Something wrong with the 1.4.5 Mac binary on CRAN with Redshift #441

ilarischeinin · 2023-09-01T13:50:14Z

I discovered an odd issue, which I thought I'd report and can help troubleshoot if needed. The tl;dr is that I cannot connect to one Redshift server with the Mac binary from CRAN, but can if I compile from source.

The current Mac binary of RPostgres (1.4.5) seems to have an issue with some, but not all, Redshift servers. The command I'm running is

DBI::dbConnect(RPostgres::Redshift(), ...) # same result with RPostgres::Postgres()

and the error I get is:

connection to server at "<host>" (<ip>), port 5439 failed: server closed the connection unexpectedly
	This probably means the server terminated abnormally
	before or while processing the request.

The error message is the same also when I don't provide a valid username/password, so it seems to happen before authentication.

The weird thing is that I get this error with only one Redshift server (our production one) and not with another (our staging) – even though they're both running the same version. Also, I can connect to both with psql from my Mac, and can connect to both with RPostgres from within a docker container running on my Mac. This made me suspect it is somehow related to the Mac version of RPostgres. And in fact, if instead of installing the binary from CRAN I compile it from source with (I have libpq installed with Homebrew):

install.packages("RPostgres", type = "source")

The problem goes away and I can connect to both servers.

I don't know if the solution is to simply recompile the Mac binary version that is available on CRAN, and that will fix it. If not, I can help troubleshoot on my end, in case that helps. The fact that this seems to affect some, but not all, Redshift servers can understandably make this hard the reproduce. Is there something I can do to see debugging messages from RPostgres/DBI?

My sessionInfo():

R version 4.2.3 (2023-03-15)
Platform: x86_64-apple-darwin17.0 (64-bit)
Running under: macOS Big Sur ... 10.16

Matrix products: default
BLAS:   /Library/Frameworks/R.framework/Versions/4.2/Resources/lib/libRblas.0.dylib
LAPACK: /Library/Frameworks/R.framework/Versions/4.2/Resources/lib/libRlapack.dylib

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

loaded via a namespace (and not attached):
 [1] bit_4.0.5       compiler_4.2.3  cli_3.6.1       hms_1.1.3      
 [5] DBI_1.1.3       Rcpp_1.0.11     bit64_4.0.5     vctrs_0.6.3    
 [9] blob_1.2.4      lifecycle_1.0.3 pkgconfig_2.0.3 rlang_1.1.1    
[13] RPostgres_1.4.5

The text was updated successfully, but these errors were encountered:

krlmlr · 2023-09-01T19:13:30Z

Thanks. Logging must be enabled when compiling, this is not an option here. This is really weird.

We could try harder to build the package so that fails to connect to Redshift the same way the CRAN build fails. What do you see with the following?

DBI::dbGetInfo(RPostgres::Postgres())
#> $driver.version
#> [1] '1.4.5'
#> 
#> $client.version
#> [1] '14.5'

^{Created on 2023-09-01 with reprex v2.0.2}

I'm a bit lost here. Logging the network communication and looking for differences? But this looks like serious overkill.

@s-u: Any idea how to troubleshoot libpq issues that happen (only) with the CRAN build?

s-u · 2023-09-02T08:35:25Z

You can check with the same binary from https://mac.r-project.org/bin/. If this is a bug in libpq then you can suggest an update to https://github.com/R-macos/recipes/blob/master/recipes/libpq

ilarischeinin · 2023-09-04T16:01:05Z

I get the same versions as you @krlmlr :

DBI::dbGetInfo(RPostgres::Postgres())
#> $driver.version
#> [1] '1.4.5'
#> 
#> $client.version
#> [1] '14.5'

^{Created on 2023-09-04 with reprex v2.0.2}

You can check with the same binary from https://mac.r-project.org/bin/.

It doesn't seem to have RPostgres available?

source("https://mac.R-project.org/bin/install.R")
#> 
#>  Usage: install.libs(names)
#> 
#>  Example: install.libs('cairo')
#> 
#>  names can be a vector or a special value 'all'.
#>  See args(install.libs) for defaults.
install.libs("RPostgres")
#> Downloading https://mac.R-project.org/bin/REPOS ...
#> Using repository  https://mac.R-project.org/bin/darwin20/x86_64 ...
#> Downloading index  https://mac.R-project.org/bin/darwin20/x86_64/PACKAGES ...
#> Error in deps(pkgs, db): Following binaries have no download candidates: RPostgres

^{Created on 2023-09-04 with reprex v2.0.2}

If this is a bug in libpq then you can suggest an update to https://github.com/R-macos/recipes/blob/master/recipes/libpq

Happy to, if that's what I should do. But to be honest I have no idea where the problem is. From the recipe you linked:

#: we only pick libpq from the entire PostgeSQL installation...

This makes me think that I do have both libpq and popstgresql installed with Homebrew, and I wonder if it could somehow be possible that in some scenario (our production Redshift) it used one, and in another scenario (our staging Redshift) the other. FWIW, here's output for brew info:

$ brew info libpq
==> libpq: stable 15.4 (bottled) [keg-only]
Postgres C API library
https://www.postgresql.org/docs/15/libpq.html
/usr/local/Cellar/libpq/15.4 (2,370 files, 28.2MB)
  Poured from bottle using the formulae.brew.sh API on 2023-08-30 at 12:31:07
From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/lib/libpq.rb
License: PostgreSQL
==> Dependencies
Required: krb5 ✔, openssl@3 ✔
==> Caveats
libpq is keg-only, which means it was not symlinked into /usr/local,
because conflicts with postgres formula.

If you need to have libpq first in your PATH, run:
  echo 'export PATH="/usr/local/opt/libpq/bin:$PATH"' >> ~/.zshrc

For compilers to find libpq you may need to set:
  export LDFLAGS="-L/usr/local/opt/libpq/lib"
  export CPPFLAGS="-I/usr/local/opt/libpq/include"
==> Analytics
install: 72,927 (30 days), 184,647 (90 days), 309,228 (365 days)
install-on-request: 38,695 (30 days), 100,106 (90 days), 156,159 (365 days)
build-error: 9 (30 days)



$ brew info postgresql
Warning: Formula postgresql was renamed to postgresql@14.
==> postgresql@14: stable 14.9 (bottled)
Object-relational database system
https://www.postgresql.org/
/usr/local/Cellar/postgresql@14/14.9 (3,316 files, 44.6MB) *
  Poured from bottle on 2023-09-04 at 18:54:09
From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/p/[email protected]
License: PostgreSQL
==> Dependencies
Build: pkg-config ✘
Required: icu4c ✔, krb5 ✔, lz4 ✔, openssl@3 ✔, readline ✔
==> Caveats
This formula has created a default database cluster with:
  initdb --locale=C -E UTF-8 /usr/local/var/postgres
For more details, read:
  https://www.postgresql.org/docs/14/app-initdb.html

To start postgresql@14 now and restart at login:
  brew services start postgresql@14
Or, if you don't want/need a background service you can just run:
  /usr/local/opt/postgresql@14/bin/postgres -D /usr/local/var/postgres
==> Analytics
install: 55,121 (30 days), 164,673 (90 days), 268,824 (365 days)
install-on-request: 53,771 (30 days), 160,989 (90 days), 262,370 (365 days)
build-error: 6 (30 days)

ilarischeinin · 2023-09-05T07:49:09Z

I did some more testing with libpq and postgresql, and found out that:

If I have both installed and compile RPostgres from source, it uses postgresql -> works
If I have only libpq, it uses that -> works
if I have only postgresql, It uses that -> works
If I have neither, compiling RPostgres from source naturally fails

Here are the compilation outputs, in case they might be useful:

# with postgresql
install.packages("RPostgres", type = "source")
#> trying URL 'https://cran.rstudio.com/src/contrib/RPostgres_1.4.5.tar.gz'
#> Content type 'application/x-gzip' length 1473682 bytes (1.4 MB)
#> ==================================================
#> downloaded 1.4 MB
#> 
#> * installing *source* package ‘RPostgres’ ...
#> ** package ‘RPostgres’ successfully unpacked and MD5 sums checked
#> ** using staged installation
#> Found pg_config includedir and libdir (PostgreSQL 14.9 (Homebrew))!
#> Using PKG_CFLAGS=-I/usr/local/include/postgresql@14
#> Using PKG_LIBS=-L/usr/local/lib/postgresql@14 -lpq
#> Using PKG_PLOGR=
#> ** libs
#> clang++ -mmacosx-version-min=10.13 -std=gnu++14 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I/usr/local/include/postgresql@14 -Ivendor -DRCPP_DEFAULT_INCLUDE_CALL=false -DRCPP_USING_UTF8_ERROR_STRING -DBOOST_NO_AUTO_PTR  -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/plogr/include' -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/Rcpp/include' -I/usr/local/include   -fPIC  -Wall -g -O2  -c DbColumn.cpp -o DbColumn.o
#> clang++ -mmacosx-version-min=10.13 -std=gnu++14 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I/usr/local/include/postgresql@14 -Ivendor -DRCPP_DEFAULT_INCLUDE_CALL=false -DRCPP_USING_UTF8_ERROR_STRING -DBOOST_NO_AUTO_PTR  -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/plogr/include' -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/Rcpp/include' -I/usr/local/include   -fPIC  -Wall -g -O2  -c DbColumnDataSource.cpp -o DbColumnDataSource.o
#> clang++ -mmacosx-version-min=10.13 -std=gnu++14 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I/usr/local/include/postgresql@14 -Ivendor -DRCPP_DEFAULT_INCLUDE_CALL=false -DRCPP_USING_UTF8_ERROR_STRING -DBOOST_NO_AUTO_PTR  -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/plogr/include' -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/Rcpp/include' -I/usr/local/include   -fPIC  -Wall -g -O2  -c DbColumnDataSourceFactory.cpp -o DbColumnDataSourceFactory.o
#> clang++ -mmacosx-version-min=10.13 -std=gnu++14 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I/usr/local/include/postgresql@14 -Ivendor -DRCPP_DEFAULT_INCLUDE_CALL=false -DRCPP_USING_UTF8_ERROR_STRING -DBOOST_NO_AUTO_PTR  -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/plogr/include' -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/Rcpp/include' -I/usr/local/include   -fPIC  -Wall -g -O2  -c DbColumnStorage.cpp -o DbColumnStorage.o
#> clang++ -mmacosx-version-min=10.13 -std=gnu++14 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I/usr/local/include/postgresql@14 -Ivendor -DRCPP_DEFAULT_INCLUDE_CALL=false -DRCPP_USING_UTF8_ERROR_STRING -DBOOST_NO_AUTO_PTR  -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/plogr/include' -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/Rcpp/include' -I/usr/local/include   -fPIC  -Wall -g -O2  -c DbConnection.cpp -o DbConnection.o
#> clang++ -mmacosx-version-min=10.13 -std=gnu++14 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I/usr/local/include/postgresql@14 -Ivendor -DRCPP_DEFAULT_INCLUDE_CALL=false -DRCPP_USING_UTF8_ERROR_STRING -DBOOST_NO_AUTO_PTR  -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/plogr/include' -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/Rcpp/include' -I/usr/local/include   -fPIC  -Wall -g -O2  -c DbDataFrame.cpp -o DbDataFrame.o
#> In file included from DbDataFrame.cpp:7:
#> vendor/boost/bind.hpp:36:1: warning: The practice of declaring the Bind placeholders (_1, _2, ...) in the global namespace is deprecated. Please use <boost/bind/bind.hpp> + using namespace boost::placeholders, or define BOOST_BIND_GLOBAL_PLACEHOLDERS to retain the current behavior. [-W#pragma-messages]
#> BOOST_PRAGMA_MESSAGE(
#> ^
#> vendor/boost/config/pragma_message.hpp:24:34: note: expanded from macro 'BOOST_PRAGMA_MESSAGE'
#> # define BOOST_PRAGMA_MESSAGE(x) _Pragma(BOOST_STRINGIZE(message(x)))
#>                                  ^
#> <scratch space>:39:2: note: expanded from here
#>  message("The practice of declaring the Bind placeholders (_1, _2, ...) " "in the global namespace is deprecated. Please use " "<boost/bind/bind.hpp> + using namespace boost::placeholders, " "or define BOOST_BIND_GLOBAL_PLACEHOLDERS to retain the current behavior.")
#>  ^
#> 1 warning generated.
#> clang++ -mmacosx-version-min=10.13 -std=gnu++14 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I/usr/local/include/postgresql@14 -Ivendor -DRCPP_DEFAULT_INCLUDE_CALL=false -DRCPP_USING_UTF8_ERROR_STRING -DBOOST_NO_AUTO_PTR  -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/plogr/include' -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/Rcpp/include' -I/usr/local/include   -fPIC  -Wall -g -O2  -c DbResult.cpp -o DbResult.o
#> clang++ -mmacosx-version-min=10.13 -std=gnu++14 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I/usr/local/include/postgresql@14 -Ivendor -DRCPP_DEFAULT_INCLUDE_CALL=false -DRCPP_USING_UTF8_ERROR_STRING -DBOOST_NO_AUTO_PTR  -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/plogr/include' -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/Rcpp/include' -I/usr/local/include   -fPIC  -Wall -g -O2  -c PqColumnDataSource.cpp -o PqColumnDataSource.o
#> clang++ -mmacosx-version-min=10.13 -std=gnu++14 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I/usr/local/include/postgresql@14 -Ivendor -DRCPP_DEFAULT_INCLUDE_CALL=false -DRCPP_USING_UTF8_ERROR_STRING -DBOOST_NO_AUTO_PTR  -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/plogr/include' -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/Rcpp/include' -I/usr/local/include   -fPIC  -Wall -g -O2  -c PqColumnDataSourceFactory.cpp -o PqColumnDataSourceFactory.o
#> clang++ -mmacosx-version-min=10.13 -std=gnu++14 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I/usr/local/include/postgresql@14 -Ivendor -DRCPP_DEFAULT_INCLUDE_CALL=false -DRCPP_USING_UTF8_ERROR_STRING -DBOOST_NO_AUTO_PTR  -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/plogr/include' -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/Rcpp/include' -I/usr/local/include   -fPIC  -Wall -g -O2  -c PqDataFrame.cpp -o PqDataFrame.o
#> clang++ -mmacosx-version-min=10.13 -std=gnu++14 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I/usr/local/include/postgresql@14 -Ivendor -DRCPP_DEFAULT_INCLUDE_CALL=false -DRCPP_USING_UTF8_ERROR_STRING -DBOOST_NO_AUTO_PTR  -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/plogr/include' -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/Rcpp/include' -I/usr/local/include   -fPIC  -Wall -g -O2  -c PqResult.cpp -o PqResult.o
#> clang++ -mmacosx-version-min=10.13 -std=gnu++14 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I/usr/local/include/postgresql@14 -Ivendor -DRCPP_DEFAULT_INCLUDE_CALL=false -DRCPP_USING_UTF8_ERROR_STRING -DBOOST_NO_AUTO_PTR  -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/plogr/include' -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/Rcpp/include' -I/usr/local/include   -fPIC  -Wall -g -O2  -c PqResultImpl.cpp -o PqResultImpl.o
#> clang++ -mmacosx-version-min=10.13 -std=gnu++14 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I/usr/local/include/postgresql@14 -Ivendor -DRCPP_DEFAULT_INCLUDE_CALL=false -DRCPP_USING_UTF8_ERROR_STRING -DBOOST_NO_AUTO_PTR  -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/plogr/include' -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/Rcpp/include' -I/usr/local/include   -fPIC  -Wall -g -O2  -c PqResultSource.cpp -o PqResultSource.o
#> clang++ -mmacosx-version-min=10.13 -std=gnu++14 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I/usr/local/include/postgresql@14 -Ivendor -DRCPP_DEFAULT_INCLUDE_CALL=false -DRCPP_USING_UTF8_ERROR_STRING -DBOOST_NO_AUTO_PTR  -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/plogr/include' -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/Rcpp/include' -I/usr/local/include   -fPIC  -Wall -g -O2  -c PqUtils.cpp -o PqUtils.o
#> clang -mmacosx-version-min=10.13 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I/usr/local/include/postgresql@14 -Ivendor -DRCPP_DEFAULT_INCLUDE_CALL=false -DRCPP_USING_UTF8_ERROR_STRING -DBOOST_NO_AUTO_PTR  -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/plogr/include' -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/Rcpp/include' -I/usr/local/include   -fPIC  -Wall -g -O2  -c RPostgres-init.c -o RPostgres-init.o
#> clang++ -mmacosx-version-min=10.13 -std=gnu++14 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I/usr/local/include/postgresql@14 -Ivendor -DRCPP_DEFAULT_INCLUDE_CALL=false -DRCPP_USING_UTF8_ERROR_STRING -DBOOST_NO_AUTO_PTR  -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/plogr/include' -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/Rcpp/include' -I/usr/local/include   -fPIC  -Wall -g -O2  -c RcppExports.cpp -o RcppExports.o
#> clang++ -mmacosx-version-min=10.13 -std=gnu++14 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I/usr/local/include/postgresql@14 -Ivendor -DRCPP_DEFAULT_INCLUDE_CALL=false -DRCPP_USING_UTF8_ERROR_STRING -DBOOST_NO_AUTO_PTR  -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/plogr/include' -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/Rcpp/include' -I/usr/local/include   -fPIC  -Wall -g -O2  -c connection.cpp -o connection.o
#> clang++ -mmacosx-version-min=10.13 -std=gnu++14 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I/usr/local/include/postgresql@14 -Ivendor -DRCPP_DEFAULT_INCLUDE_CALL=false -DRCPP_USING_UTF8_ERROR_STRING -DBOOST_NO_AUTO_PTR  -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/plogr/include' -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/Rcpp/include' -I/usr/local/include   -fPIC  -Wall -g -O2  -c encode.cpp -o encode.o
#> clang++ -mmacosx-version-min=10.13 -std=gnu++14 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I/usr/local/include/postgresql@14 -Ivendor -DRCPP_DEFAULT_INCLUDE_CALL=false -DRCPP_USING_UTF8_ERROR_STRING -DBOOST_NO_AUTO_PTR  -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/plogr/include' -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/Rcpp/include' -I/usr/local/include   -fPIC  -Wall -g -O2  -c encrypt.cpp -o encrypt.o
#> clang++ -mmacosx-version-min=10.13 -std=gnu++14 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I/usr/local/include/postgresql@14 -Ivendor -DRCPP_DEFAULT_INCLUDE_CALL=false -DRCPP_USING_UTF8_ERROR_STRING -DBOOST_NO_AUTO_PTR  -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/plogr/include' -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/Rcpp/include' -I/usr/local/include   -fPIC  -Wall -g -O2  -c logging.cpp -o logging.o
#> clang++ -mmacosx-version-min=10.13 -std=gnu++14 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I/usr/local/include/postgresql@14 -Ivendor -DRCPP_DEFAULT_INCLUDE_CALL=false -DRCPP_USING_UTF8_ERROR_STRING -DBOOST_NO_AUTO_PTR  -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/plogr/include' -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/Rcpp/include' -I/usr/local/include   -fPIC  -Wall -g -O2  -c result.cpp -o result.o
#> clang++ -mmacosx-version-min=10.13 -std=gnu++14 -dynamiclib -Wl,-headerpad_max_install_names -undefined dynamic_lookup -single_module -multiply_defined suppress -L/Library/Frameworks/R.framework/Resources/lib -L/usr/local/lib -o RPostgres.so DbColumn.o DbColumnDataSource.o DbColumnDataSourceFactory.o DbColumnStorage.o DbConnection.o DbDataFrame.o DbResult.o PqColumnDataSource.o PqColumnDataSourceFactory.o PqDataFrame.o PqResult.o PqResultImpl.o PqResultSource.o PqUtils.o RPostgres-init.o RcppExports.o connection.o encode.o encrypt.o logging.o result.o -L/usr/local/lib/postgresql@14 -lpq -F/Library/Frameworks/R.framework/.. -framework R -Wl,-framework -Wl,CoreFoundation
#> ld: warning: dylib (/usr/local/lib/postgresql@14/libpq.dylib) was built for newer macOS version (12.0) than being linked (10.13)
#> installing to /Library/Frameworks/R.framework/Versions/4.2/Resources/library/00LOCK-RPostgres/00new/RPostgres/libs
#> ** R
#> ** inst
#> ** byte-compile and prepare package for lazy loading
#> ** help
#> *** installing help indices
#> ** building package indices
#> ** installing vignettes
#> ** testing if installed package can be loaded from temporary location
#> ** checking absolute paths in shared objects and dynamic libraries
#> ** testing if installed package can be loaded from final location
#> ** testing if installed package keeps a record of temporary installation path
#> * DONE (RPostgres)
#> 
#> The downloaded source packages are in
#> 	‘/private/var/folders/np/n1zvy7cs7m9blq1w1_m_bqqw0000gn/T/RtmpTWKZBP/downloaded_packages’


# with libpq:
install.packages("RPostgres", type = "source")
#> trying URL 'https://cran.rstudio.com/src/contrib/RPostgres_1.4.5.tar.gz'
#> Content type 'application/x-gzip' length 1473682 bytes (1.4 MB)
#> ==================================================
#> downloaded 1.4 MB
#> 
#> * installing *source* package ‘RPostgres’ ...
#> ** package ‘RPostgres’ successfully unpacked and MD5 sums checked
#> ** using staged installation
#> Homebrew 4.1.7
#> Homebrew/homebrew-core (git revision c1f15f343bb; last commit 2023-09-04)
#> Homebrew/homebrew-cask (git revision 7eae8590f2; last commit 2023-09-03)
#> Using PKG_CFLAGS=-I/usr/local/opt/libpq/include
#> Using PKG_LIBS=-L/usr/local/opt/libpq/lib -lpq
#> Using PKG_PLOGR=
#> ** libs
#> clang++ -mmacosx-version-min=10.13 -std=gnu++14 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I/usr/local/opt/libpq/include -Ivendor -DRCPP_DEFAULT_INCLUDE_CALL=false -DRCPP_USING_UTF8_ERROR_STRING -DBOOST_NO_AUTO_PTR  -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/plogr/include' -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/Rcpp/include' -I/usr/local/include   -fPIC  -Wall -g -O2  -c DbColumn.cpp -o DbColumn.o
#> clang++ -mmacosx-version-min=10.13 -std=gnu++14 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I/usr/local/opt/libpq/include -Ivendor -DRCPP_DEFAULT_INCLUDE_CALL=false -DRCPP_USING_UTF8_ERROR_STRING -DBOOST_NO_AUTO_PTR  -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/plogr/include' -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/Rcpp/include' -I/usr/local/include   -fPIC  -Wall -g -O2  -c DbColumnDataSource.cpp -o DbColumnDataSource.o
#> clang++ -mmacosx-version-min=10.13 -std=gnu++14 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I/usr/local/opt/libpq/include -Ivendor -DRCPP_DEFAULT_INCLUDE_CALL=false -DRCPP_USING_UTF8_ERROR_STRING -DBOOST_NO_AUTO_PTR  -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/plogr/include' -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/Rcpp/include' -I/usr/local/include   -fPIC  -Wall -g -O2  -c DbColumnDataSourceFactory.cpp -o DbColumnDataSourceFactory.o
#> clang++ -mmacosx-version-min=10.13 -std=gnu++14 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I/usr/local/opt/libpq/include -Ivendor -DRCPP_DEFAULT_INCLUDE_CALL=false -DRCPP_USING_UTF8_ERROR_STRING -DBOOST_NO_AUTO_PTR  -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/plogr/include' -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/Rcpp/include' -I/usr/local/include   -fPIC  -Wall -g -O2  -c DbColumnStorage.cpp -o DbColumnStorage.o
#> clang++ -mmacosx-version-min=10.13 -std=gnu++14 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I/usr/local/opt/libpq/include -Ivendor -DRCPP_DEFAULT_INCLUDE_CALL=false -DRCPP_USING_UTF8_ERROR_STRING -DBOOST_NO_AUTO_PTR  -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/plogr/include' -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/Rcpp/include' -I/usr/local/include   -fPIC  -Wall -g -O2  -c DbConnection.cpp -o DbConnection.o
#> clang++ -mmacosx-version-min=10.13 -std=gnu++14 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I/usr/local/opt/libpq/include -Ivendor -DRCPP_DEFAULT_INCLUDE_CALL=false -DRCPP_USING_UTF8_ERROR_STRING -DBOOST_NO_AUTO_PTR  -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/plogr/include' -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/Rcpp/include' -I/usr/local/include   -fPIC  -Wall -g -O2  -c DbDataFrame.cpp -o DbDataFrame.o
#> In file included from DbDataFrame.cpp:7:
#> vendor/boost/bind.hpp:36:1: warning: The practice of declaring the Bind placeholders (_1, _2, ...) in the global namespace is deprecated. Please use <boost/bind/bind.hpp> + using namespace boost::placeholders, or define BOOST_BIND_GLOBAL_PLACEHOLDERS to retain the current behavior. [-W#pragma-messages]
#> BOOST_PRAGMA_MESSAGE(
#> ^
#> vendor/boost/config/pragma_message.hpp:24:34: note: expanded from macro 'BOOST_PRAGMA_MESSAGE'
#> # define BOOST_PRAGMA_MESSAGE(x) _Pragma(BOOST_STRINGIZE(message(x)))
#>                                  ^
#> <scratch space>:39:2: note: expanded from here
#>  message("The practice of declaring the Bind placeholders (_1, _2, ...) " "in the global namespace is deprecated. Please use " "<boost/bind/bind.hpp> + using namespace boost::placeholders, " "or define BOOST_BIND_GLOBAL_PLACEHOLDERS to retain the current behavior.")
#>  ^
#> 1 warning generated.
#> clang++ -mmacosx-version-min=10.13 -std=gnu++14 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I/usr/local/opt/libpq/include -Ivendor -DRCPP_DEFAULT_INCLUDE_CALL=false -DRCPP_USING_UTF8_ERROR_STRING -DBOOST_NO_AUTO_PTR  -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/plogr/include' -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/Rcpp/include' -I/usr/local/include   -fPIC  -Wall -g -O2  -c DbResult.cpp -o DbResult.o
#> clang++ -mmacosx-version-min=10.13 -std=gnu++14 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I/usr/local/opt/libpq/include -Ivendor -DRCPP_DEFAULT_INCLUDE_CALL=false -DRCPP_USING_UTF8_ERROR_STRING -DBOOST_NO_AUTO_PTR  -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/plogr/include' -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/Rcpp/include' -I/usr/local/include   -fPIC  -Wall -g -O2  -c PqColumnDataSource.cpp -o PqColumnDataSource.o
#> clang++ -mmacosx-version-min=10.13 -std=gnu++14 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I/usr/local/opt/libpq/include -Ivendor -DRCPP_DEFAULT_INCLUDE_CALL=false -DRCPP_USING_UTF8_ERROR_STRING -DBOOST_NO_AUTO_PTR  -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/plogr/include' -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/Rcpp/include' -I/usr/local/include   -fPIC  -Wall -g -O2  -c PqColumnDataSourceFactory.cpp -o PqColumnDataSourceFactory.o
#> clang++ -mmacosx-version-min=10.13 -std=gnu++14 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I/usr/local/opt/libpq/include -Ivendor -DRCPP_DEFAULT_INCLUDE_CALL=false -DRCPP_USING_UTF8_ERROR_STRING -DBOOST_NO_AUTO_PTR  -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/plogr/include' -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/Rcpp/include' -I/usr/local/include   -fPIC  -Wall -g -O2  -c PqDataFrame.cpp -o PqDataFrame.o
#> clang++ -mmacosx-version-min=10.13 -std=gnu++14 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I/usr/local/opt/libpq/include -Ivendor -DRCPP_DEFAULT_INCLUDE_CALL=false -DRCPP_USING_UTF8_ERROR_STRING -DBOOST_NO_AUTO_PTR  -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/plogr/include' -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/Rcpp/include' -I/usr/local/include   -fPIC  -Wall -g -O2  -c PqResult.cpp -o PqResult.o
#> clang++ -mmacosx-version-min=10.13 -std=gnu++14 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I/usr/local/opt/libpq/include -Ivendor -DRCPP_DEFAULT_INCLUDE_CALL=false -DRCPP_USING_UTF8_ERROR_STRING -DBOOST_NO_AUTO_PTR  -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/plogr/include' -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/Rcpp/include' -I/usr/local/include   -fPIC  -Wall -g -O2  -c PqResultImpl.cpp -o PqResultImpl.o
#> clang++ -mmacosx-version-min=10.13 -std=gnu++14 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I/usr/local/opt/libpq/include -Ivendor -DRCPP_DEFAULT_INCLUDE_CALL=false -DRCPP_USING_UTF8_ERROR_STRING -DBOOST_NO_AUTO_PTR  -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/plogr/include' -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/Rcpp/include' -I/usr/local/include   -fPIC  -Wall -g -O2  -c PqResultSource.cpp -o PqResultSource.o
#> clang++ -mmacosx-version-min=10.13 -std=gnu++14 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I/usr/local/opt/libpq/include -Ivendor -DRCPP_DEFAULT_INCLUDE_CALL=false -DRCPP_USING_UTF8_ERROR_STRING -DBOOST_NO_AUTO_PTR  -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/plogr/include' -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/Rcpp/include' -I/usr/local/include   -fPIC  -Wall -g -O2  -c PqUtils.cpp -o PqUtils.o
#> clang -mmacosx-version-min=10.13 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I/usr/local/opt/libpq/include -Ivendor -DRCPP_DEFAULT_INCLUDE_CALL=false -DRCPP_USING_UTF8_ERROR_STRING -DBOOST_NO_AUTO_PTR  -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/plogr/include' -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/Rcpp/include' -I/usr/local/include   -fPIC  -Wall -g -O2  -c RPostgres-init.c -o RPostgres-init.o
#> clang++ -mmacosx-version-min=10.13 -std=gnu++14 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I/usr/local/opt/libpq/include -Ivendor -DRCPP_DEFAULT_INCLUDE_CALL=false -DRCPP_USING_UTF8_ERROR_STRING -DBOOST_NO_AUTO_PTR  -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/plogr/include' -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/Rcpp/include' -I/usr/local/include   -fPIC  -Wall -g -O2  -c RcppExports.cpp -o RcppExports.o
#> clang++ -mmacosx-version-min=10.13 -std=gnu++14 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I/usr/local/opt/libpq/include -Ivendor -DRCPP_DEFAULT_INCLUDE_CALL=false -DRCPP_USING_UTF8_ERROR_STRING -DBOOST_NO_AUTO_PTR  -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/plogr/include' -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/Rcpp/include' -I/usr/local/include   -fPIC  -Wall -g -O2  -c connection.cpp -o connection.o
#> clang++ -mmacosx-version-min=10.13 -std=gnu++14 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I/usr/local/opt/libpq/include -Ivendor -DRCPP_DEFAULT_INCLUDE_CALL=false -DRCPP_USING_UTF8_ERROR_STRING -DBOOST_NO_AUTO_PTR  -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/plogr/include' -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/Rcpp/include' -I/usr/local/include   -fPIC  -Wall -g -O2  -c encode.cpp -o encode.o
#> clang++ -mmacosx-version-min=10.13 -std=gnu++14 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I/usr/local/opt/libpq/include -Ivendor -DRCPP_DEFAULT_INCLUDE_CALL=false -DRCPP_USING_UTF8_ERROR_STRING -DBOOST_NO_AUTO_PTR  -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/plogr/include' -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/Rcpp/include' -I/usr/local/include   -fPIC  -Wall -g -O2  -c encrypt.cpp -o encrypt.o
#> clang++ -mmacosx-version-min=10.13 -std=gnu++14 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I/usr/local/opt/libpq/include -Ivendor -DRCPP_DEFAULT_INCLUDE_CALL=false -DRCPP_USING_UTF8_ERROR_STRING -DBOOST_NO_AUTO_PTR  -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/plogr/include' -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/Rcpp/include' -I/usr/local/include   -fPIC  -Wall -g -O2  -c logging.cpp -o logging.o
#> clang++ -mmacosx-version-min=10.13 -std=gnu++14 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I/usr/local/opt/libpq/include -Ivendor -DRCPP_DEFAULT_INCLUDE_CALL=false -DRCPP_USING_UTF8_ERROR_STRING -DBOOST_NO_AUTO_PTR  -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/plogr/include' -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/Rcpp/include' -I/usr/local/include   -fPIC  -Wall -g -O2  -c result.cpp -o result.o
#> clang++ -mmacosx-version-min=10.13 -std=gnu++14 -dynamiclib -Wl,-headerpad_max_install_names -undefined dynamic_lookup -single_module -multiply_defined suppress -L/Library/Frameworks/R.framework/Resources/lib -L/usr/local/lib -o RPostgres.so DbColumn.o DbColumnDataSource.o DbColumnDataSourceFactory.o DbColumnStorage.o DbConnection.o DbDataFrame.o DbResult.o PqColumnDataSource.o PqColumnDataSourceFactory.o PqDataFrame.o PqResult.o PqResultImpl.o PqResultSource.o PqUtils.o RPostgres-init.o RcppExports.o connection.o encode.o encrypt.o logging.o result.o -L/usr/local/opt/libpq/lib -lpq -F/Library/Frameworks/R.framework/.. -framework R -Wl,-framework -Wl,CoreFoundation
#> ld: warning: dylib (/usr/local/opt/libpq/lib/libpq.dylib) was built for newer macOS version (12.0) than being linked (10.13)
#> installing to /Library/Frameworks/R.framework/Versions/4.2/Resources/library/00LOCK-RPostgres/00new/RPostgres/libs
#> ** R
#> ** inst
#> ** byte-compile and prepare package for lazy loading
#> ** help
#> *** installing help indices
#> ** building package indices
#> ** installing vignettes
#> ** testing if installed package can be loaded from temporary location
#> ** checking absolute paths in shared objects and dynamic libraries
#> ** testing if installed package can be loaded from final location
#> ** testing if installed package keeps a record of temporary installation path
#> * DONE (RPostgres)
#> 
#> The downloaded source packages are in
#> 	‘/private/var/folders/np/n1zvy7cs7m9blq1w1_m_bqqw0000gn/T/RtmpcjWwNf/downloaded_packages’

krlmlr · 2024-04-01T15:21:47Z

Is this still a problem with the most recent CRAN binary?

krlmlr · 2024-04-01T17:12:50Z

A duplicate of #404 perhaps?

ilarischeinin · 2024-04-02T08:42:47Z

Is this still a problem with the most recent CRAN binary?

It is.

Error: connection to server at "<host>" (<ip>), port 5439 failed: server closed the connection unexpectedly
	This probably means the server terminated abnormally
	before or while processing the request.

I'm currently on macOS 14.4.1 and R 4.3.3, and it remains to be the case that the binary version (now 1.4.6) of RPostgres installed from CRAN does not work with our Redshift. When I compile RPostgres from source, it does work. (I have postgresql@14 version 14.11 installed via Homebrew, which is used when compiling.)

I'm on an Intel Mac, whereas the linked issue seemed to be specific to Apple Silicon, so I think this is something different.

While the problem is easy enough to fix for me by compiling form source, I recently had a colleague run into the same issue, and they were unable to compile from source on an Apple Silicon Mac. I didn't look into that in more detail yet, as I will be upgrading from Intel later this month. I will then see if I can get to a similar fix on Apple Silicon too, but of course it would be ideal if the CRAN binary worked out of the box.

One thing that has changed since originally reporting this is that it now affects both our Redshift instances (staging and production). So something (like an update, cert change, …) must have happened on the AWS side to the staging instance, as that used to work.

Something perhaps worth repeating is that the error happens before authentication, as the error message is the same regardless of whether I'm using valid username/password or not. (And is also different from the one I get with invalid credentials when using the compiled-from-source version of the package.)

krlmlr · 2024-04-13T09:29:02Z

I now understood Simon's comment.

Can you please try compiling with libpq from https://mac.r-project.org/bin/darwin20/x86_64/ (or whatever subdirectory is appropriate for your platform)? If we need a newer/different libpq, we could send a pull request to https://github.com/R-macos/recipes/blob/master/recipes/libpq .

ilarischeinin · 2024-04-16T08:36:19Z

OK, I:

removed any libpq and postgresql installations I had via brew
downloaded libpq-14.5-darwin.20-x86_64.tar.xz from the above link
placed the files inside the corresponding directories under /opt
compiled RPostgres from source with install.packages("RPostgres", type="source")

And as a result I am running into the issue described above. So I take it the hunch is correct there that libpq version used needs to be bumped from the current 14.5.

One thing I am confused by is that the compilation output mentions "autobrew bundle version 15.2":

> install.packages("RPostgres", type="source")
trying URL 'https://cran.rstudio.com/src/contrib/RPostgres_1.4.6.tar.gz'
Content type 'application/x-gzip' length 1474867 bytes (1.4 MB)
==================================================
downloaded 1.4 MB

* installing *source* package ‘RPostgres’ ...
** package ‘RPostgres’ successfully unpacked and MD5 sums checked
** using staged installation
Homebrew 4.2.18
Using autobrew bundle: libpq-15.2-universal.tar.xz

But the entire output doesn't make it clear to me if it's actually using the 14.5 I installed manually, or the 15.2 it might download as part of the compilation. There are lines that point to directories where I put the manual installation, e.g.

clude' -I/opt/R/x86_64/include    -fPIC  -falign-functions=64 -Wall -g -O2  -c DbConnection.cpp -o DbConnection.o

So my questions:

Could the source of the problem be the discrepancy (14.5 vs. 15.2), rather than simply an outdated version?
If I submit a PR to bump the version from the current 14.5, should I go for the 15.2 mentioned related to autobrew, or to 16.2, which looks to be the current both for brew and source? (And if 16.2, should something be done to also get the autobrew from 15.2 to 16.2?)

krlmlr · 2024-04-18T16:35:44Z

I believe the autobrew source is here: https://github.com/autobrew/scripts/blob/master/libpq . In some cases (High Sierra?), we still seem to be using 14.5 here. The autobrew mechanism can be disabled by setting the DISABLE_AUTOBREW environment variable.

The library used should be reported with DBI::dbGetInfo(RPostgres::Postgres()) . What are you seeing? I understand it doesn't work with 14.5, does it work for you with 15.2? I understand it works for you with 16.2, can you confirm?

@jeroen: Can/should we bump to 16.2 here? Would that work for the High Sierra case?

jeroen · 2024-04-18T22:28:55Z

High-sierra is no longer in use on CRAN or GHA or anywhere else so we don't need to worry about that.

I have updated the script to libpq 16.0 (for which big-sur binaries were already available): autobrew/scripts@ea29fea.

However, the problem is that the CRAN macbuilders now have another version of libpq 14.5 preinstalled on the server (via the recipes system) and therefore autobrew is not used. I think you can fix this by increasing to --atleast-version=15 in the configure script here:

RPostgres/configure

Lines 21 to 22 in 2b2081c

    
           if [ "`uname`" = Darwin ] && [ "`command -v pkg-config`" ]; then 
        
             if pkg-config --atleast-version=12 libpq; then

Fixes #441

jeroen · 2024-04-19T09:20:23Z

@ilarischeinin could you test the binary package from r-universe? If that one works, it is very likely that the next CRAN release will also work for you:

install.packages("RPostgres", repos = c("https://r-dbi.r-universe.dev", "https://cran.r-project.org"))

krlmlr · 2024-04-19T17:17:23Z

Let's keep it open for now until the fixes are confirmed.

ilarischeinin · 2024-04-22T12:11:49Z

Thanks both!

could you test the binary package from r-universe?

Unfortunately it does not work. However, I'm not sure if I've somehow messed up my system when I did this earlier:

downloaded libpq-14.5-darwin.20-x86_64.tar.xz from the above link

placed the files inside the corresponding directories under /opt

As now I can't get compiling from source to produce a working version either. I did delete the files I manually copied to /opt, so not sure what is going on. But as the source code on CRAN is exactly the same as before, I feel like the issue must be on my end. The only other change is the one Jeroen made to the autobrew script, the effect of which I do see in the output:

Using autobrew bundle: libpq-16.0-universal.tar.xz

Whereas earlier it was 15.2 as mentioned above. If I set DISABLE_AUTOBREW=true, then compiling from source fails, even though I have postgresql installed via brew. It can't find libpq whereas this used to work, and pg_config is indeed in my PATH.

If libpq is already installed, check that either:
(i)  'pkg-config' is in your PATH AND PKG_CONFIG_PATH contains
     a libpq.pc file; or
(ii) 'pg_config' is in your PATH.
If neither can detect , you can set INCLUDE_DIR
and LIB_DIR manually via:
R CMD INSTALL --configure-vars='INCLUDE_DIR=... LIB_DIR=...'
--------------------------[ ERROR MESSAGE ]----------------------------
<stdin>:1:10: fatal error: 'libpq-fe.h' file not found
#include <libpq-fe.h>
         ^~~~~~~~~~~~
1 error generated.
-----------------------------------------------------------------------
ERROR: configuration failed for package ‘RPostgres’

This is becoming a real head-scratcher for me. One reason being the extra complexity brought in by the autobrew setup. Whereas I said that earlier compiling from source did work with postgresql installed via brew, now I wonder if that was actually some combination of that plus something downloaded via autobrew.

There is still a way I can get RPostgres compiled so it works with our Redshift, and it's like this: When I install libpq with brew, I see it says this:

libpq is keg-only, which means it was not symlinked into /usr/local,
because conflicts with postgres formula.

If you need to have libpq first in your PATH, run:
  echo 'export PATH="/usr/local/opt/libpq/bin:$PATH"' >> ~/.zshrc

For compilers to find libpq you may need to set:
  export LDFLAGS="-L/usr/local/opt/libpq/lib"
  export CPPFLAGS="-I/usr/local/opt/libpq/include"

For pkg-config to find libpq you may need to set:
  export PKG_CONFIG_PATH="/usr/local/opt/libpq/lib/pkgconfig"

If I include those exports, as well as the DISABLE_AUTOBREW=true one, then I can get RPostgres to compile from source and again working with my Redshift.

jeroen · 2024-04-22T12:23:03Z

But can you test the binary package? You should not have to compile anything. Which build of R do you use? Do you use an official R for MacOS from CRAN or the homebrew version of R?

ilarischeinin · 2024-04-22T12:27:07Z

The working version compiled from source against libpq from brew has

> DBI::dbGetInfo(RPostgres::Postgres())
$driver.version
[1] ‘1.4.6’

$client.version
[1] ‘16.2’

Whereas the one that uses autobrew (and doesn't work) has:

DBI::dbGetInfo(RPostgres::Postgres())
$driver.version
[1] ‘1.4.6’

$client.version
[1] ‘16.0’

But can you test the binary package?

That one doesn't work. That also has

> DBI::dbGetInfo(RPostgres::Postgres())
$driver.version
[1] ‘1.4.6.9007’

$client.version
[1] ‘16.0’

Perhaps autobrew should have 16.2 instead of 16.0?

ilarischeinin · 2024-04-22T12:27:43Z

Do you use an official R for MacOS from CRAN or the homebrew version of R?

The former. Version 4.3.3.

jeroen · 2024-04-22T12:40:07Z

Perhaps autobrew should have 16.2 instead of 16.0?

autobrew literally just has a copy of the last binary in homebrew when they still supported macos-11 (big-sur) a few months ago. I doubt that this is the issue, unless something was fixed very very recently...

ilarischeinin · 2024-04-22T12:59:30Z

I'm on Sonoma, so should I be seeing that "Using autobrew" line at all?

jeroen · 2024-04-22T13:04:19Z

Well not if you use the binary packages. If you install from source, and no suitable libpq is found locally, it will get libpq from autobrew indeed.

ilarischeinin · 2024-04-22T13:13:29Z

Right now

binary from CRAN has this issue
binary from r-universe has this issue
"regular" compile from source has this issue
installing libpq 16.2 with brew, turning off autobrew, and compiling from source does not have this issue

I'm trying to go up that list getting more paths working. While fixing 1. would of course be the best, even getting 3. to work would be nice. That's why I'm wondering if bumping the version in autobrew from 16.0 to 16.2 would help with that.

ilarischeinin · 2024-04-22T13:55:43Z

Collecting from the various messages above

14.5: not working
14.9: working
15.2: not working
15.4: working
16.0: not working
16.2: working

I'm not 100% sure that's causation there, or if there might be other moving parts too that are the real cause. Still, this makes me think if this might be something that was fixed separately in the 14.x/15.x/16.x major version paths in some x. And that's why bumping to e.g. 16.0 does not help.

ilarischeinin · 2024-04-25T06:26:12Z

@ilarischeinin could you test the binary package from r-universe? If that one works, it is very likely that the next CRAN release will also work for you:
install.packages("RPostgres", repos = c("https://r-dbi.r-universe.dev", "https://cran.r-project.org"))

I asked a colleague who's on Apple Silicon to test this too (I'm still on Intel). The outcome was the same:

this binary from r-universe.dev does not fix the issue
installing libpq 16.2 with brew and compiling against that does fix it

jeroen · 2024-04-25T07:55:35Z

OK I'll try to build a static version of 16.2, so that we can rule that out.

Another moving piece may be the gss implementation. We can link against the one included with MacOS, and another one that is in homebrew. We use the one from MacOS because krb5 in macos does not have static libs, but there is a comment in homebrew that suggests it may not always work well.

jeroen · 2024-04-25T08:39:32Z

OK I have updated the binaries on https://r-dbi.r-universe.dev/RPostgres to 16.2. Could you test if this changes anything?

install.packages("RPostgres", repos = "https://r-dbi.r-universe.dev")

# Confirm that it is indeed 16.2
library(RPostgres)
DBI::dbGetInfo(RPostgres::Postgres())

ilarischeinin · 2024-04-25T09:08:23Z

Thank you! But it does not! 😭

Still the same error

Error: connection to server at "<host>" (<ip>), port 5439 failed: server closed the connection unexpectedly
	This probably means the server terminated abnormally
	before or while processing the request.

And can confirm it is indeed 16.2:

DBI::dbGetInfo(RPostgres::Postgres())
# $driver.version
# [1] ‘1.4.6.9008’
# 
# $client.version
# [1] ‘16.2’

So at least we know this is not about the version.

That krb5 thing does sound really interesting, especially considering that at first, the problem was only with one of our two Redshift servers (staging worked while production did not), even though they were running the same Redshift version. And then some time later this started to affect both. I wonder if that could be because at first they had somehow different types of certs (production had renewed while staging had not), and then later as staging renewed as well, it also started to be affected with this issue.

With your better understanding on how those pre-compiled versions are generated vs. how it might happen when I compile locally, can you see a relevant difference between the two? Anything I could test, or anything I could do to increase logging to surface something more useful as for the error?

ilarischeinin · 2024-04-25T09:09:44Z

FWIW, here's the full output from manual compilation, which results in a working binary.

> install.packages("RPostgres", type="source")
trying URL 'https://cran.rstudio.com/src/contrib/RPostgres_1.4.6.tar.gz'
Content type 'application/x-gzip' length 1474867 bytes (1.4 MB)
==================================================
downloaded 1.4 MB

* installing *source* package ‘RPostgres’ ...
** package ‘RPostgres’ successfully unpacked and MD5 sums checked
** using staged installation
Local libpq has SSL support
Found pkg-config cflags and libs (libpq 16.2)!
Using PKG_CFLAGS=-I/usr/local/opt/libpq/include -I/usr/local/Cellar/openssl@3/3.3.0/include
Using PKG_LIBS=-L/usr/local/opt/libpq/lib -lpq
Using PKG_PLOGR=
** libs
using C compiler: ‘Apple clang version 15.0.0 (clang-1500.3.9.4)’
using C++ compiler: ‘Apple clang version 15.0.0 (clang-1500.3.9.4)’
using SDK: ‘’
clang++ -arch x86_64 -std=gnu++17 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I/usr/local/opt/libpq/include -I/usr/local/Cellar/openssl@3/3.3.0/include -Ivendor -DRCPP_DEFAULT_INCLUDE_CALL=false -DRCPP_USING_UTF8_ERROR_STRING -DBOOST_NO_AUTO_PTR  -I'/Library/Frameworks/R.framework/Versions/4.3-x86_64/Resources/library/cpp11/include' -I'/Library/Frameworks/R.framework/Versions/4.3-x86_64/Resources/library/plogr/include' -I/opt/R/x86_64/include    -fPIC  -falign-functions=64 -Wall -g -O2  -c DbColumn.cpp -o DbColumn.o
In file included from DbColumn.cpp:2:
In file included from ./DbColumn.h:8:
In file included from vendor/boost/shared_ptr.hpp:17:
In file included from vendor/boost/smart_ptr/shared_ptr.hpp:17:
In file included from vendor/boost/smart_ptr/detail/shared_count.hpp:25:
In file included from vendor/boost/smart_ptr/bad_weak_ptr.hpp:20:
In file included from vendor/boost/config.hpp:48:
vendor/boost/config/stdlib/libcpp.hpp:98:11: warning: 
      'BOOST_NO_AUTO_PTR' macro redefined [-Wmacro-redefined]
#  define BOOST_NO_AUTO_PTR
          ^
<command line>:4:9: note: previous definition is here
#define BOOST_NO_AUTO_PTR 1
        ^
1 warning generated.
clang++ -arch x86_64 -std=gnu++17 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I/usr/local/opt/libpq/include -I/usr/local/Cellar/openssl@3/3.3.0/include -Ivendor -DRCPP_DEFAULT_INCLUDE_CALL=false -DRCPP_USING_UTF8_ERROR_STRING -DBOOST_NO_AUTO_PTR  -I'/Library/Frameworks/R.framework/Versions/4.3-x86_64/Resources/library/cpp11/include' -I'/Library/Frameworks/R.framework/Versions/4.3-x86_64/Resources/library/plogr/include' -I/opt/R/x86_64/include    -fPIC  -falign-functions=64 -Wall -g -O2  -c DbColumnDataSource.cpp -o DbColumnDataSource.o
clang++ -arch x86_64 -std=gnu++17 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I/usr/local/opt/libpq/include -I/usr/local/Cellar/openssl@3/3.3.0/include -Ivendor -DRCPP_DEFAULT_INCLUDE_CALL=false -DRCPP_USING_UTF8_ERROR_STRING -DBOOST_NO_AUTO_PTR  -I'/Library/Frameworks/R.framework/Versions/4.3-x86_64/Resources/library/cpp11/include' -I'/Library/Frameworks/R.framework/Versions/4.3-x86_64/Resources/library/plogr/include' -I/opt/R/x86_64/include    -fPIC  -falign-functions=64 -Wall -g -O2  -c DbColumnDataSourceFactory.cpp -o DbColumnDataSourceFactory.o
clang++ -arch x86_64 -std=gnu++17 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I/usr/local/opt/libpq/include -I/usr/local/Cellar/openssl@3/3.3.0/include -Ivendor -DRCPP_DEFAULT_INCLUDE_CALL=false -DRCPP_USING_UTF8_ERROR_STRING -DBOOST_NO_AUTO_PTR  -I'/Library/Frameworks/R.framework/Versions/4.3-x86_64/Resources/library/cpp11/include' -I'/Library/Frameworks/R.framework/Versions/4.3-x86_64/Resources/library/plogr/include' -I/opt/R/x86_64/include    -fPIC  -falign-functions=64 -Wall -g -O2  -c DbColumnStorage.cpp -o DbColumnStorage.o
clang++ -arch x86_64 -std=gnu++17 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I/usr/local/opt/libpq/include -I/usr/local/Cellar/openssl@3/3.3.0/include -Ivendor -DRCPP_DEFAULT_INCLUDE_CALL=false -DRCPP_USING_UTF8_ERROR_STRING -DBOOST_NO_AUTO_PTR  -I'/Library/Frameworks/R.framework/Versions/4.3-x86_64/Resources/library/cpp11/include' -I'/Library/Frameworks/R.framework/Versions/4.3-x86_64/Resources/library/plogr/include' -I/opt/R/x86_64/include    -fPIC  -falign-functions=64 -Wall -g -O2  -c DbConnection.cpp -o DbConnection.o
In file included from DbConnection.cpp:2:
In file included from ./DbConnection.h:4:
In file included from vendor/boost/noncopyable.hpp:15:
In file included from vendor/boost/core/noncopyable.hpp:12:
In file included from vendor/boost/config.hpp:48:
vendor/boost/config/stdlib/libcpp.hpp:98:11: warning: 
      'BOOST_NO_AUTO_PTR' macro redefined [-Wmacro-redefined]
#  define BOOST_NO_AUTO_PTR
          ^
<command line>:4:9: note: previous definition is here
#define BOOST_NO_AUTO_PTR 1
        ^
1 warning generated.
clang++ -arch x86_64 -std=gnu++17 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I/usr/local/opt/libpq/include -I/usr/local/Cellar/openssl@3/3.3.0/include -Ivendor -DRCPP_DEFAULT_INCLUDE_CALL=false -DRCPP_USING_UTF8_ERROR_STRING -DBOOST_NO_AUTO_PTR  -I'/Library/Frameworks/R.framework/Versions/4.3-x86_64/Resources/library/cpp11/include' -I'/Library/Frameworks/R.framework/Versions/4.3-x86_64/Resources/library/plogr/include' -I/opt/R/x86_64/include    -fPIC  -falign-functions=64 -Wall -g -O2  -c DbDataFrame.cpp -o DbDataFrame.o
In file included from DbDataFrame.cpp:2:
In file included from ./DbDataFrame.h:4:
In file included from vendor/boost/container/stable_vector.hpp:23:
In file included from vendor/boost/config.hpp:48:
vendor/boost/config/stdlib/libcpp.hpp:98:11: warning: 
      'BOOST_NO_AUTO_PTR' macro redefined [-Wmacro-redefined]
#  define BOOST_NO_AUTO_PTR
          ^
<command line>:4:9: note: previous definition is here
#define BOOST_NO_AUTO_PTR 1
        ^
In file included from DbDataFrame.cpp:2:
In file included from ./DbDataFrame.h:4:
In file included from vendor/boost/container/stable_vector.hpp:42:
In file included from vendor/boost/container/detail/allocator_version_traits.hpp:27:
In file included from vendor/boost/container/detail/multiallocation_chain.hpp:32:
vendor/boost/intrusive/slist.hpp:1939:14: warning: 
      variable 'node_count' set but not used [-Wunused-but-set-variable]
      size_t node_count = 0;
             ^
In file included from DbDataFrame.cpp:7:
vendor/boost/bind.hpp:36:1: warning: The practice of
      declaring the Bind placeholders (_1, _2, ...) in the global namespace is
      deprecated. Please use <boost/bind/bind.hpp> + using namespace
      boost::placeholders, or define BOOST_BIND_GLOBAL_PLACEHOLDERS to retain
      the current behavior. [-W#pragma-messages]
BOOST_PRAGMA_MESSAGE(
^
vendor/boost/config/pragma_message.hpp:24:34: note: 
      expanded from macro 'BOOST_PRAGMA_MESSAGE'
# define BOOST_PRAGMA_MESSAGE(x) _Pragma(BOOST_STRINGIZE(message(x)))
                                 ^
<scratch space>:3:2: note: expanded from here
 message("The practice of declaring the Bind placeholders (_1, _2, ...) ...
 ^
3 warnings generated.
clang++ -arch x86_64 -std=gnu++17 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I/usr/local/opt/libpq/include -I/usr/local/Cellar/openssl@3/3.3.0/include -Ivendor -DRCPP_DEFAULT_INCLUDE_CALL=false -DRCPP_USING_UTF8_ERROR_STRING -DBOOST_NO_AUTO_PTR  -I'/Library/Frameworks/R.framework/Versions/4.3-x86_64/Resources/library/cpp11/include' -I'/Library/Frameworks/R.framework/Versions/4.3-x86_64/Resources/library/plogr/include' -I/opt/R/x86_64/include    -fPIC  -falign-functions=64 -Wall -g -O2  -c DbResult.cpp -o DbResult.o
In file included from DbResult.cpp:2:
In file included from ./DbResult.h:4:
In file included from vendor/boost/noncopyable.hpp:15:
In file included from vendor/boost/core/noncopyable.hpp:12:
In file included from vendor/boost/config.hpp:48:
vendor/boost/config/stdlib/libcpp.hpp:98:11: warning: 
      'BOOST_NO_AUTO_PTR' macro redefined [-Wmacro-redefined]
#  define BOOST_NO_AUTO_PTR
          ^
<command line>:4:9: note: previous definition is here
#define BOOST_NO_AUTO_PTR 1
        ^
1 warning generated.
clang++ -arch x86_64 -std=gnu++17 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I/usr/local/opt/libpq/include -I/usr/local/Cellar/openssl@3/3.3.0/include -Ivendor -DRCPP_DEFAULT_INCLUDE_CALL=false -DRCPP_USING_UTF8_ERROR_STRING -DBOOST_NO_AUTO_PTR  -I'/Library/Frameworks/R.framework/Versions/4.3-x86_64/Resources/library/cpp11/include' -I'/Library/Frameworks/R.framework/Versions/4.3-x86_64/Resources/library/plogr/include' -I/opt/R/x86_64/include    -fPIC  -falign-functions=64 -Wall -g -O2  -c PqColumnDataSource.cpp -o PqColumnDataSource.o
In file included from PqColumnDataSource.cpp:2:
In file included from vendor/boost/lexical_cast.hpp:21:
In file included from vendor/boost/config.hpp:48:
vendor/boost/config/stdlib/libcpp.hpp:98:11: warning: 
      'BOOST_NO_AUTO_PTR' macro redefined [-Wmacro-redefined]
#  define BOOST_NO_AUTO_PTR
          ^
<command line>:4:9: note: previous definition is here
#define BOOST_NO_AUTO_PTR 1
        ^
1 warning generated.
clang++ -arch x86_64 -std=gnu++17 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I/usr/local/opt/libpq/include -I/usr/local/Cellar/openssl@3/3.3.0/include -Ivendor -DRCPP_DEFAULT_INCLUDE_CALL=false -DRCPP_USING_UTF8_ERROR_STRING -DBOOST_NO_AUTO_PTR  -I'/Library/Frameworks/R.framework/Versions/4.3-x86_64/Resources/library/cpp11/include' -I'/Library/Frameworks/R.framework/Versions/4.3-x86_64/Resources/library/plogr/include' -I/opt/R/x86_64/include    -fPIC  -falign-functions=64 -Wall -g -O2  -c PqColumnDataSourceFactory.cpp -o PqColumnDataSourceFactory.o
clang++ -arch x86_64 -std=gnu++17 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I/usr/local/opt/libpq/include -I/usr/local/Cellar/openssl@3/3.3.0/include -Ivendor -DRCPP_DEFAULT_INCLUDE_CALL=false -DRCPP_USING_UTF8_ERROR_STRING -DBOOST_NO_AUTO_PTR  -I'/Library/Frameworks/R.framework/Versions/4.3-x86_64/Resources/library/cpp11/include' -I'/Library/Frameworks/R.framework/Versions/4.3-x86_64/Resources/library/plogr/include' -I/opt/R/x86_64/include    -fPIC  -falign-functions=64 -Wall -g -O2  -c PqDataFrame.cpp -o PqDataFrame.o
In file included from PqDataFrame.cpp:2:
In file included from ./PqDataFrame.h:4:
In file included from ./DbDataFrame.h:4:
In file included from vendor/boost/container/stable_vector.hpp:23:
In file included from vendor/boost/config.hpp:48:
vendor/boost/config/stdlib/libcpp.hpp:98:11: warning: 
      'BOOST_NO_AUTO_PTR' macro redefined [-Wmacro-redefined]
#  define BOOST_NO_AUTO_PTR
          ^
<command line>:4:9: note: previous definition is here
#define BOOST_NO_AUTO_PTR 1
        ^
In file included from PqDataFrame.cpp:2:
In file included from ./PqDataFrame.h:4:
In file included from ./DbDataFrame.h:4:
In file included from vendor/boost/container/stable_vector.hpp:42:
In file included from vendor/boost/container/detail/allocator_version_traits.hpp:27:
In file included from vendor/boost/container/detail/multiallocation_chain.hpp:32:
vendor/boost/intrusive/slist.hpp:1939:14: warning: 
      variable 'node_count' set but not used [-Wunused-but-set-variable]
      size_t node_count = 0;
             ^
2 warnings generated.
clang++ -arch x86_64 -std=gnu++17 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I/usr/local/opt/libpq/include -I/usr/local/Cellar/openssl@3/3.3.0/include -Ivendor -DRCPP_DEFAULT_INCLUDE_CALL=false -DRCPP_USING_UTF8_ERROR_STRING -DBOOST_NO_AUTO_PTR  -I'/Library/Frameworks/R.framework/Versions/4.3-x86_64/Resources/library/cpp11/include' -I'/Library/Frameworks/R.framework/Versions/4.3-x86_64/Resources/library/plogr/include' -I/opt/R/x86_64/include    -fPIC  -falign-functions=64 -Wall -g -O2  -c PqResult.cpp -o PqResult.o
In file included from PqResult.cpp:2:
In file included from ./PqResult.h:4:
In file included from ./DbResult.h:4:
In file included from vendor/boost/noncopyable.hpp:15:
In file included from vendor/boost/core/noncopyable.hpp:12:
In file included from vendor/boost/config.hpp:48:
vendor/boost/config/stdlib/libcpp.hpp:98:11: warning: 
      'BOOST_NO_AUTO_PTR' macro redefined [-Wmacro-redefined]
#  define BOOST_NO_AUTO_PTR
          ^
<command line>:4:9: note: previous definition is here
#define BOOST_NO_AUTO_PTR 1
        ^
1 warning generated.
clang++ -arch x86_64 -std=gnu++17 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I/usr/local/opt/libpq/include -I/usr/local/Cellar/openssl@3/3.3.0/include -Ivendor -DRCPP_DEFAULT_INCLUDE_CALL=false -DRCPP_USING_UTF8_ERROR_STRING -DBOOST_NO_AUTO_PTR  -I'/Library/Frameworks/R.framework/Versions/4.3-x86_64/Resources/library/cpp11/include' -I'/Library/Frameworks/R.framework/Versions/4.3-x86_64/Resources/library/plogr/include' -I/opt/R/x86_64/include    -fPIC  -falign-functions=64 -Wall -g -O2  -c PqResultImpl.cpp -o PqResultImpl.o
In file included from PqResultImpl.cpp:2:
In file included from ./PqResultImpl.h:4:
In file included from vendor/boost/noncopyable.hpp:15:
In file included from vendor/boost/core/noncopyable.hpp:12:
In file included from vendor/boost/config.hpp:48:
vendor/boost/config/stdlib/libcpp.hpp:98:11: warning: 
      'BOOST_NO_AUTO_PTR' macro redefined [-Wmacro-redefined]
#  define BOOST_NO_AUTO_PTR
          ^
<command line>:4:9: note: previous definition is here
#define BOOST_NO_AUTO_PTR 1
        ^
In file included from PqResultImpl.cpp:6:
In file included from ./PqDataFrame.h:4:
In file included from ./DbDataFrame.h:4:
In file included from vendor/boost/container/stable_vector.hpp:42:
In file included from vendor/boost/container/detail/allocator_version_traits.hpp:27:
In file included from vendor/boost/container/detail/multiallocation_chain.hpp:32:
vendor/boost/intrusive/slist.hpp:1939:14: warning: 
      variable 'node_count' set but not used [-Wunused-but-set-variable]
      size_t node_count = 0;
             ^
2 warnings generated.
clang++ -arch x86_64 -std=gnu++17 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I/usr/local/opt/libpq/include -I/usr/local/Cellar/openssl@3/3.3.0/include -Ivendor -DRCPP_DEFAULT_INCLUDE_CALL=false -DRCPP_USING_UTF8_ERROR_STRING -DBOOST_NO_AUTO_PTR  -I'/Library/Frameworks/R.framework/Versions/4.3-x86_64/Resources/library/cpp11/include' -I'/Library/Frameworks/R.framework/Versions/4.3-x86_64/Resources/library/plogr/include' -I/opt/R/x86_64/include    -fPIC  -falign-functions=64 -Wall -g -O2  -c PqResultSource.cpp -o PqResultSource.o
clang++ -arch x86_64 -std=gnu++17 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I/usr/local/opt/libpq/include -I/usr/local/Cellar/openssl@3/3.3.0/include -Ivendor -DRCPP_DEFAULT_INCLUDE_CALL=false -DRCPP_USING_UTF8_ERROR_STRING -DBOOST_NO_AUTO_PTR  -I'/Library/Frameworks/R.framework/Versions/4.3-x86_64/Resources/library/cpp11/include' -I'/Library/Frameworks/R.framework/Versions/4.3-x86_64/Resources/library/plogr/include' -I/opt/R/x86_64/include    -fPIC  -falign-functions=64 -Wall -g -O2  -c PqUtils.cpp -o PqUtils.o
clang -arch x86_64 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I/usr/local/opt/libpq/include -I/usr/local/Cellar/openssl@3/3.3.0/include -Ivendor -DRCPP_DEFAULT_INCLUDE_CALL=false -DRCPP_USING_UTF8_ERROR_STRING -DBOOST_NO_AUTO_PTR  -I'/Library/Frameworks/R.framework/Versions/4.3-x86_64/Resources/library/cpp11/include' -I'/Library/Frameworks/R.framework/Versions/4.3-x86_64/Resources/library/plogr/include' -I/opt/R/x86_64/include    -fPIC  -falign-functions=64 -Wall -g -O2  -c RPostgres-init.c -o RPostgres-init.o
clang++ -arch x86_64 -std=gnu++17 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I/usr/local/opt/libpq/include -I/usr/local/Cellar/openssl@3/3.3.0/include -Ivendor -DRCPP_DEFAULT_INCLUDE_CALL=false -DRCPP_USING_UTF8_ERROR_STRING -DBOOST_NO_AUTO_PTR  -I'/Library/Frameworks/R.framework/Versions/4.3-x86_64/Resources/library/cpp11/include' -I'/Library/Frameworks/R.framework/Versions/4.3-x86_64/Resources/library/plogr/include' -I/opt/R/x86_64/include    -fPIC  -falign-functions=64 -Wall -g -O2  -c connection.cpp -o connection.o
In file included from connection.cpp:2:
In file included from ./RPostgres_types.h:6:
In file included from ./DbConnection.h:4:
In file included from vendor/boost/noncopyable.hpp:15:
In file included from vendor/boost/core/noncopyable.hpp:12:
In file included from vendor/boost/config.hpp:48:
vendor/boost/config/stdlib/libcpp.hpp:98:11: warning: 
      'BOOST_NO_AUTO_PTR' macro redefined [-Wmacro-redefined]
#  define BOOST_NO_AUTO_PTR
          ^
<command line>:4:9: note: previous definition is here
#define BOOST_NO_AUTO_PTR 1
        ^
1 warning generated.
clang++ -arch x86_64 -std=gnu++17 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I/usr/local/opt/libpq/include -I/usr/local/Cellar/openssl@3/3.3.0/include -Ivendor -DRCPP_DEFAULT_INCLUDE_CALL=false -DRCPP_USING_UTF8_ERROR_STRING -DBOOST_NO_AUTO_PTR  -I'/Library/Frameworks/R.framework/Versions/4.3-x86_64/Resources/library/cpp11/include' -I'/Library/Frameworks/R.framework/Versions/4.3-x86_64/Resources/library/plogr/include' -I/opt/R/x86_64/include    -fPIC  -falign-functions=64 -Wall -g -O2  -c cpp11.cpp -o cpp11.o
In file included from cpp11.cpp:4:
In file included from ./RPostgres_types.h:6:
In file included from ./DbConnection.h:4:
In file included from vendor/boost/noncopyable.hpp:15:
In file included from vendor/boost/core/noncopyable.hpp:12:
In file included from vendor/boost/config.hpp:48:
vendor/boost/config/stdlib/libcpp.hpp:98:11: warning: 
      'BOOST_NO_AUTO_PTR' macro redefined [-Wmacro-redefined]
#  define BOOST_NO_AUTO_PTR
          ^
<command line>:4:9: note: previous definition is here
#define BOOST_NO_AUTO_PTR 1
        ^
1 warning generated.
clang++ -arch x86_64 -std=gnu++17 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I/usr/local/opt/libpq/include -I/usr/local/Cellar/openssl@3/3.3.0/include -Ivendor -DRCPP_DEFAULT_INCLUDE_CALL=false -DRCPP_USING_UTF8_ERROR_STRING -DBOOST_NO_AUTO_PTR  -I'/Library/Frameworks/R.framework/Versions/4.3-x86_64/Resources/library/cpp11/include' -I'/Library/Frameworks/R.framework/Versions/4.3-x86_64/Resources/library/plogr/include' -I/opt/R/x86_64/include    -fPIC  -falign-functions=64 -Wall -g -O2  -c encode.cpp -o encode.o
clang++ -arch x86_64 -std=gnu++17 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I/usr/local/opt/libpq/include -I/usr/local/Cellar/openssl@3/3.3.0/include -Ivendor -DRCPP_DEFAULT_INCLUDE_CALL=false -DRCPP_USING_UTF8_ERROR_STRING -DBOOST_NO_AUTO_PTR  -I'/Library/Frameworks/R.framework/Versions/4.3-x86_64/Resources/library/cpp11/include' -I'/Library/Frameworks/R.framework/Versions/4.3-x86_64/Resources/library/plogr/include' -I/opt/R/x86_64/include    -fPIC  -falign-functions=64 -Wall -g -O2  -c encrypt.cpp -o encrypt.o
clang++ -arch x86_64 -std=gnu++17 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I/usr/local/opt/libpq/include -I/usr/local/Cellar/openssl@3/3.3.0/include -Ivendor -DRCPP_DEFAULT_INCLUDE_CALL=false -DRCPP_USING_UTF8_ERROR_STRING -DBOOST_NO_AUTO_PTR  -I'/Library/Frameworks/R.framework/Versions/4.3-x86_64/Resources/library/cpp11/include' -I'/Library/Frameworks/R.framework/Versions/4.3-x86_64/Resources/library/plogr/include' -I/opt/R/x86_64/include    -fPIC  -falign-functions=64 -Wall -g -O2  -c logging.cpp -o logging.o
clang++ -arch x86_64 -std=gnu++17 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I/usr/local/opt/libpq/include -I/usr/local/Cellar/openssl@3/3.3.0/include -Ivendor -DRCPP_DEFAULT_INCLUDE_CALL=false -DRCPP_USING_UTF8_ERROR_STRING -DBOOST_NO_AUTO_PTR  -I'/Library/Frameworks/R.framework/Versions/4.3-x86_64/Resources/library/cpp11/include' -I'/Library/Frameworks/R.framework/Versions/4.3-x86_64/Resources/library/plogr/include' -I/opt/R/x86_64/include    -fPIC  -falign-functions=64 -Wall -g -O2  -c result.cpp -o result.o
In file included from result.cpp:2:
In file included from ./RPostgres_types.h:6:
In file included from ./DbConnection.h:4:
In file included from vendor/boost/noncopyable.hpp:15:
In file included from vendor/boost/core/noncopyable.hpp:12:
In file included from vendor/boost/config.hpp:48:
vendor/boost/config/stdlib/libcpp.hpp:98:11: warning: 
      'BOOST_NO_AUTO_PTR' macro redefined [-Wmacro-redefined]
#  define BOOST_NO_AUTO_PTR
          ^
<command line>:4:9: note: previous definition is here
#define BOOST_NO_AUTO_PTR 1
        ^
1 warning generated.
clang++ -arch x86_64 -std=gnu++17 -dynamiclib -Wl,-headerpad_max_install_names -undefined dynamic_lookup -L/Library/Frameworks/R.framework/Resources/lib -L/opt/R/x86_64/lib -o RPostgres.so DbColumn.o DbColumnDataSource.o DbColumnDataSourceFactory.o DbColumnStorage.o DbConnection.o DbDataFrame.o DbResult.o PqColumnDataSource.o PqColumnDataSourceFactory.o PqDataFrame.o PqResult.o PqResultImpl.o PqResultSource.o PqUtils.o RPostgres-init.o connection.o cpp11.o encode.o encrypt.o logging.o result.o -L/usr/local/opt/libpq/lib -lpq -F/Library/Frameworks/R.framework/.. -framework R -Wl,-framework -Wl,CoreFoundation
installing to /Library/Frameworks/R.framework/Versions/4.3-x86_64/Resources/library/00LOCK-RPostgres/00new/RPostgres/libs
** R
** inst
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
** building package indices
** installing vignettes
** testing if installed package can be loaded from temporary location
** checking absolute paths in shared objects and dynamic libraries
** testing if installed package can be loaded from final location
** testing if installed package keeps a record of temporary installation path
* DONE (RPostgres)

The downloaded source packages are in
	‘/private/var/folders/np/n1zvy7cs7m9blq1w1_m_bqqw0000gn/T/RtmpYGGBVV/downloaded_packages’

jeroen · 2024-04-25T09:41:12Z

Have you looked in your redshift server logs why it is rejecting the connection? That would probably give a clue where to look.

Also did you try if the version of MacOS mattes?

I do suspect the kerberos version could be related. On MacOS I see:

krb5-config --version
# Kerberos 5 release 1.7-prerelease

Which is a pretty old. Homebrew has krb5 1.21. I can try to build a static version of krb5 that we can use in the R package, but that might take a while.

s-u · 2024-04-26T03:45:21Z

Can someone distill the bottom line here for me here, please? I'm about to bump the libraries on the CRAN build machines since we're about to do a full rebuild for R 4.4.0, so I'd like to know

what is the actual issue here
which version are required (or optional flags or whatever it is)
how can I tell if a binary is functioning as required (has to be self-contained, so don't assume anything else is installed)

jeroen · 2024-04-26T08:09:26Z

We don't know... It seems like an auth failure for specific servers, I suspect krb5 included with MacOS is too old (it's about 15 years old). So we would need to build against a more recent kerberos, however building that as a static lib is not easy.

ilarischeinin · 2024-04-26T08:19:19Z

Have you looked in your redshift server logs why it is rejecting the connection?

I don't see anything in the connectionlogs for this :/

Also did you try if the version of MacOS mattes?

Nowadays I'm on Sonoma (as is the colleague who also tried this), but it looks like I had Big Sur when I originally filed the issue.

I do suspect the kerberos version could be related.

As I've installed libpq with brew, I also have brew's krb5 (1.21.2) installed as it's a dependency. So is that something that is different between when binaries are built from CRAN vs. compiling locally? I mean that CRAN binaries are not built with libpq coming from Homebrew, and as a consequence the krb5 is different and older?

ilarischeinin · 2024-04-26T08:26:45Z

how can I tell if a binary is functioning as required (has to be self-contained, so don't assume anything else is installed)

One big challenge for troubleshooting is that this affects Redshit, i.e. a managed, commercial service. If it was e.g. Postgres, anyone could run the server via docker and test things. But I don't know if there is a Redshift instance running somewhere that would be publicly accessible. I can try to see if it would be possible for my workplace to temporarily turn on such a cluster.

markhwhiteii · 2024-07-10T20:57:29Z

Read through all of this and am wondering if there is a solution yet? I ran into the same error:

Error: connection to server at "<host>" (<ip>), port 5439 failed: server closed the connection unexpectedly
	This probably means the server terminated abnormally
	before or while processing the request.

See related StackOverflow question.

ilarischeinin mentioned this issue Apr 18, 2024

Update libpq to fix issue with RPostgres and Redshift R-macos/recipes#51

Closed

jeroen added a commit that referenced this issue Apr 19, 2024

Bump preferred libpq version on MacOS to 15

4d6ca6e

Fixes #441

jeroen mentioned this issue Apr 19, 2024

chore: Bump preferred libpq version on MacOS to 15 #465

Merged

aviator-app bot closed this as completed in #465 Apr 19, 2024

krlmlr reopened this Apr 19, 2024

krlmlr added the help wanted label Jul 20, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Something wrong with the 1.4.5 Mac binary on CRAN with Redshift #441

Something wrong with the 1.4.5 Mac binary on CRAN with Redshift #441

ilarischeinin commented Sep 1, 2023

krlmlr commented Sep 1, 2023

s-u commented Sep 2, 2023

ilarischeinin commented Sep 4, 2023

ilarischeinin commented Sep 5, 2023

krlmlr commented Apr 1, 2024

krlmlr commented Apr 1, 2024

ilarischeinin commented Apr 2, 2024

krlmlr commented Apr 13, 2024

ilarischeinin commented Apr 16, 2024

krlmlr commented Apr 18, 2024

jeroen commented Apr 18, 2024 •

edited

Loading

jeroen commented Apr 19, 2024

krlmlr commented Apr 19, 2024

ilarischeinin commented Apr 22, 2024

jeroen commented Apr 22, 2024 •

edited

Loading

ilarischeinin commented Apr 22, 2024

ilarischeinin commented Apr 22, 2024

jeroen commented Apr 22, 2024 •

edited

Loading

ilarischeinin commented Apr 22, 2024

jeroen commented Apr 22, 2024 •

edited

Loading

ilarischeinin commented Apr 22, 2024

ilarischeinin commented Apr 22, 2024

ilarischeinin commented Apr 25, 2024

jeroen commented Apr 25, 2024 •

edited

Loading

jeroen commented Apr 25, 2024 •

edited

Loading

ilarischeinin commented Apr 25, 2024

ilarischeinin commented Apr 25, 2024 •

edited

Loading

jeroen commented Apr 25, 2024

s-u commented Apr 26, 2024

jeroen commented Apr 26, 2024

ilarischeinin commented Apr 26, 2024

ilarischeinin commented Apr 26, 2024

markhwhiteii commented Jul 10, 2024 •

edited

Loading

Something wrong with the 1.4.5 Mac binary on CRAN with Redshift #441

Something wrong with the 1.4.5 Mac binary on CRAN with Redshift #441

Comments

ilarischeinin commented Sep 1, 2023

krlmlr commented Sep 1, 2023

s-u commented Sep 2, 2023

ilarischeinin commented Sep 4, 2023

ilarischeinin commented Sep 5, 2023

krlmlr commented Apr 1, 2024

krlmlr commented Apr 1, 2024

ilarischeinin commented Apr 2, 2024

krlmlr commented Apr 13, 2024

ilarischeinin commented Apr 16, 2024

krlmlr commented Apr 18, 2024

jeroen commented Apr 18, 2024 • edited Loading

jeroen commented Apr 19, 2024

krlmlr commented Apr 19, 2024

ilarischeinin commented Apr 22, 2024

jeroen commented Apr 22, 2024 • edited Loading

ilarischeinin commented Apr 22, 2024

ilarischeinin commented Apr 22, 2024

jeroen commented Apr 22, 2024 • edited Loading

ilarischeinin commented Apr 22, 2024

jeroen commented Apr 22, 2024 • edited Loading

ilarischeinin commented Apr 22, 2024

ilarischeinin commented Apr 22, 2024

ilarischeinin commented Apr 25, 2024

jeroen commented Apr 25, 2024 • edited Loading

jeroen commented Apr 25, 2024 • edited Loading

ilarischeinin commented Apr 25, 2024

ilarischeinin commented Apr 25, 2024 • edited Loading

jeroen commented Apr 25, 2024

s-u commented Apr 26, 2024

jeroen commented Apr 26, 2024

ilarischeinin commented Apr 26, 2024

ilarischeinin commented Apr 26, 2024

markhwhiteii commented Jul 10, 2024 • edited Loading

jeroen commented Apr 18, 2024 •

edited

Loading

jeroen commented Apr 22, 2024 •

edited

Loading

jeroen commented Apr 22, 2024 •

edited

Loading

jeroen commented Apr 22, 2024 •

edited

Loading

jeroen commented Apr 25, 2024 •

edited

Loading

jeroen commented Apr 25, 2024 •

edited

Loading

ilarischeinin commented Apr 25, 2024 •

edited

Loading

markhwhiteii commented Jul 10, 2024 •

edited

Loading