I have twice encountered problems in R/RStudio when attempting to install packages compiled from source -- rstan specifically -- due to R persistently looking in the wrong place for the g++ compiler. This post is intended to document the problems for others. I encountered these issues in R 4.3.1 the first time and R 4.3.2 the second, both with rtools43. I am on Windows 10 64 bit and using RStudio 2023.12.0.369 OCean Storm.
In both cases the solution was to go to the Makeconf file in C:\Program Files\R\R-4.3.2\etc\x64 and modify the BINPREF variable manually from BINPREF ?= to the appropriate location for the most up-to-date installation of rbuildtools. At time of writing that is C:\rtools43\x86_64-w64-mingw32.static.posix\bin. This solution was identified through this answer https://stackoverflow.com/a/46619260/22555498 but finding it each time was tortuous.
The first manifestation is documented here: https://stackoverflow.com/a/77101426/22555498
The second manifestation started when I attempted to update to R 4.3.2 from 4.3.1, and was initially a failure of stan models to compile through R. I attempted a clean installation of rstan which failed to fix the problem, so I did a complete reinstallation of R, RStudio, rtools, and rstan. In this instance compilation would fail for rstan specifically at the stage of loading libraries as R kept looking for C:/RBuildTools/3.5/mingw_64/bin/g++ in order to compile from source with g++, when the correct route was through C:\rtools43\x86_64-w64-mingw32.static.posix\bin. Error output looked like
* installing *source* package 'rstan' ...
** package 'rstan' successfully unpacked and MD5 sums checked
** using staged installation
** libs
C:/RBuildTools/3.5/mingw_64/bin/g++ -std=gnu++17 -I"C:/PROGRA~1/R/R-43~1.2/include" -DNDEBUG -I"../inst/include" -I"../inst/include/boost_not_in_BH" -I"C:/Users/there/AppData/Local/R/win-library/4.3/StanHeaders/include/src" -DBOOST_DISABLE_ASSERTS -DBOOST_PHOENIX_NO_VARIADIC_EXPRESSION -D_REENTRANT -DSTAN_THREADS -DUSE_STANC3 -DSTRICT_R_HEADERS -D_HAS_AUTO_PTR_ETC=0 -DRCPP_PARALLEL_USE_TBB=1 -I'C:/Users/there/AppData/Local/R/win-library/4.3/Rcpp/include' -I'C:/Users/there/AppData/Local/R/win-library/4.3/RcppEigen/include' -I'C:/Users/there/AppData/Local/R/win-library/4.3/BH/include' -I'C:/Users/there/AppData/Local/R/win-library/4.3/StanHeaders/include' -I'C:/Users/there/AppData/Local/R/win-library/4.3/RcppParallel/include' -I"C:/rtools43/x86_64-w64-mingw32.static.posix/include" -DRCPP_PARALLEL_USE_TBB=1 -DSTRICT_R_HEADERS -O2 -Wall -mfpmath=sse -msse2 -mstackrealign -c Module.cpp -o Module.o
/bin/sh: line 1: C:/RBuildTools/3.5/mingw_64/bin/g++: No such file or directory
make: *** [C:/PROGRA~1/R/R-43~1.2/etc/x64/Makeconf:272: Module.o] Error 127
ERROR: compilation failed for package 'rstan'
* removing 'C:/Users/there/AppData/Local/R/win-library/4.3/rstan'
Warning in install.packages :
installation of package ‘rstan’ had non-zero exit status
The above error comes from recreating the error by reverting the Makeconf file to have BINPREF ?= and may not perfectly match what was originally seen but looks how I remember.
Solutions such as setting PATH and BINPREF through the Sys.setenv() function failed to correct the problem due to the hard coding in the Makeconf file. Running Sys.getenv("PATH") pointed to c:\\rtools43 which existed and was correct, but was missing C:\\rtools43\\x86_64-w64-mingw32.static.posix\\bin which I did not determine was the issue because I could not find that answer anywhere.
Sys.which("make") pointed to C:\\rtools43\\usr\\bin\\make.exe which was also correct.
Searches related to C:/RBuildTools/3.5/mingw_64/bin/g++ did not turn up useful results as the file structure seems to have changed since the first time I encountered the problem, now directing to C:/rtools43. Similar question pointed at C:/RBuildTools/4.3/mingw_64/bin/g++ which is not the default location for installation when I did it this time.