Address providing compilers in TrilinosConfig.cmake and <Package>Config.cmake files

[bartlettroscoe]

CC: @sebrowne, @ccober6, @jwillenbring, @rppawlo, @jhux2

Description

The TriBITS-generated TrilinosConfig.cmake and <Package>Config.cmake files have long included variables that gave the full paths to the different compilers used to build and install Trilinos as well as the complier options as the variables:

set(<prefix>_CMAKE_C_COMPILER ...)
set(<prefix>_CMAKE_CXX_COMPILER ...)
set(<prefix>_CMAKE_Fortaran_COMPILER ...)

set(<prefix>_CMAKE_C_FLAGS ...)
set(<prefix>_CMAKE_CXX_FLAGS ...)
set(<prefix>_CMAKE_Fortaran_FLAGS ...)

This allowed downstream projects like Albany to call find_package(Trilinos) and then use the compilers defined by that call. However, this does not work when using GNUInstallDirs.cmake (see sandialabs/Albany#982) or when using a CUDA build with Kokkos (see #11863 (comment) and TriBITSPub/TriBITS#545).

Therefore, to avoid these problems when moving to modern CMake (see TriBITSPub/TriBITS#411), these compiler-related variables should be removed from these files.

However, if it is desired to have a way to provide these compilers and compiler options to downstream, another solution is suggested in Proposed Solution 2 in:

• Address providing compilers to downstream CMake projects TriBITSPub/TriBITS#545

[bartlettroscoe]

@sebrowne, @ccober6, @jwillenbring, with the merge of:

• Famework: Change defaults for Trilinos-specific CMake flags #12258

it has just been discovered that this change that will result in libs and <Pacakg>Config.cmake files being installed under <prefix>/lib64/ on most Linux systems will break the usage of Trilinos for downstream CMake projects that are getting the compilers from Trilinos when calling find_package(Trilinos). See:

• https://gitlab.kitware.com/cmake/cmake/-/issues/25157#note_1417970

I think that we should ASAP update the Trilinos RELEASE_NOTES to state that Trilinos 15.0+ will no longer support providing the compilers in the installed <Package>Config.cmake files. Then we can rip out support for that from Trilinos and TriBITS (see TriBITSPub/TriBITS#545).

Note that when doing a CUDA build, you have to define the compilers first before calling find_package(Kokkos) or find_package(Trilinos) (see #11955 (comment)).

With Trilinos 15.0 this is a chance to officially remove this functionality with explanation!

Should I put in a PR that makes this change and drop support for putting the compilers in the <Package>Config.cmake files from TriBITS? Should we pull the trigger on:

• Address providing compilers to downstream CMake projects TriBITSPub/TriBITS#545

and either remove support for providing compilers altogether or provide them by a separate call to find_package(TrilinosEnv)?

This seems like the best time to address this (or back out the change to use GNUInstallDirs until Trilinos 16.0).

[bartlettroscoe]

@sebrowne, @ccober6, @jwillenbring, @rppawlo, @jhux2,

So given that at least Albany and Nalu still want to be able to get compilers from the Trilinos installation, we may need to determine how many other customers want to keep this functionality. And if they want to keep that functionality, we should consider pulling the trigger on Proposed Solution 2 in:

• Address providing compilers to downstream CMake projects TriBITSPub/TriBITS#545

Someone should bring this up at the next SART meeting.

[bartlettroscoe]

FYI: At the Trilinos Developers meeting yesterday, it was mentioned that this issue was discussed with the SART meeting and that @glhenni specifically asked about this.

@glhenni, if you have a strong opinion on this, please let us know.

[glhenni]

FYI: At the Trilinos Developers meeting yesterday, it was mentioned that this issue was discussed with the SART meeting and that @glhenni specifically asked about this.

@glhenni, if you have a strong opinion on this, please let us know.

I guess I don't see the problem, admittedly probably because I don't have the depth of knowledge required. If a project chooses to use, for example, Trilinos_CXX_COMPILER, but that doesn't always work, like in the case of cuda, then why isn't that burden on the project to figure out? The compiler and associated flags used to build Trilinos seems like pretty useful information to have.

[jhux2]

@bartlettroscoe This has also affected Sierra. @vbrunini can give more details.

[bartlettroscoe]

@bartlettroscoe This has also affected Sierra. @vbrunini can give more details.

@vbrunini, if I had to guess, this would be related to the <prefix>/lib to <prefix>/lib64 issue with the change to GNUInstlalDirs.cmake. See:

Trilinos/RELEASE_NOTES

Lines 18 to 47 in 55108f7

	- Change the default for `Trilinos_USE_GNUINSTALLDIRS` from `OFF` to `ON`,
	in the goal to move Trilinos and TriBITS to modern CMake.
	TriBITS has had the ability to use that paths selected by the standard
	CMake module `GNUInstallDirs.cmake` for a long time. But it is turned off
	in TriBITS by default and was never turned on in Trilinos, both for the
	sake of backward compatibility.
	This may break people's existing configurations because it will install
	libs in `<prefix>/libs64/` instead of in `<prefix>/libs/` on many systems
	(e.g. Linux systems). For example, this will break downstream CMake
	projects that call `find_package(Trilinos ...)` before defining the
	compilers (e.g. so they can get the compilers from Trilinos). If the
	compilers are not defined, `find_package()` will not search
	`<prefix>/lib64`. To revert back to using `<prefix>/lib` but still use
	`GNUInstallDirs.cmake` for Trilinos, set `-D
	CMAKE_INSTALL_LIBDIR:STRING=lib` when configuring Trilinos. To avoid the
	`find_package(Trilinos ...)` problem not searching `<prefix>/lib64`,
	consider explicitly specifying the compiler to and having the downstream
	CMake project define the compilers first with `project(<ProjectName>
	COMPILERS C CXX ...)` before calling `find_package(Trilinos ...)`. (That
	is, don't try to get the compilers from the installed Trilinos, see
	https://github.com/trilinos/Trilinos/issues/12306.)
	NOTE: The setting `-D Trilinos_USE_GNUINSTALLDIRS=OFF` is deprecated and
	may be removed in the future. (I.e. the usage of `GNUInstallDirs.cmake`
	may be hard-coded in the future so please try adjusting to the usage of
	`GNUInstallDirs.cmake` by Trilinos.) See
	https://github.com/trilinos/Trilinos/issues/12104#issuecomment-1691945033
	for additional details and instructions.

[vbrunini]

@bartlettroscoe This has also affected Sierra. @vbrunini can give more details.

@vbrunini, if I had to guess, this would be related to the <prefix>/lib to <prefix>/lib64 issue with the change to GNUInstlalDirs.cmake. See:

Trilinos/RELEASE_NOTES

Lines 18 to 47 in 55108f7

	- Change the default for `Trilinos_USE_GNUINSTALLDIRS` from `OFF` to `ON`,
	in the goal to move Trilinos and TriBITS to modern CMake.
	TriBITS has had the ability to use that paths selected by the standard
	CMake module `GNUInstallDirs.cmake` for a long time. But it is turned off
	in TriBITS by default and was never turned on in Trilinos, both for the
	sake of backward compatibility.
	This may break people's existing configurations because it will install
	libs in `<prefix>/libs64/` instead of in `<prefix>/libs/` on many systems
	(e.g. Linux systems). For example, this will break downstream CMake
	projects that call `find_package(Trilinos ...)` before defining the
	compilers (e.g. so they can get the compilers from Trilinos). If the
	compilers are not defined, `find_package()` will not search
	`<prefix>/lib64`. To revert back to using `<prefix>/lib` but still use
	`GNUInstallDirs.cmake` for Trilinos, set `-D
	CMAKE_INSTALL_LIBDIR:STRING=lib` when configuring Trilinos. To avoid the
	`find_package(Trilinos ...)` problem not searching `<prefix>/lib64`,
	consider explicitly specifying the compiler to and having the downstream
	CMake project define the compilers first with `project(<ProjectName>
	COMPILERS C CXX ...)` before calling `find_package(Trilinos ...)`. (That
	is, don't try to get the compilers from the installed Trilinos, see
	https://github.com/trilinos/Trilinos/issues/12306.)
	NOTE: The setting `-D Trilinos_USE_GNUINSTALLDIRS=OFF` is deprecated and
	may be removed in the future. (I.e. the usage of `GNUInstallDirs.cmake`
	may be hard-coded in the future so please try adjusting to the usage of
	`GNUInstallDirs.cmake` by Trilinos.) See
	https://github.com/trilinos/Trilinos/issues/12104#issuecomment-1691945033
	for additional details and instructions.

Yes it was from the move to <prefix>/lib64, I've got a fix working it's way through our CI now so I don't think there are any significant concerns on our end. Thanks.

[bartlettroscoe]

Yes it was from the move to <prefix>/lib64, I've got a fix working it's way through our CI now so I don't think there are any significant concerns on our end. Thanks.

@vbrunini, just curious, but what fix the Sierra go with?

[vbrunini]

Yes it was from the move to <prefix>/lib64, I've got a fix working it's way through our CI now so I don't think there are any significant concerns on our end. Thanks.

@vbrunini, just curious, but what fix the Sierra go with?

Previously we used a dummy CMakeLists.txt like:

cmake_minimum_required(VERSION 3.20)
project(dummyfinder LANGUAGES)

find_package(Trilinos PATHS {prefix} NO_MODULE NO_DEFAULT_PATH NO_SYSTEM_ENVIRONMENT_PATH)

# Code to extract include paths & library names from cmake targets so that we can pass it on to sierra's non-cmake build system here

and I just added CXX to the list of languages so cmake would pick up that it should look in lib64.

[bartlettroscoe]

@glhenni, if you have a strong opinion on this, please let us know.

I guess I don't see the problem, admittedly probably because I don't have the depth of knowledge required. If a project chooses to use, for example, Trilinos_CXX_COMPILER, but that doesn't always work, like in the case of cuda, then why isn't that burden on the project to figure out?

Right, but what good is a process that does not work for all situations? If you need to do something completely different for CUDA, then what is the value of trying to rely on Trilinos complilers?

The compiler and associated flags used to build Trilinos seems like pretty useful information to have.

Okay, so if that is useful, then the current system breaks when Trilinos is using the standard CMake module GNUInstallDirs.cmake (and installs under <prefix>/lib64) and find_package(Trilinos ...) can't find the installed Trilinos because find_package() will not search under <prefix>/lib64 until the compilers are defined. So the whole darn thing is broken.

The idea with the Proposed Solution 2 in:

• Address providing compilers to downstream CMake projects TriBITSPub/TriBITS#545

provides as small tweak to this process that will work for CUDA and when using GNUInstallDirs.cmake with the TriBITS project (i.e. Trilinos).

Is that worth implementing and supporting if it means that Trilinos customer projects can use the compilers and tools that were used in the installed Trilinos in all cases, even CUDA? That is the question because even before switching to GNUInstallDirs.cmake that was not longer possible due to find_package(CUDAToolkit)! This would fix this and the issue with GNUInstallDirs.cmake.

Make sense? If not, I should explain this at the next SART meeting.

[glhenni]

@rabartl
Make sense? If not, I should explain this at the next SART meeting.

I'm okay with Proposed Solution 2.

[bartlettroscoe]

@rabartl
Make sense? If not, I should explain this at the next SART meeting.

I'm okay with Proposed Solution 2.

@glhenni, if you want that, then as an important user of Trilinos, you should request that via the Trilinos HelpDesk (an internal SNL site). If the request comes from an important user, then it will carry more weight with the Trilinos prioritization process. The Trilinos framework team does not directly manage issues on GitHub, only on the Trilinos HelpDesk (or at least that was their policy).

[github-actions]

This issue has had no activity for 365 days and is marked for closure. It will be closed after an additional 30 days of inactivity.
If you would like to keep this issue open please add a comment and/or remove the MARKED_FOR_CLOSURE label.
If this issue should be kept open even with no activity beyond the time limits you can add the label DO_NOT_AUTOCLOSE.
If it is ok for this issue to be closed, feel free to go ahead and close it. Please do not add any comments or change any labels or otherwise touch this issue unless your intention is to reset the inactivity counter for an additional year.