Mpi omp support #336
Merged
Mpi omp support #336
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
… already part of the config toolbox).
…ccept a function that can return file-specific transformation scripts
Add support for specifying MPI and OpenMP in the config file
|
Forgot to request a review sigh |
MatthewHambley
requested changes
Aug 29, 2024
There was a problem hiding this comment.
The reformatting does add many extra changes but it needs to be done so I'm content with including it.
I think this change takes is in the right direction, my primary concern is the fact that tools for which "MPI" and "OpenMP" have no meaning have gained arguments for them.
Please give this some consideration. If tackling this issue would be a considerable amount of work I'm happy for you to raise an issue for it and link that to this change.
source/fab/tools/tool_repository.py
Outdated
Comment on lines
134
to
140
| def get_default(self, category: Category, | ||
| mpi: Optional[bool] = None): | ||
| '''Returns the default tool for a given category. For most tools | ||
| that will be the first entry in the list of tools. The exception | ||
| are compilers and linker: in this case it must be specified if | ||
| MPI support is required or not. And the default return will be | ||
| the first tool that either supports MPI or not. |
There was a problem hiding this comment.
Your comment raises an important point. The concept of "MPI" makes no sense for most tools. We may need to rethink this whole API but maybe not for this change.
Can the MPI-ness be made part of the category? Something like tool_repo.get_default(FortranCompiler(mpi=True)).
There was a problem hiding this comment.
@MatthewHambley
Fair point, I have half a dozen alternatives, here the few that I think are actually sensible:
- We add separate
get_compiler/get_linkermethods (which take mpi as parameter), nothing else does. - We abort if a value for mpi is specified for any non-compiler/linker too. Or we turn this into kargs, meaning we can add tool-specific flags in the future, and for now only accept MPI for compiler/linker.
- We add the
BuildConfigas argument instead of mpi only. We can access the mpi status this way, and whatever any tool might need in the future could be done (admittedly I can't think of anything useful here :) ... maybe, and I am clutching at straws here, a tool could access the number of cores to use in a build, if a tool itself can run in parallel??)
Would one of these solution work for you? Or do you have a better idea?
There was a problem hiding this comment.
@MatthewHambley That's the question I need your preference so that I can update this.
There was a problem hiding this comment.
Of the options given I think I prefer number 1.
I don't like exceptions on invalid arguments, I would rather the arguments were valid. I'm also wary of using kargs as it throws away all the advantages of static call signature checking. Also of static type analysis.
Throwing BuildConfig at the problem also suffers from removing static checking.
One possible solution was hinted at earlier. The Category enumeration could take an MPI argument for compiler and linker tools. I think this would work and would be limited to just the categories which care about the concept of MPI.
When calling: tool_repo.get_default(FortranCompiler(mpi=True)). In the method: if isinstance(category, FortranCompiler): needs_mpi = category.needs_mpi().
There was a problem hiding this comment.
I don't understand what you mean with Category .. enums are afaik not configurable. We could add a method get_fortran(mpi) etc to Category that would return FORTRAN_MPI or FORTRAN_NO_MPI, but given that we need this three times (Fortran, C, linker), this feels like a huge amount of unnecessary code (given the current size of category.py). Plus, if we test if a tool is really a Fortran compiler, we then need to test for two potential values :(
Adding get_compiler/get_linker also feels odd when I tried to do this: the get_compiler would still need to have a category (to get Fortran or c), or we add get_c_compiler and get_fortran _compiler. This then also need to be duplicated, since we need the same functions then in ToolBox and ToolRepository.
This all gets very messy if we want to add different language (C++ anyone?) - atm that might be reasonable easy to do (add the allowed suffixes, add a category, and pick C or C++ compiler based on suffix,... done .. mostly, I am sure I missed something :) But compared with then having to add get_cpp_compiler twice, ...) - feels wrong. I think that would couple the language way too much into otherwise generic tool handling classes.
My suggestion would then be to add the config object as parameter - either optional (I can't think of many other tools needing this :) ), or mandatory.
There was a problem hiding this comment.
I had a deeper look into Enum and you are right. They can take parameters but only across all members of the Enum. i.e. every option carries the same parameters.
I've raised #350 to consider an alternative approach.
Yes. How did you do #323? Just running black with certain option? If you give us the option, we can do this between our upcoming PRs, to get it out of the way once. And I really appreciate not merging #323 in, as you can see, that will potentially cause us quite a bit of headache in terms of possible conflicts :( |
I did it by hand, directed by |
|
OK, I'll just declare this ready for next review after giving mpi/openmp a default, to avoid the confusion about there his PR is at :) |
|
@MatthewHambley , I can't re-request a review, but I assumed you were done. From my point of view, this is back to you. |
MatthewHambley
approved these changes
Nov 19, 2024
hiker
added a commit
that referenced
this pull request
Nov 21, 2024
* Made mpi and openmp default to False in the BuildConfig constructor. * Removed white space. * Support compilers that do not support OpenMP. * Added documentation for openmp parameter. * Mpi omp support (#343) * #3 Introduce fixtures for Fortran compiler and tool_box. * #3 Remove explicit compiler information from MpCommonArgs (since it's already part of the config toolbox). * #3 Added linker as tool. * #3 Added test for linking shared libraries. * #3 Pass compiler flags to the linker if a compiler was specified. * #3 Remove unused function. * #3 Removed more unused code. * #3 Automatically add a linker for each compiler. * #3 Fixed typo. * #3 Support vendor for compiler and linker. * #3 Make linker having a vendor, too. * #3 Add set_default_vendor method to tool repository. * Ignore build directory for git. * Updated test. * # Fix some mypy errors and warnings. * Avoid using get() for singleton, instead use __new__ which makes mypy happier. * Changed the transformation_script parameter of function psyclone to accept a function that can return file-specific transformation scripts * Make mypy happy by using patch.object. * Remove more comments and confusion about mypy :) * Try to make mypy happy on older python versions. * Make flake8 happy. * Sort imported name alphabetically. * Try to fix failing hash test (and add some additional improvements in the test). * Removed fpath= for input transformation_script function to pass mypy test for Python 3.7; Moved transformation_script_hash test to unit test from system test * Fix mypy typing check errors for psyclone unit test * Fix config typing issue with mypy in psyclone unit test * Fix flake8 issues; Revert Config mypy typing fix * Add comment to ignore typing check for fpath parameter of input transformation_script function * Fix assert check after transformation_script function is changed from being called twice to once * Filter out 'no transformation script' warning for psyclone system test * Replace 'ignore' typing of fpath of transformation_script with removing keyword argument * #3 Support proper tests to check if tools are available. * 1. Updated transformation_script description; 2. Modified mock_transformation_script; 3.Removed redundant _analysis_for_prebuilds * Updated lfric/atm.py and lfric/gungho.py examples to pass in transformation_script functions * Added description for the psyclone step to instructions on writing a config * #3 Added git as a tool. * #3 Fix incorrect | usage in typing. * #3 Added unit tests for git. * #3 Renamed git.py to versioning.py, to avoid name clash with the corresponding test_git.py tests. * #3 Converted svn and fcm to tools. * #3 Fixed missing whitespace. * Modified the documentation for writing a config with PSyclone * Add config as a parameter for run_psyclone for the transformation_script to use;Updated the related functions and tests; Changed the logic of the transformation_script examples * #3 Replaced ar with tool object. * #3 Added tests for ar.py. * #3 Removed debug output. * #3 Converted PSyclone to be a tool. * #3 Removed debug print, fixed python 3.7 typing information. * #3 Updated comments. * Modified the get_optimisation_script function examples and updated the doc formatting * #3 Add Rsync tool. * #3 Removed now unused function. * #3 Added test for rsync. * #3 Fixed all mypy warnings about functions not checked. * #3 Replace all mock-tests to use subprocess so the name of the executable is tested as well. * #3 Remove duplicated flags. * #3 Fixed changed order of linking. * #3 Removed run_command function. * #3 Fixed 3.8 typing error. * #3 Fixed unused imports. * #3 Move flags checksum into Flags, and remove now unused tools.py file. * #3 Renamed newtools to tools. * #3 Made custom function for all git functions called (instead of just calling run). * #3 Updated and fixed comments. * #3 Fixed errors in comments. * Fixed minor errors in documentation. * #3 Make it easier to create wrapper around standard compiler. * #3 Added documentation for all tool related classes and their usage. * #3 Added MISC category. * Addressed reviewer's comments. * Updated cli to properly use ToolBox etc, removing hard-coded gnu command linker option. * Fixed mypy failures, including changes to import statement to avoid cyclic imports :(. * #3 Fix circular import. * Added #TODO so that this can be removed once fparser supports sentinels. * Fix typing problems by ignoring fparser. * Replaced more string names for artefacts with enums. * Removed EXECUTABLES from constants. * Moved Artefact class out of ArtefactStore and renamed it to ArtefactSet. * Moved OBJECT_FILES from constants into ArtefactSet. * Moved OBJECT_ARCHIVES from constants to ArtefactSet. * Moved PRAGMAD_C from constants to ArtefactSet. * Turned 'all_source' into an enum. * Allow integer as revision. * Fixed flake8 error. * Removed specific functions to add/get fortran source files etc. * Removed non-existing and unneccessary collections. * Try to fix all run_configs. * Fixed rebase issues. * Added replace functionality to ArtefactStore, updated test_artefacts to cover all lines in that file. * Started to replace artefacts when files are pre-processed. * Removed linker argument from linking step in all examples. * Try to get jules to link. * Fixed build_jules. * Fixed other issues raised in reviews. * Try to get jules to link. * Fixed other issues raised in reviews. * Simplify handling of X90 files by replacing the X90 with x90, meaning only one artefact set is involved when running PSyclone. * Make OBJECT_ARCHIVES also a dict, migrate more code to replace/add files to the default build artefact collections. * Fixed some examples. * Fix flake8 error. * Fixed failing tests. * Support empty comments. * Fix preprocessor to not unnecessary remove and add files that are already in the output directory. * Allow find_soure_files to be called more than once by adding files (not replacing artefact). * Updated lfric_common so that files created by configurator are written in build (not source). * Use c_build_files instead of pragmad_c. * Removed unnecessary str. * Documented the new artefact set handling. * Fixed typo. * Make the PSyclone API configurable. * Fixed formatting of documentation, properly used ArtefactSet names. * Support .f and .F Fortran files. * Removed setter for tool.is_available, which was only used for testing. * #3 Fix documentation and coding style issues from review. * Renamed Categories into Category. * Minor coding style cleanup. * Removed more unnecessary (). * Re-added (invalid) grab_pre_build call. * Fixed typo. * Renamed set_default_vendor to set_default_compiler_suite. * Renamed VendorTool to CompilerSuiteTool. * Also accept a Path as exec_name specification for a tool. * Move the check_available function into the base class. * Fixed some types and documentation. * Fix typing error. * Added explanation for meta-compiler. * Improved error handling and documentation. * Replace mpiifort with mpifort to be a tiny bit more portable. * Use classes to group tests for git/svn/fcm together. * Fixed issue in get_transformation script, and moved script into lfric_common to remove code duplication. * Code improvement as suggested by review. * Fixed run config * Added reference to ticket. * Updated type information. * More typing fixes. * Fixed typing warnings. * As requested by reviewer removed is_working_copy functionality. * Issue a warning (which can be silenced) when a tool in a toolbox is replaced. * Fixed flake8. * Fixed flake8. * Fixed failing test. * Addressed issues raised in review. * Removed now unnecessary operations. * Updated some type information. * Fixed all references to APIs to be consistent with PSyclone 2.5. * Added api to the checksum computation. * Fixed type information. * Added test to verify that changing the api changes the checksum. * Make compiler version a tuple of integers * Update some tests to use tuple versions * Explicitly test handling of bad version format * Fix formatting * Tidying up * Make compiler raise an error for any invalid version string Assume these compilers don't need to be hashed. Saves dealing with empty tuples. * Check compiler version string for compiler name * Fix formatting * Add compiler.get_version_string() method Includes other cleanup from PR comments * Add mpi and openmp settings to BuildConfig, made compiler MPI aware. * Looks like the circular dependency has been fixed. * Revert "Looks like the circular dependency has been fixed." ... while it works with the tests, a real application still triggered it. This reverts commit 150dc37. * Don't even try to find a C compiler if no C files are to be compiled. * Updated gitignore to ignore (recently renamed) documentation. * Fixed failing test. * Return from compile Fortran early if there are no files to compiles. Fixed coding style. * Add MPI enables wrapper for intel and gnu compiler. * Fixed test. * Automatically add openmp flag to compiler and linker based on BuildConfig. * Removed enforcement of keyword parameters, which is not supported in python 3.7. * Fixed failing test. * Support more than one tool of a given suite by sorting them. * Use different version checkout for each compiler vendor with mixins * Refactoring, remove unittest compiler class * Fix some mypy errors * Use 'Union' type hint to fix build checks * Return run_version_command to base Compiler class Provides default version command that can be overridden for other compilers. Also fix some incorrect tests Other tidying * Add a missing type hint * Remove inheritance from mixins and use protocol * Simplify compiler inheritance Mixins have static methods with unique names, overrides only happen in concrete classes * Simplify usage of compiler-specific parsing mixins. * Test for missing mixin. * Fixed test. * Added more tests for invalid version numbers. * Added more test cases for invalid version number, improved regex to work as expected. * Fixed typo in test. * Fixed test. * Split tests into smaller individual ones, fixed missing asssert in test. * Parameterised compiler version tests to also test wrapper. * Added missing MPI parameter when getting the compiler. * Fixed comments. * Order parameters to be in same order for various compiler classes. * Remove stray character * Made mpi and openmp default to False in the BuildConfig constructor. * Removed white space. * Support compilers that do not support OpenMP. * Added documentation for openmp parameter. --------- Co-authored-by: Junwei Lyu <[email protected]> Co-authored-by: Junwei Lyu <[email protected]> Co-authored-by: Junwei Lyu <[email protected]> Co-authored-by: Junwei Lyu <[email protected]> Co-authored-by: jasonjunweilyu <[email protected]> Co-authored-by: Luke Hoffmann <[email protected]> Co-authored-by: Luke Hoffmann <[email protected]> * Mpi omp support (#346) * #3 Introduce fixtures for Fortran compiler and tool_box. * #3 Remove explicit compiler information from MpCommonArgs (since it's already part of the config toolbox). * #3 Added linker as tool. * #3 Added test for linking shared libraries. * #3 Pass compiler flags to the linker if a compiler was specified. * #3 Remove unused function. * #3 Removed more unused code. * #3 Automatically add a linker for each compiler. * #3 Fixed typo. * #3 Support vendor for compiler and linker. * #3 Make linker having a vendor, too. * #3 Add set_default_vendor method to tool repository. * Ignore build directory for git. * Updated test. * # Fix some mypy errors and warnings. * Avoid using get() for singleton, instead use __new__ which makes mypy happier. * Changed the transformation_script parameter of function psyclone to accept a function that can return file-specific transformation scripts * Make mypy happy by using patch.object. * Remove more comments and confusion about mypy :) * Try to make mypy happy on older python versions. * Make flake8 happy. * Sort imported name alphabetically. * Try to fix failing hash test (and add some additional improvements in the test). * Removed fpath= for input transformation_script function to pass mypy test for Python 3.7; Moved transformation_script_hash test to unit test from system test * Fix mypy typing check errors for psyclone unit test * Fix config typing issue with mypy in psyclone unit test * Fix flake8 issues; Revert Config mypy typing fix * Add comment to ignore typing check for fpath parameter of input transformation_script function * Fix assert check after transformation_script function is changed from being called twice to once * Filter out 'no transformation script' warning for psyclone system test * Replace 'ignore' typing of fpath of transformation_script with removing keyword argument * #3 Support proper tests to check if tools are available. * 1. Updated transformation_script description; 2. Modified mock_transformation_script; 3.Removed redundant _analysis_for_prebuilds * Updated lfric/atm.py and lfric/gungho.py examples to pass in transformation_script functions * Added description for the psyclone step to instructions on writing a config * #3 Added git as a tool. * #3 Fix incorrect | usage in typing. * #3 Added unit tests for git. * #3 Renamed git.py to versioning.py, to avoid name clash with the corresponding test_git.py tests. * #3 Converted svn and fcm to tools. * #3 Fixed missing whitespace. * Modified the documentation for writing a config with PSyclone * Add config as a parameter for run_psyclone for the transformation_script to use;Updated the related functions and tests; Changed the logic of the transformation_script examples * #3 Replaced ar with tool object. * #3 Added tests for ar.py. * #3 Removed debug output. * #3 Converted PSyclone to be a tool. * #3 Removed debug print, fixed python 3.7 typing information. * #3 Updated comments. * Modified the get_optimisation_script function examples and updated the doc formatting * #3 Add Rsync tool. * #3 Removed now unused function. * #3 Added test for rsync. * #3 Fixed all mypy warnings about functions not checked. * #3 Replace all mock-tests to use subprocess so the name of the executable is tested as well. * #3 Remove duplicated flags. * #3 Fixed changed order of linking. * #3 Removed run_command function. * #3 Fixed 3.8 typing error. * #3 Fixed unused imports. * #3 Move flags checksum into Flags, and remove now unused tools.py file. * #3 Renamed newtools to tools. * #3 Made custom function for all git functions called (instead of just calling run). * #3 Updated and fixed comments. * #3 Fixed errors in comments. * Fixed minor errors in documentation. * #3 Make it easier to create wrapper around standard compiler. * #3 Added documentation for all tool related classes and their usage. * #3 Added MISC category. * Addressed reviewer's comments. * Updated cli to properly use ToolBox etc, removing hard-coded gnu command linker option. * Fixed mypy failures, including changes to import statement to avoid cyclic imports :(. * #3 Fix circular import. * Added #TODO so that this can be removed once fparser supports sentinels. * Fix typing problems by ignoring fparser. * Replaced more string names for artefacts with enums. * Removed EXECUTABLES from constants. * Moved Artefact class out of ArtefactStore and renamed it to ArtefactSet. * Moved OBJECT_FILES from constants into ArtefactSet. * Moved OBJECT_ARCHIVES from constants to ArtefactSet. * Moved PRAGMAD_C from constants to ArtefactSet. * Turned 'all_source' into an enum. * Allow integer as revision. * Fixed flake8 error. * Removed specific functions to add/get fortran source files etc. * Removed non-existing and unneccessary collections. * Try to fix all run_configs. * Fixed rebase issues. * Added replace functionality to ArtefactStore, updated test_artefacts to cover all lines in that file. * Started to replace artefacts when files are pre-processed. * Removed linker argument from linking step in all examples. * Try to get jules to link. * Fixed build_jules. * Fixed other issues raised in reviews. * Try to get jules to link. * Fixed other issues raised in reviews. * Simplify handling of X90 files by replacing the X90 with x90, meaning only one artefact set is involved when running PSyclone. * Make OBJECT_ARCHIVES also a dict, migrate more code to replace/add files to the default build artefact collections. * Fixed some examples. * Fix flake8 error. * Fixed failing tests. * Support empty comments. * Fix preprocessor to not unnecessary remove and add files that are already in the output directory. * Allow find_soure_files to be called more than once by adding files (not replacing artefact). * Updated lfric_common so that files created by configurator are written in build (not source). * Use c_build_files instead of pragmad_c. * Removed unnecessary str. * Documented the new artefact set handling. * Fixed typo. * Make the PSyclone API configurable. * Fixed formatting of documentation, properly used ArtefactSet names. * Support .f and .F Fortran files. * Removed setter for tool.is_available, which was only used for testing. * #3 Fix documentation and coding style issues from review. * Renamed Categories into Category. * Minor coding style cleanup. * Removed more unnecessary (). * Re-added (invalid) grab_pre_build call. * Fixed typo. * Renamed set_default_vendor to set_default_compiler_suite. * Renamed VendorTool to CompilerSuiteTool. * Also accept a Path as exec_name specification for a tool. * Move the check_available function into the base class. * Fixed some types and documentation. * Fix typing error. * Added explanation for meta-compiler. * Improved error handling and documentation. * Replace mpiifort with mpifort to be a tiny bit more portable. * Use classes to group tests for git/svn/fcm together. * Fixed issue in get_transformation script, and moved script into lfric_common to remove code duplication. * Code improvement as suggested by review. * Fixed run config * Added reference to ticket. * Updated type information. * More typing fixes. * Fixed typing warnings. * As requested by reviewer removed is_working_copy functionality. * Issue a warning (which can be silenced) when a tool in a toolbox is replaced. * Fixed flake8. * Fixed flake8. * Fixed failing test. * Addressed issues raised in review. * Removed now unnecessary operations. * Updated some type information. * Fixed all references to APIs to be consistent with PSyclone 2.5. * Added api to the checksum computation. * Fixed type information. * Added test to verify that changing the api changes the checksum. * Make compiler version a tuple of integers * Update some tests to use tuple versions * Explicitly test handling of bad version format * Fix formatting * Tidying up * Make compiler raise an error for any invalid version string Assume these compilers don't need to be hashed. Saves dealing with empty tuples. * Check compiler version string for compiler name * Fix formatting * Add compiler.get_version_string() method Includes other cleanup from PR comments * Add mpi and openmp settings to BuildConfig, made compiler MPI aware. * Looks like the circular dependency has been fixed. * Revert "Looks like the circular dependency has been fixed." ... while it works with the tests, a real application still triggered it. This reverts commit 150dc37. * Don't even try to find a C compiler if no C files are to be compiled. * Updated gitignore to ignore (recently renamed) documentation. * Fixed failing test. * Return from compile Fortran early if there are no files to compiles. Fixed coding style. * Add MPI enables wrapper for intel and gnu compiler. * Fixed test. * Automatically add openmp flag to compiler and linker based on BuildConfig. * Removed enforcement of keyword parameters, which is not supported in python 3.7. * Fixed failing test. * Support more than one tool of a given suite by sorting them. * Use different version checkout for each compiler vendor with mixins * Refactoring, remove unittest compiler class * Fix some mypy errors * Use 'Union' type hint to fix build checks * Return run_version_command to base Compiler class Provides default version command that can be overridden for other compilers. Also fix some incorrect tests Other tidying * Add a missing type hint * Remove inheritance from mixins and use protocol * Simplify compiler inheritance Mixins have static methods with unique names, overrides only happen in concrete classes * Simplify usage of compiler-specific parsing mixins. * Test for missing mixin. * Fixed test. * Added more tests for invalid version numbers. * Added more test cases for invalid version number, improved regex to work as expected. * Fixed typo in test. * Fixed test. * Split tests into smaller individual ones, fixed missing asssert in test. * Parameterised compiler version tests to also test wrapper. * Added missing MPI parameter when getting the compiler. * Fixed comments. * Order parameters to be in same order for various compiler classes. * Remove stray character * Made mpi and openmp default to False in the BuildConfig constructor. * Removed white space. * Support compilers that do not support OpenMP. * Added documentation for openmp parameter. --------- Co-authored-by: Junwei Lyu <[email protected]> Co-authored-by: Junwei Lyu <[email protected]> Co-authored-by: Junwei Lyu <[email protected]> Co-authored-by: Junwei Lyu <[email protected]> Co-authored-by: jasonjunweilyu <[email protected]> Co-authored-by: Luke Hoffmann <[email protected]> Co-authored-by: Luke Hoffmann <[email protected]> * Compiler wrapper (#347) * #3 Converted svn and fcm to tools. * #3 Fixed missing whitespace. * Modified the documentation for writing a config with PSyclone * Add config as a parameter for run_psyclone for the transformation_script to use;Updated the related functions and tests; Changed the logic of the transformation_script examples * #3 Replaced ar with tool object. * #3 Added tests for ar.py. * #3 Removed debug output. * #3 Converted PSyclone to be a tool. * #3 Removed debug print, fixed python 3.7 typing information. * #3 Updated comments. * Modified the get_optimisation_script function examples and updated the doc formatting * #3 Add Rsync tool. * #3 Removed now unused function. * #3 Added test for rsync. * #3 Fixed all mypy warnings about functions not checked. * #3 Replace all mock-tests to use subprocess so the name of the executable is tested as well. * #3 Remove duplicated flags. * #3 Fixed changed order of linking. * #3 Removed run_command function. * #3 Fixed 3.8 typing error. * #3 Fixed unused imports. * #3 Move flags checksum into Flags, and remove now unused tools.py file. * #3 Renamed newtools to tools. * #3 Made custom function for all git functions called (instead of just calling run). * #3 Updated and fixed comments. * #3 Fixed errors in comments. * Fixed minor errors in documentation. * #3 Make it easier to create wrapper around standard compiler. * #3 Added documentation for all tool related classes and their usage. * #3 Added MISC category. * Addressed reviewer's comments. * Updated cli to properly use ToolBox etc, removing hard-coded gnu command linker option. * Fixed mypy failures, including changes to import statement to avoid cyclic imports :(. * #3 Fix circular import. * Added #TODO so that this can be removed once fparser supports sentinels. * Fix typing problems by ignoring fparser. * Replaced more string names for artefacts with enums. * Removed EXECUTABLES from constants. * Moved Artefact class out of ArtefactStore and renamed it to ArtefactSet. * Moved OBJECT_FILES from constants into ArtefactSet. * Moved OBJECT_ARCHIVES from constants to ArtefactSet. * Moved PRAGMAD_C from constants to ArtefactSet. * Turned 'all_source' into an enum. * Allow integer as revision. * Fixed flake8 error. * Removed specific functions to add/get fortran source files etc. * Removed non-existing and unneccessary collections. * Try to fix all run_configs. * Fixed rebase issues. * Added replace functionality to ArtefactStore, updated test_artefacts to cover all lines in that file. * Started to replace artefacts when files are pre-processed. * Removed linker argument from linking step in all examples. * Try to get jules to link. * Fixed build_jules. * Fixed other issues raised in reviews. * Try to get jules to link. * Fixed other issues raised in reviews. * Simplify handling of X90 files by replacing the X90 with x90, meaning only one artefact set is involved when running PSyclone. * Make OBJECT_ARCHIVES also a dict, migrate more code to replace/add files to the default build artefact collections. * Fixed some examples. * Fix flake8 error. * Fixed failing tests. * Support empty comments. * Fix preprocessor to not unnecessary remove and add files that are already in the output directory. * Allow find_soure_files to be called more than once by adding files (not replacing artefact). * Updated lfric_common so that files created by configurator are written in build (not source). * Use c_build_files instead of pragmad_c. * Removed unnecessary str. * Documented the new artefact set handling. * Fixed typo. * Make the PSyclone API configurable. * Fixed formatting of documentation, properly used ArtefactSet names. * Support .f and .F Fortran files. * Removed setter for tool.is_available, which was only used for testing. * #3 Fix documentation and coding style issues from review. * Renamed Categories into Category. * Minor coding style cleanup. * Removed more unnecessary (). * Re-added (invalid) grab_pre_build call. * Fixed typo. * Renamed set_default_vendor to set_default_compiler_suite. * Renamed VendorTool to CompilerSuiteTool. * Also accept a Path as exec_name specification for a tool. * Move the check_available function into the base class. * Fixed some types and documentation. * Fix typing error. * Added explanation for meta-compiler. * Improved error handling and documentation. * Replace mpiifort with mpifort to be a tiny bit more portable. * Use classes to group tests for git/svn/fcm together. * Fixed issue in get_transformation script, and moved script into lfric_common to remove code duplication. * Code improvement as suggested by review. * Fixed run config * Added reference to ticket. * Updated type information. * More typing fixes. * Fixed typing warnings. * As requested by reviewer removed is_working_copy functionality. * Issue a warning (which can be silenced) when a tool in a toolbox is replaced. * Fixed flake8. * Fixed flake8. * Fixed failing test. * Addressed issues raised in review. * Removed now unnecessary operations. * Updated some type information. * Fixed all references to APIs to be consistent with PSyclone 2.5. * Added api to the checksum computation. * Fixed type information. * Added test to verify that changing the api changes the checksum. * Make compiler version a tuple of integers * Update some tests to use tuple versions * Explicitly test handling of bad version format * Fix formatting * Tidying up * Make compiler raise an error for any invalid version string Assume these compilers don't need to be hashed. Saves dealing with empty tuples. * Check compiler version string for compiler name * Fix formatting * Add compiler.get_version_string() method Includes other cleanup from PR comments * Add mpi and openmp settings to BuildConfig, made compiler MPI aware. * Looks like the circular dependency has been fixed. * Revert "Looks like the circular dependency has been fixed." ... while it works with the tests, a real application still triggered it. This reverts commit 150dc37. * Don't even try to find a C compiler if no C files are to be compiled. * Updated gitignore to ignore (recently renamed) documentation. * Fixed failing test. * Return from compile Fortran early if there are no files to compiles. Fixed coding style. * Add MPI enables wrapper for intel and gnu compiler. * Fixed test. * Automatically add openmp flag to compiler and linker based on BuildConfig. * Removed enforcement of keyword parameters, which is not supported in python 3.7. * Fixed failing test. * Support more than one tool of a given suite by sorting them. * Use different version checkout for each compiler vendor with mixins * Refactoring, remove unittest compiler class * Fix some mypy errors * Use 'Union' type hint to fix build checks * Added option to add flags to a tool. * Introduce proper compiler wrapper, used this to implement properly wrapper MPI compiler. * Fixed typo in types. * Return run_version_command to base Compiler class Provides default version command that can be overridden for other compilers. Also fix some incorrect tests Other tidying * Add a missing type hint * Added (somewhat stupid) 'test' to reach 100% coverage of PSyclone tool. * Simplified MPI support in wrapper. * More compiler wrapper coverage. * Removed duplicated function. * Removed debug print. * Removed permanently changing compiler attributes, which can cause test failures later. * More test for C compiler wrapper. * More work on compiler wrapper tests. * Fixed version and availability handling, added missing tests for 100% coverage. * Fixed typing error. * Try to fix python 3.7. * Tried to fix failing tests. * Remove inheritance from mixins and use protocol * Simplify compiler inheritance Mixins have static methods with unique names, overrides only happen in concrete classes * Updated wrapper and tests to handle error raised in get_version. * Simplified regular expressions (now tests cover detection of version numbers with only a major version). * Test for missing mixin. * Use the parsing mixing from the compiler in a compiler wrapper. * Use setattr instead of assignment to make mypy happy. * Simplify usage of compiler-specific parsing mixins. * Minor code cleanup. * Updated documentation. * Simplify usage of compiler-specific parsing mixins. * Test for missing mixin. * Fixed test. * Added missing openmp_flag property to compiler wrapper. * Don't use isinstance for consistency check, which does not work for CompilerWrappers. * Fixed isinstance test for C compilation which doesn't work with a CompilerWrapper. * Use a linker's compiler to determine MPI support. Removed mpi property from CompilerSuite. * Added more tests for invalid version numbers. * Added more test cases for invalid version number, improved regex to work as expected. * Fixed typo in test. * Fixed flake/mypy errors. * Combine wrapper flags with flags from wrapped compiler. * Made mypy happy. * Fixed test. * Split tests into smaller individual ones, fixed missing asssert in test. * Parameterised compiler version tests to also test wrapper. * Added missing MPI parameter when getting the compiler. * Fixed comments. * Order parameters to be in same order for various compiler classes. * Remove stray character * Added getter for wrapped compiler. * Fixed small error that would prevent nested compiler wrappers from being used. * Added a cast to make mypy happy. * Fixed spelling mistake in option. * Clarified documentation. * Removed unnecessary functions in CompilerWrapper. * Made mpi and openmp default to False in the BuildConfig constructor. * Removed white space. * Support compilers that do not support OpenMP. * Added documentation for openmp parameter. --------- Co-authored-by: Junwei Lyu <[email protected]> Co-authored-by: Junwei Lyu <[email protected]> Co-authored-by: jasonjunweilyu <[email protected]> Co-authored-by: Luke Hoffmann <[email protected]> Co-authored-by: Luke Hoffmann <[email protected]> * Improved MPI/OpenMP support (#336) * Recovering anything of value from the old tests (#339) * Skip test dependent on clang module if it is not installed. * Inherit slightly more extensive testing of C pragma injector. * Migrated some real-life testing of Subversion interface. * Updated comment. * Just remove old tests. * Reverted mistaken clean-up. * Removed unused import. --------- Co-authored-by: Junwei Lyu <[email protected]> Co-authored-by: Junwei Lyu <[email protected]> Co-authored-by: Junwei Lyu <[email protected]> Co-authored-by: Junwei Lyu <[email protected]> Co-authored-by: jasonjunweilyu <[email protected]> Co-authored-by: Luke Hoffmann <[email protected]> Co-authored-by: Luke Hoffmann <[email protected]> Co-authored-by: Matthew Hambley <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds configuration settings that enable/disable MPI and OpenMP to BuildConfig.
This means that an application can now specify that it should be compiled with OpenMP, without having to specify compiler-specific OpenMP flags - the Compiler object will get the information from the BuildConfig if it should add the flag or not.
Similarly, if an MPI build is requested, the tool repository will return an MPI enabled compiler (but the user can always overwrite this by hand-selecting a compiler explicitly).
Note that the MPI compiler wrapper added here will get a nicer (decorator-based) implementation in the next PR, I just tried to keep the PRs small. This PR is still quite big, but that's because I made the MPI and OpenMP settings mandatory (imho it makes sense to get the user to specify what exactly they want). The CLI defaults both to False. Additionally, I started to fix the line length (according to the discussion in #323