Improve artefact workflow

Documentation/source/advanced_config.rst

@@ -254,7 +254,7 @@ in particular where it creates files within it.
                 *.o (compiled object files)
                 *.mod (mod files)
           metrics/
-          my_program.exe
+          my_program
           log.txt
 
 The *project workspace* folder takes its name from the project label passed in to the build configuration.

Documentation/source/glossary.rst

@@ -29,7 +29,7 @@ Glossary
         Fab's built-in steps come with sensible defaults so the user doesn't have to write unnecessary config.
 
         As an example, the Fortran preprocessor has a default artefact getter which reads *".F90"* files
-        from the :term:`Artefact Collection` called ``"all_source"``.
+        from the :term:`Artefact Collection` called ``"INITIAL_SOURCE"``.
 
         Artefact getters are derived from :class:`~fab.artefacts.ArtefactsGetter`.
 
@@ -65,7 +65,7 @@ Glossary
         A folder inside the :term:`Fab Workspace`, containing all source and output from a build config.
 
     Root Symbol
-        The name of a Fortran PROGRAM, or ``"main"`` for C code. Fab builds an exe for every root symbol it's given.
+        The name of a Fortran PROGRAM, or ``"main"`` for C code. Fab builds an executable for every root symbol it's given.
 
     Source Tree
         The :class:`~fab.steps.analyse.analyse` step produces a dependency tree of the entire project source.

Documentation/source/writing_config.rst

@@ -79,7 +79,7 @@ Please see the documentation for :func:`~fab.steps.find_source_files.find_source
 including how to exclude certain source code from the build. More grab steps can be found in the :mod:`~fab.steps.grab`
 module.
 
-After the find_source_files step, there will be a collection called ``"all_source"``, in the artefact store.
+After the find_source_files step, there will be a collection called ``"INITIAL_SOURCE"``, in the artefact store.
 
 .. [1] See :func:`~fab.steps.c_pragma_injector.c_pragma_injector` for an example of a step which
     creates artefacts in the source folder.
@@ -94,7 +94,7 @@ which must happen before we analyse it.
 
 Steps generally create and find artefacts in the :term:`Artefact Store`, arranged into named collections.
 The :func:`~fab.steps.preprocess.preprocess_fortran`
-automatically looks for Fortran source code in a collection named `'all_source'`,
+automatically looks for Fortran source code in a collection named `'INITIAL_SOURCE'`,
 which is the default output from the preceding :funcfind_source_files step.
 It filters just the (uppercase) ``.F90`` files.
 
@@ -179,7 +179,7 @@ before you run the :func:`~fab.steps.analyse.analyse` step below.
 
 
 After the psyclone step, two new source files will be created for each .x90 file in the `'build_output'` folder.
-These two output files will be added under ``"psyclone_output"`` collection to the artefact store.
+These two output files will be added under ``FORTRAN_BUILD_FILES`` collection to the artefact store.
 
 
 .. _Analyse Overview:
@@ -190,11 +190,10 @@ Analyse
 We must :func:`~fab.steps.analyse.analyse` the source code to determine which
 Fortran files to compile, and in which order.
 
-The Analyse step looks for source to analyse in several collections:
+The Analyse step looks for source to analyse in two collections:
 
-* ``.f90`` found in the source
-* ``.F90`` we pre-processed into ``.f90``
-* preprocessed c
+* ``FORTRAN_BUILD_FILES``, which contains all ``.f90`` found in the source, all ``.F90`` files we pre-processed into ``.f90``, and files created by any additional step (e.g. PSyclone).
+* ``C_BUILD_FILES``, all preprocessed c files.
 
 .. code-block::
     :linenos:
@@ -227,14 +226,14 @@ The Analyse step looks for source to analyse in several collections:
 Here we tell the analyser which :term:`Root Symbol` we want to build into an executable.
 Alternatively, we can use the ``find_programs`` flag for Fab to discover and build all programs.
 
-After the Analyse step, there will be a collection called ``"build_trees"``, in the artefact store.
+After the Analyse step, there will be a collection called ``BUILD_TREES``, in the artefact store.
 
 
 Compile and Link
 ================
 
 The :func:`~fab.steps.compile_fortran.compile_fortran` step compiles files in
-the ``"build_trees"`` collection. The :func:`~fab.steps.link.link_exe` step
+the ``BUILD_TREES`` collection. The :func:`~fab.steps.link.link_exe` step
 then creates the executable.
 
 .. code-block::
@@ -269,7 +268,42 @@ then creates the executable.
             link_exe(state)
 
 
-After the :func:`~fab.steps.link.link_exe` step, the executable name can be found in a collection called ``"executables"``.
+After the :func:`~fab.steps.link.link_exe` step, the executable name can be found in a collection called ``EXECUTABLES``.
+
+ArtefactStore
+=============
+Each build configuration contains an artefact store, containing various
+sets of artefacts. The artefact sets used by Fab are defined in the
+enum :class:`~fab.artefacts.ArtefactSet`. The most important sets are ``FORTRAN_BUILD_FILES``, 
+``C_BUILD_FILES``, which will always contain all known source files that
+will need to be analysed for dependencies, compiled, and linked. All existing
+steps in Fab will make sure to maintain these artefact sets consistently,
+for example, if a ``.F90`` file is preprocessed, the ``.F90`` file in
+``FORTRAN_BUILD_FILES`` will be replaced with the corresponding preprocessed
+``.f90`` file. Similarly, new files (for examples created by PSyclone)
+will be added to ``FORTRAN_BUILD_FILES``. A user script can adds its own
+artefacts using strings as keys if required.
+
+The exact flow of artefact sets is as follows. Note that any artefact
+sets mentioned here can typically be overwritten by the user, but then
+it is the user's responsibility to maintain the default artefact sets
+(or change them all):
+
+..
+  My apologies for the LONG lines, they were the only way I could find
+  to have properly indented paragraphs :(
+
+1. :func:`~fab.steps.find_source_files.find_source_files` will add all source files it finds to ``INITIAL_SOURCE`` (by default, can be overwritten by the user). Any ``.F90`` and ``.f90`` file will also be added to ``FORTRAN_BUILD_FILES``, any ``.c`` file to ``C_BUILD_FILES``, and any ``.x90`` or ``.X90`` file to ``X90_BUILD_FILES``. It can be called several times if files from different root directories need to be added, and it will automatically update the ``*_BUILD_FILES`` sets.
+2. Any user script that creates new files can add files to ``INITIAL_SOURCE`` if required, but also to the corresponding ``*_BUILD_FILES``. This will happen automatically if :func:`~fab.steps.find_source_files.find_source_files` is called to add these newly created files.
+3. If :func:`~fab.steps.c_pragma_injector.c_pragma_injector` is being called, it will handle all files in ``C_BUILD_FILES``, and will replace all the original C files with the newly created ones. For backward compatibility it will also store the new objects in the ``PRAGMAD_C`` set.
+4. If :func:`~fab.steps.preprocess.preprocess_c` is called, it will preprocess all files in ``C_BUILD_FILES`` (at this stage typically preprocess the files in the original source folder, writing the output files to the build folder), and update that artefact set accordingly. For backward compatibility it will also store the preprocessed files in ``PREPROCESSED_C``.
+5. If :func:`~fab.steps.preprocess.preprocess_fortran` is called, it will preprocess all files in ``FORTRAN_BUILD_FILES`` that end on ``.F90``, creating new ``.f90`` files in the build folder. These files will be added to ``PREPROCESSED_FORTRAN``. Then the original ``.F90`` are removed from ``FORTRAN_BUILD_FILES``, and the new preprocessed files (which are in ``PREPROCESSED_FORTRAN``) will be added. Then any ``.f90`` files that are not already in the build folder (an example of this are files created by a user script) are copied from the original source folder into the build folder, and ``FORTRAN_BUILD_FILES`` is updated to use the files in the new location.
+6. If :func:`~fab.steps.psyclone.preprocess_x90` is called, it will similarly preprocess all ``.X90`` files in ``X90_BUILD_FILES``, creating the output files in the build folder, and replacing the files in ``X90_BUILD_FILES``.
+7. If :func:`~fab.steps.psyclone.psyclone` is called, it will process all files in ``X90_BUILD_FILES`` and add any newly created file to ``FORTRAN_BUILD_FILES``, and removing them from ``X90_BUILD_FILES``.
+8. The :func:`~fab.steps.analyse.analyse` step analyses all files in ``FORTRAN_BUILD_FILES`` and ``C_BUILD_FILES``, and add all dependencies to ``BUILD_TREES``.
+9. The :func:`~fab.steps.compile_c.compile_c` and :func:`~fab.steps.compile_fortran.compile_fortran` steps will compile all files from ``C_BUILD_FILES`` and ``FORTRAN_BUILD_FILES``, and add them to ``OBJECT_FILES``.
+10. If :func:`~fab.steps.archive_objects.archive_objects` is called, it will create libraries based on ``OBJECT_FILES``, adding the libraries to ``OBJECT_ARCHIVES``.
+11. If :func:`~fab.steps.link.link_exe` is called, it will either use ``OBJECT_ARCHIVES``, or if this is empty, use ``OBJECT_FILES``, create the binaries, and add them to ``EXECUTABLES``.
 
 
 Flags

run_configs/lfric/atm.py

@@ -1,4 +1,8 @@
 #!/usr/bin/env python3
+
+'''Example LFRic_atm build script.
+'''
+
 import logging
 
 from fab.build_config import BuildConfig, AddFlags
@@ -16,7 +20,8 @@
 from fab.tools import ToolBox
 
 from grab_lfric import lfric_source_config, gpl_utils_source_config
-from lfric_common import configurator, fparser_workaround_stop_concatenation
+from lfric_common import (configurator, fparser_workaround_stop_concatenation,
+                          get_transformation_script)
 
 logger = logging.getLogger('fab')
 
@@ -162,26 +167,6 @@ def file_filtering(config):
     ]
 
 
-def get_transformation_script(fpath, config):
-    ''':returns: the transformation script to be used by PSyclone.
-    :rtype: Path
-
-    '''
-    optimisation_path = config.source_root / 'optimisation' / 'meto-spice'
-    for base_path in [config.source_root, config.build_output]:
-        try:
-            relative_path = fpath.relative_to(base_path)
-        except ValueError:
-            pass
-    local_transformation_script = optimisation_path / (relative_path.with_suffix('.py'))
-    if local_transformation_script.exists():
-        return local_transformation_script
-    global_transformation_script = optimisation_path / 'global.py'
-    if global_transformation_script.exists():
-        return global_transformation_script
-    return ""
-
-
 if __name__ == '__main__':
     lfric_source = lfric_source_config.source_root / 'lfric'
     gpl_utils_source = gpl_utils_source_config.source_root / 'gpl_utils'

run_configs/lfric/gungho.py

@@ -4,6 +4,10 @@
 #  For further details please refer to the file COPYRIGHT
 #  which you should have received as part of this distribution
 # ##############################################################################
+'''
+A simple build script for gungho_model
+'''
+
 import logging
 
 from fab.build_config import BuildConfig
@@ -18,31 +22,12 @@
 from fab.tools import ToolBox
 
 from grab_lfric import lfric_source_config, gpl_utils_source_config
-from lfric_common import configurator, fparser_workaround_stop_concatenation
+from lfric_common import (configurator, fparser_workaround_stop_concatenation,
+                          get_transformation_script)
 
 logger = logging.getLogger('fab')
 
 
-def get_transformation_script(fpath, config):
-    ''':returns: the transformation script to be used by PSyclone.
-    :rtype: Path
-
-    '''
-    optimisation_path = config.source_root / 'optimisation' / 'meto-spice'
-    for base_path in [config.source_root, config.build_output]:
-        try:
-            relative_path = fpath.relative_to(base_path)
-        except ValueError:
-            pass
-    local_transformation_script = optimisation_path / (relative_path.with_suffix('.py'))
-    if local_transformation_script.exists():
-        return local_transformation_script
-    global_transformation_script = optimisation_path / 'global.py'
-    if global_transformation_script.exists():
-        return global_transformation_script
-    return ""
-
-
 if __name__ == '__main__':
     lfric_source = lfric_source_config.source_root / 'lfric'
     gpl_utils_source = gpl_utils_source_config.source_root / 'gpl_utils'

run_configs/lfric/lfric_common.py

@@ -1,9 +1,13 @@
 import logging
 import os
 import shutil
+from typing import Optional
 from pathlib import Path
 
+from fab.artefacts import ArtefactSet
+from fab.build_config import BuildConfig
 from fab.steps import step
+from fab.steps.find_source_files import find_source_files
 from fab.tools import Category, Tool
 
 logger = logging.getLogger('fab')
@@ -21,79 +25,113 @@ def check_available(self):
         return True
 
 
-# todo: is this part of psyclone? if so, put  it in the psyclone step module?
+# ============================================================================
 @step
 def configurator(config, lfric_source: Path, gpl_utils_source: Path, rose_meta_conf: Path, config_dir=None):
 
     rose_picker_tool = gpl_utils_source / 'rose_picker/rose_picker'
     gen_namelist_tool = lfric_source / 'infrastructure/build/tools/GenerateNamelist'
     gen_loader_tool = lfric_source / 'infrastructure/build/tools/GenerateLoader'
     gen_feigns_tool = lfric_source / 'infrastructure/build/tools/GenerateFeigns'
-
     config_dir = config_dir or config.source_root / 'configuration'
+    config_dir.mkdir(parents=True, exist_ok=True)
 
     env = os.environ.copy()
     rose_lfric_path = gpl_utils_source / 'lib/python'
     env['PYTHONPATH'] += f':{rose_lfric_path}'
 
-    # "rose picker"
-    # creates rose-meta.json and config_namelists.txt in gungho/source/configuration
+    # rose picker
+    # -----------
+    # creates rose-meta.json and config_namelists.txt in
+    # gungho/build
     logger.info('rose_picker')
     rose_picker = Script(rose_picker_tool)
-    rose_picker.run(additional_parameters=[str(rose_meta_conf),
-                                           '-directory', str(config_dir),
+    rose_picker.run(additional_parameters=[rose_meta_conf,
+                                           '-directory', config_dir,
                                            '-include_dirs', lfric_source],
                     env=env)
+    rose_meta = config_dir / 'rose-meta.json'
 
-    # "build_config_loaders"
+    # build_config_loaders
+    # --------------------
     # builds a bunch of f90s from the json
     logger.info('GenerateNamelist')
     gen_namelist = Script(gen_namelist_tool)
-    gen_namelist.run(additional_parameters=['-verbose',
-                                            str(config_dir / 'rose-meta.json'),
-                                            '-directory', str(config_dir)])
+    gen_namelist.run(additional_parameters=['-verbose', rose_meta,
+                                            '-directory', config_dir],
+                     cwd=config_dir)
 
     # create configuration_mod.f90 in source root
+    # -------------------------------------------
     logger.info('GenerateLoader')
+    names = [name.strip() for name in
+             open(config_dir / 'config_namelists.txt').readlines()]
+    configuration_mod_fpath = config_dir / 'configuration_mod.f90'
     gen_loader = Script(gen_loader_tool)
-    names = [name.strip() for name in open(config_dir / 'config_namelists.txt').readlines()]
-    configuration_mod_fpath = config.source_root / 'configuration_mod.f90'
     gen_loader.run(additional_parameters=[configuration_mod_fpath,
                                           *names])
 
     # create feign_config_mod.f90 in source root
+    # ------------------------------------------
     logger.info('GenerateFeigns')
-    feign_config = Script(gen_feigns_tool)
-    feign_config_mod_fpath = config.source_root / 'feign_config_mod.f90'
-    feign_config.run(additional_parameters=[str(config_dir / 'rose-meta.json'),
-                                            '-output', feign_config_mod_fpath])
+    feign_config_mod_fpath = config_dir / 'feign_config_mod.f90'
+    gft = Script(gen_feigns_tool)
+    gft.run(additional_parameters=[rose_meta,
+                                   '-output', feign_config_mod_fpath])
 
-    # put the generated source into an artefact
-    # todo: we shouldn't need to do this, should we?
-    #       it's just going to be found in the source folder with everything else.
-    config._artefact_store['configurator_output'] = [
-        configuration_mod_fpath,
-        feign_config_mod_fpath
-    ]
+    find_source_files(config, source_root=config_dir)
 
 
+# ============================================================================
 @step
 def fparser_workaround_stop_concatenation(config):
     """
-    fparser can't handle string concat in a stop statement. This step is a workaround.
+    fparser can't handle string concat in a stop statement. This step is
+    a workaround.
 
     https://github.com/stfc/fparser/issues/330
 
     """
-    feign_config_mod_fpath = config.source_root / 'feign_config_mod.f90'
+    feign_path = None
+    for file_path in config.artefact_store[ArtefactSet.FORTRAN_BUILD_FILES]:
+        if file_path.name == 'feign_config_mod.f90':
+            feign_path = file_path
+            break
+    else:
+        raise RuntimeError("Could not find 'feign_config_mod.f90'.")
 
     # rename "broken" version
-    broken_version = feign_config_mod_fpath.with_suffix('.broken')
-    shutil.move(feign_config_mod_fpath, broken_version)
+    broken_version = feign_path.with_suffix('.broken')
+    shutil.move(feign_path, broken_version)
 
     # make fixed version
     bad = "_config: '// &\n        'Unable to close temporary file'"
     good = "_config: Unable to close temporary file'"
 
-    open(feign_config_mod_fpath, 'wt').write(
+    open(feign_path, 'wt').write(
         open(broken_version, 'rt').read().replace(bad, good))
+
+
+# ============================================================================
+def get_transformation_script(fpath: Path,
+                              config: BuildConfig) -> Optional[Path]:
+    ''':returns: the transformation script to be used by PSyclone.
+    '''
+
+    optimisation_path = config.source_root / 'optimisation' / 'meto-spice'
+    relative_path = None
+    for base_path in [config.source_root, config.build_output]:
+        try:
+            relative_path = fpath.relative_to(base_path)
+        except ValueError:
+            pass
+    if relative_path:
+        local_transformation_script = (optimisation_path /
+                                       (relative_path.with_suffix('.py')))
+        if local_transformation_script.exists():
+            return local_transformation_script
+
+    global_transformation_script = optimisation_path / 'global.py'
+    if global_transformation_script.exists():
+        return global_transformation_script
+    return None

run_configs/um/build_um.py

@@ -13,9 +13,8 @@
 import re
 import warnings
 
-from fab.artefacts import CollectionGetter
+from fab.artefacts import ArtefactSet, CollectionGetter
 from fab.build_config import AddFlags, BuildConfig
-from fab.constants import PRAGMAD_C
 from fab.steps import step
 from fab.steps.analyse import analyse
 from fab.steps.archive_objects import archive_objects
@@ -177,7 +176,7 @@ def replace_in_file(inpath, outpath, find, replace):
 
         preprocess_c(
             state,
-            source=CollectionGetter(PRAGMAD_C),
+            source=CollectionGetter(ArtefactSet.C_BUILD_FILES),
             path_flags=[
                 # todo: this is a bit "codey" - can we safely give longer strings and split later?
                 AddFlags(match="$source/um/*", flags=[