Contributing to the Fermitools
Thank you for contributing to the Fermitools project and making Fermi data analysis even better! This page is meant to assist developers who wish to work on or test the Fermitools package as a whole. If you wish to contribute code to individual repositories you can do so using standard git methodology.
Note: This page is meant to act as a quick development environment setup guide. For best practices and procedures with regards to Fermitools development it is HIGHLY RECOMMENDED that you read the Fermi Developer Documentation.
Developers wishing to develop packages in the Fermitools should follow the workflow outlined below:
Except in cases where developers are debugging issues with the Conda Build packaging process itself, the process illustrated on the right side of the diagram should be followed. In summary,
- Checkout the tools using a
repomaninvocation as outlined above - Make any source code changes in the checked out code
- Compile the code using SCons to make sure that the code compiles without errors
- Check the code into a branch and issue a Pull Request to alert the FSSC that you would like to merge changes back into the
masterbranch of a Fermitools package repository
If building the tools on a local machine from source, refer to the updated Build-from-Source-(Developer-Workflow) wiki page and the associated helper utilities at https://github.com/fermi-lat/fermi_dev_scripts. If however you want to do it the hard way, keep reading.
To develop the tools locally using SCons one must first install Fermitools' dependencies into a conda environment. To do this without installing the tools themselves execute:
conda create --name dev --only-deps -c conda-forge/label/cf201901 -c fermi fermitools
The above command will create a new environment named dev and install the Fermitools runtime dependencies into it without installing the Fermitools executables or libraries themselves. The installed libraries and headers can be found in the directory
${CONDA_PREFIX}/envs/dev/
Note: Because the --deps-only flag installs the runtime dependencies of the Fermitools as opposed to the build dependencies, you may need to install additional libraries into your development environment.
In order to checkout the multiple repositories which make up the Fermitools you must use the repoman tool set up by Brian Van Klaveran. Repoman is a tool which checks out container packages (such as ScienceTools or GlastRelease) along with their related dependencies. To get repoman, one need only to install the Fermi version of it via pip:
pip install fermi-repoman
It is recommended that you have SSH keys set up for use with your Github account. For instructions on how to do that see the Github documentation on adding a new SSH key to your GitHub account.
If you prefer to use https protocol you will need to specify the remote base you wish repoman to use by either setting the REMOTE_BASE environment variable or providing repoman the flag --remote-base followed by the location of the fermi-lat github organization (i.e. --remote-base https://github.com/fermi-lat).
Repoman has a number of flags which can be passed to it to customize its operation. For the sake of brevity this document will only cover what you need to know to check out the Fermitools, however, if you would like to know more about repoman's operation you can reference the official repoman documentation.
To check out the most recent master branches of all of the Fermitools packages use the following command:
repoman checkout --develop ScienceTools conda
The --develop flag tells repoman to grab the master branch versions of all the ScienceTools constituant packages. In order to grab any changes that are specific to Fermitools (the ScienceTools conda distribution) the conda branch name is appended to the end of the command.
Repoman grabs branches in a superseding order from left to right. The --develop flag tells repoman to start with the master branch of every repository. Adding conda at the end of the command tells repoman to check each repository for a branch named 'conda' (which has been used for conda development/patches by the dev team) and check that out instead if it exists. This is an extremely useful feature in the common case where you may have a test branch (hypothetically named patch-1) located in one or more repositories that you wish to check out if it exists instead of the master branch. In that case you could use the command repoman checkout --develop ScienceTools conda patch-1. The branch hierarchy for that checkout would therefore be patch-1>conda>master.
SCons Build (No longer supported. See https://github.com/fermi-lat/fermi_dev_scripts)
After obtaining the pre-compiled dependencies from conda and desired source packages with repoman you can now perform a local build using SCons. For example you can set an environment variable $PREFIX to the location of your anaconda dev environment and run:
scons -C ScienceTools --site-dir=../SConsShared/site_scons --conda=$PREFIX --use-path all
to build the entire suite of the tools. SCons can be invoked either within a specific package's directory to compile only that package or within the ScienceTools directory to compile the entirety of the Fermitools. SCons will need to be provided with the locations of libraries which are necessary dependencies for the package that it is attempting to build via the --conda argument.
Further information about using SCons can be found at the SLAC Sciencetools Developer's page
Fermitools is integrated into a distributable package using the Conda Build utility. When developing the Fermitools distribution in a holistic manner it is often very useful to run checkouts and builds entirely via conda build.
To obtain conda build, simply install it via conda install. It is recommended that developers use the version of conda build that is available via the conda-forge channel as opposed to the default conda channel. This version can be obtained via this command:
conda install -c conda-forge conda-build
If you are attempting to debug problems with Fermitools Conda Build packaging, you will need to compile the tools using the conda build utility as opposed to invoking SCons directly. To build the Fermitools with Conda Build you must first clone the Fermitools-conda git repository located here:
https://github.com/fermi-lat/Fermitools-conda
This repository contains a number of files which tell conda build how to checkout, compile, and package the Fermitools. Each file will be covered briefly here, however, it is highly recommended that any developer of the Fermitools conda distribution familiarize themselves with conda's documentation on building packages.
activate.csh/activate.sh --- These scripts are run when the environment that the Fermitools are installed in is activated. They set a variety of environment variables, alias, and run checks which are required for the safe execution of the Fermitools. There are two versions of this script in order to ensure compatibility with both C/TC Shell and Bash Shell.
meta.yaml --- This file, written in yaml, provides conda build with metadata information on the package it is building along with a list of required dependencies for building and running the produced binaries. More information on the meta.yaml file can be found in the conda build documentation.
build.sh --- This script provides all of the build instructions that conda build executes when run in the recipe directory, including source code checkout, compiler flag setting, and compilation via SCons.
deactivate.csh/deactivate.sh --- These scripts are run when the environment Fermitools is in is deactivated. In a nutshell they undo all of the changes made by the corresponding activate scripts.
After installing conda build and cloning the ScienceTools-conda-recipe repository a build can be executed by entering the repository and running the conda build command:
conda build --python=3.9 -c conda-forge -c fermi .
This command tells conda build to pull required dependencies as outlined in the meta.yaml file from the conda-forge/label/cf201901 and fermi channels.
The build takes a little while to execute so sit tight as conda build does its stuff. Conda build will install dependencies (as outlined in meta.yaml), execute the build.sh script, and package the compiled binaries with necessary metadata (again from the meta.yaml file). At the end of the build, should it complete successfully, conda build will display an upload command with a path to the location of the build. Congratulations! You've just built and packaged the Fermitools conda distribution. The upload command will upload the packaged binaries to the Anaconda Cloud. Don't do this unless you know what you are doing and/or have the blessing of the Fermi-LAT software development team.
When we decide to generate a build for release an FSSC developer will update the version string of Fermitools to to reflect what changes have been made. This change will occur in the Fermitools-conda meta.yaml file, which updates the metadata of the binary which will be uploaded to the Fermi Anaconda Cloud channel for distribution, and will be used in the tagging of the Fermitools constituent package repositories in the Fermi-Lat github.com organization using repoman's mass tagging ability. Release tags will be of the form: Fermitools-x.x.x
The first public release started the Fermitools at version Fermitools-1.0.0. The FSSC uses the convention:
v.r.p->version.revision.patch->Major Update.Minor Update.Patch
Major Update: Iteration implies large functionality changes, new tools introduced, api changes, or other major change
Minor Update: Iteration implies new data files/models, updates to specific individual tools, technical changes that obviously change the user interface (to a smaller extent than a major change, etc.)
Patch: Iteration implies bug fixes/patches to technical problems brought to the attention of the developers
Tagging will occur at three distinct stages within the development process: Automatically by the build pipeline during PR-triggered builds, manually by FSSC staff when promoting a build to the testing stage, and manually by the FSSC staff when formally releasing a build to the public.
After a PR-triggered build completes in the development pipeline the built code will automatically be checked out and tagged with a tag of the form Fermitools-xx.xx.xx-$BuildNumber. The BuildNumber variable is automatically generated by the pipeling during the build process and is an incrementing integer that distinguishes one build from another. These can be seen as the last part of the filenames on the Fermi Anaconda Cloud Channel's Files page.
Once a build has been deemed stable enough to proceed to a pre-release testing stage, the codebase included in the build will be manually tagged by the FSSC staff with a tag of the form Fermitools-xx.xx.xx-Dev. While code with this tag can be viewed as more stable than code with just the above pipeline tag, it should not be considered release-ready.
Releases will be tagged just before release to the public by FSSC staff with a tag of the form Fermitools-xx.xx.xx. Code with this tag is production-ready.
Developers may tag their individual repositories with any schema they like. However, if they would like a specific tag included in a build the packageList.txt file in the ScienceTools Repository must be updated to include the new tag. The proper procedure for this is to
- Create a branch of the repository and modify the file there
- Alert FSSC staff about the requested change
- Submit a PR to pull the change into master
This file specifies the default tags that repoman pulls when performing a Fermitools checkout. Once a release occurs all listed tags in the
packagelist.txtwill be updated to the last release tag.
To add a new spectral model to the tools the following repositories need to be updated:
- The model definitions must be added to a header file and committed to the
Likelihood/Likelihooddirectory. - The implementation of the model functions must be added to a
.cxxfile and committed to theLikelihood/srcdirectory. -
Likelihood/src/AppHelpers.cxxmust be modified to add the new header file for the model as a#includereference. Anm_funcfactoryreference to the new the new model in functionGtExposure::prepareFunctionFactory()also needs to be added. Use the previous model entries as templates.
-
pyLikelihood/src/pyLikelihood.ishould be modified to include the model in two places. First, it should '#include' the header file that define the model; second you should also add a '%include Likelihood/Filename.h' directive to instruct swig to include the file in the python bindings.
pyExposure/src/gtexposure/gtexposure.cxx
This file needs to have the new header file for the model added as a #include reference. An funcfactory reference to the new the new model in function addFunctionPrototypes(optimizers::FunctionFactory * funcFactory) also needs to be added. Use the previous model entries as templates.
ModelEditor needs to be updated in order to include the new model. This is done by updating the Spectrum.py file in the modelEditor package. The Spectrum class in that file has a _defaultParameters object that lists all the models and the values of their default parameters. Simply add the new model to this list.
The FSSC Source Model Definitions page will need to be updated with the mathematical model definition (in latex) and the XML model definition.
The Fermitools use a calibration database that is fully encapsulated within the software itself. The database itself is located within the irfs repository in irfs/caldb.
Note: The caldb_release directory is deprecated and will be removed from the master branch in future updates. Right now it is maintained for historical reasons related to the glastrelease build.
In order to update the CALDB the following steps must be taken in order. All changes should be made on a separate branch and thoroughly tested before merging.
To update the CALDB one must have the FTOOLS installed, as the tool udcif must be accessible for this process
The IRFs are kept within three subdirectories of irfs/caldb/CALDB/data/glast/lat/bcf/: ea, edisp, and psf. Add the new irfs to the appropriate subdirectories. Based on nomenclature it should be obvious which irfs go in which directory. The new irfs should comply with the existing naming scheme. This is very important. If the new irf names are in a different format than the existing irfs this must be corrected before updating the caldb.
The caldb.indx file provides the caldb with a complete list of valid irfs and their properties. In order for the caldb to know that the new irfs exist and properly select them on demand this file needs to be updated with the new irfs. There is a script named update_caldb_indx.py in irfs/caldb/python which can be used to assist this process. To use the script, update it first with the suffix of the new files (e.g. "P8R3_SOURCE_V3"). After updating the suffixes, call the script from the irfs/caldb/CALDB/data/glast/lat/bcf/ directory to have it iterate through the irf directories and update the caldb.indx file.
The irf_index.fits file in contains a list of IRFs and their evclass/evtype bitmask value conversions. The make_irf_index.py in irfs/caldb/python assists in updating this file. To use the script, update the irf_index.txt file in the python directory to include the new irfs. Maintain the order of the irfs to ensure that the correct values are assigned to the irfs.
Run the script in the python directory. The script will read the list of irfs and mappings from the text file and produce a file named irf_index_candidate.fits. Verify that this file has been updated properly using a tools such as fv and, once verified, rename it to irf_index.fits and replace the old version of the file in irfs/caldc/CALDB/data/glast/lat/bcf/ with this new version.
Sometimes caldb index update script splits the path of the new irfs in the filepath column of the updated caldb.indx. This can cause issues with the caldb finding the locations of files.
Make sure that the irf_index and caldb.indx agree in their filelist. If files appear in one but not the other the caldb will crash when attempting to find valid irfs.