diff --git a/lib/spack/docs/environments.rst b/lib/spack/docs/environments.rst new file mode 100644 index 00000000000000..5ae543072b9d70 --- /dev/null +++ b/lib/spack/docs/environments.rst @@ -0,0 +1,317 @@ +.. _environments: + +Environments +============ + +An environment is used to group together a set of specs for the +purpose of building, rebuilding and deploying in a coherent fashion. +Environments provide a number of advantages over the the a la carte +approach of building and loading individual Spack modules: + +#. Environments eliminate the non-determinism encountered when + installing packages and then loading what was just created. + Consider the following sequence: + + .. code-block:: console + + $ spack install mypackage + $ spack load mypackage + + Sometime later, somebody else installs an earlier version of ``mypackage``: + + .. code-block:: console + + $ spack install mypackage@0.9 + + Now the ``spack load mypackage`` command will fail because more + than one version of ``mypackage`` is installed. Thus, ``spack + load`` is non-deterministic, and not a reliable way to load + specific sets of modules one wishes to build and load. + Environments solve this problem by creating a localized context + that is immune to what other versions of packages might be + installed in the global Spack prefix. +#. Environments separate the steps of (a) choosing what to + install, (b) concretizing, and (c) installing. This allows + Environments to remain stable and repeatable, even if Spack packages + are upgraded: specs are only re-concretized when the user + explicitly asks for it. It should even be possible to reliably + transport environments between different computers running + different versions of Spack! +#. Since re-concretization only occurs at controlled times, the + Environment equivalent of many Spack commands are far faster. For + example, ``spack env list -Ilr`` vs. ``spack spec``. +#. Environments allow several specs to be built at once; a more robust + solution than ad-hoc scripts making multiple calls to ``spack + install``. +#. An Environment that is built as a whole can be loaded as a whole. + Spack can generate a script of ``module load`` commands that load + the appropraite environment. And that script will work, without + running Spack, as long as the underlying modules remain installed. + A fast-running ``spack env load`` command would also be + possible. + +Other packaging systems also provide environments that are similar in +some ways to Spack environments; for example, `Conda environments +`_ or +`Python Virtual Environments +`_. Spack environments +some distinctive features: + +#. A spec installed "in" an environment is no different form the same + spec installed anywhere else in Spack. Environments are assembled + simply by collecting together a set of specs. Environments are + loaded by generating a set of ``module load`` commands. (In the + future, environments can also be assembled into a single tree using + techniques from ``spack view``.) +#. Spack Environments may contain more than one version of the same + package; but only a single module for a package may be loaded. + Modules that occur in earlier specs listed in an environment take + precedence over modules that occur later. + +Using Environments +------------------ + +Here we follow a typical use case of creating, concretizing, +installing and loading an environment. + +.. note:: + + Environments are currently in Beta. We expect the user interface + to change. Future changes might also break existing environment + files, particularly ``environment.json``. + + +Creating an Environment +^^^^^^^^^^^^^^^^^^^^^^^ + +An environment is created by: + +.. code-block:: console + + $ spack env myenv create + +Spack then creates the following files: + +* The directory ``var/spack/environments/myenv``. This is the public + part of the environment. + + .. note:: + + All environments are stored in the ``var/spack/environments`` folder. + + The following files may be added to this directory by the user or + Spack: + + * ``env.yaml``: Addition environment specification and configuration + * ``loads``: A script of ``module load`` commands, generated by + ``spack env myenv loads``, which the user may source to use the + built environment. + * ``logs/``: Symbolic links to build logs for specs that are part of + the environment. + * Anything else the user wishes to put in this directory! + +* The directory ``var/spack/environments/myenv/.env``. This directory + is "owned" by Spack, and may be rewritten from time to time. The + user should not put anything in it. It contains: + + * ``environment.json``: Concretized specs for the environment. + * ``repo/``: A repo consisting of the Spack packages used in this + environment. This allows the environment to build the same, in + theory, even on different verions of Spack with different + packages! + +Adding User Specs +^^^^^^^^^^^^^^^^^ + +Once an environment is created, it is time to add some user specs. A +*user spec* is simply a spec typed by the user; for example, what one +would type as part of ``spack install``. An environment can consist +of as many user specs as one likes, and the specs can contain +conflicting versions of packages. For example: + +.. code-block:: console + + $ spack env myenv add mpileaks + $ spack env myenv add python + +Concretizing +^^^^^^^^^^^^ + +Once some user specs have been added to an environment, they can be +concretized. The following command will concretize all user specs +that have been added and not yet concretized: + +.. code-block:: console + + $ spack env myenv concretize + +This command will re-concretize all specs: + +.. code-block:: console + + $ spack env myenv concretize + +Re-concretizing can be useful if packages have changed and one wishes +them to take effect in an environment; however it can lead to +additional re-building of packages. + +Listing +^^^^^^^ + +The ``spack env concretize`` command reports on its results. +This report can be repeated at any time with: + +.. code-block:: console + + $ spack env myenv list -Ilr + +This version of the command reports on hashes and install status of +all the specs in an environment. See ``spack env myenv list -h`` for +customization options. + +Installing +^^^^^^^^^^ + +Once an environment has been concretized, it can be installed. One +can think of this as running ``spack install`` on every concretized +spec in the environment: + +.. code-block:: console + + $ spack env myenv install + +As it installs, ``spack env install`` creates symbolic links in +the ``logs/`` directory in the environment, allowing for easy +inspection of build logs related to that environment. + +Loading +^^^^^^^ + +Once an environment has been installed, the following creates a load script for it: + +.. code-block:: console + + $ spack env myenv loads -r + +This creates a file called ``loads`` in the environment directory. +Sourcing that file in Bash will make the environment available to the +user; and can be included in ``.bashrc`` files, etc. The ``loads`` +file may also be copied out of the environment, renamed, etc. + +Environment Configs +------------------- + +A variety of Spack behaviors are changed through Spack configs. Many +Spack users use a ``~/.spack/packages.yaml`` file to customize or +otherwise direct the behavior of the concretizer; for example, to +choose a specific version of a package, or to use an external package. + +Spack environments can use a config as well; the user just has to +create a ``config/`` sub-directory in the environment, and then add +configuration files to it (eg: ``packages.yaml``). For example, the +following ``config/packages.yaml`` file will direct the concretizer to +use ``python@3.5.2`` in an environment: + +.. code-block:: yaml + + packages: + python: + version: [3.5.2] + +An environment's config will be loaded last; and will have precedence +over all other configs loaded by Spack; for example, the config files +in ``~/.spack``. + +env.yaml +-------- + +Further customization is also possible through a user-created +``env.yaml`` file in the environment's directory. This currently +allows for two further features: + +#. Loading (stacking) more than one config; or loading them from + locations outside the environment. +#. Recording a list of user specs to be part of the environment. + +An example ``env.yaml`` file: + +.. code-block:: yaml + + env: + configs: + - '{HOME}/spscopes/centos7' # Lowest precedence + - '{HOME}/spscopes/gissversions' + - '{HOME}/spscopes/twoway' + - '{HOME}/spscopes/develop' + - config # Highest precedence + specs: + ncview: # Highest precedence + netcdf: + nco: # Lowest precedence + py-sphinx: + +.. note:: + + #. If ``env.yaml`` exists, then Spack will no longer automatically + load from the default environment ``config/`` directory. This is a + bug. To enable the default behavior, use the following in + ``env.yaml``: + + .. code-block:: yaml + + env: + configs: + - config + + #. The effect of configs on concretization can be tested using the + ``spack env spec`` command. + + #. As shown above, the syntax ``{X}`` in config paths will + substitute the system environent variable named ``X``. This + allows configs to be referenced either relative to the + environment directory, or somewhere else on the filesystem. + + +Loading Specs from env.yaml +^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +If a list of user specs is included in ``env.yaml``, it can be added +to the environment as follows: + +.. code-block:: console + + $ spack env myenv add --all + +This is equivalent to a series of ``spack env myenv add`` calls. +Similarly, all user specs may be removed from the environment with: + +.. code-block:: console + + $ spack env myenv remove --all + +In this way, the list of user specs associated with an environment can +be kept in the environment; rather than in a separate script +elsewhere. + + +Initializing an Environment from a Template +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +When creating an environment with ``spack env create`` a user can +provide a template file with the ``--init-file`` option which will +add a set of pre-specified specs and initialize the environment's +configuration. The file has the following format: + +.. code-block:: yaml + + user_specs: + - python + - mpileaks + packages: + ... + compilers: + ... + +Each of the configuration sections (after ``user_specs``) will be +extracted into a corresponding configuration file in the environment; +Spack will not make further edits to the environment configuration. diff --git a/lib/spack/docs/index.rst b/lib/spack/docs/index.rst index 16dc69de4d1e1d..68ed9823e1cd33 100644 --- a/lib/spack/docs/index.rst +++ b/lib/spack/docs/index.rst @@ -64,6 +64,7 @@ or refer to the full manual below. module_file_support repositories binary_caches + environments command_index package_list