Garbage collection for Spack Environments

.gitignore

@@ -1,6 +1,7 @@
 /db
 /var/spack/stage
 /var/spack/cache
+/var/spack/environments
 /var/spack/repos/*/index.yaml
 /var/spack/repos/*/lock
 *.pyc

lib/spack/docs/environments.rst

@@ -0,0 +1,317 @@
+.. _environments:
+
+Environments
+============
+
+An environment is used to group together a set of specs for the
+purpose of building, rebuilding and deploying in a coherent fashion.
+Environments provide a number of advantages over the the a la carte
+approach of building and loading individual Spack modules:
+
+#. Environments eliminate the non-determinism encountered when
+   installing packages and then loading what was just created.
+   Consider the following sequence:
+
+   .. code-block:: console
+
+      $ spack install mypackage
+      $ spack load mypackage
+
+   Sometime later, somebody else installs an earlier version of ``mypackage``:
+
+   .. code-block:: console
+
+      $ spack install [email protected]
+
+   Now the ``spack load mypackage`` command will fail because more
+   than one version of ``mypackage`` is installed.  Thus, ``spack
+   load`` is non-deterministic, and not a reliable way to load
+   specific sets of modules one wishes to build and load.
+   Environments solve this problem by creating a localized context
+   that is immune to what other versions of packages might be
+   installed in the global Spack prefix.
+#. Environments separate the steps of (a) choosing what to
+   install, (b) concretizing, and (c) installing.  This allows
+   Environments to remain stable and repeatable, even if Spack packages
+   are upgraded: specs are only re-concretized when the user
+   explicitly asks for it.  It should even be possible to reliably
+   transport environments between different computers running
+   different versions of Spack!
+#. Since re-concretization only occurs at controlled times, the
+   Environment equivalent of many Spack commands are far faster.  For
+   example, ``spack env <env> list -Ilr`` vs. ``spack spec``.
+#. Environments allow several specs to be built at once; a more robust
+   solution than ad-hoc scripts making multiple calls to ``spack
+   install``.
+#. An Environment that is built as a whole can be loaded as a whole.
+   Spack can generate a script of ``module load`` commands that load
+   the appropraite environment.  And that script will work, without
+   running Spack, as long as the underlying modules remain installed.
+   A fast-running ``spack env <env> load`` command would also be
+   possible.
+
+Other packaging systems also provide environments that are similar in
+some ways to Spack environments; for example, `Conda environments
+<https://conda.io/docs/user-guide/tasks/manage-environments.html>`_ or
+`Python Virtual Environments
+<https://docs.python.org/3/tutorial/venv.html>`_.  Spack environments
+some distinctive features:
+
+#. A spec installed "in" an environment is no different form the same
+   spec installed anywhere else in Spack.  Environments are assembled
+   simply by collecting together a set of specs.  Environments are
+   loaded by generating a set of ``module load`` commands.  (In the
+   future, environments can also be assembled into a single tree using
+   techniques from ``spack view``.)
+#. Spack Environments may contain more than one version of the same
+   package; but only a single module for a package may be loaded.
+   Modules that occur in earlier specs listed in an environment take
+   precedence over modules that occur later.
+
+Using Environments
+------------------
+
+Here we follow a typical use case of creating, concretizing,
+installing and loading an environment.
+
+.. note::
+
+   Environments are currently in Beta.  We expect the user interface
+   to change.  Future changes might also break existing environment
+   files, particularly ``environment.json``.
+
+
+Creating an Environment
+^^^^^^^^^^^^^^^^^^^^^^^
+
+An environment is created by:
+
+.. code-block:: console
+
+   $ spack env myenv create
+
+Spack then creates the following files:
+
+* The directory ``var/spack/environments/myenv``.  This is the public
+  part of the environment.
+
+  .. note::
+
+     All environments are stored in the ``var/spack/environments`` folder.
+
+  The following files may be added to this directory by the user or
+  Spack:
+
+  * ``env.yaml``: Addition environment specification and configuration
+  * ``loads``: A script of ``module load`` commands, generated by
+    ``spack env myenv loads``, which the user may source to use the
+    built environment.
+  * ``logs/``: Symbolic links to build logs for specs that are part of
+    the environment.
+  * Anything else the user wishes to put in this directory!
+
+* The directory ``var/spack/environments/myenv/.env``.  This directory
+  is "owned" by Spack, and may be rewritten from time to time.  The
+  user should not put anything in it.  It contains:
+
+  * ``environment.json``: Concretized specs for the environment.
+  * ``repo/``: A repo consisting of the Spack packages used in this
+    environment.  This allows the environment to build the same, in
+    theory, even on different verions of Spack with different
+    packages!
+
+Adding User Specs
+^^^^^^^^^^^^^^^^^
+
+Once an environment is created, it is time to add some user specs.  A
+*user spec* is simply a spec typed by the user; for example, what one
+would type as part of ``spack install``.  An environment can consist
+of as many user specs as one likes, and the specs can contain
+conflicting versions of packages.  For example:
+
+.. code-block:: console
+
+   $ spack env myenv add mpileaks
+   $ spack env myenv add python
+
+Concretizing
+^^^^^^^^^^^^
+
+Once some user specs have been added to an environment, they can be
+concretized.  The following command will concretize all user specs
+that have been added and not yet concretized:
+
+.. code-block:: console
+
+   $ spack env myenv concretize
+
+This command will re-concretize all specs:
+
+.. code-block:: console
+
+   $ spack env myenv concretize
+
+Re-concretizing can be useful if packages have changed and one wishes
+them to take effect in an environment; however it can lead to
+additional re-building of packages.
+
+Listing
+^^^^^^^
+
+The ``spack env <env> concretize`` command reports on its results.
+This report can be repeated at any time with:
+
+.. code-block:: console
+
+   $ spack env myenv list -Ilr
+
+This version of the command reports on hashes and install status of
+all the specs in an environment.  See ``spack env myenv list -h`` for
+customization options.
+
+Installing
+^^^^^^^^^^
+
+Once an environment has been concretized, it can be installed.  One
+can think of this as running ``spack install`` on every concretized
+spec in the environment:
+
+.. code-block:: console
+
+   $ spack env myenv install
+
+As it installs, ``spack env <env> install`` creates symbolic links in
+the ``logs/`` directory in the environment, allowing for easy
+inspection of build logs related to that environment.
+
+Loading
+^^^^^^^
+
+Once an environment has been installed, the following creates a load script for it:
+
+.. code-block:: console
+
+   $ spack env myenv loads -r
+
+This creates a file called ``loads`` in the environment directory.
+Sourcing that file in Bash will make the environment available to the
+user; and can be included in ``.bashrc`` files, etc.  The ``loads``
+file may also be copied out of the environment, renamed, etc.
+
+Environment Configs
+-------------------
+
+A variety of Spack behaviors are changed through Spack configs.  Many
+Spack users use a ``~/.spack/packages.yaml`` file to customize or
+otherwise direct the behavior of the concretizer; for example, to
+choose a specific version of a package, or to use an external package.
+
+Spack environments can use a config as well; the user just has to
+create a ``config/`` sub-directory in the environment, and then add
+configuration files to it (eg: ``packages.yaml``).  For example, the
+following ``config/packages.yaml`` file will direct the concretizer to
+use ``[email protected]`` in an environment:
+
+.. code-block:: yaml
+
+   packages:
+       python:
+           version: [3.5.2]
+
+An environment's config will be loaded last; and will have precedence
+over all other configs loaded by Spack; for example, the config files
+in ``~/.spack``.
+
+env.yaml
+--------
+
+Further customization is also possible through a user-created
+``env.yaml`` file in the environment's directory.  This currently
+allows for two further features:
+
+#. Loading (stacking) more than one config; or loading them from
+   locations outside the environment.
+#. Recording a list of user specs to be part of the environment.
+
+An example ``env.yaml`` file:
+
+.. code-block:: yaml
+
+   env:
+       configs:
+           - '{HOME}/spscopes/centos7'            # Lowest precedence
+           - '{HOME}/spscopes/gissversions'
+           - '{HOME}/spscopes/twoway'
+           - '{HOME}/spscopes/develop'
+           - config                               # Highest precedence
+       specs:
+           ncview:                                # Highest precedence
+           netcdf:
+           nco:                                   # Lowest precedence
+           py-sphinx:
+
+.. note::
+
+   #. If ``env.yaml`` exists, then Spack will no longer automatically
+      load from the default environment ``config/`` directory.  This is a
+      bug.  To enable the default behavior, use the following in
+      ``env.yaml``:
+
+      .. code-block:: yaml
+
+         env:
+             configs:
+                 - config
+
+   #. The effect of configs on concretization can be tested using the
+      ``spack env <env> spec`` command.
+
+   #. As shown above, the syntax ``{X}`` in config paths will
+      substitute the system environent variable named ``X``.  This
+      allows configs to be referenced either relative to the
+      environment directory, or somewhere else on the filesystem.
+
+
+Loading Specs from env.yaml
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+If a list of user specs is included in ``env.yaml``, it can be added
+to the environment as follows:
+
+.. code-block:: console
+
+   $ spack env myenv add --all
+
+This is equivalent to a series of ``spack env myenv add`` calls.
+Similarly, all user specs may be removed from the environment with:
+
+.. code-block:: console
+
+   $ spack env myenv remove --all
+
+In this way, the list of user specs associated with an environment can
+be kept in the environment; rather than in a separate script
+elsewhere.
+
+
+Initializing an Environment from a Template
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+When creating an environment with ``spack env <env> create`` a user can
+provide a template file with the ``--init-file`` option which will
+add a set of pre-specified specs and initialize the environment's
+configuration. The file has the following format:
+
+.. code-block:: yaml
+
+  user_specs:
+    - python
+    - mpileaks
+  packages:
+    ...
+  compilers:
+    ...
+
+Each of the configuration sections (after ``user_specs``) will be
+extracted into a corresponding configuration file in the environment;
+Spack will not make further edits to the environment configuration.

lib/spack/docs/index.rst

@@ -64,6 +64,7 @@ or refer to the full manual below.
    module_file_support
    repositories
    binary_caches
+   environments
    command_index
    package_list