docs/v2-3/user-guide.html

<!DOCTYPE html>
<head>
    <meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1">

<meta name="description" content="   Singularity User Guide                                                 This document will cover the usage of Singularity, working with containers, and all of the user facing features. There is a separate Singularity Administration Guide which targets system administrators, so if you are a service provider, or an interested user, it is encouraged that you read that document as well.Welcome to Singularity!Singularity is a container solution created by necessity for scientific and application driven workloads.Over the past decade and a half, virtualization has gone from an engineering toy to a global infrastructure necessity and the evolution of enabling technologies has flourished. Most recently, we have seen the introduction of the latest spin on virtualization…  “containers”. People’s general conception of containers carry the heredity of its lineage and thus has influenced its features and use cases. This is both a good and a bad thing…For the industry at the forefront of the virtualization front this is a good thing. The enterprise and web enabled cloud requirements are very much in alignment with the feature set of virtual machines, and thus the preceding container technologies, but this does not bode as well for the scientific world and specifically the high performance computation (HPC) use case. While there are many overlapping features of these two fields, they differ in ways that make a shared implementation generally incompatible. While some have been able to leverage custom built resources that can operate on a lower performance scale, a proper integration is difficult and perhaps impossible with today’s technology.Scientists are a resourceful bunch and many of the features which exist both purposefully and incidentally via commonly used container technologies are not only desired, they are required for scientific use cases. This is the necessity which drove the creation of Singularity and articulated its four primary functions:Mobility Of ComputeMobility of compute is defined as the ability to define, create and maintain a workflow and be confident that the workflow can be executed on different hosts, operating systems (as long as it is Linux) and service providers. Being able to contain the entire software stack, from data files to library stack, and portably move it from system to system is true mobility.Singularity achieves this by utilizing a distributable image format that contains the entire container and stack into a single file. This file can be copied, shared, archived, and thus standard UNIX file permissions also apply. Additionally containers are portable (even across different C library versions and implementations) which makes sharing and copying an image as easy as cp or scp or ftp.ReproducibilityAs mentioned above, Singularity containers utilize a single file which is the complete representation of all the files within the container. The same features which facilitate mobility also facilitate reproducibility. Once a contained workflow has been defined, the container image can be snapshotted, archived, and locked down such that it can be used later and you can be confident that the code within the container has not changed. The container is not subject to any external influence from the host operating system (aside from the kernel).User FreedomSystem integrators, administrators, and engineers spend a lot of effort maintaining the operating systems on the resources they are responsible for, and as a result tend to take a cautious approach on their systems. As a result, you may see hosts installed with a production, mission critical operating system that is “old” and may not have a lot of packages available for it. Or you may see software or libraries that are too old or incompatible with the software you need to run, or maybe just haven’t installed the software stack you need due to complexities with building, specific software knowledge, incompatibilities or conflicts with other installed programs.Singularity can give the user the freedom they need to install the applications, versions, and dependencies for their workflows without impacting the system in any way. Users can define their own working environment and literally copy that environment image (single file) to a shared resource, and run their workflow inside that image.Support On Existing Traditional HPCThere are a lot of container systems presently available which are designed either for the enterprise, a replacement for virtual machines, cloud focused, or requires kernel features which are either not stable yet, or not available on your distribution of choice (or both).Replicating a virtual machine cloud like environment within an existing HPC resource is not a reasonable task, but this is the direction one would need to take to integrate OpenStack or Docker into traditional HPC. The use cases do not overlap nicely, nor can the solutions be forcibly wed.The goal of Singularity is to support existing and traditional HPC resources as easily as installing a single package onto the host operating system. Some configuration maybe required via a single configuration file, but the defaults are tuned to be generally applicable for shared environments.Singularity can run on host Linux distributions from RHEL6 (RHEL5 for versions lower then 2.2) and similar vintages, and the contained images have been tested as far back as Linux 2.2 (approximately 14 years old). Singularity natively supports InfiniBand, Lustre, and works seamlessly with all resource managers (e.g. SLURM, Torque, SGE, etc.) because it works like running any other command on the system.A High Level View of SingularitySecurity and privilege escalationA user inside a Singularity container is the same user as outside the containerThis is one of Singularities defining characteristics. It allows a user (that may already have shell access to a particular host) to simply run a command inside of a container image as themselves. Here is a scenario to help articulate this:  %SERVER is a shared multi-tenant resource to a number of users and as a result it is a large expensive resource far exceeding the resources of my personal workstation. But because it is a shared system, no users have root access and it is a controlled environment managed by a staff of system administrators. To keep the system secure, only the system administrators are granted root access and they control the state of the operating system. If a user is able to escalate to root (even within a container) on %SERVER, they can do bad things to the network, cause denial of service to the host (as well as other hosts on the same network), and may have unrestricted access to file systems reachable by the container.To mitigate security concerns like this, Singularity limits one’s ability to escalate permission inside a container. For example, if I do not have root access on the target system, I should not be able to escalate my privileges within the container to root either. This is semi-antagonistic to Singularity’s 3rd tenant; allowing the users to have freedom of their own environments. Because if a user has the freedom to create and manipulate their own container environment, surely they know how to escalate their privileges to root within that container. Possible means could be setting the root user’s password, or enabling themselves to have sudo access. For these reasons, Singularity prevents user context escalation within the container, and thus makes it possible to run user supplied containers on shared infrastructures.But this mitigation dictates the Singularity workflow. If a user needs to be root in order to make changes to their containers, then they need to have an endpoint (a local workstation, laptop, or server) where they have root access. Considering almost everybody at least has a laptop, this is not an unreasonable or unmanageable mitigation, but it must be defined and articulated.The Singularity container imageSingularity makes use of a container image file, which physically contains the container. This file is a physical representation of the container environment itself. If you obtain an interactive shell within a Singularity container, you are literally running within that file.This simplifies management of files to the element of least surprise, basic file permission. If you either own a container image, or have read access to that container image, you can start a shell inside that image. If you wish to disable or limit access to a shared image, you simply change the permission ACLs to that file.There are numerous benefits for using a single file image for the entire container and is summarized here:  Copying or branching an entire container is as simple as cp  Permission/access to the container is managed via standard file system permissions  Large scale performance (especially over parallel file systems) is very efficient  No caching of the image contents to run (especially nice on clusters)  Container is a sparse file so it only consumes the disk space actually used  Changes are implemented in real time (image grows and shrinks as needed)  Images can serve as stand-alone programs, and can be executed like any other program on the hostOther container formats supportedIn addition to the default Singularity container image, the following other formats are supported:  directory: standard Unix directories containing a root container image  tar.gz: zlib compressed tar archives  tar.bz2: bzip2 compressed tar archives  tar: uncompressed tar archives  cpio.gz: zlib compressed CPIO archives  cpio: uncompressed CPIO archivesnote: the suffix for the formats (except directory) are necessary as that is how Singularity identifies the image type.Supported URIsSingularity also supports several different mechanisms for obtaining the images using a standard URI format  http:// Singularity will use Curl to download the image locally, and then run from the local image  https:// same as above using encryption  docker:// Singularity can pull Docker images from a Docker registry, and will run them non-persistently (e.g. changes are not persisted as they can not be saved upstream)  shub:// Singularity Hub is our own registry for Singularity containers. If you want to publish a container, or give easy access to others from their command line, or enable automatic builds, you should build it on Singularity Hub.Name-spaces and isolationWhen asked, “What namespaces does Singularity virtualize?”, the most appropriate response from a Singularity use case is “As few as possible!”. This is because the goals of Singularity are mobility, reproducibility and freedom, not full isolation (as you would expect from industry driven container technologies). Singularity only separates the needed namespaces in order to satisfy our primary goals.So considering that, and considering that the user inside the container is the same user outside the container, allows us to blur the lines between what is contained and what is on the host. When this is done properly, using Singularity feels more like running in a parallel universe, where there are two timelines. One timeline, is the one we are familiar with, where the system administrators installed their operating system of choice. But on this alternate timeline, we bribed the system administrators and they installed our favorite operating system, and gave us full control but configured the rest of the system identically. And Singularity gives us the power to pick between these two timelines.Or in summary, Singularity allows us to virtually swap out the underlying operating system for one that we defined without affecting anything else on the system and still having all of the host resources available to us.It can also be described as ssh’ing into another identical host running a different operating system. One moment you are on Centos-6 and the next minute you are on the latest version of Ubuntu that has Tensorflow installed, or Debian with the latest OpenFoam, or a custom workflow that you installed.Additionally what name-spaces are selected for virtualization can be dynamic or conditional. For example, the PID namespace is not separated from the host by default, but if you want to separate it, you can with a command line (or environment variable) setting. You can also decide you want to contain a process so it can not reach out to the host file system if you don’t know if you trust the image. But by default, you are allowed to interface with all of the resources, devices and network inside the container as you are outside the container.Compatibility with standard work-flows, pipes and IOSingularity does its best to abstract the complications of running an application in a different environment then what is expected on the host. For example, applications or scripts within a Singularity container can easily be part of a pipeline that is being executed on the host. Singularity containers can also be executed from a batch script or other program (e.g. an HPC system’s resource manager) natively.Some usage examples of Singularity can be seen as follows:$ singularity exec dummpy.img xterm$ singularity exec dummy.img python script.py$ singularity exec dummy.img python &lt; /path/to/python/script.py$ cat /path/to/python/script.py | singularity exec dummy.img pythonYou can even run MPI executables within the container as simply as:$ mpirun -np X singularity exec /path/to/container.img /usr/bin/mpi_program_inside_container (mpi program args)The Singularity Process FlowWhen executing container commands, the Singularity process flow can be generalized as follows:  Singularity application is invoked  Global options are parsed and activated  The Singularity command (subcommand) process is activated  Subcommand options are parsed  The appropriate sanity checks are made  Environment variables are set  The Singularity Execution binary is called (sexec)  Sexec determines if it is running privileged and calls the SUID code if necessary  Namespaces are created depending on configuration and process requirements  The Singularity image is checked, parsed, and mounted in the CLONE_NEWNS namespace  Bind mount points are setup  The namespace CLONE_FS is used to virtualize a new root file system  Singularity calls execvp() and Singularity process itself is replaced by the process inside the container  When the process inside the container exits, all namespaces collapse with that process, leaving a clean systemAll of the above steps take approximately 15-25 thousandths of a second to run, which is fast enough to seem instantaneous.The Singularity Usage WorkflowThe security model of Singularity (as described above, “A user inside a Singularity container is the same user as outside the container”) defines the Singularity workflow. There are generally two classifications of actions you would implement on a container; modification (which encompasses creation, bootstrapping, installing, admin) and using the container.Modification of containers (new or existing) generally require root administrative privileges just like these actions would require on any system, container, or virtual machine. This means that a user must have a system that they have root access on. This could be a server, workstation, or even a laptop. If you are using OS X or Windows on your laptop, it is recommended to setup Vagrant, and run Singularity from there (there are recipes for this which can be found at http://singularity.lbl.gov/). Once you have Singularity installed on your endpoint of choice, this is where you will do the bulk of your container development.This workflow can be described visually as follows:One the left side, you have your laptop, workstation, or a server that you control. Here you will create your containers, modify and update your containers as you need. Once you have the container with the necessary applications, libraries and data inside it can be easily shared to other hosts and executed without have root access. But if you need to make changes again to your container, you must go back to an endpoint or system that you have root on, make the necessary changes, and then re-upload the container to the computation resource you wish to execute it on.ExamplesHow do the commands work? We recommend you look at examples for each:  shell  exec  run  bootstrapSupportHave a question, or need further information? Reach out to us.                Previous        Next                 Edit me                                                                                                                                                 Site last generated: Jul 26, 2017                             ">


<meta name="name" content="Singularity User Guide">

<meta name="thumbnail" content="http://singularity.lbl.gov/images/logo/logo.svg">


<title>Singularity User Guide | Singularity</title>
<link rel="stylesheet" href="assets/css/syntax.css">


<link rel="stylesheet" type="text/css" href="https://maxcdn.bootstrapcdn.com/font-awesome/4.5.0/css/font-awesome.min.css">
<link href="https://fonts.googleapis.com/css?family=Open+Sans" rel="stylesheet">
<!--<link rel="stylesheet" type="text/css" href="css/bootstrap.min.css">-->
<link rel="stylesheet" href="assets/css/modern-business.css">
<link rel="stylesheet" href="assets/css/lavish-bootstrap.css">
<link rel="stylesheet" href="assets/css/customstyles.css">
<link rel="stylesheet" href="assets/css/theme-blue.css">

<link rel="stylesheet" type="text/css" href="assets/css/asciinema-player.css" />

<script src="https://cdnjs.cloudflare.com/ajax/libs/jquery/2.1.4/jquery.min.js"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/jquery-cookie/1.4.1/jquery.cookie.min.js"></script>
<script src="assets/js/jquery.navgoco.min.js"></script>


<script src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.4/js/bootstrap.min.js"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/anchor-js/2.0.0/anchor.min.js"></script>
<script src="assets/js/toc.js"></script>
<script src="assets/js/customscripts.js"></script>

<link rel="shortcut icon" href="images/favicon/favicon.ico">

<!-- HTML5 Shim and Respond.js IE8 support of HTML5 elements and media queries -->
<!-- WARNING: Respond.js doesn't work if you view the page via file:// -->
<!--[if lt IE 9]>
<script src="https://oss.maxcdn.com/libs/html5shiv/3.7.0/html5shiv.js"></script>
<script src="https://oss.maxcdn.com/libs/respond.js/1.4.2/respond.min.js"></script>
<![endif]-->

<link rel="alternate" type="application/rss+xml" title="" href="http://localhost:4005feed.xml">

    <script>
        $(document).ready(function() {

            // Initialize navgoco with default options
            $("#mysidebar").navgoco({
                caretHtml: '',
                accordion: true,
                openClass: 'active', // open
                save: false, // leave false or nav highlighting doesn't work right
                cookie: {
                    name: 'navgoco',
                    expires: false,
                    path: '/'
                },
                slide: {
                    duration: 400,
                    easing: 'swing'
                }
            });

            $("#collapseAll").click(function(e) {
                e.preventDefault();
                $("#mysidebar").navgoco('toggle', false);
            });

            $("#expandAll").click(function(e) {
                e.preventDefault();
                $("#mysidebar").navgoco('toggle', true);
            });

        });

    </script>
    <script>
        $(function () {
            $('[data-toggle="tooltip"]').tooltip()
        })
    </script>
    

</head>
<body>
<!-- asciinema player -->
<script src="assets/js/asciinema-player.js"></script>

<!-- Show or hide players on button clicks-->
<script>
$( document ).ready(function() {
    $(".asciinema-button").click(function(){
        var asciinemaVideo = "#asciinema-" + $(this).attr('id');

        if ($(asciinemaVideo).hasClass('hidden')){
            $(asciinemaVideo).removeClass('hidden');
            $(this).text('Hide Tutorial')
        } else {
            $(asciinemaVideo).addClass('hidden');
            $(this).text('Show Tutorial')
        }
        
    });
});
</script>

<!-- Navigation -->
<nav class="navbar navbar-inverse navbar-fixed-top">
    <div class="container topnavlinks">
        <div class="navbar-header">
            <button type="button" class="navbar-toggle" data-toggle="collapse" data-target="#bs-example-navbar-collapse-1">
                <span class="sr-only">Toggle navigation</span>
                <span class="icon-bar"></span>
                <span class="icon-bar"></span>
                <span class="icon-bar"></span>
            </button>
            <a class="navbar-brand" href="index.html">&nbsp;<span class="projectTitle"> Singularity</span></a>
        </div>
        <div class="collapse navbar-collapse" id="bs-example-navbar-collapse-1">
            <ul class="nav navbar-nav navbar-right">
                <!-- entries without drop-downs appear here -->
                
                
                <li><a href="blog">News</a></li>
                
                
                <!-- entries with drop-downs appear here -->
                <!-- conditional logic to control which topnav appears for the audience defined in the configuration file.-->
                
                
                <li class="dropdown">
                    <a href="#" class="dropdown-toggle" data-toggle="dropdown">Docs<b class="caret"></b></a>
                    <ul class="dropdown-menu">
                        
                        
                        <li><a href="admin-guide">Admin Guide</a></li>
                        
                        
                        <li class="dropdownActive"><a href="user-guide">User Guide</a></li>
                        
                        
                        <li><a href="links">Contributed Content</a></li>
                        
                        
                    </ul>
                </li>
                
                <li class="dropdown">
                    <a href="#" class="dropdown-toggle" data-toggle="dropdown">Quick Links<b class="caret"></b></a>
                    <ul class="dropdown-menu">
                        
                        
                        <li><a href="https://github.com/singularityware/singularity" target="_blank">Github Repo</a></li>
                        
                        
                        <li><a href="https://groups.google.com/a/lbl.gov/forum/#!forum/singularity" target="_blank">Google Group</a></li>
                        
                        
                        <li><a href="http://stackoverflow.com/questions/tagged/singularity" target="_blank">Singularity on Stack Overflow</a></li>
                        
                        
                        <li><a href="https://singularity-hub.org/faq" target="_blank">Singularity Hub</a></li>
                        
                        
                        <li><a href="https://singularity-container.slack.com" target="_blank">Slack</a></li>
                        
                        
                        <li><a href="faq#troubleshooting">Troubleshooting</a></li>
                        
                        
                    </ul>
                </li>
                
                <li class="dropdown">
                    <a href="#" class="dropdown-toggle" data-toggle="dropdown">People<b class="caret"></b></a>
                    <ul class="dropdown-menu">
                        
                        
                        <li><a href="https://github.com/gmkurtzer" target="_blank">Gregory M. Kurtzer</a></li>
                        
                        
                        <li><a href="https://github.com/vsoch" target="_blank">Vanessa Sochat</a></li>
                        
                        
                        <li><a href="https://github.com/bauerm97" target="_blank">Michael Bauer</a></li>
                        
                        
                        <li><a href="https://github.com/bbockelm" target="_blank">Brian Bockelman</a></li>
                        
                        
                        <li><a href="https://github.com/singularityware/singularity/blob/master/AUTHORS" target="_blank">Complete Authors List</a></li>
                        
                        
                    </ul>
                </li>
                
                
                <li><a href="/search"><i class="fa fa-search"></i></li>
                <!-- jekyll search hidden in favor of google
                <li>
                    <div id="search-demo-container">
                        <input type="text" id="search-input" placeholder="search...">
                        <ul id="results-container"></ul>
                    </div>
                    <script src="assets/js/jekyll-search.js" type="text/javascript"></script>
                    <script type="text/javascript">
                            SimpleJekyllSearch.init({
                                searchInput: document.getElementById('search-input'),
                                resultsContainer: document.getElementById('results-container'),
                                dataSource: 'search.json',
                                searchResultTemplate: '<li><a href="{url}" title="Singularity User Guide">{title}</a></li>',
                    noResultsText: 'No results found.',
                            limit: 10,
                            fuzzy: true,
                    })
                    </script>
                    end search-->
                </li>
            </ul>
        </div>
        </div>
        <!-- /.container -->
</nav>

<!-- Page Content -->
<div class="container">
    <div class="col-lg-12">&nbsp;</div>
    <!-- Content Row -->
    <div class="row">
        <!-- Sidebar Column -->
        <div class="col-md-3">
          

<div class="shiny"><a href="\"><figure><img src="/images/logo/logo.svg" class="sidebar-logo"/></figure></a></div>


<ul id="mysidebar" class="nav">
    <li class="sidebarTitle">User Guide</li>
    
    
    <li>
        <a href="#">Getting Started</a>
        <ul>
            
            
            <li class="active"><a href="user-guide">Introduction</a></li>
            
            
            <li><a href="docs-quick-start-installation">Quick Start Installation</a></li>
            
            
            <li><a href="create-image">Create an Image</a></li>
            
            
            <li><a href="bootstrap-image">Bootstrap an Image</a></li>
            
            
            <li><a href="docs-content">Adding Content</a></li>
            
            
            <li><a href="docs-mount">Bind Paths and Files</a></li>
            
            
            <li><a href="docs-environment-metadata">Environment and Metadata</a></li>
            
            
            <li><a href="docs-changing-containers">Change an Existing Container</a></li>
            
            
            <li><a href="docs-docker">Singularity and Docker</a></li>
            
            
            <li><a href="faq#troubleshooting">Troubleshooting</a></li>
            
            
        </ul>
        
        
    <li>
        <a href="#">Commands</a>
        <ul>
            
            
            <li><a href="docs-usage">Command Usage</a></li>
            
            
            <li><a href="docs-bootstrap">bootstrap</a></li>
            
            
            <li><a href="docs-exec">exec</a></li>
            
            
            <li><a href="docs-export">export</a></li>
            
            
            <li><a href="docs-import">import</a></li>
            
            
            <li><a href="docs-inspect">inspect</a></li>
            
            
            <li><a href="docs-pull">pull</a></li>
            
            
            <li><a href="docs-run">run</a></li>
            
            
            <li><a href="docs-shell">shell</a></li>
            
            
        </ul>
        
        
        <!-- if you aren't using the accordion, uncomment this block:
           <p class="external">
               <a href="#" id="collapseAll">Collapse All</a> | <a href="#" id="expandAll">Expand All</a>
           </p>
           -->
    </li>
</ul>
</div>

<!-- this highlights the active parent class in the navgoco sidebar. this is critical so that the parent expands when you're viewing a page. This must appear below the sidebar code above. Otherwise, if placed inside customscripts.js, the script runs before the sidebar code runs and the class never gets inserted.-->
<script>$("li.active").parents('li').toggleClass("active");</script>

    <!-- Content Column -->
    <div class="col-md-9">
        <div class="post-header">
   <h1 class="post-title-main">Singularity User Guide</h1>
</div>


<div class="post-content">

   
    <!-- Previous and next buttons-->
    <div class="row" style="padding-top:30px; margin-bottom:10px"><div class="col-md-12">
        <a href="#"><button style="float:left" class="hidden previous-button btn btn-circle btn-default"><i class="fa-2x fa fa-angle-double-left"></i></button></a>
        <a href="#"><button style="float:right" class="hidden next-button btn btn-circle btn-default"><i class="fa fa-angle-double-right fa-2x"></i></button></a>
    </div></div>

    <script>
    $(document).ready(function(){

        var next = $("li.active").next().last().find('a').attr('href');
        var previous = $("li.active").prev().last().find('a').attr('href');

        // NEXT BUTTON
        if (typeof next == 'undefined'){
            console.log("disabling next button")
            $(".next-button").addClass("hidden")
        } else if (next == "#") {
            next = $("li.active").next().find("li").first().find('a').attr('href');
            $(".next-button").closest('a').attr('href', next)
            $(".next-button").removeClass('hidden')
        } else {
            $(".next-button").closest('a').attr('href', next)
            $(".next-button").removeClass('hidden')
        }

        // PREVIOUS BUTTON
        if (typeof previous == 'undefined'){
            console.log("disabling previous button")
            $(".previous-button").addClass("hidden")
        } else if (previous == "#") {
            previous = $("li.active").prev().find("li").last().find('a').attr('href')
            $(".previous-button").closest('a').attr('href', previous)
            $(".previous-button").removeClass('hidden')
        } else {
            $(".previous-button").closest('a').attr('href', previous)
            $(".previous-button").removeClass('hidden')
        }

    })
    </script>

    
<!-- this handles the automatic toc. use ## for subheads to auto-generate the on-page minitoc. if you use html tags, you must supply an ID for the heading element in order for it to appear in the minitoc. -->
<script>
$( document ).ready(function() {
  // Handler for .ready() called.

$('#toc').toc({ minimumHeaders: 0, listType: 'ul', showSpeed: 0, headers: 'h2,h3,h4' });

/* this offset helps account for the space taken up by the floating toolbar. */
$('#toc').on('click', 'a', function() {
  var target = $(this.getAttribute('href'))
    , scroll_target = target.offset().top

  $(window).scrollTop(scroll_target - 10);
  return false
})
  
});
</script>

<div id="toc"></div>

    
  <p>This document will cover the usage of Singularity, working with containers, and all of the user facing features. There is a separate <a href="/admin-guide">Singularity Administration Guide</a> which targets system administrators, so if you are a service provider, or an interested user, it is encouraged that you read that document as well.</p>

<h2 id="welcome-to-singularity">Welcome to Singularity!</h2>
<p>Singularity is a container solution created by necessity for scientific and application driven workloads.</p>

<p>Over the past decade and a half, virtualization has gone from an engineering toy to a global infrastructure necessity and the evolution of enabling technologies has flourished. Most recently, we have seen the introduction of the latest spin on virtualization…  “containers”. People’s general conception of containers carry the heredity of its lineage and thus has influenced its features and use cases. This is both a good and a bad thing…</p>

<p>For the industry at the forefront of the virtualization front this is a good thing. The enterprise and web enabled cloud requirements are very much in alignment with the feature set of virtual machines, and thus the preceding container technologies, but this does not bode as well for the scientific world and specifically the high performance computation (HPC) use case. While there are many overlapping features of these two fields, they differ in ways that make a shared implementation generally incompatible. While some have been able to leverage custom built resources that can operate on a lower performance scale, a proper integration is difficult and perhaps impossible with today’s technology.</p>

<p>Scientists are a resourceful bunch and many of the features which exist both purposefully and incidentally via commonly used container technologies are not only desired, they are required for scientific use cases. This is the necessity which drove the creation of Singularity and articulated its four primary functions:</p>

<h3 id="mobility-of-compute">Mobility Of Compute</h3>

<p>Mobility of compute is defined as the ability to define, create and maintain a workflow and be confident that the workflow can be executed on different hosts, operating systems (as long as it is Linux) and service providers. Being able to contain the entire software stack, from data files to library stack, and portably move it from system to system is true mobility.</p>

<p>Singularity achieves this by utilizing a distributable image format that contains the entire container and stack into a single file. This file can be copied, shared, archived, and thus standard UNIX file permissions also apply. Additionally containers are portable (even across different C library versions and implementations) which makes sharing and copying an image as easy as <code class="highlighter-rouge">cp</code> or <code class="highlighter-rouge">scp</code> or <code class="highlighter-rouge">ftp</code>.</p>

<h3 id="reproducibility">Reproducibility</h3>

<p>As mentioned above, Singularity containers utilize a single file which is the complete representation of all the files within the container. The same features which facilitate mobility also facilitate reproducibility. Once a contained workflow has been defined, the container image can be snapshotted, archived, and locked down such that it can be used later and you can be confident that the code within the container has not changed. The container is not subject to any external influence from the host operating system (aside from the kernel).</p>

<h3 id="user-freedom">User Freedom</h3>

<p>System integrators, administrators, and engineers spend a lot of effort maintaining the operating systems on the resources they are responsible for, and as a result tend to take a cautious approach on their systems. As a result, you may see hosts installed with a production, mission critical operating system that is “old” and may not have a lot of packages available for it. Or you may see software or libraries that are too old or incompatible with the software you need to run, or maybe just haven’t installed the software stack you need due to complexities with building, specific software knowledge, incompatibilities or conflicts with other installed programs.</p>

<p>Singularity can give the user the freedom they need to install the applications, versions, and dependencies for their workflows without impacting the system in any way. Users can define their own working environment and literally copy that environment image (single file) to a shared resource, and run their workflow inside that image.</p>

<h3 id="support-on-existing-traditional-hpc">Support On Existing Traditional HPC</h3>

<p>There are a lot of container systems presently available which are designed either for the enterprise, a replacement for virtual machines, cloud focused, or requires kernel features which are either not stable yet, or not available on your distribution of choice (or both).</p>

<p>Replicating a virtual machine cloud like environment within an existing HPC resource is not a reasonable task, but this is the direction one would need to take to integrate OpenStack or Docker into traditional HPC. The use cases do not overlap nicely, nor can the solutions be forcibly wed.</p>

<p>The goal of Singularity is to support existing and traditional HPC resources as easily as installing a single package onto the host operating system. Some configuration maybe required via a single configuration file, but the defaults are tuned to be generally applicable for shared environments.</p>

<p>Singularity can run on host Linux distributions from RHEL6 (RHEL5 for versions lower then 2.2) and similar vintages, and the contained images have been tested as far back as Linux 2.2 (approximately 14 years old). Singularity natively supports InfiniBand, Lustre, and works seamlessly with all resource managers (e.g. SLURM, Torque, SGE, etc.) because it works like running any other command on the system.</p>

<h2 id="a-high-level-view-of-singularity">A High Level View of Singularity</h2>

<h3 id="security-and-privilege-escalation">Security and privilege escalation</h3>

<p><em>A user inside a Singularity container is the same user as outside the container</em></p>

<p>This is one of Singularities defining characteristics. It allows a user (that may already have shell access to a particular host) to simply run a command inside of a container image as themselves. Here is a scenario to help articulate this:</p>

<blockquote>
  <p>%SERVER is a shared multi-tenant resource to a number of users and as a result it is a large expensive resource far exceeding the resources of my personal workstation. But because it is a shared system, no users have root access and it is a controlled environment managed by a staff of system administrators. To keep the system secure, only the system administrators are granted root access and they control the state of the operating system. If a user is able to escalate to root (even within a container) on %SERVER, they can do bad things to the network, cause denial of service to the host (as well as other hosts on the same network), and may have unrestricted access to file systems reachable by the container.</p>
</blockquote>

<p>To mitigate security concerns like this, Singularity limits one’s ability to escalate permission inside a container. For example, if I do not have root access on the target system, I should not be able to escalate my privileges within the container to root either. This is semi-antagonistic to Singularity’s 3rd tenant; allowing the users to have freedom of their own environments. Because if a user has the freedom to create and manipulate their own container environment, surely they know how to escalate their privileges to root within that container. Possible means could be setting the root user’s password, or enabling themselves to have sudo access. For these reasons, Singularity prevents user context escalation within the container, and thus makes it possible to run user supplied containers on shared infrastructures.</p>

<p>But this mitigation dictates the Singularity workflow. If a user needs to be root in order to make changes to their containers, then they need to have an endpoint (a local workstation, laptop, or server) where they have root access. Considering almost everybody at least has a laptop, this is not an unreasonable or unmanageable mitigation, but it must be defined and articulated.</p>

<h3 id="the-singularity-container-image">The Singularity container image</h3>
<p>Singularity makes use of a container image file, which physically contains the container. This file is a physical representation of the container environment itself. If you obtain an interactive shell within a Singularity container, you are literally running within that file.</p>

<p>This simplifies management of files to the element of least surprise, basic file permission. If you either own a container image, or have read access to that container image, you can start a shell inside that image. If you wish to disable or limit access to a shared image, you simply change the permission ACLs to that file.</p>

<p>There are numerous benefits for using a single file image for the entire container and is summarized here:</p>

<ul>
  <li>Copying or branching an entire container is as simple as <code class="highlighter-rouge">cp</code></li>
  <li>Permission/access to the container is managed via standard file system permissions</li>
  <li>Large scale performance (especially over parallel file systems) is very efficient</li>
  <li>No caching of the image contents to run (especially nice on clusters)</li>
  <li>Container is a sparse file so it only consumes the disk space actually used</li>
  <li>Changes are implemented in real time (image grows and shrinks as needed)</li>
  <li>Images can serve as stand-alone programs, and can be executed like any other program on the host</li>
</ul>

<h4 id="other-container-formats-supported">Other container formats supported</h4>
<p>In addition to the default Singularity container image, the following other formats are supported:</p>

<ul>
  <li><strong>directory</strong>: standard Unix directories containing a root container image</li>
  <li><strong>tar.gz</strong>: zlib compressed tar archives</li>
  <li><strong>tar.bz2</strong>: bzip2 compressed tar archives</li>
  <li><strong>tar</strong>: uncompressed tar archives</li>
  <li><strong>cpio.gz</strong>: zlib compressed CPIO archives</li>
  <li><strong>cpio</strong>: uncompressed CPIO archives</li>
</ul>

<p><em>note: the suffix for the formats (except directory) are necessary as that is how Singularity identifies the image type.</em></p>

<h4 id="supported-uris">Supported URIs</h4>
<p>Singularity also supports several different mechanisms for obtaining the images using a standard URI format</p>

<ul>
  <li><strong>http://</strong> Singularity will use Curl to download the image locally, and then run from the local image</li>
  <li><strong>https://</strong> same as above using encryption</li>
  <li><strong>docker://</strong> Singularity can pull Docker images from a Docker registry, and will run them non-persistently (e.g. changes are not persisted as they can not be saved upstream)</li>
  <li><strong>shub://</strong> Singularity Hub is our own registry for Singularity containers. If you want to publish a container, or give easy access to others from their command line, or enable automatic builds, you should build it on <a href="https://singularity-hub.org" target="_blank">Singularity Hub</a>.</li>
</ul>

<h3 id="name-spaces-and-isolation">Name-spaces and isolation</h3>
<p>When asked, “What namespaces does Singularity virtualize?”, the most appropriate response from a Singularity use case is “As few as possible!”. This is because the goals of Singularity are mobility, reproducibility and freedom, not full isolation (as you would expect from industry driven container technologies). Singularity only separates the needed namespaces in order to satisfy our primary goals.</p>

<p>So considering that, and considering that the user inside the container is the same user outside the container, allows us to blur the lines between what is contained and what is on the host. When this is done properly, using Singularity feels more like running in a parallel universe, where there are two timelines. One timeline, is the one we are familiar with, where the system administrators installed their operating system of choice. But on this alternate timeline, we bribed the system administrators and they installed our favorite operating system, and gave us full control but configured the rest of the system identically. And Singularity gives us the power to pick between these two timelines.</p>

<p>Or in summary, Singularity allows us to virtually swap out the underlying operating system for one that we defined without affecting anything else on the system and still having all of the host resources available to us.</p>

<p>It can also be described as ssh’ing into another identical host running a different operating system. One moment you are on Centos-6 and the next minute you are on the latest version of Ubuntu that has Tensorflow installed, or Debian with the latest OpenFoam, or a custom workflow that you installed.</p>

<p>Additionally what name-spaces are selected for virtualization can be dynamic or conditional. For example, the PID namespace is not separated from the host by default, but if you want to separate it, you can with a command line (or environment variable) setting. You can also decide you want to contain a process so it can not reach out to the host file system if you don’t know if you trust the image. But by default, you are allowed to interface with all of the resources, devices and network inside the container as you are outside the container.</p>

<h3 id="compatibility-with-standard-work-flows-pipes-and-io">Compatibility with standard work-flows, pipes and IO</h3>
<p>Singularity does its best to abstract the complications of running an application in a different environment then what is expected on the host. For example, applications or scripts within a Singularity container can easily be part of a pipeline that is being executed on the host. Singularity containers can also be executed from a batch script or other program (e.g. an HPC system’s resource manager) natively.</p>

<p>Some usage examples of Singularity can be seen as follows:</p>

<div class="language-bash highlighter-rouge"><pre class="highlight"><code><span class="gp">$ </span>singularity <span class="nb">exec </span>dummpy.img xterm
<span class="gp">$ </span>singularity <span class="nb">exec </span>dummy.img python script.py
<span class="gp">$ </span>singularity <span class="nb">exec </span>dummy.img python &lt; /path/to/python/script.py
<span class="gp">$ </span>cat /path/to/python/script.py | singularity <span class="nb">exec </span>dummy.img python
</code></pre>
</div>

<p>You can even run MPI executables within the container as simply as:</p>

<div class="language-bash highlighter-rouge"><pre class="highlight"><code><span class="gp">$ </span>mpirun -np X singularity <span class="nb">exec</span> /path/to/container.img /usr/bin/mpi_program_inside_container <span class="o">(</span>mpi program args<span class="o">)</span>
</code></pre>
</div>

<h3 id="the-singularity-process-flow">The Singularity Process Flow</h3>
<p>When executing container commands, the Singularity process flow can be generalized as follows:</p>

<ol>
  <li>Singularity application is invoked</li>
  <li>Global options are parsed and activated</li>
  <li>The Singularity command (subcommand) process is activated</li>
  <li>Subcommand options are parsed</li>
  <li>The appropriate sanity checks are made</li>
  <li>Environment variables are set</li>
  <li>The Singularity Execution binary is called (<code class="highlighter-rouge">sexec</code>)</li>
  <li>Sexec determines if it is running privileged and calls the <code class="highlighter-rouge">SUID</code> code if necessary</li>
  <li>Namespaces are created depending on configuration and process requirements</li>
  <li>The Singularity image is checked, parsed, and mounted in the <code class="highlighter-rouge">CLONE_NEWNS</code> namespace</li>
  <li>Bind mount points are setup</li>
  <li>The namespace <code class="highlighter-rouge">CLONE_FS</code> is used to virtualize a new root file system</li>
  <li>Singularity calls <code class="highlighter-rouge">execvp()</code> and Singularity process itself is replaced by the process inside the container</li>
  <li>When the process inside the container exits, all namespaces collapse with that process, leaving a clean system</li>
</ol>

<p>All of the above steps take approximately 15-25 thousandths of a second to run, which is fast enough to seem instantaneous.</p>

<h2 id="the-singularity-usage-workflow">The Singularity Usage Workflow</h2>
<p>The security model of Singularity (as described above, “<em>A user inside a Singularity container is the same user as outside the container</em>”) defines the Singularity workflow. There are generally two classifications of actions you would implement on a container; modification (which encompasses creation, bootstrapping, installing, admin) and using the container.</p>

<p>Modification of containers (new or existing) generally require root administrative privileges just like these actions would require on any system, container, or virtual machine. This means that a user must have a system that they have root access on. This could be a server, workstation, or even a laptop. If you are using OS X or Windows on your laptop, it is recommended to setup Vagrant, and run Singularity from there (there are recipes for this which can be found at http://singularity.lbl.gov/). Once you have Singularity installed on your endpoint of choice, this is where you will do the bulk of your container development.</p>

<p>This workflow can be described visually as follows:</p>

<p><img src="/images/docs/overview/workflow-overview.png" alt="Singularity Workflow" /></p>

<p>One the left side, you have your laptop, workstation, or a server that you control. Here you will create your containers, modify and update your containers as you need. Once you have the container with the necessary applications, libraries and data inside it can be easily shared to other hosts and executed without have root access. But if you need to make changes again to your container, you must go back to an endpoint or system that you have root on, make the necessary changes, and then re-upload the container to the computation resource you wish to execute it on.</p>

<h3 id="examples">Examples</h3>
<p>How do the commands work? We recommend you look at examples for each:</p>

<ul>
  <li><a href="/docs-shell">shell</a></li>
  <li><a href="/docs-exec">exec</a></li>
  <li><a href="/docs-run">run</a></li>
  <li><a href="/docs-bootstrap">bootstrap</a></li>
</ul>

<h2 id="support">Support</h2>

<p>Have a question, or need further information? <a href="/support">Reach out to us.</a></p>


    <!-- More navigation on the bottom -->
    <div class="row" style="padding-top:30px; margin-bottom:10px"><div class="col-md-12">
        <a href="#"><button style="width:20%; height: 70px; float:left" class="hidden previous-button btn btn-lg btn-default">Previous</button></a>
        <a href="#"><button style="width:20%; height: 70px; float:right" class="hidden next-button btn btn-lg btn-default">Next</button></a>
    </div></div>


    <a style="margin-top:10px;margin-bottom:10px" target="_blank" href="https://github.com/singularityware/singularityware.github.io/blob/master/pages/_pages/docs/user-guide.md" class="btn btn-default btn-xs githubEditButton" role="button"><i class="fa fa-github fa-lg"></i> Edit me</a>
    

    <div class="tags">
        
    </div>

    
</div>

<hr class="shaded"/>

<footer>
            <div class="row">

               <!-- Social Media links, etc -->
               <div class="col-lg-6 footer">
               
    <a class="no-after social-icon" href="https://twitter.com/SingularityApp">
      <i class="fa fa-4x fa-twitter no-after"></i>
    </a>


    <a class="no-after social-icon" href="https://github.com/singularityware">
      <i class="fa fa-4x fa-github no-after"></i>
    </a>


               </div>

                <div class="col-lg-6 footer">
<p><img src="images/logo/logo.png" alt="Company logo" style="width:40px;padding-bottom:10px"/></p>
 Site last generated: Jul 26, 2017 <br />
                </div>
            </div>
</footer>


    </div>
    <!-- /.row -->
</div>
<!-- /.container -->
    </div>

</body>

<!-- the google_analytics_id gets auto inserted from the config file -->


<script>(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)})(window,document,'script','//www.google-analytics.com/analytics.js','ga');ga('create','UA-84672381-1','auto');ga('require','displayfeatures');ga('send','pageview');</script>


</html>