Skip to main content

Singularity and Apptainer

Apptainer is a fork of Singularity project, both projects are maintained separately. We can use apptainer and singularity commands interchangeably.

Install Apptainer in RHEL based systems:

dnf install -y epel-release
dnf install -y apptainer

Install in Debian based systems:

sudo apt update && sudo apt install -y \
    wget \
    curl \
    fuse3 \
    libfuse3-3 \
    uidmap \
    squashfs-tools

# download latest version of apptainer
VER=$(curl -s https://api.github.com/repos/apptainer/apptainer/releases/latest | grep tag_name | cut -d '"' -f 4 | sed 's/v//')
wget https://github.com/apptainer/apptainer/releases/download/v${VER}/apptainer_${VER}_amd64.deb

sudo apt install -y ./apptainer_${VER}_amd64.deb
rm apptainer_${VER}_amd64.deb

Pull a docker image (it will convert and write to SIF format):

apptainer pull docker://nvcr.io/hpc/quantum_espresso:qe-7.3.1

Run a program via apptainer:

apptainer exec /qe-7.4.sif /opt/q-e-qe-7.4.1/bin/pw.x -in pw.in | tee pw.out

Bind a directory (some directories are mapped automatically, such as current working directory):

apptainer exec --bind /export:/export /qe-7.4.sif /opt/q-e-qe-7.4.1/bin/pw.x -in pw.in | tee pw.out

Launch apptainer via MPI:

mpirun -np 2 apptainer exec /qe-7.4.sif /opt/q-e-qe-7.4.1/bin/pw.x -in pw.in | tee pw.out

How to pass environment variable to the container:

  1. We can pass variable with --env flag.
apptainer exec --env MY_VARIABLE="some_value" my-container.sif echo $MY_VARIABLE
  1. We can set variables with APPTAINERENV_ or SINGULARITYENV_ prefix:
export APPTAINERENV_MY_VARIABLE="some_value"
apptainer exec my-container.sif env | grep MY_VARIABLE

Also there are APPTAINERENV_PREPEND_PATH and APPTAINERENV_APPEND_PATH to modify PATH variable.

Example Apptainer definition file:

espresso.def
Bootstrap: docker
From: almalinux:9

%labels
    Maintainer Pranab Das
    Version QE-7.4.1

%environment
    export OMP_NUM_THREADS=1
    export PATH=/usr/lib64/openmpi/bin:$PATH
    export LD_LIBRARY_PATH=/usr/lib64/openmpi/lib:$LD_LIBRARY_PATH

%post
    dnf install -y epel-release
    dnf config-manager --set-enabled crb

    dnf groupinstall -y "Development Tools"
    dnf install -y wget \
        gcc \
        gcc-c++ \
        gcc-gfortran \
        openblas-devel \
        lapack-devel \
        fftw-devel \
        openmpi-devel \
        scalapack-openmpi-devel \
        libomp-devel

    wget https://github.com/QEF/q-e/archive/refs/tags/qe-7.4.1.tar.gz
    tar -xf qe-7.4.1.tar.gz
    rm -f qe-7.4.1.tar.gz
    cd q-e-qe-7.4.1

    export OMP_NUM_THREADS=1
    export PATH=/usr/lib64/openmpi/bin:$PATH
    export LD_LIBRARY_PATH=/usr/lib64/openmpi/lib:$LD_LIBRARY_PATH

    ./configure CC=mpicc \
        FC=mpifort \
        F77=mpifort \
        F90=mpifort \
        MPIF90=mpif90 \
        --enable-parallel \
        --enable-openmp \
        --with-scalapack=yes \
        --prefix=/opt/qe-7.4.1

    make all -j$(nproc)
    make install

    rm ..
    rm -rf q-e-qe-7.4.1

Build container:

apptainer build espresso.sif espresso.def

Recover apptainer definition file from an image. Apptainer file is embedded into the SIF image.

apptainer exec <image_name.sif> cat /.singularity.d/Singularity

Sandbox mode​

We can test commands interactively in sandbox mode. Create a sandbox with --sandbox or -s:

apptainer build --sandbox test_sandbox/ docker://ubuntu:latest

This will create a standard directory named test_sandbox that contains full Linux OS tree (/bin, /etc, /usr).

Now to install packages and save them to the sandbox folder, we need to enter into the container in writable mode (use --writable or -w flag). We will also need --fakeroot or -f flag to install software as root inside the container:

apptainer shell --writable --fakeroot test_sandbox/

Inside the apptainer shell, we can install packages and experiment interactively, for example:

dnf install python3

We can package the sandbox directory into a SIF image:

apptainer build -f espresso.sif test_sandbox/

After the container is build and saved as SIF image, we may delete our sandbox folder. We need to set appropriate permission in order to be able to delete:

chmod -R u+rwX test_sandbox
rm -rf test_sandbox

Cleanup apptainer cache:

apptainer cache clean --force

Managing ENV variables​

Apptainer stores ENV variables in severals files under /.singularity.d/env.

$ ls -l /.singularity.d/env
total 10
-rwxr-xr-x 1 root root 1337 Aug  1 01:32 01-base.sh
-rwxr-xr-x 1 root root   85 Aug  1 01:32 10-docker2singularity.sh
-rwxr-xr-x 1 root root 1707 Dec  6 06:18 90-environment.sh
-rwxr-xr-x 1 root root    0 Dec  6 06:09 94-appsbase.sh
-rwxr-xr-x 1 root root 3052 Aug  1 01:32 95-apps.sh
-rwxr-xr-x 1 root root 1568 Aug  1 01:32 99-base.sh
-rwxr-xr-x 1 root root  922 Aug  1 01:32 99-runtimevars.sh

Notice the numerical prefix, they are sourced in alphabetical order by the shell. The ENV variables set in the %environment definition, goes to 90-environment.sh. This file is also symlinked from /environment:

$ apptainer exec base.sif ls -l /environment
lrwxrwxrwx 1 root root 36 Dec  6 13:29 /environment -> .singularity.d/env/90-environment.sh

However, above files are regenerated/overwritten during the build, they are not carried over from the base image. We can save ENV variables to custom files under the above folder, such files are copied to subsequent child images, and automatically sourced by apptainer.

Such a file could be 91-environment.sh where we can write variables during the build (%post section):

%post
    echo 'export PATH=/test/path:$PATH' >> $APPTAINER_ENVIRONMENT

or,

%post
    cat >> $APPTAINER_ENVIRONMENT <<'EOF'
export ONEAPI_ROOT=/opt/intel-2023.1
export TBBROOT=$ONEAPI_ROOT/tbb/2021.11
EOF

Notice the single quoted 'EOF', this prevents variable substitution in the heredoc block.

We can inspect ENV variables with:

apptainer inspect --environment <image-name.sif>
apptainer exec <image-name.sif> printenv PATH

During build phase, if required, we can source variables with:

if [ -f /.singularity.d/env/91-environment.sh ]; then
    . /.singularity.d/env/91-environment.sh
fi

or source all ENV files:

for script in /.singularity.d/env/*.sh; do
    if [ -f "$script" ]; then
        . "$script"
    fi
done

Optionally, we can save variables in custom files, e.g., 91-oneapi.sh, 92-nvhpc.sh etc.

Managing SIF images with ORAS​

Install ORAS in macos:

brew install oras

Login to GitHub Container Registry with ORAS (requires GitHub personal access token with sufficient permission):

echo $CR_PAT | oras login ghcr.io -u pranabdas --password-stdin

Push a file to GHCR:

oras push ghcr.io/pranabdas/drive:1.0.0 \
    --artifact-type application/text \
    ./sample.txt

Fetch details about an object:

oras manifest fetch ghcr.io/pranabdas/drive:1.0.0

Print SHA-256-SUM of an object:

oras manifest fetch ghcr.io/pranabdas/drive:1.0.0 | jq '.layers[0].digest' | awk -F: '{print $2}'

Pull an object:

oras pull ghcr.io/pranabdas/drive:1.0.0

Pull and write object to a specific directory:

oras pull ghcr.io/pranabdas/drive:1.0.0 --output /opt

An ORAS image can be made associated with a repository by uploading to the repository namespace:

ghcr.io/pranabdas/<repository>/<image>:<tag>

We may need to specify per file media type <file-name>:<media-type> to set artifact type correctly:

oras push ghcr.io/pranabdas/ubuntu.sif \
  --artifact-type "application/vnd.sylabs.sif.layer.v1.sif" \
  "qe.sif:application/vnd.sylabs.sif.layer.v1.sif"

Run apptainer in GitHub Codespaces​

  1. Enable via docker-in-docker:
"features": {
    "ghcr.io/devcontainers/features/docker-in-docker:2": {
      "enableNonRootDocker": "true"
    }
  },
  1. If you do not need full docker-in-docker functionality, execute following commands to enable required permissions:
{
  "name": "My Codespace",
  "runArgs": [
      "--cap-add=SYS_ADMIN",
      "--security-opt", "apparmor=unconfined",
      "--device=/dev/fuse",
      "--security-opt", "systempaths=unconfined"
  ],
}
  1. Run --privileged mode (less secure than option 2):
{
  "name": "My Codespace",
  "runArgs": [
    "--privileged"
  ],
}

Resources​