Setting up OpenMX

Please refer to the OpenMX website for detailed documentations. The installation options will depend on your system architecture and available libraries. I am installing OpenMX version 3.9.9 at one of NUS HPC clusters.

Get the source code:

wget http://t-ozaki.issp.u-tokyo.ac.jp/openmx3.9.tar.gz
wget http://www.openmx-square.org/bugfixed/21Oct17/patch3.9.9.tar.gz
tar -zxvf openmx3.9.tar.gz
rm openmx3.9.tar.gz
mv patch3.9.9.tar.gz openmx3.9/source
cd openmx3.9/source
tar -zxvf patch3.9.9.tar.gz
rm patch3.9.9.tar.gz

Using Intel libraries

If you already have Intel OneAPI/Math Kernel libraries installed, load the respective modulefiles or set environment variables:

module load xe_2015

You may check the necessary environment variables:

echo $LD_LIBRARY_PATH
echo $MKLROOT
env

Open the makefile and specify CC, FC, and LIB flags according to your system and libraries. I am using MPI and Intel Math Kernel Libraries.

CC = mpicc -O3 -qopenmp -I${MKLROOT}/include/fftw -I${MKLROOT}/include
FC = mpif90 -O3 -qopenmp -I${MKLROOT}/include/fftw
LIB= -L${MKLROOT}/include/fftw -lfftw3 -L$MKLROOT/lib/intel64 -lmkl_blacs_intelmpi_lp64 -lmkl_scalapack_lp64 -lmkl_intel_lp64 -lmkl_core -lmkl_intel_thread -lpthread -lifcore

warning

Currently I found some issues with the Intel license server while using xe_2015 module at NUS HPC clusters. After the module load, you need to export following ENV variable:

export "INTEL_LICENSE_FILE=/app1/centos6.3/Intel/xe_2015/composer_xe_2015.3.187/licenses"

If you have xe_2018 (instead of xe_2015):

module load xe_2018

Use following flags:

MKLROOT = /opt/intel/mkl
CC = mpiicc -O3 -xHOST -ip -no-prec-div -qopenmp -I${MKLROOT}/include/fftw -I${MKLROOT}/include
FC = mpiifort -O3 -xHOST -ip -no-prec-div -qopenmp
LIB= -L${MKLROOT}/include/fftw -lfftw3 -L$MKLROOT/lib/intel64 -lmkl_blacs_intelmpi_lp64 -lmkl_scalapack_lp64 -lmkl_intel_lp64 -lmkl_core -lmkl_intel_thread -lpthread -lifcore

Once you are set, compile and build the executables:

make -j8 all
make install

Compile the DosMain program:

make DosMain

There is bandgnu13.c in the source directory, which resulted in error. It needs be compiled using gcc:

module purge
gcc bandgnu13.c -lm -o bandgnu13
cp bandgnu13 ../work/

Optionally you may add the openmx3.9/work PATH to your .bashrc.

export PATH="/home/svu/{username}/openmx3.9/work:$PATH"

Install Intel oneAPI

You may use below script to download and install 2023.1 version of Intel oneAPI libraries:

scripts/intel_oneapi_components_2023.1.sh
#!/bin/bash
# Installation script for various Intel oneAPI components
# INSTALL_PATH="/workspaces/compilers"
CWD=${PWD}
BUILD_DIR=/tmp
cd ${BUILD_DIR}

sudo apt update
sudo apt install -y --no-install-recommends autoconf gawk gcc g++ make

wget https://registrationcenter-download.intel.com/akdlm/IRC_NAS/89283df8-c667-47b0-b7e1-c4573e37bd3e/l_dpcpp-cpp-compiler_p_2023.1.0.46347_offline.sh
# for custom location: --install-dir ${INSTALL_PATH}/intel
# sh ./l_dpcpp-cpp-compiler_p_2023.1.0.46347_offline.sh -a --silent --eula accept --install-dir ${INSTALL_PATH}/intel
sudo sh ./l_dpcpp-cpp-compiler_p_2023.1.0.46347_offline.sh -a --silent --eula accept
rm l_dpcpp-cpp-compiler_p_2023.1.0.46347_offline.sh

wget https://registrationcenter-download.intel.com/akdlm/IRC_NAS/150e0430-63df-48a0-8469-ecebff0a1858/l_fortran-compiler_p_2023.1.0.46348_offline.sh
# sh ./l_fortran-compiler_p_2023.1.0.46348_offline.sh -a --silent --eula accept --install-dir ${INSTALL_PATH}/intel
sudo sh ./l_fortran-compiler_p_2023.1.0.46348_offline.sh -a --silent --eula accept
rm l_fortran-compiler_p_2023.1.0.46348_offline.sh

wget https://registrationcenter-download.intel.com/akdlm/IRC_NAS/7dcd261b-12fa-418a-b61b-b3dd4d597466/l_tbb_oneapi_p_2021.9.0.43484_offline.sh
# sh ./l_tbb_oneapi_p_2021.9.0.43484_offline.sh -a --silent --eula accept --install-dir ${INSTALL_PATH}/intel
sudo sh ./l_tbb_oneapi_p_2021.9.0.43484_offline.sh -a --silent --eula accept
rm l_tbb_oneapi_p_2021.9.0.43484_offline.sh

wget https://registrationcenter-download.intel.com/akdlm/IRC_NAS/718d6f8f-2546-4b36-b97b-bc58d5482ebf/l_mpi_oneapi_p_2021.9.0.43482_offline.sh
# sh ./l_mpi_oneapi_p_2021.9.0.43482_offline.sh -a --silent --eula accept --install-dir ${INSTALL_PATH}/intel
sudo sh ./l_mpi_oneapi_p_2021.9.0.43482_offline.sh -a --silent --eula accept
rm l_mpi_oneapi_p_2021.9.0.43482_offline.sh

wget https://registrationcenter-download.intel.com/akdlm/IRC_NAS/cd17b7fe-500e-4305-a89b-bd5b42bfd9f8/l_onemkl_p_2023.1.0.46342_offline.sh
# sh ./l_onemkl_p_2023.1.0.46342_offline.sh -a --silent --eula accept --install-dir ${INSTALL_PATH}/intel
sudo sh ./l_onemkl_p_2023.1.0.46342_offline.sh -a --silent --eula accept
rm l_onemkl_p_2023.1.0.46342_offline.sh

cd ${CWD}

Initialize OneAPI/MKL env:

source /opt/intel/oneapi/setvars.sh

OpenMX makefile configuration:

MKLROOT = /opt/intel/oneapi/mkl/2023.1.0
CC = mpiicc -O3 -xHOST -ip -no-prec-div -qopenmp -I${MKLROOT}/include/fftw -I${MKLROOT}/include
FC = mpiifort -O3 -xHOST -ip -no-prec-div -qopenmp
LIB= -L${MKLROOT}/lib/intel64 -lmkl_scalapack_lp64 -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -lifcore -lmkl_blacs_intelmpi_lp64 -liomp5 -lpthread -lm -ldl

Standalone script to perform OpenMX installations:

scripts/openmx_intel.sh
#!/bin/bash
# this script is not compatible with Bourne shell (sh), use bash instead:
# bash openmx_intel.sh
# tested on ubuntu 22.04

# exit upon any command failure
set -e

OPENMX_VER="3.9.9"
MINOR_VER="3.9"
PATCH_URL="https://www.openmx-square.org/bugfixed/21Oct17/patch${OPENMX_VER}.tar.gz"
DOWNLOAD_URL="https://www.openmx-square.org/openmx${MINOR_VER}.tar.gz"
INSTALL_DIR="${HOME}/openmx/${OPENMX_VER}"
NUM_PROCS=$(nproc)

BUILD_DIR=/tmp/_build_$(date +'%Y%m%d%H%M%S')
CWD=${PWD}
mkdir ${BUILD_DIR} && cd $_

source /opt/intel/oneapi/setvars.sh

wget ${DOWNLOAD_URL}
tar -xf openmx${MINOR_VER}.tar.gz
rm openmx${MINOR_VER}.tar.gz
cd openmx${MINOR_VER}/source
wget ${PATCH_URL}
tar -xf patch${OPENMX_VER}.tar.gz
rm patch${OPENMX_VER}.tar.gz
mv kpoint.in ../work/

# edit makefile
# MKLROOT = /opt/intel/mkl
# CC = mpiicc -O3 -xHOST -ip -no-prec-div -qopenmp -I${MKLROOT}/include/fftw -I${MKLROOT}/include
# FC = mpiifort -O3 -xHOST -ip -no-prec-div -qopenmp
# LIB = -L${MKLROOT}/include/fftw -lfftw3 -L$MKLROOT/lib/intel64 -lmkl_blacs_intelmpi_lp64 -lmkl_scalapack_lp64 -lmkl_intel_lp64 -lmkl_core -lmkl_intel_thread -lpthread -lifcore

sed -i 's/^\s*MKLROOT\s*=.*/MKLROOT = \/opt\/intel\/oneapi\/mkl\/2023\.1\.0/' ${BUILD_DIR}/openmx${MINOR_VER}/source/makefile
sed -i 's/^\s*CC\s*=.*/CC = mpiicc -O3 -xHOST -ip -no-prec-div -qopenmp -I${MKLROOT}\/include\/fftw -I${MKLROOT}\/include/' ${BUILD_DIR}/openmx${MINOR_VER}/source/makefile
sed -i 's/^\s*FC\s*=.*/FC = mpiifort -O3 -xHOST -ip -no-prec-div -qopenmp/' ${BUILD_DIR}/openmx${MINOR_VER}/source/makefile
sed -i 's/^\s*LIB\s*=.*/LIB = -L${MKLROOT}\/lib\/intel64 -lmkl_scalapack_lp64 -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -lifcore -lmkl_blacs_intelmpi_lp64 -liomp5 -lpthread -lm -ldl/' ${BUILD_DIR}/openmx${MINOR_VER}/source/makefile

make DosMain
make -j${NUM_PROCS} all
sudo make install
mpiicc bandgnu13.c -lm -o bandgnu13
cp bandgnu13 ../work/

if [ ! -d ${INSTALL_DIR} ]; then
  mkdir -p ${INSTALL_DIR}
fi

cd ${CWD}
cp -r ${BUILD_DIR}/openmx${MINOR_VER}/* ${INSTALL_DIR}
rm -rf ${BUILD_DIR}

# run tests (calculations need to be launched from ${INSTALL_DIR}/work)
# cd ${INSTALL_DIR}/work
# source /opt/intel/oneapi/setvars.sh
# export OMP_NUM_THREADS=1
# mpirun -np ${NUM_PROCS} ./openmx -runtest

Using GNU libraries

Alternatively, you may use following script to install OpenMX using GNU libraries:

scripts/openmx_gnu.sh
#!/bin/bash
# this script is not compatible with Bourne shell (sh), use bash instead:
# bash openmx_gnu.sh
# tested on ubuntu 22.04

# exit upon any command failure
set -e

OPENMX_VER="3.9.9"
MINOR_VER="3.9"
PATCH_URL="https://www.openmx-square.org/bugfixed/21Oct17/patch${OPENMX_VER}.tar.gz"
DOWNLOAD_URL="https://www.openmx-square.org/openmx${MINOR_VER}.tar.gz"
INSTALL_DIR="${HOME}/openmx/${OPENMX_VER}"
NUM_PROCS=$(nproc)

BUILD_DIR=/tmp/_build_$(date +'%Y%m%d%H%M%S')
CWD=${PWD}
mkdir ${BUILD_DIR} && cd $_

sudo apt update && sudo apt upgrade -y
sudo apt install --no-install-recommends -y \
  autoconf \
  build-essential \
  ca-certificates \
  gfortran \
  libblas3 \
  libopenblas-dev \
  libc6 \
  libfftw3-dev \
  libgcc-s1 \
  liblapack-dev \
  libopenmpi-dev \
  libscalapack-openmpi-dev \
  libelpa-dev \
  wget

if ! [ -f /usr/lib/x86_64-linux-gnu/libscalapack.so ] ; then
  if ! [ -h /usr/lib/x86_64-linux-gnu/libscalapack.so ] ; then
    sudo ln -s /usr/lib/x86_64-linux-gnu/libscalapack-openmpi.so /usr/lib/x86_64-linux-gnu/libscalapack.so
  fi
fi

wget ${DOWNLOAD_URL}
tar -xf openmx${MINOR_VER}.tar.gz
rm openmx${MINOR_VER}.tar.gz
cd openmx${MINOR_VER}/source
wget ${PATCH_URL}
tar -xf patch${OPENMX_VER}.tar.gz
rm patch${OPENMX_VER}.tar.gz
mv kpoint.in ../work/

# edit makefile
# to find specific paths try: apt search fftw3; dpkg -L libfftw3-dev; ompi_info

# CC = mpicc -O3 -ffast-math -fcommon -fopenmp
# FC = mpif90 -O3 -ffast-math -fallow-argument-mismatch -fopenmp
# LIB = -lfftw3 -lmpi -lmpi_mpifh -lscalapack -llapack -lblas -lgfortran
sed -i 's/^\s*CC\s*=.*/CC = mpicc -O3 -ffast-math -fcommon -fopenmp/' ${BUILD_DIR}/openmx${MINOR_VER}/source/makefile
sed -i '/^\s*MKLROOT/d' ${BUILD_DIR}/openmx${MINOR_VER}/source/makefile
sed -i 's/^\s*FC\s*=.*/FC = mpif90 -O3 -ffast-math -fallow-argument-mismatch -fopenmp/' ${BUILD_DIR}/openmx${MINOR_VER}/source/makefile
sed -i 's/^\s*LIB\s*=.*/LIB = -lfftw3 -lmpi -lmpi_mpifh -lscalapack -llapack -lblas -lgfortran/' ${BUILD_DIR}/openmx${MINOR_VER}/source/makefile

make DosMain
make -j${NUM_PROCS} all
sudo make install
mpicc bandgnu13.c -lm -o bandgnu13
cp bandgnu13 ../work/

if [ ! -d ${INSTALL_DIR} ]; then
  mkdir -p ${INSTALL_DIR}
fi

cp -r ${BUILD_DIR}/openmx${MINOR_VER}/* ${INSTALL_DIR}
rm -rf ${BUILD_DIR}

# run tests (calculations need to be launched from ${INSTALL_DIR}/work)
# cd ${INSTALL_DIR}/work
# export OMP_NUM_THREADS=1
# mpirun -np ${NUM_PROCS} ./openmx -runtest
cd ${CWD}

Verify installation

If you like to run tests to verify your installation:

cd openmx3.9/work
mpirun -np 4 ./openmx -runtest

You may compare CPU times with other machines.

Example PBS script

Here is a sample PBS job script for NUS HPC cluster:

scripts/job.pbs
#!/bin/bash
#PBS -q parallel12
#PBS -l select=2:ncpus=12:mpiprocs=12:mem=45GB
#PBS -j eo
#PBS -N openmx-test
source /etc/profile.d/rec_modules.sh
module load xe_2018
cd $PBS_O_WORKDIR;
np=$( cat ${PBS_NODEFILE} | wc -l );
mpirun -np $np -f ${PBS_NODEFILE} openmx input.txt > output.txt

Input files, Jupyter notebooks, and other source files used in this tutorial can be found in my GitHub repository: pranabdas/openmx.

tip

Read through the OpenMX manual for various details.
OpenMX viewer can help you choose certain parameters for the input file e.g., basis configuration. You may also consult this reference table.
SeeK-path tool can help you build the k-path.
The work directory under OpenMX installation contains lots of example files for your reference.

Using Intel libraries​

Install Intel oneAPI​

Using GNU libraries​

Verify installation​

Example PBS script​

Using Intel libraries

Install Intel oneAPI

Using GNU libraries

Verify installation

Example PBS script