1. dokumentasi instalasi (2)

HIGH PERFORMANCE (HPC)

Dokumentasi instalasi

Dibuat oleh : Reza Aditya Rahmat
Tanggal : 20 Mei 2021

1. Daftar Isi
1. Daftar Isi ............................................................................................................................2
3. Instalasi Master Node ........................................................................................................3

3.a. Instalasi OS .................................................................................................................3
3.b. Persiapan repository ....................................................................................................3
3.c. Persiapan Kernel CentOS 7.3......................................................................................3
3.d. Instalasi Mellanox OFED............................................................................................4
3.e. Install OpenHPC Components ....................................................................................6
3.f. Master node configurations.........................................................................................7
4. Mempersiapkan Image Untuk Compute Node ..................................................................7
4.a. Inisialisasi BOS image ................................................................................................8
4.b. Install Mellanox driver to BOS image ........................................................................8
4.c. Install OpenHPC Components pada BOS Image........................................................9
4.d. Menambahkan compute node pada xCAT database .................................................10
5. Penutup ............................................................................................................................14

DOKUMENTASI INSTALASI OPENHPC 2

3. Instalasi Master Node

3.a. Instalasi OS

• Gunakan CentOS 7.3 dengan konfigurasi Server with GUI
• Partisi harddisk menggunakan software raid. Software raid dipilih karena lebih unggul

daripada hardware raid dari segala sisi. Dan mudah untuk recovery
• Set interface eno1 menjadi interface internal.
• Set interface eno3 menjadi interface public
• Pastikan terhubung dengan internet pada saat melakukan instalasi.
• Disable selinux menggunakan perintah

sed 's/^\(SELINUX=\).*$/\1disabled/' -i /etc/selinux/config

• Masukkan eno1 ke trusted zone

firewall-cmd --zone=trusted --change-zone=eno1 --permanent
# atau
nmcli connection modify eno1 connection.zone trusted

3.b. Persiapan repository
• Disable base repo:

yum -y install yum-utils yum-plugin-versionlock
yum-config-manager --disable base updates extras

• Enable Vault repo:

sh -c 'cat >/etc/yum.repos.d/vault73.repo' <<EOF
[vault-base]
name=CentOS-$releasever - Vault
baseurl=http://vault.centos.org/7.3.1611/os/x86_64/
gpgcheck=0
EOF

• Enable EPEL release

yum -y --enablerepo=extras install epel-release

3.c. Persiapan Kernel CentOS 7.3
• Set time and NTP

timedatectl set-timezone Asia/Jakarta
yum install chrony
sh -c 'cat >>/etc/chrony.conf' <<EOF
allow 10.10.0.0/16
local stratum 10
EOF
systemctl enable chronyd.service
systemctl restart chronyd.service

• Instal semua kebutuhan kernel

DOKUMENTASI INSTALASI OPENHPC 3

yum -y install \
kernel \
kernel-devel \
kernel-headers \
kernel-abi-whitelists \
kernel-tools \
kernel-tools-libs \
kernel-tools-libs-devel

• Lock versi kernel dari um agar kernel tidak terupdate lagi.

yum versionlock \
kernel \
kernel-devel \
kernel-headers \
kernel-abi-whitelists \
kernel-tools \
kernel-tools-libs \
kernel-tools-libs-devel

3.d. Instalasi Mellanox OFED

• Instalasi kebutuhan menggunakan yum

yum -y install perl wget gtk2 atk cairo gcc-gfortran libxml2-python tcsh libnl lsof tcl
tk

• Download Mellanox OFED dan ekstrak.

export wdir=/root/src/
mkdir -p $wdir
cd $wdir
wget http://ds.manvis.web.id/share/MLNX_OFED_LINUX-3.4-2.0.0.0-rhel7.3-x86_64.tgz
tar -xvzf MLNX_OFED_LINUX-3.4-2.0.0.0-rhel7.3-x86_64.tgz

• Install GPG Key

wget http://www.mellanox.com/downloads/ofed/RPM-GPG-KEY-Mellanox
sudo rpm --import RPM-GPG-KEY-Mellanox

• Masukkan Mellanox OFED ke dalam yum repository
Untuk menjalankan ini diiamge:
1. Masuk ke image dengan perintah chroot /install/netboot/centos7.5.2020/rootimg/
2. Edit yum.repoo sperti dibawah:

sh -c 'cat >/etc/yum.repos.d/mlnx_ofed.repo' <<EOF
[mlnx_ofed]
name=MLNX_OFED Repository
baseurl=file:///root/src/MLNX_OFED_LINUX-3.4-2.0.0.0-rhel7.3-x86_64/RPMS
enabled=1
gpgkey=file:///root/src
gpgcheck=1
EOF

• Install Mellanox OFED melalui yum

yum -y install mlnx-ofed-basic perftest ibutils ibutils2 infiniband-diags infiniband-
diags-comp opensm

DOKUMENTASI INSTALASI OPENHPC 4

DOKUMENTASI INSTALASI OPENHPC 5

• Buka kembali base repo

yum-config-manager --enable base updates extras
yum-config-manager --disable vault-base

• Daftarkan openibd ke systemctl lalu reboot

systemctl enable openibd

# Reboot
reboot

3.e. Install OpenHPC Components

• Setelah memastikan driver Mellanox dan kernel terinstall dan infiniband sudah dapat
dijalankan, anda dapat melakukan update yum dengan perintah

yum update

• Export local variables naming.
Pada resep kali ini kita akan memakai beberapa variabel bash yang berulang-ulang.
Akan sangat mudah bila mendefinisikannya terlebih dahulu

export sms_name=hpc2018
export sms_ip=10.10.0.19
export sms_eth_internal=eno1 enp17s0f0
export internal_netmask=255.255.0.0
export sms_ipoib=20.20.0.19
export ipoib_netmask=255.255.0.0
export ipoib_network=20.20.0.0
export domain_name=lapan.go.id
export bmc_username=ADMIN
export bmc_password=ADMIN

• Enable OpenHPC repository for local use

yum install http://build.openhpc.community/OpenHPC:/1.3/CentOS_7/x86_64/ohpc-release-
1.3-1.el7.x86_64.rpm

• Enable xCAT repository for local use
Pada instalasi openHPC kita diberi dua pilihan provisioning software, yang berguna
untuk melakukan konfigurasi slave-node, instalasi, dan managemen. Pada resep ini,
akan digunakan xCAT dengan konfigurasi stateless (tanpa harddisk pada slave node).

yum -y install yum-utils
wget -P /etc/yum.repos.d https://xcat.org/files/xcat/repos/yum/latest/xcat-core/xcat-
core.repo
wget -P /etc/yum.repos.d https://xcat.org/files/xcat/repos/yum/xcat-
dep/rh7/x86_64/xcat-dep.repo

• Add provisioning services on master node

yum -y install ohpc-base
yum -y install xCAT

# Enable xCAT tools for use in current shell
. /etc/profile.d/xcat.sh

DOKUMENTASI INSTALASI OPENHPC 6

• Add resource management services on master node
OpenHPC menyediakan beberapa pilihan untuk resource management. Pada resep ini
akan digunakan slurm sebagai resource management dari HPC ini.

yum -y install ohpc-slurm-server

# set nama master node pada file konfigurasi slurm
perl -pi -e "s/ControlMachine=\S+/ControlMachine=${sms_name}/" /etc/slurm/slurm.conf

• Setting eno1

# Setting
sh -c 'cat >/etc/sysconfig/network-scripts/ifcfg-eno1' <<EOF
DEVICE=eno1
IPADDR=${sms_ip}
NETMASK=${internal_netmask}
BOOTPROTO=none
ONBOOT=yes
MTU=1500
EOF

# Bring up the interface
ifup eno1

• Setting IPoIB

# Copy template
cp /opt/ohpc/pub/examples/network/centos/ifcfg-ib0 /etc/sysconfig/network-scripts

# Setting
perl -pi -e "s/master_ipoib/${sms_ipoib}/" /etc/sysconfig/network-scripts/ifcfg-ib0
perl -pi -e "s/ipoib_netmask/${ipoib_netmask}/" /etc/sysconfig/network-scripts/ifcfg-
ib0

# Bring up the interface
ifup ib0

• Register internal provisioning interface with xCAT for DHCP

chdef -t site dhcpinterfaces="${sms_eth_internal}"

3.f. Master node configurations

• Enable NFS server for home and OpenHPC public packages

# Export /home and OpenHPC public packages from master server
echo "/home *(rw,no_subtree_check,fsid=10,no_root_squash)" >> /etc/exports
echo "/opt/ohpc/pub *(ro,no_subtree_check,fsid=11)" >> /etc/exports
exportfs -a
systemctl restart nfs-server
systemctl enable nfs-server

4. Mempersiapkan Image Untuk Compute Node

Resep ini menggunakan sistem stateless untuk instalasi. Yaitu, compute node akan selalu boot
melalui PXE dengan image custom yang sudah kita buat. Compute node tidak memiliki
instalasi OS sendiri, dan tidak membutuhkan harddisk. Jika kita ingin mengubah konfigurasi
compute node, kita hanya butuh mengubah imagenya, dan mereboot semua compute node.

DOKUMENTASI INSTALASI OPENHPC 7

Pada bagian ini akan dijelaskan cara-cara untuk membuat image provisioning dari nol.
4.a. Inisialisasi BOS image

• Download iso image. Bisa melalui mirror repository, atau dari website resmi.

mkdir -p ~/src/
cd ~/src
wget http://202.90.199.146/iso/CentOS-7-x86_64-DVD-1611.iso

• Build initial BOS image

copycds -n centos7.5 -a x86_64 ~/src/CentOS-7-x86_64-DVD-1611.iso

• Save chroot location for compute image

export CHROOT=/install/netboot/centos7.6/x86_64/compute/rootimg/

4.b. Install Mellanox driver to BOS image

Instalasi driver harus dilakukan bersamaan dengan genimage. Maka langkah-langkah instalasi
harus di definisikan di xCAT sebelum genimage dilakukan

# Copy postscripts from xcat share directory to xCAT working directory
cp /opt/xcat/share/xcat/ib/scripts/Mellanox/mlnxofed_ib_install.v2
/install/postscripts/mlnxofed_ib_install

# Download ISO Driver from mellanox
cd /install/custom/
wget http://www.mellanox.com/downloads/ofed/MLNX_OFED-3.4-2.0.0.0/MLNX_OFED_LINUX-3.4-
2.0.0.0-rhel7.3-x86_64.iso
mkdir -p /install/custom/netboot/centos/

# Menambahkan kebutuhan Mellanox ofed ke pkglist. Pkglist akan diinstall pada saat
pertama kali melakukan genimage
sh -c 'cat >/install/custom/netboot/centos/compute.centos7.pkglist' <<EOF
bash
dracut-network
nfs-utils
openssl
dhclient
kernel
openssh-server
openssh-clients
iputils
bc
irqbalance
procps-ng
wget
vim-minimal
ntp
rpm
rsync
rsyslog
e2fsprogs
parted
net-tools
gzip
tar
xz
ethtool
perl
wget
gtk2
atk
cairo

DOKUMENTASI INSTALASI OPENHPC 8

gcc-gfortran
libxml2-python
tcsh
libnl
lsof
tcl
tk
EOF
# Menambahkan perintah install Mellanox ofed pada saat postinstall
rm -rf /install/custom/netboot/centos/compute.centos7.postinstall
cp -f /opt/xcat/share/xcat/netboot/centos/compute.centos7.postinstall
/install/custom/netboot/centos/
sh -c 'cat >>/install/custom/netboot/centos/compute.centos7.postinstall' <<EOF
/install/postscripts/mlnxofed_ib_install -p /install/custom/MLNX_OFED_LINUX-3.4-
2.0.0.0-rhel7.3-x86_64.iso -m '--force --basic' -n genimage -i installroot
EOF
# Mendefinisikan pkglist dan postinstall yang baru ke dalam xCAT
chdef -t osimage -o centos7.6-x86_64-netboot-compute
pkglist=/install/custom/netboot/centos/compute.centos7.pkglist
postinstall=/install/custom/netboot/centos/compute.centos7.postinstall

• Enable services

chroot $CHROOT systemctl enable openibd

4.c. Install OpenHPC Components pada BOS Image

• Build initial chroot image

genimage centos7.6-x86_64-netboot-compute

• Prepare provisioning repository

cp /etc/yum.repos.d/OpenHPC.repo $CHROOT/etc/yum.repos.d/
cp /etc/yum.repos.d/epel.repo $CHROOT/etc/yum.repos.d/

• Update BOS image

yum -y --installroot=$CHROOT update

• Install base compute package

yum -y --installroot=$CHROOT install ohpc-base-compute
# Add Slurm client support meta-package
yum -y --installroot=$CHROOT install ohpc-slurm-client
# Add Network Time Protocol (chrony) support
yum -y --installroot=$CHROOT install chrony
# Include modules user environment
yum -y --installroot=$CHROOT install lmod-ohpc

• Setting time

cd $CHROOT/etc/
ln -sf ../usr/share/zoneinfo/Asia/Jakarta localtime

DOKUMENTASI INSTALASI OPENHPC 9

sh -c 'cat >$CHROOT/etc/chrony.conf' <<EOF
server 10.10.0.19
EOF
chroot $CHROOT systemctl enable chronyd.service

• Enable NFS client

# Add NFS client mounts of /home and /opt/ohpc/pub to base image
echo "${sms_ip}:/home /home nfs nfsvers=3,nodev,nosuid,noatime 0 0" >> $CHROOT/etc/fstab
echo "${sms_ip}:/opt/ohpc/pub /opt/ohpc/pub nfs nfsvers=3,nodev,noatime 0 0" >>
$CHROOT/etc/fstab

• Identifikasi file untuk di sinkronisasi
xCAT memerlukan pendefinisian untuk file apasaja yang akan di sinkronisasi. Contoh
pada resep ini adalah file untuk pembuatan user, dan file konfigurasi slurm.

# Define path for xCAT synclist file
mkdir -p /install/custom/netboot
chdef -t osimage -o centos7.6-x86_64-netboot-compute
synclists="/install/custom/netboot/compute.synclist"

# Add desired credential files to synclist
echo "/etc/passwd -> /etc/passwd" > /install/custom/netboot/compute.synclist
echo "/etc/group -> /etc/group" >> /install/custom/netboot/compute.synclist
echo "/etc/shadow -> /etc/shadow" >> /install/custom/netboot/compute.synclist

# Add slurm configuration to synclist
echo "/etc/slurm/slurm.conf -> /etc/slurm/slurm.conf " >>
/install/custom/netboot/compute.synclist
echo "/etc/munge/munge.key -> /etc/munge/munge.key " >>
/install/custom/netboot/compute.synclist

• CATATAN!: Jika anda ingin menambahkan fitur-fitur optional pada compute node.
Lakukan pada bagian ini.

• Finalizing provisioning configuration
Jika anda sudah selesai melakukan kostumisasi fitur pada compute node, silahkan
jalankan perintah berikut untuk mencompile BOS image

packimage centos7.6-x86_64-netboot-compute

4.d. Menambahkan compute node pada xCAT database

sh -c 'cat >/root/compute-prepare.sh' <<EOF
num_computes=18
bmc_username=ADMIN
bmc_password=ADMIN

for ((i=0; i<$num_computes; i++)) ; do
let "ips = $i + 100"
let "bmcips = $i + 200"
cname=compute-$i
ip=10.10.0.$ips
ipoib=20.20.0.$ips
bmcip=10.10.0.$bmcips
macspace=`ipmitool -U ADMIN -P ADMIN -H $bmcip raw 0x30 0x21 | tail -c 18`
mac=${macspace// /:}
echo $cname $ip $bmcip $mac
mkdef -t node $cname groups=compute,all ip=$ip mac=$mac netboot=xnba \
arch=x86_64 bmc=$bmcip bmcusername=${bmc_username} bmcpassword=${bmc_password} \
nicips.ib0=${ipoib} nictypes.ib0="InfiniBand" nicnetworks.ib0=ib0 \
mgt=ipmi

done
EOF

DOKUMENTASI INSTALASI OPENHPC 10

/bin/bash /root/compute-prepare.sh

• Mendefiniskan ib0 pada xCAT

chdef -t network -o ib0 mask=$ipoib_netmask net=${ipoib_network}

# Enable secondary NIC configuration
chdef compute -p postbootscripts=confignics

• Complete network service configurations

makenetworks
chdef -t network 10_10_0_0-255_255_0_0 //dynamicrange=10.10.0.2-10.10.0.99
makedhcp -n
makehosts
makedns -n
makedhcp -a

• Set compute node ke BOS image yang telah dibuat

nodeset compute osimage=centos7.6-x86_64-netboot-compute

• Boot compute node

# set bios agar masuk ke network boot (PXE)
rsetboot compute net

# Menyalakan compute node melalui ipmi
rpower compute reset

3. Instal SLURM

#instal slurm di master node
yum –y install ohpc-slurm-server
perl -pi -e "s/ControlMachine=\S+/ControlMachine=${sms_name}/" /etc/slurm/slurm.conf

#instal slurm di compute node
export CHROOT=/install/netboot/centos7.5/x86_64/compute/rootimg/
yum -y --installroot=$CHROOT install ohpc-slurm-client

export CHROOT=/install/netboot/centos7.6/x86_64/compute/rootimg/
yum -y --installroot=$CHROOT install ohpc-slurm-client

#instal MPI Stacks
yum -y install openmpi3-gnu7-ohpc mpich-gnu7-ohpc
yum -y install mvapich2-gnu7-ohpc

#config slurm.conf
NodeName=compute-[0-17] Sockets=2 CoresPerSocket=24 ThreadsPerCore=2
Feature=HyperThread
NodeName=hpc2018 Sockets=2 CoresPerSocket=24 ThreadsPerCore=2 Feature=HyperThread
PartitionName=operasional Nodes=ALL Default=YES State=UP
PartitionName=debug Nodes=ALL Default=YES MaxTime=INFINITE State=UP

cp /etc/slurm/slurm.conf $CHROOT/etc/slurm/

# Start munge and slurm controller on master host
[sms]# systemctl enable munge
[sms]# systemctl enable slurmctld
[sms]# systemctl start munge
[sms]# systemctl start slurmctld

# Start slurm clients on compute hosts
[sms]# pdsh -w $compute_prefix[1-4] systemctl start slurmd

DOKUMENTASI INSTALASI OPENHPC 11

4. Instal WRF (Weather Research and Forecasting Model)

• Instal Library

# instal compiler (GNU Compiler)
yum –y install gnu7-compiler-ohpc
yum –y install llvm5-compilers-ohpc

# instal MPI
export DIR=/usr/local/mpich

./configure –prefix=$DIR
make
make install
export PATH=$DIR/bin:$PATH

# instal NETCDF
export DIR=/usr/local/netcdf; export CC=gcc; export CXX=g++; export FC=gfortran; export
FCFLAGS=-m64; export F77=gfortran; export FFLAGS=-m64
./configure –prefix=$DIR --disable-dap --disable-netcdf-4 --disable-shared
make
make install
export PATH=$DIR/bin:$PATH
export NETCDF=$DIR

# instal zlib
export DIR=/usr/local/zlib
./configure --prefix=$DIR
make
make install

# instal libpng
export DIR=/usr/local/libpng
./configure -–prefix=/$DIR
make
make install

# instal jasper
export DIR=/usr/local/jasper
./configure –prefix=/$DIR
make
make install

• Instal WRF

./configure
Enter selection [1-75]: 34 (dmpar GNU (gfortran/gcc))
Compile for nesting? : 1 (default)
./compile em_real >& compile.log

• Instal WPS

./configure
Enter selection [1-40]: 3 (Linux x86_64, gfortran (dmpar)))
./compile >& compile.log

• Instal ARWPOST

./configure
Enter selection [1-3]: 3 (PC Linux i486 i586 i686 x86_64, gfortran compiler)
./compile >& compile.log

DOKUMENTASI INSTALASI OPENHPC 12

• Pengaturan PATH di bashrc

##COMPILER
CC="gcc"
FC="gfortran"
F77="gfortran"
F90="gfortran"
CXX="g++"
FCFLAGS="-fPIC"
FFLAGS="-fPIC"
export CC FC F77 F90 CXX FCFLAGS FFLAGS

##NETCDF
NETCDF=/usr/local/netcdf
PATH=$NETCDF/bin:$PATH
LD_LIBRARY_PATH=$NETCDF/lib:$LD_LIBRARY_PATH
MANPATH=$NETCDF/share/man:$MAN_PATH
NETCDF_INC=$NETCDF/include
NETCDF_LIB=$NETCDF/lib
WRFIO_NCD_NO_LARGE_FILE_SUPPORT=1
export NETCDF PATH LD_LIBRARY_PATH MANPATH NETCDF_INC
export NETCDF_LIB WRFIO_NCD_LARGE_FILE_SUPPORT

##MPICH
MPICH=/usr/local/mpich
PATH=$MPICH/bin:$PATH
LD_LIBRARY_PATH=$MPICH/lib:$LD_LIBRARY_PATH
MANPATH=$MPICH/share/man:$MANPATH
INC_MPI=$MPICH/include
LIB_MPI=$MPICH/lib
export MPICH PATH LD_LIBRARY_PATH MANPATH INC_MPI LIB_MPI

##ZLIB
ZLIB=/usr/local/zlib
PATH=$ZLIB/bin:$PATH
LD_LIBRARY_PATH=$ZLIB/lib:$LD_LIBRARY_PATH
export ZLIB LD_LIBRARY_PATH PATH

##LIBPNG
LIBPNG=/usr/local/libpng
PATH=$LIBPNG/bin:$PATH
LD_LIBRARY_PATH=$LIBPNG/lib:$LD_LIBRARY_PATH
export LIBPNG LD_LIBRARY_PATH PATH

##JASPER
JASPER=/usr/local/jasper
PATH=$JASPER/bin:$PATH
LD_LIBRARY_PATH=$JASPER/lib:$LD_LIBRARY_PATH
export JASPER LD_LIBRARY_PATH PATH

ulimit -s unlimited

DOKUMENTASI INSTALASI OPENHPC 13

5. Penutup
Demikian penyusunan Laporan ini dibuat agar dapat digunakan sebagai pedoman.

DOKUMENTASI INSTALASI OPENHPC 14