Page 1 of 1

Mini Parallella Cluster managed with SLURM

PostPosted: Tue Aug 12, 2014 9:01 pm
by censix
Hi

I have installed and configured SLURM https://computing.llnl.gov/linux/slurm/
as a resource manager for my 4 parallella boards with my desktop as the controller node.
And so far it looks ok (havent done much testing yet).
However, since I do not have any previous experience with SLURM and it required quite a bit of fiddling
around to make it work, I was hoping that someone else with experience could help me validate the setup
that I have created....

If there is interest, I can post a short (and ugly) write-up of the steps I followed.

Cheers

S<

http://censix.com

Re: Mini Parallella Cluster managed with SLURM

PostPosted: Tue Aug 12, 2014 9:44 pm
by aolofsson
Cencix,
That's great! I know someone else was working on slurm for an 8 board cluster, I will check on the status.
I am sure it would be helpful to get the discussion started if you would post the steps you went through.
Andreas

Re: Mini Parallella Cluster managed with SLURM

PostPosted: Wed Aug 13, 2014 10:57 am
by 9600
Would be great to get a how-to for this up on the wiki!

Cheers,

Andrew

Re: Mini Parallella Cluster managed with SLURM

PostPosted: Wed Aug 13, 2014 12:06 pm
by censix
Andreas & Andrew

I will post what I have over the coming days. I would say that it isn't "how-to wiki material" yet until someone else replicates it. Lets see.

Soren

Re: Mini Parallella Cluster managed with SLURM

PostPosted: Wed Aug 13, 2014 12:47 pm
by 9600
censix wrote:I would say that it isn't "how-to wiki material" yet until someone else replicates it. Lets see.


I'd be interested in looking at doing this with my cluster of 4 boards in due course.

Cheers,

Andrew

Re: Mini Parallella Cluster managed with SLURM

PostPosted: Fri Aug 15, 2014 7:01 pm
by optimaler
Well, I knew I would eventually get scooped on the SLURM thing if I sat around too much, but hey, this is great!

I will try can get back on top of my parallella project after I go on vacation next week and see if I can also get SLURM working in some fashion.

Re: Mini Parallella Cluster managed with SLURM

PostPosted: Sat Aug 16, 2014 9:08 am
by censix
This post is about setting up slurm https://computing.llnl.gov/linux/slurm/ for managing a mini-cluster of 4 parallella boards. slurm is a cluster computing resource management tool used in large cluster setups in several supercomputing centres around the globe.

I am posting this to see if anyone would like to replicate what I have done and/or make suggestions on how to improve it,
since at this point I consider this to be a proof-of-concept type setup, more than anything else.


The steps are:
1) Network: Assigning proper network names to the parallella boards and the controller node
2) Installing 'munge' (binary, from a repository)
3) Installing 'slurm' (build from source)
4) Configuration
5) Launch & Basic test

1) Network:
========================================
We have four parallella boards and one desktop which will act as the controller and to all of them
we need to assign a proper name. For this setup we assume that the boards are on ip range
10.0.0.101-*.104 and the controller is at 10.0.0.100 .

The easy way. On every board (parallella??) and on the controller (desktop?) add the following to /etc/hosts:

Code: Select all
   # Add for slurm cluster
   10.0.0.101      parallella01
   10.0.0.102      parallella02
   10.0.0.103      parallella03
   10.0.0.104      parallella04
   10.0.0.100      desktop1


Also make sure that in /etc/hostname you have set the name correctly for every board/controller.

To check if this was done properly try pinging other boards by using their names, i.e.

Code: Select all
   ping parallella01
   ...
   ping parallella04



2) Installing 'munge':
========================================
The munge crtypo tool can be installed from a repository. It needs to be installed on all boards and
the controller.
Code: Select all
   sudo apt-get install munge libmunge-dev

(The slurm documentation says that instead of munge we could also use OpenSSL. https://code.google.com/p/munge/wiki/InstallationGuide)

Generate a munge key on the controller.
Code: Select all
   dd if=/dev/urandom bs=1 count=1024 >/etc/munge/munge.key
   sudo chown munge /etc/munge/munge.key
   sudo chgrp munge /etc/munge/munge.key
   sudo chmod go-rw /etc/munge/munge.key
   sudo /etc/init.d/munge start

Copy the key to the all the parallella boards.
Code: Select all
   cat /etc/munge/munge.key | ssh linaro@parallella01 'cat >> munge.key'
   ....
   cat /etc/munge/munge.key | ssh linaro@parallella04 'cat >> munge.key'

Once there, on each board, move the key to /etc/munge and start the munge demon
Code: Select all
   sudo mv ~/munge.key /etc/munge
   sudo chown munge /etc/munge/munge.key
   sudo chgrp munge /etc/munge/munge.key
   sudo chmod go-rw /etc/munge/munge.key
   sudo chmod g-w /etc  # This is only needed on the parallellas
   sudo /etc/init.d/munge start

3) Installing 'slurm' (building from source)
============================================
While slurm exists in the package repositories (as 'slurm-llnl', not to be mistaken for the network monitor called 'slurm')
I have found that if the controller node (desktop1) and the parallellas run a different OS, as is likely the case, this may
lead to problems in controller-to-board communication when we use the binary packages. So we really need to build from source.

On all boards and on the controller, first create a 'slurm' user. you can change the UID:GID (== 64030:64030) but it has to be
*the same* UID:GID on each board and on the controller, otherwise slurm will show authentication errors later on.
Code: Select all
   sudo su
   useradd slurm
   groupadd slurm
   usermod -u 64030 slurm
   groupmod -g 64030 slurm

With this done, get the latest slurm source, build and install it. Again, same procedure on all boards and on the controller.
Note that before building slurm, you have to have munge installed and running!
Code: Select all
   cd /tmp
   wget http://www.schedmd.com/download/latest/slurm-14.03.6.tar.bz2
   tar jxf *.bz2
   cd slurm*6
   ./configure
   make
   sudo make install
   sudo mkdir /var/log/slurm-llnl
   sudo chown slurm /var/log/slurm-llnl
   sudo chgrp slurm /var/log/slurm-llnl
   sudo chmod 755 /var/log/slurm-llnl

Very good. At this point we should have all necessary software installed on the boards and on the controller.


4) Configuration
============================================
Take this configuration file ('slurm.conf') and place it in /usr/local/etc on all boards and on the controller.
copy 'slurm.conf' from the next post in this thread

5) Launch & Basic test
============================================

On the controller (desktop1) start the slurm control demon.
Code: Select all
   sudo /etc/init.d/munge start
   sudo /usr/local/sbin/slurmctld

Try 'sinfo' to see the status of the cluster. since we haven't added any boards yet, this should show the parallellas in
state *Unknown*.
Code: Select all
   sinfo

On parallella01 start the slurm demon
Code: Select all
   sudo /etc/init.d/munge start
   sudo /usr/local/sbin/slurmd

Try 'sinfo' again and if everything has gone well, you should see 'parallella01' in state *Idle*.


Start the slurm demon on the remaining parallellas and 'sinfo' should show you all four nodes in state *Idle*.


On the controller you can now do.
Code: Select all
   srun -n4 hostname

Re: Mini Parallella Cluster managed with SLURM

PostPosted: Sat Aug 16, 2014 9:12 am
by censix
The 'slurm.conf' file: (adapted from the examples given by the authors of 'slurm')
=========================================

Code: Select all
###############################################################################
#                 Sample configuration file for SLURM 2
###############################################################################
#
# This file holds the system-wide SLURM configuration. It is read
# by SLURM clients, daemons, and the SLURM API to determine where
# and how to contact the SLURM controller, what other nodes reside
# in the current cluster, and various other configuration information.
#
# SLURM configuration parameters take the form Keyword=Value, where
# at this time, no spacing is allowed to surround the equals (=) sign.
# Many of the config values are not mandatory, and so may be left
# out of the config file. We will attempt to list the default
# values for those parameters in this file.
#
# This simple configuration provides a control machine named "desktop1"
# to run the Slurm's central management daemon and a single node
# named "parallella01" which execute jobs. Both machine should have Slurm
# installed and use this configuration file. If you have a similar
# configuration just change the values of ControlMachine, for the
# control machine and PartitionName and NodeName for job execution
#
###############################################################################
#
ControlMachine=desktop1
#ControlAddr=
#BackupController=
#BackupAddr=
#
#AuthType=auth/munge
AuthType=auth/none
CacheGroups=0
#CheckpointType=checkpoint/none
CryptoType=crypto/munge
#CryptoType=crypto/openssl
#DisableRootJobs=NO
#EnforcePartLimits=NO
#Epilog=
#PrologSlurmctld=
#FirstJobId=1
JobCheckpointDir=/var/lib/slurm-llnl/checkpoint
#JobCredentialPrivateKey=
#JobCredentialPublicCertificate=
#JobFileAppend=0
#JobRequeue=1
#KillOnBadExit=0
#Licenses=foo*4,bar
#MailProg=/usr/bin/mail
MailProg=none
#MaxJobCount=5000
MpiDefault=none
#MpiParams=ports:#-#
#PluginDir=
#PlugStackConfig=
#PrivateData=jobs
ProctrackType=proctrack/pgid
#Prolog=
#PrologSlurmctld=
#PropagatePrioProcess=0
#PropagateResourceLimits=
#PropagateResourceLimitsExcept=
ReturnToService=1
#SallocDefaultCommand=
SlurmctldPidFile=/var/run/slurm-llnl/slurmctld.pid
SlurmctldPort=6817
SlurmdPidFile=/var/run/slurm-llnl/slurmd.pid
SlurmdPort=6818
SlurmdSpoolDir=/var/lib/slurm-llnl/slurmd
SlurmUser=slurm
#SrunEpilog=
#SrunProlog=
StateSaveLocation=/var/lib/slurm-llnl/slurmctld
SwitchType=switch/none
#TaskEpilog=
TaskPlugin=task/none
#TaskPluginParam=
#TaskProlog=
#TopologyPlugin=topology/tree
#TmpFs=/tmp
#TrackWCKey=no
#TreeWidth=
#UnkillableStepProgram=
#UnkillableStepTimeout=
#UsePAM=0
#
#
# TIMERS
#BatchStartTimeout=10
#CompleteWait=0
#EpilogMsgTime=2000
#GetEnvTimeout=2
#HealthCheckInterval=0
#HealthCheckProgram=
InactiveLimit=0
KillWait=30
#MessageTimeout=10
#ResvOverRun=0
MinJobAge=300
#OverTimeLimit=0
SlurmctldTimeout=300
SlurmdTimeout=300
#UnkillableStepProgram=
#UnkillableStepTimeout=60
Waittime=0
#
#
# SCHEDULING
#DefMemPerCPU=0
FastSchedule=1
#MaxMemPerCPU=0
#SchedulerRootFilter=1
#SchedulerTimeSlice=30
SchedulerType=sched/backfill
SchedulerPort=7321
SelectType=select/linear
#SelectTypeParameters=
#
#
# JOB PRIORITY
#PriorityType=priority/basic
#PriorityDecayHalfLife=
#PriorityFavorSmall=
#PriorityMaxAge=
#PriorityUsageResetPeriod=
#PriorityWeightAge=
#PriorityWeightFairshare=
#PriorityWeightJobSize=
#PriorityWeightPartition=
#PriorityWeightQOS=
#
#
# LOGGING AND ACCOUNTING
#AccountingStorageEnforce=0
#AccountingStorageHost=
#AccountingStorageLoc=
#AccountingStoragePass=
#AccountingStoragePort=
AccountingStorageType=accounting_storage/none
#AccountingStorageUser=
ClusterName=parallellacluster
#DebugFlags=
#JobCompHost=
#JobCompLoc=
#JobCompPass=
#JobCompPort=
JobCompType=jobcomp/none
#JobCompUser=
JobAcctGatherFrequency=30
JobAcctGatherType=jobacct_gather/none
SlurmctldDebug=3
SlurmctldLogFile=/var/log/slurm-llnl/slurmctld.log
SlurmdDebug=3
SlurmdLogFile=/var/log/slurm-llnl/slurmd.log
#
#
# POWER SAVE SUPPORT FOR IDLE NODES (optional)
#SuspendProgram=
#ResumeProgram=
#SuspendTimeout=
#ResumeTimeout=
#ResumeRate=
#SuspendExcNodes=
#SuspendExcParts=
#SuspendRate=
#SuspendTime=
#
#
# COMPUTE NODES
NodeName=parallella01  Procs=1 State=UNKNOWN
NodeName=parallella02  Procs=1 State=UNKNOWN
NodeName=parallella03  Procs=1 State=UNKNOWN
NodeName=parallella04  Procs=1 State=UNKNOWN
#
# PARTITION
#PartitionName=allboards Nodes=parallella01  Default=YES  MaxTime=INFINITE  State=UP
PartitionName=allboards Nodes=parallella01,parallella02,parallella03,parallella04  Default=YES  MaxTime=INFINITE  State=UP