Batch Processing on the Calclabs

Last update: 13Jul2015

Long running batch jobs may be submitted from any of the Calclab login servers, calclabnx.math.tamu.edu. These jobs will be scheduled on available desktops in the Calclabs. "Availability" is defined as a block of time greater than 3 hours for which no class is scheduled for a particular room. We are using SLURM for batch scheduling. You may login to calclabnx using SSH or X2Go. There should be NO processing on the login servers, as they are for edit/compile/debug, job submission, and data collection purposes only. Any long running processes on the login servers will be terminated without warning and your account will be closed if the abuse continues.

Please consider checkpointing your application appropriately as nodes may go down during the running of your job.

The queues are configured as follows:

Queue Time limit Soft Mem limit
night 9 hours 1.9GB/job
weekend 42 hours 1.9GB/job

Please check this page for updates to the queue configuration.

Serial Job Example

Jobs are submitted using the sbatch command.
Example (serial job):
sbatch -p night myserjob.slrm

The contents of the myserjob.slrm file for a serial (single node) application may look something like this:

#SLURM --time=02:00:00
#SLURM -p night
cd $HOME/mysubdir
./myprog -j 1 -f outfile.lis << EOT
2
45.5 62
infile.txt
14
EOT
exit 0

In the above example, the #SLURM directives are interpreted by sbatch command and do not need to be specified on the command line. Here we're setting a walltime limit of 2 hours and using the night queue. When the job starts we change to the $HOME/mysubdir subdirectory and execute myprog in that directory with the command line arguments -j 1 -f outfile.lis. The lines between the two EOT tags contain the input to the program that would normally be read when myprog is executed interactively.

Job Status

You can see the status of your job using the squeue command.

Cancel a Job

You can cancel your queued jobs ('Q' status from squeue) by using the scancel jobID where jobID is also found from the qstat command. You can send Unix signal 15 to a running job ('R' status) with scancel -s 15 jobID.

Matlab Job Example

Matlab is only available to Texas A&M University faculty, staff, and students. This example is similar to the one above. We will be running a serial job in which we call Matlab.

#SLURM --time=00:01:00
#SLURM -p night

matlab -nojvm < $HOME/myfile.m > $HOME/matlab.out

exit 0

We start matlab with the -nojvm option to prevent loading the Java VM. The input for matlab is read from $HOME/myfile.m and output is stored in $HOME/matlab.out.

Job Arrays

A job array is created using the --array=<indices> option to sbatch. The indices can be a range such as 0-7, or distinct values, 0,2,4,6,8. The stdout and stderr files can use SLURM variables and can be specified as -o jarray-%A-%a.out. The %A represents the Array Job ID, and %a is the task ID within the job.

Consider the example where a 3-task job is submitted with sbatch --array=0-2. If the SLURM job ID assigned to this is 151, then the following environment variables would be set in each task within the job array:

#!/bin/bash
#SBATCH -J jarray
#SBATCH -p night
#SBATCH -o jarray-%A-%a.out
#SBATCH --time=00:01:00
#SBATCH -N1
#SBATCH --ntasks-per-core=1
#SBATCH --mem-per-cpu=100

echo "starting at `date` on `hostname`"

echo "SLURM_JOBID=$SLURM_JOBID"
echo "SLURM_ARRAY_JOB_ID=$SLURM_ARRAY_JOB_ID"
echo "SLURM_ARRAH_TASK_ID=$SLURM_ARRAY_TASK_ID"

echo "srun -l /bin/hostname"
srun -l /bin/hostname
sleep 30
echo "ended at `date` on `hostname`"
exit 0

Here's the command sequences for submitting the job:

$ sbatch --array=0-2 jarray.slrm
Submitted batch job 44
$ squeue
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
              44_0     night   jarray s-johnso  R       0:02      1 bloc122-00
              44_1     night   jarray s-johnso  R       0:02      1 bloc122-00
              44_2     night   jarray s-johnso  R       0:02      1 bloc122-00
$ squeue   # no output, job is complete
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
$ ls jarray-*
jarray-44-0.out  jarray-44-1.out  jarray-44-2.out
$ cat jarray-44-1.out
starting at Tue Jan  7 14:13:57 CST 2014 on bloc122-00
SLURM_JOBID=45
SLURM_ARRAY_JOB_ID=44
SLURM_ARRAH_TASK_ID=1
srun -l /bin/hostname
0: bloc122-00
ended at Tue Jan  7 14:14:27 CST 2014 on bloc122-00

To cancel one or more (or all) elements of job array:

# Submit job array and check queue
$ sbatch --array=0-7 jarray.slrm
$ squeue
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
              47_0     night   jarray s-johnso  R       0:02      1 bloc122-00
              47_1     night   jarray s-johnso  R       0:02      1 bloc122-00
              47_2     night   jarray s-johnso  R       0:02      1 bloc122-00
              47_3     night   jarray s-johnso  R       0:02      1 bloc122-00
              47_4     night   jarray s-johnso  R       0:02      1 bloc122-00
              47_5     night   jarray s-johnso  R       0:02      1 bloc122-00
              47_6     night   jarray s-johnso  R       0:02      1 bloc122-00
              47_7     night   jarray s-johnso  R       0:02      1 bloc122-00
# Cancel elements 5,6,7
$ scancel 47_[5-7]
# See that they're gone
$ squeue
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
              47_0     night   jarray s-johnso  R       0:16      1 bloc122-00
              47_1     night   jarray s-johnso  R       0:16      1 bloc122-00
              47_2     night   jarray s-johnso  R       0:16      1 bloc122-00
              47_3     night   jarray s-johnso  R       0:16      1 bloc122-00
              47_4     night   jarray s-johnso  R       0:16      1 bloc122-00
# Cancel the entire remaining array
$ scancel 47
# All gone!
$ squeue
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)

Parallel Job Example

For a parallel job using MPI for message passing, the script myparajob.pbs would issue the mpirun command as follows:

#SLURM -N 8
#SLURM --mem=4000
#SLURM --time=00:10:00
#SLURM -p weekend
cd $HOME/mysubdir
mpirun -np 8 ./myparaprog -j 1 -f outfile.list << EOT
6.6
infile.txt
-1 20
qfile.out
EOT
exit 0

This job is submitted with:
sbatch myparajob.pbs

In this example, we set a walltime limit of 10 minutes and request the job be started from the night queue on 8 nodes. When SLURM starts your job, it will allocate the nodes to be used.

Node Status

You can view the status of the cluster on the Status page. This page is only available on campus and via TAMU VPN.

Appropriate Use

The system administrators reserve the right to monitor all processes for appropriate use of the resources. Appropriate use is defined as legitimate academic work appropriate for Texas A&M University. Cracking MD5, mining bitcoins, or running apps such as Folding@Home are not considered legitimate. Abuse of the batch system will result in account termination.