Q-Logic IB6054601-00 D Manuel D’Utilisation

B – Integration with a Batch Queuing System

A Batch Queuing Script

IB6054601-00 D

B-3

Q

by 

mpirun.

Each line consists of a node name, a colon, and the number of processes 

to start on that node.

This is one of two formats that the file may use. See 

 for more 

information.

At this point, your script has enough information to be able to run an MPI program. 
All that remains is to start the program when the batch system tells us that we can 
do so, and notify the batch system when the job completes. This is done in the final 
part of 

batch_mpirun

:

mpirun -np $np -m $mpihosts_file "$mpi_prog" $@

exit_code=$?

scancel ${SLURM_JOBID}

rm -f $mpihosts_file

exit $exit_code

The InfiniPath software will normally ensure clean termination of all MPI programs 
when a job ends, but in some rare circumstances an MPI process will remain alive, 
and potentially interfere with future MPI jobs. To avoid this problem, the usual 
solution is to run a script before and after each batch job which kills all unwanted 
processes. QLogic does not provide such a script, but it is useful to know how to 
find out which processes on a node are using the InfiniPath interconnect. The easiest 
way to do this is through use of the fuser command, which is normally installed in 

/sbin.

Run as root:                                                                                                      

# /sbin/fuser -v /dev/ipath   

/dev/ipath:    22648m 22651m                                                                                                    

In this example, processes 22648 and 22651 are using the InfiniPath interconnect. 
It is also possible to use this command (as root):

# lsof /dev/ipath

This gets a list of processes using InfiniPath. Additionally, to get all processes, 
including stats programs, ipath_sma, diags, and others, run the program in this 
way:

# /sbin/fuser -v /dev/ipath*

losf

 can also take the same form:

# lsof /dev/ipath*