Escali 4.4 User Manual

Section: 2.2 SMC network devices

• root# detstat -r det0 # reset statistics for the det0 device.
• root# detstat -r -a # resets statistics for all DET devices.

This is a RDMA capable device that uses the Myricom GM driver and library. A GM release above 
2.0 is required. This device is straight forward and requires no configuration other than the 
presence of the libgm.so library in the library path (see /etc/ld.so.conf).

Note: 
Myricom GM software is not provided by Scali. If you have purchased a Myrinet interconnect 
you have the right to use the GM source, and a source tar ball is available from Myricom. It is 
necessary to obtain the GM source since it must be compiled per kernel version. Scali provides 
tools for generating binary RPMs to ease installing and management. These tools are provided 
in the scagmbuilder package; see the Release Notes/Readme file for detailed instructions. 

Infiniband is a relatively new interconnect that has been available since 2002, and became 
affordable in 2003. On PCI-X based systems you can expect latencies around 5

υ

S and 

bandwidth up to 700-800Mb/s (please note that performance results may vary based on 
processors, memory sub system, and the PCI bridge in the chipsets).  
 

There are various Infiniband vendors that provide slightly different hardware and software 
environments. Scali have established relationships with the following vendors: Mellanox, 
Silverstorm, Cisco, and Voltaire.  

See release notes on the exact versions of software stack that is supported. Scali provide a 
utility known as ScaIBbuilder that does an automated install of some of these stacks. (See 
IBbuilders release notes).  

 

The different vendors’ InfiniBand switches vary in feature sets, but the most important 
difference is whether they have a built in subnet manager or not. An InfiniBand network must 
have a subnet manager (SM) and if the switches don't come with a builtin SM, one has to be 
started on a node attached to the IB network. The SMs of choice for software SMs are OpenSM 
or minism. If you have SM-less switches your vendor will provide one as part of their software 
bundle.  
 

SMC uses either the uDAPL (User DAT Provider Library) supplied by the IB vendor, or the low 
level VAPI/IBA layer. DAT is an established standard and is guaranteed to work with SMC. 
However better performance is usually achieved with the VAPI/IBT interfaces. However, VAPI 
is an API that is in flux and SMC is not guaranteed to work with all (current nor future) versions 
of VAPI.