Escali 4.4 User Manual
Section: 3.8 Controlling communication resources
Scali MPI Connect Release 4.4 Users Guide
31
3.8 Controlling communication resources
Even though it is normally not necessary to set buffer parameters when running applications,
it can be done, e.g., for performance reasons. Scali MPI Connect automatically adjusts
communication resources based on the number of processes in each node and based on
pool_size and chunk_size.
it can be done, e.g., for performance reasons. Scali MPI Connect automatically adjusts
communication resources based on the number of processes in each node and based on
pool_size and chunk_size.
The built-in devices SMP and TCP/IP use a simplified protocol based on serial transfers. This
can be visualized as data being written into one end of a pipe and read from the other end.
Messages arriving out-of-order are buffered by the reader. The names of these standard
devices are SMP for intra-node-communication and TCP for node-to-node-communication.
can be visualized as data being written into one end of a pipe and read from the other end.
Messages arriving out-of-order are buffered by the reader. The names of these standard
devices are SMP for intra-node-communication and TCP for node-to-node-communication.
The size of the buffer inside the pipe can be adjusted by setting the following environment
variables:
variables:
• SCAFUN_TCP_TXBUFSZ - Sets the size of the transmit buffer.
• SCAFUN_TCP_RXBUFSZ - Sets the size of the receive buffer.
• SCAFUN_SMP_BUFSZ - Sets the size of the buffer for intranode-communication.
• SCAFUN_TCP_RXBUFSZ - Sets the size of the receive buffer.
• SCAFUN_SMP_BUFSZ - Sets the size of the buffer for intranode-communication.
The ringbuffers are divided into equally sized entries. The size varies differs for different
architectures and networks; see “Scali MPI Connect Release Notes” for details. An entry in the
ringbuffer, which is used to hold the information forming the message envelope, is reserved
each time a message is being sent, and is used by the inline protocol, the eagerbuffering
protocol, and the transporter protocol. In addition, one ore more entries are used by the inline
protocol for application data being transmitted.
architectures and networks; see “Scali MPI Connect Release Notes” for details. An entry in the
ringbuffer, which is used to hold the information forming the message envelope, is reserved
each time a message is being sent, and is used by the inline protocol, the eagerbuffering
protocol, and the transporter protocol. In addition, one ore more entries are used by the inline
protocol for application data being transmitted.
mpimon has the following interface for the eagerbuffer and channel thresholds:
• Channel threshold definitions
-channel_inline_threshold <size> to set threshold for inlining
• Eager threshold definitions
-eager_threshold <size> to set threshold for eager buffering
3.8.1 Communication resources on DAT-devices
All resources (buffers) used by SMC reside in shared memory in the nodes. This way multiple
processes (typically when a node has multiple CPUs) can share the communication resources.
processes (typically when a node has multiple CPUs) can share the communication resources.
SMC operates on a buffer pool. The pool is divided into equally sized parts called chunks. SMC
uses one chunk per connection to other processes. The mpimon option “pool_ size” limits the
total size of the pool and the “chunk_size” limits the block of memory that can be allocated for
a single connection.
uses one chunk per connection to other processes. The mpimon option “pool_ size” limits the
total size of the pool and the “chunk_size” limits the block of memory that can be allocated for
a single connection.
To set the pool size and the chunk size, specify:
-pool_size <size>
to set the buffer pool size
-chunk_size <size>
to set the chunk size