用户手册 — Q-Logic IB6054601-00 D

目录

• InfiniPath User Guide

  1
• Table of Contents

  5
• Section 1 Introduction

  11
  • 1.1 Who Should Read this Guide

    11
  • 1.2 How this Guide is Organized

    11
  • 1.3 Overview

    12
  • 1.4 Switches

    12
  • 1.5 Interoperability

    12
  • 1.6 What’s New in this Release

    13
  • 1.7 Supported Distributions and Kernels

    14
  • 1.8 Software Components

    15
  • 1.9 Conventions Used in this Document

    16
  • 1.10 Documentation and Technical Support

    16
• Section 2 InfiniPath Cluster Administration

  19
  • 2.1 Introduction

    19
  • 2.2 Installed Layout

    19
  • 2.3 Memory Footprint

    20
  • 2.4 Configuration and Startup

    22
    • 2.4.1 BIOS Settings

      22
    • 2.4.2 InfiniPath Driver Startup

      23
    • 2.4.3 InfiniPath Driver Software Configuration

      23
    • 2.4.4 InfiniPath Driver Filesystem

      23
    • 2.4.5 Subnet Management Agent

      24
    • 2.4.6 Layered Ethernet Driver

      24
      • 2.4.6.1 ipath_ether Configuration on Fedora and RHEL4

        25
      • 2.4.6.2 ipath_ether Configuration on SUSE 9.3, SLES 9, and SLES 10

        26
    • 2.4.7 OpenFabrics Configuration and Startup

      29
      • 2.4.7.1 Configuring the IPoIB Network Interface

        29
    • 2.4.8 OpenSM

      30
  • 2.5 SRP

    31
  • 2.6 Further Information on Configuring and Loading Drivers

    31
  • 2.7 Starting and Stopping the InfiniPath Software

    31
  • 2.8 Software Status

    33
  • 2.9 Configuring ssh and sshd Using shosts.equiv

    33
    • 2.9.1 Process Limitation with ssh

      35
  • 2.10 Performance and Management Tips

    35
    • 2.10.1 Remove Unneeded Services

      35
    • 2.10.2 Disable Powersaving Features

      36
    • 2.10.3 Balanced Processor Power

      37
    • 2.10.4 SDP Module Parameters for Best Performance

      37
    • 2.10.5 CPU Affinity

      37
    • 2.10.6 Hyper-Threading

      38
    • 2.10.7 Homogeneous Nodes

      38
  • 2.11 Customer Acceptance Utility

    40
• Section 3 Using InfiniPath MPI

  43
  • 3.1 InfiniPath MPI

    43
  • 3.2 Other MPI Implementations

    43
  • 3.3 Getting Started with MPI

    43
    • 3.3.1 An Example C Program

      44
    • 3.3.2 Examples Using Other Languages

      45
  • 3.4 Configuring MPI Programs for InfiniPath MPI

    46
  • 3.5 InfiniPath MPI Details

    47
    • 3.5.1 Configuring for ssh Using ssh-agent

      47
    • 3.5.2 Compiling and Linking

      49
    • 3.5.3 To Use Another Compiler

      50
      • 3.5.3.1 Compiler and Linker Variables

        51
    • 3.5.4 Cross-compilation Issues

      51
    • 3.5.5 Running MPI Programs

      52
    • 3.5.6 The mpihosts File

      53
    • 3.5.7 Console I/O in MPI Programs

      54
    • 3.5.8 Environment for Node Programs

      54
      • 3.5.8.1 Environment for Multiple Versions of InfiniPath or MPI

        55
    • 3.5.9 Multiprocessor Nodes

      56
    • 3.5.10 mpirun Options

      56
  • 3.6 Using Other MPI Implementations

    59
  • 3.7 MPI Over uDAPL

    59
  • 3.8 MPD

    59
    • 3.8.1 MPD Description

      60
    • 3.8.2 Using MPD

      60
  • 3.9 File I/O in MPI

    60
    • 3.9.1 Linux File I/O in MPI Programs

      60
    • 3.9.2 MPI-IO with ROMIO

      61
  • 3.10 InfiniPath MPI and Hybrid MPI/OpenMP Applications

    61
  • 3.11 Debugging MPI Programs

    62
    • 3.11.1 MPI Errors

      62
    • 3.11.2 Using Debuggers

      62
  • 3.12 InfiniPath MPI Limitations

    63
• Appendix A Benchmark Programs

  65
  • A.1 Benchmark 1: Measuring MPI Latency Between Two Nodes

    65
  • A.2 Benchmark 2: Measuring MPI Bandwidth Between Two Nodes

    66
  • A.3 Benchmark 3: Messaging Rate Microbenchmarks

    67
  • A.4 Benchmark 4: Measuring MPI Latency in Host Rings

    69
• Appendix B Integration with a Batch Queuing System

  71
  • B.1 A Batch Queuing Script

    71
    • B.1.1 Allocating Resources

      71
    • B.1.2 Generating the mpihosts File

      72
    • B.1.3 Simple Process Management

      73
    • B.1.4 Clean Termination of MPI Processes

      73
  • B.2 Lock Enough Memory on Nodes When Using SLURM

    74
• Appendix C Troubleshooting

  75
  • C.1 Troubleshooting InfiniPath Adapter Installation

    75
    • C.1.1 Mechanical and Electrical Considerations

      75
    • C.1.2 Some HTX Motherboards May Need 2 or More CPUs in Use

      76
  • C.2 BIOS Settings

    76
    • C.2.1 MTRR Mapping and Write Combining

      77
    • C.2.2 Incorrect MTRR Mapping

      77
    • C.2.3 Incorrect MTRR Mapping Causes Unexpected Low Bandwidth

      78
    • C.2.4 Change Setting for Mapping Memory

      78
    • C.2.5 Issue with SuperMicro H8DCE-HTe and QHT7040

      78
  • C.3 Software Installation Issues

    79
    • C.3.1 OpenFabrics Dependencies

      79
    • C.3.2 Install Warning with RHEL4U2

      79
    • C.3.3 mpirun Installation Requires 32-bit Support

      79
    • C.3.4 Installing Newer Drivers from Other Distributions

      80
    • C.3.5 Installing for Your Distribution

      81
  • C.4 Kernel and Initialization Issues

    81
    • C.4.1 Kernel Needs CONFIG_PCI_MSI=y

      82
    • C.4.2 pci_msi_quirk

      82
    • C.4.3 Driver Load Fails Due to Unsupported Kernel

      83
    • C.4.4 InfiniPath Interrupts Not Working

      83
    • C.4.5 OpenFabrics Load Errors If ib_ipath Driver Load Fails

      84
    • C.4.6 InfiniPath ib_ipath Initialization Failure

      85
    • C.4.7 MPI Job Failures Due to Initialization Problems

      85
  • C.5 OpenFabrics Issues

    86
    • C.5.1 Stop OpenSM Before Stopping/Restarting InfiniPath

      86
    • C.5.2 Load and Configure IPoIB Before Loading SDP

      86
    • C.5.3 Set $IBPATH for OpenFabrics Scripts

      86
  • C.6 System Administration Troubleshooting

    86
    • C.6.1 Broken Intermediate Link

      87
  • C.7 Performance Issues

    87
    • C.7.1 MVAPICH Performance Issues

      87
  • C.8 InfiniPath MPI Troubleshooting

    87
    • C.8.1 Mixed Releases of MPI RPMs

      87
    • C.8.2 Cross-compilation Issues

      88
    • C.8.3 Compiler/Linker Mismatch

      89
    • C.8.4 Compiler Can’t Find Include, Module or Library Files

      89
    • C.8.5 Compiling on Development Nodes

      90
    • C.8.6 Specifying the Run-time Library Path

      90
    • C.8.7 Run Time Errors With Different MPI Implementations

      91
    • C.8.8 Process Limitation with ssh

      93
    • C.8.9 Using MPI.mod Files

      93
    • C.8.10 Extending MPI Modules

      94
    • C.8.11 Lock Enough Memory on Nodes When Using a Batch Queuing System

      95
    • C.8.12 Error Messages Generated by mpirun

      96
      • C.8.12.1 Messages from the InfiniPath Library

        96
      • C.8.12.2 MPI Messages

        98
      • C.8.12.3 Driver and Link Error Messages Reported by MPI Programs

        101
    • C.8.13 MPI Stats

      102
  • C.9 Useful Programs and Files for Debugging

    103
    • C.9.1 Check Cluster Homogeneity with ipath_checkout

      103
    • C.9.2 Restarting InfiniPath

      103
    • C.9.3 Summary of Useful Programs and Files

      104
    • C.9.4 boardversion

      105
    • C.9.5 ibstatus

      106
    • C.9.6 ibv_devinfo

      106
    • C.9.7 ident

      106
    • C.9.8 ipath_checkout

      107
    • C.9.9 ipath_control

      108
    • C.9.10 ipathbug-helper

      109
    • C.9.11 ipath_pkt_test

      109
    • C.9.12 ipathstats

      109
    • C.9.13 lsmod

      110
    • C.9.14 mpirun

      110
    • C.9.15 rpm

      110
    • C.9.16 status_str

      110
    • C.9.17 strings

      112
    • C.9.18 version

      112
• Appendix D Recommended Reading

  113
  • D.1 References for MPI

    113
  • D.2 Books for Learning MPI Programming

    113
  • D.3 Reference and Source for SLURM

    113
  • D.4 InfiniBand

    113
  • D.5 OpenFabrics

    113
  • D.6 Clusters

    114
  • D.7 Rocks

    114
• Appendix E Glossary

  115
• Index

  121