综述
MPI这个东西总体来看还是相当依赖于具体实现和机器架构的。NAMD（一个分子模拟软件）文档中有下面这段话：
For workstation clusters and other massively parallel machines with special high-performance networking, NAMD uses the system-provided MPI library (with a few exceptions) and standard system tools such as mpirun are used to launch jobs. Since MPI libraries are very often incompatible between versions, you will likely need to recompile NAMD and its underlying Charm++ libraries to use these machines in parallel.
大意是说，MPI在版本之间以及厂商之间存在较大的兼容性差异，所以很多时候为了能够在高速网络互联的大型Cluster系统上面正常运行数值模式软件，不得不采用合适的编译器重新编译。
文档细节
两种进程调度器
这部分就不翻译了，文档里面说得很清楚。很多老的PBS脚本里面都默认使用MPD作为进程调度器，但是这种用法在新的编译器下即将被废弃。不过貌似老集群一般也不怎么升级编译器的样子，毕竟多数模式代码的编译器兼容性有点惨。
Hydra
Hydra is a simplified, scalable process manager. Hydra will check for known resource managers to determine where the processes may be run and to distribute the processes among the targets using proxies on each host. These proxies will be used for process launching, cleanup, I/O forwarding, signal forwarding, and other tasks.
You can start Hydra by using
mpiexec.hydra. See Scalable Process Management System (Hydra) Commands topic for a detailed list of options in the Intel® MPI Library Reference Manual.
The process manager can also be selected by directly calling the appropriate mpiexec file:mpiexec.hydrafor Hydra ormpiexecfor MPD.
MPD
MPD stands for Multipurpose Daemon. This is the Intel® MPI Library process management system for starting parallel jobs, which have to run on all nodes. MPDs gather information about the system and hardware, as well as communicate with each other to exchange required information. For example, an MPD ring is required for correct pinning under the MPD process manager.
NOTE
Multipurpose daemon (MPD) has been deprecated starting with Intel® MPI Library 5.0 release. Convert to using the scalable process management system (Hydra) instead to start parallel jobs.
参数不兼容
大意就是说两个进程调度器采用不同的参数，并且会默默地忽略掉对方的一些特有参数：
When running under a job manager, the mpirun command ignores the -r | --rsh option if Hydra* is used as the underlying process manager. In this case, the corresponding Hydra* bootstrap server is used. Use the bootstrap specific options or corresponding environment variables explicitly to override the auto detected bootstrap server.
The mpirun command silently ignores the MPD specific options for compatibility reasons if you select Hydra* as the active process manager. The following table provides the list of silently ignored and unsupported MPD* options. Avoid these unsupported options if the Hydra* process manager is used.
命令行参数
归纳汇总的命令行参数如下：
mpiexec.hydra -n 96 -hostfile ~/mpi_hosts -perhost 12 -genv I_MPI_FABRICS shm:dapl ./test
关闭架构选择时的Fallback功能
意思是说，默认情况下MPI自动选择架构，并且按照一个架构列表自上向下尝试。如果其中一个运行成功，就不会报错，但有可能会导致实际运行时的方式并非你所期望的。为了关闭这一特性，可以设置export I_MPI_FALLBACK=0，或者指定I_MPI_FABRICS环境变量。
By default, if I_MPI_FABRICS is not set, fallback will be enabled. If I_MPI_FABRICS is set, fallback will be disabled.
架构选择
I_MPI_FABRICSSelect the particular network fabrics to be used.
这是非常重要的一部分，指定程序运行时的架构。一般的架构描述是两个缩写，用冒号分割，如：shm:dapl。前者表示节点内部的连接架构，后者表示节点间的连接架构。例如，对于我们常用的Cluster架构，单个节点是普通Server相当于多核共享内存的SMP架构；节点之间采用高速Infiniband网络互联，属于DAPL-capable network fabrics，于是就有上述的架构描述。具体的可用架构列表详见文档。
调整进程分布
-perhost <# of processes >, -ppn <# of processes >, or -grr <# of processes>
Use this option to place the indicated number of consecutive MPI processes on every host in the group using round robin scheduling. See the I_MPI_PERHOST environment variable for more details.
首先这里指出了MPI在调度进程的时候所采用的算法，简而言之就是各个节点轮流获取一个进程的意思。这个参数用来设置每个节点运行多少个进程。
参数调节
Intel MPI 实现中有一个自动调优MPI参数的功能，虽然实际测试的效果一般，不过总算聊胜于无。MPI User Guide 中的"Tuning with mpitune Utility"章节描述了这个工具的用法。这个东西主要有两种工作模式：针对集群架构的参数调优，以及针对某个具体应用的参数调优。对于前者，顾名思义就是在特定集群上面跑起某个自带的应用（当然也可以指定自己的），然后看哪个配置速度快，就生成一份对应的配置文件。对于后者，则是跑一个用户自定义的