Empi

Introduction

Motivation

There are numerous MPI (Message Passing Interface) implementations in the portage tree, and far more in general. While it would be nice if everyone could settle on just one, there are a number of reasons this isn't going to happen anytime soon. For instance, binary-only applications that rely on a specific implementation/version, early support for new hardware, and, of course, developer preference.

The goal of empi is to ease the administrator burden inherent in managing numerous HPC applications and libraries, each of which support some subset of the available MPI implementations. Also, empi strives to ensure that any user of an HPC system can happily work with their favorite implementation and supporting applications, while another user may utilize an entirely different implementation, with potentially overlapping applications.

If you are only concerned with quickly getting up and running, you may skip the rest of this section as it only serves to provide a description of how empi works behind the scenes.

Empi

Heavily based on sys-devel/crossdev, empi is a script wrapping portage that uses a small bit of trickery to emerge MPI implementations and any dependent packages to unique directories in /usr/lib/mpi/. Quickly explained, this is done by making a local overlay, creating new categories that define a set consisting of a single version of one MPI implementation and any packages that utilize this implementation, and then using that category to find the correct install locations. The bulk of the work involving portage is handled by mpi.eclass.

Empi itself can handle creating the overlay directory structure, merging definitions in package.use and package.keywords, and calling emerge for the required packages. The overlay is created using symlinks, so any update to the portage tree will be replicated in the empi overlay.

mpi.eclass

The eclass uses the package category to decide where a package should be installed inside of /usr/lib/mpi/. Only packages inheriting this eclass will ever be installed in this separate root. The advantage over using ROOT=/some/where/ emerge stuff being we only need to install packages that actually require MPI or are implementations into this separate root directory, everything else is provided by the standard system.

Note that it is required that any ebuild will have to be ported to use mpi.eclass, just inheriting in anything but the simplest ebuild will not work automatically. Currently autotools based packages can be ported with minimal effort. Sadly, these are in the minority in the HPC package space, so mpi.eclass also provides mechanisms for getting install paths as well as wrappers around the standard ebuild utilities, do* and new*, with the prefix mpi_. For instance, if an ebuild needs to call dobin, it should be ported to use mpi_dobin instead.

The eclass should be transparent if empi is not being used; packages can still be emerged normally onto the system without any trace of mpi.eclass being used.

eselect-mpi

Recall that the grand unified goal is to enable users to easily work with different implementations at the same time on the same system. Therefore, unlike the majority of the other eselect modules, eselect-mpi is designed to administer a users' personal MPI environment. This is done by writing the required environment variables and other implementation specific details to ${HOME}/.env.d/mpi.sh and ${HOME}/.env.d/mpi.csh which the user must remember to source if they plan on utilizing the implementation. For the curious, all we're really doing is providing the environment variables such as PATH, MANPATH and LD_LIBRARY_PATH in this file.

A user may also use eselect-mpi to unselect an implementation. Then, after re-sourcing the appropriate file in ${HOME}/.env.d/, their environment is returned to its previous (empi-less) state and they may continue on with their work using the system MPI implementation (if installed).

Using empi

Installing empi

The above introduction makes a lot more sense once we actually get some implementations installed and start working with them. To begin, add the science overlay, which is where empi and the related ebuilds are currently being maintained.

root #emerge sys-cluster/empi

Adding an implementation and packages

To get started, we'll add a single implementation based on sys-cluster/openmpi as well as the sys-cluster/hpl package that will be built using openmpi and installed into the correct root.

Note
For the duration of this documentation, implementation will be used to refer to a package providing MPI, whereas class will be used to refer to the set of a single implementation and any packages using it.

root #/usr/bin/empi --create --class mpi-openmpi =sys-cluster/openmpi-1.2.6-r1

The above says that we want to create a new class, based on sys-cluster/openmpi-1.2.6-r1 and that we're going to call this class mpi-openmpi from now on. Empi will happily create the overlay structure in MPI_OVERLAY_DIR, duplicate anything related to openmpi in package.keywords or package.use, and emerge mpi-openmpi/openmpi-1.2.6-r1.

Important
Class names must always be prefixed with 'mpi-'

Note
Ebuilds for all implementations are actually symlinks back to the main portage tree. Therefore, any updates to the main tree will also update the empi overlay

Note that empi emerged mpi-openmpi/openmpi-1.2.6-r1 and not sys-cluster/openmpi-1.2.6-r1. By changing the category, empi is alerting mpi.eclass that it needs to handle separating this install from the standard system root. This new package can be manipulated using portage via all the normal mechanisms. Next, we add sys-cluster/hpl to the mpi-openmpi class.

root #/usr/bin/empi --class mpi-openmpi --add sys-cluster/hpl

Both openmpi and hpl are now installed in the /usr/lib/mpi/mpi-openmpi root. However, a user still needs to actually select this class in order to start using it.

user $eselect mpi set mpi-openmpi

user $eselect mpi list

Available MPI classes:
  mpi-openmpi               Enabled

user $source /etc/profile

user $eselect mpi list

Available MPI classes:
  mpi-openmpi               Enabled, In Use