The MPI model

CPMD is an MPI program: one SCF is spread over the ranks of a communicator, and one of those ranks, the parent, owns input and output. Embedding it in a host that is itself started under mpirun raises three questions: which ranks form one CPMD calculation, which rank’s result counts, and who ends MPI. This page explains the answers cpmdc gives and why.

Every rank is a copy of the host

Under mpirun, each rank runs the whole host program, and each rank’s copy of libcpmdc calls into its own copy of CPMD. CPMD’s collective operations tie those copies together: an SCF can only advance when every rank of its communicator takes part. So every rank must make the same cpmdc calls, with the same messages, in the same order. A rank that skips a call, or evaluates a different geometry, leaves the others waiting inside CPMD.

The communicator CPMD uses

cpmdc has no with_mpi option. The engine takes MPI from the CPMD archive it is linked into.

OpenCPMD keeps its world communicator in mp_comm_world. mp_start assigns that variable only when CPMD itself calls MPI_Init. When MPI is already running, mp_start leaves mp_comm_world as the host set it.

cpmdc_bind_calculator(ranks_per_calc) initialises MPI when it is not running, then splits MPI_COMM_WORLD with MPI_Comm_split, colour world_rank / ranks_per_calc and key world_rank mod ranks_per_calc, and installs the new communicator as mp_comm_world before mp_start. Each colour is one calculator: its own CPMD setup, its own SCF, its own stored orbitals, all driven by the same deck. The split happens once per process. embed_calculator_bound, a module variable in cpmd_embed_c_api, records it, and a second bind only reports the group index. A host that already holds the calculator communicator calls cpmdc_adopt_calculator_comm instead. That call reads rank and size from the adopted communicator, copies it into mp_comm_world on an OpenCPMD build, and does not call MPI_Init or MPI_Comm_split. cpmdc_adopted_comm copies the stored handle back out. A null communicator, a size that is not ranks_per_calc when that argument is positive, MPI that has not been initialised, or a build without MPI returns -1. The text is cpmdc_last_error(). A later adopt replaces the communicator when no CPMDCSession exists. If any rank of the group has a live session, every rank returns -1 and leaves the handle alone.

World size

ranks_per_calc

Calculators

Parent ranks

16

16, 0, or no call

1

0

16

4

4

0, 4, 8, 12

16

5

refused, returns -1

The adopted index

The calculator index is the lowest MPI_COMM_WORLD rank of the group divided by the adopted size. The group must be one aligned block of that many consecutive world ranks, starting at a multiple of the size. Any other layout returns -1 on every rank of the group. Indices are unique when every group has the same size. Passing the same ranks_per_calc > 0 on every rank enforces that size. A striped split of 4 ranks into groups {0, 2} and {1, 3} is refused: both have size 2 and lowest ranks 0 and 1, so both would have been index 0. Reversing the key order inside an aligned block does not change the index. One group of 4, two groups of 2, and four groups of 1 are the same rule. The mpirun how-to shows the call. rgpot is the host that loads this library and may already have split the ranks; its group contract is the rgpot MPI reference.

Open MPI’s Fortran handle for MPI_COMM_WORLD is 0, and MPI_COMM_NULL is a different value. The stored handle uses that numbering, so 0 means the world communicator. The index is the lowest world rank of an aligned block divided by that size. cpmdc does not make the SCF energy or the forces bitwise identical across rank counts. Those reductions run inside OpenCPMD (mp_sum is MPI_Allreduce, the FFT plane split follows the rank count, and OpenMP REDUCTION clauses run inside the archive). A single point at one rank and at two ranks can be compared by the maximum absolute deviation of the energy and of the forces. That comparison is a report, not a bitwise claim.

Adopted size

ranks_per_calc

Result

2

2

index = lowest world rank / 2, the same on both ranks

2

4

-1, cpmdc_last_error() names 2 and 4

4

0 or less

index = lowest world rank / 4

1

1

index = that rank’s world rank

2, striped {0, 2}

2

-1, not an aligned block

Why the parent rank’s result counts

CPMD computes the energy and ionic forces for the calculation as a whole, and its parent rank, rank 0 of mp_comm_world, is the one CPMD itself treats as holding them. cpmd.x writes the output, the restart file, and GEOMETRY from that rank. A library force call writes none of those files. The other ranks return from the same call with values of their own. They agreed with the parent in a 4-rank water single point. Over the many steps of a search they need not: in a 4-rank dimer search where every rank also ran the host’s optimiser, the ranks’ paths diverged, and only rank 0’s results matched the file route. The robust rule is that the host takes the parent’s result and broadcasts it, so every rank’s optimiser steps from the same numbers. The MPI standard gives the same advice: an application that needs a reproducible result computes it at one process and broadcasts it (MPI 4.1, section 6.9.1). cpmdc does not broadcast by default. CPMDC_RANK_CONSENSUS=1 makes it broadcast inside each calculator: after each evaluation, rank 0 of the calculator communicator sends its energy, gradient, and result snapshots to the other ranks of that communicator. cpmdc_last_rank_spread() reports how far the other ranks were from those values, which is the size of the disagreement the broadcast removed. A host that needs every calculator’s result on every world rank still broadcasts across calculators, because only the host knows which of its communicators should receive them.

One session per calculator

CPMD’s module state and the stored orbitals live in the process, one set per rank. A calculator therefore evaluates one CPMDCSession. A second session in the same process keeps the stored orbitals when the functional, deck, cutoff, cell, charge, multiplicity, and elemental composition match. Switching to a different method or composition drops them on the next evaluation. Independent calculations belong on separate calculators: one per image of a nudged elastic band, one per end of a dimer.

Who ends MPI

CPMD calls MPI_Init in its setup when MPI is not running yet, and cpmdc_bind_calculator does the same. Neither cpmdc nor the patched CPMD calls MPI_Finalize: in a host, the host decides when MPI ends, and a library that finalized MPI would break a host that still needs it. The host has to finalize on every rank before exit. Open MPI 5 counts a rank that exits without it as an abnormal termination and kills the others, which in a 4-rank single point killed rank 0 before it had written its result. cpmdc_finalize() marks the library finalized and leaves MPI alone for the same reason.

Threads inside ranks

An OpenCPMD build with OpenMP adds threads inside each rank. Ranks, threads, and calculators multiply: 16 cores can hold one calculator of 16 single-threaded ranks, four calculators of four ranks, or other combinations, set with mpirun -np, cpmdc_bind_calculator, and OMP_NUM_THREADS. The mpirun how-to shows the calls.