Generator Guide
Introduction to the generators
The main generator within WI4MPI is responsible of the MPI functions generation used to pass the appropriate arguments to the underlying runtime MPI function. It handle the generation of MPI C routines for the interface and preload version of WI4MPI.
These file generations are driven by the python command <wi4mpi_dir>/src/generator/generator.py.
This contains the Generator class that will generate the C code files and headers.
The different generation tasks have been organized in several python modules and submodules all present in the ‘src/generator’ folder.
The python command has several options such as target versions of OpenMPI, MPICH and IntelMPI or the target MPI norm.
Some functions are not implemented yet due to their complexity, but in order to have a Frozen API, they have to be added to the generated files. Those functions trigger an MPI_Abort when they are used.
Quick generation
Requirements
Prepend
wi4mpi_dir/src/generatorin yourPYTHONPATHThe generator uses
clang-format -style=LLVMto format C files. So you must haveclang-formatin yourPATHThe generator needs following python modules:
docopt jsonschema logging jinja2
Execution
wi4mpi_dir/src/generator/generate.py will print in the standard output the following lines:
generator_logger - INFO - Starting to generate.
header_logger - INFO - MpcHeaderGenerator in progress.
header_logger - WARNING - The generation of 'src/interface/header/_MPC/wrapper_f.h' have to be done locally.
A MPC program has to be executed in order to catch MPI_MODE_XXX values.
Have a look to generator/FORTRAN/MPI_XXX_generator/MPC/gen_MPC_vars.sh
header_logger - INFO - IntelHeaderGenerator in progress.
header_logger - INFO - MpichHeaderGenerator in progress.
header_logger - INFO - OmpiHeaderGenerator in progress.
header_logger - INFO - IntelIntelHeaderGenerator in progress.
header_logger - INFO - IntelMpichHeaderGenerator in progress.
header_logger - INFO - MpichIntelHeaderGenerator in progress.
header_logger - INFO - MpichMpichHeaderGenerator in progress.
header_logger - INFO - IntelOmpiHeaderGenerator in progress.
header_logger - INFO - MpichOmpiHeaderGenerator in progress.
header_logger - INFO - OmpiIntelHeaderGenerator in progress.
header_logger - INFO - OmpiMpichHeaderGenerator in progress.
header_logger - INFO - OmpiOmpiHeaderGenerator in progress.
code_logger - INFO - Generation of preload C file.
code_logger - INFO - Generation of interface C file.
generator_logger - INFO - End
All log messages are written in generator.log.
Without any options, the generator will overwrite the header files in directories
<build_dir>/src/interface/header/_*<build_dir>/src/preload/header/*_*
and generated code files in directories
<build_dir>/src/interface/gen/<build_dir>/src/preload/gen/
Header generator
Dedicated settings
The following options are dedicated to the header generator:
--interface_header_dir=<interface_header_dir> Path to header interface generation folder.
--preload_header_dir=<preload_header_dir> Path to header preload generation folder.
--openmpi_version=<openmpi_version> Version of the target OpenMPI
--mpich_version=<mpich_version> Version of the target MPICH
--intelmpi_version=<intelmpi_version> Version of the target IntelMPI
The options for defining the paths of the generation folders are detailed in Output directories.
Options for defining versions of MPI implementations are detailed in Version numbers
Output directories
The paths to the directories where the interface and preload headers are written are configurable by the following options:
--interface_header_dirset the Generator class attributeinterface_header_dir. Default value:src/interface/header--preload_header_dirset the Generator class attributepreload_header_dir. Default value:src/preload/header
The subfolders in which the headers are actually written follow the nomenclature <APP_side_name>_<RUN_side_name>.
Table 2 and Table 3 can be consulted for an overview of the distribution of headers in the subfolders of interface_header_dir and preload_header_dir.
Version numbers
The version numbers of the MPI implementations whose headers are used as the basis for producing the wi4mpi headers are configurable by the following options:
--openmpi_versionsetopenmpi_version, the version of the target OpenMPI. Default value: 5.0.3--mpich_versionsetmpich_version, the version of the target MPICH. Default value: 4.2.0--intelmpi_versionsetintelmpi_version, the version of the target IntelMPI. Default value: 24.0.0
For each of the previous options, if defined, the value of the dictionary mpi_target_version (attribute of the Generator class) corresponding to the name of the implementation will be updated
The version number of each implementation will then be retrieved in the following way within the Generator class and its subclasses:
self.mpi_target_version["<implementation_name>"]
Warning
A check for version support for each implementation is performed in the initialization of the Generator class.
If the number is not in the mpi_availabe_target_version dictionary (attribute of the Generator class) then it will return an error and stop the generation.
Currently supported versions are:
OpenMPI: 1.8.8, 2.1.6, 4.1.6, 5.0.3
MPICH: 3.1.2, 3.4.3, 4.2.0
IntelMPI: 20.0.0, 24.0.0
Aliases
An alias system has been implemented in the
generator.pyfile and allows only the major number of the MPI implementation version to be passed in argument. Concretely, this involves associating the major number with the highest supported version number. For this thealias_<implementation_name>dictionaries are used:
alias_openmpi = {
"1": "1.8.8",
"2": "2.1.6",
"4": "4.1.6",
"5": "5.0.3",
}
alias_mpich = {
"3": "3.4.3",
"4": "4.2.0",
}
alias_intelmpi = {
"20": "20.0.0",
"24": "24.0.0",
}
Workflow
As can be seen in the graph below, the --interface_header_dir and --preload_header_dir options pass the prefixes of the writing directories of the preload and interface headers to the generator when initializing an object of the Generator class.
Their value is stored in the interface_header_dir and preload_header_dir class attributes when calling the set_directories method.
Fig. 5 Set header directories prefix
Tip
Default values are src/interface/header and src/preload/header
Then, the creation of these folders is done during the initialization of each header generation class during the execution of the Generator.generate() method.
We can see for example the case of the generation of Intel-Intel preload headers on the graph Create header directories prefix.
The value of preload_header_dir previously defined during the initialization step of the Generator class is transmitted to the set_directories method of the main HeaderGenerator class which then calls the os.makedirs builtin to create the <preload_header_dir>/INTEL_INTEL subfolder.
Fig. 6 Create header directories prefix
Once the folders are created, it is the turn of the header files.
The actions concerning them, illustrated on the graph, are also triggered by the generate method of each generation class.
The basic headers are first copied from <dir_input> to <dir_output>.
To do this, the app and run attributes are used to form the strings of the file names to be copied.
This excerpt from the code of the copy_files method illustrates the Intel-Intel case:
input_file_name = f"{self.app}-{self.mpi_target_version[f'{self.app}']}_mpio.h"
self.copy_file(input_file_name, "app_mpio.h")
input_file_name = f"{self.app}-{self.mpi_target_version[f'{self.app}']}_mpi.h"
self.copy_file(input_file_name, "app_mpi.h")
input_file_name = f"{self.run}-{self.mpi_target_version[f'{self.run}']}_mpio.h"
self.copy_file(input_file_name, "run_mpio.h")
input_file_name = f"{self.run}-{self.mpi_target_version[f'{self.run}']}_mpi.h"
self.copy_file(input_file_name, "run_mpi.h")
The names and paths of the files to be copied and their destination are passed one by one to the copy_file method.
Tip
The names of the final files are defined in the HeaderGenerator class.
_run_mpi_header_file = "run_mpi.h"
_run_mpio_header_file = "run_mpio.h"
_app_mpi_header_file = "app_mpi.h"
_app_mpio_header_file = "app_mpio.h"
_wrapper_f_header_file = "wrapper_f.h"
_run_mpi_proto_header_file = "run_mpi_proto.h"
_app_mpi_proto_header_file = "app_mpi_proto.h"
Fig. 7 Copy header files
Then the copied files are modified by the following methods:
_generate_run_mpih
_generate_run_mpioh
_generate_app_mpih
_generate_app_mpioh
_generate_run_mpi_protoh
_generate_app_mpi_protoh
_generate_wrapper_fh
All these methods are defined in the main class HeaderGenerator.
They can then be overridden in subclasses dedicated to each combination <APP>-<RUN> of MPI implementation.
Various methods are used to pool file modifications.
Thus, the complete description of the workflow for generating each header can be complex.
Preload mode: IntelMPI application side – IntelMPI runtime side
run_mpi.h
The generation of the header run_mpi.h in the Intel-Intel case is illustrated in the graph below.
An object of the class IntelIntelHeaderGenerator is initialized in the method generate_header of the class Generator.
The method generate of IntelIntelHeaderGenerator is inherited from the class HeaderGenerator by the class IntelHeaderGenerator.
We find there the call to the method dedicated to the generation of the file run_mpi.h: _generate_run_mpih.
This will successively call intel_generate_run_mpih (from the IntelHeaderGenerator class) and intel_preload_exception_header_run_mpih.
The first method applies the common modifications made by _replace_mpi_with_rmpi from the HeaderGenerator class and those of intel_exceptions_run_mpih.
The modifications are then saved in run_mpi.h.
Finally, intel_preload_exception_header_run_mpih applies the latest modifications. To do this, a file from src/resources/generator_datader containing substitution instructions is passed to the replacement_from_conf_file function and lines are deleted by the delete_lines function. These two functions belong to the textoperator module.
Fig. 8 Generating run_mpi.h for IntelMPI application side – IntelMPI runtime side
app_mpi.h
Fig. 9 Generating app_mpi.h for IntelMPI application side – IntelMPI runtime side
run_mpio.h
Fig. 10 Generating run_mpio.h for IntelMPI application side – IntelMPI runtime side
app_mpio.h
Fig. 11 Generating app_mpio.h for IntelMPI application side – IntelMPI runtime side (short workflow)
Fig. 12 Generating app_mpio.h for IntelMPI application side – IntelMPI runtime side
IntelMPI interface mode
run_mpi.h
Fig. 13 Generating run_mpi.h for IntelMPI
app_mpi.h
Fig. 14 Generating app_mpi.h for IntelMPI
run_mpio.h
Fig. 15 Generating run_mpio.h for IntelMPI
OpenMPI interface mode
run_mpi.h
Fig. 16 Generating run_mpi.h for OpenMPI
app_mpi.h
Fig. 17 Generating app_mpi.h for OpenMPI
MPICH interface mode
run_mpi.h
Fig. 18 Generating run_mpi.h for MPICH
Code generator
Dedicated settings
The following options are dedicated to the code generator:
--c_preload_gen_dir=<c_preload_gen_dir> Path to C preload generation folder.
--c_interface_gen_dir=<c_interface_gen_dir> Path to C interface generation folder.
--mpi_norm=<mpi_norm> Version of MPI norm to use
The options for defining the paths of the generation folders are detailed in Output directories.
Options for defining version of MPI norm are detailed in MPI norm
Output directories
The paths to the directories where the interface and preload generated C code are written are configurable by the following options:
--c_interface_header_dirset the Generator class attributec_interface_header_dir. Default value:src/interface/gen--c_preload_header_dirset the Generator class attributec_preload_header_dir. Default value:src/preload/gen
MPI norm
The version number of the MPI standard in which the user wants to generate the C code (see C files) can be set by the --mpi_norm option.
This option set the Generator class attribute mpi_norm which transmits the version through the generator.
By default its value is 3.1
The list of supported versions is contained in mpi_available_norm:
mpi_availabe_norm = [ "1.0", "1.1", "1.2", "2.0", "2.1", "2.2", "3.0", "3.1", "4.0"]
This value is used to select the functions implemented in the chosen standard.
To do this, mpi_norm traverses the generator following the path of Fig. 19.
Its value is transmitted to the header and code generator at the initialisation of a Generator object.
It is compared to the MPImin and MPImax values of each object of the JSON file
<wi4mpi_dir>/src/resources/generator_data/code/common/jsons/functions.json (see functions.json – C)
This comparison is performed when the JSON is loaded in load_json_file.
The result is a dictionary data["function"] containing all the functions of the standard.
Fig. 19 Worflow of mpi_norm
Note
If a json schema is given as an argument to the load_json_file function then the python jsonschema module will be used to validate the json file given for reading.
Currently the schema used to validate the previous functions.json is
<wi4mpi_dir>/src/resources/generator_data/code/common/jsons/schemas/schema_functions.json
In particular, it requires the presence of the keyword MPImin.
Generation template
File template handling C MPI routines for interface and preload version:
Non generated function integration
Normal MPI_… declaration
Function pointer to the underlying runtime MPI routine declaration
ASM code chooser
A_MPI_… declaration + function construction
Header
Temporary variable assignment, and translation
Call to the MPI runtime function
Footer
return
R_MPI_… declaration + function construction
Header
Call to the MPI runtime function
Footer
return
Attribute constructor init generation
Generating function connection:
print_symbol_c
print_symbol_c
object_gen.generate_func_asmK_tls / generate_func_asmK_tls_updated_for_interface
generate_func_c
header_func
print_temporary_decl_c + affect_temp_conv_c
print_symbol_c + affect_val_conv_c
footer_func
generate_func_r
header_func
print_symbol_c
footer_func
The Fortran MPI routines template is quite similar. The step 6 and 4 are specific to this version.
Function and mappers dictionaries
functions.json – C
Example:
MPI_Init(int *, char***);
{
"args": [
{ **1st argument**
"var": "argc", **name**
"arg_dep": "", **dependency**
"In": 1, **The argument need to be converted before any call to the underlying MPI runtime call**
"name": "int_ptr_mapper", **name of the mapper corresponding to that argument (mappers are responsible for the translation)**
"Out": 0 **The argument do not need to be converted after the MPI runtime call**
},
{ **2nd argument**
"var": "argv",
"arg_dep": "",
"In": 1,
"name": "char_ppp_converter",
"Out": 0
}
],
"name": "MPI_Init",
"ret":
{
"var": "ret",
"arg_dep": "",
"In": 0,
"name": "error_converter",
"Out": 1
},
"MPImin": 1.0,
"MPImax": 4.0
}
Some additional key words to deal with some special cases:
if : Tell to the generating process that the argument needs to be translated only if the condition within the if statements is true.
if_null : Same as ‘if’ keyword but dedicated to NULL constants.
if_dep : If provided, then the generator automatically understand that the argument tested in the “if” condition is an array, and so a loop is generated from 0 to “if_dep” (‘if_dep’ works hand in hand with ‘if’).
if_null_dep : Same as ‘if_dep’ but works with ‘if_null’
if_err: Handle special case MPI_Errhandler_set.
del : The argument needs to be deleted from the mechanism managing the database (hashtable)
del2 : Same as ‘del’.
arg_dep: Same as ‘if_dep’ but works on its own.
See Table 1 to overview the association keywords.
Wait & Test |
Waitany & Waitany |
Waitsome & Testsome |
Waitall & Testall |
|
|---|---|---|---|---|
if |
R_MPI_SUCCESS |
|||
if_null |
R_MPI_REQUEST_NULL |
|||
if_dep |
NONE |
NONE |
(*)outcount |
count |
if_null_dep |
NONE |
NONE |
array_of_indices |
NONE |
del |
request_ptr_delete |
|||
del2 |
NONE |
|||
mappers.json – C
This file contains all different metadata about mappers that needs to be called to performed any conversion. Each entry is corresponding to the mappers name which is referenced in the function.json “name” keywords within any arguments function. Those entries provide metadata relevant for the generator as represented bellow:
"int_ptr_mapper": { **name**
"local_alloc": 0, **Does the variable needs to be allocated locally. 0=no; 1=yes**
"a2r": "int_ptr_conv_a2r", **If 'in=1' from functions.json is set, then this function needs to be called**
"type": "int (*)", **type of the argument**
"r2a": "int_ptr_conv_r2a", **If 'out=1' from functions.json is set, then this function needs to be called**
"no_map": "TRUE" **The argument does not need to be converted if 'TRUE'**
}
All relevent keywords that a mappers can contain are:
no_map : indicate if the argument needs to be converted
assign : indicate that the arguments simply needs a cast
local_alloc : indicate that the variable needs to be allocated locally
wrap : Special case where the argument ‘wrap’ is a function pointer. (example MPI_Op_create)
wrapped : ‘wrapped’ contain the function name which is called to translate the arguments of the function referenced by ‘wrap’.
Example of “wrap” and “wrapped”: “wrapper_user_function”
When a call to A_MPI_Op_create(A_MPI_User_function * user_fn,int commute,A_MPI_Op * op);, the user_fn et op arguments needs to be converted
int A_MPI_Op_create(A_MPI_User_function * user_fn,int commute,A_MPI_Op * op)
{
in_w=1;
ptr_user_func=(A_MPI_User_function * )user_fn;
R_MPI_Op op_ltmp;
R_MPI_Op * op_tmp=&op_ltmp;
int ret_tmp= LOCAL_MPI_Op_create( (R_MPI_User_function * ) wrapper_user_function, commute, op_tmp);
op_conv_r2a(op,op_tmp); **conversion de op**
in_w=0;
return error_code_conv_r2a(ret_tmp);
}
user_fn is a function pointer where (MPI_Datatype * ) is referenced:
typedef void (MPI_User_function) (void * , void * , int * , MPI_Datatype * );
Conversion de user_fn :
void wrapper_user_function(void * invec, void * inoutvec, int * len,R_MPI_Datatype * type)
{
A_MPI_Datatype datatype_tmp;
datatype_conv_r2a( &datatype_tmp,type);
(ptr_user_func)(invec, inoutvec, len, & datatype_tmp);
}
The following keywords are set for user_fn in MPI_Op_create:
“wrap” : “user_fn”
“wrapped” : “wrapper_user_function”
functions.json – Fortran special case
Some special cases are handle thanks to the “assoc” field which allow to make some bounds between the hash table and both of the following parameters.
Exemple:
"assoc":[
{
"func":"Keyval_translation_del",
"key":"keyval_tmp"
}
]
mappers.json – Fortran special cases
The fields ‘nomap’ and ‘argdep’ got the same goal as ‘no_map’ and ‘arg_dep’ of C mappers.
Frozen API
In order to get the frozen API proceed as follow:
Just copy the contents of A the file into the B file:
A |
B |
|
interface_api_fige.c |
<—> |
interface/gen/test_wrapper_generation.c |
interface_api_fige_fortran.c |
<—> |
interface/gen/wrapper.c |
interface_api_fige_fortran_interface.c |
<—> |
interface/gen/interface_fort.c |
preload_api_fige.c |
<—> |
preload/gen/test_wrapper_generation.c |
preload_api_fige_fortran.c |
<—> |
preload/gen/wrapper.c |
Input files
The generator will open several files during the proccess:
C header
JSON
jinja
raw text
Headers generator inputs
Base headers
The headers of each supported implementation are in <wi4mpi_dir>/src/resources/MPI_headers/ folder.
Here is an overview of the tree structure:
<implementation_name>/
└── <version>
└── mpi.h
with <implementation_name> equal to openmpi, mpich or intelmpi.
These files are used as a base from which Wi4MPI’s own headers will be generated.
How to add a new base header
Below is the procedure to follow to add a base header for the implementation <implementation_name> in the version <version>.
[global]: Create the folder
src/resources/MPI_headers/<implementation_name>/<version>[global]: Copy the headers inside the previous folder. The existing names are:
mpi.hmpio.h(IntelMPI, MPICH)mpi_proto.h(MPICH since 4.2.2)
[
src/generator/generator.py]: Add the<version>into the available versions dictionary:mpi_availabe_target_version[
src/generator/generator.py]: Update the default version of the implementation by editing the dictionarympi_target_version.[
src/generator/generator.py]: Complete the helpers in the module description and in the docopt strings (below the__main__check)[global]: If the implementation is new, create a file in the
src/resources/MPI_headers/wrapperf/directory similar to those existing
Now the new header is ready to be processed by the generator.
Files for regular expression
The re module is used to perform line-by-line or block-by-block replacements.
The re.sub command is used directly in the code for small replacements.
For larger replacements, lists of commands are written in files placed in src/resources/generator_data/headers. Here is the exhaustive list of these files:
header._common_generate_app_mpih.replaceintelintelheader.__aux_generate_run_mpioh.replaceintelintelheader._common_generate_app_mpih.replaceintelintelheader._common_generate_app_mpioh.replaceintelintelheader._preload_exception_header_run_mpih.replaceintelompiheader.ompi_replace_mpi_with_rmpi.replacempcheader._mpc_exceptions_run_mpih.replaceompiheader._replace_mpi_with_rmpi.ompiompi.replaceompiheader._replace_mpi_with_rmpi.replaceompiintelheader._app_to_run.replaceompiintelheader._run_to_app.bloc_p0.replaceompiintelheader._run_to_app.bloc_r0.replaceompiintelheader._run_to_app.replace
They are the inputs to the textoperator.replacement_from_conf_file command.
Note
The separator used in replacement is @.
It is also possible to perform block replacements. For this, blocks of text to be searched and replaced are written in separate files.
The following files are the inputs to the textoperator.replace_bloc_from_conf_file command:
header._common_generate_app_mpih.bloc_00.patternheader._common_generate_app_mpih.bloc_00.replaceheader._common_generate_app_mpih.bloc_01.patternheader._common_generate_app_mpih.bloc_01.replaceheader._common_generate_app_mpih.bloc_02.patternheader._common_generate_app_mpih.bloc_02.replaceheader._common_generate_app_mpih.bloc_06.patternheader._common_generate_app_mpih.bloc_06.replaceheader._common_generate_app_mpih.bloc_08a.patternheader._common_generate_app_mpih.bloc_08.patternheader._common_generate_app_mpih.bloc_08.replaceheader._common_generate_app_mpih.bloc_09.patternheader._common_generate_app_mpih.bloc_09.replace
Finally, the textoperator.delete_bloc_from_conf_file command, which is a special case of the previous command, allows you to search for and delete blocks of lines.
The following files are used for this purpose:
header._common_generate_app_mpih.bloc_03.deleteheader._common_generate_app_mpih.bloc_04.deleteheader._common_generate_app_mpih.bloc_05.deleteheader._common_generate_app_mpih.bloc_06.deleteheader._common_generate_app_mpih.bloc_07.deleteheader._common_generate_app_mpih.bloc_10.delete
Output files
The generator writes several files:
headers files
C files
log file
Header files
The concerned header files are
app_mpi.h: application side MPI headerrun_mpi.h: runtime side MPI headerwrapper_f.h: interface version of the Fortran header
and, if applicable,
run_mpio.hmpcmp.h: dedicated to MPCsctk_types.h: dedicated to MPCapp_mpio.happ_mpi_proto.h: additional file for MPICH from version 4.2.2run_mpio.hrun_mpi_proto.h: additional file for MPICH from version 4.2.2
see Table 2 and Table 3 for an overview.
Folder |
app_mpi.h |
run_mpi.h |
wrapper_f.h |
run_mpio.h |
mpcmp.h |
sctk_types.h |
|---|---|---|---|---|---|---|
_INTEL |
✓ |
✓ |
✓ |
✓ |
||
_MPC |
✓ |
✓ |
✓ |
✓ |
✓ |
✓ |
_MPICH |
✓ |
✓ (*) |
✓ |
✓ |
||
_OMPI |
✓ |
✓ |
✓ |
Note
(*) additional file
run_mpi_proto.hfor MPICH from version 4.2.2
Folder |
app_mpi.h |
run_mpi.h |
wrapper_f.h |
app_mpio.h |
run_mpio.h |
|---|---|---|---|---|---|
INTEL_INTEL |
✓ |
✓ |
✓ |
✓ |
✓ |
INTEL_MPICH |
✓ |
✓ (*) |
✓ |
✓ |
✓ |
INTEL_OMPI |
✓ |
✓ |
✓ |
✓ |
|
MPICH_INTEL |
✓ (**) |
✓ |
✓ |
✓ |
✓ |
MPICH_MPICH |
✓ (**) |
✓ (*) |
✓ |
✓ |
✓ |
MPICH_OMPI |
✓ (**) |
✓ |
✓ |
✓ |
|
OMPI_INTEL |
✓ |
✓ |
✓ |
✓ |
|
OMPI_MPICH |
✓ |
✓ (*) |
✓ |
✓ |
|
OMPI_OMPI |
✓ |
✓ |
✓ |
Note
(*) additional file
run_mpi_proto.hfor MPICH from version 4.2.2(**) additional file
app_mpi_proto.hfor MPICH from version 4.2.2
C files
The concerned code files are
<wi4mpi_dir>/src/preload/gen/mpi_translation_c.c<wi4mpi_dir>/src/interface/gen/mpi_translation_c.c<wi4mpi_dir>/src/interface/gen/interface_c.c
Log file
Warning, debug, info, error messages are written in file generator.log.
They are managed by the python module logging and configured by <wi4mpi_dir>/src/generator/logging.conf.