Writing Custom Drivers (Advanced)

Note

Most users should use generic_cylinders.py (see generic_cylinders.py). This section is for developers who need features not yet exposed through generic_cylinders.py or who want to understand the internal architecture.

Advanced users might want features that are not yet supported in mpisppy.generic_cylinders.py and so they might want to create their own driver. Also, users who want to maintain, or modify a copy of mpisppy.generic_cylinders.py may want some background on drivers, which is provided in this section.

Nearly the last step in most drivers is a call to mpisppy.utils.sputils.spin_the_wheel that calls the hub and spokes. Many of the example drivers take advantage of shared code in the mpisppy.utils directory.

We now proceed to consider driver examples that do not use the shared services provided with the examples; however, many developers will prefer to mimic the examples.

Starting with Examples

The utils directory has utilities that set up command line options and create dictionaries used to create hubs and spokes. The main shared utilities are

  • config.py that creates a Pyomo Config object and command line options.

  • cfg_vanilla.py (was vanilla.py) that creates dictionaries used for hub and spoke creation. These dictionaries are ultimately fed to sputils.spin_the_wheel.

The constructors for the vanilla spokes take arguments that vary slightly depending on the spoke, but all want a configuration object, followed by scenario_creator function, a scenario_denoument function (that can be None), a list of scenario names as all_scenario_names, and scenario_creator_kwargs. Other arguments can be seen in the file mpisppy.utils.vanilla.py or in the the *_cylinders.py files that use it. Since all require the first four, in the examples, they are often collected into a tuple called beans so they can be passed to the constructor for every vanilla spoke.

Extending Examples

Many developers will need to add extensions. Here are few examples:

  • The model file examples.farmer.farmer.py defines an inparser_adder function that registers a --crops-multiplier option on the Config, and a kw_creator function that forwards that value to the scenario creator via the scenario_creator_kwargs dictionary. This is the recommended pattern for adding problem-specific command-line arguments when driving a model through generic_cylinders.py. (For comparison, the older hand-wired equivalent is preserved in examples.farmer.archive.farmer_cylinders.py.)

  • In the hydro_cylinders.py example (which has three stages). The branching factors are obtained from the command line and passed to the scenario constructor via scenario_creator_kwargs and also passed to sputils.create_nodenames_from_BFs to create a node list.

  • The uc_cylinders.py example adds arguments that are used to provide data or trigger the inclusion of extensions. The extension specifications and arguments are added to the dictionaries (e.g., hub_dict) created by vanilla.py.

Not Using Examples Utilities

Some users who want to use mpi-sppy with hubs and spokes embedded in other examples and other users may want a deeper understanding because they want to extend the shared services in examples. For these users, we provide a few examples that work directly with lower- and intermediate-level APIs.

No Spokes

There is a sizes example that does not use the hub and spoke system, it just calls PH (or the EF) without any spokes. The code can be found in examples.sizes.sizes_demo.py and it also demonstrates the use of extensions (see Extensions).

Hub and Spokes

An example of a hub with spokes that does not use the shared services provided in the examples directory is acopf3.ccopf2wood.py. This shows an example of creating a dictionaries (e.g. hub_dict) without using vanilla.py. For more examples of dictionaries for hub and spoke creation, vanilla.py itself is a good place to look.

A Note about Options

As a matter of common practice, there are no checks for unrecognized keys in options dictionaries in mpi-sppy. This has the advantage that new spokes and extensions can simply look for any options that they like. The disadvantage is that developers who use mpi-sppy cannot count on it to detect spelling errors in options names.

In contrast, Config objects will check spelling if you assign to an existing option, but functions like quick_assign are more popular and this function does will create a new option if the option does not exist.

Contrasting _mpisppy_node_list and all_node_names

Note that _mpisppy_node_list that is attached to scenarios does not have the leaf nodes, but all_node_names that is part of the hub dictionary does.