APIs

High-level

For most developers, understanding the API for the high level classes is not necessary except perhaps when they want to bypass the examples shared services.

EF class

This class forms the extensive form and makes it available to send to a solver. It also provides some methods to look at the solution.

class mpisppy.opt.ef.ExtensiveForm(options, all_scenario_names, scenario_creator, scenario_creator_kwargs=None, all_nodenames=None, model_name=None, suppress_warnings=False)

Bases: SPBase

Create and solve an extensive form.

ef

Pyomo model of the extensive form.

Type:

pyomo.environ.ConcreteModel

solver

Solver produced by the Pyomo solver factory.

Parameters:

• options (dict) – Dictionary of options. May include a solver key to specify which solver name to use on the EF.

• all_scenario_names (list) – List of the names of each scenario in the EF (strings).

• scenario_creator (callable) – Scenario creator function, which takes as input a scenario name, and returns a Pyomo model of that scenario.

• scenario_creator_kwargs (dict, optional) – Keyword arguments passed to scenario_creator.

• all_nodenames (list, optional) – List of all node names, incl. leaves. Can be None for two-stage problem.

• model_name (str, optional) – Name of the resulting EF model object.

• suppress_warnings (bool, optional) – Boolean to suppress warnings when building the EF. Default is False.

Note: allowing use of the “solver” option key is for backward compatibility

get_objective_value()

Retrieve the objective value.

Returns:

Objective value.

Return type:

float

Raises:

ValueError – If optimal objective value could not be retrieved.

get_root_solution()

Get the value of the variables at the root node.

Returns:

Dictionary mapping variable name (str) to variable value (float) for all variables at the root node.

Return type:

dict

nonants()

An iterator to give representative Vars subject to non-anticipitivity Args: None

Yields:

tree node name, full EF Var name, Var value

nonants_to_csv(filename)

Dump the nonant vars from an ef to a csv file; truly a dump… :param filename: the full name of the csv output file :type filename: str

scenarios()

An iterator to give the scenario sub-models in an ef Args: None

Yields:

scenario name, scenario instance (str, ConcreteModel)

solve_extensive_form(solver_options=None, tee=False)

Solve the extensive form.

Parameters:

• solver_options (dict, optional) – Dictionary of solver-specific options (e.g. Gurobi options, CPLEX options, etc.).

• tee (bool, optional) – If True, displays solver output. Default False.

Returns:

Result returned by the Pyomo solve method.

Return type:

pyomo.opt.results.results_.SolverResults

PH class

This is the class for PH hub.

class mpisppy.opt.ph.PH(options, all_scenario_names, scenario_creator, scenario_denouement=None, all_nodenames=None, mpicomm=None, scenario_creator_kwargs=None, extensions=None, extension_kwargs=None, ph_converger=None, rho_setter=None, variable_probability=None)

Bases: PHBase

PH. See PHBase for list of args.

ph_main(finalize=True, attach_prox=True)

Execute the PH algorithm.

Parameters:

• finalize (bool, optional, default=True) – If True, call PH.post_loops(), if False, do not, and return None for Eobj

• attach_prox (bool, optional, default=True) – If True, use a proximal term, if False, no proximal term is added to the objective function

Returns:

Tuple containing

conv (float):

The convergence value (not easily interpretable).

Eobj (float or None):

If finalize=True, this is the expected, weighted objective value with the proximal term included. This value is not directly useful. If finalize=False, this value is None.

trivial_bound (float):

The “trivial bound”, computed by solving the model with no nonanticipativity constraints (immediately after iter 0).

Return type:

tuple

Note

You need an xhat finder either in denoument or in an extension.

mpisppy.opt.ph.profile(filename=None, comm=<mpisppy.MPI._MockMPIComm object>)

lshaped class

This is the class for L-shaped hub.

class mpisppy.opt.lshaped.LShapedMethod(options, all_scenario_names, scenario_creator, scenario_denouement=None, all_nodenames=None, mpicomm=None, scenario_creator_kwargs=None)

Bases: SPBase

Base class for the L-shaped method for two-stage stochastic programs.

Warning

This class explicitly assumes minimization.

Parameters:

• options (dict) –

  Dictionary of options. Possible (optional) options include

  • root_scenarios (list) - List of scenario names to include as part of the root problem (default [])

  • store_subproblems (boolean) - If True, the BendersDecomp object will maintain a dictionary containing the subproblems created by the BendersCutGenerator.

  • relax_root (boolean) - If True, the LP relaxation of the root problem is solved (i.e. integer variables in the root problem are relaxed).

  • scenario_creator_kwargs (dict) - Keyword args to pass to the scenario_creator.

  • valid_eta_lb (dict) - Dictionary mapping scenario names to valid lower bounds for the eta variables–i.e., a valid lower (outer) bound on the optimal objective value for each scenario. If none are provided, the lower bound is set to -sys.maxsize * scenario_prob, which may cause numerical errors.

  • indx_to_stage (dict) - Dictionary mapping the index of every variable in the model to the stage they belong to.

• all_scenario_names (list) – List of all scenarios names present in the model (strings).

• scenario_creator (callable) – Function which take a scenario name (string) and returns a Pyomo Concrete model with some things attached.

• scenario_denouement (callable, optional) – Function which does post-processing and reporting.

• all_nodenames (list, optional) – List of all node name (strings). Can be None for two-stage problems.

• mpicomm (MPI comm, optional) – MPI communicator to use between all scenarios. Default is MPI.COMM_WORLD.

• scenario_creator_kwargs (dict, optional) – Keyword arguments to pass to scenario_creator.

attach_nonant_var_map(scenario_name)

create_root()

creates a ConcreteModel from one of the problem scenarios then modifies the model to serve as the root problem

create_subproblem(scenario_name)

the subproblem creation function passed into the BendersCutsGenerator

lshaped_algorithm(converger=None)

function that runs the lshaped.py algorithm

options_check()

Check to ensure that the user-specified options are valid. Requried options are:

• root_solver (string) - Solver to use for the root problem.

• sp_solver (string) - Solver to use for the subproblems.

set_eta_bounds()

mpisppy.opt.lshaped.main()

Mid-level

Most developers will not want to interact directly with mid-level classes; however, they are important to code contributors and to developers who want to create their own extensions.

phbase.py

This is the base class for PH and PH-like algorithms.

class mpisppy.phbase.PHBase(options, all_scenario_names, scenario_creator, scenario_denouement=None, all_nodenames=None, mpicomm=None, scenario_creator_kwargs=None, extensions=None, extension_kwargs=None, ph_converger=None, rho_setter=None, variable_probability=None)

Bases: SPOpt

Base class for all PH-based algorithms.

Based on mpi4py (but should run with, or without, mpi) EVERY INDEX IS ZERO-BASED! (Except stages, which are one based).

Node names other than ROOT, although strings, must be a number or end in a number because mpi4py comms need a number. PH using a smart referencemodel that knows how to make its own tree nodes and just wants a trailing number in the scenario name. Assume we have only non-leaf nodes.

To check for rank 0 use self.cylinder_rank == 0.

local_scenarios

Dictionary mapping scenario names (strings) to scenarios (Pyomo conrete model objects). These are only the scenarios managed by the current rank (not all scenarios in the entire model).

Type:

dict

comms

Dictionary mapping node names (strings) to MPI communicator objects.

Type:

dict

local_scenario_names

List of local scenario names (strings). Should match the keys of the local_scenarios dict.

Type:

list

current_solver_options

from options, but callbacks might Dictionary of solver options provided in options. Note that callbacks could change these options.

Type:

dict

Parameters:

• options (dict) – Options for the PH algorithm.

• all_scenario_names (list) – List of all scenario names in the model (strings).

• scenario_creator (callable) – Function which take a scenario name (string) and returns a Pyomo Concrete model with some things attached.

• scenario_denouement (callable, optional) – Function which does post-processing and reporting.

• all_nodenames (list, optional) – List of all node name (strings). Can be None for two-stage problems.

• mpicomm (MPI comm, optional) – MPI communicator to use between all scenarios. Default is MPI.COMM_WORLD.

• scenario_creator_kwargs (dict, optional) – Keyword arguments passed to scenario_creator.

• extensions (object, optional) – PH extension object.

• extension_kwargs (dict, optional) – Keyword arguments to pass to the extensions.

• ph_converger (object, optional) – PH converger object.

• rho_setter (callable, optional) – Function to set rho values throughout the PH algorithm.

• variable_probability (callable, optional) – Function to set variable specific probabilities.

Compute_Xbar(verbose=False)

Gather xbar and x squared bar for each node in the list and distribute the values back to the scenarios.

Parameters:

verbose (boolean) – If True, prints verbose output.

Iter0()

Create solvers and perform the initial PH solve (with no dual weights or prox terms).

This function quits() if the scenario probabilities do not sum to one, or if any of the scenario subproblems are infeasible. It also calls the post_iter0 method of any extensions, and uses the rho setter (if present) after the inital solve.

Returns:

The so-called “trivial bound”, i.e., the objective value of the stochastic program with the nonanticipativity constraints removed.

Return type:

float

PH_Prep(attach_duals=True, attach_prox=True, attach_smooth=0)

Set up PH objectives (duals and prox terms), and prepare extensions, if available.

Parameters:

• add_duals (boolean, optional) – If True, adds dual weight (Ws) to the objective. Default True.

• add_prox (boolean, optional) – If True, adds prox terms to the objective. Default True.

• attach_smooth (int, optional) – If 0, no smoothing; if 1, p_value is used; if 2, p_ratio is used.

Note

This function constructs an Extension object if one was specified at the time the PH object was created.

Update_W(verbose)

Update the dual weights during the PH algorithm.

Parameters:

verbose (bool) – If True, displays verbose output during update.

Update_z(verbose)

Update the smoothing variable z during the PH algorithm.

Parameters:

verbose (bool) – If True, displays verbose output during update.

property W_disabled

W_from_flat_list(flat_list)

Set the dual weight values (Ws) for all local scenarios from a flat list.

Parameters:

flat_list (list) – One-dimensional list of dual weights.

Warning

We are counting on Pyomo indices not to change order between list creation and use.

attach_PH_to_objective(add_duals, add_prox, add_smooth=0)

Attach dual weight and prox terms to the objective function of the models in local_scenarios.

Parameters:

• add_duals (boolean) – If True, adds dual weight (Ws) to the objective.

• add_prox (boolean) – If True, adds the prox term to the objective.

attach_Ws_and_prox()

Attach the dual and prox terms to the models in local_scenarios.

attach_smoothing()

Attach the smoothing terms to the models in local_scenarios.

attach_xbars()

Attach xbar and xbar^2 Pyomo parameters to each model in local_scenarios.

convergence_diff()

Compute the convergence metric ||x_s - bar{x}||_1 / num_scenarios.

Returns:

The convergence metric ||x_s - bar{x}||_1 / num_scenarios.

Return type:

float

disable_W_and_prox()

iterk_loop()

Perform all PH iterations after iteration 0.

This function terminates if any of the following occur:

1. The maximum number of iterations is reached.

2. The user specifies a converger, and the is_converged() method of that converger returns True.

3. The hub tells it to terminate.

4. The user does not specify a converger, and the default convergence criteria are met (i.e. the convergence value falls below the user-specified threshold).

Args: None

options_check()

Check whether the options in the options attribute are acceptable.

Required options are

• solver_name (string): The name of the solver to use.

• PHIterLimit (int): The maximum number of PH iterations to execute.

• defaultPHrho (float): The default value of rho (penalty parameter) to use for PH.

• convthresh (float): The convergence tolerance of the PH algorithm.

• verbose (boolean): Flag indicating whether to display verbose output.

• display_progress (boolean): Flag indicating whether to display information about the progression of the algorithm.

• iter0_solver_options (dict): Dictionary of solver options to use on the first solve loop.

• iterk_solver_options (dict): Dictionary of solver options to use on subsequent solve loops (after iteration 0).

post_loops(extensions=None)

Call scenario denouement methods, and report the expected objective value.

Parameters:

extensions (object, optional) – PH extension object.

Returns:

Pretty useless weighted, proxed objective value.

Return type:

float

post_solve_bound(solver_options=None, verbose=False)

Compute a bound Lagrangian bound using the existing weights.

Parameters:

• solver_options (dict, optional) – Options for these solves.

• verbose (boolean, optional) – If True, displays verbose output. Default False.

Returns:

An outer bound on the optimal objective function value.

Return type:

float

Note

This function overwrites current variable values. This is only suitable for use at the end of the solves, or if you really know what you are doing. It is not suitable as a general, per-iteration Lagrangian bound solver.

property prox_disabled

reenable_W_and_prox()

solve_loop(solver_options=None, use_scenarios_not_subproblems=False, dtiming=False, dis_W=False, dis_prox=False, gripe=False, disable_pyomo_signal_handling=False, tee=False, verbose=False, need_solution=True, warmstart=WarmstartStatus.FALSE)

Loop over local_subproblems and solve them in a manner dicated by the arguments.

In addition to changing the Var values in the scenarios, this function also updates the _PySP_feas_indictor to indicate which scenarios were feasible/infeasible.

Parameters:

• solver_options (dict, optional) – The scenario solver options.

• use_scenarios_not_subproblems (boolean, optional) – If True, solves individual scenario problems, not subproblems. This distinction matters when using bundling. Default is False.

• dtiming (boolean, optional) – If True, reports solve timing information. Default is False.

• dis_W (boolean, optional) – If True, duals weights (Ws) are disabled before solve, then re-enabled after solve. Default is False.

• dis_prox (boolean, optional) – If True, prox terms are disabled before solve, then re-enabled after solve. Default is False.

• gripe (boolean, optional) – If True, output a message when a solve fails. Default is False.

• disable_pyomo_signal_handling (boolean, optional) – True for asynchronous PH; ignored for persistent solvers. Default False.

• tee (boolean, optional) – If True, displays solver output. Default False.

• verbose (boolean, optional) – If True, displays verbose output. Default False.

• need_solution (boolean, optional) – If True, raises an exception if a solution is not available. Default True

• warmstart (bool, optional) – If True, warmstart the subproblem solves. Default False.

mpisppy.phbase.profile(filename=None, comm=<mpisppy.MPI._MockMPIComm object>)

spbase.py

SPBase is the base class for many algorithms.

class mpisppy.spbase.SPBase(options, all_scenario_names, scenario_creator, scenario_denouement=None, all_nodenames=None, mpicomm=None, scenario_creator_kwargs=None, variable_probability=None, E1_tolerance=1e-05)

Bases: object

Defines an interface to all strata (hubs and spokes)

Parameters:

• options (dict) – options

• all_scenario_names (list) – all scenario names

• scenario_creator (fct) – returns a concrete model with special things

• scenario_denouement (fct) – for post processing and reporting

• all_nodenames (list) – all node names (including leaves); can be None for 2 Stage

• mpicomm (MPI comm) – if not given, use the global fullcomm

• scenario_creator_kwargs (dict) – kwargs passed directly to scenario_creator.

• variable_probability (fct) – returns a list of tuples of (id(var), prob) to set variable-specific probability (similar to PHBase.rho_setter).

local_scenarios

concrete models with extra data, key is name

Type:

dict of scenario objects

comms

keys are node names values are comm objects.

Type:

dict

local_scenario_names

names of locals

Type:

list

allreduce_or(val)

gather_var_values_to_rank0(get_zero_prob_values=False, fixed_vars=True)

Gather the values of the nonanticipative variables to the root of the mpicomm for the cylinder

Returns:

On the root (rank0), returns a dictionary mapping (scenario_name, variable_name) pairs to their values. On other ranks, returns None.

Return type:

dict or None

is_zero_prob(scenario_model, var)

Parameters:

• scenario_model – a value in SPBase.local_scenarios

• var – a Pyomo Var on the scenario_model

Returns:

True if the variable has 0 probability, False otherwise

load_best_solution()

report_var_values_at_rank0(header='', print_zero_prob_values=False, fixed_vars=True)

Pretty-print the values and associated statistics for non-anticipative variables across all scenarios.

property spcomm

update_best_solution_if_improving(obj_val)

Call if the variable values have a nonanticipative solution with associated obj_val. Will update the best_solution_cache if the solution is better than the existing cached solution

write_first_stage_solution(file_name, first_stage_solution_writer=<function first_stage_nonant_writer>)

Writes the first-stage solution, if this object reports one available.

Parameters:

• file_name – path of file to write first stage solution to

• first_stage_solution_writer (optional) – custom first stage solution writer function

write_tree_solution(directory_name, scenario_tree_solution_writer=<function scenario_tree_solution_writer>)

Writes the tree solution, if this object reports one available. Raises a RuntimeError if it is not.

Parameters:

• directory_name – directory to write tree solution to

• scenario_tree_solution_writer (optional) – custom scenario solution writer function

scenario_tree.py

This provides services for the scenario tree.

class mpisppy.scenario_tree.ScenarioNode(name, cond_prob, stage, cost_expression, nonant_list, scen_model, nonant_ef_suppl_list=None, surrogate_nonant_list=None, parent_name=None)

Bases: object

Store a node in the scenario tree.

Note

This can only be created programatically from a scenario creation function. (maybe that function reads data)

Parameters:

• name (str) – name of the node; one node must be named “ROOT”

• cond_prob (float) – conditional probability

• stage (int) – stage number (root is 1)

• cost_expression (pyo Expression or Var) – stage cost

• nonant_list (list of pyo Var, VarData or slices) – the Vars that require nonanticipativity at the node (might not be a list)

• scen_model (pyo concrete model) – the (probably not ‘a’) concrete model

• nonant_ef_suppl_list (list of pyo Var, Vardata or slices) –

  Vars for which nonanticipativity constraints will only be added to the extensive form (important for bundling), but for which mpi-sppy will not enforce them as nonanticipative elsewhere. NOTE: These types of variables are often indicator variables

  that are already present in the deterministic model.

• surrogate_nonant_list (list of pyo Var, VarData or slices) –

  Vars for which nonanticipativity constraints are enforced implicitly by the vars in varlist, but which may speed PH convergence and/or aid in cut generation when considered explicitly. These vars will be ignored for fixers, incumbent finders which fix nonants to calculate solutions, and the EF creator. NOTE: These types of variables are typically artificially added

  to the model to capture hierarchical model features.

• parent_name (str) – name of the parent node

Lists:

nonant_vardata(list of vardata objects): vardatas to blend x_bar_list(list of floats): bound by index to nonant_vardata

mpisppy.scenario_tree.build_vardatalist(self, model, varlist=None)

Convert a list of pyomo variables to a list of SimpleVar and _GeneralVarData. If varlist is none, builds a list of all variables in the model. Written by CD Laird

Parameters:

• model (ConcreteModel)

• varlist (None or list of pyo.Var)

Low Level

Most developers will not need to understand the low-level classes.

spcommunicator

This class handles communication between ranks.

spoke

hub