APIs
High-level
For most developers, understanding the API for the high level classes is not necessary except perhaps when they want to bypass the examples shared services.
EF class
This class forms the extensive form and makes it available to send to a solver. It also provides some methods to look at the solution.
- class mpisppy.opt.ef.ExtensiveForm(options, all_scenario_names, scenario_creator, scenario_creator_kwargs=None, all_nodenames=None, model_name=None, suppress_warnings=False)[source]
Bases:
SPBase
Create and solve an extensive form.
- ef
Pyomo model of the extensive form.
- Type:
pyomo.environ.ConcreteModel
- solver
Solver produced by the Pyomo solver factory.
- Parameters:
options (dict) – Dictionary of options. May include a solver key to specify which solver name to use on the EF.
all_scenario_names (list) – List of the names of each scenario in the EF (strings).
scenario_creator (callable) – Scenario creator function, which takes as input a scenario name, and returns a Pyomo model of that scenario.
scenario_creator_kwargs (dict, optional) – Keyword arguments passed to scenario_creator.
all_nodenames (list, optional) – List of all node names, incl. leaves. Can be None for two-stage problem.
model_name (str, optional) – Name of the resulting EF model object.
suppress_warnings (bool, optional) – Boolean to suppress warnings when building the EF. Default is False.
Note: allowing use of the “solver” option key is for backward compatibility
- get_objective_value()[source]
Retrieve the objective value.
- Returns:
Objective value.
- Return type:
float
- Raises:
ValueError – If optimal objective value could not be retrieved.
- get_root_solution()[source]
Get the value of the variables at the root node.
- Returns:
Dictionary mapping variable name (str) to variable value (float) for all variables at the root node.
- Return type:
dict
- nonants()[source]
An iterator to give representative Vars subject to non-anticipitivity Args: None
- Yields:
tree node name, full EF Var name, Var value
- nonants_to_csv(filename)[source]
Dump the nonant vars from an ef to a csv file; truly a dump… :param filename: the full name of the csv output file :type filename: str
- scenarios()[source]
An iterator to give the scenario sub-models in an ef Args: None
- Yields:
scenario name, scenario instance (str, ConcreteModel)
- solve_extensive_form(solver_options=None, tee=False)[source]
Solve the extensive form.
- Parameters:
solver_options (dict, optional) – Dictionary of solver-specific options (e.g. Gurobi options, CPLEX options, etc.).
tee (bool, optional) – If True, displays solver output. Default False.
- Returns:
Result returned by the Pyomo solve method.
- Return type:
pyomo.opt.results.results_.SolverResults
PH class
This is the class for PH hub.
- class mpisppy.opt.ph.PH(options, all_scenario_names, scenario_creator, scenario_denouement=None, all_nodenames=None, mpicomm=None, scenario_creator_kwargs=None, extensions=None, extension_kwargs=None, ph_converger=None, rho_setter=None, variable_probability=None)[source]
Bases:
PHBase
PH. See PHBase for list of args.
- ph_main(finalize=True)[source]
Execute the PH algorithm.
- Parameters:
finalize (bool, optional, default=True) – If True, call PH.post_loops(), if False, do not, and return None for Eobj
- Returns:
Tuple containing
- conv (float):
The convergence value (not easily interpretable).
- Eobj (float or None):
If finalize=True, this is the expected, weighted objective value with the proximal term included. This value is not directly useful. If finalize=False, this value is None.
- trivial_bound (float):
The “trivial bound”, computed by solving the model with no nonanticipativity constraints (immediately after iter 0).
- Return type:
tuple
Note
You need an xhat finder either in denoument or in an extension.
lshaped class
This is the class for L-shaped hub.
- class mpisppy.opt.lshaped.LShapedMethod(options, all_scenario_names, scenario_creator, scenario_denouement=None, all_nodenames=None, mpicomm=None, scenario_creator_kwargs=None)[source]
Bases:
SPBase
Base class for the L-shaped method for two-stage stochastic programs.
Warning
This class explicitly assumes minimization.
- Parameters:
options (dict) –
Dictionary of options. Possible (optional) options include
root_scenarios (list) - List of scenario names to include as part of the root problem (default [])
store_subproblems (boolean) - If True, the BendersDecomp object will maintain a dictionary containing the subproblems created by the BendersCutGenerator.
relax_root (boolean) - If True, the LP relaxation of the root problem is solved (i.e. integer variables in the root problem are relaxed).
scenario_creator_kwargs (dict) - Keyword args to pass to the scenario_creator.
valid_eta_lb (dict) - Dictionary mapping scenario names to valid lower bounds for the eta variables–i.e., a valid lower (outer) bound on the optimal objective value for each scenario. If none are provided, the lower bound is set to -sys.maxsize * scenario_prob, which may cause numerical errors.
indx_to_stage (dict) - Dictionary mapping the index of every variable in the model to the stage they belong to.
all_scenario_names (list) – List of all scenarios names present in the model (strings).
scenario_creator (callable) – Function which take a scenario name (string) and returns a Pyomo Concrete model with some things attached.
scenario_denouement (callable, optional) – Function which does post-processing and reporting.
all_nodenames (list, optional) – List of all node name (strings). Can be None for two-stage problems.
mpicomm (MPI comm, optional) – MPI communicator to use between all scenarios. Default is MPI.COMM_WORLD.
scenario_creator_kwargs (dict, optional) – Keyword arguments to pass to scenario_creator.
- create_root()[source]
creates a ConcreteModel from one of the problem scenarios then modifies the model to serve as the root problem
- create_subproblem(scenario_name)[source]
the subproblem creation function passed into the BendersCutsGenerator
Mid-level
Most developers will not want to interact directly with mid-level classes; however, they are important to code contributors and to developers who want to create their own extensions.
phbase.py
This is the base class for PH and PH-like algorithms.
- class mpisppy.phbase.PHBase(options, all_scenario_names, scenario_creator, scenario_denouement=None, all_nodenames=None, mpicomm=None, scenario_creator_kwargs=None, extensions=None, extension_kwargs=None, ph_converger=None, rho_setter=None, variable_probability=None)[source]
Bases:
SPOpt
Base class for all PH-based algorithms.
Based on mpi4py (but should run with, or without, mpi) EVERY INDEX IS ZERO-BASED! (Except stages, which are one based).
Node names other than ROOT, although strings, must be a number or end in a number because mpi4py comms need a number. PH using a smart referencemodel that knows how to make its own tree nodes and just wants a trailing number in the scenario name. Assume we have only non-leaf nodes.
To check for rank 0 use self.cylinder_rank == 0.
- local_scenarios
Dictionary mapping scenario names (strings) to scenarios (Pyomo conrete model objects). These are only the scenarios managed by the current rank (not all scenarios in the entire model).
- Type:
dict
- comms
Dictionary mapping node names (strings) to MPI communicator objects.
- Type:
dict
- local_scenario_names
List of local scenario names (strings). Should match the keys of the local_scenarios dict.
- Type:
list
- current_solver_options
from options, but callbacks might Dictionary of solver options provided in options. Note that callbacks could change these options.
- Type:
dict
- Parameters:
options (dict) – Options for the PH algorithm.
all_scenario_names (list) – List of all scenario names in the model (strings).
scenario_creator (callable) – Function which take a scenario name (string) and returns a Pyomo Concrete model with some things attached.
scenario_denouement (callable, optional) – Function which does post-processing and reporting.
all_nodenames (list, optional) – List of all node name (strings). Can be None for two-stage problems.
mpicomm (MPI comm, optional) – MPI communicator to use between all scenarios. Default is MPI.COMM_WORLD.
scenario_creator_kwargs (dict, optional) – Keyword arguments passed to scenario_creator.
extensions (object, optional) – PH extension object.
extension_kwargs (dict, optional) – Keyword arguments to pass to the extensions.
ph_converger (object, optional) – PH converger object.
rho_setter (callable, optional) – Function to set rho values throughout the PH algorithm.
variable_probability (callable, optional) – Function to set variable specific probabilities.
cfg (config object, optional?) controls (mainly from user) – (Maybe this should move up to spbase)
- Compute_Xbar(verbose=False)[source]
Gather xbar and x squared bar for each node in the list and distribute the values back to the scenarios.
- Parameters:
verbose (boolean) – If True, prints verbose output.
- Iter0()[source]
Create solvers and perform the initial PH solve (with no dual weights or prox terms).
This function quits() if the scenario probabilities do not sum to one, or if any of the scenario subproblems are infeasible. It also calls the post_iter0 method of any extensions, and uses the rho setter (if present) after the inital solve.
- Returns:
The so-called “trivial bound”, i.e., the objective value of the stochastic program with the nonanticipativity constraints removed.
- Return type:
float
- PH_Prep(attach_duals=True, attach_prox=True, attach_smooth=0)[source]
Set up PH objectives (duals and prox terms), and prepare extensions, if available.
- Parameters:
add_duals (boolean, optional) – If True, adds dual weight (Ws) to the objective. Default True.
add_prox (boolean, optional) – If True, adds prox terms to the objective. Default True.
attach_smooth (int, optional) – If 0, no smoothing; if 1, p_value is used; if 2, p_ratio is used.
Note
This function constructs an Extension object if one was specified at the time the PH object was created.
- Update_W(verbose)[source]
Update the dual weights during the PH algorithm.
- Parameters:
verbose (bool) – If True, displays verbose output during update.
- Update_z(verbose)[source]
Update the smoothing variable z during the PH algorithm.
- Parameters:
verbose (bool) – If True, displays verbose output during update.
- property W_disabled
- W_from_flat_list(flat_list)[source]
Set the dual weight values (Ws) for all local scenarios from a flat list.
- Parameters:
flat_list (list) – One-dimensional list of dual weights.
Warning
We are counting on Pyomo indices not to change order between list creation and use.
- attach_PH_to_objective(add_duals, add_prox, add_smooth=0)[source]
Attach dual weight and prox terms to the objective function of the models in local_scenarios.
- Parameters:
add_duals (boolean) – If True, adds dual weight (Ws) to the objective.
add_prox (boolean) – If True, adds the prox term to the objective.
- convergence_diff()[source]
Compute the convergence metric ||x_s - bar{x}||_1 / num_scenarios.
- Returns:
The convergence metric ||x_s - bar{x}||_1 / num_scenarios.
- Return type:
float
- iterk_loop()[source]
Perform all PH iterations after iteration 0.
This function terminates if any of the following occur:
The maximum number of iterations is reached.
The user specifies a converger, and the is_converged() method of that converger returns True.
The hub tells it to terminate.
The user does not specify a converger, and the default convergence criteria are met (i.e. the convergence value falls below the user-specified threshold).
Args: None
- options_check()[source]
Check whether the options in the options attribute are acceptable.
Required options are
solver_name (string): The name of the solver to use.
PHIterLimit (int): The maximum number of PH iterations to execute.
defaultPHrho (float): The default value of rho (penalty parameter) to use for PH.
convthresh (float): The convergence tolerance of the PH algorithm.
verbose (boolean): Flag indicating whether to display verbose output.
display_progress (boolean): Flag indicating whether to display information about the progression of the algorithm.
iter0_solver_options (dict): Dictionary of solver options to use on the first solve loop.
iterk_solver_options (dict): Dictionary of solver options to use on subsequent solve loops (after iteration 0).
- post_loops(extensions=None)[source]
Call scenario denouement methods, and report the expected objective value.
- Parameters:
extensions (object, optional) – PH extension object.
- Returns:
Pretty useless weighted, proxed objective value.
- Return type:
float
- post_solve_bound(solver_options=None, verbose=False)[source]
Compute a bound Lagrangian bound using the existing weights.
- Parameters:
solver_options (dict, optional) – Options for these solves.
verbose (boolean, optional) – If True, displays verbose output. Default False.
- Returns:
An outer bound on the optimal objective function value.
- Return type:
float
Note
This function overwrites current variable values. This is only suitable for use at the end of the solves, or if you really know what you are doing. It is not suitable as a general, per-iteration Lagrangian bound solver.
- property prox_disabled
- solve_loop(solver_options=None, use_scenarios_not_subproblems=False, dtiming=False, dis_W=False, dis_prox=False, gripe=False, disable_pyomo_signal_handling=False, tee=False, verbose=False, need_solution=True)[source]
Loop over local_subproblems and solve them in a manner dicated by the arguments.
In addition to changing the Var values in the scenarios, this function also updates the _PySP_feas_indictor to indicate which scenarios were feasible/infeasible.
- Parameters:
solver_options (dict, optional) – The scenario solver options.
use_scenarios_not_subproblems (boolean, optional) – If True, solves individual scenario problems, not subproblems. This distinction matters when using bundling. Default is False.
dtiming (boolean, optional) – If True, reports solve timing information. Default is False.
dis_W (boolean, optional) – If True, duals weights (Ws) are disabled before solve, then re-enabled after solve. Default is False.
dis_prox (boolean, optional) – If True, prox terms are disabled before solve, then re-enabled after solve. Default is False.
gripe (boolean, optional) – If True, output a message when a solve fails. Default is False.
disable_pyomo_signal_handling (boolean, optional) – True for asynchronous PH; ignored for persistent solvers. Default False.
tee (boolean, optional) – If True, displays solver output. Default False.
verbose (boolean, optional) – If True, displays verbose output. Default False.
need_solution (boolean, optional) – If True, raises an exception if a solution is not available. Default True
spbase.py
SPBase is the base class for many algorithms.
- class mpisppy.spbase.SPBase(options, all_scenario_names, scenario_creator, scenario_denouement=None, all_nodenames=None, mpicomm=None, scenario_creator_kwargs=None, variable_probability=None, E1_tolerance=1e-05)[source]
Bases:
object
Defines an interface to all strata (hubs and spokes)
- Parameters:
options (dict) – options
all_scenario_names (list) – all scenario names
scenario_creator (fct) – returns a concrete model with special things
scenario_denouement (fct) – for post processing and reporting
all_nodenames (list) – all node names (including leaves); can be None for 2 Stage
mpicomm (MPI comm) – if not given, use the global fullcomm
scenario_creator_kwargs (dict) – kwargs passed directly to scenario_creator.
variable_probability (fct) – returns a list of tuples of (id(var), prob) to set variable-specific probability (similar to PHBase.rho_setter).
- local_scenarios
concrete models with extra data, key is name
- Type:
dict of scenario objects
- comms
keys are node names values are comm objects.
- Type:
dict
- local_scenario_names
names of locals
- Type:
list
- gather_var_values_to_rank0(get_zero_prob_values=False, fixed_vars=True)[source]
Gather the values of the nonanticipative variables to the root of the mpicomm for the cylinder
- Returns:
On the root (rank0), returns a dictionary mapping (scenario_name, variable_name) pairs to their values. On other ranks, returns None.
- Return type:
dict or None
- is_zero_prob(scenario_model, var)[source]
- Parameters:
scenario_model – a value in SPBase.local_scenarios
var – a Pyomo Var on the scenario_model
- Returns:
True if the variable has 0 probability, False otherwise
- report_var_values_at_rank0(header='', print_zero_prob_values=False, fixed_vars=True)[source]
Pretty-print the values and associated statistics for non-anticipative variables across all scenarios.
- property spcomm
- write_first_stage_solution(file_name, first_stage_solution_writer=<function first_stage_nonant_writer>)[source]
Writes the first-stage solution, if this object reports one available.
- Parameters:
file_name – path of file to write first stage solution to
first_stage_solution_writer (optional) – custom first stage solution writer function
- write_tree_solution(directory_name, scenario_tree_solution_writer=<function scenario_tree_solution_writer>)[source]
Writes the tree solution, if this object reports one available. Raises a RuntimeError if it is not.
- Parameters:
directory_name – directory to write tree solution to
scenario_tree_solution_writer (optional) – custom scenario solution writer function
scenario_tree.py
This provides services for the scenario tree.
- class mpisppy.scenario_tree.ScenarioNode(name, cond_prob, stage, cost_expression, nonant_list, scen_model, nonant_ef_suppl_list=None, parent_name=None)[source]
Bases:
object
Store a node in the scenario tree.
Note
This can only be created programatically from a scenario creation function. (maybe that function reads data)
- Parameters:
name (str) – name of the node; one node must be named “ROOT”
cond_prob (float) – conditional probability
stage (int) – stage number (root is 1)
cost_expression (pyo Expression or Var) – stage cost
nonant_list (list of pyo Var, Vardata or slices) – the Vars that require nonanticipativity at the node (might not be a list)
scen_model (pyo concrete model) – the (probably not ‘a’) concrete model
nonant_ef_suppl_list (list of pyo Var, Vardata or slices) – vars for which nonanticipativity constraints tighten the EF (important for bundling)
parent_name (str) – name of the parent node
- Lists:
nonant_vardata(list of vardata objects): vardatas to blend x_bar_list(list of floats): bound by index to nonant_vardata
- mpisppy.scenario_tree.build_vardatalist(self, model, varlist=None)[source]
Convert a list of pyomo variables to a list of SimpleVar and _GeneralVarData. If varlist is none, builds a list of all variables in the model. Written by CD Laird
- Parameters:
model (ConcreteModel)
varlist (None or list of pyo.Var)
Low Level
Most developers will not need to understand the low-level classes.
spcommunicator
This class handles communication between ranks.
Conventional wisdom seems to be that we should use Put calls locally (i.e. a process should Put() into its own buffer), and Get calls for communication (i.e. call Get on a remote target, rather than your local buffer). The following implementation uses this paradigm.
The communication in this paradigm is a star graph, with the hub at the center and the spokes on the outside. Each spoke is concerned only with the hub, but the hub must track information about all of the spokes.
Separate hub and spoke classes for memory/window management?
- class mpisppy.cylinders.spcommunicator.SPCommunicator(spbase_object, fullcomm, strata_comm, cylinder_comm, options=None)[source]
Bases:
object
Notes: TODO
- finalize()[source]
Every hub/spoke may have a finalize function, which does some final calculations/flushing to disk after convergence
spoke
- class mpisppy.cylinders.spoke.ConvergerSpokeType(value)[source]
Bases:
Enum
An enumeration.
- INNER_BOUND = 2
- NONANT_GETTER = 4
- OUTER_BOUND = 1
- W_GETTER = 3
- class mpisppy.cylinders.spoke.InnerBoundNonantSpoke(spbase_object, fullcomm, strata_comm, cylinder_comm, options=None)[source]
Bases:
_BoundNonantSpoke
For Spokes that provide an inner (incumbent) bound through self.bound to the Hub, and receive the nonants from the main SPOpt hub.
Includes some helpful methods for saving and restoring results
- converger_spoke_char = 'I'
- converger_spoke_types = (ConvergerSpokeType.INNER_BOUND, ConvergerSpokeType.NONANT_GETTER)
- class mpisppy.cylinders.spoke.InnerBoundSpoke(spbase_object, fullcomm, strata_comm, cylinder_comm, options=None)[source]
Bases:
_BoundSpoke
For Spokes that provide an inner bound through self.bound to the Hub, and do not need information from the main PH OPT hub.
- converger_spoke_char = 'I'
- converger_spoke_types = (ConvergerSpokeType.INNER_BOUND,)
- class mpisppy.cylinders.spoke.OuterBoundNonantSpoke(spbase_object, fullcomm, strata_comm, cylinder_comm, options=None)[source]
Bases:
_BoundNonantSpoke
For Spokes that provide an outer bound through self.bound to the Hub, and receive the nonants from the main OPT hub.
- converger_spoke_char = 'A'
- converger_spoke_types = (ConvergerSpokeType.OUTER_BOUND, ConvergerSpokeType.NONANT_GETTER)
- class mpisppy.cylinders.spoke.OuterBoundSpoke(spbase_object, fullcomm, strata_comm, cylinder_comm, options=None)[source]
Bases:
_BoundSpoke
For Spokes that provide an outer bound through self.bound to the Hub, and do not need information from the main PH OPT hub.
- converger_spoke_char = 'O'
- converger_spoke_types = (ConvergerSpokeType.OUTER_BOUND,)
- class mpisppy.cylinders.spoke.OuterBoundWSpoke(spbase_object, fullcomm, strata_comm, cylinder_comm, options=None)[source]
Bases:
_BoundWSpoke
For Spokes that provide an outer bound through self.bound to the Hub, and receive the Ws (or weights) from the main PH OPT hub.
- converger_spoke_char = 'O'
- converger_spoke_types = (ConvergerSpokeType.OUTER_BOUND, ConvergerSpokeType.W_GETTER)
- class mpisppy.cylinders.spoke.Spoke(spbase_object, fullcomm, strata_comm, cylinder_comm, options=None)[source]
Bases:
SPCommunicator
- got_kill_signal()[source]
Spoke should call this method at least every iteration to see if the Hub terminated
hub
- class mpisppy.cylinders.hub.APHHub(spbase_object, fullcomm, strata_comm, cylinder_comm, spokes, options=None)[source]
Bases:
PHHub
- class mpisppy.cylinders.hub.Hub(spbase_object, fullcomm, strata_comm, cylinder_comm, spokes, options=None)[source]
Bases:
SPCommunicator
- compute_gaps()[source]
Compute the current absolute and relative gaps, using the current self.BestInnerBound and self.BestOuterBound
- abstract current_iteration()[source]
Returns the current iteration count - however the hub defines it.
- hub_finalize()[source]
Every hub may have another finalize function, which collects any results from finalize
- hub_from_spoke(values, spoke_num)[source]
spoke_num is the rank in the strata_comm, so it is 1-based not 0-based
- Returns:
- Indicates whether the “gotten” values are new,
based on the write_id.
- Return type:
is_new (bool)
- hub_to_spoke(values, spoke_strata_rank)[source]
Put the specified values into the specified locally-owned buffer for the spoke to pick up.
Notes
This automatically does the -1 indexing
This assumes that values contains a slot at the end for the write_id
- initialize_boundsout()[source]
Initialize the buffer for the hub to send bounds to bounds only spokes
- initialize_inner_bound_buffers()[source]
Initialize value of BestInnerBound, and inner bound receive buffers
- initialize_nonants()[source]
Initialize the buffer for the hub to send nonants to the appropriate spokes
- initialize_outer_bound_buffers()[source]
Initialize value of BestOuterBound, and outer bound receive buffers
- initialize_spoke_indices()[source]
Figure out what types of spokes we have, and sort them into the appropriate classes.
Note
Some spokes may be multiple types (e.g. outerbound and nonant), though not all combinations are supported.
- abstract is_converged()[source]
The hub has the ability to halt the optimization algorithm on the hub before any local convergers.
- receive_innerbounds()[source]
Get inner bounds from inner bound spokes NOTE: Does not check if there _are_ innerbound spokes (but should be harmless to call if there are none)
- receive_outerbounds()[source]
Get outer bounds from outer bound spokes NOTE: Does not check if there _are_ outerbound spokes (but should be harmless to call if there are none)
- send_boundsout()[source]
Send bounds to the appropriate spokes This is called only for spokes which are bounds only. w and nonant spokes are passed bounds through the w and nonant buffers
- class mpisppy.cylinders.hub.LShapedHub(spbase_object, fullcomm, strata_comm, cylinder_comm, spokes, options=None)[source]
Bases:
Hub
- is_converged()[source]
Returns a boolean. If True, then LShaped will terminate
- Side-effects:
The L-shaped method produces outer bounds during execution, so we will check it as well.
- send_nonants()[source]
Gather nonants and send them to the appropriate spokes TODO: Will likely fail with bundling
- class mpisppy.cylinders.hub.PHHub(spbase_object, fullcomm, strata_comm, cylinder_comm, spokes, options=None)[source]
Bases:
Hub
- initialize_ws()[source]
Initialize the buffer for the hub to send dual weights to the appropriate spokes
- is_converged()[source]
The hub has the ability to halt the optimization algorithm on the hub before any local convergers.
- send_nonants()[source]
Gather nonants and send them to the appropriate spokes TODO: Will likely fail with bundling