Proper Bundles
==============

Prior to 2024, bundles were constructed for the purpose of solves, but
all other processing (e.g., computing W values) was done on individual
scenarios. We will refer to these as `loose bundles`. This bundling scheme
is very flexible with respect to the numbers of scenarios in each bundle.
There are various if-blocks in the mpisppy code to support this type of bundle.

.. Warning::
   In relase 1.0, loose bundles scheduled to be deprecated.

In 2024, `proper bundles` were supported. After the extensive form
for a proper bundle is created, the original scenarios are more or less
forgotten and all processing takes place for the bundle. At the time
of this writing, these bundles are a little less flexible in that
the number of scenarios per bundle must divide the number of scenarios
and randomizing the assignment of scenarios to bundles is left to the
user (e.g., by using a pseudo-random vector to provide one level
of indirection for the scenario number in the ``scenario_creator`` function).
As of the time of this writing, only two-stage problems are easily supported.
Proper bundles result in faster execution than loose bundles.

See ``mpisppy.generic_cylinders.py`` for an example of their use in
code and see ``examples.generic_cylinders.bash`` for a few proper
bundle command lines.  In addition to being created on the fly and
used, they can be written (but not used in the same run) with
``--pickle-bundles-dir`` (note the the directory specified will be
overwritten), and read before use with ``--unpickle-bundles-dir``.  In
all uses of bundles in ``mpisppy.generic_cylinders.py`` the
``--scenarios-per-bundle`` option must be specified (even when
reading).

.. Note::
   When writing bundles in ``mpisppy.generic_cylinders.py``, all
   ranks are used for forming and writing bundles. Command line
   options related to anything other than proper bundles are ignored.

.. Note::
   Reading and writing bundle pickle files only works with proper bundles, not
   loose bundles.

.. Note::
   If you do pseudo random number generation on-the-fly during scenario creation,
   very careful management of random seeds is required if you want to
   get the same scenarios with proper  bundles that you get without them.

.. Note::
   Unpickled scenarios in proper bundles are not supported in generic_cyliners.
   (The wrappers would need to be more sophisticated.)

.. Note::
   The `scenario_denouement` function might not be called when pickling bundles.

.. Warning::
   Helper functions are *not* pickled, so there is a loose linkage with the
   helper functions in the module.


Modules
-------

In addition to command line options specified in ``mpisppy.utils.config.py``
and supported in ``mpisppy.generic_cylinders.py'',
there are two modules that have most of the support for proper bundles:

  - ``mpisppy.utils.pickle_bundle.py`` has miscellaneous utilities related to picking and other data processing
  - ``mpisppy.utils.proper_bundler.py`` has wrappers for cylinder programs


Multistage
----------

The most flexible way to create proper bundles is to write
your own problem-specific code to do it. The
file ``aircond_cylinders.py`` in the aircond example directory
provides an example.  The latter part of the ``allways.bash`` script
demonstrates how to run it.

There is support for multi-stage bundles in mpi-sppy, but the bundles must span the same number
of entire second stage nodes.

Notes
-----

Pickled bundles are clearly useful for algorithm tuning and algorithm
experimentation. In some, but not all, settings they can also improve
wall-clock performance for a single optimization run. The pickler
(e.g., ``bundle_pickler.py`` in the aircond example) does not use a
solver and can be run once to provide bundles to all cylinders. It can
often be assigned as many ranks as the total number of CPUs
available. Reading the bundles from a pickle file is much faster
than creating them.

The trick is that the bundles must contain entire second stage nodes
so the resulting bundles represent a two-stage problem.