diff --git a/docs/Maestro/meps/index.md b/docs/Maestro/meps/index.md new file mode 100644 index 00000000..373d9cd9 --- /dev/null +++ b/docs/Maestro/meps/index.md @@ -0,0 +1,12 @@ +# Maestro enhancement proposals (MEPs) + +## Open + +[MEP 001](mep-001-encore-study-iteration.md) Encore: Chaining studies and iteration + +[MEP 002](mep-002-parameter-composition.md) Parameter composition + +[MEP 003](mep-003-human-readable-hashing.md) Human readable hashing + +[MEP 004](mep-004-step-dependency-execution-policy.md) Step Dependency Execution Policy + diff --git a/docs/Maestro/meps/mep-001-encore-study-iteration.md b/docs/Maestro/meps/mep-001-encore-study-iteration.md new file mode 100644 index 00000000..0afb59a6 --- /dev/null +++ b/docs/Maestro/meps/mep-001-encore-study-iteration.md @@ -0,0 +1,282 @@ + + +# MEP 001 - Encore: Study Chaning and Iteration + +## Abstract + +Optimization/iteration are common workflow patterns, and this Encore feature aims to +partially address that as a next layer on top of Maestro's existing behavior where +the unit of work being chained together is a Maestro study. The most basic usecase +here is rerunning the the same study process, but passing different parameter values through +it, i.e. refining parameters in a grid search to converge upon some optimum, or to +reduce uncertainty metrics by adding additional samples in the parameter space. Despite +aiming this at studies calling themselves, this also enables branching workflows, and +studies calling other comletely different ones. The proposed changes outlined here will +focus primarily on the user facing/specification side of it, with an emphasis on how +to get data from previous studies in the chain, update variables and other env block +tokens, how to control the iterations/chaining, and global parameters. New, always present +tokens will also be detailed. + + +## User Interface + +### New Tokens + +Encore introduces a handful of new tokens with reserved names that are always available: + +**Name** | **Description** | **Notes** | +:- | :- | :- | +`$(STUDY_ITER)` | Current iteration of this study specification | +`$(ENCORE_ITER)` | Current iteration of this study + counts of all prev studies in the chain | +`$(ENCORE.parent.workspace)`| Path to parent studies' ``encore`` step workspace | Useful focus point for reaching back into studies to get data not easily passed by other tokens/parameters | +`$(ENCORE_PREV_RESULTS)` | Path to parent study's ``encore.yaml`` | +`$(ENCORE_ROOT)` | Path to root of Encore study workspaces, which contains logs and per iteration study workspaces | + +### Encore Step, Encore.yaml + +Triggering an 'Encore' involves adding a special step named Encore and at a minimum +an ``encore.yaml`` file written inside it which passes information to Maestro about +whether to run an Encore, stop, and update any scalars/tokens in the subsequent +iteration. + +At a minimum, this `encore.yaml` needs to contain one piece of information: +```yaml title="Minimal encore.yaml" +is_done: false +``` + +Setting this ``is_done`` key to true or false is how you tell Maestro whether your study needs +an 'encore' (another iteration) or not. If no other information is provided, Maestro assumes +you want to iterate on the current study specification. However, as study's cannot generate +parameters during execution, you will often want to pass in new values to parameters or update +env block tokens. + +### Changing Parameters + +Changing parameters can be done via two methods, just like a standard Maestro study. The two examples +below assume a sample study that has a single parameter 'PARAM1', and we'll generate 4 new values +with both methods. + +=== "``global.parameters``" + + In this scenario, the format to update parameters is a mapping of parameters contained in the + ``parameters`` key in the list of child_specs, with a structure identical to what you see in + the Maestro study specification + + ```yaml title="Explicit parameter value specification in encore.yaml" + is_done: false + child_specs: + - name: $(CURRENT_SPEC) + parameters: + PARAM1: + values: [0, 0.2, 0.7, 0.01] + labels: PARAM1.%% + ``` + + !!! note + Might want some 'helper functions' from maestro to write new ones into ``encore.yaml`` here? + +=== "``pgen``" + + Invoke pgen, with or without args, using the ``pgen`` and ``pgen_args`` mappings in the list of + child_specs. + + ```yaml title="Call pgen with pargs to setup next iterations' parameters" + is_done: false + child_specs: + - name: $(CURRENT_SPEC) + pgen: pgen.py + pgen_args: + - name: num_values + value: "4" + - name: value_range + value: "0,1" + ``` + + This is equivalent to the cli invocation of a pgen that has pargs named 'num_values' and 'value_range': + + ```shell + maestro run encore_study.yaml --pgen pgen.py --parg "num_values:10" --parg "value_range:0,1" + ``` + + +### Updating existing ``env`` block tokens + + + + +It is also possible to update the configuration set in the [Environment Block (``env``)](../specification#Environment Tokens) via the encore.yaml, on a per child study specification basis. + +```yaml title="Parent study specification env block" +env: + variables: + TOKEN1: 1.0 + + dependencies: + paths: + - name: INPUT_DATA_FILE + path: initial_input_data.csv +``` + +You can reference these tokens in the encore.yaml and change the values: + +```yaml title="Updating env tokens for the next iteration" +is_done: false +child_specs: + - name: $(CURRENT_STUDY_SPECIFICATION) + env: + variables: + TOKEN1: 2.0 + dependencies: + paths: + - name: INPUT_DATA_FILE + path: iteration_2_input_data.csv +``` + +!!! note + Should we require users to write the `$(ENCORE.parent.workspace)` token in the `path` or just + inline that in the generated study spec for that next iteration automatically? + +We can enable error checking/error messaging by comparing these tokens with what's in the child_study, +making it easy to catch typos that result in new unused tokens. + +### Dispatching multiple new studies + +This results file can also be used to launch multiple new studies at once, which is most useful +in the case that the child study specifications aren't the same as the current one, i.e. where you +are changing more than just the parameter values. This use case is where the list structure +of the ``child_specs`` item becomes important + +=== "Dispatch two children with no parameters" + + In this scenario, the format to update parameters is a mapping of parameters contained in the + ``parameters`` key in the list of child_specs, with a structure identical to what you see in + the Maestro study specification + + ```yaml title="Multiple child studies with no value/config inputs" + is_done: false + child_specs: + - name: child_study_A.yaml + + - name: child_study_B.yaml + ``` + + !!! note + Find a better use case to document this and help sort out passing data between these when + it's not just parameters (i.e. workspace paths, consolidated data files, ...) + +=== "Pass data to child specs" + + Placeholder for example to pass some sort of data to each child + + !!! note + What about dependencies here? -> three children, with one dependent upon the other two for + a case with multiple parents? + +### Communicating/documenting the process + +A core philosophy of Maestro is enabling reproducible science, and that means clearly documenting +the process. To that end, a major question mark on this feature is what/if any indicators might +be useful in the spec to show that particular tokens are intended for use in the encore feature? +Seralization/creation of the ``encore.yaml`` file is likely to often be done in friendlier languages +than bash, such as python, which obscures the details of it from view in the study specification. +Thus, how are future users to know which/if any ``env`` block tokens are updated in each iteration? + +As a motivating example, consider a simple Newton style optimizer which requires previous iteration +data to compute the new search direction and step size. We will solve a simple system to illustrate +this: find the minimum value of a quadratic function + +$$ +y = ax^2 + bx + c +$$ + +A newton step (dx) for this would be + +$$ +\begin{gather*} +y_i &=& a x_i^2 + b x_i + c \\ +dx &=& -\frac{2.0*a x_i + b}{2.0*c} \\ +x_{i+1} &=& x_i + dx \\ +y_{i+1} &=& a x_{i+1}^2 + b x_{i+1} + c \\ +\end{gather*} +$$ + +For this to work we need an initial guess $x_i$, and we need to update that every iteration. So +on iteration two, the $x_{i+1}$ computed in iteration 1 would need to be passed in to provide the +$x_i$. Thus might have something like this in the ``env`` block of the spec to kick things off + +```yaml title="Newton example's env block" +env: + variables: + X_INIT: 1.0 +``` + +and then the following in the ``encore.yaml`` to update the token's value for the next iteration + +```yaml title="Newton example's encore.yaml" +is_done: false +child_specs: + - name: $(CURRENT_SPEC) + env: + variables: + X_INIT: +``` + +With only the information visible in the initial specification, there's no real indication that +``X_INIT`` is a token that will be updated every iteration: that info remains buried in the +supporting python script used to write out the ``encore.yaml`` with the new value from the current +iterations computed $x_{i+1}$. Is there something we can/should tag/mark some tokens as +overridable/updateable? + +=== "Tag it with 'reserved' encore attributes" + + ```yaml + env: + variables: + X_INIT: + encore_variable: True + value: 1.0 + ``` + +=== "Generic interactive cli override tags" + + ```yaml + env: + variables: + X_INIT: + value: 1.0 + prompt: "Enter a starting point for the newton solver: single floating point number" + ``` + + Here, ``prompt`` would be a way to both mark this as an overridable parameter, communicate that + to future users, as well as provide a means of asking for values to be input when you call + `maestro run ..` with some text to help guide the input. Such a feature could also mark it + for use by encore if a convention is adopted that only overridable tokens can be passed via + ``encore.yaml``. The fact that encore may be modifying this is potentialy less immediately + clear when viewing a spec however. + + +### Optional Capabilities + +Enabling multiple study dispatch upon the second iteration does open up a potentialy interesting +possibility: enabling study dependencies as each Maestro study does with steps. I.e. in the below +example we can launch independent iterations of study1 and study2, and then execute study3 upon +completion of both of those + +```yaml title="encore.yaml" +is_done: false +child_specs: + - name: study2.yaml + pgen: chained_pgen.py + - name: study1.yaml + pgen: chained_pgen.py + pgen_args: + - name: num_values + value: "10" + - name: value_range + value: "0, 1" + - name: study3.yaml + depends: [study2, study3] + ... +``` + +An open question remains: is this use case really needed/useful, or just adding extra complexity? diff --git a/docs/Maestro/meps/mep-002-parameter-composition.md b/docs/Maestro/meps/mep-002-parameter-composition.md new file mode 100644 index 00000000..4921b325 --- /dev/null +++ b/docs/Maestro/meps/mep-002-parameter-composition.md @@ -0,0 +1,377 @@ + + +# MEP 002 - Parameter Composition: Add common operators to study specification an pgen + +## Abstract + +The default parameter value construction in Maestro is limited to explicit value lists in the study specification. While pgen facilitates arbitrary construction via python, there are a variety of common operations that are used in composing parameters that could benefit from an api to reduce user boilerplate code. This proposal outlines a set of basic operations to enable support for along with a UI for working with them directly in the yaml formatted study specification and a corresponding set of functionality available in pgen. Maestro aims to be minimal on dependencies and not pin all users to particular solutions where possible. This proposal takes care to maintain that. The goal is for the core features and operations to depend only upon standard library utilities, with the excellent itertools library facilitating much of this. There are still many operations that cannot be supported this way, such as the expansive space around statistical methods for generating values, e.g. latin hypercube, random number generation, etc. For such capabilities a plugin interface will be detailed to enable seamless addition of workflow specific operations that don't require all Maestro users to use the same sampling libraries. + +## Conventions + +### Assigning Combinations + +A special key is used to mark the final/selected combination set: `PARAMETER.COMBINATIONS`, which can either use an operator directly or using the `composition_id` key: + +``` yaml + INITIAL_VELOCITY: + values: [0.1, 0.2, 0.3] + labels: "INIT_VEL.%%" + + STOP_TIME: + values: [4.0, 2.0, 1.0] + labels: "ST.%%" + + PARAMETERS: + operator: zip + inputs: [INITIAL_VELOCITY, STOP_TIME] + + PARAMETER.COMBINATIONS: # (1) + composition_id: PARAMETERS +``` + +2. Here we identify what composition defines the parameter combinations for this study + + +The resulting set of parameter combinations: + +| **Parameter** | **Combo 1** | **Combo 2** | **Combo 3** | +| :-----------: | :---------: | :---------: | :---------: | +| INITIAL_VELOCITY | 0.1 | 0.2 | 0.3 | +| STOP_TIME | 4.0 | 2.0 | 1.0 | + +### Labels + +Only parameters with labels keys will show up in the final parameter combination set, e.g. INITIAL_VELOCITY shows up, but PARAMETERS +does not in the snippet below. + +``` yaml +INITIAL_VELOCITY: + values: [0.1, 0.2, 0.3] + labels: INIT_VEL.%% + +REVERSED_VELS: + operator: reverse + input: INITIAL_VELOCITY +``` + +Resulting parameter combinations (after assigning REVERSED_VELS to PARAMETER.COMBINATIONS) + +| **Parameter** | **Combo 1** | **Combo 2** | **Combo 3** | +| :-----------: | :---------: | :---------: | :---------: | +| INITIAL_VELOCITY | 0.1 | 0.2 | 0.3 | + +## Common Operations + +Here we detail a base set of operators to provide out of the box for building and composing parameter combinations. + +`List` + +: Explicit list of values (currently the only method available in `global.parameters` block in the study specification) + + ``` yaml + INITIAL_VELOCITY: + values: [0.1, 0.2, 0.3] + labels: INIT_VEL.%% + ``` + + | **Parameter** | **Combo 1** | **Combo 2** | **Combo 3** | + | :-----------: | :---------: | :---------: | :---------: | + | INITIAL_VELOCITY | 0.1 | 0.2 | 0.3 | + +`Range` + +: Create value lists using start, stop, and increment. Similar to `range` python function, + but is inclusive of the stop. + + ``` yaml + RESOLUTION: + operator: range + start: 1 + stop: 4 + increment: 1 + labels: RES.%% + ``` + + | **Parameter** | **Combo 1** | **Combo 2** | **Combo 3** | **Combo 4** | + | :-----------: | :---------: | :---------: | :---------: | :---------: | + | RESOLUTION | 1 | 2 | 3 | 4 | + +`Linspace` + +: Linear sampling between start, stop points, creating `N` intervals, inclusive of start/stop values + + ``` yaml + INITIAL_VELOCITY: + operator: linspace + start: 0.1 + stop: 0.4 + intervals: 4 + labels: INIT_VEL.%% + ``` + + | **Parameter** | **Combo 1** | **Combo 2** | **Combo 3** | **Combo 4** | + | :-----------: | :---------: | :---------: | :---------: | :---------: | + | INITIAL_VELOCITY | 0.1 | 0.2 | 0.3 | 0.4 | + +`Random` + +: TODO: fill out the many ways to build random numbers using the standard lib (distributions, seeds, float vs int vs strings (i.e. sample from a list of values))... + +`Zip` + +: Compositional operator, as with pythons' zip function, used to create lists of tuples built from other lists. This is the default + operator in global.parameters for combining individual parameters into the parameter combinations. + + ``` yaml + INITIAL_VELOCITY: + values: [0.1, 0.2, 0.3] + labels: "INIT_VEL.%%" + + STOP_TIME: + values: [4.0, 2.0, 1.0] + labels: "ST.%%" + + PARAMETERS: + operator: zip + inputs: [INITIAL_VELOCITY, STOP_TIME] + ``` + + | **Parameter** | **Combo 1** | **Combo 2** | **Combo 3** | + | :-----------: | :---------: | :---------: | :---------: | + | INITIAL_VELOCITY | 0.1 | 0.2 | 0.3 | + | STOP_TIME | 4.0 | 2.0 | 1.0 | + + !!! question + + Should this have a strict mode switch? global.parameters uses strict mode, meaning unequal lengths + is an error, while python's zip silently truncates to the shortest parameter + +`Batched` + +: Build lists of tuples from a single list, (see itertools' batched) + + ``` yaml + MIXED_PARAMS: + operator: batched + input_values: [4, A, 3, B, 2, C] + batch_size: 2 + labels: ['INTPARAM.%%', 'STRPARAM.%%'] # (1) + ``` + + 1. Labels is now a tuple/list, same size as batch_size, giving each param in the batch a label + + | **Parameter** | **Combo 1** | **Combo 2** | **Combo 3** | + | :-----------: | :---------: | :---------: | :---------: | + | INTPARAM | 4 | 3 | 2 | + | STRPARAM | A | B | C | + + !!! question + + Is this really a useful operator? Seems very nice and ~convoluted for also getting the labels in there + + +`Repeat` + +: Replicate a constant value: i.e. wrap a single value parameter before handing off to zip to combine it with another explicit list. (NOTE: should this just take the 'N' argument instead of requiring going through zip?) + + ``` yaml + RESOLUTION: + operator: repeat + value: 1 + # count: 3 # (1) + labels: RES.%% + + INITIAL_VELOCITY: + values: [0.1, 0.2, 0.3] + labels: INITVEL.%% + + PARAMETERS: + operator: zip # (2) + inputs: [INITIAL_VELOCITY, RESOLUTION] + ``` + + 1. Optional manual count if expansion is needed before combining with other operators + 2. Zip triggers repeats as many times as needed if count is absent + + | **Parameter** | **Combo 1** | **Combo 2** | **Combo 3** | + | :-----------: | :---------: | :---------: | :---------: | + | INITIAL_VELOCITY | 0.1 | 0.2 | 0.3 | + | RESOLUTION | 1 | 1 | 1 | + + !!! question + + Another option for auto count is using cycle instead: should repeat just be manual only? + + +`Reverse` + +: Transformational operator, reversing an existing list of values before combining with other parameters + + ``` yaml + INITIAL_VELOCITY: + values: [0.1, 0.2, 0.3] + labels: INITVEL.%% + + PARAMETERS: + operator: reverse + input: INITIAL_VELOCITY + ``` + + + | **Parameter** | **Combo 1** | **Combo 2** | **Combo 3** | + | :-----------: | :---------: | :---------: | :---------: | + | INITIAL_VELOCITY | 0.3 | 0.2 | 0.1 | + + +`Slice` + +: Slice and dice and existing list of values to subset and exsiting list of values + +`Sort` + +: Rearrange an existing list of values on some criteria + +`Randomize` + +: Randomize the order of an existing list of values + +`Uniquify` + +: Reduce a list of values to the set of unique values in it + +`Cycle` + +: Enable indexing beyond the end of a list of values; e.g. wrap a parameter in cycle before combining it with another longer list of values in the zip operator + +`Product` + +: Take a cross product of two or more parameters (should we have inner and outer products?) + +`Permute` + +: Generate permutations from a list of values, or between values of multiple lists + +`Combinations` + +: See itertools.combinations.... + +## Study Specification Interface + +A new block is proposed for the spec to avoid adding Maestro version specific behavior changes to the `global.parameters` block, leaving that as the default interface for explicit lists of parameters/parameter values. An intial name for this block of `parameters.compose` will be used in the examples below. THere are two important differences between this block and `global.parameters`: + +1. Intermediate, anonymous/temporary parameters are supported. This facilitates parameter construction using multiple chained operations without either making those chained operations overly complex or requiring that these intermediates show up in the final parameter combinations. + +2. A reserved key for assigning a specific named chain of operations to be used as the set of parameter combinations in this study. This facilitates having multiple operations defined and being able to subselect them by changing/overriding one value instead of swapping out the entire block. + + !!! warning + + This block structure/key may change to be more like separate blocks, chosing one of those by name modulo feedback on the interfaces. Could be helpful to use separate blocks for more clear organization of parameters, but may still want to share between blocks to reduce duplication. + + +### Examples + +#### `global.parameters` behavior in composition block + +This simple case shows how to replicate the behavior of the existing `global.parameters` block via composition operations using two slightly different options, both resulting in the same set of parameter combinations + +=== "Direct Operator" + + ``` yaml + INITIAL_VELOCITY: + values: [0.1, 0.2, 0.3] + labels: "INIT_VEL.%%" + + STOP_TIME: + values: [4.0, 2.0, 1.0] + labels: "ST.%%" + + PARAMETER.COMBINATIONS: # (1) + operator: zip + inputs: [INITIAL_VELOCITY, STOP_TIME] + ``` + + 1. Here we identify what composition defines the parameter combinations for this study + +=== "Intermediate Composition" + + ``` yaml + INITIAL_VELOCITY: + values: [0.1, 0.2, 0.3] + labels: "INIT_VEL.%%" + + STOP_TIME: + values: [4.0, 2.0, 1.0] + labels: "ST.%%" + + PARAMETERS: # (1) + operator: zip + inputs: [INITIAL_VELOCITY, STOP_TIME] + + PARAMETER.COMBINATIONS: # (2) + composition_id: PARAMETERS + ``` + + 1. An intermediate combination + 2. Here we identify what composition defines the parameter combinations for this study + + +The resulting set of parameter combinations: + +| **Parameter** | **Combo 1** | **Combo 2** | **Combo 3** | +| :-----------: | :---------: | :---------: | :---------: | +| INITIAL_VELOCITY | 0.1 | 0.2 | 0.3 | +| STOP_TIME | 4.0 | 2.0 | 1.0 | + +#### Tuples and cross products + +Consider the case of having an existing set of configurations and you want to perturb each one the same way, such as a resolution study + +``` yaml +parameters.compose: + + INITIAL_VELOCITY: + values: [0.1, 0.2, 0.3] # NOTE: should we replace values with list here? force operators in this block all the time? + labels: "INIT_VEL.%%" # Use familiar Maestro syntax for generating human readable string representation of values + + STOP_TIME: + values: [4.0, 2.0, 1.0] + labels: "ST.%%" + + RESOLUTION: + values: [1, 2] + labels: "RES.%%" + + GROUP1: # (1) + operator: zip + inputs: [INITIAL_VELOCITY, STOP_TIME] # (2) + + RES_STUDY: + operator: product + inputs: [GROUP1, RESOLUTION] + + PARAMETER.COMBINATIONS: # (3) + composition_id: RES_STUDY +``` + +1. An intermediate combination +2. Apply zip to the list of named parameters or compositions +3. Here we identify what composition defines the parameter combinations for this study + +The resulting set of parameter combinations: + +| **Parameter** | **Combo 1** | **Combo 2** | **Combo 3** | **Combo 4** | **Combo 5** | **Combo 6** | +| :-----------: | :---------: | :---------: | :---------: | :---------: | :---------: | :---------: | +| INITIAL_VELOCITY | 0.1 | 0.1 | 0.2 | 0.2 | 0.3 | 0.3 | +| STOP_TIME | 4.0 | 4.0 | 2.0 | 2.0 | 1.0 | 1.0 | +| RESOLUTION | 1 | 2 | 1 | 2 | 1 | 2 | + + +## Register Custom Operators + +A plugin interface will be provided to facilitate registering custom operators that go beyond what the standard library +can provide. This will facilitate hooking up the statistical package of your choice to add custom sampling operations +such as best candidate or latin hypercubes. + +!!! warning + + Under construction!! diff --git a/docs/Maestro/meps/mep-003-human-readable-hashing.md b/docs/Maestro/meps/mep-003-human-readable-hashing.md new file mode 100644 index 00000000..e68aa9ff --- /dev/null +++ b/docs/Maestro/meps/mep-003-human-readable-hashing.md @@ -0,0 +1,192 @@ +# MEP 003 - Human Readable Hashing: Compact and sortable step name hashing + +## Abstract + +Step naming and workspace naming behaviors in Maestro is a frequent pain point in many studies. The default naming convention aimed to provide quick lookup by encoding parameter names and values in the step name and workspace names. There are multiple issues that can occur with this scheme that are the target of this enhancement proposal: + +* Floating point numbers + * String versions of floats chosen by a human are not always representable exactly once converted into binary, e.g. 0.1, which is an infinitely repeating binary sequence, which converted back to a string with 17 digits of precision is 0.10000000000000001, 18 yields 0.100000000000000006, ... Default Maestro label construction would thus have varying numbers of digits in the parameter combination's label/id, frequently far more than what appears in the study specification + * Primarily a readability problem which gets worse with more digits + +* Many parameters + * Many parameters lead to ~unreadable step id's/workspace names, whether that's reading from the command line (`ls`) one of the status command's tables, or some other tabular report/document outside of Maestro (documents, dashboards, ...) + * Using 10's of parameters, especially with floats, can quickly yield final workspace names that blow out system path length limits. The only current solution is hashing + * Both a readability problem and cause of crashing workflows; auto-hashing may help here, but that has it's own issues (see next item) + +* Existing hashing + * While this solves the path length issues, it also produces a human unfriendly string which is generally not sortable, not amenable to tab completion, and is quite unreadable and difficult to use even to look up parameter names/values out of a table + * Existing method uses md5, but other hashing options aren't really any more readable/human friendly + * Trades readability for compactness + + +This proposal aims to relieve these tensions, maintaining compactness, human readability, and still guaranteeing uniqueness of the hash. + +!!! danger + + This proposal does not really detail a proper 'hash', as individual step's information (step name, parameter names/values, ...) is not + enough info by itself to determine the resulting 'hash'. Rather the hash is dependent upon the number of instances of a step, i.e. + an ordering/enumeration. + +## Parameter Combinations + +A core identifier of a Maestro step instance is the parameter combination. We shall refer to a simple study below to illustrate how parameter combinations work and the different kinds that can be attached to a study step instance. + +### Demo Study Spec +``` yaml +description: + name: parameter_combo_demo + description: | + Simple study used to demonstrate parameter combinations and a new + workspace/step hashing implementation + +study: + - name: donor-sim + description: Simple step using a subset of parameters + run: + cmd: | + echo "Used Parameters: RES: $(RES)" + + - name: acceptor-sim + description: Simple step using all parameters + run: + cmd: | + echo "Used Parameters: RES: $(RES), SHIFT_X: $(SHIFT_X)" + +global.parameters: + RES: + values: [1, 1, 2, 2] + labels: RES.%% + + SHIFT_X: + values: [3, 5, 3, 5] +``` + +### Demo Study Parameter Combinations +This results in the following set of parameter combinations: + +| **Parameter** | **Combo 1** | **Combo 2** | **Combo 3** | **Combo 4** | +| :-----------: | :---------: | :---------: | :---------: | :---------: | +| RES | 1 | 1 | 2 | 2 | +| SHIFT_X | 3 | 5 | 3 | 5 | + +Now we take into account the concept of 'used parameters', which is what Maestro uses under the covers to build the graph of instantiated steps. Each column in this table represents one parameter combination, for a toatal of four. As there are four unique values of these tuples, any step using both parameters (the steps' used parameters) will have four instances. We see `acceptor-sim` uses both, so we have four instances of this step in the study. However, `donor-sim` only uses one of the parameters, `RES`, and we can see there are only 2 unique values, leading to only two instances of `donor-sim`, as shown in the topology below: + +### Demo Study Topology +``` mermaid +flowchart TD; + A(study-root) --> donor_sim_1; + subgraph donor_sim_1 [donor-sim] + subgraph S1COMBO1 [Donor-Sim Used Combo 1] + B(RES = 1); + end + end + A --> donor_sim_2; + subgraph donor_sim_2 [donor-sim] + subgraph S1COMBO2 [Donor-Sim Used Combo 2] + C(RES = 2); + end + end + donor_sim_1 --> step_2_1; + subgraph step_2_1 [acceptor-sim] + subgraph S2COMBO1 [Acceptor-Sim Used Combo 1] + D(RES = 1\nSHIFT_X = 3); + end + end + donor_sim_1 --> step_2_2; + subgraph step_2_2 [acceptor-sim] + subgraph S2COMBO2 [Acceptor-Sim Used Combo 2] + E(RES = 1\nSHIFT_X = 5); + end + end + donor_sim_2 --> step_2_3; + subgraph step_2_3 [acceptor-sim] + subgraph S2COMBO3 [Acceptor-Sim Used Combo 3] + F(RES = 2\nSHIFT_X = 3); + end + end + donor_sim_2 --> step_2_4; + subgraph step_2_4 [acceptor-sim] + subgraph S2COMBO4 [Acceptor-Sim Used Combo 4] + G(RES = 2\nSHIFT_X = 5); + end + end +``` + +### Demo Step Names + +#### Default Naming ~v1.1 + +Default names for steps, and workspaces, uses the parameter labels as of v1.1.11, current release as of this draft. These labels are meant to be more human friendly formats of parameter name.values that identify a specific parameter value. Step/workspace naming uses the used parameter combinations' labels, as shown below: + +| **Used Combo \#** | **Step Name/Workspace Name** | +| :-----------: | :---------: | +| donor-sim used combo 1 | donor-sim_RES.1 | +| donor-sim used combo 2 | donor-sim_RES.2 | +| acceptor-sim used combo 1 | acceptor-sim_RES.1.SHIFT_X.3 | +| acceptor-sim used combo 2 | acceptor-sim_RES.1.SHIFT_X.5 | +| acceptor-sim used combo 3 | acceptor-sim_RES.2.SHIFT_X.3 | +| acceptor-sim used combo 4 | acceptor-sim_RES.2.SHIFT_X.5 | + +## Proposed New Hashing Scheme + +A simple, readable hashing scheme is evident in the examples above: `_`, where `used_combination_ID` is the step specific used combination numbers, since each step can have it's own set of combinations, and varying numbers of combinations per step. This per-step nature leads to the base step name being retained as a prefix to ensure it's clear that `used_combination_1` in `donor-sim`'s workspaces is not the same as that within `acceptor-sim`'s workspaces. + +### Demo Hashing/Workspaces + +| **Used Combo \#** | **Sorted Parameter Values** | **Hashed step id/workspace** | +| :-----------: | :---------: | :---------: | +| donor-sim used combo 1 | RES: 1 | donor-sim_used_combination_1 | +| donor-sim used combo 2 | RES: 2 | donor-sim_used_combination_2 | +| acceptor-sim used combo 1 | RES: 1, SHIFT_X: 3 | acceptor-sim_used_combination_1 | +| acceptor-sim used combo 2 | RES: 1, SHIFT_X: 5 | acceptor-sim_used_combination_2 | +| acceptor-sim used combo 3 | RES: 2, SHIFT_X: 3 | acceptor-sim_used_combination_3 | +| acceptor-sim used combo 4 | RES: 2, SHIFT_X: 5 | acceptor-sim_used_combination_4 | + +### Hash construction/parameter ordering + +Owing to the use of set intersections for determining id's and connectivity, much order information is lost once graph expasion is done. To keep things simple and ~intuitive, all the used combinations will use sorting basd on parameter names, and then values, with used_combination number counting up from one from that sorted list. This is reflected in the prior workspace/combo table + +### Study Metadata + +There will be some corresponding tweaks to the parameters.yaml metadata to expose the step specific used parameter combinations in addition to the full set of parameter combinations. This is to facilitate quick lookups of parameter values, e.g. bash or zsh shell functions to quickly get this information via yq , pipe it into fzf and then onto cd/pushd for study workspace navigation where you can select step workspaces based on human readable parameter name: value tables, or other quick lookups. This expands upon the current parameters.yaml which only contained the full parameter combinations, requiring sometimes expensive operations to find the one corresponding to a current step that only uses a subset of parameters. + +!!! question + + Add snippets of parameters.yaml, both old and new, and also demo gif of yq + fzf dir navigation workflow? + +## Additional Considerations + +### Adding parameter combinations to existing study + +Consider a hypothetical feature to enable running new parameter combinations through an existing completed study. The additional step instances such a process creates makes a bit of a mess of the naming convention given there are no guarantees on the ordering of the values in such new combinations being > what's already there. So what solutions could there be for this: + +- Don't allow adding new parameter combinations to existing study workspace (current behavior) +- Dynamic renaming: this would be very disruptive and make a mess of any metadata slurped up into anything else as both that and the workspace paths would be changed +- Ignore the tuple style ordering on the names and values after the initial batch: i.e. compute that up front with local ordering per batch of parameters, but join them based on insert order. + - Not very intuitive for users given lack of indicator when that order assumption changes if we don't also perturb the name of the combination +- Add a new suffix/prefix to the new steps to indicate the break in ordering. There are a few options: + - **group**: Pretty self explanatory marker for a new batch of parameter combinations + - **batch**: Same as above + - **movement**: More music themed name for the different groups/sets + - **set**: Still ~self explanatory, but also overloaded with the music theme. Bonus for being more compact than movement + - Others, that start deviating: **chorus** (repeating sections of music), **verse** (..), **wave**, ... + + +Here's what a few of the options might look like, either omitting or includign the suffix on even the initial batch if such a feature is enabled, or making it implicit (first batch, i.e. no suffix on the first group, only adding it if a second batch of parameter combinations is run thorugh the study): + + + +### Multi-machine workflows + +Given multi-machine workflows are on the roadmap, there's yet another wrench to throw into the works: what to do with the resource sets, i.e. procs, nodes, ... These are frequently templated with parameters just like the body of the steps. However, what does this specific set of parameters, or even a condensed 'resource_set_id' grouping of them mean in this context? Simply removing them from this naming, i.e. filtering them from the combination id/sorting order, is not an option as that may interfere with resolution and scaling studies where they may be the source of uniqueness relative to other combinations in the set. Further complicating this is that they may not be single valued per step instance in a given study when extrapolated to the multi-machine context. Currently available hardware spans a range of resource sets that may fit a given step's requirements, limiting our selves to exclusive usage for now as on a node-scheduled HPC cluster: + + +| **Resource Set ID** | **Tasks** | **Cores per node** | **Nodes** | **GPUS** | +| :-----------------: | :-------: | :----------------: | :-------: | :------: | +| rs_cpu_1 | 144 | 36 | 4 | 0 | +| rs_cpu_2 | 112 | 56 | 2 | 0 | +| rs_cpu_3 | 112 | 112 | 1 | 0 | +| rs_cpu_4 | 192 | 96 | 2 | 0 | +| rs_cpu_5 | 128 | 128 | 1 | 0 | +| rs_gpu_1 | 4 | 112 | 1 | 4 | +| rs_gpu_2 | 4 | 96 | 1 | 4 | diff --git a/docs/Maestro/meps/mep-004-step-dependency-execution-policy.md b/docs/Maestro/meps/mep-004-step-dependency-execution-policy.md new file mode 100644 index 00000000..e73a0085 --- /dev/null +++ b/docs/Maestro/meps/mep-004-step-dependency-execution-policy.md @@ -0,0 +1,339 @@ +# MEP 004 - Step Dependency Execution Policy + +## Abstract + +Maestro step execution policies are currently hardwired to successful/unsuccessful step states, which are themselves tightly coupled to the states returned by the scheduler. User control over this is limited to presence of a `restart` block in a step to take advantage of Timeout states, or manipulating the return state/exit code of the step scripts themselves to affect the state reported by the scheduler as detailed in the how-to-guides here [INSERT LINK]. Multiple mechanisms are needed to enhance this capability and move towards a decoupling of the workflow state from the scheduler and step task states. This proposal details a hook in the study specification to set an execution policy on a per step basis. This is intentionally decoupled from the step/scheduler execution layers to better deal with cases where the how-to-guides' recipes [INSERT LINK] fail to execute due to OOM's, Hardware Failures, or other states that preclude complete execution of a study step. + +The states that feed into this control mechanism are the 'final outcomes' of the state. These final outcomes are divided +into two groups: + + +| **Final Outcome** | **States** | **Meaning** | +| :---------------: | :---------: | :---------: | +| **COMPLETED** | Success, Failed, Out of Memory, Hardware Failure, Timeout, Restart Limit Reached | Execution and restart processing have concluded. Completion does not imply success. | +| **CANCELLED** | Cancelled | Execution was cancelled and downstream execution is prohibited | + + +Restarts can still occur with this proposed mechanism, currently triggering on receipt of hardware failure and timeout states from the scheduler. If the restart block is present in the step in question, restarts will be submitted until restart limit is reached, success, or failure. When either restart limit or other completed outcome is reached, or the restart block is absent, timeout and hardware failure states are promoted to a completed outcome. + +The proposed execution policy is simply stating what to do when parent steps have one of these final outcomes, signaling the step (and workflow) to either continue or stop. Current maestro behavior will only execute children if their parent(s) have a Success state. This new mechanism will optionally allow execution a step if any selected completed outcome is reached by its parent(s). As `depends` is itself a condtion upon which to control the execution of a step, we propose naming this new control aspect, `condition`, with a limited set of scalar values in this intial implementation: + +| **Condition** | **Semantics** | +| :-----------: | :-----------: | +| `all-succeeded` | Every parent must finish successfully. This preserves current behavior and will be the default. | +| `all-completed` | Every parent must reach a non-cancelled final outcome. | + +### Proposed syntax + +The proposed syntax for this new capability in the study specification extends the depends key to allow mappings +for the value in addition to the current list shape. The mapping retains the familiar topological constraint for +the list of step names defining it's parents (and the topology of the graph), and the new `condition` key here. + +``` yaml linenums="1" hl_lines="21-22" +description: + name: simple_study + description: | + Simple study used to demonstrate step dependency execution + policy. + +study: + - name: run-simulation + description: Step that executes a simulation + run: + cmd: | + echo "Used Parameters: RES: $(RES)" + + - name: process-simulation + description: Simple step that processes the outputs of a simulation + run: + cmd: | + echo "Processing simulation in $(run-simulation.workspace)" + + depends: + steps: [run-simulation] + condition: all-completed + + - name: report + description: Simple step that generates a report of processed data + run: + cmd: | + echo "Generating report of processed simulation data in $(process-simulation)" + + depends: + steps: [process-simulation] + condition: all-completed + +global.parameters: + RES: + values: [1, 2] + labels: RES.%% +``` + +### Multiple conditions + +#### Option 1 + +A simple extension using the syntax in `global.parameters` blocks would enable setting different conditions on different +steps for cases where a child has many parent steps where we allow the `all-completed` state for the `run-simulation-A` +step, but we want to require all instances of the `run-simulation-B` step to meet the more stringent `all-succeeded` +condition. + +``` Yaml linenums="1" hl_lines="27-28" +description: + name: simple_study + description: | + Simple study used to demonstrate step dependency execution + policy. + +study: + - name: run-simulation-A + description: Step that executes simulation-A + run: + cmd: | + echo "Used Parameters: RES_A: $(RES_A)" + + - name: run-simulation-B + description: Step that executes simulation-B + run: + cmd: | + echo "Used Parameters: RES_B: $(RES_B)" + + - name: process-simulation + description: Simple step that processes the outputs of a simulation + run: + cmd: | + echo "Processing simulation in $(run-simulation.workspace)" + + depends: + steps: [run-simulation-A, run-simulation-B] + condition: [all-completed, all-succeeded] + + - name: report + description: Simple step that generates a report of processed data + run: + cmd: | + echo "Generating report of processed simulation data in $(process-simulation)" + + depends: + steps: [process-simulation] + condition: [all-completed] + +global.parameters: + RES_A: + values: [1, 2] + labels: RES.%% + + RES_B: + values: [1, 2] + labels: RES.%% +``` + +#### Option 2 + +This option may be more readable in cases where you have many parent steps; two parenst as shown here isn't too stressful, +but once the lists start wrapping in the editor it becomes more cumbersome to map the condition to the step. This list +of mappings syntax change things slightly, using `step` instead of `steps`. + +!!! note + + Could potentially mix the two by allowing each mapping to apply a condition to many steps, more like the single condition syntax? + +``` Yaml linenums="1" hl_lines="28-31" +description: + name: simple_study + description: | + Simple study used to demonstrate step dependency execution + policy. + +study: + - name: run-simulation-A + description: Step that executes simulation-A + run: + cmd: | + echo "Used Parameters: RES_A: $(RES_A)" + + - name: run-simulation-B + description: Step that executes simulation-B + run: + cmd: | + echo "Used Parameters: RES_B: $(RES_B)" + + - name: process-simulation + description: Simple step that processes the outputs of a simulation + run: + cmd: | + echo "Processing simulations in $(run-simulation-A.workspace)" + echo "Processing simulations in $(run-simulation-B.workspace)" + + depends: + - step: run-simulation-A_* + condition: all-completed + - step: run-simulation-B_* + condition: all-succeeded + + - name: report + description: Simple step that generates a report of processed data + run: + cmd: | + echo "Generating report of processed simulation data in $(process-simulation)" + + depends: + - step: process-simulation + condition: all-completed + + +global.parameters: + RES_A: + values: [1, 2] + labels: RES.%% + + RES_B: + values: [1, 2] + labels: RES.%% +``` + +### Legacy syntax + +``` yaml +depends: [run-simulation] # run-simulation is parent step name +``` + +Equivalent normalized form: +``` yaml +depends: + steps: [run-simulation] + condition: all-succeeded +``` + + +### State Semantics + +| **Parent state after restart processing** | `all-succeeded` | `all-completed` | +| :---------------------------------------: | :-------------: | :-------------: | +| Success | Satisfied | Satisfied | +| Failed | Not satisfied | Satisfied | +| Out of Memory | Not satisfied | Satisfied | +| Hardware Failure | Not satisfied | Satisfied | +| Timeout | Not satisfied | Satisfied | +| Restart Limit Reached | Not satisfied | Satisfied | +| Cancelled | Not satisfied | Not satisfied | +| Running or restart pending | Wait | Wait | + +Hardware Failure and Timeout are not considered completed until no further restarts can be scheduled. + +### Application to various topologies + +#### Workflow topology + +Topology of our sample study, unexecuted + +```mermaid +flowchart LR + A1(["run-simulation-A - RES=1"]) + A2(["run-simulation-A - RES=2"]) + B1["run-simulation-B - RES=1"] + B2["run-simulation-B - RES=2"] + P["process-simulation - funnel step"] + R["report"] + + A1 --> P + A2 --> P + B1 --> P + B2 --> P + P --> R + + classDef simulation fill:#e8f4fd,stroke:#2471a3,color:#154360,stroke-width:2px + classDef process fill:#fcf3cf,stroke:#b7950b,color:#7d6608,stroke-width:2px + + class A1,A2,B1,B2 simulation + class P process + class R simulation +``` + +#### `all-succeeded`, one parent fails + +This scenario applies `all-succeeded` conditions to both simulation steps, and shows execution states +if one of those parents fails. + +```mermaid +flowchart LR + A1(["run-simulation-A - RES=1 - Success"]) + A2(["run-simulation-A - RES=2 - Success"]) + B1["run-simulation-B - RES=1 - Failed"] + B2["run-simulation-B - RES=2 - Success"] + P["❌ process-simulation - Not run - all-succeeded unmet"] + R["❌ report - Not run - upstream dependency blocked"] + + A1 --> P + A2 --> P + B1 --> P + B2 --> P + P --> R + + classDef success fill:#d5f5e3,stroke:#1e8449,color:#145a32,stroke-width:2px + classDef failed fill:#fadbd8,stroke:#c0392b,color:#7b241c,stroke-width:3px + classDef blocked fill:#f2f3f4,stroke:#5d6d7e,color:#273746,stroke-width:2px,stroke-dasharray:6 4 + + class A1,A2,B2 success + class B1 failed + class P,R blocked +``` + +#### `all-completed`, one parent fails + +This scenario applies `all-completed` conditions to both simulation steps, and shows execution states +if one of those parents fails. + +```mermaid +flowchart LR + A1(["run-simulation-A - RES=1 - Success"]) + A2(["run-simulation-A - RES=2 - Success"]) + B1["run-simulation-B - RES=1 - Failed"] + B2["run-simulation-B - RES=2 - Success"] + P["process-simulation - Success - all-completed satisfied"] + R["report - Success"] + + A1 --> P + A2 --> P + B1 --> P + B2 --> P + P --> R + + classDef success fill:#d5f5e3,stroke:#1e8449,color:#145a32,stroke-width:2px + classDef failed fill:#fadbd8,stroke:#c0392b,color:#7b241c,stroke-width:3px + + class A1,A2,B2,P,R success + class B1 failed +``` + +#### `all-completed`, one parent is cancelled + +This scenario applies `all-completed` conditions to both simulation steps, and shows execution states +if one of those parents is cancelled. Note that in this case, the outcome would be the same using +the `all-succeeded` conditions as cancellation is the one state that will halt execution in both proposed +conditions. + +```mermaid +flowchart LR + A1(["run-simulation-A - RES=1 - Success"]) + A2(["run-simulation-A - RES=2 - Success"]) + B1["run-simulation-B - RES=1 - Cancelled"] + B2["run-simulation-B - RES=2 - Success"] + P["❌ process-simulation - Not run - cancellation barrier"] + R["❌ report - Not run - upstream dependency blocked"] + + A1 --> P + A2 --> P + B1 --> P + B2 --> P + P --> R + + classDef success fill:#d5f5e3,stroke:#1e8449,color:#145a32,stroke-width:2px + classDef cancelled fill:#ede7f6,stroke:#6a1b9a,color:#4a148c,stroke-width:3px + classDef blocked fill:#f2f3f4,stroke:#5d6d7e,color:#273746,stroke-width:2px,stroke-dasharray:6 4 + + class A1,A2,B2 success + class B1 cancelled + class P,R blocked +``` + diff --git a/docs/changelog_placeholder.md b/docs/changelog_placeholder.md new file mode 100644 index 00000000..5f67cf32 --- /dev/null +++ b/docs/changelog_placeholder.md @@ -0,0 +1,3 @@ +# COMING SOON! + +This page is currently under construction diff --git a/docs/extra/mathjax.js b/docs/extra/mathjax.js new file mode 100644 index 00000000..0be88e04 --- /dev/null +++ b/docs/extra/mathjax.js @@ -0,0 +1,19 @@ +window.MathJax = { + tex: { + inlineMath: [["\\(", "\\)"]], + displayMath: [["\\[", "\\]"]], + processEscapes: true, + processEnvironments: true + }, + options: { + ignoreHtmlClass: ".*|", + processHtmlClass: "arithmatex" + } +}; + +document$.subscribe(() => { + MathJax.startup.output.clearCache() + MathJax.typesetClear() + MathJax.texReset() + MathJax.typesetPromise() +}) diff --git a/mkdocs.yml b/mkdocs.yml index 54819295..eb4ba0db 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -3,6 +3,7 @@ use_directory_urls: false markdown_extensions: - admonition - attr_list + - def_list - pymdownx.highlight: anchor_linenums: true - pymdownx.inlinehilite @@ -21,6 +22,8 @@ markdown_extensions: format: !!python/name:pymdownx.superfences.fence_code_format - pymdownx.tabbed: alternate_style: true + - pymdownx.arithmatex: + generic: true plugins: - search @@ -46,6 +49,8 @@ plugins: extra_javascript: - https://unpkg.com/mermaid@10.9.3/dist/mermaid.min.js + - extra/mathjax.js + - https://unpkg.com/mathjax@3/es5/tex-mml-chtml.js extra_css: - custom.css @@ -103,7 +108,16 @@ nav: - Maestro Reference: Maestro/reference_guide/index.md - Design Reference: Maestro/reference_guide/design_reference/ - API Reference: Maestro/reference_guide/api_reference/ - + - What's New: + - Index: 'Maestro/meps/index.md' + # - Maestro/ + # - Release Notes: Maestro/release_notes/index.md + - Changelog: 'changelog_placeholder.md' + - Maestro Enhancement Proposals: + - MEP 001: 'Maestro/meps/mep-001-encore-study-iteration.md' + - MEP 002: 'Maestro/meps/mep-002-parameter-composition.md' + - MEP 003: 'Maestro/meps/mep-003-human-readable-hashing.md' + - MEP 004: 'Maestro/meps/mep-004-step-dependency-execution-policy.md' # - API Reference: 'reference/' extra: