Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 9 additions & 2 deletions doc/ch_conceptual_design/data_organization.rst
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ The `Job` layer always includes a single data cell.
Rectangles with labels :math:`\textsf{Run}_i`, :math:`\textsf{Spill}_{i,j}`, and :math:`\textsf{APA}_{i,j,k}` represent data cells.
The pale green rectangles show two data-cell families; these are identified as families because they are the result of executing the :mathfunc:`unfold(into_apas)` node shown in :numref:`workflow`.
A solid line from one data cell to another data cell represents a logical association between the two data cells.
The bottom rectangle shows that :math:`\textsf{Waveforms}_{1,1,1}` is in the data cell :math:`\textsf{APA}_{1,1,1}`, etc.
Each dashed line between a data product and a data cell (e.g., between :math:`\textsf{Waveforms}_{1,1,1}` and :math:`\textsf{APA}_{1,1,1}`) indicates that the data product is logically contained by the data cell (see text).
Each pale purple rectangle indicates the data-product family created by unfolding each :product:`SimDepos` object as shown in :numref:`workflow`.
:name: data-organization

Expand Down Expand Up @@ -70,6 +70,13 @@ As illustrated in :numref:`data-organization`, data products are organized into
They can be unfolded into finer-grained units, enabling detailed analysis or reprocessing at different scales :need:`DUNE 43`.
This provides the ability to process data too large to fit into memory at one time :need:`DUNE 25`.

.. admonition:: Logical vs. in-memory containment

From a mathematical perspective, data cells *logically* contain data products.
This does not mean, however, that the implementation must represent a given data cell as an object that keeps all of its logically-contained data products in memory at one time.
Instead, the implementation is free to retain in memory only those data products necessary for performing a particular task (see :numref:`ch_conceptual_design/data_organization:Data Product Management`).
It is from this mathematical viewpoint that a data product equivalently *belongs to*, *is associated with*, *is contained by*, *is in*, or *is a member of* one or more data cells.

Data Product Management
-----------------------

Expand All @@ -92,7 +99,7 @@ Such metadata include:
- the *creator*, the name of the algorithm that created the data product
- an identifier for the *data cells* with which the data product is associated (e.g. `Spill`, `Run`, `Calibration Interval`, or other experiment-defined layer)
- the *stage name*, an identifier for the job in which the data product was created
- an individual *name* for the data product (which may be empty), to distinguish between multiple products of the same type created by the same algorithm.
- an individual *suffix* for the data product (which may be empty), to distinguish between multiple products of the same type created by the same algorithm.

In addition to these metadata, a data product is also specified by its *type*.

Expand Down
15 changes: 8 additions & 7 deletions doc/ch_conceptual_design/hofs/transforms.rst
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@ To illustrate the different ways a transform's algorithm can be registered with
// Return type: first number = number of good hits
// second number = number of all hits

**Transform with one argument (default output product name)**
**Transform with one argument (default output-product suffix)**

.. code:: c++

Expand All @@ -63,7 +63,7 @@ To illustrate the different ways a transform's algorithm can be registered with
.input_family(product_selector{.suffix = "Waveforms", .layer = "APA"});
}

**Transform with one argument (user-specified output product name)**
**Transform with one argument (user-specified output-product suffix)**

*As shown in* :numref:`workflow` *and described in* :numref:`ch_conceptual_design/registration:Framework Registration`

Expand All @@ -76,7 +76,7 @@ To illustrate the different ways a transform's algorithm can be registered with
.output_product_suffixes("GoodHits");
}

**Transform with two arguments (default output product name)**
**Transform with two arguments (default output-product suffix)**

*As shown in* :numref:`workflow` *and described in* :numref:`ch_conceptual_design/registration:Data Products from Different Data Layers`

Expand All @@ -88,17 +88,18 @@ To illustrate the different ways a transform's algorithm can be registered with
.input_family(
product_selector{.suffix = "Geometry", .layer = "Job"},
product_selector{.suffix = "GoodTracks", .layer = "APA"}
)
.output_product_suffixes("Vertices");
);
}

**Transform creating two data products (user-specified output product names)**
**Transform creating two data products (user-specified output-product suffixes)**

.. code:: c++

PHLEX_REGISTER_ALGORITHMS(m)
{
m.transform("hit_counter", count_good_hits, concurrency::unlimited)
.input_family(product_selector{.suffix = "GoodHits", .layer = "APA"})
.output_product_suffixes("NumGoodHits", "NumAllHits"); // <= One name per tuple slot
.output_product_suffixes(
"NumGoodHits", "NumAllHits" // <= One suffix per tuple slot
);
}
4 changes: 2 additions & 2 deletions doc/ch_preliminaries/functional_programming.rst
Original file line number Diff line number Diff line change
Expand Up @@ -86,10 +86,10 @@ Such a HOF is called a map or *transform*:
In such a scenario, the average :math:`\overline{c}` could be expressed as:

.. math::
\overline{c} = \frac{1}{n}\ \fold{+}{0}\ \transform{w}\ \fami{e} = \frac{1}{n}\ \fold{+ \comp w}{0}\ \fami{e}
\overline{c} = \frac{1}{n}\ \fold{+}{0}\ \transform{w}\ \fami{e} = \frac{1}{n}\ \fold{+ \comp (w \times w)}{0}\ \fami{e}

The second equality holds by the fold-map fusion law [Bird]_, which states that the application of a :math:`\text{transform}` followed by a :math:`\text{fold}` can be reduced to a single :math:`\text{fold}`.
The operator to this single fold is ':math:`+ \comp w`', indicating that the function :math:`w` should be applied first before invoking the :math:`+` operation.
The operator to this single fold is ':math:`+ \comp (w \times w)`', indicating that the function :math:`w` is applied to both input arguments of the :math:`+` operation.
Relying on such mathematical laws permits the replacement of chained calculations with a single calculation, often leading to efficiency improvements without affecting the result.

.. _hofs_in_phlex:
Expand Down
Loading