Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 22 additions & 0 deletions .readthedocs.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
# Read the Docs configuration file
# See https://docs.readthedocs.io/en/stable/config-file/v2.html for details

# Required
version: 2

# Set the OS, Python version, and other tools you might need
build:
os: ubuntu-24.04
tools:
python: "3.13"

# Build documentation in the "docs/" directory with Sphinx
sphinx:
configuration: docs/conf.py

# Optionally, but recommended,
# declare the Python requirements required to build your documentation
# See https://docs.readthedocs.io/en/stable/guides/reproducible-builds.html
# python:
# install:
# - requirements: docs/requirements.txt
42 changes: 41 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,47 @@

[![CI](https://github.com/BBVA/montest/actions/workflows/ci.yml/badge.svg)](https://github.com/BBVA/montest/actions/workflows/ci.yml)

A stochastic BDD framework for Python designed to test non-deterministic systems (e.g., LLMs) by evaluating statistical distributions rather than binary outcomes.
A stochastic testing framework for Python designed to test non-deterministic systems by evaluating statistical evidence across repeated observations instead of relying on one-shot binary assertions.

## Core install

```bash
pip install montest
```

The base install ships the core stochastic engine only and has no runtime dependencies.

## Quick start

```python
import math
import random

from montest import Decision, SequentialIterator, sprt


def bernoulli_llr(value: int) -> float:
return math.log(0.6 / 0.3) if value else math.log(0.4 / 0.7)


rng = random.Random(42)
criterion = sprt(llr=bernoulli_llr, alpha=0.05, beta=0.10)

for sample in SequentialIterator(lambda: int(rng.random() < 0.6), criterion):
print(sample.index, sample.value, sample.decision.value)
if sample.decision is not Decision.CONTINUE:
break
```

Future testing-tool integrations are intended to live behind optional install groups such as `montest[pytest]` and `montest[behave]`; this release only ships the zero-dependency core.

## Documentation

Build local documentation with:

```bash
task docs
```

## Development Setup

Expand Down
5 changes: 5 additions & 0 deletions Taskfile.yml
Original file line number Diff line number Diff line change
Expand Up @@ -22,3 +22,8 @@ tasks:
TOX_ENV: '{{.TOX_ENV | default "py311"}}'
cmds:
- uv run tox -e {{.TOX_ENV}}

docs:
desc: "Build Sphinx documentation"
cmds:
- uv run sphinx-build -W --keep-going -b html docs docs/_build/html
40 changes: 40 additions & 0 deletions docs/api.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
API reference
=============

Core API
--------

.. autoclass:: montest.Decision
:members:

.. autoclass:: montest.ObservationResult
:members:

.. autoclass:: montest.StoppingCriterion
:members:

.. autoclass:: montest.SequentialIterator
:members:

.. autoclass:: montest.AsyncSequentialIterator
:members:

.. autoclass:: montest.AllOf
:members:

.. autoclass:: montest.AnyOf
:members:

.. autoclass:: montest.CompositeResult
:members:

SPRT
----

.. autoclass:: montest.SPRT
:members:

.. autoclass:: montest.SPRTResult
:members:

.. autofunction:: montest.sprt
81 changes: 81 additions & 0 deletions docs/composition.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,81 @@
Composition
===========

Montest composes criteria explicitly with ``AllOf`` and ``AnyOf``. Composite
keys are owned by the mapping passed to the composite, not by child criteria.

Default resolver
----------------

When a composite reaches a terminal state, the default resolver applies this
order:

.. code-block:: python

if any(decision is Decision.ACCEPT_H1 for decision in decisions):
return Decision.ACCEPT_H1
if any(decision is Decision.INCONCLUSIVE for decision in decisions):
return Decision.INCONCLUSIVE
return Decision.ACCEPT_H0

AllOf
-----

``AllOf`` observes every non-terminal direct child on every sample. It continues
until all direct children are terminal.

Rules:

* already-terminal children are not observed again;
* already-terminal child result entries are ``None`` on later observations;
* ``n_decided`` counts direct children terminal after the current sample;
* ``n_total`` counts direct children;
* the terminal decision is resolved from all direct terminal decisions.

AnyOf
-----

``AnyOf`` observes non-terminal children in mapping order. It stops immediately
when any child returns ``Decision.ACCEPT_H1``.

Rules:

* ``ACCEPT_H1`` short-circuits the composite;
* unobserved or non-terminal child result entries are ``None`` in an early
terminal result;
* ``ACCEPT_H0`` and ``INCONCLUSIVE`` do not short-circuit;
* if no child accepts H1, the composite continues until all direct children are
terminal;
* on terminal output, ``n_decided`` is ``n_total``.

Nested composites
-----------------

Nested composites are stored under their direct mapping key. Their child keys are
not flattened into the parent result.

.. code-block:: python

criterion = AllOf(
{
"primary": sprt(llr=primary_llr),
"secondary-group": AnyOf(
{
"secondary-a": sprt(llr=secondary_a_llr),
"secondary-b": sprt(llr=secondary_b_llr),
}
),
}
)

The parent result has direct keys only:

.. code-block:: python

result.results.keys() == {"primary", "secondary-group"}

Operator overloads
------------------

Montest does not expose ``&`` or ``|`` composition. Use explicit ``AllOf`` and
``AnyOf`` mappings.
51 changes: 51 additions & 0 deletions docs/concepts.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
Core concepts
=============

Observations
------------

A sample is generated by user code and passed to a stopping criterion. The
criterion returns an observation result describing the sample, its index, and the
current decision.

``ObservationResult`` contains:

* ``value``: the observed sample;
* ``index``: the zero-based observation index;
* ``decision``: a :class:`montest.Decision`.

Decisions
---------

``Decision`` has four values:

* ``CONTINUE``: more observations are needed;
* ``ACCEPT_H1``: terminal acceptance of the alternative hypothesis;
* ``ACCEPT_H0``: terminal acceptance of the null hypothesis;
* ``INCONCLUSIVE``: terminal result when sampling stops without enough evidence.

Any decision other than ``CONTINUE`` is terminal for the built-in iterators.

Stopping criteria
-----------------

A stopping criterion implements this protocol:

.. code-block:: python

class StoppingCriterion(Protocol[S, R]):
def observe(self, sample: S, *, index: int) -> R: ...
def reset(self) -> None: ...

``observe`` consumes one sample. ``reset`` restores reusable initial state. The
current criteria raise ``RuntimeError`` if observed after a terminal decision
before ``reset`` is called.

Lifecycle
---------

.. code-block:: text

running --CONTINUE--> running
running --ACCEPT_H1/ACCEPT_H0/INCONCLUSIVE--> terminal
terminal --reset()--> running
37 changes: 37 additions & 0 deletions docs/conf.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
# Copyright 2026 Banco Bilbao Vizcaya Argentaria, S.A.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

from __future__ import annotations

import sys
from pathlib import Path

sys.path.insert(0, str(Path("..", "src").resolve()))

project = "montest"
author = "Banco Bilbao Vizcaya Argentaria, S.A."
copyright = "2026, Banco Bilbao Vizcaya Argentaria, S.A."

extensions = [
"sphinx.ext.autodoc",
"sphinx.ext.autosummary",
"sphinx.ext.intersphinx",
]

html_theme = "alabaster"

autodoc_typehints = "description"
intersphinx_mapping = {
"python": ("https://docs.python.org/3", None),
}
35 changes: 35 additions & 0 deletions docs/index.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
montest documentation
=====================

Montest is a stochastic testing framework for Python. It tests
non-deterministic systems by evaluating statistical evidence across repeated
observations instead of relying on one-shot binary assertions.

Current scope
-------------

The current package ships the zero-dependency stochastic core:

* sequential sampling helpers;
* sync and async iterators;
* typed observation results;
* stopping criteria;
* Wald SPRT;
* explicit ``AllOf`` and ``AnyOf`` composition.

Testing-tool integrations are intentionally not part of the current package.
Future pytest and behave integrations should live behind optional install
groups only after those adapters exist and are tested.

.. toctree::
:maxdepth: 2

installation
quickstart
concepts
sprt
composition
iterators
integrations
testing
api
58 changes: 58 additions & 0 deletions docs/installation.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
Installation
============

Core install
------------

Install Montest with pip:

.. code-block:: bash

pip install montest

The base install has no runtime dependencies. It contains only the stochastic
core: decisions, observation results, stopping criteria, sequential iterators,
SPRT, and explicit composites.

Development setup
-----------------

Install development dependencies with the repository task runner:

.. code-block:: bash

task sync

Common local checks:

.. code-block:: bash

task lint
task typecheck
task test
task docs

Nix development shell
---------------------

If Nix flakes are enabled, enter the dev shell first:

.. code-block:: bash

nix develop

The shell provides ``uv``, ``task``, and the supported Python versions. The same
``task`` commands apply inside the shell.

Documentation tooling
---------------------

Sphinx is a development dependency. It is not a runtime dependency of the base
package.

Future integration extras
-------------------------

pytest and behave integrations are planned but not present. Do not document or
use runnable ``montest[pytest]`` or ``montest[behave]`` install commands until
those adapters exist.
Loading