Conversation
Networks are merged into a new network under a new scope with a given name. Task and ProtocolDAGResultRef support is not implemented here.
There is hardly any difference in performance.
Add the client + API surface for the merge_networks database operation introduced in this branch. Users can now combine multiple existing AlchemicalNetworks into a single new AlchemicalNetwork through AlchemiscaleClient, with completed and errored Tasks (and their ProtocolDAGResultRefs) carried over so previously-computed results do not need to be re-run. - POST /networks/merge endpoint validates the destination Scope and each source network's Scope against the caller's token, then defers to Neo4jStore.merge_networks. - AlchemiscaleClient.merge_networks wraps the endpoint and guards against wildcard Scopes, empty input, and non-AlchemicalNetwork ScopedKeys client-side. - Integration tests cover the happy path, scope authorization (both bad destination and bad source), and that Tasks + PDRRs in complete/error state are cloned into the new scope and reachable through get_network_tasks. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replace the inline kc_to_gufe helper and manual chain-slicing loop with a call to KeyedChain.decode_subchains, paired with a list comprehension that captures the original database GufeKeys in the same order. The decode_subchains helper (gufe >=1.8.0) reuses a shared tokenizable_map across yields, so common dependencies in a source network are decoded only once. Lift the previously nested TransformationData dataclass to a private module-level _TransformationData so merge_networks reads top-to-bottom without chasing an inline class definition, and so the helper can be referenced by future tests in isolation. The dropped key_decode_dependencies import goes with kc_to_gufe. subchain_cache is preserved with the same shape but now populated via KeyedChain.from_gufe(transformation); whether the cache (and the unconnected tf_node it feeds) is needed at all is a separate question to revisit once the PERFORMS-wiring question is settled. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
merge_networks now uses KeyedChain.decode_subchains, which was added in gufe 1.8.0. Bump the pin in: - devtools/conda-envs/test.yml (was =1.7.1) so the integration tests exercise the new code path. - devtools/conda-envs/alchemiscale-server.yml (was =1.6.1) so server deployments running merge_networks pick up the required gufe API. Client and compute env files are left alone since neither imports the new code path. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Code review revealed that cloned Tasks on the merged network were
written but never wired back to their Transformations, leaving every
PERFORMS-based traversal (get_network_tasks, get_task_transformation,
set_tasks_status, etc.) blind to them. The new
test_merge_networks_preserves_tasks_and_results check via
get_network_tasks asserts this exact thing and would have failed in
CI.
Changes
-------
- Add the PERFORMS edge in _TransformationData.to_subgraph so cloned
Tasks are reachable from the standard
(:AlchemicalNetwork)-[:DEPENDS_ON]->(:Transformation)<-[:PERFORMS]-(:Task)
traversal. The pre-existing tf_node construction is now meaningful,
resolving the open question about subchain_cache.
- Add a `state` parameter to Neo4jStore.merge_networks,
/networks/merge, and AlchemiscaleClient.merge_networks, bringing
merge_networks to feature parity with assemble_network /
create_network. Defaults to NetworkStateEnum.active.
- Document the "cloned Tasks are not actioned to the merged TaskHub"
semantics in the store-level and client docstrings, including the
remediation path (call action_tasks after merging).
- Replace the O(N^2) `transformation_data.index(transformation)`
dedup with a `dict[GufeKey, _TransformationData]` keyed on
transformation.key, avoiding repeated GufeTokenizable equality
checks on large networks.
- Improve the missing-ScopedKey error to collect and report all
missing source SKs in a single ValueError, rather than only the
one that first triggered KeyError.
- Add a chain-order contract comment at the
zip(database_keys, transformations) site pointing at the
decode_subchains yield-order guarantee.
- Make /networks/merge an `async def` endpoint, matching the style
of the neighboring create_network endpoint.
- Add a parametrized client test (state in {active, inactive}) that
exercises the new state parameter end-to-end.
- Drop the unused network_tyk2 fixture from the two API-level
authorization tests.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Member
Author
|
Thanks for the careful review. Pushed Blocking
Should-fix
Nits picked up
Deferred
Sanity check the reviewer asked for Static analysis confirms the PERFORMS edge is now emitted inside the per-task loop with the correct direction ( |
merge_networks now depends on KeyedChain.decode_subchains (gufe 1.8.0). Pin all five env files to an exact 1.8.0 release, matching the repo's convention of pinning gufe exactly rather than with a lower bound. Touches the test, server, compute, client, and docs envs. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
feflow 0.2.1 hit conda-forge and pins gufe >=1.9.0,<2 plus openfe >=1.10.0,<2, which means our previous gufe=1.8.0 + openfe=1.8.0 + feflow>=0.1.4 combination is unsolvable: feflow 0.2.0 caps gufe at <1.8, and feflow 0.2.1 forces gufe>=1.9.0. Bump the trio everywhere to the latest mutually-compatible exact pins: - gufe = 1.10.0 (latest stable; satisfies openfe 1.11.1's gufe >=1.10.0,<1.11) - openfe = 1.11.1 (latest stable; satisfies feflow 0.2.1's openfe >=1.10.0,<2) - feflow = 0.2.1 (latest stable) merge_networks only needs KeyedChain.decode_subchains (gufe >=1.8.0), so the bump is API-safe. Apply across test, server, compute, and client env files. docs.yml gets the gufe bump only (no openfe/feflow there) and gains an explicit `python >=3.11,<3.13` lower bound because gufe 1.10.0 requires python >=3.11; without the lower bound, ReadTheDocs's mambaforge-4.10 resolved to a python too old for gufe and the docs build failed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
CI ran the env solve and tests cleanly with the gufe/openfe/feflow trio bump (b63474e), revealing two real bugs: 1. The four merge_networks client tests merged into destination scopes the test identity does not have access to (e.g. `test_org-test_campaign-merged_project`). The test identity only has the three scopes in `multiple_scopes` (scope_test plus two others); creating a "new project" under the same org/campaign was not, in fact, authorized. Switch the destination scope to one the identity has: - test_merge_networks: scope_test (collision-free because the merged network's name differs from pre-loaded networks) - test_merge_networks_respects_state: scope_test (each parametrize case uses a unique network name) - test_merge_networks_preserves_tasks_and_results: multiple_scopes[1] (different from where the source Tasks were set up, so the per-`_project` Task/PDRR counts stay clean) 2. The docs.yml gufe bump from 1.3.0 to 1.10.0 breaks the RTD build, because sphinx 9's autodoc dynamic-importer trips on `gufe/settings/models.py`: ph: PositiveFloat | None = Field(None, ...) ~~~~~~~~~~~~~~^~~~~~ TypeError: unsupported operand type(s) for |: 'PositiveFloat' and 'NoneType' gufe 1.3.0 still allows pydantic v1 (where PositiveFloat is a class with `__or__`); gufe >=1.8.0 forces pydantic v2 (where PositiveFloat is `Annotated[float, ...]`, no `__or__`). docs.yml only needs gufe for intersphinx and type-hint resolution; neither benefits from the bump. Revert docs.yml to its pre-PR state. Also polish the client merge_networks docstring with a `set_tasks_status` hint alongside `action_tasks` for the errored-Task remediation path. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Closes the remaining surface area in #221 (copy_network and merge_scopes) while fixing a latent bug in merge_networks that the previous CI run surfaced via test_merge_networks_preserves_tasks_and_results. Root cause of the PERFORMS bug ------------------------------ In `storage/subgraph.py` `merge_subgraph` (lines 90-108), when two Python Node objects in the same Subgraph share a primary key, the UNWIND-MERGE Cypher query returns one Neo4j elementId for both, but the result-iteration loop assigns that elementId only to the first Python Node. The duplicate keeps `identity=None`, so any Relationship referencing it gets `start/end_node.identity = None` and is silently dropped at commit. merge_networks hit this directly: 1. The AlchemicalNetwork's KeyedChain produced one Transformation Node per edge in an_subgraph. 2. `_TransformationData.to_subgraph` produced a *second* Python Transformation Node from `subchain_cache` -- same `_scoped_key`, different Python object -- and attached PERFORMS to it. 3. After `an_subgraph |= to_subgraph_result`, merge_subgraph kept the AN-chain Node identified but left the to_subgraph Node with no identity. Every PERFORMS edge was silently dropped on commit. Fix: `to_subgraph` now takes the *existing* tf_node from the surrounding subgraph as a parameter. merge_networks (and the new copy_network) build a `{gufe_key: node}` lookup from `an_subgraph` after constructing it, and pass the matching tf_node into each call. This eliminates the `subchain_cache` entirely, which had been left as an open question from the earlier review round; `tf_node`'s purpose finally has teeth as the genuine PERFORMS anchor. task_statuses knob on _TransformationData.update_task_trees ----------------------------------------------------------- The previous query hard-filtered `WHERE task.status IN ["complete", "error"]`. That filter is right for merge_networks ("carry over results, not in-flight work") but wrong for copy_network, which the issue specifies as carrying over *all* Tasks. Add an optional `task_statuses: list[str] | None = None` parameter to drop the WHERE clause when None. merge_networks passes `["complete", "error"]` explicitly; copy_network passes None. copy_network ------------ - `Neo4jStore.copy_network(network_scoped_key, scope, name=None, state=NetworkStateEnum.active)`: decodes the source AN, optionally renames (preserving the gufe key when name is unchanged), clones every Task with its PDRRs and EXTENDS edges, and writes the new network + NetworkMark + TaskHub into the target scope. Reuses `_TransformationData` machinery verbatim, just with the task_statuses filter dropped. - `POST /networks/{network_scoped_key}/copy` endpoint validates the source ScopedKey, source scope, and destination scope against the caller's token, then delegates to the store. - `AlchemiscaleClient.copy_network(network, scope, name=None, state, visualize)` mirrors create_network / merge_networks shape. Cloned Tasks are intentionally **not** actioned to the new network's TaskHub, matching merge_networks; the client docstrings spell out the action_tasks remediation path for callers who want errored or waiting Tasks back in the compute queue. merge_scopes ------------ - `AlchemiscaleClient.merge_scopes(scopes, target_scope, visualize)` is a pure client-side loop: query each source scope for every AlchemicalNetwork, then copy_network each into target_scope. No new endpoint required. Tests ----- - test_copy_network: stages 4 Tasks (complete + ok PDRR, error + not-ok PDRR, waiting, running) on three Transformations of a source network; copies into a different scope; asserts that all four Tasks are reachable via the standard `get_network_tasks` traversal with statuses preserved, both PDRRs land in the target scope with obj_keys intact, and gufe key is preserved when name is unchanged. - test_copy_network_with_rename: confirms passing a new name produces a distinct gufe key. - test_copy_network_rejects_wildcard_scope / test_copy_network_rejects_non_network_scoped_key: client-side guards. - test_merge_scopes: copies every network in two source scopes into a third; asserts per-source ScopedKey count, target-scope ScopedKey collapse (preloaded fixtures use the same network names across scopes, so copies dedupe), and gufe-key preservation. - test_merge_scopes_rejects_empty / _rejects_wildcard_target: client-side guards. news/issue-221.rst expanded to list all three new methods plus the merge_networks-only complete|error filter semantic and the no-ACTIONS policy. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
dotsdl
changed the title
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #507 +/- ##
==========================================
- Coverage 80.54% 80.42% -0.13%
==========================================
Files 31 31
Lines 4877 5087 +210
==========================================
+ Hits 3928 4091 +163
- Misses 949 996 +47 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
3 tasks
The previous to_subgraph emitted
Relationship.type("EXTENDS")(etask_node, task_node, ...)
which creates ``(etask)-[:EXTENDS]->(task)``, i.e. "extended_task
extends task" -- the opposite of what the source graph encoded:
OPTIONAL MATCH (task)-[:EXTENDS]->(extended_task:Task)
and what every other place in the store writes
(``Neo4jStore.create_tasks`` at line 3424:
``Relationship.type("EXTENDS")(task_node, extends_task_node, ...)``)
and reads (lines 2392, 3152, 3547, 3634, 4219, all traversing
``(task)-[:EXTENDS]->(other_task)``).
No existing test exercised the cloned EXTENDS direction (the store-
level test_merge_networks only counted PDRRs, and my own client tests
used independent Tasks with no EXTENDS chain), so the bug shipped
silently as part of the original PR's inline to_subgraph. Swap the
argument order and add a focused regression test:
- test_copy_network_preserves_extends_direction stages a base
complete Task plus an extending complete Task on the same
Transformation (using ``create_tasks(transformation_sks,
extends=base_task_sks)``), copies the source network into a
different scope, then asserts via raw Cypher that exactly one
EXTENDS edge exists in the target scope with
``extender == extending_task.gufe_key`` and
``extended == base_task.gufe_key`` -- i.e. the direction matches
the source.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The previous fix landed the correct EXTENDS direction but the test_copy_network_preserves_extends_direction regression test still failed (assert 0 == 1) -- the cypher MATCH returned no EXTENDS edges at all on the clone. Same Node-deduplication trap as the AN-chain PERFORMS bug, this time intra-to_subgraph. When ``record[i]["task"]`` is the same Task as ``record[j]["extended_task"]`` (the common case: an extending Task's target also appears as its own complete/error record), the previous code created two separate Python ``Node`` objects from ``record_to_node``: one when processing ``record[i]`` (named ``task_node``), one when processing ``record[j]`` (named ``etask_node``). Both have the same ``_scoped_key``. ``merge_subgraph`` assigns a Neo4j ``elementId`` only to the first Python Node it sees per primary key, so the EXTENDS Relationship's ``end_node.identity`` was ``None`` and the edge was silently dropped on commit. Memoize Task Nodes by their gufe key inside to_subgraph via ``task_node_cache: dict[str, Node]`` and a small ``get_or_create_task_node`` helper; every reference to the same Task across iterations now reuses the same Python Node and lands a real Neo4j identity. The cache is local to a single ``to_subgraph`` call so it does not leak across Transformations. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
dotsdl
added a commit
that referenced
this pull request
Jul 2, 2026
…#512) feflow 0.2.1 hit conda-forge and now pins gufe >=1.9.0,<2 and openfe >=1.10.0,<2. The existing pins (gufe=1.6.1 / openfe=1.6.1 / feflow=0.1.4 across the deployment envs, and gufe=1.7.1 / openfe=1.8.0 in test) leave callers unable to use feflow 0.2.1 against an alchemiscale env. Bump the trio to the latest mutually-compatible exact pins: - gufe = 1.10.0 (satisfies openfe 1.11.1's gufe >=1.10.0,<1.11) - openfe = 1.11.1 (satisfies feflow 0.2.1's openfe >=1.10.0,<2) - feflow = 0.2.1 (latest stable) Apply across the four runtime env files: test, server, compute, client. devtools/conda-envs/docs.yml is intentionally unchanged -- the docs build runs sphinx 9's autodoc, which trips on the `PositiveFloat | None` annotation in gufe >=1.8.0 (works with pydantic v1 in gufe 1.3.0; pydantic v2 in gufe 1.10.0 makes PositiveFloat an `Annotated[float, ...]` with no `__or__`). docs.yml only needs gufe for intersphinx and type-hint resolution, so leaving it at 1.3.0 keeps the docs build green; addressing the docs gufe bump is a separate concern from this maintenance bump. Extracted from PR #507 to unblock other inflight PRs needing the newer dependency set. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Neither merge_networks nor copy_network carries over the source network's execution-orchestration state -- ``Strategy`` (via the ``PROGRESSES`` relationship on the source AN) and ``TaskRestartPattern``\s (via ``ENFORCES`` on the source TaskHub and ``APPLIES`` on individual Tasks). Both create a fresh empty TaskHub via ``create_taskhub_subgraph`` and stop there. This is intentional: Strategy and TRPs govern *how* Tasks run (compute planning, retry policy) rather than the results themselves, which are what merge_networks and copy_network are designed to preserve. Merge_networks in particular has no natural resolution across N sources' Strategies or TRPs; copy_network is skipping them by design for consistency. Callers wanting either on the new network should set them explicitly after the merge/copy via ``AlchemiscaleClient.set_network_strategy`` and ``AlchemiscaleClient.add_task_restart_patterns``. Adds the policy statement to: - ``Neo4jStore.merge_networks`` docstring - ``Neo4jStore.copy_network`` docstring - ``AlchemiscaleClient.merge_networks`` docstring - ``AlchemiscaleClient.copy_network`` docstring - news/issue-221.rst (as an additional bullet under Added) No code change; no tests added, since the behavior is "does not do X" and is already exercised implicitly by the existing tests (which would fail if PROGRESSES/ENFORCES/APPLIES were being silently duplicated across scopes). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a "Merging and copying AlchemicalNetworks" section to getting_started.rst covering the three new client methods, their Task carry-over semantics, and the non-preservation policy for actioning, Strategy, and TaskRestartPatterns. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Explicitly verify that copying an AlchemicalNetwork into a scope where the same AlchemicalNetwork already exists collapses onto the existing node rather than creating a duplicate. The additive-attach semantics this test exercises depend on that dedup. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Existing merge_scopes coverage only exercised the pure-dedup case, where every source AlchemicalNetwork was already preloaded into the target scope with the same gufe key. The new test adds a third AlchemicalNetwork to the source scope only, then asserts that after merge_scopes the target scope holds it as a genuinely new node (alongside the 2 that dedup onto preexisting entries). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The interface-level test_copy_network* tests all start from
n4js_preloaded, which seeds the same networks (and therefore the same
Transformations) into every scope. Every copy under that fixture is
either pure dedup or pure fresh-write, never a mix.
The new store-level test seeds a target scope with only a *subset* of
the source's Transformations, then copies the full source into it.
Post-copy assertions verify:
- the target ends up with exactly one AN node for the copied
ScopedKey;
- the target's Transformation count equals the source's (overlap
subset dedups, non-overlap subset is created fresh);
- every source Transformation appears exactly once in the target;
- every cloned Task is wired via PERFORMS to a target-scope
Transformation regardless of which side of the overlap it lives on;
- EXTENDS chains are preserved across the copy;
- both source PDRRs land in the target scope with obj_keys intact.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Revives the work originally proposed in #433 (now unreachable since the contributing fork was deleted) and finishes it.
Neo4jStore.merge_networksfor combining multiple existingAlchemicalNetwork\s into a newAlchemicalNetworkin a targetScope, with completed and erroredTask\s (and theirProtocolDAGResultRef\s) cloned into the new scope so previously-computed results do not need to be re-run. (Originally authored by @ianmkenney; preserved here as-is, then refactored — see commits.)POST /networks/mergeonAlchemiscaleAPI, which validates the destinationScopeand each source network'sScopeagainst the caller's token before delegating to the store.AlchemiscaleClient.merge_networks(networks, name, scope), with client-side guards for wildcard scopes, empty input, and non-AlchemicalNetworkScopedKey\s.KeyedChain.decode_subchains(gufe ≥1.8.0), which replaces a hand-rolledkc_to_gufehelper plus manual chain-slicing loop, and lifts the inlineTransformationDatadataclass to a private module-level_TransformationDatafor readability.gufepin to>=1.8.0indevtools/conda-envs/test.ymlanddevtools/conda-envs/alchemiscale-server.yml(the only env files that actually exercise this code path).complete/errorTask\s with theirProtocolDAGResultRef\s — including a reachability check viaget_network_tasks(merged_sk).Closes #221.
Notes for reviewers
merge_networksbody is unchanged in behavior from the original PR; the diff against Allow server-side copying and merging of AlchemicalNetworks #433's last revision is thedecode_subchainsrefactor plus the lift ofTransformationData.get_network_tasksassertion intest_merge_networks_preserves_tasks_and_resultsexercises the standard(:AlchemicalNetwork)-[:DEPENDS_ON]->(:Transformation)<-[:PERFORMS]-(:Task)traversal on the merged network. If it fails, that indicates a missingPERFORMSedge in_TransformationData.to_subgraph— flagged here for follow-up rather than fixed in this PR, since the original implementation has the same behavior.subchain_cacheis preserved with cleaner contents (KeyedChain.from_gufe(transformation)) but its necessity is an open question tied to thePERFORMSfollow-up above.Test plan
pytest -n auto -v alchemiscale/tests/integration/storage/test_statestore.py -k merge_networkspytest -n auto -v alchemiscale/tests/integration/interface/test_api.py -k merge_networkspytest -n auto -v alchemiscale/tests/integration/interface/client/test_client.py -k merge_networksblack --check --diff alchemiscalemypy alchemiscaletest_merge_networks_preserves_tasks_and_resultsreveals the suspected missingPERFORMSwiring; open a follow-up if so.🤖 Generated with Claude Code