Store optimization data using blobs

[frode-aarstad]

Issue
Resolves #13901

Approach
Short description of the approach

(Screenshot of new behavior in GUI if applicable)

• PR title captures the intent of the changes, and is fitting for release notes.
• Added appropriate release note label
• Commit history is consistent and clean, in line with the contribution guidelines.
• Make sure unit tests pass locally after every commit (git rebase -i main --exec 'just rapid-tests')

When applicable

• When there are user facing changes: Updated documentation
• New behavior or changes to existing untested code: Ensured that unit tests are added (See Ground Rules).
• Large PR: Prepare changes in small commits for more convenient review
• Bug fix: Add regression test for the bug
• Bug fix: Add backport label to latest release (format: 'backport release-branch-name')

[codecov-commenter]

Codecov Report

❌ Patch coverage is 98.24561% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 91.39%. Comparing base (ed77de3) to head (6fafe67).

Files with missing lines	Patch %	Lines
src/ert/storage/migration/to37.py	96.66%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #13948      +/-   ##
==========================================
- Coverage   91.40%   91.39%   -0.01%     
==========================================
  Files         481      482       +1     
  Lines       34213    34255      +42     
==========================================
+ Hits        31272    31309      +37     
- Misses       2941     2946       +5     

Flag	Coverage Δ
cli-tests	35.93% <10.52%> (-0.04%)	⬇️
fuzz	43.56% <28.07%> (-0.02%)	⬇️
gui-tests	59.16% <10.52%> (-0.07%)	⬇️
performance-and-unit-tests	79.49% <89.47%> (+0.02%)	⬆️
test	46.07% <47.36%> (+0.04%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
src/ert/storage/blob_data.py	100.00% <100.00%> (ø)
src/ert/storage/local_ensemble.py	96.84% <100.00%> (+0.03%)	⬆️
src/ert/storage/local_storage.py	92.90% <100.00%> (ø)
src/ert/storage/migration/to37.py	96.66% <96.66%> (ø)

... and 1 file with indirect coverage changes

[codspeed-hq]

Merging this PR will not alter performance

✅ 36 untouched benchmarks

---

_{Comparing frode-aarstad:everest-data-in-blobs (6fafe67) with main (ed77de3)}

[Copilot]

Pull request overview

This PR updates ERT’s local storage format to store Everest optimization “batch” parquet dataframes as ensemble blobs (instead of loose *.parquet files under the ensemble directory), including a migration to move existing batch parquet files into blob storage and tests validating the new behavior (including dark-storage HTTP access).

Changes:

• Add a new blob metadata type (EVEREST_BATCH_DATA / EverestBatchData) and persist Everest batch dataframes as blobs via LocalEnsemble.save_batch_dataframes().
• Introduce local-storage schema version 36 and a to36 migration that moves legacy batch_*.parquet files into ensembles/<ens>/blobs/.
• Extend unit tests to cover blob persistence, migration behavior, and dark-storage /ensembles/{id}/blobs + /blobs/{uri} endpoints for Everest batch data.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
tests/ert/unit_tests/storage/test_local_storage.py	Adds tests that batch dataframes are saved as Everest batch blobs and that missing batch data returns None.
tests/ert/unit_tests/storage/migration/test_to36.py	Adds migration tests ensuring batch parquet files are moved into blob storage and non-batch parquet files remain untouched.
tests/ert/unit_tests/dark_storage/test_http_endpoints.py	Adds endpoint tests for listing Everest batch dataframe blobs and fetching parquet bytes via blob URI.
src/ert/storage/migration/to36.py	Implements the v36 migration that moves batch_*.parquet into blobs/ with Everest batch blob metadata.
src/ert/storage/local_storage.py	Bumps local storage version to 36 and wires in the to36 migration.
src/ert/storage/local_ensemble.py	Writes Everest batch dataframes into blob storage and reads them back from blobs via new helper logic.
src/ert/storage/blob_data.py	Adds the EVEREST_BATCH_DATA discriminator and EverestBatchData model to the blob metadata union.

[Copilot]

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.

[xjules]

I'm wondering here if we should have the same pattern like in ert?
Each of the update algorithm are not storing the data, but just send a dedicated event to a common broker (update_run_model) - which then internalizes the data.
Not sure how doable that is or possible at all.

[frode-aarstad]

I think we should create a new issue for this

[ertomatic]

Screenshots differ from baselines. A baseline update PR has been prepared: equinor/ert-testdata#49