Is your feature request related to a problem?
The current manual upload form requires users to download case directory data from an HPC filesystem to their local machine, then upload it through the SimBoard upload page.
This adds friction because users must move data through multiple steps before SimBoard can ingest it. It can also be inefficient for large case directories or experiment outputs that already exist on DOE filesystems.
This should be scoped as an exploratory design and prototype task.
Describe the solution you'd like
Add a command-line upload tool that users can run directly on DOE machines, similar to PACE.
The tool would allow users to submit a case directory or experiment archive from the machine where the data already resides, avoiding the need to download data locally and re-upload it through the web UI.
Example usage:
simboard upload /path/to/case-directory
The CLI should support:
- Uploading a case directory or single execution directory from a DOE machine
- Validating required metadata before upload
- Packaging only the data required for SimBoard ingestion
- Authenticating with SimBoard, likely through GitHub OAuth device flow and a SimBoard-scoped upload token
- Supporting a dry-run mode for validation before upload
- Returning clear success or failure messages
- Being installable as a conda package, if feasible
A conda package would make the tool easier to install on shared DOE/HPC systems without requiring users to clone the SimBoard repository or manually manage dependencies. It can also provide a reusable interface that could later be integrated into E3SM workflow tools such as zppy.
Describe alternatives you've considered
One alternative is to let users enter a DOE filesystem path directly in the SimBoard web UI.
This may work for NERSC because the SimBoard backend can access the NERSC performance_archive mount. However, SimBoard does not have direct access to filesystems at other DOE sites.
Supporting this across sites would likely require site-local ingestion services with filesystem access (#154), plus APIs for the central SimBoard backend to communicate with them. That would add deployment, security, networking, monitoring, and maintenance overhead.
A CLI-based upload workflow may be simpler because users run it where the data already exists, while SimBoard continues to ingest through the existing backend.
Additional context
Open questions:
- Should the CLI upload full archives, extracted metadata, or artifact references?
- Should it live in the main SimBoard repo or a standalone package?
- How should it be packaged and distributed, such as conda-forge or an E3SM conda channel?
- How should API URL, machine name, and authentication be configured?
- How should authentication, authorization, token storage, and audit logging work?
- Which DOE machine should be supported first after NERSC?
- How should failed or partial uploads be retried?
Is your feature request related to a problem?
The current manual upload form requires users to download case directory data from an HPC filesystem to their local machine, then upload it through the SimBoard upload page.
This adds friction because users must move data through multiple steps before SimBoard can ingest it. It can also be inefficient for large case directories or experiment outputs that already exist on DOE filesystems.
This should be scoped as an exploratory design and prototype task.
Describe the solution you'd like
Add a command-line upload tool that users can run directly on DOE machines, similar to PACE.
The tool would allow users to submit a case directory or experiment archive from the machine where the data already resides, avoiding the need to download data locally and re-upload it through the web UI.
Example usage:
The CLI should support:
A conda package would make the tool easier to install on shared DOE/HPC systems without requiring users to clone the SimBoard repository or manually manage dependencies. It can also provide a reusable interface that could later be integrated into E3SM workflow tools such as
zppy.Describe alternatives you've considered
One alternative is to let users enter a DOE filesystem path directly in the SimBoard web UI.
This may work for NERSC because the SimBoard backend can access the NERSC
performance_archivemount. However, SimBoard does not have direct access to filesystems at other DOE sites.Supporting this across sites would likely require site-local ingestion services with filesystem access (#154), plus APIs for the central SimBoard backend to communicate with them. That would add deployment, security, networking, monitoring, and maintenance overhead.
A CLI-based upload workflow may be simpler because users run it where the data already exists, while SimBoard continues to ingest through the existing backend.
Additional context
Open questions: