[DevOps]: Explore simple cron usage for data collection on Chrysalis

[tomvothecoder]

Task Details

SimBoard utilizes a simple combination of a Python environment, a Python collection and ingestion script, and a bash wrapper for the remote upload process for ingestion.

One suggested route was deploying this process on the E3SM Sandia Jenkins infrastructure, but it might be overkill and less efficient to deploy right now since changes may need to be made quickly in the SimBoard ingestion workflow.

@TonyB9000 suggested exploring how to this SimBoard job using a cron process on Chrysalis as it is simpler and allows us to iterate more quickly without bugging Wade or pushing things onto the e3sm-test repo.

Steps to Complete

• Reach out to the POC for Chrysalis on how to deploy a cron job
  • Azamat Mametjanov (https://e3sm.atlassian.net/wiki/spaces/IPD/overview#Machine-POCs)
  • e.g., does it require sudo privileges, can it be done with the e3sm-test account, etc.
• Attempt cron job deployment if possible

Additional Notes

No response

[TonyB9000]

@tomvothecoder Almost all reasons for placing a job under s given crontab account involve inheriting the permissions and default environment of that account. One other reason would be persistence - you would not want a system process cron'd under a summer student's crontab account, if that student (and their account) disappear after the summer.

ANY user with an account has their own crontab, and can schedule runs as they please. These entries persist after reboots, etc.

As simboard does not appear to require any root or "sudo -u root" permissions to do its job (all ingestion directories are readable), the choices are one of two for "persistence":

1. Create a "simboard" user and group account, and always run under those credentials. You would have a consistent environment and permissions, and simboard would have its own crontab. Specific users can be given the right to "sudo simboard" and edit the simboard crontab, do git-pull to update the "installed" code etc.
2. Pick another "stable" account (e.g., e3sm-test), and then engage that account owner whenever something needs modification. That would include not only schedule changes, but also "git pull" and related activity.

I would go with (1). This requires each machine's system admin to create a "simboard" user account, and a "simboard-admins" group, initially with YOU as the founding member.

According to LivChat ... Here is the layout we should follow:

Recommended pattern
1) Bootstrap once as root

System admins create:

    simboard service account
    simboard-admins group
    a limited sudoers policy, if needed

Thereafter, a trusted admin (member of the simboad-admins group) can "install" or update the code by running:

    sudo -u simboard git clone <repo-url> /path_to/simboard/app
and 
    sudo -u simboard git -C /path_to/simboard/app pull

For the cron job, a simboard-admin can issue

    sudo -u simboard crontab -e
    (edit desired crontab entry)

The cron'd script can then issue

    cd /path_to/simboard/app && python -m yourmodule

This design only (further) "bothers" the machine sys-admins when a user must be added or removed from the "simboard-admins" group, and if finer control is desired, from the simboard "sudoers" file.

[TonyB9000]

@tomvothecoder @chengzhuzhang For your consideration:

SimBoard (core DB) runs at NERSC. It is intended to collect simulation metadata output from multiple machines, the primary generation channel being zppy output, and mitigated (interleaved) with PACE post-processing. (As PACE moves materials, simboard necessarily must be prepared to process both pre-PACE “staging” and post-PACE “archive” execution directories).

Where remoteness implies that data collection be routinely shipped to the NERSC SimBoard backend for ingestion, cron jobs will be established on these remote systems to:

1. Ensure the latest released collection codes are used (pulled from git repository)
2. Run these codes with consistent application environment variable values

Uniformity, consistency, security and persistence demand that each such system have a “simboard” account, together with a “simboard-admins” group whose members are allowed to issue “sudo -u simboard” commands. This will allow the local simboard repository and runtime directories to be “simboard owned and operated”, and for simple scheduling to be accomplished (as the simboard account will have its own personal crontab file). The sys admins for each remote machine/system will need to create the simboard service account, the simboard-admins group, and populate that group with the appropriate simboard operators.

By this scheme, we reduce the need to involve varied sys-admins whenever simboard updates or scheduling are changed. Only the future addition or removal of personnel from the simboard-admins group should require their involvement.

RESIDUAL QUESTIONS

A.	Sites, Systems and Machines:  Which of these require a separate “simboard” collector instance?
    a.	NERSC/(Perlmutter, etc)
    b.	Argonne/(Chrysalis, etc)
    c.	PNNL/(Compy, etc)
    d.	etc
B.	Sys-Admins:  Who are they?
C.	When to establish?

Most of this can, and should be set-up in advance of SimBoard deployment, so that deployment gains the established consistency from the outset.

I would be happy to contact the respective administrators to have these accounts established. As SimBoard (collection) requires no special privileges, it should be trivial to add these accounts.

[chengzhuzhang]

Thanks @TonyB9000 for laying this out. the service account + simboard-admins group approach is the right model, and it probably match what we already do elsewhere in the project: PACE data collection and E3SM automated testing both rely on service accounts running scheduled jobs (cron/Jenkins) on the target machines. SimBoard should follow the same mechanism. Maybe @rljacob can help us to use or be added to these service account?

[tomvothecoder]

@chengzhuzhang @TonyB9000 @rljacob As an alternative to system-level cron and accounts, I wonder if we can use containers to easily deploy these across different machines. Something worth looking into.

[TonyB9000]

@tomvothecoder The above plan does not involve "system-level" accounts - as they are not needed. We simply create an ordinary user-level account called "simboard" to own and operate collection. We don't even need to log-in as "simboard" (no sharing passwords) because we belong to a simboard "sudoers" file, and can issue "sudo -u simboard" commands.

Just to clarify.

[TonyB9000]

@tomvothecoder Yes, containers are a mechanism one can use. How does this change management as far as ownerships, authorities, scheduling. When code is updated - what process (and owner) creates and deploys the new containers? When scheduling is to be adjusted, what authority is employed? What part of the file-space is used, and who owns it? I am sufficiently unfamiliar with containers that I can't answer this.

[TonyB9000]

@tomvothecoder After Rob's directive on scheduling SimBoard collection at Chrysalis, to wit:

"Tony, you should run all that under your user id. If you leave the project, we transfer it to someone else. "

I conducted a small test.

1. I used "crontab -e" to add this solitary line to my crontab file:

0,5,10,15,20,25,30,35,40,45,50,55 * * * * /home/ac.bartoletti1/Ops/simboard/launch_crontest.sh

This crontab entry attempts to run the "launch_crontest.sh" script every 5 minutes. The script specifies:

#!/usr/bin/env bash
set -euo pipefail

workdir=/home/ac.bartoletti1/Ops/simboard
LOCK_FILE="$workdir/myjob.lock"
LOG_FILE="$workdir/myjob.log"

exec 200>"$LOCK_FILE"
if ! flock -n 200; then
  echo "[$(date -Is)] SKIP, lock already held, pid $$" >> "$LOG_FILE"
  exit 0
fi

cleanup() {
  echo "[$(date -Is)] Job exiting, pid $$" >> "$LOG_FILE"
}
trap cleanup EXIT

ts=`date -Is`
echo "LAUNCHING crontest at $ts" >> "$LOG_FILE"
exec /home/ac.bartoletti1/Ops/simboard/do_sleep_420.sh

The purpose is to configure and launch the script at the bottom, "do_sleep_420.sh" (a stand-in for "chrysalis.sh"), which deliberately sleeps for 7 minutes and then exits. At start, it opens an I/O channel "200" (arbitrary) which the system will retain as long the "exec'd" process is active. While active, no other process can open an "I/O channel 200", so a premature restart will fail and exit.

This "job" was running over 15 hours, survived log-off, and as you can see from the log lines below, "aborts" every other time it is launched, because 5 minutes < 7 minutes.

. . .
LAUNCHING crontest at 2026-07-22T09:45:01-05:00
[2026-07-22T09:50:01-05:00] SKIP, lock already held, pid 2562971
LAUNCHING crontest at 2026-07-22T09:55:01-05:00
[2026-07-22T10:00:01-05:00] SKIP, lock already held, pid 2567156
LAUNCHING crontest at 2026-07-22T10:05:01-05:00
[2026-07-22T10:10:01-05:00] SKIP, lock already held, pid 2570609
LAUNCHING crontest at 2026-07-22T10:15:02-05:00
[2026-07-22T10:20:02-05:00] SKIP, lock already held, pid 2572907

I intend to replace "launch_crontest.sh" with a "launch_simboard_collection.sh" as a further test. It will "cd" to my "gitrepo/simboard/backend" and launch the "chrysalis.sh" script in place of "do_sleep_420.sh".

Also, I will modify the crontab entry to run 4 times per hour, offset 5 minutes (5, 20, 35, 50).

[TonyB9000]

@tomvothecoder I am using cron with this line:

5,20,35,50 * * * * /home/ac.bartoletti1/Ops/simboard/launch_simboard_collection.sh staging
# 0 14 * * * /home/ac.bartoletti1/Ops/simboard/launch_simboard_collection.sh archive

This collection-launch script will exit with an ERROR if neither "staging" nor "archive" is supplied on the command line. This way, a single collection launch script can be cron'd with different schedule, parameters, etc.

The script does launch "chrysalis.sh" successfully, but (of course) fails to fetch the "state" because the NERSC backend is currently offline.

I was about to push my "launch_simboard_collection.sh" script to ingestion/sites, when it occurred to me that it poorly separates "site-agnostic" materials from the site-specific. I intend to tabulate this distinction and then ensure they are treated separately.

[TonyB9000]

@tomvothecoder In an attempt to better rationalize the remote collection configuration, I began to itemize each component (environment to activate, ENV variables to instantiate, etc) and for each, determine if it is "site-agnostic" (the same value will suffice for all sites) or "site-specific" (need a potentially custom value or file per site). I soon realized that each such component has two aspects, the value itself, and the means or path by which it will be located. (This would indeed be simplified with a containerized approach!,)

For instance, the "SIMBOARD_API_TOKEN" is site-agnostic. But we don't want it held on a common server, and at run-time it must be accessed locally. Can the "path" or "method" be site-agnostic? With my Chrysalis testing, I have placed the value (the "export" command to be sourced) in a hidden file. The method of access can be common (the site-unique script contains the path to the hidden command file), but the precise path will be site-specific. Even here, if we agree that such path be fixed relative to the operator's "$HOME", we can have site-specific paths (each HOME may be different) but the script securing access can be common across different sites.

I am working on a thorough breakdown on this topic, and will post my assessment here (for want of a better place).

[tomvothecoder]

@TonyB9000 Thanks for the update Tony! I'm looking forward to hearing more about your findings.

[TonyB9000]

@tomvothecoder OK, I think I have a working system now. I have utilities that wipe, set and dump the whole set of simboard-related environment variables, and used them to test this system extensively, and everything now checks out.

First Principles: We want the "backend" git repository to host maximum generic (non-site-specific) code, and I have reduced this to the point that site-specific materials are contained in site-maintained file "<sitename>.config" (that gets installed as "$HOME/.simboard.config" to support uniform generic access. SItes (or simboard admins) need only update the corresponding config file and not touch the codes to make changes.

In this scheme, the repository ( . . . backend/app/scripts/ingestion/sites/) contains the following:

    simboard_pa_collection.sh    (generic)
    chrysalis.config             (site-specific)
    compy.config                 (site-specific)
    perlmutter.config            (site-specific)
    (etc)

I have added some variables and renamed some for specificity. The variables that are "generic" (site-agnostic) appear and are set in the generic simboard_pa_collection.sh script. All others are obtained within by sourcing the site configuration file, with minor exception described below.

Example: The file "chrysalis.config" contains the following lines:

    export SIMBOARD_WORKDIR="${HOME}/Ops/simboard"
    export SIMBOARD_REPODIR="${HOME}/gitrepo/simboard/backend"
    export SIMBOARD_ENV_CMD="source $HOME/envs/test_simboard/bin/activate"
    export SIMBOARD_API_TOKEN_CMD="$SIMBOARD_WORKDIR/.api_token_export"
    export DRY_RUN="${DRY_RUN:-true}"
    export PERF_ARCHIVE_ROOT="/lcrc/group/e3sm/PERF_Chrysalis/performance_archive"
    export OLD_PERF_ARCHIVE_ROOT="/lcrc/group/e3sm/PERF_Chrysalis/OLD_PERF"

(I would prefer any override of most of these to be changed in the config here, rather than simply be overridden by a command-line export.)

This configuration allows each site to decide where to have the crontab make the "launch directory" for their operations, where to find the local simboard repository "backend" for the final collection launch, what command activates the desired (conda, venv, pixi) environment, and how to obtain the API token without revealing its value.

The variable SIMBOARD_ENV_CMD is designed to accommodate cases both where one might simply issue "activate ", and those where more elaborate activations are required.

The remaining variables are:

    SCAN_MODE                (set by the launch script via the command line argument)
    SIMBOARD_API_BASE_URL     (fixed and set in the launch script)
    MAX_CASES_PER_RUN      (assumed to be controlled by the central backend ingestion)

The Launch Script:

Here is the generic launch script, cron'd as follows

    <schedule_one>  <path_to>/simboard_pa_collection.sh staging
    <schedule_two>  <path_to>/simboard_pa_collection.sh archive

It ensures that one of "staging" or "archive" is specified, sources the local site-config material, sets up process lock and logging.

[simboard_ps_collection.sh]
=====================================================================
#!/usr/bin/env bash
set -euo pipefail

if (( $# < 1 )) || [[ $1 != "archive" && $1 != "staging" ]]; then
    echo "[$(date -Is)] ERROR: Must indicate \"staging\" or \"archive\" on the command line"
    exit 1
fi

scan_mode=$1
export SCAN_MODE="$scan_mode"
if [[ $scan_mode == "archive" ]]; then
  export ARCHIVE_YEAR_START="${ARCHIVE_YEAR_START:-2025-01}"

export MAX_CASES_PER_RUN=50

# Obtain user's simboard github repository, operations directory, and work environment:
#  (SIMBOARD_REPODIR, SIMBOARD_WORKDIR, SIMBOARD_ENV_CMD, SIMBOARD_API_TOKEN_CMD, )
site_config=$HOME/.simboard.config
source $site_config
echo "Issuing SIMBOARD_ENV_CMD = $SIMBOARD_ENV_CMD"
$SIMBOARD_ENV_CMD

export PYTHON_BIN="${PYTHON_BIN:-${SIMBOARD_REPODIR}/.venv/bin/python}"
if [[ ! -x "${PYTHON_BIN}" ]]; then
  echo "Expected Python interpreter at ${PYTHON_BIN}" >&2
  echo "Run 'make install' from the repository root to create it." >&2
  exit 1
fi

export SIMBOARD_API_BASE_URL="https://simboard-dev-api.e3sm.org"
source $SIMBOARD_API_TOKEN_CMD
: "${SIMBOARD_API_TOKEN:?SIMBOARD_API_TOKEN failed to be set.}"

LOCK_FILE="$SIMBOARD_WORKDIR/SBCS.lock"
ts=`date -u +%Y%m%d_%H%M%S`
LOG_FILE="$SIMBOARD_WORKDIR/SBCS-$ts.log"

exec 200>"$LOCK_FILE"
if ! flock -n 200; then
  echo "[$(date -Is)] SKIP launch simboard collection, lock already held, pid $$" >> "$LOG_FILE"
  exit 0
fi

cleanup() {
  echo "[$(date -Is)] simboard collection launcher exiting, pid $$" >> "$LOG_FILE"
}
trap cleanup EXIT

# Run the app
cd "${SIMBOARD_REPODIR}"
exec "${PYTHON_BIN}" -m app.scripts.ingestion.hpc_upload_archive_ingestor >> "$LOG_FILE" 2>&1

I hope to test this thoroughly once the NERSC backend is up.