diff --git a/docs/source/dev_guide/index.rst b/docs/source/dev_guide/index.rst index 9ff5ade8..42b780ec 100644 --- a/docs/source/dev_guide/index.rst +++ b/docs/source/dev_guide/index.rst @@ -12,9 +12,8 @@ This guide covers everything needed to develop, test, and release ``zppy``. parameters provenance tasks/index - test - update_expected_results - releases/index.rst + tests/index + releases/index new_task new_diags_set archive/index diff --git a/docs/source/dev_guide/tests/automated_test.rst b/docs/source/dev_guide/tests/automated_test.rst new file mode 100644 index 00000000..739ce7a3 --- /dev/null +++ b/docs/source/dev_guide/tests/automated_test.rst @@ -0,0 +1,406 @@ +.. _automated-testing-zppy: + +************************* +Automated testing of zppy +************************* + +Follow the steps below to test ``zppy``. As you do so, please produce a Markdown report summarizing your results. + +Step 1: Determine what the current expected results are +======================================================= + +Machine-specific setup +~~~~~~~~~~~~~~~~~~~~~~ + +Chrysalis: + +.. code-block:: bash + + expected_results_dir=/lcrc/group/e3sm/public_html/zppy_test_resources + expected_results_records_dir=/lcrc/group/e3sm/public_html/zppy_test_resources_previous + +Compy: + +.. code-block:: bash + + expected_results_dir=/compyfs/www/zppy_test_resources + expected_results_records_dir=/compyfs/fors729/zppy_test_resources_previous + +Note that Compy doesn't give write access to ``/compyfs/www/``, so we can't add a new directory there. That's why ``zppy_test_resources_previous`` is in a separate path. + +Perlmutter: + +.. code-block:: bash + + expected_results_dir=/global/cfs/cdirs/e3sm/www/zppy_test_resources + expected_results_records_dir=/global/cfs/cdirs/e3sm/www/zppy_test_resources_previous + +Process +~~~~~~~ + +.. code-block:: bash + + ls -lt ${expected_results_dir} + +In your Markdown report, note the date the expected results were last updated. + +Step 2: Review changes since expected results were updated +========================================================== + +Now that we know the date the expected results are from, we can review what changes we'll be testing. + +Review each of the following commit logs and note commits made since the date the expected results were updated: + +* For the ``e3sm_to_cmip`` task: `e3sm_to_cmip `_ +* For the ``e3sm_diags`` task: `e3sm_diags `_ +* For the ``mpas_analysis`` task: `MPAS-Analysis `_ +* For the ``global_time_series`` and ``pcmdi_diags`` tasks: `zppy-interfaces `_ +* For ``zppy`` itself: `zppy `_ + +For the remaining tasks (``climo``, ``ts``, ``tc_analysis``, ``ilamb``, ``livvkit``), we typically just use the associated package's latest release rather than making dev environments. As such, their latest development will have no impact on our tests unless we have started using one of their newer releases. + +In your Markdown report, make a table like: + +.. code-block:: + + | Package | Changes since expected results were updated | + | --- | --- | + | [package name](link to package's commit log) | Links to all PRs merged since the expected results were updated | + ... + +Step 2.5: Refresh frozen dependency lock files (if needed) +============================================================ + +The test script freezes dependencies for ``e3sm_to_cmip``, ``e3sm_diags``, ``mpas_analysis``, ``zppy-interfaces``, and ``zppy`` so that ``test_images.py`` diffs can be attributed to the package under test rather than to an unrelated dependency (e.g. ``matplotlib``) that happened to move between runs. Each of these five gets its own dedicated, fully-resolved conda env (a "frozen base"); the test env for a given run is created by *cloning* that frozen base and ``pip install``-ing the branch under test on top, so nothing else in the environment can drift. + +This means each of the five needs a lock file -- an exact-version ``conda list --explicit`` snapshot, not a ``dev.yml`` (a ``dev.yml`` re-solves and can drift between runs even unmodified). You only need to regenerate a component's lock file when: + +* You don't have one yet (first-time setup), or +* That component's ``dev.yml`` changed in a way that should be picked up (a new or updated dependency), or +* The test script warned that ``pip install .`` pulled in something beyond the frozen base for that component (see Step C below) and you've decided to bake that change in. + +To (re)generate a lock file for a component: + +.. code-block:: bash + + cd ${repo_parent_dir}/ # e.g. e3sm_diags, MPAS-Analysis, zppy-interfaces, zppy, e3sm_to_cmip + conda env create -f /dev.yml -n tmp-lock-gen # or --file dev-spec.txt for mpas_analysis + conda activate tmp-lock-gen + conda list --explicit > ${EZ_DIR}/frozen-base-.txt + conda deactivate + conda remove --yes --all --name tmp-lock-gen + +Repeat for each of the five components, using the exact filenames referenced by ``BASE_ENV_LOCK_FILE_`` in your ``zppy_test.cfg`` (see Section B below). If ``FREEZE_DEPENDENCIES=false`` in your config, you can skip this step entirely -- every component will solve its own ``dev.yml`` fresh, as before. + +The automated test script +========================= + +The automated test script handles the following steps from the manual testing process: + +* Step 3: Set up environments for called packages +* Step 4: Set up zppy environment +* Step 5: Launch zppy jobs +* Step 6: Launch zppy jobs – bundles part 2 +* Step 7: Review finished returns +* Step 8: Run Python tests (excluding the final ``pytest tests/integration/test_images.py`` call from a compute node) + +A. Set up the test script +~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. code-block:: bash + + cd ${repo_parent_dir}/zppy + git status # Check for uncommitted changes + + # If there are uncommitted changes, + # commit them so we can move cleanly to a new branch: + git add -A + git commit -m "Checkpoint" + + git fetch upstream main # This assumes you've named your remote for the main repo as "upstream" + git checkout -b test-zppy-yyyymmdd upstream/main # Use today's date + git log --oneline | head -n 1 + # Check that this matches the corresponding commit log: + # https://github.com/E3SM-Project/zppy/commits/main + + # Now, copy the test script and cfg from the zppy repo into the directory + # that you'll be running the test script from. + mkdir -p ${test_script_dir}/test_yyyymmdd_runN + cd ${test_script_dir}/test_yyyymmdd_runN + cp ${repo_parent_dir}/zppy/tests/main_branch_testing/run_integration_test.bash . + cp ${repo_parent_dir}/zppy/tests/main_branch_testing/zppy_test.cfg . + + # Now, edit the test cfg as needed + emacs zppy_test.cfg + +B. Set up the test cfg +~~~~~~~~~~~~~~~~~~~~~~ + +Let's examine the parts of the test cfg. + +You'll likely just need to update the ``MACHINE`` name if you're not running on Chrysalis. + +.. code-block:: + + # For these, + + MACHINE=chrysalis # chrysalis | compy | perlmutter + START_PHASE=1 # 1 | 2 | 3 + AUTO_MODE=true # true = skip all interactive checkpoints + EXPLICIT_TAG="" # Leave empty to auto-generate; set to resume a prior run + +Update the ``RUN_NUMBER`` if you've already done a test run today. + +.. code-block:: + + RUN_NUMBER=1 + +Update the ``_BASE_BRANCH`` parameters if you plan to test new features or bug fixes that aren't yet included on whatever the repo calls its "official" branch. + +.. code-block:: + + DIAGS_BASE_BRANCH="main" + E3SM_TO_CMIP_BASE_BRANCH="master" + MPAS_BASE_BRANCH="develop" + ZI_BASE_BRANCH="main" + ZPPY_BASE_BRANCH="main" + +Update the ``_ENV_TYPE`` parameters if you want to use E3SM-Unified rather than a dev environment. If you plan to only run a subset of tasks, you can set the ones you aren't running to use E3SM-Unified, so that the script doesn't spend time building a dev environment that won't be used. + +.. code-block:: + + # "dev" = build a dedicated conda env from the repo's dev.yml + # "unified" = use the machine's e3sm-unified env (UNIFIED_ENV_CMD) + DIAGS_ENV_TYPE="dev" + E3SM_TO_CMIP_ENV_TYPE="dev" + MPAS_ENV_TYPE="dev" + ZI_ENV_TYPE="dev" + +Update the ``_EXISTING_ENV`` parameters if you already have an environment from a previous test run to use. + +.. code-block:: + + # Optional: reuse an existing named conda env instead of creating a new one. + # When non-empty AND the corresponding ENV_TYPE is "dev", the script skips + # conda env creation and activates this env directly. + # Leave empty to let the script auto-name and create the env as usual. + DIAGS_EXISTING_ENV="" + E3SM_TO_CMIP_EXISTING_ENV="" + MPAS_EXISTING_ENV="" + ZI_EXISTING_ENV="" + ZPPY_EXISTING_ENV="" + +Update these two parameters to configure which jobs run. + +.. code-block:: + + # Comma-separated list of zppy cfg names to generate and submit. + # These correspond to generated filenames: test_weekly__.cfg + # Any name containing "bundle" is treated as a bundle cfg and re-submitted in Phase 2. + CFGS_TO_RUN="weekly_bundles,weekly_comprehensive_v2,weekly_comprehensive_v3,weekly_legacy_3.1.0_bundles,weekly_legacy_3.1.0_comprehensive_v2,weekly_legacy_3.1.0_comprehensive_v3,weekly_legacy_3.0.0_bundles,weekly_legacy_3.0.0_comprehensive_v2,weekly_legacy_3.0.0_comprehensive_v3" + + # Comma-separated list of tasks to enable in utils.py. + TASKS_TO_RUN="e3sm_diags,mpas_analysis,global_time_series,ilamb,livvkit,pcmdi_diags" + +These parameters control the frozen dependency base discussed in Step 2.5 above. In most runs you won't need to touch these beyond making sure ``FREEZE_DEPENDENCIES=true`` and that a lock file exists for each component in ``FROZEN_BASE_COMPONENTS``. + +.. code-block:: + + # Master switch. false = every component solves its own dev.yml fresh, + # as before this feature existed. + FREEZE_DEPENDENCIES=true + + # Which components get their own dedicated frozen base. Defaults to all + # five dev-env components, since image-check diffs can come from any of + # them, not just e3sm_diags/pcmdi_diags. + FROZEN_BASE_COMPONENTS="e3sm_to_cmip,e3sm_diags,mpas_analysis,zppy_interfaces,zppy" + +Further down, after ``EZ_DIR`` is defined (see below), each frozen component's lock file path is set: + +.. code-block:: + + BASE_ENV_LOCK_FILE_E3SM_TO_CMIP="$EZ_DIR/frozen-base-e3sm_to_cmip.txt" + BASE_ENV_LOCK_FILE_E3SM_DIAGS="$EZ_DIR/frozen-base-e3sm_diags.txt" + BASE_ENV_LOCK_FILE_MPAS_ANALYSIS="$EZ_DIR/frozen-base-mpas_analysis.txt" + BASE_ENV_LOCK_FILE_ZPPY_INTERFACES="$EZ_DIR/frozen-base-zppy_interfaces.txt" + BASE_ENV_LOCK_FILE_ZPPY="$EZ_DIR/frozen-base-zppy.txt" + +If ``FREEZE_DEPENDENCIES=false``, or a component is removed from ``FROZEN_BASE_COMPONENTS``, its corresponding ``BASE_ENV_LOCK_FILE_*`` value is simply ignored. + +These parameters are unlikely to change between runs. They just let the test script know where to find files in your particular workspace. It is recommended to clone a new copy of the repos and use that for each ``_DIR`` parameter listed below. The script will change branches, so using a distinct copy means you won't get your work overwritten. + +.. code-block:: + + HOME_DIR="$HOME" + EZ_DIR="$HOME_DIR/ez" + + E3SM_DIAGS_DIR="$EZ_DIR/e3sm_diags" + E3SM_TO_CMIP_DIR="$EZ_DIR/e3sm_to_cmip" + MPAS_ANALYSIS_DIR="$EZ_DIR/MPAS-Analysis" + ZPPY_INTERFACES_DIR="$EZ_DIR/zppy-interfaces" + ZPPY_DIR="$EZ_DIR/zppy" + + CONDA_PROFILE="$HOME_DIR/miniforge3/etc/profile.d/conda.sh" + TAG_CACHE_FILE="$HOME_DIR/.zppy_test_tag" + + +C. Run the test script +~~~~~~~~~~~~~~~~~~~~~~ + +Now that we have the test cfg set up, we can run it. + +.. code-block:: bash + + screen # Use `screen`` so that even if the terminal connection is interrupted, the script will keep running. + ulimit -s unlimited # This is necessary for MPAS-Analysis to work inside `screen` + cd ${test_script_dir}/test_yyyymmdd_runN + cat zppy_test.cfg # Make sure changes are there + time ./run_integration_test.bash --config zppy_test.cfg 2>&1 | tee integration_test_runN.log + # Ctrl-A D to detach from screen + screen -ls # See what screen sessions you have + tail -f integration_test_runN.log + +Follow the ``tail`` output until you get to: + +.. code-block:: + + ✓ Phase 3 automated tests complete! + ✓ Remember to run test_images.py manually from a compute node. + ✓ Integration test automation complete! + +If ``FREEZE_DEPENDENCIES=true``, watch for warnings like the following while Phase 1 sets up environments: + +.. code-block:: + + ⚠ 'test-diags-main-yyyymmdd_runN': package set changed beyond the frozen base after 'pip install .'. + ⚠ This is expected ONLY if 'e3sm_diags' itself added or bumped a dependency: + < some diff lines > + +This means the branch under test needed something beyond what's pinned in that component's lock file. It's not necessarily a problem -- just confirm the added/bumped package is one you'd expect that branch to need, and consider regenerating that component's lock file (Step 2.5) so future runs pick it up without the warning. The full diff is also saved as ``pre_install__.txt`` / ``post_install__.txt`` in your run directory. + +D. Review the output +~~~~~~~~~~~~~~~~~~~~ + +.. code-block:: bash + + # CTRL C # Exit tail + screen -R # The script should have finished and there should be ``time`` output: real, user, sys + exit # Exit screen + cd ${test_script_dir}/test_yyyymmdd_runN + cat integration_test_runN.log + +Let's review the test script's output log. + +First, the unit tests. There are two blocks, starting with: + +.. code-block:: + + Running zppy-interfaces unit tests... + +and + +.. code-block:: + + Running zppy unit tests... + +Second, the output directories status. It should look like the following: + +.. code-block:: + + Checking all status files... + ... + ✓ All status files clean! + +If some status files were unsuccessful, you'll want to run the following to review the errors: + +.. code-block:: bash + + cd ${dir_with_failures} + grep -v "OK" * status # See what jobs failed + # Review errors: + tail ${job_that_failed}.o${id_of_job_that_failed} + grep -i error ${job_that_failed}.o${id_of_job_that_failed} + +Third, the integration tests. + +.. code-block:: + + test_last_year.py + test_bash_generation.py + test_campaign.py + test_defaults.py + test_bundles.py + +Errors here may actually be expected if the expected results haven't been updated yet to reflect a recently merged pull request. Another reason for errors on ``test_bundles.py`` in particular is if you didn't run all the jobs necessary (i.e., if you're running a partial test). + +If all 3 pieces look good, you can proceed with the final integration test, the image checker. + +Step 8: Run Python tests +======================== + +Machine-specific setup +~~~~~~~~~~~~~~~~~~~~~~ + +Chrysalis: + +.. code-block:: bash + + launch_compute_node() + { + salloc --nodes=1 --partition=debug --time=02:00:00 --account=e3sm + } + +Compy: + +.. code-block:: bash + + launch_compute_node() + { + salloc --nodes=1 --partition=short --time=01:00:00 --account=e3sm + } + +Perlmutter: + +.. code-block:: bash + + launch_compute_node() + { + salloc --nodes=1 --qos=interactive --time=01:00:00 --constraint=cpu --account=e3sm + } + +Process +~~~~~~~ + +.. code-block:: bash + + cd ${repo_parent_dir}/zppy + git status + # You might have changed branches while you were waiting for jobs to finish. + # Make sure you're now back on the correct branch: test-zppy-yyyymmdd + # Also confirm you're back in the correct env: zppy-yyyymmdd or the Unified env + + # The image checker test, which we'll run from a compute node: + launch_compute_node + + start_bash_subshell + # EITHER: + # Activate EITHER a dev environment or the Unified env: + conda activate zppy-yyyymmdd + # OR: the command from `activate_unified_env` + + pytest tests/integration/test_images.py + # Typically takes between 10 and 20 minutes on Chrysalis and Perlmutter. + # Typically takes closer to 50 minutes on Compy. + cat test_images_summary.md + exit # Exit bash shell + exit # Exit compute note + +If ``FREEZE_DEPENDENCIES=true`` was used for this run, any diffs reported here should trace back to the branches under test in ``FROZEN_BASE_COMPONENTS`` rather than to incidental dependency movement -- that's the isolation this feature is for. If a diff still looks like it could be dependency-related, double check the ``pre_install_*``/``post_install_*`` files and any warnings from Step C for that component. + +In your Markdown report: + +* From the ``pytest tests/integration/test_images.py `` command-line output, copy everything after ``Captured stdout call`` to a code block labeled "Output" +* Copy the results of ``cat test_images_summary.md`` to a section labeled "Complete summary table" +* Make a new section named "Summary table -- only failing image-check tests, sorted by task". For each task that has missing and/or mismatched images, copy the relevant rows from the summary table. Skip this section if there were no failing image-check tests. +* Note any test failures from the other Python tests. +* If there were no failures at all, print "All tests pass" diff --git a/docs/source/dev_guide/tests/index.rst b/docs/source/dev_guide/tests/index.rst new file mode 100644 index 00000000..f0119ea8 --- /dev/null +++ b/docs/source/dev_guide/tests/index.rst @@ -0,0 +1,14 @@ +.. _tests: + +******* +Tests +******* + +This page collects documentation on testing ``zppy``. + +.. toctree:: + :maxdepth: 1 + + manual_test + automated_test + update_expected_results diff --git a/docs/source/dev_guide/test.rst b/docs/source/dev_guide/tests/manual_test.rst similarity index 98% rename from docs/source/dev_guide/test.rst rename to docs/source/dev_guide/tests/manual_test.rst index ff57f678..c4d3ba02 100644 --- a/docs/source/dev_guide/test.rst +++ b/docs/source/dev_guide/tests/manual_test.rst @@ -1,8 +1,8 @@ -.. _testing-zppy: +.. _manually-testing-zppy: -************* -Testing zppy -************* +********************* +Manually testing zppy +********************* Follow the steps below to test ``zppy``. As you do so, please produce a Markdown report summarizing your results. @@ -93,6 +93,11 @@ Chrysalis: lcrc_conda # Or however you activate conda rm -rf build conda clean --all --y + # The dev.yml file may be in a different directory. + # e3sm_to_cmip, e3sm_diags: use conda-dev/ + # zppy-interfaces, zppy: use conda/ + # MPAS-Analysis: don't use this command, instead use: + # conda create --name ${env_name} --file dev-spec.txt --yes conda env create -f conda/dev.yml -n ${env_name} conda activate ${env_name} pre-commit run --all-files # Confirm this passes diff --git a/docs/source/dev_guide/update_expected_results.rst b/docs/source/dev_guide/tests/update_expected_results.rst similarity index 100% rename from docs/source/dev_guide/update_expected_results.rst rename to docs/source/dev_guide/tests/update_expected_results.rst diff --git a/tests/integration/template_weekly_bundles.cfg b/tests/integration/template_weekly_bundles.cfg index ae94c1d8..9f164ebb 100644 --- a/tests/integration/template_weekly_bundles.cfg +++ b/tests/integration/template_weekly_bundles.cfg @@ -51,10 +51,11 @@ bundle = "bundle1" years = "1985:1989:2", [[ atm_monthly_180x360_aave ]] + active = #expand active_climo_month_atm# frequency = "monthly" [[ atm_monthly_diurnal_8xdaily_180x360_aave ]] - active = #expand active_e3sm_diags# + active = #expand active_climo_diurnal_atm# frequency = "diurnal_8xdaily" input_files = "eam.h3" input_subdir = "archive/atm/hist" @@ -66,12 +67,13 @@ bundle = "bundle1" years = "1985:1989:2", [[ atm_monthly_180x360_aave ]] + active = #expand active_ts_month_atm# frequency = "monthly" input_files = "eam.h0" input_subdir = "archive/atm/hist" [[ atm_monthly_glb ]] - active = #expand active_global_time_series# + active = #expand active_ts_month_atm_glb# bundle = "bundle2" # Override bundle1 frequency = "monthly" input_files = "eam.h0" @@ -80,6 +82,7 @@ years = "1985:1989:2", years = "1985:1995:5", [[ land_monthly ]] + active = #expand active_ts_month_lnd# extra_vars = "landfrac" frequency = "monthly" input_files = "elm.h0" @@ -88,7 +91,7 @@ years = "1985:1989:2", vars = "LAISHA,LAISUN" [[ rof_monthly ]] - active = #expand active_e3sm_diags# + active = #expand active_ts_month_rof# bundle = "bundle3" # Override bundle1, let bundle1 finish first because "e3sm_diags: atm_monthly_180x360_aave_mvm" requires "ts: atm_monthly_180x360_aave" extra_vars = 'areatotal2' frequency = "monthly" @@ -98,7 +101,7 @@ years = "1985:1989:2", vars = "RIVER_DISCHARGE_OVER_LAND_LIQ" [e3sm_to_cmip] -active = #expand active_e3sm_to_cmip# +active = True bundle = "bundle1" environment_commands = "#expand e3sm_to_cmip_environment_commands#" frequency = "monthly" @@ -106,14 +109,16 @@ ts_num_years = 2 years = "1985:1989:2", [[ atm_monthly_180x360_aave ]] + active = #expand active_e3sm_to_cmip_month_atm# input_files = "eam.h0" [[ land_monthly ]] + active = #expand active_e3sm_to_cmip_month_lnd# input_files = "elm.h0" # TODO: Add "tc_analysis" back in after empty dat is resolved. # [tc_analysis] -# active = True +# active = #expand active_tc_analysis# # bundle = "bundle3" # Let bundle1 finish first because "e3sm_diags: atm_monthly_180x360_aave_mvm" requires "ts: atm_monthly_180x360_aave" # years = "1985:1989:2", diff --git a/tests/integration/template_weekly_comprehensive_v2.cfg b/tests/integration/template_weekly_comprehensive_v2.cfg index 7ba7aef8..9f3cec9b 100644 --- a/tests/integration/template_weekly_comprehensive_v2.cfg +++ b/tests/integration/template_weekly_comprehensive_v2.cfg @@ -19,20 +19,21 @@ active = True walltime = "00:30:00" [[ atm_monthly_180x360_aave ]] + active = #expand active_climo_month_atm# frequency = "monthly" input_files = "eam.h0" input_subdir = "archive/atm/hist" vars = "" [[ atm_monthly_diurnal_8xdaily_180x360_aave ]] - active = #expand active_e3sm_diags# + active = #expand active_climo_diurnal_atm# frequency = "diurnal_8xdaily" input_files = "eam.h3" input_subdir = "archive/atm/hist" vars = "PRECT" [[ land_monthly_climo ]] - active = #expand active_e3sm_diags# + active = #expand active_climo_month_lnd# frequency = "monthly" input_files = "elm.h0" input_subdir = "archive/lnd/hist" @@ -43,19 +44,20 @@ active = True walltime = "00:30:00" [[ atm_monthly_180x360_aave ]] + active = #expand active_ts_month_atm# frequency = "monthly" input_files = "eam.h0" input_subdir = "archive/atm/hist" [[ atm_daily_180x360_aave ]] - active = #expand active_e3sm_diags# + active = #expand active_ts_daily_atm# frequency = "daily" input_files = "eam.h1" input_subdir = "archive/atm/hist" vars = "PRECT" [[ rof_monthly ]] - active = #expand active_e3sm_diags# + active = #expand active_ts_month_rof# extra_vars = 'areatotal2' frequency = "monthly" input_files = "mosart.h0" @@ -65,7 +67,7 @@ walltime = "00:30:00" [[ atm_monthly_glb ]] # Note global average won't work for 3D variables. - active = #expand active_global_time_series# + active = #expand active_ts_month_atm_glb# frequency = "monthly" input_files = "eam.h0" input_subdir = "archive/atm/hist" @@ -73,7 +75,7 @@ walltime = "00:30:00" years = "1980:1990:5", [[ lnd_monthly_glb ]] - active = #expand active_global_time_series# + active = #expand active_ts_month_lnd_glb# frequency = "monthly" input_files = "elm.h0" input_subdir = "archive/lnd/hist" @@ -82,6 +84,7 @@ walltime = "00:30:00" years = "1980:1990:5", [[ land_monthly ]] + active = #expand active_ts_month_lnd# extra_vars = "landfrac" frequency = "monthly" input_files = "elm.h0" @@ -89,20 +92,22 @@ walltime = "00:30:00" vars = "LAISHA,LAISUN" [e3sm_to_cmip] -active = #expand active_e3sm_to_cmip# +active = True environment_commands = "#expand e3sm_to_cmip_environment_commands#" frequency = "monthly" ts_num_years=2 walltime = "00:30:00" [[ atm_monthly_180x360_aave ]] + active = #expand active_e3sm_to_cmip_month_atm# input_files = "eam.h0" [[ land_monthly ]] + active = #expand active_e3sm_to_cmip_month_lnd# input_files = "elm.h0" [tc_analysis] -active = #expand active_e3sm_diags# +active = #expand active_tc_analysis# walltime = "00:30:00" [e3sm_diags] diff --git a/tests/integration/template_weekly_comprehensive_v3.cfg b/tests/integration/template_weekly_comprehensive_v3.cfg index 5c3d1f0b..75535312 100644 --- a/tests/integration/template_weekly_comprehensive_v3.cfg +++ b/tests/integration/template_weekly_comprehensive_v3.cfg @@ -22,26 +22,27 @@ frequency = "monthly" walltime = "00:30:00" [[ atm_monthly_180x360_aave ]] + active = #expand active_climo_month_atm# input_files = "eam.h0" input_subdir = "archive/atm/hist" vars = "" [[ atm_monthly_diurnal_8xdaily_180x360_aave ]] - active = #expand active_e3sm_diags# + active = #expand active_climo_diurnal_atm# frequency = "diurnal_8xdaily" input_files = "eam.h3" input_subdir = "archive/atm/hist" vars = "PRECT" [[ land_monthly_climo ]] - active = True + active = #expand active_climo_month_lnd# input_files = "elm.h0" input_subdir = "archive/lnd/hist" mapping_file = "map_r05_to_cmip6_180x360_aave.20231110.nc" vars = "" [[ land_monthly_180x360_traave ]] - active = #expand active_livvkit# + active = #expand active_climo_month_lnd_for_livvkit# input_files = "elm.h0" input_subdir = "archive/lnd/hist" mapping_file = "#expand livvkit_mapping_file_path#/map_r05_to_cmip6_180x360_traave.20231110.nc" @@ -49,7 +50,7 @@ walltime = "00:30:00" years = "1985:1994:10", [[ land_monthly_climo_native ]] - active = #expand active_livvkit# + active = #expand active_climo_month_lnd_for_livvkit# climo_jobs = 12 input_files = "elm.h0" input_subdir = "archive/lnd/hist" @@ -58,7 +59,7 @@ walltime = "00:30:00" years = "1985:1994:10" [[ land_monthly_climo_racmo_gis ]] - active = #expand active_livvkit# + active = #expand active_climo_month_lnd_for_livvkit# climo_jobs = 12 climo_subsection = "racmo_gis" grid = "racmo_gis" @@ -69,7 +70,7 @@ walltime = "00:30:00" years = "1985:1994:10" [[ land_monthly_climo_racmo_ais ]] - active = #expand active_livvkit# + active = #expand active_climo_month_lnd_for_livvkit# climo_jobs = 12 input_files = "elm.h0" climo_subsection = "racmo_ais" @@ -80,7 +81,7 @@ walltime = "00:30:00" years = "1985:1994:10" [[ land_monthly_climo_merra2 ]] - active = #expand active_livvkit# + active = #expand active_climo_month_lnd_for_livvkit# climo_jobs = 12 input_files = "elm.h0" climo_subsection = "merra2" @@ -91,7 +92,7 @@ walltime = "00:30:00" years = "1985:1994:10" [[ land_monthly_climo_era5 ]] - active = #expand active_livvkit# + active = #expand active_climo_month_lnd_for_livvkit# climo_jobs = 12 input_files = "elm.h0" climo_subsection = "era5" @@ -106,6 +107,7 @@ active = True walltime = "00:30:00" [[ atm_monthly_180x360_aave ]] + active = #expand active_ts_month_atm# frequency = "monthly" input_files = "eam.h0" input_subdir = "archive/atm/hist" @@ -115,14 +117,14 @@ walltime = "00:30:00" vars = "FSNTOA,FLUT,FSNT,FLNT,FSNS,FLNS,SHFLX,QFLX,TAUX,TAUY,PRECC,PRECL,PRECSC,PRECSL,TS,TREFHT,CLDTOT,CLDHGH,CLDMED,CLDLOW,U,PSL,LANDFRAC,CLD_CAL_TMPICE,CLD_CAL_TMPLIQ,CLDLIQ,T" [[ atm_daily_180x360_aave ]] - active = #expand active_e3sm_diags# + active = #expand active_ts_daily_atm# frequency = "daily" input_files = "eam.h1" input_subdir = "archive/atm/hist" vars = "PRECT" [[ rof_monthly ]] - active = #expand active_e3sm_diags# + active = #expand active_ts_month_rof# extra_vars = 'areatotal2' frequency = "monthly" input_files = "mosart.h0" @@ -132,7 +134,7 @@ walltime = "00:30:00" [[ atm_monthly_glb ]] # Note global average won't work for 3D variables. - active = #expand active_global_time_series# + active = #expand active_ts_month_atm_glb# frequency = "monthly" input_files = "eam.h0" input_subdir = "archive/atm/hist" @@ -140,7 +142,7 @@ walltime = "00:30:00" years = "1985:1995:5", [[ lnd_monthly_glb ]] - active = #expand active_global_time_series# + active = #expand active_ts_month_lnd_glb# frequency = "monthly" input_files = "elm.h0" input_subdir = "archive/lnd/hist" @@ -149,6 +151,7 @@ walltime = "00:30:00" years = "1985:1995:5", [[ land_monthly ]] + active = #expand active_ts_month_lnd# extra_vars = "landfrac" frequency = "monthly" input_files = "elm.h0" @@ -157,7 +160,7 @@ walltime = "00:30:00" vars = "FSH,RH2M,LAISHA,LAISUN,QINTR,QOVER,QRUNOFF,QSOIL,QVEGE,QVEGT,SOILICE,SOILLIQ,SOILWATER_10CM,TSA,TSOI,H2OSNO,TOTLITC,CWDC,SOIL1C,SOIL2C,SOIL3C,SOIL4C,WOOD_HARVESTC,TOTVEGC,NBP,GPP,AR,HR" [[ land_monthly_energy ]] - active = #expand active_livvkit# + active = #expand active_ts_month_lnd_for_livvkit# frequency = "monthly" input_files = "elm.h0" input_subdir = "archive/lnd/hist" @@ -166,8 +169,7 @@ walltime = "00:30:00" years = "1985:1994:10" [[ land_monthly_smb ]] - # The data necessary for this is not found in the data input path. - active = False + active = False # The data necessary for this is not found in the data input path. frequency = "monthly" input_files = "elm.h0" input_subdir = "archive/lnd/hist" @@ -176,13 +178,14 @@ walltime = "00:30:00" years = "1985:1994:10" [e3sm_to_cmip] -active = #expand active_e3sm_to_cmip# +active = True environment_commands = "#expand e3sm_to_cmip_environment_commands#" frequency = "monthly" ts_num_years=2 walltime = "00:30:00" [[ atm_monthly_180x360_aave ]] + active = #expand active_e3sm_to_cmip_month_atm# cmip_plevdata = "#expand diagnostics_base_path#/e3sm_to_cmip_data/maps/vrt_remap_plev19.nc" cmip_vars = "ua, va, ta, wap, zg, hur, tas, ts, psl, ps, sfcWind, huss, pr, prc, prsn, evspsbl, tauu, tauv, hfls, clt, rlus, rsds, rsus, hfss, clivi, clwvi, rlut, rsdt, rsuscs, rsut, rtmt, abs550aer, od550aer, rsdscs, tasmax, tasmin" input_files = "eam.h0" @@ -191,12 +194,13 @@ walltime = "00:30:00" years = "1985:1995:2", # Need 10 years for pcmdi_diags task [[ land_monthly ]] + active = #expand active_e3sm_to_cmip_month_lnd# input_files = "elm.h0" ts_subsection = "land_monthly" # TODO: Add "tc_analysis" back in after empty dat is resolved. # [tc_analysis] -# active = True +# active = #expand active_tc_analysis# # walltime = "00:30:00" [e3sm_diags] diff --git a/tests/integration/template_weekly_legacy_3.0.0_bundles.cfg b/tests/integration/template_weekly_legacy_3.0.0_bundles.cfg index fac3483b..800a3c8b 100644 --- a/tests/integration/template_weekly_legacy_3.0.0_bundles.cfg +++ b/tests/integration/template_weekly_legacy_3.0.0_bundles.cfg @@ -53,10 +53,11 @@ bundle = "bundle1" years = "1985:1989:2", [[ atm_monthly_180x360_aave ]] + active = #expand active_climo_month_atm# frequency = "monthly" [[ atm_monthly_diurnal_8xdaily_180x360_aave ]] - active = #expand active_e3sm_diags# + active = #expand active_climo_diurnal_atm# frequency = "diurnal_8xdaily" input_files = "eam.h3" input_subdir = "archive/atm/hist" @@ -68,12 +69,13 @@ bundle = "bundle1" years = "1985:1989:2", [[ atm_monthly_180x360_aave ]] + active = #expand active_ts_month_atm# frequency = "monthly" input_files = "eam.h0" input_subdir = "archive/atm/hist" [[ atm_monthly_glb ]] - active = #expand active_global_time_series# + active = #expand active_ts_month_atm_glb# bundle = "bundle2" # Override bundle1 frequency = "monthly" input_files = "eam.h0" @@ -82,6 +84,7 @@ years = "1985:1989:2", years = "1985:1995:5", [[ land_monthly ]] + active = #expand active_ts_month_lnd# extra_vars = "landfrac" frequency = "monthly" input_files = "elm.h0" @@ -90,7 +93,7 @@ years = "1985:1989:2", vars = "LAISHA,LAISUN" [[ rof_monthly ]] - active = #expand active_e3sm_diags# + active = #expand active_ts_month_rof# bundle = "bundle3" # Override bundle1, let bundle1 finish first because "e3sm_diags: atm_monthly_180x360_aave_mvm" requires "ts: atm_monthly_180x360_aave" extra_vars = 'areatotal2' frequency = "monthly" @@ -100,7 +103,7 @@ years = "1985:1989:2", vars = "RIVER_DISCHARGE_OVER_LAND_LIQ" [e3sm_to_cmip] -active = #expand active_e3sm_to_cmip# +active = True bundle = "bundle1" environment_commands = "#expand e3sm_to_cmip_environment_commands#" frequency = "monthly" @@ -108,14 +111,16 @@ ts_num_years = 2 years = "1985:1989:2", [[ atm_monthly_180x360_aave ]] + active = #expand active_e3sm_to_cmip_month_atm# input_files = "eam.h0" [[ land_monthly ]] + active = #expand active_e3sm_to_cmip_month_lnd# input_files = "elm.h0" # TODO: Add "tc_analysis" back in after empty dat is resolved. # [tc_analysis] -# active = True +# active = #expand active_tc_analysis# # bundle = "bundle3" # Let bundle1 finish first because "e3sm_diags: atm_monthly_180x360_aave_mvm" requires "ts: atm_monthly_180x360_aave" # years = "1985:1989:2", diff --git a/tests/integration/template_weekly_legacy_3.0.0_comprehensive_v2.cfg b/tests/integration/template_weekly_legacy_3.0.0_comprehensive_v2.cfg index 2fd687c8..559e574f 100644 --- a/tests/integration/template_weekly_legacy_3.0.0_comprehensive_v2.cfg +++ b/tests/integration/template_weekly_legacy_3.0.0_comprehensive_v2.cfg @@ -21,20 +21,21 @@ active = True walltime = "00:30:00" [[ atm_monthly_180x360_aave ]] + active = #expand active_climo_month_atm# frequency = "monthly" input_files = "eam.h0" input_subdir = "archive/atm/hist" vars = "" [[ atm_monthly_diurnal_8xdaily_180x360_aave ]] - active = #expand active_e3sm_diags# + active = #expand active_climo_diurnal_atm# frequency = "diurnal_8xdaily" input_files = "eam.h3" input_subdir = "archive/atm/hist" vars = "PRECT" [[ land_monthly_climo ]] - active = #expand active_e3sm_diags# + active = #expand active_climo_month_lnd# frequency = "monthly" input_files = "elm.h0" input_subdir = "archive/lnd/hist" @@ -45,19 +46,20 @@ active = True walltime = "00:30:00" [[ atm_monthly_180x360_aave ]] + active = #expand active_ts_month_atm# frequency = "monthly" input_files = "eam.h0" input_subdir = "archive/atm/hist" [[ atm_daily_180x360_aave ]] - active = #expand active_e3sm_diags# + active = #expand active_ts_daily_atm# frequency = "daily" input_files = "eam.h1" input_subdir = "archive/atm/hist" vars = "PRECT" [[ rof_monthly ]] - active = #expand active_e3sm_diags# + active = #expand active_ts_month_rof# extra_vars = 'areatotal2' frequency = "monthly" input_files = "mosart.h0" @@ -67,7 +69,7 @@ walltime = "00:30:00" [[ atm_monthly_glb ]] # Note global average won't work for 3D variables. - active = #expand active_global_time_series# + active = #expand active_ts_month_atm_glb# frequency = "monthly" input_files = "eam.h0" input_subdir = "archive/atm/hist" @@ -75,7 +77,7 @@ walltime = "00:30:00" years = "1980:1990:5", [[ lnd_monthly_glb ]] - active = #expand active_global_time_series# + active = #expand active_ts_month_lnd_glb# frequency = "monthly" input_files = "elm.h0" input_subdir = "archive/lnd/hist" @@ -84,6 +86,7 @@ walltime = "00:30:00" years = "1980:1990:5", [[ land_monthly ]] + active = #expand active_ts_month_lnd# extra_vars = "landfrac" frequency = "monthly" input_files = "elm.h0" @@ -91,20 +94,22 @@ walltime = "00:30:00" vars = "LAISHA,LAISUN" [e3sm_to_cmip] -active = #expand active_e3sm_to_cmip# +active = True environment_commands = "#expand e3sm_to_cmip_environment_commands#" frequency = "monthly" ts_num_years=2 walltime = "00:30:00" [[ atm_monthly_180x360_aave ]] + active = #expand active_e3sm_to_cmip_month_atm# input_files = "eam.h0" [[ land_monthly ]] + active = #expand active_e3sm_to_cmip_month_lnd# input_files = "elm.h0" [tc_analysis] -active = #expand active_e3sm_diags# +active = #expand active_tc_analysis# walltime = "00:30:00" [e3sm_diags] diff --git a/tests/integration/template_weekly_legacy_3.0.0_comprehensive_v3.cfg b/tests/integration/template_weekly_legacy_3.0.0_comprehensive_v3.cfg index 8b57e400..b87057c0 100644 --- a/tests/integration/template_weekly_legacy_3.0.0_comprehensive_v3.cfg +++ b/tests/integration/template_weekly_legacy_3.0.0_comprehensive_v3.cfg @@ -28,20 +28,21 @@ active = True walltime = "00:30:00" [[ atm_monthly_180x360_aave ]] + active = #expand active_climo_month_atm# frequency = "monthly" input_files = "eam.h0" input_subdir = "archive/atm/hist" vars = "" [[ atm_monthly_diurnal_8xdaily_180x360_aave ]] - active = #expand active_e3sm_diags# + active = #expand active_climo_diurnal_atm# frequency = "diurnal_8xdaily" input_files = "eam.h3" input_subdir = "archive/atm/hist" vars = "PRECT" [[ land_monthly_climo ]] - active = #expand active_e3sm_diags# + active = #expand active_climo_month_lnd# frequency = "monthly" input_files = "elm.h0" input_subdir = "archive/lnd/hist" @@ -53,19 +54,20 @@ active = True walltime = "00:30:00" [[ atm_monthly_180x360_aave ]] + active = #expand active_ts_month_atm# frequency = "monthly" input_files = "eam.h0" input_subdir = "archive/atm/hist" [[ atm_daily_180x360_aave ]] - active = #expand active_e3sm_diags# + active = #expand active_ts_daily_atm# frequency = "daily" input_files = "eam.h1" input_subdir = "archive/atm/hist" vars = "PRECT" [[ rof_monthly ]] - active = #expand active_e3sm_diags# + active = #expand active_ts_month_rof# extra_vars = 'areatotal2' frequency = "monthly" input_files = "mosart.h0" @@ -75,7 +77,7 @@ walltime = "00:30:00" [[ atm_monthly_glb ]] # Note global average won't work for 3D variables. - active = #expand active_global_time_series# + active = #expand active_ts_month_atm_glb# frequency = "monthly" input_files = "eam.h0" input_subdir = "archive/atm/hist" @@ -83,7 +85,7 @@ walltime = "00:30:00" years = "1985:1995:5", [[ lnd_monthly_glb ]] - active = #expand active_global_time_series# + active = #expand active_ts_month_lnd_glb# frequency = "monthly" input_files = "elm.h0" input_subdir = "archive/lnd/hist" @@ -92,6 +94,7 @@ walltime = "00:30:00" years = "1985:1995:5", [[ land_monthly ]] + active = #expand active_ts_month_lnd# extra_vars = "landfrac" frequency = "monthly" input_files = "elm.h0" @@ -100,23 +103,25 @@ walltime = "00:30:00" vars = "FSH,RH2M,LAISHA,LAISUN,QINTR,QOVER,QRUNOFF,QSOIL,QVEGE,QVEGT,SOILICE,SOILLIQ,SOILWATER_10CM,TSA,TSOI,H2OSNO,TOTLITC,CWDC,SOIL1C,SOIL2C,SOIL3C,SOIL4C,WOOD_HARVESTC,TOTVEGC,NBP,GPP,AR,HR" [e3sm_to_cmip] -active = #expand active_e3sm_to_cmip# +active = True environment_commands = "#expand e3sm_to_cmip_environment_commands#" frequency = "monthly" ts_num_years=2 walltime = "00:30:00" [[ atm_monthly_180x360_aave ]] + active = #expand active_e3sm_to_cmip_month_atm# input_files = "eam.h0" ts_subsection = "atm_monthly_180x360_aave" [[ land_monthly ]] + active = #expand active_e3sm_to_cmip_month_lnd# input_files = "elm.h0" ts_subsection = "land_monthly" # TODO: Add "tc_analysis" back in after empty dat is resolved. # [tc_analysis] -# active = True +# active = #expand active_tc_analysis# # walltime = "00:30:00" [e3sm_diags] diff --git a/tests/integration/template_weekly_legacy_3.1.0_bundles.cfg b/tests/integration/template_weekly_legacy_3.1.0_bundles.cfg index 368303dc..60cdf6cb 100644 --- a/tests/integration/template_weekly_legacy_3.1.0_bundles.cfg +++ b/tests/integration/template_weekly_legacy_3.1.0_bundles.cfg @@ -53,10 +53,11 @@ bundle = "bundle1" years = "1985:1989:2", [[ atm_monthly_180x360_aave ]] + active = #expand active_climo_month_atm# frequency = "monthly" [[ atm_monthly_diurnal_8xdaily_180x360_aave ]] - active = #expand active_e3sm_diags# + active = #expand active_climo_diurnal_atm# frequency = "diurnal_8xdaily" input_files = "eam.h3" input_subdir = "archive/atm/hist" @@ -68,12 +69,13 @@ bundle = "bundle1" years = "1985:1989:2", [[ atm_monthly_180x360_aave ]] + active = #expand active_ts_month_atm# frequency = "monthly" input_files = "eam.h0" input_subdir = "archive/atm/hist" [[ atm_monthly_glb ]] - active = #expand active_global_time_series# + active = #expand active_ts_month_atm_glb# bundle = "bundle2" # Override bundle1 frequency = "monthly" input_files = "eam.h0" @@ -82,6 +84,7 @@ years = "1985:1989:2", years = "1985:1995:5", [[ land_monthly ]] + active = #expand active_ts_month_lnd# extra_vars = "landfrac" frequency = "monthly" input_files = "elm.h0" @@ -90,7 +93,7 @@ years = "1985:1989:2", vars = "LAISHA,LAISUN" [[ rof_monthly ]] - active = #expand active_e3sm_diags# + active = #expand active_ts_month_rof# bundle = "bundle3" # Override bundle1, let bundle1 finish first because "e3sm_diags: atm_monthly_180x360_aave_mvm" requires "ts: atm_monthly_180x360_aave" extra_vars = 'areatotal2' frequency = "monthly" @@ -100,7 +103,7 @@ years = "1985:1989:2", vars = "RIVER_DISCHARGE_OVER_LAND_LIQ" [e3sm_to_cmip] -active = #expand active_e3sm_to_cmip# +active = True bundle = "bundle1" environment_commands = "#expand e3sm_to_cmip_environment_commands#" frequency = "monthly" @@ -108,14 +111,16 @@ ts_num_years = 2 years = "1985:1989:2", [[ atm_monthly_180x360_aave ]] + active = #expand active_e3sm_to_cmip_month_atm# input_files = "eam.h0" [[ land_monthly ]] + active = #expand active_e3sm_to_cmip_month_lnd# input_files = "elm.h0" # TODO: Add "tc_analysis" back in after empty dat is resolved. # [tc_analysis] -# active = True +# active = #expand active_tc_analysis# # bundle = "bundle3" # Let bundle1 finish first because "e3sm_diags: atm_monthly_180x360_aave_mvm" requires "ts: atm_monthly_180x360_aave" # years = "1985:1989:2", diff --git a/tests/integration/template_weekly_legacy_3.1.0_comprehensive_v2.cfg b/tests/integration/template_weekly_legacy_3.1.0_comprehensive_v2.cfg index 6d570441..ea171c41 100644 --- a/tests/integration/template_weekly_legacy_3.1.0_comprehensive_v2.cfg +++ b/tests/integration/template_weekly_legacy_3.1.0_comprehensive_v2.cfg @@ -21,20 +21,21 @@ active = True walltime = "00:30:00" [[ atm_monthly_180x360_aave ]] + active = #expand active_climo_month_atm# frequency = "monthly" input_files = "eam.h0" input_subdir = "archive/atm/hist" vars = "" [[ atm_monthly_diurnal_8xdaily_180x360_aave ]] - active = #expand active_e3sm_diags# + active = #expand active_climo_diurnal_atm# frequency = "diurnal_8xdaily" input_files = "eam.h3" input_subdir = "archive/atm/hist" vars = "PRECT" [[ land_monthly_climo ]] - active = #expand active_e3sm_diags# + active = #expand active_climo_month_lnd# frequency = "monthly" input_files = "elm.h0" input_subdir = "archive/lnd/hist" @@ -45,19 +46,20 @@ active = True walltime = "00:30:00" [[ atm_monthly_180x360_aave ]] + active = #expand active_ts_month_atm# frequency = "monthly" input_files = "eam.h0" input_subdir = "archive/atm/hist" [[ atm_daily_180x360_aave ]] - active = #expand active_e3sm_diags# + active = #expand active_ts_daily_atm# frequency = "daily" input_files = "eam.h1" input_subdir = "archive/atm/hist" vars = "PRECT" [[ rof_monthly ]] - active = #expand active_e3sm_diags# + active = #expand active_ts_month_rof# extra_vars = 'areatotal2' frequency = "monthly" input_files = "mosart.h0" @@ -67,7 +69,7 @@ walltime = "00:30:00" [[ atm_monthly_glb ]] # Note global average won't work for 3D variables. - active = #expand active_global_time_series# + active = #expand active_ts_month_atm_glb# frequency = "monthly" input_files = "eam.h0" input_subdir = "archive/atm/hist" @@ -75,7 +77,7 @@ walltime = "00:30:00" years = "1980:1990:5", [[ lnd_monthly_glb ]] - active = #expand active_global_time_series# + active = #expand active_ts_month_lnd_glb# frequency = "monthly" input_files = "elm.h0" input_subdir = "archive/lnd/hist" @@ -84,6 +86,7 @@ walltime = "00:30:00" years = "1980:1990:5", [[ land_monthly ]] + active = #expand active_ts_month_lnd# extra_vars = "landfrac" frequency = "monthly" input_files = "elm.h0" @@ -91,20 +94,22 @@ walltime = "00:30:00" vars = "LAISHA,LAISUN" [e3sm_to_cmip] -active = #expand active_e3sm_to_cmip# +active = True environment_commands = "#expand e3sm_to_cmip_environment_commands#" frequency = "monthly" ts_num_years=2 walltime = "00:30:00" [[ atm_monthly_180x360_aave ]] + active = #expand active_e3sm_to_cmip_month_atm# input_files = "eam.h0" [[ land_monthly ]] + active = #expand active_e3sm_to_cmip_month_lnd# input_files = "elm.h0" [tc_analysis] -active = #expand active_e3sm_diags# +active = #expand active_tc_analysis# walltime = "00:30:00" [e3sm_diags] @@ -174,6 +179,7 @@ active = #expand active_mpas_analysis# anomalyRefYear = 1980 climo_years ="1980-1984", "1985-1990", enso_years = "1980-1984", "1985-1990", +environment_commands = "#expand mpas_analysis_environment_commands#" mesh = "EC30to60E2r2" parallelTaskCount = 6 partition = "#expand partition_long#" diff --git a/tests/integration/template_weekly_legacy_3.1.0_comprehensive_v3.cfg b/tests/integration/template_weekly_legacy_3.1.0_comprehensive_v3.cfg index ad39a309..2c1e65de 100644 --- a/tests/integration/template_weekly_legacy_3.1.0_comprehensive_v3.cfg +++ b/tests/integration/template_weekly_legacy_3.1.0_comprehensive_v3.cfg @@ -23,20 +23,21 @@ active = True walltime = "00:30:00" [[ atm_monthly_180x360_aave ]] + active = #expand active_climo_month_atm# frequency = "monthly" input_files = "eam.h0" input_subdir = "archive/atm/hist" vars = "" [[ atm_monthly_diurnal_8xdaily_180x360_aave ]] - active = #expand active_e3sm_diags# + active = #expand active_climo_diurnal_atm# frequency = "diurnal_8xdaily" input_files = "eam.h3" input_subdir = "archive/atm/hist" vars = "PRECT" [[ land_monthly_climo ]] - active = #expand active_e3sm_diags# + active = #expand active_climo_month_lnd# frequency = "monthly" input_files = "elm.h0" input_subdir = "archive/lnd/hist" @@ -48,20 +49,21 @@ active = True walltime = "00:30:00" [[ atm_monthly_180x360_aave ]] + active = #expand active_ts_month_atm# frequency = "monthly" input_files = "eam.h0" input_subdir = "archive/atm/hist" years = "1985:1995:2", # Need 10 years for pcmdi_diags task [[ atm_daily_180x360_aave ]] - active = #expand active_e3sm_diags# + active = #expand active_ts_daily_atm# frequency = "daily" input_files = "eam.h1" input_subdir = "archive/atm/hist" vars = "PRECT" [[ rof_monthly ]] - active = #expand active_e3sm_diags# + active = #expand active_ts_month_rof# extra_vars = 'areatotal2' frequency = "monthly" input_files = "mosart.h0" @@ -71,7 +73,7 @@ walltime = "00:30:00" [[ atm_monthly_glb ]] # Note global average won't work for 3D variables. - active = #expand active_global_time_series# + active = #expand active_ts_month_atm_glb# frequency = "monthly" input_files = "eam.h0" input_subdir = "archive/atm/hist" @@ -79,7 +81,7 @@ walltime = "00:30:00" years = "1985:1995:5", [[ lnd_monthly_glb ]] - active = #expand active_global_time_series# + active = #expand active_ts_month_lnd_glb# frequency = "monthly" input_files = "elm.h0" input_subdir = "archive/lnd/hist" @@ -88,6 +90,7 @@ walltime = "00:30:00" years = "1985:1995:5", [[ land_monthly ]] + active = #expand active_ts_month_lnd# extra_vars = "landfrac" frequency = "monthly" input_files = "elm.h0" @@ -96,13 +99,14 @@ walltime = "00:30:00" vars = "FSH,RH2M,LAISHA,LAISUN,QINTR,QOVER,QRUNOFF,QSOIL,QVEGE,QVEGT,SOILICE,SOILLIQ,SOILWATER_10CM,TSA,TSOI,H2OSNO,TOTLITC,CWDC,SOIL1C,SOIL2C,SOIL3C,SOIL4C,WOOD_HARVESTC,TOTVEGC,NBP,GPP,AR,HR" [e3sm_to_cmip] -active = #expand active_e3sm_to_cmip# +active = True environment_commands = "#expand e3sm_to_cmip_environment_commands#" frequency = "monthly" ts_num_years=2 walltime = "00:30:00" [[ atm_monthly_180x360_aave ]] + active = #expand active_e3sm_to_cmip_month_atm# cmip_plevdata = "#expand diagnostics_base_path#/e3sm_to_cmip_data/maps/vrt_remap_plev19.nc" cmip_vars = "ua, va, ta, wap, zg, hur, tas, ts, psl, ps, sfcWind, huss, pr, prc, prsn, evspsbl, tauu, tauv, hfls, clt, rlus, rsds, rsus, hfss, clivi, clwvi, rlut, rsdt, rsuscs, rsut, rtmt, abs550aer, od550aer, rsdscs, tasmax, tasmin" input_files = "eam.h0" @@ -111,12 +115,13 @@ walltime = "00:30:00" years = "1985:1995:2", # Need 10 years for pcmdi_diags task [[ land_monthly ]] + active = #expand active_e3sm_to_cmip_month_lnd# input_files = "elm.h0" ts_subsection = "land_monthly" # TODO: Add "tc_analysis" back in after empty dat is resolved. # [tc_analysis] -# active = True +# active = #expand active_tc_analysis# # walltime = "00:30:00" [e3sm_diags] @@ -207,6 +212,7 @@ active = #expand active_mpas_analysis# anomalyRefYear = 1985 climo_years = "1985-1989", "1990-1995", enso_years = "1985-1989", "1990-1995", +environment_commands = "#expand mpas_analysis_environment_commands#" mesh = "IcoswISC30E3r5" parallelTaskCount = 6 partition = "#expand partition_long#" diff --git a/tests/integration/utils.py b/tests/integration/utils.py index 17519e47..b82c9dec 100644 --- a/tests/integration/utils.py +++ b/tests/integration/utils.py @@ -180,29 +180,81 @@ def get_expansions(): expansions["environment_commands"] = TEST_SPECIFICS["environment_commands"] # Activate requested tests - expansions["active_e3sm_to_cmip"] = "False" + + # Dependencies + expansions["active_climo_month_atm"] = "False" # For e3sm_diags + expansions["active_climo_month_lnd"] = "False" # For e3sm_diags + expansions["active_climo_month_lnd_for_livvkit"] = "False" # For livvkit + expansions["active_climo_diurnal_atm"] = "False" # For e3sm_diags + + expansions["active_ts_daily_atm"] = "False" # For e3sm_diags + expansions["active_ts_month_atm"] = "False" # For e3sm_diags, ilamb, pcmdi_diags + expansions["active_ts_month_atm_glb"] = "False" # For global_time_series + expansions["active_ts_month_lnd"] = "False" # For ilamb + expansions["active_ts_month_lnd_for_livvkit"] = "False" # For livvkit + expansions["active_ts_month_lnd_glb"] = "False" # For global_time_series + expansions["active_ts_month_rof"] = "False" # For e3sm_diags + + expansions["active_e3sm_to_cmip_month_atm"] = "False" # For ilamb, pcmdi_diags + expansions["active_e3sm_to_cmip_month_lnd"] = "False" # For ilamb + + expansions["active_tc_analysis"] = "False" # For e3sm_diags + + # Plotting packages expansions["active_e3sm_diags"] = "False" - expansions["active_mpas_analysis"] = "False" + expansions["active_mpas_analysis"] = "False" # Also used for global_time_series expansions["active_global_time_series"] = "False" expansions["active_ilamb"] = "False" expansions["active_livvkit"] = "False" expansions["active_pcmdi_diags"] = "False" + if "e3sm_diags" in TEST_SPECIFICS["tasks_to_run"]: expansions["active_e3sm_diags"] = "True" + + expansions["active_climo_month_atm"] = "True" + expansions["active_climo_month_lnd"] = "True" + expansions["active_climo_diurnal_atm"] = "True" + + expansions["active_ts_month_atm"] = "True" + expansions["active_ts_month_rof"] = "True" + expansions["active_ts_daily_atm"] = "True" + + expansions["active_tc_analysis"] = "True" + if "mpas_analysis" in TEST_SPECIFICS["tasks_to_run"]: expansions["active_mpas_analysis"] = "True" + if "global_time_series" in TEST_SPECIFICS["tasks_to_run"]: expansions["active_global_time_series"] = "True" - expansions["active_mpas_analysis"] = "True" # For ocn plots - expansions["active_e3sm_to_cmip"] = "True" # For lnd plots - if "livvkit" in TEST_SPECIFICS["tasks_to_run"]: - expansions["active_livvkit"] = "True" + + expansions["active_ts_month_atm_glb"] = "True" + expansions["active_ts_month_lnd_glb"] = "True" + + expansions["active_mpas_analysis"] = "True" + if "ilamb" in TEST_SPECIFICS["tasks_to_run"]: expansions["active_ilamb"] = "True" - expansions["active_e3sm_to_cmip"] = "True" + + expansions["active_ts_month_atm"] = "True" + expansions["active_ts_month_lnd"] = "True" + + expansions["active_e3sm_to_cmip_month_atm"] = "True" + expansions["active_e3sm_to_cmip_month_lnd"] = "True" + + if "livvkit" in TEST_SPECIFICS["tasks_to_run"]: + expansions["active_livvkit"] = "True" + + expansions["active_climo_month_lnd_for_livvkit"] = "True" + + expansions["active_ts_month_lnd_for_livvkit"] = "True" + if "pcmdi_diags" in TEST_SPECIFICS["tasks_to_run"]: expansions["active_pcmdi_diags"] = "True" - expansions["active_e3sm_to_cmip"] = "True" + + expansions["active_ts_month_atm"] = "True" + + expansions["active_e3sm_to_cmip_month_atm"] = "True" + expansions["cfgs_to_run"] = TEST_SPECIFICS["cfgs_to_run"] expansions["tasks_to_run"] = TEST_SPECIFICS["tasks_to_run"] diff --git a/tests/main_branch_testing/README.md b/tests/main_branch_testing/README.md new file mode 100644 index 00000000..0e9f2d8a --- /dev/null +++ b/tests/main_branch_testing/README.md @@ -0,0 +1,270 @@ +# zppy Integration Test Automation + +This automation system streamlines the zppy integration testing workflow, reducing manual steps and wait time while maintaining quality control checkpoints. + +## Quick Start + +### Basic Usage (Interactive Mode) +```bash +./run_integration_test.bash --config zppy_test.cfg +``` + +### Skip Straight to a Later Phase +Set `START_PHASE=2` or `START_PHASE=3` in your config file, then re-run: +```bash +./run_integration_test.bash --config zppy_test.cfg +``` + +### Non-Interactive (Auto) Mode +Set `AUTO_MODE=true` in your config file. All checkpoints are bypassed and the script runs end-to-end without waiting for user input. + +## Configuration + +Copy `zppy_test.cfg` and edit it before each test run. It has four sections: + +### Runtime settings (update as needed each run) + +| Variable | Description | +| --- | --- | +| `MACHINE` | `chrysalis`, `compy`, or `perlmutter` | +| `START_PHASE` | `1`, `2`, or `3` | +| `AUTO_MODE` | `true` to skip all interactive checkpoints | +| `EXPLICIT_TAG` | Leave empty to auto-generate; set to a prior TAG to resume | +| `RUN_NUMBER` | Increment if you run multiple tests on the same day | +| `DIAGS_BASE_BRANCH` | Branch to test for e3sm_diags (usually `main`) | +| `E3SM_TO_CMIP_BASE_BRANCH` | Branch to test for e3sm_to_cmip (usually `master`) | +| `MPAS_BASE_BRANCH` | Branch to test for MPAS-Analysis (usually `develop`) | +| `ZI_BASE_BRANCH` | Branch to test for zppy-interfaces (usually `main`) | +| `ZPPY_BASE_BRANCH` | Branch to test for zppy (usually `main`) | +| `DIAGS_ENV_TYPE` | `"dev"` to build a dedicated conda env; `"unified"` to use e3sm-unified | +| `E3SM_TO_CMIP_ENV_TYPE` | `"dev"` to build a dedicated conda env; `"unified"` to use e3sm-unified | +| `MPAS_ENV_TYPE` | `"dev"` to build a dedicated conda env; `"unified"` to use e3sm-unified | +| `ZI_ENV_TYPE` | `"dev"` to build a dedicated conda env; `"unified"` to use e3sm-unified | +| `DIAGS_EXISTING_ENV` | *(optional)* Name of an existing conda env to reuse for e3sm_diags; skips creation | +| `E3SM_TO_CMIP_EXISTING_ENV` | *(optional)* Name of an existing conda env to reuse for e3sm_to_cmip; skips creation | +| `MPAS_EXISTING_ENV` | *(optional)* Name of an existing conda env to reuse for MPAS-Analysis; skips creation | +| `ZI_EXISTING_ENV` | *(optional)* Name of an existing conda env to reuse for zppy-interfaces; skips creation | +| `ZPPY_EXISTING_ENV` | *(optional)* Name of an existing conda env to reuse for zppy; skips creation | +| `CFGS_TO_RUN` | Comma-separated list of zppy cfg names to generate and submit (see below) | +| `TASKS_TO_RUN` | Comma-separated list of tasks to enable (e.g. `e3sm_diags,mpas_analysis`) | + +The `*_EXISTING_ENV` variables only take effect when the corresponding `*_ENV_TYPE` is `"dev"`. When set, the script skips conda env creation entirely and activates the named env directly (still running `pip install .` to pick up any local changes). Leave them empty to let the script auto-name and create environments as usual. + +Example — reuse envs from a prior run on the same day: +``` +DIAGS_EXISTING_ENV="test-diags-main-20250601_run1" +E3SM_TO_CMIP_EXISTING_ENV="test-e3sm-to-cmip-master-20250601_run1" +MPAS_EXISTING_ENV="test-mpas-develop-20250601_run1" +ZI_EXISTING_ENV="test-zi-main-20250601_run1" +ZPPY_EXISTING_ENV="test-zppy-main-20250601_run1" +``` + +`CFGS_TO_RUN` values correspond to generated filenames `test__.cfg`. Any name containing `bundle` is automatically treated as a bundle cfg and re-submitted in Phase 2. Example to run only the v3 comprehensive and bundle cfgs: +``` +CFGS_TO_RUN="weekly_bundles,weekly_comprehensive_v3,weekly_legacy_3.1.0_bundles,weekly_legacy_3.1.0_comprehensive_v3,weekly_legacy_3.0.0_bundles,weekly_legacy_3.0.0_comprehensive_v3" +``` + +### Frozen dependency base (isolates image-diff root causes) + +`test_images.py` can show diffs for `e3sm_diags` or `pcmdi_diags` plots that are actually caused by an unrelated dependency (e.g. `matplotlib` moving a plot a pixel to the right) rather than by the package under test. To isolate root causes, any component listed in `FROZEN_BASE_COMPONENTS` gets its **own dedicated, fully-resolved conda env**, built once per run from that component's lock file. A test env for that component is then created by **cloning** its frozen base (fast — no solve, no network) and `pip install`ing the branch under test on top, so every dependency other than that component's own code stays pinned to exactly what's in its lock file. + +Each component's frozen base is independent — `e3sm_diags`'s cdat-based stack, `mpas_analysis`'s ESMF/MPAS libs, and `e3sm_to_cmip`'s `cmor` never need to be resolved into one shared environment together. + +| Variable | Description | +| --- | --- | +| `FREEZE_DEPENDENCIES` | Master switch. `false` restores the original behavior for every component (solve its own `dev.yml` fresh each run) regardless of the settings below. | +| `FROZEN_BASE_COMPONENTS` | Comma-separated subset of `e3sm_to_cmip,e3sm_diags,mpas_analysis,zppy_interfaces,zppy` that should use a frozen base. Defaults to all five, since image diffs can come from any plotting-adjacent tool, not just `e3sm_diags`/`pcmdi_diags`. | +| `BASE_ENV_LOCK_FILE_` | Path to a fully-resolved environment spec (uppercase component name, e.g. `BASE_ENV_LOCK_FILE_E3SM_DIAGS`). Required for every component listed in `FROZEN_BASE_COMPONENTS`. | + +**Generating/updating a lock file** for a component (repeat per component in `FROZEN_BASE_COMPONENTS`): +```bash +conda env create -f -n tmp-lock-gen +conda activate tmp-lock-gen +conda list --explicit > $EZ_DIR/frozen-base-.txt +conda deactivate && conda remove --yes --all --name tmp-lock-gen +``` +A `dev.yml` is **not** sufficient on its own — it re-solves and can drift between runs even when unmodified, which is exactly the noise this feature exists to remove. Regenerate a component's lock file whenever that component's branch genuinely needs a new or updated dependency; the script will tell you when that's happened (see below). + +If a component's branch pulls in a dependency beyond what's frozen, `pip install .` still installs it — the script just makes that visible instead of letting it pass silently. It snapshots `conda list --explicit` before and after `pip install .` for every frozen component and logs a warning with the diff (also saved to `pre_install__.txt` / `post_install__.txt` in the run directory) whenever the two don't match. + +### One-time setup (paths that rarely change) + +| Variable | Description | +| --- | --- | +| `HOME_DIR` | Your home directory (default: `$HOME`) | +| `EZ_DIR` | Parent directory for all repos (default: `$HOME/ez`) | +| `E3SM_DIAGS_DIR` | Path to e3sm_diags repo | +| `E3SM_TO_CMIP_DIR` | Path to e3sm_to_cmip repo | +| `MPAS_ANALYSIS_DIR` | Path to MPAS-Analysis repo | +| `ZPPY_INTERFACES_DIR` | Path to zppy-interfaces repo | +| `ZPPY_DIR` | Path to zppy repo | +| `CONDA_PROFILE` | Path to your conda profile script | +| `TAG_CACHE_FILE` | Where the TAG is saved between phases (default: `~/.zppy_test_tag`) | + +Machine-specific settings (`OUTPUT_WORKSPACE`, conda activation command, unified environment path, `salloc` command) are derived automatically from `MACHINE` and do not appear in the config. + +## Workflow Phases + +### Phase 1: Setup +- Creates conda environments for each component where `ENV_TYPE="dev"` and no `*_EXISTING_ENV` is set; reuses the named env when `*_EXISTING_ENV` is set; skips env handling entirely for any component where `ENV_TYPE="unified"` + - For components in `FROZEN_BASE_COMPONENTS` (when `FREEZE_DEPENDENCIES=true`), the env is created by cloning that component's dedicated frozen base instead of solving `dev.yml` fresh +- Runs unit tests for zppy-interfaces and zppy +- Patches `tests/integration/utils.py` with test-specific environment commands, config list, and unique ID +- Generates config files via `python tests/integration/utils.py` +- Submits all 9 initial SLURM jobs (3 current + 3 legacy 3.1.0 + 3 legacy 3.0.0) +- Waits for jobs to complete (polls every 10 min, 4-hour max) + +### Phase 2: Bundles Part 2 +- Checks status files for all three bundles output directories; warns if any are non-OK before proceeding +- Submits bundles part 2 jobs (`weekly_bundles`, `legacy_3.1.0_bundles`, `legacy_3.0.0_bundles`) +- Waits for completion (polls every 10 min, 1-hour max) + +### Phase 3: Validation +- Checks status files for all 9 output directories +- Runs pytest integration tests: + - `test_last_year.py` + - `test_bash_generation.py` + - `test_campaign.py` + - `test_defaults.py` + - `test_bundles.py` +- Prints instructions for running `test_images.py` manually on a compute node + +## Output and Logs + +The script provides color-coded output: +- 🔵 **Blue**: Informational messages +- 🟢 **Green**: Success messages +- 🟡 **Yellow**: Warnings and checkpoints +- 🔴 **Red**: Errors + +### SLURM Job Monitoring +The script automatically monitors SLURM jobs and shows: +``` +Jobs remaining: 42 (elapsed: 1234s / max: 14400s) +``` +If all remaining jobs enter `DependencyNeverSatisfied`, they are cancelled automatically and the script exits with an error. + +### Status File Checking +Automatically checks for non-OK entries in all status directories: +``` +✓ v2: All status files OK +✓ Legacy 3.1.0 v2: All status files OK +✓ Legacy 3.0.0 v2: All status files OK +... +``` + +## Customization + +### Adjust Timeouts + +```bash +# In phase_1_setup(): +wait_for_slurm_jobs 600 14400 # Check every 10 min, max 4 hours + +# In phase_2_bundles_part2(): +wait_for_slurm_jobs 600 3600 # Check every 10 min, max 1 hour +``` + +## Troubleshooting + +### Script Exits Early +Check the error message. The script uses `set -e`, so it exits on any error. Common causes: +- A unit test failure during Phase 1 setup +- A SLURM timeout (increase the max-wait argument to `wait_for_slurm_jobs`) +- `DependencyNeverSatisfied` on all queued jobs (check your cfg files and SLURM account) +- `FREEZE_DEPENDENCIES=true` with a missing/not-yet-generated `BASE_ENV_LOCK_FILE_` for a component in `FROZEN_BASE_COMPONENTS` (the error message names the exact variable and gives the `conda list --explicit` command to generate it) + +Phase 2 checks bundle status files before resubmitting and warns if any are non-OK. Resolve any failures in the Phase 1 bundle runs before proceeding. You can restart from Phase 2 by setting `START_PHASE=2` in your config (the TAG from Phase 1 is saved in `~/.zppy_test_tag` and picked up automatically, or set `EXPLICIT_TAG` to be explicit): +```bash +./run_integration_test.bash --config zppy_test.cfg +``` + +### Environment Issues +```bash +# Remove and rebuild stale environments (only applies to components with ENV_TYPE="dev" +# and no *_EXISTING_ENV set) +conda remove --yes --all --name test-e3sm-to-cmip-master-YYYYMMDD_runN +conda remove --yes --all --name test-diags-main-YYYYMMDD_runN +conda remove --yes --all --name test-mpas-develop-YYYYMMDD_runN +conda remove --yes --all --name test-zi-main-YYYYMMDD_runN +conda remove --yes --all --name test-zppy-main-YYYYMMDD_runN + +# If using FREEZE_DEPENDENCIES=true, also remove that run's frozen base envs +# (one per component in FROZEN_BASE_COMPONENTS): +conda remove --yes --all --name test-frozen-base-e3sm_to_cmip-YYYYMMDD_runN +conda remove --yes --all --name test-frozen-base-e3sm_diags-YYYYMMDD_runN +conda remove --yes --all --name test-frozen-base-mpas_analysis-YYYYMMDD_runN +conda remove --yes --all --name test-frozen-base-zppy_interfaces-YYYYMMDD_runN +conda remove --yes --all --name test-frozen-base-zppy-YYYYMMDD_runN + +# Then re-run from Phase 1 +./run_integration_test.bash --config zppy_test.cfg +``` + +## Files Created + +``` +~/ez/zppy/ +├── tests/integration/generated/ +│ ├── test_weekly_bundles_chrysalis.cfg +│ ├── test_weekly_comprehensive_v2_chrysalis.cfg +│ ├── test_weekly_comprehensive_v3_chrysalis.cfg +│ ├── test_weekly_legacy_3.1.0_bundles_chrysalis.cfg +│ ├── test_weekly_legacy_3.1.0_comprehensive_v2_chrysalis.cfg +│ ├── test_weekly_legacy_3.1.0_comprehensive_v3_chrysalis.cfg +│ ├── test_weekly_legacy_3.0.0_bundles_chrysalis.cfg +│ ├── test_weekly_legacy_3.0.0_comprehensive_v2_chrysalis.cfg +│ └── test_weekly_legacy_3.0.0_comprehensive_v3_chrysalis.cfg +└── test_images_summary.md + +/lcrc/group/e3sm// +├── zppy_weekly_bundles_output/zppy_main_branch_test_YYYYMMDD_runN/ +├── zppy_weekly_comprehensive_v2_output/zppy_main_branch_test_YYYYMMDD_runN/ +├── zppy_weekly_comprehensive_v3_output/zppy_main_branch_test_YYYYMMDD_runN/ +├── zppy_weekly_legacy_3.1.0_bundles_output/zppy_main_branch_test_YYYYMMDD_runN/ +├── zppy_weekly_legacy_3.1.0_comprehensive_v2_output/zppy_main_branch_test_YYYYMMDD_runN/ +├── zppy_weekly_legacy_3.1.0_comprehensive_v3_output/zppy_main_branch_test_YYYYMMDD_runN/ +├── zppy_weekly_legacy_3.0.0_bundles_output/zppy_main_branch_test_YYYYMMDD_runN/ +├── zppy_weekly_legacy_3.0.0_comprehensive_v2_output/zppy_main_branch_test_YYYYMMDD_runN/ +└── zppy_weekly_legacy_3.0.0_comprehensive_v3_output/zppy_main_branch_test_YYYYMMDD_runN/ +``` + +If `FREEZE_DEPENDENCIES=true`, each frozen component's run directory also gains: +``` +pre_install__.txt # `conda list --explicit` right after cloning the frozen base +post_install__.txt # `conda list --explicit` right after `pip install .` +``` +These only differ if the branch under test pulled in a new or updated dependency — check `integration_test.log` for a warning with the diff when that happens. + +## Example Run + +```bash +# Confirm repos have no uncommitted changes +cd ~/ez/e3sm_to_cmip && git status +cd ~/ez/e3sm_diags && git status +cd ~/ez/MPAS-Analysis && git status +cd ~/ez/zppy-interfaces && git status +cd ~/ez/zppy && git status + +# Confirm no jobs are currently running +squeue -u $USER + +# Copy the script and config out of the repo (Phase 1 will change branches) +mkdir -p ~/ez/zppy_main_branch_tests/test_YYYYMMDD +cd ~/ez/zppy_main_branch_tests/test_YYYYMMDD +cp ~/ez/zppy/tests/main_branch_testing/run_integration_test.bash . +cp ~/ez/zppy/tests/main_branch_testing/zppy_test.cfg . + +# Edit configuration parameters +emacs zppy_test.cfg + +# Run inside a screen session so it will survive disconnections. +screen +cd ~/ez/zppy_main_branch_tests/test_YYYYMMDD +time ./run_integration_test.bash --config zppy_test.cfg 2>&1 | tee integration_test.log +# Ctrl-A D to detach from screen + +# Monitor progress from another terminal +screen -ls # Find the screen session +tail -f integration_test.log # Follow log output +``` diff --git a/tests/main_branch_testing/run_integration_test.bash b/tests/main_branch_testing/run_integration_test.bash new file mode 100755 index 00000000..8b8f944a --- /dev/null +++ b/tests/main_branch_testing/run_integration_test.bash @@ -0,0 +1,1063 @@ +#!/bin/bash +# zppy Integration Test Automation Script +# +# Usage: +# 1. Copy this file AND the sample config OUT of the zppy repo +# (this script will change branches). +# 2. Edit your config file (see zppy_test.cfg). +# 3. Run: ./run_integration_test.bash --config path/to/your.cfg +# +# Phases (set START_PHASE in your config): +# 1 - Full setup: build envs, run unit tests, generate configs, submit SLURM jobs +# 2 - Bundles Part 2 (run after Phase 1 jobs finish) +# 3 - Validation: status checks + pytest integration tests +# +# Notes: +# - test_images.py must be run manually from a compute node (see Phase 3 output). +# - To resume from Phase 2 or 3 on a later day, set EXPLICIT_TAG in your config +# to the TAG printed at the start of Phase 1 (or stored in ~/.zppy_test_tag), +# and set START_PHASE accordingly. +# +# Frozen dependency base (see FREEZE_DEPENDENCIES in the config): +# - Each component listed in FROZEN_BASE_COMPONENTS gets its OWN dedicated, +# fully-resolved conda env (BASE_ENV_LOCK_FILE_) instead of +# solving its dev.yml fresh every run. A test env for that component is +# then built by cloning its frozen base and `pip install`ing the branch +# under test on top, so unrelated dependency drift (e.g. matplotlib) +# can't produce false-positive diffs in test_images.py. See the README +# section "Regenerating a component's frozen base lock file" for how/when +# to update a BASE_ENV_LOCK_FILE_ entry. + +set -e # Exit on error +set -u # Exit on undefined variable + +SCRIPT_RUN_DIR="$PWD" + +# ============================================================================ +# Parse arguments +# ============================================================================ + +CONFIG_FILE="" + +while [[ $# -gt 0 ]]; do + case "$1" in + --config) CONFIG_FILE="$2"; shift 2 ;; + *) echo "Unknown argument: $1"; exit 1 ;; + esac +done + +if [[ -z "$CONFIG_FILE" ]]; then + echo "Error: --config is required." + echo "Usage: $0 --config path/to/your.cfg" + exit 1 +fi + +if [[ ! -f "$CONFIG_FILE" ]]; then + echo "Error: Config file not found: $CONFIG_FILE" + exit 1 +fi + +# Source the config. Variables defined there become the script's environment. +# shellcheck disable=SC1090 +source "$CONFIG_FILE" + +# Validate required config keys. +_required_vars=( + MACHINE START_PHASE AUTO_MODE EXPLICIT_TAG + RUN_NUMBER + DIAGS_BASE_BRANCH E3SM_TO_CMIP_BASE_BRANCH MPAS_BASE_BRANCH ZI_BASE_BRANCH ZPPY_BASE_BRANCH + DIAGS_ENV_TYPE E3SM_TO_CMIP_ENV_TYPE MPAS_ENV_TYPE ZI_ENV_TYPE + CFGS_TO_RUN TASKS_TO_RUN + HOME_DIR EZ_DIR + E3SM_DIAGS_DIR E3SM_TO_CMIP_DIR MPAS_ANALYSIS_DIR ZPPY_INTERFACES_DIR ZPPY_DIR + CONDA_PROFILE TAG_CACHE_FILE +) +_missing=() +for _var in "${_required_vars[@]}"; do + if [[ -z "${!_var+x}" ]]; then + _missing+=("$_var") + fi +done +if [[ ${#_missing[@]} -gt 0 ]]; then + echo "Error: The following required variables are missing from ${CONFIG_FILE}:" + printf ' %s\n' "${_missing[@]}" + exit 1 +fi + +# Apply defaults for optional *_EXISTING_ENV variables so the rest of the +# script can reference them unconditionally. +DIAGS_EXISTING_ENV="${DIAGS_EXISTING_ENV:-}" +E3SM_TO_CMIP_EXISTING_ENV="${E3SM_TO_CMIP_EXISTING_ENV:-}" +MPAS_EXISTING_ENV="${MPAS_EXISTING_ENV:-}" +ZI_EXISTING_ENV="${ZI_EXISTING_ENV:-}" +ZPPY_EXISTING_ENV="${ZPPY_EXISTING_ENV:-}" + +# Apply defaults for the optional frozen-dependency-base variables. +FREEZE_DEPENDENCIES="${FREEZE_DEPENDENCIES:-false}" +FROZEN_BASE_COMPONENTS="${FROZEN_BASE_COMPONENTS:-}" + +# Validate MACHINE value. +case "$MACHINE" in + chrysalis|compy|perlmutter) ;; + *) echo "Error: Unknown MACHINE '${MACHINE}'. Valid values: chrysalis | compy | perlmutter"; exit 1 ;; +esac + +# ============================================================================ +# Machine-specific settings +# ============================================================================ + +case "$MACHINE" in + chrysalis) + OUTPUT_WORKSPACE="/lcrc/group/e3sm/${USER}" + CONDA_ACTIVATION_CMD="lcrc_conda" + UNIFIED_ENV_CMD="source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_chrysalis.sh" + SALLOC_CMD="salloc --nodes=1 --partition=debug --time=02:00:00 --account=e3sm" + ;; + compy) + OUTPUT_WORKSPACE="/compyfs/${USER}" + CONDA_ACTIVATION_CMD="compy_conda" + UNIFIED_ENV_CMD="source /share/apps/E3SM/conda_envs/load_latest_e3sm_unified_compy.sh" + SALLOC_CMD="salloc --nodes=1 --partition=short --time=01:00:00 --account=e3sm" + ;; + perlmutter) + OUTPUT_WORKSPACE="/global/cfs/cdirs/e3sm/${USER}" + CONDA_ACTIVATION_CMD="nersc_conda" + UNIFIED_ENV_CMD="source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_pm-cpu.sh" + SALLOC_CMD="salloc --nodes=1 --qos=interactive --time=01:00:00 --constraint=cpu --account=e3sm" + ;; +esac + +# Derive the filename suffix used by generated zppy cfg files. +case "$MACHINE" in + chrysalis) MACHINE_CFG_SUFFIX="chrysalis" ;; + compy) MACHINE_CFG_SUFFIX="compy" ;; + perlmutter) MACHINE_CFG_SUFFIX="pm-cpu" ;; +esac + +# Split comma-separated config lists into bash arrays. +# IFS = Internal Field Separator +IFS=',' read -ra CFGS_ARRAY <<< "$CFGS_TO_RUN" +IFS=',' read -ra TASKS_ARRAY <<< "$TASKS_TO_RUN" +IFS=',' read -ra FROZEN_BASE_ARRAY <<< "$FROZEN_BASE_COMPONENTS" +# ============================================================================ +# +# Priority: +# 1. EXPLICIT_TAG from config (always wins when non-empty) +# 2. $TAG_CACHE_FILE written by a prior Phase 1 run (auto-resume) +# 3. Fresh timestamp (Phase 1 first run) +# +# Phase 1 always writes the resolved TAG to $TAG_CACHE_FILE so later phases +# can pick it up automatically without needing EXPLICIT_TAG. + +if [[ -n "$EXPLICIT_TAG" ]]; then + TAG="$EXPLICIT_TAG" + DATE_STAMP="${TAG%%_run*}" +elif [[ "$START_PHASE" -gt 1 && -f "$TAG_CACHE_FILE" ]]; then + TAG="$(cat "$TAG_CACHE_FILE")" + DATE_STAMP="${TAG%%_run*}" + echo "Loaded TAG from ${TAG_CACHE_FILE}: ${TAG}" + echo "(Set EXPLICIT_TAG in your config to override.)" +else + DATE_STAMP="$(date +%Y%m%d)" + TAG="${DATE_STAMP}_run${RUN_NUMBER}" +fi + +# ============================================================================ +# Derived (probably no edits needed) +# ============================================================================ + +UNIQUE_ID="zppy_main_branch_test_${TAG}" + +ZPPY_ENV="test-zppy-${ZPPY_BASE_BRANCH}-${TAG}" + +# Output directories (status file locations) +BUNDLES_OUTPUT="${OUTPUT_WORKSPACE}/zppy_weekly_bundles_output/${UNIQUE_ID}/v3.LR.historical_0051/post/scripts" +LEGACY_310_BUNDLES_OUTPUT="${OUTPUT_WORKSPACE}/zppy_weekly_legacy_3.1.0_bundles_output/${UNIQUE_ID}/v3.LR.historical_0051/post/scripts" +LEGACY_300_BUNDLES_OUTPUT="${OUTPUT_WORKSPACE}/zppy_weekly_legacy_3.0.0_bundles_output/${UNIQUE_ID}/v3.LR.historical_0051/post/scripts" +V2_OUTPUT="${OUTPUT_WORKSPACE}/zppy_weekly_comprehensive_v2_output/${UNIQUE_ID}/v2.LR.historical_0201/post/scripts" +LEGACY_310_V2_OUTPUT="${OUTPUT_WORKSPACE}/zppy_weekly_legacy_3.1.0_comprehensive_v2_output/${UNIQUE_ID}/v2.LR.historical_0201/post/scripts" +LEGACY_300_V2_OUTPUT="${OUTPUT_WORKSPACE}/zppy_weekly_legacy_3.0.0_comprehensive_v2_output/${UNIQUE_ID}/v2.LR.historical_0201/post/scripts" +V3_OUTPUT="${OUTPUT_WORKSPACE}/zppy_weekly_comprehensive_v3_output/${UNIQUE_ID}/v3.LR.historical_0051/post/scripts" +LEGACY_310_V3_OUTPUT="${OUTPUT_WORKSPACE}/zppy_weekly_legacy_3.1.0_comprehensive_v3_output/${UNIQUE_ID}/v3.LR.historical_0051/post/scripts" +LEGACY_300_V3_OUTPUT="${OUTPUT_WORKSPACE}/zppy_weekly_legacy_3.0.0_comprehensive_v3_output/${UNIQUE_ID}/v3.LR.historical_0051/post/scripts" + +# Colors for output +RED='\033[0;31m' +GREEN='\033[0;32m' +YELLOW='\033[1;33m' +BLUE='\033[0;34m' +NC='\033[0m' # No Color + +# ============================================================================ +# Helper Functions +# ============================================================================ + +log() { + echo -e "${BLUE}[$(date +'%Y-%m-%d %H:%M:%S')]${NC} $*" +} + +log_success() { + echo -e "${GREEN}[$(date +'%Y-%m-%d %H:%M:%S')] ✓${NC} $*" +} + +log_warning() { + echo -e "${YELLOW}[$(date +'%Y-%m-%d %H:%M:%S')] ⚠${NC} $*" +} + +log_error() { + echo -e "${RED}[$(date +'%Y-%m-%d %H:%M:%S')] ✗${NC} $*" +} + +checkpoint() { + local message="$1" + if [ "$AUTO_MODE" = false ]; then + log_warning "CHECKPOINT: $message" + read -rp "Press Enter to continue or Ctrl+C to abort..." + else + log "AUTO MODE: Passing checkpoint -- $message" + fi +} + +# ============================================================================ +# Debug Helper Functions +# ============================================================================ + +# Capture a full environment snapshot to both stdout and a log file. +# Usage: debug_env_snapshot "label-string" +debug_env_snapshot() { + local label="$1" + local out_file="${SCRIPT_RUN_DIR}/debug_env_${TAG}.txt" + { + echo "========================================" + echo "ENV SNAPSHOT: $label" + echo "Timestamp: $(date)" + echo "----------------------------------------" + echo "--- conda info ---" + conda info 2>/dev/null || echo "(conda not available)" + echo "--- CONDA_DEFAULT_ENV: ${CONDA_DEFAULT_ENV:-}" + echo "--- CONDA_PREFIX: ${CONDA_PREFIX:-}" + echo "--- Active python: $(which python 2>/dev/null || echo '')" + echo "--- Python version: $(python --version 2>/dev/null || echo '')" + echo "--- LD_LIBRARY_PATH: ${LD_LIBRARY_PATH:-}" + echo "--- PYTHONPATH: ${PYTHONPATH:-}" + echo "--- PYTHONHOME: ${PYTHONHOME:-}" + echo "--- PATH (first 5 entries):" + echo "$PATH" | tr ':' '\n' | head -5 + echo "--- ulimit -s (stack size): $(ulimit -s)" + echo "--- ulimit -v (virtual mem): $(ulimit -v)" + echo "--- OMP_NUM_THREADS: ${OMP_NUM_THREADS:-}" + echo "--- MKL_NUM_THREADS: ${MKL_NUM_THREADS:-}" + echo "--- OPENBLAS_NUM_THREADS: ${OPENBLAS_NUM_THREADS:-}" + echo "========================================" + } | tee -a "$out_file" +} + +# Capture MPAS-specific preflight diagnostics to both stdout and a log file. +# Checks the mpas_analysis binary, shared library linkage, and key Python +# packages (numpy BLAS config, ESMF, cartopy) that are common segfault sources. +# Usage: debug_mpas_preflight "label-string" +debug_mpas_preflight() { + local label="$1" + local out_file="${SCRIPT_RUN_DIR}/debug_mpas_${TAG}.txt" + { + echo "========================================" + echo "MPAS PREFLIGHT: $label" + echo "Timestamp: $(date)" + echo "----------------------------------------" + echo "--- mpas_analysis binary: $(which mpas_analysis 2>/dev/null || echo '')" + echo "--- mpas_analysis version: $(mpas_analysis --version 2>/dev/null || echo '')" + echo "--- ldd on mpas_analysis binary (if applicable):" + _mpas_bin="$(which mpas_analysis 2>/dev/null || true)" + if [[ -n "$_mpas_bin" && -f "$_mpas_bin" ]]; then + ldd "$_mpas_bin" 2>/dev/null | grep -i "not found" && echo "(^ missing libs above)" || echo "(all libs found)" + else + echo "(mpas_analysis not a regular file or not found, skipping ldd)" + fi + echo "--- numpy config (segfaults often trace to BLAS/LAPACK mismatches):" + python -c "import numpy; numpy.show_config()" 2>/dev/null || echo "(numpy not importable)" + echo "--- ESMF version:" + python -c "import ESMF; print(ESMF.__version__)" 2>/dev/null || echo "(ESMF not importable)" + echo "--- cartopy version:" + python -c "import cartopy; print(cartopy.__version__)" 2>/dev/null || echo "(cartopy not importable)" + echo "========================================" + } | tee -a "$out_file" +} + +# Activate conda and (optionally) a named environment. +activate_env() { + local env_name="${1:-}" + set +u + # shellcheck disable=SC1090 + source ~/.bashrc + $CONDA_ACTIVATION_CMD # Machine-specific conda init (lcrc_conda / compy_conda / nersc_conda) + + if [ -n "$env_name" ]; then + conda activate "$env_name" + # Log every activation so we can spot env-stack contamination between subshells. + echo "[DEBUG activate_env] $(date) | activated: $env_name | python: $(which python 2>/dev/null || echo '') | CONDA_PREFIX: ${CONDA_PREFIX:-} | LD_LIBRARY_PATH: ${LD_LIBRARY_PATH:-}" \ + >> "${SCRIPT_RUN_DIR}/debug_activate_log_${TAG}.txt" + log "Installing/updating package in '$env_name'..." + python -m pip install . + fi + set -u +} + +# Activate the machine-specific unified environment. +# UNIFIED_ENV_CMD is always "source /path/to/script.sh" (set in the +# machine-specific case block above), so we strip the leading "source " +# and source the path directly -- no eval required. +activate_unified_env() { + set +u + # shellcheck disable=SC1090 + source ~/.bashrc + $CONDA_ACTIVATION_CMD + # shellcheck disable=SC1090 + source "${UNIFIED_ENV_CMD#source }" + set -u +} + +# Return 0 (true) if the given component key uses a dedicated frozen base +# env (rather than solving its own dev.yml fresh each run). component_key is +# one of: e3sm_to_cmip, e3sm_diags, mpas_analysis, zppy_interfaces, zppy +uses_frozen_base() { + local component="$1" + [[ "$FREEZE_DEPENDENCIES" == true ]] || return 1 + local c + for c in "${FROZEN_BASE_ARRAY[@]}"; do + c="${c// /}" + if [[ "$c" == "$component" ]]; then + return 0 + fi + done + return 1 +} + +# Resolve the lock file for a component's dedicated frozen base. +# Each component in FROZEN_BASE_COMPONENTS must have its own +# BASE_ENV_LOCK_FILE_ (uppercase) set in the config -- there is no +# shared/fallback lock file. Each component gets its own resolved environment +# because their dependency stacks don't need to (and may not be able to) +# co-resolve into one universal env -- e3sm_diags's cdat-based stack, +# mpas_analysis's ESMF/MPAS libs, and e3sm_to_cmip's cmor can each be frozen +# independently without needing to agree with one another. +get_component_lock_file() { + local component="$1" + local var="BASE_ENV_LOCK_FILE_${component^^}" + echo "${!var:-}" +} + +# Build (once per TAG) the dedicated, pinned base env for one component. +# The lock file must be a FULLY RESOLVED spec (e.g. the output of +# `conda list --explicit` from a known-good env for that component), not a +# dev.yml -- a dev.yml re-solves and can silently drift between runs even +# unmodified. +# +# This is what makes "freeze everything else" possible: every test env +# cloned from base_env_name starts from byte-identical package versions, so +# any image diff that shows up after `pip install .`-ing the branch under +# test can be attributed to that branch rather than to incidental dependency +# movement. +setup_base_env() { + local component="$1" + local base_env_name="$2" + local lock_file="$3" + + activate_env # Ensure conda itself is available + + if conda env list | grep -q "^${base_env_name} "; then + log "Base env '$base_env_name' already exists, skipping creation" + return 0 + fi + + if [[ -z "$lock_file" || ! -f "$lock_file" ]]; then + log_error "FREEZE_DEPENDENCIES=true and '${component}' is in FROZEN_BASE_COMPONENTS, but BASE_ENV_LOCK_FILE_${component^^} is missing or not found: '${lock_file}'" + log_error "Generate one from a known-good env for this component, e.g.:" + log_error " conda env create -f <${component}'s dev.yml> -n tmp-lock-gen" + log_error " conda activate tmp-lock-gen" + log_error " conda list --explicit > /path/to/frozen-base-${component}.txt" + log_error "Then set BASE_ENV_LOCK_FILE_${component^^} in your config to that path." + exit 1 + fi + + log "Creating dedicated frozen base '$base_env_name' for '${component}' from ${lock_file}..." + conda create --name "$base_env_name" --file "$lock_file" --yes + log_success "Base env '$base_env_name' ready" +} + +# Create (if needed) and activate a conda environment for a component. +# +# component_key: one of e3sm_to_cmip, e3sm_diags, mpas_analysis, +# zppy_interfaces, zppy -- used to check FROZEN_BASE_COMPONENTS +# via uses_frozen_base() and to look up this component's own +# BASE_ENV_LOCK_FILE_. +# conda_dir: directory containing dev.yml (e.g. "conda" or "conda-env"), +# or "none" to use dev-spec.txt instead. Ignored when this +# component is using its frozen base. +# env_name: name of the environment to create/activate. +setup_conda_env() { + local component_key="$1" + local conda_dir="$2" + local env_name="$3" + + activate_env # Ensure conda itself is available + + if conda env list | grep -q "^${env_name} "; then + log "Environment '$env_name' already exists, skipping creation" + elif uses_frozen_base "$component_key"; then + local base_env_name lock_file + base_env_name="test-frozen-base-${component_key}-${TAG}" + lock_file="$(get_component_lock_file "$component_key")" + setup_base_env "$component_key" "$base_env_name" "$lock_file" # no-op if it already exists this run + log "Cloning '$env_name' from '${component_key}'s frozen base '$base_env_name' (deps pinned)..." + conda create --name "$env_name" --clone "$base_env_name" --yes + else + log "Creating environment '$env_name' from ${conda_dir}/dev.yml..." + rm -rf build + conda clean --all --yes + if [[ "$conda_dir" == "none" ]]; then + conda create --name "$env_name" --file dev-spec.txt --yes + else + conda env create -f "${conda_dir}/dev.yml" -n "$env_name" + fi + fi + + # For frozen-base envs, snapshot the exact package set before the + # `pip install .` that activate_env() below performs, so we can tell + # whether the package under test pulled in anything beyond itself. + local _pre_install_file="${SCRIPT_RUN_DIR}/pre_install_${env_name}_${TAG}.txt" + if uses_frozen_base "$component_key"; then + conda list --explicit > "$_pre_install_file" 2>/dev/null || true + fi + + activate_env "$env_name" + log_success "Environment '$env_name' ready" + + if uses_frozen_base "$component_key"; then + local _post_install_file="${SCRIPT_RUN_DIR}/post_install_${env_name}_${TAG}.txt" + conda list --explicit > "$_post_install_file" 2>/dev/null || true + if [[ -f "$_pre_install_file" ]] && ! diff -q "$_pre_install_file" "$_post_install_file" > /dev/null 2>&1; then + log_warning "'$env_name': package set changed beyond the frozen base after 'pip install .'." + log_warning "This is expected ONLY if '${component_key}' itself added or bumped a dependency:" + diff "$_pre_install_file" "$_post_install_file" || true + fi + fi +} + + +# Checkout test branch, creating it from upstream/ if it doesn't exist. +# Stashes/commits any in-progress work first. +ensure_test_branch() { + local test_branch="$1" + local base_branch="$2" + local current_branch + current_branch=$(git rev-parse --abbrev-ref HEAD 2>/dev/null) + + if [ "$current_branch" = "$test_branch" ]; then + log "Already on branch '$test_branch'" + return 0 + fi + + log "Saving current work before switching branches..." + git add -A + git commit -m "Auto-save before test" --no-verify || true + + if git show-ref --verify --quiet "refs/heads/$test_branch"; then + log "Checking out existing branch '$test_branch'..." + git checkout "$test_branch" + else + log "Creating new branch '$test_branch' from upstream/${base_branch}..." + git fetch upstream "${base_branch}" + git checkout -b "$test_branch" "upstream/${base_branch}" + fi + log_success "On branch '$test_branch'" +} + +# Return the environment_commands string for a component. +# Usage: get_env_cmd "dev" "$ENV_NAME" +# get_env_cmd "unified" "" +get_env_cmd() { + local env_type="$1" + local env_name="$2" + if [[ "$env_type" == "dev" ]]; then + echo "source ${CONDA_PROFILE}; conda activate ${env_name}" + else + echo "$UNIFIED_ENV_CMD" + fi +} + +# Poll squeue until no user jobs remain (or timeout). +wait_for_slurm_jobs() { + local check_interval=${1:-600} # seconds between checks (default 10 min) + local max_wait=${2:-14400} # max total wait seconds (default 4 hours) + + log "Waiting for SLURM jobs to complete (checking every ${check_interval}s, max ${max_wait}s)..." + local elapsed=0 + local prev_failed_count=0 + + while true; do + local job_count + job_count=$(squeue -u "${USER}" | wc -l) + job_count=$((job_count - 1)) # subtract header + + # Detect DependencyNeverSatisfied + local failed_jobs + failed_jobs=$(squeue -u "${USER}" | grep "DependencyNeverSatisfied" || true) + local failed_count=0 + if [ -n "$failed_jobs" ]; then + failed_count=$(echo "$failed_jobs" | wc -l) + fi + + if [ "$failed_count" -gt "$prev_failed_count" ]; then + log_error "Jobs with DependencyNeverSatisfied:" + echo "$failed_jobs" + if [ "$job_count" -eq "$failed_count" ]; then + log_error "All remaining jobs have DependencyNeverSatisfied -- cancelling." + scancel -u "${USER}" + return 1 + fi + fi + prev_failed_count=$failed_count + + if [ "$job_count" -eq 0 ]; then + log_success "All SLURM jobs completed!" + return 0 + fi + + if [ "$elapsed" -ge "$max_wait" ]; then + log_error "Timeout after ${max_wait}s waiting for SLURM jobs" + log_error "This script is going to exit now. However, the jobs in the queue will NOT be terminated. Once they finish, you may re-invoke this script with START_PHASE=2 or START_PHASE=3 in your config to continue." + log_error " TAG for this run: ${TAG}" + log_error " Resume: set START_PHASE=2 and EXPLICIT_TAG=${TAG} in your config, then re-run." + log_error " (TAG is also saved in ${TAG_CACHE_FILE})" + return 1 + fi + + echo -ne "\r${YELLOW}Jobs remaining: $job_count${NC} (elapsed: ${elapsed}s / max: ${max_wait}s)" + sleep "$check_interval" + elapsed=$((elapsed + check_interval)) + done + echo "" +} + +# Grep status files in a directory for any non-OK lines. +# Returns 0 if all OK, 1 if any failures found. +check_status_files() { + local dir="$1" + local name="$2" + + if [ ! -d "$dir" ]; then + log_warning "$name: Directory not found: $dir" + return 1 + fi + + if ! compgen -G "${dir}/*status" > /dev/null; then + log_warning "$name: No status files found in ${dir}" + return 1 + fi + + local errors + errors=$(grep -v "OK" "${dir}"/*status 2>/dev/null || true) + + if [ -z "$errors" ]; then + log_success "$name: All status files OK in ${dir}" + return 0 + else + log_error "$name: Non-OK statuses found in ${dir}:" + echo "$errors" + return 1 + fi +} + +# ============================================================================ +# Phase 1: Environment Setup + Initial SLURM Jobs +# ============================================================================ + +phase_1_setup() { + # Save TAG immediately so later phases can find it even if the date changes. + echo "$TAG" > "$TAG_CACHE_FILE" + + log "=========================================" + log "Phase 1: Setup" + log "Config file: $CONFIG_FILE" + log "Date stamp: $DATE_STAMP" + log "TAG: $TAG (saved to ${TAG_CACHE_FILE})" + log "Unique ID: $UNIQUE_ID" + if [[ "$FREEZE_DEPENDENCIES" == true ]]; then + log "Frozen base: ENABLED for components: ${FROZEN_BASE_COMPONENTS} (each has its own dedicated frozen base env)" + else + log "Frozen base: disabled (each dev env solves its own dev.yml)" + fi + log "" + log "To resume from a later phase, set in your config:" + log " START_PHASE=2" + log " EXPLICIT_TAG=${TAG}" + log "=========================================" + + # Capture the baseline environment before any component setup so we have + # a clean reference to diff against later snapshots. + debug_env_snapshot "phase-1-start" + + # ------------------------------------------------------------------ + # e3sm_to_cmip + # ------------------------------------------------------------------ + log "Setting up e3sm_to_cmip..." + + local E3SM_TO_CMIP_ENV="" + if [[ "$E3SM_TO_CMIP_ENV_TYPE" == "dev" ]]; then + if [[ -n "$E3SM_TO_CMIP_EXISTING_ENV" ]]; then + E3SM_TO_CMIP_ENV="$E3SM_TO_CMIP_EXISTING_ENV" + else + E3SM_TO_CMIP_ENV="test-e3sm-to-cmip-${E3SM_TO_CMIP_BASE_BRANCH}-${TAG}" + fi + fi + + ( + cd "$E3SM_TO_CMIP_DIR" + ensure_test_branch "test_e3sm_to_cmip_${TAG}" "$E3SM_TO_CMIP_BASE_BRANCH" + + log "Latest e3sm_to_cmip commit (should match https://github.com/E3SM-Project/e3sm_to_cmip/commits/${E3SM_TO_CMIP_BASE_BRANCH}):" + git log -1 --oneline + + if [[ "$E3SM_TO_CMIP_ENV_TYPE" == "dev" ]]; then + if [[ -n "$E3SM_TO_CMIP_EXISTING_ENV" ]]; then + log "Reusing existing 'e3sm_to_cmip' env: $E3SM_TO_CMIP_EXISTING_ENV (skipping creation)" + activate_env "$E3SM_TO_CMIP_EXISTING_ENV" + else + setup_conda_env "e3sm_to_cmip" "conda-env" "$E3SM_TO_CMIP_ENV" + fi + else + log "Using unified env for e3sm_to_cmip (skipping conda env creation)" + fi + ) + + # ------------------------------------------------------------------ + # e3sm_diags + # ------------------------------------------------------------------ + log "Setting up e3sm_diags..." + + local DIAGS_ENV="" + if [[ "$DIAGS_ENV_TYPE" == "dev" ]]; then + if [[ -n "$DIAGS_EXISTING_ENV" ]]; then + DIAGS_ENV="$DIAGS_EXISTING_ENV" + else + DIAGS_ENV="test-diags-${DIAGS_BASE_BRANCH}-${TAG}" + fi + fi + + ( + cd "$E3SM_DIAGS_DIR" + ensure_test_branch "test_e3sm_diags_${TAG}" "$DIAGS_BASE_BRANCH" + + log "Latest e3sm_diags commit (should match https://github.com/E3SM-Project/e3sm_diags/commits/${DIAGS_BASE_BRANCH}):" + git log -1 --oneline + + if [[ "$DIAGS_ENV_TYPE" == "dev" ]]; then + if [[ -n "$DIAGS_EXISTING_ENV" ]]; then + log "Reusing existing 'e3sm_diags' env: $DIAGS_EXISTING_ENV (skipping creation)" + activate_env "$DIAGS_EXISTING_ENV" + else + setup_conda_env "e3sm_diags" "conda-env" "$DIAGS_ENV" + fi + else + log "Using unified env for e3sm_diags (skipping conda env creation)" + fi + ) + + # ------------------------------------------------------------------ + # MPAS-Analysis + # ------------------------------------------------------------------ + log "Setting up MPAS-Analysis..." + + # Snapshot the environment immediately before entering the MPAS subshell. + # Compare this against the post-activate snapshot to spot any leakage + # from the e3sm_to_cmip or e3sm_diags subshells above (LD_LIBRARY_PATH, + # CONDA_PREFIX nesting, PYTHONPATH, etc.). + debug_env_snapshot "before-mpas-subshell" + + local MPAS_ENV="" + if [[ "$MPAS_ENV_TYPE" == "dev" ]]; then + if [[ -n "$MPAS_EXISTING_ENV" ]]; then + MPAS_ENV="$MPAS_EXISTING_ENV" + else + MPAS_ENV="test-mpas-${MPAS_BASE_BRANCH}-${TAG}" + fi + fi + + ( + cd "$MPAS_ANALYSIS_DIR" + ensure_test_branch "test_mpas_${TAG}" "$MPAS_BASE_BRANCH" + + log "Latest MPAS-Analysis commit (should match https://github.com/MPAS-Dev/MPAS-Analysis/commits/${MPAS_BASE_BRANCH}):" + git log -1 --oneline + + if [[ "$MPAS_ENV_TYPE" == "dev" ]]; then + if [[ -n "$MPAS_EXISTING_ENV" ]]; then + log "Reusing existing 'MPAS-Analysis' env: $MPAS_EXISTING_ENV (skipping creation)" + activate_env "$MPAS_EXISTING_ENV" + # Snapshot after activating the pre-existing env. A mismatch + # between this and the before-mpas-subshell snapshot (especially + # in LD_LIBRARY_PATH) is a likely segfault cause. + debug_env_snapshot "mpas-subshell-after-activate-existing" + debug_mpas_preflight "mpas-existing-env" + else + setup_conda_env "mpas_analysis" "none" "$MPAS_ENV" + # Snapshot after a fresh env creation + activation. Check that + # CONDA_PREFIX is clean and LD_LIBRARY_PATH only contains paths + # from this env, not any prior one. + debug_env_snapshot "mpas-subshell-after-setup-conda" + debug_mpas_preflight "mpas-new-env" + fi + else + log "Using unified env for MPAS-Analysis (skipping conda env creation)" + # Snapshot even for the unified-env path -- the unified env loader + # modifies LD_LIBRARY_PATH in ways that can conflict with prior envs. + debug_env_snapshot "mpas-subshell-unified-env" + debug_mpas_preflight "mpas-unified-env" + fi + ) + + # ------------------------------------------------------------------ + # zppy-interfaces (includes unit tests) + # ------------------------------------------------------------------ + log "Setting up zppy-interfaces..." + + local ZI_ENV="" + if [[ "$ZI_ENV_TYPE" == "dev" ]]; then + if [[ -n "$ZI_EXISTING_ENV" ]]; then + ZI_ENV="$ZI_EXISTING_ENV" + else + ZI_ENV="test-zi-${ZI_BASE_BRANCH}-${TAG}" + fi + fi + + ( + cd "$ZPPY_INTERFACES_DIR" + ensure_test_branch "test_zi_${TAG}" "$ZI_BASE_BRANCH" + + log "Latest zppy-interfaces commit (should match https://github.com/E3SM-Project/zppy-interfaces/commits/${ZI_BASE_BRANCH}):" + git log -1 --oneline + + if [[ "$ZI_ENV_TYPE" == "dev" ]]; then + if [[ -n "$ZI_EXISTING_ENV" ]]; then + log "Reusing existing 'zppy-interfaces' env: $ZI_EXISTING_ENV (skipping creation)" + activate_env "$ZI_EXISTING_ENV" + else + setup_conda_env "zppy_interfaces" "conda" "$ZI_ENV" + fi + else + log "Using unified env for zppy-interfaces..." + activate_unified_env + fi + + log "Running zppy-interfaces unit tests..." + pytest tests/unit/global_time_series/test_*.py + pytest tests/unit/pcmdi_diags/test_*.py + log_success "zppy-interfaces unit tests passed" + ) + + # ------------------------------------------------------------------ + # zppy (includes unit tests + config generation) + # ------------------------------------------------------------------ + log "Setting up zppy..." + + # Resolve ZPPY_ENV name before the subshell so it's available for + # config generation and later phases. + if [[ -n "$ZPPY_EXISTING_ENV" ]]; then + ZPPY_ENV="$ZPPY_EXISTING_ENV" + fi + # (If ZPPY_EXISTING_ENV is empty, ZPPY_ENV retains the auto-generated + # name set at the top of the script.) + + ( + cd "$ZPPY_DIR" + ensure_test_branch "test_zppy_${TAG}" "$ZPPY_BASE_BRANCH" + + log "Latest zppy commit (should match https://github.com/E3SM-Project/zppy/commits/${ZPPY_BASE_BRANCH}):" + git log -1 --oneline + + if [[ -n "$ZPPY_EXISTING_ENV" ]]; then + log "Reusing existing 'zppy' env: $ZPPY_EXISTING_ENV (skipping creation)" + activate_env "$ZPPY_ENV" + else + setup_conda_env "zppy" "conda" "$ZPPY_ENV" + fi + + log "Running zppy unit tests..." + pytest tests/test_*.py + log_success "zppy unit tests passed" + ) + + # ------------------------------------------------------------------ + # Generate config files (update utils.py TEST_SPECIFICS, then run it) + # ------------------------------------------------------------------ + log "Generating config files..." + + local E3SM_TO_CMIP_CMD + local DIAGS_CMD + local MPAS_CMD + local ZI_CMD + E3SM_TO_CMIP_CMD=$(get_env_cmd "$E3SM_TO_CMIP_ENV_TYPE" "$E3SM_TO_CMIP_ENV") + DIAGS_CMD=$(get_env_cmd "$DIAGS_ENV_TYPE" "$DIAGS_ENV") + MPAS_CMD=$(get_env_cmd "$MPAS_ENV_TYPE" "$MPAS_ENV") + ZI_CMD=$(get_env_cmd "$ZI_ENV_TYPE" "$ZI_ENV") + + # Config generation and job submission run in the parent shell so that + # the zppy command is available and cd/env state is consistent. + cd "$ZPPY_DIR" + activate_env "$ZPPY_ENV" + ensure_test_branch "test_zppy_${TAG}" "$ZPPY_BASE_BRANCH" + + UTILS_FILE="tests/integration/utils.py" + + # Build Python list literals from the bash arrays for injection into the heredoc. + local CFGS_PY_LIST TASKS_PY_LIST cfg task + CFGS_PY_LIST="" + for cfg in "${CFGS_ARRAY[@]}"; do + cfg="${cfg// /}" # strip any accidental whitespace + CFGS_PY_LIST+=" \"${cfg}\","$'\n' + done + TASKS_PY_LIST="" + for task in "${TASKS_ARRAY[@]}"; do + task="${task// /}" + TASKS_PY_LIST+="\"${task}\", " + done + TASKS_PY_LIST="${TASKS_PY_LIST%, }" # strip trailing comma+space + + python - < "${SCRIPT_RUN_DIR}/env_${TAG}.txt" + local cfg cfg_path + for cfg in "${CFGS_ARRAY[@]}"; do + cfg="${cfg// /}" + cfg_path="tests/integration/generated/test_${cfg}_${MACHINE_CFG_SUFFIX}.cfg" + if [[ ! -f "$cfg_path" ]]; then + log_error "Config file not found: $cfg_path" + log_error "Check that each CFGS_TO_RUN entry matches a cfg name generated by tests/integration/utils.py (e.g., weekly_comprehensive_v3)." + exit 1 + fi + log "Submitting: $cfg_path" + zppy -c "$cfg_path" + done + + local job_count + job_count=$(squeue -u "${USER}" | wc -l) + job_count=$((job_count - 1)) + log_success "Submitted jobs. Current queue depth: $job_count" + + checkpoint "Phase 1 jobs submitted. Waiting for them to finish..." + wait_for_slurm_jobs 600 14400 # Check every 10 min, max 4 hours + + log_success "Phase 1 complete!" +} + +# ============================================================================ +# Phase 2: Bundles Part 2 +# ============================================================================ + +phase_2_bundles_part2() { + log "=========================================" + log "Phase 2: Bundles Part 2" + log "TAG: $TAG" + log "=========================================" + + cd "$ZPPY_DIR" + activate_env "$ZPPY_ENV" + ensure_test_branch "test_zppy_${TAG}" "$ZPPY_BASE_BRANCH" + + # Verify bundle status files are clean before submitting part 2. + # These paths are fixed regardless of CFGS_TO_RUN; skip any that don't exist yet. + log "Checking bundle status files before submitting part 2..." + local all_ok=true + check_status_files "$BUNDLES_OUTPUT" "Bundles" || all_ok=false + check_status_files "$LEGACY_310_BUNDLES_OUTPUT" "Legacy 3.1.0 Bundles" || all_ok=false + check_status_files "$LEGACY_300_BUNDLES_OUTPUT" "Legacy 3.0.0 Bundles" || all_ok=false + + if [ "$all_ok" = false ]; then + log_error "One or more bundle status files have non-OK entries." + checkpoint "Errors found. Continue anyway?" + else + log_success "Bundle status files look clean -- safe to submit part 2." + fi + + log "Submitting bundles part 2..." + local cfg cfg_path + for cfg in "${CFGS_ARRAY[@]}"; do + cfg="${cfg// /}" + if [[ "$cfg" == *bundle* ]]; then + cfg_path="tests/integration/generated/test_${cfg}_${MACHINE_CFG_SUFFIX}.cfg" + if [[ ! -f "$cfg_path" ]]; then + log_error "Config file not found: $cfg_path" + log_error "Check that each CFGS_TO_RUN entry matches a cfg name generated by tests/integration/utils.py (e.g., weekly_bundles)." + exit 1 + fi + log "Submitting: $cfg_path" + zppy -c "$cfg_path" + fi + done + + local job_count + job_count=$(squeue -u "${USER}" | wc -l) + job_count=$((job_count - 1)) + log_success "Bundles part 2 submitted. Current queue depth: $job_count" + + wait_for_slurm_jobs 600 3600 # Check every 10 min, max 1 hour + + log_success "Phase 2 complete!" +} + +# ============================================================================ +# Phase 3: Validation (status checks + pytest integration tests) +# ============================================================================ + +phase_3_validation() { + log "=========================================" + log "Phase 3: Validation" + log "TAG: $TAG" + log "=========================================" + + cd "$ZPPY_DIR" + activate_env "$ZPPY_ENV" + ensure_test_branch "test_zppy_${TAG}" "$ZPPY_BASE_BRANCH" + + # ------------------------------------------------------------------ + # Status file checks + # ------------------------------------------------------------------ + log "Checking all status files..." + local all_good=true + + check_status_files "$V2_OUTPUT" "v2" || all_good=false + check_status_files "$LEGACY_310_V2_OUTPUT" "Legacy 3.1.0 v2" || all_good=false + check_status_files "$LEGACY_300_V2_OUTPUT" "Legacy 3.0.0 v2" || all_good=false + check_status_files "$V3_OUTPUT" "v3" || all_good=false + check_status_files "$LEGACY_310_V3_OUTPUT" "Legacy 3.1.0 v3" || all_good=false + check_status_files "$LEGACY_300_V3_OUTPUT" "Legacy 3.0.0 v3" || all_good=false + check_status_files "$BUNDLES_OUTPUT" "Bundles" || all_good=false + check_status_files "$LEGACY_310_BUNDLES_OUTPUT" "Legacy 3.1.0 Bundles" || all_good=false + check_status_files "$LEGACY_300_BUNDLES_OUTPUT" "Legacy 3.0.0 Bundles" || all_good=false + + if [ "$all_good" = false ]; then + log_error "Some status checks failed!" + checkpoint "Errors found in status files. Continue to pytest anyway?" + else + log_success "All status files clean!" + fi + + # ------------------------------------------------------------------ + # pytest integration tests + # ------------------------------------------------------------------ + log "Running integration tests..." + + log "Running test_last_year.py (no expected results dir)..." + pytest tests/integration/test_last_year.py \ + || log_warning "test_last_year.py had failures" + + log "Running test_bash_generation.py..." + pytest tests/integration/test_bash_generation.py \ + || log_warning "test_bash_generation.py had failures" + + log "Running test_campaign.py..." + pytest tests/integration/test_campaign.py \ + || log_warning "test_campaign.py had failures" + + log "Running test_defaults.py..." + pytest tests/integration/test_defaults.py \ + || log_warning "test_defaults.py had failures" + + log "Running test_bundles.py..." + pytest tests/integration/test_bundles.py \ + || log_warning "test_bundles.py had failures" + + # ------------------------------------------------------------------ + # test_images.py -- must run from a compute node + # ------------------------------------------------------------------ + log_warning "test_images.py requires a compute node and must be run manually." + log "To run it on ${MACHINE}:" + log " ${SALLOC_CMD}" + log " source ${CONDA_PROFILE}" + log " conda activate ${ZPPY_ENV}" + log " cd ${ZPPY_DIR}" + log " pytest tests/integration/test_images.py" + log " cat test_images_summary.md" + if [[ "$FREEZE_DEPENDENCIES" == true ]]; then + log " (Frozen base was used for: ${FROZEN_BASE_COMPONENTS} -- any diffs here" + log " should trace back to those packages, not unrelated dependency drift.)" + fi + + log_success "Phase 3 automated tests complete!" + log_success "Remember to run test_images.py manually from a compute node." +} + +# ============================================================================ +# Main +# ============================================================================ + +main() { + log "Starting zppy integration test automation" + log "Config file: $CONFIG_FILE" + log "Machine: $MACHINE" + log "TAG: $TAG" + log "Auto mode: $AUTO_MODE" + log "Start phase: $START_PHASE" + + case "$START_PHASE" in + 1) + phase_1_setup + phase_2_bundles_part2 + phase_3_validation + ;; + 2) + phase_2_bundles_part2 + phase_3_validation + ;; + 3) + phase_3_validation + ;; + *) + log_error "Invalid START_PHASE: $START_PHASE (must be 1, 2, or 3)" + exit 1 + ;; + esac + + log_success "Integration test automation complete!" +} + +main diff --git a/tests/main_branch_testing/zppy_test.cfg b/tests/main_branch_testing/zppy_test.cfg new file mode 100644 index 00000000..850884ac --- /dev/null +++ b/tests/main_branch_testing/zppy_test.cfg @@ -0,0 +1,122 @@ +# zppy integration test configuration +# Source: run_integration_test.bash --config path/to/this.cfg +# +# Sections are comments only -- bash sources this file as KEY=VALUE pairs. + +# ---------------------------------------------------------------------------- +# Runtime (formerly "Parse arguments") +# ---------------------------------------------------------------------------- + +MACHINE=chrysalis # chrysalis | compy | perlmutter +START_PHASE=1 # 1 | 2 | 3 +AUTO_MODE=true # true = skip all interactive checkpoints +EXPLICIT_TAG="" # Leave empty to auto-generate; set to resume a prior run + +# ---------------------------------------------------------------------------- +# Per-run settings +# ---------------------------------------------------------------------------- + +RUN_NUMBER=1 + +DIAGS_BASE_BRANCH="main" +E3SM_TO_CMIP_BASE_BRANCH="master" +MPAS_BASE_BRANCH="develop" +ZI_BASE_BRANCH="main" +ZPPY_BASE_BRANCH="main" + +# "dev" = build a dedicated conda env from the repo's dev.yml +# "unified" = use the machine's e3sm-unified env (UNIFIED_ENV_CMD) +DIAGS_ENV_TYPE="dev" +E3SM_TO_CMIP_ENV_TYPE="dev" +MPAS_ENV_TYPE="dev" +ZI_ENV_TYPE="dev" + +# Optional: reuse an existing named conda env instead of creating a new one. +# When non-empty AND the corresponding ENV_TYPE is "dev", the script skips +# conda env creation and activates this env directly. +# Leave empty to let the script auto-name and create the env as usual. +DIAGS_EXISTING_ENV="" +E3SM_TO_CMIP_EXISTING_ENV="" +MPAS_EXISTING_ENV="" +ZI_EXISTING_ENV="" +ZPPY_EXISTING_ENV="" + +# Comma-separated list of zppy cfg names to generate and submit. +# These correspond to generated filenames: test_weekly__.cfg +# Any name containing "bundle" is treated as a bundle cfg and re-submitted in Phase 2. +CFGS_TO_RUN="weekly_bundles,weekly_comprehensive_v2,weekly_comprehensive_v3,weekly_legacy_3.1.0_bundles,weekly_legacy_3.1.0_comprehensive_v2,weekly_legacy_3.1.0_comprehensive_v3,weekly_legacy_3.0.0_bundles,weekly_legacy_3.0.0_comprehensive_v2,weekly_legacy_3.0.0_comprehensive_v3" + +# Comma-separated list of tasks to enable in utils.py. +TASKS_TO_RUN="e3sm_diags,mpas_analysis,global_time_series,ilamb,livvkit,pcmdi_diags" + +# ---------------------------------------------------------------------------- +# Frozen dependency base (isolates image-diff root causes) +# ---------------------------------------------------------------------------- +# +# Problem this solves: test_images.py can show diffs for e3sm_diags or +# pcmdi_diags plots that are actually caused by an unrelated dependency +# (e.g. matplotlib moving a plot a pixel to the right) rather than by the +# package under test. Freezing a component's dependencies isolates root +# causes and skips redundant env solves/builds. +# +# How it works: each component listed in FROZEN_BASE_COMPONENTS gets its OWN +# dedicated, fully-resolved conda env, built once per run from that +# component's BASE_ENV_LOCK_FILE_. Test envs for that component +# are then created by CLONING its frozen base (fast, no solve) and +# `pip install`ing the branch under test on top -- so every dependency other +# than that component's own code stays pinned to exactly what's in its lock +# file. Each component's frozen base is independent: e3sm_diags's cdat-based +# stack, mpas_analysis's ESMF/MPAS libs, and e3sm_to_cmip's cmor never need to +# be resolved into a single shared environment together. +# +# Caveat: if a component's branch genuinely needs a new or newer dependency, +# `pip install .` will pull it in on top of its frozen base -- the script +# logs a diff when that happens (see pre_install_*/post_install_* files in +# the run directory) so it's a visible, deliberate exception rather than +# silent drift. When that happens, regenerate that component's lock file to +# bake the new dependency in for future runs. + +# Master switch. When false, every component below falls back to its +# original behavior (solve its own dev.yml), regardless of the other +# settings in this section. +FREEZE_DEPENDENCIES=true + +# Which components get their own dedicated frozen base instead of solving +# their dev.yml fresh each run. Comma-separated subset of: +# e3sm_to_cmip,e3sm_diags,mpas_analysis,zppy_interfaces,zppy +# Defaults to every component this script builds a dev env for -- image +# diffs can come from any plotting-adjacent tool (e3sm_diags, pcmdi_diags via +# zppy_interfaces, mpas_analysis), not just the ones flagged in the original +# bug report, so there's no reason to leave any of them unfrozen by default. +FROZEN_BASE_COMPONENTS="e3sm_to_cmip,e3sm_diags,mpas_analysis,zppy_interfaces,zppy" + +# ---------------------------------------------------------------------------- +# One-time setup +# ---------------------------------------------------------------------------- + +HOME_DIR="$HOME" +EZ_DIR="$HOME_DIR/ez" + +E3SM_DIAGS_DIR="$EZ_DIR/e3sm_diags" +E3SM_TO_CMIP_DIR="$EZ_DIR/e3sm_to_cmip" +MPAS_ANALYSIS_DIR="$EZ_DIR/MPAS-Analysis" +ZPPY_INTERFACES_DIR="$EZ_DIR/zppy-interfaces" +ZPPY_DIR="$EZ_DIR/zppy" + +CONDA_PROFILE="$HOME_DIR/miniforge3/etc/profile.d/conda.sh" +TAG_CACHE_FILE="$HOME_DIR/.zppy_test_tag" + +# One FULLY RESOLVED environment spec per frozen component -- required for +# each name listed in FROZEN_BASE_COMPONENTS above. Must be exact-version +# output, e.g.: +# conda env create -f -n tmp-lock-gen +# conda activate tmp-lock-gen +# conda list --explicit > /home/$USER/ez/frozen-base-.txt +# A dev.yml is NOT sufficient here -- it re-solves and can drift between +# runs even unmodified. Regenerate a component's file whenever that +# component needs a new/updated dependency baked into its frozen base. +BASE_ENV_LOCK_FILE_E3SM_TO_CMIP="$EZ_DIR/frozen-base-e3sm_to_cmip.txt" +BASE_ENV_LOCK_FILE_E3SM_DIAGS="$EZ_DIR/frozen-base-e3sm_diags.txt" +BASE_ENV_LOCK_FILE_MPAS_ANALYSIS="$EZ_DIR/frozen-base-mpas_analysis.txt" +BASE_ENV_LOCK_FILE_ZPPY_INTERFACES="$EZ_DIR/frozen-base-zppy_interfaces.txt" +BASE_ENV_LOCK_FILE_ZPPY="$EZ_DIR/frozen-base-zppy.txt"