Add main branch testing automation

docs/source/dev_guide/index.rst

@@ -12,9 +12,8 @@ This guide covers everything needed to develop, test, and release ``zppy``.
    parameters
    provenance
    tasks/index
-   test
-   update_expected_results
-   releases/index.rst
+   tests/index
+   releases/index
    new_task
    new_diags_set
    archive/index

docs/source/dev_guide/tests/automated_test.rst

@@ -0,0 +1,346 @@
+.. _automated-testing-zppy:
+
+*************************
+Automated testing of zppy
+*************************
+
+Follow the steps below to test ``zppy``. As you do so, please produce a Markdown report summarizing your results.
+
+Step 1: Determine what the current expected results are
+=======================================================
+
+Machine-specific setup
+~~~~~~~~~~~~~~~~~~~~~~
+
+Chrysalis:
+
+.. code-block:: bash
+
+    expected_results_dir=/lcrc/group/e3sm/public_html/zppy_test_resources
+    expected_results_records_dir=/lcrc/group/e3sm/public_html/zppy_test_resources_previous
+
+Compy:
+
+.. code-block:: bash
+
+    expected_results_dir=/compyfs/www/zppy_test_resources
+    expected_results_records_dir=/compyfs/fors729/zppy_test_resources_previous
+
+Note that Compy doesn't give write access to ``/compyfs/www/``, so we can't add a new directory there. That's why ``zppy_test_resources_previous`` is in a separate path.
+
+Perlmutter:
+
+.. code-block:: bash
+
+    expected_results_dir=/global/cfs/cdirs/e3sm/www/zppy_test_resources
+    expected_results_records_dir=/global/cfs/cdirs/e3sm/www/zppy_test_resources_previous
+
+Process
+~~~~~~~
+
+.. code-block:: bash
+
+    ls -lt ${expected_results_dir}
+
+In your Markdown report, note the date the expected results were last updated.
+
+Step 2: Review changes since expected results were updated
+==========================================================
+
+Now that we know the date the expected results are from, we can review what changes we'll be testing.
+
+Review each of the following commit logs and note commits made since the date the expected results were updated:
+
+* For the ``e3sm_to_cmip`` task: `e3sm_to_cmip <https://github.com/E3SM-Project/e3sm_to_cmip/commits/master>`_
+* For the ``e3sm_diags`` task: `e3sm_diags <https://github.com/E3SM-Project/e3sm_diags/commits/main>`_
+* For the ``mpas_analysis`` task: `MPAS-Analysis <https://github.com/MPAS-Dev/MPAS-Analysis/commits/develop/>`_
+* For the ``global_time_series`` and ``pcmdi_diags`` tasks: `zppy-interfaces <https://github.com/E3SM-Project/zppy-interfaces/commits/main>`_
+* For ``zppy`` itself: `zppy <https://github.com/E3SM-Project/zppy/commits/main>`_
+
+For the remaining tasks (``climo``, ``ts``, ``tc_analysis``, ``ilamb``, ``livvkit``), we typically just use the associated package's latest release rather than making dev environments. As such, their latest development will have no impact on our tests unless we have started using one of their newer releases.
+
+In your Markdown report, make a table like:
+
+.. code-block::
+
+    | Package | Changes since expected results were updated |
+    | --- | --- |
+    | [package name](link to package's commit log) | Links to all PRs merged since the expected results were updated |
+    ...
+
+The automated test script
+=========================
+
+The automated test script handles the following steps from the manual testing process:
+
+* Step 3: Set up environments for called packages
+* Step 4: Set up zppy environment
+* Step 5: Launch zppy jobs
+* Step 6: Launch zppy jobs – bundles part 2
+* Step 7: Review finished returns
+* Step 8: Run Python tests (excluding the final ``pytest tests/integration/test_images.py`` call from a compute node)
+
+A. Set up the test script
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. code-block:: bash
+
+    cd ${repo_parent_dir}/zppy
+    git status # Check for uncommitted changes
+
+    # If there are uncommitted changes,
+    # commit them so we can move cleanly to a new branch:
+    git add -A
+    git commit -m "Checkpoint"
+
+    git fetch upstream main # This assumes you've named your remote for the main repo as "upstream"
+    git checkout -b test-zppy-yyyymmdd upstream/main # Use today's date
+    git log --oneline | head -n 1
+    # Check that this matches the corresponding commit log:
+    # https://github.com/E3SM-Project/zppy/commits/main
+
+    # Now, copy the test script and cfg from the zppy repo into the directory
+    # that you'll be running the test script from.
+    mkdir -p ${test_script_dir}/test_yyyymmdd_runN
+    cd ${test_script_dir}/test_yyyymmdd_runN
+    cp ${repo_parent_dir}/zppy/tests/main_branch_testing/run_integration_test.bash .
+    cp ${repo_parent_dir}/zppy/tests/main_branch_testing/zppy_test.cfg .
+
+    # Now, edit the test cfg as needed
+    emacs zppy_test.cfg
+
+B. Set up the test cfg
+~~~~~~~~~~~~~~~~~~~~~~
+
+Let's examine the parts of the test cfg.
+
+You'll likely just need to update the ``MACHINE`` name if you're not running on Chrysalis.
+
+.. code-block::
+
+    # For these,
+
+    MACHINE=chrysalis       # chrysalis | compy | perlmutter
+    START_PHASE=1           # 1 | 2 | 3
+    AUTO_MODE=true          # true = skip all interactive checkpoints
+    EXPLICIT_TAG=""         # Leave empty to auto-generate; set to resume a prior run
+
+Update the ``RUN_NUMBER`` if you've already done a test run today.
+
+.. code-block::
+
+    RUN_NUMBER=1
+
+Update the ``_BASE_BRANCH`` parameters if you plan to test new features or bug fixes that aren't yet included on whatever the repo calls its "official" branch.
+
+.. code-block::
+
+    DIAGS_BASE_BRANCH="main"
+    E3SM_TO_CMIP_BASE_BRANCH="master"
+    MPAS_BASE_BRANCH="develop"
+    ZI_BASE_BRANCH="main"
+    ZPPY_BASE_BRANCH="main"
+
+Update the ``_ENV_TYPE`` parameters if you want to use E3SM-Unified rather than a dev environment. If you plan to only run a subset of tasks, you can set the ones you aren't running to use E3SM-Unified, so that the script doesn't spend time building a dev environment that won't be used.
+
+.. code-block::
+
+    # "dev"     = build a dedicated conda env from the repo's dev.yml
+    # "unified" = use the machine's e3sm-unified env (UNIFIED_ENV_CMD)
+    DIAGS_ENV_TYPE="dev"
+    E3SM_TO_CMIP_ENV_TYPE="dev"
+    MPAS_ENV_TYPE="dev"
+    ZI_ENV_TYPE="dev"
+
+Update the ``_EXISTING_ENV`` parameters if you already have an environment from a previous test run to use.
+
+.. code-block::
+
+    # Optional: reuse an existing named conda env instead of creating a new one.
+    # When non-empty AND the corresponding ENV_TYPE is "dev", the script skips
+    # conda env creation and activates this env directly.
+    # Leave empty to let the script auto-name and create the env as usual.
+    DIAGS_EXISTING_ENV=""
+    E3SM_TO_CMIP_EXISTING_ENV=""
+    MPAS_EXISTING_ENV=""
+    ZI_EXISTING_ENV=""
+    ZPPY_EXISTING_ENV=""
+
+Update these two parameters to configure which jobs run.
+
+.. code-block::
+
+    # Comma-separated list of zppy cfg names to generate and submit.
+    # These correspond to generated filenames: test_weekly_<name>_<machine>.cfg
+    # Any name containing "bundle" is treated as a bundle cfg and re-submitted in Phase 2.
+    CFGS_TO_RUN="weekly_bundles,weekly_comprehensive_v2,weekly_comprehensive_v3,weekly_legacy_3.1.0_bundles,weekly_legacy_3.1.0_comprehensive_v2,weekly_legacy_3.1.0_comprehensive_v3,weekly_legacy_3.0.0_bundles,weekly_legacy_3.0.0_comprehensive_v2,weekly_legacy_3.0.0_comprehensive_v3"
+
+    # Comma-separated list of tasks to enable in utils.py.
+    TASKS_TO_RUN="e3sm_diags,mpas_analysis,global_time_series,ilamb,livvkit,pcmdi_diags"
+
+
+These parameters are unlikely to change between runs. They just let the test script know where to find files in your particular workspace. It is recommended to clone a new copy of the repos and use that for each ``_DIR`` parameter listed below. The script will change branches, so using a distinct copy means you won't get your work overwritten.
+
+.. code-block::
+
+    HOME_DIR="$HOME"
+    EZ_DIR="$HOME_DIR/ez"
+
+    E3SM_DIAGS_DIR="$EZ_DIR/e3sm_diags"
+    E3SM_TO_CMIP_DIR="$EZ_DIR/e3sm_to_cmip"
+    MPAS_ANALYSIS_DIR="$EZ_DIR/MPAS-Analysis"
+    ZPPY_INTERFACES_DIR="$EZ_DIR/zppy-interfaces"
+    ZPPY_DIR="$EZ_DIR/zppy"
+
+    CONDA_PROFILE="$HOME_DIR/miniforge3/etc/profile.d/conda.sh"
+    TAG_CACHE_FILE="$HOME_DIR/.zppy_test_tag"
+
+
+C. Run the test script
+~~~~~~~~~~~~~~~~~~~~~~
+
+Now that we have the test cfg set up, we can run it.
+
+.. code-block:: bash
+
+    screen # Use `screen`` so that even if the terminal connection is interrupted, the script will keep running.
+    ulimit -s unlimited # This is necessary for MPAS-Analysis to work inside `screen`
+    cd ${test_script_dir}/test_yyyymmdd_runN
+    cat zppy_test.cfg # Make sure changes are there
+    time ./run_integration_test.bash --config zppy_test.cfg 2>&1 | tee integration_test_runN.log
+    # Ctrl-A D to detach from screen
+    screen -ls # See what screen sessions you have
+    tail -f integration_test_runN.log
+
+Follow the ``tail`` output until you get to:
+
+.. code-block::
+
+    ✓ Phase 3 automated tests complete!
+    ✓ Remember to run test_images.py manually from a compute node.
+    ✓ Integration test automation complete!
+
+D. Review the output
+~~~~~~~~~~~~~~~~~~~~
+
+.. code-block:: bash
+
+    # CTRL C # Exit tail
+    screen -R # The script should have finished and there should be ``time`` output: real, user, sys
+    exit # Exit screen
+    cd ${test_script_dir}/test_yyyymmdd_runN
+    cat integration_test_runN.log
+
+Let's review the test script's output log.
+
+First, the unit tests. There are two blocks, starting with:
+
+.. code-block::
+
+    Running zppy-interfaces unit tests...
+
+and
+
+.. code-block::
+
+    Running zppy unit tests...
+
+Second, the output directories status. It should look like the following:
+
+.. code-block::
+
+    Checking all status files...
+    ...
+    ✓ All status files clean!
+
+If some status files were unsuccessful, you'll want to run the following to review the errors:
+
+.. code-block:: bash
+
+    cd ${dir_with_failures}
+    grep -v "OK" * status # See what jobs failed
+    # Review errors:
+    tail ${job_that_failed}.o${id_of_job_that_failed}
+    grep -i error ${job_that_failed}.o${id_of_job_that_failed}
+
+Third, the integration tests.
+
+.. code-block::
+
+    test_last_year.py
+    test_bash_generation.py
+    test_campaign.py
+    test_defaults.py
+    test_bundles.py
+
+Errors here may actually be expected if the expected results haven't been updated yet to reflect a recently merged pull request. Another reason for errors on ``test_bundles.py`` in particular is if you didn't run all the jobs necessary (i.e., if you're running a partial test).
+
+If all 3 pieces look good, you can proceed with the final integration test, the image checker.
+
+Step 8: Run Python tests
+========================
+
+Machine-specific setup
+~~~~~~~~~~~~~~~~~~~~~~
+
+Chrysalis:
+
+.. code-block:: bash
+
+    launch_compute_node()
+    {
+        salloc --nodes=1 --partition=debug --time=02:00:00 --account=e3sm
+    }
+
+Compy:
+
+.. code-block:: bash
+
+    launch_compute_node()
+    {
+        salloc --nodes=1 --partition=short --time=01:00:00 --account=e3sm
+    }
+
+Perlmutter:
+
+.. code-block:: bash
+
+    launch_compute_node()
+    {
+        salloc --nodes=1 --qos=interactive --time=01:00:00 --constraint=cpu --account=e3sm
+    }
+
+Process
+~~~~~~~
+
+.. code-block:: bash
+
+    cd ${repo_parent_dir}/zppy
+    git status
+    # You might have changed branches while you were waiting for jobs to finish.
+    # Make sure you're now back on the correct branch: test-zppy-yyyymmdd
+    # Also confirm you're back in the correct env: zppy-yyyymmdd or the Unified env
+
+    # The image checker test, which we'll run from a compute node:
+    launch_compute_node
+
+    start_bash_subshell
+    # EITHER:
+    # Activate EITHER a dev environment or the Unified env:
+    conda activate zppy-yyyymmdd
+    # OR: the command from `activate_unified_env`
+
+    pytest tests/integration/test_images.py
+    # Typically takes between 10 and 20 minutes on Chrysalis and Perlmutter.
+    # Typically takes closer to 50 minutes on Compy.
+    cat test_images_summary.md
+    exit # Exit bash shell
+    exit # Exit compute note
+
+In your Markdown report:
+
+* From the ``pytest tests/integration/test_images.py `` command-line output, copy everything after ``Captured stdout call`` to a code block labeled "Output"
+* Copy the results of ``cat test_images_summary.md`` to a section labeled "Complete summary table"
+* Make a new section named "Summary table -- only failing image-check tests, sorted by task". For each task that has missing and/or mismatched images, copy the relevant rows from the summary table. Skip this section if there were no failing image-check tests.
+* Note any test failures from the other Python tests.
+* If there were no failures at all, print "All tests pass"

docs/source/dev_guide/tests/index.rst

@@ -0,0 +1,14 @@
+.. _tests:
+
+*******
+Tests
+*******
+
+This page collects documentation on testing ``zppy``.
+
+.. toctree::
+   :maxdepth: 1
+
+   manual_test
+   automated_test
+   update_expected_results

docs/source/dev_guide/test.rst → docs/source/dev_guide/tests/manual_test.rst

@@ -1,8 +1,8 @@
-.. _testing-zppy:
+.. _manually-testing-zppy:
 
-*************
-Testing zppy
-*************
+*********************
+Manually testing zppy
+*********************
 
 Follow the steps below to test ``zppy``. As you do so, please produce a Markdown report summarizing your results.
 
@@ -93,6 +93,11 @@ Chrysalis:
         lcrc_conda # Or however you activate conda
         rm -rf build
         conda clean --all --y
+        # The dev.yml file may be in a different directory.
+        # e3sm_to_cmip, e3sm_diags: use conda-dev/
+        # zppy-interfaces, zppy: use conda/
+        # MPAS-Analysis: don't use this command, instead use:
+        # conda create --name ${env_name} --file dev-spec.txt --yes
         conda env create -f conda/dev.yml -n ${env_name}
         conda activate ${env_name}
         pre-commit run --all-files # Confirm this passes