Skip to content

CTSM5.4.0xx Fix problems encountered with running PLUMBER2 sites - #4092

Open
olyson wants to merge 44 commits into
ESCOMP:masterfrom
olyson:PLUMBERwetland
Open

CTSM5.4.0xx Fix problems encountered with running PLUMBER2 sites#4092
olyson wants to merge 44 commits into
ESCOMP:masterfrom
olyson:PLUMBERwetland

Conversation

@olyson

@olyson olyson commented Jun 11, 2026

Copy link
Copy Markdown
Contributor

Description of changes

This PR fixes failures in running PLUMBER2 sites. There are four categories of failures.

  1. Wetland sites fail at case creation because the usermods are requesting a change to baseflow_scalar in the user_nl_clm namelist, however, this parameter is now on the parameter file. This issue is described in PLUMBER2 wetland sites fail at case creation #4023 . The sites are:
        "CZ-wet",
        "DE-SfN",
        "FI-Kaa",
        "FI-Lom",
        "RU-Che",
        "SE-Deg",
        "US-Los",
        "US-Myb",
        "US-Tw4",
        "PL-wet"
  1. Several sites fail with a negative 2-m relatve humidity error in HumanIndexMod. The negative humidity is generated from either BareGroundFluxesMod or CanopyFluxesMod and passed into HumanIndexMod where there is an error check for this condition (and endrun). This issue is described in Several PLUMBER2 sites fail at run time #4081 . The sites are
        "ES-ES2",
        "ES-LgS",
        "ES-VDA",
        "SE-Deg",
        "US-Cop"
  1. Several sites fail with calendar errors. These are due to the fact that each site may require a different case setup configuration for spinup. This issue is described in Several PLUMBER2 sites fail at run time #4081 . The sites are
        "AT-Neu",
        "AU-How",
        "BE-Vie",
        "CH-Dav",
        "DE-Geb",
        "DE-Hai",
        "DE-Tha",
        "DK-Sor",
        "FR-Pue",
        "IT-MBo",
        "IT-SRo",
        "NL-Loo",
        "US-NR1",
        "US-FPa"
  1. After fixing 3. above, five of those sites run successfully but nine fail with a different calendar error. This is related to the fact that the time dimension in the atmospheric forcing files is float, not double, consequently, the time of day calculated in CDEPS is off by several seconds both for the lower bound and the upper bound. This can cause problems for leap day transitions. This issue is described in Several PLUMBER2 sites fail at run time #4081. The sites are:
        "AU-How",
        "BE-Vie",
        "CH-Dav",
        "DE-Geb",
        "DE-Tha",
        "DK-Sor",
        "FR-Pue",
        "NL-Loo",
        "US-NR1"

Specific notes

The fixes for these issues are:

  1. Here, tower_sites.py is modified so that for wetland sites, the parameter file for the base case is queried (regardless if it is wetland or not), and then the parameter file is replaced with a modified parameter file with baseflow_scalar=0 for any requested wetland sites in the run_tower submission.
    The baseflow_scalar namelist change request was deleted from the wetland sites shell commands by running tools/site_and_regional/plumber2_usermods. These modified shell commands are part of this commit.

  2. Per conversation with @wwieder , this PR turns off HumanIndexMod for all the PLUMBER2 sites for now, to avoid the negative relative humidity error check. This is accomplished in cime_config/usermods_dirs/clm/PLUMBER2/defaults/user_nl_clm.

  3. For the sites failing with the calendar error during spinup, these are the xml variables that need to be changed on a site by site basis for spinup:

RUN_STARTDATE
CALENDAR
DATM_YR_START
DATM_YR_END
DATM_YR_ALIGN
START_TOD (may already be set correctly though)

My run tower script (/glade/work/oleson/release-clm5.0.34/tools/PTCLM/run_clmtowers_plumber2.csh) handled this complexity, but this was not implemented in the run_tower infrastructure for PLUMBER2.
I've combined the new spinup variables into create_PLUMBER2_sites_csv.ncl -> PLUMBER2_sites.csv and modified the code to read these new variables in and generate the required shell commands.

  1. I have changed the time dimension in all of the forcing files from float to double and regenerated the time data. The nine sites now run successfully.
  • The new forcing data is at /glade/work/oleson/PLUMBER2/scripts/MAKE_TIME_DOUBLE/ and will need to be copied (and rimported) to /glade/campaign/cesm/cesmdata/cseg/inputdata/atm/datm7/CLM1PT_data/PLUMBER2.

A one line change to plumber2_usermods.py was also required to get these sites to run. This line had been commented out pending a CDEPS mod, which appears to have been implemented.

  • A couple of other CDEPS changes were required to get these sites to run, a change in components/cdeps/datm/cime_config/stream_definition_datm.xml to set dtlimit to 50 for 2000 compsets (spinup), and setting taxmode to extend and dtlimit to 1.e30 for HIST compsets.

Contributors other than yourself, if any: @wwieder (python help)

CTSM issues resolved or otherwise addressed, if any:
Resolves #4023
Resolves #4081
Resolves #4105

If answers are expected to change, describe (delete this line otherwise): The sites listed above will now run to completion. Sites that had run successfully prior to these changes may not have run correctly, so answers will likely change for those sites.

Any user interface changes (namelist or namelist defaults changes)? None to the defaults namelist. For wetlands, we only create a temporary modified parameter file for the requested run(s) and point to it in the run directory. Other changes here are accomplished by the PLUMBER2 shell commands.

Testing planned or performed, if any:
I've tested all of the sites with various submissions of the type:
./run_tower --plumber-sites {XX-XXX} --run-type ad --xmlchange STOP_OPTION=nyears,STOP_N=100,RESUBMIT=2,JOB_WALLCLOCK_TIME=06:00:00,PROJECT=P93300041 --overwrite.

  • It would be good to test all of the sites in one submission of run_tower once all of the new forcing files are in place in the proper location, in order to fully test this branch.

I also tested a NEON site (ABBY) to make sure I didn't break that configuration.

Requirements before merge:

  • I have followed the CTSM contribution guidelines.
  • The code in this PR branch builds with no errors.
  • The code in this PR branch runs with no errors. Briefly describe tested configuration(s): See above
  • This either (a) does not change answers, (b) it only changes answers at roundoff level, or (c) I have performed a scientific evaluation of the answer changes. Which?: I guess this is (d) We actually now get answers

@olyson olyson self-assigned this Jun 11, 2026
@olyson olyson added bug something is working incorrectly next this should get some attention in the next week or two. Normally each Thursday SE meeting. labels Jun 11, 2026
@olyson

olyson commented Jun 11, 2026

Copy link
Copy Markdown
Contributor Author

I see that there are some python errors in the automated testing, I'll address those next week once I figure out what they are about. Although my code was rated 9.99/10, ha ha.

@olyson

olyson commented Jun 16, 2026

Copy link
Copy Markdown
Contributor Author

Ok, I've fixed all of the python-lint-and-black errors.

@wwieder
wwieder requested a review from samsrabin June 24, 2026 16:07
@wwieder

wwieder commented Jun 24, 2026

Copy link
Copy Markdown
Contributor

@samsrabin can you review this. Keith's done functional tests to check this works, but would like a review on the python. We can also discuss testing (which may be outside the scope of this PR)?

@samsrabin

Copy link
Copy Markdown
Member

Can do, sure. What sort of timeline were y'all hoping for?

@olyson

olyson commented Jun 24, 2026

Copy link
Copy Markdown
Contributor Author

Can do, sure. What sort of timeline were y'all hoping for?

Maybe by the end of next week?

@olyson

olyson commented Jun 25, 2026

Copy link
Copy Markdown
Contributor Author

Per discussion at CTSM software meeting, I'm going to expand this PR to include other fixes to problems I've encountered with running PLUMBER2 sites.

@olyson
olyson marked this pull request as draft June 25, 2026 16:08
@olyson olyson changed the title Get PLUMBER2 wetland sites to run Fix problems encountered with running PLUMBER2 sites Jun 25, 2026
@olyson olyson removed the next this should get some attention in the next week or two. Normally each Thursday SE meeting. label Jun 25, 2026
@olyson

olyson commented Jul 18, 2026

Copy link
Copy Markdown
Contributor Author

All 170 sites have now run AD (300 years), postAD (300 years), and transient modes successfully.
I'm consolidating remaining tasks here that need to be completed before/after this PR is merged, because there is some duplication of tasks throughout the PR.

@olyson olyson added the next this should get some attention in the next week or two. Normally each Thursday SE meeting. label Jul 21, 2026
@slevis-lmwg

Copy link
Copy Markdown
Contributor

Regarding the ./rimport task above, Keith and I tried
./rimport --list plumber2_list
in /glade/campaign/cesm/cesmdata/cseg/inputdata
and got the following error for all the files in the list (only one shown here):

'/glade/campaign/cesm/cesmdata/cseg/inputdata/atm/datm7/CLM1PT_data/PLUMBER2/ZM-Mon/CLM1PT_data/CTSM_DATM_ZM-Mon_2008-2008.nc':
File is already published but NOT linked; linking now.
    Error deleting file /glade/campaign/cesm/cesmdata/cseg/inputdata/atm/datm7/CLM1PT_data/PLUMBER2/ZM-Mon/CLM1PT_data/CTSM_DATM_ZM-Mon_2008-2008.nc: [Errno 13] Permission denied: '/glade/campaign/cesm/cesmdata/cseg/inputdata/atm/datm7/CLM1PT_data/PLUMBER2/ZM-Mon/CLM1PT_data/CTSM_DATM_ZM-Mon_2008-2008.nc' -> '/glade/campaign/cesm/cesmdata/cseg/inputdata/atm/datm7/CLM1PT_data/PLUMBER2/ZM-Mon/CLM1PT_data/CTSM_DATM_ZM-Mon_2008-2008.nc.tmp'. Skipping relink.
    rimport: error processing /glade/campaign/cesm/cesmdata/cseg/inputdata/atm/datm7/CLM1PT_data/PLUMBER2/ZM-Mon/CLM1PT_data/CTSM_DATM_ZM-Mon_2008-2008.nc: Error relinking during rimport

This means that I we are unable to rimport new files by the same name to the same location due to the presence of the old files. Keith and I agreed that we do NOT want to rename the files or place them in a new location, so I sent a request for help to help@ucar.edu.

@ekluzek

ekluzek commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

I suggest adding some PLUMBER2 tests similar to how we have NEON tests. For example the NEON test

  <test name="SMS_Ld10_D_Mmpi-serial" grid="CLM_USRDAT" compset="I1PtClm60Bgc" testmods="clm/default--clm/NEON/HARV">

for PLUMBER2 would become...

  <test name="SMS_Ld10_D_Mmpi-serial" grid="CLM_USRDAT" compset="I1PtClm60Bgc" testmods="clm/default--clm/PLUMBER2/PL-wet">

We should have at least one test from the top groups that fail.

@olyson

olyson commented Jul 23, 2026

Copy link
Copy Markdown
Contributor Author

@ekluzek ,
I've added two tests in testlist_clm.xml:

<test name="SMS_Ly2_D_Mmpi-serial" grid="CLM_USRDAT" compset="I1PtClm60Bgc" testmods="clm/default--clm/PLUMBER2/ARSLu--clm/ADspinup_PLUMBER2">
<test name="SMS_Ly1_D_Mmpi-serial" grid="CLM_USRDAT" compset="IHist1PtClm60Bgc" testmods="clm/default--clm/PLUMBER2/ARSLu">

The first tests all of the logic involved in running a PLUMBER2 spinup. It runs a site that that has one year of forcing and loops over the forcing twice. This should basically cover all of the spinup problems that were encountered.

The second runs the same site in transient mode which basically tests that the model successfully runs the last time step, which was fixed by the CDEPS change.

I'm not convinced about the usefulness of other tests. These tests operate outside of the run_tower infrastructure.
The wetland sites were only failing because run_tower was trying to set baseflow_scalar in user_nl_clm.
The negative humidity error was occurring within HumanIndexMod. The solution was to turn it off through the PLUMBER2 usermods in user_nl_clm. Is it worth adding a test of one of those sites just to verify that HumanIndexMod is off, I'm not sure.

@ekluzek

ekluzek commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

@olyson I just approved and made the CDEP tag. It's cdeps1.0.102. It would be good to include the update to it in this PR. But, that's also a lot of CDEP tags and there might be other things that pop up with updating to it. So we could make that update seperate from this one if it causes any problems.

So how about you try updating to 1.0.102 and if everything works and appears fine -- we include it. But, if you run into any trouble -- we hold off and make that update seperate?

@olyson

olyson commented Jul 23, 2026

Copy link
Copy Markdown
Contributor Author

@ekluzek , sounds like it is worth a try to update to that CDEPS tag, I'll let you know how it goes...

@ekluzek

ekluzek commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

@ekluzek , I've added two tests in testlist_clm.xml:

<test name="SMS_Ly2_D_Mmpi-serial" grid="CLM_USRDAT" compset="I1PtClm60Bgc" testmods="clm/default--clm/PLUMBER2/ARSLu--clm/ADspinup_PLUMBER2">
<test name="SMS_Ly1_D_Mmpi-serial" grid="CLM_USRDAT" compset="IHist1PtClm60Bgc" testmods="clm/default--clm/PLUMBER2/ARSLu">

The first tests all of the logic involved in running a PLUMBER2 spinup. It runs a site that that has one year of forcing and loops over the forcing twice. This should basically cover all of the spinup problems that were encountered.

The second runs the same site in transient mode which basically tests that the model successfully runs the last time step, which was fixed by the CDEPS change.

I'm not convinced about the usefulness of other tests. These tests operate outside of the run_tower infrastructure. The wetland sites were only failing because run_tower was trying to set baseflow_scalar in user_nl_clm. The negative humidity error was occurring within HumanIndexMod. The solution was to turn it off through the PLUMBER2 usermods in user_nl_clm. Is it worth adding a test of one of those sites just to verify that HumanIndexMod is off, I'm not sure.

Perfect, this is awesome. The main point is to add some tests that will be run with aux_clm that would've caught some of the problems. And to be thoughtful about how best to do that which is exactly what you did. It's good to know that others were part of using run_tower. This is why longer term we also need #2438.

We have some tests that ensure that HumanIndex can be turned off. You could add one of those sites just because the negative humidty shows a strange sensitivity to those sites and testing it would catch if someone were to change that. But, that strange sensitivity might show up in other ways, and possibily catch other problems down the line? So probably worth adding, but not a big deal either way. The bigger problem would be to add too many tests to aux_clm. In that case we'd want to split them into different groups. But, adding three new single-point tests to aux_clm is not a problem. But, I'm good either way if you decide to add it, or leave it off...

Thanks for working on this and coming up with carefully crafted tests that test the important things. That's a shining example of how we want to do our testing.

@olyson

olyson commented Jul 24, 2026

Copy link
Copy Markdown
Contributor Author

Thanks @ekluzek , I agree about adding a third test for one of the sites exhibiting the negative humidity problem.

@olyson

olyson commented Jul 25, 2026

Copy link
Copy Markdown
Contributor Author

@ekluzek , all of the sites ran successfully with cdeps1.0.102.

@ekluzek ekluzek mentioned this pull request Jul 27, 2026
10 tasks
@olyson

olyson commented Jul 28, 2026

Copy link
Copy Markdown
Contributor Author

@ekluzek , I've committed the three new tests and associated testmods and linked usermods, if you (or @slevis-lmwg ) would like to review.
8ca4334
3077b69

@olyson

olyson commented Jul 29, 2026

Copy link
Copy Markdown
Contributor Author

New datm forcing data has now been placed at

/glade/campaign/cesm/cesmdata/inputdata/atm/datm7/CLM1PT_data/PLUMBER2.c20260729

and rimported.
A CDEPS PR was issued to update the path to the above: ESCOMP/CDEPS#424

@ekluzek ekluzek removed the next this should get some attention in the next week or two. Normally each Thursday SE meeting. label Jul 30, 2026
@ekluzek
ekluzek requested a review from samsrabin July 30, 2026 15:47
@olyson

olyson commented Jul 30, 2026

Copy link
Copy Markdown
Contributor Author

Latest CDEPS tag has been pushed to this branch.

@samsrabin samsrabin left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, Keith! All my original comments have now been addressed.

@ekluzek ekluzek left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK I had a few changes to do. Some of them need to be done when #4110 comes in as it fixes a problem you ran into here.

I also looked at cdeps tags to make sure there weren't additional answer changes that came in from cdeps1.0.96 to cdeps1.0.103 and there weren't. I'll have a seperate PR to update submodules to show those updates are b4b, that will likely be after this, but help give confidence that we don't have anything else that changes answers.

<machine name="izumi" compiler="nag" category="aux_clm"/>
<machine name="izumi" compiler="nag" category="aux_clm_mpi_serial"/>
<machine name="izumi" compiler="nag" category="prealpha"/>
<machine name="izumi" compiler="nag" category="aux_cime_baselines"/>

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove this from aux_cime_baselines, it turns out they aren't doing those tests on Izumi.

<machine name="izumi" compiler="nag" category="aux_clm"/>
<machine name="izumi" compiler="nag" category="aux_clm_mpi_serial"/>
<machine name="izumi" compiler="nag" category="prealpha"/>
<machine name="izumi" compiler="nag" category="aux_cime_baselines"/>

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove this from aux_cime_baselines as above.

<machine name="izumi" compiler="nag" category="aux_clm"/>
<machine name="izumi" compiler="nag" category="aux_clm_mpi_serial"/>
<machine name="izumi" compiler="nag" category="prealpha"/>
<machine name="izumi" compiler="nag" category="aux_cime_baselines"/>

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove from aux_cime_baselines as above.

@@ -0,0 +1 @@
../default

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ahhh, this is here, so that MOSART_MODE isn't set. Like in ADspinup. I've fixed this problem a different way in #4110 by checking if MOSART is active first. Once you update to that you can change the tests to use ADspinup rather than ADspinup_PLUMBER2. And then this file and the ADspinup_PLUMBER2 testmod directory can be replaced.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove this file when the above changes happen.

</options>
</test>

<test name="SMS_Ly1_D_Mmpi-serial" grid="CLM_USRDAT" compset="I1PtClm60Bgc" testmods="clm/default--clm/PLUMBER2/ESLgS--clm/ADspinup_PLUMBER2">

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
<test name="SMS_Ly1_D_Mmpi-serial" grid="CLM_USRDAT" compset="I1PtClm60Bgc" testmods="clm/default--clm/PLUMBER2/ESLgS--clm/ADspinup_PLUMBER2">
<test name="SMS_Ly1_D_Mmpi-serial" grid="CLM_USRDAT" compset="I1PtClm60Bgc" testmods="clm/default--clm/PLUMBER2/ESLgS--clm/ADspinup">

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This suggestion can be applied once #4110 comes in.

</options>
</test>

<test name="SMS_Ly2_D_Mmpi-serial" grid="CLM_USRDAT" compset="I1PtClm60Bgc" testmods="clm/default--clm/PLUMBER2/ARSLu--clm/ADspinup_PLUMBER2">

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
<test name="SMS_Ly2_D_Mmpi-serial" grid="CLM_USRDAT" compset="I1PtClm60Bgc" testmods="clm/default--clm/PLUMBER2/ARSLu--clm/ADspinup_PLUMBER2">
<test name="SMS_Ly2_D_Mmpi-serial" grid="CLM_USRDAT" compset="I1PtClm60Bgc" testmods="clm/default--clm/PLUMBER2/ARSLu--clm/ADspinup">

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See discussion on this below

@github-project-automation github-project-automation Bot moved this from In progress - master to Stalled (needs review, blocked etc.) in CTSM: Upcoming tags Jul 31, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug something is working incorrectly

Projects

Status: Stalled (needs review, blocked etc.)

5 participants