Crash with shallow bedrock in FATES non-SP

[samsrabin]

Brief summary of bug

As part of my testing of #3927, I'm adding some (probably temporary) tests at Hui's ALP2 site. These tests replicate Hui's setup exactly, which includes use_bedrock = .true.. This worked fine in SP mode, but it causes a crash when I try to run in no-comp/fixed-biogeography mode.

General bug information

CTSM version you are using: ctsm5.4.028

Does this bug cause significantly incorrect results in the model's science? No

Configurations affected: FATES runs with use_bedrock true and any site/gridcell with zbedrock shallower than the deepest decomp layer.

Important output or errors that show the problem

At line 1538 of file /glade/work/samrabin/ctsm_hui_moss/src/utils/clmfates_interfaceMod.F90
Fortran runtime error: Index '20' of dimension 1 of array 'this...%litt_flux_lab_c_si' outside of expected range (6:1)

Error termination. Backtrace:
#0  0x781b9f in __clmfatesinterfacemod_MOD_updateclitterfluxes
        at /glade/work/samrabin/ctsm_hui_moss/src/utils/clmfates_interfaceMod.F90:1538
#1  0x1c2bdb1 in __cncstateupdate1mod_MOD_cstateupdate1
        at /glade/work/samrabin/ctsm_hui_moss/src/biogeochem/CNCStateUpdate1Mod.F90:202
#2  0x1c70242 in __cndrivermod_MOD_cndrivernoleaching
        at /glade/work/samrabin/ctsm_hui_moss/src/biogeochem/CNDriverMod.F90:626
#3  0xd647c0 in __cnvegetationfacade_MOD_ecosystemdynamicspredrainage
        at /glade/work/samrabin/ctsm_hui_moss/src/biogeochem/CNVegetationFacade.F90:1057
#4  0x6ebf34 in __clm_driver_MOD_clm_drv
        at /glade/work/samrabin/ctsm_hui_moss/src/main/clm_driver.F90:1112
#5  0x6691ff in modeladvance
        at /glade/work/samrabin/ctsm_hui_moss/src/cpl/nuopc/lnd_comp_nuopc.F90:913

Here's the failing line (note that line number is different from the one in the error message):

CTSM/src/utils/clmfates_interfaceMod.F90

Lines 1514 to 1517 in a96fb3f

	! (gC/m3/timestep)
	cf_soil%decomp_cpools_sourcesink_col(c,1:nlevdecomp,i_met_lit) = &
	cf_soil%decomp_cpools_sourcesink_col(c,1:nlevdecomp,i_met_lit) + &
	this%fates(ci)%bc_out(s)%litt_flux_lab_c_si(1:nlevdecomp)*dtime

That line of code is not reached in SP mode, which explains why I only started seeing the error at higher complexity.

Details of bug

litt_flux_lab_c_si is allocated in FATES, in allocate_bcout(). It's set to the value passed in to the third argument of that subroutine, which in the subroutine definition is given the dummy variable name nlevdecomp_in. The subroutine is called in CLM's CLMFatesInterfaceMod here, where you can see it's actually set to col%nbedrock, not nlevdecomp:

CTSM/src/utils/clmfates_interfaceMod.F90

Lines 985 to 990 in a96fb3f

	ndecomp = col%nbedrock(c)
	call allocate_bcin(this%fates(nc)%bc_in(s),col%nbedrock(c),ndecomp, &
	num_harvest_inst, num_landuse_state_vars, num_landuse_transition_vars, &
	surfpft_lb,surfpft_ub)
	call allocate_bcout(this%fates(nc)%bc_out(s),col%nbedrock(c),ndecomp)

My error won't happen, then, unless nlevdecomp is less than col%nbedrock. It happens in my run because the fsurdat file for this site specifies zbedrock = 0.4, which results in col%nbedrock = 6. Since nlevdecomp is 20, it triggers the crash.

[samsrabin]

Suggested fix: Ensure that, whenever the size of any such array in bc_in or bc_out is needed, it is obtained from bc_in%nlevdecomp.

Initial attempt on a moss development branch here. ALP2 site testing shows:

• One of the non SP–mode tests completes successfully for the first time!
• Other non SP–mode tests fail for unrelated reasons (i.e., this crash was at least resolved).
• SP-mode tests with moss off pass b4b with their baselines.
• SP-mode tests with moss on differ from their baselines.

To me, this suggests that this solution is the way to go. The diffs in SP-mode tests with moss on probably represent an issue with moss, not with this solution.