RC for ESACCI-SCFG-MODIS-v4-0

[heienv]

1. source_id['source_id'][key]['source_name'] = 'ESACCI-SCFG-MODIS-v4-0'
2. source_id['source_id'][key]['release_year'] = '2025'
3. source_id['source_id'][key]['source_description'] = 'Snow Cover Fraction on Ground (SCFG) derived from Terra MODIS data from the ESA SNOW CCI Climate Data Record version 4.0'
4. source_id['source_id'][key]['source_version_number'] = '4.0'
5. source_id['source_id'][key]['institution_id'] = 'ENVEO'
6. source_id['source_id'][key]['region'] = 'global_land'
7. source_id['source_id'][key]['source_type'] = 'satellite_merged'
8. list of CMIP variable_ids: scfg

---

COMMENT:

• new variable_id: scfg (long name: Snow Cover Fraction on Ground; unit %)
• new source_type: satellite_merged (as in example but not in obs4MIPs_source_type.json)

[Darwyn72]

@heienv there is 'satellite_blended' in the obs4MIPs_source_type.json the definition is: "gridded product based on both in-situ instruments and satellite measurements" or you could use 'satellite_retrieval' if it's just "gridded product based on satellite measurements"

[gleckler1]

@heienv @Darwyn72 The source_type is an attribute intended to provide convenience, e.g., enabling users to search for one "type" or another. Beyond that, it is not a critical/unique identifier for describing the data so it is reasonable to choose "satellite_retrieval" even if there are multiple satellites used. These options may be revisited and improved in the future but since its not critical its fine to proceed.

The variable_id is however a critical. I think CMIP7 is sticking with "snc" as the variable_id for snow cover (%). For this reason, I suggest we denote these two variables "sncfg" and "sncfv". I think this will help clarify the direct relationship with the "snc" CMIP(3-7) model output. @heienv Please let us know if you have concerns about this - we could think about this more. In any case, it would be reasonable to note in the obs4MIPs-compliant files (perhaps as part of the "comment" attribute), what the acronyms were in the original data.

[taylor13]

Perhaps, this has already been discussed elsewhere, but how exactly are "sncfg" and "sncfv" defined? I assume sncfv is the fraction of area where snow is found on vegetation.

If these quantities are well-defined, new standard names should be proposed for them, unless they are accurately described by one of the following existing standard names:

• "surface_snow_area_fraction": Surface snow refers to the snow on the solid ground or on surface ice cover, but excludes, for example, falling snowflakes and snow on plants.
• "snow_area_fraction_viewable_from_above": Snow "viewable from above" refers to the snow on objects or the ground as viewed from above, which excludes, for example, falling snow flakes and snow obscured by a canopy, vegetative cover, or other features resting on the surface.

[gleckler1]

@taylor13 Good point. I think "snow_area_fraction_viewable_from_above" may be appropriate for the sncvf/scvf product, but I am less sure sncfg/scfg dataset corresponds to "surface_snow_area_fraction" or is directly comparable to the model output of snc.

[gleckler1]

@heienv I have added your institution_id to the master branch and draft variable entries in the Aday table. From the discussion above we may make some changes particularly for the standard names, but in the mean time you should have enough to begin testing the preparation of your obs4MIPs-compliant version of these datasets.
We encourage you to follow the recommended setup for the 2d demo corresponds to your data structure. There is also a recent contribution of daily snow cover which might be helpful. If you are familiar with Github you can create a branch on this repo to include your draft "runCMOR.py" and "input.json" files in a directory inputs/ENVEO/ESA-snc ... and of course, please feel free to ask any questions!

[gabrieleschwaizer]

The following CF standard names were defined for the SCFV and SCFG products, respectively, generated within ESA CCI Snow:

• snow_area_fraction_viewable_from_above: "Area fraction" is the fraction of a grid cell's horizontal area that has some characteristic of interest. It is evaluated as the area of interest divided by the grid cell area, or if the cell_methods restricts the evaluation to some portion of that grid cell (e.g. "where sea_ice"), then it is the area of interest divided by the area of the identified portion. It may be expressed as a fraction, a percentage, or any other dimensionless representation of a fraction. Snow "viewable from above" refers to the snow on objects or the ground as viewed from above, which excludes, for example, falling snow flakes and snow obscured by a canopy, vegetative cover, or other features resting on the surface.

• surface_snow_area_fraction: "Area fraction" is the fraction of a grid cell's horizontal area that has some characteristic of interest. It is evaluated as the area of interest divided by the grid cell area, or if the cell_methods restricts the evaluation to some portion of that grid cell (e.g. "where sea_ice"), then it is the area of interest divided by the area of the identified portion. It may be expressed as a fraction, a percentage, or any other dimensionless representation of a fraction. Surface snow refers to the snow on the solid ground or on surface ice cover, but excludes, for example, falling snowflakes and snow on plants.

[taylor13]

In CMIP, the root name of the variable (i.e., "variable name", or variable_id) representing surface_snow_area_fraction is "snc" reported in units of "%". I would suggest following that precedent.

snow_area_fraction_viewable_from_above is not a CMIP variable, so a new variable name is needed. I would vote for to append some appropriate suffix modifer to "snc" (e.g., "sncvfa"???, where "vfa" = viewable from above???). For consistency with nearly all the cmip area fractions, I would express it as a percentage.

[gleckler1]

I have modified the standard name for the new variable sncfv to be snow_area_fraction_viewable_from_above. Perhaps there will be further discussion about this but you should have all you need to begin preparation of the data. I am taking a break until 25 March, but will be checking in periodically so may respond to any further comments or questions.

[heienv]

Hi @taylor13 @Darwyn72 @gleckler1 @gabrieleschwaizer

thank you for your feedback and defining variables.

We have a couple of questions / comments related to the variable_ids and the python script(s).

1) VARIABLE ID

• We are fine with sncfg & sncfv if it is best fitting for you.
• "cell_methods":"area: time: mean" should be "cell_methods":"area: time: min"
• The out_name for sncfv & sncfg is still snc and should be sncfv & sncfg, am I right?
• The long_name is currently defined as 'Snow Area Percentage'. The long_name have already been properly defined and discussed within the Snow CCI project (see comment of @gabrieleschwaizer). We suggest 'Snow Cover Fraction Viewable' for scnfv and 'Snow Cover Fraction on Ground' for scnfg.
• Do you add the source_id with region, source, source_type and source_verion number to obs4MIPs_CV.json or can we implement it via pull request together with the python scripts and json input table? We already added it locally for the python scripts.
• Same question for changes in the variable_id in obs4MIPs_Aday.json. We can also suggest our changes via pull request.
• The 'type' is defined as 'real' and the layer is stored as float. The type be 'integer' to write integer (this depend on your answer for the question raised within 2a) Python script CMOR DataType)

2) Python script CMOR

We are finalizing step 5 in your recipe (Prepare python script for reading in data an writing with CMOR) and are additionally preparing the python script for reading and writing the uncertainty which is not an explicit step but related to step 5. The scripts are running propery and the output seems good, so far. Anyway, during these preparation steps the following questions have arisen:

a. DataType
The input data for SnowCCI CRDP SCFV and SCFG is of type uint8, whereas CMOR currently supports only the following data types: 'd' (double), 'f' (float), 'l' (long), and 'i' (int). Is it possible to extend the supported list to include 'u1' (uint8)?
Additionally, data with the datatype 'int' appear to be converted to 'float' within cmor.write(). Is this conversion required by design, or is it unintended? We investigated that the storage can be significantly reduced without loosing information if we use uint8 instead of int or float. The corresponding no data value is 255UB. Therefore we strongly recommend the data type uint8.

b. netCDF attributes
The input data (uint8) contain values in the valid_range 0–254, where 255 represents the official missing value and should not occur. The actual_range for SCF itself is 0–100. In addition to the numeric range, several flag values are used to represent different mask conditions (flag_meanings: Cloud, Polar_Night_or_Night, Water, Salt_Lake, Permanent_Snow_and_Ice, Classification_failed, Input_Data_Error, No_Satellite_Acquisition). In the original CCI netCDF files, these flags are described using the attributes flag_values (uint8) and flag_meanings as strings, following the CF conventions, which allow metadata to describe the meaning of encoded values in a variable.

Within cmor the set_variable_attributes() is the only possibility to add attributes to the variable (set_cur_dataset_attributes() does not make sense for the variable but the global dataset). However, CMOR currently does not support lists (e.g., flag_values) or string (within flag_meanings) attributes. Instead, it only allows attributes corresponding to the datatype of the output variable, which is always converted to float.

Questions

• Would it be possible to extend CMOR to support writing the following attributes: actual_range (min, max), flag_values and flag_meanings?
• Or have we overlook anything and you can recommend a proper solution?
• Alternatively we can not discriminate the masked pixels and have to set every not observed or masked value (unequal 0-100 %) to no Data (-999 for int). What do you prefer?

Notes related to DataType & netCDF attributes

• We already noticed the example and obs4MIPs_Aday.json using valid_min and valid_max, but not actual_range what is commonly used based on the (netCDF) CF conventions. Anyway, the valid range depends on the data type, the actual range is related to the data (here 0-100 %).
• Furthermore, the type is defined as 'real' what propably causes the convertion to float. Changing the type to 'integer' leads 'int' as output. See question in variable_id.
• Consequently, although the intended datatype would be defined as uint8 (or int32), the workflow only works with float ('f'), because the data are automatically converted to float using cmor.

3) Python script uncertainty

In your ODS2.6 it is mentioned that the 'uncertainty information can be included as seperate files'. The global attributes are copied directly from the CMOR netCDF, lat, lon and time are identical. As far as we saw the uncertainty and auxilliary data are appended seperately without using the CMOR library.

Related to the uncertainty we have the following questions:

• Can the uncertainty also be included as additional layer within the cmor netCDF with the layer name as combination of aux_uncertainty_id in front of the variable_id ({aux_uncertainty_id}_{variable_id}). This would reduce the total storage significantly since lat, lon & time would not be duplicated.
• Can the valid_range, actual_range, flag_values and flag_meanings be added as variable attributes, even if the variable attributes differ from the main netCDF with the SCF dataset?
• dtype: we provide the uncertainty in percent as uint8 with a actual range 0-100. Missing value is 255. Is this format suitable for the uncertainty or should it be similar to the dtype of the main variable? Usually it should be the same data type, but since the originally data are uint8 we questioned if the standard error can be uint8.
• Since the aux_uncertainty_id is new within ODS2.6, will there be an automatic mechanism in future that we can follow as guidance? Alternatively, do you have any best practices following the newest approaches from ODS2.6?

4) Ancillary Data
We also provide the solar zenith angle for each pixel. Are you also interested in providing these ancillary data or is this unnecessary?

[Darwyn72]

@heienv @gabrieleschwaizer apologies for the delay in responding to your enquiry. I've copied in @jingbowuCU @Kate-Willett @msteckle @simon-pinnock @flicj191 @dwest77a here as well to get feedback. I've answered where I can below - could be wrong!

1)
i OK
ii You could could suggest changes via a pull request to modify these to 'min' - assume this refers to 'minimum'
iii You can specify the out_name in your CMOR script as outputVarName
iv I'd suggest changes via a pull request to change these long_names
v Normally we add the source_id to the CV.json - correct @gleckler1 ?
vi Yes, you could suggest changes via a pull-request
vii I'm not sure what you are referring to here?

2)
Apologies, I don't know about changing how CMOR functions and writes out variables, but as it's open source and run locally on your machine or network, I'd assume you could modify your copy.

3)
CMOR doesn't support multiple variables per file, so the uncertainty data needs to be submitted as a separate file. For the first file you set 'has_aux_unc' to TRUE, same for subsequent files, and for example 'aux_uncertianty_is' is set to 'ustd', subsequent file to 'stderr' etc. The Appendix 5 in the latest ODS2.6.1 covers the filename layout.

4)
I don't think this data is necessary

[gleckler1]

@heienv @gabrieleschwaizer Lots of good questions. I'm still thinking about some of them so my remarks below are incomplete - more soon as I am catching up after a break.

) VARIABLE ID

We are fine with sncfg & sncfv if it is best fitting for you.
"cell_methods":"area: time: mean" should be "cell_methods":"area: time: min"

We can make this change
The out_name for sncfv & sncfg is still snc and should be sncfv & sncfg, am I right?
Thanks for pointing this out. Yes, we can make this change on the table.
The long_name is currently defined as 'Snow Area Percentage'. The long_name have already been properly defined and discussed within the Snow CCI project (see comment of @gabrieleschwaizer). We suggest 'Snow Cover Fraction Viewable' for scnfv and 'Snow Cover Fraction on Ground' for scnfg.
Thanks for pointing this out. Yes, we can make this change on the table.
Do you add the source_id with region, source, source_type and source_verion number to obs4MIPs_CV.json or can we implement it via pull request together with the python scripts and json input table? We already added it locally for the python scripts.
This information goes in the json input table. We do not modify the obs4MIPs_CV.json manually.

Same question for changes in the variable_id in obs4MIPs_Aday.json. We can also suggest our changes via pull request.

Yes, you can do this via pull request.

[gleckler1]

@heienv @gabrieleschwaizer Source_id has been added as well as other suggested changes mentioned above under 1) VARIABLE_ID. Exception is data type. @taylor13 or @mauzey1 may have some thoughts on options currently available in CMOR for similar savings to uint8 or whether or not it is feasible to add this to CMOR.

[heienv]

@gleckler1 @taylor13 @mauzey1
Thank you for your feedback. We are adapting to the current situation and look forward to your feedback on the open questions.

[heienv]

Hi @gleckler1, @taylor13 and @mauzey1, (@gabrieleschwaizer)

we already have a working version locally but still have open questions.

data_type uint8
Do you have any new information about adding the data type (uint8) to CMOR?(#658 (comment) & #658 (comment)):
Of course, we can enable uint8 in CMOR on our side and locally. However, we would first like to confirm that it is also supported on your side. Ideally, it should also be implemented in the open-source, publicly available library to avoid confusion for external users.

**Slight suggestions for Tables/obs4MIPs_Aday.json and obs4MIPs_CV.json will be send via pull request. **
obs4MIPs_CV.json: adding SCFV as source_id
Tables/obs4MIPs_Aday.json: standard_name for sncfg (surface_snow_area_fraction_on_ground) & acronym for SCFV & type (int or uint depending on CMOR data_type)

[mauzey1]

@heienv CMOR currently does not natively support uint8 data types for variables and axes.