-
Notifications
You must be signed in to change notification settings - Fork 313
feat: add ERA5 CDS ↔ CF variable name mapping utility #3922
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
Akash-paluvai
wants to merge
2
commits into
PecanProject:develop
Choose a base branch
from
Akash-paluvai:GH-3605-era5-cf-varname-map
base: develop
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,89 @@ | ||
| #' Translate meteorological variable names between naming conventions | ||
| #' | ||
| #' Looks up variable names in one column of the PEcAn standard meteorology | ||
| #' table and returns the corresponding names from another column. | ||
| #' | ||
| #' @param vars Character vector of variable names to translate. | ||
| #' @param from Character scalar. Column in \code{table} containing the | ||
| #' input naming convention. | ||
| #' @param to Character scalar. Column in \code{table} containing the | ||
| #' output naming convention. | ||
| #' @param table Data frame used as the lookup source. Defaults to | ||
| #' \code{pecan_standard_met_table}. | ||
| #' | ||
| #' @return Named character vector of translated variable names. Names are | ||
| #' the input values; elements are the mapped values. Variables not found | ||
| #' in \code{table[[from]]} return \code{NA} and emit a log warning. | ||
| #' Returns \code{character(0)} for empty input. | ||
| #' | ||
| #' @details | ||
| #' Rows where \code{table[[from]]} is \code{NA} or empty string are excluded | ||
| #' from the lookup. Translation is not guaranteed to be one-to-one; ensure | ||
| #' uniqueness in \code{table[[to]]} if round-trip mapping is required. | ||
| #' | ||
| #' @examples | ||
| #' # ERA5 CDS name to CF standard name | ||
| #' translate_met_varnames( | ||
| #' "surface_solar_radiation_downwards", | ||
| #' from = "era5_cds", | ||
| #' to = "cf_standard_name" | ||
| #' ) | ||
| #' | ||
| #' # CF standard name to ERA5 CDS name | ||
| #' translate_met_varnames( | ||
| #' "surface_downwelling_shortwave_flux_in_air", | ||
| #' from = "cf_standard_name", | ||
| #' to = "era5_cds" | ||
| #' ) | ||
| #' | ||
| #' @export | ||
| translate_met_varnames <- function(vars, | ||
| from, | ||
| to, | ||
| table = pecan_standard_met_table) { | ||
| if (length(vars) == 0L) return(character(0)) | ||
|
|
||
| lookup <- table |> | ||
| dplyr::select(dplyr::all_of(c(from, to))) |> | ||
| dplyr::filter(!is.na(.data[[from]]), .data[[from]] != "") | ||
|
|
||
| result <- lookup[[to]] | ||
| names(result) <- lookup[[from]] | ||
|
|
||
| translated <- result[vars] | ||
|
|
||
| unknown <- vars[is.na(translated)] | ||
| if (length(unknown) > 0L) { | ||
| PEcAn.logger::logger.warn( | ||
| "translate_met_varnames: no", to, "mapping for", from, | ||
| "variable(s):", paste(unknown, collapse = ", "), | ||
| "-- returning NA for those entries" | ||
| ) | ||
| } | ||
|
|
||
| translated | ||
| } | ||
|
|
||
|
|
||
| #' Translate ERA5 CDS API names to CF standard names | ||
| #' | ||
| #' @param cds_vars Character vector of ERA5 CDS API variable names. | ||
| #' @return Named character vector of CF standard names. | ||
| #' @noRd | ||
| cds_to_cf_varnames <- function(cds_vars) { | ||
| translate_met_varnames(cds_vars, | ||
| from = "era5_cds", | ||
| to = "cf_standard_name") | ||
| } | ||
|
|
||
|
|
||
| #' Translate CF standard names to ERA5 CDS API names | ||
| #' | ||
| #' @param cf_vars Character vector of CF standard names. | ||
| #' @return Named character vector of ERA5 CDS variable names. | ||
| #' @noRd | ||
| cf_to_cds_varnames <- function(cf_vars) { | ||
| translate_met_varnames(cf_vars, | ||
| from = "cf_standard_name", | ||
| to = "era5_cds") | ||
| } |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What's the difference between the existing ERA column and the new ERA5_cds?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The
era5andera5_cdscolumns represent two different stages of the ERA5 workflow and are not interchangeable.era5: Short variable names (e.g., swvl1, ssrd) as stored in downloaded NetCDF files. Used byextract.nc.ERA5()to read variables from disk.era5_cds: Full variable names required by the Copernicus CDS API (e.g., volumetric_soil_water_layer_1). Used bydownload.ERA5_cds()via thevariables=argument when requesting data.The CDS API requires the long-form names, while NetCDF files store variables using short names, so both are needed.
Most entries in
era5_cdsare intentionallyNA. Only variables explicitly requested by the fallback pipeline viadownload.ERA5_cds()require a value.Variables such as t2m or strd have
era5short names because they can be extracted from existing files, but they are not directly requested by the fallback pipeline. Their CDS names are therefore omitted to avoid implying otherwise.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I really do not think we want to be adding a separate column to this table for every different source of the same dataset.
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@infotroph how do you suggest handling different versions of the same dataset that use different naming conventions? What are the advantages and disadvantages of each approach? More generally, what is your suggested fix for this bug?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Given we have 6 columns in this lookup table and 11 met2CF functions, it seems clear the de facto system is already for each function to handle its own name conversions -- and any unit conversions, which this table doesn't cover -- for itself.
Having all the PEcAn standard output names and units in a lookup table is definitely useful. I just don't think it's useful to make every input be listed here when all the logic for it already needs to be present in the relevant met2CF function.