add ca fertilization harmonization and ncc compost support

[divine7022]

Description

folds standalone N fertilization harmonization (was living at /projectnb/dietzelab/ccmmf/management/fertilization/harmonization.R) into data.land data-raw pipeline, then layers ncc (compost) sampling support on top and extends the events schema with an ncc event type

events_schema_v0.1.2.json picks up ncc as an allowed event_type with material, ncc_subtype, fert_subtype, and pft properties

separate workflow PR will consume these samplers from workflows/fertilization-statewide and workflows/ncc-statewide to emit ensemble events

cc @sarahkanee @mdietze @infotroph @dlebauer

Motivation and Context

Review Time Estimate

• Immediately
• Within one week
• When possible

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

• My change requires a change to the documentation.
• My name is in the list of CITATION.cff
• I agree that PEcAn Project may distribute my contribution under any or all of
  • the same license as the existing code,
  • and/or the BSD 3-clause license.
• I have updated the CHANGELOG.md.
• I have updated the documentation accordingly.
• I have read the CONTRIBUTING document.
• I have added tests to cover my changes.
• All new and existing tests passed.

[divine7022]

high level, priors are out now, two ref tables (ca_compost_amendment, ca_n_application_rate) stay as bundled data for now. and new ag/management package vs bety priors vs whatever is separate conv

[dlebauer]

new ag/management package vs bety priors vs whatever is separate conv

@divine7022 Could you please clarify?

[dlebauer]

Thank you for these changes.

General changes requested:

Better documentation

(Make sure to review the data packaging guidance from the Data chapter of the "R-Packages" book)

Documentation should cover both the source (TSV) and package (RDA) data:

package datasets should be documented in roxygen, and should include

• definitions of each field
• information about where the data came from:
  • references
  • how the dataset was compiled, including assumptions
  • link to more details in source data README.

Source datasets can be documented in a data-raw/README.md and should include:
- where data came from
- what decisions were made during curation
- note that for package datasets that do not have a manual curation step, the provenance of package data is documented by R scripts.
- how to update both the TSV and .rda files

dataset names

Make sure naming is consistent, e.g. create_n_rate_data.R reads a file n_fertilizaton.tsv and generates a dataset ca_n_application_rate. This can get confusing.

consider renaming compost.tsv to organic_ammendments.tsv to bettr reflect the diversity of materials represented in the dataset.

[dlebauer]

This table combines stoichiometric properties with assumed application-rate
scenarios.

• would it make more sense to separate material properties from application-rates?
• See more general requests about dataset documentation in my general review comments. Some specific questions are below
• what columns come from the source, and what information was added during manual curation?
  • e.g. are the Rows* and Trees* application-rate values directly supported by
    the cited sources, or did sources provide more granular, species level rates?
• Why does the build script expose only the Rows* values in
  ca_compost_amendment?
• Is the intended distinction that synthetic N rates are crop-specific, while
  compost/organic amendment rates are only available at a broader crop-structure
  level?
• Given the contents include manure, blood meal, paper, wood chips, straw, and
  other organic materials, would organic_amendments.tsv or
  compost_amendments.tsv be a clearer filename than compost.tsv?

[divine7022]

yeh did the split. one table for the properties (ca_organic_amendment_properties, so C:N, total N, 4-week PAN, material_class, source), one for the rates (ca_organic_amendment_app_rate, long by material and crop_structure). they join on material

to your other questions:

what columns come from the source, and what information was added during manual curation?

pretty much all of it's straight off the sheet, except material_class (we set that by hand, more on it below), the g/m2 columns (converted from lbs/acre with ud_convert), and the short citations (we mapped those from the source urls). it's laid out in the readme.

are the Rows* and Trees* application-rate values directly supported by
the cited sources, or did sources provide more granular, species level rates?

sheet has both. for 30 of the 32 it's the same number copied across, but 2 of em (blood meal, fresh poultry manure) really do differ and trees comes in lower. the sources only go down to crop-structure, not species, so that's as fine as it gets.

Why does the build script expose only the Rows* values..
yeh that's flagged, old script dropped Trees. both are in now.

Is the intended distinction that synthetic N rates are crop-specific, while
compost/organic amendment rates are only available at a broader crop-structure
level?

ryt, exactly. the N rates are per crop, but amendment rates only come per crop structure (rows = annual, trees = orchard), so the two tables get keyed differently on purpose

[dlebauer]

where does m come from? I don't see it in compost.tsv

[divine7022]

yeh good catch, that whole function's gone. back when the harmonizing was in the build script, material_to_class() was working out the class on the fly with grepl. m was just the material name getting passed in, but yeh, you'd have to go chase the call to see that, which is the whole problem. now it's a plain column in the source data, one class per material, mapped to CalRecycle list (14 CCR 17852). just data you can read instead of code you have to trace

[dlebauer]

why are only Rows[MIN|MAX] and not Trees[MIN|MAX] used here? (see also general question about documentation)

[divine7022]

not on purpose. same deal as the m thing, when the harmonizing was jammed into the build script it only wired up the Rows columns and quietly dropped Trees. now the harmonize step is separate script on scc and it splits Rows/Trees into the long table by crop_structure, so both stick around. most materials have the same rate either way, but 2 that don't (blood meal, fresh poultry manure) keep their tree rate now instead of losing it

[dlebauer]

Do crop-names come from a controlled vocabulary? Any reason not to use the LandIQ names here?

Where do PFTs come from? On one hand, row/woody is a simple designation, and there may be reasons to include a grouping in this dataset. But caution should be used and rationale/motivation documented because PFT groupings are opinionated, and in this case alfalfa is not typically considered a 'row' crop (it is forage).

[divine7022]

two things here

crop names: yeh those are FREP / UC ANR names off the extension pubs, not LandIQ, and that's on purpose. the rates get published at that level, and some landIQ subclasses (like "miscellaneous deciduous") you can't really pin one rate to. landIQ to FREP mapping lives over in the workflow's Crop_types crosswalk, not in this table

pft: fair, and yeh ryt, alfalfa's forage not row crop. honestly pft_group is just hand set in the spreedsheet, it's a rough row/woody/rice bucket that lines up with grass vs deciduous pfts, not a real botanical/ag call. I kept it matching the canonical cadwr_pfts.csv, which also calls alfalfa row (it's under the pasture class there). I did try pulling pft_group straight from cadwr during harmonize so it'd be one source of truth, but it came out worse, the crosswalk sends oats into the misc grain and hay bucket so it flips to "hay", and about 9 crops don't match cleanly and just drop out. so it stays hand set, and I wrote the reasoning plus the alfalfa caveat into the docs

[dlebauer]

@divine7022

To unblock #4003, which currently depends on this PR, you could

• copy the .rda and .R files from this PR to the workflows/[fertilization|ncc]-statewide directories
• update #4003 to use those resources

This is in line with one of the motivations for the workflows directory - as a place to maintain and develop workflows until individual components can be generalized and extracted into PEcAn package functions and data

[divine7022]

quick context first, since this turned into a structural change. where we started, the build scripts here were reading the curated fertilization data straight from Mike's team's spreedsheet, and doing the harmonization in place mapping the source URLs to citations, the unit conversions, the property vs rate columns, the material classes. So the .rda we shipped was harmonized version, but all that harmonization logic was baked into package build

per discussion, now I flipped it. harmonization moved out to a standalone harmonize_fertilization_data.R on SCC. It reads original TSVs and writes already harmonized CSVs, those CSVs land in data-raw/, and create_*.R scripts just read them and call use_data(). No curation left in the package, it's purely packaging clean data now. That's the "no manual curation step, provenance is the R script" case you pointed at

regarding docs, rewrote the roxygen for all three datasets, and added a data-raw/README.md covering where the raw sheet comes from, the curation decisions, and how to update

Make sure naming is consistent, e.g. create_n_rate_data.R reads a file n_fertilizaton.tsv and generates a dataset ca_n_application_rate. This can get confusing.

consider renaming compost.tsv to organic_ammendments.tsv to bettr reflect the diversity of materials represented in the dataset.

corrected, and went past the rename. now names lines up: ca_n_application_rate.csv, create_ca_n_application_rate.R, ca_n_application_rate, same for amendments. I have dropped compost.tsv, instead of just renaming it I split it into ca_organic_amendment_properties and ca_organic_amendment_app_rate (more in ^^ thread) and dropped "compost" name since the table's mostly manure, straw, wood chips, paper, blood meal