Eurostat Data Marriages_by_sex_and_previous_marital_status & Population_on_1_January_by_age_and_sex

statvar_imports/eurostat/Marriages_by_sex_and_previous_marital_status/README.md

@@ -0,0 +1,48 @@
+# Eurostat Marriages by sex and previous marital status Import
+
+## Overview
+This dataset contains annual marriage statistics at the national level, sourced from Eurostat. The data tracks marital dynamics across European countries, breaking down marriage metrics by sex and the previous marital status of the individuals entering into marriage.
+
+**type of place:** Country
+**years:** Historical data to present (1960-2024) 
+**place_resolution:** Resolved to DCIDs (e.g., dcid:country/ARM, dcid:country/EST, dcid:country/BEL)
+
+## Data Source
+**Source URL:**
+https://ec.europa.eu/eurostat/api/dissemination/sdmx/2.1/data/DEMO_NMSTA/?format=SDMX-CSV&compressed=false
+
+**Provenance Description:**
+The data is provided by Eurostat, the statistical office of the European Union. It originates from the "Demography, population stock and balance" database under the "Marriages" statistical framework, specifically the "Marriages by sex and previous marital status" (DEMO_NMSTA) dataset.
+
+## Refresh Type
+Automatic Refresh
+The refresh is automated using the provided run.sh script, which handles both data download and processing.
+
+## How To Run Import
+To execute the complete import process (download and processing), run:
+./run.sh
+
+### Script Details:
+- **Download**: Uses `curl` to fetch the latest SDMX-CSV data from Eurostat's dissemination API.
+- **Processing**: Uses `stat_var_processor.py` to map raw data to Data Commons StatVarObservations using the PV map and metadata configuration.
+
+## Key Files
+- `run.sh`: Main execution script for download and processing.
+- `output_pvmap_cleaned.csv`: Property-Value mapping for StatVar definitions and dimensions.
+- `output_metadata.csv`: Configuration parameters for the processor.
+- `places_resolved_runtime.csv`: Mapping of place codes to Data Commons DCIDs.
+- `output.csv`: Processed statistical observations.
+- `output.tmcf`: Template MCF mapping the CSV columns to Data Commons schema.
+
+## Validation
+To validate the generated data, use the Data Commons import tool (lint mode):
+```bash
+java -jar datacommons-import-tool.jar lint output.csv output.tmcf
+```
+The resulting reports (`report.json`, `summary_report.html`) in `dc_generated/` provide detailed insights into data quality and validation status.
+
+## Testing
+Testing is performed using the `test_data` directory:
+- Raw Input: `test_data/Marriages_by_sex_and_previous_marital_status_data_raw.csv`
+- Expected Output: `test_data/output.csv`
+- Expected TMCF: `test_data/output.tmcf`

statvar_imports/eurostat/Marriages_by_sex_and_previous_marital_status/manifest.json

@@ -0,0 +1,26 @@
+{
+  "import_specifications": [
+    {
+      "import_name": "EuroStat_Marriages_by_sex_and_previous_marital_status",
+      "curator_emails": [
+        "support@datacommons.org"
+      ],
+      "provenance_url": "https://ec.europa.eu/eurostat/databrowser/view/demo_nmsta/default/table?lang=en",
+      "provenance_description": "The Eurostat Marriages by sex and previous marital status indicator offers standardized, long-term national demographic insights into social structure developments across European countries. Updated annually, it details the yearly count of marriage ceremonies stratified by the sex of the individuals and classified by their prior legal marital background (such as never-married single persons, divorcees, or widowed individuals).",
+      "scripts": [
+        "run.sh"
+      ],
+      "source_files": [
+        "Marriages_by_sex_and_previous_marital_status_data_raw.csv"
+      ],
+      "import_inputs": [
+        {
+          "template_mcf": "final_output/output.tmcf",
+          "cleaned_csv": "final_output/output.csv"
+        }
+      ],
+      "cron_schedule": "5 1 1,15 * *",
+      "resource_limits": {"cpu": 4, "memory": 8, "disk":100}
+        }
+   ]
+}

statvar_imports/eurostat/Marriages_by_sex_and_previous_marital_status/output_metadata.csv

@@ -0,0 +1,2 @@
+parameter,value
+header_rows,1

statvar_imports/eurostat/Marriages_by_sex_and_previous_marital_status/output_pvmap_cleaned.csv

@@ -0,0 +1,52 @@
+key,property1,value1,property2,value2,property3,value3,property4,value4,property5,value5
+# Dataflow and global properties
+DATAFLOW:ESTAT:DEMO_NMSTA(1.0),populationType,dcs:Person,measuredProperty,dcs:count,statType,dcs:measuredValue,event,dcs:MarriageEvent
+
+
+
+# Frequency
+freq:A,observationPeriod,P1Y
+freq:M,observationPeriod,P1M
+freq:Q,observationPeriod,P3M
+
+# Unit
+unit:NR,unit,dcs:Number
+
+# Sex / Gender
+sex:T,gender,""
+sex:M,gender,dcs:Male
+sex:F,gender,dcs:Female
+# Marital Status (Previous)
+
+marsta:TOTAL,maritalStatus,""
+marsta:SIN,maritalStatus,dcs:Single
+marsta:LUN,maritalStatus,dcs:Married
+marsta:MAR,maritalStatus,dcs:Married
+marsta:WID,maritalStatus,dcs:Widowed
+marsta:DIV,maritalStatus,dcs:Divorced
+marsta:DISLUN,maritalStatus,dcs:Divorced
+marsta:DIV_DISREP,maritalStatus,dcs:Divorced
+marsta:DISREP,maritalStatus,dcs:Divorced
+marsta:WID_DTHREP,maritalStatus,dcs:Widowed
+marsta:MAR_REP,maritalStatus,dcs:Married
+marsta:SEP,maritalStatus,dcs:Separated
+marsta:DTHLUN,maritalStatus,dcs:Widowed
+marsta:REP,maritalStatus,dcs:Married
+marsta:UNK,maritalStatus,dcs:UnknownMaritalStatus
+
+# Geography
+geo,observationAbout,{Data},,,,,,,,
+
+# Time
+TIME_PERIOD,observationDate,{Data}
+# Value
+OBS_VALUE,value,{Number}
+# Silencing warnings for unmapped columns
+LAST UPDATE,LastUpdate,{Data}
+CONF_STATUS,ConfStatus,{Data}
+
+OBS_FLAG,ObsFlag,{Data}
+# Missing values / Flags
+OBS_VALUE::,#ignore,Empty value
+OBS_FLAG:M,MissingValue,''
+OBS_FLAG:NA,NotApplicable,''

statvar_imports/eurostat/Marriages_by_sex_and_previous_marital_status/places_resolved_runtime.csv

@@ -0,0 +1,264 @@
+place_name,dcid
+AD,dcid:country/AND
+AE,dcid:country/ARE
+AF,dcid:country/AFG
+AG,dcid:country/ATG
+AI,dcid:country/AIA
+AL,dcid:country/ALB
+AM,dcid:country/ARM
+AN,dcid:country/ANT
+AO,dcid:country/AGO
+AQ,dcid:country/ATA
+AR,dcid:country/ARG
+AS,dcid:country/ASM
+AT,dcid:country/AUT
+AU,dcid:country/AUS
+AW,dcid:country/ABW
+AZ,dcid:country/AZE
+BA,dcid:country/BIH
+BB,dcid:country/BRB
+BD,dcid:country/BGD
+BE,dcid:country/BEL
+BF,dcid:country/BFA
+BG,dcid:country/BGR
+BH,dcid:country/BHR
+BI,dcid:country/BDI
+BJ,dcid:country/BEN
+BL,dcid:country/BLM
+BM,dcid:country/BMU
+BN,dcid:country/BRN
+BO,dcid:country/BOL
+BR,dcid:country/BRA
+BS,dcid:country/BHS
+BT,dcid:country/BTN
+BV,dcid:country/BVT
+BW,dcid:country/BWA
+BY,dcid:country/BLR
+BZ,dcid:country/BLZ
+CA,dcid:country/CAN
+CC,dcid:country/CCK
+CD,dcid:country/COD
+CF,dcid:country/CAF
+CG,dcid:country/COG
+CH,dcid:country/CHE
+CI,dcid:country/CIV
+CK,dcid:country/COK
+CL,dcid:country/CHL
+CM,dcid:country/CMR
+CN,dcid:country/CHN
+CO,dcid:country/COL
+CR,dcid:country/CRI
+CU,dcid:country/CUB
+CV,dcid:country/CPV
+CX,dcid:country/CXR
+CY,dcid:country/CYP
+CZ,dcid:country/CZE
+DE,dcid:country/DEU
+DJ,dcid:country/DJI
+DK,dcid:country/DNK
+DM,dcid:country/DMA
+DO,dcid:country/DOM
+DZ,dcid:country/DZA
+EC,dcid:country/ECU
+EE,dcid:country/EST
+EG,dcid:country/EGY
+EH,dcid:country/ESH
+EL,dcid:country/GRC
+ER,dcid:country/ERI
+ES,dcid:country/ESP
+ET,dcid:country/ETH
+FI,dcid:country/FIN
+FJ,dcid:country/FJI
+FK,dcid:country/FLK
+FM,dcid:country/FSM
+FO,dcid:country/FRO
+FR,dcid:country/FRA
+FX,dcid:country/FXX
+GA,dcid:country/GAB
+UK,dcid:country/GBR
+GD,dcid:country/GRD
+GE,dcid:country/GEO
+GF,dcid:country/GUF
+GG,dcid:country/GGY
+GH,dcid:country/GHA
+GI,dcid:country/GIB
+GL,dcid:country/GRL
+GM,dcid:country/GMB
+GN,dcid:country/GIN
+GQ,dcid:country/GNQ
+GS,dcid:country/SGS
+GT,dcid:country/GTM
+GU,dcid:country/GUM
+GW,dcid:country/GNB
+GY,dcid:country/GUY
+HK,dcid:country/HKG
+HM,dcid:country/HMD
+HN,dcid:country/HND
+HR,dcid:country/HRV
+HT,dcid:country/HTI
+HU,dcid:country/HUN
+ID,dcid:country/IDN
+IE,dcid:country/IRL
+IL,dcid:country/ISR
+IM,dcid:country/IMN
+IN,dcid:country/IND
+IO,dcid:country/IOT
+IQ,dcid:country/IRQ
+IR,dcid:country/IRN
+IS,dcid:country/ISL
+IT,dcid:country/ITA
+JE,dcid:country/JEY
+JM,dcid:country/JAM
+JO,dcid:country/JOR
+JP,dcid:country/JPN
+KE,dcid:country/KEN
+KG,dcid:country/KGZ
+KH,dcid:country/KHM
+KI,dcid:country/KIR
+KM,dcid:country/COM
+KN,dcid:country/KNA
+KP,dcid:country/PRK
+KR,dcid:country/KOR
+KW,dcid:country/KWT
+KY,dcid:country/CYM
+KZ,dcid:country/KAZ
+LA,dcid:country/LAO
+LB,dcid:country/LBN
+LC,dcid:country/LCA
+LI,dcid:country/LIE
+LK,dcid:country/LKA
+LR,dcid:country/LBR
+LS,dcid:country/LSO
+LT,dcid:country/LTU
+LU,dcid:country/LUX
+LV,dcid:country/LVA
+LY,dcid:country/LBY
+MA,dcid:country/MAR
+MC,dcid:country/MCO
+MD,dcid:country/MDA
+ME,dcid:country/MNE
+MF,dcid:country/MAF
+MG,dcid:country/MDG
+MH,dcid:country/MHL
+MK,dcid:country/MKD
+ML,dcid:country/MLI
+MM,dcid:country/MMR
+MN,dcid:country/MNG
+MO,dcid:country/MAC
+MP,dcid:country/MNP
+MQ,dcid:country/MTQ
+MR,dcid:country/MRT
+MS,dcid:country/MSR
+MT,dcid:country/MLT
+MU,dcid:country/MUS
+MV,dcid:country/MDV
+MW,dcid:country/MWI
+MX,dcid:country/MEX
+MY,dcid:country/MYS
+MZ,dcid:country/MOZ
+NA,dcid:country/NAM
+NC,dcid:country/NCL
+NE,dcid:country/NER
+NF,dcid:country/NFK
+NG,dcid:country/NGA
+NI,dcid:country/NIC
+NL,dcid:country/NLD
+NO,dcid:country/NOR
+NP,dcid:country/NPL
+NR,dcid:country/NRU
+NU,dcid:country/NIU
+NZ,dcid:country/NZL
+OM,dcid:country/OMN
+PA,dcid:country/PAN
+PE,dcid:country/PER
+PF,dcid:country/PYF
+PG,dcid:country/PNG
+PH,dcid:country/PHL
+PK,dcid:country/PAK
+PL,dcid:country/POL
+PM,dcid:country/SPM
+PN,dcid:country/PCN
+PR,dcid:country/PRI
+PS,dcid:country/PSE
+PT,dcid:country/PRT
+PW,dcid:country/PLW
+PY,dcid:country/PRY
+QA,dcid:country/QAT
+RE,dcid:country/REU
+RO,dcid:country/ROU
+RS,dcid:country/SRB
+RU,dcid:country/RUS
+RW,dcid:country/RWA
+SA,dcid:country/SAU
+SB,dcid:country/SLB
+SC,dcid:country/SYC
+SD,dcid:country/SDN
+SE,dcid:country/SWE
+SG,dcid:country/SGP
+SH,dcid:country/SHN
+SI,dcid:country/SVN
+SJ,dcid:country/SJM
+SK,dcid:country/SVK
+SL,dcid:country/SLE
+SM,dcid:country/SMR
+SN,dcid:country/SEN
+SO,dcid:country/SOM
+SR,dcid:country/SUR
+ST,dcid:country/STP
+SV,dcid:country/SLV
+SX,dcid:country/SXM
+SY,dcid:country/SYR
+SZ,dcid:country/SWZ
+TC,dcid:country/TCA
+TD,dcid:country/TCD
+TF,dcid:country/ATF
+TG,dcid:country/TGO
+TH,dcid:country/THA
+TJ,dcid:country/TJK
+TK,dcid:country/TKL
+TL,dcid:country/TLS
+TM,dcid:country/TKM
+TN,dcid:country/TUN
+TO,dcid:country/TON
+TR,dcid:country/TUR
+TT,dcid:country/TTO
+TV,dcid:country/TUV
+TW,dcid:country/TWN
+TZ,dcid:country/TZA
+UA,dcid:country/UKR
+UG,dcid:country/UGA
+UM,dcid:country/UMI
+US,dcid:country/USA
+UY,dcid:country/URY
+UZ,dcid:country/UZB
+VA,dcid:country/VAT
+VC,dcid:country/VCT
+VE,dcid:country/VEN
+VG,dcid:country/VGB
+VI,dcid:country/VIR
+VN,dcid:country/VNM
+VU,dcid:country/VUT
+WF,dcid:country/WLF
+WS,dcid:country/WSM
+XK,dcid:country/XKS
+YE,dcid:country/YEM
+YT,dcid:country/MYT
+ZA,dcid:country/ZAF
+ZM,dcid:country/ZMB
+ZW,dcid:country/ZWE
+DE_TOT,dcid:nuts/DE_TOT
+EA,dcid:nuts/EA
+EA12,dcid:nuts/EA12
+EA18,dcid:nuts/EA18
+EA19,dcid:nuts/EA19
+EEA,dcid:nuts/EEA
+EEA30_2007,dcid:nuts/EEA30_2007
+EEA31,dcid:nuts/EEA31
+EFTA,dcid:nuts/EFTA
+EU,dcid:nuts/EU
+EU27,dcid:nuts/EU27
+EU27_2007,dcid:nuts/EU27_2007
+EU27_2020,dcid:nuts/EU27_2020
+EA20,dcid:nuts/EA20
+EA21,dcid:nuts/EA21
+EU28,dcid:nuts/EU28