Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions .github/workflows/build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -137,6 +137,11 @@ jobs:
path: astro-data
enableCrossOsArchive: true
key: satkit-data-${{ hashFiles('data/manifest.json') }}
# On a manifest change the exact key misses; seed from the newest
# older cache so EOP/SW survive a CelesTrak outage (the download
# step then verifies/updates the manifest files, and the refresh
# step updates EOP/SW when CelesTrak is reachable).
restore-keys: satkit-data-

- name: Download Satkit Data
if: steps.cache-satkit-data.outputs.cache-hit != 'true'
Expand Down Expand Up @@ -232,6 +237,11 @@ jobs:
path: astro-data
enableCrossOsArchive: true
key: satkit-data-${{ hashFiles('data/manifest.json') }}
# On a manifest change the exact key misses; seed from the newest
# older cache so EOP/SW survive a CelesTrak outage (the download
# step then verifies/updates the manifest files, and the refresh
# step updates EOP/SW when CelesTrak is reachable).
restore-keys: satkit-data-

- name: Download Satkit Data
if: steps.cache-satkit-data.outputs.cache-hit != 'true'
Expand Down
3 changes: 3 additions & 0 deletions .github/workflows/docs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,9 @@ jobs:
with:
path: astro-data
key: satkit-data-${{ hashFiles('data/manifest.json') }}
# Seed from the newest older cache on a manifest change, so EOP/SW
# survive a CelesTrak outage.
restore-keys: satkit-data-

- name: Download satkit data (manifest-verified)
if: steps.cache-astro-data.outputs.cache-hit != 'true'
Expand Down
3 changes: 3 additions & 0 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -72,6 +72,9 @@ jobs:
path: astro-data
enableCrossOsArchive: true
key: satkit-data-${{ hashFiles('data/manifest.json') }}
# Seed from the newest older cache on a manifest change, so EOP/SW
# survive a CelesTrak outage.
restore-keys: satkit-data-

- name: Download Satkit Data
if: steps.cache-satkit-data.outputs.cache-hit != 'true'
Expand Down
15 changes: 15 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -38,3 +38,18 @@ site
docs/images/*.svg
# GMAT scripts kept by tests/gmat/generate.py --keep-scripts
tests/gmat/scripts/

# Python tooling caches
.mypy_cache/
.ruff_cache/
dist/

# Jupyter checkpoints (docs/tutorials notebooks)
.ipynb_checkpoints/

# proptest failure artifacts (counterexamples get pinned as named tests instead)
proptest-regressions/
**/*.proptest-regressions

# scratch scripts stay out of commits
/scratch/
6 changes: 6 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,12 @@

Only recent releases are listed. Older entries are in this file's git history (`git show vX.Y.Z:CHANGELOG.md`) and on the [GitHub Releases](https://github.com/ssmichael1/satkit/releases) page.

## Unreleased

### Changed

- `update_datafiles()` no longer downloads files that are compiled into the library: the IERS tables and gravity models are `default: false` in the manifest (still pinned and fetchable by name), and the unused `leap-seconds.list` (nothing ever read it — the runtime leap-second table is a compiled-in constant) is removed from the manifest entirely. The only static download left is the JPL ephemeris, alongside the daily EOP / space-weather / solar-cycle refreshes ([#163](https://github.com/ssmichael1/satkit/pull/163))

## 0.21.2 - 2026-08-30

### Changed
Expand Down
28 changes: 17 additions & 11 deletions data/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,15 +48,18 @@ All manifest URLs must be `https://` (validated on load).
|---|---|---|---|---|
| `linux_p1550p2650.440` | 102.3 MB | JPL | DE440 (Park et al. 2021). US Government work, public domain. Origin URL verified byte-identical | ephemeris (default) |
| `lnxp1900p2053.421` | 14.0 MB | JPL | DE421 (Folkner et al. 2009). US Government work, public domain. `default: false` — fetched only by name (e.g. the conda package ships this one) | ephemeris |
| `tab5.2a.txt`, `tab5.2b.txt`, `tab5.2d.txt` | 171 / 137 / 9 KB | IERS | IERS Conventions (2010), TN 36, Tables 5.2a/b/d. Freely redistributable. Origin URLs verified byte-identical | core |
| `EGM96.gfc` | 5.6 MB | ICGEM (GFZ) | EGM96, Lemoine et al. 1998, NASA GSFC/NIMA — US Government work | core |
| `JGM2.gfc`, `JGM3.gfc` | 118 / 215 KB | ICGEM (GFZ) | JGM-2 (Nerem et al. 1994), JGM-3 (Tapley et al. 1996), NASA GSFC / UT CSR — US Government work | core |
| `ITU_GRACE16.gfc` | 1.8 MB | ICGEM (GFZ) | Akyilmaz et al. 2016, GFZ Data Services, **CC BY 4.0** — keep the file's header block, it carries the attribution | core |
| `leap-seconds.list` | 11 KB | IERS / IETF | Public data. Reference only: the runtime leap-second table is compiled in | reference |

`tier` is informational today: `core` = small files needed for frames and
gravity, `ephemeris` = the large JPL files, `reference` = not read at runtime.
Phase 2 (below) would embed the `core` tier in the binary.
| `tab5.2a.txt`, `tab5.2b.txt`, `tab5.2d.txt` | 171 / 137 / 9 KB | IERS | IERS Conventions (2010), TN 36, Tables 5.2a/b/d. Freely redistributable. Origin URLs verified byte-identical. `default: false` — embedded byte-identical in the binary | core |
| `EGM96.gfc` | 5.6 MB | ICGEM (GFZ) | EGM96, Lemoine et al. 1998, NASA GSFC/NIMA — US Government work. `default: false` — embedded to degree 70 | core |
| `JGM2.gfc`, `JGM3.gfc` | 118 / 215 KB | ICGEM (GFZ) | JGM-2 (Nerem et al. 1994), JGM-3 (Tapley et al. 1996), NASA GSFC / UT CSR — US Government work. `default: false` — embedded to degree 70 | core |
| `ITU_GRACE16.gfc` | 1.8 MB | ICGEM (GFZ) | Akyilmaz et al. 2016, GFZ Data Services, **CC BY 4.0** — keep the file's header block, it carries the attribution. `default: false` — embedded to degree 70 | core |

`tier` is informational: `core` = the small files frames and gravity need
(embedded in the binary since Phase 2, below — which is why their manifest
entries are `default: false`: pinned and fetchable by name, but pointless to
download while evaluation is capped at degree 40), `ephemeris` = the large
JPL files. The only `default: true` entry — the only file
`update_datafiles()` downloads besides the daily refreshes — is the DE440
ephemeris.

## Files deliberately excluded

Expand All @@ -71,6 +74,10 @@ Phase 2 (below) would embed the `core` tier in the binary.
daily, so they are never pinned: `update_datafiles()` fetches them from
CelesTrak every run via the manifest's `refresh` list.
- **`sw19571001.txt`** — an orphan on the old bucket; nothing reads it.
- **`leap-seconds.list`** — nothing reads it: the runtime leap-second table
is a compiled-in constant (`src/time/instant.rs`). It was pinned
(`tier: reference`) through 0.21.2 and remains an asset on the immutable
`data-v1` tag, but the manifest entry is gone.
- **`predicted-solar-cycle.json`** — fetched directly from NOAA/SWPC by
`solar_cycle_forecast::update()`; not a bundle file.

Expand All @@ -89,8 +96,7 @@ gh release create data-v1 --repo ssmichael1/satkit-data --latest=false \
--notes "Static data files pinned by satkit's data/manifest.json (sizes and SHA-256 there). Sources and licences: see data/README.md in ssmichael1/satkit." \
"$D/linux_p1550p2650.440" "$D/lnxp1900p2053.421" \
"$D/tab5.2a.txt" "$D/tab5.2b.txt" "$D/tab5.2d.txt" \
"$D/EGM96.gfc" "$D/ITU_GRACE16.gfc" "$D/JGM2.gfc" "$D/JGM3.gfc" \
"$D/leap-seconds.list"
"$D/EGM96.gfc" "$D/ITU_GRACE16.gfc" "$D/JGM2.gfc" "$D/JGM3.gfc"
```

(`lnxp1900p2053.421` can be fetched first with
Expand Down
30 changes: 9 additions & 21 deletions data/manifest.json
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,8 @@
"URL order per file: GitHub release asset, then the origin server when it serves byte-identical bytes, then the legacy GCS bucket (transitional).",
"SATKIT_DATA_URL, if set, is a base URL tried before all of these.",
"Regenerate with tools/make_manifest.py; see data/README.md.",
"Deliberately excluded: msis21.parm (NRLMSIS 2.1 licence is academic/non-commercial), EOP-All.csv / SW-All.csv (CelesTrak asks that its compiled files not be mirrored; fetched at runtime via `refresh`), sw19571001.txt (orphan). See data/README.md."
"Deliberately excluded: msis21.parm (NRLMSIS 2.1 licence is academic/non-commercial), EOP-All.csv / SW-All.csv (CelesTrak asks that its compiled files not be mirrored; fetched at runtime via `refresh`), sw19571001.txt (orphan), leap-seconds.list (nothing reads it; the runtime leap-second table is compiled into src/time/instant.rs). See data/README.md.",
"default: false = pinned and fetchable by name, but skipped by update_datafiles: the IERS tables are embedded byte-identical, and the full gravity files add nothing while evaluation is capped at degree 40 (the embedded copies carry degree 70)."
],
"files": [
{
Expand Down Expand Up @@ -49,7 +50,7 @@
"source": "IERS",
"license": "IERS Conventions (2010), Technical Note 36, Table 5.2a (X series); freely redistributable",
"tier": "core",
"default": true
"default": false
},
{
"name": "tab5.2b.txt",
Expand All @@ -63,7 +64,7 @@
"source": "IERS",
"license": "IERS Conventions (2010), Technical Note 36, Table 5.2b (Y series); freely redistributable",
"tier": "core",
"default": true
"default": false
},
{
"name": "tab5.2d.txt",
Expand All @@ -77,7 +78,7 @@
"source": "IERS",
"license": "IERS Conventions (2010), Technical Note 36, Table 5.2d (s + XY/2 series); freely redistributable",
"tier": "core",
"default": true
"default": false
},
{
"name": "EGM96.gfc",
Expand All @@ -90,7 +91,7 @@
"source": "ICGEM-GFZ",
"license": "EGM96 (Lemoine et al. 1998, NASA GSFC/NIMA) as distributed by ICGEM; US Government work",
"tier": "core",
"default": true
"default": false
},
{
"name": "ITU_GRACE16.gfc",
Expand All @@ -103,7 +104,7 @@
"source": "ICGEM-GFZ",
"license": "ITU_GRACE16 (Akyilmaz et al. 2016, GFZ Data Services doi:10.5880/icgem.2016.006), CC BY 4.0",
"tier": "core",
"default": true
"default": false
},
{
"name": "JGM2.gfc",
Expand All @@ -116,7 +117,7 @@
"source": "ICGEM-GFZ",
"license": "JGM-2 (Nerem et al. 1994, NASA GSFC / UT CSR) as distributed by ICGEM; US Government work",
"tier": "core",
"default": true
"default": false
},
{
"name": "JGM3.gfc",
Expand All @@ -129,20 +130,7 @@
"source": "ICGEM-GFZ",
"license": "JGM-3 (Tapley et al. 1996, NASA GSFC / UT CSR) as distributed by ICGEM; US Government work",
"tier": "core",
"default": true
},
{
"name": "leap-seconds.list",
"size": 10665,
"sha256": "14b4faab51b1885680a704744863bda8b5728a0f0a5e07a15c8469532318b305",
"urls": [
"https://github.com/ssmichael1/satkit-data/releases/download/data-v1/leap-seconds.list",
"https://storage.googleapis.com/astrokit-astro-data/leap-seconds.list"
],
"source": "IERS",
"license": "IERS/USNO leap-second list; public data. Reference only: satkit's runtime leap-second table is compiled in",
"tier": "reference",
"default": true
"default": false
}
],
"refresh": [
Expand Down
12 changes: 9 additions & 3 deletions docs/getting-started/datafiles.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ Two caveats on "offline". Frame transforms need Earth-orientation parameters as

- **EOP-All.csv** — Earth orientation parameters. This includes $\Delta UT1$, the difference between $UT1$ and $UTC$, as well as $x_p$ and $y_p$, the polar "wander" of the Earth rotation axis. This file is updated daily with most-recent values at [celestrak.org](https://www.celestrak.org) (which repackages the IERS Bulletin A / finals series) and carries IERS predictions roughly six months ahead. For dates beyond the file, the last entry's values are used (constant extrapolation) — see [EOP coverage](#eop-coverage) below.

- **leap-seconds.list** — Downloaded by `update_datafiles()` for reference only. The UTC↔TAI leap-second table that `satkit` actually uses is compiled into the library (current through the most recent leap second, 2017-01-01, when UTC began lagging TAI by 37 s); this file is not read at runtime, and a future leap second will require a new `satkit` release. The table is transcribed from [IERS Bulletin C](../guide/references.md#bulletinc); UTC and leap seconds are defined by [ITU-R TF.460-6](../guide/references.md#itu460).
- **Leap seconds** — not a file: the UTC↔TAI leap-second table is compiled into the library (current through the most recent leap second, 2017-01-01, when UTC began lagging TAI by 37 s), and a future leap second will require a new `satkit` release. The table is transcribed from [IERS Bulletin C](../guide/references.md#bulletinc); UTC and leap seconds are defined by [ITU-R TF.460-6](../guide/references.md#itu460). (Releases through 0.21.2 also downloaded a reference `leap-seconds.list`; nothing ever read it, and it is no longer fetched.)

## Where satkit looks for data, and where it writes

Expand Down Expand Up @@ -116,7 +116,7 @@ data bytes.
Sources and attribution: DE440 / DE421 — JPL (Park et al. 2021; Folkner et al.
2009), US Government work; `tab5.2a/b/d.txt` — IERS Conventions (2010), TN 36;
EGM96, JGM-2, JGM-3 — NASA GSFC (public), via ICGEM; ITU_GRACE16 — Akyilmaz et
al. 2016, GFZ Data Services, CC BY 4.0; `leap-seconds.list` — IERS/IETF. The
al. 2016, GFZ Data Services, CC BY 4.0. The
Earth-orientation and space-weather files are fetched from CelesTrak on every
update and are not pinned (they change daily). The full table, with licences,
is in `data/README.md`.
Expand All @@ -139,13 +139,19 @@ a CI job, or a machine that will later be offline:

```python
import satkit as sk
sk.utils.update_datafiles() # ephemeris + full-degree gravity files + IERS tables + EOP/SW, verified
sk.utils.update_datafiles() # ephemeris (verified) + EOP/SW + solar-cycle forecast
```

Files already present with the right hash are skipped; the space-weather and
Earth-orientation files are always refreshed. `update_datafiles(dir="...")`
writes somewhere else; `overwrite=True` re-downloads even verified files.

The IERS tables and gravity models are **not** downloaded — they are compiled
in (the tables byte-identical, gravity to degree 70, identical results at the
degree-40 evaluation cap). The full-degree `.gfc` files remain pinned in the
manifest and hosted on the `data-v1` release; drop one into a search
directory and it takes precedence over the compiled-in copy.

### The optional `satkit-data` bundle

`pip install satkit[data]` installs the `satkit-data` package (~110 MB: the
Expand Down
2 changes: 1 addition & 1 deletion docs/getting-started/installation.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ To provision everything up front (a Docker image, a CI job, a machine that will

```python
import satkit as sk
sk.utils.update_datafiles() # ephemeris + IERS tables + gravity files + EOP/SW, verified
sk.utils.update_datafiles() # ephemeris (verified) + EOP/SW + solar-cycle forecast
```

### Offline and air-gapped use
Expand Down
22 changes: 10 additions & 12 deletions python/satkit/utils.pyi
Original file line number Diff line number Diff line change
Expand Up @@ -27,19 +27,17 @@ def update_datafiles(**kwargs) -> None:
manifest. Files already present with the right hash are skipped.

Notes:
- Files include:
- ``EGM96.gfc`` : EGM-96 Gravity Model Coefficients
- ``JGM3.gfc`` : JGM-3 Gravity Model Coefficients
- ``JGM2.gfc`` : JGM-2 Gravity Model Coefficients
- ``ITU_GRACE16.gfc`` : ITU Grace 16 Gravity
- ``tab5.2a.txt`` : Coefficients for GCRS to GCRF conversion
- ``tab5.2b.txt`` : Coefficients for GCRS to GCRF conversion
- ``tab5.2d.txt`` : Coefficients for GCRS to GCRF conversion
- ``SW-ALL.csv`` : Space weather data, updated daily
- ``predicted-solar-cycle.json`` : NOAA/SWPC solar cycle forecast (~5 years of predicted F10.7)
- ``leap-seconds.list`` : Leap seconds (UTC vs TAI); reference only — the runtime table is compiled in
- ``EOP-All.csv`` : Earth orientation parameters, updated daily
- Files downloaded:
- ``linux_p1550p2650.440`` : JPL Ephemeris version 440 (~ 100 MB)
- ``EOP-All.csv`` : Earth orientation parameters, updated daily
- ``SW-All.csv`` : Space weather data, updated daily
- ``predicted-solar-cycle.json`` : NOAA/SWPC solar cycle forecast (~5 years of predicted F10.7)

- The IERS nutation tables (``tab5.2a/b/d.txt``) and the gravity
models (EGM96, JGM2, JGM3, ITU_GRACE16 — to degree 70) are compiled
into satkit and are not downloaded. A full-degree gravity file or an
updated IERS table placed in the data directory still takes
precedence over the compiled-in copy.

- The space weather and earth orientation parameters files are updated
daily and will always be downloaded regardless of the overwrite flag
Expand Down
23 changes: 11 additions & 12 deletions python/src/mod_utils.rs
Original file line number Diff line number Diff line change
Expand Up @@ -21,19 +21,18 @@ use anyhow::Result;
/// dir (str): Target directory for files. Uses ``datadir()`` if not specified
///
///
/// Files include:
///
/// * EGM96.gfc :: EGM-96 Gravity Model Coefficients
/// * JGM3.gfc :: JGM-3 Gravity Model Coefficients
/// * JGM2.gfc :: JGM-2 Gravity Model Coefficients
/// * ITU_GRACE16.gfc :: ITU Grace 16 Gravity Model Coefficients
/// * tab5.2a.txt :: Coefficients for GCRS to GCRF conversion
/// * tab5.2b.txt :: Coefficients for GCRS to GCRF conversion
/// * tab5.2d.txt :: Coefficients for GCRS to GCRF conversion
/// * SW-All.csv :: Space weather data, updated daily
/// * leap-seconds.list :: Leap seconds (UTC vs TAI); reference only, the runtime table is compiled in
/// * EOP_All.csv :: Earth orientation parameters, updated daily
/// Files downloaded:
///
/// * linux_p1550p2650.440 :: JPL Ephemeris version 440 (~ 100 MB)
/// * EOP-All.csv :: Earth orientation parameters, updated daily
/// * SW-All.csv :: Space weather data, updated daily
/// * predicted-solar-cycle.json :: NOAA/SWPC solar cycle forecast
///
/// The IERS nutation tables (tab5.2a/b/d.txt) and the gravity models
/// (EGM96, JGM2, JGM3, ITU_GRACE16 — to degree 70) are compiled into
/// satkit and are not downloaded. A full-degree gravity file or an updated
/// IERS table placed in the data directory still takes precedence over the
/// compiled-in copy.
///
/// Static files are fetched per the compiled-in data manifest
/// (`data/manifest.json`): `SATKIT_DATA_URL` mirror first if set, then the
Expand Down
13 changes: 10 additions & 3 deletions src/utils/manifest.rs
Original file line number Diff line number Diff line change
Expand Up @@ -476,6 +476,10 @@ mod tests {
m.entry("msis21.parm").is_none(),
"NRLMSIS 2.1 licence forbids redistribution"
);
assert!(
m.entry("leap-seconds.list").is_none(),
"nothing reads it; the runtime leap-second table is compiled in"
);
// Refresh files must not be pinned.
assert!(m.entry("EOP-All.csv").is_none());
assert!(m.entry("SW-All.csv").is_none());
Expand All @@ -488,9 +492,12 @@ mod tests {
e.name
);
}
// The default set includes the DE440 ephemeris but not DE421.
assert!(m.entry("linux_p1550p2650.440").unwrap().default);
assert!(!m.entry("lnxp1900p2053.421").unwrap().default);
// The only default download is the DE440 ephemeris: everything else
// pinned here is either embedded in the binary (IERS tables,
// gravity to degree 70) or an alternative (DE421), kept fetchable
// by name but not worth downloading on every install.
let defaults: Vec<&str> = m.default_files().map(|e| e.name.as_str()).collect();
assert_eq!(defaults, ["linux_p1550p2650.440"]);
assert_eq!(embedded().data_version, m.data_version);
}

Expand Down
Loading