Promote dev

.github/workflows/backend-ci.yml

@@ -34,7 +34,7 @@ jobs:
 
       # Matches template.yaml's Runtime and the deploy workflow.
       - name: Set up Python
-        uses: actions/setup-python@v5
+        uses: actions/setup-python@v7
         with:
           python-version: '3.13'
           cache: pip

.github/workflows/codeql.yml

@@ -24,7 +24,7 @@ jobs:
 
     steps:
     - name: Checkout repository
-      uses: actions/checkout@v5
+      uses: actions/checkout@v7
 
     - name: Initialize CodeQL
       uses: github/codeql-action/init@v4

.github/workflows/deploy-aws-backend.yml

@@ -13,17 +13,17 @@ jobs:
       id-token: write
       contents: read
     steps:
-      - uses: actions/checkout@v5
+      - uses: actions/checkout@v7
 
-      - uses: actions/setup-python@v5
+      - uses: actions/setup-python@v7
         with:
           python-version: '3.13'
 
       - uses: aws-actions/setup-sam@v3
         with:
           use-installer: true
 
-      - uses: aws-actions/configure-aws-credentials@v5
+      - uses: aws-actions/configure-aws-credentials@v6
         with:
           role-to-assume: ${{ secrets.AWS_ROLE_ARN }}
           aws-region: us-east-1

README.md

@@ -62,6 +62,7 @@ pnpm install
 # AWS_ACCESS_KEY_ID=your_key
 # AWS_SECRET_ACCESS_KEY=your_secret
 # EXOPLANETS_DATABASE_TABLE=exoplanets-dev
+# EXOPLANETS_RECORDS_TABLE=exoplanet-records-dev
 
 pnpm dev
 ```

ROADMAP.md

@@ -4,14 +4,14 @@
 
 ExoplanetHub is an archive first: a complete, queryable, shareable view of NASA's confirmed
 exoplanets. What the site presents as new is derived from the scheduled sync's diff against the
-archive, never hand-written. Today that is the latest-confirmations feed; retractions and records
-broken follow from the same diff. There is no blog and no editorial pipeline; if the data does not
-say it, the site does not claim it.
+archive, never hand-written. Today that is the latest-confirmations feed and records broken;
+retractions follow from the same diff. There is no blog and no editorial pipeline; if the data does
+not say it, the site does not claim it.
 
 ## Shipped
 
 ### 1. Earth-Similarity Index (ESI) Badge & Sorting
-Add visual indicators and sorting for planets most similar to Earth based on radius, density, and temperature.
+Add visual indicators and sorting for planets most similar to Earth based on radius, mass, and temperature.
 
 **Tasks:**
 - Calculate ESI score for each planet
@@ -50,6 +50,18 @@ non-astronomer and unfurls with a real preview wherever the link is shared.
 - Link planet names from the table, cards, modal and Latest Discoveries
 - Sitemap and robots entries so the pages are indexable
 
+### 5. Records Broken
+Persist the archive's superlatives — hottest, largest, smallest, most massive, shortest year,
+nearest and most Earth-like — on every sync, record each change of holder with a date, and surface
+them as the second feed derived from the sync's diff rather than written by hand.
+
+**Tasks:**
+- One ESI rule everywhere: a score exists only when radius, mass and temperature are all measured
+- Sync computes each record's holder and persists holder, tenure and displaced holders in DynamoDB
+- `/records` page: every record's holder, value, tenure and previous holders, each linking to its planet page
+- Homepage strip of the three most recently changed records, linking to `/records`
+- "Most Earth-like" framed as closest to Earth's conditions, explicitly not "habitable"
+
 ## Future Ideas
 
 - 3D visualization of exoplanet systems

aws-backend/README.md

@@ -7,10 +7,39 @@ archive no longer lists, so the table converges to the archive's count unless th
 sweep refuses to delete more than 5% of the table in one run — a truncated fetch aborts it with an
 error log and leaves the drift in place rather than emptying the table, and the upserts still stand.
 
-The invocation result reports `total_synced`, `removals_submitted` and `sweep_aborted`; every
-removed `pl_name` is logged individually. `removals_submitted` counts what was handed to DynamoDB:
-exact when `sweep_aborted` is false, an upper bound when it is true, because a batch can commit and
-still fail. A run that aborts mid-sweep is safe to retry — deletions are idempotent.
+Deletion is the one step the next run cannot recompute, so before any record is deleted the sweep
+writes a tombstone to `exoplanet-tombstones-<env>` — `{pl_name, removed_at, last_known_snapshot}`,
+where the snapshot is the full item as last stored and `removed_at` is the run's `last_updated`
+timestamp. If the tombstone batch fails, nothing is deleted. A planet the archive drops, relists and
+drops again keeps only its latest tombstone. The table is retained if the stack is deleted or the
+resource replaced, because a resync cannot rebuild it.
+
+After the sweep, the sync reconciles the records table, `exoplanet-records-<env>` (env
+`RECORDS_TABLE_NAME`): one item per superlative in `records.py`'s `RECORDS` registry, keyed by
+`record_id` — `{record_id, holder: {pl_name, value}, since, previous, updated_at}`. Each record's
+holder is computed from the fetched archive (finite, positive values only; Most Earth-like ranks on
+the unrounded ESI similarity, so its candidates are exactly the scored planets; ties go to the
+alphabetically first name). A new holder sets `since` to the run's timestamp and prepends the
+displaced holder `{pl_name, value, since, until}` to `previous`, which keeps the latest 20. The same
+holder with a moved value refreshes `holder.value` only; an identical value writes nothing, and a
+record with no measurable candidates is left untouched. The first sync — or a record id added to the
+registry later — writes a baseline with an empty `previous`, so its `since` is when tracking began,
+not when the holder won. Values are stored as `Decimal` through `values.to_decimal`. The table is
+retained like the tombstones: a resync recomputes the holder but never the history, which is also
+why the Lambda gets read and write access but not `DynamoDBCrudPolicy`: it never needs `DeleteItem`.
+
+Records are skipped when the sweep aborted, because a truncated fetch would fabricate a broken
+record that flips back on the next run. The step never fails the sync: a failure is logged and
+reported as `records_aborted`.
+
+The invocation result reports `total_synced`, `removals_submitted`, `sweep_aborted`,
+`records_changed` and `records_aborted`; every removed `pl_name` and every record written is logged
+individually. `removals_submitted` counts what was handed to DynamoDB: exact when `sweep_aborted` is
+false, an upper bound when it is true, because a batch can commit and still fail. `records_changed`
+counts items written — a new holder or a refreshed value — and is a lower bound when
+`records_aborted` is true. A run that aborts mid-sweep is safe to retry — tombstones and deletions
+are idempotent, though a retry re-puts each tombstone with its own `removed_at`, so the recorded
+time is the last attempt's, not the first run that saw the record stale.
 
 ## Local Deployment
 

aws-backend/lambda/sync/app.py

@@ -3,15 +3,19 @@
 import os
 import urllib.request
 import boto3
-from decimal import Decimal
 from datetime import datetime
 from esi import compute_esi
+from records import update_records_after_sweep
 from sweep import sweep_removed
+from values import to_decimal
 
 logging.getLogger('sweep').setLevel(logging.INFO)
+logging.getLogger('records').setLevel(logging.INFO)
 
 dynamodb = boto3.resource('dynamodb')
 table = dynamodb.Table(os.environ['TABLE_NAME'])
+tombstones = dynamodb.Table(os.environ['TOMBSTONES_TABLE_NAME'])
+records_table = dynamodb.Table(os.environ['RECORDS_TABLE_NAME'])
 
 def lambda_handler(event, context):
     nasa_url = "https://exoplanetarchive.ipac.caltech.edu/TAP/sync?query=select+pl_name,hostname,sy_snum,sy_pnum,sy_dist,discoverymethod,disc_year,disc_facility,pl_orbper,pl_orbsmax,pl_rade,pl_bmasse,pl_dens,pl_eqt,pl_insol,st_teff,st_rad,st_mass,st_logg,st_age+from+ps+where+default_flag=1&format=json"
@@ -54,18 +58,16 @@ def lambda_handler(event, context):
 
             batch.put_item(Item=item)
 
-    sweep = sweep_removed(table, archive_names)
+    sweep = sweep_removed(table, tombstones, archive_names, removed_at=timestamp)
+    records = update_records_after_sweep(sweep, records_table, data, timestamp)
 
     return {
         'statusCode': 200,
         'body': json.dumps({
             'total_synced': len(data),
             'removals_submitted': len(sweep.submitted),
-            'sweep_aborted': sweep.aborted
+            'sweep_aborted': sweep.aborted,
+            'records_changed': len(records.changed),
+            'records_aborted': records.aborted
         })
     }
-
-def to_decimal(value):
-    if value is None:
-        return None
-    return Decimal(str(value))

aws-backend/lambda/sync/dynamo.py

@@ -0,0 +1,9 @@
+def scan_all(table, key):
+    request = {}
+    items = {}
+    while True:
+        page = table.scan(**request)
+        items.update((item[key], item) for item in page['Items'])
+        if not page.get('LastEvaluatedKey'):
+            return items
+        request['ExclusiveStartKey'] = page['LastEvaluatedKey']

aws-backend/lambda/sync/esi.py

@@ -1,34 +1,27 @@
 import math
 
+from values import measured
+
 EARTH_REFERENCES = {
     'pl_rade': 1.0,
     'pl_eqt': 288.0,
     'pl_bmasse': 1.0,
 }
 
-MINIMUM_COMPONENTS = 2
-
 
-def compute_esi(record):
+def esi_similarity(record):
     similarities = []
     for field, earth_value in EARTH_REFERENCES.items():
         value = record.get(field)
-        if _is_measured(value):
-            similarities.append(1 - abs((value - earth_value) / (value + earth_value)))
+        if not measured(value):
+            return None
+        similarities.append(1 - abs((value - earth_value) / (value + earth_value)))
 
-    if len(similarities) < MINIMUM_COMPONENTS:
-        return None
+    return math.prod(similarities) ** (1 / len(similarities))
 
-    geometric_mean = math.prod(similarities) ** (1 / len(similarities))
-    return round(100 * geometric_mean)
 
-
-def _is_measured(value):
-    # bool is an int subclass; the kernel is undefined at x <= 0; huge ints overflow float().
-    if isinstance(value, bool) or not isinstance(value, (int, float)):
-        return False
-    try:
-        number = float(value)
-    except OverflowError:
-        return False
-    return math.isfinite(number) and number > 0
+def compute_esi(record):
+    similarity = esi_similarity(record)
+    if similarity is None:
+        return None
+    return round(100 * similarity)

aws-backend/lambda/sync/records.py

@@ -0,0 +1,120 @@
+import logging
+from collections.abc import Callable
+from dataclasses import dataclass
+
+from dynamo import scan_all
+from esi import esi_similarity
+from values import measured, to_decimal
+
+logger = logging.getLogger(__name__)
+
+PREVIOUS_HOLDERS_KEPT = 20
+
+
+# `measure` maps a raw archive planet to the value ranked on, or None when it has none; `pick` is max or min.
+@dataclass(frozen=True)
+class RecordSpec:
+    id: str
+    measure: Callable[[dict], float | None]
+    pick: Callable
+
+
+def archive_value(field):
+    return lambda planet: planet.get(field)
+
+
+# The registry is the contract with the frontend: these ids are what it presents.
+RECORDS = (
+    RecordSpec('most-earth-like', esi_similarity, max),
+    RecordSpec('hottest', archive_value('pl_eqt'), max),
+    RecordSpec('largest', archive_value('pl_rade'), max),
+    RecordSpec('smallest', archive_value('pl_rade'), min),
+    RecordSpec('most-massive', archive_value('pl_bmasse'), max),
+    RecordSpec('shortest-year', archive_value('pl_orbper'), min),
+    RecordSpec('nearest', archive_value('sy_dist'), min),
+)
+
+
+# `changed` is every record id whose item was written: exact when `aborted` is False, a lower bound when True.
+@dataclass(frozen=True)
+class RecordsResult:
+    changed: tuple[str, ...]
+    aborted: bool
+
+
+# A sweep abort means the fetch may be incomplete, and a missing holder would fabricate a broken record.
+def update_records_after_sweep(sweep, records_table, data, timestamp):
+    if sweep.aborted:
+        logger.warning('records skipped: the sweep aborted, so the archive fetch may be incomplete')
+        return RecordsResult(changed=(), aborted=True)
+    return update_records(records_table, data, timestamp)
+
+
+# Never raises: the ingest has already committed and must still report success.
+def update_records(records_table, data, timestamp):
+    changed = []
+    try:
+        stored = scan_all(records_table, 'record_id')
+        for spec in RECORDS:
+            if _reconcile(spec, records_table, stored.get(spec.id), data, timestamp):
+                changed.append(spec.id)
+    except Exception:
+        logger.exception('records aborted after %d writes: reconcile failed against %d archive records', len(changed), len(data))
+        return RecordsResult(changed=tuple(changed), aborted=True)
+
+    return RecordsResult(changed=tuple(changed), aborted=False)
+
+
+def _reconcile(spec, records_table, stored, data, timestamp):
+    holder = _current_holder(spec, data)
+    if holder is None:
+        logger.warning('%s: no measurable candidates, left untouched', spec.id)
+        return False
+
+    item = _next_item(spec.id, holder, stored, timestamp)
+    if item is None:
+        return False
+
+    records_table.put_item(Item=item)
+    return True
+
+
+def _current_holder(spec, data):
+    candidates = [(planet['pl_name'], spec.measure(planet)) for planet in data]
+    measurable = [candidate for candidate in candidates if measured(candidate[1])]
+    if not measurable:
+        return None
+
+    # Sorted by name first, so among tied values the alphabetically first wins and a tie never flaps.
+    return spec.pick(sorted(measurable), key=lambda candidate: candidate[1])
+
+
+def _next_item(record_id, holder, stored, timestamp):
+    name, value = holder[0], to_decimal(holder[1])
+
+    if stored is None:
+        logger.info('%s: tracking %s (%s)', record_id, name, value)
+        return {
+            'record_id': record_id,
+            'holder': {'pl_name': name, 'value': value},
+            'since': timestamp,
+            'previous': [],
+            'updated_at': timestamp,
+        }
+
+    if stored['holder']['pl_name'] != name:
+        displaced = {**stored['holder'], 'since': stored['since'], 'until': timestamp}
+        logger.info('%s: %s (%s) takes the record from %s (%s)', record_id, name, value, displaced['pl_name'], displaced['value'])
+        return {
+            'record_id': record_id,
+            'holder': {'pl_name': name, 'value': value},
+            'since': timestamp,
+            'previous': [displaced, *stored['previous']][:PREVIOUS_HOLDERS_KEPT],
+            'updated_at': timestamp,
+        }
+
+    if stored['holder']['value'] == value:
+        return None
+
+    logger.info('%s: %s moved from %s to %s', record_id, name, stored['holder']['value'], value)
+    return {**stored, 'holder': {'pl_name': name, 'value': value}, 'updated_at': timestamp}