Skip to content

Optimise missing sources - #899

Open
mauch wants to merge 11 commits into
v2.0from
optimise_missing_sources
Open

Optimise missing sources#899
mauch wants to merge 11 commits into
v2.0from
optimise_missing_sources

Conversation

@mauch

@mauch mauch commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

Modify get_src_skyregion_merged_df to improve speed and memory efficiency. This also preps this function for furture checkpoint and restart operations.

Changes are:

  • Replace source/image/epoch strings with smaller int32 indices for the duration of the function and use int64 for datetime sorting rather than a Timestamp object
  • Use numpy arrays and vectorised lookups and tricks inplace where possible to avoid large pandas dataframe copies during merge/groupby/sort operations
  • Convert the output missing sources dataframe to dask before exploding the image_diff column and returning source IDs to srtings at the end of the function
  • Get rid of the two explodes on missing_sources further downstream in new_sources and forced_extraction

These changes mean that the function now returns a persisted dask dataframe instead of a pandas dataframe, so I removed the subsequent lines in main.py which convert this dataframe to dask.

@mauch
mauch marked this pull request as draft August 19, 2026 23:44
@mauch
mauch marked this pull request as ready for review August 20, 2026 00:07
@mauch
mauch requested a review from ddobie August 20, 2026 00:08
@ddobie

ddobie commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

The code looks good. I've just tried to put a run through and I'm hitting an error, but it doesn't seem to be related to the changes - will try and take a look at it in detail soon.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants