Missing K0/K1/B0 Calibration tables during 1GC

[NeliswaBhengu]

Hi,
I was running 1GC. Initially everything ran successfully all the jobs were stating that they ran successfully and have completed but the LOGS contained repeated severe errors and missing calibration table.

LOGS/slurm_CL1GC_l0.log:2026-05-12 02:09:52 SEVERE MeasTable::dUTC(Double) (file ../../measures/Measures/MeasTable.cc, line 4290) Leap second table TAI_UTC seems out-of-date. LOGS/slurm_CL1GC_l0.log:2026-05-12 02:09:52 SEVERE MeasTable::dUTC(Double) (file ../../measures/Measures/MeasTable.cc, line 4290)+ Until the table is updated (see the CASA documentation or your system admin), LOGS/slurm_CL1GC_l0.log:2026-05-12 02:09:52 SEVERE MeasTable::dUTC(Double) (file ../../measures/Measures/MeasTable.cc, line 4290)+ times and coordinates derived from UTC could be wrong by 1s or more. LOGS/slurm_CL1GC_l0.log:2026-05-12 02:18:25 SEVERE Exception Reported: Table /scratch3/users/neliswa/j0400_oxkat/GAINTABLES/cal_1GC_1768412772-sdp-l0_2026-04-15T20-57-57_lZP_1024ch_1024ch.ms_2026-05-12-04-09-48.K0 does not exist. LOGS/slurm_CL1GC_l0.log:2026-05-12 02:18:25 SEVERE gaincal:::: Task gaincal raised an exception of class RuntimeError with the following message: Table /scratch3/users/neliswa/j0400_oxkat/GAINTABLES/cal_1GC_1768412772-sdp-l0_2026-04-15T20-57-57_lZP_1024ch_1024ch.ms_2026-05-12-04-09-48.K0 does not exist. LOGS/slurm_CL1GC_l0.log:2026-05-12 02:18:26 SEVERE Exception Reported: Table /scratch3/users/neliswa/j0400_oxkat/GAINTABLES/cal_1GC_1768412772-sdp-l0_2026-04-15T20-57-57_lZP_1024ch_1024ch.ms_2026-05-12-04-09-48.K0 does not exist. LOGS/slurm_CL1GC_l0.log:2026-05-12 02:18:26 SEVERE bandpass:::: Task bandpass raised an exception of class RuntimeError with the following message: Table /scratch3/users/neliswa/j0400_oxkat/GAINTABLES/cal_1GC_1768412772-sdp-l0_2026-04-15T20-57-57_lZP_1024ch_1024ch.ms_2026-05-12-04-09-48.K0 does not exist. LOGS/slurm_CL1GC_l0.log:2026-05-12 02:18:26 SEVERE flagdata:::: Task flagdata raised an exception of class RuntimeError with the following message: Parameter verification failed LOGS/slurm_CL1GC_l0.log:2026-05-12 02:18:26 SEVERE flagdata:::: Task flagdata raised an exception of class RuntimeError with the following message: Parameter verification failed LOGS/slurm_CL1GC_l0.log:2026-05-12 02:18:46 SEVERE Exception Reported: Table /scratch3/users/neliswa/j0400_oxkat/GAINTABLES/cal_1GC_1768412772-sdp-l0_2026-04-15T20-57-57_lZP_1024ch_1024ch.ms_2026-05-12-04-09-48.K0 does not exist. LOGS/slurm_CL1GC_l0.log:2026-05-12 02:18:46 SEVERE applycal:::: Task applycal raised an exception of class RuntimeError with the following message: Table /scratch3/users/neliswa/j0400_oxkat/GAINTABLES/cal_1GC_1768412772-sdp-l0_2026-04-15T20-57-57_lZP_1024ch_1024ch.ms_2026-05-12-04-09-48.K0 does not exist. LOGS/slurm_CL1GC_l0.log:2026-05-12 02:30:44 SEVERE Exception Reported: Table /scratch3/users/neliswa/j0400_oxkat/GAINTABLES/cal_1GC_1768412772-sdp-l0_2026-04-15T20-57-57_lZP_1024ch_1024ch.ms_2026-05-12-04-09-48.B0 does not exist. LOGS/slurm_CL1GC_l0.log:2026-05-12 02:30:44 SEVERE gaincal:::: Task gaincal raised an exception of class RuntimeError with the following message: Table /scratch3/users/neliswa/j0400_oxkat/GAINTABLES/cal_1GC_1768412772-sdp-l0_2026-04-15T20-57-57_lZP_1024ch_1024ch.ms_2026-05-12-04-09-48.B0 does not exist. LOGS/slurm_CL1GC_l0.log:2026-05-12 02:30:46 SEVERE Exception Reported: Table /scratch3/users/neliswa/j0400_oxkat/GAINTABLES/cal_1GC_1768412772-sdp-l0_2026-04-15T20-57-57_lZP_1024ch_1024ch.ms_2026-05-12-04-09-48.K1 does not exist. LOGS/slurm_CL1GC_l0.log:2026-05-12 02:30:46 SEVERE gaincal:::: Task gaincal raised an exception of class RuntimeError with the following message: Table /scratch3/users/neliswa/j0400_oxkat/GAINTABLES/cal_1GC_1768412772-sdp-l0_2026-04-15T20-57-57_lZP_1024ch_1024ch.ms_2026-05-12-04-09-48.K1 does not exist. LOGS/slurm_CL1GC_l0.log:2026-05-12 02:30:47 SEVERE Exception Reported: Table /scratch3/users/neliswa/j0400_oxkat/GAINTABLES/cal_1GC_1768412772-sdp-l0_2026-04-15T20-57-57_lZP_1024ch_1024ch.ms_2026-05-12-04-09-48.K1 does not exist. LOGS/slurm_CL1GC_l0.log:2026-05-12 02:30:47 SEVERE bandpass:::: Task bandpass raised an exception of class RuntimeError with the following message: Table /scratch3/users/neliswa/j0400_oxkat/GAINTABLES/cal_1GC_1768412772-sdp-l0_2026-04-15T20-57-57_lZP_1024ch_1024ch.ms_2026-05-12-04-09-48.K1 does not exist. LOGS/slurm_CL1GC_l0.log:2026-05-12 02:30:47 SEVERE flagdata:::: Task flagdata raised an exception of class RuntimeError with the following message: Parameter verification failed LOGS/slurm_CL1GC_l0.log:2026-05-12 02:30:47 SEVERE flagdata:::: Task flagdata raised an exception of class RuntimeError with the following message: Parameter verification failed LOGS/slurm_CL1GC_l0.log:2026-05-12 02:31:06 SEVERE Exception Reported: Table /scratch3/users/neliswa/j0400_oxkat/GAINTABLES/cal_1GC_1768412772-sdp-l0_2026-04-15T20-57-57_lZP_1024ch_1024ch.ms_2026-05-12-04-09-48.K1 does not exist. LOGS/slurm_CL1GC_l0.log:2026-05-12 02:31:06 SEVERE applycal:::: Task applycal raised an exception of class RuntimeError with the following message: Table /scratch3/users/neliswa/j0400_oxkat/GAINTABLES/cal_1GC_1768412772-sdp-l0_2026-04-15T20-57-57_lZP_1024ch_1024ch.ms_2026-05-12-04-09-48.K1 does not exist. LOGS/slurm_CL1GC_l0.log:2026-05-12 02:31:07 SEVERE Exception Reported: Table /scratch3/users/neliswa/j0400_oxkat/GAINTABLES/cal_1GC_1768412772-sdp-l0_2026-04-15T20-57-57_lZP_1024ch_1024ch.ms_2026-05-12-04-09-48.K1 does not exist. LOGS/slurm_CL1GC_l0.log:2026-05-12 02:31:07 SEVERE gaincal:::: Task gaincal raised an exception of class RuntimeError with the following message: Table /scratch3/users/neliswa/j0400_oxkat/GAINTABLES/cal_1GC_1768412772-sdp-l0_2026-04-15T20-57-57_lZP_1024ch_1024ch.ms_2026-05-12-04-09-48.K1 does not exist.

I do not understand why the K0/K1/B0 calibration table not generated. Please assist.

[IanHeywood]

Hi @NeliswaBhengu, there could be many reasons that the calibration tables have failed to generate. If you look at the logs sequentially to identify the problematic step then please post the relevant part and I might be able to advise further. The output of the ms_info file might also be informative.

[NeliswaBhengu]

@IanHeywood thanks for the response. I checked the logs sequentially and the first problematic part seems to occur during the 1GC calibration stage in CL1GC_l0.

The log first reports:

Found no unflagged data at:
(time=2026/01/14/17:52:40.6 field=0 spw=0 chan=0)

Found no unflagged data at:
(time=2026/01/14/21:05:31.7 field=0 spw=0 chan=0)

WARN Calibrater::solve
could not be solved due to missing (pre-)calibration

After that, the calibration tables fail to generate:

Exception Reported:
Table ...K0 does not exist.

and later similarly for K1 and B0, which eventually causes the pipeline to crash with:

OSError: [Errno 2] No such file or directory: '...K1'

I also checked the msinfo output. The fields are:

• J0408-6545 → PRIMARY CALIBRATOR
• J0447-2203 → SECONDARY CALIBRATOR
• ACJ0400M2926 → TARGET
• J0521+1638 → None

The MS otherwise appears normal.

[IanHeywood]

An entirely flagged calibrator scan would certainly break things, but the "no unflagged data" messages are generally harmless, it's just CASA complaining about entirely flagged regions (about half your data will be flagged for a typical L-band observation).

But K0 is the first thing that gets solved for and if that fails everything else will. There might not be anything informative in the CASA log for the gaincal operation that solves for K0 but it's worth a look.

[NeliswaBhengu]

I checked the logs around the K0 solve step. and the first I saw

Spectral window(s) 0 could not be solved due to missing (pre-)calibration

followed by:

Table ...K0 does not exist

So it seems the initial K0 delay calibration table was never generated, and then all later steps (B0, K1, applycal) failed because of that.

I also checked the GAINTABLES directory and no K0 table was created at all.

There are only a few of “Found no unflagged data” messages earlier in the log for specific times/channels, but otherwise nothing more informative appears before the K0 failure.

2026-05-12 02:09:52 SEVERE MeasTable::dUTC(Double) (file ../../measures/Measures/MeasTable.cc, line 4290) Leap second table TAI_UTC seems out-of-date. 2026-05-12 02:09:52 SEVERE MeasTable::dUTC(Double) (file ../../measures/Measures/MeasTable.cc, line 4290)+ Until the table is updated (see the CASA documentation or your system admin), 2026-05-12 02:09:52 SEVERE MeasTable::dUTC(Double) (file ../../measures/Measures/MeasTable.cc, line 4290)+ times and coordinates derived from UTC could be wrong by 1s or more. Found no unflagged data at: (time=2026/01/14/17:52:40.6 field=0 spw=0 chan=0) Found no unflagged data at: (time=2026/01/14/21:05:31.7 field=0 spw=0 chan=0) 2026-05-12 02:18:23 WARN Calibrater::solve Spectral window(s) 0, 2026-05-12 02:18:23 WARN Calibrater::solve+ could not be solved due to missing (pre-)calibration 2026-05-12 02:18:23 WARN Calibrater::solve+ in one or more of the specified tables. 2026-05-12 02:18:23 WARN Calibrater::solve+ Please check your results carefully!

Its possible that maybe this points to a fully flagged calibrator scan.

[IanHeywood]

Are you looking at the CASA logs themselves, not just the slurm logs? Can you give me access (read only is fine) to the data area please? I'll take a look when I get time.

[NeliswaBhengu]

I’ve mainly been looking at the slurm logs so far, but I’ll also check the CASA logs directly.

I can give you access to the project area. Please send me your ilifu/IDIA username and I’ll set the permissions.

[IanHeywood]

My username is ianh. Thanks.

[NeliswaBhengu]

Access granted. Thank you.

[NeliswaBhengu]

The data area is under:

/scratch3/users/neliswa/j0400_oxkat

[IanHeywood]

I had a look and the reason for the initial K0 solve is that your primary calibrator is 100% flagged. What is the provenance of the input MS? I see it already has a _1024ch suffix, so I'm assuming it's already been through some processing. I suggest going back to the original fresh-from-the-archive Measurement Set and restarting the process.