From d165be12f796aec658ad5185da406ac1e8bb4ca2 Mon Sep 17 00:00:00 2001 From: Aghiles Kheffache Date: Wed, 8 Jul 2026 21:26:39 +0000 Subject: [PATCH 01/40] Add an in-process E-PVM scheduler with reservations, backfill and lottery A tick-driven, in-process scheduler (scheduler.enabled) that plans placements by E-PVM opportunity cost instead of relying on the report-driven dispatcher: - E-PVM placement scored on cores/memory/GPU stranding, with a same-host locality bonus so a completing proc's cores tend to refill from the same layer. - Whole-host reservations + EASY backfill so wide jobs never strand behind narrow work; dispatcher.frame_cores_max raises the per-frame core clamp for whole-host jobs. - Priority-weighted lottery (Efraimidis-Spirakis) for booking order, so a sustained high-priority backlog cannot starve lower-priority work. Priority becomes a rate, not a rank (see Scheduler.md section 3.5). - License limits and folder/group core ceilings honored in-tick. The candidate query skips layers whose limit (limit_record.int_max_value) is full or whose folder (folder_resource.int_max_cores) is at its ceiling, so no doomed bookings are planned. Limits stay exact through the existing downstream frame query; the folder ceiling is held exactly by a pre-commit pass that trims any planned frame which would push a capped folder past its cap (planHost has no folder clause, so the batch must be trimmed before it commits). - Per-frame OOM memory handling: bump the offending frame and escalate the whole layer only after repeated OOMs, instead of ratcheting the layer on every kill. - Batch-commit robustness: skip a full host instead of aborting the whole tick, plus reserved/backfilled-core instrumentation on the scheduler stat line. Gated behind scheduler.enabled: with the scheduler off and booking on, the legacy dispatcher path is unchanged. The standalone-scheduler handoff mode (scheduler off + dispatcher.turn_off_booking=true) additionally suppresses FrameCompleteHandler's reactive rebook paths, so cuebot only reconciles RQD reports while an external planner owns booking. --- .../com/imageworks/spcue/VirtualProc.java | 22 +- .../com/imageworks/spcue/dao/FrameDao.java | 13 + .../com/imageworks/spcue/dao/ProcDao.java | 36 + .../spcue/dao/postgres/DispatcherDaoJdbc.java | 6 +- .../spcue/dao/postgres/FrameDaoJdbc.java | 102 + .../spcue/dao/postgres/ProcDaoJdbc.java | 103 + .../spcue/dispatcher/CoreUnitDispatcher.java | 150 +- .../spcue/dispatcher/DispatchSupport.java | 14 + .../dispatcher/DispatchSupportService.java | 90 + .../spcue/dispatcher/Dispatcher.java | 20 + .../spcue/dispatcher/FrameBooking.java | 35 + .../dispatcher/FrameCompleteHandler.java | 91 +- .../spcue/dispatcher/HostReportHandler.java | 8 +- .../spcue/dispatcher/LocalDispatcher.java | 8 + .../spcue/dispatcher/OomMemoryTracker.java | 86 + .../spcue/dispatcher/Scheduler.java | 2331 +++++++++++++++++ .../imageworks/spcue/dispatcher/Scheduler.md | 559 ++++ .../spcue/dispatcher/SchedulerMode.java | 79 + .../imageworks/spcue/rqd/RqdClientGrpc.java | 6 + .../com/imageworks/spcue/service/JobSpec.java | 15 +- .../spring/applicationContext-service.xml | 28 + cuebot/src/main/resources/opencue.properties | 76 + .../spcue/dispatcher/SchedulerTests.java | 396 +++ 23 files changed, 4257 insertions(+), 17 deletions(-) create mode 100644 cuebot/src/main/java/com/imageworks/spcue/dispatcher/FrameBooking.java create mode 100644 cuebot/src/main/java/com/imageworks/spcue/dispatcher/OomMemoryTracker.java create mode 100644 cuebot/src/main/java/com/imageworks/spcue/dispatcher/Scheduler.java create mode 100644 cuebot/src/main/java/com/imageworks/spcue/dispatcher/Scheduler.md create mode 100644 cuebot/src/main/java/com/imageworks/spcue/dispatcher/SchedulerMode.java create mode 100644 cuebot/src/test/java/com/imageworks/spcue/dispatcher/SchedulerTests.java diff --git a/cuebot/src/main/java/com/imageworks/spcue/VirtualProc.java b/cuebot/src/main/java/com/imageworks/spcue/VirtualProc.java index c48b8ff426..34f6c33606 100644 --- a/cuebot/src/main/java/com/imageworks/spcue/VirtualProc.java +++ b/cuebot/src/main/java/com/imageworks/spcue/VirtualProc.java @@ -85,6 +85,26 @@ public String getName() { */ public static final VirtualProc build(DispatchHost host, DispatchFrame frame, String... selfishServices) { + // Legacy/reactive dispatch: allow the thread-mode "grab idle cores" + // expansion so a single threadable frame can swell to fill the host. + return build(host, frame, true, selfishServices); + } + + /** + * Build a proc for a frame on a host. + * + * @param expandThreadable when {@code false}, reserve exactly + * {@code frame.minCores} and SKIP the thread-mode idle-core expansion. + * The whole-farm Scheduler uses this: it has already scored the + * placement and decremented its in-memory snapshot by the requested + * cores, so letting build() silently reserve more (grab-idle) would + * corrupt that accounting and make it over-plan a host whose cores a + * prior frame already swallowed. The Scheduler fills hosts by planning + * several placements, not by one frame ballooning. Explicit whole-host + * requests ({@code minCores <= 0}) are honored regardless. + */ + public static final VirtualProc build(DispatchHost host, DispatchFrame frame, + boolean expandThreadable, String... selfishServices) { VirtualProc proc = new VirtualProc(); proc.allocationId = host.getAllocationId(); proc.hostId = host.getHostId(); @@ -124,7 +144,7 @@ public static final VirtualProc build(DispatchHost host, DispatchFrame frame, } else if (proc.coresReserved < 0) { logger.debug("Reserving all cores minus " + proc.coresReserved); proc.coresReserved = host.cores + proc.coresReserved; - } else if (proc.coresReserved >= 100) { + } else if (proc.coresReserved >= 100 && expandThreadable) { int originalCores = proc.coresReserved; diff --git a/cuebot/src/main/java/com/imageworks/spcue/dao/FrameDao.java b/cuebot/src/main/java/com/imageworks/spcue/dao/FrameDao.java index f907536b25..3a401fc89f 100644 --- a/cuebot/src/main/java/com/imageworks/spcue/dao/FrameDao.java +++ b/cuebot/src/main/java/com/imageworks/spcue/dao/FrameDao.java @@ -167,6 +167,19 @@ public interface FrameDao { */ void updateFrameStarted(VirtualProc proc, FrameInterface frame); + /** + * Batch variant of {@link #updateFrameStarted}: marks many frames RUNNING in + * one round-trip with the same per-row optimistic version+state guard. No + * SELECT ... FOR UPDATE is taken; a frame whose state/version changed since + * planning simply updates zero rows and is reported as a loser. + * + * @param bookings the planned (frame, proc) pairs to start + * @return a mask, aligned to {@code bookings}, true where the frame was won + * (transitioned WAITING -> RUNNING by this call) + */ + boolean[] batchUpdateFramesStarted(java.util.List + bookings); + /** * Updates a frame to the stopped state. The frame MUST be in the Running state to be stopped. * diff --git a/cuebot/src/main/java/com/imageworks/spcue/dao/ProcDao.java b/cuebot/src/main/java/com/imageworks/spcue/dao/ProcDao.java index bf4ad01b42..07088d6109 100644 --- a/cuebot/src/main/java/com/imageworks/spcue/dao/ProcDao.java +++ b/cuebot/src/main/java/com/imageworks/spcue/dao/ProcDao.java @@ -96,6 +96,42 @@ public interface ProcDao { */ void insertVirtualProc(VirtualProc proc); + /** + * Batch variant of {@link #insertVirtualProc}: inserts many procs in one + * round-trip. Does NOT touch host idle resources (those are reserved up-front + * by {@link #reserveHostResourcesBatch}) nor the + * subscription/layer/job/folder/point counters, the Scheduler batches + * those separately. Each proc is assigned a fresh id. Intended for the + * Scheduler's batch commit path, where the frames were already won via a + * version-guarded update so duplicate inserts cannot occur. + * + * @param procs the procs to insert (non-local) + */ + void batchInsertVirtualProcs(java.util.List procs); + + /** + * Atomically reserve each host's aggregated idle-resource share for this tick's + * procs, using a guarded decrement that only books a host that currently has + * room for its whole share. Returns the set of host ids that had room (were + * decremented); procs on any other host must NOT be booked. This is what keeps + * the Scheduler from sending a proc to a host that cannot hold it: an + * unguarded decrement would drive idle negative, trip the + * verify_host_resources trigger, and abort the whole batched tick. + * + * @param procs the procs whose hosts to reserve (non-local) + * @return host ids that were successfully reserved + */ + java.util.Set reserveHostResourcesBatch(java.util.List procs); + + /** + * Return host idle resources reserved by {@link #reserveHostResourcesBatch} for + * procs that ended up not being booked (e.g. their frame lost the version + * race). A pure re-increment, so it can never drive idle negative. + * + * @param procs the procs whose host reservation to release + */ + void refundHostResourcesBatch(java.util.List procs); + /** * Deletes an existing virtual proc * diff --git a/cuebot/src/main/java/com/imageworks/spcue/dao/postgres/DispatcherDaoJdbc.java b/cuebot/src/main/java/com/imageworks/spcue/dao/postgres/DispatcherDaoJdbc.java index a7f9285f9a..6b1547b5b1 100644 --- a/cuebot/src/main/java/com/imageworks/spcue/dao/postgres/DispatcherDaoJdbc.java +++ b/cuebot/src/main/java/com/imageworks/spcue/dao/postgres/DispatcherDaoJdbc.java @@ -437,10 +437,14 @@ public List findNextDispatchFrames(LayerInterface layer, Dispatch layer.getLayerId(), limit); } else { + // Bind order must match the query: outer layer.pk_layer=?, then + // the tag subquery's h.str_name=?, then its l.pk_layer=?. The host + // name and the second layer id were transposed, so the tag + // subquery matched no host/layer and the query returned no frames. frames = getJdbcTemplate().query(FIND_DISPATCH_FRAME_BY_LAYER_AND_HOST, FrameDaoJdbc.DISPATCH_FRAME_MAPPER, host.idleCores, host.idleMemory, threadMode(host.threadMode), host.idleGpus, host.idleGpuMemory, - layer.getLayerId(), layer.getLayerId(), host.getName(), limit); + layer.getLayerId(), host.getName(), layer.getLayerId(), limit); } prometheusMetrics.setBookingDurationMetric("findNextDispatchFrames by layer and host query", diff --git a/cuebot/src/main/java/com/imageworks/spcue/dao/postgres/FrameDaoJdbc.java b/cuebot/src/main/java/com/imageworks/spcue/dao/postgres/FrameDaoJdbc.java index 418487458c..608df2c947 100644 --- a/cuebot/src/main/java/com/imageworks/spcue/dao/postgres/FrameDaoJdbc.java +++ b/cuebot/src/main/java/com/imageworks/spcue/dao/postgres/FrameDaoJdbc.java @@ -235,6 +235,108 @@ public void updateFrameStarted(VirtualProc proc, FrameInterface frame) { } } + /** + * Pre-acquire, in a deterministic global order, the layer_stat and job_stat + * counter rows that the batch's frame-start triggers will update. Locks all + * distinct layer_stat rows first (ordered by pk_layer), then all distinct + * job_stat rows (ordered by pk_job), the same "layer-before-job" order + * every single-frame transaction follows via the trigger, so this batch + * can never deadlock against a concurrent frame completion. SELECT ... FOR + * UPDATE inside the batch's transaction; rows are released at commit. + */ + private void lockStatRowsForBatch( + java.util.List bookings) { + java.util.SortedSet layerIds = new java.util.TreeSet<>(); + java.util.SortedSet jobIds = new java.util.TreeSet<>(); + for (com.imageworks.spcue.dispatcher.FrameBooking b : bookings) { + layerIds.add(b.frame.getLayerId()); + jobIds.add(b.frame.getJobId()); + } + if (!layerIds.isEmpty()) { + String in = String.join(",", java.util.Collections.nCopies(layerIds.size(), "?")); + getJdbcTemplate().query( + "SELECT pk_layer FROM layer_stat WHERE pk_layer IN (" + in + ") " + + "ORDER BY pk_layer FOR UPDATE", + rs -> {}, layerIds.toArray()); + } + if (!jobIds.isEmpty()) { + String in = String.join(",", java.util.Collections.nCopies(jobIds.size(), "?")); + getJdbcTemplate().query( + "SELECT pk_job FROM job_stat WHERE pk_job IN (" + in + ") " + + "ORDER BY pk_job FOR UPDATE", + rs -> {}, jobIds.toArray()); + } + } + + @Override + public boolean[] batchUpdateFramesStarted( + java.util.List bookings) { + + boolean[] won = new boolean[bookings.size()]; + if (bookings.isEmpty()) { + return won; + } + + // 0. Deadlock-free lock ordering. The frame-start UPDATE below fires + // trigger__update_frame_status_counts, which UPDATEs layer_stat then + // job_stat for each frame's layer/job, always layer first, then job. + // Every single-frame transaction (dispatch, completion, kill, ...) + // therefore acquires those counter rows in "layer-before-job" order. + // This multi-frame batch is the only writer that interleaves them: it + // would hold job_stat[J] (taken for an early frame) while still + // acquiring layer_stat rows for later frames of the same job, and a + // concurrent frame completion holding one of those layer_stat rows and + // reaching for job_stat[J] closes a deadlock cycle. + // + // Fix: pre-acquire every counter row this batch will touch, in the same + // global order the triggers use, all layer_stat rows (sorted), then + // all job_stat rows (sorted), so the batch and every single-frame + // transaction acquire on one total order and can never form a cycle. + // The trigger's later UPDATEs are then no-op re-locks on rows we hold. + lockStatRowsForBatch(bookings); + + // 1. Version+state-guarded RUNNING transition for every frame in one + // batch. Each row updates 0 (lost the race / limit hit) or 1 (won). + java.util.List startParams = new java.util.ArrayList<>(bookings.size()); + for (com.imageworks.spcue.dispatcher.FrameBooking b : bookings) { + VirtualProc proc = b.proc; + DispatchFrame frame = b.frame; + startParams.add(new Object[] {FrameState.RUNNING.toString(), proc.hostName, + proc.coresReserved, proc.memoryReserved, proc.gpusReserved, + proc.gpuMemoryReserved, frame.getFrameId(), FrameState.WAITING.toString(), + frame.getVersion()}); + } + int[] counts; + try { + counts = getJdbcTemplate().batchUpdate(UPDATE_FRAME_STARTED, startParams); + } catch (DataAccessException e) { + throw new FrameReservationException(e.getCause()); + } + + java.util.List retryParams = new java.util.ArrayList<>(); + for (int i = 0; i < bookings.size(); i++) { + // A JDBC batch may report SUCCESS_NO_INFO (-2); treat anything but an + // explicit 0 as a win, since the WHERE clause matches at most one row. + won[i] = counts[i] != 0; + if (won[i]) { + retryParams.add(new Object[] {bookings.get(i).frame.getFrameId(), -1, + FrameExitStatus.SKIP_RETRY_VALUE, FrameExitStatus.FAILED_LAUNCH_VALUE, + Dispatcher.EXIT_STATUS_FRAME_CLEARED, Dispatcher.EXIT_STATUS_FRAME_ORPHAN, + Dispatcher.EXIT_STATUS_FAILED_KILL, Dispatcher.EXIT_STATUS_DOWN_HOST}); + } + } + + // 2. Bump the retry counter for the winners, also batched. + if (!retryParams.isEmpty()) { + try { + getJdbcTemplate().batchUpdate(UPDATE_FRAME_RETRIES, retryParams); + } catch (DataAccessException e) { + throw new FrameReservationException(e.getCause()); + } + } + return won; + } + // spotless:off private static final String UPDATE_FRAME_FIXED = "UPDATE frame " diff --git a/cuebot/src/main/java/com/imageworks/spcue/dao/postgres/ProcDaoJdbc.java b/cuebot/src/main/java/com/imageworks/spcue/dao/postgres/ProcDaoJdbc.java index 3400fd5e46..27958a1004 100644 --- a/cuebot/src/main/java/com/imageworks/spcue/dao/postgres/ProcDaoJdbc.java +++ b/cuebot/src/main/java/com/imageworks/spcue/dao/postgres/ProcDaoJdbc.java @@ -20,9 +20,13 @@ import java.sql.ResultSet; import java.sql.SQLException; import java.util.ArrayList; +import java.util.Collections; import java.util.HashMap; +import java.util.HashSet; +import java.util.LinkedHashMap; import java.util.List; import java.util.Map; +import java.util.Set; import org.apache.logging.log4j.LogManager; import org.apache.logging.log4j.Logger; @@ -186,6 +190,105 @@ public void insertVirtualProc(VirtualProc proc) { } } + // Guarded per-host reservation: only decrements a host that currently has room + // for the whole aggregated share (the four ">= ?" guards). A host without room + // matches 0 rows instead of going negative, so the verify_host_resources + // trigger never fires and the batched tick is never aborted. + private static final String RESERVE_HOST_RESOURCES_GUARDED = "UPDATE host SET " + + "int_cores_idle = int_cores_idle - ?, int_mem_idle = int_mem_idle - ?, " + + "int_gpus_idle = int_gpus_idle - ?, int_gpu_mem_idle = int_gpu_mem_idle - ? " + + "WHERE pk_host = ? AND int_cores_idle >= ? AND int_mem_idle >= ? " + + "AND int_gpus_idle >= ? AND int_gpu_mem_idle >= ?"; + + // Release resources reserved for procs that were not booked after all (pure + // re-increment, so it can never drive idle negative). + private static final String REFUND_HOST_RESOURCES = "UPDATE host SET " + + "int_cores_idle = int_cores_idle + ?, int_mem_idle = int_mem_idle + ?, " + + "int_gpus_idle = int_gpus_idle + ?, int_gpu_mem_idle = int_gpu_mem_idle + ? " + + "WHERE pk_host = ?"; + + @Override + public void batchInsertVirtualProcs(List procs) { + if (procs == null || procs.isEmpty()) { + return; + } + long memReservedMin = + env.getRequiredProperty("dispatcher.memory.mem_reserved_min", Long.class); + long memGpuReservedMin = + env.getRequiredProperty("dispatcher.memory.mem_gpu_reserved_min", Long.class); + + List procRows = new ArrayList(procs.size()); + for (VirtualProc proc : procs) { + proc.id = SqlUtil.genKeyRandom(); + procRows.add(new Object[] {proc.getProcId(), proc.getHostId(), proc.getShowId(), + proc.getLayerId(), proc.getJobId(), proc.getFrameId(), proc.coresReserved, + proc.memoryReserved, proc.memoryReserved, memReservedMin, proc.gpusReserved, + proc.gpuMemoryReserved, proc.gpuMemoryReserved, memGpuReservedMin, + proc.isLocalDispatch}); + } + + // Host idle is reserved up-front by reserveHostResourcesBatch (a guarded, + // per-host decrement), so this only writes the proc rows. + getJdbcTemplate().batchUpdate(INSERT_VIRTUAL_PROC, procRows); + } + + @Override + public Set reserveHostResourcesBatch(List procs) { + if (procs == null || procs.isEmpty()) { + return Collections.emptySet(); + } + // Aggregate this tick's demand per host (a host commonly gets several procs). + Map hostDelta = new LinkedHashMap(); + for (VirtualProc proc : procs) { + long[] d = hostDelta.computeIfAbsent(proc.getHostId(), k -> new long[4]); + d[0] += proc.coresReserved; + d[1] += proc.memoryReserved; + d[2] += proc.gpusReserved; + d[3] += proc.gpuMemoryReserved; + } + List hostIds = new ArrayList(hostDelta.keySet()); + List rows = new ArrayList(hostIds.size()); + for (String hostId : hostIds) { + long[] d = hostDelta.get(hostId); + // SET deltas (4), pk_host, then the four ">= ?" guards (same deltas). + rows.add(new Object[] {d[0], d[1], d[2], d[3], hostId, d[0], d[1], d[2], d[3]}); + } + int[] updated = getJdbcTemplate().batchUpdate(RESERVE_HOST_RESOURCES_GUARDED, rows); + Set affordable = new HashSet(); + for (int i = 0; i < hostIds.size(); i++) { + // A JDBC batch may report SUCCESS_NO_INFO (-2); treat anything but an + // explicit 0 (the guard matched no row -> not enough idle) as reserved, + // matching FrameDaoJdbc's win test since each row keys on pk_host. A + // '> 0' test would drop a -2 host with its idle already decremented, + // and refundHostResourcesBatch only refunds race losers, so it would leak. + if (updated[i] != 0) { + affordable.add(hostIds.get(i)); + } + } + return affordable; + } + + @Override + public void refundHostResourcesBatch(List procs) { + if (procs == null || procs.isEmpty()) { + return; + } + Map hostDelta = new LinkedHashMap(); + for (VirtualProc proc : procs) { + long[] d = hostDelta.computeIfAbsent(proc.getHostId(), k -> new long[4]); + d[0] += proc.coresReserved; + d[1] += proc.memoryReserved; + d[2] += proc.gpusReserved; + d[3] += proc.gpuMemoryReserved; + } + List rows = new ArrayList(hostDelta.size()); + for (Map.Entry e : hostDelta.entrySet()) { + long[] d = e.getValue(); + rows.add(new Object[] {d[0], d[1], d[2], d[3], e.getKey()}); + } + getJdbcTemplate().batchUpdate(REFUND_HOST_RESOURCES, rows); + } + // spotless:off private static final String UPDATE_VIRTUAL_PROC_ASSIGN = "UPDATE " diff --git a/cuebot/src/main/java/com/imageworks/spcue/dispatcher/CoreUnitDispatcher.java b/cuebot/src/main/java/com/imageworks/spcue/dispatcher/CoreUnitDispatcher.java index 7b3cd911fa..2e723ff12c 100644 --- a/cuebot/src/main/java/com/imageworks/spcue/dispatcher/CoreUnitDispatcher.java +++ b/cuebot/src/main/java/com/imageworks/spcue/dispatcher/CoreUnitDispatcher.java @@ -77,6 +77,7 @@ public class CoreUnitDispatcher implements Dispatcher { private static final Logger logger = LogManager.getLogger(CoreUnitDispatcher.class); + private DispatchSupport dispatchSupport; private JobManager jobManager; @@ -345,7 +346,89 @@ public void wrapDispatchFrame() { @Override public List dispatchHost(DispatchHost host, LayerInterface layer) { - throw new RuntimeException("not implemented)"); + + // Layer-exact dispatch for the Scheduler: book frames of the specific + // layer it scored and reserved this host for, not the whole job. This + // mirrors dispatchHost(host, job) but scopes the frame query to the + // layer; a LayerInterface is also a JobInterface, so the existing + // burst / job-bookable checks and dispatch machinery work unchanged. + // Kept as a self-contained method so the legacy dispatchHost(host, job) + // path is left exactly as-is. + List procs = new ArrayList(); + + if (host.strandedCores == 0 && dispatchSupport.isShowAtOrOverBurst(layer, host)) { + return procs; + } + + List frames = dispatchSupport.findNextDispatchFrames(layer, host, + getIntProperty("dispatcher.frame_query_max")); + + logger.info("Frames found: " + frames.size() + " for host " + host.getName() + " " + + host.idleCores + "/" + host.idleMemory + " on layer " + layer.getName()); + + String[] selfishServices = + env.getProperty("dispatcher.frame.selfish.services", "").split(","); + for (DispatchFrame frame : frames) { + + VirtualProc proc = VirtualProc.build(host, frame, selfishServices); + + if (frame.minCores <= 0 && !proc.canHandleNegativeCoresRequest) { + logger.debug("Cannot dispatch layer, host is busy."); + break; + } + + if (host.idleCores < host.handleNegativeCoresRequirement(frame.minCores) + || host.idleMemory < frame.getMinMemory() || host.idleGpus < frame.minGpus + || host.idleGpuMemory < frame.minGpuMemory) { + logger.debug("Cannot dispatch, insufficient resources."); + break; + } + + if (!dispatchSupport.isJobBookable(layer, proc.coresReserved, proc.gpusReserved)) { + break; + } + + if (host.strandedCores == 0 && dispatchSupport.isShowAtOrOverBurst(layer, host)) { + return procs; + } + + boolean success = new DispatchFrameTemplate(proc, layer, frame, false) { + public void wrapDispatchFrame() { + logger.debug("Dispatching frame with " + frame.minCores + + " minCores on proc with " + proc.coresReserved + " coresReserved"); + dispatch(frame, proc); + dispatchSummary(proc, frame, "Booking"); + return; + } + }.execute(); + + if (success) { + procs.add(proc); + + DispatchSupport.bookedProcs.getAndIncrement(); + DispatchSupport.bookedCores.addAndGet(proc.coresReserved); + DispatchSupport.bookedGpus.addAndGet(proc.gpusReserved); + + if (host.strandedCores > 0) { + dispatchSupport.pickupStrandedCores(host); + break; + } + + host.useResources(proc.coresReserved, proc.memoryReserved, proc.gpusReserved, + proc.gpuMemoryReserved); + if (!host.hasAdditionalResources(Dispatcher.CORE_POINTS_RESERVED_MIN, + MEM_RESERVED_MIN, Dispatcher.GPU_UNITS_RESERVED_MIN, + MEM_GPU_RESERVED_MIN)) { + break; + } else if (procs.size() >= getIntProperty("dispatcher.job_frame_dispatch_max")) { + break; + } else if (procs.size() >= getIntProperty("dispatcher.host_frame_dispatch_max")) { + break; + } + } + } + + return procs; } @Override @@ -353,6 +436,71 @@ public List dispatchHost(DispatchHost host, FrameInterface frame) { throw new RuntimeException("not implemented)"); } + @Override + public List planHost(DispatchHost host, LayerInterface layer) { + // Scheduler-native lean read. The planner already loaded this host and + // already enforced show-burst and job caps in-tick, so we skip the + // per-frame isShowAtOrOverBurst / isJobBookable DB round-trips the + // legacy dispatchHost makes (~15 per placement). One candidate query, + // then build procs and apply the in-memory resource fit checks; the + // Scheduler commits the bookings in bulk. No writes, no RQD launch. + List bookings = new ArrayList(); + + List frames = dispatchSupport.findNextDispatchFrames(layer, host, + getIntProperty("dispatcher.frame_query_max")); + + String[] selfishServices = + env.getProperty("dispatcher.frame.selfish.services", "").split(","); + for (DispatchFrame frame : frames) { + + VirtualProc proc; + try { + // expandThreadable=false: reserve exactly the requested cores. The + // planner scored this placement and decremented its snapshot by the + // frame's cores, so the thread-mode grab-idle expansion would + // over-reserve and corrupt that accounting. The planner fills hosts + // by planning multiple placements, not by one frame ballooning. + proc = VirtualProc.build(host, frame, false, selfishServices); + } catch (RuntimeException e) { + // build() can still throw on edge cases; stop planning this host + // and let a later tick retry. + break; + } + + if (frame.minCores <= 0 && !proc.canHandleNegativeCoresRequest) { + break; + } + + if (host.idleCores < host.handleNegativeCoresRequirement(frame.minCores) + || host.idleMemory < frame.getMinMemory() || host.idleGpus < frame.minGpus + || host.idleGpuMemory < frame.minGpuMemory) { + break; + } + + bookings.add(new FrameBooking(frame, proc)); + DispatchSupport.bookedProcs.getAndIncrement(); + DispatchSupport.bookedCores.addAndGet(proc.coresReserved); + DispatchSupport.bookedGpus.addAndGet(proc.gpusReserved); + + if (host.strandedCores > 0) { + dispatchSupport.pickupStrandedCores(host); + break; + } + + host.useResources(proc.coresReserved, proc.memoryReserved, proc.gpusReserved, + proc.gpuMemoryReserved); + if (!host.hasAdditionalResources(Dispatcher.CORE_POINTS_RESERVED_MIN, MEM_RESERVED_MIN, + Dispatcher.GPU_UNITS_RESERVED_MIN, MEM_GPU_RESERVED_MIN)) { + break; + } else if (bookings.size() >= getIntProperty("dispatcher.job_frame_dispatch_max")) { + break; + } else if (bookings.size() >= getIntProperty("dispatcher.host_frame_dispatch_max")) { + break; + } + } + return bookings; + } + @Override public void dispatch(DispatchFrame frame, VirtualProc proc) { // Allocate frame on the database diff --git a/cuebot/src/main/java/com/imageworks/spcue/dispatcher/DispatchSupport.java b/cuebot/src/main/java/com/imageworks/spcue/dispatcher/DispatchSupport.java index 88ca36ace4..c0b5678062 100644 --- a/cuebot/src/main/java/com/imageworks/spcue/dispatcher/DispatchSupport.java +++ b/cuebot/src/main/java/com/imageworks/spcue/dispatcher/DispatchSupport.java @@ -181,6 +181,20 @@ public interface DispatchSupport { */ public void startFrameAndProc(VirtualProc proc, DispatchFrame frame); + /** + * Batch variant of {@link #startFrameAndProc}: commits many planned bookings + * in one transaction with batched statements, version-guarded frame + * RUNNING transition, proc INSERT, and host idle decrement, instead of one + * transaction and ~6 round-trips per frame. The subscription/layer/job/ + * folder/point counters are NOT written here; the Scheduler batches those + * separately. Frames that lost their optimistic version race are dropped. + * + * @param bookings the planned (frame, proc) pairs from the planning phase + * @return the subset of bookings that were actually committed (winners) + */ + public java.util.List startFramesAndProcsBatch( + java.util.List bookings); + /** * This method clears out a proc that was lost track of. This can happen if the host fails and * the proc fails to report in, a network outage occurs, or something of that nature. diff --git a/cuebot/src/main/java/com/imageworks/spcue/dispatcher/DispatchSupportService.java b/cuebot/src/main/java/com/imageworks/spcue/dispatcher/DispatchSupportService.java index 75d9f43c81..42aee9724a 100644 --- a/cuebot/src/main/java/com/imageworks/spcue/dispatcher/DispatchSupportService.java +++ b/cuebot/src/main/java/com/imageworks/spcue/dispatcher/DispatchSupportService.java @@ -15,6 +15,7 @@ package com.imageworks.spcue.dispatcher; +import java.util.ArrayList; import java.util.List; import java.util.Set; import java.util.concurrent.ConcurrentHashMap; @@ -239,6 +240,95 @@ public void startFrameAndProc(VirtualProc proc, DispatchFrame frame) { publishFrameStartedEvent(frame, proc, previousState); } + @Override + @Transactional(propagation = Propagation.REQUIRED) + public List startFramesAndProcsBatch(List bookings) { + if (bookings == null || bookings.isEmpty()) { + return java.util.Collections.emptyList(); + } + + // 1. CAPACITY GATE (before any frame is marked RUNNING): atomically reserve + // each host's aggregated share with a guarded decrement. A host that cannot + // currently hold its share returns 0 rows and is left untouched, so we never + // overcommit a host -- an unguarded decrement would drive idle negative, + // trip the verify_host_resources trigger, and abort this whole batched tick. + // Bookings on a host without room are deferred (stay WAITING) and re-planned + // next tick against fresh idle counts. + List demanded = new ArrayList(bookings.size()); + for (FrameBooking b : bookings) { + // Apply any per-frame OOM memory bump BEFORE the capacity gate, so the host + // reservation and the proc agree on the bumped amount (else the host would be + // under-reserved for RAM). Outlier frames climb here without touching the layer. + long bump = OomMemoryTracker.INSTANCE.frameBumpKb(b.frame.getFrameId()); + if (bump > b.proc.memoryReserved) { + b.proc.memoryReserved = bump; + } + demanded.add(b.proc); + } + Set affordableHosts = procDao.reserveHostResourcesBatch(demanded); + + List affordable = new ArrayList(bookings.size()); + for (FrameBooking b : bookings) { + if (affordableHosts.contains(b.proc.getHostId())) { + affordable.add(b); + } + } + if (affordable.isEmpty()) { + return java.util.Collections.emptyList(); + } + + // 2. Version-guarded RUNNING transition for the affordable bookings. The + // returned mask tells us which frames we actually won. + boolean[] won = frameDao.batchUpdateFramesStarted(affordable); + + List winners = new ArrayList(affordable.size()); + List winnerProcs = new ArrayList(affordable.size()); + List raceLosers = new ArrayList(); + for (int i = 0; i < affordable.size(); i++) { + FrameBooking b = affordable.get(i); + if (won[i]) { + // Stamp the frame linkage onto the proc before the batch insert. + // The inline path does this in reserveProc(); the planning path + // (CoreUnitDispatcher.planHost) builds the proc but never sets it, + // so without this every batch-inserted proc lands with + // pk_frame=NULL, backing no frame while holding its host cores + // (the frame shows RUNNING but its cores are stranded). + b.proc.frameId = b.frame.getFrameId(); + b.proc.jobId = b.frame.getJobId(); + b.proc.layerId = b.frame.getLayerId(); + b.proc.showId = b.frame.getShowId(); + winners.add(b); + winnerProcs.add(b.proc); + } else { + // Capacity was reserved in step 1 but the frame was lost to the + // version race (rare): give that host's reservation back so its + // idle count is not leaked. + raceLosers.add(b.proc); + } + } + + // 3. Refund hosts for the rare race-losers (re-increment only -> never + // negative, never trips the trigger). + if (!raceLosers.isEmpty()) { + procDao.refundHostResourcesBatch(raceLosers); + } + + if (winnerProcs.isEmpty()) { + return winners; + } + + // 4. Insert the winner procs. Host idle was already decremented in step 1, + // so this only writes the proc rows. The subscription/layer/job/folder/ + // point counters are batched by the Scheduler from the winners returned here. + procDao.batchInsertVirtualProcs(winnerProcs); + + // 5. Publish FRAME_STARTED events (WAITING -> RUNNING). + for (FrameBooking b : winners) { + publishFrameStartedEvent(b.frame, b.proc, FrameState.WAITING); + } + return winners; + } + @Transactional(propagation = Propagation.REQUIRED, readOnly = true) public boolean isCueBookable(FacilityInterface f) { return jobDao.cueHasPendingJobs(f); diff --git a/cuebot/src/main/java/com/imageworks/spcue/dispatcher/Dispatcher.java b/cuebot/src/main/java/com/imageworks/spcue/dispatcher/Dispatcher.java index 634502625c..217231442c 100644 --- a/cuebot/src/main/java/com/imageworks/spcue/dispatcher/Dispatcher.java +++ b/cuebot/src/main/java/com/imageworks/spcue/dispatcher/Dispatcher.java @@ -32,6 +32,12 @@ public interface Dispatcher { // Maximum number of core points that can be assigned to a frame public static final int CORE_POINTS_RESERVED_MAX = 2400; + // Maximum core points per frame when the in-process Scheduler is enabled. + // The Scheduler places whole-host-sized layers (it tracks contiguous idle + // cores per host and reserves/drains hosts for wide jobs), so it can honor + // much wider per-frame reservations than the legacy dispatcher. + public static final int CORE_POINTS_RESERVED_MAX_NEW = 6400; + // The default number of core points assigned to a frame, if no core // point value is specified public static final int CORE_POINTS_RESERVED_DEFAULT = 100; @@ -160,6 +166,20 @@ public interface Dispatcher { */ List dispatchHost(DispatchHost host, LayerInterface layer); + /** + * Plan (but do not commit) the frames that would be booked for a layer on a + * host. Runs the same placement logic as + * {@link #dispatchHost(DispatchHost, LayerInterface)}, candidate query, + * fit checks, in-memory host resource decrement, but instead of writing + * each booking, collects the planned (frame, proc) pairs for the Scheduler + * to commit in bulk. No DB writes and no RQD launch happen here. + * + * @param host + * @param layer + * @return the planned bookings, in placement order. + */ + List planHost(DispatchHost host, LayerInterface layer); + /** * Dispatch a host to the specified job. * diff --git a/cuebot/src/main/java/com/imageworks/spcue/dispatcher/FrameBooking.java b/cuebot/src/main/java/com/imageworks/spcue/dispatcher/FrameBooking.java new file mode 100644 index 0000000000..ed39cdcb5d --- /dev/null +++ b/cuebot/src/main/java/com/imageworks/spcue/dispatcher/FrameBooking.java @@ -0,0 +1,35 @@ + +/* + * Copyright Contributors to the OpenCue Project + * + * Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except + * in compliance with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software distributed under the License + * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express + * or implied. See the License for the specific language governing permissions and limitations under + * the License. + */ + +package com.imageworks.spcue.dispatcher; + +import com.imageworks.spcue.DispatchFrame; +import com.imageworks.spcue.VirtualProc; + +/** + * A planned (frame, proc) pairing produced by the Scheduler's planning phase + * and committed in bulk by the batch booking path. Carries both the frame + * (needed for the version-guarded frame UPDATE and the RQD launch) and the + * proc built for it (the INSERT row plus host/accounting deltas). + */ +public final class FrameBooking { + public final DispatchFrame frame; + public final VirtualProc proc; + + public FrameBooking(DispatchFrame frame, VirtualProc proc) { + this.frame = frame; + this.proc = proc; + } +} diff --git a/cuebot/src/main/java/com/imageworks/spcue/dispatcher/FrameCompleteHandler.java b/cuebot/src/main/java/com/imageworks/spcue/dispatcher/FrameCompleteHandler.java index ce745f6186..f2c3ac8910 100644 --- a/cuebot/src/main/java/com/imageworks/spcue/dispatcher/FrameCompleteHandler.java +++ b/cuebot/src/main/java/com/imageworks/spcue/dispatcher/FrameCompleteHandler.java @@ -175,7 +175,23 @@ public void handleFrameCompleteReport(final FrameCompleteReport report) { if (dispatchSupport.stopFrame(frame, newFrameState, exitStatus, report.getFrame().getMaxRss())) { - if (dispatcher.isTestMode()) { + // In-process Scheduler only (scheduler.enabled): cuebot plans in-process and never + // same-host-rebooks -- the Scheduler rebooks fresh next tick (the reactive + // DispatchNextFrame path is gone, now a locality score bonus) -- so on + // frame-complete the proc is ALWAYS released. The legacy async hop to + // dispatchQueue exists only to defer the rebook-or-release decision off the + // report thread; with no rebook to decide it just leaves the proc in limbo + // (pk_frame NULL, cores still held) until the task runs -- a standing backlog + // under load, and a stranded zombie if the task is ever lost. Run it inline + // (exactly as test mode does) so the proc is reaped within the completion + // itself: no limbo. The legacy dispatcher and Rust (dispatcher.turn_off_booking) are + // deliberately left on the async path here -- out of scope for this change. + // Inline post-complete only for shows the in-process Scheduler owns + // (facility, or a 'managed' show); legacy shows and Rust + // (dispatcher.turn_off_booking) stay on the async path. + boolean schedulerOwnsShow = + SchedulerMode.schedules(env, showDao, proc.getShowId()); + if (dispatcher.isTestMode() || schedulerOwnsShow) { // Database modifications on a threadpool cannot be captured by the test thread handlePostFrameCompleteOperations(proc, report, job, frame, newFrameState, frameDetail); @@ -279,6 +295,17 @@ public void handlePostFrameCompleteOperations(VirtualProc proc, FrameCompleteRep */ boolean unbookProc = proc.unbooked; + /* + * When booking is disabled, or the new Scheduler is enabled, suppress the legacy + * per-host BookingQueue enqueues below so the two booking paths never both run. The + * Scheduler reaches this host on its own tick. This mirrors the guard in + * HostReportHandler and prevents legacy booking threads from racing the Scheduler's + * batched commit for the same frames. + */ + boolean bookingOff = + env.getProperty("dispatcher.turn_off_booking", Boolean.class, false) + || SchedulerMode.facility(env); + dispatchSupport.updateUsageCounters(frame, report.getExitStatus()); boolean isLayerComplete = false; @@ -327,6 +354,12 @@ public void handlePostFrameCompleteOperations(VirtualProc proc, FrameCompleteRep */ jobManager.optimizeLayer(frame, report.getFrame().getNumCores(), report.getFrame().getMaxRss(), report.getRunTime()); + if (SchedulerMode.enabled(env)) { + // With the in-process Scheduler, a success means the layer is not + // systematically under-sized now, so reset its OOM streak and this + // frame's bump. + OomMemoryTracker.INSTANCE.onSuccess(frame.getFrameId(), frame.getLayerId()); + } } /* @@ -344,8 +377,12 @@ public void handlePostFrameCompleteOperations(VirtualProc proc, FrameCompleteRep /* * Some exit statuses indicate that a frame was killed by the application due to a - * memory issue and should be retried. In this case, disable the optimizer and raise the - * memory by what is specified in the show's service override, service or 2GB. + * memory issue and should be retried, by raising the memory (service override, + * service, or 2GB). The legacy dispatcher raises the whole LAYER and disables its + * optimizer -- the original behavior, kept unchanged. The in-process Scheduler + * instead bumps per FRAME so one hungry or spuriously-killed frame does not + * inflate every other frame and strand cores, escalating to the layer only after + * repeated OOMs in a row (see OomMemoryTracker). */ if (report.getExitStatus() == Dispatcher.EXIT_STATUS_MEMORY_FAILURE || report.getExitSignal() == Dispatcher.EXIT_STATUS_MEMORY_FAILURE @@ -377,9 +414,29 @@ public void handlePostFrameCompleteOperations(VirtualProc proc, FrameCompleteRep } unbookProc = true; - jobManager.enableMemoryOptimizer(frame, false); - jobManager.increaseLayerMemoryRequirement(frame, proc.memoryReserved + increase); - logger.info("Increased mem usage to: " + (proc.memoryReserved + increase)); + long newReserved = proc.memoryReserved + increase; + if (SchedulerMode.enabled(env)) { + // In-process Scheduler: bump per FRAME, escalate to the layer only after + // it OOMs oom_layer_escalate_threshold times in a row. Leaves the layer + // optimizer on, so an escalated layer later settles at its true size. + int oomThreshold = env.getProperty( + "dispatcher.oom_layer_escalate_threshold", Integer.class, 3); + if (OomMemoryTracker.INSTANCE.onOom(frame.getFrameId(), frame.getLayerId(), + newReserved, oomThreshold)) { + jobManager.increaseLayerMemoryRequirement(frame, newReserved); + logger.info("Layer " + frame.getLayerId() + " OOMed " + oomThreshold + + "x in a row; raised layer mem to: " + newReserved); + } else { + logger.info("Frame " + frame.getFrameId() + + " OOM; per-frame mem bump to: " + newReserved); + } + } else { + // Legacy dispatcher: original behavior, unchanged -- disable the layer + // optimizer and raise the whole layer. + jobManager.enableMemoryOptimizer(frame, false); + jobManager.increaseLayerMemoryRequirement(frame, newReserved); + logger.info("Increased mem usage to: " + newReserved); + } } /* @@ -469,7 +526,7 @@ else if (report.getHost().getNimbyLocked()) { * fractional can cause storms of booking requests that don't have a chance of * finding a suitable frame to run. */ - if (!proc.isLocalDispatch && proc.coresReserved >= 100 + if (!bookingOff && !proc.isLocalDispatch && proc.coresReserved >= 100 && dispatchSupport.isCueBookable(job)) { bookingQueue.execute(new DispatchBookHost( @@ -489,7 +546,8 @@ else if (report.getHost().getNimbyLocked()) { * This will handle show balancing in the future. */ - if (!proc.isLocalDispatch && randomNumber.nextInt(100) <= Dispatcher.UNBOOK_FREQUENCY + if (!bookingOff && !proc.isLocalDispatch + && randomNumber.nextInt(100) <= Dispatcher.UNBOOK_FREQUENCY && System.currentTimeMillis() > lastUnbook.get()) { // First make sure all jobs have their min cores @@ -543,7 +601,7 @@ else if (report.getHost().getNimbyLocked()) { /* * Check for stranded cores on the host. */ - if (!proc.isLocalDispatch && dispatchSupport.hasStrandedCores(proc) + if (!bookingOff && !proc.isLocalDispatch && dispatchSupport.hasStrandedCores(proc) && jobManager.isLayerThreadable(frame) && dispatchSupport.isJobBookable(job)) { @@ -558,11 +616,22 @@ else if (report.getHost().getNimbyLocked()) { } } - // Book the next frame of this job on the same proc + // Book the next frame of this job on the same proc. + // + // With the in-process Scheduler the reactive booking path is off: + // rebooking here would race the Scheduler's batched commit and + // dispatch outside the single batched commit path (the source + // of the inline-path deadlock). Instead, unbook the proc so its + // cores return to the host's idle pool; the Scheduler rebooks on + // its next tick, preferring the same host via a locality score + // bonus (see Scheduler placement). stopFrame only nulled the + // proc's pk_frame, so without this the reserved cores would leak. if (proc.isLocalDispatch) { dispatchQueue.execute(new DispatchNextFrame(job, proc, localDispatcher)); - } else { + } else if (!bookingOff) { dispatchQueue.execute(new DispatchNextFrame(job, proc, dispatcher)); + } else { + dispatchSupport.unbookProc(proc); } } else { dispatchSupport.unbookProc(proc, "frame state was " + newFrameState.toString()); diff --git a/cuebot/src/main/java/com/imageworks/spcue/dispatcher/HostReportHandler.java b/cuebot/src/main/java/com/imageworks/spcue/dispatcher/HostReportHandler.java index 6277b5de34..daacb53016 100644 --- a/cuebot/src/main/java/com/imageworks/spcue/dispatcher/HostReportHandler.java +++ b/cuebot/src/main/java/com/imageworks/spcue/dispatcher/HostReportHandler.java @@ -318,8 +318,14 @@ public void handleHostReport(HostReport report, boolean isBoot) { msg = "The cue has no pending jobs"; } + // When the Scheduler owns the whole facility it owns dispatch: + // suppress the legacy per-host BookingQueue enqueue so the two paths + // never both run. In 'managed' (per-show) mode the legacy dispatcher + // still runs for non-managed shows -- its query already excludes + // b_scheduler_managed shows -- so we do NOT suppress it globally there. boolean bookingOff = - env.getProperty("dispatcher.turn_off_booking", Boolean.class, false); + env.getProperty("dispatcher.turn_off_booking", Boolean.class, false) + || SchedulerMode.facility(env); /* * If a message was set, the host is not bookable. Log the message and move on. */ diff --git a/cuebot/src/main/java/com/imageworks/spcue/dispatcher/LocalDispatcher.java b/cuebot/src/main/java/com/imageworks/spcue/dispatcher/LocalDispatcher.java index d443cb941d..9b1c19cebc 100644 --- a/cuebot/src/main/java/com/imageworks/spcue/dispatcher/LocalDispatcher.java +++ b/cuebot/src/main/java/com/imageworks/spcue/dispatcher/LocalDispatcher.java @@ -244,6 +244,14 @@ private List dispatchHost(DispatchHost host, LayerInterface layer, return procs; } + @Override + public List planHost(DispatchHost host, LayerInterface layer) { + // The plan/batch-commit path is used only by the in-process Scheduler + // via CoreUnitDispatcher; local dispatch always commits inline. + throw new UnsupportedOperationException( + "planHost is not supported for local dispatch"); + } + @Override public List dispatchHost(DispatchHost host, LayerInterface layer) { diff --git a/cuebot/src/main/java/com/imageworks/spcue/dispatcher/OomMemoryTracker.java b/cuebot/src/main/java/com/imageworks/spcue/dispatcher/OomMemoryTracker.java new file mode 100644 index 0000000000..85cdf0db10 --- /dev/null +++ b/cuebot/src/main/java/com/imageworks/spcue/dispatcher/OomMemoryTracker.java @@ -0,0 +1,86 @@ +package com.imageworks.spcue.dispatcher; + +import java.util.concurrent.TimeUnit; + +import com.google.common.cache.Cache; +import com.google.common.cache.CacheBuilder; + +/** + * Transient, per-JVM tracker that keeps the OOM memory ratchet off the layer. + * + *

The legacy behaviour raised the whole layer's memory (+2GB) on every OOM and + * disabled the layer optimizer, so a single hungry or spuriously-killed frame + * inflated every other frame's reservation and stranded cores, permanently + * (reservations could only ever climb). This tracker splits the two cases: + * + *

    + *
  • Outlier (one frame OOMs): bump the memory for just THAT frame + * ({@link #frameBumpKb}), leave the layer alone. The hungry frame climbs on its + * own; nothing else over-reserves; the bump is transient.
    • + *
  • Systematic (a layer OOMs {@code threshold} times in a row, with no + * success in between): the layer really is under-sized, so the caller raises the + * whole layer once. Any success of the layer resets the streak, so scattered or + * spurious OOMs never reach the threshold and never ratchet the layer.
    • + *
+ * + *

Everything is in-JVM and bounded, so it is ephemeral (a restart forgets it) + * and cannot grow without bound. No optimizer disable is needed: escalation is a + * real pattern, so the optimizer settles the raised layer at its true size. + */ +public final class OomMemoryTracker { + + public static final OomMemoryTracker INSTANCE = new OomMemoryTracker(); + + /** pk_frame -> bumped reserved memory (kB) for that frame's next booking. */ + private final Cache frameBump = CacheBuilder.newBuilder() + .maximumSize(200_000) + .expireAfterAccess(1, TimeUnit.HOURS) + .build(); + + /** pk_layer -> consecutive OOM count, reset on any success of the layer. */ + private final Cache layerStreak = CacheBuilder.newBuilder() + .maximumSize(100_000) + .expireAfterAccess(1, TimeUnit.HOURS) + .build(); + + private OomMemoryTracker() {} + + /** Reserved-memory bump (kB) recorded for this frame, or 0 if none. */ + public long frameBumpKb(String frameId) { + Long v = frameBump.getIfPresent(frameId); + return v == null ? 0L : v; + } + + /** + * Record an OOM. Returns true if the LAYER should be raised (it has OOMed + * {@code threshold} times in a row, so it is systematically under-sized); + * false if the OOM was handled per-frame (the outlier path). + */ + public boolean onOom(String frameId, String layerId, long newReservedKb, int threshold) { + // Atomic read-modify-write: several report threads can complete OOM frames + // of the SAME layer at once, and a getIfPresent/+1/put would lose + // increments (two OOMs counted as one). asMap() is a ConcurrentMap, so + // merge() increments the streak atomically. + int n = layerStreak.asMap().merge(layerId, 1, Integer::sum); + if (n >= threshold) { + // Escalate exactly once at the boundary: only the thread that removes + // the at-threshold mapping raises the layer; a racing thread's remove + // fails (value already changed/reset) and it falls through to the + // per-frame outlier path instead of raising the layer a second time. + if (layerStreak.asMap().remove(layerId, n)) { + return true; + } + } + frameBump.put(frameId, newReservedKb); // outlier: bump just this frame + return false; + } + + /** + * A frame of the layer succeeded: the layer is not systematically broken right + * now, so forget the streak and this frame's bump. + */ + public void onSuccess(String frameId, String layerId) { + layerStreak.invalidate(layerId); + frameBump.invalidate(frameId); + } +} diff --git a/cuebot/src/main/java/com/imageworks/spcue/dispatcher/Scheduler.java b/cuebot/src/main/java/com/imageworks/spcue/dispatcher/Scheduler.java new file mode 100644 index 0000000000..da39441e8e --- /dev/null +++ b/cuebot/src/main/java/com/imageworks/spcue/dispatcher/Scheduler.java @@ -0,0 +1,2331 @@ + +/* + * Copyright Contributors to the OpenCue Project + * + * Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except + * in compliance with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software distributed under the License + * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express + * or implied. See the License for the specific language governing permissions and limitations under + * the License. + */ + +package com.imageworks.spcue.dispatcher; + +import java.sql.Connection; +import java.sql.PreparedStatement; +import java.sql.ResultSet; +import java.sql.SQLException; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Comparator; +import java.util.HashMap; +import java.util.HashSet; +import java.util.Iterator; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; +import java.util.Objects; +import java.util.Set; +import java.util.concurrent.BlockingQueue; +import java.util.concurrent.Callable; +import java.util.concurrent.ConcurrentHashMap; +import java.util.concurrent.ExecutionException; +import java.util.concurrent.ExecutorService; +import java.util.concurrent.Executors; +import java.util.concurrent.Future; +import java.util.concurrent.LinkedBlockingQueue; +import java.util.concurrent.ThreadPoolExecutor; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicBoolean; +import java.util.stream.Collectors; + +import org.apache.logging.log4j.LogManager; +import org.apache.logging.log4j.Logger; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.core.env.Environment; +import org.springframework.jdbc.core.RowMapper; +import org.springframework.jdbc.core.support.JdbcDaoSupport; +import org.springframework.transaction.PlatformTransactionManager; +import org.springframework.transaction.support.TransactionTemplate; + +import com.imageworks.spcue.DispatchHost; +import com.imageworks.spcue.LayerInterface; +import com.imageworks.spcue.VirtualProc; +import com.imageworks.spcue.service.HostManager; +import com.imageworks.spcue.rqd.RqdClient; +import com.imageworks.spcue.service.JobManager; + +/** + * Scheduler: single-threaded planner with persistent reservations, parallel + * per-host plan reads, and a single batched commit. + * + * Replaces the multi-threaded BookingQueue path. Each tick: + * + * 1. Acquire a Postgres advisory lock so only one Cuebot plans at a time. + * 2. Read bookable hosts. + * 3. Group them by static spec (alloc, normalized tags, os, has_gpu). + * 4. For each group: + * - one candidate-layer query + * - for each candidate in priority order, run the dispatch loop: score + * every fitting host (respecting reservations, overriding a + * lower-priority reservation on successful dispatch), record each + * (host, layer) pairing to plan, and decrement in-memory accounting + * from an estimate. A layer that stays blocked long enough and is wide + * enough records a reservation REQUEST. + * 5. Grant reservations: process the requests highest-priority-then-widest, + * reconciling each grantee's reservation count under the per-class and + * max-grantees caps (EASY/Maui backfill). + * 6. PLAN the recorded pairings in parallel (by host), commit them in one + * batched transaction, then fire the RQD launches fire-and-forget. + * 7. Sweep reservations whose layer no longer appears in any candidate set. + * 8. Release the lock. + * + * Planning stays single-threaded so decisions never race on shared state. The + * only parallelism is on the plan-phase reads: once placements are chosen, the + * per-host frame selection (planHost) runs concurrently on a read pool. The + * writes are not fanned out: every booking for the tick lands in one batched + * transaction. Frames lost to a frame.int_version race are dropped from the + * batch (they stay WAITING and are retried next tick); the RQD launches fire + * afterward fire-and-forget so a slow RQD never stalls the tick. + * + * Reservation invariant: a host's reservation belongs to the highest-priority + * layer that has claimed it. A reservation persists across ticks until the + * owning layer's pending unfittable frames reach zero, the layer leaves the + * dispatchable set, or a higher-priority layer overrides the claim. This + * prevents blocked-layer starvation: any layer that doesn't fit anywhere + * claims hosts so they aren't re-consumed by lower-priority work between + * the moment the layer becomes blocked and the moment hosts free up + * enough cores. + * + * Gated by scheduler.enabled (default false). When true, HostReportHandler + * suppresses the legacy BookingQueue enqueue via the existing booking-off + * branch, so the two paths never both run. + */ +public class Scheduler extends JdbcDaoSupport { + + private static final Logger logger = LogManager.getLogger(Scheduler.class); + + /** + * Postgres advisory lock key. Must be the same constant on every Cuebot + * that shares a database. Arbitrary 64-bit integer; chosen as ASCII + * "OpenCue" for visibility in pg_locks. + */ + private static final long SCHEDULER_LOCK_KEY = 0x4F70656E437565L; + + // ---- placementScore: E-PVM dimension weights -------------------------- + // + // placementScore sums a convex per-dimension cost whose argument is the + // dimensionless utilization fraction used_D/total_D (see the method's + // Javadoc), so no per-dimension unit conversion is needed, the weights + // below set only the RELATIVE importance of the dimensions on that common + // scale. Cores and memory contribute equally; GPUs are weighted higher so + // a GPU layer prefers the host where it strands the least GPU capacity. + private static final double W_CORES = 1.0; + private static final double W_MEM = 1.0; + private static final double W_GPUS = 4.0; + private static final double W_GPU_MEM = 1.0; + + // Locality bonus. The reactive path used to rebook the next frame of a job + // on the same proc the moment a frame finished (same-machine locality). + // Under the Scheduler that path is gone, a completing proc is unbooked and its cores + // return to the host. To preserve locality, placement subtracts this bonus + // from a host's score when the host ALREADY runs the candidate's layer, so + // the freed core is preferentially refilled by the same layer next tick. A + // multi-frame layer keeps >=1 proc on its host between ticks, so the signal + // persists without tracking individual completions. Sized to dominate the + // marginal stranding cost (deltaCost terms are ~e^util, single digits) but + // bounded; reservations and fit are filtered BEFORE scoring, so the bonus + // can never override them. + private volatile double localityBonus = 8.0; + private volatile boolean localityEnabled = true; + + @Autowired + private Environment env; + + @Autowired + private PlatformTransactionManager transactionManager; + + /** Built lazily from the injected manager; wraps each resource-delta flush so + * its several UPDATEs commit atomically (partial commit + retry would + * double-apply the sub-batches that had already succeeded). */ + private volatile TransactionTemplate txTemplate; + + private TransactionTemplate txTemplate() { + TransactionTemplate t = txTemplate; + if (t == null) { + t = new TransactionTemplate(transactionManager); + txTemplate = t; + } + return t; + } + + private Dispatcher dispatcher; + private DispatchSupport dispatchSupport; + private HostManager hostManager; + private JobManager jobManager; + private RqdClient rqdClient; + + private final AtomicBoolean tickInFlight = new AtomicBoolean(false); + + /** + * Live host reservations, persistent across ticks. Key: host id. Value: + * the (layer, priority) pair that has claimed the host. A reservation + * is created when a blocked layer's reconcile claims a target host, and + * removed when the layer's pending unfittable frame count reaches zero, + * the layer leaves the dispatchable set entirely, or a higher-priority + * layer overrides the claim. + * + * Single-writer: the planner thread only. HashMap is not + * thread-safe; reading or writing this field from any other thread + * (e.g. FrameCompleteHandler) requires either replacing it with a + * ConcurrentHashMap or wrapping access in an explicit lock. + * Failover via the advisory lock means a new leader starts with an + * empty map. The block-time bucket (blockedDebtMs) is in-memory too and + * resets with it, so reservations do NOT rebuild in a tick or two: each + * blocked layer must re-accrue reservation_block_seconds before it can + * reserve again, so protection re-arms only over that window after a + * failover (placement itself is unaffected and resumes immediately). + */ + private final Map reservations = new HashMap<>(); + + // ---- plan / batch-commit / launch ------------------------------------- + // + // The planner is single-threaded. During placement it only RECORDS the + // (host, layer) pairings it wants (plannedByHost). After placement it + // reads each pairing's next frames (planHost, no writes), commits them all + // in one batched transaction (startFramesAndProcsBatch: batched frame + // UPDATE + proc INSERT + host UPDATE, ~4 statements instead of ~6 per + // frame), then fires the RQD launches on a small pool. This replaced the + // per-frame commit-worker pool whose ~6 round-trips per frame dominated + // tick time at scale. + + // Per-tick placements to commit, grouped host id -> layer ids. Planner-thread + // only; cleared each tick after the batch commit. + private final Map> plannedByHost = new LinkedHashMap<>(); + // Small pool for the post-commit RQD launches (runFrame gRPC), which are + // inherently one call per frame. Bounded with caller-runs backpressure so a + // slow RQD cannot let launches pile up unbounded. + private volatile ExecutorService launchPool; + // Count of launches dropped because the launch queue was full (RQD sink too + // slow). The frame is already RUNNING in the DB; reconciliation recovers it. + private final java.util.concurrent.atomic.AtomicLong launchDropped = + new java.util.concurrent.atomic.AtomicLong(0); + // Pool for the PLAN phase: the per-host candidate reads (planHost) are + // read-only and the dominant tick cost as the farm fills, so they run in + // parallel, one task per host (a host's layers stay serial within the task + // so the in-memory capacity decrement is correct; different hosts run + // concurrently). The single batched commit still happens after those reads. + private volatile ExecutorService readPool; + // Written once inside synchronized startSchedulerPoolsIfNeeded(); read on + // the planner thread in the dispatch loop without holding the lock. + // volatile ensures the written value is immediately visible to the + // planner thread after startSchedulerPoolsIfNeeded() returns. + private volatile int jobFrameDispatchMax; + // When false, the planner ignores reservations entirely (no claims made, + // none enforced), the bare placement core. Lets us isolate core + // scheduling behaviour from the reservation logic. Read once in + // startSchedulerPoolsIfNeeded(); volatile for visibility on the planner thread. + private volatile boolean reservationsEnabled = true; + // EASY/Maui-style reservation gating (Lifka 1995; Jackson, Snell, Clement + // 2001), with a per-host-class cap so reservations can never consume a + // whole machine type ("conservative backfill", the low-utilization + // extreme). Three rules: + // + // - Time gate: a layer must be BLOCKED (waiting frames, not capped, but + // no host fits even one) continuously for reservationBlockMs before it + // may reserve. Wall-clock, so it is independent of the tick rate and + // matches operator intuition ("still stuck after 5 minutes -> act"). + // Ignores momentary saturation, which clears within a tick or two. + // + // - Capacity cap: reservations may hold at most reservationMaxFraction of + // the hosts that can fit a given layer, so the rest of that class always + // stays open and the farm cannot deadlock on reservations. + // + // - Width gate: a reservation exists to DRAIN a host so a wide frame can + // assemble a contiguous block. Only a layer whose per-frame core request + // is at least RESERVATION_MIN_HOST_FRACTION of the LARGEST host in its + // group may reserve; narrow layers run the instant any core frees and must + // not consume the scarce reservation budget. A fraction of the biggest + // host (self-tuning across host classes), not an absolute core count. + // + // Granting is priority-first (EASY/Maui): candidates are processed in + // priority+age order, each reconciling up to its capacity-aware need until + // the cap is reached, so high-priority blocked work gets first claim on the + // limited reservation budget. Read once in startSchedulerPoolsIfNeeded(); + // volatile for planner-thread visibility. + private volatile long reservationBlockMs = 300_000; // 5 minutes + private volatile double reservationMaxFraction = 0.5; + // Width gate (always on, not configurable): a layer may reserve only if its + // per-frame cores are at least this fraction of the largest host in its group. + private static final double RESERVATION_MIN_HOST_FRACTION = 0.5; + // EASY backfill (Lifka 1995): rather than freezing a reserved host idle + // while it drains, let a lower-priority frame run on its free cores, but + // ONLY if that frame's worst-case runtime (layer_usage.int_clock_time_high) + // finishes before the host is projected to free enough cores for its + // reserving (wide) layer, so the reserved job is never delayed. Recovers + // the utilization a pure freeze wastes. Read once in + // startSchedulerPoolsIfNeeded(); volatile for planner-thread visibility. + private volatile boolean backfillEnabled = true; + // Per-layer leaky bucket of NET blocked time (ms): grows while the layer is + // blocked, decays (1:1) while it places. A layer qualifies to reserve once + // its debt reaches reservationBlockMs, so a job that only CRAWLS, winning + // the odd gap, which would reset a "continuously blocked" timer, still + // accumulates and reserves, while a healthy layer stays near zero. + // lastSeenMs gives the per-layer time delta between ticks. Planner-thread only. + private final Map blockedDebtMs = new HashMap<>(); + private final Map lastSeenMs = new HashMap<>(); + + // ---- log throttling + per-window stat line ----------------------------- + // A sub-second tick would write tens of thousands of INFO lines a day if it + // logged every tick, so per-tick detail goes to DEBUG and INFO gets ONE + // consolidated stat line per statIntervalMs (scheduler.stat_interval_seconds, + // default 5 minutes). The line is a full snapshot meant to be pasted straight + // into a bug report: planner health and HA leadership, farm fill at the last + // planned tick, throughput and loss, and reservation/backfill activity (see + // maybeLogStat). It is emitted on EVERY tick attempt, leader or standby, so a + // standby that never wins the lock still logs a heartbeat (ticks=0, + // lockLost>0) proving it is alive rather than dead. + // + // Core points per whole core: OpenCue stores host/proc cores as cores * 100. + private static final int CORE_POINTS_PER_CORE = 100; + // Every field below is planner-thread only EXCEPT summarySkipped, which is + // bumped by the concurrent trigger thread whose tick overlapped (the CAS + // loser never holds tickInFlight), hence atomic. The summary* accumulators + // are summed across the window; the last* fields hold the most-recent planned + // tick's snapshot; the tick* fields are per-tick outputs doTick hands back to + // runTick. maybeLogStat emits the line and resets the window. + private volatile long statIntervalMs = 300_000; + private long lastSummaryMs = 0; + private int summaryTicks = 0; // ticks this Cuebot won and planned + private long summaryDispatched = 0; // procs committed (won the version race) + private long summaryTickMs = 0; // summed tick wall time (for the mean) + private long summaryMaxTickMs = 0; // slowest single tick in the window + private int summaryLockLost = 0; // attempts another Cuebot held the lock + private long summaryPlanned = 0; // frames the plan phase produced + private int summaryGranted = 0; // new reservations granted + private int summaryBackfilled = 0; // frames placed onto a reserved host + private long summaryBackfilledCores = 0; // core-points placed via EASY backfill + private long summaryLaunchDroppedAt = 0; // launchDropped count at window start + private final java.util.concurrent.atomic.AtomicInteger summarySkipped = + new java.util.concurrent.atomic.AtomicInteger(0); + // Most-recent planned tick's farm snapshot (set at the top of doTick, BEFORE + // placement mutates the in-memory idle counts), reported as the point-in-time + // view in the stat line. + private int lastHosts = 0; // schedulable hosts + private int lastIdleHosts = 0; // hosts with >= reservable-min idle + private long lastCoresTotalCp = 0; // total cores, core points + private long lastCoresIdleCp = 0; // idle cores, core points + private int lastGroups = 0; // host-spec groups + private int lastReservationReqs = 0; // layers requesting a reservation last tick + // Per-tick outputs set by doTick(), folded into the window by runTick(). + private long tickPlanned = 0; + private int tickGranted = 0; + private int tickBackfilled = 0; + private long tickBackfilledCores = 0; + // Guardrail: a handful of host-spec groups is expected. A count anywhere near + // the host count means hosts are fragmenting into near-per-host groups (the + // classic cause is a host name leaking into the tag set), which collapses the + // scheduler back into the per-host query storm it exists to avoid. Warn + // loudly, but throttled so it does not itself spam the log. + private static final int GROUP_COUNT_WARN_THRESHOLD = 100; + private static final long GROUP_WARN_INTERVAL_MS = 300_000; // at most every 5 min + private long lastGroupWarnMs = 0; + + // ---- batched resource accounting -------------------------------------- + // + // Booking a proc the legacy way fires five single-row UPDATEs + // (subscription, layer_resource, job_resource, folder_resource, point) + // inside the booking transaction. At full-farm scale a tick books + // thousands of procs, and they all target the same handful of hot rows + // (one point row per show/dept, one folder_resource per folder, ...), so + // they serialize on those row locks, the dominant commit cost at scale. + // Instead, the batched commit path (batchInsertVirtualProcs) never issues + // those per-proc writes, and the planner records the equivalent deltas here; + // doTick flushes one UPDATE per row right after the commit, collapsing + // thousands of contended writes into a few dozen. Proc release (frame + // complete) and legacy dispatch keep their per-proc updates. + // + // Always on for the new Scheduler, EXCEPT when scheduler_manages_resources + // is true, then the Rust scheduler's periodic recompute owns these tables + // and we must not write them at all. Set in startSchedulerPoolsIfNeeded. + private volatile boolean batchResourceAccounting = true; + // Per-row delta buffers: value is {cores, gpus}. Written on the planner + // thread when the batch commit's winners are accounted, then drained in + // flushResourceDeltas right after the commit. + // subDeltas key: pkShow + '\t' + pkAlloc + // layerDeltas key: pkLayer + // jobDeltas key: pkJob (drives job_resource, folder_resource, point) + private final Map subDeltas = new ConcurrentHashMap<>(); + private final Map layerDeltas = new ConcurrentHashMap<>(); + private final Map jobDeltas = new ConcurrentHashMap<>(); + + /** + * Lazy launch-pool init on the first runTick. Avoids touching Spring + * XML wiring for an init-method, and Cuebot is well past startup by + * the time scheduler.enabled is flipped on. + */ + private synchronized void startSchedulerPoolsIfNeeded() { + if (launchPool != null) return; + int launchSize = env.getProperty("scheduler.launch_pool_size", Integer.class, 8); + jobFrameDispatchMax = env.getProperty("dispatcher.job_frame_dispatch_max", + Integer.class, 8); + reservationsEnabled = env.getProperty("scheduler.reservations_enabled", + Boolean.class, true); + reservationBlockMs = 1000L * env.getProperty("scheduler.reservation_block_seconds", + Integer.class, 300); + reservationMaxFraction = env.getProperty("scheduler.reservation_max_fraction", + Double.class, 0.5); + backfillEnabled = env.getProperty("scheduler.backfill_enabled", + Boolean.class, true); + localityEnabled = env.getProperty("scheduler.locality_enabled", + Boolean.class, true); + localityBonus = env.getProperty("scheduler.locality_bonus", + Double.class, 8.0); + // Cadence of the consolidated INFO stat line (see maybeLogStat). Default + // 5 minutes; lower it for a live incident, raise it to quiet the log. + statIntervalMs = 1000L * env.getProperty("scheduler.stat_interval_seconds", + Integer.class, 300); + // Batch resource accounting unless the Rust scheduler owns those tables + // via its periodic recompute (scheduler_manages_resources). In that mode + // procCreated writes nothing and we must not either. + batchResourceAccounting = !env.getProperty( + "dispatcher.scheduler_manages_resources", Boolean.class, false); + // Bounded pool for post-commit RQD launches. Launches must NEVER run on + // the tick thread: a slow RQD sink would otherwise stall the whole tick + // (caller-runs put thousands of serial gRPC calls on the planner). On a + // full queue we DROP the launch and count it, the frame is already + // RUNNING in the DB, so RQD report reconciliation recovers it, so the + // tick is pure fire-and-forget and its latency never depends on RQD. + int launchQueueSize = env.getProperty("scheduler.launch_queue_size", + Integer.class, 16384); + ThreadPoolExecutor pool = new ThreadPoolExecutor(launchSize, launchSize, 0L, + TimeUnit.MILLISECONDS, new LinkedBlockingQueue<>(launchQueueSize), r -> { + Thread t = new Thread(r); + t.setName("Scheduler-launch-" + t.getId()); + t.setDaemon(true); + return t; + }, (r, ex) -> { + long n = launchDropped.incrementAndGet(); + if (n % 1000 == 1) { + logger.warn("Scheduler: launch queue full, dropping launch" + + " (total dropped=" + n + "); RQD reconciliation will recover"); + } + }); + launchPool = pool; + // Read pool for the parallel PLAN phase. Reads are DB-bound (they block + // on Postgres, not the CPU), so sizing above the core count is fine. + int readSize = env.getProperty("scheduler.read_pool_size", Integer.class, launchSize); + readPool = Executors.newFixedThreadPool(readSize, r -> { + Thread t = new Thread(r); + t.setName("Scheduler-read-" + t.getId()); + t.setDaemon(true); + return t; + }); + logger.info("Scheduler: launch pool started with " + launchSize + + " workers, read pool with " + readSize + " workers"); + } + + // ---- snapshot queries ------------------------------------------------- + + /** + * Bookable hosts: UP, lock state OPEN, with at least the minimum bookable + * cores. Returns enough columns to compute the spec key and run the + * per-host fit check without a second lookup. + */ + private static final String SELECT_ALL_HOSTS = + "SELECT " + + " h.pk_host, " + + " h.str_name, " + + " h.pk_alloc, " + + " h.int_cores, " + + " h.int_cores_idle, " + + " h.int_mem, " + + " h.int_mem_idle, " + + " h.int_gpus, " + + " h.int_gpus_idle, " + + " h.int_gpu_mem, " + + " h.int_gpu_mem_idle, " + + " h.int_procs, " + + " h.str_tags, " + + " hs.str_os " + + "FROM host h, host_stat hs " + + "WHERE h.pk_host = hs.pk_host " + + " AND hs.str_state = 'UP' " + + " AND h.str_lock_state = 'OPEN' "; + + /** + * Candidate layers for a host spec group. One query per group. Filters: + * - job PENDING and unpaused + * - tag regex match against the group's normalized tag string + * - OS match (or any if the job is OS-agnostic) + * - job under int_max_cores + * - show under subscription burst on this alloc + * - at least one WAITING, depend-resolved frame on the layer + * - layer.int_cores_min fits the group's max host TOTAL cores (not idle + * , a blocked layer waiting on a reserved host stays in the candidate + * set even when no host has it idle right now) + * Ranked by the priority-weighted lottery (power(random(),1/priority)) and + * capped by LIMIT, NOT strictly by priority. waiting_frame_count is the + * number of dispatchable frames on the layer at query time; reconciliation + * uses it to decide how many hosts the layer should reserve. + */ + private static final String SELECT_CANDIDATES_FOR_GROUP = + "SELECT " + + " l.pk_layer, " + + " l.pk_job, " + + " j.pk_show, " + + " l.int_cores_min, " + + " l.int_mem_min, " + + " l.int_gpus_min, " + + " l.int_gpu_mem_min, " + + " jr.int_priority, " + + " jr.int_cores AS job_cores_in_use, " + + " jr.int_max_cores AS job_max_cores, " + + " sub.int_cores AS show_cores_in_use, " + + " sub.int_burst AS show_burst, " + + " COALESCE(ls.int_waiting_count, 0) AS waiting_frame_count, " + + " COALESCE(lu.int_clock_time_high, 0) AS clock_time_high, " + + " COALESCE(lu.int_frame_success_count, 0) AS frame_success_count, " + // Limit (license-cap) accounting: the layer's most-constraining limit, + // its cap, and how many frames of that limit run farm-wide right now. + + " lim.pk_limit_record AS limit_id, " + + " COALESCE(lim.int_max_value, 0) AS limit_max, " + + " COALESCE(lu2.int_sum_running, 0) AS limit_running, " + // Folder (group/dept) core cap: the job's folder, its ceiling, and the + // folder's current running cores (ground truth = SUM of the folder's jobs). + + " j.pk_folder AS folder_id, " + + " COALESCE(fr.int_max_cores, -1) AS folder_max, " + + " COALESCE(fu.folder_cores, 0) AS folder_running " + + "FROM layer l " + + "JOIN job j ON j.pk_job = l.pk_job " + + "JOIN job_resource jr ON jr.pk_job = j.pk_job " + + "JOIN show sh ON sh.pk_show = j.pk_show " + + "JOIN subscription sub ON sub.pk_show = j.pk_show AND sub.pk_alloc = ? " + + "LEFT JOIN layer_usage lu ON lu.pk_layer = l.pk_layer " + + "LEFT JOIN layer_stat ls ON ls.pk_layer = l.pk_layer " + // The layer's most-constraining limit (smallest cap), one row per layer. + + "LEFT JOIN LATERAL (" + + " SELECT ll.pk_limit_record, lr.int_max_value " + + " FROM layer_limit ll " + + " JOIN limit_record lr ON lr.pk_limit_record = ll.pk_limit_record " + + " WHERE ll.pk_layer = l.pk_layer " + + " ORDER BY lr.int_max_value LIMIT 1) lim ON true " + // Farm-wide running count per limit (computed once, not per row). + + "LEFT JOIN (" + + " SELECT ll2.pk_limit_record, SUM(ls2.int_running_count) AS int_sum_running " + + " FROM layer_limit ll2 " + + " JOIN layer_stat ls2 ON ls2.pk_layer = ll2.pk_layer " + + " GROUP BY ll2.pk_limit_record) lu2 " + + " ON lu2.pk_limit_record = lim.pk_limit_record " + // Folder core ceiling + the folder's current running cores. Derived from + // layer_stat.int_running_count (running frames x per-frame cores) -- the + // same trigger-maintained counter the limit cap uses. It is robust to frame + // completion (int_running_count drops automatically) and, at a tick + // boundary, equals SUM(job_resource.int_cores) (one proc per running + // frame), the figure the folder cap is measured against. Computed once, + // not per row. + + "LEFT JOIN folder_resource fr ON fr.pk_folder = j.pk_folder " + + "LEFT JOIN (" + + " SELECT j2.pk_folder, " + + " SUM(ls2.int_running_count * l2.int_cores_min) AS folder_cores " + + " FROM job j2 " + // Only aggregate CAPPED folders (int_max_cores <> -1). Every job has a + // folder but almost none are capped, so without this join the subquery + // would sum layer_stat across the whole farm every candidate query; this + // keeps it empty (free) when no folder has a ceiling. + + " JOIN folder_resource fr2 ON fr2.pk_folder = j2.pk_folder " + + " AND fr2.int_max_cores <> -1 " + + " JOIN layer l2 ON l2.pk_job = j2.pk_job " + + " JOIN layer_stat ls2 ON ls2.pk_layer = l2.pk_layer " + + " WHERE j2.str_state = 'PENDING' " + + " GROUP BY j2.pk_folder) fu ON fu.pk_folder = j.pk_folder " + + "WHERE j.str_state = 'PENDING' " + + " AND j.b_paused = false " + + " AND (j.str_os IS NULL OR j.str_os = '' OR j.str_os = ?) " + + " AND ? ~* ('(?x)' || l.str_tags || '\\y') " + + " AND jr.int_cores < jr.int_max_cores " + + " AND sub.int_cores < sub.int_burst " + + " AND l.int_cores_min <= ? " + // Dispatchable-frame test AND waiting_frame_count both come from + // layer_stat.int_waiting_count (maintained by core trigger + // trigger__update_frame_status_counts; WAITING frames are depend-resolved, + // DEPEND is a separate state). Backed by the partial index + // idx_layer_stat_waiting (V44). Replaces a correlated COUNT(*) + EXISTS + // over frame that scanned every frame of each candidate layer per tick. + + " AND COALESCE(ls.int_waiting_count, 0) > 0 " + // Skip layers whose limit (license cap) is already full farm-wide. Their + // frames get filtered out downstream by findNextDispatchFrames anyway, so + // scoring a host + running the plan read for them only burns a cycle that + // returns nothing (it surfaces as raceLost). A limit-less layer (NULL) + // always passes. Not a correctness gate -- the downstream query still + // enforces the cap -- purely an efficiency filter. + + " AND (lim.pk_limit_record IS NULL " + + " OR COALESCE(lu2.int_sum_running, 0) < lim.int_max_value) " + // Skip jobs whose FOLDER (group/dept) core ceiling is already reached -- + // folder_resource.int_max_cores, another core cap the legacy dispatcher + // enforces. -1 = unlimited. Same rationale as the limit filter: purely an + // efficiency gate (don't plan bookings a full folder can't take). The exact + // ceiling is enforced by the post-plan folder trim in doTick, which -- + // unlike this filter -- also binds the frames planHost books in the tick + // that crosses the cap. + + " AND (COALESCE(fr.int_max_cores, -1) = -1 " + + " OR COALESCE(fu.folder_cores, 0) + l.int_cores_min <= fr.int_max_cores) " + // Progressive rollout: in 'managed' mode only shows flagged + // b_scheduler_managed are planned here (the legacy dispatch query excludes + // exactly those, so the two partition); in 'facility' mode the bound flag + // is true and this short-circuits to plan every show. + + " AND (? OR sh.b_scheduler_managed = true) " + // Priority-WEIGHTED LOTTERY, not a strict priority sort. Each eligible + // layer gets a random key random()^(1/priority) -- Efraimidis-Spirakis + // weighted reservoir sampling -- and we take the top-LIMIT by that key. + // ORDER BY ranks the WHOLE eligible set before LIMIT (sort-then-limit), + // so a low-priority layer always keeps a real, smaller chance of being + // selected: its expected share is proportional to its priority, so it is + // never starved by a sustained higher-priority stream. The old strict + // "int_priority DESC" starved it outright -- pri-100 work never ran while + // a pri-300 backlog kept the farm saturated. GREATEST(...,1) floors the + // weight so priority 0/negative still gets the minimum (nonzero) share + // rather than divide-by-zero or starvation. Reservation GRANTING stays + // strict priority-first (the requests are re-sorted by priority below), + // so wide-job rescue is unaffected by this booking-order change. + + "ORDER BY power(random(), 1.0 / GREATEST(jr.int_priority, 1)) DESC " + + "LIMIT ? "; + + // ---- row mappers ------------------------------------------------------ + + private static final RowMapper HOST_MAPPER = new RowMapper() { + public BookableHost mapRow(ResultSet rs, int i) throws SQLException { + BookableHost h = new BookableHost(); + h.hostId = rs.getString("pk_host"); + h.hostName = rs.getString("str_name"); + h.pkAlloc = rs.getString("pk_alloc"); + h.coresTotal = rs.getInt("int_cores"); + h.coresIdle = rs.getInt("int_cores_idle"); + h.memTotal = rs.getLong("int_mem"); + h.memIdle = rs.getLong("int_mem_idle"); + h.gpusTotal = rs.getInt("int_gpus"); + h.gpusIdle = rs.getInt("int_gpus_idle"); + h.gpuMemTotal = rs.getLong("int_gpu_mem"); + h.gpuMemIdle = rs.getLong("int_gpu_mem_idle"); + h.runningProcs = rs.getInt("int_procs"); + h.tagsRaw = rs.getString("str_tags"); + h.os = rs.getString("str_os"); + return h; + } + }; + + private static final RowMapper CANDIDATE_MAPPER = + new RowMapper() { + public LayerCandidate mapRow(ResultSet rs, int i) throws SQLException { + LayerCandidate c = new LayerCandidate(); + c.layerId = rs.getString("pk_layer"); + c.jobId = rs.getString("pk_job"); + c.showId = rs.getString("pk_show"); + c.layerCoresMin = rs.getInt("int_cores_min"); + c.layerMemMin = rs.getLong("int_mem_min"); + c.layerGpusMin = rs.getInt("int_gpus_min"); + c.layerGpuMemMin = rs.getLong("int_gpu_mem_min"); + c.priority = rs.getInt("int_priority"); + c.jobCoresInUse = rs.getInt("job_cores_in_use"); + c.jobMaxCores = rs.getInt("job_max_cores"); + c.showCoresInUse = rs.getInt("show_cores_in_use"); + c.showBurstCores = rs.getInt("show_burst"); + c.waitingFrameCount = rs.getInt("waiting_frame_count"); + c.clockTimeHighSec = rs.getInt("clock_time_high"); + c.frameSuccessCount = rs.getInt("frame_success_count"); + c.limitId = rs.getString("limit_id"); // null when no limit + c.limitMax = rs.getInt("limit_max"); + c.limitRunning = rs.getInt("limit_running"); + c.folderId = rs.getString("folder_id"); + c.folderMax = rs.getInt("folder_max"); // -1 = unlimited + c.folderRunning = rs.getInt("folder_running"); // core-points + return c; + } + }; + + // ---- tick ------------------------------------------------------------- + + /** Public entry point. Invoked by the Quartz trigger. */ + public void runTick() { + if (!isEnabled()) return; + if (!tickInFlight.compareAndSet(false, true)) { + logger.debug("Scheduler: previous tick still running, skipping"); + summarySkipped.incrementAndGet(); + return; + } + startSchedulerPoolsIfNeeded(); + long t0 = System.currentTimeMillis(); + try { + // A Postgres advisory lock is session-scoped: it can only be + // released by the same physical connection that took it. + // getJdbcTemplate() borrows a fresh pooled connection per + // statement, so acquire and release would land on different + // connections, leaking the lock and stalling the scheduler. + // Pin one connection for the whole tick and run lock/unlock on + // it directly; the planning queries can still use the pool. + Connection lockConn = null; + try { + lockConn = getDataSource().getConnection(); + if (!acquireLeaderLock(lockConn)) { + logger.debug("Scheduler: another Cuebot holds the planning lock"); + summaryLockLost++; + return; + } + try { + int dispatched = doTick(); + long ms = System.currentTimeMillis() - t0; + // Per-tick detail at DEBUG; INFO gets one consolidated stat + // line per window (maybeLogStat, called in the finally below). + logger.debug("Scheduler tick: dispatched " + dispatched + + " procs, " + ms + " ms, reservations=" + reservations.size()); + summaryTicks++; + summaryDispatched += dispatched; + summaryTickMs += ms; + if (ms > summaryMaxTickMs) summaryMaxTickMs = ms; + summaryPlanned += tickPlanned; + summaryGranted += tickGranted; + summaryBackfilled += tickBackfilled; + summaryBackfilledCores += tickBackfilledCores; + } finally { + releaseLeaderLock(lockConn); + } + } finally { + if (lockConn != null) { + try { + lockConn.close(); + } catch (SQLException e) { + logger.debug("Scheduler: closing lock connection failed: " + + e.getMessage()); + } + } + } + } catch (RuntimeException | SQLException e) { + logger.error("Scheduler tick failed", e); + } finally { + // One consolidated stat line per window, on EVERY tick attempt + // (leader or standby) so a standby Cuebot still emits a heartbeat. + // Reached only by the thread that held tickInFlight (the CAS loser + // returned earlier), so the plain summary fields stay single-writer. + maybeLogStat(); + tickInFlight.set(false); + } + } + + /** + * Emit the consolidated per-window stat line at most once per + * {@link #statIntervalMs}, then reset the window accumulators. A full + * snapshot meant to be pasted into a bug report: + * + * 

+     * Scheduler stat: win=300s ticks=920 skipped=0 lockLost=12 avgTick=556ms maxTick=1840ms
+     *   | farm hosts=1553 idleHosts=9 cores=57088 idleCores=74 util=99.9% groups=5
+     *   | flow committed=98210 planned=104900 raceLost=6690 launchDropped=0
+     *   | resv held=52 reservedCores=3328 granted=31 reqs=11 backfilled=88 backfilledCores=512
+     *
+ * + *
    + *
  • health/HA: win (window seconds), ticks won, skipped (fired + * while the previous tick still ran, so falling behind), lockLost + * (another Cuebot held the advisory lock, so this one was standby for + * that tick), avgTick/maxTick.
    • + *
  • farm: the last planned tick's host/core fill and host-spec + * group count (a count near the host count is the tag-leak bug the + * guardrail warns on).
    • + *
  • flow: committed procs, frames the plan phase produced, the + * gap lost to the frame version race (contention), and RQD launches + * dropped because the launch queue was full.
    • + *
  • resv: reservations held, the whole cores those held reservations + * account for (reservedCores), reservations newly granted, requested last + * tick, frames placed onto a reserved host via EASY backfill, and the + * cores (whole cores, not core-points) those backfilled frames borrowed.
    • + *
+ * + * Called from runTick's finally on the thread that held tickInFlight, so the + * plain fields are single-writer (summarySkipped is atomic because the CAS + * loser bumps it from another thread). + */ + private void maybeLogStat() { + long nowMs = System.currentTimeMillis(); + if (lastSummaryMs == 0) { // first call: start the window, do not emit + lastSummaryMs = nowMs; + summaryLaunchDroppedAt = launchDropped.get(); + return; + } + if (nowMs - lastSummaryMs < statIntervalMs) return; + + long win = (nowMs - lastSummaryMs) / 1000; + long coresTotal = lastCoresTotalCp / CORE_POINTS_PER_CORE; + long idleCores = lastCoresIdleCp / CORE_POINTS_PER_CORE; + double util = lastCoresTotalCp > 0 + ? 100.0 * (lastCoresTotalCp - lastCoresIdleCp) / lastCoresTotalCp : 0.0; + long avgTick = summaryTicks > 0 ? summaryTickMs / summaryTicks : 0; + long raceLost = Math.max(0, summaryPlanned - summaryDispatched); + long dropNow = launchDropped.get(); + long droppedInWindow = dropNow - summaryLaunchDroppedAt; + int skipped = summarySkipped.getAndSet(0); + + // Cores currently held by the wide-job reservation feature: the sum of the + // reserved layers' per-frame core requirement across all live reservations. + // A point-in-time level (like held), not a window flow, so it is summed + // here from the reservations map rather than accumulated per tick. + long reservedCp = 0; + for (Reservation r : reservations.values()) { + reservedCp += r.layerCoresMin; + } + + logger.info(String.format( + "Scheduler stat: win=%ds ticks=%d skipped=%d lockLost=%d avgTick=%dms maxTick=%dms" + + " | farm hosts=%d idleHosts=%d cores=%d idleCores=%d util=%.1f%% groups=%d" + + " | flow committed=%d planned=%d raceLost=%d launchDropped=%d" + + " | resv held=%d reservedCores=%d granted=%d reqs=%d backfilled=%d backfilledCores=%d", + win, summaryTicks, skipped, summaryLockLost, avgTick, summaryMaxTickMs, + lastHosts, lastIdleHosts, coresTotal, idleCores, util, lastGroups, + summaryDispatched, summaryPlanned, raceLost, droppedInWindow, + reservations.size(), reservedCp / CORE_POINTS_PER_CORE, summaryGranted, + lastReservationReqs, summaryBackfilled, + summaryBackfilledCores / CORE_POINTS_PER_CORE)); + + lastSummaryMs = nowMs; + summaryTicks = 0; + summaryDispatched = 0; + summaryTickMs = 0; + summaryMaxTickMs = 0; + summaryLockLost = 0; + summaryPlanned = 0; + summaryGranted = 0; + summaryBackfilled = 0; + summaryBackfilledCores = 0; + summaryLaunchDroppedAt = dropNow; + } + + /** + * One scheduling tick. The algorithm in order: + * + * 1. SNAPSHOT + * Read all bookable hosts (UP, OPEN, with at least the minimum + * bookable cores) in one SQL query. Each row carries the host's + * static spec (alloc, tags, OS), its current idle resources, its + * total capacity, and its running proc count. + * + * 2. GROUP + * Bucket hosts by their static spec key (alloc, normalized tags, + * os, has-gpu). Hosts in the same group share the same set of + * candidate layers, so one candidate query per group instead of + * per host. + * + * 3. FOR EACH GROUP: + * a. CANDIDATE QUERY + * One SQL per group, returning up to + * scheduler.layer_candidates_per_group_max layers, ranked by + * the priority-weighted lottery (§3.5), not strict priority. The + * filter "int_cores_min <= group's + * MAX TOTAL cores" includes blocked layers whose reserved + * hosts are partially loaded; using max IDLE would let them + * drop out of the candidate set and be swept incorrectly. + * + * b. DISPATCH AND RECONCILE (priority order) + * Implemented in dispatchGroupWithScoring. For each candidate: + * - Drain by best-fit onto fitting hosts. Reservation + * rules apply: a host reserved at priority >= c.priority + * for another layer is skipped; a host reserved at lower + * priority is usable, and on dispatch c takes ownership. + * - Reconcile c's reservation count to exactly c's + * remaining pending unfittable frame count. + * Layer ids encountered are added to seenLayerIds for the + * end-of-tick sweep. + * + * 4. SWEEP + * Any reservation whose layer didn't appear in any candidate set + * this tick is dropped. That layer is no longer dispatchable + * (job paused, completed, deleted, or its int_cores_min exceeds + * every host's total capacity), so its claim is stale. + * + * The reservation map persists across ticks. The single invariant is + * that a host's reservation belongs to the highest-priority layer + * that has claimed it; every operation above respects this. A new + * leader after failover starts with an empty map; because the block-time + * bucket resets with it, reservations re-arm only as blocked layers + * re-accrue reservation_block_seconds, not within a tick or two. + * + * @return total number of procs dispatched this tick + */ + private int doTick() { + long tStart = System.currentTimeMillis(); + // Reset per-tick stat outputs before any early return, so a host-less + // tick contributes zero to the window rather than last tick's values. + tickPlanned = 0; + tickGranted = 0; + tickBackfilled = 0; + tickBackfilledCores = 0; + // 1. SNAPSHOT. Read ALL schedulable hosts (UP + OPEN), busy or idle. + // Placement only uses the idle ones, but reservations must see BUSY + // hosts too, a reservation's whole purpose is to hold a host that is + // full now until it drains (EASY/conservative backfill: you reserve a + // node that is currently running work, not one that is already free). + List allHosts = readAllHosts(); + if (allHosts.isEmpty()) { + // No hosts at all. Leave existing reservations alone; they belong + // to layers whose hosts are simply unavailable this tick. + return 0; + } + + // Farm snapshot for the per-window stat line, taken BEFORE this tick's + // placement mutates the in-memory idle counts (placement decrements + // h.coresIdle). Core points; maybeLogStat converts to whole cores. + lastHosts = allHosts.size(); + lastIdleHosts = 0; + lastCoresTotalCp = 0; + lastCoresIdleCp = 0; + for (BookableHost h : allHosts) { + lastCoresTotalCp += h.coresTotal; + lastCoresIdleCp += h.coresIdle; + if (h.coresIdle >= Dispatcher.CORE_POINTS_RESERVED_MIN) lastIdleHosts++; + } + + // 2. GROUP all hosts by spec. Each group carries its full host set; the + // idle subset (for placement) is derived per group below. + Map> groups = groupByHostSpec(allHosts); + lastGroups = groups.size(); + // Guardrail: too many groups means the spec key is fragmenting per host + // (usually a host name leaked into the tag set), which defeats the whole + // point of grouping. Warn loudly but throttled so we never log it per tick. + if (groups.size() >= GROUP_COUNT_WARN_THRESHOLD) { + long nowMs = System.currentTimeMillis(); + if (nowMs - lastGroupWarnMs >= GROUP_WARN_INTERVAL_MS) { + lastGroupWarnMs = nowMs; + logger.warn("Scheduler: " + groups.size() + " host-spec groups for " + + allHosts.size() + " hosts (a handful is expected). A count near" + + " the host count means hosts are fragmenting into near-per-host" + + " groups, commonly a host name leaking into the tag set, which" + + " collapses planning into one candidate query per host, the very" + + " query storm the scheduler avoids. Check tag normalization" + + " (normalizeTags / groupByHostSpec)."); + } + } + Set seenLayerIds = new HashSet<>(); + + // EASY-backfill deadlines: for each host reserved on a PRIOR tick, the + // estimated seconds until it frees enough cores for its reserving layer. + // Computed from the untouched snapshot (before this tick's dispatch + // mutates idle counts). A borrowed frame may run on a reserved host only + // if it finishes before this deadline (backfillAllows). + Map hostById = new HashMap<>(); + for (BookableHost h : allHosts) hostById.put(h.hostId, h); + Map tReadyByHost = computeHostReadySeconds(hostById); + + // Host->layer affinity for the locality bonus. Read once per tick from + // the live proc table (before this tick's commits mutate it). + Map> hostLayerAffinity = readHostLayerAffinity(); + + // Job/show core accounting shared across every candidate this tick. + // The candidate query seeds each candidate from a point-in-time DB + // snapshot, so two layers of the same job (or two jobs of the same + // show), in one group or across groups, would each book up to the + // full cap. These tick-scoped maps carry the accumulated in-tick + // usage so the caps are enforced once, in aggregate. Planner-thread + // only, like reservations. + Map jobCoresUsed = new HashMap<>(); + Map showCoresUsed = new HashMap<>(); + // Tick-scoped running count per limit (license cap), seeded from the + // farm-wide count and incremented as this tick books, so several layers + // sharing a limit don't each fill it to the cap. Enforced in aggregate. + Map limitUsed = new HashMap<>(); + // Tick-scoped running CORES per folder (group/dept ceiling), seeded from + // the farm-wide count and enforced in aggregate -- like limits, but the + // folder cap is in cores, not frames. + Map folderUsed = new HashMap<>(); + // Capped-folder bookkeeping for the exact post-plan trim below. planHost + // has no folder clause, so within one tick it can book a capped folder past + // its ceiling; these carry, per capped folder, its ceiling and its running + // cores at tick start, so the trim can hold the committed total to the cap. + Map folderMaxCp = new HashMap<>(); // folderId -> ceiling (core-points) + Map folderRunSeed = new HashMap<>(); // folderId -> running at tick start + Map jobFolderCap = new HashMap<>(); // jobId -> its capped folderId + + // Layers that want reservations this tick (blocked long enough, or + // already holding). Collected during placement and granted afterwards + // priority-first then widest-job, so the scarce reservation budget goes + // to the highest-priority work that actually cannot fit (wide jobs) + // rather than to whichever small layer happened to be oldest. + List reservationReqs = new ArrayList<>(); + + int dispatched = 0; + for (Map.Entry> g : groups.entrySet()) { + HostSpecKey spec = g.getKey(); + List fullGroup = g.getValue(); + + // Idle subset for placement: hosts with at least the minimum + // reservable cores free. The full group is kept for reservations. + List idleGroup = new ArrayList<>(); + for (BookableHost h : fullGroup) { + if (h.coresIdle >= Dispatcher.CORE_POINTS_RESERVED_MIN) idleGroup.add(h); + } + + // 3a. CANDIDATE QUERY (one per group) + // Filter against max host *total* cores in the group, not max + // idle. A blocked layer waiting on a partially-loaded reserved + // host must remain in the candidate set so its reservation + // survives sweep. + int maxCoresTotalInGroup = fullGroup.stream() + .mapToInt(h -> h.coresTotal).max().orElse(0); + + List candidates = + readLayerCandidatesForGroup(spec, maxCoresTotalInGroup); + if (candidates.isEmpty()) continue; + // Record capped-folder ceilings + running cores for the exact post-plan + // trim. folder_running is folder-global (identical across a folder's + // candidates), so the first value seen wins. + for (LayerCandidate lc : candidates) { + if (lc.folderMax >= 0) { + folderMaxCp.putIfAbsent(lc.folderId, lc.folderMax); + folderRunSeed.putIfAbsent(lc.folderId, lc.folderRunning); + jobFolderCap.putIfAbsent(lc.jobId, lc.folderId); + } + } + // Log per-group candidate summary so we can verify wide-job layers + // are included (not cut by LIMIT or maxCores filter). + if (logger.isDebugEnabled()) { + int wideCount = 0; + for (LayerCandidate lc : candidates) + if (lc.layerCoresMin > 100) wideCount++; + logger.debug("Scheduler group: " + spec + + " hosts=" + fullGroup.size() + " idle=" + idleGroup.size() + + " maxCoresTotal=" + maxCoresTotalInGroup + + " candidates=" + candidates.size() + " wide(>100cores)=" + wideCount); + } + + // 3b. DISPATCH AND RECONCILE (priority order). Placement uses the + // idle subset; reservations use the full group (busy hosts too). + dispatched += dispatchGroupWithScoring(idleGroup, fullGroup, candidates, + seenLayerIds, jobCoresUsed, showCoresUsed, limitUsed, folderUsed, + reservationReqs, + tReadyByHost, hostLayerAffinity); + } + + // 3c. GRANT RESERVATIONS, highest priority first (then widest job). + // reconcile reads the live reservation map, so the per-class cap fills + // up for wide jobs before narrow ones are even considered, the scarce + // budget lands on work that cannot fit, not on small layers that will + // get a gap on their own. + // + // Top-K cap: at full-farm scale hundreds of layers may qualify + // simultaneously; reconciling all of them is O(layers × hosts) and + // blows up tick time. Split into two passes: + // 1. Existing reservers (maintenance): always reconcile so held + // reservations are refreshed or released promptly. + // 2. New qualifiers: only the top-K widest/highest-priority ones + // receive a grant this tick. The rest will qualify again next tick. + int reservationMaxGrantees = env.getProperty( + "scheduler.reservation_max_grantees", Integer.class, 8); + reservationReqs.sort( + Comparator.comparingInt(r -> -r.candidate.priority) + .thenComparingInt(r -> -r.candidate.layerCoresMin)); + // Verbose per-tick reservation summary at DEBUG only (guarded so the + // string is not built when DEBUG is off); INFO sees new grants below and + // the held count in the minute heartbeat. + if (logger.isDebugEnabled() && (!reservationReqs.isEmpty() || !reservations.isEmpty())) { + StringBuilder sb = new StringBuilder(); + sb.append("Scheduler resv-tick: requests=").append(reservationReqs.size()) + .append(" cap=").append(reservationMaxGrantees) + .append(" held=").append(reservations.size()) + .append(" blockThresholdMs=").append(reservationBlockMs); + if (!reservationReqs.isEmpty()) { + sb.append(" top=["); + int show = Math.min(3, reservationReqs.size()); + for (int i = 0; i < show; i++) { + ReservationRequest rr = reservationReqs.get(i); + long debt = blockedDebtMs.getOrDefault(rr.candidate.layerId, 0L); + sb.append("layer=").append(rr.candidate.layerId) + .append("(cores=").append(rr.candidate.layerCoresMin) + .append(",waiting=").append(rr.candidate.waitingFrameCount) + .append(",debt=").append(debt).append("ms)"); + if (i < show - 1) sb.append(", "); + } + sb.append("]"); + } + logger.debug(sb.toString()); + } + int newGrantees = 0; + for (ReservationRequest r : reservationReqs) { + boolean alreadyHolds = layerHoldsReservation(r.candidate.layerId); + if (alreadyHolds) { + reconcileReservationsForLayer(r.candidate, r.fullHosts); + } else if (newGrantees < reservationMaxGrantees) { + reconcileReservationsForLayer(r.candidate, r.fullHosts); + newGrantees++; + } + } + // Log only when a reservation is actually granted (an event), not every + // tick that reservations merely exist, so this stays off the hot path. + if (newGrantees > 0) { + logger.info("Scheduler resv-grant: newGrantees=" + newGrantees + + " totalHeld=" + reservations.size()); + } + tickGranted = newGrantees; + lastReservationReqs = reservationReqs.size(); + + // 4. PLAN bookings: read each placement's next frames and build procs + // in memory (no writes). Parallelized across hosts, the candidate + // reads are the dominant tick cost as the farm fills. + long tPlan = System.currentTimeMillis(); + // plannedByHost is already grouped host -> layer ids. Each host's layers + // must be planned serially on one thread: planHost decrements the + // DispatchHost's idle fields as it books (host.useResources), so a later + // layer on the same host sees the cores an earlier one already took. If + // two threads planned the same host concurrently they would both book + // against full capacity, the aggregated batch host UPDATE would overrun, + // and trigger__verify_host_resources would abort the whole batch (and + // tick). Different hosts are independent, so they run concurrently. + int placements = 0; + for (List ls : plannedByHost.values()) placements += ls.size(); + + List>> tasks = new ArrayList<>(plannedByHost.size()); + for (Map.Entry> e : plannedByHost.entrySet()) { + final String hostId = e.getKey(); + final List layerIds = e.getValue(); + tasks.add(() -> { + List out = new ArrayList<>(); + DispatchHost host = hostManager.getDispatchHost(hostId); + for (String layerId : layerIds) { + LayerInterface layer = jobManager.getLayer(layerId); + out.addAll(dispatcher.planHost(host, layer)); + } + return out; + }); + } + plannedByHost.clear(); + + List planned = new ArrayList<>(); + try { + for (Future> f : readPool.invokeAll(tasks)) { + try { + planned.addAll(f.get()); + } catch (ExecutionException ee) { + logger.debug("Scheduler: plan task failed: " + + (ee.getCause() != null ? ee.getCause().getMessage() + : ee.getMessage())); + } + } + } catch (InterruptedException ie) { + Thread.currentThread().interrupt(); + return dispatched; + } + + // FOLDER CEILING (exact). planHost books up to job_frame_dispatch_max + // frames per (host,layer) from findNextDispatchFrames, which -- unlike the + // limit filter that lives inside that query -- has no folder clause. So one + // tick's batch can carry more of a capped folder's frames than + // folder_resource.int_max_cores allows; the candidate filter and the + // in-tick estimate bound what we PLAN, not what actually commits. Hold the + // ceiling on the frames about to commit, single-threaded: walk them in plan + // order and drop any that would push a capped folder past its cap. + // folderRunSeed is the folder's running cores at tick start (the resource + // flush is synchronous within a tick, so it is current); count committed + // cores up from there. Folders with no cap are never touched. + if (!folderMaxCp.isEmpty() && !planned.isEmpty()) { + Map folderCommit = new HashMap<>(folderRunSeed); + List keep = new ArrayList<>(planned.size()); + int folderTrimmed = 0; + for (FrameBooking b : planned) { + String fid = jobFolderCap.get(b.proc.getJobId()); + if (fid == null) { + keep.add(b); // folder has no cap + continue; + } + int cap = folderMaxCp.get(fid); + int used = folderCommit.getOrDefault(fid, 0); + int cp = b.proc.coresReserved; + if (used + cp <= cap) { + folderCommit.put(fid, used + cp); + keep.add(b); + } else { + folderTrimmed++; + } + } + if (folderTrimmed > 0) { + planned = keep; + logger.debug("Scheduler: folder ceiling trimmed " + folderTrimmed + + " planned frame(s) over cap this tick"); + } + } + + long tRead = System.currentTimeMillis(); + tickPlanned = planned.size(); + + // 4b. BATCH COMMIT: one transaction, batched frame UPDATE + proc INSERT + // + host UPDATE. Returns the winners (frames not lost to a version race). + List committed = planned.isEmpty() + ? java.util.Collections.emptyList() + : dispatchSupport.startFramesAndProcsBatch(planned); + long tCommit = System.currentTimeMillis(); + + // 4c. Accounting deltas from the winners, then flush one UPDATE per row. + if (batchResourceAccounting && !committed.isEmpty()) { + List procs = new ArrayList<>(committed.size()); + for (FrameBooking b : committed) procs.add(b.proc); + accumulateResourceDeltas(procs); + } + flushResourceDeltas(); + + // 4d. LAUNCH: fire the RQD launches post-commit on the launch pool. + for (FrameBooking b : committed) { + final FrameBooking fb = b; + launchPool.execute(() -> { + try { + dispatchSupport.runFrame(fb.proc, fb.frame); + } catch (RuntimeException e) { + // The commit already succeeded (frame RUNNING + proc), but the + // launch failed. Match the inline path's compensation so the + // slot is freed immediately instead of waiting for the reaper: + // unbook the proc, return the frame to WAITING, and kill it on + // RQD in case it actually started. This runs on the launch + // thread, so the tick is never blocked by it. + logger.warn("Scheduler: RQD launch failed for " + fb.proc.getName() + + " on frame " + fb.frame.getFrameId() + ": " + e.getMessage() + + ", unbooking and clearing frame"); + try { + dispatchSupport.unbookProc(fb.proc); + dispatchSupport.clearFrame(fb.frame); + rqdClient.killFrame(fb.proc, + "launch failed during scheduler dispatch"); + } catch (RuntimeException ce) { + // killFrame failing is expected when the frame never + // launched; the unbook/clear above already freed the slot. + logger.debug("Scheduler: launch-failure cleanup partial for " + + fb.frame.getFrameId() + ": " + ce.getMessage()); + } + } + }); + } + int dispatchedNow = committed.size(); + long tFlush = System.currentTimeMillis(); + if (tFlush - tStart > 1000) { + logger.info("Scheduler tick breakdown: place=" + (tPlan - tStart) + "ms, read=" + + (tRead - tPlan) + "ms, batchCommit=" + (tCommit - tRead) + "ms, flush+launch=" + + (tFlush - tCommit) + "ms | placements=" + placements + " planned=" + + planned.size() + " committed=" + dispatchedNow); + } + dispatched = dispatchedNow; + + // 5. SWEEP orphans + reservations.entrySet().removeIf(e -> !seenLayerIds.contains(e.getValue().layerId)); + // Drop blocked-debt state for layers that left the dispatchable set, + // so a layer that disappears and returns starts its block timer fresh. + blockedDebtMs.keySet().removeIf(id -> !seenLayerIds.contains(id)); + lastSeenMs.keySet().removeIf(id -> !seenLayerIds.contains(id)); + + return dispatched; + } + + // ---- leader lock ------------------------------------------------------ + + private boolean acquireLeaderLock(Connection conn) throws SQLException { + try (PreparedStatement ps = + conn.prepareStatement("SELECT pg_try_advisory_lock(?)")) { + ps.setLong(1, SCHEDULER_LOCK_KEY); + try (ResultSet rs = ps.executeQuery()) { + return rs.next() && rs.getBoolean(1); + } + } + } + + private void releaseLeaderLock(Connection conn) { + try (PreparedStatement ps = + conn.prepareStatement("SELECT pg_advisory_unlock(?)")) { + ps.setLong(1, SCHEDULER_LOCK_KEY); + ps.execute(); + } catch (SQLException e) { + // If the connection dropped, the backend session ended and the + // lock was released automatically. + logger.debug("Scheduler: pg_advisory_unlock failed (probably connection drop): " + + e.getMessage()); + } + } + + // ---- snapshot reads --------------------------------------------------- + + /** All schedulable hosts (UP + OPEN), busy or idle. The idle subset used + * for placement is derived per group in doTick(). */ + private List readAllHosts() { + return getJdbcTemplate().query(SELECT_ALL_HOSTS, HOST_MAPPER); + } + + /** + * Host->layer affinity: which layers currently have at least one proc + * booked on each host. Drives the locality bonus (see {@link #localityBonus}). + * A proc carries pk_layer directly, so no join is needed. A just-completed + * proc has been unbooked (row deleted), so only live placements appear. + */ + private Map> readHostLayerAffinity() { + Map> affinity = new HashMap<>(); + if (!localityEnabled) return affinity; + getJdbcTemplate().query( + "SELECT pk_host, pk_layer FROM proc WHERE pk_layer IS NOT NULL", + rs -> { + affinity.computeIfAbsent(rs.getString("pk_host"), k -> new HashSet<>()) + .add(rs.getString("pk_layer")); + }); + return affinity; + } + + private List readLayerCandidatesForGroup(HostSpecKey spec, + int maxIdleInGroup) { + int limit = env.getProperty("scheduler.layer_candidates_per_group_max", + Integer.class, 2000); + return getJdbcTemplate().query( + SELECT_CANDIDATES_FOR_GROUP, + CANDIDATE_MAPPER, + spec.pkAlloc, + spec.os, + spec.tagsNormalized, + maxIdleInGroup, + SchedulerMode.facility(env), + limit); + } + + /** + * EASY backfill deadline per reserved host: seconds from now until the host + * is projected to have freed enough cores for its reserving (wide) layer. + * + * For each reserved host, the running procs are projected to finish at + * {@code avg_layer_runtime - elapsed} (avg = layer_usage.int_clock_time_success + * / int_frame_success_count). Procs are taken soonest-first and their cores + * accumulated until the reserving layer's core deficit is covered; that + * proc's projected finish is the host's ready time. The deadline is the bar + * a borrowed frame must beat (see {@link #backfillAllows}). + * + * Conservative by construction: + * - A host whose needed procs lack a runtime estimate maps to + * {@link Integer#MAX_VALUE} ("unknown -> never backfill"). + * - Ready time uses the procs' AVERAGE finish, while the borrowed frame is + * bounded by its WORST case (int_clock_time_high). Requiring + * worst(frame) <= avg(host-ready) heavily biases against delaying the + * reserved job, since high is typically well above avg. + * + * Cores-only: cores are the binding dimension for the wide-job stranding + * this targets; memory/GPU readiness is not modelled here. + * + * Uses the snapshot's idle values, so it must run before the tick mutates + * them. Empty when backfill is off or there are no reservations. + */ + private Map computeHostReadySeconds(Map hostById) { + Map ready = new HashMap<>(); + if (!backfillEnabled || reservations.isEmpty()) return ready; + + List hostIds = new ArrayList<>(reservations.keySet()); + String in = hostIds.stream().map(x -> "?").collect(Collectors.joining(",")); + String sql = + "SELECT p.pk_host AS pk_host, p.int_cores_reserved AS cores, " + + "EXTRACT(EPOCH FROM (now() - p.ts_dispatched))::int AS elapsed_sec, " + + "CASE WHEN lu.int_frame_success_count > 0 " + + " THEN lu.int_clock_time_success / lu.int_frame_success_count " + + " ELSE -1 END AS avg_sec " + + "FROM proc p " + + "LEFT JOIN layer_usage lu ON lu.pk_layer = p.pk_layer " + + "WHERE p.pk_host IN (" + in + ")"; + + // Per host: list of {coresFreed, secondsUntilFree} for each running proc. + Map> procsByHost = new HashMap<>(); + getJdbcTemplate().query(sql, rs -> { + String hid = rs.getString("pk_host"); + int cores = rs.getInt("cores"); + int elapsed = rs.getInt("elapsed_sec"); + int avg = rs.getInt("avg_sec"); + int remaining = (avg < 0) ? Integer.MAX_VALUE : Math.max(0, avg - elapsed); + procsByHost.computeIfAbsent(hid, k -> new ArrayList<>()) + .add(new int[] {cores, remaining}); + }, hostIds.toArray()); + + for (Map.Entry e : reservations.entrySet()) { + String hid = e.getKey(); + BookableHost h = hostById.get(hid); + if (h == null) continue; // reserved host not in this tick's snapshot + int need = e.getValue().layerCoresMin - h.coresIdle; + ready.put(hid, hostReadySeconds(need, procsByHost.get(hid))); + } + return ready; + } + + // ---- grouping --------------------------------------------------------- + + static Map> groupByHostSpec(List hosts) { + Map> groups = new LinkedHashMap<>(); + for (BookableHost h : hosts) { + HostSpecKey k = new HostSpecKey( + h.pkAlloc, + // Cuebot auto-adds each host's own name as a tag. Drop it + // from the grouping key, otherwise every host falls into a + // group of one and the per-group candidate query runs once + // per host instead of once per real spec. + normalizeTags(h.tagsRaw, h.hostName), + h.os, + // GPU presence is a static hardware property: use totals, + // not idle. A fully-booked GPU host (gpusIdle == 0) must + // still group as a GPU host so its candidate query filters + // for GPU layers and the GPU-weighted score protects it. + h.gpusTotal > 0 || h.gpuMemTotal > 0); + groups.computeIfAbsent(k, x -> new ArrayList<>()).add(h); + } + return groups; + } + + /** + * Normalize a host's whitespace-separated tag string so equivalent + * sets ("linux desktop" and "desktop linux") group together. Duplicate + * tags are collapsed. + */ + static String normalizeTags(String raw) { + return normalizeTags(raw, null); + } + + /** + * As {@link #normalizeTags(String)}, but also drops any tag equal to + * {@code excludeName} (case-insensitive), used to strip a host's + * auto-added name tag so it doesn't fracture the grouping. + */ + static String normalizeTags(String raw, String excludeName) { + if (raw == null || raw.trim().isEmpty()) return ""; + return Arrays.stream(raw.trim().split("\\s+")) + .filter(t -> !t.equalsIgnoreCase(excludeName)) + .distinct() + .sorted() + .collect(Collectors.joining(" ")); + } + + // ---- placement: layer-driven, best-fit ------------------------------- + + /** + * Layer-driven placement with persistent reservations. For each + * candidate in priority order: + * + * 1. Dispatch loop: score every fitting host (respecting reservations) + * with {@link #placementScore} and pick the one with the lowest + * score. Dispatch via {@code dispatcher.dispatchHost(host, layer)}. + * If the chosen host carried a lower-priority reservation, override + * it to c. Loop until no fitting host, no waiting frames, or the + * job/show cap is reached. + * 2. Reconcile: c's reservation count should equal c.waitingFrameCount + * (decremented as we dispatched). Drop excess; claim more if short. + * + * Layer ids are recorded in {@code seenLayerIds} so the end-of-tick sweep + * can drop reservations for layers that left the dispatchable set. + */ + private int dispatchGroupWithScoring(List hosts, + List fullHosts, + List candidates, + Set seenLayerIds, + Map jobCoresUsed, + Map showCoresUsed, + Map limitUsed, + Map folderUsed, + List reservationReqs, + Map tReadyByHost, + Map> hostLayerAffinity) { + int dispatched = 0; + // Largest host in this group, for the reservation width gate below: a + // layer may reserve only if its per-frame cores are a big enough fraction + // of this. Uses fullHosts (idle + busy) so the bar reflects the class's + // real top-end capacity, not just what happens to be idle this tick. + int maxGroupHostCores = 0; + for (BookableHost h : fullHosts) { + if (h.coresTotal > maxGroupHostCores) maxGroupHostCores = h.coresTotal; + } + for (LayerCandidate c : candidates) { + seenLayerIds.add(c.layerId); + + // Sync this candidate's job/show usage with the tick-wide totals + // before any cap check: seed from the DB snapshot the first time + // a job/show is seen, then read back the accumulated value so + // earlier dispatches of the same job/show (here or in another + // group) count against this candidate's caps. + c.jobCoresInUse = jobCoresUsed.computeIfAbsent(c.jobId, k -> c.jobCoresInUse); + c.showCoresInUse = showCoresUsed.computeIfAbsent(c.showId, k -> c.showCoresInUse); + // Seed this limit's tick-wide running count from the farm-wide count + // the first time it is seen this tick (candidate query already + // excluded limits that were full at query time; this catches a limit + // filling DURING the tick as sibling layers book against it). + int limitInUse = (c.limitId != null) + ? limitUsed.computeIfAbsent(c.limitId, k -> c.limitRunning) : 0; + // Same for the folder core ceiling (cores, not frames). Only tracked + // when the folder actually has a cap (folderMax >= 0; -1 = unlimited). + int folderInUse = (c.folderMax >= 0) + ? folderUsed.computeIfAbsent(c.folderId, k -> c.folderRunning) : 0; + + // A layer at its job or show cap, whose limit (license cap) is full, or + // whose folder (group/dept) core ceiling is reached, cannot run any more + // frames this tick, so it must not dispatch, but it must still reconcile, + // which now drops the reservations it can no longer use (see + // reconcileReservationsForLayer). Skipping reconcile here would leave a + // capped layer holding hosts that lower-priority work could otherwise + // consume. + boolean capped = + c.jobCoresInUse + c.layerCoresMin > c.jobMaxCores + || c.showCoresInUse + c.layerCoresMin > c.showBurstCores + || (c.limitId != null && limitInUse >= c.limitMax) + || (c.folderMax >= 0 && folderInUse + c.layerCoresMin > c.folderMax); + + boolean placed = false; + while (!capped) { + BookableHost best = null; + double bestScore = Double.POSITIVE_INFINITY; + for (BookableHost h : hosts) { + if (!fitsOnHost(c, h)) continue; + // A host reserved for a higher/equal-priority blocked layer + // is normally off-limits. EASY backfill lets c borrow it + // anyway, but only if c's worst-case runtime finishes before + // the host is projected to free enough cores for its owner + // (backfillAllows), so the reserved (wide) job is never + // delayed. That time check IS the whole guarantee: a frame + // that ends by the host's ready time has vacated its cores + // before the owner needs them, so backfilling a still- + // draining host does not delay the drain. + // + // (Do NOT also gate on "host already has enough idle for the + // owner": that made backfill dead. Once idle covers the + // owner, the higher-priority owner books the host itself, so + // the only moment backfill can place anything is WHILE the + // host is still draining -- exactly what such a guard forbids, + // leaving the sub-owner-width idle on reserved hosts to + // strand instead of being backfilled.) + if (!reservationAllows(h, c)) { + if (!backfillAllows(h, c, tReadyByHost)) continue; + } + double score = placementScore(h, c); + // Locality bonus: prefer a host already running this layer so + // a freed core is refilled by the same layer (same-machine + // locality, formerly the reactive DispatchNextFrame path). + if (localityEnabled) { + Set layersHere = hostLayerAffinity.get(h.hostId); + if (layersHere != null && layersHere.contains(c.layerId)) { + score -= localityBonus; + } + } + if (score < bestScore) { + bestScore = score; + best = h; + } + } + if (best == null) break; // no host can fit this layer + + // Estimate how many frames this commit will book. The + // dispatcher books up to job_frame_dispatch_max per call, + // bounded by the same fit checks placementScore uses. + long maxMore = computeMaxMore(best, c); + int estFrames = (int) Math.min(jobFrameDispatchMax, maxMore + 1); + // Never dispatch more frames than the layer actually has + // waiting. computeMaxMore only bounds by host capacity and + // job/show caps, so without this a layer with a few waiting + // frames on a large host would emit commit after commit (up + // to the job cap) for frames that do not exist, inflating + // the dispatch estimate and padding the batch commit with + // bookings that find nothing. + if (estFrames > c.waitingFrameCount) estFrames = c.waitingFrameCount; + if (estFrames <= 0) break; + // Cap the commit to the limit's remaining headroom. The tick-wide + // count is authoritative: sibling layers of the same limit may + // already have booked against it this tick. If it is now full, + // stop booking this layer (its later frames would find nothing). + if (c.limitId != null) { + int limHeadroom = c.limitMax - limitUsed.get(c.limitId); + if (limHeadroom <= 0) break; + if (estFrames > limHeadroom) estFrames = limHeadroom; + } + // Cap the commit to the folder's remaining CORE headroom (this cap + // is in cores, not frames). If one more frame's cores won't fit, + // stop booking this layer this tick. + if (c.folderMax >= 0) { + int folderHeadroom = c.folderMax - folderUsed.get(c.folderId); + if (folderHeadroom < c.layerCoresMin) break; + int maxByFolder = folderHeadroom / c.layerCoresMin; + if (estFrames > maxByFolder) estFrames = maxByFolder; + } + + int estCores = estFrames * c.layerCoresMin; + long estMem = (long) estFrames * c.layerMemMin; + int estGpus = estFrames * c.layerGpusMin; + long estGpuMem = (long) estFrames * c.layerGpuMemMin; + + best.coresIdle -= estCores; + best.memIdle -= estMem; + best.gpusIdle -= estGpus; + best.gpuMemIdle -= estGpuMem; + c.jobCoresInUse += estCores; + c.showCoresInUse += estCores; + c.waitingFrameCount -= estFrames; + // Publish back so other candidates of the same job/show this + // tick see the updated usage. + jobCoresUsed.put(c.jobId, c.jobCoresInUse); + showCoresUsed.put(c.showId, c.showCoresInUse); + if (c.limitId != null) + limitUsed.merge(c.limitId, estFrames, Integer::sum); + if (c.folderMax >= 0) + folderUsed.merge(c.folderId, estCores, Integer::sum); + + // Count an EASY-backfill borrow for the stat line: c is committing + // onto a host reserved at >= its priority for another layer + // (reservationAllows == false), which only got past the scoring + // loop because backfillAllows cleared it. Checked before the + // ownership block below, which never fires for this case (it + // overrides only strictly-lower-priority reservations). + if (!reservationAllows(best, c)) { + tickBackfilled++; + tickBackfilledCores += estCores; + } + + // If the host carried a lower-priority reservation, + // take ownership. A reservation by c or by anyone equal + // or higher is preserved (the second case can't happen + // here because reservationAllows already excluded it). + Reservation existing = reservations.get(best.hostId); + if (existing != null && existing.priority < c.priority) { + reservations.put(best.hostId, + new Reservation(c.layerId, c.priority, c.layerCoresMin)); + } + + submitCommit(best.hostId, c.layerId); + dispatched += estFrames; + placed = true; + + // One commit per layer per tick. Fanning a layer onto many + // hosts in a single tick would make the parallel per-host plan + // reads (planHost) all re-query "the layer's next frames" and + // grab the SAME frames: mass frame.int_version collisions and + // lost placements. Booking one host now and the rest on later + // ticks keeps each layer's frames read by exactly one plan task; + // the next tick's fresh snapshot continues where this one left + // off. A layer spreads across hosts over a few ticks instead of + // all at once. + break; + } + + if (reservationsEnabled) { + // A layer is "blocked" this tick if it still has waiting frames, + // is not capped, and could not place even one (no host fit -> + // placed == false). Accumulate net blocked time (leaky bucket). + boolean blocked = !capped && !placed && c.waitingFrameCount > 0; + long now = System.currentTimeMillis(); + long dt = now - lastSeenMs.getOrDefault(c.layerId, now); + lastSeenMs.put(c.layerId, now); + long debt = blockedDebtMs.getOrDefault(c.layerId, 0L); + debt = blocked ? debt + dt : Math.max(0, debt - dt); + blockedDebtMs.put(c.layerId, debt); + + // Record a reservation request if the layer already holds + // reservations (keep maintaining them, need shrinks as frames + // place, until the job drains, so an in-progress big job is + // never re-stolen by the small-frame stream) OR it NEWLY qualifies: + // blocked past the time threshold AND wide enough to warrant a + // reservation. The width gate keeps the narrow small-frame stream + // (which runs the instant any core frees) from flooding the budget + // meant for wide, fragmentation-starved jobs. Granting happens + // after all groups, priority-first then widest (see the sort), so + // the scarce budget goes to the highest-priority widest work. + boolean wideEnough = maxGroupHostCores > 0 + && c.layerCoresMin >= RESERVATION_MIN_HOST_FRACTION * maxGroupHostCores; + boolean qualified = + blocked && debt >= reservationBlockMs && wideEnough; + boolean holdsResv = layerHoldsReservation(c.layerId); + if (holdsResv || qualified) { + reservationReqs.add(new ReservationRequest(c, fullHosts)); + } + // Trace reservation decisions for every candidate so we can + // see why wide-job layers never accumulate enough debt. + if (logger.isDebugEnabled()) { + logger.debug("Scheduler resv-candidate: layer=" + c.layerId + + " coresMin=" + c.layerCoresMin + + " waiting=" + c.waitingFrameCount + + " capped=" + capped + " placed=" + placed + " blocked=" + blocked + + " debt=" + debt + "ms threshold=" + reservationBlockMs + "ms" + + " wide=" + wideEnough + " (coresMin=" + c.layerCoresMin + + " gate=" + (RESERVATION_MIN_HOST_FRACTION * maxGroupHostCores) + ")" + + " holds=" + holdsResv + " qualifies=" + (holdsResv || qualified)); + } + } + } + return dispatched; + } + + /** True if any host is currently reserved for this layer. */ + private boolean layerHoldsReservation(String layerId) { + for (Reservation r : reservations.values()) { + if (r.layerId.equals(layerId)) return true; + } + return false; + } + + /** Whether host h has enough TOTAL capacity to run a frame of c when idle. */ + private static boolean hostCanEverFit(LayerCandidate c, BookableHost h) { + return h.coresTotal >= c.layerCoresMin + && h.memTotal >= c.layerMemMin + && h.gpusTotal >= c.layerGpusMin + && h.gpuMemTotal >= c.layerGpuMemMin; + } + + /** How many frames of c fit on a fully-idle host h (min over dimensions). */ + private static int framesThatFit(LayerCandidate c, BookableHost h) { + long f = Long.MAX_VALUE; + if (c.layerCoresMin > 0) f = Math.min(f, h.coresTotal / c.layerCoresMin); + if (c.layerMemMin > 0) f = Math.min(f, h.memTotal / c.layerMemMin); + if (c.layerGpusMin > 0) f = Math.min(f, h.gpusTotal / c.layerGpusMin); + if (c.layerGpuMemMin > 0) f = Math.min(f, h.gpuMemTotal / c.layerGpuMemMin); + return (f == Long.MAX_VALUE) ? 1 : (int) f; // unconstrained -> 1 + } + + /** + * Frames of c that fit on one reservation-eligible host, taken as the + * MINIMUM across fitting hosts so we never under-reserve (the dangerous + * direction, too few reserved hosts and the layer stays starved). + * Returns 0 when no host can fit c (caller then falls back to a per-frame + * count). + */ + private static int framesPerFittingHost(LayerCandidate c, List hosts) { + int min = 0; + for (BookableHost h : hosts) { + if (!hostCanEverFit(c, h)) continue; + int f = framesThatFit(c, h); + if (f > 0 && (min == 0 || f < min)) min = f; + } + return min; + } + + /** + * A host's reservation lets c through if there is no reservation, the + * reservation belongs to c, or the existing reservation is strictly + * lower priority (in which case c may override on successful dispatch). + */ + private boolean reservationAllows(BookableHost h, LayerCandidate c) { + if (!reservationsEnabled) return true; + Reservation r = reservations.get(h.hostId); + return r == null + || r.layerId.equals(c.layerId) + || r.priority < c.priority; + } + + /** + * EASY backfill (Lifka 1995): may c borrow reserved host h without delaying + * its owner? Yes iff c's worst-case runtime + * (layer_usage.int_clock_time_high) finishes before h is projected to free + * enough cores for its reserving layer ({@link #computeHostReadySeconds}). + * Borrowing never takes ownership, the dispatch loop's override only fires + * for strictly-lower-priority reservations, which this host is not. + * + * Refuses to backfill when c has no runtime history (cannot bound its + * occupancy) or the host's ready time is unknown, keeping the no-delay + * guarantee conservative under soft (non-killed) estimates. + */ + private boolean backfillAllows(BookableHost h, LayerCandidate c, + Map tReadyByHost) { + if (!backfillEnabled) return false; + Integer tReady = tReadyByHost.get(h.hostId); + if (tReady == null) return false; + return backfillFits(c.hasRuntimeEstimate(), c.clockTimeHighSec, tReady); + } + + /** + * The EASY no-delay test, factored out for testing: a frame may backfill iff + * its layer has a runtime estimate, the host's ready time is known (not + * {@link Integer#MAX_VALUE}), and the frame's worst-case runtime finishes at + * or before that ready time. + */ + static boolean backfillFits(boolean hasEstimate, int clockTimeHighSec, + int tReadySeconds) { + if (!hasEstimate) return false; + if (tReadySeconds == Integer.MAX_VALUE) return false; + return clockTimeHighSec <= tReadySeconds; + } + + /** + * Seconds until a host frees {@code needCores} cores, given its running + * procs as {@code {coresFreed, secondsUntilFree}} pairs. Procs finish + * soonest-first; the crossing proc's time is the answer. Returns 0 when no + * cores are needed and {@link Integer#MAX_VALUE} when the cores can never be + * freed from the known procs (too few, or a needed proc has an unknown + * finish time, encoded as {@link Integer#MAX_VALUE}). Factored out of + * {@link #computeHostReadySeconds} for testing. + */ + static int hostReadySeconds(int needCores, List procs) { + if (needCores <= 0) return 0; + if (procs == null || procs.isEmpty()) return Integer.MAX_VALUE; + List sorted = new ArrayList<>(procs); + sorted.sort(Comparator.comparingInt(p -> p[1])); + int freed = 0; + for (int[] p : sorted) { + if (p[1] == Integer.MAX_VALUE) break; // unknown proc needed + freed += p[0]; + if (freed >= needCores) return p[1]; + } + return Integer.MAX_VALUE; + } + + /** + * Ensure c holds the right number of reservations. The target count is: + * + * 1. Frames the layer could still run: waiting frames, clamped by the job + * and show core caps (a capped layer must not hold hosts it cannot + * legally use, which would block lower-priority work). + * 2. Made CAPACITY-AWARE: a fitting host runs several frames of the layer + * (e.g. two 64-core frames on a 128-core host), so the number of HOSTS + * needed is frames / frames-per-host, not one host per frame. + * 3. Bounded by the per-class CAP: reservations (this layer's plus any + * already held by others on hosts that fit c) may cover at most + * reservationMaxFraction of the hosts that can fit c, so the class can + * never be fully reserved. Callers grant priority-first then + * widest-job, so the cap fills for high-priority wide jobs before + * narrow ones are considered. + * + * Drops excess reservations (frames dispatched, layer capped, or cap + * shrunk) or claims more via {@link #pickReservationTarget}. + */ + private void reconcileReservationsForLayer(LayerCandidate c, List hosts) { + List mine = new ArrayList<>(); + for (Map.Entry e : reservations.entrySet()) { + if (e.getValue().layerId.equals(c.layerId)) mine.add(e.getKey()); + } + + int have = mine.size(); + int framesNeed = Math.max(0, c.waitingFrameCount); + if (c.layerCoresMin > 0) { + int jobFramesLeft = Math.max(0, + (c.jobMaxCores - c.jobCoresInUse) / c.layerCoresMin); + int showFramesLeft = Math.max(0, + (c.showBurstCores - c.showCoresInUse) / c.layerCoresMin); + framesNeed = Math.min(framesNeed, Math.min(jobFramesLeft, showFramesLeft)); + } + + // Capacity-aware: how many hosts does framesNeed actually require? + // Use the smallest fitting host's capacity so we never under-reserve. + int framesPerHost = framesPerFittingHost(c, hosts); + int need = framesPerHost > 0 + ? (framesNeed + framesPerHost - 1) / framesPerHost // ceil + : framesNeed; + + // Per-class cap: at most reservationMaxFraction of the hosts that can + // fit c may be reserved at once. Count fitting hosts and how many of + // them are already reserved by OTHER layers; c may use the remainder. + int fittingTotal = 0, reservedByOthers = 0; + for (BookableHost h : hosts) { + if (!hostCanEverFit(c, h)) continue; + fittingTotal++; + Reservation r = reservations.get(h.hostId); + if (r != null && !r.layerId.equals(c.layerId)) reservedByOthers++; + } + int capTotal = (int) Math.floor(reservationMaxFraction * fittingTotal); + int capForC = Math.max(0, capTotal - reservedByOthers); + need = Math.min(need, capForC); + + if (have > need) { + for (String hostId : mine.subList(need, have)) { + reservations.remove(hostId); + } + } else if (have < need) { + int want = need - have; + for (int i = 0; i < want; i++) { + BookableHost t = pickReservationTarget(c, hosts); + if (t == null) break; // no more eligible host + Reservation existing = reservations.get(t.hostId); + if (existing != null && existing.priority < c.priority) { + logger.info("Scheduler: override reservation host=" + t.hostName + + " layer=" + existing.layerId + "(p=" + existing.priority + + ") -> " + c.layerId + "(p=" + c.priority + ")"); + } + reservations.put(t.hostId, + new Reservation(c.layerId, c.priority, c.layerCoresMin)); + } + } + } + + /** + * Pick the host most likely to become available for c soonest, expressed + * as "host with the fewest running procs": fewer running frames means + * fewer to wait on before the host frees up enough cores for c. The + * host must (a) be tag/OS-compatible (granted by group membership), + * (b) have enough TOTAL capacity for c when fully idle, (c) not already + * be reserved by c (reconcile only expands the set, never re-claims), + * and (d) not be reserved at equal or higher priority for a different + * layer. + */ + private BookableHost pickReservationTarget(LayerCandidate c, List hosts) { + BookableHost best = null; + int bestProcs = Integer.MAX_VALUE; + for (BookableHost h : hosts) { + if (!hostCanEverFit(c, h)) continue; + Reservation r = reservations.get(h.hostId); + // Skip hosts c already owns: they are counted in 'have' by + // reconcileReservationsForLayer and must not be re-claimed. + // Without this check, reservationAllows returns true for + // c's own reservations and the loop re-picks the same best + // host on every iteration, never actually expanding the set. + if (r != null && r.layerId.equals(c.layerId)) continue; + // Skip hosts held by an equal- or higher-priority layer. + if (r != null && r.priority >= c.priority) continue; + if (h.runningProcs < bestProcs) { + bestProcs = h.runningProcs; + best = h; + } + } + return best; + } + + /** + * Placement score for a (host, layer) pair. Lower is better. Callers MUST + * call {@link #fitsOnHost} first; this function assumes the layer fits. + * + * Real E-PVM (after Amir, Awerbuch, Barak, Borgstrom & Keren 2000): + * the farm carries a convex "cost" potential, summed over every host and + * every resource dimension D: + * + * C = sum_hosts sum_D e^( used_D / total_D ) + * + * Placing a frame on host h raises only h's usage, so the marginal cost + * of accepting it, the score, is the rise in that one host's terms: + * + * score(h) = sum_D W_D * ( e^(after_D/total_D) - e^(before_D/total_D) ) + * before_D = total_D - idle_D (currently reserved) + * after_D = before_D + layer.min_D (with this frame added) + * + * We pick the host with the smallest score (argmin of the marginal cost). + * E-PVM is a load-BALANCING heuristic, not a bin-packing one, and three + * properties fall out of the convex, capacity-normalized form, each + * fixing a failure of the old absolute-stranding score: + * + * - Proportional balancing. Because the exponent is used/total, the same + * frame is a smaller fraction of a large host, so a fresh big host has + * a lower marginal cost than a fresh small one and is filled first -- + * but only until its fraction catches up. Under a sustained backlog + * e^x's convexity drives every host toward the SAME fractional + * utilization, so a 128-core host ends up carrying ~8x the frames of a + * 16-core host instead of sitting idle. The old absolute-stranding + * score did the opposite: it consolidated memory-light work onto small + * hosts and left ~55% of the farm's cores (the big hosts) unused. + * + * - Size neutrality of the steady state. The cost is a fraction, so + * "balanced" means equal utilization PERCENT across heterogeneous + * hosts, not equal frame counts. That is the right target for a farm + * whose goal is to keep all hardware busy. + * + * - Memory-bound hosts read as full. A host with idle cores but + * saturated memory sits at e^(~1.0) on the memory axis; its marginal + * cost there is enormous, so it stops attracting work even though + * cores look free. The old linear score could not see this and would + * keep stranding cores behind full memory. + * + * This is a one-step lookahead: "after" is the state with just THIS frame + * added, not an end-of-tick projection. We do not need computeMaxMore's + * pile-up estimate here, the dispatch loop decrements h.*Idle after each + * commit, so the next frame in the tick sees a higher "before" and a + * steeper delta automatically (convexity handles the pile-up). + * + * Units cancel: used_D and total_D are in the same units per dimension + * (core points, KB, count), so the exponent is dimensionless and no + * per-dimension unit conversion is needed. The W_D weights set the + * relative importance of the dimensions on that common, dimensionless + * scale. + */ + static double placementScore(BookableHost h, LayerCandidate c) { + return W_CORES * deltaCost(h.coresTotal, h.coresIdle, c.layerCoresMin) + + W_MEM * deltaCost(h.memTotal, h.memIdle, c.layerMemMin) + + W_GPUS * deltaCost(h.gpusTotal, h.gpusIdle, c.layerGpusMin) + + W_GPU_MEM * deltaCost(h.gpuMemTotal, h.gpuMemIdle, c.layerGpuMemMin); + } + + /** + * Marginal rise of one resource dimension's convex cost term when a + * reservation of {@code add} is placed on a host that has {@code idle} + * free out of {@code total}. Returns 0 when the layer does not use the + * dimension (add <= 0) or the host has no capacity there. + */ + private static double deltaCost(double total, double idle, double add) { + if (add <= 0 || total <= 0) return 0; + double usedBefore = total - idle; + double usedAfter = usedBefore + add; + return Math.exp(usedAfter / total) - Math.exp(usedBefore / total); + } + + /** + * Predict the number of ADDITIONAL frames of c (beyond the first) that + * could be dispatched to h within this tick. Shared by placementScore + * (which uses it to compute stranding) and the dispatch loop (which + * uses it to estimate the frames a single commit will book). + * + * Caps applied (mirroring the dispatcher's per-frame fit checks): + * - physical fit on each dimension + * - job int_max_cores (matches isJobBookable) + * - show int_burst (matches isShowAtOrOverBurst) + * + * Per-call caps host_frame_dispatch_max and job_frame_dispatch_max + * are NOT applied here because they bound a single dispatch CALL, not + * the per-tick total. The dispatch loop applies job_frame_dispatch_max + * when estimating a single commit's worth of frames. + */ + static long computeMaxMore(BookableHost h, LayerCandidate c) { + long remCores = h.coresIdle - c.layerCoresMin; + long remMem = h.memIdle - c.layerMemMin; + long remGpus = h.gpusIdle - c.layerGpusMin; + long remGpuMem = h.gpuMemIdle - c.layerGpuMemMin; + + long maxMore = Long.MAX_VALUE; + if (c.layerCoresMin > 0) maxMore = Math.min(maxMore, remCores / c.layerCoresMin); + if (c.layerMemMin > 0) maxMore = Math.min(maxMore, remMem / c.layerMemMin); + if (c.layerGpusMin > 0) maxMore = Math.min(maxMore, remGpus / c.layerGpusMin); + if (c.layerGpuMemMin > 0) maxMore = Math.min(maxMore, remGpuMem / c.layerGpuMemMin); + + if (c.layerCoresMin > 0) { + long jobRem = (long) c.jobMaxCores - c.jobCoresInUse - c.layerCoresMin; + if (jobRem < 0) jobRem = 0; + maxMore = Math.min(maxMore, jobRem / c.layerCoresMin); + } + if (c.layerCoresMin > 0) { + long showRem = (long) c.showBurstCores - c.showCoresInUse - c.layerCoresMin; + if (showRem < 0) showRem = 0; + maxMore = Math.min(maxMore, showRem / c.layerCoresMin); + } + if (maxMore == Long.MAX_VALUE) maxMore = 0; + return maxMore; + } + + // ---- plan / batch-commit: submission ---------------------------------- + + /** + * Record a (host, layer) placement to commit at the end of this tick. + * Planner-thread only; doTick drains plannedByHost via planHost + + * startFramesAndProcsBatch. + */ + private void submitCommit(String hostId, String layerId) { + plannedByHost.computeIfAbsent(hostId, k -> new ArrayList<>()).add(layerId); + } + + // ---- batched resource accounting: accumulate + flush ------------------ + + /** + * Record the resource deltas for the procs the batch commit just booked. + * Called on the planner thread right after startFramesAndProcsBatch. The + * passed list is the batch's winners (only successfully committed procs), so + * rolled-back bookings are never counted. Local dispatches keep their own + * accounting path and are skipped here. + */ + private void accumulateResourceDeltas(List procs) { + if (procs == null || procs.isEmpty()) { + return; + } + for (VirtualProc p : procs) { + if (p.isLocalDispatch) { + continue; + } + long cores = p.coresReserved; + long gpus = p.gpusReserved; + addDelta(subDeltas, p.getShowId() + "\t" + p.getAllocationId(), cores, gpus); + addDelta(layerDeltas, p.getLayerId(), cores, gpus); + addDelta(jobDeltas, p.getJobId(), cores, gpus); + } + } + + private static void addDelta(Map buf, String key, long cores, long gpus) { + buf.merge(key, new long[] {cores, gpus}, (a, b) -> { + a[0] += b[0]; + a[1] += b[1]; + return a; + }); + } + + /** + * Apply this tick's accumulated resource deltas as one UPDATE per row. + * Runs on the planner thread right after the batch commit, so no + * accumulation races it. On a SQL error the deltas are merged back so the + * next tick retries them rather than silently dropping accounting. + * Subscription/layer rows missing (deleted mid-tick) simply update zero + * rows; folder/point use the job subquery and likewise no-op if the job is + * gone. + */ + private void flushResourceDeltas() { + if (!batchResourceAccounting) { + return; + } + flushSubDeltas(); + flushLayerDeltas(); + flushJobDeltas(); + } + + private void flushSubDeltas() { + if (subDeltas.isEmpty()) { + return; + } + Map snap = drain(subDeltas); + List batch = new ArrayList<>(snap.size()); + for (Map.Entry e : snap.entrySet()) { + String[] k = e.getKey().split("\t", 2); + long[] d = e.getValue(); + batch.add(new Object[] {(int) d[0], (int) d[1], k[0], k[1]}); + } + try { + txTemplate().execute(status -> { + getJdbcTemplate().batchUpdate( + "UPDATE subscription SET int_cores = int_cores + ?, " + + "int_gpus = int_gpus + ? WHERE pk_show = ? AND pk_alloc = ?", + batch); + return null; + }); + } catch (RuntimeException ex) { + logger.warn("Scheduler: subscription delta flush failed, retrying next tick: " + + ex.getMessage()); + for (Map.Entry e : snap.entrySet()) { + addDelta(subDeltas, e.getKey(), e.getValue()[0], e.getValue()[1]); + } + } + } + + private void flushLayerDeltas() { + if (layerDeltas.isEmpty()) { + return; + } + Map snap = drain(layerDeltas); + List batch = new ArrayList<>(snap.size()); + for (Map.Entry e : snap.entrySet()) { + long[] d = e.getValue(); + batch.add(new Object[] {(int) d[0], (int) d[1], e.getKey()}); + } + try { + txTemplate().execute(status -> { + getJdbcTemplate().batchUpdate( + "UPDATE layer_resource SET int_cores = int_cores + ?, " + + "int_gpus = int_gpus + ? WHERE pk_layer = ?", + batch); + return null; + }); + } catch (RuntimeException ex) { + logger.warn("Scheduler: layer_resource delta flush failed, retrying next tick: " + + ex.getMessage()); + for (Map.Entry e : snap.entrySet()) { + addDelta(layerDeltas, e.getKey(), e.getValue()[0], e.getValue()[1]); + } + } + } + + private void flushJobDeltas() { + if (jobDeltas.isEmpty()) { + return; + } + Map snap = drain(jobDeltas); + List jobBatch = new ArrayList<>(snap.size()); + List pointBatch = new ArrayList<>(snap.size()); + for (Map.Entry e : snap.entrySet()) { + long[] d = e.getValue(); + int cores = (int) d[0]; + int gpus = (int) d[1]; + String jobId = e.getKey(); + jobBatch.add(new Object[] {cores, gpus, jobId}); + pointBatch.add(new Object[] {cores, gpus, jobId, jobId}); + } + try { + // One transaction for all three UPDATEs: on a mid-flush error the whole + // set rolls back, so the retry (which re-queues the drained deltas) can + // never double-apply a sub-batch that had already committed. + txTemplate().execute(status -> { + getJdbcTemplate().batchUpdate( + "UPDATE job_resource SET int_cores = int_cores + ?, " + + "int_gpus = int_gpus + ? WHERE pk_job = ?", + jobBatch); + getJdbcTemplate().batchUpdate( + "UPDATE folder_resource SET int_cores = int_cores + ?, " + + "int_gpus = int_gpus + ? " + + "WHERE pk_folder = (SELECT pk_folder FROM job WHERE pk_job = ?)", + jobBatch); + getJdbcTemplate().batchUpdate( + "UPDATE point SET int_cores = int_cores + ?, int_gpus = int_gpus + ? " + + "WHERE pk_dept = (SELECT pk_dept FROM job WHERE pk_job = ?) " + + "AND pk_show = (SELECT pk_show FROM job WHERE pk_job = ?)", + pointBatch); + return null; + }); + } catch (RuntimeException ex) { + logger.warn("Scheduler: job/folder/point delta flush failed, retrying next tick: " + + ex.getMessage()); + for (Map.Entry e : snap.entrySet()) { + addDelta(jobDeltas, e.getKey(), e.getValue()[0], e.getValue()[1]); + } + } + } + + /** Copy out the current deltas and clear the buffer for the next tick. */ + private static Map drain(Map buf) { + Map snap = new HashMap<>(); + for (Iterator> it = buf.entrySet().iterator(); it.hasNext();) { + Map.Entry e = it.next(); + snap.put(e.getKey(), e.getValue()); + it.remove(); + } + return snap; + } + + static boolean fitsOnHost(LayerCandidate c, BookableHost h) { + if (h.coresIdle < c.layerCoresMin) return false; + if (h.memIdle < c.layerMemMin) return false; + if (h.gpusIdle < c.layerGpusMin) return false; + if (h.gpuMemIdle < c.layerGpuMemMin) return false; + return true; + } + + // ---- config ----------------------------------------------------------- + + private boolean isEnabled() { + return SchedulerMode.enabled(env); + } + + // ---- POJOs ------------------------------------------------------------ + + static final class BookableHost { + String hostId; + String hostName; + String pkAlloc; + // Total capacity. Used by pickReservationTarget to check whether the + // host could fit a layer when fully idle, independent of the host's + // current load. + int coresTotal; + long memTotal; + int gpusTotal; + long gpuMemTotal; + // Current idle resources. Decremented as we dispatch within a tick. + int coresIdle; + long memIdle; + int gpusIdle; + long gpuMemIdle; + // Current running proc count. Used as the "soonest-to-free" heuristic + // for reservation target selection. + int runningProcs; + String tagsRaw; + String os; + } + + static final class LayerCandidate { + String layerId; + String jobId; + String showId; + int layerCoresMin; + long layerMemMin; + int layerGpusMin; + long layerGpuMemMin; + int priority; + // Mutable in-tick accounting. + int jobCoresInUse; + int jobMaxCores; + int showCoresInUse; + int showBurstCores; + // Number of pending unfittable frames. Initialized from + // waiting_frame_count in the candidate query; decremented as the + // layer dispatches in this tick. Reconcile keeps the layer's + // reservation count equal to this value. + int waitingFrameCount; + // EASY-backfill runtime estimate (from layer_usage). clockTimeHighSec + // is the worst single-frame wall-clock time ever recorded for the + // layer; frameSuccessCount is how many successful frames produced it. + // Both 0 when the layer has no history. Used only as the conservative + // upper bound on how long a frame of this layer would occupy a host it + // backfills onto (see backfillAllows). + int clockTimeHighSec; + int frameSuccessCount; + // Limit (license-cap) accounting. limitId is the layer's most-constraining + // limit (null = no limit); limitMax is that limit's int_max_value; + // limitRunning is how many frames of it run farm-wide right now (seed for + // the tick-wide limitUsed cap in dispatchGroupWithScoring). + String limitId; + int limitMax; + int limitRunning; + // Folder (group/dept) core cap. folderId is the job's folder; folderMax is + // folder_resource.int_max_cores (-1 = unlimited, core-points); folderRunning + // is the folder's current running cores (core-points), seed for the + // tick-wide folderUsed cap in dispatchGroupWithScoring. + String folderId; + int folderMax; + int folderRunning; + + /** Whether the layer has enough history to bound a frame's runtime. */ + boolean hasRuntimeEstimate() { + return frameSuccessCount > 0 && clockTimeHighSec > 0; + } + } + + /** + * A claim on a host by a specific layer at a specific priority. Stored + * by host id. Persistent across ticks. The priority is what the override + * comparison uses; storing it on the reservation (rather than looking it + * up from the current candidate set) means an override decision works + * even when the owner layer doesn't appear in the current group's + * candidates. + */ + static final class Reservation { + final String layerId; + final int priority; + // The reserving layer's per-frame core requirement, so a host's + // earliest-ready time can be estimated (how many running procs must + // finish to free this many cores) without re-finding the owner layer. + final int layerCoresMin; + Reservation(String layerId, int priority, int layerCoresMin) { + this.layerId = layerId; + this.priority = priority; + this.layerCoresMin = layerCoresMin; + } + } + + + /** + * A layer that wants reservations this tick, paired with the full host set + * of its group. Collected during placement and processed after all groups, + * sorted priority-first then widest-job, so the capped reservation budget is + * granted to the highest-priority work that cannot fit (wide jobs) rather + * than to the oldest layer. + */ + static final class ReservationRequest { + final LayerCandidate candidate; + final List fullHosts; + ReservationRequest(LayerCandidate candidate, List fullHosts) { + this.candidate = candidate; + this.fullHosts = fullHosts; + } + } + + static final class HostSpecKey { + final String pkAlloc; + final String tagsNormalized; + final String os; + final boolean hasGpu; + + HostSpecKey(String pkAlloc, String tagsNormalized, String os, boolean hasGpu) { + this.pkAlloc = pkAlloc; + this.tagsNormalized = tagsNormalized; + this.os = os; + this.hasGpu = hasGpu; + } + + @Override public boolean equals(Object o) { + if (this == o) return true; + if (!(o instanceof HostSpecKey)) return false; + HostSpecKey k = (HostSpecKey) o; + return hasGpu == k.hasGpu + && Objects.equals(pkAlloc, k.pkAlloc) + && Objects.equals(tagsNormalized, k.tagsNormalized) + && Objects.equals(os, k.os); + } + + @Override public int hashCode() { + return Objects.hash(pkAlloc, tagsNormalized, os, hasGpu); + } + + @Override public String toString() { + return "HostSpec(alloc=" + pkAlloc + + ", tags=" + tagsNormalized + + ", os=" + os + + ", gpu=" + hasGpu + ")"; + } + } + + // ---- Spring setters --------------------------------------------------- + + public void setDispatcher(Dispatcher d) { this.dispatcher = d; } + public void setDispatchSupport(DispatchSupport d) { this.dispatchSupport = d; } + public void setHostManager(HostManager m) { this.hostManager = m; } + public void setJobManager(JobManager m) { this.jobManager = m; } + public void setRqdClient(RqdClient r) { this.rqdClient = r; } +} diff --git a/cuebot/src/main/java/com/imageworks/spcue/dispatcher/Scheduler.md b/cuebot/src/main/java/com/imageworks/spcue/dispatcher/Scheduler.md new file mode 100644 index 0000000000..fae9ed452f --- /dev/null +++ b/cuebot/src/main/java/com/imageworks/spcue/dispatcher/Scheduler.md @@ -0,0 +1,559 @@ +# Scheduler (Planner) + +A whole-farm scheduler for OpenCue, gated behind `scheduler.enabled` +(default **off**). It is an alternative to the legacy per-host dispatcher. +When enabled it owns dispatch and the legacy `BookingQueue` path is +suppressed. Placement decisions are made single-threaded over an in-memory +snapshot; only the per-host plan reads run in parallel (see section 4). + +The keystone of the design is that the planner is **stateless between ticks**: +each tick re-derives its entire picture from a fresh database snapshot and keeps +no booking state across ticks — the database is the single source of truth. +Fire-and-forget launches, instant failover, and self-healing after a crash all +fall out of that one decision (see the keystone note in section 2, and section 4). + +This document explains what it does, how it works, the concurrency model +that keeps it correct, how to configure it, and what is planned next. + +--- + +## 1. Background: dispatcher vs scheduler + +The legacy path is a **dispatcher**. When a host reports in, it runs +`findDispatchJobs(host)` (a heavy multi-table join) and books the first +frame that fits, in priority order, one host at a time. It never sees more +than one host at once, so it cannot reason across the farm: it cannot hold +a big machine open for a wide job that is queued, or steer a small frame +onto a small machine instead of wasting a big one. + +In practice that gap has been filled outside Cuebot, by people: operators +hand-tune each layer's core request and tags ahead of time so the naive +dispatcher behaves. That works, but it means humans do most of the real +scheduling, following a hand-maintained, per-show rulebook. + +A **scheduler** makes those decisions itself. It takes a snapshot of the +whole farm each cycle, scores every candidate placement by how much +capacity it would strand, holds reservations for work that would otherwise +starve, and books accordingly. That is what this component does. + +--- + +## 2. Architecture: the tick loop + +The scheduler runs a periodic tick (`runTick` → `doTick`). Each tick: + +1. **Snapshot**: read all bookable hosts in one query (`readBookableHosts`, + `SELECT_BOOKABLE_HOSTS`). Hosts that are UP, OPEN, and have at least the + minimum bookable cores. +2. **Group**: bucket hosts by spec key `(alloc, normalized_tags, os, + has_gpu)` (`groupByHostSpec`). On a homogeneous farm this is a handful of + groups, which is what collapses the per-host query storm into a few + queries per tick. +3. **For each group:** + 1. **Candidate query**: one query per group + (`readLayerCandidatesForGroup`, `SELECT_CANDIDATES_FOR_GROUP`) for the + dispatchable layers that match the group, ranked by a **priority-weighted + lottery** (§3.5), not a strict priority sort. + 2. **Dispatch** (`dispatchGroupWithScoring`): for each candidate in that + lottery order, score every fitting host, pick the lowest score, record the + placement, and decrement the in-memory snapshot. A candidate that stays + blocked long enough and is wide enough records a reservation *request*. +4. **Grant reservations**: after all groups, process the requests + highest-priority-then-widest and reconcile each grantee's reservation count + under the per-class and max-grantees caps (section 3.2). +5. **Commit**: read each recorded placement's frames in parallel by host + (`planHost`, read-only), write them all in one batched transaction + (`startFramesAndProcsBatch`), then fire the RQD launches fire-and-forget. +6. **Sweep**: drop reservations whose layer no longer appears in any + candidate set. + +Steps 1-4 run single-threaded, so the decisions never race. The only +parallelism is in step 5's plan-phase reads (one task per host); the write is a +single batched commit and the launches fire afterward fire-and-forget. + +### The keystone: stateless between ticks + +The planner holds **no durable booking state**. Each tick rebuilds its world from +the fresh host snapshot (step 1) and the per-group candidate queries (step 3); the +only thing carried across ticks is the soft reservation map, and even that is just +a hint the planner rebuilds from the database within a tick or two. **The database +is the single source of truth.** Three properties fall out of that one decision — +and they are why the rest of the design stays simple: + +- **Fire-and-forget launches.** The launch outcome never feeds back into planning + state, so the tick never waits on RQD. A dropped or lost launch leaves a frame + RUNNING in the DB that RQD never received; the orphaned-proc reaper resets it and + the *next* snapshot re-reads the corrected state. Launch latency never gates + booking (sections 4 and 7). +- **Stateless failover.** The leader keeps only the advisory lock and the in-memory + reservation hint. If it dies, the next Cuebot takes the lock and reconstructs an + identical picture from the DB within a tick or two — nothing to persist, migrate, + or replay. +- **Self-healing.** Any transient inconsistency — a partial commit, a lost launch, + snapshot drift — is erased by the next snapshot. Failures bias toward + *under*-booking for one tick, never over-booking or lost work. + +The cost is the full host snapshot every tick (the dominant read load, section 5). +That is the deliberate trade: **pay a re-read each tick in order to own no state.** + +### Operational footprint + +Owning no durable state keeps the planner small and cheap to run. It lives +**inside Cuebot** — no new deploy unit, no separate service to run or fail over, +no new infrastructure, and no schema migrations. It is gated by a single flag +(`scheduler.enabled`) and rolled back by flipping that flag off. The whole thing +is ~3,700 lines, almost all in four new files (the planner, a value type, this +doc, and a test), with ~16 existing files lightly touched. There is nowhere to +keep live state, no process to run it, and nothing to rebuild on failover — the +database it already uses is the only state there is. + +--- + +## 3. Components + +### 3.1 Placement score (multi-resource E-PVM) + +`placementScore(host, layer)`, lower is better. It is the **marginal cost** +of placing one frame on a host, under a convex potential summed over every +host and every resource dimension: + +``` +C = sum_hosts sum_D e^( used_D / total_D ) +score(h) = sum_D W_D * ( e^(after_D/total_D) - e^(before_D/total_D) ) + before_D = total_D - idle_D (currently reserved) + after_D = before_D + layer.min_D (with this frame added) +``` + +We pick the host with the smallest score. Because the exponent is the +**utilization fraction** `used_D/total_D`, the score is dimensionless and +free of host-size bias, and the convex `e^x` makes a dimension that is +already near-full (e.g. a host with idle cores but saturated memory) cost a +great deal more to load further. Default weights: `W_CORES=1`, `W_MEM=1`, +`W_GPUS=4`, `W_GPU_MEM=1`, cores and memory equal; GPUs weighted higher so a +GPU layer strands the least GPU capacity. + +This is **E-PVM / opportunity cost** (Amir, Awerbuch, Barak, Borgström & +Keren 2000; multi-resource in Verma et al., Borg, EuroSys 2015). It is a +load-**balancing** heuristic, not bin-packing: the same frame is a smaller +fraction of a larger host, so big hosts are filled first but only until their +fraction catches up. Under a sustained backlog this drives every host toward +the **same utilization percentage**, so a 128-core host carries ~8× the +frames of a 16-core host instead of sitting idle. + +**Co-locality.** The score also carries a locality bonus. The legacy reactive +path rebooked the next frame of a job onto the same proc the instant a frame +finished, keeping a job's frames together on a machine (cache coherence: +textures, geometry, KSM-shared pages, warm filesystem cache). The planner +unbooks a completing proc instead, so to preserve that it subtracts +`scheduler.locality_bonus` (default 8.0) from a host's score when the host +already runs at least one frame of the candidate's layer, read once per tick +as a host->layer affinity map (`readHostLayerAffinity`). A freed core is then +preferentially refilled by the same layer on the next tick, so a layer stays +clustered on the hosts it already occupies instead of scattering across the +farm; a multi-frame layer keeps at least one proc on its host between ticks, +so the signal persists without tracking individual completions. The bonus is +sized to outweigh the marginal stranding terms (which are ~e^util, single +digits) but it is applied AFTER fit and reservation filtering, so it can never +place a frame that does not fit or override a reservation. Disable with +`scheduler.locality_enabled=false`. + +### 3.2 Reservations (EASY/Maui backfill) + +A wide layer can be starved indefinitely by a stream of small frames: every +core that frees is grabbed by a one-core frame before enough cores ever +accumulate on a single host. The classic fix is **reservations with +backfill** (Lifka 1995, the EASY scheduler; Jackson, Snell & Clement 2001, +Maui): give the blocked wide job a future claim on a host, let that host +drain toward it, and let shorter low-priority work run on the draining host +in the meantime as long as it finishes before the host is needed. + +The planner implements this with four guards so reservations never freeze the +farm reserved-but-idle (the classic low-utilization failure mode of naive +conservative backfill): + +- **Blocked-debt gate (when may a layer reserve).** Each layer carries a + leaky bucket of net blocked time (`blockedDebtMs`): it grows while the + layer is blocked (waiting frames, dispatched nothing this tick) and decays + 1:1 while it places. A layer qualifies to reserve only once its debt + reaches `reservationBlockMs` (default 5 min). Using net debt rather than a + "continuously blocked" timer means a job that only *crawls*, winning the + odd gap that would reset a continuous timer, still earns a reservation. +- **Width gate (which layers may reserve), always on.** A reservation exists + to drain a host for a frame too wide to fit otherwise. A layer may reserve + only if its per-frame core request is at least + `RESERVATION_MIN_HOST_FRACTION` (0.5) of the **largest** host in its group. + Without this gate the narrow small-frame stream, which never actually + needed a reservation because it runs the instant any core frees, floods the + reservation budget and locks out the wide jobs the budget was meant for. + This gate is not configurable. +- **Capacity cap.** Reservations may hold at most `reservationMaxFraction` + (default 0.5) of the hosts that can fit the layer, and at most + `scheduler.reservation_max_grantees` (default 8) distinct layers may hold + reservations farm-wide. So a class of machines can never be fully reserved, + and the farm cannot deadlock on reservations. Granting is priority-first + (then widest-job-first), so high-priority blocked work gets first claim on + the limited budget. +- **Drain guard.** A reserved host that has not yet drained enough cores for + its owner (`idle < owner.coresMin`) refuses all backfill, even work that + would finish in time. Otherwise backfill keeps refilling the gap the host + is trying to open and the owner never gets in. + +Backfill itself (`backfillAllows` / `backfillFits`) is the EASY no-delay +test: a strictly-lower-priority frame may borrow a draining host's free cores +only if its runtime estimate (from `layer_usage`) shows it finishing before +the host is needed; a frame with no runtime history is refused, since its +finish time cannot be bounded. Borrowing never takes ownership. + +Supporting machinery: `reservationAllows` decides whether a reservation lets +a layer through (no reservation, owns it, or the existing one is strictly +lower priority and gets overridden on successful dispatch); +`reconcileReservationsForLayer` brings a qualified layer's reservation count +toward its pending frame count under the caps; `pickReservationTarget` +chooses the host likely to free soonest (fewest running procs) among hosts +that fit the layer when fully idle. + +The reservation map persists across ticks. The invariant: a host's +reservation belongs to the highest-priority layer that has claimed it. The +end-of-tick sweep drops reservations (and blocked-debt) for layers that left +the dispatchable set. + +This is the mechanism that retires the human-driven "save this machine for +the big job" practice. + +### 3.3 Plan reads and batched commit + +The planner never writes bookings during placement; it just records the +`(host, layer)` pairings it chose. After all groups, `doTick` reads each +pairing's frames in parallel by host (`planHost`, read-only, on a small read +pool), then writes every booking for the tick in one batched transaction +(`startFramesAndProcsBatch`: batched frame UPDATE + proc INSERT + host UPDATE). +Frames lost to a `frame.int_version` race are dropped from the batch and retried +next tick. The RQD launches fire afterward fire-and-forget on a launch pool, so +a slow RQD never stalls the tick. Each frame reserves exactly the layer's requested cores: `planHost` builds +procs with the dispatcher's thread-mode idle-core expansion (grab-idle) turned +off, so the cores committed match the cores the planner scored and decremented. +Grab-idle would silently reserve more than planned and corrupt the snapshot; +the planner fills hosts by planning several placements, not by one frame +ballooning to consume the box. + +This keeps the *decisions* on one thread (no races) while parallelizing the part +that dominates tick time as the farm fills: the per-host reads. The writes stay +one batched, atomic commit. + +### 3.4 Leader election + +Only one Cuebot may plan at a time. `runTick` takes a Postgres advisory lock +(`pg_try_advisory_lock`); a Cuebot that does not hold it returns +immediately. The lock is released at the end of each tick and is released +automatically by Postgres if the leader's session drops, so failover is +automatic. A new leader starts with an empty reservation map and no +persistent state to migrate; placement resumes immediately. One caveat: the +block-time bucket is in-memory too, so after a failover reservations re-arm +only as blocked layers re-accrue `reservation_block_seconds` — over that +window, not a tick or two. + +### 3.5 Priority: a weighted lottery (rate, not rank) + +The candidate query does **not** order strictly by priority. It draws a +**priority-weighted lottery** — Efraimidis-Spirakis weighted reservoir sampling: +each eligible layer gets a random key `power(random(), 1.0 / GREATEST(priority, 1))` +and the top `layer_candidates_per_group_max` by that key are taken +(`ORDER BY power(random(), 1.0/…) DESC LIMIT …` in `SELECT_CANDIDATES_FOR_GROUP`). +A layer's expected selection rate is proportional to its priority, so **priority +is a rate, not a rank.** This is the single most operator-visible change from the +legacy dispatcher, which sorted strictly by `priority DESC` and so gave every free +core to the highest-priority work until it drained — starving everything below it +while a high-priority backlog stayed full. + +**What this means for operators.** Priority now buys a *share*, not dominance. A +show at priority 120 vs one at 100 wins roughly `120/(120+100) ≈ 55%` of the +contested selections, not 100%. Two consequences: + +- **Re-spread clustered values.** If your priority numbers were calibrated for + rank semantics they often cluster in a narrow band (e.g. 90–110). Under the + lottery that band barely differentiates — 110 vs 90 is only a `≈1.22×` rate + edge. To get meaningful separation, spread the values (e.g. 50 / 100 / 400). +- **It is a rate, not a guarantee.** The *realized* share of completed frames also + depends on backlog composition: a stream with far more waiting layers is + over-represented in the candidate pool, so it lands more selections than its + bare priority ratio suggests, and a thin low-priority stream lands fewer. The + firm guarantee the lottery provides is **anti-starvation** — any eligible layer + keeps a nonzero, priority-weighted chance every tick and never waits behind a + saturating higher-priority backlog forever. `GREATEST(priority, 1)` floors the + weight so priority 0 or negative still draws the minimum nonzero share. + +**Reservations are the exception.** Reservation *granting* stays strict +priority-first (the requests are re-sorted by priority before the per-class caps +fill; §3.2), so the lottery changes only booking *order*, never which wide job +gets rescued first. + +--- + +## 4. Concurrency model + +The planner reasons over an in-memory snapshot while commits and external +events change the database in the background. This is safe by design. + +**Single-booker invariant.** Three guards ensure nothing competes to +*consume* capacity behind the planner's back: + +1. `tickInFlight` compare-and-set, one Cuebot never overlaps its own ticks. +2. Leader advisory lock, only one Cuebot plans across the deployment. +3. In `facility` mode `scheduler.enabled` suppresses the legacy `BookingQueue` + enqueue in `HostReportHandler`; in `managed` mode the legacy dispatcher keeps + running but its query excludes `b_scheduler_managed` shows, so the two never + book the same show. + +So the only things that can change host state during a tick are: + +- **The planner's own commits**: already accounted for, because the planner + decrements its in-memory snapshot for each decision as it makes it. +- **External frame completions**: these only *free* cores, i.e. the + snapshot is conservative (it under-counts free capacity). Safe. + +**Snapshot drift is a quality issue, not a correctness one.** The commit +treats the database as the source of truth, not the snapshot: +`startFramesAndProcsBatch` books against real current state, with two hard +guards underneath: +- the atomic host update + `UPDATE host SET int_cores_idle = int_cores_idle - ? WHERE ... >= ?`, + which makes physical over-booking impossible, and +- the `frame.int_version` optimistic lock, which rejects any overlapping + frame grab. A frame that loses the version race is dropped from the batch + (comparing affected-row counts) and stays WAITING for the next tick. + +If the batch books fewer frames than the planner estimated (a host had less +room than the snapshot, or a frame lost its version race), the planner merely +over-decremented its in-memory copy and leaves that host slightly +under-packed for the rest of the tick. The next tick's fresh snapshot +corrects it. Failures bias toward **under-booking** (waste a little capacity +for one tick), never over-booking. + +**Drift is bounded to a single tick** because the batched commit is +synchronous on the planner thread: when it returns, the database fully +reflects this tick's bookings, so the next snapshot re-grounds on reality. +Only the RQD launches run afterward, fire-and-forget on the launch pool, so a +slow or sluggish RQD never stalls the next tick. There is no commit worker +pool and no drain barrier to wait on; the single transaction is the +synchronization point. + +--- + +## 5. Performance + +The planner is built to take load off the database, the scaling bottleneck of +the legacy path, and to fill capacity the instant it exists. + +**It fills the farm immediately.** Because the planner places across the whole +farm in a single tick, rather than booking one host at a time as each host +reports, it saturates idle capacity in one pass instead of waiting for a report +from every host. From a cold start in the DB-backed simulator it drove 1553 +hosts (~57k cores) from idle to ~100% utilization in about 40 seconds, booking +on the order of hundreds of frames per second, and holds a steady backlog at +full utilization thereafter. The legacy report-driven path can only book a host +when that host next reports, so a cold farm fills at the report rate. + +**One batched commit instead of a transaction per frame.** The legacy path +books each frame in its own `@Transactional` call, so booking N frames is N +transactions, every one taking row locks on the same few hot accounting rows +(subscription, folder_resource, point, layer_stat, job_stat). Under load those +transactions serialize on the shared rows and per-transaction BEGIN/COMMIT +overhead dominates. The planner lands every booking for a tick in a single +transaction (`startFramesAndProcsBatch`, section 3.3): a batched frame UPDATE (a +VALUES join keyed on `(pk_frame, int_version)`), a multi-row proc INSERT, one +summed `UPDATE host` per host, and the resource-counter deltas accumulated in +memory and flushed as one UPDATE per hot row instead of one per proc. Thousands +of contended per-proc writes collapse into a few dozen, and a deadlock-free lock +order (layer_stat then job_stat, both sorted; `lockStatRowsForBatch`) keeps the +batch from ever deadlocking against a concurrent frame completion. The effect is +a dramatic drop in row-lock contention on exactly the rows every booking and +every completion must touch. + +**Query count scales with host-spec groups, not hosts.** The legacy dispatcher +is reactive and per-host: every host report runs `findDispatchJobs(host)`, a +heavy multi-table join, so the count of heavy candidate queries grows with the +host count and the report rate, a per-host "query storm" that worsens as the +farm grows. The planner is proactive and farm-wide: one host-snapshot query per +tick, hosts bucketed into a few static spec groups, then one candidate-layer +query per group. On a homogeneous farm that is O(G) heavy queries per tick +(G = distinct host specs, a small constant) instead of O(H) per report cycle +(H = hosts), so heavy DB query load stops scaling with farm size. The only +per-host work left is the read-only plan phase, which is light and runs in +parallel; placement scoring is O(candidates x hosts), but that is in-memory +arithmetic over the snapshot, not database work. + +**Roughly 10x less DB traffic overall.** Together these move the design from "a +transaction per booking decision plus a heavy join per host report" to +"in-memory planning with one batched commit per tick." In the DB-backed +simulator that is about an order of magnitude less database traffic in steady +state, and more than that on the worst-case per-frame row-fetch: the legacy path +fetched on the order of ~75,000 rows per completed frame, the planner ~1,000. +The database is still the remaining ceiling, not the planner, but the planner +already takes most of the load off it. + +--- + +## 6. Configuration + +| Property | Default | Meaning | +|---|---|---| +| `scheduler.enabled` | `no` | Rollout switch: `no` (off, legacy owns every show), `facility` (planner owns all shows, legacy BookingQueue globally suppressed), or `managed` (planner owns only shows flagged `b_scheduler_managed=true` -- set per show via the show API -- legacy keeps the rest). Back-compat: `true`=facility, `false`=no. | +| `scheduler.read_pool_size` | = launch pool size | Threads for the parallel per-host plan reads (read-only, DB-bound). | +| `scheduler.launch_pool_size` | `8` | Threads for the fire-and-forget RQD launches after the batched commit. | +| `scheduler.launch_queue_size` | `16384` | Bound on queued launches; on overflow a launch is dropped and recovered by RQD report reconciliation. | +| `scheduler.layer_candidates_per_group_max` | `2000` | Cap on candidate layers fetched per group per tick. | +| `scheduler.reservations_enabled` | `true` | Enable reservations and backfill. When off, pure placement scoring. | +| `scheduler.reservation_block_seconds` | `300` | Net blocked time a layer must accrue before it may reserve. | +| `scheduler.reservation_max_fraction` | `0.5` | Max fraction of a layer's fitting hosts that reservations may hold. | +| `scheduler.reservation_max_grantees` | `8` | Max distinct layers holding reservations farm-wide. | +| `scheduler.backfill_enabled` | `true` | Allow lower-priority frames to backfill draining reserved hosts. | +| `scheduler.locality_enabled` | `true` | Prefer hosts already running the layer (co-locality / cache coherence). | +| `scheduler.locality_bonus` | `8.0` | Score bonus for a co-located host. Applied after fit/reservation filtering, so it never overrides them. | +| `scheduler.stat_interval_seconds` | `300` | Cadence of the consolidated INFO `Scheduler stat:` line (planner health, farm fill, throughput, reservations). Lower it for live debugging. | +| `dispatcher.job_frame_dispatch_max` | `8` | Max frames of one job booked onto a host per tick. | +| `dispatcher.host_frame_dispatch_max` | `12` | Max frames booked onto a host per tick. | + +The reservation **width gate** (`RESERVATION_MIN_HOST_FRACTION`, 0.5 of the +largest host in a group) is deliberately a fixed constant, not a property: +loosening it reintroduces the small-frame flooding it exists to prevent. + +No schema changes. The only new SQL is the host-snapshot query and the +per-group candidate query, both against existing tables and indexes. + +**Rollback** is a single flag: set `scheduler.enabled=no` and the legacy +dispatcher resumes. Progressive rollout works the same way in reverse: in +`managed` mode, clearing a show's `b_scheduler_managed` flag hands it straight +back to the legacy dispatcher with no restart. + +--- + +## 7. Failure modes + +- **Commit collision** (`frame.int_version` / resource guard): the frame is + dropped from the batch, stays WAITING, picked up next tick. Logged at debug. +- **Leader loss** (session drop): the advisory lock releases automatically; + another Cuebot becomes leader on its next tick. Placement resumes at once, + but reservations re-arm only as blocked layers re-accrue + `reservation_block_seconds` (the block-time bucket is in-memory). +- **Slow batched commit**: the commit is synchronous, so a slow transaction + delays the next tick directly (no worker pool hides it). This is the one + place where DB latency gates the tick rate; the future-work batching and + row-fetch reductions (sections 8 and 9) target it. +- **Slow RQD launch**: absorbed by the fire-and-forget launch pool; on a full + launch queue the launch is dropped and recovered by RQD report + reconciliation, so it never stalls the tick. +- **Empty snapshot** (no bookable hosts): tick is a no-op; reservations are + left intact. +- **Spec-group explosion**: if the host-spec group count approaches the host + count (commonly a host name leaking into the tag set), planning degrades to + one candidate query per host, the very storm grouping avoids. The scheduler + logs a throttled WARNING (at most once every few minutes) so it is caught + without flooding the log. +- **Bare-hostname tag pins are not honored**: cuebot auto-adds each host's own + name as a tag, and `normalizeTags` strips it from the group key (that is what + prevents the group explosion above). As a result a layer tagged with *only* a + bare hostname (`layer.tags == ""`, the legacy exclusive-pin idiom) + matches no group and never dispatches under the scheduler — its frames sit + `WAITING`. The legacy dispatcher honors such pins (it matches the host's raw + tags), so this is a silent difference for `scheduler.enabled` shows. A layer + that carries a shared tag alongside the hostname still dispatches on the shared + tag. If exclusive hostname pinning is needed, keep those shows on the legacy + dispatcher (or route via a dedicated allocation/tag instead of a host name). + +**Observability.** Per-tick detail is DEBUG; INFO carries one consolidated +`Scheduler stat:` line per `scheduler.stat_interval_seconds` (default 5 minutes): + +``` +Scheduler stat: win=300s ticks=920 skipped=0 lockLost=12 avgTick=556ms maxTick=1840ms + | farm hosts=1553 idleHosts=9 cores=57088 idleCores=74 util=99.9% groups=5 + | flow committed=98210 planned=104900 raceLost=6690 launchDropped=0 + | resv held=52 granted=31 reqs=11 backfilled=88 +``` + +It groups planner health and HA leadership (ticks won, `skipped` = fired while +the previous tick still ran, `lockLost` = another Cuebot held the lock, avg/max +tick), farm fill (hosts, idle hosts, cores, idle cores, utilization, host-spec +group count), throughput and loss (committed procs, frames planned, `raceLost` = +frames lost to the version race, RQD launches dropped), and reservation/backfill +activity (held, newly granted, requested, backfilled). Every Cuebot emits it, +leader or standby, so a standby that never wins the lock still heartbeats +(`ticks=0`, `lockLost` high), distinguishing a standby from a dead process. The +line is meant to be pasted straight into a bug report. + +--- + +## 8. Simulator (scheduler-sim) + +The scheduler ships with a full DB-backed simulator under `scheduler-sim/`. It +is not a model of Cuebot, it *is* Cuebot: a real Cuebot process and a real +Postgres, driven over gRPC by a fake render farm, so every booking goes through +the exact production path (the real Scheduler, the real SQL, the real +frame-complete handler). That makes it the integration test unit tests cannot +be, and the place to measure behaviour that only shows up under load. + +One command brings the whole stack up from nothing: + + scheduler-sim/simulate.py --mode new --feed 240 --metrics 220 + +It initdb's a Postgres cluster, applies the cuebot schema (Flyway migrations, +tracked so a cluster that survives across runs still picks up new ones), seeds +it, builds cuebot, registers the farm, starts a fake RQD plus a continuous +status reporter, feeds a workload, prints live metrics, and tears the run down +on the next invocation. Runs are reproducible (seeded RNG) and need no manual +setup on a fresh box. + +What it reproduces faithfully: + + - **A real farm's shape.** 1553 hosts and ~57k cores across three real host + classes (jaime 128c, ram 32c, elk 16c), with realistic usable-memory cuts + for the OS, RQD and cache, so memory binds, not just cores. + - **A production-shaped workload.** Per-layer core counts, memory and frame + durations are sampled from real-farm CSV distributions, with heavy-tailed + (lognormal) per-layer memory. + - **A concurrent driver.** Frame completions and host status are reported in + parallel, the way thousands of independent RQDs do, so a slow serial + reporter cannot accidentally flatter a worse scheduler. + +What it can simulate (the knobs): + + - `--mode new|old`: A/B the new scheduler against the legacy dispatcher on an + identical workload. + - `--feed` / `--jobs`: a sustained saturated backlog or a fixed job set, for + steady-state utilization and drain time. + - `--compress`: scale frame durations to find the utilization plateau and the + cuebot/DB frame-lifecycle ceiling. + - `--strand`: the starvation test, inject wide 64-core jobs (mixed equal/high + priority) and watch them strand or get rescued by reservations and backfill. + - `--reservations`, `--reservation-block-seconds`, `--reservation-max-fraction`, + `--reservation-max-grantees`, `--no-backfill`: exercise the full EASY/Maui + reservation and backfill machinery. + - `--mem-heavy`: the low-utilization test, flood the farm with small, + single-threaded, memory-hungry jobs so RAM binds and cores strand; the GB + value dials the plateau (around 16 GB gives ~27% core utilization, 32 GB + ~17%, while memory pins near 100%). + - `--tags`: the production capability-routing model (size classes tied to host + types). + - `--gpu F`: a fraction of layers become GPU layers (few cores + 1 GPU + GPU + memory), runnable only on GPU hosts. + - `--mem-failure-rate`: inject OOM failures so cuebot bumps layer memory and + requeues, exercising the retry path. + - `--hosts`: shrink the farm to a few machines for legible, watchable debugging. + +What it measures, live (`live_stats`): real core utilization (cores actually +backing a frame), running/waiting/orphaned-proc counts, frames completing and +booking per second, Postgres commits per second, cache-hit ratio and connection +states, per-priority big-job stranding, and a DB-load view (transactions and +rows fetched per completed frame, busiest tables). That is how the numbers in +section 5 was produced. + +See `cuebot/scheduler-sim/Scheduler-simulator.md` +for what the simulator does and the full flag reference. + +## 9. Testing + +An offline simulator that A/B tests the legacy dispatcher and this planner +against a production-shaped workload lives on the `sim` branch under +`benchmarks/sim_cpp/`. It models the workload service +mix, the operator core/tag pre-pass, KSM co-location, and simulated DB time. +Use it for algorithmic experiments; use a real small-facility trial for +production validation. diff --git a/cuebot/src/main/java/com/imageworks/spcue/dispatcher/SchedulerMode.java b/cuebot/src/main/java/com/imageworks/spcue/dispatcher/SchedulerMode.java new file mode 100644 index 0000000000..c933c7cbba --- /dev/null +++ b/cuebot/src/main/java/com/imageworks/spcue/dispatcher/SchedulerMode.java @@ -0,0 +1,79 @@ + +/* + * Copyright Contributors to the OpenCue Project + * + * Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except + * in compliance with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software distributed under the License + * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express + * or implied. See the License for the specific language governing permissions and limitations under + * the License. + */ + +package com.imageworks.spcue.dispatcher; + +import com.imageworks.spcue.dao.ShowDao; +import org.springframework.core.env.Environment; + +/** + * Interprets {@code scheduler.enabled} as a tri-state progressive-rollout switch + * (rather than a plain boolean), so the in-process Scheduler can be turned on for + * one show at a time -- the same per-show model the standalone Rust scheduler uses + * via {@code show.b_scheduler_managed}: + * + *
    + *
  • {@code no} -- Scheduler off; the legacy dispatcher owns every show.
    • + *
  • {@code facility} -- Scheduler plans ALL shows; legacy booking globally + * suppressed (this is the old {@code scheduler.enabled=true} behaviour).
    • + *
  • {@code managed} -- Scheduler plans only shows flagged + * {@code b_scheduler_managed=true} (set per show via the show API, exactly + * like Rust); the legacy dispatcher keeps the rest. The legacy dispatch + * query already excludes managed shows, so the two partition cleanly.
    • + *
+ * + * Back-compat: {@code "true"} maps to {@code facility}, {@code "false"} to + * {@code no}. Show selection lives in the per-show flag, NOT in this string, so + * Cuebot never has to reconcile a config value into the database. + */ +public final class SchedulerMode { + + private SchedulerMode() {} + + public static String mode(Environment env) { + String m = env.getProperty("scheduler.enabled", "no"); + return (m == null || m.trim().isEmpty()) ? "no" : m.trim(); + } + + /** True when the in-process Scheduler runs at all (facility or managed). */ + public static boolean enabled(Environment env) { + String m = mode(env); + return !(m.equalsIgnoreCase("no") || m.equalsIgnoreCase("false")); + } + + /** + * True when the Scheduler owns EVERY show and the legacy BookingQueue is + * globally suppressed (facility-wide rollout / the old boolean {@code true}). + */ + public static boolean facility(Environment env) { + String m = mode(env); + return m.equalsIgnoreCase("facility") || m.equalsIgnoreCase("true"); + } + + /** + * Whether the in-process Scheduler -- not the legacy dispatcher -- owns this + * show. In {@code managed} mode this defers to the per-show + * {@code b_scheduler_managed} flag. + */ + public static boolean schedules(Environment env, ShowDao showDao, String showId) { + if (!enabled(env)) { + return false; + } + if (facility(env)) { + return true; + } + return showDao.isSchedulerManaged(showId); + } +} diff --git a/cuebot/src/main/java/com/imageworks/spcue/rqd/RqdClientGrpc.java b/cuebot/src/main/java/com/imageworks/spcue/rqd/RqdClientGrpc.java index 9c7f4d6a96..691755ac80 100644 --- a/cuebot/src/main/java/com/imageworks/spcue/rqd/RqdClientGrpc.java +++ b/cuebot/src/main/java/com/imageworks/spcue/rqd/RqdClientGrpc.java @@ -208,6 +208,12 @@ public void launchFrame(final RunFrame frame, final VirtualProc proc) { try { getStub(proc.hostName).launchFrame(request); } catch (StatusRuntimeException | ExecutionException e) { + // Log the underlying cause: the caller only sees a generic + // RqdClientException, which hides why the launch failed (e.g. an + // EMFILE "Too many open files" once the per-host channel cache + // exhausts the process FD limit at large farm scale). + logger.warn("failed to launch frame on " + proc.hostName + ":" + + rqdServerPort + ": " + e, e); throw new RqdClientException("failed to launch frame", e); } } diff --git a/cuebot/src/main/java/com/imageworks/spcue/service/JobSpec.java b/cuebot/src/main/java/com/imageworks/spcue/service/JobSpec.java index d4ff4e6b5d..4a54c52557 100644 --- a/cuebot/src/main/java/com/imageworks/spcue/service/JobSpec.java +++ b/cuebot/src/main/java/com/imageworks/spcue/service/JobSpec.java @@ -49,6 +49,7 @@ import com.imageworks.spcue.ServiceEntity; import com.imageworks.spcue.SpecBuilderException; import com.imageworks.spcue.dispatcher.Dispatcher; +import com.imageworks.spcue.dispatcher.SchedulerMode; import com.imageworks.spcue.grpc.depend.DependType; import com.imageworks.spcue.grpc.job.JobState; import com.imageworks.spcue.grpc.job.LayerType; @@ -601,10 +602,20 @@ private void determineMinimumCores(Element layerTag, LayerDetail layer) { corePoints = Integer.valueOf(cores); } + // The in-process Scheduler can place much wider per-frame reservations + // than the legacy dispatcher (it reserves/drains whole hosts for wide + // jobs), so it gets a higher clamp. dispatcher.frame_cores_max raises it + // further for whole-host jobs. Legacy keeps the conservative cap. + boolean schedulerEnabled = SchedulerMode.enabled(env); + int coreMax = schedulerEnabled + ? env.getProperty("dispatcher.frame_cores_max", Integer.class, + Dispatcher.CORE_POINTS_RESERVED_MAX_NEW) + : Dispatcher.CORE_POINTS_RESERVED_MAX; + if (corePoints > 0 && corePoints < Dispatcher.CORE_POINTS_RESERVED_MIN) { corePoints = Dispatcher.CORE_POINTS_RESERVED_DEFAULT; - } else if (corePoints > Dispatcher.CORE_POINTS_RESERVED_MAX) { - corePoints = Dispatcher.CORE_POINTS_RESERVED_MAX; + } else if (corePoints > coreMax) { + corePoints = coreMax; } layer.minimumCores = corePoints; diff --git a/cuebot/src/main/resources/conf/spring/applicationContext-service.xml b/cuebot/src/main/resources/conf/spring/applicationContext-service.xml index 18be8179f9..72c00ed79d 100644 --- a/cuebot/src/main/resources/conf/spring/applicationContext-service.xml +++ b/cuebot/src/main/resources/conf/spring/applicationContext-service.xml @@ -415,6 +415,20 @@ + + + + + + + + + + @@ -518,6 +532,19 @@ + + + + + + + + + + + + @@ -642,6 +669,7 @@ + diff --git a/cuebot/src/main/resources/opencue.properties b/cuebot/src/main/resources/opencue.properties index 1782177c5a..42c2b13692 100644 --- a/cuebot/src/main/resources/opencue.properties +++ b/cuebot/src/main/resources/opencue.properties @@ -113,6 +113,82 @@ dispatcher.frame_query_max=20 dispatcher.job_frame_dispatch_max=8 # Maximum number of frames to dispatch from a host at one time. dispatcher.host_frame_dispatch_max=12 + +# ---- E-PVM Scheduler (Planner) ---------------------------------------------- +# A whole-farm scheduler that replaces the legacy per-host BookingQueue dispatch +# path. Each tick it snapshots the whole farm, groups hosts by static spec +# (alloc, tags, os, gpu) and runs one candidate-layer query per group (instead +# of one query per host), then scores every placement by opportunity cost -- how +# much future capacity it would strand (E-PVM) -- rather than booking the first +# frame that fits. It also reserves hosts for wide layers that would otherwise +# starve and EASY-style backfills lower-priority frames around those +# reservations. Placement decisions are made by a single writer thread over the +# snapshot (only the per-host plan reads run in parallel); cross-Cuebot leader +# election is via a Postgres advisory lock. +# +# Tri-state progressive-rollout switch (default "no"): +# no -- off; the legacy dispatcher owns every show. +# facility -- the scheduler owns ALL shows; HostReportHandler suppresses the +# legacy BookingQueue enqueue (the old scheduler.enabled=true). +# managed -- the scheduler owns only shows flagged b_scheduler_managed=true +# (set per show via the show API, same as the Rust scheduler); the +# legacy dispatcher keeps the rest. +# Back-compat: "true" maps to facility, "false" to no. +scheduler.enabled=no +# Tick interval, in milliseconds. Render frames run for minutes to hours, so a +# few seconds between planning passes is plenty responsive and keeps the +# per-tick farm-wide query off the database hot path. +scheduler.interval_ms=3000 +# Maximum candidate layers fetched per host group per tick. Layers below +# this cut are by definition lower priority and considered on a later tick. +scheduler.layer_candidates_per_group_max=2000 +# Thread-pool sizing. The planner makes every placement decision on one thread; +# the only parallelism is the read-only per-host plan phase and the +# fire-and-forget RQD launches after the single batched commit (there is no +# commit worker pool). read_pool_size: threads for the parallel plan reads +# (defaults to launch_pool_size). launch_pool_size: threads for the post-commit +# RQD launches. launch_queue_size: bound on queued launches; on overflow a +# launch is dropped and recovered by RQD report reconciliation, so a slow RQD +# never stalls the tick. +scheduler.read_pool_size=8 +scheduler.launch_pool_size=8 +scheduler.launch_queue_size=16384 +# Whether the planner makes and enforces persistent host reservations for +# blocked layers. When false, the scheduler runs as a pure placement core +# (no reservations made or enforced), useful for isolating core behaviour. +scheduler.reservations_enabled=true +# Reservation gating (EASY/Maui-style: Lifka 1995; Jackson et al. 2001). +# reservation_block_seconds: a layer must accrue this much NET blocked time +# before it may reserve. Blocked time is a leaky bucket that grows while the +# layer is blocked (waiting frames it cannot place anywhere) and decays while +# it places, so transient saturation never triggers a reservation but a job +# that only crawls still earns one. reservation_max_fraction: reservations may +# hold at most this fraction of the hosts that can fit a given layer, so a host +# class can never be fully reserved and the farm cannot deadlock on +# reservations. A layer must also be WIDE enough to reserve (per-frame request +# at least half the largest host in its group); this width gate is a fixed +# constant, not a property, because loosening it lets a narrow small-frame +# stream flood the budget. Qualifying layers are granted reservations highest +# priority first, then widest, until the cap is reached. +scheduler.reservation_block_seconds=300 +scheduler.reservation_max_fraction=0.5 +# Maximum number of NEW reservation grants per tick. Existing holders always +# reconcile. Caps the O(layers x hosts) work at full-farm scale. +scheduler.reservation_max_grantees=8 +# EASY backfill (Lifka 1995): instead of leaving a reserved host idle while it +# drains toward its (wide) reserving layer, let a lower-priority frame run on +# its free cores, but only if that frame's worst-case runtime +# (layer_usage.int_clock_time_high) finishes before the host is projected to +# free enough cores for its owner, so the reserved job is never delayed. This +# recovers the utilization a pure freeze would waste. Requires runtime history +# on the borrowing layer; layers with no history are never backfilled. +scheduler.backfill_enabled=true +# Cadence (seconds) of the consolidated INFO "Scheduler stat:" line: one full +# snapshot per window of planner health and HA leadership, farm fill, throughput +# and loss, and reservation/backfill activity (per-tick detail stays at DEBUG). +# Emitted by every Cuebot, leader or standby, so a standby still heartbeats. +# Lower it during a live incident, raise it to quiet the log. Default 5 minutes. +scheduler.stat_interval_seconds=300 # Choose between different scheduling strategies: # - PRIORITY_ONLY: Sort by priority only # - FIFO: Whether or not to enable FIFO scheduling in the same priority. diff --git a/cuebot/src/test/java/com/imageworks/spcue/dispatcher/SchedulerTests.java b/cuebot/src/test/java/com/imageworks/spcue/dispatcher/SchedulerTests.java new file mode 100644 index 0000000000..81e006e2e4 --- /dev/null +++ b/cuebot/src/test/java/com/imageworks/spcue/dispatcher/SchedulerTests.java @@ -0,0 +1,396 @@ +/* + * Copyright Contributors to the OpenCue Project + * + * Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except + * in compliance with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software distributed under the License + * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express + * or implied. See the License for the specific language governing permissions and limitations under + * the License. + */ + +package com.imageworks.spcue.dispatcher; + +import java.util.Arrays; +import java.util.List; +import java.util.Map; + +import org.junit.Test; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertTrue; + +/** + * Unit tests for the pure, side-effect-free logic in {@link Scheduler}: + * tag normalization, host grouping, the fit check, the per-tick frame + * prediction, and the E-PVM placement score. These need no Spring context + * or database; the test lives in the dispatcher package so it can reach the + * package-private static helpers and POJOs. + */ +public class SchedulerTests { + + /** Core points per whole core (host int_cores_idle is in core points). */ + private static final int CORE = 100; + /** One gigabyte expressed in kilobytes (host/layer mem values are in KB). */ + private static final long GB = 1024L * 1024L; + + private static Scheduler.BookableHost host(String alloc, String tags, String os, + int coresIdle, long memIdle, int gpusIdle, long gpuMemIdle, + int coresTotal, long memTotal, int gpusTotal, long gpuMemTotal) { + Scheduler.BookableHost h = new Scheduler.BookableHost(); + h.hostId = "host"; + h.hostName = "host"; + h.pkAlloc = alloc; + h.tagsRaw = tags; + h.os = os; + h.coresIdle = coresIdle; + h.memIdle = memIdle; + h.gpusIdle = gpusIdle; + h.gpuMemIdle = gpuMemIdle; + h.coresTotal = coresTotal; + h.memTotal = memTotal; + h.gpusTotal = gpusTotal; + h.gpuMemTotal = gpuMemTotal; + h.runningProcs = 0; + return h; + } + + /** Host whose idle resources equal its totals (fully free). */ + private static Scheduler.BookableHost freeHost(int cores, long mem, int gpus, long gpuMem) { + return host("alloc", "tags", "Linux", cores, mem, gpus, gpuMem, + cores, mem, gpus, gpuMem); + } + + private static Scheduler.LayerCandidate layer(int coresMin, long memMin, + int gpusMin, long gpuMemMin) { + Scheduler.LayerCandidate c = new Scheduler.LayerCandidate(); + c.layerId = "layer"; + c.jobId = "job"; + c.showId = "show"; + c.layerCoresMin = coresMin; + c.layerMemMin = memMin; + c.layerGpusMin = gpusMin; + c.layerGpuMemMin = gpuMemMin; + c.priority = 100; + c.jobCoresInUse = 0; + c.jobMaxCores = Integer.MAX_VALUE; + c.showCoresInUse = 0; + c.showBurstCores = Integer.MAX_VALUE; + c.waitingFrameCount = 100; + return c; + } + + // ---- normalizeTags ---------------------------------------------------- + + @Test + public void normalizeTagsHandlesNullAndBlank() { + assertEquals("", Scheduler.normalizeTags(null)); + assertEquals("", Scheduler.normalizeTags("")); + assertEquals("", Scheduler.normalizeTags(" ")); + } + + @Test + public void normalizeTagsSortsAndCollapsesWhitespace() { + assertEquals("desktop linux", Scheduler.normalizeTags("linux desktop")); + assertEquals("a b c", Scheduler.normalizeTags("c a b")); + } + + @Test + public void normalizeTagsIsOrderIndependent() { + assertEquals(Scheduler.normalizeTags("linux desktop"), + Scheduler.normalizeTags("desktop linux")); + } + + @Test + public void normalizeTagsDedupsAndStripsHostName() { + // cuebot stores "general general "; grouping must reduce + // that to just "general" so same-spec hosts group together. + assertEquals("general", + Scheduler.normalizeTags("general general elk0001", "elk0001")); + // host-name exclusion is case-insensitive. + assertEquals("general", Scheduler.normalizeTags("general ELK0001", "elk0001")); + } + + @Test + public void groupByHostSpecCollapsesHostsThatDifferOnlyByNameTag() { + // Two hosts, identical spec, each carrying its own name as a tag. + Scheduler.BookableHost a = + host("alloc", "general a-host", "Linux", 100, GB, 0, 0, 100, GB, 0, 0); + a.hostName = "a-host"; + Scheduler.BookableHost b = + host("alloc", "general b-host", "Linux", 100, GB, 0, 0, 100, GB, 0, 0); + b.hostName = "b-host"; + + Map> groups = + Scheduler.groupByHostSpec(Arrays.asList(a, b)); + + assertEquals(1, groups.size()); + assertEquals(2, groups.values().iterator().next().size()); + } + + // ---- groupByHostSpec -------------------------------------------------- + + @Test + public void groupByHostSpecUsesGpuTotalsNotIdle() { + // A fully-booked GPU host (gpusIdle == 0) must still group as a GPU + // host: GPU presence is static hardware, keyed off totals. This is + // the regression guard for the idle-vs-total grouping fix. + Scheduler.BookableHost gpuFullyBooked = + host("a", "t", "Linux", 100, GB, 0, 0, 200, 2 * GB, 2, 8 * GB); + Scheduler.BookableHost cpuOnly = + host("a", "t", "Linux", 100, GB, 0, 0, 200, 2 * GB, 0, 0); + + Map> groups = + Scheduler.groupByHostSpec(Arrays.asList(gpuFullyBooked, cpuOnly)); + + assertEquals(2, groups.size()); + + boolean gpuKeyHasGpu = false; + boolean cpuKeyHasGpu = true; + for (Map.Entry> e : groups.entrySet()) { + if (e.getValue().contains(gpuFullyBooked)) { + gpuKeyHasGpu = e.getKey().hasGpu; + } + if (e.getValue().contains(cpuOnly)) { + cpuKeyHasGpu = e.getKey().hasGpu; + } + } + assertTrue("fully-booked GPU host must group as GPU host", gpuKeyHasGpu); + assertFalse("CPU-only host must not group as GPU host", cpuKeyHasGpu); + } + + @Test + public void groupByHostSpecMergesEquivalentTagOrderings() { + Scheduler.BookableHost a = + host("a", "linux desktop", "Linux", 100, GB, 0, 0, 100, GB, 0, 0); + Scheduler.BookableHost b = + host("a", "desktop linux", "Linux", 100, GB, 0, 0, 100, GB, 0, 0); + + Map> groups = + Scheduler.groupByHostSpec(Arrays.asList(a, b)); + + assertEquals(1, groups.size()); + assertEquals(2, groups.values().iterator().next().size()); + } + + @Test + public void groupByHostSpecSeparatesAllocAndOs() { + Scheduler.BookableHost alloc1 = + host("a1", "t", "Linux", 100, GB, 0, 0, 100, GB, 0, 0); + Scheduler.BookableHost alloc2 = + host("a2", "t", "Linux", 100, GB, 0, 0, 100, GB, 0, 0); + Scheduler.BookableHost otherOs = + host("a1", "t", "Windows", 100, GB, 0, 0, 100, GB, 0, 0); + + Map> groups = + Scheduler.groupByHostSpec(Arrays.asList(alloc1, alloc2, otherOs)); + + assertEquals(3, groups.size()); + } + + // ---- fitsOnHost ------------------------------------------------------- + + @Test + public void fitsOnHostAcceptsExactFit() { + assertTrue(Scheduler.fitsOnHost(layer(CORE, GB, 0, 0), freeHost(CORE, GB, 0, 0))); + assertTrue(Scheduler.fitsOnHost(layer(CORE, GB, 1, GB), freeHost(CORE, GB, 1, GB))); + } + + @Test + public void fitsOnHostRejectsWhenAnyDimensionShort() { + Scheduler.LayerCandidate cpu = layer(CORE, GB, 0, 0); + assertFalse(Scheduler.fitsOnHost(cpu, freeHost(CORE - 1, GB, 0, 0))); + assertFalse(Scheduler.fitsOnHost(cpu, freeHost(CORE, GB - 1, 0, 0))); + + Scheduler.LayerCandidate gpu = layer(CORE, GB, 1, GB); + assertFalse(Scheduler.fitsOnHost(gpu, freeHost(CORE, GB, 0, GB))); + assertFalse(Scheduler.fitsOnHost(gpu, freeHost(CORE, GB, 1, GB - 1))); + } + + // ---- computeMaxMore --------------------------------------------------- + + @Test + public void computeMaxMoreIsBoundedByThePhysicalDimension() { + // 10 cores idle, 1-core layer, ample memory: 9 additional frames fit. + assertEquals(9L, + Scheduler.computeMaxMore(freeHost(10 * CORE, 100 * GB, 0, 0), layer(CORE, GB, 0, 0))); + } + + @Test + public void computeMaxMoreRespectsJobMaxCores() { + Scheduler.LayerCandidate c = layer(CORE, GB, 0, 0); + c.jobMaxCores = 3 * CORE; // room for 3 cores total + // first frame consumes 1 core, leaving room for 2 more. + assertEquals(2L, + Scheduler.computeMaxMore(freeHost(10 * CORE, 100 * GB, 0, 0), c)); + } + + @Test + public void computeMaxMoreRespectsShowBurst() { + Scheduler.LayerCandidate c = layer(CORE, GB, 0, 0); + c.showBurstCores = 2 * CORE; // room for 2 cores total + assertEquals(1L, + Scheduler.computeMaxMore(freeHost(10 * CORE, 100 * GB, 0, 0), c)); + } + + // ---- placementScore --------------------------------------------------- + // + // Real E-PVM: score is the marginal rise of a convex cost + // sum_D W_D * ( e^(after_D/total_D) - e^(before_D/total_D) ) + // with before_D = total_D - idle_D and after_D = before_D + layer.min_D. + // Lower is better. Weights cores=1, mem=1, gpus=4, gpu_mem=1. + + /** Host with explicit idle resources (idle <= total). */ + private static Scheduler.BookableHost loadedHost(int cores, long mem, + int coresIdle, long memIdle) { + return host("alloc", "tags", "Linux", coresIdle, memIdle, 0, 0, + cores, mem, 0, 0); + } + + @Test + public void placementScoreOnEmptyHostIsExpFractionPerDimension() { + // 4-core/4GB layer on an empty 4-core/4GB host: each dimension goes + // 0 -> full, so its term is e^1 - 1; cores + mem give 2*(e-1). + Scheduler.LayerCandidate c = layer(4 * CORE, 4 * GB, 0, 0); + double term = Math.exp(1.0) - 1.0; + assertEquals(2 * term, Scheduler.placementScore(freeHost(4 * CORE, 4 * GB, 0, 0), c), 1e-9); + } + + @Test + public void placementScorePrefersLargerEmptyHostForSameFrame() { + // E-PVM is load-balancing: the same frame is a smaller fraction of a + // bigger host, so an empty 64-core/64GB host scores LOWER (is filled + // first) than an empty 4-core/4GB host. This is what stops big hosts + // from sitting idle under the old absolute-stranding score. + Scheduler.LayerCandidate c = layer(4 * CORE, 4 * GB, 0, 0); + double small = Scheduler.placementScore(freeHost(4 * CORE, 4 * GB, 0, 0), c); + double big = Scheduler.placementScore(freeHost(64 * CORE, 64 * GB, 0, 0), c); + assertTrue("bigger empty host should score lower", big < small); + // Exact: 2*(e^(4/64) - 1) on the 64-core host. + assertEquals(2 * (Math.exp(4.0 / 64.0) - 1.0), big, 1e-9); + } + + @Test + public void placementScoreRisesAsAHostFillsUp() { + // Convexity: adding the same frame to a host that is already loaded + // costs more than adding it to the same-size empty host, so work + // spreads across hosts instead of piling onto one. + Scheduler.LayerCandidate c = layer(CORE, GB, 0, 0); + double empty = Scheduler.placementScore(loadedHost(8 * CORE, 8 * GB, 8 * CORE, 8 * GB), c); + double loaded = Scheduler.placementScore(loadedHost(8 * CORE, 8 * GB, 2 * CORE, 2 * GB), c); + assertTrue("loaded host should score higher than empty", loaded > empty); + } + + @Test + public void placementScoreTreatsMemorySaturatedHostAsFull() { + // A host with idle cores but nearly saturated memory sits high on the + // memory axis, so its marginal cost is dominated by the steep e^x + // region, far higher than a balanced host with the same idle cores. + Scheduler.LayerCandidate c = layer(CORE, GB, 0, 0); + double balanced = Scheduler.placementScore(loadedHost(8 * CORE, 8 * GB, 4 * CORE, 4 * GB), c); + double memTight = Scheduler.placementScore(loadedHost(8 * CORE, 8 * GB, 4 * CORE, 1 * GB), c); + assertTrue("memory-tight host should score higher", memTight > balanced); + } + + @Test + public void placementScoreWeightsGpuDimensions() { + // 1-core/1GB/1-gpu/1GB-gpumem layer on an empty 1-core/1GB/2-gpu/4GB + // host. Each dimension's term is e^(add/total) - 1: + // cores : e^1 - 1 + // mem : e^1 - 1 + // gpus : (e^0.5 - 1) * 4 (W_GPUS = 4) + // gpu_mem : (e^0.25 - 1) * 1 + Scheduler.LayerCandidate gpu = layer(CORE, GB, 1, GB); + double expected = (Math.exp(1.0) - 1.0) + + (Math.exp(1.0) - 1.0) + + 4.0 * (Math.exp(0.5) - 1.0) + + 1.0 * (Math.exp(0.25) - 1.0); + assertEquals(expected, Scheduler.placementScore(freeHost(CORE, GB, 2, 4 * GB), gpu), 1e-9); + } + + @Test + public void placementScoreDoesNotPenalizeGpuSurplusForNonGpuLayer() { + // A non-GPU layer adds nothing on the GPU dimensions (add <= 0), so a + // GPU host's idle GPUs contribute 0 to its score: GPU hosts are + // protected from non-GPU work by grouping (has_gpu in the spec key), + // not by the score. The score equals the cores+mem terms only. + Scheduler.LayerCandidate cpu = layer(CORE, GB, 0, 0); + double expected = 2 * (Math.exp(1.0) - 1.0); + assertEquals(expected, Scheduler.placementScore(freeHost(CORE, GB, 4, 16 * GB), cpu), 1e-9); + } + + // ---- EASY backfill: hostReadySeconds ---------------------------------- + + @Test + public void hostReadySecondsIsZeroWhenNoCoresNeeded() { + // The host already has enough free cores for its reserving layer. + assertEquals(0, Scheduler.hostReadySeconds(0, Arrays.asList(new int[] {100, 30}))); + assertEquals(0, Scheduler.hostReadySeconds(-100, null)); + } + + @Test + public void hostReadySecondsReturnsCrossingProcsFinishTime() { + // Need 2 cores' worth (200 points). Procs free at 10s (1 core) and 30s + // (1 core); the second crosses the threshold, so the host is ready at 30s. + List procs = Arrays.asList(new int[] {100, 30}, new int[] {100, 10}); + assertEquals(30, Scheduler.hostReadySeconds(200, procs)); + } + + @Test + public void hostReadySecondsStopsAtFirstSufficientProc() { + // One big proc frees 4 cores at 20s; that alone covers the 3-core need. + List procs = Arrays.asList(new int[] {400, 20}, new int[] {100, 5}); + assertEquals(20, Scheduler.hostReadySeconds(300, procs)); + } + + @Test + public void hostReadySecondsUnknownWhenProcsCannotFreeEnough() { + // Procs free only 2 cores total but 5 are needed -> never ready. + List procs = Arrays.asList(new int[] {100, 10}, new int[] {100, 20}); + assertEquals(Integer.MAX_VALUE, Scheduler.hostReadySeconds(500, procs)); + } + + @Test + public void hostReadySecondsUnknownWhenNoProcs() { + assertEquals(Integer.MAX_VALUE, Scheduler.hostReadySeconds(100, null)); + } + + @Test + public void hostReadySecondsUnknownWhenANeededProcHasNoEstimate() { + // First proc (10s) frees 1 core; the next needed proc has an unknown + // finish (MAX_VALUE) so the host's ready time is unknown, not optimistic. + List procs = Arrays.asList( + new int[] {100, 10}, new int[] {100, Integer.MAX_VALUE}); + assertEquals(Integer.MAX_VALUE, Scheduler.hostReadySeconds(200, procs)); + } + + // ---- EASY backfill: backfillFits -------------------------------------- + + @Test + public void backfillFitsWhenFrameFinishesBeforeHostIsNeeded() { + // Worst-case frame runtime 30s, host free for its owner in 60s -> safe. + assertTrue(Scheduler.backfillFits(true, 30, 60)); + // Exactly equal is allowed (<=). + assertTrue(Scheduler.backfillFits(true, 60, 60)); + } + + @Test + public void backfillRefusedWhenFrameOutlastsTheReservation() { + assertFalse(Scheduler.backfillFits(true, 90, 60)); + } + + @Test + public void backfillRefusedWithoutRuntimeHistory() { + // No estimate -> cannot bound the frame, never borrow a reserved host. + assertFalse(Scheduler.backfillFits(false, 10, 10_000)); + } + + @Test + public void backfillRefusedWhenHostReadyTimeIsUnknown() { + assertFalse(Scheduler.backfillFits(true, 1, Integer.MAX_VALUE)); + } +} From da7740f1288275f5c1f16cdd674db6fdc65df774 Mon Sep 17 00:00:00 2001 From: Aghiles Kheffache Date: Wed, 8 Jul 2026 21:27:19 +0000 Subject: [PATCH 02/40] Add a DB-backed scheduler simulator with graphs and a --verify self-test A one-command simulator that runs a real cuebot + Postgres against a fake RQD and a synthetic farm (up to ~1553 hosts) to exercise the scheduler under load: - Workload feeders: steady fill, priority streams, and wide-job strand tests; a per-layer memory model (baseline + jitter) so a layer's frames cluster realistically instead of drawing memory independently. - Live stats and end-of-run graphs (utilization, throughput, cores-vs-memory, reservation subsystem, DB load), each stamped with the simulate.py command line so every graph is traceable to the config that produced it. - --strand-cores for whole-host (128-core) reservation tests. --verify is the recommended way to run it: one command that exercises the scheduler end to end. It runs six scenarios back-to-back, each a fresh, fully torn-down sim that writes its own graphs, then prints a PASS/FAIL summary (nonzero exit if any scenario fails): OOM -- memory failures bump the layer's memory per-frame, no legacy ratchet, and frames retry PRIORITY -- completion share is ordered by priority across 10 classes (Spearman rho) PRIORITY_STARVING -- a low-priority stream survives a high-priority flood (stays above a 3% floor) RESERVATIONS -- stranded wide jobs are rescued by reservations + backfill and actually run LIMIT -- a global license cap (limit_record.int_max_value) holds concurrent running frames at the cap under a deep backlog FOLDER -- a folder/group core ceiling (folder_resource.int_max_cores) holds the folder's running cores at the cap under a deep backlog Run it exactly as `simulate.py --verify`, with no other flags. Each scenario is tuned (farm size, oversubscription, frame length) so its verdict is meaningful; changing the options on a --verify run is unsupported and easily misleading (e.g. PRIORITY only shows proportional shares on a small, heavily oversubscribed farm -- on the full farm priority looks absent though the scheduler is correct). Only SIM_VERIFY_SECONDS (per-scenario length) is meant to be adjusted. The simulator's what/how, the --verify guidance, and the full flag reference live in cuebot/scheduler-sim/README.md. --- cuebot/scheduler-sim/.gitignore | 7 + cuebot/scheduler-sim/BUILD.md | 78 + cuebot/scheduler-sim/README.md | 283 +++ cuebot/scheduler-sim/analysis/.gitignore | 4 + cuebot/scheduler-sim/analysis/README.md | 40 + .../scheduler-sim/analysis/analyze_sweep.py | 66 + cuebot/scheduler-sim/analysis/cpu_sampler.py | 24 + cuebot/scheduler-sim/analysis/db_sampler.py | 20 + cuebot/scheduler-sim/analysis/make_graphs.py | 90 + .../scheduler-sim/analysis/make_graphs_ba.py | 87 + cuebot/scheduler-sim/analysis/plot_run.py | 267 +++ cuebot/scheduler-sim/analysis/util_sampler.py | 64 + cuebot/scheduler-sim/drain_test.py | 67 + cuebot/scheduler-sim/fake_rqd.py | 237 +++ cuebot/scheduler-sim/farm_spec.py | 203 ++ cuebot/scheduler-sim/feed.py | 163 ++ cuebot/scheduler-sim/folder_watch.py | 116 ++ cuebot/scheduler-sim/gen_jobs.py | 100 + cuebot/scheduler-sim/inject_big.py | 240 +++ cuebot/scheduler-sim/inject_folder.py | 130 ++ cuebot/scheduler-sim/inject_limit.py | 135 ++ .../scheduler-sim/inject_priority_spread.py | 151 ++ .../scheduler-sim/inject_priority_starve.py | 167 ++ cuebot/scheduler-sim/kill_all_jobs.py | 24 + cuebot/scheduler-sim/limit_watch.py | 104 + cuebot/scheduler-sim/live_stats.py | 144 ++ cuebot/scheduler-sim/metrics.py | 103 + cuebot/scheduler-sim/priority_spread_watch.py | 140 ++ cuebot/scheduler-sim/priority_starve_watch.py | 147 ++ cuebot/scheduler-sim/register_hosts.py | 111 + cuebot/scheduler-sim/resolve_local.c | 31 + cuebot/scheduler-sim/rqd_complete.py | 180 ++ cuebot/scheduler-sim/rqd_report.py | 210 ++ cuebot/scheduler-sim/setup.sh | 89 + cuebot/scheduler-sim/sim_mem.py | 107 + cuebot/scheduler-sim/sim_metrics.py | 121 ++ cuebot/scheduler-sim/sim_model.py | 197 ++ cuebot/scheduler-sim/sim_seed.sql | 46 + cuebot/scheduler-sim/simulate.py | 1810 +++++++++++++++++ cuebot/scheduler-sim/stats.py | 153 ++ cuebot/scheduler-sim/status_pinger.py | 64 + cuebot/scheduler-sim/status_pinger_fast.py | 64 + cuebot/scheduler-sim/strand_dur_watch.py | 165 ++ cuebot/scheduler-sim/strand_watch.py | 146 ++ cuebot/scheduler-sim/util_test.py | 112 + .../imageworks/spcue/dispatcher/Scheduler.md | 4 +- 46 files changed, 7009 insertions(+), 2 deletions(-) create mode 100644 cuebot/scheduler-sim/.gitignore create mode 100644 cuebot/scheduler-sim/BUILD.md create mode 100644 cuebot/scheduler-sim/README.md create mode 100644 cuebot/scheduler-sim/analysis/.gitignore create mode 100644 cuebot/scheduler-sim/analysis/README.md create mode 100644 cuebot/scheduler-sim/analysis/analyze_sweep.py create mode 100644 cuebot/scheduler-sim/analysis/cpu_sampler.py create mode 100644 cuebot/scheduler-sim/analysis/db_sampler.py create mode 100644 cuebot/scheduler-sim/analysis/make_graphs.py create mode 100644 cuebot/scheduler-sim/analysis/make_graphs_ba.py create mode 100644 cuebot/scheduler-sim/analysis/plot_run.py create mode 100644 cuebot/scheduler-sim/analysis/util_sampler.py create mode 100644 cuebot/scheduler-sim/drain_test.py create mode 100644 cuebot/scheduler-sim/fake_rqd.py create mode 100644 cuebot/scheduler-sim/farm_spec.py create mode 100644 cuebot/scheduler-sim/feed.py create mode 100644 cuebot/scheduler-sim/folder_watch.py create mode 100644 cuebot/scheduler-sim/gen_jobs.py create mode 100644 cuebot/scheduler-sim/inject_big.py create mode 100644 cuebot/scheduler-sim/inject_folder.py create mode 100644 cuebot/scheduler-sim/inject_limit.py create mode 100644 cuebot/scheduler-sim/inject_priority_spread.py create mode 100644 cuebot/scheduler-sim/inject_priority_starve.py create mode 100644 cuebot/scheduler-sim/kill_all_jobs.py create mode 100644 cuebot/scheduler-sim/limit_watch.py create mode 100644 cuebot/scheduler-sim/live_stats.py create mode 100644 cuebot/scheduler-sim/metrics.py create mode 100644 cuebot/scheduler-sim/priority_spread_watch.py create mode 100644 cuebot/scheduler-sim/priority_starve_watch.py create mode 100644 cuebot/scheduler-sim/register_hosts.py create mode 100644 cuebot/scheduler-sim/resolve_local.c create mode 100644 cuebot/scheduler-sim/rqd_complete.py create mode 100644 cuebot/scheduler-sim/rqd_report.py create mode 100755 cuebot/scheduler-sim/setup.sh create mode 100644 cuebot/scheduler-sim/sim_mem.py create mode 100644 cuebot/scheduler-sim/sim_metrics.py create mode 100644 cuebot/scheduler-sim/sim_model.py create mode 100644 cuebot/scheduler-sim/sim_seed.sql create mode 100644 cuebot/scheduler-sim/simulate.py create mode 100644 cuebot/scheduler-sim/stats.py create mode 100644 cuebot/scheduler-sim/status_pinger.py create mode 100644 cuebot/scheduler-sim/status_pinger_fast.py create mode 100644 cuebot/scheduler-sim/strand_dur_watch.py create mode 100644 cuebot/scheduler-sim/strand_watch.py create mode 100644 cuebot/scheduler-sim/util_test.py diff --git a/cuebot/scheduler-sim/.gitignore b/cuebot/scheduler-sim/.gitignore new file mode 100644 index 0000000000..32955b51aa --- /dev/null +++ b/cuebot/scheduler-sim/.gitignore @@ -0,0 +1,7 @@ +# Generated by setup.sh / simulate.py — never commit these. +venv/ +opencue_proto/ +sim_hosts +scheduler_sim.yaml +resolve_local.so +*.log diff --git a/cuebot/scheduler-sim/BUILD.md b/cuebot/scheduler-sim/BUILD.md new file mode 100644 index 0000000000..e8ea219e69 --- /dev/null +++ b/cuebot/scheduler-sim/BUILD.md @@ -0,0 +1,78 @@ +# Building & running cuebot in this dev box (hard-won notes) + +This box has a toolchain trap. These are the exact steps that work. + +## The toolchain trap +- The repo pins **Gradle 7.6.2** (cuebot/gradle/wrapper). It does NOT run on the + box's default **JDK 21** ("Unsupported class file major version 65" — bundled + ASM too old). The standalone **Gradle 8.14.3** in /opt runs on 21 but is too + new for the Spring Boot 2.2.1 plugin ("ArchivePublishArtifact"). +- Correct combo: **wrapper Gradle 7.6.2 + JDK 17**. + +## JDK 17 (with the proxy CA) +A vanilla JDK 17 download can't fetch deps: the env's outbound proxy uses a TLS +CA that vanilla cacerts don't trust (Gradle reports "plugin not found"). Fix: +copy the managed JDK 21 truststore into the JDK 17. +``` +# /tmp/jdk-17.0.2 was unpacked from openjdk-17.0.2_linux-x64; then: +cp /usr/lib/jvm/java-21-openjdk-amd64/lib/security/cacerts /tmp/jdk-17.0.2/lib/security/cacerts +``` + +## Repos: drop the dead ones (build-time only, do NOT commit) +cuebot/settings.gradle (pluginManagement) and build.gradle list `jcenter()` and +`repo.spring.io/plugins-snapshot`, which are dead and break resolution on 7.6.2. +Strip them before building: +``` +# in cuebot/: remove the 'maven { url ".../plugins-snapshot" }' line and 'jcenter()' lines +``` + +## Postgres (must run as non-root; refuses root) +``` +# run these as your own (non-root) user; postgres refuses root, no sudo needed +PGBIN=/usr/lib/postgresql/16/bin +rm -rf /tmp/pgdata && mkdir -p /tmp/pgdata /tmp/pgrun +$PGBIN/initdb -D /tmp/pgdata -U cue --auth=trust +$PGBIN/pg_ctl -D /tmp/pgdata -o "-p 5433 -k /tmp/pgrun -c listen_addresses=127.0.0.1" -l /tmp/pg.log start +$PGBIN/psql -h127.0.0.1 -p5433 -Ucue -dpostgres -c "CREATE DATABASE cuebot;" +# apply migrations in version order: +cd cuebot/src/main/resources/conf/ddl/postgres/migrations +for f in $(ls *.sql | sort -t_ -k1.2 -n); do $PGBIN/psql -h127.0.0.1 -p5433 -Ucue -dcuebot -v ON_ERROR_STOP=1 -q -f "$f"; done +# base data: dept/services/config from seed_data.sql + scheduler-sim/sim_seed.sql +``` + +## Build / run cuebot (as your own user, JDK 17, wrapper 7.6.2) +A dedicated gradle home /tmp/ghome-$USER holds the resolved deps. Build as the +user that owns the checkout (a fresh `git clone` already is); no specific account +is required. Remove cuebot/.gradle if you hit `checksums.lock (Permission denied)`. +``` +cd cuebot && env \ + CUEBOT_DB_URL="jdbc:postgresql://127.0.0.1:5433/cuebot" CUEBOT_DB_USER=cue CUEBOT_DB_PASSWORD= \ + SCHEDULER_ENABLED=true SCHEDULER_INTERVAL_MS=250 SCHEDULER_RESERVATIONS_ENABLED=false \ + ./gradlew bootRun -g /tmp/ghome-$USER -Dorg.gradle.java.home=/tmp/jdk-17.0.2 --console=plain >/tmp/cuebot.log 2>&1 +``` +gRPC serves on **8443**. Compile-only check: swap `bootRun` for `compileJava` +(note `-Werror -Xlint:all` is on — warnings fail the build). Unit tests: +`./gradlew test --tests "...SchedulerTests"`. + +## CRITICAL launch pattern (process management) +Launch long-running procs (cuebot, pinger, fake_rqd) with the Bash tool's +`run_in_background: true` and **NO inner `&`**. An inner `&` double-backgrounds +and the JVM/Python gets SIGKILLed when the wrapper shell exits. +After a cuebot restart, restart status_pinger.py too (its gRPC channel goes +stale -> all ReportStatus fail -> hosts age to DOWN). + +## Reset the farm between runs +``` +psql ... -c "DELETE FROM proc;" +psql ... -c "UPDATE host SET int_cores_idle=int_cores,int_mem_idle=int_mem,int_gpus_idle=int_gpus,int_gpu_mem_idle=int_gpu_mem;" +psql ... -c "UPDATE subscription SET int_cores=0,int_gpus=0;" +# clear the job backlog (frames then layers) for the sim show: +psql ... -c "DELETE FROM frame f USING job j WHERE f.pk_job=j.pk_job AND j.pk_show='10000000-0000-0000-0000-000000000003';" +psql ... -c "DELETE FROM layer l USING job j WHERE l.pk_job=j.pk_job AND j.pk_show='10000000-0000-0000-0000-000000000003';" +``` + +## Gotcha: "unable to allocate additional memory" +That is NOT a Postgres OOM. It's `trigger__verify_host_resources` raising when a +booking pushes a host's int_*_idle below 0 (overbooking protection). Treat it as +an overbooking/accounting signal, not a memory problem. +``` diff --git a/cuebot/scheduler-sim/README.md b/cuebot/scheduler-sim/README.md new file mode 100644 index 0000000000..6820f2027d --- /dev/null +++ b/cuebot/scheduler-sim/README.md @@ -0,0 +1,283 @@ +# Scheduler farm simulator + +A DB-backed integration harness for the cuebot `Scheduler`. It is not a model of +cuebot, it **is** cuebot: a **real cuebot + Postgres** driven over gRPC by a +**fake render farm**, so every booking goes through the exact production path +(the real `Scheduler`, the real SQL, the real frame-complete handler) — no +database writes from the driver. Hosts register like RQD, jobs are submitted like +a client, and a fake RQD runs/completes frames. It is the integration test unit +tests cannot be, and the place to observe behaviour that only shows up under +load: utilization, co-locality, throughput, reservations, dependency handling, +and big-job placement. + +This exercises the *real* booking path end to end and has already surfaced +several bugs. + +## Pieces +| file | role | +|------|------| +| **`simulate.py`** | **one command: tears down, resets DB, brings the whole stack up fresh, starts a workload** | +| `farm_spec.py` | the farm: 246 large/128c, 303 medium/32c, 1004 small/16c; mem 4 GB/core | +| `sim_model.py` | frame cores/mem/duration distribution (from real-farm CSVs); `SIM_COMPRESS` env scales durations | +| `sim_seed.sql` | one-time base data: facility/alloc/show/subscription | +| `register_hosts.py` | register all hosts via RQD ReportRqdStartup | +| `rqd_report.py [int]` | faithful host **+ running-frame** status heartbeat; refreshes `proc.ts_ping` so the 300s orphan sweep behaves like prod | +| `status_pinger.py` / `status_pinger_fast.py` | older empty-frame heartbeat (kept; superseded by `rqd_report.py`) | +| `fake_rqd.py [threads]` | fake RQD gRPC server on :8444; runs+completes frames. Used by new, old, and `rust --rust-real-launch`. `threads`=completion-report concurrency (1=serial, 64=concurrent RQDs) | +| `rqd_complete.py [int] [memfail]` | default `--mode rust`: polls the proc table for frames the Rust scheduler booked (dry-run) and reports them complete to cuebot after their `sim_model` run-time — the DB-poll analogue of `fake_rqd.py` | +| `gen_jobs.py` | submit a realistic job mix via LaunchSpec | +| `feed.py [dur] [target]` | paced feeder: hold a sustained backlog of ~`target` waiting frames | +| `drain_test.py