bundle: record and read deployment state via DMS - #6094
Open
shreyas-goenka wants to merge 182 commits into
Open
Conversation
…d IDs Wire the direct engine into the Deployment Metadata Service (DMS) so that a `record_deployment_history`-enabled bundle records each deploy/destroy as a version and can read its resource state back from DMS. The deployment ID is now assigned by the server: the first deploy calls CreateDeployment with an empty ID, reads the assigned ID back from the response, and persists it in the direct-engine state header (Header.DeploymentID). Later deploys pass the stored ID back, so a bundle maps one-to-one to a DMS deployment even after the local cache is deleted (the ID rides along in the workspace-synced state file). - libs/dms: Recorder creates the deployment (server-assigned ID) + version, heartbeats the lease, completes it, and deletes the deployment on destroy. - bundle/direct: operationRecorder reports each applied resource operation; the wire resource_key drops the CLI-internal "resources." prefix. - bundle/direct/dstate: Open takes a DMS client and overlays DMS resource state when DMS holds a successful version; deployment ID persisted in the header. - bundle/phases: create the version after plan approval, complete it under the lock, record operations during apply. - libs/testserver: stateful fake DMS (deployments/versions/operations/resources) with server-generated IDs; acceptance test covers deploy, cache-loss redeploy, and destroy. Co-authored-by: Isaac
The read overlay decided whether DMS owns a deployment's state by listing versions and scanning for a successful one. The deployment now exposes last_successful_version_id directly, so a single GetDeployment answers the same question — no version listing. The field is still stage:DEVELOPMENT in the proto and therefore stripped from the generated SDK, so this reads the deployment via a raw GET into a local struct as a temporary stub. Once the field is promoted to PRIVATE_PREVIEW and regenerated, the raw call collapses to client.GetDeployment(...). LastSuccessfulVersionId and the threaded config argument goes away (see the TODO in deploymentHasSuccessfulVersion). The testserver's GetDeployment now serves last_successful_version_id (tracked on version completion), and the now-unused ListVersions fake is removed. Co-authored-by: Isaac
Five fixes to the DMS state recording added in #6052: 1. The fake DMS server dropped last_successful_version_id. GetDeployment serialized the response through a struct embedding bundledeployments.Deployment, whose promoted MarshalJSON silently discards sibling fields. The CLI reads a missing value as "DMS does not own the state", so the entire read/overlay path (overlayDMSState, fetchDeploymentResources, deploymentHasSuccessfulVersion) never ran in any test. Serialize through a map instead, and unit-test the shape. 2. A bundle with no resources leaked a deployment record per deploy. Such a deploy writes no WAL entries, and the state file was only persisted when the WAL carried entries, so the server-assigned deployment ID was dropped and the next deploy created a second deployment. Track a dirty header so Finalize persists an ID change on its own. A header-only WAL that changed nothing still skips the write, keeping the serial in step (acceptance/bundle/deploy/wal/header-only-wal). 3. Deploy after destroy failed permanently. A successful destroy deletes the deployment record but leaves its ID in local state, so the next deploy's GetDeployment 404'd and the error was fatal — unrecoverable on retry. Treat a missing deployment as "create a new one"; any other read error stays fatal. 4. The overlay dropped depends_on. DMS does not record dependency edges, so replacing local state with DMS resources lost them, affecting delete ordering, the apply graph, and --select expansion. Carry depends_on over from the local entry. Masked by (1) until now. 5. Recording bypassed secret redaction. dstate.SaveState redacts bundle:"sensitive" fields before writing state, but the operation recorder marshalled raw, so a secret would be sent to DMS in plaintext and read back into local state. Route through structwalk.RedactSensitiveFields. Latent today: no resource state type carries a sensitive field yet. Adds acceptance coverage for deploy-destroy-deploy and for a bundle with no resources, both of which now exercise the read path (visible as GET .../resources in the recorded requests). Co-authored-by: Isaac
Recording an operation with the deployment metadata service used to happen inline on the apply worker, so every resource paid a CreateOperation round trip before the worker moved on to the next one. Queue the operations instead and upload them from a small pool of background workers (operationQueue, in the new opqueue.go). The queue holds resource keys rather than operations, so an operation recorded for a resource that is still waiting replaces the queued one: DMS keeps one state per resource key, so the later operation supersedes the earlier one and a single request records both. This is best effort - only operations that no worker has picked up yet are coalesced. Uploads are not fire-and-forget. Apply drains the queue before returning and reports the first failure, because a version that completes successfully makes DMS authoritative for resource state; dropping an operation would leave DMS with an incomplete resource set and the next deploy would plan to create resources that already exist. At most one upload per resource key runs at a time, so the last operation recorded for a resource is also the last one the service sees. Co-authored-by: Isaac
Recording deployment history is implemented end to end, but it cannot be exposed to users yet: enabling it makes the deployment metadata service the source of truth for resource state, and there is no upgrade path from an existing direct-engine state file to a DMS-owned one. Setting the flag is now an error. DATABRICKS_BUNDLE_ENABLE_RECORD_DEPLOYMENT_HISTORY lifts the error so the CLI's own acceptance tests and DMS development can exercise the feature until the direct state upgrade lands. Co-authored-by: Isaac
DATABRICKS_BUNDLE_ENABLE_RECORD_DEPLOYMENT_HISTORY read as if it were the switch that turns the feature on. It is not: the flag in databricks.yml does that, and this variable only permits the flag to be set while the feature is gated off. Name it DATABRICKS_BUNDLE_FORCE_ALLOW_RECORD_DEPLOYMENT_HISTORY. Co-authored-by: Isaac
Replace the blanket gate on experimental.record_deployment_history with a narrower check in dstate.DeploymentState.Open: recording is refused only when the state file already tracks deployed resources that DMS does not know about. Once DMS holds a successful version it is authoritative for resource state even when its resource set is empty, so adopting a state file written by a CLI that predates DMS would make already-owned resources look absent and create them a second time. A state file that DMS already owns is fine, and so is one with no resources (e.g. left behind by a destroy), which is what makes the error's destroy-and-redeploy advice work. The check keys off len(State) rather than the file existing because destroy leaves resources.json in place with an empty resource set. This drops validate.ValidateRecordDeploymentHistory and the DATABRICKS_BUNDLE_FORCE_ALLOW_RECORD_DEPLOYMENT_HISTORY escape hatch, which are no longer needed. Upgrading an existing state in place (state v3 with a feature flag plus per-resource tombstones so older clients refuse the state) is left as a TODO. Co-authored-by: Isaac
The concurrent-producer test tripped three linters: - modernize/revive want wg.Go instead of wg.Add + go func + defer wg.Done. - testifylint's go-require flags recordState, which calls require inside the spawned goroutines. testify assertions may only run on the goroutine running the test function. Record inline and send each error to a buffered channel that the test goroutine drains after wg.Wait, so the assertions stay on the test goroutine. Co-authored-by: Isaac
The operation queue collapses repeated writes to the same resource key by replacing the queued operation wholesale, so a create followed by an update was uploaded as an update. That tells DMS the resource already existed before this deploy, when in fact this deploy created it. Merge the actions instead: the state uploaded is still the later one, but a queued create or recreate wins over a subsequent update. A delete still wins over anything queued before it, since the resource is gone. This is not reachable from Apply today - each resource is recorded once per deploy, because there is one record call per graph node and dagrun visits each node exactly once - so this is about the queue being correct for any caller that records a resource more than once. Co-authored-by: Isaac
Co-authored-by: Isaac
The deployment ID was persisted in the local state file header, which made
the CLI the source of truth for a value the service mints. DMS registers
each deployment as a workspace node named resources.deployment.json under
initial_parent_path, and the node's ID *is* the deployment ID, so it can be
resolved from the workspace instead.
ResolveDeploymentID does a get-status on <state_path>/resources.deployment.json
and returns the node ID, or empty when the node is absent. Deploy, destroy,
and the read path all resolve the ID that way and pass it down; the read path
then constructs state from GetDeployment + ListResources as before.
Consequences:
- Header.DeploymentID, GetDeploymentID, and SetDeploymentID are gone,
along with the headerDirty machinery that only existed to persist the ID
on a resource-less deploy.
- Open takes a *DMSSource instead of a (client, config) pair, since the
resolved ID now has to be threaded in too.
- CreateDeployment sets initial_parent_path, which the service requires
and the CLI never set.
- createDeploymentVersion no longer recovers from a 404 on GetDeployment by
creating a second deployment. A destroy trashes the node, so a resolved
ID whose record is missing means the two are out of sync, and creating
another deployment would collide on the same node path.
The testserver models the real derivation: CreateDeployment creates the
workspace node and uses its object ID as the deployment ID, so the acceptance
tests exercise get-status resolution end to end. dms/record now wipes the
local cache before redeploying and records zero operations, which is the read
path reconstructing state entirely from DMS.
Co-authored-by: Isaac
- Limit recorded state to 64 KB, checked when the payload is built so an
oversized resource fails itself rather than the drain at close.
- Collapse the operation queue's pending/inflight pair into one `owned` set.
The two maps encoded a single question ("is this key already claimed?") and
had to be read together; `take` now only releases ownership.
- Only log "Coalescing" when an operation was actually merged. The old code
logged it for in-flight keys too, where nothing was coalesced.
- Restore mergeWalIntoState's `hasEntries` naming and comment from main. The
`persist` rename existed for the headerDirty case, which is gone.
- Trim the comments added by this PR.
Also note in fetchDeploymentResources that DMS has no field for dependency
edges and they cannot be recovered from the recorded state (references are
resolved to literals before serialization), so depends_on is carried over from
the local state file.
Co-authored-by: Isaac
The read path carried depends_on over from the local state file, which is empty exactly when it matters: a fresh checkout reconstructs state from DMS and got no dependency edges. Deletes are the one case that cannot recompute them, because the resource is gone from config, so two dropped resources with a real dependency could be deleted in the wrong order. DMS has no field for dependency edges, and they cannot be recovered from the recorded config either: references are resolved to literals before it is serialized. So Operation.State now carries an envelope, dstate.RecordedState, holding the config plus depends_on. Nesting depends_on inside the config would have collided with resource fields of the same name (jobs.Task.depends_on). The envelope mirrors the local ResourceEntry, so both sides of the round trip have the same shape. acceptance/bundle/dms/depends-on covers it: a job referencing another records its edge, and after wiping the local state a destroy still deletes the referencing job first. Co-authored-by: Isaac
Migrating an existing deployment is not supported: Open rejects a bundle that already has resources in state, so a deployment that exists in DMS was created by an opted-in CLI and DMS owns its resource set outright. The last_successful_version_id probe that decided whether to trust DMS was therefore always true by the time it ran. Removing it takes with it the raw GET that read the field (it is stage:DEVELOPMENT and stripped from the generated SDK) and DMSSource.Config, which existed only to make that call. overlayDMSState is now readDMSState, since it no longer overlays anything conditionally: it just reads. Recording stays opt-in via experimental.record_deployment_history; that flag is what makes the caller pass a DMSSource at all. acceptance/bundle/dms/existing-state also now covers wiping the local cache: deploy pulls the state file back from the workspace, so the resources stay tracked and opting in is still rejected. Co-authored-by: Isaac
Reading resource state from DMS now requires the state file to record a
"deployment_history" feature flag, rather than inferring eligibility from the
resource count.
The old check refused a state that had resources and no DMS deployment ID. That
happened to be right, but it inferred intent from a side effect: a state with
resources could equally be one DMS already owns. The flag says so directly.
Header.Features was already scaffolded for exactly this, so this fills it in:
- Open writes the flag when recording is enabled, and refuses a state that
has resources without it. Migrating such a target is not supported, so the
error names the target and the three ways out: use a new target, destroy
this one and redeploy, or unset the feature.
- A state recording any feature is written at featureStateVersion, so a CLI
that predates the flag refuses it (see migrateState) instead of deploying
against a resource set that lives in DMS and looks empty on disk.
- migrateState accepts features it implements and refuses only unknown ones,
naming just those in the error.
- WAL replay carries the features forward, so recovering a WAL from a
recording deploy still produces a state marked as DMS-owned.
The flag is per target, which matches how experimental.record_deployment_history
is set.
Co-authored-by: Isaac
Coalescing now keeps the newest operation outright instead of merging fields. Each operation carries the resource's full state rather than a delta, so a newer one entirely supersedes an older one - including its resource_id, which is the field that actually changes between two records (a create learns the ID only after the API call returns it). The previous code merged the action and overwrote the ID, which is backwards: the action cannot differ between records of the same resource, while the ID can. mergeAction is gone. Also renames `owned` to `queuedOrUploading`. Nothing is owned by a particular worker: a key can be handled by one worker, released, and picked up later by another. The mark only means "some worker will get to this", which is all record needs to know in order to not queue the key twice. The doc comments now lead with the two rules that shape the design (no overlapping uploads per resource; only the newest operation matters) so the mechanism reads as a consequence of them rather than as bookkeeping. Co-authored-by: Isaac
Collaborator
Integration test reportCommit: 200b7cc
Top 3 slowest tests (at least 2 minutes):
|
The dms tests pass workspace paths to $CLI (`workspace get-status /Workspace/...`). On Windows, Git Bash rewrites a leading-'/' argument into a Windows path before the CLI sees it, so the lookup goes to C:/Program Files/Git/Workspace/... and 404s, failing record, no-resources and redeploy-after-destroy on that platform only. Same fix and reason as acceptance/cmd/workspace/export-dir-*/test.toml. Co-authored-by: Isaac
Setting it in test.toml applied it to the whole test, which stopped Git Bash
converting the PATH too - so python3 could not find print_requests.py
("can't open file 'C:\\c\\a\\cli\\cli\\acceptance\\bin\\print_requests.py'")
and every dms test failed on Windows, including the three that were passing.
trace exports leading KEY=value pairs in a subshell, so setting it there fixes
the CLI's leading-'/' argument without reaching the helpers.
The precedent this was copied from
(acceptance/cmd/workspace/export-dir-*/test.toml) uses no python helpers, which
is why the test.toml form is safe there but not here.
Co-authored-by: Isaac
close runs on one goroutine after every apply worker has returned, so nothing else touches the queue by then and the closed flag needs no protection. The wg.Wait orders the upload workers' writes to err before it is read, so that read needs none either. Also tightens the coalescing test to count uploads for the resource instead of only checking the payload of the last one. It asserted the merged content but would have passed if the operations had been uploaded twice, which is the thing coalescing exists to prevent. Co-authored-by: Isaac
Drops the RedactSensitiveFields call, leaving a TODO: fields marked bundle:"sensitive" now reach DMS in plaintext, and the read path writes them back into the local state file unredacted. This has to be restored before the feature ships to users. Also documents why take leaves the key in queuedOrUploading when it hands an operation to a worker, and covers the case the comment describes: recording while that key's upload is in flight. The operation cannot join the in-flight request, so it is uploaded next by the same worker rather than being dropped or picked up concurrently by a second one. Co-authored-by: Isaac
Co-authored-by: Isaac
Drops the record_deployment_history annotation change (and the generated schema that followed from it), leaving both files as they are on main. CompleteVersion now keys its no-op on versionNum rather than on the heartbeat handle. Both are set together by CreateVersion, so the behaviour is the same - a deploy that was cancelled, or whose CreateVersion failed, does not complete a version that was never created - but the check now names the thing it is actually guarding. Callers defer CompleteVersion unconditionally, so this is the only thing standing between a failed CreateVersion and a CompleteVersion call against a nonexistent version. Co-authored-by: Isaac
Restores validate.ValidateRecordDeploymentHistory and the hidden DATABRICKS_BUNDLE_FORCE_ALLOW_RECORD_DEPLOYMENT_HISTORY escape hatch, so setting experimental.record_deployment_history is an error unless that variable is set. The service side is not ready for users: DMS is only deployed to dev and staging, and reading state back needs the workspace APIs to expose the deployment's tree node, which is still behind a flag. With the flag unreachable, the state feature flag added earlier is not needed yet, so dstate is back to the resource-count check in Open. The deployment_history feature, hasFeature/setFeature, the version bump on write and the WAL carry-over all come back with the state upgrade in a follow-up. The dms acceptance tests force allow the flag, and bundle/dms/not-supported covers the error users see. Operation requests in bundle/dms/depends-on print multi-line now: the state envelope nests two levels, which --oneline made unreadable. Co-authored-by: Isaac
An upload failure was only reported at close, so a DMS outage let apply deploy every remaining resource and fail at the end. That leaves resources in the workspace that DMS has no record of, and since a completed version makes DMS the source of truth for resource state, the next deploy would create them again. record now returns the first upload error, which the apply worker turns into a failed node, so the deploy stops shortly after the failure instead of running to completion. It refuses new work only: operations already recorded still upload, because close drains them, so the records DMS ends up with match the resources that were actually applied. Resources already mid-apply also finish. Also repeats the one test whose bug depends on a scheduler interleaving rather than on a forced handshake, so a single run gets many chances to hit the bad ordering. The other tests pin their interleaving with the started/block channels, so repetition would not add coverage; the new error tests use a `done` channel to wait for an upload to have finished rather than merely started. Co-authored-by: Isaac
The previous commit made record return the upload error, but record runs after the resource has already been created or updated, so every node that started before the failure was noticed still modified the workspace. Apply now checks for a recorded failure before it touches anything, right after the dependency check, so a node that has not started yet is refused rather than applied. Resources already mid-apply still finish - the check cannot unwind those - but the deploy no longer runs to completion against a service that is rejecting its records. acceptance/bundle/dms/operation-upload-fails covers it. Which resources get refused depends on how far apply got before a background upload failed, so the per-resource errors go to a LOG file and requests are not recorded; the test asserts the deploy fails and reports the upload error. Also drops --sort from the dms tests that deploy zero or one resource: their request order is already deterministic, and the unsorted output reads in chronological order. Co-authored-by: Isaac
shreyas-goenka
commented
Jul 29, 2026
| // of and the next deploy would create them a second time. Checked here rather | ||
| // than only where operations are recorded, which is after the resource has | ||
| // already been modified. | ||
| if err := opQueue.firstErr(); err != nil { |
Contributor
Author
There was a problem hiding this comment.
we could eventually extend this to record and return all multiple errors that happened.
shreyas-goenka
commented
Jul 29, 2026
| @@ -0,0 +1,4 @@ | |||
|
|
|||
| === An operation upload failure fails the deploy instead of reporting only at the end | |||
Contributor
Author
There was a problem hiding this comment.
it's hard to make a assert more here because we cannot control how many requests went through. We could harden this test by making the number of workers configurable and 1. Omitting for now.
CreateDeployment now only registers the workspace node whose ID names the
deployment; the record itself is created by the first CreateVersion. A client
that registers a deployment and then fails before recording a version leaves
just the node behind, not an empty deployment.
That state is reachable, so both sides of the CLI handle it:
- the recorder starts at version 1 under the ID the node already names,
instead of failing on the missing record or creating a second deployment
that would collide on the same node path
- the read path keeps the local (empty) state instead of surfacing the 404
from ListResources
acceptance/bundle/dms/version-never-created covers it end to end: the first
version fails, and the next deploy reuses the same deployment ID.
Also drops libs/testserver/bundle_test.go. The fake is exercised by every dms
acceptance test, so unit tests for it only duplicate that coverage.
Co-authored-by: Isaac
shreyas-goenka
commented
Aug 26, 2026
| "sequence_id": "0", | ||
| "state": "", | ||
| "status": "OPERATION_STATUS_IN_PROGRESS" | ||
| "status": "OPERATION_STATUS_PENDING" |
Contributor
Author
There was a problem hiding this comment.
we can eventually make this an IN_PROGRESS status once we support this.
Contributor
There was a problem hiding this comment.
Could you put this comments (with some more context) in the code? They will get lost if it's only on the PR.
shreyas-goenka
commented
Aug 26, 2026
| @@ -52,6 +52,16 @@ func NewStateUpdate(resourceID string, state json.RawMessage, inProgress bool) ( | |||
| status = StatusPending | |||
Contributor
Author
There was a problem hiding this comment.
This will eventually become StatusInProgress once the backend supports it.
shreyas-goenka
force-pushed
the
isaac/pr6052-fixes
branch
from
August 26, 2026 00:33
8360fe4 to
d7ab4a2
Compare
Recording a delete sent the same mask as every other write, with an empty state and the id of what had just been removed. A real workspace refuses that: a delete that succeeded has no resource left to describe, so an empty state is a state it will not take, and an id is only accepted alongside one it was given. Destroy therefore failed for every bundle that records history - the resources went, the operation stayed pending, and the version completed as a failure. The service reads a masked state with no value as a clear, which is also what drops the resource from the deployment. So a delete now names state in the mask and leaves the value out, and does not name the id. Verified against a real workspace: deploy then destroy both complete VERSION_COMPLETE_SUCCESS, the deployment ends DELETED, and a delete of one resource among several drops only that one from the deployment. The fake took the old shape without complaint, which is why this only ever failed against a workspace. It now reads a masked state with no value as the clear the service reads it as, and counts only a state it was given when deciding whether an id may accompany it. Co-authored-by: Isaac <no-reply@databricks.com>
shreyas-goenka
force-pushed
the
isaac/pr6052-fixes
branch
from
August 26, 2026 00:53
d7ab4a2 to
435683c
Compare
A failed UPDATE left its operation PENDING with no error. The recorder sent update_mask=error_message,status, and DMS refuses a failure that would leave an operation acting on a live resource described by nothing, so the write bounced and the error was dropped. Re-state what the resource still has, with the id the service requires to travel alongside it. An empty resource id means there is nothing left to describe -- a create that never landed, a recreate whose delete already went through -- and those stay state-less. libs/testserver accepted the state-less write, which is why every local test passed; it now enforces the same rule, verbatim error included. Reproduced and verified on Azure dogfood: the operation now ends OPERATION_STATUS_FAILED carrying the 404 from jobs/reset. Co-authored-by: Isaac <no-reply@databricks.com>
The DMS env matrix this branch adds applies to every bundle test, so tests that landed on main since the last merge need it materialized. Co-authored-by: Isaac <no-reply@databricks.com>
bundle/dms/record now prints GETs alongside the writes. A fresh deploy makes no reads; once the deployment exists each run makes exactly two, and an extra round-trip in any phase shows up as a diff in this golden. Co-authored-by: Isaac <no-reply@databricks.com>
A permissions node carries a path for its resource ID and its permission list as state, neither of which the existing failure tests exercise. Pins that such a failure records the state it still has rather than bouncing off the service. Co-authored-by: Isaac <no-reply@databricks.com>
The suite only ever ran against the fake, which is laxer than the service, so nine of
these cases could not have passed on cloud. Turning Cloud on for the subtree:
- Every bundle name now carries $UNIQUE_NAME, so each test gets its own root path and
the prefix sweep in bundle_clean_test.go reclaims it. Without that the tests inherited
each other's deployments between runs and would fight the nightly for the same lock.
- UC schema names carry it too: those are global, so a second run hit SCHEMA_ALREADY_EXISTS.
- The five cases that force a failure with a [[Server]] stub stay local. The cloud
harness does not pass config.Server to its proxy, so a stub there is silently ignored
and the failure the test asserts never happens.
- The fake registered the deployment node as a plain FILE; get-status reports
BUNDLE_DEPLOYMENT, which is what the goldens now record.
Co-authored-by: Isaac <no-reply@databricks.com>
A test that forces a failure cannot run against a real workspace: fault.py registers its
rule on the fake, and a [[Server]] stub is dropped because the cloud harness never passes
config.Server to its proxy. Either way the failure the test asserts never happens.
So the suite is split. The nine cases that need an injected failure say Cloud = false and
say why. Simulating them for real does not work either: pointing at a catalog or a
principal that does not exist fails on cloud but not locally, because the fake accepts
both, and teaching it otherwise would change every test that uses main without creating it.
The rest now run on cloud, which needed fixtures the real APIs accept:
- A pipeline carries a library. The API rejects one without.
- Grants name deco-test-user@databricks.com, the principal the test envs have.
- A recreate moves its schema into a catalog the test creates and drops, rather than a
catalog named "other" that does not exist.
Co-authored-by: Isaac <no-reply@databricks.com>
The path is pinned so renaming the bundle keeps the same deployment, which is the point of the test. Sharing it across runs is not: a previous run leaves a schema under it that this run then plans to delete, and a destructive change with no console to approve it fails the deploy. Co-authored-by: Isaac <no-reply@databricks.com>
Recording deployment history is matrixed over every bundle test, which doubles all of them and rewrites 927 out.test.toml files. That belongs with enabling those runs, not with the feature, so it moves to the change that turns them on. The tests under bundle/dms cover the recording itself and stay here. Co-authored-by: Isaac <no-reply@databricks.com>
Recording a deployment stamps every job and pipeline, so the suite-wide run needed a --nostamp filter, a nostamp() helper and a handful of matrix excludes to keep 100 unrelated goldens byte-identical. None of that belongs with the feature: it exists to make the other tests tolerate the stamp, so it moves to the change that turns those runs on. print_requests.py keeps --dms, which the tests under bundle/dms filter on, and loses --nostamp, which only they do not use. Co-authored-by: Isaac <no-reply@databricks.com>
Three things the tests got wrong, all of them here rather than in the change that turns the suite-wide run on, because the fixtures are here. The pipeline in no-drift was named bar. Unlike jobs, the Pipelines API refuses a duplicate name, so two overlapping cloud runs collided with RESOURCE_CONFLICT. It carries $UNIQUE_NAME now, as the bundle and the schemas already did. The catalog cleanup a recreate borrows ran under `&>> LOG.catalog`, which bash gained in 4.0. macOS CI runs 3.2, so the EXIT handler failed to parse and took out_requests.txt with it. `>> LOG.catalog 2>&1` reads the same on both, matching the note already in bundle/resources/volumes/uppercase-name/script. The comment headers added above Cloud = false left a trailing blank line, which the whitespace check rewrites. Co-authored-by: Isaac <no-reply@databricks.com>
Earlier in this branch the script piped the recorded-request jq through nostamp. In a pipeline the trailing redirect binds to the last command, so trace's own ">>> ..." echo stopped landing in out.requests.$ENGINE.txt and leaked to the test's stdout instead, and the regenerated goldens moved those two lines from out.requests.*.txt into output.txt. Dropping nostamp again put the echo back where it came from, and out.requests.*.txt was restored to match, but output.txt kept the two lines it should never have had. Every test cell diffed on that file, which is why all nine were red on a change that has nothing to do with this test. Co-authored-by: Isaac <no-reply@databricks.com>
denik
reviewed
Sep 1, 2026
| } | ||
|
|
||
| // Keep going after a failure, so one bad write does not drop everything behind it. | ||
| if err := c.write(ctx, resourceKey, update); err != nil { |
Contributor
There was a problem hiding this comment.
does calling Create/UpdateOperation resets the timeout timer on the backend as well, thus behaving like a hearbeat?
Perhaps we can use a single thread combining operations and heartbeats?
have something in queue? record operations
no action for N seconds? send heartbeat
then it naturally skips heartbeats if there are enough operations going through it.
it also becomes a single place to notice when backend is down and deployment should be stopped.
Currently both threads have to have this logic but to make a good decision you need both:
- if heartbeat endpoint is not working but operations going through at regular intervals, we can continue with deployment).
- if heartbeat endpoint is working but operations endpoint report retriable errors, we can wait for operations for longer time, since we know that versions is kept open.
Note, I recognize this adds scope to this PR -- we can do it as follow up but in this case let's keep current PR simple and remove heartbeats completely.
Contributor
Author
There was a problem hiding this comment.
does calling Create/UpdateOperation resets the timeout timer on the backend as well, thus behaving like a hearbeat?
Not yet. We could make that the behaviour if necessary.
We can't removed heartbeat, by design the backend requires them. We can defer this as a future optimizations. heartbeats are cheap API calls made every 30 seconds, so I'm not super concerned about them yet. We'll get a lot more operation API calls than heartbeats.
Contributor
There was a problem hiding this comment.
Can you add as a comment what is the exact requirement backend has for heatbeats? and what happens if it's not met.
Contributor
Author
There was a problem hiding this comment.
Removed the heartbeat logic as discussed.
shreyas-goenka
force-pushed
the
isaac/pr6052-fixes
branch
from
September 2, 2026 02:05
6eb7a3c to
a5b1a71
Compare
- Trim the buffered client to operation buffering; the deploy/destroy phases call dms.Client directly for CreateDeployment/CreateVersion/CompleteVersion. - Drop the deployment-history mutator and the FORCE_ALLOW env var; set bundle.deployment.history inline and read the deployment once, explicitly. - Destroy no longer creates or updates the deployment, and skips when none exists. - Stop storing the deployment record on DeploymentBundle. Co-authored-by: Isaac <no-reply@databricks.com>
shreyas-goenka
force-pushed
the
isaac/pr6052-fixes
branch
from
September 2, 2026 02:30
a5b1a71 to
58257c8
Compare
Drop the background heartbeat that renewed the version lease, the Client.Heartbeat method, and its Stop() call site. Co-authored-by: Isaac <no-reply@databricks.com>
Stamp the DMS deployment_id and version_id a plan targets into the saved plan, and reject it at deploy --plan time if the deployment changed or advanced past that version. Closes a replay hole: a first-deploy plan has empty lineage, so the existing serial check skips it. The fields are gated on the experimental.record_deployment_history opt-in, so the DMS test matrix (which forces recording via env var) stays transparent. Bumps plan_version to 3. Co-authored-by: Isaac <no-reply@databricks.com>
shreyas-goenka
force-pushed
the
isaac/pr6052-fixes
branch
from
September 2, 2026 13:11
84eb12b to
faf0962
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds code to read and write state using DMS, behind
DATABRICKS_BUNDLE_RECORD_DEPLOYMENT_HISTORY.Design decisions:
CreateVersionstages one operation per planned resource, so the CLI only ever callsUpdateOperation— there is noCreateOperationcall. The service creates each staged operation asPENDINGatsequence_id = 0, which is the precondition the CLI uses for its first update of a resource. Needs databricks-eng/universe#2420238 (merged).Also adds an env var to toggle DMS, and records the API status and error code with a failure.
Testing Strategy
The whole acceptance suite runs a second time with recording on (
EnvMatrix.DATABRICKS_BUNDLE_RECORD_DEPLOYMENT_HISTORY), so every bundle test exercises DMS against the local test server and asserts the same golden files either way — thenostamphelper strips the deployment stamp for that. Focused coverage of the recorded calls themselves lives underacceptance/bundle/dms.Before private preview we will:
What is missing?