Support tgz artifacts built from a directory or git ref

[vinchenzo-db]

Adds a tgz artifact type that packages source files into a gzipped tarball as part of bundle deploy, alongside the existing whl and jar types. Unlike those, DABs builds the tarball itself — there is no user build command.
Two sources are supported:

• include: pack paths under the artifact's path from the working tree (.gitignore honored).
• git: snapshot a committed ref (branch/commit) via git archive.

Archive entries are named relative to the artifact's path, so the layout is controlled by config. The built tarball is uploaded and referenced like any other artifact file.

artifacts:
  code:
     type: tgz
     path: .
     include: [src]
     files:
       - source: ./dist/code.tgz

Co-authored-by: Isaac

Changes

Why

Tests

Show that the include field is respected, we notice the ./dist/code.tgz is properly uploaded.

 v.chen at https://dbc-04ac0685-8857 in ~/.worktrees/cli-artifacts (git:air-artifacts-code-source) [20:04:39]
$ rm -rf /tmp/demo-include && mkdir -p /tmp/demo-include/src && cd /tmp/demo-include
printf 'print("hello from packaged code")\n' > src/train.py
printf '#!/bin/bash\ncd "$(dirname "$0")"\npython src/train.py\n' > command.sh
cat > databricks.yml <<YAML
bundle: {name: demo-include}
targets: {dev: {mode: development, default: true, workspace: {host: $HOST}}}
artifacts:
  code_source:
    type: tgz
    include: [src]
    files: [{source: ./dist/code.tgz}]
resources:
  jobs:
    demo_include:
      name: demo-include
      tasks:
        - task_key: train
          environment_key: default
          ai_runtime_task:
            experiment: demo-include
            deployments: [{command_path: ./command.sh, compute: {accelerator_type: GPU_1xA10, accelerator_count: 1}}]
            code_source_path: ./dist/code.tgz
      environments: [{environment_key: default, spec: {environment_version: "5", dependencies: []}}]
YAML

$CLI bundle deploy -t dev                       # → "Building code_source..." → "Uploading dist/code.tgz..."
tar tzf dist/code.tgz                            # → src/train.py   (DABs built it from ./src)
JID=$($CLI bundle summary -t dev | grep -oE 'jobs/[0-9]+' | head -1 | cut -d/ -f2)
$CLI jobs get $JID | grep code_source_path       # → rewritten to /Workspace/.../artifacts/.internal/code.tgz
$CLI bundle destroy -t dev --auto-approve

Building code_source...
Uploading dist/code.tgz...
Uploading bundle files to /Workspace/Users/v.chen@databricks.com/.bundle/demo-include/dev/files...
Created jobs.demo_include
Files: 4 uploaded, 0 deleted
Resources: 1 created, 0 changed, 0 deleted, 0 unchanged
src/train.py
          "code_source_path": "/Workspace/Users/v.chen@databricks.com/.bundle/demo-include/dev/artifacts/.internal/code.tgz",
The following resources will be deleted:
  delete resources.jobs.demo_include

All files and directories at the following location will be deleted: /Workspace/Users/v.chen@databricks.com/.bundle/demo-include/dev

Destroy: 1 deleted

Show that git refs are respected:

# v.chen at https://dbc-04ac0685-8857 in /tmp/demo-include (git:) [20:04:57]
$ rm -rf /tmp/demo-git && mkdir -p /tmp/demo-git/src && cd /tmp/demo-git
git init -q && git config user.email t@t.co && git config user.name t
printf 'print("committed code")\n' > src/train.py
printf '#!/bin/bash\ncd "$(dirname "$0")"\npython src/train.py\n' > command.sh
git add -A && git commit -qm init
BR=$(git rev-parse --abbrev-ref HEAD)
cat > databricks.yml <<YAML
bundle: {name: demo-git}
targets: {dev: {mode: development, default: true, workspace: {host: $HOST}}}
artifacts:
  code_source:
    type: tgz
    git: {branch: $BR}          # or  commit: <sha>
    include: [src]
    files: [{source: ./dist/code.tgz}]
resources:
  jobs:
    demo_git:
      name: demo-git
      tasks:
        - task_key: train
          environment_key: default
          ai_runtime_task:
            experiment: demo-git
            deployments: [{command_path: ./command.sh, compute: {accelerator_type: GPU_1xA10, accelerator_count: 1}}]
            code_source_path: ./dist/code.tgz
      environments: [{environment_key: default, spec: {environment_version: "5", dependencies: []}}]
YAML

$CLI bundle deploy -t dev
tar tzf dist/code.tgz                            # → src/train.py   (from `git archive` of the ref)
$CLI bundle destroy -t dev --auto-approve
Databricks pre-commit Git Hook V2.5.0
Running secret scanning on changes staged for commit.
secret-scan hook completed in 126 ms
Unknown project name: None, skipping linting.
pre-commit-total hook completed in 153 ms
Databricks commit-msg Git Hook V2.5.0
Running secret scanning on commit message.
secret-scan hook completed in 40 ms
commit-msg-total hook completed in 50 ms
Building code_source...
Uploading dist/code.tgz...
Uploading bundle files to /Workspace/Users/v.chen@databricks.com/.bundle/demo-git/dev/files...
Created jobs.demo_git
Files: 4 uploaded, 0 deleted
Resources: 1 created, 0 changed, 0 deleted, 0 unchanged
src/
src/train.py
The following resources will be deleted:
  delete resources.jobs.demo_git

All files and directories at the following location will be deleted: /Workspace/Users/v.chen@databricks.com/.bundle/demo-git/dev

Destroy: 1 deleted

[eng-dev-ecosystem-bot]

Integration test report

Commit: f2a8fd9

Run: 33203627766

	Env	💚​RECOVERED	🙈​SKIP	✅​pass	🙈​skip	Time
💚​	aws linux	1	1	274	1207	3:59
💚​	aws windows	1	1	276	1205	4:06
💚​	azure linux	1	1	273	1207	3:59
💚​	azure windows	1	1	275	1205	3:38
💚​	gcp linux	1	1	274	1207	4:18
💚​	gcp windows	1	1	276	1205	3:44

	Test Name	aws linux	aws windows	azure linux	azure windows	gcp linux	gcp windows
💚​	TestAccept	💚​R	💚​R	💚​R	💚​R	💚​R	💚​R
🙈​	TestAccept/ssh/connection	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S

Top 3 slowest tests (at least 2 minutes):

duration	env	testname
4:01	aws windows	TestAccept
3:39	gcp windows	TestAccept
3:32	azure windows	TestAccept

[ben-hansen-db]

Looks good, let's follow the file generation flow for the autogen files

[ben-hansen-db]

have we tested equivalence of tarballs from here vs CLI? Note also the tarball changes that landed in CLI recently to respect gitignore

[vinchenzo-db]

Made them share code instead of testing equivalence

[ben-hansen-db]

is semantics for include the same as in air cli? There it is relative to code source root, is include relative to bundle root in DABs? Can we make sure that is documented well somewhere?

[vinchenzo-db]

include is now relative to the code-source root, same as the air CLI. The reason I decided this is so that convert-to-dabs can map air CLI's include_paths straight across later.

[ben-hansen-db]

above it says this is autogenerated. Did claude update this file or was it infact autogenerated?

[ben-hansen-db]

there's probably a flow for autogenerating this file to follow

[vinchenzo-db]

^ claude ran the autogeneration script

[ben-hansen-db]

LGTM, nice work!

[pietern]

load-bearing? 😂

[vinchenzo-db]

removed, lol

[pietern]

Why do we need the output path to be specified?

We can interpolate a temporary path, under .databricks for example, such that this file is never included in the archive it produces on the second run.

Artifacts are uploaded separately from regular files. The places that refer to artifacts do path rewriting to make sure the final path we use in the workspace is the right one.

[vinchenzo-db]

For now I think this is still necessary. We can mark it as a follow up if you feel it shouldn't be, but right now the BYOT API specifies code_source_path, which is where the user runs their code. Without this output path I think it becomes a little opaque to the user that code_source_path would still say ./dist/code.tgz, but the artifact now lives somewhere else and nothing links them.

I asked claude what it would take to bridge this gap and we would need to let code_source_path reference the artifact by name, but alledgely its not a very easy fix.

[pietern]

Thanks, yeah I agree with the opaque bit, and that referencing via interpolation is the right solution.

You'd get something like this:

code_source_path: ${artifacts.air_code_source.output}

[pietern]

Thanks, yeah I agree with the opaque bit, and that referencing via interpolation is the right solution.

You'd get something like this:

code_source_path: ${artifacts.air_code_source.output}