convert-to-dabs calls tgz artifact, drop aicode mutator entirely

[vinchenzo-db]

Basically, remove the aicode mutator entirely. That way convert-to-dabs just calls the artifacts block. Clean up the code.

Reverts the user-facing local-dir shorthand from #6110 / #6494.

Changes

Why

We should simplify the UJ to limit ways to upload code.

Tests

Setup

# v.chen at ip-10-90-19-147 in ~ (git:) [19:17:23]
$ cd ~/.worktrees/cli-rm-aicode && go build -o /tmp/dbcli-rmaicode .
# v.chen at ip-10-90-19-147 in ~/.worktrees/cli-rm-aicode (git:vchen/air-rm-aicode-mutator) [19:17:24]
$ mkdir -p ~/air-rmaicode-e2e/src && cd ~/air-rmaicode-e2e

cat > run.yaml <<'EOF'
experiment_name: rmaicode-e2e
command: cd "$CODE_SOURCE_PATH" && python train.py
compute:
  accelerator_type: GPU_1xA10
  num_accelerators: 1
environment:
  version: 5
  dependencies:
    - numpy
code_source:
  type: snapshot
  snapshot:
    root_path: ./src
EOF

cat > src/train.py <<'EOF'
import os, mlflow
print("hello from delivered code_source", flush=True)
print("CODE_SOURCE_PATH=" + os.environ.get("CODE_SOURCE_PATH", "<unset>"), flush=True)
mlflow.log_metric("ok", 1)
print("done", flush=True)
EOF

Run convert-to-dabs:

$ /tmp/dbcli-rmaicode experimental air convert-to-dabs run.yaml --force                                                                                     
cat databricks.yml                                      
Wrote a Databricks Asset Bundle to .:
  databricks.yml
  generated_artifacts/training_config.yaml
  generated_artifacts/command.sh

To deploy and run this workload as a bundle:
  1. /tmp/dbcli-rmaicode bundle validate
  2. /tmp/dbcli-rmaicode bundle deploy
  3. /tmp/dbcli-rmaicode bundle run rmaicode-e2e --no-wait

bundle deploy uploads the code source and launch scripts automatically.
To see what it deployed and where: /tmp/dbcli-rmaicode bundle summary

Unlike `air run` (which submits an ephemeral run), bundle deploy creates a
persistent job that is not garbage-collected. When you are done, remove the
job and its uploaded files with:
  /tmp/dbcli-rmaicode bundle destroy
bundle:
  name: rmaicode-e2e
sync:
  paths:
    - generated_artifacts
artifacts:
  code_source:
    type: tgz
    path: .
    include:
      - src
    files:
      - source: ./dist/code_source.tgz
targets:
  dev:
    mode: development
    default: true
resources:
  jobs:
    rmaicode-e2e:
      name: rmaicode-e2e
      tasks:
        - task_key: rmaicode-e2e
          environment_key: default
          max_retries: 3
          ai_runtime_task:
            experiment: rmaicode-e2e
            deployments:
              - command_path: ./generated_artifacts/command.sh
                compute:
                  accelerator_type: GPU_1xA10
                  accelerator_count: 1
            code_source_path: ./dist/code_source.tgz
      environments:
        - environment_key: default
          spec:
            environment_version: "5"
            dependencies:
              - numpy

Validate and deploy:

$ /tmp/dbcli-rmaicode bundle validate -t dev -p e2-dogfood                                                                      
/tmp/dbcli-rmaicode bundle deploy   -t dev -p e2-dogfood
Name: rmaicode-e2e
Target: dev
Workspace:
  User: v.chen@databricks.com
  Path: /Workspace/Users/v.chen@databricks.com/.bundle/rmaicode-e2e/dev

Validation OK!
Building code_source...
Uploading dist/code_source.tgz...
Uploading bundle files to /Workspace/Users/v.chen@databricks.com/.bundle/rmaicode-e2e/dev/files...
Files: 2 uploaded, 0 deleted
Resources: 0 created, 0 changed, 0 deleted, 1 unchanged

Run:

# v.chen at ip-10-90-19-147 in ~/air-rmaicode-e2e (git:) [19:20:10]
$ JOBID=$(/tmp/dbcli-rmaicode bundle summary -t dev -p e2-dogfood -o json | grep -oE '"id":[^,]*' | head -1 | grep -oE '[0-9]+')
/tmp/dbcli-rmaicode jobs get $JOBID -p e2-dogfood | grep -oE '"code_source_path":[^,]*'
"code_source_path": "/Workspace/Users/v.chen@databricks.com/.bundle/rmaicode-e2e/dev/artifacts/.internal/code_source.tgz"
# v.chen at ip-10-90-19-147 in ~/air-rmaicode-e2e (git:) [19:20:16]
$ /tmp/dbcli-rmaicode bundle run rmaicode-e2e -t dev -p e2-dogfood --no-wait                                                    
Run URL: https://e2-dogfood.staging.cloud.databricks.com/jobs/1103803560415808/runs/174957625358063?o=6051921418418893

[github-actions]

Approval status: pending

/acceptance/bundle/ - needs approval

17 files changed
Suggested: @denik
Also eligible: @shreyas-goenka, @andrewnester, @janniklasrose, @anton-107, @lennartkats-db, @pietern

/acceptance/experimental/air/ - approved by @ben-hansen-db

Files: acceptance/experimental/air/convert-to-dabs/output.txt, acceptance/experimental/air/convert-to-dabs/script

/bundle/ - needs approval

5 files changed
Suggested: @denik
Also eligible: @shreyas-goenka, @andrewnester, @janniklasrose, @anton-107, @lennartkats-db, @pietern

/experimental/air/ - approved by @ben-hansen-db

Files: experimental/air/cmd/convert_to_dabs.go, experimental/air/cmd/convert_to_dabs_test.go

_{Any maintainer (@andrewnester, @anton-107, @denik, @pietern, @shreyas-goenka, @simonfaltum, @renaudhartert-db, @janniklasrose, @lennartkats-db, @rugpanov, @rclarey) can approve all areas.
See OWNERS for ownership rules.}

[eng-dev-ecosystem-bot]

Integration test report

Commit: b045b7e

Run: 33909563361

	Env	💚​RECOVERED	✅​pass	🙈​skip	Time
💚​	aws linux	1	275	16	7:36
💚​	aws windows	1	277	14	4:12
💚​	azure linux	1	274	16	6:38
💚​	azure windows	1	276	14	4:49
💚​	gcp linux	1	275	16	8:51
💚​	gcp windows	1	277	14	5:01

	Test Name	aws linux	aws windows	azure linux	azure windows	gcp linux	gcp windows
💚​	TestAccept	💚​R	💚​R	💚​R	💚​R	💚​R	💚​R

Top 6 slowest tests (at least 2 minutes):

duration	env	testname
4:47	azure windows	TestAccept
4:17	gcp windows	TestAccept
4:09	aws windows	TestAccept
3:59	azure linux	TestAccept
3:58	aws linux	TestAccept
3:50	gcp linux	TestAccept

[ben-hansen-db]

LGTM as a git revert