Skip to content

helm_resource upgrades unchanged releases on every tilt up #695

Description

@azmeuk

helm_resource runs helm upgrade --install on every tilt up. helm has no notion of a no-op: it re-renders the chart, builds both the stored and the target manifests, diffs them, patches every object, records a new revision and runs the hooks. When nothing has changed, all of that is spent for nothing.

Actually, it costs more than the run that installed the release, because an upgrade computes a three-way merge that an install does not.

Numbers

A chart of N namespaced ConfigMaps and nothing else (MRE below), on kind, measured in one pass:

ConfigMaps 1st tilt ci (installs) 2nd tilt ci (nothing changed) of which the helm upgrade
50 1.79 s 2.12 s 0.63 s
200 2.25 s 3.81 s 2.21 s
500 4.12 s 7.86 s 5.68 s

The upgrade grows with the number of objects in the release, and takes the resource over as the chart gets bigger: 30% of the second run at 50 objects, 72% at 500. The rest is Tilt's own startup, which is why this barely shows on a small chart. The release revision also goes from 1 to 2 on the second run, at all three sizes.

Real chart, for scale: the kyverno 3.8.2 chart puts 5.7 MB of CRDs in its release. In our dev environment tilt up on an already-installed kyverno spends about 13 of the 15 seconds that resource takes on the upgrade, every single time.

This bites in Tilt specifically because tilt up re-applies every resource at startup and no build state survives the previous session. deps only covers the session Tilt is running. A dev who restarts Tilt a few times a day pays this on each restart. The extension's own README already has the goal, under Future Work:

In the past, we've found it difficult to efficiently manage this information
from the Tiltfile. Ideally, `tilt up` against an existing environment should be
well-cached and fast. But maybe there are ways to do this with UIButtons that
manually trigger a repo update.

MRE

mkdir -p helm-noop/chart/templates && cd helm-noop
printf 'apiVersion: v2\nname: noop\nversion: 0.0.1\n' > chart/Chart.yaml
python3 - <<'EOF'
open("chart/templates/configmaps.yaml", "w").write("".join(
    "---\napiVersion: v1\nkind: ConfigMap\nmetadata:\n  name: noop-%d\ndata:\n  payload: %s\n"
    % (i, "x" * 2048) for i in range(200)))
EOF
cat > Tiltfile <<'EOF'
load('ext://helm_resource', 'helm_resource')

helm_resource('noop', './chart', namespace='helm-noop', deps=['./chart'],
              flags=['--create-namespace'], pod_readiness='ignore')
EOF

time tilt ci      # installs the release
time tilt ci      # nothing changed
tilt down

For the third column, on the release the second run leaves in place:

time helm upgrade --install --namespace helm-noop noop ./chart

Vary the range(200) above to get the other rows of the table.

pod_readiness='ignore' is only there because the chart has no pods; without it tilt ci waits for pods that never come.

Not fixable in helm

For what it is worth, I looked for a way to have helm answer "nothing to do" cheaply, and there is none:

Would you take a fix here?

The check that would work is on the input side. Digest everything handed to helm: the chart reference, every flag, the contents of every values file, and the chart directory itself when it is a local path. Compare that against what the release records, before spending anything on a render.

That is cheap next to the upgrade it would stand in for. On the three charts above, one helm status --output json plus a kubectl get of every object the release holds came to 0.13 s, 0.37 s and 0.93 s, against the upgrade's 0.63 s, 2.21 s and 5.68 s.

Environment: helm 3.19.0, tilt 0.37.6, kind, Linux.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions