👷 skip telemetry error checks when no datacenter has credentials - #4997
Conversation
The deploy-prod-dc telemetry gate runs for 30 minutes even when every datacenter lacks a configured API/application key, with each iteration just logging 'skipping...' and the loop spinning uselessly. Detect this upfront via hasTelemetryCredentials() and skip the pre-deploy check and the gate entirely when no datacenter can be queried.
Bundles Sizes Evolution
|
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6b051e3811
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if (shouldCheckTelemetryErrors && !(await hasTelemetryCredentials(datacenters))) { | ||
| printLog('No telemetry credentials found for any datacenter, skipping telemetry error checks.') | ||
| shouldCheckTelemetryErrors = false |
There was a problem hiding this comment.
issue: When AWS SSM is temporarily unavailable or the job loses permission, both telemetry credential getters catch every command failure and return undefined, so this condition disables the pre-deploy check and the entire post-deploy gate while the production workflow proceeds with deployment. Before this change, the gate retried credential reads on every iteration and could recover from a transient failure; distinguish a genuinely missing parameter from lookup errors, or keep reevaluating credentials during the gate.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
@BenoitZugmeyer should we only skip when crendentials are confirmed to be missing ?
There was a problem hiding this comment.
I added a commit for that. The command will now crash if something goes wrong instead of proceed forward.
|
✅ All CI checks and tests passed. 🎉 All green!🧪 All tests passed 🎯 Code Coverage (details) 🔗 Commit SHA: 9c94412 | Docs | View more details | Give us feedback! |
6b051e3 to
ba5c967
Compare
When the logs API returns no buckets, the computed count is undefined, producing logs like 'found undefined event(s)'. Default to 0 instead.
ba5c967 to
9c94412
Compare
2 participants
Motivation
Production datacenter deployments run a 30-minute telemetry error gate after deploying. When none of the targeted datacenters have a configured telemetry API/application key, each iteration of the gate silently logs "skipping..." and the loop spins for the full 30 minutes doing nothing, needlessly delaying the pipeline.
Related: the telemetry error log line reported
found undefined event(s)when the logs API returned no buckets, making the output misleading.See job: https://gitlab.ddbuild.io/DataDog/browser-sdk/-/jobs/1993061816 . It's waiting 30 minutes when no datacenter has a key, so it doesn't run any query.
Changes
0when the logs API returns no buckets, so log lines readfound 0 event(s)instead offound undefined event(s).Test instructions
yarn test:scriptand verify thedeploy-prod-dcandcheck-telemetry-errorssuites pass.Checklist