CVE pipeline #2614
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,133 @@ | ||
| - job: | ||
| name: cve-pipeline | ||
| properties: | ||
| - authorization: | ||
| inheritance-strategy: none | ||
| GROUP:ceph*security: | ||
| - job-read | ||
| - job-discover | ||
| - job-build | ||
| - job-cancel | ||
| - job-configure | ||
| - job-workspace | ||
| - run-replay | ||
| - run-update | ||
| - run-delete | ||
| - scm-tag | ||
| description: | | ||
| This Jenkins pipeline is only used to develop CVE fixes. Its differences are: | ||
| - The job itself is not public-facing. It does not inherit the RBAC under https://jenkins.ceph.com/manage/configureSecurity/ | ||
| - ceph-dev-pipeline pushes to the internal/private pulp instance instead of a public chacra instance | ||
| project-type: pipeline | ||
| quiet-period: 1 | ||
| concurrent: true | ||
| pipeline-scm: | ||
| scm: | ||
| - git: | ||
| url: https://github.com/ceph/ceph-build | ||
| branches: | ||
| - ${{CEPH_BUILD_BRANCH}} | ||
| shallow-clone: true | ||
| submodule: | ||
| disable: true | ||
| wipe-workspace: true | ||
| script-path: ceph-dev-pipeline/build/Jenkinsfile | ||
| lightweight-checkout: true | ||
| do-not-fetch-tags: true | ||
|
|
||
| parameters: | ||
| - string: | ||
| name: BRANCH | ||
| description: "The branch from ceph-private.git to build" | ||
| default: main | ||
|
|
||
| - choice: | ||
| name: SHA1 | ||
| description: "Intentionally blank. ceph-dev-pipeline requires this" | ||
| choices: | ||
| - '' | ||
|
|
||
| - choice: | ||
| name: CEPH_REPO | ||
| choices: | ||
| - git@github.com:ceph/ceph-private.git | ||
|
|
||
| - string: | ||
| name: DISTROS | ||
| description: "A list of distros to build for. Available options are: centos9, noble, jammy, bookworm, trixie" | ||
| default: "noble centos9 bookworm trixie" | ||
|
|
||
| - string: | ||
| name: ARCHS | ||
| description: "A list of architectures to build for. Available options are: x86_64, and arm64" | ||
| default: "x86_64 arm64" | ||
|
|
||
| - bool: | ||
| name: CI_COMPILE | ||
| description: "Whether to compile and build packages" | ||
| default: true | ||
|
|
||
| - bool: | ||
|
djgalloway marked this conversation as resolved.
|
||
| name: THROWAWAY | ||
| description: "Whether to push any binaries to Chacra or Pulp. Overrides CHACRA_UPLOAD and PULP_UPLOAD if either are true." | ||
| default: false | ||
|
|
||
| - bool: | ||
| name: CHACRA_UPLOAD | ||
| description: "DO NOT CHECK" | ||
| default: false | ||
|
|
||
| - bool: | ||
| name: PULP_UPLOAD | ||
| description: "If you want packages, this must be true. We do not push to chacra for CVE builds." | ||
| default: true | ||
|
|
||
| - choice: | ||
| name: FLAVORS | ||
| choices: | ||
| - default | ||
|
|
||
| - bool: | ||
|
dmick marked this conversation as resolved.
|
||
| name: CI_CONTAINER | ||
| description: "Whether to build and push container images" | ||
| default: true | ||
|
|
||
| - string: | ||
| name: CONTAINER_REPO_HOSTNAME | ||
| description: "FQDN of container repo server (e.g. 'quay.io')" | ||
| default: "quay-int.front.sepia.ceph.com" | ||
|
|
||
| - string: | ||
| name: CONTAINER_REPO_ORGANIZATION | ||
| description: "Name of container repo organization (e.g. 'ceph-ci')" | ||
| default: "ceph-ci" | ||
|
|
||
| - bool: | ||
| name: DWZ | ||
| description: "Use dwz to make debuginfo packages smaller" | ||
| default: false | ||
|
|
||
| - bool: | ||
| name: SCCACHE | ||
| description: "Use sccache to speed up compilation" | ||
| default: true | ||
|
|
||
| - string: | ||
| name: SETUP_BUILD_ID | ||
| description: "Reuse the source distribution from this cve-source-dist build instead of creating a new one" | ||
| default: "" | ||
|
|
||
| - choice: | ||
| name: SETUP_JOB | ||
| choices: | ||
| - cve-source-dist | ||
|
|
||
| - string: | ||
| name: CEPH_BUILD_BRANCH | ||
| description: "Use the Jenkinsfile from this ceph-build branch" | ||
| default: main | ||
|
|
||
| wrappers: | ||
| - build-name: | ||
| name: "#${{BUILD_NUMBER}} ${{BRANCH}}, ${{VERSION}}" | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,78 @@ | ||
| - job: | ||
| name: cve-source-dist | ||
| description: This job is only called from https://jenkins.ceph.com/job/cve-pipeline. The entire job and its artifacts are only visible to the ceph/security Github Team. | ||
| project-type: pipeline | ||
| concurrent: true | ||
| pipeline-scm: | ||
| scm: | ||
| - git: | ||
| url: https://github.com/ceph/ceph-build | ||
| branches: | ||
| - ${{CEPH_BUILD_BRANCH}} | ||
| shallow-clone: true | ||
| submodule: | ||
| disable: true | ||
| wipe-workspace: true | ||
| script-path: ceph-source-dist/build/Jenkinsfile | ||
| lightweight-checkout: true | ||
| do-not-fetch-tags: true | ||
| properties: | ||
| - build-discarder: | ||
| days-to-keep: -1 | ||
| num-to-keep: 100 | ||
| artifact-days-to-keep: -1 | ||
| artifact-num-to-keep: 50 | ||
| - copyartifact: | ||
| projects: cve-pipeline | ||
| - authorization: | ||
| inheritance-strategy: none | ||
| GROUP:ceph*security: | ||
| - job-read | ||
| - job-discover | ||
| - job-build | ||
| - job-cancel | ||
| - job-configure | ||
| - job-workspace | ||
| - run-replay | ||
| - run-update | ||
| - run-delete | ||
| - scm-tag | ||
|
|
||
| parameters: | ||
| - choice: | ||
| name: CEPH_REPO | ||
| choices: | ||
| - git@github.com:ceph/ceph-private.git | ||
|
|
||
| - string: | ||
| name: BRANCH | ||
| description: "The Ceph branch to build" | ||
|
|
||
| - string: | ||
| name: SHA1 | ||
| description: "The specific commit to build" | ||
|
|
||
| - string: | ||
| name: CEPH_BUILD_BRANCH | ||
| description: "Use the Jenkinsfile from this ceph-build branch" | ||
| default: main | ||
|
|
||
| scm: | ||
| - git: | ||
| url: ${{CEPH_REPO}} | ||
| # Use the SSH key attached to the ceph-jenkins GitHub account. | ||
| credentials-id: "jenkins-build" | ||
| branches: | ||
| - $BRANCH | ||
| timeout: 20 | ||
| skip-tag: true | ||
| wipe-workspace: true | ||
|
|
||
| wrappers: | ||
| - inject-passwords: | ||
| global: true | ||
| mask-password-params: true | ||
| - credentials-binding: | ||
| - text: | ||
| credential-id: shaman-api-key | ||
| variable: SHAMAN_API_KEY |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,65 @@ | ||
| #!/bin/bash | ||
| # vim: ts=4 sw=4 expandtab | ||
|
|
||
| submit_repo_status() { | ||
|
|
||
| # A helper script to post (create) the status of a repo in shaman. | ||
| # 'state' is the repo status (e.g. 'ready'). | ||
| # 'project' is used to post to the right url in shaman. | ||
| # shaman keys repos by 'chacra_url' and builds Arch rows from the 'archs' | ||
| # list, so the arch must be sent as a JSON array (not 'distro_arch'). | ||
| http_method=$1 | ||
| state=$2 | ||
| project=$3 | ||
| distro=$4 | ||
| distro_version=$5 | ||
| arch=$6 | ||
| url=$7 | ||
|
|
||
| # RPM builds also ship source RPMs (the SRPMS/ subdir), so advertise a | ||
| # "source" arch to shaman alongside the binary arch, mirroring chacra. | ||
| # Binary debs have no corresponding source repo. OS_PKG_TYPE is exported | ||
| # by the calling Jenkins step. | ||
| archs="\"$arch\"" | ||
| if [ "$OS_PKG_TYPE" = "rpm" ]; then | ||
| archs="${archs},\"source\"" | ||
| fi | ||
|
|
||
| # package_manager_version is computed by pulp_upload.sh (a separate | ||
| # process) and handed off via this file in the shared workspace. | ||
| PACKAGE_MANAGER_VERSION="" | ||
| if [ -r "$WORKSPACE/pulp_repo_info" ]; then | ||
| source "$WORKSPACE/pulp_repo_info" | ||
| fi | ||
|
Member
There was a problem hiding this comment. is it an error not to have a PACKAGE_MANAGER_VERSION? How does this synchronize?
Contributor
Author
There was a problem hiding this comment. It gets written here: https://github.com/ceph/ceph-build/pull/2614/changes/BASE..045d140e7cc55c72739aef8c6fbf3af18d7ac860#diff-e43616de828404d15446d65b52a9351b6042eefe915702c04074e6b674e46d71R424-R425. pulp_upload.sh runs immediately before notify_shaman_pulp_repo.sh in the same step: ceph-build/ceph-dev-pipeline/build/Jenkinsfile Lines 621 to 622 in 49b2430
Member
There was a problem hiding this comment. OK. So maybe mention in a comment "PACKAGE_MANAGER_VERSION is set in this file I'm about to source" and have an error condition in case it doesn't arrive for whatever reason, just to harden it a bit. Two scripts changing independently are easier to break in future |
||
|
|
||
| cat > $WORKSPACE/repo_status.json << EOF | ||
| { | ||
| "url":"$url", | ||
| "chacra_url":"$url", | ||
| "status":"$state", | ||
| "distro":"$distro", | ||
| "distro_version":"$distro_version", | ||
| "archs":[$archs], | ||
| "ref":"$BRANCH", | ||
| "sha1":"$SHA1", | ||
| "flavor":"$FLAVOR", | ||
| "extra":{ | ||
| "version":"$CEPH_VERSION", | ||
| "package_manager_version":"$PACKAGE_MANAGER_VERSION", | ||
| "build_url":"$BUILD_URL", | ||
| "root_build_cause":"$ROOT_BUILD_CAUSE", | ||
| "node_name":"$NODE_NAME", | ||
| "job_name":"$JOB_NAME" | ||
| } | ||
| } | ||
| EOF | ||
|
|
||
| SHAMAN_URL="https://shaman.ceph.com/api/repos/$project/" | ||
| # post the repo information as JSON to shaman | ||
| curl -X $http_method -H "Content-Type:application/json" --data "@$WORKSPACE/repo_status.json" -u $SHAMAN_API_USER:$SHAMAN_API_KEY ${SHAMAN_URL} | ||
| } | ||
|
|
||
| # If the script is executed (as opposed to sourced), run the function now | ||
| if [ "$(basename -- "${0#-}")" = "$(basename -- "${BASH_SOURCE}")" ]; then | ||
| submit_repo_status "POST" "$@" | ||
| fi | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
can we find a way to just use
spec_project_url?