Travis is set to run one every push to a branch or PR. The results of the builds can be found here for branches and here for PRs.
In Travis CI, 4 jobs are run to test the sdk:
For the Go, Ansible, and Helm tests, the before_install and install stages are the same:
- Check if non documentation files have been updated.
- If only documentation has been updated, skip these tests.
- Run
make tidyto ensurego.modandgo.sumare up-to-date. - Build and install the sdk using
make install. - Install ansible using
sudo pip install ansible. - Run the
hack/ci/setup-openshiftscript, which spins up an openshift cluster by configuring docker and then downloading theocv3.11 binary and runningoc cluster up.
The Go, Ansible, and Helm tests then differ in what tests they run.
- Run some basic sanity checks.
- Run
go vet. - Check that all source files have a license.
- Check that all error messages start with a lower case alphabetical character and do not end with punctuation, and log messages start with an upper case alphabetical character.
- Make sure the repo is in a clean state (this is particularly useful for making sure
go.modandgo.sumare up-to-date after runningmake tidy).
- Run
- Run unit tests.
- Run
make test.
- Run
- Run subcommand tests.
- Run
test localwith no flags enabled. - Run
test localwith most configuration flags enabled. - Run
test localin single namespace mode. - Run
test localwith--up-localflag. - Run
test localwith both--up-localand--kubeconfigflags. - Create all test resources with kubectl and run
test localwith--no-setupflag. - Run
scorecardsubcommand and check that expected score matches actual score. - Run
scorecardsubcommand with json output enabled and verify the output.
- Run
- Run go e2e tests.
- Scaffold a project using
hack/tests/scaffolding/e2e-go-scaffold.sh - Build
memcached-operatorimage to be used in tests - Run scaffolded project e2e tests using
operator-sdk up local- Run cluster test (namespace is auto-generated and deleted by test framework).
- Deploy operator and required resources to the cluster.
- Run the leader election test.
- Verify that operator deployment is ready.
- Verify that leader configmap specifies 1 leader and that the memcached operator has 2 pods (configuration for this is done in step 4.1).
- Delete current leader and wait for memcached-operator deployment to become ready again.
- Verify that leader configmap specifies 1 leader and that the memcached-operator has 2 pods.
- Verify that the name of the new leader is different from the name of the old leader.
- Run the memcached scale test.
- Create memcached CR specifying a desired cluster size of 3 and wait until memcached cluster is of size 3.
- Increase desired cluster size to 4 and wait until memcached cluster is of size 4.
- Run the memcached metrics test.
- Make sure the metrics Service was created.
- Get metrics via proxy pod and make sure they are present.
- Perform linting of the existing metrics.
- Run the memcached custom resource metrics test.
- Make sure the metrics Service was created.
- Get metrics via proxy pod and make sure they are present.
- Perform linting of the existing metrics.
- Perform checks on each custom resource generated metric and makes sure the name, type, value, labels and metric are correct.
- Run local test (namespace is auto-generated and deleted by test framework).
- Start operator using
up localsubcommand. - Run memcached scale test (described in step 4.3.1.3)
- Start operator using
- Run cluster test (namespace is auto-generated and deleted by test framework).
- Run TLS library tests.
- This test runs multiple simple tests of the operator-sdk's TLS library. The tests run in parallel and each tests runs in its own namespace.
- Scaffold a project using
- Run ansible molecule tests. (`make test/e2e/ansible-molecule)
- Create and configure a new ansible type memcached-operator.
- Create cluster resources.
- Run
operator-sdk test localto run ansible molecule tests - Change directory to
test/ansible-inventoryand runoperator-sdk test local
NOTE: All created resources, including the namespace, are deleted using a bash trap when the test finishes
NOTE If you are using a MacOSX SO then will be required replace the sed command in the hack/tests/e2e-ansible-molecule.sh as the following example.
# Use the following sed command to check it on macOsX.
# More info: https://www.mkyong.com/mac/sed-command-hits-undefined-label-error-on-mac-os-x/
sed -i "" 's|\(FROM quay.io/operator-framework/ansible-operator\)\(:.*\)\?|\1:dev|g' build/Dockerfile
# The following code is the default used (Not valid for MacOSX)
# sed -i 's|\(FROM quay.io/operator-framework/ansible-operator\)\(:.*\)\?|\1:dev|g' build/Dockerfile- Run helm e2e tests.
- Create base helm operator project by running
hack/image/helm/scaffold-helm-image.go. - Build base helm operator image.
- Create and configure a new helm type nginx-operator.
- Create cluster resources.
- Wait for operator to be ready.
- Create nginx CR and wait for it to be ready.
- Scale up the dependent deployment and verify the operator reconciles it back down.
- Scale up the CR and verify the dependent deployment scales up accordingly.
- Delete nginx CR and verify that finalizer (which writes a message in the operator logs) ran.
- Run
operator-sdk migrateto add go source to the operator (see this note on dependency management first). - Run
operator-sdk buildto compile the new binary and build a new image. - Re-run steps 4-9 to test the migrated operator.
- Create base helm operator project by running
NOTE: All created resources, including the namespace, are deleted using a bash trap when the test finishes
The markdown test does not create a new cluster and runs in a barebones Travis VM configured only for bash. This allows documentation PRs to pass quickly, as they don't require code tests. The markdown checker uses a precompiled version of marker stored in hack/ci/marker to check the validity and correctness of the links in all markdown files in the doc directory.
NOTE: There is currently a bug in marker that causes link with underscores (_) to not be checked correctly.