-
Notifications
You must be signed in to change notification settings - Fork 247
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
This adds a test suite for e2e tests. So far there is only a few tests related to inspection, but it should be enough to validate the approach. My hope is that this framework will be flexible enough to allow tests both in CI, using VMs, and on actual hardware. The tests can be configured through a config file. They can run on an existing cluster or set up a temporary kind cluster automatically. Similarly, BMO, Ironic and Cert-manager can all be deployed or not depending on configuration. Logs are collected for BMO and Ironic when deployed automatically, even in existing clusters.
- Loading branch information
Showing
25 changed files
with
2,057 additions
and
3 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,10 @@ | ||
| HTTP_PORT=6180 | ||
| PROVISIONING_IP=192.168.222.199 | ||
| DEPLOY_KERNEL_URL=http://192.168.222.199:6180/images/ironic-python-agent.kernel | ||
| DEPLOY_RAMDISK_URL=http://192.168.222.199:6180/images/ironic-python-agent.initramfs | ||
| IRONIC_ENDPOINT=http://192.168.222.199:6385/v1/ | ||
| IRONIC_INSPECTOR_ENDPOINT=http://192.168.222.199:5050/v1/ | ||
| CACHEURL=http://192.168.222.199/images | ||
| IRONIC_FAST_TRACK=true | ||
| IRONIC_KERNEL_PARAMS=console=ttyS0 | ||
| IRONIC_INSPECTOR_VLAN_INTERFACES=all |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,26 @@ | ||
| apiVersion: kustomize.config.k8s.io/v1beta1 | ||
| kind: Kustomization | ||
|
|
||
| resources: | ||
| - ../../namespace | ||
| - ../../default | ||
|
|
||
| configMapGenerator: | ||
| - behavior: replace | ||
| envs: | ||
| - ironic.env | ||
| name: ironic | ||
|
|
||
| patches: | ||
| - patch: | | ||
| # Don't try to pull again the pre-loaded image | ||
| - op: replace | ||
| path: /spec/template/spec/containers/0/imagePullPolicy | ||
| value: IfNotPresent | ||
| target: | ||
| kind: Deployment | ||
| name: controller-manager | ||
|
|
||
| images: | ||
| - name: quay.io/metal3-io/baremetal-operator | ||
| newTag: e2e |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,24 @@ | ||
| apiVersion: kustomize.config.k8s.io/v1beta1 | ||
| kind: Kustomization | ||
|
|
||
| resources: | ||
| - ../../namespace | ||
| - ../../default | ||
|
|
||
| patches: | ||
| - patch: | | ||
| # Enable test mode (fixture provider instead of ironic) | ||
| - op: add | ||
| path: /spec/template/spec/containers/0/args/- | ||
| value: --test-mode | ||
| # Don't try to pull again the pre-loaded image | ||
| - op: replace | ||
| path: /spec/template/spec/containers/0/imagePullPolicy | ||
| value: IfNotPresent | ||
| target: | ||
| kind: Deployment | ||
| name: controller-manager | ||
|
|
||
| images: | ||
| - name: quay.io/metal3-io/baremetal-operator | ||
| newTag: e2e |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,70 @@ | ||
| #!/usr/bin/env bash | ||
|
|
||
| set -x | ||
| set -o nounset | ||
| set -o pipefail | ||
|
|
||
| REPO_ROOT=$(dirname "${BASH_SOURCE[0]}")/.. | ||
| cd "${REPO_ROOT}" || exit 1 | ||
|
|
||
| # Ensure requirements are installed | ||
| "${REPO_ROOT}/hack/e2e/ensure_go.sh" | ||
| export PATH=$PATH:/usr/local/go/bin | ||
| "${REPO_ROOT}/hack/e2e/ensure_minikube.sh" | ||
| "${REPO_ROOT}/hack/e2e/ensure_kubectl.sh" | ||
| "${REPO_ROOT}/hack/e2e/ensure_cmctl.sh" | ||
|
|
||
| # Build the container image with e2e tag (used in tests) | ||
| IMG=quay.io/metal3-io/baremetal-operator:e2e make docker | ||
|
|
||
| # Set up minikube | ||
| minikube start --driver=kvm2 | ||
|
|
||
| virsh -c qemu:///system net-destroy baremetal-e2e | ||
| virsh -c qemu:///system net-define ${REPO_ROOT}/hack/e2e/net.xml | ||
| virsh -c qemu:///system net-start baremetal-e2e | ||
| # Attach baremetal-e2e interface to minikube with specific mac. | ||
| # This will give minikube a known reserved IP address that we can use for Ironic | ||
| virsh -c qemu:///system attach-interface --domain minikube --mac="52:54:00:6c:3c:01" \ | ||
| --model virtio --source baremetal-e2e --type network --config | ||
|
|
||
| # Restart minikube to apply the changes | ||
| minikube stop | ||
| minikube start | ||
|
|
||
| # Load the BMO e2e image into it | ||
| minikube image load quay.io/metal3-io/baremetal-operator:e2e | ||
|
|
||
| # Start VBMC | ||
| docker run --name vbmc --network host -d \ | ||
| -v /var/run/libvirt/libvirt-sock:/var/run/libvirt/libvirt-sock \ | ||
| -v /var/run/libvirt/libvirt-sock-ro:/var/run/libvirt/libvirt-sock-ro \ | ||
| quay.io/metal3-io/vbmc | ||
|
|
||
| # Create libvirt domain | ||
| NAME="bmo-e2e-0" | ||
| BOOT_MAC_ADDRESS="00:60:2f:31:81:01" | ||
| VBMC_PORT="16230" | ||
| virt-install --connect qemu:///system -n "${NAME}" --description "Virtualized BareMetalHost" --osinfo=ubuntu-lts-latest \ | ||
| --ram=4096 --vcpus=2 --disk size=20 --graphics=none --console pty --serial pty --pxe \ | ||
| --network network=baremetal-e2e,mac="${BOOT_MAC_ADDRESS}" --noautoconsole | ||
|
|
||
| # Add BMH VM to VBMC | ||
| docker exec vbmc vbmc add "${NAME}" --port "${VBMC_PORT}" | ||
| docker exec vbmc vbmc start "${NAME}" | ||
| docker exec vbmc vbmc list | ||
|
|
||
| # These variables are used by the tests. They override variables in the config file. | ||
| # This IP is defined by the network we created above. | ||
| # Together with the VBMC_PORT this becomes the BMC_ADDRESS used by the BMH in the test. | ||
| export BMC_ADDRESS="ipmi://192.168.222.1:${VBMC_PORT}" | ||
| export BOOT_MAC_ADDRESS | ||
| # These are the VBMC defaults (used since we did not specify anything else for `vbmc add`). | ||
| export BMC_USER=admin | ||
| export BMC_PASSWORD=password | ||
|
|
||
| # Run the e2e tests | ||
| make test-e2e | ||
|
|
||
| # Collect all artifacts | ||
| tar --directory test/e2e/ -czf artifacts.tar.gz _artifacts |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,13 @@ | ||
| #!/usr/bin/env bash | ||
|
|
||
| REPO_ROOT=$(dirname "${BASH_SOURCE[0]}")/.. | ||
| cd "${REPO_ROOT}" || exit 1 | ||
|
|
||
| minikube delete | ||
| docker stop vbmc | ||
| docker rm vbmc | ||
| virsh destroy --domain bmo-e2e-0 | ||
| virsh undefine --domain bmo-e2e-0 --remove-all-storage | ||
| virsh net-undefine baremetal-e2e | ||
|
|
||
| rm -rf "${REPO_ROOT}/test/e2e/_artifacts" |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,42 @@ | ||
| #!/usr/bin/env bash | ||
|
|
||
| set -o errexit | ||
| set -o nounset | ||
| set -o pipefail | ||
|
|
||
| # GOPATH_BIN="$(go env GOPATH)/bin/" | ||
| GOPATH_BIN="/usr/local/bin/" | ||
| OS=$(go env GOOS) | ||
| ARCH=$(go env GOARCH) | ||
| MINIMUM_CMCTL_VERSION=v1.13.0 | ||
|
|
||
| verify_cmctl_version() { | ||
| if ! [ -x "$(command -v cmctl)" ]; then | ||
| if [[ "${OSTYPE}" == "linux-gnu" ]]; then | ||
| if ! [ -d "${GOPATH_BIN}" ]; then | ||
| mkdir -p "${GOPATH_BIN}" | ||
| fi | ||
| echo 'cmctl not found, installing' | ||
| curl -fsSL -o cmctl.tar.gz https://github.com/cert-manager/cert-manager/releases/download/${MINIMUM_CMCTL_VERSION}/cmctl-${OS}-${ARCH}.tar.gz | ||
| tar xzf cmctl.tar.gz | ||
| sudo mv cmctl "${GOPATH_BIN}/cmctl" | ||
| rm cmctl.tar.gz | ||
| else | ||
| echo "Missing required binary in path: cmctl" | ||
| return 2 | ||
| fi | ||
| fi | ||
|
|
||
| local cmctl_version | ||
| IFS=" " read -ra cmctl_version <<< "$(cmctl version --client --short)" | ||
| if [[ "${MINIMUM_CMCTL_VERSION}" != $(echo -e "${MINIMUM_CMCTL_VERSION}\n${cmctl_version[2]}" | sort -s -t. -k 1,1 -k 2,2n -k 3,3n | head -n1) ]]; then | ||
| cat << EOF | ||
| Detected cmctl version: ${cmctl_version[2]}. | ||
| Requires ${MINIMUM_CMCTL_VERSION} or greater. | ||
| Please install ${MINIMUM_CMCTL_VERSION} or later. | ||
| EOF | ||
| return 2 | ||
| fi | ||
| } | ||
|
|
||
| verify_cmctl_version |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,38 @@ | ||
| #!/usr/bin/env bash | ||
|
|
||
| set -o errexit | ||
| set -o nounset | ||
| set -o pipefail | ||
|
|
||
| MINIMUM_GO_VERSION=go1.20.8 | ||
|
|
||
| # Ensure the go tool exists and is a viable version, or installs it | ||
| verify_go_version() | ||
| { | ||
| # If go is not available on the path, get it | ||
| if ! [ -x "$(command -v go)" ]; then | ||
| if [[ "${OSTYPE}" == "linux-gnu" ]]; then | ||
| echo 'go not found, installing' | ||
| curl -sLo "/tmp/${MINIMUM_GO_VERSION}.linux-amd64.tar.gz" "https://go.dev/dl/${MINIMUM_GO_VERSION}.linux-amd64.tar.gz" | ||
| sudo tar -C /usr/local -xzf "/tmp/${MINIMUM_GO_VERSION}.linux-amd64.tar.gz" | ||
| echo 'export PATH=$PATH:/usr/local/go/bin' >> ~/.bashrc | ||
| export PATH=$PATH:/usr/local/go/bin | ||
| else | ||
| echo "Missing required binary in path: go" | ||
| return 2 | ||
| fi | ||
| fi | ||
|
|
||
| local go_version | ||
| IFS=" " read -ra go_version <<< "$(go version)" | ||
| if ! [[ "${MINIMUM_GO_VERSION}" =~ .*"${go_version[2]}".* ]]; then | ||
| cat << EOF | ||
| Detected go version: ${go_version[2]}. | ||
| Requires ${MINIMUM_GO_VERSION} or greater. | ||
| Please install ${MINIMUM_GO_VERSION} or later. | ||
| EOF | ||
| return 2 | ||
| fi | ||
| } | ||
|
|
||
| verify_go_version |
Oops, something went wrong.