Skip to content

Commit

Permalink
Merge pull request #3700 from eslutsky/WorkloadPartitioning_tests
Browse files Browse the repository at this point in the history 
USHIFT-3788: Workload partitioning RF tests
  • Loading branch information
@openshift-merge-bot
openshift-merge-bot[bot] committed Aug 8, 2024
2 parents 672d1f2 + 573eac9 commit 3d590fd
Show file tree
Hide file tree
Showing 4 changed files with 275 additions and 49 deletions.
10 changes: 9 additions & 1 deletion test/resources/common.resource
View file
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,7 @@ Upload String To File # robocop: disable=too-many-calls-in-keyword
Put File ${local_tmp} ${remote_tmp} mode=0644
Remove File ${local_tmp}
${stdout} ${stderr} ${rc}= Execute Command
... mv ${remote_tmp} ${remote_filename}
... mv -Z ${remote_tmp} ${remote_filename}
... sudo=True return_rc=True return_stdout=True return_stderr=True
Should Be Equal As Integers 0 ${rc}
${stdout} ${stderr} ${rc}= Execute Command
Expand Down Expand Up @@ -120,3 +120,11 @@ Create Random Temp Directory
${path}= Join Path ${root_path} ${rand}
Create Directory ${path}
RETURN ${path}

Create Remote Dir For Path
[Documentation] Create parent dirname if doesnt exist
[Arguments] ${file_path}
${stdout} ${stderr} ${rc}= Execute Command
... mkdir -pZ $(dirname ${file_path})
... sudo=True return_rc=True return_stdout=True return_stderr=True
Should Be Equal As Integers 0 ${rc}
3 changes: 2 additions & 1 deletion test/scenarios/presubmits/el94-src@low-latency.sh
View file
Original file line number Diff line number Diff line change
Expand Up @@ -37,5 +37,6 @@ scenario_run_tests() {
run_tests host1 \
--exitonfailure \
suites/tuned/profile.robot \
suites/tuned/microshift-tuned.robot
suites/tuned/microshift-tuned.robot \
suites/tuned/workload-partitioning.robot
}
47 changes: 0 additions & 47 deletions test/suites/standard2/workload-partitioning.robot
View file

This file was deleted.

264 changes: 264 additions & 0 deletions test/suites/tuned/workload-partitioning.robot
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,264 @@
*** Settings ***
Documentation Tests for Workload partitioning
Resource ../../resources/microshift-config.resource
Resource ../../resources/common.resource
Resource ../../resources/systemd.resource
Resource ../../resources/microshift-process.resource
Resource ../../resources/microshift-network.resource
Resource ../../resources/ostree-health.resource

Suite Setup Setup Suite And Wait For Greenboot
Suite Teardown Teardown Suite


*** Variables ***
${MANAGEMENT_CPU} 0
${SYSTEMD_CRIO_DROPIN} /etc/systemd/system/crio.service.d/microshift-cpuaffinity.conf
${SYSTEMD_MICROSHIFT_DROPIN} /etc/systemd/system/microshift.service.d/microshift-cpuaffinity.conf
${SYSTEMD_OVS_DROPIN} /etc/systemd/system/ovs-vswitchd.service.d/microshift-cpuaffinity.conf
${SYSTEMD_OVSDB_DROPIN} /etc/systemd/system/ovsdb-server.service.d/microshift-cpuaffinity.conf
${CRIO_CONFIG_DROPIN} /etc/crio/crio.conf.d/20-microshift-wp.conf
${KUBELET_CPU_STATE_FILE} /var/lib/kubelet/cpu_manager_state
${KUBELET_WORKLOAD_PINNING_CONFIG} /etc/kubernetes/openshift-workload-pinning


*** Test Cases ***
Control Plane Pods Must Be Annotated
[Documentation] Verify that all the Control Plane pods are properly annotated.
All Pods Should Be Annotated As Management

Workload Partitioning Should Work
[Documentation] Verify that all the Control Plane pods are properly annotated.
[Setup] Setup For Workload Partitioning
Microshift Services Should Be Running On Reserved CPU ${MANAGEMENT_CPU}
All Pods Should Run On Reserved CPU ${MANAGEMENT_CPU}
Create Hello MicroShift Pod
All Pods Should Run On Reserved CPU ${MANAGEMENT_CPU} true
[Teardown] Teardown For Workload Partitioning


*** Keywords ***
Setup For Workload Partitioning
[Documentation] Setup for Workload Partitioning
Save Default MicroShift Config
Configure Kubelet For Workload Partitioning ${MANAGEMENT_CPU}
Configure CRIO For Workload Partitioning ${MANAGEMENT_CPU}
Configure CPUAffinity In Systemd ${MANAGEMENT_CPU} ${SYSTEMD_CRIO_DROPIN}
Configure CPUAffinity In Systemd ${MANAGEMENT_CPU} ${SYSTEMD_MICROSHIFT_DROPIN}
Configure CPUAffinity In Systemd ${MANAGEMENT_CPU} ${SYSTEMD_OVS_DROPIN}
Configure CPUAffinity In Systemd ${MANAGEMENT_CPU} ${SYSTEMD_OVSDB_DROPIN}
Systemctl Daemon Reload
Systemctl restart crio.service
Cleanup And Create NS

Teardown For Workload Partitioning
[Documentation] Setup for Workload Partitioning
Run Keyword And Ignore Error
... SSHLibrary.Get File ${DEBUG_OUTPUT_FILE} ${OUTPUTDIR}/pod-crio-inspect-output.json
Cleanup MicroShift --all --keep-images
Remove Files ${KUBELET_CPU_STATE_FILE}
... ${SYSTEMD_CRIO_DROPIN}
... ${SYSTEMD_MICROSHIFT_DROPIN}
... ${CRIO_CONFIG_DROPIN}
Systemctl Daemon Reload
Restore Default MicroShift Config
Systemctl restart crio.service
Restart MicroShift

Configure Kubelet For Workload Partitioning
[Documentation] configure microshift with kubelet CPU configuration
[Arguments] ${cpus}

${kubelet_config}= CATENATE SEPARATOR=\n
... ---
... kubelet:
... \ \ reservedSystemCPUs: "${cpus}"
... \ \ cpuManagerPolicy: static
... \ \ cpuManagerPolicyOptions:
... \ \ \ \ full-pcpus-only: "true"
... \ \ cpuManagerReconcilePeriod: 5s

${merged}= Extend MicroShift Config ${kubelet_config}
Upload MicroShift Config ${merged}

${kubelet_pinning_config}= CATENATE SEPARATOR=\n
... {
... \ \ "management": {
... \ \ \ \ "cpuset": "${cpus}"
... \ \ }
... }

Upload String To File ${kubelet_pinning_config} ${KUBELET_WORKLOAD_PINNING_CONFIG}

Configure CRIO For Workload Partitioning
[Documentation] add crio Dropin configuration
[Arguments] ${cpus}
${crio_configuration}= CATENATE SEPARATOR=\n
... [crio.runtime]
... infra_ctr_cpuset = "${cpus}"
... [crio.runtime.workloads.management]
... activation_annotation = "target.workload.openshift.io/management"
... annotation_prefix = "resources.workload.openshift.io"
... resources = { "cpushares" = 0, "cpuset" = "${cpus}" }

Upload String To File ${crio_configuration} ${CRIO_CONFIG_DROPIN}

Configure CPUAffinity In Systemd
[Documentation] add CPUAffinity in systemd unit dropin
[Arguments] ${cpus} ${target_file}
Create Remote Dir For Path ${target_file}
${systemd_cpu_affinity}= CATENATE SEPARATOR=\n
... [Service]
... CPUAffinity=${cpus}
Upload String To File ${systemd_cpu_affinity} ${target_file}

All Pods Should Run On Reserved CPU
[Documentation] Verify all the PODs runs on explicit reserved CPU
[Arguments] ${configured_cpus} ${is_workloads}=${EMPTY}
${crio_json}= Get Json From Crio Output ${is_workloads}

${json_status}= Json Parse ${crio_json}
FOR ${pod} IN @{json_status}
${pod_id} ${pod_pid} ${pod_cpuset} ${pod_info}= Construct Pod Info ${pod}

IF "${is_workloads}"=="${EMPTY}"
IF ${pod_cpuset} != ${configured_cpus}
Crio Save Pod Manifest ${pod_id}
Fail
... Management ${pod_info} running on CPU ${pod_cpuset} instead of ${configured_cpus}
END
Proccess Should Be Running On Host CPU ${pod_pid} ${configured_cpus}
ELSE IF "${pod_cpuset}" == "${configured_cpus}"
Crio Save Pod Manifest ${pod_id}
Fail
... Workload ${pod_info} running on CPU ${pod_cpuset} instead of ${configured_cpus}
END
END

Construct Pod Info
[Documentation] json
[Arguments] ${pod_json}
${pod_cpuset}= Evaluate "${pod_json}[info][runtimeSpec][linux][resources][cpu][cpus]"
${pod_name}= Evaluate "${pod_json}[status][metadata][name]"
${container_name}= Evaluate "${pod_json}[info][runtimeSpec][annotations][io.kubernetes.pod.name]"
${pod_pid}= Evaluate "${pod_json}[info][pid]"
${pod_id}= Evaluate "${pod_json}[status][id]"
${namespace}= Evaluate "${pod_json}[info][runtimeSpec][annotations][io.kubernetes.pod.namespace]"
${pod_info}= Catenate SEPARATOR=
... container: ${container_name}
... ${EMPTY} pod: ${pod_name}
... ${EMPTY} pid: ${pod_pid}
... ${EMPTY} namespace: ${namespace}
RETURN ${pod_id} ${pod_pid} ${pod_cpuset} ${pod_info}

Get Json From Crio Output
[Documentation] get json from the crio command
[Arguments] ${is_workloads}
Set Global Variable ${NOT_WORKLOADS} ${EMPTY}
IF "${is_workloads}"!="${EMPTY}"
Set Global Variable ${NOT_WORKLOADS} | not
END

${stdout} ${stderr} ${rc}= Execute Command
... crictl ps -q | xargs sudo crictl inspect | jq -rs '[.[] | select(.info.runtimeSpec.annotations["target.workload.openshift.io/management"] ${NOT_WORKLOADS})]'
... sudo=True
... return_stdout=True
... return_stderr=True
... return_rc=True
Should Be Equal As Integers ${rc} 0
RETURN ${stdout}

Crio Save Pod Manifest
[Documentation] Saves running pod manifest using crio inspect
[Arguments] ${pod_id}
${path}= Create Random Temp File
Set Global Variable ${DEBUG_OUTPUT_FILE} ${path}
${stdout} ${stderr} ${rc}= Execute Command
... crictl ps -q | xargs sudo crictl inspect | jq -rs '[.[] | select(.status.id=="${pod_id}")]' >${DEBUG_OUTPUT_FILE} 2>&1
... sudo=True
... return_stdout=True
... return_stderr=True
... return_rc=True

Microshift Services Should Be Running On Reserved CPU
[Documentation] Verify all the Microshift Services runs on explicit reserved CPU
[Arguments] ${cpus}
${pid}= MicroShift Process ID
Proccess Should Be Running On Host CPU ${pid} ${cpus}

${pid}= Crio Process ID
Proccess Should Be Running On Host CPU ${pid} ${cpus}

${pid}= MicroShift Etcd Process ID
Proccess Should Be Running On Host CPU ${pid} ${cpus}

Proccess Should Be Running On Host CPU
[Documentation] Verify all the PODs runs on explicit reserved CPU
[Arguments] ${pid} ${cpus}
${stdout} ${stderr} ${rc}= Execute Command
... taskset -cp ${pid} | awk '{print $6}'
... sudo=True return_stdout=True return_stderr=True return_rc=True
Should Be Equal As Integers ${rc} 0
Should Be Equal As Strings ${stdout} ${cpus}

All Pods Should Be Annotated As Management
[Documentation] Obtains list of Deployments created by CSV.
${pods_raw}= Oc Get All Pods
@{pods}= Split String ${pods_raw}
FOR ${pod} IN @{pods}
${ns} ${pod}= Split String ${pod} \@
Pod Must Be Annotated ${ns} ${pod}
END

Pod Must Be Annotated
[Documentation] Check management annotation for specified pod and namespace.
[Arguments] ${ns} ${pod}
${management_annotation}= Oc Get JsonPath
... pod
... ${ns}
... ${pod}
... .metadata.annotations.target\\.workload\\.openshift\\.io/management
Should Not Be Empty ${management_annotation}

Oc Get All Pods
[Documentation] Returns the running pods across all namespaces,
... Returns the command output as formatted string <name-space>@<pod-name>
${data}= Oc Get JsonPath
... pods
... ${EMPTY}
... ${EMPTY}
... range .items[*]}{\.metadata\.namespace}{"@"}{\.metadata\.name}{"\\n"}{end
RETURN ${data}

Crio Process ID
[Documentation] Return the current crio process ID
${stdout} ${stderr} ${rc}= Execute Command
... pidof crio
... return_stdout=True return_stderr=True return_rc=True
Log ${stderr}
RETURN ${stdout}

Remove Files
[Documentation] removes files from the microshit host
[Arguments] @{files}
Log ${files}
${files_path}= Catenate SEPARATOR=${SPACE} @{files}
${stdout} ${stderr} ${rc}= Execute Command
... rm -f ${files_path}
... sudo=True return_stdout=True return_stderr=True return_rc=True
Should Be Equal As Integers ${rc} 0

Cleanup And Create NS
[Documentation] cleanup microshift and recreate the namespace for workloads
Cleanup MicroShift --all --keep-images
Remove Files ${KUBELET_CPU_STATE_FILE}
Restart MicroShift
${ns}= Create Unique Namespace
Set Suite Variable \${NAMESPACE} ${ns}

Setup Suite And Wait For Greenboot
[Documentation] Run setup suit and wait for greenboot to become ready
Setup Suite
Restart Greenboot And Wait For Success

0 comments on commit 3d590fd

Please sign in to comment.