[release-4.21] OCPBUGS-67016: Isolate and reduce parallelism for OrderedNamespaceDeletion tests.

[benluddy]

These tests directly exercise the namespace controller (in kube-controller-manager), which can fall behind due to a combination of master node CPU saturation and the ephemeral namespace churn generated during parallel E2E runs. We observe flakes (timeouts) when this occurs. The tests are sensitive to CPU saturation, but not causing CPU saturation, so the goal of this change is to improve the CI signal until such time as the incidence of CPU saturation events in CI is reduced.

[openshift-ci-robot]

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: automatic mode

[openshift-ci]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[openshift-ci-robot]

@benluddy: This pull request references Jira Issue OCPBUGS-67016, which is invalid:

• expected the bug to target the "4.22.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

These tests directly exercise the namespace controller (in kube-controller-manager), which can fall behind due to a combination of master node CPU saturation and the ephemeral namespace churn generated during parallel E2E runs. We observe flakes (timeouts) when this occurs. The tests are sensitive to CPU saturation, but not causing CPU saturation, so the goal of this change is to improve the CI signal until such time as the incidence of CPU saturation events in CI is reduced.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[benluddy]

/cc @bertinatto

[benluddy]

/test ?

[openshift-ci]

@benluddy: The following commands are available to trigger required jobs:

/test e2e-aws-csi

/test e2e-aws-jenkins

/test e2e-aws-ovn-fips

/test e2e-aws-ovn-image-registry

/test e2e-aws-ovn-microshift

/test e2e-aws-ovn-microshift-serial

/test e2e-aws-ovn-serial-1of2

/test e2e-aws-ovn-serial-2of2

/test e2e-gcp-csi

/test e2e-gcp-ovn

/test e2e-gcp-ovn-builds

/test e2e-gcp-ovn-image-ecosystem

/test e2e-gcp-ovn-upgrade

/test e2e-metal-ipi-ovn-ipv6

/test e2e-vsphere-ovn

/test e2e-vsphere-ovn-upi

/test go-verify-deps

/test images

/test lint

/test okd-scos-images

/test unit

/test verify

/test verify-deps

The following commands are available to trigger optional jobs:

/test e2e-agnostic-ovn-cmd

/test e2e-aws-disruptive

/test e2e-aws-etcd-certrotation

/test e2e-aws-etcd-recovery

/test e2e-aws-ovn

/test e2e-aws-ovn-cgroupsv2

/test e2e-aws-ovn-edge-zones

/test e2e-aws-ovn-etcd-scaling

/test e2e-aws-ovn-kube-apiserver-rollout

/test e2e-aws-ovn-kubevirt

/test e2e-aws-ovn-serial-fast

/test e2e-aws-ovn-serial-ipsec

/test e2e-aws-ovn-serial-publicnet-1of2

/test e2e-aws-ovn-serial-publicnet-2of2

/test e2e-aws-ovn-single-node

/test e2e-aws-ovn-single-node-serial

/test e2e-aws-ovn-single-node-techpreview

/test e2e-aws-ovn-single-node-techpreview-serial

/test e2e-aws-ovn-single-node-upgrade

/test e2e-aws-ovn-upgrade

/test e2e-aws-ovn-upgrade-rollback

/test e2e-aws-ovn-upi

/test e2e-aws-proxy

/test e2e-azure

/test e2e-azure-ovn-etcd-scaling

/test e2e-azure-ovn-upgrade

/test e2e-baremetalds-kubevirt

/test e2e-external-aws

/test e2e-external-aws-ccm

/test e2e-external-vsphere-ccm

/test e2e-gcp-disruptive

/test e2e-gcp-fips-serial-1of2

/test e2e-gcp-fips-serial-2of2

/test e2e-gcp-ovn-etcd-scaling

/test e2e-gcp-ovn-kube-apiserver-rollout

/test e2e-gcp-ovn-rt-upgrade

/test e2e-gcp-ovn-techpreview

/test e2e-gcp-ovn-techpreview-serial-1of2

/test e2e-gcp-ovn-techpreview-serial-2of2

/test e2e-gcp-ovn-usernamespace

/test e2e-hypershift-conformance

/test e2e-metal-ipi-ovn

/test e2e-metal-ipi-ovn-bgp-virt-dualstack

/test e2e-metal-ipi-ovn-bgp-virt-dualstack-techpreview

/test e2e-metal-ipi-ovn-dualstack

/test e2e-metal-ipi-ovn-dualstack-bgp

/test e2e-metal-ipi-ovn-dualstack-bgp-local-gw

/test e2e-metal-ipi-ovn-dualstack-local-gateway

/test e2e-metal-ipi-ovn-kube-apiserver-rollout

/test e2e-metal-ipi-serial-1of2

/test e2e-metal-ipi-serial-2of2

/test e2e-metal-ipi-serial-ovn-ipv6-1of2

/test e2e-metal-ipi-serial-ovn-ipv6-2of2

/test e2e-metal-ipi-virtualmedia

/test e2e-metal-ovn-single-node-live-iso

/test e2e-metal-ovn-single-node-with-worker-live-iso

/test e2e-metal-ovn-two-node-arbiter

/test e2e-metal-ovn-two-node-fencing

/test e2e-openstack-ovn

/test e2e-openstack-serial

/test e2e-vsphere-ovn-dualstack-primaryv6

/test e2e-vsphere-ovn-etcd-scaling

/test okd-scos-e2e-aws-ovn

Use /test all to run the following jobs that were automatically triggered:

pull-ci-openshift-origin-main-go-verify-deps

pull-ci-openshift-origin-main-images

pull-ci-openshift-origin-main-lint

pull-ci-openshift-origin-main-okd-scos-images

pull-ci-openshift-origin-main-unit

pull-ci-openshift-origin-main-verify

pull-ci-openshift-origin-main-verify-deps

Details

In response to this:

/test ?

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[openshift-ci-robot]

@benluddy: This pull request references Jira Issue OCPBUGS-67016, which is invalid:

• expected the bug to target the "4.21.0" version, but no target version was set
• release note text must be set and not match the template OR release note type must be set to "Release Note Not Required". For more information you can reference the OpenShift Bug Process.
• expected Jira Issue OCPBUGS-67016 to depend on a bug targeting a version in 4.22.0 and in one of the following states: MODIFIED, ON_QA, VERIFIED, but no dependents were found

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

Details

In response to this:

These tests directly exercise the namespace controller (in kube-controller-manager), which can fall behind due to a combination of master node CPU saturation and the ephemeral namespace churn generated during parallel E2E runs. We observe flakes (timeouts) when this occurs. The tests are sensitive to CPU saturation, but not causing CPU saturation, so the goal of this change is to improve the CI signal until such time as the incidence of CPU saturation events in CI is reduced.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[benluddy]

/hold

Opening this directly against 4.21 instead of main for testing... will need a 4.22 PR if we use it.

[benluddy]

/payload-aggregate periodic-ci-openshift-release-master-ci-4.21-e2e-gcp-ovn-techpreview 10

[openshift-ci]

@benluddy: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

• periodic-ci-openshift-release-master-ci-4.21-e2e-gcp-ovn-techpreview

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/1fa0a4f0-ec17-11f0-8cad-bac4f4997d65-0

[benluddy]

/jira refresh

[openshift-ci-robot]

@benluddy: This pull request references Jira Issue OCPBUGS-67016, which is invalid:

• expected the bug to target the "4.21.0" version, but no target version was set
• release note text must be set and not match the template OR release note type must be set to "Release Note Not Required". For more information you can reference the OpenShift Bug Process.
• expected dependent Jira Issue OCPBUGS-72547 to be in one of the following states: MODIFIED, ON_QA, VERIFIED, but it is POST instead

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[benluddy]

/hold

[benluddy]

/verified by me

#30661 (comment)

10/10 with this patch vs. 50-60% pass rate on the test details report.

[openshift-ci-robot]

@benluddy: This PR has been marked as verified by me.

Details

In response to this:

/verified by me

#30661 (comment)

10/10 with this patch vs. 50-60% pass rate on the test details report.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[dgoodwin]

/lgtm
/label backport-risk-assessed
/label acknowledge-critical-fixes-only

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: benluddy, dgoodwin

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [dgoodwin]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[benluddy]

/payload-aggregate periodic-ci-openshift-release-master-ci-4.21-e2e-gcp-ovn-techpreview 10

[openshift-ci]

@benluddy: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

• periodic-ci-openshift-release-master-ci-4.21-e2e-gcp-ovn-techpreview

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/1f7d9060-f0a8-11f0-8655-d9f8718d04bc-0

[benluddy]

/verified by @benluddy

It's definitely being isolated as expected and there are no failures in the aggregated runs: #30661 (comment). Since our current baseline is around 90% pass, the aggregated results could still be due to chance, but I don't think it's worth waiting for more pre-merge test runs.

/jira refresh

[openshift-ci-robot]

@benluddy: This pull request references Jira Issue OCPBUGS-67016, which is valid. The bug has been moved to the POST state.

7 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target version (4.21.0) matches configured target version for branch (4.21.0)
• bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, POST)
• release note type set to "Release Note Not Required"
• dependent bug Jira Issue OCPBUGS-72547 is in the state MODIFIED, which is one of the valid states (MODIFIED, ON_QA, VERIFIED)
• dependent Jira Issue OCPBUGS-72547 targets the "4.22.0" version, which is one of the valid target versions: 4.22.0
• bug has dependents

Details

In response to this:

/verified by @benluddy

It's definitely being isolated as expected and there are no failures in the aggregated runs: #30661 (comment). Since our current baseline is around 90% pass, the aggregated results could still be due to chance, but I don't think it's worth waiting for more pre-merge test runs.

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[benluddy]

/hold cancel

[benluddy]

/verified by me

[openshift-ci-robot]

@benluddy: This PR has been marked as verified by me.

Details

In response to this:

/verified by me

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci]

@benluddy: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[openshift-ci-robot]

@benluddy: Jira Issue Verification Checks: Jira Issue OCPBUGS-67016
✔️ This pull request was pre-merge verified.
✔️ All associated pull requests have merged.
✔️ All associated, merged pull requests were pre-merge verified.

Jira Issue OCPBUGS-67016 has been moved to the MODIFIED state and will move to the VERIFIED state when the change is available in an accepted nightly payload. 🕓

Details

In response to this:

These tests directly exercise the namespace controller (in kube-controller-manager), which can fall behind due to a combination of master node CPU saturation and the ephemeral namespace churn generated during parallel E2E runs. We observe flakes (timeouts) when this occurs. The tests are sensitive to CPU saturation, but not causing CPU saturation, so the goal of this change is to improve the CI signal until such time as the incidence of CPU saturation events in CI is reduced.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.