test: fix smoke test failures on xen/vmware by weizhouapache · Pull Request #13628 · apache/cloudstack

weizhouapache · 2026-07-16T11:34:21Z

Description

This PR fixes some smoke test failures on vmware or xenserver/xcpng

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)
Build/CI
Test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

weizhouapache · 2026-07-16T11:34:36Z

@blueorangutan package

blueorangutan · 2026-07-16T11:36:05Z

@weizhouapache a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.

Copilot

Pull request overview

This PR targets integration smoke test stability across hypervisors (notably XenServer/XCP-ng and VMware) by skipping KVM-specific checks/tests when not running on KVM, and by adjusting a few timing/host prerequisite assumptions that can cause intermittent failures.

Changes:

Skip KVM-only paths in smoke tests when the detected hypervisor is not KVM.
Add additional host-tool prerequisite checks for a VPC source NAT IP update test.
Add delays around affinity-group migration steps and make the events-resource test more tolerant around volume attach/detach operations.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.

File	Description
test/integration/smoke/test_nonstrict_affinity_group.py	Adds waits before migration steps in non-strict affinity group tests.
test/integration/smoke/test_network_extension_namespace.py	Skips KVM prerequisite checks on non-KVM hypervisors; expands required host tools for one test.
test/integration/smoke/test_kms_lifecycle.py	Skips a VM root-disk encryption test on non-KVM hypervisors.
test/integration/smoke/test_events_resource.py	Wraps volume attach/detach operations to avoid hard failures in some environments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

codecov · 2026-07-16T11:43:10Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 19.90%. Comparing base (5812290) to head (f947c85).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff            @@
##               main   #13628   +/-   ##
=========================================
  Coverage     19.90%   19.90%           
- Complexity    20171    20177    +6     
=========================================
  Files          6372     6372           
  Lines        577180   577180           
  Branches      70693    70693           
=========================================
+ Hits         114869   114910   +41     
+ Misses       449755   449702   -53     
- Partials      12556    12568   +12

Flag	Coverage Δ
uitests	`3.71% <ø> (ø)`
unittests	`21.17% <100.00%> (+<0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

blueorangutan · 2026-07-16T12:24:11Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 18593

weizhouapache · 2026-07-16T12:35:32Z

@blueorangutan test ubuntu26 xcpng82 keepEnv

blueorangutan · 2026-07-16T12:36:03Z

@weizhouapache a [SL] Trillian-Jenkins test job (ubuntu26 mgmt + xcpng82) has been kicked to run smoke tests

weizhouapache · 2026-07-16T12:37:19Z

@blueorangutan test ubuntu26 vmware-80u3e keepEnv

blueorangutan · 2026-07-16T12:38:05Z

@weizhouapache a [SL] Trillian-Jenkins test job (ubuntu26 mgmt + vmware-80u3e) has been kicked to run smoke tests

Copilot

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.

weizhouapache · 2026-07-17T11:34:58Z

@blueorangutan package

blueorangutan · 2026-07-17T11:36:04Z

@weizhouapache a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2026-07-17T12:24:46Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 18602

weizhouapache · 2026-07-17T12:45:07Z

@blueorangutan test ubuntu26 xcpng82

blueorangutan · 2026-07-17T12:46:03Z

@weizhouapache a [SL] Trillian-Jenkins test job (ubuntu26 mgmt + xcpng82) has been kicked to run smoke tests

weizhouapache · 2026-07-17T12:46:57Z

@blueorangutan test ubuntu26 vmware-80u3e

blueorangutan · 2026-07-17T12:48:04Z

@weizhouapache a [SL] Trillian-Jenkins test job (ubuntu26 mgmt + vmware-80u3e) has been kicked to run smoke tests

blueorangutan · 2026-07-17T13:46:10Z

[SF] Trillian Build Failed (tid-7)

weizhouapache · 2026-07-17T15:36:37Z

@blueorangutan test ubuntu26 vmware-80u3

blueorangutan · 2026-07-17T15:38:03Z

@weizhouapache a [SL] Trillian-Jenkins test job (ubuntu26 mgmt + vmware-80u3) has been kicked to run smoke tests

blueorangutan · 2026-07-17T23:16:50Z

[SF] Trillian Build Failed (tid-9)

blueorangutan · 2026-09-24T15:27:16Z

[SF] Trillian Build Failed (tid-48)

blueorangutan · 2026-09-24T18:16:15Z

[SF] Trillian Build Failed (tid-50)

blueorangutan · 2026-09-25T11:01:02Z

[SF] Trillian test result (tid-46)
Environment: xcpng83 (x2), zone: Advanced Networking with Mgmt server u26
Total time taken: 76081 seconds
Marvin logs: [archive removed]
Smoke tests completed. 155 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_1_userdata_on_ssvm	`Error`	8.62	test_systemvm_userdata.py

blueorangutan · 2026-09-25T17:32:59Z

[SF] Trillian test result (tid-49)
Environment: vmware-80u3 (x2), zone: Advanced Networking with Mgmt server u26
Total time taken: 91199 seconds
Marvin logs: [archive removed]
Smoke tests completed. 153 look OK, 3 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_08_upgrade_kubernetes_ha_cluster	`Failure`	3698.26	test_kubernetes_clusters.py
test_13_test_add_external_nodes_to_cluster	`Failure`	1009.38	test_kubernetes_clusters.py
ContextSuite context=TestKubernetesCluster>:teardown	`Error`	1097.86	test_kubernetes_clusters.py
test_05_list_volumes_isrecursive	`Failure`	0.06	test_list_volumes.py
test_07_list_volumes_listall	`Failure`	0.05	test_list_volumes.py
test_list_vms_metrics_admin	`Error`	3627.05	test_metrics_api.py
test_list_vms_metrics_history	`Error`	8.74	test_metrics_api.py
test_list_volumes_metrics_history	`Error`	3625.81	test_metrics_api.py

weizhouapache · 2026-09-30T13:31:02Z

@blueorangutan test ubuntu26 vmware-70u3

blueorangutan · 2026-09-30T13:34:04Z

@weizhouapache a [SL] Trillian-Jenkins test job (ubuntu26 mgmt + vmware-70u3) has been kicked to run smoke tests

blueorangutan · 2026-09-30T13:49:41Z

[SF] Trillian Build Failed (tid-55)

weizhouapache · 2026-09-30T13:55:52Z

@blueorangutan package

blueorangutan · 2026-09-30T13:58:05Z

@weizhouapache a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2026-09-30T14:51:47Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 19357

weizhouapache · 2026-09-30T16:19:05Z

@blueorangutan test ubuntu26 vmware-70u3

blueorangutan · 2026-09-30T16:30:04Z

@weizhouapache a [SL] Trillian-Jenkins test job (ubuntu26 mgmt + vmware-70u3) has been kicked to run smoke tests

blueorangutan · 2026-10-01T12:44:43Z

[SF] Trillian test result (tid-56)
Environment: vmware-70u3 (x2), zone: Advanced Networking with Mgmt server u26
Total time taken: 69822 seconds
Marvin logs: [archive removed]
Smoke tests completed. 153 look OK, 3 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_13_test_add_external_nodes_to_cluster	`Error`	831.30	test_kubernetes_clusters.py
ContextSuite context=TestKubernetesCluster>:teardown	`Error`	920.38	test_kubernetes_clusters.py
test_05_list_volumes_isrecursive	`Failure`	0.06	test_list_volumes.py
test_07_list_volumes_listall	`Failure`	0.06	test_list_volumes.py
test_08_migrate_vm	`Error`	29.32	test_vm_life_cycle.py

blueorangutan · 2026-10-03T21:10:31Z

[SF] Trillian test result (tid-57)
Environment: vmware-70u3 (x2), zone: Advanced Networking with Mgmt server u26
Total time taken: 101910 seconds
Marvin logs: [archive removed]
Smoke tests completed. 152 look OK, 3 have errors, 1 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_08_upgrade_kubernetes_ha_cluster	`Failure`	3689.65	test_kubernetes_clusters.py
test_13_test_add_external_nodes_to_cluster	`Error`	834.28	test_kubernetes_clusters.py
ContextSuite context=TestKubernetesCluster>:teardown	`Error`	910.87	test_kubernetes_clusters.py
test_01_deploy_vm_on_specific_host	`Error`	3603.93	test_vm_deployment_planner.py
test_02_deploy_vm_on_specific_cluster	`Error`	2.48	test_vm_deployment_planner.py
test_03_deploy_vm_on_specific_pod	`Error`	2.56	test_vm_deployment_planner.py
test_04_deploy_vm_on_host_override_pod_and_cluster	`Error`	1.43	test_vm_deployment_planner.py
test_05_deploy_vm_on_cluster_override_pod	`Error`	2.49	test_vm_deployment_planner.py
test_09_expunge_vm	`Failure`	423.76	test_vm_life_cycle.py
all_test_vm_strict_host_tags	`Skipped`	---	test_vm_strict_host_tags.py

weizhouapache · 2026-10-22T09:06:38Z

@blueorangutan package

blueorangutan · 2026-10-22T09:08:05Z

@weizhouapache a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2026-10-22T10:44:30Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 19673

weizhouapache · 2026-10-22T10:51:53Z

@blueorangutan test ol8 vmware-70u3

blueorangutan · 2026-10-22T10:54:03Z

@weizhouapache a [SL] Trillian-Jenkins test job (ol8 mgmt + vmware-70u3) has been kicked to run smoke tests

weizhouapache · 2026-10-22T11:40:17Z

@blueorangutan test ol8 xcpng82

blueorangutan · 2026-10-22T11:42:05Z

@weizhouapache a [SL] Trillian-Jenkins test job (ol8 mgmt + xcpng82) has been kicked to run smoke tests

blueorangutan · 2026-10-23T11:37:21Z

[SF] Trillian test result (tid-17210)
Environment: vmware-70u3 (x2), zone: Advanced Networking with Mgmt server ol8
Total time taken: 85281 seconds
Marvin logs: [archive removed]
Smoke tests completed. 155 look OK, 2 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_08_upgrade_kubernetes_ha_cluster	`Failure`	3706.60	test_kubernetes_clusters.py
test_05_list_volumes_isrecursive	`Failure`	0.06	test_list_volumes.py
test_07_list_volumes_listall	`Failure`	0.05	test_list_volumes.py

weizhouapache · 2026-10-23T11:56:34Z

@nvazquez @RosiKyu
there are some intermittent failures on vmware and xcpng. most of the test failures are gone.
do you think we can merge this PR to 24.0 ?

nvazquez · 2026-10-23T13:02:07Z

Thanks @weizhouapache, looks good to me

weizhouapache · 2026-10-23T14:47:50Z

thanks @nvazquez @DaanHoogland
I am merging . If there are new failures, we could address in another PR

Copilot AI lite review requested due to automatic review settings July 16, 2026 11:34

boring-cyborg Bot added component:integration-test Python Warning... Python code Ahead! labels Jul 16, 2026

weizhouapache added this to the 4.23.0 milestone Jul 16, 2026

Copilot started reviewing on behalf of weizhouapache July 16, 2026 11:34

View session

weizhouapache removed this from the 4.23.0 milestone Jul 16, 2026

weizhouapache added this to the 4.23.0 milestone Jul 16, 2026

Copilot AI reviewed Jul 16, 2026

View reviewed changes

Comment thread test/integration/smoke/test_nonstrict_affinity_group.py

Comment thread test/integration/smoke/test_nonstrict_affinity_group.py

Comment thread test/integration/smoke/test_events_resource.py Outdated

Copilot AI review requested due to automatic review settings July 17, 2026 09:41

Copilot started reviewing on behalf of weizhouapache July 17, 2026 09:42

View session

Copilot AI reviewed Jul 17, 2026

View reviewed changes

Comment thread test/integration/smoke/test_network_extension_namespace.py

Comment thread test/integration/smoke/test_kms_lifecycle.py

Comment thread test/integration/smoke/test_events_resource.py

weizhouapache mentioned this pull request Oct 2, 2026

[WIP] Ceph improvements shapeblue/cloudstack#147

Draft

3 of 14 tasks

nvazquez approved these changes Oct 23, 2026

View reviewed changes

weizhouapache merged commit 1b4f19e into apache:main Oct 23, 2026
26 of 28 checks passed

github-project-automation Bot moved this from In review to Done in CloudStack Testing Oct 23, 2026

weizhouapache deleted the 4.23-fix-smoke-test-failures branch October 23, 2026 14:48

Conversation

weizhouapache commented Jul 16, 2026

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

Uh oh!

weizhouapache commented Jul 16, 2026

Uh oh!

blueorangutan commented Jul 16, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

codecov Bot commented Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

blueorangutan commented Jul 16, 2026

Uh oh!

weizhouapache commented Jul 16, 2026

Uh oh!

blueorangutan commented Jul 16, 2026

Uh oh!

weizhouapache commented Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

blueorangutan commented Jul 16, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

weizhouapache commented Jul 17, 2026

Uh oh!

blueorangutan commented Jul 17, 2026

Uh oh!

blueorangutan commented Jul 17, 2026

Uh oh!

weizhouapache commented Jul 17, 2026

Uh oh!

blueorangutan commented Jul 17, 2026

Uh oh!

weizhouapache commented Jul 17, 2026

Uh oh!

blueorangutan commented Jul 17, 2026

Uh oh!

blueorangutan commented Jul 17, 2026

Uh oh!

weizhouapache commented Jul 17, 2026

Uh oh!

blueorangutan commented Jul 17, 2026

Uh oh!

blueorangutan commented Jul 17, 2026

Uh oh!

blueorangutan commented Sep 24, 2026

Uh oh!

blueorangutan commented Sep 24, 2026

Uh oh!

blueorangutan commented Sep 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

blueorangutan commented Sep 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

weizhouapache commented Sep 30, 2026

Uh oh!

blueorangutan commented Sep 30, 2026

codecov Bot commented Jul 16, 2026 •

edited

Loading

weizhouapache commented Jul 16, 2026 •

edited

Loading

blueorangutan commented Sep 25, 2026 •

edited

Loading

blueorangutan commented Sep 25, 2026 •

edited

Loading

blueorangutan commented Oct 1, 2026 •

edited

Loading

blueorangutan commented Oct 3, 2026 •

edited

Loading

blueorangutan commented Oct 23, 2026 •

edited

Loading