ci: improve Azure test reliability (#2890)

* ci: improve Azure test reliability

Azure has intermittent failures, often the macOS framework-Python
`posix_spawn: Undefined error: 0` spawn race under load, plus transient
download flakiness. Address both:

- Retry integration tests via pytest-rerunfailures (already required):
  add `--reruns=2 --reruns-delay=5` to the serial and non-serial runs so a
  single transient flake self-heals instead of failing the whole job.
- Cap Azure at `--num-processes 3` to reduce concurrent process spawning
  (the macOS runner was using 4 xdist workers), lowering the spawn-race
  probability and peak resource pressure.
- Cache downloads across runs: relocate CIBW_CACHE_PATH to a stable path
  and add a Cache@2 task (interpreter downloads + per-worker test pip
  caches live under it), cutting the network-download flake surface.
- Bump the Azure host interpreter to 3.13.

Assisted-by: ClaudeCode:claude-opus-4.8

* ci: stabilize iOS tests, bound rerun cost

An iOS run on Azure (build 9043) hung for the full 40-min pytest timeout
on the second build config of test_ios_platforms, then reran twice
(reruns=2), blowing past the 180-min job cap with no useful signal. The
hang was a stuck simulator/xcodebuild on the second config, after the
first config had run.

- Shut down running simulators before each parametrized config via a
  `clean_ios_simulators` fixture, so a simulator left booted/wedged by the
  previous config isn't reused.
- Drop iOS reruns from 2 to 1. One retry still covers the documented
  "fails the first time" simulator-boot flake, while halving the
  worst-case retry cost (3x40=120min -> 2x40=80min) on a hang.

Assisted-by: ClaudeCode:claude-opus-4.8

* ci: don't limit num processes

Signed-off-by: Henry Schreiner <henryfs@princeton.edu>

* ci: rerun OCI unit tests that pull from Docker Hub

test_local_image et al. occasionally fail on a transient Docker Hub
anonymous-pull blip ("unauthorized: authentication required") when
pulling debian:trixie-slim. The integration runs already self-heal via
pytest-rerunfailures, but the unit run had no reruns, so a single flake
failed the whole job. Mark the three network-dependent OCI tests
(test_local_image, test_enter_error, test_multiarch_image) flaky so they
retry instead of blanket-rerunning the suite.

Assisted-by: ClaudeCode:claude-opus-4.8

* Drop iOS clean step

Co-authored-by: Henry Schreiner <HenrySchreinerIII@gmail.com>

* ci: scope integration reruns to test_0_basic::test

Drop the blanket --reruns from the serial and non-serial integration
runs and instead mark test_0_basic.py::test as flaky, so only the known
flaky basic test is retried.

Assisted-by: ClaudeCode:claude-opus-4.8

---------

Signed-off-by: Henry Schreiner <henryfs@princeton.edu>
This commit is contained in:
Henry Schreiner
2026-06-05 08:09:16 -04:00
committed by GitHub
parent 29888149e1
commit eec3efa84e
4 changed files with 33 additions and 14 deletions
+3
View File
@@ -546,6 +546,7 @@ def test_disable_host_mount(
container.call(["cat", host_mount_path], capture_output=True)
@pytest.mark.flaky(reruns=2, reruns_delay=5)
@pytest.mark.parametrize("platform", list(OCIPlatform))
def test_local_image(
container_engine: OCIContainerEngineConfig, platform: OCIPlatform, tmp_path: Path
@@ -587,6 +588,7 @@ def test_local_image(
assert container._get_platform_args() == expected_platform_args
@pytest.mark.flaky(reruns=2, reruns_delay=5)
def test_enter_error(container_engine: OCIContainerEngineConfig, tmp_path: Path) -> None:
remote_image = "debian:trixie-slim"
platform = DEFAULT_OCI_PLATFORM
@@ -745,6 +747,7 @@ def test_exit_kills_process_on_shutdown_timeout(monkeypatch: pytest.MonkeyPatch)
assert bash_stdout.closed
@pytest.mark.flaky(reruns=2, reruns_delay=5)
@pytest.mark.parametrize("platform", list(OCIPlatform))
def test_multiarch_image(container_engine: OCIContainerEngineConfig, platform: OCIPlatform) -> None:
if detect_ci_provider() == CIProvider.travis_ci and DEFAULT_OCI_PLATFORM not in {