Files
cibuildwheel/test/test_0_basic.py
T
Henry Schreiner eec3efa84e ci: improve Azure test reliability (#2890)
* ci: improve Azure test reliability

Azure has intermittent failures, often the macOS framework-Python
`posix_spawn: Undefined error: 0` spawn race under load, plus transient
download flakiness. Address both:

- Retry integration tests via pytest-rerunfailures (already required):
  add `--reruns=2 --reruns-delay=5` to the serial and non-serial runs so a
  single transient flake self-heals instead of failing the whole job.
- Cap Azure at `--num-processes 3` to reduce concurrent process spawning
  (the macOS runner was using 4 xdist workers), lowering the spawn-race
  probability and peak resource pressure.
- Cache downloads across runs: relocate CIBW_CACHE_PATH to a stable path
  and add a Cache@2 task (interpreter downloads + per-worker test pip
  caches live under it), cutting the network-download flake surface.
- Bump the Azure host interpreter to 3.13.

Assisted-by: ClaudeCode:claude-opus-4.8

* ci: stabilize iOS tests, bound rerun cost

An iOS run on Azure (build 9043) hung for the full 40-min pytest timeout
on the second build config of test_ios_platforms, then reran twice
(reruns=2), blowing past the 180-min job cap with no useful signal. The
hang was a stuck simulator/xcodebuild on the second config, after the
first config had run.

- Shut down running simulators before each parametrized config via a
  `clean_ios_simulators` fixture, so a simulator left booted/wedged by the
  previous config isn't reused.
- Drop iOS reruns from 2 to 1. One retry still covers the documented
  "fails the first time" simulator-boot flake, while halving the
  worst-case retry cost (3x40=120min -> 2x40=80min) on a hang.

Assisted-by: ClaudeCode:claude-opus-4.8

* ci: don't limit num processes

Signed-off-by: Henry Schreiner <henryfs@princeton.edu>

* ci: rerun OCI unit tests that pull from Docker Hub

test_local_image et al. occasionally fail on a transient Docker Hub
anonymous-pull blip ("unauthorized: authentication required") when
pulling debian:trixie-slim. The integration runs already self-heal via
pytest-rerunfailures, but the unit run had no reruns, so a single flake
failed the whole job. Mark the three network-dependent OCI tests
(test_local_image, test_enter_error, test_multiarch_image) flaky so they
retry instead of blanket-rerunning the suite.

Assisted-by: ClaudeCode:claude-opus-4.8

* Drop iOS clean step

Co-authored-by: Henry Schreiner <HenrySchreinerIII@gmail.com>

* ci: scope integration reruns to test_0_basic::test

Drop the blanket --reruns from the serial and non-serial integration
runs and instead mark test_0_basic.py::test as flaky, so only the known
flaky basic test is retried.

Assisted-by: ClaudeCode:claude-opus-4.8

---------

Signed-off-by: Henry Schreiner <henryfs@princeton.edu>
2026-06-05 08:09:16 -04:00

121 lines
3.8 KiB
Python

from __future__ import annotations
import textwrap
import packaging.utils
import pytest
from cibuildwheel.logger import Logger
from cibuildwheel.selector import EnableGroup
from . import test_projects, utils
TYPE_CHECKING = False
if TYPE_CHECKING:
from pathlib import Path
basic_project = test_projects.new_c_project(
setup_py_add=textwrap.dedent(
"""
import os
if os.environ.get("CIBUILDWHEEL", "0") != "1":
raise Exception("CIBUILDWHEEL environment variable is not set to 1")
"""
)
)
@pytest.mark.serial
def test_dummy_serial() -> None:
"""A no-op test to ensure that at least one serial test is always found.
Without this no-op test, CI fails on CircleCI because no serial tests are
found, and pytest errors if a test suite finds no tests.
"""
@pytest.mark.flaky(reruns=2, reruns_delay=5)
def test(
tmp_path: Path, build_frontend_env: dict[str, str], capfd: pytest.CaptureFixture[str]
) -> None:
project_dir = tmp_path / "project"
basic_project.generate(project_dir)
# build the wheels
actual_wheels = utils.cibuildwheel_run(project_dir, add_env=build_frontend_env)
# check that the expected wheels are produced
expected_wheels = utils.expected_wheels("spam", "0.1.0")
actual_wheels_normalized = {packaging.utils.parse_wheel_filename(w) for w in actual_wheels}
expected_wheels_normalized = {packaging.utils.parse_wheel_filename(w) for w in expected_wheels}
assert actual_wheels_normalized == expected_wheels_normalized
enable_groups = utils.get_enable_groups()
if EnableGroup.GraalPy not in enable_groups:
# Verify pip warning not shown
captured = capfd.readouterr()
for stream in (captured.err, captured.out):
assert "WARNING: Running pip as the 'root' user can result" not in stream
assert "A new release of pip available" not in stream
@pytest.mark.skip(reason="to keep test output clean")
def test_sample_build(tmp_path: Path, capfd: pytest.CaptureFixture[str]) -> None:
project_dir = tmp_path / "project"
basic_project.generate(project_dir)
# build the wheels, and let the output passthrough to the caller, so
# we can see how it looks
with capfd.disabled():
logger = Logger()
logger.step("test_sample_build")
try:
utils.cibuildwheel_run(project_dir)
finally:
logger.step_end()
@pytest.mark.parametrize("enable_setting", ["", "cpython-prerelease", "pypy"])
def test_build_identifiers(
tmp_path: Path, enable_setting: str, monkeypatch: pytest.MonkeyPatch
) -> None:
project_dir = tmp_path / "project"
basic_project.generate(project_dir)
monkeypatch.setenv("CIBW_ENABLE", enable_setting)
# check that the number of expected wheels matches the number of build
# identifiers
expected_wheels = utils.expected_wheels("spam", "0.1.0")
build_identifiers = utils.cibuildwheel_get_build_identifiers(project_dir)
assert len(expected_wheels) == len(build_identifiers), (
f"{expected_wheels} vs {build_identifiers}"
)
@pytest.mark.parametrize(
("add_args", "env_allow_empty"),
[
(["--allow-empty"], {}),
(["--allow-empty"], {"CIBW_ALLOW_EMPTY": "0"}),
(None, {"CIBW_ALLOW_EMPTY": "1"}),
],
)
def test_allow_empty(
tmp_path: Path, add_args: list[str] | None, env_allow_empty: dict[str, str]
) -> None:
project_dir = tmp_path / "project"
basic_project.generate(project_dir)
# Sanity check - --allow-empty should cause a no-op build to complete
# without error
actual_wheels = utils.cibuildwheel_run(
project_dir,
add_env={"CIBW_SKIP": "*", **env_allow_empty},
add_args=add_args,
)
# check that nothing was built
assert len(actual_wheels) == 0