Files
cibuildwheel/cibuildwheel/extra.py
T
Henry SchreinerCopilotcopilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>henryiii
cd38ee1548 feat: add SHA256 verification for direct downloads (#2873)
* feat: add SHA256 verification for direct downloads

Store SHA256 hashes when running update scripts and verify them
when downloading files at build time. This improves security by
detecting unexpected changes to downloaded artifacts.

Platforms covered: macOS (CPython, PyPy, GraalPy), iOS, Android,
virtualenv, and python-build-standalone. Windows (nuget) and
Linux (Docker) are excluded.

SHA256 sources per platform:
- macOS/iOS/Android CPython (python.org): sha256_sum from API
- GraalPy: .sha256 sidecar assets from GitHub releases
- python-build-standalone: SHA256SUMS file in release
- PyPy, BeeWare iOS, Maven (Chaquopy): stream-download and compute

Changes:
- cibuildwheel/util/file.py: add sha256 param to download()
- cibuildwheel/platforms/{macos,ios,android}.py: add sha256 to
  PythonConfiguration and pass to download()
- cibuildwheel/venv.py: read sha256 from toml and pass to download()
- cibuildwheel/util/python_build_standalone.py: add sha256 to
  PythonBuildStandaloneAsset and pass to download()
- cibuildwheel/resources/build-platforms.toml: add sha256 fields
- cibuildwheel/resources/virtualenv.toml: add sha256 field
- cibuildwheel/resources/python-build-standalone-releases.json: add sha256
- bin/update_pythons.py: compute/store sha256 per source strategy
- bin/update_virtualenv.py: compute sha256 by streaming download
- bin/update_python_build_standalone.py: parse SHA256SUMS file

Closes #908

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Assisted-by: copilot-cli:claude-sonnet-4.6

* fix: populate sha256 in resource files and fix Windows PythonConfiguration

- Add sha256 field to Windows PythonConfiguration (PyPy/GraalPy have
  direct download URLs on Windows too)
- Pass sha256 to install_pypy() and install_graalpy() in windows.py
- Fix update_pythons.py: handle empty sha256 from CPython API (older
  versions) by streaming download to compute it; fix condition to
  check 'not sha256' rather than 'not in dict'
- Fix update_virtualenv.py: compute sha256 even when version unchanged
  but sha256 is empty (first-time population)
- Fix update_python_build_standalone.py: resolve file path relative to
  the script itself (not the installed package) so writes go to source
  checkout, not the uv cache
- Populate actual sha256 values by running all three update scripts

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Assisted-by: copilot-cli:claude-sonnet-4.6

* fix: also include pyodide

Assisted-by: CopilotCLI:gpt-5.3-codex
Signed-off-by: Henry Schreiner <henryfs@princeton.edu>

* fix: PR review comments for cache verification and docs wording

Co-authored-by: henryiii <4616906+henryiii@users.noreply.github.com>
Signed-off-by: Henry Schreiner <henryfs@princeton.edu>

* fix: require sha256 for download configs

Require sha256 for URL-backed Python and virtualenv download configs. Update the GraalPy updater to refresh macOS x86_64 entries by selecting the latest release that still has a matching asset, and fill the two missing GraalPy checksums in build-platforms.toml.

Assisted-by: CopilotCLI:gpt-5.4

* ci: remove unit test for bin item

Signed-off-by: Henry Schreiner <henryfs@princeton.edu>

* refactor: combine sha256 unit tests into test_sha256.py

Merge pyodide_test.py and python_build_standalone_test.py into a
single unit_test/test_sha256.py since both test sha256-related
behaviour.

Assisted-by: opencode:glm-5

---------

Signed-off-by: Henry Schreiner <henryfs@princeton.edu>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: henryiii <4616906+henryiii@users.noreply.github.com>
2026-05-29 15:02:33 -04:00

110 lines
3.6 KiB
Python

"""
These are utilities for the `/bin` scripts, not for the `cibuildwheel` program.
"""
from __future__ import annotations
import json
import time
import typing
import urllib.error
import urllib.request
from io import StringIO
from typing import NotRequired, Protocol
from cibuildwheel import __version__ as cibw_version
TYPE_CHECKING = False
if TYPE_CHECKING:
from collections.abc import Mapping, Sequence
from typing import Any
__all__ = ("Printable", "dump_python_configurations")
class Printable(Protocol):
def __str__(self) -> str: ...
def dump_python_configurations(
inp: Mapping[str, Mapping[str, Sequence[Mapping[str, Printable]]]],
) -> str:
output = StringIO()
for header, values in inp.items():
output.write(f"[{header}]\n")
for inner_header, listing in values.items():
output.write(f"{inner_header} = [\n")
for item in listing:
output.write(" { ")
dict_contents = (f'{key} = "{value}"' for key, value in item.items())
output.write(", ".join(dict_contents))
output.write(" },\n")
output.write("]\n")
output.write("\n")
# Strip the final newline, to avoid two blank lines at the end.
return output.getvalue()[:-1]
def _json_request(request: urllib.request.Request, timeout: int = 30) -> dict[str, Any]:
with urllib.request.urlopen(request, timeout=timeout) as response:
return typing.cast("dict[str, Any]", json.load(response))
def github_api_request(path: str, *, max_retries: int = 3) -> dict[str, Any]:
"""
Makes a GitHub API request to the given path and returns the JSON response.
"""
api_url = f"https://api.github.com/{path}"
headers = {
"Accept": "application/vnd.github.v3+json",
"User-Agent": f"cibuildwheel/{cibw_version}",
}
request = urllib.request.Request(api_url, headers=headers)
for retry_count in range(max_retries):
try:
return _json_request(request)
except (urllib.error.URLError, TimeoutError) as e:
# pylint: disable=E1101
if (
isinstance(e, urllib.error.HTTPError)
and (e.code in {403, 429})
and e.headers.get("x-ratelimit-remaining") == "0"
):
reset_time = int(e.headers.get("x-ratelimit-reset", 0))
wait_time = max(0, reset_time - int(e.headers.get("date", 0)))
print(f"Github rate limit exceeded. Waiting for {wait_time} seconds.")
time.sleep(wait_time)
else:
print(f"Retrying GitHub API request due to error: {e}")
if retry_count == max_retries - 1:
print(f"GitHub API request failed (Network error: {e}). Check network connection.")
raise e
# Should never be reached but to keep the type checker happy
msg = "Unexpected execution path in github_api_request"
raise RuntimeError(msg)
class PyodideXBuildEnvRelease(typing.TypedDict):
version: str
sha256: str
python_version: str
emscripten_version: str
min_pyodide_build_version: NotRequired[str]
max_pyodide_build_version: NotRequired[str]
class PyodideXBuildEnvInfo(typing.TypedDict):
releases: dict[str, PyodideXBuildEnvRelease]
def get_pyodide_xbuildenv_info() -> PyodideXBuildEnvInfo:
xbuildenv_info_url = (
"https://pyodide.github.io/pyodide/api/pyodide-cross-build-environments.json"
)
with urllib.request.urlopen(xbuildenv_info_url) as response:
return typing.cast("PyodideXBuildEnvInfo", json.loads(response.read().decode("utf-8")))