Hongru/cibuildwheel
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

feat: add SHA256 verification for direct downloads (#2873)

Browse Source 
* feat: add SHA256 verification for direct downloads

Store SHA256 hashes when running update scripts and verify them
when downloading files at build time. This improves security by
detecting unexpected changes to downloaded artifacts.

Platforms covered: macOS (CPython, PyPy, GraalPy), iOS, Android,
virtualenv, and python-build-standalone. Windows (nuget) and
Linux (Docker) are excluded.

SHA256 sources per platform:
- macOS/iOS/Android CPython (python.org): sha256_sum from API
- GraalPy: .sha256 sidecar assets from GitHub releases
- python-build-standalone: SHA256SUMS file in release
- PyPy, BeeWare iOS, Maven (Chaquopy): stream-download and compute

Changes:
- cibuildwheel/util/file.py: add sha256 param to download()
- cibuildwheel/platforms/{macos,ios,android}.py: add sha256 to
  PythonConfiguration and pass to download()
- cibuildwheel/venv.py: read sha256 from toml and pass to download()
- cibuildwheel/util/python_build_standalone.py: add sha256 to
  PythonBuildStandaloneAsset and pass to download()
- cibuildwheel/resources/build-platforms.toml: add sha256 fields
- cibuildwheel/resources/virtualenv.toml: add sha256 field
- cibuildwheel/resources/python-build-standalone-releases.json: add sha256
- bin/update_pythons.py: compute/store sha256 per source strategy
- bin/update_virtualenv.py: compute sha256 by streaming download
- bin/update_python_build_standalone.py: parse SHA256SUMS file

Closes #908

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Assisted-by: copilot-cli:claude-sonnet-4.6

* fix: populate sha256 in resource files and fix Windows PythonConfiguration

- Add sha256 field to Windows PythonConfiguration (PyPy/GraalPy have
  direct download URLs on Windows too)
- Pass sha256 to install_pypy() and install_graalpy() in windows.py
- Fix update_pythons.py: handle empty sha256 from CPython API (older
  versions) by streaming download to compute it; fix condition to
  check 'not sha256' rather than 'not in dict'
- Fix update_virtualenv.py: compute sha256 even when version unchanged
  but sha256 is empty (first-time population)
- Fix update_python_build_standalone.py: resolve file path relative to
  the script itself (not the installed package) so writes go to source
  checkout, not the uv cache
- Populate actual sha256 values by running all three update scripts

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Assisted-by: copilot-cli:claude-sonnet-4.6

* fix: also include pyodide

Assisted-by: CopilotCLI:gpt-5.3-codex
Signed-off-by: Henry Schreiner <henryfs@princeton.edu>

* fix: PR review comments for cache verification and docs wording

Co-authored-by: henryiii <4616906+henryiii@users.noreply.github.com>
Signed-off-by: Henry Schreiner <henryfs@princeton.edu>

* fix: require sha256 for download configs

Require sha256 for URL-backed Python and virtualenv download configs. Update the GraalPy updater to refresh macOS x86_64 entries by selecting the latest release that still has a matching asset, and fill the two missing GraalPy checksums in build-platforms.toml.

Assisted-by: CopilotCLI:gpt-5.4

* ci: remove unit test for bin item

Signed-off-by: Henry Schreiner <henryfs@princeton.edu>

* refactor: combine sha256 unit tests into test_sha256.py

Merge pyodide_test.py and python_build_standalone_test.py into a
single unit_test/test_sha256.py since both test sha256-related
behaviour.

Assisted-by: opencode:glm-5

---------

Signed-off-by: Henry Schreiner <henryfs@princeton.edu>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: henryiii <4616906+henryiii@users.noreply.github.com>
This commit is contained in:
Henry Schreiner
2026-05-29 15:02:33 -04:00
committed by GitHub
co-authored by Copilot copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> henryiii
parent 322b876891
commit cd38ee1548
18 changed files with 874 additions and 480 deletions
Download Patch File Download Diff File Expand all files Collapse all files
bin/update_pythons.py
+62 -17
View File
@@ -15,12 +15,13 @@
from __future__ import annotations
import difflib
import hashlib
import logging
import operator
import re
import tomllib
from pathlib import Path
from typing import Any, Final, Literal, TypedDict
from typing import Any, Final, Literal, NotRequired, TypedDict
from xml.etree import ElementTree as ET
import click
@@ -56,11 +57,13 @@ class Config(TypedDict):
class ConfigUrl(Config):
url: str
sha256: NotRequired[str]
class ConfigPyodide(Config):
default_pyodide_version: str
node_version: str
sha256: str
# The following set of "Versions" classes allow the initial call to the APIs to
@@ -155,10 +158,6 @@ class GraalPyVersions:
msg = f"GraalPy {arch} not found for {spec}!"
raise RuntimeError(msg)
release = releases[-1]
version = release["python_version"]
gpversion = release["graalpy_version"]
if "macosx" in identifier:
arch = "x86_64" if "x86_64" in identifier else "arm64"
platform = "macos"
@@ -171,19 +170,38 @@ class GraalPyVersions:
arch = "amd64" if arch == "x86_64" else "aarch64"
ext = "zip" if "win" in identifier else "tar.gz"
urls = [
rf["browser_download_url"]
for rf in release["assets"]
if rf["name"].endswith(f"{platform}-{arch}.{ext}")
and rf["name"].startswith(f"graalpy-{gpversion.major}")
]
if urls:
for release in reversed(releases):
version = release["python_version"]
gpversion = release["graalpy_version"]
urls = [
rf["browser_download_url"]
for rf in release["assets"]
if rf["name"].endswith(f"{platform}-{arch}.{ext}")
and rf["name"].startswith(f"graalpy-{gpversion.major}")
]
if not urls:
continue
(url,) = urls
# Fetch sha256 from the ".sha256" sidecar asset in the same release.
sha256 = ""
sha256_asset_name = url.rsplit("/", 1)[-1] + ".sha256"
sha256_urls = [
rf["browser_download_url"]
for rf in release["assets"]
if rf["name"] == sha256_asset_name
]
if sha256_urls:
sha256_response = requests.get(sha256_urls[0])
sha256_response.raise_for_status()
sha256 = sha256_response.text.strip().split()[0]
return ConfigUrl(
identifier=identifier,
version=f"{version.major}.{version.minor}",
url=url,
sha256=sha256,
)
return None
@@ -298,12 +316,14 @@ class CPythonVersions:
uri = self.versions_dict[new_version]
files = [rf for rf in self.files_info if rf["release"] == uri]
urls = [rf["url"] for rf in files if file_ident in rf["url"]]
if urls:
matching = [rf for rf in files if file_ident in rf["url"]]
if matching:
rf = matching[0]
return ConfigUrl(
identifier=identifier,
version=f"{new_version.major}.{new_version.minor}",
url=urls[0],
url=rf["url"],
sha256=rf.get("sha256_sum", ""),
)
return None
@@ -420,6 +440,7 @@ class PyodideVersions:
version=str(version),
default_pyodide_version=release["version"],
node_version=node_version,
sha256=release["sha256"],
)
@@ -448,6 +469,16 @@ class AllVersions:
self.pyodide = PyodideVersions()
def _stream_sha256(self, url: str) -> str:
"""Download a file (streaming) and return its SHA256 hex digest."""
log.debug("Computing sha256 for %s by streaming download...", url)
response = requests.get(url, stream=True)
response.raise_for_status()
hasher = hashlib.sha256()
for chunk in response.iter_content(65536):
hasher.update(chunk)
return hasher.hexdigest()
def update_config(self, config: MutableMapping[str, str]) -> None:
identifier = config["identifier"]
version = Version(config["version"])
@@ -465,8 +496,6 @@ class AllVersions:
elif "macosx_arm64" in identifier:
config_update = self.macos_pypy_arm64.update_version_macos(spec)
elif identifier.startswith("gp"):
if "macosx_x86_64" in identifier:
return
config_update = self.graalpy.update_version(identifier, spec)
elif "t-win32" in identifier and identifier.startswith("cp"):
config_update = self.windows_t_32.update_version_windows(spec)
@@ -505,6 +534,22 @@ class AllVersions:
)
assert config_update is not None, f"{identifier} not found!"
# Fill in sha256 for URL-based configs if not already provided by the
# update_version_* method (e.g. PyPy, BeeWare iOS, Maven have no sidecar).
# Also fills in sha256 when the CPython API doesn't return a sha256_sum
# (e.g. for older releases).
# Widen the type to allow arbitrary key access on the underlying dict.
config_update_dict: dict[str, str] = config_update # type: ignore[assignment]
if "url" in config_update_dict and not config_update_dict.get("sha256"):
url = config_update_dict["url"]
existing_sha256 = config.get("sha256", "")
if url == config.get("url") and existing_sha256:
# URL unchanged — preserve the existing sha256
config_update_dict["sha256"] = existing_sha256
else:
config_update_dict["sha256"] = self._stream_sha256(url)
if config_update != config:
log.info(" Updated %s to %s", config, config_update)
config.clear()