feat: Pyodide improvements: version setting, standalone environments (#2002)

* Fix a typo: pyoodide ➡️ pyodide * Add `pyodide_build_version` attribute * Add version to xbuildenv log step * Add version to Emscripten log step * Use `pyodide-build`'s version for updating constraints * Bump Pyodide constraints by updating `pyodide-build` * Add a schema for `pyodide-version` * Update Pyodide constraints * Bump `pyodide-build` to new 0.29.0 * Test out another Pyodide identifier * Update outdated Pyodide constraints * Add Pyodide version to temp directory name * Remove Pyodide 0.26.1 from build configurations * Retrieve + validate + install specific xbuildenvs * Test wheel builds with Pyodide 0.26.2 * Add correct Pyodide version to identifier temp dir * Don't pre-call Pyodide xbuildenv search * Fetch just the stable Pyodide versions * Refactor search + validation + install into one step * Move all of it under a lock * Reorder xbuildenv installation * Add env and cwd to xbuildenv search call * Temporarily lower to 0.26.2 target * Separate out search, validate, install; again * Run xbuildenv search in `CIBW_CACHE_PATH` * Remove prior `PYODIDE_ROOT` env vars, copy envs * Validate doesn't need to depend on searching * Add file lock when searching xbuildenvs * Test the original version: 0.26.1 * Update Pyodide constraints * Update constraints for `pyodide-build` 0.29.0 again * Bump Pyodide from version 0.26.1 ➡️ version 0.26.4 * Add note on compatibility for macOS + other archs * Note Pyodide version for Pyodide identifier * Docs about `CIBW_PYODIDE_VERSION` * Don't fetch just the stable versions * Discard a variable that's not used later * Rename `search_xbuildenv` ➡️ `get_xbuildenv_versions` * `validate_xbuildenv` ➡️ `validate_xbuildenv_version` * Replace ordered comment, add newline * Replace sentence on macOS support Co-Authored-By: Hood Chatham <roberthoodchatham@gmail.com> * Capitalise: "pyodide" ➡️ "Pyodide" Co-Authored-By: Hood Chatham <roberthoodchatham@gmail.com> * "work" ➡️ "may succeed" Co-Authored-By: Hood Chatham <roberthoodchatham@gmail.com> * Add another job to test a custom Pyodide version * Handle "v"-prefixed + non-prefixed versions * Convert to a proper toml-able option, and remove some hardcoded versions This removes the enscripten and pyodide-build version specs from pyproject.toml - pyodide-build is spec'd in the constraints file, and the emscripten version can be read from the pyodide-build output. * Add a schema entry * Add docs for CIBW_PYODIDE_VERSION * Rephrase * Add tests for pyodide-version * Apply suggestions from code review * Add python_build_standalone util * Hook up to python-build-standalone, removing dependency on host python * Remove python hard-code in action.yml * Add log step * Add workaround for https://github.com/pyodide/pyodide-build/issues/143 * Add emscripten pytest test * Remove unneeded checks * Fix a pytest invoke for emscripten * Generate pyodide-build constraints from the pinned pyodide version * Remove pyodide python-build-standalone workaround * Fixup paths from newer version of pyodide-build * Fix/skip some failing tests * Use `python -m pytest` on pyodide, even on Linux * Docs fixes * Don't call the github API at runtime, cache the release assets instead * Ignore pylint false positive * Add version auto-updating for pyodide * Add support for pyodide 3.13. * Fix tests for multiple pyodide wheels * Remove workaround for unreleased pyodide-build * Rename to "test_pyodide" * Fix pathname confusion * Remove extra github actions job * Fix expectation for test_abi_none * Fix the custom_repair_wheel test to actually have clashing names * Fix pinned version test * Document test-command limitation * Remove pyodide 0.28.0a1 for now * Update constraints files * Docs/test fixes post removing pyodide cp313 * Fix ABI test expectation * Docs improvements * Improve some comments * Remove logic duplication * remove pyodide special casing * Refactor constraints code to use a utility script, circumventing import issues * chore: nicer nox env Signed-off-by: Henry Schreiner <henryschreineriii@gmail.com> * fix: typo in variable name found by copilot Signed-off-by: Henry Schreiner <henryschreineriii@gmail.com> * Apply suggestions from code review * Some more Pyodide version updates in the docs section * We haven't released Pyodide v0.27.6 yet * Back to the Github URL for cross-build-environments * `pyodide-build`, not `emsdk` for Windows skips Co-Authored-By: Hood Chatham <roberthoodchatham@gmail.com> * Move to a separate `_json_request` function Co-Authored-By: Hood Chatham <roberthoodchatham@gmail.com> * Rename "retries" ➡️ "retry_count" * Add some type hints * Copy env vars before `UV_CUSTOM_COMPILE_COMMAND` * Use `HTTPError.headers.get` instead * Remove extra end quote * Change download tests URL to `https://badssl.com/` Co-Authored-By: Joe Rickerby <1244307+joerick@users.noreply.github.com> * Download size changes, too * Use jsdelivr for Github asset mirroring * Bump to Pyodide v0.27.6 * Fix unit tests * Move to pyodide v0.27.6 again * Bump to pyodide-build 0.30.4 * Use new URL for cross-build environments metadata Co-authored-by: Joe Rickerby <joerick@mac.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: Henry Schreiner <henryschreineriii@gmail.com> Co-authored-by: Hood Chatham <roberthoodchatham@gmail.com> Co-authored-by: Joe Rickerby <joerick@mac.com> Co-authored-by: Henry Schreiner <henryschreineriii@gmail.com> Co-authored-by: Joe Rickerby <1244307+joerick@users.noreply.github.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2025-05-20 16:59:58 -04:00
parent c3c260dff7
commit 163864158d
30 changed files with 1298 additions and 115 deletions
@@ -0,0 +1,69 @@
+#!/usr/bin/env python3
+
+import sys
+import textwrap
+from pathlib import Path
+
+import click
+
+from cibuildwheel.extra import get_pyodide_xbuildenv_info
+
+
+@click.command()
+@click.argument(
+    "pyodide-version",
+    type=str,
+)
+@click.option(
+    "--output-file",
+    type=click.Path(),
+    default=None,
+    help="Output file to write the constraints to. If not provided, the constraints will be printed to stdout.",
+)
+def generate_pyodide_constraints(pyodide_version: str, output_file: str | None = None) -> None:
+    """
+    Generate constraints for a specific Pyodide version. The constraints are
+    generated based on the Pyodide version's xbuildenv info, which is retrieved
+    from the Pyodide repository.
+
+    These constraints should then be 'pinned' using `uv pip compile`.
+
+    Example usage:
+
+        bin/generate_pyodide_constraints.py 0.27.0
+    """
+    xbuildenv_info = get_pyodide_xbuildenv_info()
+    try:
+        pyodide_version_xbuildenv_info = xbuildenv_info["releases"][pyodide_version]
+    except KeyError as e:
+        msg = f"Pyodide version {pyodide_version} not found in xbuildenv info. Versions available: {', '.join(xbuildenv_info['releases'].keys())}"
+        raise click.BadParameter(msg) from e
+
+    pyodide_build_min_version = pyodide_version_xbuildenv_info.get("min_pyodide_build_version")
+    pyodide_build_max_version = pyodide_version_xbuildenv_info.get("max_pyodide_build_version")
+
+    pyodide_build_specifier_parts: list[str] = []
+
+    if pyodide_build_min_version:
+        pyodide_build_specifier_parts.append(f">={pyodide_build_min_version}")
+    if pyodide_build_max_version:
+        pyodide_build_specifier_parts.append(f"<={pyodide_build_max_version}")
+
+    pyodide_build_specifier = ",".join(pyodide_build_specifier_parts)
+
+    constraints_txt = textwrap.dedent(f"""
+        pip
+        build[virtualenv]
+        pyodide-build{pyodide_build_specifier}
+        click<8.2
+    """)
+
+    if output_file is None:
+        print(constraints_txt)
+    else:
+        Path(output_file).write_text(constraints_txt)
+        print(f"Constraints written to {output_file}", file=sys.stderr)
+
+
+if __name__ == "__main__":
+    generate_pyodide_constraints()
@@ -197,6 +197,9 @@ properties:
  xbuild-tools:
    description: Binaries on the path that should be included in an isolated cross-build environment
    type: string_array
+  pyodide-version:
+    type: string
+    description: Specify the version of Pyodide to use
  repair-wheel-command:
    description: Execute a shell command to repair each built wheel.
    type: string_array
@@ -0,0 +1,59 @@
+#!/usr/bin/env python3
+
+import json
+
+from cibuildwheel.extra import github_api_request
+from cibuildwheel.util.python_build_standalone import (
+    PythonBuildStandaloneAsset,
+    PythonBuildStandaloneReleaseData,
+)
+from cibuildwheel.util.resources import PYTHON_BUILD_STANDALONE_RELEASES
+
+
+def main() -> None:
+    """
+    This script updates the vendored list of release assets to the latest
+    version of astral-sh/python-build-standalone.
+    """
+
+    # Get the latest release tag from the GitHub API
+    latest_release = github_api_request("repos/astral-sh/python-build-standalone/releases/latest")
+    latest_tag = latest_release["tag_name"]
+
+    # Get the list of assets for the latest release
+    github_assets = github_api_request(
+        f"repos/astral-sh/python-build-standalone/releases/tags/{latest_tag}"
+    )["assets"]
+
+    assets: list[PythonBuildStandaloneAsset] = []
+
+    for github_asset in github_assets:
+        name = github_asset["name"]
+        if not name.endswith("install_only.tar.gz"):
+            continue
+        url = github_asset["browser_download_url"]
+        assets.append({"name": name, "url": url})
+
+    # Write the assets to the JSON file. One day, we might need to support
+    # multiple releases, but for now, we only support the latest one
+    json_file_contents: PythonBuildStandaloneReleaseData = {
+        "releases": [
+            {
+                "tag": latest_tag,
+                "assets": assets,
+            }
+        ]
+    }
+
+    with PYTHON_BUILD_STANDALONE_RELEASES.open("w", encoding="utf-8") as f:
+        json.dump(json_file_contents, f, indent=2)
+        # Add a trailing newline, our pre-commit hook requires it
+        f.write("\n")
+
+    print(
+        f"Updated {PYTHON_BUILD_STANDALONE_RELEASES.name} with {len(assets)} assets for tag {latest_tag}"
+    )
+
+
+if __name__ == "__main__":
+    main()
@@ -19,7 +19,7 @@ from packaging.version import Version
 from rich.logging import RichHandler
 from rich.syntax import Syntax

-from cibuildwheel.extra import dump_python_configurations
+from cibuildwheel.extra import dump_python_configurations, get_pyodide_xbuildenv_info

 log = logging.getLogger("cibw")

@@ -57,7 +57,14 @@ class ConfigApple(TypedDict):
    url: str


-AnyConfig = ConfigWinCP | ConfigWinPP | ConfigWinGP | ConfigApple
+class ConfigPyodide(TypedDict):
+    identifier: str
+    version: str
+    default_pyodide_version: str
+    node_version: str
+
+
+AnyConfig = ConfigWinCP | ConfigWinPP | ConfigWinGP | ConfigApple | ConfigPyodide


 # The following set of "Versions" classes allow the initial call to the APIs to
@@ -347,6 +354,39 @@ class CPythonIOSVersions:
        return None


+class PyodideVersions:
+    def __init__(self) -> None:
+        xbuildenv_info = get_pyodide_xbuildenv_info()
+        self.releases = xbuildenv_info["releases"]
+
+    def update_version_pyodide(
+        self, identifier: str, version: Version, spec: Specifier, node_version: str
+    ) -> ConfigPyodide | None:
+        # get releases that match the python version
+        releases = [
+            r for r in self.releases.values() if spec.contains(Version(r["python_version"]))
+        ]
+        # sort by version, latest first
+        releases.sort(key=lambda r: Version(r["version"]), reverse=True)
+
+        if not releases:
+            msg = f"Pyodide not found for {spec}!"
+            raise ValueError(msg)
+
+        final_releases = [r for r in releases if not Version(r["version"]).is_prerelease]
+
+        # prefer a final release if available, otherwise use the latest
+        # pre-release
+        release = final_releases[0] if final_releases else releases[0]
+
+        return ConfigPyodide(
+            identifier=identifier,
+            version=str(version),
+            default_pyodide_version=release["version"],
+            node_version=node_version,
+        )
+
+
 # This is a universal interface to all the above Versions classes. Given an
 # identifier, it updates a config dict.

@@ -369,6 +409,8 @@ class AllVersions:

        self.graalpy = GraalPyVersions()

+        self.pyodide = PyodideVersions()
+
    def update_config(self, config: MutableMapping[str, str]) -> None:
        identifier = config["identifier"]
        version = Version(config["version"])
@@ -407,6 +449,10 @@ class AllVersions:
            config_update = self.windows_arm64.update_version_windows(spec)
        elif "ios" in identifier:
            config_update = self.ios_cpython.update_version_ios(identifier, version)
+        elif "pyodide" in identifier:
+            config_update = self.pyodide.update_version_pyodide(
+                identifier, version, spec, config["node_version"]
+            )

        assert config_update is not None, f"{identifier} not found!"
        config.update(**config_update)
@@ -445,6 +491,9 @@ def update_pythons(force: bool, level: str) -> None:
    for config in configs["ios"]["python_configurations"]:
        all_versions.update_config(config)

+    for config in configs["pyodide"]["python_configurations"]:
+        all_versions.update_config(config)
+
    result_toml = dump_python_configurations(configs)

    rich.print()  # spacer