feat: Pyodide improvements: version setting, standalone environments (#2002)

* Fix a typo: pyoodide ➡️ pyodide

* Add `pyodide_build_version` attribute

* Add version to xbuildenv log step

* Add version to Emscripten log step

* Use `pyodide-build`'s version for updating constraints

* Bump Pyodide constraints by updating `pyodide-build`

* Add a schema for `pyodide-version`

* Update Pyodide constraints

* Bump `pyodide-build` to new 0.29.0

* Test out another Pyodide identifier

* Update outdated Pyodide constraints

* Add Pyodide version to temp directory name

* Remove Pyodide 0.26.1 from build configurations

* Retrieve + validate + install specific xbuildenvs

* Test wheel builds with Pyodide 0.26.2

* Add correct Pyodide version to identifier temp dir

* Don't pre-call Pyodide xbuildenv search

* Fetch just the stable Pyodide versions

* Refactor search + validation + install into one step

* Move all of it under a lock

* Reorder xbuildenv installation

* Add env and cwd to xbuildenv search call

* Temporarily lower to 0.26.2 target

* Separate out search, validate, install; again

* Run xbuildenv search in `CIBW_CACHE_PATH`

* Remove prior `PYODIDE_ROOT` env vars, copy envs

* Validate doesn't need to depend on searching

* Add file lock when searching xbuildenvs

* Test the original version: 0.26.1

* Update Pyodide constraints

* Update constraints for `pyodide-build` 0.29.0 again

* Bump Pyodide from version 0.26.1 ➡️ version 0.26.4

* Add note on compatibility for macOS + other archs

* Note Pyodide version for Pyodide identifier

* Docs about `CIBW_PYODIDE_VERSION`

* Don't fetch just the stable versions

* Discard a variable that's not used later

* Rename `search_xbuildenv` ➡️ `get_xbuildenv_versions`

* `validate_xbuildenv` ➡️ `validate_xbuildenv_version`

* Replace ordered comment, add newline

* Replace sentence on macOS support

Co-Authored-By: Hood Chatham <roberthoodchatham@gmail.com>

* Capitalise: "pyodide" ➡️ "Pyodide"

Co-Authored-By: Hood Chatham <roberthoodchatham@gmail.com>

* "work" ➡️ "may succeed"

Co-Authored-By: Hood Chatham <roberthoodchatham@gmail.com>

* Add another job to test a custom Pyodide version

* Handle "v"-prefixed + non-prefixed versions

* Convert to a proper toml-able option, and remove some hardcoded versions

This removes the enscripten and pyodide-build version specs from
pyproject.toml - pyodide-build is spec'd in the constraints file, and
the emscripten version can be read from the pyodide-build output.

* Add a schema entry

* Add docs for CIBW_PYODIDE_VERSION

* Rephrase

* Add tests for pyodide-version

* Apply suggestions from code review

* Add python_build_standalone util

* Hook up to python-build-standalone, removing dependency on host python

* Remove python hard-code in action.yml

* Add log step

* Add workaround for https://github.com/pyodide/pyodide-build/issues/143

* Add emscripten pytest test

* Remove unneeded checks

* Fix a pytest invoke for emscripten

* Generate pyodide-build constraints from the pinned pyodide version

* Remove pyodide python-build-standalone workaround

* Fixup paths from newer version of pyodide-build

* Fix/skip some failing tests

* Use `python -m pytest` on pyodide, even on Linux

* Docs fixes

* Don't call the github API at runtime, cache the release assets instead

* Ignore pylint false positive

* Add version auto-updating for pyodide

* Add support for pyodide 3.13.

* Fix tests for multiple pyodide wheels

* Remove workaround for unreleased pyodide-build

* Rename to "test_pyodide"

* Fix pathname confusion

* Remove extra github actions job

* Fix expectation for test_abi_none

* Fix the custom_repair_wheel test to actually have clashing names

* Fix pinned version test

* Document test-command limitation

* Remove pyodide 0.28.0a1 for now

* Update constraints files

* Docs/test fixes post removing pyodide cp313

* Fix ABI test expectation

* Docs improvements

* Improve some comments

* Remove logic duplication

* remove pyodide special casing

* Refactor constraints code to use a utility script, circumventing import issues

* chore: nicer nox env

Signed-off-by: Henry Schreiner <henryschreineriii@gmail.com>

* fix: typo in variable name found by copilot

Signed-off-by: Henry Schreiner <henryschreineriii@gmail.com>

* Apply suggestions from code review

* Some more Pyodide version updates in the docs section

* We haven't released Pyodide v0.27.6 yet

* Back to the Github URL for cross-build-environments

* `pyodide-build`, not `emsdk` for Windows skips

Co-Authored-By: Hood Chatham <roberthoodchatham@gmail.com>

* Move to a separate `_json_request` function

Co-Authored-By: Hood Chatham <roberthoodchatham@gmail.com>

* Rename "retries" ➡️ "retry_count"

* Add some type hints

* Copy env vars before `UV_CUSTOM_COMPILE_COMMAND`

* Use `HTTPError.headers.get` instead

* Remove extra end quote

* Change download tests URL to `https://badssl.com/`

Co-Authored-By: Joe Rickerby <1244307+joerick@users.noreply.github.com>

* Download size changes, too

* Use jsdelivr for Github asset mirroring

* Bump to Pyodide v0.27.6

* Fix unit tests

* Move to pyodide v0.27.6 again

* Bump to pyodide-build 0.30.4

* Use new URL for cross-build environments metadata

Co-authored-by: Joe Rickerby <joerick@mac.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Henry Schreiner <henryschreineriii@gmail.com>
Co-authored-by: Hood Chatham <roberthoodchatham@gmail.com>
Co-authored-by: Joe Rickerby <joerick@mac.com>
Co-authored-by: Henry Schreiner <henryschreineriii@gmail.com>
Co-authored-by: Joe Rickerby <1244307+joerick@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

bin/update_pythons.py

@@ -19,7 +19,7 @@ from packaging.version import Version
 from rich.logging import RichHandler
 from rich.syntax import Syntax
 
-from cibuildwheel.extra import dump_python_configurations
+from cibuildwheel.extra import dump_python_configurations, get_pyodide_xbuildenv_info
 
 log = logging.getLogger("cibw")
 
@@ -57,7 +57,14 @@ class ConfigApple(TypedDict):
     url: str
 
 
-AnyConfig = ConfigWinCP | ConfigWinPP | ConfigWinGP | ConfigApple
+class ConfigPyodide(TypedDict):
+    identifier: str
+    version: str
+    default_pyodide_version: str
+    node_version: str
+
+
+AnyConfig = ConfigWinCP | ConfigWinPP | ConfigWinGP | ConfigApple | ConfigPyodide
 
 
 # The following set of "Versions" classes allow the initial call to the APIs to
@@ -347,6 +354,39 @@ class CPythonIOSVersions:
         return None
 
 
+class PyodideVersions:
+    def __init__(self) -> None:
+        xbuildenv_info = get_pyodide_xbuildenv_info()
+        self.releases = xbuildenv_info["releases"]
+
+    def update_version_pyodide(
+        self, identifier: str, version: Version, spec: Specifier, node_version: str
+    ) -> ConfigPyodide | None:
+        # get releases that match the python version
+        releases = [
+            r for r in self.releases.values() if spec.contains(Version(r["python_version"]))
+        ]
+        # sort by version, latest first
+        releases.sort(key=lambda r: Version(r["version"]), reverse=True)
+
+        if not releases:
+            msg = f"Pyodide not found for {spec}!"
+            raise ValueError(msg)
+
+        final_releases = [r for r in releases if not Version(r["version"]).is_prerelease]
+
+        # prefer a final release if available, otherwise use the latest
+        # pre-release
+        release = final_releases[0] if final_releases else releases[0]
+
+        return ConfigPyodide(
+            identifier=identifier,
+            version=str(version),
+            default_pyodide_version=release["version"],
+            node_version=node_version,
+        )
+
+
 # This is a universal interface to all the above Versions classes. Given an
 # identifier, it updates a config dict.
 
@@ -369,6 +409,8 @@ class AllVersions:
 
         self.graalpy = GraalPyVersions()
 
+        self.pyodide = PyodideVersions()
+
     def update_config(self, config: MutableMapping[str, str]) -> None:
         identifier = config["identifier"]
         version = Version(config["version"])
@@ -407,6 +449,10 @@ class AllVersions:
             config_update = self.windows_arm64.update_version_windows(spec)
         elif "ios" in identifier:
             config_update = self.ios_cpython.update_version_ios(identifier, version)
+        elif "pyodide" in identifier:
+            config_update = self.pyodide.update_version_pyodide(
+                identifier, version, spec, config["node_version"]
+            )
 
         assert config_update is not None, f"{identifier} not found!"
         config.update(**config_update)
@@ -445,6 +491,9 @@ def update_pythons(force: bool, level: str) -> None:
     for config in configs["ios"]["python_configurations"]:
         all_versions.update_config(config)
 
+    for config in configs["pyodide"]["python_configurations"]:
+        all_versions.update_config(config)
+
     result_toml = dump_python_configurations(configs)
 
     rich.print()  # spacer