feat: Pyodide improvements: version setting, standalone environments (#2002)
* Fix a typo: pyoodide ➡️ pyodide * Add `pyodide_build_version` attribute * Add version to xbuildenv log step * Add version to Emscripten log step * Use `pyodide-build`'s version for updating constraints * Bump Pyodide constraints by updating `pyodide-build` * Add a schema for `pyodide-version` * Update Pyodide constraints * Bump `pyodide-build` to new 0.29.0 * Test out another Pyodide identifier * Update outdated Pyodide constraints * Add Pyodide version to temp directory name * Remove Pyodide 0.26.1 from build configurations * Retrieve + validate + install specific xbuildenvs * Test wheel builds with Pyodide 0.26.2 * Add correct Pyodide version to identifier temp dir * Don't pre-call Pyodide xbuildenv search * Fetch just the stable Pyodide versions * Refactor search + validation + install into one step * Move all of it under a lock * Reorder xbuildenv installation * Add env and cwd to xbuildenv search call * Temporarily lower to 0.26.2 target * Separate out search, validate, install; again * Run xbuildenv search in `CIBW_CACHE_PATH` * Remove prior `PYODIDE_ROOT` env vars, copy envs * Validate doesn't need to depend on searching * Add file lock when searching xbuildenvs * Test the original version: 0.26.1 * Update Pyodide constraints * Update constraints for `pyodide-build` 0.29.0 again * Bump Pyodide from version 0.26.1 ➡️ version 0.26.4 * Add note on compatibility for macOS + other archs * Note Pyodide version for Pyodide identifier * Docs about `CIBW_PYODIDE_VERSION` * Don't fetch just the stable versions * Discard a variable that's not used later * Rename `search_xbuildenv` ➡️ `get_xbuildenv_versions` * `validate_xbuildenv` ➡️ `validate_xbuildenv_version` * Replace ordered comment, add newline * Replace sentence on macOS support Co-Authored-By: Hood Chatham <roberthoodchatham@gmail.com> * Capitalise: "pyodide" ➡️ "Pyodide" Co-Authored-By: Hood Chatham <roberthoodchatham@gmail.com> * "work" ➡️ "may succeed" Co-Authored-By: Hood Chatham <roberthoodchatham@gmail.com> * Add another job to test a custom Pyodide version * Handle "v"-prefixed + non-prefixed versions * Convert to a proper toml-able option, and remove some hardcoded versions This removes the enscripten and pyodide-build version specs from pyproject.toml - pyodide-build is spec'd in the constraints file, and the emscripten version can be read from the pyodide-build output. * Add a schema entry * Add docs for CIBW_PYODIDE_VERSION * Rephrase * Add tests for pyodide-version * Apply suggestions from code review * Add python_build_standalone util * Hook up to python-build-standalone, removing dependency on host python * Remove python hard-code in action.yml * Add log step * Add workaround for https://github.com/pyodide/pyodide-build/issues/143 * Add emscripten pytest test * Remove unneeded checks * Fix a pytest invoke for emscripten * Generate pyodide-build constraints from the pinned pyodide version * Remove pyodide python-build-standalone workaround * Fixup paths from newer version of pyodide-build * Fix/skip some failing tests * Use `python -m pytest` on pyodide, even on Linux * Docs fixes * Don't call the github API at runtime, cache the release assets instead * Ignore pylint false positive * Add version auto-updating for pyodide * Add support for pyodide 3.13. * Fix tests for multiple pyodide wheels * Remove workaround for unreleased pyodide-build * Rename to "test_pyodide" * Fix pathname confusion * Remove extra github actions job * Fix expectation for test_abi_none * Fix the custom_repair_wheel test to actually have clashing names * Fix pinned version test * Document test-command limitation * Remove pyodide 0.28.0a1 for now * Update constraints files * Docs/test fixes post removing pyodide cp313 * Fix ABI test expectation * Docs improvements * Improve some comments * Remove logic duplication * remove pyodide special casing * Refactor constraints code to use a utility script, circumventing import issues * chore: nicer nox env Signed-off-by: Henry Schreiner <henryschreineriii@gmail.com> * fix: typo in variable name found by copilot Signed-off-by: Henry Schreiner <henryschreineriii@gmail.com> * Apply suggestions from code review * Some more Pyodide version updates in the docs section * We haven't released Pyodide v0.27.6 yet * Back to the Github URL for cross-build-environments * `pyodide-build`, not `emsdk` for Windows skips Co-Authored-By: Hood Chatham <roberthoodchatham@gmail.com> * Move to a separate `_json_request` function Co-Authored-By: Hood Chatham <roberthoodchatham@gmail.com> * Rename "retries" ➡️ "retry_count" * Add some type hints * Copy env vars before `UV_CUSTOM_COMPILE_COMMAND` * Use `HTTPError.headers.get` instead * Remove extra end quote * Change download tests URL to `https://badssl.com/` Co-Authored-By: Joe Rickerby <1244307+joerick@users.noreply.github.com> * Download size changes, too * Use jsdelivr for Github asset mirroring * Bump to Pyodide v0.27.6 * Fix unit tests * Move to pyodide v0.27.6 again * Bump to pyodide-build 0.30.4 * Use new URL for cross-build environments metadata Co-authored-by: Joe Rickerby <joerick@mac.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: Henry Schreiner <henryschreineriii@gmail.com> Co-authored-by: Hood Chatham <roberthoodchatham@gmail.com> Co-authored-by: Joe Rickerby <joerick@mac.com> Co-authored-by: Henry Schreiner <henryschreineriii@gmail.com> Co-authored-by: Joe Rickerby <1244307+joerick@users.noreply.github.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
This commit is contained in:
Agriya Khetarpal
co-authored by
Hood Chatham
Joe Rickerby
Joe Rickerby
Henry Schreiner
pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Hood Chatham
Joe Rickerby
Joe Rickerby
Henry Schreiner
pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
parent
c3c260dff7
commit
163864158d
@@ -1,13 +1,15 @@
|
||||
import functools
|
||||
import json
|
||||
import os
|
||||
import shutil
|
||||
import sys
|
||||
import tomllib
|
||||
import typing
|
||||
from collections.abc import Set
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from tempfile import TemporaryDirectory
|
||||
from typing import Final
|
||||
from typing import Final, TypedDict
|
||||
|
||||
from filelock import FileLock
|
||||
|
||||
@@ -28,8 +30,12 @@ from ..util.file import (
|
||||
extract_zip,
|
||||
move_file,
|
||||
)
|
||||
from ..util.helpers import prepare_command
|
||||
from ..util.helpers import prepare_command, unwrap, unwrap_preserving_paragraphs
|
||||
from ..util.packaging import combine_constraints, find_compatible_wheel, get_pip_version
|
||||
from ..util.python_build_standalone import (
|
||||
PythonBuildStandaloneError,
|
||||
create_python_build_standalone_environment,
|
||||
)
|
||||
from ..venv import constraint_flags, virtualenv
|
||||
|
||||
IS_WIN: Final[bool] = sys.platform.startswith("win")
|
||||
@@ -39,12 +45,23 @@ IS_WIN: Final[bool] = sys.platform.startswith("win")
|
||||
class PythonConfiguration:
|
||||
version: str
|
||||
identifier: str
|
||||
pyodide_version: str
|
||||
pyodide_build_version: str
|
||||
emscripten_version: str
|
||||
default_pyodide_version: str
|
||||
node_version: str
|
||||
|
||||
|
||||
class PyodideXBuildEnvInfoVersionRange(TypedDict):
|
||||
min: str | None
|
||||
max: str | None
|
||||
|
||||
|
||||
class PyodideXBuildEnvInfo(TypedDict):
|
||||
version: str
|
||||
python: str
|
||||
emscripten: str
|
||||
pyodide_build: PyodideXBuildEnvInfoVersionRange
|
||||
compatible: bool
|
||||
|
||||
|
||||
@functools.cache
|
||||
def ensure_node(major_version: str) -> Path:
|
||||
with resources.NODEJS.open("rb") as f:
|
||||
@@ -77,8 +94,6 @@ def ensure_node(major_version: str) -> Path:
|
||||
|
||||
|
||||
def install_emscripten(tmp: Path, version: str) -> Path:
|
||||
# We don't need to match the emsdk version to the version we install, but
|
||||
# we do for stability
|
||||
url = f"https://github.com/emscripten-core/emsdk/archive/refs/tags/{version}.zip"
|
||||
installation_path = CIBW_CACHE_PATH / f"emsdk-{version}"
|
||||
emsdk_path = installation_path / f"emsdk-{version}/emsdk"
|
||||
@@ -96,16 +111,82 @@ def install_emscripten(tmp: Path, version: str) -> Path:
|
||||
return emcc_path
|
||||
|
||||
|
||||
def get_all_xbuildenv_version_info(env: dict[str, str]) -> list[PyodideXBuildEnvInfo]:
|
||||
xbuildenvs_info_str = call(
|
||||
"pyodide",
|
||||
"xbuildenv",
|
||||
"search",
|
||||
"--json",
|
||||
"--all",
|
||||
env=env,
|
||||
cwd=CIBW_CACHE_PATH,
|
||||
capture_stdout=True,
|
||||
).strip()
|
||||
|
||||
xbuildenvs_info = json.loads(xbuildenvs_info_str)
|
||||
|
||||
if "environments" not in xbuildenvs_info:
|
||||
msg = f"Invalid xbuildenvs info, got {xbuildenvs_info}"
|
||||
raise ValueError(msg)
|
||||
|
||||
return typing.cast(list[PyodideXBuildEnvInfo], xbuildenvs_info["environments"])
|
||||
|
||||
|
||||
def get_xbuildenv_version_info(
|
||||
env: dict[str, str], version: str, pyodide_build_version: str
|
||||
) -> PyodideXBuildEnvInfo:
|
||||
xbuildenvs_info = get_all_xbuildenv_version_info(env)
|
||||
for xbuildenv_info in xbuildenvs_info:
|
||||
if xbuildenv_info["version"] == version:
|
||||
return xbuildenv_info
|
||||
|
||||
msg = unwrap(f"""
|
||||
Could not find Pyodide cross-build environment version {version} in the available
|
||||
versions as reported by pyodide-build v{pyodide_build_version}.
|
||||
Available pyodide xbuildenv versions are:
|
||||
{", ".join(e["version"] for e in xbuildenvs_info if e["compatible"])}
|
||||
""")
|
||||
raise errors.FatalError(msg)
|
||||
|
||||
|
||||
# The default pyodide xbuildenv version that's specified in
|
||||
# build-platforms.toml is compatible with the pyodide-build version that's
|
||||
# pinned in the bundled constraints file. But if the user changes
|
||||
# pyodide-version and/or dependency-constraints in the cibuildwheel config, we
|
||||
# need to check if the xbuildenv version is compatible with the pyodide-build
|
||||
# version.
|
||||
def validate_pyodide_build_version(
|
||||
xbuildenv_info: PyodideXBuildEnvInfo, pyodide_build_version: str
|
||||
) -> None:
|
||||
"""
|
||||
Validate the Pyodide version is compatible with the installed
|
||||
pyodide-build version.
|
||||
"""
|
||||
|
||||
pyodide_version = xbuildenv_info["version"]
|
||||
|
||||
if not xbuildenv_info["compatible"]:
|
||||
msg = unwrap_preserving_paragraphs(f"""
|
||||
The Pyodide xbuildenv version {pyodide_version} is not compatible
|
||||
with the pyodide-build version {pyodide_build_version}. Please use
|
||||
the 'pyodide xbuildenv search --all' command to find a compatible
|
||||
version.
|
||||
|
||||
Set the pyodide-build version using the `dependency-constraints`
|
||||
option, or set the Pyodide xbuildenv version using the
|
||||
`pyodide-version` option.
|
||||
""")
|
||||
raise errors.FatalError(msg)
|
||||
|
||||
|
||||
def install_xbuildenv(env: dict[str, str], pyodide_build_version: str, pyodide_version: str) -> str:
|
||||
"""Install a particular Pyodide xbuildenv version and set a path to the Pyodide root."""
|
||||
# Since pyodide-build was unvendored from Pyodide v0.27.0, the versions of pyodide-build are
|
||||
# not guaranteed to match the versions of Pyodide or be in sync with them. Hence, we shall
|
||||
# specify the pyodide-build version in the root path, which will set up the xbuildenv for
|
||||
# the requested Pyodide version.
|
||||
pyodide_root = (
|
||||
CIBW_CACHE_PATH
|
||||
/ f".pyodide-xbuildenv-{pyodide_build_version}/{pyodide_version}/xbuildenv/pyodide-root"
|
||||
)
|
||||
# Since pyodide-build was unvendored from Pyodide v0.27.0, the versions of
|
||||
# pyodide-build are uncoupled from the versions of Pyodide. So, we specify
|
||||
# both the pyodide-build version and the Pyodide version in the temp path.
|
||||
xbuildenv_cache_path = CIBW_CACHE_PATH / f"pyodide-build-{pyodide_build_version}"
|
||||
pyodide_root = xbuildenv_cache_path / pyodide_version / "xbuildenv" / "pyodide-root"
|
||||
|
||||
with FileLock(CIBW_CACHE_PATH / "xbuildenv.lock"):
|
||||
if pyodide_root.exists():
|
||||
return str(pyodide_root)
|
||||
@@ -114,31 +195,36 @@ def install_xbuildenv(env: dict[str, str], pyodide_build_version: str, pyodide_v
|
||||
# PYODIDE_ROOT so copy it first.
|
||||
env = dict(env)
|
||||
env.pop("PYODIDE_ROOT", None)
|
||||
|
||||
# Install the xbuildenv
|
||||
call(
|
||||
"pyodide",
|
||||
"xbuildenv",
|
||||
"install",
|
||||
"--path",
|
||||
str(xbuildenv_cache_path),
|
||||
pyodide_version,
|
||||
env=env,
|
||||
cwd=CIBW_CACHE_PATH,
|
||||
)
|
||||
assert pyodide_root.exists()
|
||||
|
||||
return str(pyodide_root)
|
||||
|
||||
|
||||
def get_base_python(identifier: str) -> Path:
|
||||
implementation_id = identifier.split("-")[0]
|
||||
majorminor = implementation_id[len("cp") :]
|
||||
version_info = (int(majorminor[0]), int(majorminor[1:]))
|
||||
if version_info == sys.version_info[:2]:
|
||||
return Path(sys.executable)
|
||||
|
||||
major_minor = ".".join(str(v) for v in version_info)
|
||||
python_name = f"python{major_minor}"
|
||||
which_python = shutil.which(python_name)
|
||||
if which_python is None:
|
||||
msg = f"CPython {major_minor} is not installed."
|
||||
raise errors.FatalError(msg)
|
||||
return Path(which_python)
|
||||
def get_base_python(tmp: Path, python_configuration: PythonConfiguration) -> Path:
|
||||
try:
|
||||
return create_python_build_standalone_environment(
|
||||
python_version=python_configuration.version,
|
||||
temp_dir=tmp,
|
||||
cache_dir=CIBW_CACHE_PATH,
|
||||
)
|
||||
except PythonBuildStandaloneError as e:
|
||||
msg = unwrap(f"""
|
||||
Failed to create a Python build environment:
|
||||
{e}
|
||||
""")
|
||||
raise errors.FatalError(msg) from e
|
||||
|
||||
|
||||
def setup_python(
|
||||
@@ -146,10 +232,13 @@ def setup_python(
|
||||
python_configuration: PythonConfiguration,
|
||||
constraints_path: Path | None,
|
||||
environment: ParsedEnvironment,
|
||||
user_pyodide_version: str | None,
|
||||
) -> dict[str, str]:
|
||||
base_python = get_base_python(python_configuration.identifier)
|
||||
log.step("Installing a base python environment...")
|
||||
base_python = get_base_python(tmp / "base", python_configuration)
|
||||
|
||||
log.step("Setting up build environment...")
|
||||
pyodide_version = user_pyodide_version or python_configuration.default_pyodide_version
|
||||
venv_path = tmp / "venv"
|
||||
env = virtualenv(python_configuration.version, base_python, venv_path, None, use_uv=False)
|
||||
venv_bin_path = venv_path / "bin"
|
||||
@@ -201,15 +290,28 @@ def setup_python(
|
||||
env=env,
|
||||
)
|
||||
|
||||
log.step(f"Installing Emscripten version: {python_configuration.emscripten_version} ...")
|
||||
emcc_path = install_emscripten(tmp, python_configuration.emscripten_version)
|
||||
pyodide_build_version = call(
|
||||
"python",
|
||||
"-c",
|
||||
"from importlib.metadata import version; print(version('pyodide-build'))",
|
||||
env=env,
|
||||
capture_stdout=True,
|
||||
).strip()
|
||||
|
||||
xbuildenv_info = get_xbuildenv_version_info(env, pyodide_version, pyodide_build_version)
|
||||
validate_pyodide_build_version(
|
||||
xbuildenv_info=xbuildenv_info,
|
||||
pyodide_build_version=pyodide_build_version,
|
||||
)
|
||||
|
||||
emscripten_version = xbuildenv_info["emscripten"]
|
||||
log.step(f"Installing Emscripten version: {emscripten_version} ...")
|
||||
emcc_path = install_emscripten(tmp, emscripten_version)
|
||||
|
||||
env["PATH"] = os.pathsep.join([str(emcc_path.parent), env["PATH"]])
|
||||
|
||||
log.step(f"Installing Pyodide xbuildenv version: {python_configuration.pyodide_version} ...")
|
||||
env["PYODIDE_ROOT"] = install_xbuildenv(
|
||||
env, python_configuration.pyodide_build_version, python_configuration.pyodide_version
|
||||
)
|
||||
log.step(f"Installing Pyodide xbuildenv version: {pyodide_version} ...")
|
||||
env["PYODIDE_ROOT"] = install_xbuildenv(env, pyodide_build_version, pyodide_version)
|
||||
|
||||
return env
|
||||
|
||||
@@ -259,6 +361,7 @@ def build(options: Options, tmp_path: Path) -> None:
|
||||
log.build_start(config.identifier)
|
||||
|
||||
identifier_tmp_dir = tmp_path / config.identifier
|
||||
|
||||
built_wheel_dir = identifier_tmp_dir / "built_wheel"
|
||||
repaired_wheel_dir = identifier_tmp_dir / "repaired_wheel"
|
||||
identifier_tmp_dir.mkdir()
|
||||
@@ -270,10 +373,11 @@ def build(options: Options, tmp_path: Path) -> None:
|
||||
)
|
||||
|
||||
env = setup_python(
|
||||
identifier_tmp_dir / "build",
|
||||
config,
|
||||
constraints_path,
|
||||
build_options.environment,
|
||||
tmp=identifier_tmp_dir / "build",
|
||||
python_configuration=config,
|
||||
constraints_path=constraints_path,
|
||||
environment=build_options.environment,
|
||||
user_pyodide_version=build_options.pyodide_version,
|
||||
)
|
||||
pip_version = get_pip_version(env)
|
||||
# The Pyodide command line runner mounts all directories in the host
|
||||
|
||||
Reference in New Issue
Block a user