Compare commits

..
20 Commits
Author SHA1 Message Date
Joe Rickerby bf3a5590c9 Bump version: v1.12.0 2021-06-22 18:13:50 +01:00
Henry Schreiner 8734e0e4e6 ci: pin dockcross for tests 2021-06-21 12:27:18 -04:00
Henry Schreiner 744812b7d3 ci: pin dockcross for tests 2021-06-21 12:27:18 -04:00
Henry Schreiner 548e0cefe3 feat: support building universal2/arm64 from x64_86 Python 3.8 (#704) 2021-06-21 12:27:18 -04:00
Joe Rickerby 6c62f9ee62 Bump version: v1.11.1.post1 2021-05-28 15:58:38 +01:00
Joe Rickerby 9a608b0b94 Fix mkdocs "Edit in Github" links 2021-05-28 15:53:26 +01:00
Joe Rickerby 5ea360607d Bump version: v1.11.1 2021-05-28 15:50:45 +01:00
Matthieu Darbois dc127cd69d Revert "Temporarily disable CircleCI mac builds (#685)" (#687)
This reverts commit af45aface9.
2021-05-28 14:24:46 +01:00
Joe Rickerby d1898152f7 Fix bump_version script 2021-05-28 14:13:04 +01:00
Joe Rickerby e4a917f717 Revert "Remove travis-ci macOS from docs/examples"
This reverts commit 5de574d36820c2481f4e7557ed6aa0e431ee3e63.
2021-05-24 17:24:36 -04:00
Henry Schreiner 6d5c80c624 ci: use native actions syntax (#662) 2021-05-24 17:24:36 -04:00
Henry SchreinerandJoe Rickerby 6ab9906ff0 chore: rename joerick/cibuildwheel -> pypa/cibuildwheel (#682)
* chore: rename joerick/cibuildwheel -> pypa/cibuildwheel

* Update Appveyor status badge markup

Note that it still has joerick/cibuildwheel - confusingly it's still within my appveyor account, but this is the new project that points to the pypa github repo

* Update Appveyor badge markup again

I went through a few passes to enable that project, here's the latest.

* Update Azure badge, also

* Update Travis badge to reference the .com

Co-authored-by: Joe Rickerby <joerick@mac.com>
2021-05-24 17:24:36 -04:00
Matthieu Darbois 5150122a95 Fix Travis CI tests on ppc64le/s390x (#689)
* Disable manylinux only tests on Travis CI ppc64le

Those tests are taking too much disk space for the Travis CI runner right now.
c.f. https://travis-ci.community/t/running-out-of-disk-space-quota-when-using-docker-on-ppc64le/11634

* Use focal to run aarch64/ppc64le/s390x jobs on Travis CI

This removes allow_failures from the Travis CI s390x job

* Use `focal` distribution for all Travis CI configurations

* Revert "Disable manylinux only tests on Travis CI ppc64le"

This reverts commit d0a09c689c058985c7dcb60b1e11a85a7bfce5e0.

* Skip test_manylinuxXXXX_only on Travis CI ppc64le

* Add a comment on why test_manylinuxXXXX_only test is skipped
2021-05-24 17:24:36 -04:00
Matthieu Darbois e5556f67f4 Remove macOS build on travis-ci (#686)
* Remove macOS build on travis-ci
* Remove travis-ci macOS from docs/examples
2021-05-24 17:24:36 -04:00
Edoardopre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>Henry Schreiner
f6fc96400b docs: update projects.yml (#673)
* Update projects.yml

Added EdoPro98/SimSiPM project

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* docs: add more projects

* docs: rerun projects.py

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Henry Schreiner <henryschreineriii@gmail.com>
2021-05-24 17:24:36 -04:00
Joe Rickerby 7e3bbe6e04 Temporarily disable CircleCI mac builds (#685) 2021-05-24 17:24:36 -04:00
Henry Schreiner 13f875a04a chore: move to PyPA (PSF) CoC (#679) 2021-05-18 13:51:04 -04:00
Henry SchreinerandMatthieu Darbois 339beb061c [1.x] chore: bump to final Python 3.5 pin (#674)
* chore: bump to final Python 3.5 pin

* Update pinned_docker_images.cfg

* Apply suggestions from code review

Co-authored-by: Matthieu Darbois <mayeut@users.noreply.github.com>

Co-authored-by: Matthieu Darbois <mayeut@users.noreply.github.com>
2021-05-14 16:41:12 -04:00
Henry Schreiner fee39c2f35 fix: missing f for fstrings in bin (#659) 2021-05-07 18:17:44 -04:00
Joe Rickerby 73adde0b9f Enable tests on 1.x branch 2021-05-05 08:51:06 +01:00
179 changed files with 27443 additions and 18302 deletions
+6 -31
View File
@@ -1,10 +1,9 @@
version: 2
jobs:
osx-python3.12:
osx-python3.6:
macos:
xcode: 15.4.0
resource_class: macos.m1.medium.gen1
xcode: 9.4.1
environment:
PYTHON: python3
steps:
@@ -18,14 +17,11 @@ jobs:
command: venv/bin/python ./bin/run_tests.py
no_output_timeout: 30m
linux-python3.12:
linux-python3.6:
docker:
- image: cimg/python:3.12
- image: circleci/python:3.6
environment:
PYTHON: python3
# Temporarily restrict the tests that are run on CircleCI to prevent
# test timeouts.
PYTEST_ADDOPTS: -k "unit_test or main_tests or test_0_basic or test_docker_images"
steps:
- checkout
- setup_remote_docker
@@ -38,30 +34,9 @@ jobs:
command: venv/bin/python ./bin/run_tests.py
no_output_timeout: 30m
linux-aarch64:
machine:
image: default
resource_class: arm.medium
environment:
PYTHON: python3
# Temporarily restrict the tests that are run on CircleCI to prevent
# test timeouts.
PYTEST_ADDOPTS: -k "unit_test or main_tests or test_0_basic or test_docker_images"
steps:
- checkout
- run:
name: Prepare the environment.
command: bash .circleci/prepare.sh
- run:
name: Test.
command: venv/bin/python ./bin/run_tests.py
no_output_timeout: 30m
workflows:
version: 2
all-tests:
jobs:
- osx-python3.12
- linux-python3.12
- linux-aarch64
- osx-python3.6
- linux-python3.6
+2 -9
View File
@@ -1,16 +1,9 @@
#!/usr/bin/env bash
set -o errexit
set -o xtrace
if [ "$(uname -s)" == "Darwin" ]; then
sudo softwareupdate --install-rosetta --agree-to-license
else
docker run --rm --privileged docker.io/tonistiigi/binfmt:latest --install all
fi
$PYTHON --version
$PYTHON -m venv venv
venv/bin/python -m pip install -U pip
$PYTHON -m pip --version
$PYTHON -m virtualenv -p $PYTHON venv
venv/bin/python -m pip install -e ".[dev]"
venv/bin/python -m pip freeze
venv/bin/python --version
-76
View File
@@ -1,76 +0,0 @@
only_if: changesInclude('.cirrus.yml') || ($CIRRUS_BRANCH == "main" && !changesIncludeOnly('docs/*', '.pre-commit-config.yaml')) || $CIRRUS_BRANCH =~ 'cirrus.*'
run_tests: &RUN_TESTS
install_cibuildwheel_script:
- python -m pip install -e ".[dev]" pytest-custom-exit-code
run_cibuildwheel_tests_script:
- python ./bin/run_tests.py
linux_x86_task:
timeout_in: 120m
compute_engine_instance:
image_project: cirrus-images
image: family/docker-builder
platform: linux
cpu: 8
memory: 8G
install_pre_requirements_script:
- docker run --rm --privileged docker.io/tonistiigi/binfmt:latest --install all
- apt install -y python3-venv python-is-python3
<<: *RUN_TESTS
linux_aarch64_task:
compute_engine_instance:
image_project: cirrus-images
image: family/docker-builder-arm64
architecture: arm64
platform: linux
cpu: 4
memory: 4G
install_pre_requirements_script:
- docker run --rm --privileged docker.io/tonistiigi/binfmt:latest --install all
- apt install -y python3-venv python-is-python3
<<: *RUN_TESTS
windows_x86_task:
# The task takes ~55 minutes while the timeout happens
# after 60 minutes by default, let's allow some wiggle room.
timeout_in: 120m
windows_container:
image: cirrusci/windowsservercore:visualstudio2022
cpu: 8
memory: 8G
install_pre_requirements_script:
- choco install -y --no-progress python3 --version 3.10.6
- refreshenv
- echo PATH=%PATH% >> "%CIRRUS_ENV%"
<<: *RUN_TESTS
macos_arm64_task:
macos_instance:
image: ghcr.io/cirruslabs/macos-sonoma-xcode
env:
PATH: /opt/homebrew/opt/python@3.10/libexec/bin:$PATH
install_pre_requirements_script:
- brew install python@3.10
<<: *RUN_TESTS
macos_arm64_cp38_task:
macos_instance:
image: ghcr.io/cirruslabs/macos-sonoma-xcode
env:
PATH: /opt/homebrew/opt/python@3.10/libexec/bin:$PATH
PYTEST_ADDOPTS: --run-cp38-universal2 -k 'test_cp38_arm64_testing_universal2_installer or test_arch_auto'
install_pre_requirements_script:
- brew install python@3.10
- curl -fsSLO https://www.python.org/ftp/python/3.8.10/python-3.8.10-macos11.pkg
- sudo installer -pkg python-3.8.10-macos11.pkg -target /
- rm python-3.8.10-macos11.pkg
- sh "/Applications/Python 3.8/Install Certificates.command"
<<: *RUN_TESTS
-6
View File
@@ -1,6 +0,0 @@
* text=auto
*.py diff=python
*.md diff=markdown
*.svg -diff
-31
View File
@@ -1,31 +0,0 @@
name: General issue
description: Open a general issue
body:
- type: markdown
attributes:
value: |
Hi! If you're not sure whether your problem is a bug in cibuildwheel or something else, please consider asking a question in the Discussions tab first.
- type: textarea
id: description
attributes:
label: Description
- type: input
id: log
attributes:
label: Build log
description: If relevant, please provide a link to a build log that shows the problem
placeholder: https://...
validations:
required: false
- type: input
id: ci-config
attributes:
label: CI config
description: Link to the CI config that produced the build log above. e.g. the Github Actions workflow file
placeholder: https://...
validations:
required: false
+7
View File
@@ -0,0 +1,7 @@
# config file for https://github.com/bobvanderlinden/probot-auto-merge
# no approvals required, just the label is enough
minApprovals: {}
requiredLabels:
- merge when green
-11
View File
@@ -1,11 +0,0 @@
version: 2
updates:
# Maintain dependencies for GitHub Actions
- package-ecosystem: "github-actions"
directory: "/"
schedule:
interval: "weekly"
groups:
actions:
patterns:
- "*"
-42
View File
@@ -1,42 +0,0 @@
name: Release
on:
workflow_dispatch:
pull_request:
release:
types:
- published
jobs:
dist:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: hynek/build-and-inspect-python-package@v2
publish:
needs: [dist]
runs-on: ubuntu-latest
if: github.event_name == 'release' && github.event.action == 'published'
environment:
name: pypi
url: https://pypi.org/p/cibuildwheel
permissions:
id-token: write
attestations: write
steps:
- uses: actions/download-artifact@v4
with:
name: Packages
path: dist
- name: Generate artifact attestation for sdist and wheel
uses: actions/attest-build-provenance@1c608d11d69870c2092266b3f9a6f3abbf17002c # v1.4.3
with:
subject-path: "dist/cibuildwheel-*"
- uses: pypa/gh-action-pypi-publish@release/v1
with:
attestations: true
+36 -155
View File
@@ -3,201 +3,82 @@ name: Test
on:
push:
branches:
- main
- master
- 1.x
pull_request:
paths-ignore:
- 'docs/**'
- .pre-commit-config.yaml
workflow_dispatch:
# allow manual runs on branches without a PR
concurrency:
group: test-${{ github.ref }}
cancel-in-progress: true
jobs:
lint:
name: Linters (mypy, flake8, etc.)
pre-commit:
name: Pre-commit checks (mypy, flake8, etc.)
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
id: python
with:
python-version: "3.x"
- uses: pre-commit/action@v3.0.1
- name: Check manifest
run: pipx run --python "${{ steps.python.outputs.python-path }}" nox -s check_manifest
- name: PyLint checks
run: pipx run --python "${{ steps.python.outputs.python-path }}" nox -s pylint -- --output-format=github
- uses: actions/checkout@v2
- uses: actions/setup-python@v2
- uses: pre-commit/action@v2.0.0
test:
name: Test on ${{ matrix.os }} (${{ matrix.python_version }})
needs: lint
name: Test cibuildwheel on ${{ matrix.os }}
needs: pre-commit
runs-on: ${{ matrix.os }}
strategy:
matrix:
os: [ubuntu-latest, windows-latest, macos-13, macos-14]
python_version: ['3.13']
include:
- os: ubuntu-latest
python_version: '3.8'
os: [ubuntu-latest, windows-latest, macos-latest]
python_version: ['3.7']
timeout-minutes: 180
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
- uses: actions/checkout@v2
- uses: actions/setup-python@v2
name: Install Python ${{ matrix.python_version }}
with:
python-version: ${{ matrix.python_version }}
allow-prereleases: true
- uses: astral-sh/setup-uv@v3
# free some space to prevent reaching GHA disk space limits
- name: Clean docker images
if: runner.os == 'Linux'
run: |
docker system prune -a -f
df -h
# for oci_container unit tests
- name: Set up QEMU
if: runner.os == 'Linux'
uses: docker/setup-qemu-action@v3
- name: Install dependencies
run: |
uv pip install --system ".[test]"
python -m pip install -e ".[dev]"
- name: Generate a sample project
- name: Sample build
if: "contains(github.event.pull_request.labels.*.name, 'CI: Sample build')"
run: |
python bin/sample_build.py
- name: Get some sample wheels
run: |
python -m test.test_projects test.test_0_basic.basic_project sample_proj
- name: Run a sample build (GitHub Action)
uses: ./
with:
package-dir: sample_proj
output-dir: wheelhouse
cibuildwheel --output-dir wheelhouse sample_proj
env:
CIBW_ARCHS_MACOS: x86_64 universal2 arm64
CIBW_BUILD_FRONTEND: 'build[uv]'
CIBW_FREE_THREADED_SUPPORT: 1
CIBW_PRERELEASE_PYTHONS: 1
- name: Run a sample build (GitHub Action, only)
uses: ./
- uses: actions/upload-artifact@v2
with:
package-dir: sample_proj
output-dir: wheelhouse_only
only: cp312-${{ runner.os == 'Linux' && 'manylinux_x86_64' || (runner.os == 'Windows' && 'win_amd64' || 'macosx_x86_64') }}
- name: Create custom configuration file
shell: bash
run: |
cat > sample_proj/cibw.toml <<EOF
[tool.cibuildwheel]
# Only build on CPython 3.12 on native arch
archs = ["native"]
build = "cp312-*"
# Skip musllinux
skip = "*-musllinux*"
EOF
- name: Run a sample build (GitHub Action, config-file)
uses: ./
with:
package-dir: sample_proj
output-dir: wheelhouse_config_file
config-file: sample_proj/cibw.toml
- name: Check Action artifacts
shell: bash
run: |
test $(find wheelhouse -name '*.whl' | wc -l) -ge 1
test $(find wheelhouse_only -name '*.whl' | wc -l) -eq 1
test $(find wheelhouse_config_file -name '*.whl' | wc -l) -eq 1
- uses: actions/upload-artifact@v4
with:
name: cibw-wheels-${{ matrix.os }}-${{ strategy.job-index }}
path: wheelhouse/*.whl
name: sample_wheels
path: wheelhouse
- name: Test cibuildwheel
run: |
python ./bin/run_tests.py --run-podman
emulated-archs:
name: Get qemu emulated architectures
needs: lint
runs-on: ubuntu-latest
outputs:
archs: ${{ steps.archs.outputs.archs }}
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.x"
- name: Install dependencies
run: python -m pip install ".[test]"
- name: Get qemu emulated architectures
id: archs
run: |
OUTPUT=$(python -c "from json import dumps; from test.utils import EMULATED_ARCHS; print(dumps(EMULATED_ARCHS))")
echo "${OUTPUT}"
echo "archs=${OUTPUT}" >> "$GITHUB_OUTPUT"
python ./bin/run_tests.py
test-emulated:
name: Test Linux ${{ matrix.arch }} using qemu
needs: emulated-archs
name: Test emulated cibuildwheel using qemu
needs: pre-commit
runs-on: ubuntu-latest
timeout-minutes: 180
strategy:
matrix:
arch: ${{ fromJSON(needs.emulated-archs.outputs.archs) }}
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.x"
- uses: actions/checkout@v2
- uses: actions/setup-python@v2
- name: Install dependencies
run: python -m pip install ".[test,uv]"
run: |
python -m pip install -r requirements-dev.txt
- name: Set up QEMU
uses: docker/setup-qemu-action@v3
id: qemu
uses: docker/setup-qemu-action@v1
with:
platforms: all
- name: Run the emulation tests
run: pytest --run-emulation ${{ matrix.arch }} test/test_emulation.py
test-pyodide:
name: Test cibuildwheel building pyodide wheels
needs: lint
runs-on: ubuntu-24.04
timeout-minutes: 180
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
name: Install Python 3.12
with:
python-version: '3.12'
- name: Install dependencies
run: |
python -m pip install ".[test]"
- name: Generate a sample project
run: |
python -m test.test_projects test.test_0_basic.basic_project sample_proj
- name: Run a sample build (GitHub Action)
uses: ./
with:
package-dir: sample_proj
output-dir: wheelhouse
env:
CIBW_PLATFORM: pyodide
- name: Run tests with 'CIBW_PLATFORM' set to 'pyodide'
run: |
python ./bin/run_tests.py
env:
CIBW_PLATFORM: pyodide
pytest --run-emulation test/test_emulation.py
+11 -34
View File
@@ -1,15 +1,6 @@
name: Update dependencies
on:
pull_request:
paths:
- '.github/workflows/update-dependencies.yml'
- 'bin/update_pythons.py'
- 'bin/update_docker.py'
- 'bin/update_virtualenv.py'
- 'bin/projects.py'
- 'docs/data/projects.yml'
- 'noxfile.py'
workflow_dispatch:
schedule:
- cron: '0 6 * * 1' # "At 06:00 on Monday."
@@ -17,35 +8,23 @@ on:
jobs:
update-dependencies:
name: Update dependencies
if: github.repository_owner == 'pypa' || github.event_name != 'schedule'
runs-on: ubuntu-latest
steps:
# we use this step to grab a Github App auth token, so that PRs generated by this workflow
# run the GHA tests.
- uses: actions/create-github-app-token@v1
id: generate-token
if: github.ref == 'refs/heads/main' && github.repository == 'pypa/cibuildwheel'
- uses: actions/checkout@v2
- uses: actions/setup-python@v2
name: Install Python 3.9
with:
app_id: ${{ secrets.CIBUILDWHEEL_BOT_APP_ID }}
private_key: ${{ secrets.CIBUILDWHEEL_BOT_APP_PRIVATE_KEY }}
- uses: actions/checkout@v4
- uses: wntrblm/nox@2024.04.15
with:
python-versions: "3.7, 3.8, 3.9, 3.10, 3.11, 3.12, 3.13"
python-version: 3.9
architecture: x64
- name: Install dependencies
run: python -m pip install ".[dev]"
- name: "Run update: dependencies"
run: nox --force-color -s update_constraints
run: python ./bin/update_dependencies.py
- name: "Run update: python configs"
run: nox --force-color -s update_pins
- name: "Run update: docs user projects"
run: nox --force-color -s update_proj -- --auth=${{ secrets.GITHUB_TOKEN }}
run: python ./bin/update_pythons.py --force
- name: Create Pull Request
if: github.ref == 'refs/heads/main' && github.repository == 'pypa/cibuildwheel'
uses: peter-evans/create-pull-request@v7
if: github.ref == 'refs/heads/master' && github.repository == 'pypa/cibuildwheel'
uses: peter-evans/create-pull-request@v3
with:
commit-message: Update dependencies
title: '[Bot] Update dependencies'
@@ -54,6 +33,4 @@ jobs:
PR generated by "Update dependencies" [workflow](https://github.com/${{github.repository}}/actions/runs/${{github.run_id}}).
branch: update-dependencies-pr
sign-commits: true
token: ${{ steps.generate-token.outputs.token }}
delete-branch: true
@@ -1,30 +0,0 @@
name: Update the vX.Y tag
on:
release:
types: [released]
workflow_dispatch:
inputs:
TAG_NAME:
description: 'Tag name that the major.minor tag will point to'
required: true
env:
TAG_NAME: ${{ github.event.inputs.TAG_NAME || github.event.release.tag_name }}
jobs:
update_tag:
name: Update the major.minor tag to include the ${{ github.event.inputs.TAG_NAME || github.event.release.tag_name }} changes
environment:
name: releaseNewActionVersion
runs-on: ubuntu-latest
permissions:
contents: write
steps:
- uses: actions/checkout@v4
- name: Update the ${{ env.TAG_NAME }} tag
id: update-major-minor-tag
uses: joerick/update-vX.Y-tag-action@v1.0
with:
source-tag: ${{ env.TAG_NAME }}
-3
View File
@@ -114,6 +114,3 @@ site/
# Virtual environments
venv*
# PyCharm
.idea/
+2 -29
View File
@@ -1,5 +1,5 @@
linux:
image: python:3.12
image: python:3.8
services:
- name: docker:dind
entrypoint: ["env", "-u", "DOCKER_HOST"]
@@ -9,34 +9,7 @@ linux:
DOCKER_DRIVER: overlay2
# See https://github.com/docker-library/docker/pull/166
DOCKER_TLS_CERTDIR: ""
# skip all but the basic tests
# (comment the below line in a PR to debug a Gitlab-specific issue)
PYTEST_ADDOPTS: -k "unit_test or test_0_basic" --suppress-no-test-exit-code
script:
- curl -sSL https://get.docker.com/ | sh
- docker run --rm --privileged docker.io/tonistiigi/binfmt:latest --install all
- python -m pip install -e ".[dev]" pytest-custom-exit-code
- python -m pip install -e ".[dev]"
- python ./bin/run_tests.py
windows:
image: mcr.microsoft.com/windows/servercore:1809
variables:
PYTEST_ADDOPTS: -k "unit_test or test_0_basic" --suppress-no-test-exit-code
before_script:
- choco install python -y --version 3.12.4
script:
- py -m pip install -e ".[dev]" pytest-custom-exit-code
- py bin\run_tests.py
tags:
- saas-windows-medium-amd64
macos:
image: macos-14-xcode-15
variables:
PYTEST_ADDOPTS: -k "unit_test or test_0_basic" --suppress-no-test-exit-code
script:
- python3 -m pip install -e ".[dev]" pytest-custom-exit-code
- python3 ./bin/run_tests.py
tags:
- saas-macos-medium-m1
+41 -63
View File
@@ -1,7 +1,7 @@
repos:
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v5.0.0
rev: v3.4.0
hooks:
- id: check-case-conflict
- id: check-merge-conflict
@@ -13,81 +13,59 @@ repos:
- id: mixed-line-ending
- id: trailing-whitespace
- repo: https://github.com/astral-sh/ruff-pre-commit
rev: v0.6.9
- repo: https://github.com/PyCQA/isort
rev: 5.8.0
hooks:
- id: ruff
args: ["--fix", "--show-fixes"]
- id: ruff-format
- id: isort
- repo: https://github.com/psf/black
rev: 21.4b2
hooks:
- id: black
- repo: https://github.com/asottile/setup-cfg-fmt
rev: v1.17.0
hooks:
- id: setup-cfg-fmt
stages: [manual]
- repo: https://github.com/pre-commit/mirrors-mypy
rev: v1.11.2
rev: v0.812
hooks:
- id: mypy
name: mypy 3.8 on cibuildwheel/
exclude: ^cibuildwheel/resources/.*py|bin/generate_schema.py$
args: ["--python-version=3.8"]
additional_dependencies: &mypy-dependencies
- bracex
- nox
- packaging
- pygithub
- rich
- tomli
- tomli_w
- types-certifi
- types-click
- types-jinja2
- types-pyyaml
- types-requests
- uv
- validate-pyproject
exclude: ^(bin/|cibuildwheel/resources/).*py$
additional_dependencies: [packaging]
args: ["--python-version=3.6", "--ignore-missing-imports", "--scripts-are-modules"]
- id: mypy
name: mypy 3.12
exclude: ^cibuildwheel/resources/.*py$
args: ["--python-version=3.12"]
additional_dependencies: *mypy-dependencies
name: mypy 3.7+ on bin/
files: ^bin/.*py$
additional_dependencies: [packaging]
args: ["--python-version=3.7", "--ignore-missing-imports", "--scripts-are-modules"]
- repo: https://github.com/shellcheck-py/shellcheck-py
rev: v0.10.0.1
- repo: https://github.com/asottile/pyupgrade
rev: v2.14.0
hooks:
- id: shellcheck
- id: pyupgrade
name: PyUpgrade 3.6+
args: ["--py36-plus"]
exclude: ^bin/
- id: pyupgrade
name: PyUpgrade 3.7+ on bin
exclude: ^(cibuildwheel|unit_test|test)/
args: ["--py37-plus"]
- repo: https://github.com/PyCQA/flake8
rev: 3.9.1
hooks:
- id: flake8
exclude: cibuildwheel/resources/
additional_dependencies: [flake8-bugbear]
- repo: local
hooks:
- id: disallow-caps
name: Disallow improper capitalization
language: pygrep
entry: PyBind|Numpy|Cmake|Github|PyTest
entry: PyBind|Numpy|Cmake|Github
types:
- markdown
exclude: ^docs/working-examples\.md$ # Autogenerated
- id: update-readme-changelog
name: Update README changelog
language: python
entry: bin/update_readme_changelog.py
files: ^docs/changelog.md$
- repo: https://github.com/codespell-project/codespell
rev: v2.3.0
hooks:
- id: codespell
args: ["-L", "sur,assertin", "-w"]
exclude: ^docs/working-examples\.md$ # Autogenerated
- repo: https://github.com/python-jsonschema/check-jsonschema
rev: 0.29.3
hooks:
- id: check-dependabot
- id: check-github-actions
- id: check-github-workflows
- id: check-gitlab-ci
- id: check-readthedocs
- id: check-travis
- id: check-jsonschema
name: Check projects
args: [--schemafile, docs/data/projects.schema.json]
files: '^docs/data/projects.yml$'
- id: check-metaschema
files: '^docs/data/projects.schema.json$'
+10 -11
View File
@@ -2,14 +2,13 @@
# See https://docs.readthedocs.io/en/stable/config-file/v2.html for details
version: 2
build:
os: ubuntu-22.04
tools:
python: "3.12"
commands:
- asdf plugin add uv
- asdf install uv latest
- asdf global uv latest
- uv venv
- uv pip install -e.[docs]
- NO_COLOR=1 .venv/bin/mkdocs build --strict --site-dir $READTHEDOCS_OUTPUT/html
mkdocs:
configuration: mkdocs.yml
python:
version: 3.8
install:
- method: pip
path: .
extra_requirements:
- docs
+17 -39
View File
@@ -4,69 +4,47 @@ language: python
branches:
only:
- main
- master
jobs:
include:
- name: Linux | x86_64 + i686 | Python 3.9
python: 3.9
- name: Linux | x86_64 + i686 | Python 3.6
python: 3.6
services: docker
env: PYTHON=python
- name: Linux | arm64 | Python 3.9
python: 3.9
- name: Linux | arm64 | Python 3.6
python: 3.6
services: docker
arch: arm64-graviton2
group: edge
virt: vm
arch: arm64
env: PYTHON=python
# docker is outdated in the arm64-graviton2 vm focal image (19.x)
# we need to upgrade to get >= 24.0
addons:
apt:
sources:
- sourceline: 'deb https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable'
packages:
- docker-ce docker-ce-cli containerd.io
- name: Linux | ppc64le | Python 3.9
python: 3.9
- name: Linux | ppc64le | Python 3.6
python: 3.6
services: docker
arch: ppc64le
allow_failure: True
env:
- PYTHON=python
# skip test_manylinuxXXXX_only, it uses too much disk space
# c.f. https://travis-ci.community/t/running-out-of-disk-space-quota-when-using-docker-on-ppc64le/11634
- PYTEST_ADDOPTS='-k "not test_manylinuxXXXX_only"'
- name: Windows | x86_64 | Python 3.9
- name: Windows | x86_64 | Python 3.6
os: windows
language: shell
before_install:
- choco upgrade python3 -y --version 3.9.13 --limit-output --params "/InstallDir:C:\\Python39"
- choco upgrade python3 -y --version 3.6.8 --limit-output
# Update root certificates to fix SSL error; see http://www.chawn.com/RootCerts.htm
- powershell "md C:\temp\certs; CertUtil -generateSSTFromWU C:\temp\certs\RootStore.sst; Get-ChildItem -Path C:\\temp\certs\Rootstore.sst | Import-Certificate -CertStoreLocation Cert:\\LocalMachine\\Root\\ | out-null"
env:
- PYTHON=C:\\Python39\\python
- PYTHON=C:\\Python36\\python
- name: Linux | s390x | Python 3.9
python: 3.9
- name: Linux | s390x | Python 3.6
python: 3.6
services: docker
arch: s390x
allow_failure: True
env: PYTHON=python
install:
- if [ "${TRAVIS_OS_NAME}" == "linux" ]; then docker run --rm --privileged docker.io/tonistiigi/binfmt:latest --install all; fi
- $PYTHON -m pip install -U pip
- $PYTHON -m pip install -e ".[test]" pytest-custom-exit-code
install: $PYTHON -m pip install -e ".[dev]" pytest-custom-exit-code
script: |
# travis_wait disable the output while waiting
# use the same kind of tricks as in multibuild
(while true; do echo "travis_keep_alive"; sleep 300; done) &
SPINNER_PID=$!
disown
result=0
$PYTHON ./bin/run_tests.py --num-processes 2 || result=1
kill -9 ${SPINNER_PID}
test ${result} -eq 0
script: $PYTHON ./bin/run_tests.py
+9 -8
View File
@@ -1,11 +1,12 @@
This is a summary of the host Python versions and platforms covered by the different CI platforms:
This is a summary of the Python versions and platforms covered by the different CI platforms:
| | 3.8 | 3.9 | 3.10 | 3.11 | 3.12 |
|---------|----------------------------------|-----------|-----------|---------|--------------------------------------------------|
| Linux | Azure Pipelines / GitHub Actions | Travis CI | Cirrus CI | | AppVeyor¹ / CircleCI¹ / GitHub Actions / GitLab¹ |
| macOS | Azure Pipelines | | Cirrus CI | GitLab¹ | AppVeyor¹ /CircleCI¹ / GitHub Actions |
| Windows | Azure Pipelines | Travis CI | Cirrus CI | | AppVeyor¹ / GitHub Actions / GitLab¹ |
| | 3.6 | 3.7 | 3.8 |
|----------|------------------------------|------------------------------------------|------------------|
| Linux | Travis CI / CircleCI | AppVeyor² / GitHub Actions | Azure Pipelines |
| macOS | CircleCI | AppVeyor² / Travis CI¹ / GitHub Actions | Azure Pipelines |
| Windows | Travis CI / Azure Pipelines | AppVeyor² / GitHub Actions | Azure Pipelines |
> ¹ Runs a reduced set of tests to reduce CI load
> ¹ Python version not really pinned, but dependent on the (default) version of image used.
> ² AppVeyor only runs the "basic" test to reduce load.
Non-x86 architectures are covered on Travis CI using Python 3.9.
Non-x86 architectures are covered on Travis CI using Python 3.6.
+5 -1
View File
@@ -1,6 +1,6 @@
This project is licensed under the 'BSD 2-clause license'.
Copyright (c) 2017-2023, Joe Rickerby and contributors. All rights reserved.
Copyright (c) 2017-2019, Joe Rickerby and contributors. All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:
@@ -22,3 +22,7 @@ SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
Get-pip and Pip (bundled in cibuildwheel/resources) are licensed under the MIT
license. See https://github.com/pypa/get-pip/blob/master/LICENSE.txt
+757 -140
View File
File diff suppressed because it is too large Load Diff
+7 -71
View File
@@ -9,14 +9,6 @@ inputs:
description: 'Folder to place the outputs in, defaults to "wheelhouse"'
required: false
default: wheelhouse
config-file:
description: 'File containing the config, defaults to {package}/pyproject.toml'
required: false
default: ''
only:
description: 'Build a specific wheel only. No need for arch/platform if this is set'
required: false
default: ''
branding:
icon: package
color: yellow
@@ -24,69 +16,13 @@ branding:
runs:
using: composite
steps:
# Set up the version of Python that supports pyodide
- uses: actions/setup-python@v5
id: python
with:
python-version: "3.12"
update-environment: false
- id: cibw
run: |
# Install cibuildwheel
"${{ steps.python.outputs.python-path }}" -u << "EOF"
import os
import shutil
import sys
import venv
from pathlib import Path
from subprocess import run
class EnvBuilder(venv.EnvBuilder):
def __init__(self):
super().__init__()
def setup_scripts(self, context):
pass
def post_setup(self, context):
super().post_setup(context)
self.bin_path = Path(context.env_exe).parent
run([sys.executable, "-m", "pip", "--python", context.env_exe, "install", r"${{ github.action_path }}"], check=True)
print("::group::Install cibuildwheel")
venv_path = Path(r"${{ runner.temp }}") / "cibw"
if venv_path.exists():
shutil.rmtree(venv_path)
builder = EnvBuilder()
builder.create(venv_path)
cibw_path = [path for path in builder.bin_path.glob("cibuildwheel*") if path.stem == "cibuildwheel"][0]
with open(os.environ["GITHUB_OUTPUT"], "at") as f:
f.write(f"cibw-path={cibw_path}\n")
print("::endgroup::")
EOF
shell: bash
# Redirecting stderr to stdout to fix interleaving issue in Actions.
# This needs powershell-core because github.action path is a Windows-style
# path on Windows. Powershell-core understands both types of paths.
- run: >
"${{ steps.cibw.outputs.cibw-path }}"
"${{ inputs.package-dir }}"
${{ inputs.output-dir != '' && format('--output-dir "{0}"', inputs.output-dir) || ''}}
${{ inputs.config-file != '' && format('--config-file "{0}"', inputs.config-file) || ''}}
${{ inputs.only != '' && format('--only "{0}"', inputs.only) || ''}}
2>&1
shell: bash
if: runner.os != 'Windows'
# Windows needs powershell to interact nicely with Meson
- run: >
& "${{ steps.cibw.outputs.cibw-path }}"
"${{ inputs.package-dir }}"
${{ inputs.output-dir != '' && format('--output-dir "{0}"', inputs.output-dir) || ''}}
${{ inputs.config-file != '' && format('--config-file "{0}"', inputs.config-file) || ''}}
${{ inputs.only != '' && format('--only "{0}"', inputs.only) || ''}}
pipx run
--spec ${{ github.action_path }}
cibuildwheel
${{ inputs.package-dir }}
--output-dir ${{ inputs.output-dir }}
shell: pwsh
if: runner.os == 'Windows'
+12 -16
View File
@@ -1,29 +1,25 @@
environment:
matrix:
- APPVEYOR_BUILD_WORKER_IMAGE: Ubuntu2204
APPVEYOR_JOB_NAME: "python312-x64-ubuntu"
- APPVEYOR_BUILD_WORKER_IMAGE: Visual Studio 2022
APPVEYOR_JOB_NAME: "python312-x64-vs2022"
- APPVEYOR_BUILD_WORKER_IMAGE: macos-sonoma
APPVEYOR_JOB_NAME: "python312-x64-macos"
- APPVEYOR_BUILD_WORKER_IMAGE: Ubuntu
APPVEYOR_JOB_NAME: "python37-x64-ubuntu"
- APPVEYOR_BUILD_WORKER_IMAGE: Visual Studio 2015
APPVEYOR_JOB_NAME: "python37-x64-vs2015"
- APPVEYOR_BUILD_WORKER_IMAGE: macos-mojave
APPVEYOR_JOB_NAME: "python37-x64-macos-mojave"
stack: python 3.12
stack: python 3.7
build: off
init:
- cmd: set PATH=C:\Python37;C:\Python37\Scripts;%PATH%
- ps: |
$BRANCH = if ($env:APPVEYOR_PULL_REQUEST_HEAD_REPO_BRANCH) { $env:APPVEYOR_PULL_REQUEST_HEAD_REPO_BRANCH } else { $env:APPVEYOR_REPO_BRANCH }
if (-not ($BRANCH -eq 'main' -or $BRANCH.ToLower().StartsWith('appveyor-'))) {
$env:PYTEST_ADDOPTS = '-k "unit_test or test_0_basic" --suppress-no-test-exit-code'
}
if ($IsLinux) {
docker run --rm --privileged docker.io/tonistiigi/binfmt:latest --install all
if (-not ($BRANCH -eq 'master' -or $BRANCH.ToLower().StartsWith('appveyor-'))) {
$env:PYTEST_ADDOPTS='-k "unit_test or test_0_basic" --suppress-no-test-exit-code'
}
install:
- python -m pip install -U pip
- python -m pip install -e ".[dev]" pytest-custom-exit-code
install: python -m pip install -e ".[dev]" pytest-custom-exit-code
# the '-u' flag is required so the output is in the correct order.
# See https://github.com/pypa/cibuildwheel/pull/24 for more info.
@@ -31,7 +27,7 @@ test_script: python -u ./bin/run_tests.py
branches:
only:
- main
- master
skip_commits:
files:
+18 -15
View File
@@ -1,39 +1,42 @@
pr:
paths:
exclude:
- docs/*
- .pre-commit-config.yaml
jobs:
- job: linux_38
timeoutInMinutes: 120
pool: {vmImage: 'Ubuntu-20.04'}
pool: {vmImage: 'Ubuntu-18.04'}
steps:
- task: UsePythonVersion@0
inputs:
versionSpec: '3.8'
- bash: |
docker run --rm --privileged docker.io/tonistiigi/binfmt:latest --install all
python -m pip install -e ".[dev]"
python -m pip install -e ".[dev]" pytest-azurepipelines
python ./bin/run_tests.py
- job: macos_38
pool: {vmImage: 'macOS-12'}
pool: {vmImage: 'macOS-10.15'}
steps:
- task: UsePythonVersion@0
inputs:
versionSpec: '3.8'
- bash: |
python -m pip install -e ".[dev]"
python ./bin/run_tests.py --num-processes 2
python -m pip install -e ".[dev]" pytest-azurepipelines
python ./bin/run_tests.py
- job: windows_36
pool: {vmImage: 'vs2017-win2016'}
timeoutInMinutes: 180
steps:
- task: UsePythonVersion@0
inputs:
versionSpec: '3.6'
- bash: |
python -m pip install -e ".[dev]" pytest-azurepipelines
python ./bin/run_tests.py
- job: windows_38
pool: {vmImage: 'windows-2019'}
pool: {vmImage: 'vs2017-win2016'}
timeoutInMinutes: 180
steps:
- task: UsePythonVersion@0
inputs:
versionSpec: '3.8'
- bash: |
python -m pip install -e ".[dev]"
python -m pip install -e ".[dev]" pytest-azurepipelines
python ./bin/run_tests.py
+5 -44
View File
@@ -1,9 +1,5 @@
#!/usr/bin/env python3
# /// script
# dependencies = ["click", "packaging", "tomli; python_version<'3.11'"]
# ///
from __future__ import annotations
@@ -17,14 +13,10 @@ from pathlib import Path
import click
from packaging.version import InvalidVersion, Version
if sys.version_info < (3, 11):
import tomli as tomllib
else:
import tomllib
import cibuildwheel
config = [
# file path, version find/replace format
("pyproject.toml", 'version = "{}"'),
("README.md", "cibuildwheel=={}"),
("cibuildwheel/__init__.py", '__version__ = "{}"'),
("docs/faq.md", "cibuildwheel=={}"),
@@ -32,6 +24,7 @@ config = [
("docs/setup.md", "cibuildwheel=={}"),
("examples/*", "cibuildwheel=={}"),
("examples/*", "cibuildwheel@v{}"),
("setup.cfg", "version = {}"),
]
RED = "\u001b[31m"
@@ -41,8 +34,7 @@ OFF = "\u001b[0m"
@click.command()
def bump_version() -> None:
with open("pyproject.toml", "rb") as f:
current_version = tomllib.load(f)["project"]["version"]
current_version = cibuildwheel.__version__
try:
commit_date_str = subprocess.run(
@@ -68,7 +60,7 @@ def bump_version() -> None:
print(e)
print("Failed to get previous version tag information.")
git_changes_result = subprocess.run(["git diff-index --quiet HEAD --"], shell=True, check=False)
git_changes_result = subprocess.run(["git diff-index --quiet HEAD --"], shell=True)
repo_has_uncommitted_changes = git_changes_result.returncode != 0
if repo_has_uncommitted_changes:
@@ -76,7 +68,7 @@ def bump_version() -> None:
sys.exit(1)
# fmt: off
print( 'Current version:', current_version)
print( 'Current version:', current_version) # noqa
new_version = input(' New version: ').strip()
# fmt: on
@@ -144,26 +136,6 @@ def bump_version() -> None:
while input('Type "done" to continue: ').strip().lower() != "done":
pass
# run pre-commit to update the README changelog
subprocess.run(
[
"pre-commit",
"run",
"--files=docs/changelog.md",
],
check=False,
)
# run pre-commit to check that no errors occurred on the second run
subprocess.run(
[
"pre-commit",
"run",
"--files=docs/changelog.md",
],
check=True,
)
subprocess.run(
[
"git",
@@ -186,17 +158,6 @@ def bump_version() -> None:
)
print("Done.")
print()
print("Push the new version to GitHub with:")
print(f" git push && git push origin v{new_version}")
print()
release_url = "https://github.com/pypa/cibuildwheel/releases/new?" + urllib.parse.urlencode(
{"tag": f"v{new_version}"}
)
print("Then create a release at the URL:")
print(f" {release_url}")
if __name__ == "__main__":
+3
View File
@@ -0,0 +1,3 @@
#!/bin/bash
CIBW_PLATFORM=linux pytest "$@"
-312
View File
@@ -1,312 +0,0 @@
#!/usr/bin/env python
# /// script
# dependencies = ["pyyaml"]
# ///
import argparse
import copy
import json
from typing import Any
import yaml
parser = argparse.ArgumentParser()
parser.add_argument("--schemastore", action="store_true", help="Generate schema_store version")
args = parser.parse_args()
starter = """
$schema: http://json-schema.org/draft-07/schema
$id: https://github.com/pypa/cibuildwheel/blob/main/cibuildwheel/resources/cibuildwheel.schema.json
$defs:
inherit:
enum:
- none
- prepend
- append
default: none
description: How to inherit the parent's value.
additionalProperties: false
description: cibuildwheel's settings.
type: object
properties:
archs:
description: Change the architectures built on your machine by default.
type: string_array
before-all:
description: Execute a shell command on the build system before any wheels are built.
type: string_array
before-build:
description: Execute a shell command preparing each wheel's build.
type: string_array
before-test:
description: Execute a shell command before testing each wheel.
type: string_array
build:
default: ['*']
description: Choose the Python versions to build.
type: string_array
build-frontend:
default: default
description: Set the tool to use to build, either "pip" (default for now), "build", or "build[uv]"
oneOf:
- enum: [pip, build, "build[uv]", default]
- type: string
pattern: '^pip; ?args:'
- type: string
pattern: '^build; ?args:'
- type: string
pattern: '^build\\[uv\\]; ?args:'
- type: object
additionalProperties: false
required: [name]
properties:
name:
enum: [pip, build, "build[uv]"]
args:
type: array
items:
type: string
build-verbosity:
type: integer
minimum: -3
maximum: 3
default: 0
description: Increase/decrease the output of pip wheel.
config-settings:
description: Specify config-settings for the build backend.
type: string_table_array
container-engine:
oneOf:
- enum: [docker, podman]
- type: string
pattern: '^docker; ?(create_args|disable_host_mount):'
- type: string
pattern: '^podman; ?(create_args|disable_host_mount):'
- type: object
additionalProperties: false
required: [name]
properties:
name:
enum: [docker, podman]
create-args:
type: array
items:
type: string
disable-host-mount:
type: boolean
dependency-versions:
default: pinned
description: Specify how cibuildwheel controls the versions of the tools it uses
type: string
environment:
description: Set environment variables needed during the build.
type: string_table
environment-pass:
description: Set environment variables on the host to pass-through to the container
during the build.
type: string_array
free-threaded-support:
type: boolean
default: false
description: The project supports free-threaded builds of Python (PEP703)
manylinux-aarch64-image:
type: string
description: Specify alternative manylinux / musllinux container images
manylinux-i686-image:
type: string
description: Specify alternative manylinux / musllinux container images
manylinux-ppc64le-image:
type: string
description: Specify alternative manylinux / musllinux container images
manylinux-pypy_aarch64-image:
type: string
description: Specify alternative manylinux / musllinux container images
manylinux-pypy_i686-image:
type: string
description: Specify alternative manylinux / musllinux container images
manylinux-pypy_x86_64-image:
type: string
description: Specify alternative manylinux / musllinux container images
manylinux-s390x-image:
type: string
description: Specify alternative manylinux / musllinux container images
manylinux-x86_64-image:
type: string
description: Specify alternative manylinux / musllinux container images
musllinux-aarch64-image:
type: string
description: Specify alternative manylinux / musllinux container images
musllinux-armv7l-image:
type: string
description: Specify alternative manylinux / musllinux container images
musllinux-i686-image:
type: string
description: Specify alternative manylinux / musllinux container images
musllinux-ppc64le-image:
type: string
description: Specify alternative manylinux / musllinux container images
musllinux-s390x-image:
type: string
description: Specify alternative manylinux / musllinux container images
musllinux-x86_64-image:
type: string
description: Specify alternative manylinux / musllinux container images
repair-wheel-command:
type: string_array
description: Execute a shell command to repair each built wheel.
skip:
description: Choose the Python versions to skip.
type: string_array
test-command:
description: Execute a shell command to test each built wheel.
type: string_array
test-extras:
description: Install your wheel for testing using `extras_require`
type: string_array
test-requires:
description: Install Python dependencies before running the tests
type: string_array
test-skip:
description: Skip running tests on some builds.
type: string_array
"""
schema = yaml.safe_load(starter)
string_array = yaml.safe_load(
"""
- type: string
- type: array
items:
type: string
"""
)
string_table_array = yaml.safe_load(
"""
- type: string
- type: object
additionalProperties: false
patternProperties:
.+:
oneOf:
- type: string
- type: array
items:
type: string
"""
)
string_table = yaml.safe_load(
"""
- type: string
- type: object
additionalProperties: false
patternProperties:
.+:
type: string
"""
)
for value in schema["properties"].values():
match value:
case {"type": "string_array"}:
del value["type"]
value["oneOf"] = string_array
case {"type": "string_table"}:
del value["type"]
value["oneOf"] = string_table
case {"type": "string_table_array"}:
del value["type"]
value["oneOf"] = string_table_array
overrides = yaml.safe_load(
"""
type: array
description: An overrides array
items:
type: object
required: ["select"]
minProperties: 2
additionalProperties: false
properties:
select: {}
inherit:
type: object
additionalProperties: false
properties:
before-all: {"$ref": "#/$defs/inherit"}
before-build: {"$ref": "#/$defs/inherit"}
before-test: {"$ref": "#/$defs/inherit"}
config-settings: {"$ref": "#/$defs/inherit"}
container-engine: {"$ref": "#/$defs/inherit"}
environment: {"$ref": "#/$defs/inherit"}
environment-pass: {"$ref": "#/$defs/inherit"}
repair-wheel-command: {"$ref": "#/$defs/inherit"}
test-command: {"$ref": "#/$defs/inherit"}
test-extras: {"$ref": "#/$defs/inherit"}
test-requires: {"$ref": "#/$defs/inherit"}
"""
)
for key, value in schema["properties"].items():
value["title"] = f'CIBW_{key.replace("-", "_").upper()}'
non_global_options = {k: {"$ref": f"#/properties/{k}"} for k in schema["properties"]}
del non_global_options["build"]
del non_global_options["skip"]
del non_global_options["test-skip"]
del non_global_options["free-threaded-support"]
overrides["items"]["properties"]["select"]["oneOf"] = string_array
overrides["items"]["properties"] |= non_global_options.copy()
del overrides["items"]["properties"]["archs"]
not_linux = non_global_options.copy()
del not_linux["environment-pass"]
del not_linux["container-engine"]
for key in list(not_linux):
if "linux-" in key:
del not_linux[key]
def as_object(d: dict[str, Any]) -> dict[str, Any]:
return {
"type": "object",
"additionalProperties": False,
"properties": copy.deepcopy(d),
}
oses = {
"linux": as_object(non_global_options),
"windows": as_object(not_linux),
"macos": as_object(not_linux),
"pyodide": as_object(not_linux),
}
oses["linux"]["properties"]["repair-wheel-command"] = {
**schema["properties"]["repair-wheel-command"],
"default": "auditwheel repair -w {dest_dir} {wheel}",
}
oses["macos"]["properties"]["repair-wheel-command"] = {
**schema["properties"]["repair-wheel-command"],
"default": "delocate-wheel --require-archs {delocate_archs} -w {dest_dir} -v {wheel}",
}
del oses["linux"]["properties"]["dependency-versions"]
schema["properties"]["overrides"] = overrides
schema["properties"] |= oses
if args.schemastore:
schema["$id"] = "https://json.schemastore.org/partial-cibuildwheel.json"
schema["$schema"] = "http://json-schema.org/draft-07/schema#"
schema["description"] = (
"cibuildwheel's toml file, generated with ./bin/generate_schema.py --schemastore from cibuildwheel."
)
print(json.dumps(schema, indent=2))
+17 -34
View File
@@ -1,25 +1,13 @@
#!/usr/bin/env python3
"""
Check known projects for usage of requires-python.
Usage:
./bin/inspect_all_known_projects.py --online=$GITHUB_TOKEN
This will cache the results to all_known_setup.yaml; you can reprint
the results without the `--online` setting.
"""
from __future__ import annotations
import ast
from collections.abc import Iterable, Iterator
from pathlib import Path
from typing import Iterator
import click
import yaml
from github import Github, GithubException
from ghapi.core import GhApi, HTTP404NotFoundError # type: ignore
from rich import print
from cibuildwheel.projectfiles import Analyzer
@@ -59,44 +47,39 @@ def check_repo(name: str, contents: str) -> str:
class MaybeRemote:
github: Github | None
contents: dict[str, dict[str, str | None]]
def __init__(self, cached_file: Path | str, *, online: str | None) -> None:
if online is not None:
self.github = Github(online)
self.contents = {
def __init__(self, cached_file: Path | str, *, online: bool) -> None:
self.online = online
if self.online:
self.contents: dict[str, dict[str, str | None]] = {
"setup.py": {},
"setup.cfg": {},
"pyproject.toml": {},
}
else:
self.github = None
with open(cached_file) as f:
self.contents = yaml.safe_load(f)
def get(self, repo: str, filename: str) -> str | None:
if self.github:
if self.online:
try:
gh_file = self.github.get_repo(repo).get_contents(filename)
except GithubException:
self.contents[filename][repo] = (
GhApi(*repo.split("/")).get_content(filename).decode()
)
except HTTP404NotFoundError:
self.contents[filename][repo] = None
else:
assert not isinstance(gh_file, list)
self.contents[filename][repo] = gh_file.decoded_content.decode(encoding="utf-8")
return self.contents[filename][repo]
elif repo in self.contents[filename]:
return self.contents[filename][repo]
else:
msg = f"Trying to access {repo}:{filename} and not in cache, rebuild cache"
raise RuntimeError(msg)
raise RuntimeError(
f"Trying to access {repo}:{filename} and not in cache, rebuild cache"
)
def save(self, filename: Path | str) -> None:
with open(filename, "w") as f:
yaml.safe_dump(self.contents, f, default_flow_style=False)
def on_each(self, repos: Iterable[str]) -> Iterator[tuple[str, str, str | None]]:
def on_each(self, repos: list[str]) -> Iterator[tuple[str, str, str | None]]:
for repo in repos:
print(f"[bold]{repo}:")
for filename in sorted(self.contents, reverse=True):
@@ -104,8 +87,8 @@ class MaybeRemote:
@click.command()
@click.option("--online", help="Set to $GITHUB_TOKEN")
def main(online: str | None) -> None:
@click.option("--online", is_flag=True, help="Remember to set GITHUB_TOKEN")
def main(online: bool) -> None:
with open(DIR / "../docs/data/projects.yml") as f:
known = yaml.safe_load(f)
+7 -7
View File
@@ -12,13 +12,13 @@ from subprocess import run
import click
def shell(cmd, *, check: bool, **kwargs):
return run([cmd], shell=True, check=check, **kwargs)
def shell(cmd, **kwargs):
return run([cmd], shell=True, **kwargs)
def git_repo_has_changes():
unstaged_changes = shell("git diff-index --quiet HEAD --", check=False).returncode != 0
staged_changes = shell("git diff-index --quiet --cached HEAD --", check=False).returncode != 0
unstaged_changes = shell("git diff-index --quiet HEAD --").returncode != 0
staged_changes = shell("git diff-index --quiet --cached HEAD --").returncode != 0
return unstaged_changes or staged_changes
@@ -40,7 +40,7 @@ def main():
timestamp = time.strftime("%Y-%m-%dT%H-%M-%S", time.gmtime())
branch_name = f"update-constraints-{timestamp}"
shell(f"git checkout -b {branch_name} origin/main", check=True)
shell(f"git checkout -b {branch_name} origin/master", check=True)
try:
shell("bin/update_dependencies.py", check=True)
@@ -63,7 +63,7 @@ def main():
"pr",
"create",
"--repo=pypa/cibuildwheel",
"--base=main",
"--base=master",
"--title=Update dependencies",
f"--body='{body}'",
],
@@ -73,7 +73,7 @@ def main():
print("Done.")
finally:
# remove any local changes
shell("git checkout -- .", check=True)
shell("git checkout -- .")
shell(f"git checkout {previous_branch}", check=True)
shell(f"git branch -D --force {branch_name}", check=True)
+31 -79
View File
@@ -1,19 +1,11 @@
#!/usr/bin/env python3
# /// script
# dependencies = [
# "click",
# "pyyaml",
# "pygithub",
# ]
# ///
"""
Convert a yaml project list into a nice table.
Suggested usage:
./bin/projects.py docs/data/projects.yml --online --auth $GITHUB_API_TOKEN
./bin/projects.py docs/data/projects.yml --online --auth $GITHUB_API_TOKEN --readme README.md
git diff
"""
@@ -24,7 +16,6 @@ import functools
import textwrap
import urllib.request
import xml.dom.minidom
from collections.abc import Iterable, Mapping, Sequence
from datetime import datetime
from io import StringIO
from pathlib import Path
@@ -41,7 +32,6 @@ ICONS = (
"circleci",
"gitlab",
"travisci",
"cirrusci",
"windows",
"apple",
"linux",
@@ -51,7 +41,7 @@ ICONS = (
class Project:
NAME: int = 0
def __init__(self, config: Mapping[str, Any], github: Github | None = None):
def __init__(self, config: dict[str, Any], github: Github | None = None):
try:
self.name: str = config["name"]
self.gh: str = config["gh"]
@@ -72,7 +62,7 @@ class Project:
print(f"Broken: {self.stars_repo}")
raise
self.num_stars: int = repo.stargazers_count
self.num_stars = repo.stargazers_count
self.pushed_at = repo.pushed_at
if not self.notes:
notes = repo.description
@@ -149,35 +139,28 @@ def fetch_icon(icon_name: str) -> None:
f.write(svgElement.toxml())
def path_for_icon(icon_name: str, relative_to: Path | None = None) -> Path:
from_path = Path.cwd() if relative_to is None else relative_to.parent
absolute_path = PROJECT_ROOT / "docs" / "data" / "readme_icons" / f"{icon_name}.svg"
return absolute_path.resolve().relative_to(from_path.resolve())
def path_for_icon(icon_name: str) -> Path:
return Path(".") / "docs" / "data" / "readme_icons" / f"{icon_name}.svg"
def get_projects(
config: Iterable[Mapping[str, Any]],
def str_projects(
config: list[dict[str, Any]],
*,
online: bool = True,
auth: str | None = None,
) -> list[Project]:
) -> str:
io = StringIO()
print = functools.partial(builtins.print, file=io)
if online:
for icon in ICONS:
fetch_icon(icon)
github = Github(auth) if online else None
return sorted((Project(item, github) for item in config), reverse=online)
def render_projects(projects: Sequence[Project], *, dest_path: Path, include_info: bool = True):
io = StringIO()
print = functools.partial(builtins.print, file=io)
projects = sorted((Project(item, github) for item in config), reverse=online)
print(Project.header())
for project in projects:
print(project.table_row())
@@ -187,73 +170,42 @@ def render_projects(projects: Sequence[Project], *, dest_path: Path, include_inf
print()
for icon in ICONS:
print(f"[{icon} icon]: {path_for_icon(icon, relative_to=dest_path).as_posix()}")
print(f"[{icon} icon]: {path_for_icon(icon).as_posix()}")
if include_info:
print()
for project in projects:
print(project.info())
print()
for project in projects:
print(project.info())
return io.getvalue()
def insert_projects_table(
file: Path,
*,
projects: Sequence[Project],
input_filename: str,
include_info: bool = True,
):
text = file.read_text()
projects_table = render_projects(projects, include_info=include_info, dest_path=file)
start_str = "<!-- START bin/projects.py -->\n"
start = text.find(start_str)
assert start != -1
end = text.find("<!-- END bin/projects.py -->\n")
assert end != -1
generated_note = f"<!-- this section is generated by bin/projects.py. Don't edit it directly, instead, edit {input_filename} -->"
new_text = (
f"{text[:start + len(start_str)]}\n{generated_note}\n\n{projects_table}\n{text[end:]}"
)
file.write_text(new_text)
PROJECT_ROOT = Path(__file__).parent / ".."
README_FILE = PROJECT_ROOT / "README.md"
DOCS_PAGE = PROJECT_ROOT / "docs" / "working-examples.md"
@click.command(help="Try ./bin/projects.py docs/data/projects.yml")
@click.command(help="Try ./bin/projects.py docs/data/projects.yml --readme README.md")
@click.argument("input", type=click.File("r"))
@click.option("--online/--no-online", default=True, help="Get info from GitHub")
@click.option("--auth", help="GitHub authentication token")
@click.option(
"--dry-run", default=False, help="Print the output, rather than writing it to files in the repo"
)
@click.option("--readme", type=click.File("r+"), help="Modify a readme file if given")
def projects(
input: TextIO,
online: bool,
auth: str | None,
dry_run: bool,
readme: TextIO | None,
) -> None:
config = yaml.safe_load(input)
projects = get_projects(config, online=online, auth=auth)
output = str_projects(config, online=online, auth=auth)
if dry_run:
output = render_projects(projects, dest_path=README_FILE)
if readme is None:
print(output)
else:
insert_projects_table(
README_FILE, projects=projects[:10], input_filename=input.name, include_info=False
)
insert_projects_table(
DOCS_PAGE, projects=projects, input_filename=input.name, include_info=False
)
text = readme.read()
start_str = "<!-- START bin/projects.py -->\n"
start = text.find(start_str)
end = text.find("<!-- END bin/projects.py -->\n")
generated_note = f"<!-- this section is generated by bin/projects.py. Don't edit it directly, instead, edit {input.name} -->"
new_text = f"{text[:start + len(start_str)]}\n{generated_note}\n\n{output}\n{text[end:]}"
readme.seek(0)
readme.write(new_text)
readme.truncate()
if __name__ == "__main__":
+18 -29
View File
@@ -7,7 +7,7 @@ import shutil
import sys
import textwrap
import time
import typing
from collections import namedtuple
from glob import glob
from pathlib import Path
from subprocess import run
@@ -16,13 +16,13 @@ from urllib.parse import quote
import click
def shell(cmd, *, check: bool, **kwargs):
return run([cmd], shell=True, check=check, **kwargs)
def shell(cmd, **kwargs):
return run([cmd], shell=True, **kwargs)
def git_repo_has_changes():
unstaged_changes = shell("git diff-index --quiet HEAD --", check=False).returncode != 0
staged_changes = shell("git diff-index --quiet --cached HEAD --", check=False).returncode != 0
unstaged_changes = shell("git diff-index --quiet HEAD --").returncode != 0
staged_changes = shell("git diff-index --quiet --cached HEAD --").returncode != 0
return unstaged_changes or staged_changes
@@ -34,22 +34,17 @@ def generate_basic_project(path):
project.generate(path)
class CIService(typing.NamedTuple):
name: str
dst_config_path: str
badge_md: str
CIService = namedtuple("CIService", "name dst_config_path badge_md")
services = [
CIService(
name="appveyor",
dst_config_path="appveyor.yml",
badge_md="[![Build status](https://ci.appveyor.com/api/projects/status/gt3vwl88yt0y3hur/branch/{branch}?svg=true)](https://ci.appveyor.com/project/joerick/cibuildwheel/branch/{branch})",
badge_md="[![Build status](https://ci.appveyor.com/api/projects/status/wbsgxshp05tt1tif/branch/{branch}?svg=true)](https://ci.appveyor.com/project/pypa/cibuildwheel/branch/{branch})",
),
CIService(
name="azure-pipelines",
dst_config_path="azure-pipelines.yml",
badge_md="[![Build Status](https://dev.azure.com/joerick0429/cibuildwheel/_apis/build/status/pypa.cibuildwheel?branchName={branch})](https://dev.azure.com/joerick0429/cibuildwheel/_build/latest?definitionId=2&branchName={branch})",
badge_md="[![Build Status](https://dev.azure.com/joerick0429/cibuildwheel/_apis/build/status/joerick.cibuildwheel?branchName={branch})](https://dev.azure.com/joerick0429/cibuildwheel/_build/latest?definitionId=2&branchName={branch})",
),
CIService(
name="circleci",
@@ -64,29 +59,24 @@ services = [
CIService(
name="travis-ci",
dst_config_path=".travis.yml",
badge_md="[![Build Status](https://app.travis-ci.com/pypa/cibuildwheel.svg?branch={branch})](https://app.travis-ci.com/pypa/cibuildwheel)",
badge_md="[![Build Status](https://travis-ci.org/pypa/cibuildwheel.svg?branch={branch})](https://travis-ci.org/pypa/cibuildwheel)",
),
CIService(
name="gitlab",
dst_config_path=".gitlab-ci.yml",
badge_md="[![Gitlab](https://gitlab.com/joerick/cibuildwheel/badges/{branch}/pipeline.svg)](https://gitlab.com/joerick/cibuildwheel/-/commits/{branch})",
),
CIService(
name="cirrus-ci",
dst_config_path=".cirrus.yml",
badge_md="[![Cirrus CI](https://api.cirrus-ci.com/github/pypa/cibuildwheel.svg?branch={branch})](https://cirrus-ci.com/github/pypa/cibuildwheel/{branch})",
badge_md="[![Gitlab](https://gitlab.com/pypa/cibuildwheel/badges/{branch}/pipeline.svg)](https://gitlab.com/pypa/cibuildwheel/-/commits/{branch})",
),
]
def ci_service_for_config_file(config_file):
filename = Path(config_file).name
service_name = Path(config_file).name.rsplit("-", 1)[0]
try:
return next(s for s in services if filename.startswith(s.name))
except StopIteration:
msg = f"unknown ci service for config file {config_file}"
raise ValueError(msg) from None
for service in services:
if service.name == service_name:
return service
raise ValueError(f"unknown ci service for config file {config_file}")
@click.command()
@@ -105,8 +95,7 @@ def run_example_ci_configs(config_files=None):
for config_file in config_files:
service = ci_service_for_config_file(config_file)
if service.name in configs_by_service:
msg = "You cannot specify more than one config per CI service"
raise Exception(msg)
raise Exception("You cannot specify more than one config per CI service")
configs_by_service[service.name] = config_file
if git_repo_has_changes():
@@ -168,7 +157,7 @@ def run_example_ci_configs(config_files=None):
finally:
# remove any local changes
shutil.rmtree(example_project, ignore_errors=True)
shell("git checkout -- .", check=True)
shell("git checkout -- .")
shell(f"git checkout {previous_branch}", check=True)
shell(f"git branch -D --force {branch_name}", check=True)
+8 -41
View File
@@ -1,56 +1,23 @@
#!/usr/bin/env python3
from __future__ import annotations
import argparse
import os
import subprocess
import sys
from pathlib import Path
if __name__ == "__main__":
default_cpu_count = os.cpu_count() or 2
parser = argparse.ArgumentParser()
parser.add_argument(
"--run-podman", action="store_true", default=False, help="run podman tests (linux only)"
)
parser.add_argument(
"--num-processes",
action="store",
default=default_cpu_count,
help="number of processes to use for testing",
)
args = parser.parse_args()
# move cwd to the project root
os.chdir(Path(__file__).resolve().parents[1])
# unit tests
# run the unit tests
unit_test_args = [sys.executable, "-m", "pytest", "unit_test"]
if sys.platform.startswith("linux") and os.environ.get("CIBW_PLATFORM", "linux") == "linux":
# run the docker unit tests only on Linux
# run the docker unit tests only on Linux
if sys.platform.startswith("linux"):
unit_test_args += ["--run-docker"]
if args.run_podman:
unit_test_args += ["--run-podman"]
subprocess.run(unit_test_args, check=True)
# integration tests
integration_test_args = [
sys.executable,
"-m",
"pytest",
f"--numprocesses={args.num_processes}",
"-x",
"--durations",
"0",
"--timeout=2400",
"test",
]
if sys.platform.startswith("linux") and args.run_podman:
integration_test_args += ["--run-podman"]
subprocess.run(integration_test_args, check=True)
# run the integration tests
subprocess.run(
[sys.executable, "-m", "pytest", "-x", "--durations", "0", "--timeout=2400", "test"],
check=True,
)
+1 -5
View File
@@ -24,8 +24,4 @@ if __name__ == "__main__":
check=True,
)
sys.exit(
subprocess.run(
[sys.executable, "-m", "cibuildwheel"], cwd=project_dir, check=False
).returncode
)
sys.exit(subprocess.run([sys.executable, "-m", "cibuildwheel"], cwd=project_dir).returncode)
@@ -2,70 +2,97 @@
from __future__ import annotations
import configparser
from dataclasses import dataclass
from pathlib import Path
import os
import shutil
import subprocess
import sys
from typing import NamedTuple
import requests
DIR = Path(__file__).parent.resolve()
RESOURCES = DIR.parent / "cibuildwheel/resources"
os.chdir(os.path.dirname(__file__))
os.chdir("..")
# CUSTOM_COMPILE_COMMAND is a pip-compile option that tells users how to
# regenerate the constraints files
os.environ["CUSTOM_COMPILE_COMMAND"] = "bin/update_dependencies.py"
PYTHON_VERSIONS = ["27", "35", "36", "37", "38", "39"]
if "--no-docker" in sys.argv:
for python_version in PYTHON_VERSIONS:
subprocess.run(
[
f"./env{python_version}/bin/pip-compile",
"--allow-unsafe",
"--upgrade",
"cibuildwheel/resources/constraints.in",
f"--output-file=cibuildwheel/resources/constraints-python{python_version}.txt",
],
check=True,
)
else:
# latest manylinux2010 image with cpython 2.7 support
image_runner = "quay.io/pypa/manylinux2010_x86_64:2021-02-06-3d322a5"
subprocess.run(["docker", "pull", image_runner], check=True)
for python_version in PYTHON_VERSIONS:
abi_flags = "" if int(python_version) >= 38 else "m"
python_path = f"/opt/python/cp{python_version}-cp{python_version}{abi_flags}/bin/"
command = (
f"{python_path}pip install pip-tools && "
f"{python_path}pip-compile --allow-unsafe --upgrade "
"cibuildwheel/resources/constraints.in "
f"--output-file cibuildwheel/resources/constraints-python{python_version}.txt"
)
subprocess.run(
[
"docker",
"run",
"--rm",
"--env=CUSTOM_COMPILE_COMMAND",
f"--volume={os.getcwd()}:/volume",
"--workdir=/volume",
image_runner,
"bash",
"-c",
command,
],
check=True,
)
# default constraints.txt
shutil.copyfile(
f"cibuildwheel/resources/constraints-python{PYTHON_VERSIONS[-1]}.txt",
"cibuildwheel/resources/constraints.txt",
)
@dataclass(frozen=True)
class Image:
class Image(NamedTuple):
manylinux_version: str
platform: str
image_name: str
tag: str | None # Set this to pin the image
tag: str | None
images = [
# manylinux1 images
Image("manylinux1", "x86_64", "quay.io/pypa/manylinux1_x86_64", None),
Image("manylinux1", "i686", "quay.io/pypa/manylinux1_i686", None),
# manylinux2010 images
Image("manylinux2010", "x86_64", "quay.io/pypa/manylinux2010_x86_64", None),
Image("manylinux2010", "i686", "quay.io/pypa/manylinux2010_i686", None),
Image("manylinux2010", "pypy_x86_64", "quay.io/pypa/manylinux2010_x86_64", None),
Image("manylinux2010", "pypy_i686", "quay.io/pypa/manylinux2010_i686", None),
# manylinux2014 images
# Images for manylinux2010 are pinned to the latest tag supporting cp27
Image("manylinux2010", "x86_64", "quay.io/pypa/manylinux2010_x86_64", "2021-02-06-3d322a5"),
Image("manylinux2010", "i686", "quay.io/pypa/manylinux2010_i686", "2021-02-06-3d322a5"),
Image("manylinux2010", "pypy_x86_64", "pypywheels/manylinux2010-pypy_x86_64", None),
# 2014 images
Image("manylinux2014", "x86_64", "quay.io/pypa/manylinux2014_x86_64", None),
Image("manylinux2014", "i686", "quay.io/pypa/manylinux2014_i686", None),
Image("manylinux2014", "aarch64", "quay.io/pypa/manylinux2014_aarch64", None),
Image("manylinux2014", "ppc64le", "quay.io/pypa/manylinux2014_ppc64le", None),
Image("manylinux2014", "s390x", "quay.io/pypa/manylinux2014_s390x", None),
Image("manylinux2014", "pypy_x86_64", "quay.io/pypa/manylinux2014_x86_64", None),
Image("manylinux2014", "pypy_i686", "quay.io/pypa/manylinux2014_i686", None),
Image("manylinux2014", "pypy_aarch64", "quay.io/pypa/manylinux2014_aarch64", None),
# manylinux_2_24 images
# 2_24 images
Image("manylinux_2_24", "x86_64", "quay.io/pypa/manylinux_2_24_x86_64", None),
Image("manylinux_2_24", "i686", "quay.io/pypa/manylinux_2_24_i686", None),
Image("manylinux_2_24", "aarch64", "quay.io/pypa/manylinux_2_24_aarch64", None),
Image("manylinux_2_24", "ppc64le", "quay.io/pypa/manylinux_2_24_ppc64le", None),
Image("manylinux_2_24", "s390x", "quay.io/pypa/manylinux_2_24_s390x", None),
Image("manylinux_2_24", "pypy_x86_64", "quay.io/pypa/manylinux_2_24_x86_64", None),
Image("manylinux_2_24", "pypy_i686", "quay.io/pypa/manylinux_2_24_i686", None),
Image("manylinux_2_24", "pypy_aarch64", "quay.io/pypa/manylinux_2_24_aarch64", None),
# manylinux_2_28 images
Image("manylinux_2_28", "x86_64", "quay.io/pypa/manylinux_2_28_x86_64", None),
Image("manylinux_2_28", "aarch64", "quay.io/pypa/manylinux_2_28_aarch64", None),
Image("manylinux_2_28", "ppc64le", "quay.io/pypa/manylinux_2_28_ppc64le", None),
Image("manylinux_2_28", "s390x", "quay.io/pypa/manylinux_2_28_s390x", None),
Image("manylinux_2_28", "pypy_x86_64", "quay.io/pypa/manylinux_2_28_x86_64", None),
Image("manylinux_2_28", "pypy_aarch64", "quay.io/pypa/manylinux_2_28_aarch64", None),
# musllinux_1_1 images
Image("musllinux_1_1", "x86_64", "quay.io/pypa/musllinux_1_1_x86_64", None),
Image("musllinux_1_1", "i686", "quay.io/pypa/musllinux_1_1_i686", None),
Image("musllinux_1_1", "aarch64", "quay.io/pypa/musllinux_1_1_aarch64", None),
Image("musllinux_1_1", "ppc64le", "quay.io/pypa/musllinux_1_1_ppc64le", None),
Image("musllinux_1_1", "s390x", "quay.io/pypa/musllinux_1_1_s390x", None),
# musllinux_1_2 images
Image("musllinux_1_2", "x86_64", "quay.io/pypa/musllinux_1_2_x86_64", None),
Image("musllinux_1_2", "i686", "quay.io/pypa/musllinux_1_2_i686", None),
Image("musllinux_1_2", "aarch64", "quay.io/pypa/musllinux_1_2_aarch64", None),
Image("musllinux_1_2", "ppc64le", "quay.io/pypa/musllinux_1_2_ppc64le", None),
Image("musllinux_1_2", "s390x", "quay.io/pypa/musllinux_1_2_s390x", None),
Image("musllinux_1_2", "armv7l", "quay.io/pypa/musllinux_1_2_armv7l", None),
]
config = configparser.ConfigParser()
@@ -111,5 +138,5 @@ for image in images:
config[image.platform][image.manylinux_version] = f"{image.image_name}:{tag_name}"
with open(RESOURCES / "pinned_docker_images.cfg", "w") as f:
with open("cibuildwheel/resources/pinned_docker_images.cfg", "w") as f:
config.write(f)
-52
View File
@@ -1,52 +0,0 @@
#!/usr/bin/env python3
from __future__ import annotations
import subprocess
import sys
from pathlib import Path
try:
from html2image import Html2Image # type: ignore[import-not-found]
except ImportError:
sys.exit(
"""
html2image not found. Ensure you have Chrome (on Mac/Windows) or
Chromium (on Linux) installed, and then do:
pip install html2image
"""
)
def main():
subprocess.run(["mkdocs", "build"], check=True)
hti = Html2Image(custom_flags=["--force-device-scale-factor=2"])
html_str = Path("docs/diagram.md").read_text()
css_tags = f"""
<style>{Path("site/css/theme.css").read_text()}</style>
<style>{Path("site/css/theme_extra.css").read_text()}</style>
<style>{Path("site/extra.css").read_text()}</style>
<style>
body {{
background: white;
}}
</style>
"""
html_str = css_tags + html_str
[screenshot, *_] = hti.screenshot(
html_str=html_str,
size=(830, 405),
)
dest_path = Path("docs/data/how-it-works.png")
if dest_path.exists():
dest_path.unlink()
Path(screenshot).rename(dest_path)
if __name__ == "__main__":
main()
-148
View File
@@ -1,148 +0,0 @@
#!/usr/bin/env python3
from __future__ import annotations
import difflib
import logging
from dataclasses import dataclass
from pathlib import Path
from typing import Final
import click
import packaging.specifiers
import requests
import rich
from packaging.version import InvalidVersion, Version
from rich.logging import RichHandler
from rich.syntax import Syntax
from cibuildwheel._compat import tomllib
log = logging.getLogger("cibw")
# Looking up the dir instead of using utils.resources_dir
# since we want to write to it.
DIR: Final[Path] = Path(__file__).parent.parent.resolve()
RESOURCES_DIR: Final[Path] = DIR / "cibuildwheel/resources"
NODEJS_DIST: Final[str] = "https://nodejs.org/dist/"
NODEJS_INDEX: Final[str] = f"{NODEJS_DIST}index.json"
@dataclass(frozen=True, order=True)
class VersionTuple:
version: Version
version_string: str
def parse_nodejs_index() -> list[VersionTuple]:
versions: list[VersionTuple] = []
response = requests.get(NODEJS_INDEX)
response.raise_for_status()
versions_info = response.json()
for version_info in versions_info:
version_string = version_info.get("version", "???")
if not version_info.get("lts", False):
log.debug("Ignoring non LTS release %r", version_string)
continue
if "linux-x64" not in version_info.get("files", []):
log.warning(
"Ignoring release %r which does not include a linux-x64 binary", version_string
)
continue
try:
version = Version(version_string)
if version.is_devrelease:
log.info("Ignoring development release %r", str(version))
continue
if version.is_prerelease:
log.info("Ignoring pre-release %r", str(version))
continue
versions.append(VersionTuple(version, version_string))
except InvalidVersion:
log.warning("Ignoring release %r", version_string)
versions.sort(reverse=True)
return versions
@click.command()
@click.option("--force", is_flag=True)
@click.option(
"--level", default="INFO", type=click.Choice(["WARNING", "INFO", "DEBUG"], case_sensitive=False)
)
def update_nodejs(force: bool, level: str) -> None:
logging.basicConfig(
level="INFO",
format="%(message)s",
datefmt="[%X]",
handlers=[RichHandler(rich_tracebacks=True, markup=True)],
)
log.setLevel(level)
toml_file_path = RESOURCES_DIR / "nodejs.toml"
original_toml = toml_file_path.read_text()
with toml_file_path.open("rb") as f:
nodejs_data = tomllib.load(f)
nodejs_data.pop("url")
major_versions = [VersionTuple(Version(key), key) for key in nodejs_data]
major_versions.sort(reverse=True)
versions = parse_nodejs_index()
# update existing versions, 1 per LTS
for major_version in major_versions:
current = Version(nodejs_data[major_version.version_string])
specifier = packaging.specifiers.SpecifierSet(
specifiers=f"=={major_version.version.major}.*"
)
for version in versions:
if specifier.contains(version.version) and version.version > current:
nodejs_data[major_version.version_string] = version.version_string
break
# check for a new major LTS to insert
if versions and versions[0].version.major > major_versions[0].version.major:
major_versions.insert(
0,
VersionTuple(Version(str(versions[0].version.major)), f"v{versions[0].version.major}"),
)
nodejs_data[major_versions[0].version_string] = versions[0].version_string
versions_toml = "\n".join(
f'{major_version.version_string} = "{nodejs_data[major_version.version_string]}"'
for major_version in major_versions
)
result_toml = f'url = "{NODEJS_DIST}"\n{versions_toml}\n'
rich.print() # spacer
if original_toml == result_toml:
rich.print("[green]Check complete, nodejs version unchanged.")
return
rich.print("nodejs version updated.")
rich.print("Changes:")
rich.print()
toml_relpath = toml_file_path.relative_to(DIR).as_posix()
diff_lines = difflib.unified_diff(
original_toml.splitlines(keepends=True),
result_toml.splitlines(keepends=True),
fromfile=toml_relpath,
tofile=toml_relpath,
)
rich.print(Syntax("".join(diff_lines), "diff", theme="ansi_light"))
rich.print()
if force:
toml_file_path.write_text(result_toml)
rich.print("[green]TOML file updated.")
else:
rich.print("[yellow]File left unchanged. Use --force flag to update.")
if __name__ == "__main__":
update_nodejs()
+80 -112
View File
@@ -5,20 +5,20 @@ from __future__ import annotations
import copy
import difflib
import logging
from collections.abc import Mapping, MutableMapping
from pathlib import Path
from typing import Any, Final, Literal, TypedDict, Union
from typing import Union
import click
import requests
import rich
import toml
from packaging.specifiers import Specifier
from packaging.version import Version
from rich.logging import RichHandler
from rich.syntax import Syntax
from cibuildwheel._compat import tomllib
from cibuildwheel.extra import dump_python_configurations
from cibuildwheel.extra import InlineArrayDictEncoder
from cibuildwheel.typing import Final, Literal, TypedDict
log = logging.getLogger("cibw")
@@ -28,7 +28,7 @@ DIR: Final[Path] = Path(__file__).parent.parent.resolve()
RESOURCES_DIR: Final[Path] = DIR / "cibuildwheel/resources"
ArchStr = Literal["32", "64", "ARM64"]
ArchStr = Literal["32", "64"]
class ConfigWinCP(TypedDict):
@@ -58,7 +58,8 @@ AnyConfig = Union[ConfigWinCP, ConfigWinPP, ConfigMacOS]
class WindowsVersions:
def __init__(self, arch_str: ArchStr, free_threaded: bool) -> None:
def __init__(self, arch_str: ArchStr) -> None:
response = requests.get("https://api.nuget.org/v3/index.json")
response.raise_for_status()
api_info = response.json()
@@ -67,47 +68,41 @@ class WindowsVersions:
if resource["@type"] == "PackageBaseAddress/3.0.0":
endpoint = resource["@id"]
ARCH_DICT = {"32": "win32", "64": "win_amd64", "ARM64": "win_arm64"}
PACKAGE_DICT = {"32": "pythonx86", "64": "python", "ARM64": "pythonarm64"}
ARCH_DICT = {"32": "win32", "64": "win_amd64"}
PACKAGE_DICT = {"32": "pythonx86", "64": "python"}
self.arch_str = arch_str
self.arch = ARCH_DICT[arch_str]
self.free_threaded = free_threaded
package = PACKAGE_DICT[arch_str]
if free_threaded:
package = f"{package}-freethreaded"
response = requests.get(f"{endpoint}{package}/index.json")
response.raise_for_status()
cp_info = response.json()
self.version_dict = {Version(v): v for v in cp_info["versions"]}
versions = (Version(v) for v in cp_info["versions"])
self.versions = sorted(v for v in versions if not v.is_devrelease)
def update_version_windows(self, spec: Specifier) -> ConfigWinCP | None:
# Specifier.filter selects all non pre-releases that match the spec,
# unless there are only pre-releases, then it selects pre-releases
# instead (like pip)
unsorted_versions = spec.filter(self.version_dict)
versions = sorted(unsorted_versions, reverse=True)
log.debug("Windows %s %s has %s", self.arch, spec, ", ".join(str(v) for v in versions))
versions = sorted(v for v in self.versions if spec.contains(v))
if not all(v.is_prerelease for v in versions):
versions = [v for v in versions if not v.is_prerelease]
log.debug(f"Windows {self.arch} {spec} has {', '.join(str(v) for v in versions)}")
if not versions:
return None
flags = "t" if self.free_threaded else ""
version = versions[0]
identifier = f"cp{version.major}{version.minor}{flags}-{self.arch}"
version = versions[-1]
identifier = f"cp{version.major}{version.minor}-{self.arch}"
return ConfigWinCP(
identifier=identifier,
version=self.version_dict[version],
version=str(version),
arch=self.arch_str,
)
class PyPyVersions:
def __init__(self, arch_str: ArchStr):
response = requests.get("https://downloads.python.org/pypy/versions.json")
response.raise_for_status()
@@ -123,60 +118,48 @@ class PyPyVersions:
]
self.arch = arch_str
def get_arch_file(self, release: Mapping[str, Any]) -> str:
urls: list[str] = [
rf["download_url"]
for rf in release["files"]
if "" in rf["platform"] == f"win{self.arch}"
]
return urls[0] if urls else ""
def update_version_windows(self, spec: Specifier) -> ConfigWinCP:
if self.arch != "32":
raise RuntimeError("64 bit releases not supported yet on Windows")
releases = [r for r in self.releases if spec.contains(r["python_version"])]
releases = sorted(releases, key=lambda r: r["pypy_version"])
releases = [r for r in releases if self.get_arch_file(r)]
if not releases:
msg = f"PyPy Win {self.arch} not found for {spec}! {self.releases}"
raise RuntimeError(msg)
version_arch = "win32" if self.arch == "32" else "win_amd64"
raise RuntimeError(f"PyPy Win {self.arch} not found for {spec}! {self.releases}")
release = releases[-1]
version = release["python_version"]
identifier = f"pp{version.major}{version.minor}-{version_arch}"
url = self.get_arch_file(release)
identifier = f"pp{version.major}{version.minor}-win32"
(url,) = [rf["download_url"] for rf in release["files"] if "" in rf["platform"] == "win32"]
return ConfigWinPP(
identifier=identifier,
version=f"{version.major}.{version.minor}",
arch=self.arch,
arch="32",
url=url,
)
def update_version_macos(self, spec: Specifier) -> ConfigMacOS:
if self.arch not in {"64", "ARM64"}:
msg = f"'{self.arch}' arch not supported yet on macOS"
raise RuntimeError(msg)
if self.arch != "64":
raise RuntimeError("Other archs not supported yet on macOS")
releases = [r for r in self.releases if spec.contains(r["python_version"])]
releases = sorted(releases, key=lambda r: r["pypy_version"])
if not releases:
msg = f"PyPy macOS {self.arch} not found for {spec}!"
raise RuntimeError(msg)
raise RuntimeError(f"PyPy macOS {self.arch} not found for {spec}!")
release = releases[-1]
version = release["python_version"]
arch = "x86_64" if self.arch == "64" else self.arch.lower()
identifier = f"pp{version.major}{version.minor}-macosx_{arch}"
identifier = f"pp{version.major}{version.minor}-macosx_x86_64"
arch = "x64" if self.arch == "64" else self.arch.lower()
(url,) = (
(url,) = [
rf["download_url"]
for rf in release["files"]
if "" in rf["platform"] == "darwin" and rf["arch"] == arch
)
if "" in rf["platform"] == "darwin" and rf["arch"] == "x64"
]
return ConfigMacOS(
identifier=identifier,
@@ -187,6 +170,7 @@ class PyPyVersions:
class CPythonVersions:
def __init__(self) -> None:
response = requests.get(
"https://www.python.org/api/v2/downloads/release/?is_published=true"
)
@@ -199,35 +183,31 @@ class CPythonVersions:
# Removing the prefix, Python 3.9 would use: release["name"].removeprefix("Python ")
version = Version(release["name"][7:])
uri = int(release["resource_uri"].rstrip("/").split("/")[-1])
self.versions_dict[version] = uri
if not version.is_prerelease and not version.is_devrelease:
uri = int(release["resource_uri"].rstrip("/").split("/")[-1])
self.versions_dict[version] = uri
def update_version_macos(
self, identifier: str, version: Version, spec: Specifier
) -> ConfigMacOS | None:
# see note above on Specifier.filter
unsorted_versions = spec.filter(self.versions_dict)
sorted_versions = sorted(unsorted_versions, reverse=True)
def update_version_macos(self, identifier: str, spec: Specifier) -> ConfigMacOS | None:
file_idents = ("macos11.pkg", "macosx10.9.pkg", "macosx10.6.pkg")
sorted_versions = sorted(v for v in self.versions_dict if spec.contains(v))
macver = "x10.9" if version <= Version("3.8.9999") else "11"
file_ident = f"macos{macver}.pkg"
for new_version in sorted_versions:
for version in reversed(sorted_versions):
# Find the first patch version that contains the requested file
uri = self.versions_dict[new_version]
uri = self.versions_dict[version]
response = requests.get(
f"https://www.python.org/api/v2/downloads/release_file/?release={uri}"
)
response.raise_for_status()
file_info = response.json()
urls = [rf["url"] for rf in file_info if file_ident in rf["url"]]
if urls:
return ConfigMacOS(
identifier=identifier,
version=f"{new_version.major}.{new_version.minor}",
url=urls[0],
)
for file_ident in file_idents:
urls = [rf["url"] for rf in file_info if file_ident in rf["url"]]
if urls:
return ConfigMacOS(
identifier=identifier,
version=f"{version.major}.{version.minor}",
url=urls[0],
)
return None
@@ -238,64 +218,53 @@ class CPythonVersions:
class AllVersions:
def __init__(self) -> None:
self.windows_32 = WindowsVersions("32", False)
self.windows_t_32 = WindowsVersions("32", True)
self.windows_64 = WindowsVersions("64", False)
self.windows_t_64 = WindowsVersions("64", True)
self.windows_arm64 = WindowsVersions("ARM64", False)
self.windows_t_arm64 = WindowsVersions("ARM64", True)
self.windows_pypy_64 = PyPyVersions("64")
self.windows_32 = WindowsVersions("32")
self.windows_64 = WindowsVersions("64")
self.windows_pypy = PyPyVersions("32")
self.macos_cpython = CPythonVersions()
self.macos_pypy = PyPyVersions("64")
self.macos_pypy_arm64 = PyPyVersions("ARM64")
def update_config(self, config: MutableMapping[str, str]) -> None:
def update_config(self, config: dict[str, str]) -> None:
identifier = config["identifier"]
version = Version(config["version"])
spec = Specifier(f"=={version.major}.{version.minor}.*")
log.info("Reading in %r -> %s @ %s", str(identifier), spec, version)
log.info(f"Reading in '{identifier}' -> {spec} @ {version}")
orig_config = copy.copy(config)
config_update: AnyConfig | None = None
config_update: AnyConfig | None
# We need to use ** in update due to MyPy (probably a bug)
if "macosx" in identifier:
if identifier.startswith("cp"):
config_update = self.macos_cpython.update_version_macos(identifier, version, spec)
elif identifier.startswith("pp"):
if "macosx_x86_64" in identifier:
config_update = self.macos_pypy.update_version_macos(spec)
elif "macosx_arm64" in identifier:
config_update = self.macos_pypy_arm64.update_version_macos(spec)
elif "t-win32" in identifier and identifier.startswith("cp"):
config_update = self.windows_t_32.update_version_windows(spec)
elif "win32" in identifier and identifier.startswith("cp"):
config_update = self.windows_32.update_version_windows(spec)
elif "t-win_amd64" in identifier and identifier.startswith("cp"):
config_update = self.windows_t_64.update_version_windows(spec)
elif "win_amd64" in identifier:
if identifier.startswith("cp"):
config_update = self.windows_64.update_version_windows(spec)
elif identifier.startswith("pp"):
config_update = self.windows_pypy_64.update_version_windows(spec)
elif "t-win_arm64" in identifier and identifier.startswith("cp"):
config_update = self.windows_t_arm64.update_version_windows(spec)
elif "win_arm64" in identifier and identifier.startswith("cp"):
config_update = self.windows_arm64.update_version_windows(spec)
if "macos" in identifier:
if identifier.startswith("pp"):
config_update = self.macos_pypy.update_version_macos(spec)
else:
config_update = self.macos_cpython.update_version_macos(identifier, spec)
assert config_update is not None, f"{identifier} not found!"
config.update(**config_update)
assert config_update is not None, f"MacOS {spec} not found!"
config.update(**config_update)
elif "win32" in identifier:
if identifier.startswith("pp"):
config.update(**self.windows_pypy.update_version_windows(spec))
else:
config_update = self.windows_32.update_version_windows(spec)
if config_update:
config.update(**config_update)
elif "win_amd64" in identifier:
config_update = self.windows_64.update_version_windows(spec)
if config_update:
config.update(**config_update)
if config != orig_config:
log.info(" Updated %s to %s", orig_config, config)
log.info(f" Updated {orig_config} to {config}")
@click.command()
@click.option("--force", is_flag=True)
@click.option(
"--level", default="INFO", type=click.Choice(["WARNING", "INFO", "DEBUG"], case_sensitive=False)
"--level", default="INFO", type=click.Choice(["INFO", "DEBUG", "TRACE"], case_sensitive=False)
)
def update_pythons(force: bool, level: str) -> None:
logging.basicConfig(
level="INFO",
format="%(message)s",
@@ -308,8 +277,7 @@ def update_pythons(force: bool, level: str) -> None:
toml_file_path = RESOURCES_DIR / "build-platforms.toml"
original_toml = toml_file_path.read_text()
with toml_file_path.open("rb") as f:
configs = tomllib.load(f)
configs = toml.loads(original_toml)
for config in configs["windows"]["python_configurations"]:
all_versions.update_config(config)
@@ -317,7 +285,7 @@ def update_pythons(force: bool, level: str) -> None:
for config in configs["macos"]["python_configurations"]:
all_versions.update_config(config)
result_toml = dump_python_configurations(configs)
result_toml = toml.dumps(configs, encoder=InlineArrayDictEncoder()) # type: ignore
rich.print() # spacer
-49
View File
@@ -1,49 +0,0 @@
#!/usr/bin/env python3
from __future__ import annotations
import re
import sys
from pathlib import Path
PROJECT_ROOT = Path(__file__).parent / ".."
CHANGELOG_FILE = PROJECT_ROOT / "docs" / "changelog.md"
README_FILE = PROJECT_ROOT / "README.md"
# https://regexr.com/622ds
FIRST_5_CHANGELOG_ENTRIES_REGEX = re.compile(r"""(^###.*?(?=###)){5}""", re.DOTALL | re.MULTILINE)
# https://regexr.com/622e5
README_CHANGELOG_SECTION = re.compile(
r"""(?<=<!-- START bin\/update_readme_changelog.py -->\n).*(?=<!-- END bin\/update_readme_changelog.py -->)""",
re.DOTALL,
)
def main():
changelog_text = CHANGELOG_FILE.read_text()
readme_text = README_FILE.read_text()
mini_changelog_match = FIRST_5_CHANGELOG_ENTRIES_REGEX.search(changelog_text)
assert mini_changelog_match, "Failed to find the first few changelog entries"
mini_changelog = "\n".join(
[
"",
"<!-- this section was generated by bin/update_readme_changelog.py -- do not edit manually -->",
"",
mini_changelog_match.group(0).strip(),
"",
"",
]
)
if not re.search(README_CHANGELOG_SECTION, readme_text):
sys.exit("Changelog section not found in README")
readme_text = re.sub(README_CHANGELOG_SECTION, mini_changelog, readme_text)
README_FILE.write_text(readme_text)
if __name__ == "__main__":
main()
-125
View File
@@ -1,125 +0,0 @@
#!/usr/bin/env python3
from __future__ import annotations
import difflib
import logging
import subprocess
from dataclasses import dataclass
from pathlib import Path
from typing import Final
import click
import rich
from packaging.version import InvalidVersion, Version
from rich.logging import RichHandler
from rich.syntax import Syntax
from cibuildwheel._compat import tomllib
log = logging.getLogger("cibw")
# Looking up the dir instead of using utils.resources_dir
# since we want to write to it.
DIR: Final[Path] = Path(__file__).parent.parent.resolve()
RESOURCES_DIR: Final[Path] = DIR / "cibuildwheel/resources"
GET_VIRTUALENV_GITHUB: Final[str] = "https://github.com/pypa/get-virtualenv"
GET_VIRTUALENV_URL_TEMPLATE: Final[str] = (
f"{GET_VIRTUALENV_GITHUB}/blob/{{version}}/public/virtualenv.pyz?raw=true"
)
@dataclass(frozen=True, order=True)
class VersionTuple:
version: Version
version_string: str
def git_ls_remote_versions(url) -> list[VersionTuple]:
versions: list[VersionTuple] = []
tags = subprocess.run(
["git", "ls-remote", "--tags", url], check=True, text=True, capture_output=True
).stdout.splitlines()
for tag in tags:
_, ref = tag.split()
assert ref.startswith("refs/tags/")
version_string = ref[10:]
try:
version = Version(version_string)
if version.is_devrelease:
log.info("Ignoring development release %r", str(version))
continue
if version.is_prerelease:
log.info("Ignoring pre-release %r", str(version))
continue
versions.append(VersionTuple(version, version_string))
except InvalidVersion:
log.warning("Ignoring ref %r", ref)
versions.sort(reverse=True)
return versions
@click.command()
@click.option("--force", is_flag=True)
@click.option(
"--level", default="INFO", type=click.Choice(["WARNING", "INFO", "DEBUG"], case_sensitive=False)
)
def update_virtualenv(force: bool, level: str) -> None:
logging.basicConfig(
level="INFO",
format="%(message)s",
datefmt="[%X]",
handlers=[RichHandler(rich_tracebacks=True, markup=True)],
)
log.setLevel(level)
toml_file_path = RESOURCES_DIR / "virtualenv.toml"
original_toml = toml_file_path.read_text()
with toml_file_path.open("rb") as f:
configurations = tomllib.load(f)
default = configurations.pop("default")
version = str(default["version"])
versions = git_ls_remote_versions(GET_VIRTUALENV_GITHUB)
if versions[0].version > Version(version):
version = versions[0].version_string
configurations["default"] = {
"version": version,
"url": GET_VIRTUALENV_URL_TEMPLATE.format(version=version),
}
result_toml = "".join(
f'{key} = {{ version = "{value["version"]}", url = "{value["url"]}" }}\n'
for key, value in configurations.items()
)
rich.print() # spacer
if original_toml == result_toml:
rich.print("[green]Check complete, virtualenv version unchanged.")
return
rich.print("virtualenv version updated.")
rich.print("Changes:")
rich.print()
toml_relpath = toml_file_path.relative_to(DIR).as_posix()
diff_lines = difflib.unified_diff(
original_toml.splitlines(keepends=True),
result_toml.splitlines(keepends=True),
fromfile=toml_relpath,
tofile=toml_relpath,
)
rich.print(Syntax("".join(diff_lines), "diff", theme="ansi_light"))
rich.print()
if force:
toml_file_path.write_text(result_toml)
rich.print("[green]TOML file updated.")
else:
rich.print("[yellow]File left unchanged. Use --force flag to update.")
if __name__ == "__main__":
update_virtualenv()
+1 -3
View File
@@ -1,3 +1 @@
from __future__ import annotations
__version__ = "2.21.3"
__version__ = "1.12.0"
+310 -271
View File
@@ -1,89 +1,90 @@
from __future__ import annotations
import argparse
import dataclasses
import os
import shutil
import sys
import tarfile
import textwrap
import traceback
import typing
from collections.abc import Iterable, Sequence, Set
from configparser import ConfigParser
from pathlib import Path
from tempfile import mkdtemp
from typing import Protocol
from typing import Dict, List, Optional, Set, Union, overload
from packaging.specifiers import SpecifierSet
import cibuildwheel
import cibuildwheel.linux
import cibuildwheel.macos
import cibuildwheel.pyodide
import cibuildwheel.util
import cibuildwheel.windows
from cibuildwheel import errors
from cibuildwheel._compat.typing import assert_never
from cibuildwheel.architecture import Architecture, allowed_architectures_check
from cibuildwheel.logger import log
from cibuildwheel.options import CommandLineArguments, Options, compute_options
from cibuildwheel.typing import PLATFORMS, GenericPythonConfiguration, PlatformName
from cibuildwheel.environment import EnvironmentParseError, parse_environment
from cibuildwheel.projectfiles import get_requires_python_str
from cibuildwheel.typing import PLATFORMS, PlatformName, assert_never
from cibuildwheel.util import (
CIBW_CACHE_PATH,
BuildOptions,
BuildSelector,
CIProvider,
DependencyConstraints,
TestSelector,
Unbuffered,
chdir,
detect_ci_provider,
fix_ansi_codes_for_github_actions,
strtobool,
resources_dir,
)
@dataclasses.dataclass
class GlobalOptions:
print_traceback_on_error: bool = True # decides what happens when errors are hit.
@overload
def get_option_from_environment(
option_name: str, *, platform: Optional[str] = None, default: str
) -> str:
... # noqa: E704
@overload
def get_option_from_environment(
option_name: str, *, platform: Optional[str] = None, default: None = None
) -> Optional[str]:
... # noqa: E704 E302
def get_option_from_environment(
option_name: str, *, platform: Optional[str] = None, default: Optional[str] = None
) -> Optional[str]: # noqa: E302
"""
Returns an option from the environment, optionally scoped by the platform.
Example:
get_option_from_environment('CIBW_COLOR', platform='macos')
This will return the value of CIBW_COLOR_MACOS if it exists, otherwise the value of
CIBW_COLOR.
"""
if platform:
option = os.environ.get(f"{option_name}_{platform.upper()}")
if option is not None:
return option
return os.environ.get(option_name, default)
def main() -> None:
global_options = GlobalOptions()
try:
main_inner(global_options)
except errors.FatalError as e:
message = e.args[0]
if log.step_active:
log.step_end_with_error(message)
else:
print(f"cibuildwheel: {message}", file=sys.stderr)
if global_options.print_traceback_on_error:
traceback.print_exc(file=sys.stderr)
sys.exit(e.return_code)
def main_inner(global_options: GlobalOptions) -> None:
"""
`main_inner` is the same as `main`, but it raises FatalError exceptions
rather than exiting directly.
"""
platform: PlatformName
parser = argparse.ArgumentParser(
description="Build wheels for all the platforms.",
epilog="""
Most options are supplied via environment variables or in
--config-file (pyproject.toml usually). See
https://github.com/pypa/cibuildwheel#options for info.
Most options are supplied via environment variables.
See https://github.com/pypa/cibuildwheel#options for info.
""",
)
parser.add_argument(
"--platform",
choices=["auto", "linux", "macos", "windows", "pyodide"],
default=None,
choices=["auto", "linux", "macos", "windows"],
default=os.environ.get("CIBW_PLATFORM", "auto"),
help="""
Platform to build for. Use this option to override the
auto-detected platform. Specifying "macos" or "windows" only works
on that operating system, but "linux" works on all three, as long
as Docker/Podman is installed. Default: auto.
Platform to build for. For "linux" you need docker running, on Mac
or Linux. For "macos", you need a Mac machine, and note that this
script is going to automatically install MacPython on your system,
so don't run on your development machine. For "windows", you need to
run in Windows, and it will build and test for all versions of
Python. Default: auto.
""",
)
@@ -101,47 +102,21 @@ def main_inner(global_options: GlobalOptions) -> None:
""",
)
parser.add_argument(
"--only",
default=None,
help="""
Force a single wheel build when given an identifier. Overrides
CIBW_BUILD/CIBW_SKIP. --platform and --arch cannot be specified
if this is given.
""",
)
parser.add_argument(
"--output-dir",
type=Path,
default=Path(os.environ.get("CIBW_OUTPUT_DIR", "wheelhouse")),
help="Destination folder for the wheels. Default: wheelhouse.",
)
parser.add_argument(
"--config-file",
default="",
help="""
TOML config file. Default: "", meaning {package}/pyproject.toml, if
it exists. To refer to a project inside your project, use {package};
this matters if you build from an SDist.
""",
default=os.environ.get("CIBW_OUTPUT_DIR", "wheelhouse"),
help="Destination folder for the wheels.",
)
parser.add_argument(
"package_dir",
metavar="PACKAGE",
default=Path("."),
type=Path,
default=".",
nargs="?",
help="""
Path to the package that you want wheels for. Default: the working
directory. Can be a directory inside the working directory, or an
sdist. When set to a directory, the working directory is still
considered the 'project' and is copied into the build container
on Linux. When set to a tar.gz sdist file, --config-file
and --output-dir are relative to the current directory, and other
paths are relative to the expanded SDist directory.
Path to the package that you want wheels for. Must be a subdirectory of
the working directory. When set, the working directory is still
considered the 'project' and is copied into the Docker container on
Linux. Default: the working directory.
""",
)
@@ -157,207 +132,258 @@ def main_inner(global_options: GlobalOptions) -> None:
help="Do not report an error code if the build does not match any wheels.",
)
parser.add_argument(
"--prerelease-pythons",
action="store_true",
help="Enable pre-release Python versions if available.",
)
args = parser.parse_args()
parser.add_argument(
"--debug-traceback",
action="store_true",
default=strtobool(os.environ.get("CIBW_DEBUG_TRACEBACK", "0")),
help="Print a full traceback for all errors",
)
detect_obsolete_options()
args = CommandLineArguments(**vars(parser.parse_args()))
global_options.print_traceback_on_error = args.debug_traceback
args.package_dir = args.package_dir.resolve()
# This are always relative to the base directory, even in SDist builds
args.output_dir = args.output_dir.resolve()
# Standard builds if a directory or non-existent path is given
if not args.package_dir.is_file() and not args.package_dir.name.endswith("tar.gz"):
build_in_directory(args)
return
# Tarfile builds require extraction and changing the directory
temp_dir = Path(mkdtemp(prefix="cibw-sdist-")).resolve(strict=True)
try:
with tarfile.open(args.package_dir) as tar:
tar.extractall(path=temp_dir)
# The extract directory is now the project dir
try:
(project_dir,) = temp_dir.iterdir()
except ValueError:
msg = "invalid sdist: didn't contain a single dir"
raise SystemExit(msg) from None
# This is now the new package dir
args.package_dir = project_dir.resolve()
with chdir(project_dir):
build_in_directory(args)
finally:
# avoid https://github.com/python/cpython/issues/86962 by performing
# cleanup manually
shutil.rmtree(temp_dir, ignore_errors=sys.platform.startswith("win"))
if temp_dir.exists():
log.warning(f"Can't delete temporary folder '{temp_dir}'")
def _compute_platform_only(only: str) -> PlatformName:
if "linux_" in only:
return "linux"
if "macosx_" in only:
return "macos"
if "win_" in only or "win32" in only:
return "windows"
if "pyodide_" in only:
return "pyodide"
msg = f"Invalid --only='{only}', must be a build selector with a known platform"
raise errors.ConfigurationError(msg)
def _compute_platform_auto() -> PlatformName:
if sys.platform.startswith("linux"):
return "linux"
elif sys.platform == "darwin":
return "macos"
elif sys.platform == "win32":
return "windows"
if args.platform != "auto":
platform = args.platform
else:
msg = (
'cibuildwheel: Unable to detect platform from "sys.platform". cibuildwheel doesn\'t '
"support building wheels for this platform. You might be able to build for a different "
"platform using the --platform argument. Check --help output for more information."
)
raise errors.ConfigurationError(msg)
ci_provider = detect_ci_provider()
if ci_provider is None:
print(
textwrap.dedent(
"""
cibuildwheel: Unable to detect platform. cibuildwheel should run on your CI server;
Travis CI, AppVeyor, Azure Pipelines, GitHub Actions, CircleCI, and Gitlab are
supported. You can run on your development machine or other CI providers using the
--platform argument. Check --help output for more information.
"""
),
file=sys.stderr,
)
sys.exit(2)
if sys.platform.startswith("linux"):
platform = "linux"
elif sys.platform == "darwin":
platform = "macos"
elif sys.platform == "win32":
platform = "windows"
else:
print(
'cibuildwheel: Unable to detect platform from "sys.platform" in a CI environment. You can run '
"cibuildwheel using the --platform argument. Check --help output for more information.",
file=sys.stderr,
)
sys.exit(2)
def _compute_platform(args: CommandLineArguments) -> PlatformName:
platform_option_value = args.platform or os.environ.get("CIBW_PLATFORM", "") or "auto"
if args.only and args.platform is not None:
msg = "--platform cannot be specified with --only, it is computed from --only"
raise errors.ConfigurationError(msg)
if args.only and args.archs is not None:
msg = "--arch cannot be specified with --only, it is computed from --only"
raise errors.ConfigurationError(msg)
if platform_option_value not in PLATFORMS | {"auto"}:
msg = f"Unsupported platform: {platform_option_value}"
raise errors.ConfigurationError(msg)
if args.only:
return _compute_platform_only(args.only)
elif platform_option_value != "auto":
return typing.cast(PlatformName, platform_option_value)
return _compute_platform_auto()
class PlatformModule(Protocol):
# note that as per PEP544, the self argument is ignored when the protocol
# is applied to a module
def get_python_configurations(
self, build_selector: BuildSelector, architectures: Set[Architecture]
) -> Sequence[GenericPythonConfiguration]: ...
def build(self, options: Options, tmp_path: Path) -> None: ...
def get_platform_module(platform: PlatformName) -> PlatformModule:
if platform == "linux":
return cibuildwheel.linux
if platform == "windows":
return cibuildwheel.windows
if platform == "macos":
return cibuildwheel.macos
if platform == "pyodide":
return cibuildwheel.pyodide
assert_never(platform)
def build_in_directory(args: CommandLineArguments) -> None:
platform: PlatformName = _compute_platform(args)
if platform == "pyodide" and sys.platform == "win32":
msg = "cibuildwheel: Building for pyodide is not supported on Windows"
print(msg, file=sys.stderr)
if platform not in PLATFORMS:
print(f"cibuildwheel: Unsupported platform: {platform}", file=sys.stderr)
sys.exit(2)
options = compute_options(platform=platform, command_line_arguments=args, env=os.environ)
package_dir = Path(args.package_dir)
output_dir = Path(args.output_dir)
if platform == "linux":
repair_command_default = "auditwheel repair -w {dest_dir} {wheel}"
elif platform == "macos":
repair_command_default = "delocate-listdeps {wheel} && delocate-wheel --require-archs {delocate_archs} -w {dest_dir} {wheel}"
elif platform == "windows":
repair_command_default = ""
else:
assert_never(platform)
build_config = os.environ.get("CIBW_BUILD") or "*"
skip_config = os.environ.get("CIBW_SKIP", "")
test_skip = os.environ.get("CIBW_TEST_SKIP", "")
environment_config = get_option_from_environment(
"CIBW_ENVIRONMENT", platform=platform, default=""
)
before_all = get_option_from_environment("CIBW_BEFORE_ALL", platform=platform, default="")
before_build = get_option_from_environment("CIBW_BEFORE_BUILD", platform=platform)
repair_command = get_option_from_environment(
"CIBW_REPAIR_WHEEL_COMMAND", platform=platform, default=repair_command_default
)
dependency_versions = get_option_from_environment(
"CIBW_DEPENDENCY_VERSIONS", platform=platform, default="pinned"
)
test_command = get_option_from_environment("CIBW_TEST_COMMAND", platform=platform)
before_test = get_option_from_environment("CIBW_BEFORE_TEST", platform=platform)
test_requires = get_option_from_environment(
"CIBW_TEST_REQUIRES", platform=platform, default=""
).split()
test_extras = get_option_from_environment("CIBW_TEST_EXTRAS", platform=platform, default="")
build_verbosity_str = get_option_from_environment(
"CIBW_BUILD_VERBOSITY", platform=platform, default=""
)
package_dir = options.globals.package_dir
package_files = {"setup.py", "setup.cfg", "pyproject.toml"}
if not any(package_dir.joinpath(name).exists() for name in package_files):
names = ", ".join(sorted(package_files, reverse=True))
msg = f"Could not find any of {{{names}}} at root of package"
raise errors.ConfigurationError(msg)
print(
f"cibuildwheel: Could not find any of {{{names}}} at root of package", file=sys.stderr
)
sys.exit(2)
platform_module = get_platform_module(platform)
identifiers = get_build_identifiers(
platform_module=platform_module,
build_selector=options.globals.build_selector,
architectures=options.globals.architectures,
# Passing this in as an environment variable will override pyproject.toml, setup.cfg, or setup.py
requires_python_str: Optional[str] = os.environ.get(
"CIBW_PROJECT_REQUIRES_PYTHON"
) or get_requires_python_str(package_dir)
requires_python = None if requires_python_str is None else SpecifierSet(requires_python_str)
build_selector = BuildSelector(
build_config=build_config, skip_config=skip_config, requires_python=requires_python
)
test_selector = TestSelector(skip_config=test_skip)
try:
environment = parse_environment(environment_config)
except (EnvironmentParseError, ValueError):
print(f'cibuildwheel: Malformed environment option "{environment_config}"', file=sys.stderr)
traceback.print_exc(None, sys.stderr)
sys.exit(2)
if dependency_versions == "pinned":
dependency_constraints: Optional[
DependencyConstraints
] = DependencyConstraints.with_defaults()
elif dependency_versions == "latest":
dependency_constraints = None
else:
dependency_versions_path = Path(dependency_versions)
dependency_constraints = DependencyConstraints(dependency_versions_path)
if test_extras:
test_extras = f"[{test_extras}]"
try:
build_verbosity = min(3, max(-3, int(build_verbosity_str)))
except ValueError:
build_verbosity = 0
# Add CIBUILDWHEEL environment variable
# This needs to be passed on to the docker container in linux.py
os.environ["CIBUILDWHEEL"] = "1"
if args.archs is not None:
archs_config_str = args.archs
else:
archs_config_str = get_option_from_environment(
"CIBW_ARCHS", platform=platform, default="auto"
)
archs = Architecture.parse_config(archs_config_str, platform=platform)
identifiers = get_build_identifiers(platform, build_selector, archs)
if args.print_build_identifiers:
for identifier in identifiers:
print(identifier)
sys.exit(0)
# Add CIBUILDWHEEL environment variable
os.environ["CIBUILDWHEEL"] = "1"
manylinux_images: Optional[Dict[str, str]] = None
if platform == "linux":
pinned_docker_images_file = resources_dir / "pinned_docker_images.cfg"
all_pinned_docker_images = ConfigParser()
all_pinned_docker_images.read(pinned_docker_images_file)
# all_pinned_docker_images looks like a dict of dicts, e.g.
# { 'x86_64': {'manylinux1': '...', 'manylinux2010': '...', 'manylinux2014': '...'},
# 'i686': {'manylinux1': '...', 'manylinux2010': '...', 'manylinux2014': '...'},
# 'pypy_x86_64': {'manylinux2010': '...' }
# ... }
# Python is buffering by default when running on the CI platforms, giving
# problems interleaving subprocess call output with unflushed calls to
# 'print'
sys.stdout = Unbuffered(sys.stdout)
manylinux_images = {}
# create the cache dir before it gets printed & builds performed
CIBW_CACHE_PATH.mkdir(parents=True, exist_ok=True)
for build_platform in ["x86_64", "i686", "pypy_x86_64", "aarch64", "ppc64le", "s390x"]:
pinned_images = all_pinned_docker_images[build_platform]
print_preamble(platform=platform, options=options, identifiers=identifiers)
config_name = f"CIBW_MANYLINUX_{build_platform.upper()}_IMAGE"
config_value = os.environ.get(config_name)
if config_value is None:
# default to manylinux2010 if it's available, otherwise manylinux2014
image = pinned_images.get("manylinux2010") or pinned_images.get("manylinux2014")
elif config_value in pinned_images:
image = pinned_images[config_value]
else:
image = config_value
manylinux_images[build_platform] = image
build_options = BuildOptions(
architectures=archs,
package_dir=package_dir,
output_dir=output_dir,
test_command=test_command,
test_requires=test_requires,
test_extras=test_extras,
before_test=before_test,
before_build=before_build,
before_all=before_all,
build_verbosity=build_verbosity,
build_selector=build_selector,
test_selector=test_selector,
repair_command=repair_command,
environment=environment,
dependency_constraints=dependency_constraints,
manylinux_images=manylinux_images,
)
# Python is buffering by default when running on the CI platforms, giving problems interleaving subprocess call output with unflushed calls to 'print'
sys.stdout = Unbuffered(sys.stdout) # type: ignore
print_preamble(platform, build_options)
try:
options.check_for_invalid_configuration(identifiers)
allowed_architectures_check(platform, options.globals.architectures)
allowed_architectures_check(platform, build_options.architectures)
except ValueError as err:
raise errors.DeprecationError(*err.args) from err
print("cibuildwheel:", *err.args, file=sys.stderr)
sys.exit(4)
if not identifiers:
message = f"No build identifiers selected: {options.globals.build_selector}"
if options.globals.allow_empty:
print(f"cibuildwheel: {message}", file=sys.stderr)
else:
raise errors.NothingToDoError(message)
output_dir = options.globals.output_dir
print(f"cibuildwheel: No build identifiers selected: {build_selector}", file=sys.stderr)
if not args.allow_empty:
sys.exit(3)
if not output_dir.exists():
output_dir.mkdir(parents=True)
tmp_path = Path(mkdtemp(prefix="cibw-run-")).resolve(strict=True)
try:
with cibuildwheel.util.print_new_wheels(
"\n{n} wheels produced in {m:.0f} minutes:", output_dir
):
platform_module.build(options, tmp_path)
finally:
# avoid https://github.com/python/cpython/issues/86962 by performing
# cleanup manually
shutil.rmtree(tmp_path, ignore_errors=sys.platform.startswith("win"))
if tmp_path.exists():
log.warning(f"Can't delete temporary folder '{tmp_path}'")
with cibuildwheel.util.print_new_wheels(
"\n{n} wheels produced in {m:.0f} minutes:", output_dir
):
if platform == "linux":
cibuildwheel.linux.build(build_options)
elif platform == "windows":
cibuildwheel.windows.build(build_options)
elif platform == "macos":
cibuildwheel.macos.build(build_options)
else:
assert_never(platform)
def print_preamble(platform: str, options: Options, identifiers: Sequence[str]) -> None:
def detect_obsolete_options() -> None:
# Check the old 'MANYLINUX1_*_IMAGE' options
for (deprecated, alternative) in [
("CIBW_MANYLINUX1_X86_64_IMAGE", "CIBW_MANYLINUX_X86_64_IMAGE"),
("CIBW_MANYLINUX1_I686_IMAGE", "CIBW_MANYLINUX_I686_IMAGE"),
]:
if deprecated in os.environ:
print(
f"'{deprecated}' has been deprecated, and will be removed in a future release. Use the option '{alternative}' instead."
)
if alternative not in os.environ:
print(f"Using value of option '{deprecated}' as replacement for '{alternative}'")
os.environ[alternative] = os.environ[deprecated]
else:
print(f"Option '{alternative}' is not empty. Please unset '{deprecated}'")
sys.exit(2)
# Check for deprecated identifiers in 'CIBW_BUILD' and 'CIBW_SKIP' options
for option in ["CIBW_BUILD", "CIBW_SKIP"]:
for deprecated, alternative in [
("manylinux1", "manylinux"),
("macosx_10_6_intel", "macosx_x86_64"),
("macosx_10_9_x86_64", "macosx_x86_64"),
]:
if option in os.environ and deprecated in os.environ[option]:
print(
f"Build identifiers with '{deprecated}' have been deprecated. Replacing all occurences of '{deprecated}' with '{alternative}' in the option '{option}'"
)
os.environ[option] = os.environ[option].replace(deprecated, alternative)
def print_preamble(platform: str, build_options: BuildOptions) -> None:
print(
textwrap.dedent(
"""
@@ -372,16 +398,11 @@ def print_preamble(platform: str, options: Options, identifiers: Sequence[str])
print(f"cibuildwheel version {cibuildwheel.__version__}\n")
print("Build options:")
print(f" platform: {platform}")
options_summary = textwrap.indent(options.summary(identifiers), " ")
if detect_ci_provider() == CIProvider.github_actions:
options_summary = fix_ansi_codes_for_github_actions(options_summary)
print(options_summary)
print(f" platform: {platform!r}")
for option, value in sorted(build_options._asdict().items()):
print(f" {option}: {value!r}")
print()
print(f"Cache folder: {CIBW_CACHE_PATH}")
warnings = detect_warnings(options=options, identifiers=identifiers)
warnings = detect_warnings(platform, build_options)
if warnings:
print("\nWarnings:")
for warning in warnings:
@@ -391,22 +412,40 @@ def print_preamble(platform: str, options: Options, identifiers: Sequence[str])
def get_build_identifiers(
platform_module: PlatformModule,
build_selector: BuildSelector,
architectures: Set[Architecture],
) -> list[str]:
python_configurations = platform_module.get_python_configurations(build_selector, architectures)
platform: PlatformName, build_selector: BuildSelector, architectures: Set[Architecture]
) -> List[str]:
python_configurations: Union[
List[cibuildwheel.linux.PythonConfiguration],
List[cibuildwheel.windows.PythonConfiguration],
List[cibuildwheel.macos.PythonConfiguration],
]
if platform == "linux":
python_configurations = cibuildwheel.linux.get_python_configurations(
build_selector, architectures
)
elif platform == "windows":
python_configurations = cibuildwheel.windows.get_python_configurations(
build_selector, architectures
)
elif platform == "macos":
python_configurations = cibuildwheel.macos.get_python_configurations(
build_selector, architectures
)
else:
assert_never(platform)
return [config.identifier for config in python_configurations]
def detect_warnings(*, options: Options, identifiers: Iterable[str]) -> list[str]:
def detect_warnings(platform: str, build_options: BuildOptions) -> List[str]:
warnings = []
# warn about deprecated {python} and {pip}
for option_name in ["test_command", "before_build"]:
option_values = [getattr(options.build_options(i), option_name) for i in identifiers]
option_value = getattr(build_options, option_name)
if any(o and ("{python}" in o or "{pip}" in o) for o in option_values):
if option_value and ("{python}" in option_value or "{pip}" in option_value):
# Reminder: in an f-string, double braces means literal single brace
msg = (
f"{option_name}: '{{python}}' and '{{pip}}' are no longer needed, "
-1
View File
@@ -1 +0,0 @@
from __future__ import annotations
-10
View File
@@ -1,10 +0,0 @@
from __future__ import annotations
import sys
if sys.version_info >= (3, 11):
from tomllib import load, loads
else:
from tomli import load, loads
__all__ = ["load", "loads"]
-14
View File
@@ -1,14 +0,0 @@
from __future__ import annotations
import sys
if sys.version_info < (3, 11):
from typing_extensions import NotRequired, Self, assert_never
else:
from typing import NotRequired, Self, assert_never
__all__ = (
"assert_never",
"NotRequired",
"Self",
)
+32 -84
View File
@@ -1,28 +1,12 @@
from __future__ import annotations
import functools
import platform as platform_module
import re
import sys
from collections.abc import Set
from enum import Enum
from typing import Final, Literal
from typing import Set
from ._compat.typing import assert_never
from .typing import PlatformName
from .typing import Literal, PlatformName, assert_never
PRETTY_NAMES: Final[dict[PlatformName, str]] = {
"linux": "Linux",
"macos": "macOS",
"windows": "Windows",
"pyodide": "Pyodide",
}
ARCH_SYNONYMS: Final[list[dict[PlatformName, str | None]]] = [
{"linux": "x86_64", "macos": "x86_64", "windows": "AMD64"},
{"linux": "i686", "macos": None, "windows": "x86"},
{"linux": "aarch64", "macos": "arm64", "windows": "ARM64"},
]
PRETTY_NAMES = {"linux": "Linux", "macos": "macOS", "windows": "Windows"}
@functools.total_ordering
@@ -37,7 +21,6 @@ class Architecture(Enum):
aarch64 = "aarch64"
ppc64le = "ppc64le"
s390x = "s390x"
armv7l = "armv7l"
# mac archs
universal2 = "universal2"
@@ -46,28 +29,19 @@ class Architecture(Enum):
# windows archs
x86 = "x86"
AMD64 = "AMD64"
ARM64 = "ARM64"
# WebAssembly
wasm32 = "wasm32"
# Allow this to be sorted
def __lt__(self, other: Architecture) -> bool:
def __lt__(self, other: "Architecture") -> bool:
return self.value < other.value
def __str__(self) -> str:
return self.name
@staticmethod
def parse_config(config: str, platform: PlatformName) -> set[Architecture]:
def parse_config(config: str, platform: PlatformName) -> "Set[Architecture]":
result = set()
for arch_str in re.split(r"[\s,]+", config):
if arch_str == "auto":
result |= Architecture.auto_archs(platform=platform)
elif arch_str == "native":
native_arch = Architecture.native_arch(platform=platform)
if native_arch:
result.add(native_arch)
result.add(Architecture(platform_module.machine()))
elif arch_str == "all":
result |= Architecture.all_archs(platform=platform)
elif arch_str == "auto64":
@@ -79,84 +53,58 @@ class Architecture(Enum):
return result
@staticmethod
def native_arch(platform: PlatformName) -> Architecture | None:
if platform == "pyodide":
return Architecture.wasm32
def auto_archs(platform: PlatformName) -> "Set[Architecture]":
native_architecture = Architecture(platform_module.machine())
result = {native_architecture}
# Cross-platform support. Used for --print-build-identifiers or docker builds.
host_platform: PlatformName = (
"windows"
if sys.platform.startswith("win")
else ("macos" if sys.platform.startswith("darwin") else "linux")
)
native_machine = platform_module.machine()
native_architecture = Architecture(native_machine)
# we might need to rename the native arch to the machine we're running
# on, as the same arch can have different names on different platforms
if host_platform != platform:
for arch_synonym in ARCH_SYNONYMS:
if native_machine == arch_synonym.get(host_platform):
synonym = arch_synonym[platform]
if synonym is None:
# can't build anything on this platform
return None
native_architecture = Architecture(synonym)
return native_architecture
@staticmethod
def auto_archs(platform: PlatformName) -> set[Architecture]:
native_arch = Architecture.native_arch(platform)
if native_arch is None:
return set() # can't build anything on this platform
result = {native_arch}
if platform == "linux" and Architecture.x86_64 in result:
# x86_64 machines can run i686 containers
if platform == "linux" and native_architecture == Architecture.x86_64:
# x86_64 machines can run i686 docker containers
result.add(Architecture.i686)
if platform == "windows" and Architecture.AMD64 in result:
if platform == "windows" and native_architecture == Architecture.AMD64:
result.add(Architecture.x86)
if platform == "macos" and native_architecture == Architecture.arm64:
# arm64 can build and test both archs of a universal2 wheel.
result.add(Architecture.universal2)
return result
@staticmethod
def all_archs(platform: PlatformName) -> set[Architecture]:
all_archs_map = {
"linux": {
def all_archs(platform: PlatformName) -> "Set[Architecture]":
if platform == "linux":
return {
Architecture.x86_64,
Architecture.i686,
Architecture.aarch64,
Architecture.ppc64le,
Architecture.s390x,
Architecture.armv7l,
},
"macos": {Architecture.x86_64, Architecture.arm64, Architecture.universal2},
"windows": {Architecture.x86, Architecture.AMD64, Architecture.ARM64},
"pyodide": {Architecture.wasm32},
}
return all_archs_map[platform]
}
elif platform == "macos":
return {Architecture.x86_64, Architecture.arm64, Architecture.universal2}
elif platform == "windows":
return {Architecture.x86, Architecture.AMD64}
else:
assert_never(platform)
@staticmethod
def bitness_archs(platform: PlatformName, bitness: Literal["64", "32"]) -> set[Architecture]:
archs_32 = {Architecture.i686, Architecture.x86, Architecture.armv7l}
def bitness_archs(platform: PlatformName, bitness: Literal["64", "32"]) -> "Set[Architecture]":
archs_32 = {Architecture.i686, Architecture.x86}
auto_archs = Architecture.auto_archs(platform)
if bitness == "64":
return auto_archs - archs_32
if bitness == "32":
elif bitness == "32":
return auto_archs & archs_32
assert_never(bitness)
else:
assert_never(bitness)
def allowed_architectures_check(
platform: PlatformName,
architectures: Set[Architecture],
) -> None:
allowed_architectures = Architecture.all_archs(platform)
msg = f"{PRETTY_NAMES[platform]} only supports {sorted(allowed_architectures)} at the moment."
+18 -25
View File
@@ -1,9 +1,5 @@
from __future__ import annotations
import subprocess
from collections.abc import Iterable, Mapping, Sequence
from dataclasses import dataclass
from typing import Callable, Dict, List # noqa: TID251
from typing import Callable, Dict, List, NamedTuple, Optional, Sequence
import bashlex
@@ -11,19 +7,20 @@ import bashlex
EnvironmentExecutor = Callable[[List[str], Dict[str, str]], str]
def local_environment_executor(command: Sequence[str], env: Mapping[str, str]) -> str:
return subprocess.run(command, env=env, text=True, stdout=subprocess.PIPE, check=True).stdout
def local_environment_executor(command: List[str], env: Dict[str, str]) -> str:
return subprocess.run(
command, env=env, universal_newlines=True, stdout=subprocess.PIPE, check=True
).stdout
@dataclass(frozen=True)
class NodeExecutionContext:
environment: dict[str, str]
class NodeExecutionContext(NamedTuple):
environment: Dict[str, str]
input: str
executor: EnvironmentExecutor
def evaluate(
value: str, environment: Mapping[str, str], executor: EnvironmentExecutor | None = None
value: str, environment: Dict[str, str], executor: Optional[EnvironmentExecutor] = None
) -> str:
if not value:
# empty string evaluates to empty string
@@ -33,17 +30,14 @@ def evaluate(
command_node = bashlex.parsesingle(value)
if len(command_node.parts) != 1:
msg = f"{value!r} has too many parts"
raise ValueError(msg)
raise ValueError(f'"{value}" has too many parts')
value_word_node = command_node.parts[0]
return evaluate_node(
value_word_node,
context=NodeExecutionContext(
environment=dict(environment),
input=value,
executor=executor or local_environment_executor,
environment=environment, input=value, executor=executor or local_environment_executor
),
)
@@ -58,8 +52,7 @@ def evaluate_node(node: bashlex.ast.node, context: NodeExecutionContext) -> str:
elif node.kind == "parameter":
return evaluate_parameter_node(node, context=context)
else:
msg = f"Unsupported bash construct: {node.kind!r}"
raise ValueError(msg)
raise ValueError(f'Unsupported bash construct: "{node.kind}"')
def evaluate_word_node(node: bashlex.ast.node, context: NodeExecutionContext) -> str:
@@ -70,8 +63,10 @@ def evaluate_word_node(node: bashlex.ast.node, context: NodeExecutionContext) ->
part_value = evaluate_node(part, context=context)
if part_string not in value:
msg = f"bash parse failed. part {part_string!r} not found in {value!r}. Word was {node.word!r}. Full input was {context.input!r}"
raise RuntimeError(msg)
raise RuntimeError(
f'bash parse failed. part "{part_string}" not found in "{value}". '
f'Word was "{node.word}". Full input was "{context.input}"'
)
value = value.replace(part_string, part_value, 1)
@@ -98,17 +93,15 @@ def evaluate_nodes_as_compound_command(
result += evaluate_command_node(node, context=context)
elif node.kind == "operator":
if node.op != ";":
msg = f"Unsupported bash operator: {node.op!r}"
raise ValueError(msg)
raise ValueError(f'Unsupported bash operator: "{node.op}"')
else:
msg = f"Unsupported bash node in compound command: {node.kind!r}"
raise ValueError(msg)
raise ValueError(f'Unsupported bash node in compound command: "{node.kind}"')
return result
def evaluate_nodes_as_simple_command(
nodes: Iterable[bashlex.ast.node], context: NodeExecutionContext
nodes: List[bashlex.ast.node], context: NodeExecutionContext
) -> str:
command = [evaluate_node(part, context=context) for part in nodes]
return context.executor(command, context.environment)
+242
View File
@@ -0,0 +1,242 @@
import io
import json
import os
import shlex
import subprocess
import sys
import uuid
from pathlib import Path, PurePath
from types import TracebackType
from typing import IO, Dict, List, Optional, Sequence, Type, cast
from .typing import PathOrStr, PopenBytes
class DockerContainer:
"""
An object that represents a running Docker container.
Intended for use as a context manager e.g.
`with DockerContainer('ubuntu') as docker:`
A bash shell is running in the remote container. When `call()` is invoked,
the command is relayed to the remote shell, and the results are streamed
back to cibuildwheel.
"""
UTILITY_PYTHON = "/opt/python/cp38-cp38/bin/python"
process: PopenBytes
bash_stdin: IO[bytes]
bash_stdout: IO[bytes]
def __init__(
self, docker_image: str, simulate_32_bit: bool = False, cwd: Optional[PathOrStr] = None
):
if not docker_image:
raise ValueError("Must have a non-empty docker image to run.")
self.docker_image = docker_image
self.simulate_32_bit = simulate_32_bit
self.cwd = cwd
self.name: Optional[str] = None
def __enter__(self) -> "DockerContainer":
self.name = f"cibuildwheel-{uuid.uuid4()}"
cwd_args = ["-w", str(self.cwd)] if self.cwd else []
shell_args = ["linux32", "/bin/bash"] if self.simulate_32_bit else ["/bin/bash"]
subprocess.run(
[
"docker",
"create",
"--env=CIBUILDWHEEL",
f"--name={self.name}",
"--interactive",
"--volume=/:/host", # ignored on CircleCI
*cwd_args,
self.docker_image,
*shell_args,
],
check=True,
)
self.process = subprocess.Popen(
[
"docker",
"start",
"--attach",
"--interactive",
self.name,
],
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
)
assert self.process.stdin and self.process.stdout
self.bash_stdin = self.process.stdin
self.bash_stdout = self.process.stdout
# run a noop command to block until the container is responding
self.call(["/bin/true"])
return self
def __exit__(
self,
exc_type: Optional[Type[BaseException]],
exc_val: Optional[BaseException],
exc_tb: Optional[TracebackType],
) -> None:
self.bash_stdin.close()
self.process.terminate()
self.process.wait()
assert isinstance(self.name, str)
subprocess.run(["docker", "rm", "--force", "-v", self.name], stdout=subprocess.DEVNULL)
self.name = None
def copy_into(self, from_path: Path, to_path: PurePath) -> None:
# `docker cp` causes 'no space left on device' error when
# a container is running and the host filesystem is
# mounted. https://github.com/moby/moby/issues/38995
# Use `docker exec` instead.
if from_path.is_dir():
self.call(["mkdir", "-p", to_path])
subprocess.run(
f"tar cf - . | docker exec -i {self.name} tar -xC {shell_quote(to_path)} -f -",
shell=True,
check=True,
cwd=from_path,
)
else:
subprocess.run(
f'cat {shell_quote(from_path)} | docker exec -i {self.name} sh -c "cat > {shell_quote(to_path)}"',
shell=True,
check=True,
)
def copy_out(self, from_path: PurePath, to_path: Path) -> None:
# note: we assume from_path is a dir
to_path.mkdir(parents=True, exist_ok=True)
subprocess.run(
f"docker exec -i {self.name} tar -cC {shell_quote(from_path)} -f - . | tar -xf -",
shell=True,
check=True,
cwd=to_path,
)
def glob(self, path: PurePath, pattern: str) -> List[PurePath]:
glob_pattern = os.path.join(str(path), pattern)
path_strs = json.loads(
self.call(
[
self.UTILITY_PYTHON,
"-c",
f"import sys, json, glob; json.dump(glob.glob({glob_pattern!r}), sys.stdout)",
],
capture_output=True,
)
)
return [PurePath(p) for p in path_strs]
def call(
self,
args: Sequence[PathOrStr],
env: Optional[Dict[str, str]] = None,
capture_output: bool = False,
cwd: Optional[PathOrStr] = None,
) -> str:
chdir = f"cd {cwd}" if cwd else ""
env_assignments = (
" ".join(f"{shlex.quote(k)}={shlex.quote(v)}" for k, v in env.items())
if env is not None
else ""
)
command = " ".join(shlex.quote(str(a)) for a in args)
end_of_message = str(uuid.uuid4())
# log the command we're executing
print(f" + {command}")
# Write a command to the remote shell. First we change the
# cwd, if that's required. Then, we use the `env` utility to run
# `command` inside the specified environment. We use `env` because it
# can cope with spaces and strange characters in the name or value.
# Finally, the remote shell is told to write a footer - this will show
# up in the output so we know when to stop reading, and will include
# the returncode of `command`.
self.bash_stdin.write(
bytes(
f"""(
{chdir}
env {env_assignments} {command}
printf "%04d%s\n" $? {end_of_message}
)
""",
encoding="utf8",
errors="surrogateescape",
)
)
self.bash_stdin.flush()
if capture_output:
output_io: IO[bytes] = io.BytesIO()
else:
output_io = sys.stdout.buffer
while True:
line = self.bash_stdout.readline()
if line.endswith(bytes(end_of_message, encoding="utf8") + b"\n"):
# fmt: off
footer_offset = (
len(line)
- 1 # newline character
- len(end_of_message) # delimiter
- 4 # 4 returncode decimals
)
# fmt: on
returncode_str = line[footer_offset : footer_offset + 4]
returncode = int(returncode_str)
# add the last line to output, without the footer
output_io.write(line[0:footer_offset])
break
else:
output_io.write(line)
if isinstance(output_io, io.BytesIO):
output = str(output_io.getvalue(), encoding="utf8", errors="surrogateescape")
else:
output = ""
if returncode != 0:
raise subprocess.CalledProcessError(returncode, args, output)
return output
def get_environment(self) -> Dict[str, str]:
env = json.loads(
self.call(
[
self.UTILITY_PYTHON,
"-c",
"import sys, json, os; json.dump(os.environ.copy(), sys.stdout)",
],
capture_output=True,
)
)
return cast(Dict[str, str], env)
def environment_executor(self, command: List[str], environment: Dict[str, str]) -> str:
# used as an EnvironmentExecutor to evaluate commands and capture output
return self.call(command, env=environment, capture_output=True)
def shell_quote(path: PurePath) -> str:
return shlex.quote(str(path))
+19 -72
View File
@@ -1,11 +1,6 @@
from __future__ import annotations
import dataclasses
from collections.abc import Mapping, Sequence
from typing import Any, Protocol
from typing import Dict, List, Mapping, Optional
import bashlex
import bashlex.errors
from . import bashlex_eval
@@ -14,7 +9,7 @@ class EnvironmentParseError(Exception):
pass
def split_env_items(env_string: str) -> list[str]:
def split_env_items(env_string: str) -> List[str]:
"""Splits space-separated variable assignments into a list of individual assignments.
>>> split_env_items('VAR=abc')
@@ -33,11 +28,7 @@ def split_env_items(env_string: str) -> list[str]:
if not env_string:
return []
try:
command_node = bashlex.parsesingle(env_string)
except bashlex.errors.ParsingError as e:
raise EnvironmentParseError(env_string) from e
command_node = bashlex.parsesingle(env_string)
result = []
for word_node in command_node.parts:
@@ -47,40 +38,7 @@ def split_env_items(env_string: str) -> list[str]:
return result
class EnvironmentAssignment(Protocol):
name: str
def evaluated_value(
self,
*,
environment: Mapping[str, str],
executor: bashlex_eval.EnvironmentExecutor | None = None,
) -> str:
"""Returns the value of this assignment, as evaluated in the environment"""
class EnvironmentAssignmentRaw:
"""
An environment variable - a simple name/value pair
"""
def __init__(self, name: str, value: str):
self.name = name
self.value = value
def __repr__(self) -> str:
return f"{self.name}={self.value}"
def evaluated_value(self, **_: Any) -> str:
return self.value
class EnvironmentAssignmentBash:
"""
An environment variable, in bash syntax. The value can use bash constructs
like "$OTHER_VAR" and "$(command arg1 arg2)".
"""
class EnvironmentAssignment:
def __init__(self, assignment: str):
name, equals, value = assignment.partition("=")
if not equals:
@@ -90,33 +48,29 @@ class EnvironmentAssignmentBash:
def evaluated_value(
self,
environment: Mapping[str, str],
executor: bashlex_eval.EnvironmentExecutor | None = None,
environment: Dict[str, str],
executor: Optional[bashlex_eval.EnvironmentExecutor] = None,
) -> str:
"""Returns the value of this assignment, as evaluated in the environment"""
return bashlex_eval.evaluate(self.value, environment=environment, executor=executor)
def as_shell_assignment(self) -> str:
return f"export {self.name}={self.value}"
def __repr__(self) -> str:
return f"{self.name}={self.value}"
def __eq__(self, other: object) -> bool:
if isinstance(other, EnvironmentAssignmentBash):
return self.name == other.name and self.value == other.value
return False
@dataclasses.dataclass
class ParsedEnvironment:
assignments: list[EnvironmentAssignment]
def __init__(self, assignments: Sequence[EnvironmentAssignment]) -> None:
self.assignments = list(assignments)
def __init__(self, assignments: List[EnvironmentAssignment]):
self.assignments = assignments
def as_dictionary(
self,
prev_environment: Mapping[str, str],
executor: bashlex_eval.EnvironmentExecutor | None = None,
) -> dict[str, str]:
environment = {**prev_environment}
executor: Optional[bashlex_eval.EnvironmentExecutor] = None,
) -> Dict[str, str]:
environment = dict(**prev_environment)
for assignment in self.assignments:
value = assignment.evaluated_value(environment=environment, executor=executor)
@@ -124,21 +78,14 @@ class ParsedEnvironment:
return environment
def add(self, name: str, value: str, prepend: bool = False) -> None:
assignment = EnvironmentAssignmentRaw(name=name, value=value)
if prepend:
self.assignments.insert(0, assignment)
else:
self.assignments.append(assignment)
def as_shell_commands(self) -> List[str]:
return [a.as_shell_assignment() for a in self.assignments]
def __repr__(self) -> str:
return f"{self.__class__.__name__}({[repr(a) for a in self.assignments]!r})"
def options_summary(self) -> Any:
return self.assignments
return f"ParsedEnvironment({[repr(a) for a in self.assignments]!r})"
def parse_environment(env_string: str) -> ParsedEnvironment:
env_items = split_env_items(env_string)
assignments = [EnvironmentAssignmentBash(item) for item in env_items]
assignments = [EnvironmentAssignment(item) for item in env_items]
return ParsedEnvironment(assignments=assignments)
-66
View File
@@ -1,66 +0,0 @@
"""
Errors that can cause the build to fail. Each subclass of FatalError has
a different return code, by defining them all here, we can ensure that they're
semantically clear and unique.
"""
import textwrap
class FatalError(BaseException):
"""
Raising an error of this type will cause the message to be printed to
stderr and the process to be terminated. Within cibuildwheel, raising this
exception produces a better error message, and optional traceback.
"""
return_code: int = 1
class ConfigurationError(FatalError):
return_code = 2
class NothingToDoError(FatalError):
return_code = 3
class DeprecationError(FatalError):
return_code = 4
class NonPlatformWheelError(FatalError):
def __init__(self) -> None:
message = textwrap.dedent(
"""
Build failed because a pure Python wheel was generated.
If you intend to build a pure-Python wheel, you don't need cibuildwheel - use
`pip wheel -w DEST_DIR .` instead.
If you expected a platform wheel, check your project configuration, or run
cibuildwheel with CIBW_BUILD_VERBOSITY=1 to view build logs.
"""
)
super().__init__(message)
self.return_code = 5
class AlreadyBuiltWheelError(FatalError):
def __init__(self, wheel_name: str) -> None:
message = textwrap.dedent(
f"""
Build failed because a wheel named {wheel_name} was already generated in the current run.
If you expected another wheel to be generated, check your project configuration, or run
cibuildwheel with CIBW_BUILD_VERBOSITY=1 to view build logs.
"""
)
super().__init__(message)
self.return_code = 6
class OCIEngineTooOldError(FatalError):
def __init__(self, message: str) -> None:
super().__init__(message)
self.return_code = 7
+15 -26
View File
@@ -2,33 +2,22 @@
These are utilities for the `/bin` scripts, not for the `cibuildwheel` program.
"""
from __future__ import annotations
from typing import Any, Dict
from collections.abc import Mapping, Sequence
from io import StringIO
from typing import Protocol
__all__ = ("Printable", "dump_python_configurations")
import toml.encoder
from packaging.version import Version
class Printable(Protocol):
def __str__(self) -> str: ...
class InlineArrayDictEncoder(toml.encoder.TomlEncoder): # type: ignore
def __init__(self) -> None:
super().__init__()
self.dump_funcs[Version] = lambda v: f'"{v}"'
def dump_python_configurations(
inp: Mapping[str, Mapping[str, Sequence[Mapping[str, Printable]]]],
) -> str:
output = StringIO()
for header, values in inp.items():
output.write(f"[{header}]\n")
for inner_header, listing in values.items():
output.write(f"{inner_header} = [\n")
for item in listing:
output.write(" { ")
dict_contents = (f'{key} = "{value}"' for key, value in item.items())
output.write(", ".join(dict_contents))
output.write(" },\n")
output.write("]\n")
output.write("\n")
# Strip the final newline, to avoid two blank lines at the end.
return output.getvalue()[:-1]
def dump_sections(self, o: Dict[str, Any], sup: str) -> Any:
if not all(isinstance(a, list) for a in o.values()):
return super().dump_sections(o, sup)
val = ""
for k, v in o.items():
inner = ",\n ".join(self.dump_inline_table(d_i).strip() for d_i in v)
val += f"{k} = [\n {inner},\n]\n"
return val, self._dict()
+263 -450
View File
@@ -1,67 +1,39 @@
from __future__ import annotations
import subprocess
import sys
import textwrap
from collections.abc import Iterable, Iterator, Sequence, Set
from dataclasses import dataclass
from pathlib import Path, PurePath, PurePosixPath
from typing import OrderedDict, Tuple
from pathlib import Path, PurePath
from typing import List, NamedTuple, Set
from packaging.version import Version
from . import errors
from ._compat.typing import assert_never
from .architecture import Architecture
from .docker_container import DockerContainer
from .logger import log
from .oci_container import OCIContainer, OCIContainerEngineConfig, OCIPlatform
from .options import BuildOptions, Options
from .typing import PathOrStr
from .util import (
BuildFrontendConfig,
BuildOptions,
BuildSelector,
find_compatible_wheel,
NonPlatformWheelError,
get_build_verbosity_extra_flags,
prepare_command,
read_python_configs,
split_config_settings,
test_fail_cwd_file,
unwrap,
resources_dir,
)
ARCHITECTURE_OCI_PLATFORM_MAP = {
Architecture.x86_64: OCIPlatform.AMD64,
Architecture.i686: OCIPlatform.i386,
Architecture.aarch64: OCIPlatform.ARM64,
Architecture.ppc64le: OCIPlatform.PPC64LE,
Architecture.s390x: OCIPlatform.S390X,
Architecture.armv7l: OCIPlatform.ARMV7,
}
@dataclass(frozen=True)
class PythonConfiguration:
class PythonConfiguration(NamedTuple):
version: str
identifier: str
path_str: str
@property
def path(self) -> PurePosixPath:
return PurePosixPath(self.path_str)
@dataclass(frozen=True)
class BuildStep:
platform_configs: list[PythonConfiguration]
platform_tag: str
container_engine: OCIContainerEngineConfig
container_image: str
def path(self) -> PurePath:
return PurePath(self.path_str)
def get_python_configurations(
build_selector: BuildSelector,
architectures: Set[Architecture],
) -> list[PythonConfiguration]:
) -> List[PythonConfiguration]:
full_python_configs = read_python_configs("linux")
python_configurations = [PythonConfiguration(**item) for item in full_python_configs]
@@ -76,424 +48,272 @@ def get_python_configurations(
]
def container_image_for_python_configuration(
config: PythonConfiguration, build_options: BuildOptions
) -> str:
# e.g
# identifier is 'cp310-manylinux_x86_64'
# platform_tag is 'manylinux_x86_64'
# platform_arch is 'x86_64'
_, platform_tag = config.identifier.split("-", 1)
_, platform_arch = platform_tag.split("_", 1)
assert build_options.manylinux_images is not None
assert build_options.musllinux_images is not None
return (
build_options.manylinux_images[platform_arch]
if platform_tag.startswith("manylinux")
else build_options.musllinux_images[platform_arch]
)
def get_build_steps(
options: Options, python_configurations: list[PythonConfiguration]
) -> Iterator[BuildStep]:
"""
Groups PythonConfigurations into BuildSteps. Each BuildStep represents a
separate container instance.
"""
steps = OrderedDict[Tuple[str, str, str, OCIContainerEngineConfig], BuildStep]()
for config in python_configurations:
_, platform_tag = config.identifier.split("-", 1)
build_options = options.build_options(config.identifier)
before_all = build_options.before_all
container_image = container_image_for_python_configuration(config, build_options)
container_engine = build_options.container_engine
step_key = (platform_tag, container_image, before_all, container_engine)
if step_key in steps:
steps[step_key].platform_configs.append(config)
else:
steps[step_key] = BuildStep(
platform_configs=[config],
platform_tag=platform_tag,
container_engine=container_engine,
container_image=container_image,
)
yield from steps.values()
def check_all_python_exist(
*, platform_configs: Iterable[PythonConfiguration], container: OCIContainer
) -> None:
exist = True
has_manylinux_interpreters = True
messages = []
def build(options: BuildOptions) -> None:
try:
# use capture_output to keep quiet
container.call(["manylinux-interpreters", "--help"], capture_output=True)
except subprocess.CalledProcessError:
has_manylinux_interpreters = False
for config in platform_configs:
python_path = config.path / "bin" / "python"
try:
if has_manylinux_interpreters:
container.call(["manylinux-interpreters", "ensure", config.path.name])
container.call(["test", "-x", python_path])
except subprocess.CalledProcessError:
messages.append(
f" '{python_path}' executable doesn't exist in image '{container.image}' to build '{config.identifier}'."
)
exist = False
if not exist:
message = "\n".join(messages)
raise errors.FatalError(message)
def build_in_container(
*,
options: Options,
platform_configs: Sequence[PythonConfiguration],
container: OCIContainer,
container_project_path: PurePath,
container_package_dir: PurePath,
) -> None:
container_output_dir = PurePosixPath("/output")
check_all_python_exist(platform_configs=platform_configs, container=container)
log.step("Copying project into container...")
container.copy_into(Path.cwd(), container_project_path)
before_all_options_identifier = platform_configs[0].identifier
before_all_options = options.build_options(before_all_options_identifier)
if before_all_options.before_all:
log.step("Running before_all...")
env = container.get_environment()
env["PATH"] = f'/opt/python/cp38-cp38/bin:{env["PATH"]}'
env["PIP_DISABLE_PIP_VERSION_CHECK"] = "1"
env["PIP_ROOT_USER_ACTION"] = "ignore"
env = before_all_options.environment.as_dictionary(
env, executor=container.environment_executor
# check docker is installed
subprocess.run(["docker", "--version"], check=True, stdout=subprocess.DEVNULL)
except Exception:
print(
"cibuildwheel: Docker not found. Docker is required to run Linux builds. "
"If you're building on Travis CI, add `services: [docker]` to your .travis.yml."
"If you're building on Circle CI in Linux, add a `setup_remote_docker` step to your .circleci/config.yml",
file=sys.stderr,
)
sys.exit(2)
before_all_prepared = prepare_command(
before_all_options.before_all,
project=container_project_path,
package=container_package_dir,
)
container.call(["sh", "-c", before_all_prepared], env=env)
built_wheels: list[PurePosixPath] = []
for config in platform_configs:
log.build_start(config.identifier)
build_options = options.build_options(config.identifier)
build_frontend = build_options.build_frontend or BuildFrontendConfig("pip")
use_uv = build_frontend.name == "build[uv]" and Version(config.version) >= Version("3.8")
pip = ["uv", "pip"] if use_uv else ["pip"]
dependency_constraint_flags: list[PathOrStr] = []
log.step("Setting up build environment...")
if build_options.dependency_constraints:
constraints_file = build_options.dependency_constraints.get_for_python_version(
config.version
)
container_constraints_file = PurePosixPath("/constraints.txt")
container.copy_into(constraints_file, container_constraints_file)
dependency_constraint_flags = ["-c", container_constraints_file]
env = container.get_environment()
env["PIP_DISABLE_PIP_VERSION_CHECK"] = "1"
env["PIP_ROOT_USER_ACTION"] = "ignore"
# put this config's python top of the list
python_bin = config.path / "bin"
env["PATH"] = f'{python_bin}:{env["PATH"]}'
env = build_options.environment.as_dictionary(env, executor=container.environment_executor)
# check config python is still on PATH
which_python = container.call(["which", "python"], env=env, capture_output=True).strip()
if PurePosixPath(which_python) != python_bin / "python":
msg = "python available on PATH doesn't match our installed instance. If you have modified PATH, ensure that you don't overwrite cibuildwheel's entry or insert python above it."
raise errors.FatalError(msg)
if use_uv:
which_uv = container.call(["which", "uv"], env=env, capture_output=True).strip()
if not which_uv:
msg = "uv not found on PATH. You must use a supported manylinux or musllinux environment with uv."
raise errors.FatalError(msg)
else:
which_pip = container.call(["which", "pip"], env=env, capture_output=True).strip()
if PurePosixPath(which_pip) != python_bin / "pip":
msg = "pip available on PATH doesn't match our installed instance. If you have modified PATH, ensure that you don't overwrite cibuildwheel's entry or insert pip above it."
raise errors.FatalError(msg)
compatible_wheel = find_compatible_wheel(built_wheels, config.identifier)
if compatible_wheel:
log.step_end()
print(
f"\nFound previously built wheel {compatible_wheel.name}, that's compatible with {config.identifier}. Skipping build step..."
)
repaired_wheels = [compatible_wheel]
else:
if build_options.before_build:
log.step("Running before_build...")
before_build_prepared = prepare_command(
build_options.before_build,
project=container_project_path,
package=container_package_dir,
)
container.call(["sh", "-c", before_build_prepared], env=env)
log.step("Building wheel...")
temp_dir = PurePosixPath("/tmp/cibuildwheel")
built_wheel_dir = temp_dir / "built_wheel"
container.call(["rm", "-rf", built_wheel_dir])
container.call(["mkdir", "-p", built_wheel_dir])
extra_flags = split_config_settings(build_options.config_settings, build_frontend.name)
extra_flags += build_frontend.args
if build_frontend.name == "pip":
extra_flags += get_build_verbosity_extra_flags(build_options.build_verbosity)
container.call(
[
"python",
"-m",
"pip",
"wheel",
container_package_dir,
f"--wheel-dir={built_wheel_dir}",
"--no-deps",
*extra_flags,
],
env=env,
)
elif build_frontend.name == "build" or build_frontend.name == "build[uv]":
if not 0 <= build_options.build_verbosity < 2:
msg = f"build_verbosity {build_options.build_verbosity} is not supported for build frontend. Ignoring."
log.warning(msg)
if use_uv and "--no-isolation" not in extra_flags and "-n" not in extra_flags:
extra_flags += ["--installer=uv"]
container.call(
[
"python",
"-m",
"build",
container_package_dir,
"--wheel",
f"--outdir={built_wheel_dir}",
*extra_flags,
],
env=env,
)
else:
assert_never(build_frontend)
built_wheel = container.glob(built_wheel_dir, "*.whl")[0]
repaired_wheel_dir = temp_dir / "repaired_wheel"
container.call(["rm", "-rf", repaired_wheel_dir])
container.call(["mkdir", "-p", repaired_wheel_dir])
if built_wheel.name.endswith("none-any.whl"):
raise errors.NonPlatformWheelError()
if build_options.repair_command:
log.step("Repairing wheel...")
repair_command_prepared = prepare_command(
build_options.repair_command, wheel=built_wheel, dest_dir=repaired_wheel_dir
)
container.call(["sh", "-c", repair_command_prepared], env=env)
else:
container.call(["mv", built_wheel, repaired_wheel_dir])
repaired_wheels = container.glob(repaired_wheel_dir, "*.whl")
for repaired_wheel in repaired_wheels:
if repaired_wheel.name in {wheel.name for wheel in built_wheels}:
raise errors.AlreadyBuiltWheelError(repaired_wheel.name)
if build_options.test_command and build_options.test_selector(config.identifier):
log.step("Testing wheel...")
# set up a virtual environment to install and test from, to make sure
# there are no dependencies that were pulled in at build time.
if not use_uv:
container.call(
["pip", "install", "virtualenv", *dependency_constraint_flags], env=env
)
testing_temp_dir = PurePosixPath(
container.call(["mktemp", "-d"], capture_output=True).strip()
)
venv_dir = testing_temp_dir / "venv"
if use_uv:
container.call(["uv", "venv", venv_dir, "--python", python_bin / "python"], env=env)
else:
# Use embedded dependencies from virtualenv to ensure determinism
venv_args = ["--no-periodic-update", "--pip=embed"]
# In Python<3.12, setuptools & wheel are installed as well
if Version(config.version) < Version("3.12"):
venv_args.extend(("--setuptools=embed", "--wheel=embed"))
container.call(["python", "-m", "virtualenv", *venv_args, venv_dir], env=env)
virtualenv_env = env.copy()
virtualenv_env["PATH"] = f"{venv_dir / 'bin'}:{virtualenv_env['PATH']}"
virtualenv_env["VIRTUAL_ENV"] = str(venv_dir)
if build_options.before_test:
before_test_prepared = prepare_command(
build_options.before_test,
project=container_project_path,
package=container_package_dir,
)
container.call(["sh", "-c", before_test_prepared], env=virtualenv_env)
# Install the wheel we just built
# Note: If auditwheel produced two wheels, it's because the earlier produced wheel
# conforms to multiple manylinux standards. These multiple versions of the wheel are
# functionally the same, differing only in name, wheel metadata, and possibly include
# different external shared libraries. so it doesn't matter which one we run the tests on.
# Let's just pick the first one.
wheel_to_test = repaired_wheels[0]
container.call(
[*pip, "install", str(wheel_to_test) + build_options.test_extras],
env=virtualenv_env,
)
# Install any requirements to run the tests
if build_options.test_requires:
container.call([*pip, "install", *build_options.test_requires], env=virtualenv_env)
# Run the tests from a different directory
test_command_prepared = prepare_command(
build_options.test_command,
project=container_project_path,
package=container_package_dir,
wheel=wheel_to_test,
)
test_cwd = testing_temp_dir / "test_cwd"
container.call(["mkdir", "-p", test_cwd])
container.copy_into(test_fail_cwd_file, test_cwd / "test_fail.py")
container.call(["sh", "-c", test_command_prepared], cwd=test_cwd, env=virtualenv_env)
# clean up test environment
container.call(["rm", "-rf", testing_temp_dir])
# move repaired wheels to output
if compatible_wheel is None:
container.call(["mkdir", "-p", container_output_dir])
container.call(["mv", *repaired_wheels, container_output_dir])
built_wheels.extend(
container_output_dir / repaired_wheel.name for repaired_wheel in repaired_wheels
)
log.build_end()
log.step("Copying wheels back to host...")
# copy the output back into the host
container.copy_out(container_output_dir, options.globals.output_dir)
log.step_end()
def build(options: Options, tmp_path: Path) -> None: # noqa: ARG001
python_configurations = get_python_configurations(
options.globals.build_selector, options.globals.architectures
)
assert options.manylinux_images is not None
python_configurations = get_python_configurations(options.build_selector, options.architectures)
platforms = [
("cp", "manylinux_x86_64", options.manylinux_images["x86_64"]),
("cp", "manylinux_i686", options.manylinux_images["i686"]),
("cp", "manylinux_aarch64", options.manylinux_images["aarch64"]),
("cp", "manylinux_ppc64le", options.manylinux_images["ppc64le"]),
("cp", "manylinux_s390x", options.manylinux_images["s390x"]),
("pp", "manylinux_x86_64", options.manylinux_images["pypy_x86_64"]),
]
cwd = Path.cwd()
abs_package_dir = options.globals.package_dir.resolve()
abs_package_dir = options.package_dir.resolve()
if cwd != abs_package_dir and cwd not in abs_package_dir.parents:
msg = "package_dir must be inside the working directory"
raise Exception(msg)
raise Exception("package_dir must be inside the working directory")
container_project_path = PurePosixPath("/project")
container_project_path = PurePath("/project")
container_package_dir = container_project_path / abs_package_dir.relative_to(cwd)
container_output_dir = PurePath("/output")
for build_step in get_build_steps(options, python_configurations):
try:
# check the container engine is installed
subprocess.run(
[build_step.container_engine.name, "--version"],
check=True,
stdout=subprocess.DEVNULL,
)
except subprocess.CalledProcessError as error:
msg = unwrap(
f"""
cibuildwheel: {build_step.container_engine.name} not found. An
OCI exe like Docker or Podman is required to run Linux builds.
If you're building on Travis CI, add `services: [docker]` to
your .travis.yml. If you're building on Circle CI in Linux,
add a `setup_remote_docker` step to your .circleci/config.yml.
If you're building on Cirrus CI, use `docker_builder` task.
"""
)
raise errors.ConfigurationError(msg) from error
for implementation, platform_tag, docker_image in platforms:
platform_configs = [
c
for c in python_configurations
if c.identifier.startswith(implementation) and c.identifier.endswith(platform_tag)
]
if not platform_configs:
continue
try:
ids_to_build = [x.identifier for x in build_step.platform_configs]
log.step(f"Starting container image {build_step.container_image}...")
print(f"info: This container will host the build for {', '.join(ids_to_build)}...")
architecture = Architecture(build_step.platform_tag.split("_", 1)[1])
with OCIContainer(
image=build_step.container_image,
oci_platform=ARCHITECTURE_OCI_PLATFORM_MAP[architecture],
log.step(f"Starting Docker image {docker_image}...")
with DockerContainer(
docker_image,
simulate_32_bit=platform_tag.endswith("i686"),
cwd=container_project_path,
engine=build_step.container_engine,
) as container:
build_in_container(
options=options,
platform_configs=build_step.platform_configs,
container=container,
container_project_path=container_project_path,
container_package_dir=container_package_dir,
)
) as docker:
log.step("Copying project into Docker...")
docker.copy_into(Path.cwd(), container_project_path)
if options.before_all:
log.step("Running before_all...")
env = docker.get_environment()
env["PATH"] = f'/opt/python/cp38-cp38/bin:{env["PATH"]}'
env["PIP_DISABLE_PIP_VERSION_CHECK"] = "1"
env = options.environment.as_dictionary(
env, executor=docker.environment_executor
)
before_all_prepared = prepare_command(
options.before_all,
project=container_project_path,
package=container_package_dir,
)
docker.call(["sh", "-c", before_all_prepared], env=env)
for config in platform_configs:
log.build_start(config.identifier)
dependency_constraint_flags: List[PathOrStr] = []
if config.identifier.startswith("pp"):
# Patch PyPy to make sure headers get installed into a venv
patch_version = "_27" if config.version == "2.7" else ""
patch_path = resources_dir / f"pypy_venv{patch_version}.patch"
patch_docker_path = PurePath("/pypy_venv.patch")
docker.copy_into(patch_path, patch_docker_path)
try:
docker.call(
[
"patch",
"--force",
"-p1",
"-d",
config.path,
"-i",
patch_docker_path,
]
)
except subprocess.CalledProcessError:
print("PyPy patch not applied", file=sys.stderr)
if options.dependency_constraints:
constraints_file = options.dependency_constraints.get_for_python_version(
config.version
)
container_constraints_file = PurePath("/constraints.txt")
docker.copy_into(constraints_file, container_constraints_file)
dependency_constraint_flags = ["-c", container_constraints_file]
log.step("Setting up build environment...")
env = docker.get_environment()
# put this config's python top of the list
python_bin = config.path / "bin"
env["PATH"] = f'{python_bin}:{env["PATH"]}'
env = options.environment.as_dictionary(
env, executor=docker.environment_executor
)
# check config python and pip are still on PATH
which_python = docker.call(
["which", "python"], env=env, capture_output=True
).strip()
if PurePath(which_python) != python_bin / "python":
print(
"cibuildwheel: python available on PATH doesn't match our installed instance. If you have modified PATH, ensure that you don't overwrite cibuildwheel's entry or insert python above it.",
file=sys.stderr,
)
sys.exit(1)
which_pip = docker.call(["which", "pip"], env=env, capture_output=True).strip()
if PurePath(which_pip) != python_bin / "pip":
print(
"cibuildwheel: pip available on PATH doesn't match our installed instance. If you have modified PATH, ensure that you don't overwrite cibuildwheel's entry or insert pip above it.",
file=sys.stderr,
)
sys.exit(1)
if options.before_build:
log.step("Running before_build...")
before_build_prepared = prepare_command(
options.before_build,
project=container_project_path,
package=container_package_dir,
)
docker.call(["sh", "-c", before_build_prepared], env=env)
log.step("Building wheel...")
temp_dir = PurePath("/tmp/cibuildwheel")
built_wheel_dir = temp_dir / "built_wheel"
docker.call(["rm", "-rf", built_wheel_dir])
docker.call(["mkdir", "-p", built_wheel_dir])
docker.call(
[
"pip",
"wheel",
container_package_dir,
"--wheel-dir",
built_wheel_dir,
"--no-deps",
*get_build_verbosity_extra_flags(options.build_verbosity),
],
env=env,
)
built_wheel = docker.glob(built_wheel_dir, "*.whl")[0]
repaired_wheel_dir = temp_dir / "repaired_wheel"
docker.call(["rm", "-rf", repaired_wheel_dir])
docker.call(["mkdir", "-p", repaired_wheel_dir])
if built_wheel.name.endswith("none-any.whl"):
raise NonPlatformWheelError()
if options.repair_command:
log.step("Repairing wheel...")
repair_command_prepared = prepare_command(
options.repair_command, wheel=built_wheel, dest_dir=repaired_wheel_dir
)
docker.call(["sh", "-c", repair_command_prepared], env=env)
else:
docker.call(["mv", built_wheel, repaired_wheel_dir])
repaired_wheels = docker.glob(repaired_wheel_dir, "*.whl")
if options.test_command and options.test_selector(config.identifier):
log.step("Testing wheel...")
# set up a virtual environment to install and test from, to make sure
# there are no dependencies that were pulled in at build time.
docker.call(
["pip", "install", "virtualenv", *dependency_constraint_flags], env=env
)
venv_dir = (
PurePath(docker.call(["mktemp", "-d"], capture_output=True).strip())
/ "venv"
)
docker.call(
["python", "-m", "virtualenv", "--no-download", venv_dir], env=env
)
virtualenv_env = env.copy()
virtualenv_env["PATH"] = f"{venv_dir / 'bin'}:{virtualenv_env['PATH']}"
if options.before_test:
before_test_prepared = prepare_command(
options.before_test,
project=container_project_path,
package=container_package_dir,
)
docker.call(["sh", "-c", before_test_prepared], env=virtualenv_env)
# Install the wheel we just built
# Note: If auditwheel produced two wheels, it's because the earlier produced wheel
# conforms to multiple manylinux standards. These multiple versions of the wheel are
# functionally the same, differing only in name, wheel metadata, and possibly include
# different external shared libraries. so it doesn't matter which one we run the tests on.
# Let's just pick the first one.
wheel_to_test = repaired_wheels[0]
docker.call(
["pip", "install", str(wheel_to_test) + options.test_extras],
env=virtualenv_env,
)
# Install any requirements to run the tests
if options.test_requires:
docker.call(
["pip", "install", *options.test_requires], env=virtualenv_env
)
# Run the tests from a different directory
test_command_prepared = prepare_command(
options.test_command,
project=container_project_path,
package=container_package_dir,
)
docker.call(
["sh", "-c", test_command_prepared], cwd="/root", env=virtualenv_env
)
# clean up test environment
docker.call(["rm", "-rf", venv_dir])
# move repaired wheels to output
docker.call(["mkdir", "-p", container_output_dir])
docker.call(["mv", *repaired_wheels, container_output_dir])
log.build_end()
log.step("Copying wheels back to host...")
# copy the output back into the host
docker.copy_out(container_output_dir, options.output_dir)
log.step_end()
except subprocess.CalledProcessError as error:
troubleshoot(options, error)
msg = f"Command {error.cmd} failed with code {error.returncode}. {error.stdout or ''}"
raise errors.FatalError(msg) from error
log.step_end_with_error(
f"Command {error.cmd} failed with code {error.returncode}. {error.stdout}"
)
troubleshoot(options.package_dir, error)
sys.exit(1)
def _matches_prepared_command(error_cmd: Sequence[str], command_template: str) -> bool:
if len(error_cmd) < 3 or error_cmd[0:2] != ["sh", "-c"]:
return False
command_prefix = command_template.split("{", maxsplit=1)[0].strip()
return error_cmd[2].startswith(command_prefix)
def troubleshoot(options: Options, error: Exception) -> None:
if isinstance(error, subprocess.CalledProcessError) and (
error.cmd[0:4] == ["python", "-m", "pip", "wheel"]
or error.cmd[0:3] == ["python", "-m", "build"]
or _matches_prepared_command(
error.cmd, options.build_options(None).repair_command
) # TODO allow matching of overrides too?
):
# the wheel build step or the repair step failed
so_files = list(options.globals.package_dir.glob("**/*.so"))
def troubleshoot(package_dir: Path, error: Exception) -> None:
if isinstance(error, subprocess.CalledProcessError) and error.cmd[0:2] == ["pip", "wheel"]:
# the 'pip wheel' step failed.
print("Checking for common errors...")
so_files = list(package_dir.glob("**/*.so"))
if so_files:
print(
@@ -502,17 +322,10 @@ def troubleshoot(options: Options, error: Exception) -> None:
NOTE: Shared object (.so) files found in this project.
These files might be built against the wrong OS, causing problems with
auditwheel. If possible, run cibuildwheel in a clean checkout.
auditwheel.
If you're using Cython and have previously done an in-place build,
remove those build files (*.so and *.c) before starting cibuildwheel.
setuptools uses the build/ folder to store its build cache. It
may be necessary to remove those build files (*.so and *.o) before
starting cibuildwheel.
Files that belong to a virtual environment are probably not an issue
unless you used a custom command telling cibuildwheel to activate it.
"""
),
file=sys.stderr,
@@ -520,4 +333,4 @@ def troubleshoot(options: Options, error: Exception) -> None:
print(" Files detected:")
print("\n".join(f" {f}" for f in so_files))
print()
print("")
+30 -52
View File
@@ -1,41 +1,30 @@
from __future__ import annotations
import codecs
import os
import re
import sys
import time
from typing import IO, AnyStr, Final, Tuple
from typing import IO, AnyStr, Optional, Union
from .util import CIProvider, detect_ci_provider
from cibuildwheel.util import CIProvider, detect_ci_provider
FoldPattern = Tuple[str, str]
DEFAULT_FOLD_PATTERN: Final[FoldPattern] = ("{name}", "")
FOLD_PATTERNS: Final[dict[str, FoldPattern]] = {
DEFAULT_FOLD_PATTERN = ("{name}", "")
FOLD_PATTERNS = {
"azure": ("##[group]{name}", "##[endgroup]"),
"travis": ("travis_fold:start:{identifier}\n{name}", "travis_fold:end:{identifier}"),
"github": ("::group::{name}", "::endgroup::{name}"),
}
PLATFORM_IDENTIFIER_DESCRIPTIONS: Final[dict[str, str]] = {
PLATFORM_IDENTIFIER_DESCIPTIONS = {
"manylinux_x86_64": "manylinux x86_64",
"manylinux_i686": "manylinux i686",
"manylinux_aarch64": "manylinux aarch64",
"manylinux_ppc64le": "manylinux ppc64le",
"manylinux_s390x": "manylinux s390x",
"musllinux_x86_64": "musllinux x86_64",
"musllinux_i686": "musllinux i686",
"musllinux_aarch64": "musllinux aarch64",
"musllinux_ppc64le": "musllinux ppc64le",
"musllinux_s390x": "musllinux s390x",
"musllinux_armv7l": "musllinux armv7l",
"win32": "Windows 32bit",
"win_amd64": "Windows 64bit",
"win_arm64": "Windows on ARM 64bit",
"macosx_x86_64": "macOS x86_64",
"macosx_universal2": "macOS Universal 2 - x86_64 and arm64",
"macosx_arm64": "macOS arm64 - Apple Silicon",
"pyodide_wasm32": "Pyodide",
}
@@ -43,10 +32,10 @@ class Logger:
fold_mode: str
colors_enabled: bool
unicode_enabled: bool
active_build_identifier: str | None = None
build_start_time: float | None = None
step_start_time: float | None = None
active_fold_group_name: str | None = None
active_build_identifier: Optional[str] = None
build_start_time: Optional[float] = None
step_start_time: Optional[float] = None
active_fold_group_name: Optional[str] = None
def __init__(self) -> None:
if sys.platform == "win32" and hasattr(sys.stdout, "reconfigure"):
@@ -124,21 +113,10 @@ class Logger:
self.step_start_time = None
def step_end_with_error(self, error: BaseException | str) -> None:
def step_end_with_error(self, error: Union[BaseException, str]) -> None:
self.step_end(success=False)
self.error(error)
def quiet(self, message: str) -> None:
c = self.colors
print(f"{c.gray}{message}{c.end}", file=sys.stderr)
def notice(self, message: str) -> None:
if self.fold_mode == "github":
print(f"::notice::{message}\n", file=sys.stderr)
else:
c = self.colors
print(f"{c.bold}Note{c.end}: {message}\n", file=sys.stderr)
def warning(self, message: str) -> None:
if self.fold_mode == "github":
print(f"::warning::{message}\n", file=sys.stderr)
@@ -146,17 +124,13 @@ class Logger:
c = self.colors
print(f"{c.yellow}Warning{c.end}: {message}\n", file=sys.stderr)
def error(self, error: BaseException | str) -> None:
def error(self, error: Union[BaseException, str]) -> None:
if self.fold_mode == "github":
print(f"::error::{error}\n", file=sys.stderr)
else:
c = self.colors
print(f"{c.bright_red}Error{c.end}: {error}\n", file=sys.stderr)
@property
def step_active(self) -> bool:
return self.step_start_time is not None
def _start_fold_group(self, name: str) -> None:
self._end_fold_group()
self.active_fold_group_name = name
@@ -177,8 +151,7 @@ class Logger:
sys.stdout.flush()
self.active_fold_group_name = None
@staticmethod
def _fold_group_identifier(name: str) -> str:
def _fold_group_identifier(self, name: str) -> str:
"""
Travis doesn't like fold groups identifiers that have spaces in. This
method converts them to ascii identifiers
@@ -193,12 +166,18 @@ class Logger:
return identifier.lower()[:20]
@property
def colors(self) -> Colors:
return Colors(enabled=self.colors_enabled)
def colors(self) -> "Colors":
if self.colors_enabled:
return Colors(enabled=True)
else:
return Colors(enabled=False)
@property
def symbols(self) -> Symbols:
return Symbols(unicode=self.unicode_enabled)
def symbols(self) -> "Symbols":
if self.unicode_enabled:
return Symbols(unicode=True)
else:
return Symbols(unicode=False)
def build_description_from_identifier(identifier: str) -> str:
@@ -207,23 +186,21 @@ def build_description_from_identifier(identifier: str) -> str:
build_description = ""
python_interpreter = python_identifier[0:2]
python_version = python_identifier[2:]
python_version = python_identifier[2:4]
if python_interpreter == "cp":
build_description += "CPython"
elif python_interpreter == "pp":
build_description += "PyPy"
else:
msg = f"unknown python {python_interpreter!r}"
raise Exception(msg)
raise Exception("unknown python")
build_description += f" {python_version[0]}.{python_version[1:]} "
build_description += f" {python_version[0]}.{python_version[1]} "
try:
build_description += PLATFORM_IDENTIFIER_DESCRIPTIONS[platform_identifier]
build_description += PLATFORM_IDENTIFIER_DESCIPTIONS[platform_identifier]
except KeyError as e:
msg = f"unknown platform {platform_identifier!r}"
raise Exception(msg) from e
raise Exception("unknown platform") from e
return build_description
@@ -238,7 +215,6 @@ class Colors:
self.bright_red = "\033[91m" if enabled else ""
self.bright_green = "\033[92m" if enabled else ""
self.white = "\033[37m\033[97m" if enabled else ""
self.gray = "\033[38;5;244m" if enabled else ""
self.bg_grey = "\033[48;5;235m" if enabled else ""
@@ -280,6 +256,8 @@ def file_supports_unicode(file_obj: IO[AnyStr]) -> bool:
return "utf" in codec_info.name
# Global instance of the Logger.
"""
Global instance of the Logger.
"""
# (there's only one stdout per-process, so a global instance is justified)
log = Logger()
+321 -492
View File
File diff suppressed because it is too large Load Diff
-515
View File
@@ -1,515 +0,0 @@
from __future__ import annotations
import io
import json
import os
import platform
import shlex
import shutil
import subprocess
import sys
import textwrap
import typing
import uuid
from collections.abc import Mapping, Sequence
from dataclasses import dataclass, field
from enum import Enum
from pathlib import Path, PurePath, PurePosixPath
from types import TracebackType
from typing import IO, Dict, Literal
from ._compat.typing import Self, assert_never
from .errors import OCIEngineTooOldError
from .logger import log
from .typing import PathOrStr, PopenBytes
from .util import (
CIProvider,
FlexibleVersion,
call,
detect_ci_provider,
parse_key_value_string,
strtobool,
)
ContainerEngineName = Literal["docker", "podman"]
# Order of the enum matters for tests. 386 shall appear before amd64.
class OCIPlatform(Enum):
i386 = "linux/386"
AMD64 = "linux/amd64"
ARMV7 = "linux/arm/v7"
ARM64 = "linux/arm64"
PPC64LE = "linux/ppc64le"
S390X = "linux/s390x"
@dataclass(frozen=True)
class OCIContainerEngineConfig:
name: ContainerEngineName
create_args: tuple[str, ...] = field(default_factory=tuple)
disable_host_mount: bool = False
@staticmethod
def from_config_string(config_string: str) -> OCIContainerEngineConfig:
config_dict = parse_key_value_string(
config_string,
["name"],
["create_args", "create-args", "disable_host_mount", "disable-host-mount"],
)
name = " ".join(config_dict["name"])
if name not in {"docker", "podman"}:
msg = f"unknown container engine {name}"
raise ValueError(msg)
name = typing.cast(ContainerEngineName, name)
# some flexibility in the option names to cope with TOML conventions
create_args = config_dict.get("create_args") or config_dict.get("create-args") or []
disable_host_mount_options = (
config_dict.get("disable_host_mount") or config_dict.get("disable-host-mount") or []
)
disable_host_mount = (
strtobool(disable_host_mount_options[-1]) if disable_host_mount_options else False
)
if "--platform" in create_args or any(arg.startswith("--platform=") for arg in create_args):
msg = "Using '--platform' in 'container-engine::create_args' is deprecated. It will be ignored."
log.warning(msg)
if "--platform" in create_args:
index = create_args.index("--platform")
create_args.pop(index)
create_args.pop(index)
else:
create_args = [arg for arg in create_args if not arg.startswith("--platform=")]
return OCIContainerEngineConfig(
name=name, create_args=tuple(create_args), disable_host_mount=disable_host_mount
)
def options_summary(self) -> str | dict[str, str]:
if not self.create_args:
return self.name
else:
return {
"name": self.name,
"create_args": repr(self.create_args),
"disable_host_mount": str(self.disable_host_mount),
}
DEFAULT_ENGINE = OCIContainerEngineConfig("docker")
def _check_engine_version(engine: OCIContainerEngineConfig) -> None:
try:
version_string = call(engine.name, "version", "-f", "{{json .}}", capture_stdout=True)
version_info = json.loads(version_string.strip())
if engine.name == "docker":
client_api_version = FlexibleVersion(version_info["Client"]["ApiVersion"])
server_api_version = FlexibleVersion(version_info["Server"]["ApiVersion"])
# --platform support was introduced in 1.32 as experimental, 1.41 removed the experimental flag
version = min(client_api_version, server_api_version)
minimum_version = FlexibleVersion("1.41")
minimum_version_str = "20.10.0" # docker version
error_msg = textwrap.dedent(
f"""
Build failed because {engine.name} is too old.
cibuildwheel requires {engine.name}>={minimum_version_str} running API version {minimum_version}.
The API version found by cibuildwheel is {version}.
"""
)
elif engine.name == "podman":
# podman uses the same version string for "Version" & "ApiVersion"
client_version = FlexibleVersion(version_info["Client"]["Version"])
if "Server" in version_info:
server_version = FlexibleVersion(version_info["Server"]["Version"])
else:
server_version = client_version
# --platform support was introduced in v3
version = min(client_version, server_version)
minimum_version = FlexibleVersion("3")
error_msg = textwrap.dedent(
f"""
Build failed because {engine.name} is too old.
cibuildwheel requires {engine.name}>={minimum_version}.
The version found by cibuildwheel is {version}.
"""
)
else:
assert_never(engine.name)
if version < minimum_version:
raise OCIEngineTooOldError(error_msg) from None
except (subprocess.CalledProcessError, KeyError, ValueError) as e:
msg = f"Build failed because {engine.name} is too old or is not working properly."
raise OCIEngineTooOldError(msg) from e
class OCIContainer:
"""
An object that represents a running OCI (e.g. Docker) container.
Intended for use as a context manager e.g.
`with OCIContainer(image = 'ubuntu') as docker:`
A bash shell is running in the remote container. When `call()` is invoked,
the command is relayed to the remote shell, and the results are streamed
back to cibuildwheel.
Example:
>>> from cibuildwheel.oci_container import * # NOQA
>>> from cibuildwheel.options import _get_pinned_container_images
>>> image = _get_pinned_container_images()['x86_64']['manylinux2014']
>>> # Test the default container
>>> with OCIContainer(image=image) as self:
... self.call(["echo", "hello world"])
... self.call(["cat", "/proc/1/cgroup"])
... print(self.get_environment())
... print(self.debug_info())
"""
UTILITY_PYTHON = "/opt/python/cp38-cp38/bin/python"
process: PopenBytes
bash_stdin: IO[bytes]
bash_stdout: IO[bytes]
def __init__(
self,
*,
image: str,
oci_platform: OCIPlatform,
cwd: PathOrStr | None = None,
engine: OCIContainerEngineConfig = DEFAULT_ENGINE,
):
if not image:
msg = "Must have a non-empty image to run."
raise ValueError(msg)
self.image = image
self.oci_platform = oci_platform
self.cwd = cwd
self.name: str | None = None
self.engine = engine
self.host_tar_format = ""
if sys.platform.startswith("darwin"):
self.host_tar_format = "--format gnutar"
def _get_platform_args(self, *, oci_platform: OCIPlatform | None = None) -> tuple[str, str]:
if oci_platform is None:
oci_platform = self.oci_platform
# we need '--pull=always' otherwise some images with the wrong platform get re-used (e.g. 386 image for amd64)
# c.f. https://github.com/moby/moby/issues/48197#issuecomment-2282802313
pull = "always"
try:
image_platform = call(
self.engine.name,
"image",
"inspect",
self.image,
"--format",
(
"{{.Os}}/{{.Architecture}}/{{.Variant}}"
if len(oci_platform.value.split("/")) == 3
else "{{.Os}}/{{.Architecture}}"
),
capture_stdout=True,
).strip()
if image_platform == oci_platform.value:
# in case the correct image is already present, don't pull
# this allows to run local only images
pull = "never"
except subprocess.CalledProcessError:
pass
return f"--platform={oci_platform.value}", f"--pull={pull}"
def __enter__(self) -> Self:
self.name = f"cibuildwheel-{uuid.uuid4()}"
_check_engine_version(self.engine)
# work-around for Travis-CI PPC64le Docker runs since 2021:
# this avoids network splits
# https://github.com/pypa/cibuildwheel/issues/904
# https://github.com/conda-forge/conda-smithy/pull/1520
network_args = []
if detect_ci_provider() == CIProvider.travis_ci and platform.machine() == "ppc64le":
network_args = ["--network=host"]
platform_args = self._get_platform_args()
simulate_32_bit = False
if self.oci_platform in {OCIPlatform.i386, OCIPlatform.ARMV7}:
# If the architecture running the image is already the right one
# or the image entrypoint takes care of enforcing this, then we don't need to
# simulate this
run_cmd = [self.engine.name, "run", "--rm"]
ctr_cmd = ["uname", "-m"]
try:
container_machine = call(
*run_cmd, *platform_args, self.image, *ctr_cmd, capture_stdout=True
).strip()
except subprocess.CalledProcessError:
if self.oci_platform == OCIPlatform.i386:
# The image might have been built with amd64 architecture
# Let's try that
platform_args = self._get_platform_args(oci_platform=OCIPlatform.AMD64)
container_machine = call(
*run_cmd, *platform_args, self.image, *ctr_cmd, capture_stdout=True
).strip()
simulate_32_bit = container_machine not in {"i686", "armv7l", "armv8l"}
shell_args = ["linux32", "/bin/bash"] if simulate_32_bit else ["/bin/bash"]
subprocess.run(
[
self.engine.name,
"create",
"--env=CIBUILDWHEEL",
"--env=SOURCE_DATE_EPOCH",
f"--name={self.name}",
"--interactive",
*(["--volume=/:/host"] if not self.engine.disable_host_mount else []),
*network_args,
*platform_args,
*self.engine.create_args,
self.image,
*shell_args,
],
check=True,
)
self.process = subprocess.Popen(
[
self.engine.name,
"start",
"--attach",
"--interactive",
self.name,
],
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
)
assert self.process.stdin
assert self.process.stdout
self.bash_stdin = self.process.stdin
self.bash_stdout = self.process.stdout
# run a noop command to block until the container is responding
self.call(["/bin/true"], cwd="/")
if self.cwd:
# Although `docker create -w` does create the working dir if it
# does not exist, podman does not. There does not seem to be a way
# to setup a workdir for a container running in podman.
self.call(["mkdir", "-p", os.fspath(self.cwd)], cwd="/")
return self
def __exit__(
self,
exc_type: type[BaseException] | None,
exc_val: BaseException | None,
exc_tb: TracebackType | None,
) -> None:
self.bash_stdin.write(b"exit 0\n")
self.bash_stdin.flush()
self.process.wait(timeout=30)
self.bash_stdin.close()
self.bash_stdout.close()
if self.engine.name == "podman":
# This works around what seems to be a race condition in the podman
# backend. The full reason is not understood. See PR #966 for a
# discussion on possible causes and attempts to remove this line.
# For now, this seems to work "well enough".
self.process.wait()
assert isinstance(self.name, str)
keep_container = strtobool(os.environ.get("CIBW_DEBUG_KEEP_CONTAINER", ""))
if not keep_container:
subprocess.run(
[self.engine.name, "rm", "--force", "-v", self.name],
stdout=subprocess.DEVNULL,
check=False,
)
self.name = None
def copy_into(self, from_path: Path, to_path: PurePath) -> None:
if from_path.is_dir():
self.call(["mkdir", "-p", to_path])
subprocess.run(
f"tar -c {self.host_tar_format} -f - . | {self.engine.name} exec -i {self.name} tar --no-same-owner -xC {shell_quote(to_path)} -f -",
shell=True,
check=True,
cwd=from_path,
)
else:
exec_process: subprocess.Popen[bytes]
with subprocess.Popen(
[
self.engine.name,
"exec",
"-i",
str(self.name),
"sh",
"-c",
f"cat > {shell_quote(to_path)}",
],
stdin=subprocess.PIPE,
) as exec_process:
assert exec_process.stdin
with open(from_path, "rb") as from_file:
# Bug in mypy, https://github.com/python/mypy/issues/15031
shutil.copyfileobj(from_file, exec_process.stdin) # type: ignore[misc]
exec_process.stdin.close()
exec_process.wait()
if exec_process.returncode:
raise subprocess.CalledProcessError(
exec_process.returncode, exec_process.args, None, None
)
def copy_out(self, from_path: PurePath, to_path: Path) -> None:
# note: we assume from_path is a dir
to_path.mkdir(parents=True, exist_ok=True)
call(self.engine.name, "cp", f"{self.name}:{from_path}/.", to_path)
def glob(self, path: PurePosixPath, pattern: str) -> list[PurePosixPath]:
glob_pattern = path.joinpath(pattern)
path_strings = json.loads(
self.call(
[
self.UTILITY_PYTHON,
"-c",
f"import sys, json, glob; json.dump(glob.glob({str(glob_pattern)!r}), sys.stdout)",
],
capture_output=True,
)
)
return [PurePosixPath(p) for p in path_strings]
def call(
self,
args: Sequence[PathOrStr],
env: Mapping[str, str] | None = None,
capture_output: bool = False,
cwd: PathOrStr | None = None,
) -> str:
if cwd is None:
# Podman does not start the a container in a specific working dir
# so we always need to specify it when making calls.
cwd = self.cwd
chdir = f"cd {cwd}" if cwd else ""
env_assignments = (
" ".join(f"{shlex.quote(k)}={shlex.quote(v)}" for k, v in env.items())
if env is not None
else ""
)
command = " ".join(shlex.quote(str(a)) for a in args)
end_of_message = str(uuid.uuid4())
# log the command we're executing
print(f" + {command}")
# Write a command to the remote shell. First we change the
# cwd, if that's required. Then, we use the `env` utility to run
# `command` inside the specified environment. We use `env` because it
# can cope with spaces and strange characters in the name or value.
# Finally, the remote shell is told to write a footer - this will show
# up in the output so we know when to stop reading, and will include
# the return code of `command`.
self.bash_stdin.write(
bytes(
f"""(
{chdir}
env {env_assignments} {command}
printf "%04d%s\n" $? {end_of_message}
)
""",
encoding="utf8",
errors="surrogateescape",
)
)
self.bash_stdin.flush()
if capture_output:
output_io: IO[bytes] = io.BytesIO()
else:
output_io = sys.stdout.buffer
while True:
line = self.bash_stdout.readline()
if line.endswith(bytes(end_of_message, encoding="utf8") + b"\n"):
# fmt: off
footer_offset = (
len(line)
- 1 # newline character
- len(end_of_message) # delimiter
- 4 # 4 return code decimals
)
# fmt: on
return_code_str = line[footer_offset : footer_offset + 4]
return_code = int(return_code_str)
# add the last line to output, without the footer
output_io.write(line[0:footer_offset])
output_io.flush()
break
else:
output_io.write(line)
output_io.flush()
if isinstance(output_io, io.BytesIO):
output = str(output_io.getvalue(), encoding="utf8", errors="surrogateescape")
else:
output = ""
if return_code != 0:
raise subprocess.CalledProcessError(return_code, args, output)
return output
def get_environment(self) -> dict[str, str]:
env = json.loads(
self.call(
[
self.UTILITY_PYTHON,
"-c",
"import sys, json, os; json.dump(os.environ.copy(), sys.stdout)",
],
capture_output=True,
)
)
return typing.cast(Dict[str, str], env)
def environment_executor(self, command: Sequence[str], environment: dict[str, str]) -> str:
# used as an EnvironmentExecutor to evaluate commands and capture output
return self.call(command, env=environment, capture_output=True)
def debug_info(self) -> str:
if self.engine.name == "podman":
command = f"{self.engine.name} info --debug"
else:
command = f"{self.engine.name} info"
completed = subprocess.run(
command,
shell=True,
check=True,
cwd=self.cwd,
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
)
output = str(completed.stdout, encoding="utf8", errors="surrogateescape")
return output
def shell_quote(path: PurePath) -> str:
return shlex.quote(os.fspath(path))
-951
View File
@@ -1,951 +0,0 @@
from __future__ import annotations
import collections
import configparser
import contextlib
import dataclasses
import difflib
import enum
import functools
import shlex
import textwrap
from collections.abc import Generator, Iterable, Set
from pathlib import Path
from typing import Any, Literal, Mapping, Sequence, Union # noqa: TID251
from packaging.specifiers import SpecifierSet
from . import errors
from ._compat import tomllib
from ._compat.typing import assert_never
from .architecture import Architecture
from .environment import EnvironmentParseError, ParsedEnvironment, parse_environment
from .logger import log
from .oci_container import OCIContainerEngineConfig
from .projectfiles import get_requires_python_str
from .typing import PLATFORMS, PlatformName
from .util import (
MANYLINUX_ARCHS,
MUSLLINUX_ARCHS,
BuildFrontendConfig,
BuildSelector,
DependencyConstraints,
TestSelector,
format_safe,
resources_dir,
selector_matches,
strtobool,
unwrap,
)
@dataclasses.dataclass
class CommandLineArguments:
platform: Literal["auto", "linux", "macos", "windows"] | None
archs: str | None
output_dir: Path
only: str | None
config_file: str
package_dir: Path
print_build_identifiers: bool
allow_empty: bool
prerelease_pythons: bool
debug_traceback: bool
@staticmethod
def defaults() -> CommandLineArguments:
return CommandLineArguments(
platform="auto",
allow_empty=False,
archs=None,
only=None,
config_file="",
output_dir=Path("wheelhouse"),
package_dir=Path("."),
prerelease_pythons=False,
print_build_identifiers=False,
debug_traceback=False,
)
@dataclasses.dataclass(frozen=True)
class GlobalOptions:
package_dir: Path
output_dir: Path
build_selector: BuildSelector
test_selector: TestSelector
architectures: set[Architecture]
allow_empty: bool
@dataclasses.dataclass(frozen=True)
class BuildOptions:
globals: GlobalOptions
environment: ParsedEnvironment
before_all: str
before_build: str | None
repair_command: str
manylinux_images: dict[str, str] | None
musllinux_images: dict[str, str] | None
dependency_constraints: DependencyConstraints | None
test_command: str | None
before_test: str | None
test_requires: list[str]
test_extras: str
build_verbosity: int
build_frontend: BuildFrontendConfig | None
config_settings: str
container_engine: OCIContainerEngineConfig
@property
def package_dir(self) -> Path:
return self.globals.package_dir
@property
def output_dir(self) -> Path:
return self.globals.output_dir
@property
def build_selector(self) -> BuildSelector:
return self.globals.build_selector
@property
def test_selector(self) -> TestSelector:
return self.globals.test_selector
@property
def architectures(self) -> set[Architecture]:
return self.globals.architectures
SettingLeaf = Union[str, int, bool]
SettingList = Sequence[SettingLeaf]
SettingTable = Mapping[str, Union[SettingLeaf, SettingList]]
SettingValue = Union[SettingTable, SettingList, SettingLeaf]
@dataclasses.dataclass(frozen=True)
class Override:
select_pattern: str
options: dict[str, SettingValue]
inherit: dict[str, InheritRule]
MANYLINUX_OPTIONS = {f"manylinux-{build_platform}-image" for build_platform in MANYLINUX_ARCHS}
MUSLLINUX_OPTIONS = {f"musllinux-{build_platform}-image" for build_platform in MUSLLINUX_ARCHS}
DISALLOWED_OPTIONS = {
"linux": {"dependency-versions"},
"macos": MANYLINUX_OPTIONS | MUSLLINUX_OPTIONS,
"windows": MANYLINUX_OPTIONS | MUSLLINUX_OPTIONS,
}
class OptionsReaderError(errors.ConfigurationError):
pass
class OptionFormat:
"""
Base class for option format specifiers. These objects describe how values
can be parsed from rich TOML values and how they're merged together.
"""
class NotSupported(Exception):
pass
def format_list(self, value: SettingList) -> str: # noqa: ARG002
raise OptionFormat.NotSupported
def format_table(self, table: SettingTable) -> str: # noqa: ARG002
raise OptionFormat.NotSupported
def merge_values(self, before: str, after: str) -> str: # noqa: ARG002
raise OptionFormat.NotSupported
class ListFormat(OptionFormat):
"""
A format that joins lists with a separator.
"""
def __init__(self, sep: str) -> None:
self.sep = sep
def format_list(self, value: SettingList) -> str:
return self.sep.join(str(v) for v in value)
def merge_values(self, before: str, after: str) -> str:
return f"{before}{self.sep}{after}"
class ShlexTableFormat(OptionFormat):
"""
The standard table format uses shlex.quote to quote values and shlex.split
to unquote and split them. When merging values, keys in before are
replaced by keys in after.
"""
def __init__(self, sep: str = " ", pair_sep: str = "=", allow_merge: bool = True) -> None:
self.sep = sep
self.pair_sep = pair_sep
self.allow_merge = allow_merge
def format_table(self, table: SettingTable) -> str:
assignments: list[tuple[str, str]] = []
for k, v in table.items():
if shlex.split(k) != [k]:
msg = f"Invalid table key: {k}"
raise OptionsReaderError(msg)
if isinstance(v, str):
assignments.append((k, v))
elif isinstance(v, Sequence):
for inner_v in v:
assignments.append((k, str(inner_v)))
else:
assignments.append((k, str(v)))
return self.sep.join(f"{k}{self.pair_sep}{shlex.quote(v)}" for k, v in assignments)
def merge_values(self, before: str, after: str) -> str:
if not self.allow_merge:
raise OptionFormat.NotSupported
before_dict = self.parse_table(before)
after_dict = self.parse_table(after)
return self.format_table({**before_dict, **after_dict})
def parse_table(self, table: str) -> Mapping[str, str | Sequence[str]]:
assignments: list[tuple[str, str]] = []
for assignment_str in shlex.split(table):
key, sep, value = assignment_str.partition(self.pair_sep)
if not sep:
msg = f"malformed option with value {assignment_str!r}"
raise OptionsReaderError(msg)
assignments.append((key, value))
result: dict[str, str | list[str]] = {}
for key, value in assignments:
if key in result:
existing_value = result[key]
if isinstance(existing_value, list):
result[key] = [*existing_value, value]
else:
result[key] = [existing_value, value]
else:
result[key] = value
return result
class EnvironmentFormat(OptionFormat):
"""
The environment format accepts a table of environment variables, where the
values may contain variables or command substitutions.
"""
def format_table(self, table: SettingTable) -> str:
return " ".join(f'{k}="{v}"' for k, v in table.items())
def merge_values(self, before: str, after: str) -> str:
return f"{before} {after}"
class InheritRule(enum.Enum):
NONE = enum.auto()
APPEND = enum.auto()
PREPEND = enum.auto()
def _resolve_cascade(
*pairs: tuple[SettingValue | None, InheritRule],
ignore_empty: bool = False,
option_format: OptionFormat | None = None,
) -> str:
"""
Given a cascade of values with inherit rules, resolve them into a single
value.
'None' values mean that the option was not set at that level, and are
ignored. If `ignore_empty` is True, empty values are ignored too.
Values start with defaults, followed by more specific rules. If rules are
NONE, the last non-null value is returned. If a rule is APPEND or PREPEND,
the value is concatenated with the previous value.
The following idiom can be used to get the first matching value:
_resolve_cascade(("value1", Inherit.NONE), ("value2", Inherit.NONE), ...)))
"""
if not pairs:
msg = "pairs cannot be empty"
raise ValueError(msg)
result: str | None = None
for value, rule in pairs:
if value is None:
continue
if ignore_empty and not value and value is not False:
continue
value_string = _stringify_setting(value, option_format=option_format)
result = _apply_inherit_rule(result, value_string, rule=rule, option_format=option_format)
if result is None:
msg = "a setting should at least have a default value"
raise ValueError(msg)
return result
def _apply_inherit_rule(
before: str | None, after: str, rule: InheritRule, option_format: OptionFormat | None
) -> str:
if rule == InheritRule.NONE:
return after
if not before:
# if before is None, we can just return after
# if before is an empty string, we shouldn't add any separator
return after
if not after:
# if after is an empty string, we shouldn't add any separator
return before
if not option_format:
msg = f"Don't know how to merge {before!r} and {after!r} with {rule}"
raise OptionsReaderError(msg)
if rule == InheritRule.APPEND:
return option_format.merge_values(before, after)
if rule == InheritRule.PREPEND:
return option_format.merge_values(after, before)
assert_never(rule)
def _stringify_setting(
setting: SettingValue,
option_format: OptionFormat | None,
) -> str:
if isinstance(setting, Mapping):
try:
if option_format is None:
raise OptionFormat.NotSupported
return option_format.format_table(setting)
except OptionFormat.NotSupported:
msg = f"Error converting {setting!r} to a string: this setting doesn't accept a table"
raise OptionsReaderError(msg) from None
if not isinstance(setting, str) and isinstance(setting, Sequence):
try:
if option_format is None:
raise OptionFormat.NotSupported
return option_format.format_list(setting)
except OptionFormat.NotSupported:
msg = f"Error converting {setting!r} to a string: this setting doesn't accept a list"
raise OptionsReaderError(msg) from None
if isinstance(setting, (bool, int)):
return str(setting)
return setting
class OptionsReader:
"""
Gets options from the environment, config or defaults, optionally scoped
by the platform.
Example:
>>> options_reader = OptionsReader(config_file, platform='macos')
>>> options_reader.get('cool-color')
This will return the value of CIBW_COOL_COLOR_MACOS if it exists,
otherwise the value of CIBW_COOL_COLOR, otherwise
'tool.cibuildwheel.macos.cool-color' or 'tool.cibuildwheel.cool-color'
from `config_file`, or from cibuildwheel/resources/defaults.toml. An
error is thrown if there are any unexpected keys or sections in
tool.cibuildwheel.
"""
def __init__(
self,
config_file_path: Path | None = None,
*,
platform: PlatformName,
env: Mapping[str, str],
disallow: Mapping[str, Set[str]] | None = None,
) -> None:
self.platform = platform
self.env = env
self.disallow = disallow or {}
# Open defaults.toml, loading both global and platform sections
defaults_path = resources_dir / "defaults.toml"
self.default_options, self.default_platform_options = self._load_file(defaults_path)
# Load the project config file
config_options: dict[str, Any] = {}
config_platform_options: dict[str, Any] = {}
if config_file_path is not None:
config_options, config_platform_options = self._load_file(config_file_path)
# Validate project config
for option_name in config_options:
self._validate_global_option(option_name)
for option_name in config_platform_options:
self._validate_platform_option(option_name)
self.config_options = config_options
self.config_platform_options = config_platform_options
self.overrides: list[Override] = []
self.current_identifier: str | None = None
config_overrides = self.config_options.get("overrides")
if config_overrides is not None:
if not isinstance(config_overrides, list):
msg = "'tool.cibuildwheel.overrides' must be a list"
raise OptionsReaderError(msg)
for config_override in config_overrides:
select = config_override.pop("select", None)
if not select:
msg = "'select' must be set in an override"
raise OptionsReaderError(msg)
if isinstance(select, list):
select = " ".join(select)
inherit = config_override.pop("inherit", {})
if not isinstance(inherit, dict) or not all(
i in {"none", "append", "prepend"} for i in inherit.values()
):
msg = "'inherit' must be a dict containing only {'none', 'append', 'prepend'} values"
raise OptionsReaderError(msg)
inherit_enum = {k: InheritRule[v.upper()] for k, v in inherit.items()}
self.overrides.append(Override(select, config_override, inherit_enum))
def _validate_global_option(self, name: str) -> None:
"""
Raises an error if an option with this name is not allowed in the
[tool.cibuildwheel] section of a config file.
"""
allowed_option_names = self.default_options.keys() | PLATFORMS | {"overrides"}
if name not in allowed_option_names:
msg = f"Option {name!r} not supported in a config file."
matches = difflib.get_close_matches(name, allowed_option_names, 1, 0.7)
if matches:
msg += f" Perhaps you meant {matches[0]!r}?"
raise OptionsReaderError(msg)
def _validate_platform_option(self, name: str) -> None:
"""
Raises an error if an option with this name is not allowed in the
[tool.cibuildwheel.<current-platform>] section of a config file.
"""
disallowed_platform_options = self.disallow.get(self.platform, set())
if name in disallowed_platform_options:
msg = f"{name!r} is not allowed in {disallowed_platform_options}"
raise OptionsReaderError(msg)
allowed_option_names = self.default_options.keys() | self.default_platform_options.keys()
if name not in allowed_option_names:
msg = f"Option {name!r} not supported in the {self.platform!r} section"
matches = difflib.get_close_matches(name, allowed_option_names, 1, 0.7)
if matches:
msg += f" Perhaps you meant {matches[0]!r}?"
raise OptionsReaderError(msg)
def _load_file(self, filename: Path) -> tuple[dict[str, Any], dict[str, Any]]:
"""
Load a toml file, returns global and platform as separate dicts.
"""
with filename.open("rb") as f:
config = tomllib.load(f)
global_options = config.get("tool", {}).get("cibuildwheel", {})
platform_options = global_options.get(self.platform, {})
return global_options, platform_options
@property
def active_config_overrides(self) -> list[Override]:
if self.current_identifier is None:
return []
return [
o for o in self.overrides if selector_matches(o.select_pattern, self.current_identifier)
]
@contextlib.contextmanager
def identifier(self, identifier: str | None) -> Generator[None, None, None]:
self.current_identifier = identifier
try:
yield
finally:
self.current_identifier = None
def get(
self,
name: str,
*,
env_plat: bool = True,
option_format: OptionFormat | None = None,
ignore_empty: bool = False,
) -> str:
"""
Get and return the value for the named option from environment,
configuration file, or the default. If env_plat is False, then don't
accept platform versions of the environment variable. If this is an
array it will be merged with "sep" before returning. If it is a table,
it will be formatted with "table['item']" using {k} and {v} and merged
with "table['sep']". If sep is also given, it will be used for arrays
inside the table (must match table['sep']). Empty variables will not
override if ignore_empty is True.
"""
if name not in self.default_options and name not in self.default_platform_options:
msg = f"{name!r} must be in cibuildwheel/resources/defaults.toml file to be accessed."
raise OptionsReaderError(msg)
# Environment variable form
envvar = f"CIBW_{name.upper().replace('-', '_')}"
plat_envvar = f"{envvar}_{self.platform.upper()}"
# get the option from the default, then the config file, then finally the environment.
# platform-specific options are preferred, if they're allowed.
return _resolve_cascade(
(self.default_options.get(name), InheritRule.NONE),
(self.default_platform_options.get(name), InheritRule.NONE),
(self.config_options.get(name), InheritRule.NONE),
(self.config_platform_options.get(name), InheritRule.NONE),
*[
(o.options.get(name), o.inherit.get(name, InheritRule.NONE))
for o in self.active_config_overrides
],
(self.env.get(envvar), InheritRule.NONE),
(self.env.get(plat_envvar) if env_plat else None, InheritRule.NONE),
ignore_empty=ignore_empty,
option_format=option_format,
)
class Options:
def __init__(
self,
platform: PlatformName,
command_line_arguments: CommandLineArguments,
env: Mapping[str, str],
read_config_file: bool = True,
):
self.platform = platform
self.command_line_arguments = command_line_arguments
self.env = env
self.reader = OptionsReader(
self.config_file_path if read_config_file else None,
platform=platform,
env=env,
disallow=DISALLOWED_OPTIONS,
)
@property
def config_file_path(self) -> Path | None:
args = self.command_line_arguments
if args.config_file:
return Path(format_safe(args.config_file, package=args.package_dir))
# return pyproject.toml, if it's available
pyproject_toml_path = Path(args.package_dir) / "pyproject.toml"
if pyproject_toml_path.exists():
return pyproject_toml_path
return None
@functools.cached_property
def package_requires_python_str(self) -> str | None:
args = self.command_line_arguments
return get_requires_python_str(Path(args.package_dir))
@property
def globals(self) -> GlobalOptions:
args = self.command_line_arguments
package_dir = args.package_dir
output_dir = args.output_dir
build_config = (
self.reader.get("build", env_plat=False, option_format=ListFormat(sep=" ")) or "*"
)
skip_config = self.reader.get("skip", env_plat=False, option_format=ListFormat(sep=" "))
test_skip = self.reader.get("test-skip", env_plat=False, option_format=ListFormat(sep=" "))
free_threaded_support = strtobool(
self.reader.get("free-threaded-support", env_plat=False, ignore_empty=True)
)
allow_empty = args.allow_empty or strtobool(self.env.get("CIBW_ALLOW_EMPTY", "0"))
prerelease_pythons = args.prerelease_pythons or strtobool(
self.env.get("CIBW_PRERELEASE_PYTHONS", "0")
)
# This is not supported in tool.cibuildwheel, as it comes from a standard location.
# Passing this in as an environment variable will override pyproject.toml, setup.cfg, or setup.py
requires_python_str: str | None = (
self.env.get("CIBW_PROJECT_REQUIRES_PYTHON") or self.package_requires_python_str
)
requires_python = None if requires_python_str is None else SpecifierSet(requires_python_str)
archs_config_str = args.archs or self.reader.get("archs", option_format=ListFormat(sep=" "))
architectures = Architecture.parse_config(archs_config_str, platform=self.platform)
# Process `--only`
if args.only:
build_config = args.only
skip_config = ""
architectures = Architecture.all_archs(self.platform)
prerelease_pythons = True
free_threaded_support = True
build_selector = BuildSelector(
build_config=build_config,
skip_config=skip_config,
requires_python=requires_python,
prerelease_pythons=prerelease_pythons,
free_threaded_support=free_threaded_support,
)
test_selector = TestSelector(skip_config=test_skip)
return GlobalOptions(
package_dir=package_dir,
output_dir=output_dir,
build_selector=build_selector,
test_selector=test_selector,
architectures=architectures,
allow_empty=allow_empty,
)
def build_options(self, identifier: str | None) -> BuildOptions:
"""
Compute BuildOptions for a single run configuration.
"""
with self.reader.identifier(identifier):
before_all = self.reader.get("before-all", option_format=ListFormat(sep=" && "))
environment_config = self.reader.get("environment", option_format=EnvironmentFormat())
environment_pass = self.reader.get(
"environment-pass", option_format=ListFormat(sep=" ")
).split()
before_build = self.reader.get("before-build", option_format=ListFormat(sep=" && "))
repair_command = self.reader.get(
"repair-wheel-command", option_format=ListFormat(sep=" && ")
)
config_settings = self.reader.get(
"config-settings", option_format=ShlexTableFormat(sep=" ", pair_sep="=")
)
dependency_versions = self.reader.get("dependency-versions")
test_command = self.reader.get("test-command", option_format=ListFormat(sep=" && "))
before_test = self.reader.get("before-test", option_format=ListFormat(sep=" && "))
test_requires = self.reader.get(
"test-requires", option_format=ListFormat(sep=" ")
).split()
test_extras = self.reader.get("test-extras", option_format=ListFormat(sep=","))
build_verbosity_str = self.reader.get("build-verbosity")
build_frontend_str = self.reader.get(
"build-frontend",
env_plat=False,
option_format=ShlexTableFormat(sep="; ", pair_sep=":", allow_merge=False),
)
build_frontend: BuildFrontendConfig | None
if not build_frontend_str or build_frontend_str == "default":
build_frontend = None
else:
try:
build_frontend = BuildFrontendConfig.from_config_string(build_frontend_str)
except ValueError as e:
msg = f"Failed to parse build frontend. {e}"
raise errors.ConfigurationError(msg) from e
try:
environment = parse_environment(environment_config)
except (EnvironmentParseError, ValueError) as e:
msg = f"Malformed environment option {environment_config!r}"
raise errors.ConfigurationError(msg) from e
# Pass through environment variables
if self.platform == "linux":
for env_var_name in reversed(environment_pass):
with contextlib.suppress(KeyError):
environment.add(env_var_name, self.env[env_var_name], prepend=True)
if dependency_versions == "pinned":
dependency_constraints: None | (
DependencyConstraints
) = DependencyConstraints.with_defaults()
elif dependency_versions == "latest":
dependency_constraints = None
else:
dependency_versions_path = Path(dependency_versions)
dependency_constraints = DependencyConstraints(dependency_versions_path)
if test_extras:
test_extras = f"[{test_extras}]"
try:
build_verbosity = min(3, max(-3, int(build_verbosity_str)))
except ValueError:
build_verbosity = 0
manylinux_images: dict[str, str] = {}
musllinux_images: dict[str, str] = {}
container_engine: OCIContainerEngineConfig | None = None
if self.platform == "linux":
all_pinned_container_images = _get_pinned_container_images()
for build_platform in MANYLINUX_ARCHS:
pinned_images = all_pinned_container_images[build_platform]
config_value = self.reader.get(
f"manylinux-{build_platform}-image", ignore_empty=True
)
if not config_value:
# default to manylinux2014
image = pinned_images["manylinux2014"]
elif config_value in pinned_images:
image = pinned_images[config_value]
else:
image = config_value
manylinux_images[build_platform] = image
for build_platform in MUSLLINUX_ARCHS:
pinned_images = all_pinned_container_images[build_platform]
config_value = self.reader.get(f"musllinux-{build_platform}-image")
if not config_value:
image = pinned_images["musllinux_1_2"]
elif config_value in pinned_images:
image = pinned_images[config_value]
else:
image = config_value
musllinux_images[build_platform] = image
container_engine_str = self.reader.get(
"container-engine",
option_format=ShlexTableFormat(sep="; ", pair_sep=":", allow_merge=False),
)
try:
container_engine = OCIContainerEngineConfig.from_config_string(container_engine_str)
except ValueError as e:
msg = f"Failed to parse container config. {e}"
raise errors.ConfigurationError(msg) from e
return BuildOptions(
globals=self.globals,
test_command=test_command,
test_requires=test_requires,
test_extras=test_extras,
before_test=before_test,
before_build=before_build,
before_all=before_all,
build_verbosity=build_verbosity,
repair_command=repair_command,
environment=environment,
dependency_constraints=dependency_constraints,
manylinux_images=manylinux_images or None,
musllinux_images=musllinux_images or None,
build_frontend=build_frontend,
config_settings=config_settings,
container_engine=container_engine,
)
def check_for_invalid_configuration(self, identifiers: Iterable[str]) -> None:
if self.platform in {"macos", "windows"}:
before_all_values = {self.build_options(i).before_all for i in identifiers}
if len(before_all_values) > 1:
raise ValueError(
unwrap(
f"""
before_all cannot be set to multiple values. On macOS and Windows,
before_all is only run once, at the start of the build. before_all values
are: {before_all_values!r}
"""
)
)
def check_for_deprecated_options(self) -> None:
build_selector = self.globals.build_selector
test_selector = self.globals.test_selector
deprecated_selectors("CIBW_BUILD", build_selector.build_config, error=True)
deprecated_selectors("CIBW_SKIP", build_selector.skip_config)
deprecated_selectors("CIBW_TEST_SKIP", test_selector.skip_config)
@functools.cached_property
def defaults(self) -> Options:
return Options(
platform=self.platform,
command_line_arguments=CommandLineArguments.defaults(),
env={},
read_config_file=False,
)
def summary(self, identifiers: Iterable[str]) -> str:
lines = []
global_option_names = sorted(f.name for f in dataclasses.fields(self.globals))
for option_name in global_option_names:
option_value = getattr(self.globals, option_name)
default_value = getattr(self.defaults.globals, option_name)
lines.append(self.option_summary(option_name, option_value, default_value))
build_options = self.build_options(identifier=None)
build_options_defaults = self.defaults.build_options(identifier=None)
build_options_for_identifier = {
identifier: self.build_options(identifier) for identifier in identifiers
}
build_option_names = sorted(f.name for f in dataclasses.fields(build_options))
for option_name in build_option_names:
if option_name == "globals":
continue
option_value = getattr(build_options, option_name)
default_value = getattr(build_options_defaults, option_name)
overrides = {
i: getattr(build_options_for_identifier[i], option_name) for i in identifiers
}
lines.append(
self.option_summary(option_name, option_value, default_value, overrides=overrides)
)
return "\n".join(lines)
def option_summary(
self,
option_name: str,
option_value: Any,
default_value: Any,
overrides: Mapping[str, Any] | None = None,
) -> str:
"""
Return a summary of the option value, including any overrides, with
ANSI 'dim' color if it's the default.
"""
value_str = self.option_summary_value(option_value)
default_value_str = self.option_summary_value(default_value)
overrides_value_strs = {
k: self.option_summary_value(v) for k, v in (overrides or {}).items()
}
# if the override value is the same as the non-overridden value, don't print it
overrides_value_strs = {k: v for k, v in overrides_value_strs.items() if v != value_str}
has_been_set = (value_str != default_value_str) or overrides_value_strs
c = log.colors
result = c.gray if not has_been_set else ""
result += f"{option_name}: "
if overrides_value_strs:
overrides_groups = collections.defaultdict(list)
for k, v in overrides_value_strs.items():
overrides_groups[v].append(k)
result += "\n *: "
result += self.indent_if_multiline(value_str, " ")
for override_value_str, identifiers in overrides_groups.items():
result += f"\n {', '.join(identifiers)}: "
result += self.indent_if_multiline(override_value_str, " ")
else:
result += self.indent_if_multiline(value_str, " ")
result += c.end
return result
def indent_if_multiline(self, value: str, indent: str) -> str:
if "\n" in value:
return "\n" + textwrap.indent(value.strip(), indent)
else:
return value
def option_summary_value(self, option_value: Any) -> str:
if hasattr(option_value, "options_summary"):
option_value = option_value.options_summary()
if isinstance(option_value, list):
return "".join(f"{el}\n" for el in option_value)
if isinstance(option_value, set):
return ", ".join(str(el) for el in sorted(option_value))
if isinstance(option_value, dict):
return "".join(f"{k}: {v}\n" for k, v in option_value.items())
return str(option_value)
def compute_options(
platform: PlatformName,
command_line_arguments: CommandLineArguments,
env: Mapping[str, str],
) -> Options:
options = Options(platform=platform, command_line_arguments=command_line_arguments, env=env)
options.check_for_deprecated_options()
return options
@functools.lru_cache(maxsize=None)
def _get_pinned_container_images() -> Mapping[str, Mapping[str, str]]:
"""
This looks like a dict of dicts, e.g.
{ 'x86_64': {'manylinux1': '...', 'manylinux2010': '...', 'manylinux2014': '...'},
'i686': {'manylinux1': '...', 'manylinux2010': '...', 'manylinux2014': '...'},
'pypy_x86_64': {'manylinux2010': '...' }
... }
"""
pinned_images_file = resources_dir / "pinned_docker_images.cfg"
all_pinned_images = configparser.ConfigParser()
all_pinned_images.read(pinned_images_file)
return all_pinned_images
def deprecated_selectors(name: str, selector: str, *, error: bool = False) -> None:
if "p2" in selector or "p35" in selector:
msg = f"cibuildwheel 2.x no longer supports Python < 3.6. Please use the 1.x series or update {name}"
if error:
raise errors.DeprecationError(msg)
log.warning(msg)
+46 -76
View File
@@ -1,108 +1,78 @@
from __future__ import annotations
import ast
import configparser
import contextlib
import sys
from configparser import ConfigParser
from pathlib import Path
from typing import Any, Optional
from ._compat import tomllib
import toml
if sys.version_info < (3, 8):
Constant = ast.Str
def get_constant(x: ast.Str) -> str:
return x.s
def get_parent(node: ast.AST | None, depth: int = 1) -> ast.AST | None:
for _ in range(depth):
node = getattr(node, "parent", None)
return node
else:
Constant = ast.Constant
def is_main(parent: ast.AST | None) -> bool:
if parent is None:
return False
# This would be much nicer with 3.10's pattern matching!
if not isinstance(parent, ast.If):
return False
if not isinstance(parent.test, ast.Compare):
return False
try:
(op,) = parent.test.ops
(comp,) = parent.test.comparators
except ValueError:
return False
if not isinstance(op, ast.Eq):
return False
values = {comp, parent.test.left}
mains = {x for x in values if isinstance(x, ast.Constant) and x.value == "__main__"}
if len(mains) != 1:
return False
consts = {x for x in values if isinstance(x, ast.Name) and x.id == "__name__"}
return len(consts) == 1
def get_constant(x: ast.Constant) -> Any:
return x.value
class Analyzer(ast.NodeVisitor):
def __init__(self) -> None:
self.requires_python: str | None = None
self.requires_python: Optional[str] = None
def visit(self, node: ast.AST) -> None:
for inner_node in ast.walk(node):
for child in ast.iter_child_nodes(inner_node):
child.parent = inner_node # type: ignore[attr-defined]
super().visit(node)
def visit(self, content: ast.AST) -> None:
for node in ast.walk(content):
for child in ast.iter_child_nodes(node):
child.parent = node # type: ignore
super().visit(content)
def visit_keyword(self, node: ast.keyword) -> None:
# Must not be nested except for if __name__ == "__main__"
self.generic_visit(node)
# This will be Module -> Expr -> Call -> keyword
parent = get_parent(node, 4)
unnested = parent is None
# This will be Module -> If -> Expr -> Call -> keyword
name_main_unnested = (
parent is not None and get_parent(parent) is None and is_main(get_parent(node, 3))
)
if (
node.arg == "python_requires"
and isinstance(node.value, ast.Constant)
and (unnested or name_main_unnested)
):
self.requires_python = node.value.value
if node.arg == "python_requires":
# Must not be nested in an if or other structure
# This will be Module -> Expr -> Call -> keyword
if not hasattr(node.parent.parent.parent, "parent") and isinstance( # type: ignore
node.value, Constant
):
self.requires_python = get_constant(node.value)
def setup_py_python_requires(content: str) -> str | None:
def setup_py_python_requires(content: str) -> Optional[str]:
try:
tree = ast.parse(content)
analyzer = Analyzer()
analyzer.visit(tree)
return analyzer.requires_python or None
except Exception: # pylint: disable=broad-except
except Exception:
return None
def get_requires_python_str(package_dir: Path) -> str | None:
"""Return the python requires string from the most canonical source available, or None"""
def get_requires_python_str(package_dir: Path) -> Optional[str]:
"Return the python requires string from the most canonical source available, or None"
# Read in from pyproject.toml:project.requires-python
with contextlib.suppress(FileNotFoundError):
with (package_dir / "pyproject.toml").open("rb") as f1:
info = tomllib.load(f1)
with contextlib.suppress(KeyError, IndexError, TypeError):
return str(info["project"]["requires-python"])
try:
info = toml.load(package_dir / "pyproject.toml")
return str(info["project"]["requires-python"])
except (FileNotFoundError, KeyError, IndexError, TypeError):
pass
# Read in from setup.cfg:options.python_requires
config = configparser.ConfigParser()
with contextlib.suppress(FileNotFoundError):
try:
config = ConfigParser()
config.read(package_dir / "setup.cfg")
with contextlib.suppress(KeyError, IndexError, TypeError):
return str(config["options"]["python_requires"])
return str(config["options"]["python_requires"])
except (FileNotFoundError, KeyError, IndexError, TypeError):
pass
setup_py = package_dir / "setup.py"
with contextlib.suppress(FileNotFoundError), setup_py.open(encoding="utf8") as f2:
return setup_py_python_requires(f2.read())
try:
with open(package_dir / "setup.py") as f:
return setup_py_python_requires(f.read())
except FileNotFoundError:
pass
return None
-399
View File
@@ -1,399 +0,0 @@
from __future__ import annotations
import os
import shutil
import sys
from collections.abc import Sequence, Set
from dataclasses import dataclass
from pathlib import Path
from filelock import FileLock
from . import errors
from .architecture import Architecture
from .environment import ParsedEnvironment
from .logger import log
from .options import Options
from .typing import PathOrStr
from .util import (
CIBW_CACHE_PATH,
BuildFrontendConfig,
BuildSelector,
call,
combine_constraints,
download,
ensure_node,
extract_zip,
find_compatible_wheel,
get_pip_version,
move_file,
prepare_command,
read_python_configs,
shell,
split_config_settings,
test_fail_cwd_file,
virtualenv,
)
@dataclass(frozen=True)
class PythonConfiguration:
version: str
identifier: str
pyodide_version: str
emscripten_version: str
node_version: str
def install_emscripten(tmp: Path, version: str) -> Path:
# We don't need to match the emsdk version to the version we install, but
# we do for stability
url = f"https://github.com/emscripten-core/emsdk/archive/refs/tags/{version}.zip"
installation_path = CIBW_CACHE_PATH / f"emsdk-{version}"
emsdk_path = installation_path / f"emsdk-{version}/emsdk"
emcc_path = installation_path / f"emsdk-{version}/upstream/emscripten/emcc"
with FileLock(f"{installation_path}.lock"):
if installation_path.exists():
return emcc_path
emsdk_zip = tmp / "emsdk.zip"
download(url, emsdk_zip)
installation_path.mkdir()
extract_zip(emsdk_zip, installation_path)
call(emsdk_path, "install", version)
call(emsdk_path, "activate", version)
return emcc_path
def install_xbuildenv(env: dict[str, str], pyodide_version: str) -> str:
pyodide_root = (
CIBW_CACHE_PATH
/ f".pyodide-xbuildenv-{pyodide_version}/{pyodide_version}/xbuildenv/pyodide-root"
)
with FileLock(CIBW_CACHE_PATH / "xbuildenv.lock"):
if pyodide_root.exists():
return str(pyodide_root)
# We don't want to mutate env but we need to delete any existing
# PYODIDE_ROOT so copy it first.
env = dict(env)
env.pop("PYODIDE_ROOT", None)
call(
"pyodide",
"xbuildenv",
"install",
pyodide_version,
env=env,
cwd=CIBW_CACHE_PATH,
)
return str(pyodide_root)
def get_base_python(identifier: str) -> Path:
implementation_id = identifier.split("-")[0]
majorminor = implementation_id[len("cp") :]
version_info = (int(majorminor[0]), int(majorminor[1:]))
if version_info == sys.version_info[:2]:
return Path(sys.executable)
major_minor = ".".join(str(v) for v in version_info)
python_name = f"python{major_minor}"
which_python = shutil.which(python_name)
if which_python is None:
msg = f"CPython {major_minor} is not installed."
raise errors.FatalError(msg)
return Path(which_python)
def setup_python(
tmp: Path,
python_configuration: PythonConfiguration,
dependency_constraint_flags: Sequence[PathOrStr],
environment: ParsedEnvironment,
) -> dict[str, str]:
base_python = get_base_python(python_configuration.identifier)
log.step("Setting up build environment...")
venv_path = tmp / "venv"
env = virtualenv(python_configuration.version, base_python, venv_path, [], use_uv=False)
venv_bin_path = venv_path / "bin"
assert venv_bin_path.exists()
env["PIP_DISABLE_PIP_VERSION_CHECK"] = "1"
# upgrade pip to the version matching our constraints
# if necessary, reinstall it to ensure that it's available on PATH as 'pip'
call(
"python",
"-m",
"pip",
"install",
"--upgrade",
"pip",
*dependency_constraint_flags,
env=env,
cwd=venv_path,
)
env = environment.as_dictionary(prev_environment=env)
# check what pip version we're on
assert (venv_bin_path / "pip").exists()
call("which", "pip", env=env)
call("pip", "--version", env=env)
which_pip = call("which", "pip", env=env, capture_stdout=True).strip()
if which_pip != str(venv_bin_path / "pip"):
msg = "pip available on PATH doesn't match our venv instance. If you have modified PATH, ensure that you don't overwrite cibuildwheel's entry or insert pip above it."
raise errors.FatalError(msg)
# check what Python version we're on
call("which", "python", env=env)
call("python", "--version", env=env)
which_python = call("which", "python", env=env, capture_stdout=True).strip()
if which_python != str(venv_bin_path / "python"):
msg = "python available on PATH doesn't match our venv instance. If you have modified PATH, ensure that you don't overwrite cibuildwheel's entry or insert python above it."
raise errors.FatalError(msg)
log.step("Installing build tools...")
call(
"pip",
"install",
"--upgrade",
"auditwheel-emscripten",
"build[virtualenv]",
"pyodide-build",
*dependency_constraint_flags,
env=env,
)
log.step("Installing emscripten...")
emcc_path = install_emscripten(tmp, python_configuration.emscripten_version)
env["PATH"] = os.pathsep.join([str(emcc_path.parent), env["PATH"]])
log.step("Installing Pyodide xbuildenv...")
env["PYODIDE_ROOT"] = install_xbuildenv(env, python_configuration.pyodide_version)
return env
def get_python_configurations(
build_selector: BuildSelector,
architectures: Set[Architecture], # noqa: ARG001
) -> list[PythonConfiguration]:
full_python_configs = read_python_configs("pyodide")
python_configurations = [PythonConfiguration(**item) for item in full_python_configs]
python_configurations = [c for c in python_configurations if build_selector(c.identifier)]
return python_configurations
def build(options: Options, tmp_path: Path) -> None:
python_configurations = get_python_configurations(
options.globals.build_selector, options.globals.architectures
)
if not python_configurations:
return
try:
before_all_options_identifier = python_configurations[0].identifier
before_all_options = options.build_options(before_all_options_identifier)
if before_all_options.before_all:
log.step("Running before_all...")
env = before_all_options.environment.as_dictionary(prev_environment=os.environ)
before_all_prepared = prepare_command(
before_all_options.before_all, project=".", package=before_all_options.package_dir
)
shell(before_all_prepared, env=env)
built_wheels: list[Path] = []
for config in python_configurations:
build_options = options.build_options(config.identifier)
build_frontend = build_options.build_frontend or BuildFrontendConfig("build")
if build_frontend.name == "pip":
msg = "The pyodide platform doesn't support pip frontend"
raise errors.FatalError(msg)
log.build_start(config.identifier)
identifier_tmp_dir = tmp_path / config.identifier
built_wheel_dir = identifier_tmp_dir / "built_wheel"
repaired_wheel_dir = identifier_tmp_dir / "repaired_wheel"
identifier_tmp_dir.mkdir()
built_wheel_dir.mkdir()
repaired_wheel_dir.mkdir()
dependency_constraint_flags: Sequence[PathOrStr] = []
if build_options.dependency_constraints:
constraints_path = build_options.dependency_constraints.get_for_python_version(
config.version, variant="pyodide"
)
dependency_constraint_flags = ["-c", constraints_path]
env = setup_python(
identifier_tmp_dir / "build",
config,
dependency_constraint_flags,
build_options.environment,
)
pip_version = get_pip_version(env)
# The Pyodide command line runner mounts all directories in the host
# filesystem into the Pyodide file system, except for the custom
# file systems /dev, /lib, /proc, and /tmp. Mounting the mount
# points for alternate file systems causes some mysterious failure
# of the process (it just quits without any clear error).
#
# Because of this, by default Pyodide can't see anything under /tmp.
# This environment variable tells it also to mount our temp
# directory.
oldmounts = ""
extra_mounts = [str(identifier_tmp_dir)]
if str(Path(".").resolve()).startswith("/tmp"):
extra_mounts.append(str(Path(".").resolve()))
if "_PYODIDE_EXTRA_MOUNTS" in env:
oldmounts = env["_PYODIDE_EXTRA_MOUNTS"] + ":"
env["_PYODIDE_EXTRA_MOUNTS"] = oldmounts + ":".join(extra_mounts)
compatible_wheel = find_compatible_wheel(built_wheels, config.identifier)
if compatible_wheel:
log.step_end()
print(
f"\nFound previously built wheel {compatible_wheel.name}, that's compatible with {config.identifier}. Skipping build step..."
)
built_wheel = compatible_wheel
else:
if build_options.before_build:
log.step("Running before_build...")
before_build_prepared = prepare_command(
build_options.before_build, project=".", package=build_options.package_dir
)
shell(before_build_prepared, env=env)
log.step("Building wheel...")
extra_flags = split_config_settings(build_options.config_settings, "build")
extra_flags += build_frontend.args
if not 0 <= build_options.build_verbosity < 2:
msg = f"build_verbosity {build_options.build_verbosity} is not supported for build frontend. Ignoring."
log.warning(msg)
build_env = env.copy()
if build_options.dependency_constraints:
combine_constraints(build_env, constraints_path, identifier_tmp_dir)
build_env["VIRTUALENV_PIP"] = pip_version
call(
"pyodide",
"build",
build_options.package_dir,
f"--outdir={built_wheel_dir}",
*extra_flags,
env=build_env,
)
built_wheel = next(built_wheel_dir.glob("*.whl"))
if built_wheel.name.endswith("none-any.whl"):
raise errors.NonPlatformWheelError()
if build_options.repair_command:
log.step("Repairing wheel...")
repair_command_prepared = prepare_command(
build_options.repair_command,
wheel=built_wheel,
dest_dir=repaired_wheel_dir,
)
shell(repair_command_prepared, env=env)
else:
shutil.move(str(built_wheel), repaired_wheel_dir)
repaired_wheel = next(repaired_wheel_dir.glob("*.whl"))
if repaired_wheel.name in {wheel.name for wheel in built_wheels}:
raise errors.AlreadyBuiltWheelError(repaired_wheel.name)
if build_options.test_command and build_options.test_selector(config.identifier):
log.step("Testing wheel...")
venv_dir = identifier_tmp_dir / "venv-test"
# set up a virtual environment to install and test from, to make sure
# there are no dependencies that were pulled in at build time.
virtualenv_env = env.copy()
virtualenv_env["PATH"] = os.pathsep.join(
[
str(ensure_node(config.node_version)),
virtualenv_env["PATH"],
]
)
# pyodide venv uses virtualenv under the hood
# use the pip embedded with virtualenv & disable network updates
virtualenv_create_env = virtualenv_env.copy()
virtualenv_create_env["VIRTUALENV_PIP"] = pip_version
virtualenv_create_env["VIRTUALENV_NO_PERIODIC_UPDATE"] = "1"
call("pyodide", "venv", venv_dir, env=virtualenv_create_env)
virtualenv_env["PATH"] = os.pathsep.join(
[
str(venv_dir / "bin"),
virtualenv_env["PATH"],
]
)
virtualenv_env["VIRTUAL_ENV"] = str(venv_dir)
# check that we are using the Python from the virtual environment
call("which", "python", env=virtualenv_env)
if build_options.before_test:
before_test_prepared = prepare_command(
build_options.before_test,
project=".",
package=build_options.package_dir,
)
shell(before_test_prepared, env=virtualenv_env)
# install the wheel
call(
"pip",
"install",
f"{repaired_wheel}{build_options.test_extras}",
env=virtualenv_env,
)
# test the wheel
if build_options.test_requires:
call("pip", "install", *build_options.test_requires, env=virtualenv_env)
# run the tests from a temp dir, with an absolute path in the command
# (this ensures that Python runs the tests against the installed wheel
# and not the repo code)
test_command_prepared = prepare_command(
build_options.test_command,
project=Path(".").resolve(),
package=build_options.package_dir.resolve(),
)
test_cwd = identifier_tmp_dir / "test_cwd"
test_cwd.mkdir(exist_ok=True)
(test_cwd / "test_fail.py").write_text(test_fail_cwd_file.read_text())
shell(test_command_prepared, cwd=test_cwd, env=virtualenv_env)
# we're all done here; move it to output (overwrite existing)
if compatible_wheel is None:
output_wheel = build_options.output_dir.joinpath(repaired_wheel.name)
moved_wheel = move_file(repaired_wheel, output_wheel)
if moved_wheel != output_wheel.resolve():
log.warning(
"{repaired_wheel} was moved to {moved_wheel} instead of {output_wheel}"
)
built_wheels.append(output_wheel)
finally:
pass
+30 -144
View File
@@ -1,185 +1,71 @@
[linux]
python_configurations = [
{ identifier = "cp27-manylinux_x86_64", version = "2.7", path_str = "/opt/python/cp27-cp27m" },
{ identifier = "cp27-manylinux_x86_64", version = "2.7", path_str = "/opt/python/cp27-cp27mu" },
{ identifier = "cp35-manylinux_x86_64", version = "3.5", path_str = "/opt/python/cp35-cp35m" },
{ identifier = "cp36-manylinux_x86_64", version = "3.6", path_str = "/opt/python/cp36-cp36m" },
{ identifier = "cp37-manylinux_x86_64", version = "3.7", path_str = "/opt/python/cp37-cp37m" },
{ identifier = "cp38-manylinux_x86_64", version = "3.8", path_str = "/opt/python/cp38-cp38" },
{ identifier = "cp39-manylinux_x86_64", version = "3.9", path_str = "/opt/python/cp39-cp39" },
{ identifier = "cp310-manylinux_x86_64", version = "3.10", path_str = "/opt/python/cp310-cp310" },
{ identifier = "cp311-manylinux_x86_64", version = "3.11", path_str = "/opt/python/cp311-cp311" },
{ identifier = "cp312-manylinux_x86_64", version = "3.12", path_str = "/opt/python/cp312-cp312" },
{ identifier = "cp313-manylinux_x86_64", version = "3.13", path_str = "/opt/python/cp313-cp313" },
{ identifier = "cp313t-manylinux_x86_64", version = "3.13", path_str = "/opt/python/cp313-cp313t" },
{ identifier = "cp27-manylinux_i686", version = "2.7", path_str = "/opt/python/cp27-cp27m" },
{ identifier = "cp27-manylinux_i686", version = "2.7", path_str = "/opt/python/cp27-cp27mu" },
{ identifier = "cp35-manylinux_i686", version = "3.5", path_str = "/opt/python/cp35-cp35m" },
{ identifier = "cp36-manylinux_i686", version = "3.6", path_str = "/opt/python/cp36-cp36m" },
{ identifier = "cp37-manylinux_i686", version = "3.7", path_str = "/opt/python/cp37-cp37m" },
{ identifier = "cp38-manylinux_i686", version = "3.8", path_str = "/opt/python/cp38-cp38" },
{ identifier = "cp39-manylinux_i686", version = "3.9", path_str = "/opt/python/cp39-cp39" },
{ identifier = "cp310-manylinux_i686", version = "3.10", path_str = "/opt/python/cp310-cp310" },
{ identifier = "cp311-manylinux_i686", version = "3.11", path_str = "/opt/python/cp311-cp311" },
{ identifier = "cp312-manylinux_i686", version = "3.12", path_str = "/opt/python/cp312-cp312" },
{ identifier = "cp313-manylinux_i686", version = "3.13", path_str = "/opt/python/cp313-cp313" },
{ identifier = "cp313t-manylinux_i686", version = "3.13", path_str = "/opt/python/cp313-cp313t" },
{ identifier = "pp27-manylinux_x86_64", version = "2.7", path_str = "/opt/python/pp27-pypy_73" },
{ identifier = "pp36-manylinux_x86_64", version = "3.6", path_str = "/opt/python/pp36-pypy36_pp73" },
{ identifier = "pp37-manylinux_x86_64", version = "3.7", path_str = "/opt/python/pp37-pypy37_pp73" },
{ identifier = "pp38-manylinux_x86_64", version = "3.8", path_str = "/opt/python/pp38-pypy38_pp73" },
{ identifier = "pp39-manylinux_x86_64", version = "3.9", path_str = "/opt/python/pp39-pypy39_pp73" },
{ identifier = "pp310-manylinux_x86_64", version = "3.10", path_str = "/opt/python/pp310-pypy310_pp73" },
{ identifier = "cp35-manylinux_aarch64", version = "3.5", path_str = "/opt/python/cp35-cp35m" },
{ identifier = "cp36-manylinux_aarch64", version = "3.6", path_str = "/opt/python/cp36-cp36m" },
{ identifier = "cp37-manylinux_aarch64", version = "3.7", path_str = "/opt/python/cp37-cp37m" },
{ identifier = "cp38-manylinux_aarch64", version = "3.8", path_str = "/opt/python/cp38-cp38" },
{ identifier = "cp39-manylinux_aarch64", version = "3.9", path_str = "/opt/python/cp39-cp39" },
{ identifier = "cp310-manylinux_aarch64", version = "3.10", path_str = "/opt/python/cp310-cp310" },
{ identifier = "cp311-manylinux_aarch64", version = "3.11", path_str = "/opt/python/cp311-cp311" },
{ identifier = "cp312-manylinux_aarch64", version = "3.12", path_str = "/opt/python/cp312-cp312" },
{ identifier = "cp313-manylinux_aarch64", version = "3.13", path_str = "/opt/python/cp313-cp313" },
{ identifier = "cp313t-manylinux_aarch64", version = "3.13", path_str = "/opt/python/cp313-cp313t" },
{ identifier = "cp35-manylinux_ppc64le", version = "3.5", path_str = "/opt/python/cp35-cp35m" },
{ identifier = "cp36-manylinux_ppc64le", version = "3.6", path_str = "/opt/python/cp36-cp36m" },
{ identifier = "cp37-manylinux_ppc64le", version = "3.7", path_str = "/opt/python/cp37-cp37m" },
{ identifier = "cp38-manylinux_ppc64le", version = "3.8", path_str = "/opt/python/cp38-cp38" },
{ identifier = "cp39-manylinux_ppc64le", version = "3.9", path_str = "/opt/python/cp39-cp39" },
{ identifier = "cp310-manylinux_ppc64le", version = "3.10", path_str = "/opt/python/cp310-cp310" },
{ identifier = "cp311-manylinux_ppc64le", version = "3.11", path_str = "/opt/python/cp311-cp311" },
{ identifier = "cp312-manylinux_ppc64le", version = "3.12", path_str = "/opt/python/cp312-cp312" },
{ identifier = "cp313-manylinux_ppc64le", version = "3.13", path_str = "/opt/python/cp313-cp313" },
{ identifier = "cp313t-manylinux_ppc64le", version = "3.13", path_str = "/opt/python/cp313-cp313t" },
{ identifier = "cp35-manylinux_s390x", version = "3.5", path_str = "/opt/python/cp35-cp35m" },
{ identifier = "cp36-manylinux_s390x", version = "3.6", path_str = "/opt/python/cp36-cp36m" },
{ identifier = "cp37-manylinux_s390x", version = "3.7", path_str = "/opt/python/cp37-cp37m" },
{ identifier = "cp38-manylinux_s390x", version = "3.8", path_str = "/opt/python/cp38-cp38" },
{ identifier = "cp39-manylinux_s390x", version = "3.9", path_str = "/opt/python/cp39-cp39" },
{ identifier = "cp310-manylinux_s390x", version = "3.10", path_str = "/opt/python/cp310-cp310" },
{ identifier = "cp311-manylinux_s390x", version = "3.11", path_str = "/opt/python/cp311-cp311" },
{ identifier = "cp312-manylinux_s390x", version = "3.12", path_str = "/opt/python/cp312-cp312" },
{ identifier = "cp313-manylinux_s390x", version = "3.13", path_str = "/opt/python/cp313-cp313" },
{ identifier = "cp313t-manylinux_s390x", version = "3.13", path_str = "/opt/python/cp313-cp313t" },
{ identifier = "pp37-manylinux_aarch64", version = "3.7", path_str = "/opt/python/pp37-pypy37_pp73" },
{ identifier = "pp38-manylinux_aarch64", version = "3.8", path_str = "/opt/python/pp38-pypy38_pp73" },
{ identifier = "pp39-manylinux_aarch64", version = "3.9", path_str = "/opt/python/pp39-pypy39_pp73" },
{ identifier = "pp310-manylinux_aarch64", version = "3.10", path_str = "/opt/python/pp310-pypy310_pp73" },
{ identifier = "pp37-manylinux_i686", version = "3.7", path_str = "/opt/python/pp37-pypy37_pp73" },
{ identifier = "pp38-manylinux_i686", version = "3.8", path_str = "/opt/python/pp38-pypy38_pp73" },
{ identifier = "pp39-manylinux_i686", version = "3.9", path_str = "/opt/python/pp39-pypy39_pp73" },
{ identifier = "pp310-manylinux_i686", version = "3.10", path_str = "/opt/python/pp310-pypy310_pp73" },
{ identifier = "cp36-musllinux_x86_64", version = "3.6", path_str = "/opt/python/cp36-cp36m" },
{ identifier = "cp37-musllinux_x86_64", version = "3.7", path_str = "/opt/python/cp37-cp37m" },
{ identifier = "cp38-musllinux_x86_64", version = "3.8", path_str = "/opt/python/cp38-cp38" },
{ identifier = "cp39-musllinux_x86_64", version = "3.9", path_str = "/opt/python/cp39-cp39" },
{ identifier = "cp310-musllinux_x86_64", version = "3.10", path_str = "/opt/python/cp310-cp310" },
{ identifier = "cp311-musllinux_x86_64", version = "3.11", path_str = "/opt/python/cp311-cp311" },
{ identifier = "cp312-musllinux_x86_64", version = "3.12", path_str = "/opt/python/cp312-cp312" },
{ identifier = "cp313-musllinux_x86_64", version = "3.13", path_str = "/opt/python/cp313-cp313" },
{ identifier = "cp313t-musllinux_x86_64", version = "3.13", path_str = "/opt/python/cp313-cp313t" },
{ identifier = "cp36-musllinux_i686", version = "3.6", path_str = "/opt/python/cp36-cp36m" },
{ identifier = "cp37-musllinux_i686", version = "3.7", path_str = "/opt/python/cp37-cp37m" },
{ identifier = "cp38-musllinux_i686", version = "3.8", path_str = "/opt/python/cp38-cp38" },
{ identifier = "cp39-musllinux_i686", version = "3.9", path_str = "/opt/python/cp39-cp39" },
{ identifier = "cp310-musllinux_i686", version = "3.10", path_str = "/opt/python/cp310-cp310" },
{ identifier = "cp311-musllinux_i686", version = "3.11", path_str = "/opt/python/cp311-cp311" },
{ identifier = "cp312-musllinux_i686", version = "3.12", path_str = "/opt/python/cp312-cp312" },
{ identifier = "cp313-musllinux_i686", version = "3.13", path_str = "/opt/python/cp313-cp313" },
{ identifier = "cp313t-musllinux_i686", version = "3.13", path_str = "/opt/python/cp313-cp313t" },
{ identifier = "cp36-musllinux_aarch64", version = "3.6", path_str = "/opt/python/cp36-cp36m" },
{ identifier = "cp37-musllinux_aarch64", version = "3.7", path_str = "/opt/python/cp37-cp37m" },
{ identifier = "cp38-musllinux_aarch64", version = "3.8", path_str = "/opt/python/cp38-cp38" },
{ identifier = "cp39-musllinux_aarch64", version = "3.9", path_str = "/opt/python/cp39-cp39" },
{ identifier = "cp310-musllinux_aarch64", version = "3.10", path_str = "/opt/python/cp310-cp310" },
{ identifier = "cp311-musllinux_aarch64", version = "3.11", path_str = "/opt/python/cp311-cp311" },
{ identifier = "cp312-musllinux_aarch64", version = "3.12", path_str = "/opt/python/cp312-cp312" },
{ identifier = "cp313-musllinux_aarch64", version = "3.13", path_str = "/opt/python/cp313-cp313" },
{ identifier = "cp313t-musllinux_aarch64", version = "3.13", path_str = "/opt/python/cp313-cp313t" },
{ identifier = "cp36-musllinux_ppc64le", version = "3.6", path_str = "/opt/python/cp36-cp36m" },
{ identifier = "cp37-musllinux_ppc64le", version = "3.7", path_str = "/opt/python/cp37-cp37m" },
{ identifier = "cp38-musllinux_ppc64le", version = "3.8", path_str = "/opt/python/cp38-cp38" },
{ identifier = "cp39-musllinux_ppc64le", version = "3.9", path_str = "/opt/python/cp39-cp39" },
{ identifier = "cp310-musllinux_ppc64le", version = "3.10", path_str = "/opt/python/cp310-cp310" },
{ identifier = "cp311-musllinux_ppc64le", version = "3.11", path_str = "/opt/python/cp311-cp311" },
{ identifier = "cp312-musllinux_ppc64le", version = "3.12", path_str = "/opt/python/cp312-cp312" },
{ identifier = "cp313-musllinux_ppc64le", version = "3.13", path_str = "/opt/python/cp313-cp313" },
{ identifier = "cp313t-musllinux_ppc64le", version = "3.13", path_str = "/opt/python/cp313-cp313t" },
{ identifier = "cp36-musllinux_s390x", version = "3.6", path_str = "/opt/python/cp36-cp36m" },
{ identifier = "cp37-musllinux_s390x", version = "3.7", path_str = "/opt/python/cp37-cp37m" },
{ identifier = "cp38-musllinux_s390x", version = "3.8", path_str = "/opt/python/cp38-cp38" },
{ identifier = "cp39-musllinux_s390x", version = "3.9", path_str = "/opt/python/cp39-cp39" },
{ identifier = "cp310-musllinux_s390x", version = "3.10", path_str = "/opt/python/cp310-cp310" },
{ identifier = "cp311-musllinux_s390x", version = "3.11", path_str = "/opt/python/cp311-cp311" },
{ identifier = "cp312-musllinux_s390x", version = "3.12", path_str = "/opt/python/cp312-cp312" },
{ identifier = "cp313-musllinux_s390x", version = "3.13", path_str = "/opt/python/cp313-cp313" },
{ identifier = "cp313t-musllinux_s390x", version = "3.13", path_str = "/opt/python/cp313-cp313t" },
{ identifier = "cp36-musllinux_armv7l", version = "3.6", path_str = "/opt/python/cp36-cp36m" },
{ identifier = "cp37-musllinux_armv7l", version = "3.7", path_str = "/opt/python/cp37-cp37m" },
{ identifier = "cp38-musllinux_armv7l", version = "3.8", path_str = "/opt/python/cp38-cp38" },
{ identifier = "cp39-musllinux_armv7l", version = "3.9", path_str = "/opt/python/cp39-cp39" },
{ identifier = "cp310-musllinux_armv7l", version = "3.10", path_str = "/opt/python/cp310-cp310" },
{ identifier = "cp311-musllinux_armv7l", version = "3.11", path_str = "/opt/python/cp311-cp311" },
{ identifier = "cp312-musllinux_armv7l", version = "3.12", path_str = "/opt/python/cp312-cp312" },
{ identifier = "cp313-musllinux_armv7l", version = "3.13", path_str = "/opt/python/cp313-cp313" },
{ identifier = "cp313t-musllinux_armv7l", version = "3.13", path_str = "/opt/python/cp313-cp313t" },
]
[macos]
python_configurations = [
{ identifier = "cp27-macosx_x86_64", version = "2.7", url = "https://www.python.org/ftp/python/2.7.18/python-2.7.18-macosx10.9.pkg" },
{ identifier = "cp35-macosx_x86_64", version = "3.5", url = "https://www.python.org/ftp/python/3.5.4/python-3.5.4-macosx10.6.pkg" },
{ identifier = "cp36-macosx_x86_64", version = "3.6", url = "https://www.python.org/ftp/python/3.6.8/python-3.6.8-macosx10.9.pkg" },
{ identifier = "cp37-macosx_x86_64", version = "3.7", url = "https://www.python.org/ftp/python/3.7.9/python-3.7.9-macosx10.9.pkg" },
{ identifier = "cp38-macosx_x86_64", version = "3.8", url = "https://www.python.org/ftp/python/3.8.10/python-3.8.10-macosx10.9.pkg" },
{ identifier = "cp38-macosx_arm64", version = "3.8", url = "https://www.python.org/ftp/python/3.8.10/python-3.8.10-macosx10.9.pkg" },
{ identifier = "cp38-macosx_universal2", version = "3.8", url = "https://www.python.org/ftp/python/3.8.10/python-3.8.10-macosx10.9.pkg" },
{ identifier = "cp39-macosx_x86_64", version = "3.9", url = "https://www.python.org/ftp/python/3.9.13/python-3.9.13-macos11.pkg" },
{ identifier = "cp39-macosx_arm64", version = "3.9", url = "https://www.python.org/ftp/python/3.9.13/python-3.9.13-macos11.pkg" },
{ identifier = "cp39-macosx_universal2", version = "3.9", url = "https://www.python.org/ftp/python/3.9.13/python-3.9.13-macos11.pkg" },
{ identifier = "cp310-macosx_x86_64", version = "3.10", url = "https://www.python.org/ftp/python/3.10.11/python-3.10.11-macos11.pkg" },
{ identifier = "cp310-macosx_arm64", version = "3.10", url = "https://www.python.org/ftp/python/3.10.11/python-3.10.11-macos11.pkg" },
{ identifier = "cp310-macosx_universal2", version = "3.10", url = "https://www.python.org/ftp/python/3.10.11/python-3.10.11-macos11.pkg" },
{ identifier = "cp311-macosx_x86_64", version = "3.11", url = "https://www.python.org/ftp/python/3.11.9/python-3.11.9-macos11.pkg" },
{ identifier = "cp311-macosx_arm64", version = "3.11", url = "https://www.python.org/ftp/python/3.11.9/python-3.11.9-macos11.pkg" },
{ identifier = "cp311-macosx_universal2", version = "3.11", url = "https://www.python.org/ftp/python/3.11.9/python-3.11.9-macos11.pkg" },
{ identifier = "cp312-macosx_x86_64", version = "3.12", url = "https://www.python.org/ftp/python/3.12.7/python-3.12.7-macos11.pkg" },
{ identifier = "cp312-macosx_arm64", version = "3.12", url = "https://www.python.org/ftp/python/3.12.7/python-3.12.7-macos11.pkg" },
{ identifier = "cp312-macosx_universal2", version = "3.12", url = "https://www.python.org/ftp/python/3.12.7/python-3.12.7-macos11.pkg" },
{ identifier = "cp313-macosx_x86_64", version = "3.13", url = "https://www.python.org/ftp/python/3.13.0/python-3.13.0-macos11.pkg" },
{ identifier = "cp313-macosx_arm64", version = "3.13", url = "https://www.python.org/ftp/python/3.13.0/python-3.13.0-macos11.pkg" },
{ identifier = "cp313-macosx_universal2", version = "3.13", url = "https://www.python.org/ftp/python/3.13.0/python-3.13.0-macos11.pkg" },
{ identifier = "cp313t-macosx_x86_64", version = "3.13", url = "https://www.python.org/ftp/python/3.13.0/python-3.13.0-macos11.pkg" },
{ identifier = "cp313t-macosx_arm64", version = "3.13", url = "https://www.python.org/ftp/python/3.13.0/python-3.13.0-macos11.pkg" },
{ identifier = "cp313t-macosx_universal2", version = "3.13", url = "https://www.python.org/ftp/python/3.13.0/python-3.13.0-macos11.pkg" },
{ identifier = "pp37-macosx_x86_64", version = "3.7", url = "https://downloads.python.org/pypy/pypy3.7-v7.3.9-osx64.tar.bz2" },
{ identifier = "pp38-macosx_x86_64", version = "3.8", url = "https://downloads.python.org/pypy/pypy3.8-v7.3.11-macos_x86_64.tar.bz2" },
{ identifier = "pp38-macosx_arm64", version = "3.8", url = "https://downloads.python.org/pypy/pypy3.8-v7.3.11-macos_arm64.tar.bz2" },
{ identifier = "pp39-macosx_x86_64", version = "3.9", url = "https://downloads.python.org/pypy/pypy3.9-v7.3.16-macos_x86_64.tar.bz2" },
{ identifier = "pp39-macosx_arm64", version = "3.9", url = "https://downloads.python.org/pypy/pypy3.9-v7.3.16-macos_arm64.tar.bz2" },
{ identifier = "pp310-macosx_x86_64", version = "3.10", url = "https://downloads.python.org/pypy/pypy3.10-v7.3.17-macos_x86_64.tar.bz2" },
{ identifier = "pp310-macosx_arm64", version = "3.10", url = "https://downloads.python.org/pypy/pypy3.10-v7.3.17-macos_arm64.tar.bz2" },
{ identifier = "cp39-macosx_x86_64", version = "3.9", url = "https://www.python.org/ftp/python/3.9.5/python-3.9.5-macos11.pkg" },
{ identifier = "cp39-macosx_arm64", version = "3.9", url = "https://www.python.org/ftp/python/3.9.5/python-3.9.5-macos11.pkg" },
{ identifier = "cp39-macosx_universal2", version = "3.9", url = "https://www.python.org/ftp/python/3.9.5/python-3.9.5-macos11.pkg" },
{ identifier = "pp27-macosx_x86_64", version = "2.7", url = "https://downloads.python.org/pypy/pypy2.7-v7.3.3-osx64.tar.bz2" },
{ identifier = "pp36-macosx_x86_64", version = "3.6", url = "https://downloads.python.org/pypy/pypy3.6-v7.3.3-osx64.tar.bz2" },
{ identifier = "pp37-macosx_x86_64", version = "3.7", url = "https://downloads.python.org/pypy/pypy3.7-v7.3.3-osx64.tar.bz2" },
]
[windows]
python_configurations = [
{ identifier = "cp27-win32", version = "2.7.18", arch = "32" },
{ identifier = "cp27-win_amd64", version = "2.7.18", arch = "64" },
{ identifier = "cp35-win32", version = "3.5.4", arch = "32" },
{ identifier = "cp35-win_amd64", version = "3.5.4", arch = "64" },
{ identifier = "cp36-win32", version = "3.6.8", arch = "32" },
{ identifier = "cp36-win_amd64", version = "3.6.8", arch = "64" },
{ identifier = "cp37-win32", version = "3.7.9", arch = "32" },
{ identifier = "cp37-win_amd64", version = "3.7.9", arch = "64" },
{ identifier = "cp38-win32", version = "3.8.10", arch = "32" },
{ identifier = "cp38-win_amd64", version = "3.8.10", arch = "64" },
{ identifier = "cp39-win32", version = "3.9.13", arch = "32" },
{ identifier = "cp39-win_amd64", version = "3.9.13", arch = "64" },
{ identifier = "cp310-win32", version = "3.10.11", arch = "32" },
{ identifier = "cp310-win_amd64", version = "3.10.11", arch = "64" },
{ identifier = "cp311-win32", version = "3.11.9", arch = "32" },
{ identifier = "cp311-win_amd64", version = "3.11.9", arch = "64" },
{ identifier = "cp312-win32", version = "3.12.7", arch = "32" },
{ identifier = "cp312-win_amd64", version = "3.12.7", arch = "64" },
{ identifier = "cp313-win32", version = "3.13.0", arch = "32" },
{ identifier = "cp313t-win32", version = "3.13.0", arch = "32" },
{ identifier = "cp313-win_amd64", version = "3.13.0", arch = "64" },
{ identifier = "cp313t-win_amd64", version = "3.13.0", arch = "64" },
{ identifier = "cp39-win_arm64", version = "3.9.10", arch = "ARM64" },
{ identifier = "cp310-win_arm64", version = "3.10.11", arch = "ARM64" },
{ identifier = "cp311-win_arm64", version = "3.11.9", arch = "ARM64" },
{ identifier = "cp312-win_arm64", version = "3.12.7", arch = "ARM64" },
{ identifier = "cp313-win_arm64", version = "3.13.0", arch = "ARM64" },
{ identifier = "cp313t-win_arm64", version = "3.13.0", arch = "ARM64" },
{ identifier = "pp37-win_amd64", version = "3.7", arch = "64", url = "https://downloads.python.org/pypy/pypy3.7-v7.3.9-win64.zip" },
{ identifier = "pp38-win_amd64", version = "3.8", arch = "64", url = "https://downloads.python.org/pypy/pypy3.8-v7.3.11-win64.zip" },
{ identifier = "pp39-win_amd64", version = "3.9", arch = "64", url = "https://downloads.python.org/pypy/pypy3.9-v7.3.16-win64.zip" },
{ identifier = "pp310-win_amd64", version = "3.10", arch = "64", url = "https://downloads.python.org/pypy/pypy3.10-v7.3.17-win64.zip" },
]
[pyodide]
python_configurations = [
{ identifier = "cp312-pyodide_wasm32", version = "3.12.1", pyodide_version = "0.26.1", emscripten_version = "3.1.58", node_version = "v20" },
{ identifier = "cp38-win32", version = "3.8.9", arch = "32" },
{ identifier = "cp38-win_amd64", version = "3.8.9", arch = "64" },
{ identifier = "cp39-win32", version = "3.9.4", arch = "32" },
{ identifier = "cp39-win_amd64", version = "3.9.4", arch = "64" },
{ identifier = "pp27-win32", version = "2.7", arch = "32", url = "https://downloads.python.org/pypy/pypy2.7-v7.3.3-win32.zip" },
{ identifier = "pp36-win32", version = "3.6", arch = "32", url = "https://downloads.python.org/pypy/pypy3.6-v7.3.3-win32.zip" },
{ identifier = "pp37-win32", version = "3.7", arch = "32", url = "https://downloads.python.org/pypy/pypy3.7-v7.3.3-win32.zip" },
]
@@ -1,832 +0,0 @@
{
"$schema": "http://json-schema.org/draft-07/schema",
"$id": "https://github.com/pypa/cibuildwheel/blob/main/cibuildwheel/resources/cibuildwheel.schema.json",
"$defs": {
"inherit": {
"enum": [
"none",
"prepend",
"append"
],
"default": "none",
"description": "How to inherit the parent's value."
}
},
"additionalProperties": false,
"description": "cibuildwheel's settings.",
"type": "object",
"properties": {
"archs": {
"description": "Change the architectures built on your machine by default.",
"oneOf": [
{
"type": "string"
},
{
"type": "array",
"items": {
"type": "string"
}
}
],
"title": "CIBW_ARCHS"
},
"before-all": {
"description": "Execute a shell command on the build system before any wheels are built.",
"oneOf": [
{
"type": "string"
},
{
"type": "array",
"items": {
"type": "string"
}
}
],
"title": "CIBW_BEFORE_ALL"
},
"before-build": {
"description": "Execute a shell command preparing each wheel's build.",
"oneOf": [
{
"type": "string"
},
{
"type": "array",
"items": {
"type": "string"
}
}
],
"title": "CIBW_BEFORE_BUILD"
},
"before-test": {
"description": "Execute a shell command before testing each wheel.",
"oneOf": [
{
"type": "string"
},
{
"type": "array",
"items": {
"type": "string"
}
}
],
"title": "CIBW_BEFORE_TEST"
},
"build": {
"default": [
"*"
],
"description": "Choose the Python versions to build.",
"oneOf": [
{
"type": "string"
},
{
"type": "array",
"items": {
"type": "string"
}
}
],
"title": "CIBW_BUILD"
},
"build-frontend": {
"default": "default",
"description": "Set the tool to use to build, either \"pip\" (default for now), \"build\", or \"build[uv]\"",
"oneOf": [
{
"enum": [
"pip",
"build",
"build[uv]",
"default"
]
},
{
"type": "string",
"pattern": "^pip; ?args:"
},
{
"type": "string",
"pattern": "^build; ?args:"
},
{
"type": "string",
"pattern": "^build\\[uv\\]; ?args:"
},
{
"type": "object",
"additionalProperties": false,
"required": [
"name"
],
"properties": {
"name": {
"enum": [
"pip",
"build",
"build[uv]"
]
},
"args": {
"type": "array",
"items": {
"type": "string"
}
}
}
}
],
"title": "CIBW_BUILD_FRONTEND"
},
"build-verbosity": {
"type": "integer",
"minimum": -3,
"maximum": 3,
"default": 0,
"description": "Increase/decrease the output of pip wheel.",
"title": "CIBW_BUILD_VERBOSITY"
},
"config-settings": {
"description": "Specify config-settings for the build backend.",
"oneOf": [
{
"type": "string"
},
{
"type": "object",
"additionalProperties": false,
"patternProperties": {
".+": {
"oneOf": [
{
"type": "string"
},
{
"type": "array",
"items": {
"type": "string"
}
}
]
}
}
}
],
"title": "CIBW_CONFIG_SETTINGS"
},
"container-engine": {
"oneOf": [
{
"enum": [
"docker",
"podman"
]
},
{
"type": "string",
"pattern": "^docker; ?(create_args|disable_host_mount):"
},
{
"type": "string",
"pattern": "^podman; ?(create_args|disable_host_mount):"
},
{
"type": "object",
"additionalProperties": false,
"required": [
"name"
],
"properties": {
"name": {
"enum": [
"docker",
"podman"
]
},
"create-args": {
"type": "array",
"items": {
"type": "string"
}
},
"disable-host-mount": {
"type": "boolean"
}
}
}
],
"title": "CIBW_CONTAINER_ENGINE"
},
"dependency-versions": {
"default": "pinned",
"description": "Specify how cibuildwheel controls the versions of the tools it uses",
"type": "string",
"title": "CIBW_DEPENDENCY_VERSIONS"
},
"environment": {
"description": "Set environment variables needed during the build.",
"oneOf": [
{
"type": "string"
},
{
"type": "object",
"additionalProperties": false,
"patternProperties": {
".+": {
"type": "string"
}
}
}
],
"title": "CIBW_ENVIRONMENT"
},
"environment-pass": {
"description": "Set environment variables on the host to pass-through to the container during the build.",
"oneOf": [
{
"type": "string"
},
{
"type": "array",
"items": {
"type": "string"
}
}
],
"title": "CIBW_ENVIRONMENT_PASS"
},
"free-threaded-support": {
"type": "boolean",
"default": false,
"description": "The project supports free-threaded builds of Python (PEP703)",
"title": "CIBW_FREE_THREADED_SUPPORT"
},
"manylinux-aarch64-image": {
"type": "string",
"description": "Specify alternative manylinux / musllinux container images",
"title": "CIBW_MANYLINUX_AARCH64_IMAGE"
},
"manylinux-i686-image": {
"type": "string",
"description": "Specify alternative manylinux / musllinux container images",
"title": "CIBW_MANYLINUX_I686_IMAGE"
},
"manylinux-ppc64le-image": {
"type": "string",
"description": "Specify alternative manylinux / musllinux container images",
"title": "CIBW_MANYLINUX_PPC64LE_IMAGE"
},
"manylinux-pypy_aarch64-image": {
"type": "string",
"description": "Specify alternative manylinux / musllinux container images",
"title": "CIBW_MANYLINUX_PYPY_AARCH64_IMAGE"
},
"manylinux-pypy_i686-image": {
"type": "string",
"description": "Specify alternative manylinux / musllinux container images",
"title": "CIBW_MANYLINUX_PYPY_I686_IMAGE"
},
"manylinux-pypy_x86_64-image": {
"type": "string",
"description": "Specify alternative manylinux / musllinux container images",
"title": "CIBW_MANYLINUX_PYPY_X86_64_IMAGE"
},
"manylinux-s390x-image": {
"type": "string",
"description": "Specify alternative manylinux / musllinux container images",
"title": "CIBW_MANYLINUX_S390X_IMAGE"
},
"manylinux-x86_64-image": {
"type": "string",
"description": "Specify alternative manylinux / musllinux container images",
"title": "CIBW_MANYLINUX_X86_64_IMAGE"
},
"musllinux-aarch64-image": {
"type": "string",
"description": "Specify alternative manylinux / musllinux container images",
"title": "CIBW_MUSLLINUX_AARCH64_IMAGE"
},
"musllinux-armv7l-image": {
"type": "string",
"description": "Specify alternative manylinux / musllinux container images",
"title": "CIBW_MUSLLINUX_ARMV7L_IMAGE"
},
"musllinux-i686-image": {
"type": "string",
"description": "Specify alternative manylinux / musllinux container images",
"title": "CIBW_MUSLLINUX_I686_IMAGE"
},
"musllinux-ppc64le-image": {
"type": "string",
"description": "Specify alternative manylinux / musllinux container images",
"title": "CIBW_MUSLLINUX_PPC64LE_IMAGE"
},
"musllinux-s390x-image": {
"type": "string",
"description": "Specify alternative manylinux / musllinux container images",
"title": "CIBW_MUSLLINUX_S390X_IMAGE"
},
"musllinux-x86_64-image": {
"type": "string",
"description": "Specify alternative manylinux / musllinux container images",
"title": "CIBW_MUSLLINUX_X86_64_IMAGE"
},
"repair-wheel-command": {
"description": "Execute a shell command to repair each built wheel.",
"oneOf": [
{
"type": "string"
},
{
"type": "array",
"items": {
"type": "string"
}
}
],
"title": "CIBW_REPAIR_WHEEL_COMMAND"
},
"skip": {
"description": "Choose the Python versions to skip.",
"oneOf": [
{
"type": "string"
},
{
"type": "array",
"items": {
"type": "string"
}
}
],
"title": "CIBW_SKIP"
},
"test-command": {
"description": "Execute a shell command to test each built wheel.",
"oneOf": [
{
"type": "string"
},
{
"type": "array",
"items": {
"type": "string"
}
}
],
"title": "CIBW_TEST_COMMAND"
},
"test-extras": {
"description": "Install your wheel for testing using `extras_require`",
"oneOf": [
{
"type": "string"
},
{
"type": "array",
"items": {
"type": "string"
}
}
],
"title": "CIBW_TEST_EXTRAS"
},
"test-requires": {
"description": "Install Python dependencies before running the tests",
"oneOf": [
{
"type": "string"
},
{
"type": "array",
"items": {
"type": "string"
}
}
],
"title": "CIBW_TEST_REQUIRES"
},
"test-skip": {
"description": "Skip running tests on some builds.",
"oneOf": [
{
"type": "string"
},
{
"type": "array",
"items": {
"type": "string"
}
}
],
"title": "CIBW_TEST_SKIP"
},
"overrides": {
"type": "array",
"description": "An overrides array",
"items": {
"type": "object",
"required": [
"select"
],
"minProperties": 2,
"additionalProperties": false,
"properties": {
"select": {
"oneOf": [
{
"type": "string"
},
{
"type": "array",
"items": {
"type": "string"
}
}
]
},
"inherit": {
"type": "object",
"additionalProperties": false,
"properties": {
"before-all": {
"$ref": "#/$defs/inherit"
},
"before-build": {
"$ref": "#/$defs/inherit"
},
"before-test": {
"$ref": "#/$defs/inherit"
},
"config-settings": {
"$ref": "#/$defs/inherit"
},
"container-engine": {
"$ref": "#/$defs/inherit"
},
"environment": {
"$ref": "#/$defs/inherit"
},
"environment-pass": {
"$ref": "#/$defs/inherit"
},
"repair-wheel-command": {
"$ref": "#/$defs/inherit"
},
"test-command": {
"$ref": "#/$defs/inherit"
},
"test-extras": {
"$ref": "#/$defs/inherit"
},
"test-requires": {
"$ref": "#/$defs/inherit"
}
}
},
"before-all": {
"$ref": "#/properties/before-all"
},
"before-build": {
"$ref": "#/properties/before-build"
},
"before-test": {
"$ref": "#/properties/before-test"
},
"build-frontend": {
"$ref": "#/properties/build-frontend"
},
"build-verbosity": {
"$ref": "#/properties/build-verbosity"
},
"config-settings": {
"$ref": "#/properties/config-settings"
},
"container-engine": {
"$ref": "#/properties/container-engine"
},
"dependency-versions": {
"$ref": "#/properties/dependency-versions"
},
"environment": {
"$ref": "#/properties/environment"
},
"environment-pass": {
"$ref": "#/properties/environment-pass"
},
"manylinux-aarch64-image": {
"$ref": "#/properties/manylinux-aarch64-image"
},
"manylinux-i686-image": {
"$ref": "#/properties/manylinux-i686-image"
},
"manylinux-ppc64le-image": {
"$ref": "#/properties/manylinux-ppc64le-image"
},
"manylinux-pypy_aarch64-image": {
"$ref": "#/properties/manylinux-pypy_aarch64-image"
},
"manylinux-pypy_i686-image": {
"$ref": "#/properties/manylinux-pypy_i686-image"
},
"manylinux-pypy_x86_64-image": {
"$ref": "#/properties/manylinux-pypy_x86_64-image"
},
"manylinux-s390x-image": {
"$ref": "#/properties/manylinux-s390x-image"
},
"manylinux-x86_64-image": {
"$ref": "#/properties/manylinux-x86_64-image"
},
"musllinux-aarch64-image": {
"$ref": "#/properties/musllinux-aarch64-image"
},
"musllinux-armv7l-image": {
"$ref": "#/properties/musllinux-armv7l-image"
},
"musllinux-i686-image": {
"$ref": "#/properties/musllinux-i686-image"
},
"musllinux-ppc64le-image": {
"$ref": "#/properties/musllinux-ppc64le-image"
},
"musllinux-s390x-image": {
"$ref": "#/properties/musllinux-s390x-image"
},
"musllinux-x86_64-image": {
"$ref": "#/properties/musllinux-x86_64-image"
},
"repair-wheel-command": {
"$ref": "#/properties/repair-wheel-command"
},
"test-command": {
"$ref": "#/properties/test-command"
},
"test-extras": {
"$ref": "#/properties/test-extras"
},
"test-requires": {
"$ref": "#/properties/test-requires"
}
}
}
},
"linux": {
"type": "object",
"additionalProperties": false,
"properties": {
"archs": {
"$ref": "#/properties/archs"
},
"before-all": {
"$ref": "#/properties/before-all"
},
"before-build": {
"$ref": "#/properties/before-build"
},
"before-test": {
"$ref": "#/properties/before-test"
},
"build-frontend": {
"$ref": "#/properties/build-frontend"
},
"build-verbosity": {
"$ref": "#/properties/build-verbosity"
},
"config-settings": {
"$ref": "#/properties/config-settings"
},
"container-engine": {
"$ref": "#/properties/container-engine"
},
"environment": {
"$ref": "#/properties/environment"
},
"environment-pass": {
"$ref": "#/properties/environment-pass"
},
"manylinux-aarch64-image": {
"$ref": "#/properties/manylinux-aarch64-image"
},
"manylinux-i686-image": {
"$ref": "#/properties/manylinux-i686-image"
},
"manylinux-ppc64le-image": {
"$ref": "#/properties/manylinux-ppc64le-image"
},
"manylinux-pypy_aarch64-image": {
"$ref": "#/properties/manylinux-pypy_aarch64-image"
},
"manylinux-pypy_i686-image": {
"$ref": "#/properties/manylinux-pypy_i686-image"
},
"manylinux-pypy_x86_64-image": {
"$ref": "#/properties/manylinux-pypy_x86_64-image"
},
"manylinux-s390x-image": {
"$ref": "#/properties/manylinux-s390x-image"
},
"manylinux-x86_64-image": {
"$ref": "#/properties/manylinux-x86_64-image"
},
"musllinux-aarch64-image": {
"$ref": "#/properties/musllinux-aarch64-image"
},
"musllinux-armv7l-image": {
"$ref": "#/properties/musllinux-armv7l-image"
},
"musllinux-i686-image": {
"$ref": "#/properties/musllinux-i686-image"
},
"musllinux-ppc64le-image": {
"$ref": "#/properties/musllinux-ppc64le-image"
},
"musllinux-s390x-image": {
"$ref": "#/properties/musllinux-s390x-image"
},
"musllinux-x86_64-image": {
"$ref": "#/properties/musllinux-x86_64-image"
},
"repair-wheel-command": {
"description": "Execute a shell command to repair each built wheel.",
"oneOf": [
{
"type": "string"
},
{
"type": "array",
"items": {
"type": "string"
}
}
],
"title": "CIBW_REPAIR_WHEEL_COMMAND",
"default": "auditwheel repair -w {dest_dir} {wheel}"
},
"test-command": {
"$ref": "#/properties/test-command"
},
"test-extras": {
"$ref": "#/properties/test-extras"
},
"test-requires": {
"$ref": "#/properties/test-requires"
}
}
},
"windows": {
"type": "object",
"additionalProperties": false,
"properties": {
"archs": {
"$ref": "#/properties/archs"
},
"before-all": {
"$ref": "#/properties/before-all"
},
"before-build": {
"$ref": "#/properties/before-build"
},
"before-test": {
"$ref": "#/properties/before-test"
},
"build-frontend": {
"$ref": "#/properties/build-frontend"
},
"build-verbosity": {
"$ref": "#/properties/build-verbosity"
},
"config-settings": {
"$ref": "#/properties/config-settings"
},
"dependency-versions": {
"$ref": "#/properties/dependency-versions"
},
"environment": {
"$ref": "#/properties/environment"
},
"repair-wheel-command": {
"$ref": "#/properties/repair-wheel-command"
},
"test-command": {
"$ref": "#/properties/test-command"
},
"test-extras": {
"$ref": "#/properties/test-extras"
},
"test-requires": {
"$ref": "#/properties/test-requires"
}
}
},
"macos": {
"type": "object",
"additionalProperties": false,
"properties": {
"archs": {
"$ref": "#/properties/archs"
},
"before-all": {
"$ref": "#/properties/before-all"
},
"before-build": {
"$ref": "#/properties/before-build"
},
"before-test": {
"$ref": "#/properties/before-test"
},
"build-frontend": {
"$ref": "#/properties/build-frontend"
},
"build-verbosity": {
"$ref": "#/properties/build-verbosity"
},
"config-settings": {
"$ref": "#/properties/config-settings"
},
"dependency-versions": {
"$ref": "#/properties/dependency-versions"
},
"environment": {
"$ref": "#/properties/environment"
},
"repair-wheel-command": {
"description": "Execute a shell command to repair each built wheel.",
"oneOf": [
{
"type": "string"
},
{
"type": "array",
"items": {
"type": "string"
}
}
],
"title": "CIBW_REPAIR_WHEEL_COMMAND",
"default": "delocate-wheel --require-archs {delocate_archs} -w {dest_dir} -v {wheel}"
},
"test-command": {
"$ref": "#/properties/test-command"
},
"test-extras": {
"$ref": "#/properties/test-extras"
},
"test-requires": {
"$ref": "#/properties/test-requires"
}
}
},
"pyodide": {
"type": "object",
"additionalProperties": false,
"properties": {
"archs": {
"$ref": "#/properties/archs"
},
"before-all": {
"$ref": "#/properties/before-all"
},
"before-build": {
"$ref": "#/properties/before-build"
},
"before-test": {
"$ref": "#/properties/before-test"
},
"build-frontend": {
"$ref": "#/properties/build-frontend"
},
"build-verbosity": {
"$ref": "#/properties/build-verbosity"
},
"config-settings": {
"$ref": "#/properties/config-settings"
},
"dependency-versions": {
"$ref": "#/properties/dependency-versions"
},
"environment": {
"$ref": "#/properties/environment"
},
"repair-wheel-command": {
"$ref": "#/properties/repair-wheel-command"
},
"test-command": {
"$ref": "#/properties/test-command"
},
"test-extras": {
"$ref": "#/properties/test-extras"
},
"test-requires": {
"$ref": "#/properties/test-requires"
}
}
}
}
}
@@ -1,123 +0,0 @@
# This file was autogenerated by uv via the following command:
# nox -s update_constraints
annotated-types==0.7.0
# via pydantic
anyio==4.6.0
# via httpx
auditwheel-emscripten==0.0.16
# via pyodide-build
build==1.2.2.post1
# via
# -r .nox/update_constraints/tmp/constraints-pyodide.in
# pyodide-build
certifi==2024.8.30
# via
# httpcore
# httpx
# requests
charset-normalizer==3.3.2
# via requests
click==8.1.7
# via typer
cloudpickle==3.0.0
# via loky
cmake==3.30.4
# via pyodide-build
distlib==0.3.8
# via virtualenv
filelock==3.16.1
# via virtualenv
h11==0.14.0
# via httpcore
httpcore==1.0.6
# via httpx
httpx==0.27.2
# via unearth
idna==3.10
# via
# anyio
# httpx
# requests
leb128==1.0.8
# via auditwheel-emscripten
loky==3.4.1
# via pyodide-build
markdown-it-py==3.0.0
# via rich
mdurl==0.1.2
# via markdown-it-py
packaging==24.1
# via
# auditwheel-emscripten
# build
# pyodide-build
# unearth
pip==24.2
# via -r .nox/update_constraints/tmp/constraints-pyodide.in
platformdirs==4.3.6
# via virtualenv
pydantic==2.9.2
# via
# pyodide-build
# pyodide-lock
pydantic-core==2.23.4
# via pydantic
pygments==2.18.0
# via rich
pyodide-build==0.26.1
# via -r .nox/update_constraints/tmp/constraints-pyodide.in
pyodide-cli==0.2.4
# via
# auditwheel-emscripten
# pyodide-build
pyodide-lock==0.1.0a6
# via pyodide-build
pyproject-hooks==1.2.0
# via build
pyyaml==6.0.2
# via pyodide-build
requests==2.32.3
# via pyodide-build
resolvelib==1.0.1
# via pyodide-build
rich==13.9.2
# via
# pyodide-build
# pyodide-cli
# typer
ruamel-yaml==0.18.6
# via pyodide-build
ruamel-yaml-clib==0.2.8
# via ruamel-yaml
shellingham==1.5.4
# via typer
sniffio==1.3.1
# via
# anyio
# httpx
typer==0.12.5
# via
# auditwheel-emscripten
# pyodide-build
# pyodide-cli
types-requests==2.32.0.20240914
# via pyodide-build
typing-extensions==4.12.2
# via
# pydantic
# pydantic-core
# typer
unearth==0.17.2
# via pyodide-build
urllib3==2.2.3
# via
# requests
# types-requests
virtualenv==20.26.6
# via
# build
# pyodide-build
wheel==0.44.0
# via
# auditwheel-emscripten
# pyodide-build
@@ -0,0 +1,55 @@
#
# This file is autogenerated by pip-compile
# To update, run:
#
# bin/update_dependencies.py
#
appdirs==1.4.4
# via virtualenv
configparser==4.0.2
# via importlib-metadata
contextlib2==0.6.0.post1
# via
# importlib-metadata
# importlib-resources
delocate==0.8.2
# via -r cibuildwheel/resources/constraints.in
distlib==0.3.1
# via virtualenv
filelock==3.0.12
# via virtualenv
importlib-metadata==2.1.1
# via virtualenv
importlib-resources==3.3.1
# via virtualenv
pathlib2==2.3.5
# via
# importlib-metadata
# importlib-resources
# virtualenv
scandir==1.10.0
# via pathlib2
singledispatch==3.6.1
# via importlib-resources
six==1.15.0
# via
# pathlib2
# virtualenv
typing==3.7.4.3
# via importlib-resources
virtualenv==20.4.3
# via -r cibuildwheel/resources/constraints.in
wheel==0.36.2
# via
# -r cibuildwheel/resources/constraints.in
# delocate
zipp==1.2.0
# via
# importlib-metadata
# importlib-resources
# The following packages are considered to be unsafe in a requirements file:
pip==20.3.4
# via -r cibuildwheel/resources/constraints.in
setuptools==44.1.1
# via -r cibuildwheel/resources/constraints.in
@@ -1,34 +0,0 @@
# This file was autogenerated by uv via the following command:
# nox -s update_constraints
altgraph==0.17.4
# via macholib
build==1.2.2.post1
# via -r cibuildwheel/resources/constraints.in
delocate==0.12.0
# via -r cibuildwheel/resources/constraints.in
distlib==0.3.8
# via virtualenv
filelock==3.16.1
# via virtualenv
importlib-metadata==8.5.0
# via build
macholib==1.16.3
# via delocate
packaging==24.1
# via
# build
# delocate
pip==24.2
# via -r cibuildwheel/resources/constraints.in
platformdirs==4.3.6
# via virtualenv
pyproject-hooks==1.2.0
# via build
tomli==2.0.2
# via build
typing-extensions==4.12.2
# via delocate
virtualenv==20.26.6
# via -r cibuildwheel/resources/constraints.in
zipp==3.20.2
# via importlib-metadata
@@ -1,28 +0,0 @@
# This file was autogenerated by uv via the following command:
# nox -s update_constraints
altgraph==0.17.4
# via macholib
build==1.2.2.post1
# via -r cibuildwheel/resources/constraints.in
delocate==0.12.0
# via -r cibuildwheel/resources/constraints.in
distlib==0.3.8
# via virtualenv
filelock==3.16.1
# via virtualenv
macholib==1.16.3
# via delocate
packaging==24.1
# via
# build
# delocate
pip==24.2
# via -r cibuildwheel/resources/constraints.in
platformdirs==4.3.6
# via virtualenv
pyproject-hooks==1.2.0
# via build
typing-extensions==4.12.2
# via delocate
virtualenv==20.26.6
# via -r cibuildwheel/resources/constraints.in
@@ -1,28 +0,0 @@
# This file was autogenerated by uv via the following command:
# nox -s update_constraints
altgraph==0.17.4
# via macholib
build==1.2.2.post1
# via -r cibuildwheel/resources/constraints.in
delocate==0.12.0
# via -r cibuildwheel/resources/constraints.in
distlib==0.3.8
# via virtualenv
filelock==3.16.1
# via virtualenv
macholib==1.16.3
# via delocate
packaging==24.1
# via
# build
# delocate
pip==24.2
# via -r cibuildwheel/resources/constraints.in
platformdirs==4.3.6
# via virtualenv
pyproject-hooks==1.2.0
# via build
typing-extensions==4.12.2
# via delocate
virtualenv==20.26.6
# via -r cibuildwheel/resources/constraints.in
@@ -1,28 +0,0 @@
# This file was autogenerated by uv via the following command:
# nox -s update_constraints
altgraph==0.17.4
# via macholib
build==1.2.2.post1
# via -r cibuildwheel/resources/constraints.in
delocate==0.12.0
# via -r cibuildwheel/resources/constraints.in
distlib==0.3.8
# via virtualenv
filelock==3.16.1
# via virtualenv
macholib==1.16.3
# via delocate
packaging==24.1
# via
# build
# delocate
pip==24.2
# via -r cibuildwheel/resources/constraints.in
platformdirs==4.3.6
# via virtualenv
pyproject-hooks==1.2.0
# via build
typing-extensions==4.12.2
# via delocate
virtualenv==20.26.6
# via -r cibuildwheel/resources/constraints.in
@@ -0,0 +1,36 @@
#
# This file is autogenerated by pip-compile
# To update, run:
#
# bin/update_dependencies.py
#
appdirs==1.4.4
# via virtualenv
delocate==0.8.2
# via -r cibuildwheel/resources/constraints.in
distlib==0.3.1
# via virtualenv
filelock==3.0.12
# via virtualenv
importlib-metadata==2.1.1
# via virtualenv
importlib-resources==3.2.1
# via virtualenv
six==1.15.0
# via virtualenv
virtualenv==20.4.3
# via -r cibuildwheel/resources/constraints.in
wheel==0.36.2
# via
# -r cibuildwheel/resources/constraints.in
# delocate
zipp==1.2.0
# via
# importlib-metadata
# importlib-resources
# The following packages are considered to be unsafe in a requirements file:
pip==20.3.4
# via -r cibuildwheel/resources/constraints.in
setuptools==50.3.2
# via -r cibuildwheel/resources/constraints.in
+23 -37
View File
@@ -1,52 +1,38 @@
#
# This file is autogenerated by pip-compile with python 3.6
# This file is autogenerated by pip-compile
# To update, run:
#
# nox -s update_constraints-3.6
# bin/update_dependencies.py
#
build==0.9.0
appdirs==1.4.4
# via virtualenv
delocate==0.8.2
# via -r cibuildwheel/resources/constraints.in
delocate==0.10.2
distlib==0.3.1
# via virtualenv
filelock==3.0.12
# via virtualenv
importlib-metadata==3.10.0
# via virtualenv
importlib-resources==5.1.2
# via virtualenv
six==1.15.0
# via virtualenv
typing-extensions==3.7.4.3
# via importlib-metadata
virtualenv==20.4.3
# via -r cibuildwheel/resources/constraints.in
distlib==0.3.8
# via virtualenv
filelock==3.4.1
# via virtualenv
importlib-metadata==4.8.3
# via
# build
# pep517
# virtualenv
importlib-resources==5.4.0
# via virtualenv
packaging==21.3
# via build
pep517==0.13.1
# via build
platformdirs==2.4.0
# via virtualenv
pyparsing==3.1.1
# via packaging
tomli==1.2.3
# via
# build
# pep517
typing-extensions==4.1.1
wheel==0.36.2
# via
# -r cibuildwheel/resources/constraints.in
# delocate
# importlib-metadata
virtualenv==20.17.1
# via -r cibuildwheel/resources/constraints.in
wheel==0.37.1
# via delocate
zipp==3.6.0
zipp==3.4.1
# via
# importlib-metadata
# importlib-resources
# pep517
# The following packages are considered to be unsafe in a requirements file:
pip==21.3.1
pip==21.0.1
# via -r cibuildwheel/resources/constraints.in
setuptools==59.6.0
setuptools==54.2.0
# via -r cibuildwheel/resources/constraints.in
+30 -35
View File
@@ -1,39 +1,34 @@
# This file was autogenerated by uv via the following command:
# nox -s update_constraints
altgraph==0.17.4
# via macholib
build==1.1.1
# via -r cibuildwheel/resources/constraints.in
delocate==0.12.0
# via -r cibuildwheel/resources/constraints.in
distlib==0.3.8
#
# This file is autogenerated by pip-compile
# To update, run:
#
# bin/update_dependencies.py
#
appdirs==1.4.4
# via virtualenv
filelock==3.12.2
# via virtualenv
importlib-metadata==6.7.0
# via
# build
# virtualenv
macholib==1.16.3
# via delocate
packaging==24.0
# via
# build
# delocate
pip==24.0
delocate==0.8.2
# via -r cibuildwheel/resources/constraints.in
platformdirs==4.0.0
distlib==0.3.1
# via virtualenv
pyproject-hooks==1.2.0
# via build
tomli==2.0.1
# via build
typing-extensions==4.7.1
# via
# delocate
# importlib-metadata
# platformdirs
virtualenv==20.26.6
# via -r cibuildwheel/resources/constraints.in
zipp==3.15.0
filelock==3.0.12
# via virtualenv
importlib-metadata==3.10.0
# via virtualenv
six==1.15.0
# via virtualenv
typing-extensions==3.7.4.3
# via importlib-metadata
virtualenv==20.4.3
# via -r cibuildwheel/resources/constraints.in
wheel==0.36.2
# via
# -r cibuildwheel/resources/constraints.in
# delocate
zipp==3.4.1
# via importlib-metadata
# The following packages are considered to be unsafe in a requirements file:
pip==21.0.1
# via -r cibuildwheel/resources/constraints.in
setuptools==54.2.0
# via -r cibuildwheel/resources/constraints.in
+22 -28
View File
@@ -1,34 +1,28 @@
# This file was autogenerated by uv via the following command:
# nox -s update_constraints
altgraph==0.17.4
# via macholib
build==1.2.2.post1
# via -r cibuildwheel/resources/constraints.in
delocate==0.12.0
# via -r cibuildwheel/resources/constraints.in
distlib==0.3.8
#
# This file is autogenerated by pip-compile
# To update, run:
#
# bin/update_dependencies.py
#
appdirs==1.4.4
# via virtualenv
filelock==3.16.1
delocate==0.8.2
# via -r cibuildwheel/resources/constraints.in
distlib==0.3.1
# via virtualenv
importlib-metadata==8.5.0
# via build
macholib==1.16.3
# via delocate
packaging==24.1
filelock==3.0.12
# via virtualenv
six==1.15.0
# via virtualenv
virtualenv==20.4.3
# via -r cibuildwheel/resources/constraints.in
wheel==0.36.2
# via
# build
# -r cibuildwheel/resources/constraints.in
# delocate
pip==24.2
# The following packages are considered to be unsafe in a requirements file:
pip==21.0.1
# via -r cibuildwheel/resources/constraints.in
platformdirs==4.3.6
# via virtualenv
pyproject-hooks==1.2.0
# via build
tomli==2.0.2
# via build
typing-extensions==4.12.2
# via delocate
virtualenv==20.26.6
setuptools==54.2.0
# via -r cibuildwheel/resources/constraints.in
zipp==3.20.2
# via importlib-metadata
+22 -28
View File
@@ -1,34 +1,28 @@
# This file was autogenerated by uv via the following command:
# nox -s update_constraints
altgraph==0.17.4
# via macholib
build==1.2.2.post1
# via -r cibuildwheel/resources/constraints.in
delocate==0.12.0
# via -r cibuildwheel/resources/constraints.in
distlib==0.3.8
#
# This file is autogenerated by pip-compile
# To update, run:
#
# bin/update_dependencies.py
#
appdirs==1.4.4
# via virtualenv
filelock==3.16.1
delocate==0.8.2
# via -r cibuildwheel/resources/constraints.in
distlib==0.3.1
# via virtualenv
importlib-metadata==8.5.0
# via build
macholib==1.16.3
# via delocate
packaging==24.1
filelock==3.0.12
# via virtualenv
six==1.15.0
# via virtualenv
virtualenv==20.4.3
# via -r cibuildwheel/resources/constraints.in
wheel==0.36.2
# via
# build
# -r cibuildwheel/resources/constraints.in
# delocate
pip==24.2
# The following packages are considered to be unsafe in a requirements file:
pip==21.0.1
# via -r cibuildwheel/resources/constraints.in
platformdirs==4.3.6
# via virtualenv
pyproject-hooks==1.2.0
# via build
tomli==2.0.2
# via build
typing-extensions==4.12.2
# via delocate
virtualenv==20.26.6
setuptools==54.2.0
# via -r cibuildwheel/resources/constraints.in
zipp==3.20.2
# via importlib-metadata
+2 -1
View File
@@ -1,4 +1,5 @@
pip
build
setuptools
wheel
delocate
virtualenv
+22 -22
View File
@@ -1,28 +1,28 @@
# This file was autogenerated by uv via the following command:
# nox -s update_constraints
altgraph==0.17.4
# via macholib
build==1.2.2.post1
# via -r cibuildwheel/resources/constraints.in
delocate==0.12.0
# via -r cibuildwheel/resources/constraints.in
distlib==0.3.8
#
# This file is autogenerated by pip-compile
# To update, run:
#
# bin/update_dependencies.py
#
appdirs==1.4.4
# via virtualenv
filelock==3.16.1
delocate==0.8.2
# via -r cibuildwheel/resources/constraints.in
distlib==0.3.1
# via virtualenv
macholib==1.16.3
# via delocate
packaging==24.1
filelock==3.0.12
# via virtualenv
six==1.15.0
# via virtualenv
virtualenv==20.4.3
# via -r cibuildwheel/resources/constraints.in
wheel==0.36.2
# via
# build
# -r cibuildwheel/resources/constraints.in
# delocate
pip==24.2
# The following packages are considered to be unsafe in a requirements file:
pip==21.0.1
# via -r cibuildwheel/resources/constraints.in
platformdirs==4.3.6
# via virtualenv
pyproject-hooks==1.2.0
# via build
typing-extensions==4.12.2
# via delocate
virtualenv==20.26.6
setuptools==54.2.0
# via -r cibuildwheel/resources/constraints.in
-51
View File
@@ -1,51 +0,0 @@
[tool.cibuildwheel]
build = "*"
skip = ""
test-skip = ""
free-threaded-support = false
archs = ["auto"]
build-frontend = "default"
config-settings = {}
dependency-versions = "pinned"
environment = {}
environment-pass = []
build-verbosity = 0
before-all = ""
before-build = ""
repair-wheel-command = ""
test-command = ""
before-test = ""
test-requires = []
test-extras = []
container-engine = "docker"
manylinux-x86_64-image = "manylinux2014"
manylinux-i686-image = "manylinux2014"
manylinux-aarch64-image = "manylinux2014"
manylinux-ppc64le-image = "manylinux2014"
manylinux-s390x-image = "manylinux2014"
manylinux-pypy_x86_64-image = "manylinux2014"
manylinux-pypy_i686-image = "manylinux2014"
manylinux-pypy_aarch64-image = "manylinux2014"
musllinux-x86_64-image = "musllinux_1_2"
musllinux-i686-image = "musllinux_1_2"
musllinux-aarch64-image = "musllinux_1_2"
musllinux-ppc64le-image = "musllinux_1_2"
musllinux-s390x-image = "musllinux_1_2"
musllinux-armv7l-image = "musllinux_1_2"
[tool.cibuildwheel.linux]
repair-wheel-command = "auditwheel repair -w {dest_dir} {wheel}"
[tool.cibuildwheel.macos]
repair-wheel-command = "delocate-wheel --require-archs {delocate_archs} -w {dest_dir} -v {wheel}"
[tool.cibuildwheel.windows]
[tool.cibuildwheel.pyodide]
@@ -1,14 +0,0 @@
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<array>
<dict>
<key>attributeSetting</key>
<integer>1</integer>
<key>choiceAttribute</key>
<string>selected</string>
<key>choiceIdentifier</key>
<string>org.python.Python.PythonTFramework-3.13</string>
</dict>
</array>
</plist>
File diff suppressed because it is too large Load Diff
+6 -2
View File
@@ -6,7 +6,6 @@
# for the ssl module. Uses the certificates provided by the certifi package:
# https://pypi.org/project/certifi/
import contextlib
import os
import os.path
import ssl
@@ -25,6 +24,9 @@ STAT_0o775 = (
| stat.S_IXOTH
)
if sys.version_info[0] == 2:
FileNotFoundError = OSError
def main():
openssl_dir, openssl_cafile = os.path.split(ssl.get_default_verify_paths().openssl_cafile)
@@ -40,8 +42,10 @@ def main():
relpath_to_certifi_cafile = os.path.relpath(certifi.where())
print(" -- removing any existing file or link")
with contextlib.suppress(FileNotFoundError):
try:
os.remove(openssl_cafile)
except FileNotFoundError:
pass
print(" -- creating symlink to certifi certificate bundle")
os.symlink(relpath_to_certifi_cafile, openssl_cafile)
-2
View File
@@ -1,2 +0,0 @@
url = "https://nodejs.org/dist/"
v20 = "v20.18.0"
+15 -46
View File
@@ -1,57 +1,26 @@
[x86_64]
manylinux1 = quay.io/pypa/manylinux1_x86_64:2024-04-29-76807b8
manylinux2010 = quay.io/pypa/manylinux2010_x86_64:2022-08-05-4535177
manylinux2014 = quay.io/pypa/manylinux2014_x86_64:2024.10.07-1
manylinux_2_24 = quay.io/pypa/manylinux_2_24_x86_64:2022-12-26-0d38463
manylinux_2_28 = quay.io/pypa/manylinux_2_28_x86_64:2024.10.07-1
musllinux_1_1 = quay.io/pypa/musllinux_1_1_x86_64:2024.10.07-1
musllinux_1_2 = quay.io/pypa/musllinux_1_2_x86_64:2024.10.07-1
manylinux1 = quay.io/pypa/manylinux1_x86_64:2021-05-05-b64d921
manylinux2010 = quay.io/pypa/manylinux2010_x86_64:2021-02-06-3d322a5
manylinux2014 = quay.io/pypa/manylinux2014_x86_64:2021-05-01-28d233a
manylinux_2_24 = quay.io/pypa/manylinux_2_24_x86_64:2021-05-01-28d233a
[i686]
manylinux1 = quay.io/pypa/manylinux1_i686:2024-04-29-76807b8
manylinux2010 = quay.io/pypa/manylinux2010_i686:2022-08-05-4535177
manylinux2014 = quay.io/pypa/manylinux2014_i686:2024.10.07-1
manylinux_2_24 = quay.io/pypa/manylinux_2_24_i686:2022-12-26-0d38463
musllinux_1_1 = quay.io/pypa/musllinux_1_1_i686:2024.10.07-1
musllinux_1_2 = quay.io/pypa/musllinux_1_2_i686:2024.10.07-1
manylinux1 = quay.io/pypa/manylinux1_i686:2021-05-05-b64d921
manylinux2010 = quay.io/pypa/manylinux2010_i686:2021-02-06-3d322a5
manylinux2014 = quay.io/pypa/manylinux2014_i686:2021-05-01-28d233a
manylinux_2_24 = quay.io/pypa/manylinux_2_24_i686:2021-05-01-28d233a
[pypy_x86_64]
manylinux2010 = quay.io/pypa/manylinux2010_x86_64:2022-08-05-4535177
manylinux2014 = quay.io/pypa/manylinux2014_x86_64:2024.10.07-1
manylinux_2_24 = quay.io/pypa/manylinux_2_24_x86_64:2022-12-26-0d38463
manylinux_2_28 = quay.io/pypa/manylinux_2_28_x86_64:2024.10.07-1
[pypy_i686]
manylinux2010 = quay.io/pypa/manylinux2010_i686:2022-08-05-4535177
manylinux2014 = quay.io/pypa/manylinux2014_i686:2024.10.07-1
manylinux_2_24 = quay.io/pypa/manylinux_2_24_i686:2022-12-26-0d38463
manylinux2010 = pypywheels/manylinux2010-pypy_x86_64:2020-12-11-f1e0e80
[aarch64]
manylinux2014 = quay.io/pypa/manylinux2014_aarch64:2024.10.07-1
manylinux_2_24 = quay.io/pypa/manylinux_2_24_aarch64:2022-12-26-0d38463
manylinux_2_28 = quay.io/pypa/manylinux_2_28_aarch64:2024.10.07-1
musllinux_1_1 = quay.io/pypa/musllinux_1_1_aarch64:2024.10.07-1
musllinux_1_2 = quay.io/pypa/musllinux_1_2_aarch64:2024.10.07-1
manylinux2014 = quay.io/pypa/manylinux2014_aarch64:2021-05-01-28d233a
manylinux_2_24 = quay.io/pypa/manylinux_2_24_aarch64:2021-05-01-28d233a
[ppc64le]
manylinux2014 = quay.io/pypa/manylinux2014_ppc64le:2024.10.07-1
manylinux_2_24 = quay.io/pypa/manylinux_2_24_ppc64le:2022-12-26-0d38463
manylinux_2_28 = quay.io/pypa/manylinux_2_28_ppc64le:2024.10.07-1
musllinux_1_1 = quay.io/pypa/musllinux_1_1_ppc64le:2024.10.07-1
musllinux_1_2 = quay.io/pypa/musllinux_1_2_ppc64le:2024.10.07-1
manylinux2014 = quay.io/pypa/manylinux2014_ppc64le:2021-05-01-28d233a
manylinux_2_24 = quay.io/pypa/manylinux_2_24_ppc64le:2021-05-01-28d233a
[s390x]
manylinux2014 = quay.io/pypa/manylinux2014_s390x:2024.10.07-1
manylinux_2_24 = quay.io/pypa/manylinux_2_24_s390x:2022-12-26-0d38463
manylinux_2_28 = quay.io/pypa/manylinux_2_28_s390x:2024.10.07-1
musllinux_1_1 = quay.io/pypa/musllinux_1_1_s390x:2024.10.07-1
musllinux_1_2 = quay.io/pypa/musllinux_1_2_s390x:2024.10.07-1
[pypy_aarch64]
manylinux2014 = quay.io/pypa/manylinux2014_aarch64:2024.10.07-1
manylinux_2_24 = quay.io/pypa/manylinux_2_24_aarch64:2022-12-26-0d38463
manylinux_2_28 = quay.io/pypa/manylinux_2_28_aarch64:2024.10.07-1
[armv7l]
musllinux_1_2 = quay.io/pypa/musllinux_1_2_armv7l:2024.10.07-1
manylinux2014 = quay.io/pypa/manylinux2014_s390x:2021-05-01-28d233a
manylinux_2_24 = quay.io/pypa/manylinux_2_24_s390x:2021-05-01-28d233a
+36
View File
@@ -0,0 +1,36 @@
diff --git a/lib-python/3/sysconfig.py b/lib-python/3/sysconfig.py
index 0d08cde66b4150e1772c13fdc6fe34d079c2b10d..30c77d165a2505d98bc7660ff0cc3b728f56c682 100644
--- a/lib-python/3/sysconfig.py
+++ b/lib-python/3/sysconfig.py
@@ -618,9 +618,6 @@ def get_config_vars(*args):
import _osx_support
_osx_support.customize_config_vars(_CONFIG_VARS)
- _CONFIG_VARS['INCLUDEPY'] = os.path.join(_CONFIG_VARS['prefix'],
- 'include')
-
if args:
vals = []
for name in args:
diff --git a/lib_pypy/_sysconfigdata.py b/lib_pypy/_sysconfigdata.py
index f1cca41a4b7553819bb1a2a63af1c1a9b8591ce9..a34216e8e12b82abd72fabdd7793b3e8bb19bbcf 100644
--- a/lib_pypy/_sysconfigdata.py
+++ b/lib_pypy/_sysconfigdata.py
@@ -5,6 +5,7 @@ from distutils.spawn import find_executable
so_ext = _imp.extension_suffixes()[0]
+mybase = os.path.dirname(os.path.dirname(__file__))
build_time_vars = {
# SOABI is PEP 3149 compliant, but CPython3 has so_ext.split('.')[1]
@@ -25,7 +26,8 @@ build_time_vars = {
'AR': "ar",
'ARFLAGS': "rc",
'EXE': "",
- 'LIBDIR': os.path.join(sys.prefix, 'bin'),
+ 'LIBDIR': os.path.join(mybase, 'lib'),
+ 'INCLUDEPY': os.path.join(mybase, 'include'),
}
if find_executable("gcc"):
+27
View File
@@ -0,0 +1,27 @@
--- a/lib-python/2.7/sysconfig.py
+++ b/lib-python/2.7/sysconfig.py
@@ -553,9 +553,6 @@ def get_config_vars(*args):
if type_ == imp.C_EXTENSION:
_CONFIG_VARS['SOABI'] = suffix.split('.')[1]
break
- _CONFIG_VARS['INCLUDEPY'] = os.path.join(_CONFIG_VARS['prefix'],
- 'include')
-
if args:
vals = []
for name in args:
--- a/lib_pypy/_sysconfigdata.py
+++ b/lib_pypy/_sysconfigdata.py
@@ -1,5 +1,11 @@
-import imp
+import imp, os
+
+mybase = os.path.dirname(os.path.dirname(__file__))
build_time_vars = {
"SO": [s[0] for s in imp.get_suffixes() if s[2] == imp.C_EXTENSION][0]
}
+
+build_time_vars['LIBDIR'] = os.path.join(mybase, 'lib')
+build_time_vars['INCLUDEPY'] = os.path.join(mybase, 'include')
+
@@ -1,17 +0,0 @@
# this file is copied to the testing cwd, to raise the below error message if
# pytest/unittest is run from there.
import unittest
class TestStringMethods(unittest.TestCase):
def test_fail(self):
self.fail(
"cibuildwheel executes tests from a different working directory to "
"your project. This ensures only your wheel is imported, preventing "
"Python from accessing files that haven't been packaged into the "
"wheel. Please specify a path to your tests when invoking pytest "
"using the {project} placeholder, e.g. `pytest {project}` or "
"`pytest {project}/tests`. cibuildwheel will replace {project} with "
"the path to your project."
)
-2
View File
@@ -1,2 +0,0 @@
py36 = { version = "20.21.1", url = "https://github.com/pypa/get-virtualenv/blob/20.21.1/public/virtualenv.pyz?raw=true" }
default = { version = "20.26.6", url = "https://github.com/pypa/get-virtualenv/blob/20.26.6/public/virtualenv.pyz?raw=true" }
-15
View File
@@ -1,15 +0,0 @@
from __future__ import annotations
import json
from pathlib import Path
from typing import Any
DIR = Path(__file__).parent.resolve()
def get_schema(tool_name: str = "cibuildwheel") -> dict[str, Any]:
"Get the stored complete schema for cibuildwheel settings."
assert tool_name == "cibuildwheel", "Only cibuildwheel is supported."
with DIR.joinpath("resources/cibuildwheel.schema.json").open(encoding="utf-8") as f:
return json.load(f) # type: ignore[no-any-return]
+20 -12
View File
@@ -1,20 +1,29 @@
from __future__ import annotations
import os
import subprocess
import typing
from typing import Final, Literal, Protocol, Union
import sys
from typing import TYPE_CHECKING, NoReturn, Set, Union
if sys.version_info < (3, 8):
from typing_extensions import Final, Literal, TypedDict
else:
from typing import Final, Literal, TypedDict
__all__ = (
"PLATFORMS",
"Final",
"Literal",
"TypedDict",
"Set",
"Union",
"PopenBytes",
"PathOrStr",
"PlatformName",
"PLATFORMS",
"PopenBytes",
"assert_never",
)
if typing.TYPE_CHECKING:
if TYPE_CHECKING:
PopenBytes = subprocess.Popen[bytes]
PathOrStr = Union[str, os.PathLike[str]]
else:
@@ -22,10 +31,9 @@ else:
PathOrStr = Union[str, "os.PathLike[str]"]
PlatformName = Literal["linux", "macos", "windows", "pyodide"]
PLATFORMS: Final[set[PlatformName]] = {"linux", "macos", "windows", "pyodide"}
PlatformName = Literal["linux", "macos", "windows"]
PLATFORMS: Final[Set[PlatformName]] = {"linux", "macos", "windows"}
class GenericPythonConfiguration(Protocol):
@property
def identifier(self) -> str: ...
def assert_never(value: NoReturn) -> NoReturn:
assert False, f"Unhandled value: {value} ({type(value).__name__})" # noqa: B011
+90 -763
View File
@@ -1,188 +1,31 @@
from __future__ import annotations
import contextlib
import fnmatch
import itertools
import os
import re
import shlex
import shutil
import ssl
import subprocess
import sys
import tarfile
import textwrap
import time
import typing
import urllib.request
from collections import defaultdict
from collections.abc import Generator, Iterable, Mapping, MutableMapping, Sequence
from dataclasses import dataclass
from enum import Enum
from functools import lru_cache, total_ordering
from pathlib import Path, PurePath
from tempfile import TemporaryDirectory
from pathlib import Path
from time import sleep
from typing import Any, ClassVar, Final, Literal, TextIO, TypeVar
from zipfile import ZipFile
from typing import Dict, Iterator, List, NamedTuple, Optional, Set
import bracex
import certifi
from filelock import FileLock
from packaging.requirements import InvalidRequirement, Requirement
import toml
from packaging.specifiers import SpecifierSet
from packaging.utils import parse_wheel_filename
from packaging.version import Version
from platformdirs import user_cache_path
from ._compat import tomllib
from .architecture import Architecture
from .environment import ParsedEnvironment
from .typing import PathOrStr, PlatformName
__all__ = [
"MANYLINUX_ARCHS",
"call",
"chdir",
"combine_constraints",
"find_compatible_wheel",
"find_uv",
"format_safe",
"get_build_verbosity_extra_flags",
"prepare_command",
"read_python_configs",
"resources_dir",
"selector_matches",
"shell",
"split_config_settings",
"strtobool",
]
resources_dir = Path(__file__).parent / "resources"
resources_dir: Final[Path] = Path(__file__).parent / "resources"
install_certifi_script: Final[Path] = resources_dir / "install_certifi.py"
free_thread_enable_313: Final[Path] = resources_dir / "free-threaded-enable-313.xml"
test_fail_cwd_file: Final[Path] = resources_dir / "testing_temp_dir_file.py"
MANYLINUX_ARCHS: Final[tuple[str, ...]] = (
"x86_64",
"i686",
"pypy_x86_64",
"aarch64",
"ppc64le",
"s390x",
"pypy_aarch64",
"pypy_i686",
)
MUSLLINUX_ARCHS: Final[tuple[str, ...]] = (
"x86_64",
"i686",
"aarch64",
"ppc64le",
"s390x",
"armv7l",
)
DEFAULT_CIBW_CACHE_PATH: Final[Path] = user_cache_path(appname="cibuildwheel", appauthor="pypa")
CIBW_CACHE_PATH: Final[Path] = Path(
os.environ.get("CIBW_CACHE_PATH", DEFAULT_CIBW_CACHE_PATH)
).resolve()
IS_WIN: Final[bool] = sys.platform.startswith("win")
@typing.overload
def call(
*args: PathOrStr,
env: Mapping[str, str] | None = None,
cwd: PathOrStr | None = None,
capture_stdout: Literal[False] = ...,
) -> None: ...
@typing.overload
def call(
*args: PathOrStr,
env: Mapping[str, str] | None = None,
cwd: PathOrStr | None = None,
capture_stdout: Literal[True],
) -> str: ...
def call(
*args: PathOrStr,
env: Mapping[str, str] | None = None,
cwd: PathOrStr | None = None,
capture_stdout: bool = False,
) -> str | None:
"""
Run subprocess.run, but print the commands first. Takes the commands as
*args. Uses shell=True on Windows due to a bug. Also converts to
Paths to strings, due to Windows behavior at least on older Pythons.
https://bugs.python.org/issue8557
"""
args_ = [str(arg) for arg in args]
# print the command executing for the logs
print("+ " + " ".join(shlex.quote(a) for a in args_))
kwargs: dict[str, Any] = {}
if capture_stdout:
kwargs["universal_newlines"] = True
kwargs["stdout"] = subprocess.PIPE
result = subprocess.run(args_, check=True, shell=IS_WIN, env=env, cwd=cwd, **kwargs)
if not capture_stdout:
return None
return typing.cast(str, result.stdout)
def shell(
*commands: str, env: Mapping[str, str] | None = None, cwd: PathOrStr | None = None
) -> None:
command = " ".join(commands)
print(f"+ {command}")
subprocess.run(command, env=env, cwd=cwd, shell=True, check=True)
def format_safe(template: str, **kwargs: str | os.PathLike[str]) -> str:
"""
Works similarly to `template.format(**kwargs)`, except that unmatched
fields in `template` are passed through untouched.
>>> format_safe('{a} {b}', a='123')
'123 {b}'
>>> format_safe('{a} {b[4]:3f}', a='123')
'123 {b[4]:3f}'
To avoid variable expansion, precede with a single backslash e.g.
>>> format_safe('\\{a} {b}', a='123')
'{a} {b}'
"""
result = template
for key, value in kwargs.items():
find_pattern = re.compile(
rf"""
(?<!\#) # don't match if preceded by a hash
{{ # literal open curly bracket
{re.escape(key)} # the field name
}} # literal close curly bracket
""",
re.VERBOSE,
)
result = re.sub(
pattern=find_pattern,
repl=str(value).replace("\\", r"\\"),
string=result,
)
# transform escaped sequences into their literal equivalents
result = result.replace(f"#{{{key}}}", f"{{{key}}}")
return result
get_pip_script = resources_dir / "get-pip.py"
install_certifi_script = resources_dir / "install_certifi.py"
def prepare_command(command: str, **kwargs: PathOrStr) -> str:
@@ -190,12 +33,12 @@ def prepare_command(command: str, **kwargs: PathOrStr) -> str:
Preprocesses a command by expanding variables like {python}.
For example, used in the test_command option to specify the path to the
project's root. Unmatched syntax will mostly be allowed through.
project's root.
"""
return format_safe(command, python="python", pip="pip", **kwargs)
return command.format(python="python", pip="pip", **kwargs)
def get_build_verbosity_extra_flags(level: int) -> list[str]:
def get_build_verbosity_extra_flags(level: int) -> List[str]:
if level > 0:
return ["-" + level * "v"]
elif level < 0:
@@ -204,39 +47,14 @@ def get_build_verbosity_extra_flags(level: int) -> list[str]:
return []
def split_config_settings(
config_settings: str, frontend: Literal["pip", "build", "build[uv]"]
) -> list[str]:
config_settings_list = shlex.split(config_settings)
s = "s" if frontend == "pip" else ""
return [f"--config-setting{s}={setting}" for setting in config_settings_list]
def read_python_configs(config: PlatformName) -> list[dict[str, str]]:
def read_python_configs(config: PlatformName) -> List[Dict[str, str]]:
input_file = resources_dir / "build-platforms.toml"
with input_file.open("rb") as f:
loaded_file = tomllib.load(f)
results: list[dict[str, str]] = list(loaded_file[config]["python_configurations"])
loaded_file = toml.load(input_file)
results: List[Dict[str, str]] = list(loaded_file[config]["python_configurations"])
return results
def selector_matches(patterns: str, string: str) -> bool:
"""
Returns True if `string` is matched by any of the wildcard patterns in
`patterns`.
Matching is according to fnmatch, but with shell-like curly brace
expansion. For example, 'cp{36,37}-*' would match either of 'cp36-*' or
'cp37-*'.
"""
patterns_list = patterns.split()
expanded_patterns = itertools.chain.from_iterable(bracex.expand(p) for p in patterns_list)
return any(fnmatch.fnmatch(string, pat) for pat in expanded_patterns)
# Once we require Python 3.10+, we can add kw_only=True
@dataclass(frozen=True)
class BuildSelector:
class IdentifierSelector:
"""
This class holds a set of build/skip patterns. You call an instance with a
build identifier, and it returns True if that identifier should be
@@ -244,81 +62,64 @@ class BuildSelector:
numeric digits before the first dash.
"""
build_config: str
skip_config: str
requires_python: SpecifierSet | None = None
# a pattern that skips prerelease versions, when include_prereleases is False.
PRERELEASE_SKIP: ClassVar[str] = ""
prerelease_pythons: bool = False
free_threaded_support: bool = False
def __init__(
self, *, build_config: str, skip_config: str, requires_python: Optional[SpecifierSet] = None
):
self.build_patterns = build_config.split()
self.skip_patterns = skip_config.split()
self.requires_python = requires_python
def __call__(self, build_id: str) -> bool:
# Filter build selectors by python_requires if set
if self.requires_python is not None:
py_ver_str = build_id.split("-")[0]
if py_ver_str.endswith("t"):
py_ver_str = py_ver_str[:-1]
major = int(py_ver_str[2])
minor = int(py_ver_str[3:])
version = Version(f"{major}.{minor}.99")
if not self.requires_python.contains(version):
return False
# filter out the prerelease pythons if self.prerelease_pythons is False
if not self.prerelease_pythons and selector_matches(self.PRERELEASE_SKIP, build_id):
return False
build_patterns = itertools.chain.from_iterable(
bracex.expand(p) for p in self.build_patterns
)
skip_patterns = itertools.chain.from_iterable(bracex.expand(p) for p in self.skip_patterns)
# filter out free threaded pythons if self.free_threaded_support is False
if not self.free_threaded_support and selector_matches("*t-*", build_id):
return False
build: bool = any(fnmatch.fnmatch(build_id, pat) for pat in build_patterns)
skip: bool = any(fnmatch.fnmatch(build_id, pat) for pat in skip_patterns)
return build and not skip
should_build = selector_matches(self.build_config, build_id)
should_skip = selector_matches(self.skip_config, build_id)
return should_build and not should_skip
def options_summary(self) -> Any:
return {
"build_config": self.build_config,
"skip_config": self.skip_config,
"requires_python": str(self.requires_python),
"prerelease_pythons": self.prerelease_pythons,
"free_threaded_support": self.free_threaded_support,
}
def __repr__(self) -> str:
if not self.skip_patterns:
return f'{self.__class__.__name__}({" ".join(self.build_patterns)!r})'
else:
return f'{self.__class__.__name__}({" ".join(self.build_patterns)!r} - {" ".join(self.skip_patterns)!r})'
@dataclass(frozen=True)
class TestSelector:
"""
A build selector that can only skip tests according to a skip pattern.
"""
class BuildSelector(IdentifierSelector):
pass
skip_config: str
def __call__(self, build_id: str) -> bool:
should_skip = selector_matches(self.skip_config, build_id)
return not should_skip
def options_summary(self) -> Any:
return {"skip_config": self.skip_config}
# Note that requires-python is not needed for TestSelector, as you can't test
# what you can't build.
class TestSelector(IdentifierSelector):
def __init__(self, *, skip_config: str):
super().__init__(build_config="*", skip_config=skip_config)
# Taken from https://stackoverflow.com/a/107717
class Unbuffered:
def __init__(self, stream: TextIO) -> None:
def __init__(self, stream): # type: ignore
self.stream = stream
def write(self, data: str) -> None:
def write(self, data): # type: ignore
self.stream.write(data)
self.stream.flush()
def writelines(self, data: Iterable[str]) -> None:
self.stream.writelines(data)
def writelines(self, datas): # type: ignore
self.stream.writelines(datas)
self.stream.flush()
def __getattr__(self, attr: str) -> Any:
def __getattr__(self, attr): # type: ignore
return getattr(self.stream, attr)
@@ -335,63 +136,18 @@ def download(url: str, dest: Path) -> None:
repeat_num = 3
for i in range(repeat_num):
try:
with urllib.request.urlopen(url, context=context) as response:
dest.write_bytes(response.read())
return
except OSError:
response = urllib.request.urlopen(url, context=context)
except Exception:
if i == repeat_num - 1:
raise
sleep(3)
continue
break
def extract_zip(zip_src: Path, dest: Path) -> None:
with ZipFile(zip_src) as zip_:
for zinfo in zip_.filelist:
zip_.extract(zinfo, dest)
# Set permissions to the same values as they were set in the archive
# We have to do this manually due to
# https://github.com/python/cpython/issues/59999
# But some files in the zipfile seem to have external_attr with 0
# permissions. In that case just use the default value???
permissions = (zinfo.external_attr >> 16) & 0o777
if permissions != 0:
dest.joinpath(zinfo.filename).chmod(permissions)
def extract_tar(tar_src: Path, dest: Path) -> None:
with tarfile.open(tar_src) as tar_:
tar_.extraction_filter = getattr(tarfile, "tar_filter", (lambda member, _: member))
tar_.extractall(dest)
def move_file(src_file: Path, dst_file: Path) -> Path:
"""Moves a file safely while avoiding potential semantic confusion:
1. `dst_file` must point to the target filename, not a directory
2. `dst_file` will be overwritten if it already exists
3. any missing parent directories will be created
Returns the fully resolved Path of the resulting file.
Raises:
NotADirectoryError: If any part of the intermediate path to `dst_file` is an existing file
IsADirectoryError: If `dst_file` points directly to an existing directory
"""
src_file = src_file.resolve(strict=True)
dst_file = dst_file.resolve()
if dst_file.is_dir():
msg = "dst_file must be a valid target filename, not an existing directory."
raise IsADirectoryError(msg)
dst_file.unlink(missing_ok=True)
dst_file.parent.mkdir(parents=True, exist_ok=True)
# using shutil.move() as Path.rename() is not guaranteed to work across filesystem boundaries
# explicit str() needed for Python 3.8
resulting_file = shutil.move(str(src_file), str(dst_file))
return Path(resulting_file).resolve(strict=True)
try:
dest.write_bytes(response.read())
finally:
response.close()
class DependencyConstraints:
@@ -400,67 +156,60 @@ class DependencyConstraints:
self.base_file_path = base_file_path.resolve()
@staticmethod
def with_defaults() -> DependencyConstraints:
def with_defaults() -> "DependencyConstraints":
return DependencyConstraints(base_file_path=resources_dir / "constraints.txt")
def get_for_python_version(
self, version: str, *, variant: Literal["python", "pyodide"] = "python"
) -> Path:
def get_for_python_version(self, version: str) -> Path:
version_parts = version.split(".")
# try to find a version-specific dependency file e.g. if
# ./constraints.txt is the base, look for ./constraints-python36.txt
specific_stem = self.base_file_path.stem + f"-{variant}{version_parts[0]}{version_parts[1]}"
# ./constraints.txt is the base, look for ./constraints-python27.txt
specific_stem = self.base_file_path.stem + f"-python{version_parts[0]}{version_parts[1]}"
specific_name = specific_stem + self.base_file_path.suffix
specific_file_path = self.base_file_path.with_name(specific_name)
if specific_file_path.exists():
return specific_file_path
else:
return self.base_file_path
def __repr__(self) -> str:
return f"{self.__class__.__name__}({self.base_file_path!r})"
def __eq__(self, o: object) -> bool:
if not isinstance(o, DependencyConstraints):
return False
return self.base_file_path == o.base_file_path
def options_summary(self) -> Any:
if self == DependencyConstraints.with_defaults():
return "pinned"
else:
return self.base_file_path.name
return f"{self.__class__.__name__}{self.base_file_path!r})"
BuildFrontendName = Literal["pip", "build", "build[uv]"]
class BuildOptions(NamedTuple):
package_dir: Path
output_dir: Path
build_selector: BuildSelector
architectures: Set[Architecture]
environment: ParsedEnvironment
before_all: str
before_build: Optional[str]
repair_command: str
manylinux_images: Optional[Dict[str, str]]
dependency_constraints: Optional[DependencyConstraints]
test_command: Optional[str]
test_selector: TestSelector
before_test: Optional[str]
test_requires: List[str]
test_extras: str
build_verbosity: int
@dataclass(frozen=True)
class BuildFrontendConfig:
name: BuildFrontendName
args: Sequence[str] = ()
class NonPlatformWheelError(Exception):
def __init__(self) -> None:
message = textwrap.dedent(
"""
cibuildwheel: Build failed because a pure Python wheel was generated.
@staticmethod
def from_config_string(config_string: str) -> BuildFrontendConfig:
config_dict = parse_key_value_string(config_string, ["name"], ["args"])
name = " ".join(config_dict["name"])
if name not in {"pip", "build", "build[uv]"}:
msg = f"Unrecognised build frontend {name!r}, only 'pip', 'build', and 'build[uv]' are supported"
raise ValueError(msg)
If you intend to build a pure-Python wheel, you don't need cibuildwheel - use
`pip wheel -w DEST_DIR .` instead.
name = typing.cast(BuildFrontendName, name)
If you expected a platform wheel, check your project configuration, or run
cibuildwheel with CIBW_BUILD_VERBOSITY=1 to view build logs.
"""
)
args = config_dict.get("args") or []
return BuildFrontendConfig(name=name, args=args)
def options_summary(self) -> str | dict[str, str]:
if not self.args:
return self.name
else:
return {"name": self.name, "args": repr(self.args)}
super().__init__(message)
def strtobool(val: str) -> bool:
@@ -474,11 +223,10 @@ class CIProvider(Enum):
azure_pipelines = "azure_pipelines"
github_actions = "github_actions"
gitlab = "gitlab"
cirrus_ci = "cirrus_ci"
other = "other"
def detect_ci_provider() -> CIProvider | None:
def detect_ci_provider() -> Optional[CIProvider]:
if "TRAVIS" in os.environ:
return CIProvider.travis_ci
elif "APPVEYOR" in os.environ:
@@ -491,8 +239,6 @@ def detect_ci_provider() -> CIProvider | None:
return CIProvider.github_actions
elif "GITLAB_CI" in os.environ:
return CIProvider.gitlab
elif "CIRRUS_CI" in os.environ:
return CIProvider.cirrus_ci
elif strtobool(os.environ.get("CI", "false")):
return CIProvider.other
else:
@@ -511,14 +257,8 @@ def unwrap(text: str) -> str:
return re.sub(r"\s+", " ", text)
@dataclass(frozen=True)
class FileReport:
name: str
size: str
@contextlib.contextmanager
def print_new_wheels(msg: str, output_dir: Path) -> Generator[None, None, None]:
def print_new_wheels(msg: str, output_dir: Path) -> Iterator[None]:
"""
Prints the new items in a directory upon exiting. The message to display
can include {n} for number of wheels, {s} for total number of seconds,
@@ -530,421 +270,8 @@ def print_new_wheels(msg: str, output_dir: Path) -> Generator[None, None, None]:
existing_contents = set(output_dir.iterdir())
yield
final_contents = set(output_dir.iterdir())
new_contents = [
FileReport(wheel.name, f"{(wheel.stat().st_size + 1023) // 1024:,d}")
for wheel in final_contents - existing_contents
]
if not new_contents:
return
max_name_len = max(len(f.name) for f in new_contents)
max_size_len = max(len(f.size) for f in new_contents)
new_contents = final_contents - existing_contents
n = len(new_contents)
s = time.time() - start_time
m = s / 60
print(
msg.format(n=n, s=s, m=m),
*sorted(
f" {f.name:<{max_name_len}s} {f.size:>{max_size_len}s} kB" for f in new_contents
),
sep="\n",
)
def get_pip_version(env: Mapping[str, str]) -> str:
versions_output_text = call(
"python", "-m", "pip", "freeze", "--all", capture_stdout=True, env=env
)
(pip_version,) = (
version[5:]
for version in versions_output_text.strip().splitlines()
if version.startswith("pip==")
)
return pip_version
@lru_cache(maxsize=None)
def ensure_node(major_version: str) -> Path:
input_file = resources_dir / "nodejs.toml"
with input_file.open("rb") as f:
loaded_file = tomllib.load(f)
version = str(loaded_file[major_version])
base_url = str(loaded_file["url"])
ext = "zip" if IS_WIN else "tar.xz"
platform = "win" if IS_WIN else ("darwin" if sys.platform.startswith("darwin") else "linux")
linux_arch = Architecture.native_arch("linux")
assert linux_arch is not None
arch = {"x86_64": "x64", "i686": "x86", "aarch64": "arm64"}.get(
linux_arch.value, linux_arch.value
)
name = f"node-{version}-{platform}-{arch}"
path = CIBW_CACHE_PATH / name
with FileLock(str(path) + ".lock"):
if not path.exists():
url = f"{base_url}{version}/{name}.{ext}"
with TemporaryDirectory() as tmp_path:
archive = Path(tmp_path) / f"{name}.{ext}"
download(url, archive)
if ext == "zip":
extract_zip(archive, path.parent)
else:
extract_tar(archive, path.parent)
assert path.exists()
if not IS_WIN:
return path / "bin"
return path
@lru_cache(maxsize=None)
def _ensure_virtualenv(version: str) -> Path:
version_parts = version.split(".")
key = f"py{version_parts[0]}{version_parts[1]}"
input_file = resources_dir / "virtualenv.toml"
with input_file.open("rb") as f:
loaded_file = tomllib.load(f)
configuration = loaded_file.get(key, loaded_file["default"])
version = str(configuration["version"])
url = str(configuration["url"])
path = CIBW_CACHE_PATH / f"virtualenv-{version}.pyz"
with FileLock(str(path) + ".lock"):
if not path.exists():
download(url, path)
return path
def _parse_constraints_for_virtualenv(
seed_packages: list[str],
dependency_constraint_flags: Sequence[PathOrStr],
) -> dict[str, str]:
"""
Parses the constraints file referenced by `dependency_constraint_flags` and returns a dict where
the key is the package name, and the value is the constraint version.
If a package version cannot be found, its value is "embed" meaning that virtualenv will install
its bundled version, already available locally.
The function does not try to be too smart and just handles basic constraints.
If it can't get an exact version, the real constraint will be handled by the
{macos|windows}.setup_python function.
"""
assert len(dependency_constraint_flags) in {0, 2}
# only seed pip if other seed packages do not appear in a constraint file
constraints_dict = {"pip": "embed"}
if len(dependency_constraint_flags) == 2:
assert dependency_constraint_flags[0] == "-c"
constraint_path = Path(dependency_constraint_flags[1])
assert constraint_path.exists()
with constraint_path.open(encoding="utf-8") as constraint_file:
for line_ in constraint_file:
line = line_.strip()
if not line:
continue
if line.startswith("#"):
continue
try:
requirement = Requirement(line)
package = requirement.name
if (
package not in seed_packages
or requirement.url is not None
or requirement.marker is not None
or len(requirement.extras) != 0
or len(requirement.specifier) != 1
):
continue
specifier = next(iter(requirement.specifier))
if specifier.operator != "==":
continue
constraints_dict[package] = specifier.version
except InvalidRequirement:
continue
return constraints_dict
def virtualenv(
version: str,
python: Path,
venv_path: Path,
dependency_constraint_flags: Sequence[PathOrStr],
*,
use_uv: bool,
) -> dict[str, str]:
"""
Create a virtual environment. If `use_uv` is True,
dependency_constraint_flags are ignored since nothing is installed in the
venv. Otherwise, pip is installed, and setuptools + wheel if Python < 3.12.
"""
assert python.exists()
if use_uv:
call("uv", "venv", venv_path, "--python", python)
else:
virtualenv_app = _ensure_virtualenv(version)
allowed_seed_packages = ["pip", "setuptools", "wheel"]
constraints = _parse_constraints_for_virtualenv(
allowed_seed_packages, dependency_constraint_flags
)
additional_flags: list[str] = []
for package in allowed_seed_packages:
if package in constraints:
additional_flags.append(f"--{package}={constraints[package]}")
else:
additional_flags.append(f"--no-{package}")
# Using symlinks to pre-installed seed packages is really the fastest way to get a virtual
# environment. The initial cost is a bit higher but reusing is much faster.
# Windows does not always allow symlinks so just disabling for now.
# Requires pip>=19.3 so disabling for "embed" because this means we don't know what's the
# version of pip that will end-up installed.
# c.f. https://virtualenv.pypa.io/en/latest/cli_interface.html#section-seeder
if (
not IS_WIN
and constraints["pip"] != "embed"
and Version(constraints["pip"]) >= Version("19.3")
):
additional_flags.append("--symlink-app-data")
call(
sys.executable,
"-sS", # just the stdlib, https://github.com/pypa/virtualenv/issues/2133#issuecomment-1003710125
virtualenv_app,
"--activators=",
"--no-periodic-update",
*additional_flags,
"--python",
python,
venv_path,
)
paths = [str(venv_path), str(venv_path / "Scripts")] if IS_WIN else [str(venv_path / "bin")]
env = os.environ.copy()
env["PATH"] = os.pathsep.join([*paths, env["PATH"]])
env["VIRTUAL_ENV"] = str(venv_path)
return env
T = TypeVar("T", bound=PurePath)
def find_compatible_wheel(wheels: Sequence[T], identifier: str) -> T | None:
"""
Finds a wheel with an abi3 or a none ABI tag in `wheels` compatible with the Python interpreter
specified by `identifier` that is previously built.
"""
interpreter, platform = identifier.split("-")
free_threaded = interpreter.endswith("t")
if free_threaded:
interpreter = interpreter[:-1]
for wheel in wheels:
_, _, _, tags = parse_wheel_filename(wheel.name)
for tag in tags:
if tag.abi == "abi3" and not free_threaded:
# ABI3 wheels must start with cp3 for impl and tag
if not (interpreter.startswith("cp3") and tag.interpreter.startswith("cp3")):
continue
elif tag.abi == "none":
# CPythonless wheels must include py3 tag
if tag.interpreter[:3] != "py3":
continue
else:
# Other types of wheels are not detected, this is looking for previously built wheels.
continue
if tag.interpreter != "py3" and int(tag.interpreter[3:]) > int(interpreter[3:]):
# If a minor version number is given, it has to be lower than the current one.
continue
if platform.startswith(("manylinux", "musllinux", "macosx")):
# Linux, macOS require the beginning and ending match (macos/manylinux version doesn't need to)
os_, arch = platform.split("_", 1)
if not tag.platform.startswith(os_):
continue
if not tag.platform.endswith(f"_{arch}"):
continue
else:
# Windows should exactly match
if tag.platform != platform:
continue
# If all the filters above pass, then the wheel is a previously built compatible wheel.
return wheel
return None
# Can be replaced by contextlib.chdir in Python 3.11
@contextlib.contextmanager
def chdir(new_path: Path | str) -> Generator[None, None, None]:
"""Non thread-safe context manager to change the current working directory."""
cwd = os.getcwd()
try:
os.chdir(new_path)
yield
finally:
os.chdir(cwd)
def fix_ansi_codes_for_github_actions(text: str) -> str:
"""
Github Actions forgets the current ANSI style on every new line. This
function repeats the current ANSI style on every new line.
"""
ansi_code_regex = re.compile(r"(\033\[[0-9;]*m)")
ansi_codes: list[str] = []
output = ""
for line in text.splitlines(keepends=True):
# add the current ANSI codes to the beginning of the line
output += "".join(ansi_codes) + line
# split the line at each ANSI code
parts = ansi_code_regex.split(line)
# if there are any ANSI codes, save them
if len(parts) > 1:
# iterate over the ANSI codes in this line
for code in parts[1::2]:
if code == "\033[0m":
# reset the list of ANSI codes when the clear code is found
ansi_codes = []
else:
ansi_codes.append(code)
return output
def parse_key_value_string(
key_value_string: str,
positional_arg_names: Sequence[str] | None = None,
kw_arg_names: Sequence[str] | None = None,
) -> dict[str, list[str]]:
"""
Parses a string like "docker; create_args: --some-option=value another-option"
"""
if positional_arg_names is None:
positional_arg_names = []
if kw_arg_names is None:
kw_arg_names = []
all_field_names = [*positional_arg_names, *kw_arg_names]
shlexer = shlex.shlex(key_value_string, posix=True, punctuation_chars=";")
shlexer.commenters = ""
shlexer.whitespace_split = True
parts = list(shlexer)
# parts now looks like
# ['docker', ';', 'create_args:', '--some-option=value', 'another-option']
# split by semicolon
fields = [list(group) for k, group in itertools.groupby(parts, lambda x: x == ";") if not k]
result: defaultdict[str, list[str]] = defaultdict(list)
for field_i, field in enumerate(fields):
# check to see if the option name is specified
field_name, sep, first_value = field[0].partition(":")
if sep:
if field_name not in all_field_names:
msg = f"Failed to parse {key_value_string!r}. Unknown field name {field_name!r}"
raise ValueError(msg)
values = ([first_value] if first_value else []) + field[1:]
else:
try:
field_name = positional_arg_names[field_i]
except IndexError:
msg = f"Failed to parse {key_value_string!r}. Too many positional arguments - expected a maximum of {len(positional_arg_names)}"
raise ValueError(msg) from None
values = field
result[field_name] += values
return dict(result)
def find_uv() -> Path | None:
# Prefer uv in our environment
with contextlib.suppress(ImportError, FileNotFoundError):
# pylint: disable-next=import-outside-toplevel
from uv import find_uv_bin
return Path(find_uv_bin())
uv_on_path = shutil.which("uv")
return Path(uv_on_path) if uv_on_path else None
def combine_constraints(
env: MutableMapping[str, str], /, constraints_path: Path, tmp_dir: Path | None
) -> None:
"""
This will workaround a bug in pip<=21.1.1 or uv<=0.2.0 if a tmp_dir is given.
If set to None, this will use the modern URI method.
"""
if tmp_dir:
if " " in str(constraints_path):
assert " " not in str(tmp_dir)
tmp_file = tmp_dir / "constraints.txt"
tmp_file.write_bytes(constraints_path.read_bytes())
constraints_path = tmp_file
our_constraints = str(constraints_path)
else:
our_constraints = (
constraints_path.as_uri() if " " in str(constraints_path) else str(constraints_path)
)
user_constraints = env.get("PIP_CONSTRAINT")
env["UV_CONSTRAINT"] = env["PIP_CONSTRAINT"] = " ".join(
c for c in [our_constraints, user_constraints] if c
)
@total_ordering
class FlexibleVersion:
version_str: str
version_parts: tuple[int, ...]
suffix: str
def __init__(self, version_str: str) -> None:
self.version_str = version_str
# Split into numeric parts and the optional suffix
match = re.match(r"^[v]?(\d+(\.\d+)*)(.*)$", version_str)
if not match:
msg = f"Invalid version string: {version_str}"
raise ValueError(msg)
version_part, _, suffix = match.groups()
# Convert numeric version part into a tuple of integers
self.version_parts = tuple(map(int, version_part.split(".")))
self.suffix = suffix.strip() if suffix else ""
# Normalize by removing trailing zeros
self.version_parts = self._remove_trailing_zeros(self.version_parts)
def _remove_trailing_zeros(self, parts: tuple[int, ...]) -> tuple[int, ...]:
# Remove trailing zeros for accurate comparisons
# without this, "3.0" would be considered greater than "3"
while parts and parts[-1] == 0:
parts = parts[:-1]
return parts
def __eq__(self, other: object) -> bool:
if not isinstance(other, FlexibleVersion):
raise NotImplementedError()
return (self.version_parts, self.suffix) == (other.version_parts, other.suffix)
def __lt__(self, other: object) -> bool:
if not isinstance(other, FlexibleVersion):
raise NotImplementedError()
return (self.version_parts, self.suffix) < (other.version_parts, other.suffix)
def __repr__(self) -> str:
return f"FlexibleVersion('{self.version_str}')"
def __str__(self) -> str:
return self.version_str
print(msg.format(n=n, s=s, m=m), *sorted(f" {f.name}" for f in new_contents), sep="\n")
+246 -448
View File
@@ -1,90 +1,88 @@
from __future__ import annotations
import os
import platform as platform_module
import shutil
import subprocess
import textwrap
from collections.abc import MutableMapping, Sequence, Set
from dataclasses import dataclass
from functools import lru_cache
import sys
import tempfile
from pathlib import Path
from typing import Dict, List, NamedTuple, Optional, Sequence, Set
from zipfile import ZipFile
from filelock import FileLock
from packaging.version import Version
import toml
from . import errors
from ._compat.typing import assert_never
from .architecture import Architecture
from .environment import ParsedEnvironment
from .logger import log
from .options import Options
from .typing import PathOrStr
from .util import (
CIBW_CACHE_PATH,
BuildFrontendConfig,
BuildFrontendName,
BuildOptions,
BuildSelector,
call,
combine_constraints,
NonPlatformWheelError,
download,
extract_zip,
find_compatible_wheel,
find_uv,
get_build_verbosity_extra_flags,
get_pip_version,
move_file,
get_pip_script,
prepare_command,
read_python_configs,
shell,
split_config_settings,
test_fail_cwd_file,
unwrap,
virtualenv,
)
IS_RUNNING_ON_AZURE = Path("C:\\hostedtoolcache").exists()
def get_nuget_args(
version: str, arch: str, free_threaded: bool, output_directory: Path
) -> list[str]:
package_name = {
"32": "pythonx86",
"64": "python",
"ARM64": "pythonarm64",
# Aliases for platform.machine() return values
"x86": "pythonx86",
"AMD64": "python",
}[arch]
if free_threaded:
package_name = f"{package_name}-freethreaded"
def call(
args: Sequence[PathOrStr], env: Optional[Dict[str, str]] = None, cwd: Optional[str] = None
) -> None:
print("+ " + " ".join(str(a) for a in args))
# we use shell=True here, even though we don't need a shell due to a bug
# https://bugs.python.org/issue8557
subprocess.run([str(a) for a in args], env=env, cwd=cwd, shell=True, check=True)
def shell(command: str, env: Optional[Dict[str, str]] = None, cwd: Optional[str] = None) -> None:
print(f"+ {command}")
subprocess.run(command, env=env, cwd=cwd, shell=True, check=True)
def get_nuget_args(version: str, arch: str) -> List[str]:
python_name = "python" if version[0] == "3" else "python2"
if arch == "32":
python_name += "x86"
return [
package_name,
python_name,
"-Version",
version,
"-FallbackSource",
"https://api.nuget.org/v3/index.json",
"-OutputDirectory",
str(output_directory),
"C:\\cibw\\python",
]
@dataclass(frozen=True)
class PythonConfiguration:
class PythonConfiguration(NamedTuple):
version: str
arch: str
identifier: str
url: str | None = None
url: Optional[str] = None
def get_python_configurations(
build_selector: BuildSelector,
architectures: Set[Architecture],
) -> list[PythonConfiguration]:
) -> List[PythonConfiguration]:
full_python_configs = read_python_configs("windows")
python_configurations = [PythonConfiguration(**item) for item in full_python_configs]
map_arch = {"32": Architecture.x86, "64": Architecture.AMD64, "ARM64": Architecture.ARM64}
map_arch = {
"32": Architecture.x86,
"64": Architecture.AMD64,
}
# Only supported with custom compiler, since MS removed the 2008 compiler download
custom_compiler = os.environ.get("DISTUTILS_USE_SDK") and os.environ.get("MSSdk")
if not custom_compiler:
python_configurations = [
c for c in python_configurations if not c.version.startswith("2.7")
]
# skip builds as required
python_configurations = [
@@ -96,435 +94,253 @@ def get_python_configurations(
return python_configurations
@lru_cache(maxsize=None)
def _ensure_nuget() -> Path:
nuget = CIBW_CACHE_PATH / "nuget.exe"
with FileLock(str(nuget) + ".lock"):
if not nuget.exists():
download("https://dist.nuget.org/win-x86-commandline/latest/nuget.exe", nuget)
return nuget
def extract_zip(zip_src: Path, dest: Path) -> None:
with ZipFile(zip_src) as zip:
zip.extractall(dest)
def install_cpython(configuration: PythonConfiguration, arch: str | None = None) -> Path:
version = configuration.version
free_threaded = "t-" in configuration.identifier
if arch is None:
arch = configuration.arch
base_output_dir = CIBW_CACHE_PATH / "nuget-cpython"
nuget_args = get_nuget_args(version, arch, free_threaded, base_output_dir)
installation_path = base_output_dir / (nuget_args[0] + "." + version) / "tools"
free_threaded_str = "-freethreaded" if free_threaded else ""
with FileLock(str(base_output_dir) + f"-{version}{free_threaded_str}-{arch}.lock"):
if not installation_path.exists():
nuget = _ensure_nuget()
call(nuget, "install", *nuget_args)
return installation_path / "python.exe"
def install_cpython(version: str, arch: str, nuget: Path) -> Path:
nuget_args = get_nuget_args(version, arch)
installation_path = Path(nuget_args[-1]) / (nuget_args[0] + "." + version) / "tools"
call([nuget, "install", *nuget_args])
return installation_path
def install_pypy(tmp: Path, arch: str, url: str) -> Path:
assert arch == "64"
assert "win64" in url
def install_pypy(version: str, arch: str, url: str) -> Path:
assert arch == "32"
# Inside the PyPy zip file is a directory with the same name
zip_filename = url.rsplit("/", 1)[-1]
extension = ".zip"
assert zip_filename.endswith(extension)
installation_path = CIBW_CACHE_PATH / zip_filename[: -len(extension)]
with FileLock(str(installation_path) + ".lock"):
if not installation_path.exists():
pypy_zip = tmp / zip_filename
download(url, pypy_zip)
# Extract to the parent directory because the zip file still contains a directory
extract_zip(pypy_zip, installation_path.parent)
return installation_path / "python.exe"
def setup_setuptools_cross_compile(
tmp: Path,
python_configuration: PythonConfiguration,
python_libs_base: Path,
env: MutableMapping[str, str],
) -> None:
distutils_cfg = tmp / "extra-setup.cfg"
env["DIST_EXTRA_CONFIG"] = str(distutils_cfg)
log.notice(f"Setting DIST_EXTRA_CONFIG={distutils_cfg} for cross-compilation")
# Ensure our additional import libraries are made available, and explicitly
# set the platform name
map_plat = {"32": "win32", "64": "win-amd64", "ARM64": "win-arm64"}
plat_name = map_plat[python_configuration.arch]
# Set environment variable so that setuptools._distutils.get_platform()
# identifies the target, not the host
vscmd_arg_tgt_arch = {"32": "x86", "64": "x64", "ARM64": "arm64"}
current_tgt_arch = vscmd_arg_tgt_arch[python_configuration.arch]
if (env.get("VSCMD_ARG_TGT_ARCH") or current_tgt_arch) != current_tgt_arch:
msg = f"VSCMD_ARG_TGT_ARCH must be set to {current_tgt_arch!r}, got {env['VSCMD_ARG_TGT_ARCH']!r}. If you're setting up MSVC yourself (e.g. using vcvarsall.bat or msvc-dev-cmd), make sure to target the right architecture. Alternatively, run cibuildwheel without configuring MSVC, and let the build backend handle it."
raise errors.FatalError(msg)
env["VSCMD_ARG_TGT_ARCH"] = current_tgt_arch
# (This file must be default/locale encoding, so we can't pass 'encoding')
distutils_cfg.write_text(
textwrap.dedent(
f"""\
[build]
plat_name={plat_name}
[build_ext]
library_dirs={python_libs_base}
plat_name={plat_name}
[bdist_wheel]
plat_name={plat_name}
"""
)
)
# setuptools builds require explicit override of PYD extension
# This is because it always gets the extension from the running
# interpreter, and has no logic to construct it. Currently, CPython's
# extensions follow our identifiers, but if they ever diverge in the
# future, we will need to store new data
log.notice(
f"Setting SETUPTOOLS_EXT_SUFFIX=.{python_configuration.identifier}.pyd for cross-compilation"
)
env["SETUPTOOLS_EXT_SUFFIX"] = f".{python_configuration.identifier}.pyd"
# Cross-compilation requires fixes that only exist in setuptools's copy of
# distutils, so ensure that it is activated
# Since not all projects can handle the newer distutils, display a warning
# to help them figure out what may have gone wrong if this breaks for them
log.notice("Setting SETUPTOOLS_USE_DISTUTILS=local as it is required for cross-compilation")
env["SETUPTOOLS_USE_DISTUTILS"] = "local"
# These cross-compile setup functions have the same signature by design
def setup_rust_cross_compile(
tmp: Path, # noqa: ARG001
python_configuration: PythonConfiguration,
python_libs_base: Path, # noqa: ARG001
env: MutableMapping[str, str],
) -> None:
# Assume that MSVC will be used, because we already know that we are
# cross-compiling. MinGW users can set CARGO_BUILD_TARGET themselves
# and we will respect the existing value.
cargo_target = {
"64": "x86_64-pc-windows-msvc",
"32": "i686-pc-windows-msvc",
"ARM64": "aarch64-pc-windows-msvc",
}.get(python_configuration.arch)
# CARGO_BUILD_TARGET is the variable used by Cargo and setuptools_rust
if env.get("CARGO_BUILD_TARGET"):
if env["CARGO_BUILD_TARGET"] != cargo_target:
log.notice("Not overriding CARGO_BUILD_TARGET as it has already been set")
# No message if it was set to what we were planning to set it to
elif cargo_target:
log.notice(f"Setting CARGO_BUILD_TARGET={cargo_target} for cross-compilation")
env["CARGO_BUILD_TARGET"] = cargo_target
else:
log.warning(
f"Unable to configure Rust cross-compilation for architecture {python_configuration.arch}"
)
def can_use_uv(python_configuration: PythonConfiguration) -> bool:
conditions = (
Version(python_configuration.version) >= Version("3.8"),
not python_configuration.identifier.startswith("pp38-"),
)
return all(conditions)
installation_path = Path("C:\\cibw") / zip_filename[: -len(extension)]
if not installation_path.exists():
pypy_zip = Path("C:\\cibw") / zip_filename
download(url, pypy_zip)
# Extract to the parent directory because the zip file still contains a directory
extract_zip(pypy_zip, installation_path.parent)
pypy_exe = "pypy3.exe" if version[0] == "3" else "pypy.exe"
(installation_path / "python.exe").symlink_to(installation_path / pypy_exe)
return installation_path
def setup_python(
tmp: Path,
python_configuration: PythonConfiguration,
dependency_constraint_flags: Sequence[PathOrStr],
environment: ParsedEnvironment,
build_frontend: BuildFrontendName,
) -> tuple[Path, dict[str, str]]:
tmp.mkdir()
) -> Dict[str, str]:
nuget = Path("C:\\cibw\\nuget.exe")
if not nuget.exists():
log.step("Downloading nuget...")
download("https://dist.nuget.org/win-x86-commandline/latest/nuget.exe", nuget)
implementation_id = python_configuration.identifier.split("-")[0]
python_libs_base = None
log.step(f"Installing Python {implementation_id}...")
if implementation_id.startswith("cp"):
native_arch = platform_module.machine()
base_python = install_cpython(python_configuration)
if python_configuration.arch == "ARM64" != native_arch:
# To cross-compile for ARM64, we need a native CPython to run the
# build, and a copy of the ARM64 import libraries ('.\libs\*.lib')
# for any extension modules.
python_libs_base = base_python.parent / "libs"
log.step(f"Installing native Python {native_arch} for cross-compilation...")
base_python = install_cpython(python_configuration, arch=native_arch)
installation_path = install_cpython(
python_configuration.version, python_configuration.arch, nuget
)
elif implementation_id.startswith("pp"):
assert python_configuration.url is not None
base_python = install_pypy(tmp, python_configuration.arch, python_configuration.url)
installation_path = install_pypy(
python_configuration.version, python_configuration.arch, python_configuration.url
)
else:
msg = "Unknown Python implementation"
raise ValueError(msg)
assert base_python.exists()
raise ValueError("Unknown Python implementation")
if build_frontend == "build[uv]" and not can_use_uv(python_configuration):
build_frontend = "build"
use_uv = build_frontend == "build[uv]"
uv_path = find_uv()
assert (installation_path / "python.exe").exists()
log.step("Setting up build environment...")
venv_path = tmp / "venv"
env = virtualenv(
python_configuration.version,
base_python,
venv_path,
dependency_constraint_flags,
use_uv=use_uv,
)
# set up environment variables for run_with_env
# set up PATH and environment variables for run_with_env
env = os.environ.copy()
env["PYTHON_VERSION"] = python_configuration.version
env["PYTHON_ARCH"] = python_configuration.arch
env["PATH"] = os.pathsep.join(
[str(installation_path), str(installation_path / "Scripts"), env["PATH"]]
)
env["PIP_DISABLE_PIP_VERSION_CHECK"] = "1"
# upgrade pip to the version matching our constraints
# if necessary, reinstall it to ensure that it's available on PATH as 'pip.exe'
if not use_uv:
call(
"python",
"-m",
"pip",
"install",
"--upgrade",
"pip",
*dependency_constraint_flags,
env=env,
cwd=venv_path,
)
# update env with results from CIBW_ENVIRONMENT
env = environment.as_dictionary(prev_environment=env)
# check what Python version we're on
call("where", "python", env=env)
call("python", "--version", env=env)
call("python", "-c", "\"import struct; print(struct.calcsize('P') * 8)\"", env=env)
where_python = call("where", "python", env=env, capture_stdout=True).splitlines()[0].strip()
if where_python != str(venv_path / "Scripts" / "python.exe"):
msg = "python available on PATH doesn't match our installed instance. If you have modified PATH, ensure that you don't overwrite cibuildwheel's entry or insert python above it."
raise errors.FatalError(msg)
# for the logs - check we're running the right version of python
call(["where", "python"], env=env)
call(["python", "--version"], env=env)
call(["python", "-c", "\"import struct; print(struct.calcsize('P') * 8)\""], env=env)
where_python = (
subprocess.run(
["where", "python"],
env=env,
universal_newlines=True,
check=True,
stdout=subprocess.PIPE,
)
.stdout.splitlines()[0]
.strip()
)
if where_python != str(installation_path / "python.exe"):
print(
"cibuildwheel: python available on PATH doesn't match our installed instance. If you have modified PATH, ensure that you don't overwrite cibuildwheel's entry or insert python above it.",
file=sys.stderr,
)
sys.exit(1)
# check what pip version we're on
if not use_uv:
assert (venv_path / "Scripts" / "pip.exe").exists()
where_pip = call("where", "pip", env=env, capture_stdout=True).splitlines()[0].strip()
if where_pip.strip() != str(venv_path / "Scripts" / "pip.exe"):
msg = "pip available on PATH doesn't match our installed instance. If you have modified PATH, ensure that you don't overwrite cibuildwheel's entry or insert pip above it."
raise errors.FatalError(msg)
call("pip", "--version", env=env)
# make sure pip is installed
if not (installation_path / "Scripts" / "pip.exe").exists():
call(["python", get_pip_script, *dependency_constraint_flags], env=env, cwd="C:\\cibw")
assert (installation_path / "Scripts" / "pip.exe").exists()
where_pip = (
subprocess.run(
["where", "pip"], env=env, universal_newlines=True, check=True, stdout=subprocess.PIPE
)
.stdout.splitlines()[0]
.strip()
)
if where_pip.strip() != str(installation_path / "Scripts" / "pip.exe"):
print(
"cibuildwheel: pip available on PATH doesn't match our installed instance. If you have modified PATH, ensure that you don't overwrite cibuildwheel's entry or insert pip above it.",
file=sys.stderr,
)
sys.exit(1)
log.step("Installing build tools...")
if build_frontend == "build":
call(
"pip",
"install",
"--upgrade",
"build[virtualenv]",
*dependency_constraint_flags,
env=env,
)
elif build_frontend == "build[uv]":
assert uv_path is not None
call(
uv_path,
"pip",
"install",
"--upgrade",
"build[virtualenv]",
*dependency_constraint_flags,
env=env,
)
if python_libs_base:
# Set up the environment for various backends to enable cross-compilation
setup_setuptools_cross_compile(tmp, python_configuration, python_libs_base, env)
setup_rust_cross_compile(tmp, python_configuration, python_libs_base, env)
return base_python, env
def build(options: Options, tmp_path: Path) -> None:
python_configurations = get_python_configurations(
options.globals.build_selector, options.globals.architectures
call(
["python", "-m", "pip", "install", "--upgrade", "pip", *dependency_constraint_flags],
env=env,
)
call(["pip", "--version"], env=env)
call(
["pip", "install", "--upgrade", "setuptools", "wheel", *dependency_constraint_flags],
env=env,
)
if not python_configurations:
return
return env
def pep_518_cp35_workaround(package_dir: Path, env: Dict[str, str]) -> None:
"""
Python 3.5 PEP 518 hack (see https://github.com/pypa/pip/issues/8392#issuecomment-639563494)
Basically, nuget's Python is an embedded Python distribution, which is not supported by pip.
Before version 3.6, there was no way to disable the "embedded" behavior, including the ignoring
of environment variables, including the ones pip uses to setup PEP 518 builds.
The fix here is as suggested in that issue; we manually setup the PEP 518 requirements. Since we
are in a fresh environment (except for pinned cibuildweel dependencies), the build is already
mostly "isolated".
"""
pyproject_path = package_dir / "pyproject.toml"
if pyproject_path.exists():
data = toml.load(pyproject_path)
requirements = data["build-system"].get("requires", []) if "build-system" in data else []
if requirements:
log.step("Performing PEP518 workaround...")
with tempfile.TemporaryDirectory() as d:
reqfile = Path(d) / "requirements.txt"
with reqfile.open("w") as f:
for r in requirements:
print(r, file=f)
call(["pip", "install", "-r", reqfile], env=env)
def build(options: BuildOptions) -> None:
temp_dir = Path(tempfile.mkdtemp(prefix="cibuildwheel"))
built_wheel_dir = temp_dir / "built_wheel"
repaired_wheel_dir = temp_dir / "repaired_wheel"
try:
before_all_options_identifier = python_configurations[0].identifier
before_all_options = options.build_options(before_all_options_identifier)
if before_all_options.before_all:
if options.before_all:
log.step("Running before_all...")
env = before_all_options.environment.as_dictionary(prev_environment=os.environ)
env = options.environment.as_dictionary(prev_environment=os.environ)
before_all_prepared = prepare_command(
before_all_options.before_all, project=".", package=options.globals.package_dir
options.before_all, project=".", package=options.package_dir
)
shell(before_all_prepared, env=env)
built_wheels: list[Path] = []
python_configurations = get_python_configurations(
options.build_selector, options.architectures
)
for config in python_configurations:
build_options = options.build_options(config.identifier)
build_frontend = build_options.build_frontend or BuildFrontendConfig("pip")
use_uv = build_frontend.name == "build[uv]" and can_use_uv(config)
log.build_start(config.identifier)
identifier_tmp_dir = tmp_path / config.identifier
identifier_tmp_dir.mkdir()
built_wheel_dir = identifier_tmp_dir / "built_wheel"
repaired_wheel_dir = identifier_tmp_dir / "repaired_wheel"
dependency_constraint_flags: Sequence[PathOrStr] = []
if build_options.dependency_constraints:
if options.dependency_constraints:
dependency_constraint_flags = [
"-c",
build_options.dependency_constraints.get_for_python_version(config.version),
options.dependency_constraints.get_for_python_version(config.version),
]
# install Python
base_python, env = setup_python(
identifier_tmp_dir / "build",
config,
dependency_constraint_flags,
build_options.environment,
build_frontend.name,
env = setup_python(config, dependency_constraint_flags, options.environment)
# run the before_build command
if options.before_build:
log.step("Running before_build...")
before_build_prepared = prepare_command(
options.before_build, project=".", package=options.package_dir
)
shell(before_build_prepared, env=env)
# activate the PEP 518 patch if on Windows Python 3.5
# (will only have an effect if PEP 517 builds are used):
if config.version.startswith("3.5"):
pep_518_cp35_workaround(options.package_dir, env)
log.step("Building wheel...")
if built_wheel_dir.exists():
shutil.rmtree(built_wheel_dir)
built_wheel_dir.mkdir(parents=True)
# Path.resolve() is needed. Without it pip wheel may try to fetch package from pypi.org
# see https://github.com/pypa/cibuildwheel/pull/369
call(
[
"pip",
"wheel",
options.package_dir.resolve(),
"-w",
built_wheel_dir,
"--no-deps",
*get_build_verbosity_extra_flags(options.build_verbosity),
],
env=env,
)
if not use_uv:
pip_version = get_pip_version(env)
compatible_wheel = find_compatible_wheel(built_wheels, config.identifier)
if compatible_wheel:
log.step_end()
print(
f"\nFound previously built wheel {compatible_wheel.name}, that's compatible with {config.identifier}. Skipping build step..."
built_wheel = next(built_wheel_dir.glob("*.whl"))
# repair the wheel
if repaired_wheel_dir.exists():
shutil.rmtree(repaired_wheel_dir)
repaired_wheel_dir.mkdir(parents=True)
if built_wheel.name.endswith("none-any.whl"):
raise NonPlatformWheelError()
if options.repair_command:
log.step("Repairing wheel...")
repair_command_prepared = prepare_command(
options.repair_command, wheel=built_wheel, dest_dir=repaired_wheel_dir
)
repaired_wheel = compatible_wheel
shell(repair_command_prepared, env=env)
else:
# run the before_build command
if build_options.before_build:
log.step("Running before_build...")
before_build_prepared = prepare_command(
build_options.before_build,
project=".",
package=options.globals.package_dir,
)
shell(before_build_prepared, env=env)
shutil.move(str(built_wheel), repaired_wheel_dir)
log.step("Building wheel...")
built_wheel_dir.mkdir()
repaired_wheel = next(repaired_wheel_dir.glob("*.whl"))
extra_flags = split_config_settings(
build_options.config_settings, build_frontend.name
)
extra_flags += build_frontend.args
build_env = env.copy()
if not use_uv:
build_env["VIRTUALENV_PIP"] = pip_version
if build_options.dependency_constraints:
constraints_path = build_options.dependency_constraints.get_for_python_version(
config.version
)
combine_constraints(build_env, constraints_path, identifier_tmp_dir)
if build_frontend.name == "pip":
extra_flags += get_build_verbosity_extra_flags(build_options.build_verbosity)
# Path.resolve() is needed. Without it pip wheel may try to fetch package from pypi.org
# see https://github.com/pypa/cibuildwheel/pull/369
call(
"python",
"-m",
"pip",
"wheel",
options.globals.package_dir.resolve(),
f"--wheel-dir={built_wheel_dir}",
"--no-deps",
*extra_flags,
env=build_env,
)
elif build_frontend.name == "build" or build_frontend.name == "build[uv]":
if not 0 <= build_options.build_verbosity < 2:
msg = f"build_verbosity {build_options.build_verbosity} is not supported for build frontend. Ignoring."
log.warning(msg)
if use_uv and "--no-isolation" not in extra_flags and "-n" not in extra_flags:
extra_flags.append("--installer=uv")
call(
"python",
"-m",
"build",
build_options.package_dir,
"--wheel",
f"--outdir={built_wheel_dir}",
*extra_flags,
env=build_env,
)
else:
assert_never(build_frontend)
built_wheel = next(built_wheel_dir.glob("*.whl"))
# repair the wheel
repaired_wheel_dir.mkdir()
if built_wheel.name.endswith("none-any.whl"):
raise errors.NonPlatformWheelError()
if build_options.repair_command:
log.step("Repairing wheel...")
repair_command_prepared = prepare_command(
build_options.repair_command,
wheel=built_wheel,
dest_dir=repaired_wheel_dir,
)
shell(repair_command_prepared, env=env)
else:
shutil.move(str(built_wheel), repaired_wheel_dir)
repaired_wheel = next(repaired_wheel_dir.glob("*.whl"))
if repaired_wheel.name in {wheel.name for wheel in built_wheels}:
raise errors.AlreadyBuiltWheelError(repaired_wheel.name)
test_selected = options.globals.test_selector(config.identifier)
if test_selected and config.arch == "ARM64" != platform_module.machine():
log.warning(
unwrap(
"""
While arm64 wheels can be built on other platforms, they cannot
be tested. An arm64 runner is required. To silence this warning,
set `CIBW_TEST_SKIP: "*-win_arm64"`.
"""
)
)
# skip this test
elif test_selected and build_options.test_command:
if options.test_command and options.test_selector(config.identifier):
log.step("Testing wheel...")
# set up a virtual environment to install and test from, to make sure
# there are no dependencies that were pulled in at build time.
if not use_uv:
call("pip", "install", "virtualenv", *dependency_constraint_flags, env=env)
call(["pip", "install", "virtualenv", *dependency_constraint_flags], env=env)
venv_dir = Path(tempfile.mkdtemp())
venv_dir = identifier_tmp_dir / "venv-test"
if use_uv:
call("uv", "venv", venv_dir, f"--python={base_python}", env=env)
else:
# Use pip version from the initial env to ensure determinism
venv_args = ["--no-periodic-update", f"--pip={pip_version}"]
# In Python<3.12, setuptools & wheel are installed as well, use virtualenv embedded ones
if Version(config.version) < Version("3.12"):
venv_args.extend(("--setuptools=embed", "--wheel=embed"))
call("python", "-m", "virtualenv", *venv_args, venv_dir, env=env)
# Use --no-download to ensure determinism by using seed libraries
# built into virtualenv
call(["python", "-m", "virtualenv", "--no-download", venv_dir], env=env)
virtualenv_env = env.copy()
virtualenv_env["PATH"] = os.pathsep.join(
@@ -533,64 +349,46 @@ def build(options: Options, tmp_path: Path) -> None:
virtualenv_env["PATH"],
]
)
virtualenv_env["VIRTUAL_ENV"] = str(venv_dir)
# check that we are using the Python from the virtual environment
call("where", "python", env=virtualenv_env)
call(["where", "python"], env=virtualenv_env)
if build_options.before_test:
if options.before_test:
before_test_prepared = prepare_command(
build_options.before_test,
options.before_test,
project=".",
package=build_options.package_dir,
package=options.package_dir,
)
shell(before_test_prepared, env=virtualenv_env)
pip = ["uv", "pip"] if use_uv else ["pip"]
# install the wheel
call(
*pip,
"install",
str(repaired_wheel) + build_options.test_extras,
["pip", "install", str(repaired_wheel) + options.test_extras],
env=virtualenv_env,
)
# test the wheel
if build_options.test_requires:
call(*pip, "install", *build_options.test_requires, env=virtualenv_env)
if options.test_requires:
call(["pip", "install"] + options.test_requires, env=virtualenv_env)
# run the tests from a temp dir, with an absolute path in the command
# run the tests from c:\, with an absolute path in the command
# (this ensures that Python runs the tests against the installed wheel
# and not the repo code)
test_command_prepared = prepare_command(
build_options.test_command,
options.test_command,
project=Path(".").resolve(),
package=options.globals.package_dir.resolve(),
wheel=repaired_wheel,
package=options.package_dir.resolve(),
)
test_cwd = identifier_tmp_dir / "test_cwd"
test_cwd.mkdir()
(test_cwd / "test_fail.py").write_text(test_fail_cwd_file.read_text())
shell(test_command_prepared, cwd="c:\\", env=virtualenv_env)
shell(test_command_prepared, cwd=test_cwd, env=virtualenv_env)
# clean up
shutil.rmtree(venv_dir)
# we're all done here; move it to output (remove if already exists)
if compatible_wheel is None:
output_wheel = build_options.output_dir.joinpath(repaired_wheel.name)
moved_wheel = move_file(repaired_wheel, output_wheel)
if moved_wheel != output_wheel.resolve():
log.warning(
"{repaired_wheel} was moved to {moved_wheel} instead of {output_wheel}"
)
built_wheels.append(output_wheel)
# clean up
# (we ignore errors because occasionally Windows fails to unlink a file and we
# don't want to abort a build because of that)
shutil.rmtree(identifier_tmp_dir, ignore_errors=True)
shutil.move(str(repaired_wheel), options.output_dir)
log.build_end()
except subprocess.CalledProcessError as error:
msg = f"Command {error.cmd} failed with code {error.returncode}. {error.stdout or ''}"
raise errors.FatalError(msg) from error
log.step_end_with_error(
f"Command {error.cmd} failed with code {error.returncode}. {error.stdout}"
)
sys.exit(1)
+5 -1063
View File
File diff suppressed because it is too large Load Diff
+24 -91
View File
@@ -2,8 +2,6 @@
title: Contributing
---
# Contributing
Wheel-building can be pretty complex. We expect users to find edge-cases - please help the rest of the community out by documenting these, adding features to support them, and reporting bugs.
If you have an idea for a modification or feature, it's probably best to raise an issue first and discuss it with the maintainer team. Once we have rough consensus on a design, begin work in a PR.
@@ -12,7 +10,8 @@ If you have an idea for a modification or feature, it's probably best to raise a
Everyone contributing to the cibuildwheel project is expected to follow the [PSF Code of Conduct](https://github.com/pypa/.github/blob/main/CODE_OF_CONDUCT.md).
## Design Goals
Design Goals
------------
- `cibuildwheel` should wrap the complexity of wheel building.
- The user interface to `cibuildwheel` is the build script (e.g. `.travis.yml`). Feature additions should not increase the complexity of this script.
@@ -21,7 +20,7 @@ Everyone contributing to the cibuildwheel project is expected to follow the [PSF
Other notes:
- The platforms are very similar, until they're not. I'd rather have straight-forward code than totally DRY code, so let's keep airy platform abstractions to a minimum.
- The platforms are very similar, until they're not. I'd rather have straight-forward code than totally DRY code, so let's keep airy platfrom abstractions to a minimum.
- I might want to break the options into a shared config file one day, so that config is more easily shared. That has motivated some of the design decisions.
### cibuildwheel's relationship with build errors
@@ -32,94 +31,20 @@ We're not responsible for errors in those tools, for fixing errors/crashes there
So, if we can, I'd like to improve the experience on errors as well. In [this](https://github.com/pypa/cibuildwheel/issues/139) case, it takes a bit of knowledge to understand that the Linux builds are happening in a different OS via Docker, that the linked symbols won't match, that auditwheel will fail because of this. A problem with how the tools fit together, instead of the tools themselves.
## Development
Maintainer notes
----------------
### Running the tests
### Local testing
When making a change to the codebase, you can run tests locally for quicker feedback than the CI runs on a PR. You can [run them directly](#making-a-venv), but the easiest way to run tests is using [nox](https://nox.thea.codes/).
You should run:
You can run all the tests locally by doing:
```bash
nox -s tests
```
However, because this takes a while, you might prefer to be more specific.
#### Unit tests
To run the project's unit tests, do:
```bash
nox -s tests -- unit_test
```
There are a few custom options to enable different parts of the test suite - check `nox -s tests -- unit_test --help` for details.
If you're calling this a lot, you might consider using the `-r` or `-R` arguments to nox to make it a bit faster. This calls pytest under the hood, so to target a specific test, use pytest's `-k` option after the `--` above to select a specific test.
#### Integration tests
To run the project's integration tests, do:
```bash
nox -s tests -- test
```
The integration test suite is big - it can take more than 30 minutes to run the whole thing.
Because it takes such a long time, normally you'd choose specific tests to run locally, and rely on the project's CI for the rest. Use pytest's `-k` option to choose specific tests. You can pass a test name or a filename, it'll run everything that matches.
```bash
nox -s tests -- test -k <test_name_or_filename>
# e.g.
nox -s tests -- test -k before_build
```
A few notes-
- Because they run inside a container, Linux tests can run on all platforms where Docker is installed, so they're convenient for running integration tests locally. Set CIBW_PLATFORM to do this: `CIBW_PLATFORM=linux nox -s tests -- test`.
- Running the macOS integration tests requires _system installs_ of Python from python.org for all the versions that are tested. We won't attempt to install these when running locally, but you can do so manually using the URL in the error message that is printed when the install is not found.
#### Making a venv
More advanced users might prefer to invoke pytest directly-
```bash
python3 -m venv .venv
source .venv/bin/activate
```python
python3 -m venv venv
. venv/bin/activate
pip install -e .[dev]
# run the unit tests
pytest unit_test
# run the whole integration test suite
pytest test
# run a specific integration test
pytest test -k test_build_frontend_args
# run a specific integration test on a different platform
CIBW_PLATFORM=linux pytest test -k test_build_frontend_args
```
### Linting, docs
Most developer tasks have a nox interface. This allows you to very simply run tasks without worrying about setting up a development environment (as shown below). This is a slower than setting up a development environment and reusing it, but has the (important) benefit of being highly reproducible; an earlier run does not affect a current run, or anything else on your machine.
You can see a list of sessions by typing `nox -l`; here are a few common ones:
```console
nox -s lint # Run the linters (default)
nox -s tests [-- PYTEST-ARGS] # Run the tests (default)
nox -s docs # Build and serve the documentation
nox -s build # Make SDist and wheel
```
More advanced users can run the update scripts. `update_pins` should work directly, but `update_constraints` needs all versions of Python installed. If you don't want to do that locally, a fast way to run it to use docker to run nox:
```console
docker run --rm -itv $PWD:/src -w /src quay.io/pypa/manylinux_2_24_x86_64:latest pipx run nox -s update_constraints
```
## Maintainer notes
To prepare a development environment.
### Testing sample configs
@@ -148,6 +73,9 @@ This has been moved to using docker, so you only need the following instructions
The dependency update script in the next section requires multiple python versions installed. One way to do this is to use `pyenv`:
```bash
pyenv install 2.7.18
pyenv install 3.5.9
pyenv install 3.6.11
pyenv install 3.7.8
# Optionally add 3.8 and make it the local version;
# otherwise assuming 3.8+ already is your current python version
@@ -156,6 +84,10 @@ pyenv install 3.7.8
Then, you need to make the required virtual environments:
```bash
$(pyenv prefix 2.7.18)/bin/python -m pip install virtualenv
$(pyenv prefix 2.7.18)/bin/python -m virtualenv env27
$(pyenv prefix 3.5.9)/bin/python -m venv env35
$(pyenv prefix 3.6.11)/bin/python -m venv env36
$(pyenv prefix 3.7.8)/bin/python -m venv env37
```
@@ -177,14 +109,15 @@ Then, increment the project version number using:
bin/bump_version.py
```
(or `nox -s bump_version`) You'll be prompted to enter the new version number. Update the changelog when prompted. The script will create a 'bump version' commit and version tag.
You'll be prompted to enter the new version number. Update the changelog when prompted. The script will create a 'bump version' commit and version tag.
Finally, cut the release and push to GitHub.
Finally, cut the release and upload to PyPI/GitHub.
```bash
rm -rf dist
python setup.py sdist bdist_wheel
twine upload dist/*
git push && git push --tags
```
Then head to https://github.com/pypa/cibuildwheel/releases and create a GitHub release from the new tag, pasting in the changelog entry. Once the release is created inside GitHub, a CI job will create the assets and upload them to PyPI.
If there were any schema updates, run `pipx run ./bin/generate_schema.py --schemastore > partial-cibuildwheel.json` and contribute the changes to SchemaStore.
Then head to https://github.com/pypa/cibuildwheel/releases and create a GitHub release from the new tag, pasting in the changelog entry.
+50 -5
View File
@@ -2,13 +2,15 @@
title: Modern C++ standards
---
# Modern C++ standards
Building Python wheels with modern C++ standards (C++11 and later) requires a few tricks.
## Python 2.7 and C++17
The Python 2.7 header files use the `register` keyword, which is [reserved and unused from C+17 onwards](https://en.cppreference.com/w/cpp/keyword/register). Compiling a wheel for Python 2.7 with the C++17 standard is still possible to allow usage of `register` using proper flag `-Wno-register` for gcc/clang and `/wd5033` for MSVC.
## manylinux1 and C++14
The past end-of-life `manylinux1` image (based on CentOS 5) contains a version of GCC and libstdc++ that only supports C++11 and earlier standards. There are however ways to compile wheels with the C++14 standard (and later): https://github.com/pypa/manylinux/issues/118
The default `manylinux1` image (based on CentOS 5) contains a version of GCC and libstdc++ that only supports C++11 and earlier standards. There are however ways to compile wheels with the C++14 standard (and later): https://github.com/pypa/manylinux/issues/118
`manylinux2010` and `manylinux2014` are newer and support all C++ standards (up to C++17).
@@ -16,11 +18,54 @@ The past end-of-life `manylinux1` image (based on CentOS 5) contains a version o
OS X/macOS allows you to specify a so-called "deployment target" version that will ensure backwards compatibility with older versions of macOS. One way to do this is by setting the `MACOSX_DEPLOYMENT_TARGET` environment variable.
However, to enable modern C++ standards, the deployment target needs to be set high enough (since older OS X/macOS versions did not have the necessary modern C++ standard library).
However, to enable modern C++ standards, the deploment target needs to be set high enough (since older OS X/macOS versions did not have the necessary modern C++ standard library).
To get C++11 and C++14 support, `MACOSX_DEPLOYMENT_TARGET` needs to be set to (at least) `"10.9"`. By default, `cibuildwheel` already does this, building 64-bit-only wheels for macOS 10.9 and later.
To get C++17 support, Xcode 9.3+ is needed, requiring at least macOS 10.13 on the build machine. To use C++17 library features and link against the C++ runtime library, set `MACOSX_DEPLOYMENT_TARGET` to `"10.13"` or `"10.14"` (or higher) - macOS 10.13 offers partial C++17 support (e.g., the filesystem header is in experimental, offering `#include <experimental/filesystem>` instead of `#include <filesystem>`); macOS 10.14 has full C++17 support. CPython 3.12+ require 10.13+ anyway.
To get C++17 support, Xcode 9.3+ is needed, requiring at least macOS 10.13 on the build machine. To use C++17 library features and link against the C++ runtime library, set `MACOSX_DEPLOYMENT_TARGET` to `"10.13"` or `"10.14"` (or higher) - macOS 10.13 offers partial C++17 support (e.g., the filesystem header is in experimental, offering `#include <experimental/filesystem>` instead of `#include <filesystem>`); macOS 10.14 has full C++17 support.
However, if only C++17 compiler and standard template library (STL) features are used (not needing a C++17 runtime) it might be possible to set `MACOSX_DEPLOYMENT_TARGET` to a lower value, such as `"10.9"`. To find out if this is the case, try compiling and running with a lower `MACOSX_DEPLOYMENT_TARGET`: if C++17 features are used that require a more recent deployment target, building the wheel should fail.
For more details see https://en.cppreference.com/w/cpp/compiler_support, https://en.wikipedia.org/wiki/Xcode, and https://xcodereleases.com/: Xcode 10 needs macOS 10.13 and Xcode 11 needs macOS 10.14.
## Windows and Python 2.7
In previous years, Microsoft distributed a compiler toolchain called 'Visual C++ for Python 2.7', which was a distribution of MSVC 2008 that was created to make it easier to build Python 2.7 extensions on Windows, because it was fully compatible with the toolchain that built Python 2.7.
This toolchain does not support modern C++ standards (i.e., C++11 and later). And it is hard to find this toolchain these days, since Microsoft removed the download for the required Visual Studio 2008 needed to build a native extension in April, 2021. So, by default, cibuildwheel does not attempt to build Python 2.7 extensions on Windows.
There is an optional workaround, though: the pybind11 project argues and shows that it is [possible to compile Python 2.7 extension with a newer compiler](https://pybind11.readthedocs.io/en/stable/faq.html#working-with-ancient-visual-studio-2008-builds-on-windows) and has an example project showing how to do this: https://github.com/pybind/python_example. The main catch is that a user might need to install [a newer "Microsoft Visual C++ Redistributable"](https://support.microsoft.com/en-us/help/2977003/the-latest-supported-visual-c-downloads), since the newer C++ standard library's binaries are not included by default with the Python 2.7 installation.
Forcing `distutils` or `setuptools` to use a more recent version of MSVC that supports modern C++ can be done in the following way:
1. Set the environment variables `DISTUTILS_USE_SDK=1` and `MSSdk=1`. These two environment variables will tell `distutils`/`setuptools` to not search and set up a build environment that uses Visual C++ for Python 2.7 (aka. MSVC 9).
2. Set up the build Visual Studio build environment you want to use, making sure that e.g. `PATH` contains `cl`, `link`, etc.
- Usually, this can be done through `vcvarsall.bat x86` or `vcvarsall.bat x64`. The exact location of this file depends on the installation, but the default path for VS 2019 Community (e.g. used in AppVeyor's `Visual Studio 2019` image) is `C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Auxiliary\Build\vcvarsall.bat`. **Note**: `vcvarsall.bat` changes the environment variables, so this cannot be run in a subprocess/subshell and consequently running `vsvarsall.bat` in `CIBW_BEFORE_BUILD` does not have any effect.
- In Azure Pipelines, [a `VSBuild` task is available](https://docs.microsoft.com/en-us/azure/devops/pipelines/tasks/build/visual-studio-build)
- In GitHub Actions, the default shell is powershell, so you'll need to use `shell: cmd` to source a `.bat` file, and the directory is `"C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Auxiliary\Build\vcvarsall.bat"`. Or you can use an action, such as the [`ilammy/msvc-dev-cmd@v1`](https://github.com/ilammy/msvc-dev-cmd) action to setup the environment (see example below).
3. Next, call `cibuildwheel`. Unfortunately, MSVC has separate toolchains for compiling 32-bit and 64-bit, so you will need to run `cibuildwheel` twice: once with `CIBW_BUILD=*-win32` after setting up the `x86` build environment, and once with `CIBW_BUILD=*-win_amd64` in a `x64` environment (see previous step).
**GitHub Action example with ilammy/msvc-dev-cmd**:
```yaml
- uses: ilammy/msvc-dev-cmd@v1
- name: Build 64-bit wheel
run: python -m cibuildwheel --output-dir wheelhouse
env:
CIBW_BUILD: cp27-win_amd64
DISTUTILS_USE_SDK: 1
MSSdk: 1
- uses: ilammy/msvc-dev-cmd@v1
with:
arch: x86
- name: Build 32-bit wheel
run: python -m cibuildwheel --output-dir wheelhouse
env:
CIBW_BUILD: cp27-win32
DISTUTILS_USE_SDK: 1
MSSdk: 1
```
Binary file not shown.

Before

Width:  |  Height:  |  Size: 67 KiB

-37
View File
@@ -1,37 +0,0 @@
{
"$schema": "http://json-schema.org/draft-07/schema#",
"$id": "https://github.com/pypa/cibuildwheel/blob/main/cibuildwheel/docs/data/projects.schema.json",
"type": "array",
"description": "The projects file for cibuildwheel",
"items": {
"type": "object",
"additionalProperties": false,
"properties": {
"name": { "type": "string" },
"gh": { "type": "string", "pattern": "[^/]*/[^/]*" },
"stars": { "$ref": "#/items/properties/gh" },
"pypi": { "type": "string" },
"notes": { "type": "string" },
"ci": {
"type": "array",
"items": {
"enum": [
"github",
"travisci",
"appveyor",
"circleci",
"gitlab",
"cirrusci",
"azurepipelines"
]
}
},
"ci_config": { "type": "string" },
"os": {
"type": "array",
"items": { "enum": ["windows", "apple", "linux"] }
}
},
"required": ["name", "gh", "ci", "os"]
}
}
+12 -313
View File
@@ -4,15 +4,9 @@
# stars: GitHub repo (optional, if different from package, such as for Twisted)
# pypi: The pypi name, if different from the GitHub package name
# os: Operating system list, [windows, apple, linux] (optional)
# ci: [appveyor, github, azurepipelines, circleci, gitlab, travisci, cirrusci] (optional)
# ci: [appveyor, github, azurepipelines, circleci, gitlab, travisci] (optional)
# notes: (text, optional)
- name: abess
gh: abess-team/abess
notes: A fast best-subset selection library. It uses cibuildwheel to build a large project with C++ extensions.
ci: [github]
os: [windows, apple, linux]
- name: Matplotlib
gh: matplotlib/matplotlib
notes: The venerable Matplotlib, a Python library with C++ portions
@@ -26,7 +20,7 @@
notes: A simple C extension, without external dependencies
- name: websockets
gh: python-websockets/websockets
gh: aaugustin/websockets
ci: [travisci]
ci_config: .travis.yml
os: [apple, linux]
@@ -108,7 +102,7 @@
- name: pikepdf
gh: pikepdf/pikepdf
ci: [github]
ci: [azurepipelines]
os: [windows, apple, linux]
- name: fathon
@@ -172,13 +166,11 @@
- name: pyzmq
gh: zeromq/pyzmq
ci: [github, circleci]
ci: [github]
os: [windows, apple, linux]
ci_config: .github/workflows/wheels.yml
notes: |
Python bindings for zeromq, the networking library.
Uses Cython on CPython and CFFI on PyPy.
ARM wheels for linux are built natively on CircleCI.
Python bindings for zeromq, the networking library. Uses Cython and CFFI.
- name: python-rapidjson
gh: python-rapidjson/python-rapidjson
@@ -206,12 +198,6 @@
ci: [github]
os: [windows, apple, linux]
- name: pybind11 cross build example
gh: wbarnha/pybind_cmake_example_crossbuild
ci: [github, gitlab]
os: [windows, apple, linux]
notes: Same as pybind11 cmake_example but used to demo Linux ARM + Windows + macOS builds on GitLab
- name: iminuit
gh: scikit-hep/iminuit
ci: [github]
@@ -270,8 +256,8 @@
- name: Tokenizer
gh: OpenNMT/Tokenizer
ci: [github]
os: [windows, apple, linux]
ci: [github, travisci]
os: [apple, linux]
- name: matrixprofile
gh: matrix-profile-foundation/matrixprofile
@@ -286,8 +272,8 @@
- name: CTranslate2
gh: OpenNMT/CTranslate2
ci: [github]
os: [windows, apple, linux]
notes: Includes libraries from the [Intel oneAPI toolkit](https://www.intel.com/content/www/us/en/developer/tools/oneapi/base-toolkit.html) and CUDA kernels compiled for multiple GPU architectures.
os: [apple, linux]
notes: Includes libraries from the [Intel oneAPI toolkit](https://software.intel.com/content/www/us/en/develop/tools/oneapi/base-toolkit.html). The Linux wheels also include CUDA libraries for GPU execution.
- name: MyPy
gh: mypyc/mypy_mypyc-wheels
@@ -295,7 +281,7 @@
pypi: mypy
ci: [github]
os: [apple, linux, windows]
notes: The compiled version of MyPy using MyPyC.
notes: MyPyC, the compiled component of MyPy.
- name: Imagecodecs (fork)
gh: czaki/imagecodecs_build
@@ -325,12 +311,12 @@
notes: Full range of wheels for setuptools rust, with auto release and PyPI deploy.
- name: python-snappy
gh: intake/python-snappy
gh: andrix/python-snappy
ci: [github]
os: [apple, linux, windows]
- name: sourmash
gh: sourmash-bio/sourmash
gh: dib-lab/sourmash
ci: [github]
os: [apple, linux, windows]
@@ -377,290 +363,3 @@
gh: LAMDA-NJU/Deep-Forest
ci: [github]
os: [apple, linux, windows]
- name: cmake
gh: scikit-build/cmake-python-distributions
stars: Kitware/CMake
ci: [github, travisci]
os: [apple, linux, windows]
notes: Multitagged binary builds for all supported platforms, using cibw 2 config configuration.
- name: ninja
gh: scikit-build/ninja-python-distributions
ci: [github, travisci]
os: [apple, linux, windows]
notes: Multitagged binary builds for all supported platforms, using cibw 2 config configuration.
- name: tgcalls
gh: MarshalX/tgcalls
pypi: pytgcalls
ci: [github]
os: [apple, windows]
notes: Python `pybind11` binding to Telegram's WebRTC library with third party dependencies like `OpenSSL`, `MozJPEG`, `FFmpeg`, etc.
- name: pillow-heif
gh: bigcat88/pillow_heif
pypi: pillow-heif
ci: [github, cirrusci]
os: [apple, linux, windows]
notes: Bindings to libheif library with third party dependencies. Fully automated CI for tests and publishing including Apple Silicon builds.
- name: clang-format
gh: ssciwr/clang-format-wheel
ci: [github]
os: [apple, linux, windows]
notes: Scikit-build wrapper around LLVM's CMake, all platforms, generic wheels.
- name: Tornado
gh: tornadoweb/tornado
ci: [github]
os: [linux, apple, windows]
notes: Tornado is a Python web framework and asynchronous networking library. Uses stable ABI for a small C extension.
- name: pytorch-fairseq
gh: facebookresearch/fairseq
ci: [github]
os: [apple, linux]
- name: uvloop
gh: MagicStack/uvloop
ci: [github]
os: [apple, linux]
- name: asyncpg
gh: MagicStack/asyncpg
ci: [github]
os: [apple, linux, windows]
- name: vaex
gh: vaexio/vaex
ci: [github]
os: [apple, linux, windows]
- name: cvxpy
gh: cvxpy/cvxpy
ci: [github]
os: [apple, linux, windows]
- name: duckdb
gh: duckdb/duckdb
ci: [github]
os: [apple, linux, windows]
- name: Triton
gh: openai/triton
ci: [github]
os: [linux]
notes: Self hosted runners
- name: vispy
gh: vispy/vispy
ci: [github]
os: [apple, linux, windows]
- name: Confluent client for Kafka
gh: confluentinc/confluent-kafka-python
ci: [travisci]
os: [apple, linux, windows]
notes: setup in `tools/wheels/build-wheels.bat`
- name: tinyobjloader
gh: tinyobjloader/tinyobjloader
ci: [azurepipelines]
os: [apple, linux, windows]
- name: PyOxidizer
gh: indygreg/PyOxidizer
ci: [github]
os: [apple, linux, windows]
- name: OpenSpiel
gh: google-deepmind/open_spiel
ci: [github]
os: [apple, linux]
- name: Google Benchmark
gh: google/benchmark
ci: [github]
os: [apple, linux, windows]
- name: Apache Beam
gh: apache/beam
ci: [github]
os: [apple, linux, windows]
- name: PyCryptodome
gh: Legrandin/pycryptodome
ci: [github]
os: [apple, linux, windows]
- name: PyYAML
gh: yaml/pyyaml
ci: [github]
os: [apple]
- name: Line Profiler
gh: pyutils/line_profiler
ci: [github]
os: [apple, linux, windows]
- name: OpenTimelineIO
gh: AcademySoftwareFoundation/OpenTimelineIO
ci: [github]
os: [apple, linux, windows]
- name: OpenColorIO
gh: AcademySoftwareFoundation/OpenColorIO
ci: [github]
os: [apple, linux, windows]
- name: Psycopg 3
gh: psycopg/psycopg
pypi: psycopg_binary
notes: A modern implementation of a PostgreSQL adapter for Python
ci: [github]
os: [windows, apple, linux]
- name: Arbor
gh: arbor-sim/arbor
ci: [github]
os: [apple, linux]
notes: >
Arbor is a multi-compartment neuron simulation library; compatible with
next-generation accelerators; best-practices applied to research software;
focused on community-driven development. Includes a
[small script](https://github.com/arbor-sim/arbor/blob/master/scripts/patchwheel.py)
patching `rpath` in bundled libraries.
- name: Kivy
gh: kivy/kivy
ci: [github]
os: [windows, apple, linux]
- name: NCNN
gh: Tencent/ncnn
ci: [github]
os: [windows, apple, linux]
- name: Prophet
gh: facebook/prophet
ci: [github]
os: [windows, apple, linux]
- name: MemRay
gh: bloomberg/memray
ci: [github]
os: [linux]
- name: PyGame
gh: pygame/pygame
ci: [github]
os: [apple, linux]
- name: UltraJSON
gh: ultrajson/ultrajson
ci: [github]
os: [windows, apple, linux]
- name: NumPy
gh: numpy/numpy
ci: [github, travisci]
os: [windows, apple, linux]
- name: Wrapt
gh: GrahamDumpleton/wrapt
ci: [github]
os: [windows, apple, linux]
- name: SimpleJSON
gh: simplejson/simplejson
ci: [github]
os: [windows, apple, linux]
- name: Implicit
gh: benfred/implicit
ci: [github]
os: [windows, apple, linux]
notes: Includes GPU support for linux wheels
- name: power-grid-model
gh: PowerGridModel/power-grid-model
ci: [github]
os: [windows, apple, linux]
notes: Python/C++ library for distribution power system analysis
- name: Python-WebRTC
gh: MarshalX/python-webrtc
ci: [github]
os: [windows, apple, linux]
- name: cf-units
gh: SciTools/cf-units
ci: [github]
os: [apple, linux]
- name: envd
gh: tensorchord/envd
ci: [github]
os: [apple, linux, windows]
notes: A machine learning development environment build tool
- name: mosec
gh: mosecorg/mosec
ci: [github]
os: [linux, apple]
notes: A machine learning model serving framework powered by Rust
- name: ril
gh: Cryptex-github/ril-py
ci: [github]
os: [windows, apple, linux]
pypi: pyril
notes: A python binding to Rust Imaging library using maturin and Pyo3, utilizes Github Action cache to improve speed. Builds abi3 wheels.
- name: Picologging
gh: microsoft/picologging
ci: [github]
os: [windows, apple, linux]
pypi: picologging
notes: A high-performance logging library for Python.
- name: aalink
gh: artfwo/aalink
ci: [github]
os: [windows, apple, linux]
pypi: aalink
notes: Async Python interface for Ableton Link.
- name: werpy
gh: analyticsinmotion/werpy
ci: [github]
os: [windows, linux, apple]
pypi: werpy
notes: An ultra-fast python package using optimized dynamic programming to compute the Word Error Rate (WER).
- name: keyvi
gh: KeyviDev/keyvi
ci: [github]
os: [linux, apple]
pypi: keyvi
notes: FST based key value index highly optimized for size and lookup performance, utilizes ccache action for improved runtime
- name: pedalboard
gh: spotify/pedalboard
ci: [github]
os: [windows, linux, apple]
pypi: pedalboard
notes: >
A Python library for working with audio data and audio plugins
by wrapping the [JUCE](https://github.com/juce-framework/JUCE/)
C++ framework. Uses cibuildwheel to deploy on as many operating
systems and Python versions as possible with only one dependency
(any NumPy).
- name: streaming-form-data
gh: siddhantgoel/streaming-form-data
ci: [github]
os: [apple, linux, windows]
pypi: streaming-form-data
notes: Streaming parser for multipart/form-data written in Cython
+1 -1
View File
@@ -1 +1 @@
<svg xmlns="http://www.w3.org/2000/svg" role="img" viewBox="0 0 24 24" width="16px" fill="#606060"><title>Azure Pipelines icon</title><path d="M1.7997 22.1998h4.2001V24H.0004v-6.0002h1.7993v4.2zM12.6229 9.8975l-7.6353 7.6353 1.4792 1.4792 7.6345-7.6362-1.4784-1.4783zM4.365 15.1793l-.391-.3908c-.5042-.5042-.5828-1.3004-.1869-1.8934l4.6036-6.8958H.9979c-.5513.0014-.9975.4487-.9975 1v5.5002l3.5021 3.6036.8625-.9238zM23.9996.9999v8.9306a2 2 0 0 1-.8893 1.6632l-12.2834 8.2005a.9999.9999 0 0 1-1.2622-.1246L8.04 18.1457c.3313-.3313 6.4656-6.4672 6.7684-6.7698l-2.1855-2.1855L5.8533 15.96l-1.5258-1.525a1.0001 1.0001 0 0 1-.1246-1.2623L12.4034.8893A2 2 0 0 1 14.0666 0h8.9331c.5522 0 .9999.4477.9999.9999zm-2.9997 5.0003c0-1.6567-1.343-2.9997-2.9997-2.9997s-2.9997 1.343-2.9997 2.9997 1.343 2.9997 2.9997 2.9997 2.9997-1.343 2.9997-2.9997zM10.272 20.4625c-.4008 0-.7776-.156-1.0609-.4392l-.368-.3679-.8435.8417 3.4996 3.4996h5.5003c.5522 0 .9999-.4477.9999-.9999v-7.3899L11.1045 20.21a1.4944 1.4944 0 0 1-.8325.2525z"/></svg>
<svg xmlns="http://www.w3.org/2000/svg" role="img" viewBox="0 0 24 24" width="16px" fill="#606060"><title>Azure Pipelines icon</title><path d="M1.8 18H0v6h6v-1.8H1.8zM23 0h-8.93a2 2 0 0 0-1.67.89L9 5.99H1a1 1 0 0 0-1 1v5.5l3.5 3.61 1.2-1.29 1.5 1.5L5 17.53l1.48 1.48 1.22-1.22 1.5 1.51L8 20.5l3.5 3.5H17a1 1 0 0 0 1-1v-8l5.11-3.4A2 2 0 0 0 24 9.92V1a1 1 0 0 0-1-1zm-5 9a3 3 0 1 1 0-6 3 3 0 0 1 0 6z"/></svg>

Before

Width:  |  Height:  |  Size: 1023 B

After

Width:  |  Height:  |  Size: 407 B

-1
View File
@@ -1 +0,0 @@
<svg xmlns="http://www.w3.org/2000/svg" role="img" viewBox="0 0 24 24" width="16px" fill="#606060"><title>Cirrus CI icon</title><path d="M19.422.453a1.113 1.113 0 0 0-1.113 1.115 1.113 1.113 0 0 0 1.112 1.114c1.31 0 2.35 1.042 2.35 2.363 0 1.32-1.04 2.363-2.35 2.363H1.112A1.113 1.113 0 0 0 0 8.52a1.113 1.113 0 0 0 1.113 1.117h18.31c1.308 0 2.35 1.042 2.35 2.363 0 1.32-1.042 2.363-2.35 2.363H1.112A1.113 1.113 0 0 0 0 15.48a1.113 1.113 0 0 0 1.113 1.112h18.31c1.308 0 2.35 1.042 2.35 2.363 0 1.32-1.042 2.363-2.35 2.363H1.112A1.113 1.113 0 0 0 0 22.432a1.113 1.113 0 0 0 1.113 1.115h18.31a1.113 1.113 0 0 0 .206-.022c2.42-.112 4.37-2.12 4.37-4.57 0-1.393-.642-2.634-1.63-3.478C23.356 14.632 24 13.393 24 12c0-1.393-.643-2.632-1.63-3.477C23.357 7.68 24 6.438 24 5.045c0-2.52-2.06-4.592-4.578-4.592z"/></svg>

Before

Width:  |  Height:  |  Size: 808 B

+11 -76
View File
@@ -2,100 +2,35 @@
title: Delivering to PyPI
---
# Delivering to PyPI
After you've built your wheels, you'll probably want to deliver them to PyPI.
## Manual method
### Manual method
On your development machine, install [pipx](https://pypa.github.io/pipx/) and do the following:
On your development machine, do the following...
```bash
# Either download the SDist from your CI, or make it:
# Clear out your 'dist' folder.
rm -rf dist
# Make a source distribution
pipx run build --sdist
python setup.py sdist
# 🏃🏻
# Go and download your wheel files from wherever you put them. e.g. your CI
# provider can be configured to store them for you. Put them all into the
# 'dist' folder.
# Upload using 'twine'
pipx run twine upload dist/*
# Upload using 'twine' (you may need to 'pip install twine')
twine upload dist/*
```
## Automatic method
### Semi-automatic method using wheelhouse-uploader
If you don't need much control over the release of a package, you can set up
cibuildwheel to deliver the wheels straight to PyPI. You just need to bump the
version and tag it.
Obviously, manual steps are for chumps, so we can automate this a little by using [wheelhouse-uploader](https://github.com/ogrisel/wheelhouse-uploader).
### Generic instructions
> Quick note from me - using S3 as a storage didn't work due to a [bug](https://issues.apache.org/jira/browse/LIBCLOUD-792) in libcloud. Feel free to use my fork of that package that fixes the bug `pip install https://github.com/joerick/libcloud/archive/v1.5.0-s3fix.zip`
Make your SDist with the [build](https://github.com/pypa/build) tool, and your wheels with cibuildwheel. If you can make the files available as
downloadable artifacts, this make testing before releases easier (depending on your CI provider's options). The "publish" job/step should collect the
files, and then run `twine upload <paths>` (possibly via [pipx](https://github.com/pypa/pipx)); this should only happen on tags or "releases".
### Automatic method
### GitHub Actions
If you don't need much control over the release of a package, you can set up cibuildwheel to deliver the wheels straight to PyPI. This doesn't require anycloud storage to work - you just need to bump the version and tag it.
GitHub actions has pipx in all the runners as a supported package manager, as
well as several useful actions. Alongside your existing job(s) that runs cibuildwheel to make wheels, you will probably want to build an SDist:
```yaml
make_sdist:
name: Make SDist
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0 # Optional, use if you use setuptools_scm
submodules: true # Optional, use if you have submodules
- name: Build SDist
run: pipx run build --sdist
- uses: actions/upload-artifact@v4
with:
name: cibw-sdist
path: dist/*.tar.gz
```
Then, you need to publish the artifacts that the previous jobs have built. This final job should run only on release or tag, depending on your preference. It gathers the artifacts from the sdist and wheel jobs and uploads them to PyPI. The release environment (`pypi` in the example below) will be created the first time this workflow runs.
This requires setting this GitHub workflow in your project's PyPI settings (for a [new project](https://docs.pypi.org/trusted-publishers/creating-a-project-through-oidc)/[existing project](https://docs.pypi.org/trusted-publishers/adding-a-publisher)).
```yaml
upload_all:
needs: [build_wheels, make_sdist]
environment: pypi
permissions:
id-token: write
runs-on: ubuntu-latest
if: github.event_name == 'release' && github.event.action == 'published'
steps:
- uses: actions/download-artifact@v4
with:
pattern: cibw-*
path: dist
merge-multiple: true
- uses: pypa/gh-action-pypi-publish@release/v1
```
You should use Dependabot to keep the publish action up to date. In the above
example, the same name (the default, "artifact" is used for all upload-artifact
runs, so we can just download all of the in one step into a common directory.
See
[`examples/github-deploy.yml`](https://github.com/pypa/cibuildwheel/blob/main/examples/github-deploy.yml)
for an example configuration that automatically upload wheels to PyPI. Also see
[scikit-hep.org/developer/gha_wheels](https://scikit-hep.org/developer/gha_wheels)
for a complete guide.
### TravisCI
See
[`examples/travis-ci-deploy.yml`](https://github.com/pypa/cibuildwheel/blob/main/examples/travis-ci-deploy.yml)
for an example configuration.
[`examples/travis-ci-deploy.yml`](https://github.com/pypa/cibuildwheel/blob/master/examples/travis-ci-deploy.yml) and [`examples/github-deploy.yml`](https://github.com/pypa/cibuildwheel/blob/master/examples/github-deploy.yml) are example configurations that automatically upload wheels to PyPI. Also check out [this example repo](https://github.com/pypa/cibuildwheel-autopypi-example) for more detailed instructions on how to set this up.
-487
View File
@@ -1,487 +0,0 @@
<div id="flow-diagram">
<div class="diagram-grid" v-bind:style="gridStyle">
<!-- BEHIND THE GRID CONTENT -->
<!-- line for each platform -->
<div class="grid-line" style="grid-row-start: 3"></div>
<div class="grid-line" style="grid-row-start: 4"></div>
<div class="grid-line" style="grid-row-start: 5"></div>
<!-- the docker container outline -->
<div class="grid-outline docker">
<div class="outline">
Manylinux container
</div>
</div>
<!-- the venv outlines -->
<div class="grid-outline testVenv" style="grid-row-start: 3">
<div class="outline">Test virtualenv</div>
</div>
<div class="grid-outline testVenv" style="grid-row-start: 4"><div class="outline"></div></div>
<div class="grid-outline testVenv" style="grid-row-start: 5"><div class="outline"></div></div>
<!-- THE GRID CONTENT -->
<!-- the row labels -->
<div class="grid-row-label" style="grid-row-start: 3">Linux</div>
<div class="grid-row-label" style="grid-row-start: 4">macOS</div>
<div class="grid-row-label" style="grid-row-start: 5">Windows</div>
<!-- the column labels -->
<div class="grid-column-label"
style="grid-row: 1 / span 1; grid-column: 5 / -3">
<div class="label">For each version of Python</div>
</div>
<div class="grid-column-label"
style="grid-row: 2 / span 1;
grid-column: 9 / -3;
margin-bottom: 0.5em;">
<div class="label">If tests are configured</div>
</div>
<!-- the steps -->
<div class="step" v-for="(step, sIndex) in diagram.steps">
<component v-for="action in step"
class="action"
v-bind:is="action.href ? 'a' : 'div'"
v-bind:href="action.href">
<div class="dots" v-if="action.style == 'dot'">
<div class="dot"
v-info="action"
v-for="(platform, pIndex) in platforms"
v-if="action.platforms.includes(platform)"
v-bind:style="{gridRowStart: `${pIndex+3}`,
gridColumnStart: `${sIndex+2}`}">
<div class="dot-graphic"></div>
</div>
</div>
<div class="block-container" v-if="action.style == 'block'">
<div class="block"
v-info="action"
v-bind:style="{...blockStyle(action),
gridColumnStart: `${sIndex+2}`}">
{{action.label}}
</div>
</div>
</a>
</div>
</div>
</div>
<noscript>
<img src="data/how-it-works.png" alt="Diagram showing the steps for each platform">
<p><strong>Enable JavaScript for an interactive version of this diagram.</strong></p>
</noscript>
<script type="module" async>
import Vue from 'https://cdn.jsdelivr.net/npm/vue@2.6.14/dist/vue.esm.browser.js'
import 'https://unpkg.com/@popperjs/core@2/dist/umd/popper.min.js'
import 'https://unpkg.com/tippy.js@6/dist/tippy-bundle.umd.js'
const diagram = {
steps: [
[
{
label: 'copy project into container',
platforms: ['linux'],
style: 'block',
width: 2,
},
],
[], // empty column so that the manylinux outline intersects the previous column in the middle.
[
{
href: 'options/#before-all',
platforms: ['linux', 'macos', 'windows'],
style: 'dot',
tooltip: {
title: 'CIBW_BEFORE_ALL',
tag: 'Optional step',
description: 'Execute a shell command on the build system before any wheels are built.'
},
},
],
[
{
label: 'Install Python',
platforms: ['macos', 'windows'],
style: 'block',
tooltip: {
description: 'Install the version of Python required to build this wheel.'
},
},
],
[
{
href: 'options/#before-build',
platforms: ['linux', 'macos', 'windows'],
style: 'dot',
tooltip: {
title: 'CIBW_BEFORE_BUILD',
tag: 'Optional step',
description: "Execute a shell command preparing each wheel's build.",
},
},
],
[
{
label: 'Build wheel',
href: 'options/#build-frontend',
platforms: ['linux', 'macos', 'windows'],
style: 'block',
tooltip: {
title: 'CIBW_BUILD_FRONTEND',
tag: 'Customisable step',
description: 'Build the wheel according to your package configuration, using the frontend of your choice - pip or build.'
},
},
],
[
{
label: 'Repair using auditwheel',
href: 'options/#repair-wheel-command',
platforms: ['linux'],
style: 'block',
tooltip: {
title: 'CIBW_REPAIR_WHEEL_COMMAND',
tag: 'Customisable step',
description: 'Bundle shared libraries and ensure manylinux compliance by running auditwheel on each built wheel.'
},
},
{
label: 'Repair using delocate',
href: 'options/#repair-wheel-command',
platforms: ['macos'],
style: 'block',
tooltip: {
title: 'CIBW_REPAIR_WHEEL_COMMAND',
tag: 'Customisable step',
description: 'Bundle shared libraries by running delocate on each built wheel.'
},
},
{
env: "CIBW_REPAIR_WHEEL_COMMAND",
href: 'options/#repair-wheel-command',
label: 'repair wheel',
platforms: ['windows'],
style: 'dot',
optional: true,
tooltip: {
title: 'CIBW_REPAIR_WHEEL_COMMAND',
tag: 'Optional step',
description: 'Execute a shell command to repair each built wheel'
},
},
],
[
{
href: 'options/#before-test',
platforms: ['linux', 'macos', 'windows'],
style: 'dot',
tooltip: {
title: 'CIBW_BEFORE_TEST',
tag: 'Optional step',
description: 'Execute a shell command before testing each wheel'
},
},
],
[
{
label: 'Install wheel',
platforms: ['linux', 'macos', 'windows'],
style: 'block',
tooltip: {
description: 'Install the wheel we just built into the test virtualenv.'
},
},
],
[
{
label: 'Test wheel',
href: 'options/#test-command',
platforms: ['linux', 'macos', 'windows'],
style: 'block',
tooltip: {
title: 'CIBW_TEST_COMMAND',
tag: 'Optional step',
description: 'Execute a shell command to test each built wheel'
},
},
],
[
{
label: 'Copy wheels out of container',
platforms: ['linux'],
style: 'block',
width: 2,
},
],
[],
]
}
const diagramComponent = new Vue({
el: '#flow-diagram',
data() {
return {
diagram,
platforms: ['linux', 'macos', 'windows'],
}
},
methods: {
blockStyle(action) {
let start, end
if (action.platforms.includes('linux')) {
start = 3
} else if (action.platforms.includes('macos')) {
start = 4
} else if (action.platforms.includes('windows')) {
start = 5
}
if (action.platforms.includes('windows')) {
end = 6
} else if (action.platforms.includes('macos')) {
end = 5
} else if (action.platforms.includes('linux')) {
end = 4
}
return {
gridRowStart: start.toString(),
gridRowEnd: end.toString(),
gridColumnEnd: `span ${action.width || 1}`,
}
}
},
computed: {
gridStyle() {
return {
gridTemplateRows: `auto auto repeat(${this.platforms.length}, 1fr)`,
gridTemplateColumns: `repeat(${this.diagram.steps.length+1}, auto)`,
}
},
},
directives: {
info: {
inserted(el, binding) {
const action = binding.value
const {env, label, optional=false, description='', href=''} = action
const tooltip = action.tooltip
if (tooltip) {
const tippyInstance = tippy(el, {
content: `
<a class="tooltip-contents" href="${href || ''}">
<div class="tooltip-title">
${tooltip.title || ''}
</div>
<div class="tooltip-tag">
${tooltip.tag || ''}
</div>
<div class="tooltip-description">
${tooltip.description}
</div>
</a>
`,
placement: 'right-start',
allowHTML: true,
maxWidth: 'none',
appendTo: document.getElementById('flow-diagram'),
offset: [0, 10],
onShow(instance) {
const stepEl = el.closest('.action')
stepEl.classList.add('tooltip-open')
instance.setProps({
interactive: tippy.currentInput.isTouch
})
},
onHide() {
const stepEl = el.closest('.action')
stepEl.classList.remove('tooltip-open')
}
})
el.addEventListener('click', e => {
// click event should just open the tooltip on touch devices
if (tippy.currentInput.isTouch) {
e.preventDefault()
}
})
}
}
}
}
})
</script>
<style>
#flow-diagram {
background: #fcfcfc;
/* font-family: Inter; */
/* font-size: 10px; */
font-size: 0.7em;
padding-bottom: 2em;
}
.no-js #flow-diagram {
display: none
}
.diagram-grid {
display: grid;
overflow-x: auto;
}
.platform, .step, .dots, .block-container, .action {
display: contents;
}
.block, .dot {
position: relative;
margin: 2em 0.5em;
}
.block {
background-color: #eee;
/* margin: 0.5em auto; */
padding: 0.5em;
max-width: 6em;
display: flex;
align-items: center;
justify-content: center;
text-align: center;
}
.dot {
/* margin: 0.5em; */
display: flex;
align-items: center;
justify-content: center;
}
.dot-graphic {
width: 1.3em;
height: 1.3em;
border-radius: 50%;
background: #DADADA;
}
a.action {
color: inherit;
}
.action.tooltip-open .dot-graphic {
background-color: #416EDA;
}
.action.tooltip-open .block {
background-color: #416EDA;
color: white;
}
.grid-column-label {
height: 2em;
position: relative;
display: flex;
align-items: center;
justify-content: center;
margin: 0 0.5em;
}
.grid-column-label::before {
content: "";
height: 1px;
background-color: currentColor;
position: absolute;
top: calc(50% - 0.5px);
left: 0;
right: 0;
}
.grid-column-label .label {
background: #fcfcfc;
padding: 0 1em;
position: relative;
}
.grid-row-label {
grid-column: 1 / span 1;
display: flex;
align-items: center;
justify-content: flex-end;
text-align: right;
background: #fcfcfc;
position: relative;
padding-right: 1em;
font-size: 1.1em;
}
.grid-line {
grid-column: 1 / -1;
grid-row-end: span 1;
position: relative;
}
.grid-line::after {
content: "";
height: 1px;
background-color: #979797;
position: absolute;
top: calc(50% - 1px);
left: 0;
right: 0;
}
.grid-outline {
grid-row-end: span 1;
position: relative;
color: #8f8f8f;
}
.grid-outline.docker {
grid-row: 3 / span 1;
grid-column: 3 / -2;
}
.grid-outline.testVenv {
grid-column: 9 / span 3;
}
.grid-outline .outline {
position: absolute;
border: 1px solid #e5e5e5;
padding: 1px 3px;
}
.grid-outline.docker .outline {
left: 0;
right: 0;
top: 0;
bottom: 0;
}
.grid-outline.testVenv .outline {
left: 0;
right: 0;
top: 0.5em;
bottom: 0.5em;
}
/* style the tooltip */
.tippy-box {
background: white;
box-shadow: 0 2px 6px -1px rgba(0,0,0,0.12);
border-radius: 0;
text-align: left;
font-size: 1em;
color: #1F1F1F;
border: 1px solid rgba(0, 0, 0, 0.02);
width: 14em;
min-width: min-content;
}
.tippy-box[data-placement^='top'] > .tippy-arrow::before {
border-top-color: white;
}
.tippy-box[data-placement^='bottom'] > .tippy-arrow::before {
border-bottom-color: white;
}
.tippy-box[data-placement^='left'] > .tippy-arrow::before {
border-left-color: white;
}
.tippy-box[data-placement^='right'] > .tippy-arrow::before {
border-right-color: white;
}
a.tooltip-contents {
color: inherit;
text-decoration: none;
display: block;
}
.tooltip-title {
font-weight: 600;
font-size: 1.1em;
color: #416EDA;
}
.tooltip-tag {
font-weight: 500;
text-transform: uppercase;
font-size: 0.9em;
color: #C9C9C9;
}
.tooltip-description {
margin-top: 1px;
margin-bottom: 2px;
}
</style>
+11 -188
View File
@@ -1,25 +1,11 @@
/* Global styles */
body {
overflow-wrap: break-word;
overflow-x: hidden;
-webkit-font-smoothing: antialiased;
}
p {
margin-bottom: 12px;
line-height: 26px;
}
pre {
margin: 1em 0;
}
pre:first-child {
margin-top: 0;
}
code {
font-size: 85%;
background-color: rgba(27,31,35,.05);
@@ -32,12 +18,11 @@ code {
code, pre, tt {
font-family: SFMono-Regular, Consolas, Liberation Mono, Menlo, monospace;
line-height: 1.4;
}
pre code, pre code.hljs {
padding: 1em;
background-color: #f6f8fa;
.hljs {
padding: 1em;
background-color: #f6f8fa;
}
h1, h2, h3, h4, h5, h6 {
@@ -63,13 +48,6 @@ h1, h2, h3, h4, h5, h6 {
border: none;
}
@media screen and (max-width: 768px) {
.wy-nav-content-wrap .wy-nav-content {
padding: 1em;
}
}
/*
Code block filename style
@@ -113,10 +91,10 @@ h1, h2, h3, h4, h5, h6 {
.code-block-filename + pre code {
padding-top: 0.9em;
}
/*
.toctree-l3 {
border-left: 10px solid transparent;
} */
}
/* import font awesome 4 for icons */
@import url(https://cdnjs.cloudflare.com/ajax/libs/font-awesome/4.7.0/css/font-awesome.min.css);
@@ -134,21 +112,7 @@ h1, h2, h3, h4, h5, h6 {
background-color: white;
display: flex;
justify-content: flex-start;
overflow-x: auto;
overflow-y: hidden;
position: relative;
}
.tabs-header::before {
position: absolute;
content: "";
bottom: 0;
left: 0;
right: 0;
height: 2px;
background: #ccc;
}
.tabs-header > * {
position: relative;
border-bottom: 2px solid #ccc;
}
/* Style the buttons that are used to open the tab content */
@@ -162,13 +126,15 @@ h1, h2, h3, h4, h5, h6 {
font-size: 0.9em;
font-weight: 600;
padding: 8px 16px;
border-bottom: 2px solid #ccc;
border-bottom: 2px solid transparent;
/* put the border on top of the parent border */
margin-bottom: -2px;
}
.tabs-header button:focus-visible {
/* preserve an outline for accessibility purposes */
/* preserve an outline for accesibility purposes */
outline: 1px solid currentColor;
outline-offset: -1px;
}
/* Change background color of buttons on hover */
@@ -187,147 +153,4 @@ h1, h2, h3, h4, h5, h6 {
background-color: #f1f6fa;
padding: 1px 1em;
padding-top: 0.8em;
/* don't collapse inner margins */
overflow-y: hidden;
}
/* Examples tabs styling */
.tabs.examples .tabs-header {
justify-content: flex-end;
background-color: transparent;
border-bottom-style: none;
padding: 0;
}
.tabs.examples .tabs-header::before {
height: 0;
}
.tabs.examples .tabs-header button {
padding: 6px 12px;
border-bottom: none;
color: rgba(0, 0, 0, 0.3);
font-size: 0.8em;
border-top-left-radius: 4px;
border-top-right-radius: 4px;
}
.tabs-header button:hover {
background-color: unset;
}
.tabs.examples .tabs-header button.active {
border-bottom-style: none;
background: #f6f8fa;
color: inherit;
}
.tabs.examples .tabs-content {
padding: 0;
background-color: transparent;
background-color: #f6f8fa;
}
.tabs.examples .tabs-content .tab > * {
padding-left: 0.8em;
padding-right: 0.8em;
}
.tabs.examples .tabs-content .tab > pre {
padding-left: 0;
padding-right: 0;
margin-bottom: 0.5em;
}
/* mkdocs update theme fix-ups */
.rst-content code, .rst-content tt, code {
/* border: none; */
/* background-color: #f0f1f1; */
color: inherit;
}
.rst-content pre code {
border: none;
}
.rst-content h1 code,
.rst-content h2 code,
.rst-content h3 code,
.rst-content h4 code,
.rst-content h5 code,
.rst-content h6 code {
border: none;
color: inherit;
background-color: #f0f1f1;
font-size: 80%;
}
.rst-content table.docutils td code,
.rst-content table.docutils th code,
.rst-content table.field-list td code,
.rst-content table.field-list th code,
.wy-table td code,
.wy-table th code {
/* table elements are already made smaller, the code styling on top of that makes the text too small */
font-size: 82.5%;
}
/* expand all the toctree entries */
.wy-menu-vertical .toctree-l1.current .toctree-l2>ul,
.wy-menu-vertical .toctree-l2.current .toctree-l3>ul,
.wy-menu-vertical .toctree-l3.current .toctree-l4>ul,
.wy-menu-vertical .toctree-l4.current .toctree-l5>ul,
.wy-menu-vertical .toctree-l5.current .toctree-l6>ul,
.wy-menu-vertical .toctree-l6.current .toctree-l7>ul,
.wy-menu-vertical .toctree-l7.current .toctree-l8>ul,
.wy-menu-vertical .toctree-l8.current .toctree-l9>ul,
.wy-menu-vertical .toctree-l9.current .toctree-l10>ul,
.wy-menu-vertical .toctree-l10.current .toctree-l11>ul {
display: block;
}
/* hide all the buttons */
.wy-menu-vertical li.current>a button.toctree-expand,
.wy-menu-vertical li.on a button.toctree-expand,
.wy-menu-vertical li.toctree-l2 button.toctree-expand,
.wy-menu-vertical li a button.toctree-expand {
display: none;
}
/* toctree layout improvements */
.wy-menu-vertical a {
padding: 0.4em 1.2em;
}
.wy-menu-vertical li.current>a,
.wy-menu-vertical li.on a {
font-weight: normal;
padding: 0.4em 1.2em;
/* border-right: 1px solid #f0f0f0; */
}
.wy-menu-vertical li.current a {
padding: 0.4em 1.2em;
}
.wy-menu-vertical li.toctree-l3 a,
.wy-menu-vertical li.toctree-l2.current li.toctree-l3>a {
padding-left: 1.8em;
}
.wy-menu-vertical li.toctree-l2.current>a {
padding: 0.4em 1.2em;
}
.wy-menu-vertical li.toctree-l2.current>a,
.wy-menu-vertical li.toctree-l2.current li.toctree-l3>a {
background: transparent;
}
.wy-menu-vertical li.toctree-l2.current>a:hover,
.wy-menu-vertical li.toctree-l2.current li.toctree-l3>a:hover {
background: #d6d6d6;
}
.wy-menu-vertical li.current>a.current {
background: #fafafa !important;
}
.wy-menu-vertical li.current a {
}
/* word wrap in table cells */
.wy-table-responsive table td, .wy-table-responsive table th {
white-space: normal;
line-height: 1.4;
}
-11
View File
@@ -20,17 +20,6 @@ while (true) {
tabContainer.insertBefore(firstTab);
tabContainer.append(headerContainer, contentContainer)
// add extra classes from the first tab to the container
const classes = Array.from(firstTab[0].classList)
for (let i = 0; i < classes.length; i++) {
const element = classes[i];
if (element == 'tab' || element == 'admonition') {
continue
}
tabContainer.addClass(element)
}
const selectTab = function (index) {
headerContainer.children().removeClass('active')
headerContainer.children().eq(index).addClass('active')
+160 -303
View File
@@ -2,31 +2,33 @@
title: Tips and tricks
---
# Tips and tricks
### Troubleshooting
## Tips
If your wheel didn't compile, check the list below for some debugging tips.
### Linux builds in containers
- A mistake in your config. To quickly test your config without doing a git push and waiting for your code to build on CI, you can test the Linux build in a Docker container. On Mac or Linux, with Docker running, try `cibuildwheel --platform linux`. You'll have to bring your config into the current environment first.
Linux wheels are built in [`manylinux`/`musllinux` containers](https://github.com/pypa/manylinux) to provide binary compatible wheels on Linux, according to [PEP 600](https://www.python.org/dev/peps/pep-0600/) / [PEP 656](https://www.python.org/dev/peps/pep-0656/). Because of this, when building with `cibuildwheel` on Linux, a few things should be taken into account:
- Missing dependency. You might need to install something on the build machine. You can do this in `.travis.yml`, `appveyor.yml`, or `.circleci/config.yml`, with apt-get, brew or choco. Given how the Linux build works, you'll need to use the [`CIBW_BEFORE_BUILD`](options.md#before-build) option.
- Programs and libraries are not installed on the CI runner host, but rather should be installed inside the container - using `yum` for `manylinux2010` or `manylinux2014`, `apt-get` for `manylinux_2_24`, `dnf` for `manylinux_2_28` and `apk` for `musllinux_1_1` or `musllinux_1_2`, or manually. The same goes for environment variables that are potentially needed to customize the wheel building.
- Windows: missing C feature. The Windows C compiler doesn't support C language features invented after 1990, so you'll have to backport your C code to C90. For me, this mostly involved putting my variable declarations at the top of the function like an animal.
`cibuildwheel` supports this by providing the [`CIBW_ENVIRONMENT`](options.md#environment) and [`CIBW_BEFORE_ALL`](options.md#before-all) options to setup the build environment inside the running container.
- MacOS: calling cibuildwheel from a python3 script and getting a `ModuleNotFoundError`? Due to a (fixed) [bug](https://bugs.python.org/issue22490) in CPython, you'll need to [unset the `__PYVENV_LAUNCHER__` variable](https://github.com/pypa/cibuildwheel/issues/133#issuecomment-478288597) before activating a venv.
- The project directory is copied into the container as `/project`, the output directory for the wheels to be copied out is `/output`. In general, this is handled transparently by `cibuildwheel`. For a more finegrained level of control however, the root of the host file system is mounted as `/host`, allowing for example to access shared files, caches, etc. on the host file system. Note that `/host` is not available on CircleCI and GitLab CI due to their Docker policies.
### Linux builds on Docker
- Alternative Docker images can be specified with the `CIBW_MANYLINUX_*_IMAGE`/`CIBW_MUSLLINUX_*_IMAGE` options to allow for a custom, preconfigured build environment for the Linux builds. See [options](options.md#linux-image) for more details.
Linux wheels are built in the [`manylinux` docker images](https://github.com/pypa/manylinux) to provide binary compatible wheels on Linux, according to [PEP 571](https://www.python.org/dev/peps/pep-0571/). Because of this, when building with `cibuildwheel` on Linux, a few things should be taken into account:
- Programs and libraries are not installed on the Travis CI Ubuntu host, but rather should be installed inside of the Docker image (using `yum` for `manylinux2010` or `manylinux2014`, and `apt-get` for `manylinux_2_24`) or manually. The same goes for environment variables that are potentially needed to customize the wheel building. `cibuildwheel` supports this by providing the `CIBW_ENVIRONMENT` and `CIBW_BEFORE_BUILD` options to setup the build environment inside the running Docker image. See [the options docs](options.md#build-environment) for details on these options.
- The project directory is mounted in the running Docker instance as `/project`, the output directory for the wheels as `/output`. In general, this is handled transparently by `cibuildwheel`. For a more finegrained level of control however, the root of the host file system is mounted as `/host`, allowing for example to access shared files, caches, etc. on the host file system. Note that this is not available on CircleCI due to their Docker policies.
- Alternative dockers images can be specified with the `CIBW_MANYLINUX_X86_64_IMAGE`, `CIBW_MANYLINUX_I686_IMAGE`, and `CIBW_MANYLINUX_PYPY_X86_64_IMAGE` options to allow for a custom, preconfigured build environment for the Linux builds. See [options](options.md#manylinux-image) for more details.
### Building macOS wheels for Apple Silicon {: #apple-silicon}
`cibuildwheel` supports both native builds and cross-compiling between `arm64` (Apple Silicon) and `x86_64` (Intel) architectures, including the cross-compatible `universal2` format.
`cibuildwheel` supports cross-compiling `universal2` and `arm64` wheels on `x86_64` runners. With the introduction of Apple Silicon, you now have several choices for wheels for Python 3.9+:
#### Overview of Mac architectures
You have several choices for wheels for Python 3.8+:
##### `x86_64`
#### `x86_64`
The traditional wheel for Apple, loads on Intel machines, and on
Apple Silicon when running Python under Rosetta 2 emulation.
@@ -34,61 +36,60 @@ Apple Silicon when running Python under Rosetta 2 emulation.
Due to a change in naming, Pip 20.3+ (or an installer using packaging 20.5+)
is required to install a binary wheel on macOS Big Sur.
##### `arm64`
#### `arm64`
The native wheel for macOS on Apple Silicon.
Requires Pip 20.3+ (or packaging 20.5+) to install.
##### `universal2`
#### `universal2`
This wheel contains both architectures, causing it to be up to twice the
size (data files do not get doubled, only compiled code). It requires
Pip 20.3 (Packaging 20.6+) to load on Intel, and Pip 21.0.1 (Packaging 20.9+)
to load on Apple Silicon.
The dual-architecture `universal2` has a few benefits, but a key benefit
to a universal wheel is that a user can bundle these wheels into an
application and ship a single binary.
!!! note
The dual-architecture `universal2` has a few benefits, but a key benefit
to a universal wheel is that a user can bundle these wheels into an
application and ship a single binary.
However, if you have a large library, then you might prefer to ship
the two single-arch wheels instead - `x86_64` and `arm64`. In rare cases,
you might want to build all three, but in that case, pip will not download
the universal wheels, because it prefers the most specific wheel
available.
#### What to provide?
However, if you have a large library, then you might prefer to ship
the two single-arch wheels instead - `x86_64` and `arm64`. In rare cases,
you might want to build all three, but in that case, pip will not download
the universal wheels, because it prefers the most specific wheel
available.
Generally speaking, because Pip 20.3 is required for the `universal2` wheel,
most packages should provide both `x86_64` and one of `universal2`/`arm64`
wheels. When Pip 20.3+ is common on macOS, then it might be possible to ship
only the `universal2` wheel.
most packages should provide both `x86_64` and `universal2` wheels for now.
Once Pip 20.3+ is common on macOS, then it should be possible to ship only the
`universal2` wheel.
Opinions vary on which of arch-specific or `universal2` wheels are best - some packagers prefer `universal2` because it's one wheel for all Mac users, so simpler, and easier to build into apps for downstream users. However, because they contain code for both architectures, their file size is larger, meaning they consume more disk space and bandwidth, and are harder to build for some projects.
**Apple Silicon wheels are not built by default**, but can be enabled by adding extra archs to the [`CIBW_ARCHS_MACOS` option](options.md#archs) - e.g. `x86_64 arm64 universal2`. Cross-compilation is provided by the Xcode toolchain.
See [GitHub issue 1333](https://github.com/pypa/cibuildwheel/issues/1333) for more discussion.
!!! important
When cross-compiling on Intel, it is not possible to test `arm64` and the `arm64` part of a `universal2` wheel.
#### How?
`cibuildwheel` will raise a warning to notify you of this - these warnings be be silenced by skipping testing on these platforms: `CIBW_TEST_SKIP: *_arm64 *_universal2:arm64`.
It's easiest to build `x86_64` wheels on `x86_64` runners, and `arm64` wheels on `arm64` runners.
On GitHub Actions, `macos-14` runners are `arm64`, and `macos-13` runners are `x86_64`. So all you need to do is ensure both are in your build matrix.
#### Cross-compiling
If your CI provider doesn't offer arm64 runners yet, or you want to create `universal2`, you'll have to cross-compile. Cross-compilation can be enabled by adding extra archs to the [`CIBW_ARCHS_MACOS` option](options.md#archs) - e.g. `CIBW_ARCHS_MACOS="x86_64 universal2"`. Cross-compilation is provided by Xcode toolchain v12.2+.
Regarding testing,
- On an arm64 runner, it is possible to test x86_64 wheels and both parts of a universal2 wheel using Rosetta 2 emulation.
- On an x86_64 runner, arm64 code can be compiled but it can't be tested. `cibuildwheel` will raise a warning to notify you of this - these warnings can be silenced by skipping testing on these platforms: `CIBW_TEST_SKIP: "*_arm64 *_universal2:arm64"`.
Hopefully, cross-compilation is a temporary situation. Once we have widely
available Apple Silicon CI runners, we can build and test `arm64` and
`universal2` wheels natively. That's why `universal2` wheels are not yet built
by default, and require opt-in by setting `CIBW_ARCHS_MACOS`.
!!! note
If your project uses **Poetry** as a build backend, cross-compiling on macOS [does not currently work](https://github.com/python-poetry/poetry/issues/7107). In some cases arm64 wheels can be built but their tags will be incorrect, with the platform tag showing `x86_64` instead of `arm64`.
Your runner needs Xcode Command Line Tools 12.2 or later to build `universal2` or `arm64`.
As a workaround, the tag can be fixed before running delocate to repair the wheel. The [`wheel tags`](https://wheel.readthedocs.io/en/stable/reference/wheel_tags.html) command is ideal for this. See [this workflow](https://gist.github.com/anderssonjohan/49f07e33fc5cb2420515a8ac76dc0c95#file-build-pendulum-wheels-yml-L39-L53) for an example usage of `wheel tags`.
So far, only CPython 3.9 supports `universal2` and `arm64` wheels.
### Building Linux wheels for non-native archs using emulation {: #emulation}
Here's an example GitHub Actions workflow with a job that builds for Apple Silicon:
> .github/workflows/build_macos.yml
```yml
{% include "../examples/github-apple-silicon.yml" %}
```
### Building non-native architectures using emulation {: #emulation}
cibuildwheel supports building non-native architectures on Linux, via
emulation through the binfmt_misc kernel feature. The easiest way to use this
@@ -109,15 +110,63 @@ Linux), and the other architectures are emulated automatically.
{% include "../examples/github-with-qemu.yml" %}
```
### Building CPython ABI3 wheels (Limited API) {: #abi3}
### Building Apple Silicon wheels on Intel {: #apple-silicon}
The CPython Limited API is a subset of the Python C Extension API that's declared to be forward-compatible, meaning you can compile wheels for one version of Python, and they'll be compatible with future versions. Wheels that use the Limited API are known as ABI3 wheels.
`cibuildwheel` supports cross-compiling `universal2` and `arm64` wheels on `x86_64` runners.
To create a package that builds ABI3 wheels, you'll need to configure your build backend to compile libraries correctly create wheels with the right tags. [Check this repo](https://github.com/joerick/python-abi3-package-sample) for an example of how to do this with setuptools.
These wheels are not built by default, but can be enabled by setting the [`CIBW_ARCHS_MACOS` option](options.md#archs) to `x86_64 arm64 universal2`. Cross-compilation is provided by the Xcode toolchain.
You could also consider running [abi3audit](https://github.com/trailofbits/abi3audit) against the produced wheels in order to check for abi3 violations or inconsistencies. You can run it alongside the default in your [CIBW_REPAIR_WHEEL_COMMAND](options.md#repair-wheel-command).
!!! important
When cross-compiling on Intel, it is not possible to test `arm64` and the `arm64` part of a `universal2` wheel.
### Packages with optional C extensions {: #optional-extensions}
`cibuildwheel` will raise a warning to notify you of this - these warnings be be silenced by skipping testing on these platforms: `CIBW_TEST_SKIP: *_arm64 *_universal2:arm64`.
Hopefully, this is a temporary situation. Once we have widely available Apple Silicon CI runners, we can build and test `arm64` and `universal2` wheels more natively. That's why `universal2` wheels are not yet built by default, and require opt-in by setting `CIBW_ARCHS_MACOS`.
!!! note
Your runner image needs Xcode Command Line Tools 12.2 or later to build `universal2` and `arm64`.
So far, only CPython 3.9 supports `universal2` and `arm64` wheels.
Here's an example GitHub Actions workflow with a job that builds for Apple Silicon:
> .github/workflows/build_macos.yml
```yml
{% include "../examples/github-apple-silicon.yml" %}
```
### Windows and Python 2.7
Building 2.7 extensions on Windows is difficult, because the VS 2008 compiler that was used for the original Python compilation was discontinued in 2018, and has since been removed from Microsoft's downloads. Most people choose to not build Windows 2.7 wheels, or to override the compiler to something more modern.
To override, you need to have a modern compiler toolchain activated, and set `DISTUTILS_USE_SDK=1` and `MSSdk=1`. For example, on GitHub Actions, you would add these steps:
```yaml
- name: Prepare compiler environment for Windows
if: runner.os == 'Windows'
uses: ilammy/msvc-dev-cmd@v1
with:
arch: x64
- name: Set Windows environment variables
if: runner.os == 'Windows'
shell: bash
run: |
echo "DISTUTILS_USE_SDK=1" >> $GITHUB_ENV
echo "MSSdk=1" >> $GITHUB_ENV
# invoke cibuildwheel...
```
cibuildwheel will not try to build 2.7 on Windows unless it detects that the above two variables are set. Note that changing to a more modern compiler will mean your wheel picks up a runtime dependency to a different [Visual C++ Redistributable][]. You also need to be [a bit careful][] in designing your extension; some major binding tools like pybind11 do this for you.
More on setting a custom Windows toolchain in our docs on modern C++ standards [here](cpp_standards.md#windows-and-python-27).
[Visual C++ Redistributable]: https://support.microsoft.com/en-us/topic/the-latest-supported-visual-c-downloads-2647da03-1eea-4433-9aff-95f26a218cc0
[a bit careful]: https://pybind11.readthedocs.io/en/stable/faq.html#working-with-ancient-visual-studio-2008-builds-on-windows
### Building packages with optional C extensions
`cibuildwheel` defines the environment variable `CIBUILDWHEEL` to the value `1` allowing projects for which the C extension is optional to make it mandatory when building wheels.
@@ -131,183 +180,7 @@ myextension = Extension(
)
```
### Automatic updates using Dependabot {: #automatic-updates}
Selecting a moving target (like the latest release) is generally a bad idea in CI. If something breaks, you can't tell whether it was your code or an upstream update that caused the breakage, and in a worse-case scenario, it could occur during a release.
There are two suggested methods for keeping cibuildwheel up to date that instead involve scheduled pull requests using GitHub's Dependabot.
#### Option 1: GitHub Action
If you use GitHub Actions for builds, you can use cibuildwheel as an action:
```yaml
uses: pypa/cibuildwheel@v2.21.3
```
This is a composite step that just runs cibuildwheel using pipx. You can set command-line options as `with:` parameters, and use `env:` as normal.
Then, your `.github/dependabot.yml` file could look like this:
```yaml
version: 2
updates:
- package-ecosystem: "github-actions"
directory: "/"
schedule:
interval: "weekly"
```
#### Option 2: Requirement files
The second option, and the only one that supports other CI systems, is using a `requirements-*.txt` file. The file should have a distinct name and have only one entry:
```bash
# requirements-cibw.txt
cibuildwheel==2.21.3
```
Then your install step would have `python -m pip install -r requirements-cibw.txt` in it. Your `.github/dependabot.yml` file could look like this:
```yaml
version: 2
updates:
- package-ecosystem: "pip"
directory: "/"
schedule:
interval: "daily"
```
This will also try to update other pins in all requirement files, so be sure you want to do that. The only control you have over the files used is via the directory option.
### Alternatives to cibuildwheel options {: #cibw-options-alternatives}
cibuildwheel provides lots of opportunities to configure the build
environment. However, you might consider adding this build configuration into
the package itself - in general, this is preferred, because users of your
package 'sdist' will also benefit.
#### Missing build dependencies {: #cibw-options-alternatives-deps}
If your build needs Python dependencies, rather than using `CIBW_BEFORE_BUILD`, it's best to add these to the
[`build-system.requires`](https://www.python.org/dev/peps/pep-0518/#build-system-table)
section of your pyproject.toml. For example, if your project requires Cython
to build, your pyproject.toml might include a section like this:
```toml
[build-system]
requires = [
"setuptools>=42",
"Cython",
]
build-backend = "setuptools.build_meta"
```
#### Actions you need to perform before building
You might need to run some other commands before building, like running a
script that performs codegen or downloading some data that's not stored in
your source tree.
Rather than using `CIBW_BEFORE_ALL` or `CIBW_BEFORE_BUILD`, you could incorporate
these steps into your package's build process. For example, if you're using
setuptools, you can add steps to your package's `setup.py` using a structure
like this:
```python
import subprocess
import setuptools
import setuptools.command.build_py
class BuildPyCommand(setuptools.command.build_py.build_py):
"""Custom build command."""
def run(self):
# your custom build steps here
# e.g.
# subprocess.run(['python', 'scripts/my_custom_script.py'], check=True)
setuptools.command.build_py.build_py.run(self)
setuptools.setup(
cmdclass={
'build_py': BuildPyCommand,
},
# Usual setup() args.
# ...
)
```
#### Compiler flags
Your build might need some compiler flags to be set through environment variables.
Consider incorporating these into your package, for example, in `setup.py` using [`extra_compile_args` or
`extra_link_args`](https://docs.python.org/3/distutils/setupscript.html#other-options).
### Python 2.7 / PyPy2 wheels
See the [cibuildwheel version 1 docs](https://cibuildwheel.pypa.io/en/1.x/) for information about building Python 2.7 or PyPy2 wheels. There are lots of tricks and workaround there that are no longer required for Python 3 in cibuildwheel 2.
## Troubleshooting
If your wheel didn't compile, you might have a mistake in your config.
To quickly test your config without doing a git push and waiting for your code to build on CI, you can [test the Linux build in a local Docker container](setup.md#local).
### Missing dependencies
Sometimes a build will fail due to a missing dependency.
**If the build is missing a Python package**, you should [add it to pyproject.toml](#cibw-options-alternatives-deps).
**If you need a build tool** (e.g. cmake, automake, ninja), you can install it through a package manager like apt/yum, brew or choco, using the [`CIBW_BEFORE_ALL`](options.md#before-all) option.
**If your build is linking into a native library dependency**, you can build/install that in [`CIBW_BEFORE_ALL`](options.md#before-all). However, on Linux, Mac (and Windows if you're using [delvewheel]), the library that you install will be bundled into the wheel in the [repair step]. So take care to ensure that
- the bundled library doesn't accidentally increase the minimum system requirements (such as the minimum macOS version)
- the bundled library matches the architecture of the wheel you're building when cross-compiling
This is particularly an issue on macOS, where de facto package manager Homebrew will install libraries that are compiled for the specific version of macOS that the build machine is running, rendering the wheels useless for any previous version. And brew will not install the right arch for cross compilation of Apple Silicon wheels.
For these reasons, it's strongly recommended to not use brew for native library dependencies. Instead, we recommend compiling the library yourself. If you compile in the [`CIBW_BEFORE_ALL`](options.md#before-all) step, cibuildwheel will have already set the appropriate `MACOSX_DEPLOYMENT_TARGET` env var, so the library will target the correct version of macOS.
!!! tip
For build steps, Homebrew is still a great resource - you can [look up the build formula](https://formulae.brew.sh/) and use that as a starting point.
[delvewheel]: https://github.com/adang1345/delvewheel
[repair step]: options.md#repair-wheel-command
[Homebrew]: https://brew.sh/
[delocate]: https://github.com/matthew-brett/delocate
### Building Rust wheels
If you build Rust wheels, you need to download the Rust compilers in manylinux.
If you support 32-bit Windows, you need to add this as a potential target. You
can do this on GitHub Actions, for example, with:
```yaml
CIBW_BEFORE_ALL_LINUX: curl -sSf https://sh.rustup.rs | sh -s -- -y
CIBW_BEFORE_ALL_WINDOWS: rustup target add i686-pc-windows-msvc
CIBW_ENVIRONMENT_LINUX: "PATH=$HOME/.cargo/bin:$PATH"
```
Rust does not provide Cargo for musllinux 32-bit, so that needs to be skipped:
```toml
[tool.cibuildwheel]
skip = ["*-musllinux_i686"]
```
Also see [maturin-action](https://github.com/PyO3/maturin-action) which is optimized for Rust wheels, builds the non-Python Rust modules once, and can cross-compile (and can build 32-bit musl, for example).
### macOS: ModuleNotFoundError
Calling cibuildwheel from a python3 script and getting a `ModuleNotFoundError`? Due to a (fixed) [bug](https://bugs.python.org/issue22490) in CPython, you'll need to [unset the `__PYVENV_LAUNCHER__` variable](https://github.com/pypa/cibuildwheel/issues/133#issuecomment-478288597) before activating a venv.
### macOS: 'No module named XYZ' errors after running cibuildwheel
### 'No module named XYZ' errors after running cibuildwheel on macOS
`cibuildwheel` on Mac installs the distributions from Python.org system-wide during its operation. This is necessary, but it can cause some confusing errors after cibuildwheel has finished.
@@ -322,81 +195,20 @@ python3 -m twine upload wheelhouse/*.whl
This doesn't work because while `cibuildwheel` was running, it installed a few new versions of 'python3', so the `python3` run on line 3 isn't the same as the `python3` that ran on line 1.
Solutions to this vary, but the simplest is to use pipx:
Solutions to this vary, but the simplest is to install tools immediately before they're used:
```bash
# most runners have pipx preinstalled, but in case you don't
python3 -m pip install pipx
pipx run cibuildwheel==2.21.3 --output-dir wheelhouse
pipx run twine upload wheelhouse/*.whl
python3 -m pip install cibuildwheel
python3 -m cibuildwheel --output-dir wheelhouse
python3 -m pip install twine
python3 -m twine upload wheelhouse/*.whl
```
### macOS: Passing DYLD_LIBRARY_PATH to delocate
### 'ImportError: DLL load failed: The specific module could not be found' error on Windows
macOS has built-in [System Integrity protections](https://developer.apple.com/library/archive/documentation/Security/Conceptual/System_Integrity_Protection_Guide/RuntimeProtections/RuntimeProtections.html) which limits the use of `DYLD_LIBRARY_PATH` and `LD_LIBRARY_PATH` so that it does not automatically pass to children processes. This means if you set `DYLD_LIBRARY_PATH` before running cibuildwheel, or even set it in `CIBW_ENVIRONMENT`, it will be stripped out of the environment before delocate is called.
Visual Studio and MSVC link the compiled binary wheels to the Microsoft Visual C++ Runtime. Normally, these are included with Python, but when compiling with a newer version of Visual Studio, it is possible users will run into problems on systems that do not have these runtime libraries installed. The solution is to ask users to download the corresponding Visual C++ Redistributable from the [Microsoft website](https://support.microsoft.com/en-us/help/2977003/the-latest-supported-visual-c-downloads) and install it. Since a Python installation normally includes these VC++ Redistributable files for [the version of the MSVC compiler used to compile Python](https://wiki.python.org/moin/WindowsCompilers), this is typically only a problem when compiling a Python 2.7 C extension with a newer compiler, e.g. to support a modern C++ standard (see [the section on modern C++ standards for Python 2.7](cpp_standards.md#windows-and-python-27) for more details).
To work around this, use a different environment variable such as `REPAIR_LIBRARY_PATH` to store the library path, and set `DYLD_LIBRARY_PATH` in [`CIBW_REPAIR_WHEEL_COMMAND_MACOS`](https://cibuildwheel.pypa.io/en/stable/options/#repair-wheel-command), like this:
!!! tab examples "Environment variables"
```yaml
CIBW_REPAIR_WHEEL_COMMAND_MACOS: >
DYLD_LIBRARY_PATH=$REPAIR_LIBRARY_PATH delocate-wheel --require-archs {delocate_archs} -w {dest_dir} -v {wheel}
```
!!! tab examples "pyproject.toml"
```toml
[tool.cibuildwheel.macos]
repair-wheel-command = """\
DYLD_LIBRARY_PATH=$REPAIR_LIBRARY_PATH delocate-wheel \
--require-archs {delocate_archs} -w {dest_dir} -v {wheel}\
"""
```
See [#816](https://github.com/pypa/cibuildwheel/issues/816), thanks to @phoerious for reporting.
### macOS: Building CPython 3.8 wheels on arm64
If you're building on an arm64 runner, you might notice something strange about CPython 3.8 - unlike Python 3.9+, it's cross-compiled to arm64 from an x86_64 version of Python running under Rosetta emulation. This is because (despite the prevalence of arm64 versions of Python 3.8 from Apple and Homebrew) there is no officially supported Python.org installer of Python 3.8 for arm64.
This is fine for simple C extensions, but for more complicated builds on arm64 it becomes an issue.
So, if you want to build macOS arm64 wheels on an arm64 runner (e.g., `macos-14`) on Python 3.8, before invoking cibuildwheel, you should install a native arm64 Python 3.8 interpreter on the runner:
!!! tab "GitHub Actions"
```yaml
- uses: actions/setup-python@v5
with:
python-version: 3.8
if: runner.os == 'macOS' && runner.arch == 'ARM64'
```
!!! tab "Generic"
```bash
curl -o /tmp/Python38.pkg https://www.python.org/ftp/python/3.8.10/python-3.8.10-macos11.pkg
sudo installer -pkg /tmp/Python38.pkg -target /
sh "/Applications/Python 3.8/Install Certificates.command"
```
Then cibuildwheel will detect that it's installed and use it instead. However, you probably don't want to build x86_64 wheels on this Python, unless you're happy with them only supporting macOS 11+.
### macOS: Library dependencies do not satisfy target MacOS
Since delocate 0.11.0 there is added verification that the library binary dependencies match the target macOS version. This is to prevent the situation where a wheel platform tag is lower than the actual minimum macOS version required by the library. To resolve this error you need to build the library to the same macOS version as the target wheel (for example using `MACOSX_DEPLOYMENT_TARGET` environment variable).
Alternatively, you could set `MACOSX_DEPLOYMENT_TARGET` in `CIBW_ENVIRONMENT` to correctly label the wheel as incompatible with older macOS versions.
This error may happen when you install a library using a package manager like Homebrew, which compiles the library for the macOS version of the build machine. This is not suitable for wheels, as the library will only work on the same macOS version as the build machine. You should compile the library yourself, or use a precompiled binary that matches the target macOS version.
### Windows: 'ImportError: DLL load failed: The specific module could not be found'
Visual Studio and MSVC link the compiled binary wheels to the Microsoft Visual C++ Runtime. Normally, the C parts of the runtime are included with Python, but the C++ components are not. When compiling modules using C++, it is possible users will run into problems on systems that do not have the full set of runtime libraries installed. The solution is to ask users to download the corresponding Visual C++ Redistributable from the [Microsoft website](https://support.microsoft.com/en-us/help/2977003/the-latest-supported-visual-c-downloads) and install it.
Additionally, Visual Studio 2019 started linking to an even newer DLL, `VCRUNTIME140_1.dll`, besides the `VCRUNTIME140.dll` that is included with recent Python versions (starting from Python 3.5; see [here](https://wiki.python.org/moin/WindowsCompilers) for more details on the corresponding Visual Studio & MSVC versions used to compile the different Python versions). To avoid this extra dependency on `VCRUNTIME140_1.dll`, the [`/d2FH4-` flag](https://devblogs.microsoft.com/cppblog/making-cpp-exception-handling-smaller-x64/) can be added to the MSVC invocations (check out [this issue](https://github.com/pypa/cibuildwheel/issues/423) for details and references). CPython 3.8.3 and all versions after it have this extra DLL, so it is only needed for 3.8 and earlier.
Additionally, Visual Studio 2019 started linking to an even newer DLL, `VCRUNTIME140_1.dll`, besides the `VCRUNTIME140.dll` that is included with recent Python versions (starting from Python 3.5; see [here](https://wiki.python.org/moin/WindowsCompilers) for more details on the corresponding Visual Studio & MSVC versions used to compile the different Python versions). To avoid this extra dependency on `VCRUNTIME140_1.dll`, the [`/d2FH4-` flag](https://devblogs.microsoft.com/cppblog/making-cpp-exception-handling-smaller-x64/) can be added to the MSVC invocations (check out [this issue](https://github.com/pypa/cibuildwheel/issues/423) for details and references).
To add the `/d2FH4-` flag to a standard `setup.py` using `setuptools`, the `extra_compile_args` option can be used:
@@ -410,12 +222,57 @@ To add the `/d2FH4-` flag to a standard `setup.py` using `setuptools`, the `extr
],
```
To investigate the dependencies of a C extension (i.e., the `.pyd` file, a DLL in disguise) on Windows, [Dependency Walker](http://www.dependencywalker.com/) is a great tool. For diagnosing a failing import, the [dlltracer](https://pypi.org/project/dlltracer/) tool may also provide additional details.
To investigate the dependencies of a C extension (i.e., the `.pyd` file, a DLL in disguise) on Windows, [Dependency Walker](http://www.dependencywalker.com/) is a great tool.
### Windows ARM64 builds {: #windows-arm64}
`cibuildwheel` supports cross-compiling `ARM64` wheels on all Windows runners, but a native ARM64 runner is required for testing. On non-native runners, tests for ARM64 wheels will be automatically skipped with a warning. Add `"*-win_arm64"` to your `CIBW_TEST_SKIP` setting to suppress the warning.
### Automatic updates {: #automatic-updates}
Cross-compilation on Windows relies on a supported build backend. Supported backends use an environment variable to specify their target platform (the one they are compiling native modules for, as opposed to the one they are running on), which is set in [cibuildwheels/windows.py](https://github.com/pypa/cibuildwheel/blob/main/cibuildwheel/windows.py) before building. Currently, `setuptools>=65.4.1` and `setuptools_rust` are the only supported backends.
Selecting a moving target (like the latest release) is generally a bad idea in CI. If something breaks, you can't tell whether it was your code or an upstream update that caused the breakage, and in a worse-case scenario, it could occur during a release.
There are two suggested methods for keeping cibuildwheel up to date that instead involve scheduled pull requests using GitHub's dependabot.
By default, `ARM64` is not enabled when running on non-ARM64 runners. Use [`CIBW_ARCHS`](options.md#archs) to select it.
#### Option 1: GitHub Action
If you use GitHub Actions for builds, you can use cibuildwheel as an action:
```yaml
uses: pypa/cibuildwheel@v1.12.0
```
This is a composite step that just runs cibuildwheel using pipx. You can set command-line options as `with:` parameters, and use `env:` as normal.
Then, your `dependabot.yml` file could look like this:
```yaml
version: 2
updates:
- package-ecosystem: "github-actions"
directory: "/"
schedule:
interval: "weekly"
ignore:
# Optional: Official actions have moving tags like v1;
# if you use those, you don't need updates.
- dependency-name: "actions/*"
```
#### Option 2: Requirement files
The second option, and the only one that supports other CI systems, is using a `requirements-*.txt` file. The file should have a distinct name and have only one entry:
```bash
# requirements-cibw.txt
cibuildwheel==1.12.0
```
Then your install step would have `python -m pip install -r requirements-cibw.txt` in it. Your `dependabot.yml` file could look like this:
```yaml
version: 2
updates:
- package-ecosystem: "pip"
directory: "/"
schedule:
interval: "daily"
```
This will also try to update other pins in all requirement files, so be sure you want to do that. The only control you have over the files used is via the directory option.
-13
View File
@@ -2,8 +2,6 @@
title: Home
---
# cibuildwheel
{%
include-markdown "../README.md"
start="<!--intro-start-->"
@@ -11,14 +9,3 @@ title: Home
%}
To get started, head over to the [setup guide](setup.md).
How it works
------------
This diagram summarises the steps that cibuildwheel takes on each platform to build your package's wheels.
{%
include "diagram.html"
%}
This isn't exhaustive, for a full list of the things cibuildwheel can do, check the [options](options.md) page.
-18
View File
@@ -1,18 +0,0 @@
from __future__ import annotations
import os
import subprocess
import sysconfig
from typing import Any
def define_env(env: Any) -> None:
"Hook function for mkdocs-macros"
@env.macro
def subprocess_run(*args: str) -> str:
"Run a subprocess and return the stdout"
env = os.environ.copy()
scripts = sysconfig.get_path("scripts")
env["PATH"] = f"{scripts}{os.pathsep}{env.get('PATH', '')}"
return subprocess.run(args, check=True, capture_output=True, text=True, env=env).stdout
+293 -1260
View File
File diff suppressed because it is too large Load Diff

Some files were not shown because too many files have changed in this diff Show More