Compare commits

..
Author SHA1 Message Date
Henry Schreiner da84127892 docs: add Python 3.8 to the identifer list (#728) 2021-06-22 18:15:25 -04:00
Joe Rickerby bf3a5590c9 Bump version: v1.12.0 2021-06-22 18:13:50 +01:00
Henry Schreiner 8734e0e4e6 ci: pin dockcross for tests 2021-06-21 12:27:18 -04:00
Henry Schreiner 744812b7d3 ci: pin dockcross for tests 2021-06-21 12:27:18 -04:00
Henry Schreiner 548e0cefe3 feat: support building universal2/arm64 from x64_86 Python 3.8 (#704) 2021-06-21 12:27:18 -04:00
Joe Rickerby 6c62f9ee62 Bump version: v1.11.1.post1 2021-05-28 15:58:38 +01:00
Joe Rickerby 9a608b0b94 Fix mkdocs "Edit in Github" links 2021-05-28 15:53:26 +01:00
Joe Rickerby 5ea360607d Bump version: v1.11.1 2021-05-28 15:50:45 +01:00
Matthieu Darbois dc127cd69d Revert "Temporarily disable CircleCI mac builds (#685)" (#687)
This reverts commit af45aface9.
2021-05-28 14:24:46 +01:00
Joe Rickerby d1898152f7 Fix bump_version script 2021-05-28 14:13:04 +01:00
Joe Rickerby e4a917f717 Revert "Remove travis-ci macOS from docs/examples"
This reverts commit 5de574d36820c2481f4e7557ed6aa0e431ee3e63.
2021-05-24 17:24:36 -04:00
Henry Schreiner 6d5c80c624 ci: use native actions syntax (#662) 2021-05-24 17:24:36 -04:00
Henry SchreinerandJoe Rickerby 6ab9906ff0 chore: rename joerick/cibuildwheel -> pypa/cibuildwheel (#682)
* chore: rename joerick/cibuildwheel -> pypa/cibuildwheel

* Update Appveyor status badge markup

Note that it still has joerick/cibuildwheel - confusingly it's still within my appveyor account, but this is the new project that points to the pypa github repo

* Update Appveyor badge markup again

I went through a few passes to enable that project, here's the latest.

* Update Azure badge, also

* Update Travis badge to reference the .com

Co-authored-by: Joe Rickerby <joerick@mac.com>
2021-05-24 17:24:36 -04:00
Matthieu Darbois 5150122a95 Fix Travis CI tests on ppc64le/s390x (#689)
* Disable manylinux only tests on Travis CI ppc64le

Those tests are taking too much disk space for the Travis CI runner right now.
c.f. https://travis-ci.community/t/running-out-of-disk-space-quota-when-using-docker-on-ppc64le/11634

* Use focal to run aarch64/ppc64le/s390x jobs on Travis CI

This removes allow_failures from the Travis CI s390x job

* Use `focal` distribution for all Travis CI configurations

* Revert "Disable manylinux only tests on Travis CI ppc64le"

This reverts commit d0a09c689c058985c7dcb60b1e11a85a7bfce5e0.

* Skip test_manylinuxXXXX_only on Travis CI ppc64le

* Add a comment on why test_manylinuxXXXX_only test is skipped
2021-05-24 17:24:36 -04:00
Matthieu Darbois e5556f67f4 Remove macOS build on travis-ci (#686)
* Remove macOS build on travis-ci
* Remove travis-ci macOS from docs/examples
2021-05-24 17:24:36 -04:00
Edoardopre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>Henry Schreiner
f6fc96400b docs: update projects.yml (#673)
* Update projects.yml

Added EdoPro98/SimSiPM project

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* docs: add more projects

* docs: rerun projects.py

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Henry Schreiner <henryschreineriii@gmail.com>
2021-05-24 17:24:36 -04:00
Joe Rickerby 7e3bbe6e04 Temporarily disable CircleCI mac builds (#685) 2021-05-24 17:24:36 -04:00
Henry Schreiner 13f875a04a chore: move to PyPA (PSF) CoC (#679) 2021-05-18 13:51:04 -04:00
Henry SchreinerandMatthieu Darbois 339beb061c [1.x] chore: bump to final Python 3.5 pin (#674)
* chore: bump to final Python 3.5 pin

* Update pinned_docker_images.cfg

* Apply suggestions from code review

Co-authored-by: Matthieu Darbois <mayeut@users.noreply.github.com>

Co-authored-by: Matthieu Darbois <mayeut@users.noreply.github.com>
2021-05-14 16:41:12 -04:00
Henry Schreiner fee39c2f35 fix: missing f for fstrings in bin (#659) 2021-05-07 18:17:44 -04:00
Joe Rickerby 73adde0b9f Enable tests on 1.x branch 2021-05-05 08:51:06 +01:00
179 changed files with 27451 additions and 17811 deletions
+6 -31
View File
@@ -1,10 +1,9 @@
version: 2 version: 2
jobs: jobs:
osx-python3.12: osx-python3.6:
macos: macos:
xcode: 15.4.0 xcode: 9.4.1
resource_class: macos.m1.medium.gen1
environment: environment:
PYTHON: python3 PYTHON: python3
steps: steps:
@@ -18,14 +17,11 @@ jobs:
command: venv/bin/python ./bin/run_tests.py command: venv/bin/python ./bin/run_tests.py
no_output_timeout: 30m no_output_timeout: 30m
linux-python3.12: linux-python3.6:
docker: docker:
- image: cimg/python:3.12 - image: circleci/python:3.6
environment: environment:
PYTHON: python3 PYTHON: python3
# Temporarily restrict the tests that are run on CircleCI to prevent
# test timeouts.
PYTEST_ADDOPTS: -k "unit_test or main_tests or test_0_basic or test_docker_images"
steps: steps:
- checkout - checkout
- setup_remote_docker - setup_remote_docker
@@ -38,30 +34,9 @@ jobs:
command: venv/bin/python ./bin/run_tests.py command: venv/bin/python ./bin/run_tests.py
no_output_timeout: 30m no_output_timeout: 30m
linux-aarch64:
machine:
image: default
resource_class: arm.medium
environment:
PYTHON: python3
# Temporarily restrict the tests that are run on CircleCI to prevent
# test timeouts.
PYTEST_ADDOPTS: -k "unit_test or main_tests or test_0_basic or test_docker_images"
steps:
- checkout
- run:
name: Prepare the environment.
command: bash .circleci/prepare.sh
- run:
name: Test.
command: venv/bin/python ./bin/run_tests.py
no_output_timeout: 30m
workflows: workflows:
version: 2 version: 2
all-tests: all-tests:
jobs: jobs:
- osx-python3.12 - osx-python3.6
- linux-python3.12 - linux-python3.6
- linux-aarch64
+2 -7
View File
@@ -1,14 +1,9 @@
#!/usr/bin/env bash
set -o errexit set -o errexit
set -o xtrace set -o xtrace
if [ "$(uname -s)" == "Darwin" ]; then
sudo softwareupdate --install-rosetta --agree-to-license
fi
$PYTHON --version $PYTHON --version
$PYTHON -m venv venv $PYTHON -m pip --version
venv/bin/python -m pip install -U pip $PYTHON -m virtualenv -p $PYTHON venv
venv/bin/python -m pip install -e ".[dev]" venv/bin/python -m pip install -e ".[dev]"
venv/bin/python -m pip freeze venv/bin/python -m pip freeze
venv/bin/python --version venv/bin/python --version
-74
View File
@@ -1,74 +0,0 @@
only_if: changesInclude('.cirrus.yml') || ($CIRRUS_BRANCH == "main" && !changesIncludeOnly('docs/*', '.pre-commit-config.yaml')) || $CIRRUS_BRANCH =~ 'cirrus.*'
run_tests: &RUN_TESTS
install_cibuildwheel_script:
- python -m pip install -e ".[dev]" pytest-custom-exit-code
run_cibuildwheel_tests_script:
- python ./bin/run_tests.py
linux_x86_task:
timeout_in: 120m
compute_engine_instance:
image_project: cirrus-images
image: family/docker-builder
platform: linux
cpu: 8
memory: 8G
install_pre_requirements_script:
- apt install -y python3-venv python-is-python3
<<: *RUN_TESTS
linux_aarch64_task:
compute_engine_instance:
image_project: cirrus-images
image: family/docker-builder-arm64
architecture: arm64
platform: linux
cpu: 4
memory: 4G
install_pre_requirements_script:
- apt install -y python3-venv python-is-python3
<<: *RUN_TESTS
windows_x86_task:
# The task takes ~55 minutes while the timeout happens
# after 60 minutes by default, let's allow some wiggle room.
timeout_in: 120m
windows_container:
image: cirrusci/windowsservercore:visualstudio2022
cpu: 8
memory: 8G
install_pre_requirements_script:
- choco install -y --no-progress python3 --version 3.10.6
- refreshenv
- echo PATH=%PATH% >> "%CIRRUS_ENV%"
<<: *RUN_TESTS
macos_arm64_task:
macos_instance:
image: ghcr.io/cirruslabs/macos-sonoma-xcode
env:
PATH: /opt/homebrew/opt/python@3.10/libexec/bin:$PATH
install_pre_requirements_script:
- brew install python@3.10
<<: *RUN_TESTS
macos_arm64_cp38_task:
macos_instance:
image: ghcr.io/cirruslabs/macos-sonoma-xcode
env:
PATH: /opt/homebrew/opt/python@3.10/libexec/bin:$PATH
PYTEST_ADDOPTS: --run-cp38-universal2 -k 'test_cp38_arm64_testing_universal2_installer or test_arch_auto'
install_pre_requirements_script:
- brew install python@3.10
- curl -fsSLO https://www.python.org/ftp/python/3.8.10/python-3.8.10-macos11.pkg
- sudo installer -pkg python-3.8.10-macos11.pkg -target /
- rm python-3.8.10-macos11.pkg
- sh "/Applications/Python 3.8/Install Certificates.command"
<<: *RUN_TESTS
-6
View File
@@ -1,6 +0,0 @@
* text=auto
*.py diff=python
*.md diff=markdown
*.svg -diff
-31
View File
@@ -1,31 +0,0 @@
name: General issue
description: Open a general issue
body:
- type: markdown
attributes:
value: |
Hi! If you're not sure whether your problem is a bug in cibuildwheel or something else, please consider asking a question in the Discussions tab first.
- type: textarea
id: description
attributes:
label: Description
- type: input
id: log
attributes:
label: Build log
description: If relevant, please provide a link to a build log that shows the problem
placeholder: https://...
validations:
required: false
- type: input
id: ci-config
attributes:
label: CI config
description: Link to the CI config that produced the build log above. e.g. the Github Actions workflow file
placeholder: https://...
validations:
required: false
+7
View File
@@ -0,0 +1,7 @@
# config file for https://github.com/bobvanderlinden/probot-auto-merge
# no approvals required, just the label is enough
minApprovals: {}
requiredLabels:
- merge when green
-11
View File
@@ -1,11 +0,0 @@
version: 2
updates:
# Maintain dependencies for GitHub Actions
- package-ecosystem: "github-actions"
directory: "/"
schedule:
interval: "weekly"
groups:
actions:
patterns:
- "*"
-50
View File
@@ -1,50 +0,0 @@
name: Release
on:
workflow_dispatch:
pull_request:
release:
types:
- published
jobs:
dist:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Build SDist and wheel
run: pipx run build
- uses: actions/upload-artifact@v4
with:
name: cibw-sdist
path: dist/*
- name: Check metadata
run: pipx run twine check dist/*
publish:
needs: [dist]
runs-on: ubuntu-latest
if: github.event_name == 'release' && github.event.action == 'published'
environment:
name: pypi
url: https://pypi.org/p/cibuildwheel
permissions:
id-token: write
attestations: write
steps:
- uses: actions/download-artifact@v4
with:
pattern: cibw-*
path: dist
merge-multiple: true
- name: Generate artifact attestation for sdist and wheel
uses: actions/attest-build-provenance@6149ea5740be74af77f260b9db67e633f6b0a9a1 # v1.4.2
with:
subject-path: "dist/cibuildwheel-*"
- uses: pypa/gh-action-pypi-publish@release/v1
+33 -149
View File
@@ -3,198 +3,82 @@ name: Test
on: on:
push: push:
branches: branches:
- main - master
- 1.x
pull_request: pull_request:
paths-ignore: paths-ignore:
- 'docs/**' - 'docs/**'
- .pre-commit-config.yaml
workflow_dispatch: workflow_dispatch:
# allow manual runs on branches without a PR # allow manual runs on branches without a PR
concurrency:
group: test-${{ github.ref }}
cancel-in-progress: true
jobs: jobs:
lint: pre-commit:
name: Linters (mypy, flake8, etc.) name: Pre-commit checks (mypy, flake8, etc.)
runs-on: ubuntu-latest runs-on: ubuntu-latest
steps: steps:
- uses: actions/checkout@v4 - uses: actions/checkout@v2
- uses: actions/setup-python@v5 - uses: actions/setup-python@v2
id: python - uses: pre-commit/action@v2.0.0
with:
python-version: "3.x"
- uses: pre-commit/action@v3.0.1
- name: Check manifest
run: pipx run --python "${{ steps.python.outputs.python-path }}" nox -s check_manifest
- name: PyLint checks
run: pipx run --python "${{ steps.python.outputs.python-path }}" nox -s pylint -- --output-format=github
test: test:
name: Test on ${{ matrix.os }} (${{ matrix.python_version }}) name: Test cibuildwheel on ${{ matrix.os }}
needs: lint needs: pre-commit
runs-on: ${{ matrix.os }} runs-on: ${{ matrix.os }}
strategy: strategy:
matrix: matrix:
os: [ubuntu-latest, windows-latest, macos-13, macos-14] os: [ubuntu-latest, windows-latest, macos-latest]
python_version: ['3.12'] python_version: ['3.7']
include:
- os: ubuntu-latest
python_version: '3.8'
timeout-minutes: 180 timeout-minutes: 180
steps: steps:
- uses: actions/checkout@v4 - uses: actions/checkout@v2
- uses: actions/setup-python@v5 - uses: actions/setup-python@v2
name: Install Python ${{ matrix.python_version }} name: Install Python ${{ matrix.python_version }}
with: with:
python-version: ${{ matrix.python_version }} python-version: ${{ matrix.python_version }}
- uses: yezz123/setup-uv@v4
# free some space to prevent reaching GHA disk space limits
- name: Clean docker images
if: runner.os == 'Linux'
run: |
docker system prune -a -f
df -h
- name: Install dependencies - name: Install dependencies
run: | run: |
uv pip install --system ".[test]" python -m pip install -e ".[dev]"
- name: Generate a sample project - name: Sample build
if: "contains(github.event.pull_request.labels.*.name, 'CI: Sample build')"
run: |
python bin/sample_build.py
- name: Get some sample wheels
run: | run: |
python -m test.test_projects test.test_0_basic.basic_project sample_proj python -m test.test_projects test.test_0_basic.basic_project sample_proj
cibuildwheel --output-dir wheelhouse sample_proj
- name: Run a sample build (GitHub Action)
uses: ./
with:
package-dir: sample_proj
output-dir: wheelhouse
env: env:
CIBW_ARCHS_MACOS: x86_64 universal2 arm64 CIBW_ARCHS_MACOS: x86_64 universal2 arm64
CIBW_BUILD_FRONTEND: 'build[uv]'
CIBW_FREE_THREADED_SUPPORT: 1
CIBW_PRERELEASE_PYTHONS: 1
- name: Run a sample build (GitHub Action, only) - uses: actions/upload-artifact@v2
uses: ./
with: with:
package-dir: sample_proj name: sample_wheels
output-dir: wheelhouse_only path: wheelhouse
only: cp312-${{ runner.os == 'Linux' && 'manylinux_x86_64' || (runner.os == 'Windows' && 'win_amd64' || 'macosx_x86_64') }}
- name: Create custom configuration file
shell: bash
run: |
cat > sample_proj/cibw.toml <<EOF
[tool.cibuildwheel]
# Only build on CPython 3.12 on native arch
archs = ["native"]
build = "cp312-*"
# Skip musllinux
skip = "*-musllinux*"
EOF
- name: Run a sample build (GitHub Action, config-file)
uses: ./
with:
package-dir: sample_proj
output-dir: wheelhouse_config_file
config-file: sample_proj/cibw.toml
- name: Check Action artefacts
shell: bash
run: |
test $(find wheelhouse -name '*.whl' | wc -l) -ge 1
test $(find wheelhouse_only -name '*.whl' | wc -l) -eq 1
test $(find wheelhouse_config_file -name '*.whl' | wc -l) -eq 1
- uses: actions/upload-artifact@v4
with:
name: cibw-wheels-${{ matrix.os }}-${{ strategy.job-index }}
path: wheelhouse/*.whl
- name: Test cibuildwheel - name: Test cibuildwheel
run: | run: |
python ./bin/run_tests.py --run-podman python ./bin/run_tests.py
emulated-archs:
name: Get qemu emulated architectures
needs: lint
runs-on: ubuntu-latest
outputs:
archs: ${{ steps.archs.outputs.archs }}
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.x"
- name: Install dependencies
run: python -m pip install ".[test]"
- name: Get qemu emulated architectures
id: archs
run: |
OUTPUT=$(python -c "from json import dumps; from test.utils import EMULATED_ARCHS; print(dumps(EMULATED_ARCHS))")
echo "${OUTPUT}"
echo "archs=${OUTPUT}" >> "$GITHUB_OUTPUT"
test-emulated: test-emulated:
name: Test Linux ${{ matrix.arch }} using qemu name: Test emulated cibuildwheel using qemu
needs: emulated-archs needs: pre-commit
runs-on: ubuntu-latest runs-on: ubuntu-latest
timeout-minutes: 180 timeout-minutes: 180
strategy:
matrix:
arch: ${{ fromJSON(needs.emulated-archs.outputs.archs) }}
steps: steps:
- uses: actions/checkout@v4 - uses: actions/checkout@v2
- uses: actions/setup-python@v5 - uses: actions/setup-python@v2
with:
python-version: "3.x"
- name: Install dependencies - name: Install dependencies
run: python -m pip install ".[test,uv]" run: |
python -m pip install -r requirements-dev.txt
- name: Set up QEMU - name: Set up QEMU
id: qemu id: qemu
uses: docker/setup-qemu-action@v3 uses: docker/setup-qemu-action@v1
with: with:
platforms: all platforms: all
- name: Run the emulation tests - name: Run the emulation tests
run: pytest --run-emulation ${{ matrix.arch }} test/test_emulation.py
test-pyodide:
name: Test cibuildwheel building pyodide wheels
needs: lint
runs-on: ubuntu-24.04
timeout-minutes: 180
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
name: Install Python 3.12
with:
python-version: '3.12'
- name: Install dependencies
run: | run: |
python -m pip install ".[test]" pytest --run-emulation test/test_emulation.py
- name: Generate a sample project
run: |
python -m test.test_projects test.test_0_basic.basic_project sample_proj
- name: Run a sample build (GitHub Action)
uses: ./
with:
package-dir: sample_proj
output-dir: wheelhouse
env:
CIBW_PLATFORM: pyodide
- name: Run tests with 'CIBW_PLATFORM' set to 'pyodide'
run: |
python ./bin/run_tests.py
env:
CIBW_PLATFORM: pyodide
+11 -34
View File
@@ -1,15 +1,6 @@
name: Update dependencies name: Update dependencies
on: on:
pull_request:
paths:
- '.github/workflows/update-dependencies.yml'
- 'bin/update_pythons.py'
- 'bin/update_docker.py'
- 'bin/update_virtualenv.py'
- 'bin/projects.py'
- 'docs/data/projects.yml'
- 'noxfile.py'
workflow_dispatch: workflow_dispatch:
schedule: schedule:
- cron: '0 6 * * 1' # "At 06:00 on Monday." - cron: '0 6 * * 1' # "At 06:00 on Monday."
@@ -17,34 +8,23 @@ on:
jobs: jobs:
update-dependencies: update-dependencies:
name: Update dependencies name: Update dependencies
if: github.repository_owner == 'pypa' || github.event_name != 'schedule'
runs-on: ubuntu-latest runs-on: ubuntu-latest
steps: steps:
- uses: actions/checkout@v4 - uses: actions/checkout@v2
- uses: actions/setup-python@v2
- uses: wntrblm/nox@2024.04.15 name: Install Python 3.9
with: with:
python-versions: "3.7, 3.8, 3.9, 3.10, 3.11, 3.12, 3.13" python-version: 3.9
architecture: x64
- name: Install dependencies
run: python -m pip install ".[dev]"
- name: "Run update: dependencies" - name: "Run update: dependencies"
run: nox --force-color -s update_constraints run: python ./bin/update_dependencies.py
- name: "Run update: python configs" - name: "Run update: python configs"
run: nox --force-color -s update_pins run: python ./bin/update_pythons.py --force
- name: "Run update: docs user projects"
run: nox --force-color -s update_proj -- --auth=${{ secrets.GITHUB_TOKEN }}
# we use this step to grab a Github App auth token, so that PRs generated by this workflow
# run the GHA tests.
- uses: tibdex/github-app-token@v2
id: generate-token
if: github.ref == 'refs/heads/main' && github.repository == 'pypa/cibuildwheel'
with:
app_id: ${{ secrets.CIBUILDWHEEL_BOT_APP_ID }}
private_key: ${{ secrets.CIBUILDWHEEL_BOT_APP_PRIVATE_KEY }}
- name: Create Pull Request - name: Create Pull Request
if: github.ref == 'refs/heads/main' && github.repository == 'pypa/cibuildwheel' if: github.ref == 'refs/heads/master' && github.repository == 'pypa/cibuildwheel'
uses: peter-evans/create-pull-request@v6 uses: peter-evans/create-pull-request@v3
with: with:
commit-message: Update dependencies commit-message: Update dependencies
title: '[Bot] Update dependencies' title: '[Bot] Update dependencies'
@@ -53,7 +33,4 @@ jobs:
PR generated by "Update dependencies" [workflow](https://github.com/${{github.repository}}/actions/runs/${{github.run_id}}). PR generated by "Update dependencies" [workflow](https://github.com/${{github.repository}}/actions/runs/${{github.run_id}}).
branch: update-dependencies-pr branch: update-dependencies-pr
committer: "cibuildwheel-bot[bot] <83877280+cibuildwheel-bot[bot]@users.noreply.github.com>"
author: "cibuildwheel-bot[bot] <83877280+cibuildwheel-bot[bot]@users.noreply.github.com>"
token: ${{ steps.generate-token.outputs.token }}
delete-branch: true delete-branch: true
@@ -1,30 +0,0 @@
name: Update the vX.Y tag
on:
release:
types: [released]
workflow_dispatch:
inputs:
TAG_NAME:
description: 'Tag name that the major.minor tag will point to'
required: true
env:
TAG_NAME: ${{ github.event.inputs.TAG_NAME || github.event.release.tag_name }}
jobs:
update_tag:
name: Update the major.minor tag to include the ${{ github.event.inputs.TAG_NAME || github.event.release.tag_name }} changes
environment:
name: releaseNewActionVersion
runs-on: ubuntu-latest
permissions:
contents: write
steps:
- uses: actions/checkout@v4
- name: Update the ${{ env.TAG_NAME }} tag
id: update-major-minor-tag
uses: joerick/update-vX.Y-tag-action@v1.0
with:
source-tag: ${{ env.TAG_NAME }}
-3
View File
@@ -114,6 +114,3 @@ site/
# Virtual environments # Virtual environments
venv* venv*
# PyCharm
.idea/
+3 -28
View File
@@ -1,40 +1,15 @@
linux: linux:
image: python:3.12 image: python:3.8
services: services:
- name: docker:dind - name: docker:dind
entrypoint: ["env", "-u", "DOCKER_HOST"] entrypoint: ["env", "-u", "DOCKER_HOST"]
command: ["dockerd-entrypoint.sh"] command: ["dockerd-entrypoint.sh"]
variables: variables:
DOCKER_HOST: tcp://docker:2375/
DOCKER_DRIVER: overlay2 DOCKER_DRIVER: overlay2
# See https://github.com/docker-library/docker/pull/166 # See https://github.com/docker-library/docker/pull/166
DOCKER_TLS_CERTDIR: "" DOCKER_TLS_CERTDIR: ""
# skip all but the basic tests
# (comment the below line in a PR to debug a Gitlab-specific issue)
PYTEST_ADDOPTS: -k "unit_test or test_0_basic" --suppress-no-test-exit-code
script: script:
- curl -sSL https://get.docker.com/ | sh - curl -sSL https://get.docker.com/ | sh
- python -m pip install -e ".[dev]" pytest-custom-exit-code - python -m pip install -e ".[dev]"
- python ./bin/run_tests.py - python ./bin/run_tests.py
windows:
image: mcr.microsoft.com/windows/servercore:1809
variables:
PYTEST_ADDOPTS: -k "unit_test or test_0_basic" --suppress-no-test-exit-code
before_script:
- choco install python -y --version 3.12.4
script:
- py -m pip install -e ".[dev]" pytest-custom-exit-code
- py bin\run_tests.py
tags:
- saas-windows-medium-amd64
macos:
image: macos-14-xcode-15
variables:
PYTEST_ADDOPTS: -k "unit_test or test_0_basic" --suppress-no-test-exit-code
script:
- python3 -m pip install -e ".[dev]" pytest-custom-exit-code
- python3 ./bin/run_tests.py
tags:
- saas-macos-medium-m1
+41 -63
View File
@@ -1,7 +1,7 @@
repos: repos:
- repo: https://github.com/pre-commit/pre-commit-hooks - repo: https://github.com/pre-commit/pre-commit-hooks
rev: v4.6.0 rev: v3.4.0
hooks: hooks:
- id: check-case-conflict - id: check-case-conflict
- id: check-merge-conflict - id: check-merge-conflict
@@ -13,81 +13,59 @@ repos:
- id: mixed-line-ending - id: mixed-line-ending
- id: trailing-whitespace - id: trailing-whitespace
- repo: https://github.com/astral-sh/ruff-pre-commit - repo: https://github.com/PyCQA/isort
rev: v0.6.3 rev: 5.8.0
hooks: hooks:
- id: ruff - id: isort
args: ["--fix", "--show-fixes"]
- id: ruff-format - repo: https://github.com/psf/black
rev: 21.4b2
hooks:
- id: black
- repo: https://github.com/asottile/setup-cfg-fmt
rev: v1.17.0
hooks:
- id: setup-cfg-fmt
stages: [manual]
- repo: https://github.com/pre-commit/mirrors-mypy - repo: https://github.com/pre-commit/mirrors-mypy
rev: v1.11.2 rev: v0.812
hooks: hooks:
- id: mypy - id: mypy
name: mypy 3.8 on cibuildwheel/ exclude: ^(bin/|cibuildwheel/resources/).*py$
exclude: ^cibuildwheel/resources/.*py|bin/generate_schema.py$ additional_dependencies: [packaging]
args: ["--python-version=3.8"] args: ["--python-version=3.6", "--ignore-missing-imports", "--scripts-are-modules"]
additional_dependencies: &mypy-dependencies
- bracex
- nox
- packaging
- pygithub
- rich
- tomli
- tomli_w
- types-certifi
- types-click
- types-jinja2
- types-pyyaml
- types-requests
- uv
- validate-pyproject
- id: mypy - id: mypy
name: mypy 3.12 name: mypy 3.7+ on bin/
exclude: ^cibuildwheel/resources/.*py$ files: ^bin/.*py$
args: ["--python-version=3.12"] additional_dependencies: [packaging]
additional_dependencies: *mypy-dependencies args: ["--python-version=3.7", "--ignore-missing-imports", "--scripts-are-modules"]
- repo: https://github.com/shellcheck-py/shellcheck-py - repo: https://github.com/asottile/pyupgrade
rev: v0.10.0.1 rev: v2.14.0
hooks: hooks:
- id: shellcheck - id: pyupgrade
name: PyUpgrade 3.6+
args: ["--py36-plus"]
exclude: ^bin/
- id: pyupgrade
name: PyUpgrade 3.7+ on bin
exclude: ^(cibuildwheel|unit_test|test)/
args: ["--py37-plus"]
- repo: https://github.com/PyCQA/flake8
rev: 3.9.1
hooks:
- id: flake8
exclude: cibuildwheel/resources/
additional_dependencies: [flake8-bugbear]
- repo: local - repo: local
hooks: hooks:
- id: disallow-caps - id: disallow-caps
name: Disallow improper capitalization name: Disallow improper capitalization
language: pygrep language: pygrep
entry: PyBind|Numpy|Cmake|Github|PyTest entry: PyBind|Numpy|Cmake|Github
types: types:
- markdown - markdown
exclude: ^docs/working-examples\.md$ # Autogenerated
- id: update-readme-changelog
name: Update README changelog
language: python
entry: bin/update_readme_changelog.py
files: ^docs/changelog.md$
- repo: https://github.com/codespell-project/codespell
rev: v2.3.0
hooks:
- id: codespell
args: ["-L", "sur,assertin", "-w"]
exclude: ^docs/working-examples\.md$ # Autogenerated
- repo: https://github.com/python-jsonschema/check-jsonschema
rev: 0.29.2
hooks:
- id: check-dependabot
- id: check-github-actions
- id: check-github-workflows
- id: check-gitlab-ci
- id: check-readthedocs
- id: check-travis
- id: check-jsonschema
name: Check projects
args: [--schemafile, docs/data/projects.schema.json]
files: '^docs/data/projects.yml$'
- id: check-metaschema
files: '^docs/data/projects.schema.json$'
+10 -11
View File
@@ -2,14 +2,13 @@
# See https://docs.readthedocs.io/en/stable/config-file/v2.html for details # See https://docs.readthedocs.io/en/stable/config-file/v2.html for details
version: 2 version: 2
build: mkdocs:
os: ubuntu-22.04 configuration: mkdocs.yml
tools:
python: "3.12" python:
commands: version: 3.8
- asdf plugin add uv install:
- asdf install uv latest - method: pip
- asdf global uv latest path: .
- uv venv extra_requirements:
- uv pip install -e.[docs] - docs
- NO_COLOR=1 .venv/bin/mkdocs build --strict --site-dir $READTHEDOCS_OUTPUT/html
+17 -30
View File
@@ -4,60 +4,47 @@ language: python
branches: branches:
only: only:
- main - master
jobs: jobs:
include: include:
- name: Linux | x86_64 + i686 | Python 3.9 - name: Linux | x86_64 + i686 | Python 3.6
python: 3.9 python: 3.6
services: docker services: docker
env: PYTHON=python env: PYTHON=python
- name: Linux | arm64 | Python 3.9 - name: Linux | arm64 | Python 3.6
python: 3.9 python: 3.6
services: docker services: docker
arch: arm64-graviton2 arch: arm64
group: edge
virt: vm
env: PYTHON=python env: PYTHON=python
- name: Linux | ppc64le | Python 3.9 - name: Linux | ppc64le | Python 3.6
python: 3.9 python: 3.6
services: docker services: docker
arch: ppc64le arch: ppc64le
allow_failure: True
env: env:
- PYTHON=python - PYTHON=python
# skip test_manylinuxXXXX_only, it uses too much disk space # skip test_manylinuxXXXX_only, it uses too much disk space
# c.f. https://travis-ci.community/t/running-out-of-disk-space-quota-when-using-docker-on-ppc64le/11634 # c.f. https://travis-ci.community/t/running-out-of-disk-space-quota-when-using-docker-on-ppc64le/11634
- PYTEST_ADDOPTS='-k "not test_manylinuxXXXX_only"' - PYTEST_ADDOPTS='-k "not test_manylinuxXXXX_only"'
- name: Windows | x86_64 | Python 3.9 - name: Windows | x86_64 | Python 3.6
os: windows os: windows
language: shell language: shell
before_install: before_install:
- choco upgrade python3 -y --version 3.9.13 --limit-output --params "/InstallDir:C:\\Python39" - choco upgrade python3 -y --version 3.6.8 --limit-output
# Update root certificates to fix SSL error; see http://www.chawn.com/RootCerts.htm
- powershell "md C:\temp\certs; CertUtil -generateSSTFromWU C:\temp\certs\RootStore.sst; Get-ChildItem -Path C:\\temp\certs\Rootstore.sst | Import-Certificate -CertStoreLocation Cert:\\LocalMachine\\Root\\ | out-null"
env: env:
- PYTHON=C:\\Python39\\python - PYTHON=C:\\Python36\\python
- name: Linux | s390x | Python 3.9 - name: Linux | s390x | Python 3.6
python: 3.9 python: 3.6
services: docker services: docker
arch: s390x arch: s390x
allow_failure: True
env: PYTHON=python env: PYTHON=python
install: install: $PYTHON -m pip install -e ".[dev]" pytest-custom-exit-code
- $PYTHON -m pip install -U pip
- $PYTHON -m pip install -e ".[dev]" pytest-custom-exit-code
script: | script: $PYTHON ./bin/run_tests.py
# travis_wait disable the output while waiting
# use the same kind of tricks as in multibuild
(while true; do echo "travis_keep_alive"; sleep 300; done) &
SPINNER_PID=$!
disown
result=0
$PYTHON ./bin/run_tests.py --num-processes 2 || result=1
kill -9 ${SPINNER_PID}
test ${result} -eq 0
+9 -8
View File
@@ -1,11 +1,12 @@
This is a summary of the host Python versions and platforms covered by the different CI platforms: This is a summary of the Python versions and platforms covered by the different CI platforms:
| | 3.8 | 3.9 | 3.10 | 3.11 | 3.12 | | | 3.6 | 3.7 | 3.8 |
|---------|----------------------------------|-----------|-----------|---------|--------------------------------------------------| |----------|------------------------------|------------------------------------------|------------------|
| Linux | Azure Pipelines / GitHub Actions | Travis CI | Cirrus CI | | AppVeyor¹ / CircleCI¹ / GitHub Actions / GitLab¹ | | Linux | Travis CI / CircleCI | AppVeyor² / GitHub Actions | Azure Pipelines |
| macOS | Azure Pipelines | | Cirrus CI | GitLab¹ | AppVeyor¹ /CircleCI¹ / GitHub Actions | | macOS | CircleCI | AppVeyor² / Travis CI¹ / GitHub Actions | Azure Pipelines |
| Windows | Azure Pipelines | Travis CI | Cirrus CI | | AppVeyor¹ / GitHub Actions / GitLab¹ | | Windows | Travis CI / Azure Pipelines | AppVeyor² / GitHub Actions | Azure Pipelines |
> ¹ Runs a reduced set of tests to reduce CI load > ¹ Python version not really pinned, but dependent on the (default) version of image used.
> ² AppVeyor only runs the "basic" test to reduce load.
Non-x86 architectures are covered on Travis CI using Python 3.9. Non-x86 architectures are covered on Travis CI using Python 3.6.
+5 -1
View File
@@ -1,6 +1,6 @@
This project is licensed under the 'BSD 2-clause license'. This project is licensed under the 'BSD 2-clause license'.
Copyright (c) 2017-2023, Joe Rickerby and contributors. All rights reserved. Copyright (c) 2017-2019, Joe Rickerby and contributors. All rights reserved.
Redistribution and use in source and binary forms, with or without Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met: modification, are permitted provided that the following conditions are met:
@@ -22,3 +22,7 @@ SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
Get-pip and Pip (bundled in cibuildwheel/resources) are licensed under the MIT
license. See https://github.com/pypa/get-pip/blob/master/LICENSE.txt
+758 -151
View File
File diff suppressed because it is too large Load Diff
+7 -71
View File
@@ -9,14 +9,6 @@ inputs:
description: 'Folder to place the outputs in, defaults to "wheelhouse"' description: 'Folder to place the outputs in, defaults to "wheelhouse"'
required: false required: false
default: wheelhouse default: wheelhouse
config-file:
description: 'File containing the config, defaults to {package}/pyproject.toml'
required: false
default: ''
only:
description: 'Build a specific wheel only. No need for arch/platform if this is set'
required: false
default: ''
branding: branding:
icon: package icon: package
color: yellow color: yellow
@@ -24,69 +16,13 @@ branding:
runs: runs:
using: composite using: composite
steps: steps:
# Set up the version of Python that supports pyodide
- uses: actions/setup-python@v5
id: python
with:
python-version: "3.12"
update-environment: false
- id: cibw # This needs powershell-core because github.action path is a Windows-style
run: | # path on Windows. Powershell-core understands both types of paths.
# Install cibuildwheel
"${{ steps.python.outputs.python-path }}" -u << "EOF"
import os
import shutil
import sys
import venv
from pathlib import Path
from subprocess import run
class EnvBuilder(venv.EnvBuilder):
def __init__(self):
super().__init__()
def setup_scripts(self, context):
pass
def post_setup(self, context):
super().post_setup(context)
self.bin_path = Path(context.env_exe).parent
run([sys.executable, "-m", "pip", "--python", context.env_exe, "install", r"${{ github.action_path }}"], check=True)
print("::group::Install cibuildwheel")
venv_path = Path(r"${{ runner.temp }}") / "cibw"
if venv_path.exists():
shutil.rmtree(venv_path)
builder = EnvBuilder()
builder.create(venv_path)
cibw_path = [path for path in builder.bin_path.glob("cibuildwheel*") if path.stem == "cibuildwheel"][0]
with open(os.environ["GITHUB_OUTPUT"], "at") as f:
f.write(f"cibw-path={cibw_path}\n")
print("::endgroup::")
EOF
shell: bash
# Redirecting stderr to stdout to fix interleaving issue in Actions.
- run: > - run: >
"${{ steps.cibw.outputs.cibw-path }}" pipx run
"${{ inputs.package-dir }}" --spec ${{ github.action_path }}
${{ inputs.output-dir != '' && format('--output-dir "{0}"', inputs.output-dir) || ''}} cibuildwheel
${{ inputs.config-file != '' && format('--config-file "{0}"', inputs.config-file) || ''}} ${{ inputs.package-dir }}
${{ inputs.only != '' && format('--only "{0}"', inputs.only) || ''}} --output-dir ${{ inputs.output-dir }}
2>&1
shell: bash
if: runner.os != 'Windows'
# Windows needs powershell to interact nicely with Meson
- run: >
& "${{ steps.cibw.outputs.cibw-path }}"
"${{ inputs.package-dir }}"
${{ inputs.output-dir != '' && format('--output-dir "{0}"', inputs.output-dir) || ''}}
${{ inputs.config-file != '' && format('--config-file "{0}"', inputs.config-file) || ''}}
${{ inputs.only != '' && format('--only "{0}"', inputs.only) || ''}}
shell: pwsh shell: pwsh
if: runner.os == 'Windows'
+11 -12
View File
@@ -1,26 +1,25 @@
environment: environment:
matrix: matrix:
- APPVEYOR_BUILD_WORKER_IMAGE: Ubuntu2204 - APPVEYOR_BUILD_WORKER_IMAGE: Ubuntu
APPVEYOR_JOB_NAME: "python312-x64-ubuntu" APPVEYOR_JOB_NAME: "python37-x64-ubuntu"
- APPVEYOR_BUILD_WORKER_IMAGE: Visual Studio 2022 - APPVEYOR_BUILD_WORKER_IMAGE: Visual Studio 2015
APPVEYOR_JOB_NAME: "python312-x64-vs2022" APPVEYOR_JOB_NAME: "python37-x64-vs2015"
- APPVEYOR_BUILD_WORKER_IMAGE: macos-sonoma - APPVEYOR_BUILD_WORKER_IMAGE: macos-mojave
APPVEYOR_JOB_NAME: "python312-x64-macos" APPVEYOR_JOB_NAME: "python37-x64-macos-mojave"
stack: python 3.12 stack: python 3.7
build: off build: off
init: init:
- cmd: set PATH=C:\Python37;C:\Python37\Scripts;%PATH%
- ps: | - ps: |
$BRANCH = if ($env:APPVEYOR_PULL_REQUEST_HEAD_REPO_BRANCH) { $env:APPVEYOR_PULL_REQUEST_HEAD_REPO_BRANCH } else { $env:APPVEYOR_REPO_BRANCH } $BRANCH = if ($env:APPVEYOR_PULL_REQUEST_HEAD_REPO_BRANCH) { $env:APPVEYOR_PULL_REQUEST_HEAD_REPO_BRANCH } else { $env:APPVEYOR_REPO_BRANCH }
if (-not ($BRANCH -eq 'main' -or $BRANCH.ToLower().StartsWith('appveyor-'))) { if (-not ($BRANCH -eq 'master' -or $BRANCH.ToLower().StartsWith('appveyor-'))) {
$env:PYTEST_ADDOPTS='-k "unit_test or test_0_basic" --suppress-no-test-exit-code' $env:PYTEST_ADDOPTS='-k "unit_test or test_0_basic" --suppress-no-test-exit-code'
} }
install: install: python -m pip install -e ".[dev]" pytest-custom-exit-code
- python -m pip install -U pip
- python -m pip install -e ".[dev]" pytest-custom-exit-code
# the '-u' flag is required so the output is in the correct order. # the '-u' flag is required so the output is in the correct order.
# See https://github.com/pypa/cibuildwheel/pull/24 for more info. # See https://github.com/pypa/cibuildwheel/pull/24 for more info.
@@ -28,7 +27,7 @@ test_script: python -u ./bin/run_tests.py
branches: branches:
only: only:
- main - master
skip_commits: skip_commits:
files: files:
+18 -14
View File
@@ -1,38 +1,42 @@
pr:
paths:
exclude:
- docs/*
- .pre-commit-config.yaml
jobs: jobs:
- job: linux_38 - job: linux_38
timeoutInMinutes: 120 pool: {vmImage: 'Ubuntu-18.04'}
pool: {vmImage: 'Ubuntu-20.04'}
steps: steps:
- task: UsePythonVersion@0 - task: UsePythonVersion@0
inputs: inputs:
versionSpec: '3.8' versionSpec: '3.8'
- bash: | - bash: |
python -m pip install -e ".[dev]" python -m pip install -e ".[dev]" pytest-azurepipelines
python ./bin/run_tests.py python ./bin/run_tests.py
- job: macos_38 - job: macos_38
pool: {vmImage: 'macOS-12'} pool: {vmImage: 'macOS-10.15'}
steps: steps:
- task: UsePythonVersion@0 - task: UsePythonVersion@0
inputs: inputs:
versionSpec: '3.8' versionSpec: '3.8'
- bash: | - bash: |
python -m pip install -e ".[dev]" python -m pip install -e ".[dev]" pytest-azurepipelines
python ./bin/run_tests.py --num-processes 2 python ./bin/run_tests.py
- job: windows_36
pool: {vmImage: 'vs2017-win2016'}
timeoutInMinutes: 180
steps:
- task: UsePythonVersion@0
inputs:
versionSpec: '3.6'
- bash: |
python -m pip install -e ".[dev]" pytest-azurepipelines
python ./bin/run_tests.py
- job: windows_38 - job: windows_38
pool: {vmImage: 'windows-2019'} pool: {vmImage: 'vs2017-win2016'}
timeoutInMinutes: 180 timeoutInMinutes: 180
steps: steps:
- task: UsePythonVersion@0 - task: UsePythonVersion@0
inputs: inputs:
versionSpec: '3.8' versionSpec: '3.8'
- bash: | - bash: |
python -m pip install -e ".[dev]" python -m pip install -e ".[dev]" pytest-azurepipelines
python ./bin/run_tests.py python ./bin/run_tests.py
+5 -44
View File
@@ -1,9 +1,5 @@
#!/usr/bin/env python3 #!/usr/bin/env python3
# /// script
# dependencies = ["click", "packaging", "tomli; python_version<'3.11'"]
# ///
from __future__ import annotations from __future__ import annotations
@@ -17,14 +13,10 @@ from pathlib import Path
import click import click
from packaging.version import InvalidVersion, Version from packaging.version import InvalidVersion, Version
if sys.version_info < (3, 11): import cibuildwheel
import tomli as tomllib
else:
import tomllib
config = [ config = [
# file path, version find/replace format # file path, version find/replace format
("pyproject.toml", 'version = "{}"'),
("README.md", "cibuildwheel=={}"), ("README.md", "cibuildwheel=={}"),
("cibuildwheel/__init__.py", '__version__ = "{}"'), ("cibuildwheel/__init__.py", '__version__ = "{}"'),
("docs/faq.md", "cibuildwheel=={}"), ("docs/faq.md", "cibuildwheel=={}"),
@@ -32,6 +24,7 @@ config = [
("docs/setup.md", "cibuildwheel=={}"), ("docs/setup.md", "cibuildwheel=={}"),
("examples/*", "cibuildwheel=={}"), ("examples/*", "cibuildwheel=={}"),
("examples/*", "cibuildwheel@v{}"), ("examples/*", "cibuildwheel@v{}"),
("setup.cfg", "version = {}"),
] ]
RED = "\u001b[31m" RED = "\u001b[31m"
@@ -41,8 +34,7 @@ OFF = "\u001b[0m"
@click.command() @click.command()
def bump_version() -> None: def bump_version() -> None:
with open("pyproject.toml", "rb") as f: current_version = cibuildwheel.__version__
current_version = tomllib.load(f)["project"]["version"]
try: try:
commit_date_str = subprocess.run( commit_date_str = subprocess.run(
@@ -68,7 +60,7 @@ def bump_version() -> None:
print(e) print(e)
print("Failed to get previous version tag information.") print("Failed to get previous version tag information.")
git_changes_result = subprocess.run(["git diff-index --quiet HEAD --"], shell=True, check=False) git_changes_result = subprocess.run(["git diff-index --quiet HEAD --"], shell=True)
repo_has_uncommitted_changes = git_changes_result.returncode != 0 repo_has_uncommitted_changes = git_changes_result.returncode != 0
if repo_has_uncommitted_changes: if repo_has_uncommitted_changes:
@@ -76,7 +68,7 @@ def bump_version() -> None:
sys.exit(1) sys.exit(1)
# fmt: off # fmt: off
print( 'Current version:', current_version) print( 'Current version:', current_version) # noqa
new_version = input(' New version: ').strip() new_version = input(' New version: ').strip()
# fmt: on # fmt: on
@@ -144,26 +136,6 @@ def bump_version() -> None:
while input('Type "done" to continue: ').strip().lower() != "done": while input('Type "done" to continue: ').strip().lower() != "done":
pass pass
# run pre-commit to update the README changelog
subprocess.run(
[
"pre-commit",
"run",
"--files=docs/changelog.md",
],
check=False,
)
# run pre-commit to check that no errors occurred on the second run
subprocess.run(
[
"pre-commit",
"run",
"--files=docs/changelog.md",
],
check=True,
)
subprocess.run( subprocess.run(
[ [
"git", "git",
@@ -186,17 +158,6 @@ def bump_version() -> None:
) )
print("Done.") print("Done.")
print()
print("Push the new version to GitHub with:")
print(f" git push && git push origin v{new_version}")
print()
release_url = "https://github.com/pypa/cibuildwheel/releases/new?" + urllib.parse.urlencode(
{"tag": f"v{new_version}"}
)
print("Then create a release at the URL:")
print(f" {release_url}")
if __name__ == "__main__": if __name__ == "__main__":
+3
View File
@@ -0,0 +1,3 @@
#!/bin/bash
CIBW_PLATFORM=linux pytest "$@"
-309
View File
@@ -1,309 +0,0 @@
#!/usr/bin/env python
# /// script
# dependencies = ["pyyaml"]
# ///
import argparse
import copy
import json
from typing import Any
import yaml
parser = argparse.ArgumentParser()
parser.add_argument("--schemastore", action="store_true", help="Generate schema_store version")
args = parser.parse_args()
starter = """
$schema: http://json-schema.org/draft-07/schema
$id: https://github.com/pypa/cibuildwheel/blob/main/cibuildwheel/resources/cibuildwheel.schema.json
$defs:
inherit:
enum:
- none
- prepend
- append
default: none
description: How to inherit the parent's value.
additionalProperties: false
description: cibuildwheel's settings.
type: object
properties:
archs:
description: Change the architectures built on your machine by default.
type: string_array
before-all:
description: Execute a shell command on the build system before any wheels are built.
type: string_array
before-build:
description: Execute a shell command preparing each wheel's build.
type: string_array
before-test:
description: Execute a shell command before testing each wheel.
type: string_array
build:
default: ['*']
description: Choose the Python versions to build.
type: string_array
build-frontend:
default: default
description: Set the tool to use to build, either "pip" (default for now), "build", or "build[uv]"
oneOf:
- enum: [pip, build, "build[uv]", default]
- type: string
pattern: '^pip; ?args:'
- type: string
pattern: '^build; ?args:'
- type: string
pattern: '^build\\[uv\\]; ?args:'
- type: object
additionalProperties: false
required: [name]
properties:
name:
enum: [pip, build, "build[uv]"]
args:
type: array
items:
type: string
build-verbosity:
type: integer
minimum: -3
maximum: 3
default: 0
description: Increase/decrease the output of pip wheel.
config-settings:
description: Specify config-settings for the build backend.
type: string_table_array
container-engine:
oneOf:
- enum: [docker, podman]
- type: string
pattern: '^docker; ?(create_args|disable_host_mount):'
- type: string
pattern: '^podman; ?(create_args|disable_host_mount):'
- type: object
additionalProperties: false
required: [name]
properties:
name:
enum: [docker, podman]
create-args:
type: array
items:
type: string
disable-host-mount:
type: boolean
dependency-versions:
default: pinned
description: Specify how cibuildwheel controls the versions of the tools it uses
type: string
environment:
description: Set environment variables needed during the build.
type: string_table
environment-pass:
description: Set environment variables on the host to pass-through to the container
during the build.
type: string_array
free-threaded-support:
type: boolean
default: false
description: The project supports free-threaded builds of Python (PEP703)
manylinux-aarch64-image:
type: string
description: Specify alternative manylinux / musllinux container images
manylinux-i686-image:
type: string
description: Specify alternative manylinux / musllinux container images
manylinux-ppc64le-image:
type: string
description: Specify alternative manylinux / musllinux container images
manylinux-pypy_aarch64-image:
type: string
description: Specify alternative manylinux / musllinux container images
manylinux-pypy_i686-image:
type: string
description: Specify alternative manylinux / musllinux container images
manylinux-pypy_x86_64-image:
type: string
description: Specify alternative manylinux / musllinux container images
manylinux-s390x-image:
type: string
description: Specify alternative manylinux / musllinux container images
manylinux-x86_64-image:
type: string
description: Specify alternative manylinux / musllinux container images
musllinux-aarch64-image:
type: string
description: Specify alternative manylinux / musllinux container images
musllinux-i686-image:
type: string
description: Specify alternative manylinux / musllinux container images
musllinux-ppc64le-image:
type: string
description: Specify alternative manylinux / musllinux container images
musllinux-s390x-image:
type: string
description: Specify alternative manylinux / musllinux container images
musllinux-x86_64-image:
type: string
description: Specify alternative manylinux / musllinux container images
repair-wheel-command:
type: string_array
description: Execute a shell command to repair each built wheel.
skip:
description: Choose the Python versions to skip.
type: string_array
test-command:
description: Execute a shell command to test each built wheel.
type: string_array
test-extras:
description: Install your wheel for testing using `extras_require`
type: string_array
test-requires:
description: Install Python dependencies before running the tests
type: string_array
test-skip:
description: Skip running tests on some builds.
type: string_array
"""
schema = yaml.safe_load(starter)
string_array = yaml.safe_load(
"""
- type: string
- type: array
items:
type: string
"""
)
string_table_array = yaml.safe_load(
"""
- type: string
- type: object
additionalProperties: false
patternProperties:
.+:
oneOf:
- type: string
- type: array
items:
type: string
"""
)
string_table = yaml.safe_load(
"""
- type: string
- type: object
additionalProperties: false
patternProperties:
.+:
type: string
"""
)
for value in schema["properties"].values():
match value:
case {"type": "string_array"}:
del value["type"]
value["oneOf"] = string_array
case {"type": "string_table"}:
del value["type"]
value["oneOf"] = string_table
case {"type": "string_table_array"}:
del value["type"]
value["oneOf"] = string_table_array
overrides = yaml.safe_load(
"""
type: array
description: An overrides array
items:
type: object
required: ["select"]
minProperties: 2
additionalProperties: false
properties:
select: {}
inherit:
type: object
additionalProperties: false
properties:
before-all: {"$ref": "#/$defs/inherit"}
before-build: {"$ref": "#/$defs/inherit"}
before-test: {"$ref": "#/$defs/inherit"}
config-settings: {"$ref": "#/$defs/inherit"}
container-engine: {"$ref": "#/$defs/inherit"}
environment: {"$ref": "#/$defs/inherit"}
environment-pass: {"$ref": "#/$defs/inherit"}
repair-wheel-command: {"$ref": "#/$defs/inherit"}
test-command: {"$ref": "#/$defs/inherit"}
test-extras: {"$ref": "#/$defs/inherit"}
test-requires: {"$ref": "#/$defs/inherit"}
"""
)
for key, value in schema["properties"].items():
value["title"] = f'CIBW_{key.replace("-", "_").upper()}'
non_global_options = {k: {"$ref": f"#/properties/{k}"} for k in schema["properties"]}
del non_global_options["build"]
del non_global_options["skip"]
del non_global_options["test-skip"]
del non_global_options["free-threaded-support"]
overrides["items"]["properties"]["select"]["oneOf"] = string_array
overrides["items"]["properties"] |= non_global_options.copy()
del overrides["items"]["properties"]["archs"]
not_linux = non_global_options.copy()
del not_linux["environment-pass"]
del not_linux["container-engine"]
for key in list(not_linux):
if "linux-" in key:
del not_linux[key]
def as_object(d: dict[str, Any]) -> dict[str, Any]:
return {
"type": "object",
"additionalProperties": False,
"properties": copy.deepcopy(d),
}
oses = {
"linux": as_object(non_global_options),
"windows": as_object(not_linux),
"macos": as_object(not_linux),
"pyodide": as_object(not_linux),
}
oses["linux"]["properties"]["repair-wheel-command"] = {
**schema["properties"]["repair-wheel-command"],
"default": "auditwheel repair -w {dest_dir} {wheel}",
}
oses["macos"]["properties"]["repair-wheel-command"] = {
**schema["properties"]["repair-wheel-command"],
"default": "delocate-wheel --require-archs {delocate_archs} -w {dest_dir} -v {wheel}",
}
del oses["linux"]["properties"]["dependency-versions"]
schema["properties"]["overrides"] = overrides
schema["properties"] |= oses
if args.schemastore:
schema["$id"] = "https://json.schemastore.org/partial-cibuildwheel.json"
schema["$schema"] = "http://json-schema.org/draft-07/schema#"
schema["description"] = (
"cibuildwheel's toml file, generated with ./bin/generate_schema.py --schemastore from cibuildwheel."
)
print(json.dumps(schema, indent=2))
+17 -34
View File
@@ -1,25 +1,13 @@
#!/usr/bin/env python3 #!/usr/bin/env python3
"""
Check known projects for usage of requires-python.
Usage:
./bin/inspect_all_known_projects.py --online=$GITHUB_TOKEN
This will cache the results to all_known_setup.yaml; you can reprint
the results without the `--online` setting.
"""
from __future__ import annotations from __future__ import annotations
import ast import ast
from collections.abc import Iterable, Iterator
from pathlib import Path from pathlib import Path
from typing import Iterator
import click import click
import yaml import yaml
from github import Github, GithubException from ghapi.core import GhApi, HTTP404NotFoundError # type: ignore
from rich import print from rich import print
from cibuildwheel.projectfiles import Analyzer from cibuildwheel.projectfiles import Analyzer
@@ -59,44 +47,39 @@ def check_repo(name: str, contents: str) -> str:
class MaybeRemote: class MaybeRemote:
github: Github | None def __init__(self, cached_file: Path | str, *, online: bool) -> None:
contents: dict[str, dict[str, str | None]] self.online = online
if self.online:
def __init__(self, cached_file: Path | str, *, online: str | None) -> None: self.contents: dict[str, dict[str, str | None]] = {
if online is not None:
self.github = Github(online)
self.contents = {
"setup.py": {}, "setup.py": {},
"setup.cfg": {}, "setup.cfg": {},
"pyproject.toml": {}, "pyproject.toml": {},
} }
else: else:
self.github = None
with open(cached_file) as f: with open(cached_file) as f:
self.contents = yaml.safe_load(f) self.contents = yaml.safe_load(f)
def get(self, repo: str, filename: str) -> str | None: def get(self, repo: str, filename: str) -> str | None:
if self.github: if self.online:
try: try:
gh_file = self.github.get_repo(repo).get_contents(filename) self.contents[filename][repo] = (
except GithubException: GhApi(*repo.split("/")).get_content(filename).decode()
)
except HTTP404NotFoundError:
self.contents[filename][repo] = None self.contents[filename][repo] = None
else:
assert not isinstance(gh_file, list)
self.contents[filename][repo] = gh_file.decoded_content.decode(encoding="utf-8")
return self.contents[filename][repo] return self.contents[filename][repo]
elif repo in self.contents[filename]: elif repo in self.contents[filename]:
return self.contents[filename][repo] return self.contents[filename][repo]
else: else:
msg = f"Trying to access {repo}:{filename} and not in cache, rebuild cache" raise RuntimeError(
raise RuntimeError(msg) f"Trying to access {repo}:{filename} and not in cache, rebuild cache"
)
def save(self, filename: Path | str) -> None: def save(self, filename: Path | str) -> None:
with open(filename, "w") as f: with open(filename, "w") as f:
yaml.safe_dump(self.contents, f, default_flow_style=False) yaml.safe_dump(self.contents, f, default_flow_style=False)
def on_each(self, repos: Iterable[str]) -> Iterator[tuple[str, str, str | None]]: def on_each(self, repos: list[str]) -> Iterator[tuple[str, str, str | None]]:
for repo in repos: for repo in repos:
print(f"[bold]{repo}:") print(f"[bold]{repo}:")
for filename in sorted(self.contents, reverse=True): for filename in sorted(self.contents, reverse=True):
@@ -104,8 +87,8 @@ class MaybeRemote:
@click.command() @click.command()
@click.option("--online", help="Set to $GITHUB_TOKEN") @click.option("--online", is_flag=True, help="Remember to set GITHUB_TOKEN")
def main(online: str | None) -> None: def main(online: bool) -> None:
with open(DIR / "../docs/data/projects.yml") as f: with open(DIR / "../docs/data/projects.yml") as f:
known = yaml.safe_load(f) known = yaml.safe_load(f)
+7 -7
View File
@@ -12,13 +12,13 @@ from subprocess import run
import click import click
def shell(cmd, *, check: bool, **kwargs): def shell(cmd, **kwargs):
return run([cmd], shell=True, check=check, **kwargs) return run([cmd], shell=True, **kwargs)
def git_repo_has_changes(): def git_repo_has_changes():
unstaged_changes = shell("git diff-index --quiet HEAD --", check=False).returncode != 0 unstaged_changes = shell("git diff-index --quiet HEAD --").returncode != 0
staged_changes = shell("git diff-index --quiet --cached HEAD --", check=False).returncode != 0 staged_changes = shell("git diff-index --quiet --cached HEAD --").returncode != 0
return unstaged_changes or staged_changes return unstaged_changes or staged_changes
@@ -40,7 +40,7 @@ def main():
timestamp = time.strftime("%Y-%m-%dT%H-%M-%S", time.gmtime()) timestamp = time.strftime("%Y-%m-%dT%H-%M-%S", time.gmtime())
branch_name = f"update-constraints-{timestamp}" branch_name = f"update-constraints-{timestamp}"
shell(f"git checkout -b {branch_name} origin/main", check=True) shell(f"git checkout -b {branch_name} origin/master", check=True)
try: try:
shell("bin/update_dependencies.py", check=True) shell("bin/update_dependencies.py", check=True)
@@ -63,7 +63,7 @@ def main():
"pr", "pr",
"create", "create",
"--repo=pypa/cibuildwheel", "--repo=pypa/cibuildwheel",
"--base=main", "--base=master",
"--title=Update dependencies", "--title=Update dependencies",
f"--body='{body}'", f"--body='{body}'",
], ],
@@ -73,7 +73,7 @@ def main():
print("Done.") print("Done.")
finally: finally:
# remove any local changes # remove any local changes
shell("git checkout -- .", check=True) shell("git checkout -- .")
shell(f"git checkout {previous_branch}", check=True) shell(f"git checkout {previous_branch}", check=True)
shell(f"git branch -D --force {branch_name}", check=True) shell(f"git branch -D --force {branch_name}", check=True)
+28 -76
View File
@@ -1,19 +1,11 @@
#!/usr/bin/env python3 #!/usr/bin/env python3
# /// script
# dependencies = [
# "click",
# "pyyaml",
# "pygithub",
# ]
# ///
""" """
Convert a yaml project list into a nice table. Convert a yaml project list into a nice table.
Suggested usage: Suggested usage:
./bin/projects.py docs/data/projects.yml --online --auth $GITHUB_API_TOKEN ./bin/projects.py docs/data/projects.yml --online --auth $GITHUB_API_TOKEN --readme README.md
git diff git diff
""" """
@@ -24,7 +16,6 @@ import functools
import textwrap import textwrap
import urllib.request import urllib.request
import xml.dom.minidom import xml.dom.minidom
from collections.abc import Iterable, Mapping, Sequence
from datetime import datetime from datetime import datetime
from io import StringIO from io import StringIO
from pathlib import Path from pathlib import Path
@@ -41,7 +32,6 @@ ICONS = (
"circleci", "circleci",
"gitlab", "gitlab",
"travisci", "travisci",
"cirrusci",
"windows", "windows",
"apple", "apple",
"linux", "linux",
@@ -51,7 +41,7 @@ ICONS = (
class Project: class Project:
NAME: int = 0 NAME: int = 0
def __init__(self, config: Mapping[str, Any], github: Github | None = None): def __init__(self, config: dict[str, Any], github: Github | None = None):
try: try:
self.name: str = config["name"] self.name: str = config["name"]
self.gh: str = config["gh"] self.gh: str = config["gh"]
@@ -72,7 +62,7 @@ class Project:
print(f"Broken: {self.stars_repo}") print(f"Broken: {self.stars_repo}")
raise raise
self.num_stars: int = repo.stargazers_count self.num_stars = repo.stargazers_count
self.pushed_at = repo.pushed_at self.pushed_at = repo.pushed_at
if not self.notes: if not self.notes:
notes = repo.description notes = repo.description
@@ -149,35 +139,28 @@ def fetch_icon(icon_name: str) -> None:
f.write(svgElement.toxml()) f.write(svgElement.toxml())
def path_for_icon(icon_name: str, relative_to: Path | None = None) -> Path: def path_for_icon(icon_name: str) -> Path:
from_path = Path.cwd() if relative_to is None else relative_to.parent return Path(".") / "docs" / "data" / "readme_icons" / f"{icon_name}.svg"
absolute_path = PROJECT_ROOT / "docs" / "data" / "readme_icons" / f"{icon_name}.svg"
return absolute_path.resolve().relative_to(from_path.resolve())
def get_projects( def str_projects(
config: Iterable[Mapping[str, Any]], config: list[dict[str, Any]],
*, *,
online: bool = True, online: bool = True,
auth: str | None = None, auth: str | None = None,
) -> list[Project]: ) -> str:
io = StringIO()
print = functools.partial(builtins.print, file=io)
if online: if online:
for icon in ICONS: for icon in ICONS:
fetch_icon(icon) fetch_icon(icon)
github = Github(auth) if online else None github = Github(auth) if online else None
return sorted((Project(item, github) for item in config), reverse=online) projects = sorted((Project(item, github) for item in config), reverse=online)
def render_projects(projects: Sequence[Project], *, dest_path: Path, include_info: bool = True):
io = StringIO()
print = functools.partial(builtins.print, file=io)
print(Project.header()) print(Project.header())
for project in projects: for project in projects:
print(project.table_row()) print(project.table_row())
@@ -187,9 +170,8 @@ def render_projects(projects: Sequence[Project], *, dest_path: Path, include_inf
print() print()
for icon in ICONS: for icon in ICONS:
print(f"[{icon} icon]: {path_for_icon(icon, relative_to=dest_path).as_posix()}") print(f"[{icon} icon]: {path_for_icon(icon).as_posix()}")
if include_info:
print() print()
for project in projects: for project in projects:
print(project.info()) print(project.info())
@@ -197,63 +179,33 @@ def render_projects(projects: Sequence[Project], *, dest_path: Path, include_inf
return io.getvalue() return io.getvalue()
def insert_projects_table( @click.command(help="Try ./bin/projects.py docs/data/projects.yml --readme README.md")
file: Path,
*,
projects: Sequence[Project],
input_filename: str,
include_info: bool = True,
):
text = file.read_text()
projects_table = render_projects(projects, include_info=include_info, dest_path=file)
start_str = "<!-- START bin/projects.py -->\n"
start = text.find(start_str)
assert start != -1
end = text.find("<!-- END bin/projects.py -->\n")
assert end != -1
generated_note = f"<!-- this section is generated by bin/projects.py. Don't edit it directly, instead, edit {input_filename} -->"
new_text = (
f"{text[:start + len(start_str)]}\n{generated_note}\n\n{projects_table}\n{text[end:]}"
)
file.write_text(new_text)
PROJECT_ROOT = Path(__file__).parent / ".."
README_FILE = PROJECT_ROOT / "README.md"
DOCS_PAGE = PROJECT_ROOT / "docs" / "working-examples.md"
@click.command(help="Try ./bin/projects.py docs/data/projects.yml")
@click.argument("input", type=click.File("r")) @click.argument("input", type=click.File("r"))
@click.option("--online/--no-online", default=True, help="Get info from GitHub") @click.option("--online/--no-online", default=True, help="Get info from GitHub")
@click.option("--auth", help="GitHub authentication token") @click.option("--auth", help="GitHub authentication token")
@click.option( @click.option("--readme", type=click.File("r+"), help="Modify a readme file if given")
"--dry-run", default=False, help="Print the output, rather than writing it to files in the repo"
)
def projects( def projects(
input: TextIO, input: TextIO,
online: bool, online: bool,
auth: str | None, auth: str | None,
dry_run: bool, readme: TextIO | None,
) -> None: ) -> None:
config = yaml.safe_load(input) config = yaml.safe_load(input)
projects = get_projects(config, online=online, auth=auth) output = str_projects(config, online=online, auth=auth)
if dry_run: if readme is None:
output = render_projects(projects, dest_path=README_FILE)
print(output) print(output)
else: else:
insert_projects_table( text = readme.read()
README_FILE, projects=projects[:10], input_filename=input.name, include_info=False start_str = "<!-- START bin/projects.py -->\n"
) start = text.find(start_str)
insert_projects_table( end = text.find("<!-- END bin/projects.py -->\n")
DOCS_PAGE, projects=projects, input_filename=input.name, include_info=False generated_note = f"<!-- this section is generated by bin/projects.py. Don't edit it directly, instead, edit {input.name} -->"
) new_text = f"{text[:start + len(start_str)]}\n{generated_note}\n\n{output}\n{text[end:]}"
readme.seek(0)
readme.write(new_text)
readme.truncate()
if __name__ == "__main__": if __name__ == "__main__":
+18 -29
View File
@@ -7,7 +7,7 @@ import shutil
import sys import sys
import textwrap import textwrap
import time import time
import typing from collections import namedtuple
from glob import glob from glob import glob
from pathlib import Path from pathlib import Path
from subprocess import run from subprocess import run
@@ -16,13 +16,13 @@ from urllib.parse import quote
import click import click
def shell(cmd, *, check: bool, **kwargs): def shell(cmd, **kwargs):
return run([cmd], shell=True, check=check, **kwargs) return run([cmd], shell=True, **kwargs)
def git_repo_has_changes(): def git_repo_has_changes():
unstaged_changes = shell("git diff-index --quiet HEAD --", check=False).returncode != 0 unstaged_changes = shell("git diff-index --quiet HEAD --").returncode != 0
staged_changes = shell("git diff-index --quiet --cached HEAD --", check=False).returncode != 0 staged_changes = shell("git diff-index --quiet --cached HEAD --").returncode != 0
return unstaged_changes or staged_changes return unstaged_changes or staged_changes
@@ -34,22 +34,17 @@ def generate_basic_project(path):
project.generate(path) project.generate(path)
class CIService(typing.NamedTuple): CIService = namedtuple("CIService", "name dst_config_path badge_md")
name: str
dst_config_path: str
badge_md: str
services = [ services = [
CIService( CIService(
name="appveyor", name="appveyor",
dst_config_path="appveyor.yml", dst_config_path="appveyor.yml",
badge_md="[![Build status](https://ci.appveyor.com/api/projects/status/gt3vwl88yt0y3hur/branch/{branch}?svg=true)](https://ci.appveyor.com/project/joerick/cibuildwheel/branch/{branch})", badge_md="[![Build status](https://ci.appveyor.com/api/projects/status/wbsgxshp05tt1tif/branch/{branch}?svg=true)](https://ci.appveyor.com/project/pypa/cibuildwheel/branch/{branch})",
), ),
CIService( CIService(
name="azure-pipelines", name="azure-pipelines",
dst_config_path="azure-pipelines.yml", dst_config_path="azure-pipelines.yml",
badge_md="[![Build Status](https://dev.azure.com/joerick0429/cibuildwheel/_apis/build/status/pypa.cibuildwheel?branchName={branch})](https://dev.azure.com/joerick0429/cibuildwheel/_build/latest?definitionId=2&branchName={branch})", badge_md="[![Build Status](https://dev.azure.com/joerick0429/cibuildwheel/_apis/build/status/joerick.cibuildwheel?branchName={branch})](https://dev.azure.com/joerick0429/cibuildwheel/_build/latest?definitionId=2&branchName={branch})",
), ),
CIService( CIService(
name="circleci", name="circleci",
@@ -64,29 +59,24 @@ services = [
CIService( CIService(
name="travis-ci", name="travis-ci",
dst_config_path=".travis.yml", dst_config_path=".travis.yml",
badge_md="[![Build Status](https://app.travis-ci.com/pypa/cibuildwheel.svg?branch={branch})](https://app.travis-ci.com/pypa/cibuildwheel)", badge_md="[![Build Status](https://travis-ci.org/pypa/cibuildwheel.svg?branch={branch})](https://travis-ci.org/pypa/cibuildwheel)",
), ),
CIService( CIService(
name="gitlab", name="gitlab",
dst_config_path=".gitlab-ci.yml", dst_config_path=".gitlab-ci.yml",
badge_md="[![Gitlab](https://gitlab.com/joerick/cibuildwheel/badges/{branch}/pipeline.svg)](https://gitlab.com/joerick/cibuildwheel/-/commits/{branch})", badge_md="[![Gitlab](https://gitlab.com/pypa/cibuildwheel/badges/{branch}/pipeline.svg)](https://gitlab.com/pypa/cibuildwheel/-/commits/{branch})",
),
CIService(
name="cirrus-ci",
dst_config_path=".cirrus.yml",
badge_md="[![Cirrus CI](https://api.cirrus-ci.com/github/pypa/cibuildwheel.svg?branch={branch})](https://cirrus-ci.com/github/pypa/cibuildwheel/{branch})",
), ),
] ]
def ci_service_for_config_file(config_file): def ci_service_for_config_file(config_file):
filename = Path(config_file).name service_name = Path(config_file).name.rsplit("-", 1)[0]
try: for service in services:
return next(s for s in services if filename.startswith(s.name)) if service.name == service_name:
except StopIteration: return service
msg = f"unknown ci service for config file {config_file}"
raise ValueError(msg) from None raise ValueError(f"unknown ci service for config file {config_file}")
@click.command() @click.command()
@@ -105,8 +95,7 @@ def run_example_ci_configs(config_files=None):
for config_file in config_files: for config_file in config_files:
service = ci_service_for_config_file(config_file) service = ci_service_for_config_file(config_file)
if service.name in configs_by_service: if service.name in configs_by_service:
msg = "You cannot specify more than one config per CI service" raise Exception("You cannot specify more than one config per CI service")
raise Exception(msg)
configs_by_service[service.name] = config_file configs_by_service[service.name] = config_file
if git_repo_has_changes(): if git_repo_has_changes():
@@ -168,7 +157,7 @@ def run_example_ci_configs(config_files=None):
finally: finally:
# remove any local changes # remove any local changes
shutil.rmtree(example_project, ignore_errors=True) shutil.rmtree(example_project, ignore_errors=True)
shell("git checkout -- .", check=True) shell("git checkout -- .")
shell(f"git checkout {previous_branch}", check=True) shell(f"git checkout {previous_branch}", check=True)
shell(f"git branch -D --force {branch_name}", check=True) shell(f"git branch -D --force {branch_name}", check=True)
+7 -40
View File
@@ -1,56 +1,23 @@
#!/usr/bin/env python3 #!/usr/bin/env python3
from __future__ import annotations
import argparse
import os import os
import subprocess import subprocess
import sys import sys
from pathlib import Path from pathlib import Path
if __name__ == "__main__": if __name__ == "__main__":
default_cpu_count = os.cpu_count() or 2
parser = argparse.ArgumentParser()
parser.add_argument(
"--run-podman", action="store_true", default=False, help="run podman tests (linux only)"
)
parser.add_argument(
"--num-processes",
action="store",
default=default_cpu_count,
help="number of processes to use for testing",
)
args = parser.parse_args()
# move cwd to the project root # move cwd to the project root
os.chdir(Path(__file__).resolve().parents[1]) os.chdir(Path(__file__).resolve().parents[1])
# unit tests # run the unit tests
unit_test_args = [sys.executable, "-m", "pytest", "unit_test"] unit_test_args = [sys.executable, "-m", "pytest", "unit_test"]
if sys.platform.startswith("linux"):
# run the docker unit tests only on Linux # run the docker unit tests only on Linux
if sys.platform.startswith("linux"):
unit_test_args += ["--run-docker"] unit_test_args += ["--run-docker"]
if args.run_podman:
unit_test_args += ["--run-podman"]
subprocess.run(unit_test_args, check=True) subprocess.run(unit_test_args, check=True)
# integration tests # run the integration tests
integration_test_args = [ subprocess.run(
sys.executable, [sys.executable, "-m", "pytest", "-x", "--durations", "0", "--timeout=2400", "test"],
"-m", check=True,
"pytest", )
f"--numprocesses={args.num_processes}",
"-x",
"--durations",
"0",
"--timeout=2400",
"test",
]
if sys.platform.startswith("linux") and args.run_podman:
integration_test_args += ["--run-podman"]
subprocess.run(integration_test_args, check=True)
+1 -5
View File
@@ -24,8 +24,4 @@ if __name__ == "__main__":
check=True, check=True,
) )
sys.exit( sys.exit(subprocess.run([sys.executable, "-m", "cibuildwheel"], cwd=project_dir).returncode)
subprocess.run(
[sys.executable, "-m", "cibuildwheel"], cwd=project_dir, check=False
).returncode
)
@@ -2,69 +2,97 @@
from __future__ import annotations from __future__ import annotations
import configparser import configparser
from dataclasses import dataclass import os
from pathlib import Path import shutil
import subprocess
import sys
from typing import NamedTuple
import requests import requests
DIR = Path(__file__).parent.resolve() os.chdir(os.path.dirname(__file__))
RESOURCES = DIR.parent / "cibuildwheel/resources" os.chdir("..")
# CUSTOM_COMPILE_COMMAND is a pip-compile option that tells users how to
# regenerate the constraints files
os.environ["CUSTOM_COMPILE_COMMAND"] = "bin/update_dependencies.py"
PYTHON_VERSIONS = ["27", "35", "36", "37", "38", "39"]
if "--no-docker" in sys.argv:
for python_version in PYTHON_VERSIONS:
subprocess.run(
[
f"./env{python_version}/bin/pip-compile",
"--allow-unsafe",
"--upgrade",
"cibuildwheel/resources/constraints.in",
f"--output-file=cibuildwheel/resources/constraints-python{python_version}.txt",
],
check=True,
)
else:
# latest manylinux2010 image with cpython 2.7 support
image_runner = "quay.io/pypa/manylinux2010_x86_64:2021-02-06-3d322a5"
subprocess.run(["docker", "pull", image_runner], check=True)
for python_version in PYTHON_VERSIONS:
abi_flags = "" if int(python_version) >= 38 else "m"
python_path = f"/opt/python/cp{python_version}-cp{python_version}{abi_flags}/bin/"
command = (
f"{python_path}pip install pip-tools && "
f"{python_path}pip-compile --allow-unsafe --upgrade "
"cibuildwheel/resources/constraints.in "
f"--output-file cibuildwheel/resources/constraints-python{python_version}.txt"
)
subprocess.run(
[
"docker",
"run",
"--rm",
"--env=CUSTOM_COMPILE_COMMAND",
f"--volume={os.getcwd()}:/volume",
"--workdir=/volume",
image_runner,
"bash",
"-c",
command,
],
check=True,
)
# default constraints.txt
shutil.copyfile(
f"cibuildwheel/resources/constraints-python{PYTHON_VERSIONS[-1]}.txt",
"cibuildwheel/resources/constraints.txt",
)
@dataclass(frozen=True) class Image(NamedTuple):
class Image:
manylinux_version: str manylinux_version: str
platform: str platform: str
image_name: str image_name: str
tag: str | None # Set this to pin the image tag: str | None
images = [ images = [
# manylinux1 images
Image("manylinux1", "x86_64", "quay.io/pypa/manylinux1_x86_64", None), Image("manylinux1", "x86_64", "quay.io/pypa/manylinux1_x86_64", None),
Image("manylinux1", "i686", "quay.io/pypa/manylinux1_i686", None), Image("manylinux1", "i686", "quay.io/pypa/manylinux1_i686", None),
# manylinux2010 images # Images for manylinux2010 are pinned to the latest tag supporting cp27
Image("manylinux2010", "x86_64", "quay.io/pypa/manylinux2010_x86_64", None), Image("manylinux2010", "x86_64", "quay.io/pypa/manylinux2010_x86_64", "2021-02-06-3d322a5"),
Image("manylinux2010", "i686", "quay.io/pypa/manylinux2010_i686", None), Image("manylinux2010", "i686", "quay.io/pypa/manylinux2010_i686", "2021-02-06-3d322a5"),
Image("manylinux2010", "pypy_x86_64", "quay.io/pypa/manylinux2010_x86_64", None), Image("manylinux2010", "pypy_x86_64", "pypywheels/manylinux2010-pypy_x86_64", None),
Image("manylinux2010", "pypy_i686", "quay.io/pypa/manylinux2010_i686", None), # 2014 images
# manylinux2014 images
Image("manylinux2014", "x86_64", "quay.io/pypa/manylinux2014_x86_64", None), Image("manylinux2014", "x86_64", "quay.io/pypa/manylinux2014_x86_64", None),
Image("manylinux2014", "i686", "quay.io/pypa/manylinux2014_i686", None), Image("manylinux2014", "i686", "quay.io/pypa/manylinux2014_i686", None),
Image("manylinux2014", "aarch64", "quay.io/pypa/manylinux2014_aarch64", None), Image("manylinux2014", "aarch64", "quay.io/pypa/manylinux2014_aarch64", None),
Image("manylinux2014", "ppc64le", "quay.io/pypa/manylinux2014_ppc64le", None), Image("manylinux2014", "ppc64le", "quay.io/pypa/manylinux2014_ppc64le", None),
Image("manylinux2014", "s390x", "quay.io/pypa/manylinux2014_s390x", None), Image("manylinux2014", "s390x", "quay.io/pypa/manylinux2014_s390x", None),
Image("manylinux2014", "pypy_x86_64", "quay.io/pypa/manylinux2014_x86_64", None), # 2_24 images
Image("manylinux2014", "pypy_i686", "quay.io/pypa/manylinux2014_i686", None),
Image("manylinux2014", "pypy_aarch64", "quay.io/pypa/manylinux2014_aarch64", None),
# manylinux_2_24 images
Image("manylinux_2_24", "x86_64", "quay.io/pypa/manylinux_2_24_x86_64", None), Image("manylinux_2_24", "x86_64", "quay.io/pypa/manylinux_2_24_x86_64", None),
Image("manylinux_2_24", "i686", "quay.io/pypa/manylinux_2_24_i686", None), Image("manylinux_2_24", "i686", "quay.io/pypa/manylinux_2_24_i686", None),
Image("manylinux_2_24", "aarch64", "quay.io/pypa/manylinux_2_24_aarch64", None), Image("manylinux_2_24", "aarch64", "quay.io/pypa/manylinux_2_24_aarch64", None),
Image("manylinux_2_24", "ppc64le", "quay.io/pypa/manylinux_2_24_ppc64le", None), Image("manylinux_2_24", "ppc64le", "quay.io/pypa/manylinux_2_24_ppc64le", None),
Image("manylinux_2_24", "s390x", "quay.io/pypa/manylinux_2_24_s390x", None), Image("manylinux_2_24", "s390x", "quay.io/pypa/manylinux_2_24_s390x", None),
Image("manylinux_2_24", "pypy_x86_64", "quay.io/pypa/manylinux_2_24_x86_64", None),
Image("manylinux_2_24", "pypy_i686", "quay.io/pypa/manylinux_2_24_i686", None),
Image("manylinux_2_24", "pypy_aarch64", "quay.io/pypa/manylinux_2_24_aarch64", None),
# manylinux_2_28 images
Image("manylinux_2_28", "x86_64", "quay.io/pypa/manylinux_2_28_x86_64", None),
Image("manylinux_2_28", "aarch64", "quay.io/pypa/manylinux_2_28_aarch64", None),
Image("manylinux_2_28", "ppc64le", "quay.io/pypa/manylinux_2_28_ppc64le", None),
Image("manylinux_2_28", "s390x", "quay.io/pypa/manylinux_2_28_s390x", None),
Image("manylinux_2_28", "pypy_x86_64", "quay.io/pypa/manylinux_2_28_x86_64", None),
Image("manylinux_2_28", "pypy_aarch64", "quay.io/pypa/manylinux_2_28_aarch64", None),
# musllinux_1_1 images
Image("musllinux_1_1", "x86_64", "quay.io/pypa/musllinux_1_1_x86_64", None),
Image("musllinux_1_1", "i686", "quay.io/pypa/musllinux_1_1_i686", None),
Image("musllinux_1_1", "aarch64", "quay.io/pypa/musllinux_1_1_aarch64", None),
Image("musllinux_1_1", "ppc64le", "quay.io/pypa/musllinux_1_1_ppc64le", None),
Image("musllinux_1_1", "s390x", "quay.io/pypa/musllinux_1_1_s390x", None),
# musllinux_1_2 images
Image("musllinux_1_2", "x86_64", "quay.io/pypa/musllinux_1_2_x86_64", None),
Image("musllinux_1_2", "i686", "quay.io/pypa/musllinux_1_2_i686", None),
Image("musllinux_1_2", "aarch64", "quay.io/pypa/musllinux_1_2_aarch64", None),
Image("musllinux_1_2", "ppc64le", "quay.io/pypa/musllinux_1_2_ppc64le", None),
Image("musllinux_1_2", "s390x", "quay.io/pypa/musllinux_1_2_s390x", None),
] ]
config = configparser.ConfigParser() config = configparser.ConfigParser()
@@ -110,5 +138,5 @@ for image in images:
config[image.platform][image.manylinux_version] = f"{image.image_name}:{tag_name}" config[image.platform][image.manylinux_version] = f"{image.image_name}:{tag_name}"
with open(RESOURCES / "pinned_docker_images.cfg", "w") as f: with open("cibuildwheel/resources/pinned_docker_images.cfg", "w") as f:
config.write(f) config.write(f)
-52
View File
@@ -1,52 +0,0 @@
#!/usr/bin/env python3
from __future__ import annotations
import subprocess
import sys
from pathlib import Path
try:
from html2image import Html2Image # type: ignore[import-not-found]
except ImportError:
sys.exit(
"""
html2image not found. Ensure you have Chrome (on Mac/Windows) or
Chromium (on Linux) installed, and then do:
pip install html2image
"""
)
def main():
subprocess.run(["mkdocs", "build"], check=True)
hti = Html2Image(custom_flags=["--force-device-scale-factor=2"])
html_str = Path("docs/diagram.md").read_text()
css_tags = f"""
<style>{Path("site/css/theme.css").read_text()}</style>
<style>{Path("site/css/theme_extra.css").read_text()}</style>
<style>{Path("site/extra.css").read_text()}</style>
<style>
body {{
background: white;
}}
</style>
"""
html_str = css_tags + html_str
[screenshot, *_] = hti.screenshot(
html_str=html_str,
size=(830, 405),
)
dest_path = Path("docs/data/how-it-works.png")
if dest_path.exists():
dest_path.unlink()
Path(screenshot).rename(dest_path)
if __name__ == "__main__":
main()
-148
View File
@@ -1,148 +0,0 @@
#!/usr/bin/env python3
from __future__ import annotations
import difflib
import logging
from dataclasses import dataclass
from pathlib import Path
from typing import Final
import click
import packaging.specifiers
import requests
import rich
from packaging.version import InvalidVersion, Version
from rich.logging import RichHandler
from rich.syntax import Syntax
from cibuildwheel._compat import tomllib
log = logging.getLogger("cibw")
# Looking up the dir instead of using utils.resources_dir
# since we want to write to it.
DIR: Final[Path] = Path(__file__).parent.parent.resolve()
RESOURCES_DIR: Final[Path] = DIR / "cibuildwheel/resources"
NODEJS_DIST: Final[str] = "https://nodejs.org/dist/"
NODEJS_INDEX: Final[str] = f"{NODEJS_DIST}index.json"
@dataclass(frozen=True, order=True)
class VersionTuple:
version: Version
version_string: str
def parse_nodejs_index() -> list[VersionTuple]:
versions: list[VersionTuple] = []
response = requests.get(NODEJS_INDEX)
response.raise_for_status()
versions_info = response.json()
for version_info in versions_info:
version_string = version_info.get("version", "???")
if not version_info.get("lts", False):
log.debug("Ignoring non LTS release %r", version_string)
continue
if "linux-x64" not in version_info.get("files", []):
log.warning(
"Ignoring release %r which does not include a linux-x64 binary", version_string
)
continue
try:
version = Version(version_string)
if version.is_devrelease:
log.info("Ignoring development release %r", str(version))
continue
if version.is_prerelease:
log.info("Ignoring pre-release %r", str(version))
continue
versions.append(VersionTuple(version, version_string))
except InvalidVersion:
log.warning("Ignoring release %r", version_string)
versions.sort(reverse=True)
return versions
@click.command()
@click.option("--force", is_flag=True)
@click.option(
"--level", default="INFO", type=click.Choice(["WARNING", "INFO", "DEBUG"], case_sensitive=False)
)
def update_nodejs(force: bool, level: str) -> None:
logging.basicConfig(
level="INFO",
format="%(message)s",
datefmt="[%X]",
handlers=[RichHandler(rich_tracebacks=True, markup=True)],
)
log.setLevel(level)
toml_file_path = RESOURCES_DIR / "nodejs.toml"
original_toml = toml_file_path.read_text()
with toml_file_path.open("rb") as f:
nodejs_data = tomllib.load(f)
nodejs_data.pop("url")
major_versions = [VersionTuple(Version(key), key) for key in nodejs_data]
major_versions.sort(reverse=True)
versions = parse_nodejs_index()
# update existing versions, 1 per LTS
for major_version in major_versions:
current = Version(nodejs_data[major_version.version_string])
specifier = packaging.specifiers.SpecifierSet(
specifiers=f"=={major_version.version.major}.*"
)
for version in versions:
if specifier.contains(version.version) and version.version > current:
nodejs_data[major_version.version_string] = version.version_string
break
# check for a new major LTS to insert
if versions and versions[0].version.major > major_versions[0].version.major:
major_versions.insert(
0,
VersionTuple(Version(str(versions[0].version.major)), f"v{versions[0].version.major}"),
)
nodejs_data[major_versions[0].version_string] = versions[0].version_string
versions_toml = "\n".join(
f'{major_version.version_string} = "{nodejs_data[major_version.version_string]}"'
for major_version in major_versions
)
result_toml = f'url = "{NODEJS_DIST}"\n{versions_toml}\n'
rich.print() # spacer
if original_toml == result_toml:
rich.print("[green]Check complete, nodejs version unchanged.")
return
rich.print("nodejs version updated.")
rich.print("Changes:")
rich.print()
toml_relpath = toml_file_path.relative_to(DIR).as_posix()
diff_lines = difflib.unified_diff(
original_toml.splitlines(keepends=True),
result_toml.splitlines(keepends=True),
fromfile=toml_relpath,
tofile=toml_relpath,
)
rich.print(Syntax("".join(diff_lines), "diff", theme="ansi_light"))
rich.print()
if force:
toml_file_path.write_text(result_toml)
rich.print("[green]TOML file updated.")
else:
rich.print("[yellow]File left unchanged. Use --force flag to update.")
if __name__ == "__main__":
update_nodejs()
+70 -102
View File
@@ -5,20 +5,20 @@ from __future__ import annotations
import copy import copy
import difflib import difflib
import logging import logging
from collections.abc import Mapping, MutableMapping
from pathlib import Path from pathlib import Path
from typing import Any, Final, Literal, TypedDict, Union from typing import Union
import click import click
import requests import requests
import rich import rich
import toml
from packaging.specifiers import Specifier from packaging.specifiers import Specifier
from packaging.version import Version from packaging.version import Version
from rich.logging import RichHandler from rich.logging import RichHandler
from rich.syntax import Syntax from rich.syntax import Syntax
from cibuildwheel._compat import tomllib from cibuildwheel.extra import InlineArrayDictEncoder
from cibuildwheel.extra import dump_python_configurations from cibuildwheel.typing import Final, Literal, TypedDict
log = logging.getLogger("cibw") log = logging.getLogger("cibw")
@@ -28,7 +28,7 @@ DIR: Final[Path] = Path(__file__).parent.parent.resolve()
RESOURCES_DIR: Final[Path] = DIR / "cibuildwheel/resources" RESOURCES_DIR: Final[Path] = DIR / "cibuildwheel/resources"
ArchStr = Literal["32", "64", "ARM64"] ArchStr = Literal["32", "64"]
class ConfigWinCP(TypedDict): class ConfigWinCP(TypedDict):
@@ -58,7 +58,8 @@ AnyConfig = Union[ConfigWinCP, ConfigWinPP, ConfigMacOS]
class WindowsVersions: class WindowsVersions:
def __init__(self, arch_str: ArchStr, free_threaded: bool) -> None: def __init__(self, arch_str: ArchStr) -> None:
response = requests.get("https://api.nuget.org/v3/index.json") response = requests.get("https://api.nuget.org/v3/index.json")
response.raise_for_status() response.raise_for_status()
api_info = response.json() api_info = response.json()
@@ -67,47 +68,41 @@ class WindowsVersions:
if resource["@type"] == "PackageBaseAddress/3.0.0": if resource["@type"] == "PackageBaseAddress/3.0.0":
endpoint = resource["@id"] endpoint = resource["@id"]
ARCH_DICT = {"32": "win32", "64": "win_amd64", "ARM64": "win_arm64"} ARCH_DICT = {"32": "win32", "64": "win_amd64"}
PACKAGE_DICT = {"32": "pythonx86", "64": "python", "ARM64": "pythonarm64"} PACKAGE_DICT = {"32": "pythonx86", "64": "python"}
self.arch_str = arch_str self.arch_str = arch_str
self.arch = ARCH_DICT[arch_str] self.arch = ARCH_DICT[arch_str]
self.free_threaded = free_threaded
package = PACKAGE_DICT[arch_str] package = PACKAGE_DICT[arch_str]
if free_threaded:
package = f"{package}-freethreaded"
response = requests.get(f"{endpoint}{package}/index.json") response = requests.get(f"{endpoint}{package}/index.json")
response.raise_for_status() response.raise_for_status()
cp_info = response.json() cp_info = response.json()
self.version_dict = {Version(v): v for v in cp_info["versions"]} versions = (Version(v) for v in cp_info["versions"])
self.versions = sorted(v for v in versions if not v.is_devrelease)
def update_version_windows(self, spec: Specifier) -> ConfigWinCP | None: def update_version_windows(self, spec: Specifier) -> ConfigWinCP | None:
# Specifier.filter selects all non pre-releases that match the spec, versions = sorted(v for v in self.versions if spec.contains(v))
# unless there are only pre-releases, then it selects pre-releases if not all(v.is_prerelease for v in versions):
# instead (like pip) versions = [v for v in versions if not v.is_prerelease]
unsorted_versions = spec.filter(self.version_dict) log.debug(f"Windows {self.arch} {spec} has {', '.join(str(v) for v in versions)}")
versions = sorted(unsorted_versions, reverse=True)
log.debug("Windows %s %s has %s", self.arch, spec, ", ".join(str(v) for v in versions))
if not versions: if not versions:
return None return None
flags = "t" if self.free_threaded else "" version = versions[-1]
version = versions[0] identifier = f"cp{version.major}{version.minor}-{self.arch}"
identifier = f"cp{version.major}{version.minor}{flags}-{self.arch}"
return ConfigWinCP( return ConfigWinCP(
identifier=identifier, identifier=identifier,
version=self.version_dict[version], version=str(version),
arch=self.arch_str, arch=self.arch_str,
) )
class PyPyVersions: class PyPyVersions:
def __init__(self, arch_str: ArchStr): def __init__(self, arch_str: ArchStr):
response = requests.get("https://downloads.python.org/pypy/versions.json") response = requests.get("https://downloads.python.org/pypy/versions.json")
response.raise_for_status() response.raise_for_status()
@@ -123,60 +118,48 @@ class PyPyVersions:
] ]
self.arch = arch_str self.arch = arch_str
def get_arch_file(self, release: Mapping[str, Any]) -> str:
urls: list[str] = [
rf["download_url"]
for rf in release["files"]
if "" in rf["platform"] == f"win{self.arch}"
]
return urls[0] if urls else ""
def update_version_windows(self, spec: Specifier) -> ConfigWinCP: def update_version_windows(self, spec: Specifier) -> ConfigWinCP:
if self.arch != "32":
raise RuntimeError("64 bit releases not supported yet on Windows")
releases = [r for r in self.releases if spec.contains(r["python_version"])] releases = [r for r in self.releases if spec.contains(r["python_version"])]
releases = sorted(releases, key=lambda r: r["pypy_version"]) releases = sorted(releases, key=lambda r: r["pypy_version"])
releases = [r for r in releases if self.get_arch_file(r)]
if not releases: if not releases:
msg = f"PyPy Win {self.arch} not found for {spec}! {self.releases}" raise RuntimeError(f"PyPy Win {self.arch} not found for {spec}! {self.releases}")
raise RuntimeError(msg)
version_arch = "win32" if self.arch == "32" else "win_amd64"
release = releases[-1] release = releases[-1]
version = release["python_version"] version = release["python_version"]
identifier = f"pp{version.major}{version.minor}-{version_arch}" identifier = f"pp{version.major}{version.minor}-win32"
url = self.get_arch_file(release)
(url,) = [rf["download_url"] for rf in release["files"] if "" in rf["platform"] == "win32"]
return ConfigWinPP( return ConfigWinPP(
identifier=identifier, identifier=identifier,
version=f"{version.major}.{version.minor}", version=f"{version.major}.{version.minor}",
arch=self.arch, arch="32",
url=url, url=url,
) )
def update_version_macos(self, spec: Specifier) -> ConfigMacOS: def update_version_macos(self, spec: Specifier) -> ConfigMacOS:
if self.arch not in {"64", "ARM64"}: if self.arch != "64":
msg = f"'{self.arch}' arch not supported yet on macOS" raise RuntimeError("Other archs not supported yet on macOS")
raise RuntimeError(msg)
releases = [r for r in self.releases if spec.contains(r["python_version"])] releases = [r for r in self.releases if spec.contains(r["python_version"])]
releases = sorted(releases, key=lambda r: r["pypy_version"]) releases = sorted(releases, key=lambda r: r["pypy_version"])
if not releases: if not releases:
msg = f"PyPy macOS {self.arch} not found for {spec}!" raise RuntimeError(f"PyPy macOS {self.arch} not found for {spec}!")
raise RuntimeError(msg)
release = releases[-1] release = releases[-1]
version = release["python_version"] version = release["python_version"]
arch = "x86_64" if self.arch == "64" else self.arch.lower() identifier = f"pp{version.major}{version.minor}-macosx_x86_64"
identifier = f"pp{version.major}{version.minor}-macosx_{arch}"
arch = "x64" if self.arch == "64" else self.arch.lower() (url,) = [
(url,) = (
rf["download_url"] rf["download_url"]
for rf in release["files"] for rf in release["files"]
if "" in rf["platform"] == "darwin" and rf["arch"] == arch if "" in rf["platform"] == "darwin" and rf["arch"] == "x64"
) ]
return ConfigMacOS( return ConfigMacOS(
identifier=identifier, identifier=identifier,
@@ -187,6 +170,7 @@ class PyPyVersions:
class CPythonVersions: class CPythonVersions:
def __init__(self) -> None: def __init__(self) -> None:
response = requests.get( response = requests.get(
"https://www.python.org/api/v2/downloads/release/?is_published=true" "https://www.python.org/api/v2/downloads/release/?is_published=true"
) )
@@ -199,33 +183,29 @@ class CPythonVersions:
# Removing the prefix, Python 3.9 would use: release["name"].removeprefix("Python ") # Removing the prefix, Python 3.9 would use: release["name"].removeprefix("Python ")
version = Version(release["name"][7:]) version = Version(release["name"][7:])
if not version.is_prerelease and not version.is_devrelease:
uri = int(release["resource_uri"].rstrip("/").split("/")[-1]) uri = int(release["resource_uri"].rstrip("/").split("/")[-1])
self.versions_dict[version] = uri self.versions_dict[version] = uri
def update_version_macos( def update_version_macos(self, identifier: str, spec: Specifier) -> ConfigMacOS | None:
self, identifier: str, version: Version, spec: Specifier file_idents = ("macos11.pkg", "macosx10.9.pkg", "macosx10.6.pkg")
) -> ConfigMacOS | None: sorted_versions = sorted(v for v in self.versions_dict if spec.contains(v))
# see note above on Specifier.filter
unsorted_versions = spec.filter(self.versions_dict)
sorted_versions = sorted(unsorted_versions, reverse=True)
macver = "x10.9" if version <= Version("3.8.9999") else "11" for version in reversed(sorted_versions):
file_ident = f"macos{macver}.pkg"
for new_version in sorted_versions:
# Find the first patch version that contains the requested file # Find the first patch version that contains the requested file
uri = self.versions_dict[new_version] uri = self.versions_dict[version]
response = requests.get( response = requests.get(
f"https://www.python.org/api/v2/downloads/release_file/?release={uri}" f"https://www.python.org/api/v2/downloads/release_file/?release={uri}"
) )
response.raise_for_status() response.raise_for_status()
file_info = response.json() file_info = response.json()
for file_ident in file_idents:
urls = [rf["url"] for rf in file_info if file_ident in rf["url"]] urls = [rf["url"] for rf in file_info if file_ident in rf["url"]]
if urls: if urls:
return ConfigMacOS( return ConfigMacOS(
identifier=identifier, identifier=identifier,
version=f"{new_version.major}.{new_version.minor}", version=f"{version.major}.{version.minor}",
url=urls[0], url=urls[0],
) )
@@ -238,64 +218,53 @@ class CPythonVersions:
class AllVersions: class AllVersions:
def __init__(self) -> None: def __init__(self) -> None:
self.windows_32 = WindowsVersions("32", False) self.windows_32 = WindowsVersions("32")
self.windows_t_32 = WindowsVersions("32", True) self.windows_64 = WindowsVersions("64")
self.windows_64 = WindowsVersions("64", False) self.windows_pypy = PyPyVersions("32")
self.windows_t_64 = WindowsVersions("64", True)
self.windows_arm64 = WindowsVersions("ARM64", False)
self.windows_t_arm64 = WindowsVersions("ARM64", True)
self.windows_pypy_64 = PyPyVersions("64")
self.macos_cpython = CPythonVersions() self.macos_cpython = CPythonVersions()
self.macos_pypy = PyPyVersions("64") self.macos_pypy = PyPyVersions("64")
self.macos_pypy_arm64 = PyPyVersions("ARM64")
def update_config(self, config: MutableMapping[str, str]) -> None: def update_config(self, config: dict[str, str]) -> None:
identifier = config["identifier"] identifier = config["identifier"]
version = Version(config["version"]) version = Version(config["version"])
spec = Specifier(f"=={version.major}.{version.minor}.*") spec = Specifier(f"=={version.major}.{version.minor}.*")
log.info("Reading in %r -> %s @ %s", str(identifier), spec, version) log.info(f"Reading in '{identifier}' -> {spec} @ {version}")
orig_config = copy.copy(config) orig_config = copy.copy(config)
config_update: AnyConfig | None = None config_update: AnyConfig | None
# We need to use ** in update due to MyPy (probably a bug) # We need to use ** in update due to MyPy (probably a bug)
if "macosx" in identifier: if "macos" in identifier:
if identifier.startswith("cp"): if identifier.startswith("pp"):
config_update = self.macos_cpython.update_version_macos(identifier, version, spec)
elif identifier.startswith("pp"):
if "macosx_x86_64" in identifier:
config_update = self.macos_pypy.update_version_macos(spec) config_update = self.macos_pypy.update_version_macos(spec)
elif "macosx_arm64" in identifier: else:
config_update = self.macos_pypy_arm64.update_version_macos(spec) config_update = self.macos_cpython.update_version_macos(identifier, spec)
elif "t-win32" in identifier and identifier.startswith("cp"):
config_update = self.windows_t_32.update_version_windows(spec)
elif "win32" in identifier and identifier.startswith("cp"):
config_update = self.windows_32.update_version_windows(spec)
elif "t-win_amd64" in identifier and identifier.startswith("cp"):
config_update = self.windows_t_64.update_version_windows(spec)
elif "win_amd64" in identifier:
if identifier.startswith("cp"):
config_update = self.windows_64.update_version_windows(spec)
elif identifier.startswith("pp"):
config_update = self.windows_pypy_64.update_version_windows(spec)
elif "t-win_arm64" in identifier and identifier.startswith("cp"):
config_update = self.windows_t_arm64.update_version_windows(spec)
elif "win_arm64" in identifier and identifier.startswith("cp"):
config_update = self.windows_arm64.update_version_windows(spec)
assert config_update is not None, f"{identifier} not found!" assert config_update is not None, f"MacOS {spec} not found!"
config.update(**config_update)
elif "win32" in identifier:
if identifier.startswith("pp"):
config.update(**self.windows_pypy.update_version_windows(spec))
else:
config_update = self.windows_32.update_version_windows(spec)
if config_update:
config.update(**config_update)
elif "win_amd64" in identifier:
config_update = self.windows_64.update_version_windows(spec)
if config_update:
config.update(**config_update) config.update(**config_update)
if config != orig_config: if config != orig_config:
log.info(" Updated %s to %s", orig_config, config) log.info(f" Updated {orig_config} to {config}")
@click.command() @click.command()
@click.option("--force", is_flag=True) @click.option("--force", is_flag=True)
@click.option( @click.option(
"--level", default="INFO", type=click.Choice(["WARNING", "INFO", "DEBUG"], case_sensitive=False) "--level", default="INFO", type=click.Choice(["INFO", "DEBUG", "TRACE"], case_sensitive=False)
) )
def update_pythons(force: bool, level: str) -> None: def update_pythons(force: bool, level: str) -> None:
logging.basicConfig( logging.basicConfig(
level="INFO", level="INFO",
format="%(message)s", format="%(message)s",
@@ -308,8 +277,7 @@ def update_pythons(force: bool, level: str) -> None:
toml_file_path = RESOURCES_DIR / "build-platforms.toml" toml_file_path = RESOURCES_DIR / "build-platforms.toml"
original_toml = toml_file_path.read_text() original_toml = toml_file_path.read_text()
with toml_file_path.open("rb") as f: configs = toml.loads(original_toml)
configs = tomllib.load(f)
for config in configs["windows"]["python_configurations"]: for config in configs["windows"]["python_configurations"]:
all_versions.update_config(config) all_versions.update_config(config)
@@ -317,7 +285,7 @@ def update_pythons(force: bool, level: str) -> None:
for config in configs["macos"]["python_configurations"]: for config in configs["macos"]["python_configurations"]:
all_versions.update_config(config) all_versions.update_config(config)
result_toml = dump_python_configurations(configs) result_toml = toml.dumps(configs, encoder=InlineArrayDictEncoder()) # type: ignore
rich.print() # spacer rich.print() # spacer
-49
View File
@@ -1,49 +0,0 @@
#!/usr/bin/env python3
from __future__ import annotations
import re
import sys
from pathlib import Path
PROJECT_ROOT = Path(__file__).parent / ".."
CHANGELOG_FILE = PROJECT_ROOT / "docs" / "changelog.md"
README_FILE = PROJECT_ROOT / "README.md"
# https://regexr.com/622ds
FIRST_5_CHANGELOG_ENTRIES_REGEX = re.compile(r"""(^###.*?(?=###)){5}""", re.DOTALL | re.MULTILINE)
# https://regexr.com/622e5
README_CHANGELOG_SECTION = re.compile(
r"""(?<=<!-- START bin\/update_readme_changelog.py -->\n).*(?=<!-- END bin\/update_readme_changelog.py -->)""",
re.DOTALL,
)
def main():
changelog_text = CHANGELOG_FILE.read_text()
readme_text = README_FILE.read_text()
mini_changelog_match = FIRST_5_CHANGELOG_ENTRIES_REGEX.search(changelog_text)
assert mini_changelog_match, "Failed to find the first few changelog entries"
mini_changelog = "\n".join(
[
"",
"<!-- this section was generated by bin/update_readme_changelog.py -- do not edit manually -->",
"",
mini_changelog_match.group(0).strip(),
"",
"",
]
)
if not re.search(README_CHANGELOG_SECTION, readme_text):
sys.exit("Changelog section not found in README")
readme_text = re.sub(README_CHANGELOG_SECTION, mini_changelog, readme_text)
README_FILE.write_text(readme_text)
if __name__ == "__main__":
main()
-125
View File
@@ -1,125 +0,0 @@
#!/usr/bin/env python3
from __future__ import annotations
import difflib
import logging
import subprocess
from dataclasses import dataclass
from pathlib import Path
from typing import Final
import click
import rich
from packaging.version import InvalidVersion, Version
from rich.logging import RichHandler
from rich.syntax import Syntax
from cibuildwheel._compat import tomllib
log = logging.getLogger("cibw")
# Looking up the dir instead of using utils.resources_dir
# since we want to write to it.
DIR: Final[Path] = Path(__file__).parent.parent.resolve()
RESOURCES_DIR: Final[Path] = DIR / "cibuildwheel/resources"
GET_VIRTUALENV_GITHUB: Final[str] = "https://github.com/pypa/get-virtualenv"
GET_VIRTUALENV_URL_TEMPLATE: Final[str] = (
f"{GET_VIRTUALENV_GITHUB}/blob/{{version}}/public/virtualenv.pyz?raw=true"
)
@dataclass(frozen=True, order=True)
class VersionTuple:
version: Version
version_string: str
def git_ls_remote_versions(url) -> list[VersionTuple]:
versions: list[VersionTuple] = []
tags = subprocess.run(
["git", "ls-remote", "--tags", url], check=True, text=True, capture_output=True
).stdout.splitlines()
for tag in tags:
_, ref = tag.split()
assert ref.startswith("refs/tags/")
version_string = ref[10:]
try:
version = Version(version_string)
if version.is_devrelease:
log.info("Ignoring development release %r", str(version))
continue
if version.is_prerelease:
log.info("Ignoring pre-release %r", str(version))
continue
versions.append(VersionTuple(version, version_string))
except InvalidVersion:
log.warning("Ignoring ref %r", ref)
versions.sort(reverse=True)
return versions
@click.command()
@click.option("--force", is_flag=True)
@click.option(
"--level", default="INFO", type=click.Choice(["WARNING", "INFO", "DEBUG"], case_sensitive=False)
)
def update_virtualenv(force: bool, level: str) -> None:
logging.basicConfig(
level="INFO",
format="%(message)s",
datefmt="[%X]",
handlers=[RichHandler(rich_tracebacks=True, markup=True)],
)
log.setLevel(level)
toml_file_path = RESOURCES_DIR / "virtualenv.toml"
original_toml = toml_file_path.read_text()
with toml_file_path.open("rb") as f:
configurations = tomllib.load(f)
default = configurations.pop("default")
version = str(default["version"])
versions = git_ls_remote_versions(GET_VIRTUALENV_GITHUB)
if versions[0].version > Version(version):
version = versions[0].version_string
configurations["default"] = {
"version": version,
"url": GET_VIRTUALENV_URL_TEMPLATE.format(version=version),
}
result_toml = "".join(
f'{key} = {{ version = "{value["version"]}", url = "{value["url"]}" }}\n'
for key, value in configurations.items()
)
rich.print() # spacer
if original_toml == result_toml:
rich.print("[green]Check complete, virtualenv version unchanged.")
return
rich.print("virtualenv version updated.")
rich.print("Changes:")
rich.print()
toml_relpath = toml_file_path.relative_to(DIR).as_posix()
diff_lines = difflib.unified_diff(
original_toml.splitlines(keepends=True),
result_toml.splitlines(keepends=True),
fromfile=toml_relpath,
tofile=toml_relpath,
)
rich.print(Syntax("".join(diff_lines), "diff", theme="ansi_light"))
rich.print()
if force:
toml_file_path.write_text(result_toml)
rich.print("[green]TOML file updated.")
else:
rich.print("[yellow]File left unchanged. Use --force flag to update.")
if __name__ == "__main__":
update_virtualenv()
+1 -3
View File
@@ -1,3 +1 @@
from __future__ import annotations __version__ = "1.12.0"
__version__ = "2.20.0"
+308 -270
View File
@@ -1,89 +1,90 @@
from __future__ import annotations
import argparse import argparse
import dataclasses
import os import os
import shutil
import sys import sys
import tarfile
import textwrap import textwrap
import traceback import traceback
import typing from configparser import ConfigParser
from collections.abc import Iterable, Sequence, Set
from pathlib import Path from pathlib import Path
from tempfile import mkdtemp from typing import Dict, List, Optional, Set, Union, overload
from typing import Protocol
from packaging.specifiers import SpecifierSet
import cibuildwheel import cibuildwheel
import cibuildwheel.linux import cibuildwheel.linux
import cibuildwheel.macos import cibuildwheel.macos
import cibuildwheel.pyodide
import cibuildwheel.util import cibuildwheel.util
import cibuildwheel.windows import cibuildwheel.windows
from cibuildwheel import errors
from cibuildwheel._compat.typing import assert_never
from cibuildwheel.architecture import Architecture, allowed_architectures_check from cibuildwheel.architecture import Architecture, allowed_architectures_check
from cibuildwheel.logger import log from cibuildwheel.environment import EnvironmentParseError, parse_environment
from cibuildwheel.options import CommandLineArguments, Options, compute_options from cibuildwheel.projectfiles import get_requires_python_str
from cibuildwheel.typing import PLATFORMS, GenericPythonConfiguration, PlatformName from cibuildwheel.typing import PLATFORMS, PlatformName, assert_never
from cibuildwheel.util import ( from cibuildwheel.util import (
CIBW_CACHE_PATH, BuildOptions,
BuildSelector, BuildSelector,
CIProvider, DependencyConstraints,
TestSelector,
Unbuffered, Unbuffered,
chdir,
detect_ci_provider, detect_ci_provider,
fix_ansi_codes_for_github_actions, resources_dir,
strtobool,
) )
@dataclasses.dataclass @overload
class GlobalOptions: def get_option_from_environment(
print_traceback_on_error: bool = True # decides what happens when errors are hit. option_name: str, *, platform: Optional[str] = None, default: str
) -> str:
... # noqa: E704
@overload
def get_option_from_environment(
option_name: str, *, platform: Optional[str] = None, default: None = None
) -> Optional[str]:
... # noqa: E704 E302
def get_option_from_environment(
option_name: str, *, platform: Optional[str] = None, default: Optional[str] = None
) -> Optional[str]: # noqa: E302
"""
Returns an option from the environment, optionally scoped by the platform.
Example:
get_option_from_environment('CIBW_COLOR', platform='macos')
This will return the value of CIBW_COLOR_MACOS if it exists, otherwise the value of
CIBW_COLOR.
"""
if platform:
option = os.environ.get(f"{option_name}_{platform.upper()}")
if option is not None:
return option
return os.environ.get(option_name, default)
def main() -> None: def main() -> None:
global_options = GlobalOptions() platform: PlatformName
try:
main_inner(global_options)
except errors.FatalError as e:
message = e.args[0]
if log.step_active:
log.step_end_with_error(message)
else:
print(f"cibuildwheel: {message}", file=sys.stderr)
if global_options.print_traceback_on_error:
traceback.print_exc(file=sys.stderr)
sys.exit(e.return_code)
def main_inner(global_options: GlobalOptions) -> None:
"""
`main_inner` is the same as `main`, but it raises FatalError exceptions
rather than exiting directly.
"""
parser = argparse.ArgumentParser( parser = argparse.ArgumentParser(
description="Build wheels for all the platforms.", description="Build wheels for all the platforms.",
epilog=""" epilog="""
Most options are supplied via environment variables or in Most options are supplied via environment variables.
--config-file (pyproject.toml usually). See See https://github.com/pypa/cibuildwheel#options for info.
https://github.com/pypa/cibuildwheel#options for info.
""", """,
) )
parser.add_argument( parser.add_argument(
"--platform", "--platform",
choices=["auto", "linux", "macos", "windows", "pyodide"], choices=["auto", "linux", "macos", "windows"],
default=None, default=os.environ.get("CIBW_PLATFORM", "auto"),
help=""" help="""
Platform to build for. Use this option to override the Platform to build for. For "linux" you need docker running, on Mac
auto-detected platform. Specifying "macos" or "windows" only works or Linux. For "macos", you need a Mac machine, and note that this
on that operating system, but "linux" works on all three, as long script is going to automatically install MacPython on your system,
as Docker/Podman is installed. Default: auto. so don't run on your development machine. For "windows", you need to
run in Windows, and it will build and test for all versions of
Python. Default: auto.
""", """,
) )
@@ -101,47 +102,21 @@ def main_inner(global_options: GlobalOptions) -> None:
""", """,
) )
parser.add_argument(
"--only",
default=None,
help="""
Force a single wheel build when given an identifier. Overrides
CIBW_BUILD/CIBW_SKIP. --platform and --arch cannot be specified
if this is given.
""",
)
parser.add_argument( parser.add_argument(
"--output-dir", "--output-dir",
type=Path, default=os.environ.get("CIBW_OUTPUT_DIR", "wheelhouse"),
default=Path(os.environ.get("CIBW_OUTPUT_DIR", "wheelhouse")), help="Destination folder for the wheels.",
help="Destination folder for the wheels. Default: wheelhouse.",
)
parser.add_argument(
"--config-file",
default="",
help="""
TOML config file. Default: "", meaning {package}/pyproject.toml, if
it exists. To refer to a project inside your project, use {package};
this matters if you build from an SDist.
""",
) )
parser.add_argument( parser.add_argument(
"package_dir", "package_dir",
metavar="PACKAGE", default=".",
default=Path("."),
type=Path,
nargs="?", nargs="?",
help=""" help="""
Path to the package that you want wheels for. Default: the working Path to the package that you want wheels for. Must be a subdirectory of
directory. Can be a directory inside the working directory, or an the working directory. When set, the working directory is still
sdist. When set to a directory, the working directory is still considered the 'project' and is copied into the Docker container on
considered the 'project' and is copied into the build container Linux. Default: the working directory.
on Linux. When set to a tar.gz sdist file, --config-file
and --output-dir are relative to the current directory, and other
paths are relative to the expanded SDist directory.
""", """,
) )
@@ -157,208 +132,258 @@ def main_inner(global_options: GlobalOptions) -> None:
help="Do not report an error code if the build does not match any wheels.", help="Do not report an error code if the build does not match any wheels.",
) )
parser.add_argument( args = parser.parse_args()
"--prerelease-pythons",
action="store_true",
help="Enable pre-release Python versions if available.",
)
parser.add_argument( detect_obsolete_options()
"--debug-traceback",
action="store_true",
default=strtobool(os.environ.get("CIBW_DEBUG_TRACEBACK", "0")),
help="Print a full traceback for all errors",
)
args = CommandLineArguments(**vars(parser.parse_args())) if args.platform != "auto":
platform = args.platform
global_options.print_traceback_on_error = args.debug_traceback
args.package_dir = args.package_dir.resolve()
# This are always relative to the base directory, even in SDist builds
args.output_dir = args.output_dir.resolve()
# Standard builds if a directory or non-existent path is given
if not args.package_dir.is_file() and not args.package_dir.name.endswith("tar.gz"):
build_in_directory(args)
return
# Tarfile builds require extraction and changing the directory
temp_dir = Path(mkdtemp(prefix="cibw-sdist-")).resolve(strict=True)
try:
with tarfile.open(args.package_dir) as tar:
tar.extractall(path=temp_dir)
# The extract directory is now the project dir
try:
(project_dir,) = temp_dir.iterdir()
except ValueError:
msg = "invalid sdist: didn't contain a single dir"
raise SystemExit(msg) from None
# This is now the new package dir
args.package_dir = project_dir.resolve()
with chdir(project_dir):
build_in_directory(args)
finally:
# avoid https://github.com/python/cpython/issues/86962 by performing
# cleanup manually
shutil.rmtree(temp_dir, ignore_errors=sys.platform.startswith("win"))
if temp_dir.exists():
log.warning(f"Can't delete temporary folder '{temp_dir}'")
def _compute_platform_only(only: str) -> PlatformName:
if "linux_" in only:
return "linux"
if "macosx_" in only:
return "macos"
if "win_" in only or "win32" in only:
return "windows"
if "pyodide_" in only:
return "pyodide"
msg = f"Invalid --only='{only}', must be a build selector with a known platform"
raise errors.ConfigurationError(msg)
def _compute_platform_auto() -> PlatformName:
if sys.platform.startswith("linux"):
return "linux"
elif sys.platform == "darwin":
return "macos"
elif sys.platform == "win32":
return "windows"
else: else:
msg = ( ci_provider = detect_ci_provider()
'cibuildwheel: Unable to detect platform from "sys.platform". cibuildwheel doesn\'t ' if ci_provider is None:
"support building wheels for this platform. You might be able to build for a different " print(
"platform using the --platform argument. Check --help output for more information." textwrap.dedent(
"""
cibuildwheel: Unable to detect platform. cibuildwheel should run on your CI server;
Travis CI, AppVeyor, Azure Pipelines, GitHub Actions, CircleCI, and Gitlab are
supported. You can run on your development machine or other CI providers using the
--platform argument. Check --help output for more information.
"""
),
file=sys.stderr,
)
sys.exit(2)
if sys.platform.startswith("linux"):
platform = "linux"
elif sys.platform == "darwin":
platform = "macos"
elif sys.platform == "win32":
platform = "windows"
else:
print(
'cibuildwheel: Unable to detect platform from "sys.platform" in a CI environment. You can run '
"cibuildwheel using the --platform argument. Check --help output for more information.",
file=sys.stderr,
) )
raise errors.ConfigurationError(msg)
def _compute_platform(args: CommandLineArguments) -> PlatformName:
platform_option_value = args.platform or os.environ.get("CIBW_PLATFORM", "") or "auto"
if args.only and args.platform is not None:
msg = "--platform cannot be specified with --only, it is computed from --only"
raise errors.ConfigurationError(msg)
if args.only and args.archs is not None:
msg = "--arch cannot be specified with --only, it is computed from --only"
raise errors.ConfigurationError(msg)
if platform_option_value not in PLATFORMS | {"auto"}:
msg = f"Unsupported platform: {platform_option_value}"
raise errors.ConfigurationError(msg)
if args.only:
return _compute_platform_only(args.only)
elif platform_option_value != "auto":
return typing.cast(PlatformName, platform_option_value)
return _compute_platform_auto()
class PlatformModule(Protocol):
# note that as per PEP544, the self argument is ignored when the protocol
# is applied to a module
def get_python_configurations(
self, build_selector: BuildSelector, architectures: Set[Architecture]
) -> Sequence[GenericPythonConfiguration]: ...
def build(self, options: Options, tmp_path: Path) -> None: ...
# pylint: disable-next=inconsistent-return-statements
def get_platform_module(platform: PlatformName) -> PlatformModule:
if platform == "linux":
return cibuildwheel.linux
if platform == "windows":
return cibuildwheel.windows
if platform == "macos":
return cibuildwheel.macos
if platform == "pyodide":
return cibuildwheel.pyodide
assert_never(platform)
def build_in_directory(args: CommandLineArguments) -> None:
platform: PlatformName = _compute_platform(args)
if platform == "pyodide" and sys.platform == "win32":
msg = "cibuildwheel: Building for pyodide is not supported on Windows"
print(msg, file=sys.stderr)
sys.exit(2) sys.exit(2)
options = compute_options(platform=platform, command_line_arguments=args, env=os.environ) if platform not in PLATFORMS:
print(f"cibuildwheel: Unsupported platform: {platform}", file=sys.stderr)
sys.exit(2)
package_dir = Path(args.package_dir)
output_dir = Path(args.output_dir)
if platform == "linux":
repair_command_default = "auditwheel repair -w {dest_dir} {wheel}"
elif platform == "macos":
repair_command_default = "delocate-listdeps {wheel} && delocate-wheel --require-archs {delocate_archs} -w {dest_dir} {wheel}"
elif platform == "windows":
repair_command_default = ""
else:
assert_never(platform)
build_config = os.environ.get("CIBW_BUILD") or "*"
skip_config = os.environ.get("CIBW_SKIP", "")
test_skip = os.environ.get("CIBW_TEST_SKIP", "")
environment_config = get_option_from_environment(
"CIBW_ENVIRONMENT", platform=platform, default=""
)
before_all = get_option_from_environment("CIBW_BEFORE_ALL", platform=platform, default="")
before_build = get_option_from_environment("CIBW_BEFORE_BUILD", platform=platform)
repair_command = get_option_from_environment(
"CIBW_REPAIR_WHEEL_COMMAND", platform=platform, default=repair_command_default
)
dependency_versions = get_option_from_environment(
"CIBW_DEPENDENCY_VERSIONS", platform=platform, default="pinned"
)
test_command = get_option_from_environment("CIBW_TEST_COMMAND", platform=platform)
before_test = get_option_from_environment("CIBW_BEFORE_TEST", platform=platform)
test_requires = get_option_from_environment(
"CIBW_TEST_REQUIRES", platform=platform, default=""
).split()
test_extras = get_option_from_environment("CIBW_TEST_EXTRAS", platform=platform, default="")
build_verbosity_str = get_option_from_environment(
"CIBW_BUILD_VERBOSITY", platform=platform, default=""
)
package_dir = options.globals.package_dir
package_files = {"setup.py", "setup.cfg", "pyproject.toml"} package_files = {"setup.py", "setup.cfg", "pyproject.toml"}
if not any(package_dir.joinpath(name).exists() for name in package_files): if not any(package_dir.joinpath(name).exists() for name in package_files):
names = ", ".join(sorted(package_files, reverse=True)) names = ", ".join(sorted(package_files, reverse=True))
msg = f"Could not find any of {{{names}}} at root of package" print(
raise errors.ConfigurationError(msg) f"cibuildwheel: Could not find any of {{{names}}} at root of package", file=sys.stderr
platform_module = get_platform_module(platform)
identifiers = get_build_identifiers(
platform_module=platform_module,
build_selector=options.globals.build_selector,
architectures=options.globals.architectures,
) )
sys.exit(2)
# Passing this in as an environment variable will override pyproject.toml, setup.cfg, or setup.py
requires_python_str: Optional[str] = os.environ.get(
"CIBW_PROJECT_REQUIRES_PYTHON"
) or get_requires_python_str(package_dir)
requires_python = None if requires_python_str is None else SpecifierSet(requires_python_str)
build_selector = BuildSelector(
build_config=build_config, skip_config=skip_config, requires_python=requires_python
)
test_selector = TestSelector(skip_config=test_skip)
try:
environment = parse_environment(environment_config)
except (EnvironmentParseError, ValueError):
print(f'cibuildwheel: Malformed environment option "{environment_config}"', file=sys.stderr)
traceback.print_exc(None, sys.stderr)
sys.exit(2)
if dependency_versions == "pinned":
dependency_constraints: Optional[
DependencyConstraints
] = DependencyConstraints.with_defaults()
elif dependency_versions == "latest":
dependency_constraints = None
else:
dependency_versions_path = Path(dependency_versions)
dependency_constraints = DependencyConstraints(dependency_versions_path)
if test_extras:
test_extras = f"[{test_extras}]"
try:
build_verbosity = min(3, max(-3, int(build_verbosity_str)))
except ValueError:
build_verbosity = 0
# Add CIBUILDWHEEL environment variable
# This needs to be passed on to the docker container in linux.py
os.environ["CIBUILDWHEEL"] = "1"
if args.archs is not None:
archs_config_str = args.archs
else:
archs_config_str = get_option_from_environment(
"CIBW_ARCHS", platform=platform, default="auto"
)
archs = Architecture.parse_config(archs_config_str, platform=platform)
identifiers = get_build_identifiers(platform, build_selector, archs)
if args.print_build_identifiers: if args.print_build_identifiers:
for identifier in identifiers: for identifier in identifiers:
print(identifier) print(identifier)
sys.exit(0) sys.exit(0)
# Add CIBUILDWHEEL environment variable manylinux_images: Optional[Dict[str, str]] = None
os.environ["CIBUILDWHEEL"] = "1" if platform == "linux":
pinned_docker_images_file = resources_dir / "pinned_docker_images.cfg"
all_pinned_docker_images = ConfigParser()
all_pinned_docker_images.read(pinned_docker_images_file)
# all_pinned_docker_images looks like a dict of dicts, e.g.
# { 'x86_64': {'manylinux1': '...', 'manylinux2010': '...', 'manylinux2014': '...'},
# 'i686': {'manylinux1': '...', 'manylinux2010': '...', 'manylinux2014': '...'},
# 'pypy_x86_64': {'manylinux2010': '...' }
# ... }
# Python is buffering by default when running on the CI platforms, giving manylinux_images = {}
# problems interleaving subprocess call output with unflushed calls to
# 'print'
sys.stdout = Unbuffered(sys.stdout)
# create the cache dir before it gets printed & builds performed for build_platform in ["x86_64", "i686", "pypy_x86_64", "aarch64", "ppc64le", "s390x"]:
CIBW_CACHE_PATH.mkdir(parents=True, exist_ok=True) pinned_images = all_pinned_docker_images[build_platform]
print_preamble(platform=platform, options=options, identifiers=identifiers) config_name = f"CIBW_MANYLINUX_{build_platform.upper()}_IMAGE"
config_value = os.environ.get(config_name)
if config_value is None:
# default to manylinux2010 if it's available, otherwise manylinux2014
image = pinned_images.get("manylinux2010") or pinned_images.get("manylinux2014")
elif config_value in pinned_images:
image = pinned_images[config_value]
else:
image = config_value
manylinux_images[build_platform] = image
build_options = BuildOptions(
architectures=archs,
package_dir=package_dir,
output_dir=output_dir,
test_command=test_command,
test_requires=test_requires,
test_extras=test_extras,
before_test=before_test,
before_build=before_build,
before_all=before_all,
build_verbosity=build_verbosity,
build_selector=build_selector,
test_selector=test_selector,
repair_command=repair_command,
environment=environment,
dependency_constraints=dependency_constraints,
manylinux_images=manylinux_images,
)
# Python is buffering by default when running on the CI platforms, giving problems interleaving subprocess call output with unflushed calls to 'print'
sys.stdout = Unbuffered(sys.stdout) # type: ignore
print_preamble(platform, build_options)
try: try:
options.check_for_invalid_configuration(identifiers) allowed_architectures_check(platform, build_options.architectures)
allowed_architectures_check(platform, options.globals.architectures)
except ValueError as err: except ValueError as err:
raise errors.DeprecationError(*err.args) from err print("cibuildwheel:", *err.args, file=sys.stderr)
sys.exit(4)
if not identifiers: if not identifiers:
message = f"No build identifiers selected: {options.globals.build_selector}" print(f"cibuildwheel: No build identifiers selected: {build_selector}", file=sys.stderr)
if options.globals.allow_empty: if not args.allow_empty:
print(f"cibuildwheel: {message}", file=sys.stderr) sys.exit(3)
else:
raise errors.NothingToDoError(message)
output_dir = options.globals.output_dir
if not output_dir.exists(): if not output_dir.exists():
output_dir.mkdir(parents=True) output_dir.mkdir(parents=True)
tmp_path = Path(mkdtemp(prefix="cibw-run-")).resolve(strict=True)
try:
with cibuildwheel.util.print_new_wheels( with cibuildwheel.util.print_new_wheels(
"\n{n} wheels produced in {m:.0f} minutes:", output_dir "\n{n} wheels produced in {m:.0f} minutes:", output_dir
): ):
platform_module.build(options, tmp_path) if platform == "linux":
finally: cibuildwheel.linux.build(build_options)
# avoid https://github.com/python/cpython/issues/86962 by performing elif platform == "windows":
# cleanup manually cibuildwheel.windows.build(build_options)
shutil.rmtree(tmp_path, ignore_errors=sys.platform.startswith("win")) elif platform == "macos":
if tmp_path.exists(): cibuildwheel.macos.build(build_options)
log.warning(f"Can't delete temporary folder '{tmp_path}'") else:
assert_never(platform)
def print_preamble(platform: str, options: Options, identifiers: Sequence[str]) -> None: def detect_obsolete_options() -> None:
# Check the old 'MANYLINUX1_*_IMAGE' options
for (deprecated, alternative) in [
("CIBW_MANYLINUX1_X86_64_IMAGE", "CIBW_MANYLINUX_X86_64_IMAGE"),
("CIBW_MANYLINUX1_I686_IMAGE", "CIBW_MANYLINUX_I686_IMAGE"),
]:
if deprecated in os.environ:
print(
f"'{deprecated}' has been deprecated, and will be removed in a future release. Use the option '{alternative}' instead."
)
if alternative not in os.environ:
print(f"Using value of option '{deprecated}' as replacement for '{alternative}'")
os.environ[alternative] = os.environ[deprecated]
else:
print(f"Option '{alternative}' is not empty. Please unset '{deprecated}'")
sys.exit(2)
# Check for deprecated identifiers in 'CIBW_BUILD' and 'CIBW_SKIP' options
for option in ["CIBW_BUILD", "CIBW_SKIP"]:
for deprecated, alternative in [
("manylinux1", "manylinux"),
("macosx_10_6_intel", "macosx_x86_64"),
("macosx_10_9_x86_64", "macosx_x86_64"),
]:
if option in os.environ and deprecated in os.environ[option]:
print(
f"Build identifiers with '{deprecated}' have been deprecated. Replacing all occurences of '{deprecated}' with '{alternative}' in the option '{option}'"
)
os.environ[option] = os.environ[option].replace(deprecated, alternative)
def print_preamble(platform: str, build_options: BuildOptions) -> None:
print( print(
textwrap.dedent( textwrap.dedent(
""" """
@@ -373,16 +398,11 @@ def print_preamble(platform: str, options: Options, identifiers: Sequence[str])
print(f"cibuildwheel version {cibuildwheel.__version__}\n") print(f"cibuildwheel version {cibuildwheel.__version__}\n")
print("Build options:") print("Build options:")
print(f" platform: {platform}") print(f" platform: {platform!r}")
options_summary = textwrap.indent(options.summary(identifiers), " ") for option, value in sorted(build_options._asdict().items()):
if detect_ci_provider() == CIProvider.github_actions: print(f" {option}: {value!r}")
options_summary = fix_ansi_codes_for_github_actions(options_summary)
print(options_summary)
print() warnings = detect_warnings(platform, build_options)
print(f"Cache folder: {CIBW_CACHE_PATH}")
warnings = detect_warnings(options=options, identifiers=identifiers)
if warnings: if warnings:
print("\nWarnings:") print("\nWarnings:")
for warning in warnings: for warning in warnings:
@@ -392,22 +412,40 @@ def print_preamble(platform: str, options: Options, identifiers: Sequence[str])
def get_build_identifiers( def get_build_identifiers(
platform_module: PlatformModule, platform: PlatformName, build_selector: BuildSelector, architectures: Set[Architecture]
build_selector: BuildSelector, ) -> List[str]:
architectures: Set[Architecture], python_configurations: Union[
) -> list[str]: List[cibuildwheel.linux.PythonConfiguration],
python_configurations = platform_module.get_python_configurations(build_selector, architectures) List[cibuildwheel.windows.PythonConfiguration],
List[cibuildwheel.macos.PythonConfiguration],
]
if platform == "linux":
python_configurations = cibuildwheel.linux.get_python_configurations(
build_selector, architectures
)
elif platform == "windows":
python_configurations = cibuildwheel.windows.get_python_configurations(
build_selector, architectures
)
elif platform == "macos":
python_configurations = cibuildwheel.macos.get_python_configurations(
build_selector, architectures
)
else:
assert_never(platform)
return [config.identifier for config in python_configurations] return [config.identifier for config in python_configurations]
def detect_warnings(*, options: Options, identifiers: Iterable[str]) -> list[str]: def detect_warnings(platform: str, build_options: BuildOptions) -> List[str]:
warnings = [] warnings = []
# warn about deprecated {python} and {pip} # warn about deprecated {python} and {pip}
for option_name in ["test_command", "before_build"]: for option_name in ["test_command", "before_build"]:
option_values = [getattr(options.build_options(i), option_name) for i in identifiers] option_value = getattr(build_options, option_name)
if any(o and ("{python}" in o or "{pip}" in o) for o in option_values): if option_value and ("{python}" in option_value or "{pip}" in option_value):
# Reminder: in an f-string, double braces means literal single brace # Reminder: in an f-string, double braces means literal single brace
msg = ( msg = (
f"{option_name}: '{{python}}' and '{{pip}}' are no longer needed, " f"{option_name}: '{{python}}' and '{{pip}}' are no longer needed, "
-1
View File
@@ -1 +0,0 @@
from __future__ import annotations
-10
View File
@@ -1,10 +0,0 @@
from __future__ import annotations
import sys
if sys.version_info >= (3, 11):
from tomllib import load, loads
else:
from tomli import load, loads
__all__ = ["load", "loads"]
-14
View File
@@ -1,14 +0,0 @@
from __future__ import annotations
import sys
if sys.version_info < (3, 11):
from typing_extensions import NotRequired, Self, assert_never
else:
from typing import NotRequired, Self, assert_never
__all__ = (
"assert_never",
"NotRequired",
"Self",
)
+26 -76
View File
@@ -1,28 +1,12 @@
from __future__ import annotations
import functools import functools
import platform as platform_module import platform as platform_module
import re import re
import sys
from collections.abc import Set
from enum import Enum from enum import Enum
from typing import Final, Literal from typing import Set
from ._compat.typing import assert_never from .typing import Literal, PlatformName, assert_never
from .typing import PlatformName
PRETTY_NAMES: Final[dict[PlatformName, str]] = { PRETTY_NAMES = {"linux": "Linux", "macos": "macOS", "windows": "Windows"}
"linux": "Linux",
"macos": "macOS",
"windows": "Windows",
"pyodide": "Pyodide",
}
ARCH_SYNONYMS: Final[list[dict[PlatformName, str | None]]] = [
{"linux": "x86_64", "macos": "x86_64", "windows": "AMD64"},
{"linux": "i686", "macos": None, "windows": "x86"},
{"linux": "aarch64", "macos": "arm64", "windows": "ARM64"},
]
@functools.total_ordering @functools.total_ordering
@@ -45,20 +29,13 @@ class Architecture(Enum):
# windows archs # windows archs
x86 = "x86" x86 = "x86"
AMD64 = "AMD64" AMD64 = "AMD64"
ARM64 = "ARM64"
# WebAssembly
wasm32 = "wasm32"
# Allow this to be sorted # Allow this to be sorted
def __lt__(self, other: Architecture) -> bool: def __lt__(self, other: "Architecture") -> bool:
return self.value < other.value return self.value < other.value
def __str__(self) -> str:
return self.name
@staticmethod @staticmethod
def parse_config(config: str, platform: PlatformName) -> set[Architecture]: def parse_config(config: str, platform: PlatformName) -> "Set[Architecture]":
result = set() result = set()
for arch_str in re.split(r"[\s,]+", config): for arch_str in re.split(r"[\s,]+", config):
if arch_str == "auto": if arch_str == "auto":
@@ -76,70 +53,42 @@ class Architecture(Enum):
return result return result
@staticmethod @staticmethod
def native_arch(platform: PlatformName) -> Architecture | None: def auto_archs(platform: PlatformName) -> "Set[Architecture]":
if platform == "pyodide": native_architecture = Architecture(platform_module.machine())
return Architecture.wasm32 result = {native_architecture}
# Cross-platform support. Used for --print-build-identifiers or docker builds. if platform == "linux" and native_architecture == Architecture.x86_64:
host_platform: PlatformName = ( # x86_64 machines can run i686 docker containers
"windows"
if sys.platform.startswith("win")
else ("macos" if sys.platform.startswith("darwin") else "linux")
)
native_machine = platform_module.machine()
native_architecture = Architecture(native_machine)
# we might need to rename the native arch to the machine we're running
# on, as the same arch can have different names on different platforms
if host_platform != platform:
for arch_synonym in ARCH_SYNONYMS:
if native_machine == arch_synonym.get(host_platform):
synonym = arch_synonym[platform]
if synonym is None:
# can't build anything on this platform
return None
native_architecture = Architecture(synonym)
return native_architecture
@staticmethod
def auto_archs(platform: PlatformName) -> set[Architecture]:
native_arch = Architecture.native_arch(platform)
if native_arch is None:
return set() # can't build anything on this platform
result = {native_arch}
if platform == "linux" and Architecture.x86_64 in result:
# x86_64 machines can run i686 containers
result.add(Architecture.i686) result.add(Architecture.i686)
if platform == "windows" and Architecture.AMD64 in result: if platform == "windows" and native_architecture == Architecture.AMD64:
result.add(Architecture.x86) result.add(Architecture.x86)
if platform == "macos" and native_architecture == Architecture.arm64:
# arm64 can build and test both archs of a universal2 wheel.
result.add(Architecture.universal2)
return result return result
@staticmethod @staticmethod
def all_archs(platform: PlatformName) -> set[Architecture]: def all_archs(platform: PlatformName) -> "Set[Architecture]":
all_archs_map = { if platform == "linux":
"linux": { return {
Architecture.x86_64, Architecture.x86_64,
Architecture.i686, Architecture.i686,
Architecture.aarch64, Architecture.aarch64,
Architecture.ppc64le, Architecture.ppc64le,
Architecture.s390x, Architecture.s390x,
},
"macos": {Architecture.x86_64, Architecture.arm64, Architecture.universal2},
"windows": {Architecture.x86, Architecture.AMD64, Architecture.ARM64},
"pyodide": {Architecture.wasm32},
} }
return all_archs_map[platform] elif platform == "macos":
return {Architecture.x86_64, Architecture.arm64, Architecture.universal2}
elif platform == "windows":
return {Architecture.x86, Architecture.AMD64}
else:
assert_never(platform)
@staticmethod @staticmethod
# pylint: disable-next=inconsistent-return-statements def bitness_archs(platform: PlatformName, bitness: Literal["64", "32"]) -> "Set[Architecture]":
def bitness_archs(platform: PlatformName, bitness: Literal["64", "32"]) -> set[Architecture]:
archs_32 = {Architecture.i686, Architecture.x86} archs_32 = {Architecture.i686, Architecture.x86}
auto_archs = Architecture.auto_archs(platform) auto_archs = Architecture.auto_archs(platform)
@@ -155,6 +104,7 @@ def allowed_architectures_check(
platform: PlatformName, platform: PlatformName,
architectures: Set[Architecture], architectures: Set[Architecture],
) -> None: ) -> None:
allowed_architectures = Architecture.all_archs(platform) allowed_architectures = Architecture.all_archs(platform)
msg = f"{PRETTY_NAMES[platform]} only supports {sorted(allowed_architectures)} at the moment." msg = f"{PRETTY_NAMES[platform]} only supports {sorted(allowed_architectures)} at the moment."
+18 -25
View File
@@ -1,9 +1,5 @@
from __future__ import annotations
import subprocess import subprocess
from collections.abc import Iterable, Mapping, Sequence from typing import Callable, Dict, List, NamedTuple, Optional, Sequence
from dataclasses import dataclass
from typing import Callable, Dict, List # noqa: TID251
import bashlex import bashlex
@@ -11,19 +7,20 @@ import bashlex
EnvironmentExecutor = Callable[[List[str], Dict[str, str]], str] EnvironmentExecutor = Callable[[List[str], Dict[str, str]], str]
def local_environment_executor(command: Sequence[str], env: Mapping[str, str]) -> str: def local_environment_executor(command: List[str], env: Dict[str, str]) -> str:
return subprocess.run(command, env=env, text=True, stdout=subprocess.PIPE, check=True).stdout return subprocess.run(
command, env=env, universal_newlines=True, stdout=subprocess.PIPE, check=True
).stdout
@dataclass(frozen=True) class NodeExecutionContext(NamedTuple):
class NodeExecutionContext: environment: Dict[str, str]
environment: dict[str, str]
input: str input: str
executor: EnvironmentExecutor executor: EnvironmentExecutor
def evaluate( def evaluate(
value: str, environment: Mapping[str, str], executor: EnvironmentExecutor | None = None value: str, environment: Dict[str, str], executor: Optional[EnvironmentExecutor] = None
) -> str: ) -> str:
if not value: if not value:
# empty string evaluates to empty string # empty string evaluates to empty string
@@ -33,17 +30,14 @@ def evaluate(
command_node = bashlex.parsesingle(value) command_node = bashlex.parsesingle(value)
if len(command_node.parts) != 1: if len(command_node.parts) != 1:
msg = f"{value!r} has too many parts" raise ValueError(f'"{value}" has too many parts')
raise ValueError(msg)
value_word_node = command_node.parts[0] value_word_node = command_node.parts[0]
return evaluate_node( return evaluate_node(
value_word_node, value_word_node,
context=NodeExecutionContext( context=NodeExecutionContext(
environment=dict(environment), environment=environment, input=value, executor=executor or local_environment_executor
input=value,
executor=executor or local_environment_executor,
), ),
) )
@@ -58,8 +52,7 @@ def evaluate_node(node: bashlex.ast.node, context: NodeExecutionContext) -> str:
elif node.kind == "parameter": elif node.kind == "parameter":
return evaluate_parameter_node(node, context=context) return evaluate_parameter_node(node, context=context)
else: else:
msg = f"Unsupported bash construct: {node.kind!r}" raise ValueError(f'Unsupported bash construct: "{node.kind}"')
raise ValueError(msg)
def evaluate_word_node(node: bashlex.ast.node, context: NodeExecutionContext) -> str: def evaluate_word_node(node: bashlex.ast.node, context: NodeExecutionContext) -> str:
@@ -70,8 +63,10 @@ def evaluate_word_node(node: bashlex.ast.node, context: NodeExecutionContext) ->
part_value = evaluate_node(part, context=context) part_value = evaluate_node(part, context=context)
if part_string not in value: if part_string not in value:
msg = f"bash parse failed. part {part_string!r} not found in {value!r}. Word was {node.word!r}. Full input was {context.input!r}" raise RuntimeError(
raise RuntimeError(msg) f'bash parse failed. part "{part_string}" not found in "{value}". '
f'Word was "{node.word}". Full input was "{context.input}"'
)
value = value.replace(part_string, part_value, 1) value = value.replace(part_string, part_value, 1)
@@ -98,17 +93,15 @@ def evaluate_nodes_as_compound_command(
result += evaluate_command_node(node, context=context) result += evaluate_command_node(node, context=context)
elif node.kind == "operator": elif node.kind == "operator":
if node.op != ";": if node.op != ";":
msg = f"Unsupported bash operator: {node.op!r}" raise ValueError(f'Unsupported bash operator: "{node.op}"')
raise ValueError(msg)
else: else:
msg = f"Unsupported bash node in compound command: {node.kind!r}" raise ValueError(f'Unsupported bash node in compound command: "{node.kind}"')
raise ValueError(msg)
return result return result
def evaluate_nodes_as_simple_command( def evaluate_nodes_as_simple_command(
nodes: Iterable[bashlex.ast.node], context: NodeExecutionContext nodes: List[bashlex.ast.node], context: NodeExecutionContext
) -> str: ) -> str:
command = [evaluate_node(part, context=context) for part in nodes] command = [evaluate_node(part, context=context) for part in nodes]
return context.executor(command, context.environment) return context.executor(command, context.environment)
+242
View File
@@ -0,0 +1,242 @@
import io
import json
import os
import shlex
import subprocess
import sys
import uuid
from pathlib import Path, PurePath
from types import TracebackType
from typing import IO, Dict, List, Optional, Sequence, Type, cast
from .typing import PathOrStr, PopenBytes
class DockerContainer:
"""
An object that represents a running Docker container.
Intended for use as a context manager e.g.
`with DockerContainer('ubuntu') as docker:`
A bash shell is running in the remote container. When `call()` is invoked,
the command is relayed to the remote shell, and the results are streamed
back to cibuildwheel.
"""
UTILITY_PYTHON = "/opt/python/cp38-cp38/bin/python"
process: PopenBytes
bash_stdin: IO[bytes]
bash_stdout: IO[bytes]
def __init__(
self, docker_image: str, simulate_32_bit: bool = False, cwd: Optional[PathOrStr] = None
):
if not docker_image:
raise ValueError("Must have a non-empty docker image to run.")
self.docker_image = docker_image
self.simulate_32_bit = simulate_32_bit
self.cwd = cwd
self.name: Optional[str] = None
def __enter__(self) -> "DockerContainer":
self.name = f"cibuildwheel-{uuid.uuid4()}"
cwd_args = ["-w", str(self.cwd)] if self.cwd else []
shell_args = ["linux32", "/bin/bash"] if self.simulate_32_bit else ["/bin/bash"]
subprocess.run(
[
"docker",
"create",
"--env=CIBUILDWHEEL",
f"--name={self.name}",
"--interactive",
"--volume=/:/host", # ignored on CircleCI
*cwd_args,
self.docker_image,
*shell_args,
],
check=True,
)
self.process = subprocess.Popen(
[
"docker",
"start",
"--attach",
"--interactive",
self.name,
],
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
)
assert self.process.stdin and self.process.stdout
self.bash_stdin = self.process.stdin
self.bash_stdout = self.process.stdout
# run a noop command to block until the container is responding
self.call(["/bin/true"])
return self
def __exit__(
self,
exc_type: Optional[Type[BaseException]],
exc_val: Optional[BaseException],
exc_tb: Optional[TracebackType],
) -> None:
self.bash_stdin.close()
self.process.terminate()
self.process.wait()
assert isinstance(self.name, str)
subprocess.run(["docker", "rm", "--force", "-v", self.name], stdout=subprocess.DEVNULL)
self.name = None
def copy_into(self, from_path: Path, to_path: PurePath) -> None:
# `docker cp` causes 'no space left on device' error when
# a container is running and the host filesystem is
# mounted. https://github.com/moby/moby/issues/38995
# Use `docker exec` instead.
if from_path.is_dir():
self.call(["mkdir", "-p", to_path])
subprocess.run(
f"tar cf - . | docker exec -i {self.name} tar -xC {shell_quote(to_path)} -f -",
shell=True,
check=True,
cwd=from_path,
)
else:
subprocess.run(
f'cat {shell_quote(from_path)} | docker exec -i {self.name} sh -c "cat > {shell_quote(to_path)}"',
shell=True,
check=True,
)
def copy_out(self, from_path: PurePath, to_path: Path) -> None:
# note: we assume from_path is a dir
to_path.mkdir(parents=True, exist_ok=True)
subprocess.run(
f"docker exec -i {self.name} tar -cC {shell_quote(from_path)} -f - . | tar -xf -",
shell=True,
check=True,
cwd=to_path,
)
def glob(self, path: PurePath, pattern: str) -> List[PurePath]:
glob_pattern = os.path.join(str(path), pattern)
path_strs = json.loads(
self.call(
[
self.UTILITY_PYTHON,
"-c",
f"import sys, json, glob; json.dump(glob.glob({glob_pattern!r}), sys.stdout)",
],
capture_output=True,
)
)
return [PurePath(p) for p in path_strs]
def call(
self,
args: Sequence[PathOrStr],
env: Optional[Dict[str, str]] = None,
capture_output: bool = False,
cwd: Optional[PathOrStr] = None,
) -> str:
chdir = f"cd {cwd}" if cwd else ""
env_assignments = (
" ".join(f"{shlex.quote(k)}={shlex.quote(v)}" for k, v in env.items())
if env is not None
else ""
)
command = " ".join(shlex.quote(str(a)) for a in args)
end_of_message = str(uuid.uuid4())
# log the command we're executing
print(f" + {command}")
# Write a command to the remote shell. First we change the
# cwd, if that's required. Then, we use the `env` utility to run
# `command` inside the specified environment. We use `env` because it
# can cope with spaces and strange characters in the name or value.
# Finally, the remote shell is told to write a footer - this will show
# up in the output so we know when to stop reading, and will include
# the returncode of `command`.
self.bash_stdin.write(
bytes(
f"""(
{chdir}
env {env_assignments} {command}
printf "%04d%s\n" $? {end_of_message}
)
""",
encoding="utf8",
errors="surrogateescape",
)
)
self.bash_stdin.flush()
if capture_output:
output_io: IO[bytes] = io.BytesIO()
else:
output_io = sys.stdout.buffer
while True:
line = self.bash_stdout.readline()
if line.endswith(bytes(end_of_message, encoding="utf8") + b"\n"):
# fmt: off
footer_offset = (
len(line)
- 1 # newline character
- len(end_of_message) # delimiter
- 4 # 4 returncode decimals
)
# fmt: on
returncode_str = line[footer_offset : footer_offset + 4]
returncode = int(returncode_str)
# add the last line to output, without the footer
output_io.write(line[0:footer_offset])
break
else:
output_io.write(line)
if isinstance(output_io, io.BytesIO):
output = str(output_io.getvalue(), encoding="utf8", errors="surrogateescape")
else:
output = ""
if returncode != 0:
raise subprocess.CalledProcessError(returncode, args, output)
return output
def get_environment(self) -> Dict[str, str]:
env = json.loads(
self.call(
[
self.UTILITY_PYTHON,
"-c",
"import sys, json, os; json.dump(os.environ.copy(), sys.stdout)",
],
capture_output=True,
)
)
return cast(Dict[str, str], env)
def environment_executor(self, command: List[str], environment: Dict[str, str]) -> str:
# used as an EnvironmentExecutor to evaluate commands and capture output
return self.call(command, env=environment, capture_output=True)
def shell_quote(path: PurePath) -> str:
return shlex.quote(str(path))
+18 -71
View File
@@ -1,11 +1,6 @@
from __future__ import annotations from typing import Dict, List, Mapping, Optional
import dataclasses
from collections.abc import Mapping, Sequence
from typing import Any, Protocol
import bashlex import bashlex
import bashlex.errors
from . import bashlex_eval from . import bashlex_eval
@@ -14,7 +9,7 @@ class EnvironmentParseError(Exception):
pass pass
def split_env_items(env_string: str) -> list[str]: def split_env_items(env_string: str) -> List[str]:
"""Splits space-separated variable assignments into a list of individual assignments. """Splits space-separated variable assignments into a list of individual assignments.
>>> split_env_items('VAR=abc') >>> split_env_items('VAR=abc')
@@ -33,11 +28,7 @@ def split_env_items(env_string: str) -> list[str]:
if not env_string: if not env_string:
return [] return []
try:
command_node = bashlex.parsesingle(env_string) command_node = bashlex.parsesingle(env_string)
except bashlex.errors.ParsingError as e:
raise EnvironmentParseError(env_string) from e
result = [] result = []
for word_node in command_node.parts: for word_node in command_node.parts:
@@ -47,40 +38,7 @@ def split_env_items(env_string: str) -> list[str]:
return result return result
class EnvironmentAssignment(Protocol): class EnvironmentAssignment:
name: str
def evaluated_value(
self,
*,
environment: Mapping[str, str],
executor: bashlex_eval.EnvironmentExecutor | None = None,
) -> str:
"""Returns the value of this assignment, as evaluated in the environment"""
class EnvironmentAssignmentRaw:
"""
An environment variable - a simple name/value pair
"""
def __init__(self, name: str, value: str):
self.name = name
self.value = value
def __repr__(self) -> str:
return f"{self.name}={self.value}"
def evaluated_value(self, **_: Any) -> str:
return self.value
class EnvironmentAssignmentBash:
"""
An environment variable, in bash syntax. The value can use bash constructs
like "$OTHER_VAR" and "$(command arg1 arg2)".
"""
def __init__(self, assignment: str): def __init__(self, assignment: str):
name, equals, value = assignment.partition("=") name, equals, value = assignment.partition("=")
if not equals: if not equals:
@@ -90,33 +48,29 @@ class EnvironmentAssignmentBash:
def evaluated_value( def evaluated_value(
self, self,
environment: Mapping[str, str], environment: Dict[str, str],
executor: bashlex_eval.EnvironmentExecutor | None = None, executor: Optional[bashlex_eval.EnvironmentExecutor] = None,
) -> str: ) -> str:
"""Returns the value of this assignment, as evaluated in the environment"""
return bashlex_eval.evaluate(self.value, environment=environment, executor=executor) return bashlex_eval.evaluate(self.value, environment=environment, executor=executor)
def as_shell_assignment(self) -> str:
return f"export {self.name}={self.value}"
def __repr__(self) -> str: def __repr__(self) -> str:
return f"{self.name}={self.value}" return f"{self.name}={self.value}"
def __eq__(self, other: object) -> bool:
if isinstance(other, EnvironmentAssignmentBash):
return self.name == other.name and self.value == other.value
return False
@dataclasses.dataclass
class ParsedEnvironment: class ParsedEnvironment:
assignments: list[EnvironmentAssignment] def __init__(self, assignments: List[EnvironmentAssignment]):
self.assignments = assignments
def __init__(self, assignments: Sequence[EnvironmentAssignment]) -> None:
self.assignments = list(assignments)
def as_dictionary( def as_dictionary(
self, self,
prev_environment: Mapping[str, str], prev_environment: Mapping[str, str],
executor: bashlex_eval.EnvironmentExecutor | None = None, executor: Optional[bashlex_eval.EnvironmentExecutor] = None,
) -> dict[str, str]: ) -> Dict[str, str]:
environment = {**prev_environment} environment = dict(**prev_environment)
for assignment in self.assignments: for assignment in self.assignments:
value = assignment.evaluated_value(environment=environment, executor=executor) value = assignment.evaluated_value(environment=environment, executor=executor)
@@ -124,21 +78,14 @@ class ParsedEnvironment:
return environment return environment
def add(self, name: str, value: str, prepend: bool = False) -> None: def as_shell_commands(self) -> List[str]:
assignment = EnvironmentAssignmentRaw(name=name, value=value) return [a.as_shell_assignment() for a in self.assignments]
if prepend:
self.assignments.insert(0, assignment)
else:
self.assignments.append(assignment)
def __repr__(self) -> str: def __repr__(self) -> str:
return f"{self.__class__.__name__}({[repr(a) for a in self.assignments]!r})" return f"ParsedEnvironment({[repr(a) for a in self.assignments]!r})"
def options_summary(self) -> Any:
return self.assignments
def parse_environment(env_string: str) -> ParsedEnvironment: def parse_environment(env_string: str) -> ParsedEnvironment:
env_items = split_env_items(env_string) env_items = split_env_items(env_string)
assignments = [EnvironmentAssignmentBash(item) for item in env_items] assignments = [EnvironmentAssignment(item) for item in env_items]
return ParsedEnvironment(assignments=assignments) return ParsedEnvironment(assignments=assignments)
-60
View File
@@ -1,60 +0,0 @@
"""
Errors that can cause the build to fail. Each subclass of FatalError has
a different return code, by defining them all here, we can ensure that they're
semantically clear and unique.
"""
import textwrap
class FatalError(BaseException):
"""
Raising an error of this type will cause the message to be printed to
stderr and the process to be terminated. Within cibuildwheel, raising this
exception produces a better error message, and optional traceback.
"""
return_code: int = 1
class ConfigurationError(FatalError):
return_code = 2
class NothingToDoError(FatalError):
return_code = 3
class DeprecationError(FatalError):
return_code = 4
class NonPlatformWheelError(FatalError):
def __init__(self) -> None:
message = textwrap.dedent(
"""
Build failed because a pure Python wheel was generated.
If you intend to build a pure-Python wheel, you don't need cibuildwheel - use
`pip wheel -w DEST_DIR .` instead.
If you expected a platform wheel, check your project configuration, or run
cibuildwheel with CIBW_BUILD_VERBOSITY=1 to view build logs.
"""
)
super().__init__(message)
self.return_code = 5
class AlreadyBuiltWheelError(FatalError):
def __init__(self, wheel_name: str) -> None:
message = textwrap.dedent(
f"""
Build failed because a wheel named {wheel_name} was already generated in the current run.
If you expected another wheel to be generated, check your project configuration, or run
cibuildwheel with CIBW_BUILD_VERBOSITY=1 to view build logs.
"""
)
super().__init__(message)
self.return_code = 6
+15 -26
View File
@@ -2,33 +2,22 @@
These are utilities for the `/bin` scripts, not for the `cibuildwheel` program. These are utilities for the `/bin` scripts, not for the `cibuildwheel` program.
""" """
from __future__ import annotations from typing import Any, Dict
from collections.abc import Mapping, Sequence import toml.encoder
from io import StringIO from packaging.version import Version
from typing import Protocol
__all__ = ("Printable", "dump_python_configurations")
class Printable(Protocol): class InlineArrayDictEncoder(toml.encoder.TomlEncoder): # type: ignore
def __str__(self) -> str: ... def __init__(self) -> None:
super().__init__()
self.dump_funcs[Version] = lambda v: f'"{v}"'
def dump_sections(self, o: Dict[str, Any], sup: str) -> Any:
def dump_python_configurations( if not all(isinstance(a, list) for a in o.values()):
inp: Mapping[str, Mapping[str, Sequence[Mapping[str, Printable]]]], return super().dump_sections(o, sup)
) -> str: val = ""
output = StringIO() for k, v in o.items():
for header, values in inp.items(): inner = ",\n ".join(self.dump_inline_table(d_i).strip() for d_i in v)
output.write(f"[{header}]\n") val += f"{k} = [\n {inner},\n]\n"
for inner_header, listing in values.items(): return val, self._dict()
output.write(f"{inner_header} = [\n")
for item in listing:
output.write(" { ")
dict_contents = (f'{key} = "{value}"' for key, value in item.items())
output.write(", ".join(dict_contents))
output.write(" },\n")
output.write("]\n")
output.write("\n")
# Strip the final newline, to avoid two blank lines at the end.
return output.getvalue()[:-1]
+164 -341
View File
@@ -1,58 +1,39 @@
from __future__ import annotations
import subprocess import subprocess
import sys import sys
import textwrap import textwrap
from collections.abc import Iterable, Iterator, Sequence, Set from pathlib import Path, PurePath
from dataclasses import dataclass from typing import List, NamedTuple, Set
from pathlib import Path, PurePath, PurePosixPath
from typing import OrderedDict, Tuple
from packaging.version import Version
from . import errors
from ._compat.typing import assert_never
from .architecture import Architecture from .architecture import Architecture
from .docker_container import DockerContainer
from .logger import log from .logger import log
from .oci_container import OCIContainer, OCIContainerEngineConfig
from .options import BuildOptions, Options
from .typing import PathOrStr from .typing import PathOrStr
from .util import ( from .util import (
BuildFrontendConfig, BuildOptions,
BuildSelector, BuildSelector,
find_compatible_wheel, NonPlatformWheelError,
get_build_verbosity_extra_flags, get_build_verbosity_extra_flags,
prepare_command, prepare_command,
read_python_configs, read_python_configs,
split_config_settings, resources_dir,
test_fail_cwd_file,
unwrap,
) )
@dataclass(frozen=True) class PythonConfiguration(NamedTuple):
class PythonConfiguration:
version: str version: str
identifier: str identifier: str
path_str: str path_str: str
@property @property
def path(self) -> PurePosixPath: def path(self) -> PurePath:
return PurePosixPath(self.path_str) return PurePath(self.path_str)
@dataclass(frozen=True)
class BuildStep:
platform_configs: list[PythonConfiguration]
platform_tag: str
container_engine: OCIContainerEngineConfig
container_image: str
def get_python_configurations( def get_python_configurations(
build_selector: BuildSelector, build_selector: BuildSelector,
architectures: Set[Architecture], architectures: Set[Architecture],
) -> list[PythonConfiguration]: ) -> List[PythonConfiguration]:
full_python_configs = read_python_configs("linux") full_python_configs = read_python_configs("linux")
python_configurations = [PythonConfiguration(**item) for item in full_python_configs] python_configurations = [PythonConfiguration(**item) for item in full_python_configs]
@@ -67,296 +48,217 @@ def get_python_configurations(
] ]
def container_image_for_python_configuration( def build(options: BuildOptions) -> None:
config: PythonConfiguration, build_options: BuildOptions try:
) -> str: # check docker is installed
# e.g subprocess.run(["docker", "--version"], check=True, stdout=subprocess.DEVNULL)
# identifier is 'cp310-manylinux_x86_64' except Exception:
# platform_tag is 'manylinux_x86_64' print(
# platform_arch is 'x86_64' "cibuildwheel: Docker not found. Docker is required to run Linux builds. "
_, platform_tag = config.identifier.split("-", 1) "If you're building on Travis CI, add `services: [docker]` to your .travis.yml."
_, platform_arch = platform_tag.split("_", 1) "If you're building on Circle CI in Linux, add a `setup_remote_docker` step to your .circleci/config.yml",
file=sys.stderr,
assert build_options.manylinux_images is not None
assert build_options.musllinux_images is not None
return (
build_options.manylinux_images[platform_arch]
if platform_tag.startswith("manylinux")
else build_options.musllinux_images[platform_arch]
) )
sys.exit(2)
assert options.manylinux_images is not None
python_configurations = get_python_configurations(options.build_selector, options.architectures)
platforms = [
("cp", "manylinux_x86_64", options.manylinux_images["x86_64"]),
("cp", "manylinux_i686", options.manylinux_images["i686"]),
("cp", "manylinux_aarch64", options.manylinux_images["aarch64"]),
("cp", "manylinux_ppc64le", options.manylinux_images["ppc64le"]),
("cp", "manylinux_s390x", options.manylinux_images["s390x"]),
("pp", "manylinux_x86_64", options.manylinux_images["pypy_x86_64"]),
]
def get_build_steps( cwd = Path.cwd()
options: Options, python_configurations: list[PythonConfiguration] abs_package_dir = options.package_dir.resolve()
) -> Iterator[BuildStep]: if cwd != abs_package_dir and cwd not in abs_package_dir.parents:
""" raise Exception("package_dir must be inside the working directory")
Groups PythonConfigurations into BuildSteps. Each BuildStep represents a
separate container instance.
"""
steps = OrderedDict[Tuple[str, str, str, OCIContainerEngineConfig], BuildStep]()
for config in python_configurations: container_project_path = PurePath("/project")
_, platform_tag = config.identifier.split("-", 1) container_package_dir = container_project_path / abs_package_dir.relative_to(cwd)
container_output_dir = PurePath("/output")
build_options = options.build_options(config.identifier) for implementation, platform_tag, docker_image in platforms:
platform_configs = [
before_all = build_options.before_all c
container_image = container_image_for_python_configuration(config, build_options) for c in python_configurations
container_engine = build_options.container_engine if c.identifier.startswith(implementation) and c.identifier.endswith(platform_tag)
]
step_key = (platform_tag, container_image, before_all, container_engine) if not platform_configs:
continue
if step_key in steps:
steps[step_key].platform_configs.append(config)
else:
steps[step_key] = BuildStep(
platform_configs=[config],
platform_tag=platform_tag,
container_engine=container_engine,
container_image=container_image,
)
yield from steps.values()
def check_all_python_exist(
*, platform_configs: Iterable[PythonConfiguration], container: OCIContainer
) -> None:
exist = True
has_manylinux_interpreters = True
messages = []
try: try:
# use capture_output to keep quiet log.step(f"Starting Docker image {docker_image}...")
container.call(["manylinux-interpreters", "--help"], capture_output=True) with DockerContainer(
except subprocess.CalledProcessError: docker_image,
has_manylinux_interpreters = False simulate_32_bit=platform_tag.endswith("i686"),
cwd=container_project_path,
) as docker:
for config in platform_configs: log.step("Copying project into Docker...")
python_path = config.path / "bin" / "python" docker.copy_into(Path.cwd(), container_project_path)
try:
if has_manylinux_interpreters:
container.call(["manylinux-interpreters", "ensure", config.path.name])
container.call(["test", "-x", python_path])
except subprocess.CalledProcessError:
messages.append(
f" '{python_path}' executable doesn't exist in image '{container.image}' to build '{config.identifier}'."
)
exist = False
if not exist:
message = "\n".join(messages)
raise errors.FatalError(message)
if options.before_all:
def build_in_container(
*,
options: Options,
platform_configs: Sequence[PythonConfiguration],
container: OCIContainer,
container_project_path: PurePath,
container_package_dir: PurePath,
) -> None:
container_output_dir = PurePosixPath("/output")
check_all_python_exist(platform_configs=platform_configs, container=container)
log.step("Copying project into container...")
container.copy_into(Path.cwd(), container_project_path)
before_all_options_identifier = platform_configs[0].identifier
before_all_options = options.build_options(before_all_options_identifier)
if before_all_options.before_all:
log.step("Running before_all...") log.step("Running before_all...")
env = container.get_environment() env = docker.get_environment()
env["PATH"] = f'/opt/python/cp38-cp38/bin:{env["PATH"]}' env["PATH"] = f'/opt/python/cp38-cp38/bin:{env["PATH"]}'
env["PIP_DISABLE_PIP_VERSION_CHECK"] = "1" env["PIP_DISABLE_PIP_VERSION_CHECK"] = "1"
env["PIP_ROOT_USER_ACTION"] = "ignore" env = options.environment.as_dictionary(
env = before_all_options.environment.as_dictionary( env, executor=docker.environment_executor
env, executor=container.environment_executor
) )
before_all_prepared = prepare_command( before_all_prepared = prepare_command(
before_all_options.before_all, options.before_all,
project=container_project_path, project=container_project_path,
package=container_package_dir, package=container_package_dir,
) )
container.call(["sh", "-c", before_all_prepared], env=env) docker.call(["sh", "-c", before_all_prepared], env=env)
built_wheels: list[PurePosixPath] = []
for config in platform_configs: for config in platform_configs:
log.build_start(config.identifier) log.build_start(config.identifier)
build_options = options.build_options(config.identifier)
build_frontend = build_options.build_frontend or BuildFrontendConfig("pip")
use_uv = build_frontend.name == "build[uv]" and Version(config.version) >= Version("3.8")
pip = ["uv", "pip"] if use_uv else ["pip"]
dependency_constraint_flags: list[PathOrStr] = [] dependency_constraint_flags: List[PathOrStr] = []
if config.identifier.startswith("pp"):
# Patch PyPy to make sure headers get installed into a venv
patch_version = "_27" if config.version == "2.7" else ""
patch_path = resources_dir / f"pypy_venv{patch_version}.patch"
patch_docker_path = PurePath("/pypy_venv.patch")
docker.copy_into(patch_path, patch_docker_path)
try:
docker.call(
[
"patch",
"--force",
"-p1",
"-d",
config.path,
"-i",
patch_docker_path,
]
)
except subprocess.CalledProcessError:
print("PyPy patch not applied", file=sys.stderr)
if build_options.dependency_constraints: if options.dependency_constraints:
constraints_file = build_options.dependency_constraints.get_for_python_version( constraints_file = options.dependency_constraints.get_for_python_version(
config.version config.version
) )
container_constraints_file = PurePosixPath("/constraints.txt") container_constraints_file = PurePath("/constraints.txt")
container.copy_into(constraints_file, container_constraints_file) docker.copy_into(constraints_file, container_constraints_file)
dependency_constraint_flags = ["-c", container_constraints_file] dependency_constraint_flags = ["-c", container_constraints_file]
log.step("Setting up build environment...") log.step("Setting up build environment...")
env = container.get_environment() env = docker.get_environment()
env["PIP_DISABLE_PIP_VERSION_CHECK"] = "1"
env["PIP_ROOT_USER_ACTION"] = "ignore"
# put this config's python top of the list # put this config's python top of the list
python_bin = config.path / "bin" python_bin = config.path / "bin"
env["PATH"] = f'{python_bin}:{env["PATH"]}' env["PATH"] = f'{python_bin}:{env["PATH"]}'
env = build_options.environment.as_dictionary(env, executor=container.environment_executor) env = options.environment.as_dictionary(
env, executor=docker.environment_executor
# check config python is still on PATH
which_python = container.call(["which", "python"], env=env, capture_output=True).strip()
if PurePosixPath(which_python) != python_bin / "python":
msg = "python available on PATH doesn't match our installed instance. If you have modified PATH, ensure that you don't overwrite cibuildwheel's entry or insert python above it."
raise errors.FatalError(msg)
if use_uv:
which_uv = container.call(["which", "uv"], env=env, capture_output=True).strip()
if not which_uv:
msg = "uv not found on PATH. You must use a supported manylinux or musllinux environment with uv."
raise errors.FatalError(msg)
else:
which_pip = container.call(["which", "pip"], env=env, capture_output=True).strip()
if PurePosixPath(which_pip) != python_bin / "pip":
msg = "pip available on PATH doesn't match our installed instance. If you have modified PATH, ensure that you don't overwrite cibuildwheel's entry or insert pip above it."
raise errors.FatalError(msg)
compatible_wheel = find_compatible_wheel(built_wheels, config.identifier)
if compatible_wheel:
log.step_end()
print(
f"\nFound previously built wheel {compatible_wheel.name}, that's compatible with {config.identifier}. Skipping build step..."
) )
repaired_wheels = [compatible_wheel]
else: # check config python and pip are still on PATH
if build_options.before_build: which_python = docker.call(
["which", "python"], env=env, capture_output=True
).strip()
if PurePath(which_python) != python_bin / "python":
print(
"cibuildwheel: python available on PATH doesn't match our installed instance. If you have modified PATH, ensure that you don't overwrite cibuildwheel's entry or insert python above it.",
file=sys.stderr,
)
sys.exit(1)
which_pip = docker.call(["which", "pip"], env=env, capture_output=True).strip()
if PurePath(which_pip) != python_bin / "pip":
print(
"cibuildwheel: pip available on PATH doesn't match our installed instance. If you have modified PATH, ensure that you don't overwrite cibuildwheel's entry or insert pip above it.",
file=sys.stderr,
)
sys.exit(1)
if options.before_build:
log.step("Running before_build...") log.step("Running before_build...")
before_build_prepared = prepare_command( before_build_prepared = prepare_command(
build_options.before_build, options.before_build,
project=container_project_path, project=container_project_path,
package=container_package_dir, package=container_package_dir,
) )
container.call(["sh", "-c", before_build_prepared], env=env) docker.call(["sh", "-c", before_build_prepared], env=env)
log.step("Building wheel...") log.step("Building wheel...")
temp_dir = PurePosixPath("/tmp/cibuildwheel") temp_dir = PurePath("/tmp/cibuildwheel")
built_wheel_dir = temp_dir / "built_wheel" built_wheel_dir = temp_dir / "built_wheel"
container.call(["rm", "-rf", built_wheel_dir]) docker.call(["rm", "-rf", built_wheel_dir])
container.call(["mkdir", "-p", built_wheel_dir]) docker.call(["mkdir", "-p", built_wheel_dir])
extra_flags = split_config_settings(build_options.config_settings, build_frontend.name) docker.call(
extra_flags += build_frontend.args
if build_frontend.name == "pip":
extra_flags += get_build_verbosity_extra_flags(build_options.build_verbosity)
container.call(
[ [
"python",
"-m",
"pip", "pip",
"wheel", "wheel",
container_package_dir, container_package_dir,
f"--wheel-dir={built_wheel_dir}", "--wheel-dir",
built_wheel_dir,
"--no-deps", "--no-deps",
*extra_flags, *get_build_verbosity_extra_flags(options.build_verbosity),
], ],
env=env, env=env,
) )
elif build_frontend.name == "build" or build_frontend.name == "build[uv]":
if not 0 <= build_options.build_verbosity < 2:
msg = f"build_verbosity {build_options.build_verbosity} is not supported for build frontend. Ignoring."
log.warning(msg)
if use_uv and "--no-isolation" not in extra_flags and "-n" not in extra_flags:
extra_flags += ["--installer=uv"]
container.call(
[
"python",
"-m",
"build",
container_package_dir,
"--wheel",
f"--outdir={built_wheel_dir}",
*extra_flags,
],
env=env,
)
else:
assert_never(build_frontend)
built_wheel = container.glob(built_wheel_dir, "*.whl")[0] built_wheel = docker.glob(built_wheel_dir, "*.whl")[0]
repaired_wheel_dir = temp_dir / "repaired_wheel" repaired_wheel_dir = temp_dir / "repaired_wheel"
container.call(["rm", "-rf", repaired_wheel_dir]) docker.call(["rm", "-rf", repaired_wheel_dir])
container.call(["mkdir", "-p", repaired_wheel_dir]) docker.call(["mkdir", "-p", repaired_wheel_dir])
if built_wheel.name.endswith("none-any.whl"): if built_wheel.name.endswith("none-any.whl"):
raise errors.NonPlatformWheelError() raise NonPlatformWheelError()
if build_options.repair_command: if options.repair_command:
log.step("Repairing wheel...") log.step("Repairing wheel...")
repair_command_prepared = prepare_command( repair_command_prepared = prepare_command(
build_options.repair_command, wheel=built_wheel, dest_dir=repaired_wheel_dir options.repair_command, wheel=built_wheel, dest_dir=repaired_wheel_dir
) )
container.call(["sh", "-c", repair_command_prepared], env=env) docker.call(["sh", "-c", repair_command_prepared], env=env)
else: else:
container.call(["mv", built_wheel, repaired_wheel_dir]) docker.call(["mv", built_wheel, repaired_wheel_dir])
repaired_wheels = container.glob(repaired_wheel_dir, "*.whl") repaired_wheels = docker.glob(repaired_wheel_dir, "*.whl")
for repaired_wheel in repaired_wheels: if options.test_command and options.test_selector(config.identifier):
if repaired_wheel.name in {wheel.name for wheel in built_wheels}:
raise errors.AlreadyBuiltWheelError(repaired_wheel.name)
if build_options.test_command and build_options.test_selector(config.identifier):
log.step("Testing wheel...") log.step("Testing wheel...")
# set up a virtual environment to install and test from, to make sure # set up a virtual environment to install and test from, to make sure
# there are no dependencies that were pulled in at build time. # there are no dependencies that were pulled in at build time.
if not use_uv: docker.call(
container.call(
["pip", "install", "virtualenv", *dependency_constraint_flags], env=env ["pip", "install", "virtualenv", *dependency_constraint_flags], env=env
) )
venv_dir = (
testing_temp_dir = PurePosixPath( PurePath(docker.call(["mktemp", "-d"], capture_output=True).strip())
container.call(["mktemp", "-d"], capture_output=True).strip() / "venv"
) )
venv_dir = testing_temp_dir / "venv"
if use_uv: docker.call(
container.call(["uv", "venv", venv_dir], env=env) ["python", "-m", "virtualenv", "--no-download", venv_dir], env=env
else: )
# Use embedded dependencies from virtualenv to ensure determinism
venv_args = ["--no-periodic-update", "--pip=embed"]
# In Python<3.12, setuptools & wheel are installed as well
if Version(config.version) < Version("3.12"):
venv_args.extend(("--setuptools=embed", "--wheel=embed"))
container.call(["python", "-m", "virtualenv", *venv_args, venv_dir], env=env)
virtualenv_env = env.copy() virtualenv_env = env.copy()
virtualenv_env["PATH"] = f"{venv_dir / 'bin'}:{virtualenv_env['PATH']}" virtualenv_env["PATH"] = f"{venv_dir / 'bin'}:{virtualenv_env['PATH']}"
virtualenv_env["VIRTUAL_ENV"] = str(venv_dir)
if build_options.before_test: if options.before_test:
before_test_prepared = prepare_command( before_test_prepared = prepare_command(
build_options.before_test, options.before_test,
project=container_project_path, project=container_project_path,
package=container_package_dir, package=container_package_dir,
) )
container.call(["sh", "-c", before_test_prepared], env=virtualenv_env) docker.call(["sh", "-c", before_test_prepared], env=virtualenv_env)
# Install the wheel we just built # Install the wheel we just built
# Note: If auditwheel produced two wheels, it's because the earlier produced wheel # Note: If auditwheel produced two wheels, it's because the earlier produced wheel
@@ -365,125 +267,53 @@ def build_in_container(
# different external shared libraries. so it doesn't matter which one we run the tests on. # different external shared libraries. so it doesn't matter which one we run the tests on.
# Let's just pick the first one. # Let's just pick the first one.
wheel_to_test = repaired_wheels[0] wheel_to_test = repaired_wheels[0]
container.call( docker.call(
[*pip, "install", str(wheel_to_test) + build_options.test_extras], ["pip", "install", str(wheel_to_test) + options.test_extras],
env=virtualenv_env, env=virtualenv_env,
) )
# Install any requirements to run the tests # Install any requirements to run the tests
if build_options.test_requires: if options.test_requires:
container.call([*pip, "install", *build_options.test_requires], env=virtualenv_env) docker.call(
["pip", "install", *options.test_requires], env=virtualenv_env
)
# Run the tests from a different directory # Run the tests from a different directory
test_command_prepared = prepare_command( test_command_prepared = prepare_command(
build_options.test_command, options.test_command,
project=container_project_path, project=container_project_path,
package=container_package_dir, package=container_package_dir,
wheel=wheel_to_test,
) )
test_cwd = testing_temp_dir / "test_cwd" docker.call(
container.call(["mkdir", "-p", test_cwd]) ["sh", "-c", test_command_prepared], cwd="/root", env=virtualenv_env
container.copy_into(test_fail_cwd_file, test_cwd / "test_fail.py") )
container.call(["sh", "-c", test_command_prepared], cwd=test_cwd, env=virtualenv_env)
# clean up test environment # clean up test environment
container.call(["rm", "-rf", testing_temp_dir]) docker.call(["rm", "-rf", venv_dir])
# move repaired wheels to output # move repaired wheels to output
if compatible_wheel is None: docker.call(["mkdir", "-p", container_output_dir])
container.call(["mkdir", "-p", container_output_dir]) docker.call(["mv", *repaired_wheels, container_output_dir])
container.call(["mv", *repaired_wheels, container_output_dir])
built_wheels.extend(
container_output_dir / repaired_wheel.name for repaired_wheel in repaired_wheels
)
log.build_end() log.build_end()
log.step("Copying wheels back to host...") log.step("Copying wheels back to host...")
# copy the output back into the host # copy the output back into the host
container.copy_out(container_output_dir, options.globals.output_dir) docker.copy_out(container_output_dir, options.output_dir)
log.step_end() log.step_end()
def build(options: Options, tmp_path: Path) -> None: # noqa: ARG001
python_configurations = get_python_configurations(
options.globals.build_selector, options.globals.architectures
)
cwd = Path.cwd()
abs_package_dir = options.globals.package_dir.resolve()
if cwd != abs_package_dir and cwd not in abs_package_dir.parents:
msg = "package_dir must be inside the working directory"
raise Exception(msg)
container_project_path = PurePosixPath("/project")
container_package_dir = container_project_path / abs_package_dir.relative_to(cwd)
for build_step in get_build_steps(options, python_configurations):
try:
# check the container engine is installed
subprocess.run(
[build_step.container_engine.name, "--version"],
check=True,
stdout=subprocess.DEVNULL,
)
except subprocess.CalledProcessError as error: except subprocess.CalledProcessError as error:
msg = unwrap( log.step_end_with_error(
f""" f"Command {error.cmd} failed with code {error.returncode}. {error.stdout}"
cibuildwheel: {build_step.container_engine.name} not found. An
OCI exe like Docker or Podman is required to run Linux builds.
If you're building on Travis CI, add `services: [docker]` to
your .travis.yml. If you're building on Circle CI in Linux,
add a `setup_remote_docker` step to your .circleci/config.yml.
If you're building on Cirrus CI, use `docker_builder` task.
"""
) )
raise errors.ConfigurationError(msg) from error troubleshoot(options.package_dir, error)
sys.exit(1)
try:
ids_to_build = [x.identifier for x in build_step.platform_configs]
log.step(f"Starting container image {build_step.container_image}...")
print(f"info: This container will host the build for {', '.join(ids_to_build)}...")
with OCIContainer(
image=build_step.container_image,
enforce_32_bit=build_step.platform_tag.endswith("i686"),
cwd=container_project_path,
engine=build_step.container_engine,
) as container:
build_in_container(
options=options,
platform_configs=build_step.platform_configs,
container=container,
container_project_path=container_project_path,
container_package_dir=container_package_dir,
)
except subprocess.CalledProcessError as error:
troubleshoot(options, error)
msg = f"Command {error.cmd} failed with code {error.returncode}. {error.stdout or ''}"
raise errors.FatalError(msg) from error
def _matches_prepared_command(error_cmd: Sequence[str], command_template: str) -> bool: def troubleshoot(package_dir: Path, error: Exception) -> None:
if len(error_cmd) < 3 or error_cmd[0:2] != ["sh", "-c"]: if isinstance(error, subprocess.CalledProcessError) and error.cmd[0:2] == ["pip", "wheel"]:
return False # the 'pip wheel' step failed.
command_prefix = command_template.split("{", maxsplit=1)[0].strip() print("Checking for common errors...")
return error_cmd[2].startswith(command_prefix) so_files = list(package_dir.glob("**/*.so"))
def troubleshoot(options: Options, error: Exception) -> None:
if isinstance(error, subprocess.CalledProcessError) and (
error.cmd[0:4] == ["python", "-m", "pip", "wheel"]
or error.cmd[0:3] == ["python", "-m", "build"]
or _matches_prepared_command(
error.cmd, options.build_options(None).repair_command
) # TODO allow matching of overrides too?
):
# the wheel build step or the repair step failed
so_files = list(options.globals.package_dir.glob("**/*.so"))
if so_files: if so_files:
print( print(
@@ -492,17 +322,10 @@ def troubleshoot(options: Options, error: Exception) -> None:
NOTE: Shared object (.so) files found in this project. NOTE: Shared object (.so) files found in this project.
These files might be built against the wrong OS, causing problems with These files might be built against the wrong OS, causing problems with
auditwheel. If possible, run cibuildwheel in a clean checkout. auditwheel.
If you're using Cython and have previously done an in-place build, If you're using Cython and have previously done an in-place build,
remove those build files (*.so and *.c) before starting cibuildwheel. remove those build files (*.so and *.c) before starting cibuildwheel.
setuptools uses the build/ folder to store its build cache. It
may be necessary to remove those build files (*.so and *.o) before
starting cibuildwheel.
Files that belong to a virtual environment are probably not an issue
unless you used a custom command telling cibuildwheel to activate it.
""" """
), ),
file=sys.stderr, file=sys.stderr,
@@ -510,4 +333,4 @@ def troubleshoot(options: Options, error: Exception) -> None:
print(" Files detected:") print(" Files detected:")
print("\n".join(f" {f}" for f in so_files)) print("\n".join(f" {f}" for f in so_files))
print() print("")
+30 -51
View File
@@ -1,40 +1,30 @@
from __future__ import annotations
import codecs import codecs
import os import os
import re import re
import sys import sys
import time import time
from typing import IO, AnyStr, Final, Tuple from typing import IO, AnyStr, Optional, Union
from .util import CIProvider, detect_ci_provider from cibuildwheel.util import CIProvider, detect_ci_provider
FoldPattern = Tuple[str, str] DEFAULT_FOLD_PATTERN = ("{name}", "")
DEFAULT_FOLD_PATTERN: Final[FoldPattern] = ("{name}", "") FOLD_PATTERNS = {
FOLD_PATTERNS: Final[dict[str, FoldPattern]] = {
"azure": ("##[group]{name}", "##[endgroup]"), "azure": ("##[group]{name}", "##[endgroup]"),
"travis": ("travis_fold:start:{identifier}\n{name}", "travis_fold:end:{identifier}"), "travis": ("travis_fold:start:{identifier}\n{name}", "travis_fold:end:{identifier}"),
"github": ("::group::{name}", "::endgroup::{name}"), "github": ("::group::{name}", "::endgroup::{name}"),
} }
PLATFORM_IDENTIFIER_DESCRIPTIONS: Final[dict[str, str]] = { PLATFORM_IDENTIFIER_DESCIPTIONS = {
"manylinux_x86_64": "manylinux x86_64", "manylinux_x86_64": "manylinux x86_64",
"manylinux_i686": "manylinux i686", "manylinux_i686": "manylinux i686",
"manylinux_aarch64": "manylinux aarch64", "manylinux_aarch64": "manylinux aarch64",
"manylinux_ppc64le": "manylinux ppc64le", "manylinux_ppc64le": "manylinux ppc64le",
"manylinux_s390x": "manylinux s390x", "manylinux_s390x": "manylinux s390x",
"musllinux_x86_64": "musllinux x86_64",
"musllinux_i686": "musllinux i686",
"musllinux_aarch64": "musllinux aarch64",
"musllinux_ppc64le": "musllinux ppc64le",
"musllinux_s390x": "manylinux s390x",
"win32": "Windows 32bit", "win32": "Windows 32bit",
"win_amd64": "Windows 64bit", "win_amd64": "Windows 64bit",
"win_arm64": "Windows on ARM 64bit",
"macosx_x86_64": "macOS x86_64", "macosx_x86_64": "macOS x86_64",
"macosx_universal2": "macOS Universal 2 - x86_64 and arm64", "macosx_universal2": "macOS Universal 2 - x86_64 and arm64",
"macosx_arm64": "macOS arm64 - Apple Silicon", "macosx_arm64": "macOS arm64 - Apple Silicon",
"pyodide_wasm32": "Pyodide",
} }
@@ -42,10 +32,10 @@ class Logger:
fold_mode: str fold_mode: str
colors_enabled: bool colors_enabled: bool
unicode_enabled: bool unicode_enabled: bool
active_build_identifier: str | None = None active_build_identifier: Optional[str] = None
build_start_time: float | None = None build_start_time: Optional[float] = None
step_start_time: float | None = None step_start_time: Optional[float] = None
active_fold_group_name: str | None = None active_fold_group_name: Optional[str] = None
def __init__(self) -> None: def __init__(self) -> None:
if sys.platform == "win32" and hasattr(sys.stdout, "reconfigure"): if sys.platform == "win32" and hasattr(sys.stdout, "reconfigure"):
@@ -123,21 +113,10 @@ class Logger:
self.step_start_time = None self.step_start_time = None
def step_end_with_error(self, error: BaseException | str) -> None: def step_end_with_error(self, error: Union[BaseException, str]) -> None:
self.step_end(success=False) self.step_end(success=False)
self.error(error) self.error(error)
def quiet(self, message: str) -> None:
c = self.colors
print(f"{c.gray}{message}{c.end}", file=sys.stderr)
def notice(self, message: str) -> None:
if self.fold_mode == "github":
print(f"::notice::{message}\n", file=sys.stderr)
else:
c = self.colors
print(f"{c.bold}Note{c.end}: {message}\n", file=sys.stderr)
def warning(self, message: str) -> None: def warning(self, message: str) -> None:
if self.fold_mode == "github": if self.fold_mode == "github":
print(f"::warning::{message}\n", file=sys.stderr) print(f"::warning::{message}\n", file=sys.stderr)
@@ -145,17 +124,13 @@ class Logger:
c = self.colors c = self.colors
print(f"{c.yellow}Warning{c.end}: {message}\n", file=sys.stderr) print(f"{c.yellow}Warning{c.end}: {message}\n", file=sys.stderr)
def error(self, error: BaseException | str) -> None: def error(self, error: Union[BaseException, str]) -> None:
if self.fold_mode == "github": if self.fold_mode == "github":
print(f"::error::{error}\n", file=sys.stderr) print(f"::error::{error}\n", file=sys.stderr)
else: else:
c = self.colors c = self.colors
print(f"{c.bright_red}Error{c.end}: {error}\n", file=sys.stderr) print(f"{c.bright_red}Error{c.end}: {error}\n", file=sys.stderr)
@property
def step_active(self) -> bool:
return self.step_start_time is not None
def _start_fold_group(self, name: str) -> None: def _start_fold_group(self, name: str) -> None:
self._end_fold_group() self._end_fold_group()
self.active_fold_group_name = name self.active_fold_group_name = name
@@ -176,8 +151,7 @@ class Logger:
sys.stdout.flush() sys.stdout.flush()
self.active_fold_group_name = None self.active_fold_group_name = None
@staticmethod def _fold_group_identifier(self, name: str) -> str:
def _fold_group_identifier(name: str) -> str:
""" """
Travis doesn't like fold groups identifiers that have spaces in. This Travis doesn't like fold groups identifiers that have spaces in. This
method converts them to ascii identifiers method converts them to ascii identifiers
@@ -192,12 +166,18 @@ class Logger:
return identifier.lower()[:20] return identifier.lower()[:20]
@property @property
def colors(self) -> Colors: def colors(self) -> "Colors":
return Colors(enabled=self.colors_enabled) if self.colors_enabled:
return Colors(enabled=True)
else:
return Colors(enabled=False)
@property @property
def symbols(self) -> Symbols: def symbols(self) -> "Symbols":
return Symbols(unicode=self.unicode_enabled) if self.unicode_enabled:
return Symbols(unicode=True)
else:
return Symbols(unicode=False)
def build_description_from_identifier(identifier: str) -> str: def build_description_from_identifier(identifier: str) -> str:
@@ -206,23 +186,21 @@ def build_description_from_identifier(identifier: str) -> str:
build_description = "" build_description = ""
python_interpreter = python_identifier[0:2] python_interpreter = python_identifier[0:2]
python_version = python_identifier[2:] python_version = python_identifier[2:4]
if python_interpreter == "cp": if python_interpreter == "cp":
build_description += "CPython" build_description += "CPython"
elif python_interpreter == "pp": elif python_interpreter == "pp":
build_description += "PyPy" build_description += "PyPy"
else: else:
msg = f"unknown python {python_interpreter!r}" raise Exception("unknown python")
raise Exception(msg)
build_description += f" {python_version[0]}.{python_version[1:]} " build_description += f" {python_version[0]}.{python_version[1]} "
try: try:
build_description += PLATFORM_IDENTIFIER_DESCRIPTIONS[platform_identifier] build_description += PLATFORM_IDENTIFIER_DESCIPTIONS[platform_identifier]
except KeyError as e: except KeyError as e:
msg = f"unknown platform {platform_identifier!r}" raise Exception("unknown platform") from e
raise Exception(msg) from e
return build_description return build_description
@@ -237,7 +215,6 @@ class Colors:
self.bright_red = "\033[91m" if enabled else "" self.bright_red = "\033[91m" if enabled else ""
self.bright_green = "\033[92m" if enabled else "" self.bright_green = "\033[92m" if enabled else ""
self.white = "\033[37m\033[97m" if enabled else "" self.white = "\033[37m\033[97m" if enabled else ""
self.gray = "\033[38;5;244m" if enabled else ""
self.bg_grey = "\033[48;5;235m" if enabled else "" self.bg_grey = "\033[48;5;235m" if enabled else ""
@@ -279,6 +256,8 @@ def file_supports_unicode(file_obj: IO[AnyStr]) -> bool:
return "utf" in codec_info.name return "utf" in codec_info.name
# Global instance of the Logger. """
Global instance of the Logger.
"""
# (there's only one stdout per-process, so a global instance is justified) # (there's only one stdout per-process, so a global instance is justified)
log = Logger() log = Logger()
+277 -447
View File
@@ -1,56 +1,49 @@
from __future__ import annotations
import functools
import os import os
import platform import platform
import re import re
import shlex
import shutil import shutil
import subprocess import subprocess
import sys import sys
import typing import tempfile
from collections.abc import Sequence, Set
from dataclasses import dataclass
from pathlib import Path from pathlib import Path
from typing import Literal, Tuple from typing import Any, Dict, List, NamedTuple, Optional, Sequence, Set, Tuple, cast
from filelock import FileLock
from packaging.version import Version
from . import errors
from ._compat.typing import assert_never
from .architecture import Architecture from .architecture import Architecture
from .environment import ParsedEnvironment from .environment import ParsedEnvironment
from .logger import log from .logger import log
from .options import Options from .typing import Literal, PathOrStr
from .typing import PathOrStr
from .util import ( from .util import (
CIBW_CACHE_PATH, BuildOptions,
BuildFrontendConfig,
BuildFrontendName,
BuildSelector, BuildSelector,
call, NonPlatformWheelError,
combine_constraints,
detect_ci_provider,
download, download,
find_compatible_wheel,
find_uv,
free_thread_enable_313,
get_build_verbosity_extra_flags, get_build_verbosity_extra_flags,
get_pip_version, get_pip_script,
install_certifi_script, install_certifi_script,
move_file,
prepare_command, prepare_command,
read_python_configs, read_python_configs,
shell, resources_dir,
split_config_settings,
test_fail_cwd_file,
unwrap, unwrap,
virtualenv,
) )
@functools.lru_cache(maxsize=None) def call(
def get_macos_version() -> tuple[int, int]: args: Sequence[PathOrStr],
env: Optional[Dict[str, str]] = None,
cwd: Optional[str] = None,
shell: bool = False,
) -> None:
# print the command executing for the logs
if shell:
print(f"+ {args}")
else:
print("+ " + " ".join(shlex.quote(str(a)) for a in args))
subprocess.run(args, env=env, cwd=cwd, shell=shell, check=True)
def get_macos_version() -> Tuple[int, int]:
""" """
Returns the macOS major/minor version, as a tuple, e.g. (10, 15) or (11, 0) Returns the macOS major/minor version, as a tuple, e.g. (10, 15) or (11, 0)
@@ -61,36 +54,21 @@ def get_macos_version() -> tuple[int, int]:
""" """
version_str, _, _ = platform.mac_ver() version_str, _, _ = platform.mac_ver()
version = tuple(map(int, version_str.split(".")[:2])) version = tuple(map(int, version_str.split(".")[:2]))
if (10, 15) < version < (11, 0): return cast(Tuple[int, int], version)
# When built against an older macOS SDK, Python will report macOS 10.16
# instead of the real version.
version_str = call(
sys.executable,
"-sS",
"-c",
"import platform; print(platform.mac_ver()[0])",
env={"SYSTEM_VERSION_COMPAT": "0"},
capture_stdout=True,
)
version = tuple(map(int, version_str.split(".")[:2]))
return typing.cast(Tuple[int, int], version)
@functools.lru_cache(maxsize=None) def get_macos_sdks() -> List[str]:
def get_test_macosx_deployment_target() -> str: output = subprocess.run(
version = get_macos_version() ["xcodebuild", "-showsdks"],
if version >= (11, 0): universal_newlines=True,
return f"{version[0]}.0" check=True,
return f"{version[0]}.{version[1]}" stdout=subprocess.PIPE,
).stdout
def get_macos_sdks() -> list[str]:
output = call("xcodebuild", "-showsdks", capture_stdout=True)
return [m.group(1) for m in re.finditer(r"-sdk (macosx\S+)", output)] return [m.group(1) for m in re.finditer(r"-sdk (macosx\S+)", output)]
@dataclass(frozen=True) class PythonConfiguration(NamedTuple):
class PythonConfiguration:
version: str version: str
identifier: str identifier: str
url: str url: str
@@ -98,7 +76,8 @@ class PythonConfiguration:
def get_python_configurations( def get_python_configurations(
build_selector: BuildSelector, architectures: Set[Architecture] build_selector: BuildSelector, architectures: Set[Architecture]
) -> list[PythonConfiguration]: ) -> List[PythonConfiguration]:
full_python_configs = read_python_configs("macos") full_python_configs = read_python_configs("macos")
python_configurations = [PythonConfiguration(**item) for item in full_python_configs] python_configurations = [PythonConfiguration(**item) for item in full_python_configs]
@@ -113,130 +92,152 @@ def get_python_configurations(
# skip builds as required by BUILD/SKIP # skip builds as required by BUILD/SKIP
python_configurations = [c for c in python_configurations if build_selector(c.identifier)] python_configurations = [c for c in python_configurations if build_selector(c.identifier)]
# filter-out some cross-compilation configs with PyPy: # When running on macOS 11 and x86_64, the reported OS is '10.16', but
# can't build arm64 on x86_64 # there is no such OS - it really means macOS 11.
# rosetta allows to build x86_64 on arm64 if get_macos_version() >= (10, 16):
if platform.machine() == "x86_64": if any(c.identifier.startswith("pp") for c in python_configurations):
python_configurations_before = set(python_configurations) # pypy doesn't work on macOS 11 yet
python_configurations = [ # See https://foss.heptapod.net/pypy/pypy/-/issues/3314
c log.warning(
for c in python_configurations
if not (c.identifier.startswith("pp") and c.identifier.endswith("arm64"))
]
removed_elements = python_configurations_before - set(python_configurations)
if removed_elements:
ids = ", ".join(c.identifier for c in removed_elements)
log.quiet(
unwrap( unwrap(
f""" """
Note: {ids} {'was' if len(removed_elements) == 1 else 'were'} PyPy is currently unsupported when building on macOS 11. To build macOS PyPy
selected, but can't be built on x86_64 so will be skipped automatically. wheels, build on an older OS, such as macOS 10.15. To silence this warning,
deselect PyPy by adding "pp*-macosx*" to your CIBW_SKIP option.
""" """
) )
) )
python_configurations = [
c for c in python_configurations if not c.identifier.startswith("pp")
]
if any(c.identifier.startswith("cp35") for c in python_configurations):
# CPython 3.5 doesn't work on macOS 11
log.warning(
unwrap(
"""
CPython 3.5 is unsupported when building on macOS 11. To build CPython 3.5
wheels, build on an older OS, such as macOS 10.15. To silence this warning,
deselect CPython 3.5 by adding "cp35-macosx_x86_64" to your CIBW_SKIP option.
"""
)
)
python_configurations = [
c for c in python_configurations if not c.identifier.startswith("cp35")
]
return python_configurations return python_configurations
def install_cpython(tmp: Path, version: str, url: str, free_threading: bool) -> Path: SYMLINKS_DIR = Path("/tmp/cibw_bin")
ft = "T" if free_threading else ""
installation_path = Path(f"/Library/Frameworks/Python{ft}.framework/Versions/{version}")
with FileLock(CIBW_CACHE_PATH / f"cpython{version}.lock"): def make_symlinks(installation_bin_path: Path, python_executable: str, pip_executable: str) -> None:
installed_system_packages = call("pkgutil", "--pkgs", capture_stdout=True).splitlines() assert (installation_bin_path / python_executable).exists()
# if this version of python isn't installed, get it from python.org and install
python_package_identifier = f"org.python.Python.Python{ft}Framework-{version}" # Python bin folders on Mac don't symlink `python3` to `python`, and neither
if python_package_identifier not in installed_system_packages: # does PyPy for `pypy` or `pypy3`, so we do that so `python` and `pip` always
if detect_ci_provider() is None: # point to the active configuration.
# if running locally, we don't want to install CPython with sudo if SYMLINKS_DIR.exists():
# let the user know & provide a link to the installer shutil.rmtree(SYMLINKS_DIR)
msg = ( SYMLINKS_DIR.mkdir(parents=True)
f"Error: CPython {version} is not installed.\n"
"cibuildwheel will not perform system-wide installs when running outside of CI.\n" (SYMLINKS_DIR / "python").symlink_to(installation_bin_path / python_executable)
f"To build locally, install CPython {version} on this machine, or, disable this version of Python using CIBW_SKIP=cp{version.replace('.', '')}-macosx_*\n" (SYMLINKS_DIR / "python-config").symlink_to(
f"\nDownload link: {url}" installation_bin_path / (python_executable + "-config")
) )
raise errors.FatalError(msg) (SYMLINKS_DIR / "pip").symlink_to(installation_bin_path / pip_executable)
pkg_path = tmp / "Python.pkg"
def install_cpython(version: str, url: str) -> Path:
installed_system_packages = subprocess.run(
["pkgutil", "--pkgs"], universal_newlines=True, check=True, stdout=subprocess.PIPE
).stdout.splitlines()
# if this version of python isn't installed, get it from python.org and install
python_package_identifier = f"org.python.Python.PythonFramework-{version}"
python_executable = "python3" if version[0] == "3" else "python"
installation_bin_path = Path(f"/Library/Frameworks/Python.framework/Versions/{version}/bin")
if python_package_identifier not in installed_system_packages:
# download the pkg # download the pkg
download(url, pkg_path) download(url, Path("/tmp/Python.pkg"))
# install # install
args = [] call(["sudo", "installer", "-pkg", "/tmp/Python.pkg", "-target", "/"])
if version.startswith("3.13"): # patch open ssl
# Python 3.13 is the first version to have a free-threading option if version == "3.5":
args += ["-applyChoiceChangesXML", str(free_thread_enable_313.resolve())] open_ssl_patch_url = f"https://github.com/mayeut/patch-macos-python-openssl/releases/download/v1.1.1h/patch-macos-python-{version}-openssl-v1.1.1h.tar.gz"
call("sudo", "installer", "-pkg", pkg_path, *args, "-target", "/") download(open_ssl_patch_url, Path("/tmp/python-patch.tar.gz"))
pkg_path.unlink() call(
env = os.environ.copy() [
env["PIP_DISABLE_PIP_VERSION_CHECK"] = "1" "sudo",
"tar",
"-C",
f"/Library/Frameworks/Python.framework/Versions/{version}/",
"-xmf",
"/tmp/python-patch.tar.gz",
]
)
if free_threading: call(["sudo", str(installation_bin_path / python_executable), str(install_certifi_script)])
call(installation_path / f"bin/python{version}t", "-m", "ensurepip", env=env)
call(installation_path / f"bin/python{version}t", install_certifi_script, env=env)
else:
call(installation_path / "bin/python3", install_certifi_script, env=env)
return installation_path / "bin" / (f"python{version}t" if free_threading else "python3") pip_executable = "pip3" if version[0] == "3" else "pip"
make_symlinks(installation_bin_path, python_executable, pip_executable)
return installation_bin_path
def install_pypy(tmp: Path, url: str) -> Path: def install_pypy(version: str, url: str) -> Path:
pypy_tar_bz2 = url.rsplit("/", 1)[-1] pypy_tar_bz2 = url.rsplit("/", 1)[-1]
extension = ".tar.bz2" extension = ".tar.bz2"
assert pypy_tar_bz2.endswith(extension) assert pypy_tar_bz2.endswith(extension)
installation_path = CIBW_CACHE_PATH / pypy_tar_bz2[: -len(extension)] pypy_base_filename = pypy_tar_bz2[: -len(extension)]
with FileLock(str(installation_path) + ".lock"): installation_path = Path("/tmp") / pypy_base_filename
if not installation_path.exists(): if not installation_path.exists():
downloaded_tar_bz2 = tmp / pypy_tar_bz2 downloaded_tar_bz2 = Path("/tmp") / pypy_tar_bz2
download(url, downloaded_tar_bz2) download(url, downloaded_tar_bz2)
installation_path.parent.mkdir(parents=True, exist_ok=True) call(["tar", "-C", "/tmp", "-xf", downloaded_tar_bz2])
call("tar", "-C", installation_path.parent, "-xf", downloaded_tar_bz2) # Patch PyPy to make sure headers get installed into a venv
downloaded_tar_bz2.unlink() patch_version = "_27" if version == "2.7" else ""
return installation_path / "bin" / "pypy3" patch_path = resources_dir / f"pypy_venv{patch_version}.patch"
call(["patch", "--force", "-p1", "-d", installation_path, "-i", patch_path])
installation_bin_path = installation_path / "bin"
python_executable = "pypy3" if version[0] == "3" else "pypy"
pip_executable = "pip3" if version[0] == "3" else "pip"
make_symlinks(installation_bin_path, python_executable, pip_executable)
return installation_bin_path
def setup_python( def setup_python(
tmp: Path,
python_configuration: PythonConfiguration, python_configuration: PythonConfiguration,
dependency_constraint_flags: Sequence[PathOrStr], dependency_constraint_flags: Sequence[PathOrStr],
environment: ParsedEnvironment, environment: ParsedEnvironment,
build_frontend: BuildFrontendName, ) -> Dict[str, str]:
) -> tuple[Path, dict[str, str]]:
if build_frontend == "build[uv]" and Version(python_configuration.version) < Version("3.8"):
build_frontend = "build"
uv_path = find_uv()
use_uv = build_frontend == "build[uv]" and Version(python_configuration.version) >= Version(
"3.8"
)
tmp.mkdir()
implementation_id = python_configuration.identifier.split("-")[0] implementation_id = python_configuration.identifier.split("-")[0]
log.step(f"Installing Python {implementation_id}...") log.step(f"Installing Python {implementation_id}...")
if implementation_id.startswith("cp"):
free_threading = "t-macos" in python_configuration.identifier
base_python = install_cpython(
tmp, python_configuration.version, python_configuration.url, free_threading
)
if implementation_id.startswith("cp"):
installation_bin_path = install_cpython(
python_configuration.version, python_configuration.url
)
elif implementation_id.startswith("pp"): elif implementation_id.startswith("pp"):
base_python = install_pypy(tmp, python_configuration.url) installation_bin_path = install_pypy(python_configuration.version, python_configuration.url)
else: else:
msg = "Unknown Python implementation" raise ValueError("Unknown Python implementation")
raise ValueError(msg)
assert (
base_python.exists()
), f"{base_python.name} not found, has {list(base_python.parent.iterdir())}"
log.step("Setting up build environment...") log.step("Setting up build environment...")
venv_path = tmp / "venv"
env = virtualenv( env = os.environ.copy()
python_configuration.version, env["PATH"] = os.pathsep.join(
base_python, [
venv_path, str(SYMLINKS_DIR),
dependency_constraint_flags, str(installation_bin_path),
use_uv=use_uv, env["PATH"],
]
) )
venv_bin_path = venv_path / "bin"
assert venv_bin_path.exists()
# Fix issue with site.py setting the wrong `sys.prefix`, `sys.exec_prefix`, # Fix issue with site.py setting the wrong `sys.prefix`, `sys.exec_prefix`,
# `sys.path`, ... for PyPy: https://foss.heptapod.net/pypy/pypy/issues/3175 # `sys.path`, ... for PyPy: https://foss.heptapod.net/pypy/pypy/issues/3175
# Also fix an issue with the shebang of installed scripts inside the # Also fix an issue with the shebang of installed scripts inside the
@@ -244,77 +245,55 @@ def setup_python(
# https://github.com/pypa/virtualenv/issues/620 # https://github.com/pypa/virtualenv/issues/620
# Also see https://github.com/python/cpython/pull/9516 # Also see https://github.com/python/cpython/pull/9516
env.pop("__PYVENV_LAUNCHER__", None) env.pop("__PYVENV_LAUNCHER__", None)
env = environment.as_dictionary(prev_environment=env)
# we version pip ourselves, so we don't care about pip version checking # we version pip ourselves, so we don't care about pip version checking
env["PIP_DISABLE_PIP_VERSION_CHECK"] = "1" env["PIP_DISABLE_PIP_VERSION_CHECK"] = "1"
# upgrade pip to the version matching our constraints # check what version we're on
# if necessary, reinstall it to ensure that it's available on PATH as 'pip' call(["which", "python"], env=env)
if build_frontend == "build[uv]": call(["python", "--version"], env=env)
assert uv_path is not None which_python = subprocess.run(
pip = [str(uv_path), "pip"] ["which", "python"], env=env, universal_newlines=True, check=True, stdout=subprocess.PIPE
else: ).stdout.strip()
pip = ["python", "-m", "pip"] if which_python != "/tmp/cibw_bin/python":
print(
if not use_uv: "cibuildwheel: python available on PATH doesn't match our installed instance. If you have modified PATH, ensure that you don't overwrite cibuildwheel's entry or insert python above it.",
call( file=sys.stderr,
*pip,
"install",
"--upgrade",
"pip",
*dependency_constraint_flags,
env=env,
cwd=venv_path,
) )
sys.exit(1)
# Apply our environment after pip is ready # install pip & wheel
env = environment.as_dictionary(prev_environment=env) call(["python", get_pip_script, *dependency_constraint_flags], env=env, cwd="/tmp")
assert (installation_bin_path / "pip").exists()
call(["which", "pip"], env=env)
call(["pip", "--version"], env=env)
which_pip = subprocess.run(
["which", "pip"], env=env, universal_newlines=True, check=True, stdout=subprocess.PIPE
).stdout.strip()
if which_pip != "/tmp/cibw_bin/pip":
print(
"cibuildwheel: pip available on PATH doesn't match our installed instance. If you have modified PATH, ensure that you don't overwrite cibuildwheel's entry or insert pip above it.",
file=sys.stderr,
)
sys.exit(1)
# check what pip version we're on # Set MACOSX_DEPLOYMENT_TARGET to 10.9, if the user didn't set it.
if not use_uv: # CPython 3.5 defaults to 10.6, and pypy defaults to 10.7, causing
assert (venv_bin_path / "pip").exists() # inconsistencies if it's left unset.
call("which", "pip", env=env) env.setdefault("MACOSX_DEPLOYMENT_TARGET", "10.9")
call("pip", "--version", env=env)
which_pip = call("which", "pip", env=env, capture_stdout=True).strip()
if which_pip != str(venv_bin_path / "pip"):
msg = "cibuildwheel: pip available on PATH doesn't match our installed instance. If you have modified PATH, ensure that you don't overwrite cibuildwheel's entry or insert pip above it."
raise errors.FatalError(msg)
# check what Python version we're on
call("which", "python", env=env)
call("python", "--version", env=env)
which_python = call("which", "python", env=env, capture_stdout=True).strip()
if which_python != str(venv_bin_path / "python"):
msg = "cibuildwheel: python available on PATH doesn't match our installed instance. If you have modified PATH, ensure that you don't overwrite cibuildwheel's entry or insert python above it."
raise errors.FatalError(msg)
config_is_arm64 = python_configuration.identifier.endswith("arm64") config_is_arm64 = python_configuration.identifier.endswith("arm64")
config_is_universal2 = python_configuration.identifier.endswith("universal2") config_is_universal2 = python_configuration.identifier.endswith("universal2")
# Set MACOSX_DEPLOYMENT_TARGET, if the user didn't set it. if python_configuration.version == "3.5":
# For arm64, the minimal deployment target is 11.0. # Cross-compilation platform override - CPython 3.5 has an
# On x86_64 (or universal2), use 10.9 as a default. # i386/x86_64 version of Python, but we only want a x64_64 build
# CPython 3.13 needs 10.13. env.setdefault("_PYTHON_HOST_PLATFORM", "macosx-10.9-x86_64")
if config_is_arm64: # https://github.com/python/cpython/blob/a5ed2fe0eedefa1649aa93ee74a0bafc8e628a10/Lib/_osx_support.py#L260
default_target = "11.0" env.setdefault("ARCHFLAGS", "-arch x86_64")
elif Version(python_configuration.version) >= Version("3.13"):
default_target = "10.13"
elif python_configuration.identifier.startswith("pp") and Version(
python_configuration.version
) >= Version("3.9"):
default_target = "10.15"
else:
default_target = "10.9"
env.setdefault("MACOSX_DEPLOYMENT_TARGET", default_target)
# This is a floor, it can't be set lower than the default_target. elif python_configuration.version not in {"3.6", "3.7"}:
if Version(env["MACOSX_DEPLOYMENT_TARGET"]) < Version(default_target):
log.warning(
f"Bumping MACOSX_DEPLOYMENT_TARGET ({env['MACOSX_DEPLOYMENT_TARGET']}) to the minimum required ({default_target})."
)
env["MACOSX_DEPLOYMENT_TARGET"] = default_target
if python_configuration.version not in {"3.6", "3.7"}:
if config_is_arm64: if config_is_arm64:
# macOS 11 is the first OS with arm64 support, so the wheels # macOS 11 is the first OS with arm64 support, so the wheels
# have that as a minimum. # have that as a minimum.
@@ -352,180 +331,93 @@ def setup_python(
env.setdefault("SDKROOT", arm64_compatible_sdks[0]) env.setdefault("SDKROOT", arm64_compatible_sdks[0])
log.step("Installing build tools...") log.step("Installing build tools...")
if build_frontend == "pip":
call( call(
[
"pip", "pip",
"install", "install",
"--upgrade", "--upgrade",
"setuptools",
"wheel",
"delocate", "delocate",
*dependency_constraint_flags, *dependency_constraint_flags,
],
env=env, env=env,
) )
elif build_frontend == "build":
call(
"pip",
"install",
"--upgrade",
"delocate",
"build[virtualenv]",
*dependency_constraint_flags,
env=env,
)
elif build_frontend == "build[uv]":
assert uv_path is not None
call(
uv_path,
"pip",
"install",
"--upgrade",
"delocate",
"build[virtualenv, uv]",
*dependency_constraint_flags,
env=env,
)
else:
assert_never(build_frontend)
return base_python, env return env
def build(options: Options, tmp_path: Path) -> None: def build(options: BuildOptions) -> None:
python_configurations = get_python_configurations( temp_dir = Path(tempfile.mkdtemp(prefix="cibuildwheel"))
options.globals.build_selector, options.globals.architectures built_wheel_dir = temp_dir / "built_wheel"
) repaired_wheel_dir = temp_dir / "repaired_wheel"
if not python_configurations:
return
try: try:
before_all_options_identifier = python_configurations[0].identifier if options.before_all:
before_all_options = options.build_options(before_all_options_identifier)
if before_all_options.before_all:
log.step("Running before_all...") log.step("Running before_all...")
env = before_all_options.environment.as_dictionary(prev_environment=os.environ) env = options.environment.as_dictionary(prev_environment=os.environ)
env.setdefault("MACOSX_DEPLOYMENT_TARGET", "10.9") env.setdefault("MACOSX_DEPLOYMENT_TARGET", "10.9")
before_all_prepared = prepare_command( before_all_prepared = prepare_command(
before_all_options.before_all, project=".", package=before_all_options.package_dir options.before_all, project=".", package=options.package_dir
) )
shell(before_all_prepared, env=env) call([before_all_prepared], shell=True, env=env)
built_wheels: list[Path] = [] python_configurations = get_python_configurations(
options.build_selector, options.architectures
)
for config in python_configurations: for config in python_configurations:
build_options = options.build_options(config.identifier)
build_frontend = build_options.build_frontend or BuildFrontendConfig("pip")
use_uv = build_frontend.name == "build[uv]" and Version(config.version) >= Version(
"3.8"
)
uv_path = find_uv()
if use_uv and uv_path is None:
msg = "uv not found"
raise AssertionError(msg)
pip = ["pip"] if not use_uv else [str(uv_path), "pip"]
log.build_start(config.identifier) log.build_start(config.identifier)
identifier_tmp_dir = tmp_path / config.identifier
identifier_tmp_dir.mkdir()
built_wheel_dir = identifier_tmp_dir / "built_wheel"
repaired_wheel_dir = identifier_tmp_dir / "repaired_wheel"
config_is_arm64 = config.identifier.endswith("arm64") config_is_arm64 = config.identifier.endswith("arm64")
config_is_universal2 = config.identifier.endswith("universal2") config_is_universal2 = config.identifier.endswith("universal2")
dependency_constraint_flags: Sequence[PathOrStr] = [] dependency_constraint_flags: Sequence[PathOrStr] = []
if build_options.dependency_constraints: if options.dependency_constraints:
dependency_constraint_flags = [ dependency_constraint_flags = [
"-c", "-c",
build_options.dependency_constraints.get_for_python_version(config.version), options.dependency_constraints.get_for_python_version(config.version),
] ]
base_python, env = setup_python( env = setup_python(config, dependency_constraint_flags, options.environment)
identifier_tmp_dir / "build",
config,
dependency_constraint_flags,
build_options.environment,
build_frontend.name,
)
if not use_uv:
pip_version = get_pip_version(env)
compatible_wheel = find_compatible_wheel(built_wheels, config.identifier) if options.before_build:
if compatible_wheel:
log.step_end()
print(
f"\nFound previously built wheel {compatible_wheel.name}, that's compatible with {config.identifier}. Skipping build step..."
)
repaired_wheel = compatible_wheel
else:
if build_options.before_build:
log.step("Running before_build...") log.step("Running before_build...")
before_build_prepared = prepare_command( before_build_prepared = prepare_command(
build_options.before_build, project=".", package=build_options.package_dir options.before_build, project=".", package=options.package_dir
) )
shell(before_build_prepared, env=env) call(before_build_prepared, env=env, shell=True)
log.step("Building wheel...") log.step("Building wheel...")
built_wheel_dir.mkdir() if built_wheel_dir.exists():
shutil.rmtree(built_wheel_dir)
built_wheel_dir.mkdir(parents=True)
extra_flags = split_config_settings(
build_options.config_settings, build_frontend.name
)
extra_flags += build_frontend.args
build_env = env.copy()
if not use_uv:
build_env["VIRTUALENV_PIP"] = pip_version
if build_options.dependency_constraints:
constraint_path = build_options.dependency_constraints.get_for_python_version(
config.version
)
combine_constraints(
build_env, constraint_path, identifier_tmp_dir if use_uv else None
)
if build_frontend.name == "pip":
extra_flags += get_build_verbosity_extra_flags(build_options.build_verbosity)
# Path.resolve() is needed. Without it pip wheel may try to fetch package from pypi.org # Path.resolve() is needed. Without it pip wheel may try to fetch package from pypi.org
# see https://github.com/pypa/cibuildwheel/pull/369 # see https://github.com/pypa/cibuildwheel/pull/369
call( call(
"python", [
"-m",
"pip", "pip",
"wheel", "wheel",
build_options.package_dir.resolve(), options.package_dir.resolve(),
f"--wheel-dir={built_wheel_dir}", "--wheel-dir",
built_wheel_dir,
"--no-deps", "--no-deps",
*extra_flags, *get_build_verbosity_extra_flags(options.build_verbosity),
env=build_env, ],
env=env,
) )
elif build_frontend.name == "build" or build_frontend.name == "build[uv]":
if not 0 <= build_options.build_verbosity < 2:
msg = f"build_verbosity {build_options.build_verbosity} is not supported for build frontend. Ignoring."
log.warning(msg)
if use_uv and "--no-isolation" not in extra_flags and "-n" not in extra_flags:
extra_flags.append("--installer=uv")
call(
"python",
"-m",
"build",
build_options.package_dir,
"--wheel",
f"--outdir={built_wheel_dir}",
*extra_flags,
env=build_env,
)
else:
assert_never(build_frontend)
built_wheel = next(built_wheel_dir.glob("*.whl")) built_wheel = next(built_wheel_dir.glob("*.whl"))
repaired_wheel_dir.mkdir() if repaired_wheel_dir.exists():
shutil.rmtree(repaired_wheel_dir)
repaired_wheel_dir.mkdir(parents=True)
if built_wheel.name.endswith("none-any.whl"): if built_wheel.name.endswith("none-any.whl"):
raise errors.NonPlatformWheelError() raise NonPlatformWheelError()
if build_options.repair_command: if options.repair_command:
log.step("Repairing wheel...") log.step("Repairing wheel...")
if config_is_universal2: if config_is_universal2:
@@ -536,32 +428,22 @@ def build(options: Options, tmp_path: Path) -> None:
delocate_archs = "x86_64" delocate_archs = "x86_64"
repair_command_prepared = prepare_command( repair_command_prepared = prepare_command(
build_options.repair_command, options.repair_command,
wheel=built_wheel, wheel=built_wheel,
dest_dir=repaired_wheel_dir, dest_dir=repaired_wheel_dir,
delocate_archs=delocate_archs, delocate_archs=delocate_archs,
) )
shell(repair_command_prepared, env=env) call(repair_command_prepared, env=env, shell=True)
else: else:
shutil.move(str(built_wheel), repaired_wheel_dir) shutil.move(str(built_wheel), repaired_wheel_dir)
repaired_wheel = next(repaired_wheel_dir.glob("*.whl")) repaired_wheel = next(repaired_wheel_dir.glob("*.whl"))
if repaired_wheel.name in {wheel.name for wheel in built_wheels}:
raise errors.AlreadyBuiltWheelError(repaired_wheel.name)
log.step_end() log.step_end()
if build_options.test_command and build_options.test_selector(config.identifier): if options.test_command and options.test_selector(config.identifier):
machine_arch = platform.machine() machine_arch = platform.machine()
python_arch = call( testing_archs: List[Literal["x86_64", "arm64"]] = []
"python",
"-sSc",
"import platform; print(platform.machine())",
env=env,
capture_stdout=True,
).strip()
testing_archs: list[Literal["x86_64", "arm64"]]
if config_is_arm64: if config_is_arm64:
testing_archs = ["arm64"] testing_archs = ["arm64"]
@@ -573,7 +455,7 @@ def build(options: Options, tmp_path: Path) -> None:
for testing_arch in testing_archs: for testing_arch in testing_archs:
if config_is_universal2: if config_is_universal2:
arch_specific_identifier = f"{config.identifier}:{testing_arch}" arch_specific_identifier = f"{config.identifier}:{testing_arch}"
if not build_options.test_selector(arch_specific_identifier): if not options.test_selector(arch_specific_identifier):
continue continue
if machine_arch == "x86_64" and testing_arch == "arm64": if machine_arch == "x86_64" and testing_arch == "arm64":
@@ -582,9 +464,10 @@ def build(options: Options, tmp_path: Path) -> None:
unwrap( unwrap(
""" """
While arm64 wheels can be built on x86_64, they cannot be While arm64 wheels can be built on x86_64, they cannot be
tested. Consider building arm64 wheels natively, if your CI tested. The ability to test the arm64 wheels will be added in a
provider offers this. To silence this warning, set future release of cibuildwheel, once Apple Silicon CI runners
`CIBW_TEST_SKIP: "*-macosx_arm64"`. are widely available. To silence this warning, set
`CIBW_TEST_SKIP: *-macosx_arm64`.
""" """
) )
) )
@@ -593,35 +476,16 @@ def build(options: Options, tmp_path: Path) -> None:
unwrap( unwrap(
""" """
While universal2 wheels can be built on x86_64, the arm64 part While universal2 wheels can be built on x86_64, the arm64 part
of the wheel cannot be tested on x86_64. Consider building of them cannot currently be tested. The ability to test the
universal2 wheels on an arm64 runner, if your CI provider offers arm64 part of a universal2 wheel will be added in a future
this. Notably, an arm64 runner can also test the x86_64 part of release of cibuildwheel, once Apple Silicon CI runners are
the wheel, through Rosetta emulation. To silence this warning, widely available. To silence this warning, set
set `CIBW_TEST_SKIP: "*-macosx_universal2:arm64"`. `CIBW_TEST_SKIP: *-macosx_universal2:arm64`.
""" """
) )
) )
else: else:
msg = "unreachable" raise RuntimeError("unreachable")
raise RuntimeError(msg)
# skip this test
continue
is_cp38 = config.identifier.startswith("cp38-")
if testing_arch == "arm64" and is_cp38 and python_arch != "arm64":
log.warning(
unwrap(
"""
While cibuildwheel can build CPython 3.8 universal2/arm64 wheels, we
cannot test the arm64 part of them, even when running on an Apple
Silicon machine. This is because we use the x86_64 installer of
CPython 3.8. See the discussion in
https://github.com/pypa/cibuildwheel/pull/1169 for the details. To
silence this warning, set `CIBW_TEST_SKIP: "cp38-macosx_*:arm64"`.
"""
)
)
# skip this test # skip this test
continue continue
@@ -634,119 +498,85 @@ def build(options: Options, tmp_path: Path) -> None:
# set up a virtual environment to install and test from, to make sure # set up a virtual environment to install and test from, to make sure
# there are no dependencies that were pulled in at build time. # there are no dependencies that were pulled in at build time.
if not use_uv: call(["pip", "install", "virtualenv", *dependency_constraint_flags], env=env)
call("pip", "install", "virtualenv", *dependency_constraint_flags, env=env) venv_dir = Path(tempfile.mkdtemp())
venv_dir = identifier_tmp_dir / f"venv-test-{testing_arch}"
arch_prefix = [] arch_prefix = []
uv_arch_args = []
if testing_arch != machine_arch: if testing_arch != machine_arch:
if machine_arch == "arm64" and testing_arch == "x86_64": if machine_arch == "arm64" and testing_arch == "x86_64":
# rosetta2 will provide the emulation with just the arch prefix. # rosetta2 will provide the emulation with just the arch prefix.
arch_prefix = ["arch", "-x86_64"] arch_prefix = ["arch", "-x86_64"]
uv_arch_args = ["--python-platform", "x86_64-apple-darwin"]
else: else:
msg = f"don't know how to emulate {testing_arch} on {machine_arch}" raise RuntimeError(
raise RuntimeError(msg) "don't know how to emulate {testing_arch} on {machine_arch}"
)
# define a custom 'call' function that adds the arch prefix each time # define a custom 'call' function that adds the arch prefix each time
call_with_arch = functools.partial(call, *arch_prefix) def call_with_arch(args: Sequence[PathOrStr], **kwargs: Any) -> None:
shell_with_arch = functools.partial(call, *arch_prefix, "/bin/sh", "-c") if isinstance(args, str):
args = " ".join(arch_prefix) + " " + args
if use_uv:
pip_install = functools.partial(call, *pip, "install", *uv_arch_args)
call("uv", "venv", venv_dir, f"--python={base_python}", env=env)
else: else:
pip_install = functools.partial(call_with_arch, *pip, "install") args = [*arch_prefix, *args]
# Use pip version from the initial env to ensure determinism call(args, **kwargs)
venv_args = ["--no-periodic-update", f"--pip={pip_version}"]
# In Python<3.12, setuptools & wheel are installed as well, use virtualenv embedded ones # Use --no-download to ensure determinism by using seed libraries
if Version(config.version) < Version("3.12"): # built into virtualenv
venv_args.extend(("--setuptools=embed", "--wheel=embed")) call_with_arch(
call_with_arch("python", "-m", "virtualenv", *venv_args, venv_dir, env=env) ["python", "-m", "virtualenv", "--no-download", venv_dir], env=env
)
virtualenv_env = env.copy() virtualenv_env = env.copy()
virtualenv_env["MACOSX_DEPLOYMENT_TARGET"] = get_test_macosx_deployment_target()
virtualenv_env["PATH"] = os.pathsep.join( virtualenv_env["PATH"] = os.pathsep.join(
[ [
str(venv_dir / "bin"), str(venv_dir / "bin"),
virtualenv_env["PATH"], virtualenv_env["PATH"],
] ]
) )
virtualenv_env["VIRTUAL_ENV"] = str(venv_dir)
# check that we are using the Python from the virtual environment # check that we are using the Python from the virtual environment
call_with_arch("which", "python", env=virtualenv_env) call_with_arch(["which", "python"], env=virtualenv_env)
if build_options.before_test: if options.before_test:
before_test_prepared = prepare_command( before_test_prepared = prepare_command(
build_options.before_test, options.before_test, project=".", package=options.package_dir
project=".",
package=build_options.package_dir,
) )
shell_with_arch(before_test_prepared, env=virtualenv_env) call_with_arch(before_test_prepared, env=virtualenv_env, shell=True)
# install the wheel # install the wheel
if is_cp38 and python_arch == "x86_64": call_with_arch(
virtualenv_env_install_wheel = virtualenv_env.copy() ["pip", "install", f"{repaired_wheel}{options.test_extras}"],
virtualenv_env_install_wheel["SYSTEM_VERSION_COMPAT"] = "0" env=virtualenv_env,
log.notice(
unwrap(
"""
Setting SYSTEM_VERSION_COMPAT=0 to ensure CPython 3.8 can get
correct macOS version and allow installation of wheels with
MACOSX_DEPLOYMENT_TARGET >= 11.0.
See https://github.com/pypa/cibuildwheel/issues/1767 for the
details.
"""
)
)
else:
virtualenv_env_install_wheel = virtualenv_env
pip_install(
f"{repaired_wheel}{build_options.test_extras}",
env=virtualenv_env_install_wheel,
) )
# test the wheel # test the wheel
if build_options.test_requires: if options.test_requires:
pip_install( call_with_arch(
*build_options.test_requires, ["pip", "install"] + options.test_requires, env=virtualenv_env
env=virtualenv_env_install_wheel,
) )
# run the tests from a temp dir, with an absolute path in the command # run the tests from $HOME, with an absolute path in the command
# (this ensures that Python runs the tests against the installed wheel # (this ensures that Python runs the tests against the installed wheel
# and not the repo code) # and not the repo code)
test_command_prepared = prepare_command( test_command_prepared = prepare_command(
build_options.test_command, options.test_command,
project=Path(".").resolve(), project=Path(".").resolve(),
package=build_options.package_dir.resolve(), package=options.package_dir.resolve(),
wheel=repaired_wheel,
) )
call_with_arch(
test_cwd = identifier_tmp_dir / "test_cwd" test_command_prepared,
test_cwd.mkdir(exist_ok=True) cwd=os.environ["HOME"],
(test_cwd / "test_fail.py").write_text(test_fail_cwd_file.read_text()) env=virtualenv_env,
shell=True,
shell_with_arch(test_command_prepared, cwd=test_cwd, env=virtualenv_env)
# we're all done here; move it to output (overwrite existing)
if compatible_wheel is None:
output_wheel = build_options.output_dir.joinpath(repaired_wheel.name)
moved_wheel = move_file(repaired_wheel, output_wheel)
if moved_wheel != output_wheel.resolve():
log.warning(
"{repaired_wheel} was moved to {moved_wheel} instead of {output_wheel}"
) )
built_wheels.append(output_wheel)
# clean up # clean up
shutil.rmtree(identifier_tmp_dir) shutil.rmtree(venv_dir)
# we're all done here; move it to output (overwrite existing)
shutil.move(str(repaired_wheel), options.output_dir)
log.build_end() log.build_end()
except subprocess.CalledProcessError as error: except subprocess.CalledProcessError as error:
msg = f"Command {error.cmd} failed with code {error.returncode}. {error.stdout or ''}" log.step_end_with_error(
raise errors.FatalError(msg) from error f"Command {error.cmd} failed with code {error.returncode}. {error.stdout}"
)
sys.exit(1)
-425
View File
@@ -1,425 +0,0 @@
from __future__ import annotations
import io
import json
import os
import platform
import shlex
import shutil
import subprocess
import sys
import typing
import uuid
from collections.abc import Mapping, Sequence
from dataclasses import dataclass, field
from pathlib import Path, PurePath, PurePosixPath
from types import TracebackType
from typing import IO, Dict, Literal
from ._compat.typing import Self
from .typing import PathOrStr, PopenBytes
from .util import (
CIProvider,
call,
detect_ci_provider,
parse_key_value_string,
strtobool,
)
ContainerEngineName = Literal["docker", "podman"]
@dataclass(frozen=True)
class OCIContainerEngineConfig:
name: ContainerEngineName
create_args: tuple[str, ...] = field(default_factory=tuple)
disable_host_mount: bool = False
@staticmethod
def from_config_string(config_string: str) -> OCIContainerEngineConfig:
config_dict = parse_key_value_string(
config_string,
["name"],
["create_args", "create-args", "disable_host_mount", "disable-host-mount"],
)
name = " ".join(config_dict["name"])
if name not in {"docker", "podman"}:
msg = f"unknown container engine {name}"
raise ValueError(msg)
name = typing.cast(ContainerEngineName, name)
# some flexibility in the option names to cope with TOML conventions
create_args = config_dict.get("create_args") or config_dict.get("create-args") or []
disable_host_mount_options = (
config_dict.get("disable_host_mount") or config_dict.get("disable-host-mount") or []
)
disable_host_mount = (
strtobool(disable_host_mount_options[-1]) if disable_host_mount_options else False
)
return OCIContainerEngineConfig(
name=name, create_args=tuple(create_args), disable_host_mount=disable_host_mount
)
def options_summary(self) -> str | dict[str, str]:
if not self.create_args:
return self.name
else:
return {
"name": self.name,
"create_args": repr(self.create_args),
"disable_host_mount": str(self.disable_host_mount),
}
DEFAULT_ENGINE = OCIContainerEngineConfig("docker")
class OCIContainer:
"""
An object that represents a running OCI (e.g. Docker) container.
Intended for use as a context manager e.g.
`with OCIContainer(image = 'ubuntu') as docker:`
A bash shell is running in the remote container. When `call()` is invoked,
the command is relayed to the remote shell, and the results are streamed
back to cibuildwheel.
Example:
>>> from cibuildwheel.oci_container import * # NOQA
>>> from cibuildwheel.options import _get_pinned_container_images
>>> image = _get_pinned_container_images()['x86_64']['manylinux2014']
>>> # Test the default container
>>> with OCIContainer(image=image) as self:
... self.call(["echo", "hello world"])
... self.call(["cat", "/proc/1/cgroup"])
... print(self.get_environment())
... print(self.debug_info())
"""
UTILITY_PYTHON = "/opt/python/cp38-cp38/bin/python"
process: PopenBytes
bash_stdin: IO[bytes]
bash_stdout: IO[bytes]
def __init__(
self,
*,
image: str,
enforce_32_bit: bool = False,
cwd: PathOrStr | None = None,
engine: OCIContainerEngineConfig = DEFAULT_ENGINE,
):
if not image:
msg = "Must have a non-empty image to run."
raise ValueError(msg)
self.image = image
self.enforce_32_bit = enforce_32_bit
self.cwd = cwd
self.name: str | None = None
self.engine = engine
def __enter__(self) -> Self:
self.name = f"cibuildwheel-{uuid.uuid4()}"
# work-around for Travis-CI PPC64le Docker runs since 2021:
# this avoids network splits
# https://github.com/pypa/cibuildwheel/issues/904
# https://github.com/conda-forge/conda-smithy/pull/1520
network_args = []
if detect_ci_provider() == CIProvider.travis_ci and platform.machine() == "ppc64le":
network_args = ["--network=host"]
simulate_32_bit = False
if self.enforce_32_bit:
# If the architecture running the image is already the right one
# or the image entrypoint takes care of enforcing this, then we don't need to
# simulate this
container_machine = call(
self.engine.name, "run", "--rm", self.image, "uname", "-m", capture_stdout=True
).strip()
simulate_32_bit = container_machine != "i686"
shell_args = ["linux32", "/bin/bash"] if simulate_32_bit else ["/bin/bash"]
subprocess.run(
[
self.engine.name,
"create",
"--env=CIBUILDWHEEL",
"--env=SOURCE_DATE_EPOCH",
f"--name={self.name}",
"--interactive",
*(["--volume=/:/host"] if not self.engine.disable_host_mount else []),
*network_args,
*self.engine.create_args,
self.image,
*shell_args,
],
check=True,
)
self.process = subprocess.Popen(
[
self.engine.name,
"start",
"--attach",
"--interactive",
self.name,
],
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
)
assert self.process.stdin
assert self.process.stdout
self.bash_stdin = self.process.stdin
self.bash_stdout = self.process.stdout
# run a noop command to block until the container is responding
self.call(["/bin/true"], cwd="/")
if self.cwd:
# Although `docker create -w` does create the working dir if it
# does not exist, podman does not. There does not seem to be a way
# to setup a workdir for a container running in podman.
self.call(["mkdir", "-p", os.fspath(self.cwd)], cwd="/")
return self
def __exit__(
self,
exc_type: type[BaseException] | None,
exc_val: BaseException | None,
exc_tb: TracebackType | None,
) -> None:
self.bash_stdin.write(b"exit 0\n")
self.bash_stdin.flush()
self.process.wait(timeout=30)
self.bash_stdin.close()
self.bash_stdout.close()
if self.engine.name == "podman":
# This works around what seems to be a race condition in the podman
# backend. The full reason is not understood. See PR #966 for a
# discussion on possible causes and attempts to remove this line.
# For now, this seems to work "well enough".
self.process.wait()
assert isinstance(self.name, str)
keep_container = strtobool(os.environ.get("CIBW_DEBUG_KEEP_CONTAINER", ""))
if not keep_container:
subprocess.run(
[self.engine.name, "rm", "--force", "-v", self.name],
stdout=subprocess.DEVNULL,
check=False,
)
self.name = None
def copy_into(self, from_path: Path, to_path: PurePath) -> None:
# `docker cp` causes 'no space left on device' error when
# a container is running and the host filesystem is
# mounted. https://github.com/moby/moby/issues/38995
# Use `docker exec` instead.
if from_path.is_dir():
self.call(["mkdir", "-p", to_path])
subprocess.run(
f"tar cf - . | {self.engine.name} exec -i {self.name} tar --no-same-owner -xC {shell_quote(to_path)} -f -",
shell=True,
check=True,
cwd=from_path,
)
else:
exec_process: subprocess.Popen[bytes]
with subprocess.Popen(
[
self.engine.name,
"exec",
"-i",
str(self.name),
"sh",
"-c",
f"cat > {shell_quote(to_path)}",
],
stdin=subprocess.PIPE,
) as exec_process:
assert exec_process.stdin
with open(from_path, "rb") as from_file:
# Bug in mypy, https://github.com/python/mypy/issues/15031
shutil.copyfileobj(from_file, exec_process.stdin) # type: ignore[misc]
exec_process.stdin.close()
exec_process.wait()
if exec_process.returncode:
raise subprocess.CalledProcessError(
exec_process.returncode, exec_process.args, None, None
)
def copy_out(self, from_path: PurePath, to_path: Path) -> None:
# note: we assume from_path is a dir
to_path.mkdir(parents=True, exist_ok=True)
if self.engine.name == "podman":
subprocess.run(
[
self.engine.name,
"cp",
f"{self.name}:{from_path}/.",
str(to_path),
],
check=True,
cwd=to_path,
)
elif self.engine.name == "docker":
# There is a bug in docker that prevents a simple 'cp' invocation
# from working https://github.com/moby/moby/issues/38995
command = f"{self.engine.name} exec -i {self.name} tar -cC {shell_quote(from_path)} -f - . | tar -xf -"
subprocess.run(
command,
shell=True,
check=True,
cwd=to_path,
)
else:
raise KeyError(self.engine.name)
def glob(self, path: PurePosixPath, pattern: str) -> list[PurePosixPath]:
glob_pattern = path.joinpath(pattern)
path_strings = json.loads(
self.call(
[
self.UTILITY_PYTHON,
"-c",
f"import sys, json, glob; json.dump(glob.glob({str(glob_pattern)!r}), sys.stdout)",
],
capture_output=True,
)
)
return [PurePosixPath(p) for p in path_strings]
def call(
self,
args: Sequence[PathOrStr],
env: Mapping[str, str] | None = None,
capture_output: bool = False,
cwd: PathOrStr | None = None,
) -> str:
if cwd is None:
# Podman does not start the a container in a specific working dir
# so we always need to specify it when making calls.
cwd = self.cwd
chdir = f"cd {cwd}" if cwd else ""
env_assignments = (
" ".join(f"{shlex.quote(k)}={shlex.quote(v)}" for k, v in env.items())
if env is not None
else ""
)
command = " ".join(shlex.quote(str(a)) for a in args)
end_of_message = str(uuid.uuid4())
# log the command we're executing
print(f" + {command}")
# Write a command to the remote shell. First we change the
# cwd, if that's required. Then, we use the `env` utility to run
# `command` inside the specified environment. We use `env` because it
# can cope with spaces and strange characters in the name or value.
# Finally, the remote shell is told to write a footer - this will show
# up in the output so we know when to stop reading, and will include
# the return code of `command`.
self.bash_stdin.write(
bytes(
f"""(
{chdir}
env {env_assignments} {command}
printf "%04d%s\n" $? {end_of_message}
)
""",
encoding="utf8",
errors="surrogateescape",
)
)
self.bash_stdin.flush()
if capture_output:
output_io: IO[bytes] = io.BytesIO()
else:
output_io = sys.stdout.buffer
while True:
line = self.bash_stdout.readline()
if line.endswith(bytes(end_of_message, encoding="utf8") + b"\n"):
# fmt: off
footer_offset = (
len(line)
- 1 # newline character
- len(end_of_message) # delimiter
- 4 # 4 return code decimals
)
# fmt: on
return_code_str = line[footer_offset : footer_offset + 4]
return_code = int(return_code_str)
# add the last line to output, without the footer
output_io.write(line[0:footer_offset])
output_io.flush()
break
else:
output_io.write(line)
output_io.flush()
if isinstance(output_io, io.BytesIO):
output = str(output_io.getvalue(), encoding="utf8", errors="surrogateescape")
else:
output = ""
if return_code != 0:
raise subprocess.CalledProcessError(return_code, args, output)
return output
def get_environment(self) -> dict[str, str]:
env = json.loads(
self.call(
[
self.UTILITY_PYTHON,
"-c",
"import sys, json, os; json.dump(os.environ.copy(), sys.stdout)",
],
capture_output=True,
)
)
return typing.cast(Dict[str, str], env)
def environment_executor(self, command: Sequence[str], environment: dict[str, str]) -> str:
# used as an EnvironmentExecutor to evaluate commands and capture output
return self.call(command, env=environment, capture_output=True)
def debug_info(self) -> str:
if self.engine.name == "podman":
command = f"{self.engine.name} info --debug"
else:
command = f"{self.engine.name} info"
completed = subprocess.run(
command,
shell=True,
check=True,
cwd=self.cwd,
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
)
output = str(completed.stdout, encoding="utf8", errors="surrogateescape")
return output
def shell_quote(path: PurePath) -> str:
return shlex.quote(os.fspath(path))
-952
View File
@@ -1,952 +0,0 @@
from __future__ import annotations
import collections
import configparser
import contextlib
import dataclasses
import difflib
import enum
import functools
import shlex
import textwrap
from collections.abc import Generator, Iterable, Set
from pathlib import Path
from typing import Any, Literal, Mapping, Sequence, Union # noqa: TID251
from packaging.specifiers import SpecifierSet
from . import errors
from ._compat import tomllib
from ._compat.typing import assert_never
from .architecture import Architecture
from .environment import EnvironmentParseError, ParsedEnvironment, parse_environment
from .logger import log
from .oci_container import OCIContainerEngineConfig
from .projectfiles import get_requires_python_str
from .typing import PLATFORMS, PlatformName
from .util import (
MANYLINUX_ARCHS,
MUSLLINUX_ARCHS,
BuildFrontendConfig,
BuildSelector,
DependencyConstraints,
TestSelector,
format_safe,
resources_dir,
selector_matches,
strtobool,
unwrap,
)
@dataclasses.dataclass
class CommandLineArguments:
platform: Literal["auto", "linux", "macos", "windows"] | None
archs: str | None
output_dir: Path
only: str | None
config_file: str
package_dir: Path
print_build_identifiers: bool
allow_empty: bool
prerelease_pythons: bool
debug_traceback: bool
@staticmethod
def defaults() -> CommandLineArguments:
return CommandLineArguments(
platform="auto",
allow_empty=False,
archs=None,
only=None,
config_file="",
output_dir=Path("wheelhouse"),
package_dir=Path("."),
prerelease_pythons=False,
print_build_identifiers=False,
debug_traceback=False,
)
@dataclasses.dataclass(frozen=True)
class GlobalOptions:
package_dir: Path
output_dir: Path
build_selector: BuildSelector
test_selector: TestSelector
architectures: set[Architecture]
allow_empty: bool
@dataclasses.dataclass(frozen=True)
class BuildOptions:
globals: GlobalOptions
environment: ParsedEnvironment
before_all: str
before_build: str | None
repair_command: str
manylinux_images: dict[str, str] | None
musllinux_images: dict[str, str] | None
dependency_constraints: DependencyConstraints | None
test_command: str | None
before_test: str | None
test_requires: list[str]
test_extras: str
build_verbosity: int
build_frontend: BuildFrontendConfig | None
config_settings: str
container_engine: OCIContainerEngineConfig
@property
def package_dir(self) -> Path:
return self.globals.package_dir
@property
def output_dir(self) -> Path:
return self.globals.output_dir
@property
def build_selector(self) -> BuildSelector:
return self.globals.build_selector
@property
def test_selector(self) -> TestSelector:
return self.globals.test_selector
@property
def architectures(self) -> set[Architecture]:
return self.globals.architectures
SettingLeaf = Union[str, int, bool]
SettingList = Sequence[SettingLeaf]
SettingTable = Mapping[str, Union[SettingLeaf, SettingList]]
SettingValue = Union[SettingTable, SettingList, SettingLeaf]
@dataclasses.dataclass(frozen=True)
class Override:
select_pattern: str
options: dict[str, SettingValue]
inherit: dict[str, InheritRule]
MANYLINUX_OPTIONS = {f"manylinux-{build_platform}-image" for build_platform in MANYLINUX_ARCHS}
MUSLLINUX_OPTIONS = {f"musllinux-{build_platform}-image" for build_platform in MUSLLINUX_ARCHS}
DISALLOWED_OPTIONS = {
"linux": {"dependency-versions"},
"macos": MANYLINUX_OPTIONS | MUSLLINUX_OPTIONS,
"windows": MANYLINUX_OPTIONS | MUSLLINUX_OPTIONS,
}
class OptionsReaderError(errors.ConfigurationError):
pass
class OptionFormat:
"""
Base class for option format specifiers. These objects describe how values
can be parsed from rich TOML values and how they're merged together.
"""
class NotSupported(Exception):
pass
def format_list(self, value: SettingList) -> str: # noqa: ARG002
raise OptionFormat.NotSupported
def format_table(self, table: SettingTable) -> str: # noqa: ARG002
raise OptionFormat.NotSupported
def merge_values(self, before: str, after: str) -> str: # noqa: ARG002
raise OptionFormat.NotSupported
class ListFormat(OptionFormat):
"""
A format that joins lists with a separator.
"""
def __init__(self, sep: str) -> None:
self.sep = sep
def format_list(self, value: SettingList) -> str:
return self.sep.join(str(v) for v in value)
def merge_values(self, before: str, after: str) -> str:
return f"{before}{self.sep}{after}"
class ShlexTableFormat(OptionFormat):
"""
The standard table format uses shlex.quote to quote values and shlex.split
to unquote and split them. When merging values, keys in before are
replaced by keys in after.
"""
def __init__(self, sep: str = " ", pair_sep: str = "=", allow_merge: bool = True) -> None:
self.sep = sep
self.pair_sep = pair_sep
self.allow_merge = allow_merge
def format_table(self, table: SettingTable) -> str:
assignments: list[tuple[str, str]] = []
for k, v in table.items():
if shlex.split(k) != [k]:
msg = f"Invalid table key: {k}"
raise OptionsReaderError(msg)
if isinstance(v, str):
assignments.append((k, v))
elif isinstance(v, Sequence):
for inner_v in v:
assignments.append((k, str(inner_v)))
else:
assignments.append((k, str(v)))
return self.sep.join(f"{k}{self.pair_sep}{shlex.quote(v)}" for k, v in assignments)
def merge_values(self, before: str, after: str) -> str:
if not self.allow_merge:
raise OptionFormat.NotSupported
before_dict = self.parse_table(before)
after_dict = self.parse_table(after)
return self.format_table({**before_dict, **after_dict})
def parse_table(self, table: str) -> Mapping[str, str | Sequence[str]]:
assignments: list[tuple[str, str]] = []
for assignment_str in shlex.split(table):
key, sep, value = assignment_str.partition(self.pair_sep)
if not sep:
msg = f"malformed option with value {assignment_str!r}"
raise OptionsReaderError(msg)
assignments.append((key, value))
result: dict[str, str | list[str]] = {}
for key, value in assignments:
if key in result:
existing_value = result[key]
if isinstance(existing_value, list):
result[key] = [*existing_value, value]
else:
result[key] = [existing_value, value]
else:
result[key] = value
return result
class EnvironmentFormat(OptionFormat):
"""
The environment format accepts a table of environment variables, where the
values may contain variables or command substitutions.
"""
def format_table(self, table: SettingTable) -> str:
return " ".join(f'{k}="{v}"' for k, v in table.items())
def merge_values(self, before: str, after: str) -> str:
return f"{before} {after}"
class InheritRule(enum.Enum):
NONE = enum.auto()
APPEND = enum.auto()
PREPEND = enum.auto()
def _resolve_cascade(
*pairs: tuple[SettingValue | None, InheritRule],
ignore_empty: bool = False,
option_format: OptionFormat | None = None,
) -> str:
"""
Given a cascade of values with inherit rules, resolve them into a single
value.
'None' values mean that the option was not set at that level, and are
ignored. If `ignore_empty` is True, empty values are ignored too.
Values start with defaults, followed by more specific rules. If rules are
NONE, the last non-null value is returned. If a rule is APPEND or PREPEND,
the value is concatenated with the previous value.
The following idiom can be used to get the first matching value:
_resolve_cascade(("value1", Inherit.NONE), ("value2", Inherit.NONE), ...)))
"""
if not pairs:
msg = "pairs cannot be empty"
raise ValueError(msg)
result: str | None = None
for value, rule in pairs:
if value is None:
continue
if ignore_empty and not value and value is not False:
continue
value_string = _stringify_setting(value, option_format=option_format)
result = _apply_inherit_rule(result, value_string, rule=rule, option_format=option_format)
if result is None:
msg = "a setting should at least have a default value"
raise ValueError(msg)
return result
# pylint: disable-next=inconsistent-return-statements
def _apply_inherit_rule(
before: str | None, after: str, rule: InheritRule, option_format: OptionFormat | None
) -> str:
if rule == InheritRule.NONE:
return after
if not before:
# if before is None, we can just return after
# if before is an empty string, we shouldn't add any separator
return after
if not after:
# if after is an empty string, we shouldn't add any separator
return before
if not option_format:
msg = f"Don't know how to merge {before!r} and {after!r} with {rule}"
raise OptionsReaderError(msg)
if rule == InheritRule.APPEND:
return option_format.merge_values(before, after)
elif rule == InheritRule.PREPEND:
return option_format.merge_values(after, before)
else:
assert_never(rule)
def _stringify_setting(
setting: SettingValue,
option_format: OptionFormat | None,
) -> str:
if isinstance(setting, Mapping):
try:
if option_format is None:
raise OptionFormat.NotSupported
return option_format.format_table(setting)
except OptionFormat.NotSupported:
msg = f"Error converting {setting!r} to a string: this setting doesn't accept a table"
raise OptionsReaderError(msg) from None
if not isinstance(setting, str) and isinstance(setting, Sequence):
try:
if option_format is None:
raise OptionFormat.NotSupported
return option_format.format_list(setting)
except OptionFormat.NotSupported:
msg = f"Error converting {setting!r} to a string: this setting doesn't accept a list"
raise OptionsReaderError(msg) from None
if isinstance(setting, (bool, int)):
return str(setting)
return setting
class OptionsReader:
"""
Gets options from the environment, config or defaults, optionally scoped
by the platform.
Example:
>>> options_reader = OptionsReader(config_file, platform='macos')
>>> options_reader.get('cool-color')
This will return the value of CIBW_COOL_COLOR_MACOS if it exists,
otherwise the value of CIBW_COOL_COLOR, otherwise
'tool.cibuildwheel.macos.cool-color' or 'tool.cibuildwheel.cool-color'
from `config_file`, or from cibuildwheel/resources/defaults.toml. An
error is thrown if there are any unexpected keys or sections in
tool.cibuildwheel.
"""
def __init__(
self,
config_file_path: Path | None = None,
*,
platform: PlatformName,
env: Mapping[str, str],
disallow: Mapping[str, Set[str]] | None = None,
) -> None:
self.platform = platform
self.env = env
self.disallow = disallow or {}
# Open defaults.toml, loading both global and platform sections
defaults_path = resources_dir / "defaults.toml"
self.default_options, self.default_platform_options = self._load_file(defaults_path)
# Load the project config file
config_options: dict[str, Any] = {}
config_platform_options: dict[str, Any] = {}
if config_file_path is not None:
config_options, config_platform_options = self._load_file(config_file_path)
# Validate project config
for option_name in config_options:
self._validate_global_option(option_name)
for option_name in config_platform_options:
self._validate_platform_option(option_name)
self.config_options = config_options
self.config_platform_options = config_platform_options
self.overrides: list[Override] = []
self.current_identifier: str | None = None
config_overrides = self.config_options.get("overrides")
if config_overrides is not None:
if not isinstance(config_overrides, list):
msg = "'tool.cibuildwheel.overrides' must be a list"
raise OptionsReaderError(msg)
for config_override in config_overrides:
select = config_override.pop("select", None)
if not select:
msg = "'select' must be set in an override"
raise OptionsReaderError(msg)
if isinstance(select, list):
select = " ".join(select)
inherit = config_override.pop("inherit", {})
if not isinstance(inherit, dict) or not all(
i in {"none", "append", "prepend"} for i in inherit.values()
):
msg = "'inherit' must be a dict containing only {'none', 'append', 'prepend'} values"
raise OptionsReaderError(msg)
inherit_enum = {k: InheritRule[v.upper()] for k, v in inherit.items()}
self.overrides.append(Override(select, config_override, inherit_enum))
def _validate_global_option(self, name: str) -> None:
"""
Raises an error if an option with this name is not allowed in the
[tool.cibuildwheel] section of a config file.
"""
allowed_option_names = self.default_options.keys() | PLATFORMS | {"overrides"}
if name not in allowed_option_names:
msg = f"Option {name!r} not supported in a config file."
matches = difflib.get_close_matches(name, allowed_option_names, 1, 0.7)
if matches:
msg += f" Perhaps you meant {matches[0]!r}?"
raise OptionsReaderError(msg)
def _validate_platform_option(self, name: str) -> None:
"""
Raises an error if an option with this name is not allowed in the
[tool.cibuildwheel.<current-platform>] section of a config file.
"""
disallowed_platform_options = self.disallow.get(self.platform, set())
if name in disallowed_platform_options:
msg = f"{name!r} is not allowed in {disallowed_platform_options}"
raise OptionsReaderError(msg)
allowed_option_names = self.default_options.keys() | self.default_platform_options.keys()
if name not in allowed_option_names:
msg = f"Option {name!r} not supported in the {self.platform!r} section"
matches = difflib.get_close_matches(name, allowed_option_names, 1, 0.7)
if matches:
msg += f" Perhaps you meant {matches[0]!r}?"
raise OptionsReaderError(msg)
def _load_file(self, filename: Path) -> tuple[dict[str, Any], dict[str, Any]]:
"""
Load a toml file, returns global and platform as separate dicts.
"""
with filename.open("rb") as f:
config = tomllib.load(f)
global_options = config.get("tool", {}).get("cibuildwheel", {})
platform_options = global_options.get(self.platform, {})
return global_options, platform_options
@property
def active_config_overrides(self) -> list[Override]:
if self.current_identifier is None:
return []
return [
o for o in self.overrides if selector_matches(o.select_pattern, self.current_identifier)
]
@contextlib.contextmanager
def identifier(self, identifier: str | None) -> Generator[None, None, None]:
self.current_identifier = identifier
try:
yield
finally:
self.current_identifier = None
def get(
self,
name: str,
*,
env_plat: bool = True,
option_format: OptionFormat | None = None,
ignore_empty: bool = False,
) -> str:
"""
Get and return the value for the named option from environment,
configuration file, or the default. If env_plat is False, then don't
accept platform versions of the environment variable. If this is an
array it will be merged with "sep" before returning. If it is a table,
it will be formatted with "table['item']" using {k} and {v} and merged
with "table['sep']". If sep is also given, it will be used for arrays
inside the table (must match table['sep']). Empty variables will not
override if ignore_empty is True.
"""
if name not in self.default_options and name not in self.default_platform_options:
msg = f"{name!r} must be in cibuildwheel/resources/defaults.toml file to be accessed."
raise OptionsReaderError(msg)
# Environment variable form
envvar = f"CIBW_{name.upper().replace('-', '_')}"
plat_envvar = f"{envvar}_{self.platform.upper()}"
# get the option from the default, then the config file, then finally the environment.
# platform-specific options are preferred, if they're allowed.
return _resolve_cascade(
(self.default_options.get(name), InheritRule.NONE),
(self.default_platform_options.get(name), InheritRule.NONE),
(self.config_options.get(name), InheritRule.NONE),
(self.config_platform_options.get(name), InheritRule.NONE),
*[
(o.options.get(name), o.inherit.get(name, InheritRule.NONE))
for o in self.active_config_overrides
],
(self.env.get(envvar), InheritRule.NONE),
(self.env.get(plat_envvar) if env_plat else None, InheritRule.NONE),
ignore_empty=ignore_empty,
option_format=option_format,
)
class Options:
def __init__(
self,
platform: PlatformName,
command_line_arguments: CommandLineArguments,
env: Mapping[str, str],
read_config_file: bool = True,
):
self.platform = platform
self.command_line_arguments = command_line_arguments
self.env = env
self.reader = OptionsReader(
self.config_file_path if read_config_file else None,
platform=platform,
env=env,
disallow=DISALLOWED_OPTIONS,
)
@property
def config_file_path(self) -> Path | None:
args = self.command_line_arguments
if args.config_file:
return Path(format_safe(args.config_file, package=args.package_dir))
# return pyproject.toml, if it's available
pyproject_toml_path = Path(args.package_dir) / "pyproject.toml"
if pyproject_toml_path.exists():
return pyproject_toml_path
return None
@functools.cached_property
def package_requires_python_str(self) -> str | None:
args = self.command_line_arguments
return get_requires_python_str(Path(args.package_dir))
@property
def globals(self) -> GlobalOptions:
args = self.command_line_arguments
package_dir = args.package_dir
output_dir = args.output_dir
build_config = (
self.reader.get("build", env_plat=False, option_format=ListFormat(sep=" ")) or "*"
)
skip_config = self.reader.get("skip", env_plat=False, option_format=ListFormat(sep=" "))
test_skip = self.reader.get("test-skip", env_plat=False, option_format=ListFormat(sep=" "))
free_threaded_support = strtobool(
self.reader.get("free-threaded-support", env_plat=False, ignore_empty=True)
)
allow_empty = args.allow_empty or strtobool(self.env.get("CIBW_ALLOW_EMPTY", "0"))
prerelease_pythons = args.prerelease_pythons or strtobool(
self.env.get("CIBW_PRERELEASE_PYTHONS", "0")
)
# This is not supported in tool.cibuildwheel, as it comes from a standard location.
# Passing this in as an environment variable will override pyproject.toml, setup.cfg, or setup.py
requires_python_str: str | None = (
self.env.get("CIBW_PROJECT_REQUIRES_PYTHON") or self.package_requires_python_str
)
requires_python = None if requires_python_str is None else SpecifierSet(requires_python_str)
archs_config_str = args.archs or self.reader.get("archs", option_format=ListFormat(sep=" "))
architectures = Architecture.parse_config(archs_config_str, platform=self.platform)
# Process `--only`
if args.only:
build_config = args.only
skip_config = ""
architectures = Architecture.all_archs(self.platform)
prerelease_pythons = True
free_threaded_support = True
build_selector = BuildSelector(
build_config=build_config,
skip_config=skip_config,
requires_python=requires_python,
prerelease_pythons=prerelease_pythons,
free_threaded_support=free_threaded_support,
)
test_selector = TestSelector(skip_config=test_skip)
return GlobalOptions(
package_dir=package_dir,
output_dir=output_dir,
build_selector=build_selector,
test_selector=test_selector,
architectures=architectures,
allow_empty=allow_empty,
)
def build_options(self, identifier: str | None) -> BuildOptions:
"""
Compute BuildOptions for a single run configuration.
"""
with self.reader.identifier(identifier):
before_all = self.reader.get("before-all", option_format=ListFormat(sep=" && "))
environment_config = self.reader.get("environment", option_format=EnvironmentFormat())
environment_pass = self.reader.get(
"environment-pass", option_format=ListFormat(sep=" ")
).split()
before_build = self.reader.get("before-build", option_format=ListFormat(sep=" && "))
repair_command = self.reader.get(
"repair-wheel-command", option_format=ListFormat(sep=" && ")
)
config_settings = self.reader.get(
"config-settings", option_format=ShlexTableFormat(sep=" ", pair_sep="=")
)
dependency_versions = self.reader.get("dependency-versions")
test_command = self.reader.get("test-command", option_format=ListFormat(sep=" && "))
before_test = self.reader.get("before-test", option_format=ListFormat(sep=" && "))
test_requires = self.reader.get(
"test-requires", option_format=ListFormat(sep=" ")
).split()
test_extras = self.reader.get("test-extras", option_format=ListFormat(sep=","))
build_verbosity_str = self.reader.get("build-verbosity")
build_frontend_str = self.reader.get(
"build-frontend",
env_plat=False,
option_format=ShlexTableFormat(sep="; ", pair_sep=":", allow_merge=False),
)
build_frontend: BuildFrontendConfig | None
if not build_frontend_str or build_frontend_str == "default":
build_frontend = None
else:
try:
build_frontend = BuildFrontendConfig.from_config_string(build_frontend_str)
except ValueError as e:
msg = f"Failed to parse build frontend. {e}"
raise errors.ConfigurationError(msg) from e
try:
environment = parse_environment(environment_config)
except (EnvironmentParseError, ValueError) as e:
msg = f"Malformed environment option {environment_config!r}"
raise errors.ConfigurationError(msg) from e
# Pass through environment variables
if self.platform == "linux":
for env_var_name in reversed(environment_pass):
with contextlib.suppress(KeyError):
environment.add(env_var_name, self.env[env_var_name], prepend=True)
if dependency_versions == "pinned":
dependency_constraints: None | (
DependencyConstraints
) = DependencyConstraints.with_defaults()
elif dependency_versions == "latest":
dependency_constraints = None
else:
dependency_versions_path = Path(dependency_versions)
dependency_constraints = DependencyConstraints(dependency_versions_path)
if test_extras:
test_extras = f"[{test_extras}]"
try:
build_verbosity = min(3, max(-3, int(build_verbosity_str)))
except ValueError:
build_verbosity = 0
manylinux_images: dict[str, str] = {}
musllinux_images: dict[str, str] = {}
container_engine: OCIContainerEngineConfig | None = None
if self.platform == "linux":
all_pinned_container_images = _get_pinned_container_images()
for build_platform in MANYLINUX_ARCHS:
pinned_images = all_pinned_container_images[build_platform]
config_value = self.reader.get(
f"manylinux-{build_platform}-image", ignore_empty=True
)
if not config_value:
# default to manylinux2014
image = pinned_images["manylinux2014"]
elif config_value in pinned_images:
image = pinned_images[config_value]
else:
image = config_value
manylinux_images[build_platform] = image
for build_platform in MUSLLINUX_ARCHS:
pinned_images = all_pinned_container_images[build_platform]
config_value = self.reader.get(f"musllinux-{build_platform}-image")
if not config_value:
image = pinned_images["musllinux_1_2"]
elif config_value in pinned_images:
image = pinned_images[config_value]
else:
image = config_value
musllinux_images[build_platform] = image
container_engine_str = self.reader.get(
"container-engine",
option_format=ShlexTableFormat(sep="; ", pair_sep=":", allow_merge=False),
)
try:
container_engine = OCIContainerEngineConfig.from_config_string(container_engine_str)
except ValueError as e:
msg = f"Failed to parse container config. {e}"
raise errors.ConfigurationError(msg) from e
return BuildOptions(
globals=self.globals,
test_command=test_command,
test_requires=test_requires,
test_extras=test_extras,
before_test=before_test,
before_build=before_build,
before_all=before_all,
build_verbosity=build_verbosity,
repair_command=repair_command,
environment=environment,
dependency_constraints=dependency_constraints,
manylinux_images=manylinux_images or None,
musllinux_images=musllinux_images or None,
build_frontend=build_frontend,
config_settings=config_settings,
container_engine=container_engine,
)
def check_for_invalid_configuration(self, identifiers: Iterable[str]) -> None:
if self.platform in {"macos", "windows"}:
before_all_values = {self.build_options(i).before_all for i in identifiers}
if len(before_all_values) > 1:
raise ValueError(
unwrap(
f"""
before_all cannot be set to multiple values. On macOS and Windows,
before_all is only run once, at the start of the build. before_all values
are: {before_all_values!r}
"""
)
)
def check_for_deprecated_options(self) -> None:
build_selector = self.globals.build_selector
test_selector = self.globals.test_selector
deprecated_selectors("CIBW_BUILD", build_selector.build_config, error=True)
deprecated_selectors("CIBW_SKIP", build_selector.skip_config)
deprecated_selectors("CIBW_TEST_SKIP", test_selector.skip_config)
@functools.cached_property
def defaults(self) -> Options:
return Options(
platform=self.platform,
command_line_arguments=CommandLineArguments.defaults(),
env={},
read_config_file=False,
)
def summary(self, identifiers: Iterable[str]) -> str:
lines = []
global_option_names = sorted(f.name for f in dataclasses.fields(self.globals))
for option_name in global_option_names:
option_value = getattr(self.globals, option_name)
default_value = getattr(self.defaults.globals, option_name)
lines.append(self.option_summary(option_name, option_value, default_value))
build_options = self.build_options(identifier=None)
build_options_defaults = self.defaults.build_options(identifier=None)
build_options_for_identifier = {
identifier: self.build_options(identifier) for identifier in identifiers
}
build_option_names = sorted(f.name for f in dataclasses.fields(build_options))
for option_name in build_option_names:
if option_name == "globals":
continue
option_value = getattr(build_options, option_name)
default_value = getattr(build_options_defaults, option_name)
overrides = {
i: getattr(build_options_for_identifier[i], option_name) for i in identifiers
}
lines.append(
self.option_summary(option_name, option_value, default_value, overrides=overrides)
)
return "\n".join(lines)
def option_summary(
self,
option_name: str,
option_value: Any,
default_value: Any,
overrides: Mapping[str, Any] | None = None,
) -> str:
"""
Return a summary of the option value, including any overrides, with
ANSI 'dim' color if it's the default.
"""
value_str = self.option_summary_value(option_value)
default_value_str = self.option_summary_value(default_value)
overrides_value_strs = {
k: self.option_summary_value(v) for k, v in (overrides or {}).items()
}
# if the override value is the same as the non-overridden value, don't print it
overrides_value_strs = {k: v for k, v in overrides_value_strs.items() if v != value_str}
has_been_set = (value_str != default_value_str) or overrides_value_strs
c = log.colors
result = c.gray if not has_been_set else ""
result += f"{option_name}: "
if overrides_value_strs:
overrides_groups = collections.defaultdict(list)
for k, v in overrides_value_strs.items():
overrides_groups[v].append(k)
result += "\n *: "
result += self.indent_if_multiline(value_str, " ")
for override_value_str, identifiers in overrides_groups.items():
result += f"\n {', '.join(identifiers)}: "
result += self.indent_if_multiline(override_value_str, " ")
else:
result += self.indent_if_multiline(value_str, " ")
result += c.end
return result
def indent_if_multiline(self, value: str, indent: str) -> str:
if "\n" in value:
return "\n" + textwrap.indent(value.strip(), indent)
else:
return value
def option_summary_value(self, option_value: Any) -> str:
if hasattr(option_value, "options_summary"):
option_value = option_value.options_summary()
if isinstance(option_value, list):
return "".join(f"{el}\n" for el in option_value)
if isinstance(option_value, set):
return ", ".join(str(el) for el in sorted(option_value))
if isinstance(option_value, dict):
return "".join(f"{k}: {v}\n" for k, v in option_value.items())
return str(option_value)
def compute_options(
platform: PlatformName,
command_line_arguments: CommandLineArguments,
env: Mapping[str, str],
) -> Options:
options = Options(platform=platform, command_line_arguments=command_line_arguments, env=env)
options.check_for_deprecated_options()
return options
@functools.lru_cache(maxsize=None)
def _get_pinned_container_images() -> Mapping[str, Mapping[str, str]]:
"""
This looks like a dict of dicts, e.g.
{ 'x86_64': {'manylinux1': '...', 'manylinux2010': '...', 'manylinux2014': '...'},
'i686': {'manylinux1': '...', 'manylinux2010': '...', 'manylinux2014': '...'},
'pypy_x86_64': {'manylinux2010': '...' }
... }
"""
pinned_images_file = resources_dir / "pinned_docker_images.cfg"
all_pinned_images = configparser.ConfigParser()
all_pinned_images.read(pinned_images_file)
return all_pinned_images
def deprecated_selectors(name: str, selector: str, *, error: bool = False) -> None:
if "p2" in selector or "p35" in selector:
msg = f"cibuildwheel 2.x no longer supports Python < 3.6. Please use the 1.x series or update {name}"
if error:
raise errors.DeprecationError(msg)
log.warning(msg)
+42 -72
View File
@@ -1,108 +1,78 @@
from __future__ import annotations
import ast import ast
import configparser import sys
import contextlib from configparser import ConfigParser
from pathlib import Path from pathlib import Path
from typing import Any, Optional
from ._compat import tomllib import toml
if sys.version_info < (3, 8):
Constant = ast.Str
def get_constant(x: ast.Str) -> str:
return x.s
def get_parent(node: ast.AST | None, depth: int = 1) -> ast.AST | None: else:
for _ in range(depth): Constant = ast.Constant
node = getattr(node, "parent", None)
return node
def get_constant(x: ast.Constant) -> Any:
def is_main(parent: ast.AST | None) -> bool: return x.value
if parent is None:
return False
# This would be much nicer with 3.10's pattern matching!
if not isinstance(parent, ast.If):
return False
if not isinstance(parent.test, ast.Compare):
return False
try:
(op,) = parent.test.ops
(comp,) = parent.test.comparators
except ValueError:
return False
if not isinstance(op, ast.Eq):
return False
values = {comp, parent.test.left}
mains = {x for x in values if isinstance(x, ast.Constant) and x.value == "__main__"}
if len(mains) != 1:
return False
consts = {x for x in values if isinstance(x, ast.Name) and x.id == "__name__"}
return len(consts) == 1
class Analyzer(ast.NodeVisitor): class Analyzer(ast.NodeVisitor):
def __init__(self) -> None: def __init__(self) -> None:
self.requires_python: str | None = None self.requires_python: Optional[str] = None
def visit(self, node: ast.AST) -> None: def visit(self, content: ast.AST) -> None:
for inner_node in ast.walk(node): for node in ast.walk(content):
for child in ast.iter_child_nodes(inner_node): for child in ast.iter_child_nodes(node):
child.parent = inner_node # type: ignore[attr-defined] child.parent = node # type: ignore
super().visit(node) super().visit(content)
def visit_keyword(self, node: ast.keyword) -> None: def visit_keyword(self, node: ast.keyword) -> None:
# Must not be nested except for if __name__ == "__main__"
self.generic_visit(node) self.generic_visit(node)
if node.arg == "python_requires":
# Must not be nested in an if or other structure
# This will be Module -> Expr -> Call -> keyword # This will be Module -> Expr -> Call -> keyword
parent = get_parent(node, 4) if not hasattr(node.parent.parent.parent, "parent") and isinstance( # type: ignore
unnested = parent is None node.value, Constant
# This will be Module -> If -> Expr -> Call -> keyword
name_main_unnested = (
parent is not None and get_parent(parent) is None and is_main(get_parent(node, 3))
)
if (
node.arg == "python_requires"
and isinstance(node.value, ast.Constant)
and (unnested or name_main_unnested)
): ):
self.requires_python = node.value.value self.requires_python = get_constant(node.value)
def setup_py_python_requires(content: str) -> str | None: def setup_py_python_requires(content: str) -> Optional[str]:
try: try:
tree = ast.parse(content) tree = ast.parse(content)
analyzer = Analyzer() analyzer = Analyzer()
analyzer.visit(tree) analyzer.visit(tree)
return analyzer.requires_python or None return analyzer.requires_python or None
except Exception: # pylint: disable=broad-except except Exception:
return None return None
def get_requires_python_str(package_dir: Path) -> str | None: def get_requires_python_str(package_dir: Path) -> Optional[str]:
"""Return the python requires string from the most canonical source available, or None""" "Return the python requires string from the most canonical source available, or None"
# Read in from pyproject.toml:project.requires-python # Read in from pyproject.toml:project.requires-python
with contextlib.suppress(FileNotFoundError): try:
with (package_dir / "pyproject.toml").open("rb") as f1: info = toml.load(package_dir / "pyproject.toml")
info = tomllib.load(f1)
with contextlib.suppress(KeyError, IndexError, TypeError):
return str(info["project"]["requires-python"]) return str(info["project"]["requires-python"])
except (FileNotFoundError, KeyError, IndexError, TypeError):
pass
# Read in from setup.cfg:options.python_requires # Read in from setup.cfg:options.python_requires
config = configparser.ConfigParser() try:
with contextlib.suppress(FileNotFoundError): config = ConfigParser()
config.read(package_dir / "setup.cfg") config.read(package_dir / "setup.cfg")
with contextlib.suppress(KeyError, IndexError, TypeError):
return str(config["options"]["python_requires"]) return str(config["options"]["python_requires"])
except (FileNotFoundError, KeyError, IndexError, TypeError):
pass
setup_py = package_dir / "setup.py" try:
with contextlib.suppress(FileNotFoundError), setup_py.open(encoding="utf8") as f2: with open(package_dir / "setup.py") as f:
return setup_py_python_requires(f2.read()) return setup_py_python_requires(f.read())
except FileNotFoundError:
pass
return None return None
-399
View File
@@ -1,399 +0,0 @@
from __future__ import annotations
import os
import shutil
import sys
from collections.abc import Sequence, Set
from dataclasses import dataclass
from pathlib import Path
from filelock import FileLock
from . import errors
from .architecture import Architecture
from .environment import ParsedEnvironment
from .logger import log
from .options import Options
from .typing import PathOrStr
from .util import (
CIBW_CACHE_PATH,
BuildFrontendConfig,
BuildSelector,
call,
combine_constraints,
download,
ensure_node,
extract_zip,
find_compatible_wheel,
get_pip_version,
move_file,
prepare_command,
read_python_configs,
shell,
split_config_settings,
test_fail_cwd_file,
virtualenv,
)
@dataclass(frozen=True)
class PythonConfiguration:
version: str
identifier: str
pyodide_version: str
emscripten_version: str
node_version: str
def install_emscripten(tmp: Path, version: str) -> Path:
# We don't need to match the emsdk version to the version we install, but
# we do for stability
url = f"https://github.com/emscripten-core/emsdk/archive/refs/tags/{version}.zip"
installation_path = CIBW_CACHE_PATH / f"emsdk-{version}"
emsdk_path = installation_path / f"emsdk-{version}/emsdk"
emcc_path = installation_path / f"emsdk-{version}/upstream/emscripten/emcc"
with FileLock(f"{installation_path}.lock"):
if installation_path.exists():
return emcc_path
emsdk_zip = tmp / "emsdk.zip"
download(url, emsdk_zip)
installation_path.mkdir()
extract_zip(emsdk_zip, installation_path)
call(emsdk_path, "install", version)
call(emsdk_path, "activate", version)
return emcc_path
def install_xbuildenv(env: dict[str, str], pyodide_version: str) -> str:
pyodide_root = (
CIBW_CACHE_PATH
/ f".pyodide-xbuildenv-{pyodide_version}/{pyodide_version}/xbuildenv/pyodide-root"
)
with FileLock(CIBW_CACHE_PATH / "xbuildenv.lock"):
if pyodide_root.exists():
return str(pyodide_root)
# We don't want to mutate env but we need to delete any existing
# PYODIDE_ROOT so copy it first.
env = dict(env)
env.pop("PYODIDE_ROOT", None)
call(
"pyodide",
"xbuildenv",
"install",
pyodide_version,
env=env,
cwd=CIBW_CACHE_PATH,
)
return str(pyodide_root)
def get_base_python(identifier: str) -> Path:
implementation_id = identifier.split("-")[0]
majorminor = implementation_id[len("cp") :]
version_info = (int(majorminor[0]), int(majorminor[1:]))
if version_info == sys.version_info[:2]:
return Path(sys.executable)
major_minor = ".".join(str(v) for v in version_info)
python_name = f"python{major_minor}"
which_python = shutil.which(python_name)
if which_python is None:
msg = f"CPython {major_minor} is not installed."
raise errors.FatalError(msg)
return Path(which_python)
def setup_python(
tmp: Path,
python_configuration: PythonConfiguration,
dependency_constraint_flags: Sequence[PathOrStr],
environment: ParsedEnvironment,
) -> dict[str, str]:
base_python = get_base_python(python_configuration.identifier)
log.step("Setting up build environment...")
venv_path = tmp / "venv"
env = virtualenv(python_configuration.version, base_python, venv_path, [], use_uv=False)
venv_bin_path = venv_path / "bin"
assert venv_bin_path.exists()
env["PIP_DISABLE_PIP_VERSION_CHECK"] = "1"
# upgrade pip to the version matching our constraints
# if necessary, reinstall it to ensure that it's available on PATH as 'pip'
call(
"python",
"-m",
"pip",
"install",
"--upgrade",
"pip",
*dependency_constraint_flags,
env=env,
cwd=venv_path,
)
env = environment.as_dictionary(prev_environment=env)
# check what pip version we're on
assert (venv_bin_path / "pip").exists()
call("which", "pip", env=env)
call("pip", "--version", env=env)
which_pip = call("which", "pip", env=env, capture_stdout=True).strip()
if which_pip != str(venv_bin_path / "pip"):
msg = "pip available on PATH doesn't match our venv instance. If you have modified PATH, ensure that you don't overwrite cibuildwheel's entry or insert pip above it."
raise errors.FatalError(msg)
# check what Python version we're on
call("which", "python", env=env)
call("python", "--version", env=env)
which_python = call("which", "python", env=env, capture_stdout=True).strip()
if which_python != str(venv_bin_path / "python"):
msg = "python available on PATH doesn't match our venv instance. If you have modified PATH, ensure that you don't overwrite cibuildwheel's entry or insert python above it."
raise errors.FatalError(msg)
log.step("Installing build tools...")
call(
"pip",
"install",
"--upgrade",
"auditwheel-emscripten",
"build[virtualenv]",
"pyodide-build",
*dependency_constraint_flags,
env=env,
)
log.step("Installing emscripten...")
emcc_path = install_emscripten(tmp, python_configuration.emscripten_version)
env["PATH"] = os.pathsep.join([str(emcc_path.parent), env["PATH"]])
log.step("Installing Pyodide xbuildenv...")
env["PYODIDE_ROOT"] = install_xbuildenv(env, python_configuration.pyodide_version)
return env
def get_python_configurations(
build_selector: BuildSelector,
architectures: Set[Architecture], # noqa: ARG001
) -> list[PythonConfiguration]:
full_python_configs = read_python_configs("pyodide")
python_configurations = [PythonConfiguration(**item) for item in full_python_configs]
python_configurations = [c for c in python_configurations if build_selector(c.identifier)]
return python_configurations
def build(options: Options, tmp_path: Path) -> None:
python_configurations = get_python_configurations(
options.globals.build_selector, options.globals.architectures
)
if not python_configurations:
return
try:
before_all_options_identifier = python_configurations[0].identifier
before_all_options = options.build_options(before_all_options_identifier)
if before_all_options.before_all:
log.step("Running before_all...")
env = before_all_options.environment.as_dictionary(prev_environment=os.environ)
before_all_prepared = prepare_command(
before_all_options.before_all, project=".", package=before_all_options.package_dir
)
shell(before_all_prepared, env=env)
built_wheels: list[Path] = []
for config in python_configurations:
build_options = options.build_options(config.identifier)
build_frontend = build_options.build_frontend or BuildFrontendConfig("build")
if build_frontend.name == "pip":
msg = "The pyodide platform doesn't support pip frontend"
raise errors.FatalError(msg)
log.build_start(config.identifier)
identifier_tmp_dir = tmp_path / config.identifier
built_wheel_dir = identifier_tmp_dir / "built_wheel"
repaired_wheel_dir = identifier_tmp_dir / "repaired_wheel"
identifier_tmp_dir.mkdir()
built_wheel_dir.mkdir()
repaired_wheel_dir.mkdir()
dependency_constraint_flags: Sequence[PathOrStr] = []
if build_options.dependency_constraints:
constraints_path = build_options.dependency_constraints.get_for_python_version(
config.version, variant="pyodide"
)
dependency_constraint_flags = ["-c", constraints_path]
env = setup_python(
identifier_tmp_dir / "build",
config,
dependency_constraint_flags,
build_options.environment,
)
pip_version = get_pip_version(env)
# The Pyodide command line runner mounts all directories in the host
# filesystem into the Pyodide file system, except for the custom
# file systems /dev, /lib, /proc, and /tmp. Mounting the mount
# points for alternate file systems causes some mysterious failure
# of the process (it just quits without any clear error).
#
# Because of this, by default Pyodide can't see anything under /tmp.
# This environment variable tells it also to mount our temp
# directory.
oldmounts = ""
extra_mounts = [str(identifier_tmp_dir)]
if str(Path(".").resolve()).startswith("/tmp"):
extra_mounts.append(str(Path(".").resolve()))
if "_PYODIDE_EXTRA_MOUNTS" in env:
oldmounts = env["_PYODIDE_EXTRA_MOUNTS"] + ":"
env["_PYODIDE_EXTRA_MOUNTS"] = oldmounts + ":".join(extra_mounts)
compatible_wheel = find_compatible_wheel(built_wheels, config.identifier)
if compatible_wheel:
log.step_end()
print(
f"\nFound previously built wheel {compatible_wheel.name}, that's compatible with {config.identifier}. Skipping build step..."
)
built_wheel = compatible_wheel
else:
if build_options.before_build:
log.step("Running before_build...")
before_build_prepared = prepare_command(
build_options.before_build, project=".", package=build_options.package_dir
)
shell(before_build_prepared, env=env)
log.step("Building wheel...")
extra_flags = split_config_settings(build_options.config_settings, "build")
extra_flags += build_frontend.args
if not 0 <= build_options.build_verbosity < 2:
msg = f"build_verbosity {build_options.build_verbosity} is not supported for build frontend. Ignoring."
log.warning(msg)
build_env = env.copy()
if build_options.dependency_constraints:
combine_constraints(build_env, constraints_path, identifier_tmp_dir)
build_env["VIRTUALENV_PIP"] = pip_version
call(
"pyodide",
"build",
build_options.package_dir,
f"--outdir={built_wheel_dir}",
*extra_flags,
env=build_env,
)
built_wheel = next(built_wheel_dir.glob("*.whl"))
if built_wheel.name.endswith("none-any.whl"):
raise errors.NonPlatformWheelError()
if build_options.repair_command:
log.step("Repairing wheel...")
repair_command_prepared = prepare_command(
build_options.repair_command,
wheel=built_wheel,
dest_dir=repaired_wheel_dir,
)
shell(repair_command_prepared, env=env)
else:
shutil.move(str(built_wheel), repaired_wheel_dir)
repaired_wheel = next(repaired_wheel_dir.glob("*.whl"))
if repaired_wheel.name in {wheel.name for wheel in built_wheels}:
raise errors.AlreadyBuiltWheelError(repaired_wheel.name)
if build_options.test_command and build_options.test_selector(config.identifier):
log.step("Testing wheel...")
venv_dir = identifier_tmp_dir / "venv-test"
# set up a virtual environment to install and test from, to make sure
# there are no dependencies that were pulled in at build time.
virtualenv_env = env.copy()
virtualenv_env["PATH"] = os.pathsep.join(
[
str(ensure_node(config.node_version)),
virtualenv_env["PATH"],
]
)
# pyodide venv uses virtualenv under the hood
# use the pip embedded with virtualenv & disable network updates
virtualenv_create_env = virtualenv_env.copy()
virtualenv_create_env["VIRTUALENV_PIP"] = pip_version
virtualenv_create_env["VIRTUALENV_NO_PERIODIC_UPDATE"] = "1"
call("pyodide", "venv", venv_dir, env=virtualenv_create_env)
virtualenv_env["PATH"] = os.pathsep.join(
[
str(venv_dir / "bin"),
virtualenv_env["PATH"],
]
)
virtualenv_env["VIRTUAL_ENV"] = str(venv_dir)
# check that we are using the Python from the virtual environment
call("which", "python", env=virtualenv_env)
if build_options.before_test:
before_test_prepared = prepare_command(
build_options.before_test,
project=".",
package=build_options.package_dir,
)
shell(before_test_prepared, env=virtualenv_env)
# install the wheel
call(
"pip",
"install",
f"{repaired_wheel}{build_options.test_extras}",
env=virtualenv_env,
)
# test the wheel
if build_options.test_requires:
call("pip", "install", *build_options.test_requires, env=virtualenv_env)
# run the tests from a temp dir, with an absolute path in the command
# (this ensures that Python runs the tests against the installed wheel
# and not the repo code)
test_command_prepared = prepare_command(
build_options.test_command,
project=Path(".").resolve(),
package=build_options.package_dir.resolve(),
)
test_cwd = identifier_tmp_dir / "test_cwd"
test_cwd.mkdir(exist_ok=True)
(test_cwd / "test_fail.py").write_text(test_fail_cwd_file.read_text())
shell(test_command_prepared, cwd=test_cwd, env=virtualenv_env)
# we're all done here; move it to output (overwrite existing)
if compatible_wheel is None:
output_wheel = build_options.output_dir.joinpath(repaired_wheel.name)
moved_wheel = move_file(repaired_wheel, output_wheel)
if moved_wheel != output_wheel.resolve():
log.warning(
"{repaired_wheel} was moved to {moved_wheel} instead of {output_wheel}"
)
built_wheels.append(output_wheel)
finally:
pass
+30 -135
View File
@@ -1,176 +1,71 @@
[linux] [linux]
python_configurations = [ python_configurations = [
{ identifier = "cp27-manylinux_x86_64", version = "2.7", path_str = "/opt/python/cp27-cp27m" },
{ identifier = "cp27-manylinux_x86_64", version = "2.7", path_str = "/opt/python/cp27-cp27mu" },
{ identifier = "cp35-manylinux_x86_64", version = "3.5", path_str = "/opt/python/cp35-cp35m" },
{ identifier = "cp36-manylinux_x86_64", version = "3.6", path_str = "/opt/python/cp36-cp36m" }, { identifier = "cp36-manylinux_x86_64", version = "3.6", path_str = "/opt/python/cp36-cp36m" },
{ identifier = "cp37-manylinux_x86_64", version = "3.7", path_str = "/opt/python/cp37-cp37m" }, { identifier = "cp37-manylinux_x86_64", version = "3.7", path_str = "/opt/python/cp37-cp37m" },
{ identifier = "cp38-manylinux_x86_64", version = "3.8", path_str = "/opt/python/cp38-cp38" }, { identifier = "cp38-manylinux_x86_64", version = "3.8", path_str = "/opt/python/cp38-cp38" },
{ identifier = "cp39-manylinux_x86_64", version = "3.9", path_str = "/opt/python/cp39-cp39" }, { identifier = "cp39-manylinux_x86_64", version = "3.9", path_str = "/opt/python/cp39-cp39" },
{ identifier = "cp310-manylinux_x86_64", version = "3.10", path_str = "/opt/python/cp310-cp310" }, { identifier = "cp27-manylinux_i686", version = "2.7", path_str = "/opt/python/cp27-cp27m" },
{ identifier = "cp311-manylinux_x86_64", version = "3.11", path_str = "/opt/python/cp311-cp311" }, { identifier = "cp27-manylinux_i686", version = "2.7", path_str = "/opt/python/cp27-cp27mu" },
{ identifier = "cp312-manylinux_x86_64", version = "3.12", path_str = "/opt/python/cp312-cp312" }, { identifier = "cp35-manylinux_i686", version = "3.5", path_str = "/opt/python/cp35-cp35m" },
{ identifier = "cp313-manylinux_x86_64", version = "3.13", path_str = "/opt/python/cp313-cp313" },
{ identifier = "cp313t-manylinux_x86_64", version = "3.13", path_str = "/opt/python/cp313-cp313t" },
{ identifier = "cp36-manylinux_i686", version = "3.6", path_str = "/opt/python/cp36-cp36m" }, { identifier = "cp36-manylinux_i686", version = "3.6", path_str = "/opt/python/cp36-cp36m" },
{ identifier = "cp37-manylinux_i686", version = "3.7", path_str = "/opt/python/cp37-cp37m" }, { identifier = "cp37-manylinux_i686", version = "3.7", path_str = "/opt/python/cp37-cp37m" },
{ identifier = "cp38-manylinux_i686", version = "3.8", path_str = "/opt/python/cp38-cp38" }, { identifier = "cp38-manylinux_i686", version = "3.8", path_str = "/opt/python/cp38-cp38" },
{ identifier = "cp39-manylinux_i686", version = "3.9", path_str = "/opt/python/cp39-cp39" }, { identifier = "cp39-manylinux_i686", version = "3.9", path_str = "/opt/python/cp39-cp39" },
{ identifier = "cp310-manylinux_i686", version = "3.10", path_str = "/opt/python/cp310-cp310" }, { identifier = "pp27-manylinux_x86_64", version = "2.7", path_str = "/opt/python/pp27-pypy_73" },
{ identifier = "cp311-manylinux_i686", version = "3.11", path_str = "/opt/python/cp311-cp311" }, { identifier = "pp36-manylinux_x86_64", version = "3.6", path_str = "/opt/python/pp36-pypy36_pp73" },
{ identifier = "cp312-manylinux_i686", version = "3.12", path_str = "/opt/python/cp312-cp312" },
{ identifier = "cp313-manylinux_i686", version = "3.13", path_str = "/opt/python/cp313-cp313" },
{ identifier = "cp313t-manylinux_i686", version = "3.13", path_str = "/opt/python/cp313-cp313t" },
{ identifier = "pp37-manylinux_x86_64", version = "3.7", path_str = "/opt/python/pp37-pypy37_pp73" }, { identifier = "pp37-manylinux_x86_64", version = "3.7", path_str = "/opt/python/pp37-pypy37_pp73" },
{ identifier = "pp38-manylinux_x86_64", version = "3.8", path_str = "/opt/python/pp38-pypy38_pp73" }, { identifier = "cp35-manylinux_aarch64", version = "3.5", path_str = "/opt/python/cp35-cp35m" },
{ identifier = "pp39-manylinux_x86_64", version = "3.9", path_str = "/opt/python/pp39-pypy39_pp73" },
{ identifier = "pp310-manylinux_x86_64", version = "3.10", path_str = "/opt/python/pp310-pypy310_pp73" },
{ identifier = "cp36-manylinux_aarch64", version = "3.6", path_str = "/opt/python/cp36-cp36m" }, { identifier = "cp36-manylinux_aarch64", version = "3.6", path_str = "/opt/python/cp36-cp36m" },
{ identifier = "cp37-manylinux_aarch64", version = "3.7", path_str = "/opt/python/cp37-cp37m" }, { identifier = "cp37-manylinux_aarch64", version = "3.7", path_str = "/opt/python/cp37-cp37m" },
{ identifier = "cp38-manylinux_aarch64", version = "3.8", path_str = "/opt/python/cp38-cp38" }, { identifier = "cp38-manylinux_aarch64", version = "3.8", path_str = "/opt/python/cp38-cp38" },
{ identifier = "cp39-manylinux_aarch64", version = "3.9", path_str = "/opt/python/cp39-cp39" }, { identifier = "cp39-manylinux_aarch64", version = "3.9", path_str = "/opt/python/cp39-cp39" },
{ identifier = "cp310-manylinux_aarch64", version = "3.10", path_str = "/opt/python/cp310-cp310" }, { identifier = "cp35-manylinux_ppc64le", version = "3.5", path_str = "/opt/python/cp35-cp35m" },
{ identifier = "cp311-manylinux_aarch64", version = "3.11", path_str = "/opt/python/cp311-cp311" },
{ identifier = "cp312-manylinux_aarch64", version = "3.12", path_str = "/opt/python/cp312-cp312" },
{ identifier = "cp313-manylinux_aarch64", version = "3.13", path_str = "/opt/python/cp313-cp313" },
{ identifier = "cp313t-manylinux_aarch64", version = "3.13", path_str = "/opt/python/cp313-cp313t" },
{ identifier = "cp36-manylinux_ppc64le", version = "3.6", path_str = "/opt/python/cp36-cp36m" }, { identifier = "cp36-manylinux_ppc64le", version = "3.6", path_str = "/opt/python/cp36-cp36m" },
{ identifier = "cp37-manylinux_ppc64le", version = "3.7", path_str = "/opt/python/cp37-cp37m" }, { identifier = "cp37-manylinux_ppc64le", version = "3.7", path_str = "/opt/python/cp37-cp37m" },
{ identifier = "cp38-manylinux_ppc64le", version = "3.8", path_str = "/opt/python/cp38-cp38" }, { identifier = "cp38-manylinux_ppc64le", version = "3.8", path_str = "/opt/python/cp38-cp38" },
{ identifier = "cp39-manylinux_ppc64le", version = "3.9", path_str = "/opt/python/cp39-cp39" }, { identifier = "cp39-manylinux_ppc64le", version = "3.9", path_str = "/opt/python/cp39-cp39" },
{ identifier = "cp310-manylinux_ppc64le", version = "3.10", path_str = "/opt/python/cp310-cp310" }, { identifier = "cp35-manylinux_s390x", version = "3.5", path_str = "/opt/python/cp35-cp35m" },
{ identifier = "cp311-manylinux_ppc64le", version = "3.11", path_str = "/opt/python/cp311-cp311" },
{ identifier = "cp312-manylinux_ppc64le", version = "3.12", path_str = "/opt/python/cp312-cp312" },
{ identifier = "cp313-manylinux_ppc64le", version = "3.13", path_str = "/opt/python/cp313-cp313" },
{ identifier = "cp313t-manylinux_ppc64le", version = "3.13", path_str = "/opt/python/cp313-cp313t" },
{ identifier = "cp36-manylinux_s390x", version = "3.6", path_str = "/opt/python/cp36-cp36m" }, { identifier = "cp36-manylinux_s390x", version = "3.6", path_str = "/opt/python/cp36-cp36m" },
{ identifier = "cp37-manylinux_s390x", version = "3.7", path_str = "/opt/python/cp37-cp37m" }, { identifier = "cp37-manylinux_s390x", version = "3.7", path_str = "/opt/python/cp37-cp37m" },
{ identifier = "cp38-manylinux_s390x", version = "3.8", path_str = "/opt/python/cp38-cp38" }, { identifier = "cp38-manylinux_s390x", version = "3.8", path_str = "/opt/python/cp38-cp38" },
{ identifier = "cp39-manylinux_s390x", version = "3.9", path_str = "/opt/python/cp39-cp39" }, { identifier = "cp39-manylinux_s390x", version = "3.9", path_str = "/opt/python/cp39-cp39" },
{ identifier = "cp310-manylinux_s390x", version = "3.10", path_str = "/opt/python/cp310-cp310" },
{ identifier = "cp311-manylinux_s390x", version = "3.11", path_str = "/opt/python/cp311-cp311" },
{ identifier = "cp312-manylinux_s390x", version = "3.12", path_str = "/opt/python/cp312-cp312" },
{ identifier = "cp313-manylinux_s390x", version = "3.13", path_str = "/opt/python/cp313-cp313" },
{ identifier = "cp313t-manylinux_s390x", version = "3.13", path_str = "/opt/python/cp313-cp313t" },
{ identifier = "pp37-manylinux_aarch64", version = "3.7", path_str = "/opt/python/pp37-pypy37_pp73" },
{ identifier = "pp38-manylinux_aarch64", version = "3.8", path_str = "/opt/python/pp38-pypy38_pp73" },
{ identifier = "pp39-manylinux_aarch64", version = "3.9", path_str = "/opt/python/pp39-pypy39_pp73" },
{ identifier = "pp310-manylinux_aarch64", version = "3.10", path_str = "/opt/python/pp310-pypy310_pp73" },
{ identifier = "pp37-manylinux_i686", version = "3.7", path_str = "/opt/python/pp37-pypy37_pp73" },
{ identifier = "pp38-manylinux_i686", version = "3.8", path_str = "/opt/python/pp38-pypy38_pp73" },
{ identifier = "pp39-manylinux_i686", version = "3.9", path_str = "/opt/python/pp39-pypy39_pp73" },
{ identifier = "pp310-manylinux_i686", version = "3.10", path_str = "/opt/python/pp310-pypy310_pp73" },
{ identifier = "cp36-musllinux_x86_64", version = "3.6", path_str = "/opt/python/cp36-cp36m" },
{ identifier = "cp37-musllinux_x86_64", version = "3.7", path_str = "/opt/python/cp37-cp37m" },
{ identifier = "cp38-musllinux_x86_64", version = "3.8", path_str = "/opt/python/cp38-cp38" },
{ identifier = "cp39-musllinux_x86_64", version = "3.9", path_str = "/opt/python/cp39-cp39" },
{ identifier = "cp310-musllinux_x86_64", version = "3.10", path_str = "/opt/python/cp310-cp310" },
{ identifier = "cp311-musllinux_x86_64", version = "3.11", path_str = "/opt/python/cp311-cp311" },
{ identifier = "cp312-musllinux_x86_64", version = "3.12", path_str = "/opt/python/cp312-cp312" },
{ identifier = "cp313-musllinux_x86_64", version = "3.13", path_str = "/opt/python/cp313-cp313" },
{ identifier = "cp313t-musllinux_x86_64", version = "3.13", path_str = "/opt/python/cp313-cp313t" },
{ identifier = "cp36-musllinux_i686", version = "3.6", path_str = "/opt/python/cp36-cp36m" },
{ identifier = "cp37-musllinux_i686", version = "3.7", path_str = "/opt/python/cp37-cp37m" },
{ identifier = "cp38-musllinux_i686", version = "3.8", path_str = "/opt/python/cp38-cp38" },
{ identifier = "cp39-musllinux_i686", version = "3.9", path_str = "/opt/python/cp39-cp39" },
{ identifier = "cp310-musllinux_i686", version = "3.10", path_str = "/opt/python/cp310-cp310" },
{ identifier = "cp311-musllinux_i686", version = "3.11", path_str = "/opt/python/cp311-cp311" },
{ identifier = "cp312-musllinux_i686", version = "3.12", path_str = "/opt/python/cp312-cp312" },
{ identifier = "cp313-musllinux_i686", version = "3.13", path_str = "/opt/python/cp313-cp313" },
{ identifier = "cp313t-musllinux_i686", version = "3.13", path_str = "/opt/python/cp313-cp313t" },
{ identifier = "cp36-musllinux_aarch64", version = "3.6", path_str = "/opt/python/cp36-cp36m" },
{ identifier = "cp37-musllinux_aarch64", version = "3.7", path_str = "/opt/python/cp37-cp37m" },
{ identifier = "cp38-musllinux_aarch64", version = "3.8", path_str = "/opt/python/cp38-cp38" },
{ identifier = "cp39-musllinux_aarch64", version = "3.9", path_str = "/opt/python/cp39-cp39" },
{ identifier = "cp310-musllinux_aarch64", version = "3.10", path_str = "/opt/python/cp310-cp310" },
{ identifier = "cp311-musllinux_aarch64", version = "3.11", path_str = "/opt/python/cp311-cp311" },
{ identifier = "cp312-musllinux_aarch64", version = "3.12", path_str = "/opt/python/cp312-cp312" },
{ identifier = "cp313-musllinux_aarch64", version = "3.13", path_str = "/opt/python/cp313-cp313" },
{ identifier = "cp313t-musllinux_aarch64", version = "3.13", path_str = "/opt/python/cp313-cp313t" },
{ identifier = "cp36-musllinux_ppc64le", version = "3.6", path_str = "/opt/python/cp36-cp36m" },
{ identifier = "cp37-musllinux_ppc64le", version = "3.7", path_str = "/opt/python/cp37-cp37m" },
{ identifier = "cp38-musllinux_ppc64le", version = "3.8", path_str = "/opt/python/cp38-cp38" },
{ identifier = "cp39-musllinux_ppc64le", version = "3.9", path_str = "/opt/python/cp39-cp39" },
{ identifier = "cp310-musllinux_ppc64le", version = "3.10", path_str = "/opt/python/cp310-cp310" },
{ identifier = "cp311-musllinux_ppc64le", version = "3.11", path_str = "/opt/python/cp311-cp311" },
{ identifier = "cp312-musllinux_ppc64le", version = "3.12", path_str = "/opt/python/cp312-cp312" },
{ identifier = "cp313-musllinux_ppc64le", version = "3.13", path_str = "/opt/python/cp313-cp313" },
{ identifier = "cp313t-musllinux_ppc64le", version = "3.13", path_str = "/opt/python/cp313-cp313t" },
{ identifier = "cp36-musllinux_s390x", version = "3.6", path_str = "/opt/python/cp36-cp36m" },
{ identifier = "cp37-musllinux_s390x", version = "3.7", path_str = "/opt/python/cp37-cp37m" },
{ identifier = "cp38-musllinux_s390x", version = "3.8", path_str = "/opt/python/cp38-cp38" },
{ identifier = "cp39-musllinux_s390x", version = "3.9", path_str = "/opt/python/cp39-cp39" },
{ identifier = "cp310-musllinux_s390x", version = "3.10", path_str = "/opt/python/cp310-cp310" },
{ identifier = "cp311-musllinux_s390x", version = "3.11", path_str = "/opt/python/cp311-cp311" },
{ identifier = "cp312-musllinux_s390x", version = "3.12", path_str = "/opt/python/cp312-cp312" },
{ identifier = "cp313-musllinux_s390x", version = "3.13", path_str = "/opt/python/cp313-cp313" },
{ identifier = "cp313t-musllinux_s390x", version = "3.13", path_str = "/opt/python/cp313-cp313t" },
] ]
[macos] [macos]
python_configurations = [ python_configurations = [
{ identifier = "cp27-macosx_x86_64", version = "2.7", url = "https://www.python.org/ftp/python/2.7.18/python-2.7.18-macosx10.9.pkg" },
{ identifier = "cp35-macosx_x86_64", version = "3.5", url = "https://www.python.org/ftp/python/3.5.4/python-3.5.4-macosx10.6.pkg" },
{ identifier = "cp36-macosx_x86_64", version = "3.6", url = "https://www.python.org/ftp/python/3.6.8/python-3.6.8-macosx10.9.pkg" }, { identifier = "cp36-macosx_x86_64", version = "3.6", url = "https://www.python.org/ftp/python/3.6.8/python-3.6.8-macosx10.9.pkg" },
{ identifier = "cp37-macosx_x86_64", version = "3.7", url = "https://www.python.org/ftp/python/3.7.9/python-3.7.9-macosx10.9.pkg" }, { identifier = "cp37-macosx_x86_64", version = "3.7", url = "https://www.python.org/ftp/python/3.7.9/python-3.7.9-macosx10.9.pkg" },
{ identifier = "cp38-macosx_x86_64", version = "3.8", url = "https://www.python.org/ftp/python/3.8.10/python-3.8.10-macosx10.9.pkg" }, { identifier = "cp38-macosx_x86_64", version = "3.8", url = "https://www.python.org/ftp/python/3.8.10/python-3.8.10-macosx10.9.pkg" },
{ identifier = "cp38-macosx_arm64", version = "3.8", url = "https://www.python.org/ftp/python/3.8.10/python-3.8.10-macosx10.9.pkg" }, { identifier = "cp38-macosx_arm64", version = "3.8", url = "https://www.python.org/ftp/python/3.8.10/python-3.8.10-macosx10.9.pkg" },
{ identifier = "cp38-macosx_universal2", version = "3.8", url = "https://www.python.org/ftp/python/3.8.10/python-3.8.10-macosx10.9.pkg" }, { identifier = "cp38-macosx_universal2", version = "3.8", url = "https://www.python.org/ftp/python/3.8.10/python-3.8.10-macosx10.9.pkg" },
{ identifier = "cp39-macosx_x86_64", version = "3.9", url = "https://www.python.org/ftp/python/3.9.13/python-3.9.13-macos11.pkg" }, { identifier = "cp39-macosx_x86_64", version = "3.9", url = "https://www.python.org/ftp/python/3.9.5/python-3.9.5-macos11.pkg" },
{ identifier = "cp39-macosx_arm64", version = "3.9", url = "https://www.python.org/ftp/python/3.9.13/python-3.9.13-macos11.pkg" }, { identifier = "cp39-macosx_arm64", version = "3.9", url = "https://www.python.org/ftp/python/3.9.5/python-3.9.5-macos11.pkg" },
{ identifier = "cp39-macosx_universal2", version = "3.9", url = "https://www.python.org/ftp/python/3.9.13/python-3.9.13-macos11.pkg" }, { identifier = "cp39-macosx_universal2", version = "3.9", url = "https://www.python.org/ftp/python/3.9.5/python-3.9.5-macos11.pkg" },
{ identifier = "cp310-macosx_x86_64", version = "3.10", url = "https://www.python.org/ftp/python/3.10.11/python-3.10.11-macos11.pkg" }, { identifier = "pp27-macosx_x86_64", version = "2.7", url = "https://downloads.python.org/pypy/pypy2.7-v7.3.3-osx64.tar.bz2" },
{ identifier = "cp310-macosx_arm64", version = "3.10", url = "https://www.python.org/ftp/python/3.10.11/python-3.10.11-macos11.pkg" }, { identifier = "pp36-macosx_x86_64", version = "3.6", url = "https://downloads.python.org/pypy/pypy3.6-v7.3.3-osx64.tar.bz2" },
{ identifier = "cp310-macosx_universal2", version = "3.10", url = "https://www.python.org/ftp/python/3.10.11/python-3.10.11-macos11.pkg" }, { identifier = "pp37-macosx_x86_64", version = "3.7", url = "https://downloads.python.org/pypy/pypy3.7-v7.3.3-osx64.tar.bz2" },
{ identifier = "cp311-macosx_x86_64", version = "3.11", url = "https://www.python.org/ftp/python/3.11.9/python-3.11.9-macos11.pkg" },
{ identifier = "cp311-macosx_arm64", version = "3.11", url = "https://www.python.org/ftp/python/3.11.9/python-3.11.9-macos11.pkg" },
{ identifier = "cp311-macosx_universal2", version = "3.11", url = "https://www.python.org/ftp/python/3.11.9/python-3.11.9-macos11.pkg" },
{ identifier = "cp312-macosx_x86_64", version = "3.12", url = "https://www.python.org/ftp/python/3.12.5/python-3.12.5-macos11.pkg" },
{ identifier = "cp312-macosx_arm64", version = "3.12", url = "https://www.python.org/ftp/python/3.12.5/python-3.12.5-macos11.pkg" },
{ identifier = "cp312-macosx_universal2", version = "3.12", url = "https://www.python.org/ftp/python/3.12.5/python-3.12.5-macos11.pkg" },
{ identifier = "cp313-macosx_x86_64", version = "3.13", url = "https://www.python.org/ftp/python/3.13.0/python-3.13.0rc1-macos11.pkg" },
{ identifier = "cp313-macosx_arm64", version = "3.13", url = "https://www.python.org/ftp/python/3.13.0/python-3.13.0rc1-macos11.pkg" },
{ identifier = "cp313-macosx_universal2", version = "3.13", url = "https://www.python.org/ftp/python/3.13.0/python-3.13.0rc1-macos11.pkg" },
{ identifier = "cp313t-macosx_x86_64", version = "3.13", url = "https://www.python.org/ftp/python/3.13.0/python-3.13.0rc1-macos11.pkg" },
{ identifier = "cp313t-macosx_arm64", version = "3.13", url = "https://www.python.org/ftp/python/3.13.0/python-3.13.0rc1-macos11.pkg" },
{ identifier = "cp313t-macosx_universal2", version = "3.13", url = "https://www.python.org/ftp/python/3.13.0/python-3.13.0rc1-macos11.pkg" },
{ identifier = "pp37-macosx_x86_64", version = "3.7", url = "https://downloads.python.org/pypy/pypy3.7-v7.3.9-osx64.tar.bz2" },
{ identifier = "pp38-macosx_x86_64", version = "3.8", url = "https://downloads.python.org/pypy/pypy3.8-v7.3.11-macos_x86_64.tar.bz2" },
{ identifier = "pp38-macosx_arm64", version = "3.8", url = "https://downloads.python.org/pypy/pypy3.8-v7.3.11-macos_arm64.tar.bz2" },
{ identifier = "pp39-macosx_x86_64", version = "3.9", url = "https://downloads.python.org/pypy/pypy3.9-v7.3.16-macos_x86_64.tar.bz2" },
{ identifier = "pp39-macosx_arm64", version = "3.9", url = "https://downloads.python.org/pypy/pypy3.9-v7.3.16-macos_arm64.tar.bz2" },
{ identifier = "pp310-macosx_x86_64", version = "3.10", url = "https://downloads.python.org/pypy/pypy3.10-v7.3.17-macos_x86_64.tar.bz2" },
{ identifier = "pp310-macosx_arm64", version = "3.10", url = "https://downloads.python.org/pypy/pypy3.10-v7.3.17-macos_arm64.tar.bz2" },
] ]
[windows] [windows]
python_configurations = [ python_configurations = [
{ identifier = "cp27-win32", version = "2.7.18", arch = "32" },
{ identifier = "cp27-win_amd64", version = "2.7.18", arch = "64" },
{ identifier = "cp35-win32", version = "3.5.4", arch = "32" },
{ identifier = "cp35-win_amd64", version = "3.5.4", arch = "64" },
{ identifier = "cp36-win32", version = "3.6.8", arch = "32" }, { identifier = "cp36-win32", version = "3.6.8", arch = "32" },
{ identifier = "cp36-win_amd64", version = "3.6.8", arch = "64" }, { identifier = "cp36-win_amd64", version = "3.6.8", arch = "64" },
{ identifier = "cp37-win32", version = "3.7.9", arch = "32" }, { identifier = "cp37-win32", version = "3.7.9", arch = "32" },
{ identifier = "cp37-win_amd64", version = "3.7.9", arch = "64" }, { identifier = "cp37-win_amd64", version = "3.7.9", arch = "64" },
{ identifier = "cp38-win32", version = "3.8.10", arch = "32" }, { identifier = "cp38-win32", version = "3.8.9", arch = "32" },
{ identifier = "cp38-win_amd64", version = "3.8.10", arch = "64" }, { identifier = "cp38-win_amd64", version = "3.8.9", arch = "64" },
{ identifier = "cp39-win32", version = "3.9.13", arch = "32" }, { identifier = "cp39-win32", version = "3.9.4", arch = "32" },
{ identifier = "cp39-win_amd64", version = "3.9.13", arch = "64" }, { identifier = "cp39-win_amd64", version = "3.9.4", arch = "64" },
{ identifier = "cp310-win32", version = "3.10.11", arch = "32" }, { identifier = "pp27-win32", version = "2.7", arch = "32", url = "https://downloads.python.org/pypy/pypy2.7-v7.3.3-win32.zip" },
{ identifier = "cp310-win_amd64", version = "3.10.11", arch = "64" }, { identifier = "pp36-win32", version = "3.6", arch = "32", url = "https://downloads.python.org/pypy/pypy3.6-v7.3.3-win32.zip" },
{ identifier = "cp311-win32", version = "3.11.9", arch = "32" }, { identifier = "pp37-win32", version = "3.7", arch = "32", url = "https://downloads.python.org/pypy/pypy3.7-v7.3.3-win32.zip" },
{ identifier = "cp311-win_amd64", version = "3.11.9", arch = "64" },
{ identifier = "cp312-win32", version = "3.12.5", arch = "32" },
{ identifier = "cp312-win_amd64", version = "3.12.5", arch = "64" },
{ identifier = "cp313-win32", version = "3.13.0-rc1", arch = "32" },
{ identifier = "cp313t-win32", version = "3.13.0-rc1", arch = "32" },
{ identifier = "cp313-win_amd64", version = "3.13.0-rc1", arch = "64" },
{ identifier = "cp313t-win_amd64", version = "3.13.0-rc1", arch = "64" },
{ identifier = "cp39-win_arm64", version = "3.9.10", arch = "ARM64" },
{ identifier = "cp310-win_arm64", version = "3.10.11", arch = "ARM64" },
{ identifier = "cp311-win_arm64", version = "3.11.9", arch = "ARM64" },
{ identifier = "cp312-win_arm64", version = "3.12.5", arch = "ARM64" },
{ identifier = "cp313-win_arm64", version = "3.13.0-rc1", arch = "ARM64" },
{ identifier = "cp313t-win_arm64", version = "3.13.0-rc1", arch = "ARM64" },
{ identifier = "pp37-win_amd64", version = "3.7", arch = "64", url = "https://downloads.python.org/pypy/pypy3.7-v7.3.9-win64.zip" },
{ identifier = "pp38-win_amd64", version = "3.8", arch = "64", url = "https://downloads.python.org/pypy/pypy3.8-v7.3.11-win64.zip" },
{ identifier = "pp39-win_amd64", version = "3.9", arch = "64", url = "https://downloads.python.org/pypy/pypy3.9-v7.3.16-win64.zip" },
{ identifier = "pp310-win_amd64", version = "3.10", arch = "64", url = "https://downloads.python.org/pypy/pypy3.10-v7.3.17-win64.zip" },
]
[pyodide]
python_configurations = [
{ identifier = "cp312-pyodide_wasm32", version = "3.12.1", pyodide_version = "0.26.1", emscripten_version = "3.1.58", node_version = "v20" },
] ]
@@ -1,821 +0,0 @@
{
"$schema": "http://json-schema.org/draft-07/schema",
"$id": "https://github.com/pypa/cibuildwheel/blob/main/cibuildwheel/resources/cibuildwheel.schema.json",
"$defs": {
"inherit": {
"enum": [
"none",
"prepend",
"append"
],
"default": "none",
"description": "How to inherit the parent's value."
}
},
"additionalProperties": false,
"description": "cibuildwheel's settings.",
"type": "object",
"properties": {
"archs": {
"description": "Change the architectures built on your machine by default.",
"oneOf": [
{
"type": "string"
},
{
"type": "array",
"items": {
"type": "string"
}
}
],
"title": "CIBW_ARCHS"
},
"before-all": {
"description": "Execute a shell command on the build system before any wheels are built.",
"oneOf": [
{
"type": "string"
},
{
"type": "array",
"items": {
"type": "string"
}
}
],
"title": "CIBW_BEFORE_ALL"
},
"before-build": {
"description": "Execute a shell command preparing each wheel's build.",
"oneOf": [
{
"type": "string"
},
{
"type": "array",
"items": {
"type": "string"
}
}
],
"title": "CIBW_BEFORE_BUILD"
},
"before-test": {
"description": "Execute a shell command before testing each wheel.",
"oneOf": [
{
"type": "string"
},
{
"type": "array",
"items": {
"type": "string"
}
}
],
"title": "CIBW_BEFORE_TEST"
},
"build": {
"default": [
"*"
],
"description": "Choose the Python versions to build.",
"oneOf": [
{
"type": "string"
},
{
"type": "array",
"items": {
"type": "string"
}
}
],
"title": "CIBW_BUILD"
},
"build-frontend": {
"default": "default",
"description": "Set the tool to use to build, either \"pip\" (default for now), \"build\", or \"build[uv]\"",
"oneOf": [
{
"enum": [
"pip",
"build",
"build[uv]",
"default"
]
},
{
"type": "string",
"pattern": "^pip; ?args:"
},
{
"type": "string",
"pattern": "^build; ?args:"
},
{
"type": "string",
"pattern": "^build\\[uv\\]; ?args:"
},
{
"type": "object",
"additionalProperties": false,
"required": [
"name"
],
"properties": {
"name": {
"enum": [
"pip",
"build",
"build[uv]"
]
},
"args": {
"type": "array",
"items": {
"type": "string"
}
}
}
}
],
"title": "CIBW_BUILD_FRONTEND"
},
"build-verbosity": {
"type": "integer",
"minimum": -3,
"maximum": 3,
"default": 0,
"description": "Increase/decrease the output of pip wheel.",
"title": "CIBW_BUILD_VERBOSITY"
},
"config-settings": {
"description": "Specify config-settings for the build backend.",
"oneOf": [
{
"type": "string"
},
{
"type": "object",
"additionalProperties": false,
"patternProperties": {
".+": {
"oneOf": [
{
"type": "string"
},
{
"type": "array",
"items": {
"type": "string"
}
}
]
}
}
}
],
"title": "CIBW_CONFIG_SETTINGS"
},
"container-engine": {
"oneOf": [
{
"enum": [
"docker",
"podman"
]
},
{
"type": "string",
"pattern": "^docker; ?(create_args|disable_host_mount):"
},
{
"type": "string",
"pattern": "^podman; ?(create_args|disable_host_mount):"
},
{
"type": "object",
"additionalProperties": false,
"required": [
"name"
],
"properties": {
"name": {
"enum": [
"docker",
"podman"
]
},
"create-args": {
"type": "array",
"items": {
"type": "string"
}
},
"disable-host-mount": {
"type": "boolean"
}
}
}
],
"title": "CIBW_CONTAINER_ENGINE"
},
"dependency-versions": {
"default": "pinned",
"description": "Specify how cibuildwheel controls the versions of the tools it uses",
"type": "string",
"title": "CIBW_DEPENDENCY_VERSIONS"
},
"environment": {
"description": "Set environment variables needed during the build.",
"oneOf": [
{
"type": "string"
},
{
"type": "object",
"additionalProperties": false,
"patternProperties": {
".+": {
"type": "string"
}
}
}
],
"title": "CIBW_ENVIRONMENT"
},
"environment-pass": {
"description": "Set environment variables on the host to pass-through to the container during the build.",
"oneOf": [
{
"type": "string"
},
{
"type": "array",
"items": {
"type": "string"
}
}
],
"title": "CIBW_ENVIRONMENT_PASS"
},
"free-threaded-support": {
"type": "boolean",
"default": false,
"description": "The project supports free-threaded builds of Python (PEP703)",
"title": "CIBW_FREE_THREADED_SUPPORT"
},
"manylinux-aarch64-image": {
"type": "string",
"description": "Specify alternative manylinux / musllinux container images",
"title": "CIBW_MANYLINUX_AARCH64_IMAGE"
},
"manylinux-i686-image": {
"type": "string",
"description": "Specify alternative manylinux / musllinux container images",
"title": "CIBW_MANYLINUX_I686_IMAGE"
},
"manylinux-ppc64le-image": {
"type": "string",
"description": "Specify alternative manylinux / musllinux container images",
"title": "CIBW_MANYLINUX_PPC64LE_IMAGE"
},
"manylinux-pypy_aarch64-image": {
"type": "string",
"description": "Specify alternative manylinux / musllinux container images",
"title": "CIBW_MANYLINUX_PYPY_AARCH64_IMAGE"
},
"manylinux-pypy_i686-image": {
"type": "string",
"description": "Specify alternative manylinux / musllinux container images",
"title": "CIBW_MANYLINUX_PYPY_I686_IMAGE"
},
"manylinux-pypy_x86_64-image": {
"type": "string",
"description": "Specify alternative manylinux / musllinux container images",
"title": "CIBW_MANYLINUX_PYPY_X86_64_IMAGE"
},
"manylinux-s390x-image": {
"type": "string",
"description": "Specify alternative manylinux / musllinux container images",
"title": "CIBW_MANYLINUX_S390X_IMAGE"
},
"manylinux-x86_64-image": {
"type": "string",
"description": "Specify alternative manylinux / musllinux container images",
"title": "CIBW_MANYLINUX_X86_64_IMAGE"
},
"musllinux-aarch64-image": {
"type": "string",
"description": "Specify alternative manylinux / musllinux container images",
"title": "CIBW_MUSLLINUX_AARCH64_IMAGE"
},
"musllinux-i686-image": {
"type": "string",
"description": "Specify alternative manylinux / musllinux container images",
"title": "CIBW_MUSLLINUX_I686_IMAGE"
},
"musllinux-ppc64le-image": {
"type": "string",
"description": "Specify alternative manylinux / musllinux container images",
"title": "CIBW_MUSLLINUX_PPC64LE_IMAGE"
},
"musllinux-s390x-image": {
"type": "string",
"description": "Specify alternative manylinux / musllinux container images",
"title": "CIBW_MUSLLINUX_S390X_IMAGE"
},
"musllinux-x86_64-image": {
"type": "string",
"description": "Specify alternative manylinux / musllinux container images",
"title": "CIBW_MUSLLINUX_X86_64_IMAGE"
},
"repair-wheel-command": {
"description": "Execute a shell command to repair each built wheel.",
"oneOf": [
{
"type": "string"
},
{
"type": "array",
"items": {
"type": "string"
}
}
],
"title": "CIBW_REPAIR_WHEEL_COMMAND"
},
"skip": {
"description": "Choose the Python versions to skip.",
"oneOf": [
{
"type": "string"
},
{
"type": "array",
"items": {
"type": "string"
}
}
],
"title": "CIBW_SKIP"
},
"test-command": {
"description": "Execute a shell command to test each built wheel.",
"oneOf": [
{
"type": "string"
},
{
"type": "array",
"items": {
"type": "string"
}
}
],
"title": "CIBW_TEST_COMMAND"
},
"test-extras": {
"description": "Install your wheel for testing using `extras_require`",
"oneOf": [
{
"type": "string"
},
{
"type": "array",
"items": {
"type": "string"
}
}
],
"title": "CIBW_TEST_EXTRAS"
},
"test-requires": {
"description": "Install Python dependencies before running the tests",
"oneOf": [
{
"type": "string"
},
{
"type": "array",
"items": {
"type": "string"
}
}
],
"title": "CIBW_TEST_REQUIRES"
},
"test-skip": {
"description": "Skip running tests on some builds.",
"oneOf": [
{
"type": "string"
},
{
"type": "array",
"items": {
"type": "string"
}
}
],
"title": "CIBW_TEST_SKIP"
},
"overrides": {
"type": "array",
"description": "An overrides array",
"items": {
"type": "object",
"required": [
"select"
],
"minProperties": 2,
"additionalProperties": false,
"properties": {
"select": {
"oneOf": [
{
"type": "string"
},
{
"type": "array",
"items": {
"type": "string"
}
}
]
},
"inherit": {
"type": "object",
"additionalProperties": false,
"properties": {
"before-all": {
"$ref": "#/$defs/inherit"
},
"before-build": {
"$ref": "#/$defs/inherit"
},
"before-test": {
"$ref": "#/$defs/inherit"
},
"config-settings": {
"$ref": "#/$defs/inherit"
},
"container-engine": {
"$ref": "#/$defs/inherit"
},
"environment": {
"$ref": "#/$defs/inherit"
},
"environment-pass": {
"$ref": "#/$defs/inherit"
},
"repair-wheel-command": {
"$ref": "#/$defs/inherit"
},
"test-command": {
"$ref": "#/$defs/inherit"
},
"test-extras": {
"$ref": "#/$defs/inherit"
},
"test-requires": {
"$ref": "#/$defs/inherit"
}
}
},
"before-all": {
"$ref": "#/properties/before-all"
},
"before-build": {
"$ref": "#/properties/before-build"
},
"before-test": {
"$ref": "#/properties/before-test"
},
"build-frontend": {
"$ref": "#/properties/build-frontend"
},
"build-verbosity": {
"$ref": "#/properties/build-verbosity"
},
"config-settings": {
"$ref": "#/properties/config-settings"
},
"container-engine": {
"$ref": "#/properties/container-engine"
},
"dependency-versions": {
"$ref": "#/properties/dependency-versions"
},
"environment": {
"$ref": "#/properties/environment"
},
"environment-pass": {
"$ref": "#/properties/environment-pass"
},
"manylinux-aarch64-image": {
"$ref": "#/properties/manylinux-aarch64-image"
},
"manylinux-i686-image": {
"$ref": "#/properties/manylinux-i686-image"
},
"manylinux-ppc64le-image": {
"$ref": "#/properties/manylinux-ppc64le-image"
},
"manylinux-pypy_aarch64-image": {
"$ref": "#/properties/manylinux-pypy_aarch64-image"
},
"manylinux-pypy_i686-image": {
"$ref": "#/properties/manylinux-pypy_i686-image"
},
"manylinux-pypy_x86_64-image": {
"$ref": "#/properties/manylinux-pypy_x86_64-image"
},
"manylinux-s390x-image": {
"$ref": "#/properties/manylinux-s390x-image"
},
"manylinux-x86_64-image": {
"$ref": "#/properties/manylinux-x86_64-image"
},
"musllinux-aarch64-image": {
"$ref": "#/properties/musllinux-aarch64-image"
},
"musllinux-i686-image": {
"$ref": "#/properties/musllinux-i686-image"
},
"musllinux-ppc64le-image": {
"$ref": "#/properties/musllinux-ppc64le-image"
},
"musllinux-s390x-image": {
"$ref": "#/properties/musllinux-s390x-image"
},
"musllinux-x86_64-image": {
"$ref": "#/properties/musllinux-x86_64-image"
},
"repair-wheel-command": {
"$ref": "#/properties/repair-wheel-command"
},
"test-command": {
"$ref": "#/properties/test-command"
},
"test-extras": {
"$ref": "#/properties/test-extras"
},
"test-requires": {
"$ref": "#/properties/test-requires"
}
}
}
},
"linux": {
"type": "object",
"additionalProperties": false,
"properties": {
"archs": {
"$ref": "#/properties/archs"
},
"before-all": {
"$ref": "#/properties/before-all"
},
"before-build": {
"$ref": "#/properties/before-build"
},
"before-test": {
"$ref": "#/properties/before-test"
},
"build-frontend": {
"$ref": "#/properties/build-frontend"
},
"build-verbosity": {
"$ref": "#/properties/build-verbosity"
},
"config-settings": {
"$ref": "#/properties/config-settings"
},
"container-engine": {
"$ref": "#/properties/container-engine"
},
"environment": {
"$ref": "#/properties/environment"
},
"environment-pass": {
"$ref": "#/properties/environment-pass"
},
"manylinux-aarch64-image": {
"$ref": "#/properties/manylinux-aarch64-image"
},
"manylinux-i686-image": {
"$ref": "#/properties/manylinux-i686-image"
},
"manylinux-ppc64le-image": {
"$ref": "#/properties/manylinux-ppc64le-image"
},
"manylinux-pypy_aarch64-image": {
"$ref": "#/properties/manylinux-pypy_aarch64-image"
},
"manylinux-pypy_i686-image": {
"$ref": "#/properties/manylinux-pypy_i686-image"
},
"manylinux-pypy_x86_64-image": {
"$ref": "#/properties/manylinux-pypy_x86_64-image"
},
"manylinux-s390x-image": {
"$ref": "#/properties/manylinux-s390x-image"
},
"manylinux-x86_64-image": {
"$ref": "#/properties/manylinux-x86_64-image"
},
"musllinux-aarch64-image": {
"$ref": "#/properties/musllinux-aarch64-image"
},
"musllinux-i686-image": {
"$ref": "#/properties/musllinux-i686-image"
},
"musllinux-ppc64le-image": {
"$ref": "#/properties/musllinux-ppc64le-image"
},
"musllinux-s390x-image": {
"$ref": "#/properties/musllinux-s390x-image"
},
"musllinux-x86_64-image": {
"$ref": "#/properties/musllinux-x86_64-image"
},
"repair-wheel-command": {
"description": "Execute a shell command to repair each built wheel.",
"oneOf": [
{
"type": "string"
},
{
"type": "array",
"items": {
"type": "string"
}
}
],
"title": "CIBW_REPAIR_WHEEL_COMMAND",
"default": "auditwheel repair -w {dest_dir} {wheel}"
},
"test-command": {
"$ref": "#/properties/test-command"
},
"test-extras": {
"$ref": "#/properties/test-extras"
},
"test-requires": {
"$ref": "#/properties/test-requires"
}
}
},
"windows": {
"type": "object",
"additionalProperties": false,
"properties": {
"archs": {
"$ref": "#/properties/archs"
},
"before-all": {
"$ref": "#/properties/before-all"
},
"before-build": {
"$ref": "#/properties/before-build"
},
"before-test": {
"$ref": "#/properties/before-test"
},
"build-frontend": {
"$ref": "#/properties/build-frontend"
},
"build-verbosity": {
"$ref": "#/properties/build-verbosity"
},
"config-settings": {
"$ref": "#/properties/config-settings"
},
"dependency-versions": {
"$ref": "#/properties/dependency-versions"
},
"environment": {
"$ref": "#/properties/environment"
},
"repair-wheel-command": {
"$ref": "#/properties/repair-wheel-command"
},
"test-command": {
"$ref": "#/properties/test-command"
},
"test-extras": {
"$ref": "#/properties/test-extras"
},
"test-requires": {
"$ref": "#/properties/test-requires"
}
}
},
"macos": {
"type": "object",
"additionalProperties": false,
"properties": {
"archs": {
"$ref": "#/properties/archs"
},
"before-all": {
"$ref": "#/properties/before-all"
},
"before-build": {
"$ref": "#/properties/before-build"
},
"before-test": {
"$ref": "#/properties/before-test"
},
"build-frontend": {
"$ref": "#/properties/build-frontend"
},
"build-verbosity": {
"$ref": "#/properties/build-verbosity"
},
"config-settings": {
"$ref": "#/properties/config-settings"
},
"dependency-versions": {
"$ref": "#/properties/dependency-versions"
},
"environment": {
"$ref": "#/properties/environment"
},
"repair-wheel-command": {
"description": "Execute a shell command to repair each built wheel.",
"oneOf": [
{
"type": "string"
},
{
"type": "array",
"items": {
"type": "string"
}
}
],
"title": "CIBW_REPAIR_WHEEL_COMMAND",
"default": "delocate-wheel --require-archs {delocate_archs} -w {dest_dir} -v {wheel}"
},
"test-command": {
"$ref": "#/properties/test-command"
},
"test-extras": {
"$ref": "#/properties/test-extras"
},
"test-requires": {
"$ref": "#/properties/test-requires"
}
}
},
"pyodide": {
"type": "object",
"additionalProperties": false,
"properties": {
"archs": {
"$ref": "#/properties/archs"
},
"before-all": {
"$ref": "#/properties/before-all"
},
"before-build": {
"$ref": "#/properties/before-build"
},
"before-test": {
"$ref": "#/properties/before-test"
},
"build-frontend": {
"$ref": "#/properties/build-frontend"
},
"build-verbosity": {
"$ref": "#/properties/build-verbosity"
},
"config-settings": {
"$ref": "#/properties/config-settings"
},
"dependency-versions": {
"$ref": "#/properties/dependency-versions"
},
"environment": {
"$ref": "#/properties/environment"
},
"repair-wheel-command": {
"$ref": "#/properties/repair-wheel-command"
},
"test-command": {
"$ref": "#/properties/test-command"
},
"test-extras": {
"$ref": "#/properties/test-extras"
},
"test-requires": {
"$ref": "#/properties/test-requires"
}
}
}
}
}
@@ -1,123 +0,0 @@
# This file was autogenerated by uv via the following command:
# nox -s update_constraints
annotated-types==0.7.0
# via pydantic
anyio==4.4.0
# via httpx
auditwheel-emscripten==0.0.16
# via pyodide-build
build==1.2.1
# via
# -r .nox/update_constraints/tmp/constraints-pyodide.in
# pyodide-build
certifi==2024.8.30
# via
# httpcore
# httpx
# requests
charset-normalizer==3.3.2
# via requests
click==8.1.7
# via typer
cloudpickle==3.0.0
# via loky
cmake==3.30.2
# via pyodide-build
distlib==0.3.8
# via virtualenv
filelock==3.15.4
# via virtualenv
h11==0.14.0
# via httpcore
httpcore==1.0.5
# via httpx
httpx==0.27.2
# via unearth
idna==3.8
# via
# anyio
# httpx
# requests
leb128==1.0.8
# via auditwheel-emscripten
loky==3.4.1
# via pyodide-build
markdown-it-py==3.0.0
# via rich
mdurl==0.1.2
# via markdown-it-py
packaging==24.1
# via
# auditwheel-emscripten
# build
# pyodide-build
# unearth
pip==24.2
# via -r .nox/update_constraints/tmp/constraints-pyodide.in
platformdirs==4.2.2
# via virtualenv
pydantic==2.8.2
# via
# pyodide-build
# pyodide-lock
pydantic-core==2.20.1
# via pydantic
pygments==2.18.0
# via rich
pyodide-build==0.26.1
# via -r .nox/update_constraints/tmp/constraints-pyodide.in
pyodide-cli==0.2.4
# via
# auditwheel-emscripten
# pyodide-build
pyodide-lock==0.1.0a6
# via pyodide-build
pyproject-hooks==1.1.0
# via build
pyyaml==6.0.2
# via pyodide-build
requests==2.32.3
# via pyodide-build
resolvelib==1.0.1
# via pyodide-build
rich==13.8.0
# via
# pyodide-build
# pyodide-cli
# typer
ruamel-yaml==0.18.6
# via pyodide-build
ruamel-yaml-clib==0.2.8
# via ruamel-yaml
shellingham==1.5.4
# via typer
sniffio==1.3.1
# via
# anyio
# httpx
typer==0.12.5
# via
# auditwheel-emscripten
# pyodide-build
# pyodide-cli
types-requests==2.32.0.20240712
# via pyodide-build
typing-extensions==4.12.2
# via
# pydantic
# pydantic-core
# typer
unearth==0.17.2
# via pyodide-build
urllib3==2.2.2
# via
# requests
# types-requests
virtualenv==20.26.3
# via
# build
# pyodide-build
wheel==0.44.0
# via
# auditwheel-emscripten
# pyodide-build
@@ -0,0 +1,55 @@
#
# This file is autogenerated by pip-compile
# To update, run:
#
# bin/update_dependencies.py
#
appdirs==1.4.4
# via virtualenv
configparser==4.0.2
# via importlib-metadata
contextlib2==0.6.0.post1
# via
# importlib-metadata
# importlib-resources
delocate==0.8.2
# via -r cibuildwheel/resources/constraints.in
distlib==0.3.1
# via virtualenv
filelock==3.0.12
# via virtualenv
importlib-metadata==2.1.1
# via virtualenv
importlib-resources==3.3.1
# via virtualenv
pathlib2==2.3.5
# via
# importlib-metadata
# importlib-resources
# virtualenv
scandir==1.10.0
# via pathlib2
singledispatch==3.6.1
# via importlib-resources
six==1.15.0
# via
# pathlib2
# virtualenv
typing==3.7.4.3
# via importlib-resources
virtualenv==20.4.3
# via -r cibuildwheel/resources/constraints.in
wheel==0.36.2
# via
# -r cibuildwheel/resources/constraints.in
# delocate
zipp==1.2.0
# via
# importlib-metadata
# importlib-resources
# The following packages are considered to be unsafe in a requirements file:
pip==20.3.4
# via -r cibuildwheel/resources/constraints.in
setuptools==44.1.1
# via -r cibuildwheel/resources/constraints.in
@@ -1,34 +0,0 @@
# This file was autogenerated by uv via the following command:
# nox -s update_constraints
altgraph==0.17.4
# via macholib
build==1.2.1
# via -r cibuildwheel/resources/constraints.in
delocate==0.12.0
# via -r cibuildwheel/resources/constraints.in
distlib==0.3.8
# via virtualenv
filelock==3.15.4
# via virtualenv
importlib-metadata==8.4.0
# via build
macholib==1.16.3
# via delocate
packaging==24.1
# via
# build
# delocate
pip==24.2
# via -r cibuildwheel/resources/constraints.in
platformdirs==4.2.2
# via virtualenv
pyproject-hooks==1.1.0
# via build
tomli==2.0.1
# via build
typing-extensions==4.12.2
# via delocate
virtualenv==20.26.3
# via -r cibuildwheel/resources/constraints.in
zipp==3.20.1
# via importlib-metadata
@@ -1,28 +0,0 @@
# This file was autogenerated by uv via the following command:
# nox -s update_constraints
altgraph==0.17.4
# via macholib
build==1.2.1
# via -r cibuildwheel/resources/constraints.in
delocate==0.12.0
# via -r cibuildwheel/resources/constraints.in
distlib==0.3.8
# via virtualenv
filelock==3.15.4
# via virtualenv
macholib==1.16.3
# via delocate
packaging==24.1
# via
# build
# delocate
pip==24.2
# via -r cibuildwheel/resources/constraints.in
platformdirs==4.2.2
# via virtualenv
pyproject-hooks==1.1.0
# via build
typing-extensions==4.12.2
# via delocate
virtualenv==20.26.3
# via -r cibuildwheel/resources/constraints.in
@@ -1,28 +0,0 @@
# This file was autogenerated by uv via the following command:
# nox -s update_constraints
altgraph==0.17.4
# via macholib
build==1.2.1
# via -r cibuildwheel/resources/constraints.in
delocate==0.12.0
# via -r cibuildwheel/resources/constraints.in
distlib==0.3.8
# via virtualenv
filelock==3.15.4
# via virtualenv
macholib==1.16.3
# via delocate
packaging==24.1
# via
# build
# delocate
pip==24.2
# via -r cibuildwheel/resources/constraints.in
platformdirs==4.2.2
# via virtualenv
pyproject-hooks==1.1.0
# via build
typing-extensions==4.12.2
# via delocate
virtualenv==20.26.3
# via -r cibuildwheel/resources/constraints.in
@@ -1,28 +0,0 @@
# This file was autogenerated by uv via the following command:
# nox -s update_constraints
altgraph==0.17.4
# via macholib
build==1.2.1
# via -r cibuildwheel/resources/constraints.in
delocate==0.12.0
# via -r cibuildwheel/resources/constraints.in
distlib==0.3.8
# via virtualenv
filelock==3.15.4
# via virtualenv
macholib==1.16.3
# via delocate
packaging==24.1
# via
# build
# delocate
pip==24.2
# via -r cibuildwheel/resources/constraints.in
platformdirs==4.2.2
# via virtualenv
pyproject-hooks==1.1.0
# via build
typing-extensions==4.12.2
# via delocate
virtualenv==20.26.3
# via -r cibuildwheel/resources/constraints.in
@@ -0,0 +1,36 @@
#
# This file is autogenerated by pip-compile
# To update, run:
#
# bin/update_dependencies.py
#
appdirs==1.4.4
# via virtualenv
delocate==0.8.2
# via -r cibuildwheel/resources/constraints.in
distlib==0.3.1
# via virtualenv
filelock==3.0.12
# via virtualenv
importlib-metadata==2.1.1
# via virtualenv
importlib-resources==3.2.1
# via virtualenv
six==1.15.0
# via virtualenv
virtualenv==20.4.3
# via -r cibuildwheel/resources/constraints.in
wheel==0.36.2
# via
# -r cibuildwheel/resources/constraints.in
# delocate
zipp==1.2.0
# via
# importlib-metadata
# importlib-resources
# The following packages are considered to be unsafe in a requirements file:
pip==20.3.4
# via -r cibuildwheel/resources/constraints.in
setuptools==50.3.2
# via -r cibuildwheel/resources/constraints.in
+23 -37
View File
@@ -1,52 +1,38 @@
# #
# This file is autogenerated by pip-compile with python 3.6 # This file is autogenerated by pip-compile
# To update, run: # To update, run:
# #
# nox -s update_constraints-3.6 # bin/update_dependencies.py
# #
build==0.9.0 appdirs==1.4.4
# via virtualenv
delocate==0.8.2
# via -r cibuildwheel/resources/constraints.in # via -r cibuildwheel/resources/constraints.in
delocate==0.10.2 distlib==0.3.1
# via virtualenv
filelock==3.0.12
# via virtualenv
importlib-metadata==3.10.0
# via virtualenv
importlib-resources==5.1.2
# via virtualenv
six==1.15.0
# via virtualenv
typing-extensions==3.7.4.3
# via importlib-metadata
virtualenv==20.4.3
# via -r cibuildwheel/resources/constraints.in # via -r cibuildwheel/resources/constraints.in
distlib==0.3.8 wheel==0.36.2
# via virtualenv
filelock==3.4.1
# via virtualenv
importlib-metadata==4.8.3
# via
# build
# pep517
# virtualenv
importlib-resources==5.4.0
# via virtualenv
packaging==21.3
# via build
pep517==0.13.1
# via build
platformdirs==2.4.0
# via virtualenv
pyparsing==3.1.1
# via packaging
tomli==1.2.3
# via
# build
# pep517
typing-extensions==4.1.1
# via # via
# -r cibuildwheel/resources/constraints.in
# delocate # delocate
# importlib-metadata zipp==3.4.1
virtualenv==20.17.1
# via -r cibuildwheel/resources/constraints.in
wheel==0.37.1
# via delocate
zipp==3.6.0
# via # via
# importlib-metadata # importlib-metadata
# importlib-resources # importlib-resources
# pep517
# The following packages are considered to be unsafe in a requirements file: # The following packages are considered to be unsafe in a requirements file:
pip==21.3.1 pip==21.0.1
# via -r cibuildwheel/resources/constraints.in # via -r cibuildwheel/resources/constraints.in
setuptools==59.6.0 setuptools==54.2.0
# via -r cibuildwheel/resources/constraints.in # via -r cibuildwheel/resources/constraints.in
+30 -35
View File
@@ -1,39 +1,34 @@
# This file was autogenerated by uv via the following command: #
# nox -s update_constraints # This file is autogenerated by pip-compile
altgraph==0.17.4 # To update, run:
# via macholib #
build==1.1.1 # bin/update_dependencies.py
# via -r cibuildwheel/resources/constraints.in #
delocate==0.12.0 appdirs==1.4.4
# via -r cibuildwheel/resources/constraints.in
distlib==0.3.8
# via virtualenv # via virtualenv
filelock==3.12.2 delocate==0.8.2
# via virtualenv
importlib-metadata==6.7.0
# via
# build
# virtualenv
macholib==1.16.3
# via delocate
packaging==24.0
# via
# build
# delocate
pip==24.0
# via -r cibuildwheel/resources/constraints.in # via -r cibuildwheel/resources/constraints.in
platformdirs==4.0.0 distlib==0.3.1
# via virtualenv # via virtualenv
pyproject-hooks==1.1.0 filelock==3.0.12
# via build # via virtualenv
tomli==2.0.1 importlib-metadata==3.10.0
# via build # via virtualenv
typing-extensions==4.7.1 six==1.15.0
# via # via virtualenv
# delocate typing-extensions==3.7.4.3
# importlib-metadata
# platformdirs
virtualenv==20.26.3
# via -r cibuildwheel/resources/constraints.in
zipp==3.15.0
# via importlib-metadata # via importlib-metadata
virtualenv==20.4.3
# via -r cibuildwheel/resources/constraints.in
wheel==0.36.2
# via
# -r cibuildwheel/resources/constraints.in
# delocate
zipp==3.4.1
# via importlib-metadata
# The following packages are considered to be unsafe in a requirements file:
pip==21.0.1
# via -r cibuildwheel/resources/constraints.in
setuptools==54.2.0
# via -r cibuildwheel/resources/constraints.in
+22 -28
View File
@@ -1,34 +1,28 @@
# This file was autogenerated by uv via the following command: #
# nox -s update_constraints # This file is autogenerated by pip-compile
altgraph==0.17.4 # To update, run:
# via macholib #
build==1.2.1 # bin/update_dependencies.py
# via -r cibuildwheel/resources/constraints.in #
delocate==0.12.0 appdirs==1.4.4
# via -r cibuildwheel/resources/constraints.in
distlib==0.3.8
# via virtualenv # via virtualenv
filelock==3.15.4 delocate==0.8.2
# via -r cibuildwheel/resources/constraints.in
distlib==0.3.1
# via virtualenv # via virtualenv
importlib-metadata==8.4.0 filelock==3.0.12
# via build # via virtualenv
macholib==1.16.3 six==1.15.0
# via delocate # via virtualenv
packaging==24.1 virtualenv==20.4.3
# via -r cibuildwheel/resources/constraints.in
wheel==0.36.2
# via # via
# build # -r cibuildwheel/resources/constraints.in
# delocate # delocate
pip==24.2
# The following packages are considered to be unsafe in a requirements file:
pip==21.0.1
# via -r cibuildwheel/resources/constraints.in # via -r cibuildwheel/resources/constraints.in
platformdirs==4.2.2 setuptools==54.2.0
# via virtualenv
pyproject-hooks==1.1.0
# via build
tomli==2.0.1
# via build
typing-extensions==4.12.2
# via delocate
virtualenv==20.26.3
# via -r cibuildwheel/resources/constraints.in # via -r cibuildwheel/resources/constraints.in
zipp==3.20.1
# via importlib-metadata
+22 -28
View File
@@ -1,34 +1,28 @@
# This file was autogenerated by uv via the following command: #
# nox -s update_constraints # This file is autogenerated by pip-compile
altgraph==0.17.4 # To update, run:
# via macholib #
build==1.2.1 # bin/update_dependencies.py
# via -r cibuildwheel/resources/constraints.in #
delocate==0.12.0 appdirs==1.4.4
# via -r cibuildwheel/resources/constraints.in
distlib==0.3.8
# via virtualenv # via virtualenv
filelock==3.15.4 delocate==0.8.2
# via -r cibuildwheel/resources/constraints.in
distlib==0.3.1
# via virtualenv # via virtualenv
importlib-metadata==8.4.0 filelock==3.0.12
# via build # via virtualenv
macholib==1.16.3 six==1.15.0
# via delocate # via virtualenv
packaging==24.1 virtualenv==20.4.3
# via -r cibuildwheel/resources/constraints.in
wheel==0.36.2
# via # via
# build # -r cibuildwheel/resources/constraints.in
# delocate # delocate
pip==24.2
# The following packages are considered to be unsafe in a requirements file:
pip==21.0.1
# via -r cibuildwheel/resources/constraints.in # via -r cibuildwheel/resources/constraints.in
platformdirs==4.2.2 setuptools==54.2.0
# via virtualenv
pyproject-hooks==1.1.0
# via build
tomli==2.0.1
# via build
typing-extensions==4.12.2
# via delocate
virtualenv==20.26.3
# via -r cibuildwheel/resources/constraints.in # via -r cibuildwheel/resources/constraints.in
zipp==3.20.1
# via importlib-metadata
+2 -1
View File
@@ -1,4 +1,5 @@
pip pip
build setuptools
wheel
delocate delocate
virtualenv virtualenv
+22 -22
View File
@@ -1,28 +1,28 @@
# This file was autogenerated by uv via the following command: #
# nox -s update_constraints # This file is autogenerated by pip-compile
altgraph==0.17.4 # To update, run:
# via macholib #
build==1.2.1 # bin/update_dependencies.py
# via -r cibuildwheel/resources/constraints.in #
delocate==0.12.0 appdirs==1.4.4
# via -r cibuildwheel/resources/constraints.in
distlib==0.3.8
# via virtualenv # via virtualenv
filelock==3.15.4 delocate==0.8.2
# via -r cibuildwheel/resources/constraints.in
distlib==0.3.1
# via virtualenv # via virtualenv
macholib==1.16.3 filelock==3.0.12
# via delocate # via virtualenv
packaging==24.1 six==1.15.0
# via virtualenv
virtualenv==20.4.3
# via -r cibuildwheel/resources/constraints.in
wheel==0.36.2
# via # via
# build # -r cibuildwheel/resources/constraints.in
# delocate # delocate
pip==24.2
# The following packages are considered to be unsafe in a requirements file:
pip==21.0.1
# via -r cibuildwheel/resources/constraints.in # via -r cibuildwheel/resources/constraints.in
platformdirs==4.2.2 setuptools==54.2.0
# via virtualenv
pyproject-hooks==1.1.0
# via build
typing-extensions==4.12.2
# via delocate
virtualenv==20.26.3
# via -r cibuildwheel/resources/constraints.in # via -r cibuildwheel/resources/constraints.in
-50
View File
@@ -1,50 +0,0 @@
[tool.cibuildwheel]
build = "*"
skip = ""
test-skip = ""
free-threaded-support = false
archs = ["auto"]
build-frontend = "default"
config-settings = {}
dependency-versions = "pinned"
environment = {}
environment-pass = []
build-verbosity = 0
before-all = ""
before-build = ""
repair-wheel-command = ""
test-command = ""
before-test = ""
test-requires = []
test-extras = []
container-engine = "docker"
manylinux-x86_64-image = "manylinux2014"
manylinux-i686-image = "manylinux2014"
manylinux-aarch64-image = "manylinux2014"
manylinux-ppc64le-image = "manylinux2014"
manylinux-s390x-image = "manylinux2014"
manylinux-pypy_x86_64-image = "manylinux2014"
manylinux-pypy_i686-image = "manylinux2014"
manylinux-pypy_aarch64-image = "manylinux2014"
musllinux-x86_64-image = "musllinux_1_2"
musllinux-i686-image = "musllinux_1_2"
musllinux-aarch64-image = "musllinux_1_2"
musllinux-ppc64le-image = "musllinux_1_2"
musllinux-s390x-image = "musllinux_1_2"
[tool.cibuildwheel.linux]
repair-wheel-command = "auditwheel repair -w {dest_dir} {wheel}"
[tool.cibuildwheel.macos]
repair-wheel-command = "delocate-wheel --require-archs {delocate_archs} -w {dest_dir} -v {wheel}"
[tool.cibuildwheel.windows]
[tool.cibuildwheel.pyodide]
@@ -1,14 +0,0 @@
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<array>
<dict>
<key>attributeSetting</key>
<integer>1</integer>
<key>choiceAttribute</key>
<string>selected</string>
<key>choiceIdentifier</key>
<string>org.python.Python.PythonTFramework-3.13</string>
</dict>
</array>
</plist>
File diff suppressed because it is too large Load Diff
+6 -2
View File
@@ -6,7 +6,6 @@
# for the ssl module. Uses the certificates provided by the certifi package: # for the ssl module. Uses the certificates provided by the certifi package:
# https://pypi.org/project/certifi/ # https://pypi.org/project/certifi/
import contextlib
import os import os
import os.path import os.path
import ssl import ssl
@@ -25,6 +24,9 @@ STAT_0o775 = (
| stat.S_IXOTH | stat.S_IXOTH
) )
if sys.version_info[0] == 2:
FileNotFoundError = OSError
def main(): def main():
openssl_dir, openssl_cafile = os.path.split(ssl.get_default_verify_paths().openssl_cafile) openssl_dir, openssl_cafile = os.path.split(ssl.get_default_verify_paths().openssl_cafile)
@@ -40,8 +42,10 @@ def main():
relpath_to_certifi_cafile = os.path.relpath(certifi.where()) relpath_to_certifi_cafile = os.path.relpath(certifi.where())
print(" -- removing any existing file or link") print(" -- removing any existing file or link")
with contextlib.suppress(FileNotFoundError): try:
os.remove(openssl_cafile) os.remove(openssl_cafile)
except FileNotFoundError:
pass
print(" -- creating symlink to certifi certificate bundle") print(" -- creating symlink to certifi certificate bundle")
os.symlink(relpath_to_certifi_cafile, openssl_cafile) os.symlink(relpath_to_certifi_cafile, openssl_cafile)
-2
View File
@@ -1,2 +0,0 @@
url = "https://nodejs.org/dist/"
v20 = "v20.17.0"
+15 -43
View File
@@ -1,54 +1,26 @@
[x86_64] [x86_64]
manylinux1 = quay.io/pypa/manylinux1_x86_64:2024-04-29-76807b8 manylinux1 = quay.io/pypa/manylinux1_x86_64:2021-05-05-b64d921
manylinux2010 = quay.io/pypa/manylinux2010_x86_64:2022-08-05-4535177 manylinux2010 = quay.io/pypa/manylinux2010_x86_64:2021-02-06-3d322a5
manylinux2014 = quay.io/pypa/manylinux2014_x86_64:2024.09.01-2 manylinux2014 = quay.io/pypa/manylinux2014_x86_64:2021-05-01-28d233a
manylinux_2_24 = quay.io/pypa/manylinux_2_24_x86_64:2022-12-26-0d38463 manylinux_2_24 = quay.io/pypa/manylinux_2_24_x86_64:2021-05-01-28d233a
manylinux_2_28 = quay.io/pypa/manylinux_2_28_x86_64:2024.09.01-2
musllinux_1_1 = quay.io/pypa/musllinux_1_1_x86_64:2024.09.01-2
musllinux_1_2 = quay.io/pypa/musllinux_1_2_x86_64:2024.09.01-2
[i686] [i686]
manylinux1 = quay.io/pypa/manylinux1_i686:2024-04-29-76807b8 manylinux1 = quay.io/pypa/manylinux1_i686:2021-05-05-b64d921
manylinux2010 = quay.io/pypa/manylinux2010_i686:2022-08-05-4535177 manylinux2010 = quay.io/pypa/manylinux2010_i686:2021-02-06-3d322a5
manylinux2014 = quay.io/pypa/manylinux2014_i686:2024.09.01-2 manylinux2014 = quay.io/pypa/manylinux2014_i686:2021-05-01-28d233a
manylinux_2_24 = quay.io/pypa/manylinux_2_24_i686:2022-12-26-0d38463 manylinux_2_24 = quay.io/pypa/manylinux_2_24_i686:2021-05-01-28d233a
musllinux_1_1 = quay.io/pypa/musllinux_1_1_i686:2024.09.01-2
musllinux_1_2 = quay.io/pypa/musllinux_1_2_i686:2024.09.01-2
[pypy_x86_64] [pypy_x86_64]
manylinux2010 = quay.io/pypa/manylinux2010_x86_64:2022-08-05-4535177 manylinux2010 = pypywheels/manylinux2010-pypy_x86_64:2020-12-11-f1e0e80
manylinux2014 = quay.io/pypa/manylinux2014_x86_64:2024.09.01-2
manylinux_2_24 = quay.io/pypa/manylinux_2_24_x86_64:2022-12-26-0d38463
manylinux_2_28 = quay.io/pypa/manylinux_2_28_x86_64:2024.09.01-2
[pypy_i686]
manylinux2010 = quay.io/pypa/manylinux2010_i686:2022-08-05-4535177
manylinux2014 = quay.io/pypa/manylinux2014_i686:2024.09.01-2
manylinux_2_24 = quay.io/pypa/manylinux_2_24_i686:2022-12-26-0d38463
[aarch64] [aarch64]
manylinux2014 = quay.io/pypa/manylinux2014_aarch64:2024.09.01-2 manylinux2014 = quay.io/pypa/manylinux2014_aarch64:2021-05-01-28d233a
manylinux_2_24 = quay.io/pypa/manylinux_2_24_aarch64:2022-12-26-0d38463 manylinux_2_24 = quay.io/pypa/manylinux_2_24_aarch64:2021-05-01-28d233a
manylinux_2_28 = quay.io/pypa/manylinux_2_28_aarch64:2024.09.01-2
musllinux_1_1 = quay.io/pypa/musllinux_1_1_aarch64:2024.09.01-2
musllinux_1_2 = quay.io/pypa/musllinux_1_2_aarch64:2024.09.01-2
[ppc64le] [ppc64le]
manylinux2014 = quay.io/pypa/manylinux2014_ppc64le:2024.09.01-2 manylinux2014 = quay.io/pypa/manylinux2014_ppc64le:2021-05-01-28d233a
manylinux_2_24 = quay.io/pypa/manylinux_2_24_ppc64le:2022-12-26-0d38463 manylinux_2_24 = quay.io/pypa/manylinux_2_24_ppc64le:2021-05-01-28d233a
manylinux_2_28 = quay.io/pypa/manylinux_2_28_ppc64le:2024.09.01-2
musllinux_1_1 = quay.io/pypa/musllinux_1_1_ppc64le:2024.09.01-2
musllinux_1_2 = quay.io/pypa/musllinux_1_2_ppc64le:2024.09.01-2
[s390x] [s390x]
manylinux2014 = quay.io/pypa/manylinux2014_s390x:2024.09.01-2 manylinux2014 = quay.io/pypa/manylinux2014_s390x:2021-05-01-28d233a
manylinux_2_24 = quay.io/pypa/manylinux_2_24_s390x:2022-12-26-0d38463 manylinux_2_24 = quay.io/pypa/manylinux_2_24_s390x:2021-05-01-28d233a
manylinux_2_28 = quay.io/pypa/manylinux_2_28_s390x:2024.09.01-2
musllinux_1_1 = quay.io/pypa/musllinux_1_1_s390x:2024.09.01-2
musllinux_1_2 = quay.io/pypa/musllinux_1_2_s390x:2024.09.01-2
[pypy_aarch64]
manylinux2014 = quay.io/pypa/manylinux2014_aarch64:2024.09.01-2
manylinux_2_24 = quay.io/pypa/manylinux_2_24_aarch64:2022-12-26-0d38463
manylinux_2_28 = quay.io/pypa/manylinux_2_28_aarch64:2024.09.01-2
+36
View File
@@ -0,0 +1,36 @@
diff --git a/lib-python/3/sysconfig.py b/lib-python/3/sysconfig.py
index 0d08cde66b4150e1772c13fdc6fe34d079c2b10d..30c77d165a2505d98bc7660ff0cc3b728f56c682 100644
--- a/lib-python/3/sysconfig.py
+++ b/lib-python/3/sysconfig.py
@@ -618,9 +618,6 @@ def get_config_vars(*args):
import _osx_support
_osx_support.customize_config_vars(_CONFIG_VARS)
- _CONFIG_VARS['INCLUDEPY'] = os.path.join(_CONFIG_VARS['prefix'],
- 'include')
-
if args:
vals = []
for name in args:
diff --git a/lib_pypy/_sysconfigdata.py b/lib_pypy/_sysconfigdata.py
index f1cca41a4b7553819bb1a2a63af1c1a9b8591ce9..a34216e8e12b82abd72fabdd7793b3e8bb19bbcf 100644
--- a/lib_pypy/_sysconfigdata.py
+++ b/lib_pypy/_sysconfigdata.py
@@ -5,6 +5,7 @@ from distutils.spawn import find_executable
so_ext = _imp.extension_suffixes()[0]
+mybase = os.path.dirname(os.path.dirname(__file__))
build_time_vars = {
# SOABI is PEP 3149 compliant, but CPython3 has so_ext.split('.')[1]
@@ -25,7 +26,8 @@ build_time_vars = {
'AR': "ar",
'ARFLAGS': "rc",
'EXE': "",
- 'LIBDIR': os.path.join(sys.prefix, 'bin'),
+ 'LIBDIR': os.path.join(mybase, 'lib'),
+ 'INCLUDEPY': os.path.join(mybase, 'include'),
}
if find_executable("gcc"):
+27
View File
@@ -0,0 +1,27 @@
--- a/lib-python/2.7/sysconfig.py
+++ b/lib-python/2.7/sysconfig.py
@@ -553,9 +553,6 @@ def get_config_vars(*args):
if type_ == imp.C_EXTENSION:
_CONFIG_VARS['SOABI'] = suffix.split('.')[1]
break
- _CONFIG_VARS['INCLUDEPY'] = os.path.join(_CONFIG_VARS['prefix'],
- 'include')
-
if args:
vals = []
for name in args:
--- a/lib_pypy/_sysconfigdata.py
+++ b/lib_pypy/_sysconfigdata.py
@@ -1,5 +1,11 @@
-import imp
+import imp, os
+
+mybase = os.path.dirname(os.path.dirname(__file__))
build_time_vars = {
"SO": [s[0] for s in imp.get_suffixes() if s[2] == imp.C_EXTENSION][0]
}
+
+build_time_vars['LIBDIR'] = os.path.join(mybase, 'lib')
+build_time_vars['INCLUDEPY'] = os.path.join(mybase, 'include')
+
@@ -1,17 +0,0 @@
# this file is copied to the testing cwd, to raise the below error message if
# pytest/unittest is run from there.
import unittest
class TestStringMethods(unittest.TestCase):
def test_fail(self):
self.fail(
"cibuildwheel executes tests from a different working directory to "
"your project. This ensures only your wheel is imported, preventing "
"Python from accessing files that haven't been packaged into the "
"wheel. Please specify a path to your tests when invoking pytest "
"using the {project} placeholder, e.g. `pytest {project}` or "
"`pytest {project}/tests`. cibuildwheel will replace {project} with "
"the path to your project."
)
-2
View File
@@ -1,2 +0,0 @@
py36 = { version = "20.21.1", url = "https://github.com/pypa/get-virtualenv/blob/20.21.1/public/virtualenv.pyz?raw=true" }
default = { version = "20.26.3", url = "https://github.com/pypa/get-virtualenv/blob/20.26.3/public/virtualenv.pyz?raw=true" }
-15
View File
@@ -1,15 +0,0 @@
from __future__ import annotations
import json
from pathlib import Path
from typing import Any
DIR = Path(__file__).parent.resolve()
def get_schema(tool_name: str = "cibuildwheel") -> dict[str, Any]:
"Get the stored complete schema for cibuildwheel settings."
assert tool_name == "cibuildwheel", "Only cibuildwheel is supported."
with DIR.joinpath("resources/cibuildwheel.schema.json").open(encoding="utf-8") as f:
return json.load(f) # type: ignore[no-any-return]
+20 -12
View File
@@ -1,20 +1,29 @@
from __future__ import annotations
import os import os
import subprocess import subprocess
import typing import sys
from typing import Final, Literal, Protocol, Union from typing import TYPE_CHECKING, NoReturn, Set, Union
if sys.version_info < (3, 8):
from typing_extensions import Final, Literal, TypedDict
else:
from typing import Final, Literal, TypedDict
__all__ = ( __all__ = (
"PLATFORMS", "Final",
"Literal",
"TypedDict",
"Set",
"Union",
"PopenBytes",
"PathOrStr", "PathOrStr",
"PlatformName", "PlatformName",
"PLATFORMS", "PLATFORMS",
"PopenBytes", "assert_never",
) )
if typing.TYPE_CHECKING: if TYPE_CHECKING:
PopenBytes = subprocess.Popen[bytes] PopenBytes = subprocess.Popen[bytes]
PathOrStr = Union[str, os.PathLike[str]] PathOrStr = Union[str, os.PathLike[str]]
else: else:
@@ -22,10 +31,9 @@ else:
PathOrStr = Union[str, "os.PathLike[str]"] PathOrStr = Union[str, "os.PathLike[str]"]
PlatformName = Literal["linux", "macos", "windows", "pyodide"] PlatformName = Literal["linux", "macos", "windows"]
PLATFORMS: Final[set[PlatformName]] = {"linux", "macos", "windows", "pyodide"} PLATFORMS: Final[Set[PlatformName]] = {"linux", "macos", "windows"}
class GenericPythonConfiguration(Protocol): def assert_never(value: NoReturn) -> NoReturn:
@property assert False, f"Unhandled value: {value} ({type(value).__name__})" # noqa: B011
def identifier(self) -> str: ...
+90 -714
View File
@@ -1,187 +1,31 @@
from __future__ import annotations
import contextlib import contextlib
import fnmatch import fnmatch
import itertools import itertools
import os import os
import re import re
import shlex
import shutil
import ssl import ssl
import subprocess
import sys
import tarfile
import textwrap import textwrap
import time import time
import typing
import urllib.request import urllib.request
from collections import defaultdict
from collections.abc import Generator, Iterable, Mapping, MutableMapping, Sequence
from dataclasses import dataclass
from enum import Enum from enum import Enum
from functools import lru_cache from pathlib import Path
from pathlib import Path, PurePath
from tempfile import TemporaryDirectory
from time import sleep from time import sleep
from typing import Any, ClassVar, Final, Literal, TextIO, TypeVar from typing import Dict, Iterator, List, NamedTuple, Optional, Set
from zipfile import ZipFile
import bracex import bracex
import certifi import certifi
from filelock import FileLock import toml
from packaging.requirements import InvalidRequirement, Requirement
from packaging.specifiers import SpecifierSet from packaging.specifiers import SpecifierSet
from packaging.utils import parse_wheel_filename
from packaging.version import Version from packaging.version import Version
from platformdirs import user_cache_path
from ._compat import tomllib
from .architecture import Architecture from .architecture import Architecture
from .environment import ParsedEnvironment
from .typing import PathOrStr, PlatformName from .typing import PathOrStr, PlatformName
__all__ = [ resources_dir = Path(__file__).parent / "resources"
"MANYLINUX_ARCHS",
"call",
"chdir",
"combine_constraints",
"find_compatible_wheel",
"find_uv",
"format_safe",
"get_build_verbosity_extra_flags",
"prepare_command",
"read_python_configs",
"resources_dir",
"selector_matches",
"shell",
"split_config_settings",
"strtobool",
]
resources_dir: Final[Path] = Path(__file__).parent / "resources" get_pip_script = resources_dir / "get-pip.py"
install_certifi_script = resources_dir / "install_certifi.py"
install_certifi_script: Final[Path] = resources_dir / "install_certifi.py"
free_thread_enable_313: Final[Path] = resources_dir / "free-threaded-enable-313.xml"
test_fail_cwd_file: Final[Path] = resources_dir / "testing_temp_dir_file.py"
MANYLINUX_ARCHS: Final[tuple[str, ...]] = (
"x86_64",
"i686",
"pypy_x86_64",
"aarch64",
"ppc64le",
"s390x",
"pypy_aarch64",
"pypy_i686",
)
MUSLLINUX_ARCHS: Final[tuple[str, ...]] = (
"x86_64",
"i686",
"aarch64",
"ppc64le",
"s390x",
)
DEFAULT_CIBW_CACHE_PATH: Final[Path] = user_cache_path(appname="cibuildwheel", appauthor="pypa")
CIBW_CACHE_PATH: Final[Path] = Path(
os.environ.get("CIBW_CACHE_PATH", DEFAULT_CIBW_CACHE_PATH)
).resolve()
IS_WIN: Final[bool] = sys.platform.startswith("win")
@typing.overload
def call(
*args: PathOrStr,
env: Mapping[str, str] | None = None,
cwd: PathOrStr | None = None,
capture_stdout: Literal[False] = ...,
) -> None: ...
@typing.overload
def call(
*args: PathOrStr,
env: Mapping[str, str] | None = None,
cwd: PathOrStr | None = None,
capture_stdout: Literal[True],
) -> str: ...
def call(
*args: PathOrStr,
env: Mapping[str, str] | None = None,
cwd: PathOrStr | None = None,
capture_stdout: bool = False,
) -> str | None:
"""
Run subprocess.run, but print the commands first. Takes the commands as
*args. Uses shell=True on Windows due to a bug. Also converts to
Paths to strings, due to Windows behavior at least on older Pythons.
https://bugs.python.org/issue8557
"""
args_ = [str(arg) for arg in args]
# print the command executing for the logs
print("+ " + " ".join(shlex.quote(a) for a in args_))
kwargs: dict[str, Any] = {}
if capture_stdout:
kwargs["universal_newlines"] = True
kwargs["stdout"] = subprocess.PIPE
result = subprocess.run(args_, check=True, shell=IS_WIN, env=env, cwd=cwd, **kwargs)
if not capture_stdout:
return None
return typing.cast(str, result.stdout)
def shell(
*commands: str, env: Mapping[str, str] | None = None, cwd: PathOrStr | None = None
) -> None:
command = " ".join(commands)
print(f"+ {command}")
subprocess.run(command, env=env, cwd=cwd, shell=True, check=True)
def format_safe(template: str, **kwargs: str | os.PathLike[str]) -> str:
"""
Works similarly to `template.format(**kwargs)`, except that unmatched
fields in `template` are passed through untouched.
>>> format_safe('{a} {b}', a='123')
'123 {b}'
>>> format_safe('{a} {b[4]:3f}', a='123')
'123 {b[4]:3f}'
To avoid variable expansion, precede with a single backslash e.g.
>>> format_safe('\\{a} {b}', a='123')
'{a} {b}'
"""
result = template
for key, value in kwargs.items():
find_pattern = re.compile(
rf"""
(?<!\#) # don't match if preceded by a hash
{{ # literal open curly bracket
{re.escape(key)} # the field name
}} # literal close curly bracket
""",
re.VERBOSE,
)
result = re.sub(
pattern=find_pattern,
repl=str(value).replace("\\", r"\\"),
string=result,
)
# transform escaped sequences into their literal equivalents
result = result.replace(f"#{{{key}}}", f"{{{key}}}")
return result
def prepare_command(command: str, **kwargs: PathOrStr) -> str: def prepare_command(command: str, **kwargs: PathOrStr) -> str:
@@ -189,12 +33,12 @@ def prepare_command(command: str, **kwargs: PathOrStr) -> str:
Preprocesses a command by expanding variables like {python}. Preprocesses a command by expanding variables like {python}.
For example, used in the test_command option to specify the path to the For example, used in the test_command option to specify the path to the
project's root. Unmatched syntax will mostly be allowed through. project's root.
""" """
return format_safe(command, python="python", pip="pip", **kwargs) return command.format(python="python", pip="pip", **kwargs)
def get_build_verbosity_extra_flags(level: int) -> list[str]: def get_build_verbosity_extra_flags(level: int) -> List[str]:
if level > 0: if level > 0:
return ["-" + level * "v"] return ["-" + level * "v"]
elif level < 0: elif level < 0:
@@ -203,39 +47,14 @@ def get_build_verbosity_extra_flags(level: int) -> list[str]:
return [] return []
def split_config_settings( def read_python_configs(config: PlatformName) -> List[Dict[str, str]]:
config_settings: str, frontend: Literal["pip", "build", "build[uv]"]
) -> list[str]:
config_settings_list = shlex.split(config_settings)
s = "s" if frontend == "pip" else ""
return [f"--config-setting{s}={setting}" for setting in config_settings_list]
def read_python_configs(config: PlatformName) -> list[dict[str, str]]:
input_file = resources_dir / "build-platforms.toml" input_file = resources_dir / "build-platforms.toml"
with input_file.open("rb") as f: loaded_file = toml.load(input_file)
loaded_file = tomllib.load(f) results: List[Dict[str, str]] = list(loaded_file[config]["python_configurations"])
results: list[dict[str, str]] = list(loaded_file[config]["python_configurations"])
return results return results
def selector_matches(patterns: str, string: str) -> bool: class IdentifierSelector:
"""
Returns True if `string` is matched by any of the wildcard patterns in
`patterns`.
Matching is according to fnmatch, but with shell-like curly brace
expansion. For example, 'cp{36,37}-*' would match either of 'cp36-*' or
'cp37-*'.
"""
patterns_list = patterns.split()
expanded_patterns = itertools.chain.from_iterable(bracex.expand(p) for p in patterns_list)
return any(fnmatch.fnmatch(string, pat) for pat in expanded_patterns)
# Once we require Python 3.10+, we can add kw_only=True
@dataclass(frozen=True)
class BuildSelector:
""" """
This class holds a set of build/skip patterns. You call an instance with a This class holds a set of build/skip patterns. You call an instance with a
build identifier, and it returns True if that identifier should be build identifier, and it returns True if that identifier should be
@@ -243,81 +62,64 @@ class BuildSelector:
numeric digits before the first dash. numeric digits before the first dash.
""" """
build_config: str def __init__(
skip_config: str self, *, build_config: str, skip_config: str, requires_python: Optional[SpecifierSet] = None
requires_python: SpecifierSet | None = None ):
self.build_patterns = build_config.split()
# a pattern that skips prerelease versions, when include_prereleases is False. self.skip_patterns = skip_config.split()
PRERELEASE_SKIP: ClassVar[str] = "" self.requires_python = requires_python
prerelease_pythons: bool = False
free_threaded_support: bool = False
def __call__(self, build_id: str) -> bool: def __call__(self, build_id: str) -> bool:
# Filter build selectors by python_requires if set # Filter build selectors by python_requires if set
if self.requires_python is not None: if self.requires_python is not None:
py_ver_str = build_id.split("-")[0] py_ver_str = build_id.split("-")[0]
if py_ver_str.endswith("t"):
py_ver_str = py_ver_str[:-1]
major = int(py_ver_str[2]) major = int(py_ver_str[2])
minor = int(py_ver_str[3:]) minor = int(py_ver_str[3:])
version = Version(f"{major}.{minor}.99") version = Version(f"{major}.{minor}.99")
if not self.requires_python.contains(version): if not self.requires_python.contains(version):
return False return False
# filter out the prerelease pythons if self.prerelease_pythons is False build_patterns = itertools.chain.from_iterable(
if not self.prerelease_pythons and selector_matches(self.PRERELEASE_SKIP, build_id): bracex.expand(p) for p in self.build_patterns
return False )
skip_patterns = itertools.chain.from_iterable(bracex.expand(p) for p in self.skip_patterns)
# filter out free threaded pythons if self.free_threaded_support is False build: bool = any(fnmatch.fnmatch(build_id, pat) for pat in build_patterns)
if not self.free_threaded_support and selector_matches("*t-*", build_id): skip: bool = any(fnmatch.fnmatch(build_id, pat) for pat in skip_patterns)
return False return build and not skip
should_build = selector_matches(self.build_config, build_id) def __repr__(self) -> str:
should_skip = selector_matches(self.skip_config, build_id) if not self.skip_patterns:
return f'{self.__class__.__name__}({" ".join(self.build_patterns)!r})'
return should_build and not should_skip else:
return f'{self.__class__.__name__}({" ".join(self.build_patterns)!r} - {" ".join(self.skip_patterns)!r})'
def options_summary(self) -> Any:
return {
"build_config": self.build_config,
"skip_config": self.skip_config,
"requires_python": str(self.requires_python),
"prerelease_pythons": self.prerelease_pythons,
"free_threaded_support": self.free_threaded_support,
}
@dataclass(frozen=True) class BuildSelector(IdentifierSelector):
class TestSelector: pass
"""
A build selector that can only skip tests according to a skip pattern.
"""
skip_config: str
def __call__(self, build_id: str) -> bool: # Note that requires-python is not needed for TestSelector, as you can't test
should_skip = selector_matches(self.skip_config, build_id) # what you can't build.
return not should_skip class TestSelector(IdentifierSelector):
def __init__(self, *, skip_config: str):
def options_summary(self) -> Any: super().__init__(build_config="*", skip_config=skip_config)
return {"skip_config": self.skip_config}
# Taken from https://stackoverflow.com/a/107717 # Taken from https://stackoverflow.com/a/107717
class Unbuffered: class Unbuffered:
def __init__(self, stream: TextIO) -> None: def __init__(self, stream): # type: ignore
self.stream = stream self.stream = stream
def write(self, data: str) -> None: def write(self, data): # type: ignore
self.stream.write(data) self.stream.write(data)
self.stream.flush() self.stream.flush()
def writelines(self, data: Iterable[str]) -> None: def writelines(self, datas): # type: ignore
self.stream.writelines(data) self.stream.writelines(datas)
self.stream.flush() self.stream.flush()
def __getattr__(self, attr: str) -> Any: def __getattr__(self, attr): # type: ignore
return getattr(self.stream, attr) return getattr(self.stream, attr)
@@ -334,63 +136,18 @@ def download(url: str, dest: Path) -> None:
repeat_num = 3 repeat_num = 3
for i in range(repeat_num): for i in range(repeat_num):
try: try:
with urllib.request.urlopen(url, context=context) as response: response = urllib.request.urlopen(url, context=context)
dest.write_bytes(response.read()) except Exception:
return
except OSError:
if i == repeat_num - 1: if i == repeat_num - 1:
raise raise
sleep(3) sleep(3)
continue
break
try:
def extract_zip(zip_src: Path, dest: Path) -> None: dest.write_bytes(response.read())
with ZipFile(zip_src) as zip_: finally:
for zinfo in zip_.filelist: response.close()
zip_.extract(zinfo, dest)
# Set permissions to the same values as they were set in the archive
# We have to do this manually due to
# https://github.com/python/cpython/issues/59999
# But some files in the zipfile seem to have external_attr with 0
# permissions. In that case just use the default value???
permissions = (zinfo.external_attr >> 16) & 0o777
if permissions != 0:
dest.joinpath(zinfo.filename).chmod(permissions)
def extract_tar(tar_src: Path, dest: Path) -> None:
with tarfile.open(tar_src) as tar_:
tar_.extraction_filter = getattr(tarfile, "tar_filter", (lambda member, _: member))
tar_.extractall(dest)
def move_file(src_file: Path, dst_file: Path) -> Path:
"""Moves a file safely while avoiding potential semantic confusion:
1. `dst_file` must point to the target filename, not a directory
2. `dst_file` will be overwritten if it already exists
3. any missing parent directories will be created
Returns the fully resolved Path of the resulting file.
Raises:
NotADirectoryError: If any part of the intermediate path to `dst_file` is an existing file
IsADirectoryError: If `dst_file` points directly to an existing directory
"""
src_file = src_file.resolve(strict=True)
dst_file = dst_file.resolve()
if dst_file.is_dir():
msg = "dst_file must be a valid target filename, not an existing directory."
raise IsADirectoryError(msg)
dst_file.unlink(missing_ok=True)
dst_file.parent.mkdir(parents=True, exist_ok=True)
# using shutil.move() as Path.rename() is not guaranteed to work across filesystem boundaries
# explicit str() needed for Python 3.8
resulting_file = shutil.move(str(src_file), str(dst_file))
return Path(resulting_file).resolve(strict=True)
class DependencyConstraints: class DependencyConstraints:
@@ -399,67 +156,60 @@ class DependencyConstraints:
self.base_file_path = base_file_path.resolve() self.base_file_path = base_file_path.resolve()
@staticmethod @staticmethod
def with_defaults() -> DependencyConstraints: def with_defaults() -> "DependencyConstraints":
return DependencyConstraints(base_file_path=resources_dir / "constraints.txt") return DependencyConstraints(base_file_path=resources_dir / "constraints.txt")
def get_for_python_version( def get_for_python_version(self, version: str) -> Path:
self, version: str, *, variant: Literal["python", "pyodide"] = "python"
) -> Path:
version_parts = version.split(".") version_parts = version.split(".")
# try to find a version-specific dependency file e.g. if # try to find a version-specific dependency file e.g. if
# ./constraints.txt is the base, look for ./constraints-python36.txt # ./constraints.txt is the base, look for ./constraints-python27.txt
specific_stem = self.base_file_path.stem + f"-{variant}{version_parts[0]}{version_parts[1]}" specific_stem = self.base_file_path.stem + f"-python{version_parts[0]}{version_parts[1]}"
specific_name = specific_stem + self.base_file_path.suffix specific_name = specific_stem + self.base_file_path.suffix
specific_file_path = self.base_file_path.with_name(specific_name) specific_file_path = self.base_file_path.with_name(specific_name)
if specific_file_path.exists(): if specific_file_path.exists():
return specific_file_path return specific_file_path
else: else:
return self.base_file_path return self.base_file_path
def __repr__(self) -> str: def __repr__(self) -> str:
return f"{self.__class__.__name__}({self.base_file_path!r})" return f"{self.__class__.__name__}{self.base_file_path!r})"
def __eq__(self, o: object) -> bool:
if not isinstance(o, DependencyConstraints):
return False
return self.base_file_path == o.base_file_path
def options_summary(self) -> Any:
if self == DependencyConstraints.with_defaults():
return "pinned"
else:
return self.base_file_path.name
BuildFrontendName = Literal["pip", "build", "build[uv]"] class BuildOptions(NamedTuple):
package_dir: Path
output_dir: Path
build_selector: BuildSelector
architectures: Set[Architecture]
environment: ParsedEnvironment
before_all: str
before_build: Optional[str]
repair_command: str
manylinux_images: Optional[Dict[str, str]]
dependency_constraints: Optional[DependencyConstraints]
test_command: Optional[str]
test_selector: TestSelector
before_test: Optional[str]
test_requires: List[str]
test_extras: str
build_verbosity: int
@dataclass(frozen=True) class NonPlatformWheelError(Exception):
class BuildFrontendConfig: def __init__(self) -> None:
name: BuildFrontendName message = textwrap.dedent(
args: Sequence[str] = () """
cibuildwheel: Build failed because a pure Python wheel was generated.
@staticmethod If you intend to build a pure-Python wheel, you don't need cibuildwheel - use
def from_config_string(config_string: str) -> BuildFrontendConfig: `pip wheel -w DEST_DIR .` instead.
config_dict = parse_key_value_string(config_string, ["name"], ["args"])
name = " ".join(config_dict["name"])
if name not in {"pip", "build", "build[uv]"}:
msg = f"Unrecognised build frontend {name!r}, only 'pip', 'build', and 'build[uv]' are supported"
raise ValueError(msg)
name = typing.cast(BuildFrontendName, name) If you expected a platform wheel, check your project configuration, or run
cibuildwheel with CIBW_BUILD_VERBOSITY=1 to view build logs.
"""
)
args = config_dict.get("args") or [] super().__init__(message)
return BuildFrontendConfig(name=name, args=args)
def options_summary(self) -> str | dict[str, str]:
if not self.args:
return self.name
else:
return {"name": self.name, "args": repr(self.args)}
def strtobool(val: str) -> bool: def strtobool(val: str) -> bool:
@@ -473,11 +223,10 @@ class CIProvider(Enum):
azure_pipelines = "azure_pipelines" azure_pipelines = "azure_pipelines"
github_actions = "github_actions" github_actions = "github_actions"
gitlab = "gitlab" gitlab = "gitlab"
cirrus_ci = "cirrus_ci"
other = "other" other = "other"
def detect_ci_provider() -> CIProvider | None: def detect_ci_provider() -> Optional[CIProvider]:
if "TRAVIS" in os.environ: if "TRAVIS" in os.environ:
return CIProvider.travis_ci return CIProvider.travis_ci
elif "APPVEYOR" in os.environ: elif "APPVEYOR" in os.environ:
@@ -490,8 +239,6 @@ def detect_ci_provider() -> CIProvider | None:
return CIProvider.github_actions return CIProvider.github_actions
elif "GITLAB_CI" in os.environ: elif "GITLAB_CI" in os.environ:
return CIProvider.gitlab return CIProvider.gitlab
elif "CIRRUS_CI" in os.environ:
return CIProvider.cirrus_ci
elif strtobool(os.environ.get("CI", "false")): elif strtobool(os.environ.get("CI", "false")):
return CIProvider.other return CIProvider.other
else: else:
@@ -510,14 +257,8 @@ def unwrap(text: str) -> str:
return re.sub(r"\s+", " ", text) return re.sub(r"\s+", " ", text)
@dataclass(frozen=True)
class FileReport:
name: str
size: str
@contextlib.contextmanager @contextlib.contextmanager
def print_new_wheels(msg: str, output_dir: Path) -> Generator[None, None, None]: def print_new_wheels(msg: str, output_dir: Path) -> Iterator[None]:
""" """
Prints the new items in a directory upon exiting. The message to display Prints the new items in a directory upon exiting. The message to display
can include {n} for number of wheels, {s} for total number of seconds, can include {n} for number of wheels, {s} for total number of seconds,
@@ -529,373 +270,8 @@ def print_new_wheels(msg: str, output_dir: Path) -> Generator[None, None, None]:
existing_contents = set(output_dir.iterdir()) existing_contents = set(output_dir.iterdir())
yield yield
final_contents = set(output_dir.iterdir()) final_contents = set(output_dir.iterdir())
new_contents = final_contents - existing_contents
new_contents = [
FileReport(wheel.name, f"{(wheel.stat().st_size + 1023) // 1024:,d}")
for wheel in final_contents - existing_contents
]
if not new_contents:
return
max_name_len = max(len(f.name) for f in new_contents)
max_size_len = max(len(f.size) for f in new_contents)
n = len(new_contents) n = len(new_contents)
s = time.time() - start_time s = time.time() - start_time
m = s / 60 m = s / 60
print( print(msg.format(n=n, s=s, m=m), *sorted(f" {f.name}" for f in new_contents), sep="\n")
msg.format(n=n, s=s, m=m),
*sorted(
f" {f.name:<{max_name_len}s} {f.size:>{max_size_len}s} kB" for f in new_contents
),
sep="\n",
)
def get_pip_version(env: Mapping[str, str]) -> str:
versions_output_text = call(
"python", "-m", "pip", "freeze", "--all", capture_stdout=True, env=env
)
(pip_version,) = (
version[5:]
for version in versions_output_text.strip().splitlines()
if version.startswith("pip==")
)
return pip_version
@lru_cache(maxsize=None)
def ensure_node(major_version: str) -> Path:
input_file = resources_dir / "nodejs.toml"
with input_file.open("rb") as f:
loaded_file = tomllib.load(f)
version = str(loaded_file[major_version])
base_url = str(loaded_file["url"])
ext = "zip" if IS_WIN else "tar.xz"
platform = "win" if IS_WIN else ("darwin" if sys.platform.startswith("darwin") else "linux")
linux_arch = Architecture.native_arch("linux")
assert linux_arch is not None
arch = {"x86_64": "x64", "i686": "x86", "aarch64": "arm64"}.get(
linux_arch.value, linux_arch.value
)
name = f"node-{version}-{platform}-{arch}"
path = CIBW_CACHE_PATH / name
with FileLock(str(path) + ".lock"):
if not path.exists():
url = f"{base_url}{version}/{name}.{ext}"
with TemporaryDirectory() as tmp_path:
archive = Path(tmp_path) / f"{name}.{ext}"
download(url, archive)
if ext == "zip":
extract_zip(archive, path.parent)
else:
extract_tar(archive, path.parent)
assert path.exists()
if not IS_WIN:
return path / "bin"
return path
@lru_cache(maxsize=None)
def _ensure_virtualenv(version: str) -> Path:
version_parts = version.split(".")
key = f"py{version_parts[0]}{version_parts[1]}"
input_file = resources_dir / "virtualenv.toml"
with input_file.open("rb") as f:
loaded_file = tomllib.load(f)
configuration = loaded_file.get(key, loaded_file["default"])
version = str(configuration["version"])
url = str(configuration["url"])
path = CIBW_CACHE_PATH / f"virtualenv-{version}.pyz"
with FileLock(str(path) + ".lock"):
if not path.exists():
download(url, path)
return path
def _parse_constraints_for_virtualenv(
seed_packages: list[str],
dependency_constraint_flags: Sequence[PathOrStr],
) -> dict[str, str]:
"""
Parses the constraints file referenced by `dependency_constraint_flags` and returns a dict where
the key is the package name, and the value is the constraint version.
If a package version cannot be found, its value is "embed" meaning that virtualenv will install
its bundled version, already available locally.
The function does not try to be too smart and just handles basic constraints.
If it can't get an exact version, the real constraint will be handled by the
{macos|windows}.setup_python function.
"""
assert len(dependency_constraint_flags) in {0, 2}
# only seed pip if other seed packages do not appear in a constraint file
constraints_dict = {"pip": "embed"}
if len(dependency_constraint_flags) == 2:
assert dependency_constraint_flags[0] == "-c"
constraint_path = Path(dependency_constraint_flags[1])
assert constraint_path.exists()
with constraint_path.open(encoding="utf-8") as constraint_file:
for line_ in constraint_file:
line = line_.strip()
if not line:
continue
if line.startswith("#"):
continue
try:
requirement = Requirement(line)
package = requirement.name
if (
package not in seed_packages
or requirement.url is not None
or requirement.marker is not None
or len(requirement.extras) != 0
or len(requirement.specifier) != 1
):
continue
specifier = next(iter(requirement.specifier))
if specifier.operator != "==":
continue
constraints_dict[package] = specifier.version
except InvalidRequirement:
continue
return constraints_dict
def virtualenv(
version: str,
python: Path,
venv_path: Path,
dependency_constraint_flags: Sequence[PathOrStr],
*,
use_uv: bool,
) -> dict[str, str]:
"""
Create a virtual environment. If `use_uv` is True,
dependency_constraint_flags are ignored since nothing is installed in the
venv. Otherwise, pip is installed, and setuptools + wheel if Python < 3.12.
"""
assert python.exists()
if use_uv:
call("uv", "venv", venv_path, "--python", python)
else:
virtualenv_app = _ensure_virtualenv(version)
allowed_seed_packages = ["pip", "setuptools", "wheel"]
constraints = _parse_constraints_for_virtualenv(
allowed_seed_packages, dependency_constraint_flags
)
additional_flags: list[str] = []
for package in allowed_seed_packages:
if package in constraints:
additional_flags.append(f"--{package}={constraints[package]}")
else:
additional_flags.append(f"--no-{package}")
# Using symlinks to pre-installed seed packages is really the fastest way to get a virtual
# environment. The initial cost is a bit higher but reusing is much faster.
# Windows does not always allow symlinks so just disabling for now.
# Requires pip>=19.3 so disabling for "embed" because this means we don't know what's the
# version of pip that will end-up installed.
# c.f. https://virtualenv.pypa.io/en/latest/cli_interface.html#section-seeder
if (
not IS_WIN
and constraints["pip"] != "embed"
and Version(constraints["pip"]) >= Version("19.3")
):
additional_flags.append("--symlink-app-data")
call(
sys.executable,
"-sS", # just the stdlib, https://github.com/pypa/virtualenv/issues/2133#issuecomment-1003710125
virtualenv_app,
"--activators=",
"--no-periodic-update",
*additional_flags,
"--python",
python,
venv_path,
)
paths = [str(venv_path), str(venv_path / "Scripts")] if IS_WIN else [str(venv_path / "bin")]
env = os.environ.copy()
env["PATH"] = os.pathsep.join([*paths, env["PATH"]])
env["VIRTUAL_ENV"] = str(venv_path)
return env
T = TypeVar("T", bound=PurePath)
def find_compatible_wheel(wheels: Sequence[T], identifier: str) -> T | None:
"""
Finds a wheel with an abi3 or a none ABI tag in `wheels` compatible with the Python interpreter
specified by `identifier` that is previously built.
"""
interpreter, platform = identifier.split("-")
free_threaded = interpreter.endswith("t")
if free_threaded:
interpreter = interpreter[:-1]
for wheel in wheels:
_, _, _, tags = parse_wheel_filename(wheel.name)
for tag in tags:
if tag.abi == "abi3" and not free_threaded:
# ABI3 wheels must start with cp3 for impl and tag
if not (interpreter.startswith("cp3") and tag.interpreter.startswith("cp3")):
continue
elif tag.abi == "none":
# CPythonless wheels must include py3 tag
if tag.interpreter[:3] != "py3":
continue
else:
# Other types of wheels are not detected, this is looking for previously built wheels.
continue
if tag.interpreter != "py3" and int(tag.interpreter[3:]) > int(interpreter[3:]):
# If a minor version number is given, it has to be lower than the current one.
continue
if platform.startswith(("manylinux", "musllinux", "macosx")):
# Linux, macOS require the beginning and ending match (macos/manylinux version doesn't need to)
os_, arch = platform.split("_", 1)
if not tag.platform.startswith(os_):
continue
if not tag.platform.endswith(f"_{arch}"):
continue
else:
# Windows should exactly match
if tag.platform != platform:
continue
# If all the filters above pass, then the wheel is a previously built compatible wheel.
return wheel
return None
# Can be replaced by contextlib.chdir in Python 3.11
@contextlib.contextmanager
def chdir(new_path: Path | str) -> Generator[None, None, None]:
"""Non thread-safe context manager to change the current working directory."""
cwd = os.getcwd()
try:
os.chdir(new_path)
yield
finally:
os.chdir(cwd)
def fix_ansi_codes_for_github_actions(text: str) -> str:
"""
Github Actions forgets the current ANSI style on every new line. This
function repeats the current ANSI style on every new line.
"""
ansi_code_regex = re.compile(r"(\033\[[0-9;]*m)")
ansi_codes: list[str] = []
output = ""
for line in text.splitlines(keepends=True):
# add the current ANSI codes to the beginning of the line
output += "".join(ansi_codes) + line
# split the line at each ANSI code
parts = ansi_code_regex.split(line)
# if there are any ANSI codes, save them
if len(parts) > 1:
# iterate over the ANSI codes in this line
for code in parts[1::2]:
if code == "\033[0m":
# reset the list of ANSI codes when the clear code is found
ansi_codes = []
else:
ansi_codes.append(code)
return output
def parse_key_value_string(
key_value_string: str,
positional_arg_names: Sequence[str] | None = None,
kw_arg_names: Sequence[str] | None = None,
) -> dict[str, list[str]]:
"""
Parses a string like "docker; create_args: --some-option=value another-option"
"""
if positional_arg_names is None:
positional_arg_names = []
if kw_arg_names is None:
kw_arg_names = []
all_field_names = [*positional_arg_names, *kw_arg_names]
shlexer = shlex.shlex(key_value_string, posix=True, punctuation_chars=";")
shlexer.commenters = ""
shlexer.whitespace_split = True
parts = list(shlexer)
# parts now looks like
# ['docker', ';', 'create_args:', '--some-option=value', 'another-option']
# split by semicolon
fields = [list(group) for k, group in itertools.groupby(parts, lambda x: x == ";") if not k]
result: defaultdict[str, list[str]] = defaultdict(list)
for field_i, field in enumerate(fields):
# check to see if the option name is specified
field_name, sep, first_value = field[0].partition(":")
if sep:
if field_name not in all_field_names:
msg = f"Failed to parse {key_value_string!r}. Unknown field name {field_name!r}"
raise ValueError(msg)
values = ([first_value] if first_value else []) + field[1:]
else:
try:
field_name = positional_arg_names[field_i]
except IndexError:
msg = f"Failed to parse {key_value_string!r}. Too many positional arguments - expected a maximum of {len(positional_arg_names)}"
raise ValueError(msg) from None
values = field
result[field_name] += values
return dict(result)
def find_uv() -> Path | None:
# Prefer uv in our environment
with contextlib.suppress(ImportError, FileNotFoundError):
# pylint: disable-next=import-outside-toplevel
from uv import find_uv_bin
return Path(find_uv_bin())
uv_on_path = shutil.which("uv")
return Path(uv_on_path) if uv_on_path else None
def combine_constraints(
env: MutableMapping[str, str], /, constraints_path: Path, tmp_dir: Path | None
) -> None:
"""
This will workaround a bug in pip<=21.1.1 or uv<=0.2.0 if a tmp_dir is given.
If set to None, this will use the modern URI method.
"""
if tmp_dir:
if " " in str(constraints_path):
assert " " not in str(tmp_dir)
tmp_file = tmp_dir / "constraints.txt"
tmp_file.write_bytes(constraints_path.read_bytes())
constraints_path = tmp_file
our_constraints = str(constraints_path)
else:
our_constraints = (
constraints_path.as_uri() if " " in str(constraints_path) else str(constraints_path)
)
user_constraints = env.get("PIP_CONSTRAINT")
env["UV_CONSTRAINT"] = env["PIP_CONSTRAINT"] = " ".join(
c for c in [our_constraints, user_constraints] if c
)
+211 -410
View File
@@ -1,90 +1,88 @@
from __future__ import annotations
import os import os
import platform as platform_module
import shutil import shutil
import subprocess import subprocess
import textwrap import sys
from collections.abc import MutableMapping, Sequence, Set import tempfile
from dataclasses import dataclass
from functools import lru_cache
from pathlib import Path from pathlib import Path
from typing import Dict, List, NamedTuple, Optional, Sequence, Set
from zipfile import ZipFile
from filelock import FileLock import toml
from packaging.version import Version
from . import errors
from ._compat.typing import assert_never
from .architecture import Architecture from .architecture import Architecture
from .environment import ParsedEnvironment from .environment import ParsedEnvironment
from .logger import log from .logger import log
from .options import Options
from .typing import PathOrStr from .typing import PathOrStr
from .util import ( from .util import (
CIBW_CACHE_PATH, BuildOptions,
BuildFrontendConfig,
BuildFrontendName,
BuildSelector, BuildSelector,
call, NonPlatformWheelError,
combine_constraints,
download, download,
extract_zip,
find_compatible_wheel,
find_uv,
get_build_verbosity_extra_flags, get_build_verbosity_extra_flags,
get_pip_version, get_pip_script,
move_file,
prepare_command, prepare_command,
read_python_configs, read_python_configs,
shell,
split_config_settings,
test_fail_cwd_file,
unwrap,
virtualenv,
) )
IS_RUNNING_ON_AZURE = Path("C:\\hostedtoolcache").exists()
def get_nuget_args(
version: str, arch: str, free_threaded: bool, output_directory: Path def call(
) -> list[str]: args: Sequence[PathOrStr], env: Optional[Dict[str, str]] = None, cwd: Optional[str] = None
package_name = { ) -> None:
"32": "pythonx86", print("+ " + " ".join(str(a) for a in args))
"64": "python", # we use shell=True here, even though we don't need a shell due to a bug
"ARM64": "pythonarm64", # https://bugs.python.org/issue8557
# Aliases for platform.machine() return values subprocess.run([str(a) for a in args], env=env, cwd=cwd, shell=True, check=True)
"x86": "pythonx86",
"AMD64": "python",
}[arch] def shell(command: str, env: Optional[Dict[str, str]] = None, cwd: Optional[str] = None) -> None:
if free_threaded: print(f"+ {command}")
package_name = f"{package_name}-freethreaded" subprocess.run(command, env=env, cwd=cwd, shell=True, check=True)
def get_nuget_args(version: str, arch: str) -> List[str]:
python_name = "python" if version[0] == "3" else "python2"
if arch == "32":
python_name += "x86"
return [ return [
package_name, python_name,
"-Version", "-Version",
version, version,
"-FallbackSource", "-FallbackSource",
"https://api.nuget.org/v3/index.json", "https://api.nuget.org/v3/index.json",
"-OutputDirectory", "-OutputDirectory",
str(output_directory), "C:\\cibw\\python",
] ]
@dataclass(frozen=True) class PythonConfiguration(NamedTuple):
class PythonConfiguration:
version: str version: str
arch: str arch: str
identifier: str identifier: str
url: str | None = None url: Optional[str] = None
def get_python_configurations( def get_python_configurations(
build_selector: BuildSelector, build_selector: BuildSelector,
architectures: Set[Architecture], architectures: Set[Architecture],
) -> list[PythonConfiguration]: ) -> List[PythonConfiguration]:
full_python_configs = read_python_configs("windows") full_python_configs = read_python_configs("windows")
python_configurations = [PythonConfiguration(**item) for item in full_python_configs] python_configurations = [PythonConfiguration(**item) for item in full_python_configs]
map_arch = {"32": Architecture.x86, "64": Architecture.AMD64, "ARM64": Architecture.ARM64} map_arch = {
"32": Architecture.x86,
"64": Architecture.AMD64,
}
# Only supported with custom compiler, since MS removed the 2008 compiler download
custom_compiler = os.environ.get("DISTUTILS_USE_SDK") and os.environ.get("MSSdk")
if not custom_compiler:
python_configurations = [
c for c in python_configurations if not c.version.startswith("2.7")
]
# skip builds as required # skip builds as required
python_configurations = [ python_configurations = [
@@ -96,392 +94,236 @@ def get_python_configurations(
return python_configurations return python_configurations
@lru_cache(maxsize=None) def extract_zip(zip_src: Path, dest: Path) -> None:
def _ensure_nuget() -> Path: with ZipFile(zip_src) as zip:
nuget = CIBW_CACHE_PATH / "nuget.exe" zip.extractall(dest)
with FileLock(str(nuget) + ".lock"):
if not nuget.exists():
download("https://dist.nuget.org/win-x86-commandline/latest/nuget.exe", nuget)
return nuget
def install_cpython(configuration: PythonConfiguration, arch: str | None = None) -> Path: def install_cpython(version: str, arch: str, nuget: Path) -> Path:
version = configuration.version nuget_args = get_nuget_args(version, arch)
free_threaded = "t-" in configuration.identifier installation_path = Path(nuget_args[-1]) / (nuget_args[0] + "." + version) / "tools"
if arch is None: call([nuget, "install", *nuget_args])
arch = configuration.arch return installation_path
base_output_dir = CIBW_CACHE_PATH / "nuget-cpython"
nuget_args = get_nuget_args(version, arch, free_threaded, base_output_dir)
installation_path = base_output_dir / (nuget_args[0] + "." + version) / "tools"
free_threaded_str = "-freethreaded" if free_threaded else ""
with FileLock(str(base_output_dir) + f"-{version}{free_threaded_str}-{arch}.lock"):
if not installation_path.exists():
nuget = _ensure_nuget()
call(nuget, "install", *nuget_args)
return installation_path / "python.exe"
def install_pypy(tmp: Path, arch: str, url: str) -> Path: def install_pypy(version: str, arch: str, url: str) -> Path:
assert arch == "64" assert arch == "32"
assert "win64" in url
# Inside the PyPy zip file is a directory with the same name # Inside the PyPy zip file is a directory with the same name
zip_filename = url.rsplit("/", 1)[-1] zip_filename = url.rsplit("/", 1)[-1]
extension = ".zip" extension = ".zip"
assert zip_filename.endswith(extension) assert zip_filename.endswith(extension)
installation_path = CIBW_CACHE_PATH / zip_filename[: -len(extension)] installation_path = Path("C:\\cibw") / zip_filename[: -len(extension)]
with FileLock(str(installation_path) + ".lock"):
if not installation_path.exists(): if not installation_path.exists():
pypy_zip = tmp / zip_filename pypy_zip = Path("C:\\cibw") / zip_filename
download(url, pypy_zip) download(url, pypy_zip)
# Extract to the parent directory because the zip file still contains a directory # Extract to the parent directory because the zip file still contains a directory
extract_zip(pypy_zip, installation_path.parent) extract_zip(pypy_zip, installation_path.parent)
return installation_path / "python.exe" pypy_exe = "pypy3.exe" if version[0] == "3" else "pypy.exe"
(installation_path / "python.exe").symlink_to(installation_path / pypy_exe)
return installation_path
def setup_setuptools_cross_compile(
tmp: Path,
python_configuration: PythonConfiguration,
python_libs_base: Path,
env: MutableMapping[str, str],
) -> None:
distutils_cfg = tmp / "extra-setup.cfg"
env["DIST_EXTRA_CONFIG"] = str(distutils_cfg)
log.notice(f"Setting DIST_EXTRA_CONFIG={distutils_cfg} for cross-compilation")
# Ensure our additional import libraries are made available, and explicitly
# set the platform name
map_plat = {"32": "win32", "64": "win-amd64", "ARM64": "win-arm64"}
plat_name = map_plat[python_configuration.arch]
# Set environment variable so that setuptools._distutils.get_platform()
# identifies the target, not the host
vscmd_arg_tgt_arch = {"32": "x86", "64": "x64", "ARM64": "arm64"}
current_tgt_arch = vscmd_arg_tgt_arch[python_configuration.arch]
if (env.get("VSCMD_ARG_TGT_ARCH") or current_tgt_arch) != current_tgt_arch:
msg = f"VSCMD_ARG_TGT_ARCH must be set to {current_tgt_arch!r}, got {env['VSCMD_ARG_TGT_ARCH']!r}. If you're setting up MSVC yourself (e.g. using vcvarsall.bat or msvc-dev-cmd), make sure to target the right architecture. Alternatively, run cibuildwheel without configuring MSVC, and let the build backend handle it."
raise errors.FatalError(msg)
env["VSCMD_ARG_TGT_ARCH"] = current_tgt_arch
# (This file must be default/locale encoding, so we can't pass 'encoding')
distutils_cfg.write_text(
textwrap.dedent(
f"""\
[build]
plat_name={plat_name}
[build_ext]
library_dirs={python_libs_base}
plat_name={plat_name}
[bdist_wheel]
plat_name={plat_name}
"""
)
)
# setuptools builds require explicit override of PYD extension
# This is because it always gets the extension from the running
# interpreter, and has no logic to construct it. Currently, CPython's
# extensions follow our identifiers, but if they ever diverge in the
# future, we will need to store new data
log.notice(
f"Setting SETUPTOOLS_EXT_SUFFIX=.{python_configuration.identifier}.pyd for cross-compilation"
)
env["SETUPTOOLS_EXT_SUFFIX"] = f".{python_configuration.identifier}.pyd"
# Cross-compilation requires fixes that only exist in setuptools's copy of
# distutils, so ensure that it is activated
# Since not all projects can handle the newer distutils, display a warning
# to help them figure out what may have gone wrong if this breaks for them
log.notice("Setting SETUPTOOLS_USE_DISTUTILS=local as it is required for cross-compilation")
env["SETUPTOOLS_USE_DISTUTILS"] = "local"
# These cross-compile setup functions have the same signature by design
def setup_rust_cross_compile(
tmp: Path, # noqa: ARG001
python_configuration: PythonConfiguration,
python_libs_base: Path, # noqa: ARG001
env: MutableMapping[str, str],
) -> None:
# Assume that MSVC will be used, because we already know that we are
# cross-compiling. MinGW users can set CARGO_BUILD_TARGET themselves
# and we will respect the existing value.
cargo_target = {
"64": "x86_64-pc-windows-msvc",
"32": "i686-pc-windows-msvc",
"ARM64": "aarch64-pc-windows-msvc",
}.get(python_configuration.arch)
# CARGO_BUILD_TARGET is the variable used by Cargo and setuptools_rust
if env.get("CARGO_BUILD_TARGET"):
if env["CARGO_BUILD_TARGET"] != cargo_target:
log.notice("Not overriding CARGO_BUILD_TARGET as it has already been set")
# No message if it was set to what we were planning to set it to
elif cargo_target:
log.notice(f"Setting CARGO_BUILD_TARGET={cargo_target} for cross-compilation")
env["CARGO_BUILD_TARGET"] = cargo_target
else:
log.warning(
f"Unable to configure Rust cross-compilation for architecture {python_configuration.arch}"
)
def setup_python( def setup_python(
tmp: Path,
python_configuration: PythonConfiguration, python_configuration: PythonConfiguration,
dependency_constraint_flags: Sequence[PathOrStr], dependency_constraint_flags: Sequence[PathOrStr],
environment: ParsedEnvironment, environment: ParsedEnvironment,
build_frontend: BuildFrontendName, ) -> Dict[str, str]:
) -> tuple[Path, dict[str, str]]: nuget = Path("C:\\cibw\\nuget.exe")
tmp.mkdir() if not nuget.exists():
log.step("Downloading nuget...")
download("https://dist.nuget.org/win-x86-commandline/latest/nuget.exe", nuget)
implementation_id = python_configuration.identifier.split("-")[0] implementation_id = python_configuration.identifier.split("-")[0]
python_libs_base = None
log.step(f"Installing Python {implementation_id}...") log.step(f"Installing Python {implementation_id}...")
if implementation_id.startswith("cp"): if implementation_id.startswith("cp"):
native_arch = platform_module.machine() installation_path = install_cpython(
base_python = install_cpython(python_configuration) python_configuration.version, python_configuration.arch, nuget
if python_configuration.arch == "ARM64" != native_arch: )
# To cross-compile for ARM64, we need a native CPython to run the
# build, and a copy of the ARM64 import libraries ('.\libs\*.lib')
# for any extension modules.
python_libs_base = base_python.parent / "libs"
log.step(f"Installing native Python {native_arch} for cross-compilation...")
base_python = install_cpython(python_configuration, arch=native_arch)
elif implementation_id.startswith("pp"): elif implementation_id.startswith("pp"):
assert python_configuration.url is not None assert python_configuration.url is not None
base_python = install_pypy(tmp, python_configuration.arch, python_configuration.url) installation_path = install_pypy(
else: python_configuration.version, python_configuration.arch, python_configuration.url
msg = "Unknown Python implementation"
raise ValueError(msg)
assert base_python.exists()
use_uv = (
build_frontend == "build[uv]"
and Version(python_configuration.version) >= Version("3.8")
and not python_configuration.identifier.startswith("pp38-")
) )
uv_path = find_uv() else:
raise ValueError("Unknown Python implementation")
assert (installation_path / "python.exe").exists()
log.step("Setting up build environment...") log.step("Setting up build environment...")
venv_path = tmp / "venv"
env = virtualenv(
python_configuration.version,
base_python,
venv_path,
dependency_constraint_flags,
use_uv=use_uv,
)
# set up environment variables for run_with_env # set up PATH and environment variables for run_with_env
env = os.environ.copy()
env["PYTHON_VERSION"] = python_configuration.version env["PYTHON_VERSION"] = python_configuration.version
env["PYTHON_ARCH"] = python_configuration.arch env["PYTHON_ARCH"] = python_configuration.arch
env["PIP_DISABLE_PIP_VERSION_CHECK"] = "1" env["PATH"] = os.pathsep.join(
[str(installation_path), str(installation_path / "Scripts"), env["PATH"]]
# upgrade pip to the version matching our constraints
# if necessary, reinstall it to ensure that it's available on PATH as 'pip.exe'
if not use_uv:
call(
"python",
"-m",
"pip",
"install",
"--upgrade",
"pip",
*dependency_constraint_flags,
env=env,
cwd=venv_path,
) )
env["PIP_DISABLE_PIP_VERSION_CHECK"] = "1"
# update env with results from CIBW_ENVIRONMENT # update env with results from CIBW_ENVIRONMENT
env = environment.as_dictionary(prev_environment=env) env = environment.as_dictionary(prev_environment=env)
# check what Python version we're on # for the logs - check we're running the right version of python
call("where", "python", env=env) call(["where", "python"], env=env)
call("python", "--version", env=env) call(["python", "--version"], env=env)
call("python", "-c", "\"import struct; print(struct.calcsize('P') * 8)\"", env=env) call(["python", "-c", "\"import struct; print(struct.calcsize('P') * 8)\""], env=env)
where_python = call("where", "python", env=env, capture_stdout=True).splitlines()[0].strip() where_python = (
if where_python != str(venv_path / "Scripts" / "python.exe"): subprocess.run(
msg = "python available on PATH doesn't match our installed instance. If you have modified PATH, ensure that you don't overwrite cibuildwheel's entry or insert python above it." ["where", "python"],
raise errors.FatalError(msg) env=env,
universal_newlines=True,
check=True,
stdout=subprocess.PIPE,
)
.stdout.splitlines()[0]
.strip()
)
if where_python != str(installation_path / "python.exe"):
print(
"cibuildwheel: python available on PATH doesn't match our installed instance. If you have modified PATH, ensure that you don't overwrite cibuildwheel's entry or insert python above it.",
file=sys.stderr,
)
sys.exit(1)
# check what pip version we're on # make sure pip is installed
if not use_uv: if not (installation_path / "Scripts" / "pip.exe").exists():
assert (venv_path / "Scripts" / "pip.exe").exists() call(["python", get_pip_script, *dependency_constraint_flags], env=env, cwd="C:\\cibw")
where_pip = call("where", "pip", env=env, capture_stdout=True).splitlines()[0].strip() assert (installation_path / "Scripts" / "pip.exe").exists()
if where_pip.strip() != str(venv_path / "Scripts" / "pip.exe"): where_pip = (
msg = "pip available on PATH doesn't match our installed instance. If you have modified PATH, ensure that you don't overwrite cibuildwheel's entry or insert pip above it." subprocess.run(
raise errors.FatalError(msg) ["where", "pip"], env=env, universal_newlines=True, check=True, stdout=subprocess.PIPE
)
call("pip", "--version", env=env) .stdout.splitlines()[0]
.strip()
)
if where_pip.strip() != str(installation_path / "Scripts" / "pip.exe"):
print(
"cibuildwheel: pip available on PATH doesn't match our installed instance. If you have modified PATH, ensure that you don't overwrite cibuildwheel's entry or insert pip above it.",
file=sys.stderr,
)
sys.exit(1)
log.step("Installing build tools...") log.step("Installing build tools...")
if build_frontend == "build":
call( call(
"pip", ["python", "-m", "pip", "install", "--upgrade", "pip", *dependency_constraint_flags],
"install",
"--upgrade",
"build[virtualenv]",
*dependency_constraint_flags,
env=env, env=env,
) )
elif build_frontend == "build[uv]": call(["pip", "--version"], env=env)
assert uv_path is not None
call( call(
uv_path, ["pip", "install", "--upgrade", "setuptools", "wheel", *dependency_constraint_flags],
"pip",
"install",
"--upgrade",
"build[virtualenv]",
*dependency_constraint_flags,
env=env, env=env,
) )
if python_libs_base: return env
# Set up the environment for various backends to enable cross-compilation
setup_setuptools_cross_compile(tmp, python_configuration, python_libs_base, env)
setup_rust_cross_compile(tmp, python_configuration, python_libs_base, env)
return base_python, env
def build(options: Options, tmp_path: Path) -> None: def pep_518_cp35_workaround(package_dir: Path, env: Dict[str, str]) -> None:
python_configurations = get_python_configurations( """
options.globals.build_selector, options.globals.architectures Python 3.5 PEP 518 hack (see https://github.com/pypa/pip/issues/8392#issuecomment-639563494)
) Basically, nuget's Python is an embedded Python distribution, which is not supported by pip.
Before version 3.6, there was no way to disable the "embedded" behavior, including the ignoring
of environment variables, including the ones pip uses to setup PEP 518 builds.
if not python_configurations: The fix here is as suggested in that issue; we manually setup the PEP 518 requirements. Since we
return are in a fresh environment (except for pinned cibuildweel dependencies), the build is already
mostly "isolated".
"""
pyproject_path = package_dir / "pyproject.toml"
if pyproject_path.exists():
data = toml.load(pyproject_path)
requirements = data["build-system"].get("requires", []) if "build-system" in data else []
if requirements:
log.step("Performing PEP518 workaround...")
with tempfile.TemporaryDirectory() as d:
reqfile = Path(d) / "requirements.txt"
with reqfile.open("w") as f:
for r in requirements:
print(r, file=f)
call(["pip", "install", "-r", reqfile], env=env)
def build(options: BuildOptions) -> None:
temp_dir = Path(tempfile.mkdtemp(prefix="cibuildwheel"))
built_wheel_dir = temp_dir / "built_wheel"
repaired_wheel_dir = temp_dir / "repaired_wheel"
try: try:
before_all_options_identifier = python_configurations[0].identifier if options.before_all:
before_all_options = options.build_options(before_all_options_identifier)
if before_all_options.before_all:
log.step("Running before_all...") log.step("Running before_all...")
env = before_all_options.environment.as_dictionary(prev_environment=os.environ) env = options.environment.as_dictionary(prev_environment=os.environ)
before_all_prepared = prepare_command( before_all_prepared = prepare_command(
before_all_options.before_all, project=".", package=options.globals.package_dir options.before_all, project=".", package=options.package_dir
) )
shell(before_all_prepared, env=env) shell(before_all_prepared, env=env)
built_wheels: list[Path] = [] python_configurations = get_python_configurations(
options.build_selector, options.architectures
)
for config in python_configurations: for config in python_configurations:
build_options = options.build_options(config.identifier)
build_frontend = build_options.build_frontend or BuildFrontendConfig("pip")
use_uv = (
build_frontend.name == "build[uv]"
and Version(config.version) >= Version("3.8")
and not config.identifier.startswith("pp38-")
)
log.build_start(config.identifier) log.build_start(config.identifier)
identifier_tmp_dir = tmp_path / config.identifier
identifier_tmp_dir.mkdir()
built_wheel_dir = identifier_tmp_dir / "built_wheel"
repaired_wheel_dir = identifier_tmp_dir / "repaired_wheel"
dependency_constraint_flags: Sequence[PathOrStr] = [] dependency_constraint_flags: Sequence[PathOrStr] = []
if build_options.dependency_constraints: if options.dependency_constraints:
dependency_constraint_flags = [ dependency_constraint_flags = [
"-c", "-c",
build_options.dependency_constraints.get_for_python_version(config.version), options.dependency_constraints.get_for_python_version(config.version),
] ]
# install Python # install Python
base_python, env = setup_python( env = setup_python(config, dependency_constraint_flags, options.environment)
identifier_tmp_dir / "build",
config,
dependency_constraint_flags,
build_options.environment,
build_frontend.name,
)
if not use_uv:
pip_version = get_pip_version(env)
compatible_wheel = find_compatible_wheel(built_wheels, config.identifier)
if compatible_wheel:
log.step_end()
print(
f"\nFound previously built wheel {compatible_wheel.name}, that's compatible with {config.identifier}. Skipping build step..."
)
repaired_wheel = compatible_wheel
else:
# run the before_build command # run the before_build command
if build_options.before_build: if options.before_build:
log.step("Running before_build...") log.step("Running before_build...")
before_build_prepared = prepare_command( before_build_prepared = prepare_command(
build_options.before_build, options.before_build, project=".", package=options.package_dir
project=".",
package=options.globals.package_dir,
) )
shell(before_build_prepared, env=env) shell(before_build_prepared, env=env)
# activate the PEP 518 patch if on Windows Python 3.5
# (will only have an effect if PEP 517 builds are used):
if config.version.startswith("3.5"):
pep_518_cp35_workaround(options.package_dir, env)
log.step("Building wheel...") log.step("Building wheel...")
built_wheel_dir.mkdir() if built_wheel_dir.exists():
shutil.rmtree(built_wheel_dir)
extra_flags = split_config_settings( built_wheel_dir.mkdir(parents=True)
build_options.config_settings, build_frontend.name
)
extra_flags += build_frontend.args
build_env = env.copy()
if not use_uv:
build_env["VIRTUALENV_PIP"] = pip_version
if build_options.dependency_constraints:
constraints_path = build_options.dependency_constraints.get_for_python_version(
config.version
)
combine_constraints(build_env, constraints_path, identifier_tmp_dir)
if build_frontend.name == "pip":
extra_flags += get_build_verbosity_extra_flags(build_options.build_verbosity)
# Path.resolve() is needed. Without it pip wheel may try to fetch package from pypi.org # Path.resolve() is needed. Without it pip wheel may try to fetch package from pypi.org
# see https://github.com/pypa/cibuildwheel/pull/369 # see https://github.com/pypa/cibuildwheel/pull/369
call( call(
"python", [
"-m",
"pip", "pip",
"wheel", "wheel",
options.globals.package_dir.resolve(), options.package_dir.resolve(),
f"--wheel-dir={built_wheel_dir}", "-w",
built_wheel_dir,
"--no-deps", "--no-deps",
*extra_flags, *get_build_verbosity_extra_flags(options.build_verbosity),
env=build_env, ],
env=env,
) )
elif build_frontend.name == "build" or build_frontend.name == "build[uv]":
if not 0 <= build_options.build_verbosity < 2:
msg = f"build_verbosity {build_options.build_verbosity} is not supported for build frontend. Ignoring."
log.warning(msg)
if use_uv and "--no-isolation" not in extra_flags and "-n" not in extra_flags:
extra_flags.append("--installer=uv")
call(
"python",
"-m",
"build",
build_options.package_dir,
"--wheel",
f"--outdir={built_wheel_dir}",
*extra_flags,
env=build_env,
)
else:
assert_never(build_frontend)
built_wheel = next(built_wheel_dir.glob("*.whl")) built_wheel = next(built_wheel_dir.glob("*.whl"))
# repair the wheel # repair the wheel
repaired_wheel_dir.mkdir() if repaired_wheel_dir.exists():
shutil.rmtree(repaired_wheel_dir)
repaired_wheel_dir.mkdir(parents=True)
if built_wheel.name.endswith("none-any.whl"): if built_wheel.name.endswith("none-any.whl"):
raise errors.NonPlatformWheelError() raise NonPlatformWheelError()
if build_options.repair_command: if options.repair_command:
log.step("Repairing wheel...") log.step("Repairing wheel...")
repair_command_prepared = prepare_command( repair_command_prepared = prepare_command(
build_options.repair_command, options.repair_command, wheel=built_wheel, dest_dir=repaired_wheel_dir
wheel=built_wheel,
dest_dir=repaired_wheel_dir,
) )
shell(repair_command_prepared, env=env) shell(repair_command_prepared, env=env)
else: else:
@@ -489,39 +331,16 @@ def build(options: Options, tmp_path: Path) -> None:
repaired_wheel = next(repaired_wheel_dir.glob("*.whl")) repaired_wheel = next(repaired_wheel_dir.glob("*.whl"))
if repaired_wheel.name in {wheel.name for wheel in built_wheels}: if options.test_command and options.test_selector(config.identifier):
raise errors.AlreadyBuiltWheelError(repaired_wheel.name)
test_selected = options.globals.test_selector(config.identifier)
if test_selected and config.arch == "ARM64" != platform_module.machine():
log.warning(
unwrap(
"""
While arm64 wheels can be built on other platforms, they cannot
be tested. An arm64 runner is required. To silence this warning,
set `CIBW_TEST_SKIP: "*-win_arm64"`.
"""
)
)
# skip this test
elif test_selected and build_options.test_command:
log.step("Testing wheel...") log.step("Testing wheel...")
# set up a virtual environment to install and test from, to make sure # set up a virtual environment to install and test from, to make sure
# there are no dependencies that were pulled in at build time. # there are no dependencies that were pulled in at build time.
if not use_uv: call(["pip", "install", "virtualenv", *dependency_constraint_flags], env=env)
call("pip", "install", "virtualenv", *dependency_constraint_flags, env=env) venv_dir = Path(tempfile.mkdtemp())
venv_dir = identifier_tmp_dir / "venv-test" # Use --no-download to ensure determinism by using seed libraries
# built into virtualenv
if use_uv: call(["python", "-m", "virtualenv", "--no-download", venv_dir], env=env)
call("uv", "venv", venv_dir, f"--python={base_python}", env=env)
else:
# Use pip version from the initial env to ensure determinism
venv_args = ["--no-periodic-update", f"--pip={pip_version}"]
# In Python<3.12, setuptools & wheel are installed as well, use virtualenv embedded ones
if Version(config.version) < Version("3.12"):
venv_args.extend(("--setuptools=embed", "--wheel=embed"))
call("python", "-m", "virtualenv", *venv_args, venv_dir, env=env)
virtualenv_env = env.copy() virtualenv_env = env.copy()
virtualenv_env["PATH"] = os.pathsep.join( virtualenv_env["PATH"] = os.pathsep.join(
@@ -530,64 +349,46 @@ def build(options: Options, tmp_path: Path) -> None:
virtualenv_env["PATH"], virtualenv_env["PATH"],
] ]
) )
virtualenv_env["VIRTUAL_ENV"] = str(venv_dir)
# check that we are using the Python from the virtual environment # check that we are using the Python from the virtual environment
call("where", "python", env=virtualenv_env) call(["where", "python"], env=virtualenv_env)
if build_options.before_test: if options.before_test:
before_test_prepared = prepare_command( before_test_prepared = prepare_command(
build_options.before_test, options.before_test,
project=".", project=".",
package=build_options.package_dir, package=options.package_dir,
) )
shell(before_test_prepared, env=virtualenv_env) shell(before_test_prepared, env=virtualenv_env)
pip = ["uv", "pip"] if use_uv else ["pip"]
# install the wheel # install the wheel
call( call(
*pip, ["pip", "install", str(repaired_wheel) + options.test_extras],
"install",
str(repaired_wheel) + build_options.test_extras,
env=virtualenv_env, env=virtualenv_env,
) )
# test the wheel # test the wheel
if build_options.test_requires: if options.test_requires:
call(*pip, "install", *build_options.test_requires, env=virtualenv_env) call(["pip", "install"] + options.test_requires, env=virtualenv_env)
# run the tests from a temp dir, with an absolute path in the command # run the tests from c:\, with an absolute path in the command
# (this ensures that Python runs the tests against the installed wheel # (this ensures that Python runs the tests against the installed wheel
# and not the repo code) # and not the repo code)
test_command_prepared = prepare_command( test_command_prepared = prepare_command(
build_options.test_command, options.test_command,
project=Path(".").resolve(), project=Path(".").resolve(),
package=options.globals.package_dir.resolve(), package=options.package_dir.resolve(),
wheel=repaired_wheel,
) )
test_cwd = identifier_tmp_dir / "test_cwd" shell(test_command_prepared, cwd="c:\\", env=virtualenv_env)
test_cwd.mkdir()
(test_cwd / "test_fail.py").write_text(test_fail_cwd_file.read_text())
shell(test_command_prepared, cwd=test_cwd, env=virtualenv_env)
# we're all done here; move it to output (remove if already exists)
if compatible_wheel is None:
output_wheel = build_options.output_dir.joinpath(repaired_wheel.name)
moved_wheel = move_file(repaired_wheel, output_wheel)
if moved_wheel != output_wheel.resolve():
log.warning(
"{repaired_wheel} was moved to {moved_wheel} instead of {output_wheel}"
)
built_wheels.append(output_wheel)
# clean up # clean up
# (we ignore errors because occasionally Windows fails to unlink a file and we shutil.rmtree(venv_dir)
# don't want to abort a build because of that)
shutil.rmtree(identifier_tmp_dir, ignore_errors=True)
# we're all done here; move it to output (remove if already exists)
shutil.move(str(repaired_wheel), options.output_dir)
log.build_end() log.build_end()
except subprocess.CalledProcessError as error: except subprocess.CalledProcessError as error:
msg = f"Command {error.cmd} failed with code {error.returncode}. {error.stdout or ''}" log.step_end_with_error(
raise errors.FatalError(msg) from error f"Command {error.cmd} failed with code {error.returncode}. {error.stdout}"
)
sys.exit(1)
+5 -1014
View File
File diff suppressed because it is too large Load Diff
+24 -91
View File
@@ -2,8 +2,6 @@
title: Contributing title: Contributing
--- ---
# Contributing
Wheel-building can be pretty complex. We expect users to find edge-cases - please help the rest of the community out by documenting these, adding features to support them, and reporting bugs. Wheel-building can be pretty complex. We expect users to find edge-cases - please help the rest of the community out by documenting these, adding features to support them, and reporting bugs.
If you have an idea for a modification or feature, it's probably best to raise an issue first and discuss it with the maintainer team. Once we have rough consensus on a design, begin work in a PR. If you have an idea for a modification or feature, it's probably best to raise an issue first and discuss it with the maintainer team. Once we have rough consensus on a design, begin work in a PR.
@@ -12,7 +10,8 @@ If you have an idea for a modification or feature, it's probably best to raise a
Everyone contributing to the cibuildwheel project is expected to follow the [PSF Code of Conduct](https://github.com/pypa/.github/blob/main/CODE_OF_CONDUCT.md). Everyone contributing to the cibuildwheel project is expected to follow the [PSF Code of Conduct](https://github.com/pypa/.github/blob/main/CODE_OF_CONDUCT.md).
## Design Goals Design Goals
------------
- `cibuildwheel` should wrap the complexity of wheel building. - `cibuildwheel` should wrap the complexity of wheel building.
- The user interface to `cibuildwheel` is the build script (e.g. `.travis.yml`). Feature additions should not increase the complexity of this script. - The user interface to `cibuildwheel` is the build script (e.g. `.travis.yml`). Feature additions should not increase the complexity of this script.
@@ -21,7 +20,7 @@ Everyone contributing to the cibuildwheel project is expected to follow the [PSF
Other notes: Other notes:
- The platforms are very similar, until they're not. I'd rather have straight-forward code than totally DRY code, so let's keep airy platform abstractions to a minimum. - The platforms are very similar, until they're not. I'd rather have straight-forward code than totally DRY code, so let's keep airy platfrom abstractions to a minimum.
- I might want to break the options into a shared config file one day, so that config is more easily shared. That has motivated some of the design decisions. - I might want to break the options into a shared config file one day, so that config is more easily shared. That has motivated some of the design decisions.
### cibuildwheel's relationship with build errors ### cibuildwheel's relationship with build errors
@@ -32,94 +31,20 @@ We're not responsible for errors in those tools, for fixing errors/crashes there
So, if we can, I'd like to improve the experience on errors as well. In [this](https://github.com/pypa/cibuildwheel/issues/139) case, it takes a bit of knowledge to understand that the Linux builds are happening in a different OS via Docker, that the linked symbols won't match, that auditwheel will fail because of this. A problem with how the tools fit together, instead of the tools themselves. So, if we can, I'd like to improve the experience on errors as well. In [this](https://github.com/pypa/cibuildwheel/issues/139) case, it takes a bit of knowledge to understand that the Linux builds are happening in a different OS via Docker, that the linked symbols won't match, that auditwheel will fail because of this. A problem with how the tools fit together, instead of the tools themselves.
## Development Maintainer notes
----------------
### Running the tests ### Local testing
When making a change to the codebase, you can run tests locally for quicker feedback than the CI runs on a PR. You can [run them directly](#making-a-venv), but the easiest way to run tests is using [nox](https://nox.thea.codes/). You should run:
You can run all the tests locally by doing: ```python
python3 -m venv venv
```bash . venv/bin/activate
nox -s tests
```
However, because this takes a while, you might prefer to be more specific.
#### Unit tests
To run the project's unit tests, do:
```bash
nox -s tests -- unit_test
```
There are a few custom options to enable different parts of the test suite - check `nox -s tests -- unit_test --help` for details.
If you're calling this a lot, you might consider using the `-r` or `-R` arguments to nox to make it a bit faster. This calls pytest under the hood, so to target a specific test, use pytest's `-k` option after the `--` above to select a specific test.
#### Integration tests
To run the project's integration tests, do:
```bash
nox -s tests -- test
```
The integration test suite is big - it can take more than 30 minutes to run the whole thing.
Because it takes such a long time, normally you'd choose specific tests to run locally, and rely on the project's CI for the rest. Use pytest's `-k` option to choose specific tests. You can pass a test name or a filename, it'll run everything that matches.
```bash
nox -s tests -- test -k <test_name_or_filename>
# e.g.
nox -s tests -- test -k before_build
```
A few notes-
- Because they run inside a container, Linux tests can run on all platforms where Docker is installed, so they're convenient for running integration tests locally. Set CIBW_PLATFORM to do this: `CIBW_PLATFORM=linux nox -s tests -- test`.
- Running the macOS integration tests requires _system installs_ of Python from python.org for all the versions that are tested. We won't attempt to install these when running locally, but you can do so manually using the URL in the error message that is printed when the install is not found.
#### Making a venv
More advanced users might prefer to invoke pytest directly-
```bash
python3 -m venv .venv
source .venv/bin/activate
pip install -e .[dev] pip install -e .[dev]
# run the unit tests
pytest unit_test
# run the whole integration test suite
pytest test
# run a specific integration test
pytest test -k test_build_frontend_args
# run a specific integration test on a different platform
CIBW_PLATFORM=linux pytest test -k test_build_frontend_args
``` ```
### Linting, docs To prepare a development environment.
Most developer tasks have a nox interface. This allows you to very simply run tasks without worrying about setting up a development environment (as shown below). This is a slower than setting up a development environment and reusing it, but has the (important) benefit of being highly reproducible; an earlier run does not affect a current run, or anything else on your machine.
You can see a list of sessions by typing `nox -l`; here are a few common ones:
```console
nox -s lint # Run the linters (default)
nox -s tests [-- PYTEST-ARGS] # Run the tests (default)
nox -s docs # Build and serve the documentation
nox -s build # Make SDist and wheel
```
More advanced users can run the update scripts. `update_pins` should work directly, but `update_constraints` needs all versions of Python installed. If you don't want to do that locally, a fast way to run it to use docker to run nox:
```console
docker run --rm -itv $PWD:/src -w /src quay.io/pypa/manylinux_2_24_x86_64:latest pipx run nox -s update_constraints
```
## Maintainer notes
### Testing sample configs ### Testing sample configs
@@ -148,6 +73,9 @@ This has been moved to using docker, so you only need the following instructions
The dependency update script in the next section requires multiple python versions installed. One way to do this is to use `pyenv`: The dependency update script in the next section requires multiple python versions installed. One way to do this is to use `pyenv`:
```bash ```bash
pyenv install 2.7.18
pyenv install 3.5.9
pyenv install 3.6.11
pyenv install 3.7.8 pyenv install 3.7.8
# Optionally add 3.8 and make it the local version; # Optionally add 3.8 and make it the local version;
# otherwise assuming 3.8+ already is your current python version # otherwise assuming 3.8+ already is your current python version
@@ -156,6 +84,10 @@ pyenv install 3.7.8
Then, you need to make the required virtual environments: Then, you need to make the required virtual environments:
```bash ```bash
$(pyenv prefix 2.7.18)/bin/python -m pip install virtualenv
$(pyenv prefix 2.7.18)/bin/python -m virtualenv env27
$(pyenv prefix 3.5.9)/bin/python -m venv env35
$(pyenv prefix 3.6.11)/bin/python -m venv env36
$(pyenv prefix 3.7.8)/bin/python -m venv env37 $(pyenv prefix 3.7.8)/bin/python -m venv env37
``` ```
@@ -177,14 +109,15 @@ Then, increment the project version number using:
bin/bump_version.py bin/bump_version.py
``` ```
(or `nox -s bump_version`) You'll be prompted to enter the new version number. Update the changelog when prompted. The script will create a 'bump version' commit and version tag. You'll be prompted to enter the new version number. Update the changelog when prompted. The script will create a 'bump version' commit and version tag.
Finally, cut the release and push to GitHub. Finally, cut the release and upload to PyPI/GitHub.
```bash ```bash
rm -rf dist
python setup.py sdist bdist_wheel
twine upload dist/*
git push && git push --tags git push && git push --tags
``` ```
Then head to https://github.com/pypa/cibuildwheel/releases and create a GitHub release from the new tag, pasting in the changelog entry. Once the release is created inside GitHub, a CI job will create the assets and upload them to PyPI. Then head to https://github.com/pypa/cibuildwheel/releases and create a GitHub release from the new tag, pasting in the changelog entry.
If there were any schema updates, run `pipx run ./bin/generate_schema.py --schemastore > partial-cibuildwheel.json` and contribute the changes to SchemaStore.
+48 -4
View File
@@ -2,13 +2,15 @@
title: Modern C++ standards title: Modern C++ standards
--- ---
# Modern C++ standards
Building Python wheels with modern C++ standards (C++11 and later) requires a few tricks. Building Python wheels with modern C++ standards (C++11 and later) requires a few tricks.
## Python 2.7 and C++17
The Python 2.7 header files use the `register` keyword, which is [reserved and unused from C+17 onwards](https://en.cppreference.com/w/cpp/keyword/register). Compiling a wheel for Python 2.7 with the C++17 standard is still possible to allow usage of `register` using proper flag `-Wno-register` for gcc/clang and `/wd5033` for MSVC.
## manylinux1 and C++14 ## manylinux1 and C++14
The old `manylinux1` image (based on CentOS 5) contains a version of GCC and libstdc++ that only supports C++11 and earlier standards. There are however ways to compile wheels with the C++14 standard (and later): https://github.com/pypa/manylinux/issues/118 The default `manylinux1` image (based on CentOS 5) contains a version of GCC and libstdc++ that only supports C++11 and earlier standards. There are however ways to compile wheels with the C++14 standard (and later): https://github.com/pypa/manylinux/issues/118
`manylinux2010` and `manylinux2014` are newer and support all C++ standards (up to C++17). `manylinux2010` and `manylinux2014` are newer and support all C++ standards (up to C++17).
@@ -16,7 +18,7 @@ The old `manylinux1` image (based on CentOS 5) contains a version of GCC and lib
OS X/macOS allows you to specify a so-called "deployment target" version that will ensure backwards compatibility with older versions of macOS. One way to do this is by setting the `MACOSX_DEPLOYMENT_TARGET` environment variable. OS X/macOS allows you to specify a so-called "deployment target" version that will ensure backwards compatibility with older versions of macOS. One way to do this is by setting the `MACOSX_DEPLOYMENT_TARGET` environment variable.
However, to enable modern C++ standards, the deployment target needs to be set high enough (since older OS X/macOS versions did not have the necessary modern C++ standard library). However, to enable modern C++ standards, the deploment target needs to be set high enough (since older OS X/macOS versions did not have the necessary modern C++ standard library).
To get C++11 and C++14 support, `MACOSX_DEPLOYMENT_TARGET` needs to be set to (at least) `"10.9"`. By default, `cibuildwheel` already does this, building 64-bit-only wheels for macOS 10.9 and later. To get C++11 and C++14 support, `MACOSX_DEPLOYMENT_TARGET` needs to be set to (at least) `"10.9"`. By default, `cibuildwheel` already does this, building 64-bit-only wheels for macOS 10.9 and later.
@@ -25,3 +27,45 @@ To get C++17 support, Xcode 9.3+ is needed, requiring at least macOS 10.13 on th
However, if only C++17 compiler and standard template library (STL) features are used (not needing a C++17 runtime) it might be possible to set `MACOSX_DEPLOYMENT_TARGET` to a lower value, such as `"10.9"`. To find out if this is the case, try compiling and running with a lower `MACOSX_DEPLOYMENT_TARGET`: if C++17 features are used that require a more recent deployment target, building the wheel should fail. However, if only C++17 compiler and standard template library (STL) features are used (not needing a C++17 runtime) it might be possible to set `MACOSX_DEPLOYMENT_TARGET` to a lower value, such as `"10.9"`. To find out if this is the case, try compiling and running with a lower `MACOSX_DEPLOYMENT_TARGET`: if C++17 features are used that require a more recent deployment target, building the wheel should fail.
For more details see https://en.cppreference.com/w/cpp/compiler_support, https://en.wikipedia.org/wiki/Xcode, and https://xcodereleases.com/: Xcode 10 needs macOS 10.13 and Xcode 11 needs macOS 10.14. For more details see https://en.cppreference.com/w/cpp/compiler_support, https://en.wikipedia.org/wiki/Xcode, and https://xcodereleases.com/: Xcode 10 needs macOS 10.13 and Xcode 11 needs macOS 10.14.
## Windows and Python 2.7
In previous years, Microsoft distributed a compiler toolchain called 'Visual C++ for Python 2.7', which was a distribution of MSVC 2008 that was created to make it easier to build Python 2.7 extensions on Windows, because it was fully compatible with the toolchain that built Python 2.7.
This toolchain does not support modern C++ standards (i.e., C++11 and later). And it is hard to find this toolchain these days, since Microsoft removed the download for the required Visual Studio 2008 needed to build a native extension in April, 2021. So, by default, cibuildwheel does not attempt to build Python 2.7 extensions on Windows.
There is an optional workaround, though: the pybind11 project argues and shows that it is [possible to compile Python 2.7 extension with a newer compiler](https://pybind11.readthedocs.io/en/stable/faq.html#working-with-ancient-visual-studio-2008-builds-on-windows) and has an example project showing how to do this: https://github.com/pybind/python_example. The main catch is that a user might need to install [a newer "Microsoft Visual C++ Redistributable"](https://support.microsoft.com/en-us/help/2977003/the-latest-supported-visual-c-downloads), since the newer C++ standard library's binaries are not included by default with the Python 2.7 installation.
Forcing `distutils` or `setuptools` to use a more recent version of MSVC that supports modern C++ can be done in the following way:
1. Set the environment variables `DISTUTILS_USE_SDK=1` and `MSSdk=1`. These two environment variables will tell `distutils`/`setuptools` to not search and set up a build environment that uses Visual C++ for Python 2.7 (aka. MSVC 9).
2. Set up the build Visual Studio build environment you want to use, making sure that e.g. `PATH` contains `cl`, `link`, etc.
- Usually, this can be done through `vcvarsall.bat x86` or `vcvarsall.bat x64`. The exact location of this file depends on the installation, but the default path for VS 2019 Community (e.g. used in AppVeyor's `Visual Studio 2019` image) is `C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Auxiliary\Build\vcvarsall.bat`. **Note**: `vcvarsall.bat` changes the environment variables, so this cannot be run in a subprocess/subshell and consequently running `vsvarsall.bat` in `CIBW_BEFORE_BUILD` does not have any effect.
- In Azure Pipelines, [a `VSBuild` task is available](https://docs.microsoft.com/en-us/azure/devops/pipelines/tasks/build/visual-studio-build)
- In GitHub Actions, the default shell is powershell, so you'll need to use `shell: cmd` to source a `.bat` file, and the directory is `"C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Auxiliary\Build\vcvarsall.bat"`. Or you can use an action, such as the [`ilammy/msvc-dev-cmd@v1`](https://github.com/ilammy/msvc-dev-cmd) action to setup the environment (see example below).
3. Next, call `cibuildwheel`. Unfortunately, MSVC has separate toolchains for compiling 32-bit and 64-bit, so you will need to run `cibuildwheel` twice: once with `CIBW_BUILD=*-win32` after setting up the `x86` build environment, and once with `CIBW_BUILD=*-win_amd64` in a `x64` environment (see previous step).
**GitHub Action example with ilammy/msvc-dev-cmd**:
```yaml
- uses: ilammy/msvc-dev-cmd@v1
- name: Build 64-bit wheel
run: python -m cibuildwheel --output-dir wheelhouse
env:
CIBW_BUILD: cp27-win_amd64
DISTUTILS_USE_SDK: 1
MSSdk: 1
- uses: ilammy/msvc-dev-cmd@v1
with:
arch: x86
- name: Build 32-bit wheel
run: python -m cibuildwheel --output-dir wheelhouse
env:
CIBW_BUILD: cp27-win32
DISTUTILS_USE_SDK: 1
MSSdk: 1
```
Binary file not shown.

Before

Width:  |  Height:  |  Size: 67 KiB

-37
View File
@@ -1,37 +0,0 @@
{
"$schema": "http://json-schema.org/draft-07/schema#",
"$id": "https://github.com/pypa/cibuildwheel/blob/main/cibuildwheel/docs/data/projects.schema.json",
"type": "array",
"description": "The projects file for cibuildwheel",
"items": {
"type": "object",
"additionalProperties": false,
"properties": {
"name": { "type": "string" },
"gh": { "type": "string", "pattern": "[^/]*/[^/]*" },
"stars": { "$ref": "#/items/properties/gh" },
"pypi": { "type": "string" },
"notes": { "type": "string" },
"ci": {
"type": "array",
"items": {
"enum": [
"github",
"travisci",
"appveyor",
"circleci",
"gitlab",
"cirrusci",
"azurepipelines"
]
}
},
"ci_config": { "type": "string" },
"os": {
"type": "array",
"items": { "enum": ["windows", "apple", "linux"] }
}
},
"required": ["name", "gh", "ci", "os"]
}
}
+12 -313
View File
@@ -4,15 +4,9 @@
# stars: GitHub repo (optional, if different from package, such as for Twisted) # stars: GitHub repo (optional, if different from package, such as for Twisted)
# pypi: The pypi name, if different from the GitHub package name # pypi: The pypi name, if different from the GitHub package name
# os: Operating system list, [windows, apple, linux] (optional) # os: Operating system list, [windows, apple, linux] (optional)
# ci: [appveyor, github, azurepipelines, circleci, gitlab, travisci, cirrusci] (optional) # ci: [appveyor, github, azurepipelines, circleci, gitlab, travisci] (optional)
# notes: (text, optional) # notes: (text, optional)
- name: abess
gh: abess-team/abess
notes: A fast best-subset selection library. It uses cibuildwheel to build a large project with C++ extensions.
ci: [github]
os: [windows, apple, linux]
- name: Matplotlib - name: Matplotlib
gh: matplotlib/matplotlib gh: matplotlib/matplotlib
notes: The venerable Matplotlib, a Python library with C++ portions notes: The venerable Matplotlib, a Python library with C++ portions
@@ -26,7 +20,7 @@
notes: A simple C extension, without external dependencies notes: A simple C extension, without external dependencies
- name: websockets - name: websockets
gh: python-websockets/websockets gh: aaugustin/websockets
ci: [travisci] ci: [travisci]
ci_config: .travis.yml ci_config: .travis.yml
os: [apple, linux] os: [apple, linux]
@@ -108,7 +102,7 @@
- name: pikepdf - name: pikepdf
gh: pikepdf/pikepdf gh: pikepdf/pikepdf
ci: [github] ci: [azurepipelines]
os: [windows, apple, linux] os: [windows, apple, linux]
- name: fathon - name: fathon
@@ -172,13 +166,11 @@
- name: pyzmq - name: pyzmq
gh: zeromq/pyzmq gh: zeromq/pyzmq
ci: [github, circleci] ci: [github]
os: [windows, apple, linux] os: [windows, apple, linux]
ci_config: .github/workflows/wheels.yml ci_config: .github/workflows/wheels.yml
notes: | notes: |
Python bindings for zeromq, the networking library. Python bindings for zeromq, the networking library. Uses Cython and CFFI.
Uses Cython on CPython and CFFI on PyPy.
ARM wheels for linux are built natively on CircleCI.
- name: python-rapidjson - name: python-rapidjson
gh: python-rapidjson/python-rapidjson gh: python-rapidjson/python-rapidjson
@@ -206,12 +198,6 @@
ci: [github] ci: [github]
os: [windows, apple, linux] os: [windows, apple, linux]
- name: pybind11 cross build example
gh: wbarnha/pybind_cmake_example_crossbuild
ci: [github, gitlab]
os: [windows, apple, linux]
notes: Same as pybind11 cmake_example but used to demo Linux ARM + Windows + macOS builds on GitLab
- name: iminuit - name: iminuit
gh: scikit-hep/iminuit gh: scikit-hep/iminuit
ci: [github] ci: [github]
@@ -270,8 +256,8 @@
- name: Tokenizer - name: Tokenizer
gh: OpenNMT/Tokenizer gh: OpenNMT/Tokenizer
ci: [github] ci: [github, travisci]
os: [windows, apple, linux] os: [apple, linux]
- name: matrixprofile - name: matrixprofile
gh: matrix-profile-foundation/matrixprofile gh: matrix-profile-foundation/matrixprofile
@@ -286,8 +272,8 @@
- name: CTranslate2 - name: CTranslate2
gh: OpenNMT/CTranslate2 gh: OpenNMT/CTranslate2
ci: [github] ci: [github]
os: [windows, apple, linux] os: [apple, linux]
notes: Includes libraries from the [Intel oneAPI toolkit](https://www.intel.com/content/www/us/en/developer/tools/oneapi/base-toolkit.html) and CUDA kernels compiled for multiple GPU architectures. notes: Includes libraries from the [Intel oneAPI toolkit](https://software.intel.com/content/www/us/en/develop/tools/oneapi/base-toolkit.html). The Linux wheels also include CUDA libraries for GPU execution.
- name: MyPy - name: MyPy
gh: mypyc/mypy_mypyc-wheels gh: mypyc/mypy_mypyc-wheels
@@ -295,7 +281,7 @@
pypi: mypy pypi: mypy
ci: [github] ci: [github]
os: [apple, linux, windows] os: [apple, linux, windows]
notes: The compiled version of MyPy using MyPyC. notes: MyPyC, the compiled component of MyPy.
- name: Imagecodecs (fork) - name: Imagecodecs (fork)
gh: czaki/imagecodecs_build gh: czaki/imagecodecs_build
@@ -325,12 +311,12 @@
notes: Full range of wheels for setuptools rust, with auto release and PyPI deploy. notes: Full range of wheels for setuptools rust, with auto release and PyPI deploy.
- name: python-snappy - name: python-snappy
gh: intake/python-snappy gh: andrix/python-snappy
ci: [github] ci: [github]
os: [apple, linux, windows] os: [apple, linux, windows]
- name: sourmash - name: sourmash
gh: sourmash-bio/sourmash gh: dib-lab/sourmash
ci: [github] ci: [github]
os: [apple, linux, windows] os: [apple, linux, windows]
@@ -377,290 +363,3 @@
gh: LAMDA-NJU/Deep-Forest gh: LAMDA-NJU/Deep-Forest
ci: [github] ci: [github]
os: [apple, linux, windows] os: [apple, linux, windows]
- name: cmake
gh: scikit-build/cmake-python-distributions
stars: Kitware/CMake
ci: [github, travisci]
os: [apple, linux, windows]
notes: Multitagged binary builds for all supported platforms, using cibw 2 config configuration.
- name: ninja
gh: scikit-build/ninja-python-distributions
ci: [github, travisci]
os: [apple, linux, windows]
notes: Multitagged binary builds for all supported platforms, using cibw 2 config configuration.
- name: tgcalls
gh: MarshalX/tgcalls
pypi: pytgcalls
ci: [github]
os: [apple, windows]
notes: Python `pybind11` binding to Telegram's WebRTC library with third party dependencies like `OpenSSL`, `MozJPEG`, `FFmpeg`, etc.
- name: pillow-heif
gh: bigcat88/pillow_heif
pypi: pillow-heif
ci: [github, cirrusci]
os: [apple, linux, windows]
notes: Bindings to libheif library with third party dependencies. Fully automated CI for tests and publishing including Apple Silicon builds.
- name: clang-format
gh: ssciwr/clang-format-wheel
ci: [github]
os: [apple, linux, windows]
notes: Scikit-build wrapper around LLVM's CMake, all platforms, generic wheels.
- name: Tornado
gh: tornadoweb/tornado
ci: [github]
os: [linux, apple, windows]
notes: Tornado is a Python web framework and asynchronous networking library. Uses stable ABI for a small C extension.
- name: pytorch-fairseq
gh: facebookresearch/fairseq
ci: [github]
os: [apple, linux]
- name: uvloop
gh: MagicStack/uvloop
ci: [github]
os: [apple, linux]
- name: asyncpg
gh: MagicStack/asyncpg
ci: [github]
os: [apple, linux, windows]
- name: vaex
gh: vaexio/vaex
ci: [github]
os: [apple, linux, windows]
- name: cvxpy
gh: cvxpy/cvxpy
ci: [github]
os: [apple, linux, windows]
- name: duckdb
gh: duckdb/duckdb
ci: [github]
os: [apple, linux, windows]
- name: Triton
gh: openai/triton
ci: [github]
os: [linux]
notes: Self hosted runners
- name: vispy
gh: vispy/vispy
ci: [github]
os: [apple, linux, windows]
- name: Confluent client for Kafka
gh: confluentinc/confluent-kafka-python
ci: [travisci]
os: [apple, linux, windows]
notes: setup in `tools/wheels/build-wheels.bat`
- name: tinyobjloader
gh: tinyobjloader/tinyobjloader
ci: [azurepipelines]
os: [apple, linux, windows]
- name: PyOxidizer
gh: indygreg/PyOxidizer
ci: [github]
os: [apple, linux, windows]
- name: OpenSpiel
gh: google-deepmind/open_spiel
ci: [github]
os: [apple, linux]
- name: Google Benchmark
gh: google/benchmark
ci: [github]
os: [apple, linux, windows]
- name: Apache Beam
gh: apache/beam
ci: [github]
os: [apple, linux, windows]
- name: PyCryptodome
gh: Legrandin/pycryptodome
ci: [github]
os: [apple, linux, windows]
- name: PyYAML
gh: yaml/pyyaml
ci: [github]
os: [apple]
- name: Line Profiler
gh: pyutils/line_profiler
ci: [github]
os: [apple, linux, windows]
- name: OpenTimelineIO
gh: AcademySoftwareFoundation/OpenTimelineIO
ci: [github]
os: [apple, linux, windows]
- name: OpenColorIO
gh: AcademySoftwareFoundation/OpenColorIO
ci: [github]
os: [apple, linux, windows]
- name: Psycopg 3
gh: psycopg/psycopg
pypi: psycopg_binary
notes: A modern implementation of a PostgreSQL adapter for Python
ci: [github]
os: [windows, apple, linux]
- name: Arbor
gh: arbor-sim/arbor
ci: [github]
os: [apple, linux]
notes: >
Arbor is a multi-compartment neuron simulation library; compatible with
next-generation accelerators; best-practices applied to research software;
focused on community-driven development. Includes a
[small script](https://github.com/arbor-sim/arbor/blob/master/scripts/patchwheel.py)
patching `rpath` in bundled libraries.
- name: Kivy
gh: kivy/kivy
ci: [github]
os: [windows, apple, linux]
- name: NCNN
gh: Tencent/ncnn
ci: [github]
os: [windows, apple, linux]
- name: Prophet
gh: facebook/prophet
ci: [github]
os: [windows, apple, linux]
- name: MemRay
gh: bloomberg/memray
ci: [github]
os: [linux]
- name: PyGame
gh: pygame/pygame
ci: [github]
os: [apple, linux]
- name: UltraJSON
gh: ultrajson/ultrajson
ci: [github]
os: [windows, apple, linux]
- name: NumPy
gh: numpy/numpy
ci: [github, travisci]
os: [windows, apple, linux]
- name: Wrapt
gh: GrahamDumpleton/wrapt
ci: [github]
os: [windows, apple, linux]
- name: SimpleJSON
gh: simplejson/simplejson
ci: [github]
os: [windows, apple, linux]
- name: Implicit
gh: benfred/implicit
ci: [github]
os: [windows, apple, linux]
notes: Includes GPU support for linux wheels
- name: power-grid-model
gh: PowerGridModel/power-grid-model
ci: [github]
os: [windows, apple, linux]
notes: Python/C++ library for distribution power system analysis
- name: Python-WebRTC
gh: MarshalX/python-webrtc
ci: [github]
os: [windows, apple, linux]
- name: cf-units
gh: SciTools/cf-units
ci: [github]
os: [apple, linux]
- name: envd
gh: tensorchord/envd
ci: [github]
os: [apple, linux, windows]
notes: A machine learning development environment build tool
- name: mosec
gh: mosecorg/mosec
ci: [github]
os: [linux, apple]
notes: A machine learning model serving framework powered by Rust
- name: ril
gh: Cryptex-github/ril-py
ci: [github]
os: [windows, apple, linux]
pypi: pyril
notes: A python binding to Rust Imaging library using maturin and Pyo3, utilizes Github Action cache to improve speed. Builds abi3 wheels.
- name: Picologging
gh: microsoft/picologging
ci: [github]
os: [windows, apple, linux]
pypi: picologging
notes: A high-performance logging library for Python.
- name: aalink
gh: artfwo/aalink
ci: [github]
os: [windows, apple, linux]
pypi: aalink
notes: Async Python interface for Ableton Link.
- name: werpy
gh: analyticsinmotion/werpy
ci: [github]
os: [windows, linux, apple]
pypi: werpy
notes: An ultra-fast python package using optimized dynamic programming to compute the Word Error Rate (WER).
- name: keyvi
gh: KeyviDev/keyvi
ci: [github]
os: [linux, apple]
pypi: keyvi
notes: FST based key value index highly optimized for size and lookup performance, utilizes ccache action for improved runtime
- name: pedalboard
gh: spotify/pedalboard
ci: [github]
os: [windows, linux, apple]
pypi: pedalboard
notes: >
A Python library for working with audio data and audio plugins
by wrapping the [JUCE](https://github.com/juce-framework/JUCE/)
C++ framework. Uses cibuildwheel to deploy on as many operating
systems and Python versions as possible with only one dependency
(any NumPy).
- name: streaming-form-data
gh: siddhantgoel/streaming-form-data
ci: [github]
os: [apple, linux, windows]
pypi: streaming-form-data
notes: Streaming parser for multipart/form-data written in Cython
+1 -1
View File
@@ -1 +1 @@
<svg xmlns="http://www.w3.org/2000/svg" role="img" viewBox="0 0 24 24" width="16px" fill="#606060"><title>Azure Pipelines icon</title><path d="M1.7997 22.1998h4.2001V24H.0004v-6.0002h1.7993v4.2zM12.6229 9.8975l-7.6353 7.6353 1.4792 1.4792 7.6345-7.6362-1.4784-1.4783zM4.365 15.1793l-.391-.3908c-.5042-.5042-.5828-1.3004-.1869-1.8934l4.6036-6.8958H.9979c-.5513.0014-.9975.4487-.9975 1v5.5002l3.5021 3.6036.8625-.9238zM23.9996.9999v8.9306a2 2 0 0 1-.8893 1.6632l-12.2834 8.2005a.9999.9999 0 0 1-1.2622-.1246L8.04 18.1457c.3313-.3313 6.4656-6.4672 6.7684-6.7698l-2.1855-2.1855L5.8533 15.96l-1.5258-1.525a1.0001 1.0001 0 0 1-.1246-1.2623L12.4034.8893A2 2 0 0 1 14.0666 0h8.9331c.5522 0 .9999.4477.9999.9999zm-2.9997 5.0003c0-1.6567-1.343-2.9997-2.9997-2.9997s-2.9997 1.343-2.9997 2.9997 1.343 2.9997 2.9997 2.9997 2.9997-1.343 2.9997-2.9997zM10.272 20.4625c-.4008 0-.7776-.156-1.0609-.4392l-.368-.3679-.8435.8417 3.4996 3.4996h5.5003c.5522 0 .9999-.4477.9999-.9999v-7.3899L11.1045 20.21a1.4944 1.4944 0 0 1-.8325.2525z"/></svg> <svg xmlns="http://www.w3.org/2000/svg" role="img" viewBox="0 0 24 24" width="16px" fill="#606060"><title>Azure Pipelines icon</title><path d="M1.8 18H0v6h6v-1.8H1.8zM23 0h-8.93a2 2 0 0 0-1.67.89L9 5.99H1a1 1 0 0 0-1 1v5.5l3.5 3.61 1.2-1.29 1.5 1.5L5 17.53l1.48 1.48 1.22-1.22 1.5 1.51L8 20.5l3.5 3.5H17a1 1 0 0 0 1-1v-8l5.11-3.4A2 2 0 0 0 24 9.92V1a1 1 0 0 0-1-1zm-5 9a3 3 0 1 1 0-6 3 3 0 0 1 0 6z"/></svg>

Before

Width:  |  Height:  |  Size: 1023 B

After

Width:  |  Height:  |  Size: 407 B

-1
View File
@@ -1 +0,0 @@
<svg xmlns="http://www.w3.org/2000/svg" role="img" viewBox="0 0 24 24" width="16px" fill="#606060"><title>Cirrus CI icon</title><path d="M19.422.453a1.113 1.113 0 0 0-1.113 1.115 1.113 1.113 0 0 0 1.112 1.114c1.31 0 2.35 1.042 2.35 2.363 0 1.32-1.04 2.363-2.35 2.363H1.112A1.113 1.113 0 0 0 0 8.52a1.113 1.113 0 0 0 1.113 1.117h18.31c1.308 0 2.35 1.042 2.35 2.363 0 1.32-1.042 2.363-2.35 2.363H1.112A1.113 1.113 0 0 0 0 15.48a1.113 1.113 0 0 0 1.113 1.112h18.31c1.308 0 2.35 1.042 2.35 2.363 0 1.32-1.042 2.363-2.35 2.363H1.112A1.113 1.113 0 0 0 0 22.432a1.113 1.113 0 0 0 1.113 1.115h18.31a1.113 1.113 0 0 0 .206-.022c2.42-.112 4.37-2.12 4.37-4.57 0-1.393-.642-2.634-1.63-3.478C23.356 14.632 24 13.393 24 12c0-1.393-.643-2.632-1.63-3.477C23.357 7.68 24 6.438 24 5.045c0-2.52-2.06-4.592-4.578-4.592z"/></svg>

Before

Width:  |  Height:  |  Size: 808 B

+11 -76
View File
@@ -2,100 +2,35 @@
title: Delivering to PyPI title: Delivering to PyPI
--- ---
# Delivering to PyPI
After you've built your wheels, you'll probably want to deliver them to PyPI. After you've built your wheels, you'll probably want to deliver them to PyPI.
## Manual method ### Manual method
On your development machine, install [pipx](https://pypa.github.io/pipx/) and do the following: On your development machine, do the following...
```bash ```bash
# Either download the SDist from your CI, or make it:
# Clear out your 'dist' folder. # Clear out your 'dist' folder.
rm -rf dist rm -rf dist
# Make a source distribution # Make a source distribution
pipx run build --sdist python setup.py sdist
# 🏃🏻 # 🏃🏻
# Go and download your wheel files from wherever you put them. e.g. your CI # Go and download your wheel files from wherever you put them. e.g. your CI
# provider can be configured to store them for you. Put them all into the # provider can be configured to store them for you. Put them all into the
# 'dist' folder. # 'dist' folder.
# Upload using 'twine' # Upload using 'twine' (you may need to 'pip install twine')
pipx run twine upload dist/* twine upload dist/*
``` ```
## Automatic method ### Semi-automatic method using wheelhouse-uploader
If you don't need much control over the release of a package, you can set up Obviously, manual steps are for chumps, so we can automate this a little by using [wheelhouse-uploader](https://github.com/ogrisel/wheelhouse-uploader).
cibuildwheel to deliver the wheels straight to PyPI. You just need to bump the
version and tag it.
### Generic instructions > Quick note from me - using S3 as a storage didn't work due to a [bug](https://issues.apache.org/jira/browse/LIBCLOUD-792) in libcloud. Feel free to use my fork of that package that fixes the bug `pip install https://github.com/joerick/libcloud/archive/v1.5.0-s3fix.zip`
Make your SDist with the [build](https://github.com/pypa/build) tool, and your wheels with cibuildwheel. If you can make the files available as ### Automatic method
downloadable artifacts, this make testing before releases easier (depending on your CI provider's options). The "publish" job/step should collect the
files, and then run `twine upload <paths>` (possibly via [pipx](https://github.com/pypa/pipx)); this should only happen on tags or "releases".
### GitHub Actions If you don't need much control over the release of a package, you can set up cibuildwheel to deliver the wheels straight to PyPI. This doesn't require anycloud storage to work - you just need to bump the version and tag it.
GitHub actions has pipx in all the runners as a supported package manager, as [`examples/travis-ci-deploy.yml`](https://github.com/pypa/cibuildwheel/blob/master/examples/travis-ci-deploy.yml) and [`examples/github-deploy.yml`](https://github.com/pypa/cibuildwheel/blob/master/examples/github-deploy.yml) are example configurations that automatically upload wheels to PyPI. Also check out [this example repo](https://github.com/pypa/cibuildwheel-autopypi-example) for more detailed instructions on how to set this up.
well as several useful actions. Alongside your existing job(s) that runs cibuildwheel to make wheels, you will probably want to build an SDist:
```yaml
make_sdist:
name: Make SDist
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0 # Optional, use if you use setuptools_scm
submodules: true # Optional, use if you have submodules
- name: Build SDist
run: pipx run build --sdist
- uses: actions/upload-artifact@v4
with:
name: cibw-sdist
path: dist/*.tar.gz
```
Then, you need to publish the artifacts that the previous jobs have built. This final job should run only on release or tag, depending on your preference. It gathers the artifacts from the sdist and wheel jobs and uploads them to PyPI. The release environment (`pypi` in the example below) will be created the first time this workflow runs.
This requires setting this GitHub workflow in your project's PyPI settings (for a [new project](https://docs.pypi.org/trusted-publishers/creating-a-project-through-oidc)/[existing project](https://docs.pypi.org/trusted-publishers/adding-a-publisher)).
```yaml
upload_all:
needs: [build_wheels, make_sdist]
environment: pypi
permissions:
id-token: write
runs-on: ubuntu-latest
if: github.event_name == 'release' && github.event.action == 'published'
steps:
- uses: actions/download-artifact@v4
with:
pattern: cibw-*
path: dist
merge-multiple: true
- uses: pypa/gh-action-pypi-publish@release/v1
```
You should use Dependabot to keep the publish action up to date. In the above
example, the same name (the default, "artifact" is used for all upload-artifact
runs, so we can just download all of the in one step into a common directory.
See
[`examples/github-deploy.yml`](https://github.com/pypa/cibuildwheel/blob/main/examples/github-deploy.yml)
for an example configuration that automatically upload wheels to PyPI. Also see
[scikit-hep.org/developer/gha_wheels](https://scikit-hep.org/developer/gha_wheels)
for a complete guide.
### TravisCI
See
[`examples/travis-ci-deploy.yml`](https://github.com/pypa/cibuildwheel/blob/main/examples/travis-ci-deploy.yml)
for an example configuration.
-487
View File
@@ -1,487 +0,0 @@
<div id="flow-diagram">
<div class="diagram-grid" v-bind:style="gridStyle">
<!-- BEHIND THE GRID CONTENT -->
<!-- line for each platform -->
<div class="grid-line" style="grid-row-start: 3"></div>
<div class="grid-line" style="grid-row-start: 4"></div>
<div class="grid-line" style="grid-row-start: 5"></div>
<!-- the docker container outline -->
<div class="grid-outline docker">
<div class="outline">
Manylinux container
</div>
</div>
<!-- the venv outlines -->
<div class="grid-outline testVenv" style="grid-row-start: 3">
<div class="outline">Test virtualenv</div>
</div>
<div class="grid-outline testVenv" style="grid-row-start: 4"><div class="outline"></div></div>
<div class="grid-outline testVenv" style="grid-row-start: 5"><div class="outline"></div></div>
<!-- THE GRID CONTENT -->
<!-- the row labels -->
<div class="grid-row-label" style="grid-row-start: 3">Linux</div>
<div class="grid-row-label" style="grid-row-start: 4">macOS</div>
<div class="grid-row-label" style="grid-row-start: 5">Windows</div>
<!-- the column labels -->
<div class="grid-column-label"
style="grid-row: 1 / span 1; grid-column: 5 / -3">
<div class="label">For each version of Python</div>
</div>
<div class="grid-column-label"
style="grid-row: 2 / span 1;
grid-column: 9 / -3;
margin-bottom: 0.5em;">
<div class="label">If tests are configured</div>
</div>
<!-- the steps -->
<div class="step" v-for="(step, sIndex) in diagram.steps">
<component v-for="action in step"
class="action"
v-bind:is="action.href ? 'a' : 'div'"
v-bind:href="action.href">
<div class="dots" v-if="action.style == 'dot'">
<div class="dot"
v-info="action"
v-for="(platform, pIndex) in platforms"
v-if="action.platforms.includes(platform)"
v-bind:style="{gridRowStart: `${pIndex+3}`,
gridColumnStart: `${sIndex+2}`}">
<div class="dot-graphic"></div>
</div>
</div>
<div class="block-container" v-if="action.style == 'block'">
<div class="block"
v-info="action"
v-bind:style="{...blockStyle(action),
gridColumnStart: `${sIndex+2}`}">
{{action.label}}
</div>
</div>
</a>
</div>
</div>
</div>
<noscript>
<img src="data/how-it-works.png" alt="Diagram showing the steps for each platform">
<p><strong>Enable JavaScript for an interactive version of this diagram.</strong></p>
</noscript>
<script type="module" async>
import Vue from 'https://cdn.jsdelivr.net/npm/vue@2.6.14/dist/vue.esm.browser.js'
import 'https://unpkg.com/@popperjs/core@2/dist/umd/popper.min.js'
import 'https://unpkg.com/tippy.js@6/dist/tippy-bundle.umd.js'
const diagram = {
steps: [
[
{
label: 'copy project into container',
platforms: ['linux'],
style: 'block',
width: 2,
},
],
[], // empty column so that the manylinux outline intersects the previous column in the middle.
[
{
href: 'options/#before-all',
platforms: ['linux', 'macos', 'windows'],
style: 'dot',
tooltip: {
title: 'CIBW_BEFORE_ALL',
tag: 'Optional step',
description: 'Execute a shell command on the build system before any wheels are built.'
},
},
],
[
{
label: 'Install Python',
platforms: ['macos', 'windows'],
style: 'block',
tooltip: {
description: 'Install the version of Python required to build this wheel.'
},
},
],
[
{
href: 'options/#before-build',
platforms: ['linux', 'macos', 'windows'],
style: 'dot',
tooltip: {
title: 'CIBW_BEFORE_BUILD',
tag: 'Optional step',
description: "Execute a shell command preparing each wheel's build.",
},
},
],
[
{
label: 'Build wheel',
href: 'options/#build-frontend',
platforms: ['linux', 'macos', 'windows'],
style: 'block',
tooltip: {
title: 'CIBW_BUILD_FRONTEND',
tag: 'Customisable step',
description: 'Build the wheel according to your package configuration, using the frontend of your choice - pip or build.'
},
},
],
[
{
label: 'Repair using auditwheel',
href: 'options/#repair-wheel-command',
platforms: ['linux'],
style: 'block',
tooltip: {
title: 'CIBW_REPAIR_WHEEL_COMMAND',
tag: 'Customisable step',
description: 'Bundle shared libraries and ensure manylinux compliance by running auditwheel on each built wheel.'
},
},
{
label: 'Repair using delocate',
href: 'options/#repair-wheel-command',
platforms: ['macos'],
style: 'block',
tooltip: {
title: 'CIBW_REPAIR_WHEEL_COMMAND',
tag: 'Customisable step',
description: 'Bundle shared libraries by running delocate on each built wheel.'
},
},
{
env: "CIBW_REPAIR_WHEEL_COMMAND",
href: 'options/#repair-wheel-command',
label: 'repair wheel',
platforms: ['windows'],
style: 'dot',
optional: true,
tooltip: {
title: 'CIBW_REPAIR_WHEEL_COMMAND',
tag: 'Optional step',
description: 'Execute a shell command to repair each built wheel'
},
},
],
[
{
href: 'options/#before-test',
platforms: ['linux', 'macos', 'windows'],
style: 'dot',
tooltip: {
title: 'CIBW_BEFORE_TEST',
tag: 'Optional step',
description: 'Execute a shell command before testing each wheel'
},
},
],
[
{
label: 'Install wheel',
platforms: ['linux', 'macos', 'windows'],
style: 'block',
tooltip: {
description: 'Install the wheel we just built into the test virtualenv.'
},
},
],
[
{
label: 'Test wheel',
href: 'options/#test-command',
platforms: ['linux', 'macos', 'windows'],
style: 'block',
tooltip: {
title: 'CIBW_TEST_COMMAND',
tag: 'Optional step',
description: 'Execute a shell command to test each built wheel'
},
},
],
[
{
label: 'Copy wheels out of container',
platforms: ['linux'],
style: 'block',
width: 2,
},
],
[],
]
}
const diagramComponent = new Vue({
el: '#flow-diagram',
data() {
return {
diagram,
platforms: ['linux', 'macos', 'windows'],
}
},
methods: {
blockStyle(action) {
let start, end
if (action.platforms.includes('linux')) {
start = 3
} else if (action.platforms.includes('macos')) {
start = 4
} else if (action.platforms.includes('windows')) {
start = 5
}
if (action.platforms.includes('windows')) {
end = 6
} else if (action.platforms.includes('macos')) {
end = 5
} else if (action.platforms.includes('linux')) {
end = 4
}
return {
gridRowStart: start.toString(),
gridRowEnd: end.toString(),
gridColumnEnd: `span ${action.width || 1}`,
}
}
},
computed: {
gridStyle() {
return {
gridTemplateRows: `auto auto repeat(${this.platforms.length}, 1fr)`,
gridTemplateColumns: `repeat(${this.diagram.steps.length+1}, auto)`,
}
},
},
directives: {
info: {
inserted(el, binding) {
const action = binding.value
const {env, label, optional=false, description='', href=''} = action
const tooltip = action.tooltip
if (tooltip) {
const tippyInstance = tippy(el, {
content: `
<a class="tooltip-contents" href="${href || ''}">
<div class="tooltip-title">
${tooltip.title || ''}
</div>
<div class="tooltip-tag">
${tooltip.tag || ''}
</div>
<div class="tooltip-description">
${tooltip.description}
</div>
</a>
`,
placement: 'right-start',
allowHTML: true,
maxWidth: 'none',
appendTo: document.getElementById('flow-diagram'),
offset: [0, 10],
onShow(instance) {
const stepEl = el.closest('.action')
stepEl.classList.add('tooltip-open')
instance.setProps({
interactive: tippy.currentInput.isTouch
})
},
onHide() {
const stepEl = el.closest('.action')
stepEl.classList.remove('tooltip-open')
}
})
el.addEventListener('click', e => {
// click event should just open the tooltip on touch devices
if (tippy.currentInput.isTouch) {
e.preventDefault()
}
})
}
}
}
}
})
</script>
<style>
#flow-diagram {
background: #fcfcfc;
/* font-family: Inter; */
/* font-size: 10px; */
font-size: 0.7em;
padding-bottom: 2em;
}
.no-js #flow-diagram {
display: none
}
.diagram-grid {
display: grid;
overflow-x: auto;
}
.platform, .step, .dots, .block-container, .action {
display: contents;
}
.block, .dot {
position: relative;
margin: 2em 0.5em;
}
.block {
background-color: #eee;
/* margin: 0.5em auto; */
padding: 0.5em;
max-width: 6em;
display: flex;
align-items: center;
justify-content: center;
text-align: center;
}
.dot {
/* margin: 0.5em; */
display: flex;
align-items: center;
justify-content: center;
}
.dot-graphic {
width: 1.3em;
height: 1.3em;
border-radius: 50%;
background: #DADADA;
}
a.action {
color: inherit;
}
.action.tooltip-open .dot-graphic {
background-color: #416EDA;
}
.action.tooltip-open .block {
background-color: #416EDA;
color: white;
}
.grid-column-label {
height: 2em;
position: relative;
display: flex;
align-items: center;
justify-content: center;
margin: 0 0.5em;
}
.grid-column-label::before {
content: "";
height: 1px;
background-color: currentColor;
position: absolute;
top: calc(50% - 0.5px);
left: 0;
right: 0;
}
.grid-column-label .label {
background: #fcfcfc;
padding: 0 1em;
position: relative;
}
.grid-row-label {
grid-column: 1 / span 1;
display: flex;
align-items: center;
justify-content: flex-end;
text-align: right;
background: #fcfcfc;
position: relative;
padding-right: 1em;
font-size: 1.1em;
}
.grid-line {
grid-column: 1 / -1;
grid-row-end: span 1;
position: relative;
}
.grid-line::after {
content: "";
height: 1px;
background-color: #979797;
position: absolute;
top: calc(50% - 1px);
left: 0;
right: 0;
}
.grid-outline {
grid-row-end: span 1;
position: relative;
color: #8f8f8f;
}
.grid-outline.docker {
grid-row: 3 / span 1;
grid-column: 3 / -2;
}
.grid-outline.testVenv {
grid-column: 9 / span 3;
}
.grid-outline .outline {
position: absolute;
border: 1px solid #e5e5e5;
padding: 1px 3px;
}
.grid-outline.docker .outline {
left: 0;
right: 0;
top: 0;
bottom: 0;
}
.grid-outline.testVenv .outline {
left: 0;
right: 0;
top: 0.5em;
bottom: 0.5em;
}
/* style the tooltip */
.tippy-box {
background: white;
box-shadow: 0 2px 6px -1px rgba(0,0,0,0.12);
border-radius: 0;
text-align: left;
font-size: 1em;
color: #1F1F1F;
border: 1px solid rgba(0, 0, 0, 0.02);
width: 14em;
min-width: min-content;
}
.tippy-box[data-placement^='top'] > .tippy-arrow::before {
border-top-color: white;
}
.tippy-box[data-placement^='bottom'] > .tippy-arrow::before {
border-bottom-color: white;
}
.tippy-box[data-placement^='left'] > .tippy-arrow::before {
border-left-color: white;
}
.tippy-box[data-placement^='right'] > .tippy-arrow::before {
border-right-color: white;
}
a.tooltip-contents {
color: inherit;
text-decoration: none;
display: block;
}
.tooltip-title {
font-weight: 600;
font-size: 1.1em;
color: #416EDA;
}
.tooltip-tag {
font-weight: 500;
text-transform: uppercase;
font-size: 0.9em;
color: #C9C9C9;
}
.tooltip-description {
margin-top: 1px;
margin-bottom: 2px;
}
</style>
+9 -176
View File
@@ -1,25 +1,11 @@
/* Global styles */ /* Global styles */
body {
overflow-wrap: break-word;
overflow-x: hidden;
-webkit-font-smoothing: antialiased;
}
p { p {
margin-bottom: 12px; margin-bottom: 12px;
line-height: 26px; line-height: 26px;
} }
pre {
margin: 1em 0;
}
pre:first-child {
margin-top: 0;
}
code { code {
font-size: 85%; font-size: 85%;
background-color: rgba(27,31,35,.05); background-color: rgba(27,31,35,.05);
@@ -32,10 +18,9 @@ code {
code, pre, tt { code, pre, tt {
font-family: SFMono-Regular, Consolas, Liberation Mono, Menlo, monospace; font-family: SFMono-Regular, Consolas, Liberation Mono, Menlo, monospace;
line-height: 1.4;
} }
pre code, pre code.hljs { .hljs {
padding: 1em; padding: 1em;
background-color: #f6f8fa; background-color: #f6f8fa;
} }
@@ -63,13 +48,6 @@ h1, h2, h3, h4, h5, h6 {
border: none; border: none;
} }
@media screen and (max-width: 768px) {
.wy-nav-content-wrap .wy-nav-content {
padding: 1em;
}
}
/* /*
Code block filename style Code block filename style
@@ -113,10 +91,10 @@ h1, h2, h3, h4, h5, h6 {
.code-block-filename + pre code { .code-block-filename + pre code {
padding-top: 0.9em; padding-top: 0.9em;
} }
/*
.toctree-l3 { .toctree-l3 {
border-left: 10px solid transparent; border-left: 10px solid transparent;
} */ }
/* import font awesome 4 for icons */ /* import font awesome 4 for icons */
@import url(https://cdnjs.cloudflare.com/ajax/libs/font-awesome/4.7.0/css/font-awesome.min.css); @import url(https://cdnjs.cloudflare.com/ajax/libs/font-awesome/4.7.0/css/font-awesome.min.css);
@@ -134,21 +112,7 @@ h1, h2, h3, h4, h5, h6 {
background-color: white; background-color: white;
display: flex; display: flex;
justify-content: flex-start; justify-content: flex-start;
overflow-x: auto; border-bottom: 2px solid #ccc;
overflow-y: hidden;
position: relative;
}
.tabs-header::before {
position: absolute;
content: "";
bottom: 0;
left: 0;
right: 0;
height: 2px;
background: #ccc;
}
.tabs-header > * {
position: relative;
} }
/* Style the buttons that are used to open the tab content */ /* Style the buttons that are used to open the tab content */
@@ -162,13 +126,15 @@ h1, h2, h3, h4, h5, h6 {
font-size: 0.9em; font-size: 0.9em;
font-weight: 600; font-weight: 600;
padding: 8px 16px; padding: 8px 16px;
border-bottom: 2px solid #ccc; border-bottom: 2px solid transparent;
/* put the border on top of the parent border */
margin-bottom: -2px;
} }
.tabs-header button:focus-visible { .tabs-header button:focus-visible {
/* preserve an outline for accessibility purposes */ /* preserve an outline for accesibility purposes */
outline: 1px solid currentColor; outline: 1px solid currentColor;
outline-offset: -1px;
} }
/* Change background color of buttons on hover */ /* Change background color of buttons on hover */
@@ -187,137 +153,4 @@ h1, h2, h3, h4, h5, h6 {
background-color: #f1f6fa; background-color: #f1f6fa;
padding: 1px 1em; padding: 1px 1em;
padding-top: 0.8em; padding-top: 0.8em;
/* don't collapse inner margins */
overflow-y: hidden;
}
/* Examples tabs styling */
.tabs.examples .tabs-header {
justify-content: flex-end;
background-color: transparent;
border-bottom-style: none;
padding: 0;
}
.tabs.examples .tabs-header::before {
height: 0;
}
.tabs.examples .tabs-header button {
padding: 6px 12px;
border-bottom: none;
color: rgba(0, 0, 0, 0.3);
font-size: 0.8em;
border-top-left-radius: 4px;
border-top-right-radius: 4px;
}
.tabs-header button:hover {
background-color: unset;
}
.tabs.examples .tabs-header button.active {
border-bottom-style: none;
background: #f6f8fa;
color: inherit;
}
.tabs.examples .tabs-content {
padding: 0;
background-color: transparent;
background-color: #f6f8fa;
}
.tabs.examples .tabs-content .tab > * {
padding-left: 0.8em;
padding-right: 0.8em;
}
.tabs.examples .tabs-content .tab > pre {
padding-left: 0;
padding-right: 0;
margin-bottom: 0.5em;
}
/* mkdocs update theme fix-ups */
.rst-content code, .rst-content tt, code {
/* border: none; */
/* background-color: #f0f1f1; */
color: inherit;
}
.rst-content pre code {
border: none;
}
.rst-content h1 code,
.rst-content h2 code,
.rst-content h3 code,
.rst-content h4 code,
.rst-content h5 code,
.rst-content h6 code {
border: none;
color: inherit;
background-color: #f0f1f1;
font-size: 80%;
}
/* expand all the toctree entries */
.wy-menu-vertical .toctree-l1.current .toctree-l2>ul,
.wy-menu-vertical .toctree-l2.current .toctree-l3>ul,
.wy-menu-vertical .toctree-l3.current .toctree-l4>ul,
.wy-menu-vertical .toctree-l4.current .toctree-l5>ul,
.wy-menu-vertical .toctree-l5.current .toctree-l6>ul,
.wy-menu-vertical .toctree-l6.current .toctree-l7>ul,
.wy-menu-vertical .toctree-l7.current .toctree-l8>ul,
.wy-menu-vertical .toctree-l8.current .toctree-l9>ul,
.wy-menu-vertical .toctree-l9.current .toctree-l10>ul,
.wy-menu-vertical .toctree-l10.current .toctree-l11>ul {
display: block;
}
/* hide all the buttons */
.wy-menu-vertical li.current>a button.toctree-expand,
.wy-menu-vertical li.on a button.toctree-expand,
.wy-menu-vertical li.toctree-l2 button.toctree-expand,
.wy-menu-vertical li a button.toctree-expand {
display: none;
}
/* toctree layout improvements */
.wy-menu-vertical a {
padding: 0.4em 1.2em;
}
.wy-menu-vertical li.current>a,
.wy-menu-vertical li.on a {
font-weight: normal;
padding: 0.4em 1.2em;
/* border-right: 1px solid #f0f0f0; */
}
.wy-menu-vertical li.current a {
padding: 0.4em 1.2em;
}
.wy-menu-vertical li.toctree-l3 a,
.wy-menu-vertical li.toctree-l2.current li.toctree-l3>a {
padding-left: 1.8em;
}
.wy-menu-vertical li.toctree-l2.current>a {
padding: 0.4em 1.2em;
}
.wy-menu-vertical li.toctree-l2.current>a,
.wy-menu-vertical li.toctree-l2.current li.toctree-l3>a {
background: transparent;
}
.wy-menu-vertical li.toctree-l2.current>a:hover,
.wy-menu-vertical li.toctree-l2.current li.toctree-l3>a:hover {
background: #d6d6d6;
}
.wy-menu-vertical li.current>a.current {
background: #fafafa !important;
}
.wy-menu-vertical li.current a {
}
/* word wrap in table cells */
.wy-table-responsive table td, .wy-table-responsive table th {
white-space: normal;
line-height: 1.4;
} }
-11
View File
@@ -20,17 +20,6 @@ while (true) {
tabContainer.insertBefore(firstTab); tabContainer.insertBefore(firstTab);
tabContainer.append(headerContainer, contentContainer) tabContainer.append(headerContainer, contentContainer)
// add extra classes from the first tab to the container
const classes = Array.from(firstTab[0].classList)
for (let i = 0; i < classes.length; i++) {
const element = classes[i];
if (element == 'tab' || element == 'admonition') {
continue
}
tabContainer.addClass(element)
}
const selectTab = function (index) { const selectTab = function (index) {
headerContainer.children().removeClass('active') headerContainer.children().removeClass('active')
headerContainer.children().eq(index).addClass('active') headerContainer.children().eq(index).addClass('active')
+152 -295
View File
@@ -2,31 +2,33 @@
title: Tips and tricks title: Tips and tricks
--- ---
# Tips and tricks ### Troubleshooting
## Tips If your wheel didn't compile, check the list below for some debugging tips.
### Linux builds in containers - A mistake in your config. To quickly test your config without doing a git push and waiting for your code to build on CI, you can test the Linux build in a Docker container. On Mac or Linux, with Docker running, try `cibuildwheel --platform linux`. You'll have to bring your config into the current environment first.
Linux wheels are built in [`manylinux`/`musllinux` containers](https://github.com/pypa/manylinux) to provide binary compatible wheels on Linux, according to [PEP 600](https://www.python.org/dev/peps/pep-0600/) / [PEP 656](https://www.python.org/dev/peps/pep-0656/). Because of this, when building with `cibuildwheel` on Linux, a few things should be taken into account: - Missing dependency. You might need to install something on the build machine. You can do this in `.travis.yml`, `appveyor.yml`, or `.circleci/config.yml`, with apt-get, brew or choco. Given how the Linux build works, you'll need to use the [`CIBW_BEFORE_BUILD`](options.md#before-build) option.
- Programs and libraries are not installed on the CI runner host, but rather should be installed inside the container - using `yum` for `manylinux2010` or `manylinux2014`, `apt-get` for `manylinux_2_24`, `dnf` for `manylinux_2_28` and `apk` for `musllinux_1_1` or `musllinux_1_2`, or manually. The same goes for environment variables that are potentially needed to customize the wheel building. - Windows: missing C feature. The Windows C compiler doesn't support C language features invented after 1990, so you'll have to backport your C code to C90. For me, this mostly involved putting my variable declarations at the top of the function like an animal.
`cibuildwheel` supports this by providing the [`CIBW_ENVIRONMENT`](options.md#environment) and [`CIBW_BEFORE_ALL`](options.md#before-all) options to setup the build environment inside the running container. - MacOS: calling cibuildwheel from a python3 script and getting a `ModuleNotFoundError`? Due to a (fixed) [bug](https://bugs.python.org/issue22490) in CPython, you'll need to [unset the `__PYVENV_LAUNCHER__` variable](https://github.com/pypa/cibuildwheel/issues/133#issuecomment-478288597) before activating a venv.
- The project directory is copied into the container as `/project`, the output directory for the wheels to be copied out is `/output`. In general, this is handled transparently by `cibuildwheel`. For a more finegrained level of control however, the root of the host file system is mounted as `/host`, allowing for example to access shared files, caches, etc. on the host file system. Note that `/host` is not available on CircleCI and GitLab CI due to their Docker policies. ### Linux builds on Docker
- Alternative Docker images can be specified with the `CIBW_MANYLINUX_*_IMAGE`/`CIBW_MUSLLINUX_*_IMAGE` options to allow for a custom, preconfigured build environment for the Linux builds. See [options](options.md#linux-image) for more details. Linux wheels are built in the [`manylinux` docker images](https://github.com/pypa/manylinux) to provide binary compatible wheels on Linux, according to [PEP 571](https://www.python.org/dev/peps/pep-0571/). Because of this, when building with `cibuildwheel` on Linux, a few things should be taken into account:
- Programs and libraries are not installed on the Travis CI Ubuntu host, but rather should be installed inside of the Docker image (using `yum` for `manylinux2010` or `manylinux2014`, and `apt-get` for `manylinux_2_24`) or manually. The same goes for environment variables that are potentially needed to customize the wheel building. `cibuildwheel` supports this by providing the `CIBW_ENVIRONMENT` and `CIBW_BEFORE_BUILD` options to setup the build environment inside the running Docker image. See [the options docs](options.md#build-environment) for details on these options.
- The project directory is mounted in the running Docker instance as `/project`, the output directory for the wheels as `/output`. In general, this is handled transparently by `cibuildwheel`. For a more finegrained level of control however, the root of the host file system is mounted as `/host`, allowing for example to access shared files, caches, etc. on the host file system. Note that this is not available on CircleCI due to their Docker policies.
- Alternative dockers images can be specified with the `CIBW_MANYLINUX_X86_64_IMAGE`, `CIBW_MANYLINUX_I686_IMAGE`, and `CIBW_MANYLINUX_PYPY_X86_64_IMAGE` options to allow for a custom, preconfigured build environment for the Linux builds. See [options](options.md#manylinux-image) for more details.
### Building macOS wheels for Apple Silicon {: #apple-silicon} ### Building macOS wheels for Apple Silicon {: #apple-silicon}
`cibuildwheel` supports both native builds and cross-compiling between `arm64` (Apple Silicon) and `x86_64` (Intel) architectures, including the cross-compatible `universal2` format. `cibuildwheel` supports cross-compiling `universal2` and `arm64` wheels on `x86_64` runners. With the introduction of Apple Silicon, you now have several choices for wheels for Python 3.9+:
#### Overview of Mac architectures #### `x86_64`
You have several choices for wheels for Python 3.8+:
##### `x86_64`
The traditional wheel for Apple, loads on Intel machines, and on The traditional wheel for Apple, loads on Intel machines, and on
Apple Silicon when running Python under Rosetta 2 emulation. Apple Silicon when running Python under Rosetta 2 emulation.
@@ -34,19 +36,20 @@ Apple Silicon when running Python under Rosetta 2 emulation.
Due to a change in naming, Pip 20.3+ (or an installer using packaging 20.5+) Due to a change in naming, Pip 20.3+ (or an installer using packaging 20.5+)
is required to install a binary wheel on macOS Big Sur. is required to install a binary wheel on macOS Big Sur.
##### `arm64` #### `arm64`
The native wheel for macOS on Apple Silicon. The native wheel for macOS on Apple Silicon.
Requires Pip 20.3+ (or packaging 20.5+) to install. Requires Pip 20.3+ (or packaging 20.5+) to install.
##### `universal2` #### `universal2`
This wheel contains both architectures, causing it to be up to twice the This wheel contains both architectures, causing it to be up to twice the
size (data files do not get doubled, only compiled code). It requires size (data files do not get doubled, only compiled code). It requires
Pip 20.3 (Packaging 20.6+) to load on Intel, and Pip 21.0.1 (Packaging 20.9+) Pip 20.3 (Packaging 20.6+) to load on Intel, and Pip 21.0.1 (Packaging 20.9+)
to load on Apple Silicon. to load on Apple Silicon.
!!! note
The dual-architecture `universal2` has a few benefits, but a key benefit The dual-architecture `universal2` has a few benefits, but a key benefit
to a universal wheel is that a user can bundle these wheels into an to a universal wheel is that a user can bundle these wheels into an
application and ship a single binary. application and ship a single binary.
@@ -57,38 +60,36 @@ you might want to build all three, but in that case, pip will not download
the universal wheels, because it prefers the most specific wheel the universal wheels, because it prefers the most specific wheel
available. available.
#### What to provide?
Generally speaking, because Pip 20.3 is required for the `universal2` wheel, Generally speaking, because Pip 20.3 is required for the `universal2` wheel,
most packages should provide both `x86_64` and one of `universal2`/`arm64` most packages should provide both `x86_64` and `universal2` wheels for now.
wheels. When Pip 20.3+ is common on macOS, then it might be possible to ship Once Pip 20.3+ is common on macOS, then it should be possible to ship only the
only the `universal2` wheel. `universal2` wheel.
Opinions vary on which of arch-specific or `universal2` wheels are best - some packagers prefer `universal2` because it's one wheel for all Mac users, so simpler, and easier to build into apps for downstream users. However, because they contain code for both architectures, their file size is larger, meaning they consume more disk space and bandwidth, and are harder to build for some projects. **Apple Silicon wheels are not built by default**, but can be enabled by adding extra archs to the [`CIBW_ARCHS_MACOS` option](options.md#archs) - e.g. `x86_64 arm64 universal2`. Cross-compilation is provided by the Xcode toolchain.
See [GitHub issue 1333](https://github.com/pypa/cibuildwheel/issues/1333) for more discussion. !!! important
When cross-compiling on Intel, it is not possible to test `arm64` and the `arm64` part of a `universal2` wheel.
#### How? `cibuildwheel` will raise a warning to notify you of this - these warnings be be silenced by skipping testing on these platforms: `CIBW_TEST_SKIP: *_arm64 *_universal2:arm64`.
It's easiest to build `x86_64` wheels on `x86_64` runners, and `arm64` wheels on `arm64` runners. Hopefully, cross-compilation is a temporary situation. Once we have widely
available Apple Silicon CI runners, we can build and test `arm64` and
On GitHub Actions, `macos-14` runners are `arm64`, and `macos-13` runners are `x86_64`. So all you need to do is ensure both are in your build matrix. `universal2` wheels natively. That's why `universal2` wheels are not yet built
by default, and require opt-in by setting `CIBW_ARCHS_MACOS`.
#### Cross-compiling
If your CI provider doesn't offer arm64 runners yet, or you want to create `universal2`, you'll have to cross-compile. Cross-compilation can be enabled by adding extra archs to the [`CIBW_ARCHS_MACOS` option](options.md#archs) - e.g. `CIBW_ARCHS_MACOS="x86_64 universal2"`. Cross-compilation is provided by Xcode toolchain v12.2+.
Regarding testing,
- On an arm64 runner, it is possible to test x86_64 wheels and both parts of a universal2 wheel using Rosetta 2 emulation.
- On an x86_64 runner, arm64 code can be compiled but it can't be tested. `cibuildwheel` will raise a warning to notify you of this - these warnings can be silenced by skipping testing on these platforms: `CIBW_TEST_SKIP: "*_arm64 *_universal2:arm64"`.
!!! note !!! note
If your project uses **Poetry** as a build backend, cross-compiling on macOS [does not currently work](https://github.com/python-poetry/poetry/issues/7107). In some cases arm64 wheels can be built but their tags will be incorrect, with the platform tag showing `x86_64` instead of `arm64`. Your runner needs Xcode Command Line Tools 12.2 or later to build `universal2` or `arm64`.
As a workaround, the tag can be fixed before running delocate to repair the wheel. The [`wheel tags`](https://wheel.readthedocs.io/en/stable/reference/wheel_tags.html) command is ideal for this. See [this workflow](https://gist.github.com/anderssonjohan/49f07e33fc5cb2420515a8ac76dc0c95#file-build-pendulum-wheels-yml-L39-L53) for an example usage of `wheel tags`. So far, only CPython 3.9 supports `universal2` and `arm64` wheels.
### Building Linux wheels for non-native archs using emulation {: #emulation} Here's an example GitHub Actions workflow with a job that builds for Apple Silicon:
> .github/workflows/build_macos.yml
```yml
{% include "../examples/github-apple-silicon.yml" %}
```
### Building non-native architectures using emulation {: #emulation}
cibuildwheel supports building non-native architectures on Linux, via cibuildwheel supports building non-native architectures on Linux, via
emulation through the binfmt_misc kernel feature. The easiest way to use this emulation through the binfmt_misc kernel feature. The easiest way to use this
@@ -109,15 +110,63 @@ Linux), and the other architectures are emulated automatically.
{% include "../examples/github-with-qemu.yml" %} {% include "../examples/github-with-qemu.yml" %}
``` ```
### Building CPython ABI3 wheels (Limited API) {: #abi3} ### Building Apple Silicon wheels on Intel {: #apple-silicon}
The CPython Limited API is a subset of the Python C Extension API that's declared to be forward-compatible, meaning you can compile wheels for one version of Python, and they'll be compatible with future versions. Wheels that use the Limited API are known as ABI3 wheels. `cibuildwheel` supports cross-compiling `universal2` and `arm64` wheels on `x86_64` runners.
To create a package that builds ABI3 wheels, you'll need to configure your build backend to compile libraries correctly create wheels with the right tags. [Check this repo](https://github.com/joerick/python-abi3-package-sample) for an example of how to do this with setuptools. These wheels are not built by default, but can be enabled by setting the [`CIBW_ARCHS_MACOS` option](options.md#archs) to `x86_64 arm64 universal2`. Cross-compilation is provided by the Xcode toolchain.
You could also consider running [abi3audit](https://github.com/trailofbits/abi3audit) against the produced wheels in order to check for abi3 violations or inconsistencies. You can run it alongside the default in your [CIBW_REPAIR_WHEEL_COMMAND](options.md#repair-wheel-command). !!! important
When cross-compiling on Intel, it is not possible to test `arm64` and the `arm64` part of a `universal2` wheel.
### Packages with optional C extensions {: #optional-extensions} `cibuildwheel` will raise a warning to notify you of this - these warnings be be silenced by skipping testing on these platforms: `CIBW_TEST_SKIP: *_arm64 *_universal2:arm64`.
Hopefully, this is a temporary situation. Once we have widely available Apple Silicon CI runners, we can build and test `arm64` and `universal2` wheels more natively. That's why `universal2` wheels are not yet built by default, and require opt-in by setting `CIBW_ARCHS_MACOS`.
!!! note
Your runner image needs Xcode Command Line Tools 12.2 or later to build `universal2` and `arm64`.
So far, only CPython 3.9 supports `universal2` and `arm64` wheels.
Here's an example GitHub Actions workflow with a job that builds for Apple Silicon:
> .github/workflows/build_macos.yml
```yml
{% include "../examples/github-apple-silicon.yml" %}
```
### Windows and Python 2.7
Building 2.7 extensions on Windows is difficult, because the VS 2008 compiler that was used for the original Python compilation was discontinued in 2018, and has since been removed from Microsoft's downloads. Most people choose to not build Windows 2.7 wheels, or to override the compiler to something more modern.
To override, you need to have a modern compiler toolchain activated, and set `DISTUTILS_USE_SDK=1` and `MSSdk=1`. For example, on GitHub Actions, you would add these steps:
```yaml
- name: Prepare compiler environment for Windows
if: runner.os == 'Windows'
uses: ilammy/msvc-dev-cmd@v1
with:
arch: x64
- name: Set Windows environment variables
if: runner.os == 'Windows'
shell: bash
run: |
echo "DISTUTILS_USE_SDK=1" >> $GITHUB_ENV
echo "MSSdk=1" >> $GITHUB_ENV
# invoke cibuildwheel...
```
cibuildwheel will not try to build 2.7 on Windows unless it detects that the above two variables are set. Note that changing to a more modern compiler will mean your wheel picks up a runtime dependency to a different [Visual C++ Redistributable][]. You also need to be [a bit careful][] in designing your extension; some major binding tools like pybind11 do this for you.
More on setting a custom Windows toolchain in our docs on modern C++ standards [here](cpp_standards.md#windows-and-python-27).
[Visual C++ Redistributable]: https://support.microsoft.com/en-us/topic/the-latest-supported-visual-c-downloads-2647da03-1eea-4433-9aff-95f26a218cc0
[a bit careful]: https://pybind11.readthedocs.io/en/stable/faq.html#working-with-ancient-visual-studio-2008-builds-on-windows
### Building packages with optional C extensions
`cibuildwheel` defines the environment variable `CIBUILDWHEEL` to the value `1` allowing projects for which the C extension is optional to make it mandatory when building wheels. `cibuildwheel` defines the environment variable `CIBUILDWHEEL` to the value `1` allowing projects for which the C extension is optional to make it mandatory when building wheels.
@@ -131,183 +180,7 @@ myextension = Extension(
) )
``` ```
### Automatic updates using Dependabot {: #automatic-updates} ### 'No module named XYZ' errors after running cibuildwheel on macOS
Selecting a moving target (like the latest release) is generally a bad idea in CI. If something breaks, you can't tell whether it was your code or an upstream update that caused the breakage, and in a worse-case scenario, it could occur during a release.
There are two suggested methods for keeping cibuildwheel up to date that instead involve scheduled pull requests using GitHub's Dependabot.
#### Option 1: GitHub Action
If you use GitHub Actions for builds, you can use cibuildwheel as an action:
```yaml
uses: pypa/cibuildwheel@v2.20.0
```
This is a composite step that just runs cibuildwheel using pipx. You can set command-line options as `with:` parameters, and use `env:` as normal.
Then, your `.github/dependabot.yml` file could look like this:
```yaml
version: 2
updates:
- package-ecosystem: "github-actions"
directory: "/"
schedule:
interval: "weekly"
```
#### Option 2: Requirement files
The second option, and the only one that supports other CI systems, is using a `requirements-*.txt` file. The file should have a distinct name and have only one entry:
```bash
# requirements-cibw.txt
cibuildwheel==2.20.0
```
Then your install step would have `python -m pip install -r requirements-cibw.txt` in it. Your `.github/dependabot.yml` file could look like this:
```yaml
version: 2
updates:
- package-ecosystem: "pip"
directory: "/"
schedule:
interval: "daily"
```
This will also try to update other pins in all requirement files, so be sure you want to do that. The only control you have over the files used is via the directory option.
### Alternatives to cibuildwheel options {: #cibw-options-alternatives}
cibuildwheel provides lots of opportunities to configure the build
environment. However, you might consider adding this build configuration into
the package itself - in general, this is preferred, because users of your
package 'sdist' will also benefit.
#### Missing build dependencies {: #cibw-options-alternatives-deps}
If your build needs Python dependencies, rather than using `CIBW_BEFORE_BUILD`, it's best to add these to the
[`build-system.requires`](https://www.python.org/dev/peps/pep-0518/#build-system-table)
section of your pyproject.toml. For example, if your project requires Cython
to build, your pyproject.toml might include a section like this:
```toml
[build-system]
requires = [
"setuptools>=42",
"Cython",
]
build-backend = "setuptools.build_meta"
```
#### Actions you need to perform before building
You might need to run some other commands before building, like running a
script that performs codegen or downloading some data that's not stored in
your source tree.
Rather than using `CIBW_BEFORE_ALL` or `CIBW_BEFORE_BUILD`, you could incorporate
these steps into your package's build process. For example, if you're using
setuptools, you can add steps to your package's `setup.py` using a structure
like this:
```python
import subprocess
import setuptools
import setuptools.command.build_py
class BuildPyCommand(setuptools.command.build_py.build_py):
"""Custom build command."""
def run(self):
# your custom build steps here
# e.g.
# subprocess.run(['python', 'scripts/my_custom_script.py'], check=True)
setuptools.command.build_py.build_py.run(self)
setuptools.setup(
cmdclass={
'build_py': BuildPyCommand,
},
# Usual setup() args.
# ...
)
```
#### Compiler flags
Your build might need some compiler flags to be set through environment variables.
Consider incorporating these into your package, for example, in `setup.py` using [`extra_compile_args` or
`extra_link_args`](https://docs.python.org/3/distutils/setupscript.html#other-options).
### Python 2.7 / PyPy2 wheels
See the [cibuildwheel version 1 docs](https://cibuildwheel.pypa.io/en/1.x/) for information about building Python 2.7 or PyPy2 wheels. There are lots of tricks and workaround there that are no longer required for Python 3 in cibuildwheel 2.
## Troubleshooting
If your wheel didn't compile, you might have a mistake in your config.
To quickly test your config without doing a git push and waiting for your code to build on CI, you can [test the Linux build in a local Docker container](setup.md#local).
### Missing dependencies
Sometimes a build will fail due to a missing dependency.
**If the build is missing a Python package**, you should [add it to pyproject.toml](#cibw-options-alternatives-deps).
**If you need a build tool** (e.g. cmake, automake, ninja), you can install it through a package manager like apt/yum, brew or choco, using the [`CIBW_BEFORE_ALL`](options.md#before-all) option.
**If your build is linking into a native library dependency**, you can build/install that in [`CIBW_BEFORE_ALL`](options.md#before-all). However, on Linux, Mac (and Windows if you're using [delvewheel]), the library that you install will be bundled into the wheel in the [repair step]. So take care to ensure that
- the bundled library doesn't accidentally increase the minimum system requirements (such as the minimum macOS version)
- the bundled library matches the architecture of the wheel you're building when cross-compiling
This is particularly an issue on macOS, where de facto package manager Homebrew will install libraries that are compiled for the specific version of macOS that the build machine is running, rendering the wheels useless for any previous version. And brew will not install the right arch for cross compilation of Apple Silicon wheels.
For these reasons, it's strongly recommended to not use brew for native library dependencies. Instead, we recommend compiling the library yourself. If you compile in the [`CIBW_BEFORE_ALL`](options.md#before-all) step, cibuildwheel will have already set the appropriate `MACOSX_DEPLOYMENT_TARGET` env var, so the library will target the correct version of macOS.
!!! tip
For build steps, Homebrew is still a great resource - you can [look up the build formula](https://formulae.brew.sh/) and use that as a starting point.
[delvewheel]: https://github.com/adang1345/delvewheel
[repair step]: options.md#repair-wheel-command
[Homebrew]: https://brew.sh/
[delocate]: https://github.com/matthew-brett/delocate
### Building Rust wheels
If you build Rust wheels, you need to download the Rust compilers in manylinux.
If you support 32-bit Windows, you need to add this as a potential target. You
can do this on GitHub Actions, for example, with:
```yaml
CIBW_BEFORE_ALL_LINUX: curl -sSf https://sh.rustup.rs | sh -s -- -y
CIBW_BEFORE_ALL_WINDOWS: rustup target add i686-pc-windows-msvc
CIBW_ENVIRONMENT_LINUX: "PATH=$HOME/.cargo/bin:$PATH"
```
Rust does not provide Cargo for musllinux 32-bit, so that needs to be skipped:
```toml
[tool.cibuildwheel]
skip = ["*-musllinux_i686"]
```
Also see [maturin-action](https://github.com/PyO3/maturin-action) which is optimized for Rust wheels, builds the non-Python Rust modules once, and can cross-compile (and can build 32-bit musl, for example).
### macOS: ModuleNotFoundError
Calling cibuildwheel from a python3 script and getting a `ModuleNotFoundError`? Due to a (fixed) [bug](https://bugs.python.org/issue22490) in CPython, you'll need to [unset the `__PYVENV_LAUNCHER__` variable](https://github.com/pypa/cibuildwheel/issues/133#issuecomment-478288597) before activating a venv.
### macOS: 'No module named XYZ' errors after running cibuildwheel
`cibuildwheel` on Mac installs the distributions from Python.org system-wide during its operation. This is necessary, but it can cause some confusing errors after cibuildwheel has finished. `cibuildwheel` on Mac installs the distributions from Python.org system-wide during its operation. This is necessary, but it can cause some confusing errors after cibuildwheel has finished.
@@ -322,81 +195,20 @@ python3 -m twine upload wheelhouse/*.whl
This doesn't work because while `cibuildwheel` was running, it installed a few new versions of 'python3', so the `python3` run on line 3 isn't the same as the `python3` that ran on line 1. This doesn't work because while `cibuildwheel` was running, it installed a few new versions of 'python3', so the `python3` run on line 3 isn't the same as the `python3` that ran on line 1.
Solutions to this vary, but the simplest is to use pipx: Solutions to this vary, but the simplest is to install tools immediately before they're used:
```bash ```bash
# most runners have pipx preinstalled, but in case you don't python3 -m pip install cibuildwheel
python3 -m pip install pipx python3 -m cibuildwheel --output-dir wheelhouse
python3 -m pip install twine
pipx run cibuildwheel==2.20.0 --output-dir wheelhouse python3 -m twine upload wheelhouse/*.whl
pipx run twine upload wheelhouse/*.whl
``` ```
### macOS: Passing DYLD_LIBRARY_PATH to delocate ### 'ImportError: DLL load failed: The specific module could not be found' error on Windows
macOS has built-in [System Integrity protections](https://developer.apple.com/library/archive/documentation/Security/Conceptual/System_Integrity_Protection_Guide/RuntimeProtections/RuntimeProtections.html) which limits the use of `DYLD_LIBRARY_PATH` and `LD_LIBRARY_PATH` so that it does not automatically pass to children processes. This means if you set `DYLD_LIBRARY_PATH` before running cibuildwheel, or even set it in `CIBW_ENVIRONMENT`, it will be stripped out of the environment before delocate is called. Visual Studio and MSVC link the compiled binary wheels to the Microsoft Visual C++ Runtime. Normally, these are included with Python, but when compiling with a newer version of Visual Studio, it is possible users will run into problems on systems that do not have these runtime libraries installed. The solution is to ask users to download the corresponding Visual C++ Redistributable from the [Microsoft website](https://support.microsoft.com/en-us/help/2977003/the-latest-supported-visual-c-downloads) and install it. Since a Python installation normally includes these VC++ Redistributable files for [the version of the MSVC compiler used to compile Python](https://wiki.python.org/moin/WindowsCompilers), this is typically only a problem when compiling a Python 2.7 C extension with a newer compiler, e.g. to support a modern C++ standard (see [the section on modern C++ standards for Python 2.7](cpp_standards.md#windows-and-python-27) for more details).
To work around this, use a different environment variable such as `REPAIR_LIBRARY_PATH` to store the library path, and set `DYLD_LIBRARY_PATH` in [`CIBW_REPAIR_WHEEL_COMMAND_MACOS`](https://cibuildwheel.pypa.io/en/stable/options/#repair-wheel-command), like this: Additionally, Visual Studio 2019 started linking to an even newer DLL, `VCRUNTIME140_1.dll`, besides the `VCRUNTIME140.dll` that is included with recent Python versions (starting from Python 3.5; see [here](https://wiki.python.org/moin/WindowsCompilers) for more details on the corresponding Visual Studio & MSVC versions used to compile the different Python versions). To avoid this extra dependency on `VCRUNTIME140_1.dll`, the [`/d2FH4-` flag](https://devblogs.microsoft.com/cppblog/making-cpp-exception-handling-smaller-x64/) can be added to the MSVC invocations (check out [this issue](https://github.com/pypa/cibuildwheel/issues/423) for details and references).
!!! tab examples "Environment variables"
```yaml
CIBW_REPAIR_WHEEL_COMMAND_MACOS: >
DYLD_LIBRARY_PATH=$REPAIR_LIBRARY_PATH delocate-wheel --require-archs {delocate_archs} -w {dest_dir} -v {wheel}
```
!!! tab examples "pyproject.toml"
```toml
[tool.cibuildwheel.macos]
repair-wheel-command = """\
DYLD_LIBRARY_PATH=$REPAIR_LIBRARY_PATH delocate-wheel \
--require-archs {delocate_archs} -w {dest_dir} -v {wheel}\
"""
```
See [#816](https://github.com/pypa/cibuildwheel/issues/816), thanks to @phoerious for reporting.
### macOS: Building CPython 3.8 wheels on arm64
If you're building on an arm64 runner, you might notice something strange about CPython 3.8 - unlike Python 3.9+, it's cross-compiled to arm64 from an x86_64 version of Python running under Rosetta emulation. This is because (despite the prevalence of arm64 versions of Python 3.8 from Apple and Homebrew) there is no officially supported Python.org installer of Python 3.8 for arm64.
This is fine for simple C extensions, but for more complicated builds on arm64 it becomes an issue.
So, if you want to build macOS arm64 wheels on an arm64 runner (e.g., `macos-14`) on Python 3.8, before invoking cibuildwheel, you should install a native arm64 Python 3.8 interpreter on the runner:
!!! tab "GitHub Actions"
```yaml
- uses: actions/setup-python@v5
with:
python-version: 3.8
if: runner.os == 'macOS' && runner.arch == 'ARM64'
```
!!! tab "Generic"
```bash
curl -o /tmp/Python38.pkg https://www.python.org/ftp/python/3.8.10/python-3.8.10-macos11.pkg
sudo installer -pkg /tmp/Python38.pkg -target /
sh "/Applications/Python 3.8/Install Certificates.command"
```
Then cibuildwheel will detect that it's installed and use it instead. However, you probably don't want to build x86_64 wheels on this Python, unless you're happy with them only supporting macOS 11+.
### macOS: Library dependencies do not satisfy target MacOS
Since delocate 0.11.0 there is added verification that the library binary dependencies match the target macOS version. This is to prevent the situation where a wheel platform tag is lower than the actual minimum macOS version required by the library. To resolve this error you need to build the library to the same macOS version as the target wheel (for example using `MACOSX_DEPLOYMENT_TARGET` environment variable).
Alternatively, you could set `MACOSX_DEPLOYMENT_TARGET` in `CIBW_ENVIRONMENT` to correctly label the wheel as incompatible with older macOS versions.
This error may happen when you install a library using a package manager like Homebrew, which compiles the library for the macOS version of the build machine. This is not suitable for wheels, as the library will only work on the same macOS version as the build machine. You should compile the library yourself, or use a precompiled binary that matches the target macOS version.
### Windows: 'ImportError: DLL load failed: The specific module could not be found'
Visual Studio and MSVC link the compiled binary wheels to the Microsoft Visual C++ Runtime. Normally, the C parts of the runtime are included with Python, but the C++ components are not. When compiling modules using C++, it is possible users will run into problems on systems that do not have the full set of runtime libraries installed. The solution is to ask users to download the corresponding Visual C++ Redistributable from the [Microsoft website](https://support.microsoft.com/en-us/help/2977003/the-latest-supported-visual-c-downloads) and install it.
Additionally, Visual Studio 2019 started linking to an even newer DLL, `VCRUNTIME140_1.dll`, besides the `VCRUNTIME140.dll` that is included with recent Python versions (starting from Python 3.5; see [here](https://wiki.python.org/moin/WindowsCompilers) for more details on the corresponding Visual Studio & MSVC versions used to compile the different Python versions). To avoid this extra dependency on `VCRUNTIME140_1.dll`, the [`/d2FH4-` flag](https://devblogs.microsoft.com/cppblog/making-cpp-exception-handling-smaller-x64/) can be added to the MSVC invocations (check out [this issue](https://github.com/pypa/cibuildwheel/issues/423) for details and references). CPython 3.8.3 and all versions after it have this extra DLL, so it is only needed for 3.8 and earlier.
To add the `/d2FH4-` flag to a standard `setup.py` using `setuptools`, the `extra_compile_args` option can be used: To add the `/d2FH4-` flag to a standard `setup.py` using `setuptools`, the `extra_compile_args` option can be used:
@@ -410,12 +222,57 @@ To add the `/d2FH4-` flag to a standard `setup.py` using `setuptools`, the `extr
], ],
``` ```
To investigate the dependencies of a C extension (i.e., the `.pyd` file, a DLL in disguise) on Windows, [Dependency Walker](http://www.dependencywalker.com/) is a great tool. For diagnosing a failing import, the [dlltracer](https://pypi.org/project/dlltracer/) tool may also provide additional details. To investigate the dependencies of a C extension (i.e., the `.pyd` file, a DLL in disguise) on Windows, [Dependency Walker](http://www.dependencywalker.com/) is a great tool.
### Windows ARM64 builds {: #windows-arm64}
`cibuildwheel` supports cross-compiling `ARM64` wheels on all Windows runners, but a native ARM64 runner is required for testing. On non-native runners, tests for ARM64 wheels will be automatically skipped with a warning. Add `"*-win_arm64"` to your `CIBW_TEST_SKIP` setting to suppress the warning. ### Automatic updates {: #automatic-updates}
Cross-compilation on Windows relies on a supported build backend. Supported backends use an environment variable to specify their target platform (the one they are compiling native modules for, as opposed to the one they are running on), which is set in [cibuildwheels/windows.py](https://github.com/pypa/cibuildwheel/blob/main/cibuildwheel/windows.py) before building. Currently, `setuptools>=65.4.1` and `setuptools_rust` are the only supported backends. Selecting a moving target (like the latest release) is generally a bad idea in CI. If something breaks, you can't tell whether it was your code or an upstream update that caused the breakage, and in a worse-case scenario, it could occur during a release.
There are two suggested methods for keeping cibuildwheel up to date that instead involve scheduled pull requests using GitHub's dependabot.
By default, `ARM64` is not enabled when running on non-ARM64 runners. Use [`CIBW_ARCHS`](options.md#archs) to select it. #### Option 1: GitHub Action
If you use GitHub Actions for builds, you can use cibuildwheel as an action:
```yaml
uses: pypa/cibuildwheel@v1.12.0
```
This is a composite step that just runs cibuildwheel using pipx. You can set command-line options as `with:` parameters, and use `env:` as normal.
Then, your `dependabot.yml` file could look like this:
```yaml
version: 2
updates:
- package-ecosystem: "github-actions"
directory: "/"
schedule:
interval: "weekly"
ignore:
# Optional: Official actions have moving tags like v1;
# if you use those, you don't need updates.
- dependency-name: "actions/*"
```
#### Option 2: Requirement files
The second option, and the only one that supports other CI systems, is using a `requirements-*.txt` file. The file should have a distinct name and have only one entry:
```bash
# requirements-cibw.txt
cibuildwheel==1.12.0
```
Then your install step would have `python -m pip install -r requirements-cibw.txt` in it. Your `dependabot.yml` file could look like this:
```yaml
version: 2
updates:
- package-ecosystem: "pip"
directory: "/"
schedule:
interval: "daily"
```
This will also try to update other pins in all requirement files, so be sure you want to do that. The only control you have over the files used is via the directory option.
-13
View File
@@ -2,8 +2,6 @@
title: Home title: Home
--- ---
# cibuildwheel
{% {%
include-markdown "../README.md" include-markdown "../README.md"
start="<!--intro-start-->" start="<!--intro-start-->"
@@ -11,14 +9,3 @@ title: Home
%} %}
To get started, head over to the [setup guide](setup.md). To get started, head over to the [setup guide](setup.md).
How it works
------------
This diagram summarises the steps that cibuildwheel takes on each platform to build your package's wheels.
{%
include "diagram.html"
%}
This isn't exhaustive, for a full list of the things cibuildwheel can do, check the [options](options.md) page.
-18
View File
@@ -1,18 +0,0 @@
from __future__ import annotations
import os
import subprocess
import sysconfig
from typing import Any
def define_env(env: Any) -> None:
"Hook function for mkdocs-macros"
@env.macro
def subprocess_run(*args: str) -> str:
"Run a subprocess and return the stdout"
env = os.environ.copy()
scripts = sysconfig.get_path("scripts")
env["PATH"] = f"{scripts}{os.pathsep}{env.get('PATH', '')}"
return subprocess.run(args, check=True, capture_output=True, text=True, env=env).stdout
+222 -1173
View File
File diff suppressed because it is too large Load Diff

Some files were not shown because too many files have changed in this diff Show More