From 837ea5f59cdd2840431c1274b4ad9379c27a8543 Mon Sep 17 00:00:00 2001 From: Henry Schreiner Date: Thu, 4 Feb 2021 22:03:46 -0500 Subject: [PATCH 01/14] docs: tips and tricks for macOS AS --- docs/faq.md | 45 ++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 44 insertions(+), 1 deletion(-) diff --git a/docs/faq.md b/docs/faq.md index 95a99be9..74b44743 100644 --- a/docs/faq.md +++ b/docs/faq.md @@ -12,7 +12,7 @@ If your wheel didn't compile, check the list below for some debugging tips. - Windows: missing C feature. The Windows C compiler doesn't support C language features invented after 1990, so you'll have to backport your C code to C90. For me, this mostly involved putting my variable declarations at the top of the function like an animal. -- MacOS: calling cibuildwheel from a python3 script and getting a `ModuleNotFoundError`? Due to a [bug](https://bugs.python.org/issue22490) in CPython, you'll need to [unset the `__PYVENV_LAUNCHER__` variable](https://github.com/joerick/cibuildwheel/issues/133#issuecomment-478288597) before activating a venv. +- MacOS: calling cibuildwheel from a python3 script and getting a `ModuleNotFoundError`? Due to a (fixed) [bug](https://bugs.python.org/issue22490) in CPython, you'll need to [unset the `__PYVENV_LAUNCHER__` variable](https://github.com/joerick/cibuildwheel/issues/133#issuecomment-478288597) before activating a venv. ### Linux builds on Docker @@ -24,6 +24,49 @@ Linux wheels are built in the [`manylinux` docker images](https://github.com/pyp - Alternative dockers images can be specified with the `CIBW_MANYLINUX_X86_64_IMAGE`, `CIBW_MANYLINUX_I686_IMAGE`, and `CIBW_MANYLINUX_PYPY_X86_64_IMAGE` options to allow for a custom, preconfigured build environment for the Linux builds. See [options](options.md#manylinux-image) for more details. +### Building macOS wheels for Apple Silicon {: #as-wheels} + +With the introduction of Apple Silicon, you now have several choices for wheels +for Python 3.9+: + +`x86_64` +: The traditional wheel for Apple, loads on Intel machines, and on + Apple Silicon if you use Intel emulation for your stack (not ideal!). Due + to the change in naming, you have to have a tool based on Packaging 20.5+, + such as Pip 20.3+, to load a binary wheel on macOS Big Sur, even the Intel + version. + +`arm64` +: This is the native wheel for macOS on Apple Silicon. Requires Packaging + 20.5+ and therefore pip 20.3+ to load. + +`universal2` +: This wheel holds both architectures in it, causing it to be up to twice the + size (data files do not get doubled, only compiled code). It requires + Packaging 20.6+ to load on Intel (Pip 20.3), and Packaging 20.9+ to load on + Apple Silicon (Pip 21.0.1). + + +The general consensus is that most packages should provide a `x86_64` wheel +and a `universal2` wheel for now. Once Pip 20.3+ is common on macOS, then +`x86_64` wheels would no longer need to be shipped. The key benefit to a +universal wheel is that a user can bundle wheels into an application and ship a +single binary. If you have a large application, then you might prefer to ship +the two native wheels instead. Pip always chooses the most specific wheel +available. In rare cases, you might want to ship all three, but in that case +pip will never download the universal wheels. + +In cibuildwheel, you need to ask for `universal2` or `arm64` explicitly: + +```yaml +CIBW_ARCHS_MACOS: x86_64 universal2 +# Or +CIBW_ARCHS_MACOS: x86_64 arm64 +``` + +Keep in mind, on Intel runners, only the Intel part of a universal2 wheel is +tested, and an `arm64` wheel is not tested at all. + ### Building non-native architectures using emulation {: #emulation} cibuildwheel supports building non-native architectures on Linux, via From 49738c0ceefe1c4848d9ec64f8c44244ca457c6a Mon Sep 17 00:00:00 2001 From: Henry Schreiner Date: Thu, 4 Feb 2021 22:08:18 -0500 Subject: [PATCH 02/14] docs: Github -> GitHub, and ensure it doesn't happen again --- .pre-commit-config.yaml | 11 +++++++++++ README.md | 12 ++++++------ docs/contributing.md | 2 +- docs/data/projects.yml | 4 ++-- docs/faq.md | 6 +++--- docs/setup.md | 4 ++-- 6 files changed, 25 insertions(+), 14 deletions(-) diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index 59ddddee..3451efc6 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -47,3 +47,14 @@ repos: - id: flake8 exclude: cibuildwheel/resources/ additional_dependencies: [flake8-bugbear] + + +# The original pybind11 checks for a few C++ style items +- repo: local + hooks: + - id: disallow-caps + name: Disallow improper capitalization + language: pygrep + entry: PyBind|Numpy|Cmake|Github + types: + - markdown diff --git a/README.md b/README.md index 6b46c70e..c9add123 100644 --- a/README.md +++ b/README.md @@ -59,7 +59,7 @@ Usage Example setup ------------- -To build manylinux, macOS, and Windows wheels on Github Actions, you could use this `.github/workflows/wheels.yml`: +To build manylinux, macOS, and Windows wheels on GitHub Actions, you could use this `.github/workflows/wheels.yml`: ```yaml name: Build @@ -337,7 +337,7 @@ _1 January 2021_ - 🛠 Added a patch for Pypy to ensure header files are available for building in a virtualenv. (#502) -- 🛠 Some preparatory work towards using cibuildwheel as a Github Action. +- 🛠 Some preparatory work towards using cibuildwheel as a GitHub Action. Check out [the FAQ](https://cibuildwheel.readthedocs.io/en/stable/faq/#option-1-github-action) for information on how to use it. We'll be fully updating the docs to this @@ -349,7 +349,7 @@ _21 December 2020_ - 🛠 Update dependencies, notably wheel==0.36.2 and pip==20.3.3, and CPython to their latest bugfix releases (#489) -- 📚 Switch to a Github example in the README (#479) +- 📚 Switch to a GitHub example in the README (#479) - 📚 Create Working Examples table, with many projects that use cibuildwheel (#474) - 📚 Import Working Examples table and Changelog to docs @@ -454,7 +454,7 @@ _8 July 2020_ _25 June 2020_ - 🐛 Fix "OSError: [WinError 17] The system cannot move the file to a different - disk drive" on Github Actions (#388, #389) + disk drive" on GitHub Actions (#388, #389) ### 1.5.0 @@ -468,7 +468,7 @@ _24 June 2020_ (#360, #358) - ✨ Added workaround to allow Python 3.5 on Windows to pull dependencies from pyproject.toml. (#358) -- 📚 Improved Github Actions examples and docs (#354, #362) +- 📚 Improved GitHub Actions examples and docs (#354, #362) - 🐛 Ensure pip wheel uses the specified package, and doesn't build a wheel from PyPI (#369) - 🛠 Internal changes: using pathlib.Path, precommit hooks, testing @@ -526,7 +526,7 @@ _2 May 2020_ _12 March 2020_ -- 🌟 Add support for building on Github Actions! Check out the +- 🌟 Add support for building on GitHub Actions! Check out the [docs](https://cibuildwheel.readthedocs.io/en/stable/setup/#github-actions) for information on how to set it up. (#194) - ✨ Add the `CIBW_BEFORE_TEST` option, which lets you run a command to diff --git a/docs/contributing.md b/docs/contributing.md index 329a355c..1307c8e5 100644 --- a/docs/contributing.md +++ b/docs/contributing.md @@ -111,7 +111,7 @@ bin/bump_version.py You'll be prompted to enter the new version number. Update the changelog when prompted. The script will create a 'bump version' commit and version tag. -Finally, cut the release and upload to PyPI/Github. +Finally, cut the release and upload to PyPI/GitHub. ```bash rm -rf dist diff --git a/docs/data/projects.yml b/docs/data/projects.yml index 37a3114d..b032fb67 100644 --- a/docs/data/projects.yml +++ b/docs/data/projects.yml @@ -1,7 +1,7 @@ # Fields: # name: required, printed name -# gh: Github repo (eventually a url: could be added for non-github projects) -# stars: Github repo (optional, if different from package, such as for Twisted) +# gh: GitHub repo (eventually a url: could be added for non-github projects) +# stars: GitHub repo (optional, if different from package, such as for Twisted) # os: Operating system list, [windows, apple, linux] (optional) # ci: [appveyor, github, azurepipelines, circleci, gitlab, travisci] (optional) # notes: (text, optional) diff --git a/docs/faq.md b/docs/faq.md index 74b44743..3ab9f8a4 100644 --- a/docs/faq.md +++ b/docs/faq.md @@ -71,13 +71,13 @@ tested, and an `arm64` wheel is not tested at all. cibuildwheel supports building non-native architectures on Linux, via emulation through the binfmt_misc kernel feature. The easiest way to use this -is via the [docker/setup-qemu-action][setup-qemu-action] on Github Actions or +is via the [docker/setup-qemu-action][setup-qemu-action] on GitHub Actions or [tonistiigi/binfmt][binfmt]. [setup-qemu-action]: https://github.com/docker/setup-qemu-action [binfmt]: https://hub.docker.com/r/tonistiigi/binfmt -Check out the following config for an example of how to set it up on Github +Check out the following config for an example of how to set it up on GitHub Actions. Once QEMU is set up and registered, you just need to set the `CIBW_ARCHS_LINUX` environment variable (or use the `--archs` option on Linux), and the other architectures are emulated automatically. @@ -105,7 +105,7 @@ Hopefully, this is a temporary situation. Once we have widely available Apple Si So far, only CPython 3.9 supports `universal2` and `arm64` wheels. -Here's an example Github Actions workflow with a job that builds for Apple Silicon: +Here's an example GitHub Actions workflow with a job that builds for Apple Silicon: > .github/workflows/build_macos.yml ```yml diff --git a/docs/setup.md b/docs/setup.md index b07a0744..934d782b 100644 --- a/docs/setup.md +++ b/docs/setup.md @@ -4,14 +4,14 @@ title: 'Setup' # GitHub Actions [linux/mac/windows] {: #github-actions} -To build Linux, Mac, and Windows wheels using Github Actions, create a `.github/workflows/build.yml` file in your repo. +To build Linux, Mac, and Windows wheels using GitHub Actions, create a `.github/workflows/build.yml` file in your repo. > build.yml ```yaml {% include "../examples/github-minimal.yml" %} ``` -Commit this file, and push to Github - either to your default branch, or to a PR branch. The build should start automatically. +Commit this file, and push to GitHub - either to your default branch, or to a PR branch. The build should start automatically. For more info on this file, check out the [docs](https://help.github.com/en/actions/reference/workflow-syntax-for-github-actions). From d871a15e3c34b909e994291b42fbd174218b5638 Mon Sep 17 00:00:00 2001 From: Henry Schreiner Date: Thu, 4 Feb 2021 22:21:14 -0500 Subject: [PATCH 03/14] examples: update GitHub examples to be a bit cleaner --- examples/github-deploy.yml | 13 +++++-------- examples/github-minimal.yml | 11 ++++------- examples/github-with-qemu.yml | 9 +++------ 3 files changed, 12 insertions(+), 21 deletions(-) diff --git a/examples/github-deploy.yml b/examples/github-deploy.yml index b83c4bbd..aed86393 100644 --- a/examples/github-deploy.yml +++ b/examples/github-deploy.yml @@ -24,20 +24,17 @@ jobs: - uses: actions/setup-python@v2 name: Install Python with: - python-version: '3.7' + python-version: '3.8' - name: Install cibuildwheel - run: | - python -m pip install cibuildwheel==1.8.0 + run: python -m pip install cibuildwheel==1.8.0 - name: Install Visual C++ for Python 2.7 if: runner.os == 'Windows' - run: | - choco install vcpython27 -f -y + run: choco install vcpython27 -f -y - name: Build wheels - run: | - python -m cibuildwheel --output-dir wheelhouse + run: python -m cibuildwheel --output-dir wheelhouse - uses: actions/upload-artifact@v2 with: @@ -52,7 +49,7 @@ jobs: - uses: actions/setup-python@v2 name: Install Python with: - python-version: '3.7' + python-version: '3.8' - name: Build sdist run: python setup.py sdist diff --git a/examples/github-minimal.yml b/examples/github-minimal.yml index 03ce1724..af82797f 100644 --- a/examples/github-minimal.yml +++ b/examples/github-minimal.yml @@ -16,20 +16,17 @@ jobs: - uses: actions/setup-python@v2 name: Install Python with: - python-version: '3.7' + python-version: '3.8' - name: Install cibuildwheel - run: | - python -m pip install cibuildwheel==1.8.0 + run: python -m pip install cibuildwheel==1.8.0 - name: Install Visual C++ for Python 2.7 if: runner.os == 'Windows' - run: | - choco install vcpython27 -f -y + run: choco install vcpython27 -f -y - name: Build wheels - run: | - python -m cibuildwheel --output-dir wheelhouse + run: python -m cibuildwheel --output-dir wheelhouse - uses: actions/upload-artifact@v2 with: diff --git a/examples/github-with-qemu.yml b/examples/github-with-qemu.yml index 2b3f55af..cccc363f 100644 --- a/examples/github-with-qemu.yml +++ b/examples/github-with-qemu.yml @@ -19,13 +19,11 @@ jobs: python-version: '3.7' - name: Install cibuildwheel - run: | - python -m pip install cibuildwheel==1.8.0 + run: python -m pip install cibuildwheel==1.8.0 - name: Install Visual C++ for Python 2.7 if: runner.os == 'Windows' - run: | - choco install vcpython27 -f -y + run: choco install vcpython27 -f -y - name: Set up QEMU if: runner.os == 'Linux' @@ -34,8 +32,7 @@ jobs: platforms: all - name: Build wheels - run: | - python -m cibuildwheel --output-dir wheelhouse + run: python -m cibuildwheel --output-dir wheelhouse env: # configure cibuildwheel to build native archs ('auto'), and some # emulated ones From ca9d8d5d2c0472170871dd099e3a166801e35bac Mon Sep 17 00:00:00 2001 From: Henry Schreiner Date: Thu, 4 Feb 2021 22:21:38 -0500 Subject: [PATCH 04/14] examples: bump versions in a few places --- examples/travis-ci-deploy.yml | 2 +- examples/travis-ci-minimal.yml | 2 +- examples/travis-ci-test-and-deploy.yml | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/examples/travis-ci-deploy.yml b/examples/travis-ci-deploy.yml index 6d624ab8..a6e6b15b 100644 --- a/examples/travis-ci-deploy.yml +++ b/examples/travis-ci-deploy.yml @@ -14,7 +14,7 @@ jobs: - os: windows language: shell before_install: - - choco install python --version 3.8.0 + - choco install python --version 3.8.6 - export PATH="/c/Python38:/c/Python38/Scripts:$PATH" # make sure it's on PATH as 'python3' - ln -s /c/Python38/python.exe /c/Python38/python3.exe diff --git a/examples/travis-ci-minimal.yml b/examples/travis-ci-minimal.yml index c7cfb91e..0d9b7e0b 100644 --- a/examples/travis-ci-minimal.yml +++ b/examples/travis-ci-minimal.yml @@ -20,7 +20,7 @@ jobs: - os: windows language: shell before_install: - - choco install python --version 3.8.0 + - choco install python --version 3.8.6 - export PATH="/c/Python38:/c/Python38/Scripts:$PATH" # make sure it's on PATH as 'python3' - ln -s /c/Python38/python.exe /c/Python38/python3.exe diff --git a/examples/travis-ci-test-and-deploy.yml b/examples/travis-ci-test-and-deploy.yml index 31627c1d..83fed3e9 100644 --- a/examples/travis-ci-test-and-deploy.yml +++ b/examples/travis-ci-test-and-deploy.yml @@ -14,7 +14,7 @@ python: before_install: - | if [[ "$TRAVIS_OS_NAME" = windows ]]; then - choco install python --version 3.8.0 + choco install python --version 3.8.6 export PATH="/c/Python38:/c/Python38/Scripts:$PATH" # make sure it's on PATH as 'python3' ln -s /c/Python38/python.exe /c/Python38/python3.exe From 0c30df5c96b9d8147be8b05678bdb31d518da05f Mon Sep 17 00:00:00 2001 From: Henry Schreiner Date: Thu, 4 Feb 2021 22:21:57 -0500 Subject: [PATCH 05/14] docs: update msg about running on a development machine --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index c9add123..e38f07f0 100644 --- a/README.md +++ b/README.md @@ -52,7 +52,7 @@ Usage ¹ Requires a "third-party build host"; expected to work with cibuildwheel but not directly tested by our CI.
-`cibuildwheel` is not intended to run on your development machine. Because it uses system Python from Python.org it will try to install packages globally - not what you expect from a build tool! Instead, isolated CI services like those mentioned above are ideal. +`cibuildwheel` is not intended to run on your development machine. Because it uses system Python from Python.org on macOS and Windows, it will try to install packages globally - not what you expect from a build tool! Instead, isolated CI services like those mentioned above are ideal. For Linux builds, it uses manylinux docker images, so those can be done locally for testing in a pinch. From 8fe6520ef404d3865c0273b2df880e5ca088abb7 Mon Sep 17 00:00:00 2001 From: Henry Schreiner Date: Thu, 4 Feb 2021 22:27:18 -0500 Subject: [PATCH 06/14] examples: move github-minimal to action --- examples/github-minimal.yml | 11 +---------- 1 file changed, 1 insertion(+), 10 deletions(-) diff --git a/examples/github-minimal.yml b/examples/github-minimal.yml index af82797f..db5cf436 100644 --- a/examples/github-minimal.yml +++ b/examples/github-minimal.yml @@ -13,20 +13,11 @@ jobs: steps: - uses: actions/checkout@v2 - - uses: actions/setup-python@v2 - name: Install Python - with: - python-version: '3.8' - - - name: Install cibuildwheel - run: python -m pip install cibuildwheel==1.8.0 - - name: Install Visual C++ for Python 2.7 if: runner.os == 'Windows' run: choco install vcpython27 -f -y - - name: Build wheels - run: python -m cibuildwheel --output-dir wheelhouse + - uses: joerick/cibuildwheel@v1.8.0 - uses: actions/upload-artifact@v2 with: From 3163743cba265fd2cdda9ba8f4c6dc8c1526c0c0 Mon Sep 17 00:00:00 2001 From: Henry Schreiner Date: Thu, 4 Feb 2021 22:35:42 -0500 Subject: [PATCH 07/14] docs: some README touchup --- README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index e38f07f0..98f6481a 100644 --- a/README.md +++ b/README.md @@ -31,7 +31,7 @@ What does it do? ² Only supported on Travis
³ Beta support until Travis CI fixes a bug
-- Builds manylinux, macOS, and Windows wheels for CPython and PyPy +- Builds manylinux, macOS 10.9+, and Windows wheels for CPython and PyPy - Works on GitHub Actions, Azure Pipelines, Travis CI, AppVeyor, CircleCI, and GitLab CI - Bundles shared library dependencies on Linux and macOS through [auditwheel](https://github.com/pypa/auditwheel) and [delocate](https://github.com/matthew-brett/delocate) - Runs the library test suite against the wheel-installed version of your library @@ -77,8 +77,8 @@ jobs: steps: - uses: actions/checkout@v2 + # Used to host cibuildwheel - uses: actions/setup-python@v2 - name: Install Python - name: Install cibuildwheel run: python -m pip install cibuildwheel==1.8.0 @@ -93,7 +93,7 @@ jobs: path: ./wheelhouse/*.whl ``` -For more information, including building on Python 2, PyPI deployment, and the use of other CI services, check out the [documentation](https://cibuildwheel.readthedocs.org) and the [examples](https://github.com/joerick/cibuildwheel/tree/master/examples). +For more information, including building on Python 2, PyPI deployment, and the use of other CI services or the dedicated GitHub Action, check out the [documentation](https://cibuildwheel.readthedocs.org) and the [examples](https://github.com/joerick/cibuildwheel/tree/master/examples). Options ------- From 160d1db6be213780833c3eb43d72b19413649e7c Mon Sep 17 00:00:00 2001 From: Henry Schreiner Date: Fri, 5 Feb 2021 08:48:35 -0500 Subject: [PATCH 08/14] docs: add GHA badge, normalize CircleCI badge --- README.md | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 98f6481a..9b1a975f 100644 --- a/README.md +++ b/README.md @@ -1,7 +1,14 @@ cibuildwheel ============ -[![PyPI](https://img.shields.io/pypi/v/cibuildwheel.svg)](https://pypi.python.org/pypi/cibuildwheel) [![Documentation Status](https://readthedocs.org/projects/cibuildwheel/badge/?version=stable)](https://cibuildwheel.readthedocs.io/en/stable/?badge=stable) [![Build Status](https://travis-ci.org/joerick/cibuildwheel.svg?branch=master)](https://travis-ci.org/joerick/cibuildwheel) [![Build status](https://ci.appveyor.com/api/projects/status/wbsgxshp05tt1tif/branch/master?svg=true)](https://ci.appveyor.com/project/joerick/cibuildwheel/branch/master) [![CircleCI](https://circleci.com/gh/joerick/cibuildwheel.svg?style=svg)](https://circleci.com/gh/joerick/cibuildwheel) [![Build Status](https://dev.azure.com/joerick0429/cibuildwheel/_apis/build/status/joerick.cibuildwheel?branchName=master)](https://dev.azure.com/joerick0429/cibuildwheel/_build/latest?definitionId=2&branchName=master) +[![PyPI](https://img.shields.io/pypi/v/cibuildwheel.svg)](https://pypi.python.org/pypi/cibuildwheel) +[![Documentation Status](https://readthedocs.org/projects/cibuildwheel/badge/?version=stable)](https://cibuildwheel.readthedocs.io/en/stable/?badge=stable) +[![Actions Status](https://github.com/joerick/cibuildwheel/workflows/Test/badge.svg)](https://github.com/joerick/cibuildwheel/actions) +[![Travis Status](https://img.shields.io/travis/joerick/cibuildwheel/master?logo=travis)](https://travis-ci.org/joerick/cibuildwheel) +[![AppVeyor status](https://ci.appveyor.com/api/projects/status/wbsgxshp05tt1tif/branch/master?svg=true)](https://ci.appveyor.com/project/joerick/cibuildwheel/branch/master) +[![CircleCI Status](https://img.shields.io/circleci/build/gh/joerick/cibuildwheel/master?logo=circleci)](https://circleci.com/gh/joerick/cibuildwheel) +[![Azure Status](https://dev.azure.com/joerick0429/cibuildwheel/_apis/build/status/joerick.cibuildwheel?branchName=master)](https://dev.azure.com/joerick0429/cibuildwheel/_build/latest?definitionId=2&branchName=master) + [Documentation](https://cibuildwheel.readthedocs.org) From adc6af9e7cf595361c62848fc24fd981d6c1a88f Mon Sep 17 00:00:00 2001 From: Joe Rickerby Date: Fri, 5 Feb 2021 19:18:24 +0000 Subject: [PATCH 09/14] Combine the two Apple Silicon FAQ entries into one and make a few tweaks --- docs/faq.md | 88 +++++++++++++++++++------------ examples/github-apple-silicon.yml | 2 +- 2 files changed, 56 insertions(+), 34 deletions(-) diff --git a/docs/faq.md b/docs/faq.md index 3ab9f8a4..ddec5893 100644 --- a/docs/faq.md +++ b/docs/faq.md @@ -24,49 +24,71 @@ Linux wheels are built in the [`manylinux` docker images](https://github.com/pyp - Alternative dockers images can be specified with the `CIBW_MANYLINUX_X86_64_IMAGE`, `CIBW_MANYLINUX_I686_IMAGE`, and `CIBW_MANYLINUX_PYPY_X86_64_IMAGE` options to allow for a custom, preconfigured build environment for the Linux builds. See [options](options.md#manylinux-image) for more details. -### Building macOS wheels for Apple Silicon {: #as-wheels} +### Building macOS wheels for Apple Silicon {: #apple-silicon} -With the introduction of Apple Silicon, you now have several choices for wheels -for Python 3.9+: +`cibuildwheel` supports cross-compiling `universal2` and `arm64` wheels on `x86_64` runners. With the introduction of Apple Silicon, you now have several choices for wheels for Python 3.9+: -`x86_64` -: The traditional wheel for Apple, loads on Intel machines, and on - Apple Silicon if you use Intel emulation for your stack (not ideal!). Due - to the change in naming, you have to have a tool based on Packaging 20.5+, - such as Pip 20.3+, to load a binary wheel on macOS Big Sur, even the Intel - version. +#### `x86_64` -`arm64` -: This is the native wheel for macOS on Apple Silicon. Requires Packaging - 20.5+ and therefore pip 20.3+ to load. +The traditional wheel for Apple, loads on Intel machines, and on +Apple Silicon when running Python under Rosetta 2 emulation. -`universal2` -: This wheel holds both architectures in it, causing it to be up to twice the - size (data files do not get doubled, only compiled code). It requires - Packaging 20.6+ to load on Intel (Pip 20.3), and Packaging 20.9+ to load on - Apple Silicon (Pip 21.0.1). +Due to a change in naming, Pip 20.3+ (or an installer using packaging 20.5+) +is required to install a binary wheel on macOS Big Sur. +#### `arm64` -The general consensus is that most packages should provide a `x86_64` wheel -and a `universal2` wheel for now. Once Pip 20.3+ is common on macOS, then -`x86_64` wheels would no longer need to be shipped. The key benefit to a -universal wheel is that a user can bundle wheels into an application and ship a -single binary. If you have a large application, then you might prefer to ship -the two native wheels instead. Pip always chooses the most specific wheel -available. In rare cases, you might want to ship all three, but in that case -pip will never download the universal wheels. +The native wheel for macOS on Apple Silicon. -In cibuildwheel, you need to ask for `universal2` or `arm64` explicitly: +Requires Pip 20.3+ (or packaging 20.5+) to install. -```yaml -CIBW_ARCHS_MACOS: x86_64 universal2 -# Or -CIBW_ARCHS_MACOS: x86_64 arm64 +#### `universal2` + +This wheel contains both architectures, causing it to be up to twice the +size (data files do not get doubled, only compiled code). It requires +Pip 20.3 (Packaging 20.6+) to load on Intel, and Pip 21.0.1 (Packaging 20.9+) +to load on Apple Silicon. + +!!! note + The dual-architecture `universal2` has a few benefits, but a key benefit + to a universal wheel is that a user can bundle these wheels into an + application and ship a single binary. + + However, if you have a large library, then you might prefer to ship + the two single-arch wheels instead - `x86_64` and `arm64`. In rare cases, + you might want to build all three, but in that case, pip will not download + the universal wheels, because it prefers the most specific wheel + available. + +Generally speaking, because Pip 20.3 is required for the `universal2` wheel, +most packages should provide both `x86_64` and `universal2` wheels for now. +Once Pip 20.3+ is common on macOS, then it should be possible to ship only the +`universal2` wheel. + +**Apple Silicon wheels are not built by default**, but can be enabled by setting the [`CIBW_ARCHS_MACOS` option](options.md#archs) to `x86_64 arm64 universal2`. Cross-compilation is provided by the Xcode toolchain. + +!!! important + When cross-compiling on Intel, it is not possible to test `arm64` and the `arm64` part of a `universal2` wheel. + + `cibuildwheel` will raise a warning to notify you of this - these warnings be be silenced by skipping testing on these platforms: `CIBW_TEST_SKIP: *_arm64 *_universal2:arm64`. + +Hopefully, cross-compilation is a temporary situation. Once we have widely +available Apple Silicon CI runners, we can build and test `arm64` and +`universal2` wheels natively. That's why `universal2` wheels are not yet built +by default, and require opt-in by setting `CIBW_ARCHS_MACOS`. + +!!! note + Your runner needs Xcode Command Line Tools 12.2 or later to build `universal2` or `arm64`. + + So far, only CPython 3.9 supports `universal2` and `arm64` wheels. + +Here's an example GitHub Actions workflow with a job that builds for Apple Silicon: + +> .github/workflows/build_macos.yml +```yml +{% include "../examples/github-apple-silicon.yml" %} ``` -Keep in mind, on Intel runners, only the Intel part of a universal2 wheel is -tested, and an `arm64` wheel is not tested at all. - ### Building non-native architectures using emulation {: #emulation} cibuildwheel supports building non-native architectures on Linux, via diff --git a/examples/github-apple-silicon.yml b/examples/github-apple-silicon.yml index e1ec7087..905028e5 100644 --- a/examples/github-apple-silicon.yml +++ b/examples/github-apple-silicon.yml @@ -13,7 +13,7 @@ jobs: run: | pipx run cibuildwheel==1.8.0 --output-dir wheelhouse env: - CIBW_ARCHS_MACOS: x86_64 universal2 arm64 + CIBW_ARCHS_MACOS: x86_64 universal2 - uses: actions/upload-artifact@v2 with: From b6eca6ca485dcc3ee7d09c099f9fb13d04dea877 Mon Sep 17 00:00:00 2001 From: Joe Rickerby Date: Fri, 5 Feb 2021 19:19:19 +0000 Subject: [PATCH 10/14] Use the composite-steps action in more examples --- examples/github-deploy.yml | 5 +---- examples/github-minimal.yml | 6 +++++- examples/github-with-qemu.yml | 5 +---- 3 files changed, 7 insertions(+), 9 deletions(-) diff --git a/examples/github-deploy.yml b/examples/github-deploy.yml index aed86393..81fea41d 100644 --- a/examples/github-deploy.yml +++ b/examples/github-deploy.yml @@ -26,15 +26,12 @@ jobs: with: python-version: '3.8' - - name: Install cibuildwheel - run: python -m pip install cibuildwheel==1.8.0 - - name: Install Visual C++ for Python 2.7 if: runner.os == 'Windows' run: choco install vcpython27 -f -y - name: Build wheels - run: python -m cibuildwheel --output-dir wheelhouse + uses: joerick/cibuildwheel@v1.8.0 - uses: actions/upload-artifact@v2 with: diff --git a/examples/github-minimal.yml b/examples/github-minimal.yml index db5cf436..1c054686 100644 --- a/examples/github-minimal.yml +++ b/examples/github-minimal.yml @@ -17,7 +17,11 @@ jobs: if: runner.os == 'Windows' run: choco install vcpython27 -f -y - - uses: joerick/cibuildwheel@v1.8.0 + - name: Build wheels + uses: joerick/cibuildwheel@v1.8.0 + env: + # if you need to supply options, put them here. e.g. + # CIBW_SOME_OPTION: value - uses: actions/upload-artifact@v2 with: diff --git a/examples/github-with-qemu.yml b/examples/github-with-qemu.yml index cccc363f..ad03e490 100644 --- a/examples/github-with-qemu.yml +++ b/examples/github-with-qemu.yml @@ -18,9 +18,6 @@ jobs: with: python-version: '3.7' - - name: Install cibuildwheel - run: python -m pip install cibuildwheel==1.8.0 - - name: Install Visual C++ for Python 2.7 if: runner.os == 'Windows' run: choco install vcpython27 -f -y @@ -32,7 +29,7 @@ jobs: platforms: all - name: Build wheels - run: python -m cibuildwheel --output-dir wheelhouse + uses: joerick/cibuildwheel@v1.8.0 env: # configure cibuildwheel to build native archs ('auto'), and some # emulated ones From 566754fdbafb66e7b7e94576dd38fedb4002b71e Mon Sep 17 00:00:00 2001 From: Joe Rickerby Date: Fri, 5 Feb 2021 19:20:11 +0000 Subject: [PATCH 11/14] README: add Apple Silicon and emulation references + a few minor tweaks --- README.md | 40 ++++++++++++++++++---------------------- 1 file changed, 18 insertions(+), 22 deletions(-) diff --git a/README.md b/README.md index 9b1a975f..09104162 100644 --- a/README.md +++ b/README.md @@ -22,26 +22,24 @@ Python wheels are great. Building them across **Mac, Linux, Windows**, on **mult What does it do? ---------------- -| | macOS x86_64 | Windows 64bit | Windows 32bit | manylinux x86_64 | manylinux i686 | manylinux aarch64 | manylinux ppc64le | manylinux s390x | -|---|---|---|---|---|---|---|---|---| -| CPython 2.7 | ✅ | ✅¹ | ✅¹ | ✅ | ✅ | | | | -| CPython 3.5 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅² | ✅² | ✅³ | -| CPython 3.6 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅² | ✅² | ✅³ | -| CPython 3.7 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅² | ✅² | ✅³ | -| CPython 3.8 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅² | ✅² | ✅³ | -| CPython 3.9 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅² | ✅² | ✅³ | -| PyPy 2.7 v7.3.3 | ✅ | | ✅ | ✅ | | | | | -| PyPy 3.6 v7.3.3 | ✅ | | ✅ | ✅ | | | | | -| PyPy 3.7 (beta) v7.3.3 | ✅ | | ✅ | ✅ | | | | | +| | macOS Intel | macOS Apple Silicon | Windows 64bit | Windows 32bit | manylinux x86_64 | manylinux i686 | manylinux aarch64 | manylinux ppc64le | manylinux s390x | +|---|---|---|---|---|---|---|---|---|---| +| CPython 2.7 | ✅ | | ✅¹ | ✅¹ | ✅ | ✅ | | | | +| CPython 3.5 | ✅ | | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | +| CPython 3.6 | ✅ | | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | +| CPython 3.7 | ✅ | | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | +| CPython 3.8 | ✅ | | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | +| CPython 3.9 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | +| PyPy 2.7 v7.3.3 | ✅ | | | ✅ | ✅ | | | | | +| PyPy 3.6 v7.3.3 | ✅ | | | ✅ | ✅ | | | | | +| PyPy 3.7 (beta) v7.3.3 | ✅ | | | ✅ | ✅ | | | | | ¹ Not supported on Travis
-² Only supported on Travis
-³ Beta support until Travis CI fixes a bug
-- Builds manylinux, macOS 10.9+, and Windows wheels for CPython and PyPy +- Builds manylinux, macOS, and Windows wheels for CPython and PyPy - Works on GitHub Actions, Azure Pipelines, Travis CI, AppVeyor, CircleCI, and GitLab CI - Bundles shared library dependencies on Linux and macOS through [auditwheel](https://github.com/pypa/auditwheel) and [delocate](https://github.com/matthew-brett/delocate) -- Runs the library test suite against the wheel-installed version of your library +- Runs your library's tests against the wheel-installed version of your library Usage ----- @@ -50,14 +48,14 @@ Usage | | Linux | macOS | Windows | Linux ARM | |-----------------|-------|-------|---------|--------------| -| GitHub Actions | ✅ | ✅ | ✅ | ✴️¹ | -| Azure Pipelines | ✅ | ✅ | ✅ | ✴️¹ | +| GitHub Actions | ✅ | ✅ | ✅ | ✅¹ | +| Azure Pipelines | ✅ | ✅ | ✅ | | | Travis CI | ✅ | ✅ | ✅ | ✅ | | AppVeyor | ✅ | ✅ | ✅ | | | CircleCI | ✅ | ✅ | | | | Gitlab CI | ✅ | | | | -¹ Requires a "third-party build host"; expected to work with cibuildwheel but not directly tested by our CI.
+¹ [Requires emulation](https://cibuildwheel.readthedocs.io/en/stable/faq/#emulation), distributed separately. Other services may also support Linux ARM through emulation or third-party build hosts, but these are not tested in our CI.
`cibuildwheel` is not intended to run on your development machine. Because it uses system Python from Python.org on macOS and Windows, it will try to install packages globally - not what you expect from a build tool! Instead, isolated CI services like those mentioned above are ideal. For Linux builds, it uses manylinux docker images, so those can be done locally for testing in a pinch. @@ -84,16 +82,14 @@ jobs: steps: - uses: actions/checkout@v2 - # Used to host cibuildwheel - - uses: actions/setup-python@v2 - - name: Install cibuildwheel run: python -m pip install cibuildwheel==1.8.0 - name: Build wheels run: python -m cibuildwheel --output-dir wheelhouse env: - CIBW_SKIP: "cp27-* pp27-*" # skip Python 2.7 wheels + # if you need to supply options, put them here. e.g. + # CIBW_SOME_OPTION: value - uses: actions/upload-artifact@v2 with: From c7ad0c7654b7fda526b3537ad84b697461c67428 Mon Sep 17 00:00:00 2001 From: Joe Rickerby Date: Fri, 5 Feb 2021 19:43:19 +0000 Subject: [PATCH 12/14] Review tweaks from henryiii --- README.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 09104162..d9fd5e86 100644 --- a/README.md +++ b/README.md @@ -36,7 +36,7 @@ What does it do? ¹ Not supported on Travis
-- Builds manylinux, macOS, and Windows wheels for CPython and PyPy +- Builds manylinux, macOS 10.9+, and Windows wheels for CPython and PyPy - Works on GitHub Actions, Azure Pipelines, Travis CI, AppVeyor, CircleCI, and GitLab CI - Bundles shared library dependencies on Linux and macOS through [auditwheel](https://github.com/pypa/auditwheel) and [delocate](https://github.com/matthew-brett/delocate) - Runs your library's tests against the wheel-installed version of your library @@ -82,6 +82,9 @@ jobs: steps: - uses: actions/checkout@v2 + # Used to host cibuildwheel + - uses: actions/setup-python@v2 + - name: Install cibuildwheel run: python -m pip install cibuildwheel==1.8.0 From 3940d38b4df408fc4448cec54492bb919ccd3209 Mon Sep 17 00:00:00 2001 From: Joe Rickerby Date: Fri, 5 Feb 2021 20:17:53 +0000 Subject: [PATCH 13/14] Fix GHA example syntax --- README.md | 6 +++--- examples/github-minimal.yml | 6 +++--- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/README.md b/README.md index d9fd5e86..4e802498 100644 --- a/README.md +++ b/README.md @@ -90,9 +90,9 @@ jobs: - name: Build wheels run: python -m cibuildwheel --output-dir wheelhouse - env: - # if you need to supply options, put them here. e.g. - # CIBW_SOME_OPTION: value + # to supply options, put them in 'env', like: + # env: + # CIBW_SOME_OPTION: value - uses: actions/upload-artifact@v2 with: diff --git a/examples/github-minimal.yml b/examples/github-minimal.yml index 1c054686..c6884fcd 100644 --- a/examples/github-minimal.yml +++ b/examples/github-minimal.yml @@ -19,9 +19,9 @@ jobs: - name: Build wheels uses: joerick/cibuildwheel@v1.8.0 - env: - # if you need to supply options, put them here. e.g. - # CIBW_SOME_OPTION: value + # to supply options, put them in 'env', like: + # env: + # CIBW_SOME_OPTION: value - uses: actions/upload-artifact@v2 with: From efc97b80ac3ce74ced4f344a13cac9e3e78fefaa Mon Sep 17 00:00:00 2001 From: Joe Rickerby Date: Fri, 5 Feb 2021 20:25:03 +0000 Subject: [PATCH 14/14] Update docs/faq.md --- docs/faq.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/faq.md b/docs/faq.md index ddec5893..d0175775 100644 --- a/docs/faq.md +++ b/docs/faq.md @@ -65,7 +65,7 @@ most packages should provide both `x86_64` and `universal2` wheels for now. Once Pip 20.3+ is common on macOS, then it should be possible to ship only the `universal2` wheel. -**Apple Silicon wheels are not built by default**, but can be enabled by setting the [`CIBW_ARCHS_MACOS` option](options.md#archs) to `x86_64 arm64 universal2`. Cross-compilation is provided by the Xcode toolchain. +**Apple Silicon wheels are not built by default**, but can be enabled by adding extra archs to the [`CIBW_ARCHS_MACOS` option](options.md#archs) - e.g. `x86_64 arm64 universal2`. Cross-compilation is provided by the Xcode toolchain. !!! important When cross-compiling on Intel, it is not possible to test `arm64` and the `arm64` part of a `universal2` wheel.