Hongru/cibuildwheel
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Merge pull request #577 from joerick/docs/u2

Browse Source 
docs: Universal2 discussion and more updates
This commit is contained in:
Joe Rickerby
2021-02-05 21:12:29 +00:00
committed by GitHub
parent 99acc0abac efc97b80ac
commit 3fa7e596af
13 changed files with 136 additions and 74 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.pre-commit-config.yaml
+11
View File
@@ -47,3 +47,14 @@ repos:
- id: flake8
exclude: cibuildwheel/resources/
additional_dependencies: [flake8-bugbear]
# The original pybind11 checks for a few C++ style items
- repo: local
hooks:
- id: disallow-caps
name: Disallow improper capitalization
language: pygrep
entry: PyBind|Numpy|Cmake|Github
types:
- markdown
README.md
+36 -30
View File
@@ -1,7 +1,14 @@
cibuildwheel
============
[![PyPI](https://img.shields.io/pypi/v/cibuildwheel.svg)](https://pypi.python.org/pypi/cibuildwheel) [![Documentation Status](https://readthedocs.org/projects/cibuildwheel/badge/?version=stable)](https://cibuildwheel.readthedocs.io/en/stable/?badge=stable) [![Build Status](https://travis-ci.org/joerick/cibuildwheel.svg?branch=master)](https://travis-ci.org/joerick/cibuildwheel) [![Build status](https://ci.appveyor.com/api/projects/status/wbsgxshp05tt1tif/branch/master?svg=true)](https://ci.appveyor.com/project/joerick/cibuildwheel/branch/master) [![CircleCI](https://circleci.com/gh/joerick/cibuildwheel.svg?style=svg)](https://circleci.com/gh/joerick/cibuildwheel) [![Build Status](https://dev.azure.com/joerick0429/cibuildwheel/_apis/build/status/joerick.cibuildwheel?branchName=master)](https://dev.azure.com/joerick0429/cibuildwheel/_build/latest?definitionId=2&branchName=master)
[![PyPI](https://img.shields.io/pypi/v/cibuildwheel.svg)](https://pypi.python.org/pypi/cibuildwheel)
[![Documentation Status](https://readthedocs.org/projects/cibuildwheel/badge/?version=stable)](https://cibuildwheel.readthedocs.io/en/stable/?badge=stable)
[![Actions Status](https://github.com/joerick/cibuildwheel/workflows/Test/badge.svg)](https://github.com/joerick/cibuildwheel/actions)
[![Travis Status](https://img.shields.io/travis/joerick/cibuildwheel/master?logo=travis)](https://travis-ci.org/joerick/cibuildwheel)
[![AppVeyor status](https://ci.appveyor.com/api/projects/status/wbsgxshp05tt1tif/branch/master?svg=true)](https://ci.appveyor.com/project/joerick/cibuildwheel/branch/master)
[![CircleCI Status](https://img.shields.io/circleci/build/gh/joerick/cibuildwheel/master?logo=circleci)](https://circleci.com/gh/joerick/cibuildwheel)
[![Azure Status](https://dev.azure.com/joerick0429/cibuildwheel/_apis/build/status/joerick.cibuildwheel?branchName=master)](https://dev.azure.com/joerick0429/cibuildwheel/_build/latest?definitionId=2&branchName=master)
[Documentation](https://cibuildwheel.readthedocs.org)
@@ -15,26 +22,24 @@ Python wheels are great. Building them across **Mac, Linux, Windows**, on **mult
What does it do?
----------------
| | macOS x86_64 | Windows 64bit | Windows 32bit | manylinux x86_64 | manylinux i686 | manylinux aarch64 | manylinux ppc64le | manylinux s390x |
|---|---|---|---|---|---|---|---|---|
| CPython 2.7 | ✅ | ✅¹ | ✅¹ | ✅ | ✅ | | | |
| CPython 3.5 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅² | ✅² | ✅³ |
| CPython 3.6 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅² | ✅² | ✅³ |
| CPython 3.7 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅² | ✅² | ✅³ |
| CPython 3.8 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅² | ✅² | ✅³ |
| CPython 3.9 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅² | ✅² | ✅³ |
| PyPy 2.7 v7.3.3 | ✅ | | ✅ | ✅ | | | | |
| PyPy 3.6 v7.3.3 | ✅ | | ✅ | ✅ | | | | |
| PyPy 3.7 (beta) v7.3.3 | ✅ | | ✅ | ✅ | | | | |
| | macOS Intel | macOS Apple Silicon | Windows 64bit | Windows 32bit | manylinux x86_64 | manylinux i686 | manylinux aarch64 | manylinux ppc64le | manylinux s390x |
|---|---|---|---|---|---|---|---|---|---|
| CPython 2.7 | ✅ | | ✅¹ | ✅¹ | ✅ | ✅ | | | |
| CPython 3.5 | ✅ | | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| CPython 3.6 | ✅ | | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| CPython 3.7 | ✅ | | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| CPython 3.8 | ✅ | | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| CPython 3.9 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| PyPy 2.7 v7.3.3 | ✅ | | | ✅ | ✅ | | | | |
| PyPy 3.6 v7.3.3 | ✅ | | | ✅ | ✅ | | | | |
| PyPy 3.7 (beta) v7.3.3 | ✅ | | | ✅ | ✅ | | | | |
<sup>¹ Not supported on Travis</sup><br>
<sup>² Only supported on Travis</sup><br>
<sup>³ Beta support until Travis CI fixes <a href="https://travis-ci.community/t/no-space-left-on-device-for-system-z/5954/11">a bug</a></sup><br>
- Builds manylinux, macOS, and Windows wheels for CPython and PyPy
- Builds manylinux, macOS 10.9+, and Windows wheels for CPython and PyPy
- Works on GitHub Actions, Azure Pipelines, Travis CI, AppVeyor, CircleCI, and GitLab CI
- Bundles shared library dependencies on Linux and macOS through [auditwheel](https://github.com/pypa/auditwheel) and [delocate](https://github.com/matthew-brett/delocate)
- Runs the library test suite against the wheel-installed version of your library
- Runs your library's tests against the wheel-installed version of your library
Usage
-----
@@ -43,23 +48,23 @@ Usage
| | Linux | macOS | Windows | Linux ARM |
|-----------------|-------|-------|---------|--------------|
| GitHub Actions | ✅ | ✅ | ✅ | ✴️¹ |
| Azure Pipelines | ✅ | ✅ | ✅ | ✴️¹ |
| GitHub Actions | ✅ | ✅ | ✅ | ✅¹ |
| Azure Pipelines | ✅ | ✅ | ✅ | |
| Travis CI | ✅ | ✅ | ✅ | ✅ |
| AppVeyor | ✅ | ✅ | ✅ | |
| CircleCI | ✅ | ✅ | | |
| Gitlab CI | ✅ | | | |
<sup>¹ Requires a "third-party build host"; expected to work with cibuildwheel but not directly tested by our CI.</sup><br>
<sup>¹ [Requires emulation](https://cibuildwheel.readthedocs.io/en/stable/faq/#emulation), distributed separately. Other services may also support Linux ARM through emulation or third-party build hosts, but these are not tested in our CI.</sup><br>
`cibuildwheel` is not intended to run on your development machine. Because it uses system Python from Python.org it will try to install packages globally - not what you expect from a build tool! Instead, isolated CI services like those mentioned above are ideal.
`cibuildwheel` is not intended to run on your development machine. Because it uses system Python from Python.org on macOS and Windows, it will try to install packages globally - not what you expect from a build tool! Instead, isolated CI services like those mentioned above are ideal. For Linux builds, it uses manylinux docker images, so those can be done locally for testing in a pinch.
<!--intro-end-->
Example setup
-------------
To build manylinux, macOS, and Windows wheels on Github Actions, you could use this `.github/workflows/wheels.yml`:
To build manylinux, macOS, and Windows wheels on GitHub Actions, you could use this `.github/workflows/wheels.yml`:
```yaml
name: Build
@@ -77,23 +82,24 @@ jobs:
steps:
- uses: actions/checkout@v2
# Used to host cibuildwheel
- uses: actions/setup-python@v2
name: Install Python
- name: Install cibuildwheel
run: python -m pip install cibuildwheel==1.8.0
- name: Build wheels
run: python -m cibuildwheel --output-dir wheelhouse
env:
CIBW_SKIP: "cp27-* pp27-*" # skip Python 2.7 wheels
# to supply options, put them in 'env', like:
# env:
# CIBW_SOME_OPTION: value
- uses: actions/upload-artifact@v2
with:
path: ./wheelhouse/*.whl
```
For more information, including building on Python 2, PyPI deployment, and the use of other CI services, check out the [documentation](https://cibuildwheel.readthedocs.org) and the [examples](https://github.com/joerick/cibuildwheel/tree/master/examples).
For more information, including building on Python 2, PyPI deployment, and the use of other CI services or the dedicated GitHub Action, check out the [documentation](https://cibuildwheel.readthedocs.org) and the [examples](https://github.com/joerick/cibuildwheel/tree/master/examples).
Options
-------
@@ -337,7 +343,7 @@ _1 January 2021_
- 🛠 Added a patch for Pypy to ensure header files are available for building
in a virtualenv. (#502)
- 🛠 Some preparatory work towards using cibuildwheel as a Github Action.
- 🛠 Some preparatory work towards using cibuildwheel as a GitHub Action.
Check out
[the FAQ](https://cibuildwheel.readthedocs.io/en/stable/faq/#option-1-github-action)
for information on how to use it. We'll be fully updating the docs to this
@@ -349,7 +355,7 @@ _21 December 2020_
- 🛠 Update dependencies, notably wheel==0.36.2 and pip==20.3.3, and CPython to
their latest bugfix releases (#489)
- 📚 Switch to a Github example in the README (#479)
- 📚 Switch to a GitHub example in the README (#479)
- 📚 Create Working Examples table, with many projects that use cibuildwheel (#474)
- 📚 Import Working Examples table and Changelog to docs
@@ -454,7 +460,7 @@ _8 July 2020_
_25 June 2020_
- 🐛 Fix "OSError: [WinError 17] The system cannot move the file to a different
disk drive" on Github Actions (#388, #389)
disk drive" on GitHub Actions (#388, #389)
### 1.5.0
@@ -468,7 +474,7 @@ _24 June 2020_
(#360, #358)
- ✨ Added workaround to allow Python 3.5 on Windows to pull dependencies from
pyproject.toml. (#358)
- 📚 Improved Github Actions examples and docs (#354, #362)
- 📚 Improved GitHub Actions examples and docs (#354, #362)
- 🐛 Ensure pip wheel uses the specified package, and doesn't build a wheel
from PyPI (#369)
- 🛠 Internal changes: using pathlib.Path, precommit hooks, testing
@@ -526,7 +532,7 @@ _2 May 2020_
_12 March 2020_
- 🌟 Add support for building on Github Actions! Check out the
- 🌟 Add support for building on GitHub Actions! Check out the
[docs](https://cibuildwheel.readthedocs.io/en/stable/setup/#github-actions)
for information on how to set it up. (#194)
- ✨ Add the `CIBW_BEFORE_TEST` option, which lets you run a command to
docs/contributing.md
+1 -1
View File
@@ -111,7 +111,7 @@ bin/bump_version.py
You'll be prompted to enter the new version number. Update the changelog when prompted. The script will create a 'bump version' commit and version tag.
Finally, cut the release and upload to PyPI/Github.
Finally, cut the release and upload to PyPI/GitHub.
```bash
rm -rf dist
docs/data/projects.yml
+2 -2
View File
@@ -1,7 +1,7 @@
# Fields:
# name: required, printed name
# gh: Github repo (eventually a url: could be added for non-github projects)
# stars: Github repo (optional, if different from package, such as for Twisted)
# gh: GitHub repo (eventually a url: could be added for non-github projects)
# stars: GitHub repo (optional, if different from package, such as for Twisted)
# os: Operating system list, [windows, apple, linux] (optional)
# ci: [appveyor, github, azurepipelines, circleci, gitlab, travisci] (optional)
# notes: (text, optional)
docs/faq.md
+69 -4
View File
@@ -12,7 +12,7 @@ If your wheel didn't compile, check the list below for some debugging tips.
- Windows: missing C feature. The Windows C compiler doesn't support C language features invented after 1990, so you'll have to backport your C code to C90. For me, this mostly involved putting my variable declarations at the top of the function like an animal.
- MacOS: calling cibuildwheel from a python3 script and getting a `ModuleNotFoundError`? Due to a [bug](https://bugs.python.org/issue22490) in CPython, you'll need to [unset the `__PYVENV_LAUNCHER__` variable](https://github.com/joerick/cibuildwheel/issues/133#issuecomment-478288597) before activating a venv.
- MacOS: calling cibuildwheel from a python3 script and getting a `ModuleNotFoundError`? Due to a (fixed) [bug](https://bugs.python.org/issue22490) in CPython, you'll need to [unset the `__PYVENV_LAUNCHER__` variable](https://github.com/joerick/cibuildwheel/issues/133#issuecomment-478288597) before activating a venv.
### Linux builds on Docker
@@ -24,17 +24,82 @@ Linux wheels are built in the [`manylinux` docker images](https://github.com/pyp
- Alternative dockers images can be specified with the `CIBW_MANYLINUX_X86_64_IMAGE`, `CIBW_MANYLINUX_I686_IMAGE`, and `CIBW_MANYLINUX_PYPY_X86_64_IMAGE` options to allow for a custom, preconfigured build environment for the Linux builds. See [options](options.md#manylinux-image) for more details.
### Building macOS wheels for Apple Silicon {: #apple-silicon}
`cibuildwheel` supports cross-compiling `universal2` and `arm64` wheels on `x86_64` runners. With the introduction of Apple Silicon, you now have several choices for wheels for Python 3.9+:
#### `x86_64`
The traditional wheel for Apple, loads on Intel machines, and on
Apple Silicon when running Python under Rosetta 2 emulation.
Due to a change in naming, Pip 20.3+ (or an installer using packaging 20.5+)
is required to install a binary wheel on macOS Big Sur.
#### `arm64`
The native wheel for macOS on Apple Silicon.
Requires Pip 20.3+ (or packaging 20.5+) to install.
#### `universal2`
This wheel contains both architectures, causing it to be up to twice the
size (data files do not get doubled, only compiled code). It requires
Pip 20.3 (Packaging 20.6+) to load on Intel, and Pip 21.0.1 (Packaging 20.9+)
to load on Apple Silicon.
!!! note
The dual-architecture `universal2` has a few benefits, but a key benefit
to a universal wheel is that a user can bundle these wheels into an
application and ship a single binary.
However, if you have a large library, then you might prefer to ship
the two single-arch wheels instead - `x86_64` and `arm64`. In rare cases,
you might want to build all three, but in that case, pip will not download
the universal wheels, because it prefers the most specific wheel
available.
Generally speaking, because Pip 20.3 is required for the `universal2` wheel,
most packages should provide both `x86_64` and `universal2` wheels for now.
Once Pip 20.3+ is common on macOS, then it should be possible to ship only the
`universal2` wheel.
**Apple Silicon wheels are not built by default**, but can be enabled by adding extra archs to the [`CIBW_ARCHS_MACOS` option](options.md#archs) - e.g. `x86_64 arm64 universal2`. Cross-compilation is provided by the Xcode toolchain.
!!! important
When cross-compiling on Intel, it is not possible to test `arm64` and the `arm64` part of a `universal2` wheel.
`cibuildwheel` will raise a warning to notify you of this - these warnings be be silenced by skipping testing on these platforms: `CIBW_TEST_SKIP: *_arm64 *_universal2:arm64`.
Hopefully, cross-compilation is a temporary situation. Once we have widely
available Apple Silicon CI runners, we can build and test `arm64` and
`universal2` wheels natively. That's why `universal2` wheels are not yet built
by default, and require opt-in by setting `CIBW_ARCHS_MACOS`.
!!! note
Your runner needs Xcode Command Line Tools 12.2 or later to build `universal2` or `arm64`.
So far, only CPython 3.9 supports `universal2` and `arm64` wheels.
Here's an example GitHub Actions workflow with a job that builds for Apple Silicon:
> .github/workflows/build_macos.yml
```yml
{% include "../examples/github-apple-silicon.yml" %}
```
### Building non-native architectures using emulation {: #emulation}
cibuildwheel supports building non-native architectures on Linux, via
emulation through the binfmt_misc kernel feature. The easiest way to use this
is via the [docker/setup-qemu-action][setup-qemu-action] on Github Actions or
is via the [docker/setup-qemu-action][setup-qemu-action] on GitHub Actions or
[tonistiigi/binfmt][binfmt].
[setup-qemu-action]: https://github.com/docker/setup-qemu-action
[binfmt]: https://hub.docker.com/r/tonistiigi/binfmt
Check out the following config for an example of how to set it up on Github
Check out the following config for an example of how to set it up on GitHub
Actions. Once QEMU is set up and registered, you just need to set the
`CIBW_ARCHS_LINUX` environment variable (or use the `--archs` option on
Linux), and the other architectures are emulated automatically.
@@ -62,7 +127,7 @@ Hopefully, this is a temporary situation. Once we have widely available Apple Si
So far, only CPython 3.9 supports `universal2` and `arm64` wheels.
Here's an example Github Actions workflow with a job that builds for Apple Silicon:
Here's an example GitHub Actions workflow with a job that builds for Apple Silicon:
> .github/workflows/build_macos.yml
```yml
docs/setup.md
+2 -2
View File
@@ -4,14 +4,14 @@ title: 'Setup'
# GitHub Actions [linux/mac/windows] {: #github-actions}
To build Linux, Mac, and Windows wheels using Github Actions, create a `.github/workflows/build.yml` file in your repo.
To build Linux, Mac, and Windows wheels using GitHub Actions, create a `.github/workflows/build.yml` file in your repo.
> build.yml
```yaml
{% include "../examples/github-minimal.yml" %}
```
Commit this file, and push to Github - either to your default branch, or to a PR branch. The build should start automatically.
Commit this file, and push to GitHub - either to your default branch, or to a PR branch. The build should start automatically.
For more info on this file, check out the [docs](https://help.github.com/en/actions/reference/workflow-syntax-for-github-actions).
examples/github-apple-silicon.yml
+1 -1
View File
@@ -13,7 +13,7 @@ jobs:
run: |
pipx run cibuildwheel==1.8.0 --output-dir wheelhouse
env:
CIBW_ARCHS_MACOS: x86_64 universal2 arm64
CIBW_ARCHS_MACOS: x86_64 universal2
- uses: actions/upload-artifact@v2
with:
examples/github-deploy.yml
+4 -10
View File
@@ -24,20 +24,14 @@ jobs:
- uses: actions/setup-python@v2
name: Install Python
with:
python-version: '3.7'
- name: Install cibuildwheel
run: |
python -m pip install cibuildwheel==1.8.0
python-version: '3.8'
- name: Install Visual C++ for Python 2.7
if: runner.os == 'Windows'
run: |
choco install vcpython27 -f -y
run: choco install vcpython27 -f -y
- name: Build wheels
run: |
python -m cibuildwheel --output-dir wheelhouse
uses: joerick/cibuildwheel@v1.8.0
- uses: actions/upload-artifact@v2
with:
@@ -52,7 +46,7 @@ jobs:
- uses: actions/setup-python@v2
name: Install Python
with:
python-version: '3.7'
python-version: '3.8'
- name: Build sdist
run: python setup.py sdist
examples/github-minimal.yml
+5 -13
View File
@@ -13,23 +13,15 @@ jobs:
steps:
- uses: actions/checkout@v2
- uses: actions/setup-python@v2
name: Install Python
with:
python-version: '3.7'
- name: Install cibuildwheel
run: |
python -m pip install cibuildwheel==1.8.0
- name: Install Visual C++ for Python 2.7
if: runner.os == 'Windows'
run: |
choco install vcpython27 -f -y
run: choco install vcpython27 -f -y
- name: Build wheels
run: |
python -m cibuildwheel --output-dir wheelhouse
uses: joerick/cibuildwheel@v1.8.0
# to supply options, put them in 'env', like:
# env:
# CIBW_SOME_OPTION: value
- uses: actions/upload-artifact@v2
with:
examples/github-with-qemu.yml
+2 -8
View File
@@ -18,14 +18,9 @@ jobs:
with:
python-version: '3.7'
- name: Install cibuildwheel
run: |
python -m pip install cibuildwheel==1.8.0
- name: Install Visual C++ for Python 2.7
if: runner.os == 'Windows'
run: |
choco install vcpython27 -f -y
run: choco install vcpython27 -f -y
- name: Set up QEMU
if: runner.os == 'Linux'
@@ -34,8 +29,7 @@ jobs:
platforms: all
- name: Build wheels
run: |
python -m cibuildwheel --output-dir wheelhouse
uses: joerick/cibuildwheel@v1.8.0
env:
# configure cibuildwheel to build native archs ('auto'), and some
# emulated ones
examples/travis-ci-deploy.yml
+1 -1
View File
@@ -14,7 +14,7 @@ jobs:
- os: windows
language: shell
before_install:
- choco install python --version 3.8.0
- choco install python --version 3.8.6
- export PATH="/c/Python38:/c/Python38/Scripts:$PATH"
# make sure it's on PATH as 'python3'
- ln -s /c/Python38/python.exe /c/Python38/python3.exe
examples/travis-ci-minimal.yml
+1 -1
View File
@@ -20,7 +20,7 @@ jobs:
- os: windows
language: shell
before_install:
- choco install python --version 3.8.0
- choco install python --version 3.8.6
- export PATH="/c/Python38:/c/Python38/Scripts:$PATH"
# make sure it's on PATH as 'python3'
- ln -s /c/Python38/python.exe /c/Python38/python3.exe
examples/travis-ci-test-and-deploy.yml
+1 -1
View File
@@ -14,7 +14,7 @@ python:
before_install:
- |
if [[ "$TRAVIS_OS_NAME" = windows ]]; then
choco install python --version 3.8.0
choco install python --version 3.8.6
export PATH="/c/Python38:/c/Python38/Scripts:$PATH"
# make sure it's on PATH as 'python3'
ln -s /c/Python38/python.exe /c/Python38/python3.exe