From 86b00da7546ce2b47ce99bcc8aced3ca0416a319 Mon Sep 17 00:00:00 2001 From: Joe Rickerby Date: Fri, 26 Feb 2021 16:36:35 +0000 Subject: [PATCH 01/12] Minor docs updates --- docs/contributing.md | 34 +++++++++++++++++----------------- 1 file changed, 17 insertions(+), 17 deletions(-) diff --git a/docs/contributing.md b/docs/contributing.md index 1307c8e5..4e3ae227 100644 --- a/docs/contributing.md +++ b/docs/contributing.md @@ -2,9 +2,9 @@ title: Contributing --- -Wheel-building can be pretty complex. I expect users to find many edge-cases - please help the rest of the community out by documenting these, adding features to support them, and reporting bugs. +Wheel-building can be pretty complex. We expect users to find edge-cases - please help the rest of the community out by documenting these, adding features to support them, and reporting bugs. -I plan to be pretty liberal in accepting pull requests, as long as they align with the design goals below. +If you have an idea for a modification or feature, it's probably best to raise an issue first and discuss it with the maintainer team. Once we have rough consensus on a design, begin work in a PR. `cibuildwheel` is indie open source. We're not paid to work on this. @@ -27,12 +27,12 @@ cibuildwheel doesn't really do anything itself - it's always deferring to other We're not responsible for errors in those tools, for fixing errors/crashes there. But cibuildwheel's job is providing users with an 'integrated' user experience across those tools. We provide an abstraction. The user says 'build me some wheels', not 'open the docker container, build a wheel with pip, fix up the symbols with auditwheel' etc. However, errors have a habit of breaking abstractions. And this is where users get confused, because the mechanism of cibuildwheel is laid bare, and they must understand a little bit how it works to debug. -So, if we can, I'd like to improve the experience on errors as well. In [this](https://github.com/joerick/cibuildwheel/issues/139) case, it takes a bit of knowledge to understand that the linux builds are happening in a totally different OS via docker, that the linked symbols won't match, that auditwheel will fail because of this. A problem with how the tools fit together, instead of the tools themselves. +So, if we can, I'd like to improve the experience on errors as well. In [this](https://github.com/joerick/cibuildwheel/issues/139) case, it takes a bit of knowledge to understand that the Linux builds are happening in a different OS via Docker, that the linked symbols won't match, that auditwheel will fail because of this. A problem with how the tools fit together, instead of the tools themselves. Maintainer notes ---------------- -## Local testing +### Local testing You should run: @@ -44,10 +44,9 @@ pip install -e .[dev] To prepare a development environment. -## Testing minimal configs +### Testing sample configs - -cibuildwheel's _minimal_ example configs can be tested on a simple project on cibuildwheel's existing CI. These should be run whenever the minimal configs change. +cibuildwheel's example configs can be tested on a simple project on cibuildwheel's existing CI. These should be run whenever the minimal configs change. To test minimal configs, make sure you have a clean git repo, then run the script: @@ -57,9 +56,15 @@ bin/run_example_ci_configs.py The script will create an isolated 'orphan' commit containing all the minimal config CI files, and a simple C extension project, and push that to a branch on the `origin` repo. The project's CI is already set up to run on branch push, so will begin testing. +You can test any other configs using `bin/run_example_ci_configs.py CONFIG_PATH`, e.g. + +```bash +bin/run_example_ci_configs.py examples/github-with-qemu.yml +``` + The script then outputs a Markdown table that can be copy/pasted into a PR to monitor and record the test. -## Preparing environments +### Preparing environments This has been moved to using docker, so you only need the following instructions if you add `--no-docker` to avoid using docker. @@ -92,16 +97,9 @@ And, you need to install the requirements into each environment: for f in env*/bin/pip; do $f install pip-tools; done ``` +### Making a release -## Making a release - -Before making a release, ensure pinned dependencies are up-to-date. Run the script: - -```bash -bin/make_dependency_update_pr.py -``` - -If updates are needed, this will push a PR with those updates for the CI to test. Once green, merge this PR. +Before making a release, ensure pinned dependencies are up-to-date. Autoupdates are run weekly, with a PR being raised with any changes as required, so just make sure the latest one is merged before continuing. Then, increment the project version number using: @@ -119,3 +117,5 @@ python setup.py sdist bdist_wheel twine upload dist/* git push && git push --tags ``` + +Then head to https://github.com/joerick/cibuildwheel/releases and create a GitHub release from the new tag, pasting in the changelog entry. From 9c797c271bf510bff4a6fe0fe9006cf9dcb8ecfa Mon Sep 17 00:00:00 2001 From: Joe Rickerby Date: Fri, 23 Apr 2021 10:28:49 +0100 Subject: [PATCH 02/12] Add delvewheel shoutout Fix #459 --- docs/options.md | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/docs/options.md b/docs/options.md index 425fe0e7..464d1c85 100644 --- a/docs/options.md +++ b/docs/options.md @@ -432,14 +432,28 @@ The command is run in a shell, so you can run multiple commands like `cmd1 && cm Platform-specific variants also available:
`CIBW_REPAIR_WHEEL_COMMAND_MACOS` | `CIBW_REPAIR_WHEEL_COMMAND_WINDOWS` | `CIBW_REPAIR_WHEEL_COMMAND_LINUX` +!!! tip + cibuildwheel doesn't yet ship a default repair command for Windows. + + **If that's an issue for you, check out [delvewheel]** - a new package that aims to do the same as `auditwheel` or `delocate` for Windows. + + Because delvewheel is still relatively early-stage, cibuildwheel does not yet run it by default. However, we'd recommend giving it a try! See the examples below for usage. + + [Delvewheel]: https://github.com/adang1345/delvewheel + #### Examples ```yaml +# use delvewheel on windows +CIBW_BEFORE_BUILD_WINDOWS: "pip install delvewheel" +CIBW_REPAIR_WHEEL_COMMAND_WINDOWS: "delvewheel repair -w {dest_dir} {wheel}" + # don't repair macOS wheels CIBW_REPAIR_WHEEL_COMMAND_MACOS: "" # pass the `--lib-sdir .` flag to auditwheel on Linux CIBW_REPAIR_WHEEL_COMMAND_LINUX: "auditwheel repair --lib-sdir . -w {dest_dir} {wheel}" + ``` From 940b48f7e6b04c33729475e92ca5c0c1388b7cc5 Mon Sep 17 00:00:00 2001 From: Joe Rickerby Date: Fri, 23 Apr 2021 10:33:05 +0100 Subject: [PATCH 03/12] add note about 3.6+ --- docs/options.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/options.md b/docs/options.md index 464d1c85..86f20ce0 100644 --- a/docs/options.md +++ b/docs/options.md @@ -437,14 +437,14 @@ Platform-specific variants also available:
**If that's an issue for you, check out [delvewheel]** - a new package that aims to do the same as `auditwheel` or `delocate` for Windows. - Because delvewheel is still relatively early-stage, cibuildwheel does not yet run it by default. However, we'd recommend giving it a try! See the examples below for usage. + Because delvewheel is still relatively early-stage, cibuildwheel does not yet run it by default. However, we'd recommend giving it a try! See the examples below for usage. [Delvewheel]: https://github.com/adang1345/delvewheel #### Examples ```yaml -# use delvewheel on windows +# use delvewheel on windows (only works on Python 3.6+) CIBW_BEFORE_BUILD_WINDOWS: "pip install delvewheel" CIBW_REPAIR_WHEEL_COMMAND_WINDOWS: "delvewheel repair -w {dest_dir} {wheel}" From e9df889f5641d8130fdf8f6f70d6149d672a67a8 Mon Sep 17 00:00:00 2001 From: Joe Rickerby Date: Fri, 23 Apr 2021 11:05:27 +0100 Subject: [PATCH 04/12] Typography tweaks --- docs/extra.css | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/extra.css b/docs/extra.css index ab62371e..0be08ca3 100644 --- a/docs/extra.css +++ b/docs/extra.css @@ -3,13 +3,14 @@ p { margin-bottom: 12px; + line-height: 26px; } code { font-size: 85%; background-color: rgba(27,31,35,.05); border-radius: 3px; - padding: .2em .4em; + padding: .15em .4em; color: inherit; border: none; margin: 0 1px; From 04837dd593c0c94ec66d752ca8027c028d61f250 Mon Sep 17 00:00:00 2001 From: Joe Rickerby Date: Fri, 23 Apr 2021 11:05:53 +0100 Subject: [PATCH 05/12] Get rid of huge repetitive MANYLINUX_IMAGE header --- docs/options.md | 23 ++++++++++++++++------- 1 file changed, 16 insertions(+), 7 deletions(-) diff --git a/docs/options.md b/docs/options.md index 86f20ce0..cba662fe 100644 --- a/docs/options.md +++ b/docs/options.md @@ -457,17 +457,26 @@ CIBW_REPAIR_WHEEL_COMMAND_LINUX: "auditwheel repair --lib-sdir . -w {dest_dir} { ``` -### `CIBW_MANYLINUX_X86_64_IMAGE`, `CIBW_MANYLINUX_I686_IMAGE`, `CIBW_MANYLINUX_PYPY_X86_64_IMAGE`, `CIBW_MANYLINUX_AARCH64_IMAGE`, `CIBW_MANYLINUX_PPC64LE_IMAGE`, `CIBW_MANYLINUX_S390X_IMAGE` {: #manylinux-image} -> Specify alternative manylinux docker images +### CIBW_MANYLINUX_*_IMAGE {: #manylinux-image} +> Specify alternative manylinux Docker images -An alternative Docker image to be used for building [`manylinux`](https://github.com/pypa/manylinux) wheels. `cibuildwheel` will then pull these instead of the default images, [`quay.io/pypa/manylinux2010_x86_64`](https://quay.io/pypa/manylinux2010_x86_64), [`quay.io/pypa/manylinux2010_i686`](https://quay.io/pypa/manylinux2010_i686), [`pypywheels/manylinux2010-pypy_x86_64`](https://hub.docker.com/r/pypywheels/manylinux2010-pypy_x86_64), [`quay.io/pypa/manylinux2014_aarch64`](https://quay.io/pypa/manylinux2014_aarch64), [`quay.io/pypa/manylinux2014_ppc64le`](https://quay.io/pypa/manylinux2014_ppc64le), and [`quay.io/pypa/manylinux2014_s390x`](https://quay.io/pypa/manylinux2010_s390x). +The available options are: -The value of this option can either be set to `manylinux1`, `manylinux2010`, `manylinux2014` or `manylinux_2_24` to use a pinned version of the [official `manylinux` images](https://github.com/pypa/manylinux) and [PyPy `manylinux` images](https://github.com/pypy/manylinux). Alternatively, set these options to any other valid Docker image name. Note that for PyPy, only the official `manylinux2010` image is currently available. For architectures other +- `CIBW_MANYLINUX_X86_64_IMAGE` +- `CIBW_MANYLINUX_I686_IMAGE` +- `CIBW_MANYLINUX_PYPY_X86_64_IMAGE` +- `CIBW_MANYLINUX_AARCH64_IMAGE` +- `CIBW_MANYLINUX_PPC64LE_IMAGE` +- `CIBW_MANYLINUX_S390X_IMAGE` + +Set an alternative Docker image to be used for building [manylinux](https://github.com/pypa/manylinux) wheels. `cibuildwheel` will then pull these instead of the default images, [`quay.io/pypa/manylinux2010_x86_64`](https://quay.io/pypa/manylinux2010_x86_64), [`quay.io/pypa/manylinux2010_i686`](https://quay.io/pypa/manylinux2010_i686), [`pypywheels/manylinux2010-pypy_x86_64`](https://hub.docker.com/r/pypywheels/manylinux2010-pypy_x86_64), [`quay.io/pypa/manylinux2014_aarch64`](https://quay.io/pypa/manylinux2014_aarch64), [`quay.io/pypa/manylinux2014_ppc64le`](https://quay.io/pypa/manylinux2014_ppc64le), and [`quay.io/pypa/manylinux2014_s390x`](https://quay.io/pypa/manylinux2010_s390x). + +The value of this option can either be set to `manylinux1`, `manylinux2010`, `manylinux2014` or `manylinux_2_24` to use a pinned version of the [official manylinux images](https://github.com/pypa/manylinux) and [PyPy manylinux images](https://github.com/pypy/manylinux). Alternatively, set these options to any other valid Docker image name. Note that for PyPy, only the official `manylinux2010` image is currently available. For architectures other than x86 (x86\_64 and i686) `manylinux2014` or `manylinux_2_24` must be used because the first version of the manylinux specification that supports additional architectures is `manylinux2014`. -Beware to specify a valid Docker image that can be used in the same way as the official, default Docker images: all necessary Python and pip versions need to be present in `/opt/python/`, and the `auditwheel` tool needs to be present for `cibuildwheel` to work. Apart from that, the architecture and relevant shared system libraries need to be manylinux1-, manylinux2010- or manylinux2014-compatible in order to produce valid `manylinux1`/`manylinux2010`/`manylinux2014`/`manylinux_2_24` wheels (see [pypa/manylinux on GitHub](https://github.com/pypa/manylinux), [PEP 513](https://www.python.org/dev/peps/pep-0513/), [PEP 571](https://www.python.org/dev/peps/pep-0571/), [PEP 599](https://www.python.org/dev/peps/pep-0599/) and [PEP 600](https://www.python.org/dev/peps/pep-0600/) for more details). +You'll need to set a Docker image that can be used in the same way as the official, default Docker images: all necessary Python and pip versions need to be present in `/opt/python/`, and the `auditwheel` tool needs to be present for `cibuildwheel` to work. Apart from that, the architecture and relevant shared system libraries need to be manylinux1-, manylinux2010- or manylinux2014-compatible in order to produce valid `manylinux1`/`manylinux2010`/`manylinux2014`/`manylinux_2_24` wheels (see [pypa/manylinux on GitHub](https://github.com/pypa/manylinux), [PEP 513](https://www.python.org/dev/peps/pep-0513/), [PEP 571](https://www.python.org/dev/peps/pep-0571/), [PEP 599](https://www.python.org/dev/peps/pep-0599/) and [PEP 600](https://www.python.org/dev/peps/pep-0600/) for more details). -Note that `auditwheel` detects the version of the `manylinux` standard in the Docker image through the `AUDITWHEEL_PLAT` environment variable, as `cibuildwheel` has no way of detecting the correct `--plat` command line argument to pass to `auditwheel` for a custom image. If a Docker image does not correctly set this `AUDITWHEEL_PLAT` environment variable, the `CIBW_ENVIRONMENT` option can be used to do so (e.g., `CIBW_ENVIRONMENT='AUDITWHEEL_PLAT="manylinux2010_$(uname -m)"'`). +Note that `auditwheel` detects the version of the manylinux standard in the Docker image through the `AUDITWHEEL_PLAT` environment variable, as `cibuildwheel` has no way of detecting the correct `--plat` command line argument to pass to `auditwheel` for a custom image. If a Docker image does not correctly set this `AUDITWHEEL_PLAT` environment variable, the `CIBW_ENVIRONMENT` option can be used to do so (e.g., `CIBW_ENVIRONMENT='AUDITWHEEL_PLAT="manylinux2010_$(uname -m)"'`). Note that `manylinux2014`/`manylinux_2_24` don't support builds with Python 2.7 - when building with `manylinux2014`/`manylinux_2_24`, skip Python 2.7 using `CIBW_SKIP` (see example below). @@ -764,7 +773,7 @@ optional arguments: $('.rst-content h3') .filter(function (i, el) { - return !!$(el).text().match(/(^([A-Z0-9, _]| and )+)¶$/); + return !!$(el).text().match(/(^([A-Z0-9, _*]| and )+)¶$/); }) .each(function (i, el) { var optionName = $(el).text().replace('¶', ''); From c6ede9b542ae6becaa7482016f7d9b3c37662b26 Mon Sep 17 00:00:00 2001 From: Joe Rickerby Date: Fri, 23 Apr 2021 12:11:43 +0100 Subject: [PATCH 06/12] Remove some of the backtick token formatting. MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Using the backtick too much breaks up the flow of the text and makes it hard to read. Here, I'm deciding that project names are not backtick'd, unless they're referring to something the user would type. e.g. ❌ the `manylinux` project ✅ the manylinux project ❌ set `CIBW_MANYLINUX_IMAGE` to manylinux2010 ✅ set `CIBW_MANYLINUX_IMAGE` to `manylinux2010` --- docs/options.md | 24 ++++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/docs/options.md b/docs/options.md index cba662fe..a3125f5d 100644 --- a/docs/options.md +++ b/docs/options.md @@ -90,7 +90,7 @@ Default: `auto` `auto` will auto-detect platform using environment variables, such as `TRAVIS_OS_NAME`/`APPVEYOR`/`CIRCLECI`. -For `linux` you need Docker running, on macOS or Linux. For `macos`, you need a Mac machine, and note that this script is going to automatically install MacPython on your system, so don't run on your development machine. For `windows`, you need to run in Windows, and `cibuildwheel` will install required versions of Python to `C:\cibw\python` using NuGet. +For `linux` you need Docker running, on macOS or Linux. For `macos`, you need a Mac machine, and note that this script is going to automatically install MacPython on your system, so don't run on your development machine. For `windows`, you need to run in Windows, and cibuildwheel will install required versions of Python to `C:\cibw\python` using NuGet. This option can also be set using the [command-line option](#command-line) `--platform`. @@ -99,7 +99,7 @@ This option can also be set using the [command-line option](#command-line) `--pl > Choose the Python versions to build -Space-separated list of builds to build and skip. Each build has an identifier like `cp27-manylinux_x86_64` or `cp35-macosx_x86_64` - you can list specific ones to build and `cibuildwheel` will only build those, and/or list ones to skip and `cibuildwheel` won't try to build them. +Space-separated list of builds to build and skip. Each build has an identifier like `cp27-manylinux_x86_64` or `cp35-macosx_x86_64` - you can list specific ones to build and cibuildwheel will only build those, and/or list ones to skip and cibuildwheel won't try to build them. When both options are specified, both conditions are applied and only builds with a tag that matches `CIBW_BUILD` and does not match `CIBW_SKIP` will be built. @@ -120,7 +120,7 @@ When setting the options, you can use shell-style globbing syntax, as per [`fnma | PyPy3.7 v7.3 | pp37-macosx_x86_64 | pp37-win32 | pp37-manylinux_x86_64 | | -The list of supported and currently selected build identifiers can also be retrieved by passing the `--print-build-identifiers` flag to `cibuildwheel`. +The list of supported and currently selected build identifiers can also be retrieved by passing the `--print-build-identifiers` flag to cibuildwheel. The format is `python_tag-platform_tag`, with tags similar to those in [PEP 425](https://www.python.org/dev/peps/pep-0425/#details). For CPython, the minimally supported macOS version is 10.9; for PyPy 2.7 and PyPy 3.6/3.7, respectively macOS 10.7 and 10.13 or higher is required. @@ -329,7 +329,7 @@ CIBW_ENVIRONMENT: "BUILD_TIME=$(date) SAMPLE_TEXT=\"sample text\"" ``` !!! note - `cibuildwheel` always defines the environment variable `CIBUILDWHEEL=1`. This can be useful for [building wheels with optional extensions](faq.md#building-packages-with-optional-c-extensions). + cibuildwheel always defines the environment variable `CIBUILDWHEEL=1`. This can be useful for [building wheels with optional extensions](faq.md#building-packages-with-optional-c-extensions). ### `CIBW_BEFORE_ALL` {: #before-all} > Execute a shell command on the build system before any wheels are built. @@ -338,7 +338,7 @@ Shell command to prepare a common part of the project (e.g. build or install lib This option is very useful for the Linux build, where builds take place in isolated Docker containers managed by cibuildwheel. This command will run inside the container before the wheel builds start. Note, if you're building both x86_64 and i686 wheels (the default), your build uses two different Docker images. In that case, this command will execute twice - once per build container. -The placeholder `{package}` can be used here; it will be replaced by the path to the package being built by `cibuildwheel`. +The placeholder `{package}` can be used here; it will be replaced by the path to the package being built by cibuildwheel. On Windows and macOS, the version of Python available inside `CIBW_BEFORE_ALL` is whatever is available on the host machine. On Linux, a modern Python version is available on PATH. @@ -361,7 +361,7 @@ A shell command to run before building the wheel. This option allows you to run If dependencies are required to build your wheel (for example if you include a header from a Python module), instead of using this command, we recommend adding requirements to a pyproject.toml file. This is reproducible, and users who do not get your wheels (such as Alpine or ClearLinux users) will still benefit. -The active Python binary can be accessed using `python`, and pip with `pip`; `cibuildwheel` makes sure the right version of Python and pip will be executed. The placeholder `{package}` can be used here; it will be replaced by the path to the package being built by `cibuildwheel`. +The active Python binary can be accessed using `python`, and pip with `pip`; cibuildwheel makes sure the right version of Python and pip will be executed. The placeholder `{package}` can be used here; it will be replaced by the path to the package being built by cibuildwheel. The command is run in a shell, so you can write things like `cmd1 && cmd2`. @@ -421,7 +421,7 @@ Default: A shell command to repair a built wheel by copying external library dependencies into the wheel tree and relinking them. The command is run on each built wheel (except for pure Python ones) before testing it. -The following placeholders must be used inside the command and will be replaced by `cibuildwheel`: +The following placeholders must be used inside the command and will be replaced by cibuildwheel: - `{wheel}` for the absolute path to the built wheel - `{dest_dir}` for the absolute path of the directory where to create the repaired wheel @@ -435,7 +435,7 @@ Platform-specific variants also available:
!!! tip cibuildwheel doesn't yet ship a default repair command for Windows. - **If that's an issue for you, check out [delvewheel]** - a new package that aims to do the same as `auditwheel` or `delocate` for Windows. + **If that's an issue for you, check out [delvewheel]** - a new package that aims to do the same as auditwheel or delocate for Windows. Because delvewheel is still relatively early-stage, cibuildwheel does not yet run it by default. However, we'd recommend giving it a try! See the examples below for usage. @@ -469,14 +469,14 @@ The available options are: - `CIBW_MANYLINUX_PPC64LE_IMAGE` - `CIBW_MANYLINUX_S390X_IMAGE` -Set an alternative Docker image to be used for building [manylinux](https://github.com/pypa/manylinux) wheels. `cibuildwheel` will then pull these instead of the default images, [`quay.io/pypa/manylinux2010_x86_64`](https://quay.io/pypa/manylinux2010_x86_64), [`quay.io/pypa/manylinux2010_i686`](https://quay.io/pypa/manylinux2010_i686), [`pypywheels/manylinux2010-pypy_x86_64`](https://hub.docker.com/r/pypywheels/manylinux2010-pypy_x86_64), [`quay.io/pypa/manylinux2014_aarch64`](https://quay.io/pypa/manylinux2014_aarch64), [`quay.io/pypa/manylinux2014_ppc64le`](https://quay.io/pypa/manylinux2014_ppc64le), and [`quay.io/pypa/manylinux2014_s390x`](https://quay.io/pypa/manylinux2010_s390x). +Set an alternative Docker image to be used for building [manylinux](https://github.com/pypa/manylinux) wheels. cibuildwheel will then pull these instead of the default images, [`quay.io/pypa/manylinux2010_x86_64`](https://quay.io/pypa/manylinux2010_x86_64), [`quay.io/pypa/manylinux2010_i686`](https://quay.io/pypa/manylinux2010_i686), [`pypywheels/manylinux2010-pypy_x86_64`](https://hub.docker.com/r/pypywheels/manylinux2010-pypy_x86_64), [`quay.io/pypa/manylinux2014_aarch64`](https://quay.io/pypa/manylinux2014_aarch64), [`quay.io/pypa/manylinux2014_ppc64le`](https://quay.io/pypa/manylinux2014_ppc64le), and [`quay.io/pypa/manylinux2014_s390x`](https://quay.io/pypa/manylinux2010_s390x). The value of this option can either be set to `manylinux1`, `manylinux2010`, `manylinux2014` or `manylinux_2_24` to use a pinned version of the [official manylinux images](https://github.com/pypa/manylinux) and [PyPy manylinux images](https://github.com/pypy/manylinux). Alternatively, set these options to any other valid Docker image name. Note that for PyPy, only the official `manylinux2010` image is currently available. For architectures other than x86 (x86\_64 and i686) `manylinux2014` or `manylinux_2_24` must be used because the first version of the manylinux specification that supports additional architectures is `manylinux2014`. -You'll need to set a Docker image that can be used in the same way as the official, default Docker images: all necessary Python and pip versions need to be present in `/opt/python/`, and the `auditwheel` tool needs to be present for `cibuildwheel` to work. Apart from that, the architecture and relevant shared system libraries need to be manylinux1-, manylinux2010- or manylinux2014-compatible in order to produce valid `manylinux1`/`manylinux2010`/`manylinux2014`/`manylinux_2_24` wheels (see [pypa/manylinux on GitHub](https://github.com/pypa/manylinux), [PEP 513](https://www.python.org/dev/peps/pep-0513/), [PEP 571](https://www.python.org/dev/peps/pep-0571/), [PEP 599](https://www.python.org/dev/peps/pep-0599/) and [PEP 600](https://www.python.org/dev/peps/pep-0600/) for more details). +You'll need to set a Docker image that can be used in the same way as the official, default Docker images: all necessary Python and pip versions need to be present in `/opt/python/`, and the auditwheel tool needs to be present for cibuildwheel to work. Apart from that, the architecture and relevant shared system libraries need to be manylinux1-, manylinux2010- or manylinux2014-compatible in order to produce valid manylinux1/manylinux2010/manylinux2014/manylinux_2_24 wheels (see [pypa/manylinux on GitHub](https://github.com/pypa/manylinux), [PEP 513](https://www.python.org/dev/peps/pep-0513/), [PEP 571](https://www.python.org/dev/peps/pep-0571/), [PEP 599](https://www.python.org/dev/peps/pep-0599/) and [PEP 600](https://www.python.org/dev/peps/pep-0600/) for more details). -Note that `auditwheel` detects the version of the manylinux standard in the Docker image through the `AUDITWHEEL_PLAT` environment variable, as `cibuildwheel` has no way of detecting the correct `--plat` command line argument to pass to `auditwheel` for a custom image. If a Docker image does not correctly set this `AUDITWHEEL_PLAT` environment variable, the `CIBW_ENVIRONMENT` option can be used to do so (e.g., `CIBW_ENVIRONMENT='AUDITWHEEL_PLAT="manylinux2010_$(uname -m)"'`). +Note that auditwheel detects the version of the manylinux standard in the Docker image through the `AUDITWHEEL_PLAT` environment variable, as cibuildwheel has no way of detecting the correct `--plat` command line argument to pass to auditwheel for a custom image. If a Docker image does not correctly set this `AUDITWHEEL_PLAT` environment variable, the `CIBW_ENVIRONMENT` option can be used to do so (e.g., `CIBW_ENVIRONMENT='AUDITWHEEL_PLAT="manylinux2010_$(uname -m)"'`). Note that `manylinux2014`/`manylinux_2_24` don't support builds with Python 2.7 - when building with `manylinux2014`/`manylinux_2_24`, skip Python 2.7 using `CIBW_SKIP` (see example below). @@ -597,7 +597,7 @@ CIBW_TEST_COMMAND: "echo Wheel installed" A shell command to run in **each** test virtual environment, before your wheel is installed and tested. This is useful if you need to install a non pip package, change values of environment variables or perform multi step pip installation (e.g. installing `scikit-build` or `cython` before install test package) -The active Python binary can be accessed using `python`, and pip with `pip`; `cibuildwheel` makes sure the right version of Python and pip will be executed. The placeholder `{package}` can be used here; it will be replaced by the path to the package being built by `cibuildwheel`. +The active Python binary can be accessed using `python`, and pip with `pip`; cibuildwheel makes sure the right version of Python and pip will be executed. The placeholder `{package}` can be used here; it will be replaced by the path to the package being built by cibuildwheel. The command is run in a shell, so you can write things like `cmd1 && cmd2`. From a50dffd844fd8617c11b0a713772c2397e7ac45f Mon Sep 17 00:00:00 2001 From: Joe Rickerby Date: Fri, 23 Apr 2021 12:42:44 +0100 Subject: [PATCH 07/12] Fix formatting of code blocks in admonitions --- docs/extra.css | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/docs/extra.css b/docs/extra.css index 0be08ca3..3fc5c37d 100644 --- a/docs/extra.css +++ b/docs/extra.css @@ -40,6 +40,14 @@ h1, h2, h3, h4, h5, h6 { white-space: normal; } +.rst-content .admonition { + line-height: inherit; +} + +.admonition code { + border: none; +} + /* Code block filename style From 9975f9c9bb689f1b336f949bf3db6be7020df36e Mon Sep 17 00:00:00 2001 From: Joe Rickerby Date: Fri, 23 Apr 2021 12:43:56 +0100 Subject: [PATCH 08/12] Copy pass over Options page - readability improvements and extra info --- docs/options.md | 42 ++++++++++++++++++++++++++---------------- 1 file changed, 26 insertions(+), 16 deletions(-) diff --git a/docs/options.md b/docs/options.md index a3125f5d..00d90523 100644 --- a/docs/options.md +++ b/docs/options.md @@ -4,7 +4,7 @@ ## Setting options -cibuildwheel is configured using environment variables, that can be set using +cibuildwheel is configured using environment variables that can be set using your CI config. For example, to configure cibuildwheel to run tests, add the following YAML to @@ -90,10 +90,21 @@ Default: `auto` `auto` will auto-detect platform using environment variables, such as `TRAVIS_OS_NAME`/`APPVEYOR`/`CIRCLECI`. -For `linux` you need Docker running, on macOS or Linux. For `macos`, you need a Mac machine, and note that this script is going to automatically install MacPython on your system, so don't run on your development machine. For `windows`, you need to run in Windows, and cibuildwheel will install required versions of Python to `C:\cibw\python` using NuGet. +- For `linux`, you need Docker running, on macOS or Linux. +- For `macos`, you need a Mac machine. Note that cibuildwheel is going to install MacPython on your system, so you probably don't want to run this on your development machine. +- For `windows`, you need to run in Windows. cibuildwheel will install required versions of Python to `C:\cibw\python` using NuGet. This option can also be set using the [command-line option](#command-line) `--platform`. +!!! tip + If you have Docker installed on macOS or Linux, you can locally debug your cibuildwheel Linux config, instead of pushing to CI to test every change. For example: + + ```bash + export CIBW_BUILD='cp37-*' + export CIBW_TEST_COMMAND='pytest {package}/tests' + cibuildwheel --platform linux . + ``` + ### `CIBW_BUILD`, `CIBW_SKIP` {: #build-skip} @@ -103,7 +114,7 @@ Space-separated list of builds to build and skip. Each build has an identifier l When both options are specified, both conditions are applied and only builds with a tag that matches `CIBW_BUILD` and does not match `CIBW_SKIP` will be built. -When setting the options, you can use shell-style globbing syntax, as per [`fnmatch`](https://docs.python.org/3/library/fnmatch.html) with the addition of curly bracket syntax `{option1,option2}`, provided by [`bracex`](https://pypi.org/project/bracex/). All the build identifiers supported by cibuildwheel are shown below: +When setting the options, you can use shell-style globbing syntax, as per [fnmatch](https://docs.python.org/3/library/fnmatch.html) with the addition of curly bracket syntax `{option1,option2}`, provided by [bracex](https://pypi.org/project/bracex/). All the build identifiers supported by cibuildwheel are shown below:
@@ -276,8 +287,8 @@ the package is compatible with all versions of Python that it can build. Currently, setuptools has not yet added support for reading this value from pyproject.toml yet, and so does not copy it to Requires-Python in the wheel - metadata. This mechanism is used by `pip` to scan through older versions of - your package until it finds a release compatible with the curernt version + metadata. This mechanism is used by pip to scan through older versions of + your package until it finds a release compatible with the current version of Python compatible when installing, so it is an important value to set if you plan to drop support for a version of Python in the future. @@ -471,15 +482,15 @@ The available options are: Set an alternative Docker image to be used for building [manylinux](https://github.com/pypa/manylinux) wheels. cibuildwheel will then pull these instead of the default images, [`quay.io/pypa/manylinux2010_x86_64`](https://quay.io/pypa/manylinux2010_x86_64), [`quay.io/pypa/manylinux2010_i686`](https://quay.io/pypa/manylinux2010_i686), [`pypywheels/manylinux2010-pypy_x86_64`](https://hub.docker.com/r/pypywheels/manylinux2010-pypy_x86_64), [`quay.io/pypa/manylinux2014_aarch64`](https://quay.io/pypa/manylinux2014_aarch64), [`quay.io/pypa/manylinux2014_ppc64le`](https://quay.io/pypa/manylinux2014_ppc64le), and [`quay.io/pypa/manylinux2014_s390x`](https://quay.io/pypa/manylinux2010_s390x). -The value of this option can either be set to `manylinux1`, `manylinux2010`, `manylinux2014` or `manylinux_2_24` to use a pinned version of the [official manylinux images](https://github.com/pypa/manylinux) and [PyPy manylinux images](https://github.com/pypy/manylinux). Alternatively, set these options to any other valid Docker image name. Note that for PyPy, only the official `manylinux2010` image is currently available. For architectures other -than x86 (x86\_64 and i686) `manylinux2014` or `manylinux_2_24` must be used because the first version of the manylinux specification that supports additional architectures is `manylinux2014`. - -You'll need to set a Docker image that can be used in the same way as the official, default Docker images: all necessary Python and pip versions need to be present in `/opt/python/`, and the auditwheel tool needs to be present for cibuildwheel to work. Apart from that, the architecture and relevant shared system libraries need to be manylinux1-, manylinux2010- or manylinux2014-compatible in order to produce valid manylinux1/manylinux2010/manylinux2014/manylinux_2_24 wheels (see [pypa/manylinux on GitHub](https://github.com/pypa/manylinux), [PEP 513](https://www.python.org/dev/peps/pep-0513/), [PEP 571](https://www.python.org/dev/peps/pep-0571/), [PEP 599](https://www.python.org/dev/peps/pep-0599/) and [PEP 600](https://www.python.org/dev/peps/pep-0600/) for more details). - -Note that auditwheel detects the version of the manylinux standard in the Docker image through the `AUDITWHEEL_PLAT` environment variable, as cibuildwheel has no way of detecting the correct `--plat` command line argument to pass to auditwheel for a custom image. If a Docker image does not correctly set this `AUDITWHEEL_PLAT` environment variable, the `CIBW_ENVIRONMENT` option can be used to do so (e.g., `CIBW_ENVIRONMENT='AUDITWHEEL_PLAT="manylinux2010_$(uname -m)"'`). +The value of this option can either be set to `manylinux1`, `manylinux2010`, `manylinux2014` or `manylinux_2_24` to use a pinned version of the [official manylinux images](https://github.com/pypa/manylinux) and [PyPy manylinux images](https://github.com/pypy/manylinux). Alternatively, set these options to any other valid Docker image name. For PyPy, only the official `manylinux2010` image is currently available. For architectures other +than x86 (x86\_64 and i686) `manylinux2014` or `manylinux_2_24` must be used, because the first version of the manylinux specification that supports additional architectures is `manylinux2014`. Note that `manylinux2014`/`manylinux_2_24` don't support builds with Python 2.7 - when building with `manylinux2014`/`manylinux_2_24`, skip Python 2.7 using `CIBW_SKIP` (see example below). +If setting a custom Docker image, you'll need to make sure it can be used in the same way as the official, default Docker images: all necessary Python and pip versions need to be present in `/opt/python/`, and the auditwheel tool needs to be present for cibuildwheel to work. Apart from that, the architecture and relevant shared system libraries need to be compatible to the relevant standard to produce valid manylinux1/manylinux2010/manylinux2014/manylinux_2_24 wheels (see [pypa/manylinux on GitHub](https://github.com/pypa/manylinux), [PEP 513](https://www.python.org/dev/peps/pep-0513/), [PEP 571](https://www.python.org/dev/peps/pep-0571/), [PEP 599](https://www.python.org/dev/peps/pep-0599/) and [PEP 600](https://www.python.org/dev/peps/pep-0600/) for more details). + +Auditwheel detects the version of the manylinux standard in the Docker image through the `AUDITWHEEL_PLAT` environment variable, as cibuildwheel has no way of detecting the correct `--plat` command line argument to pass to auditwheel for a custom image. If a Docker image does not correctly set this `AUDITWHEEL_PLAT` environment variable, the `CIBW_ENVIRONMENT` option can be used to do so (e.g., `CIBW_ENVIRONMENT='AUDITWHEEL_PLAT="manylinux2010_$(uname -m)"'`). + #### Examples ```yaml @@ -562,8 +573,7 @@ CIBW_DEPENDENCY_VERSIONS: ./constraints.txt Shell command to run tests after the build. The wheel will be installed automatically and available for import from the tests. To ensure the wheel is -imported by your tests (instead of your source copy), tests are run from a -different directory. Use the placeholders `{project}` and `{package}` when +imported by your tests (instead of your source copy), **tests are not run from your project directory**. Use the placeholders `{project}` and `{package}` when specifying paths in your project. If this variable is not set, your wheel will not be installed after building. @@ -594,8 +604,8 @@ CIBW_TEST_COMMAND: "echo Wheel installed" ### `CIBW_BEFORE_TEST` {: #before-test} > Execute a shell command before testing each wheel -A shell command to run in **each** test virtual environment, before your wheel is installed and tested. This is useful if you need to install a non pip package, change values of environment variables -or perform multi step pip installation (e.g. installing `scikit-build` or `cython` before install test package) +A shell command to run in **each** test virtual environment, before your wheel is installed and tested. This is useful if you need to install a non-pip package, invoke pip with different environment variables, +or perform a multi-step pip installation (e.g. installing scikit-build or Cython before installing test package). The active Python binary can be accessed using `python`, and pip with `pip`; cibuildwheel makes sure the right version of Python and pip will be executed. The placeholder `{package}` can be used here; it will be replaced by the path to the package being built by cibuildwheel. @@ -662,7 +672,7 @@ CIBW_TEST_EXTRAS: test,qt ### `CIBW_TEST_SKIP` {: #test-skip} > Skip running tests on some builds -This will skip testing on any identifiers that match the given skip patterns (see [`CIBW_SKIP`](#build-skip)). This can be used to mask out tests for wheels that have missing dependencies upstream that are slow or hard to build, or to mask up slow tests on emulated architectures. +This will skip testing on any identifiers that match the given skip patterns (see [`CIBW_SKIP`](#build-skip)). This can be used to mask out tests for wheels that have missing dependencies upstream that are slow or hard to build, or to skip slow tests on emulated architectures. With macOS `universal2` wheels, you can also skip the the individual archs inside the wheel using an `:arch` suffix. For example, `cp39-macosx_universal2:x86_64` or `cp39-macosx_universal2:arm64`. From d8dd54af1162f89f6acdd16564c14312680c4d1c Mon Sep 17 00:00:00 2001 From: Joe Rickerby Date: Fri, 23 Apr 2021 15:38:01 +0100 Subject: [PATCH 09/12] Update options.md --- docs/options.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/options.md b/docs/options.md index 00d90523..9c580a47 100644 --- a/docs/options.md +++ b/docs/options.md @@ -97,7 +97,7 @@ Default: `auto` This option can also be set using the [command-line option](#command-line) `--platform`. !!! tip - If you have Docker installed on macOS or Linux, you can locally debug your cibuildwheel Linux config, instead of pushing to CI to test every change. For example: + If you have Docker installed, you can locally debug your cibuildwheel Linux config, instead of pushing to CI to test every change. For example: ```bash export CIBW_BUILD='cp37-*' From 0d08a1c85ede401ba3670675bf7185a729f9d59e Mon Sep 17 00:00:00 2001 From: Joe Rickerby Date: Tue, 27 Apr 2021 08:55:39 +0100 Subject: [PATCH 10/12] Apply suggestions from code review Co-authored-by: Yannick Jadoul --- docs/options.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/options.md b/docs/options.md index 9c580a47..edea4d36 100644 --- a/docs/options.md +++ b/docs/options.md @@ -487,7 +487,7 @@ than x86 (x86\_64 and i686) `manylinux2014` or `manylinux_2_24` must be used, be Note that `manylinux2014`/`manylinux_2_24` don't support builds with Python 2.7 - when building with `manylinux2014`/`manylinux_2_24`, skip Python 2.7 using `CIBW_SKIP` (see example below). -If setting a custom Docker image, you'll need to make sure it can be used in the same way as the official, default Docker images: all necessary Python and pip versions need to be present in `/opt/python/`, and the auditwheel tool needs to be present for cibuildwheel to work. Apart from that, the architecture and relevant shared system libraries need to be compatible to the relevant standard to produce valid manylinux1/manylinux2010/manylinux2014/manylinux_2_24 wheels (see [pypa/manylinux on GitHub](https://github.com/pypa/manylinux), [PEP 513](https://www.python.org/dev/peps/pep-0513/), [PEP 571](https://www.python.org/dev/peps/pep-0571/), [PEP 599](https://www.python.org/dev/peps/pep-0599/) and [PEP 600](https://www.python.org/dev/peps/pep-0600/) for more details). +If setting a custom Docker image, you'll need to make sure it can be used in the same way as the official, default Docker images: all necessary Python and pip versions need to be present in `/opt/python/`, and the auditwheel tool needs to be present for cibuildwheel to work. Apart from that, the architecture and relevant shared system libraries need to be compatible to the relevant standard to produce valid manylinux1/manylinux2010/manylinux2014/manylinux_2_24 wheels (see [pypa/manylinux on GitHub](https://github.com/pypa/manylinux), [PEP 513](https://www.python.org/dev/peps/pep-0513/), [PEP 571](https://www.python.org/dev/peps/pep-0571/), [PEP 599](https://www.python.org/dev/peps/pep-0599/) and [PEP 600](https://www.python.org/dev/peps/pep-0600/) for more details). Auditwheel detects the version of the manylinux standard in the Docker image through the `AUDITWHEEL_PLAT` environment variable, as cibuildwheel has no way of detecting the correct `--plat` command line argument to pass to auditwheel for a custom image. If a Docker image does not correctly set this `AUDITWHEEL_PLAT` environment variable, the `CIBW_ENVIRONMENT` option can be used to do so (e.g., `CIBW_ENVIRONMENT='AUDITWHEEL_PLAT="manylinux2010_$(uname -m)"'`). @@ -573,7 +573,8 @@ CIBW_DEPENDENCY_VERSIONS: ./constraints.txt Shell command to run tests after the build. The wheel will be installed automatically and available for import from the tests. To ensure the wheel is -imported by your tests (instead of your source copy), **tests are not run from your project directory**. Use the placeholders `{project}` and `{package}` when +imported by your tests (instead of your source copy), **tests are not run from +your project directory**. Use the placeholders `{project}` and `{package}` when specifying paths in your project. If this variable is not set, your wheel will not be installed after building. From 07b87ea651deb1c9f68055f1b3307fd7d251d074 Mon Sep 17 00:00:00 2001 From: Joe Rickerby Date: Tue, 27 Apr 2021 19:05:31 +0100 Subject: [PATCH 11/12] Add Windows 2.7 FAQ entry. --- docs/faq.md | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/docs/faq.md b/docs/faq.md index a43f4ae2..223d913e 100644 --- a/docs/faq.md +++ b/docs/faq.md @@ -136,6 +136,29 @@ Here's an example GitHub Actions workflow with a job that builds for Apple Silic {% include "../examples/github-apple-silicon.yml" %} ``` +### Windows and Python 2.7 + +Building 2.7 extensions on Windows is difficult, because the VS 2008 compiler that was used for the original Python compilation was discontinued in 2018, and has since been removed from Microsoft's downloads. Most people choose to not build Windows 2.7 wheels, or to override the compiler to something more modern. + +To override, you need to have a modern compiler toolchain activated, and set `DISTUTILS_USE_SDK=1` and `MSSdk=1`. For example, on GitHub Actions, you would add these steps: + +```yaml + - name: Prepare compiler environment for Windows + if: runner.os == 'Windows' + uses: ilammy/msvc-dev-cmd@v1 + with: + arch: x64 + + - name: Set Windows environment variables + if: runner.os == 'Windows' + shell: bash + run: | + echo "DISTUTILS_USE_SDK=1" >> $GITHUB_ENV + echo "MSSdk=1" >> $GITHUB_ENV +``` + +cibuildwheel will not try to build 2.7 on Windows unless it detects that the above two variables are set. Note that changing to a more modern compiler will mean your wheel picks up a runtime dependency to a different [Visual C++ Redistributable](https://support.microsoft.com/en-us/topic/the-latest-supported-visual-c-downloads-2647da03-1eea-4433-9aff-95f26a218cc0). + ### Building packages with optional C extensions `cibuildwheel` defines the environment variable `CIBUILDWHEEL` to the value `1` allowing projects for which the C extension is optional to make it mandatory when building wheels. From e11c9070ce02e79a0a008db60f30b7034c8ab8cd Mon Sep 17 00:00:00 2001 From: Joe Rickerby Date: Fri, 30 Apr 2021 11:10:54 +0100 Subject: [PATCH 12/12] Add a cross-link between the CPP discussion and the FAQ entry. --- docs/cpp_standards.md | 4 +++- docs/faq.md | 9 ++++++++- 2 files changed, 11 insertions(+), 2 deletions(-) diff --git a/docs/cpp_standards.md b/docs/cpp_standards.md index fa26dfc7..137872cb 100644 --- a/docs/cpp_standards.md +++ b/docs/cpp_standards.md @@ -30,7 +30,9 @@ For more details see https://en.cppreference.com/w/cpp/compiler_support, https:/ ## Windows and Python 2.7 -Visual C++ for Python 2.7 does not support modern C++ standards (i.e., C++11 and later). When building on Appveyor, you will need to either use the "Visual Studio 2017" or "Visual Studio 2019" image, but Python 2.7 is not supported on these images - skip it by setting `CIBW_SKIP=cp27-win*`. +In previous years, Microsoft distributed a compiler toolchain called 'Visual C++ for Python 2.7', which was a distribution of MSVC 9 that was created to make it easier to build Python 2.7 extensions on Windows, because it was fully compatible with the toolchain that built Python 2.7. + +This toolchain does not support modern C++ standards (i.e., C++11 and later). And it is hard to find this toolchain these days, since Microsoft have stopped distributing it. So, by default, cibuildwheel does not attempt to build Python 2.7 extensions on Windows. There is an optional workaround for this, though: the pybind11 project argues and shows that it is [possible to compile Python 2.7 extension with a newer compiler](https://pybind11.readthedocs.io/en/stable/faq.html#working-with-ancient-visual-studio-2008-builds-on-windows) and has an example project showing how to do this: https://github.com/pybind/python_example. The main catch is that a user might need to install [a newer "Microsoft Visual C++ Redistributable"](https://support.microsoft.com/en-us/help/2977003/the-latest-supported-visual-c-downloads), since the newer C++ standard library's binaries are not included by default with the Python 2.7 installation. diff --git a/docs/faq.md b/docs/faq.md index 223d913e..b1b6bb48 100644 --- a/docs/faq.md +++ b/docs/faq.md @@ -155,9 +155,16 @@ To override, you need to have a modern compiler toolchain activated, and set `DI run: | echo "DISTUTILS_USE_SDK=1" >> $GITHUB_ENV echo "MSSdk=1" >> $GITHUB_ENV + + # invoke cibuildwheel... ``` -cibuildwheel will not try to build 2.7 on Windows unless it detects that the above two variables are set. Note that changing to a more modern compiler will mean your wheel picks up a runtime dependency to a different [Visual C++ Redistributable](https://support.microsoft.com/en-us/topic/the-latest-supported-visual-c-downloads-2647da03-1eea-4433-9aff-95f26a218cc0). +cibuildwheel will not try to build 2.7 on Windows unless it detects that the above two variables are set. Note that changing to a more modern compiler will mean your wheel picks up a runtime dependency to a different [Visual C++ Redistributable][]. You also need to be [a bit careful][] in designing your extension; some major binding tools like pybind11 do this for you. + +More on setting a custom Windows toolchain in our docs on modern C++ standards [here](cpp_standards.md#windows-and-python-27). + +[Visual C++ Redistributable]: https://support.microsoft.com/en-us/topic/the-latest-supported-visual-c-downloads-2647da03-1eea-4433-9aff-95f26a218cc0 +[a bit careful]: https://pybind11.readthedocs.io/en/stable/faq.html#working-with-ancient-visual-studio-2008-builds-on-windows ### Building packages with optional C extensions