diff --git a/docs/cpp_standards.md b/docs/cpp_standards.md index 1b33f685..4887502d 100644 --- a/docs/cpp_standards.md +++ b/docs/cpp_standards.md @@ -30,7 +30,9 @@ For more details see https://en.cppreference.com/w/cpp/compiler_support, https:/ ## Windows and Python 2.7 -Visual C++ for Python 2.7 does not support modern C++ standards (i.e., C++11 and later), and Microsoft has removed the download for the required Visual Studio 2008 needed to build a native extension in April, 2021. +In previous years, Microsoft distributed a compiler toolchain called 'Visual C++ for Python 2.7', which was a distribution of MSVC 2008 that was created to make it easier to build Python 2.7 extensions on Windows, because it was fully compatible with the toolchain that built Python 2.7. + +This toolchain does not support modern C++ standards (i.e., C++11 and later). And it is hard to find this toolchain these days, since Microsoft removed the download for the required Visual Studio 2008 needed to build a native extension in April, 2021. So, by default, cibuildwheel does not attempt to build Python 2.7 extensions on Windows. There is an optional workaround, though: the pybind11 project argues and shows that it is [possible to compile Python 2.7 extension with a newer compiler](https://pybind11.readthedocs.io/en/stable/faq.html#working-with-ancient-visual-studio-2008-builds-on-windows) and has an example project showing how to do this: https://github.com/pybind/python_example. The main catch is that a user might need to install [a newer "Microsoft Visual C++ Redistributable"](https://support.microsoft.com/en-us/help/2977003/the-latest-supported-visual-c-downloads), since the newer C++ standard library's binaries are not included by default with the Python 2.7 installation. diff --git a/docs/extra.css b/docs/extra.css index ab62371e..3fc5c37d 100644 --- a/docs/extra.css +++ b/docs/extra.css @@ -3,13 +3,14 @@ p { margin-bottom: 12px; + line-height: 26px; } code { font-size: 85%; background-color: rgba(27,31,35,.05); border-radius: 3px; - padding: .2em .4em; + padding: .15em .4em; color: inherit; border: none; margin: 0 1px; @@ -39,6 +40,14 @@ h1, h2, h3, h4, h5, h6 { white-space: normal; } +.rst-content .admonition { + line-height: inherit; +} + +.admonition code { + border: none; +} + /* Code block filename style diff --git a/docs/faq.md b/docs/faq.md index 1c6acbb2..2d8b834f 100644 --- a/docs/faq.md +++ b/docs/faq.md @@ -136,6 +136,36 @@ Here's an example GitHub Actions workflow with a job that builds for Apple Silic {% include "../examples/github-apple-silicon.yml" %} ``` +### Windows and Python 2.7 + +Building 2.7 extensions on Windows is difficult, because the VS 2008 compiler that was used for the original Python compilation was discontinued in 2018, and has since been removed from Microsoft's downloads. Most people choose to not build Windows 2.7 wheels, or to override the compiler to something more modern. + +To override, you need to have a modern compiler toolchain activated, and set `DISTUTILS_USE_SDK=1` and `MSSdk=1`. For example, on GitHub Actions, you would add these steps: + +```yaml + - name: Prepare compiler environment for Windows + if: runner.os == 'Windows' + uses: ilammy/msvc-dev-cmd@v1 + with: + arch: x64 + + - name: Set Windows environment variables + if: runner.os == 'Windows' + shell: bash + run: | + echo "DISTUTILS_USE_SDK=1" >> $GITHUB_ENV + echo "MSSdk=1" >> $GITHUB_ENV + + # invoke cibuildwheel... +``` + +cibuildwheel will not try to build 2.7 on Windows unless it detects that the above two variables are set. Note that changing to a more modern compiler will mean your wheel picks up a runtime dependency to a different [Visual C++ Redistributable][]. You also need to be [a bit careful][] in designing your extension; some major binding tools like pybind11 do this for you. + +More on setting a custom Windows toolchain in our docs on modern C++ standards [here](cpp_standards.md#windows-and-python-27). + +[Visual C++ Redistributable]: https://support.microsoft.com/en-us/topic/the-latest-supported-visual-c-downloads-2647da03-1eea-4433-9aff-95f26a218cc0 +[a bit careful]: https://pybind11.readthedocs.io/en/stable/faq.html#working-with-ancient-visual-studio-2008-builds-on-windows + ### Building packages with optional C extensions `cibuildwheel` defines the environment variable `CIBUILDWHEEL` to the value `1` allowing projects for which the C extension is optional to make it mandatory when building wheels. diff --git a/docs/options.md b/docs/options.md index 5cf97803..a8aa125c 100644 --- a/docs/options.md +++ b/docs/options.md @@ -4,7 +4,7 @@ ## Setting options -cibuildwheel is configured using environment variables, that can be set using +cibuildwheel is configured using environment variables that can be set using your CI config. For example, to configure cibuildwheel to run tests, add the following YAML to @@ -90,20 +90,31 @@ Default: `auto` `auto` will auto-detect platform using environment variables, such as `TRAVIS_OS_NAME`/`APPVEYOR`/`CIRCLECI`. -For `linux` you need Docker running, on macOS or Linux. For `macos`, you need a Mac machine, and note that this script is going to automatically install MacPython on your system, so don't run on your development machine. For `windows`, you need to run in Windows, and `cibuildwheel` will install required versions of Python to `C:\cibw\python` using NuGet. +- For `linux`, you need Docker running, on macOS or Linux. +- For `macos`, you need a Mac machine. Note that cibuildwheel is going to install MacPython on your system, so you probably don't want to run this on your development machine. +- For `windows`, you need to run in Windows. cibuildwheel will install required versions of Python to `C:\cibw\python` using NuGet. This option can also be set using the [command-line option](#command-line) `--platform`. +!!! tip + If you have Docker installed, you can locally debug your cibuildwheel Linux config, instead of pushing to CI to test every change. For example: + + ```bash + export CIBW_BUILD='cp37-*' + export CIBW_TEST_COMMAND='pytest {package}/tests' + cibuildwheel --platform linux . + ``` + ### `CIBW_BUILD`, `CIBW_SKIP` {: #build-skip} > Choose the Python versions to build -Space-separated list of builds to build and skip. Each build has an identifier like `cp27-manylinux_x86_64` or `cp35-macosx_x86_64` - you can list specific ones to build and `cibuildwheel` will only build those, and/or list ones to skip and `cibuildwheel` won't try to build them. +Space-separated list of builds to build and skip. Each build has an identifier like `cp27-manylinux_x86_64` or `cp35-macosx_x86_64` - you can list specific ones to build and cibuildwheel will only build those, and/or list ones to skip and cibuildwheel won't try to build them. When both options are specified, both conditions are applied and only builds with a tag that matches `CIBW_BUILD` and does not match `CIBW_SKIP` will be built. -When setting the options, you can use shell-style globbing syntax, as per [`fnmatch`](https://docs.python.org/3/library/fnmatch.html) with the addition of curly bracket syntax `{option1,option2}`, provided by [`bracex`](https://pypi.org/project/bracex/). All the build identifiers supported by cibuildwheel are shown below: +When setting the options, you can use shell-style globbing syntax, as per [fnmatch](https://docs.python.org/3/library/fnmatch.html) with the addition of curly bracket syntax `{option1,option2}`, provided by [bracex](https://pypi.org/project/bracex/). All the build identifiers supported by cibuildwheel are shown below:
@@ -120,7 +131,7 @@ When setting the options, you can use shell-style globbing syntax, as per [`fnma | PyPy3.7 v7.3 | pp37-macosx_x86_64 | pp37-win32 | pp37-manylinux_x86_64 | | -The list of supported and currently selected build identifiers can also be retrieved by passing the `--print-build-identifiers` flag to `cibuildwheel`. +The list of supported and currently selected build identifiers can also be retrieved by passing the `--print-build-identifiers` flag to cibuildwheel. The format is `python_tag-platform_tag`, with tags similar to those in [PEP 425](https://www.python.org/dev/peps/pep-0425/#details). For CPython, the minimally supported macOS version is 10.9; for PyPy 2.7 and PyPy 3.6/3.7, respectively macOS 10.7 and 10.13 or higher is required. @@ -276,8 +287,8 @@ the package is compatible with all versions of Python that it can build. Currently, setuptools has not yet added support for reading this value from pyproject.toml yet, and so does not copy it to Requires-Python in the wheel - metadata. This mechanism is used by `pip` to scan through older versions of - your package until it finds a release compatible with the curernt version + metadata. This mechanism is used by pip to scan through older versions of + your package until it finds a release compatible with the current version of Python compatible when installing, so it is an important value to set if you plan to drop support for a version of Python in the future. @@ -329,7 +340,7 @@ CIBW_ENVIRONMENT: "BUILD_TIME=$(date) SAMPLE_TEXT=\"sample text\"" ``` !!! note - `cibuildwheel` always defines the environment variable `CIBUILDWHEEL=1`. This can be useful for [building wheels with optional extensions](faq.md#building-packages-with-optional-c-extensions). + cibuildwheel always defines the environment variable `CIBUILDWHEEL=1`. This can be useful for [building wheels with optional extensions](faq.md#building-packages-with-optional-c-extensions). ### `CIBW_BEFORE_ALL` {: #before-all} > Execute a shell command on the build system before any wheels are built. @@ -338,7 +349,7 @@ Shell command to prepare a common part of the project (e.g. build or install lib This option is very useful for the Linux build, where builds take place in isolated Docker containers managed by cibuildwheel. This command will run inside the container before the wheel builds start. Note, if you're building both x86_64 and i686 wheels (the default), your build uses two different Docker images. In that case, this command will execute twice - once per build container. -The placeholder `{package}` can be used here; it will be replaced by the path to the package being built by `cibuildwheel`. +The placeholder `{package}` can be used here; it will be replaced by the path to the package being built by cibuildwheel. On Windows and macOS, the version of Python available inside `CIBW_BEFORE_ALL` is whatever is available on the host machine. On Linux, a modern Python version is available on PATH. @@ -366,7 +377,7 @@ A shell command to run before building the wheel. This option allows you to run If dependencies are required to build your wheel (for example if you include a header from a Python module), instead of using this command, we recommend adding requirements to a pyproject.toml file. This is reproducible, and users who do not get your wheels (such as Alpine or ClearLinux users) will still benefit. -The active Python binary can be accessed using `python`, and pip with `pip`; `cibuildwheel` makes sure the right version of Python and pip will be executed. The placeholder `{package}` can be used here; it will be replaced by the path to the package being built by `cibuildwheel`. +The active Python binary can be accessed using `python`, and pip with `pip`; cibuildwheel makes sure the right version of Python and pip will be executed. The placeholder `{package}` can be used here; it will be replaced by the path to the package being built by cibuildwheel. The command is run in a shell, so you can write things like `cmd1 && cmd2`. @@ -426,7 +437,7 @@ Default: A shell command to repair a built wheel by copying external library dependencies into the wheel tree and relinking them. The command is run on each built wheel (except for pure Python ones) before testing it. -The following placeholders must be used inside the command and will be replaced by `cibuildwheel`: +The following placeholders must be used inside the command and will be replaced by cibuildwheel: - `{wheel}` for the absolute path to the built wheel - `{dest_dir}` for the absolute path of the directory where to create the repaired wheel @@ -437,31 +448,54 @@ The command is run in a shell, so you can run multiple commands like `cmd1 && cm Platform-specific variants also available:
`CIBW_REPAIR_WHEEL_COMMAND_MACOS` | `CIBW_REPAIR_WHEEL_COMMAND_WINDOWS` | `CIBW_REPAIR_WHEEL_COMMAND_LINUX` +!!! tip + cibuildwheel doesn't yet ship a default repair command for Windows. + + **If that's an issue for you, check out [delvewheel]** - a new package that aims to do the same as auditwheel or delocate for Windows. + + Because delvewheel is still relatively early-stage, cibuildwheel does not yet run it by default. However, we'd recommend giving it a try! See the examples below for usage. + + [Delvewheel]: https://github.com/adang1345/delvewheel + #### Examples ```yaml +# use delvewheel on windows (only works on Python 3.6+) +CIBW_BEFORE_BUILD_WINDOWS: "pip install delvewheel" +CIBW_REPAIR_WHEEL_COMMAND_WINDOWS: "delvewheel repair -w {dest_dir} {wheel}" + # don't repair macOS wheels CIBW_REPAIR_WHEEL_COMMAND_MACOS: "" # pass the `--lib-sdir .` flag to auditwheel on Linux CIBW_REPAIR_WHEEL_COMMAND_LINUX: "auditwheel repair --lib-sdir . -w {dest_dir} {wheel}" + ``` -### `CIBW_MANYLINUX_X86_64_IMAGE`, `CIBW_MANYLINUX_I686_IMAGE`, `CIBW_MANYLINUX_PYPY_X86_64_IMAGE`, `CIBW_MANYLINUX_AARCH64_IMAGE`, `CIBW_MANYLINUX_PPC64LE_IMAGE`, `CIBW_MANYLINUX_S390X_IMAGE` {: #manylinux-image} -> Specify alternative manylinux docker images +### CIBW_MANYLINUX_*_IMAGE {: #manylinux-image} +> Specify alternative manylinux Docker images -An alternative Docker image to be used for building [`manylinux`](https://github.com/pypa/manylinux) wheels. `cibuildwheel` will then pull these instead of the default images, [`quay.io/pypa/manylinux2010_x86_64`](https://quay.io/pypa/manylinux2010_x86_64), [`quay.io/pypa/manylinux2010_i686`](https://quay.io/pypa/manylinux2010_i686), [`pypywheels/manylinux2010-pypy_x86_64`](https://hub.docker.com/r/pypywheels/manylinux2010-pypy_x86_64), [`quay.io/pypa/manylinux2014_aarch64`](https://quay.io/pypa/manylinux2014_aarch64), [`quay.io/pypa/manylinux2014_ppc64le`](https://quay.io/pypa/manylinux2014_ppc64le), and [`quay.io/pypa/manylinux2014_s390x`](https://quay.io/pypa/manylinux2010_s390x). +The available options are: -The value of this option can either be set to `manylinux1`, `manylinux2010`, `manylinux2014` or `manylinux_2_24` to use a pinned version of the [official `manylinux` images](https://github.com/pypa/manylinux) and [PyPy `manylinux` images](https://github.com/pypy/manylinux). Alternatively, set these options to any other valid Docker image name. Note that for PyPy, only the official `manylinux2010` image is currently available. For architectures other -than x86 (x86\_64 and i686) `manylinux2014` or `manylinux_2_24` must be used because the first version of the manylinux specification that supports additional architectures is `manylinux2014`. +- `CIBW_MANYLINUX_X86_64_IMAGE` +- `CIBW_MANYLINUX_I686_IMAGE` +- `CIBW_MANYLINUX_PYPY_X86_64_IMAGE` +- `CIBW_MANYLINUX_AARCH64_IMAGE` +- `CIBW_MANYLINUX_PPC64LE_IMAGE` +- `CIBW_MANYLINUX_S390X_IMAGE` -Beware to specify a valid Docker image that can be used in the same way as the official, default Docker images: all necessary Python and pip versions need to be present in `/opt/python/`, and the `auditwheel` tool needs to be present for `cibuildwheel` to work. Apart from that, the architecture and relevant shared system libraries need to be manylinux1-, manylinux2010- or manylinux2014-compatible in order to produce valid `manylinux1`/`manylinux2010`/`manylinux2014`/`manylinux_2_24` wheels (see [pypa/manylinux on GitHub](https://github.com/pypa/manylinux), [PEP 513](https://www.python.org/dev/peps/pep-0513/), [PEP 571](https://www.python.org/dev/peps/pep-0571/), [PEP 599](https://www.python.org/dev/peps/pep-0599/) and [PEP 600](https://www.python.org/dev/peps/pep-0600/) for more details). +Set an alternative Docker image to be used for building [manylinux](https://github.com/pypa/manylinux) wheels. cibuildwheel will then pull these instead of the default images, [`quay.io/pypa/manylinux2010_x86_64`](https://quay.io/pypa/manylinux2010_x86_64), [`quay.io/pypa/manylinux2010_i686`](https://quay.io/pypa/manylinux2010_i686), [`pypywheels/manylinux2010-pypy_x86_64`](https://hub.docker.com/r/pypywheels/manylinux2010-pypy_x86_64), [`quay.io/pypa/manylinux2014_aarch64`](https://quay.io/pypa/manylinux2014_aarch64), [`quay.io/pypa/manylinux2014_ppc64le`](https://quay.io/pypa/manylinux2014_ppc64le), and [`quay.io/pypa/manylinux2014_s390x`](https://quay.io/pypa/manylinux2010_s390x). -Note that `auditwheel` detects the version of the `manylinux` standard in the Docker image through the `AUDITWHEEL_PLAT` environment variable, as `cibuildwheel` has no way of detecting the correct `--plat` command line argument to pass to `auditwheel` for a custom image. If a Docker image does not correctly set this `AUDITWHEEL_PLAT` environment variable, the `CIBW_ENVIRONMENT` option can be used to do so (e.g., `CIBW_ENVIRONMENT='AUDITWHEEL_PLAT="manylinux2010_$(uname -m)"'`). +The value of this option can either be set to `manylinux1`, `manylinux2010`, `manylinux2014` or `manylinux_2_24` to use a pinned version of the [official manylinux images](https://github.com/pypa/manylinux) and [PyPy manylinux images](https://github.com/pypy/manylinux). Alternatively, set these options to any other valid Docker image name. For PyPy, only the official `manylinux2010` image is currently available. For architectures other +than x86 (x86\_64 and i686) `manylinux2014` or `manylinux_2_24` must be used, because the first version of the manylinux specification that supports additional architectures is `manylinux2014`. Note that `manylinux2014`/`manylinux_2_24` don't support builds with Python 2.7 - when building with `manylinux2014`/`manylinux_2_24`, skip Python 2.7 using `CIBW_SKIP` (see example below). +If setting a custom Docker image, you'll need to make sure it can be used in the same way as the official, default Docker images: all necessary Python and pip versions need to be present in `/opt/python/`, and the auditwheel tool needs to be present for cibuildwheel to work. Apart from that, the architecture and relevant shared system libraries need to be compatible to the relevant standard to produce valid manylinux1/manylinux2010/manylinux2014/manylinux_2_24 wheels (see [pypa/manylinux on GitHub](https://github.com/pypa/manylinux), [PEP 513](https://www.python.org/dev/peps/pep-0513/), [PEP 571](https://www.python.org/dev/peps/pep-0571/), [PEP 599](https://www.python.org/dev/peps/pep-0599/) and [PEP 600](https://www.python.org/dev/peps/pep-0600/) for more details). + +Auditwheel detects the version of the manylinux standard in the Docker image through the `AUDITWHEEL_PLAT` environment variable, as cibuildwheel has no way of detecting the correct `--plat` command line argument to pass to auditwheel for a custom image. If a Docker image does not correctly set this `AUDITWHEEL_PLAT` environment variable, the `CIBW_ENVIRONMENT` option can be used to do so (e.g., `CIBW_ENVIRONMENT='AUDITWHEEL_PLAT="manylinux2010_$(uname -m)"'`). + #### Examples ```yaml @@ -544,8 +578,8 @@ CIBW_DEPENDENCY_VERSIONS: ./constraints.txt Shell command to run tests after the build. The wheel will be installed automatically and available for import from the tests. To ensure the wheel is -imported by your tests (instead of your source copy), tests are run from a -different directory. Use the placeholders `{project}` and `{package}` when +imported by your tests (instead of your source copy), **tests are not run from +your project directory**. Use the placeholders `{project}` and `{package}` when specifying paths in your project. If this variable is not set, your wheel will not be installed after building. @@ -576,10 +610,10 @@ CIBW_TEST_COMMAND: "echo Wheel installed" ### `CIBW_BEFORE_TEST` {: #before-test} > Execute a shell command before testing each wheel -A shell command to run in **each** test virtual environment, before your wheel is installed and tested. This is useful if you need to install a non pip package, change values of environment variables -or perform multi step pip installation (e.g. installing `scikit-build` or `cython` before install test package) +A shell command to run in **each** test virtual environment, before your wheel is installed and tested. This is useful if you need to install a non-pip package, invoke pip with different environment variables, +or perform a multi-step pip installation (e.g. installing scikit-build or Cython before installing test package). -The active Python binary can be accessed using `python`, and pip with `pip`; `cibuildwheel` makes sure the right version of Python and pip will be executed. The placeholder `{package}` can be used here; it will be replaced by the path to the package being built by `cibuildwheel`. +The active Python binary can be accessed using `python`, and pip with `pip`; cibuildwheel makes sure the right version of Python and pip will be executed. The placeholder `{package}` can be used here; it will be replaced by the path to the package being built by cibuildwheel. The command is run in a shell, so you can write things like `cmd1 && cmd2`. @@ -644,7 +678,7 @@ CIBW_TEST_EXTRAS: test,qt ### `CIBW_TEST_SKIP` {: #test-skip} > Skip running tests on some builds -This will skip testing on any identifiers that match the given skip patterns (see [`CIBW_SKIP`](#build-skip)). This can be used to mask out tests for wheels that have missing dependencies upstream that are slow or hard to build, or to mask up slow tests on emulated architectures. +This will skip testing on any identifiers that match the given skip patterns (see [`CIBW_SKIP`](#build-skip)). This can be used to mask out tests for wheels that have missing dependencies upstream that are slow or hard to build, or to skip slow tests on emulated architectures. With macOS `universal2` wheels, you can also skip the the individual archs inside the wheel using an `:arch` suffix. For example, `cp39-macosx_universal2:x86_64` or `cp39-macosx_universal2:arm64`. @@ -755,7 +789,7 @@ optional arguments: $('.rst-content h3') .filter(function (i, el) { - return !!$(el).text().match(/(^([A-Z0-9, _]| and )+)¶$/); + return !!$(el).text().match(/(^([A-Z0-9, _*]| and )+)¶$/); }) .each(function (i, el) { var optionName = $(el).text().replace('¶', '');