Merge remote-tracking branch 'origin/master' into gitlab-ci

This commit is contained in:
Joe Rickerby
2020-09-07 20:25:14 +01:00
19 changed files with 143 additions and 75 deletions
-17
View File
@@ -1,22 +1,6 @@
version: 2
jobs:
flake8-mypy:
docker:
- image: circleci/python:3.6
steps:
- checkout
- run:
name: Install flake8 & mypy
command: sudo python -m pip install flake8 mypy
- run:
name: flake8
command: flake8
- run:
name: mypy
command: mypy
osx-python3.6:
macos:
xcode: "9.4.1"
@@ -54,6 +38,5 @@ workflows:
version: 2
all-tests:
jobs:
- flake8-mypy
- osx-python3.6
- linux-python3.6
+8
View File
@@ -7,6 +7,14 @@ on:
- 'docs/**'
jobs:
pre-commit:
name: Pre-commit checks (mypy, flake8, etc.)
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- uses: actions/setup-python@v2
- uses: pre-commit/action@v2.0.0
test:
# skip branch builds on joerick/cibuildwheel, but always build `master` and pull requests
if: github.repository != 'joerick/cibuildwheel' || github.ref == 'refs/heads/master' || github.event_name == 'pull_request'
+24 -20
View File
@@ -1,21 +1,25 @@
repos:
- repo: https://gitlab.com/pycqa/flake8
rev: 3.8.1
hooks:
- id: flake8
pass_filenames: true
# this seems to need to be here in addition to setup.cfg
exclude: cibuildwheel/resources/
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v2.3.0
hooks:
- id: check-yaml
- id: check-merge-conflict
- id: end-of-file-fixer
exclude: cibuildwheel/resources/pinned_docker_images.cfg
- repo: https://github.com/pre-commit/mirrors-mypy
rev: '' # Use the sha / tag you want to point at
hooks:
- id: mypy
files: cibuildwheel|test
exclude: cibuildwheel/resources
- repo: https://gitlab.com/pycqa/flake8
rev: 3.8.3
hooks:
- id: flake8
exclude: cibuildwheel/resources/
additional_dependencies: [flake8-bugbear]
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v3.2.0
hooks:
- id: check-case-conflict
- id: check-merge-conflict
- id: check-yaml
- id: end-of-file-fixer
exclude: cibuildwheel/resources/pinned_docker_images.cfg
- id: mixed-line-ending
- id: trailing-whitespace
- repo: https://github.com/pre-commit/mirrors-mypy
rev: v0.782
hooks:
- id: mypy
files: cibuildwheel/|test/
pass_filenames: false
+10 -10
View File
@@ -9,7 +9,7 @@ cibuildwheel
Python wheels are great. Building them across **Mac, Linux, Windows**, on **multiple versions of Python**, is not.
`cibuildwheel` is here to help. `cibuildwheel` runs on your CI server - currently it supports Azure Pipelines, Travis CI, AppVeyor, GitHub Actions, CircleCI, and Gitlab CI - and it builds and tests your wheels across all of your platforms.
`cibuildwheel` is here to help. `cibuildwheel` runs on your CI server - currently it supports Azure Pipelines, Travis CI, AppVeyor, GitHub Actions and CircleCI - and it builds and tests your wheels across all of your platforms.
What does it do?
@@ -17,19 +17,19 @@ What does it do?
| | macOS x86_64 | Windows 64bit | Windows 32bit | manylinux x86_64 | manylinux i686 | manylinux aarch64 | manylinux ppc64le | manylinux s390x |
|---|---|---|---|---|---|---|---|---|
| CPython 2.7 | ✅ | ✅¹ | ✅¹ | ✅ | ✅ | | | |
| CPython 3.5 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅² | ✅² | ✅³ |
| CPython 3.6 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅² | ✅² | ✅³ |
| CPython 3.7 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅² | ✅² | ✅³ |
| CPython 3.8 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅² | ✅² | ✅³ |
| CPython 3.9 | 🛠 | 🛠 | 🛠 | 🛠 | 🛠 | 🛠² | 🛠² | |
| PyPy 2.7 v7.3.0 | ✅ | | ✅ | ✅ | | | | |
| PyPy 3.6 v7.3.0 | ✅ | | ✅ | ✅ | | | | |
| CPython 2.7 | ✅ | ✅¹ | ✅¹ | ✅ | ✅ | | | |
| CPython 3.5 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅² | ✅² | ✅³ |
| CPython 3.6 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅² | ✅² | ✅³ |
| CPython 3.7 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅² | ✅² | ✅³ |
| CPython 3.8 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅² | ✅² | ✅³ |
| CPython 3.9 | | | | | | ✅² | ✅⁴ | ✅³ |
| PyPy 2.7 v7.3.0 | ✅ | | ✅ | ✅ | | | | |
| PyPy 3.6 v7.3.0 | ✅ | | ✅ | ✅ | | | | |
<sup>¹ Not supported on Travis</sup><br>
<sup>² Only supported on Travis</sup><br>
<sup>³ Beta support until Travis CI fixes <a href="https://travis-ci.community/t/no-space-left-on-device-for-system-z/5954/11">a bug</a></sup><br>
<sup>⁴ Python 3.9 is not yet ABI stable, so you shouldn't publish wheels with it yet. But if you want to check that your wheels build on Python 3.9, try our [`python3.9` branch](https://github.com/joerick/cibuildwheel/pull/382)!</sup>
<sup>⁴ Only Python 3.9 beta 3 is supported, until <a href="https://github.com/pypa/manylinux/issues/758">a bug</a> in building the manylinux2014 is fixed</sup><br>
- Builds manylinux, macOS and Windows wheels for CPython and PyPy using Azure Pipelines, Travis CI, AppVeyor, and CircleCI
- Bundles shared library dependencies on Linux and macOS through [auditwheel](https://github.com/pypa/auditwheel) and [delocate](https://github.com/matthew-brett/delocate)
+3 -3
View File
@@ -1,5 +1,5 @@
environment:
# workaround an appveyor bug on macos
# workaround an appveyor bug on macos
# see https://github.com/appveyor/ci/issues/3376#issuecomment-611314299
APPVEYOR_CONSOLE_DISABLE_PTY: true
matrix:
@@ -11,8 +11,8 @@ environment:
APPVEYOR_JOB_NAME: "python37-x64-macos-mojave"
# while pypy libffi dependency is failing the build, allow failures on macos
matrix:
allow_failures:
matrix:
allow_failures:
- APPVEYOR_BUILD_WORKER_IMAGE: macos-mojave
stack: python 3.7
+3 -2
View File
@@ -117,11 +117,12 @@ class DockerContainer:
return [PurePath(p) for p in path_strs]
def call(self, args: Sequence[Union[str, PathLike]], env: Dict[str, str] = {},
def call(self, args: Sequence[Union[str, PathLike]], env: Optional[Dict[str, str]] = None,
capture_output=False, cwd: Optional[Union[str, PathLike]] = None) -> str:
chdir = f'cd {cwd}' if cwd else ''
env_assignments = ' '.join(f'{shlex.quote(k)}={shlex.quote(v)}'
for k, v in env.items())
for k, v in env.items()) if env is not None else ''
command = ' '.join(shlex.quote(str(a)) for a in args)
end_of_message = str(uuid.uuid4())
+8 -3
View File
@@ -50,26 +50,31 @@ def get_python_configurations(build_selector: BuildSelector) -> List[PythonConfi
PythonConfiguration(version='3.6', identifier='cp36-manylinux_x86_64', path_str='/opt/python/cp36-cp36m'),
PythonConfiguration(version='3.7', identifier='cp37-manylinux_x86_64', path_str='/opt/python/cp37-cp37m'),
PythonConfiguration(version='3.8', identifier='cp38-manylinux_x86_64', path_str='/opt/python/cp38-cp38'),
PythonConfiguration(version='3.9', identifier='cp39-manylinux_x86_64', path_str='/opt/python/cp39-cp39'),
PythonConfiguration(version='2.7', identifier='cp27-manylinux_i686', path_str='/opt/python/cp27-cp27m'),
PythonConfiguration(version='2.7', identifier='cp27-manylinux_i686', path_str='/opt/python/cp27-cp27mu'),
PythonConfiguration(version='3.5', identifier='cp35-manylinux_i686', path_str='/opt/python/cp35-cp35m'),
PythonConfiguration(version='3.6', identifier='cp36-manylinux_i686', path_str='/opt/python/cp36-cp36m'),
PythonConfiguration(version='3.7', identifier='cp37-manylinux_i686', path_str='/opt/python/cp37-cp37m'),
PythonConfiguration(version='3.8', identifier='cp38-manylinux_i686', path_str='/opt/python/cp38-cp38'),
PythonConfiguration(version='3.9', identifier='cp39-manylinux_i686', path_str='/opt/python/cp39-cp39'),
PythonConfiguration(version='2.7', identifier='pp27-manylinux_x86_64', path_str='/opt/python/pp27-pypy_73'),
PythonConfiguration(version='3.6', identifier='pp36-manylinux_x86_64', path_str='/opt/python/pp36-pypy36_pp73'),
PythonConfiguration(version='3.5', identifier='cp35-manylinux_aarch64', path_str='/opt/python/cp35-cp35m'),
PythonConfiguration(version='3.6', identifier='cp36-manylinux_aarch64', path_str='/opt/python/cp36-cp36m'),
PythonConfiguration(version='3.7', identifier='cp37-manylinux_aarch64', path_str='/opt/python/cp37-cp37m'),
PythonConfiguration(version='3.8', identifier='cp38-manylinux_aarch64', path_str='/opt/python/cp38-cp38'),
PythonConfiguration(version='3.9', identifier='cp39-manylinux_aarch64', path_str='/opt/python/cp39-cp39'),
PythonConfiguration(version='3.5', identifier='cp35-manylinux_ppc64le', path_str='/opt/python/cp35-cp35m'),
PythonConfiguration(version='3.6', identifier='cp36-manylinux_ppc64le', path_str='/opt/python/cp36-cp36m'),
PythonConfiguration(version='3.7', identifier='cp37-manylinux_ppc64le', path_str='/opt/python/cp37-cp37m'),
PythonConfiguration(version='3.8', identifier='cp38-manylinux_ppc64le', path_str='/opt/python/cp38-cp38'),
PythonConfiguration(version='3.9', identifier='cp39-manylinux_ppc64le', path_str='/opt/python/cp39-cp39'),
PythonConfiguration(version='3.5', identifier='cp35-manylinux_s390x', path_str='/opt/python/cp35-cp35m'),
PythonConfiguration(version='3.6', identifier='cp36-manylinux_s390x', path_str='/opt/python/cp36-cp36m'),
PythonConfiguration(version='3.7', identifier='cp37-manylinux_s390x', path_str='/opt/python/cp37-cp37m'),
PythonConfiguration(version='3.8', identifier='cp38-manylinux_s390x', path_str='/opt/python/cp38-cp38'),
PythonConfiguration(version='3.9', identifier='cp39-manylinux_s390x', path_str='/opt/python/cp39-cp39'),
]
# skip builds as required
return [c for c in python_configurations if matches_platform(c.identifier) and build_selector(c.identifier)]
@@ -233,8 +238,8 @@ def build(options: BuildOptions) -> None:
def troubleshoot(package_dir: Path, error: Exception) -> None:
if (isinstance(error, subprocess.CalledProcessError) and 'exec' in error.cmd):
# the bash script failed
if (isinstance(error, subprocess.CalledProcessError) and error.cmd[0:2] == ['pip', 'wheel']):
# the 'pip wheel' step failed.
print('Checking for common errors...')
so_files = list(package_dir.glob('**/*.so'))
@@ -247,7 +252,7 @@ def troubleshoot(package_dir: Path, error: Exception) -> None:
If you're using Cython and have previously done an in-place build,
remove those build files (*.so and *.c) before starting cibuildwheel.
'''))
'''), file=sys.stderr)
print(' Files detected:')
print('\n'.join([f' {f}' for f in so_files]))
+1
View File
@@ -38,6 +38,7 @@ def get_python_configurations(build_selector: BuildSelector) -> List[PythonConfi
PythonConfiguration(version='3.6', identifier='cp36-macosx_x86_64', url='https://www.python.org/ftp/python/3.6.8/python-3.6.8-macosx10.9.pkg'),
PythonConfiguration(version='3.7', identifier='cp37-macosx_x86_64', url='https://www.python.org/ftp/python/3.7.8/python-3.7.8-macosx10.9.pkg'),
PythonConfiguration(version='3.8', identifier='cp38-macosx_x86_64', url='https://www.python.org/ftp/python/3.8.4/python-3.8.4-macosx10.9.pkg'),
PythonConfiguration(version='3.9', identifier='cp39-macosx_x86_64', url='https://www.python.org/ftp/python/3.9.0/python-3.9.0rc1-macosx10.9.pkg'),
# PyPy
PythonConfiguration(version='2.7', identifier='pp27-macosx_x86_64', url='https://downloads.python.org/pypy/pypy2.7-v7.3.1-osx64.tar.bz2'),
PythonConfiguration(version='3.6', identifier='pp36-macosx_x86_64', url='https://downloads.python.org/pypy/pypy3.6-v7.3.1-osx64.tar.bz2'),
+10 -10
View File
@@ -1,22 +1,22 @@
[x86_64]
manylinux1 = quay.io/pypa/manylinux1_x86_64:2020-05-25-b56702d
manylinux2010 = quay.io/pypa/manylinux2010_x86_64:2020-05-20-c7bbd11
manylinux2014 = quay.io/pypa/manylinux2014_x86_64:2020-05-17-2f8ac3b
manylinux1 = quay.io/pypa/manylinux1_x86_64:2020-08-20-a535ba4
manylinux2010 = quay.io/pypa/manylinux2010_x86_64:2020-08-20-df89e22
manylinux2014 = quay.io/pypa/manylinux2014_x86_64:2020-08-29-f97fd86
[i686]
manylinux1 = quay.io/pypa/manylinux1_i686:2020-05-25-b56702d
manylinux2010 = quay.io/pypa/manylinux2010_i686:2020-05-20-c7bbd11
manylinux2014 = quay.io/pypa/manylinux2014_i686:2020-05-17-2f8ac3b
manylinux1 = quay.io/pypa/manylinux1_i686:2020-08-20-a535ba4
manylinux2010 = quay.io/pypa/manylinux2010_i686:2020-08-20-df89e22
manylinux2014 = quay.io/pypa/manylinux2014_i686:2020-08-29-f97fd86
[pypy_x86_64]
manylinux2010 = pypywheels/manylinux2010-pypy_x86_64:2020-04-25-eb2cdff
manylinux2010 = pypywheels/manylinux2010-pypy_x86_64:2020-07-02-fd8c128
[aarch64]
manylinux2014 = quay.io/pypa/manylinux2014_aarch64:2020-05-17-2f8ac3b
manylinux2014 = quay.io/pypa/manylinux2014_aarch64:2020-08-29-f97fd86
[ppc64le]
manylinux2014 = quay.io/pypa/manylinux2014_ppc64le:2020-05-17-2f8ac3b
manylinux2014 = quay.io/pypa/manylinux2014_ppc64le:2020-08-03-bb4aa00
[s390x]
manylinux2014 = quay.io/pypa/manylinux2014_s390x:2020-05-17-2f8ac3b
manylinux2014 = quay.io/pypa/manylinux2014_s390x:2020-08-29-f97fd86
+9 -1
View File
@@ -58,6 +58,8 @@ def get_python_configurations(build_selector: BuildSelector) -> List[PythonConfi
PythonConfiguration(version='3.7.8', arch='64', identifier='cp37-win_amd64', url=None),
PythonConfiguration(version='3.8.4', arch='32', identifier='cp38-win32', url=None),
PythonConfiguration(version='3.8.4', arch='64', identifier='cp38-win_amd64', url=None),
PythonConfiguration(version='3.9.0-rc1', arch='32', identifier='cp39-win32', url=None),
PythonConfiguration(version='3.9.0-rc1', arch='64', identifier='cp39-win_amd64', url=None),
# PyPy
PythonConfiguration(version='2.7', arch='32', identifier='pp27-win32', url='https://downloads.python.org/pypy/pypy2.7-v7.3.1-win32.zip'),
PythonConfiguration(version='3.6', arch='32', identifier='pp36-win32', url='https://downloads.python.org/pypy/pypy3.6-v7.3.1-win32.zip'),
@@ -177,8 +179,14 @@ def pep_518_cp35_workaround(package_dir: Path, env: Dict[str, str]) -> None:
if 'build-system' in data
else []
)
if requirements:
shell(['pip', 'install'] + requirements, env=env)
with tempfile.TemporaryDirectory() as d:
reqfile = Path(d) / "requirements.txt"
with reqfile.open("w") as f:
for r in requirements:
print(r, file=f)
call(['pip', 'install', '-r', reqfile], env=env)
def build(options: BuildOptions) -> None:
+2 -2
View File
@@ -9,14 +9,14 @@ After you've built your wheels, you'll probably want to deliver them to PyPI.
On your development machine, do the following...
```bash
# Clear out your 'dist' folder.
# Clear out your 'dist' folder.
rm -rf dist
# Make a source distribution
python setup.py sdist
# 🏃🏻
# Go and download your wheel files from wherever you put them. e.g. your CI
# provider can be configured to store them for you. Put them all into the
# provider can be configured to store them for you. Put them all into the
# 'dist' folder.
# Upload using 'twine' (you may need to 'pip install twine')
+1 -1
View File
@@ -59,7 +59,7 @@ h1, h2, h3, h4, h5, h6 {
background-color: #eff1f3;
font-size: 85%;
font-weight: bold;
border-top-left-radius: 3px;
border-top-right-radius: 3px;
+20
View File
@@ -61,3 +61,23 @@ python3 -m cibuildwheel --output-dir wheelhouse
python3 -m pip install twine
python3 -m twine upload wheelhouse/*.whl
```
### 'ImportError: DLL load failed: The specific module could not be found' error on Windows
Visual Studio and MSVC link the compiled binary wheels to the Microsoft Visual C++ Runtime. Normally, these are included with Python, but when compiling with a newer version of Visual Studio, it is possible users will run into problems on systems that do not have these runtime libraries installed. The solution is to ask users to download the corresponding Visual C++ Redistributable from the [Microsoft website](https://support.microsoft.com/en-us/help/2977003/the-latest-supported-visual-c-downloads) and install it. Since a Python installation normally includes these VC++ Redistributable files for [the version of the MSVC compiler used to compile Python](https://wiki.python.org/moin/WindowsCompilers), this is typically only a problem when compiling a Python 2.7 C extension with a newer compiler, e.g. to support a modern C++ standard (see [the section on modern C++ standards for Python 2.7](cpp_standards.md#windows-and-python-27) for more details).
Additionally, Visual Studio 2019 started linking to an even newer DLL, `VCRUNTIME140_1.dll`, besides the `VCRUNTIME140.dll` that is included with recent Python versions (starting from Python 3.5; see [here](https://wiki.python.org/moin/WindowsCompilers) for more details on the corresponding Visual Studio & MSVC versions used to compile the different Python versions). To avoid this extra dependency on `VCRUNTIME140_1.dll`, the [`/d2FH4-` flag](https://devblogs.microsoft.com/cppblog/making-cpp-exception-handling-smaller-x64/) can be added to the MSVC invocations (check out [this issue](https://github.com/joerick/cibuildwheel/issues/423) for details and references).
To add the `/d2FH4-` flag to a standard `setup.py` using `setuptools`, the `extra_compile_args` option can be used:
```python
ext_modules=[
Extension(
'c_module',
sources=['extension.c'],
extra_compile_args=['/d2FH4-'] if sys.platform == 'win32' else []
)
],
```
To investigate the dependencies of a C extension (i.e., the `.pyd` file, a DLL in disguise) on Windows, [Dependency Walker](http://www.dependencywalker.com/) is a great tool.
+2 -2
View File
@@ -2,9 +2,9 @@
title: Home
---
{%
{%
includemarkdown "../README.md"
start="<!--intro-start-->"
start="<!--intro-start-->"
end="<!--intro-end-->"
%}
+2 -1
View File
@@ -1,5 +1,6 @@
[flake8]
ignore = E501,W503,E741,E226
ignore = E501,W503,E741,E226,B950
select = C,E,F,W,B,B9
application-import-names = cibuildwheel
exclude =
cibuildwheel/resources/,
+2 -2
View File
@@ -32,10 +32,10 @@ def test(tmp_path):
actual_wheels = utils.cibuildwheel_run(project_dir, add_env={
'CIBW_MANYLINUX_X86_64_IMAGE': 'dockcross/manylinux2010-x64',
'CIBW_MANYLINUX_I686_IMAGE': 'dockcross/manylinux2010-x86',
'CIBW_SKIP': 'pp*',
'CIBW_SKIP': 'pp* cp39-*',
})
# also check that we got the right wheels built
expected_wheels = [w for w in utils.expected_wheels('spam', '0.1.0')
if '-pp' not in w]
if '-pp' not in w and '-cp39-' not in w]
assert set(actual_wheels) == set(expected_wheels)
+9
View File
@@ -1,6 +1,7 @@
import textwrap
from . import test_projects
from . import utils
import os
basic_project = test_projects.new_c_project(
setup_py_add=textwrap.dedent(
@@ -27,6 +28,7 @@ basic_project.files[
[build-system]
requires = [
"setuptools >= 42",
"setuptools_scm[toml]>=4.1.2",
"wheel",
"requests==2.22.0; python_version<'3.6'",
"requests==2.23.0; python_version>='3.6'"
@@ -47,3 +49,10 @@ def test_pep518(tmp_path):
# check that the expected wheels are produced
expected_wheels = utils.expected_wheels("spam", "0.1.0")
assert set(actual_wheels) == set(expected_wheels)
# These checks ensure an extra file is not created when using custom
# workaround; see https://github.com/joerick/cibuildwheel/issues/421
assert not (project_dir / "42").exists()
assert not (project_dir / "4.1.2").exists()
assert len(os.listdir(project_dir)) == len(basic_project.files)
+28
View File
@@ -0,0 +1,28 @@
import subprocess
import pytest
from .test_projects import TestProject
from . import utils
so_file_project = TestProject()
so_file_project.files['libnothing.so'] = ''
so_file_project.files['setup.py'] = '''
raise Exception('this build will fail')
'''
def test_failed_project_with_so_files(tmp_path, capfd):
if utils.platform != 'linux':
pytest.skip('this test is only relevant to the linux build')
project_dir = tmp_path / 'project'
so_file_project.generate(project_dir)
with pytest.raises(subprocess.CalledProcessError):
utils.cibuildwheel_run(project_dir)
captured = capfd.readouterr()
print('out', captured.out)
print('err', captured.err)
assert "NOTE: Shared object (.so) files found in this project." in captured.err
+1 -1
View File
@@ -91,7 +91,7 @@ def expected_wheels(package_name, package_version, manylinux_versions=None,
else:
manylinux_versions = ['manylinux2014']
python_abi_tags = ['cp35-cp35m', 'cp36-cp36m', 'cp37-cp37m', 'cp38-cp38']
python_abi_tags = ['cp35-cp35m', 'cp36-cp36m', 'cp37-cp37m', 'cp38-cp38', 'cp39-cp39']
if pm.machine() in ['x86_64', 'AMD64', 'x86']:
python_abi_tags += ['cp27-cp27m', 'pp27-pypy_73', 'pp36-pypy36_pp73']