Merge pull request #155 from YannickJadoul/manylinux2010-alternative

Adding support for building manylinux2010 wheels - alternative
This commit is contained in:
Joe Rickerby
2019-10-20 19:29:19 +01:00
committed by GitHub
9 changed files with 187 additions and 43 deletions
+19 -12
View File
@@ -221,8 +221,8 @@ All being well, you should get wheels delivered to you in a few minutes.
| **Build parameters** | `CIBW_BUILD_VERBOSITY` | Increase or decrease the output of `pip wheel` |
| **Build environment** | `CIBW_ENVIRONMENT` | Set environment variables needed during the build |
| | `CIBW_BEFORE_BUILD` | Execute a shell command preparing each wheel's build |
| | `CIBW_MANYLINUX1_X86_64_IMAGE` | Specify an alternative manylinx1 x86_64 docker image |
| | `CIBW_MANYLINUX1_I686_IMAGE` | Specify an alternative manylinux1 i686 docker image |
| | `CIBW_MANYLINUX_X86_64_IMAGE` | Specify an alternative manylinux x86_64 docker image |
| | `CIBW_MANYLINUX_I686_IMAGE` | Specify an alternative manylinux i686 docker image |
| **Tests** | `CIBW_TEST_COMMAND` | Execute a shell command to test all built wheels |
| | `CIBW_TEST_REQUIRES` | Install Python dependencies before running the tests |
| | `CIBW_TEST_EXTRAS` | Install Python dependencies before running the tests using ``extras_require``|
@@ -231,10 +231,10 @@ A more detailed description of the options, the allowed values, and some example
### Linux builds on Docker
Linux wheels are built in the [`manylinux1` docker images](https://github.com/pypa/manylinux) to provide binary compatible wheels on Linux, according to [PEP 513](https://www.python.org/dev/peps/pep-0513/). Because of this, when building with `cibuildwheel` on Linux, a few things should be taken into account:
Linux wheels are built in the [`manylinux` docker images](https://github.com/pypa/manylinux) to provide binary compatible wheels on Linux, according to [PEP 571](https://www.python.org/dev/peps/pep-0571/). Because of this, when building with `cibuildwheel` on Linux, a few things should be taken into account:
- Programs and libraries cannot be installed on the Travis CI Ubuntu host with `apt-get`, but can be installed inside of the Docker image using `yum` or manually. The same goes for environment variables that are potentially needed to customize the wheel building. `cibuildwheel` supports this by providing the `CIBW_ENVIRONMENT` and `CIBW_BEFORE_BUILD` options to setup the build environment inside the running Docker image. See [below](#options) for details on these options.
- The project directory is mounted in the running Docker instance as `/project`, the output directory for the wheels as `/output`. In general, this is handled transparently by `cibuildwheel`. For a more finegrained level of control however, the root of the host file system is mounted as `/host`, allowing for example to access shared files, caches, etc. on the host file system. Note that this is not available on CircleCI due to their Docker policies.
- Alternative dockers images can be specified with the `CIBW_MANYLINUX1_X86_64_IMAGE` and `CIBW_MANYLINUX1_I686_IMAGE` options to allow for a custom, preconfigured build environment for the Linux builds. See [below](#options) for more details.
- Alternative dockers images can be specified with the `CIBW_MANYLINUX_X86_64_IMAGE` and `CIBW_MANYLINUX_I686_IMAGE` options to allow for a custom, preconfigured build environment for the Linux builds. See [below](#options) for more details.
Options
@@ -291,17 +291,19 @@ For `linux` you need Docker running, on Mac or Linux. For `macos`, you need a Ma
Optional.
Space-separated list of builds to build and skip. Each build has an identifier like `cp27-manylinux1_x86_64` or `cp34-macosx_10_6_intel` - you can list specific ones to build and `cibuildwheel` will only build those, and/or list ones to skip and `cibuildwheel` won't try to build them.
Space-separated list of builds to build and skip. Each build has an identifier like `cp27-manylinux_x86_64` or `cp34-macosx_10_6_intel` - you can list specific ones to build and `cibuildwheel` will only build those, and/or list ones to skip and `cibuildwheel` won't try to build them.
When both options are specified, both conditions are applied and only builds with a tag that matches `CIBW_BUILD` and does not match `CIBW_SKIP` will be built.
The format is `python_tag-platform_tag`. The tags are as defined in [PEP 0425](https://www.python.org/dev/peps/pep-0425/#details).
The format is `python_tag-platform_tag`. The tags are similar but not identical to the ones defined in [PEP 425](https://www.python.org/dev/peps/pep-0425/#details).
Python tags look like `cp27` `cp34` `cp35` `cp36` `cp37`
Platform tags look like `macosx_10_6_intel` `manylinux1_x86_64` `manylinux1_i686` `win32` `win_amd64`
Platform tags look like `macosx_10_6_intel` `manylinux_x86_64` `manylinux_i686` `win32` `win_amd64`
You can also use shell-style globbing syntax (as per `fnmatch`)
You can also use shell-style globbing syntax (as per `fnmatch`).
The list of supported and currently selected build identifiers can be retrieved by passing the `--print-build-identifiers` flag to `cibuildwheel`.
Examples:
- Only build on Python 3.6: `CIBW_BUILD`:`cp36-*`
@@ -311,7 +313,7 @@ Examples:
- Skip Python 2.7 on 32-bit Windows: `CIBW_SKIP`:`cp27-win32`
- Skip Python 3.4 and Python 3.5: `CIBW_SKIP`:`cp34-* cp35-*`
- Skip Python 3.6 on Linux: `CIBW_SKIP`:`cp36-manylinux*`
- Only build on Python 3 and skip 32-bit builds: `CIBW_BUILD`:`cp3?-*` and `CIBW_SKIP`:`*-win32 *-manylinux1_i686`
- Only build on Python 3 and skip 32-bit builds: `CIBW_BUILD`:`cp3?-*` and `CIBW_SKIP`:`*-win32 *-manylinux_i686`
***
@@ -370,15 +372,20 @@ Platform-specific variants also available:
***
| Environment variables: `CIBW_MANYLINUX1_X86_64_IMAGE` and `CIBW_MANYLINUX1_I686_IMAGE`
| Environment variables: `CIBW_MANYLINUX_X86_64_IMAGE` and `CIBW_MANYLINUX_I686_IMAGE`
| ---
Optional.
An alternative docker image to be used for building [`manylinux1`](https://github.com/pypa/manylinux) wheels. `cibuildwheel` will then pull these instead of the official images, [`quay.io/pypa/manylinux1_x86_64`](https://quay.io/pypa/manylinux1_i686) and [`quay.io/pypa/manylinux1_i686`](https://quay.io/pypa/manylinux1_i686).
An alternative Docker image to be used for building [`manylinux`](https://github.com/pypa/manylinux) wheels. `cibuildwheel` will then pull these instead of the default images, [`quay.io/pypa/manylinux2010_x86_64`](https://quay.io/pypa/manylinux2010_x86_64) and [`quay.io/pypa/manylinux2010_i686`](https://quay.io/pypa/manylinux2010_i686).
Beware to specify a valid docker image that can be used the same as the official, default docker images: all necessary Python and pip versions need to be present in `/opt/python/`, and the `auditwheel` tool needs to be present for `cibuildwheel` to work. Apart from that, the architecture and relevant shared system libraries need to be manylinux1-compatible in order to produce valid `manylinux1` wheels (see https://github.com/pypa/manylinux and [PEP 513](https://www.python.org/dev/peps/pep-0513/) for more details).
The value of this option can either be set to `manylinux1` or `manylinux2010` to use the [official `manylinux` images](https://github.com/pypa/manylinux), or any other valid Docker image name.
Beware to specify a valid Docker image that can be used in the same way as the official, default Docker images: all necessary Python and pip versions need to be present in `/opt/python/`, and the `auditwheel` tool needs to be present for `cibuildwheel` to work. Apart from that, the architecture and relevant shared system libraries need to be manylinux1- or manylinux2010-compatible in order to produce valid `manylinux1`/`manylinux2010` wheels (see https://github.com/pypa/manylinux, [PEP 513](https://www.python.org/dev/peps/pep-0513/), and [PEP 571](https://www.python.org/dev/peps/pep-0571/) for more details).
Note that `auditwheel` detects the version of the `manylinux` standard in the Docker image through the `AUDITWHEEL_PLAT` environment variable, as `cibuildwheel` has no way of detecting the correct `--plat` command line argument to pass to `auditwheel` for a custom image. If a Docker image does not correctly set this `AUDITWHEEL_PLAT` environment variable, the `CIBW_ENVIRONMENT` option can be used to do so (e.g., `CIBW_ENVIRONMENT="manylinux2010_$(uname -m)"`).
Example: `manylinux1`
Example: `dockcross/manylinux-x64`
Example: `dockcross/manylinux-x86`
+31 -3
View File
@@ -54,6 +54,8 @@ def main():
args = parser.parse_args()
detect_obsolete_options()
if args.platform != 'auto':
platform = args.platform
else:
@@ -141,11 +143,17 @@ def main():
)
if platform == 'linux':
manylinux1_x86_64_image = os.environ.get('CIBW_MANYLINUX1_X86_64_IMAGE', None)
manylinux1_i686_image = os.environ.get('CIBW_MANYLINUX1_I686_IMAGE', None)
manylinux_x86_64_image = os.environ.get('CIBW_MANYLINUX_X86_64_IMAGE', 'manylinux2010')
manylinux_i686_image = os.environ.get('CIBW_MANYLINUX_I686_IMAGE', 'manylinux2010')
default_manylinux_images_x86_64 = {'manylinux1': 'quay.io/pypa/manylinux1_x86_64',
'manylinux2010': 'quay.io/pypa/manylinux2010_x86_64'}
default_manylinux_images_i686 = {'manylinux1': 'quay.io/pypa/manylinux1_i686',
'manylinux2010': 'quay.io/pypa/manylinux2010_i686'}
build_options.update(
manylinux1_images={'x86_64': manylinux1_x86_64_image, 'i686': manylinux1_i686_image},
manylinux_images={'x86_64': default_manylinux_images_x86_64.get(manylinux_x86_64_image) or manylinux_x86_64_image,
'i686': default_manylinux_images_i686.get(manylinux_i686_image) or manylinux_i686_image},
)
elif platform == 'macos':
pass
@@ -170,6 +178,26 @@ def main():
raise Exception('Unsupported platform')
def detect_obsolete_options():
# Check the old 'MANYLINUX1_*_IMAGE' options
for (deprecated, alternative) in [('CIBW_MANYLINUX1_X86_64_IMAGE', 'CIBW_MANYLINUX_X86_64_IMAGE'),
('CIBW_MANYLINUX1_I686_IMAGE', 'CIBW_MANYLINUX_I686_IMAGE')]:
if deprecated in os.environ:
print("'{}' has been deprecated, and will be removed in a future release. Use the option '{}' instead.".format(deprecated, alternative))
if alternative not in os.environ:
print("Using value of option '{}' as replacement for '{}'".format(deprecated, alternative))
os.environ[alternative] = os.environ[deprecated]
else:
print("Option '{}' is not empty. Please unset '{}'".format(alternative, deprecated))
exit(2)
# Check for 'manylinux1' in the 'CIBW_BUILD' and 'CIBW_SKIP' options
for deprecated in ['CIBW_BUILD', 'CIBW_SKIP']:
if deprecated in os.environ and 'manylinux1' in os.environ[deprecated]:
print("Build identifiers with 'manylinux1' been deprecated. Replacing all occurences of 'manylinux1' by 'manylinux' in the option '{}'".format(deprecated))
os.environ[deprecated] = os.environ[deprecated].replace('manylinux1', 'manylinux')
def print_preamble(platform, build_options):
print(textwrap.dedent('''
_ _ _ _ _ _ _
+28 -23
View File
@@ -12,25 +12,25 @@ except ImportError:
def get_python_configurations(build_selector):
PythonConfiguration = namedtuple('PythonConfiguration', ['identifier', 'path'])
python_configurations = [
PythonConfiguration(identifier='cp27-manylinux1_x86_64', path='/opt/python/cp27-cp27m'),
PythonConfiguration(identifier='cp27-manylinux1_x86_64', path='/opt/python/cp27-cp27mu'),
PythonConfiguration(identifier='cp34-manylinux1_x86_64', path='/opt/python/cp34-cp34m'),
PythonConfiguration(identifier='cp35-manylinux1_x86_64', path='/opt/python/cp35-cp35m'),
PythonConfiguration(identifier='cp36-manylinux1_x86_64', path='/opt/python/cp36-cp36m'),
PythonConfiguration(identifier='cp37-manylinux1_x86_64', path='/opt/python/cp37-cp37m'),
PythonConfiguration(identifier='cp27-manylinux1_i686', path='/opt/python/cp27-cp27m'),
PythonConfiguration(identifier='cp27-manylinux1_i686', path='/opt/python/cp27-cp27mu'),
PythonConfiguration(identifier='cp34-manylinux1_i686', path='/opt/python/cp34-cp34m'),
PythonConfiguration(identifier='cp35-manylinux1_i686', path='/opt/python/cp35-cp35m'),
PythonConfiguration(identifier='cp36-manylinux1_i686', path='/opt/python/cp36-cp36m'),
PythonConfiguration(identifier='cp37-manylinux1_i686', path='/opt/python/cp37-cp37m'),
PythonConfiguration(identifier='cp27-manylinux_x86_64', path='/opt/python/cp27-cp27m'),
PythonConfiguration(identifier='cp27-manylinux_x86_64', path='/opt/python/cp27-cp27mu'),
PythonConfiguration(identifier='cp34-manylinux_x86_64', path='/opt/python/cp34-cp34m'),
PythonConfiguration(identifier='cp35-manylinux_x86_64', path='/opt/python/cp35-cp35m'),
PythonConfiguration(identifier='cp36-manylinux_x86_64', path='/opt/python/cp36-cp36m'),
PythonConfiguration(identifier='cp37-manylinux_x86_64', path='/opt/python/cp37-cp37m'),
PythonConfiguration(identifier='cp27-manylinux_i686', path='/opt/python/cp27-cp27m'),
PythonConfiguration(identifier='cp27-manylinux_i686', path='/opt/python/cp27-cp27mu'),
PythonConfiguration(identifier='cp34-manylinux_i686', path='/opt/python/cp34-cp34m'),
PythonConfiguration(identifier='cp35-manylinux_i686', path='/opt/python/cp35-cp35m'),
PythonConfiguration(identifier='cp36-manylinux_i686', path='/opt/python/cp36-cp36m'),
PythonConfiguration(identifier='cp37-manylinux_i686', path='/opt/python/cp37-cp37m'),
]
# skip builds as required
return [c for c in python_configurations if build_selector(c.identifier)]
def build(project_dir, output_dir, test_command, test_requires, test_extras, before_build, build_verbosity, build_selector, environment, manylinux1_images):
def build(project_dir, output_dir, test_command, test_requires, test_extras, before_build, build_verbosity, build_selector, environment, manylinux_images):
try:
subprocess.check_call(['docker', '--version'])
except:
@@ -42,8 +42,8 @@ def build(project_dir, output_dir, test_command, test_requires, test_extras, bef
python_configurations = get_python_configurations(build_selector)
platforms = [
('manylinux1_x86_64', manylinux1_images.get('x86_64') or 'quay.io/pypa/manylinux1_x86_64'),
('manylinux1_i686', manylinux1_images.get('i686') or 'quay.io/pypa/manylinux1_i686'),
('manylinux_x86_64', manylinux_images['x86_64']),
('manylinux_i686', manylinux_images['i686']),
]
for platform_tag, docker_image in platforms:
@@ -62,9 +62,9 @@ def build(project_dir, output_dir, test_command, test_requires, test_extras, bef
for PYBIN in {pybin_paths}; do
# Setup
rm -rf /tmp/built_wheel
rm -rf /tmp/delocated_wheel
rm -rf /tmp/delocated_wheels
mkdir /tmp/built_wheel
mkdir /tmp/delocated_wheel
mkdir /tmp/delocated_wheels
if [ ! -z {before_build} ]; then
PATH="$PYBIN:$PATH" sh -c {before_build}
@@ -79,11 +79,11 @@ def build(project_dir, output_dir, test_command, test_requires, test_extras, bef
# the first element
if [[ "$built_wheel" == *none-any.whl ]]; then
# pure python wheel - just copy
mv "$built_wheel" /tmp/delocated_wheel
mv "$built_wheel" /tmp/delocated_wheels
else
auditwheel repair "$built_wheel" -w /tmp/delocated_wheel
auditwheel repair "$built_wheel" -w /tmp/delocated_wheels
fi
delocated_wheel=(/tmp/delocated_wheel/*.whl)
delocated_wheels=(/tmp/delocated_wheels/*.whl)
if [ ! -z {test_command} ]; then
# Set up a virtual environment to install and test from, to make sure
@@ -100,7 +100,12 @@ def build(project_dir, output_dir, test_command, test_requires, test_extras, bef
echo "Running tests using `which python`"
# Install the wheel we just built
pip install "$delocated_wheel"{test_extras}
# Note: If auditwheel produced two wheels, it's because the earlier produced wheel
# conforms to multiple manylinux standards. These multiple versions of the wheel are
# functionally the same, differing only in name, wheel metadata, and possibly include
# different external shared libraries. so it doesn't matter which one we run the tests on.
# Let's just pick the first one.
pip install "${{delocated_wheels[0]}}"{test_extras}
# Install any requirements to run the tests
if [ ! -z "{test_requires}" ]; then
@@ -118,8 +123,8 @@ def build(project_dir, output_dir, test_command, test_requires, test_extras, bef
fi
# we're all done here; move it to output
mv "$delocated_wheel" /output
chown {uid}:{gid} "/output/$(basename "$delocated_wheel")"
mv "${{delocated_wheels[@]}}" /output
for delocated_wheel in "${{delocated_wheels[@]}}"; do chown {uid}:{gid} "/output/$(basename "$delocated_wheel")"; done
done
'''.format(
pybin_paths=' '.join(c.path+'/bin' for c in platform_configs),
+5 -1
View File
@@ -17,6 +17,10 @@ def test():
def test_build_identifiers():
# check that the number of expected wheels matches the number of build
# identifiers
expected_wheels = utils.expected_wheels('spam', '0.1.0')
# after adding CIBW_MANYLINUX_IMAGE to support manylinux2010, there
# can be multiple wheels for each wheel, though, so we need to limit
# the expected wheels
expected_wheels = [w for w in utils.expected_wheels('spam', '0.1.0')
if not '-manylinux' in w or '-manylinux1' in w]
build_identifiers = utils.cibuildwheel_get_build_identifiers(project_dir)
assert len(expected_wheels) == len(build_identifiers)
+7 -4
View File
@@ -5,14 +5,17 @@ def test():
project_dir = os.path.dirname(__file__)
if utils.platform != 'linux':
pytest.skip('the docker test is only relevant to the linux build')
pytest.skip('the test is only relevant to the linux build')
utils.cibuildwheel_run(project_dir, add_env={
'CIBW_MANYLINUX1_X86_64_IMAGE': 'dockcross/manylinux-x64',
'CIBW_MANYLINUX1_I686_IMAGE': 'dockcross/manylinux-x86',
'CIBW_MANYLINUX_X86_64_IMAGE': 'dockcross/manylinux2010-x64',
'CIBW_MANYLINUX_I686_IMAGE': 'dockcross/manylinux1-x86',
'CIBW_BEFORE_BUILD': '/opt/python/cp36-cp36m/bin/pip install -U auditwheel', # Currently necessary on dockcross images to get auditwheel 2.1 supporting AUDITWHEEL_PLAT
'CIBW_ENVIRONMENT': 'AUDITWHEEL_PLAT=`if [ $(uname -i) == "x86_64" ]; then echo "manylinux2010_x86_64"; else echo "manylinux1_i686"; fi`',
})
# also check that we got the right wheels built
expected_wheels = utils.expected_wheels('spam', '0.1.0')
expected_wheels = [w for w in utils.expected_wheels('spam', '0.1.0')
if '-manylinux2010_i686' not in w]
actual_wheels = os.listdir('wheelhouse')
assert set(actual_wheels) == set(expected_wheels)
@@ -0,0 +1,21 @@
import os, pytest
import utils
def test():
project_dir = os.path.dirname(__file__)
if utils.platform != 'linux':
pytest.skip('the docker test is only relevant to the linux build')
# build the wheels
# CFLAGS environment veriable is ecessary to fail on 'malloc_info' (on manylinux1) during compilation/linking,
# rather than when dynamically loading the Python
utils.cibuildwheel_run(project_dir, add_env={
'CIBW_ENVIRONMENT': 'CFLAGS="$CFLAGS -Werror=implicit-function-declaration"',
})
# also check that we got the right wheels
expected_wheels = [w for w in utils.expected_wheels('spam', '0.1.0')
if not '-manylinux' in w or '-manylinux2010' in w]
actual_wheels = os.listdir('wheelhouse')
assert set(actual_wheels) == set(expected_wheels)
+7
View File
@@ -0,0 +1,7 @@
from setuptools import setup, Extension
setup(
name="spam",
ext_modules=[Extension('spam', sources=['spam.c'])],
version="0.1.0",
)
+57
View File
@@ -0,0 +1,57 @@
#include <Python.h>
#if defined(__linux__)
#include <malloc.h>
#endif
static PyObject *
spam_system(PyObject *self, PyObject *args)
{
const char *command;
int sts = 0;
if (!PyArg_ParseTuple(args, "s", &command))
return NULL;
#if defined(__linux__)
sts = malloc_info(0, stdout);
#endif
if (sts == 0) {
sts = system(command);
}
return PyLong_FromLong(sts);
}
/* Module initialization */
#if PY_MAJOR_VERSION >= 3
#define MOD_INIT(name) PyMODINIT_FUNC PyInit_##name(void)
#define MOD_DEF(m, name, doc, methods, module_state_size) \
static struct PyModuleDef moduledef = { \
PyModuleDef_HEAD_INIT, name, doc, module_state_size, methods, }; \
m = PyModule_Create(&moduledef);
#define MOD_RETURN(m) return m;
#else
#define MOD_INIT(name) PyMODINIT_FUNC init##name(void)
#define MOD_DEF(m, name, doc, methods, module_state_size) \
m = Py_InitModule3(name, methods, doc);
#define MOD_RETURN(m) return;
#endif
static PyMethodDef module_methods[] = {
{"system", (PyCFunction)spam_system, METH_VARARGS,
"Execute a shell command."},
{NULL} /* Sentinel */
};
MOD_INIT(spam)
{
PyObject* m;
MOD_DEF(m,
"spam",
"Example module",
module_methods,
-1)
MOD_RETURN(m)
}
+12
View File
@@ -54,12 +54,24 @@ def expected_wheels(package_name, package_version):
'{package_name}-{package_version}-cp35-cp35m-manylinux1_x86_64.whl',
'{package_name}-{package_version}-cp36-cp36m-manylinux1_x86_64.whl',
'{package_name}-{package_version}-cp37-cp37m-manylinux1_x86_64.whl',
'{package_name}-{package_version}-cp27-cp27m-manylinux2010_x86_64.whl',
'{package_name}-{package_version}-cp27-cp27mu-manylinux2010_x86_64.whl',
'{package_name}-{package_version}-cp34-cp34m-manylinux2010_x86_64.whl',
'{package_name}-{package_version}-cp35-cp35m-manylinux2010_x86_64.whl',
'{package_name}-{package_version}-cp36-cp36m-manylinux2010_x86_64.whl',
'{package_name}-{package_version}-cp37-cp37m-manylinux2010_x86_64.whl',
'{package_name}-{package_version}-cp27-cp27m-manylinux1_i686.whl',
'{package_name}-{package_version}-cp27-cp27mu-manylinux1_i686.whl',
'{package_name}-{package_version}-cp34-cp34m-manylinux1_i686.whl',
'{package_name}-{package_version}-cp35-cp35m-manylinux1_i686.whl',
'{package_name}-{package_version}-cp36-cp36m-manylinux1_i686.whl',
'{package_name}-{package_version}-cp37-cp37m-manylinux1_i686.whl',
'{package_name}-{package_version}-cp27-cp27m-manylinux2010_i686.whl',
'{package_name}-{package_version}-cp27-cp27mu-manylinux2010_i686.whl',
'{package_name}-{package_version}-cp34-cp34m-manylinux2010_i686.whl',
'{package_name}-{package_version}-cp35-cp35m-manylinux2010_i686.whl',
'{package_name}-{package_version}-cp36-cp36m-manylinux2010_i686.whl',
'{package_name}-{package_version}-cp37-cp37m-manylinux2010_i686.whl',
]
elif platform == 'windows':
templates = [