Bump version

macOS: Fix Pip bugs resulting from PyPI TLS 1.2 enforcement
macOS: Fix brew python version

.travis.yml

@@ -32,7 +32,7 @@ matrix:
         - "PYTHON=python3"
       before_install:
         - brew update
-        - brew install python3
+        - brew outdated python || brew upgrade python
 
 script:
   - |

README.md

@@ -36,15 +36,20 @@ Usage
 - Create a `.travis.yml` file in your repo.
 
     ```
+    language: python
+    
     matrix:
       include:
         - sudo: required
           services:
             - docker
+          env: PIP=pip
         - os: osx
+          language: generic
+          env: PIP=pip2
 
     script:
-      - pip install cibuildwheel==0.5.1
+      - $PIP install cibuildwheel==0.7.1
       - cibuildwheel --output-dir wheelhouse
     ```
 
@@ -54,7 +59,7 @@ Usage
 
     ```
     build_script:
-      - pip install cibuildwheel==0.5.1
+      - pip install cibuildwheel==0.7.1
       - cibuildwheel --output-dir wheelhouse
     artifacts:
       - path: "wheelhouse\\*.whl"
@@ -79,11 +84,20 @@ All being well, you should get wheels delivered to you in a few minutes.
 |   | `CIBW_SKIP` | Skip certain Python versions |
 | **Build environment** | `CIBW_ENVIRONMENT` | Set environment variables needed during the build |
 |   | `CIBW_BEFORE_BUILD` | Execute a shell command preparing each wheel's build |
+|   | `CIBW_MANYLINUX1_X86_64_IMAGE` | Specify an alternative manylinx1 x86_64 docker image |
+|   | `CIBW_MANYLINUX1_I686_IMAGE` | Specify an alternative manylinux1 i686 docker image |
 | **Tests** | `CIBW_TEST_COMMAND` | Execute a shell command to test all built wheels |
 |   | `CIBW_TEST_REQUIRES` | Install Python dependencies before running the tests |
 
 A more detailed description of the options, the allowed values, and some examples can be found in the [Options](#options) section.
 
+### Linux builds on Docker
+
+Linux wheels are built in the [`manylinux1` docker images](https://github.com/pypa/manylinux) to provide binary compatible wheels on Linux, according to [PEP 513](https://www.python.org/dev/peps/pep-0513/). Because of this, when building with `cibuildwheel` on Linux, a few things should be taken into account:
+- Progams and libraries cannot be installed on the Travis CI Ubuntu host with `apt-get`, but can be installed inside of the Docker image using `yum` or manually. The same goes for environment variables that are potentially needed to customize the wheel building. `cibuildwheel` supports this by providing the `CIBW_ENVIRONMENT` and `CIBW_BEFORE_BUILD` options to setup the build environment inside the running Docker image. See [below](#options) for details on these options.
+- The project directory is mounted in the running Docker instance as `/project`, the output directory for the wheels as `/output`. In general, this is handled transparently by `cibuildwheel`. For a more finegrained level of control however, the root of the host file system is mounted as `/host`, allowing for example to access shared files, caches, etc. on the host file system.
+- Alternative dockers images can be specified with the `CIBW_MANYLINUX1_X86_64_IMAGE` and `CIBW_MANYLINUX1_I686_IMAGE` options to allow for a custom, preconfigured build environment for the Linux builds. See [below](#options) for more details.
+
 
 Options
 -------
@@ -139,13 +153,18 @@ The format is `python_tag-platform_tag`. The tags are as defined in [PEP 0425](h
 
 Python tags look like `cp27` `cp34` `cp35` `cp36`
 
-Platform tags look like `macosx_10_6_intel` `manylinux1_x86_64` `manylinux1_i386` `win32` `win_amd64`
+Platform tags look like `macosx_10_6_intel` `manylinux1_x86_64` `manylinux1_i686` `win32` `win_amd64`
 
 You can also use shell-style globbing syntax (as per `fnmatch`) 
 
-Example: `cp27-macosx_10_6_intel`  (don't build on Python 2 on Mac)  
-Example: `cp27-win*`  (don't build on Python 2.7 on Windows)  
-Example: `cp34-* cp35-*`  (don't build on Python 3.4 or Python 3.5)  
+Examples:
+- Skip building on Python 2.7 on the Mac: `cp27-macosx_10_6_intel`  
+- Skip building on Python 2.7 on all platforms: `cp27-*`  
+- Skip Python 2.7 on Windows: `cp27-win*`  
+- Skip Python 2.7 on 32bit Windows: `cp27-win32`  
+- Skip Python 3.4 and Python 3.5: `cp34-* cp35-*`
+- Skip Python 3.6 on Linux: `cp36-manylinux*`
+- Only build on Python 3.6: `cp27-* cp34-* cp35-*`
 
 | Environment variable: `CIBW_ENVIRONMENT`
 | ---
@@ -186,6 +205,18 @@ Example: `yum install -y libffi-dev && {pip} install .`
 Platform-specific variants also available:  
  `CIBW_BEFORE_BUILD_MACOS` | `CIBW_BEFORE_BUILD_WINDOWS` | `CIBW_BEFORE_BUILD_LINUX`
 
+| Environment variables: `CIBW_MANYLINUX1_X86_64_IMAGE` and `CIBW_MANYLINUX1_I686_IMAGE`
+| ---
+
+Optional.
+
+An alternative docker image to be used for building [`manylinux1`](https://github.com/pypa/manylinux) wheels. `cibuildwheel` will then pull these instead of the official images, [`quay.io/pypa/manylinux1_x86_64`](https://quay.io/pypa/manylinux1_i686) and [`quay.io/pypa/manylinux1_i686`](https://quay.io/pypa/manylinux1_i686).
+
+Beware to specify a valid docker image that can be used the same as the official, default docker images: all necessary Python and pip versions need to be present in `/opt/python/`, and the `auditwheel` tool needs to be present for `cibuildwheel` to work. Apart from that, the architecture and relevant shared system libraries need to be manylinux1-compatible in order to produce valid `manylinux1` wheels (see https://github.com/pypa/manylinux and [PEP 513](https://www.python.org/dev/peps/pep-0513/) for more details).
+
+Example: `dockcross/manylinux-x64`  
+Example: `dockcross/manylinux-x86`
+
 | Environment variable: `CIBW_TEST_COMMAND`
 | ---
 
@@ -273,6 +304,8 @@ Here are some repos that use cibuildwheel.
 
 - [pyinstrument_cext](https://github.com/joerick/pyinstrument_cext)
 - [websockets](https://github.com/aaugustin/websockets)
+- [Parselmouth](https://github.com/YannickJadoul/Parselmouth)
+- [python-admesh](https://github.com/admesh/python-admesh)
 
 > Add repo here! Send a PR.
 
@@ -284,6 +317,21 @@ Since `cibuildwheel` runs the wheel through delocate or auditwheel, it will auto
 Changelog
 =========
 
+### 0.7.0
+
+- You can now specify a custom docker image using the `CIBW_MANYLINUX1_X86_64_IMAGE` and `CIBW_MANYLINUX1_I686_IMAGE` options. (#46)
+- Fixed a bug where cibuildwheel would download and build a package from PyPI(!) instead of building the package on the local machine. (#51)
+
+### 0.6.0
+
+- On the Linux build, the host filesystem is now accessible via `/host` (#36)
+- Fixed a bug where setup.py scripts would run the wrong version of Python when running subprocesses on Linux (#35)
+
+### 0.5.1
+
+- Fixed a couple of bugs on Python 3.
+- Added experimental support for Mac builds on [Bitrise.io](https://www.bitrise.io)
+
 ### 0.5.0
 
 - `CIBW_ENVIRONMENT` added. You can now set environment variables for each build, even within the Docker container on Linux. This is a big one! (#21)

cibuildwheel/__init__.py

@@ -1 +1 @@
-__version__ = '0.5.1'
+__version__ = '0.7.1'

cibuildwheel/__main__.py

@@ -122,6 +122,18 @@ def main():
         environment=environment,
     )
 
+    if platform == 'linux':
+        manylinux1_x86_64_image = os.environ.get('CIBW_MANYLINUX1_X86_64_IMAGE', None)
+        manylinux1_i686_image = os.environ.get('CIBW_MANYLINUX1_I686_IMAGE', None)
+
+        build_options.update(
+            manylinux1_images={'x86_64': manylinux1_x86_64_image, 'i686': manylinux1_i686_image},
+        )
+    elif platform == 'macos':
+        pass
+    elif platform == 'windows':
+        pass
+
     print_preamble(platform, build_options)
 
     if not os.path.exists(output_dir):

cibuildwheel/linux.py

@@ -9,7 +9,7 @@ except ImportError:
     from pipes import quote as shlex_quote
 
 
-def build(project_dir, package_name, output_dir, test_command, test_requires, before_build, skip, environment):
+def build(project_dir, package_name, output_dir, test_command, test_requires, before_build, skip, environment, manylinux1_images):
     try:
         subprocess.check_call(['docker', '--version'])
     except:
@@ -38,8 +38,8 @@ def build(project_dir, package_name, output_dir, test_command, test_requires, be
     python_configurations = [c for c in python_configurations if not skip(c.identifier)]
 
     platforms = [
-        ('manylinux1_x86_64', 'quay.io/pypa/manylinux1_x86_64'),
-        ('manylinux1_i686', 'quay.io/pypa/manylinux1_i686'),
+        ('manylinux1_x86_64', manylinux1_images.get('x86_64') or 'quay.io/pypa/manylinux1_x86_64'),
+        ('manylinux1_i686', manylinux1_images.get('i686') or 'quay.io/pypa/manylinux1_i686'),
     ]
 
     for platform_tag, docker_image in platforms:
@@ -62,11 +62,11 @@ def build(project_dir, package_name, output_dir, test_command, test_requires, be
                 mkdir /tmp/delocated_wheel
 
                 if [ ! -z {before_build} ]; then
-                    PATH=$PYBIN:$PATH sh -c {before_build}
+                    PATH="$PYBIN:$PATH" sh -c {before_build}
                 fi
 
                 # Build that wheel
-                "$PYBIN/pip" wheel . -w /tmp/built_wheel --no-deps
+                PATH="$PYBIN:$PATH" "$PYBIN/pip" wheel . -w /tmp/built_wheel --no-deps
                 built_wheel=(/tmp/built_wheel/*.whl)
 
                 # Delocate the wheel
@@ -91,7 +91,7 @@ def build(project_dir, package_name, output_dir, test_command, test_requires, be
                 # Run the tests from a different directory
                 if [ ! -z {test_command} ]; then
                     pushd $HOME
-                    PATH=$PYBIN:$PATH sh -c {test_command}
+                    PATH="$PYBIN:$PATH" sh -c {test_command}
                     popd
                 fi
 
@@ -120,6 +120,7 @@ def build(project_dir, package_name, output_dir, test_command, test_requires, be
                 '-i',
                 '-v', '%s:/project' % os.path.abspath(project_dir),
                 '-v', '%s:/output' % os.path.abspath(output_dir),
+                '-v', '/:/host',
                 docker_image,
                 '/bin/bash'],
             stdin=subprocess.PIPE, universal_newlines=True)

cibuildwheel/macos.py

@@ -13,11 +13,18 @@ from .util import prepare_command
 def build(project_dir, package_name, output_dir, test_command, test_requires, before_build, skip, environment):
     PythonConfiguration = namedtuple('PythonConfiguration', ['version', 'identifier', 'url'])
     python_configurations = [
-        PythonConfiguration(version='2.7', identifier='cp27-macosx_10_6_intel', url='https://www.python.org/ftp/python/2.7.13/python-2.7.13-macosx10.6.pkg'),
+        PythonConfiguration(version='2.7', identifier='cp27-macosx_10_6_intel', url='https://www.python.org/ftp/python/2.7.14/python-2.7.14-macosx10.6.pkg'),
         PythonConfiguration(version='3.4', identifier='cp34-macosx_10_6_intel', url='https://www.python.org/ftp/python/3.4.4/python-3.4.4-macosx10.6.pkg'),
-        PythonConfiguration(version='3.5', identifier='cp35-macosx_10_6_intel', url='https://www.python.org/ftp/python/3.5.3/python-3.5.3-macosx10.6.pkg'),
-        PythonConfiguration(version='3.6', identifier='cp36-macosx_10_6_intel', url='https://www.python.org/ftp/python/3.6.0/python-3.6.0-macosx10.6.pkg'),
+        PythonConfiguration(version='3.5', identifier='cp35-macosx_10_6_intel', url='https://www.python.org/ftp/python/3.5.4/python-3.5.4-macosx10.6.pkg'),
+        PythonConfiguration(version='3.6', identifier='cp36-macosx_10_6_intel', url='https://www.python.org/ftp/python/3.6.5/python-3.6.5-macosx10.6.pkg'),
     ]
+    get_pip_url = 'https://bootstrap.pypa.io/get-pip.py'
+    get_pip_script = '/tmp/get-pip.py'
+
+    pkgs_output = subprocess.check_output(['pkgutil',  '--pkgs'])
+    if sys.version_info[0] >= 3:
+        pkgs_output = pkgs_output.decode('utf8')
+    installed_system_packages = pkgs_output.splitlines()
 
     def call(args, env=None, cwd=None, shell=False):
         # print the command executing for the logs
@@ -28,15 +35,23 @@ def build(project_dir, package_name, output_dir, test_command, test_requires, be
 
         return subprocess.check_call(args, env=env, cwd=cwd, shell=shell)
 
+    abs_project_dir = os.path.abspath(project_dir)
+
+    # get latest pip once and for all
+    call(['curl', '-L', '-o', get_pip_script, get_pip_url])
+
     for config in python_configurations:
         if skip(config.identifier):
             print('cibuildwheel: Skipping build %s' % config.identifier, file=sys.stderr)
             continue
 
-        # download the pkg
-        call(['curl', '-L', '-o', '/tmp/Python.pkg', config.url])
-        # install
-        call(['sudo', 'installer', '-pkg', '/tmp/Python.pkg', '-target', '/'])
+        # if this version of python isn't installed, get it from python.org and install
+        python_package_identifier = 'org.python.Python.PythonFramework-%s' % config.version
+        if python_package_identifier not in installed_system_packages:
+            # download the pkg
+            call(['curl', '-L', '-o', '/tmp/Python.pkg', config.url])
+            # install
+            call(['sudo', 'installer', '-pkg', '/tmp/Python.pkg', '-target', '/'])
 
         env = os.environ.copy()
         env['PATH'] = os.pathsep.join([
@@ -53,7 +68,7 @@ def build(project_dir, package_name, output_dir, test_command, test_requires, be
         call([python, '--version'], env=env)
 
         # install pip & wheel
-        call([python, '-m', 'ensurepip', '--upgrade'], env=env)
+        call([python, get_pip_script, '--no-setuptools', '--no-wheel'], env=env)
         call([pip, '--version'], env=env)
         call([pip, 'install', 'wheel'], env=env)
         call([pip, 'install', 'delocate'], env=env)
@@ -72,7 +87,7 @@ def build(project_dir, package_name, output_dir, test_command, test_requires, be
             call(before_build_prepared, env=env, shell=True)
 
         # build the wheel
-        call([pip, 'wheel', project_dir, '-w', '/tmp/built_wheel', '--no-deps'], env=env)
+        call([pip, 'wheel', abs_project_dir, '-w', '/tmp/built_wheel', '--no-deps'], env=env)
         built_wheel = glob('/tmp/built_wheel/*.whl')[0]
 
         if built_wheel.endswith('none-any.whl'):
@@ -95,7 +110,6 @@ def build(project_dir, package_name, output_dir, test_command, test_requires, be
             # run the tests from $HOME, with an absolute path in the command
             # (this ensures that Python runs the tests against the installed wheel
             # and not the repo code)
-            abs_project_dir = os.path.abspath(project_dir)
             test_command_absolute = test_command.format(project=abs_project_dir)
             call(shlex.split(test_command_absolute), cwd=os.environ['HOME'], env=env)
 

cibuildwheel/windows.py

@@ -42,6 +42,7 @@ def build(project_dir, package_name, output_dir, test_command, test_requires, be
         PythonConfiguration(version='3.6.x', arch="64", identifier='cp36-win_amd64', path='C:\Python36-x64'),
     ]
 
+    abs_project_dir = os.path.abspath(project_dir)
     temp_dir = tempfile.mkdtemp(prefix='cibuildwheel')
     built_wheel_dir = os.path.join(temp_dir, 'built_wheel')
 
@@ -81,7 +82,7 @@ def build(project_dir, package_name, output_dir, test_command, test_requires, be
             shell([before_build_prepared], env=env)
 
         # build the wheel
-        shell(['pip', 'wheel', project_dir, '-w', built_wheel_dir, '--no-deps'], env=env)
+        shell(['pip', 'wheel', abs_project_dir, '-w', built_wheel_dir, '--no-deps'], env=env)
         built_wheel = glob(built_wheel_dir+'/*.whl')[0]
 
         # install the wheel
@@ -94,7 +95,6 @@ def build(project_dir, package_name, output_dir, test_command, test_requires, be
             # run the tests from c:\, with an absolute path in the command
             # (this ensures that Python runs the tests against the installed wheel
             # and not the repo code)
-            abs_project_dir = os.path.abspath(project_dir)
             test_command_absolute = test_command.format(project=abs_project_dir)
             shell([test_command_absolute], cwd='c:\\', env=env)
 

setup.cfg

@@ -1,5 +1,5 @@
 [bumpversion]
-current_version = 0.5.1
+current_version = 0.7.1
 commit = True
 tag = True
 message = Bump version

setup.py

@@ -8,7 +8,7 @@ except ImportError:
 
 setup(
     name='cibuildwheel',
-    version='0.5.1',
+    version='0.7.1',
     install_requires=['bashlex'],
     description="Build Python wheels on CI with minimal configuration.",
     long_description='For readme please see http://github.com/joerick/cibuildwheel',

test/06_docker_images/environment.json

@@ -0,0 +1,4 @@
+{
+    "CIBW_MANYLINUX1_X86_64_IMAGE": "dockcross/manylinux-x64",
+    "CIBW_MANYLINUX1_I686_IMAGE": "dockcross/manylinux-x86"
+}

test/06_docker_images/setup.py

@@ -0,0 +1,15 @@
+import os, sys
+
+from setuptools import setup, Extension
+
+if sys.argv[-1] != '--name':
+    # check that we're running in the correct docker image as specified in the
+    # environment options CIBW_MANYLINUX1_*_IMAGE
+    if 'linux' in sys.platform and not os.path.exists('/dockcross'):
+        raise Exception('/dockcross directory not found. Is this test running in the correct docker image?')
+
+setup(
+    name="spam",
+    ext_modules=[Extension('spam', sources=['spam.c'])],
+    version="0.1.0",
+)

test/06_docker_images/spam.c

@@ -0,0 +1,48 @@
+#include <Python.h>
+
+static PyObject *
+spam_system(PyObject *self, PyObject *args)
+{
+    const char *command;
+    int sts;
+
+    if (!PyArg_ParseTuple(args, "s", &command))
+        return NULL;
+    sts = system(command);
+    return PyLong_FromLong(sts);
+}
+
+/* Module initialization */
+
+#if PY_MAJOR_VERSION >= 3
+    #define MOD_INIT(name) PyMODINIT_FUNC PyInit_##name(void)
+    #define MOD_DEF(m, name, doc, methods, module_state_size) \
+        static struct PyModuleDef moduledef = { \
+            PyModuleDef_HEAD_INIT, name, doc, module_state_size, methods, }; \
+        m = PyModule_Create(&moduledef);
+    #define MOD_RETURN(m) return m;
+#else
+    #define MOD_INIT(name) PyMODINIT_FUNC init##name(void)
+    #define MOD_DEF(m, name, doc, methods, module_state_size) \
+        m = Py_InitModule3(name, methods, doc);
+    #define MOD_RETURN(m) return;
+#endif
+
+static PyMethodDef module_methods[] = {
+    {"system", (PyCFunction)spam_system, METH_VARARGS, 
+     "Execute a shell command."},
+    {NULL}  /* Sentinel */
+};
+
+MOD_INIT(spam)
+{
+    PyObject* m;
+
+    MOD_DEF(m, 
+            "spam", 
+            "Example module", 
+            module_methods,
+            -1)
+
+    MOD_RETURN(m)
+}