Merge pull request #180 from mayeut/nuget

Adding Python 3.8, Always use nuget to install python on Windows

README.md

@@ -12,12 +12,13 @@ Python wheels are great. Building them across **Mac, Linux, Windows**, on **mult
 What does it do?
 ----------------
 
-|   | macOS 10.6+ | manylinux i686 | manylinux x86_64 |  Windows 32bit | Windows 64bit |
+|   | macOS 10.6+ intel | macOS 10.9+ x86_64 | manylinux i686 | manylinux x86_64 |  Windows 32bit | Windows 64bit |
-|---|---|---|---|---|---|
+|---|---|---|---|---|---|---|
-| Python 2.7 | ✅ | ✅ | ✅ | ✅¹ | ✅¹ |
+| Python 2.7 | ✅ |    | ✅ | ✅ | ✅¹ | ✅¹ |
-| Python 3.5 | ✅ | ✅ | ✅ | ✅  | ✅  |
+| Python 3.5 | ✅ |    | ✅ | ✅ | ✅  | ✅  |
-| Python 3.6 | ✅ | ✅ | ✅ | ✅  | ✅  |
+| Python 3.6 | ✅ |    | ✅ | ✅ | ✅  | ✅  |
-| Python 3.7 | ✅ | ✅ | ✅ | ✅  | ✅  |
+| Python 3.7 | ✅ |    | ✅ | ✅ | ✅  | ✅  |
+| Python 3.8 |    | ✅ | ✅ | ✅ | ✅  | ✅  |
 
 > ¹ Not supported on Travis
 
@@ -76,14 +77,7 @@ jobs:
 - job: windows
   pool: {vmImage: 'vs2017-win2016'}
   steps:
-    - {task: UsePythonVersion@0, inputs: {versionSpec: '2.7', architecture: x86}}
+    - task: UsePythonVersion@0
-    - {task: UsePythonVersion@0, inputs: {versionSpec: '2.7', architecture: x64}}
-    - {task: UsePythonVersion@0, inputs: {versionSpec: '3.5', architecture: x86}}
-    - {task: UsePythonVersion@0, inputs: {versionSpec: '3.5', architecture: x64}}
-    - {task: UsePythonVersion@0, inputs: {versionSpec: '3.6', architecture: x86}}
-    - {task: UsePythonVersion@0, inputs: {versionSpec: '3.6', architecture: x64}}
-    - {task: UsePythonVersion@0, inputs: {versionSpec: '3.7', architecture: x86}}
-    - {task: UsePythonVersion@0, inputs: {versionSpec: '3.7', architecture: x64}}
     - script: choco install vcpython27 -f -y
       displayName: Install Visual C++ for Python 2.7
     - bash: |
@@ -309,9 +303,9 @@ When both options are specified, both conditions are applied and only builds wit
 
 The format is `python_tag-platform_tag`. The tags are similar but not identical to the ones defined in [PEP 425](https://www.python.org/dev/peps/pep-0425/#details).
 
-Python tags look like `cp27` `cp35` `cp36` `cp37`
+Python tags look like `cp27` `cp35` `cp36` `cp37` `cp38`
 
-Platform tags look like `macosx_10_6_intel` `manylinux_x86_64` `manylinux_i686` `win32` `win_amd64`
+Platform tags look like `macosx_10_6_intel` `macosx_10_9_x86_64` `manylinux_x86_64` `manylinux_i686` `win32` `win_amd64`
 
 You can also use shell-style globbing syntax (as per `fnmatch`).
 
@@ -320,6 +314,7 @@ The list of supported and currently selected build identifiers can be retrieved
 Examples:
 - Only build on Python 3.6: `CIBW_BUILD`:`cp36-*`
 - Skip building on Python 2.7 on the Mac: `CIBW_SKIP`:`cp27-macosx_10_6_intel`
+- Skip building on Python 3.8 on the Mac: `CIBW_SKIP`:`cp38-macosx_10_9_x86_64`
 - Skip building on Python 2.7 on all platforms: `CIBW_SKIP`:`cp27-*`
 - Skip Python 2.7 on Windows: `CIBW_SKIP`:`cp27-win*`
 - Skip Python 2.7 on 32-bit Windows: `CIBW_SKIP`:`cp27-win32`
@@ -352,10 +347,10 @@ You must set this variable to pass variables to Linux builds (since they execute
 
 You can use `$PATH` syntax to insert other variables, or the `$(pwd)` syntax to insert the output of other shell commands.
 
-Example: `CFLAGS="-g -Wall" CXXFLAGS="-Wall"`
+Example: `CFLAGS="-g -Wall" CXXFLAGS="-Wall"`\
-Example: `PATH=$PATH:/usr/local/bin`
+Example: `PATH=$PATH:/usr/local/bin`\
-Example: `BUILD_TIME="$(date)"`
+Example: `BUILD_TIME="$(date)"`\
-Example: `PIP_EXTRA_INDEX_URL="https://pypi.myorg.com/simple"`
+Example: `PIP_EXTRA_INDEX_URL="https://pypi.myorg.com/simple"`\
 
 Platform-specific variants also available:
 `CIBW_ENVIRONMENT_MACOS` | `CIBW_ENVIRONMENT_WINDOWS` | `CIBW_ENVIRONMENT_LINUX`
@@ -375,11 +370,11 @@ If dependencies are required to build your wheel (for example if you include a h
 
 The active Python binary can be accessed using `python`, and pip with `pip`; `cibuildwheel` makes sure the right version of Python and pip will be executed. `{project}` can be used as a placeholder for the absolute path to the project's root.
 
-Example: `pip install .`
+Example: `pip install .`\
-Example: `pip install pybind11`
+Example: `pip install pybind11`\
 Example: `yum install -y libffi-dev && pip install .`
 
-Platform-specific variants also available:
+Platform-specific variants also available:\
  `CIBW_BEFORE_BUILD_MACOS` | `CIBW_BEFORE_BUILD_WINDOWS` | `CIBW_BEFORE_BUILD_LINUX`
 
 ***
@@ -397,8 +392,8 @@ Beware to specify a valid Docker image that can be used in the same way as the o
 
 Note that `auditwheel` detects the version of the `manylinux` standard in the Docker image through the `AUDITWHEEL_PLAT` environment variable, as `cibuildwheel` has no way of detecting the correct `--plat` command line argument to pass to `auditwheel` for a custom image. If a Docker image does not correctly set this `AUDITWHEEL_PLAT` environment variable, the `CIBW_ENVIRONMENT` option can be used to do so (e.g., `CIBW_ENVIRONMENT="manylinux2010_$(uname -m)"`).
 
-Example: `manylinux1`
+Example: `manylinux1`\
-Example: `dockcross/manylinux-x64`
+Example: `dockcross/manylinux-x64`\
 Example: `dockcross/manylinux-x86`
 
 ***
@@ -426,7 +421,7 @@ Optional.
 
 Space-separated list of dependencies required for running the tests.
 
-Example: `pytest`
+Example: `pytest`\
 Example: `nose==1.3.7 moto==0.4.31`
 
 Platform-specific variants also available:

azure-pipelines.yml

@@ -1,6 +1,6 @@
 jobs:
 - job: linux
-  pool: {vmImage: 'Ubuntu-16.04'}
+  pool: {vmImage: 'Ubuntu-18.04'}
   steps:
     - task: UsePythonVersion@0
     - bash: |
@@ -18,14 +18,7 @@ jobs:
 - job: windows
   pool: {vmImage: 'vs2017-win2016'}
   steps:
-    - {task: UsePythonVersion@0, inputs: {versionSpec: '2.7', architecture: x86}}
+    - task: UsePythonVersion@0
-    - {task: UsePythonVersion@0, inputs: {versionSpec: '2.7', architecture: x64}}
-    - {task: UsePythonVersion@0, inputs: {versionSpec: '3.5', architecture: x86}}
-    - {task: UsePythonVersion@0, inputs: {versionSpec: '3.5', architecture: x64}}
-    - {task: UsePythonVersion@0, inputs: {versionSpec: '3.6', architecture: x86}}
-    - {task: UsePythonVersion@0, inputs: {versionSpec: '3.6', architecture: x64}}
-    - {task: UsePythonVersion@0, inputs: {versionSpec: '3.7', architecture: x86}}
-    - {task: UsePythonVersion@0, inputs: {versionSpec: '3.7', architecture: x64}}
     - script: choco install vcpython27 -f -y
       displayName: Install Visual C++ for Python 2.7
     - bash: |

cibuildwheel/linux.py

@@ -17,11 +17,13 @@ def get_python_configurations(build_selector):
         PythonConfiguration(identifier='cp35-manylinux_x86_64', path='/opt/python/cp35-cp35m'),
         PythonConfiguration(identifier='cp36-manylinux_x86_64', path='/opt/python/cp36-cp36m'),
         PythonConfiguration(identifier='cp37-manylinux_x86_64', path='/opt/python/cp37-cp37m'),
+        PythonConfiguration(identifier='cp38-manylinux_x86_64', path='/opt/python/cp38-cp38'),
         PythonConfiguration(identifier='cp27-manylinux_i686', path='/opt/python/cp27-cp27m'),
         PythonConfiguration(identifier='cp27-manylinux_i686', path='/opt/python/cp27-cp27mu'),
         PythonConfiguration(identifier='cp35-manylinux_i686', path='/opt/python/cp35-cp35m'),
         PythonConfiguration(identifier='cp36-manylinux_i686', path='/opt/python/cp36-cp36m'),
         PythonConfiguration(identifier='cp37-manylinux_i686', path='/opt/python/cp37-cp37m'),
+        PythonConfiguration(identifier='cp38-manylinux_i686', path='/opt/python/cp38-cp38'),
     ]
 
     # skip builds as required

cibuildwheel/macos.py

@@ -18,6 +18,7 @@ def get_python_configurations(build_selector):
         PythonConfiguration(version='3.5', identifier='cp35-macosx_10_6_intel', url='https://www.python.org/ftp/python/3.5.4/python-3.5.4-macosx10.6.pkg'),
         PythonConfiguration(version='3.6', identifier='cp36-macosx_10_6_intel', url='https://www.python.org/ftp/python/3.6.8/python-3.6.8-macosx10.6.pkg'),
         PythonConfiguration(version='3.7', identifier='cp37-macosx_10_6_intel', url='https://www.python.org/ftp/python/3.7.5/python-3.7.5-macosx10.6.pkg'),
+        PythonConfiguration(version='3.8', identifier='cp38-macosx_10_9_x86_64', url='https://www.python.org/ftp/python/3.8.0/python-3.8.0-macosx10.9.pkg'),
     ]
 
     # skip builds as required

cibuildwheel/windows.py

@@ -8,67 +8,48 @@ try:
 except ImportError:
     from pipes import quote as shlex_quote
 
+try:
+    from urllib.request import urlopen
+except ImportError:
+    from urllib2 import urlopen
+
 from .util import prepare_command, get_build_verbosity_extra_flags
 
 
 IS_RUNNING_ON_AZURE = os.path.exists('C:\\hostedtoolcache')
 IS_RUNNING_ON_TRAVIS = os.environ.get('TRAVIS_OS_NAME') == 'windows'
 
+
 def get_python_path(config):
-    if IS_RUNNING_ON_AZURE:
+    nuget_args = get_nuget_args(config)
-        # We can't hard-code the paths because on Azure, we don't know which
+    return os.path.join(nuget_args[-1], nuget_args[0] + "." + config.version, "tools")
-        # bugfix release of Python we are getting so we need to check which
-        # ones exist. We just use the first one that is found since there should
-        # only be one.
-        path_pattern = 'C:\\hostedtoolcache\\windows\\Python\\{version}\\{arch}'.format(
-            version=config.version.replace('x', '*'),
-            arch='x86' if config.arch == '32' else 'x64'
-        )
-        try:
-            return glob(path_pattern)[0]
-        except IndexError:
-            raise Exception('Could not find a Python install at ' + path_pattern)
-    elif IS_RUNNING_ON_TRAVIS:
-        if config.version == "3.4.x":
-            return config.path
-        else:
-            nuget_args = get_nuget_args(config)
-            return os.path.join(nuget_args[-1], nuget_args[0] + "." + config.nuget_version, "tools")
-    else:
-        # Assume we're running on AppVeyor
-        major, minor = config.version.split('.')[:2]
-        return 'C:\\Python{major}{minor}{arch}'.format(
-            major=major,
-            minor=minor,
-            arch = '-x64' if config.arch == '64' else ''
-        )
 
 
 def get_nuget_args(configuration):
-    if configuration.nuget_version is None:
-        return None
     python_name = "python" if configuration.version[0] == '3' else "python2"
     if configuration.arch == "32":
         python_name = python_name + "x86"
-    return [python_name, "-Version", configuration.nuget_version, "-OutputDirectory", "C:/python"]
+    return [python_name, "-Version", configuration.version, "-OutputDirectory", "C:/python"]
 
 def get_python_configurations(build_selector):
-    PythonConfiguration = namedtuple('PythonConfiguration', ['version', 'arch', 'identifier', 'path', "nuget_version"])
+    PythonConfiguration = namedtuple('PythonConfiguration', ['version', 'arch', 'identifier'])
     python_configurations = [
-        PythonConfiguration(version='2.7.x', arch="32", identifier='cp27-win32', path='C:\\Python27', nuget_version="2.7.16"),
+        PythonConfiguration(version='2.7.17', arch="32", identifier='cp27-win32'),
-        PythonConfiguration(version='2.7.x', arch="64", identifier='cp27-win_amd64', path='C:\\Python27-x64', nuget_version="2.7.16"),
+        PythonConfiguration(version='2.7.17', arch="64", identifier='cp27-win_amd64'),
-        PythonConfiguration(version='3.5.x', arch="32", identifier='cp35-win32', path='C:\\Python35', nuget_version="3.5.4"),
+        PythonConfiguration(version='3.5.4', arch="32", identifier='cp35-win32'),
-        PythonConfiguration(version='3.5.x', arch="64", identifier='cp35-win_amd64', path='C:\\Python35-x64', nuget_version="3.5.4"),
+        PythonConfiguration(version='3.5.4', arch="64", identifier='cp35-win_amd64'),
-        PythonConfiguration(version='3.6.x', arch="32", identifier='cp36-win32', path='C:\\Python36', nuget_version="3.6.8"),
+        PythonConfiguration(version='3.6.8', arch="32", identifier='cp36-win32'),
-        PythonConfiguration(version='3.6.x', arch="64", identifier='cp36-win_amd64', path='C:\\Python36-x64', nuget_version="3.6.8"),
+        PythonConfiguration(version='3.6.8', arch="64", identifier='cp36-win_amd64'),
-        PythonConfiguration(version='3.7.x', arch="32", identifier='cp37-win32', path='C:\\Python37', nuget_version="3.7.4"),
+        PythonConfiguration(version='3.7.5', arch="32", identifier='cp37-win32'),
-        PythonConfiguration(version='3.7.x', arch="64", identifier='cp37-win_amd64', path='C:\\Python37-x64', nuget_version="3.7.4")
+        PythonConfiguration(version='3.7.5', arch="64", identifier='cp37-win_amd64'),
+        PythonConfiguration(version='3.8.0', arch="32", identifier='cp38-win32'),
+        PythonConfiguration(version='3.8.0', arch="64", identifier='cp38-win_amd64'),
     ]
 
     if IS_RUNNING_ON_TRAVIS:
         # cannot install VCForPython27.msi which is needed for compiling C software
         # try with (and similar): msiexec /i VCForPython27.msi ALLUSERS=1 ACCEPT=YES /passive
-        python_configurations = [c for c in python_configurations if c.version != '2.7.x']
+        python_configurations = [c for c in python_configurations if not c.version.startswith('2.7.')]
 
      # skip builds as required
     python_configurations = [c for c in python_configurations if build_selector(c.identifier)]
@@ -78,11 +59,20 @@ def get_python_configurations(build_selector):
 
 
 def build(project_dir, output_dir, test_command, test_requires, test_extras, before_build, build_verbosity, build_selector, environment):
+    def simple_shell(args, env=None, cwd=None):
+        print('+ ' + ' '.join(args))
+        args = ['cmd', '/E:ON', '/V:ON', '/C'] + args
+        return subprocess.check_call(' '.join(args), env=env, cwd=cwd)
+    def download(url, dest):
+        print('+ Download ' + url + ' to ' + dest)
+        response = urlopen(url)
+        try:
+            with open(dest, 'wb') as file:
+                file.write(response.read())
+        finally:
+            response.close()
     if IS_RUNNING_ON_AZURE or IS_RUNNING_ON_TRAVIS:
-        def shell(args, env=None, cwd=None):
+        shell = simple_shell
-            print('+ ' + ' '.join(args))
-            args = ['cmd', '/E:ON', '/V:ON', '/C'] + args
-            return subprocess.check_call(' '.join(args), env=env, cwd=cwd)
     else:
         run_with_env = os.path.abspath(os.path.join(os.path.dirname(__file__), 'resources', 'appveyor_run_with_env.cmd'))
 
@@ -98,21 +88,19 @@ def build(project_dir, output_dir, test_command, test_requires, test_extras, bef
     temp_dir = tempfile.mkdtemp(prefix='cibuildwheel')
     built_wheel_dir = os.path.join(temp_dir, 'built_wheel')
 
-    if IS_RUNNING_ON_TRAVIS:
+    # install nuget as best way to provide python
-        # instal nuget as best way for provide python
+    nuget = 'C:\\nuget.exe'
-        shell(["choco", "install", "nuget.commandline"])
+    download('https://dist.nuget.org/win-x86-commandline/latest/nuget.exe', nuget)
-        # get pip fo this installation which not have.
+    # get pip fo this installation which not have.
-        get_pip_url = 'https://bootstrap.pypa.io/get-pip.py'
+    get_pip_script = 'C:\\get-pip.py'
-        get_pip_script = 'C:\\get-pip.py'
+    download('https://bootstrap.pypa.io/get-pip.py', get_pip_script)
-        shell(['curl', '-L', '-o', get_pip_script, get_pip_url])
 
     python_configurations = get_python_configurations(build_selector)
     for config in python_configurations:
         config_python_path = get_python_path(config)
-        if IS_RUNNING_ON_TRAVIS and config.nuget_version is not None and not os.path.exists(config_python_path):
+        simple_shell([nuget, "install"] + get_nuget_args(config))
-            shell(["nuget", "install"] + get_nuget_args(config))
+        if not os.path.exists(os.path.join(config_python_path, 'Scripts', 'pip.exe')):
-            if not os.path.exists(os.path.join(config_python_path, 'Scripts', 'pip.exe')):
+            simple_shell([os.path.join(config_python_path, 'python.exe'), get_pip_script ])
-                shell([os.path.join(config_python_path, 'python.exe'), get_pip_script ])
 
         # check python & pip exist for this configuration
         assert os.path.exists(os.path.join(config_python_path, 'python.exe'))
@@ -139,12 +127,7 @@ def build(project_dir, output_dir, test_command, test_requires, test_extras, bef
         shell(['python', '-c', '"import struct; print(struct.calcsize(\'P\') * 8)\"'], env=env)
 
         # prepare the Python environment
-        if config.version == "3.4.x":
+        shell(['python', '-m', 'pip', 'install', '--upgrade', 'pip'], env=env)
-            shell(['python', '-m', 'pip', 'install', 'pip==19.1.1'],
-              env=env)
-        else:
-            shell(['python', '-m', 'pip', 'install', '--upgrade', 'pip'],
-                env=env)
         shell(['pip', 'install', '--upgrade', 'setuptools'], env=env)
         shell(['pip', 'install', 'wheel'], env=env)
 

test/shared/utils.py

@@ -54,21 +54,25 @@ def expected_wheels(package_name, package_version):
             '{package_name}-{package_version}-cp35-cp35m-manylinux1_x86_64.whl',
             '{package_name}-{package_version}-cp36-cp36m-manylinux1_x86_64.whl',
             '{package_name}-{package_version}-cp37-cp37m-manylinux1_x86_64.whl',
+            '{package_name}-{package_version}-cp38-cp38-manylinux1_x86_64.whl',
             '{package_name}-{package_version}-cp27-cp27m-manylinux2010_x86_64.whl',
             '{package_name}-{package_version}-cp27-cp27mu-manylinux2010_x86_64.whl',
             '{package_name}-{package_version}-cp35-cp35m-manylinux2010_x86_64.whl',
             '{package_name}-{package_version}-cp36-cp36m-manylinux2010_x86_64.whl',
             '{package_name}-{package_version}-cp37-cp37m-manylinux2010_x86_64.whl',
+            '{package_name}-{package_version}-cp38-cp38-manylinux2010_x86_64.whl',
             '{package_name}-{package_version}-cp27-cp27m-manylinux1_i686.whl',
             '{package_name}-{package_version}-cp27-cp27mu-manylinux1_i686.whl',
             '{package_name}-{package_version}-cp35-cp35m-manylinux1_i686.whl',
             '{package_name}-{package_version}-cp36-cp36m-manylinux1_i686.whl',
             '{package_name}-{package_version}-cp37-cp37m-manylinux1_i686.whl',
+            '{package_name}-{package_version}-cp38-cp38-manylinux1_i686.whl',
             '{package_name}-{package_version}-cp27-cp27m-manylinux2010_i686.whl',
             '{package_name}-{package_version}-cp27-cp27mu-manylinux2010_i686.whl',
             '{package_name}-{package_version}-cp35-cp35m-manylinux2010_i686.whl',
             '{package_name}-{package_version}-cp36-cp36m-manylinux2010_i686.whl',
             '{package_name}-{package_version}-cp37-cp37m-manylinux2010_i686.whl',
+            '{package_name}-{package_version}-cp38-cp38-manylinux2010_i686.whl',
         ]
     elif platform == 'windows':
         templates = [
@@ -76,10 +80,12 @@ def expected_wheels(package_name, package_version):
             '{package_name}-{package_version}-cp35-cp35m-win32.whl',
             '{package_name}-{package_version}-cp36-cp36m-win32.whl',
             '{package_name}-{package_version}-cp37-cp37m-win32.whl',
+            '{package_name}-{package_version}-cp38-cp38-win32.whl',
             '{package_name}-{package_version}-cp27-cp27m-win_amd64.whl',
             '{package_name}-{package_version}-cp35-cp35m-win_amd64.whl',
             '{package_name}-{package_version}-cp36-cp36m-win_amd64.whl',
             '{package_name}-{package_version}-cp37-cp37m-win_amd64.whl',
+            '{package_name}-{package_version}-cp38-cp38-win_amd64.whl',
         ]
     elif platform == 'macos':
         templates = [
@@ -87,12 +93,13 @@ def expected_wheels(package_name, package_version):
             '{package_name}-{package_version}-cp35-cp35m-macosx_10_6_intel.whl',
             '{package_name}-{package_version}-cp36-cp36m-macosx_10_6_intel.whl',
             '{package_name}-{package_version}-cp37-cp37m-macosx_10_6_intel.whl',
+            '{package_name}-{package_version}-cp38-cp38-macosx_10_9_x86_64.whl',
         ]
     else:
         raise Exception('unsupported platform')
 
     if IS_WINDOWS_RUNNING_ON_TRAVIS:
-        # Python 2.7 and 3.4 isn't supported on Travis.
+        # Python 2.7 isn't supported on Travis.
         templates = [t for t in templates if '-cp27-' not in t]
 
     return [filename.format(package_name=package_name, package_version=package_version)