docs: drop double install and add a few projects (#520)

* docs: project update

* docs: pyproject.toml mention

* Apply suggestions from code review

Co-authored-by: Joe Rickerby <joerick@mac.com>

* docs: one more change from @joerick, dropped double install example

Co-authored-by: Joe Rickerby <joerick@mac.com>

README.md

@@ -135,11 +135,13 @@ Here are some repos that use cibuildwheel.
 | [websockets][]                    | ![travisci icon][] | ![apple icon][] ![linux icon][] | Library for building WebSocket servers and clients. Mostly written in Python, with a small C 'speedups' extension module.  |
 | [pyzmq][]                         | ![github icon][] | ![windows icon][] ![apple icon][] ![linux icon][] | Python bindings for zeromq, the networking library. Uses Cython and CFFI.  |
 | [aiortc][]                        | ![github icon][] | ![apple icon][] ![linux icon][] | WebRTC and ORTC implementation for Python using asyncio. |
-| [h5py][]                          | ![azurepipelines icon][] | ![windows icon][] ![apple icon][] ![linux icon][] | HDF5 for Python -- The h5py package is a Pythonic interface to the HDF5 binary data format. |
+| [numexpr][]                       | ![github icon][] ![travisci icon][] | ![windows icon][] ![apple icon][] ![linux icon][] | Fast numerical array expression evaluator for Python, NumPy, PyTables, pandas, bcolz and more |
 | [coverage.py][]                   | ![github icon][] | ![windows icon][] ![apple icon][] ![linux icon][] | The coverage tool for Python |
+| [h5py][]                          | ![azurepipelines icon][] | ![windows icon][] ![apple icon][] ![linux icon][] | HDF5 for Python -- The h5py package is a Pythonic interface to the HDF5 binary data format. |
 | [River][]                         | ![github icon][] | ![windows icon][] ![apple icon][] ![linux icon][] | 🌊 Online machine learning in Python |
 | [PyAV][]                          | ![github icon][] | ![windows icon][] ![apple icon][] ![linux icon][] | Pythonic bindings for FFmpeg's libraries. |
 | [Dependency Injector][]           | ![travisci icon][] | ![windows icon][] ![apple icon][] ![linux icon][] | Dependency injection framework for Python, uses Windows TravisCI |
+| [PyTables][]                      | ![github icon][] | ![windows icon][] ![apple icon][] ![linux icon][] | A Python package to manage extremely large amounts of data |
 | [aioquic][]                       | ![github icon][] | ![windows icon][] ![apple icon][] ![linux icon][] | QUIC and HTTP/3 implementation in Python |
 | [google neuroglancer][]           | ![github icon][] | ![windows icon][] ![apple icon][] ![linux icon][] | WebGL-based viewer for volumetric data |
 | [AutoPy][]                        | ![travisci icon][] | ![windows icon][] ![apple icon][] ![linux icon][] | Includes a Windows Travis build. |
@@ -147,11 +149,15 @@ Here are some repos that use cibuildwheel.
 | [Parselmouth][]                   | ![github icon][] | ![windows icon][] ![apple icon][] ![linux icon][] | A Python interface to the Praat software package, using pybind11, C++17 and CMake, with the core Praat static library built only once and shared between wheels. |
 | [python-rapidjson][]              | ![travisci icon][] ![gitlab icon][] ![appveyor icon][] | ![windows icon][] ![linux icon][] | Python wrapper around rapidjson |
 | [Rtree][]                         | ![github icon][] | ![windows icon][] ![apple icon][] ![linux icon][] | Rtree: spatial index for Python GIS ¶ |
+| [cyvcf2][]                        | ![github icon][] | ![apple icon][] ![linux icon][] | cython + htslib == fast VCF and BCF processing |
 | [KDEpy][]                         | ![github icon][] | ![windows icon][] ![apple icon][] ![linux icon][] | Kernel Density Estimation in Python |
-| [pybind11 python_example][]       | ![github icon][] | ![windows icon][] ![apple icon][] ![linux icon][] | Example pybind11 module built with a Python-based build system |
 | [pybind11 cmake_example][]        | ![github icon][] | ![windows icon][] ![apple icon][] ![linux icon][] | Example pybind11 module built with a CMake-based build system |
+| [pybind11 python_example][]       | ![github icon][] | ![windows icon][] ![apple icon][] ![linux icon][] | Example pybind11 module built with a Python-based build system |
+| [dd-trace-py][]                   | ![github icon][] | ![windows icon][] ![apple icon][] ![linux icon][] | Uses custom alternate arch emulation on GitHub |
 | [iminuit][]                       | ![github icon][] | ![windows icon][] ![apple icon][] ![linux icon][] | Jupyter-friendly Python interface for C++ MINUIT2 |
 | [jq.py][]                         | ![travisci icon][] | ![apple icon][] ![linux icon][] | Python bindings for jq |
+| [matrixprofile][]                 | ![travisci icon][] | ![windows icon][] ![apple icon][] ![linux icon][] | A Python 3 library making time series data mining tasks, utilizing matrix profile algorithms, accessible to everyone. |
+| [Tokenizer][]                     | ![github icon][] ![travisci icon][] | ![apple icon][] ![linux icon][] | Fast and customizable text tokenization library with BPE and SentencePiece support |
 | [bx-python][]                     | ![travisci icon][] | ![apple icon][] ![linux icon][] | A library that includes Cython extensions. |
 | [boost-histogram][]               | ![github icon][] ![travisci icon][] | ![windows icon][] ![apple icon][] ![linux icon][] | Supports full range of wheels, including PyPy and alternate archs. |
 | [pybase64][]                      | ![github icon][] | ![windows icon][] ![apple icon][] ![linux icon][] | Fast Base64 encoding/decoding in Python |
@@ -172,11 +178,13 @@ Here are some repos that use cibuildwheel.
 [websockets]: https://github.com/aaugustin/websockets
 [pyzmq]: https://github.com/zeromq/pyzmq
 [aiortc]: https://github.com/aiortc/aiortc
-[h5py]: https://github.com/h5py/h5py
+[numexpr]: https://github.com/pydata/numexpr
 [coverage.py]: https://github.com/nedbat/coveragepy
+[h5py]: https://github.com/h5py/h5py
 [River]: https://github.com/online-ml/river
 [PyAV]: https://github.com/PyAV-Org/PyAV
 [Dependency Injector]: https://github.com/ets-labs/python-dependency-injector
+[PyTables]: https://github.com/PyTables/PyTables
 [aioquic]: https://github.com/aiortc/aioquic
 [google neuroglancer]: https://github.com/google/neuroglancer
 [AutoPy]: https://github.com/autopilot-rs/autopy
@@ -184,11 +192,15 @@ Here are some repos that use cibuildwheel.
 [Parselmouth]: https://github.com/YannickJadoul/Parselmouth
 [python-rapidjson]: https://github.com/python-rapidjson/python-rapidjson
 [Rtree]: https://github.com/Toblerity/rtree
+[cyvcf2]: https://github.com/brentp/cyvcf2
 [KDEpy]: https://github.com/tommyod/KDEpy
-[pybind11 python_example]: https://github.com/pybind/python_example
 [pybind11 cmake_example]: https://github.com/pybind/cmake_example
+[pybind11 python_example]: https://github.com/pybind/python_example
+[dd-trace-py]: https://github.com/DataDog/dd-trace-py
 [iminuit]: https://github.com/scikit-hep/iminuit
 [jq.py]: https://github.com/mwilliamson/jq.py
+[matrixprofile]: https://github.com/matrix-profile-foundation/matrixprofile
+[Tokenizer]: https://github.com/OpenNMT/Tokenizer
 [bx-python]: https://github.com/bxlab/bx-python
 [boost-histogram]: https://github.com/scikit-hep/boost-histogram
 [pybase64]: https://github.com/mayeut/pybase64
@@ -211,42 +223,48 @@ Here are some repos that use cibuildwheel.
 [apple icon]: docs/data/readme_icons/apple.svg
 [linux icon]: docs/data/readme_icons/linux.svg
 
-<!-- scikit-learn: 43477, last pushed 0 days ago -->
+<!-- scikit-learn: 43621, last pushed 0 days ago -->
-<!-- Matplotlib: 12823, last pushed 0 days ago -->
+<!-- Matplotlib: 12886, last pushed 0 days ago -->
-<!-- psutil: 6887, last pushed 0 days ago -->
+<!-- psutil: 6933, last pushed 1 days ago -->
-<!-- twisted-iocpsupport: 4114, last pushed 2 days ago -->
+<!-- twisted-iocpsupport: 4128, last pushed 1 days ago -->
-<!-- scikit-image: 4104, last pushed 0 days ago -->
+<!-- scikit-image: 4123, last pushed 0 days ago -->
-<!-- websockets: 3089, last pushed 10 days ago -->
+<!-- websockets: 3102, last pushed 3 days ago -->
-<!-- pyzmq: 2654, last pushed 20 days ago -->
+<!-- pyzmq: 2662, last pushed 6 days ago -->
-<!-- aiortc: 2073, last pushed 7 days ago -->
+<!-- aiortc: 2099, last pushed 6 days ago -->
-<!-- h5py: 1455, last pushed 12 days ago -->
+<!-- numexpr: 1504, last pushed 1 days ago -->
-<!-- coverage.py: 1441, last pushed 3 days ago -->
+<!-- coverage.py: 1461, last pushed 0 days ago -->
-<!-- River: 1219, last pushed 0 days ago -->
+<!-- h5py: 1460, last pushed 0 days ago -->
-<!-- PyAV: 1114, last pushed 14 days ago -->
+<!-- River: 1250, last pushed 0 days ago -->
-<!-- Dependency Injector: 1030, last pushed 0 days ago -->
+<!-- PyAV: 1122, last pushed 10 days ago -->
-<!-- aioquic: 552, last pushed 49 days ago -->
+<!-- Dependency Injector: 1050, last pushed 1 days ago -->
-<!-- google neuroglancer: 550, last pushed 7 days ago -->
+<!-- PyTables: 998, last pushed 0 days ago -->
-<!-- AutoPy: 507, last pushed 105 days ago -->
+<!-- aioquic: 565, last pushed 62 days ago -->
-<!-- pikepdf: 487, last pushed 0 days ago -->
+<!-- google neuroglancer: 551, last pushed 21 days ago -->
-<!-- Parselmouth: 433, last pushed 26 days ago -->
+<!-- AutoPy: 513, last pushed 119 days ago -->
-<!-- python-rapidjson: 407, last pushed 10 days ago -->
+<!-- pikepdf: 502, last pushed 0 days ago -->
-<!-- Rtree: 378, last pushed 0 days ago -->
+<!-- Parselmouth: 438, last pushed 0 days ago -->
-<!-- KDEpy: 230, last pushed 9 days ago -->
+<!-- python-rapidjson: 408, last pushed 23 days ago -->
-<!-- pybind11 python_example: 221, last pushed 44 days ago -->
+<!-- Rtree: 380, last pushed 12 days ago -->
-<!-- pybind11 cmake_example: 219, last pushed 7 days ago -->
+<!-- cyvcf2: 231, last pushed 0 days ago -->
-<!-- iminuit: 166, last pushed 0 days ago -->
+<!-- KDEpy: 230, last pushed 22 days ago -->
-<!-- jq.py: 138, last pushed 73 days ago -->
+<!-- pybind11 cmake_example: 224, last pushed 20 days ago -->
-<!-- bx-python: 94, last pushed 1 days ago -->
+<!-- pybind11 python_example: 221, last pushed 57 days ago -->
-<!-- boost-histogram: 63, last pushed 0 days ago -->
+<!-- dd-trace-py: 206, last pushed 0 days ago -->
-<!-- pybase64: 52, last pushed 2 days ago -->
+<!-- iminuit: 166, last pushed 9 days ago -->
-<!-- TgCrypto: 49, last pushed 36 days ago -->
+<!-- jq.py: 139, last pushed 86 days ago -->
-<!-- etebase-py: 40, last pushed 3 days ago -->
+<!-- matrixprofile: 119, last pushed 10 days ago -->
-<!-- pyjet: 27, last pushed 29 days ago -->
+<!-- Tokenizer: 114, last pushed 6 days ago -->
-<!-- numpythia: 23, last pushed 28 days ago -->
+<!-- bx-python: 94, last pushed 14 days ago -->
-<!-- fathon: 18, last pushed 1 days ago -->
+<!-- boost-histogram: 63, last pushed 1 days ago -->
-<!-- pyinstrument_cext: 10, last pushed 29 days ago -->
+<!-- pybase64: 52, last pushed 0 days ago -->
-<!-- xmlstarlet: 7, last pushed 0 days ago -->
+<!-- TgCrypto: 49, last pushed 49 days ago -->
-<!-- pybind11 scikit_build_example: 0, last pushed 42 days ago -->
+<!-- etebase-py: 40, last pushed 1 days ago -->
+<!-- pyjet: 27, last pushed 42 days ago -->
+<!-- numpythia: 24, last pushed 41 days ago -->
+<!-- fathon: 19, last pushed 1 days ago -->
+<!-- pyinstrument_cext: 10, last pushed 42 days ago -->
+<!-- xmlstarlet: 7, last pushed 1 days ago -->
+<!-- pybind11 scikit_build_example: 2, last pushed 3 days ago -->
 
 <!-- END bin/projects.py -->
 

bin/projects.py

@@ -20,7 +20,7 @@ from typing import Any, Dict, List, Optional, TextIO
 
 import click
 import yaml
-from github import Github
+from github import Github, GithubException
 
 ICONS = (
     "appveyor",
@@ -53,7 +53,12 @@ class Project:
 
         self.online = github is not None
         if github is not None:
-            repo = github.get_repo(self.stars_repo)
+            try:
+                repo = github.get_repo(self.stars_repo)
+            except GithubException:
+                print(f"Broken: {self.stars_repo}")
+                raise
+
             self.num_stars = repo.stargazers_count
             self.pushed_at = repo.pushed_at
             if not self.notes:
@@ -167,7 +172,7 @@ def str_projects(
     return io.getvalue()
 
 
-@click.command()
+@click.command(help="Try ./bin/projects.py docs/data/projects.yml --readme README.md")
 @click.argument("input", type=click.File("r"))
 @click.option("--online/--no-online", default=True, help="Get info from GitHub")
 @click.option("--auth", help="GitHub authentication token")

docs/data/projects.yml

@@ -229,3 +229,34 @@
   ci: [travisci]
   os: [windows, apple, linux]
   notes: Dependency injection framework for Python, uses Windows TravisCI
+
+- name: numexpr
+  gh: pydata/numexpr
+  ci: [github, travisci]
+  os: [windows, apple, linux]
+
+- name: PyTables
+  gh: PyTables/PyTables
+  ci: [github]
+  os: [windows, apple, linux]
+
+- name: dd-trace-py
+  gh: DataDog/dd-trace-py
+  ci: [github]
+  os: [windows, apple, linux]
+  notes: Uses custom alternate arch emulation on GitHub
+
+- name: Tokenizer
+  gh: OpenNMT/Tokenizer
+  ci: [github, travisci]
+  os: [apple, linux]
+
+- name: matrixprofile
+  gh: matrix-profile-foundation/matrixprofile
+  ci: [travisci]
+  os: [windows, apple, linux]
+
+- name: cyvcf2
+  gh: brentp/cyvcf2
+  ci: [github]
+  os: [apple, linux]

docs/options.md

@@ -243,7 +243,7 @@ CIBW_BEFORE_ALL_LINUX: yum install -y libffi-dev
 
 A shell command to run before building the wheel. This option allows you to run a command in **each** Python environment before the `pip wheel` command. This is useful if you need to set up some dependency so it's available during the build.
 
-If dependencies are required to build your wheel (for example if you include a header from a Python module), set this to `pip install .`, and the dependencies will be installed automatically by pip. However, this means your package will be built twice - if your package takes a long time to build, you might wish to manually list the dependencies here instead.
+If dependencies are required to build your wheel (for example if you include a header from a Python module), instead of using this command, we recommend adding requirements to a pyproject.toml file. This is reproducible, and users who do not get your wheels (such as Alpine or ClearLinux users) will still benefit.
 
 The active Python binary can be accessed using `python`, and pip with `pip`; `cibuildwheel` makes sure the right version of Python and pip will be executed. The placeholder `{package}` can be used here; it will be replaced by the path to the package being built by `cibuildwheel`.
 
@@ -254,22 +254,47 @@ Platform-specific variants also available:<br/>
 
 #### Examples
 ```yaml
-# install your project and dependencies before building
+# install something required for the build (you might want to use pyproject.toml instead)
-CIBW_BEFORE_BUILD: pip install .
-
-# install something required for the build
 CIBW_BEFORE_BUILD: pip install pybind11
 
 # chain commands using &&
-CIBW_BEFORE_BUILD: yum install -y libffi-dev && pip install .
+CIBW_BEFORE_BUILD_LINUX: yum install -y libffi-dev && make clean
 
 # run a script that's inside your project
 CIBW_BEFORE_BUILD: bash scripts/prepare_for_build.sh
 
 # if cibuildwheel is called with a package_dir argument, it's available as {package}
-CIBW_BEFORE_BUILD: "{package}/bin/prepare_for_build.sh"
+CIBW_BEFORE_BUILD: "{package}/script/prepare_for_build.sh"
 ```
 
+!!! note
+    If you need dependencies installed for the build, we recommend using pyproject.toml. This is an example pyproject.toml file:
+
+    ```toml
+    [build-system]
+    requires = [
+        "setuptools>=42",
+        "wheel",
+        "Cython",
+        "numpy==1.11.3; python_version<='3.6'",
+        "numpy==1.14.5; python_version=='3.7'",
+        "numpy==1.17.3; python_version=='3.8'",
+        "numpy==1.19.4; python_version>='3.9'",
+    ]
+
+    build-backend = "setuptools.build_meta"
+    ```
+
+    This [PEP 517][]/[PEP 518][] style build allows you to completely control the
+    build environment in cibuildwheel, [PyPA-build][], and pip, doesn't force
+    downstream users to install anything they don't need, and lets you do more
+    complex pinning (Cython, for example, requires a wheel to be built with an
+    equal or earlier version of NumPy; pinning in this way is the only way to
+    ensure your module works on all available NumPy versions).
+
+    [PyPA-build]: https://pypa-build.readthedocs.io/en/latest/
+    [PEP 517]: https://www.python.org/dev/peps/pep-0517/
+    [PEP 518]: https://www.python.org/dev/peps/pep-0517/
 
 ### `CIBW_REPAIR_WHEEL_COMMAND` {: #repair-wheel-command}
 > Execute a shell command to repair each (non-pure Python) built wheel