piwheels stats for 2018

piwheels was launched in mid-2017, and we started counting downloads in November of that year, so now we have over a year’s worth of data on which packages have been downloaded, and how much time has been saved.

If you’re not familiar, piwheels.org is a Python package repository providing natively compiled wheels for the Raspberry Pi’s Arm architecture, making pip installations fast for Raspberry Pi users.

In 2018, a total of 5,154,233 packages were downloaded, saving users 39 years, 10 months and 29 days. Some more stats for the year:

  • Average daily downloads: 14,519
  • Average monthly downloads: 429,519
  • Most downloads in one day: 25,389 on 30 November
  • Most downloads in one month: 662,621 in December (21,445 daily average)

Usage was steady throughout the year, with a rise in the final quarter:

Interestingly, the top 50 download days on record have all been since late October!

Packages

The top 30 most downloaded packages of the year are:

  1. pycparser (203,432)
  2. home-assistant-frontend (157,924)
  3. PyYAML (153,907)
  4. cffi (143,998)
  5. MarkupSafe (138,488)
  6. SQLAlchemy (133,629)
  7. multidict (132,226)
  8. numpy (130,044)
  9. aiohttp (129,322)
  10. future (121,117)
  11. cryptography (109,670)
  12. idna_ssl (109,317)
  13. mutagen (100,491)
  14. user-agents (81,608)
  15. netifaces (79,248)
  16. pycryptodome (78,503)
  17. gTTS-token (75,068)
  18. yarl (73,824)
  19. voluptuous-serialize (71,656)
  20. paho-mqtt (63,925)
  21. bcrypt (58,017)
  22. RPi.GPIO (50,246)
  23. Pillow (48,757)
  24. docopt (43,908)
  25. psutil (40,205)
  26. PyQRCode (39,436)
  27. matplotlib (38,412)
  28. tensorflow (38,335)
  29. opencv-python (37,274)
  30. ifaddr (36,804)

Architectures

The Pi 2 and Pi 3 both identify as armv7l, and that’s by far the most common architecture, followed by armv6l which is the architecture of Pi 1 and Pi Zero. Note that the time saved is much higher for Pi 1/Zero users as they don’t contain the quad-core CPU found in the Pi 2/3.

  1. armv7l (4,564,317)
  2. armv6l (481,616)
  3. x86_64 (93,142)
  4. armv8l (6,666)
  5. i686 (6,550)
  6. aarch64 (606)
  7. AMD64 (185)
  8. x86 (2)

Pi 3 users running a 64-bit operating system (not Raspbian) identify as armv8l, which is fourth in the list. We don’t provide armv8l wheels, so this number only includes pure Python packages. Higher than armv8l in third place is x86_64, which is 64-bit non-Pi platforms, i.e. PC running Windows/Mac/Linux, but again, this (as with all other architecture) only counts pure Python packages. These users will not really see any benefit to using piwheels to install packages. Pure Python wheel downloads may be slightly faster than source distribution but it’s nothing compared to packages which require compilation.

Operating systems

By far, most downloads come from Raspbian, particularly Raspbian Stretch. The next highest are Debian Stretch and then various Ubuntu distributions (each significantly smaller):

Since Raspbian Stretch comes pre-configured with settings for piwheels, but Jessie doesn’t, Jessie users have to update pip and manually set the config to use piwheels. There may be a significant number of Raspberry Pi users using Raspbian Jessie who are not aware of the piwheels project, but we would never know. There are also plenty of other random distributions appearing with just a few downloads each.

Python versions

Currently we provide wheels for the default Python 3 version in Raspbian Jessie (Python 3.4) and Stretch (Python 3.5), we don’t build Python 2 wheels, but pure Python packages are often available for both 2 and 3, so Python 2 users can pick up those packages, and hence Python 2.7 is present in our download stats. We’ve started to build Python 3.6 wheels using unreleased Raspbian Buster (testing), and by the time that’s released, it will probably ship with Python 3.7 and hopefully without Python 2.

The vast majority of our downloads are for Python 3.5 (Stretch). However, the number of Python 2 downloads (though limited to pure Python packages) is larger than any other Python 3 version which indicates a large Python 2 user base. Remember, kids, Python 2 EOL is now less than a year away!

Time saved

Since we record how long a build takes, and we log every download, we can calculate the amount of time saved:

  • January – 1 year 7 months
  • February – 1 year 8 months
  • March – 2 years 1 month
  • April – 1 year 11 months
  • May – 2 years 2 months
  • June – 2 years 8 months
  • July – 3 years 1 month
  • August – 4 years 6 months
  • September – 4 years 4 months
  • October – 5 years
  • November – 4 years 10 months
  • December – 6 years 1 month

This totals nearly 40 years saved in just one year. The most time saved in a single day was on 28 December (over 90 days).

It’s worth noting that for the only package we don’t build ourselves, Tensorflow (built by Google and uploaded to piwheels), we don’t record a build time, so this package is effectively excluded from the time saved stats. Pete Warden estimates it would take a whole day to build it yourself on a Pi 3, and this is our 28th most popular package (and climbing – 25th in the last month), so if we were to add this to the time saved stat it would increase immensely (by an estimated 141 years), which would heavily skew the stat.

Energy saved

Similarly, we can calculate how much energy has been saved by reducing the need for compilation. Roughly speaking, we estimate we’ve saved at least 276KWh of electricity, based on the idle draw of a Pi 2 for the Armv7 downloads, and a Pi Zero for the Armv6 downloads.

Downloads through the day

So what does an average day look like on piwheels? Well here’s a graph showing the number of downloads per hour throughout our busiest day of the year:

On to 2019

Debian and Raspbian Buster are expected to arrive in mid-2019, so we’ll see the arrival of Python 3.6 and 3.7, and will provide wheels for all packages in due course.

We have been working on a few new features, including HTML project pages (like PyPI does) so you don’t have to use the simple index to see which packages and versions we have. We have also made it possible to calculate apt dependencies for a wheel based on packages providing required shared object files, as described in a previous post.

If you’re interested in following piwheels project updates, we tweet daily and monthly stats at @piwheels. You can also follow me @ben_nuttall and Dave @waveform80.

The piwheels project wouldn’t be possible without considerable support from Mythic Beasts, who provide storage and cloud Pis. The Pi platform is so straightforward, it’s been a pleasure to use, allowing us to scale up builder Pis with ease. We highly recommend using this (very affordable) service for real Pi testing for your projects.

New Tensorflow releases for Raspberry Pi

Last month, the Tensorflow team at Google announced official support for Raspberry Pi, by releasing pre-build binaries of v1.9.0 to piwheels.org. Since then, two new releases (v1.10.1 and v1.11.0) have been made and they are now available on piwheels.org.

To install the latest Tensorflow on Raspbian Stretch, first install libatlas, which is a depencency for the optimised version of numpy on piwheels, the  simply use pip to install tensorflow:

sudo apt install libatlas3-base
sudo pip3 install tensorflow

If you’re using Tensorflow in your Pi project, be sure to let us know by tweeting at @piwheels.

How to work out the missing dependencies for a Python package

When you install a compiled Python wheel, whether it’s from PyPI or piwheels, it will likely depend on some shared libraries, specifically certain .so files (shared object files) in order to be used.

If you’ve ever been in the situation where you’ve installed a library but importing it fails, it’s a pretty unpleasant experience:

The next step for most people is to Google the error message. But that’s a long and slow process that often doesn’t lead to a successful import.

The best way to resolve this issue requires a couple of command line tools: ldd and apt-file.

First, navigate to the location of the package installation. This is usually /usr/local/lib/python3.5/dist-packages/<package>/. Note the package directory will be named after the import line, which may differ from the name of the package as you installed it. For example, you pip install numpy and import numpy but you pip install opencv-python and import cv2.

Run ls in that directory and look for an .so file:

pi@raspberrypi:/usr/local/lib/python3.5/dist-packages/cv2 $ ls
cv2.cpython-35m-arm-linux-gnueabihf.so data __init__.py __pycache__

Run ldd on that file:

ldd cv2.cpython-35m-arm-linux-gnueabihf.so

(tab completion is your friend)

You’ll see a lot of .so files. Those are shared objects the Python library source code refers to. Some of them will be available on your system, and will show the location they can be found:

libpthread.so.0 => /lib/arm-linux-gnueabihf/libpthread.so.0 (0x74fdb000)

Others will show “not found”. These are the ones you need to make available:

libhdf5_serial.so.100 => not found

You can use grep to filter out the found ones:

ldd cv2.cpython-35m-arm-linux-gnueabihf.so | grep "not found"

To find out which apt packages provide this .so, use apt-file. apt-file isn’t installed by default, so install it with apt:

sudo apt install apt-file

You’ll want to update apt-file‘s cache:

sudo apt-file update

Then use apt-file search on the missing .so file:

apt-file search libhdf5_serial.so.100

This will show a list of apt packages (including some duplicates):

libhdf5-100: /usr/lib/arm-linux-gnueabihf/libhdf5_serial.so.100
libhdf5-100: /usr/lib/arm-linux-gnueabihf/libhdf5_serial.so.100.0.1

This indicates that the package libhdf5 will provide the required .so file opencv refers to. So install it:

sudo apt install libhdf5-100

That’s it! Just rinse and repeat. Once you’ve made all the missing shared object files available, you’ll be able to import the module no problem:

Note that some shared objects can be provided by multiple packages. Sometimes it’s obvious which is the lighter option, i.e. libatlas3-base rather than libatlas-base-dev (avoid -dev packages if possible), or libpango-1.0-0 rather than the humongous wolfram-engine. You can see the file size with apt-cache show:

pi@raspberrypi:~ $ apt-file search libpango-1.0.so.0
libpango-1.0-0: /usr/lib/arm-linux-gnueabihf/libpango-1.0.so.0
libpango-1.0-0: /usr/lib/arm-linux-gnueabihf/libpango-1.0.so.0.4000.5
wolfram-engine: /opt/Wolfram/WolframEngine/11.3/SystemFiles/Libraries/Linux-ARM/libpango-1.0.so.0
pi@raspberrypi:~ $ apt show libpango-1.0 | grep Size

Installed-Size: 515 kB
Download-Size: 305 kB
pi@raspberrypi:~ $ apt show wolfram-engine | grep Size

Installed-Size: 829 MB
Download-Size: 306 MB

As you can see, the choice here is an easy one: 300 kB vs 300MB!

We’re planning to add project pages to piwheels.org, which will feature library dependencies so you don’t have to look them up manually. If you were looking for opencv requirements, see our blog post New opencv builds which includes the lists of dependencies for the opencv packages.

New opencv builds

The opencv maintainers don’t release source distribution for the packages, so Dave has been building Raspberry Pi wheels manually from source on GitHub. They have also chosen to split releases into four separate pacakges:

opencv-contrib includes all of opencv, plus additional modules (listed in the opencv docs). The -headless releases exclude any GUI functionality, which mean they require fewer dependencies, and are ideal if you’re using Raspbian Lite or simply not using any GUI features.

We now have Raspberry Pi wheels for versions 3.4.2.16, 3.4.2.17 and 3.4.3.18 of all four package variations on piwheels.org.

Dependencies

When you pip install an opencv package, you’ll need various apt packages installed to provide make it work. We’re planning to document this on piwheels.org for each package in future but for now, here are the full installation instructions for opencv:

opencv-python

sudo apt install libatlas3-base libwebp6 libtiff5 libjasper1 libilmbase12 libopenexr22 libilmbase12 libgstreamer1.0-0 libavcodec57 libavformat57 libavutil55 libswscale4 libqtgui4 libqt4-test libqtcore4
sudo pip3 install opencv-python

opencv-python-headless

sudo apt install libatlas3-base libwebp6 libtiff5 libjasper1 libilmbase12 libopenexr22 libilmbase12 libgstreamer1.0-0 libavcodec57 libavformat57 libavutil55 libswscale4 libgtk-3-0 libpangocairo-1.0-0 libpango-1.0-0 libatk1.0-0 libcairo-gobject2 libcairo2 libgdk-pixbuf2.0-0
sudo pip3 install opencv-python-headless

opencv-contrib-python

sudo apt install libatlas3-base libsz2 libharfbuzz0b libtiff5 libjasper1 libilmbase12 libopenexr22 libilmbase12 libgstreamer1.0-0 libavcodec57 libavformat57 libavutil55 libswscale4 libqtgui4 libqt4-test libqtcore4
sudo pip3 install opencv-contrib-python libwebp6

opencv-contrib-python-headless

sudo apt install libatlas3-base libhdf5-100 libharfbuzz0b libwebp6 libtiff5 libjasper1 libilmbase12 libopenexr22 libgstreamer1.0-0 libavcodec57 libavformat57 libavutil55 libswscale4 libgtk-3-0 libpangocairo-1.0-0 libpango-1.0-0 libatk1.0-0 libcairo-gobject2 libcairo2 libgdk-pixbuf2.0-0
sudo pip3 install opencv-contrib-python-headless

 

Official Tensorflow releases

We’re pleased to announce the immediate availability of Tensorflow 1.9 for Raspberry Pi, as officially supported by the Tensorflow team at Google.

Install Tensorflow now on your Raspberry Pi:

sudo apt install libatlas3-base
sudo pip3 install tensorflow

This news was originally announced by Pete Warden, software engineer at Google:

Tensorflow is available for Python 3.7, 3.4 and 3.5 and works on any Raspberry Pi model. Read more in Pete’s Medium post and see the official documentation.

We’re really excited to see what people do with Tensorflow on the Raspberry Pi now that an easy installation is available.

Unlike all other packages on piwheels at present, Tensorflow was not built by the piwheels team. It was built by Google and submitted directly to us for inclusion.

Happy Tensorflowing!