I'm working on my pi 'cluster', built using:
- 1x Raspberry Pi 4
- 4x Raspberry Pi 0 w
- 2x Raspberry Pi 0
Current use case is to run a python distributed task processor built on top of celery, Pi 0s and the controller on Pi4. The Backend for the processor is
- RabbitMQ for task distribution
- Redis for collecting results (celery backend)
I started with learning about celery (http://www.celeryproject.org/). When I got a simple worker up and running (following the basic tutorial) I started thinking about deployment.
The worker code, is in python, I was searching for a way to build a CI/CD pipeline from my Gitlab server, another requirement was to install worker as a service. I found this article to be the most insightful (https://www.nylas.com/blog/packaging-deploying-python/). It lists couple solutions:
- git & pip
- PEX
- deb packages (dh-virtualenv)
- docker
After investigating pros and cons (you can find them in the lined article) I decided that the minimal fuss solution is docker.
Next step was to get docker on all my pi nodes. Another problem, installing docker on pi0(w) does not work out of the box (https://www.reddit.com/r/raspberry_pi/comments/d29tg0/failing_to_get_docker_installed_on_a_pi_zero_w/) (https://github.com/moby/moby/issues/38175)
The suggested ways of getting docker on pi0 are:
- HypriotOS
- Downgrading containerd
I started with HypriotOS as it seemed straightforward. Flashed the microSD card with 1.12.0, added my cloud-init (added ssh key, set static IP and hostname)
The result was - well, it booted up on Pi0, but the cloud-init didn't work, the pirate user wasn't created and I couldn't log in to the system. Found out it's a known issue (https://github.com/hypriot/image-builder-rpi/issues/340) but the suggested workaround wasn't suitable for me.
I didn't want to mess with raspbian packages, so I decided to try and get Hypriot to work. Putting that flashed microSD card into Pi4 worked fine. I was curious, if after getting cloud-init to work properly on Pi4, it would work after transferring the card to Pi0. And it did, that way I got a working hypriotOS, with docker on my Pi0s! I flashed all 6 nodes, flashed Pi4... and discovered another problem (or actually couple of them)
- The network did not always start properly, on random occasions it would fail the wpa_supplicant init and crash. The error suggested that
/dev/stderrwas not initialised. - Second problem was:
ctrl_iface exists and seems to be in use - cannot override it
Delete '/var/run/wpa_supplicant/wlan0' manually if it is not used anymore
Failed to initialize control interface 'DIR=/var/run/wpa_supplicant GROUP=netdev'.
You may have another wpa_supplicant process already running or the file was
left by an unclean termination of wpa_supplicant in which case you will need
to manually remove this file before starting wpa_supplicant again.- Third problem was that the wlan0 interface was getting secondary IP address assigned from DHCP
I fixed the first problem by adding a little sed magic to cloud-init runcmd: (debian bug: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=838291)- [ sed, -i, -e, 's/\/dev\/stderr/\&2/g', /etc/wpa_supplicant/functions.sh] - [ sed, -i, -e, 's/\/dev\/stdout/\&1/g', /etc/wpa_supplicant/functions.sh]
Second problem, this solution worked (https://raspberrypi.stackexchange.com/questions/50967/raspi-3-wlan0-not-associated) but it is ad-hoc fix, not suitable for cluster node, I could as well just unplug and plug in the power manually to have same effect(usually after rebooting the network worked, it was really random)
The third problem was related to dhcpcd and /etc/network/interfaces miss configuration. Default cloud-init set the static IP in the /etc/network/interface.d/wlan0 there was no entry in the dhcpcd.conf for it. While debugging the issue, I stopped the dhcpcd service, and the secondary IP disappeared, together with any host resolution... So it wasn't the way. (I later discovered that adding denyinterfaces wlan0 to dhcpcd.conf, but didn't have a chance to try it)
I was able to fix 2/3 issues, so it was still not good enough for my use, I decided to try playing with Raspbian and applying everything I have learned so far.
I got the latest Raspbian buster lite, flashed the microSD card, installed it on Pi4 (important) and I was able to install docker with no problem (source)
curl -sSL https://get.docker.com | sh
sudo usermod -aG docker pi
docker run hello-world
sudo apt install -y libffi-dev libssl-dev python3 python3-pip
sudo apt-get remove python-configparser
sudo pip3 install docker-composeAfter testing that it works on Pi0 (hooray) I added zsh, ohmyzsh, powerline10k, and vim.
Having a fully operational system I started to think about how to flash all the other nodes quickly? I went with creating an image from my microSD card, shrinking it, flashing it to other cards and adjusting hostnames and ip addresses manually.
# run the commands on Linux, not WSL, if on VM, mount the SD card to the VM
sudo fdisk -l # find the device name for your card, like /dev/mmcsd
sudo dd if=/dev/mmcsd of=/path/to/your/image
# let it run, it might take couple minutes, and there is no output.
wget https://raw.githubusercontent.com/Drewsif/PiShrink/master/pishrink.sh
chmod +x pishrink.sh
sudo ./pishrink /path/to/your/image.img /path/to/your/image.shrinked.img
gzip -9 /path/to/your/image.shrinked.img
# First image for me was 32GB - as the size of the card. Shrinked version was below 4GB, gzipped around 900MB
Now you can use the image with Etcher (https://github.com/balena-io/etcher)
That's it.
Links
https://packaging.python.org/overview/
.https://raspberrypi.stackexchange.com/questions/13359/where-does-my-secondary-ip-come-from
Clone and shrink Raspbian image: https://medium.com/platformer-blog/creating-a-custom-raspbian-os-image-for-production-3fcb43ff3630