In my HomeLab I have two RancherOS hosts, and now I added 7 Raspberry Pi's and I'm planning on adding more hosts in the future. Initially I was just connecting via ssh and executing commands on the hosts. For Rancher running docker-compose is a little extra work, as it doesn't have it available. The quickest solution I have vor it is:

docker run --rm -it -v ${PWD}:/rx -v /var/run/docker.sock:/var/run/docker.sock -w /rx docker/compose up -d

This command:

• docker run gets docker-compose container from docker hub
• --rm removes the container after closing it, or after the command finishes
• -it launches interactive terminal
• -v ${PWD}:/rx mounts current directory as /rx on the container
• -v /var/run/docker.sock:/var/run/docker.sock mounts local docker socket, so that the compose container can control your docker instance
• -w sets the working directory
• docker/compose is the name of the image from docker hub
• up -d is the command we send to the container

All this is equivalent to running docker-compose up -d but without having compose installed

This approach is not optimal for having multiple remote hosts, so I was researching how could I use my local docker engine and control the remote hosts.

The quickest way is to run export DOCKER_HOST=ssh://username@<machine.ip> , that makes docker use the remote HOST. There is a newer way using docker context. It allows you to set and use multiple configs and easily switch between them.

To list the existing contexts run docker context ls

To create a new context run docker context create <context_name> --docker "host=ssh://<username>@<host or ip>"

To use the newly created context you can either run docker --context <context_name> command for each command or run docker context use <context_name>. Important note is that if you set DOCKER_HOST variable it will override docker context, so make sure to unset it.

docker-compose

What about docker-compose? The new-new version supports docker contexts, but it's still in RC as I write this post, so the only other way is to set DOCKER_HOST, it will use it automatically.

paramiko.ssh_exception.ChannelException: ChannelException(1, 'Administratively prohibited') or similar errors are caused by sshd MaxSessions default value of 10, where docker-compose tries to open 25 connections. Solution is to edit /etc/ssh/sshd_config and changing MaxSessions to something above 25.

Links:

• https://github.com/docker/compose/issues/6463
• https://github.com/docker/docker-py/issues/2246
• https://www.docker.com/blog/how-to-deploy-on-remote-docker-hosts-with-docker-compose/

I'm working on my pi 'cluster', built using:

• 1x Raspberry Pi 4
• 4x Raspberry Pi 0 w
• 2x Raspberry Pi 0

Current use case is to run a python distributed task processor built on top of celery, Pi 0s and the controller on Pi4. The Backend for the processor is

• RabbitMQ for task distribution
• Redis for collecting results (celery backend)

I started with learning about celery (http://www.celeryproject.org/). When I got a simple worker up and running (following the basic tutorial) I started thinking about deployment.

The worker code, is in python, I was searching for a way to build a CI/CD pipeline from my Gitlab server, another requirement was to install worker as a service. I found this article to be the most insightful (https://www.nylas.com/blog/packaging-deploying-python/). It lists couple solutions:

• git & pip
• PEX
• deb packages (dh-virtualenv)
• docker

After investigating pros and cons (you can find them in the lined article) I decided that the minimal fuss solution is docker.

Next step was to get docker on all my pi nodes. Another problem, installing docker on pi0(w) does not work out of the box (https://www.reddit.com/r/raspberry_pi/comments/d29tg0/failing_to_get_docker_installed_on_a_pi_zero_w/) (https://github.com/moby/moby/issues/38175)

The suggested ways of getting docker on pi0 are:

• HypriotOS
• Downgrading containerd

I started with HypriotOS as it seemed straightforward. Flashed the microSD card with 1.12.0, added my cloud-init (added ssh key, set static IP and hostname)

The result was - well, it booted up on Pi0, but the cloud-init didn't work, the pirate user wasn't created and I couldn't log in to the system. Found out it's a known issue (https://github.com/hypriot/image-builder-rpi/issues/340) but the suggested workaround wasn't suitable for me.

I didn't want to mess with raspbian packages, so I decided to try and get Hypriot to work. Putting that flashed microSD card into Pi4 worked fine. I was curious, if after getting cloud-init to work properly on Pi4, it would work after transferring the card to Pi0. And it did, that way I got a working hypriotOS, with docker on my Pi0s! I flashed all 6 nodes, flashed Pi4... and discovered another problem (or actually couple of them)

• The network did not always start properly, on random occasions it would fail the wpa_supplicant init and crash. The error suggested that /dev/stderr was not initialised.
• Second problem was:

ctrl_iface exists and seems to be in use - cannot override it
Delete '/var/run/wpa_supplicant/wlan0' manually if it is not used anymore
Failed to initialize control interface 'DIR=/var/run/wpa_supplicant GROUP=netdev'.
You may have another wpa_supplicant process already running or the file was
left by an unclean termination of wpa_supplicant in which case you will need
to manually remove this file before starting wpa_supplicant again.

• Third problem was that the wlan0 interface was getting secondary IP address assigned from DHCP

I fixed the first problem by adding a little sed magic to cloud-init runcmd: (debian bug: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=838291)
- [ sed, -i, -e, 's/\/dev\/stderr/\&2/g', /etc/wpa_supplicant/functions.sh]
- [ sed, -i, -e, 's/\/dev\/stdout/\&1/g', /etc/wpa_supplicant/functions.sh]

Second problem, this solution worked (https://raspberrypi.stackexchange.com/questions/50967/raspi-3-wlan0-not-associated) but it is ad-hoc fix, not suitable for cluster node, I could as well just unplug and plug in the power manually to have same effect(usually after rebooting the network worked, it was really random)

The third problem was related to dhcpcd and /etc/network/interfaces miss configuration. Default cloud-init set the static IP in the /etc/network/interface.d/wlan0 there was no entry in the dhcpcd.conf for it. While debugging the issue, I stopped the dhcpcd service, and the secondary IP disappeared, together with any host resolution... So it wasn't the way. (I later discovered that adding  denyinterfaces wlan0 to dhcpcd.conf, but didn't have a chance to try it)

I was able to fix 2/3 issues, so it was still not good enough for my use, I decided to try playing with Raspbian and applying everything I have learned so far.

I got the latest Raspbian buster lite, flashed the microSD card, installed it on Pi4 (important) and I was able to install docker with no problem (source)

curl -sSL https://get.docker.com | sh
sudo usermod -aG docker pi
docker run hello-world

sudo apt install -y libffi-dev libssl-dev python3 python3-pip
sudo apt-get remove python-configparser
sudo pip3 install docker-compose

After testing that it works on Pi0 (hooray) I added zsh, ohmyzsh, powerline10k, and vim.

Having a fully operational system I started to think about how to flash all the other nodes quickly? I went with creating an image from my microSD card, shrinking it, flashing it to other cards and adjusting hostnames and ip addresses manually.

# run the commands on Linux, not WSL, if on VM, mount the SD card to the VM
sudo fdisk -l # find the device name for your card, like /dev/mmcsd
sudo dd if=/dev/mmcsd of=/path/to/your/image

# let it run, it might take couple minutes, and there is no output.

wget  https://raw.githubusercontent.com/Drewsif/PiShrink/master/pishrink.sh
chmod +x pishrink.sh
sudo ./pishrink /path/to/your/image.img /path/to/your/image.shrinked.img 
gzip -9 /path/to/your/image.shrinked.img

# First image for me was 32GB - as the size of the card. Shrinked version was below 4GB, gzipped around 900MB

Now you can use the image with Etcher (https://github.com/balena-io/etcher)

That's it.

Links

https://packaging.python.org/overview/

.https://raspberrypi.stackexchange.com/questions/37920/how-do-i-set-up-networking-wifi-static-ip-address

.https://raspberrypi.stackexchange.com/questions/39785/differences-between-etc-dhcpcd-conf-and-etc-network-interfaces

.https://raspberrypi.stackexchange.com/questions/13359/where-does-my-secondary-ip-come-from

Clone and shrink Raspbian image: https://medium.com/platformer-blog/creating-a-custom-raspbian-os-image-for-production-3fcb43ff3630

I was trying to get SonarQube test instance for my HomeLab, and found out Bitnami docker image (link). I thought about using my RancherOS freshly updated VM for trying to run it.

The first obstacle happened to be the curl. There is no curl installed on bare RancherOS instance. But there is wget so a simple fix to replace those commands.

The second problem was bigger, you get docker-compose.yml file, and have to run docker-compose to start the stack. But there is no docker-compose on RancherOS...

After searching the web I discovered there is a docker image with docker compose, but no instructions on how to execute it from container.

After some googling I crafted a woring command:

docker run --rm -it -v ${PWD}:/tmp -v /var/run/docker.sock:/var/run/docker.sock -w /tmp docker/compose up

• -v ${PWD}:/tmp mounts the current directory as /tmp on docker container
• -v /var/run/docker.sock:/var/run/docker.sock allows docker-compose to talk to rancherOS docker
• -w /tmp sets the working directory for docker container
• docker/compose is the latest compose image
• up is the command

Overall it's the equivalent of running docker-compose up but without having docker-compose installed/available.

Usefull links:

Running docker-compose with Rancher OS

I am trying with aws rancher os. I want to create and run a docker-compose file with the same rancher OS. When I am trying with Docker-compose up command I am getting the error ’not recognized doc...

NGNStack Overflow

As part of the HomeLab refresh, I started updating my tools and dependencies, one of the pieces was RancherOS image that I used for running Gitlab runners.

When I tried following my old notes (link) I realised that some commands were no longer working.

I'm experimenting with XCP-ng (xen) this time, so cannot find a way to inject cloud-init on startup. I got the latest rancherOS iso, loaded it to my iso library and booted the newly created vm. Because I couldn't inject the config file directly, I had to find a way to get it to the rancherOS vm. In the old days I would just use
python -m  SimpleHTTPServer 8000 in the directory with files I wanted to use, but with python 2.x reaching EOL I moved to a modern, python 3.x syntax:
python -m http.server 8000. That command starts a http server in the directory you run it in, so I can access all the files using HTTP protocol from all the machines in my homelab network.

The config file that works, looks like:

#cloud-config

hostname: rancherX

rancher:
  network:
    interfaces:
      eth*:
        dhcp: false
      eth0:
        address: 192.168.0.2/24
        gateway: 192.168.0.1
    dns:
      nameservers:
        - 192.168.0.1

ssh_authorized_keys:
  - ssh-rsa ASDF..GF==

This config file allows you to quickly set up

• hostname
• static ip
• nameservers
• ssh key for remote access

Now to install the rancherOS you have to run:

sudo ros install -c http://put.your.ip.here:8000/rancher-cloud-config.yml -d /dev/mount_name

I noticed it doesn't allow to run without specifying the mount to install, so seems like the in-memory commands are no longer working

The time has come for me to revisit my local dev-lab, one of the things I'd like to do first is update the software versions of the tools I use. First tool is - Gitlab

When I started my lab, I installed Gitlab CE from bitnami, version 11. The current version (as of April 2020) is version 12.

The upgrade process seems quite simple

• sudo apt update
• sudo apt install gitlab-ce

Let's see what's going to break :)

The first problem I encountered is the expired PUBKEY
(The following signatures couldn't be verified because the public key is not available: NO_PUBKEY).
Fix is quite simple :

• curl --silent https://packages.gitlab.com/gpg.key | sudo apt-key add -

Now update should work. When I run install, I got a notification that I', trying to upgrade from major version 11 to major version 12, and I first have to upgrade to version 11.11 ...

Now - how do I install this version? First I have to find the specific version number

• apt-cache madison gitlab-ce and because there are a lot of version and I only care about 11.11, I will pipe it to grep | grep 11.11
• I found version 11.11.8-ce.0
• Now to install the specific verion I run sudo apt install gitlab-ce=11.11.8-ce.0

If everything worked fine, now it's time to check if the gitlab server behaves properly - open the UI, pull and push some code.

Everything was fine, so I can proceed with latest version instalation

• sudo apt install gitlab-ce

That's it! :)

Links:

How to update GitLab repository signing key

GitLab repository signing key was updated at the beginning of the April, so you can get an error during the signature verification.

Milosz Galazka

If you have a GitLab instance using your self-signed certificate, you have to add it to machines pulling the code, and to the runner, so that they can securely communicate with the server.

I'm using docker based gitlab-runner, to add the cert to it follow these steps:

Make sure you have the certificate, I'm using the root pem certificate file.

Copy the file to your docker box and rename it to ca.crt (yes, change the file extension!)

Next step is to run the temporary container, mounting the config directory and registering it with gitlab:

Run: 
docker run --rm -t -i \
    -v /srv/gitlab-runner/config:/etc/gitlab-runner \
    -v /path-to-your-cert/ca.crt:/etc/gitlab-runner/certs/ca.crt \
    --name gitlab-runner gitlab/gitlab-runner register

You are going to be asked a few questions now: (borrowed from gitlab documentation). You can get the required information from your gitlab server
https://your-gitlab.url/admin/runners

1. Enter your GitLab instance URL:
   Please enter the gitlab-ci coordinator URL (e.g. https://gitlab.com )  https://gitlab.com
2. Enter the token you obtained to register the Runner:
   Please enter the gitlab-ci token for this runner  xxx
3. Enter a description for the Runner, you can change this later in GitLab’s UI:
   Please enter the gitlab-ci description for this runner  [hostame] my-runner
4. Enter the tags associated with the Runner, you can change this later in GitLab’s UI:
   Please enter the gitlab-ci tags for this runner (comma separated):  my-tag,another-tag
5. Enter the Runner executor:
   Please enter the executor: ssh, docker+machine, docker-ssh+machine, kubernetes, docker, parallels, virtualbox, docker-ssh, shell:  docker
6. If you chose Docker as your executor, you’ll be asked for the default image to be used for projects that do not define one in .gitlab-ci.yml:
   Please enter the Docker image (eg. ruby:2.1):  alpine:latest

The final step is to run the runner:

Run:
docker run -d --name gitlab-runner --restart always \
    -v /srv/gitlab-runner/config:/etc/gitlab-runner \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -v /path-to-your-cert/ca.crt:/etc/gitlab-runner/certs/ca.crt \
    gitlab/gitlab-runner:latest

Links:

https://docs.gitlab.com/runner/register/index.html#docker

https://docs.gitlab.com/runner/install/docker.html

https://docs.gitlab.com/runner/

I wrote an updated version here: https://blog.petermartyniak.com/rancheros-cloud-init-cloud-config-setup-2020-update/

When you create a new RancherOS box, you can use cloud-config.yml file to setup your OS. There are two ways of doing it, with or without installation to the disk.

In memory option:

Create or download your cloud-config.yml file:

sudo ros install -c https://link/to/cloud-config.yml

Install to disk option:

Get the config file as in the previous section, then:

sudo ros install -c cloud-config.yml -d /dev/sda

Kernel params for ESXi 6.7 option:

Edit your Rancher VM open [VM Options] > [Advanced] > [Edit Configuration] and [Add parameter]

Key= guestinfo.cloud-init.config.url

Value= http://your.url/cloud-config.yml

Rancher is going to automatically pick up that config and apply it on boot

The cloud config file:

This file sets the persistent drive, hostname, ssh key, static ip address and custom DNS

#cloud-config
runcmd:
  -  sudo mkfs.ext4 -L RANCHER_STATE /dev/sda
hostname : your-hostname
ssh_authorized_keys:
  - ssh-rsa =your=rsa=key
rancher:
  network:
    interfaces:
      eth*:
        address: ip/netmask
        gateway: gateway
    dns:
      nameservers:
        - nameserver

Links

https://rancher.com/docs/os/v1.2/en/running-rancheros/cloud/vmware-esxi/

https://rancher.com/docs/os/v1.2/en/boot-process/cloud-init/

https://rancher.com/docs/os/v1.2/en/running-rancheros/server/install-to-disk/

https://cloudinit.readthedocs.io/en/latest/topics/examples.html

https://rancher.com/docs/os/v1.x/en/installation/configuration/

You can run RancherOS in two ways. Boot from an ISO to memory or install on disk. There are some advantages of running it from ISO, like super easy update process. The downside is - by default, after reboot, you're going to loose all your images, settings and data. To fix that, you can persist the data by creating and mounting a partition. (documentation)

sudo mkfs.ext4 -L RANCHER_STATE /dev/sda

sudo reboot

Sometimes when you experiment with some apps and VMs (like hosting gitlab on a local server) you might want to setup SSL for the app to work, to mimic the live setup and to make the browser happy. In order to do that, you need a SSL certificate.

You can buy one for your domain from a trusted CA, but if you're working on a local network, that might not be possible. The other solution is... becoming CA yourself and issuing and signing the certificate yourself!

It's pretty easy, you need a linux box with openssl installed, then follow these instructions:

CA part

To become a CA, you need a key and certificate pair. To create the key, execute:

openssl genrsa -des3 -out myCA.key 2048

To generate the certificate, execute the following:

openssl req -x509 -new -nodes -key myCA.key -sha256 -days 1925 -out myCA.pem

That's it! Now after you import the CA certificate to your machine, every certificate signed by it is going to be trusted!

CRT part

First thing you need is a private key:

openssl genrsa -out gitlab.local.key 2048

Then create the signing request:

openssl req -new -key gitlab.local.key -out gitlab.local.csr

Answer the question asked, one potentially important is the Common Name.

Now to sign it with the CA key and certificate, you need the config file with Subject Alternative Name (SAN) specified.

The config I used comes from here:

authorityKeyIdentifier=keyid,issuer
basicConstraints=CA:FALSE
keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
subjectAltName = @alt_names

[alt_names]
DNS.1 = gitlab.local

Now the final command to sign the certificate:

openssl x509 -req -in gitlab.local.csr -CA myCA.pem -CAkey myCA.key -CAcreateserial -out gitlab.local.crt -days 1825 -sha256 -extfile config.conf

Now you should have the working and signed certificate.

Links & Gotcha's

why you cannot do TLD wildcard, even with SAN (like *.local)

https://bugs.chromium.org/p/chromium/issues/detail?id=736715

https://superuser.com/questions/1305671/san-wildcard-for-whole-domain-tld

https://www.icann.org/groups/ssac/documents/sac-015-en

https://security.stackexchange.com/questions/6873/can-a-wildcard-ssl-certificate-be-issued-for-a-second-level-domain

Useful links

https://deliciousbrains.com/ssl-certificate-authority-for-local-https-development/

https://stackoverflow.com/questions/10175812/how-to-create-a-self-signed-certificate-with-openssl/27931596#27931596

https://stackoverflow.com/questions/43665243/invalid-self-signed-ssl-cert-subject-alternative-name-missing/43665244#43665244

https://unix.stackexchange.com/questions/371997/creating-a-local-ssl-certificate

http://wiki.cacert.org/FAQ/subjectAltName

https://geekflare.com/san-ssl-certificate/

https://gist.github.com/bitoiu/9e19962b991a71165268

https://blog.zencoffee.org/2013/04/creating-and-signing-an-ssl-cert-with-alternative-names/

http://grokify.github.io/security/wildcard-subject-alternative-name-ssl-tls-certificates/

https://stackoverflow.com/questions/1822268/how-do-i-create-my-own-wildcard-certificate-on-linux

In order to setup your local GitLab CE instance (inside ESXi), you need:

• Working ESXi
• Bitnami Gitlab *.OVA file (link)
• Local DNS is a bonus

VM

After you download the OVA file, open ESXi web UI, create a new VM and select Deploy a VM from UVF file, give it a name and select/drag the OVA file into the box. Follow on screen instructions.

Static IP/ DHCP

Now that you have the VM up an running, you have to make a decision, if you want to use DHCP or static IP for that box. If your choice is DHCP, setup your DHCP service to assign same IP to your VM MAC address every time, so that you can setup the URL/hostname in Gitlab. If you pick static IP, open the VM console in ESXi (or ssh to the box if enabled). You should see the login and password, use it. You might be asked to change the default password.

Now follow the instructions from here:

• check your interface name: ip a it should be something like ens32
• navigate to network settings directory cd /etc/systemd/network
• copy network config file sudo cp 99-dhcp.network 25-wired.network
• edit the network settings file:

[Match]
Name=ens32 (your interface neame here)

[Network]
Address=10.0.0.9/24 (desired IP address and netmask)
Gateway=10.0.0.1
DNS=10.0.0.1

• save file and restart the VM

Hostname, DNS

At this point you should have IP setup done and dusted, it's a good idea to give your GitLab instance some meaningful name to access, like gitlab.local

In order to do it, you need a local DNS server. Add an A record, pointing to your VM IP address, and a name.

Try accessing your GitLab instance using your newly created domain name. There are two problems.

1. SSL is not set up and the browser is complaining about self signed certificate
2. Server is redirecting to IP address, not using the domain name

To fix the second issue, follow those instructions (link), or run

sudo /opt/bitnami/apps/gitlab/bnconfig --machine_hostname NAME

where NAME is your domain name.

Rename or remove the bnconfig file to avoid the hostname being reset after reboot!

SSL

The last part is setting up SSL. As we are using the server in a local network, self signed certificate is enough.

You need:

• server.key
• server.crt

Copy both files to the server (I'm using WinSCP), copy both files to /etc/gitlab/ssl/ change owner to root:root and chmod 600.

Last two steps are - trigger gitlab script reconfigure and restart gitlab service.

mv server.* /etc/gitlab/ssl/
chown root:root /etc/gitlab/ssl/
chmod 600 /etc/gitlab/ssl/
gitlab-ctl reconfigure
gitlab-ctl restart

Git.exe and SSL

Execute git config --list --show-origin to find out the location of your git config files. Then edit it and add:

[http]
	sslCAInfo=[your location]\\cert.pem

Gotcha's

I use local DNS server, but the Bitnami image used Google's 8.8.8.8 and 8.8.4.4 DNS by default. Modifying the network setup wasn't enough. I had to manually edit /etc/resolv.conf file to add my DNS server.