Watchdog: reboot Raspberry Pi if network is down

Network down! Help!

I have an issue with one of my Raspberry Pis. It will not survive if Wi-Fi network is down for an extended period of time.

I have not been able to figure out the root cause, or how long the network needs to be unreachable before the Raspberry Pi just drops the network connection and never connects again. This happens occasionally since the Pi is located just a little too far away from my Wi-Fi access point. Due to real estate fire prevention regulation reasons I do not have a cable route available so wireless is the way to go.

Now this wouldn’t be such a big issue, but the Raspberry Pi is a headless one (i.e. it does not have a keyboard or monitor), and is only connected via Wi-Fi network. I have some sensors connected to this Pi, which in turn update a web page. Losing network connectivity causes web page data to be out of date and is quite inconvenient.

Today after work I noticed that the Pi had lost network connectivity (again) at 04:40. I had to make my way to the garage (again) and reboot the device (which has no keyboard or network connectivity) – and my frustration level rose to “I’ll fix this now!” level.

Right. After a few half-assed attempts at ping + reboot scripts I figured there must be a more elegant solution, and I was sure I’m not the first one with this problem.

Enter Watchdog.

Searching the Internet I found a couple of examples, none of which was immediately copy/paste ready for my needs.

They did lead me to right track and after reading some manpages, forums and gist.github.com code snippets, getting my Pi to a constant reboot loop etc. I finally came up with what seems to be a working solution.

The solution

First things first: I used a Raspberry Pi 2 with a USB Wi-Fi module and Raspbian Wheezy (4.1.19-v7+ #858 SMP Tue Mar 15 15:56:00 GMT 2016 armv7l GNU/Linux).

Start by installing watchdog:

sudo apt-get install watchdog

Make a backup copy of /etc/watchdog.conf file just in case:

sudo cp /etc/watchdog.conf /etc/watchdog.conf.backup

Edit the /etc/watchdog.conf file to contain the following. There is a short comment on each line about what they do:

$ sudo nano /etc/watchdog.conf
# Watchdog ping: if unresponsive, reboot:
interface = wlan0    # use interface wlan0
ping-count = 5       # ping 5 times
ping = 192.168.1.1   # ping test destination IP address
# Change default interval from 1 second to 20:
interval = 20        # perform watchdog checks every 20 seconds

then reboot (e.g. sudo reboot).

The above will ping five (5) times for destination address 192.168.1.1 every 20 seconds. I’m not sure if the interface command is actually needed, but it did not do any harm so I left it there.

192.168.1.1 is my default gateway, and I really do want to test connectivity against this address instead of some random host in the Internet, since I don’t want my Pi to reboot in case the Internet connection is down. If you insist on pinging a host in the Internet (not recommended), good choice might be Google public DNS servers (8.8.8.8 and 8.8.4.4) or any other host of your choice.

I did use Google DNS server for testing purposes, since it was easier to cut the connection to the Internet but maintain local area network (LAN) connectivity for management purposes.

Watchdog writes log to syslog (/var/log/syslog), and when the ping test fails, this is what it will look like (note the target here is Google DNS 8.8.8.8, not my internal network default gw):

Oct 24 20:35:42 localhost watchdog[2640]: ping: 8.8.8.8
Oct 24 20:35:42 localhost watchdog[2640]: no response from ping (target: 8.8.8.8)

When there is no response to any of the five (5) ping echo requests, the Raspberry will reboot. I will forget this, so I inserted the following to /home/pi/.profile:

echo "Warning: If network is down, this system will reboot in 20 seconds. Comment out ping = 192.168.1.1 line from /etc/watchdog.conf to avoid reboots."

That will print a warning message plus info which file to configure if needed – every time I log in. Even I should be now able to remember where Watchdog is configured 🙂

Further reading

Below you can find a list of some resources which did help me with the solution:

1. Watchdog man page: https://www.systutorials.com/docs/linux/man/8-watchdog/
2. Watchdog.conf man page: https://www.systutorials.com/docs/linux/man/5-watchdog.conf/
3. Good explanation of watchdog.conf ping variable: http://www.sat.dundee.ac.uk/psc/watchdog/watchdog-configure.html#Network_ping
4. Reddit, a solution which almost worked: https://www.reddit.com/r/raspberry_pi/comments/4ih9xo/id_like_my_routeronastick_vpn_to_autorestart/d2y3yj4/?context=3
5. Non-watchdog script to solve the same issue (not mine): https://gist.github.com/SandroMachado/87e591fc42f368636b251b566485ae46
6. Another non-watchdog script (again, not mine): http://weworkweplay.com/play/rebooting-the-raspberry-pi-when-it-loses-wireless-connection-wifi/

Improvements

I have only had this setup running now for a couple of hours, so it’s unclear if it will really work over time. Hope so, I’ll update this article if needed.

What else I could do with Watchdog? Well, I could certainly improve this to check that my Python programs do not die – or at least react and restart them automatically when they do die. Or if the system bogs down and stays unresponsive for extended periods of time.

What do you think? If you have suggestions, improvements or indeed more experience than I do, please do leave a comment below.

Follow-up

Update 30.3.2018: Reminded by Will in the comments (thanks!), I noticed I have not followed up on my promise to update this article.

I have not had any problems with performance or reboot loops etc. with the script. It just works as intended. The 20 second interval is a bit tight, but it’s entirely doable to stop the script in that time if needed.

How would I improve this? Well, I’d create a separate reboot log instead of relying on syslog, but it’s really not necessary. It would help for statistics collection over a longer time period but like said, not necessary.

perl: warning: Please check that your locale settings …

TLDR

My issue was solved by making sure the following content was in /etc/default/locale:

defgw@proto-pi ~ $ cat /etc/default/locale
#  File generated by update-locale
LANG=en_GB.UTF-8
LANGUAGE=en_GB:en
LC_ALL=en_GB.UTF-8

Problem

I recently installed a new Raspbian Scratch image on my protoboard, i.e. on a throwaway Raspberry Pi memory card for testing purposes.

Installing new packages came with an error like below:

$ sudo dpkg-reconfigure locales
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = "en_US.UTF-8",
LC_ALL = "en_US.UTF-8",
LC_CTYPE = "UTF-8",
LANG = "en_GB.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_GB.UTF-8").
locale: Cannot set LC_CTYPE to default locale: No such file or directory
locale: Cannot set LC_MESSAGES to default locale: No such file or directory
locale: Cannot set LC_ALL to default locale: No such file or directory
/usr/bin/locale: Cannot set LC_CTYPE to default locale: No such file or directory
/usr/bin/locale: Cannot set LC_MESSAGES to default locale: No such file or directory
/usr/bin/locale: Cannot set LC_ALL to default locale: No such file or directory

This is definitely not ideal, and googling for a solution came up with various instructions and most related to Ubuntu (maybe this is a more prevalent issue among Ubuntu community?). I did not find a good solution, so I decided to find out where these settings are located.

Fast forward here, a lot of searching, trial and error ensued…

Solution

The locales are configured in /etc/default/locale, and for some reason mine only had the following content:

#  File generated by update-locale
LANG=en_GB.UTF-8

Adding the lines in the error message and changing everything to en_GB.UTF-8 to be in line with the existing LANG setting (and then rebooting for good measure) removed the issue:

$ cat /etc/default/locale
#  File generated by update-locale
LANG=en_GB.UTF-8
LANGUAGE=en_GB:en
LC_ALL=en_GB.UTF-8

The problem is now solved and the error message is gone.

Feedback

If you know whether I have messed something up during Raspbian installation (raspi-config?) or perhaps this is default behaviour, please let me know in comments below.

 

MQTT for Raspberry Pi

Having relied on disk shares over Wi-Fi for a fleet of Raspberry Pis recording and handling sensor data, I’ve had my share of reliability issues.

Knowing basically nothing of MQTT, I did some research¹ and found that this publish-subscribe model might be suitable for my home automation projects. So here it goes, I’m trying to document my endeavours learning the ropes and giving you some insights how it went.

For now I am using a Raspberry Pi 2B with a freshly installed image of just released Raspbian Scratch Lite². The first tests will be managed locally, and I’ll add more Raspberries in to the mix as things progress.

First things first: ensure your installation is up to date:

sudo apt-get update
sudo apt-get upgrade

Install necessary components. I chose to use Mosquitto³ (https://mosquitto.org/), since there is quite a lot of documentation and helpful videos available. Note: mosquitto is the broker module and mosquitto-clients is the client part. You may not need both on each computer.

sudo apt-get install mosquitto
sudo apt-get install mosquitto-clients

Since I will also be creating some Python scripts, let’s install python2 modules for MQTT:

sudo apt-get install python-pip  # For Python3, install python3-pip instead
sudo pip install paho-mqtt # For Python3, use 'sudo pip3 install-paho-mqtt' instead

Basically that’s it. Now you have all the necessary components  to start experimenting with MQTT. Disclaimer: since I haven’t even touched at any Python scripts yet, something may be missing.

A basic test that all Mosquitto components were installed successfully is in order. SSH to the Raspberry Pi (or use desktop and terminal) and enter the following command to start subscribing to a topic called “testtopic”:

$ mosquitto_sub -t "testtopic" -h "127.0.0.1"

Note: the “-h “127.0.0.1” is unnecessary, if you run the sub and pub on the same Raspberry Pi. If you run MQTT server on another host, replace “127.0.0.1” with the actual IP address (or DNS name) of the server. 127.0.0.1 means localhost, which is the default setting should you omit the “-h” option.

The terminal will now be waiting for input to an MQTT topic called “testtopic”. You may exit the state with CTRL-C.

Now open another terminal session and type in:

$ mosquitto_pub -t "testtopic" -m "Hello world!"

You should see “Hello world!” appear on the first terminal. If not, make sure your apt-get commands were completed successfully. If you are having issues, maybe the repository does not contain Mosquitto packages. This may help, especially if you are running Wheezy or Jessie release: https://oshlab.com/install-mqtt-mosquitto-raspberry-pi/

Bonus: you can check which Raspbian release your Raspberry has with the following command.

$ /usr/bin/lsb_release -a | grep -i codename

On my Stretch the output looks like this:

No LSB modules are available.
Codename: stretch

On an older Jessie it looks like this:

No LSB modules are available.
Codename: jessie

And on an even older Wheezy it looks like this:

No LSB modules are available.
Codename: wheezy

One last thing: the mosquitto broker is now automatically started after you installed the package. You can stop and start it using the following commands:

$ sudo /etc/init.d/mosquitto stop
[ ok ] Stopping mosquitto (via systemctl): mosquitto.service.
$ sudo /etc/init.d/mosquitto start
[ ok ] Starting mosquitto (via systemctl): mosquitto.service.

 

That’s it for now, I’ll make more posts later as I stumble forward.

Edit 4.3.2018: Clarified the subscription command by including “-h” option to define the MQTT server.

Sources / references:

¹ https://en.wikipedia.org/wiki/MQTT
¹ http://www.hivemq.com/blog/mqtt-essentials-part2-publish-subscribe
¹ http://internetofthingsagenda.techtarget.com/definition/MQTT-MQ-Telemetry-Transport
² https://www.raspberrypi.org/downloads/raspbian/
³ https://mosquitto.org/

RPi firmware upgrade and how to list all running daemons?

Managed to set up avahi-daemon and CUPS daemon to publish my Wi-Fi printer as an AirPrint printer – a blog post will follow later.

After doing a firmware upgrade (sudo rpi-update) and a reboot the services were no longer running. So I stumbled upon a command which shows me all running daemons:

service --status-all

You’re welcome 🙂

EDIT 5.4.2015:
As far as I understood from searching the web: it’s unnecessary to run rpi-update, unless you want the absolutely latest possible experimental version. Running apt-get update and apt-get upgrade should be quite enough for any normal Rpi user.

First login and static IP address configuration

UPDATE 28.3.2016: Static IP address configuration has apparently changed:

1. Leave /etc/network/interfaces pretty much alone.
2. Configure /etc/dhcpcd.conf with static IP address entries.

See the bottom of this post.

---

 

After successful installation and first boot, I’m about to log in for the first time.

Logging in and changing the default password immediately:

pi@srv01 ~ $ passwd
Changing password for pi.
(current) UNIX password: raspberry
Enter new UNIX password:
Retype new UNIX password:
passwd: password updated successfully
pi@srv01 ~ $

checking if I actually have an IP address:
pi@srv01 ~ $ ifconfig
eth0 Link encap:Ethernet HWaddr b8:27:eb:xx:xx:xx
inet addr:192.168.1.29 Bcast:192.168.1.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:24799 errors:0 dropped:512 overruns:0 frame:0
TX packets:10333 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:2917645 (2.7 MiB) TX bytes:5436810 (5.1 MiB)

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:1088 errors:0 dropped:0 overruns:0 frame:0
TX packets:1088 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:121896 (119.0 KiB) TX bytes:121896 (119.0 KiB)

pi@srv01 ~ $

I do: 192.168.1.29, so let’s update for the first time:

pi@srv01 ~ $ sudo apt-get update

Uh oh, this end with an error. Can’t even ping the Internet hosts, such as http://www.google.com. Can’t be a DNS error, since 8.8.8.8 does not answer either. I can ping my defgw 192.1768.1.1 however, so the problem is not on the local LAN. Quick reboot to the cable modem, and everything is working again.

apt-get completes successfully, and I run sudo apt-get upgrade to actually upgrade the installed packages.

Time to change a static IP address so I can connect to the device remotely and disconnect the keyboard and monitor.

The IP address is set in /etc/network/interfaces text file. Checking the contents can be done with command cat /etc/network/interfaces. To actually change the contents, you must have superuser rights, and be familiar with some text editing software, such as vi or nano. I prefer to use nano on my devices, since it’s easier to use in small scale editing.

pi@srv01 ~ $ sudo nano /etc/network/interfaces

I don’t have saved the original configuration from the file, but below is a working example of all the contents of the file, with a static IP address of 192.168.1.105:

auto lo

iface lo inet loopback
iface eth0 inet static
address 192.168.1.105
netmask 255.255.255.0
network 192.168.1.0
broadcast 192.168.1.255
gateway 192.168.1.1

allow-hotplug wlan0
iface wlan0 inet manual
wpa-roam /etc/wpa_supplicant/wpa_supplicant.conf
iface default inet dhcp

Finally, using the command raspi-config I can go back to the install selection box and make sure SSH server is running.

I’m ready to power off and move the server to it’s proper location. I enter the following command to shut down the server:

pi@srv01 ~ $ sudo shutdown -h now

Disconnect cables, move the server physically to another location, connect network cable and power cable and the server will start itself with IP address 192.168.1.105. Check using my Macbook if I can ssh to it from a terminal window, and a login screen is waiting for me.
ssh pi@192.168.1.105
pi@192.168.1.105's password:

---

UPDATE 28.3.2016: Static IP address configuration has apparently changed:

1. Leave /etc/network/interfaces pretty much alone:

My config for eth0 (without wireless) is currently as follows:

auto eth0
allow-hotplug eth0
iface eth0 inet manual

2. Configure /etc/dhcpcd.conf with static IP address entries:

Add the following to the very end of /etc/dhcpcd.conf:

# Static IP configuration
interface eth0
static ip_address=192.168.1.120/24
static routers=192.168.1.1
static domain_name_servers=8.8.8.8 8.8.4.4

That’s it. For more information, see the source where I found this info:

https://www.raspberrypi.org/forums/viewtopic.php?f=63&t=110606

And scroll down to post by “paulv » Tue Oct 06, 2015 8:51 am”

I haven’t gotten around to test with wireless yet, but I’ll update the post if I get the chance.

 

First install

After successfully copying the NOOBS installation files to the SD card, I’m now ready to connect all the cables:

• Monitor to HDMI connector
• Network cable to RJ-45 connector
• Mouse and keyboard to USB
• micro-SD card to the correct location
• power adapter to power input connector and power the thing up.

Seems like everything is in place, the device boots properly. I make the following selections in the menu presented during the installation:

• Raspbian with no separate data partition. Would I need this later? Maybe, maybe not. I’ll survive without for the time being.
• Language: en
• Keyboard: fi (I use a Finnish keyboard)

That’s it. I click ‘Install’ and off we go. It takes a while to install, and eventually I get more choices to make via the raspi-config:

• Boot to command line (I intend to make a headless server, with only a network cable / wireless adapter connecting the Raspberry to the external world).
• Change time zone to correct one
• Set Hostname (I named mine srv01)
• Set memory split to 16M for GPU (read about this somewhere in the net, I should not need much memory for the GPU since most of the time I don’t need a monitor directly on the device itself).

Finish and reboot…

Superb, everything checks out fine. I even get an IP address from my DHCP server (home connection router).

SD card woes

Okay, I got the Raspberry Pi 2, 2A 5V power adapter, a micro-SD card (32 GB), HDMI cable, a D-Link DWA-121 WLAN adapter and I think I’m good to go.

First thing to do is to format the SD card. I’m using my MAC for that, since it has an SD card reader. The micro-SD card came with an adapter for the MAC to be able to read the card. I used the program called SDFormatter v4.0.

Plug the micro-SD card in the adapter, make sure the write protect is off and plug it to the Mac. So far so good, SD card visible in Finder. Now a quick format. But alas, my computer thinks the card is write-protected even though the card slider is clearly in “not protected” position.

After a long time banging my head against the write-protect wall, I finally try to move the slider to half-way locked / half-way not locked position. Great success, now I can finally format the card 🙂 I think the SD card locking lever position sensor is malfunctioning on my Mac.

While the computer keeps formatting the card, I visit http://www.raspberrypi.org/downloads/ and download NOOBS 1.4.0 Zip-file.

Formatting completes and I copy/paste the contents of the zip-file without further problems to the card. Time to power up the Raspberry!