Watchdog: reboot Raspberry Pi if network is down

Network down! Help!

I have an issue with one of my Raspberry Pis. It will not survive if Wi-Fi network is down for an extended period of time.

I have not been able to figure out the root cause, or how long the network needs to be unreachable before the Raspberry Pi just drops the network connection and never connects again. This happens occasionally since the Pi is located just a little too far away from my Wi-Fi access point. Due to real estate fire prevention regulation reasons I do not have a cable route available so wireless is the way to go.

Now this wouldn’t be such a big issue, but the Raspberry Pi is a headless one (i.e. it does not have a keyboard or monitor), and is only connected via Wi-Fi network. I have some sensors connected to this Pi, which in turn update a web page. Losing network connectivity causes web page data to be out of date and is quite inconvenient.

Today after work I noticed that the Pi had lost network connectivity (again) at 04:40. I had to make my way to the garage (again) and reboot the device (which has no keyboard or network connectivity) – and my frustration level rose to “I’ll fix this now!” level.

Right. After a few half-assed attempts at ping + reboot scripts I figured there must be a more elegant solution, and I was sure I’m not the first one with this problem.

Enter Watchdog.

Searching the Internet I found a couple of examples, none of which was immediately copy/paste ready for my needs.

They did lead me to right track and after reading some manpages, forums and gist.github.com code snippets, getting my Pi to a constant reboot loop etc. I finally came up with what seems to be a working solution.

The solution

First things first: I used a Raspberry Pi 2 with a USB Wi-Fi module and Raspbian Wheezy (4.1.19-v7+ #858 SMP Tue Mar 15 15:56:00 GMT 2016 armv7l GNU/Linux).

Start by installing watchdog:

sudo apt-get install watchdog

Make a backup copy of /etc/watchdog.conf file just in case:

sudo cp /etc/watchdog.conf /etc/watchdog.conf.backup

Edit the /etc/watchdog.conf file to contain the following. There is a short comment on each line about what they do:

$ sudo nano /etc/watchdog.conf
# Watchdog ping: if unresponsive, reboot:
interface = wlan0    # use interface wlan0
ping-count = 5       # ping 5 times
ping = 192.168.1.1   # ping test destination IP address
# Change default interval from 1 second to 20:
interval = 20        # perform watchdog checks every 20 seconds

then reboot (e.g. sudo reboot).

The above will ping five (5) times for destination address 192.168.1.1 every 20 seconds. I’m not sure if the interface command is actually needed, but it did not do any harm so I left it there.

192.168.1.1 is my default gateway, and I really do want to test connectivity against this address instead of some random host in the Internet, since I don’t want my Pi to reboot in case the Internet connection is down. If you insist on pinging a host in the Internet (not recommended), good choice might be Google public DNS servers (8.8.8.8 and 8.8.4.4) or any other host of your choice.

I did use Google DNS server for testing purposes, since it was easier to cut the connection to the Internet but maintain local area network (LAN) connectivity for management purposes.

Watchdog writes log to syslog (/var/log/syslog), and when the ping test fails, this is what it will look like (note the target here is Google DNS 8.8.8.8, not my internal network default gw):

Oct 24 20:35:42 localhost watchdog[2640]: ping: 8.8.8.8
Oct 24 20:35:42 localhost watchdog[2640]: no response from ping (target: 8.8.8.8)

When there is no response to any of the five (5) ping echo requests, the Raspberry will reboot. I will forget this, so I inserted the following to /home/pi/.profile:

echo "Warning: If network is down, this system will reboot in 20 seconds. Comment out ping = 192.168.1.1 line from /etc/watchdog.conf to avoid reboots."

That will print a warning message plus info which file to configure if needed – every time I log in. Even I should be now able to remember where Watchdog is configured 🙂

Further reading

Below you can find a list of some resources which did help me with the solution:

1. Watchdog man page: https://www.systutorials.com/docs/linux/man/8-watchdog/
2. Watchdog.conf man page: https://www.systutorials.com/docs/linux/man/5-watchdog.conf/
3. Good explanation of watchdog.conf ping variable: http://www.sat.dundee.ac.uk/psc/watchdog/watchdog-configure.html#Network_ping
4. Reddit, a solution which almost worked: https://www.reddit.com/r/raspberry_pi/comments/4ih9xo/id_like_my_routeronastick_vpn_to_autorestart/d2y3yj4/?context=3
5. Non-watchdog script to solve the same issue (not mine): https://gist.github.com/SandroMachado/87e591fc42f368636b251b566485ae46
6. Another non-watchdog script (again, not mine): http://weworkweplay.com/play/rebooting-the-raspberry-pi-when-it-loses-wireless-connection-wifi/

Improvements

I have only had this setup running now for a couple of hours, so it’s unclear if it will really work over time. Hope so, I’ll update this article if needed.

What else I could do with Watchdog? Well, I could certainly improve this to check that my Python programs do not die – or at least react and restart them automatically when they do die. Or if the system bogs down and stays unresponsive for extended periods of time.

What do you think? If you have suggestions, improvements or indeed more experience than I do, please do leave a comment below.

Follow-up

Update 30.3.2018: Reminded by Will in the comments (thanks!), I noticed I have not followed up on my promise to update this article.

I have not had any problems with performance or reboot loops etc. with the script. It just works as intended. The 20 second interval is a bit tight, but it’s entirely doable to stop the script in that time if needed.

How would I improve this? Well, I’d create a separate reboot log instead of relying on syslog, but it’s really not necessary. It would help for statistics collection over a longer time period but like said, not necessary.

perl: warning: Please check that your locale settings …

TLDR

My issue was solved by making sure the following content was in /etc/default/locale:

defgw@proto-pi ~ $ cat /etc/default/locale
#  File generated by update-locale
LANG=en_GB.UTF-8
LANGUAGE=en_GB:en
LC_ALL=en_GB.UTF-8

Problem

I recently installed a new Raspbian Scratch image on my protoboard, i.e. on a throwaway Raspberry Pi memory card for testing purposes.

Installing new packages came with an error like below:

$ sudo dpkg-reconfigure locales
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = "en_US.UTF-8",
LC_ALL = "en_US.UTF-8",
LC_CTYPE = "UTF-8",
LANG = "en_GB.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_GB.UTF-8").
locale: Cannot set LC_CTYPE to default locale: No such file or directory
locale: Cannot set LC_MESSAGES to default locale: No such file or directory
locale: Cannot set LC_ALL to default locale: No such file or directory
/usr/bin/locale: Cannot set LC_CTYPE to default locale: No such file or directory
/usr/bin/locale: Cannot set LC_MESSAGES to default locale: No such file or directory
/usr/bin/locale: Cannot set LC_ALL to default locale: No such file or directory

This is definitely not ideal, and googling for a solution came up with various instructions and most related to Ubuntu (maybe this is a more prevalent issue among Ubuntu community?). I did not find a good solution, so I decided to find out where these settings are located.

Fast forward here, a lot of searching, trial and error ensued…

Solution

The locales are configured in /etc/default/locale, and for some reason mine only had the following content:

#  File generated by update-locale
LANG=en_GB.UTF-8

Adding the lines in the error message and changing everything to en_GB.UTF-8 to be in line with the existing LANG setting (and then rebooting for good measure) removed the issue:

$ cat /etc/default/locale
#  File generated by update-locale
LANG=en_GB.UTF-8
LANGUAGE=en_GB:en
LC_ALL=en_GB.UTF-8

The problem is now solved and the error message is gone.

Feedback

If you know whether I have messed something up during Raspbian installation (raspi-config?) or perhaps this is default behaviour, please let me know in comments below.

 

apt-get search?

I wanted to know how to find out which apt-get package will give me an application x. For example, nslookup.

Running a simple “apt-get install nslookup” will not do, since there is no such package.

After some digging, I found that I can search apt-cache, and it actually contains information about applications.

Try this:

$ apt-cache search nslookup

Output:

dnsutils - Clients provided with BIND
knot-dnsutils - Clients provided with Knot DNS (kdig, knslookup, knsupdate)
libnet-nslookup-perl - simple DNS lookup module for perl

Now, installing dnsutils will install also nslookup. Handy:

apt-get install dnsutils

Collecting data from serial port and CSV handling

Hello,

I’ve managed to set up my Arduino board (http://arduino.cc/) with a temperature probe (among other things) and connected it to my Raspberry Pi via USB port. The temperature probe is a Dallas Semiconductor DS18B20 with a stainless steel casing (such as this: http://www.adafruit.com/products/381), and it’s wired outside my house through a ventilation hatch to measure external temperature. Actually there are two sensors – one to measure temperature outside, and another to measure temperature indoors next to the Arduino.

I have code on the Arduino to write out temperatures via the USB connection, and on the Raspberry Pi I have another bit of code to read the serial input and save it as a CSV file, see the code below. It’s originally by Ben Kenney, and I modified it to suit my needs. I hope the copy/paste worked, had some difficulties until I figured out I can use the pre-tag.

Update 11.2.2017: If you get this error: ImportError: No module named serial, just install Python serial module: sudo apt-get install python-serial.

#!/usr/bin/python

'''
Original data collection script by Ben Kenney - July 2012
This program reads data coming from the serial port and saves that data to a text file. It expects data in the following format with a comma (,) as a separator:
"value1,value2"
It assumes that the Arduino shows up in /dev/ttyACM0 on the Raspberry Pi which should happen if you're using Debian.
'''

import serial
from time import strftime
from datetime import datetime, time

ser = serial.Serial('/dev/ttyACM0',9600)

startTime = datetime.now()
try:
    while 1:
        line=ser.readline().rstrip()
        temp,outsidetemp=line.split(",")
        now = datetime.now()
        elapsedTime = now-startTime
        elapsedSeconds = (elapsedTime.microseconds+(elapsedTime.days*24*3600+elapsedTime.seconds)*10**6)/10**6
        f=open('/home/pi/sensors/sensordata/temperaturedata.csv','a')
        print >>f,("%s,%s,%s,%s"%(now.strftime("%Y-%m-%d %H:%M:%S"),elapsedSeconds,temp,outsidetemp))
      f.close()
except KeyboardInterrupt:
print "\ndone"

I do understand that I could also use the RPi GPIO ports, but I have no experience using GPIO. I do have some experience with Arduino, so I opted to use one of the Arduino boards lying around.

During the last few weeks of operation, I’ve accumulated quite a lot of data and realized that over time I will fill up my SD card on the RPi if I don’t find out a way to manage the gathered data.

The script that listens to the serial port writes each output from the Arduino on a separate line and adds a timestamp in the beginning of the line (now.strftime). Thus I thought I could grep my way out of this situation, but I found a bash command that seemed more elegant. Find my solution below.

I decided that I do want to save the old data in a monthly file, for possible future use. So the first thing to do is to make a copy of last months rows in my csv-file, so I can copy it to external computer for archiving purposes:

grep 2015-03 temperaturedata.csv > temperaturedata_2015-03.csv

Next, I will remove all rows with March 2015 date from the original file:

sed -i "s/.*2015-03.*//g" ./temperaturedata.csv

This will not remove the rows themselves, only the contents on that row, so I still need one more command to erase all empty rows in the file:

sed -i "/^$/d" ./temperaturedata.csv

Copy the archived file temperaturedata_2015-03.csv to another computer and delete it from local computer and we’re done for now.

Now that I have all the lines of code available, I could create a script which uses the system date as a variable and run the cleanup scripts on the 1st day of each month. Mind, removing last month data, not current month :). That will remain as an exercise for later, though. For now I’m content I managed to clean up the large CSV file without affecting the data collection operation.

Note: The commands above are the result of Googling and applying modifications to suit my needs, not that I could have written those completely myself with my close to zero experience.

First login and static IP address configuration

UPDATE 28.3.2016: Static IP address configuration has apparently changed:

1. Leave /etc/network/interfaces pretty much alone.
2. Configure /etc/dhcpcd.conf with static IP address entries.

See the bottom of this post.

---

 

After successful installation and first boot, I’m about to log in for the first time.

Logging in and changing the default password immediately:

pi@srv01 ~ $ passwd
Changing password for pi.
(current) UNIX password: raspberry
Enter new UNIX password:
Retype new UNIX password:
passwd: password updated successfully
pi@srv01 ~ $

checking if I actually have an IP address:
pi@srv01 ~ $ ifconfig
eth0 Link encap:Ethernet HWaddr b8:27:eb:xx:xx:xx
inet addr:192.168.1.29 Bcast:192.168.1.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:24799 errors:0 dropped:512 overruns:0 frame:0
TX packets:10333 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:2917645 (2.7 MiB) TX bytes:5436810 (5.1 MiB)

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:1088 errors:0 dropped:0 overruns:0 frame:0
TX packets:1088 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:121896 (119.0 KiB) TX bytes:121896 (119.0 KiB)

pi@srv01 ~ $

I do: 192.168.1.29, so let’s update for the first time:

pi@srv01 ~ $ sudo apt-get update

Uh oh, this end with an error. Can’t even ping the Internet hosts, such as http://www.google.com. Can’t be a DNS error, since 8.8.8.8 does not answer either. I can ping my defgw 192.1768.1.1 however, so the problem is not on the local LAN. Quick reboot to the cable modem, and everything is working again.

apt-get completes successfully, and I run sudo apt-get upgrade to actually upgrade the installed packages.

Time to change a static IP address so I can connect to the device remotely and disconnect the keyboard and monitor.

The IP address is set in /etc/network/interfaces text file. Checking the contents can be done with command cat /etc/network/interfaces. To actually change the contents, you must have superuser rights, and be familiar with some text editing software, such as vi or nano. I prefer to use nano on my devices, since it’s easier to use in small scale editing.

pi@srv01 ~ $ sudo nano /etc/network/interfaces

I don’t have saved the original configuration from the file, but below is a working example of all the contents of the file, with a static IP address of 192.168.1.105:

auto lo

iface lo inet loopback
iface eth0 inet static
address 192.168.1.105
netmask 255.255.255.0
network 192.168.1.0
broadcast 192.168.1.255
gateway 192.168.1.1

allow-hotplug wlan0
iface wlan0 inet manual
wpa-roam /etc/wpa_supplicant/wpa_supplicant.conf
iface default inet dhcp

Finally, using the command raspi-config I can go back to the install selection box and make sure SSH server is running.

I’m ready to power off and move the server to it’s proper location. I enter the following command to shut down the server:

pi@srv01 ~ $ sudo shutdown -h now

Disconnect cables, move the server physically to another location, connect network cable and power cable and the server will start itself with IP address 192.168.1.105. Check using my Macbook if I can ssh to it from a terminal window, and a login screen is waiting for me.
ssh pi@192.168.1.105
pi@192.168.1.105's password:

---

UPDATE 28.3.2016: Static IP address configuration has apparently changed:

1. Leave /etc/network/interfaces pretty much alone:

My config for eth0 (without wireless) is currently as follows:

auto eth0
allow-hotplug eth0
iface eth0 inet manual

2. Configure /etc/dhcpcd.conf with static IP address entries:

Add the following to the very end of /etc/dhcpcd.conf:

# Static IP configuration
interface eth0
static ip_address=192.168.1.120/24
static routers=192.168.1.1
static domain_name_servers=8.8.8.8 8.8.4.4

That’s it. For more information, see the source where I found this info:

https://www.raspberrypi.org/forums/viewtopic.php?f=63&t=110606

And scroll down to post by “paulv » Tue Oct 06, 2015 8:51 am”

I haven’t gotten around to test with wireless yet, but I’ll update the post if I get the chance.

 

First install

After successfully copying the NOOBS installation files to the SD card, I’m now ready to connect all the cables:

• Monitor to HDMI connector
• Network cable to RJ-45 connector
• Mouse and keyboard to USB
• micro-SD card to the correct location
• power adapter to power input connector and power the thing up.

Seems like everything is in place, the device boots properly. I make the following selections in the menu presented during the installation:

• Raspbian with no separate data partition. Would I need this later? Maybe, maybe not. I’ll survive without for the time being.
• Language: en
• Keyboard: fi (I use a Finnish keyboard)

That’s it. I click ‘Install’ and off we go. It takes a while to install, and eventually I get more choices to make via the raspi-config:

• Boot to command line (I intend to make a headless server, with only a network cable / wireless adapter connecting the Raspberry to the external world).
• Change time zone to correct one
• Set Hostname (I named mine srv01)
• Set memory split to 16M for GPU (read about this somewhere in the net, I should not need much memory for the GPU since most of the time I don’t need a monitor directly on the device itself).

Finish and reboot…

Superb, everything checks out fine. I even get an IP address from my DHCP server (home connection router).

SD card woes

Okay, I got the Raspberry Pi 2, 2A 5V power adapter, a micro-SD card (32 GB), HDMI cable, a D-Link DWA-121 WLAN adapter and I think I’m good to go.

First thing to do is to format the SD card. I’m using my MAC for that, since it has an SD card reader. The micro-SD card came with an adapter for the MAC to be able to read the card. I used the program called SDFormatter v4.0.

Plug the micro-SD card in the adapter, make sure the write protect is off and plug it to the Mac. So far so good, SD card visible in Finder. Now a quick format. But alas, my computer thinks the card is write-protected even though the card slider is clearly in “not protected” position.

After a long time banging my head against the write-protect wall, I finally try to move the slider to half-way locked / half-way not locked position. Great success, now I can finally format the card 🙂 I think the SD card locking lever position sensor is malfunctioning on my Mac.

While the computer keeps formatting the card, I visit http://www.raspberrypi.org/downloads/ and download NOOBS 1.4.0 Zip-file.

Formatting completes and I copy/paste the contents of the zip-file without further problems to the card. Time to power up the Raspberry!