Frustration with my home network’s WAN connectivity randomly flapping and being somewhat cheap led me to see if I could get paged without having worry about email relays or paying for your traditional notification service à la PagerDuty (pricey) or VictorOps (unreliable).

The first order of business was to write up a simple script to perform some checks against my home network.  It’s a simple shell script run through cron that maintains a running log of check attempts and results.  Each time it runs, reads the last result, performs new checks, and if the state changed from the previous attempt, then send out an alert.

The code for the script is below or check it out at GitHub.

#!/bin/bash

## Very basic check to see if home network is up.
## First attempt SSH connection to server.
## If this fails, then attempt to ping router.
## Record result of check to logfile.
## If current check status is different than last check status
##    sent message to Slack via webhook

## Host and SSH port to check
hName="[HostName]"
port="[Port]"

## Slack Hook and Channel
slackWebHook="https://hooks.slack.com/services/[WebHookLink]"
slackChannel="@[SlackUserName]"

## LogFile to maintain check state.
logFile="/[someLogDirectory/check-home.log"

## Send a color coded message to Slack
function notifySlack () {
  status=$1
  hostname=$2
  result=$3

  case "${status}" in
    OK)
      color="good"
      ;;
    WARNING)
      color="warning"
      ;;
    CRITICAL)
      color="danger"
      ;;
    *)
      color="#909090"
      ;;
  esac

  payload="\"attachments\": [{ \"title\": \"${hostname} status is ${status}\", \"text\": \"${result}\", \"color\": \"${color}\" }]"
  if [ ! -z "${slackChannel}" ]; then
     curl -s -XPOST --data-urlencode "payload={ \"channel\": \"${slackChannel}\", ${payload} }" ${slackWebHook} > /dev/null 2>&1
  else
     curl -s -XPOST --data-urlencode "payload={ ${payload} }" ${slackWebHook} > /dev/null 2>&1
  fi
}

## Get previous status
lastEvent=`tail -1 ${logFile}`
lastStatus=`echo ${lastEvent} | awk '{ print $4 }'`

### First Check Host if SSH is up.
results=`echo QUIT | nc -v -w 5 ${hName} ${port} 2>&1 | grep -v mismatch`
res=`echo ${results} | awk '{print $5}'`

## Check connection status, set to WARNING if this fails
if [ "${res}" != "open" ]; then
   echo `date +"%h %d %T"` WARNING $results >> ${logFile}
   status="WARNING"
   
   ## Try pinging host
   results2=`ping -c 5 ${hName} | grep packets`
   percent=`echo ${results2} | awk '{ print $6 }' | sed -e 's/\%//'`
   if [ "${percent}" -gt 0 ]; then
      echo `date +"%h %d %T"` CRITICAL $results2 >> ${logFile}
      status="CRITICAL"
   fi
else
   ## Otherwise we are okay
   echo `date +"%h %d %T"` OK $results >> ${logFile}
   status="OK"
fi

## Do we notify Slack?
if [ "${status}" != "${lastStatus}" ]; then
   results=`tail -1 ${logFile}`
   notifySlack "${status}" "${hName}" "${results}"
fi

The script is run simply through cron and I have it scheduled to run every 2 minutes from a machine outside my home network.  For myself, I don’t have the script post to a channel, but rather it sends a direct message.  Then in my Slack notification settings, I make sure it sends a push notification for direct messages and for certain words/phrases.

In the above, the system alerted when its SSH check against my home machine failed.  I only consider this a warning as the overall network could still be okay.  The logic flow dictates, that if the SSH fails, then it attempts to ping the router.  The below shows an alert and recovery related to the WAN interface dropping off the Internet.

Overall, it works pretty well.  Using a script and cron is pretty rudimentary monitoring, but it accomplished the basic need.  These alerts have allowed me to narrow down my search through the logs from my home router to track down its problems.

I received this fortune in a cookie from lunch the other day.
  Lunch was catered Chinese
    in the office of my current employer;
      an Internet technology start-up
        where I stand in front of computer screens
          For large portions of the day.

I manage a team of computer people.
  More often called System Administrators, or
    Network Administrators, or
      Site Reliability Engineers, or
        Site Reliability Operators, or
          IT, Ops, or DevOps*.

Some of my time involves meetings
  with software developers and engineers
    solving software and service architectural problems
      and coming to agreements on how to progress forward.

Sometimes I even write code
  something I’ve been doing since 1983.
    Other times I setup, configure, and maintain
      large scale Internet infrastructures and web services.
        The combination of those two functions has been my profession since 1993.

Over 20 years in this field.
  It doesn’t really seem that long.
    Probably because I’ve done well by it
      And for the most part enjoyed the work very much.

I guess this fortune cookie turned out true
  If maybe a bit after the fact had been well settled.
    Had I received it 30 years ago, though,
      I would’ve just said, “Duh, I’m going to do that anyway.”

* I consider DevOps to be an organizational structure and philosophy focused on development and operational practices.  It is not IMHO a job title or position.  Using it as such is frequently a sign that someone doesn’t know what they are talking about.

Alcohol content: a wee dram (Macallan Fine Oak 21 Year)

I’ve spent my life recording numbers to generate graphs to make sense of the world.  The numbers are what’s important, but humans do not really gain comprehension staring at rows and columns of numbers.  The human brain is made for visual processing.  A glance at a well plotted graph can present more value than attempting to read through the millions of data points that made up the graph.  The plot illuminates the data and should allow a reasonable person to intuit meaning.  I’ve probably tossed many kilograms of notebooks filled with data sets and plotted graphs.  Almost no one ever cared about the numbers, just the resulting graphs.

Figures at Work

I’ve used GIFs generated with ancient versions of GNU Plot  to analyze performance data.  The pain of creating the graphs turned justified when insights were gained from simple visualizations.

Certainly, in my line of work pretty pictures help explain what’s going on to a wider audience than the raw numbers.  You can immediately look at a picture and notice when something appears out of the ordinary.

Or…

Other times, I’ve needed to translate the data to justify costs or ways to lower costs.  The following graph helped start a conversation with finance to purchase Amazon AWS Reserved Instances.  A full multi-sheet spreadsheet explaining the financial breakdown was still needed for full justification.  But without this and few other pretty pictures, the discussion and the approval of the up front costs would never have happened.

But that’s just work and part of the expected drudgery of justifying your department’s daily existence.  You do them like annual reviews and power point presentations; get them done to prove your point, then try to get back to real work…  Unless you are a scientist (data or otherwise), in which case, collecting data points and graphing the results might be your real job.

Zen and the Art of Keeping Track of My Car

But even before I ever had to plot a graph in school, I watched my dad keep a small notebook in every car to record the mileage, fuel added, fuel costs, etc.  He’d calculate the MPG since the last fuel stop and record that, and then reset the tripmeter.  He’d also indicate when he’d perform an old change or some other service on the car.  It was a valuable service record for the car.  Noticing a change in MPG could also indicate that something .  I continued the tradition when I started driving my own cars.  And it didn’t take me long to realize there was so much more information in those numbers if I could plot them.

Automotive gas prices clearly cycle over time, but over the years have been trending down in Northern California.  The slight uptick at the end could mean summer prices are coming.  But in this case, it’s more a matter of which town I needed to get gas.

How’s my driving?  So as I drive more, a higher percentage of my time spent in this car is commuting.  Commuting seems to be so much more fuel efficient than track days and auto-crossing.

I’m driving this car a whole lot more now.

Example Google Sheets Logbook

Always Look on the Bright Side of Life

Maybe all this tracking, accounting, and graphing is sign of some compulsion on my part.  Certainly when it comes to question of my personal health, I become obsessive about tracking as many variables as possible.  There have been a few times when I’ve been not entirely healthy and the cause was not immediately apparent.  When a doctor or group of doctors become interested in you, life becomes less than pleasant.  On more than one occasion, after months of specialists and testing my body has sorted itself out.  Whatever was wrong resolved itself with no definitive answer for a cause.  The interminable wait between lab results and doctors visits, though, left for too much time for fidgeting and speculation.

Until wearable medical devices came around, I kept log books.  During one of these months long investigative sessions my log filled dozens of composition book pages.  Over three months I documented when symptoms would occur, the severity, duration, what remediation steps were taken, and their efficacy.  One specialist upon reading through my notes suggested I was adding to my general stress levels.  He did not come right out and suggest this additional stress was the cause of my ills, but I suspect that was on his mind.  I won’t discount that possibility entirely.  But sitting around waiting for something to happen and someone else to do something felt worse.

With a wearable, especially one that can track heart rates, the paper log book becomes a bit less necessary.

There’s still a lot of manual data entry than has to go on, but apps like Apple Health and My Fitness Pal make it less tedious.  Apple’s Health app has horrible graphing in my opinion and without paying for it, it’s difficult to get all your data out of FitBit.  Using QS Access on the iPhone helps to move the data points into a spreadsheet like Google Apps or Excel where you can manipulate and visualize the data.

I still don’t know what’s going on with my current malady.  At least by recording data points and making graphs I feel like I’m doing something for my treatment while I wait for the next test or visit.

Alcohol content: measurements not recorded