Set up Uptime Monitoring with Kibana, Heartbeat and Slack (Part 2)

In the first part of this series, we went ahead and set up heartbeat to monitor two HTTP servers and send the data to Elasticsearch. We also configured Kibana to automatically create the uptime dashboards for us. For this second part, we are going to include Slack notifications to notify us in case the server goes down.

Before we begin, we have to get the webhook url for your Slack channel. Follow the below instructions to get the webhook url.

1. Log in to slack.com as a team administrator.
2. Go to https://my.slack.com/services/new/incoming-webhook/.
3. Select a default channel for the integration.

4. Click Add Incoming Webhook Integration.

After the incoming webhook has been generated, copy the url and add it with the following configuration in the elasticsearch.yml file in the directory elasticsearch/config/.

xpack.notification.slack:
  account:
    monitoring:
      url: <WEB-HOOK-URL>
      message_defaults:
        from: x-pack
        icon: http://example.com/images/watcher-icon.jpg
        attachment:
          fallback: "X-Pack Notification"
          color: "#36a64f"
          title: "X-Pack Notification"
          title_link: "https://www.elastic.co/guide/en/x-pack/current/index.html"
          text: "One of your watches generated this notification."
          mrkdwn_in: "pretext, text"

After you add this to your elasticsearch.yml, you have to restart all the containers for the changes to take effect. Run:

$docker-compose restart $(docker-compose ps -q)

Once the containers have restarted, head over to Kibana, under Management menu, under Elasticsearch, click the Watcher option and create a new threshold alert

Setting Up Watcher

1. Using the UI (Kibana 6.6 and above)

With the Kibana 6.6, you can now configure the notifications schedule and message with the UI. After entering the name, indices and time field, you will be provided by a set of options like below:

Watcher Conditions for firing a Slack Notification

These options make Kibana aware of the conditions requires to fire the actions such as Slack notifications. For this case, the conditions are:

WHEN count() GROUPED OVER top 5 http.response.status_code IS ABOVE 299 FOR THE LAST 5 minutes

HTTP codes 200 to 299 represents a successful request, so we want to fire an action when we get responses with status codes greater than 299. Next we will move to actions.

Slack notification to be sent once condition is met

For this case, you will pick the Slack action on the drop down. (If Slack wasn’t configured properly in elasticsearch.yml, it will be grayed out). The recipient can be a user or a channel (remember to add a # symbol for channels), and the message can be customized to include the error message and code as below:

Encountered  {{ctx.payload.hits.total}} errors in the last 5 minutes: \n {{ctx.payload.hits.hits}} *Error Message*: _{{_source.error.message}}_, \n *HTTP Status*: _{{_source.http.response.status}}_ \n {{ctx.payload.hits.hits}}

2. Using JSON (Kibana 6.5 and below)

The below JSON can be used to create an advanced watch or if you’re using Kibana version 6.5 or below where there is no option to use the UI.

{
  "trigger": {
    "schedule": {
      "interval": "5m"
    }
  },
  "input": {
    "search": {
      "request": {
        "search_type": "query_then_fetch",
        "indices": [
          "heartbeat*"
        ],
        "types": [],
        "body": {
          "query": {
            "bool": {
              "must": [
                {
                  "query_string": {
                    "query": "monitor.status:down"
                  }
                },
                {
                  "range": {
                    "@timestamp": {
                      "gte": "now-5m"
                    }
                  }
                }
              ]
            }
          }
        }
      }
    }
  },
  "condition": {
    "compare": {
      "ctx.payload.hits.total": {
        "gt": 0
      }
    }
  },
  "actions": {
    "send_trigger": {
      "throttle_period_in_millis": 3600000,
      "slack": {
        "message": {
          "from": "heartbeat",
          "to": [
            "#heartbeat-monitoring"
          ],
          "text": "Heartbeat Monitoring",
          "icon": "https://raw.githubusercontent.com/elastic/elasticsearch-net/master/build/nuget-icon.png",
          "attachments": [
            {
              "color": "danger",
              "title": "Server Down",
              "text": "Encountered  {{ctx.payload.hits.total}} errors in the last 5 minutes: \n {{#ctx.payload.hits.hits}} *Error Message*: _{{_source.error.message}}_, \n *HTTP Status*: _{{_source.http.response.status}}_ \n {{/ctx.payload.hits.hits}}"
            }
          ]
        }
      }
    }
  }
}

Further explanations for each of the properties in the above JSON object can be found here.

And that’s it!

You can save your watch at this point and try to simulate it using the simulate menu in Kibana. You should receive a message like this from slackbot:

Feel free to reach out and ask any questions regarding the above set up!