Introducing Title Remapper

by Kaushik Thirthappa

Over the last few months, a number of our users have asked if we can add more context to their alerts. We spoke with them on our live chat on dashboard and brainstormed the idea of Title Remapper.

Title Remapper empowers you to programmatically change incident title in real-time for better context.

Let's take Sentry as an example. The incident title is clear but it will do us as incident responders better if we add the project name. Just makes it tad bit clear.

Without title remapper - var temp is not defined
With Title remapper - var temp is not defined for Notification service

See the difference? it immediately adds more context. So much nicer to get this additional info especially on phone call and sms alerts.

Here's how we accomplished this --


Sentry Integration Payload 

{
  project: {
    name: 'Notification service',
    id: project-987
  },
  error: {
    message: 'var temp is not defined',
    id: 1234,
    context: {
      user: 'John Doe',
      customer: 'Acme Inc',
      subscription: 'Pro'
    }
  }
}

By default, our internal parsing logic for Sentry will pick the error message and set it as incident title. However, you can now remap the title to add project name to it.

Here is the Title remapper Handelbars JS code --

{{data.error.message}} for {{project.name}}

{{data.error.message}} = var temp is not defined

{{project.name}} = Notification service

By clubbing multiple variables, we are adding better context for all alerts. Super handy and super easy to get started.

Caveats

  1. One Title remapper can be linked to multiple integrations.
  2. Ensure to link a remapper created for, say AWS, to only be linked with other AWS integrations. If not, there could be errors and your title will be missing
  3. If you link multiple Title remappers to an integration, only the last linked remapper will be taken into consideration.

Background

To test, at first, we monkey-patched the solution in the backend for select customers. Programmatic access essentially had us ideating on giving API access but it seemed impossible since we had to change the incident title while we are creating an incident.

There were 3 main factors to be considered.

  1. Speed
  2. Least learning curve
  3. Flexibility

At the end of the day, we are parsing payloads sent to us at hooks.spike.sh. Changing the title programmatically could be best suited for something like Handlebars JS. We ❤️Handlebars JS

Speed

Sending alerts as fast as I can is incredibly important.

Parsing payload sent to us, changing incident title, creating an incident, determining who to alert, and then sending alerts within 800 milliseconds is crucial. So, speed was definitely one factor to be considered.

Handlebars JS is fast. Our instant thought to create a middleware that users can write their own code came was right. We took some inspiration from Checkly on this. They have done stellar work with adding a similar layer.

Least learning curve

It's incredibly easy for anyone with pretty basic programming knowledge to pick up Handlebars JS. Created by @wycats, it reliably provides very easy templating solutions. To change dynamic titles, all users need to do it access the data and reformat it for more context. There isn't much logic layer that would be needed besides re-formatting your payload into a more meaningful title. It's also safe and reliable.

Extensibility

Since re-formatting is easy with Handlebars JS, we can further extend this feature by providing more context like total suppressed and repeated incident metrics, who resolved it the last time, current responders added, etc. Users can then use data for their ongoing incident and mix it with a simple logic layer and more context.

So, yeah, Handlebars JS ftw.

Time for some examples

All examples below are showcased with the below payload.

Common payload

data: {
  "body": {
    "event_definition_id": "this-is-a-test-notification",
    "event_definition_type": "test-dummy-v1",
    "event_definition_title": "Event Definition Test Title",
    "event_definition_description": "Event Definition Test Description",
    "job_definition_id": "163",
    "job_trigger_id": "8999",
    "event": {
      "id": "NotificationTestId",
      "event_definition_type": "notification-test-v1",
      "event_definition_id": "EventDefinitionTestId",
      "origin_context": "urn:graylog:message:es:testIndex_42:b5etest--id-4-90ed-0dbeefbaz",
      "timestamp": "2021-05-05T09:42:42.823Z",
      "timestamp_processing": "2021-05-05T09:42:42.823Z",
      "timerange_start": null,
      "timerange_end": null,
      "streams": [
        "802109582109518290"
      ],
      "source_streams": [],
      "message": "Notification test message triggered from user <richard>",
      "source": "12837126856181276589235717181276589235716",
      "key_tuple": [
        "testkey"
      ],
      "key": "testkey",
      "priority": 2,
      "alert": true,
      "fields": {
        "field1": "value1",
        "field2": "value2"
      }
    },
    "backlog": []
  },
  "message": "Notification test message triggered from user <richard>"
}

For basic control

{{data.message}}

Output
Notification test message triggered from user <richard>

{{data.body.event_definition_title}}

Output
Event Definition Test Title

[{{data.body.event_definition_type}}] has incident => {{data.body.event_definition_title}}

Output
[test-dummy-v1] has incident => Event Definition Test Title

For advanced control

You can use in-built helpers provided by Handlebars JS. Alongside, we also support a number of helpful helpers on our own fork of Swag.

Avoid #each and #unless or basically all for.. loops
{{#if data.body.event_definition_desc}}
  Title: {{data.body.event_definition_title}}
{{else if data.body.event.message}}
  Message: [{{data.body.event.key}}] - {{data.body.event.message}}
{{else}}
  {{data.message}}
{{/if}}

Output
Message: [testkey] - Notification test message triggered from user <richard>

Swag helpers example

{{uppercase data.message}} with priority {{add data.body.event.priority 1}}

Output
NOTIFICATION TEST MESSAGE TRIGGERED FROM USER <RICHARD> with priority 3

Super happy to see this feature go live for everybody. Do ping us from your dashboard of write us at hello@spike.sh with your ideas 'cos we would love to hear them.

Special thanks to Erik Symonds.