Splunk ITSI–ServiceNow Integration: Cutting Incident Delay to 1 Minute

The customer encountered a problem with their ServiceNow integration, where incidents generated through the ITSI NEAP policy were delayed. The customer was using Splunk ITSI version 4.18.  There was an average delay of five minutes from the time a noteworthy event was triggered from the correlation search. which was then grouped into an episode and ultimately triggered the ServiceNow incident creation action.

Their ServiceNow automation workflow was affected by this delay because critical services required incidents to be created almost immediately. They had already adjusted every conceivable ITSI NEAP policy parameter, but they had run out of ways to improve the workflow even more. The customer wanted to shorten the time it took to create incidents without compromising the logic used to create ITSI alerts. 

The Solution 

To optimize the incident creation process, we proposed creating the ITSI NEAP policy by triggering ServiceNow incident creation directly from the Splunk ITSI correlation search. This method eradicated the waiting time and hence reducing the delay between event detection and incident creation. 

However, this new approach bought in two different hurdles. 

• Initially, irrespective of the quantity of the results, only one incident was created when the search was run.
• Another major concern was that the customer's custom ServiceNow integration was not returning the ServiceNow incident number to the incident creation request, it was difficult to track and link the incident to the pertinent ITSI episode. We created a two-part search-based solution to address these issues: 
  1. Primary Search: This search used the sendalert and map commands in combination with the ITSI event creation and ServiceNow alert actions to produce ITSI notables and ServiceNow incidents directly from the correlation search. This ensured that incidents would be generated more quickly. Setting this primary search to run every two minutes reduced the time required to create a ServiceNow incident from five minutes to one and a half minutes. 
  2. Maintenance Search: A secondary search was created to examine the information from a custom ServiceNow table input in order to retrieve the incident ID. This search ensured that the ServiceNow incident numbers were appropriately linked to the relevant ITSI episodes, maintaining system synchronization, by utilizing the itsi_event_action_link_ticket action.

The following search examples demonstrate how we implemented Splunk ITSI ServiceNow automation using sendalert and map commands.

Below is a simplified representation of the Main search:

# Base event filter

index="itsi_summary" (indexed_is_service_aggregate::0) (alert_level>=3)

# Enrich the event with service and KPI information

| lookup service_kpi_lookup _key as itsi_service_id OUTPUT title AS service_name

| lookup itsi_kpi_attributes kpiid AS kpiid OUTPUTNEW *

# Create the ITSI notable event

| sendalert itsi_event_generator param.title=%itsiNotableTitle% param.description=%itsiDescription% param.severity=%alert_level%

# Create ServiceNOW incident for each notable

| map search="

    | makeresults

    | sendalert snow_incident_custom

        param.cmdb_ci=\"$cmdb_ci$\"

        param.impact=\"$impact$\"

        param.severity=\"$severity$\"

        param.mon_object=\"$mon_object$\"

        param.assignment_group=\"$assignment_group$\"

        param.short_description=\"$short_description$\"

        param.description=\"$description$\"

        param.incident_occured=\"$incident_occured$\"

        param.work_instruction=\"$work_instruction$\"

        param.correlation_id=\"$correlation_id$\"

# Display key tracking fields

| table groupingid snow_id alert_level incident_occured_epoch snow_correlation_id itsi_group_id itsi_group_status

# Record the results for maintenance/lookup purposes

| outputlookup append=true snow_tickets.csv

Below is a simplified representation of the Maintenance search which attached the ticket to the episode:

|inputlookup snow_tickets.csv

|join [<fetch the Snow event ID from the data input index>]

|join [<fetch the ITSI Episode ID from the ITSI grouped alerts index]

| lookup itsi_notable_group_user_lookup _key AS itsi_group_id OUTPUT status as itsi_group_status]

| lookup itsi_notable_event_external_ticket event_id as itsi_group_id OUTPUT tickets.ticket_id as external_ticket_id

| outputlookup snow_tickets.csv

| where isnull(external_ticket_id)

| eval external_ticket_id = snow_id

| map search="| makeresults

    | eval itsi_group_id=\"$itsi_group_id$\"

    | eval external_ticket_id=$external_ticket_id$

    | sendalert itsi_event_action_link_ticket

        param.operation=\"upsert\"

        param.ticket_id=\"$external_ticket_id$\"

        param.ticket_system=\"Service Now\"

       param.ticket_url=\"https://<Service-now-hostname>/incident.do?sysparm_query=number=$external_ticket_id$\""

Outcome

The itsi_event_generator action with sendalert command allowed creation of notable without adding the action. The map command allowed to trigger the snow_incident_custom action for every search result and the itsi_event_action_link_ticket allowed to link the Service Now tickets to the respective episodes. 

By adopting this two-layered Splunk ITSI and ServiceNow integration approach, we helped the customer reduce the incident creation delay to under a minute, ensuring faster response times and improved ITSI-ServiceNow workflow efficiency.  

For more information, explore our Splunk professional services.