Splunk Search History Replication in a Search Head Cluster 

Overview 

In Splunk Search Head Clustering (SHC), user search history is traditionally saved locally on each search head. This leads to fragmented user experiences as search history is only available on the node where it was generated.  The Search History Replication feature resolves this by replicating user search history across all nodes in a cluster, ensuring continuity and accessibility of search history data regardless of the search head the user connects to.

Problem Statement

Previously, search history was stored in CSV files on local file systems of individual search heads. This architecture resulted in data silos, as history was inaccessible when users switched between different SHC nodes. Search history replication centralizes and synchronizes history records across all cluster members using KVStore.

Architecture

Legacy Architecture

In the earlier implementation, search history was maintained in CSV files under $SPLUNK_HOME/etc/users/<user>/<app>/history/. The files were updated at runtime and were not replicated between search heads.

Updated Architecture

Search history replication is enabled by setting search_history_storage_mode = kvstore in limits.conf. This allows history to be shared across all search heads in the cluster.

Configuration Steps

To enable search history replication, edit limits.conf on all SHC members:

[search]
enable_history = true
search_history_storage_mode = kvstore
max_history_length = 500
max_history_time_to_keep = 90d

Once updated with the above configuration, perform a rolling restart of the SHC.

Settings Help:

enable_history = <boolean>

• Specifies whether to keep a history of the searches that are run. Default: true

search_history_storage_mode = <string>

• The storage mode by which a search head cluster saves search history.
• Valid storage modes include "csv" and "kvstore".
• This setting is valid only when the 'enable_history' setting has a value of "true".
• A value of "kvstore" means that the cluster can replicate search history across all its members using the App Key Value Store service.
• A value of "csv" means that search history is saved to CSV files only on the local search head.
• When you initially give this setting a value of "kvstore", the search head migrates the existing search history records, if they are present in existing CSV files, into the App Key Value Store service. This migration of search history can only happen once. If you later change the storage mode to "csv", then back to "kvstore", subsequent migrations do not occur.
• NOTE: In the "kvstore" storage mode, the 'max_history_length' is the maximum number of searches that the SHC can store for each user. In this case, the maximum acceptable value of 'max_history_length' cannot exceed 1000.
• Default: csv

max_history_length = <integer>

• Maximum number of searches to store in history for each user and application.
• When 'search_history_storage_mode' has a value of "kvstore", this value is applicable per user only, and not per user and application combination.
• Default: 500

max_history_storage_retention_time = <integer>[s|m|h|d]

• The maximum time to store search history records for each user and application.
• This setting and the 'max_history_length' setting determine how many search history records appear in persistent storage.
• Search stops storing search history records when either the retention time or the number of search history records exceeds the values you configure with these settings.
• A value of 0 means that search history is retained according to 'max_history_length' only.
• The time units you can specify for this setting are: s, sec, second, secs, seconds, m, min, minute, mins, minutes, h, hr, hour, hrs, hours, d, day, days.
• This setting is only applicable when 'search_history_storage_mode' has a value of "kvstore".
• Default: 90d

Search history migration fom CSV based to KVStore based

•  The first time Splunk restarts after enabling the search history replication feature, after KVStore is ready, splunkd will kick off the migration procedure to move the old CSV-based search history into KVStore based.

10-09-2024 18:12:24.081 -0700 INFO  SearchHistoryMigrate [28255 SearchHistoryMigrateTPoolWorker-0] - Checking if the serch history migration destination is ready

10-09-2024 18:13:22.087 -0700 INFO  SearchHistoryMigrate [28255 SearchHistoryMigrateTPoolWorker-0] - The search history migration destination is ready after seconds=58

• Then it will create a migration task for each search history csv file under $SPLUNK/etc/<user>/<app>/history/<servername>.csv.
  
  On first startup with KVStore mode enabled, Splunk initiates a migration of CSV-based search history to KVStore. Marker files (.sh_migrated) are used to ensure that the migration is not repeated across restarts.
  
  10-09-2022 18:13:22.088 -0700 INFO  SearchHistoryMigrate [28255 SearchHistoryMigrateTPoolWorker-0] - Have created search history migration jobs for users=1 jobs=1 in milliseconds=0
• After migrating each CSV file, a ".sh_migrated" marker file is created, so the csv file wont be migrated again after splunk restarts after it earlier stoped accidentally before finishing migrating all the csv files.

$ ls -ltra etc/users/admin/search/history/.sh_migrated

-rw------- 1 sguo domain^users 0 Oct  9 18:13 etc/users/admin/search/history/.sh_migrated

• After all csv files are migrated, a marker file is created under $SPLUNK/var/run/splunk, so that splunk wont start the migration any more when it restarts next time.

$ ls -ltra var/run/splunk/.sh_migrated

-rw------- 1 sguo domain^users 0 Oct  9 18:13 var/run/splunk/.sh_migrated

After the migration procedure, an user should be able to query the old csv based search history via REST API or the "|history" command

$ curl -k -u user1:changeme https://127.0.0.1:8098/servicesNS/user1/system/storage/collections/data/SearchHistory -XGET|python -m json.tool

OR

$ bin/splunk search "|history"

Search history maintenance

Remove a user’s search history:

A user's entire search history records can be removed by a user with admin role using Splunk REST API. For example:

curl -k -u admin -g 'https://<host>:<mgmt_port>/servicesNS/<user>/system/storage/collections/data/SearchHistory' -X DELETE

Normally a user should not delete his/her history records.  But with Splunk REST API for KV store, there is nothing prevent a user from removing his/her own history records for special purposes.  Key IDs of the records to be removed need to be identified in order to perform this task.

storage/collections/data/{collection}/{id}

•       GET: Get records in a collection by key ID.
•       DELETE: Delete a record in a collection by key ID.

For example:

# get search history records

curl -k -u <username>:<password>    "https://<host>:<port>/servicesNS/admin/system/storage/collections/data/SearchHistory"

# remove specific search history records.

curl -k -u <username>:<password> "https://<host>:<port>/servicesNS/admin/system/storage/collections/data/SearchHistory/ronnie_1672727615.4_D74F956C-BE79-4C95-A669-7E576C28B7F2" -X delete

# remove search history timestamp older in time, using url-encoded query "{"timestamp":{"lt":1672727627000}}"

curl -k -u <username>:<password> https://<host>:<port>/servicesNS/admin/system/storage/collections/data/SearchHistory?query=%7B%22timestamp%22%3A%7B%22%24lt%22%3A1672727627000%7D%7D -Xdelete

Monitoring SearchHistory collection size:

Admin users can monitor the size of SearchHistory collection size via MC (Monitoring Console). 

In Splunk Web, select Monitoring Console > Search > KV Store > KV Store: Instance, select search head instance, browse down Collection Metrics table, you will see SearchHistory collection under app system. 

For the size of records occupied by a specific user, there is no direct way, but you can get an idea by querying all search history records belong to one user via Splunk REST API for KVstore.

Typical Issues

4.1 Search history is not replicated

In most cases this is due to KV store is not started or not functioning properly. Please check if KV store is enabled and in working state.  You can run following command on a search head to check KV store status in the cluster.

$ splunk show kvstore-status

4.2 Some search history records are not replicated during/after restart

Search history replication relies on KV store to replicated data in a cluster.  During restart (including rolling restart) KV store is restarted and out of service for a short period of time, search history may not replicate to some search heads.  These search history records are still on the source search head.

4.3 How to turn on debug log for search history replication?

log-local.cfg

[splunkd]

category.SearchHistory = DEBUG

category.DispatchManager = DEBUG

To see more debug information related to KV Store, turn on the following debug channel.

[splunkd]

category.KVStorageProvider=DEBUG

4.4 Is search history saved in KV Store?

Yes. Below REST API can be used to examine the content in the SearchHistory KV Store collection.

curl -k -u <user>:<password> https://<host-ip>:<mgt-port>/servicesNS/nobody/system/storage/collections/data/SearchHistory -X GET

4.5 How to turn off search history replication?

Set search_history_storage_mode to csv and restart the search head. Remember search history records generated during the period search_history_storage_mode = tkvstore were still saved in KV store, so you cannot see those records after restart.

4.6 max_history_length does not work as expected.

When search_history_storage_mode = kvstore,  max_history_length is capped at 1000.  For example, if you set max_history_length=2000, the system will reset it to 1000.  Another thing is this setting is per user when search_history_storage_mode=kvstore is used.  If search_history_storage_mode=csv, then max_history_length is still per user per app and cap does not apply.

Storage Constraints

Search history information requires space to store.  In a large search head cluster with a large user base, the storage required to store user search history information can be large. The size of the entire collection is not a major concern because only a subset of history records is replicated. We think the concerns about KV store are more related to ES and ITSI large kvstore lookups, where the entire file needs to be replicated. 

The following estimation is based on these assumptions:

•       10 search heads
•        5000 users
•        10 apps for each user,
•        500 bytes history information per record
•        max_history_length = 500

In this feature, search history is stored per user instead of per user per app.  This is to reduce the storage required to store search history into KV store.  max_history_length changed to 500 to further reduce the storage required to store search history information.

Using CSV implementation

Assuming search history need to replicate to all search heads in the cluster

5000 users x 10 apps x 500 search history x 500bytes  =  12.5GB per SH

Total space is 12.5GB x 10 SHs = 125GB

Using KVStore implementation

Search history replicated by KV store

• 5000 users x 500 search history x 600 bytes (500 + 100 extra fields) = 1.5GB collection size
• Total space is 1.5GB x (10-) < 15 GB.

Because KV store manages storage, backup size is even smaller as this is the total size of the replicated collection.

Final Notes

- This feature requires admin privileges for maintenance tasks.
- The migration is one-time and cannot be redone by switching back and forth between CSV and KVStore modes.
- Best suited for environments with Search Head Clustering and multiple active users.

Search history replication improves user experience and reduces storage overhead compared to CSV-based storage.

For more information, explore our Splunk professional services.

Reference:

Splunk limits.conf - limits.conf - Splunk Documentation