Customizing and Enhancing Splunk

So what is Splunk? At its core Splunk is
a search engine. It was designed to
allow any data from an infrastructure device to be indexed and searched. Any output
from applications, servers and network devices can be “eaten” by Splunk. However,
Splunk has become more than just a standalone product. The current 3.x series of the
product has opened up the internal API and exposed it to allow outside development
of new applications on top of the Splunk core. This post is going to touch on some
of the capabilities available to developers looking to get even more out of their
Splunk installation.

I am going to be discussing two
elements of Splunk that a user can customize and enhance in the current product release:
Spunk UI customization and RESTful applications.

UI Customization

The
Splunk web UI would definitely be classified as a Web 2.0 application. It relies heavily
on the use of asynchronous Javascript and XML (also known as
AJAX) and cascading style sheets (CSS).

Splunk uses a set of CSS and supporting
HTML and image files to provide themes to the end user. Creating a new theme for use
within Splunk is as simple as cloning one of the existing theme’s CSS and supporting
files and editing it. I would not recommend trying to create a new theme from scratch
as you may miss one of the necessary elements used in the UI resulting in your Splunk
UI appearing to be broken.

The base directory for Splunk’s
themes is $SPLUNK_HOME/share/splunk/search_oxiclean/.

Within that directory are the CSS files, $SPLUNK_HOME/share/splunk/search_oxiclean/static/css/skins/, and supporting image files,
$SPLUNK_HOME/share/splunk/search_oxiclean/images/skins/.

To help in creating a new theme
I highly recommend using the Firefox
plugin Firebug to identify
which element in the CSS you need to modify to affect the look of the UI.

A simple example of a customization
would be to replace the Splunk logo in the upper left corner of the search area with
your organizations logo. To do this clone one of the three default themes and modify
the following sections (also don’t forget to put the image files on the Splunk server
too):

Change the background-image:url(…)to
your logo, restart Splunkweb, login and change your preference to your new theme.

Unfortunately, creating a custom
theme is the extent of the UI customization supported by the current version (3.x).
A new version, 4.0, is scheduled to be released
in early 2009 that promises
much more flexibility in building a custom UI. All elements of the UI are going to
be modular and developers will be able to not only rearrange them within the page
layout but also be able to create entirely new modules.

RESTful Applications

Splunk was built using the representational
state transfer (REST)
architecture. The newer versions (3.3 and on) began exposing Splunk’s internal
REST API for developers to build upon. A number of REST endpoints were created that
developers can directly access from external applications via simple HTTP requests.

A community driven Google code group
has been create called Splunk Labs to
enable developers to share ideas and applications built upon Splunk. There are currently
SDKs available in most major Web application languages including Python, Perl, PHP,
Java and .NET.

Out of the box, Splunk includes
a number of endpoints that can be accessed directly with a standard web browser. If
you have not modified the management port used by splunkd you can view the REST API
directly by browsing to https://localhost:8089/services/.

Here are a two of the more interesting
endpoints included:

To monitor the status of existing
jobs in the system go to https://localhost:8089/services/search/jobs.
From there you can examine what searches are running and either cancel them or pause
them (yes, you can actually pause searches!)

To view the current configuration
files Splunk is using look at https://localhost:8089/services/properties/file_name.
Replace “file_name” with any of the .conf files Splunk uses but do not include
the .conf extension. (Ex: to view inputs.conf only use /properties/input).

New endpoints can be created by
defining them in the restmap.conf file and then creating a new application under the …/etc/apps/ directory.
If you are interested in more of the details on coding a custom endpoint I would recommend
visiting the Splunk Labs website.

A final comment about
the Splunk REST API. In the upcoming 4.0 release, Splunk has stated that they will
be greatly expanding the API.