Splunk 4.0.9 Installation Guide

Splunk Installation Manual

Version: 4.0.9

Generated: 5/10/2010 03:24 amCopyright Splunk, Inc. All Rights Reserved

Table of ContentsWelcome to the Splunk Installation Manual......................................................................................1

What's in the Installation Manual.................................................................................................1System requirements..................................................................................................................1 Components of a Splunk deployment........................................................................................3 Splunk's architecture and what gets installed............................................................................4 About Splunk licenses................................................................................................................5 More about Splunk Free.............................................................................................................9 Before you install......................................................................................................................10

Step by step installation procedures...............................................................................................11 Choose your platform...............................................................................................................11 Install on Windows...................................................................................................................11 Install on Windows via the commandline.................................................................................15 Install on Linux.........................................................................................................................20 Install on Solaris.......................................................................................................................22 Install on Mac OS.....................................................................................................................25 Install on FreeBSD...................................................................................................................27 Install on AIX............................................................................................................................29 Install on HP-UX.......................................................................................................................31 Install a license.........................................................................................................................32

Start Splunk for the first time............................................................................................................35 Start Splunk for the first time....................................................................................................35

Migrate from 3.4.x or earlier..............................................................................................................37 What to expect when migrating to 4.0......................................................................................37 Migrate on UNIX.......................................................................................................................43 Migrate on Windows.................................................................................................................45 Steps for manual migration to Splunk 4.x.................................................................................47

Upgrade from 4.0 or later..................................................................................................................52 Upgrade Splunk on Windows...................................................................................................52 Upgrade Splunk on Linux, Solaris, FreeBSD, HP-UX, AIX, and MacOS.................................53

Other tasks.........................................................................................................................................55 Run Splunk as a different or non-root user..............................................................................55 Uninstall Splunk.......................................................................................................................56 Correcting the user selected during Windows installation........................................................58 Configure a standalone 3.4.x deployment server.....................................................................59

What comes next?.............................................................................................................................61 Ready to start using Splunk?...................................................................................................61 Estimating your storage requirements......................................................................................61 Hardware capacity planning for your Splunk deployment........................................................62

i

Table of ContentsReference............................................................................................................................................70

PGP Public Key........................................................................................................................70 File manifest.............................................................................................................................70

ii

Welcome to the Splunk Installation Manual

What's in the Installation ManualWhat's in the Installation Manual

Use this guide to find system requirements, licensing information, and procedures for installing ormigrating Splunk.

Find what you need

You can use the table of contents to the left of this panel, or simply search for what you want in thesearch box in the upper right.

If you're interested in more specific scenarios and best practices, you can visit the Splunk CommunityWiki to see how other users Splunk IT.

Make a PDF

If you'd like a PDF of any version of this manual, click the pdf version link above the table ofcontents bar on the left side of this page. A PDF version of the manual is generated on the fly for you,and you can save it or print it out to read later.

System requirementsSystem requirements

Before you download and install the Splunk software, read the following sections for the supportedsystem requirements. If you have ideas or requests for new features to add to future releases, emailSplunk Support. Also, you can follow our Product Roadmap.

Refer to the download page for the latest version to download. Check the release notes for details onknown and resolved issues.

For a discussion of hardware planning for deployment, check out the topic on capacity planning inthis manual.

Supported OSes

Splunk is supported on the following platforms.

Solaris 9, 10 (x86, SPARC)• Linux Kernel vers 2.6.x and above (x86: 32 and 64-bit)• FreeBSD 6.1 and 6.2 (x86: 32 and 64-bit)• Windows 2003 (x86: 64-bit, supported but not recommended on 32-bit)• Windows 2008 (x86: 64-bit, supported but not recommended on 32-bit)• WindowsXP (x86: 32-bit)• Vista (x86: 32-bit, 64-bit)•

1

MacOSX 10.5 and 10.6 (32-bit & 64bit in one download, 10.6 is only supported in 32-bitmode)

•

AIX 5.2 and 5.3• HP-UX 11iv2 (11.22) and 11iv3 (11.31) (PA-RISC or Itanium)•

FreeBSD 7.x

To run Splunk 4.x on 32-bit FreeBSD 7.x, install the compat6x libraries. Splunk Support will supply"best effort" support for users running on FreeBSD 7.x. For more information, refer to this Communitywiki topic.

Creating and editing configuration files on non-UTF-8 OSes

Splunk expects configuration files to be in ASCII/UTF-8. If you are editing or creating a configurationfile on an OS that is non-UTF-8, you must ensure that the editor you are using is configured to savein ASCII/UTF-8.

Supported browsers

Firefox 2 and 3.0.x• Firefox 3.5 (with Splunk version 4.0.6 and later)• Internet Explorer 6, 7 and 8• Safari 3•

Recommended hardware

Splunk is a high-performance application. If you are performing a comprehensive evaluation ofSplunk for production deployment, we recommend that you use hardware typical of your productionenvironment; this hardware should meet or exceed the recommended hardware capacityspecifications below.

For a discussion of hardware planning for deployment, check out the topic on capacityplanning in this manual.

Note: Running Splunk in virtual machine (VM) mode on any platform will degrade performance.

Recommended and minimum hardware capacity

Platform Recommended hardware capacity/configuration Minimum supportedhardware capacity

Non-Windowsplatforms

2x quad-core Xeon, 3GHz, 8GB RAM, RAID 0 or1+0, with a 64 bit OS installed.

1x1.4 GHz CPU, 1 GBRAM

Windowsplatforms

2x quad-core Xeon, 3GHz, 8GB RAM, RAID 0 or1+0, with a 64 bit OS installed.

Pentium 4 or equivalent at2Ghz, 2GB RAM

All configurations other than Splunk light forwarder instances require at least therecommended hardware configuration.

•

The minimum supported hardware guidelines are designed for personal use of Splunk.•

2

Important: For all installations including forwarders, a minimum of 2GB hard disk space for yourSplunk installation is required in addition to the space required for your index. Refer to this topicon estimating your index size requirements in this manual for some planning information.

Hardware requirements for Splunk light forwarders

Recommended Dual Core 1.5Ghz+ processor, 1GB+ RAM

Minimum 1.0 Ghz processor, 512MB RAMFor more information on deployment planning, refer to the Deployment section of the Splunkcommunity KnowledgeBase.

Supported file systems

Platform File systems

Linux ext2/3, reiser3, XFS, NFS 3/4

Solaris UFS, ZFS, VXFS, NFS 3/4

FreeBSD FFS, UFS, NFS 3/4

Mac OS X HFS, NFS 3/4

AIX JFS, JFS2, NFS 3/4

HP-UX VXFS, NFS 3/4

Windows NTFS, FAT32Considerations regarding NFS

NFS is usually a poor choice for Splunk indexing activity, for reasons of performance, resilience, andsemantics. In environments with very high bandwidth, very low latency links, that are kept highlyreliable, it can be an appropriate choice. Typically, this is a NAS accessed via the NFS protocol.

Note: Several other file systems are supported. If you run Splunk on a filesystem that is not listedabove, Splunk may run a startup utility named locktest. Locktest is a program that tests the startup process. If locktest runs and fails, the filesystem is not suitable for running Splunk.

Note: On FreeBSD, mounting as nullfs is not supported.

Supported server hardware architectures

32 and 64-bit architectures are supported for some platforms. download page page for details.

Components of a Splunk deploymentComponents of a Splunk deployment

Splunk is simple to deploy by design. By using a single software component and easy to understandconfigurations, Splunk can coexist with existing infrastructure or be deployed as a universal platformfor accessing IT data.

3

The simplest deployment is the one you get by default when you install Splunk: indexing andsearching on the same server. Data comes in from the sources you've configured, and you log intoSplunk Web or the CLI on this same server to search, monitor, alert, and report on your IT data.

Depending on your needs, you can also deploy components of Splunk on different servers to addressyour load and availability requirements. This section covers these potential components:

Indexer

In this mode, indexers, or index servers, provide indexing capability for local and remote data andhost the primary Splunk datastore, as well as Splunk Web. Refer to "How indexing works" in theAdmin Manual for more information.

Search head

In this mode, a Splunk instance is configured to direct user search requests to one or more indexers.Use distributed search to configure a search head to search across a pool of indexers.

Forwarder

Forwarders use the same Splunk software package but do not store indexed data locally. All indexeddata is forwarded to remote index servers. To reduce operational footprint, Splunk Web is not used.Refer to the documentation on setting up a Splunk instance as a forwarder.

Deployment server

Both indexers and forwarders can also act as deployment servers. A deployment server distributesconfiguration information to running instances of Splunk via a push mechanism which is enabledthrough configuration. Refer to the documentation on setting up a Splunk instance as a deploymentserver.

Functions at a glance

Functions Indexer Search head Forwarder Deploymentserver

Indexing xWeb xDirect search xForward to indexer xDeploy configurations x x x

Splunk's architecture and what gets installedSplunk's architecture and what gets installed

This topic discusses Splunk's internal architecture and processes at a high level. If you're looking forinformation about third-party components used in Splunk, refer to the credits section in the Releasenotes.

4

Processes

A Splunk server runs two processes on your host, splunkd and splunkweb:

splunkd is a distributed C/C++ server that accesses, processes and indexes streaming ITdata. It also handles search requests. splunkd processes and indexes your data bystreaming it through a series of pipelines, each made up of a series of processors.

Pipelines are single threads inside the splunkd process, each configured with a singlesnippet of XML.

♦

Processors are individual, reusable C or C++ functions that act on the stream of ITdata passing through a pipeline. Pipelines can pass data to one another via queues.splunkd supports a command line interface for searching and viewing results.

♦

•

splunkweb is a Python-based application server based on CherryPy that provides the SplunkWeb user interface. It allows users to search and navigate IT data stored by Splunk serversand to manage your Splunk deployment through a Web interface.

•

splunkweb and splunkd can both communicate with your Web browser via REST:

splunkd also runs a Web server on port 8089 with SSL/HTTPS turned on by default.• splunkweb runs a Web server on port 8000 without SSL/HTTPS by default.•

Architecture diagram

About Splunk licensesAbout Splunk licenses

Each instance of Splunk must have its own license. This topic discusses the different Splunklicenses, how to install or update a license, and what to do when you have a violation on your license.

Note: When running Splunk Enterprise, you must purchase a separate license for every instance ofSplunk that you deploy.

5

Enterprise vs. Free license

Splunk provides two standard types of licenses, an Enterprise license, and a Free license.

When you download Splunk for the first time, you are asked to register. Your registration authorizesyou to receive an Enterprise trial license, which allows a maximum indexing volume of 500 MB/day.The Enterprise trial license expires 60 days from download. If you are running with a Enterprise triallicense and your license expires, Splunk continues to index your data. However, you will not be ableto search until you install a new license.

Once you have installed Splunk, you can choose to run Splunk with the Enterprise trial license until itexpires, purchase an Enterprise license, or switch to the Free license, which is included.

The Free license is not a trial license and does not have an expiration date. It also allows 500MB/dayof indexing volume, but the following features that are available with the Enterprise license aredisabled:

Multiple user accounts and role-based access controls• Distributed search• Forwarding in TCP/HTTP formats (you can forward data to other Splunk instances, but not tonon-Splunk instances)

•

Deployment management (including for clients)• Scheduled saved searches (including summary indexing) and alerting/monitoring•

Learn more about Splunk with a Free license

Find more information about the different license features here. Also, read Splunk's Free licenseagreement.

Forwarding license

A license isn't required to enable forwarding, but enables security on the forwarder so that users mustsupply username and password to access it. Splunk includes a forwarder license that you can installon each Splunk forwarder. This 1 MB/day forward-only license is not subtracted from your existinglicense(s) and can be applied to multiple forwarders.

Each instance of Splunk that does any indexing must have its own license.

1. Stop Splunk: ./splunk stop

2. Copy $SPLUNK_HOME/etc/splunk-forwarder.license to$SPLUNK_HOME/etc/splunk.license

3. Start Splunk: ./splunk start

This license does not limit how much data you can forward from that machine.

6

Search head license

A search head does not normally index any data, but you don't want unrestricted access on thereeither. Apply a forwarder license to enable authentication on your search head instance.

Summary index searches count towards your indexed data volume. If you run these searches on yoursearch head, it will violate the forwarder license and search functionality will be disabled. To avoidthis, configure your summary index searches on your indexing instances instead.

Custom trial Enterprise license

You can request trial Enterprise licenses of varying size and duration. The default evaluation period is60 days. If you are preparing a pilot for a large deployment and have requirements for a longerduration or higher indexing volumes during your trial, contact Splunk Sales with your request.

Preview license

Splunk's Preview releases require a different license that is not compatible with other Splunkreleases. Also, if you are evaluating a Preview release of Splunk, it will not run with a Free orEnterprise license. Preview licenses typically enable Enterprise features, they are just restricted toPreview releases. If you are evaluating a Preview version of Splunk, it will come with its own license.

View your license and usage details

You can view details about your license from Splunk Web or the command line interface (CLI). Thedetails about your license include general information, such as the type of license, the indexing level,and the expiration date of the license.

View license information in Splunk Web

To view your license details in Splunk Web, go to Manager > License. Under "License & Usage". Inaddition to the general information about your license, the details include: the count of days until yourlicense expires, the peak indexed amount (in MB), and the count of violations against your license.

Note: You can also install and update your license from this page.

You can also use search to learn information about indexing volumes. For a report on the dailyindexed volume, use this search:

index=_internal todaysBytesIndexed LicenseManager-Audit NOTsource=*web_service.log | eval Daily_Indexing_Volume_in_MBs =todaysBytesIndexed/1024/1024 | timechart avg(Daily_Indexing_Volume_in_MBs)by hostShow license information in the CLI

If you have access to the CLI, you can view details about your license with:

./splunk show license

The CLI displays the same general details, but also includes information about the state of yourlicense, such as the maximum number of violations (Max Violations) permitted by the license within

7

the grace period moving window (Violation Period). Splunk also displays an "Expiration State", whichtells you when you license is either "7" days or "1" day from expiring; otherwise, it is "ok".

Install or update your license

All Splunk servers have a license located in $SPLUNK_HOME/etc/, whether it is a Free license(splunk-free.license) or an Enterprise license (splunk.license). You can install and updateyour licenses with the CLI or from Splunk Web's Manager > License page.

Refer to the Admin Manual for instructions to install or update your Splunk license.

Migrating to 4.0

If you migrated a 3.x Splunk instance directly to 4.0, remove the$SPLUNK_HOME/etc/splunk.license file before you run Splunk. The instance will then pick upthe 60-day Enterprise trial license that is included with 4.0.

If you have a current Enterprise license/support contract for 3.x, an updated license is waiting for you.Log into splunk.com and go to http://www.splunk.com/store/myorders to pick it up. Replace yourexisting $SPLUNK_HOME/etc/splunk.license file with the new file, or use the instructions forupdating your Splunk license.

Licenses for distributed deployments

If you have Splunk running on multiple hosts in a distributed environment, you must have a separateunique license key for each host. If you have been using a single license key for all your hosts, thiswill not work in versions 4.0 and later (except with an Enterprise Trial license, starting with version4.0.3). Contact Splunk Support to have your license broken up into multiple keys for this purpose, oremail Splunk Sales to purchase additional keys.

License violations

Violations occur when you exceed the maximum indexing volume allowed for your license. If youexceed your licensed daily volume on any one calendar day, you will get a violation warning. Themessage persists for 14 days. If you have 5 or more violations on an Enterprise license or 3violations on a Free license in a rolling 30-day period, search will be disabled. Searchcapabilities return when you have fewer than 5 (Enterprise) or 3 (Free) violations in the previous 30days or when you apply a new license with a larger volume limit.

Note: During a license violation period, Splunk does not stop indexing your data. Splunk only blocksaccess while you exceed your license.

Note: Searches to the _internal index are not disabled even during a licensing-enforcement period,so you can still access the Indexing Status dashboard, or run searches against _internal to diagnosethe licensing problem.

Got License Violations? Click here to troubleshoot.

8

More about Splunk FreeMore about Splunk Free

Splunk Free is a totally free (as in beer) version of Splunk. It allows you to index up to 500MB/dayand will never expire. If you go over 500MB/day more than 3 times in a 30 day period, Splunk willcontinue to index your data, but search will be disabled until you are back down to 3 or fewer times inthe 30 day period.

What's it for?

Splunk Free is designed for personal, ad-hoc search and visualization of IT data. You can use SplunkFree for ongoing indexing of small volumes (<500MB/day) of data. Additionally, you can use it forshort-term bulk-loading and analysis of larger data sets--Splunk Free allows you to bulk-load muchlarger data sets up to 3 times within a 30 day period. This can be useful for forensic review of largedata sets.

What is and isn't enabled

Splunk Free is a single-user product. All of Splunk's features are supported with the exception of:

Multiple user accounts and role-based access controls (there's no authentication when usingSplunk Free)

•

Distributed search• Forwarding in TCP/HTTP formats (you can forward data to other Splunk instances, but not tonon-Splunk instances)

•

Deployment management• Scheduled saved searches (including summary indexing) and alerting/monitoring•

A Free instance can be used as a forwarder (to a Splunk indexer) but may not be a client of adeployment server.

Switching to Free from an Enterprise (Trial) License

When you first download and install Splunk, you are automatically using an Enterprise Trial license.You can continue to use the Enterprise Trial License until it expires, or switch to the Free license rightaway, depending on your requirements.

What you should know about switching to Free

Splunk Enterprise Trial gives you access to a number of features that are not available in SplunkFree. When you switch, be aware of the following:

User accounts or roles that you've created will no longer work.• Anyone connecting to the instance will automatically be logged on as 'admin'. You will nolonger see a login screen, though you will see the update check occur.

•

Any knowledge objects created by any user other than 'admin' (such as event type,transaction, or source type definitions) and not already globally shared will not be available. Ifyou need these knowledge objects to continue to be available after you switch to Splunk Free,you can either

•

9

use Manager to promote them to be globally available before you switch using theinformation in this topic or

♦

hand edit the configuration files they are in to promote them as described here♦ Scheduled searches you've set up (including alerts and summary indexing searches) will nolonger fire/function.

you will no longer receive alerts from Splunk♦ searches and reports expecting summary indexes may be inaccurate, or return noresults

♦

dashboards that use search artifacts (such as via HiddenSavedSearch) will run thesearches when you load them

♦

•

Configurations in outputs.conf to forward to third-party applications in TCP or HTTP formatswill stop working.

•

When you attempt to make any of the above configurations in Manager while using an EnterpriseTrial, you will be warned about the above limitations in a Free Splunk.

How do I switch to Splunk Free?

If you currently have Splunk Enterprise (trial or not), you can either wait for your Enterprise License toexpire, or switch to a Free License at any time. To switch to a Free License:

1. Log in to Splunk Web as a user with admin privileges and navigate to Manager > License.

2. Review the text below the License and usage area, find the switch to a free license link, andclick it. A login page is displayed.

3. Select Switch to Free License and click Continue.

4. You are prompted to restart.

Before you installBefore you install

Before you install Splunk, be sure to review the system requirements and ensure that you'vedownloaded the right installation package for your system.

If you're upgrading from an earlier version of Splunk, review the information in "What to expect whenupgrading to 4.0" before proceeding.

10

Step by step installation procedures

Choose your platformChoose your platform

Choose from the list below for detailed installation procedures:

Windows• Windows commandline instructions• Linux• Solaris• MacOS• FreeBSD• AIX• HP-UX•

Install on WindowsInstall on Windows

This topic describes the procedure for installing on Windows using the GUI installer. More options areavailable for installation (such as silent installation) if you use the commandline installation.

Note: The Windows App was enabled by default in its app.conf file in versions 4.0-4.0.2. Starting inversion 4.0.3, it is disabled in this file by default. Read on for important details:

If you're upgrading from 4.0-4.0.2 to 4.0.3 or later, the Windows App will be disabled, even if itwas enabled in the version you're upgrading from.

•

If you're doing a fresh installation of 4.0.3 or later, the Windows App is enabled by default viathe MSI and if you want to install it in a disabled state, you must specify this using theSPLUNK_APP msiexec command as described in "Install on Windows via the commandline".

•

Important: Running the 32-bit version of Splunk for Windows on a 64-bit platform is notrecommended. If you can run 64-bit Splunk on 64-bit hardware, we strongly recommend it. Theperformance is greatly improved over the 32-bit version.

Choosing the user Splunk should run as

When you run the Splunk Windows installer, you are given the option to select a user Splunk will runas.

If you install as the Local System user, Splunk will have access to all or nearly all of the importantinformation on your local machine. However, the Local System user has no privileges on otherWindows machines by design.

If you intend to do any of the following things, you must give Splunk a Domain account:

11

read Event Logs remotely• collect performance counters remotely• read network shares for log files• enumerate the Active Directory schema using Active Directory monitoring•

The Domain account you use must also be a local Administrator or equivalent. Please ask yourWindows Domain administrator for an account if you are unsure of what username to run Splunkunder.

Minimum local permissions required for the two Splunk services:

Required user rights for the splunkd service:

Full control over Splunk's installation directory• Read access to any flat-files• Permission to log on as a service• Permission to log on as a batch job• Replace a process-level token• Permission to act as part of the operating system• Permission to bypass traverse checking•

Required user rights for the splunkweb service:

Full control over Splunk's installation directory• Permission to log on as a service•

Note: These are the rights that splunkd and splunkweb specifically invoke. Other rights orpermissions may be required depending on your usage and what data you want to access.Additionally, many user right assignments and other group policy restrictions can prevent Splunk fromrunning. If you have issues, consider using a tool such as Sysinternals to troubleshoot yourenvironment, or reverting to running the splunkd service as an administrator or equivalent account.

Important: If you change the user Splunk runs as after you have installed, you must ensure that theuser you create has the necessary permissions, and also ensure that that user has Full Controlpermissions to the $SPLUNK_HOME/var directory.

If you accidentally specify the wrong user the first time you install

If you specified the wrong user during the installation procedure, you'll see two popup error dialogstelling you this. Complete the installation and then use these instructions to switch to the correct user.You must not start Splunk before doing this.

Install Splunk via the GUI installer

The Windows installer is an MSI file.

1. To start the installer, double-click the splunk.msi file.

The Welcome panel is displayed.

12

2. To begin the installation, click Next.

Note: On each panel, you can click Next to continue, Back to go back a step, or Cancel to close theinstaller.

The licensing panel is displayed.

3. Read the licensing agreement and select "I accept the terms in the license agreement". Click Nextto continue installing.

The Customer Information panel is displayed.

4. Enter the requested details and click Next.

The Destination Folder panel is displayed.

Note: Splunk is installed by default into the \Program Files\Splunk.

5. Click Change... to specify a different location to install Splunk, or click Next to accept the defaultvalue.

The Logon Information panel is displayed.

Splunk installs and runs two Windows services, splunkd and splunkweb. These services will beinstalled and run as the user you specify on this panel. You can choose to run Splunk with LocalSystem credentials, or provide a specific account. That account should have local administratorprivileges, plus appropriate domain permissions if you are collecting data from other machines.

The user Splunk runs as must have permissions to:

Run as a service.• Read whatever files you are configuring it to monitor.• Collect performance or other WMI data.• Write to Splunk's directory.•

Note: If you install as the Local System user, some network resources may not be available to theSplunk application. Additionally, WMI remote authentication will not work; this user has nullcredentials and Windows servers normally disallow such connections. Only local data collection withWMI will be available. Contact your systems administrator for advice if you are unsure what user tospecify.

6. Select a user type and click Next.

Important: When migrating or upgrading, you must re-specify the user you want Splunk to runas--this information is not automatically maintained from release to release.

If you specified the local system user, proceed to step 8. Otherwise, the Logon Information: specifya username and password panel is displayed.

13

7. Specify a username and password to install and run Splunk and click Next.

Note: To use an existing user, you can enter or browse for the username and domain details. Splunkrecommends using the Browse... button to ensure that you select a valid user. If you cannot browsefor the user because that user doesn't exist in your security context, or you mistype the username,your installation will fail. Splunk cannot start without a valid username and password; browsingconfirms the user is correct.

The pre-installation summary panel is displayed.

8. Click Install to proceed.

The installer runs and displays the Installation Complete panel.

Caution: If you specified the wrong user during the installation procedure, you will see two popuperror windows explaining this. If this occurs, Splunk installs itself as the local system user by default.Splunk will not start automatically in this situation. You can proceed through the final panel of theinstallation, leaving all boxes checked. Then, use these instructions to switch to the correct userbefore starting Splunk.

9. Check the boxes to Start Splunk and Start Splunk Web now. Click Finish.

The installation completes, Splunk starts, and Splunk Web launches in a supported browser.

Note: The first time you access Splunk Web after installation, login with the default username adminand password changeme.

Launch Splunk in a Web browser

To access Splunk Web after you start Splunk on your machine, you can either:

Click the Splunk icon in Start>Programs>Splunk•

or

Open a Web browser and navigate to http://localhost:8000.•

Log in using the default credentials: username: admin and password: changeme . Be sure to changethe admin password as soon as possible and make a note of what you changed it to.

Now that you've installed Splunk, what comes next?

Change the Splunk Web or splunkd service ports

If you want the Splunk Web service or the splunkd service to use a different port, you can change thedefaults.

To change the splunk web service port:•

14

From the $SPLUNK_HOME/bin/ directory: splunk set web-port ####

To change the splunkd port:•

From the $SPLUNK_HOME/bin/ directory: splunk set splunkd-port ####

Note: If you specify a port and that port is not available, or if the default port is unavailable, Splunkwill automatically select the next available port.

Avoid IE Enhanced Security pop-ups

To avoid IE Enhanced Security pop-ups, add the following URLs to the allowed Intranet group or fullytrusted group in IE:

quickdraw.splunk.com• the URL of your Splunk instance•

Install or upgrade license

If you are performing a new installation of Splunk or switching from one license type to another, youmust install or update your license.

Uninstall Splunk

To uninstall Splunk, use the Add or Remove Programs option in the Control Panel.

What's next?

Review this topic about considerations for deciding how to monitor Windows data in the AdminManual.

Install on Windows via the commandlineInstall on Windows via the commandline

This topic describes the procedures for installing Splunk on Windows using the commandline.

Important: Running the 32-bit version of Splunk for Windows on a 64-bit platform is notrecommended. If you can run 64-bit Splunk on 64-bit hardware, we strongly recommend it. Theperformance is greatly improved over the 32-bit version.



•

If you're doing a fresh installation of 4.0.3 or later, the Windows App is enabled by default,unless you explicitly enable a different app such as SplunkLightFowarder. via the MSI.However, you can enable the forwarder apps and enable Windows Event Log explicitly. If youwant to install it in a disabled state, you must specify this using the SPLUNK_APP msiexec

•

15

command as described later in this topic.

Choosing the user Splunk should run as

When you run the Splunk Windows installer, you are given the option to select a user Splunk will runas.

If you install as the Local System user, Splunk will have access to all or nearly all of the importantinformation on your local machine. However, the Local System user has no privileges on otherWindows machines by design. If you intend to read Event Logs or performance counters from othermachines via WMI, or read network shares for log files, you will need a domain account. That accountmust be a local Administrator or equivalent, and should have rights to the external data you want toSplunk. Please ask your Windows domain administrator for an account if you are unsure of whatcredentials to give Splunk.

Minimum permissions required for the two Splunk services:

Required user rights for the splunkd service:

Full control over Splunk's installation directory• Read access to any flat-files• Permission to log on as a service• Permission to log on as a batch job• Replace a process-level token• Permission to act as part of the operating system• Permission to bypass traverse checking•

Required user rights for the splunkweb service:

Full control over Splunk's installation directory• Permission to log on as a service•

Important: If you must change the user Splunk runs as after you have installed, you must ensure thatthe user you create has the necessary permissions, and also ensure that that user has Full Controlpermissions to the $SPLUNK_HOME/var directory.

If you specified the wrong user during your installation, Splunk will not start. If this occurs, Splunk hasinstalled itself as the local system user by default. Use the instructions in these instructions to switchto the correct user before starting Splunk.

How to use the MSI on the commandline

You can install Splunk for Windows using the MSI on the commandline by typing the following:

msiexec.exe /i Splunk.msi

This section lists the available flags for doing this, and provides a few examples of doing this invarious configurations.

You can specify

16

which Windows event logs to index or not• which Windows registry hive to monitor• which WMI information to pull• the user Splunk runs as (be sure the user you specify has the appropriate permissions toaccess the content you want Splunk to index)

•

an included application configuration for Splunk to enable (such as the Splunk light forwarder)• whether or not Splunk should start up automatically when the installation is completed•

Note: The first time you access Splunk Web after installation, log in with the default username adminand password changeme.

Supported flags

The following is a list of the flags you can use when installing Splunk for Windows via thecommandline.

Use this flag to specify directory to install. Default is c:\program files\splunk.

INSTALLDIR=<directory_path>•

Use these flags to specify alternate ports for splunkd and splunkweb to use

SPLUNKD_PORT=<port number>• WEB_PORT=<port number>•

Use these flags to specify whether or not Splunk should index a particular Windows event log.

WINEVENTLOGAPPCHECK=1/0, off by default• WINEVENTLOGSECCHECK=1/0, off by default• WINEVENTLOGSYSCHECK=1/0, off by default• WINEVENTLOGFWDCHECK=1/0, off by default• WINEVENTLOGSETCHECK=1/0, off by default•

Use these flags to specify whether or not Splunk should index the Windows registry USER hive. Bydefault these are set to 0 (off).

REGISTRYCHECK_U=1/0• REGISTRYCHECK_BASELINE_U=1/0•

Use these flags to specify whether or not Splunk should index the Windows registry LocalMachinehive. By default, these are set to 0 (off).

REGISTRYCHECK_LM=1/0• REGISTRYCHECK_BASELINE_LM=1/0•

Use these flags to specify which WMI performance information to index. These are set to 0 (off) bydefault.

WMICHECK_CPUTIME=1/0•

17

WMICHECK_LOCALDISK=1/0• WMICHECK_FREEDISK=1/0• WMICHECK_MEMORY=1/0•

Use this flag to specify a user Splunk should run as. Supported values are: 1 for the LocalSystemuser and 2 for a different user. The default value is 1.

RBG_LOGON_INFO_USER_CONTEXT=1/2•

Use these flags to provide domain/username and password information for the user specified inRBG_LOGON_INFO_USER_CONTEXT. You must specify the domain with the username in theformat "domain\username".

IS_NET_API_LOGON_USERNAME="<domain\username>"• IS_NET_API_LOGON_PASSWORD="<pass>"•

Use this flag to specify an included Splunk application configuration to enable for this installation ofSplunk. Currently supported options for <SplunkApp> are: SplunkLightForwarder, SplunkForwarder.

Refer to the documentation about the Splunk forwarder and light forwarder configurations for moreinformation about the forwarders. If you specify either the Splunk forwarder or light forwarder here,you must also specify FORWARD_SERVER="<server:port>".

SPLUNK_APP=<SplunkApp>•

To install Splunk with no applications at all, specify this flag but leave the value empty (SPLUNK_APP="" ).

Use this flag *only* when you are also using SPLUNK_APP to enable either the Splunk forwarder orlight forwarder. Specify the server and port of the Splunk server to which this forwarder will send data.

FORWARD_SERVER="<server:port>"•

Use this flag to specify whether or not Splunk should start up automatically when the installationcompletes. The default value is 1 (on).

LAUNCHSPLUNK=0/1•

Important: If you are enabling an App (SPLUNK_APP), Splunk will start automatically; this cannot beoverridden.

Use these flags to specify which Splunk services start up automatically at boot time.

AUTOSTARTSERVICE=0/1 (starts both services)• AUTOSTARTSERVICE_SPLUNKD=0/1• AUTOSTARTSERVICE_SPLUNKWEB=0/1•

18

Silent installation

To run the installation silently, add /quiet to the end of your installation command string. If yoursystem is running UAC (which is sometimes on by default) you must run the installation asAdministrator. To do this: when opening a cmd prompt, right click and select "Run As Administrator".Then use this cmd window to run the silent install command.

Examples

The following are some examples of using different flags.

Install Splunk to run as the Local System user

msiexec.exe /i Splunk.msi RBG_LOGON_INFO_USER_CONTEXT=1

Specify the username and the domain the user belongs to

msiexec.exe /i Splunk.msi SPLUNK_APP="SplunkForwarder"RBG_LOGON_INFO_USER_CONTEXT=2 IS_NET_API_LOGON_USERNAME="AD\splunk"IS_NET_API_LOGON_PASSWORD="splunk123"

Enable SplunkForwarder, disable indexing of the Windows System event log, and run the installer in silent mode

msiexec.exe /i Splunk.msi SPLUNK_APP="SplunkForwarder"FORWARD_SERVER="<server:port>" WINEVENTLOGSYSCHECK=0 /quiet

Where "<server:port>" are the server and port of the Splunk server to which this machine shouldsend data.

Launch Splunk in a Web browser

To access Splunk Web after you start Splunk on your machine, you can either:

Click the Splunk icon in Start>Programs>Splunk•

or

Open a Web browser and navigate to http://localhost:8000.•

Log in using the default credentials: username: admin and password: changeme . Be sure to changethe admin password as soon as possible and make a note of what you changed it to.

Now that you're ready to use Splunk, refer to the User Manual and begin using Splunk!

Avoid IE Enhanced Security pop-ups

To avoid IE Enhanced Security pop-ups, add the following URLs to the allowed Intranet group or fullytrusted group in IE:

quickdraw.splunk.com• the URL of your Splunk instance•

19

Install or upgrade license


Uninstall Splunk

To uninstall Splunk, use the Add or Remove Programs option in the Control Panel.

You can also use msiexec from the commandline.

Install on LinuxInstall on Linux

You can install Splunk on Linux using RPM or DEB packages, or a tarball.

RedHat RPM install

To install the Splunk RPM in the default directory /opt/splunk:

rpm -i splunk_package_name.rpm

To install Splunk in a different directory, use the --prefix flag:

rpm -i --prefix=/opt/new_directory splunk_package_name.rpm

To upgrade an existing Splunk installation using the RPM:

rpm -U splunk_package_name.rpm

To upgrade an existing Splunk installation that was done in a different directory, use the --prefixflag:

rpm -U --prefix=/opt/new_directory splunk_package_name.rpm

If you want to automate your RPM install with kickstart, add the following to your kickstart file:

./splunk start --accept-license

./splunk enable boot-start

Note: The second line is optional for the kickstart file.

Debian DEB install

To install the Splunk DEB package:

dpkg -i splunk_package_name.deb

Note: You can only install the Splunk DEB package in the default location, /opt/splunk.

20

Tarball install

To install Splunk on a Linux system, expand the tarball into an appropriate directory. The defaultinstall directory is /opt/splunk.

When installing with the tarball:

Splunk does not create the splunk user automatically. If you want Splunk to run as a specificuser, you must create the user manually.

•

Be sure the disk partition has enough space to hold the uncompressed volume of the data youplan to keep indexed.

•

What gets installed

Splunk package status:

dpkg --status splunk

List all packages:

dpkg --list

Start Splunk

Splunk can run as any user on the local system. If you run Splunk as a non-root user, make sure thatSplunk has the appropriate permissions to read the inputs that you specify. Refer to the instructionsfor running Splunk as a non-root user for more information.

To start Splunk from the command line interface, run the following command from$SPLUNK_HOME/bin directory (where $SPLUNK_HOME is the directory into which you installedSplunk):

./splunk start

By convention, this document uses:

$SPLUNK_HOME to identify the path to your Splunk installation.• $SPLUNK_HOME/bin/ to indicate the location of the command line interface.•

Startup options

The first time you start Splunk after a new installation, you must accept the license agreement. Tostart Splunk and accept the license in one step:

$SPLUNK_HOME/bin/splunk start --accept-license

Note: There are two dashes before the accept-license option.

21

Launch Splunk Web and log in

After you start Splunk and accept the license agreement,

1. In a browser window, access Splunk Web at http://<hostname>:port.

hostname is the host machine.• port is the port you specified during the installation (the default port is 8000).•

2. Splunk Web prompts you for login information (default, username admin and passwordchangeme) before it launches. If you switch to Splunk Free, you will bypass this logon page in futuresessions.


Uninstall Splunk

Use your local package management commands to uninstall Splunk. In most cases, files that werenot originally installed by the package will be retained. These files include your configuration andindex files which are under your installation directory.

If you can't use package management commands, follow the instructions for manually uninstallingSplunk components.

RedHat Linux

To uninstall from RedHat Linux

rpm -e splunk_product_name

Debian Linux

To uninstall from Debian Linux:

dpkg -r splunk

To purge (delete everything, including configuration files):

dpkg -P splunk

Install on SolarisInstall on Solaris

This topic provides the procedures for installing Splunk on Solaris.

Install Splunk

Splunk for Solaris is available as a PKG file or a tarball.

22

PKG file install

The PKG installation package includes a request file that prompts you to answer a few questionsbefore Splunk installs.

pkgadd -d ./splunk_product_name.pkg

A list of the available packages is displayed.

Select the packages you wish to process (the default is "all").•

The installer then prompts you to specify a base installation directory.

To install into the default directory, /opt/splunk, leave this blank.•

PKG file upgrade

To upgrade an existing Splunk installation using a PKG file, use the same command line as youwould for a fresh install.

pkgadd -d ./splunk_product_name.pkg

You will be prompted to overwrite any changed files, answer yes to every one.

To run the upgrade silently (and not have to answer yes for every file overwrite), type:

pkgadd -n -d ./splunk_product_name.pkg

Tarball install

To install Splunk on a Solaris system, expand the tarball into an appropriate directory. By default,Splunk installs into /opt/splunk/.



•


•

What gets installed

Splunk package info:

pkginfo -l splunk

List all packages:

pkginfo

23

Start Splunk

Splunk can run as any user on the local system. If you run Splunk as a non-root user, make sure thatSplunk has the appropriate permissions to read the inputs that you specify. For more information,refer to the instructions on running Splunk as a non-root user.


./splunk start

By convention, Splunk's documentation uses:


Startup options






1. In a browser window, access Splunk Web at http://mysplunkhost:port, where:

mysplunkhost is the host machine.• port is the port you specified during the installation (8000).•



Uninstall Splunk

Use your local package management commands to uninstall Splunk. In most cases, files that werenot originally installed by the package are retained. These files include your configuration and indexfiles which are under your installation directory.

pkgrm splunk

24

Install on Mac OSInstall on Mac OS

The Mac OS build comes in two forms: a DMG package and a tarball. Below are instructions for the:

Graphical (basic) and command line installs using the DMG file.• Tarball install.•

Graphical install

1. Double-click on the DMG file.

A Finder window containing splunk.pkg opens.

2. In the FInder window, double-click on splunk.pkg.

The Splunk installer opens and displays the Introduction, which lists version and copyrightinformation.

3. Click Continue.

The Select a Destination window opens.

4. Choose a location to install Splunk.

To install in the default directory, /Applications/splunk, click on the harddrive icon.• To select a different location, click Choose Folder...•

5. Click Continue.

The pre-installation summary displays. If you need to make changes,

Click Change Install Location to choose a new folder, or• Click Back to go back a step.•

6. Click Install.

Your installation will begin. It may take a few minutes.

7. When your install completes, click Finish.

Command line install

1. To mount the dmg:

hdid splunk_package_name.dmg

2. To Install

25

To the root volume:•

installer -pkg splunk.pkg -target /

To a different disk of partition:•

installer -pkg splunk.pkg -target /Volumes\ Disk

-target specifies a target volume, such as another disk, where Splunk will be installed in/Applications/splunk.

To install into a directory other than /Applications/splunk on any volume, use the graphicalinstaller as described above.

Tarball install

To install Splunk on a Mac OS, expand the tarball into an appropriate directory. The default installdirectory is /Applications/splunk.



•


•

Start Splunk

Splunk can run as any user on the local system. If you run Splunk as a non-root user, make sure thatSplunk has the appropriate permissions to read the inputs that you specify.


./splunk start



Startup options




26







Manage your license


Uninstall Splunk


You can also simply go to $SPLUNK_HOME/bin, type ./splunk stop on the commandline andthen delete the $SPLUNK_HOME directory and everything under it.

Install on FreeBSDInstall on FreeBSD

The FreeBSD builds comes in two forms: an installer (5.4-intel) and a tarball (i386). Both are TGZfiles.

Basic install

To install FreeBSD using the intel installer:

pkg_add splunk_package_name-5.4-intel.tgz

This installs Splunk in the default directory, /opt/splunk/

To install Splunk in a different directory:

pkg_add -v -p /usr/splunk splunk_package_name-5.4-intel.tgz

27

Tarball install

To install Splunk on a FreeBSD system, expand the tarball into an appropriate directory. The defaultinstall directory is /opt/splunk.



•


•

After you install

To ensure that Splunk functions properly on FreeBSD, you must:

1. Add the following to /boot/loader.conf

kern.maxdsiz="2147483648" # 2GBkern.dfldsiz="2147483648" # 2GBmachdep.hlt_cpus=0

2. Add the following to /etc/sysctl.conf:

vm.max_proc_mmap=2147483647

A restart of the OS is required for the changes to effect.

What gets installed

To see the list of Splunk packages:

pkg_info -L splunk

To list all packages:

pkg_info

Start Splunk



./splunk start


28


Startup options










Manage your license


Uninstall Splunk


To uninstall Splunk from the default location:

pkg_delete splunk

To uninstall Splunk from a different location:

pkg_delete -p /usr/splunk splunk

Install on AIX

29

Install on AIX

This topic will guide you through installing Splunk on the AIX platform.

Note: If you are upgrading, review the upgrade documentation later in this manual and check themigration documentation for any migation considerations before proceeding.

Install Splunk

The AIX install comes in tarball form.



•


•

To install Splunk on an AIX system, expand the tarball into an appropriate directory. The defaultinstall directory is /opt/splunk.

For AIX 5.3, check to make sure your service packs are up to date. Splunk requires the followingservice level:

$ oslevel -r5300-005

Start Splunk

Splunk can run as any user on the local system. If you run Splunk as a non-root user, make sure thatSplunk has the appropriate permissions to read the inputs that you specify. Refer to the instructionsfor running Splunk as a non-root user for more information.


./splunk start



Note: The AIX version of Splunk does not register itself to auto-start on reboot.

Startup options


30



For more information, refer to "Splunk startup options" in this manual.







Manage your license

If you are performing a new installation of Splunk or switching from one license type to another, youmust update your license.

Uninstall Splunk


Install on HP-UXInstall on HP-UX

To install Splunk on an HP-UX system, expand the tarball into an appropriate directory. The defaultinstall directory is /opt/splunk.



•


•

Start Splunk


31


./splunk start



Note: The HP-UX version of Splunk does not register itself to auto-start on reboot.

Startup options










Manage your license


Install a licenseInstall a license

Each instance of Splunk server must have its own license. This topic discusses how to install orupdate a license, and what to do when you have a violation on your license.

32

For more details about license types and policies, refer to "About Splunk licenses" earlier in thismanual.

Note: You must purchase a separate license for every instance of Splunk with an Enterprise licensethat you deploy.

Install your license

All Splunk servers have a license located in $SPLUNK_HOME/etc/, whether it is a Free license(splunk-free.license) or an Enterprise license (splunk.license). You can install and updateyour licenses with the CLI or using Splunk Web.

Install a license using Splunk Web

1. Log into Splunk Web as the admin user.

2. Click Manager>License.

3. Click Change License.

4. Paste in your license key and click Save.

5. Return to the main Manager tab and click Restart Splunk.

Note - Sometimes, you may get an 'Invalid License' error when you apply a license via Splunk Web.This is usually caused by hidden characters getting picked up and added to the license string whenusing copy&paste. To resolve this, you can update the new splunk.license file manually in$SPLUNK_HOME/etc. Make sure you use a plain text editor to do this.

Pre-seeding your license before first time run

Starting with 4.0.2, by default when you start Splunk for the first time, it moves aside any existing 3.xlicense and replaces it with a temporary Enterprise trial license. This allows you to bring up the newversion of Splunk without having your license be expired until you get your new one copied in.

If you are migrating to Splunk 4.0.2 or later and have a valid 4.x license, you can pre-seed the licensefile so that it pulls in and installs your new license the first time you start Splunk 4. This is useful if youhave to deploy multiple instances and don't want to have to manually copy the new license in afterstarting Splunk on each machine.

Migrate to 4.0.2 or later, but don't start Splunk yet.• Copy your new license into $SPLUNK_HOME/etc/splunk-user.license.• Start Splunk.•

If you're making a deployable package, you can include the splunk-user.license file with your updatedlicense in it before you tar/zip it up for deployment to other systems.

33

License violations

Violations occur when you exceed the maximum indexing volume allowed for your license. If youexceed your licensed daily volume on any one calendar day, you will get a violation warning. Themessage persists for 14 days. If you have more than 5 violations in a rolling 30-day period, searchwill be disabled. Search capabilities return when you have fewer than 5 violations in the previous 30days or when you apply a new license with a larger volume limit.

Got license violations? Click here to troubleshoot

Note: During a license violation period, Splunk does not stop indexing your data. Splunk onlyblocks search access while you exceed the allowed number of license violations.

34

Start Splunk for the first time

Start Splunk for the first timeStart Splunk for the first time

To start Splunk:

On Windows You can start Splunk on Windows using either the commandline, or the WindowsServices Manager. Using the commandline offers more options, described later in this section. In acmd window, go to C:\Program Files\Splunk\bin and type:

splunk start

(For Windows users: in the subsequent examples and information, replace $SPLUNK_HOME withC:\Program Files\ if you have installed Splunk in the default location.)

On UNIX Use the Splunk command line interface (CLI):

$SPLUNK_HOME/bin/splunk start

Splunk then displays the license agreement and prompts you to accept the before the startupsequence continues.

Other start options

To automatically accept the license when you start Splunk for the first time, add theaccept-license option to the start command:


The startup sequence displays:

Checking prerequisites...Checking http port [8000]: openChecking mgmt port [8089]: openVerifying configuration. This may take a while...Finished verifying configuration.Checking index directory...Verifying databases...Verified databases: _audit, _blocksignature, _internal, _thefishbucket, history, main, sampledata, splunklogger, summaryChecking index filesAll index checks passed.All preliminary checks passed.Starting splunkd...Starting splunkweb...Splunk Server started. The Splunk web interface is at http://<hostname>:8000 If you get stuck, we're here to help. Feel free to email us at '[email protected]'.

Note: If the default ports are already in use (or are otherwise not available), Splunk will offer to usethe next available port. You can either accept this option or specify a port for Splunk to use.

35

There are two other start options: no-prompt and answer-yes:

If you run $SPLUNK_HOME/bin/splunk start --no-prompt, Splunk proceeds withstartup until it requires you to answer a question. Then, it displays the question, why it isquitting, and quits.

•

If you run SPLUNK_HOME/bin/splunk start --answer-yes, Splunk proceeds withstartup and automatically answers "yes" to all yes/no questions. Splunk displays the questionand answer as it continues.

•

If you run start with all three options in one line, for example:

$SPLUNK_HOME/bin/splunk start --answer-yes --no-prompt --accept-license

Splunk does not ask you to accept the license.• Splunk answers yes to any yes/no question.• Splunk quits when it encounters a non-yes/no question.•

Start and disable individual processes

You can start and stop individual Splunk processes by adding the process as an object to the startcommand. The objects include:

splunkd, the Splunk server daemon.• splunkweb, Splunk's Web interface process.•

For example, to start only splunkd:

$SPLUNK_HOME/bin/splunk start splunkd

To disable splunkweb:

$SPLUNK_HOME/bin/splunk disable webserver

For more information about start, refer to the CLI help page:

$SPLUNK_HOME/bin/splunk help start

Launch Splunk Web

Navigate to:

http://mysplunkhost:8000

Use whatever host and port you chose during installation.

The first time you log in to Splunk Enterprise, the default login details are:Username - admin.Password - changeme.

Splunk Free does not have access controls.

36

Migrate from 3.4.x or earlier

What to expect when migrating to 4.0What to expect when migrating to 4.0

This topic discusses the various issues and considerations you should review before migrating to 4.0from 3.4.x or earlier. You'll find information about aspects of your deployment that cannot be migratedautomatically, some changes you will need to make manually, and other changes that the migrationscript will handle.

Note: If you're migrating from versions earlier than 3.4.x, please follow the documentation for thatversion to migrate to 3.4.x and then proceed from there. Migration to 4.x is only supported from 3.4.x.

Before you proceed with migration, you should also review the Known Issues for additionalinformation.

Share your migration experiences

Did you migrate an existing Splunk installation to 4.x and run into some issues? What led to yoursuccess? Share your tips with other users on the Splunk Community Wiki "Migration experiences"page.

Considerations and support for users of Splunk 3.4.x

Splunk 4 is a huge stride forward in performance and flexibility, but there are a few interactionchanges vs. 3.4.x which upgraders should be aware of, and even some reasons why you might wantto wait for a future release before upgrading. Below are some capabilities that have changed with theintroduction of Splunk 4:

Live tail

With Splunk 4's dramatically improved search and indexing speed, along with the ability toprovide intermediate search results, you don't really need a separate live event console to seedata in near real-time. However, if your use case relies on version 3.4.x's "Live tail" feature,you may want to wait on upgrading to Splunk 4. Future roadmap plans involve re-architectingthe live tail functionality to scale across much larger data flows, and across distributedenvironments. Additionally, look out for improve real-time alerting and dashboard updatesdown the road as a result of these upcoming architectural changes.

•

Custom field actions

Based on customer feedback, we decided to re-architect this feature to improve flexibility andallow for event actions based on multiple fields. Expect this functionality to be reintroduced in anear term 4.x release. If you rely on this functionality, but still want to upgrade, you may wantto consider Splunk 4's new "Dynamic field lookups" as an alternative which allows you to mapdata from external databases and lists into Splunk.

•

Snapshots

37

In Splunk 4, we've improved upon 3.x's ability to take a timeline snapshots of individualsearches. Try out Splunk 4's new job manager which allows you to retrieve the entire cachedsearch result, including reports, from existing searches.

•

Event scrolling

In Splunk 4, the new page selector allows you to hop between results with greater flexibility,even as a search runs. However, for those who still prefer a scroll bar, expect this capability tobe re-introduced as an option in a future 4.x release.

•

Timeline and timestamp interaction

In Splunk 4, we improved the timeline to allow users to quickly view any time range withinsearch results, without having to rerun a search. Also try clicking "zoom-in" on the timeline,which now allows you to lock-in a time range, and specify follow on search.

•

We're also planning to improve the usability of some related 3.4.x functionality includingclicking on timestamps, and double clicking on timeline bars in future versions of 4.x.

•

Crawl

Crawl is no longer configurable via Splunk Web, but is still available as a search command.Based on customer feedback, we have decided to re-architect this feature to make it easierand more effective. Expect improved functionality, along with a new user interface to beintroduced in a future release.

•

FIFO inputs

This input type has been deprecated with Splunk 4, and we do not recommend using it as abest practice due to data loss considerations. If you currently rely on this input type considerwriting the output to a flat file that Splunk is monitoring.

•

Persistent Queue

This feature has been deprecated with Splunk 4. When a forwarder loses its connection to theindexer, it will block instead of continuously writing events to disk. If you are concerned aboutdata loss, consider using TCP or file inputs.

•

RSS alerts

RSS alerts are not supported for the initial realease of the 4.x line.•

Let us know when you're ready to migrate, and we'll be here to help.

Migrating your license

Splunk 4.x does not work with licenses from older releases.

If you're migrating to 4.0 or 4.0.1, AFTER YOU INSTALL, YOU WILL SEE A MESSAGESTATING THAT YOUR LICENSE IS NO LONGER VALID.

•

38

If you are an current Enterprise customer, check your splunk.com orders page for an updatedlicense.

•

If you're migrating to 4.0.2, your 3.x license is backed up and replaced with an Enterprise triallicense when you start Splunk 4.

•

If you are running with a 3.x Free or Enterprise trial license, delete the$SPLUNK_HOME/etc/splunk.license file before you start Splunk 4.x. The instance willthen pick up the 60-day Enterprise trial license.

•

Pre-seeding your license before first time run

Starting with 4.0.2, by default when you start Splunk for the first time, it moves aside any existing 3.xlicense and replaces it with a temporary Enterprise trial license. This allows you to bring up the newversion of Splunk without having your license be expired until you get your new one copied in.

If you are migrating to Splunk 4.0.2 or later and have a valid 4.x license, you can pre-seed the licensefile so that it pulls in and installs your new license the first time you start Splunk 4. This is useful if youhave to deploy multiple instances and don't want to have to manually copy the new license in afterstarting Splunk on each machine.

Migrate to 4.0.2 or later, but don't start Splunk yet.• Copy your new license into $SPLUNK_HOME/etc/splunk-user.license.• Start Splunk.•

If you're making a deployable package, you can include the splunk-user.license file with your updatedlicense in it before you tar/zip it up for deployment to other systems.

Which migration is right for you?

If you have not customized your 3.x deployment extensively, you may be able to use the automaticmigration process described in either Migrate on Windows or Migrate on UNIX.

If you have customized your 3.x deployment to any degree, (in particular, if you have customizeddeployment.conf), you may have to consider migrating your deployment manually. In this case, followthe Steps for manual migration to Splunk 4.x.

What cannot be migrated

Splunk 4.x is significantly different from earlier versions. Some configuration files from your 3.xdeployment can be copied over to your new 4.x install without any migration required. However,some aspects of your existing deployment cannot be migrated, and must be rebuilt; this is mostrelevant for 3.x deployments and configurations that have been extensively customized.

Splunk Web display customizations

All things to do with Splunk Web have been completely re-architected and rebuilt from the ground up.As a result, any customizations you've made that affect the display of Splunk Web cannot bemigrated; you must rebuild them using the 4.x architecture and tools. These include:

Form searches• Field actions (field_actions.conf)•

39

Dashboards• UI preferences (prefs.conf)• Report chart and table preferences on saved searches• Changes to UI strings (literals.conf)•

Considerations for apps

In general, apps do not get migrated; you should re-architect your apps to follow the new 4.0architecture. For more information about apps in 4.0, refer to the Developer Manual. For moreguidance on migrating your apps, consult this topic on migrating your 3.x apps in the DeveloperManual.

Not all Splunk 3.4.x apps will be immediately available for 4.0. New apps will be added to the SplunkApp Store as they become available.

Splunk for Change Management has not yet been made available for Splunk 4, If you have thisapp, contact Support before upgrading, or consider installing a parallel instance of Splunk to maintainit.

Considerations for deployment server and forwarders

4.x deployment server is not backwards compatible with 3.x deployment clients.

If you use the Splunk deployment server, you must rebuild your deployment configuration usingthe new deployment server architecture in 4.x; it cannot be migrated. If you've made changes todeployment.conf in version 3.x, you should not copy it over into your new Splunk 4.x installation.Set it aside to use as a basis for building your new deployment server configuration.

In the meantime, if you have 3.4.x deployment server and clients and do not want to migrate all theclients at this time, you can use the instructions in this topic to configure a stripped-down 3.4.xdeployment server to continue managing your deployment clients.

If you have 3.4.x forwarders, you can delay migrating them to 4.x; 3.4.x forwarders will work with a4.x version of Splunk. This is especially useful for large forwarder deployments. You can wait untilyou are comfortable with your new deployment server configuration before migrating forwarders to4.x.

In summary:

3.x forwarders will work with a 4.x indexer.• 3.x deployment clients will not work with 4.x deployment server.•

Considerations for distributed search

4.0.x distributed search is not backwards compatible with 3.x distributed search. In addition, thedeployment server classes are not directly applicable to 4.x deployment server configuration. 4.xdeployment servers use the concept of "apps" to push configuration to clients, as opposed to classes.Admins must build an application from their classes configuration to push to clients. It isrecommended to do this first, before losing configuration parity at upgrade time, for clients running4.x.

40

If you use distributed search, all your participating distributed search nodes will need to run Splunk4.x in order to successfully communicate and return results.

Using a 4.0 or 4.0.1 Splunk deployment derver with 3.x deployment clients will crash the clients

This issue was resolved in 4.0.2.

To ensure your 4.x server doesn't contact your existing 3.x deployment clients, change themanagement port on the 4.x instance. Add the following to$SPLUNK_HOME/etc/system/local/web.conf to change it from the default 8089 port to 8090:

[settings]mgmtHostPort = 127.0.0.1:8090

What you must migrate manually

Splunk 4.x differs significantly from version 3.x, and the migration script cannot convert the content ofsome configuration files. This section describes some of the changes that you must make manually.

Considerations for scheduled searches and alerts

Scheduled searches and alerts do not migrate automatically. In the 4.0 knowledge model, asearch cannot execute outside of an app context. Thus, saved searches are considered part of theSearch app; they are not global and viewable across all apps.

To migrate your saved searches:

1. Stop Splunk; execute the command:

$SPLUNK_HOME/bin/splunk stop

2. Move $SPLUNK_HOME/etc/system/local/savedsearches.conf to$SPLUNK_HOME/etc/apps/search/local/

3. Move any stanzas whose name starts with "savedsearches/" from$SPLUNK_HOME/etc/system/metadata/local.meta to$SPLUNK_HOME/etc/apps/search/metadata/local.meta

4. Start Splunk; execute the command:


Note: Scheduled searches will not run unless these 2 steps are completed.

Considerations for tags and aliases

Sourcetype renaming replaces "sourcetype aliasing" and isn't implemented by tags. For moreinformation, see "About tags and aliases" in the Knowledge Manager manual. Saved searches thatrely on aliased sourcetypes won't work without migration. For steps to migrate these savedsearches, see migration concerns for scheduled searches and alerts.

41

Considerations for crawl.conf

The crawl.conf stanza [file_crawler] should be renamed [files]. This does not happenautomatically.

No migration concerns for CLI

The command line interface, CLI, has been completely rewritten for Splunk 4.x. There are no knownmigration or backwards compatibility issues. Refer to release notes for the list of changes to the CLI,which include deprecated CLI command options and new functionality.

Changes made during automatic migration

During migration, many configuration files cannot simply be copied over. Splunk provides scripts forconverting and migrating the content of these files so that they will function correctly in 4.x. You canrun the migration preview utility to see what will be changed before you actually upgrade and migrate.The following is a list of some of the changes that take place during migration.

alert_actions.conf

This configuration file is migrated with savedsearches.conf.

indexes.conf

During migration, some attributes are added to indexes.conf while other local attributes are eitherremoved or changed to global parameters. For more details about these parameters, refer toindexes.conf in the Admin manual.

The following attributes are no longer supported and have no effect:

_actions• maxTermChars• maxTerms• maxPostings• maxValues• waitForOptimize•

The following global attributes are added with these defaults:

indexThreads = auto• maxMemMB = 5• memPoolMB = auto• maxHotSpanSecs = 7776000• maxHotIdleSecs = 0• maxHotBuckets = 1• quarantinePastSecs = 77760000• quarantineFutureSecs = 2592000•

For high-volume indexes (such as main), the following defaults are used:

maxHotBuckets = 10•

42

maxHotIdleSecs = 86400• maxMemMB = 20•

savedsearches.conf

During migration, form searches are disabled, a default global view state is set, and owner attributesare moved so that they follow the 4.0 roles and permissions model.

if the search attribute contains macros ($<strings>$), disabled = 1 is added to the stanza.• if the attribute defines a view state &mdash, which is any line that begins with "viewstate.",the line is commented out.

•

if the stanza contains 'userid' and/or 'role' attributes, a corresponding metadata owner/ACL isadded.

•

server.conf

During migration, the following attributes are moved from splunkd.xml to server.conf:

minFreeMb• pollingFrequency• serverName•

The following attributes, for the server certificate and key file and password, are renamed:

keysfile is replaced with sslkeysfile• keysfilepassword is replaced with sslkeysfilePassword•

Windows specific conf files

During migration, Windows-specific files (regmon-filters.conf, sysmon.conf, and wmi.conf) andknowledge objects are moved out of default and into the Windows app architecture.

Conf files that are removed

During migration, the following conf files are removed:

typedefs.conf• searchdata.conf•

Index data compatibility

Data indexed by version 3.x is completely compatible with 4.x and does not need to be re-indexed.However, note that it may have been indexed in a less efficient form, so searches performed overdata that was originally indexed and migrated from 3.x may not be as fast as searches overnewly-indexed 4.x data.

Migrate on UNIX

43

Migrate on UNIX

When you migrate to 4.0, your configuration files will be updated and changed. You can run themigration preview utility to see what will be changed before you actually upgrade and migrate. Whenyou do this, a file containing the changes that the script proposes to make is written to:

$SPLUNK_HOME/var/log/splunk/migration.log.<timestamp>

Before you migrate

Before you perform the migration, we strongly recommend that you review these migrationconsiderations and back up all of your files, including Splunk configurations, data and binaries.Splunk does not provide a means of downgrading to previous versions; if you need to revert to anolder Splunk release, just reinstall it.

Migration instructions

1. Execute the $SPLUNK_HOME/bin/splunk stop command.

2. Disable any 'coldToFrozen' scripts in $SPLUNK_HOME/etc/system/local/indexes.conf. If you leavethem in place they could potentially run when you start 4.0 for the first time and cause your instanceto appear unresponsive.

3. Install the Splunk package over your existing Splunk deployment.

If you are using a TAR file, expand it into the same directory as your existing Splunk instance. Thisoverwrites and replaces matching files but does not remove unique files.

If you are using a package manager, such as an RPM:

rpm -U splunk_package_name.rpm

4. Execute the $SPLUNK_HOME/bin/splunk start command.

The following output is displayed:

This appears to be an upgrade of Splunk.

--------------------------------------------------------------------------------

Splunk has detected an older version of Splunk installed on this machine. Tofinish upgrading to the new version, Splunk's installer will automaticallyupdate and alter your current configuration files. Deprecated configurationfiles will be renamed with a .deprecated extension.

You can choose to preview the changes that will be made to your configurationfiles before proceeding with the migration and upgrade:

If you want to migrate and upgrade without previewing the changes that will bemade to your existing configuration files, choose 'y'.If you want to see what changes will be made before you proceed with theupgrade, choose 'n'.

44

Perform migration and upgrade without previewing configuration changes? [y/n]

4. You're given the choice of running the migration preview script to see what changes will be madeto your existing configuration files, or proceeding with the migration and upgrade right away.

5. If you choose to view the expected changes, the script provides a list.

6. Once you've reviewed these changes and are ready to proceed with migration and upgrade, run$SPLUNK_HOME/bin/splunk start again.

Note: You can complete Steps 3 to 5 in one line:

To accept the license and view the expected changes (answer 'n') before continuing the upgrade:

$SPLUNK_HOME/bin/splunk start --accept-license --answer-no

To accept the license and begin the upgrade without viewing the changes (answer 'y'):

$SPLUNK_HOME/bin/splunk start --accept-license --answer-yes

Migrate on WindowsMigrate on Windows

When you migrate, your configuration files are updated and changed to support the new functionality.You can run the migration preview utility to see what will be changed before you actually upgrade andmigrate. When you do this, a file containing the changes that the script proposes to make is writtento:

$SPLUNK_HOME/var/log/splunk/migration.log.<timestamp>

Before you migrate

Review the migration concerns in "What to expect when upgrading to 4.0".• Back up your files, including Splunk configurations, data and binaries.• Stop Splunk either using the Windows Start menu option or by executing the$SPLUNK_HOME\bin\splunk stop command.

•

Be aware that you cannot change the user Splunk runs as during an upgrade. Do not changethe user from the Windows Service Control panel; Splunk will stop working. If you must changethe user, you must uninstall and reinstall Splunk.

•

The Windows App was enabled by default in versions 4.0-4.0.2. Starting in version 4.0.3, it isdisabled by default unless you enable it explicitly via the commandline/MSI installationprocess.

•

Disable any 'coldToFrozen' scripts in $SPLUNK_HOME\etc\system\local\indexes.conf. If youleave them in place they could potentially run when you start 4.0 for the first time and causeyour instance to appear unresponsive.

•

45

Upgrade instructions

1. Download the new MSI file from the Splunk download page.

2. Double-click the MSI file.

The Welcome panel is displayed. Follow the onscreen instructions to upgrade Splunk.

For information about each panel, refer to the installation instructions.

When you reach the Install step, you have the option to preview changes that will be made for thisupgrade.

3. Preview your upgrade and migration if desired.

The following text is displayed:

This appears to be an upgrade of Splunk.

--------------------------------------------------------------------------------

Splunk has detected an older version of Splunk installed on this machine. Tofinish upgrading to the new version, Splunk's installer will automaticallyupdate and alter your current configuration files. Deprecated configurationfiles will be renamed with a .deprecated extension.

You can choose to preview the changes that will be made to your configurationfiles before proceeding with the migration and upgrade:

If you want to migrate and upgrade without previewing the changes that will bemade to your existing configuration files, choose 'y'.If you want to see what changes will be made before you proceed with theupgrade, choose 'n'.

Perform migration and upgrade without previewing configuration changes? [y/n]

4. You're given the choice of running the migration preview script to see what changes will be madeto your existing configuration files, or proceeding with the migration and upgrade right away.

5. If you choose to view the expected changes (select N), the script provides a list.

You can scroll up to review the changes or look at them in$SPLUNK_HOME\var\log\splunk\migration.log.<timestamp>. At the end of the list, youwill see an error message, which you can ignore.

6. Press Enter to return to step 3 and finish your upgrade by typing Y.

Important: When migrating or upgrading, you must re-specify the user you want Splunk to runas--this information is not automatically maintained from release to release.

46

Start Splunk

On Windows, Splunk is installed by default into \Program Files\Splunk

You can start and stop the following Splunk processes via the Windows Services Manager:

Server daemon: splunkd• Web interface: splunkweb•

You can also start, stop, and restart both processes at once by going to \ProgramFiles\Splunk\bin and typing

# splunk.exe [start|stop|restart]

Note: If you do not select Start Splunk Services now, they will be set to manual startup andtherefore will not start after a reboot. You must start them from the Windows Service Manager MMC,and optionally configure auto-start if you want them to start automatically at boot time.

Important: After upgrading, Splunk may start reading some files incorrectly as binaries. You canoverride this behavior in props.conf by adding the following line to a source stanza:NO_BINARY_CHECK = true

Steps for manual migration to Splunk 4.xSteps for manual migration to Splunk 4.x

This topic describes the procedure for migrating a 3.x Splunk installation to version 4.x manually.Before you proceed, be sure to review "What to expect when migrating to 4.0" in this manual.

If you have not customized your 3.x deployment extensively, you may be able to use the automatic"in-place" migration process described in the Migrate on Windows or Migrate on UNIX topics. Inparticular, if you have not customized deployment.conf, you can likely use the automatic migrationprocess.

Some custom configurations cannot be migrated at all, and must be rebuilt within the new Splunkframework. Refer to the section titled "What cannot be migrated" in this topic and "What to expectwhen migrating to 4.0" in this manual for more details.

Share your migration experiences

Did you migrate an existing Splunk installation to 4.x and run into some issues? What led to yoursuccess? Share your tips with other users on the Splunk Community Wiki "Migration experiences"page.

Overview of the manual migration process

During the manual procedure, you will make a backup copy of your existing deployment, installSplunk 4.x in a separate path from 3.x, and then migrate your data and configuration files across fromyour 3.x deployment. This procedure is designed to allow for a minimum of downtime.

47

It's important to follow this procedure carefully--Splunk 4.x differs significantly from version 3.x, andmany configuration files cannot simply be copied over. Splunk provides scripts for converting andmigrating the content of these files so that they will function correctly in 4.x.

Read through this entire topic before beginning the procedure. This way, you will know whatto expect, and can plan accordingly.

What cannot be migrated

As mentioned earlier, Splunk 4.x is significantly different from earlier versions. Some aspects of yourexisting deployment cannot be migrated, and must be rebuilt.

Splunk Web display customizations

All things to do with Splunk Web have been completely re-architected and rebuilt from the ground up.As a result, any customizations you've made that affect the display of Splunk Web cannot bemigrated; you must rebuild them using the 4.x architecture and tools. These include:

Form searches• Field actions (field_actions.conf)• Dashboards - visit here for instructions on how to rebuild them• UI preferences (prefs.conf)• Report chart and table preferences on saved searches• Changes to UI strings (literals.conf)• 3.x Apps - read this Developer Manual topic for guidance on reworking them.•

Considerations for deployment server and forwarders

If you use the Splunk deployment server, you must rebuild your deployment configurationusing the new deployment server architecture in 4.x; it cannot be migrated. If you've madechanges to deployment.conf in version 3.x, you should not copy it over into your newSplunk 4.x installation. Set it aside to use as a basis for building your new deployment serverconfiguration.

•

If you have deployed forwarders, you can delay migrating them to 4.x; 3.x forwarders will workwith a 4.x version of Splunk. This is especially useful for large forwarder deployments. You canwait until you are comfortable with your new deployment server configuration before migratingforwarders to 4.x.

•

Version 4.x forwarders do not work with version 3.x deployment server.•

Before beginning the migration

Before you begin migrating your Splunk data and configuration files, make a backup copy of your 3.xinstallation. The easiest way to do this is to run the Splunk CLI diag utility, which will create acompressed (.tar) file containing your entire installation. To do this, from $SPLUNK_HOME/bin type:

On UNIX:

./splunk diag

On Windows:

48

splunk diag

If you have difficultly running diag in your environment, you can also run the python script directlyusing the cmd command:

./splunk cmd python /opt/splunk/lib/python2.5/site-packages/splunk/clilib/info_gather.py

This produces splunk-diag.tar.gz|zip that you can preserve as a backup of your 3.xdeployment.

Once you have done this, use the instructions in this manual to install version 4.0 for your platform ina different location from your existing 3.x install but do not start it yet.

Do not start your 4.x Splunk instance yet.

Import configuration files that do not require manual migration

Some of the configuration files from your 3.x deployment can be copied over to your new 4.x installwithout any migration required.

Remember to copy these files from/to $SPLUNK_HOME/etc/system/local or from your customconfig directory--do not copy over the files in /default!

Copy over all configuration files except deployment.conf.• Edit the copy of inputs.conf you put on your new 4.x Splunk instance to set all the inputs todisabled=true.

•

Copy over the etc/myinstall/splunkd.xml file•

Import user and password information

You can bring your user data, password, and other authentication information across at this point:

authentication.conf• authorize.conf• splunk.secret• passwd file•

Do not start your 4.x Splunk install yet.

Migrate your Splunk index data

This section explains how to configure your new 4.x install to read your existing Splunk index data.Data indexed by version 3.x is completely compatible with 4.x and does not need to be re-indexed.However, note that it may have been indexed in a less efficient form, so searches performed overdata that was originally indexed and migrated from 3.x may not be as fast as searches overnewly-indexed 4.x data.

Again, be sure to follow these instructions carefully to minimize downtime and data loss. Order isimportant! Additionally, be prepared to perform these steps in rapid succession. Once you roll the'hot' index data directory to 'warm' on your existing Splunk deployment, you'll want to immediately

49

move on to the next steps as quickly as possible to ensure there is a minimum of downtime in thetransition to indexing with your new 4.x Splunk install.

To understand the implications of "hot" and "warm" index directories, review the "Best practices forbacking up" topic in the Splunk Community Wiki.

1. In the root directory of your 4.x install, edit splunk-launch.conf to point the value of SPLUNK_DB topoint to the index data directories of your 3.x deployment. By default, this value is/opt/splunk/var/lib/splunk, but you can check the splunk-launch.conf in your 3.x deploymentto be sure you've got the right path.

2. Then, in your version 3.x deployment, use the CLI to force a roll of any data in your hot index datadirectory to the warm' directory. To do this, type:

./splunk search '| oldsearch !++cmd++::roll' -authadmin:<ADMIN_PASSWORD>

You'll see an error about "Search Execute failed because Hot db rolled out to warm"right afterward; you can safely ignore it. You'll also need to provide the admin passwordto execute this CLI command.

♦

To roll any other indexes you are currently writing to, use:♦ ./splunk search ' | oldsearch index=<INDEX_NAME> !++cmd++::roll'-auth admin:<ADMIN_PASSWORD>

♦

•

3. As soon as you've rolled all your indexes from hot to warm, immediately shut down your 3.xdeployment and start your 4.x install up. When you start your 4.x, some of the configuration files youcopied over may undergo an auto-migration process. Splunk will display information about thisprocess when you start 4.x for the first time.

4. As quickly as you can, log into your new 4.x install and search for data you know is in your 3.xindex, to validate all the migration steps. Ensure that your new deployment is able to search and finddata in your 3.x indexes. If the validation fails, immediately restart your 3.x deployment beforeproceeding with any troubleshooting.

5. Once you have successfully validated that your 4.x deployment can search your 3.x index data,immediately edit inputs.conf and set your inputs to disabled=false and restart your 4.x Splunkdeployment to enable them. Verify that data is coming into your new deployment.

Now you're ready to begin using your new 4.x Splunk deployment!

If you use distributed search

If you have configured distributed search, you must distribute new keys to all search peers to usedistributed search in 4.x. These keys are generated at startup if distributed search is enabled.

You can do this via the UI on the search head. For each of your search peers, re-enter the remoteusername/password through the Manager Distributed search pages.

Or you can do this manually. The keys will be generated in

50

$SPLUNK_HOME/etc/auth/distServerKeys/. Distribute the files$SPLUNK_HOME/etc/auth/distServerKeys/trusted.pem and private.pem from one hostto any others distributed search peers.

51

Upgrade from 4.0 or later

Upgrade Splunk on WindowsUpgrade Splunk on Windows

This topic describes the procedure for upgrading your Windows Splunk instance from version 4.x to alater version. You can upgrade using the GUI installer, or by running msiexec on the commandline asdescribed in "Install on Windows via the commandline".



•

If you're doing a fresh installation of 4.0.3 or later, the Windows App is enabled by default viathe MSI and if you want to install it in a disabled state, you must specify this using theSPLUNK_APP msiexec command as described in "Install on Windows via the commandline".

•

Before you upgrade

Important: When upgrading, you must explicitly specify the same domain user that you specifiedduring first time install. If you do not specify the same user, Splunk will default to using the LocalSystem User. If you accidentally specify the wrong user during your installation, use the instructionsin these instructions to switch to the correct user before starting Splunk.

Important: Before you perform the upgrade, we strongly recommend that you back up all of yourfiles, including Splunk configurations, data and binaries. Splunk does not provide a means ofdowngrading to previous versions; if you need to revert to an older Splunk release, just reinstall it.

Upgrading using the GUI installer

1. Stop Splunk either using the Windows Start menu option or by executing the$SPLUNK_HOME/bin/splunk stop command.


3. Double-click the MSI file. The Welcome panel is displayed. Follow the onscreen instructions toupgrade Splunk. For information about each panel, refer to the installation instructions.

4. Splunk will start up by default when you complete the installation.

A log of the changes made to your configuration files during the upgrade is placed in $TEMP$.

Upgrading using the commandline

1. Stop Splunk either using the Windows Start menu option or by executing the$SPLUNK_HOME/bin/splunk stop command.

52


3. Use the instructions in "Install on Windows via the commandline". If Splunk is running as a userother than the Local System user, you must explicitly specify this user in your commandline. You canchange the ports (SPLUNKD_PORT and WEB_PORT) at this time, and also use theLAUNCHSPLUNK option to specify whether Splunk should start up automatically or not when you'refinished, but you cannot change any other settings.

4. Depending on your specification, Splunk may start automatically when you complete theinstallation.

A log of the changes made to your configuration files during the upgrade is placed in $TEMP$.

Start Splunk

On Windows, Splunk is installed by default into \Program Files\Splunk and is started by default.

You can start and stop the following Splunk processes via the Windows Services Manager:

Server process: splunkd• Web interface process: splunkweb•

You can also start, stop, and restart both processes at once by going to \ProgramFiles\Splunk\bin and typing

# splunk [start|stop|restart]

Upgrade Splunk on Linux, Solaris, FreeBSD, HP-UX, AIX, andMacOSUpgrade Splunk on Linux, Solaris, FreeBSD, HP-UX, AIX, and MacOS

This topic describes the procedure for upgrading your Splunk instance from version 4.x to a laterversion.

How upgrading works

When you upgrade your configuration files are not actually changed until you start Splunk afterperforming the installation of the new version. You can run the migration preview utility at that time tosee what will be changed before the files are updated. If you choose to view the changes beforeproceeding, a file containing the changes that the upgrade script proposes to make is written to$SPLUNK_HOME/var/log/splunk/migration.log.<timestamp>

Important: Before you perform the upgrade, we strongly recommend that you back up all of yourfiles, including Splunk configurations, data and binaries. Splunk does not provide a means ofdowngrading to previous versions; if you need to revert to an older Splunk release, just reinstall it.

53

Steps for upgrading

1. Execute the $SPLUNK_HOME/bin/splunk stop command.

2. To upgrade and migrate from version 4.0 and later, install the Splunk package over your existingSplunk deployment:

If you are using a .tar file, expand it into the same directory as your existing Splunk instance.This overwrites and replaces matching files but does not remove unique files.

•

If you are using a package manager, such as an RPM, type rpm -Usplunk_package_name.rpm

•

If you are using a .dmg file (on MacOS), double-click it and follow the instructions. Be surespecify the the same installation directory as your existing installation.

•

3. Execute the $SPLUNK_HOME/bin/splunk start command.

The following output is displayed:

This appears to be an upgrade of Splunk.--------------------------------------------------------------------------------Splunk has detected an older version of Splunk installed on this machine. Tofinish upgrading to the new version, Splunk's installer will automaticallyupdate and alter your current configuration files. Deprecated configurationfiles will be renamed with a .deprecated extension.You can choose to preview the changes that will be made to your configurationfiles before proceeding with the migration and upgrade:If you want to migrate and upgrade without previewing the changes that will bemade to your existing configuration files, choose 'y'.If you want to see what changes will be made before you proceed with theupgrade, choose 'n'.Perform migration and upgrade without previewing configuration changes? [y/n]

4. Choose whether you want to run the migration preview script to see what changes will be made toyour existing configuration files, or proceed with the migration and upgrade right away.

5. If you choose to view the expected changes, the script provides a list.

6. Once you've reviewed these changes and are ready to proceed with migration and upgrade, run$SPLUNK_HOME/bin/splunk start again.

Note: You can complete Steps 3 to 5 in one line:

To accept the license and view the expected changes (answer 'n') before continuing the upgrade:

$SPLUNK_HOME/bin/splunk start --accept-license --answer-no

To accept the license and begin the upgrade without viewing the changes (answer 'y'):

$SPLUNK_HOME/bin/splunk start --accept-license --answer-yes

54

Other tasks

Run Splunk as a different or non-root userRun Splunk as a different or non-root user

You can run Splunk as any user on the local system. If you run Splunk as a non-root user, make sureSplunk has the appropriate permissions to:

Read the files and directories it is configured to watch. Some log files and directories mayrequire root or superuser access to be indexed.

•

Write to Splunk's directory and execute any scripts configured to work with your alerts orscripted input.

•

Bind to the network ports it is listening on (ports below 1024 are reserved ports that only rootcan bind to).

•

Note: Because ports below 1024 are reserved for root access only, Splunk will only be able to listenon port 514 (the default listening port for syslog) if it is running as root. You can, however installanother utility (such as syslog-ng) to write your syslog data to a file and have Splunk monitor that fileinstead.

Instructions

To run Splunk as a non-root user, you need to first install Splunk as root. Then, before you startSplunk for the first time, change the ownership of the splunk directory to the desired user. Thefollowing are instructions to install Splunk and run it as a non-root user, splunk.

1. Create the user and group, splunk.

For Linux, Solaris, and FreeBSD:

useradd splunkgroupadd splunk

For Mac OS:

You can use the System Preferences > Accounts panel to add users and groups.

2. As root and using one of the packages (not a tarball), run the installation.

Important: Do not start Splunk yet.

3. Use the chown command to change the ownership of the splunk directory and everything under itto the desired user.

chown -R splunk $SPLUNK_HOME/

Note: $SPLUNK_HOME refers to installation directory of Splunk.

55

4. Start Splunk.


Also, if you want to start Splunk as the splunk user while you are logged in as a different user, youcan use the sudo command:

sudo -H -u splunk $SPLUNK_HOME/bin/splunk start

This example command assumes:

If Splunk is installed in an alternate location, update the path in the command accordingly.• Your system may not have sudo installed. If this is the case, you can use su.• If you are installing using a tarball and want Splunk to run as a particular user (such assplunk), you must create that user manually.

•

The splunk user will need access to /dev/urandom to generate the certs for the product.•

Solaris 10 privileges

When installing on Solaris 10 as the splunk user, you must set additional privileges to startsplunkd and bind to reserved ports.

To start splunkd as the splunk user on Solaris 10, run:

# usermod -K defaultpriv=basic,net_privaddr,proc_exec,proc_fork splunk

To allow the splunk user to bind to reserved ports on Solaris 10, run (as root):

# usermod -K defaultpriv=basic,net_privaddr splunk

Uninstall SplunkUninstall Splunk

Before you uninstall, stop Splunk. Navigate to $SPLUNK_HOME/bin and type ./splunk stop (orjust splunk stop on Windows).


Note: $SPLUNK_HOME refers to the Splunk installation directory. On Windows, this is C:\ProgramFiles\Splunk by default. For most Unix platforms, the default installation directory is /opt/splunk;for Mac OS, it is /Applications/splunk.

RedHat Linux

To uninstall Splunk on RedHat:

rpm -e splunk_product_name

56

Debian Linux

To uninstall Splunk on Debian:

dpkg -r splunk

To purge (delete everything, including configuration files) on Debian:

dpkg -P splunk

FreeBSD

To uninstall Splunk from the default location on FreeBSD:

pkg_delete splunk

To uninstall Splunk from a different location on FreeBSD:

pkg_delete -p /usr/splunk splunk

Solaris

To uninstall Splunk on Solaris:

pkgrm splunk

Windows

To uninstall Splunk on Windows:

Use the Add or Remove Programs option in the Control Panel.

Uninstall Splunk manually

If you can't use package management commands, use these instructions to uninstall Splunk.

Note: These instructions will not remove any init scripts that have been created.

1. Stop Splunk.

$SPLUNK_HOME/bin/splunk stop

2. Find and kill any lingering processes that contain "splunk" in its name.

For Linux and Solaris:

kill -9 `ps -ef | grep splunk | grep -v grep | awk '{print $2;}'`

For FreeBSD and Mac OS

kill -9 `ps ax | grep splunk | grep -v grep | awk '{print $1;}'`

57

3. Remove the Splunk installation directory, $SPLUNK_HOME. For example:

rm -rf /opt/splunk

Note: For Mac OS, you can also remove the installation directory by dragging the folder into thetrash.

3. Remove any Splunk datastore or indexes outside the top-level directory, if they exist.

rm -rf /opt/splunkdata

4. Delete the splunk user and group, if they exist.

For Linux, Solaris, and FreeBSD:

userdel splunkgroupdel splunk

For Mac OS: You can use the System Preferences > Accounts panel to manage users andgroups.

For Windows: Open a command prompt and run the command msiexec /x against the msipackage that you installed.

Correcting the user selected during Windows installationCorrecting the user selected during Windows installation

If you have selected "other user" during the Windows GUI installation, and that user does not exist orperhaps you mistyped the information, you can go into the Windows Service Control Manager andspecify the correct information, as long as you have not started Splunk yet.

If you specified an invalid user during the Windows GUI installation process, you will see two popuperror windows.

To change the user:

1. In Control Panel > Administrative Tools > Services, find the Splunkd and SplunkWeb services.You'll note they are not started and are currently owned by the Local System User.

2. Right click on each service and choose Properties. The properties dialog for that service isdisplayed.

3. Select the Log On tab.

4. Select the This account radio button and fill in the correct domain\username and password.

5. Click Apply.

6. Click OK.

58

6. Repeat for the second service (you must do this for both Splunkd and Splunk Web.

7. You can now either start both services from the Service Manager or from the Splunk command lineinterface.

Configure a standalone 3.4.x deployment serverConfigure a standalone 3.4.x deployment server

If you are planning to migrate to Splunk 4.x, but do not want to migrate your deployment clients until alater time, you can set up a stripped-down, standalone 3.4.x deployment server to serve yourdeployment clients until you're ready to migrate them (Splunk 4.x deployment server is incompatiblewith clients older than 4.x).

This procedure assumes the following:

You have an existing deployment server at fflanda.splunk.com listening on port 8089 fordeployment clients.

•

Your deployment clients are all 3.x and are all polling this deployment server.• The deployment classes are in $SPLUNK_HOME/etc/modules/distributedDeployment/classes.• This Splunk instance is also an index server that must be upgraded.•

Given the above, the procedure is as following:

1. Download the latest Splunk 3.4.x build for your architecture.

2. Back up the existing $SPLUNK_HOME/etc using tar -zxvf $SPLUNK_HOME/etc >/tmp/splunk_old_etc.tgz

3. Stop Splunk, remove deployment.conf and deployment classes.

4. If $SPLUNK_HOME = /opt/splunk, mv to /opt/splunk_old. Otherwise, install the 3.4.x tarball or rpmin the default location

5. Extract the splunk_old_etc.tgz over top of the fresh installation.

6. Remove/rename any inputs.conf/outputs.conf files in /opt/splunk_depserver/etc/system/local or/opt/splunk_depserver/etc/apps/. You will probably want to keep authentication.conf, server.conf,/opt/splunk/etc/passwd, /opt/splunk/etc/auth/* - pretty much anything but inputs.conf, outputs.conf andthe unchanged splunk-launch.conf.

7. Disable Splunk Web on the newly installed instance using the CLI or web.conf.

8. Execute mv /opt/splunk /opt/splunk_depserver

9. Edit /opt/splunk_depserver/etc/splunk-launch.conf to change $SPLUNK_HOME to/opt/splunk_depserver. If $SPLUNK_DB is also set, comment out this variable so that the newinstance does not try to write to the old data store.

59

10. To ensure that this deployment server remains functional, switch its license out for a 3.x forwarderlicense. Copy $SPLUNK_HOME/etc/splunk-forwarder.license to $SPLUNK_HOME/etc/splunk.license.

11. Execute /opt/splunk_depserver/bin/splunk start

12. Execute mv /opt/splunk_old /opt/splunk, then perform migration.

13. During post-migration start up, Splunk will notice that the old management port is bound, and willprompt the admin to change the management port. Keep track of this new port as you mustupdate it in any distributed search or REST configurations.

14. Execute /opt/splunk_depserver/bin/splunk list deploy-clients -authadmin:changeme and verify that the deployment clients have been in touch with the deploymentserver.

14. Review /opt/splunk_depserver/var/log/splunk/splunkd.log and/opt/splunk/var/log/splunk/splunkd.log for errors.

60

What comes next?

Ready to start using Splunk?Ready to start using Splunk?

Now that you've got Splunk installed on one server, here're some links to get you started:

Learn what Splunk is, what it does, and how it's different.• Learn how to add your data to Splunk.• Add and manage users.• Learn how to search, monitor, report, and more• Learn about Splunk knowledge and how you can use it to maximize Splunk's value.• Plan your deployment, from gigabytes to terabytes per day.•

Estimating your storage requirementsEstimating your storage requirements

This topic describes how to estimate the size of your Splunk index on disk and associated data sothat you can plan your storage capacity requirements.

When Splunk indexes your data, the resulting data falls into two basic categories: the compressedraw data that is persisted and the indexes that point to this data. With a little experimentation, you canestimate how much disk space you will need.

Typically, the compressed, persisted data that Splunk extracts from your data inputs amounts toapproximately 10% of the raw data that comes into Splunk. The indexes that are created to accessthis data can be anywhere from 10% to 110% of the data that comes in. This value is affectedstrongly by how many unique terms occur in your data. Depending on the characteristics of your data,you might want to tune your segmentation settings later on. For an introduction to how segmentationworks and how it affects your index size, you can also watch this video on segmentation by one ofSplunk's lead developers.

The best way to get an idea of your index size is to experiment by installing a test copy of Splunksomewhere and indexing a representative sample of your data, and then checking the sizes of theresulting directories defaultdb.

Here's how:

Once you've indexed your sample:

1. Go to $SPLUNK_HOME/var/lib/splunk/defaultdb/db/db-hot.

2. Run du -shc hot_v*/rawdata to determine how large the compressed persisted raw data is.

3. Run du -ch hot_v* and look at the last total line to see the size of the index.

61

This is the persisted data to which the items in the index point. Typically, this file's size is about 10%of the size of the sample data set you indexed.

4. Add the values you get together.

This is the total size of the index and associated data for the sample you have indexed. You can nowuse this to extrapolate the size requirements of your Splunk index and rawdata directories overtime.

Hardware capacity planning for your Splunk deploymentHardware capacity planning for your Splunk deployment

Splunk is a very flexible product that can be deployed to meet almost any scale and redundancyrequirement. However, that doesn't remove the need for care and planning. This article discusseshigh level considerations for Splunk deployments, including sizing and availability.

After you've worked through the general layout of your Splunk search topology, the other sections inthis document can explain more thoroughly how to implement them, along with the formal Adminguide for Splunk.

Reference Hardware

Let's consider a common, commodity hardware server as our standard:

Intel 64-bit chip architecture• Standard Linux or Windows 64-bit distribution• 2 CPU, 4 core per CPU, 2.5-3Ghz per core• 8GB RAM• 4x300GB SAS hard disks at 10,000 rpm each in RAID 10

capable of 800 IO operations / second (IOPS)♦ •

standard 1Gb Ethernet NIC, optional 2nd NIC for a management network•

For the purposes of this discussion this will be our single server unit. Note that the only exceptionalitem here is the disk array. Splunk is often constrained by disk I/O first, so always consider that firstwhen selecting your hardware.

Performance Checklist

The first step to deciding on a reference architecture is sizing - can your Splunk handle the load? Forthe purposes of this guide we assume that managing forwarder connections and configurations (butnot their data!) to be free. Therefore we need to look at index volume and search load.

Question 1: Do you need to index more than 2GB per day?

Question 2: Do you need more than 2 concurrent users?

If the answer to both questions is 'NO' then your Splunk instance can safely share one of the aboveservers with other services, with the caveat that Splunk be allowed sufficient disk I/O on the sharedbox. If you answered yes, continue.

62

Question 3: Do you need to index more than 100GB per day?

Question 4: Do you need to have more than 4 concurrent users?

If the answer to both questions is 'NO', then a single dedicated Splunk server of our referencearchitecture should be able to handle your workload.

Question 5: Do you need more than 500GB of storage?

At a high level, total storage is calculated as follows:

daily average rate x retention policy x 1/2

You can generally safely use this simple calculation method. If you want to base your calculation onthe specific type(s) of data that you'll be feeding into Splunk, you can use the method described in"Estimating your storage requirements" in this manual.

Splunk can generally, including indexes, store raw data at approximately half the original size thanksto compression. Given allowances for operating system and disk partitioning, that suggests about500GB of usable space. In practical terms, that's ~6 months of fast storage at 5GB/day, or 10 days at100GB/day.

If you need more storage, you can either opt for more local disks for fast access (required for frequentsearching) or consider attached or network storage (acceptable for occasional searching).Low-latency connections over NFS or CIFS are acceptable for searches over long time periods whereinstant search returns can be compromised to lower cost per GB. Shares mounted over WANconnections and standby storage such as tape are never acceptable.

Beyond 100GB/Day

If you have requirements greater than 100GB/day or 4 concurrent users, you'll want to leverageSplunk's scale-out capabilities. That involves using distributed search to run searches in parallelacross multiple indexers at once, and possibly load balancing the incoming data with auto loadbalanced Splunk forwarders.

Also, at this scale it is very likely that you'll have high availability or redundancy requirements,covered in greater detail below.

Question 6: Do you need more than 300GB/day of daily indexed volume?

If you do not - i.e. you are between 100GB/day and 300GB/day - you should be able to have multipledual-purpose Splunk boxes that are searching across each other.

63

example of a search user searching on one Splunk instance and having their search distributed toother instances

Additional Considerations

you can use a third party load balancer to assign users to different Splunk instances• the recommended best practice for Splunk forwarder management is to use auto loadbalancing

•

you can use Splunk deployment server to propagate Splunk apps and user preferencesbetween instances

•

if you need more than 4 concurrent search users 'per server' this deployment is notappropriate

for example if you have 2 reference servers but need more than 8 search users♦

•

Beyond 300GB/Day

For deployments of 300GB/day or larger, consider a three tier Splunk deployment. In this model,search is separated from index by creating Splunk search heads, or instances of Splunk that only dosearching. That allows for more efficient use of hardware, and to scale search usage (mostly)independently of index volume.

Example Splunk distributed topology. This example could handle up to 400GB/day and 8 concurrentsearch users for common use cases.

Dividing Up Indexing and Searching

At daily volumes above 300GB/day, it makes sense to slightly modify our reference hardware toreflect the differing needs of indexers and search heads. Search heads do not need disk I/O, normuch local storage. However they are far more CPU bound than indexers. Therefore we can changeour recommendations to:

Search Head

64

Intel 64-bit chip architecture• Standard Linux or Windows 64-bit distribution• 4 CPU, 4 core per CPU, 2.5-3Ghz per core• 4GB RAM• 2 300GB SAS hard disks at 10,000 rpm each in RAID 0• standard 1Gb Ethernet NIC, optional 2nd NIC for a management network•

Given that a search head will be CPU bound, if fewer, more performant servers are desired, addingmore and faster CPU cores is best.

Note: The guideline of 1 core per active user still applies. Don't forget to account for scheduledsearches in your CPU allowance as well.

Indexer

Intel 64-bit chip architecture• Standard Linux or Windows 64-bit distribution• 2 CPU, 4 core per CPU, 2.5-3Ghz per core• 8GB RAM• 8 300GB SAS hard disks at 10,000 rpm each in RAID 10

capable of 1200 IO operations / second (Iopps)♦ •

standard 1Gb Ethernet NIC, optional 2nd NIC for a management network•

The indexers will be busy both writing new data and servicing the remote requests of search heads.Therefore disk I/O is the primary bottleneck.

At these daily volumes, likely local disk will not provide cost effective storage for the time frames thatspeedy search is desired, suggesting fast attached storage or networked storage. While there are toomany types of storage to be prescriptive, here are guidelines to consider:

indexers do many bulk reads• indexers do many disk seeks•

Therefore...

more disks (specifically, more spindles) are better• total throughput of the entire system is important, but...• disk to controller ratio should be higher, similar to a database•

Ratio of indexers to search heads

Technically, there is no practical Splunk limitation on the number of search heads an indexer cansupport, or the number of indexers a search head can search against. However systems limitationssuggest a ratio of approximately 8 to 1 for most use cases. That is a rough guideline however; if youhave many searchers compared to your total data volume, more search heads make sense, forexample. In general, the best use of a separate search head is to populate summary indexes. Thissearch head will then act like an indexer to the primary search head that users log into.

65

Accommodating many simultaneous searches

A common question for a large deployment is: how do I account for many concurrent users? Let'stake as an example a system that may have at peak times 48 concurrent searches. The short answeris that we can accommodate 48 simultaneous searches on a cluster of indexers and search headswhere each machine has enough RAM to prevent swapping. Assuming that each search takes200MB of RAM per system, that is roughly 10GB additional RAM (beyond indexing requirements).This is because CPU will degrade gracefully with more concurrent jobs but once the working set ofmemory for all processes exceeds the physical RAM, performance drops catastrophically withswapping.

The caveat here is that a search's run time will be longer in proportion to the number of free coreswhen no searches were running. For example, suppose the indexers were doing nothing before thesearches arrived and have 8 cores each. Suppose the first (of identical searches) takes 10s tocomplete. Then the first 8 searches will each take 10s to complete since there is no contention.However, since there are only 8 cores, if there are 48 searches running, each search will take 48/8 =6x longer than if only 1-8 searches were running. So now, every search takes ~1 minute to complete.

This leads to the observation that the most important thing to do here is add indexers. Indexers dothe bulk of the work in search (reading data off disk, decompressing it, extracting knowledge andreporting). If we want to return to the world of 10s searches, we use 6 indexers (one search head isprobably still fine, though it may be appropriate to set aside a search head for summary indexcreation) and searches 1-8 now take 10/6 = 1.6s and with 48 searches, each takes 10s.

Unfortunately, the system isn't typically idle before searches arrive. If we are indexing 150 GB/day, atpeak times, we probably are using 4 of the 8 cores doing indexing. That means that the first 4searches take 10s, and having 48 searches running takes 48/4 = 12x longer, or 2 min to completeeach.

Now one might say: let me put sixteen cores per indexer rather than eight and avoid buying somemachines. That makes a little bit of sense, but is not the best choice. The number of cores doesn'thelp searches 1-16 in this case; they still take 10s. With 48 searches, each search will take 48/16 =3x longer, which is indeed better than 6x. However, it's usually not too much more expensive to buytwo 8 core machines, which has advantages: the first few searches will now just take 5s (which is themost common case) and we now have more aggregate I/O capacity (doubling the number of coresdoes nothing for I/O, adding servers does).

The lesson here is to add indexers. Doing so reduces the load on any system from indexing, to freecores for search. Also, since the performance of almost all types of search scale with the number ofindexers, searches will be faster, which mitigates the effect of slowness from resource sharing.Additionally making every search faster, we will often avoid the case of concurrent searches withconcurrent users. In realistic situations, with hundreds of users, each user will run a search every fewminutes, though not at the exact same time as other users. By reducing the search time by a factor of6 (by adding more indexers), the concurrency factor will be reduced (not necessarily by 6x, but bysome meaningful factor). This in turn, lowers the concurrency related I/O and memory contention.

66

Summary of Performance Recommendations

Daily Volume Number of SearchUsers

RecommendedIndexers

Recommended SearchHeads

< 2GB/day < 2 1, shared N/A

2GB/day to100GB/day up to 4 1 N/A

200GB/day up to 8 2 1




1TB/day up to 24 10 2

Note that these are approximate guidelines only. You should feel free to modify based on thediscussion here for your specific use case, and to contact Splunk for more guidance if needed.

High Availability and Data Redundancy

Many Splunk deployments require some form of redundancy, either to protect the data from loss orthe search service from outage - and sometimes both. In general Splunk's solution to this problem isa straightforward matter of data duplication, however we will look at three specific deploymentpossibilities.

Data Duplication

The easiest method of ensuring data will not be lost is to have two original artifacts made by cloningdata coming from Splunk forwarders.

In this approach, the data is duplicated and available instantly, should you need to cut over to thestand-by Splunk instance. Note that while you can simply have one Splunk forward to the next Splunk(as shown here for the offsite location) to save on network usage, there is a risk on hard shutdown ofthe last few events not being sent on. If that is acceptable, the topology can be even simpler.

67

High Availability

The goal of a high availability deployment is both data survivability and service uptime. Toaccommodate this kind of deployment, you need to duplicate both the data and the physical hardwareproviding service, not unlike other web based applications. Also, redundancy needs to be consideredfor all three tiers of service - splunkweb searching, splunkd indexing and forwarding.

In this topology, there are two data complete functional groups. In the picture both groups areservicing search requests to optimize hardware costs; the second infrastructure could be idle toensure neither disruption nor degradation of search services.

Things to note about this topology

search heads can be load-balanced to allow for user redirection should one go down• as long as there is at least one indexer up in a clone group, the dataset remains intact

so long as the surviving indexers can handle the load; it is recommended to stopsearching against a degraded group to ensure it doesn't fall behind

♦ •

Performance Considerations

Splunk has three primary roles - indexer, searcher and forwarder. In many cases a single Splunkinstance may two or all three roles at once. All have their own performance requirements, andbottlenecks.

indexing, while relatively resource inexpensive, is often disk I/O bound• searching can be both CPU and disk I/O bound• forwarding uses very little resources, and is rarely a bottleneck•

As you can see, disk I/O is frequently the limiting factor in Splunk performance, and deserves extraconsideration in your planning. That also makes Splunk a poor virtualization candidate unlessdedicated disk access can be arranged.

CPU

allow 1 CPU core for every 1MB/s of indexing volume• allow 1 CPU core for Splunk's optimization routines for every 2MB/s of indexing volume• allow 1 CPU per active searcher (be sure to account for scheduled searches)•

68

Disk I/O

assume 50 Iopps per 1 MB/s of indexing volume• allow 50 Iopps for splunk's optimize routines• allow 100 Iopps per search, or an average of 200 Iopps per search user•

Memory

allow 200-300MB for indexing• allow 500MB per concurrent search user• allow 1GB for the operating system to accommodate OS caching•

Total Storage

allow 15% overhead for OS and disk partitioningon this system there is ~500GB of usable storage♦

•

conservatively Splunk can, including indexes, compress original logs by ~50%compression rates vary based on the data♦

•

Based on these estimate, this machine will be disk IO bound' if there are too many active users ortoo many searches per user. That is the most likely limitation for this hardware, possibly followed byCPU if the searches are highly computational in nature, such as many uses of stats or evalcommands in a single search.

Applied Performance

With the information above, it is possible to estimate required hardware for most Splunk use cases byconsidering the following:

the amount of daily indexed volume (disk I/O, CPU)• the required retention period (total storage)• the number of concurrent search users (disk I/O, CPU)•

Note that not all search users consume the same amount of resources. While their are in depthguides for search cost analysis available here, consider these very rough guidelines.

dashboard-heavy users trigger many searches at once• dashboards also suggest many scheduled searches• searching for rare events across large datasets (e.g all time) is disk I/O intensive• calculating summary information is CPU intensive

if done over long time intervals, can also be disk I/O intensive♦ •

alerts and scheduled searches run even if no one sees their results•

What does that mean in real life?

executive users with many dashboards and summaries require both CPU and disk I/O• operations users searching over recent and small datasets require less resources• forensic and compliance users searching over long timeframes require disk I/O• alerting and scheduled searches over short timeframes are inexpensive; over long timeframespotentially very expensive.

•

69

Reference

PGP Public KeyPGP Public Key

-----BEGIN PGP PUBLIC KEY BLOCK-----Version: GnuPG v1.4.1 (GNU/Linux)

mQGiBEbE21QRBADEMonUxCV2kQ2oxsJTjYXrYCWCtH5/OnmhK5lT2TQaE9QUTs+wnM3sVInQqwRwBDH2qsHgqjJS0PIE867n+lVuk0gSVzS5SOlYzQjnSrisvyN452MF2PgetHq8Lb884cPJnxR6xoFTHqOQueKEOXCovz1eVrjrjfpnmWKa/+5X8wCg/CJ7pT7OXHFN4XOseVQabetEbWcEAIUaazF2i2x9QDJ+6twTAlX2oqAquqtBzJX5qaHnOyRdBEU2g4ndiE3QAKybuq5f0UM7GXqdllihVUBatqafySfjlTBaMVzd4ttrDRpqWya4ppPMIWcnFG2CXf4+HuyTPgj2cry2oMBm2LMfGhxcqM5mpoyHqUiCn7591Ra/J2/FA/0c2UAUh/eSiOn89I6FhFOicT5RPtRpxMoEM1Di15zJ7EXY+xBVF9rutqhR5OI9kdHibYTwf4qjOOPOA7237N1by9GiXY/8s+rDWmSNKZB+xAaLyl7cDhYMv7CPqFTutvE8BxTsF0MgRuzIHfJQE2quuxKJFs9lkSFGuZhvRuwRcrQgS2ltIFdhbGxhY2UgPHJlbGVhc2VAc3BsdW5rLmNvbT6IXgQTEQIAHgUCRsTbVAIbAwYLCQgHAwIDFQIDAxYCAQIeAQIXgAAKCRApYLH9ZT+xEhsPAKDimP8sdCr2ecPm8mre/8TK3BhapQCg3/xEickiRKKlpKnySUNLR/ZBh3m5Ag0ERsTbbRAIAIdfWiOBeCj8BqrcTXxm6MMvdEkjdJCr4xmwaQpYmS4JKK/hJFfpyS8XUgHjBz/7zfR8Ipr2CU59Fy4vb5oUHeOecK9ag5JFdG2i/VWH/vEJAMCkbN/6aWwhHt992PUZC7EHQ5ufRdxGGap8SPZTiIKY0OrX6Km6usoVWMTYKNm/v7my8dJ2F46YJ7wIBF7arG/voMOg1Cbn7pCwCAtgjOhgjdPXRJUEzZP3AfLIc3t5iq5n5FYLGAOpT7OIroM5AkgbVLfj+cjKaGD5UZW7SO0akWhTbVHSCDJoZAGJrvJs5DHcEnCjVy9AJxTNMs9GOwWaixfyQ7jgMNWKHJp+EyMAAwYH/RLNK0HHVSByPWnS2t5sXedIGAgm0fTHhVUCWQxN3knDIRMdkqDTnDKdqcqYFsEljazI2kx1ZlWdUGmvU+Zb8FCH90ej8O6jdFLKJaq50/I/oY0+/+DRBZJG3oKu/CK2NH2VnK1KLzAYnd2wZQAEja4O1CBV0hgutVf/ZxzDUAr/XqPHy5+EYg964Xz0PdZiZKOhJ5g4QjhhOL3jQwcBuyFbJADw8+Tsk8RJqZvHfuwPouVU+8F2vLJKiF2HbKOUJvdH5GfFuk6o5V8nnir7xSrVj4abfP4xA6RVum3HtWoD7t//75gLcW77kXDR8pmmnddm5VXnAuk+GTPGACj98+eISQQYEQIACQUCRsTbbQIbDAAKCRApYLH9ZT+xEiVuAJ9INUCilkgXSNu9p27zxTZh1kL04QCg6YfWldq/MWPCwa1PgiHrVJngp4s==Mz6T-----END PGP PUBLIC KEY BLOCK-----

Installing the key

Copy and paste the key into a file. Install the key using:

rpm --import <filename>

File manifestFile manifest

A complete inventory of the files and permissions that ship with your Splunk installation can be foundin the root directory of your Splunk installation. The file will end with -manifest.

70

Splunk 4.0.9 Installation Guide

Documents