Migrating Splunk indexed data

First stop splunk.

cd into your splunk/bin directory
./splunk stop

Create a new folder (ex. /mnt/splunk_data).

cp -rp splunk/var/lib/splunk/* /mnt/splunk_data/

Change SPLUNK_DB to point to /mnt/splunk_data.

vi splunk/etc/splunk-launch.conf

Find SPLUNK_DB in the file and change the path.

SPLUNK_DB=/mnt1/splunk_data

You may also want to change the retention policy and the max storage size.

// 30 days
frozenTimePeriodInSecs = 2592000

// 90G
maxTotalDataSizeMB = 90000

It's recommended to set the size using the following formula:

Total storage = daily average rate x retention policy x 1/2 = 15 Gig

Start Splunk.

./splunk start

To tune Splunk settings, check:
http://docs.splunk.com/Documentation/Splunk/4.3.1/Installation/CapacityplanningforalargerSplunkdeployment

Installing splunk on AWS

Begin by downloading Splunk Light here: http://www.splunk.com/en_us/download.html. You will probably need to register an account on Splunk before it lets you to download it.

Upload Splunk to your ec2 instance using SCP. For example

scp -i ec2-user@:tmp

In above, I uploaded the splunk tgz file to a tmp folder in my ec2 instance.

You will need to install glibc.i686 first.

yum -y install glibc.i686

Create a folder called /opt if it doesn't exist

Extract your tgz file inside opt

tar xvzf splunklight-6.2.4-271043-Linux-i686.tgz

The splunk executable is located in /opt/splunk/bin. cd into it.

Start splunk:

sudo ./splunk start --accept-license

Start splunk on boot:

sudo ./splunk enable boot-start -user ec2-user

You should be able to view splunk's web interface at port 8000 or your ec2 public address.

Other useful commands:

./splunk stop
./splunk restart