Migrating Splunk indexed data

First stop splunk.

cd into your splunk/bin directory
./splunk stop

Create a new folder (ex. /mnt/splunk_data).

cp -rp splunk/var/lib/splunk/* /mnt/splunk_data/

Change SPLUNK_DB to point to /mnt/splunk_data.

vi splunk/etc/splunk-launch.conf

Find SPLUNK_DB in the file and change the path.

SPLUNK_DB=/mnt1/splunk_data

You may also want to change the retention policy and the max storage size.

// 30 days
frozenTimePeriodInSecs = 2592000

// 90G
maxTotalDataSizeMB = 90000

It's recommended to set the size using the following formula:

Total storage = daily average rate x retention policy x 1/2 = 15 Gig

Start Splunk.

./splunk start

To tune Splunk settings, check:
http://docs.splunk.com/Documentation/Splunk/4.3.1/Installation/CapacityplanningforalargerSplunkdeployment

Install Splunk Forwarding and Receiving

We will be using Splunk Light.

Click on the menu icon at the upper right corner. Choose Data -> Receiving.

In Configure receiving, choose 9997 as the receiving port.

In your application instance, install the universal splunk forwarder.

http://www.splunk.com/en_us/download/universal-forwarder.html

Extract it and put it in /opt/splunk_forwarder directory

sudo ./splunk start
sudo ./splunk enable boot-start -user ec2-user

List all the forward servers:
./splunk list forward-server

Active forwards:
None
Configured but inactive forwards:
None

If it prompts you for username and password, use
username: admin
password: changeme

Add the receiving server to the forwarder:

./splunk add forward-server :9997

Test the connection:
./splunk list forward-server

Active forwards:
None
Configured but inactive forwards:
:9997

If it's not active, remember to add port 9997 to your security group.

Add data to monitor

./splunk add monitor -index main -sourcetype

To list what's being monitored:

./splunk list monitor

Installing splunk on AWS

Begin by downloading Splunk Light here: http://www.splunk.com/en_us/download.html. You will probably need to register an account on Splunk before it lets you to download it.

Upload Splunk to your ec2 instance using SCP. For example

scp -i ec2-user@:tmp

In above, I uploaded the splunk tgz file to a tmp folder in my ec2 instance.

You will need to install glibc.i686 first.

yum -y install glibc.i686

Create a folder called /opt if it doesn't exist

Extract your tgz file inside opt

tar xvzf splunklight-6.2.4-271043-Linux-i686.tgz

The splunk executable is located in /opt/splunk/bin. cd into it.

Start splunk:

sudo ./splunk start --accept-license

Start splunk on boot:

sudo ./splunk enable boot-start -user ec2-user

You should be able to view splunk's web interface at port 8000 or your ec2 public address.

Other useful commands:

./splunk stop
./splunk restart

show user cronjobs in ubuntu

Show all the users and their respective cronjobs

for user in $(cut -f1 -d: /etc/passwd); do echo $user; crontab -u $user -l; done

boot2docker cannot cd into a directory

Let's say you are running your server using:

docker-compose up

You may be trying to run bash for your container,

docker ps (grab the container id)
docker exec -it 301 bash

When you cd into a mounted host volume, if you get a "killed" message or it just logs you out, try the following:

boot2docker restart

docker - error fetching ubuntu packages

If you ever see the following error and you are using boot2docker, run "boot2docker restart"

E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/libe/libevent/libevent-2.0-5_2.0.21-stable-1ubuntu1.14.04.1_amd64.deb  Could not resolve 'archive.ubuntu.com'

E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/m/memcached/memcached_1.4.14-0ubuntu9_amd64.deb  Could not resolve 'archive.ubuntu.com'

E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?

Rudix The easiest way to install unix software for OSX

Install Rudix

> curl -s https://raw.githubusercontent.com/rudix-mac/rpm/2015.5/rudix.py | sudo python - install rudix

To install any packages, for example, erlang,

> sudo rudix install erlang