First stop splunk.
cd into your splunk/bin directory
./splunk stop
Create a new folder (ex. /mnt/splunk_data).
cp -rp splunk/var/lib/splunk/* /mnt/splunk_data/
Change SPLUNK_DB to point to /mnt/splunk_data.
vi splunk/etc/splunk-launch.conf
Find SPLUNK_DB in the file and change the path.
SPLUNK_DB=/mnt1/splunk_data
You may also want to change the retention policy and the max storage size.
// 30 days
frozenTimePeriodInSecs = 2592000
// 90G
maxTotalDataSizeMB = 90000
It's recommended to set the size using the following formula:
Total storage = daily average rate x retention policy x 1/2 = 15 Gig
Start Splunk.
./splunk start
To tune Splunk settings, check:
http://docs.splunk.com/Documentation/Splunk/4.3.1/Installation/CapacityplanningforalargerSplunkdeployment
Friday, July 24, 2015
Thursday, July 23, 2015
Install Splunk Forwarding and Receiving
We will be using Splunk Light.
Click on the menu icon at the upper right corner. Choose Data -> Receiving.
In Configure receiving, choose 9997 as the receiving port.
In your application instance, install the universal splunk forwarder.
http://www.splunk.com/en_us/download/universal-forwarder.html
Extract it and put it in /opt/splunk_forwarder directory
sudo ./splunk start
sudo ./splunk enable boot-start -user ec2-user
List all the forward servers:
./splunk list forward-server
Active forwards:
None
Configured but inactive forwards:
None
If it prompts you for username and password, use
username: admin
password: changeme
Add the receiving server to the forwarder:
./splunk add forward-server:9997
Test the connection:
./splunk list forward-server
Active forwards:
None
Configured but inactive forwards:
:9997
If it's not active, remember to add port 9997 to your security group.
Add data to monitor
./splunk add monitor -index main -sourcetype
To list what's being monitored:
./splunk list monitor
Click on the menu icon at the upper right corner. Choose Data -> Receiving.
In Configure receiving, choose 9997 as the receiving port.
In your application instance, install the universal splunk forwarder.
http://www.splunk.com/en_us/download/universal-forwarder.html
Extract it and put it in /opt/splunk_forwarder directory
sudo ./splunk start
sudo ./splunk enable boot-start -user ec2-user
List all the forward servers:
./splunk list forward-server
Active forwards:
None
Configured but inactive forwards:
None
If it prompts you for username and password, use
username: admin
password: changeme
Add the receiving server to the forwarder:
./splunk add forward-server
Test the connection:
./splunk list forward-server
Active forwards:
None
Configured but inactive forwards:
If it's not active, remember to add port 9997 to your security group.
Add data to monitor
./splunk add monitor
To list what's being monitored:
./splunk list monitor
Installing splunk on AWS
Begin by downloading Splunk Light here: http://www.splunk.com/en_us/download.html. You will probably need to register an account on Splunk before it lets you to download it.
Upload Splunk to your ec2 instance using SCP. For example
scp -i ec2-user@:tmp
In above, I uploaded the splunk tgz file to a tmp folder in my ec2 instance.
You will need to install glibc.i686 first.
yum -y install glibc.i686
Create a folder called /opt if it doesn't exist
Extract your tgz file inside opt
tar xvzf splunklight-6.2.4-271043-Linux-i686.tgz
The splunk executable is located in /opt/splunk/bin. cd into it.
Start splunk:
sudo ./splunk start --accept-license
Start splunk on boot:
sudo ./splunk enable boot-start -user ec2-user
You should be able to view splunk's web interface at port 8000 or your ec2 public address.
Other useful commands:
./splunk stop
./splunk restart
Upload Splunk to your ec2 instance using SCP. For example
scp -i
In above, I uploaded the splunk tgz file to a tmp folder in my ec2 instance.
You will need to install glibc.i686 first.
yum -y install glibc.i686
Create a folder called /opt if it doesn't exist
Extract your tgz file inside opt
tar xvzf splunklight-6.2.4-271043-Linux-i686.tgz
The splunk executable is located in /opt/splunk/bin. cd into it.
Start splunk:
sudo ./splunk start --accept-license
Start splunk on boot:
sudo ./splunk enable boot-start -user ec2-user
You should be able to view splunk's web interface at port 8000 or your ec2 public address.
Other useful commands:
./splunk stop
./splunk restart
Wednesday, July 8, 2015
show user cronjobs in ubuntu
Show all the users and their respective cronjobs
for user in $(cut -f1 -d: /etc/passwd); do echo $user; crontab -u $user -l; done
for user in $(cut -f1 -d: /etc/passwd); do echo $user; crontab -u $user -l; done
Sunday, July 5, 2015
boot2docker cannot cd into a directory
Let's say you are running your server using:
docker-compose up
You may be trying to run bash for your container,
docker ps (grab the container id)
docker exec -it 301 bash
When you cd into a mounted host volume, if you get a "killed" message or it just logs you out, try the following:
boot2docker restart
docker-compose up
You may be trying to run bash for your container,
docker ps (grab the container id)
docker exec -it 301 bash
When you cd into a mounted host volume, if you get a "killed" message or it just logs you out, try the following:
boot2docker restart
docker - error fetching ubuntu packages
If you ever see the following error and you are using boot2docker, run "boot2docker restart"
E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/libe/libevent/libevent-2.0-5_2.0.21-stable-1ubuntu1.14.04.1_amd64.deb Could not resolve 'archive.ubuntu.com'
E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/m/memcached/memcached_1.4.14-0ubuntu9_amd64.deb Could not resolve 'archive.ubuntu.com'
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/libe/libevent/libevent-2.0-5_2.0.21-stable-1ubuntu1.14.04.1_amd64.deb Could not resolve 'archive.ubuntu.com'
E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/m/memcached/memcached_1.4.14-0ubuntu9_amd64.deb Could not resolve 'archive.ubuntu.com'
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
Saturday, July 4, 2015
Rudix The easiest way to install unix software for OSX
Install Rudix
> curl -s https://raw.githubusercontent.com/rudix-mac/rpm/2015.5/rudix.py | sudo python - install rudix
To install any packages, for example, erlang,
> sudo rudix install erlang
> curl -s https://raw.githubusercontent.com/rudix-mac/rpm/2015.5/rudix.py | sudo python - install rudix
To install any packages, for example, erlang,
> sudo rudix install erlang
boot startup scripts with chkconfig
All the startup scripts in ubuntu is located in /etc/init.d. You can turn these scripts on or off by using chkconfig.
> chkconfig
To turn a script on at level 2, 3, 5
> chkconfig memcached on --level 235
To turn off a script
> chkconfig memcached off
EBS expand file system to recognize volume size in Ubuntu
A common scenario when dealing using EC2 is expanding EBS/volume sizes. You may be doing a new AMI or just expanding an existing volume. This article is about how to make your file system (ex. xfs, ext) to recognize the size of your new volumes.
After you expand your volume, ssh into the instance.
Show the instance's volumes and their sizes.
> sudo lsblk -o NAME,FSTYPE,SIZE,MOUNTPOINT,LABEL
NAME FSTYPE SIZE MOUNTPOINT LABEL
xvda1 ext4 128G / /
xvdb ext3 840G /media/ephemeral0
xvdm linux_raid_member 10G ip-10-188-5-211:0
└─md127 xfs 60G /mnt/data
xvdn linux_raid_member 10G ip-10-188-5-211:0
└─md127 xfs 60G /mnt/data
xvdo linux_raid_member 10G ip-10-188-5-211:0
└─md127 xfs 60G /mnt/data
xvdl linux_raid_member 10G ip-10-188-5-211:0
└─md127 xfs 60G /mnt/data
xvdj linux_raid_member 10G ip-10-188-5-211:0
└─md127 xfs 60G /mnt/data
xvdk linux_raid_member 10G ip-10-188-5-211:0
└─md127 xfs 60G /mnt/data
xvdf ext4 30G /mnt/shared
In the example above, we see /dev/xvda1 has 128G and is using file system ext4.
If you want more details on the file system types of each volume, you can use the file command:
> sudo file -s /dev/xvd*
/dev/xvda1: Linux rev 1.0 ext4 filesystem data, UUID=ebbf1b1c-fb71-40aa-93a3-056b455e5127 (needs journal recovery) (extents) (large files) (huge files)
/dev/xvdb: Linux rev 1.0 ext3 filesystem data, UUID=07b9bb55-97cc-47e8-b968-6f158e66ff60 (needs journal recovery) (large files)
/dev/xvdf: Linux rev 1.0 ext4 filesystem data, UUID=bff77q92-806c-44a5-a260-5a50025283ba (needs journal recovery) (extents) (large files) (huge files)
/dev/xvdj: data
/dev/xvdk: data
/dev/xvdl: data
/dev/xvdm: data
/dev/xvdn: data
/dev/xvdo: data
> lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
xvda1 202:1 0 128G 0 disk /
xvdb 202:16 0 840G 0 disk /media/ephemeral0
xvdm 202:192 0 10G 0 disk
└─md127 9:127 0 60G 0 raid0 /mnt/data
xvdn 202:208 0 10G 0 disk
└─md127 9:127 0 60G 0 raid0 /mnt/data
xvdo 202:224 0 10G 0 disk
└─md127 9:127 0 60G 0 raid0 /mnt/data
xvdl 202:176 0 10G 0 disk
└─md127 9:127 0 60G 0 raid0 /mnt/data
xvdj 202:144 0 10G 0 disk
└─md127 9:127 0 60G 0 raid0 /mnt/data
xvdk 202:160 0 10G 0 disk
└─md127 9:127 0 60G 0 raid0 /mnt/data
xvdf 202:80 0 30G 0 disk /mnt/shared
> df -h
Filesystem Size Used Avail Use% Mounted on
/dev/xvda1 7.9G 4.0G 3.9G 52% /
tmpfs 17G 0 17G 0% /dev/shm
/dev/xvdb 827G 201M 785G 1% /media/ephemeral0
/dev/xvdf 30G 8.0G 21G 29% /mnt/shared
/dev/md127 60G 15G 46G 25% /mnt/data
For ext2, ext3, ext4, you can use the resize2fs command.
Resize /dev/xvda1
> sudo resize2fs /dev/xvda1
For xfs, you can do
> sudo xfs_growfs -d /mnt
After you expand your volume, ssh into the instance.
Show the instance's volumes and their sizes.
> sudo lsblk -o NAME,FSTYPE,SIZE,MOUNTPOINT,LABEL
NAME FSTYPE SIZE MOUNTPOINT LABEL
xvda1 ext4 128G / /
xvdb ext3 840G /media/ephemeral0
xvdm linux_raid_member 10G ip-10-188-5-211:0
└─md127 xfs 60G /mnt/data
xvdn linux_raid_member 10G ip-10-188-5-211:0
└─md127 xfs 60G /mnt/data
xvdo linux_raid_member 10G ip-10-188-5-211:0
└─md127 xfs 60G /mnt/data
xvdl linux_raid_member 10G ip-10-188-5-211:0
└─md127 xfs 60G /mnt/data
xvdj linux_raid_member 10G ip-10-188-5-211:0
└─md127 xfs 60G /mnt/data
xvdk linux_raid_member 10G ip-10-188-5-211:0
└─md127 xfs 60G /mnt/data
xvdf ext4 30G /mnt/shared
In the example above, we see /dev/xvda1 has 128G and is using file system ext4.
If you want more details on the file system types of each volume, you can use the file command:
> sudo file -s /dev/xvd*
/dev/xvda1: Linux rev 1.0 ext4 filesystem data, UUID=ebbf1b1c-fb71-40aa-93a3-056b455e5127 (needs journal recovery) (extents) (large files) (huge files)
/dev/xvdb: Linux rev 1.0 ext3 filesystem data, UUID=07b9bb55-97cc-47e8-b968-6f158e66ff60 (needs journal recovery) (large files)
/dev/xvdf: Linux rev 1.0 ext4 filesystem data, UUID=bff77q92-806c-44a5-a260-5a50025283ba (needs journal recovery) (extents) (large files) (huge files)
/dev/xvdj: data
/dev/xvdk: data
/dev/xvdl: data
/dev/xvdm: data
/dev/xvdn: data
/dev/xvdo: data
> lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
xvda1 202:1 0 128G 0 disk /
xvdb 202:16 0 840G 0 disk /media/ephemeral0
xvdm 202:192 0 10G 0 disk
└─md127 9:127 0 60G 0 raid0 /mnt/data
xvdn 202:208 0 10G 0 disk
└─md127 9:127 0 60G 0 raid0 /mnt/data
xvdo 202:224 0 10G 0 disk
└─md127 9:127 0 60G 0 raid0 /mnt/data
xvdl 202:176 0 10G 0 disk
└─md127 9:127 0 60G 0 raid0 /mnt/data
xvdj 202:144 0 10G 0 disk
└─md127 9:127 0 60G 0 raid0 /mnt/data
xvdk 202:160 0 10G 0 disk
└─md127 9:127 0 60G 0 raid0 /mnt/data
xvdf 202:80 0 30G 0 disk /mnt/shared
> df -h
Filesystem Size Used Avail Use% Mounted on
/dev/xvda1 7.9G 4.0G 3.9G 52% /
tmpfs 17G 0 17G 0% /dev/shm
/dev/xvdb 827G 201M 785G 1% /media/ephemeral0
/dev/xvdf 30G 8.0G 21G 29% /mnt/shared
/dev/md127 60G 15G 46G 25% /mnt/data
For ext2, ext3, ext4, you can use the resize2fs command.
Resize /dev/xvda1
> sudo resize2fs /dev/xvda1
For xfs, you can do
> sudo xfs_growfs -d /mnt
Friday, July 3, 2015
Using Zend opcache with php-fpm
Install Zend OPcache
> yum install php55-opcache
Check if the module exists:
> php -m | grep cache
Add the following to your php.ini
opcache.enable=1
opcache.memory_consumption=128
opcache.max_accelerated_files=4000
opcache.revalidate_freq=60
Check if opcache is enabled by
> php-fpm -i | grep cache
The opcache settings may be located in /etc/php-5.5.d/opcache.ini
use the following to set opcache.max_accelerated_files:
find . -type f -print | grep php | wc -l
If the number of php files is 2000, you may want to set it to some number slightly larger than that.
Thursday, July 2, 2015
Upgrading php5.4 to php5.5 in Amazon EC2
First stop apache, nginx, php-fpm if you are running them.
List all the php 5.4 modules:
> yum list installed | grep php54
php54.x86_64 5.4.21-1.46.amzn1 @amzn-updates
php54-bcmath.x86_64 5.4.21-1.46.amzn1 @amzn-updates
php54-cli.x86_64 5.4.21-1.46.amzn1 @amzn-updates
php54-common.x86_64 5.4.21-1.46.amzn1 @amzn-updates
php54-devel.x86_64 5.4.21-1.46.amzn1 @amzn-updates
php54-fpm.x86_64 5.4.21-1.46.amzn1 @amzn-updates
php54-gd.x86_64 5.4.21-1.46.amzn1 @amzn-updates
php54-intl.x86_64 5.4.21-1.46.amzn1 @amzn-updates
php54-mbstring.x86_64 5.4.21-1.46.amzn1 @amzn-updates
php54-mcrypt.x86_64 5.4.21-1.46.amzn1 @amzn-updates
php54-mysqlnd.x86_64 5.4.21-1.46.amzn1 @amzn-updates
php54-pdo.x86_64 5.4.21-1.46.amzn1 @amzn-updates
php54-pecl-apc.x86_64 3.1.13-1.12.amzn1 @amzn-updates
php54-pecl-igbinary.x86_64 1.1.2-0.2.git3b8ab7e.6.amzn1 @amzn-updates
php54-pecl-memcache.x86_64 3.0.7-3.10.amzn1 @amzn-updates
php54-pecl-memcached.x86_64 2.1.0-1.5.amzn1 @amzn-updates
php54-pecl-xdebug.x86_64 2.2.1-1.6.amzn1 @amzn-updates
php54-process.x86_64 5.4.21-1.46.amzn1 @amzn-updates
php54-soap.x86_64 5.4.21-1.46.amzn1 @amzn-updates
php54-xml.x86_64 5.4.21-1.46.amzn1 @amzn-updates
php54-xmlrpc.x86_64 5.4.21-1.46.amzn1 @amzn-updates
Remove all of them:
yum remove php54.x86_64 php54-bcmath.x86_64 php54-cli.x86_64 php54-common.x86_64 php54-devel.x86_64 php54-fpm.x86_64 php54-gd.x86_64 php54-intl.x86_64 php54-mbstring.x86_64 php54-mcrypt.x86_64 php54-mysqlnd.x86_64 php54-pdo.x86_64 php54-pecl-apc.x86_64 php54-pecl-igbinary.x86_64 php54-pecl-memcache.x86_64 php54-pecl-memcached.x86_64 php54-pecl-xdebug.x86_64 php54-process.x86_64 php54-soap.x86_64 php54-xml.x86_64 php54-xmlrpc.x86_64
Install php 5.5
yum install php55.x86_64 php55-bcmath.x86_64 php55-cli.x86_64 php55-common.x86_64 php55-devel.x86_64 php55-fpm.x86_64 php55-gd.x86_64 php55-intl.x86_64 php55-mbstring.x86_64 php55-mcrypt.x86_64 php55-mysqlnd.x86_64 php55-pdo.x86_64 php55-pecl-apc.x86_64 php55-pecl-igbinary.x86_64 php55-pecl-memcache.x86_64 php55-pecl-memcached.x86_64 php55-pecl-xdebug.x86_64 php55-process.x86_64 php55-soap.x86_64 php55-xml.x86_64 php55-xmlrpc.x86_64
You may need to tweak the php-fpm settings
List all the php 5.4 modules:
> yum list installed | grep php54
php54.x86_64 5.4.21-1.46.amzn1 @amzn-updates
php54-bcmath.x86_64 5.4.21-1.46.amzn1 @amzn-updates
php54-cli.x86_64 5.4.21-1.46.amzn1 @amzn-updates
php54-common.x86_64 5.4.21-1.46.amzn1 @amzn-updates
php54-devel.x86_64 5.4.21-1.46.amzn1 @amzn-updates
php54-fpm.x86_64 5.4.21-1.46.amzn1 @amzn-updates
php54-gd.x86_64 5.4.21-1.46.amzn1 @amzn-updates
php54-intl.x86_64 5.4.21-1.46.amzn1 @amzn-updates
php54-mbstring.x86_64 5.4.21-1.46.amzn1 @amzn-updates
php54-mcrypt.x86_64 5.4.21-1.46.amzn1 @amzn-updates
php54-mysqlnd.x86_64 5.4.21-1.46.amzn1 @amzn-updates
php54-pdo.x86_64 5.4.21-1.46.amzn1 @amzn-updates
php54-pecl-apc.x86_64 3.1.13-1.12.amzn1 @amzn-updates
php54-pecl-igbinary.x86_64 1.1.2-0.2.git3b8ab7e.6.amzn1 @amzn-updates
php54-pecl-memcache.x86_64 3.0.7-3.10.amzn1 @amzn-updates
php54-pecl-memcached.x86_64 2.1.0-1.5.amzn1 @amzn-updates
php54-pecl-xdebug.x86_64 2.2.1-1.6.amzn1 @amzn-updates
php54-process.x86_64 5.4.21-1.46.amzn1 @amzn-updates
php54-soap.x86_64 5.4.21-1.46.amzn1 @amzn-updates
php54-xml.x86_64 5.4.21-1.46.amzn1 @amzn-updates
php54-xmlrpc.x86_64 5.4.21-1.46.amzn1 @amzn-updates
Remove all of them:
yum remove php54.x86_64 php54-bcmath.x86_64 php54-cli.x86_64 php54-common.x86_64 php54-devel.x86_64 php54-fpm.x86_64 php54-gd.x86_64 php54-intl.x86_64 php54-mbstring.x86_64 php54-mcrypt.x86_64 php54-mysqlnd.x86_64 php54-pdo.x86_64 php54-pecl-apc.x86_64 php54-pecl-igbinary.x86_64 php54-pecl-memcache.x86_64 php54-pecl-memcached.x86_64 php54-pecl-xdebug.x86_64 php54-process.x86_64 php54-soap.x86_64 php54-xml.x86_64 php54-xmlrpc.x86_64
Install php 5.5
yum install php55.x86_64 php55-bcmath.x86_64 php55-cli.x86_64 php55-common.x86_64 php55-devel.x86_64 php55-fpm.x86_64 php55-gd.x86_64 php55-intl.x86_64 php55-mbstring.x86_64 php55-mcrypt.x86_64 php55-mysqlnd.x86_64 php55-pdo.x86_64 php55-pecl-apc.x86_64 php55-pecl-igbinary.x86_64 php55-pecl-memcache.x86_64 php55-pecl-memcached.x86_64 php55-pecl-xdebug.x86_64 php55-process.x86_64 php55-soap.x86_64 php55-xml.x86_64 php55-xmlrpc.x86_64
You may need to tweak the php-fpm settings
Wednesday, July 1, 2015
Configure symfony to see log errors to swiftmailer
If you want 400x and 500x errors, use action_level error, else use action_level critical
monolog:
handlers:
main:
type: fingers_crossed
action_level: error
buffer_size: 200
handler: nested
grouped:
type: group
members: [streamed, buffered]
streamed:
type: stream
path: %log_dir%/moonlight_%kernel.environment%.log
level: debug
buffered:
type: buffer
buffer_size: 200
handler: swift
swift:
type: swift_mailer
from_email:
to_email:
subject: Crtical Error Alert
level: debug
monolog:
handlers:
main:
type: fingers_crossed
action_level: error
buffer_size: 200
handler: nested
grouped:
type: group
members: [streamed, buffered]
streamed:
type: stream
path: %log_dir%/moonlight_%kernel.environment%.log
level: debug
buffered:
type: buffer
buffer_size: 200
handler: swift
swift:
type: swift_mailer
from_email:
to_email:
subject: Crtical Error Alert
level: debug
Elastic beanstalk docker - map symfony logs to S3
In config.yml
monolog:
handlers:
main:
type: fingers_crossed
action_level: error
buffer_size: 200
handler: nested
nested:
type: stream
path: %log_dir%/moonlight_%kernel.environment%.log
level: debug
monolog:
handlers:
main:
type: fingers_crossed
action_level: error
buffer_size: 200
handler: nested
nested:
type: stream
path: %log_dir%/moonlight_%kernel.environment%.log
level: debug
Make log_dir in parameter.yml to be /var/log/nginx or anywhere you want.
Create a file called Dockerrun.aws.json
{
"AWSEBDockerrunVersion": "1",
"Ports": [
{
"ContainerPort": "80"
}
],
"Logging": "/var/log/nginx"
}
The logging entry above needs to be the same as log_dir you set in parameter.log.
In Elastic Beanstalk settings, click on Configuration on the left side, then software configuration.
Check "Enable log file rotation to Amazon S3. If checked, service logs are published to S3."
If you are using a custom IAM, you will need to grant read and write permissions to S3:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "Stmt1435793320000",
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:GetObjectVersion",
"s3:ListBucket",
"s3:ListBucketVersions"
],
"Resource": [
"arn:aws:s3:::elasticbeanstalk-*/resources/environments/logs/*"
]
}
]
}
Log rotations happen about every 15 mins. You can search the s3 directory elasticbeanstalk-*/resources/environments/logs/* for logs.
If you are using a custom IAM, you will need to grant read and write permissions to S3:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "Stmt1435793320000",
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:GetObjectVersion",
"s3:ListBucket",
"s3:ListBucketVersions"
],
"Resource": [
"arn:aws:s3:::elasticbeanstalk-*/resources/environments/logs/*"
]
}
]
}
Log rotations happen about every 15 mins. You can search the s3 directory elasticbeanstalk-*/resources/environments/logs/* for logs.
Subscribe to:
Posts (Atom)