Cassandra Data Types

Internal Type	CQL Name	Description
BytesType	blob	Arbitrary hexadecimal bytes (no validation)
AsciiType	ascii	US-ASCII character string
UTF8Type	text, varchar	UTF-8 encoded string
IntegerType	varint	Arbitrary-precision integer
LongType	int, bigint	8-byte long
UUIDType	uuid	Type 1 or type 4 UUID
DateType	timestamp	Date plus time, encoded as 8 bytes since epoch
BooleanType	boolean	true or false
FloatType	float	4-byte floating point
DoubleType	double	8-byte floating point
DecimalType	decimal	Variable-precision decimal
CounterColumnType	counter	Distributed counter value (8-byte long)

The table is taken from datastax.

NoSQL comparison

Below is a very good comparison among many NoSQL databases.

http://kkovacs.eu/cassandra-vs-mongodb-vs-couchdb-vs-redis

Cassandra cqlsh - mismatched input ')' expecting EOF

The default CQLsh version is 2, and it doesn't support composite keys.

To switch to CQLsh 3,
> bin/cqlsh --cql3

To execute a script
> bin/cqlsh --cql3 -f {script_name}

Cassandra CLI basic schema commands

Launch the Cassandra CLI

> bin/cassandra-cli

Show all keyspaces:

> show keyspaces;

select a keyspace:

> use {keyspace}

show all column families of a keyspace:

> describe {keyspace}

Git - how to diff files with different eol

git diff --ignore-space-at-eol

Resources for Learning Cassandra

Setting up Cassandra

http://wiki.apache.org/cassandra/GettingStarted

The Twissandra Project

http://www.rackspace.com/blog/cassandra-by-example/?2fa943c8
http://www.datastax.com/docs/0.6/data_model/twissandra
https://github.com/twissandra/twissandra

eBay Cassandra Tips

http://www.ebaytechblog.com/2012/07/16/cassandra-data-modeling-best-practices-part-1/
http://www.ebaytechblog.com/2012/08/14/cassandra-data-modeling-best-practices-part-2/

Not connected to a cassandra instance.

Use netstat to check if Cassandra is running on port 9160.
> netstat -tulpn

Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 127.0.0.1:8005          0.0.0.0:*               LISTEN      20358/java    
tcp        0      0 127.0.0.1:9160          0.0.0.0:*               LISTEN      4951/java      
tcp        0      0 127.0.0.1:3306          0.0.0.0:*               LISTEN      18848/mysqld  
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      20358/java    
tcp        0      0 0.0.0.0:42325           0.0.0.0:*               LISTEN      4951/java      
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      619/sshd      
tcp        0      0 127.0.0.1:7000          0.0.0.0:*               LISTEN      4951/java      
tcp        0      0 0.0.0.0:7199            0.0.0.0:*               LISTEN      4951/java      
tcp        0      0 0.0.0.0:53890           0.0.0.0:*               LISTEN      4951/java      
tcp6       0      0 :::22                   :::*                    LISTEN      619/sshd      
udp        0      0 0.0.0.0:68              0.0.0.0:*                           431/dhclient3

Make sure port 9160 is running.

If not, start Cassandra:
> sudo cassandra -f

Connect to Cassandra
> sudo cassandra-cli --host localhost

Keyspace: system:
  Replication Strategy: org.apache.cassandra.locator.LocalStrategy
  Durable Writes: true
    Options: [replication_factor:1]
  Column Families:
    ColumnFamily: HintsColumnFamily (Super)
    "hinted handoff data"
      Key Validation Class: org.apache.cassandra.db.marshal.BytesType
      Default column value validator: org.apache.cassandra.db.marshal.BytesType
      Columns sorted by: org.apache.cassandra.db.marshal.BytesType/org.apache.cassandra.db.marshal.BytesType
      Row cache size / save period in seconds / keys to save : 0.0/0/all
      Row Cache Provider: org.apache.cassandra.cache.ConcurrentLinkedHashCacheProvider
      Key cache size / save period in seconds: 0.01/0
      GC grace seconds: 0
      Compaction min/max thresholds: 4/32
      Read repair chance: 0.0
      Replicate on write: true
      Bloom Filter FP chance: default
      Built indexes: []
      Compaction Strategy: org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy
    ColumnFamily: IndexInfo
    "indexes that have been completed"
      Key Validation Class: org.apache.cassandra.db.marshal.BytesType
      Default column value validator: org.apache.cassandra.db.marshal.BytesType
      Columns sorted by: org.apache.cassandra.db.marshal.UTF8Type
      Row cache size / save period in seconds / keys to save : 0.0/0/all
      Row Cache Provider: org.apache.cassandra.cache.ConcurrentLinkedHashCacheProvider
      Key cache size / save period in seconds: 0.01/0
      GC grace seconds: 0
      Compaction min/max thresholds: 4/32
      Read repair chance: 0.0
      Replicate on write: true
      Bloom Filter FP chance: default
      Built indexes: []
      Compaction Strategy: org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy
    ColumnFamily: LocationInfo
    "persistent metadata for the local node"
      Key Validation Class: org.apache.cassandra.db.marshal.BytesType
      Default column value validator: org.apache.cassandra.db.marshal.BytesType
      Columns sorted by: org.apache.cassandra.db.marshal.BytesType
      Row cache size / save period in seconds / keys to save : 0.0/0/all
      Row Cache Provider: org.apache.cassandra.cache.ConcurrentLinkedHashCacheProvider
      Key cache size / save period in seconds: 0.01/0
      GC grace seconds: 0
      Compaction min/max thresholds: 4/32
      Read repair chance: 0.0
      Replicate on write: true
      Bloom Filter FP chance: default
      Built indexes: []
      Compaction Strategy: org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy
    ColumnFamily: Migrations
    "individual schema mutations"
      Key Validation Class: org.apache.cassandra.db.marshal.BytesType
      Default column value validator: org.apache.cassandra.db.marshal.BytesType
      Columns sorted by: org.apache.cassandra.db.marshal.TimeUUIDType
      Row cache size / save period in seconds / keys to save : 0.0/0/all
      Row Cache Provider: org.apache.cassandra.cache.ConcurrentLinkedHashCacheProvider
      Key cache size / save period in seconds: 0.01/0
      GC grace seconds: 0
      Compaction min/max thresholds: 4/32
      Read repair chance: 0.0
      Replicate on write: true
      Bloom Filter FP chance: default
      Built indexes: []
      Compaction Strategy: org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy
    ColumnFamily: NodeIdInfo
    "nodeId and their metadata"
      Key Validation Class: org.apache.cassandra.db.marshal.BytesType
      Default column value validator: org.apache.cassandra.db.marshal.BytesType
      Columns sorted by: org.apache.cassandra.db.marshal.TimeUUIDType
      Row cache size / save period in seconds / keys to save : 0.0/0/all
      Row Cache Provider: org.apache.cassandra.cache.ConcurrentLinkedHashCacheProvider
      Key cache size / save period in seconds: 0.01/0
      GC grace seconds: 0
      Compaction min/max thresholds: 4/32
      Read repair chance: 0.0
      Replicate on write: true
      Bloom Filter FP chance: default
      Built indexes: []
      Compaction Strategy: org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy
    ColumnFamily: Schema
    "current state of the schema"
      Key Validation Class: org.apache.cassandra.db.marshal.BytesType
      Default column value validator: org.apache.cassandra.db.marshal.BytesType
      Columns sorted by: org.apache.cassandra.db.marshal.UTF8Type
      Row cache size / save period in seconds / keys to save : 0.0/0/all
      Row Cache Provider: org.apache.cassandra.cache.ConcurrentLinkedHashCacheProvider
      Key cache size / save period in seconds: 0.01/0
      GC grace seconds: 0
      Compaction min/max thresholds: 4/32
      Read repair chance: 0.0
      Replicate on write: true
      Bloom Filter FP chance: default
      Built indexes: []
      Compaction Strategy: org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy
    ColumnFamily: Versions
    "server version information"
      Key Validation Class: org.apache.cassandra.db.marshal.UTF8Type
      Default column value validator: org.apache.cassandra.db.marshal.BytesType
      Columns sorted by: org.apache.cassandra.db.marshal.UTF8Type
      Row cache size / save period in seconds / keys to save : 0.0/0/all
      Row Cache Provider: org.apache.cassandra.cache.ConcurrentLinkedHashCacheProvider
      Key cache size / save period in seconds: 0.01/0
      GC grace seconds: 0
      Compaction min/max thresholds: 4/32
      Read repair chance: 0.0
      Replicate on write: true
      Bloom Filter FP chance: default
      Built indexes: []
      Column Metadata:
        Column Name: version
          Validation Class: org.apache.cassandra.db.marshal.UTF8Type
      Compaction Strategy: org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy

Cassandra - installing on Ubuntu 12.04 Amazon EC2

> sudo apt-get update

> sudo vi /etc/apt/sources.list

Add the following lines:

deb http://www.apache.org/dist/cassandra/debian 12x main
deb-src http://www.apache.org/dist/cassandra/debian 12x main

> sudo apt-get update

You would get 

W: GPG error: http://www.apache.org 10x InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 4BD736A82B5C1B00

Request the public key. You may need to change the public key (bold text above) when doing a gpg.

> gpg --keyserver wwwkeys.pgp.net --recv-keys 4BD736A82B5C1B00
> sudo apt-key add ~/.gnupg/pubring.gpg
> sudo apt-get update

> sudo apt-get install cassandra

> ps auwx | grep cassandra

> kill -9 {pid_of_cassandra}

Change the data, log locations:

> sudo vi /etc/cassandra/cassandra.yaml

Change the following locations if needed. (Cassandra does not run well on EBS - refer to the Apache Cassandra Document)

data_file_directories - (/var/lib/cassandra/data)
commitlog_directory -  (/var/lib/cassandra/commitlog)
saved_caches_directory - (/var/lib/cassandra/saved_caches)

Start Cassandra with:
> sudo /etc/init.d/cassandra start

Stop Cassandra with:
> sudo /etc/init.d/cassandra stop

Note: If you see the following, that means Cassandra is already started

Error: Exception thrown by the agent : java.rmi.server.ExportException: Port already in use: 7199; nested exception is:
java.net.BindException: Address already in use

> netstat -tulpn

Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 127.0.0.1:8005          0.0.0.0:*               LISTEN      20358/java    
tcp        0      0 127.0.0.1:9160          0.0.0.0:*               LISTEN      5996/jsvc.exec
tcp        0      0 127.0.0.1:3306          0.0.0.0:*               LISTEN      18848/mysqld  
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      20358/java    
tcp        0      0 0.0.0.0:47697           0.0.0.0:*               LISTEN      5996/jsvc.exec
tcp        0      0 0.0.0.0:50324           0.0.0.0:*               LISTEN      5996/jsvc.exec
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      619/sshd      
tcp        0      0 127.0.0.1:7000          0.0.0.0:*               LISTEN      5996/jsvc.exec
tcp        0      0 0.0.0.0:7199            0.0.0.0:*               LISTEN      5996/jsvc.exec
tcp6       0      0 :::22                   :::*                    LISTEN      619/sshd      
udp        0      0 0.0.0.0:68              0.0.0.0:*                           431/dhclient3  

If Cassandra is running, you should see, port 9160, 7000 and 7199 being in used

> sudo service cassandra status
This should return the status of Cassandra

Cassandra - What do you need to keep in mind when designing family columns?

Pass the timestamp with each column value. Do not use this timestamp as data in your application. Cassandra uses it internally for conflict resolution.

Super columns may have performance issues. Use composite columns.

Design your schema based on how your application would query data. Identify the frequently used ones. Always remember Cassandra is the map of maps.

De-normalize for read performance.

Use secondary indices only when the column keys are similar (Ex. Family column of User with column key Provinces; and the secondary index is on Provinces)

It is okay to store values in column keys and it is okay to leave column values to null.

Don't store something like "item description" as the column key.

Use wide rows for ordering, grouping, and filtering.

A row is not split across nodes. Cassandra allows 2 billion columns per row.

From Relational Database to Cassandra

The following is an analogy to introduce Cassandra to people with relational database background. The left side is the Cassandra model; the right side is the relational model.

Keyspace - Database
Column Family - Table
Row Key - Primary key
Column key - Column name
Column value - Column value

However, when designing Cassandra keyspace schema, it's better to think of Cassanda column families as a map of a map:

Map < RowKey, SortedMap < ColumnKey, ColumnValue > >

Super column groups columns, turning the two nested maps into three nested map.

Map < RowKey, SortedMap < ColumnKey,
SortedMap < ColumnKey, ColumnValue> > >

Cassandra - Create a keyspace tutorial

Launch the Cassandra CLI:

> bin/cassandra-cli

Create a keyspace called "testing" (In MySQL, it's like a database).

> create keyspace testing;

Showing the keyspace (You will see the following):

> show keyspaces;

Keyspace: testing:
  Replication Strategy: org.apache.cassandra.locator.NetworkTopologyStrategy
  Durable Writes: true
    Options: [datacenter1:1]
  Column Families:

> use testing;

> create column family Users with comparator = 'UTF8Type';

> assume Users keys as utf8;

> update column family Users with column_metadata =

        [

        {column_name: first, validation_class: UTF8Type},

        {column_name: last, validation_class: UTF8Type},

        {column_name: age, validation_class: UTF8Type, index_type: KEYS}

        ];

Insert a user:

> set Users['jsmith']['first'] = 'John';

> set Users['jsmith']['last'] = 'Smith';

> set Users['jsmith']['age'] = '18';

> get Users[1]

=> (column=name, value=John, timestamp=1356649133517000)
=> (column=password, value=Smith, timestamp=1356649170278000)
Returned 2 results.
Elapsed time: 2.22 msec(s).

Update a user:

>set Users['jsmith']['first'] = 'Adam';

>get User where age = '18';

Etsy - MySQL Sharding

Below is the Esty architecture and how Etsy is handling over 1.5 billion page views per month and 15TB stored data.  The interesting part is that it is based on MySQL.

Extensive sharding is used for partitioning data.  Important concepts include building a database to handle shard_id look ups (user_id to shard_id, group_id to shard_id), and shard migration

The Etsy Shard Architecture: Starts With S and Ends With Hard from jgoulah

Setting up Cassandra - ubuntu Amazon EC2

Download Cassandra

sudo apt-get install cassandra

The config is located in:

/etc/cassandra/conf/cassandra.yml

Create the following directories or modify these settings in conf/cassandra.yml if necessary

data_file_directories (/var/lib/cassandra/data)
commitlog_directory (/var/lib/cassandra/commitlog)
saved_caches_directory (/var/lib/cassandra/saved_caches)

Modify log directory in conf/log4j-server.properties

log4j.appender.R.File=/var/log/cassandra/system.log

By default, Cassandra uses between 1/4 and 1/2 of the available RAM.

If you are getting OutOfMemory exceptions or massive garbage collections, change the following in conf/cassandra-env.sh

#MAX_HEAP_SIZE="4G"
#HEAP_NEWSIZE="800M"

Rule of thumb - HEAP_NEWSIZE = 1/4 of MAX_HEAP_SIZE

Start Cassandra (-f means run in background):

bin/cassandra -f

Accessing the Cassandra CLI:

bin/cassandra-cli

Cassandra Commands:

assume                  Apply client side validation.
connect                 Connect to a Cassandra node.
consistencylevel        Sets consisteny level for the client to use.
count                   Count columns or super columns.
create column family    Add a column family to an existing keyspace.
create keyspace         Add a keyspace to the cluster.
del                     Delete a column, super column or row.
decr                    Decrements a counter column.
describe cluster        Describe the cluster configuration.
describe                Describe a keyspace and its column families or column family in current keyspace.
drop column family      Remove a column family and its data.
drop keyspace           Remove a keyspace and its data.
drop index              Remove an existing index from specific column.
get                     Get rows and columns.
incr                    Increments a counter column.
list                    List rows in a column family.
set                     Set columns.
show api version        Show the server API version.
show cluster name       Show the cluster name.
show keyspaces          Show all keyspaces and their column families.
show schema             Show a cli script to create keyspaces and column families.
truncate                Drop the data in a column family.
update column family    Update the settings for a column family.
update keyspace         Update the settings for a keyspace.
use                     Switch to a keyspace.

Git - Fixing the crlf, linebreak, eol nightware

Problem:

Windows, Linux, and Mac have different ways to represent line breaks (end-of-line).  Windows use CRLF, linux and Mac use LF.

When developers are checking in code using different operating systems, a git diff will show lines with different line break representations to be different.

To compare without caring about the line breaks, use

git diff --ignore-space-at-eol

To clean and rebuild the directory:

Set text=auto in .gitattributes

echo "* text=auto" << .gitattributes

Remove the Git index:

rm .git/index

Rebuild the index

git reset

Commit the changes:

git add .
git commit

You should see the following:

The file will have its original line endings in your working directory.
warning: LF will be replaced by CRLF in  

Tracking a branch from Github

Deleting a local branch

git branch -d {branch_name}

Deleting a remote branch

git checkout :{branch_name}

Track a new branch from Github

git checkout origin/wundr_compilable_project -t

MySQL schema commands

Select a database:

use {database_name};

Show all tables:

show tables;

Show all columns in a table

describe {table_name};
show columns from {table_name};

Show indices:

show index from {table_name};

Show MySQL status;

show engine innodb status;

Porting MySql from Ubuntu to MacOSX - Case sensitivity, foreign Key violation

Story:

My project is using Spring, Hibernate, and MySQL. I was trying to port a MySQL database from EC2 Ubuntu to my MacOSX for local development.

When I test the application, sometimes when an insert is triggered, I get a foreign key constraint failed exception.


SQL Error: 1452, SQLState: 23000
Cannot add or update a child row: a foreign key constraint fails


l made sure the foreign keys are not violated.

Then I ran the command in mysql:

show engine innodb status

I found the following:

------------------------
LATEST FOREIGN KEY ERROR
------------------------
Foreign key constraint fails for table `store`.`checkoutevent`:
,
  CONSTRAINT `FKBD82C0D499A4F1ED` FOREIGN KEY (`id`) REFERENCES `TransactionEvent` (`id`)
But the parent table `store`.`TransactionEvent`
or its .ibd file does not currently exist!

I checked that the TransactionEvent table indeed exists.


Explanation:

After further investigation, I found that case sensitivity matters in different platforms.

Value	Meaning
0	Table and database names are stored on disk using the lettercase specified in the CREATE TABLE or CREATE DATABASE statement. Name comparisons are case sensitive. You should not set this variable to 0 if you are running MySQL on a system that has case-insensitive file names (such as Windows or Mac OS X). If you force this variable to 0 with --lower-case-table-names=0 on a case-insensitive file system and access MyISAM tablenames using different lettercases, index corruption may result.
1	Table names are stored in lowercase on disk and name comparisons are not case sensitive. MySQL converts all table names to lowercase on storage and lookup. This behavior also applies to database names and table aliases.
2	Table and database names are stored on disk using the lettercase specified in the CREATE TABLE or CREATE DATABASE statement, but MySQL converts them to lowercase on lookup. Name comparisons are not case sensitive. This works only on file systems that are not case sensitive! InnoDB table names are stored in lowercase, as for lower_case_table_names=1.

Solution:

Export the database

mysqldump -u{username} -p --databases {database_name} > {database_name}.sql

Restart MySQL with --lower-case-table-names=1

mysqld --lower-case-table-names=1

Recreate the database (you need to drop the existing database first)

mysql -u{username} -p < {database_name}.sql

IOS 6 - PNG Images in webkit showing as black background

This is a weird problem that happens to me only in IOS 6.

If you are using UIWebKit in your IOS app and also using CSS to render images (ex. button background), make sure you flatten the images.

In Adobe Photoshop, when exporting the images (Export for Web), uncheck transparency and interlaced.

The following are image samples.

Image that renders normally:

Image that renders black:

Try it out and let me know if you are facing the same problems.

Git - deleting a remote branch from github

git push origin :{branch_to_be_deleted}

Spring JavaMail - SSLHandshakeException, SunCertPathBuilderException

If you get the following exception, that means your Java run time doesn't trust the certificate.

javax.mail.MessagingException: Can't send command to SMTP host (javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target)

Get the cert:

openssl s_client -connect email-smtp.us-east-1.amazonaws.com:25 -starttls smtp > amazon_ses.cert

If you get

verify error:num=20:unable to get local issuer certificate

Open the file and copy everything between the following two lines:

-----BEGIN CERTIFICATE-----

-----END CERTIFICATE-----

Save it to amazon_ses.cert.

Import the cert to Java

sudo keytool -import -alias [alias] -file [cert_file] -keystore [java_home]/lib/security/cacerts

Fill in the correct keystore password when prompted.

MySQL - Export and import table schema

Export a database with only tables:

mysqldump --no-data --tables -u{username} -p{password} {database} > {sqldump_filename}

Import:

mysql -u{username} -p{password} {database} < {sqldump_filename}

Maven and Tomcat - UnsupportedClassVersionError unsupported major.minor version 51.0 unable to load class

This error means that the run time JDK version is older than the compiled JDK version. The solution is to make sure Maven and Tomcat pointing to the same JDK.

Check Maven version:

mvn --version

Sample Output:

Apache Maven 3.0.4 (r1232337; 2012-01-17 08:44:56+0000)
Maven home: /opt/tools/apache-maven-3.0.4
Java version: 1.7.0_09, vendor: Oracle Corporation
Java home: /usr/lib/jvm/java-7-openjdk-amd64/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "3.2.0-31-virtual", arch: "amd64", family: "unix"

Check java version

java -version

Sample Output:

java version "1.7.0_09"
OpenJDK Runtime Environment (IcedTea7 2.3.3) (7u9-2.3.3-0ubuntu1~12.04.1)
OpenJDK 64-Bit Server VM (build 23.2-b09, mixed mode)

Check if you have set the JAVA_HOME environment correctly.

echo $JAVA_HOME

Sample Output:

/usr/lib/jvm/java-7-openjdk-amd64

If you are sure everything is pointing to the same JDK and you still get the UnsupportedClassVersionError, check if Tomcat is using the correct JDK

vi /etc/init.d/tomcat7

Search For:

for jdir in $JDK_DIRS; do
    if [ -r "$jdir/bin/java" -a -z "${JAVA_HOME}" ]; then
        JAVA_HOME="$jdir"
    fi
done
export JAVA_HOME
echo $JAVA_HOME #Add this line

Now start Tomcat

service tomcat7 start 

Install Maven 3 on Amazon EC2 Ubuntu

Choose a location of your choice

cd /opt/tools

Download Maven 3

wget ftp://mirror.csclub.uwaterloo.ca/apache/maven/maven-3/3.0.4/binaries/apache-maven-3.0.4-bin.tar.gz

Unpack Maven

tar -xzvf apache-maven-3.0.4-bin.tar.gz

Add java and maven environment variables:

sudo vi ~/.bashrc
export JAVA_HOME=$(readlink -f /usr/bin/java | sed "s:bin/java::")

export M2_HOME=/opt/tools/apache-maven-3.0.4
export MAVEN_HOME=/opt/tools/apache-maven-3.0.4
export M2=/opt/tools/apache-maven-3.0.4/bin
export PATH=/opt/tools/apache-maven-3.0.4/bin:$PATH

Save ~/.bashrc

source ~/.bashrc 

Test Maven:

mvn --version

Install git and clone a project on Amazon EC2 Ubuntu

Install git:

sudo apt-get install git

Setup git profile:

git config --global user.name "{your_name}"
git config --global user.email "{your_email}"
git config --global github.user "{your_github_login}"

Clone your project:

git clone {project_url}

Renaming origin

git remote rename origin github

Install Tomcat 7 on Amazon EC2 Ubuntu

Install tomcat7

sudo apt-get install tomcat7

Start tomcat7

sudo service tomcat7 start

Stop tomcat7

sudo service tomcat7 stop

Restart tomcat7

sudo service tomcat7 restart

Tomcat7 runs on port 8080 by default. Remember to add port 8080 to the security group in the Amazon EC2 Management Console.


Change from port 8080/8443 to 80/443

sudo vi /etc/tomcat7/server.xml

Enable AUTHBIND

sudo vi /etc/default/tomcat7 

# If you run Tomcat on port numbers that are all higher than 1023, then you
# do not need authbind.  It is used for binding Tomcat to lower port numbers.
# NOTE: authbind works only with IPv4.  Do not enable it when using IPv6.
# (yes/no, default: no)
#AUTHBIND=no 

AUTHBIND=yes

Create AUTOBIND permission files

touch /etc/authbind/byport/80
touch /etc/authbind/byport/443
chmod 0755 /etc/authbind/byport/80
chmod 0755 /etc/authbind/byport/443
chown tomcat7:tomcat7 /etc/authbind/byport/80
chown tomcat7:tomcat7 /etc/authbind/byport/443

Restart Tomcat7

sudo service tomcat7 restart

Check if Tomcat is running.

netstat -tulpn

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 127.0.0.1:8005          0.0.0.0:*               LISTEN      1268/java    
tcp        0      0 127.0.0.1:3306          0.0.0.0:*               LISTEN      18848/mysqld  
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      1268/java    
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      619/sshd      
tcp6       0      0 :::22                   :::*                    LISTEN      619/sshd      
udp        0      0 0.0.0.0:68              0.0.0.0:*                           431/dhclient3  

Tomcat7 is running on port 80 as process "java".

Deployment root path: /var/lib/tomcat7/webapps
Log path: /var/log/tomcat7

You can change the deployment root path in /etc/tomcat7/server.xml. Search for "webapps".

You will also need to change logging.properties

Search for

1catalina.org.apache.juli.FileHandler.level = FINE
1catalina.org.apache.juli.FileHandler.directory = /vol1/logs
1catalina.org.apache.juli.FileHandler.prefix = catalina.
2localhost.org.apache.juli.FileHandler.level = FINE
2localhost.org.apache.juli.FileHandler.directory = /vol1/logs
2localhost.org.apache.juli.FileHandler.prefix = localhost.

CATALINA_HOME and CATALINA_BASE are defined in:

vi /etc/init.d/tomcat7 

# Directory where the Tomcat 6 binary distribution resides
CATALINA_HOME=/usr/share/$NAME
# Directory for per-instance configuration files and webapps
CATALINA_BASE=/var/lib/$NAME

$NAME is tomcat7


Change webapps location to EBS volume:

sudo vi /etc/tomcat7/server.xml

Find

Change appBase to "/vol/webapps" or your desired location.

/vol is the mounted point of a EBS volume. To create a EBS volume, read the post - Amazon EC2 - Mounting a EBS drive.

Restart tomcat7.

Deploy a war file to "/vol/webapps" and access the application at

http://{ec2-domain}/{app_name}

Install Java OpenJDK 7 on Amazon EC2 Ubuntu

Machine Specs:

• Ubuntu Server 12.04.1 LTS 64-bit

Installing OpenJDK7:

1) Download openjdk-7.

sudo apt-get install openjdk-7-jdk

Use "java -version" to check what version java is being used.

You may have multiple versions of JDK installed. Run update-alternatives to choose the java version.

sudo update-alternatives --config java 

  Selection    Path                                            Priority   Status
------------------------------------------------------------
* 0            /usr/lib/jvm/java-6-openjdk-amd64/jre/bin/java   1061      auto mode
  1            /usr/lib/jvm/java-6-openjdk-amd64/jre/bin/java   1061      manual mode
  2            /usr/lib/jvm/java-7-openjdk-amd64/jre/bin/java   1051      manual mode

Choose 2 when it prompts.


2) To find out where Java is installed:

which java

3) Setting JAVA_HOME

sudo vi ~/.bashrc
append "export JAVA_HOME=$(readlink -f /usr/bin/java | sed "s:bin/java::")" to .bashrc and save the filesource ~/.bashrc

source ~/.bashrc  

echo $JAVA_HOME

Make sure you are modifying ~/.bashrc as root. If you are doing it as a user (ex. ubuntu), the root will not have JAVA_HOME set.

Persist MySQL on EBS Amazon EC2

The goal of this post is to move MySQL files to EBS volume so the data can persist.

Start by making the following folders on a mount point called "/vol".  "/vol" maps to a EBS volume.  (For instructions on how to mount a EBS volume, refer to the post - Amazon EC2 - Mounting a EBS drive)

sudo mkdir /vol/etc
sudo mkdir /vol/lib
sudo mkdir /vol/log

Move all the default MySQL folders/files to the above folders

sudo mv /etc/mysql     /vol/etc/
sudo mv /var/lib/mysql /vol/lib/
sudo mv /var/log/mysql /vol/log/

Point the MySQL paths to the EBS paths

sudo mkdir /etc/mysql
sudo mkdir /var/lib/mysql
sudo mkdir /var/log/mysql
sudo vi /etc/fstab 

Append "/vol/etc/mysql /etc/mysql none bind" to /etc/fstab
Append "/vol/lib/mysql /var/lib/mysql none bind" to /etc/fstab
Append "/vol/log/mysql /var/log/mysql none bind" to /etc/fstab 

sudo mount /etc/mysql
sudo mount /var/lib/mysql
sudo mount /var/log/mysql

 The resulting /etc/fstab file should look similar to the following:

LABEL=cloudimg-rootfs   /        ext4   defaults        0 0
/dev/xvdb       /mnt    auto    defaults,nobootwait,comment=cloudconfig 0       2
/dev/xvda3      none    swap    sw,comment=cloudconfig  0       0
/dev/xvdf       /vol    xfs     noatime 0 0
/vol/etc/mysql /etc/mysql     none bind
/vol/lib/mysql /var/lib/mysql none bind
/vol/log/mysql /var/log/mysql none bind

Start MySQL

sudo /etc/init.d/mysql start 

Unmounting MySQL volume.

sudo /etc/init.d/mysql stop
sudo umount /etc/mysql
sudo umount /var/lib/mysql
sudo umount /var/log/mysql

 Handy command for checking the mount status

df -h

How to start and stop MySQL on ubuntu

Starting MySQL:

sudo /etc/init.d/mysql start

Stopping MySQL:

sudo /etc/init.d/mysql stop

Restart MySQL:

sudo /etc/init.d/mysql restart

Check MySQL status:

sudo netstat -tap | grep mysql

This is from a default installation of MySQL in Ubuntu Server 12.04.1 LTS on Amazon EC2

sudo apt-get install -y mysql-server

Amazon EC2 - Mounting a EBS drive

This post applies to

• mounting a new EBS volume
• mounting an existing EBS volume or snapshot

Specs:

• Ubuntu Server 12.04.1 LTS 64-bit
• XFS

1.) Creating a EBS volume (skip this if you have a volume already)

• Login to your Amazon EC2 Admin Console
• In the left sidebar menu, expand Elastic Block Volume and click on Volumes
• Click on Create Volume
• Specify a size (Ex. 8 Gib)
• Set Availability Zone to the region that your instance is in (Note that it is very important that the EBS and the instance are in the same region.)
• Click "Yes, Create"
• Wait till the console says the EBS is available (blue circle).

2.) Attaching a EBS volume to an instance

• Right click on a EBS volume
• Click Attach Volume
• Select your instance (If you want to launch a new instance, check the post - Amazon EC2 - Launching Ubuntu Server 12.04.1 LTS step by step guide)
• If the volume says "attaching" for a long time (ex. 10 mins), check this post - EBS hangs in "attaching" mode forever (will not attach) without errormsg.

3.) Mounting the EBS volume

ssh into your instance  (ex. ssh -i {key} {user}@{ec2_address})


sudo apt-get update && sudo apt-get upgrade -y


Check if the drive is available by

sudo fdisk -l

Sample output:

Disk /dev/xvdf: 8589 MB, 8589934592 bytes
255 heads, 63 sectors/track, 1044 cylinders, total 16777216 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000 

Disk /dev/xvdf doesn't contain a valid partition table

Install xfs tools.

sudo apt-get install -y xfsprogs xfs

sudo mkfs.xfs /dev/xvdf 

Add the new volume to the file system configuration file.

sudo vi /etc/fstab

Add the following to the end of the file.

/dev/xvdf /vol xfs noatime 0 0

Save the file.

/vol is the mount point for the partition /dev/xvdf.

Note that xvd[f-p] are partitions for EBS. You can choose which letter to use when creating the EBS in the EC2 admin console.


sudo mkdir -m 000 /vol

sudo mount /vol

cd into /vol to see if it's accessible.

Installing the Amazon EC2 API Tools

This post is deprecated. Please check this post here - Amazon EC2 Command Line Tools Installation Guide.

The Amazon EC2 API tools allow you to access admin console functions in command line. The following instructions are for Ubuntu Server 12.04.1 LTS.

1) Enable multiverse
sudo perl -pi.orig -e   'next if /-backports/; s/^# (deb .* multiverse)$/$1/'   /etc/apt/sources.list

2) Enable the awstools
sudo apt-add-repository ppa:awstools-dev/awstools
sudo apt-get update

3) Install AWS command line tools
sudo apt-get install ec2-api-tools ec2-ami-tools iamcli rdscli
sudo apt-get install aws-cloudformation-cli elbcli


Create the AWS X.509 certificate at https://aws-portal.amazon.com/gp/aws/securityCredentials if you don't have it already.


mkdir $HOME/.aws

Then copy the cert- and pk- files to the following directories.

$HOME/.aws/cert-{some_id}.pem
$HOME/.aws/pk-{some_id}.pem

Create the file $HOME/.aws/aws-credential.txt and put the following into the file (Your access_key and secret key are available at https://aws-portal.amazon.com/gp/aws/securityCredentials).

export AWS_ACCESS_KEY={your_access_key}
export AWS_SECRET_KEY={your_secret_key}

Ex.


AKI[...]
DPh[...]

Test the following commands:

• ec2-describe-instances
• ec2-describe-volumes
• ec2-describe-regions

sftp - uploading files to Amazon EC2

sftp -i {key} {user}@{host}

(Ex. sftp -i key.pem ubuntu@ec2-111-11-11-111.compute-1.amazonaws.com)

cd {upload_destination_directory}

lcd {upload_source_directory}

mput {filename a, filename b, ... filename n}

Popular commands:

• ls - list the current directory on the remote machine
• cd - change directories on the remote machine
• mkdir - make a directory in current directory on remote machine
• lls - list the current directory on the local machine
• lcd - change directories on the local machine
• lmkdir - make a directory in current directory on local machine
• get - get files from the remote machine
• put - put files on the remote machine
• mget - get multiple files from the remote machine
• mput - put multiple files on the remote machine
• help - display commands available in sftp

Amazon EC2 - Launching Ubuntu Server 12.04.1 LTS step by step guide

In the ec2 admin console, click on "launch instance".

Select classic wizard.

Select Ubuntu Server 12.04.1 LTS 64 bit (Note that the instances with the yellow star sides are eligible for the first year free tier.)

In the instance dropdown box, select the desired instance type (for the free tier, select Micro).

For the availability zone, choose one or leave it to no preference. Remember the availibility you choose. The important thing to remember is that the EBS volume we set up later needs to be in the same availability zone.

Click Continue.

Check "Prevention against accidental termination".

Click Continue.

Leave the drive options as default and click continue.

On the next screen, add some tags. This makes it easier to search and manage an instance. Click continue.

Create a new key pair. Note that you can only download this once. Do NOT lose this. If you lose it, you will lose access to this instance. (You can always replicate a similar instance by generating a new key pair)

Create a new Security Group by filling in the group name and description. They can be anything.

Add ports 22, 80, 443, 25.

Review your settings and when ready, click Launch.

Taking a screenshot of a window in MacOSX Mountain Lion

• command + shift + 4
• press space
• click a window using the mouse or the trackpad
• a screenshot with timestamp will be saved to the desktop

MySQL - Export and import a database

Export a MySQL database:

mysqldump -u{username} -p{password} {database} > {export_filename}

Import from a MySQL database dump:

mysql -u{username} -p{password} {database} < {export_filename}

Download files from Amazon EC2 using terminal

SCP is a handy tool for downloading files from EC2.

In the following code:

• {user} is the user for the host. In Amazon EC2, this can be "ec2-user" or "ubuntu "
• {domain} is the ec2 address. You can find this by logging into the web ec2 admin panel
• {remote_path} is the path to the file that you want to copy
• {local_path} is the save location in your local drive

Redirect from naked domain to www subdomain

This is often done for SEO.  Rather than having the naked domain and www subdomain, simply do a 301 redirect from naked to www subdomain.

Substitute example.com with your domain below.