8 posts tagged “ec2”
I did read about this a while back when the challenges using EC2 was how not lose a database in case of a machine failure. Luckily it never happened and forgot all about EBS, the persistent disk for Amazon EC2. We went live with a new customer over the weekend and EBS came up all over again for data backup. Quite simple to use it.
Interesting aspect to tackle after putting applications on EC2 in a live production environment. Our first customer has been live for over 1 year on EC2 and never been down for a single time. We didn't have to reboot the linux server though we had to restart our java processes for upgrades. Touchwood. Database incremental backups happen every minute to S3 and a daily back up at night. This is mostly an enterprise app, so not very stringent requirements on real time failover. Shortly we are going live with a new retail customer facing application - a 24 x 7 uptime requirement. Scalability is not much of an issue as our server is still operating under 65MB with response times under 50ms most of the time. Failover is a key aspect for us - passive or active, active being the preferred option.
Ant (http://ant/.apache.org) is embodies simplicity, extensibility and never in your way software. A lot of developer tasks can be automated with this tool. For quite some time we used it to just build a WAR file and deploy it Tomcat on our desktop machines. Deployment to server used to be a one hour task. Build with the server configuration files, ftp to server, extract static stuff to apache, re-deploy the database and recycle the server.
Recently my EC2 instance was hung. The website was getting timed out. A clear signal that the web server is not responding and all socket communication is not happening. As the sockets were not reactive, so the ssh server. I was not able to login using the SSH terminal - PUTTY.
Now I just can't shut down that instance and start another one as I had some production data in mysql data files which had not backed up. I had told my customer 99% uptime, 256MBps connectivity but what about reliability.. Now they are going to lose production data. Literally, I was shocked. I desparetly tried to get my data back. Did googling and got depressed reading the comments of pool. We then put a urgent request in the forum and amazon replied that nothing is possible.
After I cooled down a bit, it was time to fight back. First thing I thought of trying was restarting the instance. However, I was not sure about my data. All my mysql data files were in /mnt. So the question was am I going to loose data if I restart my instance. Some links I referred confused me more; I left with a understanding that anything at MNT has the possibility of loss. But the mnt was my target. Now I took a decision to restart my server - The last resort.
I issued restart command from my desktop. Oops... there was no effect. Then I went to my another amazon instance. There I had the EC2 tools. I used those tools and issued a restart command. Did a ssh again. The server was still hung. I sat tight for 5 mins. Again connected via my putty. Bingo, this time the server shown me the login prompt. My curiosity doubled up.. Am I going to have my data in /mnt. Then I changed directory to /mnt. Everything was there intact. I just relaxed. Thank God - I survived a crash
I remember few years back when I was running the Websphere, it was very difficult to kill a instance. On killing of one instance, a deamon was spawning another. Killing that one, one more was coming up. I had tried "kill -9", "kill -KILL" - No help. When one was going down, other was coming up. It is not like, I have 2 process instances running and on killing of both, the application is completely down. Rather, it is only one and taking birth each time it is murdered.
Why not this happens for a server. If I consider that is possible, virtually the application is immortal.
[I have taken this section out from my regular writting as I want to highlight on one particular issue. Recently I had mapped a production instance to amazon EC2 instance in web DNS mapping. I issued a shutdown command on EC2 machine by mistake. It just took me 2 mins to bring up a snapshot of the last instance as I was taking the snapshot. Then I pointed to this new server address in web DNS mapping. This process took a good 30mins and the system remained down for so long :(]
Consider a machinsm is available, where web DNS mapping is instantaneous to the depployed server. In this spectrum, I have an application which transcends through server instances. The possibilities lies on the snapshot creation. Before death can I take an exact snapshot and all next instances starts from there. Amazon EC2 has infiniband networking to S3. So, network latency is not my concern. Hadoop is there for my rescue to transfer the big files. The big file is the tar ball of the complete mnt. If you are reading my previous posts, you know you can shove everyting to mnt. That means the complete harddisk portion of my interest to the S3. This helps on saving the database files, other contents, application state and everything else.
During shutdown,
- Create a snapshot tar ball of /mnt of the portion of harddisk of my interest. - 30sec
- Copy this to S3.
- Start another instance.
- When the next instance comes up, it self configures itself during start up taking the file from S3.
I have taken snapshot and created a new instance quickly but manually. It takes just a minute. How can I do it automatically?
First I looked for linux shutdown hooks like init. No luck. If you are aware of any, please let me know. Now I am going to explore by creating a blank java application and starting it during init and putting a shutdown hook there. This shutdown hook will grap the linux shutdown. I am concluding this reading the post at http://forum.java.sun.com/thread.jspa?threadID=504889 . Once I finish this experiement, I blog about my findings :) This will open up following possibilities in EC2:
- Immortal application in a mortal server
- No information loss when instance shutted down.
- Many app instance creation when load increases. [What about the database??? I have a big blog coming on this. I am half way through]
Don't forget to help me on implementing a linux hook of shutdown rather than going via the java way... A work around not a clean solution :(
I was monitoring my EC2 linux instance. I suddenly found my free memory is depleting. I was cluless. With research I got an paper which explains about the linux cache memory. http://mail.nl.linux.org/linux-mm/2003-03/msg00077.html
Memory Cached - Any modern operating system will cache files frequently accessed.
With detail research I found out one of the major I/O operations are log files. I use log4J for logging in the application. Though the log files are rotated, the system spits out huge amount of logging information. The monitoring scripts, scans through these log files in each 5mins to get if any exception has happend. The frequent access of log files and huge log file sizes makes the OS caching these files.
Major loggers of log4J are only in WARN level. However, each request entry is written for monitoring purpose. Then I asked myself why am I writting these logs. The answer was
- For trouble shooting during a fault
- For Monitoring
- As a audit mechanism
Consider system has a fault / exception. This time I require to log the complete execution path as in case of debug to quickly identify the issue and solve it. With warning, it becomes difficult to get closure to actual circumstancecs.
Regarding this I had discussed with sunil before and he was talking about a in memory logging and on need flushing. This is the background of conceiving onelineLog -
A simple on need logging framework to optimize system resources and serving clients faster.
The definition of availability:
Present and ready for use; at hand; accessible:
Capable of being gotten; obtainable
Qualified and willing to be of service or assistance
Unavailability is not felt till the moment users reach out for the service and are not served. For me the counting of unavailability starts the moment absence of availability felt.
My application is a web based application. The potential of unavailability is bacause of:
- Absence of the desktop machines.
- Absence of network connection.
- Unable to serve due to fault on:
- Website not available
- Application servers are down or hung
- Slow performance and high response time
- Software bug and functionality is not available.
To make a high available application, the system requires:
- Mobile presence
- Smart client applications, provision to browse at client side during no network.
- The website is up all the time.All machines and mobile devices are functioning properly.
- All serves are functioning properly - Apache, tomcat, mysql, mail
- All service agents are functioning properly - sms, email
During,
- Single node failure
- High load situation
- New application version deployment
- Regular machine patch management
- Hacking attacks, intrusion attacks
- Virus Attack
- External applications unavailability
and immediately serving after a fault due to,
- Database failure
- System/node unavailability
- Application bugs
OK, the principles I am guided to provide high availability in my application are:
During Requirement
Availability for me is a functional requirement. All the requirements are analyzed from this perspective too. If system couldn't be available from world wide web, can he perform some operations from his mobile device via SMS, can he send a mail and do the same operation.
During Architecture & Design
- Stateless architecture on Amazon EC2 to quickly provide more servers as the number of users increase.
- Central point of information update and numerious points of information viewing to scale. In my application around 80% time information is viewed and 20% time it get's modified.
- Each component handles failure situation. Ex. The email agent if could not send mail using account X, it will try another email provider.
During execution - production environment
On production - During available period:
- Preparing Stand by machine to take over during failure
- Monitoring the health and providing necessary support.
- Continuous revision of server deployment according to the measured data:
Staep -1 ) Moving database and other single point of failure to an dedicated machine.
Step - 2) Up scaling te machine by adding more memory, cpu, disk and other resources in the existing machine
Step - 3) Hot backup and replication of all changed content.
On production - During Unavailable period
The Preparedness
- Failover drill process. It forces the system to shut down and waits for the failover system to take care.
See our profile for the motivation behind the blog
How to build a Linux install running on EC2
For information on how to start and logging-in to Ec2 instance from desktop, refer to amazon webservices
In this blog, I will share my experiences on baking a machine image on Ec2 with necessary application servers for a java web stack - Apache, Tomcat, Mysql and pma (php based mysql admin tool)
OK, first what problems I faced;
- When I built my first image, the server quickly shooted up of disk usage to 90%. This is because we put all our deployments in sda1. Amazon only gives around 2GB in sda1 and rest on /mnt 158GB. Anything I put on mnt is going to wash away and can't be in the image.
- Finding right executables to install.
To address this problem the strategy is to install java and hadoop in sda1 and rest under mnt. Hadoop helps on getting the rest of the deployment pack from S3 to local /mnt. I assumed this will get the sda1 disk usage to minimum. However there are more surprises waiting in store. First let me take you though steps of building an image.
-
All available base instances ec2-describe-images.cmd -o amazon
-
I chose ec2-public-images/fedora-8-i386-base-v1.06.manifest.xml amazon
-
Started instance ec2-run-instances fedora-8-i386-base-v1.06.manifest.xml -k gsg-keypair
-
mkdir /mnt/install
cd /mnt/install
yum install perl-DBI -
echo "Get Mysql"
wget http://dev.mysql.com/get/Downloads/MySQL-5.1/MySQL-server-community-5.1.24-0....
wget http://dev.mysql.com/get/Downloads/MySQL-5.1/MySQL-client-community-5.1.24-0....
wget http://dev.mysql.com/get/Downloads/MySQL-5.1/MySQL-shared-community-5.1.24-0....
wget http://dev.mysql.com/get/Downloads/MySQL-5.1/MySQL-devel-community-5.1.24-0.r... -
rpm -i MySQL-server-community-5.1.24-0.rhel3.i386.rpm
rpm -i MySQL-client-community-5.1.24-0.rhel3.i386.rpm
rpm -i MySQL-devel-community-5.1.24-0.rhel3.i386.rpm
rpm -i MySQL-shared-community-5.1.24-0.rhel3.i386.rpm -
echo "Now add it to the path"
PATH=$PATH:/usr/bin -
Hardening mysql - /usr/bin/mysql_secure_installation
-
Mysql by default gets configured to start during startup. This is not optimal. Why? I want to have mysql installation directory to mnt and once EC2 starts, I get rest of image ball from S3 and host in mnt. So I went ahead and deleted mysql from startup
-
cd /etc/init.d; mv mysql ~
-
Time to shutdown mysql and move this to mnt
mysqladmin --user=root --password=jskfk shutdown -
cp -R /var/lib/mysql /mnt/install
rm -rf /var/lib/mysql
ln -s /mnt/install/mysql /var/lib/mysql
mysqld --user=root &
Time to take care of Apache -
yum install openssl-devel
cd /mnt/install/downloads
wget http://mirror.nyi.net/apache/httpd/httpd-2.2.6.tar.gz
tar -zxvf httpd-2.2.6.tar.gz
cd /mnt/install/downloads/httpd-2.2.6 -
./configure --prefix=/mnt/install/apache --with-mpm=worker \
--enable-so \
--disable-actions \
--disable-alias \
--disable-asis \
--disable-auth \
--disable-autoindex\
--disable-cgi \
--disable-env \
--disable-imap \
--disable-include \
--disable-negotiation \
--disable-status \
--disable-userdir \
--enable-cache=shared \
--enable-disk_cache=shared \
--enable-mem_cache=shared \
--enable-deflate=shared \
--enable-expires=shared \
--enable-filter=shared \
--enable-ext_filter=shared \
--enable-headers=shared \
--enable-proxy=shared \
--enable-rewrite=shared \
--enable-ssl=shared \
--enable-usertrack=shared \
--enable-vhost_alias=shared \
--enable-auth_digest=shared \
--enable-authn_core=shared \
--enable-authz_core=shared \
--enable-authn-dbm=shared \
--enable-authn-anon=shared \
--enable-authn-dbd=shared \
--enable-authn-alias=shared \
--enable-authz-host=shared \
--enable-authz-groupfile=shared \
--enable-authz-user=shared \
--enable-authz-dbm=shared \
--enable-authz-owner=shared \
--enable-setenvif=shared \
--enable-dumpio=shared \
--enable-log_forensic=shared \
--enable-mods-shared=all \
--with-z=/usr/lib \
--with-openssl-libs=/usr/lib -
make
make install
PATH=$PATH:/mnt/install/apache/bin -
The JK connector installation from Tomcat to Apache. You may see the links are not working. There could be a potential release of a new version.
-
echo "Install JK"
cd /mnt/install/downloads
wget http://www.apache.org/dist/tomcat/tomcat-connectors/jk/source/tomcat-connecto...
tar -zxvf tomcat-connectors-1.2.26-src.tar.gz
cd /mnt/install/downloads/tomcat-connectors-1.2.26-src/native
./configure --with-apxs=/mnt/install/apache/bin/apxs --enable-EAPI
make
make install -
Install and configure Php stuffs
-
echo "Get Php"
cd /mnt/install/downloads
wget http://in2.php.net/get/php-5.2.4.tar.gz/from/us.php.net/mirror
tar -zxvf php-5.2.4.tar.gz
cd /mnt/install/downloads/php-5.2.4
yum install libxml2-devel
./configure --with-apxs2=/mnt/install/apache/bin/apxs --with-mysql=/usr/bin/mysql --with-config-file-path=/mnt/install/apache/conf --with-libxml-dir=./ext/libxmlln -s /usr/lib/libmysqlclient.so /usr/lib/mysql/libmysqlclient.so
make
make install -
Tomcat runs in Java. Java we are going to install in sda1 as hadoop also relies on Java. Why do we need hadoop and java in the main image?? Without this we can't go to S3 and get the tar ball (mnt folder). Hadoop needs Java internally. So we need both :)
-
echo "Java installation"
Goto http://java.sun.com/javase/downloads/index_jdk5.jsp
Download JDK 5.0 Update 15, Accept the License
Copy the link for jdk-1_5_0_15-linux-i586.bin
cd /mnt/install/downloadscd /usr/local
sh /mnt/install/downloads/java.bin
mv jdk1.5.0_15 jdk1.5
export JAVA_HOME=/usr/local/jdk1.5
export PATH=$PATH:$JAVA_HOME/bin -
Install tomcat now
-
cd /mnt/install/downloads
wget http://apache.mirror.facebook.com/tomcat/tomcat-6/v6.0.14/bin/apache-tomcat-6.0.14.tar.gz
tar -zxvf apache-tomcat-6.0.14.tar.gz
cd /mnt/install
mv /mnt/install/downloads/apache-tomcat-6.0.14 /mnt/install/tomcat
-
Time to set all configuration files
-
Make sure follwing lines are there in httpd.conf
# Restrict Access to sensitive files
LoadModule php5_module modules/libphp5.so
LoadModule jk_module modules/mod_jk.so
Deny from all
JkWorkersFile /mnt/install/apache/conf/workers.properties
JkShmFile /mnt/logs/apache.mod_jk.shm
JkLogFile /mnt/logs/apache.mod_jk.log
JkLogLevel error
JkLogStampFormat "[%a %b %d %H:%M:%S %Y] "
JkMount /*/*.xml ajp13
# mod_deflate (compress output for browsers that support it)
AddOutputFilterByType DEFLATE text/html text/plain text/xml text/css application/x-javascript text/javascript text/x-js BrowserMatch ^Mozilla/4 gzip-only-text/html -
workers.properties settings
worker.loadbalancer.type=lb
worker.list=ajp13
worker.ajp13.port=8009
worker.ajp13.host=localhost
worker.ajp13.type=ajp13
worker.loadbalancer.balance_workers=ajp13 -
I took out all my logs - mysql, tomcat, apache to /mnt/logs. That requires proper settings on the files present in the conf directory.
-
Bouncing once the servers:
/mnt/install/tomcat/bin/shutdown.sh
/mnt/install/apache/bin/apachectl stop
/mnt/install/tomcat/bin/startup.sh
/mnt/install/apache/bin/apachectl start
-
Well phpMyAdmin a must have to aminister the mysql database. here is what I have done
-
cd /mnt/install/downloads
wget http://prdownloads.sourceforge.net/phpmyadmin/phpMyAdmin-2.11.1-all-languages.tar.gz?download
tar -zxvf phpMyAdmin-2.11.1-all-languages.tar.gz
cd /mnt/install/downloads/phpMyAdmin-2.11.1-all-languages
mkdir /mnt/install/apache/htdocs/pma
cd /mnt/install/apache/htdocs/pma
cp -R /mnt/install/downloads/phpMyAdmin-2.11.1-all-languages/* .
cp config.sample.inc.php config.inc.php
vi config.inc.php
change $cfg['blowfish_secret'] =
mkdir config
chmod o+rw config
cp config.inc.php config/
chmod o+w config/config.inc.php
mkdir data
chmod o+rw data
-
Start mysql if not running -
mysql --password=whrshfuurw
-
GRANT USAGE ON mysql.* TO 'support'@'localhost' IDENTIFIED BY 'xxuser'
;
GRANT SELECT (
Host, User, Select_priv, Insert_priv, Update_priv, Delete_priv,
Create_priv, Drop_priv, Reload_priv, Shutdown_priv, Process_priv,
File_priv, Grant_priv, References_priv, Index_priv, Alter_priv,
Show_db_priv, Super_priv, Create_tmp_table_priv, Lock_tables_priv,
Execute_priv, Repl_slave_priv, Repl_client_priv
) ON mysql.user TO 'support'@'localhost';
GRANT SELECT ON mysql.db TO 'support'@'localhost';
GRANT SELECT ON mysql.host TO 'support'@'localhost';
GRANT SELECT (Host, Db, User, Table_name, Table_priv, Column_priv)
ON mysql.tables_priv TO 'support'@'localhost';
http://website/pma/scripts/setup.php
Config server: port 3306, authentication = http controluser = support passwd
Upload/Download data directory
mv config/config.inc.php . # move file to current directory
chmod o-w config.inc.php # remove world write permissions
-
Login to the website and ec2-XXXX-XXX-XXX-XXX/pma
-
Let's do some support accounts created:
groupadd support
useradd -g support supportadmin
passwd supportadmin
supportadminpass
ssh supportadmin@localhost
-
Never forget to set the .bashrc. This is the place, I set the timing to India. Believe me, other wise the date thing just kills due to little slippage in the development. The servers run in USA and users sitting in India.. Wow.. Got to be really careful in each and every aspect of the application.
export JAVA_OPTS="-Duser.timezone=IST"
export JAVA_HOME=/usr/local/jdk1.5
export PATH=/mnt/install/tomcat/bin:/mnt/install/apache/bin/:$JAVA_HOME/bin:/usr/local/hadoop/bin:$PATH
set -o vi
alias servers='netstat -l -n -p -t -u -w'
alias stop="apachectl stop; shutdown.sh; netstat -l -n -p -t -u -w | grep 8080 | grep LISTEN | sed 's/ [ ]*/ /g' | cut -d' ' -f7 | cut -d'/' -f1 | xargs kill"
alias start="startup.sh;apachectl start"
echo "Welcome abord.."
-
Little bit hardening of the servers
-
netstat -l -n -p -t -u -w
iptables --list
iptables -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
iptables -A OUTPUT -m state --state NEW,RELATED,ESTABLISHED -j ACCEPT
iptables -A INPUT -p tcp -m tcp --dport 22 -j ACCEPT
iptables -A INPUT -p tcp -m tcp --dport 80 -j ACCEPT
iptables -A INPUT -p tcp -m tcp --dport 443 -j ACCEPT
iptables -A INPUT -s 127.0.0.1 -j ACCEPT
iptables -A OUTPUT -p tcp -m tcp --dport 22 -j DROP
iptables -A INPUT -j DROP
iptables -A FORWARD -j DROP
-
vi /mnt/install/apache/htdocs/php.ini
expose_php = Off
display_erros = Off
log_errors = On
error_log = /mnt/logs
-
Put the Hadoop in sda1.
cd /mnt/install/downloads
wget http://apache.tradebit.com/pub/hadoop/core/stable/hadoop-0.15.3.tar.gz
gunzip hadoop-0.15.3.tar.gz
cd /usr/local
tar -xvf /mnt/install/downloads/hadoop-0.15.3.tar
rm docs and src
BrowserMatch ^Mozilla/4\.[0678] no-gzip
BrowserMatch \bMSIE\s7 !no-gzip !gzip-only-text/html
I deleted the download directories and log files. Tarred up the whole /mnt/install and sent this to s3 using hadoop. After this I stopped servers, cleaned up mnt and cut an image.
Next time I built as java and hadoop were in /usr/local, I was able to retrieve my deploy ball from s3 and unpackaging it at /mnt. Well I was ready to go with sucessful start of all servers. However another surprise was just waitting for me. I suddenly saw, the disk usage is shotting up. I couldn't figure out why.. It was from /usr/bin
Now It is time to take that out. Softlink was my solution
echo "Moving out the /usr directory from sda1 to mnt"
cp -R /usr /mnt
rm -rf /usr
ln -s /mnt/usr /usr
cd /usr/bin
This created another set of issues though reduced my disk usage to 25%. Inside /usr, there were many relative soft links like ../../. So finding wach one which is missing and fixing it was my job :(
rm -f env; ln -s /bin/env env
rm -f Mail; ln -s /bin/mail Mail
rm -f awk; ln -s /bin/gawk awk
rm -f gettext; ln -s /bin/gettext gettext
rm -f gunzip; ln -s /bin/gunzip gunzip
rm -f gzip; ln -s /bin/gzip gzip
rm -f cut; ln -s /bin/cut cut
rm -f kill; ln -s /bin/kill kill
rm -f gawk; ln -s /bin/gawk gawk
rm -f libdb-4.6.so; ln -s /lib/libdb-4.6.so libdb-4.6.so
rm -f libselinux.so; ln -s /lib/libselinux.so.1 libselinux.so
rm -f libsepol.so; ln -s /lib/libsepol.so.1 libsepol.so
rm -f libdl.so; ln -s /lib/libdl.so.2 libdl.so
rm -f libm.so; ln -s /lib/libm.so.6 libm.so
rm -f libssl.so; ln -s /lib/libssl.so.0.9.8b libssl.so
rm -f libBrokenLocale.so; ln -s /lib/libBrokenLocale.so.1 libBrokenLocale.so
rm -f libthread_db.so; ln -s /lib/libthread_db.so.1 libthread_db.so
rm -f libutil.so; ln -s /lib/libutil.so.1 libutil.so
rm -f libnsl.so; ln -s /lib/libnsl.so.1 libnsl.so
rm -f libnss_compat.so; ln -s /lib/libnss_compat.so.2 libnss_compat.so
rm -f libnss_db.so; ln -s /lib/libnss_db.so.2 libnss_db.so
rm -f libnss_dns.so; ln -s /lib/libnss_dns.so.2 libnss_dns.so
rm -f libnss_files.so; ln -s /lib/libnss_files.so.2 libnss_files.so
rm -f libnss_hesiod.so; ln -s /lib/libnss_hesiod.so.2 libnss_hesiod.so
rm -f libnss_nis.so; ln -s /lib/libnss_nis.so.2 libnss_nis.so
rm -f libnss_nisplus.so; ln -s /lib/libnss_nisplus.so.2 libnss_nisplus.so
rm -f libanl.so; ln -s /lib/libanl.so.1 libanl.so
rm -f libz.so; ln -s /lib/libz.so.1.2.3 libz.so
rm -f libcidn.so; ln -s /lib/libcidn.so.1 libcidn.so
rm -f libresolv.so; ln -s /lib/libresolv.so.2 libresolv.so
rm -f libcrypt.so; ln -s /lib/libcrypt.so.1 libcrypt.so
rm -f libcrypto.so; ln -s /lib/libcrypto.so.0.9.8b libcrypto.so
rm -f librt.so; ln -s /lib/librt.so.1 librt.so
cd /usr/lib/lsb/
rm -f install_initd; ln -s /sbin/chkconfig install_initd
rm -f remove_initd; ln -s /sbin/chkconfig remove_initd
cd /usr/sbin
rm -f accton; ln -s /sbin/accton accton
rm -f kudzu; ln -s /sbin/kudzu kudzu
rm -f rcmysql; ln -s /etc/init.d/mysql rcmysql
rm -f hwclock; ln -s /sbin/hwclock hwclock
cd /usr/share/terminfo/a
rm -f ansi; ln -s /lib/terminfo/a/ansi ansi
cd /usr/share/terminfo/d
rm -f dumb; ln -s /lib/terminfo/d/dumb dumb
cd /usr/share/terminfo/l
rm -f linux; ln -s /lib/terminfo/l/linux linux
cd /usr/share/terminfo/v
rm -f vt100-am; ln -s /lib/terminfo/v/vt100-am vt100-am
rm -f vt100; ln -s /lib/terminfo/v/vt100 vt100
rm -f vt100-nav; ln -s /lib/terminfo/v/vt100-nav vt100-nav
rm -f vt200; ln -s /lib/terminfo/v/vt200 vt200
rm -f vt220; ln -s /lib/terminfo/v/vt220 vt220
cd /usr/share/terminfo/x
rm -f xterm; ln -s /lib/terminfo/x/xterm xterm
rm -f xxterm; ln -s /lib/terminfo/x/xxterm xxterm
---------------------------------------
Now things are good. Let's grab the ball from s3
/usr/local/hadoop/bin/hadoop fs -ls /image
/usr/local/hadoop/bin/hadoop fs -copyToLocal /image/nn.tar .
tar xvf nn.tar
Finally, if you need the mail server and not running, start it,
echo "start the mail server if not running"
listeningports=`netstat -l -n -p -t -u -w | sed 's/ [ ]*/ / g' | cut -d' ' -f4`
isListening=`echo $listeningports | grep 25 | wc -l`
if [ $isListening -ne 1 ]; then
sendmail -bd &
fi
---------------------------------------
Was this blog little too detailed? Well, in this I wanted to give enough details for those wanting to start an EC2 instance can do cut n paste.
Further on I will blog on what to do when an instance is lost, hung up and other failures like that. Interestingly we were able to recover an instance after Amazon support team gave up.
If you are trying out these commands, cool. Let us know how you got on. If you are reading so far I acknowledge your patience. Let me know your view points.
To avoid spam, I am giving our mail address on a image. See below or in the top. Write a comment here or send a mail. Tomorrow I will share how we use EC2 for production hosting rather than for non critical activities such as testing, backup etc.