Home

EC2 Instance Configuration

Package Installation

$ sudo yum install emacs git

Python Packages

After following http://tumblr.kurttheviking.com/post/30920138314/painless-python-2-7-on-aws-ec2, run

$ sudo easy_install tweepy unicodecsv argparse boto

We want Python 2.7.x as the default:

$ sudo yum install python27
$ sudo rm /usr/bin/python
$ sudo ln -s `which python27` /usr/bin/python

Other Scaffolding

$ sudo mkdir /var/log/twitter
$ sudo chown $USER: /var/log/twitter
$ mkdir /var/log/twitter/old
$ touch /var/log/twitter/tweets.csv
$ sudo touch /var/lib/logrotate.status
$ sudo chown $USER: /var/lib/logrotate.status

Set up keys

Examine twitter_keys.py and .boto.sample.

Set up cron jobs

$ crontab -e

Add the following to the crontab file:

*/5 * * * * logrotate /home/ec2-user/data-gathering/logrotate.conf
*/10 * * * * python /home/ec2-user/data-gathering/s3/upload_to_s3.py -D -b [bucket] /var/log/twitter/old

Running the streaming script

I ran run_twitter_streaming_forever.sh inside a screen session, then power-detached.