Genie sandbox

[sriramkrishnan]

Since Genie has several pre-requisites such as Hadoop, Hive, Pig and Tomcat, and many configurations to tweak, we should create a sandbox (e.g. an AMI, a bootstrap action, a playbook, etc) that our users can instantiate to get Genie working easily out of the box.

[pas256]

I am happy to help with his as I am working on playbooks and AMIs for other projects as well:
https://github.com/awsanswers/netflixoss-ansible

[piaozhexiu]

Hello Peter, that would be awesome! Should I assign this to you?

[pas256]

Yeah sure. The turn around time is likely 1 month, though I hope to get to
it sooner.

Kind regards,
Peter Sankauskas

On Mon, Jul 22, 2013 at 8:21 PM, Cheolsoo Park notifications@github.comwrote:

Hello Peter, that would be awesome! Should I assign this to you?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/9#issuecomment-21390993
.

[sriramkrishnan]

Hi Peter, looks like GitHub won't let me officially assign this to you - but consider this yours. If you get sidetracked or don't want to work on it, let us know.

The high level requirement is that we just need a way to:

• Install all dependencies (Tomcat, Hadoop, etc)
• Update all properties as specified by the Wiki
• Deploy WAR file
• Bake AMI

We don't anticipate any backwards incompatible changes to anything above (although we may cleanup from of the UI/web components in genie-web).

Thanks!
Sriram

[pas256]

I have made a start on this, but there is still more to do. The most notable is that installing Hadoop as a dependency. Here is the progress so far:
https://github.com/Answers4AWS/netflixoss-ansible/blob/master/playbooks/genie-ubuntu.yml

[sriramkrishnan]

@pas256 et al, I have a first version of a bootstrap action that installs Genie on an EMR master. We can also port this to make it work via Ansible.

• Copy deploy/aws/emr_genie_bootstrap.sh to S3, and set environment var EMR_GENIE_BOOTSTRAP_LOC to point to that location
• Run deploy/aws/emr_genie_launch.sh

There is also a emr_genie_postinstall.sh script, which can be run from the master to register the EMR cluster with Genie, and to run a few jobs.

[sriramkrishnan]

QuickStart guide for EMR is now added: https://github.com/Netflix/genie/wiki/QuickStart-on-EMR

@pas256 let me know once your Ansible instructions are ready, and I will link to it from the Wiki (and call this issue closed). Thanks!

[pas256]

They are ready:

https://github.com/Answers4AWS/netflixoss-ansible#genie

[sriramkrishnan]

Added link to: https://github.com/Netflix/genie/wiki/QuickStart-on-EMR.

I think we can call this done. @pas256, thanks for your contribution!

[sriramkrishnan]

We have have two ways to install Genie easily on top of EMR. Closing this issue.