zData Ambari Extensions

Ambari is a tool which makes provisioning, managing, and monitoring of Apache Hadoop deployments easy. zData's Ambari Extensions builds atop Ambari to provide easy deployment and management of HAWQ, Chorus, and soon other Pivotal technologies.

The master branch contains code to work with Apache Ambari. There is a pivotal branch which works with Pivotal Ambari.

Visit the project's documentation for quick start guides and more information.

Getting started with Vagrant

1. Requires the following plugins: vagrant-hostmanager, vagrant-reload

   vagrant plugin install vagrant-hostmanager
   vagrant plugin install vagrant-reload
   
   vagrant plugin install vagrant-aws # Optional, provision on AWS
   vagrant plugin install vagrant-cachier # Optional, cache downloaded packages to speed up provisioning

2. Create boxes with Virtualbox

   vagrant up # Bring up master, slave1

   Note: Copy vagrant-env.conf.sample to vagrant-env.conf and modify the values to change various vagrant settings such as the number of slave machines.

3. Connect, vms created: master, slave0, slave1

   master.ambaricluster.local
   slave1.ambaricluster.local
   
   vagrant ssh master
   vagrant ssh slave1
   

Additional steps to deploy to AWS:

1. Install a dummy box:

   vagrant box add dummy https://github.com/mitchellh/vagrant-aws/raw/master/dummy.box

2. Configure Vagrant with your unique Amazon access key and secret. Creating a user Vagrantfile in ~/.vagrant.d/Vagrantfile with:

   Vagrant.configure('2') do |config|
       config.vm.provider :aws do |aws, override|
           aws.access_key_id = ENV['AWS_KEY']
           aws.secret_access_key = ENV['AWS_SECRET']
       end
   end

   Add environmental variables with your AWS_KEY and AWS_SECRET in your ~/.bashrc:

   export AWS_KEY="THEKEY"
   export AWS_SECRET="THESECRET"

3. Configure Vagrant with a keypair it can use to communicate with the created boxes. Generate a new keypair. In your user Vagrantfile add the following lines in the config.vm.provider :aws block:

   aws.keypair_name = 'vagrant'
   override.ssh.private_key_path = '~/.ssh/aws-vagrant'

   The variable keypair_name should be the name of the keypair on AWS, the variable private_key_path should be the path to the private key on your local computer.

   Your user Vagrantfile should now look like:

   Vagrant.configure('2') do |config|
       config.vm.provider :aws do |aws, override|
           aws.access_key_id = ENV['AWS_KEY']
           aws.secret_access_key = ENV['AWS_SECRET']
           aws.keypair_name = 'vagrant'
           override.ssh.private_key_path = '~/.ssh/aws-vagrant'
       end
   end

4. Create the boxes on AWS

   vagrant up --provider=aws --no-parallel
   vagrant hostmanager

More information about getting started with Ambari using vagrant is available here.

Services

Greenplum

Installs and manages the Pivotal Greenplum database software.

What The Service Does Not Do

• Does not automatically create or setup XFS filesystem.
• Does not specifiy an IO scheduler of deadline.
• Does not configure read-ahead.
• Does not disable transparent hugepage.

Chorus

Installs and manages zData Chorus.

Minimum Tuning Values

• inimum_memory = 256M
• maximum_memory = 256M
• young_heap_size = 128M
• max_perm_size = 256M

HAWQ

Installs and manages the Pivotal HAWQ Hadoop SQL engine.

PXF

Installs and manages the Pivotal Extension Framework, patches it to work with Hortonwork's Hadoop.

Development

Writing features for both Vanilla and Pivotal

Sometimes it's possible to use the same code for both Pivotal and Apache Ambari, when this is the case you can use git to help simplify merging a feature branch to both branches.

git checkout master
git checkout -b feature/##

# Do feature

# Pivotal port
git checkout -b feature/##_pivotal_port
git rebase --onto pivotal master feature/##_pivotal_port
git checkout pivotal
git merge --ff-only feature/##_pivotal_port
git branch -D feature/##_pivotal_port

# Merge to master
git checkout master
git merge feature/##_pivotal_port

Running tests

Tests are written with unittest2, and they require the ambari source code be available.

1. Set an environment variable so unittests can find the Ambari's source directory:

   git clone https://git-wip-us.apache.org/repos/asf/ambari.git ~/source-ambari
   export AMBARI_SOURCE="$HOME/source-ambari"
   

2. Install necessary python packages

   pip install unittest2 mock
   

3. Run tests:

   (cd tests; python -m unittest discover)
   

Retrieve Artifacts

To install HAWQ you will need some files from Pivotal. You can find these files at https://network.pivotal.io/products/pivotal-hd. Create an account if you don't have one already, and place the downloaded files in the artifacts folder located in the project root.

• Pivotal HD 2.1 -> Pivotal HD 2.1.0
• Pivotal HD 2.1 -> PHD 2.1.0: Pivotal HAWQ 1.2.1.0
• 4.3.5.0 Database Server -> Greenplum Database 4.3.5.0 for Red Hat Enterprise Linux 5 and 6

Licensing

Ambari is an open source deployment tool, users must follow license agreements provided by Hortonworks and Pivotal software.