Skip to content

Hadoop Versions

Matt Bossenbroek edited this page Sep 19, 2015 · 3 revisions

PigPen can also be used with Cloudera distributions of Hadoop. All you need to do is change the Hadoop and Pig dependencies to match what you're using:

(defproject pigpen-demo "0.1.0-SNAPSHOT"
  :description "FIXME: write description"
  :url "http://example.com/FIXME"
  :license {:name "Eclipse Public License"
            :url "http://www.eclipse.org/legal/epl-v10.html"}
  :main pigpen-demo.core
  :repositories [["cloudera" "https://repository.cloudera.com/artifactory/cloudera-repos/"]]
  :dependencies [[org.clojure/clojure "1.5.1"]
                 ;; See https://github.com/Netflix/PigPen for current version
                 [com.netflix.pigpen/pigpen "0.3.0"]]
  :profiles {:dev {:dependencies
                   [[org.apache.hadoop/hadoop-client "2.0.0-mr1-cdh4.4.0"]
                    [org.apache.pig/pig "0.11.0-cdh4.4.0"]]}})

This will prevent jar problems when the uberjar is submitted to the cluster.