Example Storm Topologies

Learn to use Storm!

---

Table of Contents

• Getting started
• Using storm-starter with Leiningen
• Using storm-starter with Maven
• Using storm-starter with IntelliJ IDEA

---

Getting started

Prerequisites

First, you need java and git installed and in your user's PATH. Also, two of the examples in storm-starter require Python and Ruby.

Next, make sure you have the storm-starter code available on your machine. Git/GitHub beginners may want to use the following command to download the latest storm-starter code and change to the new directory that contains the downloaded code.

$ git clone git://github.com/nathanmarz/storm-starter.git && cd storm-starter

storm-starter overview

storm-starter contains a variety of examples of using Storm. If this is your first time working with Storm, check out these topologies first:

1. ExclamationTopology: Basic topology written in all Java
2. WordCountTopology: Basic topology that makes use of multilang by implementing one bolt in Python
3. ReachTopology: Example of complex DRPC on top of Storm

After you have familiarized yourself with these topologies, take a look at the other topopologies in src/jvm/storm/starter/ such as RollingTopWords for more advanced implementations.

If you want to learn more about how Storm works, please head over to the Storm project page.

Using storm-starter with Leiningen

Install Leiningen

The storm-starter build uses Leiningen 2.0. Install Leiningen by following the leiningen installation instructions.

Running topologies with Leiningen

To run a Java topology

$ lein deps
$ lein compile
$ java -cp $(lein classpath) storm.starter.ExclamationTopology

To run a Clojure topology:

$ lein deps
$ lein compile
$ lein run -m storm.starter.clj.word-count

Using storm-starter with Maven

Install Maven

Maven is an alternative to Leiningen. Install Maven (preferably version 3.x) by following the Maven installation instructions.

Running topologies with Maven

storm-starter contains m2-pom.xml which can be used with Maven using the -f option. For example, to compile and run WordCountTopology in local mode, use the command:

$ mvn -f m2-pom.xml compile exec:java -Dstorm.topology=storm.starter.WordCountTopology

You can also run clojure topologies with Maven:

$ mvn -f m2-pom.xml compile exec:java -Dstorm.topology=storm.starter.clj.word_count

Packaging storm-starter for use on a Storm cluster

You can package a jar suitable for submitting to a Storm cluster with the command:

$ mvn -f m2-pom.xml package

This will package your code and all the non-Storm dependencies into a single "uberjar" at the path target/storm-starter-{version}-jar-with-dependencies.jar.

Running unit tests

Use the following Maven command to run the unit tests that ship with storm-starter. Unfortunately lein test does not yet run the included unit tests.

$ mvn -f m2-pom.xml test

Using storm-starter with IntelliJ IDEA

Importing storm-starter as a project in IDEA

The following instructions will import storm-starter as a new project in IntelliJ IDEA.

• Copy m2-pom.xml to pom.xml. This is requried so that IDEA (or Eclipse) can properly detect the maven configuration.
• Open File > Import Project... and navigate to the top-level directory of your storm-starter clone (e.g. ~/git/storm-starter).
• Select Import project from external model, select "Maven", and click Next.
• In the following screen, enable the checkbox Import Maven projects automatically. Leave all other values at their defaults. Click Next.
• Click Next on the following screen about selecting Maven projects to import.
• Select the JDK to be used by IDEA for storm-starter, then click Next.
  • At the time of this writing you should use JDK 6.
  • It is strongly recommended to use Sun/Oracle JDK 6 rather than OpenJDK 6.
• You may now optionally change the name of the project in IDEA. The default name suggested by IDEA is "storm-starter". Click Finish once you are done.