Be able to deploy into VPC

[greggilbert]

I've got a Storm setup that I'd like to deploy within a VPC on AWS. As is pretty standard in VPC setups, there's a bastion server that you SSH into first, and then get into the instances. I can't find a way to get sparse submit to work with this setup, though. It'd be nice if the deploy script could work with this.

Sidenote: one idea I had was to lein uberjar everything, then manually scp that up to the servers. When I do that, though, and run bin/storm jar 0.0.1-STANDALONE.jar MAINCLASS, I'm not sure what to put for MAINCLASS. My project.clj is defproject events "0.0.1-SNAPSHOT", but events doesn't work. Any thoughts on that? Is the only option to run it through lein?

I think it'd make sense to update the documentation to reflect these sort of outlying scenarios.

[floer32]

sounds like it'll be frustrating to work about but 👍 very important

[dan-blanchard]

The simplest thing to do:

1. Create a jar using lein uberjar
2. scp that jar and your topology Clojure file—I'm assuming this is called events.clj—to a server on AWS you can submit from.
3. SSH into the same server and run:

bin/storm jar events-0.0.1-SNAPSHOT-STANDALONE.jar streamparse.commands.submit_topology events.clj

Please let us know if that doesn't work.

[dan-blanchard]

Ah, I forgot one thing. You'll also need to have virtualenvs setup on all of your machines containing all of the Python prereqs you need. They will need to be located in the path indicated in your config.json file.

Once #99 is addressed, this step will no longer be necessary.

[codywilbourn]

You could also try a bit of ssh magic. This is all hypothetical because I don't have a test setup to try it on.

Add env.use_ssh_config = True to your fabfile.py. See #54 for a example snippet.

Update your ~/.ssh/config to use the bastion host for your commands.

Host *.internal.example.com
    ProxyCommand ssh bastion.example.com exec nc %h %p

If you don't have a common subdomain you'll have to list all of the hosts individually.

Host host1.example.com
    ProxyCommand ssh bastion.example.com exec nc %h %p
...

Setup your streamparse config to use all of the hosts normally (without bastion host).

[dan-blanchard]

Thanks @codywilbourn. I knew I should have just asked you. 😄

[floer32]

Yeah +1 for SSH config as a way to manage this. We've had great success with this approach.