Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Multi-tenant architecture for the app #996

Merged
merged 8 commits into from
Mar 24, 2016
Merged

Multi-tenant architecture for the app #996

merged 8 commits into from
Mar 24, 2016

Conversation

tomwilkie
Copy link
Contributor

My aim is to break this into multiple PRs:

TO DEMO:

$ make
$ cd experimental/multitenant
$ make
$ weave launch
$ ./run.sh
$ cd ../../
$ ./scope --probe-only

TODO:

  • Dynamo Collector
    • deal with periods spanning the hour
    • read row/column keys first, pick already decoded reports from a cache, and then send a multiget for the rest.
    • use sqs for sending shortcut reports.
  • Control Router
    • don't delete queues
    • correctly tidy up connections
    • use consul to keep connection state
  • Pipe Router
    • implement delete
    • make it work beyond the single replica case
    • implement garbage collection

@tomwilkie tomwilkie self-assigned this Feb 22, 2016
@tomwilkie tomwilkie changed the title Multi-tenant architecture for the app [WIP] Multi-tenant architecture for the app Feb 22, 2016
@tomwilkie tomwilkie force-pushed the multi-tenant-app branch 2 times, most recently from 6c1619b to 25d64f4 Compare February 29, 2016 11:32
@tomwilkie tomwilkie force-pushed the multi-tenant-app branch 2 times, most recently from b4e5a4a to 2bc52f4 Compare March 1, 2016 18:02
@tomwilkie tomwilkie mentioned this pull request Mar 9, 2016
Merged
@tomwilkie tomwilkie force-pushed the multi-tenant-app branch 5 times, most recently from 52b4bc4 to 339567f Compare March 10, 2016 08:44
@paulbellamy
Copy link
Contributor

Initially looks fine. fns (particularly consulpiperouter) are too large and need breaking down, IMO (but at least there are lots of comments).

@tomwilkie
Copy link
Contributor Author

Yeah I'm rewriting pipe router into mich smaller bits. Thanks!

On Friday, 11 March 2016, Paul Bellamy notifications@github.com wrote:

Initially looks fine. fns (particularly consulpiperouter) are too large
and need breaking down, IMO (but at least there are lots of comments).

—
Reply to this email directly or view it on GitHub
#996 (comment).

@tomwilkie tomwilkie force-pushed the multi-tenant-app branch 6 times, most recently from c9ab3a5 to 2451500 Compare March 17, 2016 12:20
@tomwilkie tomwilkie changed the title [WIP] Multi-tenant architecture for the app Multi-tenant architecture for the app Mar 17, 2016
paulbellamy
paulbellamy reviewed Mar 22, 2016
View reviewed changes
app/multitenant/consul_pipe_router.go
}

// Atomically modify a pipe in a callback.
// If pipe doesn't exist you'll get nil in callback.

This comment was marked as abuse.

Sign in to view

This comment was marked as abuse.

Sign in to view

This comment was marked as abuse.

Sign in to view

paulbellamy
paulbellamy reviewed Mar 22, 2016
View reviewed changes
app/multitenant/consul_pipe_router.go

// Next wait for other end to connect
// at this point we 'own' one end
pipe, err := pr.watch(key, mtime.Now().Add(10*longPollDuration), func(p *consulPipe) (bool, error) {

This comment was marked as abuse.

Sign in to view

This comment was marked as abuse.

Sign in to view

This comment was marked as abuse.

Sign in to view

This comment was marked as abuse.

Sign in to view

This comment was marked as abuse.

Sign in to view

@tomwilkie
Copy link
Contributor Author

The ConsulPipeRouter seems to be three things in one. There's the actual pipe routing, a consistent consul kv client, and an atomic-CASable value (backed by consul).

I've broken out the high-level consul client. Pipe-routing and the value data type have been left in there.

@tomwilkie
Vendor aws go library
90bc5e2
@tomwilkie
Vendor consul api
b61c5c0
@tomwilkie
Multi-tenant backends for app.
9885706 
    Add DynamoDB based collector
    - Store compressed reports in dynamodb

    Add SQS based control router.
    - Uses a queue per probe and a queue per UI for control requests & responses.

    Add Consul-based, horizontally-scalable, multi-tenant pipe router.
    - Uses consul to coordinate each end of pipe connections replicas of a pipe service.
@tomwilkie
Actor model for pipe router
0a8fd8a
@tomwilkie
Review feedback
ac638d1
@tomwilkie
Pin alpine version.
531e082
@tomwilkie
Break out factories for app backends, and encode everything in urls.
9610e95
@tomwilkie
Spelling
f0f03d3
@tomwilkie tomwilkie assigned paulbellamy and unassigned tomwilkie Mar 23, 2016
paulbellamy
paulbellamy reviewed Mar 24, 2016
View reviewed changes
app/multitenant/consul_pipe_router.go

addr := fmt.Sprintf("%s:%d", pr.advertise, privateAPIPort)
log.Infof("Serving private API on endpoint %s.", addr)
log.Infof("Private API terminated: %v", http.ListenAndServe(addr, router))

This comment was marked as abuse.

Sign in to view

This comment was marked as abuse.

Sign in to view

This comment was marked as abuse.

Sign in to view

@paulbellamy
Copy link
Contributor

LGTM

@tomwilkie tomwilkie merged commit a703fe8 into master Mar 24, 2016
@tomwilkie tomwilkie deleted the multi-tenant-app branch March 24, 2016 13:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees

@paulbellamy paulbellamy

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@tomwilkie @paulbellamy