-
Notifications
You must be signed in to change notification settings - Fork 1
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
break AWSService interface into smaller pieces (batch, log, etc) #271
Conversation
response, err := http.Get(URL) | ||
if err != nil { | ||
fmt.Printf("%s", err) | ||
w.Close() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This needs to be CloseWithError(err)
, and then don't print the error - if you were printing the error you should use log
instead of fmt
.
e89193f
to
7a1b33b
Compare
1c48e07
to
e07409a
Compare
f132b15
to
3ce4190
Compare
ef75118
to
c688e1c
Compare
These are non-semantic changes only, in an effort to reduce the size of the 'integrate fake-batch' branch in #271. The intent is that changes in this diff should be 'obviously correct' by diff inspection and not require much testing.
9f26d34
to
df220d1
Compare
OK, so I worked on this quite a bit this week, and I achieved two things:
So, what remains to be done in this PR?
I suggest doing this not by keeping the current WIP patch, but instead by doing it one bit a time and graudally reducing this patch to nothing. |
df220d1
to
ffd527c
Compare
My mind has lost track of what's to be done here so it's time to make a list:
|
Alright, it's hard to see what actually needs doing but the original goal of this PR was to make fake-batch, on-prem etc. work. In light of that I'm going to do some offline end to end testing and anywhere that doesn't 'just work' I'm going to create a github issue for. Hopefully with all the groundwork that's been laid this shouldn't be too difficult. |
Type of change
Description
This PR depends on pingproto being vendored, that's covered by this PR: #265
The implementations involved depend on no_idle, #272
TODO
Previously we had the AWS interface under service/aws. It was quite big, it covered all of our interactions with AWS and used one implementation with one client for all of this. Now that we're gearing up for on-prem there's a need to have different aspects of that interface handled by different clients, for instance storage goes to minio while batch jobs go to fake-batch.
This PR aims to break up the AWS interface into Batch, Storage and LogStreaming. This should increase flexibility. It also aims to have implementations for these new interfaces so nothing breaks during the transition. Most of the Batch implementation is the same as it was, likewise the Storage, LogStreaming for AWS has been improved and LogStreaming for fake-batch has been created.
cw_id_watcher
has been moved since it was the sole user of a function that added complexity to the AWS interface and didn't fit in the new order. That function got moved up to the cron worker level.Review Checklist
Goals
Does it solve the problem?
Is it the simplest implementation of that solution?
Does it yak shave? Does it introduce new dependencies that aren't necessary?
Does it decrease modularity?
Does the user of a module need to import another module to use this one?
If we want to delete these changes, how easy is that?
Does it clarify our domain?
What things does it refine? What things get added? How does this pave the way for new things?
Are things named in such a way that a domain expert can find them?
Does it introduce non-domain concepts?
What does the user of this need to learn outside of our domain in order to use this?
Testing
Do we integration test changes to external services?
Do we unit test code we can change?