feat: auto-create 1.x DB or 2.x bucket for flux task logs #2622
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
lesam
force-pushed
the
auto-create-db
branch
from
ee74eb3 to
bff9c92
Compare
lesam
force-pushed
the
auto-create-db
branch
from
bff9c92 to
109022a
Compare
Closes #2620 Flux task logs are stored in the "Analytic Store", which is backed by an influxdb 1.x Database or 2.x Bucket. Auto-creation gives an infinite retention policy, which can be adjusted later by the user. Note that for 1.x, "database/rp" bucket names will not be autocreated, only bare "database" bucket names. This could potentially be added in the future but the assumption is if a user wants control over the retention policy name they probably want to control the retention policy creation as well. Tested manually against 1.x and 2.x.
lesam
force-pushed
the
auto-create-db
branch
from
109022a to
3e5944a
Compare
gwossum
reviewed
Sep 15, 2021
task/backend/analytical_storage.go
Outdated
Comment on lines
35
to
57
| for try := 0; try < numTries; try++ { | ||
| // poll for influxdb to come up - usually it should be already up by the time we get here. | ||
| if try > 0 { | ||
| time.Sleep(1 * time.Second) | ||
| } | ||
| queryForError := func(q string) error { | ||
| // This query is only successful if the bucket exists | ||
| result, err := cli.QueryFlux(influxdb.FluxQuery{ | ||
| Org: dest.Org, | ||
| OrgID: dest.OrgID, | ||
| Query: q, | ||
| }) | ||
| if err != nil { | ||
| return err | ||
| } | ||
| for result.More() { | ||
| if err := result.Next().Tables().Do(func(table flux.Table) error { | ||
| return table.Do(func(col flux.ColReader) error { | ||
| return nil | ||
| }) | ||
| }); err != nil { | ||
| return err | ||
| } |
There was a problem hiding this comment.
If I'm reading this loop correctly, if we only succeed in creating the bucket on the last attempt, we will actually return an error even if the bucket was created. The bucket check and no error return is done at the top of the loop, and on the last iteration we can't get back to the top to check for success. Doesn't seem likely to happen in the real life, but might be confusing when it does.
gwossum
reviewed
Sep 15, 2021
gwossum
reviewed
Sep 16, 2021
lesam
force-pushed
the
auto-create-db
branch
from
4961834 to
651b45d
Compare
lesam
commented
Sep 16, 2021
| @@ -563,6 +563,11 @@ func (c *influxdbCluster) Open() error { | |||
|
|
|||
| c.watchSubs() | |||
|
|
|||
| // Even if we're not linking subscriptions, validate the client to ensure influxdb is actually up | |||
| if err := c.validateClientWithBackoff(ctx); err != nil { | |||
There was a problem hiding this comment.
Note this pings the configured influxdb and happens before we try to set up the flux task service.
lesam
force-pushed
the
auto-create-db
branch
from
651b45d to
04ec692
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Closes #2620
Flux task logs are stored in the "Analytic Store", which is backed
by an influxdb 1.x Database or 2.x Bucket.
Auto-creation gives an infinite retention policy, which can be adjusted later by the user.
Note that for 1.x, "database/rp" bucket names will not be autocreated, only
bare "database" bucket names. This could potentially be added in the future
but the assumption is if a user wants control over the retention policy name
they probably want to control the retention policy creation as well.
Tested manually against 1.x and 2.x.
Required for all non-trivial PRs