feat: SSAPI Pagination (BPS-276) #1965
Conversation
Caleb-Hurshman
requested review from
schmikei
and removed request for
a team and
dpaasman00
| @@ -32,6 +32,7 @@ type Config struct { | |||
| Username string `mapstructure:"splunk_username"` | |||
| Password string `mapstructure:"splunk_password"` | |||
| Searches []Search `mapstructure:"searches"` | |||
| EventBatchSize int `mapstructure:"event_batch_size"` | |||
There was a problem hiding this comment.
Should this be on the search configuration itself?
There was a problem hiding this comment.
I guess. If this is something that a user would want to change for different queries, then that's not too hard to accomodate. In my head, this setting was more useful for weighing how much load should be placed on the Splunk API vs. on the receiver and would be the same across all searches.
| if logTimestamp.UTC().After(latestTime.UTC()) { | ||
| ssapir.logger.Info("skipping log entry - timestamp after latestTime", zap.Time("time", logTimestamp.UTC()), zap.Time("latestTime", latestTime.UTC())) | ||
| // logger will only log up to 10 times for a given code block, known weird behavior | ||
| // TODO: Consider breaking, assuming all logs are in order |
There was a problem hiding this comment.
Fair thing to look in to; we'd need to make sure that the events being retrieved from Splunk are in order, which I'm not confident to say one way or another on quite yet
There was a problem hiding this comment.
Took some time to look into this, the pattern I'm seeing is that events are returned latest to earliest, which makes sense based on how the Splunk search UI works. So it would be correct to break once a log is earlier than the earliest time.
| var resultCountTracker = 0 // track number of results exported | ||
| var offset = 0 // offset for pagination | ||
| var limitReached = false | ||
| for { |
There was a problem hiding this comment.
Let's set up some time to vscode share a solution on polling; feel like having an infinite for loop in this case could actually be improved if we start looking at context.Context
Proposed Change
Paginate Splunk search result responses. Add an
event_batch_sizeconfig option to control how many events (search results) to fetch at a time.Checklist