[chore][fileconsumer] Remove generation field from reader

[djaglowski]

The fileconsumer package remembers all files it has seen for a short period of time. This ensures that we do not double-ingest files in the rare case where a file "temporarily disappears", such as may happen when with a file rotation strategy that involves use of a temp file name that is outside the include glob. In such cases, we may know of a file during one poll interval, then subsequently fail to find it in the next, then find it again in the next.

We manage this short-term memory by remember files for a given number of poll intervals, currently hardcoded to 3. This PR restructures the way that we manage these generations. Previously, the reader struct had a field called generation, which was incremented when rotating generations. All previously known readers were stored in a single slice, which was appended to and cut as appropriate to rotate the generations.

Now, each generation is stored in a separate slice, so we can manage generations without the need to increment or check a generation field. This has a couple benefits:

1. The management of generations is a concern that is outside the reader itself, so the reader should not have to carry this information or expose it externally as would be necessary if it were in a dedicated package.
2. Management of generations may change in the future, but it's likely that some notion of generations will persist. Future designs can more easily slot into a []generation pattern, vs a []reader pattern.

[github-actions]

This PR was marked stale due to lack of activity. It will be closed in 14 days.

[djaglowski]

This is failing due to race conditions in the flusher. As a result, I've taken a detour to simplify the tokenize package. (See #26241 as a potential end state.) I believe this will simplify the problem here as well.

[github-actions]

This PR was marked stale due to lack of activity. It will be closed in 14 days.

[djaglowski]

Resolved by #27396