Applying patterns line length limit on the json message key

[cyriltovena]

We drop lines that are too big default 3k chars or contains too many tokens for performance reasons in pattern ingesters.

For json log line we only use a set of keys for discovering patterns.

We could probably do the json message parsing prior that limit and apply that limit on the message key.

Notes:

• We should avoid parsing the json key twice.
• This might means we'll hold the backing array for the log buffer entirely and increase memory usage.

[ravishankar15]

Hi @cyriltovena Correct me if my understanding is wrong, Basically if its a jsonTokenizer we wanted to check MaxAllowedLineLength on the extracted line from the set of keys that is after Tokenize. For other tokenizers it should proceed as it is now ?

Can you help with the clarification. I can raise a PR for this.