Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Scanner error: token too long #70

Open
pkuca opened this issue Apr 23, 2020 · 2 comments · May be fixed by #91 or #92
Open

Scanner error: token too long #70

pkuca opened this issue Apr 23, 2020 · 2 comments · May be fixed by #91 or #92

Comments

@pkuca
Copy link

pkuca commented Apr 23, 2020

When using gron -s, I am seeing the following error:

bufio.Scanner: token too long: %!s(MISSING)

I'm happy to submit a PR increasing the default buffer size to something you'd approve. Alternatively, I was thinking about how the buffer size could be provided as an option - I noticed there's only a single option passed in to the function now.
@pkuca
Copy link
Author

pkuca commented Apr 23, 2020

Referring to: https://github.com/tomnomnom/gron/blob/master/main.go#L267

@rishabhdeepsingh
Copy link

Why create a buffer size? can't we just use dynamic?

rjp added a commit to rjp/gron that referenced this issue Jan 13, 2022
@rjp
Adds a slurp mode for reading the full file
aff8c43 
Using the slurp option -- `-S`, `-slurp` -- causes gron to read *all* the
data into RAM before processing.  Tested on a 102M JSON file - memory
usage is about 400M for parsing the first object.

Fixes tomnomnom#70 (in a fairly sledgehammer way, I suppose.)
This was referenced Jan 13, 2022
Open
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Adds a slurp mode for reading the full file rjp/gron
Adds "decode all" option rjp/gron
2 participants
@pkuca @rishabhdeepsingh