Optimize json parsing for faster indexing in elasticsearch

[dojutsu-user]

Optimizing the json parsing will result in faster indexing/reindexing.
https://github.com/readthedocs/readthedocs.org/blob/master/readthedocs/search/parse_json.py

[humitos]

I'm lacking context here. What are the ideas around this?

[dojutsu-user]

I don't currently have any "optimization ideas".
If you mean about this issue, when we do indexing of projects, json files are parsed and the relevant data is sent to elasticsearch, I believe that json parsing can be made more faster.
It will reduce the time taken for a project to get indexed into elasticsearch. Also, the time taken for indexing of all projects will also gets reduced.

[stsewd]

We use the json package from the std lib

readthedocs.org/readthedocs/search/parse_json.py

Line 93 in 34d1a15

data = json.loads(file_contents)

not sure how we can improve that

[ericholscher]

I imagine it's the HTML parsing with pyquery that is the slow part.

[dojutsu-user]

I think most of it is because the HTML parsing but we can also use a faster json library for deserialization.
Like: https://github.com/ijl/orjson

[dojutsu-user]

@ericholscher

I imagine it's the HTML parsing with pyquery that is the slow part.

pyquery by default uses html as its parser,
https://github.com/gawel/pyquery/blob/2351446c56d5c3df93e3fd81a2b828cf00e3b648/pyquery/pyquery.py#L225-L226

and it is quite slow.
I believe if we use html5_parser, this might gets faster.

Edit: html5_parser doesn't seem to work as expected. 😕

[ericholscher]

Believe we fixed this already 👍