This repository has been archived by the owner on Mar 8, 2018. It is now read-only.
Iterate over a stream of json objects + support new yajl version + bug fixes #33
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* updated build instructions
rickeyski@cf3c29d Move IssueTwentySevenTest dict into python2 to prevent python3 test from bailing on loading the test suite Update IssueTwentySevenTest to only run on python2. (probably want to run this on python3 too).
* Store decoded objects in a list
…d objects in the internal buffer.
Remove actions from _internal_decode that reset the parser state
clang -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I. -Iincludes/ -Iyajl/src -I/usr/include/python2.6 -c yajl.c -o build/temp.linux-x86_64-2.6/yajl.o -Wall -DMOD_VERSION="0.3.6-38862b0"
yajl.c:411:15: warning: incompatible pointer types initializing 'PyCFunction' (aka 'PyObject *(*)(PyObject *, PyObject *)') with
an expression of type 'PyCFunctionWithKeywords' (aka 'PyObject *(*)(PyObject *, PyObject *, PyObject *)')
{"dumps", (PyCFunctionWithKeywords)(py_dumps), METH_VARARGS | METH_KEYWORDS,
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
yajl.c:427:14: warning: incompatible pointer types initializing 'PyCFunction' (aka 'PyObject *(*)(PyObject *, PyObject *)') with
an expression of type 'PyCFunctionWithKeywords' (aka 'PyObject *(*)(PyObject *, PyObject *, PyObject *)')
{"dump", (PyCFunctionWithKeywords)(py_dump), METH_VARARGS | METH_KEYWORDS,
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
yajl.c:331:18: warning: unused function 'py_iterload' [-Wunused-function]
static PyObject *py_iterload(PYARGS)
^
3 warnings generated.
clang -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I. -Iincludes/ -Iyajl/src -I/usr/include/python2.6 -c encoder.c -o build/temp.linux-x86_64-2.6/encoder.o -Wall -DMOD_VERSION="0.3.6-38862b0"
encoder.c:316:9: warning: expression result unused [-Wunused-value]
PyObject_INIT_VAR(op, &PyString_Type, size);
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from encoder.c:32:
In file included from /usr/include/python2.6/Python.h:81:
/usr/include/python2.6/objimpl.h:159:29: note: instantiated from:
( Py_SIZE(op) = (size), PyObject_INIT((op), (typeobj)) )
^
encoder.c:316:9: note: instantiated from:
PyObject_INIT_VAR(op, &PyString_Type, size);
^ ~~
encoder.c:316:27: note: instantiated from:
PyObject_INIT_VAR(op, &PyString_Type, size);
^~
1 warning generated.
Updated runtests.sh to run issue_11 test under python3
…rieved by _fetchObject
To iterate over json items written to the stdin of a python process:
import yajl
import sys
for i in yajl.Decoder(allow_multiple_values=True,stream=sys.stdin):
print i
Python3 support is broken in this build.
Refactor py_yajldecoder_decode
add check to decoder init function to check stream has read() method
Also fix functions which were returning true/false/none types but not incrementing the refcount
Unicode buffer was not being freed after use
Merged
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Note: I've aimed to keep backwards API compatibility but I've made some additions, how these new additions actually work are open to discussion.
I think that this patch aims to address issues:
The main improvement is that you can now write something like:
Which will let you iterate over a stream of json objects read from the processes std input channel. It's not too slow either, on my core2duo a yajl based producer / consumer connected via a unix pipe with a very basic object can process about 100,000 json objects/sec.
I don't think the iterator method will handle non blocking sockets very well at the moment, it may not handle them at all - I've not tested it yet. If the file object is in blocking mode the iterator handles that fine.
I've also fixed a number of bugs relating to memory management (including issue #32) along the way.
Summary of changes.
The unit tests still pass and the current version also supports python 3. Though I really should write some more unit tests for the new features and some documentation to accompany them.