Mimic sqlite3 standard module Type extensions to DB-API

[alanjds]

Provides a dbapi.py API as close as possible from stock sqlite3 module Type extensions of stdlib of Python.

Imported test_types.py tests from https://github.com/ghaering/pysqlite.
For now, one suite (CTE) is disabled and 2 tests fails, mainly about custom types combined with Unicode conversion:

But anyway is a lot better (more compatible) than before ;)

$ py.test src/test/test_types.py
============================= test session starts ==============================
platform darwin -- Python 2.7.5, pytest-2.9.2, py-1.4.31, pluggy-0.3.1
rootdir: /Users/alanjds/src/git/pyrqlite, inifile:
plugins: cov-2.4.0
collected 32 items

src/test/test_types.py ..................F......F......

=================================== FAILURES ===================================
_________________ ColNamesTests.test_CheckCaseInConverterName __________________

self = <test_types.ColNamesTests testMethod=test_CheckCaseInConverterName>

    def test_CheckCaseInConverterName(self):
>       self.cur.execute("select 'other' as \"x [b1b1]\"")

src/test/test_types.py:283:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
src/pyrqlite/cursors.py:164: in execute
    parse_colnames=self.connection.parse_colnames)
src/pyrqlite/extensions.py:128: in _convert_to_python
    value = value.decode('base64')
../../../.virtualenvs/sandbox/lib/python2.7/encodings/base64_codec.py:42: in base64_decode
    output = base64.decodestring(input)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

s = 'other'

    def decodestring(s):
        """Decode a string."""
>       return binascii.a2b_base64(s)
E       Error: Incorrect padding

/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/base64.py:321: Error
____________ BinaryConverterTests.test_CheckBinaryInputForConverter ____________

self = <test_types.BinaryConverterTests testMethod=test_CheckBinaryInputForConverter>

    def test_CheckBinaryInputForConverter(self):
        testdata = b"abcdefg" * 10
        compressed = zlib.compress(testdata)
>       result = self.con.execute('select ? as "x [bin]"', (compressed,)).fetchone()[0]

src/test/test_types.py:381:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
src/pyrqlite/connections.py:100: in execute
    return cursor.execute(*args, **kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <pyrqlite.cursors.Cursor object at 0x10ef6c550>
operation = 'select \'x\x9cKLJNIMKO\xa4\x8c\x02\x00\xca\x0f\x1bY\' as "x [bin]"'
parameters = ('x\x9cKLJNIMKO\xa4\x8c\x02\x00\xca\x0f\x1bY',)

    def execute(self, operation, parameters=None):

        if parameters:
            operation = self._substitute_params(operation, parameters)

        command = self._get_sql_command(operation)
        if command == 'SELECT':
            payload = self._request("GET",
                                    "/db/query?" + urlencode({'q': operation}))
        else:
            payload = self._request("POST", "/db/execute?transaction",
                                    headers={'Content-Type': 'application/json'}, body=json.dumps([operation]))

        last_insert_id = None
        rows_affected = -1
        payload_rows = {}
        try:
            results = payload["results"]
        except KeyError:
            pass
        else:
            rows_affected = 0
            for item in results:
                if 'error' in item:
                    logging.error(json.dumps(item))
>                   raise Error(json.dumps(item))
E                   Error: {"error": "unrecognized token: \"'x\ufffdKLJNIMKO\ufffd\ufffd\u0002\""}

src/pyrqlite/cursors.py:120: Error
----------------------------- Captured stderr call -----------------------------
ERROR:root:{"error": "unrecognized token: \"'x\ufffdKLJNIMKO\ufffd\ufffd\u0002\""}
===================== 2 failed, 30 passed in 0.56 seconds ======================

[alanjds]

Question: on "select 'other' as \"x [b1b1]\"", why is "other" not answered as base64, as the other non-standard types answers?

[otoolep]

@alanjds -- would you mind opening an issue in the rqlite repo for your question?

[otoolep]

@alanjds -- when you say it's better than before (I don't doubt you) do you mean there is more functionality, and more tests pass?

Also, should we skip the tests that are failing?

[alanjds]

Oops, just forgot to put a default on connections, unless all tests will fail.

[alanjds]

1. No problem. Will cross-post on rqlite/rqlite repo.

2. I mean that, before the 1st change to pyrqlite besides adding tests, it only handled char and integer values correctly.

When I saw a DB-API driver for a distributed version of SQLite then I expected it to be feature-compatible with some caveats, not to implement just the minimal API to be considered PEP249-compliant. Be clear that it is not a rant, just expectations mismatch.

Then I updated the driver to be as compliant as possible with the standard sqlite3 API, that I did not knew to be not standard. It is non-standard because have extensions to work around sqlite3 lack of features like BOOL type, etc. The first time I expected pyrqlite to be an almost drop-in replacement of sqlite3, and this PR is the far I could came to fulfil this expectation for the next guy trying.

3. Maybe could just skip this 2 tests, but I advise not to. As non-ascii column names are tested and working, seems to me that rqlite server does not implement correctly the type conversions for values "verbatim-declared" on the statement with custom types declared. Seems that custom types are always returned as base64, but the 1st test exposes an exception for this rule. The 2nd test is even stranger as column names are allowed non-ascii chars. Both tests should be fixed on rqlite server instead of passed, in my opinion.