Skip to content

gh-69093: Add incremental I/O to blobs support in sqlite3 #30356

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 83 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
83 commits
Select commit Hold shift + click to select a range
4fa0e91
support BLOB incremental I/O in sqlite module
palaviv Feb 24, 2017
3fe9108
Note that blob size cannot be changed using the blob object
palaviv Feb 25, 2017
865c1c8
Use assertRaises in tests
palaviv Feb 25, 2017
788fd54
Fix doc error
palaviv Feb 25, 2017
a1361e5
blob support sequence protocol
palaviv Mar 4, 2017
5aedfba
Calculate blob length once at creation
palaviv Mar 4, 2017
db6ef32
Add initial Doc for sequence protocol
palaviv Mar 4, 2017
219f4cb
Don't support blob operation
palaviv Apr 18, 2018
6dafe0e
move news entry to blurb
palaviv Apr 18, 2018
ffac901
Add blob to PCBuild
palaviv Apr 18, 2018
3475fa1
Update version
palaviv May 8, 2019
f6015ff
Fix memory leak
palaviv May 8, 2019
01e526c
Update version
palaviv Jul 29, 2020
354bebf
Fix CR comments in documentation and testing
palaviv Aug 3, 2020
9709456
Fix CR comments in code
palaviv Aug 3, 2020
e6e5099
Make readonly and dbname keyword arguements only
palaviv Aug 3, 2020
e9a8080
Fix more CR comments
palaviv Aug 3, 2020
d4fb1b5
Add more indicative error on write bigger then blob length
palaviv Aug 3, 2020
a7288f9
Merge branch 'main' into sqlite-blob
Sep 10, 2021
525a9c3
Adapt sqlite3.Connection.open_blob() to AC
Sep 10, 2021
2f65cc8
Adapt sqlite.Blob to AC
Sep 10, 2021
0dd047f
PEP 7 and normalised naming
Sep 10, 2021
d34a77b
Use Py_NewRef and Py_IsNone
Sep 10, 2021
9d55705
Harden blob_open()
Sep 10, 2021
d47ea17
Move blob init to blob_open()
Sep 10, 2021
e07a116
initialise rc in blob_ass_subscript()
Sep 10, 2021
982c812
Improve blob.seek() parameter naming
Sep 10, 2021
e92495b
Adapt to heap types and allow calling close multiple times
Sep 10, 2021
8f2ce8a
Consolidate tests
Sep 10, 2021
0a87520
Naming: read_length => length
Sep 10, 2021
32132cf
Wrap error handling in support function
Sep 10, 2021
287803e
Add blob seek position markers as constants
Sep 10, 2021
0fc5a39
Adjust SQLITE_ABORT error message
Sep 10, 2021
cd0bde1
Remove unneeded sqlite3_blob_close() and fix GC tracking
Sep 10, 2021
cf7e15e
Use close_blob() in pysqlite_close_all_blobs()
Sep 10, 2021
7e77217
Refactor write/read
Sep 10, 2021
c57e45a
Sipmlify __exit__
Sep 10, 2021
dd76f72
Use new slice API
Sep 10, 2021
237684e
Refactor very large functions
Sep 11, 2021
58905e8
Simplify subscript slice
Sep 11, 2021
9d69fca
Consolidate more tests
Sep 11, 2021
285bb3d
Use supplied offset in inner_read()
Sep 11, 2021
2f3051a
Simplify test
Sep 11, 2021
fd7c311
Simplify assign subscript slice
Sep 11, 2021
0644e87
Early error checking, and use PyBytes_AS_STRING() when possible
Sep 11, 2021
ced431a
Expand test suite
Sep 11, 2021
2dae1b9
Remove unneeded parts of sequence protocol
Sep 11, 2021
e8fa47e
Normalise error messages
Sep 11, 2021
6441668
Improve error message/type for to large subscript assignment
Sep 11, 2021
03d2152
Normalise naming: write_inner => inner_write
Sep 11, 2021
9360e62
Adjust comment
Sep 11, 2021
8eb16f7
Move blob_seterror() closer to where it's first used
Sep 11, 2021
411d07e
Fetch state from connection in check_blob()
Sep 11, 2021
8149e36
Remove unused declaration
Sep 11, 2021
56b4caa
Doc adjustments
Sep 11, 2021
afdeb2e
Add What's New and adjust NEWS
Sep 11, 2021
b12219f
Use sqlite3_blob_bytes() iso. storing length on blob object
Sep 12, 2021
32a21d8
Merge branch 'main' into sqlite-blob
Sep 12, 2021
36b0ca1
Also remove length from blob_open()
Sep 12, 2021
3d91705
Add get subscript index helper
Sep 12, 2021
ceee315
Expand test suite
Sep 12, 2021
44f4cd3
Merge branch 'main' into sqlite-blob
Sep 21, 2021
220b576
Merge branch 'main' into sqlite-blob
Nov 17, 2021
d9b5cdf
Format docs with linewidth 80
Nov 17, 2021
aaa2721
Improve tests
Nov 17, 2021
18e5118
Merge branch 'main' into sqlite-blob
Jan 2, 2022
8f685ba
Add clinic_state() stub. (Currently not used)
Jan 2, 2022
96df661
Visit/clear blob type during module traverse/clear
Jan 2, 2022
97d12a8
Harden some tests, simplify others
Jan 2, 2022
2e63b3e
Simplify example
Jan 3, 2022
be27747
Update docs
Jan 3, 2022
5add365
Simplify tests
Jan 3, 2022
bacf087
Safe close
Jan 3, 2022
5ff202d
Condense a comment
Jan 3, 2022
1fa5901
Make sure we catch int overflow for all types of writes
Jan 3, 2022
16f4d0d
Merge branch 'main' into sqlite-blob-all
Jan 18, 2022
7aec288
Fix typo in test name
Jan 18, 2022
aaaf7ab
Improve test
Jan 18, 2022
f9e65c0
Improve tests
Jan 18, 2022
6a5c864
Always check length in inner write
Jan 18, 2022
ea1045c
Fix missing array sentinel
Jan 18, 2022
2f848d9
Match apsw's API open_blob => blobopen
Jan 19, 2022
8d906bc
Fix overflow test on win x64
Jan 19, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions Doc/includes/sqlite3/blob.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
import sqlite3

con = sqlite3.connect(":memory:")
con.execute("create table test(blob_col blob)")
con.execute("insert into test(blob_col) values (zeroblob(10))")

blob = con.blobopen("test", "blob_col", 1)
blob.write(b"Hello")
blob.write(b"World")
blob.seek(0)
print(blob.read()) # will print b"HelloWorld"
blob.close()
12 changes: 12 additions & 0 deletions Doc/includes/sqlite3/blob_with.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
import sqlite3

con = sqlite3.connect(":memory:")

con.execute("create table test(blob_col blob)")
con.execute("insert into test(blob_col) values (zeroblob(10))")

with con.blobopen("test", "blob_col", 1) as blob:
blob.write(b"Hello")
blob.write(b"World")
blob.seek(0)
print(blob.read()) # will print b"HelloWorld"
73 changes: 73 additions & 0 deletions Doc/library/sqlite3.rst
Original file line number Diff line number Diff line change
Expand Up @@ -394,6 +394,20 @@ Connection Objects
supplied, this must be a callable returning an instance of :class:`Cursor`
or its subclasses.

.. method:: blobopen(table, column, row, /, *, readonly=False, name="main")

On success, a :class:`Blob` handle to the :abbr:`BLOB (Binary Large
OBject)` located in row *row*, column *column*, table *table* in database
*name* will be returned. When *readonly* is :const:`True` the blob is
opened without write permissions.

.. note::

The BLOB size cannot be changed using the :class:`Blob` class. Use the
SQL function ``zeroblob`` to create a blob with a fixed size.

.. versionadded:: 3.11

.. method:: commit()

This method commits the current transaction. If you don't call this method,
Expand Down Expand Up @@ -1021,6 +1035,65 @@ Exceptions
transactions turned off. It is a subclass of :exc:`DatabaseError`.


.. _sqlite3-blob-objects:

Blob Objects
------------

.. versionadded:: 3.11

.. class:: Blob

A :class:`Blob` instance can read and write the data in a :abbr:`BLOB
(Binary Large OBject)`. The :class:`Blob` class implements the file and
mapping protocols.

.. method:: Blob.close()

Close the BLOB.

The BLOB will be unusable from this point forward. An
:class:`~sqlite3.Error` (or subclass) exception will be raised if any
further operation is attempted with the BLOB.

.. method:: Blob.__len__()

Return the BLOB size as length in bytes.

.. method:: Blob.read(length=-1, /)

Read *length* bytes of data from the BLOB at the current offset position.
If the end of the BLOB is reached we will return the data up to end of
file. When *size* is not specified or is negative, :meth:`~Blob.read`
will read till the end of the BLOB.

.. method:: Blob.write(data, /)

Write *data* to the BLOB at the current offset. This function cannot
change the BLOB length. Writing beyond the end of the blob will result in
an exception being raised.

.. method:: Blob.tell()

Return the current access position of the BLOB.

.. method:: Blob.seek(offset, origin=sqlite3.BLOB_SEEK_START, /)

Set the current access position of the BLOB to *offset*. The *origin*
argument defaults to :data:`os.SEEK_SET` (absolute BLOB positioning).
Other values for *origin* are :data:`os.SEEK_CUR` (seek relative to the
current position) and :data:`os.SEEK_END` (seek relative to the BLOB’s
end).

:class:`Blob` example:

.. literalinclude:: ../includes/sqlite3/blob.py

A :class:`Blob` can also be used as a :term:`context manager`:

.. literalinclude:: ../includes/sqlite3/blob_with.py


.. _sqlite3-types:

SQLite and Python types
Expand Down
4 changes: 4 additions & 0 deletions Doc/whatsnew/3.11.rst
Original file line number Diff line number Diff line change
Expand Up @@ -301,6 +301,10 @@ sqlite3
Instead we leave it to the SQLite library to handle these cases.
(Contributed by Erlend E. Aasland in :issue:`44092`.)

* Add :meth:`~sqlite3.Connection.blobopen` to :class:`sqlite3.Connection`.
:class:`sqlite3.Blob` allows incremental I/O operations on blobs.
(Contributed by Aviv Palivoda and Erlend E. Aasland in :issue:`24905`)


sys
---
Expand Down
Loading