api: native crud module support #264

GRISHNOV · 2022-11-15T20:55:26Z

Adds native api support for crud module to use it from a connection object.

Below there are examples of using the api.

>>> import tarantool
>>> from tarantool.error import CrudModuleError, CrudModuleManyError, DatabaseError
>>> conn = tarantool.Connection(host='localhost',port=3301)

>>> conn.crud_
    conn.crud_count(                conn.crud_insert(               conn.crud_insert_object_many(   
    conn.crud_min(                  conn.crud_replace_object(       conn.crud_stats(                
    conn.crud_unflatten_rows(       conn.crud_upsert_many(          conn.crud_delete(               
    conn.crud_insert_many(          conn.crud_len(                  conn.crud_replace(              
    conn.crud_replace_object_many(  conn.crud_storage_info(         conn.crud_update(               
    conn.crud_upsert_object(        conn.crud_get(                  conn.crud_insert_object(        
    conn.crud_max(                  conn.crud_replace_many(         conn.crud_select(               
    conn.crud_truncate(             conn.crud_upsert(               conn.crud_upsert_object_many(

Inserting data via crud:

# Insert without exception:
>>> res = conn.crud_insert('tester', (3500,300,'Rob'))
>>> res
<tarantool.crud.CrudResult object at 0x11a56e320>
>>> res.
res.metadata  res.rows
>>> res.rows
[[3500, 300, 'Rob']]
>>> res.metadata
[{'name': 'id', 'type': 'unsigned'}, {'name': 'bucket_id', 'type': 'unsigned'}, {'name': 'name', 'type': 'string'}]

# Insert with exception (duplicate key exists):
>>> try:
...     res = conn.crud_insert('tester', (3500,300,'Rob'))
... except CrudModuleError as e:
...     exc_crud = e
... 
>>> exc_crud
CrudModuleError(0, 'Failed to insert: Duplicate key exists in unique index "primary_index" in space "tester" with old tuple - [3500, 300, "Rob"] and new tuple - [3500, 300, "Rob"]')
>>> exc_crud.extra_info_error
<tarantool.crud.CrudError object at 0x10a276950>
>>> exc_crud.extra_info_error.
exc_crud.extra_info_error.class_name  exc_crud.extra_info_error.err         exc_crud.extra_info_error.file        exc_crud.extra_info_error.line        exc_crud.extra_info_error.str       
>>> exc_crud.extra_info_error.class_name
'InsertError'
>>> exc_crud.extra_info_error.str
'InsertError: Failed to insert: Duplicate key exists in unique index "primary_index" in space "tester" with old tuple - [3500, 300, "Rob"] and new tuple - [3500, 300, "Rob"]'

# In case of batch operation (*_many), CrudModuleManyError exception contains both result and errors (if there is a problem with at least one row).
>>> try:
...     res = conn.crud_insert_object_many('tester', ({'id':3,'bucket_id':100,'name':'Ann'}, {'id':4,'bucket_id':100,'name':'Sam'}), {'timeout':100, 'rollback_on_error':False})
... except CrudModuleManyError as e:
...     exc_crud = e
... 
>>> exc_crud
CrudModuleManyError(0, 'Got multiple errors, see errors_list')
>>> exc_crud.success_list # some of the rows were inserted.
<tarantool.crud.CrudResult object at 0x11a56f310>
>>> exc_crud.success_list.rows
[[1, 100, 'Bob'], [2, 100, 'Rob']]
>>> exc_crud.errors_list # some of the rows were not inserted.
[<tarantool.crud.CrudError object at 0x11a56e9e0>, <tarantool.crud.CrudError object at 0x11a56f490>]
>>> exc_crud.errors_list[0].str
'CallError: Failed for 037adb3a-b9e3-4f78-a6d1-9f0cdb6cbefc: Function returned an error: Duplicate key exists in unique index "primary_index" in space "tester" with old tuple - [3500, 300, "Rob"] and new tuple - [3500, 100, "Mike"]'
>>> exc_crud.errors_list[1].str
'InsertManyError: Failed to flatten object: FlattenError: Object is specified in bad format: FlattenError: Unknown field "second_name" is specified'
    
# If there are no problems with any rows, the entire response will be contained in the res variable.
>>> res = conn.crud_insert_object_many('tester', ({'id':3,'bucket_id':100,'name':'Ann'}, {'id':4,'bucket_id':100,'name':'Sam'}), {'timeout':100, 'rollback_on_error':False})
>>> res.rows
[[3, 100, 'Ann'], [4, 100, 'Sam']]

Crud not found on the router:

>>> try:
...     res = conn.crud_insert('tester', (22221,300,'Rob'))
... except DatabaseError as e:
...     exc_db = e
... 
 >>> exc_db
DatabaseError(33, "Ensure that you're calling crud.router and user has sufficient grants")
>>> exc_db.extra_info
BoxError(type='ClientError', file='/tmp/tarantool-20221003-6335-edruh3/tarantool-2.10.3/src/box/lua/call.c', line=112, message="Procedure 'crud.insert' is not defined", errno=0, errcode=33, fields=None, prev=None)

Select and unflatten_rows via crud:

>>> try:
...     res = conn.crud_select('tester', {}, {'first':2})
... except CrudModuleError as e:
...     exc_crud = e
... 
>>> res
<tarantool.crud.CrudResult object at 0x10a276d10>
>>> res.
res.metadata  res.rows      
>>> res.rows
[[1, 100, 'Mike'], [2, 100, 'Mike']]
>>> res.metadata
[{'name': 'id', 'type': 'unsigned'}, {'name': 'bucket_id', 'type': 'unsigned'}, {'name': 'name', 'type': 'string'}]
>>> r = conn.crud_unflatten_rows(res.rows, res.metadata)
>>> r
[{'id': 1, 'bucket_id': 100, 'name': 'Mike'}, {'id': 2, 'bucket_id': 100, 'name': 'Mike'}]

Truncate and len via crud:

>>> try:
...     res = conn.crud_len('tester')
... except CrudModuleError as e:
...     exc_crud = e
>>> res
26
>>> try:
...     res = conn.crud_truncate('tester')
... except CrudModuleError as e:
...     exc_crud = e
>>> res
True

Encoding=None in Connection:

>>> conn = tarantool.Connection(host='localhost',port=3301,encoding=None)

>>> try:
...     res = conn.crud_insert('tester', (3500,300,'Rob'))
... except CrudModuleError as e:
...     exc_crud = e
... 
>>> res
<tarantool.crud.CrudResult object at 0x1188a7fd0>
>>> res.
res.metadata  res.rows      
>>> res.metadata
[{b'name': b'id', b'type': b'unsigned'}, {b'name': b'bucket_id', b'type': b'unsigned'}, {b'name': b'name', b'type': b'string'}]
>>> res.rows
[[3500, 300, b'Rob']]

Closes #205

DifferentialOrange

Thank you for your draft! The task it much trickier that I initially have expected. I gave it some thought (and recalled some crud behavior) and I think that we should try to rework some things. I'll summarize my comments here.

Try to get rid of class Crud.
Try to work with calls only.
Do not expose configuration handles.
Raise errors as exceptions.
Optimize utils usage.

I see that the first one is a requirement to be able to use conn.crud.method, but is such approach really widespread in Python? I'm not well-experienced myself, but I think that I have seen providing many handles from a single place more often than using complex hierarchy. Maybe it's worth to study this question a little by using some Python modules as an example (and if you still decide to use conn.crud.method, maybe you'll see some nice implementation approaches).

It is highly likely that I have forgot to mention something important in this review, so feel free to ask me any additional questions here or in personal chat.

tarantool/crud.py

GRISHNOV · 2022-11-22T20:32:26Z

Thank you for your feedback!

I have made most of the edits (with the exception of the pairs implementation).
If the current API is suitable, I'll move on to writing documentation and tests

DifferentialOrange · 2022-11-23T09:34:49Z

>>> exc_crud.res is None # use only in case of *_many

I don't get this comment.

GRISHNOV · 2022-11-23T09:40:35Z

>>> exc_crud.res is None # use only in case of *_many

I don't get this comment.

I mean, in exc_crud.res, the result can only appear when the *_many() is called (when part of the batch is successful and part is not), in other cases exc_crud.res will always be None

DifferentialOrange

Thank you! I think current API is fine and we may continue to work with it.

tarantool/connection.py

tarantool/error.py

GRISHNOV · 2022-11-29T13:34:55Z

Thank you!
At the moment, I have implemented tests and documentation, and removed the draft status from the PR

DifferentialOrange

Thank you for the update!

See comments below. The main drawback of the current version is that tests are actually not run on CI, so I think you would be interested to fix it first and then fix everything that fails on CI.

There are some things that maybe are not covered yet, but there should be enough work for now.

docs/source/quick-start.rst

test/suites/test_crud.py

tarantool/connection.py

DifferentialOrange

Sorry for the long response. I've thought that error is less trickier, I've proposed the solution.

.github/workflows/packing.yml

.github/workflows/testing.yml

DifferentialOrange · 2022-12-13T15:36:03Z

tarantool/error.py

+            self.errs = args[1]
+        else:
+            # Sets tarantool.crud.CrudError object.
+            self.err = args[1]


There is already an exception API in the connector. Receiving a different types of exceptions may be confusing, especially if they uses the same fields with different meanings.

DifferentialOrange · 2022-12-13T15:36:16Z

tarantool/error.py

+            self.errs = args[1]
+        else:
+            # Sets tarantool.crud.CrudError object.
+            self.err = args[1]


For example

>>> c.select('myspace') Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/home/georgymoiseev/Development/github/tarantool/tarantool-python/tarantool/connection.py", line 1873, in select space_name = self.schema.get_space(space_name).sid File "/home/georgymoiseev/Development/github/tarantool/tarantool-python/tarantool/schema.py", line 216, in get_space return self.fetch_space(space) File "/home/georgymoiseev/Development/github/tarantool/tarantool-python/tarantool/schema.py", line 243, in fetch_space raise SchemaError(errmsg) tarantool.error.SchemaError: There's no space with name 'myspace' >>> c.crud_select('myspace') Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/home/georgymoiseev/Development/github/tarantool/tarantool-python/tarantool/connection.py", line 2550, in crud_select raise CrudModuleError(None, CrudError(crud_resp[1])) tarantool.error.CrudModuleError: (None, <tarantool.crud.CrudError object at 0x7f4864a525f0>)

DifferentialOrange · 2022-12-13T16:35:42Z

tarantool/crud.py

+    info = ". Ensure that you're calling crud.router and user has sufficient grants"
+    e.message = e.message + info
+    e.extra_info.message = e.extra_info.message + info
+    e.args = (e.args[0], e.args[1] + info)


Second, let's never mess with args.

>>> class MyError(Exception): ... def __init__(self, *args): ... pass ... >>> MyError(1, 2) MyError(1, 2) >>> e = MyError(1, 2) >>> e. e.args e.with_traceback( >>> e.args (1, 2)

args exception attribute is essentially a copy of "what data was used to raise exception with".

Either you forgot to commit it or lost it while doing other reworks, but it's still in call_crud

Done, deleted

tarantool/crud.py

.github/workflows/testing.yml

DifferentialOrange · 2022-12-20T07:48:58Z

tarantool/error.py

+    .. _crud: https://github.com/tarantool/crud
+    """
+
+    def __init__(self, *args):


Since there are always two positional arguments, why use *args instead of def __init__(self, success, error):

Done, now using __init__(self, success, error):

DifferentialOrange · 2022-12-20T07:50:10Z

tarantool/error.py

+        """
+
+        # Sets tarantool.crud.CrudResult object.
+        self.success = args[0]


There is no success other than nil for non-many requests, isn't it?

Done, deleted

DifferentialOrange · 2022-12-20T07:51:52Z

tarantool/error.py

+            self.errs = args[1]
+        else:
+            # Sets tarantool.crud.CrudError object.
+            self.err = args[1]


It's not yet done: CrudModuleError inherits DatabaseError, but it never calls its __init__ (like here:

tarantool-python/tarantool/error.py

Line 262 in 1154b3d

super(NetworkError, self).__init__(0, self.message)

)

DifferentialOrange · 2022-12-20T07:52:16Z

tarantool/error.py

+            self.errs = args[1]
+        else:
+            # Sets tarantool.crud.CrudError object.
+            self.err = args[1]


Same about CrudModuleManyError

DifferentialOrange · 2022-12-20T07:54:46Z

tarantool/crud.py

+    """
+
+
+def wrapCrudUndefinedProcedureError(e: DatabaseError):


There is no wrapCrudUndefinedProcedureError now, but there is also no exceptions chaining yet and you still overwrite exception fields.

DifferentialOrange · 2022-12-20T07:55:47Z

tarantool/crud.py

+    info = ". Ensure that you're calling crud.router and user has sufficient grants"
+    e.message = e.message + info
+    e.extra_info.message = e.extra_info.message + info
+    e.args = (e.args[0], e.args[1] + info)


Either you forgot to commit it or lost it while doing other reworks, but it's still in call_crud

DifferentialOrange · 2022-12-20T07:59:05Z

tarantool/connection_pool.py

+
+    def crud_insert(self, space_name, values, opts={}, *, mode=Mode.ANY):
+        """
+        Execute an CRUD_INSERT request on the pool server: 


SUCH_STYLE in other documentation strings refer to IPROTO_REQUEST codes (omitting the IPROTO_ part), so writing CRUD_INSERT here and below may be a bit confusing.

You may ignore this comment if you wish.

Done, rewritten without uppercase

GRISHNOV · 2022-12-21T14:27:58Z

Thank you for your feedback! I have updated the PR description and tried to answer all the comments on the code review

DifferentialOrange

Thank you for your updates! Let's resolve a couple of minor comments and I'll merge this one.

tarantool/crud.py

tarantool/error.py

docs/source/quick-start.rst

Adds native api support for crud module [1] to use it from a connection object. 1. github.com/tarantool/crud Closes #205

Follows #264

Docker debian/ubuntu images are sudoless. Follows #264

Set valid package manager for fedora. Fix using apt for docker: debian/ubuntu containers are sudoless. Follows #264

tarantool-python static analysis was introduced in [1], CRUD API was introduced in [2]. 1. tarantool/tarantool-python#289 2. tarantool/tarantool-python#264

* connectors: remove queue-python tarantool/queue-python [1] last commit was in 2016. Module does not support Tarantool 1.6+. 1. https://github.com/tarantool/queue-python * connectors: update aiotarantool status shveenkov/aiotarantool last commit was in 2019. 1. https://github.com/shveenkov/aiotarantool * connectors: remove gtarantool from comparison shveenkov/gtarantool [1] last commit was in 2016. It is useless for modern Tarantool users. 1. https://github.com/shveenkov/gtarantool * connectors: update comparison table tarantool-python static analysis was introduced in [1], CRUD API was introduced in [2]. 1. tarantool/tarantool-python#289 2. tarantool/tarantool-python#264

GRISHNOV requested a review from DifferentialOrange November 16, 2022 08:55

DifferentialOrange requested changes Nov 16, 2022

View reviewed changes

GRISHNOV force-pushed the igrishnov/gh-205-native-crud-api-support branch from ad145c4 to 6343199 Compare November 22, 2022 18:18

GRISHNOV requested a review from DifferentialOrange November 22, 2022 20:32

DifferentialOrange reviewed Nov 23, 2022

View reviewed changes

tarantool/connection.py Show resolved Hide resolved

tarantool/connection.py Show resolved Hide resolved

tarantool/connection.py Outdated Show resolved Hide resolved

tarantool/error.py Outdated Show resolved Hide resolved

DifferentialOrange mentioned this pull request Nov 23, 2022

Type hints and type check #266

Open

GRISHNOV force-pushed the igrishnov/gh-205-native-crud-api-support branch 4 times, most recently from 3ed26c5 to cc04dc4 Compare November 29, 2022 12:40

GRISHNOV marked this pull request as ready for review November 29, 2022 13:34

GRISHNOV requested a review from DifferentialOrange November 29, 2022 13:35

DifferentialOrange reviewed Nov 29, 2022

View reviewed changes

GRISHNOV force-pushed the igrishnov/gh-205-native-crud-api-support branch 13 times, most recently from 20f6c01 to da98380 Compare December 5, 2022 21:49

GRISHNOV force-pushed the igrishnov/gh-205-native-crud-api-support branch 3 times, most recently from b5de7af to 1611fc9 Compare December 12, 2022 09:21

GRISHNOV requested a review from DifferentialOrange December 12, 2022 09:27

DifferentialOrange reviewed Dec 13, 2022

View reviewed changes

GRISHNOV force-pushed the igrishnov/gh-205-native-crud-api-support branch from 1611fc9 to 3455edc Compare December 19, 2022 14:16

GRISHNOV requested a review from DifferentialOrange December 19, 2022 14:23

DifferentialOrange reviewed Dec 20, 2022

View reviewed changes

GRISHNOV force-pushed the igrishnov/gh-205-native-crud-api-support branch from 3455edc to 2e88041 Compare December 21, 2022 13:56

GRISHNOV requested a review from DifferentialOrange December 21, 2022 14:28

DifferentialOrange approved these changes Dec 22, 2022

View reviewed changes

tarantool/crud.py Outdated Show resolved Hide resolved

tarantool/crud.py Outdated Show resolved Hide resolved

tarantool/error.py Outdated Show resolved Hide resolved

docs/source/quick-start.rst Show resolved Hide resolved

api: native crud module support

832ab10

Adds native api support for crud module [1] to use it from a connection object. 1. github.com/tarantool/crud Closes #205

GRISHNOV force-pushed the igrishnov/gh-205-native-crud-api-support branch from 2e88041 to 832ab10 Compare December 22, 2022 08:17

DifferentialOrange approved these changes Dec 22, 2022

View reviewed changes

DifferentialOrange merged commit 4a791dc into master Dec 22, 2022

DifferentialOrange deleted the igrishnov/gh-205-native-crud-api-support branch December 22, 2022 08:45

DifferentialOrange added a commit that referenced this pull request Dec 22, 2022

ci: fix package manager for fedora

07b936d

Follows #264

DifferentialOrange added a commit that referenced this pull request Dec 22, 2022

ci: fix using apt for docker

9417581

Docker debian/ubuntu images are sudoless. Follows #264

DifferentialOrange mentioned this pull request Dec 22, 2022

Fix packing in Github Actions #272

Merged

DifferentialOrange added a commit that referenced this pull request Dec 22, 2022

ci: fix crud install

016203c

Set valid package manager for fedora. Fix using apt for docker: debian/ubuntu containers are sudoless. Follows #264

DifferentialOrange added a commit that referenced this pull request Dec 22, 2022

ci: fix crud install

e41ed24

Set valid package manager for fedora. Fix using apt for docker: debian/ubuntu containers are sudoless. Follows #264

DifferentialOrange added a commit that referenced this pull request Dec 22, 2022

ci: fix crud install

01e7161

Set valid package manager for fedora. Fix using apt for docker: debian/ubuntu containers are sudoless. Follows #264

DifferentialOrange added a commit that referenced this pull request Dec 22, 2022

ci: fix crud install

2e08b48

Set valid package manager for fedora. Fix using apt for docker: debian/ubuntu containers are sudoless. Follows #264

DifferentialOrange added a commit that referenced this pull request Dec 22, 2022

ci: fix crud install

4f6b965

Set valid package manager for fedora. Fix using apt for docker: debian/ubuntu containers are sudoless. Follows #264

DifferentialOrange added a commit that referenced this pull request Dec 22, 2022

ci: fix crud install

331daea

Set valid package manager for fedora. Fix using apt for docker: debian/ubuntu containers are sudoless. Follows #264

DifferentialOrange mentioned this pull request Feb 13, 2023

api: add CRUD module support tarantool/go-tarantool#263

Merged

24 tasks

api: native crud module support #264

api: native crud module support #264

Uh oh!

Conversation

GRISHNOV commented Nov 15, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Inserting data via crud:

Crud not found on the router:

Select and unflatten_rows via crud:

Truncate and len via crud:

Encoding=None in Connection:

Uh oh!

DifferentialOrange left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

GRISHNOV commented Nov 22, 2022

Uh oh!

DifferentialOrange commented Nov 23, 2022

Uh oh!

GRISHNOV commented Nov 23, 2022

Uh oh!

DifferentialOrange left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

GRISHNOV commented Nov 29, 2022

Uh oh!

DifferentialOrange left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

DifferentialOrange left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

GRISHNOV commented Nov 15, 2022 •

edited

Loading

DifferentialOrange left a comment •

edited

Loading