Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for pyexasol part 2/2 #320

Merged
merged 86 commits into from
May 24, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
86 commits
Select commit Hold shift + click to select a range
0fa9f0e
Add hooks for websocket based exasol dialect
Nicoretti Mar 23, 2023
4dd3517
Implement basic parsing of connection arguments
Nicoretti Mar 23, 2023
c5f0a53
Fix parameterized queries
Nicoretti Mar 23, 2023
14e2617
Add websocket dialect/connector to ci/cd matrix
Nicoretti Mar 23, 2023
00e64b6
Fix websocket dbapi signature and default implementation
Nicoretti Mar 28, 2023
69b7e88
Disable statement cache for websocket dialect
Nicoretti Mar 28, 2023
7fbec53
Fix CI/CD builds for websocket based dialect
Nicoretti Mar 28, 2023
3153755
Fix Big Integer conversion edge case for websocket dialect
Nicoretti Mar 30, 2023
fac4ceb
Update dependencies
Nicoretti Mar 30, 2023
155915f
Restructure the dbapi2 module of the websocket driver
Nicoretti Apr 3, 2023
21519e3
Move dbapi exceptions into seperate module
Nicoretti Apr 3, 2023
dd9e827
Move Connection and Cursor protocol definition different module
Nicoretti Apr 3, 2023
82a5f51
Move type definitions and type conversions to different module
Nicoretti Apr 3, 2023
6f73d2f
Improve readability of section seperators
Nicoretti Apr 3, 2023
199287e
Move Connection and Curosr implementation in different modules
Nicoretti Apr 3, 2023
be40b26
Move comment to correct location
Nicoretti Apr 4, 2023
851cb7d
Update exasol/driver/websocket/_protocols.py
Nicoretti Apr 5, 2023
851c58d
Update exasol/driver/websocket/_protocols.py
Nicoretti Apr 5, 2023
734a28b
Update exasol/driver/websocket/_protocols.py
Nicoretti Apr 5, 2023
3042ba1
Update exasol/driver/websocket/_protocols.py
Nicoretti Apr 5, 2023
785165f
Update exasol/driver/websocket/_protocols.py
Nicoretti Apr 5, 2023
5bce87d
Update exasol/driver/websocket/_protocols.py
Nicoretti Apr 5, 2023
5fa79a1
Add todo note
Nicoretti Apr 5, 2023
562a5b3
Added justification for pylint suppressions
Nicoretti Apr 5, 2023
97165bb
Add justification for linter suppression
Nicoretti Apr 5, 2023
6f78db6
Update exasol/driver/websocket/_cursor.py
Nicoretti Apr 5, 2023
dbdf318
Update dependencies
Nicoretti Apr 24, 2023
b95d6ac
Fix sqlalchemy dialect conversions
Nicoretti Apr 24, 2023
b82e49c
Exasol Websocket DBAPI datatypes
Nicoretti Apr 24, 2023
7babb59
Apply simplifications
Nicoretti Apr 26, 2023
5a45570
Rework dbapi2 types
Nicoretti Apr 26, 2023
489632c
Add hooks for type conversions between pyexasol and dbapi2 cursor
Nicoretti Apr 27, 2023
0ad928e
Fix Date edge cases
Nicoretti Apr 27, 2023
c11b35c
Fix Datetime edge cases
Nicoretti Apr 27, 2023
bf5efaf
Add basic context manager functionality to default cursor
Nicoretti May 8, 2023
7984aa9
Added notes to context manager functions
Nicoretti May 8, 2023
472856c
Update dependencies
Nicoretti May 8, 2023
e59aaca
Add exception DBAPI 2 exception translation for executed statements
Nicoretti May 8, 2023
ab8cad7
Skip test for duplicate names in result set for websocket based driver
Nicoretti May 8, 2023
b51fb32
Fix DBAPI2 inconsistency for description attribute of the cursor
Nicoretti May 11, 2023
a081b26
Remove datatypes section
Nicoretti May 11, 2023
d9cb72b
Add notes regarding prepared statement issue
Nicoretti May 11, 2023
d947100
Fix unexpected auto commits (#336)
Nicoretti May 15, 2023
c7adafe
Prepare release 4.4.0 (#337)
Nicoretti May 16, 2023
8c24452
Adjust date in change log (#338)
Nicoretti May 16, 2023
5776bfd
Update dependencies
Nicoretti May 16, 2023
08c857c
Merge branch 'master' into features/#105-pyexasol-backend-part-2
Nicoretti May 16, 2023
6568c2e
Consolidate server version detection
Nicoretti May 16, 2023
4d4ae21
Adjust test_update_executemany test
Nicoretti May 16, 2023
c86e279
Update dependencies
Nicoretti May 19, 2023
addff14
Delete implementation notes
Nicoretti May 19, 2023
61333ae
Add support for enabeling and disabeling certificate validation
Nicoretti May 19, 2023
d8a1469
Fix tls settings in websocket driver
Nicoretti May 22, 2023
7f95539
Make sure certificate tests works for all exasol dialects
Nicoretti May 22, 2023
599ccbb
Adjust DBAPI tests
Nicoretti May 22, 2023
0f92058
Update README.md
Nicoretti May 22, 2023
3eb7c96
Update pylint rating
Nicoretti May 22, 2023
70253b9
Update dependencies
Nicoretti May 22, 2023
dcff405
Fix rst formatting error
Nicoretti May 23, 2023
89388e8
Disable remaing failing SQLA compliance tests
Nicoretti May 23, 2023
5ee6a76
Update dependencies
Nicoretti May 23, 2023
55c0a02
Update exasol/driver/websocket/_connection.py
Nicoretti May 23, 2023
c949009
Update exasol/driver/websocket/_connection.py
Nicoretti May 23, 2023
da085cb
Update exasol/driver/websocket/_connection.py
Nicoretti May 23, 2023
eadfdcb
Update README.rst
Nicoretti May 23, 2023
d878dcf
Fix documentatio build
Nicoretti May 23, 2023
8c620cc
Restore old skipping behavior for pyodbc tests
Nicoretti May 23, 2023
aeb4fdb
Remove outdated TODO
Nicoretti May 23, 2023
8d61ce9
Update README.rst
Nicoretti May 23, 2023
28779db
Update README.rst
Nicoretti May 23, 2023
c21886b
Update README.rst
Nicoretti May 23, 2023
79900f0
Update README.rst
Nicoretti May 23, 2023
d470541
Update changelog
Nicoretti May 23, 2023
8bf2f39
Add rational for not following dbapi behavior in regard of the rowcount
Nicoretti May 23, 2023
ed180fe
Update README.rst
Nicoretti May 23, 2023
2516712
Update README.rst
Nicoretti May 23, 2023
5372aee
Improve error message when accessing already closed cursor
Nicoretti May 23, 2023
cc1c367
Update README
Nicoretti May 23, 2023
5ca1c14
Add client name for wss connections
Nicoretti May 23, 2023
99cb8c7
Update README
Nicoretti May 23, 2023
36355db
Address code review feedback
Nicoretti May 23, 2023
85b2616
Fix unit tests
Nicoretti May 23, 2023
5d8f7e4
Update CHANGELOG.rst
Nicoretti May 24, 2023
a2d74d3
Update CHANGELOG.rst
Nicoretti May 24, 2023
37b8053
Update README.rst
Nicoretti May 24, 2023
9eb3e1e
Add client_version to connection parameters
Nicoretti May 24, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/workflows/ci-cd.yml
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,7 @@ jobs:
connector:
- pyodbc
- turbodbc
- websocket
Nicoretti marked this conversation as resolved.
Show resolved Hide resolved
exasol_version:
- 7.1.17
- 7.0.20
Expand Down
1 change: 1 addition & 0 deletions .github/workflows/ci.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,7 @@ jobs:
connector:
- pyodbc
- turbodbc
- websocket
exasol_version:
- 7.1.17
- 7.0.20
Expand Down
11 changes: 11 additions & 0 deletions CHANGELOG.rst
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,17 @@
Unreleased
==========

* Add Beta version of websocket based dialect

**🚨 Attention:**

This feature is currently in Beta, therefore it should not be used in production.
We also recommend to have a look into the known issues, before you start using it.

If you encounter any problem, please `create an issue <https://github.com/exasol/sqlalchemy-exasol/issues/new?assignees=&labels=bug&projects=&template=bug.md&title=%F0%9F%90%9E+%3CInsert+Title%3E>`_.
With your feedback, we will be able stabilize this feature more quickly.


.. _changelog-4.4.0:

4.4.0 — 2023-05-16
Expand Down
130 changes: 111 additions & 19 deletions README.rst
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ SQLAlchemy Dialect for EXASOL DB
:target: https://pycqa.github.io/isort/
:alt: Formatter - Isort

.. image:: https://img.shields.io/badge/pylint-5.8-yellow
.. image:: https://img.shields.io/badge/pylint-6.4-yellowgreen
:target: https://github.com/PyCQA/pylint
:alt: Pylint

Expand All @@ -49,17 +49,6 @@ How to get started
We assume you have a good understanding of (unix)ODBC. If not, make sure you
read their documentation carefully - there are lot's of traps 🪤 to step into.

Meet the system requirements
````````````````````````````

On Linux/Unix like systems you need:

- Python
- An Exasol DB (e.g. `docker-db <test_docker_image_>`_ or a `cloud instance <test_drive_>`_)
- The packages unixODBC and unixODBC-dev >= 2.2.14
- The Exasol `ODBC driver <odbc_driver_>`_
- The ODBC.ini and ODBCINST.ini configurations files setup

Turbodbc support
````````````````

Expand All @@ -72,6 +61,18 @@ Turbodbc support
`test/test_update.py <test/test_update.py>`_ for an example



Meet the system requirements
````````````````````````````

On Linux/Unix like systems you need:

- Python
- An Exasol DB (e.g. `docker-db <test_docker_image_>`_ or a `cloud instance <test_drive_>`_)
- The packages unixODBC and unixODBC-dev >= 2.2.14
- The Exasol `ODBC driver <odbc_driver_>`_
- The ODBC.ini and ODBCINST.ini configurations files setup

Setup your python project and install sqlalchemy-exasol
```````````````````````````````````````````````````````

Expand Down Expand Up @@ -109,18 +110,17 @@ The dialect supports two types of connection urls creating an engine. A DSN (Dat

.. list-table::

* - Type
- Example
* - DSN URL
- 'exa+pyodbc://USER:PWD@exa_test'
* - HOST URL
- 'exa+pyodbc://USER:PWD@192.168.14.227..228:1234/my_schema?parameter'
* - Type
- Example
* - DSN URL
- 'exa+pyodbc://USER:PWD@exa_test'
* - HOST URL
- 'exa+pyodbc://USER:PWD@192.168.14.227..228:1234/my_schema?parameter'

Features
++++++++

- SELECT, INSERT, UPDATE, DELETE statements
- you can even use the MERGE statement (see unit tests for examples)

Notes
+++++
Expand All @@ -139,3 +139,95 @@ Development & Testing
`````````````````````
See `developer guide`_

Websocket support
-----------------

.. attention::

The Websocket support is currently in Beta, therefore it should not be used in production.
We also recommend to have a look into the known issues, before you start using it.

If you encounter any issue, please `create an issue <https://github.com/exasol/sqlalchemy-exasol/issues/new?assignees=&labels=bug&projects=&template=bug.md&title=%F0%9F%90%9E+%3CInsert+Title%3E>`_.
With your feedback, we will be able to stabilize this feature more quickly.

What is Websocket support?
``````````````````````````
In the context of SQLA and Exasol, Websocket support means that an SQLA dialect
supporting the `Exasol Websocket Protocol <https://github.com/exasol/websocket-api>`_
is provided.

Using the websocket based protocol instead over ODBC will provide various advantages:

* Less System Dependencies
* Easier to use than ODBC based driver(s)
* Lock free metadata calls etc.
Nicoretti marked this conversation as resolved.
Show resolved Hide resolved

For further details `Why a Websockets API <https://github.com/exasol/websocket-api#why-a-websockets-api>`_.

Examples Usage(s)
`````````````````

.. code-block:: python

from sqla import create_engine

engine = create_engine("exa+websocket://sys:exasol@127.0.0.1:8888")
with engine.connect() as con:
...

.. code-block:: python

from sqla import create_engine

# ATTENTION:
# In terms of security it is NEVER a good idea to turn of certificate validation!!
# In rare cases it may be handy for non-security related reasons.
# That said, if you are not a 100% sure about your scenario, stick with the
# secure defaults.
# In most cases, having a valid certificate and/or configuring the truststore(s)
Nicoretti marked this conversation as resolved.
Show resolved Hide resolved
# appropriately is the best/correct solution.
engine = create_engine("exa+websocket://sys:exasol@127.0.0.1:8888?SSLCertificate=SSL_VERIFY_NONE")
with engine.connect() as con:
...

Supported Connection Parameters
```````````````````````````````
.. list-table::

* - Parameter
- Values
- Comment
* - ENCRYPTION
- Y, Yes, N, No
- Y or Yes Enable Encryption (TLS) default, N or No disable Encryption
* - SSLCertificate
- SSL_VERIFY_NONE
- Disable certificate validation


Known Issues
````````````

* Literal casts within prepared statements do not work
- :code:`INSERT INTO t (x) VALUES (CAST(? AS VARCHAR(50)));`
* Various conversions regarding float, decimals
- Certain scenarios still yield a :code:`string` type instead :code:`float` or :code:`decimal` type.
* Insert
- Insert multiple rows via prepared statements does not work in all cases
- Insert from SELECT does not work
* For some prepared statements, the wss protocol type conversion does not work properly
- Error messages usually state some JSON type mismatch, e.g.: '... getString: JSON value is not a string ...'
* Known failing tests of the SQLA compliance test suite
- FAILED test/integration/sqlalchemy/test_suite.py::CastTypeDecoratorTest_exasol+exasol_driver_websocket_dbapi2::test_special_type - sqlalchemy.exc.DBAPIError: (exasol.driver.websocket._errors.Error)
- FAILED test/integration/sqlalchemy/test_suite.py::ExistsTest_exasol+exasol_driver_websocket_dbapi2::test_select_exists - sqlalchemy.exc.DBAPIError: (exasol.driver.websocket._errors.Error)
- FAILED test/integration/sqlalchemy/test_suite.py::ExistsTest_exasol+exasol_driver_websocket_dbapi2::test_select_exists_false - sqlalchemy.exc.DBAPIError: (exasol.driver.websocket._errors.Error)
- FAILED test/integration/sqlalchemy/test_suite.py::InsertBehaviorTest_exasol+exasol_driver_websocket_dbapi2::test_empty_insert_multiple - sqlalchemy.exc.DBAPIError: (exasol.driver.websocket._errors.Error)
- ERROR test/integration/sqlalchemy/test_suite.py::InsertBehaviorTest_exasol+exasol_driver_websocket_dbapi2::test_empty_insert_multiple_teardown - ERROR
- FAILED test/integration/sqlalchemy/test_suite.py::InsertBehaviorTest_exasol+exasol_driver_websocket_dbapi2::test_insert_from_select - sqlalchemy.exc.DBAPIError: (exasol.driver.websocket._errors.Error)
- FAILED test/integration/sqlalchemy/test_suite.py::InsertBehaviorTest_exasol+exasol_driver_websocket_dbapi2::test_insert_from_select_with_defaults - sqlalchemy.exc.DBAPIError: (exasol.driver.websocket._errors.Error)
- FAILED test/integration/sqlalchemy/test_suite.py::NumericTest_exasol+exasol_driver_websocket_dbapi2::test_float_as_decimal - sqlalchemy.exc.DBAPIError: (exasol.driver.websocket._errors.Error)
- FAILED test/integration/sqlalchemy/test_suite.py::NumericTest_exasol+exasol_driver_websocket_dbapi2::test_float_as_float - sqlalchemy.exc.DBAPIError: (exasol.driver.websocket._errors.Error)
- FAILED test/integration/sqlalchemy/test_suite.py::NumericTest_exasol+exasol_driver_websocket_dbapi2::test_float_coerce_round_trip - AssertionError: '15.7563' != 15.7563
- FAILED test/integration/sqlalchemy/test_suite.py::NumericTest_exasol+exasol_driver_websocket_dbapi2::test_float_custom_scale - sqlalchemy.exc.DBAPIError: (exasol.driver.websocket._errors.Error)
- FAILED test/integration/sqlalchemy/test_suite.py::NumericTest_exasol+exasol_driver_websocket_dbapi2::test_numeric_as_float - AssertionError: {'15.7563'} != {15.7563}
- FAILED test/integration/sqlalchemy/test_suite.py::NumericTest_exasol+exasol_driver_websocket_dbapi2::test_render_literal_numeric_asfloat - AssertionError: assert '15.7563' in [15.7563]
Loading