Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Migrate/Update API documentation #158

Merged
merged 24 commits into from
Dec 5, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
8d55c64
Add api documentation for connection and ExaConnection
Nicoretti Nov 29, 2024
f10530d
Migrate all pydoc comments to Napoleon docstring style comments
Nicoretti Nov 29, 2024
a8ad894
Add api documentation for connect_local_config function
Nicoretti Nov 29, 2024
835d9a8
Add api documentation for http_transport function
Nicoretti Nov 29, 2024
340b1ca
Add all relevant classes and functions to api docs
Nicoretti Nov 29, 2024
29605a9
Migrate documentation of ExaConnection to doc strings
Nicoretti Nov 29, 2024
8b2e842
Migrate documentation of ExaStatement to doc strings
Nicoretti Dec 3, 2024
ba4b057
Migrate documentation of ExaFormatter to doc strings
Nicoretti Dec 3, 2024
c306506
Migrate documentation of ExaMetaData to doc strings
Nicoretti Dec 3, 2024
23dd3be
Migrate documentation of ExaExtension to doc strings
Nicoretti Dec 4, 2024
aa1fbc8
Migrate documentation of ExaHTTPTransportWrapper to doc strings
Nicoretti Dec 4, 2024
8b6e27e
Update changelog
Nicoretti Dec 4, 2024
89a24dd
Update pyexasol/__init__.py
Nicoretti Dec 4, 2024
945320c
Update pyexasol/connection.py
Nicoretti Dec 4, 2024
95729bc
Update pyexasol/connection.py
Nicoretti Dec 4, 2024
b0d6ef5
Update pyexasol/connection.py
Nicoretti Dec 4, 2024
53fb8a6
Update pyexasol/connection.py
Nicoretti Dec 4, 2024
2bd18e9
Update pyexasol/connection.py
Nicoretti Dec 4, 2024
c2f1f23
Update pyexasol/connection.py
Nicoretti Dec 4, 2024
753fe05
Update pyexasol/formatter.py
Nicoretti Dec 4, 2024
63342d5
Update pyexasol/formatter.py
Nicoretti Dec 4, 2024
fb55fa2
Update pyexasol/formatter.py
Nicoretti Dec 5, 2024
cdc45fd
Update pyexasol/formatter.py
Nicoretti Dec 5, 2024
dc6e068
Address code review feedback
Nicoretti Dec 5, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
41 changes: 41 additions & 0 deletions doc/api.rst
Original file line number Diff line number Diff line change
Expand Up @@ -3,3 +3,44 @@
:octicon:`cpu` API Reference
=============================

.. autofunction:: pyexasol.connect

.. autofunction:: pyexasol.connect_local_config

.. autofunction:: pyexasol.http_transport

.. autoclass:: pyexasol.ExaConnection
:members:
:special-members: __init__
:undoc-members:
:show-inheritance:

.. autoclass:: pyexasol.ExaStatement
:members:
:special-members: __init__, __iter__
:undoc-members:
:show-inheritance:

.. autoclass:: pyexasol.ExaFormatter
:class-doc-from: init
:members:
:undoc-members:
:show-inheritance:

.. autoclass:: pyexasol.ExaMetaData
:class-doc-from: init
:members:
:undoc-members:
:show-inheritance:

.. autoclass:: pyexasol.ExaExtension
:class-doc-from: init
:members:
:undoc-members:
:show-inheritance:

.. autoclass:: pyexasol.ExaHTTPTransportWrapper
:class-doc-from: init
:members:
:undoc-members:
:show-inheritance:
4 changes: 4 additions & 0 deletions doc/changes/unreleased.md
Original file line number Diff line number Diff line change
@@ -1,2 +1,6 @@
# Unreleased

## 📚 Documentation

* Add sphinx based documention

2 changes: 2 additions & 0 deletions doc/user_guide/protocol_version.rst
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
.. _protocol_version:

WebSocket protocol versions
===========================

Expand Down
80 changes: 49 additions & 31 deletions pyexasol/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -62,22 +62,28 @@

def connect(**kwargs) -> ExaConnection:
"""
Constructor of connection objects
Please check ExaConnection object for list of arguments
Create a new connection object. For details regarding kwargs,
refer to the :class:`pyexasol.ExaConnection` class.
"""
return ExaConnection(**kwargs)


def connect_local_config(config_section, config_path=None, **kwargs) -> ExaConnection:
"""
Constructor of connection objects based on local config file
Default config path is ~/.pyexasol.ini
Constructor for connection objects based on a local config file.

Extra arguments override values from config
Info:
- The default config path is ~/.pyexasol.ini
- Extra arguments override values from config

:param config_section: Name of config section (required!)
:param config_path: Custom path to local config file
:param kwargs: Arguments for "connect()" function
Args:

config_section:
Name of config section (required!)
config_path:
Custom path to local config file
kwargs:
Arguments for "connect()" function
"""

conf = ExaLocalConfig(config_path)
Expand All @@ -88,28 +94,40 @@ def connect_local_config(config_section, config_path=None, **kwargs) -> ExaConne

def http_transport(ipaddr, port, compression=False, encryption=True) -> ExaHTTPTransportWrapper:
"""
Constructor of HTTP Transport wrapper for parallel HTTP Transport (EXPORT or IMPORT)
Compression and encryption arguments should match pyexasol.connect()

How to use:
1) Parent process opens main connection to Exasol with pyexasol.connect()
2)
2) Parent process creates any number of child processes (possibly on remote host or another container)
3) Every child process starts HTTP transport sub-connection with pyexasol.http_transport()
and gets "ipaddr:port" string using ExaHTTPTransportWrapper.address
4) Every child process sends address string to parent process using any communication method (Pipe, Queue, Redis, etc.)
5) Parent process runs .export_parallel() or .import_parallel(), which initiates EXPORT or IMPORT query in Exasol
6) Every child process receives or sends a chunk of data using ExaHTTPTransportWrapper.export_*() or .import_*()
7) Parent process waits for Exasol query and for child processes to finish

All child processes should run in parallel.
It is NOT possible to process some data first, and process some more data later.

If an exception is raised in child process, it will close the pipe used for HTTP transport.
Closing the pipe prematurely will cause SQL query to fail and will raise an exception in parent process.
Parent process is responsible for closing other child processes and cleaning up.

PyEXASOL does not provide a complete solution to manage child processes, only examples.
The final solution depends on your hardware, network configuration, cloud provider and container orchestration software.
Constructor for HTTP Transport wrapper for parallel HTTP Transport (EXPORT or IMPORT)

Args:
ipaddr:
IP address of one of Exasol nodes received from :meth:`pyexasol.ExaConnection.get_nodes`
port:
Port of one of Exasol nodes received from :meth:`pyexasol.ExaConnection.get_nodes`
compression:
Use zlib compression for HTTP transport, must be the same as `compression` of main connection
encryption:
Use SSL/TLS encryption for HTTP transport, must be the same as `encryption` of main connection

Info:
Compression and encryption arguments should match :func:`pyexasol.connect`

How to use:

#. Parent process opens main connection to Exasol with pyexasol.connect()
#. Parent process creates any number of child processes (possibly on remote host or another container)
#. Every child process starts HTTP transport sub-connection with pyexasol.http_transport()
#. and gets "ipaddr:port" string using ExaHTTPTransportWrapper.address
#. Every child process sends address string to parent process using any communication method (Pipe, Queue, Redis, etc.)
#. Parent process runs .export_parallel() or .import_parallel(), which initiates EXPORT or IMPORT query in Exasol
#. Every child process receives or sends a chunk of data using ExaHTTPTransportWrapper.export_*() or .import_*()
#. Parent process waits for Exasol query and for child processes to finish

All child processes should run in parallel.
It is NOT possible to process some data first, and process some more data later.

If an exception is raised in child process, it will close the pipe used for HTTP transport.
Closing the pipe prematurely will cause SQL query to fail and will raise an exception in parent process.
Parent process is responsible for closing other child processes and cleaning up.

PyEXASOL does not provide a complete solution to manage child processes, only examples.
The final solution depends on your hardware, network configuration, cloud provider and container orchestration software.
"""
return ExaHTTPTransportWrapper(ipaddr, port, compression, encryption)
Loading
Loading