embedPy

Allows the kdb+ interpreter to manipulate Python objects and call Python functions. Part of the Fusion for kdb+ interface collection.

Please direct any questions to ai@kx.com.

Please report issues in this repository.

Requirements

• kdb+ ≥ 3.5 64-bit/32-bit(Linux/Arm)
• Python ≥ 3.5.0 (macOS/Linux/Arm) ≥ 3.6.0 windows
• numpy < 1.22

⚠️ numpy version: Due to a breaking change with the numpy C api conversions between Python and q arrays is unsupported.

 q)np:.p.import[`numpy]
 q)np[`:__version__]`
 "1.22.1"
 q)np[`:arange][10]`
 foreign

Overview

You can either

• install embedPy to run on your local machine; or
• download or build a Docker image in which to run embedPy

There are three ways to install embedPy on your local machine:

1. Download and install a release

2. Clone and build from source, on your local machine or in a Docker image

3. Install with Conda - recommended for use with

   • Anaconda Python
   • mlnotebooks
   • JupyterQ

32-bit Linux/Arm builds require users to build from source, there is not currenly a conda build or provided pre-compiled binary.

Anaconda Python

If you are using Anaconda Python, we recommend installing with Conda. If, instead, you take option (1) or (2) above, and are using Linux or macOS, set your LD_LIBRARY_PATH (Linux) or DYLD_LIBRARY_PATH (macOS) to your Python distributions library directory to avoid conflicts between libraries which both q and Python use (e.g. libz, libssl). You can find this directory's location in Python.

>>> import sysconfig
>>> sysconfig.get_config_var('LIBDIR')

PyQ

If you are currently using PyQ, both interfaces use a file p.k in $QHOME/{l64,m64} which results in a conflict when both are installed.

You may want to run initially from another directory, without installing. Skip the install step above, and run q in the directory where you unzipped the release.

Test script

The test script test.q requires the packages listed in tests/requirements.txt, although embedPy does not itself require them. They can be installed using pip or conda.

pip install -r tests/requirements.txt

or

conda install --file tests/requirements.txt

If the tests all pass, no message is displayed.

Install on local machine

Download and install a release

1. Download a release archive from the releases page, and unzip it.

2. In the unzipped directory, run the tests.

   $ q test.q

3. Install: put p.q and p.k in QHOME and the library file (p.so for macOS/Linux or p.dll for Windows) in $QHOME/{l64,m64,w64}.

Clone and build from source

1. Clone this repository from GitHub.

2. To run embedPy without Internet access, download the kdb+ C API header file and place it in the build directory.

3. Build the interface and run the tests.

   $ make p.so && q test.q

4. Install: put p.q and p.k in $QHOME and p.so in $QHOME/{l64,l32,m64}.

Note

For ease of install on 32-bit Arm and linux it is suggested that a new user should use a miniconda version of python specific to the architecture being used, for example rpi for raspberry pi. This is not an explicit requirement but makes install of embedPy and python packages more seamless.

Install with Conda

This requires either macOS or Linux.

1. Download and install either the full Anaconda distribution or Miniconda for Python3

2. Use the conda command to install packages as follows:

   $ conda install -c kx embedPy

Run on local machine

Start q with embedPy

$ q p.q

Or from q, load p.q.

q)\l p.q

Documentation is on the embedPy homepage.

Run a Docker image

If you have Docker installed, instead of installing embedPy on your machine, you can run:

$ docker run -it --name myembedpy kxsys/embedpy
kdb+ on demand - Personal Edition

[snipped]

I agree to the terms of the license agreement for kdb+ on demand Personal Edition (N/y): y

If applicable please provide your company name (press enter for none): ACME Limited
Please provide your name: Bob Smith
Please provide your email (requires validation): bob@example.com
KDB+ 3.5 2018.04.25 Copyright (C) 1993-2018 Kx Systems
l64/ 4()core 7905MB kx 0123456789ab 172.17.0.2 EXPIRE 2018.12.04 bob@example.com KOD #0000000

q)

See docker/README.md for more details.

Back-incompatible changes

V1.0 -> V 1.1

.p.key and .p.value removed

V0.2-beta -> V1.0

• Attribute access from embedPy object

  q)obj`ATTRNAME   / old
  q)obj`:ATTRNAME  / new

• embedPy objects can be called directly without explicitly specifying the call return type; the default return type is an embedPy object

V0.1-beta -> V0.2beta in V0.2-beta

V0.2-beta features a number of changes back-incompatible with the previous release, V0.1-beta.

Most notably, the default type used in many operations is now the embedPy type, rather than the foreign type.