diff --git a/README.md b/README.md index 4499dd21180..52d95f1c34e 100644 --- a/README.md +++ b/README.md @@ -9,15 +9,16 @@ Deephaven includes an intuitive user experience and visualization tools. It can ingest data from a variety of sources, apply computation and analysis algorithms to that data, and build rich queries, dashboards, and representations with the results. -Deephaven Community Core is an open version of [Deephaven Enterprise](https://deephaven.io), +Deephaven Community Core is the open version of [Deephaven Enterprise](https://deephaven.io), which functions as the data backbone for prominent hedge funds, banks, and financial exchanges. -[![Join the chat at https://gitter.im/deephaven/deephaven](https://badges.gitter.im/deephaven/deephaven.svg)](https://gitter.im/deephaven/deephaven?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge) -![Build CI](https://github.com/deephaven/deephaven-core/actions/workflows/build-ci.yml/badge.svg?branch=main) -![Quick CI](https://github.com/deephaven/deephaven-core/actions/workflows/quick-ci.yml/badge.svg?branch=main) -![Docs CI](https://github.com/deephaven/deephaven-core/actions/workflows/docs-ci.yml/badge.svg?branch=main) -![Check CI](https://github.com/deephaven/deephaven-core/actions/workflows/check-ci.yml/badge.svg?branch=main) -![Nightly Check CI](https://github.com/deephaven/deephaven-core/actions/workflows/nightly-check-ci.yml/badge.svg?branch=main) +- ![Build CI](https://github.com/deephaven/deephaven-core/actions/workflows/build-ci.yml/badge.svg?branch=main) +- ![Quick CI](https://github.com/deephaven/deephaven-core/actions/workflows/quick-ci.yml/badge.svg?branch=main) +- ![Docs CI](https://github.com/deephaven/deephaven-core/actions/workflows/docs-ci.yml/badge.svg?branch=main) +- ![Check CI](https://github.com/deephaven/deephaven-core/actions/workflows/check-ci.yml/badge.svg?branch=main) +- ![Nightly Check CI](https://github.com/deephaven/deephaven-core/actions/workflows/nightly-check-ci.yml/badge.svg?branch=main) + +This README is intended to provide a high-level overview of the installation and use of Deephaven Community Core. For more detailed guides on the topics presented below, see our [Community documentation](https://deephaven.io/core/docs). ## Supported Languages @@ -27,247 +28,233 @@ which functions as the data backbone for prominent hedge funds, banks, and finan | Java / Groovy | Yes | Yes | | C++ | No | Yes | | JavaScript | No | Yes | -| gRPC | - | Yes | - -## Run Deephaven +| Go | No | Yes | +| R | No | Yes | -This section is a quick start guide for running Deephaven from pre-built images. Almost all users will want to run Deephaven using pre-built images. It is the easiest way to deploy. For detailed instructions, see [Launch Deephaven from pre-built images](https://deephaven.io/core/docs/tutorials/quickstart). +Deephaven's client APIs use [gRPC](https://grpc.io/), [protobuf](https://github.com/deephaven/deephaven-core/tree/main/proto/proto-backplane-grpc/src/main/proto/deephaven/proto), [Apache Arrow Flight](https://arrow.apache.org/docs/format/Flight.html), and [Barrage](https://github.com/deephaven/barrage) to handle ticking data. Users who wish to build their own client APIs can use these tools to do so. -Developers interested in tinkering with and modifying source code should build from the source code. For detailed instructions on how to do this, see [Build and launch Deephaven](https://deephaven.io/core/docs/how-to-guides/launch-build). +The following list contains documentation links for installation instructions and more: -If you are not sure which of the two is right for you, use the pre-built images. +- Python + - [Run from Docker](https://deephaven.io/core/docs/tutorials/quickstart/) + - [pip-installed](https://deephaven.io/core/docs/tutorials/quickstart-pip/) +- Groovy + - [Run from Docker](https://deephaven.io/core/groovy/docs/tutorials/quickstart/) +- [Python client](https://pypi.org/project/pydeephaven/) +- [Java client](https://deephaven.io/core/docs/how-to-guides/java-client/) +- [JS client](https://deephaven.io/core/docs/reference/js-api/documentation/) +- [Go client](https://pkg.go.dev/github.com/deephaven/deephaven-core/go) +- [R client](https://github.com/deephaven/deephaven-core/blob/main/R/rdeephaven/README.md) -### Required Dependencies +## Install and run Deephaven -Running Deephaven requires a few software packages. +The Deephaven server can be installed and instantiated [from Docker](#from-docker), [from Python](#from-python), or [from source code](#built-from-source). -| Package | Version | OS | -| -------------- | ----------------------------- | ------------ | -| docker | ^20.10.8 | All | -| docker-compose | ^1.29.0 | All | -| Windows | 10 (OS build 20262 or higher) | Only Windows | -| WSL | 2 | Only Windows | +### From Docker -You can check if these packages are installed and functioning by running: -``` -docker version -docker-compose version -docker run hello-world -``` +This is the easiest way to get started with Deephaven. For complete instructions, see our [quickstart for Docker](https://deephaven.io/core/docs/tutorials/quickstart/). The table below shows installation dependencies. -> :warning: **On Windows, all commands must be run inside a WSL 2 terminal.** +| Dependency | Version | OS | Required/Recommended | +| -------------- | -------- | --------| -------------------- | +| Docker | ^20.10.8 | All | Required | +| Docker compose | ^2 | All | Recommended | +| Windows | 10+ | Windows | Required | +| WSL | ^2 | Windows | Required | -If any dependencies are missing or unsupported versions are installed, see [Launch Deephaven from pre-built images](https://deephaven.io/core/docs/tutorials/quickstart#prerequisites) for installation instructions. +The quickest way to install and run Deephaven from Docker is with a single Docker command: +**Python without Docker Compose** -For running the unit tests, you will also need to install [Git LFS](https://git-lfs.com/) and fetch all the required files. Run the following commands from inside the directory: +```sh +# Python +docker run --rm --name deephaven -p 10000:10000 ghcr.io/deephaven/server:latest ``` -git lfs install -git lfs pull + +**Groovy without Docker Compose** + +```sh +# Groovy +docker run --rm name deephaven -p 10000:10000 ghcr.io/deephaven/server-slim:latest ``` -### Create deployment +Users who wish to customize their deployment should use Docker Compose. Deephaven offers a multitude of pre-made [docker-compose.yml files](https://deephaven.io/core/docs/tutorials/quickstart/#choose-a-deployment) to choose from. To get started, all that's required is to download a file, pull the images, and start the server. -A directory must be created to store files and mount points for your deployment. Here, we are using the `deephaven-deployment` directory. +**Python with Docker Compose** -You will need to `cd` into the deployment directory to launch or interact with the deployment. +The base Python `docker-compose.yml` file can be found [here](https://raw.githubusercontent.com/deephaven/deephaven-core/main/containers/python-examples/base/docker-compose.yml). -```bash +```sh mkdir deephaven-deployment cd deephaven-deployment -``` - -> :warning: **Commands in the following sections for interacting with a deployment must be run from the deployment directory.** -### Launch: Python +curl -O https://raw.githubusercontent.com/deephaven/deephaven-core/main/containers/python-examples/base/docker-compose.yml -Run the following commands to launch Deephaven for Python server applications. - -```bash -curl https://raw.githubusercontent.com/deephaven/deephaven-core/main/containers/python/base/docker-compose.yml -O -docker-compose pull -docker-compose up -d +docker compose pull +docker compose up ``` -### Launch: Python with NLTK +**Groovy with Docker Compose** -Run the following commands to launch Deephaven for Python server applications with the [NLTK](https://nltk.org/) module pre-installed. - -```bash -curl https://raw.githubusercontent.com/deephaven/deephaven-core/main/containers/python/NLTK/docker-compose.yml -O -docker-compose pull -docker-compose up -d -``` +The base Groovy `docker-compose.yml` file can be found [here](https://raw.githubusercontent.com/deephaven/deephaven-core/main/containers/groovy/docker-compose.yml). -### Launch: Python with PyTorch +```sh +mkdir deephaven-deployment +cd deephaven-deployment -Run the following commands to launch Deephaven for Python server applications with the [PyTorch](https://pytorch.org/) module pre-installed. +curl -O https://raw.githubusercontent.com/deephaven/deephaven-core/main/containers/groovy/docker-compose.yml -```bash -curl https://raw.githubusercontent.com/deephaven/deephaven-core/main/containers/python/PyTorch/docker-compose.yml -O -docker-compose pull -docker-compose up -d +docker compose pull +docker compose up ``` -### Launch: Python with SciKit-Learn +### pip-installed Deephaven -Run the following commands to launch Deephaven for Python server applications with the [SciKit-Learn](https://scikit-learn.org/stable/) module pre-installed. +Users who wish to use Python but not Docker should use [pip-installed Deephaven](https://deephaven.io/core/docs/tutorials/quickstart-pip/). -```bash -curl https://raw.githubusercontent.com/deephaven/deephaven-core/main/containers/python/SciKit-Learn/docker-compose.yml -O -docker-compose pull -docker-compose up -d +```sh +pip install --upgrade pip setuptools wheel +pip install deephaven-server deephaven-ipywidgets ``` -### Launch: Python with TensorFlow - -Run the following commands to launch Deephaven for Python server applications with the [TensorFlow](https://www.tensorflow.org/) module pre-installed. +Then, from Python: -```bash -curl https://raw.githubusercontent.com/deephaven/deephaven-core/main/containers/python/TensorFlow/docker-compose.yml -O -docker-compose pull -docker-compose up -d +```python +from deephaven_server import Server +s = Server(port=10000, jvm_args=["-Xmx4g"]).start() ``` -### Launch: Python with example data +The input arguments to `Server` specify to bind to the Deephaven server on port `10000` and to allocate 4GB of memory to the server JVM. -Run the following commands to launch Deephaven for Python server applications, with example data. +### Built from source -```bash -curl https://raw.githubusercontent.com/deephaven/deephaven-core/main/containers/python-examples/base/docker-compose.yml -O -docker-compose pull -docker-compose up -d -``` - -### Launch: Python with example data and NLTK - -Run the following commands to launch Deephaven for Python server applications, with example data and [NLTK](https://nltk.org/). +Users who wish to modify source code and contribute to the project should build Deephaven from source. For complete instructions, see [How to build Deephaven from source](https://deephaven.io/core/docs/how-to-guides/launch-build/). -```bash -curl https://raw.githubusercontent.com/deephaven/deephaven-core/main/containers/python-examples/NLTK/docker-compose.yml -O -docker-compose pull -docker-compose up -d -``` +Building and running Deephaven requires a few software packages. -### Launch: Python with example data and PyTorch +| Package | Version | OS | Required/Recommended | +| -------------- | ----------------------------- | ------------ | -------------------- | +| git | ^2.25.0 | All | Required | +| java | >=11, <20 | All | Required | +| docker | ^20.10.8 | All | Required | +| docker compose | ^2 | All | Recommended | +| Windows | 10 (OS build 20262 or higher) | Only Windows | Required | +| WSL | 2 | Only Windows | Required | -Run the following commands to launch Deephaven for Python server applications, with example data and [PyTorch](https://pytorch.org/). +You can check if these packages are installed and functioning by running: ```bash -curl https://raw.githubusercontent.com/deephaven/deephaven-core/main/containers/python-examples/PyTorch/docker-compose.yml -O -docker-compose pull -docker-compose up -d +git version +java -version +docker version +docker compose version +docker run hello-world ``` -### Launch: Python with example data and SciKit-Learn +:::note -Run the following commands to launch Deephaven for Python server applications, with example data and [SciKit-Learn](https://scikit-learn.org/stable/). +Internally, the Java build process will use [Gradle Auto Provisioning](https://docs.gradle.org/current/userguide/toolchains.html#sec:provisioning) +to download and use the appropriate Java version for building and testing. -```bash -curl https://raw.githubusercontent.com/deephaven/deephaven-core/main/containers/python-examples/SciKit-Learn/docker-compose.yml -O -docker-compose pull -docker-compose up -d -``` +::: -### Launch: Python with example data and TensorFlow +:::note -Run the following commands to launch Deephaven for Python server applications, with example data and [TensorFlow](https://www.tensorflow.org/). +On Windows, all commands must be run inside a WSL 2 terminal. -```bash -curl https://raw.githubusercontent.com/deephaven/deephaven-core/main/containers/python-examples/TensorFlow/docker-compose.yml -O -docker-compose pull -docker-compose up -d -``` +::: -### Launch: Groovy / Java +#### Python -Run the following commands to launch Deephaven for Groovy / Java server applications. +A Python virtual environment is highly recommended for building Deephaven from source. Additionally, the wheel is installed with [pip](https://pypi.org/project/pip/) and built with [Gradle](https://gradle.org/). -```bash -curl https://raw.githubusercontent.com/deephaven/deephaven-core/main/containers/groovy/docker-compose.yml -O -docker-compose pull -docker-compose up -d +```sh +git clone https://github.com/deephaven/deephaven-core.git +cd deephaven-core +python3 -m venv /tmp/my-dh-venv +source /tmp/my-dh-venv/bin/activate +./gradlew py-server:assemble +pip install "py/server/build/wheel/deephaven_core--py3-non-any.whl[autocomplete] +./gradlew server-jetty-app:run ``` -### Launch: Groovy / Java with example data +#### Groovy -Run the following commands to launch Deephaven for Groovy / Java server applications, with example data. +The Groovy server is built with [Gradle](https://gradle.org/). `-Pgroovy` builds the Groovy server instead of Python. -```bash -curl https://raw.githubusercontent.com/deephaven/deephaven-core/main/containers/groovy-examples/docker-compose.yml -O -docker-compose pull -docker-compose up -d +```sh +git clone https://github.com/deephaven/deephaven-core.git +cd deephaven-core +./gradlew server-jetty-app:run -Pgroovy ``` -### Monitor logs - -The `-d` option to `docker-compose` causes the containers to run in the background, in detached mode. This option allows you to use your shell after Docker launches the containers. +## Get the authentication key -Since the container is running detached, you will not see any logs. However, you can follow the logs by running: +Deephaven, by default, uses [pre-shared key authentication](https://deephaven.io/core/docs/how-to-guides/authentication/auth-psk/) to authenticate against unauthorized access. -```bash -docker-compose logs -f -``` +### Deephaven run from Docker -Use CTRL+C to stop monitoring the logs and return to a prompt. +The pre-shared key is printed to the Docker logs when the server is started. Set your own key with the configuration parameter `-Dauthentication.psk=`. For users running Deephaven via Docker, this is set in the `environment` section of a `docker-compose.yml` file, or as a space-separated configuration parameter at the end of the [`docker run` command](#from-docker). -### Shutdown +To find the pre-shared key in the Docker logs: -The deployment can be brought down by running: - -```bash -docker-compose down +```sh +docker compose logs -f | grep "access through pre-shared key" ``` -### Manage example data +### Deephaven run from Python -[Deephaven's examples repository](https://github.com/deephaven/examples) contains data sets that are useful when learning -to use Deephaven. These data sets are used extensively in Deephaven's documentation and are needed to run some examples. [Deephaven's examples repository](https://github.com/deephaven/examples) contains documentation on the available data sets and how to manage them. +When a Deephaven server is started from Python, executing Deephaven queries from Python does _not_ require the key. However, if you wish to connect to the IDE via your web browser, you will need the pre-shared key. You will not be able to get the pre-shared key unless you set it yourself. To set the pre-shared key, add `"-Dauthentication.psk="` as an additional JVM parameter to the server. The following example sets the key to `MyPreSharedKey`: -If you have chosen a deployment with example data, the example data sets will be downloaded. Production deployments containing your own data will not need the example data sets. - - -To upgrade a deployment to the latest example data, run: - -```bash -docker-compose run examples download -``` - -To see what other example data management commands are available, run: - -```bash -docker-compose run examples +```python +from deephaven_server import Server +s = Server(port=10000, jvm_args=["-Xmx4g", "-Dauthentication.psk=MyPreSharedKey"]).start() ``` -If your deployment does not have example data, these commands will fail with `ERROR: No such service`. - +### Client APIs -## Run Deephaven IDE +Clients that attempt to connect to a server using pre-shared key authentication will need to supply the key to complete the connection. The key is the same for a client connection as it is for connecting directly to the server. For instance, in the [above example](#deephaven-run-from-python), the key for a client connection would also be `MyPreSharedKey`. -Once Deephaven is running, you can launch a Deephaven IDE in your web browser. Deephaven IDE allows you to interactively analyze data. +## Connect to the server -- If Deephaven is running locally, navigate to [http://localhost:10000/ide/](http://localhost:10000/ide/). -- If Deephaven is running remotely, navigate to `http://:10000/ide/`, where `` is the address of the machine Deephaven is running on. +The Deephaven UI is accessible from a web browser. For a server running locally on port 10000, it can be connected to via `https://localhost:10000/ide`. For a server running remotely on port 10000, it can be connected to via `https://:10000/ide`. If using authentication, enter credentials to gain access to the IDE. For information on supported browsers, see [here](https://github.com/deephaven/web-client-ui#browser-support). -![alt_text](docs/images/ide_startup.png "Deephaven IDE") - -# First query +## First query From the Deephaven IDE, you can perform your first query. -This script creates two small tables: one for employees and one for departments. -It joins the two tables on the DeptID column to show the name of the department -where each employee works. +The scripts below create two small tables: one for employees and one for departments. They are joined on the `DeptID` column to show the name of the department where each employee works. + +### Python ```python +from deephaven import new_table +from deephaven.column import string_col, int_col +from deephaven.constants import NULL_INT + +left = new_table([ + string_col("LastName", ["Rafferty", "Jones", "Steiner", "Robins", "Smith", "Rogers"]), + int_col("DeptID", [31, 33, 33, 34, 34, NULL_INT]), + string_col("Telephone", ["(347) 555-0123", "(917) 555-0198", "(212) 555-0167", "(952) 555-0110", None, None]) + ]) + +right = new_table([ + int_col("DeptID", [31, 33, 34, 35]), + string_col("DeptName", ["Sales", "Engineering", "Clerical", "Marketing"]), + string_col("Telephone", ["(646) 555-0134", "(646) 555-0178", "(646) 555-0159", "(212) 555-0111"]) + ]) + +t = left.join(right, "DeptID", "DeptName, DeptTelephone=Telephone") +``` + +![alt_text](docs/images/ide_first_query.png "Deephaven IDE First Query") -from deephaven.TableTools import newTable, stringCol, intCol -from deephaven.conversion_utils import NULL_INT +### Groovy +```groovy left = newTable( - stringCol("LastName", "Rafferty", "Jones", "Steiner", "Robins", "Smith", "Rogers"), - intCol("DeptID", 31, 33, 33, 34, 34, NULL_INT), - stringCol("Telephone", "(347) 555-0123", "(917) 555-0198", "(212) 555-0167", "(952) 555-0110", None, None) + string_col("LastName", "Rafferty", "Jones", "Steiner", "Robins", "Smith", "Rogers"), + int_col("DeptID", 31, 33, 33, 34, 34, NULL_INT), + string_col("Telephone", "(347) 555-0123", "(917) 555-0198", "(212) 555-0167", "(952) 555-0110", null, null) ) right = newTable( @@ -276,12 +263,11 @@ right = newTable( stringCol("Telephone", "(646) 555-0134", "(646) 555-0178", "(646) 555-0159", "(212) 555-0111") ) -t = left.join(right, "DeptID", "DeptName,DeptTelephone=Telephone") +t = left.join(right, "DeptID", "DeptName, DeptTelephone=Telephone") ``` ![alt_text](docs/images/ide_first_query.png "Deephaven IDE First Query") - ## Resources * [Help!](https://github.com/deephaven/deephaven-core/discussions/969) @@ -291,14 +277,17 @@ t = left.join(right, "DeptID", "DeptName,DeptTelephone=Telephone") * [Java API docs](https://deephaven.io/core/javadoc/) * [Python API docs](https://deephaven.io/core/pydoc/) -## Code Of Conduct +## Contributing + +See [CONTRIBUTING](./CONTRIBUTING.md) for full instructions on how to contribute to this project. + +### Code Of Conduct This project has adopted the [Contributor Covenant Code of Conduct](https://www.contributor-covenant.org/version/2/0/code_of_conduct/). For more information see the [Code of Conduct](CODE_OF_CONDUCT.md) or contact [opencode@deephaven.io](mailto:opencode@deephaven.io) with any additional questions or comments. - -## License +### License Copyright © 2016-2023 Deephaven Data Labs and Patent Pending. All rights reserved.