forked from Point72/csp
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge pull request Point72#142 from Point72/pavithraes/docs-restructure
Documentation Restructure
- Loading branch information
Showing
39 changed files
with
3,864 additions
and
3,733 deletions.
There are no files selected for viewing
This file was deleted.
Oops, something went wrong.
This file was deleted.
Oops, something went wrong.
This file was deleted.
Oops, something went wrong.
This file was deleted.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,70 +1,38 @@ | ||
`csp` ("Composable Stream Processing") is a functional-like reactive | ||
language that makes time-series stream processing simple to do. The | ||
main reactive engine is a C++ based engine which has been exposed to | ||
python ( other languages may optionally be extended in future versions | ||
). `csp` applications define a connected graph of components using a | ||
declarative language (which is essentially python). Once a graph is | ||
constructed it can be run using the C++ engine. Graphs are composed of | ||
some number of "input" adapters, a set of connected calculation "nodes" | ||
and at the end sent off to "output" adapters. Inputs as well as the | ||
engine can be seamlessly run in simulation mode using historical input | ||
adapters or in realtime mode using realtime input adapters. | ||
<picture> | ||
<source media="(prefers-color-scheme: light)" srcset="https://raw.githubusercontent.com/Point72/csp/main/docs/img/csp-light.png"> | ||
<source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/Point72/csp/main/docs/img/csp-dark.png"> | ||
<img alt="CSP logo mark - text will be black in light color mode and white in dark color mode." width="50%"/> | ||
</picture> | ||
|
||
# Contents | ||
CSP (Composable Stream Processing) is a library for high-performance real-time event stream processing in Python. | ||
|
||
- [0. Introduction](https://github.com/Point72/csp/wiki/0.-Introduction) | ||
- [1. Generic Nodes (csp.baselib)](<https://github.com/Point72/csp/wiki/1.-Generic-Nodes-(csp.baselib)>) | ||
- [2. Math Nodes (csp.math)](<https://github.com/Point72/csp/wiki/2.-Math-Nodes-(csp.math)>) | ||
- [3. Statistics Nodes (csp.stats)](<https://github.com/Point72/csp/wiki/3.-Statistics-Nodes-(csp.stats)>) | ||
- [4. Random Time Series Generation](<https://github.com/Point72/csp/wiki/4.-Random-Time-Series-Generation-(csp.random)>) | ||
- [5. Adapters](https://github.com/Point72/csp/wiki/5.-Adapters) | ||
- [6. Dynamic Graphs](https://github.com/Point72/csp/wiki/6.-Dynamic-Graphs) | ||
- [7. csp.Struct](https://github.com/Point72/csp/wiki/7.-csp.Struct) | ||
- [8. Profiler](https://github.com/Point72/csp/wiki/8.-Profiler) | ||
- [9. Caching](https://github.com/Point72/csp/wiki/9.-Caching) | ||
## Key Features | ||
|
||
# Installation | ||
- **Powerful C++ Engine:** Execute the graph using CSP's C++ Graph Processing Engine | ||
- **Simulation (i.e., offline) mode:** Test workflows on historical data and quickly move to real-time data in deployment | ||
- **Infrastructure-agnostic:** Connect to any data format or storage database, using built-in (Parquet, Kafka, etc.) or custom adapters | ||
- **Highly-customizable:** Write your own input and output adapters for any data/storage formats, and real-time adapters for specific workflows | ||
- **PyData interoperability:** Use your favorite libraries from the Scientific Python Ecosystem for numerical and statistical computations | ||
- **Functional/declarative style:** Write concise and composable code for stream processing by building graphs in Python | ||
|
||
We ship binary wheels to install `csp` on MacOS and Linux via `pip`: | ||
<!-- ## Applications --> | ||
|
||
```bash | ||
pip install csp | ||
``` | ||
## Get Started | ||
|
||
Other platforms will need to see the instructions to [build `csp` from | ||
source](https://github.com/Point72/csp/wiki/98.-Building-From-Source). | ||
- [Install CSP](Installation) and [write your first CSP program](First-Steps) | ||
- Learn more about [nodes](CSP-Node), [graphs](CSP-Graph), and [execution modes](Execution-Modes) | ||
- Learn to extend CSP with [adapters](Adapters) | ||
|
||
We plan to create conda packages on conda-forge and ship binaries for Windows in | ||
the near future. | ||
<!-- - Check out the [examples](Examples) for various CSP features and use cases --> | ||
|
||
# Contributing | ||
> \[!TIP\] | ||
> Find relevant docs with GitHub’s search function, use `repo:Point72/csp type:wiki <search terms>` to search the documentation Wiki Pages. | ||
Contributions are welcome on this project. We distribute under the terms of the [Apache 2.0 license](https://github.com/Point72/csp/blob/main/LICENSE). | ||
## Community | ||
|
||
For **bug reports** or **small feature requests**, please open an issue on our [issues page](https://github.com/Point72/csp/issues). | ||
- [Contribute](Contribute) to CSP and help improve the project | ||
- Read about future plans in the [project roadmap](Roadmap) | ||
|
||
For **questions** or to discuss **larger changes or features**, please use our [discussions page](https://github.com/Point72/csp/discussions). | ||
## License | ||
|
||
For **contributions**, please see our [developer documentation](https://github.com/Point72/csp/wiki/99.-Developer). We have `help wanted` and `good first issue` tags on our issues page, so these are a great place to start. | ||
|
||
For **documentation updates**, make PRs that update the pages in `/docs/wiki`. The documentation is pushed to the GitHub wiki automatically through a GitHub workflow. Note that direct updates to this wiki will be overwritten. | ||
|
||
# Roadmap | ||
|
||
We do not have a formal roadmap, but we're happy to discuss features, improvements, new adapters, etc, in our [discussions area](https://github.com/Point72/csp/discussions). Here are some high level items we hope to accomplish in the next few months: | ||
|
||
- Support `clang` compiler and full MacOS support ([#33](https://github.com/Point72/csp/issues/33) / [#132](https://github.com/Point72/csp/pull/132)) | ||
- Support `msvc` compiler and full Windows support ([#109](https://github.com/Point72/csp/issues/109)) | ||
- Establish a better pattern for adapters ([#165](https://github.com/Point72/csp/discussions/165)) | ||
|
||
## Adapters and Extensions | ||
|
||
- Redis Pub/Sub Adapter with [Redis-plus-plus](https://github.com/sewenew/redis-plus-plus) ([#61](https://github.com/Point72/csp/issues/61)) | ||
- C++-based websocket adapter | ||
- Client adapter in [#152](https://github.com/Point72/csp/pull/152) | ||
- C++-based HTTP/SSE adapter | ||
- Add support for other graph viewers, including interactive / standalone / Jupyter | ||
|
||
## Other Open Source Projects | ||
|
||
- `csp-gateway`: Application development framework, built with [FastAPI](https://fastapi.tiangolo.com) and [Perspective](https://github.com/finos/perspective). This is a library we have built internally at Point72 on top of `csp` that we hope to open source later in 2024. It allows for easier construction of modular `csp` applications, along with a pluggable REST/WebSocket API and interactive UI. | ||
CSP is licensed under the Apache 2.0 license. See the [LICENSE](https://github.com/Point72/csp/blob/main/LICENSE) file for details. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
_This wiki is autogenerated. To made updates, open a PR against the original source file in [`docs/wiki`](https://github.com/Point72/csp/tree/main/docs/wiki)._ |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,61 @@ | ||
<!-- | ||
This sidebar is displayed on the GitHub Wiki section instead of the default sidebar. | ||
Notes for editors: | ||
- Ensure links don't have the file extensions (i.e., `.md`) | ||
- Do not use colons (':') in page titles, they don't render properly as links in the sidebar | ||
- Use only the filenames in this page (without the filepath and file extension) | ||
--> | ||
|
||
**[Home](Home)** | ||
|
||
**Get Started (Tutorials)** | ||
|
||
- [Installation](Installation) | ||
- [First steps](First-Steps) | ||
|
||
<!-- Add more pages when ready --> | ||
|
||
**Concepts** | ||
|
||
- [CSP Node](CSP-Node) | ||
- [CSP Graph](CSP-Graph) | ||
- [Historical Buffers](Historical-Buffers) | ||
- [Execution Modes](Execution-Modes) | ||
- [Adapters](Adapters) | ||
|
||
**How-to guides** | ||
|
||
- [Use Statistical Nodes](Use-Statistical-Nodes) | ||
- Use Adapters (coming soon) | ||
- [Add Cycles in Graphs](Add-Cycles-in-Graphs) | ||
- [Create Dynamic Baskets](Create-Dynamic-Baskets) | ||
- Write Adapters: | ||
- [Write Historical Input Adapters](Write-Historical-Input-Adapters) | ||
- [Write Realtime Input Adapters](Write-Realtime-Input-Adapters) | ||
- [Write Output Adapters](Write-Output-Adapters) | ||
- [Profile CSP Code](Profile-CSP-Code) | ||
|
||
**References** | ||
|
||
- API Reference | ||
- [Base Nodes API](Base-Nodes-API) | ||
- [Base Adapters API](Base-Adapters-API) | ||
- [Math and Logic Nodes API](Math-and-Logic-Nodes-API) | ||
- [Statistical Nodes API](Statistical-Nodes-API) | ||
- [Functional Methods API](Functional-Methods-API) | ||
- [Adapters (Kafka, Parquet, DBReader) API](Input-Output-Adapters-API) | ||
- [Random Time Series Generators API](Random-Time-Series-Generators-API) | ||
- [`csp.Struct` API](csp.Struct-API) | ||
- [`csp.dynamic` API](csp.dynamic-API) | ||
- [`csp.profiler` API](csp.profiler-API) | ||
- [Examples](Examples) | ||
- [Glossary of Terms](Glossary) | ||
|
||
**Developer Guide** | ||
|
||
- [Contributing](Contribute) | ||
- [Development Setup](Local-Development-Setup) | ||
- [Build CSP from Source](Build-CSP-from-Source) | ||
- [GitHub Conventions (for maintainers)](GitHub-Conventions) | ||
- [Release Process (for maintainers)](Release-Process) | ||
- [Roadmap](Roadmap) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,110 @@ | ||
`csp.baselib` defines some generally useful adapters, which are also imported directly into the CSP namespace when importing CSP. | ||
|
||
These are all graph-time constructs. | ||
|
||
## Table of Contents | ||
|
||
- [Table of Contents](#table-of-contents) | ||
- [`csp.timer`](#csptimer) | ||
- [`csp.const`](#cspconst) | ||
- [`csp.curve`](#cspcurve) | ||
- [`csp.add_graph_output`](#cspadd_graph_output) | ||
- [`csp.feedback`](#cspfeedback) | ||
|
||
## `csp.timer` | ||
|
||
```python | ||
csp.timer( | ||
interval: timedelta, | ||
value: '~T' = True, | ||
allow_deviation: bool = False | ||
) | ||
``` | ||
|
||
This will create a repeating timer edge that will tick on the given `timedelta` with the given value (value defaults to `True`, returning a `ts[bool]`) | ||
|
||
Args: | ||
|
||
- **`interval`**: how often to tick value | ||
- **`value`**: the actual value that will tick every interval (defaults to the value `True`) | ||
- **`allow_deviation`**: When running in realtime the engine will ensure timers execute exactly when they requested on their intervals. | ||
If your engine begins to lag, timers will still execute at the expected time "in the past" as the engine catches up | ||
(imagine having a `csp.timer` fire every 1/2 second but the engine becomes delayed for 1 second. | ||
By default the half seconds will still execute until time catches up to wallclock). | ||
When `allow_deviation` is `True`, and the engine is in realtime mode, subsequent timers will always be scheduled from the current wallclock + interval, | ||
so they won't end up lagging behind at the expensive of the timer skewing. | ||
|
||
## `csp.const` | ||
|
||
```python | ||
csp.const( | ||
value: '~T', | ||
delay: timedelta = timedelta() | ||
) | ||
``` | ||
|
||
This will create an edge that ticks one time with the value provided. | ||
By default this will tick at the start of the engine, delta can be provided to delay the tick | ||
|
||
## `csp.curve` | ||
|
||
```python | ||
csp.curve( | ||
typ: 'T', | ||
data: typing.Union[list, tuple] | ||
) | ||
``` | ||
|
||
This allows you to convert a list of non-CSP data into a ticking edge in CSP | ||
|
||
Args: | ||
|
||
- **`typ`**: is the type of the value of the data of this edge | ||
- **`data`**: is either a list of tuples of `(datetime, value)`, or a tuple of two equal-length numpy ndarrays, the first with datetimes and the second with values. | ||
In either case, that will tick on the returned edge into the engine, and the data must be in time order. | ||
Note that for the list of tuples case, you can also provide tuples of (timedelta, value) where timedelta will be the offset from the engine's start time. | ||
|
||
## `csp.add_graph_output` | ||
|
||
```python | ||
csp.add_graph_output( | ||
key: object, | ||
input: ts['T'], | ||
tick_count: int = -1, | ||
tick_history: timedelta = timedelta() | ||
) | ||
``` | ||
|
||
This allows you to connect an edge as a "graph output". | ||
All edges added as outputs will be returned to the caller from `csp.run` as a dictionary of `key: [(datetime, value)]` | ||
(list of datetime, values that ticked on the edge) or if `csp.run` is passed `output_numpy=True`, as a dictionary of | ||
`key: (array, array)` (tuple of two numpy arrays, one with datetimes and one with values). | ||
See [Collecting Graph Outputs](https://github.com/Point72/csp/wiki/0.-Introduction#collecting-graph-outputs) | ||
|
||
Args: | ||
|
||
- **`key`**: key to return the results as from `csp.run` | ||
- **`input`**: edge to connect | ||
- **`tick_count`**: number of ticks to keep in the buffer (defaults to -1 - all ticks) | ||
- **`tick_history`**: amount of ticks to keep by time window (defaults to keeping all history) | ||
|
||
## `csp.feedback` | ||
|
||
```python | ||
csp.feedback(typ) | ||
``` | ||
|
||
`csp.feedback` is a construct that can be used to create artificial loops in the graph. | ||
Use feedbacks in order to delay bind an input to a node in order to be able to create a loop | ||
(think of writing a simulated exchange that takes orders in and needs to feed responses back to the originating node). | ||
|
||
`csp.feedback` itself is not an edge, its a construct that allows you to access the delayed edge / bind a delayed input. | ||
|
||
Args: | ||
|
||
- **`typ`**: type of the edge's data to be bound | ||
|
||
Methods: | ||
|
||
- **`out()`**: call this method on the feedback object to get the edge which can be wired as an input | ||
- **`bind(x: ts[object])`**: call this to bind an edge to the feedback |
Oops, something went wrong.