A Data Dependency Graph Framework and Executor
DataJet abstracts over function calls by mapping inputs through a graph of functions to desired outputs. As a programmer, you declare your data transformations (functions of inputs to outputs) once, and datajet will handle mapping any input to any output reachable by the graph of functions.
Key Features
- Lazy: Only Evaluate and return the data you need
- Declarative: Declare Data and functions on the data explicitly, using plain python
- Dependency-Free: Just Python.
Requirements:
- Python >=3.8
To Get Started, Install DataJet From pypi:
pip install datajet- DataJet simplifies the codebase of dynamic systems with mutliple ways to calculate a datapoints from different inputs.
- DataJet de-couples downstream calculations from the mechanics of calculating upstream dependencies.
from datajet import execute
dollars = [7.98, 20.94, 37.9, 30.31]
units = [1, 3, 5, 4,]
def prices(dollars, units):
return [d/u for d, u in zip(dollars, units)]
def average_price(prices):
return sum(prices) / len(prices)
def average_price_rounded_down(average_price):
return average_price * 1000 // 10 / 100
datajet_map = {
"prices": prices,
"average_price": average_price,
"average_price_rounded_down": average_price_rounded_down,
}
execute(
datajet_map,
context={
"dollars": dollars,
"units": units,
},
fields=['average_price_rounded_down']
)
{'average_price_rounded_down': 7.52}And, if you have prices, you can directly get what you need:
prices = [3.99, 4.49, 2.89, 2.79, 2.99]
execute(datajet_map,context={"prices": prices,}, fields=['average_price', 'average_price_rounded_down'])
{'average_price': 3.4299999999999997, 'average_price_rounded_down': 3.42}Keys can be any hashable. The value corresponding to each key can be a function or an object. The functions can have 0 or more parameters. The parameter names must correspond to other keys in the dict if no explicitly defined inputs to the callable are declared in the map. See Datamap reference for how to explicitly define inputs.
You can also define multiple ways of calculating a piece of data via defining a list of functions as the value to the key. Again, each function's parameters must correspond to other keys in the dict, or else you can define which other keys should be inputs to the function via explicitly defining inputs.
https://bmritz.github.io/datajet/
git clone
make install
This will start a poetry shell that has all the dev dependencies installed. You can run deactivate to exit the shell.
make test
If you see:
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)>
Go to /Applications/Python3.x and run 'Install Certificates.command'