Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow for Node Attribute dependencies #255

Merged
merged 9 commits into from
Mar 16, 2022
Merged

Allow for Node Attribute dependencies #255

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
3c42549
first version of NodeAttribute dependencies
PythonFZ Mar 15, 2022
b6ba324
Merge branch 'main' into add_getdesc
PythonFZ Mar 16, 2022
03f128f
temporary fix for circular imports
PythonFZ Mar 16, 2022
c04cb48
add znjson level
PythonFZ Mar 16, 2022
1645f3e
test properties
PythonFZ Mar 16, 2022
93f08cf
replace static duck type by `Node` import via `TYPE_CHECKING`
PythonFZ Mar 16, 2022
e09af28
cleanup isinstance
PythonFZ Mar 16, 2022
e37c590
update notebooks
PythonFZ Mar 16, 2022
469491d
add tests
PythonFZ Mar 16, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 15 additions & 14 deletions examples/docs/01_Intro.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -81,20 +81,20 @@
"\n",
"You can now commit the changes to git.\n",
"\n",
"\u001B[31m+---------------------------------------------------------------------+\n",
"\u001B[0m\u001B[31m|\u001B[0m \u001B[31m|\u001B[0m\n",
"\u001B[31m|\u001B[0m DVC has enabled anonymous aggregate usage analytics. \u001B[31m|\u001B[0m\n",
"\u001B[31m|\u001B[0m Read the analytics documentation (and how to opt-out) here: \u001B[31m|\u001B[0m\n",
"\u001B[31m|\u001B[0m <\u001B[36mhttps://dvc.org/doc/user-guide/analytics\u001B[39m> \u001B[31m|\u001B[0m\n",
"\u001B[31m|\u001B[0m \u001B[31m|\u001B[0m\n",
"\u001B[31m+---------------------------------------------------------------------+\n",
"\u001B[0m\n",
"\u001B[33mWhat's next?\u001B[39m\n",
"\u001B[33m------------\u001B[39m\n",
"- Check out the documentation: <\u001B[36mhttps://dvc.org/doc\u001B[39m>\n",
"- Get help and share ideas: <\u001B[36mhttps://dvc.org/chat\u001B[39m>\n",
"- Star us on GitHub: <\u001B[36mhttps://github.com/iterative/dvc\u001B[39m>\n",
"\u001B[0m"
"\u001b[31m+---------------------------------------------------------------------+\n",
"\u001b[0m\u001b[31m|\u001b[0m \u001b[31m|\u001b[0m\n",
"\u001b[31m|\u001b[0m DVC has enabled anonymous aggregate usage analytics. \u001b[31m|\u001b[0m\n",
"\u001b[31m|\u001b[0m Read the analytics documentation (and how to opt-out) here: \u001b[31m|\u001b[0m\n",
"\u001b[31m|\u001b[0m <\u001b[36mhttps://dvc.org/doc/user-guide/analytics\u001b[39m> \u001b[31m|\u001b[0m\n",
"\u001b[31m|\u001b[0m \u001b[31m|\u001b[0m\n",
"\u001b[31m+---------------------------------------------------------------------+\n",
"\u001b[0m\n",
"\u001b[33mWhat's next?\u001b[39m\n",
"\u001b[33m------------\u001b[39m\n",
"- Check out the documentation: <\u001b[36mhttps://dvc.org/doc\u001b[39m>\n",
"- Get help and share ideas: <\u001b[36mhttps://dvc.org/chat\u001b[39m>\n",
"- Star us on GitHub: <\u001b[36mhttps://github.com/iterative/dvc\u001b[39m>\n",
"\u001b[0m"
]
}
],
Expand Down Expand Up @@ -245,6 +245,7 @@
"outputs": [],
"source": [
"import logging\n",
"\n",
"config.log_level = logging.INFO"
]
},
Expand Down
145 changes: 135 additions & 10 deletions examples/docs/03_dependencies.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,9 +15,9 @@
"## Node dependencies\n",
"We will first look at Node based dependencies starting from a RandomNumber `Hello World` example.\n",
"In our first stage we create a random number and then we add another Node that depends on this one.\n",
"We can do this very easily by setting the `dvc.deps` in the following way\n",
"We can do this very easily by setting the `zn.deps` in the following way\n",
"```py\n",
"dependency: Stage = dvc.deps(Stage.load()\n",
"dependency: Stage = zn.deps(Stage) # or zn.deps(Stage.load())\n",
"```\n",
"\n",
"This allows us to access all properties of the `dependency` attribute."
Expand Down Expand Up @@ -74,7 +74,7 @@
"name": "stdout",
"output_type": "stream",
"text": [
"Initialized empty Git repository in /tmp/tmp1cphhnad/.git/\r\n",
"Initialized empty Git repository in /tmp/tmp6z7c4790/.git/\r\n",
"Initialized DVC repository.\r\n",
"\r\n",
"You can now commit the changes to git.\r\n",
Expand Down Expand Up @@ -153,10 +153,10 @@
"name": "stdout",
"output_type": "stream",
"text": [
"2022-02-22 13:37:09,358 (WARNING): Jupyter support is an experimental feature! Please save your notebook before running this command!\n",
"2022-03-16 13:42:38,692 (WARNING): Jupyter support is an experimental feature! Please save your notebook before running this command!\n",
"Submit issues to https://github.com/zincware/ZnTrack.\n",
"2022-02-22 13:37:12,883 (WARNING): Running DVC command: 'dvc run -n RandomNumber ...'\n",
"2022-02-22 13:37:18,895 (WARNING): Running DVC command: 'dvc run -n ComputePower ...'\n"
"2022-03-16 13:42:42,495 (WARNING): Running DVC command: 'dvc run -n RandomNumber ...'\n",
"2022-03-16 13:42:49,268 (WARNING): Running DVC command: 'dvc run -n ComputePower ...'\n"
]
}
],
Expand Down Expand Up @@ -210,7 +210,7 @@
"name": "stdout",
"output_type": "stream",
"text": [
"11.0 ^ 2.0 = 121.0\n"
"7.0 ^ 2.0 = 49.0\n"
]
}
],
Expand Down Expand Up @@ -288,9 +288,9 @@
"name": "stdout",
"output_type": "stream",
"text": [
"2022-02-22 13:37:28,604 (WARNING): Running DVC command: 'dvc run -n WriteToFile ...'\n",
"2022-02-22 13:37:34,666 (WARNING): Running DVC command: 'dvc run -n PowerFromFile ...'\n",
"2022-02-22 13:37:40,425 (WARNING): Running DVC command: 'dvc run -n ComparePowers ...'\n"
"2022-03-16 13:43:00,548 (WARNING): Running DVC command: 'dvc run -n WriteToFile ...'\n",
"2022-03-16 13:43:07,632 (WARNING): Running DVC command: 'dvc run -n PowerFromFile ...'\n",
"2022-03-16 13:43:14,835 (WARNING): Running DVC command: 'dvc run -n ComparePowers ...'\n"
]
}
],
Expand Down Expand Up @@ -422,9 +422,134 @@
"display(Pretty(\"dvc.yaml\"))"
]
},
{
"cell_type": "markdown",
"source": [
"## Node attributes as dependencies\n",
"\n",
"It is also possible to specify a Node attribute as a dependency. In this case you will be able to access the value of the attribute directly instead of using the Node class.\n",
"This can be used for all `dvc.<option>` and `zn.<option>` as well as e.g. class properties.\n",
"Note that the dvc dependencies will still be written for the full Node and won't be limited to the Node attribute.\n",
"To be able to define a dependency of an attribute the `zntrack.getdeps` function is required."
],
"metadata": {
"collapsed": false,
"pycharm": {
"name": "#%% md\n"
}
}
},
{
"cell_type": "code",
"execution_count": 15,
"outputs": [],
"source": [
"from zntrack import getdeps"
],
"metadata": {
"collapsed": false,
"pycharm": {
"name": "#%%\n"
}
}
},
{
"cell_type": "code",
"execution_count": 16,
"outputs": [],
"source": [
"class ComputePowerFromNumber(Node):\n",
" number: float = zn.deps() # this will be float instead of RandomNumber\n",
"\n",
" power: int = zn.params()\n",
" result: float = zn.outs()\n",
"\n",
" def run(self):\n",
" self.result = self.number**self.power"
],
"metadata": {
"collapsed": false,
"pycharm": {
"name": "#%%\n"
}
}
},
{
"cell_type": "code",
"execution_count": 17,
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"2022-03-16 13:43:31,053 (WARNING): Running DVC command: 'dvc run -n ComputePowerFromNumber ...'\n"
]
}
],
"source": [
"ComputePowerFromNumber(number=getdeps(RandomNumber, \"number\"), power=2.0).write_graph(\n",
" run=True\n",
")"
],
"metadata": {
"collapsed": false,
"pycharm": {
"name": "#%%\n"
}
}
},
{
"cell_type": "markdown",
"source": [
"`getdeps(RandomNumber, \"number\")` can also be replaced by `getdeps(RandomNumber[\"nodename\"], \"number\")` or `getdeps(RandomNumber.load(name=\"nodename\"), \"number\")`.\n",
"The first argument represents the Node and the second argument is the attribute, similar to `getattr()`."
],
"metadata": {
"collapsed": false,
"pycharm": {
"name": "#%% md\n"
}
}
},
{
"cell_type": "code",
"execution_count": 18,
"outputs": [],
"source": [
"compute_power = ComputePowerFromNumber.load()"
],
"metadata": {
"collapsed": false,
"pycharm": {
"name": "#%%\n"
}
}
},
{
"cell_type": "code",
"execution_count": 19,
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"7.0 ^ 2.0 = 49.0\n"
]
}
],
"source": [
"print(f\"{compute_power.number} ^ {compute_power.power} = {compute_power.result}\")"
],
"metadata": {
"collapsed": false,
"pycharm": {
"name": "#%%\n"
}
}
},
{
"cell_type": "code",
"execution_count": 20,
"id": "b26fb0f0-5f3e-40d0-b1fd-8d5844f051e8",
"metadata": {
"nbsphinx": "hidden",
Expand Down
86 changes: 80 additions & 6 deletions examples/docs/06_named_nodes.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@
"name": "stdout",
"output_type": "stream",
"text": [
"Initialized empty Git repository in /tmp/tmpptggw90x/.git/\r\n",
"Initialized empty Git repository in /tmp/tmpcn8yts4z/.git/\r\n",
"Initialized DVC repository.\r\n",
"\r\n",
"You can now commit the changes to git.\r\n",
Expand Down Expand Up @@ -114,11 +114,11 @@
"name": "stdout",
"output_type": "stream",
"text": [
"2022-02-22 13:52:00,589 (WARNING): Jupyter support is an experimental feature! Please save your notebook before running this command!\n",
"2022-03-16 13:28:59,293 (WARNING): Jupyter support is an experimental feature! Please save your notebook before running this command!\n",
"Submit issues to https://github.com/zincware/ZnTrack.\n",
"2022-02-22 13:52:04,249 (WARNING): Running DVC command: 'dvc run -n HelloWorld ...'\n",
"2022-02-22 13:52:11,594 (WARNING): Running DVC command: 'dvc run -n Test01 ...'\n",
"2022-02-22 13:52:18,383 (WARNING): Running DVC command: 'dvc run -n Test02 ...'\n"
"2022-03-16 13:29:03,070 (WARNING): Running DVC command: 'dvc run -n HelloWorld ...'\n",
"2022-03-16 13:29:10,857 (WARNING): Running DVC command: 'dvc run -n Test01 ...'\n",
"2022-03-16 13:29:18,410 (WARNING): Running DVC command: 'dvc run -n Test02 ...'\n"
]
}
],
Expand Down Expand Up @@ -198,7 +198,7 @@
"name": "stdout",
"output_type": "stream",
"text": [
"2022-02-22 13:52:27,386 (WARNING): Running DVC command: 'dvc run -n FindMaximum ...'\n"
"2022-03-16 13:29:28,486 (WARNING): Running DVC command: 'dvc run -n FindMaximum ...'\n"
]
}
],
Expand Down Expand Up @@ -285,9 +285,83 @@
}
}
},
{
"cell_type": "markdown",
"source": [
"In addition to the introduced classmethod `Node.load(name=\"nodename\")` it is also possible to use `Node[\"nodename\"]`. Note that this only works for `Node[\"nodename\"]` and not for `Node()[\"nodename\"]`. Using this we can also write the following:"
],
"metadata": {
"collapsed": false,
"pycharm": {
"name": "#%% md\n"
}
}
},
{
"cell_type": "code",
"execution_count": 13,
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"17\n",
"Test01\n"
]
}
],
"source": [
"print(HelloWorld[\"Test01\"].outputs)\n",
"print(HelloWorld[\"Test01\"].node_name)"
],
"metadata": {
"collapsed": false,
"pycharm": {
"name": "#%%\n"
}
}
},
{
"cell_type": "markdown",
"source": [
"this is equivalent to the classmethod `load()`. It is also possible to pass a dictionary as kwargs which will be passed to `load(**kwargs)`."
],
"metadata": {
"collapsed": false,
"pycharm": {
"name": "#%% md\n"
}
}
},
{
"cell_type": "code",
"execution_count": 14,
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"42\n",
"Test02\n",
"42\n"
]
}
],
"source": [
"print(HelloWorld.load(\"Test02\").outputs)\n",
"print(HelloWorld.load(\"Test02\").node_name)\n",
"print(HelloWorld[{\"name\": \"Test02\"}].outputs)"
],
"metadata": {
"collapsed": false,
"pycharm": {
"name": "#%%\n"
}
}
},
{
"cell_type": "code",
"execution_count": 15,
"id": "7b59179c-0b15-45a6-97b1-b2d6cc1ebce3",
"metadata": {
"nbsphinx": "hidden",
Expand Down
7 changes: 5 additions & 2 deletions examples/docs/09_lazy.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -127,7 +127,7 @@
"outputs": [],
"source": [
"class PrintOption(zn.outs):\n",
" dvc_option = \"outs\" \n",
" dvc_option = \"outs\"\n",
" # zntrack will try dvc --PrintOption outs.json\n",
" # we must tell it to use dvc --outs outs.json instead\n",
" def get_data_from_files(self, instance):\n",
Expand Down Expand Up @@ -737,7 +737,10 @@
],
"source": [
"# with lazy we get the same number for every run which is not what we expect.\n",
"print(f\"{random_number_lazy_1.number} == {random_number_lazy_2.number} == {random_number_lazy_3.number}\")"
"print(\n",
" f\"{random_number_lazy_1.number} == {random_number_lazy_2.number} ==\"\n",
" f\" {random_number_lazy_3.number}\"\n",
")"
]
},
{
Expand Down
Loading