Replies: 5 comments 31 replies
-
|
Relevant discussions/resources:
I hadn't though of using DVC, but on the face of it, that sounds like a awesome way to implement the functionality. Funnily enough, my gist above got a comment by @bvancil (https://github.com/bvancil) that also asked about the code-versioning functionality. I thought along the same lines as you, that comparing the AST of the node would be a cool way to do this, but your point on module-level things like imports changing is something I hadn't considered. Potentially it could be node-code + module-code (minus other functions/nodes)? |
Beta Was this translation helpful? Give feedback.
-
|
@Spooky-0 -- instead of using inspect, perhaps one could leverage a "designated test generator" for a given artifact (type?) and coverage, parsing the xml or json output. Afterthought -- thinking about our own code, the problem for us with this approach would be numba.cuda code, which isn't covered. Usually we place cuda compiled code together with wrapper code in the same module, so as long as we weren't trying to resolve beyond the module level this would work. I presume that perhaps inspect could do a bit better (seeing the call, at least) but it would also have problems with calls out into c code, etc. Presumably some config/manual specification of relation between certain code and certain artifacts could close this gap. ... perhaps pipeline steps can be associated with tests (and perhaps many-many?) via a wrapper. Could use coverage contexts to mark what code was run by which of these tests (in CI/CD, for instance). |
Beta Was this translation helpful? Give feedback.
-
|
@shaunc LOVE the idea of using something like coverage to do this. This sounds much more promising to me than trying to parse the AST ourselves. Why try and reinvent the wheel to work out which lines of Python are executed when other people have already figured that out? Also might be worth considering trace and heartrate. Being compatible with CUDA would be nice, but honestly I think that's quite an unusual/advanced case you have. If we could have something working just with normal Python code then that would be amazing. Even just the naive |
Beta Was this translation helpful? Give feedback.
-
|
Heartrate seems higher level, but trace looks quite nice... hmm... so one question would be to do this "at runtime" -- as you could with trace, or via a test as I was thinking. My idea was that you either declared a test with the node, or you wrapped a test in a decorator that referenced the node. Vs CUDA or etc etc my idea was in that wrapper you could also specify some modules to track manually. I guess you could do this using trace automagically without having a designated test? I guess there is a performance hit vs not having to maintain the test and seeing what branches actually ran in a node execution. |
Beta Was this translation helpful? Give feedback.
-
|
Yeah, I was imagining something where you would do this at runtime rather than needing to write a test for each node that you want to "version control". Otherwise it seems like a bit of an abuse of the test runner, which we're only really using as a vehicle to get coverage. And you'd have to ensure that you're running the tests in between every The other possible issue I see with the test runner strategy is that you need to write test data for every single node you want to version control. I was thinking that you could avoid writing an individual test for each node by just writing one parametrised test that does something clever by going through each of kedro's nodes. But you'd still need a way to run each node function on its own test input data somehow. |
Beta Was this translation helpful? Give feedback.
-
|
Yes -- I guess you could perhaps have the best of both worlds if there was a "record mode" where you would run and write per-node-metadata on source code usage (which you could check in), and a "production mode" where you would just use this data to possibly skip nodes or not without having trace on. I personally would still like the ability to add additional code manually to the generated list (and not have it clobbered by rerecord, of course). |
Beta Was this translation helpful? Give feedback.
-
|
The flip-side of this discussion on partial rebuilds -- suppose I'm using argo-workflows to run my pipeline. I want to get all the artifacts into DVC so that I can later use them for partial rebuilds. Should I write wrappers that check in outputs? What about logs and error outputs? Also in a distributed environment I can't just rely on the local work tree -- it would seem that I need a versioning policy to guide how the commits are merged. Has any of this work been done already? Suggestions on where to look, or where to put it if I want to build something myself? Update -- dvc checkpoints seem to be the thing to integrate with. I guess the approach would be to make kedro the source of authority, and automate generation of dvc.yaml, etc.... (unless there is some reason to think about doing it the other way around?) |
Beta Was this translation helpful? Give feedback.
-
|
@Spooky-0, @AntonyMilneQB @mzjp2 -- is there any code out there that generates dvc.yaml-s from pipeline definitions? Is that part of the vision here? I would be happy to work on this -- should it go in a kedro-dvc repo/plugin? |
Beta Was this translation helpful? Give feedback.
-
|
So I'm not sure we have anything off the shelf - since joining the LF we've had a lot of activity on the topic so we'll try and consolidate it all shortly (see this discssion for details #1181) Also happy to answer any questions about the internals that may be helpful - there will be some small breaking changes in 0.18.0 that we can keep front of mind. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks, @datajoely ... on the dvc side I've gotten some encoragement -- [Update -- more info on the dvc side] |
Beta Was this translation helpful? Give feedback.
-
|
An initial stab at a design -- https://github.com/FactFiber/kedro-dvc/blob/main/doc/design.md BTW ... :( ... |
Beta Was this translation helpful? Give feedback.
-
|
Also feel free to tag us in PRs and issues on your repo if it's to do with Kedro internals and we'll do our best ot help :) |
Beta Was this translation helpful? Give feedback.
-
|
We need 3.9 support certainly, but we can try to keep 3.8 at the very least as well for the moment. Hmm... how to handle with CI? Are you perchance going to create an 0.18 pre-release (which is stable enough! :)) anytime soon? I encourage it! :) As for comments -- I'm sure there is still much to think through. The first issue that comes to mind is versioning. I haven't looked into it yet, but would hypothesize that dvc uses git versioning tools to produce hashes. Perhaps this is a good way to go? Is there a point at which I can intercept how you generate version strings and substitute in the dvc version hash? I encourage you to take the discussion to the plugin discussion board :) Thank you! |
Beta Was this translation helpful? Give feedback.
-
|
0.18 pre-release is unofficially available already in the Will take a look at the plugin discussion board for the other stuff! Very exciting 😀 |
Beta Was this translation helpful? Give feedback.
-
|
develop -- almost!: % poetry add https://github.com/kedro-org/kedro/archive/develop.zip
...
Because no versions of black match >22.1.0,<23.0.0
and black (22.1.0) depends on click (>=8.0.0), black (>=22.1.0,<23.0.0) requires click (>=8.0.0).
And because kedro (0.17.6) depends on click (<8.0), black (>=22.1.0,<23.0.0) is incompatible with kedro (0.17.6).sigh -- I wish poetry would allow overrides.... however -- works with black |
Beta Was this translation helpful? Give feedback.
-
|
Ah yes, this was pointed out last week and is on the backlog to fix 🙂 |
Beta Was this translation helpful? Give feedback.
-
|
I've written some more detail into the basic correspondence between kedro and dvc. A question -- is there a mechanism for attaching extra metadata to data catalog entries, nodes and pipelines? Perhaps a generic mechanism would suffice. E.g. a meta=dict(kedro_dvc=dict( ...)) or equiv in yaml for data catalog. This could also be provided externally, but it gets harder to be DRY that way.
|
Beta Was this translation helpful? Give feedback.
-
|
Hi @shaunc off topic for a second - it would be great to join our discord server - it may make sense to make a dedicated |
Beta Was this translation helpful? Give feedback.
-
|
@datajoely -- thanks for the invite! @AntonyMilneQB -- thanks for the replies!
|
Beta Was this translation helpful? Give feedback.
-
|
@datajoely -- do you have an "after_command_run" cli hook? We wanted to add our own conf install after |
Beta Was this translation helpful? Give feedback.
-
|
@shaunc that doesn't exist, but we could probably add it if it's useful. The |
Beta Was this translation helpful? Give feedback.
-
|
It would be useful... its not essential though. Right now I'm starting with a sequence |
Beta Was this translation helpful? Give feedback.
-
|
Hello Kedro Devs/Mods, Any updates in this work? I read that Kedro provides data versioning -- but it simply writes data to folder with timestamp. Recently, there are some demo's that show how MLFlow and DVC can work together. I am wondering are there any efforts to make DVC work with Kedro Data Catalogue. It solves many problems and Then Kedro can play nicely along with other mature open source tools in the MLOps space. Otherwise, it feels as if, everybody (tool) becomes a siwss-army knife of MLops. Appreciate any pointers or comments or suggestions! |
Beta Was this translation helpful? Give feedback.
-
|
@shaunc how are things going? Is there any support we cna provide? |
Beta Was this translation helpful? Give feedback.
-
Hi All,
I have been thinking of creating a Kedro Plugin which leverages DVCs Version Controlled Data but Kedros beautiful pipelines. Somewhere I have read a discussion of this idea before, but I can’t find it anymore. Maybe Github, Discord, etc.
Background: At work I had the problem of jobs taking a long time and failing often, but once they ran successfully, all was good. However, when changing parts of the pipeline, after a while, you don’t know if your data is up to date any more - so you have to rerun all. And this takes a long time. So I used DVC to track my input data (e.g. CSV, Hadoop Data*, etc.) but I also tracked the scripts (.py, .sh, etc.). When any one of those changed, the Pipeline would run only that what was necessary. For development this was a game changer.
*) I added an extra step, which simply called ‘DESCRIBE my_hive/impala_table’, saved the output somewhere, and then DVC’ed just this proxy. If the Data on Hive changed, DVC would not know - but this was fine.
For Kedro I was thinking of creating a Plugin/Hook which has very similar behaviour. A Nodes Input is very easily tracked by DVC (especially easy for local data). The Code is not so easily tracked.
Some difficulties: I cannot track all the code, because if I change Node17 then I would have to rerun everything, because accoarding to DVC some code changed.
Thus I want to have a DVC run only look at the current Nodes Code and Data. I was thinking of using python’s inspect module to track the code for this Node. I think this would work. The problem might be, that the Node imports code from a different file. If this file then changes, would DVC know? Is it possible to track all relevant code?
DVC can only run one task at a time. Thus integrating with kedro-accelrator would be impossible. Right?
To reiterate: The goal of kedro-dvc would not be the Data Version Control per se, but rather it would help during devlopment of kedro pipelines. You could be sure that the code/data is always up to date, but only have to run individual nodes - the rest can be skipped as the data is up to date.
The goal is not to keep the data versioned (I guess you should use versioned datasets xD)
Before I embark on this, I would like to ask the community on feedback.
Thanks!
Beta Was this translation helpful? Give feedback.
All reactions