How to store custom data for targets?

[doanac]

I'm currently carrying a patch out-of-tree I use to keep track of a Target's custom data so that I know what docker-apps to deploy:

doanac@40c51db

In its current form there are some things wrong with the patch:

1. I can/should use the ostree-mock approach to avoid my fakeGetCurrent hack
2. I'm writing a file directly to disk and this project has put in a decent amount of effort avoiding that by saving things in sqlite.

I'm curious is there's a way I could re-work this patch that might be acceptable to you guys? One idea I had was to add a column named something like custom-data to the installed_versions table. That would actually solve this problem universally and not just for docker-apps. However, the custom data would increase the size of each entry and I'm not sure any other code in the near/planned future would need this? I could ifdef something like that in and just activate for docker-apps, but that seems a little sloppy as well.

I'm open to anything here as it would be nice to get this code maintained/tested upstream.

[pattivacek]

We're still thinking about how best to address this, but one solution might be to make a new table specifically for holding that custom metadata, possibly per ECU.

One point is still confusing me a bit, though: why is re-reading the Target metadata from the json that we store in the meta table not an option?

[doanac]

One point is still confusing me a bit, though: why is re-reading the Target metadata from the json that we store in the meta table not an option?

Director sends the target w/o custom metadata. At least this was the situation when I was originally building this. I think it would be a nice improvement if Director could send that.

[pattivacek]

Director sends the target w/o custom metadata

Was the Director provided with the custom metadata? Or do you mean that the Director isn't copying the custom metadata provided to the Image repository? If the latter, that is by design, although it is presumably possible to change. (@simao may have more insight.) If the former, that is a bug. However, the only way I know to provide custom metadata to the Director is with ota-cli, which probably isn't an option if you're using our SaaS instance, but might be for you're running your own.

On a related note, we just recently implemented some logic to be smarter with URLs in the custom metadata from the Director and Image repos. I don't really see a good way to generalize that at present, but it at least starts the framework of comparing metadata received from the Director and Image repos and synchronizing them as necessary.

[doanac]

Was the Director provided with the custom metadata?

No. that's why I'm having to look things up now.

We're still thinking about how best to address this, but one solution might be to make a new table specifically for holding that custom metadata, possibly per ECU.

I guess that's fine. Seems like you'll wind up with one-to-one foreign key mappings so it might be easier to just track in the current installed_versions as its own column. I'm indifferent though.

[pattivacek]

No. that's why I'm having to look things up now.

I see. If you are able to provide the Director that metadata directly, that problem would be solved, right?

it might be easier to just track in the current installed_versions as its own column.

On second thought, you're probably right about that. How would you like to store it? Just the whole custom section as json?

[doanac]

If you are able to provide the Director that metadata directly, that problem would be solved, right?

Yes, but I think we'd have to change the director to make this work in a generic fashion.

How would you like to store it? Just the whole custom section as json?

I think so. Otherwise we are probably going to be guessing at what to store, miss something, and wind up needing some kind of migration logic.

[doanac]

Apologies, I've sort of let this die. However, I've started on a branch tonight to add a new column to installed_versions for this. I'll get a PR out once I've been able to test

[lbonn]

I think that this is now resolved, feel free to reopen if something is missing.