Writable Declarative Store

[piclemx]

Problem

Currently, we are using the mode of importing the a file (configmap k8s) into flipt. This mode would start to become a problem if we reach the limit of the size of a configmap.

Ideal Solution

We could leverage the declarative store, but in a case of an object storage (ex. minio, aws s3). The lift is on the integrators to validate their file and lock it before saving it to the object storage.

The idea solution would be that flipt can handle this.

Those endpoints should maybe have an different authorization to avoid anyone to use it.

• Flipt should lock the file
• Flipt should validate the feature flag
• Flipt should store it to the file
• Flipt should always unlock after the call have been made

I think, it should be interesting to have a way to bulk change features flag in 1 call

Search

• I searched for other open and closed issues before opening this

Additional Context

No response

[GeorgeMac]

Thanks for raising this @piclemx this is more validation that folks want the UI and writes to the declarative backends.

I guess it could be argued this is a duplicate of #2497
In that issue I did focus on Git as the first backend to target and it influenced a lot of the design decisions there.
However, it does aim to be an avenue to support the rest of the backends. So unsure if we want to merge these thoughts into that issue or not.

As you mentioned, the big challenges are around all those lovely ACID database problems when all you have is e.g. a file.
Locking on the file is one way, but care needs to be taken when mapping the existing APIs that support writes onto this.
Our APIs today aren't particularly sympathic to a process like this.

Also, some of the declarative sources have some more favourable upstream mechanisms for handling concurrency control.
With Git we have immutable objects, branches and fast-forward only push/merge operations.
With OCI we have immutable objects and references and so on (no guarantees when updating references in the upstream though).

In that issue, I propose some ideas around a new write API for Flipt.
Something that is more sympathetic to how the declarative backends store state (i.e. idempotent PUT API containing the entire snapshot of e.g. a flag, segment or potentially entire namespace).

On top of this, I have prototyped this API for Git and toyed around with the idea of having all these changes go through a kind of staging flow. When a bunch of changes are ultimately "committed". Each users UI would be staging changes locally until commit. Commit is where a write to the upstream actually takes place. This could be the point where e.g. a file is written and pushed to object storage, or a branch in Git, or an OCI tarball and a reference moved.

This would need new UI concepts to explain that you changes aren't live until commit and potentially a conflict resolution phase.

This staging / commit process could also be incorporated with an approval flow.

Just a bunch of thoughts to add on here in case they interest you.

Do any of these ideas resonate with what you're imagining?

If you're interested in some of the gritty details my WIP is here https://github.com/flipt-io/flipt/tree/gm/git-writes
But theres a lot to unpack to play with it.

[piclemx]

@GeorgeMac, I think I would go with another set of an API. Like a management API stuff. At the end, the UI would still be in read-only mode. You could split the file in multiple files. Like one per ns.

At the end, I think flipt will keep is sqlite?

Staging part

I think you could do a minor change to flipt to use a different file or something.

I'm not sure I want to put our git as the source of it. We did have bad behaviour using that approach in the past.

[GeorgeMac]

@piclemx thanks for the feedback!

Agreed that a new management API is desireable.
In general we have some limitations with the current mutating APIs that impede some future feature goals.
For example, adding approval mechanisms becomes pretty intangible with our current API.

If we are to create a new management API I would love it to be one that the rest of the Flipt backends could move towards for management of state, including the relational ones. And for it to be appropriate for the UI to consume. We've already fragmented in the way Flipt works quite a bit across the various backends. A lofty goal perhaps 😂 but the current sprawl of the ways in which Flipt works is already a bit of a maintenance challenge. So anything we can do to create some consistency is super valuable.

On using a local file synced to object store, I think there a number of non-trivial challenges here.
Off the bat there is horizontal scale, though that is a bit of a moot point with sqlite, which already has similar limitations.
It could simply be a one instance only (at-least to start).

Then there is multi-user interaction and when to syncing state to S3.
Obviously, we can use file-level locking or even just in-memory locking if we're talking about one Flipt instance to manage concurrency control over the state. Though work will need to be done to manage the various namespace to file mappings and also how the in-memory cache is synced to the file. Currently, the format in memory is very different to the file format, it is indexed to be appropriate for the API's read operations (evaluations in particular). And the locking would apply to protect the cache and the file. We're in the realms of a flat-file database. All possible, just easy to overlook or get wrong.

It might be that actually, not writing this to disk and just doing it to memory and syncing that to object could be favourable. Depending on our tolerance for failure, the writes to object store could either be in the API transactions or asynchrous. The latter obviously leaving room for lost writes.

After saying all this though, perhaps just using the sqlite backend we have and asynchronously exporting that to object store is the same experience as async writes from memory or a file 🤷‍♂️?

I think some concept of branching or staging becomes important for e.g. approval, which has been something folks have asked of late and other FF offerings provide. We need a way to represent some delta to be applied after some approval. This is something where a new managent API really brings value, and things like Git bring some nice sympathies, plus if we did it right, we could create an experience that seemlessly supports folks that want to GitOps and folks that just want a UI that has versioning and approvals and so on. That is my dream there at-least 😂.

I'm not sure I want to put our git as the source of it. We did have bad behaviour using that approach in the past.

I would love to learn more about what when wrong with Git there, if you have any details on that?

Quick aside on Git: one thing I have wondered about is if we could get to a state where Git could even be embedded (like with sqlite), so you don't have to know about it, but you could if you wanted (e.g. the local or an upstream could be the source of truth). Then object store and OCI can become distribution targets. Where Flipt syncs the head of some target branch to e.g. S3 or OCI and so on for scaling the evaluation side of the equation and removing Git from the production scenario.
(That was my long term thought on hiding Git from those that want to know about it - at least).

[piclemx]

I was more thinking something like that. Not saying that flipt locally should handle a fs.

graph TD; Operator --> Management-API;
Management-API --> sqlite;
sqlite --> S3;

Loading

[GeorgeMac]

When you say Operator, are you thinking K8s operator / controller / CRD?

One thing we could do r.e. k8s operator is simply drop the database / object store components all together and treat the CRD as the source of truth. Like how we have relation and declarative backends. K8s own API resources could be a backend in that case.

[piclemx]

Yes I was thinking about that.

The only problem with it is the need to validate on the k8s side for something external