Add import append flag to import new flags/segments only not existing in DB yet

[wtertius]

Closes: #2114

[wtertius]

Our developers keep flags and segments as code in flipt.yaml config in import format. So, it's convenient to add new flags to the same file. In some way it becomes a way to roll out migration of new flags and segments only without changing state of already existing flags and segments.

[erka]

Hey @wtertius.

Thank you for your contribution! Your use case is fascinating, and it would be helpful to understand it better. Since you already have a file containing all the flags, have you considered using a declarative storage? Is there any blocker?

[wtertius]

Since you already have a file containing all the flags, have you considered using a declarative storage? Is there any blocker?

Flipt is deployed to Kubernetes namespace with the service itself and its Posgres DB. We chose to use Postgres as Flipt storage basically because we have it already. But also there are other reasons:

1. Service owner can change flags in UI in realtime at prod environment and we won't loose data as we have automated backups
2. This allows us to keep more load using Postgres RO replicas for flag evaluating in the future (there is can be feature request for special DSN fo RO to not setup two Flipt instances - one for master and another one for evaluating)
3. This allows us to make seamless flipt rollout in Kubernetes (that achievable with declarative storage with shared volumes of cause)

[codecov]

Codecov Report

Attention: Patch coverage is 71.42857% with 16 lines in your changes missing coverage. Please review.

Project coverage is 64.22%. Comparing base (8795205) to head (116d6e4).

Files	Patch %	Lines
cmd/flipt/import.go	0.00%	8 Missing ⚠️
internal/ext/importer.go	83.33%	4 Missing and 4 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3299      +/-   ##
==========================================
+ Coverage   64.18%   64.22%   +0.03%     
==========================================
  Files         168      168              
  Lines       13506    13559      +53     
==========================================
+ Hits         8669     8708      +39     
- Misses       4174     4184      +10     
- Partials      663      667       +4     

Flag	Coverage Δ
unittests	64.22% <71.42%> (+0.03%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[markphelps]

Thank you for your contribution @wtertius !! It makes a lot of sense to me.

Correct me if Im wrong but this PR seems like it's very close to whats being asked for in #2114 ?

What do you think about instead changing this to an upsert operation? So that it adds any new flags/segments but also updates any that have changed?

I guess that would also require us to go through each of the subrecords too (like variants, rules, rollouts, constraints, etc) and check to see if they already exist as well and if so we do an update, and if not we do an create

WDYT @erka ?

[erka]

Correct me if Im wrong but this PR seems like it's very close to whats being asked for in #2114 ?

I think this will resolve that issue.

WDYT @erka ?

I am not sure about upsert. It looks like the database is the state of truth for a specific environment. This means product teams can directly modify flag rules and segments through the UI. While developers appreciate an easy way to introduce new flags/segments. I see it as mixin of declarative and relational storages or better to say a continuous deployment of Flipt for relational databases. Upsert the flags from file could revert the changes introduced by product teams and impact evaluation results.

[markphelps]

nit: i think maybe we should rename the internal value from newFlagsOnly to append or insertOnly like you have for the external facing command and update the description of the flag

[markphelps]

Correct me if Im wrong but this PR seems like it's very close to whats being asked for in #2114 ?

I think this will resolve that issue.

WDYT @erka ?

I am not sure about upsert. It looks like the database is the state of truth for a specific environment. This means product teams can directly modify flag rules and segments through the UI. While developers appreciate an easy way to introduce new flags/segments. I see it as mixin of declarative and relational storages or better to say a continuous deployment of Flipt for relational databases. Upsert the flags from file could revert the changes introduced by product teams and impact evaluation results.

yeah I think this will complete part of #2114, it wont fix the 'update' part requested in this comment:

causes the import process to update any existing flags/segments, and add new ones.

However it will fix the overall issue of not having the import fail for existing data. Im good with keeping this change as is then, just want to make sure we come up with the best name possible for the user facing CLI option and description

[markphelps]

looks great @wtertius ! one minor suggestion about err messages

[markphelps]

@all-contributors please add @wtertius for code

[allcontributors]

@markphelps

I've put up a pull request to add @wtertius! 🎉

[markphelps]

thanks @wtertius !!!