allow for loading initial information of a TestData from a file

[jmhodges-color]

Is your feature request related to a problem? Please describe.
We have very many feature flags in our codebase, many of which are permanent for partial availability reasons.

We'd like our devs' tests to be initialized with a known-good state of those feature flags that we'd store on disk and then use the TestData API to override those values in their tests.

Describe the solution you'd like

It'd be really nice for Files and TestData to have a way to cross through them.

Something like:

feature_store: FeatureStore = Files.load_from_file('/path/to/flagdata.json'); 
td = TestData(feature_store)
td.update(td.flag(...))

Describe alternatives you've considered
Currently, we try to not know much of anything about the schema of the flag data on disk from a dump of https://app.launchdarkly.com/sdk/latest-all

We have a subclass of TestData that takes a FeatureStore argument, and overrides _make_init_data to use the FeatureStore's values as the initial values in _TestDataSource's FeatureStore (with some jiggering of version). That could probably be made into how TestData operates if created with a FeatureStore.

However, getting the FeatureStore to pass into TestData filled with data is grody because there's no public way to turn file data into a FeatureStore directly without knowing details about the schema of the flag data.

One horrible hack imagined was creating a Config with an InMemoryFeatureStore as feature_store and a Files.new_data_source() as update_processor_class, passing that Config to an LDClient, and waiting for the LDClient to initialize. Then, using the contents of InMemoryFeatureStore as the initial FeatureStore our subclass uses and throwing away that Files-y LDClient.

Something like:

feature_store = InMemoryFeatureStore()
data_source_factory = Files.new_data_source(paths=["flagdata.json"], auto_update=False)
files_ldclient = LDClient(Config(sdk_key='sdk-fake', update_processor_class=data_source_factory, send_events=False, feature_store=feature_store)
td = OurTestData(feature_store) # See "Additional context"

Additional context

The subclass we wrote:

class OurTestData(TestData):
     def __init__(self, initial_feature_store: Optional[FeatureStore]):
         self.initial_feature_store = initial_feature_store
         super().__init__()
 
     def _make_init_data(self) -> dict:
         features = {}
         if self.initial_feature_store:
           features = self.initial_feature_store.all(FEATURES, lambda x: x)
         for k, v in self._current_flags.items():
             initial_v = features.get(k, None)
             if initial_v:
               v['version'] = initial_v['version']
             features[k] = v
 
         return {FEATURES: features}

[jmhodges-color]

Hm, there's a counter-argument here that's pretty strong that the default values should be enough. That's probably correct.

[eli-darkly]

The general idea of being able to combine two different data sources has been proposed before (including variations like "get real flag data from LaunchDarkly, but allow it to be selectively overridden"); it's been on the back burner, but we haven't ruled it out. However, there are some non-trivial design decisions that would have to be made to define what the behavior should be and deal with all the other parts of the SDK that have an expectation of "there is just one data store and one data source."