Support schemaless

[jacwright]

Lyra requires a schema to be defined up front, providing some friction to using it with SaaS clients who must define a schema themselves before uploading their data.

It would be great to have the schema be automatically generated based off the shape of objects that are added. E.g. the first object with the property "age" on it as a number would add "age" to the schema of type "number".

I'm not sure the optimizations that would have to be dropped or complexity to this feature which might make it impossible.

One idea I had for accomplishing this from the outside would be to set a schema like this:

const db = await create({
  schema: {
    str1: "string",
    str2: "string",
    // ...
    str20: "string",
    num1: "number",
    num2: "number",
    // ... etc.
  },
});

then creating a property name lookup that could alter the fields of the objects on their way in and out such as:

const actualSchema = {
  author: 'str1',
  quote: 'str2'
};

Intercom describes running into issues with ElasticSearch schema maps growing unbounded and using a solution like this to allow all customer data to run side-by-side in the same index. I thought it might be a solution to implementing schemaless within Lyra.

[mateonunez]

Hi @jacwright, we discuss this on #171.

I had the same problem. That's why I've created a simple plugin (lyra-schema-resolver) to generate the based on the data you're trading on.

[jacwright]

I'll check it out, thank you!