New Integration Request: Azure Table Storage

[geekwhocodes]

Tell us about the new integration you’d like to have

• Which source and which destination?
• Do you need a specific version of the underlying data source e.g: you specifically need support for an older version of the API or DB?

We use azure table storage extensively in our projects. It would be great if we add this as a source and eventually destination.

Describe the context around this new integration

• Which team in your company wants this integration, what for? This helps us understand the use case.
• How often do you want to run syncs?
• If this is an API source connector, which entities/endpoints do you need supported?

Currently, we don't see any use case around it but in my opinion this is very generic and widely used service.
There are REST APIs and python client library available.

Describe the alternative you are considering or using

What are you considering doing if you don’t have this integration through Airbyte?
I came across this awesome project from YC. Currently, I don't have use case in my mind but this is good to have byte.

Note: I would like to work on this source.

┆Issue is synchronized with this Asana task by Unito

[geekwhocodes]

Please let me know if anyone is already working on it :)

[michel-tricot]

Hey @geekwhocodes!!

I don't think anyone is working on it and you're most welcome to take a stab at it.

Do you need any help to get started?

[geekwhocodes]

@michel-tricot thanks for confirming it. I went through documentation and I think I am good for now but
I am sure I will need help in near future, will ask on Slack :)

[geekwhocodes]

folks,

I have few questions regarding Azure table storage connector. I am seeking some guidance to proceed further.

1. It does not have anything like table definition or pre-defined schema
2. There is no map reduce like functionality to fetch all properties

   You have to read all entities in side that table to fetch all properties available

Solutions :

1. Fetch first x entities to infer schema (not reliable)

2. Fetch/Read all entities in discover method/phase (not efficient)

3. There are three system properties that table storage provides - PartitionKey, RowKey, Timestamp. Can we define static schema like:
   PartitionKey
   RowKey
   Timestamp
   DynamicProperties - This will JSON serialized string which will contain all properties and values. (This breaks original document schema)

4. Instead of supporting all tables in storage account, Can we support one table at a time and ask end user to fill in column names(We will fetch first x entities and infer property type from it - just like you are doing it in MongoDB connector)

[marcosmarxm]

hey @geekwhocodes sorry for the delayed replied. I'm not super familiar with Azure Table Storage, but in a way is similar to S3/GCS storing JSON (slightly, maybe) can I suggest you take a look at File source connector implementation? https://docs.airbyte.io/integrations/sources/file
I'll take a look at it later today

[geekwhocodes]

Thanks @marcosmarxm, I will look into file connector's implementation.

[marcosmarxm]

Implemented #7095