Update schemas to the latest format

[vchrombie]

ELK keeps a description for each enriched data used to build the KIbiter dashboards. Such descriptions are stored in the folder schema as CSV files. Over time, these descriptions have evolved and the current format is defined as a list of attributes that include the name, the type, whether the field can be aggregated and a description (e.g., schema/git.csv). Nevertheless, some schemas are still not aligned with the latest format. For instance, this is the case for:

• schema/stackoverflow.csv
• schema/meetup.csv

The goal of this issue is to update the schemas to the latest format. In order to do so, given a data source (e.g., meetup, stackoverflow), micro-mordred[*] should be executed to collect and enrich the data. Then, the enriched documents should be inspected using the dev tools or the discover of Kibiter. For each attribute found in the enriched index, the corresponding schema should contain the name of the attribute, the type, whether the field can be aggregated and a description.

You can also use this script for automating the process and creating the schema file from the index: generate-es-index-schema.py

Note that some fields like the grimoire_creation_date, project, project_1, origin, etc. are shared across all enriched indexes and their descriptions can be taken from existing schemas.

[*] Details to execute micro-mordred for a given data source are available at: supported-data-sources.

Related issues

• Initially reported at Update schemas to latest format #803
• Update bugzilla csv to latest format #836
• [schema] Add description, aggregatable and remove useless fields for each data schema #283

[prokan468]

I have worked with CSV files and python. Please do assign this issue to me and I shall provide you with the results.

[vchrombie]

I have worked with CSV files and python. Please do assign this issue to me and I shall provide you with the results.

Hi @prokan468, thanks for showing interest. We cannot assign this issue since it is a long one. Feel free to choose the backend, follow the steps, update the schema and open the PR.

Please let me know if you need any help.