Extend data explorer for document-based data

[PhilippeMoussalli]

Inspecting the data can be difficult since long strings are stretched horizontally in the explorer's table. The table view needs to be automatically adjusted based on the available data

Tasks

Preview Give feedback

1. Restructure explorer #658
   +0
   
2. Enable better view of documents #631
   +0
   
3. Visualize an overview of the pipeline run #635
   +0
4. Gather feedback on additional RAG needed functionalities #660
   +0
5. Enable comparing different pipeline runs #634
   Data explorer +1
6. Enable filtering, seaching and sorting through documents #632
   +0
7. Assess options between aggrid and streamlit datasets #641
   +0
8. Redesign explorer after subset redesign #633
   +0
9. Document data explorer changes #659
   Data explorer documentation +2
   

[PhilippeMoussalli]

Things to improve

1) Enable exploring datasets prior to loading

This can be easily done if the dataset is stored in parquet format since Dask can interface with different backend storage with fsspec

2) Visualize overview of pipeline run with subsets and fields

Visualize evolution of a chosen pipeline and schema

3) Ability to export the dataset to different format for user desired visualization

** Structured data: Parquet -> pandas, csv **
** Unstructured data** -> Mainly images for now, integrate the existing script to export the dataset locally.

4) Better view of documents

• Enable hovering over a field to display the full content

• Ability to export or visualize in a separate tab (to be investigated)

5) Comparing different pipeline runs

Some runners already provide this functionalities so let's keep it simple, goal is to understand how different parameters might be affecting the data processing. Data points can be linked together by ID.

• Arguments (link arguments to certain outcome)
• Images side by side (outcome of different processing parameters/m on image processing e.g segmentation)
• Captions side by side (outcome of different processing parameters on image processing e.g. captioning )

6) Better understanding of components

• Display component spec or script for selected component

6) Other

• Filter
• Search
• Toggle

[RobbeSneyders]

1) Enable exploring datasets prior to loading

Not convinced that this is within scope. Datasets prior to loading can have any format. In any case, I don't think it's high priority.

2) Visualize overview of pipeline run with subsets and fields

I think this makes sense as an overview page indeed. Maybe even for navigation. We should think about how it fits in the explorer.

Note that we should still add this to the pipeline SDK for iterative development in the future as well.

3) Ability to export the dataset to different format for user desired visualization

Not completely sure what this means, but I would keep as much as possible inside the data explorer.

4) Better view of documents

This is the most important one for now, and we should go further. We should be able to show formatted documents.

• Properly show basic formatting like newlines etc.
• Show html documents
• Show pdf documents?

5) Comparing different pipeline runs

This could be a powerful one. We should think about:

• How to show this, because there's a third dimension (row, column, run)
• How to merge the two runs based on id

6) Better understanding of components

Not sure if this fits in the data explorer. It also doesn't have access to this content at this moment.

6) Filter, search, sort

These are important as well, and based on this link it seems like it is well supported with dataframes in Streamlit. This would be useful to eg. check all chunks of a document by filtering on the document_id and sorting on the chunk_id.

---

We should really tackle this with the RAG validation use case in mind, so I would focus on the following ranked priority-wise:

1. Better view of documents
2. Filter, search, sort
3. Comparing different pipeline runs