fuse table implementations

[dantengsky]

Summary

• Data Ingestion

  • statistics accumulator (by using facilities of common data block, not parquet) #2193
  • interagte with data accessor #2194
  • enable proper compression/encoding algos for columns #1781
  • write parquet and gather statistics in one pass #1783
  • table level OCC transaction #1785
    • enable "remote meta service" for fuse table #2361
    • retry strategy of table commit #2383

• Data Scanning

  • Integration with Indexing
    • settle down cross components border APIs #1782
    • make sure the encoding of Min/Max values make sense #1801
    • enable VerifiableExpression in fuse table #2362
    • "optimise" the min/max value if necessary (keep the size of values of boud reasonable, sacrifices some accuracy)
  • Enable caching in table data readers  #1784
    Consider caching the following items
    • Table Snapshot
    • Table SegmentInfo
    • Parquet Meta Section
    • Column Data (of a block)

• Compact

  • distributed insertion
  • simple chunked insertion (by number of blocks) #3121
  • chunked insertion (by size of blocks) #3147
  • [fuse table] impl insert overwrite stmt #3144
  • syntax sugar : compact table  #3176
  • Proposal: introducing table functions that export fuse table meta #2946
  • fine-grained compaction

• misc

  • tracings

• Refactoring

  • split fuse table utils into mods #2378

• Testing

  • Unit tests
  • Integration Tests
    • Local FileSystem
    • Mock S3 Service (MinIO?)

[Xuanwo]

@dantengsky Most work on this tracking issue has been implemented. How about closing it and start a new one if needed?

[dantengsky]

@dantengsky Most work on this tracking issue has been implemented. How about closing it and start a new one if needed?

let close it :)