You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
It would greatly simplify the whole system if format meta data was stored in the data files. This would mean that the IO layer could automatically open data files in the correct manner without having to be specifically directed by the high layers. This would simply everything by reducing coupling.
Information that should be stored in the files would include:
Whether entires are enumerated, and if enabled:
The relevant enumeration file(s)
Whether skip index is enabled on each column
The enumeration file format (JDBM vs TSV dump)
Whether compact format is enabled.
Character encoding
Ordering of the data.
Column names
Date written
Other information that may be useful but is harder to write in a header (because this information is not always available a priori) could include:
Number of records
Data statistics
Checksum
This feature could be be implemented with the addition of comments and structured escaping in the file format.
It would greatly simplify the whole system if format meta data was stored in the data files. This would mean that the IO layer could automatically open data files in the correct manner without having to be specifically directed by the high layers. This would simply everything by reducing coupling.
Information that should be stored in the files would include:
Other information that may be useful but is harder to write in a header (because this information is not always available a priori) could include:
This feature could be be implemented with the addition of comments and structured escaping in the file format.
The text was updated successfully, but these errors were encountered: