Skip to content

Conversation

@jvansanten
Copy link
Contributor

The northern tracks data sample comes in a giant structured array with some unused columns. This is inefficient in several ways:

  • Reading the array from disk takes several seconds, as even the columns that are not used have to be read
  • Reading an individual column makes poor use of caches, as adjacent values in the column are separated by strides that are large compared to a cache line

Improve both of these by reading data from an npz file if present (unused columns are never read), and storing data column wise in astropy tables.

@jvansanten jvansanten force-pushed the improve-cache-coherency branch from eaa3d41 to 1712158 Compare July 8, 2025 14:40
@jvansanten jvansanten mentioned this pull request Jul 8, 2025
@jvansanten
Copy link
Contributor Author

Close in favor of #444

@jvansanten jvansanten closed this Sep 4, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant