Read CSV files quickly

[Beliavsky]

Since R is used for analyzing large data sets, there are several libraries for reading CSV files quickly, one of them being csvread, written in C++. It would be nice to have a fast CSV file reader in Fortran, even if it's just an interface to one in C or C++.

[arjenmarkus]

Tangential remark: A problem and a rather fundamental one that occurs with CSV files is that the format leaves a lot of freedom. Just think of the use of comma's, semicolons etc. You would need some detection mechanism or options. I do not know that library, but it should not be too hard to get something reasonably fast - these degrees of freedom are the main bottleneck I'd say. Op di 5 nov 2024 om 14:46 schreef Beliavsky ***@***.***>:

…

Since R is used for analyzing large data sets, there are several libraries for reading CSV files quickly, one of them being csvread <https://cran.r-project.org/web/packages/csvread/index.html>, written in C++. It would be nice to have a fast CSV file reader in Fortran, even if it's just an interface to one in C or C++. — Reply to this email directly, view it on GitHub <#884>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAN6YR5IJ4JTBN5RT25X2ILZ7DDZRAVCNFSM6AAAAABRGSZ5CCVHI2DSMVQWIX3LMV43ASLTON2WKOZSGYZTKNJRGMYDGNY> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[sakamoti]

I have never loaded large data, so I am not sure how fast it is for your use case, but as a package for reading CSV files in Fortran, there is csv-fortran. This package is also introduced on fortran-lang.org.

[jalvesz]

This could be developed using to_num_from_stream. The fastest apporach I have found to load large numeric ASCII files into memory is to load it fully into a single large string and then stream throught the string pointer.