[feature request] Add numpy.genfromtxt()

[mw66]

https://forum.dlang.org/thread/recgifvrlcambwwkoobx@forum.dlang.org

Hi,

I'm just wondering what is the best way to read CSV data file into Mir (2d array) ndslice? Esp. if it can parse date into int/float.

I searched a bit, but can't find any example.

Thanks.

So, can we add numpy.genfromtxt():

https://numpy.org/doc/stable/reference/generated/numpy.genfromtxt.html

Esp. with column converters?

[mw66]

https://github.com/Kriyszig/magpie

BTW, found this for the reference:

from_csv(string path, int indexDepth = 1, int columnDepth = 1,int[] columns = [], char sep = ',')

[mw66]

For Mir to catch up with numpy, being able to easily read CSV to import data is a must to attract data scientists.

In numpy/pandas, it's just one liner.

[9il]

mir-ion can deserialize Mir and common 2D arrays from Binary Ion, Text Ion, JSON, MsgPack. Plus YAML support is coming soon.

We just need to add CSV there.

void serializeFromCsv(S)(scope ref S serializer, scop const(char)[] csv, /+ other params+/)
{
///on the fly
 /// - parse csv

 /// - recognise  cell patters: integers, floats, timestamps, true, false, null; all others are strings
 /// - serializer.putValue(value)
}

• The serialiser interface is here. https://github.com/libmir/mir-ion/blob/master/source/mir/ser/interfaces.d#L18
• [Timestamp](http://mir-algorithm.libmir.org/mir_timestamp.html).fromYamlString and Timestamp.fromString have recognition API
• For long and double recognition fromString

If we have this function implemented I will do the rest. The API will looks like:

auto matrix = csvText.deserializeCsv!(Slice!(2, double*));

[jmh530]

I don't really have a good sense of how to use mir.ion. I see that it references a cookbook, but it would be good to have a simple example written in D on the readme.md.

[jmh530]

auto matrix = deserializeCsv!(Slice!(2, double*));

Am I correct that the deserializeCsv would handle the transformation of what is read from a .csv file into the Slice type? It wouldn't handle the reading of the file itself.

[9il]

The fixed version should looks like:

auto matrix = csvText.deserializeCsv!(Slice!(2, double*));

deserializeCsv will call serializeFromCsv to serialize CSV to binary Ion DOM (it is super fast), and then, it will call deserializeValue, which is already implemented.

Mir always splits deserializ=sation logic in two stages: First, data to binary Ion; second, binary ion to value.

The idea is that binary ion to value works fast and unified across all types of formats.

[9il]

Assume raw major notation.

Then CSV should have the following options to be converted to Ion:

• matrix
• an array of records with inner keys in the first raw
• a record of arrays with outer keys in the first column
• a record of records with inner keys in the first raw and outer keys in the first column

These four kinds of conversion kinds allow converting CSV to Ion on the fly.

Also, a simple transposition with full memory allocation will allow other four conversions:

• transposed matrix
• an array of records with inner keys in the first column
• a record of arrays with outer keys in the first raw
• a record of records with inner keys in the first column and outer keys in the first raw

We can define CSV algebraic as follow:

module mir.algebraic_alias.csv;
/++
Definition union for $(LREF JsonAlgebraic).
+/
import mir.algebraic;
/++
CSV tagged algebraic alias.
+/
alias CsvAlgebraic = Algebraic!Csv_;

union Csv_
{
    /// Used for empty CSV scalar like one between two separators: `,,`
    typeof(null) null_;
    /// Used for false, true, False, True, and friends. Follows YAML conversion
    bool boolean;
    ///
    long integer;
    ///
    double float_;
    ///
    immutable(char)[] string;
}

[9il]

Added draft: https://github.com/libmir/mir-ion/blob/master/source/mir/csv.d

[jmh530]

I don't know if this applies or not, but one thing that is very useful in R's read.csv [1] function is the ability to identify certain strings as representing NAs. So for instance, setting it so that "NA" or "N/A" or "#N/A" (you could even do -999) are known as NAs, then the entire column will be the floating point type even if there were originally some strings in it. It makes it so you don't need to read the column as a string and process it later.

Not sure what other features that would be useful, but that's one that sticks out.

[1] https://www.rdocumentation.org/packages/utils/versions/3.6.2/topics/read.table

[mw66]

Also check numpy.genfromtxt() and pandas.read_csv().

I think handle invalid nan values (including empty entry i.e. ,,), and allow user pass in column converter callbacks are two most important features.

numpy.genfromtxt(fname, dtype=<class 'float'>, comments='#', delimiter=None, skip_header=0, skip_footer=0, converters=None, missing_values=None, filling_values=None, usecols=None, names=None, excludelist=None, deletechars=" !#$%&'()*+, -./:;<=>?@[\]^{|}~", replace_space='_', autostrip=False, case_sensitive=True, defaultfmt='f%i', unpack=None, usemask=False, loose=True, invalid_raise=True, max_rows=None, encoding='bytes', *, ndmin=0, like=None)[source]

https://pandas.pydata.org/docs/reference/api/pandas.read_csv.html

[mw66]

column converter callbacks

E.g. user can plug in his own date string to int/float converter function.

[9il]

@mw66 I can't see a reason why we may even need to provide an in-column converter callback while we have the power of mir.algebraic, mir.ndslice, and mir.functional. It may be confusing but the experience of people from scripting languages are and mine are so different that it hard to get why the want to do something one way while there is a 'common' way to do it.

Maybe we could share our experiences with each other.

Let's do the following. Please provide a CSV/TSV data sample and an example in any programming language of how you handle it. Then I will provide an example of how we can handle the data in Mir. When we do so it would be easy to figure out a good API.

[9il]

@mw66 I have updated the draft with callback support. Please check the first unittest.

[mw66]

why we may even need to provide an in-column converter callback

The reason is simple: the ndslice's type is homogeneous, it's just double/float, but the input can be mixed with date string: e.g.

Date,Open,High,Low,Close,Volume
2021-01-21 09:30:00,133.8,134.43,133.59,134.0,9166695
2021-01-21 09:35:00,134.25,135.0,134.19,134.5,4632863

You need the converter at import time, to convert the date string into double, not afterwards (how do you store string in the ndslice mixed with floats then)?

[mw66]

https://github.com/libmir/mir-ion/blob/master/source/mir/csv.d#L518

So, ndslice can have mixed types for different columns? (sorry if I miss something, I'm a newbie to mir).