ENH: Add nullable dtypes to read_csv

[lithomas1]

• closes ENH: add option to get nullable dtypes to pd.read_csv #36712
• tests added / passed
• Ensure all linting tests pass, see here for how to run them
• whatsnew entry

[gfyoung]

As a whole looks pretty good, though with the growing number of null-related parameters, I think we will at some point need to condense into one "super parameter" (e.g. dict of some sorts).

[jbrockmendel]

can this be done at a higher level (i.e. not in the cython code)?

[lithomas1]

That would probably require a substantial refactoring of the Python Parser code, which calls this directly.

FWIW, maybe_convert_objects also has a convert_to_nullable_integer param.

[jorisvandenbossche]

@lithomas1 Cool! Thanks for working on this.

We now also have a nullable float dtype nowadays. I know that this is less useful at this point (since float already supports missing values, in contrast to integer/boolean), but I think that a use_nullable_dtypes keyword should probably also result in nullable float.

EDIT: and actually also the nullable "string" dtype.

As a whole looks pretty good, though with the growing number of null-related parameters, I think we will at some point need to condense into one "super parameter" (e.g. dict of some sorts).

@gfyoung I don't think this keyword is very related to the other read_csv null-related keywords? (or which ones were you thinking about?) And the use_nullable_dtypes keyword is also consistent with other IO methods (#29752, #31242)

[jorisvandenbossche]

We should probably use "boolean" dtype if use_nullable_dtypes=True regardless of whether NA's where actually present or not?

[lithomas1]

All NAs has unknown dtype, so we follow numpy and return float64. Currently, there has to be an integer or boolean value and NAs in order to return a nullable EA type, so a column of a float or bool will just return the numpy types.

This is just here as a test case to make sure I'm not doing the inferencing stuff wrong.

[gfyoung]

@gfyoung I don't think this keyword is very related to the other read_csv null-related keywords? (or which ones were you thinking about?) And the use_nullable_dtypes keyword is also consistent with other IO methods (#29752, #31242)

@jorisvandenbossche: They are related broadly speaking, but the consistency argument is fair as well.

I was just saying it's something to consider in the future, especially if we continue to add more arguments to an already long signature for read_csv (and the other I/O methods).

[jreback]

can you do a pre-cursor PR which does the lower level changes (e.g. to maybe_convert_*) and fully unit tests those. then we can layer to reader changes in this PR.

[lithomas1]

@jreback I think I will first try to get this feature-complete by adding in support for nullable float and string dtypes. Then I can spit out the inferencing changes. Marking this as draft in the meantime.

[simonjayhawkins]

The constructors for StringArray and ArrowStringArray are incompatible. To allow the global default for StringDtype backend storage (pyarrow or python) to be used here in the future (after #39908 is merged), we should probably call StringDtype.construct_array_type._from_sequence.

[jreback]

@lithomas1 status of this?

[lithomas1]

Depends on #41412, but we need to discuss the correct behavior in #42973.

[jreback]

@lithomas1 this might be easier these days?

[lithomas1]

Still stuck by at least #45168.

[jreback]

@lithomas1 status of this?

[mroeschke]

Thanks for the pull request, but it appears to have gone stale. Feel free to reopen when you have the time to revisit. Closing to clear out the queue.