allow using Iterable in Series and DataFrame constructor #21987

holymonson · 2018-07-20T08:00:50Z

closes Cannot produce DataFrame from dict-yielding iterator. #2193
tests added / passed
passes git diff upstream/master -u -- "*.py" | flake8 --diff
whatsnew entry

@TomAugspurger Hope you can review this.

BTW, perhpas you may want to change Iterable in is_list_like() to Sequence.

gen = (i for i in range(10))
pandas.core.dtypes.inference.is_list_like(gen)
# True
len(gen)
# Traceback (most recent call last):
#   File "<stdin>", line 1, in <module>
# TypeError: object of type 'generator' has no len()
gen[0]
# Traceback (most recent call last):
#   File "<stdin>", line 1, in <module>
# TypeError: 'generator' object is not subscriptable

holymonson · 2018-07-20T10:38:01Z

This test failed. But a single-column DataFrame seems reasonable to me.

pandas/pandas/tests/frame/test_constructors.py

Lines 868 to 872 in 27ebb3e

    
           # corner, silly 
        
           # TODO: Fix this Exception to be better... 
        
           with tm.assert_raises_regex(ValueError, 'constructor not ' 
        
                                       'properly called'): 
        
               DataFrame((1, 2, 3))

pd.DataFrame((1, 2, 3))
#   0
#0  1
#1  2
#2  3

mroeschke · 2018-07-20T17:50:58Z

pandas/tests/series/test_constructors.py


        expected = Series(list(range(10)), dtype='int64')
-        result = Series(range(10), dtype='int64')
+        result = Series(Iter(), dtype='int64')


cant this simply be iter(range(10))?

range is list-like, subclass of Sequence, which doesn't need consuming by data = list(data). I'd like to add back range(10) test and keep both.

holymonson · 2018-07-21T02:58:32Z

Allow pd.DataFrame((1, 2, 3)) because pd.DataFrame([1, 2, 3]) is valid. They should be the same.

codecov · 2018-07-21T12:23:47Z

Codecov Report

Merging #21987 into master will decrease coverage by <.01%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master   #21987      +/-   ##
==========================================
- Coverage   92.02%   92.02%   -0.01%     
==========================================
  Files         170      170              
  Lines       50707    50704       -3     
==========================================
- Hits        46661    46658       -3     
  Misses       4046     4046

Flag	Coverage Δ
#multiple	`90.42% <100%> (-0.01%)`	⬇️
#single	`42.31% <80%> (ø)`	⬆️

Impacted Files	Coverage Δ
pandas/core/series.py	`94.11% <100%> (ø)`	⬆️
pandas/core/frame.py	`97.19% <100%> (-0.01%)`	⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 693e235...a2ace52. Read the comment docs.

gfyoung · 2018-07-22T22:46:31Z

pandas/tests/frame/test_constructors.py

+        expected = DataFrame([[1, 2, 3]] * 10)
+        result = DataFrame(Iter())
+        tm.assert_frame_equal(result, expected)
+


Reference the issue number in a comment below the functional definition.

Applies to all newly-added tests.

TomAugspurger · 2018-07-24T14:48:48Z

Allow pd.DataFrame((1, 2, 3)) because pd.DataFrame([1, 2, 3]) is valid. They should be the same.

Context: That goes back a long way to DataMatrix: c34ac74#diff-225c6e44f997b9b698ab52a48b223018R39

+1 for allowing it.

TomAugspurger

Merging later today, unless someone else beats me to it.

jreback · 2018-07-24T15:08:40Z

will have a look

don’t merge

jreback

pls add a whatsnew note, other enhancements in 0.24

jreback · 2018-07-25T10:07:38Z

pandas/core/frame.py

                                         copy=copy)
-        elif isinstance(data, (list, types.GeneratorType)):
-            if isinstance(data, types.GeneratorType):
+        elif (isinstance(data, collections.Iterable)


can you add a comment here (1 line above) which defines what this clause is checking

jreback · 2018-07-25T10:07:48Z

pandas/core/series.py

            elif isinstance(data, (set, frozenset)):
                raise TypeError("{0!r} type is unordered"
                                "".format(data.__class__.__name__))
+            elif (isinstance(data, collections.Iterable)


jreback · 2018-07-26T13:01:09Z

thanks @holymonson !

…21987)

holymonson force-pushed the iterable_in_constructor branch from b498a5c to d1c1bc2 Compare July 20, 2018 10:48

mroeschke reviewed Jul 20, 2018

View reviewed changes

holymonson force-pushed the iterable_in_constructor branch from d1c1bc2 to 4fb3b38 Compare July 21, 2018 02:52

gfyoung added Enhancement Internals Related to non-user accessible pandas implementation Low-Memory labels Jul 22, 2018

gfyoung requested a review from TomAugspurger July 22, 2018 22:45

gfyoung reviewed Jul 22, 2018

View reviewed changes

holymonson force-pushed the iterable_in_constructor branch from 4fb3b38 to 166f4f6 Compare July 23, 2018 03:10

TomAugspurger approved these changes Jul 24, 2018

View reviewed changes

jreback requested changes Jul 25, 2018

View reviewed changes

jreback added this to the 0.24.0 milestone Jul 25, 2018

holymonson added 3 commits July 25, 2018 18:47

allow using Iterable in Series and DataFrame constructor

5648790

add GH num to xfer

21f4b66

add doc

a2ace52

holymonson force-pushed the iterable_in_constructor branch from 166f4f6 to a2ace52 Compare July 25, 2018 11:33

jreback approved these changes Jul 26, 2018

View reviewed changes

jreback merged commit be6ad72 into pandas-dev:master Jul 26, 2018

Sup3rGeo pushed a commit to Sup3rGeo/pandas that referenced this pull request Oct 1, 2018

allow using Iterable in Series and DataFrame constructor (pandas-dev#…

e561986

…21987)

scarlehoff mentioned this pull request Aug 12, 2019

n3fit - Refactor of hyperopt NNPDF/nnpdf#516

Merged

Uh oh!

Uh oh!

allow using Iterable in Series and DataFrame constructor #21987

allow using Iterable in Series and DataFrame constructor #21987

Uh oh!

Conversation

holymonson commented Jul 20, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

holymonson commented Jul 20, 2018

Uh oh!

mroeschke Jul 20, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

holymonson Jul 21, 2018

Choose a reason for hiding this comment

Uh oh!

holymonson commented Jul 21, 2018

Uh oh!

codecov bot commented Jul 21, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

gfyoung Jul 22, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

TomAugspurger commented Jul 24, 2018

Uh oh!

TomAugspurger left a comment

Choose a reason for hiding this comment

Uh oh!

jreback commented Jul 24, 2018

Uh oh!

jreback left a comment

Choose a reason for hiding this comment

Uh oh!

jreback Jul 25, 2018

Choose a reason for hiding this comment

Uh oh!

jreback Jul 25, 2018

Choose a reason for hiding this comment

Uh oh!

jreback commented Jul 26, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

holymonson commented Jul 20, 2018 •

edited

Loading

mroeschke Jul 20, 2018 •

edited

Loading

codecov bot commented Jul 21, 2018 •

edited

Loading

gfyoung Jul 22, 2018 •

edited

Loading