add concat

[MarcoGorelli]

No description provided.

[rgommers]

Thanks Marco! concat being vertical-only matches my recollection, and overall this looks good modulo my minor comments. I'll comment on gh-136 on more of what was previously discussed about no horizontal concat.

[rgommers]

How about mentioning that column names must match? And must ordering match too?

[kkraus14]

We may want to document any row ordering guarantees or lack thereof as well. I.E. does this guarantee that for df1.concat([df2, df3]) that the row order is df1 rows, then df2 rows, then df3 rows?

[MarcoGorelli]

I presume so - or should that not be guaranteed?

[kkraus14]

I believe ANSI SQL doesn't guarantee any ordering of UNION ALL (which is the dataframe concat equivalent) unless there's also an ORDER BY clause (sort). For example, postgres doesn't guarantee any maintained ordering in a UNION ALL operation. So DataFrame APIs backed by a relational database would possibly struggle to meet that guarantee.

In general though, the often desired behavior is to maintain the order based on the passed in objects.

[MarcoGorelli]

i wasn't aware - ok thanks have added a note

[kkraus14]

I.E. Ibis (https://github.com/ibis-project/ibis) backed by basically any of its backends (https://github.com/ibis-project/ibis#backends) where the DataFrame(s) in question are the result of running joins.

There's no guarantees made in output ordering for the joins as well as no guarantees made in output ordering for the concatenation. It could potentially be worked around by doing something like using SQL ROW_NUMBER() functions to get ordering of the input tables to the concatenate call and then sort by that generated column afterwards, but that's somewhat expensive.

[kkraus14]

I think we need resolution here before we can merge.

[MarcoGorelli]

we've got Column.__getitem__, what does that return for an unordered column?

I don't know what the resolution is here, this discussion's a bit abstract for me - what do you suggest?

[kkraus14]

we've got Column.__getitem__, what does that return for an unordered column?

Potentially a random row or set of rows with no guarantee about determinism.

Maybe we can be a bit looser in the language and leave it how it currently is which is a bit underdefined which basically makes it implementation defined?

[MarcoGorelli]

ok - anything you'd like changed, or ok to merge?

[rgommers]

One thing that I missed the first time around looking at this is: why is concat a method rather than a free function? I looked for a previous discussion about it, but could not find it. The few I did find only addressed that the input should be a list of df's - I suspect most people would assume this would be a free function, just like in pandas.

[jorisvandenbossche]

I would also expect a function. But do we already have a full concept of a "namespace" from which this function can be accessed?

[rgommers]

But do we already have a full concept of a "namespace" from which this function can be accessed?

We have the concept I think, and a package where it can live. Just need to add a DataFrame.__dataframe_namespace__ method - I'll open a PR. Also need it to add a null object I believe.

[MarcoGorelli]

I wasn't aware of such a namespace - so in practice, if I wanted to concatenate a sequence of dataframes dfs, I'd do something like

    namespace = dfs[0].__dataframe_namespace__
    return namespace.concat(dfs)

?
Sounds fine if so. I'll update after Ralf's PR then

[rgommers]

I wasn't aware of such a namespace - so in practice, if I wanted to concatenate a sequence of dataframes dfs, I'd do something like

See gh-156.

[MarcoGorelli]

I've added a commit to make this a top-level function rather than a DataFrame method - OK to get this in?

[MarcoGorelli]

can add to index.rst after https://github.com/data-apis/dataframe-api/pull/157/files is merged

[MarcoGorelli]

I've amended the description to note that the ordering guarantees are only there if the DataFrame implementation has a concept of ordered columns - OK to get this in?

(arguably the same comment might have to be made for sort, which would be undefined for dataframes which don't have ordered columns)

[MarcoGorelli]

thanks all for reviews and comments!