"Concatenation will no longer sort" wording is very inconsistent

[quant5]

The wording in the update docs is (emphasis added):

In a future version of pandas pandas.concat() will no longer sort the non-concatenation axis when it is not already aligned. (...)

To keep the previous behavior (sorting) and silence the warning, pass sort=True.
To accept the future behavior (no sorting), pass sort=False

https://pandas-docs.github.io/pandas-docs-travis/whatsnew.html#concatenation-will-no-longer-sort

But the wording in the FutureWarning messages is (emphasis added):

Sorting because non-concatenation axis is not aligned. A future version
of pandas will change to not sort by default.

To accept the future behavior, pass sort=True.
To retain the current behavior and silence the warning, pass sort=False

https://github.com/pandas-dev/pandas/blob/c4da79b5b322c73d8e61d1cb98ac4ab1e2438b40/pandas/core/indexes/api.py

It needs to be clearer that current default is still sort=True but will revert to sort=False in a future version.

[TomAugspurger]

It's confusing because it's wrong :)

Duplicate of #21101 if you want to take a look at fixing it.

[techvslife]

also: Currently the warning doesn't get silenced if you pass "sort=False" (i.e. the true future default behavior)--presumably there should be no warning as long as the sort parameter is explicitly set (passed).

[TomAugspurger]

Do you have an example of a warning being emitted with `sort=False`?

…

On Mon, May 21, 2018 at 11:56 AM, techvslife ***@***.***> wrote: (also: Currently the warning doesn't get silenced if you pass "sort=False" (i.e. the true future default behavior)--presumably there should be no warning as long as the sort parameter is explicitly set (passed).) — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#21112 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABQHIpAD9lxmu-3099jbpALhS3ZyfSEAks5t0vG3gaJpZM4UD6_G> .

[techvslife]

Yes, what happens is that it does not produce the line of code in the warning (as it does when you omit the sort parameter), but it does still print the warning (this is the exact text and line spacing, not sure why the last line looks so strange):

C:\Users\mlopez\Anaconda3\lib\site-packages\pandas\core\frame.py:6201: FutureWarning: Sorting because non-concatenation axis is not aligned. A future version
of pandas will change to not sort by default.

To accept the future behavior, pass 'sort=True'.

To retain the current behavior and silence the warning, pass sort=False

  sort=sort)

[TomAugspurger]

Can you make a small reproducible example (with data) that emits the warning with `sort=False`?

…

On Mon, May 21, 2018 at 1:30 PM, techvslife ***@***.***> wrote: Yes, what happens is that it does not produce the line in the warning (as it does when you omit the sort parameter), but it does print the warning (this is the exact text and line spacing, not sure why the last line looks so strange): C:\Users\mlopez\Anaconda3\lib\site-packages\pandas\core\frame.py:6201: FutureWarning: Sorting because non-concatenation axis is not aligned. A future version of pandas will change to not sort by default. To accept the future behavior, pass 'sort=True'. To retain the current behavior and silence the warning, pass sort=False sort=sort) — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#21112 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABQHIgdb-iGD6Vf-ngWHubuYQP9gdN4uks5t0wfcgaJpZM4UD6_G> .

[techvslife]

Sorry, I'm not having luck reproducing with a small example; may not be a concat command, but a merge command instead. Unfortunately I don't know what line in code is triggering the warning from the warning.

[TomAugspurger]

You can make warnings raise. Something like ``` import warnings warnings.simplefilter("error") <your code> ```

…

On Mon, May 21, 2018 at 2:06 PM, techvslife ***@***.***> wrote: Sorry, I'm not having luck reproducing with a small example; may not be a concat command, but a merge command instead. Unfortunately I don't know what line in code is triggering the warning from the warning. — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#21112 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABQHIgfJzYneJoqF0VaPK1WMQhQlSp9lks5t0xAngaJpZM4UD6_G> .

[techvslife]

ok thanks, I'll try re-running it, but I won't be able to set up the full script today.

[techvslife]

Ok, I found it. It actually was an append command (not concat or merge). Adding the sort parameter did remove the warning. I added the sort=False parameter to this:
df_B = df_A.append(df_B, verify_integrity=True, sort=False).copy()

[TomAugspurger]

Thanks for checking.

…

On Wed, May 23, 2018 at 5:17 PM, techvslife ***@***.***> wrote: Ok, I found it. It actually was an append command (not concat or merge). Adding the sort parameter did remove the warning. I added the sort=False parameter to this: df_B = df_A.append(df_B, verify_integrity=True, sort=False).copy() — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#21112 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABQHItsIjgL3fXCxhDf7n0p4bNPqsMQnks5t1d_tgaJpZM4UD6_G> .

[allayli]

It seems that sort=None is NOT backward compatible.
Why NOT make it True by default? Or any Global config can set it be True?

[TomAugspurger]

Do you have an example where it isn't backwards compatible?

…

On Thu, Jun 7, 2018 at 10:04 AM, Li Sixiang ***@***.***> wrote: It seems that sort=None is NOT backward compatible. Why NOT make it True by default? Or any Global config can set it be True? — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#21112 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABQHIiaP5dcDeW2YvPtCfKxOfa_AcVZ6ks5t6UEFgaJpZM4UD6_G> .

[TasneemVKhan]

i had the same error I resolved it using
store_items = store_items.append(new_store , sort=True)

You just have to add sort =True to your code and the warning disappears

[Dougz00]

If you are using glob and a loop to read a large number of csv files this also works:
files = glob.glob(pattern)
df = pd.DataFrame()
for f in files:
df = df.append(pd.read_csv(f),sort=True)

[jreback]

If you are using glob and a loop to read a large number of csv files this also works:
files = glob.glob(pattern)
df = pd.DataFrame()
for f in files:
df = df.append(pd.read_csv(f),sort=True)

@Dougz00 that is exponential time

concatenation a list of data frames ; don’t append
see the append warnings in the docs