Adding overwrite or ignore behavior for row conflicts. #46232

LSturtew · 2022-03-04T21:26:16Z

@cvonsteg started the effort to implement this. Since they are unable to continue and this is a highly desired feature, I made an attempt to clean up the code and fix merge conflicts. I would like to share the credit for this PR (I do not know how this works).

A new parameter is added to to_sql: on_row_conflct

on_row_conflict : {'fail', 'overwrite', 'append'},
    default: 'fail'
    Determine insertion behavior in case of a primary key clash.
    - 'fail': Do nothing to handle primary key clashes, will raise an Error.
    - 'overwrite': Update existing rows in database with primary key clashes,
      and append the remaining rows with non-conflicting primary keys
    - 'append': Ignore incoming rows with primary key clashes, and
      insert only the incoming rows with non-conflicting primary keys

closes Adding (Insert or update if key exists) option to .to_sql #14553, continuation of Adding (Insert or update if key exists) option to .to_sql #14553 #29636
Tests added and passed if fixing a bug or adding a new feature
All code checks passed.
Added an entry in the latest doc/source/whatsnew/vX.X.X.rst file if fixing a bug or adding a new feature.

…t 88a5a481b git-subtree-dir: vendor/github.com/V0RT3X4/python_utils git-subtree-split: 88a5a481b5dbec610e762df862fd69918c1b77d4

git-vendor-name: python_utils git-vendor-dir: vendor/github.com/V0RT3X4/python_utils git-vendor-repository: git@github.com:V0RT3X4/python_utils.git git-vendor-ref: master

…tch function written down for deleting pkeys

…hed upsert ignore method

…t 88a5a481b git-subtree-dir: vendor/github.com/V0RT3X4/python_utils git-subtree-split: 88a5a481b5dbec610e762df862fd69918c1b77d4

git-vendor-name: python_utils git-vendor-dir: vendor/github.com/V0RT3X4/python_utils git-vendor-repository: git@github.com:V0RT3X4/python_utils.git git-vendor-ref: master

…ignore

jreback

i started reviewing, but this again is adding an enormous amount of complexity. can you get sqlalchemy to do this type of operation directly?

jreback · 2022-03-05T17:13:59Z

pandas/core/generic.py

@@ -2799,13 +2800,22 @@ def to_sql(
        schema : str, optional
            Specify the schema (if database flavor supports this). If None, use
            default schema.
-        if_exists : {'fail', 'replace', 'append'}, default 'fail'
+        if_exists : {'fail', 'replace', 'append'},\
+        default 'fail'


why are you u using row-continuation here? just leave it

can consider deprecating this and change the name to if_table_exists to be more obvious (but orthogonal)

pandas/core/generic.py

pandas/io/sql.py

jreback · 2022-03-05T17:15:22Z

pandas/core/generic.py

            How to behave if the table already exists.

            * fail: Raise a ValueError.
            * replace: Drop the table before inserting new values.
            * append: Insert new values to the existing table.

+        on_row_conflict : {'fail', 'overwrite', 'ignore'},\
+        default 'fail'


needs a versionadded 1.5.0

jreback · 2022-03-05T17:15:37Z

pandas/io/sql.py

    if if_exists not in ("fail", "replace", "append"):
        raise ValueError(f"'{if_exists}' is not valid for if_exists")

+    if on_row_conflict not in ("fail", "overwrite", "ignore"):
+        raise ValueError(f"'{on_row_conflict}' is not valid for on_row_conflict'")


is there a test for this?

yes:

def test_to_sql_invalid_on_row_conflict(self, pkey_frame): msg = "'notvalidvalue' is not valid for on_row_conflict" with pytest.raises(ValueError, match=msg): sql.to_sql( pkey_frame, "pkey_frame1", self.conn, if_exists="append", on_row_conflict="notvalidvalue", )

jreback · 2022-03-05T17:15:53Z

pandas/io/sql.py

    if if_exists not in ("fail", "replace", "append"):
        raise ValueError(f"'{if_exists}' is not valid for if_exists")

+    if on_row_conflict not in ("fail", "overwrite", "ignore"):
+        raise ValueError(f"'{on_row_conflict}' is not valid for on_row_conflict'")
+    # on_row_conflict only used with append


blank line before, comments can be inside the clause

jreback · 2022-03-05T17:16:13Z

pandas/io/sql.py

@@ -827,6 +850,182 @@ def create(self):
        else:
            self._execute_create()

+    def _load_existing_pkeys(self, primary_keys, primary_key_values):


type inputs & outputs

LSturtew · 2022-03-06T08:55:32Z

Sqlalchemy handles row conflicts differently depending on the database dialect. E.g: for postgress and sqlite the implementation is similar to what was originally proposed in the original PR: on_conflict_do_nothing or on_conflict_do_update, for mysql on the other hand this is handled with on_duplicate_key_update.

github-actions · 2022-04-06T00:05:01Z

This pull request is stale because it has been open for thirty days with no activity. Please update and respond to this comment if you're still interested in working on this.

maveec · 2022-04-27T12:20:55Z

@LSturtew seems little is missing to finalize this. Thank you for your effort and keep up the great work, lots of people are - and will be - grateful for this =)

hichameyessou · 2022-08-21T14:27:40Z

@LSturtew few months passed, do you still have the capacity to complete this PR?

JustinGuese · 2022-08-29T11:28:32Z

can we run this again? the logs for the tests are gone... Or what seems to be missing? I guess it is only a problem with the Docstrings and types, right?

JustinGuese · 2022-08-29T16:30:45Z

I have added a pr for some small docstring changes to the repo of @LSturtew, because if I create a new PR to the pandas main branch all the colab would be lost I guess?
Otherwise, if you (LSturtew) give editing rights to maintainers I can do it directly (https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/allowing-changes-to-a-pull-request-branch-created-from-a-fork)

LSturtew · 2022-08-29T16:51:32Z

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/allowing-changes-to-a-pull-request-branch-created-from-a-fork

The box Allow edits and access to secrets by maintainers is checked

JustinGuese · 2022-08-30T10:45:39Z

okay, i can't get it to work... maybe I'm just stuck in my thoughts

ERROR: Permission to LSturtew/pandas.git denied to JustinGuese.
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

Same when trying to do the same push to pandas-dev/pandas.git

I'm:

gh pr checkout -R pandas-dev/pandas 46232
adding my changes, git add & git commit
git push git@github.com:LSturtew/pandas.git sql-upsert:sql-upsert

what am I doing wrong? Do I need to create a separate PR? But I do not want the main work by everyone to be lost

mroeschke · 2023-01-16T19:42:54Z

Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.

cvonsteg added 30 commits October 12, 2019 16:37

updated .gitignore to ignore .python-version file

848becc

skeleton framework

d9f7c23

update docstring and add TODO

f0726b3

scratch functions added

c742f47

generator comment

d2a0c2d

Squashed 'vendor/github.com/V0RT3X4/python_utils/' content from commi…

686930c

…t 88a5a481b git-subtree-dir: vendor/github.com/V0RT3X4/python_utils git-subtree-split: 88a5a481b5dbec610e762df862fd69918c1b77d4

Add "python_utils" from "git@github.com:V0RT3X4/python_utils.git@master"

1f47d45

git-vendor-name: python_utils git-vendor-dir: vendor/github.com/V0RT3X4/python_utils git-vendor-repository: git@github.com:V0RT3X4/python_utils.git git-vendor-ref: master

Add "python_utils" from "git@github.com:V0RT3X4/python_utils.git@master"

881a934

git-vendor-name: python_utils git-vendor-dir: vendor/github.com/V0RT3X4/python_utils git-vendor-repository: git@github.com:V0RT3X4/python_utils.git git-vendor-ref: master

update scratch

c95de1d

update scratch

0082347

starting to modify sql.py adding workflow for insert method, and scra…

2d85c70

…tch function written down for deleting pkeys

starting to modify sql.py adding workflow for insert method, and scra…

0f32cd0

…tch function written down for deleting pkeys

sql.py - methods to get pkey columns, pkey iterator, and almost finis…

dc70c40

…hed upsert ignore method

sql.py - methods to get pkey columns, pkey iterator, and almost finis…

a7b8e8e

…hed upsert ignore method

changed upsert to if_exists option, beginning implementation

1d936ef

changed upsert to if_exists option, beginning implementation

9db2aaa

all workflow present - now to debugging

57a246d

all workflow present - now to debugging

c290a78

tidy up repo

35e0fc4

tidy up repo

6bcd6c2

Squashed 'vendor/github.com/V0RT3X4/python_utils/' content from commi…

00e6319

…t 88a5a481b git-subtree-dir: vendor/github.com/V0RT3X4/python_utils git-subtree-split: 88a5a481b5dbec610e762df862fd69918c1b77d4

Add "python_utils" from "git@github.com:V0RT3X4/python_utils.git@master"

8d654ca

git-vendor-name: python_utils git-vendor-dir: vendor/github.com/V0RT3X4/python_utils git-vendor-repository: git@github.com:V0RT3X4/python_utils.git git-vendor-ref: master

Add "python_utils" from "git@github.com:V0RT3X4/python_utils.git@master"

966a95c

git-vendor-name: python_utils git-vendor-dir: vendor/github.com/V0RT3X4/python_utils git-vendor-repository: git@github.com:V0RT3X4/python_utils.git git-vendor-ref: master

added df argument to stop original df from getting deleted by upsert_…

b214a68

…ignore

added df argument to stop original df from getting deleted by upsert_…

41938cd

…ignore

tidying up sql.py code and adding helpers to sql_scratch.py

78f8e86

tidying up sql.py code and adding helpers to sql_scratch.py

025b0ef

made upsert_delete mask index agnostic

bbcf92b

made upsert_delete mask index agnostic

73fea73

updated docuemntation for to_sql method

d8b7686

cvonsteg and others added 12 commits September 8, 2021 07:09

fix: tests working

8123cd7

chore: check metadata bind before reflect

26faabe

clean up docstrings

9260eca

merge: master

ad9f52f

merge: master

a63a77a

Merge branch 'main' into sql-upsert

a4df52b

Merge branch 'main' into sql-upsert

6190f6a

refactor: use

75358ac

pre-commit fix

a3e1fb6

Merge branch 'pandas-dev:main' into sql-upsert

21ec99d

clean up

53d6cc7

count updated rows for inserts with row conflict

a5fe55d

jreback requested changes Mar 5, 2022

View reviewed changes

jreback added the IO SQL to_sql, read_sql, read_sql_query label Mar 5, 2022

Luka Sturtewagen added 2 commits March 6, 2022 09:33

rename overwrite to replace

636f6c2

add type hints for new functions

67197c0

Luka Sturtewagen added 2 commits March 6, 2022 09:59

fix typo

b4b42f9

add typehints

3116cde

github-actions bot added the Stale label Apr 6, 2022

JustinGuese mentioned this pull request Aug 31, 2022

to_sql: Adding overwrite or ignore behavior for row conflicts #48331

Closed

4 tasks

mroeschke closed this Jan 16, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adding overwrite or ignore behavior for row conflicts. #46232

Adding overwrite or ignore behavior for row conflicts. #46232

LSturtew commented Mar 4, 2022

jreback left a comment

jreback Mar 5, 2022

jreback Mar 5, 2022

jreback Mar 5, 2022

LSturtew Mar 6, 2022

jreback Mar 5, 2022

jreback Mar 5, 2022

LSturtew commented Mar 6, 2022

github-actions bot commented Apr 6, 2022

maveec commented Apr 27, 2022

hichameyessou commented Aug 21, 2022

JustinGuese commented Aug 29, 2022

JustinGuese commented Aug 29, 2022

LSturtew commented Aug 29, 2022

JustinGuese commented Aug 30, 2022 •

edited

Loading

mroeschke commented Jan 16, 2023

Adding overwrite or ignore behavior for row conflicts. #46232

Adding overwrite or ignore behavior for row conflicts. #46232

Conversation

LSturtew commented Mar 4, 2022

jreback left a comment

Choose a reason for hiding this comment

jreback Mar 5, 2022

Choose a reason for hiding this comment

jreback Mar 5, 2022

Choose a reason for hiding this comment

jreback Mar 5, 2022

Choose a reason for hiding this comment

LSturtew Mar 6, 2022

Choose a reason for hiding this comment

jreback Mar 5, 2022

Choose a reason for hiding this comment

jreback Mar 5, 2022

Choose a reason for hiding this comment

LSturtew commented Mar 6, 2022

github-actions bot commented Apr 6, 2022

maveec commented Apr 27, 2022

hichameyessou commented Aug 21, 2022

JustinGuese commented Aug 29, 2022

JustinGuese commented Aug 29, 2022

LSturtew commented Aug 29, 2022

JustinGuese commented Aug 30, 2022 • edited Loading

mroeschke commented Jan 16, 2023

JustinGuese commented Aug 30, 2022 •

edited

Loading