Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PostgreSQL jsonb data type support #1335

Open
nj1973 opened this issue Nov 19, 2024 · 2 comments · May be fixed by #1339
Open

PostgreSQL jsonb data type support #1335

nj1973 opened this issue Nov 19, 2024 · 2 comments · May be fixed by #1339
Assignees
Labels
priority: p1 High priority. Fix may be included in the next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns.

Comments

@nj1973
Copy link
Contributor

nj1973 commented Nov 19, 2024

Test case

Oracle:

CREATE TABLE dvt_test.tab_json
(
  id VARCHAR2(10) NOT NULL PRIMARY KEY
, col_json1 CLOB
, col_json2 CLOB
);
ALTER TABLE dvt_test.tab_json ADD CONSTRAINT tab_json_chk1 CHECK (col_json1 IS JSON) ENABLE;
ALTER TABLE dvt_test.tab_json ADD CONSTRAINT tab_json_chk2 CHECK (col_json2 IS JSON) ENABLE;
INSERT INTO dvt_test.tab_json VALUES (1,'{"a": 1}','{"a": 1}');
COMMIT;

PostgreSQL:

CREATE TABLE dvt_test.tab_json
(
  id varchar(10) NOT NULL PRIMARY KEY
, col_json1 json
, col_json2 jsonb
);
INSERT INTO dvt_test.tab_json VALUES (1,'{"a": 1}','{"a": 1}');

Column validation
DVT command Oracle vs PostgreSQL:

$ data-validation validate column -sc ora_local -tc pg_local -tbls="dvt_test.tab_json" --count '*' --sum '*' --min '*' --max '*' --avg '*' --std '*' --filter-status fail
...
sqlalchemy.exc.ProgrammingError: (psycopg2.errors.UndefinedFunction) function length(json) does not exist
LINE 2: ..._json2 AS col_json2, length(t1.id) AS length__id, length(t1....
                                                             ^
HINT:  No function matches the given name and argument types. You might need to add explicit type casts.

The problem is that Oracle stores JSON data in a CLOB.

Row validation
DVT command Oracle vs PostgreSQL:

$ data-validation validate row -sc ora_local -tc pg_local -tbls="dvt_test.tab_json" -pk=id --hash=col_json1,col_json2
...
  File "/usr/local/google/home/neiljohnson/github/professional-services-data-validator/.venv/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
    cursor.execute(statement, parameters)
cx_Oracle.DatabaseError: ORA-00932: inconsistent datatypes: expected CLOB got CHAR
@nj1973 nj1973 added type: feature request 'Nice-to-have' improvement, new feature or different behavior or design. priority: p1 High priority. Fix may be included in the next release. labels Nov 19, 2024
@nj1973
Copy link
Contributor Author

nj1973 commented Nov 19, 2024

We should add the test columns above to the dvt_ora2pg_types integration test.

@nj1973 nj1973 self-assigned this Nov 19, 2024
@nj1973
Copy link
Contributor Author

nj1973 commented Nov 21, 2024

I feel this issue needs to be split into two.

  1. Prevent Oracle CLOB/NCLOB from generating exceptions when used in a normal context. This is the higher priority part to prevent command failures.
  2. Look at general JSON support and tests across all validation types. Including spotting Oracle CLOB/NCLOB used for JSON data. This is lower priority feature request.

I will complete item 1 on this issue now, I've opened #1338 for item 2 above.

@nj1973 nj1973 added type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns. and removed type: feature request 'Nice-to-have' improvement, new feature or different behavior or design. labels Nov 21, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
priority: p1 High priority. Fix may be included in the next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns.
Projects
None yet
1 participant