fix(ingest/dremio): Dremio software jobs retrieval SQL query fix query error

[acrylJonny]

Checklist

• The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
• Links to related issues (if applicable)
• Tests for the changes have been added/updated (if applicable)
• Docs related to the changes have been added/updated (if applicable). If a new feature has been added a Usage Guide has been added for the same.
• For any breaking change/potential downtime/deprecation/big changes an entry has been made in Updating DataHub

[mayurinehate]

Adding a comment as to inconsistent type of queries_datasets across cloud and software would help for future readers.

[acrylJonny]

Adding a comment as to inconsistent type of queries_datasets across cloud and software would help for future readers.

Added comment on dremio_sql_queries.py pointing to incorrect documentation.

[BernardToure]

@acrylJonny , @mayurinehate ,

I'm using Dremio Cloud and I have sqlglot parsing errors because :
1 - the way Dremio deal with numbers and special characters in folders and table name
2 - the addition of branch name when using arctic to version view

For point 1, Dremio is using double quotes instead of backquotes around folders name in the from part of the queries
For point 2, Dremio refer to the source branch by adding AT BRANCH main after the table name.

I was able to stop sqlglot errors by using modified VIEW_DEFINITION and query column in the metadata-ingestion/src/datahub/ingestion/source/dremio/dremio_sql_queries.py file.

Before:
SELECT * FROM
(
SELECT
RESOURCE_ID,
V.TABLE_NAME,
OWNER,
PATH AS TABLE_SCHEMA,
CONCAT(REPLACE(REPLACE(
REPLACE(V.PATH, ', ', '.'),
'[', ''), ']', ''
)) AS FULL_TABLE_PATH,
OWNER_TYPE,
LOCATION_ID,
VIEW_DEFINITION,
FORMAT_TYPE,
COLUMN_NAME,
ORDINAL_POSITION, ...

After:
SELECT * FROM
(
SELECT
RESOURCE_ID,
V.TABLE_NAME,
OWNER,
PATH AS TABLE_SCHEMA,
CONCAT(REPLACE(REPLACE(
REPLACE(V.PATH, ', ', '.'),
'[', ''), ']', ''
)) AS FULL_TABLE_PATH,
OWNER_TYPE,
LOCATION_ID,
REGEXP_REPLACE(REGEXP_REPLACE(VIEW_DEFINITION,'(?i)AT BRANCH .?MAIN.?',''),'"','`') as VIEW_DEFINITION,
FORMAT_TYPE,
COLUMN_NAME,
ORDINAL_POSITION, ...

Same for the sys.project.history.jobs, replacing :
SELECT
job_id,
user_name,
submitted_ts,
query,

by
SELECT
job_id,
user_name,
submitted_ts,
REGEXP_REPLACE(REGEXP_REPLACE(query,'(?i)AT BRANCH .?MAIN.?',''),'"','`') as query,

These change replace all doublequotes by backquotes and remove AT BRANCH MAIN (case insensitives).

Can these changes be implemented and I'll be more than happy to test if you tell me how.

Cheers,
Berni