add unsafe_dynamic_query to gcp_bigquery_select

[jem-davies]

Will enable things like:

# {"table": "test.people", "where": "city IN (?,?,?)", "columns": ["name", "age", "city"], "args": ["London", "Paris", "Dublin"]}
pipeline:
  processors:
    - gcp_bigquery_select:
        project: ${GGP_PROJECT}
        table: ${! this.table } # test.people
        columns_mapping: root = this.columns #["name", "age", "city"]
        where:  ${! "city IN ("+this.args.join(",").re_replace_all("\\b\\w+\\b","?")+")" } # city IN (?,?,?)
        args_mapping: root = this.args # ["London", "Paris", "Dublin"]
        unsafe_dynamic_query: true

So that there is greater flexibility with this processor.

[jem-davies]

Suggested change

	project: ${GGP_PROJECT}
	project: ${GCP_PROJECT}

[jem-davies]

probably need to add a linter such that only 1 columns / columns_mapping field is provided

[jem-davies]

add better description - think about how we need to set unsafe_dynamic_query for this field to be evaluated.

[gregfurman]

Why can't we just have a single column field that either supports plain string or bloblang interpolation?

[jem-davies]

Suggested change

	# {"table": "test.people", "where": "city IN (?,?,?)", "columns": ["name", "age", "city"], "args": ["London", "Paris", "Dublin"]}
	# {"table": "test.people", "columns": ["name", "age", "city"], "args": ["London", "Paris", "Dublin"]}

[gregfurman]

Nice one here. Some questions/feedback.

Also, can we write a test for this? Just a unit test with this new interpolation functionality would be fine.

Stretch goal though: am thinking it could be nice to add an integration test using that bigquery emulator. Check our the setupBigQueryEmulator function and the TestGCPBigQueryStorageOutputWriteOK test for reference.

[gregfurman]

Why can't we just have a single column field that either supports plain string or bloblang interpolation?

[gregfurman]

I think we should add an experimental note here or some admonition that specifies this approach will likely be re-worked in future releases.

[gregfurman]

Either wrap this with an error message or just pass the err

[gregfurman]

Can we consider putting this into its own function? that way instead of this msg.SetError we can just return an err and set it once.

Thinking something like:

if proc.config.unsafeDyn {
  result, err := proc.executeDynamicQuery(msg)
  if err != nil {
    msg.SetError(err)
  }
}