distsqlrun: syncFlowConsumer unset when running internal executor queries

[asubiotto]

A panic was discovered in the vectorized flow setup code where Flow.syncFlowConsumer was nil. #38557 mitigates this by checking if the consumer is nil and returning an error which started producing these errors:

I190628 15:59:45.676023 3672 sql/distsql_running.go:180 [intExec=adopt-job] error vectorizing remote flow d6d8146f-f896-41d8-8c17-82bae5f1bbd3, restarting with vectorize=off and ID 10d20b4f-490d-48b6-943a-a1bc791f5102: syncFlowConsumer unset, unable to create materializer

These seem like bad plans generated by the distsql physical planner when running internal executor queries (read-setting is another example of when this happens). Essentially the node running the internal query generates a plan that says that the final receiver of data of that query should be on the remote node, which is not a state a query should ever be in (how do you get data back to the gateway if the final receiver is on a remote node?).

[yuzefovich]

I'm running into this problem when turning vectorize to "streaming."

This can be somewhat reliably (like with 50% probability) reproduced with setting vectorize to on in exec_util.go on alter_table logic test (for example, I used make testbaselogic TESTCONFIG=fakedist FILES=alter_table TESTFLAGS='-test.v -show-logs -show-sql -vmodule=column_exec_setup=2,server=1').

[yuzefovich]

An interesting observation is that the more logging I add (by, say, adding distsql_running=1 to vmodule) and the more my laptop is loaded, the less likely this failure reproduces.

[yuzefovich]

I've just confirmed that this issue appears to be caused by not properly using/releasing TableReaderSpecs, and the fix is in #38952.

[asubiotto]

Thanks for confirming!