sql: allow virtual table descriptors in a concrete db

[knz]

Requested by @petermattis in #10317 (comment): host the tables in cdb_internal in system instead.

To do this a couple of steps are needed:

1. defer the virtual description initialization code to the point after the system descriptor is created and available.
2. change the initialization code to allow a concrete db descriptor as parent descriptor for a virtual schema.
3. change the virtual table lookup code in data_source.go

@nvanbenschoten are there any other steps needed?

[knz]

Alternatively, we can hide the concrete system descriptor and create a new virtual schema descriptor also called "system" which proxies the system tables.

[dt]

This would be a side effect of a goal @nvanbenschoten, @a-robinson, @paperstreet and I have discussed a couple times: moving to having physical descriptors for virtual tables, just like for normal tables and views, with the only difference from a view being that they'd have an enum pointing to a row production function rather than a string query.

IIRC we all think such a unification would remove some special-cases and simplify some of the descriptor handling code, but we're waiting on that until we have a better story for upgrades -- right now, since the virtual tables are entirely synthetic, we can iterate on them without having to consider migrating older, persisted versions.

[knz]

Yes I sense there is much to be gained by persisting and versioning virtual table descriptors.

Versioning is another interest of mine (which extends to other things including our grammar and builtin functions), because as we change these semantics we run the risk of invalidating existing views that use them.

[nvanbenschoten]

Yeah I think this would be a natural result of persisting virtual descriptors (see #9361). The complication, as discussed above and in the issue, is that our descriptor migration story becomes much more critical. #9404 is a big step in the right direction, but its full implementation may be a ways off, and it still adds a lot of overhead to the currently trivial task of adding virtual tables. With information_schema and pg_catalog in flux, I'd like to avoid adding such a large barrier.

In the short-term, I wouldn't be opposed to adding some aliasing logic to virtual descriptors, so that we could hang virtual tables off of system. This may present some complications if we don't special-case system and try to make this work generally for all databases, but it's certainly doable.

[dt]

my $0.02: I'd rather we not add more special-cases to descriptor handling, and instead just wait for #9404 to do this The Right Way.

[nvanbenschoten]

I agree. If this issue isn't critical to resolve soon, I'd rather we just wait.

[knz]

I've been keeping an eye on this issue and I think the initial phrasing, as-is, is not relevant any more. With #22371 merged, the ship has sailed on the idea that there are virtual schemas that automagically appear in every database; that must be true for compatibility for information_schema and pg_catalog, and we made it true (for better or for worse) for crdb_internal.

The idea to "persist" virtual schemas that are to appear in every database doesn't make sense any more. I am closing this issue as a result.