55654: xform: reorganize rules and tests for consistency r=RaduBerinde a=mgartner

#### xform: organize rules by the type of expressions they match

This commit organizes exploration rules by the type of expressions they
match. `GenerateZigzagJoins` and `GenerateInvertedZigzagJoins` have been
moved to `select.opt`. All other rules were correctly organized.

Release note: None

#### xform: reorganize tests to match the organization of the rules

This commit moves around (and makes some minor necessary changes) to
exploration tests so that tests for rules in `foo.opt` can be found in
`testdata/foo`. Most of the changes were related to zigzag join rules
which moved from `join` to `select`, and some tests for
`GenerateConstrainedScans` that have lived in `scan` for years.

The goal of this change is to make it easier to find tests for a
specific rule and to set a precedent for future tests.

Release note: None


Co-authored-by: Marcus Gartner <marcus@cockroachlabs.com>

…db#55722

55248: sql: add issue link to ALTER TABLE DROP COLUMN with sql_safe_updates r=lucy-zhang a=ajwerner

Explicit transactions with drop column are very dangerous. Inform the user.

Touches cockroachdb#51863.

Release note (sql change): Added a note to inform users with an issue link
when attempting potentially hazardous DROP COLUMN operations when
sql_safe_updates is enabled.

55655: xform: organize custom functions into general_funcs.go and scan_funcs.go r=RaduBerinde a=mgartner

#### xform: rename custom_funcs.go to general_funcs.go

This is in preparation for breaking up the custom functions into
multiple files, similar to the organization of normalization custom
functions.

Release note: None

#### xform: move scan-related custom functions to scan_funcs.go

This is one step in breaking up the exploration custom functions into
files based on the rules they facilitate, similar to the normalization
custom functions.

Release note: None


55721: roachtest: bump estimated max warehouses of tpccbench/nodes=3/cpu=16 r=nvanbenschoten a=nvanbenschoten

Now that we're importing instead of restoring and picking up a number
of optimizations along the way, this estimated max warehouse count is
too low. This is why we have been flatlining on:
https://roachperf.crdb.dev/?filter=&view=tpccbench%2Fnodes%3D3%2Fcpu%3D16&tab=gce

55722: sql: ban renaming table to a new database unless both schemas are public r=lucy-zhang a=lucy-zhang

Previously we would seemingly allow renaming tables to a new database
with arbitrary source and target schemas without ever actually
reassigning the schema ID. This could lead to a corrupted descriptor
with an invalid schema ID for the database. This PR fixes the
implementation to return an error when renaming a table to a different
database unless both the source and target schemas are `public`.

Fixes cockroachdb#55710.

Release note (bug fix): Previously, changing the parent database and
schema of a table using `RENAME` was seemingly permitted but would lead
to corruption of the table metadata. Now an error is returned when
attempting to rename a table to a different database except in the case
where both the source and target schemas are the `public` schema in each
database, which continues to be supported.

Co-authored-by: Andrew Werner <ajwerner@cockroachlabs.com>
Co-authored-by: Marcus Gartner <marcus@cockroachlabs.com>
Co-authored-by: Nathan VanBenschoten <nvanbenschoten@gmail.com>
Co-authored-by: Lucy Zhang <lucy@cockroachlabs.com>

The output of the SPLIT AT command returns keys in the form
`/Table/52/1/1` whereas the output of SHOW RANGES omits the
prefix, of table id and index id, and displays keys as `/1`.
This is a little confusing and making them consistent would
make it easier to visually compare.

This patch strips the prefix from the keys in the output of
SPLIT AT.

Release note (sql change): pretty column of SPLIT AT output was changed
by stripping prefix. This makes it consistent with output from SHOW
RANGES.

Empty (i.e. uninitialized engines) could receive tombstones before
being initialized. Initialization checks that the engine is empty
(save for the cluster version) and thus failed. Simply don't write
tombstones to uninitialized engines, which is fine since by the
time the callbacks fire, at least one is initialized anyway, and
besides, this mechanism is best effort.

The alternatives would have been to either allow tombstones to
be present on an engine that is being bootstrapped, or to give
the storage the option to defer writing to the engine once it's
bootstrapped. Neither seemed worth the extra work.

Fixes cockroachdb#55576.

Release note: None

55164: ui: extend diagnostics column to allow activate and download reports r=koorosh a=koorosh

Resolves cockroachdb#50824
~~Depends on cockroachdb/admin-ui-components#31
~~Depends on cockroachdb/yarn-vendored#42

Previously, Statements table had a Diagnostics column which allowed
users to request diagnostics reports for the first time and then
displayed status for requested report only. As result it wasn't
possible to download already generated report or request new one
report from the statements table.

With current changes, Diagnostics column allows requesting new
reports every time when previous reports are generated.
Also, it provides a list with links to download previous reports.

The main change is to provide a list of available (or requested)
reports for every statement (instead of a single, most recent
report as it was before). Then extracted `StatementsPage` component
(from `admin-ui-components` package) handles all rendering logic
for this list of reports.

Minor changes:
- `WAITING FOR QUERY` status is renamed to `WAITING` for new design
- `getDiagnosticsStatus` utility function is reused to reduce code
duplication

Release note (admin ui change): Diagnostics column (on statements table)
has been changed and includes `Activate` button and dropdown list to
download completed reports. Also, diagnostics badge status is changed from
`WAITING FOR QUERY` to `WAITING`

![Screen Shot 2020-10-01 at 4 55 32 PM](https://user-images.githubusercontent.com/3106437/94915373-77c62c80-04b5-11eb-8fef-4b33db15613b.png)


Co-authored-by: Andrii Vorobiov <and.vorobiov@gmail.com>

…tenant

Previously, accessing either of theses tables would lead to an internal error.
This was caused by defaulting to using a null node ID if the node ID was
unavailable, causing a null violation during validation.

The rest of internal tables simply use the zero node ID, so this commit fixes
crdb_internal.{leases,node_runtime_info} to use this behavior as well.

Release note: None

This commit fixes the normalization rule that converts st_distance to
st_dwithin or st_dwithinexclusive, which was broken in the case when
the use_spheroid parameter was used. Prior to this commit, the rule was
assigning the use_spheroid parameter as the 3rd parameter to st_dwithin
or st_dwithinexclusive and the distance parameter as the 4th, but that
order does not match the function signatures. This commit fixes the issue
by assigning distance as the 3rd parameter and use_spheroid as the 4th
if it exists.

Fixes cockroachdb#55675

Release note (bug fix): Fixed an internal error that could occur during
query planning when the use_spheroid parameter was used in the ST_Distance
function as part of a filter predicate. For example, `SELECT ... WHERE
ST_Distance(geog1, geog2, false) < 10` previously caused an error. This
has now been fixed.

55738: sql: fix crdb_internal.{leases,node_runtime_info} when accessed by a tenant r=tbg a=asubiotto

Previously, accessing either of theses tables would lead to an internal error.
This was caused by defaulting to using a null node ID if the node ID was
unavailable, causing a null violation during validation.

The rest of internal tables simply use the zero node ID, so this commit fixes
crdb_internal.{leases,node_runtime_info} to use this behavior as well.

Release note: None

Fixes cockroachdb#55701 

Co-authored-by: Alfonso Subiotto Marques <alfonso@cockroachlabs.com>

To implement cockroachdb#48775 I will be modifying the schema GC code to
add support for destroying tenants data. I notice3d that the
current code has redundant repetitions which makes it hard to
understand and modify.

The logic for computing new deadline and refreshing tables is
being repeated in 3 code blocks - I have merged them together.

Release note: none.

Currently, doing direct mutations on a RocksDB instance bypasses
custom batching / syncing logic that we've built on top of it.
This, or something internal to RocksDB, started leading to some bugs
when all direct mutations started passing in WriteOptions.sync = true
(see cockroachdb#55240 for when this change went in).

In this change, direct mutations still commit the batch with sync=true
to guarantee WAL syncing, but they go through the batch commit pipeline
too, just like the vast majority of operations already do.

Fixes cockroachdb#55362.

Release note: None.

55543: sql: make SPLIT AT output be consistent with SHOW_RANGES r=ajwerner a=neeral

This fixes cockroachdb#24740

The output of the SPLIT AT command returns keys in the form
`/Table/52/1/1` whereas the output of SHOW RANGES omits the
prefix, of table id and index id, and displays keys as `/1`.
This is a little confusing and making them consistent would
make it easier to visually compare.

This patch strips the prefix from the keys in the output of
SPLIT AT.

Release note (sql change): pretty column of SPLIT AT output was changed
by stripping prefix. This makes it consistent with output from SHOW
RANGES.

Co-authored-by: neeral <neeral.dodhia@gmail.com>

…db#55737

55567: geo/geomfn: implement ST_MinimumBoundingCircle r=otan a=C0rWin

This commit implements ST_MinimumBoundingCircle builtin function to
return polygon shape which approximates minimum bounding circle to
contain provided geometry.

Resolves: cockroachdb#48987

Signed-off-by: Artem Barger <bartem@il.ibm.com>

Release note (sql change): Returns polygon shape which approximates
minimum bounding circle to contain geometry

55604: kvserver: improve logging tags in multiTestContext r=andreimatei a=andreimatei

Release note: None

55616: colbuilder: disable wrapping of changefeed processors r=yuzefovich a=yuzefovich

The root of the problem is that `Columnarizer` has buffering behavior -
in 20.1 it will be hanging until `coldata.BatchSize()` (1024 by default)
rows are emitted by the changefeed. On 20.2 and master due to dynamic
batch size behavior it will still be hanging but in a slightly
different manner.

This is less of a problem on 20.2 and the current master because the
vectorized engine will not be used for the changefeed DistSQL flow since
the vectorized row count threshold is never met for it (the estimated
row count for the plan is 0, so unless a user does
`SET vectorize_row_count_threshold=0;` or
`SET vectorize=experimental_always;`, we will always use row-by-row
engine). In 20.1 the meaning of `vectorize=on` was different - we never
looked at the threshold and used the vectorized engine if it was
supported.

In order to fix this issue we simply refuse to wrap the changefeed
processors, so the row-by-row engine will be always used for changefeed
flows.

Fixes: cockroachdb#55605.

Release note (bug fix): The current implementation of changefeeds is
incompatible with the vectorized engine, so whenever the latter is being
used to run the former, it could hang indefinitely. This is now fixed.
Namely, on 20.2 releases this could happen if the user runs
`SET vectorize_row_count_threshold=0;`, and on 20.1 releases - if the
user runs `SET vectorize=on`.

55737: schemachange: refactor GC job code r=spaskob a=spaskob

To implement cockroachdb#48775 I will be modifying the schema GC code to
add support for destroying tenants data. I notice3d that the
current code has redundant repetitions which makes it hard to
understand and modify.

The logic for computing new deadline and refreshing tables is
being repeated in 3 code blocks - I have merged them together.

Release note: none.

Co-authored-by: Artem Barger <bartem@il.ibm.com>
Co-authored-by: Andrei Matei <andrei@cockroachlabs.com>
Co-authored-by: Yahor Yuzefovich <yahor@cockroachlabs.com>
Co-authored-by: Spas Bojanov <spas@cockroachlabs.com>

55708: storage: Use batches for direct RocksDB mutations r=itsbilal a=itsbilal

Currently, doing direct mutations on a RocksDB instance bypasses
custom batching / syncing logic that we've built on top of it.
This, or something internal to RocksDB, started leading to some bugs
when all direct mutations started passing in WriteOptions.sync = true
(see cockroachdb#55240 for when this change went in).

In this change, direct mutations still commit the batch with sync=true
to guarantee WAL syncing, but they go through the batch commit pipeline
too, just like the vast majority of operations already do.

Fixes cockroachdb#55362.

Release note: None.

55734: server: skip unit'ed engines in tombstone storage r=irfansharif a=tbg

Empty (i.e. uninitialized engines) could receive tombstones before
being initialized. Initialization checks that the engine is empty
(save for the cluster version) and thus failed. Simply don't write
tombstones to uninitialized engines, which is fine since by the
time the callbacks fire, at least one is initialized anyway, and
besides, this mechanism is best effort.

The alternatives would have been to either allow tombstones to
be present on an engine that is being bootstrapped, or to give
the storage the option to defer writing to the engine once it's
bootstrapped. Neither seemed worth the extra work.

Fixes cockroachdb#55576.

Release note: None


55739: opt: fix normalization of st_distance when use_spheroid parameter used r=rytaft a=rytaft

This commit fixes the normalization rule that converts `st_distance` to
`st_dwithin` or `st_dwithinexclusive`, which was broken in the case when
the `use_spheroid` parameter was used. Prior to this commit, the rule was
assigning the `use_spheroid` parameter as the 3rd parameter to `st_dwithin`
or `st_dwithinexclusive` and the `distance` parameter as the 4th, but that
order does not match the function signatures. This commit fixes the issue
by assigning `distance` as the 3rd parameter and `use_spheroid` as the 4th
if it exists.

Fixes cockroachdb#55675

Release note (bug fix): Fixed an internal error that could occur during
query planning when the use_spheroid parameter was used in the ST_Distance
function as part of a filter predicate. For example, `SELECT ... WHERE
ST_Distance(geog1, geog2, false) < 10` previously caused an error. This
has now been fixed.

Co-authored-by: Bilal Akhtar <bilal@cockroachlabs.com>
Co-authored-by: Tobias Grieger <tobias.b.grieger@gmail.com>
Co-authored-by: Rebecca Taft <becca@cockroachlabs.com>

Changes pulled in:
 - 3ff24428f10792b9a4e23129fca3d9ba47c15004 tool: Add `db checkpoint` command

Release note (cli change): Adds `cockroach debug pebble db checkpoint` debug
command to easily create a checkpoint without using rocksdb.

Release note: None

55751: vendor: Bump pebble to 3ff24428f10792b9a4e23129fca3d9ba47c15004 r=petermattis a=itsbilal

Changes pulled in:
 - 3ff24428f10792b9a4e23129fca3d9ba47c15004 tool: Add `db checkpoint` command

Release note (cli change): Adds `cockroach debug pebble db checkpoint` debug
command to easily create a checkpoint without using rocksdb.

Co-authored-by: Bilal Akhtar <bilal@cockroachlabs.com>

I noticed that there is no standardized prefix for commit
messages and PRs for RFC changes or additions. I would like to
add a recommendation to standardize the prefix to `rfc`.

Previous commits have used: `rfc`, `rfcs`, `docs`, `docs/rfcs`,
`docs/RFCs` and other variants.

Release note: None

Closes cockroachdb#55542.

This was causing node deaths to be initially ignored in test
failures like cockroachdb#55542. The monitor was not watching when the
cluster was restarted, so there was no need to inform it of
the restart. Because of this, the monitor had an accumulated
"death quota" that allowed it to ignore the first few deaths
during the actual run of the test.

I tested this 5 times without issue.

In v20.1, we increased the default max range size from 64MB to 512MB.
However, we only doubled (258b965) the default raft log truncation
threshold. This has the potential to exacerbate issues like cockroachdb#37906,
because each range will now handle on average 8 times more write load,
and will therefore be forced to truncate its log on average 4 times as
frequently as it had previously.

This commit bumps the default raft log truncation to 16MB. It doesn't go
as far as scaling the log truncation threshold by the max range size
(either automatically or with a fixed 32MB default) because the cost of
an individual log truncation is proportional to the log size and we
don't want to make each log truncation significantly more disruptive.
16MB seems like a good middle ground and we know that we have customers
already successfully running in production with this value.

Realease note: None

Release note: None

This commit re-enables the PartialIndexMutator for sqlsmith tests. This
mutator was disabled in cockroachdb#55724 after bug fixes in cockroachdb#55702 revealed
invalid schemas being created in sqlsmith. By reordering the mutators,
randomly generated schemas are no longer invalid.

Informs cockroachdb#55718

Release note: None

55755: rfc: add recommendation for RFC commit prefix to docs r=bdarnell a=angelapwen

I noticed that there is no standardized prefix for commit
messages and PRs for RFC changes or additions. I would like to
add a recommendation to standardize the prefix to `rfc`.

Previous commits have used: `rfc`, `rfcs`, `docs`, `docs/rfcs`,
`docs/RFCs` and other variants.

Release note: None

55760: authors: add amruss to authors r=otan a=amruss

Realease note: None

55763: authors: add ianjevans to authors r=otan a=ianjevans

Release note: None

Co-authored-by: angelapwen <angelaw@cockroachlabs.com>
Co-authored-by: Abigail Russell <abbey@cockroachlabs.com>
Co-authored-by: ianjevans <ian@cockroachlabs.com>

This is handy to ensure that the rate limiter does not become a bottleneck
in the face of many small acquisitions.

Release note: None

The bug was due to the fact that we were passing around the column descriptor
as a pointer to an entry in the `Columns` slice. When we overwrote that slice
we'd clobber the descriptor and, in the process, muck up the mutation which
pointed to it. We should look for more bugs like this. Prior to this change
the added logic test would fail with the below error:

```
testdata/logic_test/hash_sharded_index:589:
expected success, but found
(XXUUU) family "fam_1_k_j" contains unknown column "4"
```

Release note (bug fix): Fixed a bug which would prevent the dropping of hash
sharded indexes if they were added prior to other columns.

Currently, the databases page automatically loads stats and info
on all tables in all database simultaneously. This can cause heavy
load and contention on clusters with hundreds of tables and/or
databases.

This change introduces a quick fix to prevent the immediate problem
with the expectation that we will revisit the databases page for
a more thorough redesign in the near future.

This change adds a button next to each database in the
databases page titled "Load stats for all tables" which triggers
the queries to load table-specific stats for just that one database.
The tables will be loaded in sequence by converting the loading
loop to use async/await to ensure one query is executed at a time.

One problem this introduces is that the database stats summary
box on the right computes its data incrementally and will show
changing data as more tables load for a specific database. We have
chosen to not show the updated data until all table info has been
loaded for the given database.

Release note (admin ui change): Loading table-level stats requires
a button click per-database in order to prevent contention for
clusters with many databases and/or tables. In addition, the loading
of table data is staggered by table instead of triggered simultaneously
for all tables.

55767: authors: add Namrata Kodali to authors r=nkodali a=nkodali

Release note: None

Co-authored-by: Namrata Kodali <namrata@cockroachlabs.com>

Add constraint name to error for unique constraint, check constraint, and foreign key constraint violations. Those constraint names are then propagated over the PostgreSQL wire protocol and will show up for example in JDBC exceptions (org.postgresql.util.PSQLException#getServerErrorMessage().getConstraint()). This commit improves PostgreSQL compatibility of CockroachDB error messages.

Release note (sql change): Add constraint name to constraint errors, to increase wire level PostgreSQL compatibility.

55761: sqlsmith: enable partial index mutator r=yuzefovich a=mgartner

This commit re-enables the PartialIndexMutator for sqlsmith tests. This
mutator was disabled in cockroachdb#55724 after bug fixes in cockroachdb#55702 revealed
invalid schemas being created in sqlsmith. By reordering the mutators,
randomly generated schemas are no longer invalid.

Informs cockroachdb#55718

Release note: None

Co-authored-by: Marcus Gartner <marcus@cockroachlabs.com>

55559: ui: databases page requires click to load table info r=dhartunian a=dhartunian

Currently, the databases page automatically loads stats and info
on all tables in all database simultaneously. This can cause heavy
load and contention on clusters with hundreds of tables and/or
databases.

This change introduces a quick fix to prevent the immediate problem
with the expectation that we will revisit the databases page for
a more thorough redesign in the near future.

This change adds a button next to each database in the
databases page titled "Load stats for all tables" which triggers
the queries to load table-specific stats for just that one database.
In addition, table stats are staggered at 500ms intervals to prevent
an immediate triggering of potentially hundreds of queries
simultaneously.

One problem this introduces is that the database stats summary
box on the right computes its data incrementally and will show
changing data as more tables load for a specific database. We might
want to consider hiding the box until all the table stats load.

Release note (admin ui change): Loading table-level stats requires
a button click per-database in order to prevent contention for
clusters with many databases and/or tables. In addition, the loading
of table data is staggered by table instead of triggered simultaneously
for all tables.

Co-authored-by: David Hartunian <davidh@cockroachlabs.com>

We disallow columns using user-defined types in a different database
from the table at creation time. However, we have a loophole where it's
possible to move the table to a different database using `RENAME` and
thus create a cross-database reference. This breaks backup/restore and
is generally unintended. This PR adds a check to disallow this.

Release note (bug fix): Tables can no longer be moved to a different
database using `RENAME` if they have columns using user-defined types
(enums).

Previously, if tracing was enabled on a query that read bytes that
weren't decodable into datum, the database would crash.

Release note (bug fix): fix rare crash during tracing when reading
un-decodable data.

…db#55779

55459: kv: increase defaultRaftLogTruncationThreshold to 16MB r=nvanbenschoten a=nvanbenschoten

In v20.1, we increased the default max range size from 64MB to 512MB. However, we only doubled (258b965) the default raft log truncation threshold. This has the potential to exacerbate issues like cockroachdb#37906, because each range will now handle on average 8 times more write load, and will therefore be forced to truncate its log on average 4 times as frequently as it had previously.

This commit bumps the default raft log truncation to 16MB. It doesn't go as far as scaling the log truncation threshold by the max range size (either automatically or with a fixed 32MB default) because the cost of an individual log truncation is proportional to the log size and we don't want to make each log truncation significantly more disruptive. 16MB seems like a good middle ground and we know that we have customers already successfully running in production with this value.

55660: sql: add constraint name to constraint error r=yuzefovich a=alex-berger

Add constraint name to error for unique constraint, check constraintand foreign key constraint violations. Those constraint names are then propagated over the PostgreSQL wire protocol and will show up for example in JDBC exceptions (org.postgresql.util.PSQLException#getServerErrorMessage().getConstraint()). This commit improves PostgreSQL compatibility.

Release note: Improve PosgreSQL wire protocol compatibility by adding constraint name to sql errors.

55759: roachtest: don't expect node deaths in tpccbench r=nvanbenschoten a=nvanbenschoten

Closes cockroachdb#55542.

This was causing node deaths to be initially ignored in test
failures like cockroachdb#55542. The monitor was not watching when the
cluster was restarted, so there was no need to inform it of
the restart. Because of this, the monitor had an accumulated
"death quota" that allowed it to ignore the first few deaths
during the actual run of the test.

I tested this 5 times without issue.

55779: sql: disallow moving tables with user-defined types into a different DB r=lucy-zhang a=lucy-zhang

We disallow columns using user-defined types in a different database
from the table at creation time. However, we have a loophole where it's
possible to move the table to a different database using `RENAME` and
thus create a cross-database reference. This breaks backup/restore and
is generally unintended. This PR adds a check to disallow this.

Partially addresses cockroachdb#55709.
Related to cockroachdb#55772.

Release note (bug fix): Tables can no longer be moved to a different
database using `RENAME` if they have columns using user-defined types
(enums).

Co-authored-by: Nathan VanBenschoten <nvanbenschoten@gmail.com>
Co-authored-by: alex.berger@nexiot.ch <alex.berger@nexiot.ch>
Co-authored-by: Lucy Zhang <lucy@cockroachlabs.com>

54984: cli: add support for BIT type in dump r=pbardea a=pbardea

This commit adds support for the BIT type in dump.

Release note (bug fix): Previously, dumps of tables with a BIT type
column would result in an error. This column type is now supported.

Co-authored-by: Paul Bardea <pbardea@gmail.com>

This commit adds support for the syntax `... UNIQUE WITHOUT INDEX ...`,
both when adding UNIQUE constraints and when adding UNIQUE columns. Using
this syntax will currently return the error "unique constraints without
an index are not yet supported", but support for the syntax serves as a
starting point for adding support for these unique constraints.

Informs cockroachdb#41535

Release note (sql change): Added support for using the syntax
`... UNIQUE WITHOUT INDEX ...` in CREATE TABLE and ALTER TABLE statements,
both when defining columns and unique constraints. Although this syntax
can now be parsed successfully, using this syntax currently returns an
error "unique constraints without an index are not yet supported".

Previously, SHOW BACKUP would expect that the database and schema it
references is in the backup. In the case that we had a cross-db
reference, the schema or database that it references would not be
included in the backup, and would crash. It now reports objects that it
cannot find as NULL.

Release note (bug fix): Fix crash when performing a SHOW BACKUP against
a backup which contains a table that references a type in another
database. This state was only reachable in 20.2 release candidates and
earlier.

User facing change:
- New flag `--azure-availability-zone` lets the user supply which zone
a VM should be provisioned in.
- New flag `--azure-network-disk-type` lets the user choose which type
of network disk to use (ultra-disk, premium-ssd)

Previously, we didn't allow users to decide which availability zone a
VM was provisioned in. To achieve this, this patch adds support for
Network Security Groups. This was required as we had to update the
IP SKU from Basic (which is now deprecated) to Standard. By default,
this SKU blocks all inbound/outbound communication, which needed to be
overriden by creating network security rules that allow HTTPS/SSH/HTTP
connections. Network Security Groups are created under the resource
groups that manage VNets for a particular location the first time a
VNet is created.

Release note: None

55774: backupccl: fix crash in SHOW BACKUP when types have cross-db references r=lucy-zhang,ajwerner a=pbardea

Previously, SHOW BACKUP would expect that the database and schema it
references is in the backup. In the case that we had a cross-db
reference, the schema or database that it references would not be
included in the backup, and would crash. It now reports objects that it
cannot find as NULL.

Fixes cockroachdb#55772.

Release note (bug fix): Fix crash when performing a SHOW BACKUP against
a backup which contains a table that references a type in another
database. This state was only reachable in 20.2 release candidates and
earlier.

Co-authored-by: Paul Bardea <pbardea@gmail.com>

55700: sql: add syntax to create unique constraints without an index r=rytaft a=rytaft

This commit adds support for the syntax `... UNIQUE WITHOUT INDEX ...`,
both when adding `UNIQUE` constraints and when adding `UNIQUE` columns. Using
this syntax will currently return the error "unique constraints without
an index are not yet supported", but support for the syntax serves as a
starting point for adding support for these unique constraints.

Informs cockroachdb#41535

Release note (sql change): Added support for using the syntax
`... UNIQUE WITHOUT INDEX ...` in CREATE TABLE and ALTER TABLE statements,
both when defining columns and unique constraints. Although this syntax
can now be parsed successfully, using this syntax currently returns an
error "unique constraints without an index are not yet supported".

Co-authored-by: Rebecca Taft <becca@cockroachlabs.com>

Release note: None

55551: keys: improve printing of node tombstone key r=irfansharif a=tbg



Co-authored-by: Tobias Grieger <tobias.b.grieger@gmail.com>

StartServer was using this odd pattern where it took both a `*testing.T`
and returned an `error`. This method was then also called on a goroutine
whenever `ParallelStart` was specified.

I wasted quite a bit on these errors (things just hang, really hard to
debug) so I cleaned this method up to always return an error.

Release note: None

55550: testcluster: avoid t.Fatal on goroutine r=knz a=tbg

StartServer was using this odd pattern where it took both a `*testing.T`
and returned an `error`. This method was then also called on a goroutine
whenever `ParallelStart` was specified.

I wasted quite a bit on these errors (things just hang, really hard to
debug) so I cleaned this method up to always return an error.

Release note: None


Co-authored-by: Tobias Grieger <tobias.b.grieger@gmail.com>

Before, Session page has filters menu which used default Antd component
and it deviated common styles.

Now, existing `Dropdown` component is extended to be used in Sessions
page with extra properties to get more flexibility for customization.
And this Dropdown is used instead.

Release note: none

Previously, "back links" from details page to main
pages had regression on styles and didn't satisfy
current designs.

Now, links by default black and have blue color on
hover only. Also styles for links inside SortedTable
component is fixed to inherit font size and font styles
from parent elements

Release note (admin-ui-change): Fix link colors for "Back" link on
Details pages (Node Overview, Jobs, Sessions, and statements
details pages.

Previously, `Pagination` component was used from Antd design system
and every page component added it's own duplicate of custom styles
on top of Antd Pagination. This introduced multiple places which
have to changed in order to unify styles across entire app.

Now, main `Pagination` component is added to `admin-ui-components`
package which is basically wrapper on top of Antd component with
custom styles. Also, all existing duplicates for styles are removed
and it avoids confusion to know which Pagination is correct one.

In addition, styles which were duplicated from StatementsPage into
SessionsPage were re-imported in it's own styles to keep single
places for changes.

Release note: none

Session details page had inconsistent styles where headers for sections
had wrong sizes, spacings between sections were larger than in designs,
and positions for action buttons (Cancel query and session buttons)
had to be placed in header section.

Now, styles are adjusted according to designs and fixed in a way to
avoid global changes. The only one global change is made to reset
`line-height` property for H1 element which were inherited from <body>
element and caused wrong text positioning.

Release note (admin ui change): adjusted styles for Session details page

55766: sql: fix bug dropping shard column which is not the last column r=lucy-zhang a=ajwerner

The bug was due to the fact that we were passing around the column descriptor
as a pointer to an entry in the `Columns` slice. When we overwrote that slice
we'd clobber the descriptor and, in the process, muck up the mutation which
pointed to it. We should look for more bugs like this. Prior to this change
the added logic test would fail with the below error:

```
testdata/logic_test/hash_sharded_index:589:
expected success, but found
(XXUUU) family "fam_1_k_j" contains unknown column "4"
```

Release note (bug fix): Fixed a bug which would prevent the dropping of hash
sharded indexes if they were added prior to other columns.

Co-authored-by: Andrew Werner <ajwerner@cockroachlabs.com>

The multiregion setup is missing a comma, preventing deployments ran using this file from being accessed through `cockroachdb-public.default`.
This adds in the extra comma and fixes the issue.

55790: build: add bazel-3.7.0 to builder image r=irfansharif,jlinder a=otan

Resolves cockroachdb#55693 

Release note: None

Co-authored-by: Oliver Tan <otan@cockroachlabs.com>

55812: Fix multiregion setup r=DuskEagle a=Tayo-00

The multiregion setup is missing a comma, preventing deployments ran using this file from being accessed through `cockroachdb-public.default`.
This adds in the extra comma and fixes the issue.

Co-authored-by: Tayo <l.lee@tayo.ws>

Release note: None

This commit reworks the SQL liveness subsystem to start on the local node
following the migrations that it relies upon. This is both safe and ensures
that jobs created on the new binary will have a session ID, thus avoiding
the issue that startable jobs created on the local node may not have had
a session ID and thus would not have had a proper claim to run the job
locally.

Fixes cockroachdb#55497.

Release note (bug fix): Fixes a bug which could cause IMPORT, BACKUP or
RESTORE to experience an error when they occur concurrently to when the
cluster sets its version to upgraded.

Previously, the max mutation batch size variable was duplicated across
optimizer and execution and couldn't be modified at program start time
for metamorphic testing. This is now corrected.

Release note: None

55720: builtins: add sha224 and sha384 functions r=rafiss a=timgraham

Fixes cockroachdb#42450

Release note (sql change): Add sha224 and sha384 functions.

55815: sql: refactor max batch size to unblock metamorphic tests r=jordanlewis a=jordanlewis

Previously, the max mutation batch size variable was duplicated across
optimizer and execution and couldn't be modified at program start time
for metamorphic testing. This is now corrected.

Co-authored-by: Tim Graham <timograham@gmail.com>
Co-authored-by: Jordan Lewis <jordanthelewis@gmail.com>

Release note: None

Release note: None

…db#55824

55524: jobs,sqlliveness,server: start sqlliveness and pgserver after migrations r=spaskob a=ajwerner

This commit reworks the SQL liveness subsystem to start on the local node
following the migrations that it relies upon. This is both safe and ensures
that jobs created on the new binary will have a session ID, thus avoiding
the issue that startable jobs created on the local node may not have had
a session ID and thus would not have had a proper claim to run the job
locally.

Fixes cockroachdb#55497.

Release note (bug fix): Fixes a bug which could cause IMPORT, BACKUP or
RESTORE to experience an error when they occur concurrently to when the
cluster sets its version to upgraded.

55775: roachprod: add availability zone and network disk type support to azure r=miretskiy a=arulajmani

User facing change:
- New flag `--azure-availability-zone` lets the user supply which zone
a VM should be provisioned in.
- New flag `--azure-network-disk-type` lets the user choose which type
of network disk to use (ultra-disk, premium-ssd)

Previously, we didn't allow users to decide which availability zone a
VM was provisioned in. To achieve this, this patch adds support for
Network Security Groups. This was required as we had to update the
IP SKU from Basic (which is now deprecated) to Standard. By default,
this SKU blocks all inbound/outbound communication, which needed to be
overriden by creating network security rules that allow HTTPS/SSH/HTTP
connections. Network Security Groups are created under the resource
groups that manage VNets for a particular location the first time a
VNet is created.

Release note: None

55820: roachtest: update 20.2 predecessor version to 20.1.8 and create fixtures r=jlinder a=asubiotto

Release note: None

55824: cloud: update orchestrator configs to point to v20.1.8 r=jlinder a=asubiotto

Release note: None

Co-authored-by: Andrew Werner <ajwerner@cockroachlabs.com>
Co-authored-by: arulajmani <arulajmani@gmail.com>
Co-authored-by: Alfonso Subiotto Marques <alfonso@cockroachlabs.com>

Previously, we used RocksDB's file locking mechanism for temp directory
locking. As RocksDB is being removed from the codebase, move to using
Pebble's syscall-based mechanism.

Release note: None.

Add parsing and telemetry support for DROP OWNED BY command.
Includes small refactors of the related REASSIGN OWNED BY
command such as appending "_by" to some instances of
"reassign_owned" for readability.

Plan to follow up with implementation of DROP OWNED BY for
feature parity with PostgresQL, which will resolve the issue
linked.

Release note: None

Add a logictest configuration that has three regions, with three
nodes in three different AZs.

Release note: None

A comment said that the start time of an epoch-based lease is
ignored, but that's not true. For example, the tscache is reset to the
lease's start time.

Release note: None

55830: roachpb: fix comment on Lease.Start r=andreimatei a=andreimatei

A comment said that the start time of an epoch-based lease is
ignored, but that's not true. For example, the tscache is reset to the
lease's start time.

Release note: None

Co-authored-by: Andrei Matei <andrei@cockroachlabs.com>

This change deletes all RocksDB specific Go code, and updates
all tests that called RocksDB-specific methods to call their Pebble
versions instead.

This is a prerequisite for deleting RocksDB from the codebase.

Release note (general change): RocksDB can no longer be used as
the storage engine. Passing in --storage-engine=rocksdb will
return an error.

As --storage-engine=rocksdb no longer works as a command line
argument, update the help text for that flag to reflect its nature.

Also skip engine/switch, and inconsistency roachtests as they use
RocksDB-only functionality.

Release note (cli change): Update --storage-engine help text to
reflext RocksDB deletion.

This change updates the Makefile to not build snappy, and rocksdb.
Libroach is built without those two libraries as well.

Release note: None.

This change is a simple file deletion of:
c-deps/{rocksdb,snappy}{,-rebuild}

Release note: None

This change deletes all of Libroach except for the methods
necessary to implement DBDumpThreadStacks. Also moves the Go-side
code to pkg/storage/stacks.go.

Release note: None.

55816: sql: allow parsing of DROP OWNED BY r=otan a=angelapwen

Part 1 of addressing cockroachdb#55381.

Add parsing and telemetry support for DROP OWNED BY command.
Includes small refactors of the related REASSIGN OWNED BY
command such as appending "_by" to some instances of
"reassign_owned" for readability.

Plan to follow up with implementation of DROP OWNED BY for
feature parity with PostgresQL, which will resolve the issue
linked.

Note that this command is also making use of `role_spec_list` 
which as noted in cockroachdb#54696 does not properly parse 
`CURRENT_USER` or `SESSION_USER` and uses strings for
roles/usernames rather than expressions. This issue is out of 
scope for this PR and will be resolved separately.

Release note: None

Co-authored-by: angelapwen <angelaw@cockroachlabs.com>

55784: logictest: add multiregion configuration r=ajstorm a=otan

Add a logictest configuration that has three regions, with three
nodes in three different AZs.

Release note: None


Co-authored-by: Oliver Tan <otan@cockroachlabs.com>

Makes the `git` invocations a bit more useful.

Release note: None

Release note: None

In all uses of BuildScalar, we map variables directly to columns in
the metadata. This change simplifies the interface and makes an
indexed var container internally.

Release note: None

This change reworks the infrastructure for tests involving scalars.
Instead of passing a list of types and using thing like `@1` to refer
to variables, we define variables with names. This makes the tests a
lot cleaner and doesn't rely on an obscure syntax. It will also allow
extending the variable definitions (e.g. computed column expressions).

The new infrastructure will be used for other scalar tests (most
importantly, index constraints and partial predicate implication).

Release note: None

Release note: None

Release note: None

Release note: None

Release note: None

55843: roachpb: track a few auto-generated files as binary files r=irfansharif a=irfansharif

Makes the `git` invocations a bit more useful.

Release note: None

55844: keys: update doc.go to include node tombstone key r=irfansharif a=irfansharif

Release note: None

Co-authored-by: irfan sharif <irfanmahmoudsharif@gmail.com>

55509: *: Remove RocksDB r=itsbilal a=itsbilal

As in title. Removes RocksDB and snappy, from the `c-deps` directory,
and updates tests and roachtests to reflect a Pebble-only world. Libroach
is slimmed to just the one method (`DumpThreadStacks`) that is still necessary.

As this is a massive change, I've tried to make it as easy to review as possible by
splitting changes in different commits. The first commit just updates temp file locking,
the second commit deletes RocksDB-specific code paths and updates the few remaining
RocksDB-only tests to use equivalent Pebble code paths, the fourth commit has
Makefile changes, and the last commit is a plain file deletion in `c-deps`. 

This is ready for review, though the following relatively-minor tasks are still TODO:

- [x] Resolve merge conflicts
- [x] Pass lint
- [x] Bring back thread stack dumps as they may be necessary for inspecting other CGo code

Co-authored-by: Bilal Akhtar <bilal@cockroachlabs.com>

55499: ui: update styles for session details page r=koorosh a=koorosh

resolves cockroachdb#54513 
depends on cockroachdb/admin-ui-components#31
depends on cockroachdb/admin-ui-components#34
depends on cockroachdb/admin-ui-components#35

Current fix is quite complex and touches a lot of places because most of the issues with styles were common for all pages (Statements, Jobs, Node Overview).
Also many style duplicates required to get rid of duplicates and unify single version of proper components to use across entire app.

It's easier to review this PR per commit. Each commit describes in details what changes were made and their purpose.
 
![Screen Shot 2020-10-13 at 3 18 18 PM](https://user-images.githubusercontent.com/3106437/95859522-690a3000-0d67-11eb-94bb-1cccaac1d816.png)

![Screen Shot 2020-10-13 at 1 43 23 PM](https://user-images.githubusercontent.com/3106437/95856423-a9b37a80-0d62-11eb-8ade-54e8ed5b895c.png)


Co-authored-by: Andrii Vorobiov <and.vorobiov@gmail.com>

Release note: None

55855: build: remove unused ROCKSDB variables r=jordanlewis a=jordanlewis

Release note: None

Co-authored-by: Jordan Lewis <jordanthelewis@gmail.com>

Previously, events were not created when schema objects were
created, dropped or modified.

Fixes cockroachdb#55744

Release note (admin ui change): changing schema object
now causes an event to be logged and displayed in the admin ui

This change cleans up the treeprinter implementation and interface and
adds a "with bullets" style that looks like this:
```
  • foo
  │
  ├── • bar1
  │   │ bar2
  │   │
  │   └── • baz
  │
  └── • qux
```
This is a preliminary design from Anne for the new EXPLAIN output.

Release note: None

Changing the explain output test from an Example to a datadriven test
(which is much easier to update).

Release note: None

This change switches the plan string in EXPLAIN ANALYZE (DEBUG)
bundles to use the new style with bullets. See the updated test for
some examples.

Release note: None

55785: sql: create event log entry for schema events r=jordanlewis,arulajmani a=neeral

Previously, events were not created when schema objects were
created, dropped or modified.

Fixes cockroachdb#55744 

Release note (admin ui change): changing schema object
now causes an event to be logged and displayed in the admin ui

Co-authored-by: neeral <neeral.dodhia@gmail.com>

libcryptopp was only used for RocksDB encryption-at-rest. Now
that RocksDB is deleted (cockroachdb#55509), we can remove libcryptopp too.

Release note: None.

This commit copies the full crdb_internal test to crdb_internal_tenant, and
rewrites it to succeed with this configuration in order to increase
crdb_internal tenant test coverage.

Release note: None (testing change)

Refs: cockroachdb#50984, cockroachdb#54599

Reason: flaky test

Generated by bin/skip-test.

Release justification: non-production code changes

Release note: None

These are now converted into paired joins consisting of
a left outer inverted join followed by an left anti
lookup join. This is more efficient than the previous
approach that involved two left outer joins and removal
of the matching rows using a select.

Release note: None

This commit divides up all the session parameters into two categories:
local only (those that don't influence the execution if not propagated
to the remote nodes) and non-local (those that must be propagated). The
former are extracted into a separate struct, and the latter are now
mostly stored directly in a protobuf with a few exceptions having custom
serialization/deserialization logic. This allows us to clean up the eval
context proto and centralize the custom logic and the definition.

This commit also fixed an issue of not propagating the default int size
which - in theory - could influence the execution on the remote nodes
(e.g. when performing a cast by an internal executor).

Release note: None

55569: sessiondata: extract all non-local parameters into a protobuf r=yuzefovich a=yuzefovich

This commit divides up all the session parameters into two categories:
local only (those that don't influence the execution if not propagated
to the remote nodes) and non-local (those that must be propagated). The
former are extracted into a separate struct, and the latter are now
mostly stored directly in a protobuf with a few exceptions having custom
serialization/deserialization logic. This allows us to clean up the eval
context proto and centralize the custom logic and the definition.

This commit also fixed an issue of not propagating the default int size
which - in theory - could influence the execution on the remote nodes
(e.g. when performing a cast by an internal executor).

Fixes: cockroachdb#51075.

Release note: None

Co-authored-by: Yahor Yuzefovich <yahor@cockroachlabs.com>

One test cannot be run under the metamorphic build configuration, since
its output depends on the batch size: the test counts the number of
spans scanned, which will go up with a small batch size due to the table
reader resuming the scan until its finished.

Release note: None

55858: *: Remove cryptopp from c-deps, stop building it r=petermattis a=itsbilal

libcryptopp was only used for RocksDB encryption-at-rest. Now
that RocksDB is deleted (cockroachdb#55509), we can remove libcryptopp too.

Release note: None.

Co-authored-by: Bilal Akhtar <bilal@cockroachlabs.com>

55637: sql: use new explain format in explain bundles r=RaduBerinde a=RaduBerinde

#### util: treeprinter improvements and new style

This change cleans up the treeprinter implementation and interface and
adds a "with bullets" style that looks like this:
```
  • foo
  │
  ├── • bar1
  │   │ bar2
  │   │
  │   └── • baz
  │
  └── • qux
```
This is a preliminary design from Anne for the new EXPLAIN output.

Release note: None

#### opt: rework explain output test

Changing the explain output test from an Example to a datadriven test
(which is much easier to update).

Release note: None

#### sql: use new explain format in explain bundles

This change switches the plan string in EXPLAIN ANALYZE (DEBUG)
bundles to use the new style with bullets. See the updated test for
some examples.

Release note: None

55825: roachtest: rename AppDev team to SQL Experience r=solongordon a=rafiss

Release note: None

Co-authored-by: Radu Berinde <radu@cockroachlabs.com>
Co-authored-by: Rafi Shamim <rafi@cockroachlabs.com>

55792: opt: rework tests involving scalar expressions r=RaduBerinde a=RaduBerinde

#### opt: simplify execbuilder.BuildScalar

In all uses of BuildScalar, we map variables directly to columns in
the metadata. This change simplifies the interface and makes an
indexed var container internally.

Release note: None

#### opt: rework scalar optbuilder tests

This change reworks the infrastructure for tests involving scalars.
Instead of passing a list of types and using thing like `@1` to refer
to variables, we define variables with names. This makes the tests a
lot cleaner and doesn't rely on an obscure syntax. It will also allow
extending the variable definitions (e.g. computed column expressions).

The new infrastructure will be used for other scalar tests (most
importantly, index constraints and partial predicate implication).

Release note: None

#### opt: update memo/expr_test to use ScalarVars

Release note: None

#### opt: update composite_sensitive test to use ScalarVars

Release note: None

#### opt: update implicator tests to use ScalarVars

Release note: None

#### opt: update idxconstraint tests to use ScalarVars

Release note: None

Co-authored-by: Radu Berinde <radu@cockroachlabs.com>