Merge #54896 #55687 #55827

54896: cli: add import pgdump/mysqldump CLI command r=miretskiy,knz a=adityamaru

This change introduces a new CLI command to import
locally saved PGDUMP or MYSQLDUMP files into a running
cockroach cluster.

The underlying logic relies on user scoped userfile
storage to upload the local dump file to, and subsequently
import data from. Most of the heavy lifting was already
completed during the development of the userfile storage.

We can only support bundle formats that have the table
schemas to be imported, baked into the dump files. The newly
added CLI commands support the same options as IMPORT PGDUMP
and IMPORT MYSQLDUMP from the SQL shell do. These options
are to be passed in as CLI flags.

Following are the CLI examples:
`./cockroach import db <format> <source>`
`./cockroach import table <tablename> <format> <source>`

Release note (cli change): Adds an import CLI command which
allows users to upload and import local dump files into a
running cockroach cluster. We currently support PGDUMP and
MYSQLDUMP formats.

55687: *: bazel-ify cockroachdb r=irfansharif a=irfansharif

This commit introduces a new build system to cockroachdb: Bazel.
The hope here is to eventually replace our current Makefile. It sets the
foundation for introducing Bazel in our CI systems for parallelized
builds/test runs, artifact caching, remote builds, and many other such
profits (literally, it could reduce infra spend).

We originally prototyped a pseudo bazel-ified cockroachdb in #52824,
which resulted in a week long effort (with otan@, jamesl@, and myself) to
wrap it up in earnest in #55258. This is the polished version of the
resulting work, the write up for which can be found at
http://go.crdb.dev/bazel-hackweek.

---

Most of the BUILD files here were auto-generated through Gazelle (see
doc). Some of these auto-generated files required further manual
tweaking to get just right (see doc). These were mostly around the packages that
link into c-dependencies, but also the few packages that rely on
auto-generated code (again, see doc). These specific files are the ones
intended for review (they're also the ones responsible for
auto-generation), among a few others. I've listed them out here:

```
	new file:   .bazelrc
	modified:   .gitattributes
	modified:   .gitignore
	new file:   BUILD.bazel
	new file:   DEPS.bzl
	modified:   Makefile
	new file:   WORKSPACE
	new file:   pkg/ccl/gssapiccl/BUILD.bazel
	new file:   pkg/ccl/storageccl/engineccl/BUILD.bazel
	new file:   pkg/cli/BUILD.bazel
	new file:   pkg/cmd/cockroach-oss/BUILD.bazel
	new file:   pkg/cmd/cockroach-short/BUILD.bazel
	new file:   pkg/cmd/cockroach/BUILD.bazel
	new file:   pkg/col/coldata/BUILD.bazel
	new file:   pkg/geo/geoproj/BUILD.bazel
	modified:   pkg/geo/geoproj/geoproj.go
	modified:   pkg/geo/geoproj/proj.cc
	new file:   pkg/geo/geos/BUILD.bazel
	new file:   pkg/sql/colconv/BUILD.bazel
	new file:   pkg/sql/colexec/BUILD.bazel
	new file:   pkg/sql/colexec/COLEXEC.bzl
	modified:   pkg/sql/colexec/and_or_projection_tmpl.go
	modified:   pkg/sql/colexec/any_not_null_agg_tmpl.go
	modified:   pkg/sql/colexec/avg_agg_tmpl.go
	modified:   pkg/sql/colexec/bool_and_or_agg_tmpl.go
	modified:   pkg/sql/colexec/cast_tmpl.go
	new file:   pkg/sql/colexec/colbuilder/BUILD.bazel
	new file:   pkg/sql/colexec/colexec.go
	modified:   pkg/sql/colexec/concat_agg_tmpl.go
	modified:   pkg/sql/colexec/const_tmpl.go
	modified:   pkg/sql/colexec/count_agg_tmpl.go
	modified:   pkg/sql/colexec/default_agg_tmpl.go
	modified:   pkg/sql/colexec/default_cmp_expr_tmpl.go
	modified:   pkg/sql/colexec/default_cmp_proj_ops_tmpl.go
	modified:   pkg/sql/colexec/default_cmp_sel_ops_tmpl.go
	modified:   pkg/sql/colexec/distinct_tmpl.go
	modified:   pkg/sql/colexec/hash_aggregator_tmpl.go
	modified:   pkg/sql/colexec/hash_utils_tmpl.go
	modified:   pkg/sql/colexec/hashjoiner_tmpl.go
	modified:   pkg/sql/colexec/hashtable_tmpl.go
	modified:   pkg/sql/colexec/is_null_ops_tmpl.go
	modified:   pkg/sql/colexec/mergejoinbase_tmpl.go
	modified:   pkg/sql/colexec/mergejoiner_tmpl.go
	modified:   pkg/sql/colexec/min_max_agg_tmpl.go
	modified:   pkg/sql/colexec/ordered_synchronizer_tmpl.go
	modified:   pkg/sql/colexec/proj_const_ops_tmpl.go
	modified:   pkg/sql/colexec/proj_non_const_ops_tmpl.go
	modified:   pkg/sql/colexec/quicksort_tmpl.go
	modified:   pkg/sql/colexec/rank_tmpl.go
	modified:   pkg/sql/colexec/relative_rank_tmpl.go
	modified:   pkg/sql/colexec/row_number_tmpl.go
	modified:   pkg/sql/colexec/rowstovec_tmpl.go
	modified:   pkg/sql/colexec/select_in_tmpl.go
	modified:   pkg/sql/colexec/selection_ops_tmpl.go
	modified:   pkg/sql/colexec/sort_tmpl.go
	modified:   pkg/sql/colexec/substring_tmpl.go
	modified:   pkg/sql/colexec/sum_agg_tmpl.go
	modified:   pkg/sql/colexec/utils_tmpl.go
	modified:   pkg/sql/colexec/values_differ_tmpl.go
	modified:   pkg/sql/colexec/vec_comparators_tmpl.go
	modified:   pkg/sql/colexec/window_peer_grouper_tmpl.go
	new file:   pkg/sql/colflow/BUILD.bazel
	new file:   pkg/sql/opt/BUILD.bazel
	new file:   pkg/sql/opt/exec/BUILD.bazel
	new file:   pkg/sql/opt/exec/execbuilder/BUILD.bazel
	new file:   pkg/sql/opt/exec/explain/BUILD.bazel
	new file:   pkg/sql/opt/memo/BUILD.bazel
	new file:   pkg/sql/opt/norm/BUILD.bazel
	new file:   pkg/sql/opt/optgen/lang/BUILD.bazel
	new file:   pkg/sql/opt/xform/BUILD.bazel
	new file:   pkg/sql/parser/BUILD.bazel
	new file:   pkg/sql/parser/sql-gen.sh
	modified:   pkg/sql/parser/sql.y
	new file:   pkg/storage/BUILD.bazel
	modified:   vendor
```

The changes to all the _tmpl.go files were needed to include necessary
import analysis/compilation for bazel when generating the .eg.go files
(also summarized in doc up above), and should be uncontroversial.

I think it's easiest to pull the branch down locally and to click around
to see what works, and what doesn't. I would start at the top-level
BUILD.bazel and WORSPACE file. I've added comments to explain what most
of added machinery does.

Release note: None


55827: sql: stub ALTER TABLE ... SET LOCALITY/AFFINITY related commands r=ajstorm a=otan

Following the doc on proposed ideas and fleshed them all out. They
default to displaying REGIONAL AFFINITY for now.

Release note (sql change): Implemented ALTER TABLE ... SET
LOCALITY/REGIONAL AFFINITY commands that allow users to configure
multiregion properties of given tables. These may change later.

Co-authored-by: Aditya Maru <adityamaru@gmail.com>
Co-authored-by: irfan sharif <irfanmahmoudsharif@gmail.com>
Co-authored-by: Oliver Tan <otan@cockroachlabs.com>

.bazelrc

@@ -0,0 +1,9 @@
+# Bazel's usual sandboxing does not try to protect against accidental
+# dependencies on # files from the host filesystem. See [1]. This is a problem
+# for us, for when users have `proj` installed locally. They can run into
+# issues where bazel tries to use `/usr/local/include/proj_api.h` instead of
+# `bazel-build/.../c-deps/proj/lib`. The work around here, suggested by [1], is
+# to ensure the sandbox does not read /usr/local/include.
+#
+# [1]: https://github.com/bazelbuild/bazel/issues/8053.
+build --sandbox_block_path=/usr/local/include

.gitattributes

@@ -1,2 +1,3 @@
 *.pb.* -diff
 *.eg.go -diff
+DEPS.bzl -diff

BUILD.bazel

@@ -0,0 +1,214 @@
+load("@bazel_gazelle//:def.bzl", "gazelle")
+
+# The following directives inform gazelle how to auto-generate BUILD.bazel
+# files throughout the repo. By including them here, we can run gazelle using
+# `bazel run //:gazelle` instead of invoking gazelle directly.
+#
+# NB: Be sure to re-generate BUILD.bazel files if any of the directives below
+# are changed.
+
+# Define the gazelle prefix and what the autogenerated BUILD files should be
+# named as.
+#
+# gazelle:prefix github.com/cockroachdb/cockroach
+# gazelle:build_file_name BUILD.bazel
+
+# We disable protobuf generation for our dependencies
+#
+# gazelle:proto disable_global
+
+# Gazelle is unable to resolve this specific package.
+#
+# TODO(irfansharif): I'm not sure why this is. Is it because it's a proto only
+# package?
+#
+# gazelle:resolve go github.com/grpc-ecosystem/grpc-gateway/internal //vendor/github.com/grpc-ecosystem/grpc-gateway/internal
+
+# We exclude a few things from gazelle consideration:
+# - two of our C dependencies that are already bazel-ified, lest we overwrite
+#   those build files.
+# - all checked-in autogenerated files (they'll get autogenerated on the fly
+#   through bazel).
+# - a testdata "repo" that looks like a go package, but isn't
+#   (testdata/src/example.com)
+#
+# gazelle:exclude c-deps/googletest
+# gazelle:exclude c-deps/protobuf
+# gazelle:exclude **/zcgo_flags.go
+# gazelle:exclude **/zcgo_flags_*.go
+# gazelle:exclude **/*.og.go
+# gazelle:exclude **/*.eg.go
+# gazelle:exclude pkg/sql/parser/sql.go
+# gazelle:exclude pkg/sql/opt/rule_name_string.go
+# gazelle:exclude pkg/sql/opt/rule_name_string.go
+# gazelle:exclude pkg/cmd/prereqs/testdata
+
+# TODO(irfansharif): Today we let bazel take over the vendor directory and
+# define each vendored dependency as a bazel target. We should be able to have
+# bazel ignore all of vendor/ and create those dependencies on the fly.
+# Deleting the vendor directory and running `bazel run //:gazelle` shows what
+# that would look like (though that doesn't quite work yet).
+
+# TODO(irfansharif): Document a few usage patterns for bazel and how to
+# understand all the autogen stuff. Probably as a tech note. Here are a few
+# short hands I've used so far:
+#
+#   bazel test //pkg/kv/kvserver/concurrency/...
+#   bazel build //pkg/sql/opt
+#   bazel build //pkg/sql/opt:all
+#   bazel build //pkg/sql/colexec:gen-exec
+#   bazel build //pkg/cmd/cockroach-short
+#   bazel build //:libjemalloc
+#   bazel query //pkg/sql/colexec:all
+#   bazel run //:gazelle
+#   bazel run //pkg/cmd/cockroach-short -- demo
+#   bazel run //pkg/sql/opt/optgen/cmd/langgen -- -h
+#   bazel run //:gazelle -- update-repos -from_file=go.mod -to_macro=DEPS.bzl%go_deps
+#
+# The //<stuff> names can also be fully qualified using @cockroach, and that
+# appears in certain parts of the codebase/elsewhere. Specifically it'll look
+# like:
+#
+#   bazel build @cockroach//pkg/cmd/cockroach-short
+#
+# TODO(irfansharif): We should probably define shorthands for the more common
+# ones from up above.
+
+# TODO(irfansharif): The way we currently generate code through bazel, that
+# code is only available within the bazel sandbox. Bazel ignores all
+# the pre-generated code that is already checked into the codebase (through
+# `make generate`/etc.)through the exclude directives above. It's generating
+# everything on the fly.
+#
+# As we move towards bazel, we'll want to introduce a mechanism that implants
+# the generated code within the sandbox placing them "back into" the
+# appropriate packages. This is to ensure we don't break existing IDEs and code
+# editors which rely on files existing in the same package itself (as opposed
+# to the sandbox). This way we could continue checking in auto-generated code.
+# We should provide a bazel alternative for `make generate` that does this very
+# same thing. See [1], this is a long standing issue for folks using using Go,
+# bazel, and autogenerated code.
+#
+# [1]: https://github.com/bazelbuild/rules_go/issues/512.
+
+# TODO(irfansharif): We'll need to pin toolchains somewhere to make sure
+# everything below works as expected.
+
+gazelle(
+    name = "gazelle",
+    prefix = "github.com/cockroachdb/cockroach",
+)
+
+# Load the components that lets us use cmake/make in third party deps.
+load("@rules_foreign_cc//tools/build_defs:cmake.bzl", "cmake_external")
+load("@rules_foreign_cc//tools/build_defs:configure.bzl", "configure_make")
+
+# TODO(irfansharif): All the cmake cache entries below were cargo-culted from
+# the Makefile. We may need to programatically set them depending on the
+# machine bazel is being run on. For now the only tested variant are Macbooks.
+
+# Define the build target for libjemalloc.
+configure_make(
+    name = "libjemalloc",
+    autoconf = True,
+    configure_env_vars = {
+        "AR": "",
+    },
+    configure_in_place = True,
+    configure_options = [
+        "--enable-prof",
+    ],
+    lib_source = "@jemalloc//:all",
+    make_commands = [
+        "make build_lib_static",
+        "mkdir -p libjemalloc/lib",
+        "cp lib/libjemalloc.a libjemalloc/lib",
+        "cp -r include libjemalloc",
+    ],
+    static_libraries = ["libjemalloc.a"],
+    visibility = ["//visibility:public"],
+)
+
+# Define the build targets for libprotobuf and protoc.
+cmake_external(
+    name = "libprotobuf",
+    binaries = ["protoc"],
+    cache_entries = {
+        "CMAKE_BUILD_TYPE": "Release",
+        "CMAKE_TARGET_MESSAGES": "OFF",
+        "protobuf_BUILD_TESTS": "OFF",
+    },
+    lib_source = "@protobuf//:all",
+    static_libraries = ["libprotobuf.a"],
+    visibility = ["//visibility:public"],
+    working_directory = "cmake",
+)
+
+# Define the build target for libproj.
+cmake_external(
+    name = "libproj",
+    cache_entries = {
+        "CMAKE_BUILD_TYPE": "Release",
+        "BUILD_LIBPROJ_SHARED": "OFF",
+    },
+    lib_source = "@proj//:all",
+    static_libraries = ["libproj.a"],
+    visibility = ["//visibility:public"],
+)
+
+# TODO(irfansharif): libgeos has not been worked out yet. We'll need to
+# similarly ensure the lib/libgeos.so and lib/libegeos_c.so are in the right
+# place.
+
+# Define the build target for libroach.
+#
+# Bazel also expects the library archive and the include headers to be placed
+# in a certain path, so we fix it all up accordingly within make_commands.
+cmake_external(
+    name = "libroach",
+    cache_entries = {
+        "CMAKE_TARGET_MESSAGES": "OFF",
+        "CMAKE_BUILD_TYPE": "Release",
+    },
+    lib_source = "@libroach//:all",
+    make_commands = [
+        "make roach",
+        "mkdir -p libroach/lib",
+        "cp libroach.a libroach/lib/libroach.a",
+        "cp -r $EXT_BUILD_ROOT/external/libroach/include libroach",
+    ],
+    static_libraries = ["libroach.a"],
+    tools_deps = [
+        "@libroach//:all",
+    ],
+    visibility = ["//visibility:public"],
+    deps = [],
+)
+
+# Define the build target for kerberos.
+#
+# TODO(irfansharif): Kerboros is not used for anything other than Linux.
+# The following has not been tested as yet.
+configure_make(
+    name = "libkrb5",
+    autoreconf = True,
+    configure_command = "src/configure",
+    configure_env_vars = {
+        "CPFLAGS": "",
+        "CXXFLAGS": "",
+    },
+    configure_options = [
+        "--enable-static",
+        "--disable-shared",
+    ] + select({
+        "//conditions:default": ["AR=/usr/bin/ar"],
+    }),
+    lib_source = "@krb5//:all",
+    make_commands = [
+        "make",
+        "mkdir -p libkrb5/lib",
+        "cp libkrb5/libgssapi_krb5.a libkrb5/lib",
+    ],
+    static_libraries = ["libgssapi_krb5.a"],
+    visibility = ["//visibility:public"],
+)