Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

sql: Added new tables and columns in pg_catalog #59498

Conversation

mnovelodou
Copy link
Contributor

@mnovelodou mnovelodou commented Jan 27, 2021

Previously, added tables and columns existed in postgres but not cockroach db
This was inadequate because this may cause errors at tools that expects these
columns to exists
To address this, this patch contains the result of diff tool to add what is
missing

Release note (sql change): Added missing tables and columns at pg_catalog
New columns in table pg_attribute:
atthasmissing
New columns in table pg_class:
relrowsecurity
relforcerowsecurity
relispartition
relispopulated
relreplident
relrewrite
New columns in table pg_collation:
collprovider
collversion
collisdeterministic
New columns in table pg_index:
indnkeyatts
New columns in table pg_proc:
prokind
prosupport
New columns in table pg_aggregate:
aggmfinalmodify
aggfinalmodify
New Tables:
pg_replication_origin_roname_index
pg_config
pg_largeobject_loid_pn_index
pg_ts_config
pg_operator_oid_index
pg_conversion_oid_index
pg_depend_depender_index
pg_stat_user_functions
pg_index_indrelid_index
pg_group
pg_db_role_setting_databaseid_rol_index
pg_statio_user_tables
pg_partitioned_table_partrelid_index
pg_publication_oid_index
pg_publication
pg_publication_tables
pg_stat_xact_user_tables
pg_ts_config_map_index
pg_replication_origin_roiident_index
pg_stat_progress_vacuum
pg_file_settings
pg_statio_all_indexes
pg_shseclabel_object_index
pg_enum_oid_index
pg_statistic_relid_att_inh_index
pg_largeobject_metadata_oid_index
pg_stat_archiver
pg_statio_sys_sequences
pg_collation_oid_index
pg_subscription
pg_namespace_oid_index
pg_amop
pg_stat_database_conflicts
pg_tablespace_oid_index
pg_class_oid_index
pg_range_rngtypid_index
pg_description_o_c_o_index
pg_opclass_oid_index
pg_sequence_seqrelid_index
pg_trigger_oid_index
pg_timezone_abbrevs
pg_ts_parser_oid_index
pg_transform
pg_extension_oid_index
pg_statio_user_sequences
pg_shdepend_reference_index
pg_foreign_data_wrapper_oid_index
pg_ts_config_oid_index
pg_ts_dict_oid_index
pg_init_privs_o_c_o_index
pg_user_mappings
pg_default_acl_role_nsp_obj_index
pg_stat_slru
pg_constraint_contypid_index
pg_stat_progress_create_index
pg_transform_type_lang_index
pg_authid_oid_index
pg_shmem_allocations
pg_statio_sys_tables
pg_statio_all_sequences
pg_policy_oid_index
pg_shdepend_depender_index
pg_attribute_relid_attnum_index
pg_event_trigger_oid_index
pg_amproc
pg_cast_oid_index
pg_constraint_conparentid_index
pg_statio_sys_indexes
pg_conversion_default_index
pg_statistic_ext
pg_shadow
pg_trigger_tgconstraint_index
pg_ts_template
pg_cast_source_target_index
pg_type_oid_index
pg_amproc_fam_proc_index
pg_aggregate_fnoid_index
pg_stat_progress_basebackup
pg_default_acl_oid_index
pg_foreign_server_oid_index
pg_statistic_ext_data_stxoid_index
pg_transform_oid_index
pg_language_oid_index
pg_db_role_setting
pg_amop_opr_fam_index
pg_user_mapping_oid_index
pg_hba_file_rules
pg_am_oid_index
pg_statio_all_tables
pg_statio_user_indexes
pg_stat_progress_analyze
pg_replication_origin
pg_depend_reference_index
pg_stat_bgwriter
pg_attrdef_adrelid_adnum_index
pg_stat_sys_indexes
pg_statistic_ext_oid_index
pg_foreign_table_relid_index
pg_user_mapping_user_server_index
pg_seclabel_object_index
pg_subscription_rel_srrelid_srsubid_index
pg_publication_rel_prrelid_prpubid_index
pg_inherits_parent_index
pg_ts_dict
pg_stat_user_tables
pg_stat_progress_cluster
pg_stat_xact_all_tables
pg_stat_database
pg_shdescription_o_c_index
pg_publication_rel_oid_index
pg_largeobject
pg_publication_rel
pg_rewrite_oid_index
pg_stat_all_indexes
pg_amop_fam_strat_index
pg_proc_oid_index
pg_database_oid_index
pg_sequences
pg_subscription_oid_index
pg_enum_typid_sortorder_index
pg_ts_parser
pg_ts_config_map
pg_attrdef_oid_index
pg_timezone_names
pg_ts_template_oid_index
pg_statistic_ext_relid_index
pg_index_indexrelid_index
pg_amop_oid_index
pg_amproc_oid_index
pg_rules
pg_opfamily
pg_stat_xact_sys_tables
pg_policies
pg_constraint_oid_index
pg_stat_sys_tables
pg_stat_xact_user_functions
pg_available_extension_versions
pg_stat_all_tables
pg_auth_members_role_member_index
pg_auth_members_member_role_index
pg_inherits_relid_seqno_index
pg_opfamily_oid_index
pg_class_tblspc_relfilenode_index
pg_cursors
pg_stat_gssapi
pg_stat_ssl
pg_stat_user_indexes

Fixes #58001

@blathers-crl
Copy link

blathers-crl bot commented Jan 27, 2021

Thank you for contributing to CockroachDB. Please ensure you have followed the guidelines for creating a PR.

My owl senses detect your PR is good for review. Please keep an eye out for any test failures in CI.

🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is otan.

@blathers-crl blathers-crl bot added the O-community Originated from the community label Jan 27, 2021
@cockroach-teamcity
Copy link
Member

This change is Reviewable

@mnovelodou mnovelodou force-pushed the pg-catalog-missing-tables-and-columns branch from 60cbc32 to 43f6762 Compare February 5, 2021 22:47
@blathers-crl
Copy link

blathers-crl bot commented Feb 5, 2021

Thank you for updating your pull request.

My owl senses detect your PR is good for review. Please keep an eye out for any test failures in CI.

🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is otan.

@mnovelodou mnovelodou force-pushed the pg-catalog-missing-tables-and-columns branch 3 times, most recently from 298349f to 3e6991f Compare February 8, 2021 22:16
@mnovelodou mnovelodou requested a review from a team as a code owner February 8, 2021 22:16
Previously, added tables and columns existed in postgres but not cockroach db
This was inadequate because this may cause errors at tools that expects these
columns to exists
To address this, this patch contains the result of diff tool to add what is
missing

Release note (sql change): Added missing tables and columns at pg_catalog
New columns in table pg_attribute:
        atthasmissing
New columns in table pg_class:
        relrowsecurity
        relforcerowsecurity
        relispartition
        relispopulated
        relreplident
        relrewrite
New columns in table pg_collation:
        collprovider
        collversion
        collisdeterministic
New columns in table pg_index:
        indnkeyatts
New columns in table pg_proc:
        prokind
        prosupport
New columns in table pg_aggregate:
        aggmfinalmodify
        aggfinalmodify
New Tables:
        pg_replication_origin_roname_index
        pg_config
        pg_largeobject_loid_pn_index
        pg_ts_config
        pg_operator_oid_index
        pg_conversion_oid_index
        pg_depend_depender_index
        pg_stat_user_functions
        pg_index_indrelid_index
        pg_group
        pg_db_role_setting_databaseid_rol_index
        pg_statio_user_tables
        pg_partitioned_table_partrelid_index
        pg_publication_oid_index
        pg_publication
        pg_publication_tables
        pg_stat_xact_user_tables
        pg_ts_config_map_index
        pg_replication_origin_roiident_index
        pg_stat_progress_vacuum
        pg_file_settings
        pg_statio_all_indexes
        pg_shseclabel_object_index
        pg_enum_oid_index
        pg_statistic_relid_att_inh_index
        pg_largeobject_metadata_oid_index
        pg_stat_archiver
        pg_statio_sys_sequences
        pg_collation_oid_index
        pg_subscription
        pg_namespace_oid_index
        pg_amop
        pg_stat_database_conflicts
        pg_tablespace_oid_index
        pg_class_oid_index
        pg_range_rngtypid_index
        pg_description_o_c_o_index
        pg_opclass_oid_index
        pg_sequence_seqrelid_index
        pg_trigger_oid_index
        pg_timezone_abbrevs
        pg_ts_parser_oid_index
        pg_transform
        pg_extension_oid_index
        pg_statio_user_sequences
        pg_shdepend_reference_index
        pg_foreign_data_wrapper_oid_index
        pg_ts_config_oid_index
        pg_ts_dict_oid_index
        pg_init_privs_o_c_o_index
        pg_user_mappings
        pg_default_acl_role_nsp_obj_index
        pg_stat_slru
        pg_constraint_contypid_index
        pg_stat_progress_create_index
        pg_transform_type_lang_index
        pg_authid_oid_index
        pg_shmem_allocations
        pg_statio_sys_tables
        pg_statio_all_sequences
        pg_policy_oid_index
        pg_shdepend_depender_index
        pg_attribute_relid_attnum_index
        pg_event_trigger_oid_index
        pg_amproc
        pg_cast_oid_index
        pg_constraint_conparentid_index
        pg_statio_sys_indexes
        pg_conversion_default_index
        pg_statistic_ext
        pg_shadow
        pg_trigger_tgconstraint_index
        pg_ts_template
        pg_cast_source_target_index
        pg_type_oid_index
        pg_amproc_fam_proc_index
        pg_aggregate_fnoid_index
        pg_stat_progress_basebackup
        pg_default_acl_oid_index
        pg_foreign_server_oid_index
        pg_statistic_ext_data_stxoid_index
        pg_transform_oid_index
        pg_language_oid_index
        pg_db_role_setting
        pg_amop_opr_fam_index
        pg_user_mapping_oid_index
        pg_hba_file_rules
        pg_am_oid_index
        pg_statio_all_tables
        pg_statio_user_indexes
        pg_stat_progress_analyze
        pg_replication_origin
        pg_depend_reference_index
        pg_stat_bgwriter
        pg_attrdef_adrelid_adnum_index
        pg_stat_sys_indexes
        pg_statistic_ext_oid_index
        pg_foreign_table_relid_index
        pg_user_mapping_user_server_index
        pg_seclabel_object_index
        pg_subscription_rel_srrelid_srsubid_index
        pg_publication_rel_prrelid_prpubid_index
        pg_inherits_parent_index
        pg_ts_dict
        pg_stat_user_tables
        pg_stat_progress_cluster
        pg_stat_xact_all_tables
        pg_stat_database
        pg_shdescription_o_c_index
        pg_publication_rel_oid_index
        pg_largeobject
        pg_publication_rel
        pg_rewrite_oid_index
        pg_stat_all_indexes
        pg_amop_fam_strat_index
        pg_proc_oid_index
        pg_database_oid_index
        pg_sequences
        pg_subscription_oid_index
        pg_enum_typid_sortorder_index
        pg_ts_parser
        pg_ts_config_map
        pg_attrdef_oid_index
        pg_timezone_names
        pg_ts_template_oid_index
        pg_statistic_ext_relid_index
        pg_index_indexrelid_index
        pg_amop_oid_index
        pg_amproc_oid_index
        pg_rules
        pg_opfamily
        pg_stat_xact_sys_tables
        pg_policies
        pg_constraint_oid_index
        pg_stat_sys_tables
        pg_stat_xact_user_functions
        pg_available_extension_versions
        pg_stat_all_tables
        pg_auth_members_role_member_index
        pg_auth_members_member_role_index
        pg_inherits_relid_seqno_index
        pg_opfamily_oid_index
        pg_class_tblspc_relfilenode_index
        pg_cursors
        pg_stat_gssapi
        pg_stat_ssl
        pg_stat_user_indexes

Fixes cockroachdb#58001
@mnovelodou mnovelodou force-pushed the pg-catalog-missing-tables-and-columns branch from 3e6991f to 4be0947 Compare February 8, 2021 23:23
Copy link
Contributor

@RichardJCai RichardJCai left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The output looking pretty good! Got some minor comments

Reviewable status: :shipit: complete! 0 of 0 LGTMs obtained (waiting on @mnovelodou and @rafiss)


pkg/sql/pg_catalog.go, line 674 at r1 (raw file):

				tree.DNull,                        // attoptions
				tree.DNull,                        // attfdwoptions
				// These columns were generated automatically by diff tool

I think we should figure out a better / more specific name for diff tool, to people without context diff tool alone may be ambiguous.


pkg/sql/pg_catalog.go, line 2795 at r1 (raw file):

var pgCatalogPoliciesTable = virtualSchemaTable{
	comment: "pg_policies is not implemented yet",

Maybe we should add a comment here as well saying that it was generated by the diff tool (or whatever we choose to name it)

craig bot pushed a commit that referenced this pull request Feb 19, 2021
60758: sql: Added new tables and columns in pg_catalog r=rafiss a=RichardJCai

Taking over #59498 since @mnovelodou is out and we want to get this in before stability.

Previously, added tables and columns existed in postgres but not cockroach db
This was inadequate because this may cause errors at tools that expects these
columns to exists
To address this, this patch contains the result of diff tool to add what is
missing

Release note (sql change): Added missing tables and columns at pg_catalog
New columns in table pg_attribute:
        atthasmissing
New columns in table pg_class:
        relrowsecurity
        relforcerowsecurity
        relispartition
        relispopulated
        relreplident
        relrewrite
New columns in table pg_collation:
        collprovider
        collversion
        collisdeterministic
New columns in table pg_index:
        indnkeyatts
New columns in table pg_proc:
        prokind
        prosupport
New columns in table pg_aggregate:
        aggmfinalmodify
        aggfinalmodify
New Tables:
        pg_replication_origin_roname_index
        pg_config
        pg_largeobject_loid_pn_index
        pg_ts_config
        pg_operator_oid_index
        pg_conversion_oid_index
        pg_depend_depender_index
        pg_stat_user_functions
        pg_index_indrelid_index
        pg_group
        pg_db_role_setting_databaseid_rol_index
        pg_statio_user_tables
        pg_partitioned_table_partrelid_index
        pg_publication_oid_index
        pg_publication
        pg_publication_tables
        pg_stat_xact_user_tables
        pg_ts_config_map_index
        pg_replication_origin_roiident_index
        pg_stat_progress_vacuum
        pg_file_settings
        pg_statio_all_indexes
        pg_shseclabel_object_index
        pg_enum_oid_index
        pg_statistic_relid_att_inh_index
        pg_largeobject_metadata_oid_index
        pg_stat_archiver
        pg_statio_sys_sequences
        pg_collation_oid_index
        pg_subscription
        pg_namespace_oid_index
        pg_amop
        pg_stat_database_conflicts
        pg_tablespace_oid_index
        pg_class_oid_index
        pg_range_rngtypid_index
        pg_description_o_c_o_index
        pg_opclass_oid_index
        pg_sequence_seqrelid_index
        pg_trigger_oid_index
        pg_timezone_abbrevs
        pg_ts_parser_oid_index
        pg_transform
        pg_extension_oid_index
        pg_statio_user_sequences
        pg_shdepend_reference_index
        pg_foreign_data_wrapper_oid_index
        pg_ts_config_oid_index
        pg_ts_dict_oid_index
        pg_init_privs_o_c_o_index
        pg_user_mappings
        pg_default_acl_role_nsp_obj_index
        pg_stat_slru
        pg_constraint_contypid_index
        pg_stat_progress_create_index
        pg_transform_type_lang_index
        pg_authid_oid_index
        pg_shmem_allocations
        pg_statio_sys_tables
        pg_statio_all_sequences
        pg_policy_oid_index
        pg_shdepend_depender_index
        pg_attribute_relid_attnum_index
        pg_event_trigger_oid_index
        pg_amproc
        pg_cast_oid_index
        pg_constraint_conparentid_index
        pg_statio_sys_indexes
        pg_conversion_default_index
        pg_statistic_ext
        pg_shadow
        pg_trigger_tgconstraint_index
        pg_ts_template
        pg_cast_source_target_index
        pg_type_oid_index
        pg_amproc_fam_proc_index
        pg_aggregate_fnoid_index
        pg_stat_progress_basebackup
        pg_default_acl_oid_index
        pg_foreign_server_oid_index
        pg_statistic_ext_data_stxoid_index
        pg_transform_oid_index
        pg_language_oid_index
        pg_db_role_setting
        pg_amop_opr_fam_index
        pg_user_mapping_oid_index
        pg_hba_file_rules
        pg_am_oid_index
        pg_statio_all_tables
        pg_statio_user_indexes
        pg_stat_progress_analyze
        pg_replication_origin
        pg_depend_reference_index
        pg_stat_bgwriter
        pg_attrdef_adrelid_adnum_index
        pg_stat_sys_indexes
        pg_statistic_ext_oid_index
        pg_foreign_table_relid_index
        pg_user_mapping_user_server_index
        pg_seclabel_object_index
        pg_subscription_rel_srrelid_srsubid_index
        pg_publication_rel_prrelid_prpubid_index
        pg_inherits_parent_index
        pg_ts_dict
        pg_stat_user_tables
        pg_stat_progress_cluster
        pg_stat_xact_all_tables
        pg_stat_database
        pg_shdescription_o_c_index
        pg_publication_rel_oid_index
        pg_largeobject
        pg_publication_rel
        pg_rewrite_oid_index
        pg_stat_all_indexes
        pg_amop_fam_strat_index
        pg_proc_oid_index
        pg_database_oid_index
        pg_sequences
        pg_subscription_oid_index
        pg_enum_typid_sortorder_index
        pg_ts_parser
        pg_ts_config_map
        pg_attrdef_oid_index
        pg_timezone_names
        pg_ts_template_oid_index
        pg_statistic_ext_relid_index
        pg_index_indexrelid_index
        pg_amop_oid_index
        pg_amproc_oid_index
        pg_rules
        pg_opfamily
        pg_stat_xact_sys_tables
        pg_policies
        pg_constraint_oid_index
        pg_stat_sys_tables
        pg_stat_xact_user_functions
        pg_available_extension_versions
        pg_stat_all_tables
        pg_auth_members_role_member_index
        pg_auth_members_member_role_index
        pg_inherits_relid_seqno_index
        pg_opfamily_oid_index
        pg_class_tblspc_relfilenode_index
        pg_cursors
        pg_stat_gssapi
        pg_stat_ssl
        pg_stat_user_indexes

Fixes #58001

60765: kv: don't serve non-locking, read-write requests on followers r=nvanbenschoten a=nvanbenschoten

Discovered while investigating a test failure in #59566.

In 278a21b, we shifted from talking about read and write requests to
locking and non-locking requests when deciding whether a request could
be served on a follower. This prevented locking scans and gets from
being served on followers. However, it began letting lone HeartbeatTxn
and EndTxn requests past the old `!IsReadOnly()` check. Luckily, these
were still prevented from being served on followers because they are
only sent in read-write transactions, which were also prevented from
performing follower reads.

Yesterday, in 0ac8ab9, we lifted this second limitation, allowing
read-write transactions to perform follower reads for non-locking
batches. However, this no longer prevented HeartbeatTxn and EndTxn
requests from being routed and served on follower replicas. This
resulted in a pretty disastrous situation where in very rare cases, a
follower was proposing a write under a lease that it did not own.
Luckily, new assertions added in #59566 caught this.

This commit fixes this oversight be re-introducing "read-only" as a
condition for serving follower reads.

Release note: None

60781: cluster: use WaitConditionNextExit r=rickystewart a=tbg

There is a chance that this will address an issue that causes spurious
failures on CI, where it looks like we're janking the file system out
from under a running process. For context, see:

#58955

Release note: None


Co-authored-by: richardjcai <caioftherichard@gmail.com>
Co-authored-by: Nathan VanBenschoten <nvanbenschoten@gmail.com>
Co-authored-by: Tobias Grieger <tobias.b.grieger@gmail.com>
@RichardJCai
Copy link
Contributor

Closed by #60758

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
O-community Originated from the community
Projects
None yet
Development

Successfully merging this pull request may close these issues.

sql: create empty pg_catalog tables that are missing
3 participants