Multi tenant monitoring #6725

halilozanakgul · 2023-02-17T11:46:16Z

DESCRIPTION: Adds views that monitor statistics on tenant usages

This PR adds citus_stats_tenants view that monitors the tenants on the cluster.

citus_stats_tenants shows the node id, colocation id, tenant attribute, read count in this period and last period, and query count in this period and last period of the tenant.
Tenant attribute currently is the tenant's distribution column value, later when schema based sharding is introduced, this meaning might change.
A period is a time bucket the queries are counted by. Read and query counts for this period can increase until the current period ends. After that those counts are moved to last period's counts, which cannot change. The period length can be set using 'citus.stats_tenants_period'.

SELECT queries are counted as read queries, INSERT, UPDATE and DELETE queries are counted as write queries. So in the view read counts are SELECT counts and query counts are SELECT, INSERT, UPDATE and DELETE count.

The data is stored in shared memory, in a struct named MultiTenantMonitor.

citus_stats_tenants shows the data from local tenants.

citus_stats_tenants show up to citus.stats_tenant_limit number of tenants.
The tenants are scored based on the number of queries they run and the recency of those queries. Every query ran increases the score of tenant by ONE_QUERY_SCORE, and after every period ends the scores are halved. Halving is done lazily.
To retain information a longer the monitor keeps up to 3 times citus.stats_tenant_limit tenants. When the tenant count hits 3 * citus.stats_tenant_limit, last citus.stats_tenant_limit tenants are removed. To see all stored tenants you can use citus_stats_tenants(return_all_tenants := true)

Create collector view that gets data from all nodes. Citus stats tenants collector view #6761
Add monitoring log Add logs to citus_stats_tenants #6762
Create enable/disable GUC Citus stats tenants guc #6769
Parse the annotation string correctly Parse the annotation string correctly #6796
Add local queries and prepared statements Attribute local queries and cached plans on local execution #6797
Rename to citus_stat_statements Rename citus stats tenants #6821
Run pgbench
Fix role permissions Revoke statistics permissions from public and grant them to pg_monitor #6812

codecov · 2023-03-10T09:50:33Z

Codecov Report

Merging #6725 (5980d30) into main (d04d32b) will decrease coverage by 1.81%.
The diff coverage is 98.51%.

@@            Coverage Diff             @@
##             main    #6725      +/-   ##
==========================================
- Coverage   93.24%   91.43%   -1.81%     
==========================================
  Files         269      271       +2     
  Lines       57366    57632     +266     
==========================================
- Hits        53489    52698     -791     
- Misses       3877     4934    +1057

JelteF · 2023-03-14T08:08:12Z

src/test/regress/sql/failure_multi_dml.sql

@@ -21,7 +21,7 @@ SELECT citus.clear_network_traffic();
 ---- at each significant point. These transactions are 2pc

 -- fail at DELETE
-SELECT citus.mitmproxy('conn.onQuery(query="^DELETE").kill()');
+SELECT citus.mitmproxy('conn.onQuery(query="DELETE").kill()');


Why remove all these ^ characters?

Now the queries have the tenant attribute comment at the beginning. So they don't start with, for example, DELETE anymore.

JelteF · 2023-03-14T08:11:15Z

src/test/regress/expected/upgrade_list_citus_objects.out

@@ -316,7 +318,9 @@ ORDER BY 1;
 view citus_shards_on_worker
 view citus_stat_activity
 view citus_stat_statements
+ view citus_stats_tenants
+ view citus_stats_tenants_storage


I think this one and the related function should be removed

JelteF · 2023-03-14T08:13:04Z

src/test/regress/bin/normalize.sed

@@ -303,3 +303,5 @@ s/(NOTICE:  issuing SET LOCAL application_name TO 'citus_rebalancer gpid=)[0-9]+

 # shard_rebalancer output, flaky improvement number
 s/improvement of 0.1[0-9]* is lower/improvement of 0.1xxxxx is lower/g
+
+s/\/\* attributeTo.*\*\///g


Could use a comment what this replacement is for

I should've mentioned this.
Gökhan has another PR that fixes the string parsing. So this part will be changed a bit
#6753

marcocitus · 2023-03-14T09:45:57Z

src/include/distributed/utils/attribute.h

+	int selectsInLastPeriod;
+	int selectsInThisPeriod;
+
+	int insertCount;


I probably missed something. Why are we specifically tracking inserts and not other writes?

I think I should add them. Should it be INSERT/UPDATE/DELETE for writes and SELECT for reads

Updated my PR and added UPDATE and DELETE queries

marcocitus · 2023-03-14T09:47:01Z

src/backend/distributed/utils/attribute.c

+		}
+		TenantStats *tenantStats = &monitor->tenants[tenantIndex];
+
+		LWLockAcquire(&tenantStats->lock, LW_EXCLUSIVE);


could this cause contention?

(mainly wondering whether atomic integers would be better, though that might result in multiple locks, so might be worse overall)

Had offline discussion. We are keeping this for now, until we run some benchmarks.

marcocitus · 2023-03-21T08:14:01Z

src/backend/distributed/utils/attribute.c

+
+		if (MultiTenantMonitoringLogLevel != CITUS_LOG_LEVEL_OFF)
+		{
+			ereport(NOTICE, (errmsg("total select count = %d, total CPU time = %f "


NOTICE -> MultiTenantMonitoringLogLevel?

Is this log really necessary? I think I can remove it

JelteF

Overall I think this is a good start and we can use this as a good base for other sub-PRs to be merged against.

The main issue I see is the locking behaviour. I think we probably want to rethink that a bit. Because the current approach is quite complex and thus unsurprisingly has bugs, some of which I don't know if we can easily fix without changing the logic significantly.

I think the biggest problems stem from the fact that it's trying to divide priorities a bit too much. For example, each backend tries to reorder its own rank all the time. I think the locking logic would become much simpler if we don't do that and instead order the tenants only when we need to truncate the list.

This refactoring of the locking can be done in a separate sub-PR though, to keep it easy to review. Let's use the current branch as a sort-of main branch for now.

JelteF · 2023-03-21T09:04:19Z

src/test/regress/expected/citus_stats_tenants.out

+INSERT INTO dist_tbl VALUES (1, 'abcd');
+INSERT INTO dist_tbl VALUES (2, 'abcd');
+UPDATE dist_tbl SET b = a + 1 WHERE a = 3;
+UPDATE dist_tbl SET b = a + 1 WHERE a = 4;


Why is there no tenant 4 in the output below?

I set the citus.stats_tenants_limit to 2. Tenant 4 didn't run enough queries yet to be in top 2.

Ah, it's set to 2 in pg_regress_multi.pl. Any reason why it's set there instead of in this file? I think having it in this file would be more discoverable, and clearer when reading the test what is going on. I do realize that these tests change anyway in #6761, so maybe changing the default isn't even necessary anymore then.

JelteF · 2023-03-21T09:50:03Z

src/backend/distributed/utils/attribute.c

+void
+CitusAttributeToEnd(QueryDesc *queryDesc)


Missing comment

src/backend/distributed/shared_library_init.c

JelteF · 2023-03-21T10:41:24Z

src/backend/distributed/utils/attribute.c

+
+
+/*
+ * AttributeQueryIfAnnotated assigns the attributes of tenant if the query is annotated.


This sentence is a bit strange an it could use a longer description.

JelteF · 2023-03-21T11:19:52Z

src/include/distributed/utils/attribute.h

+typedef struct TenantStats
+{
+	char tenantAttribute[100];
+
+	int colocationGroupId;
+
+	int readCount;
+	double totalReadTime;
+	int readsInLastPeriod;
+	int readsInThisPeriod;
+
+	int writeCount;
+	double totalWriteTime;
+	int writesInLastPeriod;
+	int writesInThisPeriod;
+
+	time_t lastQueryTime;
+
+	long long score;
+	time_t lastScoreReduction;
+
+	NamedLWLockTranche namedLockTranche;
+	LWLock lock;
+} TenantStats;
+
+typedef struct MultiTenantMonitor
+{
+	time_t periodStart;
+
+	NamedLWLockTranche namedLockTranche;
+	LWLock lock;
+
+	int tenantCount;
+	TenantStats tenants[FLEXIBLE_ARRAY_MEMBER];
+} MultiTenantMonitor;


These structs could use some comments. Some things that would be useful:

What do the structs represent?

What are the different locks for?

How large is the tenants array?

JelteF · 2023-03-21T11:41:01Z

src/backend/distributed/utils/attribute.c

+
+
+/*
+ * FindTenantStats finds the dsm (dynamic shared memory) handle for the current tenant's statistics.


Suggested change

* FindTenantStats finds the dsm (dynamic shared memory) handle for the current tenant's statistics.

* FindTenantStats finds the index for the current tenant's statistics.

Good catch 👍

JelteF · 2023-03-21T11:42:29Z

src/backend/distributed/utils/attribute.c

+
+		if (tenantIndex == -1)
+		{
+			tenantIndex = CreateTenantStats(monitor);


Also a race condition if two tenants are created at the same time. One way to resolve this would be by incrementing tenantCount atomically in CreateTenantStats. That way concurrent calls would get a different index.

JelteF · 2023-03-21T12:04:06Z

src/backend/distributed/utils/attribute.c

+			TenantStats tempTenant = monitor->tenants[tenantIndex];
+			monitor->tenants[tenantIndex] = monitor->tenants[tenantIndex - 1];
+			monitor->tenants[tenantIndex - 1] = tempTenant;


This is copying the tenant lock. Copying locks is not an operation that's supported on them. They require that people look at the same memory locations. So if some other process is waiting on &monitor->tenants[tenantIndex - 1].lock while we do this swap then that's going to cause many problems.

JelteF · 2023-03-21T12:10:29Z

src/backend/distributed/utils/attribute.c

+#define ATTRIBUTE_PREFIX "/* attributeTo: "
+#define ATTRIBUTE_STRING_FORMAT "/* attributeTo: %s,%d */"
+#define CITUS_STATS_TENANTS_COLUMNS 7
+#define ONE_QUERY_SCORE 1000000000


Why is this such a high number?

This value will be halved every period, which is by default 1 minute. So in about half an hour this number will become 0.

JelteF · 2023-03-21T12:16:11Z

src/backend/distributed/utils/attribute.c

+		 * After updating the score we might need to change the rank of the tenant in the monitor
+		 */
+		while (tenantIndex != 0 &&
+			   monitor->tenants[tenantIndex - 1].score <


This is not threadsafe to do since we don't have a read lock on this tenant.

JelteF · 2023-03-24T09:56:03Z

src/backend/distributed/utils/attribute.c

+
+		MultiTenantMonitor *monitor = GetMultiTenantMonitor();
+
+		LWLockAcquire(&monitor->lock, LW_EXCLUSIVE);


This is a really broad lock that we shouldn't acquire exclusively when not necessary. Now we acquire it for every update of the stats, effectively making the finer grained per-tenant lock completely unused. We should only take this lock exclusively when modifying the order of the tenants.

Notes from meeting

SHARE_LOCK find tenants if found tenants -> modify tenant else release lock acquire EXCLUSIVE_LOCK find not find tenanants -> create tenant release ECLUSIVE LOCK SHARE_LOCK find tenant if found tenant -> modify tenant else -> stop

After discussing with @marcocitus: It's probably nice to only sort the array in the exclusive lock some percentage of the time, for the many tenants case (e.g. deviceids). This can be done either probabilistically, or based on time since last taking the lock to create a tenant, or a combination of both. So e.g. if (lastAdditionByThisProcess > 5 seconds ago || random() > 0.001). Not necessarily now, more as something to try out if benchmarks show perf issues.

JelteF · 2023-03-24T13:31:12Z

src/backend/distributed/utils/attribute.c

+		 *
+		 * Every time tenant count hits CitusStatsTenantsLimit * 3, we reduce it back to CitusStatsTenantsLimit * 2.
+		 */
+		if (monitor->tenantCount >= CitusStatsTenantsLimit * 3)


Should be part of createtenant check

JelteF · 2023-03-24T13:55:30Z

src/include/distributed/utils/attribute.h

+	/*
+	 * The time that the monitor's latest period started.
+	 * This value is incremented by citus.stats_tenants_period each time.
+	 * The increment is done lazily.
+	 */
+	time_t periodStart;


After discussion: We don't need to store this in the monitor anymore. It can be a local variable everywhere that we use it.

JelteF · 2023-03-24T14:19:30Z

src/include/distributed/utils/attribute.h

+	 * and is 3 * citus.stats_tenants_limit
+	 */
+	int tenantCount;
+	TenantStats tenants[FLEXIBLE_ARRAY_MEMBER];


After discussing with @marcocitus: Should this be a hashtable? Hashtables are O(1) for lookups, but small arrays can be faster in practice due to cache locality. We use shared memory hash table in SharedConnectionStatsShmemInit. If benchmarks show that the current approach is slow, we could try out a hashtable.

JelteF · 2023-03-24T15:15:06Z

src/backend/distributed/utils/attribute.c

+	/*
+	 * If the tenant count reached 3 * CitusStatsTenantsLimit, we evict the tenants
+	 * with the lowest score.
+	 */
+	EvictTenantsIfNecessary(queryTime);


Let's move this to before we insert the new tenant.

JelteF · 2023-03-24T15:16:22Z

src/backend/distributed/utils/attribute.c

+static void
+RecordTenantStats(TenantStats *tenantStats)
+{
+	tenantStats->score += ONE_QUERY_SCORE;


We should make sure this addition doesn't overflow. Making it clamp to MAX_INT64 seems fine.

marcocitus · 2023-03-27T09:29:23Z

src/include/distributed/utils/attribute.h

+	 * The tenant monitoring score of this tenant. This value is increased by ONE_QUERY_SCORE at every query
+	 * and halved after every period.
+	 */
+	long long score;


is this a particular established algorithm? (it looks vaguely familiar)

I made research a bit and then made this algorithm up. I'm halving just to make it more efficient by using bit shift left operator <<.
This is open to any suggestions.

I think explaining the goals of the algorithm in the comment would be good. Also probably good to mention that it's a custom algorithm. Mainly in case we realise it doesn't do what we want in some scenarios.

JelteF · 2023-03-27T09:39:48Z

src/test/regress/pg_regress_multi.pl

@@ -487,6 +487,7 @@ sub generate_hba
 push(@pgOptions, "citus.enable_manual_changes_to_shards=on");
 push(@pgOptions, "citus.allow_unsafe_locks_from_workers=on");
 push(@pgOptions, "citus.stat_statements_track = 'all'");
+push(@pgOptions, "citus.stats_tenants_limit = 2");


Any reason to have this change globally instead of only in the test that we care about it?

Since we set the shared memory at the beginning, I thought I have to set this up before setting up the cluster

ah yeah makes sense

JelteF · 2023-03-27T14:59:42Z

src/backend/distributed/utils/attribute.c

+		LWLockAcquire(&monitor->lock, LW_SHARED);
+
+		int currentTenantIndex = FindTenantStats(monitor);
+
+		if (currentTenantIndex != -1)
+		{


locking logic could definitely use some comments, because it's not trivial

JelteF · 2023-03-27T15:01:37Z

src/backend/distributed/utils/attribute.c

+
+
+/*
+ * CreateTenantStats creates the data structure for a tenant's statistics.


Suggested change

* CreateTenantStats creates the data structure for a tenant's statistics.

* CreateTenantStats creates the data structure for a tenant's statistics.

* NOTE: Calling this function requires that the monitor->lock is held in LW_EXCLUSIVE mode

marcocitus · 2023-03-28T09:31:31Z

src/backend/distributed/planner/deparse_shard_query.c

+												  partitionColumnType);
+		}
+
+		task->partitionColumn = partitionColumnString;


partitionColumn suggests that this contains a description/Var of the partition column

I can also use partitionColumnString then

or maybe partitionColumnValueString?

I changed this to partitionKeyValue and it is now a Const* value like in Job struct

marcocitus · 2023-03-28T09:34:20Z

src/backend/distributed/shared_library_init.c

+		gettext_noop("monitor limit"),
+		NULL,
+		&CitusStatsTenantsLimit,
+		10, 1, 100,


These values seem pretty low, which does not make me super confident that we will actually be able to capture the top tenants.

For comparison, our TopN extension defaults to 1000.

Let's set the default to 100, and the maximum to 10000.

marcocitus · 2023-03-28T09:40:54Z

src/backend/distributed/planner/distributed_planner.c

@@ -157,6 +158,8 @@ distributed_planner(Query *parse,
 	bool fastPathRouterQuery = false;
 	Node *distributionKeyValue = NULL;

+	AttributeQueryIfAnnotated(query_string, parse->commandType);


could we move it ot the end of this function? the logic here is quite important to understanding the planner and already quite messy and verbose

marcocitus · 2023-03-28T09:41:22Z

src/backend/distributed/utils/attribute.c

+ * for the tenant statistics monitoring this function records the tenant attributes.
+ */
+void
+AttributeQueryIfAnnotated(const char *query_string, CmdType commandType)


is there a way to disable the annotation injection & parsing?

Just merged #6769 I think it does what you're asking for.

marcocitus · 2023-03-28T09:41:48Z

src/backend/distributed/utils/attribute.c

+ * sleep_until_next_period sleeps until the next monitoring period starts.
+ */
+Datum
+sleep_until_next_period(PG_FUNCTION_ARGS)


move to a test directory

marcocitus · 2023-03-28T09:42:51Z

src/backend/distributed/utils/attribute.c

+ * clean_citus_stats_tenants cleans the citus_stats_tenants monitor.
+ */
+Datum
+clean_citus_stats_tenants(PG_FUNCTION_ARGS)


to be consistent with pg/citus_stat_statements, should we call this citus_stat_tenants_reset and make it a properly exposed function?

marcocitus · 2023-03-28T09:45:54Z

src/backend/distributed/sql/udfs/citus_stats_tenants/latest.sql

@@ -0,0 +1,66 @@
+-- cts in the query is an abbreviation for citus_stats_tenants
+CREATE OR REPLACE FUNCTION pg_catalog.citus_stats_tenants (


could we use citus_stat_tenants (without the extra s) to be consistent with existing pg_stat_ and citus_stat_ views?

@mulander this seems like a typo?

Yes it is. I have this in the task list in the description. I was waiting for other PRs to be merged to fix this. I'll get to it.

Merged a PR to fix this

marcocitus · 2023-03-28T09:48:35Z

src/backend/distributed/planner/deparse_shard_query.c

+			partitionColumnType = workerJob->partitionKeyValue->consttype;
+			partitionColumnString = DatumToString(partitionColumnValue,
+												  partitionColumnType);
+		}


maybe wrap in something like task->partitionColumnValueString = PartitionColumnValueStringFromJob(workerJob) since this seems to repeat in a few places

related, do we need to do the string conversion at this particular moment, or could we do it more lazily?

I moved it to end functions

marcocitus · 2023-03-28T09:49:50Z

src/backend/distributed/planner/deparse_shard_query.c

+		Datum partitionColumnValue;
+		Oid partitionColumnType = 0;
+		char *partitionColumnString = NULL;
+		if (workerJob->partitionKeyValue != NULL)


would it be possible to refactor partitionKeyValue out of workerJob? it seems like it's not the right place, since a job can span across many partition column values

Do you mean Task? It was already in the Job before this PR.

marcocitus · 2023-03-28T09:51:27Z

src/backend/distributed/utils/attribute.c

+
+		if (MultiTenantMonitoringLogLevel != CITUS_LOG_LEVEL_OFF)
+		{
+			ereport(NOTICE, (errmsg("attributing query to tenant: %s",


NOTICE -> MultiTenantMonitoringLogLevel ?

(otherwise, why is this a log level?)

Deleted this log

marcocitus · 2023-03-28T09:52:32Z

src/backend/distributed/utils/attribute.c

+ * AnnotateQuery annotates the query with tenant attributes.
+ */
+char *
+AnnotateQuery(char *queryString, char *partitionColumn, int colocationId)


partitionColumn sounds like it contains the name of the partition column

Changed this to partitionKeyValue

marcocitus · 2023-03-28T09:53:44Z

src/backend/distributed/utils/attribute.c

+static void
+AttributeMetricsIfApplicable()
+{
+	if (strcmp(attributeToTenant, "") != 0)


maybe

if (attributeToTenant[0] == '\0') { /* no tenant ID to attribute */ return; }

marcocitus · 2023-03-28T09:56:09Z

src/backend/distributed/utils/attribute.c

+		clock_t end = { 0 };
+
+		end = clock();
+		time_t queryTime = time(0);


we usually use postgres' time functions like GetCurrentTimestamp()

marcocitus · 2023-03-28T09:57:14Z

src/backend/distributed/utils/attribute.c

+		{
+			LWLockRelease(&monitor->lock);
+
+			LWLockAcquire(&monitor->lock, LW_EXCLUSIVE);


we should be careful that this does not happen often for typical CRUD workloads (e.g. distribution column is bigserial), otherwise we get significant contention

I'm turning the tracking off by default

JelteF · 2023-03-30T15:46:00Z

src/backend/distributed/utils/attribute.c

+char attributeToTenant[MAX_TENANT_ATTRIBUTE_LENGTH] = "";
+CmdType attributeToCommandType = CMD_UNKNOWN;
+int attributeToColocationGroupId = INVALID_COLOCATION_ID;
+
+const char *SharedMemoryNameForMultiTenantMonitor =
+	"Shared memory for multi tenant monitor";
+
+char *tenantTrancheName = "Tenant Tranche";
+char *monitorTrancheName = "Multi Tenant Monitor Tranche";


nit: These could all be static afaict, because they are not used outside of this file. Also in general we try to use CapitalizedCamelCase for global variables to make it clear that they are special.

Suggested change

char attributeToTenant[MAX_TENANT_ATTRIBUTE_LENGTH] = "";

CmdType attributeToCommandType = CMD_UNKNOWN;

int attributeToColocationGroupId = INVALID_COLOCATION_ID;

const char *SharedMemoryNameForMultiTenantMonitor =

"Shared memory for multi tenant monitor";

char *tenantTrancheName = "Tenant Tranche";

char *monitorTrancheName = "Multi Tenant Monitor Tranche";

char attributeToTenant[MAX_TENANT_ATTRIBUTE_LENGTH] = "";

CmdType attributeToCommandType = CMD_UNKNOWN;

int attributeToColocationGroupId = INVALID_COLOCATION_ID;

const char *SharedMemoryNameForMultiTenantMonitor =

"Shared memory for multi tenant monitor";

char *tenantTrancheName = "Tenant Tranche";

char *monitorTrancheName = "Multi Tenant Monitor Tranche";

JelteF · 2023-03-31T12:43:03Z

src/backend/distributed/shared_library_init.c

@@ -2283,6 +2304,37 @@ RegisterCitusConfigVariables(void)
 		GUC_STANDARD,
 		NULL, NULL, NULL);

+	DefineCustomEnumVariable(
+		"citus.stat_tenants_track",
+		gettext_noop("enable disable"),


The descriptions of these GUCs should be improved

JelteF · 2023-03-31T12:43:26Z

src/backend/distributed/shared_library_init.c

+		gettext_noop("monitor period"),
+		NULL,
+		&CitusStatsTenantsPeriod,
+		60, 1, 1000000000,


The max seems quite excessively high. I think 1 day is probably the most that it useful to anyone.

I initially made this 60 * 60, so 1 hour. But in testing I needed to make sure a period doesn't pass when I don't want it to. That's why I increased it to this number.
What do you think I can do instead?

60*60*24 seems like a good maximum

Add a view that collects statistics from all nodes

Creates log message for citus_stats_tenants

This pull request modifies our query annotation and parsing logic by using a JSON-structured annotation string. Basically, it prepends a JSON string in a multiline comment that contains `tenantId` and `colocationId` to a query string to be able to track query statistics on the worker nodes. It also parses the received annotation in the query string and sets the relevant tenantId and colocationId on the worker nodes. --------- Co-authored-by: Jelte Fennema <github-tech@jeltef.nl>

Adds a GUC for turning tenant monitoring on and off

#6812) This PR revokes permissions from stats views and functions from the public role and grants `SELECT` and `EXECUTE` to pg_monitor.

Rename some objects `*citus_stats_tenants*` to `*citus_stat_tenants*` `citus_stats_tenants*` and `citus_stat_tenants*` GUCs to `stat_tenants*` `MultiTenantMonitoringLogLevel` to `CitusStatTenantsLogLevel` `multi_tenant_monitoring_log_level` to `citus.stat_tenants_log_level` `attribute` files to `citus_stat_tenants`

Turn it back on for in tests Also address some reviews

halilozanakgul force-pushed the multi-tenant-monitoring branch from 484a6aa to 01eb68a Compare March 6, 2023 14:25

JelteF reviewed Mar 14, 2023

View reviewed changes

marcocitus reviewed Mar 14, 2023

View reviewed changes

marcocitus reviewed Mar 21, 2023

View reviewed changes

JelteF suggested changes Mar 21, 2023

View reviewed changes

JelteF reviewed Mar 24, 2023

View reviewed changes

halilozanakgul force-pushed the multi-tenant-monitoring branch 2 times, most recently from 7707875 to 51d167b Compare March 27, 2023 09:16

marcocitus reviewed Mar 27, 2023

View reviewed changes

JelteF reviewed Mar 27, 2023

View reviewed changes

halilozanakgul force-pushed the multi-tenant-monitoring branch from d786b2f to d660339 Compare March 27, 2023 14:14

JelteF reviewed Mar 27, 2023

View reviewed changes

marcocitus reviewed Mar 28, 2023

View reviewed changes

JelteF reviewed Mar 30, 2023

View reviewed changes

JelteF reviewed Mar 31, 2023

View reviewed changes

halilozanakgul force-pushed the multi-tenant-monitoring branch from 3e2cf73 to e302be1 Compare April 3, 2023 15:14

halilozanakgul marked this pull request as ready for review April 4, 2023 10:52

halilozanakgul force-pushed the multi-tenant-monitoring branch from aeb604b to 14ad3ef Compare April 4, 2023 10:54

JelteF approved these changes Apr 5, 2023

View reviewed changes

halilozanakgul and others added 12 commits April 5, 2023 17:06

Add multi tenant statistics monitoring

c210e8c

Citus stats tenants collector view (#6761)

ff0db22

Add a view that collects statistics from all nodes

Add logs to citus_stats_tenants (#6762)

14487fd

Creates log message for citus_stats_tenants

Attribute local queries and cached plans on local execution (#6797)

20a80e4

Citus stats tenants guc (#6769)

72daa79

Adds a GUC for turning tenant monitoring on and off

Revoke statistics permissions from public and grant them to pg_monitor (

089e533

#6812) This PR revokes permissions from stats views and functions from the public role and grants `SELECT` and `EXECUTE` to pg_monitor.

Address reviews on the PR

61a2f00

Turn tracking off by default

f68c828

Turn it back on for in tests Also address some reviews

Update stat_tenants_limit

4269384

Update stat_tenants_period limit

83bfe90

halilozanakgul force-pushed the multi-tenant-monitoring branch from 3df5a55 to 83bfe90 Compare April 5, 2023 14:06

Update score comment

5980d30

halilozanakgul merged commit 52ad2d0 into main Apr 5, 2023

halilozanakgul deleted the multi-tenant-monitoring branch April 5, 2023 14:44



		/*
		* AttributeQueryIfAnnotated assigns the attributes of tenant if the query is annotated.



		/*
		* FindTenantStats finds the dsm (dynamic shared memory) handle for the current tenant's statistics.

	* FindTenantStats finds the dsm (dynamic shared memory) handle for the current tenant's statistics.
	* FindTenantStats finds the index for the current tenant's statistics.


		MultiTenantMonitor *monitor = GetMultiTenantMonitor();

		LWLockAcquire(&monitor->lock, LW_EXCLUSIVE);



		/*
		* CreateTenantStats creates the data structure for a tenant's statistics.

	* CreateTenantStats creates the data structure for a tenant's statistics.
	* CreateTenantStats creates the data structure for a tenant's statistics.
	* NOTE: Calling this function requires that the monitor->lock is held in LW_EXCLUSIVE mode

		@@ -0,0 +1,66 @@
		-- cts in the query is an abbreviation for citus_stats_tenants
		CREATE OR REPLACE FUNCTION pg_catalog.citus_stats_tenants (

Multi tenant monitoring #6725

Multi tenant monitoring #6725

Conversation

halilozanakgul commented Feb 17, 2023 • edited Loading

codecov bot commented Mar 10, 2023 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JelteF Mar 14, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

marcocitus Mar 14, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JelteF left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JelteF Mar 24, 2023 • edited Loading

Choose a reason for hiding this comment

JelteF Mar 24, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JelteF Mar 24, 2023 • edited Loading

Choose a reason for hiding this comment

marcocitus Mar 27, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

marcocitus Mar 28, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

marcocitus Mar 28, 2023 • edited Loading

Choose a reason for hiding this comment

JelteF Apr 4, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

halilozanakgul commented Feb 17, 2023 •

edited

Loading

codecov bot commented Mar 10, 2023 •

edited

Loading

JelteF Mar 14, 2023 •

edited

Loading

marcocitus Mar 14, 2023 •

edited

Loading

JelteF Mar 24, 2023 •

edited

Loading

JelteF Mar 24, 2023 •

edited

Loading

JelteF Mar 24, 2023 •

edited

Loading

marcocitus Mar 27, 2023 •

edited

Loading

marcocitus Mar 28, 2023 •

edited

Loading

marcocitus Mar 28, 2023 •

edited

Loading

JelteF Apr 4, 2023 •

edited

Loading