Remove dependency on placement caches from DROP

[hanefi]

When dropping objects, citus_drop_trigger should not depend on the validity of the caches for placements as it causes may cause error messages from time to time. With this commit we remove the depencency on the validity on caches for shard placements.

However, citus_drop_trigger still depends on WorkerNodeCache in 3 different codepaths listed below:

- master_remove_distributed_table_metadata_from_workers
 -> MasterRemoveDistributedTableMetadataFromWorkers
  -> SendCommandToWorkersWithMetadata
   -> SendCommandToMetadataWorkersParams
    -> TargetWorkerSetNodeList
     -> ActivePrimaryNonCoordinatorNodeList
      -> FilterActiveNodeListFunc
       -> GetWorkerNodeHash

- citus_drop_all_shards
 -> DropShards
  -> DropTaskList
   -> ShardPlacementListIncludingOrphanedPlacementsViaCatalog
    -> LookupNodeForGroup
     -> PrepareWorkerNodeCache

- master_remove_partition_metadata
 -> DeleteColocationGroupIfNoTablesBelong
  -> DeleteColocationGroup
   -> SyncDeleteColocationGroupToNodes
    -> SendCommandToWorkersWithMetadataViaSuperUser
     -> SendCommandToMetadataWorkersParams
      -> TargetWorkerSetNodeList
       -> ActivePrimaryNonCoordinatorNodeList
        -> FilterActiveNodeListFunc
         -> GetWorkerNodeHash

Related: #5372 #6041

[hanefi]

Apparently I broke DROP commands here. Let me post some updates on the failures.

DROP Commands complete just fine, with some extra notice messages that say shards do not exist.

 DROP TABLE data_load_test1, data_load_test2;

 ... redacted some output ...

+NOTICE:  table "data_load_test1_360021" does not exist, skipping
+CONTEXT:  SQL statement "SELECT citus_drop_all_shards(v_obj.objid, v_obj.schema_name, v_obj.object_name, drop_shards_metadata_only := false)"
+PL/pgSQL function citus_drop_trigger() line 25 at PERFORM
+NOTICE:  table "data_load_test1_360021" does not exist, skipping
+CONTEXT:  SQL statement "SELECT citus_drop_all_shards(v_obj.objid, v_obj.schema_name, v_obj.object_name, drop_shards_metadata_only := false)"
+PL/pgSQL function citus_drop_trigger() line 25 at PERFORM
+NOTICE:  table "data_load_test1_360022" does not exist, skipping
+CONTEXT:  SQL statement "SELECT citus_drop_all_shards(v_obj.objid, v_obj.schema_name, v_obj.object_name, drop_shards_metadata_only := false)"
+PL/pgSQL function citus_drop_trigger() line 25 at PERFORM

Later in the same test we see that those shards are still in worker nodes, and they are not dropped.

 -- There should be no table on the worker node
 \c - - :public_worker_1_host :worker_1_port
 SELECT relname FROM pg_class WHERE relname LIKE 'data_load_test%';
         relname         
 ------------------------
-(0 rows)
+ data_load_test1_360021
+ data_load_test1_360022
+ data_load_test1_360023
+ data_load_test1_360024
+ data_load_test2_360025
+ data_load_test2_360026
+ data_load_test2_360027
+ data_load_test2_360028
+(8 rows)

I guess I sent the DROP commands for shards to the coordinator somehow instead of the correct worker node. I continue investigating.

Test failures are from this CI run

[hanefi]

This PR also aims to fix the errors that we see in the codepath shared in #5780 (comment)

I haven't tested it fully, but I will try the repro steps and other examples shared on the issue I linked above.

[hanefi]

Superseded by #6687