Partition Usage for Gathering Data on Subset of Processes

[tmarrinan]

Hello.
I am trying to repartition data published to Ascent. My use case is for only a subset (possibly only a single) of MPI processes to perform some form of data analysis on the simulation data. I therefore want to gather data from all MPI processes on just one or a few target processes.
The partition pipeline seems like it may be well-suited for this, but it's hard to tell from the documentation just how to specify the desired data layout. Can this be done, or is the partition pipeline more just used for automatic repartitioning of data for better load balancing?

I'll also provide two example (simplified) uses in case my description above isn't fully clear.

Case 1:

2D uniform grid of scalar values split amongst 4 MPI processes:

+------------+------------+
|            |            |
|     0      |     1      |
|            |            |
+------------+------------+
|            |            |
|     2      |     3      |
|            |            |
+------------+------------+

I need all data gathered on rank 0:

+-------------------------+
|                         |
|                         |
|                         |
|            0            |
|                         |
|                         |
|                         |
+-------------------------+

Case 2:

3D triangular mesh with subregions owned amongst 8 MPI processes.

I need data gathered on 2 processes so that each of those 2 processes own their original subregion along with 3 other subregions (i.e. each of the 2 processes now own half of the triangular mesh).

[JustinPrivitera]

Maybe this will help while we update the docs: https://ascent--1412.org.readthedocs.build/en/1412/Actions/Pipelines.html#partitioning

[tmarrinan]

Thank you!
I'm still not sure I 100% follow the documentation. If this is how scalar data is published to Ascent in my Case 1:

uint32_t scalar_size = dim_x * dim_y;

conduit::Node data;
data["state/domain_id"] = process_id;
    
data["coordsets/coords/type"] = "uniform";
data["coordsets/coords/dims/i"] = dim_x;
data["coordsets/coords/dims/j"] = dim_y;

data["coordsets/coords/origin/x"] = offset_x;
data["coordsets/coords/origin/y"] = offset_y;
data["coordsets/coords/spacing/dx"] = 1;
data["coordsets/coords/spacing/dy"] = 1;

data["topologies/topo/type"] = "uniform";
data["topologies/topo/coordset"] = "coords";
    
data["fields/scalar1/association"] = "vertex";
data["fields/scalar1/topology"] = "topo";
data["fields/scalar1/values"].set_external(scalar1_data, scalar_size );

data["fields/scalar2/association"] = "vertex";
data["fields/scalar2/topology"] = "topo";
data["fields/scalar2/values"].set_external(scalar2_data, scalar_size );

data["fields/scalar3/association"] = "vertex";
data["fields/scalar3/topology"] = "topo";
data["fields/scalar3/values"].set_external(scalar3_data, scalar_size );

ascent.publish(data);

Then is this how I would aggregate a single scalar field on process 0?

conduit::Node actions;
conduit::Node &add_pipelines= actions.append();
add_pipelines["action"] = "add_pipelines";
conduit::Node &pipelines= add_pipelines["piplines"];
pipelines["pl1/f1/type"] = "partition";
pipelines["pl1/f1/params/target"] = 1; // gather data on a single process (how to know which one)?
pipelines["pl1/f1/params/selections/type"] = "logical";
// for selection, is this part being sent by each process or total region that is wanted? Also, is end inclusive or exclusive?
pipelines["pl1/f1/params/selections/start"] = [0,0]; 
pipelines["pl1/f1/params/selections/end"] = [dim_x, dim_y];
// how to just gather field `scalar2`? I see selection type "field", but then I cannot specify logical region

I'm also not sure about the syntax for lists. The documentation shows:

pipelines["pl1/f1/params/fields"] = ["pink","pony","club"];

But that does not seem like valid C++. Would The following work?

const char* field_names[3] = {"pink","pony","club"};
pipelines["pl1/f1/params/fields"].set(field_names, 3);

[nicolemarsaglia]

@tmarrinan I'm currently working on a bug impacting the partition filter. Currently the filter only accepts a target value. A target=1 should default to the root process. Once I fix this bug there will be a destination_ranks that can be set.

For gathering all domains of a single field on one process I would (eventually) do it like so:

conduit::Node actions;
conduit::Node &add_pipelines= actions.append();
add_pipelines["action"] = "add_pipelines";
conduit::Node &pipelines= add_pipelines["piplines"];
pipelines["pl1/f1/type"] = "partition";
pipelines["pl1/f1/params/target"] = 1; 
pipelines["pl1/f1/params/selections/type"] = "field";
pipelines["pl1/f1/params/selections/field"] = "scalar2";
pipelines["pl1/f1/params/selections/destination_ranks"] = 0;

I'm guessing there probably is a way of doing a mesh selection of an individual field, but I'm not entirely sure. All of these parameters are optional so how they combine can be confusing. All of this uses and is based off of Conduit Blueprint's partition, which has more sufficient docs with fewer errors: https://llnl-conduit.readthedocs.io/en/latest/blueprint_mesh_partition.html#partitioning

[JustinPrivitera]

You can just use Conduit directly to re-partition for now instead of asking Ascent to call Conduit.

[tmarrinan]

Thanks for all the information.
I'm still struggling a bit. I'm trying to use Conduit directly now and have tried to set it up as follows:

conduit::Node options, output;
options["target"] = 1;
options["fields"] = {"scalar2"};
conduit::blueprint::mpi::mesh::partition(data, options, output, MPI_COMM_WORLD);

But I see a message saying "Unable to combine domains as uniform, using unstructured."

[tmarrinan]

OK - after a bit more reading and testing - I think I have it working!

There were 2 key things I needed to change (one with the mesh and one with the partition options):

1. Change scalar value association from "vertex" to "element" (this also meant that the coordinate dims needed to be increased by 1)
2. Have an array of selections (one per process), adding a proper "domain_id" and using start and end values that match the local data

Final solution:

uint32_t scalar_size = dim_x * dim_y;

conduit::Node data;
data["state/domain_id"] = process_id;

data["coordsets/coords/type"] = "uniform";
data["coordsets/coords/dims/i"] = dim_x+ 1;
data["coordsets/coords/dims/j"] = dim_y + 1;

data["coordsets/coords/origin/x"] = offset_x;
data["coordsets/coords/origin/y"] = offset_y;
data["coordsets/coords/spacing/dx"] = 1;
data["coordsets/coords/spacing/dy"] = 1;

data["topologies/topo/type"] = "uniform";
data["topologies/topo/coordset"] = "coords";

data["fields/scalar1/association"] = "element";
data["fields/scalar1/topology"] = "topo";
data["fields/scalar1/values"].set(scalar1_values, scalar_size);

data["fields/scalar2/association"] = "element";
data["fields/scalar2/topology"] = "topo";
data["fields/scalar2/values"].set(scalar2_values, scalar_size);

data["fields/scalar3/association"] = "element";
data["fields/scalar3/topology"] = "topo";
data["fields/scalar3/values"].set(scalar3_values, scalar_size);

int i;
conduit::Node options, selections, output;
for (i = 0; i < num_processes; i++)
{
    conduit::Node &selection = selections.append();
    selection["type"] = "logical";
    selection["domain_id"] = i;
    // NOTE: may need different "start" and "end" values if all processes do not own same size region
    selection["start"] = {0u, 0u, 0u};
    selection["end"] = {dim_x, dim_y, 1u};
}
options["target"] = 1;
options["fields"] = {"scalar2"};
options["selections"] = selections;
options["mapping"] = 0;

conduit::blueprint::mpi::mesh::partition(data, options, output, MPI_COMM_WORLD);

This resulted in the following output (12 processes, each responsible for a 4x4 subregion in an overall 16x12 grid; each process filled its local region with floating point values equal to its process id):

state:
  domain_id: 0
coordsets:
  coords:
    type: "uniform"
    origin:
      x: 0.0
      y: 0.0
    dims:
      i: 17
      j: 13
topologies:
  topo:
    type: "uniform"
    coordset: "coords"
fields:
  scalar2:
    topology: "topo"
    association: "element"
    values: [0.0, 0.0, 0.0, 0.0, 1.0, 1.0, 1.0, 1.0, 2.0, 2.0, 2.0, 2.0, 3.0, 3.0, 3.0, 3.0, 0.0, 0.0, 0.0, 0.0, 1.0, 1.0, 1.0, 1.0, 2.0, 2.0, 2.0, 2.0, 3.0, 3.0, 3.0, 3.0, 0.0, 0.0, 0.0, 0.0, 1.0, 1.0, 1.0, 1.0, 2.0, 2.0, 2.0, 2.0, 3.0, 3.0, 3.0, 3.0, 0.0, 0.0, 0.0, 0.0, 1.0, 1.0, 1.0, 1.0, 2.0, 2.0, 2.0, 2.0, 3.0, 3.0, 3.0, 3.0, 4.0, 4.0, 4.0, 4.0, 5.0, 5.0, 5.0, 5.0, 6.0, 6.0, 6.0, 6.0, 7.0, 7.0, 7.0, 7.0, 4.0, 4.0, 4.0, 4.0, 5.0, 5.0, 5.0, 5.0, 6.0, 6.0, 6.0, 6.0, 7.0, 7.0, 7.0, 7.0, 4.0, 4.0, 4.0, 4.0, 5.0, 5.0, 5.0, 5.0, 6.0, 6.0, 6.0, 6.0, 7.0, 7.0, 7.0, 7.0, 4.0, 4.0, 4.0, 4.0, 5.0, 5.0, 5.0, 5.0, 6.0, 6.0, 6.0, 6.0, 7.0, 7.0, 7.0, 7.0, 8.0, 8.0, 8.0, 8.0, 9.0, 9.0, 9.0, 9.0, 10.0, 10.0, 10.0, 10.0, 11.0, 11.0, 11.0, 11.0, 8.0, 8.0, 8.0, 8.0, 9.0, 9.0, 9.0, 9.0, 10.0, 10.0, 10.0, 10.0, 11.0, 11.0, 11.0, 11.0, 8.0, 8.0, 8.0, 8.0, 9.0, 9.0, 9.0, 9.0, 10.0, 10.0, 10.0, 10.0, 11.0, 11.0, 11.0, 11.0, 8.0, 8.0, 8.0, 8.0, 9.0, 9.0, 9.0, 9.0, 10.0, 10.0, 10.0, 10.0, 11.0, 11.0, 11.0, 11.0]
  original_element_ids:
    topology: "topo"
    association: "element"
    values:
      domains: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
      ids: [0, 1, 2, 3, 0, 1, 2, 3, 0, 1, 2, 3, 0, 1, 2, 3, 4, 5, 6, 7, 4, 5, 6, 7, 4, 5, 6, 7, 4, 5, 6, 7, 8, 9, 10, 11, 8, 9, 10, 11, 8, 9, 10, 11, 8, 9, 10, 11, 12, 13, 14, 15, 12, 13, 14, 15, 12, 13, 14, 15, 12, 13, 14, 15, 0, 1, 2, 3, 0, 1, 2, 3, 0, 1, 2, 3, 0, 1, 2, 3, 4, 5, 6, 7, 4, 5, 6, 7, 4, 5, 6, 7, 4, 5, 6, 7, 8, 9, 10, 11, 8, 9, 10, 11, 8, 9, 10, 11, 8, 9, 10, 11, 12, 13, 14, 15, 12, 13, 14, 15, 12, 13, 14, 15, 12, 13, 14, 15, 0, 1, 2, 3, 0, 1, 2, 3, 0, 1, 2, 3, 0, 1, 2, 3, 4, 5, 6, 7, 4, 5, 6, 7, 4, 5, 6, 7, 4, 5, 6, 7, 8, 9, 10, 11, 8, 9, 10, 11, 8, 9, 10, 11, 8, 9, 10, 11, 12, 13, 14, 15, 12, 13, 14, 15, 12, 13, 14, 15, 12, 13, 14, 15]
  original_vertex_ids:
    topology: "topo"
    association: "vertex"
    values:
      domains: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
      ids: [0, 1, 2, 3, 0, 1, 2, 3, 0, 1, 2, 3, 0, 1, 2, 3, 4, 5, 6, 7, 8, 5, 6, 7, 8, 5, 6, 7, 8, 5, 6, 7, 8, 9, 10, 11, 12, 13, 10, 11, 12, 13, 10, 11, 12, 13, 10, 11, 12, 13, 14, 15, 16, 17, 18, 15, 16, 17, 18, 15, 16, 17, 18, 15, 16, 17, 18, 19, 0, 1, 2, 3, 0, 1, 2, 3, 0, 1, 2, 3, 0, 1, 2, 3, 4, 5, 6, 7, 8, 5, 6, 7, 8, 5, 6, 7, 8, 5, 6, 7, 8, 9, 10, 11, 12, 13, 10, 11, 12, 13, 10, 11, 12, 13, 10, 11, 12, 13, 14, 15, 16, 17, 18, 15, 16, 17, 18, 15, 16, 17, 18, 15, 16, 17, 18, 19, 0, 1, 2, 3, 0, 1, 2, 3, 0, 1, 2, 3, 0, 1, 2, 3, 4, 5, 6, 7, 8, 5, 6, 7, 8, 5, 6, 7, 8, 5, 6, 7, 8, 9, 10, 11, 12, 13, 10, 11, 12, 13, 10, 11, 12, 13, 10, 11, 12, 13, 14, 15, 16, 17, 18, 15, 16, 17, 18, 15, 16, 17, 18, 15, 16, 17, 18, 19, 20, 21, 22, 23, 20, 21, 22, 23, 20, 21, 22, 23, 20, 21, 22, 23, 24]

[nicolemarsaglia]

This is great! Thanks for reporting back!

Your error with the uniform vs unstructured is certainly odd, and I would have thought that would work as well. But I'm glad you were able to figure out a working solution.

[tmarrinan]

Yes, the tricky part was realizing that which data came from which domain_id needed to be manually selected using an array of "selections" rather than just specifying the desired region and letting Conduit determine who owned that data. There were no examples in the documentation with multiple selections, so it was a bit of trial-and-error.

[tmarrinan]

So I have it working using Conduit directly, but now trying to make it an Ascent action. I am trying something like the following:

action: "add_pipelines"
  pipelines:
    pl1:
      f1:
        type: "partition"
        params:
          target: 1
          fields: "scalar2"
          selections:
            -
              type: "logical"
              domain_id: 0
              start: [0, 0, 0]
              end: [3, 3, 0]
            -
              type: "logical"
              domain_id: 1
              start: [0, 0, 0]
              end: [3, 3, 0]
            -
              type: "logical"
              domain_id: 2
              start: [0, 0, 0]
              end: [3, 3, 0]
            -
              type: "logical"
              domain_id: 3
              start: [0, 0, 0]
              end: [3, 3, 0]
          mapping: 0

However, I get an error from Ascent:

Cannot create filter pl1_f1_partition(type: blueprint_data_partition) because verify_params failed.

errors:
  - "Surprise parameter 'params/selections/[0]/type'\nSurprise parameter 'params/selections/[0]/domain_id'\nSurprise parameter 'params/selections/[0]/start'\nSurprise parameter 'params/selections/[0]/end'\nSurprise parameter 'params/selections/[1]/type'\nSurprise parameter 'params/selections/[1]/domain_id'\nSurprise parameter 'params/selections/[1]/start'\nSurprise parameter 'params/selections/[1]/end'\nSurprise parameter 'params/selections/[2]/type'\nSurprise parameter 'params/selections/[2]/domain_id'\nSurprise parameter 'params/selections/[2]/start'\nSurprise parameter 'params/selections/[2]/end'\nSurprise parameter 'params/selections/[3]/type'\nSurprise parameter 'params/selections/[3]/domain_id'\nSurprise parameter 'params/selections/[3]/start'\nSurprise parameter 'params/selections/[3]/end'\n"

I notice from the earlier conversation, that selections seemed to be a single object rather than a list. Is that the issue? If so, how would I remedy that?

[JustinPrivitera]

I think other developers are currently working on fixes to the partitioner: #1412

Maybe one of them can weigh in on your question.

[nicolemarsaglia]

Hi @tmarrinan that is not the issue, unfortunately the issue is the selection functionality is not currently working. I have a branch: task/2024_parition_docs that should allow for selection. I'm still in the process of testing unit cases.

If you do test out this branch, I think you'd want to change your selection to something like the following:

action: "add_pipelines"
  pipelines:
    pl1:
      f1:
        type: "partition"
        params:
          target: 1
          fields: "scalar2"
          selections:
              type: "logical"
              domain_id: [0,1,2,3,4]
              start: [0, 0, 0]
              end: [3, 3, 0]
          mapping: 0

[tmarrinan]

Thanks!
In my actual use, each process may have different start and end values (e.g. 2D grid not evenly divisible amongst processes, so some will be 1 cell wider). What might be the way to specify unique start and end values if not using a list for selections?

[nicolemarsaglia]

Hmm @tmarrinan great question! A list of selections seems like the way to do it. I'll have to check if the functionality exists in conduit for that option!

[tmarrinan]

Fortunately it does!
I can accomplish the partitioning as desired using conduit directly, but am hoping to eventually integrate it into my Ascent workflow.

Here's how I create the partitioning programmatically:

uint32_t *local_offsets = ... // array of x and y offsets in local data for each process (accounts for potential ghost cells at the boundaries)
uint32_t *local_sizes = ... // array of x and y dimensions of local data for each process (excludes potential ghost cells)
conduit::Node options, selections, output;
for (i = 0; i < num_ranks; i++)
{
    conduit::Node &selection = selections.append();
    uint32_t start_x = local_offsets[2 * i + 0];
    uint32_t start_y = local_offsets[2 * i + 1];
    selection["type"] = "logical";
    selection["domain_id"] = i;
    selection["start"] = {start_x, start_y, 0u};
    selection["end"] = {start_x + local_sizes[2 * i + 0] - 1u, start_y + local_sizes[2 * i + 1] - 1u, 0u};
}
options["target"] = 1;
options["fields"] = {"scalar2"};
options["selections"] = selections;
options["mapping"] = 0;

conduit::blueprint::mpi::mesh::partition(mesh, options, output, MPI_COMM_WORLD);

The resulting options yaml for a 200x80 grid split between 4 processes looks like the following:

target: 1
fields: "scalar2"
selections:
  -
    type: "logical"
    domain_id: 0
    start: [0, 0, 0]
    end: [99, 39, 0]
  -
    type: "logical"
    domain_id: 1
    start: [1, 0, 0]
    end: [100, 39, 0]
  -
    type: "logical"
    domain_id: 2
    start: [0, 1, 0]
    end: [99, 40, 0]
  -
    type: "logical"
    domain_id: 3
    start: [1, 1, 0]
    end: [100, 40, 0]
mapping: 0

[nicolemarsaglia]

oh awesome! I'll have to figure out how best to pass this to Ascent, but it's great to know this is how it works! Thanks for the info!