Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

apply_neighborhood: Data type (datacube/array) in the callback #387

Closed
m-mohr opened this issue Sep 30, 2022 · 1 comment · Fixed by #390
Closed

apply_neighborhood: Data type (datacube/array) in the callback #387

m-mohr opened this issue Sep 30, 2022 · 1 comment · Fixed by #390
Labels
breaking help wanted Extra attention is needed minor
Milestone

Comments

@m-mohr
Copy link
Member

m-mohr commented Sep 30, 2022

Process ID: apply_neighborhood

Describe the issue:
apply_neighborhood is incompatible with the array processes that we typically use in callbacks. So you can currently only use datacube-based processes such as run_udf, apply and apply_dimension in it, but not for example mean, first or array_element.

Proposed solution:
Potential solutions for it:

  1. an explicit datacube to 1D array function (e.g. datacube_to_array(data: datacube) -> labeled-array)
  2. Add labeled-array as a data type to the callback schema (see process description below)
  3. use apply_dimension on the remaining dimension (a bit cumbersome)

It doesn't really work if backends implicitly flatten a datacube into a 1D array when needed, because that doesn't fit into the strictly types process descriptions. User would not expect that. Instead, solution 2 would be required.

I have no strong preference yet.

Note: apply_neighborhood is unfortunately not in the draft state and as such a breaking change requires to wait for v2 (which is planned anyway for vector data cubes).

Additional context:
Origin: https://discuss.eodc.eu/t/moving-average-with-low-pass-filter/428/12

Proposal for 2 (only the process parameter):

{
  "name": "process",
  "description": "Process to be applied on all neighborhoods.",
  "schema": {
    "type": "object",
    "subtype": "process-graph",
    "parameters": [
      {
        "name": "data",
        "description": "The input data, which is a subset of the data cube as specified in `size` and `overlap`. If the given size and overlap result in a one-dimensional data cube it is converted to an array.",
        "schema": [
          {
            "title": "Multi-dimensional data",
            "type": "object",
            "subtype": "raster-cube"
          },
          {
            "title": "One-dimensional data",
            "type": "array",
            "subtype": "labeled-array"
          }
        ]
      },
      {
        "name": "context",
        "description": "Additional data passed by the user.",
        "schema": {
          "description": "Any data type."
        },
        "optional": true,
        "default": null
      }
    ],
    "returns": {
      "description": "An array or data cube with the newly computed values. The data type and dimensionality must correspond to the input data. For data cubes this means it must have the same dimensions and the dimension properties (name, type, labels, reference system and resolution) must remain unchanged. Otherwise, a `DataCubePropertiesImmutable` exception will be thrown.",
      "schema": [
        {
          "title": "Multi-dimensional data",
          "type": "object",
          "subtype": "raster-cube"
        },
        {
          "title": "One-dimensional data",
          "type": "array",
          "subtype": "labeled-array"
        }
      ]
    }
  }
}
@m-mohr
Copy link
Member Author

m-mohr commented Oct 5, 2022

I did not hear a clear tendency in the call, but for Huriels use case we may start with UDFs for now...

@m-mohr m-mohr added this to the 2.0.0 milestone Oct 9, 2022
@m-mohr m-mohr added help wanted Extra attention is needed breaking minor and removed bug patch labels Oct 9, 2022
@m-mohr m-mohr linked a pull request Oct 9, 2022 that will close this issue
@m-mohr m-mohr closed this as completed Jan 31, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
breaking help wanted Extra attention is needed minor
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant