Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

disambiguate origin of operation in teraslice assets #3837

Open
godber opened this issue Nov 19, 2024 · 0 comments
Open

disambiguate origin of operation in teraslice assets #3837

godber opened this issue Nov 19, 2024 · 0 comments

Comments

@godber
Copy link
Member

godber commented Nov 19, 2024

Here's what we're thinking about for this interface now, still open for discussion:

  • in assets - assetName:version or assetHash
  • in operations - operation[@assetName[:version]] OR operation[@assetHash]
    • operation[@assetName[:version]]
    • operation[@assetHash]
    • operation
  • in apis - apiName[:namespace] OR apiName[@assetName[:version]] OR apiName[@assetHash] OR apiName[@assetHash][:namespace] OR apiName[@assetName[:version]][:namespace]

notional job

{
    "name": "Reindex Events",
    "lifecycle": "once",
    "analytics": false,
    "assets": [
       "elasticsearch", 
       "asset1:v1.4.5",
       "asset1:v2.5.6",
       "asset2"
    ],
    "apis": [
        { "_name":  "example-api" },
        { "_name":  "example-api@asset2"  },
        { "_name": "example-api@asset1:v2.5.6"  },
        { "_name":  "example-api@asset1:v2.5.6:foo2"  },
    ],
    "operations": [
        {
            "_op": "elasticsearch_reader",
            "index": "events-*",
            "type": "event",
            "size": 5000,
            "date_field_name": "created"
        },
        {
            "_op": "custom_op@asset1"
            "some": "configuration1",
            "api_name":  "example-api@asset1:v2.5.6:foo2" 
        },
        {
            "_op": "custom_op@asset2",
            "some": "configuration2"
        },
        {
            "_op": "elasticsearch_bulk",
             "index": "bigdata3",
            "type": "events",
            "size": 5000
        }
    ]
}

We need to keep in mind the two scenarios:

  • job validation
  • job execution

When the execution is create from the job we will map "_op": "custom_op@asset1:version" to "_op": "custom_op@assetHash".

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants