How can we mark partitions as "materialized" in dagster if the data already exists in the DWH?

[cyberjar09]

Is there a way to tell Dagster that the first thousand partitions of a time-partitioned asset are already completed ? If I copy data manually into a table?

I'd like to mark these assets and partitions as materialized without actually running the code that overwrites their contents.

Context:

• our self hosted dagster instance is undergoing a major overhaul
  • we are migrating our dagster webserver+daemon deployment to another kubernetes cluster
  • this new daemon+webserver will also be connected to multiple code locations which have clashing names (dagster assets must be uniquely named across code locations)
    • hence we need to also rename all assets in the code locations to make them unique (we are adding a region prefix)

---

Problem Statement:

• given the above, we will lose all our dagster run and materialization history
  • asset rename
  • moving dagster to actually consolidate multiple server instances deployed across k8s clusters to 1 instance that just has multiple code locations instead
• since the historical data already exists in the DWH, we wanted the ability to just mark those partitions as "materialized" rather than re-run all the backfills just to get the UI to reflect that the data exists
• one idea was to just mark the date range and do a no_op in dagster code but was wondering if there are alternatives to consider?

[sryza]

You can report assets and asset partitions as materialized in Dagster without actually running the computation that materializes. This is possible from the UI and using Dagster's Python APIs.

From the UI

If you go to the page for the asset, you can click the dropdown next to the "Materialize" button and click "Report materialization events".

Then you'll get a dialog that allows you to choose a partition range:

Using the Python APIs

First, get your hands on a DagsterInstance: #7427.

Then you can use DagsterInstance.report_runless_asset_event:

from dagster import AssetMaterialization, AssetKey

for partition_key in partition_keys:
    asset_materialization = AssetMaterialization(AssetKey("my_asset_key"), partition=partition_key)
    instance.report_runless_asset_event(asset_materialization)

[weberdavid]

@sryza Thanks. Two questions:

• Would this work with DynamicPartitionDefinitions?
• Is there a way to do it programatically in Python? Let's say I could write a function that gets all the partition-keys out of my database, and based on that I would want to fill those keys as already materialized partitions in Dagster.

[sryza]

• This should work with DynamicPartitionDefinitions
• Yes. I updated the answer above to include it.

We also have an issue to track adding CLI support: #17202

[JasperHG90]

Hi @sryza . Stumbled upon this github issue. I'm on Dagsterr 1.6.5. and was looking for this functionality but it's not available. Do you know if this has been moved?

[sryza]

Are you able to send a screenshot. I just checked on the asset details page and saw it there.

[JasperHG90]

I found it, thank you!

[cyberjar09]

thanks a lot for the detailed response @sryza
wanted to confirm that this feature is only available 1.5.3 onwards yea?

[sryza]

That's correct

[dsgibbons]

Is there a way to do the opposite? That is, report that a previously materialized partition is now failed or missing? Both via the UI and the Python API?

[sryza]

Not currently. Here's the relevant issue: #14749.

[varunp2k]

[varunp2k]

@sryza
can you please help?