Can i caching in "dataset" ?

[AnTapTanhCode]

Hi all, when i edit dataset i see the setting to edit "cache timeout"

I have implemented cache on redis but see a part related to dataset shown on redis. So i want to ask which part is the timeout for and is there any setting to cache the dataset again? Here are my settings about cache:

 ACHE_CONFIG = {
    "CACHE_TYPE": "RedisCache",
    "CACHE_DEFAULT_TIMEOUT": 86400,
    "CACHE_KEY_PREFIX": "superset_",
    'CACHE_REDIS_URL': 'redis://redis:6379/1'
}

DATA_CACHE_CONFIG = {
    'CACHE_TYPE': 'RedisCache',
    'CACHE_DEFAULT_TIMEOUT': 86400,  
    'CACHE_KEY_PREFIX': 'data_',
    'CACHE_REDIS_URL': 'redis://redis:6379/2'
}

FILTER_STATE_CACHE_CONFIG = {
    'CACHE_TYPE': 'RedisCache',
    'CACHE_DEFAULT_TIMEOUT': 120,
    'CACHE_KEY_PREFIX': 'superset_filter_',
    'CACHE_REDIS_URL': 'redis://redis:6379/4'
}

EXPLORE_FORM_DATA_CACHE_CONFIG = {
    'CACHE_TYPE': 'RedisCache',
    'CACHE_DEFAULT_TIMEOUT': 120,
    'CACHE_KEY_PREFIX': 'superset_explore_',
    'CACHE_REDIS_URL': 'redis://redis:6379/5'
}

RESULTS_BACKEND = RedisCache(
    host= '172.18.0.3',
    port=6379,
    db=3,
    key_prefix = 'query_',
    default_timeout=120,
    )

class CeleryConfig:
    broker_url = 'redis://redis:6379/0'
    imports = (
        "superset.sql_lab",
        "superset.tasks.scheduler",
        "superset.tasks.thumbnails",
        "superset.tasks.cache",
    )
    result_backend = 'redis://redis:6379/0'
    worker_prefetch_multiplier = 1
    task_acks_late = False
    }

CELERY_CONFIG = CeleryConfig

CELERYBEAT_SCHEDULE = {
    'warm_up_cache_daily_6am': {
        'task': 'superset.tasks.warm_up_cache',
        'schedule': crontab(minute=00, hour=6'),
    },
}

[dosubot[bot]]

The "cache timeout" setting in the dataset configuration determines how long the cached data remains valid before it needs to be refreshed. If you specify a "cache timeout" in the dataset's form data, it will use that value. Otherwise, it will fall back to the datasource's cache timeout, then the database's cache timeout, and finally the default cache timeout specified in your configuration ^[1].

Regarding re-caching a dataset, Apache Superset provides a mechanism to invalidate and refresh the cache. You can use the cache invalidation API to invalidate specific cache keys associated with a dataset, which will force the application to fetch fresh data and cache it again. This is typically done by sending a request with the datasource_uids that need to be invalidated ^[2].

Additionally, the warm_up_cache task in your Celery configuration is designed to pre-populate the cache for datasets by executing the DatasetWarmUpCacheCommand. This task can be scheduled to run at specific times, as you have configured it to run daily at 6 AM ^[3].

To continue talking to Dosu, mention @dosu.

[dosubot[bot]]

When you edit a chart in Apache Superset, such as changing it from 2 columns to 3 columns, a new cache entry is created because the cache key is generated based on the chart's configuration. Any change in the chart's structure or data will result in a different cache key, leading to a new cache entry ^[1].

To delete the old cache for a chart, you can use the /invalidate endpoint of the CacheRestApi. This involves sending a POST request with the relevant datasource information to invalidate the associated cache records. This process will remove the old cache entries from Redis ^[2].

To continue talking to Dosu, mention @dosu.

[AnTapTanhCode]

do you Step-by-step presentation for it? @dosu

[dosubot[bot]]

To delete old cache entries for a chart in Apache Superset when the chart configuration changes, follow these steps using the invalidate method from the CacheRestApi class:

1. Prepare the Request: Create a JSON request body that includes a list of datasources. This can be a list of datasource UUIDs or tuples of database and datasource names.

2. Send a POST Request: Use the /invalidate endpoint to send a POST request with the prepared JSON body. This endpoint is exposed by the CacheRestApi class.

3. Process the Request: The invalidate method will:

   • Load the request data using CacheInvalidationRequestSchema.
   • Retrieve the datasource objects using the provided names or UUIDs.
   • Query the CacheKey table to find cache keys associated with the datasources.
   • Delete the cache entries using cache_manager.cache.delete_many.
   • Remove the corresponding records from the database.

4. Handle Responses: The method will return a 201 response if the cache was successfully invalidated. If there are errors in the request or during processing, it will return a 400 or 500 response, respectively.

Here is a simplified example of how you might structure the request in Python:

import requests

# Define the URL for the invalidate endpoint
url = "http://<superset-host>/api/v1/cachekey/invalidate"

# Prepare the JSON payload with the datasources
payload = {
    "datasources": [
        {
            "datasource_name": "example_datasource",
            "database_name": "example_database",
            "schema": "public",
            "catalog": "default"
        }
    ]
}

# Send the POST request
response = requests.post(url, json=payload)

# Check the response status
if response.status_code == 201:
    print("Cache successfully invalidated.")
else:
    print(f"Failed to invalidate cache: {response.status_code} - {response.text}")

Replace <superset-host> with the actual host of your Superset instance ^[1].

To continue talking to Dosu, mention @dosu.

[AnTapTanhCode]

do you config it to auto delete old cache when that have new cache? @dosu

[dosubot[bot]]

Apache Superset does not have a built-in configuration option to automatically delete old cache entries when new ones are created. Cache invalidation typically needs to be handled manually or through custom logic, such as using the cache invalidation API to remove specific cache entries when necessary ^[1].

To continue talking to Dosu, mention @dosu.