Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Is it possible to cache with a custom key? #329

Open
egeres opened this issue Aug 15, 2024 · 2 comments
Open

Is it possible to cache with a custom key? #329

egeres opened this issue Aug 15, 2024 · 2 comments

Comments

@egeres
Copy link

egeres commented Aug 15, 2024

I'm looking for a way to cache the output of processing a file, but I want to use the hash of the file, not the path. This code doesn't work, but it would be more or less what I would need, is it possible to do anything similar with this library?

# More code before this...
cache = FanoutCache(cache_dir)

def compute_file_hash(file_path: Path) -> str:
    hash_func = hashlib.sha256()
    with file_path.open("rb") as f:
        while chunk := f.read(8192):
            hash_func.update(chunk)
    return hash_func.hexdigest()

def custom_key_builder(func, *args, **kwargs):
    file_path = args[0]
    file_hash = compute_file_hash(file_path)
    return file_hash

@cache.memoize(key=custom_key_builder)
def process_file(file_path: Path):
    print("Processing!!")
@egeres
Copy link
Author

egeres commented Aug 15, 2024

I guess I could just do:

c = FanoutCache(cache_dir)

def compute_file_hash(file_path: Path) -> str:
    hash_func = hashlib.sha256()
    with file_path.open("rb") as f:
        while chunk := f.read(8192):
            hash_func.update(chunk)
    return hash_func.hexdigest()

def cache_mine(method):

    def wrapper(*args, **kwargs):
        file_path = args[0]
        assert isinstance(file_path, Path), f"Expected Path, got {type(file_path)}"
        file_hash = compute_file_hash(file_path)
        with c as reference:
            if file_hash in reference:
                return reference[file_hash]
            out = method(*args, **kwargs)
            if out is not None:
                reference.set(file_hash, out)
        return out

    return wrapper

@cache_mine
def process_file(file_path: Path):
    print("Processing!!")
    return -1

@phi-friday
Copy link

Override the __cache_key__ attribute of the function.

see more:

def decorator(func):
"""Decorator created by memoize() for callable `func`."""
base = (full_name(func),) if name is None else (name,)
@ft.wraps(func)
def wrapper(*args, **kwargs):
"""Wrapper for callable to cache arguments and return values."""
key = wrapper.__cache_key__(*args, **kwargs)
result = self.get(key, default=ENOVAL, retry=True)
if result is ENOVAL:
result = func(*args, **kwargs)
if expire is None or expire > 0:
self.set(key, result, expire, tag=tag, retry=True)
return result
def __cache_key__(*args, **kwargs):
"""Make key for cache given function arguments."""
return args_to_key(base, args, kwargs, typed, ignore)
wrapper.__cache_key__ = __cache_key__
return wrapper
return decorator

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants