Cache loaded data to avoid re-loading when cell re-runs?

[BioTurboNick]

I'm trying to determine if I can implement a cache of previously loaded data so that re-running the cell will look it up in there if it was previously loaded. The idea being that additional new data from a source folder will be loaded, but not have to reload everything each time.

The cached variable would only be modified from within the cell it is defined in to preserve reactivity as observed by every other cell.

I was trying something like this:

begin
	if !isdefined(@__MODULE__, :data_cache)
		data_cache = Dict{String, Any}()
	end
	data = map(paths) do p
		if haskey(data_cache, p)
			data_cache[p]
		else
			data_cache[p] = load(p)
		end
	end
end

But it looks like @__MODULE__ changes on each execution, and the variable is always unassigned when the cell is run.

Is there a better, or any, way to approach this?

[BioTurboNick]

This seems to work:

begin
	if !isdefined(Main, :data_cache)
		setglobal!(Main, :data_cache, Dict{String, Any}())
	end
	data = map(paths) do p
		if haskey(Main.data_cache, p)
			Main.data_cache[p]
		else
			Main.data_cache[p] = load(p)
		end
	end
end

[Pangoraw]

An alternative solution is leveraging existing Julia packages such as Memoize.jl by doing @memoize function load(...) ... end.

[BioTurboNick]

Oh thanks, I'll check that one out.