File metadata caching ?

[Adirelle]

This is one is a bit arguable. I see that file metadata are not cached, but reading them can possibly involve a lot of IO operations. Have you ever considered caching those metadata ?

[ronindev]

If you are talking about ffmpeg metadata, it uses the cache. See rrcache

[anacrolix]

I do believe it's stored in memory. One of the original frustrations I had with other servers was their scanning of everything at startup, so I tried to make this implementation do it on the fly as much as possible. If I did it again I'd probably cache in something like sqlite and try to prefetch where reasonable (immediate subdirs and root for example) on another goroutine. I did start this project a very long time ago (and ported it from Python) :)

If you want add cache persistance that sounds good, I'd only add that it would be helpful if a check is done that the file hasn't been modified since the caching date, and to rescan if it has.

[Adirelle]

@ronindev I was more thinking of the base metadata (like name, mediatype, children, ...).

@anacrolix It seems to me that no metadata are cached at all, even in memory, but I may have overlooked it.

Serializing these data in a persistent storage can be tedious. And in that case, there are pure Go stores (see https://github.com/avelino/awesome-go#database). This may be a bit too much.

My idea is to cache the metadata in memory, mainly to avoid performing the same scan several times. This is probably a good middle ground. However, there should be some way to limit the cache size (for low-end machine or very big content libraries).

Background: before I found DMS, I was tinkering with my own server but as I lacked all the UPNP knowledge and libraries, I started from the scanning parts, including updates using notify mechanisms, though it is probably overkill.

[anacrolix]

I don't think I got much benefit in caching filesystem stuff. Are you seeing some performance issues?

The ffmpeg info however, can be slow, here's where it's pulled from the cache: https://github.com/anacrolix/dms/blob/master/dlna/dms/dms.go#L954.

[pataquets]

I do believe it's stored in memory. One of the original frustrations I had with other servers was their scanning of everything at startup, so I tried to make this implementation do it on the fly as much as possible.

Found this issue while looking how to force dms to proactively scan media on service start. I've looked and there is no command line switch to force it. @anacrolix What would be needed to be invoked to add such a switch?

[anacrolix]

It should be possible to rejig the code slightly to walk the media directory and trigger fetching the metadata on everything found. Let me know if you need more assistance than that.