Improve listing/searcing performance

[bpatrik]

Opening directory with a lot (2000+) of photos or running a big search query takes too long.

The issue is two fold:

1. server side processing
   2, data download time.

It turns out that 2) is more dominant that 1) if one lists a lot of photos.

Searching for . (i.e listing my whole gallery ~60k items), downloaded a 33.3MB json in 1.5min, which is 90x longer than I would prefer it.

Response header does not show gzipping, but I run into this before and I think the OS unzips and removes it:

I see a few solutions to this:

1. Save on metadata.
   • I can save some stuff here and there, like no need to send the id down, but that would be just packet change. I want to reduce the size by 50-80%.
2. Do some json tricks
   • like renaming name to n in the json object to save on the string lengths. I do not think that would do any help. This is what gzip is for.
3. trying some other data format. I was looking at protobuf. Looks promising, but not sure if it worth the investment,

Blocked by:

• Improve SQL performance #299

[bpatrik]

Did some analysis.

I found that doing the following modification on the sent data would save this much:

action	data size	additive saving	additivesaving [%]
Original data	33.55MB	N/A	N/A
Remove DB ids	32.19MB	1.36MB	4.060%
remove not used orientation	31.17MB	2.38MB	7.100%
reduce float precision (eg.: gps coordinates do not need 10 dec. precision)	30.15MB	3.40MB	10.143%
replace readyThumnails field with a bitmap and send missingThumbnail rather	28.77MB	4.78MB	14.253%
Use list instead of object for face region bounding box	27.96MB	5.60MB	16.681%
Mapify directory: Do not send directories with each photo, rather group them	23.77MB	9.79MB	29.168%
Mapify Location: Create a Location map and use it as enum to send location info like cities, countries, etc.	23.33MB	10.22MB	30.472%
Mapify cameradata: Same as with location, but with camera model name, lens, etc..	21.19MB	12.37MB	36.858%
MappifyKeywords: same as above but with the keywords	20.66MB	12.90MB	38.439%
Estimation of using protobuf	12.93MB	20.62MB	61.456%

[bpatrik]

Latest commits did a significant improvement on searching:
Data size got reduced to 16.5 MB, witch is a 2x improvement form the previous 33MB size.

I was running the 'test' both times on my full gallery that contains 65k photos.

[bpatrik]

With the latest changes, the size got event smaller to 11.9MB (from the original 33MB) for 66K photos (my gallery got a bit bigger).

Thumbnail decoration also got faster as I replaced my hard drive with and SSD. (So that improvement is irrelevant)

[bpatrik]

So It turns out that after several years of engineering I still cant read a manuals and my nginx compression was not enabled for json and lot other MIME types. 🤦‍♂️ (My only excuse that I'm not actually a web developer by profession)

Enabling compression saved a lot of json download time:

As it compressed the previous 11MB download to1.5MB.

Not to mention it also compressed down the APP from MBs to less then 500KB significantly improving the first loadtime.

I'll update the sample nginx config with the new config.