Version 562

closes #1520, closes #1518

db/help my media files are broke.txt

@@ -15,7 +15,7 @@ FYI: In hydrus's file storage, there are two sets of 256 sub folders. One is 'fx
 
 Once you are booted, your client may throw up lots of missing thumbnail/file errors. Do not worry so much about these for now, but it would be best to close your larger pages and pause the import pages for now. Don't pause network traffic, since we'll be using that in a minute.
 
-If you needed to remap your file locations during pre-boot repair, then hit _database->migrate database_. It may still have your old damaged location set as the 'ideal', and the remapped correct locations set as just a temporary location it currently wants to move away from. Correct this by giving the new locations some weight and removing the old locations. Once you are done, you should only see good locations listed, no desire to move anything, with 'current' and 'ideal' usage at the same percentage.
+If you needed to remap your file locations during pre-boot repair, then hit _database->move media files_. It may still have your old damaged location set as the 'ideal', and the remapped correct locations set as just a temporary location it currently wants to move away from. Correct this by giving the new locations some weight and removing the old locations. Once you are done, you should only see good locations listed, no desire to move anything, with 'current' and 'ideal' usage at the same percentage.
 
 Next, we have two classes of problem: files in the database but not in storage, and files in storage but not in the database.
 

docs/changelog.md

@@ -7,6 +7,41 @@ title: Changelog
 !!! note
     This is the new changelog, only the most recent builds. For all versions, see the [old changelog](old_changelog.html).
 
+## [Version 562](https://github.com/hydrusnetwork/hydrus/releases/tag/v562)
+
+### misc
+
+* page tab drag and drops will now not start unless the click has lasted more than 100ms
+* same for thumbnail drag and drop--it perviously did a 20 pixel deadzone, but time checks detect accidental/spastic clicks better and stops false negatives when you start dragging on certain edges
+* added a 'BUGFIX: disable page tab drag and drop' setting to _options->gui pages_. while adding this, I may have accidentally fixed the issue I wanted to investigate (rare hangs on page DnD)
+* the manage tags dialog now shows the current count of tags for each page tab, and, if there are outstanding changes, shows an asterisk
+* the `migrate database` dialog is renamed `move media files`
+
+### fixes
+
+* fixed the basic copy/paste in the single 'edit datetime' panel, wich was often raising a dumb error. this thing also now exports millisecond data (issue #1520)
+* I am pretty sure I fixed the column-resizing problem in the very new PySide6 (Qt) 6.6.1, which it seems AUR users were recently updated to in an automatic OS update. all columns were setting to 100px width on initialisation. I think it is now safe to try out 6.6.1. I am still not sure why it was doing this, but some extra safeguards seem to have fixed it and also not broken things for <=6.6.0, so let me know what you run into! if you were affected by this, recall that you can right-click on any multi-column list header and say 'reset widths' to get something sensible back here
+* when exporting files, the max size is now clipped another 84 characters (64 + 20 more, which usually ends up about 150 characters max for the output filename), in order to give padding for longer sidecar suffixes and also avoid going right to the filesystem limit, which broadly isn't sensible
+* I think I fixed an issue where the mouse could stay hidden, perhaps, just on Wayland, after closing the media viewer with your keyboard (issue #1518)
+* fixed inc/dec ratings in the media viewer not updating their tooltips on new media correctly
+* if you hit 'open this location' on the export files window and the location does not exist, you now get a nice messagebox rather than a semi-silent error
+
+### analyze
+
+* background: some databases that process the PTR superfast or otherwise import a lot of data to a new file domain sometimes encounter massively massively slow tag update actions (typically tag-delete when the tags involved have siblings/parents), so I want to make the critical 'ANALYZE' call more timely
+* the 'analyze' database maintenance call will be soft-called far more regularly during normal repository processing, not just on first sync
+* sped up how some pre-analyze calculation is done
+* the size limit for automatic database analyze maintenance is raised from 100k rows to 10M
+* I hope to do more work here in future, probably making a review panel like we did for vacuum
+* if your repository processing sometimes hangs your whole damn client for 10-15 minutes, hit _database->db maintenance->analyze->full_! this job may take 30-60 minutes to finish
+
+### boring code cleanup
+
+* finished the HG->CG.client_controller refactor I started last week. this was a thousand lines changed from one braindead format to another, but it will be a useful step towards untangling the hell-nest import hierarchy
+* did a scattering of the clientinterface typing, getting a feel for where I want to take this
+* deleted the old in-client server-test's 'boot' variant; this is no longer used and was always super hacky to maintain
+* I removed an old basic error raising routine that would sometimes kick in when a hash definition is missing. this routine now always fills in the missing data with garbage and does its best to recover the invalid situation automatically, with decent logging, while still informing the user that things are well busted m8. it isn't the user's job to fix this, and there is no good fix anyway, so no point halting work and giving it to the user to figure out!
+
 ## [Version 561](https://github.com/hydrusnetwork/hydrus/releases/tag/v561)
 
 ### rearranging thumbnails
@@ -447,54 +482,3 @@ title: Changelog
 * the forced-wait throttle that happens on several exception catches is reduced from 1s to 200ms
 * I made the new job status queue properly thread-safe with a lock. I forgot to do it last week, whoops!
 * fixed the build script to construct a file named .tar.zst for the Ubuntu release, not .tar.gz
-
-## [Version 552](https://github.com/hydrusnetwork/hydrus/releases/tag/v552)
-
-### misc
-
-* 'system:has audio' and 'system:embedded metadata' are now combined under a new meta-system predicate 'system:file properties'. if you can't find your yes/no predicate, try looking there!
-* menu commands will no longer have their unadjusted label as their tooltip. all tooltips are either the full status bar description or the full label if it was long enough to be elided
-* the 'open externally' panel now shows the default filetype thumbnail for formats like zip and epub
-* 'system:number of character tags > 4' now parses correct when you type it (previously it wouldn't work with a namespace), including special handling for 'unnamespaced'
-* the various 'number of x' system predicates will now parse if you type 'num x', 'number x', or 'num of x'
-* to match the other entries, the '4k' resolution swap-in label is now '2160p'
-* added a little extra info on the manage tags dialog to 'getting started with tags'
-* if you have 'confirm sending files to trash' turned off, the delete dialog will now show on physical deletes (i.e. deletes from the trash)
-* updated the derpibooru parser to pull the new AI-based 'generator' and 'prompter' namespaces (converting both to the hydrus-appropriate 'creator')
-* thanks to a user, the Linux build is now archived with zstd instead of gzip. should be about the same size but faster to decompress
-
-### fixes
-
-* fixed a stupid typo in the folder copy/move tech last week that was not allowing some move/copies to start (as always, the thing that is so simple that you don't think to test it is the very thing that blows up). sorry for the trouble!
-* cleaned up the file/folder move/copy error statements a little more
-* fixed the 'default search page tag service' dropdown in _options->search_ not saving correctly
-* fixed the 'open externally' panel having out of position thumbnails when your thumbnail supersampling is set to other than 100%
-* fixed the import and display of images in signed 16-bit format (weird TIFFs, seems like).
-* any image with an unusual channel data type beyond uint16 and int16 is going to be, as the default thing to do, normalised to unsigned 8-bit. it may blow out the colour range, but it should show something!
-* the client handles files with (0x0) resolution better. they should now always import, and it'll _attempt_ to render them to a normal full size thumb. if it works (e.g. this is some misconfigured SVG), great, and if it doesn't, we'll get a nicely sized filetype.png or hydrus.png fallback
-* files with (0x0) resolution will now never show in the preview or media viewers. previously, the preview viewer would bail out half-way through setting the media, causing it to fall into an invalid state where it still showed the previous valid media but wouldn't 'click-off' it easily, and the media viewer would generally panic to its 'no media to show' state and lose navigation functionality. now, files that are 0x0 are included in the general 'can we show this?' pre-launch sanity checks
-
-### has_transparency
-
-* the database can now remember if a file has transparency. you can search this with the new 'system:has transparency' predicate, which is under the new 'system:file properties' and will also parse if you type 'system:has/no transparency/alpha'
-* note that my version of 'has transparency' discludes files that have an all-opaque alpha channel (i.e. one that lets no light through). RGBA is insufficient--I want an alpha channel with some actual translucency somewhere!
-* although many application image project types like PSD and XCF can have transparency, the various ways we render or thumbnail them are hacky and probably lock to RGB or RGBA always, so I'm going to start simple. this week, we test transparency for all the images that support it (basically anything but jpeg), and animated gif. the animated gif tech is new and actually looks through every frame of an RGBA gif until it hits interesting alpha to catch cases where it starts opaque and fades away
-* just like we had with 'has exif' and similar, 'has transparency' knowledge will be calculated instantly for all new files, but for the files you already have, we'll have to do some slow file maintenance in the background for a while to retroactively calculate it all. you don't have to do anything; the data will just populate over time
-* the duplicate filter now shows 'has transparency, the other is opaque' statements
-* while working on this, I encountered a number of files that seemed to be false positives--apparently normal, fully opaque images of anime girls that were somehow showing up as 'has interesting alpha'. upon inspecting them closely, I discovered the border pixels had a slight fade, or one pixel out of all of them was 98% opaque, or the single bottom right pixel was completely transparent. perhaps some of these are secret artist markers, but I imagine many are just an accidental drawing tablet smudge or dodgy crop tool calculations. I'm leaving them as 'has_transparency' for now, but maybe we'll want to tune this more in future, perhaps saying you have to be at least 0.3% transparent to count. anyway, as always, while I am interested in seeing files that seem to get a false positive/negative with this new 'has transparency' test, if you have the technical know-how, please check if they actually have no alpha yourself first. once you play around with this system, let me know what sort of pseudo-'false positive' rate you are getting, and we can talk about an appropriate threshold
-
-### client api
-
-* the 'file_metadata' call now includes a 'has_transparency' boolean! remember that it will be overly `false` for a while, until the file maintenance catches up
-* forgot to mention it last week, but thanks to a user there is a new `/manage_database/get_client_options` call that fetches a heap of different client options. this exposes a mess that may change with any update, but there may be something neat you can hook into. this week we fixed a thing that was breaking this call for probably all old clients
-* the client api version is now 56
-
-### boring cleanup
-
-* renamed JobKey to JobStatus across the program
-* in prep for Client API calls to interact with the popup system, the queue of JobStatuses waiting to be displayed in the popup toaster is now encapsulated in a separate class, outside of the Qt object dangerzone
-* sped up how the popup manager system inspects and cleans the JobStatus queue in general. should have better performance when you get hundreds or thousands of messages
-* cleaned up some awkward popup manager dismiss code
-* fixed a timing issue that meant popup messages were auto-dismissed from the popup toaster up to a second after they were being 'deleted' by their parent functiions. subscription flow felt more laggy because of this
-* fixed the file info manager's duplicate call to duplicate unusual metadata like has_exif and blurhash
-* removed some old code that isn't used any more

docs/database_migration.md

@@ -46,7 +46,7 @@ If you would like to move your files and thumbnails to new locations, I generall
 ??? note "Missing Locations"
     If your folders are in the wrong locations on a client boot, a repair dialog appears, and you can manually update the client's internal understanding. This is not impossible to figure out, _and in some tricky storage situations doing this on purpose can be faster than letting the client migrate things itself_, but generally it is best and safest to do everything through the dialog.
 
-Go _database->migrate database_, giving you this dialog:
+Go _database->move media files_, giving you this dialog:
 
 ![](images/db_migration.png)
 
@@ -56,7 +56,7 @@ The buttons let you add more locations and remove old ones. The operations on th
 
 **Weight** means the relative amount of media you would like to store in that location. It only matters if you are spreading your files across multiple locations. If location A has a weight of 1 and B has a weight of 2, A will get approximately one third of your files and B will get approximately two thirds.
 
-**Max Size** means the max total size of files the client will want to store in that location. Again, it only matters if you are spreading your files across multiple locations, but it is a simple way to ensure you don't go over a particular smaller hard drive's size. One location must always be limitless. This is not precise, so give it some padding. When one location is maxed out, the remaining locations will distribute the remainder of the files according to their respective weights. _For the meantime, this will not update by itself. If you import many files, the location may go over its limit and you will have to revisit 'migrate database' to rebalance your files again. Bear with me--I will fix this soon with the background migrate._ 
+**Max Size** means the max total size of files the client will want to store in that location. Again, it only matters if you are spreading your files across multiple locations, but it is a simple way to ensure you don't go over a particular smaller hard drive's size. One location must always be limitless. This is not precise, so give it some padding. When one location is maxed out, the remaining locations will distribute the remainder of the files according to their respective weights. _For the meantime, this will not update by itself. If you import many files, the location may go over its limit and you will have to revisit 'move media files' to rebalance your files again. Bear with me--I will fix this soon with the background migrate._ 
 
 Let's set up an example move: