Skip to content

Commit

Permalink
Version 546
Browse files Browse the repository at this point in the history 
closes #1448
  • Loading branch information
@hydrusnetwork
hydrusnetwork committed Oct 4, 2023
1 parent 59b01d7 commit 343e12a
Show file tree
Hide file tree
Showing 52 changed files with 2,084 additions and 1,793 deletions.
63 changes: 29 additions & 34 deletions docs/changelog.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,35 @@ title: Changelog
!!! note
This is the new changelog, only the most recent builds. For all versions, see the [old changelog](old_changelog.html).

## [Version 546](https://github.com/hydrusnetwork/hydrus/releases/tag/v546)

### misc

* fixed the recent messed up colours in PSD thumbnail generation. I enthusiastically 'fixed' a problem with greyscale PSD thumbs at the last minute last week and accidentally swapped the RGB colour channels on coloured ones. I changed the badly named method that caused this mixup, and all existing PSD thumbs will be regenerated (issue #1448)
* fixed up some borked button-enabling and status-displaying logic in the file history chart. the cancel process should work properly on repeat now
* made two logical fixes to the archive count in the new file history chart when you have a specific search--archive times for files you deleted are now included properly, and files that are not eligible for archiving are discluded from the initial count. this _should_ make the inbox and archive lines, which were often way too high during specific searches, a little better behaved. let me know what you see!
* added a checkbox to _options->thumbnails_ to turn off the new blurhash thumbnail fallback
* 'this has exif data, the other does not' statements are now calculated from cached knowledge--loading pairs in the duplicate filter should be faster now
* some larger image files with clever metadata should import just a little faster now
* if the process isn't explicitly frozen into an executable or a macOS App, it is now considered 'running from source'. various unusual 'running from source' modes (e.g. booting from various scripts that mess with argv) should now be recognised better

### boring code cleanup

* moved 'recent tags' code to a new client db module
* moved ratings code to a new client db module
* moved some db integrity checking code to the db maintenance module
* moved the orphan table checking code to the db maintenance module
* fixed the orphan table checking code, which was under-detecting orphan tables
* moved some final references to sibling/parent tables from main db method to sibling and parent modules
* moved most of the image metadata functions (exif, icc profile, human-readable, subsampling, quantization quality estimate) to a new `HydrusImageMetadata` file
* moved the new blurhash methods to a new `HydrusBlurhash` file
* moved various normalisation routines to a new `HydrusImageNormalisation` file
* moved various channel scanning and adjusting code to a new `HydrusImageColours` file
* moved the hydrus image files to the new 'hydrus.core.images' module
* cleaned up some image loading code
* deleted ancient and no-longer-used client db code regarding imageboard definitions, status texts, and more
* removed the ancient `OPENCV_OK` fallback code, which was only used, superfluously, in a couple of final places. OpenCV is not optional to run hydrus, server or client

## [Version 545](https://github.com/hydrusnetwork/hydrus/releases/tag/v545)

### blurhash
Expand Down Expand Up @@ -382,37 +411,3 @@ title: Changelog
* silenced the long time logspam that oftens happens when generating flash thumbnails
* fixed a stupid typo error in the routine that schedules downloading files from file repositories
* `nose`, `six`, and `zope` are no longer in any of the requirements.txts. I think these were needed a million years ago as PyInstaller hacks, but the situation is much better these days

## [Version 536](https://github.com/hydrusnetwork/hydrus/releases/tag/v536)

### more new filetypes

* thanks to a user, we have XCF and gzip filetype support!
* I rejiggered the new SVG support so there is a firmer server/client split. the new tech needs Qt, which broke the headless Docker server last week at the last minute--now the server has some sensible stubs that safely revert to the default svg thumb and give unknown resolution, and the client patches in full support dynamically
* the new SVG code now supports the 'scale to fill' thumbnail option

### misc

* I fixed the issue that was causing tags to stay in the tag autocomplete lookup despite going to 0 count. it should not happen for new cases, and **on update, a database routine will run to remove all your existing orphans. if you have ever synced with the PTR, it will take several minutes to run!**
* sending the command to set a file as the best in its duplicate group now presents a yes/no dialog to confirm
* hitting the shortcut for 'set the focused file as better than the other(s)' when you only have one file now asks if you just want to set that file as the best of its group
* fixed an erroneous 'cannot show the best quality file of this file's group here' label in the file relationships menu--a count was off
* fixed the 'set up a hydrus.desktop file' setup script to point to the new hydrus_client.sh startup script name
* thanks to a user, a situation where certain unhandled URLs that deliver JSON were parsing as mpegs by ffmpeg and causing a weird loop is now caught and stopped. more investigation is needed to fix it properly

### boring stuff

* when a problem or file maintenance job causes a new file maintenance job to be queued (e.g. if the client in a metadata scan discovers the resolution of a file was not as expected, let's say it now recognises EXIF rotation, and starts a secondary thumbnail regen job), it now wakes the file maintenance manager immediately, which should help clear out and update for these jobs quickly when you are looking at the problem thumbnails
* if you have an image type set to show as an 'open externally' button in the media viewer, then it is now no longer prefetched in the rendering system!
* I added a very simple .editorconfig file for the project. since we have a variety of weird files in the directory tree, I've made it cautious and python-specific to start with. we'll expand as needed
* I moved the similar files search tree and maintenance tracker from client.caches.db to client.db. while the former table is regeneratable, it isn't a cache or precomputation store, _per se_, so I finally agreed to move it to the main db. if you have a giganto database, it may take an extra minute to update
* added a 'requirements_server.txt' to the advanced requirements.txts directory, just for future reference, and trimmed the Server Dockerfile down to reflect it

### client api

* thanks to a user, fixed a really stupid typo in the Client API when sending the 'file_id' parameter to set the file
* wrote unit tests for file_id and file_ids parameters to stop this sort mistake in future
* if you attempt to delete a file over the Client API when one of the given files is delete-locked (this is an advanced option that stops deletion of any archived file), the request now returns a 409 Conflict response, saying which hashes were bad, and does not delete anything
* wrote a unit test to catch the new delete lock test
* deleted the old-and-deprecated-in-one-week 'pair_rows' parameter-handling code in the set_file_relationships command
* the client api version is now 49
28 changes: 28 additions & 0 deletions docs/old_changelog.html
View file
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,34 @@
<div class="content">
<h1 id="changelog"><a href="#changelog">changelog</a></h1>
<ul>
<li>
<h2 id="version_546"><a href="#version_546">version 546</a></h2>
<ul>
<li><h3>misc</h3></li>
<li>fixed the recent messed up colours in PSD thumbnail generation. I enthusiastically 'fixed' a problem with greyscale PSD thumbs at the last minute last week and accidentally swapped the RGB colour channels on coloured ones. I changed the badly named method that caused this mixup, and all existing PSD thumbs will be regenerated (issue #1448)</li>
<li>fixed up some borked button-enabling and status-displaying logic in the file history chart. the cancel process should work properly on repeat now</li>
<li>made two logical fixes to the archive count in the new file history chart when you have a specific search--archive times for files you deleted are now included properly, and files that are not eligible for archiving are discluded from the initial count. this _should_ make the inbox and archive lines, which were often way too high during specific searches, a little better behaved. let me know what you see!</li>
<li>added a checkbox to _options->thumbnails_ to turn off the new blurhash thumbnail fallback</li>
<li>'this has exif data, the other does not' statements are now calculated from cached knowledge--loading pairs in the duplicate filter should be faster now</li>
<li>some larger image files with clever metadata should import just a little faster now</li>
<li>if the process isn't explicitly frozen into an executable or a macOS App, it is now considered 'running from source'. various unusual 'running from source' modes (e.g. booting from various scripts that mess with argv) should now be recognised better</li>
<li><h3>boring code cleanup</h3></li>
<li>moved 'recent tags' code to a new client db module</li>
<li>moved ratings code to a new client db module</li>
<li>moved some db integrity checking code to the db maintenance module</li>
<li>moved the orphan table checking code to the db maintenance module</li>
<li>fixed the orphan table checking code, which was under-detecting orphan tables</li>
<li>moved some final references to sibling/parent tables from main db method to sibling and parent modules</li>
<li>moved most of the image metadata functions (exif, icc profile, human-readable, subsampling, quantization quality estimate) to a new `HydrusImageMetadata` file</li>
<li>moved the new blurhash methods to a new `HydrusBlurhash` file</li>
<li>moved various normalisation routines to a new `HydrusImageNormalisation` file</li>
<li>moved various channel scanning and adjusting code to a new `HydrusImageColours` file</li>
<li>moved the hydrus image files to the new 'hydrus.core.images' module</li>
<li>cleaned up some image loading code</li>
<li>deleted ancient and no-longer-used client db code regarding imageboard definitions, status texts, and more</li>
<li>removed the ancient `OPENCV_OK` fallback code, which was only used, superfluously, in a couple of final places. OpenCV is not optional to run hydrus, server or client</li>
</ul>
</li>
<li>
<h2 id="version_545"><a href="#version_545">version 545</a></h2>
<ul>
Expand Down
42 changes: 33 additions & 9 deletions hydrus/client/ClientDuplicates.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -7,10 +7,12 @@
from hydrus.core import HydrusData
from hydrus.core import HydrusExceptions
from hydrus.core import HydrusGlobals as HG
from hydrus.core import HydrusImageHandling
from hydrus.core import HydrusSerialisable
from hydrus.core import HydrusTags
from hydrus.core import HydrusTime
from hydrus.core.images import HydrusImageHandling
from hydrus.core.images import HydrusImageMetadata
from hydrus.core.images import HydrusImageOpening

from hydrus.client import ClientConstants as CC
from hydrus.client import ClientThreading
Expand Down Expand Up @@ -425,14 +427,36 @@ def GetDuplicateComparisonStatements( shown_media, comparison_media ):

path = HG.client_controller.client_files_manager.GetFilePath( s_hash, s_mime )

hashes_to_jpeg_quality[ s_hash ] = HydrusImageHandling.GetJPEGQuantizationQualityEstimate( path )
try:

raw_pil_image = HydrusImageOpening.RawOpenPILImage( path )

result = HydrusImageMetadata.GetJPEGQuantizationQualityEstimate( raw_pil_image )

except:

result = ( 'unknown', None )


hashes_to_jpeg_quality[ s_hash ] = result


if c_hash not in hashes_to_jpeg_quality:

path = HG.client_controller.client_files_manager.GetFilePath( c_hash, c_mime )

hashes_to_jpeg_quality[ c_hash ] = HydrusImageHandling.GetJPEGQuantizationQualityEstimate( path )
try:

raw_pil_image = HydrusImageOpening.RawOpenPILImage( path )

result = HydrusImageMetadata.GetJPEGQuantizationQualityEstimate( raw_pil_image )

except:

result = ( 'unknown', None )


hashes_to_jpeg_quality[ c_hash ] = result


( s_label, s_jpeg_quality ) = hashes_to_jpeg_quality[ s_hash ]
Expand Down Expand Up @@ -489,9 +513,9 @@ def has_exif( m ):

path = HG.client_controller.client_files_manager.GetFilePath( hash, mime )

pil_image = HydrusImageHandling.RawOpenPILImage( path )
raw_pil_image = HydrusImageOpening.RawOpenPILImage( path )

exif_dict = HydrusImageHandling.GetEXIFDict( pil_image )
exif_dict = HydrusImageMetadata.GetEXIFDict( raw_pil_image )

if exif_dict is None:

Expand All @@ -506,8 +530,8 @@ def has_exif( m ):



s_has_exif = has_exif( shown_media )
c_has_exif = has_exif( comparison_media )
s_has_exif = shown_media.GetFileInfoManager().has_exif
c_has_exif = comparison_media.GetFileInfoManager().has_exif

if s_has_exif ^ c_has_exif:

Expand All @@ -523,8 +547,8 @@ def has_exif( m ):
statements_and_scores[ 'exif_data' ] = ( exif_statement, 0 )


s_has_human_readable_embedded_metadata = shown_media.GetMediaResult().GetFileInfoManager().has_human_readable_embedded_metadata
c_has_human_readable_embedded_metadata = comparison_media.GetMediaResult().GetFileInfoManager().has_human_readable_embedded_metadata
s_has_human_readable_embedded_metadata = shown_media.GetFileInfoManager().has_human_readable_embedded_metadata
c_has_human_readable_embedded_metadata = comparison_media.GetFileInfoManager().has_human_readable_embedded_metadata

if s_has_human_readable_embedded_metadata ^ c_has_human_readable_embedded_metadata:

Expand Down
54 changes: 38 additions & 16 deletions hydrus/client/ClientFiles.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -10,12 +10,15 @@
from hydrus.core import HydrusExceptions
from hydrus.core import HydrusFileHandling
from hydrus.core import HydrusGlobals as HG
from hydrus.core import HydrusImageHandling
from hydrus.core import HydrusPSDHandling
from hydrus.core import HydrusLists
from hydrus.core import HydrusPaths
from hydrus.core import HydrusThreading
from hydrus.core import HydrusTime
from hydrus.core.images import HydrusBlurhash
from hydrus.core.images import HydrusImageHandling
from hydrus.core.images import HydrusImageMetadata
from hydrus.core.images import HydrusImageOpening
from hydrus.core.networking import HydrusNetworking

from hydrus.client import ClientConstants as CC
Expand Down Expand Up @@ -1740,7 +1743,7 @@ def RegenerateThumbnailIfWrongSize( self, media ):
thumbnail_scale_type = self._controller.new_options.GetInteger( 'thumbnail_scale_type' )
thumbnail_dpr_percent = HG.client_controller.new_options.GetInteger( 'thumbnail_dpr_percent' )

( clip_rect, ( expected_width, expected_height ) ) = HydrusImageHandling.GetThumbnailResolutionAndClipRegion( ( media_width, media_height ), bounding_dimensions, thumbnail_scale_type, thumbnail_dpr_percent )
( clip_rect, ( expected_width, expected_height ) ) = HydrusImageHandling.GetThumbnailResolutionAndClipRegion( (media_width, media_height), bounding_dimensions, thumbnail_scale_type, thumbnail_dpr_percent )

if current_width != expected_width or current_height != expected_height:

Expand Down Expand Up @@ -1812,7 +1815,16 @@ def HasHumanReadableEmbeddedMetadata( path, mime ):

else:

has_human_readable_embedded_metadata = HydrusImageHandling.HasHumanReadableEmbeddedMetadata( path )
try:

pil_image = HydrusImageOpening.RawOpenPILImage( path )

except:

return False


has_human_readable_embedded_metadata = HydrusImageMetadata.HasHumanReadableEmbeddedMetadata( pil_image )


return has_human_readable_embedded_metadata
Expand Down Expand Up @@ -2213,7 +2225,16 @@ def _HasEXIF( self, media_result ):

path = self._controller.client_files_manager.GetFilePath( hash, mime )

has_exif = HydrusImageHandling.HasEXIF( path )
try:

raw_pil_image = HydrusImageOpening.RawOpenPILImage( path )

has_exif = HydrusImageMetadata.HasEXIF( path )

except:

has_exif = False


additional_data = has_exif

Expand Down Expand Up @@ -2259,34 +2280,35 @@ def _HasICCProfile( self, media_result ):
if mime not in HC.FILES_THAT_CAN_HAVE_ICC_PROFILE:

return False


try:

path = self._controller.client_files_manager.GetFilePath( hash, mime )

if mime == HC.APPLICATION_PSD:

try:

has_icc_profile = HydrusPSDHandling.PSDHasICCProfile(path)

has_icc_profile = HydrusPSDHandling.PSDHasICCProfile( path )
except:

return None

else:

try:

pil_image = HydrusImageHandling.RawOpenPILImage( path )
raw_pil_image = HydrusImageOpening.RawOpenPILImage( path )

except:

return None


has_icc_profile = HydrusImageMetadata.HasICCProfile( raw_pil_image )


has_icc_profile = HydrusImageHandling.HasICCProfile( pil_image )


additional_data = has_icc_profile

Expand Down Expand Up @@ -2494,7 +2516,7 @@ def _RegenBlurhash( self, media ):

numpy_image = ClientImageHandling.GenerateNumPyImage( thumbnail_path, thumbnail_mime )

return HydrusImageHandling.GetBlurhashFromNumPy( numpy_image )
return HydrusBlurhash.GetBlurhashFromNumPy( numpy_image )

except:

Expand Down
3 changes: 1 addition & 2 deletions hydrus/client/ClientImageHandling.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -7,9 +7,8 @@

from hydrus.client import ClientConstants as CC
from hydrus.core import HydrusData
from hydrus.core import HydrusImageHandling
from hydrus.core import HydrusGlobals as HG
from hydrus.core import HydrusTime
from hydrus.core.images import HydrusImageHandling

cv_interpolation_enum_lookup = {}

Expand Down
Loading

0 comments on commit 343e12a

Please sign in to comment.