diff --git a/CHANGELOG.md b/CHANGELOG.md index c6e41350..975bb6e1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,78 @@ All notable changes to this project will be documented in this file. This change The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). +## 7.0.0 - 2023-06-11 + +### Added + +* added 'stats' that serves to centralise a bunch of numbers used internally that might be interesting to the user. + - see the `more stats` button in the bottom left corner. +* Github rate limit information is now fetched as part of the stats. + - but no more than once a minute. + - see: https://github.com/ogri-la/strongbox#user-content-github-api-authentication +* manually refreshing the user catalogue will now switch to the `log` pane before doing so. + - the intent is to show that *something* is happening. + - see: `Catalogue -> Refresh user catalogue` +* new (opt-in) preference to automaticaly refresh the user-catalogue every 28 days. + - the user-catalogue are addons added to strongbox using `File -> Import addon` or by 'starring' a regular addon. + - see: `Preferences -> Keep user catalogue updated`. +* added a "clear" button to the addons search that removes all search filters, including search terms. +* added new column for installed addons "starred" that will add an installed addon to the 'user-catalogue'. + - star button disabled when addon is being ignored or isn't matched against the catalogue. +* new column for installed addons "size" with the total size of the addon on disk, including any grouped addons. + - see: `View -> Columns -> size` +* `user.clj`, where the REPL will take you by default during development. + - this lets me separate some development dependencies and logic from what is released. + +### Changed + +* the main window is now always split with the bottom pane hidden by default. + - if can be dragged open or toggled open by either of the two status bar buttons. +* clicking the status bar buttons to open the bottom pane is now much quicker. +* refreshing the user-catalogue now checks imported/starred addons against the full catalogue before checking online. + - if it fails to find addon in catalogue, it will fall back to checking online like before. + - the user-catalogue was originally for imported addons without a catalogue (github, gitlab) but is now also for 'starred' addons. Refreshing it now that we have a Github catalogue is much faster. +* menu labels for the installed addons table columns have been tweaked (see `View -> Columns`) + - "installed" is now "installed version" + - "available" is now "available version" + - "version" is now "installed+available version" + - "WoW" is now "game version (WoW)" +* the "fat" column profile now uses the "installed" and "available" columns rather than the combined "version" column. +* the "fat" column profile includes the new "starred" and "size" columns. + - see: `View -> Columns -> fat` +* `jlink compress=2` changed to `jlink compress=1` during the building of the linux AppImage. + - `2` means "zip", which interferes with the final AppImage compression. + - this shaves off ~7MB from the final AppImage. +* replaced the compressed, static, "emergency" catalogue with a simple JSON string. + 1. it wasn't working at compile time like I thought. + 2. regular strings are more compressible ultimately when building an AppImage. +* bumped dependencies. + - removed `apache.commons.compressors` as no longer required. +* some dependencies used for development are no longer bundled during release. +* strongbox release data will only be downloaded once the app has finished loading. +* strongbox release data will only be downloaded once per-session. + - it would previously re-attempt to download the release data on failure endlessly. + +### Fixed + +* search tab filter buttons are now a uniform height. +* `key` column in the addon detail "raw data" widget is now wide enough for the text "supported game tracks". +* `updated` column in the installed addon tab is now wide enough for the text "12 months ago". +* possible cache stampede fetching strongbox release info. A lock is now acquired to ensure checks happen sequentially. + - it was possible for the GUI to fire off many requests to Github simultaneously, bypassing cache and overwriting each other. + +### Removed + +* Tukui support. Tukui addons are: + - no longer checked for updates. + - excluded from being imported. + - excluded from the user-catalogue. + - no longer scraped from the tukui.org API into a catalogue. + - no longer present in the "full" or "short" catalogues. + - excluded from search results. + - removed from the 'emergency' (built-in, hardcoded) catalogue (used when remote catalogues are unavailable). + - removed from lists of available addon hosts to switch an addon between. + ## 6.1.2 - 2023-05-16 * issue #402, fixed a freezing bug in the search results, introduced in 5.1.0 (2022-03-02). diff --git a/LICENCE.txt b/LICENCE.txt index d4c29436..704ccb58 100644 --- a/LICENCE.txt +++ b/LICENCE.txt @@ -688,3 +688,4 @@ Program grant you additional permission to convey the resulting work. * cljfxcss * clj-http-fake * tolitius/lasync +* clj-commons/humanize diff --git a/README.md b/README.md index 406562dd..6810104b 100644 --- a/README.md +++ b/README.md @@ -1,30 +1,42 @@ # strongbox, a World of Warcraft addon manager -`strongbox` is an **open source**, **[advertisement free](#recognition)** and **[privacy respecting](#privacy)** addon manager for World of Warcraft. +`strongbox` is an **open source**, **[advertisement free](#recognition)** and **[privacy respecting](#privacy)** addon +manager for World of Warcraft. It runs on Linux and macOS. -It supports addons hosted by ~Curseforge,~ wowinterface.com, Tukui, Github and Gitlab. +It supports addons hosted by ~Curseforge,~ wowinterface.com, ~Tukui~, Github and Gitlab. + +--- + +***Notice***: tukui.org no longer hosts addons except `elvui` and `tukui` as of **2023-06-01**. + +Unfortunately the location of these two addons has changed as well as details around their access. + +I've dropped support for tukui.org in **7.0.0** and instead mirror [tukui](https://github.com/ogri-la/tukui) and +[elvui](https://github.com/ogri-la/elvui) releases on Github. This automatically includes them in the +[Github catalogue](https://github.com/ogri-la/strongbox-catalogue/blob/master/github-catalogue.json). --- ***Notice***: Curseforge addons no longer receive updates as of version **5.0.0**, released Feb 1st, 2022. -Use the *"Source"* and *"Find similar"* actions from the addon context menu ([added **4.9.0**](https://github.com/ogri-la/strongbox/releases)) to help migrate addons away from Curseforge. +Use the *"Source"* and *"Find similar"* actions from the addon context menu +([added **4.9.0**](https://github.com/ogri-la/strongbox/releases)) to help migrate addons away from Curseforge. I also maintain a list of [other addon managers](https://ogri-la.github.io/wow-addon-managers/). --- -[![strongbox version 6.0.0](./screenshots/screenshot-6.0.0-installed-skinny-thumbnail.jpg)](./screenshots/screenshot-6.0.0-installed-skinny.png?raw=true) -[![strongbox version 6.0.0](./screenshots/screenshot-6.0.0-installed-thumbnail.jpg)](./screenshots/screenshot-6.0.0-installed.png?raw=true) -[![strongbox version 6.0.0](./screenshots/screenshot-6.0.0-installed-fat-thumbnail.jpg)](./screenshots/screenshot-6.0.0-installed-fat.png?raw=true) -[![strongbox version 6.0.0](./screenshots/screenshot-6.0.0-detail-thumbnail.jpg)](./screenshots/screenshot-6.0.0-detail.png?raw=true) +[![strongbox version 7.0.0](./screenshots/screenshot-7.0.0-installed-skinny-thumbnail.jpg)](./screenshots/screenshot-7.0.0-installed-skinny.png?raw=true) +[![strongbox version 7.0.0](./screenshots/screenshot-7.0.0-installed-thumbnail.jpg)](./screenshots/screenshot-7.0.0-installed.png?raw=true) +[![strongbox version 7.0.0](./screenshots/screenshot-7.0.0-installed-fat-thumbnail.jpg)](./screenshots/screenshot-7.0.0-installed-fat.png?raw=true) +[![strongbox version 7.0.0](./screenshots/screenshot-7.0.0-detail-thumbnail.jpg)](./screenshots/screenshot-7.0.0-detail.png?raw=true) -[![strongbox version 6.0.0](./screenshots/screenshot-6.0.0-dark-installed-skinny-thumbnail.jpg)](./screenshots/screenshot-6.0.0-dark-installed-skinny.png?raw=true) -[![strongbox version 6.0.0](./screenshots/screenshot-6.0.0-dark-installed-thumbnail.jpg)](./screenshots/screenshot-6.0.0-dark-installed.png?raw=true) -[![strongbox version 6.0.0](./screenshots/screenshot-6.0.0-dark-installed-fat-thumbnail.jpg)](./screenshots/screenshot-6.0.0-dark-installed-fat.png?raw=true) -[![strongbox version 6.0.0](./screenshots/screenshot-6.0.0-dark-detail-thumbnail.jpg)](./screenshots/screenshot-6.0.0-dark-detail.png?raw=true) +[![strongbox version 7.0.0](./screenshots/screenshot-7.0.0-dark-installed-skinny-thumbnail.jpg)](./screenshots/screenshot-7.0.0-dark-installed-skinny.png?raw=true) +[![strongbox version 7.0.0](./screenshots/screenshot-7.0.0-dark-installed-thumbnail.jpg)](./screenshots/screenshot-7.0.0-dark-installed.png?raw=true) +[![strongbox version 7.0.0](./screenshots/screenshot-7.0.0-dark-installed-fat-thumbnail.jpg)](./screenshots/screenshot-7.0.0-dark-installed-fat.png?raw=true) +[![strongbox version 7.0.0](./screenshots/screenshot-7.0.0-dark-detail-thumbnail.jpg)](./screenshots/screenshot-7.0.0-dark-detail.png?raw=true) ## Installation @@ -32,14 +44,14 @@ Arch Linux users can install `strongbox` from the [AUR](https://aur.archlinux.or For other Linux users: -1. download: [./releases/strongbox](https://github.com/ogri-la/strongbox/releases/download/6.1.2/strongbox) +1. download: [./releases/strongbox](https://github.com/ogri-la/strongbox/releases/download/7.0.0/strongbox) 2. make executable: `chmod +x strongbox` 3. run: `./strongbox` If you're on macOS or having a problem with the binary or just prefer Java `.jar` files (requires Java 11+): -1. download: [./releases/strongbox-6.1.2-standalone.jar](https://github.com/ogri-la/strongbox/releases/download/6.1.2/strongbox-6.1.2-standalone.jar) -2. run: `java -jar strongbox-6.1.2-standalone.jar` +1. download: [./releases/strongbox-7.0.0-standalone.jar](https://github.com/ogri-la/strongbox/releases/download/7.0.0/strongbox-7.0.0-standalone.jar) +2. run: `java -jar strongbox-7.0.0-standalone.jar` ## Usage @@ -68,7 +80,7 @@ Afterwards, use the `Update all` button to update all addons with new versions a * [install addons from multiple sources](#install-addons-from-multiple-sources): - ~Curseforge~ - wowinterface.com - - Tukui + - ~Tukui~ - Github (using *releases*) - Gitlab (using *releases*) * [import and export lists of addons](#import-and-export-lists-of-addons) @@ -107,9 +119,7 @@ that it's my privilege to offer this small piece back. This software interacts with the following remote hosts: -* ~Overwolf/Twitch/Curseforge [Addons API](https://addons-ecs.forgesvc.net/) and its [CDN](https://edge.forgecdn.net/)~ * [wowinterface.com](https://wowinterface.com) -* [www.tukui.org](https://www.tukui.org/api.php) * [api.github.com](https://developer.github.com/v3/repos/releases) - to download repository and release data for addons hosted on Github - to download the latest `strongbox` release information @@ -148,7 +158,8 @@ bug. *Some* of the details it contains are: ### classic and retail addon support -"Classic", "Classic (The Burning Crusade)", "Classic (Wrath of the Lich King)" and "Retail" versions of WoW are all distinct addon systems. +"Classic", "Classic (The Burning Crusade)", "Classic (Wrath of the Lich King)" and "Retail" versions of WoW are all +distinct addon systems. Some addons support all systems in a single download, some support classic as an alternate build of the same addon, some addons support classic only, some addons have been split up into multiple addons. There is a lot of variation. @@ -182,9 +193,7 @@ Click `File` from the top menu and select `Import addon` and paste the URL of th Strongbox supports searching for addons from the following addon hosts: -* ~[Curseforge](https://www.curseforge.com/wow/addons)~ * [wowinterface.com](https://wowinterface.com/addons.php) -* [Tukui](https://www.tukui.org) * [Github](https://github.com) Click the `search` tab and start typing. @@ -224,9 +233,7 @@ Click the `Update all` button next to your addon directory. Strongbox supports installing addons from the following addon hosts: -* ~[Curseforge](https://www.curseforge.com/wow/addons)~ * [wowinterface.com](https://wowinterface.com/addons.php) -* [Tukui](https://www.tukui.org) * [Github](https://www.github.com) * [Gitlab](https://gitlab.com) @@ -313,7 +320,6 @@ Right-click an addon and select `Release`. Strongbox currently supports installing previous releases for: -* ~Curseforge~ * Github * Gitlab @@ -394,6 +400,6 @@ Prior to `1.0.0`, `strongbox` was known as `wowman`. The [AUR package](https://a ## License -Copyright © 2018-2022 Torkus +Copyright © 2018-2023 Torkus Distributed under the GNU Affero General Public Licence, version 3 [with additional permissions](LICENCE.txt#L665) diff --git a/SECURITY.md b/SECURITY.md index b9b9cd37..2665c596 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -15,8 +15,9 @@ All other major versions are unsupported. | Version | Supported | | ------- | ------------------ | -| 6.x.x | :heavy_check_mark: | -| 5.x.x | :heavy_minus_sign: | +| 7.x.x | :heavy_check_mark: | +| 6.x.x | :heavy_minus_sign: | +| 5.x.x | :x: | | 4.x.x | :x: | | 3.x.x | :x: | | 2.x.x | :x: | diff --git a/TODO.md b/TODO.md index 932a26b5..7bdf9a36 100644 --- a/TODO.md +++ b/TODO.md @@ -6,25 +6,183 @@ see CHANGELOG.md for a more formal list of changes by release ## done -* fonts are screwy on some systems - - https://github.com/ogri-la/strongbox/issues/384 - - embedded an older, smaller, lighter fontawesome and switched the glyphs +* 'downloading strongbox data' shouldn't be blocking the gui from starting + - is the version check happening async too? + - done + +* size of addon on disk + - I'd like to see a column with 'size on disk' + - "1024KiB", "1MiB", "1.04MB" + - done + - I'd like to see a total size of all addons in addon dir + - "1024MiB", "1GiB" + - see :db-stats + - I'd like to see size of disk and free space + - "2TiB of 14TiB free" + - nah + - and everything mooshed together + - "1GiB of addons on /dev/foo with 2TiB of 14TiB available" + - nah + +* user catalogue, add a 'add to user-catalogue' option to make an addon always available despite selected catalogue + - done, via favouriting, but! it's not available on the installed addon pane page - done -* log pane, bug, the text 'catalogue' is being truncated in multi-line messages - - see screenshot Screenshot at 2022-09-24 08-56-35.png +* user catalogue, what is happening now that regular non-github addons can be favourited? + - do they need to have their details refreshed? + - yes, they are copies from whatever catalogue they came from. + - also, we have a github catalogue now, I bet the majority of these updates can be pulled directly from catalogues. + - import-addon will + - calls find-addon + - for github, gitlab and curseforge addons + - this involves calls to various apis and checks and things + - otherwise, looks up the addon in the catalogue + - calls expand-summary + - skips the optional dry-run installation + - calls expand-summary + - (again) + - calls add-addon to add addon to user-catalogue .. + - writes user-catalogue to disk + - forces an update of the :db by setting it to nil + + - refresh-user-catalogue + - calls refresh-user-catalogue-item on each item: + - calls find-addon + - for github, gitlab and curseforge addons + - this involves calls to various apis and checks and things + - otherwise, looks up the addon in the catalogue + - calls expand-summary + - skips the optional dry-run installation + - calls add-addon to add addon to user-catalogue .. + - writes user-catalogue to disk + + - we want to: + - check the catalogue for the addon first + - I think find-addon should be skipped as we already have the addon as an addon-summary with a url, source, source-id, etc. + - if not found in catalogue ... ? + - import it with find-addon? + - done -* dark theme, 'wow' column text is black on black for some reason +* search, a 'clear' button + - resets favourited, search input, tags, etc - done -* add a .desktop file for AUR installations +* search, buttons are slightly different sizes + - input star and dropdown are a few pixels shorter than the buttons on either side - done -## todo +* user catalogue, schedule refreshes + - ensure the user catalogue doesn't get too stale and perform an update in the background if it looks like it's very old + - update README + - perhaps a preference? + - opt-in, opt-out? + - some task on startup that examines the catalogue's age and decides to do a refresh? + - would play havoc with testing. + - disable during testing? + - done + +* gui, raw data, 'key' column too small for 'supported-game-tracks' + - done + +* gui, installed, 'updated' column too small for 'NN months ago' + - done + +* user-catalogue + - switch to log window to see progress when refresh-catalogue triggered + - only when triggered from menu + - scheduled refreshes happen in background + - done + +* display github requests remaining + - done + +* user-catalogue, don't switch to user catalogue if sub-pane is up and notice logger is showing + - done + +* 'core/state :db-stats', seems like a nice idea to put more information here + - known-hosts + - num addons per-host + - num addons favourited/user-catalogue + - num addons + - total size of addons + - ...? + - a button on the opposite of the log popup button but similar that will show some overall stats + - done +* size column should be right aligned + - done + +* remove tukui + - I'm not keeping all this logic for two addons + - stop catalogue addons from being merged into full catalogue + - done + - can the addons be mirrored/hosted on github? + - done + - follow curseforge removal pr + - done + +* update screenshots + - done + +## todo ## todo bucket (no particular order) +* search results, if there are addons from the same host (github) with the same name (tukui), disambiguate them + - 'tukui' in the search results shouldn't mean 'ogri-la/tukui' if 'tukui.org/tukui' is also available + - which it isn't, but that's not the point. + +* expand-summary, can the game track wrangling logic be made generic rather than per-host? + - this would tie in with returning *all* releases from a host + - it would insulate host logic from selected game tracks and game track strictness + - it would handle pinned logic as well + - currently handled in catalogue + +* user catalogue, refreshing may guarantee exceeding github limit. + - if we know this, add a warning? refuse? stop? + +* support NO_COLOR envvar, http://no-color.org + +* gui, bug, sorting isn't preserved between addon dir switches + - for example, sorting by 'size' in one addon dir, then switching to another will see random sorting + +* gui, 'set-icon' is taking a long time to do it's thing. + +* gui, raw data, add textual versions of this data as well + - pretty printing in a gui is one thing, but useless if it can't be copied + - have a text field with plain text and yaml or json formatted addon data could be useful as well + +* gui, better copying from the interface, especially the log box + +* manually select the primary addon in a group of addons to prevent synthetic titles + +* gui, right-click column header and show columns to disable + +* no errors displayed when installing from addon detail page + +* search, add ability to browse catalogue page by page + - returned to bucket 2022-03-02 + - the 'search' tab kinda sorta is this ... perhaps a preference to disable sampling? + +* http, add with-backoff support to download-file + - just had a wowinterface addon download timeout + +* ux, offer to clean up .nfo files when removing an addon directory + - not just .nfo files, but .zip files matching a pattern too + +* grouping + - I think the tree-table-view allows us to 'group' things now ... + - it's 'flat' at the moment, but it could be grouped by 'ignored', 'pinned', 'updates available' + - ignored are collapsed + +* gui, try replacing the auto fit columns with something like this: + - https://stackoverflow.com/questions/14650787/javafx-column-in-tableview-auto-fit-size#answer-49134109 + +# + +* bug, I noticed a weird issue on the mac where Tukui was installing an addon called "Tukui 2" + - the original "Tukui" addon indicated a release was available but another, seemingly identical addon "Tukui 2" had been installed + * add support for cloning git repositories - this is to get around addon repositories not uploading 'releases' when tagging - switch between branches @@ -42,27 +200,13 @@ see CHANGELOG.md for a more formal list of changes by release - how about a shallow clone? - just the files are cloned to a directory -* size of addon on disk - - I'd like to see a column with 'size on disk' - - "1024KiB", "1MiB", "1.04MB" - - I'd like to see a total size of all addons in addon dir - - "1024MiB", "1GiB" - - I'd like to see size of disk and free space - - "2TiB of 14TiB free" - - and everything mooshed together - - "1GiB of addons on /dev/foo with 2TiB of 14TiB available" - -* wowinterface, fetch addon data from secondary source - - *augment* what is currently implemented with my own source - - failure to fetch this other data shouldn't prevent wowi updates from working normally - - this source is hosted on github as static content, updated daily. +* zip, switch to apache commons compress for decompressing + - https://commons.apache.org/proper/commons-compress/ + - .tar.gz and 7z support would be interesting + - rar should just die already + - this would fix a major showstopper in porting to windows + - 2022-05-29: returned to bucket, gazumped by installing addon from file. -* 'core/state :db-stats', seems like a nice idea to put more information here - - known-hosts - - num addons per-host - - num addons favourited/user-catalogue - - num addons - - ...? * github, install addon from the auto-generated .zip files - it looks like auctionator has disabled/deleted their 'releases' and put a link to their curseforge page up @@ -71,14 +215,15 @@ see CHANGELOG.md for a more formal list of changes by release - https://github.com/Auctionator/Auctionator/tags - we'll have to do some directory name munging -* gui, right-click column header and show columns to disable +* wowinterface, fetch addon data from secondary source + - *augment* what is currently implemented with my own source + - failure to fetch this other data shouldn't prevent wowi updates from working normally + - this source is hosted on github as static content, updated daily. * github authentication - so user doesn't get capped - or have to wrangle GITHUB_AUTH tokens -* no errors displayed when installing from addon detail page - * bug, trade skill master string-converter changed directory names between 2.0.7 and 2.1.0 - see also Combuctor 9.1.3 vs Combuctor 8.1.1 with 'BagBrother' in old addons - BagBrother was removed but also got @@ -91,16 +236,9 @@ see CHANGELOG.md for a more formal list of changes by release install combuctor 9.1.3 find 'combuctor' and install from wowi (8.1.1) get weird orphaned BagBrother addon - - 2022-06-27, returned to bucket + - 2022-06-27, returned to bucket - I don't have a solution for this, good or bad. -* zip, switch to apache commons compress for decompressing - - https://commons.apache.org/proper/commons-compress/ - - .tar.gz and 7z support would be interesting - - rar should just die already - - this would fix a major showstopper in porting to windows - - 2022-05-29: returned to bucket, gazumped by installing addon from file. - * bug, addon detail, highlighted installed version is causing rows to be highlighted in the raw data column? - looks like a javafx problem, no idea how to fix - try reducing to smallest possible reproduction @@ -109,15 +247,13 @@ see CHANGELOG.md for a more formal list of changes by release - perhaps check the addon name isn't prefixed with '_Classic' ? - how many would be affected by this? * create a parser for that shit markup that is preventing reconcilation -* manually select the primary addon in a group of addons to prevent synthetic titles * finer grained control over grouping of addons -* gui, better copying from the interface, especially the log box + - like what? + - like a grouped addon or grouping lists of addons? * possible bug? installing combuctor 8.1.1 from file matches against the catalogue (good), then installing 9.1.3 file loses the match. - mutual dependencies information is mostly blank * catalogue, download counts for github addons -* search, add ability to browse catalogue page by page - - returned to bucket 2022-03-02 ### catalogue v3 / capture more addon data @@ -135,7 +271,6 @@ see CHANGELOG.md for a more formal list of changes by release ### - * investigate better popularity metric than 'downloads' - if we make an effort to scrape everyday, we can generate this popularity graph ourselves * wowinterface, revisit the pages that are being scraped, make sure we're not missing any @@ -148,11 +283,14 @@ see CHANGELOG.md for a more formal list of changes by release * bug, test [:core :clear-addon-ignore-flag--implicit-ignore] is printing an error when game-track-list definitely exists - what is removing it? +### major version 7 + * default to keeping last three zip files by default - stretch goal - probably not a good idea for this release where we might want to keep zips around -* 'downloading strongbox data' shouldn't be blocking the gui from starting +### + * user catalogue pane - context menu @@ -177,9 +315,6 @@ see CHANGELOG.md for a more formal list of changes by release - or the zip file is badly formed - stuff a regular user should gloss over but a dev might find useful -* http, add with-backoff support to download-file - - just had a wowinterface addon download timeout - * a more permanent store than just cached files - I want to store release data permanently - multiple pages @@ -212,23 +347,10 @@ see CHANGELOG.md for a more formal list of changes by release * gitlab, add optional API authentication like github -* ux, offer to clean up .nfo files when removing a directory - * bug, stacktrace on double refresh -* grouping - - I think the tree-table-view allows us to 'group' things now ... - - it's 'flat' at the moment, but it could be grouped by 'ignored', 'pinned', 'updates available' - - ignored are collapsed - -* add a 'add to user-catalogue' option to make an addon always available despite selected catalogue - - done, via favouriting, but! it's not available on the installed addon pane page - * add a 'catalogue is N days old' somewhere -* gui, try replacing the auto fit columns with something like this: - - https://stackoverflow.com/questions/14650787/javafx-column-in-tableview-auto-fit-size#answer-49134109 - * gui, toggleable highlighers as a menuitem - highlight unmatched - highlight updates @@ -284,9 +406,7 @@ see CHANGELOG.md for a more formal list of changes by release - feels a bit like scope creep to me - it would be nice and convenient, but a lot of work to build and maintain -* schedule user catalogue refreshes - - ensure the user catalogue doesn't get too stale and perform an update in the background if it looks like it's very old - - update README + * http, clear non-catalogue cache after session - it seems reasonable that stopping and starting the app will have it re-fetch addon summaries. diff --git a/build-linux-image.sh b/build-linux-image.sh index c05779f0..6736a41e 100755 --- a/build-linux-image.sh +++ b/build-linux-image.sh @@ -6,13 +6,16 @@ output_dir="custom-jre" rm -rf "./$output_dir" echo "--- building custom JRE ---" + +# compress=1 'constant string sharing' compresses better with AppImage than compress=2 'zip', 52MB -> 45MB +# - https://docs.oracle.com/en/java/javase/19/docs/specs/man/jlink.html#plugin-compress jlink \ --add-modules "java.sql,java.naming,java.desktop,jdk.unsupported,jdk.crypto.ec" \ --output "$output_dir" \ --strip-debug \ --no-man-pages \ --no-header-files \ - --compress=2 + --compress=1 # needed when built using Ubuntu as libjvm.so is *huge* # doesn't seem to hurt to strip the other .so files. diff --git a/cloverage.clj b/cloverage.clj index 85eba155..36eb7478 100644 --- a/cloverage.clj +++ b/cloverage.clj @@ -2,11 +2,12 @@ (:require [strongbox [main :as main] + [constants :as constants] + [utils :as utils] [http :as http] [joblib :as joblib] [logging :as logging] - [core :as core] - [catalogue :as catalogue]] + [core :as core]] [clojure.test :as test] [cloverage.coverage :as c])) @@ -22,7 +23,8 @@ http/*default-pause* 1 ;; ms http/*default-attempts* 1 ;;joblib/tick-delay joblib/*tick* - catalogue/host-disabled? (constantly false)] + utils/folder-size-bytes (constantly 0) + constants/max-user-catalogue-age 9999] (core/reset-logging!) (apply require (map symbol ns-list)) {:errors (reduce + ((juxt :error :fail) diff --git a/pom.xml b/pom.xml index bb019112..4506292e 100644 --- a/pom.xml +++ b/pom.xml @@ -4,7 +4,7 @@ ogri-la strongbox jar - 6.1.2 + 7.0.0 strongbox World Of Warcraft Addon Manager https://github.com/ogri-la/strongbox @@ -18,7 +18,7 @@ https://github.com/ogri-la/strongbox scm:git:git://github.com/ogri-la/strongbox.git scm:git:ssh://git@github.com/ogri-la/strongbox.git - 8d1220551085510b52aed14be3d79586429cea4f + 073a7afcc0b485b1e838f66efec023d0aaa5032b src @@ -83,12 +83,12 @@ org.clojure tools.cli - 1.0.206 + 1.0.214 org.clojure tools.namespace - 1.1.0 + 1.4.4 org.clojure @@ -113,7 +113,7 @@ clj-commons fs - 1.6.307 + 1.6.310 slugify @@ -128,43 +128,27 @@ org.flatland ordered - 1.5.9 + 1.15.11 clojure.java-time clojure.java-time - 0.3.3 + 1.2.0 envvar envvar 1.1.2 - - gui-diff - gui-diff - 0.6.7 - - - parsley - net.cgrant - - - - - com.taoensso - tufte - 2.2.0 - tolitius lasync - 0.1.23 + 0.1.24 cljfx cljfx - 1.7.19 + 1.7.22 javafx-web @@ -232,20 +216,15 @@ 17.0.2 mac - - org.apache.commons - commons-compress - 1.21 - org.ocpsoft.prettytime prettytime - 5.0.2.Final + 5.0.6.Final org.controlsfx controlsfx - 11.1.1 + 11.1.2 clj-http-fake @@ -253,6 +232,18 @@ 1.0.3 test + + gui-diff + gui-diff + 0.6.7 + + + parsley + net.cgrant + + + test + diff --git a/project.clj b/project.clj index e13fa5b4..4e15e895 100644 --- a/project.clj +++ b/project.clj @@ -1,4 +1,4 @@ -(defproject ogri-la/strongbox "6.1.2" +(defproject ogri-la/strongbox "7.0.0" :description "World Of Warcraft Addon Manager" :url "https://github.com/ogri-la/strongbox" :license {:name "GNU Affero General Public License (AGPL)" @@ -12,24 +12,22 @@ :pedantic? false :dependencies [[org.clojure/clojure "1.10.3"] - [org.clojure/tools.cli "1.0.206"] ;; cli arg parsing - [org.clojure/tools.namespace "1.1.0"] ;; reload code + [org.clojure/tools.cli "1.0.214"] ;; cli arg parsing + [org.clojure/tools.namespace "1.4.4"] ;; reload code [org.clojure/data.json "2.4.0"] ;; json handling [orchestra "2021.01.01-1"] ;; improved clojure.spec instrumentation ;; see lein deps :tree [com.taoensso/timbre "5.1.2"] ;; logging [clj-http "3.12.3"] ;; better http slurping - [clj-commons/fs "1.6.307"] ;; file system wrangling + [clj-commons/fs "1.6.310"] ;; file system wrangling [slugify "0.0.1"] [trptcolin/versioneer "0.2.0"] ;; version number wrangling. it's more involved than you might suspect - [org.flatland/ordered "1.5.9"] ;; better ordered map - [clojure.java-time "0.3.3"] ;; date/time handling library, https://github.com/dm3/clojure.java-time + [org.flatland/ordered "1.15.11"] ;; better ordered map + [clojure.java-time "1.2.0"] ;; date/time handling library, https://github.com/dm3/clojure.java-time [envvar "1.1.2"] ;; environment variable wrangling - [gui-diff "0.6.7" :exclusions [net.cgrant/parsley]] ;; pops up a graphical diff for test results - [com.taoensso/tufte "2.2.0"] ;; profiling - [tolitius/lasync "0.1.23"] ;; better parallel processing + [tolitius/lasync "0.1.24"] ;; better parallel processing - [cljfx "1.7.19" :exclusions [org.openjfx/javafx-web + [cljfx "1.7.22" :exclusions [org.openjfx/javafx-web org.openjfx/javafx-media]] [cljfx/css "1.1.0"] @@ -47,9 +45,8 @@ ;; GPLv3 compatible dependencies. ;; these don't need an exception in LICENCE.txt - [org.apache.commons/commons-compress "1.21"] ;; Apache 2.0 licenced, bz2 compression/decompression of static catalogue - [org.ocpsoft.prettytime/prettytime "5.0.2.Final"] ;; Apache 2.0 licenced, pretty date formatting - [org.controlsfx/controlsfx "11.1.1"] ;; BSD-3 + [org.ocpsoft.prettytime/prettytime "5.0.6.Final"] ;; Apache 2.0 licenced, pretty date formatting + [org.controlsfx/controlsfx "11.1.2"] ;; BSD-3 ;; remember to update the LICENCE.txt ;; remember to update pom file (`lein pom`) @@ -66,16 +63,21 @@ :resource-paths ["resources"] - :profiles {:dev {:plugins [[lein-ancient "0.7.0"]] - :resource-paths ["dev-resources" "resources"] ;; dev-resources take priority + :profiles {:repl {:source-paths ["repl"]} + :dev {:resource-paths ["dev-resources" "resources"] ;; dev-resources take priority :dependencies [[clj-http-fake "1.0.3"] ;; fake http responses for testing + [gui-diff "0.6.7" :exclusions [net.cgrant/parsley]] ;; pops up a graphical diff for test results ]} :uberjar {:aot :all ;; fixes hanging issue: ;; - https://github.com/cljfx/cljfx/issues/17 :injections [(javafx.application.Platform/exit)] - }} + :jvm-opts ["-Dclojure.compiler.disable-locals-clearing=true" + "-Dclojure.compiler.elide-meta=[:doc :file :line :added]" + "-Dclojure.compiler.direct-linking=true"]}} + + :repl-options {:init-ns strongbox.user} ;; see repl/strongbox/user.clj ;; debug output from JavaFX about which GTK it is looking for. ;; was useful in figuring out why javafx was failing to initialise even with xvfb. @@ -83,7 +85,8 @@ :main strongbox.main - :plugins [[lein-cljfmt "0.9.0"] + :plugins [[lein-ancient "0.7.0"] + [lein-cljfmt "0.9.0"] [jonase/eastwood "1.3.0"] [lein-cloverage "1.2.4"] [venantius/yagni "0.1.7"]] @@ -97,5 +100,6 @@ ;;:unused-fn-args ;; prefer to keep for readability ;;:keyword-typos ;; bugged with spec? ] - :only-modified true} + :only-modified true + } ) diff --git a/repl/strongbox/user.clj b/repl/strongbox/user.clj new file mode 100644 index 00000000..4ba06d2f --- /dev/null +++ b/repl/strongbox/user.clj @@ -0,0 +1,63 @@ +(ns strongbox.user + (:refer-clojure :rename {test clj-test}) + (:require + [taoensso.timbre :as timbre :refer [spy info warn error report]] + [clojure.test] + [clojure.tools.namespace.repl :as tn :refer [refresh]] + [strongbox + [addon :as addon] + [constants :as constants] + [main :as main :refer [restart stop]] + [catalogue :as catalogue] + [http :as http] + [core :as core] + [utils :as utils :refer [in?]] + ;;[cli :as cli] + ] + [gui.diff :refer [with-gui-diff]] + )) + +(defn test + [& [ns-kw fn-kw]] + (main/stop) + (clojure.tools.namespace.repl/refresh) ;; reloads all namespaces, including strongbox.whatever-test ones + (utils/instrument true) ;; always test with spec checking ON + + (try + ;; note! remember to update `cloverage.clj` with any new bindings + (with-redefs [core/*testing?* true + http/*default-pause* 1 ;; ms + http/*default-attempts* 1 + ;; don't pause while testing. nothing should depend on that pause happening. + ;; note! this is different to `joblib/tick-delay` not delaying when `joblib/*tick*` is unbound. + ;; tests still bind `joblib/*tick*` and run things in parallel. + ;;joblib/tick-delay joblib/*tick* + ;;main/*spec?* true + ;;cli/install-update-these-in-parallel cli/install-update-these-serially + ;;core/check-for-updates core/check-for-updates-serially + ;; for testing purposes, no addon host is disabled + utils/folder-size-bytes (constantly 0) + constants/max-user-catalogue-age 9999 + ] + (core/reset-logging!) + + (if ns-kw + (if (some #{ns-kw} [:main :utils :http + :core :toc :nfo :zip :config :catalogue :addon :logging :joblib + :cli :gui :jfx + :curseforge-api :wowinterface-api :gitlab-api :github-api :tukui-api + :release-json]) + (with-gui-diff + (if fn-kw + ;; `test-vars` will run the test but not give feedback if test passes OR test not found + ;; slightly better than nothing + (clojure.test/test-vars [(resolve (symbol (str "strongbox." (name ns-kw) "-test") (name fn-kw)))]) + (clojure.test/run-all-tests (re-pattern (str "strongbox." (name ns-kw) "-test")))) + ) + (error "unknown test file:" ns-kw)) + (clojure.test/run-all-tests #"strongbox\..*-test"))) + (finally + ;; use case: we run the tests from the repl and afterwards we call `restart` to start the app. + ;; `stop` inside `restart` will be outside of `with-redefs` and still have logging `:min-level` set to `:debug` + ;; it will dump a file and yadda yadda. + (core/reset-logging!)))) diff --git a/screenshots/screenshot-7.0.0-dark-detail-thumbnail.jpg b/screenshots/screenshot-7.0.0-dark-detail-thumbnail.jpg new file mode 100644 index 00000000..e1967c8f Binary files /dev/null and b/screenshots/screenshot-7.0.0-dark-detail-thumbnail.jpg differ diff --git a/screenshots/screenshot-7.0.0-dark-detail.png b/screenshots/screenshot-7.0.0-dark-detail.png new file mode 100644 index 00000000..d373a765 Binary files /dev/null and b/screenshots/screenshot-7.0.0-dark-detail.png differ diff --git a/screenshots/screenshot-7.0.0-dark-installed-fat-thumbnail.jpg b/screenshots/screenshot-7.0.0-dark-installed-fat-thumbnail.jpg new file mode 100644 index 00000000..9a311288 Binary files /dev/null and b/screenshots/screenshot-7.0.0-dark-installed-fat-thumbnail.jpg differ diff --git a/screenshots/screenshot-7.0.0-dark-installed-fat.png b/screenshots/screenshot-7.0.0-dark-installed-fat.png new file mode 100644 index 00000000..e39b1d3a Binary files /dev/null and b/screenshots/screenshot-7.0.0-dark-installed-fat.png differ diff --git a/screenshots/screenshot-7.0.0-dark-installed-skinny-thumbnail.jpg b/screenshots/screenshot-7.0.0-dark-installed-skinny-thumbnail.jpg new file mode 100644 index 00000000..3eb8a222 Binary files /dev/null and b/screenshots/screenshot-7.0.0-dark-installed-skinny-thumbnail.jpg differ diff --git a/screenshots/screenshot-7.0.0-dark-installed-skinny.png b/screenshots/screenshot-7.0.0-dark-installed-skinny.png new file mode 100644 index 00000000..4d67c2db Binary files /dev/null and b/screenshots/screenshot-7.0.0-dark-installed-skinny.png differ diff --git a/screenshots/screenshot-7.0.0-dark-installed-thumbnail.jpg b/screenshots/screenshot-7.0.0-dark-installed-thumbnail.jpg new file mode 100644 index 00000000..45ee1af3 Binary files /dev/null and b/screenshots/screenshot-7.0.0-dark-installed-thumbnail.jpg differ diff --git a/screenshots/screenshot-7.0.0-dark-installed.png b/screenshots/screenshot-7.0.0-dark-installed.png new file mode 100644 index 00000000..01926804 Binary files /dev/null and b/screenshots/screenshot-7.0.0-dark-installed.png differ diff --git a/screenshots/screenshot-7.0.0-detail-thumbnail.jpg b/screenshots/screenshot-7.0.0-detail-thumbnail.jpg new file mode 100644 index 00000000..4f2c777c Binary files /dev/null and b/screenshots/screenshot-7.0.0-detail-thumbnail.jpg differ diff --git a/screenshots/screenshot-7.0.0-detail.png b/screenshots/screenshot-7.0.0-detail.png new file mode 100644 index 00000000..0e3bf10a Binary files /dev/null and b/screenshots/screenshot-7.0.0-detail.png differ diff --git a/screenshots/screenshot-7.0.0-installed-fat-thumbnail.jpg b/screenshots/screenshot-7.0.0-installed-fat-thumbnail.jpg new file mode 100644 index 00000000..c0e35775 Binary files /dev/null and b/screenshots/screenshot-7.0.0-installed-fat-thumbnail.jpg differ diff --git a/screenshots/screenshot-7.0.0-installed-fat.png b/screenshots/screenshot-7.0.0-installed-fat.png new file mode 100644 index 00000000..82575e41 Binary files /dev/null and b/screenshots/screenshot-7.0.0-installed-fat.png differ diff --git a/screenshots/screenshot-7.0.0-installed-skinny-thumbnail.jpg b/screenshots/screenshot-7.0.0-installed-skinny-thumbnail.jpg new file mode 100644 index 00000000..f835934b Binary files /dev/null and b/screenshots/screenshot-7.0.0-installed-skinny-thumbnail.jpg differ diff --git a/screenshots/screenshot-7.0.0-installed-skinny.png b/screenshots/screenshot-7.0.0-installed-skinny.png new file mode 100644 index 00000000..87a142a6 Binary files /dev/null and b/screenshots/screenshot-7.0.0-installed-skinny.png differ diff --git a/screenshots/screenshot-7.0.0-installed-thumbnail.jpg b/screenshots/screenshot-7.0.0-installed-thumbnail.jpg new file mode 100644 index 00000000..b9241aee Binary files /dev/null and b/screenshots/screenshot-7.0.0-installed-thumbnail.jpg differ diff --git a/screenshots/screenshot-7.0.0-installed.png b/screenshots/screenshot-7.0.0-installed.png new file mode 100644 index 00000000..bb8d650f Binary files /dev/null and b/screenshots/screenshot-7.0.0-installed.png differ diff --git a/src/strongbox/addon.clj b/src/strongbox/addon.clj index 1e98b570..d8728e7b 100644 --- a/src/strongbox/addon.clj +++ b/src/strongbox/addon.clj @@ -18,6 +18,12 @@ (def dummy-dirname "not-the-addon-dir-you-are-looking-for") +(defn-spec host-disabled? boolean? + "returns `true` if the addon host has been disabled" + [addon map?] + (or (-> addon :source (= "curseforge")) + (-> addon :source (utils/in? sp/tukui-source-list)))) + (defn-spec -remove-addon! nil? "safely removes the given `addon-dirname` from `install-dir`. if the given `addon-dirname` is a mutual dependency with another addon, just remove it's entry from @@ -120,9 +126,15 @@ {:label (format "%s (group)" next-best-label) :description (format "group record for the %s addon" next-best-label)})) - addon-str (clojure.string/join ", " (map :dirname addons))] + ;; count total size in bytes, update top-level :dirsize + total-grouped-size (apply + (remove nil? (map :dirsize (:group-addons addon)))) + addon (if (> total-grouped-size 0) + (assoc addon :dirsize total-grouped-size) + addon) + + msg-str (clojure.string/join ", " (map :dirname addons))] - (logging/addon-log addon :info (format "contains %s addons: %s" (count addons) addon-str)) + (logging/addon-log addon :info (format "contains %s addons: %s" (count addons) msg-str)) addon))) ;; this flattens the newly grouped addons from a map into a list and joins the unknowns @@ -149,11 +161,9 @@ (defn-spec merge-toc-nfo (s/or :ok map?, :empty nil?) "merges `toc` data with `nfo` data with special handling for the `source-map-list`." [toc (s/nilable map?), nfo (s/nilable map?)] - (let [curse? (fn [source-map] - (= (:source source-map) "curseforge")) - source-map-list (some->> (merge-lists (extract-source-map-list toc) + (let [source-map-list (some->> (merge-lists (extract-source-map-list toc) (extract-source-map-list nfo)) - (remove curse?) + (remove host-disabled?) vec (assoc {} :source-map-list))] (merge toc nfo source-map-list))) diff --git a/src/strongbox/catalogue.clj b/src/strongbox/catalogue.clj index cbf9d668..41b4c2c4 100644 --- a/src/strongbox/catalogue.clj +++ b/src/strongbox/catalogue.clj @@ -3,7 +3,6 @@ [clojure.spec.alpha :as s] [orchestra.core :refer [defn-spec]] [taoensso.timbre :as log :refer [debug info warn error spy]] - [taoensso.tufte :as tufte :refer [p]] [java-time] [strongbox [constants :as constants] @@ -16,11 +15,6 @@ [gitlab-api :as gitlab-api] [github-api :as github-api]])) -(defn-spec host-disabled? boolean? - "returns `true` if the addon host has been disabled" - [addon map?] - (-> addon :source (= "curseforge"))) - (defn-spec -expand-summary (s/or :ok :addon/expanded, :error nil?) "fetches updates from the addon host for the given `addon` and `game-track`. does *not* support multiple game tracks or warning the user, see `expand-summary`. @@ -63,29 +57,38 @@ game-track* game-track game-track (some #{game-track} sp/game-tracks)] ;; :retail => :retail, :unknown-game-track => nil (cond - (not game-track) (error (format "unsupported game track '%s'." (str game-track*))) - (host-disabled? addon) (warn (utils/message-list (str "addon host 'curseforge' was disabled " constants/curseforge-cutoff-label ".") - ["use 'Source' and 'Find similar' from the addon context menu for alternatives."])) - :else (if-let [source-updates (if strict? - (-expand-summary addon game-track) - (utils/first-nn (partial -expand-summary addon) (get track-map game-track)))] - source-updates - - ;; "no 'Retail' release found on github" - ;; "no 'Classic' release found on wowinterface" - ;; "no 'Classic (TBC)', 'Classic' or 'Retail' release found on github" - (let [single-template "no '%s' release found on %s." - multi-template "no '%s', '%s', '%s' or '%s' release found on %s." - msg (if strict? - (format single-template (sp/game-track-labels-map game-track) (:source addon)) - (apply format multi-template (conj (mapv #(sp/game-track-labels-map %) (get track-map game-track)) - (:source addon))))] - (warn msg)))))) + (not game-track) + (error (format "unsupported game track '%s'." (str game-track*))) + + (strongbox.addon/host-disabled? addon) + (if (= (:source addon) "curseforge") + (warn (utils/message-list (str "addon host 'curseforge' was disabled " constants/curseforge-cutoff-label ".") + ["use 'Source' and 'Find similar' from the addon context menu for alternatives."])) + (warn (utils/message-list (str "addon host 'tukui' was disabled " constants/tukui-cutoff-label ".") + ["use 'Source' and 'Find similar' from the addon context menu for alternatives."]))) + + :else + (if-let [source-updates (if strict? + (-expand-summary addon game-track) + (utils/first-nn (partial -expand-summary addon) (get track-map game-track)))] + source-updates + + ;; "no 'Retail' release found on github" + ;; "no 'Classic' release found on wowinterface" + ;; "no 'Classic (TBC)', 'Classic' or 'Retail' release found on github" + (let [single-template "no '%s' release found on %s." + multi-template "no '%s', '%s', '%s' or '%s' release found on %s." + msg (if strict? + (format single-template (sp/game-track-labels-map game-track) (:source addon)) + (apply format multi-template (conj (mapv #(sp/game-track-labels-map %) (get track-map game-track)) + (:source addon))))] + (warn msg)))))) ;; (defn-spec toc2summary (s/nilable :addon/summary) - "accepts toc or toc+nfo data and emits a version of the data that validates as an `:addon/summary`" + "accepts toc or toc+nfo data and emits a version of the data that validates as an `:addon/summary`. + used as a last ditch effort to match installed addons against the catalogue by guessing a bunch of parameters." [toc (s/or :just-toc :addon/toc, :mixed :addon/toc+nfo)] (when-not (:ignore? toc) (let [sink nil @@ -130,8 +133,7 @@ (defn-spec format-catalogue-data :catalogue/catalogue "returns a correctly formatted, ordered, catalogue given a list of addons and a datestamp" [addon-list :addon/summary-list, datestamp ::sp/ymd-dt] - (let [addon-list (p :cat/sort-addons - (sort-by :name addon-list))] + (let [addon-list (sort-by :name addon-list)] {:spec {:version 2} :datestamp datestamp :total (count addon-list) @@ -160,8 +162,7 @@ (defn validate "validates the given data as a `:catalogue/catalogue`, returning nil if data is invalid" [catalogue] - (p :catalogue:validate - (sp/valid-or-nil :catalogue/catalogue catalogue))) + (sp/valid-or-nil :catalogue/catalogue catalogue)) (defn-spec write-catalogue (s/or :ok ::sp/extant-file, :error nil?) "write catalogue to given `output-file` as JSON. returns path to output file" diff --git a/src/strongbox/cli.clj b/src/strongbox/cli.clj index 00986472..faae9729 100644 --- a/src/strongbox/cli.clj +++ b/src/strongbox/cli.clj @@ -7,7 +7,6 @@ [me.raynes.fs :as fs] [strongbox [zip :as zip] - [constants :as constants] [joblib :as joblib] [github-api :as github-api] [tukui-api :as tukui-api] @@ -17,7 +16,6 @@ [logging :as logging] [addon :as addon] [specs :as sp] - [catalogue :as catalogue] [http :as http] [utils :as utils :refer [if-let* message-list]] [core :as core :refer [get-state paths]]])) @@ -31,6 +29,18 @@ (swap! core/state update-in [:gui-split-pane] not) nil) +(defn-spec set-split-pane nil? + "set `selected?` to `true` to split the main pane horizontally." + [selected? boolean?] + (swap! core/state assoc :gui-split-pane selected?) + nil) + +(defn-spec set-sub-pane nil? + "sets the content of the bottom (sub) pane. either `:notice-logger` (default) or `:stats`" + [sub-pane :gui/sub-pane-content] + (swap! core/state assoc :gui-sub-pane sub-pane) + nil) + ;; selecting addons (defn-spec select-addons-for-search! nil? @@ -70,44 +80,23 @@ ;; ui refreshing -(defn-spec hard-refresh nil? - "unlike `core/refresh`, `cli/hard-refresh` clears the http cache before checking for addon updates." - [] - ;; why can't we be more specific, like just the addons for the current addon-dir? - ;; the url used to 'expand' an addon from the catalogue isn't preserved. - ;; it may also change with the game track (tukui, historically) or not even exist (tukui, currently). - ;; a thorough inspection would be too much code. - ;; this also removes the etag cache. the etag db only applies to catalogues and downloaded zip files. - (core/delete-http-cache!) - (core/refresh)) - -(defn-spec half-refresh nil? - "like `core/refresh` but excludes reloading catalogues, focusing on re-reading installed addons, - matching them to the catalogue and reapplying host updates." - [] - (report "refresh") - (core/load-all-installed-addons) - (core/match-all-installed-addons-with-catalogue) - (core/check-for-updates) - (core/save-settings!)) - (defn-spec set-addon-dir! nil? "adds and sets the given `addon-dir`, then reloads addons." [addon-dir ::sp/addon-dir] (core/set-addon-dir! addon-dir) - (half-refresh)) + (core/half-refresh)) (defn-spec set-game-track-strictness! nil? "changes the the 'strict' flag for the current addon directory, then reloads addons." [new-strictness-level ::sp/strict?] (core/set-game-track-strictness! new-strictness-level) - (half-refresh)) + (core/half-refresh)) (defn-spec remove-addon-dir! nil? "deletes an addon-dir, selects first available addon dir, partial refresh of application state" [] (core/remove-addon-dir!) ;; the next addon dir is selected, if any - (half-refresh)) + (core/half-refresh)) ;; search @@ -199,16 +188,10 @@ (doseq [path path-list] (core/state-bind path listener)))) -(defn-spec reset-search-navigation nil? - "returns the search results to page 1" - [] - (swap! core/state assoc-in [:search :page] 0) - nil) - (defn-spec search-add-filter nil? "adds a new filter to the search `filter-by` state." [filter-by :search/filter-by, val any?] - (reset-search-navigation) + (core/reset-search-navigation) (case filter-by :source (swap! core/state assoc-in [:search :filter-by filter-by] (utils/nilable val)) :tag (swap! core/state update-in [:search :filter-by filter-by] conj val) @@ -218,17 +201,23 @@ (defn-spec search-rm-filter nil? "removes a filter from the search `filter-by` state." [filter-by :search/filter-by, val any?] - (reset-search-navigation) + (core/reset-search-navigation) (swap! core/state update-in [:search :filter-by filter-by] clojure.set/difference #{val}) nil) (defn-spec search-toggle-filter nil? "toggles boolean filters on and off" [filter-by :search/filter-by] - (reset-search-navigation) + (core/reset-search-navigation) (swap! core/state update-in [:search :filter-by filter-by] not) nil) +(defn-spec clear-search! nil? + "resets the search state to defaults and jogs the search results" + [] + (core/reset-search-state!) + (bump-search)) + ;; (defn-spec change-catalogue nil? @@ -271,7 +260,7 @@ (info (format "pinning to \"%s\"" (:installed-version addon))) (addon/pin! (core/selected-addon-dir) addon))) addon-list) - (half-refresh))) + (core/half-refresh))) (defn-spec unpin nil? "unpins the addons in given `addon-list` regardless of whether they are pinned or not. @@ -628,72 +617,12 @@ ;; importing addons -;; todo: logic might be better off in core.clj -(defn-spec find-addon (s/or :ok :addon/summary, :error nil?) - "given a URL of a supported addon host, parses it, looks for it in the catalogue, expands addon and attempts to install a dry run installation. - if successful, returns the addon-summary. - dry-run installation attempt can be skipped by setting `attempt-dry-run` to false." - [addon-url string?, attempt-dry-run? boolean?] - (binding [http/*cache* (core/cache)] - (if-let* [addon-summary-stub (catalogue/parse-user-string addon-url) - source (:source addon-summary-stub) - match-on-list [[[:source :url] [:source :url]] - [[:source :source-id] [:source :source-id]]] - addon-summary (cond - (= source "github") - (or (github-api/find-addon (:source-id addon-summary-stub)) - (error (message-list - "Failed. URL must be:" - ["valid" - "originate from github.com" - "addon uses 'releases'" - "latest release has a packaged 'asset'" - "asset must be a .zip file" - "zip file must be structured like an addon"]))) - - (= source "gitlab") - (or (gitlab-api/find-addon (:source-id addon-summary-stub)) - (error (message-list - "Failed. URL must be:" - ["valid" - "originate from gitlab.com" - "addon uses releases" - "latest release has a custom asset with a 'link'" - "link type must be either a 'package' or 'other'"]))) - - (= source "curseforge") - (error (str "addon host 'curseforge' was disabled " constants/curseforge-cutoff-label ".")) - - :else - ;; look in the current catalogue. emit an error if we fail - (or (:catalogue-match (core/-find-first-in-db (or (core/get-state :db) []) addon-summary-stub match-on-list)) - (error (format "couldn't find addon in catalogue '%s'" - (name (core/get-state :cfg :selected-catalogue)))))) - - ;; game track doesn't matter when adding it to the user catalogue. - ;; prefer retail though (it's the most common) and `strict` here is `false` - addon (or (catalogue/expand-summary addon-summary :retail false) - (error "failed to fetch details of addon")) - - ;; a dry-run is good when importing an addon for the first time but - ;; not necessary when updating the user-catalogue. - _ (if-not attempt-dry-run? - true - (or (core/install-addon-guard addon (core/selected-addon-dir) true) - (error "failed dry-run installation")))] - - ;; if-let* was successful! - addon-summary - - ;; failed if-let* :( - nil))) - (defn-spec import-addon nil? "goes looking for given `addon-url` and, if found, adds it to the user catalogue and then installs it." [addon-url string?] (binding [http/*cache* (core/cache)] (if-let* [dry-run? true - addon-summary (find-addon addon-url dry-run?) + addon-summary (core/find-addon addon-url dry-run?) addon (core/expand-summary-wrapper addon-summary)] ;; success! add to user-catalogue and proceed to install (do (core/add-user-addon! addon-summary) @@ -706,38 +635,9 @@ ;; failed to find or expand summary, probably because of selected game track. nil))) -(defn-spec refresh-user-catalogue-item nil? - "refresh the details of an individual `addon` in the user catalogue, optionally writing the updated catalogue to file." - ([addon :addon/summary] - (refresh-user-catalogue-item addon true)) - ([addon :addon/summary, write? boolean?] - (logging/with-addon addon - (info "refreshing details") - (try - (let [attempt-dry-run? false - refreshed-addon (find-addon (:url addon) attempt-dry-run?)] - (if refreshed-addon - (do (core/add-user-addon! refreshed-addon) - (when write? - (core/write-user-catalogue!)) - (info "... done!")) - (warn "failed to refresh catalogue entry"))) - (catch Exception e - (error (format "an unexpected error happened while updating the details for '%s' in the user-catalogue: %s" - (:name addon) (.getMessage e)))))))) - -(defn-spec refresh-user-catalogue nil? - "refresh the details of all addons in the user catalogue, writing the updated catalogue to file once." +(defn refresh-user-catalogue [] - (binding [http/*cache* (core/cache)] - (info (format "refreshing \"%s\", this may take a minute ..." (-> (core/paths :user-catalogue-file) fs/base-name))) - (let [write? false] - (doseq [user-addon (core/get-state :user-catalogue :addon-summary-list)] - (refresh-user-catalogue-item user-addon write?))) - (core/write-user-catalogue!)) - nil) - -;; + (core/refresh-user-catalogue)) ;; todo: shift to core.clj or addon.clj (defn-spec addon-source-map-to-url (s/or :ok ::sp/url, :error nil?) @@ -776,50 +676,6 @@ :else (or (:version row) (:installed-version row)))) -;; todo: do we really need this separate to jfx/gui-column-map ? -(def column-map - "common column names mapped to labels and value-generating functions. - jfx.clj takes this and merges/augments it with jfx-specific stuff." - {:browse-local {:label "browse" :value-fn :addon-dir} - :source {:label "source" :value-fn :source} - :source-id {:label "ID" :value-fn :source-id} - :source-map-list {:label "other sources" :value-fn (fn [row] - (->> row - :source-map-list - (map :source) - (remove #(= % (:source row))) - utils/nilable - (clojure.string/join ", ")))} - :name {:label "name" :value-fn (comp utils/no-new-lines :label)} - :description {:label "description" :value-fn (comp utils/no-new-lines :description)} - :tag-list {:label "tags" :value-fn (fn [row] - (when-not (empty? (:tag-list row)) - (str (:tag-list row))))} - :updated-date {:label "updated" :value-fn (comp utils/format-dt :updated-date)} - :created-date {:label "created" :value-fn (comp utils/format-dt :created-date)} - :installed-version {:label "installed" :value-fn :installed-version} - :available-version {:label "available" :value-fn available-versions-v1} - :combined-version {:label "version" :value-fn available-versions-v2} - :game-version {:label "WoW" - :value-fn (fn [row] - (some-> row :interface-version str utils/interface-version-to-game-version))} - - :uber-button {:label nil ;; the gui will use the column-id (`:uber-button`) for the column menu when label is `nil` - :value-fn (fn [row] - (let [queue (core/get-state :job-queue) - job-id (joblib/addon-id row)] - (if (and (core/unsteady? (:name row)) - (joblib/has-job? queue job-id)) - ;; parallel job in progress, show a ticker. - "*" - (cond - (:ignore? row) (:ignored constants/glyph-map) - (:pinned-version row) (:pinned constants/glyph-map) - (core/unsteady? (:name row)) (:unsteady constants/glyph-map) - (addon-has-errors? row) (:errors constants/glyph-map) - (addon-has-warnings? row) (:warnings constants/glyph-map) - :else (:tick constants/glyph-map)))))}}) - (defn-spec toggle-ui-column nil? "toggles the display of the given `column-id` in the user preferences." [column-id keyword?, selected? boolean?] @@ -853,7 +709,7 @@ [addon :addon/toc+nfo, new-source-map :addon/source-map] (when-not (= (:source addon) (:source new-source-map)) (addon/switch-source! (core/selected-addon-dir) addon new-source-map) - (half-refresh))) + (core/half-refresh))) ;; @@ -863,6 +719,20 @@ (core/add-user-addon! addon-summary) (core/write-user-catalogue!)) +(defn-spec add-addon-to-user-catalogue nil? + "given an `addon` with only a `:source` and `:source-id`, find it in the catalogue and add it to the user-catalogue. + fails if addon cannot be found in catalogue." + [addon map?] + (logging/with-addon addon + (if-not (s/valid? :addon/source-map addon) + (error "failed to star addon, it is missing some basic information ('source' and 'source-id').") + (let [catalogue-addon (core/db-addon-by-source-and-source-id + (core/get-state :db) (:source addon) (:source-id addon))] + (if-not (s/valid? :addon/summary catalogue-addon) + (error "failed to star addon, it is not matched to the catalogue.") + (add-summary-to-user-catalogue catalogue-addon))))) + nil) + (defn-spec remove-summary-from-user-catalogue nil? "removes an `addon-summary` (catalogue entry) from the user-catalogue, but only if it's present." [addon-summary :addon/summary] diff --git a/src/strongbox/config.clj b/src/strongbox/config.clj index 63bbfb5e..9d7009f7 100644 --- a/src/strongbox/config.clj +++ b/src/strongbox/config.clj @@ -28,7 +28,11 @@ (let [curse-idx 3] (utils/drop-idx -default-catalogue-list--v2 curse-idx))) -(def -default-catalogue-list -default-catalogue-list--v3) +(def -default-catalogue-list--v4 + (let [tukui-idx 2] + (utils/drop-idx -default-catalogue-list--v3 tukui-idx))) + +(def -default-catalogue-list -default-catalogue-list--v4) (def default-cfg {:addon-dir-list [] @@ -44,7 +48,10 @@ ;; see `specs/column-preset-list` for selectable presets ;; see `cli/column-map` for all known columns - :ui-selected-columns sp/default-column-list}}) + :ui-selected-columns sp/default-column-list + + ;; refresh the user-catalogue every N days + :keep-user-catalogue-updated false}}) (defn handle-install-dir "`:install-dir` was once supported in the user configuration but is now only supported in the command line options. @@ -121,6 +128,12 @@ (let [new-catalogue-list (vec (remove #(= :curseforge (:name %)) (:catalogue-location-list cfg)))] (assoc cfg :catalogue-location-list new-catalogue-list))) +(defn-spec remove-tukui-catalogue map? + "removes the tukui catalogue from the user config." + [cfg map?] + (let [new-catalogue-list (vec (remove #(= :tukui (:name %)) (:catalogue-location-list cfg)))] + (assoc cfg :catalogue-location-list new-catalogue-list))) + (defn-spec handle-column-preferences map? "handles upgrading of the default column list. if the config is using the v1 defaults, upgrade to v2 defaults." @@ -171,6 +184,7 @@ remove-invalid-catalogue-location-entries add-github-catalogue remove-curseforge-catalogue + remove-tukui-catalogue handle-column-preferences strip-unspecced-keys) message (format "configuration from %s is invalid and will be ignored: %s" diff --git a/src/strongbox/constants.clj b/src/strongbox/constants.clj index 09c63e62..0c96c0f2 100644 --- a/src/strongbox/constants.clj +++ b/src/strongbox/constants.clj @@ -64,6 +64,7 @@ (def glyph-map glyph-map--fontawesome) (def curseforge-cutoff-label "Feb 1st, 2022") +(def tukui-cutoff-label "June 1st, 2023") (def releases "https://wowpedia.fandom.com/wiki/Patch" @@ -122,3 +123,5 @@ "1.3" "World of Warcraft: Ruins of the Dire Maul" "1.2" "World of Warcraft: Mysteries of Maraudon" "1" "World of Warcraft"}) + +(def max-user-catalogue-age 28) diff --git a/src/strongbox/core.clj b/src/strongbox/core.clj index 31681393..d8581bbb 100644 --- a/src/strongbox/core.clj +++ b/src/strongbox/core.clj @@ -15,9 +15,11 @@ [zip :as zip] [http :as http] [logging :as logging] - [utils :as utils :refer [join nav-map delete-many-files! expand-path if-let*]] + [utils :as utils :refer [join nav-map delete-many-files! expand-path if-let* message-list]] [catalogue :as catalogue] [specs :as sp] + [github-api :as github-api] + [gitlab-api :as gitlab-api] [joblib :as joblib]]) (:import [org.apache.commons.compress.compressors CompressorStreamFactory CompressorException] @@ -32,7 +34,7 @@ (def static-catalogue "a bz2 compressed copy of the full catalogue used when the remote catalogue is unavailable or corrupt. from this we can do per-host filtering as well as shortening to generate the other catalogues." - (utils/compressed-slurp "full-catalogue.json")) + (utils/compile-time-slurp "full-catalogue.json")) (defn generate-path-map "filesystem paths whose location may vary based on the current working directory, environment variables, etc. @@ -118,7 +120,7 @@ :db nil ;; some generated stats about the db that are updated just once at load time. - :db-stats {:known-host-list []} + :db-stats nil ;; the list of addons from the user-catalogue :user-catalogue nil @@ -146,9 +148,12 @@ ;; per-tab log levels are attached to each tab in the `:tab-list` :gui-log-level :info - ;; split the gui in two horizontally with the notice logger on the bottom. + ;; split the gui in two horizontally with the sub-pane on the bottom. :gui-split-pane false + ;; default widget for the sub-pane + :gui-sub-pane :notice-logger + ;; addons in an 'unsteady' state (data being updated, addon being installed, etc) ;; allows a UI to watch and update the gui with progress. ;; also allows us to pause doing a thing until an addon is removed from the list. @@ -674,7 +679,7 @@ "derives the requested catalogue from the static catalogue." [catalogue-location :catalogue/location] (let [opts {} - catalogue (catalogue/read-catalogue (.getBytes (utils/decompress-bytes static-catalogue)) opts) + catalogue (catalogue/read-catalogue (.getBytes static-catalogue) opts) catalogue (assoc catalogue :emergency? true)] (warn (utils/message-list (format "the remote catalogue is unreachable or corrupt: %s" (:source catalogue-location)) @@ -685,7 +690,7 @@ :short (catalogue/shorten-catalogue catalogue) ;;:curseforge (catalogue/filter-catalogue catalogue "curseforge") :wowinterface (catalogue/filter-catalogue catalogue "wowinterface") - :tukui (catalogue/filter-catalogue catalogue "tukui") + ;;:tukui (catalogue/filter-catalogue catalogue "tukui") :github (catalogue/filter-catalogue catalogue "github") ;; todo: add a test for this. I expect all derivable catalogues to be available nil))) @@ -728,9 +733,7 @@ "returns the contents of the user catalogue or `nil` if it doesn't exist." [] (let [catalogue (catalogue/read-catalogue (paths :user-catalogue-file) {:bad-data? nil}) - curse? (fn [addon] - (-> addon :source (= "curseforge"))) - new-summary-list (->> catalogue :addon-summary-list (remove curse?) vec)] + new-summary-list (->> catalogue :addon-summary-list (remove addon/host-disabled?) vec)] (when catalogue (catalogue/new-catalogue new-summary-list)))) @@ -838,15 +841,23 @@ (or (-find-first-in-db db installed-addon match-on-list) installed-addon)))) +(defn-spec db-addon-by-source-and-source-id (s/nilable :addon/summary) + "returns the first addon summary from `db` whose source and source-id exactly match the given `source` and `source-id`. + there should only ever be one or zero such addons." + [db :addon/summary-list, source :addon/source, source-id :addon/source-id] + (let [xf (filter #(and (= source (:source %)) + (= source-id (:source-id %))))] + (first (into [] xf db)))) + (defn-spec db-addon-by-source-and-name :addon/summary-list - "returns a list of addon summaries from `db` whose source and name match (exactly) the given `source` and `name`" + "returns a list of addon summaries from `db` whose source and name exactly match the given `source` and `name`." [db :addon/summary-list, source :addon/source, name ::sp/name] (let [xf (filter #(and (= source (:source %)) (= name (:name %))))] (into [] xf db))) (defn-spec db-addon-by-name :addon/summary-list - "returns a list of addon summaries from `db` whose name matches (exactly) the given `name`" + "returns a list of addon summaries from `db` whose name exactly matches the given `name`." [db :addon/summary-list, name ::sp/name] (let [xf (filter #(= name (:name %)))] (into [] xf db))) @@ -915,12 +926,24 @@ (partition-all cap (seque 100 (filter match-fn db)))))))) (defn-spec empty-search-results nil? - "empties app state of search results. - this is to clear out anything between catalogue reloads." + "empties app state of search *results* but not filters. + this handles catalogue reloads but preserves user filtering." [] (swap! state update-in [:search] merge (select-keys -search-state-template [:page :results :selected-results-list])) nil) +(defn-spec reset-search-navigation nil? + "resets the search results to page 1" + [] + (swap! state assoc-in [:search :page] 0) + nil) + +(defn-spec reset-search-state! nil? + "replaces search state with default settings." + [] + (swap! state update-in [:search] merge -search-state-template) + nil) + (defn-spec db-load-user-catalogue nil? "loads the user catalogue into state, but only if it hasn't already been loaded." [] @@ -986,14 +1009,7 @@ (let [final-catalogue (load-current-catalogue)] (when-not (empty? final-catalogue) - (swap! state merge {:db (:addon-summary-list final-catalogue) - :db-stats {:num-addons (count (:addon-summary-list final-catalogue)) - :known-host-list (->> final-catalogue - :addon-summary-list - (map :source) - distinct - sort - vec)}})))) + (swap! state assoc :db (:addon-summary-list final-catalogue))))) nil) (defn-spec -match-installed-addon-list-with-catalogue :addon/installed-list @@ -1168,6 +1184,119 @@ ;; +(defn-spec find-addon (s/or :ok :addon/summary, :error nil?) + "given a URL of a supported addon host, parses it, looks for it in the catalogue, expands addon and attempts to install a dry run installation. + if successful, returns the addon-summary. + dry-run installation attempt can be skipped by setting `attempt-dry-run` to false." + [addon-url string?, attempt-dry-run? boolean?] + (binding [http/*cache* (cache)] + (if-let* [addon-summary-stub (catalogue/parse-user-string addon-url) + source (:source addon-summary-stub) + match-on-list [[[:source :url] [:source :url]] + [[:source :source-id] [:source :source-id]]] + addon-summary (cond + (= source "github") + (or (github-api/find-addon (:source-id addon-summary-stub)) + (error (message-list + "Failed. URL must be:" + ["valid" + "originate from github.com" + "addon uses 'releases'" + "latest release has a packaged 'asset'" + "asset must be a .zip file" + "zip file must be structured like an addon"]))) + + (= source "gitlab") + (or (gitlab-api/find-addon (:source-id addon-summary-stub)) + (error (message-list + "Failed. URL must be:" + ["valid" + "originate from gitlab.com" + "addon uses releases" + "latest release has a custom asset with a 'link'" + "link type must be either a 'package' or 'other'"]))) + + (= source "curseforge") + (error (str "addon host 'curseforge' was disabled " constants/curseforge-cutoff-label ".")) + + (utils/in? source sp/tukui-source-list) + (error (str "addon host 'tukui' was disabled " constants/tukui-cutoff-label ".")) + + :else + ;; look in the current catalogue. emit an error if we fail + (or (:catalogue-match (-find-first-in-db (or (get-state :db) []) addon-summary-stub match-on-list)) + (error (format "couldn't find addon in catalogue '%s'" + (name (get-state :cfg :selected-catalogue)))))) + + ;; game track doesn't matter when adding it to the user catalogue. + ;; prefer retail though (it's the most common) and `strict` here is `false` + addon (or (catalogue/expand-summary addon-summary :retail false) + (error "failed to fetch details of addon")) + + ;; a dry-run is good when importing an addon for the first time but + ;; not necessary when updating the user-catalogue. + _ (if-not attempt-dry-run? + true + (or (install-addon-guard addon (selected-addon-dir) true) + (error "failed dry-run installation")))] + + ;; if-let* was successful! + addon-summary + + ;; failed if-let* :( + nil))) + +(defn-spec refresh-user-catalogue-item nil? + "refresh the details of an individual `addon` in the user catalogue, optionally writing the updated catalogue to file." + [addon :addon/summary, db :addon/summary-list] + (logging/with-addon addon + (info "refreshing user-catalogue entry") + (try + (let [{:keys [source source-id url]} addon + refreshed-addon (db-addon-by-source-and-source-id db source source-id) + attempt-dry-run? false + refreshed-addon (or refreshed-addon + (find-addon url attempt-dry-run?))] + (if-not refreshed-addon + (warn "failed to refresh user-catalogue entry as the addon was not found in the catalogue or online") + (add-user-addon! refreshed-addon))) + + (catch Exception e + (error (format "an unexpected error happened while refreshing the user-catalogue entry: %s" (.getMessage e))))))) + +(defn-spec refresh-user-catalogue nil? + "refresh the details of all addons in the user catalogue, writing the updated catalogue to file once." + [] + (binding [http/*cache* (cache)] + (let [path (fs/base-name (paths :user-catalogue-file)) ;; "user-catalogue.json" + ;; we can't assume the full-catalogue is available. + _ (download-catalogue (get-catalogue-location :full)) + db (catalogue/read-catalogue (catalogue-local-path :full)) + full-catalogue (or (:addon-summary-list db) [])] + (info (format "refreshing \"%s\", this may take a minute ..." path)) + (doseq [user-addon (get-state :user-catalogue :addon-summary-list)] + (refresh-user-catalogue-item user-addon full-catalogue)) + (write-user-catalogue!) + (info (format "\"%s\" has been refreshed" path)))) + nil) + +(defn-spec refresh-user-catalogue? boolean? + "predicate, returns `true` if the user-catalogue needs a refresh." + [keep-user-catalogue-updated? boolean?, catalogue-datestamp (s/nilable ::sp/inst)] + (and keep-user-catalogue-updated? + catalogue-datestamp + (utils/older-than? catalogue-datestamp constants/max-user-catalogue-age :days))) + +(defn-spec scheduled-user-catalogue-refresh nil? + "checks the loaded database and calls `refresh-user-catalogue` if it's considered too old." + [] + (when (refresh-user-catalogue? (get-state :cfg :preferences :keep-user-catalogue-updated) + (get-state :user-catalogue :datestamp)) + (info (format "user-catalogue not updated in the last %s days, automatic refresh triggered." constants/max-user-catalogue-age)) + (refresh-user-catalogue))) + +;; + (defn-spec init-dirs nil? "ensure all directories in `generate-path-map` exist and are writable, creating them if necessary. this logic depends on paths that are not generated until the application has been started." @@ -1254,35 +1383,113 @@ [] (versioneer/get-version "ogri-la" "strongbox")) -(defn-spec -latest-strongbox-release (s/or :ok string?, :failed? keyword) +(def download-version-lock (Object.)) + +(defn-spec -download-strongbox-release (s/or :ok string?, :failed? keyword) "returns the most recently released version of strongbox on github. returns `:failed` if an error occurred while downloading/decoding/extracting the version name, rather than `nil`. `nil` is used to mean 'not set (yet)' in the app state." [] - (binding [http/*cache* (cache)] - (let [message "downloading strongbox version data" - url "https://api.github.com/repos/ogri-la/strongbox/releases/latest"] - (or (some-> url (http/download message) http/sink-error utils/from-json :tag_name) - :failed)))) + (locking download-version-lock + (binding [http/*cache* (cache)] + (let [message "downloading strongbox version data" + url "https://api.github.com/repos/ogri-la/strongbox/releases/latest"] + (or (some-> url (http/download message) http/sink-error utils/from-json :tag_name) + :failed))))) -(defn-spec latest-strongbox-release (s/nilable string?) - "returns the most recently released version of strongbox on github or `nil` if it can't." +(defn-spec latest-strongbox-version? boolean? + "returns `true` if the given `latest-release` is the *most recent known* version of strongbox. + when called with no parameters the `latest-release` is pulled from application state." + ([] + (latest-strongbox-version? (get-state :latest-strongbox-release))) + ([latest-release ::sp/latest-strongbox-release] + (case latest-release + nil true ;; we haven't looked yet, so yes, we're the latest :) + :failed true ;; we've already looked and failed, so as far as we know we're the latest. + (let [version-running (strongbox-version) + sorted-asc (utils/sort-semver-strings [latest-release version-running])] + (= version-running (last sorted-asc)))))) + +(defn-spec latest-strongbox-release! ::sp/latest-strongbox-release + "downloads and sets the most recently released version of strongbox on github, returning the found version or `:failed` on error." [] (let [lsr (get-state :latest-strongbox-release)] (case lsr - nil (let [lsr (-latest-strongbox-release)] + ;; we haven't looked yet, so look. + nil (let [lsr (-download-strongbox-release)] (swap! state assoc :latest-strongbox-release lsr) - (latest-strongbox-release)) ;; recurse + lsr) + ;; we've already looked and failed, don't try again this session. :failed nil + ;; we've already looked, return what we found lsr))) -(defn-spec latest-strongbox-version? boolean? - "returns true if the *running instance* of strongbox is the *most recent known* version of strongbox." +;; stats + +(defn-spec github-stats! (s/nilable :github/requests-stats) + "returns a map of github request rate limiting stats by querying the `/rate_limit` API endpoint. + doesn't make a Github `/rate_limit` request more often than once a minute." + [] + (when (and (started?) + (not *testing?*)) + (binding [http/*cache* (cache) + http/*expiry-offset-minutes* 1] + (when-let [resp (some-> "https://api.github.com/rate_limit" http/download http/sink-error utils/from-json :resources :core)] + {:github/token-set? (not (nil? (System/getenv "GITHUB_TOKEN"))) + :github/requests-limit (:limit resp) + :github/requests-limit-reset-minutes (-> resp :reset utils/unix-time-to-datetime utils/minutes-from-now) + :github/requests-remaining (:remaining resp) + :github/requests-used (:used resp)})))) + +(defn-spec app-stats map? + "summarises application state and returns a map of stats" + [state map?] + (let [num-addons (-> state :db count) + known-host-list (->> state :db (map :source) distinct sort vec) + num-addons-starred (-> state :user-catalogue-idx count) + + num-addons-installed (-> state :installed-addon-list count) + num-addons-installed-matched (->> state :installed-addon-list (filter :matched?) count) + num-addons-installed-ignored (->> state :installed-addon-list (filter addon/ignored?) count) + num-addons-installed-bytes (->> state :installed-addon-list (map :dirsize) (remove nil?) (reduce +)) + + num-addons-reducer (fn [acc addon] + (update acc (-> addon :source (or "none")) (fnil inc 0))) + num-addons-per-host (reduce num-addons-reducer {} (-> state :db)) + num-addons-installed-per-host (reduce num-addons-reducer {} (-> state :installed-addon-list))] + + (merge + {:addons/total num-addons + :addons/known-host-list known-host-list + :addons/num-by-host num-addons-per-host + :addons/total-starred num-addons-starred + + :installed-addons/total num-addons-installed + :installed-addons/num-matched num-addons-installed-matched + :installed-addons/num-by-host num-addons-installed-per-host + :installed-addons/num-ignored num-addons-installed-ignored + :installed-addons/total-bytes num-addons-installed-bytes} + + (github-stats!)))) + +(defn-spec update-stats! nil? + "summarises application state and updates a map of stats" + [] + (swap! state assoc :db-stats (app-stats (get-state))) + nil) + +(defn-spec watch-stats! nil? + "attaches a listener to sections of the state so stats are updated as they happen" [] - (let [version-running (strongbox-version) - latest-release (or (latest-strongbox-release) version-running) - sorted-asc (utils/sort-semver-strings [latest-release version-running])] - (= version-running (last sorted-asc)))) + (let [watch-these-paths [[:db] + [:installed-addon-list] + [:user-catalogue-idx]]] + (doseq [path watch-these-paths] + (state-bind path (fn [_] + (try + (update-stats!) + (catch Exception e + (error e "uncaught exception updating stats")))))))) ;; import/export @@ -1409,9 +1616,7 @@ :invalid-data? nil-me :transform-map {:game-track keyword}}) - curse? (fn [addon] - (some-> addon :source (= "curseforge"))) - addon-list (remove curse? addon-list) + addon-list (remove addon/host-disabled? addon-list) full-data? (fn [addon] (utils/all (mapv #(contains? addon %) [:source :source-id :name]))) @@ -1476,8 +1681,32 @@ ;; seems like a good place to preserve the etag-db (save-settings!) + (scheduled-user-catalogue-refresh) + nil) +(defn-spec hard-refresh nil? + "unlike `refresh`, `hard-refresh` clears the http cache before checking for addon updates." + [] + ;; why can't we be more specific, like just the addons for the current addon-dir? + ;; the url used to 'expand' an addon from the catalogue isn't preserved. + ;; it may also change with the game track (tukui, historically) or not even exist (tukui, currently). + ;; a thorough inspection would be too much code. + ;; this also removes the etag cache. the etag db only applies to catalogues and downloaded zip files. + (delete-http-cache!) + (refresh)) + +;; move to core.clj? +(defn-spec half-refresh nil? + "like `refresh` but excludes reloading catalogues, focusing on re-reading installed addons, + matching them to the catalogue and reapplying host updates." + [] + (report "refresh") + (load-all-installed-addons) + (match-all-installed-addons-with-catalogue) + (check-for-updates) + (save-settings!)) + (defn refresh-check "given a set of directories, presumably new ones introduced during an update, checks to see if any are present at all in the current set of known directories and does a `refresh` if not. @@ -1491,7 +1720,7 @@ diff (clojure.set/difference new-dirs existing-dirs)] (when-not (empty? diff) (debug "diff found between new and old, full refresh required:" diff) - ;; todo: could this be a half-refresh? should half-refresh live in core.clj ? + ;; todo: could this be a half-refresh instead? (refresh)))) ;; todo: move to ui.cli @@ -1577,6 +1806,7 @@ (init-dirs) (prune-http-cache!) (load-settings! cli-opts) + (watch-stats!) state) diff --git a/src/strongbox/http.clj b/src/strongbox/http.clj index f2b5a1cc..97be0208 100644 --- a/src/strongbox/http.clj +++ b/src/strongbox/http.clj @@ -17,7 +17,7 @@ (:import [org.apache.commons.io.input CountingInputStream])) -(def expiry-offset-hours 1) ;; hours +(def ^:dynamic *expiry-offset-minutes* 60) (def ^:dynamic *cache* nil) (def ^:dynamic *default-pause* 1000) (def ^:dynamic *default-attempts* 3) @@ -89,7 +89,7 @@ "returns `true` if the last modification time on given file is before the expiry date of +N hours" [output-file] (when (and output-file (fs/exists? output-file)) - (not (utils/file-older-than output-file expiry-offset-hours)))) + (not (utils/file-older-than output-file *expiry-offset-minutes* :minutes)))) (defn-spec url-to-filename ::sp/file "safely encode a URI to something that can live cached on the filesystem" @@ -402,12 +402,12 @@ ;; (defn-spec prune-cache-dir nil? - "deletes files in the given `cache-dir` that are older than the `expiry-offset-hours`" + "deletes files in the given `cache-dir` that are older than the `expiry-offset-minutes`" [cache-dir ::sp/extant-dir] - (let [expiry-date (* 2 expiry-offset-hours)] + (let [expiry (+ *expiry-offset-minutes* 1)] (doseq [cache-file (fs/list-dir cache-dir) :when (and (fs/file? cache-file) - (utils/file-older-than (str cache-file) expiry-date))] + (utils/file-older-than (str cache-file) expiry :minutes))] (fs/delete cache-file) (debug "deleted expired cache file:" cache-file)))) diff --git a/src/strongbox/jfx.clj b/src/strongbox/jfx.clj index 9877c3a8..5b95fd30 100644 --- a/src/strongbox/jfx.clj +++ b/src/strongbox/jfx.clj @@ -2,7 +2,6 @@ (:require [me.raynes.fs :as fs] [clojure.pprint] - [clojure.set] [clojure.java.io :as io] ;;[clojure.core.cache :as cache] [clojure.string :refer [lower-case join capitalize replace] :rename {replace str-replace}] @@ -30,15 +29,14 @@ [core :as core]]) (:import [javafx.scene.text Font] - [java.util List Calendar Locale] + [java.util List Calendar] [javafx.util Callback] [javafx.scene.control TreeTableRow TableRow TextInputDialog Alert Alert$AlertType ButtonType] [javafx.scene.input MouseButton MouseEvent KeyEvent KeyCode] [javafx.stage Stage FileChooser FileChooser$ExtensionFilter DirectoryChooser Window WindowEvent] [javafx.application Platform] [javafx.scene Node] - [javafx.event Event] - [java.text NumberFormat])) + [javafx.event Event])) (defn load-font-from-resources [resource] @@ -67,13 +65,6 @@ (delete [_ component opts] (fx.lifecycle/delete fx.lifecycle/dynamic (:child component) opts)))) -(def user-locale (Locale/getDefault)) -(def ^java.text.NumberFormat number-formatter (NumberFormat/getNumberInstance user-locale)) - -(defn format-number - [^Integer n] - (.format number-formatter n)) - (def major-theme-map {:light {:base "#ececec" @@ -307,7 +298,7 @@ {:-fx-opacity "1" ;; a disabled button already has a greying effect applied :-fx-font-style "normal"} - [".version-column" ".installed-column" ".available-version-column"] + [".version-column" ".installed-column" ".available-version-column" ".size-column"] {:-fx-alignment "center-right" :-fx-text-overrun "leading-ellipsis"} @@ -360,7 +351,18 @@ :-fx-border-width "0 1 0 0" :-fx-border-color (colour :table-border) :-fx-text-overrun "word-ellipsis" - ":hover" {:-fx-background-color (colour :row-updateable-selected)}}}} + ":hover" {:-fx-background-color (colour :row-updateable-selected)}}} + + ".star-column:hover > .button" + {:-fx-text-fill (colour :star-hover)} + + ".star-column > .button" + {:-fx-padding "1 0" + :-fx-font-size "1.3em" + :-fx-text-fill (colour :star-unstarred) + + ".starred" + {:-fx-text-fill (colour :star-starred)}}} ;; ;; installed-addons tab @@ -499,19 +501,15 @@ ;; "#search-addons " - {".star-column:hover > .button" - {:-fx-text-fill (colour :star-hover)} - - ".star-column > .button" - {:-fx-padding "1 0" - :-fx-font-size "1.3em" - :-fx-text-fill (colour :star-unstarred) + {"#search-install-button" + {:-fx-min-width "90px" + :-fx-padding ".4em 1em"} - ".starred" - {:-fx-text-fill (colour :star-starred)}} - - "#search-install-button" - {:-fx-min-width "90px"} + "#search-text-field " + {:-fx-min-width "100px" + :-fx-padding ".4em .5em" + :-fx-background-radius "0" + :-fx-text-fill (colour :table-font-colour)} "#search-random-button" {:-fx-min-width "80px"} @@ -519,7 +517,7 @@ "#search-user-catalogue-button" {:-fx-font-weight "bold" :-fx-font-size "1.2em" - :-fx-padding "2 7 " + :-fx-padding "3 7 " ".starred" {:-fx-text-fill (colour :star-starred) ;; the yellow of the star doesn't stand out from the gray gradient behind it. @@ -528,16 +526,15 @@ :-fx-stroke-width ".2" :-fx-effect (str "dropshadow( gaussian , " (colour :star-starred) " , 10, 0.0 , 0 , 0 )")}}} + "#search-addon-hosts-list" + {:-fx-pref-height "2em"} + "#search-prev-button" {:-fx-min-width "80px"} "#search-next-button" {:-fx-min-width "70px"} - "#search-text-field " - {:-fx-min-width "100px" - :-fx-text-fill (colour :table-font-colour)} - "#search-selected-tag-bar" {:-fx-padding "0 0 10 10" :-fx-spacing "10" @@ -558,22 +555,21 @@ :-fx-alignment "center-left" "#status-bar-left" - {:-fx-padding "0 10" + {:-fx-padding "0 0 0 10px" :-fx-alignment "center-left" :-fx-pref-width 9999.0 - " > .text" {;; omg, wtf does 'fx-fill' work and not 'fx-text-fill' ??? - :-fx-fill (colour :table-font-colour)}} + " > .text" {:-fx-text-fill (colour :table-font-colour) + :-fx-padding "0 0 0 10"} + + " #more-stats" {:-fx-min-width 79}} "#status-bar-right" {:-fx-min-width "130px" ;; long enough to render "warnings (999)" - :-fx-padding "5px 12px 5px 0" + :-fx-padding "5px 10px 5px 0" :-fx-alignment "center-right"} - "#status-bar-right .button" - {:-fx-padding "4 10" - ;; doesn't look right when button is coloured. - ;;:-fx-background-radius "4" - :-fx-font-size "11px" + "#status-bar-right .toggle-button" + {:-fx-background-radius "4" ;; this isn't great but it's better than nothing. revisit when it makes more sense. ":armed" @@ -587,6 +583,24 @@ {:-fx-text-fill (colour :row-error-text) :-fx-base (colour :row-error)}} + ;; + ;; widgets + ;; + + ".table-view#key-vals .column-header .label" + {:-fx-font-style "normal"} ;; column *values*, not the column *header* should be italic + + ".table-view#key-vals .key-column" + {:-fx-alignment "center-right" + :-fx-padding "0 1em 0 0" + :-fx-font-style "italic"} + + ".table-view#key-vals .key-column.column-header .label" + {:-fx-alignment "center-right"} + + ".table-view#key-vals .val-column.column-header .label" + {:-fx-alignment "center-left"} + ;; ;; addon-detail ;; @@ -662,20 +676,6 @@ :-fx-alignment "bottom-right" :-fx-pref-width 9999.0} - ".table-view#key-vals .column-header .label" - {:-fx-font-style "normal"} ;; column *values*, not the column *header* should be italic - - ".table-view#key-vals .key-column" - {:-fx-alignment "center-right" - :-fx-padding "0 1em 0 0" - :-fx-font-style "italic"} - - ".table-view#key-vals .key-column.column-header .label" - {:-fx-alignment "center-right"} - - ".table-view#key-vals .val-column.column-header .label" - {:-fx-alignment "center-left"} - "#addon-detail-big-buttons" {:-fx-padding "2em 0" " .toggle-button" {:-fx-pref-width "100pc" @@ -1091,7 +1091,7 @@ {:fx/type :text :text (format "version %s" (core/strongbox-version))} {:fx/type :text - :text (format "version %s is now available to download!" (core/latest-strongbox-release)) + :text (format "version %s is now available to download!" (core/latest-strongbox-release!)) :managed (not (core/latest-strongbox-version?)) :visible (not (core/latest-strongbox-version?))} {:fx/type :hyperlink @@ -1128,7 +1128,7 @@ (when-not (empty? error-messages) (let [msg (message-list (format "warnings/errors while installing \"%s\"" label) error-messages)] (alert :warning msg {:wait? false})))) - (cli/half-refresh))) + (core/half-refresh))) ;; @@ -1199,101 +1199,162 @@ :text "↪ browse local files"})) (defn gui-column-map - "overrides and additional column information for the GUI. see `cli/column-map`." - [queue] - (let [-gui-column-map - {:expand-group {:label "" :min-width 25 :pref-width 25 :max-width 25 :cell-value-factory (constantly "")} - :browse-local {:min-width 135 :pref-width 143 :max-width 150 + "list of columns for the installed-addons-table that needs to be separately defined so a menu can be built. + called with no arguments, the various attached functions will probably fail." + ([] + (gui-column-map nil)) + ([context] + (let [queue (when context + (fx/sub-val context get-in [:app-state :job-queue])) + user-catalogue-idx (when context + (mapv utils/source-map (fx/sub-val context get-in [:app-state, :user-catalogue :addon-summary-list]))) + starred? (fn [row] + (if (or (not context) + (not (map? row))) + false + (utils/in? (utils/source-map row) user-catalogue-idx)))] + {:browse-local {:text "browse" + :min-width 135 :pref-width 143 :max-width 150 + :cell-value-factory identity + :cell-factory {:fx/cell-type :tree-table-cell + :describe (fn [row] + {:graphic (or (addon-fs-link (:dirname row)) + {:fx/type :label + :text (get row :dirname "")})})}} + + :source {:text "source" + :min-width 130 :pref-width 135 :max-width 145 + :cell-value-factory identity + :cell-factory {:fx/cell-type :tree-table-cell + :describe (fn [row] + (when (map? row) + {:graphic (href-to-hyperlink row)}))}} + + :source-id {:text "ID" + :min-width 60 :pref-width 150 + :cell-value-factory :source-id} + + :source-map-list {:text "other sources" + :cell-value-factory identity :cell-factory {:fx/cell-type :tree-table-cell :describe (fn [row] - {:graphic (or (addon-fs-link (:dirname row)) - {:fx/type :label - :text (get row :dirname "")})})} - :cell-value-factory identity} - :source {:min-width 130 :pref-width 135 :max-width 145 - :cell-factory {:fx/cell-type :tree-table-cell - :describe (fn [row] - {:graphic (href-to-hyperlink row)})} - :cell-value-factory identity} - :source-id {:min-width 60 :pref-width 150} - :source-map-list {:cell-factory {:fx/cell-type :tree-table-cell - :describe (fn [row] - (let [urls (for [source-map (:source-map-list row) - :let [url (cli/addon-source-map-to-url row source-map)] - :when (and url - (not (= (:source row) (:source source-map))))] - (href-to-hyperlink (assoc source-map :url url))) - urls (utils/nilable (vec urls))] - (if urls - {:graphic {:fx/type :h-box - :children urls}} - {:graphic {:fx/type :label - :text ""}})))} - :cell-value-factory identity} - - :name {:min-width 100 :pref-width 300} - :description {:min-width 150 :pref-width 450} - :tag-list {:min-width 200 :pref-width 300 :style-class ["tag-button-column"] + (let [urls (for [source-map (:source-map-list row) + :let [url (cli/addon-source-map-to-url row source-map)] + :when (and url + (not (= (:source row) (:source source-map))))] + (href-to-hyperlink (assoc source-map :url url))) + urls (utils/nilable (vec urls))] + (if urls + {:graphic {:fx/type :h-box + :children urls}} + {:graphic {:fx/type :label + :text ""}})))}} + + :name {:text "name" + :min-width 100 :pref-width 300 + :cell-value-factory (comp utils/no-new-lines :label)} + + :description {:text "description" + :min-width 150 :pref-width 450 + :cell-value-factory (comp utils/no-new-lines :description)} + + :dirsize {:text "size" + :min-width 80 :pref-width 80 + :cell-value-factory :dirsize + :cell-factory {:fx/cell-type :tree-table-cell + :describe (fn [bytes] + (when (number? bytes) + {:text (utils/filesize bytes)}))}} + + :starred {:text "" :menu-label "starred" + :min-width 50 :pref-width 50 :max-width 50 :style-class ["invisible-button-column" "star-column"] + :comparator (fn [a b] + (if (starred? a) 1 0)) + :cell-value-factory identity + :cell-factory {:fx/cell-type :tree-table-cell + :describe (fn [installed-addon] + (if (or (not (map? installed-addon)) + (addon/ignored? installed-addon) + (not (:matched? installed-addon))) + {:text ""} + (let [starred (starred? installed-addon) + f (if starred + cli/remove-summary-from-user-catalogue + cli/add-addon-to-user-catalogue)] + {:graphic (button (:star constants/glyph-map) + (async-handler (partial f installed-addon)) + {:style-class (if starred "starred" "unstarred")})})))}} + + :tag-list {:text "tags" + :min-width 200 :pref-width 300 :style-class ["tag-button-column"] + :cell-value-factory identity + :cell-factory {:fx/cell-type :tree-table-cell + :describe (fn [row] + {:graphic {:fx/type :h-box + :children (mapv (fn [tag] + (button (name tag) + (async-handler #(do (switch-tab! SEARCH-TAB) + (cli/search-add-filter :tag tag))) + {:tooltip (name tag)})) + (:tag-list row))}})}} + + :updated-date {:text "updated" + :min-width 100 :pref-width 130 :max-width 150 + :cell-value-factory :updated-date + :cell-factory {:fx/cell-type :tree-table-cell + :describe (fn [dt] + {:text (if-not (string? dt) "" (utils/format-dt dt))})}} + + :created-date {:text "created" + :min-width 90 :pref-width 110 :max-width 120 + :cell-value-factory :created-date + :cell-factory {:fx/cell-type :tree-table-cell + :describe (fn [dt] + ;; for some reason I'm getting the whole row here ... (:uber button column?)! + {:text (if-not (string? dt) "" (utils/format-dt dt))})}} + + :installed-version {:text "installed" :menu-label "installed version" + :min-width 100 :pref-width 175 :max-width 250 :style-class ["installed-column"] + :cell-value-factory :installed-version} + + :available-version {:text "available" :menu-label "available version" + :min-width 100 :pref-width 175 :max-width 250 :style-class ["available-version-column"] + :cell-value-factory cli/available-versions-v1} + + :combined-version {:text "version" :menu-label "installed+available version" + :min-width 100 :pref-width 175 :max-width 250 :style-class ["version-column"] + :cell-value-factory cli/available-versions-v2} + + :game-version {:text "WoW" :menu-label "game version (WoW)" + :min-width 70 :pref-width 70 :max-width 100 + :cell-value-factory identity + :cell-factory {:fx/cell-type :tree-table-cell + :describe (fn [row] + (let [text (some-> row :interface-version str utils/interface-version-to-game-version) + text (if-not (string? text) "" text)] + {:graphic {:fx/type fx.ext.node/with-tooltip-props + :props {:tooltip {:fx/type :tooltip + :text (-> text utils/patch-name (or "?")) + ;; the tooltip will be long and intrusive, make delay longer than typical. + :show-delay 400}} + :desc {:fx/type :label + :style-class ["table-cell"] + :text text}}}))}} + + :uber-button {:text "" :menu-label "über button" + :min-width 80 :pref-width 80 :max-width 120 :style-class ["invisible-button-column"] :cell-value-factory identity :cell-factory {:fx/cell-type :tree-table-cell :describe (fn [row] - {:graphic {:fx/type :h-box - :children (mapv (fn [tag] - (button (name tag) - (async-handler #(do (switch-tab! SEARCH-TAB) - (cli/search-add-filter :tag tag))) - {:tooltip (name tag)})) - (:tag-list row))}})}} - :created-date {:min-width 90 :pref-width 110 :max-width 120 - :cell-value-factory :created-date - :cell-factory {:fx/cell-type :tree-table-cell - :describe (fn [dt] - ;; for some reason I'm getting the whole row here ... (:uber button column?)! - {:text (if-not (string? dt) "" (utils/format-dt dt))})}} - :updated-date {:min-width 90 :pref-width 110 :max-width 120 - :cell-value-factory :updated-date - :cell-factory {:fx/cell-type :tree-table-cell - :describe (fn [dt] - {:text (if-not (string? dt) "" (utils/format-dt dt))})}} - :installed-version {:min-width 100 :pref-width 175 :max-width 250 :style-class ["installed-column"]} - :available-version {:min-width 100 :pref-width 175 :max-width 250 :style-class ["available-version-column"]} - :combined-version {:min-width 100 :pref-width 175 :max-width 250 :style-class ["version-column"]} - :game-version {:min-width 70 :pref-width 70 :max-width 100 - :cell-factory {:fx/cell-type :tree-table-cell - :describe (fn [text] - ;; for some reason I'm getting the whole row here - (let [text (if-not (string? text) "" text)] - {:graphic {:fx/type fx.ext.node/with-tooltip-props - :props {:tooltip {:fx/type :tooltip - :text (-> text utils/patch-name (or "?")) - ;; the tooltip will be long and intrusive, make delay longer than typical. - :show-delay 400}} - :desc {:fx/type :label - :style-class ["table-cell"] - :text text}}}))}} - - :uber-button {:min-width 80 :pref-width 80 :max-width 120 :style-class ["invisible-button-column"] - :cell-value-factory identity - :cell-factory {:fx/cell-type :tree-table-cell - :describe (fn [row] - (if (or (not row) - (not (map? row))) - ;; for some reason I'm getting the contents of the :created-date column here - {:text ""} - - (let [job-id (joblib/addon-id row)] - {:graphic (if (and (core/unsteady? (:name row)) - (joblib/has-job? queue job-id)) - (addon-progress-bar row queue job-id) - (uber-button row))})))}}} - - ;; rename some UI column keys and then merge with the gui columns - merge-ui-gui-columns (fn [[key val]] - [key (merge - (clojure.set/rename-keys val {:label :text, :value-fn :cell-value-factory}) - (get -gui-column-map key))]) - column-map (->> cli/column-map (map merge-ui-gui-columns) (into {}))] - column-map)) + (if (not (map? row)) + ;; for some reason I'm getting the contents of the :created-date column here + {:text ""} + ;; else + (let [job-id (joblib/addon-id row)] + {:graphic (if (and (core/unsteady? (:name row)) + (joblib/has-job? queue job-id)) + (addon-progress-bar row queue job-id) + (uber-button row))})))}}}))) (defn-spec make-table-column map? "returns a description of a table column that lives within a table." @@ -1379,18 +1440,25 @@ (let [^javafx.scene.control.CheckMenuItem menu-item (.getSource ev)] (cli/set-preference :addon-zips-to-keep (if (.isSelected menu-item) 0 nil))))})) +(defn menu-item--keep-user-catalogue-updated + [{:keys [fx/context]}] + (let [selected? (fx/sub-val context get-in [:app-state :cfg :preferences :keep-user-catalogue-updated])] + {:fx/type :check-menu-item + :text "Keep user catalogue updated" + :selected selected? + :on-action (fn [_] + (cli/set-preference :keep-user-catalogue-updated (not selected?)))})) (defn-spec build-column-menu ::sp/list-of-maps "returns a list of columns that are 'selected' if present in `selected-column-list`." [selected-column-list :ui/column-list] - (let [column-list (cli/sort-column-list (keys cli/column-map)) - queue nil - gui-column-map (gui-column-map queue) + (let [gui-column-map (gui-column-map) + column-list (cli/sort-column-list (keys gui-column-map)) toggle-column-menu-item (fn [column-id] (let [column (column-id gui-column-map)] {:fx/type :check-menu-item - :text (or (:text column) (name column-id)) + :text (or (:menu-label column) (:text column) (name column-id)) :selected (utils/in? column-id selected-column-list) :on-action (async-event-handler #(cli/toggle-ui-column column-id (-> % .getTarget .isSelected)))})) @@ -1412,6 +1480,7 @@ no-addon-dir? (nil? addon-dir) selected-theme (fx/sub-val context get-in [:app-state :cfg :gui-theme]) selected-columns (fx/sub-val context get-in [:app-state :cfg :preferences :ui-selected-columns]) + split-pane (fx/sub-val context get-in [:app-state :gui-split-pane]) file-menu [(menu-item "Install addon from file" (async-handler zip-file-picker) {:disable no-addon-dir?}) (menu-item "Import addon" (async-handler import-addon-handler) @@ -1437,10 +1506,11 @@ separator (menu-item "E_xit" exit-handler {:key "Ctrl+Q"})] - prefs-menu [{:fx/type menu-item--num-zips-to-keep}] + prefs-menu [{:fx/type menu-item--num-zips-to-keep} + {:fx/type menu-item--keep-user-catalogue-updated}] view-menu (into - [(menu-item "Refresh" (async-handler cli/hard-refresh) {:key "F5"}) + [(menu-item "Refresh" (async-handler core/hard-refresh) {:key "F5"}) separator (menu-item "_Installed" (switch-tab-event-handler INSTALLED-TAB) {:key "Ctrl+I"}) (menu-item "Searc_h" (switch-tab-event-handler SEARCH-TAB) @@ -1458,7 +1528,12 @@ (fx/sub-val context get-in [:app-state :cfg :selected-catalogue]) (fx/sub-val context get-in [:app-state :cfg :catalogue-location-list])) [separator - (menu-item "Refresh user catalogue" (async-handler cli/refresh-user-catalogue))]) + (menu-item "Refresh user catalogue" (async-handler + (fn [] + (if split-pane + (cli/set-sub-pane :notice-logger) + (switch-tab! LOG-TAB)) + (cli/refresh-user-catalogue))))]) cache-menu [(menu-item "Clear http cache" (async-handler core/delete-http-cache!)) (menu-item "Clear addon zips" (async-handler core/delete-downloaded-addon-zips!) @@ -1542,7 +1617,9 @@ (defn installed-addons-menu-bar "returns a description of the installed-addons tab-pane menu." [{:keys [fx/context]}] - (let [selected-addon-dir (fx/sub-val context get-in [:app-state :cfg :selected-addon-dir])] + (let [selected-addon-dir (fx/sub-val context get-in [:app-state :cfg :selected-addon-dir]) + latest-release (fx/sub-val context get-in [:app-state :latest-strongbox-release]) + latest-version? (core/latest-strongbox-version? latest-release)] {:fx/type :h-box :padding 10 :spacing 10 @@ -1554,10 +1631,10 @@ {:fx/type wow-dir-dropdown} {:fx/type game-track-dropdown} {:fx/type :button - :text (str "Update Available: " (core/latest-strongbox-release)) + :text (str "Update Available: " latest-release) :on-action (handler #(utils/browse-to "https://github.com/ogri-la/strongbox/releases")) - :visible (not (core/latest-strongbox-version?)) - :managed (not (core/latest-strongbox-version?))}]})) + :visible (not latest-version?) + :managed (not latest-version?)}]})) (defn-spec build-release-menu ::sp/list-of-maps "returns a list of `:menu-item` maps that will update the given `addon` with @@ -1664,22 +1741,25 @@ [{:keys [fx/context]}] (fx/sub-val context get-in [:app-state :unsteady-addon-list]) ;; re-render table when addons become unsteady (fx/sub-val context get-in [:app-state :log-lines]) ;; re-render rows when addons emit warnings or errors - (let [queue (fx/sub-val context get-in [:app-state :job-queue]) - row-list (fx/sub-val context get-in [:app-state :installed-addon-list]) + (let [row-list (fx/sub-val context get-in [:app-state :installed-addon-list]) selected (fx/sub-val context get-in [:app-state :selected-addon-list]) selected-addon-dir (fx/sub-val context get-in [:app-state :cfg :selected-addon-dir]) user-selected-column-list (cli/sort-column-list (fx/sub-val context get-in [:app-state :cfg :preferences :ui-selected-columns])) - ;; can't be part of the column map because it's actually attached to the row. - ;; this is just a spacer so the arrow always has room and isn't overlapped by another column's values. - arrow-column {:fx/type :tree-table-column :cell-value-factory (constantly "") - :min-width 25 :max-width 25 :resizable false} + ;; can't be part of the column map because the arrow glyph is actually embedded in the row. + ;; this is just a spacer so the arrow always has a fixed amount of room and is not overlapped by other columns. + arrow-column {:text "" + :fx/type :tree-table-column + :min-width 25 :max-width 25 :resizable false + :cell-value-factory (constantly "")} selected-columns (or user-selected-column-list sp/default-column-list) - column-list (utils/select-vals (gui-column-map queue) selected-columns) + column-list (utils/select-vals (gui-column-map context) selected-columns) column-list (mapv make-tree-table-column column-list) + ;; the column list can be empty if the user removes all columns! column-list (if-not (empty? column-list) (into [arrow-column] column-list) []) + column-list (mapv #(dissoc % :menu-label) column-list) ;; wraps the list of addons in a :`tree-item` component to model the parent->child relationship. row-list (mapv (fn [row] @@ -1937,7 +2017,7 @@ :cell-factory {:fx/cell-type :table-cell :describe (fn [n] (when n - {:text (format-number n)}))}} + {:text (utils/format-number n)}))}} {:text "" :style-class ["wide-button-column"] :min-width 120 :pref-width 120 :max-width 120 :resizable false :cell-factory {:fx/cell-type :table-cell :describe (fn [addon] @@ -1974,8 +2054,7 @@ (defn search-addons-search-field [{:keys [fx/context]}] (let [search-state (fx/sub-val context get-in [:app-state, :search]) - ;;known-host-list (fx/sub-val context get-in [:app-state, :db-stats :known-host-list]) - known-host-list (core/get-state :db-stats :known-host-list) + known-host-list (or (fx/sub-val context get-in [:app-state :db-stats :addons/known-host-list]) []) disable-host-selector? (= 1 (count known-host-list)) tag-set (->> search-state :filter-by :tag) @@ -2024,6 +2103,7 @@ :title "addon host" :items known-host-list :show-checked-count false + :id "search-addon-hosts-list" :on-checked-items-changed (fn [val] (cli/search-add-filter :source val)) :disable disable-host-selector?} @@ -2033,6 +2113,22 @@ ;; :text "random" ;; :on-action (handler cli/random-search)} + {:fx/type :button + :id "search-clear-button" + :text "clear" + :on-action (fn [_] + (when-let [ccb (first (select "#search-addon-hosts-list"))] + (.clearChecks (.getCheckModel ccb))) + (cli/clear-search!)) + :disable (let [ss (dissoc search-state :results) + as (dissoc core/-search-state-template :results)] + (if (clojure.string/blank? (:term ss)) + ;; we use alternating `" "` and `nil` to 'bump' search results. + ;; if we have one of those, don't consider the search term. + (= (dissoc ss :term) + (dissoc as :term)) + (= ss as)))} + {:fx/type :h-box :id "spacer" :h-box/hgrow :ALWAYS} @@ -2111,11 +2207,8 @@ (defn addon-detail-key-vals-widget "displays a two-column table of `key: val` fields for what we know about an addon." [{:keys [addon]}] - (let [key-col (fn [keypair] - ;; shouldn't ever be nil but better safe than sorry - (-> keypair :key (or ":nil") str (subs 1))) - column-list [{:text "key" :min-width 150 :pref-width 150 :max-width 200 :resizable false :cell-value-factory key-col} - {:text "val" :cell-value-factory :val}] + (let [column-list [{:text "key" :min-width 220 :pref-width 250 :max-width 300 :resizable false :cell-value-factory (comp utils/pretty-print-keyword :key)} + {:text "val" :cell-value-factory (comp utils/pretty-print-value :val)}] blacklist [:group-addons :release-list :source-map-list] sanitised (apply dissoc addon blacklist) @@ -2481,7 +2574,10 @@ (let [log-lines (fx/sub-val context get-in [:app-state :log-lines]) log-lines (cli/log-entries-since-last-refresh log-lines) - toggle (fx/sub-val context get-in [:app-state :gui-split-pane]) + split-pane-state (fx/sub-val context get-in [:app-state :gui-split-pane]) + sub-pane-selected (fx/sub-val context get-in [:app-state :gui-sub-pane]) + selected? (and split-pane-state + (= sub-pane-selected :notice-logger)) ;; {:warn 1, :info 20} stats (utils/count-occurances log-lines :level) @@ -2515,46 +2611,89 @@ :show-delay 400}} :desc {:fx/type :toggle-button :text lbl - :selected (boolean toggle) + :selected selected? :style-class (utils/items ["toggle-button" (cond has-errors? "with-error" has-warnings? "with-warning")]) :on-selected-changed (async-handler (fn [] - (cli/toggle-split-pane) + (cli/set-sub-pane :notice-logger) + (cli/set-split-pane (not selected?)) (cli/change-notice-logger-level max-level)))}})) -(defn status-bar - "this is the litle strip of text at the bottom of the application." +(defn key-vals-widget + "general purpose two-column table intended for two-column key+val data." + [{:keys [column-list row-list placeholder]}] + (let [row-list (or row-list [])] + {:fx/type :table-view + :id "key-vals" + :style-class ["table-view"] + :placeholder {:fx/type :text + :style-class ["table-placeholder-text"] + :text (or placeholder "")} + :column-resize-policy javafx.scene.control.TableView/CONSTRAINED_RESIZE_POLICY + :columns (mapv make-table-column column-list) + :items row-list})) + +(defn stats-button [{:keys [fx/context]}] - (let [num-matching-template "%s of %s installed addons found in catalogue." - all-matching-template "all installed addons found in catalogue." - catalogue-count-template "%s addons in catalogue." + (let [split-pane-state (fx/sub-val context get-in [:app-state :gui-split-pane]) + sub-pane-selected (fx/sub-val context get-in [:app-state :gui-sub-pane]) + selected? (and split-pane-state + (= sub-pane-selected :stats))] - ia (fx/sub-val context get-in [:app-state :installed-addon-list]) + {:fx/type fx.ext.node/with-tooltip-props + :props {:tooltip {:fx/type :tooltip + :text "tooltip" + :show-delay 400}} + :desc {:fx/type :toggle-button + :id "more-stats" + :text "more stats" + :selected selected? + :style-class ["toggle-button"] + :on-selected-changed (async-handler (fn [] + (cli/set-sub-pane :stats) + (cli/set-split-pane (not selected?))))}})) - uia (filter :matched? ia) +(defn status-bar + "this is the litle strip of text at the bottom of the application." + [{:keys [fx/context]}] + (let [stats (fx/sub-val context get-in [:app-state :db-stats]) - a-count (count (fx/sub-val context get-in [:app-state :db])) - ia-count (count ia) - uia-count (count uia) + ;; don't show the message if we haven't finished loading yet + hide-status? (or (zero? (get stats :addons/total 0)) + (zero? (get stats :installed-addons/total 0))) - strings [(format catalogue-count-template (format-number a-count)) - (if (= ia-count uia-count) - all-matching-template - (format num-matching-template uia-count ia-count))]] + catalogue-count-status (format "%s addons in catalogue, %s addons installed, %s addons matched to catalogue" + (get stats :addons/total 0) + (get stats :installed-addons/total 0) + (get stats :installed-addons/num-matched 0))] {:fx/type :h-box :id "status-bar" :children [{:fx/type :h-box :id "status-bar-left" - :children [{:fx/type :text + :children [{:fx/type stats-button} + {:fx/type :label :style-class ["text"] - :text (join " " strings)}]} + :text (if hide-status? "" catalogue-count-status)}]} {:fx/type :h-box :id "status-bar-right" :children [{:fx/type split-pane-button}]}]})) +(defn db-stats-widget + "an instance of the `key-vals-widget` for displaying the `:db-stats` data" + [{:keys [fx/context]}] + (let [db-stats (fx/sub-val context get-in [:app-state :db-stats]) + key-vals (sort-by :key (mapv (fn [[key val]] {:key key :val val}) db-stats))] + {:fx/type key-vals-widget + :column-list [{:text "" :min-width 250 :pref-width 270 :max-width 290 + :style-class ["key-column"] + :cell-value-factory (comp utils/pretty-print-keyword :key)} + {:text "" :cell-value-factory (comp utils/pretty-print-value :val) + :style-class ["val-column"]}] + :row-list key-vals})) + ;; (defn app @@ -2565,7 +2704,12 @@ style (fx/sub-val context get :style) showing? (fx/sub-val context get-in [:app-state :gui-showing?]) theme (fx/sub-val context get-in [:app-state :cfg :gui-theme]) - split-pane-on? (fx/sub-val context get-in [:app-state :gui-split-pane])] + split-pane-on? (fx/sub-val context get-in [:app-state :gui-split-pane]) + default-sub-pane :notice-logger + sub-pane (or (fx/sub-val context get-in [:app-state :gui-sub-pane]) + default-sub-pane) + -notice-logger {:fx/type notice-logger} + -db-stats {:fx/type db-stats-widget}] {:fx/type :stage :showing showing? :on-close-request exit-handler @@ -2589,13 +2733,13 @@ :root {:fx/type :border-pane :id (name theme) :top {:fx/type menu-bar} - :center (if split-pane-on? - {:fx/type :split-pane - :orientation :vertical - :divider-positions [0.6] - :items [{:fx/type tabber} - {:fx/type notice-logger}]} - {:fx/type tabber}) + :center {:fx/type :split-pane + :orientation :vertical + :divider-positions (if split-pane-on? [0.6] [1]) + :items [{:fx/type tabber} + {:fx/type :v-box + :managed split-pane-on? + :children [(if (= sub-pane :notice-logger) -notice-logger -db-stats)]}]} :bottom {:fx/type status-bar}}}})) (defn start @@ -2621,12 +2765,6 @@ (swap! gui-state fx/swap-context assoc :style (style)))) (core/add-cleanup-fn #(remove-watch rf key))) - ;; logging to app state for use in the UI - _ (cli/init-ui-logger) - - ;; asynchronous searching. as the user types, update the state with search results asynchronously - _ (cli/-init-search-listener) - renderer (fx/create-renderer :middleware (comp fx/wrap-context-desc @@ -2641,6 +2779,7 @@ ;; don't do this, renderer has to be unmounted and the app closed before further state changes happen during cleanup. ;;_ (core/add-cleanup-fn #(fx/unmount-renderer gui-state renderer)) + _ (swap! core/state assoc :disable-gui (fn [] (fx/unmount-renderer gui-state renderer) ;; the slightest of delays allows any final rendering to happen before the exit-handler is called. @@ -2660,9 +2799,19 @@ ;; happens during testing and causes a few weird windows to hang around. ;; see `(run! (fn [_] (test :jfx)) (range 0 100))` (let [kick (future - (set-icon) + ;; roughly follows `cli/start` + + ;; logging to app state for use in the UI + (cli/init-ui-logger) + ;; asynchronous searching. as the user types, update the state with search results asynchronously + (cli/-init-search-listener) + (core/refresh) - (bump-search))] + + (bump-search) + (core/latest-strongbox-release!) + (set-icon) ;; 601ms :( + )] (core/add-cleanup-fn #(future-cancel kick))) ;; calling the `renderer` will re-render the GUI. diff --git a/src/strongbox/main.clj b/src/strongbox/main.clj index 9ca9c554..5375e15c 100644 --- a/src/strongbox/main.clj +++ b/src/strongbox/main.clj @@ -1,24 +1,17 @@ (ns strongbox.main - (:refer-clojure :rename {test clj-test}) (:require [taoensso.timbre :as timbre :refer [spy info warn error report]] - [clojure.test] [clojure.tools.cli] - [clojure.tools.namespace.repl :as tn :refer [refresh]] [clojure.string :refer [lower-case]] [me.raynes.fs :as fs] [strongbox - [catalogue :as catalogue] - [http :as http] - ;;[joblib :as joblib] [core :as core] [utils :as utils :refer [in?]] ;; warning! requiring cljfx starts the javafx application thread. ;; this is a pita for exiting a non-javafx UI (cli) and aot as it just 'hangs'. ;; hanging aot is handled in project.clj, but dynamic inclusion of jfx is handled here. ;;[jfx :as jfx] - [cli :as cli]] - [gui.diff :refer [with-gui-diff]]) + [cli :as cli]]) (:gen-class)) (Thread/setDefaultUncaughtExceptionHandler @@ -78,56 +71,12 @@ (Thread/sleep 750) ;; gives me time to switch panes (start cli-opts)) -(defn test - [& [ns-kw fn-kw]] - (stop) - (clojure.tools.namespace.repl/refresh) ;; reloads all namespaces, including strongbox.whatever-test ones - (utils/instrument true) ;; always test with spec checking ON - - (try - ;; note! remember to update `cloverage.clj` with any new bindings - (with-redefs [core/*testing?* true - http/*default-pause* 1 ;; ms - http/*default-attempts* 1 - ;; don't pause while testing. nothing should depend on that pause happening. - ;; note! this is different to `joblib/tick-delay` not delaying when `joblib/*tick*` is unbound. - ;; tests still bind `joblib/*tick*` and run things in parallel. - ;;joblib/tick-delay joblib/*tick* - ;;main/*spec?* true - ;;cli/install-update-these-in-parallel cli/install-update-these-serially - ;;core/check-for-updates core/check-for-updates-serially - ;; for testing purposes, no addon host is disabled - catalogue/host-disabled? (constantly false)] - (core/reset-logging!) - - (if ns-kw - (if (some #{ns-kw} [:main :utils :http - :core :toc :nfo :zip :config :catalogue :addon :logging :joblib - :cli :gui :jfx - :curseforge-api :wowinterface-api :gitlab-api :github-api :tukui-api - :release-json]) - (with-gui-diff - (if fn-kw - ;; `test-vars` will run the test but not give feedback if test passes OR test not found - ;; slightly better than nothing - (clojure.test/test-vars [(resolve (symbol (str "strongbox." (name ns-kw) "-test") (name fn-kw)))]) - (clojure.test/run-all-tests (re-pattern (str "strongbox." (name ns-kw) "-test"))))) - (error "unknown test file:" ns-kw)) - (clojure.test/run-all-tests #"strongbox\..*-test"))) - (finally - ;; use case: we run the tests from the repl and afterwards we call `restart` to start the app. - ;; `stop` inside `restart` will be outside of `with-redefs` and still have logging `:min-level` set to `:debug` - ;; it will dump a file and yadda yadda. - (core/reset-logging!)))) - ;; (defn usage [parsed-opts] (str "Usage: ./strongbox [--action] [--addons-dir]\n\n" (:summary parsed-opts))) -;; - (def cli-options [["-h" "--help"] diff --git a/src/strongbox/specs.clj b/src/strongbox/specs.clj index afdd2a29..433bc9fb 100644 --- a/src/strongbox/specs.clj +++ b/src/strongbox/specs.clj @@ -42,6 +42,8 @@ (s/def ::atom #(instance? clojure.lang.Atom %)) +(s/def ::gte-zero #(and (number? %) (>= % 0))) + (s/def ::list-of-strings (s/coll-of string?)) (s/def ::list-of-maps (s/coll-of map?)) (s/def ::list-of-keywords (s/coll-of keyword?)) @@ -108,6 +110,7 @@ (s/def ::ignore-flag (s/keys :req-un [::ignore?])) (s/def ::download-url ::url) (s/def ::dirname (s/and string? #(not (empty? %)))) ;; and doesn't contain any '/' characters +(s/def ::dirsize (s/and int? #(>= % 0))) (s/def ::description (s/nilable string?)) (s/def ::matched? boolean?) (s/def ::group-id string?) @@ -121,6 +124,8 @@ (s/def ::label string?) ;; name of the addon without normalisation (s/def ::release-label ::label) +(s/def ::latest-strongbox-release (s/or :set string? :not-set nil? :failed keyword?)) + ;; dates and times (s/def ::inst (s/and string? #(try @@ -133,6 +138,7 @@ false)))) (s/def ::zoned-dt-obj #(instance? java.time.ZonedDateTime %)) +(s/def ::local-dt-obj #(instance? java.time.LocalDateTime %)) ;; javafx, cljfx, gui ;; no references to cljfx or javafx please! @@ -140,6 +146,7 @@ (s/def :javafx/node #(instance? javafx.scene.Node %)) (s/def :gui/column-data (s/keys :opt-un [:gui/text :gui/cell-value-factory :gui/style-class])) +(s/def :gui/sub-pane-content #{:notice-logger :stats}) (s/def :addon/id (s/or :regular (s/keys :req-un [:addon/source :addon/source-id]) ;; installed addons and catalogue addons :edge (s/keys :req-in [::dirname]))) ;; unmatched and ignored addons @@ -155,15 +162,40 @@ ;; defined here to prevent coupling between cli.clj and config.clj (I guess?) ;; all known columns. also constitutes the column order. -(def known-column-list [:browse-local :source :source-id :source-map-list :name :description :tag-list :created-date :updated-date :installed-version :available-version :combined-version :game-version :uber-button]) +;; arrow column/group expander always comes first +(def known-column-list + [:starred :browse-local + :source :source-id :source-map-list + :name :description :tag-list + :created-date :updated-date :dirsize + :installed-version :available-version :combined-version :game-version + :uber-button]) ;; default set of columns -(def default-column-list--v1 [:source :name :description :installed-version :available-version :game-version :uber-button]) -(def default-column-list--v2 [:source :name :description :combined-version :game-version :uber-button]) +(def default-column-list--v1 + [:source + :name :description + :installed-version :available-version :game-version + :uber-button]) +(def default-column-list--v2 + [:source + :name :description + :combined-version :game-version + :uber-button]) (def default-column-list default-column-list--v2) -(def skinny-column-list [:name :version :combined-version :game-version :uber-button]) -(def fat-column-list [:browse-local :source :source-id :name :description :tag-list :created-date :updated-date :combined-version :game-version :uber-button]) +(def skinny-column-list + [:name + :version :combined-version :game-version + :uber-button]) + +(def fat-column-list + [:starred :browse-local + :source :source-id + :name :description :tag-list + :created-date :updated-date :dirsize + :installed-version :available-version :game-version + :uber-button]) (def column-preset-list [[:default default-column-list] [:skinny skinny-column-list] @@ -178,11 +210,13 @@ (s/def ::addon-dir-map (s/keys :req-un [::addon-dir :addon-dir/game-track ::strict?])) (s/def ::addon-dir-list (s/coll-of ::addon-dir-map)) (s/def ::selected-catalogue keyword?) +(s/def ::keep-user-catalogue-updated boolean?) (s/def :config/addon-zips-to-keep (s/nilable int?)) (s/def :config/ui-selected-columns :ui/column-list) (s/def :config/preferences (s/keys :req-un [:config/addon-zips-to-keep - (s/nilable :config/ui-selected-columns)])) + (s/nilable :config/ui-selected-columns) + :config/keep-user-catalogue-updated])) (s/def ::user-config (s/keys :req-un [::addon-dir-list ::selected-addon-dir ::catalogue-location-list ::selected-catalogue @@ -236,6 +270,7 @@ (s/def :addon/category string?) (s/def :addon/category-list (s/coll-of :addon/category)) +(def tukui-source-list #{"tukui" "tukui-classic" "tukui-classic-tbc" "tukui-classic-wotlk"}) (s/def :addon/source (s/or :known #{"curseforge" "wowinterface" "github" "gitlab" "tukui" "tukui-classic" "tukui-classic-tbc" "tukui-classic-wotlk"} :unknown string?)) (s/def :addon/source-id (s/or ::integer-id? int? ;; tukui has negative ids @@ -257,7 +292,8 @@ ::interface-version ::installed-version :addon/supported-game-tracks] - :opt-un [;; toc file may contain addon host information but it's not guaranteed. + :opt-un [::dirsize ;; not present on error during calculation. zero during testing. + ;; toc file may contain addon host information but it's not guaranteed. :addon/source :addon/source-id :addon/source-map-list])) @@ -453,3 +489,16 @@ ;; search (s/def :search/filter-by #{:source :tag :tag-membership :user-catalogue}) + +;; github + +(s/def :github/requests-used ::gte-zero) +(s/def :github/remaining ::gte-zero) +(s/def :github/requests-limit-reset-minutes number?) +(s/def :github/requests-limit ::gte-zero) +(s/def :github/token-set? boolean?) +(s/def :github/requests-stats (s/keys :req [:github/token-set? + :github/requests-limit + :github/requests-limit-reset-minutes + :github/requests-remaining + :github/requests-used])) diff --git a/src/strongbox/toc.clj b/src/strongbox/toc.clj index 2e2b3d25..d713347d 100644 --- a/src/strongbox/toc.clj +++ b/src/strongbox/toc.clj @@ -80,6 +80,7 @@ (mapv (fn [[filename-game-track filename]] (merge (read-toc-file (utils/join toc-dir filename)) {:dirname (fs/base-name addon-dir) ;; /foo/bar/baz => baz + :dirsize (utils/folder-size-bytes addon-dir) :-filename filename :-filename-game-track filename-game-track})) (find-toc-files toc-dir)))) @@ -172,6 +173,10 @@ ;;:required-dependencies (:requireddeps keyvals) } + addon (if-let [dirsize (:dirsize keyvals)] + (assoc addon :dirsize dirsize) + addon) + ;; prefers tukui over wowi, wowi over github. I'd like to prefer github over wowi, but github ;; requires API calls to interact with and these are limited unless authenticated. addon (merge addon diff --git a/src/strongbox/utils.clj b/src/strongbox/utils.clj index bd8bc335..bda13215 100644 --- a/src/strongbox/utils.clj +++ b/src/strongbox/utils.clj @@ -17,10 +17,13 @@ [java-time :as jt] [java-time.format]) (:import + [java.util List Calendar Locale] + [java.lang Math] [java.util Base64] [org.apache.commons.compress.compressors CompressorStreamFactory CompressorException] [org.ocpsoft.prettytime.units Decade] - [org.ocpsoft.prettytime PrettyTime])) + [org.ocpsoft.prettytime PrettyTime] + [java.text NumberFormat])) (defn repl-stack-element? [stack-element] @@ -103,18 +106,6 @@ (warn (format "deleting %s %s files" (count file-list) file-type)) (dorun (map (juxt alert fs/delete) file-list)))))))) -(defn-spec file-older-than boolean? - [file ::sp/extant-file, hours pos-int?] - (let [modtime (jt/instant (fs/mod-time file)) - now (java-time/instant) - expiry-offset (jt/hours hours) - expiry-date (jt/plus modtime expiry-offset) - expired? (jt/before? expiry-date now)] - (when expired? - ;; too noisy even for :debug when nothing has expired - (debug (format "path %s; modtime %s; expiry-offset %s; expiry-date %s; now %s; expired? %s" file modtime expiry-offset expiry-date now expired?))) - expired?)) - #_(defn-spec days-between-then-and-now int? [datestamp ::sp/inst] (let [then (java-time/local-date datestamp) @@ -126,16 +117,34 @@ (.format (java.text.SimpleDateFormat. "yyyy-MM-dd") (java.util.Date.))) (defn-spec todt ::sp/zoned-dt-obj - "takes an ISO8901 string and returns a java.time.ZonedDateTime object. - these are needed to calculate durations" + "takes an ISO8601 string and returns a java.time.ZonedDateTime object. + if the date is missing it's time portion, it's assumed to be `T00:00:00Z`." [dt ::sp/inst] - (java-time/zoned-date-time (get java-time.format/predefined-formatters "iso-zoned-date-time") dt)) + (let [dt (if (-> dt count (= 10)) (str dt "T00:00:00Z") dt)] + (java-time/zoned-date-time (get java-time.format/predefined-formatters "iso-zoned-date-time") dt))) (defn-spec dt-before? boolean? "returns `true` if `date-1` happened before `date-2`" [date-1 ::sp/inst, date-2 ::sp/inst] (jt/before? (todt date-1) (todt date-2))) +(defn-spec older-than? boolean? + "returns `true` if the `period` (hours, days) between *now* and `then` is greater than `threshold`." + [then ::sp/inst, threshold ::sp/gte-zero, period keyword?] + (let [expiry-offset + (case period + :minutes (jt/minutes threshold) + :hours (jt/hours threshold) + :days (jt/days threshold)) + now (java-time/instant) + expiry-date (jt/plus (jt/instant (todt then)) expiry-offset)] + (jt/before? expiry-date now))) + +(defn-spec file-older-than boolean? + "returns `true` if given `file` has a modification date older than given `hours`." + [file ::sp/extant-file, threshold ::sp/gte-zero, period keyword?] + (-> file fs/mod-time jt/instant str (older-than? threshold period))) + (defn-spec published-before-classic? (s/or :ok boolean?, :error nil?) [dt-string (s/nilable ::sp/inst)] (try @@ -749,39 +758,40 @@ (def -with-lock-lock (Object.)) +(def -with-lock-wait-retry-time 10) ;; ms + (defmacro with-lock "executes `form` once all items in given `user-set` are available in `lock-set-atom`." [lock-set-atom user-set & form] - `(let [wait-time# 10] ;; ms - (loop [waited# 0] + `(loop [waited# 0] ;; ensure reading the atom is single threaded (locking) and that when we read it and test the result, ;; we update it in the same operation (dosync). - (let [lock-set# - (locking -with-lock-lock - (dosync - (debug "current locks:" (deref ~lock-set-atom)) - (when (empty? (clojure.set/intersection (deref ~lock-set-atom) ~user-set)) + (let [lock-set# + (locking -with-lock-lock + (dosync + (debug "current locks:" (deref ~lock-set-atom)) + (when (empty? (clojure.set/intersection (deref ~lock-set-atom) ~user-set)) ;; there is no overlap between the locks we have and what the user wants. ;; add the user locks to the working set and execute body - (swap! ~lock-set-atom into ~user-set))))] - - (debug "acquiring locks:" ~user-set) - (if (not (nil? lock-set#)) - (try - (debug "locks acquired:" ~user-set) - ~@form - (finally + (swap! ~lock-set-atom into ~user-set))))] + + (debug "acquiring locks:" ~user-set) + (if (not (nil? lock-set#)) + (try + (debug "locks acquired:" ~user-set) + ~@form + (finally ;; when body is complete, release the locks ;; synchronised access not required ? - (debug "releasing locks:" ~user-set) - (swap! ~lock-set-atom clojure.set/difference ~user-set))) + (debug "releasing locks:" ~user-set) + (swap! ~lock-set-atom clojure.set/difference ~user-set))) ;; something else holds one or more of the desired locks! wait a duration and try again - (do (debug "blocked!") - (Thread/sleep wait-time#) - (debug (format "recurring in %s ms, have waited %s ms" wait-time# waited#)) - (recur (+ waited# wait-time#)))))))) + (do (debug "blocked!") + (Thread/sleep -with-lock-wait-retry-time) + (debug (format "recurring in %s ms, have waited %s ms" -with-lock-wait-retry-time waited#)) + (recur (+ waited# -with-lock-wait-retry-time))))))) (defn-spec patch-name (s/or :ok string?, :not-found nil?) "returns the 'patch' name for the given `game-version`, considering only the major and minor values. @@ -794,27 +804,119 @@ (or (get constants/releases major-minor) (get constants/releases major)))) -(defn-spec compressed-slurp (s/or :ok bytes?, :no-resource nil?) - "returns the bz2 compressed bytes of the given resource file `resource`. - returns `nil` if the file can't be found." - [resource string?] - (let [input-file (clojure.java.io/resource resource)] - (when input-file - (with-open [out (java.io.ByteArrayOutputStream.)] - (with-open [cos (.createCompressorOutputStream (CompressorStreamFactory.) CompressorStreamFactory/BZIP2, out)] - (clojure.java.io/copy (clojure.java.io/input-stream input-file) cos)) - ;; compressed output stream (cos) needs to be closed to flush any remaining bytes - (.toByteArray out))))) - -(defn-spec decompress-bytes (s/or :ok string?, :nil-or-empty-bytes nil?) - "decompresses the given `bz2-bytes` as bz2, returning a string. - if bytes are empty or `nil`, returns `nil`. - if bytes are not bz2 compressed, throws an `IOException`." - [bz2-bytes (s/nilable bytes?)] - (when-not (empty? bz2-bytes) - (with-open [is (clojure.java.io/input-stream bz2-bytes)] - (try - (with-open [cin (.createCompressorInputStream (CompressorStreamFactory.) CompressorStreamFactory/BZIP2, is)] - (slurp cin)) - (catch CompressorException ce - (throw (.getCause ce))))))) +(defmacro compile-time-slurp + "slurps given `resource` file at macro-expansion (compile) time." + [resource] + `(slurp (clojure.java.io/resource ~resource))) + +(defn-spec folder-size-bytes int? + "returns the size of a directory and it's contents in bytes" + [path ::sp/extant-dir] + (let [count-subdirs (fn [root dir-set] + (map #(-> root (fs/file %) .length) dir-set)) + count-files (fn [root file-list] + (map #(.length (fs/file root %)) file-list)) + count-subdirs+files (fn [root dir-set file-list] + (into (count-subdirs root dir-set) + (count-files root file-list)))] + (+ (-> path fs/file .length) + (reduce + (flatten (fs/walk count-subdirs+files path)))))) + +;; --- +;; copied from: https://github.com/clj-commons/humanize/blob/master/src/clj_commons/humanize.cljc +;; on: 2023-04-08 +;; with licence: EPL v1.0 +;; added to GPL exclusion list, see LICENCE.md. + +(defn logn [num base] + (/ (Math/round (Math/log num)) + (Math/round (Math/log base)))) + +(defn filesize + "Format a number of bytes as a human readable filesize (eg. 10 kB). + decimal suffixes (kB, MB) are used." + [bytes] + (cond + (not (number? bytes)) "" + (zero? bytes) "0" ;; special case for zero + + :else + (let [format-string "%.1f " + decimal-sizes [:B, :KB, :MB, :GB, :TB, + :PB, :EB, :ZB, :YB] + + units decimal-sizes + base 1000 + + base-pow (int (Math/floor (logn bytes base))) + ;; if base power shouldn't be larger than biggest unit + base-pow (if (< base-pow (count units)) + base-pow + (dec (count units))) + suffix (name (get units base-pow)) + ;; TODO: Math/pow isn't a drop-in for `expt`: + ;; https://github.com/clojure/math.numeric-tower/blob/97827be66f35feebc3c89ba81c546fef4adc7947/src/main/clojure/clojure/math/numeric_tower.clj#L89-L103 + value (float (/ bytes (Math/pow base base-pow)))] + + (str (format format-string value) suffix)))) + +(defn-spec now ::sp/inst + "returns the date and time right now as a datetime string" + [] + (str (java-time/instant))) + +(defn-spec unix-time-to-datetime ::sp/inst + "converts a number in the unix time format '1685623484' to milliseconds, then a `java.time.Instant` then a string" + [unix-time-seconds number?] + (-> unix-time-seconds (* 1000) java-time/instant str)) + +(defn-spec minutes-from-now number? + "returns the number of minutes between `(now)` and the given `instant`" + [instant ::sp/inst] + (let [duration (->> instant java-time/instant (java-time/duration (java-time/instant (now))))] + (java-time/as duration :minutes))) + +(defn user-locale + [] + (Locale/getDefault)) + +(defn-spec format-number string? + "locale-aware number formatting." + [^Integer n number?] + (.format ^java.text.NumberFormat (NumberFormat/getNumberInstance (user-locale)) n)) + +(defn-spec pretty-print-keyword (s/nilable string?) + "converts a keyword into a string. + hyphens are removed: :foo-bar => 'foo bar' + namespaces are handled: :foo/bar-baz => 'foo / bar baz' + non-keywords return nil. + nil returns nil." + [kw (s/nilable keyword?)] + (when (keyword? kw) ;; :foo, :foo/bar-baz + (let [ns-str (namespace kw) ;; nil, :foo + rest-str (some-> kw name (clojure.string/replace "-" " ")) ;; "bar baz" + ] + (if ns-str + (format "%s / %s" ns-str rest-str) ;; "foo / bar baz" + rest-str)))) ;; "bar baz" + +(defn-spec pretty-print-value string? + "converts any value into a friendly string. + strings are returned as-is. + integers are formatted according to locale. + lists and maps are recursively formatted. + empty lists and maps return '(empty)' + `nil` becomes the string '(none)'. + " + [v any?] + (cond + (nil? v) "(none)" + (number? v) (format-number v) + (sequential? v) (if (empty? v) "(empty)" (clojure.string/join ", " (mapv pretty-print-value v))) + (map? v) (if (empty? v) + "(empty)" + (clojure.string/join ", " (mapv (fn [[key val]] + (format "%s: %s" (pretty-print-value key) (pretty-print-value val))) (sort-by first v)))) + (boolean? v) (-> v str clojure.string/capitalize) + (keyword? v) (name v) + :else (str v))) diff --git a/src/strongbox/wowinterface_api.clj b/src/strongbox/wowinterface_api.clj index 6ec37630..f3b04f9a 100644 --- a/src/strongbox/wowinterface_api.clj +++ b/src/strongbox/wowinterface_api.clj @@ -28,25 +28,23 @@ (error (utils/reportable-error "given addon-summary has no game track list.")) (error addon-summary)) - ;; todo: this shouldn't be an 'if' if there is no 'else' + ;; when the selected game track is supported by the addon, check for updates. (when (some #{game-track} (:game-track-list addon-summary)) (let [url (str wowinterface-api "/filedetails/" (:source-id addon-summary) ".json") - result-list (some-> url http/download-with-backoff http/sink-error utils/from-json) - result (first result-list)] - (when result - - ;; has this happened before? can we find an example? - (when (> (count result-list) 1) - (warn "wowinterface api returned more than one result for addon with id:" (:source-id addon-summary))) - - (let [sid (:source-id addon-summary) - ;; rarely present. use it if found. actual value of `aid` not necessary, it seems to work when empty as well. - aid (extract-aid (:UIDownload result))] - [{:download-url (if aid - (format "https://cdn.wowinterface.com/downloads/getfile.php?id=%s&aid=%s" sid aid) - (format "https://cdn.wowinterface.com/downloads/getfile.php?id=%s" sid)) - :version (:UIVersion result) - :game-track game-track}]))))) + result-list (some-> url http/download-with-backoff http/sink-error utils/from-json)] + (when-not (empty? result-list) + + ;; 2023-06-09: we don't expect more than one result from wowi, ever, but for the sake of testing and + ;; consistency with other hosts it is now supported. + (mapv (fn [result] + (let [sid (:source-id addon-summary) + ;; rarely present. use it if found. actual value of `aid` not necessary, it seems to work when empty as well. + aid (extract-aid (:UIDownload result))] + {:download-url (if aid + (format "https://cdn.wowinterface.com/downloads/getfile.php?id=%s&aid=%s" sid aid) + (format "https://cdn.wowinterface.com/downloads/getfile.php?id=%s" sid)) + :version (:UIVersion result) + :game-track game-track})) result-list))))) (defn-spec parse-user-string (s/or :ok :addon/source-id :error nil?) "extracts the addon ID from the given `url`" diff --git a/test/fixtures/user-config-6.0.json b/test/fixtures/user-config-6.0.json new file mode 100644 index 00000000..c912808e --- /dev/null +++ b/test/fixtures/user-config-6.0.json @@ -0,0 +1,52 @@ +{ + "gui-theme": "dark-green", + "addon-dir-list": [ + { + "addon-dir": "/tmp/.strongbox-bar", + "game-track": "classic-tbc", + "strict?": true + }, + { + "addon-dir": "/tmp/.strongbox-foo", + "game-track": "retail", + "strict?": false + } + ], + "selected-addon-dir": "/tmp/.strongbox-foo", + "catalogue-location-list": [ + { + "name": "short", + "label": "Short (default)", + "source": "https://raw.githubusercontent.com/ogri-la/strongbox-catalogue/master/short-catalogue.json" + }, + { + "name": "full", + "label": "Full", + "source": "https://raw.githubusercontent.com/ogri-la/strongbox-catalogue/master/full-catalogue.json" + }, + { + "name": "tukui", + "label": "Tukui", + "source": "https://raw.githubusercontent.com/ogri-la/strongbox-catalogue/master/tukui-catalogue.json" + }, + { + "name": "wowinterface", + "label": "WoWInterface", + "source": "https://raw.githubusercontent.com/ogri-la/strongbox-catalogue/master/wowinterface-catalogue.json" + }, + { + "name": "github", + "label": "GitHub", + "source": "https://raw.githubusercontent.com/ogri-la/strongbox-catalogue/master/github-catalogue.json" + } + ], + "selected-catalogue": "full", + "preferences": + { + "addon-zips-to-keep": 3, + "ui-selected-columns": [ + "source", "name", "description", "installed-version", "available-version", "game-version", "uber-button" + ], + "keep-user-catalogue-updated": true + } +} diff --git a/test/fixtures/user-config-7.0.json b/test/fixtures/user-config-7.0.json new file mode 100644 index 00000000..9fd2635c --- /dev/null +++ b/test/fixtures/user-config-7.0.json @@ -0,0 +1,47 @@ +{ + "gui-theme": "dark-green", + "addon-dir-list": [ + { + "addon-dir": "/tmp/.strongbox-bar", + "game-track": "classic-tbc", + "strict?": true + }, + { + "addon-dir": "/tmp/.strongbox-foo", + "game-track": "retail", + "strict?": false + } + ], + "selected-addon-dir": "/tmp/.strongbox-foo", + "catalogue-location-list": [ + { + "name": "short", + "label": "Short (default)", + "source": "https://raw.githubusercontent.com/ogri-la/strongbox-catalogue/master/short-catalogue.json" + }, + { + "name": "full", + "label": "Full", + "source": "https://raw.githubusercontent.com/ogri-la/strongbox-catalogue/master/full-catalogue.json" + }, + { + "name": "wowinterface", + "label": "WoWInterface", + "source": "https://raw.githubusercontent.com/ogri-la/strongbox-catalogue/master/wowinterface-catalogue.json" + }, + { + "name": "github", + "label": "GitHub", + "source": "https://raw.githubusercontent.com/ogri-la/strongbox-catalogue/master/github-catalogue.json" + } + ], + "selected-catalogue": "full", + "preferences": + { + "addon-zips-to-keep": 3, + "ui-selected-columns": [ + "source", "name", "description", "installed-version", "available-version", "game-version", "uber-button" + ], + "keep-user-catalogue-updated": true + } +} diff --git a/test/strongbox/addon_test.clj b/test/strongbox/addon_test.clj index 2189dd2e..c7998c34 100644 --- a/test/strongbox/addon_test.clj +++ b/test/strongbox/addon_test.clj @@ -182,6 +182,7 @@ expected {:name "someaddon", :dirname "SomeAddon", + :dirsize 0 :label "SomeAddon", :description "asdf", :interface-version 80300, @@ -214,6 +215,7 @@ expected {;; toc data :name "someaddon" :dirname "SomeAddon" + :dirsize 0 :label "SomeAddon" :description "asdf" :interface-version 80300 @@ -255,6 +257,7 @@ expected {:name "someaddon", :dirname "SomeAddon", + :dirsize 0 :label "SomeAddon", :description "asdf", :interface-version 80300, @@ -279,6 +282,7 @@ expected {:name "someaddon", :dirname "SomeAddon", + :dirsize 0 :label "SomeAddon", :description "asdf", :interface-version 80300 @@ -298,6 +302,7 @@ expected [;; description has been modified in "-Classic" vs "-Vanilla" {:description "Slightly differently does what no other addon does." :dirname "EveryAddon", + :dirsize 0 :installed-version "1.2.3", :interface-version 11307, :label "EveryAddon 1.2.3", diff --git a/test/strongbox/catalogue_test.clj b/test/strongbox/catalogue_test.clj index 850e2d16..f1aefaa2 100644 --- a/test/strongbox/catalogue_test.clj +++ b/test/strongbox/catalogue_test.clj @@ -3,6 +3,7 @@ [clojure.test :refer [deftest testing is use-fixtures]] ;;[taoensso.timbre :as log :refer [debug info warn error spy]] [strongbox + [utils :as utils] [constants :as constants] [logging :as logging] [catalogue :as catalogue] @@ -280,9 +281,7 @@ strict? true fake-routes {"https://addons-ecs.forgesvc.net/api/v2/addon/281321" {:get (fn [req] {:status 502 :reason-phrase "Gateway Time-out (HTTP 502)"})}} - expected ["failed to fetch 'https://addons-ecs.forgesvc.net/api/v2/addon/281321': Gateway Time-out (HTTP 502) (HTTP 502)" - "Curseforge: the API is having problems right now. Try again later." - "no 'Retail' release found on curseforge."]] + expected ["addon host 'curseforge' was disabled Feb 1st, 2022.\n • use 'Source' and 'Find similar' from the addon context menu for alternatives."]] (with-fake-routes-in-isolation fake-routes (is (= expected (logging/buffered-log :info (catalogue/expand-summary addon game-track strict?)))))))) @@ -293,211 +292,244 @@ strict? true fake-routes {"https://addons-ecs.forgesvc.net/api/v2/addon/281321" {:get (fn [req] {:status 504 :reason-phrase "Gateway Time-out (HTTP 504)"})}} - expected ["failed to fetch 'https://addons-ecs.forgesvc.net/api/v2/addon/281321': Gateway Time-out (HTTP 504) (HTTP 504)" - "Curseforge: the API is having problems right now. Try again later." - "no 'Retail' release found on curseforge."]] + expected ["addon host 'curseforge' was disabled Feb 1st, 2022.\n • use 'Source' and 'Find similar' from the addon context menu for alternatives."]] (with-fake-routes-in-isolation fake-routes (is (= expected (logging/buffered-log :info (catalogue/expand-summary addon game-track strict?)))))))) ;; retail -(deftest expand-summary--retail-strict +(deftest expand-summary--retail-strict--just-retail (testing "when just retail is available, use it" - (let [addon {:name "foo" :label "Foo" :source "curseforge" :source-id "4646"} + (let [addon {:name "foo" :label "Foo" :source "wowinterface" :source-id "4646" :game-track-list [:retail]} game-track :retail strict? true - response (slurp (fixture-path "curseforge-api-addon--retail.json")) - fake-routes {"https://addons-ecs.forgesvc.net/api/v2/addon/4646" - {:get (fn [req] {:status 200 :body response})}} - expected {:download-url "https://edge.forgecdn.net/files/3104/62/Pawn-2.4.5.zip", - :interface-version 90000, + response [{:game-track :retail + :UIVersion "2.4.5"}] + fake-routes {"https://api.mmoui.com/v3/game/WOW/filedetails/4646.json" + {:get (fn [req] {:status 200 :body (utils/to-json response)})}} + expected {:download-url "https://cdn.wowinterface.com/downloads/getfile.php?id=4646" :version "2.4.5" :game-track :retail - :release-list [{:download-url "https://edge.forgecdn.net/files/3104/62/Pawn-2.4.5.zip", + :release-list [{:download-url "https://cdn.wowinterface.com/downloads/getfile.php?id=4646" :game-track :retail, - :interface-version 90000, - :release-label "[WoW 9.0.1] Pawn-2.4.5", :version "2.4.5"}]} expected (merge addon expected)] (with-fake-routes-in-isolation fake-routes - (is (= expected (catalogue/expand-summary addon game-track strict?)))))) + (is (= expected (catalogue/expand-summary addon game-track strict?))))))) +(deftest expand-summary--retail-strict--just-classic (testing "when just classic is available, use nothing" - (let [addon {:name "foo" :label "Foo" :source "curseforge" :source-id "4646"} + (let [addon {:name "foo" :label "Foo" :source "wowinterface" :source-id "4646" + ;; 2023-06-11: test switched from curseforge to wowinterface. + ;; wowinterface doesn't expand to different game tracks, so what we're testing here is refusing to expand, + ;; rather than filtering out a classic release. + :game-track-list [:classic]} game-track :retail strict? true - response (slurp (fixture-path "curseforge-api-addon--classic.json")) - fake-routes {"https://addons-ecs.forgesvc.net/api/v2/addon/4646" - {:get (fn [req] {:status 200 :body response})}} + response [{:game-track :classic + :UIVersion "2.4.5"}] + fake-routes {"https://api.mmoui.com/v3/game/WOW/filedetails/4646.json" + {:get (fn [req] {:status 200 :body (utils/to-json response)})}} expected nil] (with-fake-routes-in-isolation fake-routes - (is (= expected (catalogue/expand-summary addon game-track strict?)))))) + (is (= expected (catalogue/expand-summary addon game-track strict?))))))) +(deftest expand-summary--retail-strict--retail-and-classic (testing "when both retail and classic are available, use retail" - (let [addon {:name "foo" :label "Foo" :source "curseforge" :source-id "4646"} + (let [addon {:name "foo" :label "Foo" :source "wowinterface" :source-id "4646" :game-track-list [:retail :classic]} game-track :retail strict? true - response (slurp (fixture-path "curseforge-api-addon--retail-AND-classic.json")) - fake-routes {"https://addons-ecs.forgesvc.net/api/v2/addon/4646" - {:get (fn [req] {:status 200 :body response})}} - expected {:download-url "https://edge.forgecdn.net/files/3104/62/Pawn-2.4.5.zip", - :interface-version 90000, + response [{:game-track :retail + :UIVersion "2.4.5"}] + fake-routes {"https://api.mmoui.com/v3/game/WOW/filedetails/4646.json" + {:get (fn [req] {:status 200 :body (utils/to-json response)})}} + expected {:download-url "https://cdn.wowinterface.com/downloads/getfile.php?id=4646", :version "2.4.5" :game-track :retail - :release-list [{:download-url "https://edge.forgecdn.net/files/3104/62/Pawn-2.4.5.zip", + :release-list [{:download-url "https://cdn.wowinterface.com/downloads/getfile.php?id=4646", :game-track :retail, - :interface-version 90000, - :release-label "[WoW 9.0.1] Pawn-2.4.5", :version "2.4.5"}]} expected (merge addon expected)] (with-fake-routes-in-isolation fake-routes (is (= expected (catalogue/expand-summary addon game-track strict?))))))) (deftest expand-summary--retail-then-classic - (testing "when just classic is available, use it" - (let [addon {:name "foo" :label "Foo" :source "curseforge" :source-id "4646"} + (testing "when both retail and classic are available, game track is set to retail and and strict-mode is off, use retail." + (let [addon {:name "foo" :label "Foo" :source "wowinterface" :source-id "4646" :game-track-list [:retail :classic]} game-track :retail strict? false - response (slurp (fixture-path "curseforge-api-addon--classic.json")) - fake-routes {"https://addons-ecs.forgesvc.net/api/v2/addon/4646" - {:get (fn [req] {:status 200 :body response})}} - expected {:download-url "https://edge.forgecdn.net/files/3104/60/Pawn-2.4.5-Classic.zip", - :interface-version 11300, - :version "2.4.5 (Classic)" - :game-track :classic - :release-list [{:download-url "https://edge.forgecdn.net/files/3104/60/Pawn-2.4.5-Classic.zip", - :game-track :classic, - :interface-version 11300, - :release-label "[WoW 1.13.5] Pawn-2.4.5-Classic", - :version "2.4.5 (Classic)"}]} + response [{:game-track :retail + :UIVersion "2.4.5"}] + fake-routes {"https://api.mmoui.com/v3/game/WOW/filedetails/4646.json" + {:get (fn [req] {:status 200 :body (utils/to-json response)})}} + expected {:download-url "https://cdn.wowinterface.com/downloads/getfile.php?id=4646" + :version "2.4.5" + :game-track :retail + :release-list [{:download-url "https://cdn.wowinterface.com/downloads/getfile.php?id=4646" + :game-track :retail + :version "2.4.5"}]} expected (merge addon expected)] (with-fake-routes-in-isolation fake-routes (is (= expected (catalogue/expand-summary addon game-track strict?))))))) ;; classic -(deftest expand-summary--classic-strict +(deftest expand-summary--classic-strict--just-classic (testing "when just classic is available, use it" - (let [addon {:name "foo" :label "Foo" :source "curseforge" :source-id "4646"} + (let [addon {:name "foo" :label "Foo" :source "wowinterface" :source-id "4646" :game-track-list [:classic]} game-track :classic strict? true - response (slurp (fixture-path "curseforge-api-addon--classic.json")) - fake-routes {"https://addons-ecs.forgesvc.net/api/v2/addon/4646" - {:get (fn [req] {:status 200 :body response})}} - expected {:download-url "https://edge.forgecdn.net/files/3104/60/Pawn-2.4.5-Classic.zip", - :interface-version 11300, - :version "2.4.5 (Classic)" + response [{:game-track :classic + :UIVersion "2.4.5"}] + fake-routes {"https://api.mmoui.com/v3/game/WOW/filedetails/4646.json" + {:get (fn [req] {:status 200 :body (utils/to-json response)})}} + expected {:download-url "https://cdn.wowinterface.com/downloads/getfile.php?id=4646" + :version "2.4.5" :game-track :classic - :release-list [{:download-url "https://edge.forgecdn.net/files/3104/60/Pawn-2.4.5-Classic.zip", + :release-list [{:download-url "https://cdn.wowinterface.com/downloads/getfile.php?id=4646" :game-track :classic, - :interface-version 11300, - :release-label "[WoW 1.13.5] Pawn-2.4.5-Classic", - :version "2.4.5 (Classic)"}]} + :version "2.4.5"}]} expected (merge addon expected)] (with-fake-routes-in-isolation fake-routes - (is (= expected (catalogue/expand-summary addon game-track strict?)))))) + (is (= expected (catalogue/expand-summary addon game-track strict?))))))) +(deftest expand-summary--classic-strict--just-retail (testing "when just retail is available, use nothing" - (let [addon {:name "foo" :label "Foo" :source "curseforge" :source-id "4646"} + (let [addon {:name "foo" :label "Foo" :source "wowinterface" :source-id "4646" :game-track-list [:retail]} game-track :classic strict? true - response (slurp (fixture-path "curseforge-api-addon--retail.json")) - fake-routes {"https://addons-ecs.forgesvc.net/api/v2/addon/4646" - {:get (fn [req] {:status 200 :body response})}} + response [{:game-track :retail + :UIVersion "2.4.5"}] + fake-routes {"https://api.mmoui.com/v3/game/WOW/filedetails/4646.json" + {:get (fn [req] {:status 200 :body (utils/to-json response)})}} expected nil] (with-fake-routes-in-isolation fake-routes - (is (= expected (catalogue/expand-summary addon game-track strict?)))))) + (is (= expected (catalogue/expand-summary addon game-track strict?))))))) +(deftest expand-summary--classic-strict--retail-and-classic (testing "when both retail and classic are available, use classic" - (let [addon {:name "foo" :label "Foo" :source "curseforge" :source-id "4646"} + (let [addon {:name "foo" :label "Foo" :source "wowinterface" :source-id "4646" :game-track-list [:retail :classic]} game-track :classic strict? true - response (slurp (fixture-path "curseforge-api-addon--retail-AND-classic.json")) - fake-routes {"https://addons-ecs.forgesvc.net/api/v2/addon/4646" - {:get (fn [req] {:status 200 :body response})}} - expected {:download-url "https://edge.forgecdn.net/files/3104/60/Pawn-2.4.5-Classic.zip", - :interface-version 11300, - :version "2.4.5 (Classic)" + response [{:game-track :retail ;; 2023-06-09: this is interesting behaviour. retail response, classic returned + :UIVersion "2.4.5"}] + fake-routes {"https://api.mmoui.com/v3/game/WOW/filedetails/4646.json" + {:get (fn [req] {:status 200 :body (utils/to-json response)})}} + expected {:download-url "https://cdn.wowinterface.com/downloads/getfile.php?id=4646" + :version "2.4.5" :game-track :classic - :release-list [{:download-url "https://edge.forgecdn.net/files/3104/60/Pawn-2.4.5-Classic.zip", + :release-list [{:download-url "https://cdn.wowinterface.com/downloads/getfile.php?id=4646" :game-track :classic, - :interface-version 11300, - :release-label "[WoW 1.13.5] Pawn-2.4.5-Classic", - :version "2.4.5 (Classic)"}]} + :version "2.4.5"}]} expected (merge addon expected)] (with-fake-routes-in-isolation fake-routes (is (= expected (catalogue/expand-summary addon game-track strict?))))))) (deftest expand-summary--classic-then-retail - (testing "when just retail is available, use it" - (let [addon {:name "foo" :label "Foo" :source "curseforge" :source-id "4646"} + (testing "when both retail and classic are available, game track is set to classic and and strict-mode is off, use classic." + (let [addon {:name "foo" :label "Foo" :source "wowinterface" :source-id "4646" :game-track-list [:retail :classic]} game-track :classic strict? false - response (slurp (fixture-path "curseforge-api-addon--retail.json")) - fake-routes {"https://addons-ecs.forgesvc.net/api/v2/addon/4646" - {:get (fn [req] {:status 200 :body response})}} - expected {:download-url "https://edge.forgecdn.net/files/3104/62/Pawn-2.4.5.zip", - :interface-version 90000, + response [{:game-track :retail + :UIVersion "2.4.5"}] + fake-routes {"https://api.mmoui.com/v3/game/WOW/filedetails/4646.json" + {:get (fn [req] {:status 200 :body (utils/to-json response)})}} + expected {:download-url "https://cdn.wowinterface.com/downloads/getfile.php?id=4646" :version "2.4.5" - :game-track :retail - :release-list [{:download-url "https://edge.forgecdn.net/files/3104/62/Pawn-2.4.5.zip", - :game-track :retail, - :interface-version 90000, - :release-label "[WoW 9.0.1] Pawn-2.4.5", + :game-track :classic + :release-list [{:download-url "https://cdn.wowinterface.com/downloads/getfile.php?id=4646" + :game-track :classic :version "2.4.5"}]} expected (merge addon expected)] (with-fake-routes-in-isolation fake-routes (is (= expected (catalogue/expand-summary addon game-track strict?))))))) +;; --- + (deftest expand-summary--pinned--use-pinned (testing "when an addon is pinned, look for it's release in the list of releases returned from the host" - (let [addon {:name "foo" :label "Foo" :source "curseforge" :source-id "4646" - :installed-version "1.2.3" :pinned-version "1.2.0"} + (let [addon {:name "foo" + :label "Foo" + :source "wowinterface" + :source-id "4646" + :installed-version "1.2.3" + :pinned-version "1.2.0" + :game-track-list [:retail]} game-track :retail strict? true - fixture [{:download-url "https://edge.forgecdn.net/files/3104/62/Pawn-2.4.5.zip", - :game-track :retail, - :interface-version 90000, - :release-label "[WoW 9.0.1] Pawn-1.2.3.zip", - :version "1.2.3"} - ;; pinned release - {:download-url "https://edge.forgecdn.net/files/3104/062/Addon-2.4.5.zip", + expected {:name "foo" + :label "Foo" + :source "wowinterface" + :source-id "4646" + :installed-version "1.2.3" + :pinned-version "1.2.0" + :game-track-list [:retail] + + ;; pinned release merged in + :download-url "https://cdn.wowinterface.com/downloads/getfile.php?id=4646" :game-track :retail, - :interface-version 90000, - :release-label "[WoW 9.0.1] Addon-1.2.0", - :version "1.2.0"}] - expected (-> addon - (merge {:release-list fixture} - (second fixture)) - (dissoc :release-label))] - (with-fake-routes-in-isolation {} - ;; omg, with-redefs is fantastic - (with-redefs [strongbox.curseforge-api/expand-summary (constantly fixture)] - (is (= expected (catalogue/expand-summary addon game-track strict?)))))))) + :version "1.2.0" + + ;; pinned + new versions available in release list + :release-list [{:download-url "https://cdn.wowinterface.com/downloads/getfile.php?id=4646" + :game-track :retail, + :version "1.2.3"} + {:download-url "https://cdn.wowinterface.com/downloads/getfile.php?id=4646" + :game-track :retail, + :version "1.2.0"}]} + response [{:game-track :retail + :UIVersion "1.2.3"} + ;; pinned release + {:game-track :retail + :UIVersion "1.2.0"}] + fake-routes {"https://api.mmoui.com/v3/game/WOW/filedetails/4646.json" + {:get (fn [req] {:status 200 :body (utils/to-json response)})}}] + (with-fake-routes-in-isolation fake-routes + (is (= expected (catalogue/expand-summary addon game-track strict?))))))) (deftest expand-summary--pinned--use-latest (testing "when a pinned addon cannot find it's pinned release, use the latest release available" - (let [addon {:name "foo" :label "Foo" :source "curseforge" :source-id "4646" - :installed-version "0.9.9" :pinned-version "0.9.9"} + (let [addon {:name "foo" + :label "Foo" + :source "wowinterface" + :source-id "4646" + :installed-version "0.9.9" + :pinned-version "0.9.9" + :game-track-list [:retail]} game-track :retail strict? true - fixture [{:download-url "https://edge.forgecdn.net/files/3104/62/Pawn-2.4.5.zip", - :game-track :retail, - :interface-version 90000, - :release-label "[WoW 9.0.1] Pawn-1.2.3", - :version "1.2.3"} - {:download-url "https://edge.forgecdn.net/files/3104/062/Addon-2.4.5.zip", + expected {:name "foo" + :label "Foo" + :source "wowinterface" + :source-id "4646" + :installed-version "0.9.9" + :pinned-version "0.9.9" + :game-track-list [:retail] + + ;; latest release merged in + :download-url "https://cdn.wowinterface.com/downloads/getfile.php?id=4646" :game-track :retail, - :interface-version 90000, - :release-label "[WoW 9.0.1] Addon-1.2.0", - :version "1.2.0"}] - expected (-> addon - (merge {:release-list fixture} - (first fixture)) - (dissoc :release-label))] - (with-fake-routes-in-isolation {} - (with-redefs [strongbox.curseforge-api/expand-summary (constantly fixture)] - (is (= expected (catalogue/expand-summary addon game-track strict?)))))))) + :version "1.2.3" + + ;; pinned + new versions available in release list + :release-list [{:download-url "https://cdn.wowinterface.com/downloads/getfile.php?id=4646" + :game-track :retail, + :version "1.2.3"} + {:download-url "https://cdn.wowinterface.com/downloads/getfile.php?id=4646" + :game-track :retail, + :version "1.2.0"}]} + response [{:game-track :retail + :UIVersion "1.2.3"} + {:game-track :retail + :UIVersion "1.2.0"} + ;; pinned release missing from response + ;;{:game-track :retail + ;; :UIVersion "1.2.0"} + ] + fake-routes {"https://api.mmoui.com/v3/game/WOW/filedetails/4646.json" + {:get (fn [req] {:status 200 :body (utils/to-json response)})}}] + (with-fake-routes-in-isolation fake-routes + (is (= expected (catalogue/expand-summary addon game-track strict?))))))) ;; diff --git a/test/strongbox/cli_test.clj b/test/strongbox/cli_test.clj index a08c1338..d8243b82 100644 --- a/test/strongbox/cli_test.clj +++ b/test/strongbox/cli_test.clj @@ -446,6 +446,7 @@ ;; a mush of the above (.nfo written during install) and the EveryAddon .toc file expected {:description "Does what no other addon does, slightly differently", :dirname "EveryAddon", + :dirsize 0 :group-id "https://www.wowinterface.com/downloads/info25079", :installed-game-track :retail, :installed-version "1.2.3", @@ -575,6 +576,7 @@ ;; a mush of the above (.nfo written during install) and the EveryAddon .toc file expected {:description "Does what no other addon does, slightly differently", :dirname "EveryAddon", + :dirsize 0 :group-id "https://www.tukui.org/addons.php?id=98", :installed-game-track :retail, :installed-version "0.960", @@ -586,9 +588,11 @@ :source "tukui", :source-id 98 :source-map-list [{:source "tukui" :source-id 98}]} + _ expected expected-addon-dir (utils/join install-dir "EveryAddon") expected-user-catalogue [match] + _ expected-user-catalogue catalogue (utils/to-json (catalogue/new-catalogue [match])) @@ -614,34 +618,42 @@ ;; user gives us this url, we find it and install it (cli/import-addon user-url) - ;; addon was successfully download and installed - (is (fs/exists? expected-addon-dir)) + ;; addon was (not) successfully download and installed + ;;(is (fs/exists? expected-addon-dir)) + (is (not (fs/exists? expected-addon-dir))) ;; re-read install dir (core/load-all-installed-addons) - ;; we expect our mushy set of .nfo and .toc data - (is (= [expected] (core/get-state :installed-addon-list))) + ;; we expect ~our~ (no) mushy set of .nfo and .toc data + ;;(is (= [expected] (core/get-state :installed-addon-list))) + (is (= [] (core/get-state :installed-addon-list))) ;; and that the addon was added to the user catalogue - (is (= expected-user-catalogue (core/get-state :user-catalogue :addon-summary-list))) + ;;(is (= expected-user-catalogue (core/get-state :user-catalogue :addon-summary-list))) + (is (nil? (core/get-state :user-catalogue :addon-summary-list))) ;; and that the user catalogue was persisted to disk - (is (= expected-user-catalogue - (:addon-summary-list (catalogue/read-catalogue (core/paths :user-catalogue-file)))))))))) + ;;(is (= expected-user-catalogue + ;; (:addon-summary-list (catalogue/read-catalogue (core/paths :user-catalogue-file))))) + (is (nil? (:addon-summary-list (catalogue/read-catalogue (core/paths :user-catalogue-file)))))))))) ;; todo: no import-addon-gitlab ? -(deftest refresh-user-catalogue +(deftest refresh-user-catalogue--not-in-catalogue (testing "the user catalogue can be 'refreshed', pulling in updated information from github and the current catalogue" (with-running-app+opts {:ui nil} (let [;; user-catalogue with a bunch of addons across all hosts that the user has added. user-catalogue-fixture (fixture-path "user-catalogue--populated.json") - ;; default app catalogue, contains newer versions of the addon summaries in the user-catalogue. - ;; this is because the catalogue is updated periodically and the user-catalogue is not. + ;; default app catalogue. this is what the user should have selected before and after refresh, + ;; despite using the 'full' catalogue for the refresh. short-catalogue (slurp (fixture-path "user-catalogue--short-catalogue.json")) + ;; full app catalogue, contains newer versions of the addon summaries in the user-catalogue. + ;; this is because regular catalogues are updated periodically and the user-catalogue is not. + full-catalogue short-catalogue + tukui-fixture (slurp (fixture-path "user-catalogue--tukui.json")) tukui-classic-fixture (slurp (fixture-path "user-catalogue--tukui-classic.json")) tukui-classic-tbc-fixture (slurp (fixture-path "user-catalogue--tukui-classic-tbc.json")) @@ -657,7 +669,10 @@ gitlab-blob-fixture (slurp (fixture-path "user-catalogue--gitlab-blob.json")) gitlab-releases-fixture (slurp (fixture-path "user-catalogue--gitlab-releases.json")) - fake-routes {"https://raw.githubusercontent.com/ogri-la/strongbox-catalogue/master/short-catalogue.json" + fake-routes {"https://raw.githubusercontent.com/ogri-la/strongbox-catalogue/master/full-catalogue.json" + {:get (fn [req] {:status 200 :body full-catalogue})} + + "https://raw.githubusercontent.com/ogri-la/strongbox-catalogue/master/short-catalogue.json" {:get (fn [req] {:status 200 :body short-catalogue})} "https://www.tukui.org/api.php?addons" @@ -711,6 +726,9 @@ ;; sanity check, ensure user-catalogue loaded (is (= expected-num (count (core/get-state :db)))) + ;; ensure default short-catalogue is being used + (is (= :short (:name (core/current-catalogue)))) + ;; we need to load the short-catalogue using newer versions of what is in the user-catalogue ;; the user-catalogue is then matched against db, the newer summary returned and written to the user catalogue @@ -722,32 +740,12 @@ expected-user-catalogue (-> (core/get-user-catalogue) (update-in [:addon-summary-list] #(mapv inc-downloads %)) (assoc :datestamp today))] - (cli/refresh-user-catalogue) + (core/refresh-user-catalogue) ;; ensure new user-catalogue matches expectations - (is (= expected-user-catalogue (core/get-user-catalogue))))))))) + (is (= expected-user-catalogue (core/get-user-catalogue))) -(deftest refresh-user-catalogue-item - (testing "individual addons can be refreshed, writing the changes to disk afterwards." - (let [user-catalogue (catalogue/new-catalogue [helper/addon-summary]) - new-addon (merge helper/addon-summary {:updated-date "2022-02-02T02:02:02"}) - expected (assoc user-catalogue :addon-summary-list [new-addon])] - (with-running-app - (swap! core/state assoc :user-catalogue user-catalogue) - (core/write-user-catalogue!) - (with-redefs [cli/find-addon (fn [& args] new-addon)] - (cli/refresh-user-catalogue-item helper/addon-summary)) - (is (= expected (core/get-state :user-catalogue))))))) - -(deftest refresh-user-catalogue-item--no-catalogue - (testing "looking for an addon that doesn't exist in the catalogue isn't a total failure" - (with-running-app - (is (nil? (cli/refresh-user-catalogue-item helper/addon-summary)))))) - -(deftest refresh-user-catalogue-item--unhandled-exception - (testing "unhandled exceptions while refreshing a user-catalogue item isn't a total failure" - (with-running-app - (with-redefs [cli/find-addon (fn [& args] (throw (Exception. "catastrophe!")))] - (is (nil? (cli/refresh-user-catalogue-item helper/addon-summary))))))) + ;; ensure the selected catalogue hasn't changed despite downloading the full catalogue + (is (= :short (:name (core/current-catalogue)))))))))) ;; @@ -776,6 +774,25 @@ (is (= expected (core/get-state :user-catalogue))) (is (= expected (catalogue/read-catalogue user-catalogue-file))))))) +(deftest add-addon-to-user-catalogue + (testing "random addons can be added to the user catalogue, provided they can be found in the catalogue." + (with-running-app + (let [expected (catalogue/new-catalogue [helper/addon-summary]) + given (select-keys helper/addon-summary [:source :source-id]) + user-catalogue-file (core/paths :user-catalogue-file) + db [helper/addon-summary]] + + (is (nil? (core/get-state :db))) + (is (nil? (core/get-state :user-catalogue))) + (is (not (fs/exists? user-catalogue-file))) + + (swap! core/state assoc :db db) + + (cli/add-addon-to-user-catalogue given) + + (is (= expected (core/get-state :user-catalogue))) + (is (= expected (catalogue/read-catalogue user-catalogue-file))))))) + ;; test doesn't seem to live comfortably in `core_test.clj` (deftest install-update-these-in-parallel--bad-download diff --git a/test/strongbox/config_test.clj b/test/strongbox/config_test.clj index 4883b671..d6026f53 100644 --- a/test/strongbox/config_test.clj +++ b/test/strongbox/config_test.clj @@ -197,7 +197,9 @@ :ui-selected-columns [:source :name :description ;;:installed-version :available-version ;; replaced in 5.0.0 :combined-version - :game-version :uber-button]}} + :game-version :uber-button] + ;; new in 6.2.0 + :keep-user-catalogue-updated false}} :cli-opts {} :file-opts {:debug? true @@ -231,7 +233,9 @@ :ui-selected-columns [:source :name :description ;;:installed-version :available-version ;; replaced in 5.0.0 :combined-version - :game-version :uber-button]}} + :game-version :uber-button] + ;; new in 6.2.0 + :keep-user-catalogue-updated false}} :cli-opts {} :file-opts {:selected-catalogue :full @@ -266,7 +270,9 @@ :ui-selected-columns [:source :name :description ;;:installed-version :available-version ;; replaced in 5.0.0 :combined-version - :game-version :uber-button]}} + :game-version :uber-button] + ;; new in 6.2.0 + :keep-user-catalogue-updated false}} :cli-opts {} :file-opts {:gui-theme :dark @@ -302,7 +308,9 @@ :ui-selected-columns [:source :name :description ;;:installed-version :available-version ;; replaced in 5.0.0 :combined-version - :game-version :uber-button]}} + :game-version :uber-button] + ;; new in 6.2.0 + :keep-user-catalogue-updated false}} :cli-opts {} :file-opts {:gui-theme :dark @@ -337,7 +345,9 @@ :ui-selected-columns [:source :name :description ;;:installed-version :available-version ;; replaced in 5.0.0 :combined-version - :game-version :uber-button]}} + :game-version :uber-button] + ;; new in 6.2.0 + :keep-user-catalogue-updated false}} :cli-opts {} :file-opts {:gui-theme :dark @@ -350,6 +360,7 @@ ;; the set of available catalogues are added to configuration. :catalogue-location-list [{:name :short :label "Short (default)" :source "https://raw.githubusercontent.com/ogri-la/strongbox-catalogue/master/short-catalogue.json"} {:name :full :label "Full" :source "https://raw.githubusercontent.com/ogri-la/strongbox-catalogue/master/full-catalogue.json"} + ;; removed in 7.0.0 {:name :tukui :label "Tukui" :source "https://raw.githubusercontent.com/ogri-la/strongbox-catalogue/master/tukui-catalogue.json"} ;; removed in 5.0.0 {:name :curseforge :label "Curseforge" :source "https://raw.githubusercontent.com/ogri-la/strongbox-catalogue/master/curseforge-catalogue.json"} @@ -384,7 +395,9 @@ :ui-selected-columns [:source :name :description ;;:installed-version :available-version ;; replaced in 5.0.0 :combined-version - :game-version :uber-button]}} + :game-version :uber-button] + ;; new in 6.2.0 + :keep-user-catalogue-updated false}} :cli-opts {} :file-opts {:gui-theme :dark @@ -396,6 +409,7 @@ ;; new in 1.0 :catalogue-location-list [{:name :short :label "Short (default)" :source "https://raw.githubusercontent.com/ogri-la/strongbox-catalogue/master/short-catalogue.json"} {:name :full :label "Full" :source "https://raw.githubusercontent.com/ogri-la/strongbox-catalogue/master/full-catalogue.json"} + ;; removed in 7.0.0 {:name :tukui :label "Tukui" :source "https://raw.githubusercontent.com/ogri-la/strongbox-catalogue/master/tukui-catalogue.json"} ;; removed in 5.0.0 {:name :curseforge :label "Curseforge" :source "https://raw.githubusercontent.com/ogri-la/strongbox-catalogue/master/curseforge-catalogue.json"} @@ -432,7 +446,9 @@ :ui-selected-columns [:source :name :description ;;:installed-version :available-version ;; replaced in 5.0.0 :combined-version - :game-version :uber-button]}} + :game-version :uber-button] + ;; new in 6.2.0 + :keep-user-catalogue-updated false}} :cli-opts {} :file-opts {:gui-theme :dark-green @@ -444,6 +460,7 @@ ;; new in 1.0 :catalogue-location-list [{:name :short :label "Short (default)" :source "https://raw.githubusercontent.com/ogri-la/strongbox-catalogue/master/short-catalogue.json"} {:name :full :label "Full" :source "https://raw.githubusercontent.com/ogri-la/strongbox-catalogue/master/full-catalogue.json"} + ;; removed in 7.0.0 {:name :tukui :label "Tukui" :source "https://raw.githubusercontent.com/ogri-la/strongbox-catalogue/master/tukui-catalogue.json"} ;; removed in 5.0.0 {:name :curseforge :label "Curseforge" :source "https://raw.githubusercontent.com/ogri-la/strongbox-catalogue/master/curseforge-catalogue.json"} @@ -479,7 +496,9 @@ :ui-selected-columns [:source :name :description ;;:installed-version :available-version ;; replaced in 5.0.0 :combined-version - :game-version :uber-button]}} + :game-version :uber-button] + ;; new in 6.2.0 + :keep-user-catalogue-updated false}} :cli-opts {} :file-opts {:gui-theme :dark-green @@ -491,6 +510,7 @@ ;; new in 1.0 :catalogue-location-list [{:name :short :label "Short (default)" :source "https://raw.githubusercontent.com/ogri-la/strongbox-catalogue/master/short-catalogue.json"} {:name :full :label "Full" :source "https://raw.githubusercontent.com/ogri-la/strongbox-catalogue/master/full-catalogue.json"} + ;; removed in 7.0.0 {:name :tukui :label "Tukui" :source "https://raw.githubusercontent.com/ogri-la/strongbox-catalogue/master/tukui-catalogue.json"} ;; removed in 5.0.0 {:name :curseforge :label "Curseforge" :source "https://raw.githubusercontent.com/ogri-la/strongbox-catalogue/master/curseforge-catalogue.json"} @@ -523,7 +543,9 @@ ;; new in 3.1.0 :preferences {:addon-zips-to-keep 3 ;; new in 4.7.0 - :ui-selected-columns [:source :name :description :available-version :uber-button]}} + :ui-selected-columns [:source :name :description :available-version :uber-button] + ;; new in 6.2.0 + :keep-user-catalogue-updated false}} :cli-opts {} :file-opts {:gui-theme :dark-green @@ -535,6 +557,7 @@ ;; new in 1.0 :catalogue-location-list [{:name :short :label "Short (default)" :source "https://raw.githubusercontent.com/ogri-la/strongbox-catalogue/master/short-catalogue.json"} {:name :full :label "Full" :source "https://raw.githubusercontent.com/ogri-la/strongbox-catalogue/master/full-catalogue.json"} + ;; removed in 7.0.0 {:name :tukui :label "Tukui" :source "https://raw.githubusercontent.com/ogri-la/strongbox-catalogue/master/tukui-catalogue.json"} ;; removed in 5.0.0 {:name :curseforge :label "Curseforge" :source "https://raw.githubusercontent.com/ogri-la/strongbox-catalogue/master/curseforge-catalogue.json"} @@ -569,7 +592,9 @@ ;; new in 3.1.0 :preferences {:addon-zips-to-keep 3 ;; new in 4.7.0 - :ui-selected-columns [:source :name :description :available-version :uber-button]}} + :ui-selected-columns [:source :name :description :available-version :uber-button] + ;; new in 6.2.0 + :keep-user-catalogue-updated false}} :cli-opts {} :file-opts {:gui-theme :dark-green @@ -581,6 +606,7 @@ ;; new in 1.0 :catalogue-location-list [{:name :short :label "Short (default)" :source "https://raw.githubusercontent.com/ogri-la/strongbox-catalogue/master/short-catalogue.json"} {:name :full :label "Full" :source "https://raw.githubusercontent.com/ogri-la/strongbox-catalogue/master/full-catalogue.json"} + ;; removed in 7.0.0 {:name :tukui :label "Tukui" :source "https://raw.githubusercontent.com/ogri-la/strongbox-catalogue/master/tukui-catalogue.json"} ;; removed in 5.0.0 {:name :curseforge :label "Curseforge" :source "https://raw.githubusercontent.com/ogri-la/strongbox-catalogue/master/curseforge-catalogue.json"} @@ -620,7 +646,9 @@ :ui-selected-columns [:source :name :description :combined-version ;; new default in 5.0.0 - :game-version :uber-button]}} + :game-version :uber-button] + ;; new in 6.2.0 + :keep-user-catalogue-updated false}} :cli-opts {} :file-opts {:gui-theme :dark-green @@ -632,12 +660,12 @@ ;; new in 1.0 :catalogue-location-list [{:name :short :label "Short (default)" :source "https://raw.githubusercontent.com/ogri-la/strongbox-catalogue/master/short-catalogue.json"} {:name :full :label "Full" :source "https://raw.githubusercontent.com/ogri-la/strongbox-catalogue/master/full-catalogue.json"} + ;; removed in 7.0.0 {:name :tukui :label "Tukui" :source "https://raw.githubusercontent.com/ogri-la/strongbox-catalogue/master/tukui-catalogue.json"} ;; removed in 5.0.0 ;;{:name :curseforge :label "Curseforge" :source "https://raw.githubusercontent.com/ogri-la/strongbox-catalogue/master/curseforge-catalogue.json"} {:name :wowinterface :label "WoWInterface" :source "https://raw.githubusercontent.com/ogri-la/strongbox-catalogue/master/wowinterface-catalogue.json"} ;; new in 4.9 - ;; the set of available catalogues now includes a github catalogue {:name :github :label "GitHub" :source "https://raw.githubusercontent.com/ogri-la/strongbox-catalogue/master/github-catalogue.json"}] :preferences {:addon-zips-to-keep 3 @@ -648,3 +676,115 @@ :etag-db {}}] (is (= expected (config/load-settings cli-opts cfg-file etag-db-file)))))) +(deftest load-settings-6.0 + (testing "a standard config file circa 6.0 is loaded and parsed as expected" + (let [cli-opts {} + cfg-file (fixture-path "user-config-6.0.json") + etag-db-file (fixture-path "empty-map.json") + + expected {:cfg {:gui-theme :dark-green ;; new in 0.11, `:dark-green` new in 3.2.0 + :selected-catalogue :full ;; new in 0.10 + ;;:debug? true ;; removed in 0.12 + :addon-dir-list [{:addon-dir "/tmp/.strongbox-bar", :game-track :classic-tbc :strict? true} ;; `:classic-tbc` and `:strict?` added in 4.1 + {:addon-dir "/tmp/.strongbox-foo", :game-track :retail :strict? false}] + + ;; new in 1.0 + ;; new in 4.9, the set of available catalogues now includes a github catalogue + :catalogue-location-list (:catalogue-location-list config/default-cfg) + + ;; new in 0.12 + ;; moved to :cfg in 1.0 + :selected-addon-dir "/tmp/.strongbox-foo" + + ;; new in 3.1.0 + :preferences {:addon-zips-to-keep 3 + + :ui-selected-columns [:source :name :description + :combined-version ;; new default in 5.0.0 + :game-version :uber-button] + ;; new in 6.2.0 + :keep-user-catalogue-updated true}} + + :cli-opts {} + :file-opts {:gui-theme :dark-green + :selected-catalogue :full + :addon-dir-list [{:addon-dir "/tmp/.strongbox-bar", :game-track :classic-tbc, :strict? true} + {:addon-dir "/tmp/.strongbox-foo", :game-track :retail, :strict? false}] + :selected-addon-dir "/tmp/.strongbox-foo" + + ;; new in 1.0 + :catalogue-location-list [{:name :short :label "Short (default)" :source "https://raw.githubusercontent.com/ogri-la/strongbox-catalogue/master/short-catalogue.json"} + {:name :full :label "Full" :source "https://raw.githubusercontent.com/ogri-la/strongbox-catalogue/master/full-catalogue.json"} + ;; removed in 7.0.0 + {:name :tukui :label "Tukui" :source "https://raw.githubusercontent.com/ogri-la/strongbox-catalogue/master/tukui-catalogue.json"} + ;; removed in 5.0.0 + ;;{:name :curseforge :label "Curseforge" :source "https://raw.githubusercontent.com/ogri-la/strongbox-catalogue/master/curseforge-catalogue.json"} + {:name :wowinterface :label "WoWInterface" :source "https://raw.githubusercontent.com/ogri-la/strongbox-catalogue/master/wowinterface-catalogue.json"} + ;; new in 4.9 + {:name :github :label "GitHub" :source "https://raw.githubusercontent.com/ogri-la/strongbox-catalogue/master/github-catalogue.json"}] + + :preferences {:addon-zips-to-keep 3 + :ui-selected-columns [:source :name :description + :installed-version :available-version ;; replaced in 5.0.0 + :game-version :uber-button] + :keep-user-catalogue-updated true}} + + :etag-db {}}] + (is (= expected (config/load-settings cli-opts cfg-file etag-db-file)))))) + +(deftest load-settings-7.0 + (testing "a standard config file circa 7.0 is loaded and parsed as expected" + (let [cli-opts {} + cfg-file (fixture-path "user-config-7.0.json") + etag-db-file (fixture-path "empty-map.json") + + expected {:cfg {:gui-theme :dark-green ;; new in 0.11, `:dark-green` new in 3.2.0 + :selected-catalogue :full ;; new in 0.10 + ;;:debug? true ;; removed in 0.12 + :addon-dir-list [{:addon-dir "/tmp/.strongbox-bar", :game-track :classic-tbc :strict? true} ;; `:classic-tbc` and `:strict?` added in 4.1 + {:addon-dir "/tmp/.strongbox-foo", :game-track :retail :strict? false}] + + ;; new in 1.0 + ;; new in 4.9, the set of available catalogues now includes a github catalogue + :catalogue-location-list (:catalogue-location-list config/default-cfg) + + ;; new in 0.12 + ;; moved to :cfg in 1.0 + :selected-addon-dir "/tmp/.strongbox-foo" + + ;; new in 3.1.0 + :preferences {:addon-zips-to-keep 3 + + :ui-selected-columns [:source :name :description + :combined-version ;; new default in 5.0.0 + :game-version :uber-button] + ;; new in 6.2.0 + :keep-user-catalogue-updated true}} + + :cli-opts {} + :file-opts {:gui-theme :dark-green + :selected-catalogue :full + :addon-dir-list [{:addon-dir "/tmp/.strongbox-bar", :game-track :classic-tbc, :strict? true} + {:addon-dir "/tmp/.strongbox-foo", :game-track :retail, :strict? false}] + :selected-addon-dir "/tmp/.strongbox-foo" + + ;; new in 1.0 + :catalogue-location-list [{:name :short :label "Short (default)" :source "https://raw.githubusercontent.com/ogri-la/strongbox-catalogue/master/short-catalogue.json"} + {:name :full :label "Full" :source "https://raw.githubusercontent.com/ogri-la/strongbox-catalogue/master/full-catalogue.json"} + ;; removed in 7.0.0 + ;;{:name :tukui :label "Tukui" :source "https://raw.githubusercontent.com/ogri-la/strongbox-catalogue/master/tukui-catalogue.json"} + ;; removed in 5.0.0 + ;;{:name :curseforge :label "Curseforge" :source "https://raw.githubusercontent.com/ogri-la/strongbox-catalogue/master/curseforge-catalogue.json"} + {:name :wowinterface :label "WoWInterface" :source "https://raw.githubusercontent.com/ogri-la/strongbox-catalogue/master/wowinterface-catalogue.json"} + ;; new in 4.9 + {:name :github :label "GitHub" :source "https://raw.githubusercontent.com/ogri-la/strongbox-catalogue/master/github-catalogue.json"}] + + :preferences {:addon-zips-to-keep 3 + :ui-selected-columns [:source :name :description + :installed-version :available-version ;; replaced in 5.0.0 + :game-version :uber-button] + :keep-user-catalogue-updated true}} + + :etag-db {}}] + (is (= expected (config/load-settings cli-opts cfg-file etag-db-file)))))) + diff --git a/test/strongbox/core_test.clj b/test/strongbox/core_test.clj index 4cb54614..36f1af11 100644 --- a/test/strongbox/core_test.clj +++ b/test/strongbox/core_test.clj @@ -6,6 +6,7 @@ [envvar.core :refer [with-env]] [me.raynes.fs :as fs] [taoensso.timbre :as log :refer [debug info warn error spy]] + [java-time :as jt] [strongbox.cli :as cli] [strongbox [addon :as addon :refer [downloaded-addon-fname]] @@ -264,6 +265,7 @@ expected [{:created-date "2011-01-04T05:42:23Z", :description "desc", :dirname "Addon3", + :dirsize 0 :download-count 3, :download-url "https://cdn.wowinterface.com/downloads/getfile.php?id=3", :game-track :retail, @@ -288,36 +290,10 @@ :updated-date "2019-07-03T07:11:47Z", :url "https://www.wowinterface.com/downloads/info3", :version "1.2.3"} - {:created-date "2011-01-04T05:42:23Z", - :description "desc", - :dirname "Addon4", - :download-count 4, - :download-url "https://www.tukui.org/addons.php?download=4", - :game-track :retail, - :group-id "https://www.tukui.org/addons.php?id=4", - :installed-game-track :retail, - :installed-version "1.2.3", - :interface-version 80200, - :label "Addon4", - :matched? true, - :name "addon4", - :primary? true, - :release-list [{:download-url "https://www.tukui.org/addons.php?download=4", - :game-track :retail, - :interface-version 80200, - :version "1.2.3"}], - :source "tukui", - :source-id 4, - :source-map-list [{:source "tukui", :source-id 4}], - :supported-game-tracks [:retail], - :tag-list [], - :update? false, - :updated-date "2019-07-03T07:11:47Z", - :url "https://www.tukui.org/addons.php?id=4", - :version "1.2.3"} {:created-date "2011-01-04T05:42:23Z", :description "desc", :dirname "Addon5", + :dirsize 0 :download-count 5, :download-url "https://github.com/author/addon5/releases/download/Addon5-v1.2.3/Addon5-v1.2.3.zip", :game-track :classic-tbc, @@ -408,6 +384,7 @@ expected [{:created-date "2011-01-04T05:42:23Z", :description "desc", :dirname "Addon3", + :dirsize 0 :download-count 3, :download-url "https://cdn.wowinterface.com/downloads/getfile.php?id=3", :game-track :retail ;; addon supports retail and classic, addon dir game track is set to retail @@ -432,36 +409,10 @@ :updated-date "2019-07-03T07:11:47Z", :url "https://www.wowinterface.com/downloads/info3", :version "1.2.3"} - {:created-date "2011-01-04T05:42:23Z", - :description "desc", - :dirname "Addon4", - :download-count 4, - :download-url "https://www.tukui.org/addons.php?download=4", - :game-track :retail, - :group-id "https://www.tukui.org/addons.php?id=4", - :installed-game-track :retail, - :installed-version "1.2.3", - :interface-version 80200, - :label "Addon4", - :matched? true, - :name "addon4", - :primary? true, - :release-list [{:download-url "https://www.tukui.org/addons.php?download=4", - :game-track :retail, - :interface-version 80200, - :version "1.2.3"}], - :source "tukui", - :source-id 4, - :source-map-list [{:source "tukui", :source-id 4}], - :supported-game-tracks [:retail], - :tag-list [], - :update? false, - :updated-date "2019-07-03T07:11:47Z", - :url "https://www.tukui.org/addons.php?id=4", - :version "1.2.3"} {:created-date "2011-01-04T05:42:23Z", :description "desc", :dirname "Addon5", + :dirsize 0 :download-count 5, :download-url "https://github.com/author/addon5/releases/download/Addon5-v1.2.3/Addon5-v1.2.3.zip", :game-track :classic-tbc @@ -496,38 +447,37 @@ (deftest check-for-addon-update (testing "the key `:update?` is set to `true` when the installed version doesn't match the catalogue version" - (let [;; we start off with a list of these called a catalogue. it's downloaded from github - catalogue {:tag-list [:auction-house] - :download-count 1 - :label "Every Addon" - :name "every-addon", - :source "curseforge", - :source-id 0 - :updated-date "2012-09-20T05:32:00Z", - :url "https://www.curseforge.com/wow/addons/every-addon"} - - ;; this is subset of the data the remote addon host (like curseforge) serves us - api-result {:latestFiles [{:downloadUrl "https://example.org/foo" - :displayName "v8.10.00" - :gameVersionFlavor "wow_retail", - :gameVersion ["7.0.0"] - :fileDate "2001-01-03T00:00:00.000Z", - :fileName "EveryAddon.zip" - :releaseType 1, - :exposeAsAlternative nil}]} - alt-api-result (assoc-in api-result [:latestFiles 0 :displayName] "v8.20.00") - - dummy-catalogue (catalogue/new-catalogue [catalogue]) + (let [;; we start off with a list of these called a catalogue + catalogue [{:label "Every Addon" + :name "every-addon", + :description "Does foo, only better." + :source "wowinterface", + :source-id 0 + :game-track-list [:retail] + :url "https://github.com/addons/every-addon" + :download-count 1 + :tag-list [:auction-house] + :updated-date "2012-09-20T05:32:00Z", + :created-date "2023-08-01T00:00:00Z"}] + + dummy-catalogue (catalogue/new-catalogue catalogue) + + ;; this is a subset of the data the remote addon host (like wowinterface) serves us + api-result [{:game-track :retail, + :UIVersion "v8.10.00"}] + + alt-api-result (assoc-in api-result [0 :UIVersion] "v8.20.00") fake-routes {;; catalogue "https://raw.githubusercontent.com/ogri-la/strongbox-catalogue/master/short-catalogue.json" {:get (fn [req] {:status 200 :body (utils/to-json dummy-catalogue)})} - ;; every-addon - "https://addons-ecs.forgesvc.net/api/v2/addon/0" + ;; every-addon 0 + "https://api.mmoui.com/v3/game/WOW/filedetails/0.json" {:get (fn [req] {:status 200 :body (utils/to-json api-result)})} - "https://addons-ecs.forgesvc.net/api/v2/addon/1" + ;; every-addon 1 + "https://api.mmoui.com/v3/game/WOW/filedetails/1.json" {:get (fn [req] {:status 200 :body (utils/to-json alt-api-result)})}}] (with-global-fake-routes-in-isolation fake-routes @@ -549,7 +499,7 @@ :name "every-addon", :group-id "doesntmatter" :primary? true, - :source "curseforge" + :source "wowinterface" :source-id 0} ;; the nfo data is simply merged over the top of the scraped toc data @@ -559,20 +509,16 @@ ;; in this case we have a catalogue of 1 and only interested in the first result result (first (core/db-match-installed-addon-list-with-catalogue (core/get-state :db) [toc])) - ;; previously done in above step, mooshing the installed addon and catalogue item together is - ;; now a separate step + ;; the installed addon result and catalogue item result are mooshed together toc-addon (core/moosh-addons toc (:catalogue-match result)) - alt-toc-addon (assoc toc-addon :source-id 1) ;; and what we 'expand' that data into - source-updates {:download-url "https://example.org/foo", + source-updates {:download-url "https://cdn.wowinterface.com/downloads/getfile.php?id=0" :version "v8.10.00" :game-track :retail - :release-list [{:download-url "https://example.org/foo", + :release-list [{:download-url "https://cdn.wowinterface.com/downloads/getfile.php?id=0" :game-track :retail, - :interface-version 70000, - :release-label "[WoW 7.0.0] EveryAddon", :version "v8.10.00"}]} alt-source-updates (assoc source-updates :version "v8.20.00") @@ -580,7 +526,8 @@ ;; after calling `check-for-update` we expect the result to be the merged sum of the below parts expected (merge toc-addon source-updates {:update? false}) - alt-expected (merge alt-toc-addon alt-source-updates {:update? true})] + alt-expected (merge alt-toc-addon alt-source-updates {:update? true :download-url "https://cdn.wowinterface.com/downloads/getfile.php?id=1"}) + alt-expected (assoc-in alt-expected [:release-list 0 :download-url] "https://cdn.wowinterface.com/downloads/getfile.php?id=1")] (is (= expected (core/check-for-update toc-addon))) (is (= alt-expected (core/check-for-update alt-toc-addon))))))))) @@ -1266,6 +1213,7 @@ ;;:version ... :description "Does what no other addon does, slightly differently", :dirname "EveryAddon", + :dirsize 0 :group-id "https://group.id/never/fetched", :installed-game-track :retail, :installed-version "1.2.3", @@ -1300,6 +1248,7 @@ expected {;;:ignore? false, ;; removed rather than set to false. :description "Does what no other addon does, slightly differently", :dirname "EveryAddon", + :dirsize 0 :group-id "https://group.id/never/fetched", :installed-game-track :retail, :installed-version "1.2.3", @@ -1343,8 +1292,10 @@ expected {:description "group record for the fetched addon", :dirname "EveryAddon-BundledAddon", + :dirsize 0 :group-addons [{:description "A useful addon that everyone bundles with their own.", :dirname "EveryAddon-BundledAddon", + :dirsize 0 :group-id "https://group.id/also/never/fetched", :ignore? true, @@ -1362,6 +1313,7 @@ {:description "Does what every addon does, just better", :dirname "EveryOtherAddon", + :dirsize 0 :group-id "https://group.id/also/never/fetched", :installed-game-track :retail, :installed-version "5.6.7", @@ -1421,6 +1373,7 @@ expected {:ignore? false, ;; explicit `false` rather than removed :description "Does what no other addon does, slightly differently", :dirname "EveryAddon", + :dirsize 0 :group-id "https://group.id/never/fetched", :installed-game-track :retail, :installed-version "1.2.3", @@ -1453,69 +1406,129 @@ expected-empty-search-state (assoc core/-search-state-template :term search-term)] (with-global-fake-routes-in-isolation fake-routes (with-running-app + + ;; we have 4 search results to start with + (cli/bump-search) + (Thread/sleep 50) ;; searching happens in the background + (is (= 4 (-> (core/get-state :search) :results first count))) + + ;; search for 'a' + ;; we should have three results: + ;; 1. "*A* New Simple Percent" + ;; 2. "Skins for *A*ddOns" + ;; 3. "Chinchill*a*" (cli/search search-term) - ;; searching happens in the background (Thread/sleep 50) - ;; we have one search result from a catalogue of 4 addons - (is (= 1 (-> (core/get-state :search) :results count))) + (is (= 3 (-> (core/get-state :search) :results first count))) + ;; empty the stale search state (core/empty-search-results) + (Thread/sleep 50) (is (= expected-empty-search-state (core/get-state :search))) + ;; do the search again without specifying a search term + ;; we should have three search results again (cli/bump-search) (Thread/sleep 50) - (is (= 1 (-> (core/get-state :search) :results count)))))))) + (is (= 3 (-> (core/get-state :search) :results first count)))))))) -(deftest -latest-strongbox-release +(deftest reset-search-state + (testing "search state can be cleared entirely" + (let [dummy-catalogue (slurp (fixture-path "catalogue--v2.json")) + fake-routes {"https://raw.githubusercontent.com/ogri-la/strongbox-catalogue/master/short-catalogue.json" + {:get (fn [req] {:status 200 :body dummy-catalogue})}} + search-term "a"] + (with-global-fake-routes-in-isolation fake-routes + (with-running-app + (cli/bump-search) + + ;; we have four results initially + (Thread/sleep 50) + (is (= 4 (-> (core/get-state :search) :results first count))) + + ;; search for 'a' + ;; we should have three results: + ;; 1. "*A* New Simple Percent" + ;; 2. "Skins for *A*ddOns" + ;; 3. "Chinchill*a*" + (cli/search search-term) + (Thread/sleep 50) + (is (= 3 (-> (core/get-state :search) :results first count))) + + ;; filter by host and we still have one search result + (cli/search-add-filter :source ["curseforge"]) + (Thread/sleep 50) + ;;(is (= {} (core/get-state :search))) + (is (= 1 (-> (core/get-state :search) :results first count))) + (is (= ["curseforge"] (core/get-state :search :filter-by :source))) + + ;; filter by tag and we still have one search result + (cli/search-add-filter :tag :unit-frames) + (Thread/sleep 50) + (is (= 1 (-> (core/get-state :search) :results count))) + (is (= #{:unit-frames} (core/get-state :search :filter-by :tag))) + + (core/reset-search-state!) + (Thread/sleep 50) + + ;; we have 4 search results from a catalogue of 4 addons + (is (= 4 (-> (core/get-state :search) :results first count))) + + ;; and all the filters have been removed + (is (nil? (core/get-state :search :term))) + (is (empty? (core/get-state :search :filter-by :tag))) + (is (nil? (core/get-state :search :filter-by :source)))))))) + +(deftest -download-strongbox-release (testing "standard github response for strongbox release data can be parsed and the release version extracted" (let [fake-routes {"https://api.github.com/repos/ogri-la/strongbox/releases/latest" {:get (fn [req] {:status 200 :body (slurp (fixture-path "github-strongbox-release.json"))})}} expected "4.3.0"] (with-global-fake-routes-in-isolation fake-routes - (is (= expected (core/-latest-strongbox-release))))))) + (is (= expected (core/-download-strongbox-release))))))) -(deftest -latest-strongbox-release--throttled +(deftest -download-strongbox-release--throttled (testing "throttled github response status for strongbox release data returns a :failed " (let [fake-routes {"https://api.github.com/repos/ogri-la/strongbox/releases/latest" {:get (fn [req] {:status 403 :reason-phrase "asdf"})}} expected :failed] (with-global-fake-routes-in-isolation fake-routes - (is (= expected (core/-latest-strongbox-release))))))) + (is (= expected (core/-download-strongbox-release))))))) -(deftest -latest-strongbox-release--unknown +(deftest -download-strongbox-release--unknown (testing "weird github response statuses for strongbox release data returns a :failed " (let [fake-routes {"https://api.github.com/repos/ogri-la/strongbox/releases/latest" {:get (fn [req] {:status 999 :reason-phrase "asdf"})}} expected :failed] (with-global-fake-routes-in-isolation fake-routes - (is (= expected (core/-latest-strongbox-release))))))) + (is (= expected (core/-download-strongbox-release))))))) -(deftest -latest-strongbox-release--malformed +(deftest -download-strongbox-release--malformed (testing "successful but malformed/unparseable github response for strongbox release data returns a :failed " (let [fake-routes {"https://api.github.com/repos/ogri-la/strongbox/releases/latest" {:get (fn [req] {:status 200 :body "asdf"})}} expected :failed] (with-global-fake-routes-in-isolation fake-routes - (is (= expected (core/-latest-strongbox-release))))))) + (is (= expected (core/-download-strongbox-release))))))) -(deftest latest-strongbox-release +(deftest latest-strongbox-release! (testing "standard github response for strongbox release data can be parsed and the release version extracted" (let [fake-routes {"https://api.github.com/repos/ogri-la/strongbox/releases/latest" {:get (fn [req] {:status 200 :body (slurp (fixture-path "github-strongbox-release.json"))})}} expected "4.3.0"] (with-running-app (with-global-fake-routes-in-isolation fake-routes - (is (= expected (core/latest-strongbox-release)))))))) + (is (= expected (core/latest-strongbox-release!)))))))) -(deftest latest-strongbox-release--throttled - (testing "throttled github response status for strongbox release data returns `nil`" +(deftest latest-strongbox-release!--throttled + (testing "throttled github response status for strongbox release data returns `:failed`" (let [fake-routes {"https://api.github.com/repos/ogri-la/strongbox/releases/latest" {:get (fn [req] {:status 403 :reason-phrase "asdf"})}}] (with-running-app (with-global-fake-routes-in-isolation fake-routes - (is (nil? (core/latest-strongbox-release)))))))) + (is (= :failed (core/latest-strongbox-release!)))))))) -(deftest latest-strongbox-release--subsequent-failure +(deftest latest-strongbox-release!--subsequent-failure (testing "once discovered, release versions are are stored in app state and not fetched again." (let [fake-routes {"https://api.github.com/repos/ogri-la/strongbox/releases/latest" {:get (fn [req] {:status 403 :reason-phrase "asdf"})}} @@ -1523,7 +1536,7 @@ (with-running-app (swap! core/state assoc :latest-strongbox-release expected) (with-global-fake-routes-in-isolation fake-routes - (is (= expected (core/latest-strongbox-release)))))))) + (is (= expected (core/latest-strongbox-release!)))))))) (deftest unsteady? (testing "a function that operates on addons can be wrapped to mark the addon as 'unsteady'" @@ -1708,8 +1721,10 @@ ;; after installing A, then B then C, we expect C to have cleanly replaced A and B expected {:description "group record for the EveryAddonThree addon", :dirname "EveryAddonOne", + :dirsize 0 :group-addons [{:description "Does what no other addon does, slightly differently.", :dirname "EveryAddonOne", + :dirsize 0 :group-id "https://example.com/EveryAddonThree", :installed-game-track :retail, :installed-version "1.2.3", @@ -1724,6 +1739,7 @@ :supported-game-tracks [:retail]} {:description "Does what no other addon does, slightly differently.", :dirname "EveryAddonThree", + :dirsize 0 :group-id "https://example.com/EveryAddonThree", :installed-game-track :retail, :installed-version "1.2.3", @@ -1738,6 +1754,7 @@ :supported-game-tracks [:retail]} {:description "Does what no other addon does, slightly differently.", :dirname "EveryAddonTwo", + :dirsize 0 :group-id "https://example.com/EveryAddonThree", :installed-game-track :retail, :installed-version "1.2.3", @@ -1851,3 +1868,53 @@ expected [toc]] (is (= expected (core/db-match-installed-addon-list-with-catalogue db installed-addon-list)))))) + +(deftest db-addon-by-source-and-source-id + (let [expected helper/addon-summary + db [helper/addon-summary] + {:keys [source source-id]} helper/addon-summary] + (is (= expected (core/db-addon-by-source-and-source-id db source source-id))) + (is (nil? (core/db-addon-by-source-and-source-id db "wowinterface" "foo"))))) + +;; --- + +(deftest refresh-user-catalogue-item + (testing "individual addons can be refreshed, writing the changes to disk afterwards." + (let [user-catalogue (catalogue/new-catalogue [helper/addon-summary]) + new-addon (merge helper/addon-summary {:updated-date "2022-02-02T02:02:02"}) + expected (assoc user-catalogue :addon-summary-list [new-addon]) + db []] + (with-running-app + (swap! core/state assoc :user-catalogue user-catalogue) + (core/write-user-catalogue!) + (with-redefs [core/find-addon (fn [& args] new-addon)] + (core/refresh-user-catalogue-item helper/addon-summary db)) + (is (= expected (core/get-state :user-catalogue))))))) + +(deftest refresh-user-catalogue-item--no-catalogue + (testing "looking for an addon that doesn't exist in the catalogue isn't a total failure" + (with-running-app + (let [db []] + (is (nil? (core/refresh-user-catalogue-item helper/addon-summary db))))))) + +(deftest refresh-user-catalogue-item--unhandled-exception + (testing "unhandled exceptions while refreshing a user-catalogue item isn't a total failure" + (with-running-app + (with-redefs [core/find-addon (fn [& args] (throw (Exception. "catastrophe!")))] + (let [db []] + (is (nil? (core/refresh-user-catalogue-item helper/addon-summary db)))))))) + +(deftest scheduled-user-catalogue-refresh + (with-running-app + (java-time/with-clock (java-time/fixed-clock "2100-01-01T00:00:00Z") + (cli/set-preference :keep-user-catalogue-updated true) + (is (true? (core/get-state :cfg :preferences :keep-user-catalogue-updated))) + (swap! core/state assoc :user-catalogue (catalogue/new-catalogue [])) + (is (true? (core/refresh-user-catalogue? + (core/get-state :cfg :preferences :keep-user-catalogue-updated) + (core/get-state :user-catalogue :datestamp)))) + (let [expected ["user-catalogue not updated in the last 9999 days, automatic refresh triggered." + "downloading 'full' catalogue" + "refreshing \"user-catalogue.json\", this may take a minute ..." + "\"user-catalogue.json\" has been refreshed"]] + (is (= expected (logging/buffered-log :info (core/scheduled-user-catalogue-refresh)))))))) diff --git a/test/strongbox/jfx_test.clj b/test/strongbox/jfx_test.clj index a5dbb67a..322fa90c 100644 --- a/test/strongbox/jfx_test.clj +++ b/test/strongbox/jfx_test.clj @@ -78,6 +78,7 @@ {:fx/type :menu-item, :text "skinny"} {:fx/type :menu-item, :text "fat"} jfx/separator + {:fx/type :check-menu-item, :text "starred", :selected false} {:fx/type :check-menu-item, :text "browse", :selected false} {:fx/type :check-menu-item, :text "source", :selected true} {:fx/type :check-menu-item, :text "ID", :selected true} @@ -87,11 +88,12 @@ {:fx/type :check-menu-item, :text "tags", :selected false} {:fx/type :check-menu-item, :text "created", :selected false} {:fx/type :check-menu-item, :text "updated", :selected false} - {:fx/type :check-menu-item, :text "installed", :selected false} - {:fx/type :check-menu-item, :text "available", :selected false} - {:fx/type :check-menu-item, :text "version", :selected false} - {:fx/type :check-menu-item, :text "WoW", :selected false} - {:fx/type :check-menu-item, :text "uber-button", :selected false}] + {:fx/type :check-menu-item, :text "size", :selected false} + {:fx/type :check-menu-item, :text "installed version", :selected false} + {:fx/type :check-menu-item, :text "available version", :selected false} + {:fx/type :check-menu-item, :text "installed+available version", :selected false} + {:fx/type :check-menu-item, :text "game version (WoW)", :selected false} + {:fx/type :check-menu-item, :text "über button", :selected false}] selected-columns [:foo :bar :baz :source :source-id] diff --git a/test/strongbox/test_helper.clj b/test/strongbox/test_helper.clj index e958b260..fe47e23b 100644 --- a/test/strongbox/test_helper.clj +++ b/test/strongbox/test_helper.clj @@ -177,7 +177,11 @@ ;; latest strongbox version "https://api.github.com/repos/ogri-la/strongbox/releases/latest" - {:get (fn [req] {:status 200 :body "{\"tag_name\": \"0.0.0\"}"})}}] + {:get (fn [req] {:status 200 :body "{\"tag_name\": \"0.0.0\"}"})} + + ;; github requests remaining + "https://api.github.com/rate_limit" + {:get (fn [req] {:status 200 :body (utils/to-json {:resources {:core {:limit 0, :remaining 0, :reset 978307200, :used 0, :resource "core"}}})})}}] (try (debug "stopping application if it hasn't already been stopped") (main/stop) diff --git a/test/strongbox/toc_test.clj b/test/strongbox/toc_test.clj index 4b84768f..ad820cf6 100644 --- a/test/strongbox/toc_test.clj +++ b/test/strongbox/toc_test.clj @@ -87,6 +87,7 @@ SomeAddon.lua") toc-file-path (join addon-path "SomeAddon.toc") expected [{:name "addon-name" :dirname "SomeAddon" + :dirsize 0 :label "Addon Name" :description "Description of the addon here" :interface-version 80205 diff --git a/test/strongbox/utils_test.clj b/test/strongbox/utils_test.clj index f0253a55..22408312 100644 --- a/test/strongbox/utils_test.clj +++ b/test/strongbox/utils_test.clj @@ -83,17 +83,56 @@ (java-time/with-clock (java-time/fixed-clock "2001-01-02T00:00:00Z") (is (= (utils/days-between-then-and-now "2001-01-01") 1))))) +(deftest older-than? + (java-time/with-clock (java-time/fixed-clock "2023-01-01T00:00:00Z") + (let [cases [[["2001-01-01" 0 :days] true] + [["2001-01-01" 0 :hours] true] + + [["2001-01-01" 1 :days] true] + [["2001-01-01" 1 :hours] true] + + [["2079-12-31" 1 :days] false] + [["2079-12-31" 1 :hours] false] + + [["2079-12-31" 0 :days] false] + [["2079-12-31" 0 :hours] false] + + ;; right now is 2023-01-01T00:00:00Z + + [["2023-01-01T00:00:00Z" 0 :minutes] false] + + [["2022-12-31T23:58:59Z" 1 :minutes] true] + [["2023-01-01T00:00:00Z" 1 :minutes] false]]] + + (doseq [[[then threshold period] expected] cases] + (is (= expected (utils/older-than? then threshold period)), (format "case %s %s %s" then threshold period)))))) + +(deftest older-than?--bad-period + (is (thrown? java.lang.IllegalArgumentException (utils/older-than? "2001-01-01" 12 :parsecs)))) + (deftest file-older-than (testing "files whose modification times are older than N hours" - (java-time/with-clock (java-time/fixed-clock "1970-01-01T02:00:00Z") ;; jan 1st 1970, 2 am - (let [path (utils/join *temp-dir-path* "foo")] - (try - (fs/touch path 0) ;; created Jan 1st 1970 - (.setLastModified (fs/file path) 0) ;; modified Jan 1st 1970 - (is (utils/file-older-than path 1)) - (is (not (utils/file-older-than path 3))) - (finally - (fs/delete path))))))) + (let [path (utils/join *temp-dir-path* "foo")] + (fs/touch path 0) ;; created Jan 1st 1970 + (.setLastModified (fs/file path) 0) ;; modified Jan 1st 1970 + (try + ;; 2 minutes past the epoch + (java-time/with-clock (java-time/fixed-clock "1970-01-01T00:02:00Z") + (is (utils/file-older-than path 1 :minutes)) + (is (not (utils/file-older-than path 3 :minutes)))) + + ;; 2 hours past the epoch + (java-time/with-clock (java-time/fixed-clock "1970-01-01T02:00:00Z") + (is (utils/file-older-than path 1 :hours)) + (is (not (utils/file-older-than path 3 :hours)))) + + ;; 1 day, 1 second past the epoch + (java-time/with-clock (java-time/fixed-clock "1970-01-02T00:00:01Z") + (is (utils/file-older-than path 1 :days)) + (is (not (utils/file-older-than path 2 :days)))) + + (finally + (fs/delete path)))))) (deftest nilable (let [cases [[nil nil] @@ -467,14 +506,15 @@ (deftest published-before-classic? (let [cases [["2001" nil] - ["2001-01-01" nil] + ;;["2001-01-01" nil] + ["2001-01-01" true] ;; 2023-05-12: utc timezone now appended to y-m-d strings ["2001-01-01T01:00" nil] ["2001-01-01T01:00:00Z" true] ["2019-08-25T23:59:59Z" true] [constants/release-of-wow-classic false] ["2019-08-26T00:00:01Z" false]]] (doseq [[given expected] cases] - (is (= expected (utils/published-before-classic? given)))))) + (is (= expected (utils/published-before-classic? given)) (str "case: " given))))) (deftest source-map (let [cases [[nil {}] @@ -534,7 +574,7 @@ :debug (let [fn1-ref (fn1) ;; ensure fn1 is always executed first. it will always finish last. - _ (Thread/sleep 7) + _ (Thread/sleep 5) fn2-ref (fn2)] @fn1-ref @fn2-ref)) @@ -564,21 +604,94 @@ (doseq [[given expected] cases] (is (= expected (utils/patch-name given)))))) -(deftest compressed-slurp - (let [expected "foo!"] - (is (= expected (utils/decompress-bytes (utils/compressed-slurp "foo.txt")))))) +(deftest filesize + (let [cases [[0 "0"] ;; special case + [1 "1.0 B"] + [1000 "1.0 KB"] + [1024 "1.0 KB"] + [10000 "10.0 KB"] + [100000 "100.0 KB"] + [1000000 "1.0 MB"] + [1500000 "1.5 MB"] -(deftest compressed-slurp--not-found - (is (nil? (utils/compressed-slurp "file-that-definitely-does-not-exist.txt")))) + [-1 "-1.0 B"] -(deftest decompress-bytes--empty - (is (nil? (utils/decompress-bytes (.getBytes "")))) - (is (nil? (utils/decompress-bytes nil)))) + [nil ""] + ["foo" ""] + [:foo ""] + [{:foo "bar"} ""]]] + + (doseq [[given expected] cases] + (is (= expected (utils/filesize given)))))) -(deftest decompress-bytes--not-compressed - (let [result (try - (utils/decompress-bytes (.getBytes "!")) - (catch java.io.IOException ioe - ioe))] - (is (instance? java.io.IOException result)) - (is (= "Stream is not in the BZip2 format" (.getMessage result))))) +(deftest format-number + (with-redefs [utils/user-locale (constantly java.util.Locale/GERMAN)] + (let [cases [[1 "1"] + [10 "10"] + [1000 "1.000"]]] + (doseq [[given expected] cases] + (is (= expected (utils/format-number given)))))) + + (with-redefs [utils/user-locale (constantly java.util.Locale/UK)] + (let [cases [[1 "1"] + [10 "10"] + [1000 "1,000"]]] + (doseq [[given expected] cases] + (is (= expected (utils/format-number given))))))) + +(deftest unix-time-to-datetime + (let [cases [[0 "1970-01-01T00:00:00Z"] + [1 "1970-01-01T00:00:01Z"] + [61 "1970-01-01T00:01:01Z"] + [1685623484 "2023-06-01T12:44:44Z"]]] + (doseq [[given expected] cases] + (is (= expected (utils/unix-time-to-datetime given)))))) + +(deftest minutes-from-now + (let [cases [["2023-06-01T00:00:00Z" 0] + ["2023-06-01T00:00:01Z" 0] + ["2023-06-01T00:00:10Z" 0] + ["2023-06-01T00:01:00Z" 1] + ["2023-06-01T00:10:00Z" 10] + ["2023-06-01T01:00:00Z" 60] + + ["2023-05-31T23:59:59Z" 0] + ["2023-05-31T23:59:00Z" -1] + ["2023-05-31T23:50:00Z" -10] + ["2023-05-31T23:00:00Z" -60]]] + (with-redefs [utils/now (constantly "2023-06-01T00:00:00Z")] + (doseq [[given expected] cases] + (is (= expected (utils/minutes-from-now given))))))) + +(deftest pretty-print-keyword + (let [cases [[nil nil] + [:foo "foo"] + [:foo-bar "foo bar"] + [:foo/bar "foo / bar"] + [:foo/bar-baz "foo / bar baz"]]] + (doseq [[given expected] cases] + (is (= expected (utils/pretty-print-keyword given)))))) + +(deftest pretty-print-value + (let [cases [[nil "(none)"] + + [:foo "foo"] + [:foo/bar "bar"] + + [1 "1"] + [100 "100"] + + [{} "(empty)"] + [{:foo []} "foo: (empty)"] + [{:foo :bar} "foo: bar"] + [{:foo {:bar :baz}} "foo: bar: baz"] + [{:foo {:bar :baz} :bup :boo} "bup: boo, foo: bar: baz"] ;; non-deterministic ... + + [[] "(empty)"] + [[1,2,3] "1, 2, 3"] + [[{}, 1] "(empty), 1"] + + [true "True"] + [false "False"]]] + (doseq [[given expected] cases] + (is (= expected (utils/pretty-print-value given))))))