Added filter by filesystem location + bonus tooltip on overflowed filter names #518

FinalDoom · 2022-02-20T06:34:05Z

New filter section that allows viewing space usage/torrent count by directory in tree format.
Filter names overflow with ellipses, and in the case of root directories, this is common: so I added a title-based tooltip to filter elements where the text is overflowed. Only appears on overflowed elements, and on mouse event. Not sure if equivalent touch events would work the same way.

No issue submitted, just something I wanted to make.

Breaking change (changes that break backward compatibility of public API or CLI - semver MAJOR)
New feature (non-breaking change which adds functionality - semver MINOR)
Bug fix (non-breaking change which fixes an issue - semver PATCH)

codecov · 2022-02-20T06:36:54Z

Codecov Report

Attention: Patch coverage is 95.55556% with 2 lines in your changes missing coverage. Please review.

Project coverage is 73.23%. Comparing base (1510940) to head (916f8de).
Report is 46 commits behind head on master.

Files with missing lines	Patch %	Lines
server/services/taxonomyService.ts	97.67%	1 Missing ⚠️
server/util/fileUtil.ts	50.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #518      +/-   ##
==========================================
+ Coverage   72.94%   73.23%   +0.28%     
==========================================
  Files          62       62              
  Lines       11375    11409      +34     
  Branches      951      969      +18     
==========================================
+ Hits         8298     8355      +57     
+ Misses       3063     3040      -23     
  Partials       14       14

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

mouzzampk2014 · 2022-02-20T07:36:50Z

This is brilliant, thanks for working on this.

jesec · 2022-02-24T01:56:37Z

The general idea looks OK, but the operations could get prohibitively expensive when there are many entries.

FinalDoom · 2022-02-24T07:09:11Z

Open to other algorithms, but I can attest it takes minimal amount of time more than master with ~6k torrents.

And with 10^5 directories (actual insanity IMO); 10 directories each with 10 children, each with 10 children, and so on, 5 deep.

Would it be worth making a default-disabled setting to enable the feature?

jesec · 2022-02-25T21:44:50Z

Open to other algorithms, but I can attest it takes minimal amount of time more than master with ~6k torrents. And with 10^5 directories (actual insanity IMO); 10 directories each with 10 children, each with 10 children, and so on, 5 deep.

Numbers look acceptable, but I would assume there is a non-trivial amount of associated CPU usage. I am reluctant to add a potentially expensive operation to the poll routine. We can probably employ more memoization.

Would it be worth making a default-disabled setting to enable the feature?

Not a good idea when the server-side functions are involved. There could be API concerns.

I am tied up with some other matters at hand, and I will take a deeper look next week.

boeto · 2022-11-12T04:05:27Z

When will it be merge? Expect this awesome feature

trim21 · 2023-12-17T20:01:08Z

can you resolve conflicts？

FinalDoom · 2023-12-19T22:37:01Z

I can, are you looking to actually merge it? (don't wanna play the fix/wait/fix/wait game)

trim21 · 2023-12-20T04:48:55Z

I can, are you looking to actually merge it? (don't wanna play the fix/wait/fix/wait game)

yes

update master

FinalDoom · 2023-12-21T01:22:05Z

Should be good now, also changed the mouseover to a hook, should be nicer.

trim21 · 2023-12-21T01:37:53Z

give me some time to test this locally

trim21 · 2023-12-21T11:28:15Z

this doesn't look right, in my understand shouldn't this display like a tree or multiple trees? but some non-root elements are displayed at top level

download dir here are /srv/dev-disk-by-*/.../... and /export/3t-1/...

trim21 · 2023-12-21T11:43:42Z

also, just a question, is it possible to build trees at client instead of server like tags?

FinalDoom · 2023-12-22T00:35:18Z

shouldn't this display like a tree or multiple trees?

I'll have to take a look at it again, as it's been a while; but iirc the trees are built by comparing common paths, removing prefixes, and traversing down. I think the paths come from those reported on the torrent metadata.. so is it possible those torrents are using relative paths or something starting with the hex?

Which root path should those non-root elements be under?

is it possible to build trees at client instead of server like tags?

Probably? I'm guessing I mimicked something that needed to be server-side, but again it's been a while. I'll have to take a look at what metadata ends up on the client.

trim21 · 2023-12-22T07:29:54Z

so is it possible those torrents are using relative paths or something starting with the hex?

Which root path should those non-root elements be under?

no, they are all abs path.

My torrents' download directories:

/srv/dev-disk-by-uuid-58b9efe4-428e-40d1-8d3f-97d8a5bac12a/u2-small/{hex}
/srv/dev-disk-by-uuid-3c509d6a-504e-4d71-a19d-5f81c62d9902/...
/srv/dev-disk-by-uuid-41a30911-024f-4bd6-bb21-2aa90bed52e9/...
/srv/dev-disk-by-uuid-58b9efe4-428e-40d1-8d3f-97d8a5bac12a/...
/export/3t-1/downloads/...

so all hex elements and elements starts by dev-disk-by-uuid should be non-root elements, only /srv/ and /export/3t-1/downloads should be root element.

FinalDoom · 2023-12-23T00:38:48Z

Very odd. I'll have to take another look and make some test cases when I get a chance.

trim21 · 2023-12-23T14:05:20Z

looks like I have mixed case directories and they cause this bug:

/srv/dev-disk-by-uuid-58b9efe4-428e-40d1-8d3f-97d8a5bac12a/U2-small/e9b3a6c75170e0c14701e7095b94412cfd4c7c7d
/srv/dev-disk-by-uuid-58b9efe4-428e-40d1-8d3f-97d8a5bac12a/u2-small/00ac7be5eb069dd8af992fca3a40b3755aaad846

trim21 · 2023-12-23T14:25:15Z

console.log(
  [
    '/srv/dev-disk-by-uuid-58b9efe4-428e-40d1-8d3f-97d8a5bac12a/u2-small/231d4f952201ba96628a9520740554c6ae6837db',
    '/srv/dev-disk-by-uuid-58b9efe4-428e-40d1-8d3f-97d8a5bac12a/U2-small/232e75cf79187a07a7507f82275f1e8d3f7ddf73',
    '/srv/dev-disk-by-uuid-58b9efe4-428e-40d1-8d3f-97d8a5bac12a/u2-small/2364ebde4155eaf3267af1011b4dbc9cc75d99e1',
  ].sort((a, b) => {
    return a.localeCompare(b);
  }),
);

=>

[
  '/srv/dev-disk-by-uuid-58b9efe4-428e-40d1-8d3f-97d8a5bac12a/u2-small/231d4f952201ba96628a9520740554c6ae6837db',
  '/srv/dev-disk-by-uuid-58b9efe4-428e-40d1-8d3f-97d8a5bac12a/U2-small/232e75cf79187a07a7507f82275f1e8d3f7ddf73',
  '/srv/dev-disk-by-uuid-58b9efe4-428e-40d1-8d3f-97d8a5bac12a/u2-small/2364ebde4155eaf3267af1011b4dbc9cc75d99e1'
]

server/services/taxonomyService.ts

trim21 · 2023-12-24T09:10:52Z

lgtm, only one minor problem

FinalDoom · 2023-12-24T10:34:08Z

I have a reduced solution that can probably be moved client-side; but case-insensitivity causes problems when filtering torrents. I think the solution is for the tree to be client-side, and the server can tell the client if the OS it's running on is case-insensitive or not, allowing insensitive tree and filtering

The current solution means client has no clue on insensitivity, so filtering on non-matching lowercased names omits torrents incorrectly.

Another option would be to always treat everything as case sensitive, and you'd end up with duplicate paths where casing doesn't match.

But it's late, so I'll give it a shot tomorrow.

trim21 · 2023-12-24T10:44:38Z

I have a reduced solution that can probably be moved client-side; but case-insensitivity causes problems when filtering torrents. I think the solution is for the tree to be client-side, and the server can tell the client if the OS it's running on is case-insensitive or not, allowing insensitive tree and filtering

The current solution means client has no clue on insensitivity, so filtering on non-matching lowercased names omits torrents incorrectly.

Another option would be to always treat everything as case sensitive, and you'd end up with duplicate paths where casing doesn't match.

But it's late, so I'll give it a shot tomorrow.

after reviewing the code I think there is no need to move it to client side， current code looks fine

trim21 · 2023-12-24T10:46:09Z

handling case-insensitivity may introduce too much extra complexity，which will need to handle every component of path，I think current code works fine

FinalDoom · 2024-01-02T05:59:22Z

I looked a bit at moving the treeing to client-side:

I'm guessing there's some reason for building the tag list and other sidebar filters server-side, whether it's shared efficiency for multiple clients, the taxonomy diff data savings, or something else. I reduced the complexity of computing the tree and just stored the counts/data size in-tree, so it should be a little faster/simpler overall.

There may be an argument for moving filter value computation to the client, but I think matching the existing sidebar filter logic makes more sense for now.

Just treat everything as case-sensitive. Also flood could be on a different OS than the torrent application, which would also cause mismatches.

trim21 · 2024-01-02T17:29:44Z

please do not force push

FinalDoom · 2024-01-02T18:29:56Z

I don't see what's failing in the tests. It names the new test file but there's no failing test or other message. It passes locally. Github doesn't have a way to rerun workflows?

FinalDoom · 2024-01-02T19:46:01Z

please do not force push

It's in my repo and rebasing creates better changeset history, why not?

trim21 · 2024-01-03T10:26:14Z

please do not force push

It's in my repo and rebasing creates better changeset history, why not?

Force-push causes inconvenient in code reviewing

trim21 · 2024-02-11T07:55:02Z

server/util/fileUtil.ts

-      return true;
-    }
-    return false;
+    return !!realPath?.startsWith(allowedPath);


please do not touch this file

jesec force-pushed the master branch from b0ead6f to 6ff622c Compare September 13, 2022 06:36

jesec force-pushed the master branch from 07e5b1e to 8ab421a Compare June 11, 2023 08:05

FinalDoom and others added 2 commits December 20, 2023 17:12

Merge pull request #1 from jesec/master

e20a2f0

update master

Added filter by filesystem loc+overflow filt ttip

bc6b386

FinalDoom force-pushed the filter-by-location branch from 26ba089 to 4e82634 Compare December 21, 2023 01:13

Make tooltip display more reactive

f783f68

FinalDoom force-pushed the filter-by-location branch from 4e82634 to f783f68 Compare December 21, 2023 01:21

trim21 reviewed Dec 23, 2023

View reviewed changes

server/services/taxonomyService.ts Outdated Show resolved Hide resolved

Update server/services/taxonomyService.ts

5d3aaa1

trim21 reviewed Dec 24, 2023

View reviewed changes

server/services/taxonomyService.ts Outdated Show resolved Hide resolved

Fix case sensitivity, add tests, remove shift-click in tree filters

0900280

FinalDoom force-pushed the filter-by-location branch from 17ec3e4 to 0900280 Compare January 2, 2024 05:59

FinalDoom force-pushed the filter-by-location branch from a87a45f to 84c54a3 Compare January 2, 2024 17:26

Remove case-insensitivity as it breaks client-side filtering.

4a14a8d

Just treat everything as case-sensitive. Also flood could be on a different OS than the torrent application, which would also cause mismatches.

FinalDoom force-pushed the filter-by-location branch from 84c54a3 to 4a14a8d Compare January 2, 2024 17:29

Don't fail for already-exited process.

3baa607

Merge branch 'master' into filter-by-location

916f8de

trim21 requested changes Feb 11, 2024

View reviewed changes

server/util/fileUtil.ts

return true;

}

return false;

return !!realPath?.startsWith(allowedPath);

Copy link

Collaborator

trim21 Feb 11, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please do not touch this file

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added filter by filesystem location + bonus tooltip on overflowed filter names #518

Added filter by filesystem location + bonus tooltip on overflowed filter names #518

FinalDoom commented Feb 20, 2022

codecov bot commented Feb 20, 2022 •

edited

Loading

mouzzampk2014 commented Feb 20, 2022

jesec commented Feb 24, 2022

FinalDoom commented Feb 24, 2022 •

edited

Loading

jesec commented Feb 25, 2022

boeto commented Nov 12, 2022

trim21 commented Dec 17, 2023

FinalDoom commented Dec 19, 2023

trim21 commented Dec 20, 2023

FinalDoom commented Dec 21, 2023

trim21 commented Dec 21, 2023

trim21 commented Dec 21, 2023 •

edited

Loading

trim21 commented Dec 21, 2023 •

edited

Loading

FinalDoom commented Dec 22, 2023

trim21 commented Dec 22, 2023 •

edited

Loading

FinalDoom commented Dec 23, 2023

trim21 commented Dec 23, 2023

trim21 commented Dec 23, 2023 •

edited

Loading

trim21 commented Dec 24, 2023

FinalDoom commented Dec 24, 2023

trim21 commented Dec 24, 2023 •

edited

Loading

trim21 commented Dec 24, 2023 •

edited

Loading

FinalDoom commented Jan 2, 2024

trim21 commented Jan 2, 2024 •

edited

Loading

FinalDoom commented Jan 2, 2024

FinalDoom commented Jan 2, 2024

trim21 commented Jan 3, 2024

trim21 Feb 11, 2024

Added filter by filesystem location + bonus tooltip on overflowed filter names #518

Are you sure you want to change the base?

Added filter by filesystem location + bonus tooltip on overflowed filter names #518

Conversation

FinalDoom commented Feb 20, 2022

codecov bot commented Feb 20, 2022 • edited Loading

Codecov Report

mouzzampk2014 commented Feb 20, 2022

jesec commented Feb 24, 2022

FinalDoom commented Feb 24, 2022 • edited Loading

jesec commented Feb 25, 2022

boeto commented Nov 12, 2022

trim21 commented Dec 17, 2023

FinalDoom commented Dec 19, 2023

trim21 commented Dec 20, 2023

FinalDoom commented Dec 21, 2023

trim21 commented Dec 21, 2023

trim21 commented Dec 21, 2023 • edited Loading

trim21 commented Dec 21, 2023 • edited Loading

FinalDoom commented Dec 22, 2023

trim21 commented Dec 22, 2023 • edited Loading

FinalDoom commented Dec 23, 2023

trim21 commented Dec 23, 2023

trim21 commented Dec 23, 2023 • edited Loading

trim21 commented Dec 24, 2023

FinalDoom commented Dec 24, 2023

trim21 commented Dec 24, 2023 • edited Loading

trim21 commented Dec 24, 2023 • edited Loading

FinalDoom commented Jan 2, 2024

trim21 commented Jan 2, 2024 • edited Loading

FinalDoom commented Jan 2, 2024

FinalDoom commented Jan 2, 2024

trim21 commented Jan 3, 2024

trim21 Feb 11, 2024

Choose a reason for hiding this comment

codecov bot commented Feb 20, 2022 •

edited

Loading

FinalDoom commented Feb 24, 2022 •

edited

Loading

trim21 commented Dec 21, 2023 •

edited

Loading

trim21 commented Dec 21, 2023 •

edited

Loading

trim21 commented Dec 22, 2023 •

edited

Loading

trim21 commented Dec 23, 2023 •

edited

Loading

trim21 commented Dec 24, 2023 •

edited

Loading

trim21 commented Dec 24, 2023 •

edited

Loading

trim21 commented Jan 2, 2024 •

edited

Loading