Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding tag community, database and REST API endpoints #6396

Merged
merged 8 commits into from
Oct 15, 2021

Conversation

drew2a
Copy link
Contributor

@drew2a drew2a commented Sep 30, 2021

This PR contains core-side logic for #6214

Design decisions behind the Community:

  • Tag operations gossiped transitively across the network
  • Pull-based gossip is used
  • For tags' integrity check ipv8 signatures are used
  • The simplified algorithm is: "Request 10 tags from a random peer every 5 seconds"

Design decisions behind the DB:

       class TorrentTagOp(db.Entity):
            id = orm.PrimaryKey(int, auto=True)

            torrent_tag = orm.Required(lambda: TorrentTag)
            peer = orm.Required(lambda: Peer)

            operation = orm.Required(int)
            time = orm.Required(int)
            signature = orm.Required(bytes)
            updated_at = orm.Required(datetime.datetime, default=datetime.datetime.utcnow)

            orm.composite_key(torrent_tag, peer)
  • We will use two counters to access information about added and removed tags without aggregation:
        class TorrentTag(db.Entity):
            ...
            added_count = orm.Required(int, default=0)
            removed_count = orm.Required(int, default=0)
  • We will use an indicator to distinguish local user operations and remote user operations:
        class TorrentTag(db.Entity):
            ...
            local_operation = orm.Optional(int) 

@drew2a drew2a force-pushed the feature/tag_community branch from acce8af to cef9e7d Compare September 30, 2021 21:04
@drew2a drew2a force-pushed the feature/tag_community branch 2 times, most recently from d7c8429 to efde2e4 Compare October 6, 2021 17:10
@drew2a drew2a force-pushed the feature/tag_community branch 3 times, most recently from 45fe78e to bbcec9f Compare October 7, 2021 19:34
@drew2a drew2a marked this pull request as ready for review October 7, 2021 19:41
@drew2a drew2a requested review from a team, ichorid, devos50 and kozlovsky and removed request for a team October 7, 2021 19:41
@drew2a drew2a requested a review from xoriole October 7, 2021 19:47
@devos50
Copy link
Contributor

devos50 commented Oct 8, 2021

Please note: I will add more tests in the next PR. This PR should be merged to main to let @devos50 work on the GUI part.

Feel free to add these tests in this PR, I can just rebase my branch when you make changes (adding tests to this PR should not merge conflict with my work). 👍

@devos50 devos50 changed the title Add TagComponent Adding tag community, database and REST API endpoints Oct 8, 2021
@drew2a drew2a force-pushed the feature/tag_community branch 2 times, most recently from 5f95718 to 333fa1e Compare October 8, 2021 14:57
ichorid
ichorid previously approved these changes Oct 8, 2021
@devos50
Copy link
Contributor

devos50 commented Oct 8, 2021

@drew2a I notice that there are a few uncovered lines which could probably be covered with a carefully designed test (e.g., in the TagCommunity class). Were you planning on adding additional tests in this PR?

Also, I think the remaining pylint errors can be fixed 👍

@drew2a drew2a force-pushed the feature/tag_community branch from 798608d to a016b30 Compare October 11, 2021 15:37
@drew2a drew2a force-pushed the feature/tag_community branch 2 times, most recently from 09b0b88 to 7b47c18 Compare October 13, 2021 12:51
@drew2a drew2a marked this pull request as ready for review October 13, 2021 13:15
@drew2a drew2a force-pushed the feature/tag_community branch 3 times, most recently from 8f3221f to 2841e72 Compare October 14, 2021 10:28
@drew2a drew2a force-pushed the feature/tag_community branch 2 times, most recently from 50e9c9b to 8305fef Compare October 15, 2021 13:43
Filtering tags in the REST API

Simplify test_get_my_channel_tags_xxx


Add TagDb.get_tags_operations_for_gossip()


Change timer to counter


Add lower-case validation to TagValidator


Converting tags to lower-case in REST API

Ensure that random tags during tests are lowercase

Extend Tests


Apply ipv8 dataclass


Refactor TagCommunity


PR fixes


Add TagSerializer


Refactoring DB: counter for peer, timestamp for operations


Appending tags to search results

Also extended the search GUI test to return more results.

PyLint


Refactor message raw serialisation


Fix merge conflicts


PR fixes


Remove unused const
@drew2a drew2a force-pushed the feature/tag_community branch from 8305fef to cb7db06 Compare October 15, 2021 13:59
@sonarqubecloud
Copy link

Kudos, SonarCloud Quality Gate passed!    Quality Gate passed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 0 Code Smells

No Coverage information No Coverage information
0.0% 0.0% Duplication

@devos50
Copy link
Contributor

devos50 commented Oct 15, 2021

A test on Windows 64-bit is failing:

def test_calculate_category_invalid_announce_list():
        torrent_info = {b"info": {b"name": b"term1", b"length": 1234},
                        b"announce-list": [[]], b"comment": b"lorem ipsum"}
>       assert default_category_filter.calculateCategory(torrent_info, "my torrent") == "xxx"
E       AssertionError: assert 'other' == 'xxx'
E         - xxx
E         + other

src\tribler-core\tribler_core\components\metadata_store\category_filter\tests\test_category.py:33: AssertionError

That's unexpected since this test looks deterministic. Let's see if this persists.

@devos50
Copy link
Contributor

devos50 commented Oct 15, 2021

retest this please

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

5 participants