Tribler project organization: the home of Trustchain currency

[qstokkink]

Recently in IPv8 we have separated the identity derivative of Trustchain from the currency derivative of Trustchain. This enables the discussion of where the code for the Trustchain currency should be located. We have several options (feel free to contribute your own):

• Move the Trustchain currency code to our token storage and trading solution Anydex.
• Move the Trustchain currency code to Tribler.
• Keep the Trustchain currency code in IPv8.
• Remove the Trustchain currency code entirely and switch to the new Noodle protocol, optimized for currency.

Personally, I believe the Trustchain currency code is not a networking primitive nor a meta-community (either of which would qualify it for inclusion in IPv8) and should be moved to the other currency-related code in Anydex.

I'd like to use this issue for all team members to voice their opinion.

[ichorid]

Switch to Noodle New. We harvested enough experience from all the years of using the current currency code (pun intended) to understand that there is no "one size fits all" in ledger engineering.

[devos50]

Remove the Trustchain currency code entirely and switch to the new Noodle protocol, optimized for currency.

I see a full move to Noodle New as the preferred solution, but it is likely not something that will happen soon. It might also require changes to the payouts/market, and possibly break backwards compatibility. Note that Noodle New is not specifically built for currencies but will support many use-cases as far as I'm aware (e.g., channels).

Move the Trustchain currency code to Tribler.

This will break AnyDex since that module builds on top of IPv8 and expects TrustChain to be part of IPv8. If we do this, I most likely need to add Tribler as a dependency in AnyDex. If this is our preferred solution, I will try to find a solution to make sure that the market still works.

Move the Trustchain currency code to our token storage and trading solution Anydex.

I see this as a viable short-term solution. AnyDex is already part of Tribler and the modules that use TrustChain (payouts) can simply use the AnyDex code.

Keep the Trustchain currency code in IPv8.

I agree that TrustChain in IPv8 might feel off, but until Noodle New is ready, integrated and tested, I'm totally fine with keeping the code in IPv8.

[qstokkink]

Thanks for the input guys. From your inputs I gathered:

• We wait for Noodle New.
• The current Trustchain stays where it is.

This leads me to a follow-up question: where should Noodle New then be located?

[devos50]

It is currently located here. I would suggest moving that repository to the Tribler organization somewhere later.

[qstokkink]

Ah, it's in its own repository already. I wasn't quite expecting that one.

[synctext]

This enables the discussion of where the code for the Trustchain currency should be located.

Interesting topic. Where goes our token code? (its not a currency, also reputation is not yet implemented). So there is not a concrete problem to solve, but a cosmetic operation and re-factoring towards a cleaner architecture; right? "Token Management" needs a place in our architecture, just like overlay, market, risk, and reputation functions, with careful long-term implication and thoughts?

This will break AnyDex since that module builds on top of IPv8 and expects TrustChain to be part of IPv8.

Difficult. Guess its also a matter of taste, hence something called "Repo style wars". Our current polyrepo policy is more an architectural problem I believe. Please study this scientific article: Monorepos: A Multivocal Literature Review. Our infrastructure and tooling is quite complex. We have dedicated AnyDEX, Noodle, and IPv8 tooling, that creates inefficiencies, maintenance issues, single-person ownership, and hampers re-use.
Also: Why Google stores billions of lines of code in a single repository (obviously the Google repo is 18 years old and ours only 15 years; plus they have 35 million commits, we're not that big yet 😀 )

[qstokkink]

A quote from "Why Google stores billions of lines of code in a single repository":

The Git community strongly suggests and prefers developers have more and smaller repositories. A Git-clone operation requires copying all content to one’s local machine, a procedure incompatible with a large repository. To move to Git-based source hosting, it would be necessary to split Google’s repository into thousands of separate repositories to achieve reasonable performance. Such reorganization would necessitate cultural and work-flow changes for Google’s developers. As a comparison, Google’s Git-hosted Android codebase is divided into more than 800 separate repositories.

[synctext]

A Git-clone operation requires copying all content to one’s local machine, a procedure incompatible with a large repository.

Yes, seems to be a divisive issue. Fortunately, our laptop can still contain all our Tribler/IPv8/AnyDEX/... code locally. We're not 86TB repo size of Google yet. I'm also in flavor of this from a project maintenance perspective. With a year of tuning by the new scientific developers, we lose the pointer update mess and get more uniformity.

[qstokkink]

Please study this scientific article: Monorepos: A Multivocal Literature Review. Our infrastructure and tooling is quite complex. We have dedicated AnyDEX, Noodle, and IPv8 tooling, that creates inefficiencies, maintenance issues, single-person ownership, and hampers re-use.

The (unpublished arxiv) paper you cites states this about complexity:

• Codebase complexity: According study New version check from Tribler #14, the main challenge of monorepos is to manage all projects in a single repository. Although the understanding organization of the code in monolithic repositories is easy, it is a complex task to determine where new code should be placed. Besides, study Systemtap: create a Python, SQLite and C++ probe (#6) #7 criticizes monorepos since their high codebase complexity does not necessarily increase productivity

This about tooling and testing:

• Tooling investments: A huge repository requires managing tools to scale. Study #1discusses the high cost of running these tools.
• Build, Test Bloat, and Deploy: Due to its size, studies Perfect Darknet Roadmap Ticket #3 and Systemtap: exact cost of a 1h Tribler run, integrated in Jenkins #8 question the cost ofbuilding and testing on monorepos, whereas study Systemtap: create a Python, SQLite and C++ probe (#6) #7 questions the cost of deploy.

Lastly, this about actually moving to a monorepo:

• Migration: Study GUI: random walk with real-time updates #10 introduces “the monorepo problem”. It refers to the fact thatmigrating of many repositories to only one has a high cost, because it is necessary tomodularize all code. This is too critical that study 1 TByte of seeding #21 reports a migration study case.

It just seems like an overal bad fit for us, even according to the literature you posted.

[qstokkink]

See also https://sethrobertson.github.io/GitBestPractices/#divide

[synctext]

OK. "however, important configuration files or files including business critical algorithms can be more tightly controlled", Google also does not use a real monorepo.

[qstokkink]

Let me end my push-back against the monorepo here: what I would advocate for is using a per-component analysis to divide the codebase, to make it both manageable and workable: the problem of microrepositories is indeed a real one and has its issues, but the single repository is not much better. I believe a carefully balanced division of the codebase is the winning strategy.

[ichorid]

I hate submodules too. These are called "sobmodules" 😿 for a reason. However, the cure seems not much better than the ail...

[qstokkink]

Ok, I think we have a clear plan for Trustchain and no more discussion is needed in this particular issue.

Regarding the monorepo discussion, we could make another issue to discuss all of Tribler's components and tooling and discuss where they should be housed. Someone would have to make a comprehensive list of all of these components and tools first though (as we have a substantial amount of those).