Migrate to zcash_client_backend and zcash_client_sqlite architecture.

[nejucomo]

Overview

zingolib could migrate to an architecture with zcash_client_backend and zcash_client_sqlite, which are used within other shielded wallets (at least zashi).

This ticket is about considering this migration versus the status quo of zingolib architecture, gathering relevant info, and documenting the rationale for switching or keeping the current architecture (or introducing other alternatives). The actual approach of how to do such a migration is complex enough that it should probably live in future tickets so that this ticket is more focused on judiciously deciding whether or not to attempt such a thing.

In this ticket we refer to SQA to mean the "status quo (current) architecture", and ZCBA to mean "zcash_client_backend (and related crates) architecture".

If a consensus / decision to do this kind of architectural migration comes out of this ticket, a (big) next step would be to begin adding that migration to any existing plans (e.g. roadmaps, backlogs, etc…).

Note: It's very likely Zingo devs have had multiple discussions around this same topic. I didn't see anything obvious when searching the issue tracker for zcash_client_backend. I didn't search any other docs for this topic. If there's already a specific consensus that this idea is tabled indefinitely or on a someday wishlist, etc… let's summarize that here and cross-link.

Trade-offs

Here I try to enumerate the trade-offs I anticipate. Keep in mind these all assume the migration from SQA to ZCBA is complete. If ZCBA seems desirable from this analysis, a future step would be to re-examine these trade-offs for an interim migration period.

Benefits of SQA

• It already exists and works in its current form today.
• The functionality across a greater portion of zingo's behavior is within a single codebase, allowing potentially tighter integration and more control over design, release schedule, etc… for that larger portion.
• There's no need for any migration process and the risks and complexity of it.

Drawbacks of SQA

• It doesn't share as much code with other Zcash projects, which could enable more eyes for debugging or security analysis, more devs per line of code, etc…
• It may differ in observable behavior from zcash_client_backend-style wallets, which may allow privacy adversaries to determine which transactions, lightwalletd connections, or other observable components are used by which wallet vendors. This can reduce the privacy of users of wallets on both sides of a behavioral difference, because an adversary can categorize observable user behavior into smaller groups for both kinds of wallet. (If, by contrast, they behave identical from what is observable, then all users of all related wallets have stronger privacy together.)
• There is likely to be notable tech debt since Zingo inherited zecwallet-lite code, IIRC, so the initial code and architectural decisions are fairly old in the Zcash timeline (e.g. pre-Orchard / UAs / ZIP-317, etc…).
• I have the impression SQA has too much "not-invented-here syndrome", ie: the code reinvents the wheel too often. In particular, I learned that the data persistence has lots of custom / in-house code, which makes me nervous since data persistence can be crucial for ensuring wallets aren't corrupted or leaked.
• There may be more specific details I'm unaware of, since I'm not that familiar with the zingolib code.

Benefits of ZCBA

• Zingo devs can focus on less scope wherever the zcash_client_backend stack properly provides what Zingo needs. A smaller scope of code means: fewer design open questions -> more design time spent on unique/Zingo-specific needs, less complexity of code to directly manage, easier to test, less risk in releases -> faster releases.
• More dev eyes on the code: at least two or maybe more shielded wallet teams would have eyes on this code.
• More user-coverage: users of multiple wallets would rely on this common core, ensuring it gets more real-world usage from a larger variety of people who might exercise or expose different kinds of bugs, or have a broader variety of feature needs.
• Shared behavior across shielded wallets: one important category of privacy weakness is observable differences in shielded wallet behavior (examples may include: which notes are selected to construct shielded transactions, when/where lightwalletd is consulted for updates, particulars of address management such as UA components). Shielded wallets may behave differently because of user behavior differences, code differences, etc… By reusing more core wallet logic, different wallet products behave more similarly for some observable behaviors. This improves user privacy protections against this tricky category of shielded metadata analysis. (wishlist: describe this more precisely / carefully as a goal in a privacy/security model doc; maybe Zingo-specific or a shared Zcash resource.)
• Specific feature/bugfix details: there are probably specific features of zcash_client_backend that are benefits, but I'm not familiar enough with the codebase to list them out.
• Zingo devs might contribute bugfixes or feature improvements upstream to those dependencies, benefiting more Zcash wallets/users and ensuring the upstream dependencies meet Zingo's needs.

Drawbacks of ZCBA

• Safely transitioning Zingo to this new backend may be difficult and somewhat risky.
• zcash_client_backend may not provide all of the features currently provided by zingolib. (IIRC, when Zingo launched Orchard support, it was not present in zcash_client_backend. I'm not sure if that's still the case. Are there other features in this category?
• In this architecture, Zingo devs / design would have less scope, and the code architecture of Zingo would be constrained by any zcash_client_backend impositions.
• More "dependency risk": the more logic is provided dependencies, the more likely external devs might impact the roadmap. For example, compare the current version of zingolib where data persistence is internal to an architecture that relies on zcash_client_sqlite: if there's an important change or bugfix in persistence, in the former case (current zingo) the zingo-devs can potentially choose the release schedule with more freedom, whereas in the latter case (zcash_client_sqlite) a release schedule may require blocking on a new dependency release.
  • an important dependency risk is if the upstream devs do not prioritize implementing a feature or bugfix that's needed for Zingo's roadmap, so Zingo devs may need to fork or patch upstream dependencies.

Next Steps

These are written from my perspective (without a lot of specific context of zingo's current codebase nor the zcash_client_* crates):

#. Pulse check: how important does this seem to Zingo devs? Should we spend more time analyzing this idea, or table it for now?
#. Quick checklist: does zcash_client_backend provide all of the functionality zingolib needs today to support the existing latest release of Zingo wallets? Yes: proceed; No: either table this with "blocked on X, Y, and Z features shipping in crates A, B, and C" or start a more complex analysis of how complex implementing the feature in zingolib-on-zcash_client-crates.
#. Code impact - best-guess answers to questions like:

• what types/functions would be removed by a complete architecture switch?
• what types/functions would need to be modified from their current form after a complete switch to the new architecture?
• what types/functions would remain completely unmodified from such a switch?
  #. … ?
  #. Given the results of all of the previous steps, are any subsets of Zingo devs committed to working on a migration to ZCBA? If yes, proceed, otherwise document that no one sees the trade-offs as worth it, presently.