[Enhancement]: Improve the workflow for local development of Pretendo

[ashquarky]

Update: Plan of action here #57 (comment)

Checked Existing

• I have checked the repository for duplicate issues.

What enhancement would you like to see?

When developing the Pretendo Network itself, Inkay can serve as a roadblock since it makes assumptions about the production Pretendo Network - domains, certificates, etc.

This is a tracking issue to discuss specifically how Inkay can better serve developers of Pretendo Network. We do not intend to deal with applications outside of local development hosting at this time.

Any other details to share? (OPTIONAL)

Parts of this have been discussed before: #40 (comment) #39

I think it's okay to assume developers will use a proxy, but there are still issues from having to write all the domain rewrite rules in one's preferred proxy suite (Fiddler, Charles and mitmproxy are all in common use) and the proxy's self-signed certificates.

We also need to determine if we should assume developers will use a full self-hosted setup, or a "hybrid" setup (e.g. production Accounts/Friends but development Juxt/NEX). I think there is potential in a hybrid setup combined with e.g. a mode on the Juxt backend that doesn't validate account tokens, or a mode of the NEX servers that read passwords from a nex-viewer-format config file instead of querying Accounts.

@CaramelKat
@TraceEntertains
@jonbarrow (optional)

[jonbarrow]

I agree there is virtue in trying to improve the developer experience. However, I still personally feel like this is out of scope for Inkay, and that it should be kept as simple as possible and within the scope of "production patches" for us. If people need to work locally, the existing options are pretty trivial:

• Build Inkay from source (minor changes, would be better if we had explicit build instructions)
• Use a proxy server (trivial to do and is honestly the fastest way to get this going, it's how I did all of my development prior to Inkay and is how I still do things)

I understand your point on the number of rewrite rules, but really there isn't that many. Fiddler, Charles, and mitmproxy all support wild card style rewrites, so you can just do something akin to https://*.nintendo.net -> http://*.pretendo.cc (or whatever domain is being used) and for 99% of the domains it will Just Work:tm:

As for the certificates, this has really never been an issue since SSL patches have existed for a long time. In fact, we have several of them. Our org has https://github.com/PretendoNetwork/wiiu-nossl which is private for some reason (not sure why it's private? But I do give out builds of this to people at times to help with debugging, so it's a useful tool to have at our disposal), and then there's the other private repo you and Rambo have added stuff to

Opening these SSL patches to the public as (a) dedicated development tool(s) + using a proxy server with some basic rewrite rules would be a better way to go imo rather than trying to mix the scopes of "development tool" and "production user tool" into Inkay

Alternatively, part of the reason why @MatthewL246 was brought on was because of his Docker work with the idea of creating a dedicated development environment. That, also, sounds like a better plan imo than trying to add development stuff to Inkay, and https://github.com/PretendoNetwork/mitmproxy-nintendo already serves the purpose of directing Nintendo requests to a custom server which can easily be modified to use whatever domain (or do what I do and just change your hosts file)

We also need to determine if we should assume developers will use a full self-hosted setup, or a "hybrid" setup (e.g. production Accounts/Friends but development Juxt/NEX)

While there is sometimes virtue in a hybrid setup (and is, to be honest, what I use most of the time, though I'm getting myself out of that habit) I think officially we should encourage full self-hosted setups. A development environment really shouldn't be mixing with production imo, even if it is arguably more convenient. That way we reduce the risk of corrupting live production data, and the development environment is not tied to whatever is in production (as much as there is virtue in using a hybrid setup, there is just as much, if not more, virtue in having everything isolated and fully within the developer's control)

Also, we should consider moving some of these instructions (setting up the local servers, installing the languages/runtimes, getting your patches ready and compiled, etc.) to https://developer.pretendo.network (repo https://github.com/PretendoNetwork/developer-documentation), since this supposed to be the main source of information for developers eventually (a sister issue is probably in order on this repo? Though there are these 3 issues already PretendoNetwork/developer-documentation#5, PretendoNetwork/developer-documentation#6, and PretendoNetwork/developer-documentation#7 which all somewhat serve this goal)

[ashquarky]

Build Inkay from source (minor changes, would be better if we had explicit build instructions)
Use a proxy server (trivial to do and is honestly the fastest way to get this going, it's how I did all of my development prior to Inkay and is how I still do things)

I think these are both reasonable things to request developers do, but I still think streamlining is possible even for these workflows. For example, could the dockerised development environment automatically provide a suitably patched Inkay, or expose a DNS/proxy server with the proper redirections applied?

Build Inkay from source (minor changes, would be better if we had explicit build instructions)

Inkay's URLs are currently spread across several files:

• Accounts, Friends, NEX, SpotPass
• Miiverse applet (trust me, there's a URL in those bytes!)
• Miiverse API
• etc.. (Account Settings, eShop, and NNCS all have their own files with URLs)

I think it would be valuable to factor out the pretendo.cc part into one location in the root of the repository so that it can be replaced with pretendo.localhost or whatever easily or even automatically. This would not affect how production builds of Inkay operate at runtime - this is just a build-time option.

Use a proxy server (trivial to do and is honestly the fastest way to get this going, it's how I did all of my development prior to Inkay and is how I still do things)

Using a proxy means self-signed certificates, and the NoSSL plugins that exist do not reach into web applets (Miiverse, eShop, Account Settings) and thus do not help with development of these services.

Either we could refactor Inkay's handling of the production SSL cert (as has been discussed in the past), or extend one of the NoSSL plugins to work for the applets, now that Aroma has improved enough to make that possible. I'm okay with either approach - assuming you want the latter, I think we should transfer the Meowth repository to the Pretendo org and make it into a proper dev tool.

So, the setup would be something like:

• main Aroma environment
  • Prod Inkay
• test Aroma environment
  • Locally built Inkay with different domains (if needed for the individual devs' proxy setup)
  • NoSSL plugin (updated to cover all development targets)

While there is sometimes virtue in a hybrid setup (and is, to be honest, what I use most of the time, though I'm getting myself out of that habit) I think officially we should encourage full self-hosted setups.

I think we can agree to disagree on this - specifically services like Juxt are practically completely independent of the account server and should be hostable standalone. It also says a lot that as far as I know ~all of the Pretendo team are using hybrid setups on prod Accounts since the Docker stuff (as far as I'm aware) isn't actually usable yet.

Maybe a mock Accounts server that just responds with "yeah sure lol" to every request instead of requiring an actual registration, linking, email, and seperate Wii U profile? Not 100% sure, maybe we can discuss this further in another issue.

[ashquarky]

Oh and I had honestly forgotten about the developer wiki, will mull on it and read up!

[jonbarrow]

I think these are both reasonable things to request developers do, but I still think streamlining is possible even for these workflows. For example, could the dockerised development environment automatically provide a suitably patched Inkay, or expose a DNS/proxy server with the proper redirections applied?

It could, and this sounds reasonable to me. Though I still disagree with providing a custom build of Inkay in this case since it feels like it still makes it "officially" a development tool on our part, whereas I believe the scope should stay limited to just production patches (I have more on this further down)

Inkay's URLs are currently spread across several files
...
I think it would be valuable to factor out the pretendo.cc part into one location in the root of the repository so that it can be replaced with pretendo.localhost or whatever easily or even automatically. This would not affect how production builds of Inkay operate at runtime - this is just a build-time option.

Oof yeah I didn't realize just how spread it was (and I forgot about the bytes one). Valid! I agree this should be consolidated, that's a good area for this to be improved on

Using a proxy means self-signed certificates, and the NoSSL plugins that exist do not reach into web applets (Miiverse, eShop, Account Settings) and thus do not help with development of these services

For this, see my comment here PretendoNetwork/Martini#13. These applets are difficult to work with in general, even if you get the certificates part figured out. Using the Wii U as the development platform is not great and is, honestly, probably the worst way to go about developing for the applets ironically. They're slow, annoying to do basic things like refreshing, etc.

I think a better solution to this is to move that task to Cemu. Whether that be through us providing a way to load these applets in Cemu (like with Martini-style patched RPXes/RPLs) or by maintaining a dedicated fork of Cemu specifically for working in the applets (which would be more difficult on our part, but would give much more control over the workflow) and for anything not using an applet then the expectation is to either build Inkay from source or use a proxy server + an SSL patch

I also joked with @CaramelKat a few years ago about building a custom browser which uses a super old version of WebKit to try and emulate the applet environment, which would honestly be the best solution (maximum control, no reliance on console emulation, etc.), but is also very much the most unrealistic (build a dedicated browser? Even if we piggy-backed off existing tech like WebKit that's still a ton of work)

I think we should transfer the Meowth repository to the Pretendo org and make it into a proper dev tool

That sounds great to me 👍

I think we can agree to disagree on this

I think I may have misunderstood you originally. When you said "hybrid" and talked about only hosting parts of the network, I thought you were implying that there should be a mix of "this uses a local server, but this uses a production server". Your follow up now reads more like "Juxtaposition doesn't rely on NEX, and NEX doesn't rely on Juxtaposition, so those things don't necessarily need to be hosted at the same time". The latter I am in agreement with, things which don't necessarily rely on each other don't necessarily need to be hosted together either. I just figure it would be easier to spin everything up at once for the sake of simplicity. If you did mean "use local data in some places and production data in others", then I do still disagree with that and there's a number of times where that breaks things. For example, SMM stores a list of your uploaded courses in your account save file. If you try to use a production account on a local server, then the course data won't match and SMM will think your courses have been deleted on the server and mark them as such in your save file. You would need to make a fresh account to avoid this issue, and at that point just run everything locally and don't take up an account on production for that. There's also plenty of times where you would explicitly want everything locally, for example if the account server was changing the encrypted token format, you'd want to host the account server locally with those changes (which may not be in prod yet) and update another server alongside them. This introduces the need to sometimes use production data, sometimes use local data, which overall just complicates the development setup imo. I still also very much have concerns about a developer working locally accidentally fucking prod (the last thing we need is a "oops I dropped the prod database" issue at 3am, looking at you @mrjvs @binaryoverload). That just sounds like a recipe for disaster, and I think it's well worth the minor annoyance of running an extra server or two to avoid that

[jonbarrow]

Oh and I had honestly forgotten about the developer wiki, will mull on it and read up!

It's super incomplete. It's mostly on hiatus (at least the NEX parts) while we finalize our library designs. Ofc other non-NEX lib additions can still be made in the meantime

[CaramelKat]

Forgive me if this is a little rough, this is currently getting typed out on my phone at 9% lol

Regarding the reliance on other servers, Juxt is unfortunately no where near as independent as it once was. Once upon a time all of the account data was mirrored. This offered a lot of flexibility in terms of speed and how we processed the data, but had risk for getting out of sync. For the last few years, this data has been pulled directly from the account server (previously through a db connection, now via grpc and cached locally for a given session for read only operations since it's slow to get the data).

This does mean a self hosted account server is almost certainly necessary. I've avoided this for years using a prod environment, but had to cave a few months back when I tried to get back into development (and promptly gave up because I couldn't get it or the friends sever running after nearly a week of effort, even with the newer docker infrastructure). Assuming the docker setup actually works now (again, haven't tried in months) having it easier to build with local certs and domains would be a huge boost to development. I've learned the hard way that there isn't a better way to test changes in Juxt than on console, especially on the 3DS.

Cemu would also be a good solution, and I'd love to explore that as a development avenue as well

[jonbarrow]

The 3DS is good to mention, we had been ignoring that in this conversation. I still think local applet development would be best done in emulators, seeing as you get all the benefits of not running it on native hardware (faster to load, faster to iterate on, potential for making a dedicated environment through forking/stripping, etc.)

Maybe Azahar would be a good candidate for this? Or we can assume that anything the Wii U applet can run, the 3DS applet can as well (as in, functionally speaking. Ofc the 3DS has more limited resources and so things like asset sizes and all that will be different, I mean in terms of like HTML, CSS and JS support) and just use Cemu to work on both platforms?

[ashquarky]

Cemu isn't a low-level emulator and thus won't be perfectly accurate, so I think we should always have some ability to test on hardware too, even if it's just as a pre-commit smoketest.

I think just supporting everything is reasonable here and developers can pick whichever workflow works best for them.

[ashquarky]

This does mean a self hosted account server is almost certainly necessary.

I'll set aside the hybrid idea for now then and focus on full selfhosting.

[jonbarrow]

Cemu isn't a low-level emulator and thus won't be perfectly accurate, so I think we should always have some ability to test on hardware too, even if it's just as a pre-commit smoketest.

I think just supporting everything is reasonable here and developers can pick whichever workflow works best for them.

I'm not saying we should strictly disallow hardware testing, and from my tests Cemu has been perfectly fine when it comes to applets (the issue with some graphics being wonky is a little unfair on my part since the system I tested this on was barely able to run Cemu at all, it had a myriad of other issues with normal titles as well. Anyone using anything even remotely modern should have a much better experience than I), but I do think it should be our recommended "go-to" method given the benefits it brings and the way it simplifies the developer environment, and we have yet to see any real negative results from using Cemu for this outside of it having some issues on potato machines like mine (which we imo should be considered an outlier, my machine really doesn't have to be considered typical imo). Anything else (building Inkay, running on real hardware, etc.) imo should be considered more of a "you can, but you'd likely have a worse time" sort of deal

Though you do make a good point about doing pre-commit smoketests. Checking on real hardware before committing should probably be mandatory (I've only been talking about how things would be setup for active development)

The more we talk about this the more I want to just build our own dedicated applet browser emulator (/s)

The applets suck

[jonbarrow]

@ashquarky On a more serious note, you mentioned Cemu not being low-level. What about Decaf? I haven't checked up on that in a few years, but surely it's gotten better no? And iirc it was low-level?

There's also https://github.com/kinnay/Wii-U-Firmware-Emulator which specifically says:

This emulator emulates the Wii U processors and hardware at the lowest level. It's currently able to emulate all the way through boot1, IOSU and Cafe OS up to the Wii U menu

though it's nowhere near as complete as something like Cemu. But maybe it would be a better candidate for a dedicated applet tester outside of real hardware?

Just spit-balling ideas here

[ashquarky]

I think Cemu is generally better than Decaf these days for accuracy and compatibility, and while Decaf does go a bit harder on things like IOS HLE it's still ultimately a still HLE emulator and those inherently have some amount of inaccuracy or bugs.

WUFE is indeed a perfect low-level emulator, but it also doesn't have a graphics output, so stones glass houses etc.

To be clear I think Cemu will be an excellent DX speedup for the vast majority of changes, I just also think that since our target is hardware we should test on hardware too. Plus hardware is arguably easier for new contributors, since they already have the application and account files installed and ready to go, so it's a bit less setup.

[jonbarrow]

I think Cemu is generally better than Decaf these days for accuracy and compatibility, and while Decaf does go a bit harder on things like IOS HLE it's still ultimately a still HLE emulator and those inherently have some amount of inaccuracy or bugs

Damn, I was mistaken then. I thought Decaf was LLE

WUFE is indeed a perfect low-level emulator, but it also doesn't have a graphics output, so stones glass houses etc.

Damn again, forgot there was no graphics output. I mean. I guess that CAN be added, but that does place a lot more work on us (maybe this should still be added to the tasklist however? As a "in case anyone wants to go for it" thing?)

To be clear I think Cemu will be an excellent DX speedup for the vast majority of changes, I just also think that since our target is hardware we should test on hardware too

Yeah this is fair. I think we agree there. Imo the ideal workflow would be:

• Develop against Cemu, as a way to speed things up since it's much faster to iterate on
• Test changes on real hardware before committing, to confirm changes work as expected

Plus hardware is arguably easier for new contributors, since they already have the application and account files installed and ready to go, so it's a bit less setup

I think we should have some reasonable expectations of people who want to contribute in ways that require a development setup, personally. For things like minor website changes, really anyone can do that. But if you want to contribute to the core parts of the network, I think it's fine if we have some higher expectations and ask them to do things like "use Cemu", which really isn't that tall of an order to ask tbf. If you're wanting to contribute like that, you're already setting up several things on your PC. Expecting them to also setup one extra program is more than reasonable imo. I don't think we have to baby (for lack of a better word) new contributors if they're trying to contribute to stuff like this

[ashquarky]

Okay, so I think we have a provisional plan here:

• Centralise the URLs in Inkay to allow changing the base domain (pretendo.cc) easily - 9e376f7
• Investigate using Cemu for Juxtaposition development ( [Feature]: Port to PC and/or Cemu graphics pack Martini#13 )
• Clean up Meowth and add support for SSL patching the applets - https://github.com/PretendoNetwork/Meowth/commit/11cde803ff9bbde972cb9b65ec0a748cde996e52
• Document an ideal workflow on the developer wiki using the above points
• Other workflow / automation stuff where we can (e.g. pretendo-docker automatically hosting mitmproxy for you)

[jonbarrow]

That sounds good to me. I think long-term maybe we should look into having a monorepo people can pull down that has things like:

• mitmproxy-nintendo (Charles my beloved, but it is paid software)
• A custom Inkay builder (maybe with a CLI that just takes in some parameters to build Inkay with? Like the domain and such)
• The server repos themselves (git submodules?) to be worked on
• Some way to spin up the network easily from the local servers (can Docker do this cleanly?)

I've never designed anything like this before so I'm a tad out of my element here, but roughly something like:

developer-environment/
├── patches/
│   ├── wiiu/
│   │   ├── builder/
│   │   └── inkay/
│   └── 3ds/
│       ├── builder/
│       └── nimbus/
├── proxy/
│   └── mitmproxy-nintendo files here
├── repositories/
│   ├── account/
│   ├── nex-go/
│   ├── miiverse-api/
│   └── etc
└── orchestration/
    └── server runner idk

[ashquarky]

This sounds good longer-term. I might suggest that running the server binary itself on the host should be a supported case in such a setup (rather than doing a docker build cycle) since it allows for improved IDE integrations and using a debugger and this type of thing, which would be harder if the server under development is inside Docker. We're creeping pretty far out of scope for an Inkay issue though.

(That won't be a problem fwiw, it just needs to spit out a .env file with the right keys and docker IP addresses)

[jonbarrow]

I assumed we were already pretty out of scope for this issue already to be fair, and had just started using it as a place to discuss the dev env more broadly/publicly

Also, it should be noted that "server binary" isn't necessarily a thing in all places. Anything using TypeScript/JavaScript won't have a binary to run. I don't disagree with the premise this is just something to keep in mind

Will and jvs may have more opinions on this sort of long-term setup though, this sounds like something more their speed

[DaniElectra]

Speaking for the 3DS side of things: It is important to consider that we don't have native account switching there, so reusing the account from production on local servers is essentially a necessity if you don't want to go through the nightmare of handling different accounts.

As for URL patches, these are more-or-less centralized within the HTTP patch which replaces every URL with nintendo(wifi).net
to pretendo.cc. These are also patched within ACT and Friends

For Miiverse specifically, while reversing the posting applet I also found out that you can give the api_host and n3ds_host through "launch parameters" to simplify, so it should be relatively straightforward to make a "launcher" for Miiverse with a custom domain.

I also believe the "server binary" should be kept as an option, mainly for integration and debugging as Ash said. Yesterday I was trying to debug with her using GDB a local instance of the test server from nex-go with this branch: https://github.com/PretendoNetwork/nex-go/tree/work/pcap-replay using a packet dump from the prod friends server to try and figure out the stalled connections issue. The research was limited (we could only test with auth since we would need the PIDs and NEX passwords from users to investigate on the secure server), but we did confirm stalls on clients that didn't finish the connection to the server (e.g. they didn't send a CONNECT packet or it wasn't received on the server).

[jonbarrow]

Speaking for the 3DS side of things: It is important to consider that we don't have native account switching there, so reusing the account from production on local servers is essentially a necessity if you don't want to go through the nightmare of handling different accounts

It's not a necessity, it just feels like that because Nimbus only swaps between 2 accounts because it's our production patches and that's all it needs to do. This would fall under the same "we shouldn't use production tools as development tools" opinion I have for Inkay

Providing a dedicated developer tool which is essentially just an expanded version of https://github.com/zaksabeast/3ds-Friend-Account-Manager which allows for the creation of local accounts using the full range of properties would get around this issue

In September 2023 you documented all the server types which the 3DS allows https://www.3dbrew.org/wiki/Friend_Services#Server_Types. While the official servers only supported 8 letters, assuming the console itself can handle all 21 then we can have many many more local account types for the purpose of development

In the docs about server types is the line:

the production server type is "L1", so the first value is 0 and the second value is 1

and https://www.3dbrew.org/wiki/FRDA:CreateLocalAccount has parameters for a number and a letter value. I assume these parameters correlate to the line about the server type mentioned above, which means we can get many combinations of account types

Additionally, Nimbus only makes use of the test environment for our accounts, which gives us the test URLs. But the 3DS has a 3rd dev environment as well which would fit this purpose perfectly (both thematically, and because we would yet again get new URLs)

So for the 3DS this would look like:

• Expanded version of https://github.com/zaksabeast/3ds-Friend-Account-Manager which lets you create more than just 2 accounts (or even just a dedicated "create/delete/whatever your dev account" tool which explicitly says it's not meant for use on live servers)
• All accounts under this developer tool would be under the dev environment to get URLs separate from Nintendo's and ours
• The same suggestion regarding using a proxy server would apply here to direct the dev URLs to a local server

This prevents the need to reuse a production account while working locally

For Miiverse specifically, while reversing the posting applet I also found out that you can give the api_host and n3ds_host through "launch parameters" to simplify, so it should be relatively straightforward to make a "launcher" for Miiverse with a custom domain

Using the above method of getting a local dev account, this also shouldn't be necessary as you can just redirect requests to the discovery URL like normal using the proxy server. Though it's still super interesting tbh, maybe that would still be useful for something?

I also believe the "server binary" should be kept as an option, mainly for integration and debugging as Ash said

We're all in agreement there. To be clear, like I said I was not against this idea. My comment about this earlier was to temper expectations since not every server would have a "server binary" to do this with (like you can't run the accounts "server binary" since there isn't one)

[DaniElectra]

So for the 3DS this would look like:

• Expanded version of zaksabeast/3ds-Friend-Account-Manager which lets you create more than just 2 accounts (or even just a dedicated "create/delete/whatever your dev account" tool which explicitly says it's not meant for use on live servers)
• All accounts under this developer tool would be under the dev environment to get URLs separate from Nintendo's and ours
• The same suggestion regarding using a proxy server would apply here to direct the dev URLs to a local server

This prevents the need to reuse a production account while working locally

That is fair, though if we go for a dedicated tool this would mean we would stop dog-fooding Nimbus, but that's not a big deal.

I've also heard that there were some limitations when using the dev environment, but I don't have proof of this and I don't remember where this came from, so it may be completely wrong and we can probably use it just fine

Using the above method of getting a local dev account, this also shouldn't be necessary as you can just redirect requests to the discovery URL like normal using the proxy server. Though it's still super interesting tbh, maybe that would still be useful for something?

Yeah this isn't useful if we have the dev accounts, just wanted to mention it as part of my research

[jonbarrow]

I've also heard that there were some limitations when using the dev environment, but I don't have proof of this and I don't remember where this came from, so it may be completely wrong and we can probably use it just fine

If this is the case then we can always just make these accounts also use test. It wouldn't be as nice since we wouldn't get dedicated URLs, but proxy server redirects still fix this in the end

[jonbarrow]

That is fair, though if we go for a dedicated tool this would mean we would stop dog-fooding Nimbus, but that's not a big deal

I'm not super sure what this means? Are you suggesting we deprecate Nimbus? Because that's not what I was trying to suggest. My suggestion was to have this split:

• Nimbus. Keep as our production patches for standard end users. Only swap between NN and our servers (internally this swaps accounts, but to the end user this just looks like a server switch). Doesn't give advanced usage like manually creating accounts (done for you already), deleting accounts, using anything besides test and the server type Nimbus uses, etc. Basically no changes here, keep it as it is
• Dedicated developer tool. Essentially a more powerful Nimbus/3ds-Friend-Account-Manager. Not only does it let you swap between more than 2 accounts (and tells you that they are different accounts, not just servers), but also lets you create more than the standard 2 Nimbus has and lets you specify the server type and such. Uses dev rather than test unless dev has issues

[jonbarrow]

Added this to the task tracker. I think difficulty I gave it is accurate, but feel free to question that. I may be under/over estimating here

[DaniElectra]

I'm not super sure what this means? Are you suggesting we deprecate Nimbus? Because that's not what I was trying to suggest.

No, there is a misunderstanding here.

I mean "dog-fooding" as in actively using it as developers, and we may not detect bugs in features we implement since we aren't actively using it. For example this could be as subtle as some form of leak, causing a long shutdown time

[jonbarrow]

Ah I understand, my apologies

[jonbarrow]

Also I've went ahead and added the "browser app emulator" idea to the task tracker as well, with details about what our options are

I forgot that Nintendo based these browsers off open-source work and that they publish those changes as per the license requirements. So it may actually be very viable for us to snipe those changes Nintendo made and actually build a dedicated desktop browser that acts exactly the same as the browsers in apps like Miiverse

See the details here https://github.com/orgs/PretendoNetwork/projects/34?pane=issue&itemId=91100846

[mrjvs]

I was pinged for opinions, so here are my opinions:

1. Mixing production and development (named hybrid in this conversation) is not something we should be striving for. I would rather our effort is invested somewhere else. Not to mention that this makes it very hard for someone to contribute if they are not part of the team. (This was already agreed opun I think, just voicing my agreement anyway)
2. Exploring emulators sounds like a good plan for juxt, Building a browser is a lot of effort comparatively. (and something to maintain)
3. I'm not sold on a monorepo to be a home for all development. That's another something that needs active maintenance.
4. Do not use sub modules. Sub modules point to a specific commit hash and need a new git push to move that pointer to the newest commit of the branch. Realistically, nobody is gonna actually be updating the git submodule commit hash and it'll just grow stale.
5. I think focusing on running the services locally on the host (not in a docker container) is a good thing to strive for. Working in dev containers can work, but I don't think it's the right thing for pretendo due the unique requirements of the services we make.

[MatthewL246]

I'll add my experiences and main pain points with Pretendo Docker.

• On Wii U, replacing the certificate requires building a custom copy of Inkay. I don't need to go into too much detail about this because [Feature]: Ability to load certificate from a file  #40 is already tracking this issue. Loading the certificate from a file as suggested there would effectively eliminate this pain point and no longer require building a custom copy of Inkay to use Pretendo Docker.
• On 3DS, adding an additional account is a huge pain. I currently use a combination of a Friends account switcher GM9 script that swaps between different Friends and account sysmodule saves and a homebrew program that resets the "test" account that Nimbus uses. This is risky and not nearly as user-friendly as I would like it to be. You already discussed potential solutions for that, which I'll respond to below.

Opening these SSL patches to the public as (a) dedicated development tool(s) + using a proxy server with some basic rewrite rules would be a better way to go imo rather than trying to mix the scopes of "development tool" and "production user tool" into Inkay

This sounds fine to me. Either loading the certificate from a file or using Meowth to patch out verification entirely would both work equally well from my perspective, with Meowth likely being a little more convenient.

I think officially we should encourage full self-hosted setups. A development environment really shouldn't be mixing with production imo, even if it is arguably more convenient.

I completely agree. Developing this way also has the benefit of ensuring that all server features work locally, without required cloud services or other parts of our own infrastructure, by essentially dogfooding self-hosting. I think this has clear benefits for external contributors, self-hosters, and user freedom in general.

For example, could the dockerised development environment automatically provide a suitably patched Inkay, or expose a DNS/proxy server with the proper redirections applied?

Pretendo Docker already does this ;). It has a script that compiles Inkay with a custom certificate, and it also runs mitmproxy. It even has the option of running an SSSL server with a little additional setup.

I think it would be valuable to factor out the pretendo.cc part into one location in the root of the repository so that it can be replaced with pretendo.localhost or whatever easily or even automatically. This would not affect how production builds of Inkay operate at runtime - this is just a build-time option.

The whole domains thing hasn't been a problem for Pretendo Docker because everything is redirected to a local server by mitmproxy. I don't see the point in changing pretendo.cc to pretendo.local or similar when you're going to have to be connected to a proxy anyway to forward the traffic to your local server and make the subdomain routing work. I would also strongly recommend that any developer use an MITM-ing proxy due to the benefits it provides (as you already discussed).

With that being said, I am certainly not against that change because I don't see any downsides. I see it as being more useful for general self-hosters than developers, though.

The latter I am in agreement with, things which don't necessarily rely on each other don't necessarily need to be hosted together either.

One of my long-term plans for Pretendo Docker is to split the different services into different compose.ymls and make a CLI to configure which ones to run, including their dependencies. I see this as primarily a way to save system resources by not running unnecessary services.

Yeah this is fair. I think we agree there. Imo the ideal workflow would be:

• Develop against Cemu, as a way to speed things up since it's much faster to iterate on
• Test changes on real hardware before committing, to confirm changes work as expected

This sounds good to me. I would, of course, want to support both emulators and real consoles in Pretendo Docker, since both will have their own place in the development cycle.

That sounds good to me. I think long-term maybe we should look into having a monorepo people can pull down that has things like:
...
I've never designed anything like this before so I'm a tad out of my element here, but roughly something like:

I don't think this was intentional (and that makes it even funnier), but you basically just described the structure of Pretendo Docker lol. It includes a mitmproxy server, a script that compiles Inkay (I linked it above), the server repos as submodules, patches for the server code, and a setup script + Docker Compose config for spinning up the servers.

I might suggest that running the server binary itself on the host should be a supported case in such a setup (rather than doing a docker build cycle) since it allows for improved IDE integrations and using a debugger and this type of thing, which would be harder if the server under development is inside Docker.

I remember previously discussing this in PretendoNetwork/juxtaposition-ui#65 (comment), where I didn't quite understand this setup at the time (but I do now).

Personally, I'm happy with using a devcontainer setup. VSCode (my main editor for Pretendo work) has great support for it with features like remote debugging, and there are some easy speed-ups you can do like mounting your working directory inside the container and restarting the container instead of doing a full rebuild. However, I respect that others will have different preferences for IDEs and development processes, so I agree that running the server outside the containers and exposing the gRPC ports (for example) should be a supported option.

Expanded version of zaksabeast/3ds-Friend-Account-Manager which lets you create more than just 2 accounts (or even just a dedicated "create/delete/whatever your dev account" tool which explicitly says it's not meant for use on live servers)

Now we're getting back to the main 3DS pain point I mentioned in my second bullet. I just want to say that having this would be awesome to have for Pretendo Docker.

I never shared it with anyone, but when I learned that it would be possible to create more than 2 accounts, I envisioned a "Nimbus Advanced" that would allow you to create, modify the server environment, and delete arbitrary Friends accounts without requiring any hacky GM9 script stuff. I originally thought of a hidden "advanced mode" in Nimbus that you could access by pressing a secret button combination or something, but using a separate application is probably more practical.

I forgot that Nintendo based these browsers off open-source work and that they publish those changes as per the license requirements.

Very interesting! For future reference, the source code is here: https://support.nintendo.com/jp/oss/index.html. I checked and can confirm it includes WebKit, although I didn't do anything else with it.

3. I'm not sold on a monorepo to be a home for all development. That's another something that needs active maintenance.
4. Do not use sub modules. Sub modules point to a specific commit hash and need a new git push to move that pointer to the newest commit of the branch. Realistically, nobody is gonna actually be updating the git submodule commit hash and it'll just grow stale.

In my experience maintaining Pretendo Docker, it hasn't been that bad. I have Dependabot set up to automatically update the submodules daily and create an update PR. Then, when I get an email, I make sure that CI passes and quickly check the diff to make sure it doesn't change anything in places like configuration environment variables. If everything's fine, I just squash and merge. All you need is a developer or 2 who has a feeling of personal responsibility over the repo and email notifications for PRs in their inbox. (The CI currently only checks whether the setup script runs, the servers build successfully, and the servers start without crashing within 60 seconds. I think this is a pretty decent smoke test for a development environment without having actual tests in most repos.)

The main issue is when something does change in a server that would require updates to Pretendo Docker itself. To call myself out a little here, I have a submodule update PR that has been open for a month because CI is failing and I haven't had the time to investigate it and update the necessary configuration. This is clearly one of the problems with the setup you're talking about here.

However, I'm not certain that there is a better option. Ignoring the specifics like whether you use a monorepo and submodules or not: when you boil it down, you have 2 options: is your development environment pinned to specific server repo versions, or does it always use the latest version? Either way, sometimes breaking changes are going to happen to the servers. In the first way, you'll have an outdated repo until it's fixed, and in the second way, you'll have a broken environment. IMO, having an outdated server in your development environment is preferable to a broken development environment.

[ashquarky]

Pretendo Docker already does this ;). It has a script that compiles Inkay with a custom certificate, and it also runs mitmproxy. It even has the option of running an SSSL server with a little additional setup.

Dang, you're way ahead of me! That's excellent. I think improving Meowth should make it not needed though (outside of specific setups) which will be even better.

I'm not entirely clear on why, but I know @CaramelKat uses a custom miiverse.cc domain for development alongside a MITM proxy. I would guess that it makes life easier to switch between the selfhost and "prod" Pretendo?

We will also have to see how a MITM'ing proxy interacts with Cemu, if we're going to integrate that more deeply.

[MatthewL246]

Cemu works fine with mitmproxy by setting <proxy_server> in settings.xml. I don't think it actually verifies any certificates.

[ashquarky]

Some progress has been made on this:
In 9e376f7, Inkay had all the domains moved to revolve around a single #define, namely NETWORK_BASEURL. Build tooling can either create an inkay_config.local.h to define this, or via make CFLAGS. This needs to be written up and documented.
Meowth has been moved over to the Pretendo org and in https://github.com/PretendoNetwork/Meowth/commit/11cde803ff9bbde972cb9b65ec0a748cde996e52 I updated all the toolchain stuff, as well as adding SSL patches for Account Settings, Miiverse, TVii and eShop. I think those are all the relevant WebKit applets, let me know if I missed any?
That repo is probably good to make public soon, or at leat opened to the rest of the Pretendo team.

[ashquarky]

With the new Meowth setup, one can spin up a (stock) mitmproxy, enable TLSv1.1 and disable upstream cert verification, and start capturing traffic on live Pretendo. Obviously a real use case would involve hosts and redirecting away from live Pretendo, but it's a damn sight easier than compiling an Inkay.