[Question]: is it possible to build this project as c-archive?

[TheAwesome98-Real]

Checked Existing

• I have checked the repository for duplicate issues.

What would you like to know?

i'm writing a game server, and my favourite programming language is rust, so i'd like to write it in that. this is the most comprehensive nex implementation i've seen, and it would be extremely difficult for me, who is (technically) one person, to reimplement this in rust to be equal to this. so, i decided, it would be significantly easier to write rust bindings to this go package. i'd simply have to remember to free all pointers, and it shouldn't be too difficult to do. the problem arises when i actually try to build the package for rust. you see, i've only ever tried to write bindings for other c code, in which you simply call cc and tell it to build a .a file, then you tell rustc to use this .a file when linking. when looking around, i found you could tell the go compiler to output a .a file by using the command-line argument -buildmode=c-archive. the problem is, when i do this, go tells me -buildmode=c-archive requires exactly one main package. i don't really understand go, but from my experience with other languages, i would assume there is no main package, since this is a library. why does it require this? surely the option is there to build libraries? how do i fix it?

[binaryoverload]

Trying looking at this article: https://darkcoding.net/software/building-shared-libraries-in-go-part-1/

It seems to be that build mode needs to be c-shared which builds a standard library that can be linked to

I found this from https://users.rust-lang.org/t/can-rust-interface-with-go-libraries/4292/2

[jonbarrow]

What @binaryoverload said is half-correct, you have to use c-shared. Though this still requires several changes

No matter the build mode the Go module is expected to have at least a single main package. In Go libraries designed to have C bindings, but no real main package, this usually takes the form of a small "glue" package using main that manages everything exposed through the C bindings

However we are not set up for this, and there are a great number of issues with trying to migrate to this type of model:

1. Every single source file which has anything that is designed to be exposed in C bindings would need to be updated in order to import the C package and //export directives would need to be added for every single item that should be exposed. That's quite a bit to do and I'm not sure how well the export directive will play with Godoc comments
2. The entire library would need to be restructured. Go cannot have two different package names in the same directory, and right now the package at the root of the library is nex. In order to add a main package to even get C bindings to build the whole library would need to be refactored to not have these package name conflicts
3. C bindings for Go programs are imperfect and still rely on the Go runtime, which is fine in a lot of cases but has some caveats when working with code like ours. We rely extensively on goroutines for multithreading, which is a concept unique to Go. These goroutines are managed by the Go runtime, and you can shoot yourself in the foot when using bindings like these if the Go runtime is interrupted by the host language (C, Rust, whatever) which can cause issues that are difficult to debug
4. Debugging in general for bindings like these would be a huge pain in the ass for you, as would making any sort of patches to our library. While we are fairly accurate and extensive in our emulation, our implementation is imperfect and still has some issues. For instance we're pretty sure we have a memory leak here somewhere related to stale references to closed socket connections. Running into that issue in your Rust code would likely be a nightmare to debug
5. Some types would likely need to be updated and refactored (such as int being changed to C.int) which might conflict with our implementation since we rely heavily on custom types of even basic primitives like strings and numbers (see Refactor types #56)

Overall I do not think creating bindings for this library and trying to use it in other languages is a good idea and given the issues associated with it it's not a huge priority in my opinion. At least not right now. It's a bit impractical and would almost certainly have many footguns. And any issues/patches you'd have to make in the library itself to fix these bindings would require you to work in Go anyway, so you might as well just cut out that middleman tbh. I know you said that Rust is your favorite language and I agree that trying to rewrite all of this in Rust would be a large undertaking and not really worth it, but this is a Go library designed to be used by Go programs, so you would likely be better off just using it as it was intended. You would run into far less issues, and (this might sound selfish but ehh) possibly even kick back any patches to us if you find any issues (spirit of open source and all)

[TheAwesome98-Real]

fair enough. i'll probably just write my own version in rust, since having multiple implementations of standards is healthy

[jonbarrow]

I disagree that that is healthy, to be honest, but if you want to spend the time to rewrite everything then I wish you luck. I personally think the healthier thing to do would be to contribute back to a single implementation in a unified effort to create a solid implementation, rather than having many fragmented implementations of varying degrees of stability/features/API (at least until that implementation is considered truly stable, at which point ports would be much easier to make)

People have tried to make ports of our libs before, in multiple languages, and they tend to end up being half-implemented and not really going anywhere, since it is a decently large undertaking. Do keep in mind that this is not the only library needed, this only implements the lower level PRUDP connections. You also need a way to handle the protocols and the implementations of those protocols

I tend to lean towards the opinion that fragmentation like that for no real reason outside of language preference only really ends up serving to divide rather than create a better ecosystem. Having many ports in many languages, which all have varying/different levels of support/features/stability, sounds much worse imo than just using and contributing towards one implementation with the goal of making it as stable as possible through unified effort. Working together to better a single implementation has worked out for the better in the past, this implementation would not be where it is today without the efforts of others who would have liked to use different languages (including Rust). And the efforts of people like GoCentral help pushed forward things like Rendez-Vous support for non-NEX clients

We're no strangers to language preferences for sure (99% of our stack is written in 2 languages, which we do mostly for consistency reasons, which is a real issue being solved by that decision), but we also recognize that trying to reinvent the wheel in a different language tends to be not great unless it's fixing an actual problem we have. For example Pictobox is a TypeScript implementation of many image formats, but this was made to fix the issue of existing JavaScript libraries not having APIs that tie in together very well (and the stack we needed to use these in was already TS/JS based), and some images had no libraries at all. So we implemented them ourselves to fill that need. On the other hand, PKHaX (our Pokemon legality check server) is written in C#, and is the only server we have which is. This is because we rely on PKHeX internally to do the legality checks, which is written in C#. We could have rewritten PKHeX into TS/JS in order to line up with the rest of our stack, but it wouldn't really solve an objective issue outside of language preference. Doing so would have just led to further tech debt on our end, now needing to maintain our own implementation which would almost certainly become out of date (which can and does happen, PKHeX constantly updates its rules for determining the legality of Pokemon like seen here), requiring much more maintenance for basically no gain or else our implementation would become stale and further divided from the upstream version, while also contributing nothing back to the original authors

This is not meant to discourage you, to be clear (nor is it really meant to lecture, I just tend to write very long messages. It's a me thing), just giving my opinion/view on the matter as someone who has been working professionally in open source for many years now (I've been working on Pretendo since 2017, and have been an open source and freelance developer for several years prior to that) and seeing how often work ends up getting divided into 100 half-working implementations rather than people coming together to make one as good as it can be. I very often see repos with descriptions such as "project but in Rust!" (Rust ports specifically seem to be a trend as of late...?) or "library but in Python!" made by people who just like those languages, made without solving a problem outside of preference, and seeing they got some work done 5 years ago and then were never updated again past the basics

[ashquarky]

if you do actually do this (especially if it’s no_std) let me know because it would be portable to the consoles in a way that Go isn’t (for the reasons Jon described) and would be useful for homebrew I understand Jon’s point about fragmentation but there might be actual demand here which would keep the project alive

[jonbarrow]

if you do actually do this (especially if it’s no_std) let me know because it would be portable to the consoles in a way that Go isn’t (for the reasons Jon described) and would be useful for homebrewI understand Jon’s point about fragmentation but there might be actual demand here which would keep the project aliveOn 27 Nov 2024, at 06:03, lily celeste newton @.***> wrote:

fair enough. i'll probably just write my own version in rust, since having multiple implementations of standards is healthy

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you are subscribed to this thread.Message ID: @.***>

It would only be useful in the way you described if it had client support, which would need to be done from scratch seeing as this library can't act as a client right now. And if that's what you want, Cemu already has a PRUDPv0 implementation in C++ which is much more simplified than what we have here and Dani has already expressed interest in making a more robust C++ client in our private dev channels on Discord. A Rust port of this library is largely unnecessary even for what you've described

There is merit in making a C/C++/Rust/whatever client, but that's a totally different issue than the one being described here and doesn't really involve porting this library. In that case it would be more efficient to just write a dedicated client, and not worry about server support at all (like what we've done here, writing a server without worrying about client support). But again, that has already been done to some capacity by others so I still feel like starting from scratch in Rust just for the sake of using Rust (which is a trend I've noticed online as of late...?) is kinda pointless

[TheAwesome98-Real]

the problem is i'm trying to implement this into a project that i've already written that uses rust crates and c libraries. that's why i don't think it makes sense to say i should contribute back to this project, even if i want to, because there's no reasonable way i can use it in my project. but anyway, i would only need to implement the minimum set of nex stuff to support that specific game server anyway, and if i open-source the crate it might possibly benefit someone else. that's why i don't think it's useful to say things like this:

starting from scratch in Rust just for the sake of using Rust (which is a trend I've noticed online as of late...?)

you don't know why a user would want to use c/c++ or rust, just as i don't know why you want to use go for your project. i could easily say something stupid like "you use go because google made it", but that's probably not true, you probably use it because you've made go projects before, and you like that it's got garbage collection, or someone recommended it to you, or something.

[jonbarrow]

the problem is i'm trying to implement this into a project that i've already written that uses rust crates and c libraries

Then this is a real issue being solved by a language port, like the examples I mentioned before. In which case I do think rewriting this in Rust is a more than valid path to take, and like I said I wish you luck with it 👍

you don't know why a user would want to use c/c++ or rust

Correct, in most cases I likely wouldn’t. However in this case you did give us reasons. I wrote my replies based off the information you provided. Your original comment opened with i'm writing a game server, and my favourite programming language is rust, so i'd like to write it in that, which (understandably) reads as the only reason why Rust is being mentioned is due to a language preference and nothing else (which is again something I already said). That, combined with the trend I and many others have noticed with regards to people making specifically Rust ports to the point where it's become a joke in of itself ("rewrite it in rust/RIIR", "carcinization of project/language" and other similar phrases are common jokes/teases at the trend, you can find many such instances all over the internet), it's not unfounded here for us to assume that was also the main goal with this question, simply RIIR. Hence why I responded the way I responded. I can only work off information I have, and this wasn't a case where no information was given to be fair. There was enough to make a reasonable assumption, I just used deductive reasoning based on the information at hand. Which just happened to be incorrect given new information

you probably use it because you've made go projects before, and you like that it's got garbage collection, or someone recommended it to you, or something

This is actually many of our first times using Go, and while it has good recommendations generally (both from the masses online and people we know), it was not chosen specifically because of just being recommended nor was it chosen for any one specific feature. It was chosen for a number of different reasons, including (but not limited to):

• Syntax that is simple enough to write that those who had not used it before/are coming from other languages can easily pick up on it intuitively while still maintaining a robust feature set (which was only expanded upon in later updates with the introduction of generics and more powerful type aliases, etc.)
• The language offers a good balance of safety and simplicity, while still allowing you to take down many guardrails if you so choose to do so. One example being the garbage collector as you mentioned, however you can use the C and unsafe packages to manually manage your own memory if you ever need to. This makes Go incredibly flexible when you need it to be, while having sensible guardrails in place for most typical projects
• It of course has the general benefits of not being an interpreted language (speed/performance, portable once compiled without needing to install a runtime, etc.) like Python or JavaScript
• The languages goroutine concept makes (basic) concurrency trivial compared to some other languages, and when paired with channels make for almost effortless load balancing
• This one is more subjective, but the returning of errors as values from methods is a great pattern for error handling compared to other languages where you sometimes run into random exceptions you didn't even know could happen (I'm looking at you JavaScript)
• Go ships with incredible debugging tools out of the box such as pprof which you can use to gather statistics of a running process in real time including CPU performance, memory allocations, the number of goroutines and if they are parked or not, etc. The tools Go ship with also allow you to export this data in several different formats including fully labeled graphs of how data flows through the program from function to function (which was incredibly helpful for finding a memory leak we had here where one function was parking thousands of goroutines which prevented their resources from being freed)
• Etc.

The decision to use Go over a language we may have already been familiar with (JavaScript, Python, PHP, etc.) was done by considering the requirements of the servers, and comparing feature sets. Of course there are other languages which have many of these same draws, including Rust. I have nothing against the language, and I honestly hear nothing but good things from most people. However the difference between our decision to use Go, and my previous understanding of why you wanted to use Rust, was that we did not choose Go due to purely a personal language preference and to port an existing library due to only that preference. Which again, is what I had already said. My entire previous reply was against the idea of fragmentation simply for the sake of language preference and to encourage unification through combined teamwork to make one, rock-solid, implementation (at least to start with, from which other language ports could more easily be made), which is one of the biggest real strengths of open source. Which, as you've not clarified, was due to a miscommunication (though I do still stand by the original opinion that I think it's healthy to give back when working in open source, in some capacity, since you said i don't think it makes sense to say i should contribute back to this project, even if i want to, just as being good faith and in the spirit of open source)

i would only need to implement the minimum set of nex stuff to support that specific game server anyway

I have to ask, what game is this? I'm not aware of anything that needs only this library to function, it sounds like you're saying you don't need the protocols here? I'm curious what your project is! It sounds interesting!

[TheAwesome98-Real]

I'm not aware of anything that needs only this library to function

it wouldn't be just this library, there are other things i need to implement, but they're all based on this, so it's the first thing i'd have to do

(actually reading this back yeah i can see how easily what i said was misintepreted, i meant to say i'd only have to implement the minimum that that game server supports)

my project itself is just an idea i want to see the viability of - it is multiple game servers within one. i've already written some of the psn component, because that's the only other game console online service that i'm familiar with, and i thought that the next thing i'd try to implement would be the wii u since it has good emulation, development tools, and the pretendo project is open-source.

[jonbarrow]

my project itself is just an idea i want to see the viability of - it is multiple game servers within one

I can go ahead and give some insight on that now in case it prevents you from wasting time, since it's an idea that was thrown around on our end a while ago too

It's not very viable. Every game has a unique access key which is used as part of the packet validation and transport mechanisms. If you don't use the right key that the client expects, then the connection fails. And there's no way to heuristically know which client is being used outside of dedicated ports, which at that point it's no longer "in one". It might be the same process, sure, but it's not the same endpoint. Not to mention that different games use this access key in different ways, and some have completely unique ways of using it (Luigi's Mansion on the 3DS is one such game which uses its own algorithms)

On top of that, each game tends to vary in its configuration. Sometimes slightly, sometimes by a lot. Theres the difference in NEX/PRUDP versions obviously (which changes things like packet structure, acknowledgment mechanisms, RMC payload structures, etc.) but also even things like the packet MTU tend to vary, the largest variance being that of the friends server which has a very low MTU compared to other servers, though multiple other games also have different MTUs (the game clients seem kinda lenient on the MTU specifically but it's something to keep in mind). If you don't use the right library configuration for a game then the connection will be unstable at best, unusable at worst. Nintendo and 3rd parties didn't have to worry about this because the server library versions always matched the client, but for us we need to support all variants and thus each game has its own configuration

It's technically possible to get, like, super basic stuff working. You could probably get multiple games to establish a basic PRUDP connection if you patched the clients access keys to all be the same. But for actual game functionality you'd be hard pressed to get multiple actual games running on the same endpoint (PRUDP uses the concept of "endpoints" rather than "servers") unless unless their configs match 1:1 (which, again, normally isn't even possible due the differing access keys). At best you'd likely just have multiple endpoints with different configs running in the same process, but that's functionally no different than just having them in separate processes

And that's just issues with the basics. As I said PRUDP uses "endpoints" rather than servers, and advanced clients can establish multiple connections to the same address which all have their own isolated state through the virtual port system. These virtual connections can have completely different configurations as well, including their OWN access keys separate from the main one, so now you'd be managing THOSE on top of whatever you're trying to use to maintain the state of the differing client connections on the single endpoint

And that ignores the other logistical issues like general state management which sounds like it would be a nightmare

[TheAwesome98-Real]

the games i'm talking about are the same/very similar game, but on different systems. i only need nex for the wii u version, since it uses that

[jonbarrow]

Ah I see I apologize, I misunderstood. I thought the goal was to run many different NEX games via one server (such as running SMM, Splatoon, and MK8 all together at once). I see what you mean now. Sounds interesting! Good luck with that 👍