Compiling cells-sync under ARM64 (running DSM - a Linux distro for the Synology NAS): the nightmare #37
Comments
|
nice read :-) |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
In my never-ending quest of getting the Pydio Cells family of services running on as many platforms as I can, I've been tackling a years-old issue I have: how to synchronize between a Synology NAS and a remote Pydio Cells installation.
The 'obvious' choice would be just to get the Synology NAS to sync over its one-size-fits-all S3 sync solution, which works with pretty much everything... except Pydio Cells. Alas, that's a completely different issue and not relevant today.
Instead, after a complex procedure that allowed me to compile CGo packages for the ARM64 architecture on the Synology NAS — something which eluded me for months, too — I managed to get a lot more things to compile there, so I thought I could give CellsSync a try. Although Synology's DSM (= DIskStation Manager) is essentially a GNU/Linux distro, its configuration is somewhere between macOS and Debian Linux (coming closer to Debian Linux these days, since it already uses a
systemdsetup, although it doesn't ship with a package manager — deliberately so, you're not supposed to tinker with the operating system...). By far the easiest non-supported programming language to get working on the Synology DSM is Go (at least when compared to C/C++...), so, once you get Go compiling basically everything, it's supposed to compile Cells Sync as well.While I don't need the GUI version of Cells Sync, it seems that somehow the
Makefile'assumes' that the installation is 'complete', that is, that the whole UX is ready to be installed — even if it won't. Unfortunately, the Synology NAS is not meant to be an application development environment — it's way too slow for that. Fortunately, Go is blazing fast everywhere — and thus also on a slow ARM64 with a small memory footprint! — but, alas, that's not the case for practically anything else.npmtakes eternities to do anything worthy.Predictably:
(if you wish, I can send that log as well)
FYI, I've tried to 'modularise' Cells Sync and Pydio Cells, after noticing that Cells Sync relies on a
commondirectory which is used by Cells Sync... ugh. You really should put those common files on a separate repository, simply because building a CLI version for Cells Sync ought to be substantially easier to compile than the whole Pydio Cells mega-package! Or so I'd expect, anyway. After 4 days of making a serious attempt to get everything modularised, I had to give up (and trust me, I don't usually give up easily!).I believe that the main issue with the current Pydio Cells suite is its dependencies of incredibly obsolete packages, most of which have quasi-circular dependencies. There are a few ones that are simple — packages that just changed their current location, something which applies to BoltDB and Caddy, for instance — and a few that are very hard, and are at the deep core of Pydio Cells (but not Cells Sync!... alas, you can't compile it without compiling Cells Sync as well). I understand that the main reason for this obsolescence was the need to start a large Go project before modules came around. As practically everybody switched to modules (Google does a yearly survey to its community, and it seems that around 96% of the Go developer community regularly uses modules), it becomes increasingly harder and harder to keep running a complex application-building environment that relies on other sources. To be more specific, the mere idea of using
npmto handle Go module dependencies is, to be blunt, insane. Even the core Pydio Cells uses Go-specific, pre-module packages to handle those dependencies (at least, as far as I know).Particularly hard to fix are the dependencies around the
go-micropackage (extensively used by Cells to provide an environment of microservices), mostly because there are at least three different packages doing the same (and using the same name). Two of those are simply major versions of the same package — but, under the rules of Go modules, each major version (because it's API-incompatible with the previous one), gets its own package & namespace (be it a Git branch or a completely separate repository).go modcan handle these easily, but the pre-modules system has a really tough time to deal with them — simply because some packages may also be updated and require the old major version, while newer packages, which have been updated, rely on the new major version. Go modules deals with those issues easily enough; the 'vendor' approach used by Pydio Cells gets hopelessly confused. Finally, there is a third source ofgo-micro. As far as I can see, some of the concepts are similar; they might even have started as a fork of each other, and then moved on in different directions. Pydio Cells sometimes needs subpackages of one, sometimes of the other. This is sadly unfortunate for those, like me, who are unaware of the real differences between them, and, just by looking at what the Pydio Cells source is including/requiring, it's not obvious which is which. Eventually, as the dependency tree grows, it becomes obvious when the 'wrong' package had been included, and that means going back and changing the import path just for that file, and move on with the generation of the dependency tree (i.e.go mod tidy).The second big issue has to do with the usage of Google's
protobufimplementation in Go — which, again, has seen several different releases, often incompatible with each other. Google has kindly left a reasonably old implementation available on GitHub (it's 5-6 years old, at the time of writing), but they seriously warn everybody to move on to the current, official version, which is not hosted on GitHub but on Google's own Git repository for Go-related things. So far, so good — you can continue to use the old version, after all — but the problem, not surprisingly, is similar to the one above: some packages used by Pydio Cells have never been updated, and use the old version ofprotobufto generate their own protobuf bindings; others have been updated, and use the new version; and Pydio Cells itself knows nothing about 'different'protobufpackages, and gets in trouble if the 'wrong' one is included — mostly because, from the perspective of the Pydio Cells code, nothing is wrong: they just include a third-party package (which might even be up-to-date), which, in turn, is calling either the old or the new version ofprotobuf... and theoretically, from the PoV of the Cells developers, that's a problem to be fixed upstream; they just know what to call from the third-party package, and are assuming that whatever it depends upon is handled correctly. And, actually, it is... with Go modules, which tell exactly what has to be included, and the correct order for including all of that (even if different versions of the same package are used, Go modules will handle those cases easily as well). The problem is when one uses such third-party packages without using Go modules — then it means manually handling all dependencies correcly, and that will be true not only for the particular file/module/package one is working with, but for all of the others as well, since — well, since they can be importing different versions, too!Right. You can see how this can quickly become exponential. But, as you might imagine, to make matters worse,
go-microrelies heavily uponprotobuf. And now Cells is including 3 versions ofgo-microand at least two versions ofprotobuf, which means that, every time some import path requires to be fixed on a file that uses microservices, there are at least 6 different combinations to pick from, and only one of them will be correct. With a bit of luck, you might get it right for the handful of files inside one of the many Cells submodules (worst-case scenario: try 6 different combinations in turn, you'll get it right at least on the last try...).Obviously, when I started doing all the above, I really had no idea how many different versions — and combinations of versions — were available. That meant that after 'fixing' a lot of dependencies on the several submodules, I finally hit one submodule where there was a new combination of versions that I hadn't encountered before. This meant dealing with it for that particular submodule (or file inside a submodule)... and hoping against hope that I had fixed the first submodules correctly. Murphy's Law, again, showed that not to be the case, so very quickly I was running in loops — getting dependencies right, one by one; getting stuck on a difficult case; fixing it, restarting from scratch; now the ones I had fixed before were, obviously, inconsistent, so I had to change them again. Run
go mod tidyagain... getting stuck on a new submodule, with a new (previously unknown) dependency. Fix it. Again... the early modules broke... so I had to start from scratch... and so forth.Needless to say, after four days, I never managed to even get a fully working dependency tree. Every day (sometimes more than once) I had to restart with a 'clean' installation because things had become so hopelessly tangled that I lost the thread — the best choice was to start from scratch and rely on the knowledge I had acquired the day before. Wash, rinse, repeat.
Until, well, it simply was too much for me.
I now start to appreciate why @cdujeu never bothered with changing everything to Go modules :-)
On the other hand, at least
cells-synccould have a CLI that did not depend on the whole of the UX and a fully-compiled installation of Pydio Cells. That would go a long, long way to make my job easier.So I guess that I need to dig into the
cells-synccode to figure out the 'magic' of syncing (uploads and downloads are trivial, especially when usingcells-client, of course :-) ).I know this has been a very long essay, but I hope you feel my frustration. Remember what my original intention was: having a way to sync folders from my home NAS to a remote server running Pydio Cells. That seems trivial enough, right? Well, one thing is to copy files in either direction. That's the easy bit. The other issue is to make Pydio Cells understand that new files have arrived, and index them. That's the tricky bit. Of course, I can always run the scheduler (manually) every now and then, to reindex the whole folder. Unfortunately, that takes 2-3 minutes (under my current scenario), and consumes a huge amount of resources (memory, CPU...) during those 2-3 minutes. What I need is to do the same thing that Cells Sync does: figure out what are the differences between the local and remote folders, send them over, and index them on-demand as they arrive. That means near-real-time sync'ing (just as is provided by Dropbox, OneDrive, Google Drive... and Cells Sync), as opposed to having to ask people to 'wait a bit until I manually resync the folders'. Note that, from the perspective of the operating system, the folders are in sync all the time (yay
[rclone](https://github.com/rclone/rclone)!). The problem is that there is no easy and obvious way to inform Pydio Cells about the changes...Aye, I'm also aware that I could theoretically connect to Pydio Cells using the Amazon S3 protocol. That, unfortunately, only seems to work for some cases but not others; I have absolutely no idea why (there are few logs to help me with a simple 'authentication denied' error). The exact credentials have no issue whatsoever with things such as Postman or even
s3cmd; I guess that there is some step I might be missing...The text was updated successfully, but these errors were encountered: