cloud.weave.works is down + missing page in weave documentation

[jpetazzo]

Hi!

I noticed that cloud.weave.works was down.
(According to https://downforeveryoneorjustme.com/cloud.weave.works?proto=https, it's not just me, at least at the moment.)

This is (or at least, was) mentioned as the method of choice to install Weave on Kubernetes clusters:

kubever=$(kubectl version | base64 | tr -d '\n')
kubectl apply -f https://cloud.weave.works/k8s/net?k8s-version=$kubever

It worked a few days ago (at least last Monday 26-SEP-2022 according to my logs) but doesn't work today (Monday 03-OCT-2022).
I thought - maybe this was decommissioned, and I should update my deployment scripts?
But when I go to the Weave documentation (https://www.weave.works/docs/net/latest/kubernetes/), it says "The recommended way of using Weave with Kubernetes is via the new Kubernetes Addon". But the link to "Kubernetes Addon" redirects to the Weave Net overview (https://www.weave.works/docs/net/latest/overview/).

I'm not sure what to do next; I'll fetch a cached copy of the Weave YAML somewhere but I thought it might help others to report this and get it fixed ♥

[clacsinaRX-M]

Encountered the same issue this morning.

I was able to install weave on both a 1.24 and 1.25 cluster using the 1.11 yaml from the release page: https://github.com/weaveworks/weave/releases/tag/v2.8.1

kubectl apply -f https://github.com/weaveworks/weave/releases/download/v2.8.1/weave-daemonset-k8s-1.11.yaml

That seems like an older version so I'm unsure if that's going to continue being supported, but so far things are working as expected.

[prassurohith]

Thanks @clacsinaRX-M

You saved me its working for me

[jtgorny]

Oof https://www.weave.works/blog/weave-cloud-end-of-service

[fujitatomoya]

Is there any official and up-to-date one instead of #3960 (comment)?

[zenhighzer]

Oh no, I also wondered why cloud.weave.works is down.

But if I access "https://www.weave.works/docs/net/latest/kubernetes/kube-addon" I land on the Overview-Page. So actually I cant find any Page showing how to install the kubernetes addon

[monadic]

Hi, we are looking into a workaround if we can provide, or some alternatives. Watch this space.

[lizwarner-weave]

Just confirming that the workaround here #3960 (comment) should work. We are updating our documentation and will comment again on this issue in a bit

[seanmcconkey]

While we're waiting for the fix, do you not think it prudent to get someone in the organisation to update https://www.weave.works/docs/net/latest/kubernetes/kube-addon/ with the fix and a link to the EOS announcement.

[lizwarner-weave]

Updates:

1. We have updated https://www.weave.works/docs/net/latest/kubernetes/kube-addon/ so users are no longer directed to Weave Cloud to obtain Weave Net
2. Please also see Opening up development of Weave Net #3948 if you’re interested in continuing to be involved with Weave Net as a community member
3. Please note our end-of-life page which covers both Weave Cloud and Weave Net

Thanks for your feedback and patience

[kingdonb]

JFYI, this is the top result when searching for "weave net one-liner" so at least 50% of people will probably click this first: https://www.weave.works/blog/weave-net-kubernetes-integration/

I was unable to find this blog page in any of our git repos, or I'd be updating it myself right now, but it still contains the reference to cloud.weave.works – thanks Weaveworks team for handling this issue so quickly already 🙇

[dholbach]

@AhmedAtefWW can you help?

[jpetazzo]

Thanks for updating the documentation, and confirming that the YAML on the GitHub release pages is now the canonical source; that's very much appreciated! ♥

One little remark: the YAML that was just released there a few days ago (weave-daemonset-k8s.yaml) uses the :latest image tag. Personally, I would rather use a pinned version, like in the other manifests in that release page. Pinned versions avoid issues due to unexpected upgrades; they ensure consistent versions across nodes; and they're easier to troubleshoot (because kubectl describe on the DaemonSet or Pod will show you exactly what you're running).

For folks wondering about the differences between the -1.8, -1.9, -1.11 YAML manifests:

• the 1.8 manifest uses a DaemonSet apiVersion extensions/v1beta1
• the 1.9 manifest switches to DaemonSet apps/v1
• the 1.11 manifest adds priorityClassName: system-node-critical to the Weave Pods

There was no change in DaemonSet API or behavior since then, so the 1.11 manifest (in fact, the 1.9 manifest!) will work just fine on Kubernetes 1.25.

Thanks again!

[lucj]

@lizwarner-weave is there a dedicated URL to install weave scope as well ? The installation instructions still refers to cloud.weave.works (https://www.weave.works/docs/scope/latest/installing/#k8s)

[enekofb]

@lizwarner-weave is there a dedicated URL to install weave scope as well ? The installation instructions still refers to cloud.weave.works (https://www.weave.works/docs/scope/latest/installing/#k8s)

@lucj looking into this

[enekofb]

@lucj see the update on scope docs

let me know any feedback at the back of it

[fireflycons]

@monadic Any chance of creating a helm chart? Would be useful to be able to configure things like IPALLOC_RANGE

Would you accept a PR for one?

[monadic]

Sure please email me about this Alexis at Weave dot works

[jpetazzo]

One little remark: the YAML that was just released there a few days ago (weave-daemonset-k8s.yaml) uses the :latest image tag. Personally, I would rather use a pinned version, like in the other manifests in that release page. Pinned versions avoid issues due to unexpected upgrades; they ensure consistent versions across nodes; and they're easier to troubleshoot (because kubectl describe on the DaemonSet or Pod will show you exactly what you're running).

... And indeed, using the :latest version brought some unexpected problems; it looks like the :latest tag isn't properly multiarch:

ubuntu@node1:~$ uname -a
Linux node1 5.15.0-1021-oracle #27-Ubuntu SMP Fri Oct 14 20:04:20 UTC 2022 aarch64 aarch64 aarch64 GNU/Linux
ubuntu@node1:~$ sudo docker run --entrypoint sh -ti  weaveworks/weave-kube:latest -c "apk -q add file && file weaver"
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
weaver: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), statically linked, for GNU/Linux 3.2.0, BuildID[sha1]=1a29ea0e5d7e33dcee87512f32068e17dae6f78c, stripped
ubuntu@node1:~$ sudo docker run --entrypoint sh -ti  weaveworks/weave-kube:2.8.1 -c "apk -q add file && file weaver"
weaver: ELF 64-bit LSB executable, ARM aarch64, version 1 (GNU/Linux), statically linked, for GNU/Linux 3.7.0, BuildID[sha1]=892358108af8a6f0d0c5427a34943ebfe471b84c, stripped

I'll open a separate issue to track this.

[kallisti5]

I just ran into this. how lame of weave 😢

[kingdonb]

I'm pretty sure you can still:

kubectl apply -f https://github.com/weaveworks/weave/releases/download/v2.8.1/weave-daemonset-k8s.yaml

as described on:
https://www.weave.works/docs/net/latest/kubernetes/kube-addon/

I use Weave Net, but I don't e2e test it daily. If there's an issue that it can't be installed, please cc me as it will be a problem for me sooner or later. But from what I've seen, they've still been taking care of the installation blockers as they come in.

🙏 🙇 📿

[jpetazzo]

Hi @kingdonb ! 👋🏻

I'm not sure which issue @kallisti5 is talking about; but perhaps it's about the multiarch problem described just above (and tracked in #3974) - if you kubectl apply the default YAML on an ARM cluster, it won't work because that YAML is referencing :latest, which isn't multiarch.

Cheers!

[kallisti5]

I'm not sure which issue @kallisti5 is talking about;

Just the shutdown of cloud.weave.works. It was a really nice service to use.

[kallisti5]

It also looks like the last weave release was in 2021. That doesn't bode well for its future...

https://github.com/weaveworks/weave/releases

[kingdonb]

The shutdown of Weave Cloud shouldn't really have had anything to do with Weave Net, but it unfortunately did impact

There are people interested in rekindling Weave Net here:

• Opening up development of Weave Net #3948

It's going to have to come from volunteer effort though, as it always has! There have been hotfixes to keep the one-liner instructions working, and there has been upstream issues that breaking changes impacted Weave Net, that got sorted out ultimately, and there are people who are enthusiastic about keeping Weave Net going, but it's definitely been on life support!

[fujitatomoya]

i would like to revise #3960 (comment), anyone working to address this issue to support multi-arch image?

[kingdonb]

I would like to see multi-arch get resolved, and I can likely help test it if someone else is driving the effort. 🙏

[jpetazzo]

Hi @kingdonb, all that is needed for multi-arch support is to update the released YAML so that it references tag :2.8.1 instead of :latest (see #3974 for details). I don't know how to update Github releases (it looks like they were done manually, at least the last YAML) otherwise I'd offer a PR. Let me know if that's not clear, I'm happy to provide better details!

[kingdonb]

Those are pretty concise and to the point, thank you for speaking up about it!

I'll see if I can find out who has the keys to push those tags around today. ❤️

[kingdonb]

OK, so this is really bad. I think this means that new clusters on Weave Net that just follow the released instructions cannot have Arm64 nodes without some modifications. This is really bad because, the arm64 support exists, it is there, it's just obscured by some mistake that occurred during triage of the cloud.weave.works website itself sunsetting, with Weave Cloud.

And it can technically be fixed with a single change, but there is no commitment from us at Weaveworks to continue the maintenance of Weave Net distro, as has been echoed from the start of this issue. And there has been a loose formation of a new team, and then the holidays. I'm just repeating the events so far, for anyone that is just joining the conversation.

I think the instructions that are hosted at Weaveworks that "need to be updated" now are actually on the Weaveworks blog or corporate brand site. I don't believe any keys have been handed over to anyone.

So if a new team is being formed that's composed of people who are not at Weaveworks, they're going to need somewhere else to post the instructions that explain how to get at the latest release. I'm not sure if that team has a name. Does anyone know the answer yet?

So is there a GitHub org for New-Weave-Net org yet?

Or has this conversation already been covered somewhere and I need to catch up with the recording? I think the team needs a name first, (and not to put the cart before the horse, but then there is the possibility that more than one competing team springs up... I'd be happy to work with one point of contact or multiple but I'd prefer a single team with at least one committed point of contact that I can name.)

I'm doing the best to facilitate this next step right now without being antagonistic to anyone, please nobody read this note as passive-aggressive or anything else, as I genuinely don't know if there is a new maintainer yet... but I want to help! 🌷

[kingdonb]

And for anyone that's aware of the precarious maintenance status of Weave Net right now, it should not be a problem to make a one-line change to the YAML in order to get access to multi-architecture for their cluster. It is risky because there has been no maintainer, I wouldn't recommend it for production.

So I think the new release team needs to create a website where they will host the instructions for Weave Net going forward, (and I will be glad and volunteer to help with this.)

Is there such a team where the maintainers can be enumerated now, or is it public?

I would like for this to be easy, and for things to just go on working for everyone, but I also want to avoid that someone winds up with Weave Net in production, and a CVE is not addressed, and then it's then Weaveworks who is indirectly responsible (in spite of the license which says we cannot be responsible), because of no maintenance and a poorly managed handover that resulted in incorrect signaling about the current maintenance status.

I'm trying to say, I think the bad news is we cannot "just put it back" from latest even though it's clearly in error. The doc may need to remain in error, and we keep it as incentive for folks to move on to the new maintainership... once it is extant!

I'm happy to be overruled by anyone that has the keys and can actually fix it now, but I'm actually not in a position to make the change requested myself, and I'm doing my best to telegraph the disposition of engineering resources in the company who are in position to trace that down, but have been asked to focus on other stuff.

So as respectfully as possible, I think right now the blocker is: there is no new website from the new maintainer team.

[fujitatomoya]

I'm doing the best to facilitate this next step right now

thank you so much for your contribution!

but there is no commitment from us at Weaveworks to continue the maintenance of Weave Net distro, as has been echoed from the start of this issue.

off topic but https://github.com/weaveworks/weave is still maintained? i see that last commit is 3 months ago...

And for anyone that's aware of the precarious maintenance status of Weave Net right now, it should not be a problem to make a one-line change to the YAML in order to get access to multi-architecture for their cluster.

right.

It is risky because there has been no maintainer, I wouldn't recommend it for production.

I think this is one of the huge driver if we use weave... but correct.

I am happy to help anything i can do, but i am just one of the user with low bandwidth...

[kingdonb]

off topic but https://github.com/weaveworks/weave is still maintained? i see that last commit is 3 months ago...

Look closer, there have been less than 10 commits since the end of 2021, and there have been no releases other than the changes necessary to avoid leaving everyone stranded when Weave Cloud service and domain name was decommissioned.

It's not that there is no living person who has write access, it's that there is no active maintainer or "product owner" if Open Source can be considered a product, which it can.

There has been some small docs contributed yet, but I work at Weaveworks and have been in the conversations, so I can tell you with certainty, there is no commitment from Weaveworks Inc. to maintain Weave Net going on forward. It will have to come from the community, and I hope that it does. Weave Net may be formally sunset if it can't be maintained effectively.

I appreciate you @fujitatomoya I am just a user of Weave Net, too. I hope it does not come off as too presumptuous stating that I will volunteer to facilitate the new maintainer when we don't yet know who that person or group is. I cannot be the volunteer myself, I have too many other roles at Weaveworks and other commitments in open source. But I have taken over Open Source projects before that were left behind by a company when the project was in some transitionary state.

I can offer mentorship based on that experience, and material support (server hosting) to whoever wants to do the hard work – building and testing, and preparing the next release, and documenting what issues, etc. – all those jobs of a maintainer!

I can be reached on Slack (CNCF #flux) by anyone who is looking for help to answer this call.

[rajch]

@kingdonb I was one of few people who volunteered to help in any way possible. We had a couple of meetings, but beyond setting up a rough roadmap of activities, we didn't reach any conclusion.

Meanwhile, I set up a replacement endpoint for the one-line weave setup on my own. It's not "official" in any way, nor is it mentioned on any web site other than issue #3948, but me and students of the classes I teach use it. I'll make the correction suggested by @jpetazzo there.

[kingdonb]

replacement endpoint for the one-line weave setup on my own

I would like to help you amplify this, but maybe a meeting would be better first. Thank you for sharing Weave Net with your students! :)

Let's meet some time, again. I would like to meet you personally, but maybe it makes sense to put another date on the calendar for the working group, in case any other members are still interested.

🔔 🪐 ringy bell ringy It will take some time to establish a new group and decide what it should be called, as well as formalize leadership. Let's continue the conversation. 💯

[rajch]

Let's set a date. It would be nice to have some more people (perhaps @fujitatomoya ?) in the meeting.

[fujitatomoya]

yeah thanks for bringing this up, count me in 👍

[kingdonb]

This is the time:

• 2PM (14:00) Thursday (Feb 2) ET - 19:00:00 GMT

Please E-mail kingdon at weave dot works

Edit: I have two confirmed @fujitatomoya and @rajch 👍 I'll be there myself as well, of course. As far as I know, that is the team so far.

Anyone else who is interested, please e-mail me to be added to the calendar. subject: Weave Net

[kingdonb]

We had our meeting today, and we committed to join another meeting at the same time next month:

• 2PM (14:00) Thursday (Mar 3) ET - 19:00:00 GMT

Goals: to run all of the tests and build infrastructure, to provide an environment that can run all the tests (for public good) and demonstrate its use, to make an inventory of any dependencies and submit a PR for upgrading them... we can hope to bring in more volunteers next month, and share our experiences from running the tests.

Thanks for doing this Tomoya and Raj 👍

The consensus of the group was, we need more volunteers, or at least party members, in order to make this a sustainable enterprise.

The discussion continues:

• Opening up development of Weave Net #3948