FW 3.2.7 breaks RADIUS certificates #61
Comments
|
I am running into the exact same issue, also not directly using that script but I referred to it for debug since it was said to be compatible with 3.2.7. I have been looking into it for a few hours now, to no avail. Something is overwriting the certs served by freeradius in The certificates in I'm wondering how that script gets around this and manages to get its LE-backed certificates served persistently instead for EAP? |
|
@ouaibe just checked my logs and most recent cert update on my box was before updating it to 3.2.7. Updated the README to reflect that there seems to be an issue with 3.2.7 Early Access. Will need to sift through that, any hints appreciated. |
|
official UI thread for their 3.2.7 release, I posted a comment but havent heard anything back https://community.ui.com/releases/UniFi-OS-Dream-Router-3-2-7/bdc93e02-3f71-4196-8553-46664c605e13 |
yes, thanks, just saw that. On your question for downgrade - not sure if that works (for you...), but switching to the official firmware from EA may bridge it for your case? And I'm still on the fence if the suggestion to put in a feature request for this was malevolent or uninformed: such requests exist since ages and this repo will be archived the minute I see it implemented in this "semi-pro" product... |
|
@therealpaulgg just info: Running @ouaibe no chance yet to test RADIUS as I don't use it right now. |
|
See that's the unfortunate thing - I am only on official. I never participated in early access - but they've pushed it to us regardless. I dont even have a different release channel as an option interesting on your restart not changing files...maybe there's a solution for us. edit: I've been using SSH to do all this management...you use Web interface? not even sure how that works |
|
@alxwolf thanks for checking. On my end, nothing in |
|
@therealpaulgg - UniFi has three places for certs, "Web Interface" being one (which requires restarting @ouaibe and @therealpaulgg: you're both using an internal CA - a while ago, I experienced that UniFiOS will remove invalid certificates and overwrite those with default self-signed ones.
Would you mind checking your certs, if they are actually valid and include a full chain up to a trusted root certificate? (and that's where the black box is with me not knowing which root certs are trusted by UniFiOS...) Hypothesis right now is that somehow the UDM "does not like" your certs and by that overwrites them. |
|
I thought this could be a possibility, but note that it worked flawlessly for years prior to 3.2.7 and I didn't change anything before this update. The irony is that my certs validate using Here's my cert chain validation (note that I installed my CA in the local cert store using Here's Unifi's cert chain validation: Which is probably because the CA isn't in the local store, but we can specify the CA manually using Note that I had backed up the "older" certificates prior to the 3.2.7 upgrade, and they definitely changed their cert chain with that update (i.e. they did "something). Here's the info on the previous one: I also thought that maybe they had now put their root CA in their known keystore and were re-generating from this upon It's very possible that their concept of "invalid" (causing the replacement) morphed into "it's not ours", but it's hard to tell because whatever validation they're doing is completely opaque. |
|
I know this is "late" in the discussion, but I can also confirm that the certs are being replaced by From And using That being said, I'm unable to find the source of these certificates, nor which exact process does the replacement. |
|
So, seems some more people noticed it. |
|
my internal CA is as follows: Root CA the router certificate is signed by my intermediate CA. I just checked /etc/ssl/certs and looks like there's some stuff that has been deleted, including my root certificate from the keystore. 
When I have some more time I'll play around with this... |
|
Well, it seems to be a somewhat bigger mess created by UI (with production systems losing WiFi connection according to statement in UI Community linked above. If that's the case, I'd expect the release to be pulled at least, if not rolled back. Glenn knows a lot to everything about the certificate story in UniFi OS, I hope he will address the topic. And as X-Mas is approaching - maybe this leads to a well-documented and automated, GUI based approach. Or not. |
|
Added my intermediate CA in to my root keystore, verified with openssl verify. Still wasnt happy - and reset my certs. Yeah hopefully UI team addresses. Not sure what else I can do for now. |
alxwolf
changed the title
|
@ouaibe may I suggest you address UI-Glenn directly as he explicitly offers it here |
|
@alxwolf thanks, I have already posted in that thread and I'm not seeing an offer from UI-Glenn to exchange on that problem specifically, but I'll look into it. |
|
Allright, I found the solution (which I had in front of my eyes for a while, except I missed a layer of b64 encoding). Turns out, with 3.2.7, the certificates (CA, Server, and a bunch of others) have been moved to the The JSON file elements where this can be found are Upon service start, the certificates are taken from the As such, it's possible to replace these entries in the JSON file and (probably?) make them persistent in this configuration. Here are one-liners using Note that this is probably also true for the So why did Ubiquiti make that change? I can't tel, but hopefully it is because eventually that JSON config file (that seem to hold the entirety of the UDM config) will be modified by the Web UI and allow the user to load their own certificates ? One can dream :) That being said, forcing a new CA chain onto all their users without any early warning is certainly not professional, with many users complaining about connection refusals from Apple (and Android) devices that suddently get a certificate change (for another untrusted one, mind you) https://community.ui.com/questions/UniFi-OS-3-2-7-CA-Certificate-changes-break-Radius-backed-802-1x-authentication/9a5d4fb9-f791-4bec-81b5-3b9cc0c7853a#answer/23a7c99a-e1f4-410f-95fb-670b4fb426d3 |
alxwolf
changed the title
|
@ouaibe Great analysis! Will have a look into the web server certificate in the other issue e#62 - the JSON contains another cert, suitable for web server, which is (again) the standard self-signed cert by UI (at least on my machine). So far, restarting but, maybe this is just a small step we have to overcome on the path to having it managed automatically. |
|
Thanks to you for all the hard work you did put into helping users side-load their own certs, it really is appreciated! 🤗 |
All thanks @ouaibe . This did solve the problem for me. |
|
Just to finalize this, it seems that after some Unifi platform reboot happened, the It is therefore my guess that the real cold config file is In order to address that, you need to proceed with the same modification of that file: After that, you can validate that both files are the same: |
|
I thought I had it but there's some other circumstance where both of these files get overwritten with a Unifi certificate (e.g. changing the Network setting and causing a re-provision of APs/the UDM). There must be some other place where these are stored, still investigating. |
|
One thing bugging me is that the UniFi Easy Let's Encrypt script by Glenn Rietveld (what a coincidence, his name is Glenn... right?) still uses the well-known places (before 3.2.7) when updating the RADIUS certs. |
|
Indeed, and there's something going on with MongoDB: it also stores the certificates as well. I haven't had the opportunity to go through it all (I'm not super skilled in NoSQL DBs), but I think this might be the source for the replacement when the UDM restarts on its own (let's say you enable/disable a Wifi network, that triggers the issue on my end). What I recently found out is that I have some weirdnesses in the logs when the UDM restarts, something seems to not like the JSON config files and causes a "re-migration/re-init", but I have zero idea why that is, and I tried to sync I looked quickly into the The Glenn Rietveld (no clue if that's a naming coïncidence) scripts are also doing something with MongoDB. I have posted in that thread in the Ubiquiti Community forums but didn't get any response... I'll continue drilling into mongodb (using |
|
Allright, I "think" I got to the end of it by modifying the right key/values in the mongoDB. This seems to survive the System config reload, the The following one-liners are used to load the right certs in the right collection by updating existing values (this One thing I haven't tested is if the other changes to the JSON files are necessary if this is done first. It might be the case that only this change to the MongoDB is needed ultimately. Also, I'm not sure how this will fare with updates, Ubiquiti has already changed all certs w/o telling its users, so this might have to be adjusted with future updates of Unifi OS. |
|
sadly, still no clear path on how to fix this after some trial and error... |
While I don't use this script specifically (I have a internal CA that I use to generate certificates for my intranet), I do at the very least replace
/data/unifi-core/config/unifi-core.crtalong with/data/unifi-core/config/unifi-core.key. Upon restarting unifi-core withsystemctl restart unifi-core, the OS overwrites my custom certificates, effectively undoing my work.My setup has been working for months and I never had to mess with keystore or the RADIUS config (raddb). I am trying to reach out to anyone who might know more. Only reason I noticed this was because my monitoring software detected it couldn't reach my router anymore...then did some digging and found this stuff out. Router must have updated at 3am causing this to break.
Originally posted by @therealpaulgg in #60 (comment)
The text was updated successfully, but these errors were encountered: