-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Run 2 Instances Of AdGuard Home For Redundancy #573
Comments
Well, usually it is possible to configure multiple DNS servers in the router DHCP settings. However, one of them will be the primary server, and it will be used most of the time anyway. |
Think this should be reopened and made a feature request. Routers don't strictly have a primary and secondary DNS in the sense that ones a backup only server, it's just poor terminology that gets used around the place and whether a router favours one DNS server over another is up to implementation and vendor. Generally if multiple DNS servers are configured, they will both be used. Even if one was used more frequently than another, that leaves a percentage of requests going to a different DNS server and if thats not an Adguard server the security measures provided by this product are null and void for those requests. There's various reasons to want to want to run multiple instances, with the main one being people simply may need to reboot a system from time to time. Either the router will have one DNS server configured which is expected to go down occasionally, or theres multiple and traffic 24/7 will be going to both. Ideally rather than the other DNS server being a public one, it could be another Adguard instances so irrespective of what server is used, the traffic is protected. Right now thats possible no doubt, but any whitelist, blacklist etc etc will need to be manually applied to two servers. It'd be a great advantage if the web UIs allowed for synchronisation of config which would make managing multiple instances substantially easier and encourage more secure measures. FWIW this is the most upvoted, not implemented, feature over at Pi-Hole so there is demand and I suspect implementing it would win many a customer over. |
@iSmigit well, there's nothing that prevents running multiple instances of AGH as long as they have different configurations / listen ports. |
Thanks for the response. The subject title and my assumption is that the OP and myself would both like to see identically configures instances running on two systems. If one instance was to go down the endpoint or router could direct traffic to the other DNS server, which are both identically configured. As you note this is already doable, but if you have config that’s not out of the box then you need to manually apply it to both instances. In particular adding filters means managing multiple instances where I think it would be awesome if the application itself could accept the host name of a second instance of the application and have them replicate the system config (filter rules) whenever a change is made to one instance. Unsure if it’s work well if the DHCP service was enabled, but for pure DNS filtering I think being able to mirror config automatically would be awesome. |
I just think that this is a bit too complicated for the current stage of AGH development. If we're talking about running two different instances of AGH on two different machines, it should be possible to simply
Also, I think we can do it step by step and start with smaller things. For instance, we could start with implementing "reload" and "configtest" operations (like what nginx provides). With these operations it'll be easier to implement settings sync via |
I would like this feature as well, any possibility of opening this issue back up? |
Reopened as a feature request, and issue priority set to "low" for now. If you upvote this feature request, please also add a comment explaining your use case. |
The important issue to solve would be to allow multiple instances of AdGuardHome to be deployed and synced with each other, for example a Primary, Secondary instance. My use case is to make AdGuardHome to be highly available (HA), thus providing near-zero downtime to DNS requests when my main AdGuardHome instance goes offline. There are many different factors that would allow AdGuardHome to go offline like rebooting for patches or hardware failure. A stop-gap would be to use NGiNX, haproxy, keepalived or any other TCP/UPD load balancer with the combination of rsync / scripts. Having it be provided out of the box in AGH would be awesome. It's the top requested feature of the PiHole project: Edit: For those finding this issue. I have moved to Blocky. It is completely stateless and can be ran HA with minimal configuration. |
Can you tell me how to upvote it? I would really love this as well. One server goes down, wife says "The wifi is down!! What did you do?" if I have two servers which are synchronized, that buys me some WAF points. |
@jschwalbe and others in this issue. I would check out Blocky. It is completely stateless and can be ran HA with minimal configuration. |
@jschwalbe just ad a 👍 reaction to the issue |
GoalFirst, let's clearly establish the purpose of this feature. It's not about load balancing -- if you want load balancing, use HAProxy. It should be communicated that load balancing is completely out-of-scope for this project. This issue is about config synchronization, which is a blocker to running multiple instances of AdGuard for High Availability. I'd like to explore a potential solution which I think would give AdGuard a edge against competitors: webhooks. I'm interested in contributing here because I think AdGuard's code quality is leagues ahead of PiHole. I also want to stress that I don't think this issue has been adequately labeled: based on dozens of attempts by users on reddit and pihole forums and hundreds of participants on those threads, a lot of people want this. AdGuard could be the first to have it. (Note: Blocky referenced above does not have this feature, it just makes it easier to deploy 2 duplicate instances based on stateless config -- whole different ballgame) My ProposalThis is the simplest and most robust solution I could come up with. Enumerate a set of hookable events, for example: const (
DnsConfig EventType = "dns_config"
DnsRewrite EventType = "dns_rewrite"
DnsSafeBrowsing EventType = "dns_safe_browsing"
DnsAccess EventType = "dns_access"
DnsParental EventType = "dns_parental"
DnsSafeSearch EventType = "dns_safe_search"
BlockedServices EventType = "blocked_services"
Dhcp EventType = "dpcp"
Stats EventType = "stats"
Private EventType = "private"
QueryLog EventType = "query_log"
Filter EventType = "filter"
FilterRule EventType = "filter_rule"
I18N EventType = "i19n"
Client EventType = "client"
Tls EventType = "tls"
)
Then, rather than baking the config sync service into AdGuardHome, we could write another small microservice that waits for webhooks and pulls config from the primary, then pushes it to all secondary nodes. The microservice could even be very clever and do bi-directional sync if it could diff the changes. It may actually be better to eventually put this into the core, but webhooks would be a good first step. #1649 is a draft PR of my plan of attack. If approved, I can also draft up and example sync service. If we would prefer to put the sync service into this codebase, I have ideas for that too. |
@subdavis let's discuss the proposal here. Otherwise, the discussion will be fragmented. First of all, thank you for your insights. For some reason, I didn't realize that this feature is that desired. We should definitely find a way to help users configure it. Regarding webhooks, I generally like the idea of providing them as a feature by itself, it will definitely help people build projects integrated with AdGuard Home, and it will be a great addition to the API. However, I don't think webhooks should be purposefully limited to config changes. The most popular feature requests in this repo are about query logs and statistics, and I suppose webhooks should support them. About the implementation draft, I think that synchronizing configuration should not require that many different webhook event categories. I kinda understand why you did it -- I guess it is to use existing API methods. However, I think this complicates things, and there is a simpler way. Alternatively, there can be a single "config changed" event that is fired every time config file changes. The webhook consumer may react to this event, replace the config file on the secondary server, and simply reload AdGuard Home configuration (config reload was implemented in #1302). To make this all easier to implement, we may add "reload" as a separate API method. Please let me know what you think. |
@ameshkov thanks for changing the priority of this issue.
Sure, expanding the scope to include metrics reporting makes sense. I haven't looked at those feature requests, but I can.
I did this mainly because it allowed a very clean mapping between webhook event name and which API endpoint the webhook handler needs to call to fetch and propagate configuration to "followers". Here's a example:
If all the handler gets is
There is no API for fetching a server's entire config, but we could add one. Here's the example I think you're suggesting:
This raises a few questions for me:
I don't believe mixing filesystem operations, signals, and webhooks is a very good practice.
Better to just do the sync purely with REST, I think, since the concepts involved are easier and more accessible to the average user. |
I'll try to come up with a more detailed description of what I suggest tomorrow. Meanwhile, a couple of quick thoughts:
This would address most of your questions save for one:
I suppose this can be handled on the side of the sync micro-service. It's rather easy to exclude specified yaml fields from sync. |
Sure, that seems fine. A bit heavy-handed, perhaps, but definitely less complex for maintainers of both systems. It may perform badly for users with large filter lists, so that might be worth evaluating.
If you do the whole If API consumers use a different language, though, their code is just going to break silently when the schema changes. This is why I really like Openapi-codegen -- the compiler yells at you when the schema changes and breaks your consumer. Anyway, thanks for the consideration and the discussion. I'm happy to help work on this PR, and I'm planning to write the sync handler. I don't care if that lives in my own personal namespace or this organization. Cheers. |
I guess using
Well, this is one of the reasons why we're maintaining the openapi spec. We do that manually, though. |
We don't have a configuration reload now - currently there's no use for it. |
@szolin got it, thx. |
@scyto no, that's not how that works |
I've created the following setup for redundancy:
Pros:
Cons:
|
to add details for others on why it doesn't work like that- one node can lock the state of the db file - which can cause the other node to hang on start at the |
Approach i settled on is to run in a docker swarm with macvlan a single instance. So while this doesn't protect against process failure / process corruption it does protect against container failure, docker host failure, VM failure, VM host failure and physical hardware failure. I am going to continue to investigate propagation of settings between two nodes. FWIW my approach to two adguard nodes using adguard sync is documented here as part of my swarm+glusterfs setup (have swarm will use it, lol - definitely not the simplest way) |
FYI, the reason I ran BGP at home is that in testing I found that a lot of servers didn't failover to the second configured DNS server if the first one is having issues - it just stop resolving altogether. Make sure you actually test the primary failing in different ways before relying on this. |
Is it possible to add my second adguard home ip in my first adguard home's instance dhcp settings? Would be great as my router doesnt allow me to change dns so i have to rely on adguards dhcp. |
@ameshkov "If achieving it with rsync is possible then the restart problem is also easy to solve. But here, i will explain it with a simple alternative solution that i use the most i.e use unison to sync the file and use "system service" to restart on script end #------------------------------------ code starts here -------------------------------------------------------# [Unit] [Service] [Install] #------------------------------------ code ends here -------------------------------------------------------# When using delay in seconds please remember to add delay seconds according to the script execution time. If script sync finishes in 2 seconds use more than 2 seconds else it will end up crashing the system service due to overlapping. Also you can use system service to restart AGH in the same manner by creating dependent system service to restart AGH on sync finish. Edit: Explained in a hard way. Sorry for my bad englpis 😄 |
I ended up using entr to start a sync from Primary server to backup server whenever there is a change to config file. This means any changes to Primary instance is replicated to secondary within few seconds of config file being written to disk without relying on a scheduled cron. entr is available in Debian & Ubuntu repos, so you should be able to install it by running Assumptions
If the assumptions aren't true, feel free to change it to match your setup. systemd service [Unit]
Description=AdGuard Home sync service
After=AdGuardHome.service
[Service]
User=aghome
Group=aghome
StartLimitInterval=5
StartLimitBurst=10
ExecStart=/usr/local/scripts/sync-service.sh
Restart=on-failure
RestartSec=5
[Install]
WantedBy=multi-user.target Shell script systemd calls. Couldn't use pipe in systemd service file so had to use a shell script for it. #!/usr/bin/env sh
/bin/echo /opt/AdGuardHome/AdGuardHome.yaml | /usr/bin/entr -p /usr/local/scripts/agh-sync.sh
#!/usr/bin/env sh
echo "Starting AGH config sync"
ssh aghome@192.168.100.3 'sudo systemctl stop AdGuardHome'
rsync /opt/AdGuardHome/AdGuardHome.yaml aghome@192.168.100.3:/opt/AdGuardHome/AdGuardHome.yaml
ssh aghome@192.168.100.3 'sudo systemctl start AdGuardHome'
echo "Sync Complete" In the backup instance, have the below in aghome ALL=(ALL) NOPASSWD: /bin/systemctl stop AdGuardHome, /bin/systemctl start AdGuardHome Stopping & starting the service does take time and also clears the in memory cache adguardhome holds. So it would be nice at some point for AdGuard Home to support hot reloading config file without restarting the service. |
Hi all, I've been running multiple AGH instances successfully for at least a year. Config sync doesn't seem to be an issue, I'm just storing the The only issue I've had so far is databases as those are not meant to be used by multiple instances, so I have a fully working HA setup but somewhat broken stats. It would be great if that DB issue could be solved somehow but I can live with that. |
+1 to what the person above said. Right now, it is already possible to run multiple instances of AGH and make it HA by putting the instances behind a load balancer. The only thing is that the file-based database is meant to be written to by only one at a time, causing issues when multiple instances are writing to it (when you share the volumes between the two instances). Therefore, the way to add HA to AGH is to simply allow usage of MySQL/Postgres as the database of choice (and not jjst boltDB); as everything else - config files and whatnot - will be effectively "stateless". |
I think Adguard currently uses SQLite? The folks at @superfly made LiteFS which can replicate/distribute SQLite databases. It might be worth taking a look for deployments, however you'd either need to determine a hardcoded primary, setup Consul, or use Fly.io which automatically hosts Consul for you. |
...which is obviously a non-starter. I think it's much easier to just extend whatever SQL layer they're using to support other SQL databases as well, which immediately solves this (and a whole host of other related issues) in one go. |
Absolutely, there definitely should be a better solution. |
I have been running adguard home with adguardhome sync for over a year now, has worked perfectly. It is realtively simple (i still agree a simpler native solution would rock, unfortunately i cannot code so i am of no use at all). I then went one step more complicated and diud it all in docker swarm with gluserfs and macvlans if anyone is interested in that sort of thing https://gist.github.com/scyto/f4624361c4e8c3be2aad9b3f0073c7f9 |
syncing is one thing, but we need stats from both (or more) nodes, so there is no way around mysql or any other sql database |
This comment was marked as duplicate.
This comment was marked as duplicate.
Hey there, |
Jumping in on this. Grafana is good for combining the stats but I think what would be useful is to have the ability to just see those stats from your multiple AG instances in the query log of anyone one instance, you would still have the ability to perform contextual actions on those queries. Right now I have two instances running with the inotify/rsync method on the config file. If I have an item I want to block and I go to my AG query log it's 50/50 as to whether I see it there. I either need to block that item manually or wait for it to be queried by the primary AG, else the config change will just be lost when the primary is updated. |
Hello, I also successfully imported the dashboard. What I'm having trouble with is getting the dashboard to work and it always shows me: |
Hi |
Hi Guys, is it possible to run 2 instances of AdGuard Home on different PC's on the network for redundancy? I Can use the 2 DNS addresses in the router.
The text was updated successfully, but these errors were encountered: