Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

dnsmasq should have requests from the vm filtered, but all requests are allowed and they aren't shown in the UI #482

Closed
Techtonictools opened this issue Aug 27, 2021 · 20 comments

Comments

@Techtonictools
Copy link

I'm using virt-manager on a debian host to run Win10 and OpenSnitch to filter network requests that origninate from the vm. I had OpenSnitch setup so all requests from the vm (and host) are denied unless the host is in an exception list associated with the binary making the request. It was working yesterday and a few days before that, but today after resuming from S4 from yesterdays work, all requests from the vm were granted and none of the requests from the vm were being shown in the UI.

There was a previous instance where the same behavior occurred and I rebuilt the rules, did stop/starts and it started working. But today I removed the dnsmasq rules, invoked a request and did not get a prompt from OpenSnitch to add a rule. There weren't any rules for dnsmasq in the UI or in a .json file. Stop/start and close/open and then re-invoking a page load/request in the vm didn't work around the issue to get the pop up to create a new rule and there were no related events in the event tab.

To Reproduce
Install debian bullseye
Install virt-manager
Install OpenSnitch on host
Create rules pictured below and replace the vm client ip in 001-dnsmasq allow with the IP you should get for your vm
Create an exception list and put a domain name in it and save it to the exception list path
Create a vm and install win10 in it
After Win10 install completes, invoke a request in the vm that is not in the exception list. Windows will be trying to make requests by default, but to be sure launch IE and let it try to go to the pre-loaded default sites on launch (like install edge, msn, show news, etc)

Post error logs:
There is no record of any request from the vm in the log with the loglevel set to debug when in repro state.

Expected behavior
A record of the rule that was used for the request from the vm in the event tab in the UI with the status allow or deny.
A blocked request.

OS:

Additional context
If the steps to reproduce yield a successful block and the vm requests are showing in the events tab, try systemctl hibernate, power on and re-test a request from the vm.
It isn't a 100% repro and I'm not sure of the exact steps to reproduce the problem on demand. What I'm doing to reproduce the issue is to use the vm and host machine daily through the work week and at EOD, hibernate. Sometimes hibernate fails and a clean boot is performed instead but today resume from hibernate succeeded and that is when the problem reproduced.

000-dnsmasq allow
001-dnsmaq allow client IP
999-dnsmasq deny

@gustavo-iniguez-goya
Copy link
Collaborator

I removed the dnsmasq rules, invoked a request and did not get a prompt from OpenSnitch to add a rule

mmh, are the firewall rules added after resume? iptables -t mangle -L OUTPUT (or nft list ruleset).

Post error logs:
There is no record of any request from the vm in the log with the loglevel set to debug when in repro state.

What's the last line written to the log?

Also, just for testing purposes, rename the name of the rules so they don't have spaces not punctuations.

@gustavo-iniguez-goya
Copy link
Collaborator

Also take a look at the journal log to see if there's any error related to us: journalctl -ar

@Techtonictools
Copy link
Author

I didn't see any obviously related entries in the journal log. The issue is reproducing constantly even after reboots so it seems like I have it configured wrong. But firefox and qbittorrent rules are running as expected. It seems that right now it is just the dnsmasq rules that aren't working as expected.

This is the iptables output:

anon@dev:~$ sudo iptables -t mangle -L OUTPUT
[sudo] password for anon: 
Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
NFQUEUE    all  --  anywhere             anywhere             ctstate NEW,RELATED NFQUEUE num 0 bypass
opensnitch-filter-OUTPUT  all  --  anywhere             anywhere            
anon@dev:~$ 

The last line of the opensnitchd log:

[2021-08-27 21:23:02]  DBG  new connection udp6 => 9836:2600:100e:b12e:654b:8616:f9ff:fe1b:fd84 -> 2600:1700:4ca0:7000::33:1130 uid: 1000
[2021-08-27 21:23:02]  DBG  ✔ /usr/bin/qbittorrent -> 2600:1700:4ca0:7000::33:1130 (qbittorrent-allow)

Attached are all the rules loaded in case I have a mistake in there somewhere.
rules.gz

@gustavo-iniguez-goya
Copy link
Collaborator

gustavo-iniguez-goya commented Aug 27, 2021

ok, everything seems to be fine as far as I can tell. I'll try to reproduce it but it seems to be a bit tricky, I'll need your help.

As long as the other rules are working and you get prompted to allow other binaries (telnet 1.1.1.1, curl https://...) everything will be fine. Otherwise, if you don't get prompted to allow/deny connections for any command then it could be a deadlock and we'd be fckd.

The next time you reproduce it, please, do the following:

  • If all requests from the vm are granted, see if the iptables/nftables rules are added to the mangle table.

If they are, delete the rule if it exists, add it again and see if it starts working:

# iptables -t mangle -F OUTPUT
# iptables -t mangle -I OUTPUT -m conntrack --ctstate NEW,RELATED -j NFQUEUE --queue-num 0 --queue-bypass

Note: If you flush the OUTPUT table, after a few seconds the daemon should add the rules again.

  • monitor the log opensnitchd.log in DEBUG level. If there's no activity at all and you can connect to the internet, then the firewall rules are not taken place. If you cannot connect to the internet, not to domains nor to IPs, then it could be a deadlock.
  • change daemon options from the GUI (procs monitor method, log level , etc), and see if they're applied and logs are written to the file.

By the way, what process monitor method are you using? proc, ebpf.. ?

@Techtonictools
Copy link
Author

My laptop running opensnitch and virt-manager is working as expected so I put those rules on my main pc and the problem still reproduced. OpenSnitch did invoke the rule dialog when a program without a rule wanted access.

Below is what the mangle table looks like while in repro state. I also ran the flush command and then what I think is the add command, but my version of iptables doesn't have the --state input parameter.

root@dev:/etc/opensnitchd/rules# sudo iptables --table mangle --list --line-numbers
Chain PREROUTING (policy ACCEPT)
num  target     prot opt source               destination         

Chain INPUT (policy ACCEPT)
num  target     prot opt source               destination         

Chain FORWARD (policy ACCEPT)
num  target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
num  target     prot opt source               destination         
1    NFQUEUE    all  --  anywhere             anywhere             ctstate NEW,RELATED NFQUEUE num 0 bypass
2    opensnitch-filter-OUTPUT  all  --  anywhere             anywhere            

Chain POSTROUTING (policy ACCEPT)
num  target     prot opt source               destination         
1    LIBVIRT_PRT  all  --  anywhere             anywhere            

Chain LIBVIRT_PRT (1 references)
num  target     prot opt source               destination         
1    CHECKSUM   udp  --  anywhere             anywhere             udp dpt:bootpc CHECKSUM fill

Chain opensnitch-filter-OUTPUT (1 references)
num  target     prot opt source               destination         
1    ACCEPT     icmp --  anywhere             anywhere            
root@dev:/etc/opensnitchd/rules# sudo iptables -t mangle -F OUTPUT
root@dev:/etc/opensnitchd/rules# sudo iptables -t mangle -I OUTPUT -m conntrack --state NEW,RELATED -j NFQUEUE --queue-num 0 --queue-bypass
iptables v1.8.7 (nf_tables): unknown option "--state"
Try `iptables -h' or 'iptables --help' for more information.
root@dev:/etc/opensnitchd/rules#

The -F parameter to iptables says it deletes the rules one by one, but I'm still seeing the same rules.(Flush the selected chain (all the chains in the table if none is given). This is equivalent to deleting all the rules one by one.)

root@dev:/etc/opensnitchd/rules# sudo iptables --table mangle --list --line-numbers
Chain PREROUTING (policy ACCEPT)
num  target     prot opt source               destination         

Chain INPUT (policy ACCEPT)
num  target     prot opt source               destination         

Chain FORWARD (policy ACCEPT)
num  target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
num  target     prot opt source               destination         
1    NFQUEUE    all  --  anywhere             anywhere             ctstate NEW,RELATED NFQUEUE num 0 bypass
2    opensnitch-filter-OUTPUT  all  --  anywhere             anywhere            

Chain POSTROUTING (policy ACCEPT)
num  target     prot opt source               destination         
1    LIBVIRT_PRT  all  --  anywhere             anywhere            

Chain LIBVIRT_PRT (1 references)
num  target     prot opt source               destination         
1    CHECKSUM   udp  --  anywhere             anywhere             udp dpt:bootpc CHECKSUM fill

Chain opensnitch-filter-OUTPUT (1 references)
num  target     prot opt source               destination         
1    ACCEPT     icmp --  anywhere             anywhere            
root@dev:/etc/opensnitchd/rules# 

How should I delete and re-add the rule? I'm pretty new to networking stuff at this level and I haven't been using ebpf or any related tool, although at some point I'll be learning about using one.

@gustavo-iniguez-goya
Copy link
Collaborator

gustavo-iniguez-goya commented Aug 28, 2021

my version of iptables doesn't have the --state input parameter.

oopsie, sorry for that, it's --ctstate instead of --state

The -F parameter to iptables says it deletes the rules one by one, but I'm still seeing the same rules.

That's probably because the daemon re-added them. There should be a log entry like this one: IMP firewall rules changed, reloading

@Techtonictools
Copy link
Author

In OpenSnitch settings, ebpf is set for process monitor method.

I ran the iptables commands to refresh the mangle table, re-ran the scenario to invoke a request to a blocked site on the vm and there was no change in behavior. I looked in the system log and and didn't see any entries related to opensnitch. The opensnitch log didn't have any record of the vm hitting the site. Then I ran through the following steps:

Uninstalled opensnitch

anon@dev:/var/log$ sudo apt remove opensnitch python3-opensnitch-ui
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following packages were automatically installed and are no longer required:
  libabsl20200923 libnetfilter-queue1 python3-grpcio python3-protobuf python3-pyinotify python3-pyqt5.qtsql python3-slugify python3-unidecode
Use 'sudo apt autoremove' to remove them.
The following packages will be REMOVED:
  opensnitch python3-opensnitch-ui
0 upgraded, 0 newly installed, 2 to remove and 9 not upgraded.
After this operation, 27.5 MB disk space will be freed.
Do you want to continue? [Y/n] y
(Reading database ... 224230 files and directories currently installed.)
Removing opensnitch (1.4.0~rc4-1) ...
Synchronizing state of opensnitch.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install disable opensnitch
Removed /etc/systemd/system/multi-user.target.wants/opensnitch.service.
Removing python3-opensnitch-ui (1.4.0~rc4-1) ...
Processing triggers for hicolor-icon-theme (0.17-2) ...
Processing triggers for gnome-menus (3.36.0-1) ...
Processing triggers for mailcap (3.69) ...
Processing triggers for desktop-file-utils (0.26-1) ...
anon@dev:/var/log$ sudo apt remove --purge opensnitch python3-opensnitch-ui
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following packages were automatically installed and are no longer required:
  libabsl20200923 libnetfilter-queue1 python3-grpcio python3-protobuf python3-pyinotify python3-pyqt5.qtsql python3-slugify python3-unidecode
Use 'sudo apt autoremove' to remove them.
The following packages will be REMOVED:
  opensnitch* python3-opensnitch-ui*
0 upgraded, 0 newly installed, 2 to remove and 9 not upgraded.
After this operation, 0 B of additional disk space will be used.
Do you want to continue? [Y/n] y
(Reading database ... 223111 files and directories currently installed.)
Purging configuration files for opensnitch (1.4.0~rc4-1) ...
dpkg: warning: while removing opensnitch, directory '/usr/lib/systemd/system' not empty so not removed
dpkg: warning: while removing opensnitch, directory '/etc/opensnitchd' not empty so not removed
Purging configuration files for python3-opensnitch-ui (1.4.0~rc4-1) ...
anon@dev:/var/log$ 

The mangle table after install

anon@dev:/var/log$ sudo sudo iptables -t mangle -L OUTPUT
Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
anon@dev:/var/log$ 

I deleted the opensnitch log and then executed opensnitch install;

anon@dev:~/Downloads$ sudo dpkg -i opensnitch_1.4.0.rc4-1_amd64.deb python3-opensnitch-ui_1.4.0.rc4-1_all.deb; sudo apt -f install
Selecting previously unselected package opensnitch.
(Reading database ... 223104 files and directories currently installed.)
Preparing to unpack opensnitch_1.4.0.rc4-1_amd64.deb ...
Unpacking opensnitch (1.4.0~rc4-1) ...
Selecting previously unselected package python3-opensnitch-ui.
Preparing to unpack python3-opensnitch-ui_1.4.0.rc4-1_all.deb ...
Unpacking python3-opensnitch-ui (1.4.0~rc4-1) ...
Setting up opensnitch (1.4.0~rc4-1) ...
Synchronizing state of opensnitch.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable opensnitch
Created symlink /etc/systemd/system/multi-user.target.wants/opensnitch.service → /lib/systemd/system/opensnitch.service.
Setting up python3-opensnitch-ui (1.4.0~rc4-1) ...
gtk-update-icon-cache: Cache file created successfully.
Processing triggers for gnome-menus (3.36.0-1) ...
Processing triggers for desktop-file-utils (0.26-1) ...
Processing triggers for mailcap (3.69) ...
Processing triggers for hicolor-icon-theme (0.17-2) ...
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
0 upgraded, 0 newly installed, 0 to remove and 9 not upgraded.
anon@dev:~/Downloads$ 

Mangle table check

anon@dev:~/Downloads$ sudo sudo iptables -t mangle -L OUTPUT
Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
NFQUEUE    all  --  anywhere             anywhere             ctstate NEW,RELATED NFQUEUE num 0 bypass
opensnitch-filter-OUTPUT  all  --  anywhere             anywhere            
anon@dev:~/Downloads$ 

I set opensnitch log level to debug in the ui and Invoked a page load in the vm (that had been running through the uninstall/install) to msn.com and it loaded as did yahoo.com, neither are in the exception list. Below is the opensnitch log from performing those steps.

anon@dev:/var/log$ cat opensnitchd.log 
[2021-08-28 16:41:53]  IMP  Start writing logs to /var/log/opensnitchd.log
[2021-08-28 16:44:08]  IMP  Start writing logs to /var/log/opensnitchd.log
[2021-08-28 16:44:08]  IMP  Start writing logs to /var/log/opensnitchd.log
[2021-08-28 16:44:46]  DBG  new connection udp6 => 49794:::1 -> ::1:49794 uid: 119
[2021-08-28 16:44:46]  DBG  ✔ /usr/lib/postgresql/13/bin/postgres -> ::1:49794 (allow-always-simple-usr-lib-postgresql-13-bin-postgres)
[2021-08-28 16:45:22]  DBG  new connection udp6 => 53083:2600:100e:b12a:dd56:47d:177d:708d:5b41 -> 2001:738:0:860:1::2:123 uid: 101
[2021-08-28 16:45:22]  DBG  ✔ /usr/lib/systemd/systemd-timesyncd -> 2001:738:0:860:1::2:123 (allow-always-simple-usrlibsystemdsystemd-timesyncd)
[2021-08-28 16:45:46]  DBG  new connection udp6 => 49794:::1 -> ::1:49794 uid: 119
[2021-08-28 16:45:46]  DBG  ✔ /usr/lib/postgresql/13/bin/postgres -> ::1:49794 (allow-always-simple-usr-lib-postgresql-13-bin-postgres)
anon@dev:/var/log$ 

@gustavo-iniguez-goya
Copy link
Collaborator

I've replicated your setup and so far is working as expected. I'll keep an eye on it.

@gustavo-iniguez-goya
Copy link
Collaborator

@Techtonictools the rules.gz seems to be empty, could you reupload it or post the file with the exception list?

@Techtonictools
Copy link
Author

rules.zip Let me know if that doesn't go through. I had checked the rules.gz prior to uploading by extracting it and checking for the rules in the folder. I'm not sure what went wrong there.

@gustavo-iniguez-goya
Copy link
Collaborator

gustavo-iniguez-goya commented Aug 29, 2021

thank you @Techtonictools , now it's correct.

[edited] nothing to see here, it was a typo...

@gustavo-iniguez-goya
Copy link
Collaborator

The rule 000-dnsmasq-allow.json is filtering domains based on the process' command line, i.e.: it's checking connections against /proc//cmdline instead of /proc/<PID>/exe . Change in that rule process.command by process.path and test it again please.

@Techtonictools
Copy link
Author

That is definately a mistake on my part. I changed the rule in the UI to use the From this executable and put the dnsmaq path there and retried. I must be doing something else wrong since the behavior hasn't changed. I would think with the error to use the command line instead of the exe path would result in a denial. But I'm not seeing any dnsmasq entries in the Events tab. I'll try to narrow down the problem a bit more.

@gustavo-iniguez-goya
Copy link
Collaborator

gustavo-iniguez-goya commented Aug 29, 2021

That's really weird. Do you see connections in DEBUG log level in /var/log/opensnitchd.log? like new connection tcp => 45622:127.0.0.1 -> 127.0.0.1:9054 uid: 1000

You should see them regardless of the rules and the proc monitor method being used. If there're no entries of "new connection..." then the mangle rule is not added or it's being bypassed.

While you're reproducing this issue:

  • verify that the mangle rule is added, and that it's the first in the list.
  • add a rule to the FORWARD chain and see if the connections start appearing: iptables -I FORWARD -m conntrack --ctstate NEW,RELATED -j NFQUEUE --queue-num 0 --queue-bypass

I've realized that after coming back from hibernation our DNS rule is not in the first place of the filter table, and instead is one of the libvirt. Also check it, and if it's not in the first place, delete it and re-add it:

  • iptables -L INPUT
  • iptables -D INPUT 2 (if it's in the second place)
  • iptables -I INPUT -p udp --sport 53 -j NFQUEUE --queue-num 0 --queue-bypass

I'm using your ruleset now, but it's working fine for now.

What's the NIC configuration of the VM? NAT and virtio?

@Techtonictools
Copy link
Author

Techtonictools commented Aug 30, 2021

Yeah, it is weird as if I broke something. I need to come back to it a few days, but I'll try those things and post back then. The first scenario I have tried the other day and posted in this issue log. It is the post that includes the uninstall/install steps (but not limited to) and the terminal and og outputs.

@gustavo-iniguez-goya
Copy link
Collaborator

I don't know if you mentioned it already, but could you reproduce this problem using "proc" monitor method? I want to discard that it's not due to the ebpf module.

Sorry, that comment on the issue 481 was supposed to be for this one. I reproduced a similar behaviour where the daemon starts to behave in a wrong manner, it can be firewall rules monitor stop working for no reason, rules list getting corrupted, etc...

In OpenSnitch settings, ebpf is set for process monitor method.

Change it please to proc and see if you reproduce it.

@Techtonictools
Copy link
Author

I changed the method to proc today and there wasn't any change. Outside of that, I ran another repro test starting from a different point and removed virt-manager to unload and remove all related bits like the two dnsmasq instances that are running. In addition, I wanted to block all internet requests from the vm to the internet completely so after booting into the vm, no internet requests would succeed. To do that in opensnitch I set each dnsmasq related rule to deny so that the vm wouldn't be allowed to make internet requests.

I reinstalled virt-manager and after booting the vm, I tried to load a website and it didn't load. Looking in the Events tab of the UI there were two events logged (illustrated below).
Then I went back to the vm and tried to load the blocked site again and it loaded. Any internet request is granted from that point and there aren't any Events in the UI related to any of the requests that succeeded.

opensnitch_events

Rules:

{
  "created": "2021-09-03T10:11:05.034941849-07:00",
  "updated": "2021-09-03T10:11:05.035173259-07:00",
  "name": "999-dnsmasq-deny",
  "enabled": true,
  "precedence": false,
  "action": "deny",
  "duration": "always",
  "operator": {
    "type": "simple",
    "operand": "process.path",
    "sensitive": false,
    "data": "/usr/sbin/dnsmasq",
    "list": []
  }
}


{
  "created": "2021-09-08T09:41:21.280133651-07:00",
  "updated": "2021-09-08T09:41:21.280557172-07:00",
  "name": "001-dnsmasq-allow-192.168.122.11",
  "enabled": true,
  "precedence": true,
  "action": "deny",
  "duration": "always",
  "operator": {
    "type": "list",
    "operand": "list",
    "sensitive": false,
    "data": "[{\"type\": \"simple\", \"operand\": \"process.path\", \"data\": \"/usr/sbin/dnsmasq\", \"sensitive\": false}, {\"type\": \"simple\", \"operand\": \"dest.ip\", \"data\": \"192.168.122.11\", \"sensitive\": false}]",
    "list": [
      {
	"type": "simple",
	"operand": "process.path",
	"sensitive": false,
	"data": "/usr/sbin/dnsmasq",
	"list": null
      },
      {
	"type": "simple",
	"operand": "dest.ip",
	"sensitive": false,
	"data": "192.168.122.11",
	"list": null
      }
    ]
  }
}


{
  "created": "2021-08-29T16:26:42.033454793-06:00",
  "updated": "2021-08-29T16:26:45.395065435-06:00",
  "name": "000-dnsmasq-allow",
  "enabled": true,
  "precedence": true,
  "action": "allow",
  "duration": "always",
  "operator": {
    "type": "list",
    "operand": "list",
    "sensitive": false,
    "data": "[{\"type\": \"simple\", \"operand\": \"process.path\", \"data\": \"/usr/sbin/dnsmasq\", \"sensitive\": false}, {\"type\": \"lists\", \"operand\": \"lists.domains\", \"data\": \"/home/anon/opensnitch\", \"sensitive\": false}]",
    "list": [
      {
	"type": "simple",
	"operand": "process.path",
	"sensitive": false,
	"data": "/usr/sbin/dnsmasq",
	"list": null
      },
      {
	"type": "lists",
	"operand": "lists.domains",
	"sensitive": false,
	"data": "/home/anon/opensnitch",
	"list": null
      }
    ]
  }
}

@gustavo-iniguez-goya
Copy link
Collaborator

Thank you @Techtonictools . Could you upload the file with the exceptions?

@Techtonictools
Copy link
Author

testdomains.txt

I just reproduced it with a clean exception list attached. Only debian.org should load but yahoo.com loaded using that same list.

@gustavo-iniguez-goya
Copy link
Collaborator

hi! a lot has changed since v1.4.0. Please, could you test it with latest v1.6.0rc5 version https://github.com/evilsocket/opensnitch/releases/tag/v1.6.0-rc.5 ?
If this issue is not updated, it will be closed in ~2 weeks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants