Filtering requests from vmware #429
Comments
|
uh, interesting @Techtonictools . I've reproduced this problem, but curiously it has worked for the first requests to startpage.com but after a while the second rule (001-vm-deny-all) has started taking precedence...
Yes, it's just another process opening connections. I'll analyze it. |
ok, definetely there's a problem. I've reapplied the rule (by editing it and clicking on Apply) and it has started working as expected. |
I was seeing some weirdness too where a rule would get deleted. I added vmwares rule and a firefox rule was deleted and a new one created automatically. It took a big of managing to get all 6 rules for firefox, pandora and vmware. |
|
Ok, here's one bug (note to myself):
|
|
I've got a fix for this bug. We weren't resetting correctly rule data when opening the rules editor dialog, so if you had edited a rule before, when adding a new one it was detecting it as a rule change, rather than adding a new rule, thus causing to delete the "old rule". If you want to test it, these 2 lines fixes the issue: |
|
I added that change and re-launched the UI. Then deleted the vmware rules and re-created them, but the rule priority issue remains and the request to startpage.com gets denied by rule 001-vmware deny all. |
|
👎 for me, I'll try to reproduce it again |
|
one more change @Techtonictools ? Change this line from "" to "list": (Yours will be empty). And after adding it, just open each vmware related rule and click on apply, that would be enough to reapply the rules, and see if it works. It's working for me now. |
When the Duration of a rule changed (from 1h to 5m, from 5m to until restart, etc), the timer of the old rule was fired, causing deleting the rule from the list. This erroneous behaviour could be one of the reasons of #429
The operator_operand field in the rules view is now 'list' for rule 000-vmware allow after making that change. I re-tried the repro scenario and found some interesting results. The VM can load sites now like www.weather.gov. The other interesting result is something I haven't mentioned before and noticed it now and that is the host/debian machine can't get startpage.com loaded because it gets blocked. So I looked for startpage.com in the exception list and it is there. I don't recall that problem earlier but I've noticed some sites in the exception list get blocked. Using Pithos, there are a bunch of server names to add to the exception list and after adding them all, some sites get denied. It was I initially thought that having a comment at the end of the line in the exception file was causing it |
Let's see if we can find out what's going on here! Be sure that the exception rule has the checkbox
no, that's not supported, the format should be: I spotted (and fixed) a bug yesterday on how temporary rules are managed. In order to avoid weird behaviours, don't change the Duration of a rule if it's temporary. i.e.: don't change from |
|
For me, the denied requests problem where the request is set to be allowed but is denied repros easily by using the attached rules. After exporting and cleaning the rules on my laptop I realized that I had probably mozilla and the vm set to allow so I could use the machine so the rules need to be edited as well as changing the exception list directory. As you know, it is a lot of work to create an exception list so below are pandoras sites. |
|
The HOSTS formatted exception list in the post above, that is for the Pithos app on the apt store in debian. The repro with that may take a while streaming. If you're not into pandora then just surfing, adding sites and then using them would be the alternative repro scenario. The repro steps with Pithos is just to run it and when you see a song in the UI queue skips, that is when to look at the opensnitch event viewer for a request denial. If the server is in the exception list already, that is the repro. But it seems to happen with all apps I tried, VMWare, Pithos and Firefox. To clarify what I meant by a song skip, that is when a song that is shown to be played next in the UI doesn't play. Instead, it is dropped and another will load after that and start playing because a request to the server was denied. An important note is that behavior also occurs when the server isn't in the exception list yet. |
|
Thank you very much for the lists @Techtonictools ! I've realized that you're using the same directory of blocking lists in several rules, to apply it to different apps. It makes a lot of sense, but I didn't even think in that use case... I'll try to reproduce again with your rules. Edit: sorry, lists work per-rule. A rule can have multiple lists (distributed in several files in the same directory), and only non-repeated domains will be loaded for that rule. |
|
sorry @Techtonictools , I've tried to reproduce it with your rules and your list of domains, but it works fine for me using firefox. I'd have liked to test it with Pithos, but Pandora is not available in Europe. Anyways, I think I can mimik the use case with spotify or rhythmbox |
Where in the source code is the logic for determining if a request is going to be granted or denied? |
|
First of all, rules are checked in alphabetical order. There're 2 main points in he code to decide if a rule matches or not:
It receives a connection, and iterates over all the rules to see if any matches.
If there's a match, and the action of the rule is Deny or is a priority rule, it stop iterating and returns the rule that matched, applying the action of that rule. I could compile a daemon with more log traces to debug all the rules operations (listing order, what rule is checked and what no, if the domain is found in the list of domains or not, etc). I can upload it here. |
|
Looking at the repro behavior now the For the rule matching code logic I don't know I'm curious how the request (a binary from firefox wanting to reach out on the network) matches the rule when there aren't checks to find I think the logging of the rule list like you suggest would be helpful (printing the sorted rules before iterating for a match) as well as the other logging suggestions. |
nope, if a connection doesn't match any rules it should ask to allow or deny it. Here you have, the daemon compiled from latest sources with all the fixes. The extra logs are in DEBUG level, so set LogLevel to 0 or configure it from the GUI. It'll print all the rules that are evaluated + the operands tested: |
|
I performed the repro with the instrumented binary and attached the log from during the repro. In the repro spew, Lithos/Pandora requested |
|
Thank you! The domain is checked against the list of domains under But it doesn't seem to find it in the list of that directory. If it'd have been matched, there should have appeared a log like this one: Edit and save the file with the domains and see if the list is reloaded after save the file, you should see logs similar to these ones: Verify that the file is parsed and loaded and that the number of domains is not 0. |
|
How do I verify the file is parsed and loaded and how do I verify that the domains loaded is 1? The file was re-saved and there was no change. There are 28 lines in the exception file and there is nothing in the log or settings that I found that shows the exception file is loaded. |
|
When the daemon starts, there should be logs like these ones: If you don't see "monitor lists started..." then something is not working as it should. In order to verify that the lists of domains are being loaded and reloaded, try the following: create the directory: just after creating the rule, there should appear in the log something similar to this: try adding a new domain: If this works, edit your rule |
|
I don't have that loading rules nor loading domains spew. I did a |
|
That's really strange. You are using daemon v1.4.0rc2, aren't you? Try creating the dummy rule with the directory and 1 list + 1 domain, and set the log level to DEBUG (although the logs should appear with INFO) |
|
I'm using the binary you attached to this log the other day and DEBUG is on (the second line in the log reflects DBG). I'm not sure what you mean with the dummy rule. I don't understand the directions. What directory and what is 1 list + 1 domain? |
|
Sorry, I mean, in order to verify that the lists are working, create a simple rule with these options: create the directory After saving the rule, you should see in the logs these messages: (Setting the log level to INFO will ease to see only those messages). If you don't see those messages, then for some reason the lists of domains are not working. If you see those messages, open the file |
|
I revisited this and put my old exception list file back in the list directory ( that was previously not working), rebooted for the service to start correctly. Now it is fine now and pandora is working, including tuner.pandora.com which is in both exception list files. |
|
thank you very much for the feedback @Techtonictools !! let me know if you find anything else. I'll publish a new release soon with all the fixes, knowing now that it's working as expected. |
|
The problem was reproducing again. I tried to attach gdb to opensnitch and set a breakpoint on domainsListCmp so that I can see why the domain in the list isn't found, but the function wasn't found. Since golang doesn't seem to be fully supported in gdb I'm going to give delve another try after I upgrade the go version in debian so I can build it. |
|
I wanted to step through the daemon during a repro and see why domainsListCmp doesn't find the domain I want to allow, but doing so is not straightforward and requires a bigger time investment than I can spend. It isn't as simple as attaching a debugger and walking through it for me. For anyone that has this scenario working (block all requests except for domains in a list), what distro are you using? |
|
In my case is Debian Sid, and I've also tested it on Kubuntu 20.04 and Debian Buster (livecd iso). I'd try to reproduce it with just 2 rules. |
|
hey @Techtonictools , there're new packages with fixes in the lists of domains, could you install them and see if these problems are solved? |
|
I gave the new release a try. The upgrade process was seamless. The unexpected block problem is still happening though. In one case, api0.weather.com was blocked when requested by firefox on the host. The other fail case was in a VM (virtual machine manager running Windows), wunderground.com was blocked. That too is in the exception list and it had just loaded on the host in firefox. The repro scenario is to load that wunderground.com page, enter a location, click on the WUNDERMAP tab and enable radar. Doing that will invoke the api*.weather.com requests. Attached is the log file, the rules and pictures of the specific exception and the correlating denial. Let me know if you need more information. 127.0.0.1 mapbox.com |
|
As an update, I reproduced this problem where domains are blocked that are in the exception list on a new machine. I put debian on my main PC, replacing windows, and imported the opensnitch rules after installing 1.4.0.rc4-1. api[x].weather.com is getting blocked following the most recent repro scenario listed above with weather underground. I'm not sure if the following scenario would impact setting up the repro after having a different configuration for anyone else, but one thing noteworthy doing the repro on a new machine was that I initially forgot to edit the file path of the exception list in the .json rule files in /etc/opensnitchd/rules after copying them from my laptop. Once I found that issue, I did a string replace to have the proper path in the json rule and the UI was still reflecting the old path. I stopped and restarted, closed and reopened the UI and still the old path was there in the rules in the UI but the proper path was in the json files. After a few more stop/start and close/open the correct path was present in the UI. As an aside, previously (months ago?) Pandora (pithos) was getting some hosts blocked. But for the past 3 weeks I've been streaming pithos all day and it hasn't blocked any sites in the exception list for that app. However, other sites in the list do get blocked when coming from other apps. Let me know if there is anything I can do. |
|
I root caused one issue that is causing the unexpected blocks and found one whitespace char was at the end of the string and before the newline with some of the domains. That makes sense because when looking at the opensnitch logs and the code, the domain being passed to I'm not sure if that was the cause of all the blocks yet, but for sure that was the cause for some of them. I'm going to install virt-manager and get VM's making requests in parallel with other apps that are also making requests out to the internet to see if the unexpected block problem reproduces in another way. |
|
thank you @Techtonictools ! that's very interesting, I'll try to reproduce it ASAP. |
|
ok, confirmed, if the entries of a blocklist have spaces (or other control characters) the rule doesn't block/allow connections. |
|
That resolved it, thanks. |
Remove \r\n\t\s from the end of each line of a blocklist. If the entries of a list had these characters caused to not match connections and not apply the rule. closes #429
I'm looking to use opensnitch to block all requests from vmware and only allow requests in an exception list.
After creating one rule to block the vmware nat program and then another rule to hook the exception list for the vmware nat binary, I'm finding all requests are getting denied. I'm hoping to get requests to domains in the exception list to succeed and all others die. Is this a supported scenario with VM software (vmware or virtualbox)?
Post error logs:
The opensnitch events tab shows an entry of the request to the host I'm trying to access (startpage.com) from the VM and the request is blocked because of rule 001-vmware deny all
Versions :
Additional context
The same exception list is used in opensnitch rules for multiple binaries, firefox and pandora (Pithos/python3.7), to accomplish the same task of only allowing hosts I want and blocking all others. Those two scenarios are working but I'm not able to get vmware nat stuff to work using the same approach.
The text was updated successfully, but these errors were encountered: