VetIoT: Vet IoT defenses automatically

VetIoT is a research tool to evaluate policy enforcing IoT defenses empirically. VetIoT can generate testcases for policy enforcing IoT defenses, execute those testcases on an IoT platform, and empirically evaluate IoT defenses based on the result of the testcases. We created this tool as part of our paper "VetIoT: On Vetting IoT Defenses Enforcing Policies at Runtime" accepted at IEEE CNS, 2023.

We present a small example here where VetIoT evaluates an IoT defense called ExPAT with 5 randomly generated testcases.

Example: Stress testing ExPAT (an IoT defense)

To conduct stress testing on ExPAT with VetIoT, a user need to configure VetIoT by following the configuration guide provided in the vetiot/test-configs directory. For convenience, configuration used in this example is already provided in the directory vetiot/test-configs/expat/ST/ directory.

Stress testing ExPAT with 5 testcases generated by VetIoT

vagrant@vetiot:~/vetiot$ ./run.sh 
Name of the Defense:expat
Testcase generation type (A for automatic, M for Manual):A
Testcase count (5, 10, 15, 25, 35, 50):5
Name of the configuration:ST

Result of stress testing ExPAT with 5 testcases.

Results from VetIoT shows that on 2 out of 5 testcases, ExPAT successfully detected malicious testcases (testcase 2 and testcase 3). On remaining 3 testcases, ExPAT did not detect any policy violation or indeterminate scenario. As per ExPAT those testcases were not malicious.

{
    "randomExp_2": {
        "policyViolation": 1,
        "indeterminate": 0
    },
    "randomExp_3": {
        "policyViolation": 2,
        "indeterminate": 0
    }
}

Installation/Setup Environment

Users can manually create environment to run VetIoT or they can utilize provided Vagrantfile to automatically create a virtual environment to run VetIoT. Using vagrant would be easier for 2 reasons:

1. It will not modify user's own system environment
2. Users can avoid the manual effort to setup the running environment for VetIoT.

Using Vagrant to setup environment.

1. Download and install VirtualBox from VirtualBox download link and then install Vagrant following vagrant installation instructions.

2. Open a terminal and go to the root directory vetiot-release

3. Create the vagrant virtual machine using

   vagrant up
   

   At this stage, user does not need to do anything. This command will create a virtual machine with Ubuntu 20.04. After that it will prepare the virtual machine automatically following the steps mentioned below.

   Steps to setup environment for VetIoT:

   1. Install Java-11 with Azul Zulu jdk 11.0.19

   2. Download OpenHAB 3.2.0 stable runtime edition from OpenHAB download link and unzip the downloaded file in vetiot/env-setup/ directory

   3. Install Python 3.9 and Python virtual environment using these commands on terminal

      sudo apt update
      sudo apt upgrade
      sudo apt install python3.9 python3.9-venv
      sudo apt install unzip
      

Setup environment without vagrant

If users want to run VetIoT on their own machine (not using vagrant), following steps mentioned in the section Steps to setup environment for VetIoT would be enough.

Configuring VetIoT

VetIoT uses file-based configuration. To configure VetIoT a user can follow the configuration guide provided in the vetiot/test-configs directory.

Running/Testing VetIoT

VetIoT communicates with the IoT platform via rest-api. As a result, an IoT platform must be running in parallel while VetIoT is executing. Instructions to start the IoT platform and run VetIoT is availble in the user guide provided in the vetiot directory.