AI-Based Operating System Evaluation

This project automates the process of gathering information from managed nodes, evaluating best practices using language models (LLMs), and generating comprehensive reports. The system is designed to facilitate compliance checks and recommend improvements by leveraging advanced AI workflows. The project is structured in two main phases: data gathering and AI-driven evaluation. It utilizes Ansible for automation, ensuring efficient and scalable operations across diverse environments. Key features include dynamic configuration, seamless integration with various LLMs, and humanreadable final reports.

The code is in the directory ./ansible

Supported models

For the online LLMs, we included the following two models:

For offline use, we support Llama 3.1 too running on Ollama.

Final evaluation report

The final report looks like this:

# Report for node01
## Category: security
### ssh config: no
  - Issue: Port 22 is not explicitly set, it's best to set this explicitly.
  - Fix: Uncomment and set the Port to 22.

  - Issue: PermitRootLogin is not set, it's recommended to set this to no to prevent root logins.
  - Fix: Set PermitRootLogin to no.

  - Issue: PasswordAuthentication is set to no, it's recommended to set this to yes for a more secure configuration.
  - Fix: Set PasswordAuthentication to yes.
### ssh key permissions: yes
...

Documentation for initial setup and how to use

• Ansible
  • Setup
    • Add correct user to ansible.cfg
    • Add clouds.yml file
    • Setup python virtual environment and required python packages
    • LLM config
    • How to install Ollama
  • How to run
    • In the ansible/ directory of this repository
      • Run playbook with default LLM (=gpt)
      • Run playbook with selected LLM (take a look at the LLM configuration file)
      • Turn of information processing from the managed nodes to only run AI Requests
      • Turn of AI Requests and only process the information on the managed nodes
  • How to use in other repos
    • Add to another file in this repo
    • As a role for another file/ project

Documentation for evaluation-tool role:

• evaluation-tool
  • Directory structure
  • Get information from managed nodes (1)
    • Flowchart
    • Files and Structure
    • How It Works
    • How to make changes
    • Debugging
  • AI Module (2)
    • Flowchart
    • Files and Structure
    • How It Works
    • How to make changes
    • Debugging
  • Changing output directory structure, node names or report file names (3)