Add readthedocs configurations

.readthedocs.yaml

@@ -0,0 +1,32 @@
+# .readthedocs.yaml
+# Read the Docs configuration file
+# See https://docs.readthedocs.io/en/stable/config-file/v2.html for details
+
+# Required
+version: 2
+
+# Set the OS, Python version and other tools you might need
+build:
+  os: ubuntu-22.04
+  tools:
+    python: "3.11"
+    # You can also specify other tool versions:
+    # nodejs: "19"
+    # rust: "1.64"
+    # golang: "1.19"
+
+# Build documentation in the "docs/" directory with Sphinx
+sphinx:
+   configuration: docs/conf.py
+
+# Optionally build your docs in additional formats such as PDF and ePub
+formats:
+   - pdf
+#    - epub
+
+# Optional but recommended, declare the Python requirements required
+# to build your documentation
+# See https://docs.readthedocs.io/en/stable/guides/reproducible-builds.html
+python:
+   install:
+   - requirements: docs/requirements.txt

docs/OVERVIEW.md

@@ -1,26 +1,7 @@
-# STIX-SHIFTER
-
-**Table of Contents**
-
-- [Introduction](#introduction)
-  - [What is STIX?](#what-is-stix)
-  - [What is STIX-SHIFTER?](#what-is-stix-shifter)
-  - [What is STIX Patterning? What are STIX Observations?](#what-is-stix-patterning-what-are-stix-observations)
-  - [This sounds like Sigma, I already have that](#this-sounds-like-sigma-i-already-have-that)
-  - [What is a STIX-SHIFTER connector?](#what-is-a-stix-shifter-connector)
-  - [Why would I want to use this?](#why-would-i-want-to-use-this)
-- [Available Connectors](CONNECTORS.md)
-- [How to use](#How-to-use)
-  - [Translate](#translate)
-  - [Transmit](#transmit)
-  - [Execute](#execute)
-  - [Debug](#Debug)
-- [Limitations](#limitations)
-- [Glossary](#glossary)
-- [Architecture Context](#architecture-context)
-- [Contributing](#contributing)
-- [Guide for creating new connectors](adapter-guide/develop-stix-adapter.md)
-- [Licensing](#licensing)
+
+```{contents} Table of Contents
+:depth: 3
+```
 
 ## Introduction
 

docs/README.md

@@ -1,20 +1,19 @@
 [![example workflow](https://github.com/opencybersecurityalliance/stix-shifter/actions/workflows/main.yml/badge.svg)](https://github.com/opencybersecurityalliance/stix-shifter/actions)
 [![codecov](https://codecov.io/gh/opencybersecurityalliance/stix-shifter/branch/develop/graph/badge.svg?token=gQvl14peRj)](https://codecov.io/gh/opencybersecurityalliance/stix-shifter)
 
-# Introduction to STIX-Shifter
+# Introduction
 
 STIX-shifter is an open source python library allowing software to connect to products that house data repositories by using STIX Patterning, and return results as STIX Observations.
 
 This library takes in STIX 2 Patterns as input, and "finds" data that matches the patterns inside various products that house repositories of cybersecurity data. Examples of such products include SIEM systems, endpoint management systems, threat intelligence platforms, orchestration platforms, network control points, data lakes, and more.
 
 In addition to "finding" the data by using these patterns, STIX-Shifter also _transforms the output_ into STIX 2 Observations. Why would we do that you ask? To put it simply - so that all of the security data, regardless of the source, mostly looks and behaves the same.
 
-## Project Overview and CLI Commands
+***Project Documenation***
 
-For general information about STIX, this project, and the command line utilities, see the [STIX-shifter Overview](OVERVIEW.md)
+For general information about STIX, this project, and the command line utilities, see the [STIX-shifter Documenation](https://stix-shifter.readthedocs.io/)
 
-
-# Installation
+## Installation
 
 The recommended method for installing stix-shifter is via pip. Two prerequisite packages needs to be installed inlcuding the package of stix-shifter connector module to complete a stix-shifter connector installation. Run the below commands to install all the packages:
 
@@ -25,15 +24,15 @@ The recommended method for installing stix-shifter is via pip. Two prerequisite
 3. Desired stix-shifter connector module package:  `pip install stix-shifter-modules-<module name> `
    Example:  `pip install stix-shifter-modules-qradar`
 
-## Dependencies
+### Dependencies
 
 STIX-shifter requries Python 3.8 or greater. See the [requirements file](../stix_shifter/requirements.txt) for library dependencies. 
 
-# Usage
+## Usage
 
 STIX-Shifter can use used the following ways:
 
-## 1. As a command line utility
+### As a command line utility
 
 The STIX-Shifter comes with a bundled script which you can use to translate STIX Pattern to a native datasource query. It can also be used to translate a JSON data source query result to a STIX bundle of observable objects. You can also send query to a datasource by using a transmission option. 
 
@@ -54,7 +53,7 @@ In order to build `stix-shifter` packages from source follow the below prerequis
    3. Run setup: `python3 setup.py install`
 
 
-## 2. Running from the source
+### Running from the source
 
 You may also use the `python3 main.py` script. All the options are the same as the command line utility described above.
 
@@ -72,7 +71,7 @@ In order to run `python3 main.py` from the source follow the below prerequisite
 
 **Note:** setup.py only installs dependencies when INSTALL_REQUIREMENTS_ONLY=1 directive is used. This option is similar to `python3 generate_requirements.py && pip install -r requirements.txt`
 
-## 3. As a library
+### As a library
 
 You can also use this library to integrate STIX Shifter into your own tools. You can translate a STIX Pattern:
 

docs/adapter-guide/best_practices.md

@@ -1,4 +1,4 @@
-# Best practices when developing a new connector
+## Best practices
 
 An assessment of the data source APIs should be made before beginning implementation of a connector. The APIs should return a good coverage of [cyber observable](https://docs.oasis-open.org/cti/stix/v2.0/stix-v2.0-part4-cyber-observable-objects.html) data that fits within the standard STIX `observed-data` objects. If most of the data returned is getting mapped to custom STIX objects or properties, it may be an indication that the data source is not a good fit for a connector. The APIs should also allow for robust filtering of the data, or support a query language; this is essential for executing federated searches against multiple connectors using STIX patterning.
 

docs/adapter-guide/custom_mappings.md

@@ -1,4 +1,4 @@
-# Use of custom mappings
+## Use of custom mappings
 
 Follow the below steps, if a user or threat hunter wants to use custom mapping after installing stix-shifter libraries:
 

docs/adapter-guide/develop-configuration-json.md

@@ -1,20 +1,20 @@
-# Configuration Parameters
+## Configuration Parameters
 
 A json file needs to be created that contains configuration parameters for each module. The configuration json file is required in order to validate the module specific parameters for a successful translation and transmission call. Please follow this naming convention when you create the file: `config.json`
 
 A second json file is required to translate the parameters defined in `lang_en.json` for the UI. This file is necessary in order to help the UI framework show the parameters in human readable format.
 
-## File Location
+### File Location
 
 Create a directory named `configuration` in your module folder. The json files mentioned above needs to be created inside `configuration`. Make sure json files saved in the following location for your new module-
 
 ```
 /stix_shifter_modules/<module name>/configuration
 ```
 
-## JSON File Description
+### JSON File Description
 
-### config json file
+#### config json file
 
 Two top level json objects needs to be preset in the file: `connection` and `configuration`. The child attributes of the `connection` object should be the parameters required for making API calls which can be used by multiple users and role levels. The `configuration` object should contain the parameters that are required for API authentication for individual users and roles. 
 
@@ -104,7 +104,7 @@ Configuration object needs to have `auth` child object. `auth` object should con
 
 Both connection and configuration object may contain more or different parameters than that are defined in the example above based on the individual module. 
 
-### lang json file
+#### lang json file
 
 The `lang_en.json` file has the similar format like `config.json`. It has different child attributes to translate the files for UI framework.
 

docs/adapter-guide/develop-mapping-keywords.md

@@ -1,18 +1,28 @@
-# To STIX mapping Keywords
+## Mapping Keywords
 
 There are keywords which need to be specified in the `to-stix` mappings in order to perform specific operations on the datasource fields. There are two types of keywords:
 1. Required
 2. Optional 
 
 The below table contains the keywords and their usages:
 
-## Required Keywords
+### Required Keywords
 
-<table>
-<tr>
-<td> Keywords </td> <td> Type </td> <td> Descriptions </td> <td> Usage </td> <td> Example </td>
+<style>
+table, th, td {
+  border: 1px solid black;
+  border-collapse: collapse;
+}
+th, td {
+  padding: 15px;
+}
+</style>
+
+<table style="width:100%; border: 1px solid black;  border-collapse: collapse;">
+<tr border: 1px solid black;>
+<th> Keywords </th> <th> Type </th> <th> Descriptions </th> <th> Usage </th> <th> Example </th>
 </tr>
-<td><b> key </b></td> <td> String </td> <td> The STIX object and properties whose path is defined in dot notation.</td> <td> "key": "stix-object.stix_object_property.sub_property" </td>
+<td><b> key </b></td> <td> String </td> <td> The STIX object and properties whose path is defined in dot notation.</td> <td> "key": "stix-object.stix-object-property.sub-property" </td>
 <td>
 
 ```json
@@ -41,11 +51,11 @@ The below table contains the keywords and their usages:
 </table>
 
 
-## Optional Keywords
+### Optional Keywords
 
 <table>
 <tr>
-<td> Keywords </td> <td> Type </td> <td> Descriptions </td> <td> Usage </td>
+<th> Keywords </th> <th> Type </th> <th> Descriptions </th> <th> Usage </th>
 </tr>
 <tr>
 <td><b> references </b></td> <td> String/List(string) </td> <td> Specifies named objects to reference in another object. </td> <td> "references": "src_ip" <br>"references": ["dst_mac"] </td>
@@ -65,14 +75,14 @@ The below table contains the keywords and their usages:
 <tr>
 <td><b> group_ref </b></td> <td> Boolean </td> <td> This keyword needs to be used when there is a nested list of dictionaries and each dictionary item creates an object. This keyword groups together references in a list and sets where the object is mapped. </td> <td> "group_ref": true </td>
 </tr>
-<td><b> ds_key </b></td> <td> String </td> <td> This keyword is used when datasource results are formatted to modify some field names. The value assigned to the keyword determines the mapping of a STIX object. This keyword is only used in the aws_athena and aws_cloud_watch_logs modules to resolve nested dictionary mappings. <b>This keyword has been deprecated since nested dictionary mappings are now handled by the JSON to STIX translation utility.</b> </td> <td> "ds_key": "resource_instancedetails_networkinterfaces_0_networkinterfaceid" </td>
+<td><b> ds_key </b></td> <td> String </td> <td> This keyword is used when datasource results are formatted to modify some field names. The value assigned to the keyword determines the mapping of a STIX object. This keyword is only used in the aws_athena and aws_cloud_watch_logs modules to resolve nested dictionary mappings. <b>This keyword has been deprecated since nested dictionary mappings are now handled by the JSON to STIX translation utility.</b> </td> <td> "ds_key": "resource_instancedetails" </td>
 </tr>
 </table>
 
 
-## Examples of Optional keywords:
+### Examples of Optional keywords:
 
-##  unwrap 
+####  unwrap 
 
 **Mapping:**
 
@@ -140,7 +150,7 @@ This STIX bundle contains two ipv4-addr objects which are created based on `unwr
 
 <br>
 
-##  group 
+####  group 
 
 **Mapping:**
 
@@ -236,7 +246,7 @@ This STIX bundle contains two ipv4-addr objects which are created based on `unwr
 
 <br>
 
-##  group_ref 
+#### group_ref 
 
 **Mapping:**
 
@@ -337,7 +347,7 @@ List of nested dictionary in datasource results are referenced in `scanned_refs`
 
 <br>
 
-## value 
+#### value 
 
 **Mapping:**
 
@@ -397,7 +407,7 @@ List of nested dictionary in datasource results are referenced in `scanned_refs`
 
 <br>
 
-##  references 
+#### references 
 
 **Mapping:**
 
@@ -477,7 +487,7 @@ Source `ipv4-addr` object number is referenced in `network-traffic` object:
 
 <br>
 
-##   transformer  
+#### transformer  
 
 **Mapping:**
 ```

docs/adapter-guide/develop-stix-adapter.md

@@ -1,19 +1,12 @@
-# Developing a new STIX-shifter connector
+## Developing a new connector
 
-- [Introduction](../README.md)
-- [Overview of the project](../OVERVIEW.md)
-- [Scenario](#scenario)
-- [Prerequisites](#prerequisites)
-- [Best Practices for developing a connector](best_practices.md)
-- [Steps](#steps)
+### Scenario
 
-## Scenario
-
-### Participants
+***Participants***
 
 This scenario involves a software developer (_Developer A_) and an end user (_User A_). _Developer A_ wants to implement a new connector for the STIX-shifter project that can support a particular security product (_Product A_). _User A_ is another developer that uses the STIX-shifter library.
 
-### Problem to solve
+***Problem to solve***
 
 _User A_ performs security monitoring with _Product A_ and several other security products. The other products already have existing STIX-shifter connectors.
 
@@ -24,7 +17,7 @@ _User A_ would like to:
 
 By implementing a new connector, _Developer A_ allows _Product A_ to fit into the workflow.
 
-## Prerequisites
+### Prerequisites
 
 - Your development environment must use Python 3.8 or greater.
 - You must have access to the target data source. In the sample scenario, you must have access to Product A data source.
@@ -33,11 +26,11 @@ By implementing a new connector, _Developer A_ allows _Product A_ to fit into th
   - Observable objects. See [STIX™ Version 2.0. Part 4: Cyber Observable Objects](http://docs.oasis-open.org/cti/stix/v2.0/stix-v2.0-part4-cyber-observable-objects.html)
   - Stix patterning. See [STIX™ Version 2.0. Part 5: STIX Patterning](https://docs.oasis-open.org/cti/stix/v2.0/stix-v2.0-part5-stix-patterning.html)
 
-## Best practices
+### Best practices
 
 Familiarize yourself with some [best practices](best_practices.md) before beginning a new connector.
 
-## Steps
+### Steps
 
 To develop a STIX-shifter connector for a data source:
 
@@ -49,7 +42,7 @@ To develop a STIX-shifter connector for a data source:
 1. [Create the module entry points](#create-module-entry-points).
 1. Create a pull request to merge your changes in the `opencybersecurityalliance/stix-shifter` repository.
 
-### Create a module folder
+#### Create a module folder
 
 Connector modules are stored under the `stix_shifter_modules` directory. To help you get started with creating a new connector, two module templates are available. If your data source executes queries synchronously (there is no API call to check the status of the query), make a copy of the `synchronous_template` folder in the `stix_shifter_modules` directory. If your data source executes queries asynchronously, make a copy of the `async_template` folder. The instructions that follow use the async template as an example.
 
@@ -68,7 +61,7 @@ Each module contains the following directories and files:
 
 entry_point.py: Initializes classes and paths used by the connector. 
 
-### Create the module entry points
+#### Create the module entry points
 
 The `EntryPoint` class acts as a gateway to the various methods used by the translation and transmission classes. In most instances, it's fine to use the `setup_transmission_simple` and `setup_translation_simple(dialect_default='default')` methods. In cases where multiple dialects are used by the connector, the `dialect_default` argument is the dialect you wish to use as the default when the entire collection isn't passed in. See [Create a Translation module](develop-translation-module.md) to learn about dialects.
 
@@ -99,7 +92,7 @@ If the data source is synchronous, you must include `set_async(False)` in the co
 ```
 
 
-### Testing a new connector using the proxy host
+#### Testing a new connector using the proxy host
 
 Work on a new stix-shifter connector occurs after the project has been forked and cloned into a local development environment. Stix-shifter contains a **proxy** connector that facilitates a remote instance of the project calling out to a local instance. While in development, a new connector's working branch can be tested in any project using the stix-shifter library without first merging into the master branch on Github. A host is run on the local instance from the CLI. When a `proxy` data source is passed to the remote instance of stix-shifter, the real connection attributes (data source type, host, and port contained in the options) are passed onto the local instance of stix-shifter running the proxy host. The host will then use the new connector and return results back to the remote stix-shifter instance.
 
@@ -115,7 +108,7 @@ As an example:
 python main.py host '{"type": "identity","id": "identity--f431f809-377b-45e0-aa1c-6a4751cae5ff","name": "Bundle","identity_class": "events"}' "192.168.122.83:5000"
 ```
 
-#### Calling the proxy host
+##### Calling the proxy host
 
 Each of the translate and transmit CLI commands outlined in the stix-shifter overview can be used to call the proxy host.
 
@@ -125,7 +118,7 @@ As an example:
 python main.py transmit proxy '{"options": {"proxy_host": "127.0.0.1", "proxy_port": 5000, "destination": {"connection": {"options": {"result_limit": 10000, "time_range": 5, "timeout": 30}, "host": "<HOST>", "port": <PORT>, "type": "qradar"}, "configuration": {"auth": { "SEC": "<SEC TOKEN>"} } } }}' '{}' ping
 ```
 
-### Packaging individual connectors
+#### Packaging individual connectors
 
 Stix-shifter can be broken into several python whl packages by using the `setup.py` script found in the root of the project. This packaging script can be called from the CLI:
 
@@ -162,7 +155,7 @@ stix_shifter_modules =>
         entry_point
 ```
 
-### Building images of the connectors
+#### Building images of the connectors
 
 You can build the docker image your developed connector locally and publish it to your desired repository. In order to do that, follow the below steps-