misp-playbooks/pb_using_timestamps_in_MISP-with_output.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "45036329-e06b-4060-8a0f-79e74f4f45f1",
   "metadata": {},
   "source": [
    "# Using timestamps in MISP\n",
    "\n",
    "## Introduction\n",
    "\n",
    "- UUID: **7b498d8f-d1a3-4576-b894-58f908fff82c**\n",
    "- Started from [issue 42](https://github.com/MISP/misp-playbooks/issues/42)\n",
    "- State: **Published** : demo version with output\n",
    "- Purpose:\n",
    "    - A playbook that documents the different timestamps that are used in MISP.\n",
    "    - Go through the timestamp for publishing and last changes as well as how these can be used in search queries.\n",
    "    - Document what changes a timestamp in a MISP event.\n",
    "- Tags: [ \"timestamp\", \"export\" ]\n",
    "- External resources: None\n",
    "- Target audience: **CTI**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "75a71ce4-eb61-4d8a-9f32-11e274b21800",
   "metadata": {
    "tags": []
   },
   "source": [
    "# Playbook TOC\n",
    "\n",
    "- **Using timestamps in MISP**\n",
    "   - Introduction\n",
    "- **Timestamps in MISP**\n",
    "   - IN:1 Introduction\n",
    "   - IN:2 Date formats\n",
    "- **Preparation**\n",
    "   - PR:1 Initialise environment\n",
    "   - PR:2 Load helper functions\n",
    "- **Adding events and attributes to MISP**\n",
    "   - AD:1 Create an event\n",
    "   - AD:2 Add attributes to an event\n",
    "   - AD:3 Publish the event\n",
    "   - AD:4 Summary\n",
    "- **Event search context**\n",
    "   - ES:1 Search for recently published events\n",
    "   - ES:2 Search for recently changed events\n",
    "   - ES:3 Search for events on their date field\n",
    "   - ES:4 What is the \"First recorded change\" of an event?\n",
    "- **Attribute search context**\n",
    "   - AS:1 Search for attributes on their last change timestamp\n",
    "   - AS:2 Search for attributes on their first seen or last seen value\n",
    "   - AS:3 Search for attributes in events with a specific recent change or published timestamp\n",
    "   - AS:4 Search for attributes in events with a specific date field\n",
    "   - AS:5 Search for first change of an attribute\n",
    "- **Object search context**\n",
    "   - OS:1 Search for objects\n",
    "- **Sightings**\n",
    "   - SI:1 Adding sightings\n",
    "   - SI:2 Search for sightings\n",
    "- **Proposal / ShadowAttributes**\n",
    "   - PS1: Search for proposals or shadowattributes\n",
    "- **Closure**\n",
    "   - EN:1 End of the playbook\n",
    "   - External references\n",
    "   - Technical details"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a81feb43-7c2d-4e22-bb42-a99a2fb39e83",
   "metadata": {},
   "source": [
    "# Timestamps in MISP\n",
    "\n",
    "## IN:1 Introduction\n",
    "\n",
    "All the activities in MISP, such as creating events, adding attributes or editing objects, are timestamped. These timestamps are not only tracked in the audit log but are an integral part of the [MISP data model](https://www.misp-project.org/misp-training/a.11-misp-data-model.pdf) for events, attributes or objects.\n",
    "\n",
    "## IN:2 Date formats \n",
    "\n",
    "Before we continue it's useful to review the different date formats that you can use (primarily for searching) in MISP. You can use \n",
    "\n",
    "- An exact **date**, such as *2023-07-10*, sometimes with and sometimes without hour and minute representation. Corresponds with ISO 8601 date notation.\n",
    "- An exact **UNIX timestamp** match, such as *1688655953*\n",
    "- A **short-hand** notation, such as *1d* or *3h*\n",
    "- A **date range**, such as *[\"14d\", \"7d\"]*"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8dc24913",
   "metadata": {
    "tags": []
   },
   "source": [
    "# Preparation\n",
    "\n",
    "## PR:1 Initialise environment\n",
    "\n",
    "This section **initialises the playbook environment** and loads the required Python libraries. The credentials for MISP (**API key**) are loaded from the file `keys.py` in the directory **vault**. A [PyMISP](https://github.com/MISP/PyMISP) object is created to interact with MISP and the active MISP server is displayed. By printing out the server name you know that it's possible to connect to MISP. In case of a problem PyMISP will indicate the error with `PyMISPError: Unable to connect to MISP`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "5f94c336",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "The version of PyMISP recommended by the MISP instance (2.4.172) is newer than the one you're using now (2.4.168.1). Please upgrade PyMISP.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The \u001b[92mPython libraries\u001b[90m are loaded and the \u001b[92mcredentials\u001b[90m are read from the keys file.\n",
      "I will use the MISP server \u001b[92mhttps://misp.demo.cudeso.be/\u001b[90m for this playbook.\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Initialise Python environment\n",
    "import urllib3\n",
    "import sys\n",
    "import json\n",
    "from datetime import date\n",
    "import datetime\n",
    "from prettytable import PrettyTable, MARKDOWN\n",
    "from pymisp import *\n",
    "import requests\n",
    "\n",
    "# Load the credentials\n",
    "sys.path.insert(0, \"../vault/\")\n",
    "from keys import *\n",
    "if misp_verifycert is False:\n",
    "    import urllib3\n",
    "    urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)\n",
    "print(\"The \\033[92mPython libraries\\033[90m are loaded and the \\033[92mcredentials\\033[90m are read from the keys file.\")\n",
    "\n",
    "# Create the PyMISP object\n",
    "misp = PyMISP(misp_url, misp_key, misp_verifycert)\n",
    "print(\"I will use the MISP server \\033[92m{}\\033[90m for this playbook.\\n\\n\".format(misp_url))\n",
    "\n",
    "# Set the headers in case we use Python requests\n",
    "headers = {\"Authorization\": misp_key,\n",
    "           \"Accept\": \"application/json\",\n",
    "           \"Content-type\": \"application/json\"}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c576dbcd-9cf7-4acc-9773-73bf736cb7a3",
   "metadata": {},
   "source": [
    "## PR:2 Load helper functions\n",
    "\n",
    "The next cell contains **helper functions** that are used in this playbook. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "c84186a6-e0cb-4e15-a586-daaeef65351d",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Return a PrettyTable object with columns 'fields'\n",
    "def eventstable(fields):\n",
    "    table = PrettyTable()\n",
    "    table.field_names = fields\n",
    "    table.align[\"Event\"] = \"l\"\n",
    "    table.align[\"Event ID\"] = \"l\"\n",
    "    table._max_width = {\"Event\": 45}\n",
    "    return table\n",
    "\n",
    "\n",
    "# Convert a UNIX timestamp to a human readable date format \n",
    "def unix_timestamp_to_date(timestamp, display_format=\"%Y-%m-%d %H:%M:%S\"):\n",
    "    timestamp = int(timestamp)\n",
    "    dt = datetime.datetime.fromtimestamp(timestamp)\n",
    "    return dt.strftime(display_format)\n",
    "\n",
    "\n",
    "# Convert a human readable date format to timestamp\n",
    "def date_to_unix_timestamp(human_date):\n",
    "    dt_object = datetime.datetime.strptime(human_date, \"%Y-%m-%d\")\n",
    "    timestamp = dt_object.timestamp()\n",
    "    return timestamp"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "099fb1fe-639d-4567-bb19-1ad53c9f387e",
   "metadata": {},
   "source": [
    "# Adding events and attributes to MISP\n",
    "\n",
    "## AD:1 Create an event\n",
    "\n",
    "When you **add a MISP event** you need to provide a date field. This date refers to when a threat was observed or when an incident occurred. The [date](https://www.misp-standard.org/rfc/misp-standard-core.html#name-date) field is in ISO 8601 format (**YYYY-MM-DD**). When you manually add an event via the MISP user interface you can use a date picker to set the correct date. \n",
    "\n",
    "This date field is set by the person (or script) that created the threat event, this timestamp is not set automatically by MISP.\n",
    "\n",
    "![helpers/timestamp_add.png](helpers/timestamp_add.png)\n",
    "\n",
    "### Use PyMISP to create an event\n",
    "\n",
    "The example uses PyMISP to create a simple MISP event, sets a date with **set_date** and then requests the event from the MISP server."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "fda56d32-6d78-4528-b9be-7c93cf2396b8",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Created event ID \u001b[92m3019\u001b[90m with UUID \u001b[92m92bc628b-0b79-4e7d-b4c9-e61830a3f246\u001b[90m.\n",
      "\n",
      "{'id': '3019', 'orgc_id': '1', 'org_id': '1', 'date': '2023-07-23', 'threat_level_id': '4', 'info': 'Demo event for MISP', 'published': False, 'uuid': '92bc628b-0b79-4e7d-b4c9-e61830a3f246', 'attribute_count': '0', 'analysis': '1', 'timestamp': '1689079679', 'distribution': '3', 'proposal_email_lock': False, 'locked': False, 'publish_timestamp': '0', 'sharing_group_id': '0', 'disable_correlation': False, 'extends_uuid': '', 'protected': None, 'event_creator_email': 'admin@admin.test', 'Org': {'id': '1', 'name': 'DEMO-ORG', 'uuid': '5c1eb4f8-bae5-45f5-a772-06d4a3f64c3e', 'local': True}, 'Orgc': {'id': '1', 'name': 'DEMO-ORG', 'uuid': '5c1eb4f8-bae5-45f5-a772-06d4a3f64c3e', 'local': True}, 'Attribute': [], 'ShadowAttribute': [], 'RelatedEvent': [], 'Galaxy': [], 'Object': [], 'EventReport': [], 'CryptographicKey': []}\n",
      "\n",
      "Date ('date'): \u001b[92m 2023-07-23 \u001b[90m\n",
      "Publish timestamp ('publish_timestamp'): \u001b[92m 0 \u001b[90m\n",
      "Timestamp ('timestamp'): \u001b[92m 1689079679 \u001b[90m\n"
     ]
    }
   ],
   "source": [
    "event = MISPEvent()\n",
    "event.info = \"Demo event for MISP\"\n",
    "event.distribution = 3\n",
    "event.threat_level_id = 4\n",
    "event.analysis = 1\n",
    "\n",
    "# Set the MISP event date\n",
    "event.set_date(\"2023-07-23\")\n",
    "\n",
    "result = misp.add_event(event)[\"Event\"]\n",
    "event_uuid = result[\"uuid\"]\n",
    "timestamp_on_create = result[\"timestamp\"]\n",
    "\n",
    "print(\"Created event ID \\033[92m{}\\033[90m with UUID \\033[92m{}\\033[90m.\\n\\n{}\\n\".format(result[\"id\"], result[\"uuid\"], result))\n",
    "print(\"Date ('date'): \\033[92m\", result[\"date\"], \"\\033[90m\")\n",
    "print(\"Publish timestamp ('publish_timestamp'): \\033[92m\", result[\"publish_timestamp\"], \"\\033[90m\")\n",
    "print(\"Timestamp ('timestamp'): \\033[92m\", timestamp_on_create, \"\\033[90m\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fcc07b7f-99c3-4fe9-9632-b15b40437451",
   "metadata": {},
   "source": [
    "### Output from PyMISP after creating the event\n",
    "\n",
    "As you can notice from the previous output:\n",
    "- The **date** is the value supplied during event creation. It is in ISO 8601 format.\n",
    "- The **publish_timestamp** is not yet set. It is set by MISP following a publishing action and is covered in a later section.\n",
    "- The **timestamp** indicates the last change of the event, is set automatically by MISP and is returned in UNIX timestamp format."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1aefbe82-a715-47a0-81be-88f9e49eb667",
   "metadata": {},
   "source": [
    "## AD:2 Add attributes to an event\n",
    "\n",
    "Next we add an attribute to the event. There are two *optional* dates that you can supply:\n",
    "\n",
    "- The **first seen**, indicating when you have first seen the attribute\n",
    "- The **last seen**, when you have last seen the attribute\n",
    "\n",
    "These dates add contextualisation, in the sense that they allow you to indicate if a threat (an indicator) is still relevant. They are *optional*, you can combine them or only add the first seen, and not the last seen, or vice-versa. The [first seen and last seen](https://www.misp-standard.org/rfc/misp-standard-core.html#name-first_seen) are in an ISO 8601 format, up to the micro-second with time zone support.\n",
    "\n",
    "![helpers/timestamp_add_attribute.png](helpers/timestamp_add_attribute.png)\n",
    "\n",
    "When you add attributes with the *freetext import tool*, you do not get the option to add the first seen or last seen. You can still add them later on, once the attributes are added to the event. MISP also does not show the first seen or last seen value when you list the attributes in a threat event. To view them you have to click on the **Context** button. You can also view them in a more visual representation under the **Event timeline**.\n",
    "\n",
    "![helpers/timestamp_attribute_seen.png](helpers/timestamp_attribute_seen.png)\n",
    "\n",
    "### Use PyMISP to add an attribute \n",
    "\n",
    "With PyMISP you can provide the first seen (**first_seen**) and last seen (**last_seen**) value in UNIX timestamp format but also as a human readable date and time.\n",
    "\n",
    "There's one additional timestamp in the example below and that is the attribute **timestamp**. You can specify a timestamp when you create an attribute, but MISP will ignore this value. The timestamp is set **automatically** by MISP to the time when the action is executed, you cannot override this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "ded1cba2-0459-4955-8f3c-531d46384eaa",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Added attribute \u001b[92m1d94558f-5b54-4a34-8fba-29eeba691fde\u001b[90m\n",
      "\n",
      "{'id': '2633129', 'event_id': '3019', 'object_id': '0', 'object_relation': None, 'category': 'Network activity', 'type': 'ip-dst', 'value1': '1.2.2.1', 'value2': '', 'to_ids': True, 'uuid': '1d94558f-5b54-4a34-8fba-29eeba691fde', 'timestamp': '1689079680', 'distribution': '5', 'sharing_group_id': '0', 'comment': 'There is an initial request to this IP', 'deleted': False, 'disable_correlation': False, 'first_seen': '2023-07-10T20:21:34.000000+00:00', 'last_seen': '2023-07-15T00:00:00.000000+00:00', 'value': '1.2.2.1'}\n",
      "\n",
      "Timestamp ('timestamp'): \u001b[92m 1689079680 \u001b[90m\n",
      "First seen ('first_seen'): \u001b[92m 2023-07-10T20:21:34.000000+00:00 \u001b[90m\n",
      "Last seen ('last_seen'): \u001b[92m 2023-07-15T00:00:00.000000+00:00 \u001b[90m\n"
     ]
    }
   ],
   "source": [
    "attribute = MISPAttribute()\n",
    "attribute.type = \"ip-dst\"\n",
    "attribute.value = \"1.2.2.1\"\n",
    "attribute.to_ids = True\n",
    "attribute.comment = \"There is an initial request to this IP\"\n",
    "\n",
    "attribute.timestamp = 8551954799 # This will be ignored\n",
    "attribute.first_seen = 1689020494  # Provide first seen in UNIX timestamp format\n",
    "attribute.last_seen = \"2023-07-15\" # Provide last seen in human readable date format\n",
    "\n",
    "result_attr = misp.add_attribute(event_uuid, attribute)[\"Attribute\"]\n",
    "result_attr_uuid = result_attr[\"uuid\"]\n",
    "attribute_timestamp = result_attr[\"timestamp\"]\n",
    "\n",
    "print(\"Added attribute \\033[92m{}\\033[90m\\n\\n{}\\n\".format(result_attr_uuid, result_attr))\n",
    "print(\"Timestamp ('timestamp'): \\033[92m\", attribute_timestamp, \"\\033[90m\")\n",
    "print(\"First seen ('first_seen'): \\033[92m\", result_attr[\"first_seen\"], \"\\033[90m\")\n",
    "print(\"Last seen ('last_seen'): \\033[92m\", result_attr[\"last_seen\"], \"\\033[90m\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a0462aa8-a2dc-425b-80c0-58de466a5892",
   "metadata": {},
   "source": [
    "## AD:3 Publish the event\n",
    "\n",
    "When an event is created it is visible on the MISP instance (depending on the event distribution setting) but it is not exportable or synchronised with other servers. \n",
    "\n",
    "![helpers/timestamp_publishing.png](helpers/timestamp_publishing.png)\n",
    "\n",
    "It's only after **publishing the event** that it is exported or synchronised. When you publish an event its publish timestamp is **automatically** set by MISP. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4d2596d6-c391-4851-8f50-48e36a82863f",
   "metadata": {},
   "source": [
    "### Use PyMISP to publish an event\n",
    "\n",
    "Publishing a MISP event with PyMISP is easy with the **publish** function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "6ad17a6f-bd8b-4f6d-a359-c9413b651db9",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Publish MISP event \u001b[92m92bc628b-0b79-4e7d-b4c9-e61830a3f246\u001b[90m\n",
      "{'saved': True, 'success': True, 'name': 'Job queued', 'message': 'Job queued', 'url': '/events/publish/3019', 'id': '3019'}\n"
     ]
    }
   ],
   "source": [
    "print(\"Publish MISP event \\033[92m{}\\033[90m\".format(event_uuid))\n",
    "print(misp.publish(event_uuid))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e16b5d2c-a90a-4053-ac15-6ae499a064b6",
   "metadata": {},
   "source": [
    "### Use PyMISP to retrieve details of the event\n",
    "\n",
    "Next we query MISP for the event that we just published and analyse which timestamps have changed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "d8e10fc3-63f1-4534-ac38-af18030e58fc",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'id': '3019', 'orgc_id': '1', 'org_id': '1', 'date': '2023-07-23', 'threat_level_id': '4', 'info': 'Demo event for MISP', 'published': False, 'uuid': '92bc628b-0b79-4e7d-b4c9-e61830a3f246', 'attribute_count': '1', 'analysis': '1', 'timestamp': '1689079680', 'distribution': '3', 'proposal_email_lock': False, 'locked': False, 'publish_timestamp': '0', 'sharing_group_id': '0', 'disable_correlation': False, 'extends_uuid': '', 'protected': None, 'event_creator_email': 'admin@admin.test', 'Org': {'id': '1', 'name': 'DEMO-ORG', 'uuid': '5c1eb4f8-bae5-45f5-a772-06d4a3f64c3e', 'local': True}, 'Orgc': {'id': '1', 'name': 'DEMO-ORG', 'uuid': '5c1eb4f8-bae5-45f5-a772-06d4a3f64c3e', 'local': True}, 'Attribute': [{'id': '2633129', 'type': 'ip-dst', 'category': 'Network activity', 'to_ids': True, 'uuid': '1d94558f-5b54-4a34-8fba-29eeba691fde', 'event_id': '3019', 'distribution': '5', 'timestamp': '1689079680', 'comment': 'There is an initial request to this IP', 'sharing_group_id': '0', 'deleted': False, 'disable_correlation': False, 'object_id': '0', 'object_relation': None, 'first_seen': '2023-07-10T20:21:34.000000+00:00', 'last_seen': '2023-07-15T00:00:00.000000+00:00', 'value': '1.2.2.1', 'Galaxy': [], 'ShadowAttribute': []}], 'ShadowAttribute': [], 'RelatedEvent': [], 'Galaxy': [], 'Object': [], 'EventReport': [], 'CryptographicKey': []} \n",
      "\n",
      "Date ('date'): \u001b[92m 2023-07-23 \u001b[90m\n",
      "Publish timestamp ('publish_timestamp'): \u001b[92m 0 \u001b[90m\n",
      "Timestamp ('timestamp'): \u001b[92m 1689079680 \u001b[90m\n",
      "Timestamp when we created the event ('timestamp'): \u001b[92m 1689079679 \u001b[90m\n",
      "Timestamp when we added the attribute ('timestamp'): \u001b[92m 1689079680 \u001b[90m\n"
     ]
    }
   ],
   "source": [
    "result = misp.search(controller=\"events\", uuid=event_uuid)[0][\"Event\"]\n",
    "print(result, \"\\n\")\n",
    "\n",
    "print(\"Date ('date'): \\033[92m\", result[\"date\"], \"\\033[90m\")\n",
    "print(\"Publish timestamp ('publish_timestamp'): \\033[92m\", result[\"publish_timestamp\"], \"\\033[90m\")\n",
    "print(\"Timestamp ('timestamp'): \\033[92m\", result[\"timestamp\"], \"\\033[90m\")\n",
    "print(\"Timestamp when we created the event ('timestamp'): \\033[92m\", timestamp_on_create, \"\\033[90m\")\n",
    "if result[\"timestamp\"] == attribute_timestamp:\n",
    "    print(\"Timestamp when we added the attribute ('timestamp'): \\033[92m\", attribute_timestamp, \"\\033[90m\")\n",
    "else:\n",
    "    print(\"There have been other changes after adding attributes in this playbook.\\n   For reference, the timestamp when we added the attribute ('timestamp'): \\033[92m\", attribute_timestamp, \"\\033[90m\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b7b37952-69a7-400a-b6a0-af2257bdb52a",
   "metadata": {},
   "source": [
    "### Results of adding data with PyMISP\n",
    "\n",
    "As you can observe in the above example:\n",
    "\n",
    "- The **publish timestamp** is updated because of the publish action. This timestamp is set by MISP and returned in UNIX timestamp format.\n",
    "- The last change (**timestamp**) of the event corresponds with the time the attribute was added. This timestamp is also set by MISP and returned in UNIX timestamp format.\n",
    "\n",
    "## AD:4 Summary\n",
    "\n",
    "In the previous cells we breefly covered how to add events and attributes and which timestamps come into play. A lot of **use cases for timestamps** in MISP deal with searching (and filtering) for data. This is covered in the next section."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4740cc7a-313f-44b9-ba0e-5e8dbc828410",
   "metadata": {},
   "source": [
    "# Event search context\n",
    "\n",
    "## ES:1 Search for recently published events\n",
    "\n",
    "When you publish an event it's **publish timestamp** is automatically updated by MISP. Searching for events based on their publish timestamps then allows you to filter for recently created (or synchronised) events made available on your MISP server. The [publish_timestamp](https://www.misp-standard.org/rfc/misp-standard-core.html#name-publish_timestamp) is expressed in seconds (decimal) since 1st of January 1970 (Unix timestamp). At each publication of an event, the publish_timestamp is. If the published_timestamp is present and the published flag is set to false, the publish_timestamp represents the previous publication timestamp. If the event was never published, the published_timestamp is set to 0.\n",
    "\n",
    "This publish timestamp corresponds with what's on the *Published* line when you view the details of a MISP event. It is set on *your* MISP instance, when you published the MISP event on your instance (either automatically for example after syncing or manually because of a follow-up action).\n",
    "\n",
    "![helpers/timestamp_published.png](helpers/timestamp_published.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f9171481-3b74-4447-b980-9aeb66c4c9ad",
   "metadata": {},
   "source": [
    "### Search for published or unpublished events\n",
    "\n",
    "The published timestamp represents the timestamp when the event was **last published**. It is not an indication if the event is **currently published**. To clarify: an event can be in an unpublished state, and still have a published timestamp, referring to a previous action. If you do not specify to PyMISP to only return published or unpublished events, then it will return **both**, the published and unpublished events.\n",
    "\n",
    "### Use PyMISP to search for events with a recent publish timestamp and use a short-hand date format\n",
    "\n",
    "In the PyMISP search function set the controller to **events** (the default setting) and use the **publish_timestamp** parameter (technically you can also use the **last** parameter, it is a synonym for publish_timestamp but is deprecated). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "ac9b1120-1877-48f3-a45a-befd63a9cdfd",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Search for \u001b[92mevents with a recent publish timestamp\u001b[90m and use a \u001b[92mshort-hand\u001b[90m date format\n",
      "+----------+-----------------------------------------------+---------------------------+-----------+\n",
      "| Event ID | Event                                         |       Published date      | Published |\n",
      "+----------+-----------------------------------------------+---------------------------+-----------+\n",
      "| 2978     | The Curious Case of “Monti” Ransomware: A     | 2023-07-05 18:40:59+00:00 |    True   |\n",
      "|          | Real-World Doppelganger BlackBerry Logo       |                           |           |\n",
      "| 2995     | CVE investigation for CVE-2021-44228          | 2023-07-06 19:31:46+00:00 |    True   |\n",
      "| 2662     | An older threat event                         | 2023-07-10 11:48:28+00:00 |   False   |\n",
      "| 3019     | Demo event for MISP                           | 2023-07-11 12:48:03+00:00 |    True   |\n",
      "+----------+-----------------------------------------------+---------------------------+-----------+ \n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Get events with a publish timestamp of 10d or less\n",
    "print(\"Search for \\033[92mevents with a recent publish timestamp\\033[90m and use a \\033[92mshort-hand\\033[90m date format\")\n",
    "events = misp.search(\"events\", publish_timestamp=\"10d\", pythonify=True)\n",
    "table = eventstable([\"Event ID\", \"Event\", \"Published date\", \"Published\"])\n",
    "for event in events:\n",
    "    table.add_row([event.id, event.info, event.publish_timestamp, event.published])\n",
    "print(table.get_string(sortby=\"Published date\"), \"\\n\\n\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "242c5e7e-6fd1-4d67-a510-87325efca612",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Search for \u001b[92mevents with a recent publish timestamp\u001b[90m, use a \u001b[92mshort-hand\u001b[90m date format and only return \u001b[92mpublished\u001b[90m events.\n",
      "+----------+-----------------------------------------------+---------------------------+-----------+\n",
      "| Event ID | Event                                         |       Published date      | Published |\n",
      "+----------+-----------------------------------------------+---------------------------+-----------+\n",
      "| 2978     | The Curious Case of “Monti” Ransomware: A     | 2023-07-05 18:40:59+00:00 |    True   |\n",
      "|          | Real-World Doppelganger BlackBerry Logo       |                           |           |\n",
      "| 2995     | CVE investigation for CVE-2021-44228          | 2023-07-06 19:31:46+00:00 |    True   |\n",
      "| 3019     | Demo event for MISP                           | 2023-07-11 12:48:03+00:00 |    True   |\n",
      "+----------+-----------------------------------------------+---------------------------+-----------+ \n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Get published events with a publish timestamp of 10d or less\n",
    "print(\"Search for \\033[92mevents with a recent publish timestamp\\033[90m, use a \\033[92mshort-hand\\033[90m date format and only return \\033[92mpublished\\033[90m events.\")\n",
    "events = misp.search(\"events\", publish_timestamp=\"10d\", pythonify=True, published=1)\n",
    "table = eventstable([\"Event ID\", \"Event\", \"Published date\", \"Published\"])\n",
    "for event in events:\n",
    "    table.add_row([event.id, event.info, event.publish_timestamp, event.published])\n",
    "print(table.get_string(sortby=\"Published date\"), \"\\n\\n\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "79fa38b0-3f26-4a69-8cd1-d0e62bcbb100",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Search for \u001b[92mevents with a recent publish timestamp\u001b[90m, use a \u001b[92mshort-hand date\u001b[90m format and only return \u001b[92mnot-published\u001b[90m events.\n",
      "+----------+-----------------------+---------------------------+-----------+\n",
      "| Event ID | Event                 |       Published date      | Published |\n",
      "+----------+-----------------------+---------------------------+-----------+\n",
      "| 2662     | An older threat event | 2023-07-10 11:48:28+00:00 |   False   |\n",
      "+----------+-----------------------+---------------------------+-----------+ \n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Get the not-published events with a publish timestamp of 10d or less\n",
    "print(\"Search for \\033[92mevents with a recent publish timestamp\\033[90m, use a \\033[92mshort-hand date\\033[90m format and only return \\033[92mnot-published\\033[90m events.\")\n",
    "events = misp.search(\"events\", publish_timestamp=\"10d\", pythonify=True, published=0)\n",
    "table = eventstable([\"Event ID\", \"Event\", \"Published date\", \"Published\"])\n",
    "for event in events:\n",
    "    table.add_row([event.id, event.info, event.publish_timestamp, event.published])\n",
    "print(table.get_string(sortby=\"Published date\"), \"\\n\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "51ae1703-d3d5-4e95-bfde-48d72aaedd00",
   "metadata": {},
   "source": [
    "### Use PyMISP to search for published events during a specific date range specified in short-hand format\n",
    "\n",
    "Instead of specifying a date in short-hand format you can also specify a **date range** with the format *[\"startdate\",\"enddate\"]*. The below example searches for events published between 10 and 3 days ago."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "b2aa8525-291e-41b2-96cf-41902cb067f6",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Search for \u001b[92mevents with a recent publish timestamp\u001b[90m, use a date \u001b[92mrange\u001b[90m and only return \u001b[92mpublished\u001b[90m events.\n",
      "+----------+-----------------------------------------------+---------------------------+-----------+\n",
      "| Event ID | Event                                         |       Published date      | Published |\n",
      "+----------+-----------------------------------------------+---------------------------+-----------+\n",
      "| 2978     | The Curious Case of “Monti” Ransomware: A     | 2023-07-05 18:40:59+00:00 |    True   |\n",
      "|          | Real-World Doppelganger BlackBerry Logo       |                           |           |\n",
      "| 2995     | CVE investigation for CVE-2021-44228          | 2023-07-06 19:31:46+00:00 |    True   |\n",
      "+----------+-----------------------------------------------+---------------------------+-----------+ \n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(\"Search for \\033[92mevents with a recent publish timestamp\\033[90m, use a date \\033[92mrange\\033[90m and only return \\033[92mpublished\\033[90m events.\")\n",
    "events = misp.search(\"events\", publish_timestamp=[\"3d\",\"10d\"], pythonify=True, published=1)\n",
    "table = eventstable([\"Event ID\", \"Event\", \"Published date\", \"Published\"])\n",
    "for event in events:\n",
    "    table.add_row([event.id, event.info, event.publish_timestamp, event.published])\n",
    "print(table.get_string(sortby=\"Published date\"), \"\\n\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3876fcda-76a3-4ece-a1f8-4b4e8e054380",
   "metadata": {},
   "source": [
    "### Use REST API to search for published events on an exact date\n",
    "\n",
    "The REST API uses a similar syntax as PyMISP. The difference is that instead of providing the parameters to PyMISP, you now have to supply them in JSON format: `{\"returnFormat\":\"json\", \"publish_timestamp\": \"yourdate\", \"published\":\"1\"}`. The result shows one of the advantages of using PyMISP. If you use the REST API you get the timestamps in UNIX timestamp, which you then need to convert to a human readable format. With PyMISP a date object is returned, provided you use the **pythonify** setting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "2bb41ede-1280-4e5a-8ff8-a2fbecf5fabd",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Search for \u001b[92mpublished events on an exact date\u001b[90m\n",
      "+----------+---------------------+----------------+---------------------+-----------+\n",
      "| Event ID | Event               | Published date | Published Converted | Published |\n",
      "+----------+---------------------+----------------+---------------------+-----------+\n",
      "| 3019     | Demo event for MISP |   1689079683   | 2023-07-11 12:48:03 |    True   |\n",
      "+----------+---------------------+----------------+---------------------+-----------+ \n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(\"Search for \\033[92mpublished events on an exact date\\033[90m\")\n",
    "result = requests.post(\"{}/events/restSearch\".format(misp_url), headers=headers, verify=misp_verifycert,\n",
    "                           json={\"returnFormat\":\"json\", \"publish_timestamp\":date_to_unix_timestamp(\"2023-07-07\"), \"published\":\"1\"})\n",
    "table = eventstable([\"Event ID\", \"Event\", \"Published date\", \"Published Converted\", \"Published\"])\n",
    "if not \"errors\" in result.json():    \n",
    "    for event in result.json().get(\"response\", []):\n",
    "        table.add_row([event[\"Event\"][\"id\"], event[\"Event\"][\"info\"], event[\"Event\"][\"publish_timestamp\"], unix_timestamp_to_date(event[\"Event\"][\"publish_timestamp\"]), event[\"Event\"][\"published\"]])\n",
    "print(table.get_string(sortby=\"Published date\"), \"\\n\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c8e2b78e-1d17-497c-8311-f6445d430bf4",
   "metadata": {},
   "source": [
    "## ES:2 Search for recently changed events\n",
    "\n",
    "Searching for recently changed events is with the **timestamp** parameter. Similarly as with searching for the publish timestamp, you can specify a short-hand notation, an exact timestamp or a range. If you do not supply the published state as a parameter then both published and unpublished events are returned. The [timestamp](https://www.misp-standard.org/rfc/misp-standard-core.html#name-timestamp) represents a reference time when the event, or one of the attributes within the event was created, or last updated/edited on the instance. Timestamp is expressed in seconds (decimal) since 1st of January 1970 (Unix timestamp).\n",
    "\n",
    "This timestamp corresponds with the *Last change* line when you view the details of MISP event. This can be a timestamp set on *your* MISP instance (if you did a change to an event) but can also be set on a remote MISP server when you synchronise threat data.\n",
    "\n",
    "![helpers/timestamp_lastchange.png](helpers/timestamp_lastchange.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c05aa57a-c016-4e83-ae31-871b01d95cb3",
   "metadata": {},
   "source": [
    "### Use PyMISP to search for recently changed events\n",
    "\n",
    "Searching for recently changed events in PyMISP is similar as searching for events based on their publish_timestamp. Set the controller to events and use the **timestamp** parameter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "eabe6db8-979d-45a5-9f31-f744c7cd2427",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Search for \u001b[92mrecently changed\u001b[90m events with \u001b[92mshort-hand\u001b[90m format\n",
      "+----------+-----------------------------------------------+---------------------------+-----------+\n",
      "| Event ID | Event                                         |        Last change        | Published |\n",
      "+----------+-----------------------------------------------+---------------------------+-----------+\n",
      "| 2805     | X_Trader Supply Chain Attack Affects Critical | 2023-07-08 16:39:27+00:00 |   False   |\n",
      "|          | Infrastructure Organizations in U.S. and      |                           |           |\n",
      "|          | Europe                                        |                           |           |\n",
      "| 2964     | Minas – on the way to complexity              | 2023-07-08 16:55:02+00:00 |   False   |\n",
      "| 2662     | An older threat event                         | 2023-07-11 12:45:06+00:00 |   False   |\n",
      "| 3019     | Demo event for MISP                           | 2023-07-11 12:48:00+00:00 |    True   |\n",
      "+----------+-----------------------------------------------+---------------------------+-----------+ \n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(\"Search for \\033[92mrecently changed\\033[90m events with \\033[92mshort-hand\\033[90m format\")\n",
    "events = misp.search(\"events\", timestamp=\"3d\", pythonify=True)\n",
    "table = eventstable([\"Event ID\", \"Event\", \"Last change\", \"Published\"])\n",
    "for event in events:\n",
    "    table.add_row([event.id, event.info, event.timestamp, event.published])\n",
    "print(table.get_string(sortby=\"Last change\"), \"\\n\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c6dc98d0-4b6e-41d8-b82d-0e0dceec16be",
   "metadata": {},
   "source": [
    "### Use REST API to search for recently changed events\n",
    "\n",
    "Use the `timestamp` field in the JSON request that you send to the REST API: `{\"returnFormat\":\"json\", \"timestamp\": \"yourdate\"}`."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1891a428-cdd4-41e9-b271-b16a91dbf1c4",
   "metadata": {},
   "source": [
    "## ES:3 Search for events on their date field\n",
    "\n",
    "A MISP event has a date field that represents when a threat was observed or when an incident occured. Whereas the previous searches are done on the fields that are **automatically** set by MISP because of user actions, this search is done on a field **manually** set by a MISP user.\n",
    "\n",
    "This timestamp corresponds with the *Date* line when you view the details of MISP event and is set by the creator of the threat event.\n",
    "\n",
    "![helpers/timestamp_date.png](helpers/timestamp_date.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "18d632ac-109e-4185-a738-8dfb1b65c858",
   "metadata": {},
   "source": [
    "### Use PyMISP to search for events of specific dates\n",
    "\n",
    "You can search for events with a date set after the one specified with **date_from** or for events with a date set before the one specified with **date_to**. It's possible to combine both settings, allowing you to specify a date range."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "1eab02de-769a-4b97-92af-ee68768fa114",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Search for \u001b[92mevents after and before a specific date\u001b[90m\n",
      "+----------+-----------------------------------------------+------------+-----------+\n",
      "| Event ID | Event                                         |    Date    | Published |\n",
      "+----------+-----------------------------------------------+------------+-----------+\n",
      "| 2805     | X_Trader Supply Chain Attack Affects Critical | 2023-07-05 |   False   |\n",
      "|          | Infrastructure Organizations in U.S. and      |            |           |\n",
      "|          | Europe                                        |            |           |\n",
      "| 2995     | CVE investigation for CVE-2021-44228          | 2023-07-05 |    True   |\n",
      "| 2964     | Minas – on the way to complexity              | 2023-07-06 |   False   |\n",
      "+----------+-----------------------------------------------+------------+-----------+ \n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(\"Search for \\033[92mevents after and before a specific date\\033[90m\")\n",
    "events = misp.search(\"events\", date_from=\"2023-07-04\", date_to=\"2023-07-06\", pythonify=True)\n",
    "table = eventstable([\"Event ID\", \"Event\", \"Date\", \"Published\"])\n",
    "for event in events:\n",
    "    table.add_row([event.id, event.info, event.date, event.published])\n",
    "print(table.get_string(sortby=\"Date\"), \"\\n\\n\")   "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4b803a9f-b58b-4d1b-8293-ce77f67f2f58",
   "metadata": {},
   "source": [
    "### Use REST API to search for events of specific dates\n",
    "\n",
    "The previous search queries for PyMISP have a similar syntax when used directly with the REST API. In this case, when searching on the date field, the REST API does not use date_from or date_to, but **from** and **to**. For completeness, you can also use the **date** field with the REST API. Note that the REST API will not return an error if you use date_from or date_to, it will ignore these search parameters.\n",
    "\n",
    "In this case you also do not need to convert the returned date value to a human readable format, MISP does not return a UNIX timestamp, it returns the event date in a human readable (ISO 8601) format."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "e375dbca-232e-4d27-ab0e-fb113bec27e0",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Search for \u001b[92mevents after and before a specific date\u001b[90m\n",
      "+----------+-----------------------------------------------+------------+-----------+\n",
      "| Event ID | Event                                         |    Date    | Published |\n",
      "+----------+-----------------------------------------------+------------+-----------+\n",
      "| 2805     | X_Trader Supply Chain Attack Affects Critical | 2023-07-05 |   False   |\n",
      "|          | Infrastructure Organizations in U.S. and      |            |           |\n",
      "|          | Europe                                        |            |           |\n",
      "| 2995     | CVE investigation for CVE-2021-44228          | 2023-07-05 |    True   |\n",
      "| 2964     | Minas – on the way to complexity              | 2023-07-06 |   False   |\n",
      "+----------+-----------------------------------------------+------------+-----------+ \n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(\"Search for \\033[92mevents after and before a specific date\\033[90m\")\n",
    "result = requests.post(\"{}/events/restSearch\".format(misp_url), headers=headers, verify=misp_verifycert,\n",
    "                           json={\"returnFormat\":\"json\", \"from\":\"2023-07-04\", \"to\": \"2023-07-06\"})\n",
    "table = eventstable([\"Event ID\", \"Event\", \"Date\", \"Published\"])\n",
    "if not \"errors\" in result.json():    \n",
    "    for event in result.json().get(\"response\", []):\n",
    "        table.add_row([event[\"Event\"][\"id\"], event[\"Event\"][\"info\"], event[\"Event\"][\"date\"], event[\"Event\"][\"published\"]])\n",
    "else:\n",
    "    print(result.text())\n",
    "print(table.get_string(sortby=\"Date\"), \"\\n\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e020df7e-b6fb-4235-a7cc-8116042d6596",
   "metadata": {},
   "source": [
    "## ES:4 What is the \"First recorded change\" of an event?\n",
    "\n",
    "You probably noticed that when you view the details of MISP event, there's another useful timestamp, the *First recorded change*. Although this timestamp is displayed on the events detail page, it's actually calculated on the changes to **attributes** (or objects) (see [app/Controller/EventsController.php](https://github.com/MISP/MISP/blob/2.4/app/Controller/EventsController.php#L1469) ), more specifically on the first time an attribute (or object) was added to the MISP event. \n",
    "\n",
    "This timestamp is set automatically by MISP on the server where the event is created.\n",
    "\n",
    "![helpers/timestamp_firstchange.png](helpers/timestamp_firstchange.png)\n",
    "\n",
    "Because this timestamp is calculated on attribute (or object) information, you now need to set the PyMISP search controller to **attributes** (or objects) or use the attributes (or objects) REST API endpoint. In this playbook we first review how to do the search for attributes. You can apply the same approach when searching for objects."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3f603081-453d-45f1-be54-ca2ab120511e",
   "metadata": {},
   "source": [
    "# Attribute search context\n",
    "\n",
    "## AS:1 Search for attributes on their last change timestamp\n",
    "\n",
    "You can search for attributes based on the timestamp of their last change with the parameter **attribute_timestamp**. This [attribute timestamp](https://www.misp-standard.org/rfc/misp-standard-core.html#name-timestamp-2) is set automatically on the server where the attribute is last modified (or created).\n",
    "\n",
    "The attribute_timestamp corresponds with the value displayed in the date column.\n",
    "\n",
    "![helpers/timestamp_attribute_1.png](helpers/timestamp_attribute_1.png) \n",
    "\n",
    "### Search for actionable indicators only\n",
    "\n",
    "For attributes it is also useful to consider if you want to include all attributes or only those which are marked as **actionable**. If you only want the attributes marked as actionable, then add the additional filter **to_ids** with a value of True. Typically if you search for data that needs to be send to security devices then you only include the actionable indicators.\n",
    "\n",
    "### Use PyMISP to search for attributes on their last change timestamp\n",
    "\n",
    "In this section, because we search for attributes, you have to set the PyMISP search controller to use **attributes** (instead of the default events). Then use the parameter **attribute_timestamp** to search for the last change timestamps. By default the information returned by the -attribute- search will not contain event information such as the even last change timestamp or event published timestamp. In some cases it's interesting to also receive this information. Set the parameter **include_context** to True to also include the event context."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "e56bce1d-b38b-43ed-847e-402d1363834a",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Search for \u001b[92mattributes changed after a specific timestamp\u001b[90m\n",
      "+----------+----------------------------------+----------------------------------+---------------------------+-------------------+----------------+-----------+\n",
      "| Event ID | Event                            |              Value               |    Attribute timestamp    | Event last change | Published date | Published |\n",
      "+----------+----------------------------------+----------------------------------+---------------------------+-------------------+----------------+-----------+\n",
      "| 2964     | Minas – on the way to complexity | f38a1b6b132afa55ab48b4b7a8986181 | 2023-07-08 16:55:02+00:00 |      Unknown      |    Unknown     |   False   |\n",
      "| 3019     | Demo event for MISP              |             1.2.2.1              | 2023-07-11 12:48:00+00:00 |      Unknown      |    Unknown     |   False   |\n",
      "+----------+----------------------------------+----------------------------------+---------------------------+-------------------+----------------+-----------+ \n",
      "\n",
      "\n",
      "Search for \u001b[92mattributes changed after a specific timestamp\u001b[90m - with context\n",
      "+----------+----------------------------------+----------------------------------+---------------------------+---------------------------+---------------------------+-----------+\n",
      "| Event ID | Event                            |              Value               |    Attribute timestamp    |     Event last change     |       Published date      | Published |\n",
      "+----------+----------------------------------+----------------------------------+---------------------------+---------------------------+---------------------------+-----------+\n",
      "| 2964     | Minas – on the way to complexity | f38a1b6b132afa55ab48b4b7a8986181 | 2023-07-08 16:55:02+00:00 | 2023-07-08 16:55:02+00:00 | 2023-06-01 12:30:27+00:00 |   False   |\n",
      "| 3019     | Demo event for MISP              |             1.2.2.1              | 2023-07-11 12:48:00+00:00 | 2023-07-11 12:48:00+00:00 | 2023-07-11 12:48:03+00:00 |    True   |\n",
      "+----------+----------------------------------+----------------------------------+---------------------------+---------------------------+---------------------------+-----------+ \n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(\"Search for \\033[92mattributes changed after a specific timestamp\\033[90m\")\n",
    "attributes = misp.search(\"attributes\", attribute_timestamp=\"3d\", pythonify=True)\n",
    "table = eventstable([\"Event ID\", \"Event\", \"Value\", \"Attribute timestamp\", \"Event last change\", \"Published date\", \"Published\"])\n",
    "for attribute in attributes:\n",
    "    table.add_row([attribute.Event.id, attribute.Event.info, attribute.value, attribute.timestamp, \n",
    "                           attribute.Event.timestamp if hasattr(attribute.Event, \"timestamp\") else \"Unknown\", \n",
    "                           attribute.Event.publish_timestamp if hasattr(attribute.Event, \"publish_timestamp\") else \"Unknown\", attribute.Event.published])\n",
    "print(table.get_string(sortby=\"Attribute timestamp\"), \"\\n\\n\")\n",
    "\n",
    "print(\"Search for \\033[92mattributes changed after a specific timestamp\\033[90m - with context\")\n",
    "# Added event context with include_context=True\n",
    "attributes = misp.search(\"attributes\", attribute_timestamp=\"3d\", pythonify=True, include_context=True)\n",
    "table = eventstable([\"Event ID\", \"Event\", \"Value\", \"Attribute timestamp\", \"Event last change\", \"Published date\", \"Published\"])\n",
    "for attribute in attributes:\n",
    "    table.add_row([attribute.Event.id, attribute.Event.info, attribute.value, attribute.timestamp, \n",
    "                           attribute.Event.timestamp if hasattr(attribute.Event, \"timestamp\") else \"Unknown\", \n",
    "                           attribute.Event.publish_timestamp if hasattr(attribute.Event, \"publish_timestamp\") else \"Unknown\", attribute.Event.published])\n",
    "print(table.get_string(sortby=\"Attribute timestamp\"), \"\\n\\n\")  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "a2972d85-a9b7-4eb4-8fac-4dcc03b91588",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Search for \u001b[92mattributes changed after a specific timestamp\u001b[90m and only return the \u001b[92mactionable\u001b[90m indicators\n",
      "+----------+----------------------------------+----------------------------------+---------------------------+\n",
      "| Event ID | Event                            |              Value               |    Attribute timestamp    |\n",
      "+----------+----------------------------------+----------------------------------+---------------------------+\n",
      "| 2964     | Minas – on the way to complexity | f38a1b6b132afa55ab48b4b7a8986181 | 2023-07-08 16:55:02+00:00 |\n",
      "| 3019     | Demo event for MISP              |             1.2.2.1              | 2023-07-11 12:48:00+00:00 |\n",
      "+----------+----------------------------------+----------------------------------+---------------------------+ \n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(\"Search for \\033[92mattributes changed after a specific timestamp\\033[90m and only return the \\033[92mactionable\\033[90m indicators\")\n",
    "attributes = misp.search(\"attributes\", attribute_timestamp=\"3d\", to_ids=True, pythonify=True)\n",
    "table = eventstable([\"Event ID\", \"Event\", \"Value\", \"Attribute timestamp\"])\n",
    "for attribute in attributes:\n",
    "    table.add_row([attribute.Event.id, attribute.Event.info, attribute.value, attribute.timestamp])\n",
    "print(table.get_string(sortby=\"Attribute timestamp\"), \"\\n\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1f8474f0-10c8-44f9-9478-117b01a5de6b",
   "metadata": {},
   "source": [
    "### Use REST API to search for attributes on their last change timestamp\n",
    "\n",
    "The REST API is similar as the interactions with PyMISP. In this case you have to rely on the `attribute_timestamp` field in the JSON request: `{\"returnFormat\":\"json\", \"attribute_timestamp\": \"yourdate\"}`. Remember though, that in this case you have to use the **attributes** endpoint, and not the events endpoint."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "c215a676-1779-4448-8e5b-fe762d2d2a63",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Search for \u001b[92mattributes changed after a specific timestamp\u001b[90m\n",
      "+----------+----------------------------------+----------------------------------+---------------------+\n",
      "| Event ID | Event                            |              Value               | Attribute timestamp |\n",
      "+----------+----------------------------------+----------------------------------+---------------------+\n",
      "| 2964     | Minas – on the way to complexity | f38a1b6b132afa55ab48b4b7a8986181 | 2023-07-08 16:55:02 |\n",
      "| 3019     | Demo event for MISP              |             1.2.2.1              | 2023-07-11 12:48:00 |\n",
      "+----------+----------------------------------+----------------------------------+---------------------+ \n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(\"Search for \\033[92mattributes changed after a specific timestamp\\033[90m\")\n",
    "result = requests.post(\"{}/attributes/restSearch\".format(misp_url), headers=headers, verify=misp_verifycert,\n",
    "                           json={\"returnFormat\":\"json\", \"attribute_timestamp\":\"3d\"})\n",
    "table = eventstable([\"Event ID\", \"Event\", \"Value\", \"Attribute timestamp\"])\n",
    "if not \"errors\" in result.json():\n",
    "    for attribute in result.json().get(\"response\", []).get(\"Attribute\", []):\n",
    "        table.add_row([attribute[\"Event\"][\"id\"], attribute[\"Event\"][\"info\"], attribute[\"value\"], unix_timestamp_to_date(attribute[\"timestamp\"])])\n",
    "print(table.get_string(sortby=\"Attribute timestamp\"), \"\\n\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bb12f8bf-17da-45c8-88a2-941d2e5fa569",
   "metadata": {},
   "source": [
    "## AS:2 Search for attributes on their first seen or last seen value\n",
    "\n",
    "You can also search for first seen and last seen values of attributes. The first seen and last seen are set manually by the threat author.\n",
    "\n",
    "![helpers/timestamp_attribute_seen.png](helpers/timestamp_attribute_seen.png)\n",
    "\n",
    "When searching for attributes with a last seen or first seen value it makes sense to also only focus on those attributes that have been marked as actionable, meaning the **to_ids** flag is set to True.\n",
    "\n",
    "### Use PyMISP to search for attributes on their last seen or first seen value\n",
    "\n",
    "The parameters that refer to the last seen or first seen value in the PyMISP search function are **last_seen** and **first_seen**. You can combine these queries, or only filter for one of the values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "d4732408-de0d-48be-9180-b7741c74cbda",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Search for attributes with a \u001b[92mfirst seen or last seen timestamp\u001b[90m\n",
      "+----------+---------------------+---------+---------------------------+---------------------------+\n",
      "| Event ID | Event               |  Value  |    Attribute timestamp    |         First seen        |\n",
      "+----------+---------------------+---------+---------------------------+---------------------------+\n",
      "| 3019     | Demo event for MISP | 1.2.2.1 | 2023-07-11 12:48:00+00:00 | 2023-07-10 20:21:34+00:00 |\n",
      "+----------+---------------------+---------+---------------------------+---------------------------+ \n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(\"Search for attributes with a \\033[92mfirst seen or last seen timestamp\\033[90m\")\n",
    "attributes = misp.search(\"attributes\", first_seen=\"15d\", to_ids=True,  pythonify=True, include_context=True)\n",
    "table = eventstable([\"Event ID\", \"Event\", \"Value\", \"Attribute timestamp\", \"First seen\"])\n",
    "for attribute in attributes:\n",
    "    table.add_row([attribute.Event.id, attribute.Event.info, attribute.value, attribute.timestamp, attribute.first_seen])\n",
    "print(table.get_string(sortby=\"First seen\"), \"\\n\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7f6bd4e2-4144-44cd-bf9b-7a12e4f1926c",
   "metadata": {},
   "source": [
    "### Use REST API to search for attributes on their last seen or first seen value\n",
    "\n",
    "It should not come as a surprise that the REST API also uses **last_seen** and **first_seen** as search criteria. Important to remark is that the date returned by the REST API is not in UNIX timestamp format, but in ISO 8601 format (hence also the reason why we don't convert it to a human readable string with `unix_timestamp_to_date`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "7486e75f-1a49-4c68-9e5a-ed677876a159",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Search for attributes with a \u001b[92mfirst seen or last seen timestamp\u001b[90m\n",
      "+----------+---------------------+---------+---------------------+----------------------------------+\n",
      "| Event ID | Event               |  Value  | Attribute timestamp |       Attribute first seen       |\n",
      "+----------+---------------------+---------+---------------------+----------------------------------+\n",
      "| 3019     | Demo event for MISP | 1.2.2.1 | 2023-07-11 12:48:00 | 2023-07-10T20:21:34.000000+00:00 |\n",
      "+----------+---------------------+---------+---------------------+----------------------------------+ \n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(\"Search for attributes with a \\033[92mfirst seen or last seen timestamp\\033[90m\")\n",
    "result = requests.post(\"{}/attributes/restSearch\".format(misp_url), headers=headers, verify=misp_verifycert,\n",
    "                           json={\"returnFormat\":\"json\", \"first_seen\":\"10d\", \"to_ids\":\"1\"})\n",
    "table = eventstable([\"Event ID\", \"Event\", \"Value\", \"Attribute timestamp\", \"Attribute first seen\"])\n",
    "if not \"errors\" in result.json():\n",
    "    for attribute in result.json().get(\"response\", []).get(\"Attribute\", []):\n",
    "        table.add_row([attribute[\"Event\"][\"id\"], attribute[\"Event\"][\"info\"], attribute[\"value\"], unix_timestamp_to_date(attribute[\"timestamp\"]), attribute[\"first_seen\"]])\n",
    "print(table.get_string(sortby=\"Attribute timestamp\"), \"\\n\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3a329db0-e003-4896-93b9-b5a10b1f90a3",
   "metadata": {},
   "source": [
    "## AS:3 Search for attributes in events with a specific recent change or published timestamp\n",
    "\n",
    "So far we have searched for attributes and filtered on the timestamps associated with these attributes. MISP also allows you to search for attributes, and filter on the timestamps associated with the event in which these attributes are enclosed. Obviously you can also achieve this by searching for events and then working your way through the attributes (event per event), but in a lot of cases it's more convenient to search for the attributes, and use the event information  as an additional filter criteria.\n",
    "\n",
    "### Use PyMISP to search for attributes in events with a specific recent change or published timestamp\n",
    "\n",
    "In this case we have to set the controller to attributes, and then use **event_timestamp** as the filter to query for recently changed events. Likewise, if you want to filter for events with a recent published timestamp you can use the filter **publish_timestamp**. In this case we also include only the attributes that are actionable (to_ids is set to True)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "d7e9b821-bdd3-4db0-b149-35d8ba30cac1",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Search for \u001b[92mattributes\u001b[90m in events with a specific recent \u001b[92mtimestamp\u001b[90m change\n",
      "+----------+-----------------------+---------------------------+----------------------+---------------------------+\n",
      "| Event ID | Event                 |     Event last change     |        Value         |    Attribute timestamp    |\n",
      "+----------+-----------------------+---------------------------+----------------------+---------------------------+\n",
      "| 2662     | An older threat event | 2023-07-11 12:45:06+00:00 | www.doesnotexist.com | 2023-07-02 12:15:56+00:00 |\n",
      "| 2662     | An older threat event | 2023-07-11 12:45:06+00:00 |       23.2.2.2       | 2023-07-02 12:21:32+00:00 |\n",
      "| 2662     | An older threat event | 2023-07-11 12:45:06+00:00 |     29.28.27.26      | 2023-07-02 12:29:45+00:00 |\n",
      "| 2662     | An older threat event | 2023-07-11 12:45:06+00:00 |       1.2.3.4        | 2023-07-02 12:34:29+00:00 |\n",
      "| 3019     | Demo event for MISP   | 2023-07-11 12:48:00+00:00 |       1.2.2.1        | 2023-07-11 12:48:00+00:00 |\n",
      "+----------+-----------------------+---------------------------+----------------------+---------------------------+ \n",
      "\n",
      "\n",
      "Search for \u001b[92mattributes\u001b[90m in events with a specific recent \u001b[92mpublished timestamp\u001b[90m\n",
      "+----------+-----------------------+---------------------------+----------------------+---------------------------+\n",
      "| Event ID | Event                 |  Event publish timestamp  |        Value         |    Attribute timestamp    |\n",
      "+----------+-----------------------+---------------------------+----------------------+---------------------------+\n",
      "| 2662     | An older threat event | 2023-07-10 11:48:28+00:00 | www.doesnotexist.com | 2023-07-02 12:15:56+00:00 |\n",
      "| 2662     | An older threat event | 2023-07-10 11:48:28+00:00 |       23.2.2.2       | 2023-07-02 12:21:32+00:00 |\n",
      "| 2662     | An older threat event | 2023-07-10 11:48:28+00:00 |     29.28.27.26      | 2023-07-02 12:29:45+00:00 |\n",
      "| 2662     | An older threat event | 2023-07-10 11:48:28+00:00 |       1.2.3.4        | 2023-07-02 12:34:29+00:00 |\n",
      "| 3019     | Demo event for MISP   | 2023-07-11 12:48:03+00:00 |       1.2.2.1        | 2023-07-11 12:48:00+00:00 |\n",
      "+----------+-----------------------+---------------------------+----------------------+---------------------------+ \n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(\"Search for \\033[92mattributes\\033[90m in events with a specific recent \\033[92mtimestamp\\033[90m change\")\n",
    "attributes = misp.search(\"attributes\", event_timestamp=\"2d\", to_ids=True, pythonify=True, include_context=True)\n",
    "table = eventstable([\"Event ID\", \"Event\", \"Event last change\", \"Value\", \"Attribute timestamp\"])\n",
    "for attribute in attributes:\n",
    "    table.add_row([attribute.Event.id, attribute.Event.info, attribute.Event.timestamp, attribute.value, attribute.timestamp])\n",
    "print(table.get_string(sortby=\"Attribute timestamp\"), \"\\n\\n\")\n",
    "\n",
    "\n",
    "print(\"Search for \\033[92mattributes\\033[90m in events with a specific recent \\033[92mpublished timestamp\\033[90m\")\n",
    "attributes = misp.search(\"attributes\", publish_timestamp=\"2d\", to_ids=True, pythonify=True, include_context=True)\n",
    "table = eventstable([\"Event ID\", \"Event\", \"Event publish timestamp\", \"Value\", \"Attribute timestamp\"])\n",
    "for attribute in attributes:\n",
    "    table.add_row([attribute.Event.id, attribute.Event.info, attribute.Event.publish_timestamp, attribute.value, attribute.timestamp])\n",
    "print(table.get_string(sortby=\"Attribute timestamp\"), \"\\n\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0d46c72c-c3c3-4ab4-bef6-fd9b4122705e",
   "metadata": {},
   "source": [
    "### Use REST API to search for attributes in events with a specific recent change or published timestamp\n",
    "\n",
    "The REST API is similar as the interactions with PyMISP, but don't forget to use the attributes endpoint. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "972fcf90-f84e-4ebb-a5c2-6c2633be6242",
   "metadata": {},
   "source": [
    "## AS:4 Search for attributes in events with a specific date field\n",
    "\n",
    "Similarily as how you can search for events based on the **date** field you can also use this criteria to search for attributes part of events matching the **date_from** and / or **date_to** criteria.\n",
    "\n",
    "### Use PyMISP and the REST API to search for attributes in events with a specific date field"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "0a1bd125-4cd0-4671-b30c-efc058f611d2",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Search for \u001b[92mattributes\u001b[90m in events\u001b[92m after and before a specific date\u001b[90m\n",
      "+----------+----------------------------------+------------+----------------------------------+---------------------------+\n",
      "| Event ID | Event                            | Event date |              Value               |    Attribute timestamp    |\n",
      "+----------+----------------------------------+------------+----------------------------------+---------------------------+\n",
      "| 2964     | Minas – on the way to complexity | 2023-07-06 | 06fe9ab0b17f659486e3c3ace43f0e3a | 2023-05-21 13:30:40+00:00 |\n",
      "| 2964     | Minas – on the way to complexity | 2023-07-06 | 08da41489b4b68565dc77bb9acb1ecb4 | 2023-05-21 13:30:40+00:00 |\n",
      "| 2964     | Minas – on the way to complexity | 2023-07-06 | 63e0cd6475214c697c5fc115d40327b4 | 2023-05-21 13:30:40+00:00 |\n",
      "| 2964     | Minas – on the way to complexity | 2023-07-06 | f38a1b6b132afa55ab48b4b7a8986181 | 2023-07-08 16:55:02+00:00 |\n",
      "+----------+----------------------------------+------------+----------------------------------+---------------------------+ \n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(\"Search for \\033[92mattributes\\033[90m in events\\033[92m after and before a specific date\\033[90m\")\n",
    "attributes = misp.search(\"attributes\", date_from=\"2023-07-06\", date_to=\"2023-07-09\", to_ids=True,  pythonify=True, include_context=True)\n",
    "table = eventstable([\"Event ID\", \"Event\", \"Event date\", \"Value\", \"Attribute timestamp\"])\n",
    "for attribute in attributes:\n",
    "    table.add_row([attribute.Event.id, attribute.Event.info, attribute.Event.date, attribute.value, attribute.timestamp])\n",
    "print(table.get_string(sortby=\"Attribute timestamp\"), \"\\n\\n\") "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "606c1d63-27d8-490e-bf37-4b241899e516",
   "metadata": {},
   "source": [
    "## AS:5 Search for first change of an attribute\n",
    "\n",
    "We concluded the previous section on event searching with the *First recorded change* and this playbook has yet not provided an easy way to provide you this timestamp.\n",
    "\n",
    "![helpers/timestamp_firstchange.png](helpers/timestamp_firstchange.png)\n",
    "\n",
    "\n",
    "### Use PyMISP or REST API to search for the first change\n",
    "\n",
    "Unfortunately there isn't an easy way to have PyMISP or the REST API return a sorted list of attributes. In order to get to the first recorded change you have to\n",
    "- Initialise a variable on timestamp far ahead in time\n",
    "- Get all the attributes for a specific event\n",
    "    - We do not request them as Python objects, because we want only want to timestamp in *timestamp* format, not as a datetime object\n",
    "- Loop through all the attributes and whenever the timestamp value is lower, update our variable\n",
    "- In a second loop, attempt the above again, but now query for the objects in the event.\n",
    "- This will eventually get use our *First recorded change* value based on attribute and object changes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "0b29107b-4082-4fda-b4ac-a85a20859891",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Search for \u001b[92mfirst recorded change in an event\u001b[90m\n",
      "The first recorded change is: \u001b[92m2022-09-12 13:27:38\u001b[90m\n",
      "\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "event_id_to_search = \"2978\"\n",
    "\n",
    "print(\"Search for \\033[92mfirst recorded change in an event\\033[90m\")\n",
    "base_timestamp = 8551954799\n",
    "timestamp = base_timestamp  # The time is Thu Dec 31 2240 22:59:59 GMT+0000. We're using MISP v6.2.92-ish by then\n",
    "attributes = misp.search(\"attributes\", eventid=event_id_to_search)\n",
    "if len(attributes) > 0:\n",
    "    for attribute in attributes.get(\"Attribute\", []):\n",
    "        if int(attribute[\"timestamp\"]) < timestamp:\n",
    "            timestamp = int(attribute[\"timestamp\"])\n",
    "\n",
    "objects = misp.search(\"objects\", eventid=event_id_to_search)\n",
    "if len(objects) > 0:\n",
    "    for mispobject in objects:\n",
    "        if int(mispobject[\"Object\"][\"timestamp\"]) < timestamp:\n",
    "            timestamp = int(mispobject[\"Object\"][\"timestamp\"])\n",
    "        \n",
    "if timestamp < base_timestamp:     \n",
    "    print(\"The first recorded change is: \\033[92m{}\\033[90m\".format(unix_timestamp_to_date(timestamp)))\n",
    "else:\n",
    "    print(\"Unable to get the first recorded change.\")\n",
    "print(\"\\n\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "94712020-0926-4276-918b-820110abf51a",
   "metadata": {},
   "source": [
    "# Object search context\n",
    "\n",
    "## OS:1 Search for objects\n",
    "\n",
    "You already had a preview on how the PyMISP search function can be used to search for objects. It's as easy as telling the controller to use **objects**, and then use the same search criteria as you use for searching for attributes.\n",
    "\n",
    "### Use PyMISP to search for objects based on their last timestamp change\n",
    "\n",
    "A simple example demonstrate the use of PyMISP for searching of objects based on their last change timestamp with **attribute_timestamp**.\n",
    "\n",
    "There are some peculiar aspects when working with Objects contrary to Attributes\n",
    "- Contrary to with attributes, with `object.Event` you need to query for the dictionary value\n",
    "- There is no `object_timestamp`, instead use `attribute_timestamp`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "0f7ff989-210a-4df2-af47-3a9a6fe5a3b0",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Search for \u001b[92mobjects\u001b[90m with a recent \u001b[92mtimestamp change\u001b[90m\n",
      "+----------+--------------------------------------+----------------+--------------------------------------+-------------------+---------------------------+\n",
      "| Event ID | Event                                |  Object name   |             Object UUID              |      Comment      |         Timestamp         |\n",
      "+----------+--------------------------------------+----------------+--------------------------------------+-------------------+---------------------------+\n",
      "| 2662     | An older threat event                |   domain-ip    | 5e3fcb9c-04e4-4608-8b52-7483199d2bf6 |                   | 2023-07-02 12:15:56+00:00 |\n",
      "| 2662     | An older threat event                |   domain-ip    | d9cdf9a5-5111-4a94-a6b6-ce07148efcaa |     mycomment3    | 2023-07-02 12:34:29+00:00 |\n",
      "| 2995     | CVE investigation for CVE-2021-44228 | attack-pattern | 987c898a-c6f5-4d64-a190-173f687605c2 | From cve_advanced | 2023-07-05 18:28:44+00:00 |\n",
      "| 2995     | CVE investigation for CVE-2021-44228 |    weakness    | b2f4c820-90d5-485d-bf6a-3fd137d7c33c | From cve_advanced | 2023-07-05 18:28:44+00:00 |\n",
      "| 2995     | CVE investigation for CVE-2021-44228 | vulnerability  | 5fa36ead-b095-4de2-befe-032e49c0751b | From cve_advanced | 2023-07-05 18:28:45+00:00 |\n",
      "| 2995     | CVE investigation for CVE-2021-44228 | attack-pattern | 9835aad1-95a6-4730-b121-11b181a5e8b6 | From cve_advanced | 2023-07-05 18:30:09+00:00 |\n",
      "| 2995     | CVE investigation for CVE-2021-44228 | vulnerability  | a5d7ad06-fc54-47f9-ae49-ba1aaa597eef | From cve_advanced | 2023-07-05 18:30:09+00:00 |\n",
      "| 2995     | CVE investigation for CVE-2021-44228 |    weakness    | dbb7a7a1-16bf-485b-b3f6-54303201d5bc | From cve_advanced | 2023-07-05 18:30:09+00:00 |\n",
      "| 2995     | CVE investigation for CVE-2021-44228 | attack-pattern | f944333d-8694-4324-b46d-067d7c7924d4 | From cve_advanced | 2023-07-05 18:31:06+00:00 |\n",
      "| 2995     | CVE investigation for CVE-2021-44228 | vulnerability  | 7403a435-28ab-45aa-a0ff-934e70b35ad2 | From cve_advanced | 2023-07-05 18:31:06+00:00 |\n",
      "| 2995     | CVE investigation for CVE-2021-44228 |    weakness    | aa794648-2abe-401c-97c9-31d866c73702 | From cve_advanced | 2023-07-05 18:31:06+00:00 |\n",
      "+----------+--------------------------------------+----------------+--------------------------------------+-------------------+---------------------------+ \n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(\"Search for \\033[92mobjects\\033[90m with a recent \\033[92mtimestamp change\\033[90m\")\n",
    "objects = misp.search(\"objects\", attribute_timestamp=\"10d\", pythonify=True, include_context=True)\n",
    "table = eventstable([\"Event ID\", \"Event\", \"Object name\", \"Object UUID\", \"Comment\", \"Timestamp\"])\n",
    "for mispobject in objects:\n",
    "    table.add_row([mispobject.Event[\"id\"], mispobject.Event[\"info\"], mispobject.name, mispobject.uuid, mispobject.comment, mispobject.timestamp])\n",
    "print(table.get_string(sortby=\"Timestamp\"), \"\\n\\n\") "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fa9a2c17-d387-4851-b1ce-9a7f5efd3ccf",
   "metadata": {},
   "source": [
    "# Sightings\n",
    "\n",
    "A [sighting](https://www.misp-standard.org/rfc/misp-standard-core.html#name-sighting) is an ascertainment which describes whether an attribute has been seen under a given set of conditions. The sighting can include the organisation who sighted the attribute or can be anonymised. \n",
    "\n",
    "## SI:1 Adding sightings\n",
    "\n",
    "When you add sightings you can choose to add the sighting to one specific attribute in one event or add a *global* sighting. The latter means that the sighting is added to the attribute, regardless in which event the attribute is found. When you add sightings you can specify the **date_sighting**, when the referenced attribute is sighted. This date_sighting is expressed in seconds (decimal) elapsed since 1st of January 1970 (Unix timestamp).\n",
    "\n",
    "### Use PyMISP to add sightings\n",
    "\n",
    "In PyMISP you can use the function **add_sighting** and the supply a JSON describing the sighting. As you can learn from the example, you can supply the sighting date in a human readable format (ISO 8601) or as a UNIX timestamp."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "191ba7c9-e129-4434-8271-b52e2739ceeb",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Adding sightings for \u001b[92m167.89.16.17\u001b[90m\n",
      "\n",
      "\n",
      "{'Sighting': {'id': '32', 'attribute_id': '2606163', 'event_id': '2656', 'org_id': '1', 'date_sighting': '1689079721', 'uuid': 'feec5ba6-7059-4b09-b173-21fe165a1208', 'source': 'MISP playbook - human readable date', 'type': '0'}}\n",
      "{'Sighting': {'id': '36', 'attribute_id': '2606163', 'event_id': '2656', 'org_id': '1', 'date_sighting': '1689079721', 'uuid': '98d9fe7e-20b1-467f-b01a-6354a705b2d3', 'source': 'MISP playbook - UNIX timestamp', 'type': '0'}}\n"
     ]
    }
   ],
   "source": [
    "attribute_value = \"167.89.16.17\"\n",
    "attribute_uuid = \"5650a1bf-c25a-4e0a-a45a-f9dc10c0b6ff\"\n",
    "\n",
    "print(\"Adding sightings for \\033[92m{}\\033[90m\\n\\n\".format(attribute_value))\n",
    "date_human_readable = \"2023-07-11\"\n",
    "date_unix_timestamp = 1689067634\n",
    "print(misp.add_sighting({\"value\":\"167.89.16.17\", \"attribute_uuid\":attribute_uuid, \"date_sighting\":date_human_readable, \"source\":\"MISP playbook - human readable date\"}))\n",
    "print(misp.add_sighting({\"value\":\"167.89.16.17\", \"attribute_uuid\":attribute_uuid, \"date_sighting\":date_unix_timestamp, \"source\":\"MISP playbook - UNIX timestamp\"}))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6e92f900-5c29-4c3b-b2a4-ad77e7799de1",
   "metadata": {},
   "source": [
    "## SI:2 Search for sightings\n",
    "\n",
    "Unfortunately in MISP it's not possible to search for a list of \"recent\" sightings. What's more, adding a sighting to an attribute does not change the last update timestamp of the event or the attribute. There are two ways though to search (or process) sightings, either query for them based on an event ID, or have them included in the results when you search for attributes.\n",
    "\n",
    "### Use PyMISP to search for sightings\n",
    "\n",
    "The **search_sightings** function of PyMISP allows you to search for sightings. You can filter the results for attributes, events or the type of sighting. What concerns timestamps you can filter the search results on sightings in events published after a specific date (with **publish_timestamp**) or for events that have their date set before (with **date_to**) or after (with **date_from**) a given date.\n",
    "\n",
    "The returned timestamp is in UNIX timestamp format, that is why we transform it to a human readable format with `unix_timestamp_to_date`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "ffc5f2fd-33be-4195-b5d2-7d375e46ae9c",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Search for \u001b[92msightings\u001b[90m in a specific event\n",
      "+-----------------+--------------+-------------------------------------+---------------+---------------------+\n",
      "| Attribute value | Attribute ID |           Sighting source           | Sighting type |  Sighting timestamp |\n",
      "+-----------------+--------------+-------------------------------------+---------------+---------------------+\n",
      "|   167.89.16.17  |   2606141    |    MISP playbook - UNIX timestamp   |       0       | 2023-07-11 12:48:41 |\n",
      "|   167.89.16.17  |   2606141    | MISP playbook - human readable date |       0       | 2023-07-11 12:48:41 |\n",
      "|   167.89.16.17  |   2606146    |    MISP playbook - UNIX timestamp   |       0       | 2023-07-11 12:48:41 |\n",
      "|   167.89.16.17  |   2606146    | MISP playbook - human readable date |       0       | 2023-07-11 12:48:41 |\n",
      "+-----------------+--------------+-------------------------------------+---------------+---------------------+ \n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "event_id_to_search = 2655\n",
    "\n",
    "print(\"Search for \\033[92msightings\\033[90m in a specific event\")\n",
    "result = misp.search_sightings(context=\"event\", context_id=event_id_to_search, include_attribute=True, include_event_meta=True)\n",
    "table = eventstable([\"Attribute value\", \"Attribute ID\", \"Sighting source\", \"Sighting type\", \"Sighting timestamp\"])\n",
    "for sighting in result:\n",
    "    table.add_row([sighting[\"Sighting\"][\"value\"], sighting[\"Sighting\"][\"attribute_id\"], sighting[\"Sighting\"][\"source\"], sighting[\"Sighting\"][\"type\"], unix_timestamp_to_date(sighting[\"Sighting\"][\"date_sighting\"])])\n",
    "print(table.get_string(sortby=\"Sighting timestamp\"), \"\\n\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d1a501d8-7eb6-4383-a010-65fa96335caa",
   "metadata": {},
   "source": [
    "### Use the REST API to process sightings timestamps with the attributes or event endpoint\n",
    "\n",
    "If you process attributes (or events) you can check for the presence of sightings by setting the parameter **includeSightings** to True. If there are sightings, then you can use the timestamp of the sighting as the reference to when the sighting was made. \n",
    "\n",
    "The returned timestamp is in UNIX timestamp format, that is why we transform it to a human readable format with `unix_timestamp_to_date`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "ac9d3b9f-cb2f-45a3-9d46-aea618049636",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Search for attributes in a specific event and then check if there are \u001b[92msightings\u001b[90m\n",
      "+----------+---------------------+-----------------+---------------------+-------------------------------------+---------------+---------------------+\n",
      "| Event ID | Event               | Attribute value | Attribute timestamp |           Sighting source           | Sighting type |  Sighting timestamp |\n",
      "+----------+---------------------+-----------------+---------------------+-------------------------------------+---------------+---------------------+\n",
      "| 2655     | Demo Event Phishing |   167.89.16.17  | 2023-05-05 18:59:04 |    MISP playbook - UNIX timestamp   |       0       | 2023-07-11 12:48:41 |\n",
      "| 2655     | Demo Event Phishing |   167.89.16.17  | 2023-05-05 18:59:04 |    MISP playbook - UNIX timestamp   |       0       | 2023-07-11 12:48:41 |\n",
      "| 2655     | Demo Event Phishing |   167.89.16.17  | 2023-05-05 18:59:04 | MISP playbook - human readable date |       0       | 2023-07-11 12:48:41 |\n",
      "| 2655     | Demo Event Phishing |   167.89.16.17  | 2023-05-05 18:59:04 | MISP playbook - human readable date |       0       | 2023-07-11 12:48:41 |\n",
      "+----------+---------------------+-----------------+---------------------+-------------------------------------+---------------+---------------------+ \n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "event_id_to_search = \"2655\"\n",
    "\n",
    "\n",
    "print(\"Search for attributes in a specific event and then check if there are \\033[92msightings\\033[90m\")\n",
    "result = requests.post(\"{}/attributes/restSearch\".format(misp_url), headers=headers, verify=misp_verifycert,\n",
    "                           json={\"returnFormat\":\"json\", \"includeSightings\":1, \"includeContext\":1, \"eventid\":event_id_to_search})\n",
    "table = eventstable([\"Event ID\", \"Event\", \"Attribute value\", \"Attribute timestamp\", \"Sighting source\", \"Sighting type\", \"Sighting timestamp\"])\n",
    "if not \"errors\" in result.json():    \n",
    "    for attribute in result.json().get(\"response\", []).get(\"Attribute\", []):\n",
    "        if len(attribute.get(\"Sighting\", [])) > 0:\n",
    "            for sighting in attribute[\"Sighting\"]:\n",
    "                table.add_row([attribute[\"Event\"][\"id\"], attribute[\"Event\"][\"info\"], attribute[\"value\"], unix_timestamp_to_date(attribute[\"timestamp\"]), sighting[\"source\"], sighting[\"type\"], unix_timestamp_to_date(sighting[\"date_sighting\"])])\n",
    "else:\n",
    "    print(result.text())\n",
    "print(table.get_string(sortby=\"Sighting timestamp\"), \"\\n\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9061e755-a377-466d-8f96-2d5635e01f34",
   "metadata": {},
   "source": [
    "# Proposal / ShadowAttributes\n",
    "\n",
    "Proposals or [ShadowAttributes](https://www.misp-standard.org/rfc/misp-standard-core.html#name-shadowattribute-2) are 3rd party created attributes that either propose to add new information to an event or modify existing information. They are not meant to be actionable until the event creator accepts them - at which point they will be converted into attributes or modify an existing attribute. They are similar in structure to Attributes but additionally carry a reference to the creator of the ShadowAttribute as well as a revocation flag.\n",
    "\n",
    "## PS1: Search for proposals or shadowattributes\n",
    "\n",
    "Unfortunately it's not possible to search for a list of \"recent\" proposals. What's more, a proposal to an event, does not change the last update timestamp of the event or the attribute.\n",
    "\n",
    "### Use PyMISP to search for proposals\n",
    "\n",
    "The PyMISP function **attribute_proposals** searches for proposals in a specific event."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "552d0a15-7c84-42d1-a604-148079678f19",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[{'ShadowAttribute': {'id': '5', 'old_id': '2606144', 'event_id': '2655', 'type': 'email-src', 'category': 'Payload delivery', 'uuid': 'a8278fee-7ef9-49bb-ae3e-0e0a61577bbd', 'to_ids': True, 'comment': '', 'org_id': '1', 'timestamp': '1689011497', 'first_seen': None, 'last_seen': None, 'value': 'abuse@tafmail.com', 'org_uuid': '5c1eb4f8-bae5-45f5-a772-06d4a3f64c3e', 'old_uuid': 'a8278fee-7ef9-49bb-ae3e-0e0a61577bbd', 'event_uuid': 'fb4a934f-4694-406c-a26d-d148e881f875', 'Org': {'name': 'DEMO-ORG', 'uuid': '5c1eb4f8-bae5-45f5-a772-06d4a3f64c3e', 'id': '1'}}}]\n"
     ]
    }
   ],
   "source": [
    "event_id_to_search = 2655\n",
    "\n",
    "print(misp.attribute_proposals(event_id_to_search))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "692908fa-37ac-48b7-a8f6-c4c0d7e95e7d",
   "metadata": {},
   "source": [
    "### Use the REST API to process proposal timestamps\n",
    "\n",
    "If you process attributes (or events) you can check for the presence of proposals by setting the parameter **includeProposals** to True. If there are proposals, then you can use the timestamp of the proposal as the reference to when the proposal was made.\n",
    "\n",
    "The returned timestamp is in UNIX timestamp format, that is why we transform it to a human readable format with `unix_timestamp_to_date`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "17f96783-a463-4335-b0fb-b729df1a0ce4",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Search for attributes in a specific event and then check if there are \u001b[92mproposals\u001b[90m\n",
      "+----------+---------------------+----------------------+---------------------+-------------------+---------------------+\n",
      "| Event ID | Event               |   Attribute value    | Attribute timestamp |   Proposal value  |  Proposal timestamp |\n",
      "+----------+---------------------+----------------------+---------------------+-------------------+---------------------+\n",
      "| 2655     | Demo Event Phishing | no-reply@tafmail.com | 2023-05-05 18:59:04 | abuse@tafmail.com | 2023-07-10 17:51:37 |\n",
      "+----------+---------------------+----------------------+---------------------+-------------------+---------------------+ \n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "event_id_to_search = \"2655\"\n",
    "\n",
    "\n",
    "print(\"Search for attributes in a specific event and then check if there are \\033[92mproposals\\033[90m\")\n",
    "result = requests.post(\"{}/attributes/restSearch\".format(misp_url), headers=headers, verify=misp_verifycert,\n",
    "                           json={\"returnFormat\":\"json\", \"includeProposals\":1, \"includeContext\":1, \"eventid\":event_id_to_search})\n",
    "table = eventstable([\"Event ID\", \"Event\", \"Attribute value\", \"Attribute timestamp\", \"Proposal value\", \"Proposal timestamp\"])\n",
    "if not \"errors\" in result.json():    \n",
    "    for attribute in result.json().get(\"response\", []).get(\"Attribute\", []):\n",
    "        if len(attribute.get(\"ShadowAttribute\", [])) > 0:\n",
    "            for shadow in attribute[\"ShadowAttribute\"]:\n",
    "                table.add_row([attribute[\"Event\"][\"id\"], attribute[\"Event\"][\"info\"], attribute[\"value\"], unix_timestamp_to_date(attribute[\"timestamp\"]), shadow[\"value\"], unix_timestamp_to_date(shadow[\"timestamp\"])])\n",
    "else:\n",
    "    print(result.text())\n",
    "print(table.get_string(sortby=\"Proposal timestamp\"), \"\\n\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "498b8eff-7d1b-47b7-90c4-e13cc738b36c",
   "metadata": {},
   "source": [
    "# Closure \n",
    "## EN:1 End of the playbook "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "4c03bb30",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[92m End of the playbook\n"
     ]
    }
   ],
   "source": [
    "print(\"\\033[92m End of the playbook\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b17af9f2",
   "metadata": {},
   "source": [
    "## External references \n",
    "\n",
    "- [The MISP Project](https://www.misp-project.org/)\n",
    "- [MISP Cheat Sheet](https://www.misp-project.org/misp-training/cheatsheet.pdf)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e1c813f6",
   "metadata": {},
   "source": [
    "## Technical details \n",
    "\n",
    "### Documentation\n",
    "\n",
    "This playbook requires these Python **libraries** to exist in the environment where the playbook is executed. You can install them with `pip install <library>`.\n",
    "\n",
    "```\n",
    "PrettyTable\n",
    "```\n",
    "\n",
    "You need to have **network access** to \n",
    "- your MISP server (HTTP or HTTPS)\n",
    "\n",
    "You need\n",
    "- an **API key with MISP**\n",
    "- - Under Global Actions, My Profile. Add an extra authentication key.\n",
    "- - Add the API key (`misp_key`) and the MISP URL (`misp_url`) to `keys.py`\n",
    "- - If you use a self-signed certificate set `misp_verifycert` to False"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "50ad2ff2-0df9-42ae-b839-b4682cde1078",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "celltoolbar": "Attachments",
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  },
  "toc-showcode": false,
  "toc-showmarkdowntxt": false
 },
 "nbformat": 4,
 "nbformat_minor": 5
}