examples/prensor_playground.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PHuwKiTF02Zq"
      },
      "source": [
        "# Your structured data into Tensorflow.\n",
        "\n",
        "ML training often expects _flat_ data, like a line in a CSV.\n",
        "[tf.Example](https://www.tensorflow.org/api_docs/python/tf/train/Example) was\n",
        "designed to represent flat data. But the data you care about and want to predict\n",
        "things about usually starts out _structured_.\n",
        "\n",
        "Over and over again you have to write transform code that turns your structured data into Tensors. This repetitive transform code must be rewritten over and over for all your ML pipelines both for training _and_ serving! And it lets bugs slip into your ML pipeline. \n",
        "\n",
        "`struct2tensor` lets you take advantage of structured data _within_ your ML pipelines. It is:\n",
        "\n",
        "* **for**: ML Engineers \n",
        "* **who**: train models on data that starts out structured\n",
        "* **it is**: a python library \n",
        "* **that**: transforms your structured data into model-friendly (Sparse, Raggged, Dense, ...) tensors hermetically _within_ your model\n",
        "* **unlike**: writing custom transforms over and over for training and serving.\n",
        "\n",
        "\n",
        "---\n",
        "\n",
        "\n",
        "![struct2tensor diagram showing the transform happens in the model](https://imgur.com/aqOX7nS.png)\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7ozCmoF33ogj"
      },
      "source": [
        "# Demo example\n",
        "\n",
        "Suppose we have this _structured_ data we want to train on. The source example data format is a [protobuff](https://developers.google.com/protocol-buffers). `struct2tensor` was built internally and works on protobuffers now. It can be extended to parquet, json, etc. in the future.\n",
        "\n",
        "```\n",
        "# e.g. a web session\n",
        "message Session{\n",
        "  message SessionInfo {\n",
        "    string session_feature = 1;\n",
        "    double session_duration_sec = 2;\n",
        "  }\n",
        "  SessionInfo session_info = 1;\n",
        "  message Event {\n",
        "    string query = 1;\n",
        "    message Action {\n",
        "      int number_of_views = 1;\n",
        "    }\n",
        "    repeated Action action = 2;\n",
        "  }\n",
        "  repeated Event event = 2;\n",
        "}\n",
        "```\n",
        "\n",
        "\n",
        "In 3 steps we'll extract the fields we want with `struct2tensor`. We'll end up with batch-aligned `SparseTensors`:\n",
        "\n",
        "1. Tell our model what examples we care about, e.g.  **`event`** (submessage `Session::Event`).\n",
        "2. Pick the proto fields that we think are good features, say:\n",
        "  * `session_info.session_feature`\n",
        "  * `event.query`\n",
        "3. Identify the label to predict, say  **`event.action.number_of_views`**  (the actual label could be sum(action.number_of_views for action in event))\n",
        "\n",
        "\n",
        "Then we can build a struct2tensor query that:\n",
        "* parses instances of this protocol buffer\n",
        "* transforms the fields we care about\n",
        "* creates the necessary `SparseTensor`s\n",
        "\n",
        "Don't worry about some of these terms yet. We'll show you an example. And then explain the terms later."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "w2RkJ6mN2Y6-"
      },
      "source": [
        "## Install required packages (internal colab users: skip)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 461
        },
        "id": "q3REXR58msJe",
        "outputId": "9b3a6130-83ce-46af-fe61-44a8cb95b4d8"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Requirement already satisfied: struct2tensor in /usr/local/lib/python3.6/dist-packages (0.0.1.dev6)\n",
            "Requirement already satisfied: tensorflow-metadata\u003e=0.13.0 in /usr/local/lib/python3.6/dist-packages (from struct2tensor) (0.15.1)\n",
            "Requirement already satisfied: protobuf\u003e=3.8.0 in /usr/local/lib/python3.6/dist-packages (from struct2tensor) (3.10.0)\n",
            "Requirement already satisfied: tensorflow==1.15.0 in /usr/local/lib/python3.6/dist-packages (from struct2tensor) (1.15.0)\n",
            "Requirement already satisfied: googleapis-common-protos in /usr/local/lib/python3.6/dist-packages (from tensorflow-metadata\u003e=0.13.0-\u003estruct2tensor) (1.6.0)\n",
            "Requirement already satisfied: six\u003e=1.9 in /usr/local/lib/python3.6/dist-packages (from protobuf\u003e=3.8.0-\u003estruct2tensor) (1.12.0)\n",
            "Requirement already satisfied: setuptools in /usr/local/lib/python3.6/dist-packages (from protobuf\u003e=3.8.0-\u003estruct2tensor) (42.0.1)\n",
            "Requirement already satisfied: numpy\u003c2.0,\u003e=1.16.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow==1.15.0-\u003estruct2tensor) (1.17.4)\n",
            "Requirement already satisfied: opt-einsum\u003e=2.3.2 in /usr/local/lib/python3.6/dist-packages (from tensorflow==1.15.0-\u003estruct2tensor) (3.1.0)\n",
            "Requirement already satisfied: keras-preprocessing\u003e=1.0.5 in /usr/local/lib/python3.6/dist-packages (from tensorflow==1.15.0-\u003estruct2tensor) (1.1.0)\n",
            "Requirement already satisfied: termcolor\u003e=1.1.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow==1.15.0-\u003estruct2tensor) (1.1.0)\n",
            "Requirement already satisfied: wheel\u003e=0.26 in /usr/local/lib/python3.6/dist-packages (from tensorflow==1.15.0-\u003estruct2tensor) (0.33.6)\n",
            "Requirement already satisfied: grpcio\u003e=1.8.6 in /usr/local/lib/python3.6/dist-packages (from tensorflow==1.15.0-\u003estruct2tensor) (1.15.0)\n",
            "Requirement already satisfied: tensorboard\u003c1.16.0,\u003e=1.15.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow==1.15.0-\u003estruct2tensor) (1.15.0)\n",
            "Requirement already satisfied: keras-applications\u003e=1.0.8 in /usr/local/lib/python3.6/dist-packages (from tensorflow==1.15.0-\u003estruct2tensor) (1.0.8)\n",
            "Requirement already satisfied: tensorflow-estimator==1.15.1 in /usr/local/lib/python3.6/dist-packages (from tensorflow==1.15.0-\u003estruct2tensor) (1.15.1)\n",
            "Requirement already satisfied: google-pasta\u003e=0.1.6 in /usr/local/lib/python3.6/dist-packages (from tensorflow==1.15.0-\u003estruct2tensor) (0.1.8)\n",
            "Requirement already satisfied: gast==0.2.2 in /usr/local/lib/python3.6/dist-packages (from tensorflow==1.15.0-\u003estruct2tensor) (0.2.2)\n",
            "Requirement already satisfied: wrapt\u003e=1.11.1 in /usr/local/lib/python3.6/dist-packages (from tensorflow==1.15.0-\u003estruct2tensor) (1.11.2)\n",
            "Requirement already satisfied: absl-py\u003e=0.7.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow==1.15.0-\u003estruct2tensor) (0.8.1)\n",
            "Requirement already satisfied: astor\u003e=0.6.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow==1.15.0-\u003estruct2tensor) (0.8.0)\n",
            "Requirement already satisfied: werkzeug\u003e=0.11.15 in /usr/local/lib/python3.6/dist-packages (from tensorboard\u003c1.16.0,\u003e=1.15.0-\u003etensorflow==1.15.0-\u003estruct2tensor) (0.16.0)\n",
            "Requirement already satisfied: markdown\u003e=2.6.8 in /usr/local/lib/python3.6/dist-packages (from tensorboard\u003c1.16.0,\u003e=1.15.0-\u003etensorflow==1.15.0-\u003estruct2tensor) (3.1.1)\n",
            "Requirement already satisfied: h5py in /usr/local/lib/python3.6/dist-packages (from keras-applications\u003e=1.0.8-\u003etensorflow==1.15.0-\u003estruct2tensor) (2.8.0)\n",
            "Requirement already satisfied: graphviz in /usr/local/lib/python3.6/dist-packages (0.10.1)\n"
          ]
        }
      ],
      "source": [
        "#@test {\"skip\": true} \n",
        "# install struct2tensor\n",
        "!pip install struct2tensor\n",
        "# graphviz for pretty output\n",
        "!pip install graphviz"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dIxHSM3VQfUu"
      },
      "source": [
        "## Some Pretty Printing and Imports\n",
        "\n",
        "(not the \"real\" work yet)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "code",
        "colab": {
          "height": 34
        },
        "executionInfo": {
          "elapsed": 437,
          "status": "ok",
          "timestamp": 1600375610067,
          "user": {
            "displayName": "",
            "photoUrl": "",
            "userId": ""
          },
          "user_tz": 420
        },
        "id": "lc5KF8MILYrS",
        "outputId": "5b8c3534-db19-4f98-cc12-f44716bc402a",
        "scrolled": true
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "type-specific pretty printing ready to go\n"
          ]
        }
      ],
      "source": [
        "import base64\n",
        "import numpy as np\n",
        "import pprint\n",
        "import os\n",
        "import tensorflow\n",
        "from graphviz import Source\n",
        "\n",
        "import tensorflow as tf\n",
        "\n",
        "from IPython.display import Image\n",
        "from IPython.lib import pretty\n",
        "\n",
        "import struct2tensor as s2t\n",
        "from struct2tensor.test import test_pb2\n",
        "from google.protobuf import text_format\n",
        "\n",
        "\n",
        "def _display(graph):\n",
        "  \"\"\"Renders a graphviz digraph.\"\"\"\n",
        "  s = Source(graph)\n",
        "  s.format='svg'\n",
        "  return s\n",
        "    \n",
        "\n",
        "def _create_query_from_text_sessions(text_sessions):\n",
        "  \"\"\"Creates a struct2tensor query from a list of pbtxt of struct2tensor.test.Session.\"\"\"\n",
        "  sessions = tf.constant([\n",
        "      text_format.Merge(\n",
        "          text_session, \n",
        "          test_pb2.Session()\n",
        "      ).SerializeToString()\n",
        "      for text_session in text_sessions\n",
        "  ])\n",
        "  return s2t.create_expression_from_proto(\n",
        "    sessions, test_pb2.Session.DESCRIPTOR)\n",
        "\n",
        "def _prensor_pretty_printer(prensor, p, cycle):\n",
        "  \"\"\"Pretty printing function for struct2tensor.prensor.Prensor\"\"\"\n",
        "  pretty.pprint(prensor.get_sparse_tensors())\n",
        "\n",
        "def _sp_pretty_printer(sp, p, cycle):\n",
        "  \"\"\"Pretty printing function for SparseTensor.\"\"\"\n",
        "\n",
        "  del cycle\n",
        "  p.begin_group(4, \"SparseTensor(\")\n",
        "  p.text(\"values={}, \".format(sp.values.numpy().tolist()))\n",
        "  p.text(\"dense_shape={}, \".format(sp.dense_shape.numpy().tolist()))\n",
        "  p.break_()\n",
        "  p.text(\"indices={}\".format(sp.indices.numpy().tolist()))\n",
        "  p.end_group(4, \")\")\n",
        "\n",
        "\n",
        "pretty.for_type(tf.SparseTensor, _sp_pretty_printer)\n",
        "pretty.for_type(s2t.Prensor, _prensor_pretty_printer)\n",
        "\n",
        "_pretty_print = pretty.pprint\n",
        "\n",
        "print(\"type-specific pretty printing ready to go\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sEsOWNuz7jC7"
      },
      "source": [
        "## The real work:\n",
        "\n",
        "A function that parses our structured data (protobuffers) into tensors:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "height": 34
        },
        "executionInfo": {
          "elapsed": 378,
          "status": "ok",
          "timestamp": 1600375613528,
          "user": {
            "displayName": "",
            "photoUrl": "",
            "userId": ""
          },
          "user_tz": 420
        },
        "id": "ZC-oUzvBoPjA",
        "outputId": "7a19c4e5-0cb1-479d-9245-e302a932448e"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Defined the workhorse func: (structured data at rest) -\u003e (tensors)\n"
          ]
        }
      ],
      "source": [
        "@tf.function(input_signature=[tf.TensorSpec(shape=(None), dtype=tf.string)], autograph=False)\n",
        "def parse_session(serialized_sessions):\n",
        "  \"\"\"A TF function parsing a batch of serialized Session protos into tensors.\n",
        "\n",
        "  It is a TF graph that takes one 1-D tensor as input, and outputs a\n",
        "  Dict[str, tf.SparseTensor]\n",
        "  \"\"\"\n",
        "  query = s2t.create_expression_from_proto(\n",
        "      serialized_sessions, test_pb2.Session.DESCRIPTOR)\n",
        "  # Move all the fields of our interest to under \"event\".  \n",
        "  query = query.promote_and_broadcast({\n",
        "          \"session_feature\": \"session_info.session_feature\",\n",
        "          \"action_number_of_views\": \"event.action.number_of_views\" },\n",
        "          \"event\")\n",
        "  # Specify \"event\" to be examples.\n",
        "  query = query.reroot(\"event\")\n",
        "  # Extract all the fields of our interest.\n",
        "  projection = query.project([\"session_feature\", \"query\", \"action_number_of_views\"]) \n",
        "  prensors = s2t.calculate_prensors([projection])\n",
        "    \n",
        "  output_sparse_tensors = {}\n",
        "  for prensor in prensors:\n",
        "    path_to_tensor = prensor.get_sparse_tensors()\n",
        "    output_sparse_tensors.update({str(k): v for k, v in path_to_tensor.items()})\n",
        "  \n",
        "  return output_sparse_tensors\n",
        "\n",
        "print(\"Defined the workhorse func: (structured data at rest) -\u003e (tensors)\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VKp4SxTgPzpe"
      },
      "source": [
        "## Lets see it in action:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "height": 119
        },
        "executionInfo": {
          "elapsed": 569,
          "status": "ok",
          "timestamp": 1600375616071,
          "user": {
            "displayName": "",
            "photoUrl": "",
            "userId": ""
          },
          "user_tz": 420
        },
        "id": "-cIlFypdPeZX",
        "outputId": "7840d783-249c-4be2-d675-10f8e817dded"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "{'action_number_of_views': SparseTensor(values=[1, 2, 3], dense_shape=[2, 2], \n",
            "     indices=[[0, 0], [1, 0], [1, 1]]),\n",
            " 'query': SparseTensor(values=[b'Hello', b'world'], dense_shape=[2], \n",
            "     indices=[[0], [1]]),\n",
            " 'session_feature': SparseTensor(values=[b'foo', b'foo'], dense_shape=[2, 1], \n",
            "     indices=[[0, 0], [1, 0]])}\n"
          ]
        }
      ],
      "source": [
        "serialized_sessions = tf.constant([\n",
        "    text_format.Merge(\n",
        "        \"\"\"\n",
        "        session_info {\n",
        "          session_duration_sec: 1.0\n",
        "          session_feature: \"foo\"\n",
        "        }\n",
        "        event {\n",
        "          query: \"Hello\"\n",
        "          action {\n",
        "            number_of_views: 1\n",
        "          }\n",
        "          action {\n",
        "          }\n",
        "        }\n",
        "        event {\n",
        "          query: \"world\"\n",
        "          action {\n",
        "            number_of_views: 2\n",
        "          }\n",
        "          action {\n",
        "            number_of_views: 3\n",
        "          }\n",
        "        }\n",
        "        \"\"\",\n",
        "        test_pb2.Session()\n",
        "    ).SerializeToString()\n",
        "])\n",
        "\n",
        "_pretty_print(parse_session(serialized_sessions))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pW9zBjNMWMIH"
      },
      "source": [
        "See how we went from our pre-pipeline data (the Protobuffer) all the way to the structured data, packed into `SparseTensor`s?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QY7wy_6loF4v"
      },
      "source": [
        "# Digging Far Deeper\n",
        "Interested and want to learn more? Read on...\n",
        "\n",
        "Let's define several terms we mentioned before:\n",
        "\n",
        "### Prensor\n",
        "\n",
        "A Prensor (protobuffer + tensor) is a data structure storing the data we work on. We use protobuffers a lot at Google. `struct2tensor` can support other structured formats, too.\n",
        "\n",
        "For example, throughout this colab we will be using proto\n",
        "[`struct2tensor.test.Session`](http://cs/symbol:struct2tensor.test.Session). A schematic visualization\n",
        "of a selected part of the prensor from that proto looks like:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 543
        },
        "id": "ZTjNwx4bBXFk",
        "outputId": "4927fdf8-0d2c-46e6-ee2f-a1b2bf4a298f"
      },
      "outputs": [
        {
          "data": {
            "image/svg+xml": "\u003c?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?\u003e\n\u003c!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\"\u003e\n\u003c!-- Generated by graphviz version 2.40.1 (20161225.0304)\n --\u003e\n\u003c!-- Title: %3 Pages: 1 --\u003e\n\u003csvg width=\"288pt\" height=\"392pt\"\n viewBox=\"0.00 0.00 288.04 392.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\"\u003e\n\u003cg id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 388)\"\u003e\n\u003ctitle\u003e%3\u003c/title\u003e\n\u003cpolygon fill=\"#ffffff\" stroke=\"transparent\" points=\"-4,4 -4,-388 284.0386,-388 284.0386,4 -4,4\"/\u003e\n\u003c!-- root --\u003e\n\u003cg id=\"node1\" class=\"node\"\u003e\n\u003ctitle\u003eroot\u003c/title\u003e\n\u003cellipse fill=\"none\" stroke=\"#000000\" cx=\"181.643\" cy=\"-366\" rx=\"27\" ry=\"18\"/\u003e\n\u003ctext text-anchor=\"middle\" x=\"181.643\" y=\"-362.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\"\u003eroot\u003c/text\u003e\n\u003c/g\u003e\n\u003c!-- session --\u003e\n\u003cg id=\"node2\" class=\"node\"\u003e\n\u003ctitle\u003esession\u003c/title\u003e\n\u003cellipse fill=\"none\" stroke=\"#000000\" cx=\"181.643\" cy=\"-279\" rx=\"37.0935\" ry=\"18\"/\u003e\n\u003ctext text-anchor=\"middle\" x=\"181.643\" y=\"-275.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\"\u003esession\u003c/text\u003e\n\u003c/g\u003e\n\u003c!-- root\u0026#45;\u0026gt;session --\u003e\n\u003cg id=\"edge1\" class=\"edge\"\u003e\n\u003ctitle\u003eroot\u0026#45;\u0026gt;session\u003c/title\u003e\n\u003cpath fill=\"none\" stroke=\"#000000\" d=\"M181.643,-347.9735C181.643,-336.1918 181.643,-320.5607 181.643,-307.1581\"/\u003e\n\u003cpolygon fill=\"#000000\" stroke=\"#000000\" points=\"185.1431,-307.0033 181.643,-297.0034 178.1431,-307.0034 185.1431,-307.0033\"/\u003e\n\u003ctext text-anchor=\"middle\" x=\"185.643\" y=\"-318.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\"\u003e*\u003c/text\u003e\n\u003c/g\u003e\n\u003c!-- event --\u003e\n\u003cg id=\"node3\" class=\"node\"\u003e\n\u003ctitle\u003eevent\u003c/title\u003e\n\u003cellipse fill=\"none\" stroke=\"#000000\" cx=\"132.643\" cy=\"-192\" rx=\"30.5947\" ry=\"18\"/\u003e\n\u003ctext text-anchor=\"middle\" x=\"132.643\" y=\"-188.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\"\u003eevent\u003c/text\u003e\n\u003c/g\u003e\n\u003c!-- session\u0026#45;\u0026gt;event --\u003e\n\u003cg id=\"edge2\" class=\"edge\"\u003e\n\u003ctitle\u003esession\u0026#45;\u0026gt;event\u003c/title\u003e\n\u003cpath fill=\"none\" stroke=\"#000000\" d=\"M171.7271,-261.3943C164.7744,-249.0496 155.358,-232.3306 147.51,-218.3965\"/\u003e\n\u003cpolygon fill=\"#000000\" stroke=\"#000000\" points=\"150.3339,-216.2782 142.3769,-209.2827 144.2348,-219.7133 150.3339,-216.2782\"/\u003e\n\u003ctext text-anchor=\"middle\" x=\"164.643\" y=\"-231.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\"\u003e*\u003c/text\u003e\n\u003c/g\u003e\n\u003c!-- session_id --\u003e\n\u003cg id=\"node4\" class=\"node\"\u003e\n\u003ctitle\u003esession_id\u003c/title\u003e\n\u003cellipse fill=\"none\" stroke=\"#000000\" cx=\"230.643\" cy=\"-192\" rx=\"49.2915\" ry=\"18\"/\u003e\n\u003ctext text-anchor=\"middle\" x=\"230.643\" y=\"-188.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\"\u003esession_id\u003c/text\u003e\n\u003c/g\u003e\n\u003c!-- session\u0026#45;\u0026gt;session_id --\u003e\n\u003cg id=\"edge3\" class=\"edge\"\u003e\n\u003ctitle\u003esession\u0026#45;\u0026gt;session_id\u003c/title\u003e\n\u003cpath fill=\"none\" stroke=\"#000000\" d=\"M191.5589,-261.3943C198.4208,-249.211 207.6823,-232.7669 215.4675,-218.9443\"/\u003e\n\u003cpolygon fill=\"#000000\" stroke=\"#000000\" points=\"218.7159,-220.3089 220.5737,-209.8782 212.6167,-216.8737 218.7159,-220.3089\"/\u003e\n\u003ctext text-anchor=\"middle\" x=\"213.143\" y=\"-231.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\"\u003e?\u003c/text\u003e\n\u003c/g\u003e\n\u003c!-- action --\u003e\n\u003cg id=\"node5\" class=\"node\"\u003e\n\u003ctitle\u003eaction\u003c/title\u003e\n\u003cellipse fill=\"none\" stroke=\"#000000\" cx=\"78.643\" cy=\"-105\" rx=\"33.2948\" ry=\"18\"/\u003e\n\u003ctext text-anchor=\"middle\" x=\"78.643\" y=\"-101.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\"\u003eaction\u003c/text\u003e\n\u003c/g\u003e\n\u003c!-- event\u0026#45;\u0026gt;action --\u003e\n\u003cg id=\"edge4\" class=\"edge\"\u003e\n\u003ctitle\u003eevent\u0026#45;\u0026gt;action\u003c/title\u003e\n\u003cpath fill=\"none\" stroke=\"#000000\" d=\"M121.9744,-174.8116C114.2085,-162.3 103.5554,-145.1366 94.7556,-130.9592\"/\u003e\n\u003cpolygon fill=\"#000000\" stroke=\"#000000\" points=\"97.5369,-128.8033 89.2895,-122.1527 91.5894,-132.4949 97.5369,-128.8033\"/\u003e\n\u003ctext text-anchor=\"middle\" x=\"113.643\" y=\"-144.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\"\u003e*\u003c/text\u003e\n\u003c/g\u003e\n\u003c!-- query_token --\u003e\n\u003cg id=\"node6\" class=\"node\"\u003e\n\u003ctitle\u003equery_token\u003c/title\u003e\n\u003cellipse fill=\"none\" stroke=\"#000000\" cx=\"187.643\" cy=\"-105\" rx=\"57.6901\" ry=\"18\"/\u003e\n\u003ctext text-anchor=\"middle\" x=\"187.643\" y=\"-101.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\"\u003equery_token\u003c/text\u003e\n\u003c/g\u003e\n\u003c!-- event\u0026#45;\u0026gt;query_token --\u003e\n\u003cg id=\"edge5\" class=\"edge\"\u003e\n\u003ctitle\u003eevent\u0026#45;\u0026gt;query_token\u003c/title\u003e\n\u003cpath fill=\"none\" stroke=\"#000000\" d=\"M143.5093,-174.8116C151.3162,-162.4624 161.988,-145.5816 170.8819,-131.513\"/\u003e\n\u003cpolygon fill=\"#000000\" stroke=\"#000000\" points=\"174.0365,-133.0729 176.4218,-122.75 168.1197,-129.3323 174.0365,-133.0729\"/\u003e\n\u003ctext text-anchor=\"middle\" x=\"167.643\" y=\"-144.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\"\u003e*\u003c/text\u003e\n\u003c/g\u003e\n\u003c!-- number_of_views --\u003e\n\u003cg id=\"node7\" class=\"node\"\u003e\n\u003ctitle\u003enumber_of_views\u003c/title\u003e\n\u003cellipse fill=\"none\" stroke=\"#000000\" cx=\"78.643\" cy=\"-18\" rx=\"78.7863\" ry=\"18\"/\u003e\n\u003ctext text-anchor=\"middle\" x=\"78.643\" y=\"-14.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\"\u003enumber_of_views\u003c/text\u003e\n\u003c/g\u003e\n\u003c!-- action\u0026#45;\u0026gt;number_of_views --\u003e\n\u003cg id=\"edge6\" class=\"edge\"\u003e\n\u003ctitle\u003eaction\u0026#45;\u0026gt;number_of_views\u003c/title\u003e\n\u003cpath fill=\"none\" stroke=\"#000000\" d=\"M78.643,-86.9735C78.643,-75.1918 78.643,-59.5607 78.643,-46.1581\"/\u003e\n\u003cpolygon fill=\"#000000\" stroke=\"#000000\" points=\"82.1431,-46.0033 78.643,-36.0034 75.1431,-46.0034 82.1431,-46.0033\"/\u003e\n\u003ctext text-anchor=\"middle\" x=\"82.143\" y=\"-57.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\"\u003e?\u003c/text\u003e\n\u003c/g\u003e\n\u003c/g\u003e\n\u003c/svg\u003e\n",
            "text/plain": [
              "\u003cgraphviz.files.Source at 0x7efc6d4c1160\u003e"
            ]
          },
          "execution_count": 5,
          "metadata": {
            "tags": []
          },
          "output_type": "execute_result"
        }
      ],
      "source": [
        "#@title { display-mode: \"form\" }\n",
        "#@test {\"skip\": true} \n",
        "\n",
        "_display(\"\"\"\n",
        "digraph {\n",
        "  root -\u003e session [label=\"*\"];\n",
        "  session -\u003e event [label=\"*\"];\n",
        "  session -\u003e session_id [label=\"?\"];\n",
        "  event -\u003e action [label=\"*\"];\n",
        "  event -\u003e query_token [label=\"*\"]\n",
        "  action -\u003e number_of_views [label=\"?\"];\n",
        "}\n",
        "\"\"\")\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "P0aBK7NbO0wp"
      },
      "source": [
        "We will be using visualizations like this to demostrate struct2tensor queries later.\n",
        "\n",
        "Note:\n",
        "\n",
        "* The \"*\" on the edge means the pointed node has repeated values; while the \"?\" means it has an optional value.\n",
        "* There is always a \"root\" node whose only child is the root of the structure. Note that it's \"repeated\" because one struct2tensorTree can represent multiple instances of a structure.\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5stZYn1dBXdl"
      },
      "source": [
        "\n",
        "### struct2tensor Query\n",
        "A struct2tensor query transforms a Prensor into another Prensor.\n",
        "\n",
        "For example, `broadcast` is a query that replicates a node as a child of one of its siblings.\n",
        "\n",
        "Applying\n",
        "```\n",
        "broadcast(\n",
        "    source_path=\"session.session_id\",\n",
        "    sibling=\"event\",\n",
        "    new_field_name=\"session_session_id\")\n",
        "```\n",
        "\n",
        "on the previous tree gives:\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 543
        },
        "id": "zfhF-frzPxZm",
        "outputId": "36bd1e53-befe-4bfa-e7da-737bd18a2611"
      },
      "outputs": [
        {
          "data": {
            "image/svg+xml": "\u003c?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?\u003e\n\u003c!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\"\u003e\n\u003c!-- Generated by graphviz version 2.40.1 (20161225.0304)\n --\u003e\n\u003c!-- Title: %3 Pages: 1 --\u003e\n\u003csvg width=\"387pt\" height=\"392pt\"\n viewBox=\"0.00 0.00 387.44 392.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\"\u003e\n\u003cg id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 388)\"\u003e\n\u003ctitle\u003e%3\u003c/title\u003e\n\u003cpolygon fill=\"#ffffff\" stroke=\"transparent\" points=\"-4,4 -4,-388 383.4377,-388 383.4377,4 -4,4\"/\u003e\n\u003c!-- session_session_id --\u003e\n\u003cg id=\"node1\" class=\"node\"\u003e\n\u003ctitle\u003esession_session_id\u003c/title\u003e\n\u003cellipse fill=\"none\" stroke=\"#ff0000\" cx=\"80.5928\" cy=\"-105\" rx=\"80.6858\" ry=\"18\"/\u003e\n\u003ctext text-anchor=\"middle\" x=\"80.5928\" y=\"-101.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\"\u003esession_session_id\u003c/text\u003e\n\u003c/g\u003e\n\u003c!-- root --\u003e\n\u003cg id=\"node2\" class=\"node\"\u003e\n\u003ctitle\u003eroot\u003c/title\u003e\n\u003cellipse fill=\"none\" stroke=\"#000000\" cx=\"261.5928\" cy=\"-366\" rx=\"27\" ry=\"18\"/\u003e\n\u003ctext text-anchor=\"middle\" x=\"261.5928\" y=\"-362.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\"\u003eroot\u003c/text\u003e\n\u003c/g\u003e\n\u003c!-- session --\u003e\n\u003cg id=\"node3\" class=\"node\"\u003e\n\u003ctitle\u003esession\u003c/title\u003e\n\u003cellipse fill=\"none\" stroke=\"#000000\" cx=\"261.5928\" cy=\"-279\" rx=\"37.0935\" ry=\"18\"/\u003e\n\u003ctext text-anchor=\"middle\" x=\"261.5928\" y=\"-275.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\"\u003esession\u003c/text\u003e\n\u003c/g\u003e\n\u003c!-- root\u0026#45;\u0026gt;session --\u003e\n\u003cg id=\"edge1\" class=\"edge\"\u003e\n\u003ctitle\u003eroot\u0026#45;\u0026gt;session\u003c/title\u003e\n\u003cpath fill=\"none\" stroke=\"#000000\" d=\"M261.5928,-347.9735C261.5928,-336.1918 261.5928,-320.5607 261.5928,-307.1581\"/\u003e\n\u003cpolygon fill=\"#000000\" stroke=\"#000000\" points=\"265.0929,-307.0033 261.5928,-297.0034 258.0929,-307.0034 265.0929,-307.0033\"/\u003e\n\u003ctext text-anchor=\"middle\" x=\"265.5928\" y=\"-318.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\"\u003e*\u003c/text\u003e\n\u003c/g\u003e\n\u003c!-- event --\u003e\n\u003cg id=\"node4\" class=\"node\"\u003e\n\u003ctitle\u003eevent\u003c/title\u003e\n\u003cellipse fill=\"none\" stroke=\"#000000\" cx=\"212.5928\" cy=\"-192\" rx=\"30.5947\" ry=\"18\"/\u003e\n\u003ctext text-anchor=\"middle\" x=\"212.5928\" y=\"-188.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\"\u003eevent\u003c/text\u003e\n\u003c/g\u003e\n\u003c!-- session\u0026#45;\u0026gt;event --\u003e\n\u003cg id=\"edge2\" class=\"edge\"\u003e\n\u003ctitle\u003esession\u0026#45;\u0026gt;event\u003c/title\u003e\n\u003cpath fill=\"none\" stroke=\"#000000\" d=\"M251.677,-261.3943C244.7242,-249.0496 235.3078,-232.3306 227.4598,-218.3965\"/\u003e\n\u003cpolygon fill=\"#000000\" stroke=\"#000000\" points=\"230.2838,-216.2782 222.3268,-209.2827 224.1846,-219.7133 230.2838,-216.2782\"/\u003e\n\u003ctext text-anchor=\"middle\" x=\"244.5928\" y=\"-231.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\"\u003e*\u003c/text\u003e\n\u003c/g\u003e\n\u003c!-- session_id --\u003e\n\u003cg id=\"node5\" class=\"node\"\u003e\n\u003ctitle\u003esession_id\u003c/title\u003e\n\u003cellipse fill=\"none\" stroke=\"#000000\" cx=\"310.5928\" cy=\"-192\" rx=\"49.2915\" ry=\"18\"/\u003e\n\u003ctext text-anchor=\"middle\" x=\"310.5928\" y=\"-188.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\"\u003esession_id\u003c/text\u003e\n\u003c/g\u003e\n\u003c!-- session\u0026#45;\u0026gt;session_id --\u003e\n\u003cg id=\"edge3\" class=\"edge\"\u003e\n\u003ctitle\u003esession\u0026#45;\u0026gt;session_id\u003c/title\u003e\n\u003cpath fill=\"none\" stroke=\"#000000\" d=\"M271.5087,-261.3943C278.3706,-249.211 287.6322,-232.7669 295.4173,-218.9443\"/\u003e\n\u003cpolygon fill=\"#000000\" stroke=\"#000000\" points=\"298.6657,-220.3089 300.5235,-209.8782 292.5665,-216.8737 298.6657,-220.3089\"/\u003e\n\u003ctext text-anchor=\"middle\" x=\"293.0928\" y=\"-231.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\"\u003e?\u003c/text\u003e\n\u003c/g\u003e\n\u003c!-- event\u0026#45;\u0026gt;session_session_id --\u003e\n\u003cg id=\"edge5\" class=\"edge\"\u003e\n\u003ctitle\u003eevent\u0026#45;\u0026gt;session_session_id\u003c/title\u003e\n\u003cpath fill=\"none\" stroke=\"#000000\" d=\"M191.9678,-178.4063C171.4317,-164.8711 139.5358,-143.8488 115.1315,-127.7641\"/\u003e\n\u003cpolygon fill=\"#000000\" stroke=\"#000000\" points=\"116.79,-124.6654 106.5143,-122.0846 112.9378,-130.5101 116.79,-124.6654\"/\u003e\n\u003ctext text-anchor=\"middle\" x=\"158.0928\" y=\"-144.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\"\u003e?\u003c/text\u003e\n\u003c/g\u003e\n\u003c!-- action --\u003e\n\u003cg id=\"node6\" class=\"node\"\u003e\n\u003ctitle\u003eaction\u003c/title\u003e\n\u003cellipse fill=\"none\" stroke=\"#000000\" cx=\"212.5928\" cy=\"-105\" rx=\"33.2948\" ry=\"18\"/\u003e\n\u003ctext text-anchor=\"middle\" x=\"212.5928\" y=\"-101.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\"\u003eaction\u003c/text\u003e\n\u003c/g\u003e\n\u003c!-- event\u0026#45;\u0026gt;action --\u003e\n\u003cg id=\"edge4\" class=\"edge\"\u003e\n\u003ctitle\u003eevent\u0026#45;\u0026gt;action\u003c/title\u003e\n\u003cpath fill=\"none\" stroke=\"#000000\" d=\"M212.5928,-173.9735C212.5928,-162.1918 212.5928,-146.5607 212.5928,-133.1581\"/\u003e\n\u003cpolygon fill=\"#000000\" stroke=\"#000000\" points=\"216.0929,-133.0033 212.5928,-123.0034 209.0929,-133.0034 216.0929,-133.0033\"/\u003e\n\u003ctext text-anchor=\"middle\" x=\"216.5928\" y=\"-144.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\"\u003e*\u003c/text\u003e\n\u003c/g\u003e\n\u003c!-- query_token --\u003e\n\u003cg id=\"node7\" class=\"node\"\u003e\n\u003ctitle\u003equery_token\u003c/title\u003e\n\u003cellipse fill=\"none\" stroke=\"#000000\" cx=\"321.5928\" cy=\"-105\" rx=\"57.6901\" ry=\"18\"/\u003e\n\u003ctext text-anchor=\"middle\" x=\"321.5928\" y=\"-101.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\"\u003equery_token\u003c/text\u003e\n\u003c/g\u003e\n\u003c!-- event\u0026#45;\u0026gt;query_token --\u003e\n\u003cg id=\"edge6\" class=\"edge\"\u003e\n\u003ctitle\u003eevent\u0026#45;\u0026gt;query_token\u003c/title\u003e\n\u003cpath fill=\"none\" stroke=\"#000000\" d=\"M231.0832,-177.2416C247.8774,-163.8371 272.929,-143.8418 292.4215,-128.2836\"/\u003e\n\u003cpolygon fill=\"#000000\" stroke=\"#000000\" points=\"294.7624,-130.8933 300.3947,-121.9196 290.3956,-125.4223 294.7624,-130.8933\"/\u003e\n\u003ctext text-anchor=\"middle\" x=\"278.5928\" y=\"-144.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\"\u003e*\u003c/text\u003e\n\u003c/g\u003e\n\u003c!-- number_of_views --\u003e\n\u003cg id=\"node8\" class=\"node\"\u003e\n\u003ctitle\u003enumber_of_views\u003c/title\u003e\n\u003cellipse fill=\"none\" stroke=\"#000000\" cx=\"212.5928\" cy=\"-18\" rx=\"78.7863\" ry=\"18\"/\u003e\n\u003ctext text-anchor=\"middle\" x=\"212.5928\" y=\"-14.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\"\u003enumber_of_views\u003c/text\u003e\n\u003c/g\u003e\n\u003c!-- action\u0026#45;\u0026gt;number_of_views --\u003e\n\u003cg id=\"edge7\" class=\"edge\"\u003e\n\u003ctitle\u003eaction\u0026#45;\u0026gt;number_of_views\u003c/title\u003e\n\u003cpath fill=\"none\" stroke=\"#000000\" d=\"M212.5928,-86.9735C212.5928,-75.1918 212.5928,-59.5607 212.5928,-46.1581\"/\u003e\n\u003cpolygon fill=\"#000000\" stroke=\"#000000\" points=\"216.0929,-46.0033 212.5928,-36.0034 209.0929,-46.0034 216.0929,-46.0033\"/\u003e\n\u003ctext text-anchor=\"middle\" x=\"216.0928\" y=\"-57.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\"\u003e?\u003c/text\u003e\n\u003c/g\u003e\n\u003c/g\u003e\n\u003c/svg\u003e\n",
            "text/plain": [
              "\u003cgraphviz.files.Source at 0x7efc6d4c1320\u003e"
            ]
          },
          "execution_count": 6,
          "metadata": {
            "tags": []
          },
          "output_type": "execute_result"
        }
      ],
      "source": [
        "#@title { display-mode: \"form\" }\n",
        "#@test {\"skip\": true} \n",
        "\n",
        "_display(\"\"\"\n",
        "digraph {\n",
        "  session_session_id [color=\"red\"];\n",
        "  root -\u003e session [label=\"*\"];\n",
        "  session -\u003e event [label=\"*\"];\n",
        "  session -\u003e session_id [label=\"?\"];\n",
        "  event -\u003e action [label=\"*\"];\n",
        "  event -\u003e session_session_id [label=\"?\"];\n",
        "  event -\u003e query_token [label=\"*\"];\n",
        "  action -\u003e number_of_views [label=\"?\"];\n",
        "}\n",
        "\"\"\")\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eOHokTjBPyHW"
      },
      "source": [
        "We will talk about common struct2tensor queries in later sections.\n",
        "\n",
        "### Projection\n",
        "A projection of paths in a Prensor produces another Prensor with just the selected paths.\n",
        "\n",
        "#### Logical representation of a projection\n",
        "The structure of the projected path can be represented losslessly as nested lists. For example, the projection of `event.action.number_of_views` from the struct2tensorTree formed by the following two instances of `struct2tensor.test.Session`:\n",
        "```\n",
        "{\n",
        "  event { action { number_of_views: 1} action { number_of_views: 2} action {} }\n",
        "  event {}\n",
        "}, {\n",
        "  event { action { number_of_views: 3} }\n",
        "}\n",
        "```\n",
        "\n",
        "is:\n",
        "\n",
        "```\n",
        "[  # the outer list has two elements b/c there are two Session protos.\n",
        " [  # the first proto has two events\n",
        "   [[1],[2],[]],  # 3 actions, the last one does not have a number_of_views.\n",
        "   [], # the second event does not have action\n",
        " ],\n",
        " [ # the second proto has one event\n",
        "   [[3]],\n",
        " ],\n",
        "]\n",
        "```\n",
        "\n",
        "#### Representing nested lists with `tf.SparseTensor`\n",
        "\n",
        "struct2tensor uses `tf.SparseTensor` to represent the above nested list in the projection results. Note that `tf.SparseTensor` essentially enforces that the lists nested at the same level to have the same length (because the there is a certain size for each dimension), therefore this representation is lossy. The above nested lists, when written as a SparseTensor will look like:\n",
        "```\n",
        "tf.SparseTensor(\n",
        "  dense_shape=[2, 2, 3, 1],  # each is the maximum length of lists at the same nesting level.\n",
        "  values = [1, 2, 3],\n",
        "  indices = [[0, 0, 0, 0], [0, 0, 1, 0], [1, 0, 0, 0]]\n",
        ")\n",
        "```\n",
        "\n",
        "Note that the last dimension is useless: the index of that dimension will always be 0 for any present value because number_of_views is an optional field. So struct2tensors library will actually \"squeeze\" all the optional dimensions.\n",
        "\n",
        "The actual result would be:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "height": 68
        },
        "executionInfo": {
          "elapsed": 352,
          "status": "ok",
          "timestamp": 1600368919353,
          "user": {
            "displayName": "",
            "photoUrl": "",
            "userId": ""
          },
          "user_tz": 420
        },
        "id": "XvuEY3D3WIP7",
        "outputId": "ee33d486-90e7-4328-accc-11f13f82a9db"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "[{event.action.number_of_views: SparseTensor(values=[1, 2, 3], dense_shape=[2, 2, 3], \n",
            "     indices=[[0, 0, 0], [0, 0, 1], [1, 0, 0]])}\n",
            "]\n"
          ]
        }
      ],
      "source": [
        "query = _create_query_from_text_sessions(['''\n",
        "  event { action { number_of_views: 1} action { number_of_views: 2} action {} }\n",
        "  event {}\n",
        "  ''', '''\n",
        "  event { action { number_of_views: 3} }\n",
        "  ''']\n",
        "  ).project([\"event.action.number_of_views\"])\n",
        "\n",
        "prensor = s2t.calculate_prensors([query])\n",
        "pretty.pprint(prensor)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iaailOJhWLLa"
      },
      "source": [
        "struct2tensor's internal data model is closer to the above \"nested lists\" abstraction and sometimes it's easier to reason with \"nested lists\" than with `SparseTensor`s.\n",
        "\n",
        "Recently, [`tf.RaggedTensor`](https://www.tensorflow.org/guide/ragged_tensors) was introduced to represent nested lists exactly. We are working on adding support for projecting into ragged tensors."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UD48ikK4Eop-"
      },
      "source": [
        "## Common struct2tensor Queries\n",
        "\n",
        "### `promote`\n",
        "\n",
        "Promotes a node to become a sibling of its parent. If the node is repeated, then all its values are concatenated (the order is preserved)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 426
        },
        "id": "tnbD2lKoDbsk",
        "outputId": "244aeef6-48b7-4ae5-f01c-2429bb65a7ea"
      },
      "outputs": [
        {
          "data": {
            "image/svg+xml": "\u003c?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?\u003e\n\u003c!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\"\u003e\n\u003c!-- Generated by graphviz version 2.40.1 (20161225.0304)\n --\u003e\n\u003c!-- Title: %3 Pages: 1 --\u003e\n\u003csvg width=\"124pt\" height=\"305pt\"\n viewBox=\"0.00 0.00 123.69 305.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\"\u003e\n\u003cg id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 301)\"\u003e\n\u003ctitle\u003e%3\u003c/title\u003e\n\u003cpolygon fill=\"#ffffff\" stroke=\"transparent\" points=\"-4,4 -4,-301 119.6897,-301 119.6897,4 -4,4\"/\u003e\n\u003c!-- root --\u003e\n\u003cg id=\"node1\" class=\"node\"\u003e\n\u003ctitle\u003eroot\u003c/title\u003e\n\u003cellipse fill=\"none\" stroke=\"#000000\" cx=\"57.8449\" cy=\"-279\" rx=\"27\" ry=\"18\"/\u003e\n\u003ctext text-anchor=\"middle\" x=\"57.8449\" y=\"-275.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\"\u003eroot\u003c/text\u003e\n\u003c/g\u003e\n\u003c!-- session --\u003e\n\u003cg id=\"node2\" class=\"node\"\u003e\n\u003ctitle\u003esession\u003c/title\u003e\n\u003cellipse fill=\"none\" stroke=\"#000000\" cx=\"57.8449\" cy=\"-192\" rx=\"37.0935\" ry=\"18\"/\u003e\n\u003ctext text-anchor=\"middle\" x=\"57.8449\" y=\"-188.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\"\u003esession\u003c/text\u003e\n\u003c/g\u003e\n\u003c!-- root\u0026#45;\u0026gt;session --\u003e\n\u003cg id=\"edge1\" class=\"edge\"\u003e\n\u003ctitle\u003eroot\u0026#45;\u0026gt;session\u003c/title\u003e\n\u003cpath fill=\"none\" stroke=\"#000000\" d=\"M57.8449,-260.9735C57.8449,-249.1918 57.8449,-233.5607 57.8449,-220.1581\"/\u003e\n\u003cpolygon fill=\"#000000\" stroke=\"#000000\" points=\"61.345,-220.0033 57.8449,-210.0034 54.345,-220.0034 61.345,-220.0033\"/\u003e\n\u003ctext text-anchor=\"middle\" x=\"61.8449\" y=\"-231.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\"\u003e*\u003c/text\u003e\n\u003c/g\u003e\n\u003c!-- event --\u003e\n\u003cg id=\"node3\" class=\"node\"\u003e\n\u003ctitle\u003eevent\u003c/title\u003e\n\u003cellipse fill=\"none\" stroke=\"#000000\" cx=\"57.8449\" cy=\"-105\" rx=\"30.5947\" ry=\"18\"/\u003e\n\u003ctext text-anchor=\"middle\" x=\"57.8449\" y=\"-101.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\"\u003eevent\u003c/text\u003e\n\u003c/g\u003e\n\u003c!-- session\u0026#45;\u0026gt;event --\u003e\n\u003cg id=\"edge2\" class=\"edge\"\u003e\n\u003ctitle\u003esession\u0026#45;\u0026gt;event\u003c/title\u003e\n\u003cpath fill=\"none\" stroke=\"#000000\" d=\"M57.8449,-173.9735C57.8449,-162.1918 57.8449,-146.5607 57.8449,-133.1581\"/\u003e\n\u003cpolygon fill=\"#000000\" stroke=\"#000000\" points=\"61.345,-133.0033 57.8449,-123.0034 54.345,-133.0034 61.345,-133.0033\"/\u003e\n\u003ctext text-anchor=\"middle\" x=\"61.8449\" y=\"-144.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\"\u003e*\u003c/text\u003e\n\u003c/g\u003e\n\u003c!-- query_token --\u003e\n\u003cg id=\"node4\" class=\"node\"\u003e\n\u003ctitle\u003equery_token\u003c/title\u003e\n\u003cellipse fill=\"none\" stroke=\"#000000\" cx=\"57.8449\" cy=\"-18\" rx=\"57.6901\" ry=\"18\"/\u003e\n\u003ctext text-anchor=\"middle\" x=\"57.8449\" y=\"-14.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\"\u003equery_token\u003c/text\u003e\n\u003c/g\u003e\n\u003c!-- event\u0026#45;\u0026gt;query_token --\u003e\n\u003cg id=\"edge3\" class=\"edge\"\u003e\n\u003ctitle\u003eevent\u0026#45;\u0026gt;query_token\u003c/title\u003e\n\u003cpath fill=\"none\" stroke=\"#000000\" d=\"M57.8449,-86.9735C57.8449,-75.1918 57.8449,-59.5607 57.8449,-46.1581\"/\u003e\n\u003cpolygon fill=\"#000000\" stroke=\"#000000\" points=\"61.345,-46.0033 57.8449,-36.0034 54.345,-46.0034 61.345,-46.0033\"/\u003e\n\u003ctext text-anchor=\"middle\" x=\"61.8449\" y=\"-57.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\"\u003e*\u003c/text\u003e\n\u003c/g\u003e\n\u003c/g\u003e\n\u003c/svg\u003e\n",
            "text/plain": [
              "\u003cgraphviz.files.Source at 0x7efc6d4c1438\u003e"
            ]
          },
          "execution_count": 8,
          "metadata": {
            "tags": []
          },
          "output_type": "execute_result"
        }
      ],
      "source": [
        "#@title { display-mode: \"form\" }\n",
        "#@test {\"skip\": true} \n",
        "\n",
        "_display('''\n",
        "digraph {\n",
        "  root -\u003e session [label=\"*\"];\n",
        "  session -\u003e event [label=\"*\"];\n",
        "  event -\u003e query_token [label=\"*\"];\n",
        "}\n",
        "''')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OOgJlcHPDyIk"
      },
      "source": [
        "`promote(source_path=\"event.query_token\", new_field_name=\"event_query_token\")`"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 426
        },
        "id": "xKJ_EraVHyUA",
        "outputId": "64c3720d-3672-4625-d8c1-1a4b301d6a2d"
      },
      "outputs": [
        {
          "data": {
            "image/svg+xml": "\u003c?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?\u003e\n\u003c!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\"\u003e\n\u003c!-- Generated by graphviz version 2.40.1 (20161225.0304)\n --\u003e\n\u003c!-- Title: %3 Pages: 1 --\u003e\n\u003csvg width=\"281pt\" height=\"305pt\"\n viewBox=\"0.00 0.00 281.04 305.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\"\u003e\n\u003cg id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 301)\"\u003e\n\u003ctitle\u003e%3\u003c/title\u003e\n\u003cpolygon fill=\"#ffffff\" stroke=\"transparent\" points=\"-4,4 -4,-301 277.0375,-301 277.0375,4 -4,4\"/\u003e\n\u003c!-- event_query_token --\u003e\n\u003cg id=\"node1\" class=\"node\"\u003e\n\u003ctitle\u003eevent_query_token\u003c/title\u003e\n\u003cellipse fill=\"none\" stroke=\"#ff0000\" cx=\"83.1926\" cy=\"-105\" rx=\"83.3857\" ry=\"18\"/\u003e\n\u003ctext text-anchor=\"middle\" x=\"83.1926\" y=\"-101.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\"\u003eevent_query_token\u003c/text\u003e\n\u003c/g\u003e\n\u003c!-- root --\u003e\n\u003cg id=\"node2\" class=\"node\"\u003e\n\u003ctitle\u003eroot\u003c/title\u003e\n\u003cellipse fill=\"none\" stroke=\"#000000\" cx=\"98.1926\" cy=\"-279\" rx=\"27\" ry=\"18\"/\u003e\n\u003ctext text-anchor=\"middle\" x=\"98.1926\" y=\"-275.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\"\u003eroot\u003c/text\u003e\n\u003c/g\u003e\n\u003c!-- session --\u003e\n\u003cg id=\"node3\" class=\"node\"\u003e\n\u003ctitle\u003esession\u003c/title\u003e\n\u003cellipse fill=\"none\" stroke=\"#000000\" cx=\"98.1926\" cy=\"-192\" rx=\"37.0935\" ry=\"18\"/\u003e\n\u003ctext text-anchor=\"middle\" x=\"98.1926\" y=\"-188.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\"\u003esession\u003c/text\u003e\n\u003c/g\u003e\n\u003c!-- root\u0026#45;\u0026gt;session --\u003e\n\u003cg id=\"edge1\" class=\"edge\"\u003e\n\u003ctitle\u003eroot\u0026#45;\u0026gt;session\u003c/title\u003e\n\u003cpath fill=\"none\" stroke=\"#000000\" d=\"M98.1926,-260.9735C98.1926,-249.1918 98.1926,-233.5607 98.1926,-220.1581\"/\u003e\n\u003cpolygon fill=\"#000000\" stroke=\"#000000\" points=\"101.6927,-220.0033 98.1926,-210.0034 94.6927,-220.0034 101.6927,-220.0033\"/\u003e\n\u003ctext text-anchor=\"middle\" x=\"102.1926\" y=\"-231.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\"\u003e*\u003c/text\u003e\n\u003c/g\u003e\n\u003c!-- session\u0026#45;\u0026gt;event_query_token --\u003e\n\u003cg id=\"edge3\" class=\"edge\"\u003e\n\u003ctitle\u003esession\u0026#45;\u0026gt;event_query_token\u003c/title\u003e\n\u003cpath fill=\"none\" stroke=\"#000000\" d=\"M95.0846,-173.9735C93.0533,-162.1918 90.3582,-146.5607 88.0475,-133.1581\"/\u003e\n\u003cpolygon fill=\"#000000\" stroke=\"#000000\" points=\"91.4449,-132.2633 86.2966,-123.0034 84.5467,-133.4527 91.4449,-132.2633\"/\u003e\n\u003ctext text-anchor=\"middle\" x=\"96.1926\" y=\"-144.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\"\u003e*\u003c/text\u003e\n\u003c/g\u003e\n\u003c!-- event --\u003e\n\u003cg id=\"node4\" class=\"node\"\u003e\n\u003ctitle\u003eevent\u003c/title\u003e\n\u003cellipse fill=\"none\" stroke=\"#000000\" cx=\"215.1926\" cy=\"-105\" rx=\"30.5947\" ry=\"18\"/\u003e\n\u003ctext text-anchor=\"middle\" x=\"215.1926\" y=\"-101.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\"\u003eevent\u003c/text\u003e\n\u003c/g\u003e\n\u003c!-- session\u0026#45;\u0026gt;event --\u003e\n\u003cg id=\"edge2\" class=\"edge\"\u003e\n\u003ctitle\u003esession\u0026#45;\u0026gt;event\u003c/title\u003e\n\u003cpath fill=\"none\" stroke=\"#000000\" d=\"M118.5725,-176.8458C137.7919,-162.5544 166.7343,-141.0331 187.9324,-125.2705\"/\u003e\n\u003cpolygon fill=\"#000000\" stroke=\"#000000\" points=\"190.2713,-127.8928 196.2075,-119.1171 186.0944,-122.2756 190.2713,-127.8928\"/\u003e\n\u003ctext text-anchor=\"middle\" x=\"168.1926\" y=\"-144.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\"\u003e*\u003c/text\u003e\n\u003c/g\u003e\n\u003c!-- query_token --\u003e\n\u003cg id=\"node5\" class=\"node\"\u003e\n\u003ctitle\u003equery_token\u003c/title\u003e\n\u003cellipse fill=\"none\" stroke=\"#000000\" cx=\"215.1926\" cy=\"-18\" rx=\"57.6901\" ry=\"18\"/\u003e\n\u003ctext text-anchor=\"middle\" x=\"215.1926\" y=\"-14.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\"\u003equery_token\u003c/text\u003e\n\u003c/g\u003e\n\u003c!-- event\u0026#45;\u0026gt;query_token --\u003e\n\u003cg id=\"edge4\" class=\"edge\"\u003e\n\u003ctitle\u003eevent\u0026#45;\u0026gt;query_token\u003c/title\u003e\n\u003cpath fill=\"none\" stroke=\"#000000\" d=\"M215.1926,-86.9735C215.1926,-75.1918 215.1926,-59.5607 215.1926,-46.1581\"/\u003e\n\u003cpolygon fill=\"#000000\" stroke=\"#000000\" points=\"218.6927,-46.0033 215.1926,-36.0034 211.6927,-46.0034 218.6927,-46.0033\"/\u003e\n\u003ctext text-anchor=\"middle\" x=\"219.1926\" y=\"-57.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\"\u003e*\u003c/text\u003e\n\u003c/g\u003e\n\u003c/g\u003e\n\u003c/svg\u003e\n",
            "text/plain": [
              "\u003cgraphviz.files.Source at 0x7efc58c02358\u003e"
            ]
          },
          "execution_count": 9,
          "metadata": {
            "tags": []
          },
          "output_type": "execute_result"
        }
      ],
      "source": [
        "#@title { display-mode: \"form\" }\n",
        "#@test {\"skip\": true} \n",
        "\n",
        "_display('''\n",
        "digraph {\n",
        "  event_query_token [color=\"red\"];\n",
        "  root -\u003e session [label=\"*\"];\n",
        "  session -\u003e event [label=\"*\"];\n",
        "  session -\u003e event_query_token [label=\"*\"];\n",
        "  event -\u003e query_token [label=\"*\"];\n",
        "}\n",
        "''')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "height": 68
        },
        "executionInfo": {
          "elapsed": 344,
          "status": "ok",
          "timestamp": 1600296594869,
          "user": {
            "displayName": "",
            "photoUrl": "",
            "userId": ""
          },
          "user_tz": 420
        },
        "id": "oQCcVWd-JDT9",
        "outputId": "acad1b9a-0985-46bb-b895-7d085814d20e"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "[{event_query_token: SparseTensor(values=[b'abc', b'def', b'ghi'], dense_shape=[1, 3], \n",
            "     indices=[[0, 0], [0, 1], [0, 2]])}\n",
            "]\n"
          ]
        }
      ],
      "source": [
        "query = (_create_query_from_text_sessions([\n",
        "\"\"\"\n",
        "event {\n",
        "  query_token: \"abc\"\n",
        "  query_token: \"def\"\n",
        "}\n",
        "event {\n",
        "  query_token: \"ghi\"\n",
        "}\n",
        "\"\"\"])\n",
        "        .promote(source_path=\"event.query_token\", new_field_name=\"event_query_token\")\n",
        "        .project([\"event_query_token\"]))\n",
        "\n",
        "prensor = s2t.calculate_prensors([query])\n",
        "\n",
        "_pretty_print(prensor)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "av9km5d_KM8o"
      },
      "source": [
        "The projected structure is like:\n",
        "```\n",
        "{\n",
        "  # this is under Session.\n",
        "  event_query_token: \"abc\"\n",
        "  event_query_token: \"def\"\n",
        "  event_query_token: \"ghi\"\n",
        "}\n",
        "```"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "t2BU8warKqFm"
      },
      "source": [
        "### `broadcast`\n",
        "\n",
        "Broadcasts the value of a node to one of its sibling. The value will be replicated if the sibling is repeated. This is similar to TensorFlow and Numpy's [broadcasting semantics](https://docs.scipy.org/doc/numpy-1.13.0/user/basics.broadcasting.html).\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 311
        },
        "id": "1hFGEV0DMmOo",
        "outputId": "ff23c5ce-17c9-4604-f309-c9b384b6d8ca"
      },
      "outputs": [
        {
          "data": {
            "image/svg+xml": "\u003c?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?\u003e\n\u003c!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\"\u003e\n\u003c!-- Generated by graphviz version 2.40.1 (20161225.0304)\n --\u003e\n\u003c!-- Title: %3 Pages: 1 --\u003e\n\u003csvg width=\"186pt\" height=\"218pt\"\n viewBox=\"0.00 0.00 185.94 218.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\"\u003e\n\u003cg id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 214)\"\u003e\n\u003ctitle\u003e%3\u003c/title\u003e\n\u003cpolygon fill=\"#ffffff\" stroke=\"transparent\" points=\"-4,4 -4,-214 181.9429,-214 181.9429,4 -4,4\"/\u003e\n\u003c!-- root --\u003e\n\u003cg id=\"node1\" class=\"node\"\u003e\n\u003ctitle\u003eroot\u003c/title\u003e\n\u003cellipse fill=\"none\" stroke=\"#000000\" cx=\"98.3956\" cy=\"-192\" rx=\"27\" ry=\"18\"/\u003e\n\u003ctext text-anchor=\"middle\" x=\"98.3956\" y=\"-188.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\"\u003eroot\u003c/text\u003e\n\u003c/g\u003e\n\u003c!-- session --\u003e\n\u003cg id=\"node2\" class=\"node\"\u003e\n\u003ctitle\u003esession\u003c/title\u003e\n\u003cellipse fill=\"none\" stroke=\"#000000\" cx=\"98.3956\" cy=\"-105\" rx=\"37.0935\" ry=\"18\"/\u003e\n\u003ctext text-anchor=\"middle\" x=\"98.3956\" y=\"-101.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\"\u003esession\u003c/text\u003e\n\u003c/g\u003e\n\u003c!-- root\u0026#45;\u0026gt;session --\u003e\n\u003cg id=\"edge1\" class=\"edge\"\u003e\n\u003ctitle\u003eroot\u0026#45;\u0026gt;session\u003c/title\u003e\n\u003cpath fill=\"none\" stroke=\"#000000\" d=\"M98.3956,-173.9735C98.3956,-162.1918 98.3956,-146.5607 98.3956,-133.1581\"/\u003e\n\u003cpolygon fill=\"#000000\" stroke=\"#000000\" points=\"101.8957,-133.0033 98.3956,-123.0034 94.8957,-133.0034 101.8957,-133.0033\"/\u003e\n\u003ctext text-anchor=\"middle\" x=\"102.3956\" y=\"-144.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\"\u003e*\u003c/text\u003e\n\u003c/g\u003e\n\u003c!-- session_id --\u003e\n\u003cg id=\"node3\" class=\"node\"\u003e\n\u003ctitle\u003esession_id\u003c/title\u003e\n\u003cellipse fill=\"none\" stroke=\"#000000\" cx=\"49.3956\" cy=\"-18\" rx=\"49.2915\" ry=\"18\"/\u003e\n\u003ctext text-anchor=\"middle\" x=\"49.3956\" y=\"-14.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\"\u003esession_id\u003c/text\u003e\n\u003c/g\u003e\n\u003c!-- session\u0026#45;\u0026gt;session_id --\u003e\n\u003cg id=\"edge2\" class=\"edge\"\u003e\n\u003ctitle\u003esession\u0026#45;\u0026gt;session_id\u003c/title\u003e\n\u003cpath fill=\"none\" stroke=\"#000000\" d=\"M88.4797,-87.3943C81.6179,-75.211 72.3563,-58.7669 64.5711,-44.9443\"/\u003e\n\u003cpolygon fill=\"#000000\" stroke=\"#000000\" points=\"67.4219,-42.8737 59.4649,-35.8782 61.3228,-46.3089 67.4219,-42.8737\"/\u003e\n\u003ctext text-anchor=\"middle\" x=\"80.8956\" y=\"-57.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\"\u003e?\u003c/text\u003e\n\u003c/g\u003e\n\u003c!-- event --\u003e\n\u003cg id=\"node4\" class=\"node\"\u003e\n\u003ctitle\u003eevent\u003c/title\u003e\n\u003cellipse fill=\"none\" stroke=\"#000000\" cx=\"147.3956\" cy=\"-18\" rx=\"30.5947\" ry=\"18\"/\u003e\n\u003ctext text-anchor=\"middle\" x=\"147.3956\" y=\"-14.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\"\u003eevent\u003c/text\u003e\n\u003c/g\u003e\n\u003c!-- session\u0026#45;\u0026gt;event --\u003e\n\u003cg id=\"edge3\" class=\"edge\"\u003e\n\u003ctitle\u003esession\u0026#45;\u0026gt;event\u003c/title\u003e\n\u003cpath fill=\"none\" stroke=\"#000000\" d=\"M108.3115,-87.3943C115.2642,-75.0496 124.6807,-58.3306 132.5286,-44.3965\"/\u003e\n\u003cpolygon fill=\"#000000\" stroke=\"#000000\" points=\"135.8039,-45.7133 137.6617,-35.2827 129.7047,-42.2782 135.8039,-45.7133\"/\u003e\n\u003ctext text-anchor=\"middle\" x=\"129.3956\" y=\"-57.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\"\u003e*\u003c/text\u003e\n\u003c/g\u003e\n\u003c/g\u003e\n\u003c/svg\u003e\n",
            "text/plain": [
              "\u003cgraphviz.files.Source at 0x7efc58d8cc50\u003e"
            ]
          },
          "execution_count": 11,
          "metadata": {
            "tags": []
          },
          "output_type": "execute_result"
        }
      ],
      "source": [
        "#@title { display-mode: \"form\" }\n",
        "#@test {\"skip\": true} \n",
        "\n",
        "_display('''\n",
        "digraph {\n",
        "  root -\u003e session [label=\"*\"];\n",
        "  session -\u003e session_id [label=\"?\"];\n",
        "  session -\u003e event [label=\"*\"];\n",
        "}\n",
        "''')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DvBCsWBuNDKV"
      },
      "source": [
        "`broadcast(source_path=\"session_id\", sibling_field=\"event\", new_field_name=\"session_session_id\")`"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 426
        },
        "id": "wTUHyGvSNMGp",
        "outputId": "d1ef4902-72eb-4bd2-e0bb-3008a9de17e7"
      },
      "outputs": [
        {
          "data": {
            "image/svg+xml": "\u003c?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?\u003e\n\u003c!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\"\u003e\n\u003c!-- Generated by graphviz version 2.40.1 (20161225.0304)\n --\u003e\n\u003c!-- Title: %3 Pages: 1 --\u003e\n\u003csvg width=\"236pt\" height=\"305pt\"\n viewBox=\"0.00 0.00 235.99 305.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\"\u003e\n\u003cg id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 301)\"\u003e\n\u003ctitle\u003e%3\u003c/title\u003e\n\u003cpolygon fill=\"#ffffff\" stroke=\"transparent\" points=\"-4,4 -4,-301 231.9885,-301 231.9885,4 -4,4\"/\u003e\n\u003c!-- session_session_id --\u003e\n\u003cg id=\"node1\" class=\"node\"\u003e\n\u003ctitle\u003esession_session_id\u003c/title\u003e\n\u003cellipse fill=\"none\" stroke=\"#ff0000\" cx=\"147.3956\" cy=\"-18\" rx=\"80.6858\" ry=\"18\"/\u003e\n\u003ctext text-anchor=\"middle\" x=\"147.3956\" y=\"-14.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\"\u003esession_session_id\u003c/text\u003e\n\u003c/g\u003e\n\u003c!-- root --\u003e\n\u003cg id=\"node2\" class=\"node\"\u003e\n\u003ctitle\u003eroot\u003c/title\u003e\n\u003cellipse fill=\"none\" stroke=\"#000000\" cx=\"64.3956\" cy=\"-279\" rx=\"27\" ry=\"18\"/\u003e\n\u003ctext text-anchor=\"middle\" x=\"64.3956\" y=\"-275.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\"\u003eroot\u003c/text\u003e\n\u003c/g\u003e\n\u003c!-- session --\u003e\n\u003cg id=\"node3\" class=\"node\"\u003e\n\u003ctitle\u003esession\u003c/title\u003e\n\u003cellipse fill=\"none\" stroke=\"#000000\" cx=\"64.3956\" cy=\"-192\" rx=\"37.0935\" ry=\"18\"/\u003e\n\u003ctext text-anchor=\"middle\" x=\"64.3956\" y=\"-188.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\"\u003esession\u003c/text\u003e\n\u003c/g\u003e\n\u003c!-- root\u0026#45;\u0026gt;session --\u003e\n\u003cg id=\"edge1\" class=\"edge\"\u003e\n\u003ctitle\u003eroot\u0026#45;\u0026gt;session\u003c/title\u003e\n\u003cpath fill=\"none\" stroke=\"#000000\" d=\"M64.3956,-260.9735C64.3956,-249.1918 64.3956,-233.5607 64.3956,-220.1581\"/\u003e\n\u003cpolygon fill=\"#000000\" stroke=\"#000000\" points=\"67.8957,-220.0033 64.3956,-210.0034 60.8957,-220.0034 67.8957,-220.0033\"/\u003e\n\u003ctext text-anchor=\"middle\" x=\"68.3956\" y=\"-231.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\"\u003e*\u003c/text\u003e\n\u003c/g\u003e\n\u003c!-- session_id --\u003e\n\u003cg id=\"node4\" class=\"node\"\u003e\n\u003ctitle\u003esession_id\u003c/title\u003e\n\u003cellipse fill=\"none\" stroke=\"#000000\" cx=\"49.3956\" cy=\"-105\" rx=\"49.2915\" ry=\"18\"/\u003e\n\u003ctext text-anchor=\"middle\" x=\"49.3956\" y=\"-101.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\"\u003esession_id\u003c/text\u003e\n\u003c/g\u003e\n\u003c!-- session\u0026#45;\u0026gt;session_id --\u003e\n\u003cg id=\"edge2\" class=\"edge\"\u003e\n\u003ctitle\u003esession\u0026#45;\u0026gt;session_id\u003c/title\u003e\n\u003cpath fill=\"none\" stroke=\"#000000\" d=\"M61.2876,-173.9735C59.2563,-162.1918 56.5612,-146.5607 54.2505,-133.1581\"/\u003e\n\u003cpolygon fill=\"#000000\" stroke=\"#000000\" points=\"57.6479,-132.2633 52.4996,-123.0034 50.7497,-133.4527 57.6479,-132.2633\"/\u003e\n\u003ctext text-anchor=\"middle\" x=\"61.8956\" y=\"-144.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\"\u003e?\u003c/text\u003e\n\u003c/g\u003e\n\u003c!-- event --\u003e\n\u003cg id=\"node5\" class=\"node\"\u003e\n\u003ctitle\u003eevent\u003c/title\u003e\n\u003cellipse fill=\"none\" stroke=\"#000000\" cx=\"147.3956\" cy=\"-105\" rx=\"30.5947\" ry=\"18\"/\u003e\n\u003ctext text-anchor=\"middle\" x=\"147.3956\" y=\"-101.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\"\u003eevent\u003c/text\u003e\n\u003c/g\u003e\n\u003c!-- session\u0026#45;\u0026gt;event --\u003e\n\u003cg id=\"edge3\" class=\"edge\"\u003e\n\u003ctitle\u003esession\u0026#45;\u0026gt;event\u003c/title\u003e\n\u003cpath fill=\"none\" stroke=\"#000000\" d=\"M80.0073,-175.636C92.8036,-162.2229 111.0676,-143.0787 125.3545,-128.1034\"/\u003e\n\u003cpolygon fill=\"#000000\" stroke=\"#000000\" points=\"128.0426,-130.3561 132.413,-120.7047 122.9778,-125.5241 128.0426,-130.3561\"/\u003e\n\u003ctext text-anchor=\"middle\" x=\"114.3956\" y=\"-144.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\"\u003e*\u003c/text\u003e\n\u003c/g\u003e\n\u003c!-- event\u0026#45;\u0026gt;session_session_id --\u003e\n\u003cg id=\"edge4\" class=\"edge\"\u003e\n\u003ctitle\u003eevent\u0026#45;\u0026gt;session_session_id\u003c/title\u003e\n\u003cpath fill=\"none\" stroke=\"#000000\" d=\"M147.3956,-86.9735C147.3956,-75.1918 147.3956,-59.5607 147.3956,-46.1581\"/\u003e\n\u003cpolygon fill=\"#000000\" stroke=\"#000000\" points=\"150.8957,-46.0033 147.3956,-36.0034 143.8957,-46.0034 150.8957,-46.0033\"/\u003e\n\u003ctext text-anchor=\"middle\" x=\"150.8956\" y=\"-57.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\"\u003e?\u003c/text\u003e\n\u003c/g\u003e\n\u003c/g\u003e\n\u003c/svg\u003e\n",
            "text/plain": [
              "\u003cgraphviz.files.Source at 0x7efc58c25550\u003e"
            ]
          },
          "execution_count": 12,
          "metadata": {
            "tags": []
          },
          "output_type": "execute_result"
        }
      ],
      "source": [
        "#@title { display-mode: \"form\" }\n",
        "#@test {\"skip\": true} \n",
        "\n",
        "_display('''\n",
        "digraph {\n",
        "  session_session_id [color=\"red\"];\n",
        "  root -\u003e session [label=\"*\"];\n",
        "  session -\u003e session_id [label=\"?\"];\n",
        "  session -\u003e event [label=\"*\"];\n",
        "  event -\u003e session_session_id [label=\"?\"];\n",
        "}\n",
        "''')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "height": 68
        },
        "executionInfo": {
          "elapsed": 341,
          "status": "ok",
          "timestamp": 1600296607633,
          "user": {
            "displayName": "",
            "photoUrl": "",
            "userId": ""
          },
          "user_tz": 420
        },
        "id": "wQE_ceMzNjzv",
        "outputId": "d92d9977-1e4c-4138-e89c-94e9c0f4bf5e"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "[{event.session_session_id: SparseTensor(values=[8, 8], dense_shape=[1, 2], \n",
            "     indices=[[0, 0], [0, 1]])}\n",
            "]\n"
          ]
        }
      ],
      "source": [
        "query = (_create_query_from_text_sessions([\n",
        "\"\"\"\n",
        "session_id: 8\n",
        "event { }\n",
        "event { }\n",
        "\"\"\"])\n",
        "        .broadcast(source_path=\"session_id\",\n",
        "                   sibling_field=\"event\",\n",
        "                   new_field_name=\"session_session_id\")\n",
        "        .project([\"event.session_session_id\"]))\n",
        "\n",
        "prensor = s2t.calculate_prensors([query])\n",
        "_pretty_print(prensor)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7fL8pmsQObUT"
      },
      "source": [
        "The projected structure is like:\n",
        "```\n",
        "{\n",
        "  event {\n",
        "    session_session_id: 8\n",
        "  }\n",
        "  event {\n",
        "    session_session_id: 8\n",
        "  }\n",
        "}\n",
        "```"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ch2WgR9AP23Q"
      },
      "source": [
        "### `promote_and_broadcast`\n",
        "The query accepts multiple source fields and a destination field. For each source field, it first promotes it to the least common ancestor with the destination field (if necessary), then broadcasts it to the destination field (if necessary).\n",
        "\n",
        "Usually for the purpose of machine learning, this gives a reasonable flattened representation of nested structures."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-uNfasxgRsvO"
      },
      "source": [
        "```\n",
        "promote_and_broadcast(\n",
        "    path_dictionary={\n",
        "        'session_info_duration_sec': 'session_info.session_duration_sec'},\n",
        "    dest_path_parent='event.action')\n",
        "```\n",
        "is equivalent to:\n",
        "```\n",
        "promote(source_path='session_info.session_duration_sec',\n",
        "    new_field_name='anonymous_field1')\n",
        "\n",
        "broadcast(source_path='anonymous_field1',\n",
        "    sibling_field='event.action',\n",
        "    new_field_name='session_info_duration_sec')\n",
        "```\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rYkWc_u8SR_k"
      },
      "source": [
        "### `map_field_values`\n",
        "\n",
        "Creates a new node that is a sibling of a leaf node. The values of the new node are results of applying the given function to the values of the source node.\n",
        "\n",
        "Note that the function provided takes 1-D tensor that contains all the values of the source node as input and should also output a 1-D tensor of the same size, and it should build TF ops.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "height": 68
        },
        "executionInfo": {
          "elapsed": 333,
          "status": "ok",
          "timestamp": 1600296617311,
          "user": {
            "displayName": "",
            "photoUrl": "",
            "userId": ""
          },
          "user_tz": 420
        },
        "id": "DWjA2OFcS4k1",
        "outputId": "7a2d8ef5-7111-4b45-81dd-65d8222badae"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "[{session_id_plus_one: SparseTensor(values=[9, 10], dense_shape=[2], \n",
            "     indices=[[0], [1]])}\n",
            "]\n"
          ]
        }
      ],
      "source": [
        "query = (_create_query_from_text_sessions([\n",
        "\"\"\"\n",
        "session_id: 8\n",
        "\"\"\",\n",
        "\"\"\"\n",
        "session_id: 9\n",
        "\"\"\"])\n",
        "        .map_field_values(\"session_id\", lambda x: tf.add(x, 1), dtype=tf.int64,\n",
        "                          new_field_name=\"session_id_plus_one\")\n",
        "        .project([\"session_id_plus_one\"]))\n",
        "    \n",
        "prensor = s2t.calculate_prensors([query])\n",
        "\n",
        "_pretty_print(prensor)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_NusnDa1T7s5"
      },
      "source": [
        "### `reroot`\n",
        "\n",
        "Makes the given node the new root of the struct2tensorTree. This has two effects:\n",
        "\n",
        "* restricts the scope of the struct2tensorTree\n",
        "  + The field paths in all the following queries are relative to the new root\n",
        "  + There's no way to refer to nodes that are outside the subtree rooted at the new root.\n",
        "* changes the batch dimension.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 426
        },
        "id": "OBNPdOlQU6qS",
        "outputId": "df56d6d2-72c5-4c90-e2b9-b3fe2bffde00"
      },
      "outputs": [
        {
          "data": {
            "image/svg+xml": "\u003c?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?\u003e\n\u003c!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\"\u003e\n\u003c!-- Generated by graphviz version 2.40.1 (20161225.0304)\n --\u003e\n\u003c!-- Title: %3 Pages: 1 --\u003e\n\u003csvg width=\"198pt\" height=\"305pt\"\n viewBox=\"0.00 0.00 198.29 305.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\"\u003e\n\u003cg id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 301)\"\u003e\n\u003ctitle\u003e%3\u003c/title\u003e\n\u003cpolygon fill=\"#ffffff\" stroke=\"transparent\" points=\"-4,4 -4,-301 194.2918,-301 194.2918,4 -4,4\"/\u003e\n\u003c!-- root --\u003e\n\u003cg id=\"node1\" class=\"node\"\u003e\n\u003ctitle\u003eroot\u003c/title\u003e\n\u003cellipse fill=\"none\" stroke=\"#000000\" cx=\"64.3956\" cy=\"-279\" rx=\"27\" ry=\"18\"/\u003e\n\u003ctext text-anchor=\"middle\" x=\"64.3956\" y=\"-275.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\"\u003eroot\u003c/text\u003e\n\u003c/g\u003e\n\u003c!-- session --\u003e\n\u003cg id=\"node2\" class=\"node\"\u003e\n\u003ctitle\u003esession\u003c/title\u003e\n\u003cellipse fill=\"none\" stroke=\"#000000\" cx=\"64.3956\" cy=\"-192\" rx=\"37.0935\" ry=\"18\"/\u003e\n\u003ctext text-anchor=\"middle\" x=\"64.3956\" y=\"-188.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\"\u003esession\u003c/text\u003e\n\u003c/g\u003e\n\u003c!-- root\u0026#45;\u0026gt;session --\u003e\n\u003cg id=\"edge1\" class=\"edge\"\u003e\n\u003ctitle\u003eroot\u0026#45;\u0026gt;session\u003c/title\u003e\n\u003cpath fill=\"none\" stroke=\"#000000\" d=\"M64.3956,-260.9735C64.3956,-249.1918 64.3956,-233.5607 64.3956,-220.1581\"/\u003e\n\u003cpolygon fill=\"#000000\" stroke=\"#000000\" points=\"67.8957,-220.0033 64.3956,-210.0034 60.8957,-220.0034 67.8957,-220.0033\"/\u003e\n\u003ctext text-anchor=\"middle\" x=\"68.3956\" y=\"-231.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\"\u003e*\u003c/text\u003e\n\u003c/g\u003e\n\u003c!-- session_id --\u003e\n\u003cg id=\"node3\" class=\"node\"\u003e\n\u003ctitle\u003esession_id\u003c/title\u003e\n\u003cellipse fill=\"none\" stroke=\"#000000\" cx=\"49.3956\" cy=\"-105\" rx=\"49.2915\" ry=\"18\"/\u003e\n\u003ctext text-anchor=\"middle\" x=\"49.3956\" y=\"-101.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\"\u003esession_id\u003c/text\u003e\n\u003c/g\u003e\n\u003c!-- session\u0026#45;\u0026gt;session_id --\u003e\n\u003cg id=\"edge2\" class=\"edge\"\u003e\n\u003ctitle\u003esession\u0026#45;\u0026gt;session_id\u003c/title\u003e\n\u003cpath fill=\"none\" stroke=\"#000000\" d=\"M61.2876,-173.9735C59.2563,-162.1918 56.5612,-146.5607 54.2505,-133.1581\"/\u003e\n\u003cpolygon fill=\"#000000\" stroke=\"#000000\" points=\"57.6479,-132.2633 52.4996,-123.0034 50.7497,-133.4527 57.6479,-132.2633\"/\u003e\n\u003ctext text-anchor=\"middle\" x=\"61.8956\" y=\"-144.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\"\u003e?\u003c/text\u003e\n\u003c/g\u003e\n\u003c!-- event --\u003e\n\u003cg id=\"node4\" class=\"node\"\u003e\n\u003ctitle\u003eevent\u003c/title\u003e\n\u003cellipse fill=\"none\" stroke=\"#000000\" cx=\"147.3956\" cy=\"-105\" rx=\"30.5947\" ry=\"18\"/\u003e\n\u003ctext text-anchor=\"middle\" x=\"147.3956\" y=\"-101.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\"\u003eevent\u003c/text\u003e\n\u003c/g\u003e\n\u003c!-- session\u0026#45;\u0026gt;event --\u003e\n\u003cg id=\"edge3\" class=\"edge\"\u003e\n\u003ctitle\u003esession\u0026#45;\u0026gt;event\u003c/title\u003e\n\u003cpath fill=\"none\" stroke=\"#000000\" d=\"M80.0073,-175.636C92.8036,-162.2229 111.0676,-143.0787 125.3545,-128.1034\"/\u003e\n\u003cpolygon fill=\"#000000\" stroke=\"#000000\" points=\"128.0426,-130.3561 132.413,-120.7047 122.9778,-125.5241 128.0426,-130.3561\"/\u003e\n\u003ctext text-anchor=\"middle\" x=\"114.3956\" y=\"-144.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\"\u003e*\u003c/text\u003e\n\u003c/g\u003e\n\u003c!-- event_id --\u003e\n\u003cg id=\"node5\" class=\"node\"\u003e\n\u003ctitle\u003eevent_id\u003c/title\u003e\n\u003cellipse fill=\"none\" stroke=\"#000000\" cx=\"147.3956\" cy=\"-18\" rx=\"42.7926\" ry=\"18\"/\u003e\n\u003ctext text-anchor=\"middle\" x=\"147.3956\" y=\"-14.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\"\u003eevent_id\u003c/text\u003e\n\u003c/g\u003e\n\u003c!-- event\u0026#45;\u0026gt;event_id --\u003e\n\u003cg id=\"edge4\" class=\"edge\"\u003e\n\u003ctitle\u003eevent\u0026#45;\u0026gt;event_id\u003c/title\u003e\n\u003cpath fill=\"none\" stroke=\"#000000\" d=\"M147.3956,-86.9735C147.3956,-75.1918 147.3956,-59.5607 147.3956,-46.1581\"/\u003e\n\u003cpolygon fill=\"#000000\" stroke=\"#000000\" points=\"150.8957,-46.0033 147.3956,-36.0034 143.8957,-46.0034 150.8957,-46.0033\"/\u003e\n\u003ctext text-anchor=\"middle\" x=\"150.8956\" y=\"-57.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\"\u003e?\u003c/text\u003e\n\u003c/g\u003e\n\u003c/g\u003e\n\u003c/svg\u003e\n",
            "text/plain": [
              "\u003cgraphviz.files.Source at 0x7efc58c9f3c8\u003e"
            ]
          },
          "execution_count": 15,
          "metadata": {
            "tags": []
          },
          "output_type": "execute_result"
        }
      ],
      "source": [
        "#@title { display-mode: \"form\" }\n",
        "#@test {\"skip\": true} \n",
        "\n",
        "_display('''\n",
        "digraph {\n",
        "  root -\u003e session [label=\"*\"];\n",
        "  session -\u003e session_id [label=\"?\"];\n",
        "  session -\u003e event [label=\"*\"];\n",
        "  event -\u003e event_id [label=\"?\"];\n",
        "}\n",
        "''')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HFlI3zMtU6Ev"
      },
      "source": [
        "`reroot(\"event\")`"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 311
        },
        "id": "uANFWQA-T68W",
        "outputId": "659a11b1-39bb-456d-de90-b2675182ca25"
      },
      "outputs": [
        {
          "data": {
            "image/svg+xml": "\u003c?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?\u003e\n\u003c!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\"\u003e\n\u003c!-- Generated by graphviz version 2.40.1 (20161225.0304)\n --\u003e\n\u003c!-- Title: %3 Pages: 1 --\u003e\n\u003csvg width=\"94pt\" height=\"218pt\"\n viewBox=\"0.00 0.00 93.79 218.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\"\u003e\n\u003cg id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 214)\"\u003e\n\u003ctitle\u003e%3\u003c/title\u003e\n\u003cpolygon fill=\"#ffffff\" stroke=\"transparent\" points=\"-4,4 -4,-214 89.7924,-214 89.7924,4 -4,4\"/\u003e\n\u003c!-- root --\u003e\n\u003cg id=\"node1\" class=\"node\"\u003e\n\u003ctitle\u003eroot\u003c/title\u003e\n\u003cellipse fill=\"none\" stroke=\"#000000\" cx=\"42.8962\" cy=\"-192\" rx=\"27\" ry=\"18\"/\u003e\n\u003ctext text-anchor=\"middle\" x=\"42.8962\" y=\"-188.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\"\u003eroot\u003c/text\u003e\n\u003c/g\u003e\n\u003c!-- event --\u003e\n\u003cg id=\"node2\" class=\"node\"\u003e\n\u003ctitle\u003eevent\u003c/title\u003e\n\u003cellipse fill=\"none\" stroke=\"#000000\" cx=\"42.8962\" cy=\"-105\" rx=\"30.5947\" ry=\"18\"/\u003e\n\u003ctext text-anchor=\"middle\" x=\"42.8962\" y=\"-101.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\"\u003eevent\u003c/text\u003e\n\u003c/g\u003e\n\u003c!-- root\u0026#45;\u0026gt;event --\u003e\n\u003cg id=\"edge1\" class=\"edge\"\u003e\n\u003ctitle\u003eroot\u0026#45;\u0026gt;event\u003c/title\u003e\n\u003cpath fill=\"none\" stroke=\"#000000\" d=\"M42.8962,-173.9735C42.8962,-162.1918 42.8962,-146.5607 42.8962,-133.1581\"/\u003e\n\u003cpolygon fill=\"#000000\" stroke=\"#000000\" points=\"46.3963,-133.0033 42.8962,-123.0034 39.3963,-133.0034 46.3963,-133.0033\"/\u003e\n\u003ctext text-anchor=\"middle\" x=\"46.8962\" y=\"-144.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\"\u003e*\u003c/text\u003e\n\u003c/g\u003e\n\u003c!-- event_id --\u003e\n\u003cg id=\"node3\" class=\"node\"\u003e\n\u003ctitle\u003eevent_id\u003c/title\u003e\n\u003cellipse fill=\"none\" stroke=\"#000000\" cx=\"42.8962\" cy=\"-18\" rx=\"42.7926\" ry=\"18\"/\u003e\n\u003ctext text-anchor=\"middle\" x=\"42.8962\" y=\"-14.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\"\u003eevent_id\u003c/text\u003e\n\u003c/g\u003e\n\u003c!-- event\u0026#45;\u0026gt;event_id --\u003e\n\u003cg id=\"edge2\" class=\"edge\"\u003e\n\u003ctitle\u003eevent\u0026#45;\u0026gt;event_id\u003c/title\u003e\n\u003cpath fill=\"none\" stroke=\"#000000\" d=\"M42.8962,-86.9735C42.8962,-75.1918 42.8962,-59.5607 42.8962,-46.1581\"/\u003e\n\u003cpolygon fill=\"#000000\" stroke=\"#000000\" points=\"46.3963,-46.0033 42.8962,-36.0034 39.3963,-46.0034 46.3963,-46.0033\"/\u003e\n\u003ctext text-anchor=\"middle\" x=\"46.3962\" y=\"-57.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\"\u003e?\u003c/text\u003e\n\u003c/g\u003e\n\u003c/g\u003e\n\u003c/svg\u003e\n",
            "text/plain": [
              "\u003cgraphviz.files.Source at 0x7efc58c9f908\u003e"
            ]
          },
          "execution_count": 16,
          "metadata": {
            "tags": []
          },
          "output_type": "execute_result"
        }
      ],
      "source": [
        "#@title { display-mode: \"form\" }\n",
        "#@test {\"skip\": true} \n",
        "\n",
        "_display('''\n",
        "digraph {\n",
        "  root -\u003e event [label=\"*\"];\n",
        "  event -\u003e event_id [label=\"?\"];\n",
        "}\n",
        "''')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "height": 459
        },
        "executionInfo": {
          "elapsed": 408,
          "status": "ok",
          "timestamp": 1600297140484,
          "user": {
            "displayName": "",
            "photoUrl": "",
            "userId": ""
          },
          "user_tz": 420
        },
        "id": "lMoeHHllVeet",
        "outputId": "631afc7a-ebb3-456f-d0a1-11eeeccee95c"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Assume the following Sessions: \n",
            "[session_id: 1\n",
            "event {\n",
            "  event_id: \"a\"\n",
            "}\n",
            "event {\n",
            "  event_id: \"b\"\n",
            "}\n",
            ", session_id: 2\n",
            ", session_id: 3\n",
            "event {\n",
            "  event_id: \"c\"\n",
            "}\n",
            "]\n",
            "\n",
            "\n",
            "project([\"event.event_id\"]) before reroot() (the batch dimension is the index to sessions):\n",
            "[{event.event_id: SparseTensor(values=[b'a', b'b', b'c'], dense_shape=[3, 2], \n",
            "     indices=[[0, 0], [0, 1], [2, 0]])}\n",
            "]\n",
            "\n",
            "\n",
            "project([\"event_id\"]) after reroot() (the batch dimension becomes the index to events):\n",
            "[{event_id: SparseTensor(values=[b'a', b'b', b'c'], dense_shape=[3], \n",
            "     indices=[[0], [1], [2]])}\n",
            "]\n"
          ]
        }
      ],
      "source": [
        "#@title { display-mode: \"form\" }\n",
        "text_protos = [\"\"\"\n",
        "session_id: 1\n",
        "event {\n",
        "  event_id: \"a\"\n",
        "}\n",
        "event {\n",
        "  event_id: \"b\"\n",
        "}\n",
        "\"\"\",\n",
        "\"\"\"\n",
        "session_id: 2\n",
        "\"\"\",\n",
        "\"\"\"\n",
        "session_id: 3\n",
        "event {\n",
        "  event_id: \"c\"\n",
        "}\n",
        "\"\"\"\n",
        "]\n",
        "print(\"\"\"Assume the following Sessions: \"\"\")\n",
        "print([text_format.Merge(p, s2t.test.test_pb2.Session()) for p in text_protos])\n",
        "print(\"\\n\")\n",
        "reroot_example_query = _create_query_from_text_sessions(text_protos)\n",
        "\n",
        "print(\"\"\"project([\"event.event_id\"]) before reroot() (the batch dimension is the index to sessions):\"\"\")\n",
        "_pretty_print(s2t.calculate_prensors([reroot_example_query.project([\"event.event_id\"])]))\n",
        "print(\"\\n\")\n",
        "print(\"\"\"project([\"event_id\"]) after reroot() (the batch dimension becomes the index to events):\"\"\")\n",
        "_pretty_print(s2t.calculate_prensors([reroot_example_query.reroot(\"event\").project([\"event_id\"])]))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "70nd50UgLlQU"
      },
      "source": [
        "### Proto Map\n",
        "\n",
        "You can specify a key for the proto map field in a path via brackets.\n",
        "\n",
        "Given the following tf.Example:\n",
        "```\n",
        "features {\n",
        "  feature {\n",
        "    key: \"my_feature\"\n",
        "    value {\n",
        "      float_list {\n",
        "        value: 1.0\n",
        "      }\n",
        "    }\n",
        "  }\n",
        "  feature {\n",
        "    key: \"other_feature\"\n",
        "    value {\n",
        "      bytes_list {\n",
        "        value: \"my_val\"\n",
        "      }\n",
        "    }\n",
        "  }\n",
        "}\n",
        "```\n",
        "\n",
        "To get the values of `my_feature` and `other_feature`, we can `promote_and_broadcast` and `project` the following paths: `features.feature[my_feature].float_list.value` and `features.feature[other_feature].bytes_list.value`\n",
        "\n",
        "This results in the following dict of ragged tensors: \n",
        "```\n",
        "{\n",
        "  features.my_new_feature: \u003ctf.RaggedTensor [[[1.0]]]\u003e,\n",
        "  features.other_new_feature: \u003ctf.RaggedTensor [[[b'my_val']]]\u003e\n",
        "}\n",
        "```\n",
        "\n",
        "Note: we renamed `my_feature` to `my_new_feature` in the `promote_and_broadcast` (and similarly for `other_feature`)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "executionInfo": {
          "elapsed": 111,
          "status": "ok",
          "timestamp": 1632331356173,
          "user": {
            "displayName": "",
            "photoUrl": "",
            "userId": ""
          },
          "user_tz": 420
        },
        "id": "9ESrpLsEL4SO",
        "outputId": "ba002544-d4b2-4ae2-f343-f1f25ab91a4d"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "{features.my_new_feature: \u003ctf.RaggedTensor [[[1.0]]]\u003e, features.other_new_feature: \u003ctf.RaggedTensor [[[b'my_val']]]\u003e}\n"
          ]
        }
      ],
      "source": [
        "tf_example = text_format.Parse(\"\"\"\n",
        "features {\n",
        "  feature {\n",
        "    key: \"my_feature\"\n",
        "    value {\n",
        "      float_list {\n",
        "        value: 1.0\n",
        "      }\n",
        "    }\n",
        "  }\n",
        "  feature {\n",
        "    key: \"other_feature\"\n",
        "    value {\n",
        "      bytes_list {\n",
        "        value: \"my_val\"\n",
        "      }\n",
        "    }\n",
        "  }\n",
        "}\n",
        "\"\"\", tf.train.Example())\n",
        "\n",
        "query = s2t.create_expression_from_proto(\n",
        "    tf_example.SerializeToString(), tf.train.Example.DESCRIPTOR)\n",
        "query = query.promote_and_broadcast({'my_new_feature': \"features.feature[my_feature].float_list.value\", \"other_new_feature\": \"features.feature[other_feature].bytes_list.value\"}, \"features\")\n",
        "query = query.project([\"features.my_new_feature\", \"features.other_new_feature\"])\n",
        "[prensor] = s2t.calculate_prensors([query])\n",
        "ragged_tensors = prensor.get_ragged_tensors()\n",
        "\n",
        "print(ragged_tensors)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UxLZtkdmM7lj"
      },
      "source": [
        "## Apache Parquet Support\n",
        "\n",
        "`struct2tensor` offers an [Apache Parquet](https://parquet.apache.org/) [tf.DataSet](https://www.tensorflow.org/api_docs/python/tf/data/Dataset) that allows reading from a Parquet file and apply queries to manipulate the structure of the data.\n",
        "\n",
        "Because of the powerful struct2tensor library, the dataset will only read the Parquet columns that are required. This reduces I/O cost if we only need a select few columns."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7IHo8w3HMuyT"
      },
      "source": [
        "### Preparation\n",
        "\n",
        "Please run the code cell at [Some Pretty Printing and Imports](#scrollTo=dIxHSM3VQfUu\u0026line=1\u0026uniqifier=1) to ensure that all required modules are imported, and that pretty print works properly.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Vu02cwvfXVpf"
      },
      "source": [
        "#### Prepare the input data"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 68
        },
        "id": "xEDieagzNL-9",
        "outputId": "773de200-cd49-496f-ff3f-60074be06bca"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n",
            "                                 Dload  Upload   Total   Spent    Left  Speed\n",
            "100  1657  100  1657    0     0   8122      0 --:--:-- --:--:-- --:--:--  8122\n"
          ]
        }
      ],
      "source": [
        "# Download our sample data file from the struct2tensor repository. The desciption of the data is below.\n",
        "#@test {\"skip\": true} \n",
        "\n",
        "!curl -o dremel_example.parquet 'https://raw.githubusercontent.com/google/struct2tensor/master/struct2tensor/testdata/parquet_testdata/dremel_example.parquet'"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Gn5J62JkNN-P"
      },
      "source": [
        "### Example\n",
        "\n",
        "We will use a sample Parquet data file (*dremel_example.parquet*), which contains data based on the example used in this paper: https://storage.googleapis.com/pub-tools-public-publication-data/pdf/36632.pdf \n",
        "\n",
        "The file *dremel_example.parquet* has the following schema:\n",
        "\n",
        "```\n",
        "message Document {\n",
        "  required int64 DocId;\n",
        "  optional group Links {\n",
        "    repeated int64 Backward;\n",
        "    repeated int64 Forward; }\n",
        "  repeated group Name {\n",
        "    repeated group Language {\n",
        "      required string Code;\n",
        "      optional string Country; }\n",
        "    optional string Url; }}\n",
        " ```\n",
        "\n",
        "and contains the following data:\n",
        "  \n",
        "\n",
        "```\n",
        "Document\n",
        "  DocId: 10\n",
        "  Links\n",
        "    Forward: 20\n",
        "    Forward: 40\n",
        "    Forward: 60\n",
        "  Name\n",
        "    Language\n",
        "      Code: 'en-us'\n",
        "      Country: 'us'\n",
        "    Language\n",
        "      Code: 'en'\n",
        "    Url: 'http://A'\n",
        "  Name\n",
        "    Url: 'http://B'\n",
        "  Name\n",
        "    Language\n",
        "      Code: 'en-gb'\n",
        "      Country: 'gb'\n",
        "Document\n",
        "  DocId: 20\n",
        "  Links\n",
        "    Backward: 10\n",
        "    Backward: 30\n",
        "    Forward: 80\n",
        "  Name\n",
        "    Url: 'http://C'\n",
        "```\n",
        "\n",
        "\n",
        "In this example, we will promote and broadcast the field `Links.Forward` and project it.\n",
        "\n",
        "batch_size will be the number of records (`Document`) per prensor. This works with optional and repeated fields, and will be able to batch the entire record.\n",
        "\n",
        "Feel free to try `batch_size = 2` in the below code. (Note this parquet file only has 2 records (`Document`) total).\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 442
        },
        "id": "A8RyaU3EX4av",
        "outputId": "9235bd5b-cca6-45e0-8244-0f58d6bdcd9b"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/struct2tensor/expression_impl/parquet.py:65: FutureWarning: The 'field_by_name' method is deprecated, use 'field' instead\n",
            "  [arrow_schema.field_by_name(name) for name in arrow_schema.names]))\n",
            "/usr/local/lib/python3.6/dist-packages/struct2tensor/expression_impl/parquet.py:396: FutureWarning: The 'field_by_name' method is deprecated, use 'field' instead\n",
            "  for step in curr_steps_as_set\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "============================\n",
            "Schema of new_field prensor: \n",
            "RootNodeTensor root\n",
            "  repeated ChildNodeTensor Name\n",
            "    repeated \u003cdtype: 'int64'\u003e new_field\n",
            "\n",
            "Sparse tensor representation: \n",
            "{Name.new_field: SparseTensor(values=[20, 40, 60, 20, 40, 60, 20, 40, 60], dense_shape=[1, 3, 3], \n",
            "     indices=[[0, 0, 0], [0, 0, 1], [0, 0, 2], [0, 1, 0], [0, 1, 1], [0, 1, 2], [0, 2, 0], [0, 2, 1], [0, 2, 2]])}\n",
            "\n",
            "============================\n",
            "Schema of new_field prensor: \n",
            "RootNodeTensor root\n",
            "  repeated ChildNodeTensor Name\n",
            "    repeated \u003cdtype: 'int64'\u003e new_field\n",
            "\n",
            "Sparse tensor representation: \n",
            "{Name.new_field: SparseTensor(values=[80], dense_shape=[1, 1, 1], \n",
            "     indices=[[0, 0, 0]])}\n",
            "\n",
            "============================\n"
          ]
        }
      ],
      "source": [
        "#@test {\"skip\": true} \n",
        "\n",
        "from struct2tensor import expression_impl\n",
        "\n",
        "filenames = [\"dremel_example.parquet\"]\n",
        "batch_size = 1\n",
        "\n",
        "exp = s2t.expression_impl.parquet.create_expression_from_parquet_file(filenames)\n",
        "new_exp = exp.promote_and_broadcast({\"new_field\": \"Links.Forward\"}, \"Name\")\n",
        "proj_exp = new_exp.project([\"Name.new_field\"])\n",
        "proj_exp_needed = exp.project([\"Name.Url\"]) \n",
        "# Please note that currently, proj_exp_needed needs to be passed into calculate.\n",
        "# This is due to the way data is stored in parquet (values and repetition \u0026 \n",
        "# definition levels). To construct the node for \"Name\", we need to read the \n",
        "# values of a column containing \"Name\".\n",
        "pqds = s2t.expression_impl.parquet.calculate_parquet_values([proj_exp, proj_exp_needed], exp, \n",
        "                                        filenames, batch_size)\n",
        "\n",
        "for prensors in pqds:\n",
        "  new_field_prensor = prensors[0]\n",
        "  print(\"============================\")\n",
        "  print(\"Schema of new_field prensor: \")\n",
        "  print(new_field_prensor)\n",
        "  print(\"\\nSparse tensor representation: \")\n",
        "  pretty.pprint(new_field_prensor)\n",
        "print(\"============================\")"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "collapsed_sections": [],
      "last_runtime": {
        "kind": "private"
      },
      "name": "struct2tensor: ML on structured data",
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}