Hello_cu_cat.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/dcolinmorgan/grph/blob/main/Hello_cu_cat.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4Y1PPZV1dpo9"
      },
      "source": [
        "# Hello cu_cat: Easy automatic feature engineering for messy dataframes on GPUs!\n",
        "Daniel, Tanmoy, Alex, Leo\n"
      ],
      "id": "4Y1PPZV1dpo9"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LJZKwblsd-k1"
      },
      "source": [
        "[  [LOOM](https://www.loom.com/share/d7fd4980b31949b7b840b230937a636f?sid=6d56b82e-9f50-4059-af9f-bfdc32cd3509)] – press play and dive into analysis!\n",
        "\n",
        "The latest [PyGraphistry[AI]](https://github.com/graphistry/pygraphistry/) release GPU accelerates its automatic feature encoding pipeline, by introducing a new member to the open source GPU dataframe ecosystem: cu_cat.\n",
        "\n",
        "Cu_cat is our GPU-accelerated open source fork of the popular CPU Python  library dirty_cat.   Like dirty_cat, cu_cat makes it easy to convert messy dataframes filled with numbers, strings, and timestamps into numeric feature columns optimized for AI models. It adds interoperability for GPU dataframes and replaces key kernels and algorithms with faster and more scalable GPU variants. **Even on low-end GPUs, we are now able to tackle much larger datasets in the same amount of time – or for the first time! – with end-to-end pipelines. We typically save time with 3-5X speedups and will even see 10X+.**"
      ],
      "id": "LJZKwblsd-k1"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "pZ2B17Vp2EY1"
      },
      "outputs": [],
      "source": [
        "!pip install --extra-index-url=https://pypi.nvidia.com cuml-cu12==23.12.00 cudf-cu12==23.12.00 #cugraph-cu11 pylibraft_cu11 raft_dask_cu11 dask_cudf_cu11 pylibcugraph_cu11 pylibraft_cu11\n",
        "!pip install git+https://github.com/graphistry/pygraphistry.git@dev/depman_gpufeat\n",
        "\n",
        "!pip3 install --upgrade cu_cat"
      ],
      "id": "pZ2B17Vp2EY1"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 55
        },
        "id": "3EszGClHTCC_",
        "outputId": "a80627cd-29e2-45ba-82b2-b0bc0779d63d"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'23.06.00'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 2
        }
      ],
      "source": [
        "import cuml,cudf,os\n",
        "cuml.__version__"
      ],
      "id": "3EszGClHTCC_"
    },
    {
      "cell_type": "code",
      "source": [
        "!echo \"0.8.6\" > VERSION.txt\n",
        "!cp VERSION.txt /usr/local/lib/python3.10/dist-packages/cu_cat/"
      ],
      "metadata": {
        "id": "eHV1zVIxKYVb"
      },
      "id": "eHV1zVIxKYVb",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "219c6516-e381-4d41-aebb-a17704888e5a"
      },
      "outputs": [],
      "source": [
        "import locale\n",
        "def getpreferredencoding(do_setlocale = True):\n",
        "    return \"UTF-8\"\n",
        "locale.getpreferredencoding = getpreferredencoding"
      ],
      "id": "219c6516-e381-4d41-aebb-a17704888e5a"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "9da43cdd-e7c2-45e2-83cb-e7b6a3efe896"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "from collections import Counter\n",
        "import cProfile\n",
        "import pandas as pd\n",
        "import numpy as np\n",
        "import matplotlib.pyplot as plt\n",
        "from pstats import Stats\n",
        "import cudf\n",
        "from time import time\n",
        "import warnings\n",
        "warnings.filterwarnings('ignore')\n",
        "\n",
        "pd.set_option('display.max_colwidth', 200)"
      ],
      "id": "9da43cdd-e7c2-45e2-83cb-e7b6a3efe896"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 55
        },
        "id": "af47c1f6-003b-4585-9722-c13d9a93eb4b",
        "outputId": "340f447e-4455-443e-c173-20f752810dfd"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'0.29.5+84.gf0eb1bf'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 7
        }
      ],
      "source": [
        "import graphistry\n",
        "\n",
        "graphistry.register(api=3,protocol=\"https\", server=\"hub.graphistry.com\", username='dcolinmorgan', password='***') ## key id, secret key\n",
        "graphistry.__version__\n"
      ],
      "id": "af47c1f6-003b-4585-9722-c13d9a93eb4b"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "AuJH7SQMmllb",
        "outputId": "10e51c80-8034-47e2-9f79-c226a3d281b5"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "name\n",
            "Tesla T4\n"
          ]
        }
      ],
      "source": [
        "!nvidia-smi --query-gpu=gpu_name --format=csv\n",
        "# !pip install nvsmi\n",
        "# import nvsmi\n",
        "# gpu=str(nvsmi.get_gpu_processes()).split('|')[3].split(':')[1]"
      ],
      "id": "AuJH7SQMmllb"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "xrzf3ha3Vp2H"
      },
      "outputs": [],
      "source": [
        "X=1\n",
        "import time\n",
        "timestr = time.strftime(\"%Y%m%d-%H%M%S\")\n",
        "from time import time\n"
      ],
      "id": "xrzf3ha3Vp2H"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QAwlxmRDso3p"
      },
      "source": [
        "## look at dataset (select non-sparse columns)"
      ],
      "id": "QAwlxmRDso3p"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Mfs2wwMwoahF"
      },
      "outputs": [],
      "source": [
        "!wget -nc https://www.dropbox.com/s/31dx1g6g59exoc3/part.88.parquet\n",
        "winlogsA=pd.read_parquet('part.88.parquet')\n"
      ],
      "id": "Mfs2wwMwoahF"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "vAG5lO9C1RvP"
      },
      "outputs": [],
      "source": [
        "winlogs=winlogsA[['LogonID','UserName','LogHost','Time','DomainName']]#,'LogonType','SubjectLogonID','Status','Destination','ServiceName']]#,i]]\n",
        "winlogs=winlogs.sample(10000,replace=False)#.dropna()"
      ],
      "id": "vAG5lO9C1RvP"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "cULzDpRa4XBS"
      },
      "outputs": [],
      "source": [
        "import locale\n",
        "def getpreferredencoding(do_setlocale = True):\n",
        "    return \"UTF-8\"\n",
        "locale.getpreferredencoding = getpreferredencoding"
      ],
      "id": "cULzDpRa4XBS"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hoM53bR4DOTG"
      },
      "source": [
        "# Graphistry[AI] - Feature Engineering:\n",
        "## CUDA_cat GPU engine\n",
        "- data: **5million row** x dozens of sparse columns Windows log dataset\n",
        "- data subset: parred down to **20k** rows, selecting just dense columns"
      ],
      "id": "hoM53bR4DOTG"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "-ksOnvDUyYXK"
      },
      "outputs": [],
      "source": [
        "import logging\n",
        "logging.basicConfig(level=logging.DEBUG)"
      ],
      "id": "-ksOnvDUyYXK"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "85kLoG1fyY9U"
      },
      "outputs": [],
      "source": [
        "winlogs=winlogsA[['LogonID','UserName','LogHost','Time','DomainName']]#,'LogonType','SubjectLogonID','Status','Destination','ServiceName']]#,i]]\n",
        "winlogs=winlogs.sample(10000,replace=False)"
      ],
      "id": "85kLoG1fyY9U"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 561
        },
        "id": "fF4c1Fq4XAGr",
        "outputId": "89b6a930-30df-47ab-d0df-21ee723eade8"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "WARNING:graphistry.umap_utils:* Ignoring target column of shape (10000, 0) in UMAP fit, as it is not one dimensional"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "53.1531982421875\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ],
            "text/html": [
              "\n",
              "            <iframe id=\"ab8ca6d0-11b6-4c6b-95e6-b46e1d1d8f90\" src=\"https://hub.graphistry.com/graph/graph.html?dataset=a26c67274ae64051b8297d2511bc9aaf&type=arrow&viztoken=73bac216-6af7-4bc8-a4a7-19014a62e739&usertag=fd8971fa-pygraphistry-0.29.5+84.gf0eb1bf&splashAfter=1696326121&info=true&play=0\"\n",
              "                    allowfullscreen=\"true\" webkitallowfullscreen=\"true\" mozallowfullscreen=\"true\"\n",
              "                    oallowfullscreen=\"true\" msallowfullscreen=\"true\"\n",
              "                    style=\"width:100%; height:500px; border: 1px solid #DDD; overflow: hidden\"\n",
              "                    \n",
              "            >\n",
              "            </iframe>\n",
              "        \n",
              "            <script>\n",
              "                try {\n",
              "                  $(\"#ab8ca6d0-11b6-4c6b-95e6-b46e1d1d8f90\").bind('mousewheel', function(e) { e.preventDefault(); });\n",
              "                } catch (e) { console.error('exn catching scroll', e); }\n",
              "            </script>\n",
              "        "
            ]
          },
          "metadata": {},
          "execution_count": 24
        }
      ],
      "source": [
        "g1 = graphistry.nodes(winlogs)\n",
        "t=time()\n",
        "g11=g1.umap(feature_engine='cu_cat',engine='cuml',memoize=False)\n",
        "t2=time()-t\n",
        "print(t2)\n",
        "\n",
        "g11.plot()"
      ],
      "id": "fF4c1Fq4XAGr"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FQj6CzACstGG"
      },
      "source": [
        "## run loops per size and feature_engine, ie cu_cat and dirty_cat, remembering to set memoize=False"
      ],
      "id": "FQj6CzACstGG"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "_9dZ8SvQv7zf"
      },
      "outputs": [],
      "source": [
        "bench=pd.DataFrame(columns=['dataset','row','col','cc_time','dc_time'])\n",
        "# winlogsA=pd.read_parquet('part.88.parquet')\n",
        "winlogsA=winlogsA[winlogsA.columns.drop(['ProcessID','LogonID','Source','ParentProcessID']).tolist()]\n",
        "s=winlogsA.select_dtypes(include='int').columns\n",
        "winlogsA[s] = winlogsA[s].astype(\"float\")\n",
        "# winlogsA.fillna(0,inplace=True)\n",
        "# winlogsA.Time=pd.to_datetime(winlogsA.Time.astype(str).str[:6],errors='coerce')"
      ],
      "id": "_9dZ8SvQv7zf"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "c00b3f92-9e41-4e0d-82e5-70cc4fce5e91"
      },
      "outputs": [],
      "source": [
        "\n",
        "n=[2000,5000,10000,50000,500000]\n",
        "\n",
        "for ii,i in enumerate(n):\n",
        "    winlogs=winlogsA.sample(i,replace=False)#.fillna(0)\n",
        "\n",
        "    g0 = graphistry.nodes((winlogs))\n",
        "    t=time()\n",
        "    # try:\n",
        "    g00=g0.umap(feature_engine='cu_cat',engine='cuml',memoize=False,min_dist=0)\n",
        "    t1=time()-t\n",
        "    print('\\n',t1,'\\n')\n",
        "    # except:\n",
        "      # print('didnt run:',i,'/n')\n",
        "\n",
        "    g1 = graphistry.nodes(winlogs)\n",
        "    t=time()\n",
        "    g11=g1.umap(feature_engine='dirty_cat',engine='cuml',memoize=False,min_dist=0)\n",
        "    t2=time()-t\n",
        "    print('\\n',t2,'\\n')\n",
        "    bench.loc[1+bench.shape[0]]=['winlogs',winlogs.shape[0],winlogs.shape[1],t1,t2]\n",
        "    bench.to_csv('winlogs_'+str(gpu.strip())+'_wide_bench_'+timestr+'.txt',sep='\\t',mode='a')\n"
      ],
      "id": "c00b3f92-9e41-4e0d-82e5-70cc4fce5e91"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Fan7N-hlZD6L"
      },
      "outputs": [],
      "source": [
        "g00.addStyle(bg={'color': '#eee'}.plot()"
      ],
      "id": "Fan7N-hlZD6L"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jYsjKwZkskar"
      },
      "source": [
        "#plot results and export"
      ],
      "id": "jYsjKwZkskar"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "zpbJ4k2TkPhp"
      },
      "outputs": [],
      "source": [
        "bench=bench.rename(columns={'cc_time':'cu_cat','dc_time':'dirty_cat'})\n",
        "dd=bench.melt(id_vars=['dataset','row','col'])\n",
        "dd=dd[dd.dataset.str.contains('winlogs')]\n",
        "\n",
        "dd.value=dd.value.astype('float')\n",
        "dd.row=dd.row.astype('int')\n",
        "dd=dd.rename(columns={'row':'row_count','value':'time (sec)'})\n",
        "dd\n",
        "\n",
        "dd=dd[dd.row_count<1000000]\n",
        "dd=dd[dd.row_count>2000]\n",
        "\n",
        "\n",
        "plt.figure(figsize=(8, 6))\n",
        "\n",
        "import seaborn as sns\n",
        "\n",
        "capsize = .1\n",
        "\n",
        "ax = sns.barplot(data=dd,  x=\"row_count\", y=\"time (sec)\", hue=\"variable\", capsize=capsize)\n",
        "\n",
        "patches = ax.patches\n",
        "lines_per_err = 1 if capsize is None else 3\n",
        "ax.bar_label(ax.containers[0], fmt='%.1f')\n",
        "ax.bar_label(ax.containers[1], fmt='%.1f')\n",
        "\n",
        "for i, line in enumerate(ax.get_lines()):\n",
        "    newcolor = patches[i // lines_per_err]#.get_facecolor()\n",
        "    line.set_color('black')\n",
        "\n",
        "fig = ax.get_figure()\n",
        "fig.savefig('winlogs_'+str(gpu.strip())+'_bench_'+timestr+'.png')\n"
      ],
      "id": "zpbJ4k2TkPhp"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "7VzRYIAQ-nDD"
      },
      "outputs": [],
      "source": [
        "from google.colab import files\n",
        "files.download('winlogs_'+str(gpu.strip())+'_bench_'+timestr+'.txt')\n",
        "files.download('winlogs_'+str(gpu.strip())+'_bench_'+timestr+'.png')"
      ],
      "id": "7VzRYIAQ-nDD"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "NCiq7x5Vscqn"
      },
      "outputs": [],
      "source": [
        "bench"
      ],
      "id": "NCiq7x5Vscqn"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "oa5z3d07xSrb"
      },
      "outputs": [],
      "source": [
        "from importlib import reload\n",
        "# import\n",
        "reload(cu_cat)"
      ],
      "id": "oa5z3d07xSrb"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pnX4dxf2El1S"
      },
      "source": [
        "# with more columns"
      ],
      "id": "pnX4dxf2El1S"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "FYlJP98iRVAE"
      },
      "outputs": [],
      "source": [
        "\n",
        "n=[2000,10000,50000,100000,500000]\n",
        "\n",
        "import string, random\n",
        "def id_generator(size=12, chars=string.ascii_uppercase + string.digits + ' '):\n",
        "    return ''.join(random.choice(chars) for _ in range(size))\n",
        "\n",
        "for ii,i in enumerate(n):\n",
        "    winlogs=winlogsA.sample(i,replace=False)#.fillna(0)\n",
        "\n",
        "    # print(i)\n",
        "    # winlogs=winlogs[AA]\n",
        "    winlogs=winlogs[['EventID','LogHost','UserName','Time','DomainName','LogonType','SubjectLogonID','Status','Destination','ServiceName']]#,i]]\n",
        "    CC=winlogs.sample(20,replace=True,axis=1)\n",
        "\n",
        "    winlogs=pd.DataFrame((np.apply_along_axis(np.random.permutation, 0, CC)))\n",
        "    winlogs.columns=winlogs.columns.astype(str)\n",
        "\n",
        "    winlogs['str1'] = np.array([id_generator(10) for i in range(i)])#.reshape(-1,2)\n",
        "    words = [''.join(random.choice(string.ascii_uppercase + string.digits + ' ') for j in range(20)) for i in range(i)]\n",
        "    winlogs['str2']=words\n",
        "\n",
        "    g0 = graphistry.nodes((winlogs))\n",
        "    t=time()\n",
        "    # try:\n",
        "    g00=g0.umap(engine='cuda',memoize=False)\n",
        "    # g00=g0.umap(feature_engine='cu_cat',engine='cuml',memoize=False)\n",
        "    t1=time()-t\n",
        "    print('\\n',t1,'\\n')\n",
        "    # except:\n",
        "      # print('didnt run:',i,'/n')\n",
        "\n",
        "    g1 = graphistry.nodes(winlogs)\n",
        "    t=time()\n",
        "    g11=g1.umap(feature_engine='dirty_cat',engine='cuml',memoize=False)\n",
        "    t2=time()-t\n",
        "    print('\\n',t2,'\\n')\n",
        "    bench.loc[1+bench.shape[0]]=['winlogs',winlogs.shape[0],winlogs.shape[1],t1,t2]\n",
        "    bench.to_csv('winlogs_'+str(gpu.strip())+'_wide_bench_'+timestr+'.txt',sep='\\t',mode='a')\n",
        "#"
      ],
      "id": "FYlJP98iRVAE"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "rrL0vUAyH4NM"
      },
      "outputs": [],
      "source": [],
      "id": "rrL0vUAyH4NM"
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "provenance": [],
      "gpuClass": "premium",
      "gpuType": "T4",
      "include_colab_link": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.8.13"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 5
}