Example Usage.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from functools import partial\n",
    "import torch\n",
    "from torch.nn import CosineSimilarity\n",
    "import numpy as np\n",
    "\n",
    "from sentence_transformers import SentenceTransformer\n",
    "from metrics.model_free_metrics import *\n",
    "from embeddings.models import *\n",
    "from metrics.generalized_diversity import *\n",
    "\n",
    "\n",
    "device = 'cuda' if torch.cuda.is_available() else 'cpu'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "generations = ['1. \"The stars glittered in the night sky as the ship drifted silently through the void of space. Commander Jelena Banovic stood at the helm, her hands steady on the controls as she watched the endless expanse of stars passing by her window. She had been a traveler of the stars for many years now, and the excitement of venturing into the unknown still filled her with a sense of wonder. She knew that her mission was a great one, and she was determined to succeed.\"',\n",
    " '2. \"The sound of alarms filled the air as the ship lurched forward, the engines straining to keep up with the demands of the crew. Captain Zara Karim stood on the bridge, her eyes fixed on the vast display of stars beyond the viewport. This was the moment she had been waiting for, the moment she had been preparing for. It was time to leave the safety of Earth and venture into the unknown.\"',\n",
    " '3. \"The stars shone like a million tiny diamonds in the night sky as the ship hurtled through space. For the crew of the S.S. Star Seeker, the mission was simple: explore the far reaches of the universe and discover what lay beyond. On the bridge, Lieutenant Kaela Smith stood at the helm, her hands steady on the controls. She had been trained for this mission and was determined to do her best.\"',\n",
    " '4. \"The universe opened up in front of them, a vast expanse of stars and galaxies stretching out to infinity. Commander Luka Vukovic stood on the bridge of the Star Voyager, his hands moving swiftly over the controls as he plotted a course through the unknown. This was a mission of discovery and exploration, and he was determined to make the most of it.\"',\n",
    " '5. \"The void of space stretched out before them, a vast expanse of nothingness. Captain Reese Connors stood on the bridge of the U.S.S. Starfire, her hands steady on the controls as she plotted a course through the unknown. It was a mission of exploration and danger, and she was determined to succeed no matter what.\"',\n",
    " '6. \"The stars twinkled in the night sky as the ship drifted through the darkness. Lieutenant Maxine Stuart stood on the bridge, her eyes fixed on the viewport as she watched the stars pass by. She had been trained for this mission and was determined to make the most of it. Little did she know that this voyage would be full of danger and mystery.\"',\n",
    " '7. \"The stars shone like a million bright lights in the night sky as the ship hurtled through space. Commander Lucas Ramirez stood at the helm, his hands steady on the controls as he steered the ship towards its destination. This was a mission of exploration and danger, and he was determined to succeed no matter what.\"',\n",
    " '8. \"The stars stretched out before them, a seemingly endless expanse of distant suns and galaxies. Captain Gina Chen stood on the bridge of the Starcruiser, her hands moving swiftly over the controls as she plotted a course through the unknown. This was a mission of exploration and danger, and she was determined to make the most of it.\"',\n",
    " '9. \"The universe opened up before them like a vast expanse of stars and planets, beckoning them to explore. Commander Xander Ray stood on the bridge of the U.S.S. Star Seeker, his hands steady on the controls as he plotted a course through the unknown. This was a mission of exploration and discovery, and he was determined to make the most of it.\"',\n",
    " '10. \"The stars shone like a million tiny beacons in the night sky as the ship hurtled through space. Captain Ryan Moore stood on the bridge, his hands steady on the controls as he guided the ship through the unknown. This was a mission of exploration and danger, and he was determined to succeed no matter what.\"',\n",
    " '1. The future was changing, and no one was sure what to expect. The world was in chaos, and the government had no answers. People were scared, and some were even beginning to lose hope. But then something unexpected happened.',\n",
    " '2. Across the universe, a great wave of energy surged out, radiating a brilliant light. It was like nothing anyone had ever seen before. Astronomers were astounded, and scientists scrambled to try to understand what was happening.',\n",
    " '3. On Earth, it was felt everywhere. People stared up in wonder, and some even reported feeling the energy coursing through their veins. Eventually, it was determined that this wave of energy had originated from an unknown source, and it seemed to be growing stronger.',\n",
    " '4. At the same time, news of strange occurrences began to surface from all over the world. People were reporting sightings of unusual creatures, and others were experiencing inexplicable changes in their physical abilities.',\n",
    " '5. As the days passed, the wave of energy began to take on a more tangible form. It became clear that it had been sent to Earth for a purpose, and many began to suspect that it was some kind of message.',\n",
    " '6. A small group of brave scientists decided to investigate the situation further. They ventured into the unknown, searching for the source of the energy and unlocking its secrets.',\n",
    " '7. Little did they know, their journey would take them to the edges of the universe and beyond. They encountered strange new creatures and mysterious forces, all of which seemed to be connected to the wave of energy.',\n",
    " '8. Along the way, they discovered a powerful artifact, an ancient device designed to protect the Earth from a powerful threat. With the artifact in their possession, they set out to find the source of the energy and stop the impending danger.',\n",
    " \"9. But their mission was not without peril. They were pursued by a powerful enemy, determined to stop them from completing their task. In the end, the group's courage and determination would be tested to the limits.\",\n",
    " '10. With the fate of the world at stake, they must now embark on a daring journey across the universe to save humanity. What will they discover, and what dangers will they face? Only time will tell in this thrilling science fiction adventure.',\n",
    " '1. The sun shone brightly down on the barren landscape of the distant planet. It had been five years since the first settlers had touched down, and the colony had grown from a small outpost to bustling metropolis. Captain Shaela stood outside the entrance to the main city, surveying the progress that had been made. She had been part of the first wave of settlers, and the pride she felt for what they had accomplished was overwhelming.',\n",
    " '2. The heavy metallic door slammed shut behind her as she entered the control room. She took in the sight of the room, her eyes widening in awe. In the center of the room was a large console with hundreds of blinking lights and dials. The walls were lined with monitors and various other machines.',\n",
    " '3. Shaela took a deep breath and stepped forward, her heart pounding in her chest. She had been chosen to be the first human to explore the unknown planet. The mission was dangerous, but the potential rewards were too great to resist. She had to find a way to unlock the secrets of the planet and bring back something of value.',\n",
    " \"4. She made her way to the console and began the process of powering up the ship's systems. As she worked, she couldn't help but feel a sense of excitement. She was about to embark on a journey that could potentially change the course of human history.\",\n",
    " \"5. After a few minutes of preparation, she pushed the button to activate the main engine. With a loud rumble, the ship began to move forward, slowly gaining speed as it headed towards the unknown. As the ship neared the planet, the view out the window changed, and the planet's surface revealed itself.\",\n",
    " '6. Shaela had expected to find a barren wasteland, but instead she found a lush, verdant world filled with strange and unknown creatures. She was amazed at the sight, and she could not help but wonder what secrets this planet held. She was determined to find out.',\n",
    " '7. She spent the next few days exploring the planet, learning about its life forms and geography. As she made her way deeper into the planet, she discovered a strange energy emanating from the core of the planet. She realized that this energy was the key to unlocking the secrets of the planet.',\n",
    " '8. She returned to the ship and began to construct a device that would allow her to tap into the energy source. As she worked, she could feel the power of the energy calling out to her, and she felt a strange connection to the planet. She knew that she had made the right decision in coming here.',\n",
    " '9. After weeks of work, she finally finished building the device, and she was ready to make her move. She stepped back, took a deep breath, and activated the device. As the energy began to flow, she could feel the power coursing through her body.',\n",
    " '10. Suddenly, the planet began to shake violently, and she could feel the energy surging through her. She was filled with a strange, overwhelming feeling, and she knew that she had made a powerful connection to the planet. She could feel its secrets calling out to her, and she was determined to uncover them.',\n",
    " '1. It was a dark and stormy night when the spaceship first arrived. The alien vessel seemed to defy the laws of physics as it descended from the sky, its hull glowing brightly in the light of the moon. The ship was huge, dwarfing the nearby buildings, and it was accompanied by a deafening roar that echoed through the city.',\n",
    " '2. Captain Alexander Jansen and his crew were the first to arrive on the scene, and the sight of the alien ship filled them with a sense of awe and excitement. They had never seen anything like it before, and it was clear that this was no ordinary spacecraft.',\n",
    " '3. The ship was unlike anything they had ever seen before. Its hull was made of a material that seemed to absorb the light around it, but still glowed with an eerie, almost otherworldly light. It was clear that this was a ship from beyond the stars.',\n",
    " '4. Jansen and his crew cautiously approached the ship, not knowing what to expect. As they got closer, they noticed strange symbols and markings on the hull, as if someone had been trying to communicate with them.',\n",
    " '5. Suddenly, they heard a voice coming from the ship. It was a voice unlike any they had ever heard before, speaking in an unfamiliar language. It was clear that they were not alone.',\n",
    " '6. The crew cautiously made their way aboard the ship, unsure of what they would find. The interior of the ship was unlike anything they had ever seen before. The walls were lined with strange devices and glowing screens, and the air was filled with the hum of some unknown energy source.',\n",
    " '7. As they explored the ship, they discovered a strange chamber filled with strange artifacts. It was clear that the ship had been on some kind of mission, but what that mission was remained a mystery.',\n",
    " '8. Suddenly, the crew heard a loud noise coming from outside the ship. When they looked out the windows, they saw a fleet of alien vessels approaching. It was clear that they had been followed.',\n",
    " '9. Jansen and his crew had no choice but to try and outrun the alien fleet. With no time to lose, they set a course for the nearest star system and engaged the ship’s powerful engines.',\n",
    " '10. As the ship raced through space, the crew encountered strange and dangerous creatures, traveled to distant planets, and encountered powerful alien civilizations. They had no idea what was in store for them, but one thing was certain: their journey was just beginning.']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Normalized Compression Distance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average NCD is: 0.7381337543816338\n"
     ]
    }
   ],
   "source": [
    "ncds = get_pairwise_ncd(generations)\n",
    "\n",
    "print(\"Average NCD is: {}\".format(sum(ncds) / len(ncds)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# NGram Overlap Similarity"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The average 2-ngram Jaccard Similarity is: 0.05517241379310345\n"
     ]
    }
   ],
   "source": [
    "n=2 \n",
    "\n",
    "x, y = generations[0], generations[1]\n",
    "\n",
    "overlap = pairwise_ngram(n, x, y)\n",
    "\n",
    "print(\"The average {}-ngram Jaccard Similarity is: {}\".format(n, overlap))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Cosine Similarity"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The average pairwise cosine similarity is: 0.4148975610733032\n"
     ]
    }
   ],
   "source": [
    "sent_embedder = SBERTEmbedder()\n",
    "sent_embedder.model = sent_embedder.model.to(device)\n",
    "\n",
    "avg_sim = get_avg_cosine_sim(sent_embedder, generations)\n",
    "\n",
    "print(\"The average pairwise cosine similarity is: {}\".format(avg_sim))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Order *q* Diversity "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The diversity of order 2 for NCD-based similarity is: 1.4135637107558692\n",
      "The diversity of order 3 for NCD-based similarity is: 1.3925950120678359\n",
      "The diversity of order 4 for NCD-based similarity is: 1.3886314242168292\n",
      "The diversity of order 5 for NCD-based similarity is: 1.3908361377188092\n",
      "The diversity of order 6 for NCD-based similarity is: 1.367049263237418\n",
      "The diversity of order 2 for bigram Jaccard similarity is: 27.028350843791586\n",
      "The diversity of order 3 for bigram Jaccard similarity is: 30.16624345821143\n",
      "The diversity of order 4 for bigram Jaccard similarity is: 36.24725477927909\n",
      "The diversity of order 5 for bigram Jaccard similarity is: 46.115594897711624\n",
      "The diversity of order 6 for bigram Jaccard similarity is: 35.48591137193012\n",
      "The diversity of order 2 for cosine similarity is: 2.004205168379168\n",
      "The diversity of order 3 for cosine similarity is: 2.3266738047190403\n",
      "The diversity of order 4 for cosine similarity is: 2.220990352180027\n",
      "The diversity of order 5 for cosine similarity is: 2.3086272502498058\n",
      "The diversity of order 6 for cosine similarity is: 2.28618610275597\n"
     ]
    }
   ],
   "source": [
    "qs = [2,3,4,5,6]\n",
    "num_samples = 50\n",
    "\n",
    "# normalized compression distance based\n",
    "bigram_partial = partial(pairwise_ngram, 2)\n",
    "cosine_partial = partial(get_pairwise_cosine_sim, sent_embedder)\n",
    "\n",
    "for q in qs:\n",
    "    ncd_q = diversity_order_q(generations, q, ncd, num_samples)\n",
    "    print(\"The diversity of order {} for NCD-based similarity is: {}\".format(q, ncd_q))\n",
    "\n",
    "for q in qs:\n",
    "    bigram_q = diversity_order_q(generations, q, bigram_partial, num_samples)\n",
    "    print(\"The diversity of order {} for bigram Jaccard similarity is: {}\".format(q, bigram_q))\n",
    "\n",
    "for q in qs:\n",
    "    cosine_q = diversity_order_q(generations, q, cosine_partial, num_samples)\n",
    "    print(\"The diversity of order {} for cosine similarity is: {}\".format(q, cosine_q))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Few Shot",
   "language": "python",
   "name": "few_shot_word_learning"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}