diff --git a/samples/Bigtable_KNN_Example.ipynb b/samples/Bigtable_KNN_Example.ipynb new file mode 100644 index 000000000..68fe8992f --- /dev/null +++ b/samples/Bigtable_KNN_Example.ipynb @@ -0,0 +1,406 @@ +{ + "nbformat": 4, + "nbformat_minor": 0, + "metadata": { + "colab": { + "provenance": [] + }, + "kernelspec": { + "name": "python3", + "display_name": "Python 3" + }, + "language_info": { + "name": "python" + } + }, + "cells": [ + { + "cell_type": "markdown", + "source": [ + "Copyright 2024 Google LLC.
\n", + "SPDX-License-Identifier: Apache-2.0" + ], + "metadata": { + "id": "wsibGZVPWaxz" + } + }, + { + "cell_type": "code", + "source": [ + "#@title Default title text\n", + "# Licensed under the Apache License, Version 2.0 (the \"License\");\n", + "# you may not use this file except in compliance with the License.\n", + "# You may obtain a copy of the License at\n", + "#\n", + "# https://www.apache.org/licenses/LICENSE-2.0\n", + "#\n", + "# Unless required by applicable law or agreed to in writing, software\n", + "# distributed under the License is distributed on an \"AS IS\" BASIS,\n", + "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n", + "# See the License for the specific language governing permissions and\n", + "# limitations under the License." + ], + "metadata": { + "id": "noEonOrgWzjv" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "source": [ + "

Step 0: Set variables for your Bigtable instance

\n", + "If you do not yet have a Bigtable instance, please see creating an instance" + ], + "metadata": { + "id": "jxrtdYDNfNxD" + } + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "FQScT_Dd43cF" + }, + "outputs": [], + "source": [ + "PROJECT_ID = 'google.com:cloud-bigtable-dev'\n", + "INSTANCE_ID = 'crosbie-instance'\n", + "TABLE_ID = 'knn_intro'" + ] + }, + { + "cell_type": "code", + "source": [ + "#if running from colab, you will also need to authenticate the project id\n", + "from google.colab import auth\n", + "auth.authenticate_user(project_id=PROJECT_ID)" + ], + "metadata": { + "id": "tYNnjiyjchPg" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "source": [ + "

Step 1: Create a table to store the text, embeddings and search phrase

\n" + ], + "metadata": { + "id": "a9DFEgtbfjRC" + } + }, + { + "cell_type": "code", + "source": [ + "from google.cloud import bigtable\n", + "from google.cloud.bigtable import column_family\n", + "\n", + "client = bigtable.Client(project=PROJECT_ID, admin=True)\n", + "instance = client.instance(INSTANCE_ID)\n", + "table = instance.table(TABLE_ID)\n", + "\n", + "column_families = {\"docs\":column_family.MaxVersionsGCRule(2), \"search_phrase\":column_family.MaxVersionsGCRule(2)}\n", + "\n", + "if not table.exists():\n", + " table.create(column_families=column_families)\n", + "else:\n", + " print(\"Table already exists\")" + ], + "metadata": { + "id": "x6A26bH148h9" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "source": [ + "

Step 2: Embed texts with a pre-trained, foundational model from Vertex

\n", + "\n", + "Next, generate the text and embeddings which you will store in Bigtable along with the associated keys.\n", + "\n", + "For full documentation, please see get text embeddings or [get multimodal embeddings](https://cloud.google.com/vertex-ai/generative-ai/docs/embeddings/get-multimodal-embeddings).\n" + ], + "metadata": { + "id": "zqzV5RyJiucT" + } + }, + { + "cell_type": "code", + "source": [ + "from typing import List, Optional\n", + "from vertexai.language_models import TextEmbeddingInput, TextEmbeddingModel\n", + "from vertexai.generative_models import GenerativeModel\n", + "\n", + "#defines which LLM that we should use to generate the text\n", + "model = GenerativeModel(\"gemini-1.5-pro-001\")\n", + "\n", + "#first we will use generative AI to create a list of 10 chunks for phrases\n", + "#This can be replaced with a static list of text items or your own data\n", + "chunks = []\n", + "for i in range(10):\n", + " response = model.generate_content(\n", + " \"Generate a paragraph between 10 and 20 words that is about about either Bigtable or Generative AI\"\n", + " )\n", + " chunks.append(response.text)\n", + " print(response.text)\n", + "\n", + "#create embeddings for the chunks of text\n", + "def embed_text(\n", + " texts: List[str] = chunks,\n", + " task: str = \"RETRIEVAL_DOCUMENT\",\n", + " model_name: str = \"text-embedding-004\",\n", + " dimensionality: Optional[int] = 128,\n", + ") -> List[List[float]]:\n", + " \"\"\"Embeds texts with a pre-trained, foundational model.\"\"\"\n", + " model = TextEmbeddingModel.from_pretrained(model_name)\n", + " inputs = [TextEmbeddingInput(text, task) for text in texts]\n", + " kwargs = dict(output_dimensionality=dimensionality) if dimensionality else {}\n", + " embeddings = model.get_embeddings(inputs, **kwargs)\n", + " return [embedding.values for embedding in embeddings]\n", + "\n", + "embeddings = embed_text()\n", + "print(\"embeddings created for text phrases\")" + ], + "metadata": { + "id": "sNP6RGdQLMef", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "7c1fc46a-634c-4fae-c732-987c2255e561" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "Generative AI crafts realistic new content, like text or images, by learning patterns from existing data. \n", + "\n", + "Generative AI models learn patterns from data to create new content like text, images, and even music. \n", + "\n", + "Bigtable efficiently stores and manages massive datasets, enabling real-time data access and analysis. \n", + "\n", + "Generative AI models learn from existing data to create new, original content like text, images, or music. \n", + "\n", + "Generative AI models learn from vast datasets to create new, original content like text, images, and music. \n", + "\n", + "Bigtable, Google's distributed NoSQL database, handles massive datasets with high performance and scalability. \n", + "\n", + "Bigtable, Google's distributed NoSQL database, handles massive datasets with high performance and scalability. \n", + "\n", + "Generative AI models, like ChatGPT, learn from data to create new content, including text, images, and even music. \n", + "\n", + "Generative AI models learn from data to create new, original content like text, images, and even music. \n", + "\n", + "Bigtable efficiently stores massive datasets, enabling real-time access for large-scale applications. \n", + "\n", + "embeddings created for text phrases\n" + ] + } + ] + }, + { + "cell_type": "code", + "source": [ + "#create embeddings for the chunks of text\n", + "def embed_text(\n", + " texts: List[str] = chunks,\n", + " task: str = \"RETRIEVAL_DOCUMENT\",\n", + " model_name: str = \"text-embedding-004\",\n", + " dimensionality: Optional[int] = 128,\n", + ") -> List[List[float]]:\n", + " \"\"\"Embeds texts with a pre-trained, foundational model.\"\"\"\n", + " model = TextEmbeddingModel.from_pretrained(model_name)\n", + " inputs = [TextEmbeddingInput(text, task) for text in texts]\n", + " kwargs = dict(output_dimensionality=dimensionality) if dimensionality else {}\n", + " embeddings = model.get_embeddings(inputs, **kwargs)\n", + " return [embedding.values for embedding in embeddings]\n", + "\n", + "embeddings = embed_text()\n", + "print(embeddings)" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "-PjFk4BmMpLf", + "outputId": "6cd85f5e-344d-4035-c23d-4848a814e460" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "[[-0.08450032770633698, 0.02769993618130684, -0.03482632339000702, -0.017077140510082245, 0.019390733912587166, 0.04392266273498535, 0.03357764706015587, 0.015328948386013508, 0.006103283725678921, -0.014262494631111622, -0.024556264281272888, 0.016103453934192657, 0.02031383290886879, 0.011007575318217278, 0.03205610811710358, -0.07855264097452164, 0.030707791447639465, 0.0038270477671176195, -0.1282244324684143, 0.027971018105745316, 0.0429173968732357, -0.012058214284479618, -0.020504970103502274, -0.061760395765304565, -0.04263300448656082, 0.015370801091194153, 0.05843237414956093, 0.01672925055027008, 0.007277846336364746, -0.019984232261776924, 0.02035752683877945, 0.09591261297464371, 0.026950383558869362, -0.03744083270430565, -0.0034160178620368242, -0.04068847373127937, -0.007037016097456217, -0.0026350365951657295, 0.05283447727560997, -0.04651201516389847, -0.04621433466672897, 0.023651549592614174, -0.021708453074097633, 0.043830644339323044, 0.011450933292508125, -0.011279990896582603, 0.012341639958322048, 0.05122089758515358, -0.05696691945195198, 0.06189574673771858, 0.008550770580768585, -0.03774706646800041, -0.024338068440556526, -0.005989800672978163, -0.028971556574106216, -0.022401073947548866, -0.045650333166122437, 0.0029600164853036404, 0.062295567244291306, 0.004726134706288576, -0.04207140579819679, 0.007950368337333202, -0.026424679905176163, -0.020298607647418976, -0.0049337767995893955, -0.0019005639478564262, -0.022663593292236328, 0.008831997402012348, -0.0662795826792717, 0.023682232946157455, 0.01763196475803852, 0.018057800829410553, -0.040720704942941666, 0.03145000711083412, 0.00712017435580492, -0.017980607226490974, -0.031269967555999756, -0.056338123977184296, -0.04399557784199715, 0.05046271160244942, 0.0005845706909894943, 0.02893402799963951, 0.06699441373348236, 0.052571818232536316, -0.001466421177610755, -0.013394299894571304, 0.02678605541586876, -0.09429049491882324, -0.104759581387043, -0.03622598573565483, 0.08944173902273178, -0.012836132198572159, 0.020250121131539345, -0.022795412689447403, 0.03042852133512497, -0.013504283502697945, -0.05681625381112099, -0.11831095069646835, 0.05893881991505623, 0.044461678713560104, 0.012705438770353794, -0.008877890184521675, 0.02256305143237114, -0.06481199711561203, 0.07357621192932129, 0.07637888193130493, -0.001022631535306573, 0.003012050874531269, -0.04501718655228615, 0.011580892838537693, -0.011630089022219181, -0.05957183614373207, 0.00027843835414387286, -0.01866539753973484, -0.021486755460500717, 0.017088083550333977, -0.08996545523405075, 0.01969411037862301, -0.031917400658130646, -0.001033484353683889, -0.01838698238134384, -0.0007348550716415048, -0.022319208830595016, 0.04706244915723801, 0.05763157457113266, -0.0074624838307499886, 0.044900424778461456, -0.020464489236474037], [0.03288421779870987, 0.021993177011609077, -0.03025340475142002, 0.012661152519285679, 0.02722495049238205, 0.04671330749988556, 0.057840850204229355, -0.03770286217331886, -0.0038906517438590527, 0.02690144255757332, -0.03656824305653572, 0.026921013370156288, 0.034005362540483475, -0.0001731609954731539, -7.00462405802682e-05, -0.045175276696681976, 0.020177461206912994, 0.0023166921455413103, -0.03180791810154915, -0.009655234403908253, 0.0014677854487672448, -0.018724599853157997, 0.0147550692781806, -0.019778424873948097, -0.0278276689350605, -0.019930537790060043, 0.00431312620639801, -0.009892669506371021, 0.010262960568070412, 0.015673425048589706, 0.028375502675771713, 0.02992774173617363, 0.007420352194458246, -0.08993220329284668, -0.01199731882661581, 0.003996206447482109, 0.004132588393986225, 0.05582793429493904, 0.05872391536831856, -0.046186402440071106, -0.006283359136432409, 0.027917305007576942, -0.0626022145152092, 0.006784149445593357, -0.02458398975431919, 0.047997210174798965, 0.021437611430883408, 0.054494425654411316, -0.06377507746219635, 0.04894367605447769, -0.004984859377145767, 0.010942202061414719, -0.023535825312137604, 0.032498523592948914, -0.014414998702704906, -0.008071021176874638, -0.025653431192040443, -0.04322294890880585, 0.0002405418490525335, -0.0054001472890377045, -0.024918189272284508, -0.026027025654911995, 0.00011138208355987445, -0.004568804521113634, 0.004761930089443922, -0.01461446937173605, -0.024787141010165215, -0.05311122536659241, -0.05162893235683441, 0.0058540538884699345, 5.0478585762903094e-05, 0.016072731465101242, -0.0036017929669469595, 0.018004992976784706, -0.029895668849349022, 0.015325338579714298, 0.0032211083453148603, -0.036512333899736404, -0.045434292405843735, 0.05004606395959854, -0.037314463406801224, 0.030334999784827232, 0.045979611575603485, 0.04254082217812538, 0.008182959631085396, 0.00841519795358181, 0.008615066297352314, -0.05308935046195984, -0.033391524106264114, -0.01721903495490551, 0.08515728265047073, -0.011648016981780529, 0.03834306076169014, -0.002702154917642474, 0.005585528444498777, -0.06765760481357574, -0.13593952357769012, -0.1353914588689804, 0.1036500334739685, 0.039072077721357346, -0.017012016847729683, 0.03396797925233841, 0.0017852389719337225, -0.038167741149663925, 0.024172846227884293, 0.0231820996850729, 0.02442244440317154, -0.05536488816142082, -0.009793227538466454, 0.023833122104406357, -0.019103558734059334, -0.04244992882013321, 0.008560619316995144, -0.02769671566784382, 0.03391335904598236, 0.02959752455353737, 0.0006119771860539913, 0.02249145694077015, -0.004586266819387674, -0.019061578437685966, 0.010476240888237953, 0.022111613303422928, -0.02050863206386566, 0.021454667672514915, 0.04359370097517967, -0.0181434266269207, -0.00985880009829998, -0.022338978946208954], [-0.06078857183456421, 0.02353990636765957, -0.026801450178027153, -0.02456226944923401, -0.00843739788979292, 0.021139580756425858, 0.02972477301955223, 0.010893899947404861, 0.009643233381211758, -0.006152538117021322, -0.032207656651735306, 0.02367754653096199, 0.0531800352036953, 0.032293036580085754, 0.027034293860197067, -0.058666449040174484, 0.0419318713247776, -0.0018733846955001354, -0.13301880657672882, 0.03391027823090553, 0.03569146245718002, -0.021888302639126778, 0.0010070157004520297, -0.03804202005267143, -0.037917107343673706, 0.020295685157179832, 0.057564619928598404, 0.03935980796813965, 0.0034540544729679823, -0.020403947681188583, 0.030477864667773247, 0.08062468469142914, 0.0324820950627327, -0.029033053666353226, 0.009409772232174873, -0.048055071383714676, -0.02163645811378956, -0.005604191683232784, 0.029310384765267372, -0.05573151260614395, -0.0385185182094574, 0.023786641657352448, -0.03459029272198677, 0.07740714401006699, -0.012590926140546799, 0.010781713761389256, 0.019386928528547287, 0.04888157546520233, -0.03601633384823799, 0.0733419731259346, -0.004087223671376705, -0.05005636811256409, -0.02504015527665615, -0.009126170538365841, -0.03667687624692917, -0.0022418454755097628, -0.02991626039147377, 0.028756598010659218, 0.03364890068769455, -0.008844790048897266, -0.042354002594947815, 0.013724589720368385, -0.020221484825015068, -0.022553326562047005, 0.012340106070041656, -0.027229873463511467, 0.007000615820288658, -0.00437335018068552, -0.08747163414955139, 0.05686615779995918, -0.014788887463510036, 0.008272969163954258, -0.03795122355222702, 0.016777878627181053, 0.0202676709741354, -0.016952302306890488, -0.02430984191596508, -0.03832195699214935, -0.03968500345945358, 0.04166620597243309, 0.008372534066438675, 0.01462782546877861, 0.06235484778881073, 0.037777651101350784, 0.017406562343239784, -0.020295578986406326, 0.024819955229759216, -0.08151800185441971, -0.11668550968170166, -0.0042527480982244015, 0.11537011712789536, -0.014892863109707832, 0.01606971211731434, -0.027070200070738792, 0.03049718402326107, -0.008170733228325844, -0.052861008793115616, -0.09197192639112473, 0.09827377647161484, 0.045810356736183167, 0.020268719643354416, -0.00799744576215744, 0.0012385305017232895, -0.06687118113040924, 0.057647742331027985, 0.08168042451143265, -0.01628391072154045, -0.00285937893204391, -0.04744301363825798, 0.03996937721967697, -0.00254423706792295, -0.06174959987401962, 0.007929063402116299, -0.01338979322463274, -0.017765434458851814, 0.012333503924310207, -0.07243526726961136, 0.013874292373657227, -0.0407545268535614, 0.00034642466926015913, -0.016636865213513374, 0.0032880553044378757, -0.012837821617722511, 0.02665996551513672, 0.050262343138456345, 0.017946038395166397, 0.0357295460999012, -0.007143475115299225], [-0.06847218424081802, 0.036874957382678986, -0.03248230740427971, -0.03005390241742134, -0.0008860180387273431, 0.02493991144001484, 0.03015642799437046, 0.007136846426874399, 0.015834499150514603, -0.011635983362793922, -0.023426681756973267, 0.006363945081830025, 0.026770692318677902, 0.019747737795114517, 0.025018030777573586, -0.06848934292793274, 0.03224615752696991, 0.008790278807282448, -0.12833142280578613, 0.03342564031481743, 0.048698149621486664, -0.016804667189717293, -0.01981537975370884, -0.056901633739471436, -0.03598572313785553, 0.018673690035939217, 0.057080164551734924, 0.03393591195344925, 0.004871944896876812, -0.021360969170928, 0.024444352835416794, 0.0941895917057991, 0.03185967728495598, -0.04133514687418938, 0.0033061117865145206, -0.04439851641654968, -0.013959058560431004, 0.006056260317564011, 0.05369942635297775, -0.03256845846772194, -0.03505531698465347, 0.016922397539019585, -0.029344355687499046, 0.04227321222424507, 0.003560128388926387, -0.007798419799655676, 0.0052518099546432495, 0.05723845586180687, -0.050699613988399506, 0.06423664093017578, 0.00340746296569705, -0.04029220715165138, -0.02391812764108181, -0.007777650840580463, -0.02907698228955269, -0.012861157767474651, -0.04880594089627266, 0.02368948608636856, 0.050997089594602585, 0.008533514104783535, -0.05147513374686241, 0.01922735385596752, -0.03458172827959061, -0.027080422267317772, -0.012557431124150753, -0.004578156862407923, -0.0095704710111022, -0.001662466791458428, -0.06414275616407394, 0.045664675533771515, -0.005258599296212196, 0.00933118350803852, -0.04936465620994568, 0.02951420657336712, 0.023292923346161842, -0.023346830159425735, -0.04170457273721695, -0.0585801899433136, -0.04210037365555763, 0.036277830600738525, 0.008482322096824646, 0.02095283567905426, 0.057119227945804596, 0.04447251558303833, 4.884347799816169e-05, -0.009244076907634735, 0.02255321852862835, -0.08583744615316391, -0.11415613442659378, -0.01635848730802536, 0.09547705203294754, -0.015148086473345757, 0.02819075994193554, -0.0164557546377182, 0.02593742311000824, -0.014737357385456562, -0.04345235601067543, -0.10204050689935684, 0.0735623762011528, 0.04983805865049362, 0.025026727467775345, -0.0007459582411684096, 0.025268452242016792, -0.06466612219810486, 0.07015109062194824, 0.08282256126403809, -0.0054011777974665165, -0.00392462033778429, -0.04952109232544899, 0.026064403355121613, -0.011867979541420937, -0.07737120985984802, 0.001671997713856399, -0.012132421135902405, -0.024170251563191414, 0.014405445195734501, -0.09126658737659454, 0.016359098255634308, -0.033928073942661285, 0.00260285590775311, -0.011677633039653301, 0.012707340531051159, -0.012549111619591713, 0.055193059146404266, 0.047120485454797745, 0.004867863841354847, 0.039733391255140305, -0.012745831161737442], [-0.0832604467868805, 0.023033130913972855, -0.03591461479663849, -0.01631186529994011, 0.011968880891799927, 0.045666102319955826, 0.034957677125930786, 0.026531314477324486, 0.008142765611410141, -0.011480225250124931, -0.02300768531858921, 0.014716006815433502, 0.01822371408343315, 0.0065570068545639515, 0.028499536216259003, -0.07088254392147064, 0.030843306332826614, 0.0031396127305924892, -0.12400203943252563, 0.02994600497186184, 0.03872041776776314, -0.012828000821173191, -0.02136680670082569, -0.05074867233633995, -0.046846482902765274, 0.020341556519269943, 0.05183933675289154, 0.016978073865175247, 0.007244898937642574, -0.025475045666098595, 0.015436332672834396, 0.0951925739645958, 0.025686295703053474, -0.03453615680336952, -0.005534656345844269, -0.035657383501529694, -0.008774839341640472, -0.0028547707479447126, 0.05334746837615967, -0.03988794982433319, -0.05051686614751816, 0.021731045097112656, -0.02170310541987419, 0.04366104304790497, 0.005206139758229256, -0.011328227818012238, 0.01625741273164749, 0.05026169493794441, -0.05536627396941185, 0.07158339023590088, 0.005935680121183395, -0.03491698205471039, -0.02047242596745491, -0.0067781610414385796, -0.02696160040795803, -0.02352280355989933, -0.04829324781894684, 0.009076814167201519, 0.06266755610704422, 0.007700574584305286, -0.043075744062662125, 0.014500418677926064, -0.01987326145172119, -0.021106308326125145, -0.0009128159144893289, -0.00857322383671999, -0.023177454248070717, 0.009942266158759594, -0.07126551866531372, 0.02484099380671978, 0.01721320115029812, 0.028079017996788025, -0.04213978722691536, 0.03654175624251366, 0.00980630423873663, -0.012015646323561668, -0.0262577123939991, -0.057999081909656525, -0.03861434385180473, 0.04284786060452461, -0.0015353847993537784, 0.026851776987314224, 0.06184181198477745, 0.05713065341114998, 0.0008666547946631908, -0.013353238813579082, 0.03172348067164421, -0.08712438493967056, -0.10618459433317184, -0.03539332002401352, 0.08821821212768555, -0.015110976062715054, 0.015672672539949417, -0.022111622616648674, 0.0349217914044857, -0.01453952956944704, -0.051970191299915314, -0.12357059866189957, 0.0652945339679718, 0.040103960782289505, 0.0192596223205328, -0.007036934606730938, 0.023827141150832176, -0.074575275182724, 0.06680455803871155, 0.06418177485466003, 0.0014870514860376716, -0.0018351537873968482, -0.045457590371370316, 0.008453689515590668, -0.004408234730362892, -0.0644928514957428, -0.0035922881215810776, -0.016479969024658203, -0.02102629281580448, 0.01869354583323002, -0.09165934473276138, 0.016320200636982918, -0.03199587017297745, 0.0033065348397940397, -0.021047838032245636, -0.0035949803423136473, -0.020854176953434944, 0.04181322082877159, 0.053498733788728714, -0.0034310887567698956, 0.0392974354326725, -0.023559285327792168], [-0.07055750489234924, 0.027071231976151466, -0.038349080830812454, -0.023176107555627823, 0.0029575005173683167, 0.03386880084872246, 0.0349406860768795, 0.01590549945831299, 0.011641228571534157, -0.012463110499083996, -0.022392451763153076, 0.012984903529286385, 0.022389719262719154, 0.012247142381966114, 0.02707509510219097, -0.06478013843297958, 0.02837267331779003, 0.013527167961001396, -0.12594962120056152, 0.03217228129506111, 0.04138890653848648, -0.01697699911892414, -0.018982520326972008, -0.05657929182052612, -0.032851316034793854, 0.02307913452386856, 0.049700263887643814, 0.028120847418904305, 0.011450382880866528, -0.019710861146450043, 0.025761103257536888, 0.09929323196411133, 0.030499381944537163, -0.04159565642476082, -0.008390258066356182, -0.042051855474710464, -0.013964364305138588, 0.004095571115612984, 0.05775759369134903, -0.034667521715164185, -0.03500237688422203, 0.01607472635805607, -0.03643927350640297, 0.038169704377651215, 0.004736537113785744, -0.011048507876694202, 0.015365851111710072, 0.05972721055150032, -0.054701048880815506, 0.06760497391223907, 0.001888061291538179, -0.034801557660102844, -0.021543806418776512, -0.005267709959298372, -0.03099055588245392, -0.021652787923812866, -0.04248255118727684, 0.020567620173096657, 0.056372903287410736, 0.004778227303177118, -0.05434839427471161, 0.012843675911426544, -0.027429603040218353, -0.023911045864224434, -0.004342452622950077, -0.0014171571237966418, -0.01252778060734272, -0.0016842037439346313, -0.0684569776058197, 0.04337219521403313, 0.013265795074403286, 0.02245275117456913, -0.05001119151711464, 0.037518057972192764, 0.011877535842359066, -0.016742300242185593, -0.03249656781554222, -0.05364057049155235, -0.040473055094480515, 0.03383515402674675, -0.00238224770873785, 0.017812510952353477, 0.0581229068338871, 0.04472169652581215, -0.0007741840090602636, -0.014260964468121529, 0.029139330610632896, -0.08957478404045105, -0.10135997086763382, -0.03225848451256752, 0.08963300287723541, -0.012187780812382698, 0.02038552053272724, -0.017418796196579933, 0.03961922600865364, -0.018883951008319855, -0.050678420811891556, -0.11472823470830917, 0.06151053309440613, 0.04887017607688904, 0.02159186080098152, -0.0012482781894505024, 0.025494839996099472, -0.06913146376609802, 0.06998679041862488, 0.08258551359176636, -0.0037160301581025124, -0.00446908175945282, -0.04812660068273544, 0.021729670464992523, -0.004123028367757797, -0.07107995450496674, -0.007254302967339754, -0.013075150549411774, -0.01914948783814907, 0.016733631491661072, -0.09204697608947754, 0.012139315716922283, -0.03352804109454155, 0.005945014767348766, -0.011651512235403061, 0.0054581924341619015, -0.011640381067991257, 0.049067676067352295, 0.046512771397829056, 0.0006562298513017595, 0.04488394036889076, -0.01862006075680256], [-0.06750629097223282, 0.029979834333062172, -0.023873407393693924, -0.02551955357193947, 0.009883342310786247, 0.025899523869156837, 0.02706289291381836, 0.009603536687791348, 0.009308427572250366, -0.007993200793862343, -0.027766801416873932, 0.006203698925673962, 0.03445429354906082, 0.019654816016554832, 0.0226555448025465, -0.06803430616855621, 0.02664882317185402, 0.00851901713758707, -0.1326347440481186, 0.047421060502529144, 0.04709175229072571, -0.018050147220492363, -0.014436399564146996, -0.05561348795890808, -0.031165681779384613, 0.012248166836798191, 0.06275199353694916, 0.03582436218857765, 0.0036465281154960394, -0.015399710275232792, 0.013696656562387943, 0.08810453116893768, 0.02803228422999382, -0.03619308024644852, -0.009287182241678238, -0.04349767044186592, -0.007773283869028091, 0.005181537475436926, 0.05023975670337677, -0.047019120305776596, -0.03822415694594383, 0.028475500643253326, -0.027596978470683098, 0.04917384311556816, 0.009291652590036392, -0.0038874768652021885, 0.010509932413697243, 0.06264537572860718, -0.050245799124240875, 0.061208512634038925, 0.006937152706086636, -0.04229504242539406, -0.03388851881027222, 0.0010995973134413362, -0.0363873690366745, 0.0002450027968734503, -0.04007129371166229, 0.018388964235782623, 0.04647867754101753, 0.007110332138836384, -0.0579921193420887, 0.018296794965863228, -0.032156042754650116, -0.01751009374856949, -0.006702065467834473, -0.01671420969069004, -0.01649457961320877, -0.011779953725636005, -0.06279963254928589, 0.03424787148833275, -0.0078303636983037, -0.0017846496775746346, -0.04392112046480179, 0.03025701828300953, 0.014966514892876148, -0.020343557000160217, -0.033018745481967926, -0.06167527288198471, -0.04157835617661476, 0.044325925409793854, 0.0032672183588147163, 0.018025686964392662, 0.060150668025016785, 0.03744037076830864, 0.003084528259932995, -0.014236882328987122, 0.02651219628751278, -0.08444149047136307, -0.11070699989795685, -0.018889205530285835, 0.10296771675348282, -0.017088046297430992, 0.027013737708330154, -0.025094732642173767, 0.02491964027285576, -0.018686963245272636, -0.05249706283211708, -0.09722930938005447, 0.07181468605995178, 0.05524546280503273, 0.026743784546852112, -0.007950828410685062, 0.016561457887291908, -0.06991297751665115, 0.06963112205266953, 0.08972373604774475, -0.007289609406143427, -0.0053421612828969955, -0.05248326063156128, 0.03286239132285118, -0.0104910833761096, -0.053850460797548294, 0.003133697435259819, -0.007846723310649395, -0.02927633561193943, 0.00847103726118803, -0.08742033690214157, 0.008551081642508507, -0.0336105078458786, 0.008995037525892258, -0.004666196182370186, 0.012850038707256317, -0.017499320209026337, 0.05303291603922844, 0.04805891215801239, -0.0009477857383899391, 0.04886544868350029, -0.009903490543365479], [-0.08802168071269989, 0.027025088667869568, -0.031541548669338226, -0.02136027254164219, 0.01425863616168499, 0.038948021829128265, 0.035522278398275375, 0.01463734358549118, 0.006829730700701475, -0.017494836822152138, -0.022795986384153366, 0.014753896743059158, 0.0195468682795763, 0.006471413653343916, 0.03045477345585823, -0.07581829279661179, 0.03045738860964775, 0.0012872344814240932, -0.12901705503463745, 0.03001341223716736, 0.03933470696210861, -0.016197605058550835, -0.025079898536205292, -0.061596859246492386, -0.04062305763363838, 0.012969451025128365, 0.061685871332883835, 0.023175720125436783, 0.007855902425944805, -0.015420561656355858, 0.019728761166334152, 0.09216408431529999, 0.024477960541844368, -0.03451453894376755, 0.0012868058402091265, -0.042653750628232956, -0.009181234054267406, 0.0022574046161025763, 0.054741740226745605, -0.04507189616560936, -0.04829578474164009, 0.020953679457306862, -0.024213628843426704, 0.03927928954362869, 0.006679365411400795, -0.015727609395980835, 0.012828095816075802, 0.05464222654700279, -0.05586550384759903, 0.05909173563122749, 0.0025136538315564394, -0.042732834815979004, -0.02796393260359764, -0.005218395031988621, -0.030157245695590973, -0.016847964376211166, -0.04612521454691887, 0.010811598040163517, 0.059855252504348755, 0.006892528384923935, -0.040531087666749954, 0.012333492748439312, -0.026832709088921547, -0.022782843559980392, -0.008739154785871506, -0.007020080927759409, -0.025635093450546265, 0.008820454590022564, -0.0705648735165596, 0.02297165058553219, 0.015904974192380905, 0.017342692241072655, -0.046378400176763535, 0.03341757878661156, 0.010676102712750435, -0.026282014325261116, -0.03520328179001808, -0.060195647180080414, -0.04018506780266762, 0.051128845661878586, 0.001456266501918435, 0.03046419285237789, 0.0657254084944725, 0.053664203733205795, -0.007809511851519346, -0.008844050578773022, 0.025017134845256805, -0.08764605224132538, -0.1102745309472084, -0.032211411744356155, 0.09724140167236328, -0.015988726168870926, 0.02509596385061741, -0.01805783063173294, 0.028922926634550095, -0.011971882544457912, -0.05148180574178696, -0.10701295733451843, 0.06126411631703377, 0.045518647879362106, 0.015845563262701035, -0.010595332831144333, 0.024372080340981483, -0.06187495216727257, 0.0746084675192833, 0.07971738278865814, -0.0024573360569775105, 0.00898950919508934, -0.04927609860897064, 0.020641831681132317, -0.012498859316110611, -0.058536093682050705, -0.0011213197140023112, -0.015061830170452595, -0.026433590799570084, 0.014831009320914745, -0.09336697310209274, 0.018665647134184837, -0.029465269297361374, -0.005286822561174631, -0.01414266787469387, 0.002421328332275152, -0.018187370151281357, 0.05022888258099556, 0.059828903526067734, -0.0059084100648760796, 0.04919610917568207, -0.025961415842175484], [-0.0675264298915863, 0.03495296835899353, -0.030930781736969948, -0.014092613011598587, 0.013997942209243774, 0.03706035390496254, 0.035335972905159, 0.02653873711824417, 0.001021873322315514, -0.01610586792230606, -0.02927643433213234, 0.014713930897414684, 0.029660562053322792, 0.009889215230941772, 0.027923118323087692, -0.0698719471693039, 0.02745768055319786, 0.0030575350392609835, -0.11927767097949982, 0.036546457558870316, 0.04259480535984039, -0.011604289524257183, -0.009467828087508678, -0.0639779269695282, -0.03628218173980713, 0.020251506939530373, 0.05460827052593231, 0.022202758118510246, 0.01370142586529255, -0.021589798852801323, 0.022251008078455925, 0.10629713535308838, 0.030376318842172623, -0.042732447385787964, -0.008274372667074203, -0.03717458248138428, -0.01256425678730011, -0.0052057188004255295, 0.048437755554914474, -0.05051301047205925, -0.038743194192647934, 0.023813817650079727, -0.031239114701747894, 0.05213940516114235, 0.009313102811574936, -0.002751102903857827, 0.01918652467429638, 0.05882252752780914, -0.0583786740899086, 0.06810934096574783, 0.012583642266690731, -0.03950301185250282, -0.030354557558894157, -0.006789719685912132, -0.030980659648776054, -0.012632817961275578, -0.03566691279411316, 0.006352998316287994, 0.056975107640028, 0.013314157724380493, -0.06354827433824539, 0.014350559562444687, -0.03167014941573143, -0.018571412190794945, 0.008010854944586754, -0.009600391611456871, -0.016698922961950302, 0.0007728228811174631, -0.060776498168706894, 0.039497844874858856, 0.015168871730566025, 0.019687645137310028, -0.04386891424655914, 0.0331425815820694, 0.013180090114474297, -0.022044481709599495, -0.022593416273593903, -0.0571766272187233, -0.05058995634317398, 0.04099157825112343, 0.006717816926538944, 0.025933818891644478, 0.06992121785879135, 0.03791666030883789, 0.007019295822829008, -0.009054869413375854, 0.024656789377331734, -0.09182697534561157, -0.11322931200265884, -0.03840455040335655, 0.0848197340965271, -0.013003193773329258, 0.012797700241208076, -0.010614274069666862, 0.03647063300013542, -0.012720528990030289, -0.06267557293176651, -0.1130601316690445, 0.055783431977033615, 0.04963139072060585, 0.01820162683725357, -0.005135740153491497, 0.02420783042907715, -0.07007893174886703, 0.0659552738070488, 0.08340093493461609, -0.00873480923473835, 0.0011029349407181144, -0.05218726769089699, 0.028344828635454178, -0.005267107859253883, -0.052830636501312256, -0.0012868661433458328, -0.004138442687690258, -0.027390778064727783, 0.009607856161892414, -0.08856646716594696, 0.014773302711546421, -0.03953264281153679, 0.011777067556977272, -0.012297802604734898, 0.004636714700609446, -0.00935113150626421, 0.046519141644239426, 0.059070661664009094, -0.0005707071395590901, 0.04763476178050041, -0.010226924903690815], [-0.07262115180492401, 0.02641778625547886, -0.037233494222164154, -0.015044180676341057, 0.019913572818040848, 0.04513171315193176, 0.04064024239778519, 0.02321176417171955, 0.005179690662771463, -0.001570819178596139, -0.03069181926548481, 0.009808478876948357, 0.01819002255797386, 0.0034496805164963007, 0.031106147915124893, -0.06925054639577866, 0.030371777713298798, -0.006121473386883736, -0.12815529108047485, 0.039702292531728745, 0.03903184458613396, -0.015746554359793663, -0.012991579249501228, -0.05269838869571686, -0.0437283031642437, 0.0029514566995203495, 0.0561254546046257, 0.021259235218167305, 0.006793961860239506, -0.014787561260163784, 0.010191787965595722, 0.0940069854259491, 0.029590677469968796, -0.034750211983919144, -0.008279015310108662, -0.042376045137643814, -0.005578779615461826, 0.003691978519782424, 0.05156036466360092, -0.0440816730260849, -0.04908936843276024, 0.023294713348150253, -0.01997344382107258, 0.04564203694462776, 0.005938525777310133, -0.009490270167589188, 0.021430253982543945, 0.0471002496778965, -0.05032622814178467, 0.0723748654127121, 0.009758902713656425, -0.03708598017692566, -0.026287876069545746, -0.003002309938892722, -0.031297896057367325, -0.007976168766617775, -0.043171923607587814, 0.00767994811758399, 0.06346762180328369, 0.004965413827449083, -0.03978068381547928, 0.009043961763381958, -0.025563059374690056, -0.013151923194527626, 0.003197248326614499, -0.017676284536719322, -0.01720440946519375, 0.002769108861684799, -0.07054702937602997, 0.025839097797870636, 0.016409752890467644, 0.027811354026198387, -0.04578369855880737, 0.03856915608048439, 0.010609149001538754, -0.019022630527615547, -0.03153477981686592, -0.059814319014549255, -0.03934204950928688, 0.05020563304424286, -0.0007390665705315769, 0.029483025893568993, 0.06685050576925278, 0.06219407543540001, 0.004654586315155029, -0.006560273934155703, 0.03133256360888481, -0.08676977455615997, -0.10239183157682419, -0.04082522168755531, 0.09167974442243576, -0.01957445777952671, 0.02088232897222042, -0.019348109140992165, 0.03719932585954666, -0.013116889633238316, -0.04928577318787575, -0.11611621081829071, 0.06949485838413239, 0.04283612221479416, 0.016803687438368797, -0.007598348893225193, 0.025134574621915817, -0.06869832426309586, 0.06657709181308746, 0.07937946915626526, -0.003487298497930169, -0.0018057216657325625, -0.05306039750576019, 0.018617689609527588, -0.012938783504068851, -0.049972690641880035, 0.0002915595250669867, -0.013267476111650467, -0.02502572536468506, 0.01423992682248354, -0.08931529521942139, 0.018528874963521957, -0.041104648262262344, 0.00385862635448575, -0.015391242690384388, -0.0023373691365122795, -0.011209686286747456, 0.040794093161821365, 0.059711258858442307, -0.009945409372448921, 0.038816459476947784, -0.022040631622076035]]\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "

Step 3: Define functions that let you convert into byte objects

\n", + "Bigtable is optimized for key-value pairs and generally will store data as byte objects.\n", + "\n", + "You will need to convert the embeddings that come back from Vertex which are stored as a list of floats in Python, into byte objects that can be inserted into a Bigtable table. The below functions achieve this.\n" + ], + "metadata": { + "id": "0qkRf1nZjMUc" + } + }, + { + "cell_type": "code", + "source": [ + "import struct\n", + "def floats_to_bytes(float_list):\n", + " \"\"\"\n", + " Convert a list of floats to a bytes object, where each float is represented by 4 big-endian bytes.\n", + "\n", + " Parameters:\n", + " float_list (list of float): The list of floats to be converted.\n", + "\n", + " Returns:\n", + " bytes: The resulting bytes object with concatenated 4-byte big-endian representations of the floats.\n", + " \"\"\"\n", + " byte_array = bytearray()\n", + "\n", + " for value in float_list:\n", + " packed_value = struct.pack('>f', value)\n", + " byte_array.extend(packed_value)\n", + "\n", + " # Convert bytearray to bytes\n", + " return bytes(byte_array)" + ], + "metadata": { + "id": "w35KQPr3Kz5c" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "source": [ + "

Step 4: Write the embeddings to Bigtable

" + ], + "metadata": { + "id": "zIkDVD9sjWC5" + } + }, + { + "cell_type": "code", + "source": [ + "from google.cloud.bigtable.data import RowMutationEntry\n", + "from google.cloud.bigtable.data import SetCell\n", + "\n", + "mutations = []\n", + "embeddings = embed_text()\n", + "for i, embedding in enumerate(embeddings):\n", + "\n", + " #convert each embedding into a byte object\n", + " vector = floats_to_bytes(embedding)\n", + "\n", + " #set the row key which will be used to pull the range of documents (ex. doc type or user id)\n", + " row_key = f\"doc_{i}\"\n", + "\n", + " row = table.direct_row(row_key)\n", + "\n", + " #set the column for the embedding based on the byte object format of the embedding\n", + " row.set_cell(\"docs\",\"embedding\",vector)\n", + "\n", + " #store the text associated with vector in the same key\n", + " row.set_cell(\"docs\",\"text\",chunks[i])\n", + "\n", + " mutations.append(row)\n", + "\n", + "#write the rows to Bigtable\n", + "table.mutate_rows(mutations)" + ], + "metadata": { + "id": "NYDXhByAWA9j", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "27e6e457-8577-4b4e-b9f6-3c7af24ad702" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "[, , , , , , , , , ]" + ] + }, + "metadata": {}, + "execution_count": 8 + } + ] + }, + { + "cell_type": "markdown", + "source": [], + "metadata": { + "id": "TDcY7dFVmzYT" + } + }, + { + "cell_type": "markdown", + "source": [ + "

Step 5: Perform a KNN search using Bigtable SQL

\n", + "\n", + "From Python, we can use the GoogleSQL COSINE_DISTANCE function to find the similarity between our text embeddings and search phrases that we give it. Since this computation may take time to process, we will want to use Bigtable’s Asynchronous Data Client to execute the SQL query.
\n", + "\n", + "The response returned should be one of the generated text descriptions that describes Bigtable, an open source database that was originally modeled after Bigtable." + ], + "metadata": { + "id": "Ex6AGVUanzBO" + } + }, + { + "cell_type": "code", + "source": [ + "from google.cloud.bigtable.data import BigtableDataClientAsync\n", + "\n", + "#first embed the search phrase\n", + "search_embedding = embed_text(texts=[\"Apache HBase\"])\n", + "\n", + "query = \"\"\"\n", + " select _key, docs['text'] as description\n", + " FROM knn_intro\n", + " ORDER BY COSINE_DISTANCE(TO_VECTOR32(docs['embedding']), {search_embedding})\n", + " LIMIT 1;\n", + " \"\"\"\n", + "\n", + "async def execute_query():\n", + " async with BigtableDataClientAsync(project=PROJECT_ID) as client:\n", + " local_query = query\n", + " async for row in await client.execute_query(query.format(search_embedding=search_embedding[0]), INSTANCE_ID):\n", + " return(row[\"_key\"],row[\"description\"])\n", + "\n", + "await execute_query()" + ], + "metadata": { + "id": "6s8hBCi9eP-v", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "caaac461-46b1-41af-ef4d-17761d6335ed" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "(b'doc_1', b\"Bigtable is Google's distributed, NoSQL database designed to handle massive datasets with high availability. \\n\")\n" + ] + } + ] + } + ] +} \ No newline at end of file