add example for text search

Author: fogfish

text-search/README.md

@@ -0,0 +1,120 @@
+# Tutorial: "Text Search"
+
+This tutorial guides you through the process of creating instances of an approximation nearest neighbor search of natural language content.
+
+The tutorial is based on [Crime and Punishment](https://en.wikipedia.org/wiki/Crime_and_Punishment). It is a novel by the Russian author Fyodor Dostoevsky that is a psychological exploration of guilt, morality, and redemption.
+
+
+## Requirements
+
+1. Install the command-line utility from source code. It requires [Golang](https://go.dev) development environment:
+
+```bash
+go install github.com/kshard/optimum/cmd/optimum@latest
+```
+
+2. Define environment variable with values obtained from your provider. 
+
+```bash
+export AWS_REGION=us-east-1
+export AWS_ACCESS_KEY_ID=XXXXXXXXXX
+export AWS_SECRET_ACCESS_KEY=XXXXXXXXXXXXXXXXXX
+export HOST=https://example.com
+export ROLE=arn:aws:iam::000000000000:role/example-access-role
+```
+
+3. Clone this repository, examples and tutorial section.
+
+```bash
+git clone github.com/kshard/optimum-tutorials
+cd text-search
+```
+
+## Text retrieval
+
+1. Let's create new instance of `text` data structure. We call it `scap` - Sentences of Crime and Punishment. The command displays the progress of the data structure creation. Please wait until the instance is successfully provisioned.
+  
+```bash
+optimum text create -u $HOST -r $ROLE \
+  -n scap \
+  -j config.json
+
+# Output:
+# 
+# scap (vsn Njqa2b7o86O72V.0) | CREATING  ...
+# scap (vsn Njqa2b7o86O72V.0) | STARTING  ...
+# scap (vsn Njqa2b7o86O72V.0) | RUNNING   ... 
+# scap (vsn Njqa2b7o86O72V.0) | SUCCEEDED ... 
+```
+
+2. `sentences/crime-and-punishment.txt` dataset is a sequence of natural language sentences from the book. Upload this dataset.
+
+```bash
+optimum text upload -u $HOST -r $ROLE \
+  -n scap \
+  sentences/crime-and-punishment.txt
+```
+
+3. Commit uploaded dataset, making it available on the read path. The command displays the progress of the commit operation. Please wait until the dataset is successfully committed. Note: the commit takes about 10 min due to intentional rate limiting for text analysis.
+
+```bash
+optimum text commit -u $HOST -r $ROLE \
+  -n scap
+
+# Output
+# 
+# scap (vsn Njqa6b7o866fsZ.L) | COMMITTING ...
+# scap (vsn Njqa6b7o866fsZ.L) | STARTING  ...
+# scap (vsn Njqa6b7o866fsZ.L) | RUNNING   ... 
+# scap (vsn Njqa6b7o866fsZ.L) | SUCCEEDED ... 
+```
+
+4. Check that `scap` instance is active and the update of latest version has been successful.
+
+```bash
+optimum text list -u $HOST -r $ROLE
+
+# Output
+#
+# NAME  VERSION          UPDATED AT          | STATUS  PENDING |
+# scap  Njqa6b7o866fsZ.L 2024-08-19 06:52:38 | ACTIVE          |
+```
+
+5. `queries.txt` dataset consists of 100 questions about "Crime and Punishment":
+- Lines 01-15: Plot-Related Questions
+- Lines 16-30: Character-Focused Questions
+- Lines 31-50: Themes
+- Lines 51-70: Philosophical Questions
+- Lines 71-90: Symbolism and Imagery
+- Lines 91-99. Historical and Social Context
+ 
+```bash
+optimum text query -u $HOST -r $ROLE \
+  -n scap \
+  queries.txt
+```
+
+The output for each query is 10 most sentences (sha1 hashes), each is ranked with the cosine distance. You can interpret is as 
+* `[0.0, 0.2]` Highly similar items. Use this range when you need very close matches (e.g., finding duplicate documents).
+* `(0.2, 0.5]` Moderately similar items. Useful when you want to find items that are related but not identical.
+* `(0.5, 0.8]` Weakly similar items. This range could be used for exploratory results where you want to include some diversity.
+* `(0.8, 1.0]` Dissimilar items. Typically, these items are unrelated, and you might filter them out unless dissimilarity is desirable (e.g., in anomaly detection).
+
+A few examples below, shows typical output:
+
+```
+Query 1 (took 67.288517ms) | cask:vYntsiSv3o4Rga6zg7wbue4ZXR/text/scap (vsn Njzn.QG1iUgv.R.2, size 12391)
+  > How does Raskolnikov’s initial crime drive the plot of the novel?
+  0.103614 : the novel centers around the tormented mind of rodion raskolnikov, a man driven by desperation and ideology. the weight of raskolnikov's crime becomes an unbearable burden that consumes his soul. guilt, however, proves to be a relentless force that erodes raskolnikov’s belief in his own superiority. raskolnikov’s inner conflict represents the struggle between the desire for power and the need for human connection. raskolnikov’s interactions with others are marked by a sense of disconnection and misunderstanding. raskolnikov’s oscillation between confession and concealment reflects his deep internal conflict. raskolnikov’s crime is motivated by a mixture of desperation, pride, and a misguided sense of justice. raskolnikov’s crime is both a physical act of violence and a symbolic rejection of moral law. raskolnikov’s journey from crime to punishment is a reflection of the universal human experience of sin and redemption. raskolnikov’s eventual acceptance of his guilt is a painful but necessary step towards his redemption. 
+  0.151891 : raskolnikov’s murder of the pawnbroker is both a calculated act and a fevered impulse. the novel’s pacing mirrors raskolnikov’s descent into madness, with moments of intense introspection and sudden bursts of action. 
+  ...
+```
+
+```
+Query 97 (took 79.000902ms) | cask:vYntsiSv3o4Rga6zg7wbue4ZXR/text/scap (vsn Njzn.QG1iUgv.R.2, size 12391)
+  > In what ways does Dostoevsky portray the urban environment as contributing to moral decay?
+  0.216930 : the city of st. petersburg mirrors raskolnikov’s mental state, with its oppressive atmosphere and squalor. the novel’s portrayal of st. petersburg as a city of decay and corruption reflects the moral decay of its inhabitants. 
+  0.269576 : the novel explores the consequences of living according to abstract ideals without regard for human emotions and relationships. 
+  0.292030 : the novel critiques the dehumanizing effects of poverty and capitalism. the novel’s depiction of urban life is bleak and oppressive, reflecting the dehumanizing effects of poverty and isolation. 
+  ...
+```

text-search/config.json

@@ -0,0 +1,12 @@
+{
+  "embeddings": {
+    "model": "amazon.titan-embed-text-v2:0",
+    "dimension": 256
+  },
+  "hnsw": {
+    "m": 8,
+    "m0": 64,
+    "efConstruction": 400,
+    "surface": "cosine"
+  }
+}

text-search/queries.txt

@@ -0,0 +1,99 @@
+How does Raskolnikov’s initial crime drive the plot of the novel?
+Why does Raskolnikov choose to murder Alyona Ivanovna, the pawnbroker?
+How does the murder affect Raskolnikov psychologically throughout the story?
+What role does the subplot involving the Marmeladov family play in the main story?
+How does Raskolnikov’s relationship with his sister, Dunya, develop throughout the novel?
+Why does Raskolnikov confess to the murder at the end of the novel?
+How does Raskolnikov’s internal struggle between guilt and justification evolve over the course of the novel?
+What is the significance of Raskolnikov’s decision to kill Lizaveta as well as the pawnbroker?
+How does Porfiry Petrovich manipulate Raskolnikov psychologically in their encounters?
+How does the setting of St. Petersburg contribute to the atmosphere of the novel?
+How does Raskolnikov's interaction with the police develop his character arc?
+In what ways do dreams play a role in shaping Raskolnikov's choices throughout the novel?
+How do Raskolnikov's changing plans after the murder reveal his inner conflict?
+What role does Razumikhin play in balancing the dark psychological elements of the plot?
+How does Dostoevsky use suspense and pacing to intensify the psychological tension in the story?
+What motivates Raskolnikov to commit the murder beyond financial need?
+How does Sonya’s character represent a moral compass in contrast to Raskolnikov?
+What role does Dunya play in Raskolnikov’s sense of family and duty?
+In what ways is Razumikhin a foil to Raskolnikov’s character?
+What does Raskolnikov's relationship with his mother, Pulcheria Alexandrovna, reveal about his personality?
+How does the character of Svidrigailov contribute to the theme of moral ambiguity?
+In what ways is Svidrigailov a double or mirror to Raskolnikov?
+How does Porfiry’s cat-and-mouse game with Raskolnikov drive the tension in the novel?
+How does Raskolnikov’s worldview clash with that of Sonya, and how does this shape his character development?
+How does Luzhin represent the corrupting influence of wealth and power?
+Why does Sonya remain loyal to Raskolnikov, even after learning of his crime?
+What does the relationship between Raskolnikov and Sonya suggest about redemption?
+How does Dunya's engagement to Luzhin illustrate her character's values and strength?
+What drives Svidrigailov’s obsession with Dunya?
+How does Raskolnikov’s alienation from society reflect his personality and inner turmoil?
+How does the theme of guilt and conscience permeate the novel?
+In what ways does Dostoevsky explore the theme of justice in the novel?
+How does *Crime and Punishment* examine the idea of "superman theory" through Raskolnikov’s beliefs?
+What role does poverty play in shaping the characters' lives and decisions?
+How does the novel contrast rational thought with emotion and intuition?
+How does Dostoevsky critique utilitarianism through Raskolnikov’s justification of his crime?
+In what ways does *Crime and Punishment* address the theme of alienation?
+How does Dostoevsky use the motif of suffering to develop the novel’s moral and philosophical themes?
+What does the novel suggest about the possibility of redemption and atonement?
+How does *Crime and Punishment* depict the struggle between good and evil within an individual?
+How does Dostoevsky portray the legal and criminal justice system in the novel?
+How does the novel explore the theme of free will versus determinism?
+What role does fate play in shaping the events of the novel?
+How does the novel depict the consequences of pride and hubris?
+How does Dostoevsky use Raskolnikov's crime to explore moral relativism?
+How does Dostoevsky portray mental illness and psychological breakdown in the novel?
+In what ways does the novel explore the theme of nihilism and its consequences?
+How does *Crime and Punishment* reflect on the role of suffering in achieving self-awareness or salvation?
+What does the novel suggest about the nature of evil and its roots in human nature?
+How does the novel explore the tension between intellectualism and morality?
+In what ways does Raskolnikov’s crime reflect Nietzschean philosophy or the idea of a “superman”?
+How does Dostoevsky challenge utilitarianism with Raskolnikov's justifications for murder?
+How does Raskolnikov’s theory of extraordinary men lead to his downfall?
+How does the novel explore the relationship between law and morality?
+How does Dostoevsky's portrayal of suffering align with Christian existentialism?
+What role does religious symbolism play in Raskolnikov’s journey toward redemption?
+How does Sonya’s faith challenge Raskolnikov’s existential despair?
+What does the novel suggest about the nature of free will and moral responsibility?
+How does the novel engage with the philosophical ideas of nihilism?
+How does the novel critique the Enlightenment emphasis on reason and individualism?
+How does Dostoevsky explore the theme of God and forgiveness in *Crime and Punishment*?
+How does Raskolnikov’s inner dialogue reflect the philosophical conflict between rational egoism and altruism?
+What does Dostoevsky suggest about the relationship between morality and religion in the novel?
+How does the novel explore the psychological and moral implications of crime beyond legal punishment?
+In what ways does the novel question the possibility of true justice in human society?
+What does Dostoevsky suggest about the existence of absolute moral truths versus relative ones?
+How does the novel engage with the existential idea that life is inherently meaningless?
+How does Raskolnikov’s attempt to play God contribute to his psychological decline?
+How does the novel challenge the reader's understanding of morality and justice?
+What is the significance of Raskolnikov’s final epiphany in relation to philosophical themes of salvation?
+What is the symbolic meaning of the recurring image of blood in the novel?
+How does the setting of St. Petersburg function symbolically in the novel?
+What is the significance of dreams in *Crime and Punishment*, particularly Raskolnikov’s dream of the horse?
+How does the symbol of the axe represent Raskolnikov’s inner conflict?
+What is the symbolic importance of the cross that Sonya gives to Raskolnikov?
+How does Dostoevsky use light and darkness to reflect the characters’ inner states?
+How does the weather and physical environment in the novel mirror Raskolnikov’s mental and emotional condition?
+What does the pawnshop symbolize in the context of the novel’s themes of poverty and desperation?
+How does the use of colors like yellow (Alyona’s apartment, Sonya’s hair) contribute to the novel’s atmosphere?
+How does the image of the bridge serve as a metaphor for transition or liminality in the novel?
+How does the portrayal of alcohol in the novel serve as a symbol of escape and despair?
+What does Raskolnikov’s physical illness after the murder symbolize?
+How does the image of the horse in Raskolnikov’s dream reflect his internal struggle with guilt and morality?
+What is the significance of the Lazarus story, which Sonya reads to Raskolnikov?
+How does the image of Raskolnikov’s room reflect his isolation and psychological turmoil?
+How does the theme of doubling (Raskolnikov/Svidrigailov) function symbolically in the novel?
+How do the streets of St. Petersburg act as a symbolic reflection of Raskolnikov's mental state?
+How is money portrayed as a symbol of corruption and moral decay in the novel?
+What role does water symbolism (rain, rivers) play in shaping the novel’s mood and themes?
+How does Dostoevsky use the image of imprisonment to explore the theme of internal punishment?
+How does the novel reflect the socio-economic conditions of 19th-century Russia?
+In what ways does *Crime and Punishment* critique the social inequality and poverty of the time?
+How do the characters’ struggles reflect broader societal issues in pre-revolutionary Russia?
+What role does class play in shaping the moral decisions of the characters?
+How does Dostoevsky use the legal system in the novel to comment on Russian society?
+How does the novel explore the tension between Westernization and traditional Russian values?
+In what ways does Dostoevsky portray the urban environment as contributing to moral decay?
+How does the novel address the conflict between modernity and traditionalism in Russia?
+What is the significance of Dostoevsky’s personal experiences with crime and punishment in shaping the