Add js sdk folder to fish-tank (#75)

* Add js sdk folder to fish-tank * Update formatting * Add placeholder files * Update naming of the file * Add one more example * kate/tf 3913 application collect ecommerce pricing data (#81) * collect pricing data example * update lint * Update javascript-sdk/examples/collect-pricing-data/README.md Co-authored-by: R L Nabors <rachelnabors@users.noreply.github.com> --------- Co-authored-by: R L Nabors <rachelnabors@users.noreply.github.com> * Add js example for list query usage (#83) * Add js example for list query usage * lint * Address security concerns * lint * Add review comments * Lint * Lint * add save / load authenticated session js sdk example. ran script and saw successful reload (#84) * Add close-popup files (#76) * Add close-popup files * Format * Update example * format * Update javascript-sdk/examples/close-popup/README.md * Update --------- Co-authored-by: R L Nabors <rachelnabors@users.noreply.github.com> * Close cookie popup js sdk (#77) * js example cookie * readme * link * . * agentql env key * Update javascript-sdk/examples/close-cookie-dialog/README.md * remove api key setup --------- Co-authored-by: R L Nabors <rachelnabors@users.noreply.github.com> * Add log-into-sites examples (#79) * Add examples * Format * Update javascript-sdk/examples/log-into-sites/README.md * Update --------- Co-authored-by: R L Nabors <rachelnabors@users.noreply.github.com> * Compare Product Prices Example (#78) * Update * Update * Update * Update * Update * Update * Update * Update --------- Co-authored-by: R L Nabors <rachelnabors@users.noreply.github.com> * Add headless browser script (#85) * Add js headless browser script * Add Getting Started YT script (#90) * Add wait for entire page to load example script * Js stealth mode example (#80) * Add example script * Add readme * Fix await * Fix format * Fix lint error * Fix nits * Add xpath application example (#92) * Add xpath application example * Add extra comment * linting * `python` -> `Javascript` * Added javascript sentiment analysis script (#94) * Add wait for entire page to load example script (#95) * Add wait for entire page to load example script * Add `await` * `.py` -> `.js` * submit form js example (#89) * submit form js example * changes * ran lint * changes * Add get_by_prompt example * Fix formatting * add interact with existing browser example * Update interact with existing browser example * address comments & fix examples --------- Co-authored-by: Kate Zhang <54541538+KateZhang98@users.noreply.github.com> Co-authored-by: R L Nabors <rachelnabors@users.noreply.github.com> Co-authored-by: Urvish Thakker <141964033+thakkerurvish@users.noreply.github.com> Co-authored-by: Mark Peng <mark@tinyfish.io> Co-authored-by: Suveen Ellawela <suveen@tinyfish.io> Co-authored-by: Aaron Tan <157438789+aarontantf@users.noreply.github.com> Co-authored-by: Jin Yang <157438770+jinyangTF@users.noreply.github.com> Co-authored-by: Zachary Lyon <zachary@lyonfamily.org> Co-authored-by: Sergey Ryabov <944361+colriot@users.noreply.github.com> Co-authored-by: Julius Gambe <gambe@tinyfish.io> Co-authored-by: Josh Kung <josh@tinyfish.io>
tinyfish-io · Nov 6, 2024 · 4dcbde9 · 4dcbde9
1 parent 13b9029
commit 4dcbde9
Show file tree

Hide file tree

Showing 88 changed files with 7,458 additions and 17 deletions.
diff --git a/.github/workflows/precommit.yml b/.github/workflows/precommit.yml
@@ -5,7 +5,7 @@ on:
     types: [opened, synchronize, reopened]
 
 jobs:
-  pre-commit:
+  python-pre-commit:
     name: Pre-commit checks
     runs-on: ubuntu-latest
     steps:
@@ -49,3 +49,27 @@ jobs:
         continue-on-error: true
         with:
           pylance-version: latest-release
+
+  js-pre-commit:
+    name: JS Pre-commit checks
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout Repository
+        uses: actions/checkout@v4
+
+      - name: Setup Node.js
+        uses: actions/setup-node@v3
+        with:
+          node-version: "20"
+
+      - name: Install dependencies
+        working-directory: ./javascript-sdk
+        run: npm install
+
+      - name: Run ESLint
+        working-directory: ./javascript-sdk
+        run: npm run lint
+
+      - name: Run Prettier
+        working-directory: ./javascript-sdk
+        run: npx prettier --check .
diff --git a/.gitignore b/.gitignore
@@ -36,3 +36,5 @@ logs/
 tmp/
 
 *env.local
+
+javascript-sdk/node_modules
diff --git a/.vscode/extensions.json b/.vscode/extensions.json
@@ -4,6 +4,8 @@
     "GitHub.copilot",
     "esbenp.prettier-vscode",
     "ms-python.isort",
-    "ms-python.pylint"
+    "ms-python.pylint",
+    "dbaeumer.vscode-eslint",
+    "yoavbls.pretty-ts-errors",
   ]
-}
+}
diff --git a/.vscode/settings.json b/.vscode/settings.json
@@ -8,6 +8,18 @@
   "[json]": {
     "editor.defaultFormatter": "vscode.json-language-features"
   },
+  "[javascriptreact]": {
+    "editor.defaultFormatter": "esbenp.prettier-vscode"
+  },
+  "[typescript]": {
+    "editor.defaultFormatter": "esbenp.prettier-vscode"
+  },
+  "eslint.workingDirectories": [
+    {
+      "directory": "./javascript-sdk",
+      "changeProcessCWD": true
+    }
+  ],
   "editor.formatOnSave": true,
   "files.insertFinalNewline": true,
   "black-formatter.args": [

diff --git a/JS/.eslintrc.js b/JS/.eslintrc.js
@@ -0,0 +1,14 @@
+/* eslint-env node */
+module.exports = {
+  root: true,
+  env: {
+    browser: true,
+    es6: true,
+    node: true,
+  },
+  extends: ['eslint:recommended', 'prettier'],
+  parserOptions: {
+    ecmaVersion: 12,
+    sourceType: 'module',
+  },
+};
diff --git a/JS/.prettierrc.js b/JS/.prettierrc.js
@@ -0,0 +1,9 @@
+module.exports = {
+  printWidth: 100,
+  trailingComma: 'all',
+  singleQuote: true,
+  importOrder: ['<THIRD_PARTY_MODULES>', '^@/types/(.*)$', '^@/lib/(.*)$', '^[./]'],
+  importOrderSeparation: true,
+  importOrderSortSpecifiers: true,
+  importOrderCaseInsensitive: false,
+};
diff --git a/JS/application-examples/collect-youtube-comments/README.md b/JS/application-examples/collect-youtube-comments/README.md
@@ -0,0 +1,17 @@
+# Example script: interacting with YouTube website and extracting video information using AgentQL
+
+This is an example of interacting with YouTube website and extracting video information using AgentQL.
+
+## Run the script
+
+- [Install AgentQL SDK](https://docs.agentql.com/javascript-sdk/installation)
+- Save this JavaScript file locally as **main.js**
+- Run the following command from the project's folder:
+
+```bash
+node main.js
+```
+
+## Play with the query
+
+Install the [AgentQL Debugger Chrome extension](https://docs.agentql.com/installation/chrome-extension-installation) to play with the AgentQL query. [Learn more about the AgentQL query language](https://docs.agentql.com/agentql-query/query-intro)
diff --git a/JS/application-examples/collect-youtube-comments/main.js b/JS/application-examples/collect-youtube-comments/main.js
@@ -0,0 +1,96 @@
+const { chromium } = require('playwright');
+const { configure, wrap } = require('agentql');
+
+const URL = 'https://www.youtube.com/';
+
+async function main() {
+  // Configure the AgentQL API key
+  configure({
+    apiKey: process.env.AGENTQL_API_KEY, // This is the default and can be omitted.
+  });
+
+  const browser = await chromium.launch({ headless: false });
+  const page = await wrap(await browser.newPage());
+  await page.goto(URL);
+
+  const SEARCH_QUERY = `
+  {
+    search_input
+    search_btn
+  }
+  `;
+
+  const VIDEO_QUERY = `
+  {
+    videos[] {
+      video_link
+      video_title
+      channel_name
+    }
+  }
+  `;
+
+  const VIDEO_CONTROL_QUERY = `
+  {
+    play_or_pause_btn
+    expand_description_btn
+  }
+  `;
+
+  const DESCRIPTION_QUERY = `
+  {
+    description_text
+  }
+  `;
+
+  const COMMENT_QUERY = `
+  {
+    comments[] {
+      channel_name
+      comment_text
+    }
+  }
+  `;
+
+  try {
+    // search query
+    const searchResponse = await page.queryElements(SEARCH_QUERY);
+    await searchResponse.search_input.type('machine learning', { delay: 75 });
+    await searchResponse.search_btn.click();
+
+    // video query
+    const videoResponse = await page.queryElements(VIDEO_QUERY);
+    console.log(
+      `Clicking Youtube Video: ${await videoResponse.videos[0].video_title.textContent()}`,
+    );
+    await videoResponse.videos[0].video_link.click(); // click the first youtube video
+
+    // video control query
+    const controlResponse = await page.queryElements(VIDEO_CONTROL_QUERY);
+    await controlResponse.expand_description_btn.click();
+
+    // description query
+    const descriptionData = await page.queryData(DESCRIPTION_QUERY);
+    console.log(`Captured the following description:\n${descriptionData.description_text}`);
+
+    // Scroll down the page to load more comments
+    for (let i = 0; i < 3; i++) {
+      await page.keyboard.press('PageDown');
+      await page.waitForLoadState();
+    }
+
+    // comment query
+    const commentResponse = await page.queryData(COMMENT_QUERY);
+    console.log(`Captured ${commentResponse.comments?.length || 0} comments!`);
+  } catch (error) {
+    console.error(`Found Error: ${error}`);
+    throw error;
+  }
+
+  // Used only for demo purposes. It allows you to see the effect of the script.
+  await page.waitForTimeout(10000);
+
+  await browser.close();
+}
+
+main();
diff --git a/JS/application-examples/perform-sentiment-analysis/README.md b/JS/application-examples/perform-sentiment-analysis/README.md
@@ -0,0 +1,29 @@
+# Example script: perform sentiment analysis with AgentQL
+
+This example demonstrates how to perform sentiment analysis on YouTube comments with AgentQL and OpenAI's GPT-3.5 model.
+
+## Run the script
+
+- [Install AgentQL SDK](https://docs.agentql.com/javascript-sdk/installation)
+- [Install OpenAI SDK](https://www.npmjs.com/package/openai) with the following command:
+
+```bash
+npm install openai
+```
+
+- Save this JavaScript file locally as **main.js**
+- Set your OpenAI API key as an environment variable with the following command:
+
+```bash
+export OPENAI_API_KEY="My API Key"
+```
+
+- Run the following command from the project's folder:
+
+```bash
+node main.js
+```
+
+## Play with the query
+
+Install the [AgentQL Debugger Chrome extension](https://docs.agentql.com/installation/chrome-extension-installation) to play with the AgentQL query. [Learn more about the AgentQL query language](https://docs.agentql.com/agentql-query/query-intro)
diff --git a/JS/application-examples/perform-sentiment-analysis/main.js b/JS/application-examples/perform-sentiment-analysis/main.js
@@ -0,0 +1,99 @@
+/* This example demonstrates how to perform sentiment analysis on YouTube comments with AgentQL and OpenAI's GPT-3.5 model. */
+
+const { wrap, configure } = require('agentql');
+const { chromium } = require('playwright');
+
+// Import the OpenAI API client.
+const { OpenAI } = require('openai/index.mjs');
+
+// Define the URL of the page to scrape.
+const URL = 'https://www.youtube.com/watch?v=JfM1mr2bCuk';
+
+// Define a query to interact with the page.
+const QUERY = `
+{
+  video_title
+  video_channel
+  comments[] {
+    comment_text
+    author
+  }
+}
+`;
+
+async function getComments() {
+  // Configure the AgentQL API key
+  configure({
+    apiKey: process.env.AGENTQL_API_KEY, // This is the default and can be omitted.
+  });
+
+  // Launch a headless browser using Playwright.
+  const browser = await chromium.launch({ headless: false });
+
+  // Create a new page in the browser and wrap it to get access to the AgentQL's querying API
+  const page = await wrap(await browser.newPage());
+  await page.goto(URL);
+
+  for (let i = 0; i < 5; i++) {
+    // Scroll down the page to load more comments.
+    await page.waitForPageReadyState();
+
+    // Scroll down the page to load more comments
+    await page.keyboard.press('PageDown');
+  }
+
+  // Use queryData() method to fetch the video information from the page.
+  const response = await page.queryData(QUERY);
+
+  // Close the browser
+  await browser.close();
+
+  return response;
+}
+
+async function performSentimentAnalysis(comments) {
+  // User message construction
+  let USER_MESSAGE =
+    'These are the comments on the video. I am trying to understand the sentiment of the comments.';
+
+  // Append each comment's text to USER_MESSAGE
+  comments.comments.forEach((comment) => {
+    USER_MESSAGE += comment.comment_text;
+  });
+
+  // Define the system message
+  const SYSTEM_MESSAGE = `You are an expert in understanding social media analytics and specialize in analyzing the sentiment of comments.
+  Please find the comments on the video as follows:
+  `;
+
+  // Append request for a summary and takeaways
+  USER_MESSAGE +=
+    ' Could you please provide a summary of the comments on the video. Additionally, just give only 3 takeaways which would be important for me as the creator of the video.';
+
+  // Initialize OpenAI client
+  const client = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });
+
+  try {
+    const response = await client.chat.completions.create({
+      model: 'gpt-3.5-turbo',
+      messages: [
+        { role: 'system', content: SYSTEM_MESSAGE },
+        { role: 'user', content: USER_MESSAGE },
+      ],
+    });
+
+    // Return the content of the first completion choice
+    return response.choices[0].message.content;
+  } catch (error) {
+    console.error('Error during API call:', error);
+    throw error;
+  }
+}
+
+async function main() {
+  const comments = await getComments();
+  const summary = await performSentimentAnalysis(comments);
+  console.log(summary);
+}
+
+main();
diff --git a/JS/application-examples/submit-form/README.md b/JS/application-examples/submit-form/README.md
@@ -0,0 +1,26 @@
+# Example script: Automating form submission using AgentQL
+
+This is an example of automating form submission on a website using AgentQL.
+
+## Run the script
+
+- [Install AgentQL SDK](https://docs.agentql.com/javascript-sdk/installation)
+- Save this JavaScript file locally as **main.js**
+- Run the following command from the project's folder:
+
+```bash
+node main.js
+```
+
+## Code explanation
+
+This script demonstrates how to:
+
+1. Navigate to a form page
+2. Fill out form fields using AgentQL queries
+3. Submit the form and handle the confirmation
+4. Wait for the submission to complete
+
+## Play with the query
+
+Install the [AgentQL Debugger Chrome extension](https://docs.agentql.com/installation/chrome-extension-installation) to play with the AgentQL query. [Learn more about the AgentQL query language](https://docs.agentql.com/agentql-query/query-intro)