-
Notifications
You must be signed in to change notification settings - Fork 28
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add Getting Started YT script #90
Merged
Merged
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,17 @@ | ||
# Example script: interacting with YouTube website and extracting video information using AgentQL | ||
|
||
This is an example of interacting with YouTube website and extracting video information using AgentQL. | ||
|
||
## Run the script | ||
|
||
- [Install AgentQL SDK](https://docs.agentql.com/javascript-sdk/installation) | ||
- Save this JavaScript file locally as **collect_youtube_comments.js** | ||
- Run the following command from the project's folder: | ||
|
||
```bash | ||
node collect_youtube_comments.js | ||
``` | ||
|
||
## Play with the query | ||
|
||
Install the [AgentQL Debugger Chrome extension](https://docs.agentql.com/installation/chrome-extension-installation) to play with the AgentQL query. [Learn more about the AgentQL query language](https://docs.agentql.com/agentql-query/query-intro) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,94 @@ | ||
const { chromium } = require('playwright'); | ||
const { configure, wrap } = require('agentql'); | ||
|
||
const URL = 'https://www.youtube.com/'; | ||
|
||
async function main() { | ||
// Configure the AgentQL API key | ||
configure({ | ||
apiKey: process.env.AGENTQL_API_KEY, // This is the default and can be omitted. | ||
}); | ||
|
||
const browser = await chromium.launch({ headless: false }); | ||
const page = wrap(await browser.newPage()); | ||
await page.goto(URL); | ||
|
||
const SEARCH_QUERY = ` | ||
{ | ||
search_input | ||
search_btn | ||
} | ||
`; | ||
|
||
const VIDEO_QUERY = ` | ||
{ | ||
videos[] { | ||
video_link | ||
video_title | ||
channel_name | ||
} | ||
} | ||
`; | ||
|
||
const VIDEO_CONTROL_QUERY = ` | ||
{ | ||
play_or_pause_btn | ||
expand_description_btn | ||
} | ||
`; | ||
|
||
const DESCRIPTION_QUERY = ` | ||
{ | ||
description_text | ||
} | ||
`; | ||
|
||
const COMMENT_QUERY = ` | ||
{ | ||
comments[] { | ||
channel_name | ||
comment_text | ||
} | ||
} | ||
`; | ||
|
||
try { | ||
// search query | ||
const searchResponse = await page.queryElements(SEARCH_QUERY); | ||
await searchResponse.search_input.type('machine learning', { delay: 75 }); | ||
await searchResponse.search_btn.click(); | ||
|
||
// video query | ||
const videoResponse = await page.queryElements(VIDEO_QUERY); | ||
console.log( | ||
`Clicking Youtube Video: ${await videoResponse.videos[0].video_title.textContent()}`, | ||
); | ||
await videoResponse.videos[0].video_link.click(); // click the first youtube video | ||
|
||
// video control query | ||
const controlResponse = await page.queryElements(VIDEO_CONTROL_QUERY); | ||
await controlResponse.expand_description_btn.click(); | ||
|
||
// description query | ||
const descriptionData = await page.queryData(DESCRIPTION_QUERY); | ||
console.log(`Captured the following description:\n${descriptionData.description_text}`); | ||
|
||
// Scroll down the page to load more comments | ||
for (let i = 0; i < 3; i++) { | ||
await page.keyboard.press('PageDown'); | ||
await page.waitForLoadState(); | ||
} | ||
|
||
// comment query | ||
const commentResponse = await page.queryData(COMMENT_QUERY); | ||
console.log(`Captured ${commentResponse.comments?.length || 0} comments!`); | ||
} catch (error) { | ||
console.error(`Found Error: ${error}`); | ||
throw error; | ||
} | ||
|
||
// Used only for demo purposes. It allows you to see the effect of the script. | ||
await page.waitForTimeout(10000); | ||
} | ||
|
||
main(); |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@rachelnabors and I have decided to remove them from the example. We will include how to set up API key in installation guide.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This might be no so good, because some people don't read the installation guide (except the part with
npm install
), and go straight to the code examples to play with.What kind of content do you plan in installation guide? To set OS global env var or smth else?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The installation guide currently contains guide on set session env var via
export
, but this is a good point. We should instruct users to set OS global env var