Parse tokenize #1497
Merged
Parse tokenize #1497
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Adds a new tokenize command to parse text into hex tokens with options.
Contributor
|
LGTM 🚀
|
Contributor
AnnotationsLinter: no-fix-mesCaution Found a file modification, but there are no TODOs to encourage further improvements. Avoid incomplete code without tracking improvements. (packages/cli/src/cli.ts#L1 no_fix_mes) Linter: grumpy-devWarning Oh great, another CLI rewrite. The descriptions are verbose and yet somehow say nothing new. Try to be concise but meaningful. No one cares about walls of text. (packages/cli/src/cli.ts#L1 grumpy_dev_cli_file) Warning Looks like imports were deleted without regard for dependency assessments. Are you sure these constants weren't being used elsewhere? Always validate their absence before deletion. (packages/cli/src/parse.ts#L1 grumpy_dev_parse_file) Linter: variable-namesNote Since new parsing logic is tampered with in Linter: commentsWarning CLI functions lack updated comments or docstrings to explain any subtle changes or the need for this rewrite. Always keep documentation up-to-date. (packages/cli/src/cli.ts#L1 comments_missing) Linter: stats
If code deletions or additions impact critical functionality, ensure they've been tested. Provide test coverage for new code logic. Add clear comments to avoid confusion when updating CLI tools or parser constants.
|
pelikhan
added a commit
that referenced
this pull request
Apr 25, 2025
* 🐛: Handle unknown reference types in state logging Added logging for unknown reference types in activateChatParticipant. * 3rd party license update * fix #1462 * ✨ feat: enhance Azure API usage and version configuration - Added support for AZURE_AI_INFERENCE_VERSION constant. - Expanded documentation with API version and configuration examples. * ✨ Enhance API version parsing logic Refined Azure API version fallback and URL parameter parsing. * test with gpt-4.1 * 🔧 refactor: improve astgrep debug and doc formatting Adjusted debug imports and corrected doc indentation and spacing. * ✨ Improve terminal message rendering logic Modified slice behavior for balanced content display in limited rows. * using project watcher to avoid rebuild scripts on every invocation * cache workspace parsing * Release 1.129.5 * [skip ci] updated version numbers * ✨ Improve debug logging and update GCC script Enhanced debug logging across services and refined GCC script messaging. * ♻️ Enhance debugging and logging in container handling Added detailed container operations logging and improved dbg usage. * ✨ feat: improve Azure API version parsing in env.ts Enhanced parsing to ignore versions with curly braces in URLs. * Release 1.129.6 * [skip ci] updated version numbers * 🎨: Improve trimmed message rendering logic - Adjusted logic to add ellipsis only when necessary. * ✨: Improve message trimming with hidden lines indicator Added indicator for hidden lines count in trimmed message content. * ✨: Improve hidden lines rendering logic Adjusted handling to display one hidden line or summarize excess. * ✨ refactor terminal rendering for truncation logic Refined line truncation logic for better clarity and maintainability. * Release 1.129.7 * [skip ci] updated version numbers * support running .gitignored files directly * Release 1.129.8 * [skip ci] updated version numbers * toy blog narration script * test narration * add blog audio descriptions * ✨: Enhance blog narration with diverse AI voice options Add detailed voice personas, force logic, and AI disclaimers. * generating videos * ✨ Enhance Azure AI and OpenAI LLM capabilities Updated Azure AI models, pricing, and regex for expanded support. * render schema in chat preview * ✨ feat: enable blog narration and improve schema handling Added a draft blog post for narration summaries with genai and refined JSON schema handling in code, enhancing readability and functionality. * narration blog * don't send defaults in response_schema * ✨: Add options to disable trace and output generation Introduce "--no-run-trace" and "--no-output-trace" CLI options. * ✨ Update trace and output file handling conventions Updated constants and related code references for clearer filenames. * Release 1.129.9 * [skip ci] updated version numbers * ✨: Add JSONSchemaToFunctionParameters utility refinement Introduce a robust utility to convert JSON Schemas into parameters. * ✨ feat: add options parameter to copyTo method - Enhanced copyTo to support additional options for file handling. * copyTo option as optional * removing .mp4s * typos (#1465) * updating github actions docs * ✨ Add blog post on GitHub Models in Actions New blog post explains using GITHUB_TOKEN with GitHub Models. * typo * narration * fix various typechecking issues in genaiscript.d.ts * ✨ refactor code and enhance interfaces Updated interfaces, improved typings, and cleaned unused elements. * Release 1.130.0 * [skip ci] updated version numbers * ✨ improve error handling in file operations * ✨ Add workspace and haiku prompt, enhance fs function Added a VS Code workspace file and a haiku prompt. Improved `tryStat` logic. * genai: generated docs * Release 1.130.1 * [skip ci] updated version numbers * message about waiting to start * enable permissions: models: read in github actions (#1467) * enable models * save transcript as well * ✨ Optimize front matter generation and update descriptions Refactored front matter scripts and enhanced SEO metadata generation. * Docs image generation (#1468) * add images for docs * optimizing * more opti * Update genaiscript.prompt.md (#1469) The reference to the GenAIScript docs is broken. * support using selection (#1471) * support using selection * ✨ feat: add string variable initialization in switch case - Introduced a new string variable initialization in "string" case handling. * linking to neoview runner * add og:image * one more image * Release 1.130.2 * [skip ci] updated version numbers * Update file inclusion for genaisrc subfolders (#1473) * fix for #1360 * ✨ feat: update GenAI docs for TypeScript support - Enhanced TypeScript integration with explanations and examples * fixing typos * adding fetch tool (#1474) * adding fetch tool * basic fetch working * slide on ast * File-as-resources (#1478) * support uris in file descriptions * missing await * ✨ Improve git and resource handling for better integration Enhanced debugging, refactored functions, and streamlined cloning logic. * ✨ Add cancellation checks and enhance resource handling Introduced cancellation checks in critical methods and improved URI handling. * ✨ Add support for filtering by file extensions Introduced an --accept option for specifying accepted extensions. * 🐛 Fix git URL pathname parsing logic Improve regex for .git suffix detection and normalize repo data. * ✨ refactor resource resolution and add tests Separated script resolution logic and introduced gist handling. * ♻️ refactor: simplify resource imports by removing unused modules Removed unused imports including constants, crypto, workdir, host, and mathjs. * ✨ Enhance CLI run doc with resource info, fix indentation Improved documentation by adding resource resolution details and correcting indentation inconsistencies. * ✨: Add support for git repository URI pattern - Introduced `git://` URI pattern for resolving file globs * ✨ feat: enhance resource and script handling in CLI and core Added new parameters to CLI, improved resource resolution, and caching. * ✨ enhance: improve resource test cleanup and readability - Added async cleanup with recursive rmdir in tests. - Removed redundant console log statements. * ✨ feat: add directory check before processing files Added logic to skip processing when encountering directories. * ✨ feat: enhance WhisperASR with debugging and signal support Added debugging with genaiscriptDebug and support for cancellation signals. * updating packages * show size in build * ✨: Introduce advanced GitHub client features Enhanced client with APIs, job utils, and streamlined functionality. * ignore a few files * Release 1.130.3 * [skip ci] updated version numbers * ✨: Add prompty file parsing support to parsers Expanded parsers to handle prompty files and updated types accordingly. * ✨ Add onMessage handler for resourceChange messages (#1480) Enhances worker thread with onMessage callback support. * Release 1.130.4 * [skip ci] updated version numbers * round robin example * push experiment * comments * ✨ Add Prompty support and refine parsers/doc updates Expanded documentation for Prompty and parsers with cleaner formatting. * ✨ feat: add error logging to OutputTrace interface Added an `error` method to log errors with optional metadata. * Release 1.130.5 * [skip ci] updated version numbers * ✨ refactor: Rename type in PromptyDocument interface - Replaced ChatCompletionMessageParam with ChatMessage in messages field * Release 1.130.6 * [skip ci] updated version numbers * ✨ chore: update dependencies across multiple packages Upgraded various dependencies to enhance functionality and security. * Adding a basic linter (#1482) * adding base script * add error message * ✨ feat: update git diff base to HEAD^ Adjusted base reference for `git.diff` from "dev" to "HEAD^". * ✨: Enhance git diff configuration for linters Updated `git.diff` to include `ignoreSpaceChange` option. * ✨ feat: enhance linter script with base parameter support Added a configurable "base" parameter for defining git diff base commit. * 🚚 Rename and enhance workflows and linter paths Moved linters to .github directory and updated workflow logic. * ✨ feat: Enhance linting with git diff and tool support Added git diff handling with extended features and new tools. * ✨: Improve AI evaluation prompt clarity Added a note encouraging evaluation with 👍👎 feedback. * ✨: Add Discord link to social footer configuration Added a Discord link to the social array in astro.config.mjs. * ✨ feat: Update hero links with Discord integration Replaced Blog link with Discord invite link in the hero section. * fix definitions when opening .mts script * Fix draft file saving issue in genaiscript (#1486) * ✨ refactor GitClient for improved error handling Simplified branch handling and added error fallback support. * updated linter model * update reviews models * don't use color emojis (#1487) * collapse system messages (#1488) * ♻️ Simplify trace.startDetails parameters Simplified the call by removing the expanded option parameter. * ✨: Enhance debugging and error handling in core modules Improved debug logs, error reporting, and response handling refinements. * more triggers * add linter (#1489) * add linter * ✨ Add Discord server link to README Added a link to the project's Discord server for community engagement. * 💬 Add clarity to Discord server links Updated text to improve Discord join links for consistency. * add diagram * remove extension * ⚡ Update linter rules and tools configuration Enhanced diagram labeling rules and added new linter tools details. * ✨: Add PR description validation guideline Ensure PR descriptions align with changes in the diff for clarity. * 🎨: Enhance lint output formatting Use GitHub Flavored Markdown for richer annotations * ✨ feat: enhance linter tools and add stats markdown Added new linter analysis capabilities and tools for reporting stats. * Release 1.130.7 * [skip ci] updated version numbers * move linters (#1490) * move linters * add genaisrc path to linter workflow triggers * ✨ feat: Enhance GenAIScript with prompty file parser Added support for parsing .prompty files in scripts for GenAIScript. * 🎨: Refine grumpy-dev review role description Changed the role description to be more concise. Removed task section. * added discord blog post * ✨ improve argument handling in resolveImportPrompty - Enhanced logging and refined error handling for extra arguments. * ♻️ Update genai linting workflow and script configurations Integrated push triggers and expanded linter script features. * ✨ feat: Add notification for no doc changes in diff - Notify user when no changes found in docs directory. * 🎨 refactor: Update system title and description for Z3 Refined the system's title and description for clarity and accuracy. * github copilot chat as default provider (#1491) * add settings * ✨ Clarify LLM provider description in settings Updated the description to mention all VS Code-supported models. * cleanup * use option to specif provider * change default * fix config name * updated docs * fix condition * built-in z3 (#1493) * ✨ Enhance Z3 agent description for clarity Expanded the Z3 agent description to include solving constraint systems. * Release 1.131.0 * [skip ci] updated version numbers * ✨ chore: Enhance file handling, improve MIME logic Streamlined debugging imports, added MIME rules, and new sanitize script. * ✨ chore: Update devDependencies in package.json Upgraded astro and starlight-blog to newer versions. * Create no-fix-mes * Rename no-fix-mes to no-fix-mes.md * 🎨: Refactor resolveFileContent return type handling Refactored the function to improve clarity and type consistency. * ✨ chore: update MIME type for JavaScript files - Changed MIME type for .js from text/javascript to application/javascript * a few more processes using models * add docs workflow * tweak /docs action * typo * 🎨 update compile step in workflow config Replaced 'yarn compile:cli' with 'yarn compile' for build consistency. * add diff to genai:docs * add typechecks/ fetch dev * debug logging * handle nothing to do case * git status * update before generate * removed pattern * adding gpt-image-1 support (#1492) * refresh dependencies * re-neable blog generator * updated slides * Release 1.132.0 * [skip ci] updated version numbers * Parse tokenize (#1497) * ✨ feat: add tokenize command to CLI for token processing Adds a new tokenize command to parse text into hex tokens with options. * docs generator * remove comments * updated docs prompt --------- Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com> Co-authored-by: Andrew Noblet <andrewbnoblet@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
tokenizecommand in the CLI, allowing users to tokenize text from a file and display token values in hexadecimal format.parseTokenizeFunction: Implemented a new utility function to handle the tokenization logic, utilizing encoding models for token generation and output.tokenizecommand, including argument parsing and options like--model.tokenizecommand.