Enhance experience replay for priority queue buffer#306
Enhance experience replay for priority queue buffer#306pan-x-c merged 7 commits intoagentscope-ai:mainfrom
Conversation
Summary of ChangesHello @yanxi-chen, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request significantly enhances the experience replay buffer's priority queue mechanism by introducing a more flexible priority function interface. This change allows for conditional insertion of experiences into the buffer, enabling finer-grained control over which data points are retained. A new strategy is also added to limit the reuse of experiences, and the default decay rate for an existing priority function is adjusted to reflect these improvements. Highlights
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request enhances the experience replay mechanism for the priority queue buffer. The priority functions are updated to return a boolean flag, put_into_queue, which provides more control over whether an item is added to the queue. A new priority function, linear_decay_use_count_control_priority, is introduced to demonstrate this by limiting the number of times an experience can be used. The changes are logical and well-implemented. My main feedback is to add docstrings to the new priority functions to improve documentation, as noted in the PR checklist.
yanxi-chen
changed the title
|
/unittest-module-buffer |
Summary
Tests
Github Test Reporter by CTRF 💚 |
|
/unittest-all |
|
/unittest-module-common |
Summary
Tests
Github Test Reporter by CTRF 💚 |
3 participants
Description
Enhance experience replay for priority queue buffer.
linear_decay, change defaultdecayvalue from 0.1 to 2.0; this could make the replay mechanism more reliable, with less dependence on setting the cooldown-time parameter appropriatelycapacity = min(storage_config.capacity, 2 * train_batch_size)tocapacity = storage_config.capacity(and update unittests accordingly)priority_groups)Checklist
Please check the following items before code is ready to be reviewed.