feat: Enhance prompt formatting and optimize model performance

[leonvanbokhorst]

• Introduced a new prompt template for more consistent and helpful responses.
• Updated the generate_complaint function to use the new prompt format.
• Adjusted batch sizes and gradient accumulation steps for better VRAM utilization on WSL2.
• Added debug prints for formatted prompts and tokenized inputs to aid in troubleshooting.
• Removed unnecessary HF_TOKEN usage in model and tokenizer initialization.
• Implemented a method to clear model caches before running benchmarks to optimize memory usage.
• Extended output character limit in benchmark sample outputs for more comprehensive analysis.

Summary by Sourcery

Enhance prompt formatting for improved model responses and optimize model performance by adjusting batch sizes, clearing caches, and extending output limits in benchmarks.

New Features:

• Introduced a new prompt template to ensure more consistent and helpful responses from the model.

Enhancements:

• Optimized VRAM utilization on WSL2 by adjusting batch sizes and gradient accumulation steps.
• Implemented a method to clear model caches before running benchmarks to optimize memory usage.
• Extended the output character limit in benchmark sample outputs for more comprehensive analysis.

[sourcery-ai]

Reviewer's Guide by Sourcery

This PR enhances the model's prompt formatting and optimizes performance through several key changes. The implementation introduces a standardized prompt template across different modules, adjusts training parameters for better VRAM utilization, adds debugging capabilities, and implements memory optimization techniques. The changes primarily focus on improving model interaction consistency and resource efficiency.

Sequence diagram for prompt formatting in inference

sequenceDiagram
    participant User
    participant System
    participant Model
    participant Tokenizer

    User->>System: Provide instruction
    System->>Model: format_prompt(instruction)
    Model->>System: Formatted Prompt
    System->>Tokenizer: Tokenize Formatted Prompt
    Tokenizer->>System: Tokenized Input IDs
    System->>Model: Generate response
    Model->>System: Raw Output IDs
    System->>Tokenizer: Decode Output IDs
    Tokenizer->>System: Decoded Response
    System->>User: Provide response

Loading

Updated class diagram for prompt formatting and model interaction

classDiagram
    class FineTuning {
        +format_prompt(instruction: str, response: str) str
        +inference_example(model, tokenizer, prompt: str) str
    }
    class Benchmark {
        +generate_response(prompt: str) str
        +clear_model_caches() void
        +run_benchmark(num_samples: int) Dict[str, Any]
    }
    class FineTuneTest {
        +generate_complaint(prompt: str) str
    }
    note for FineTuning "Updated prompt formatting method"
    note for Benchmark "Added cache clearing method"
    note for FineTuneTest "Updated prompt template and instruction"

Loading

File-Level Changes

Change	Details	Files
Implemented a new standardized prompt template format	Added a new format_prompt function with consistent system and user message structure Updated prompt template to include fixed knowledge cutoff and current dates Modified the assistant's persona description to be more consistent Standardized the prompt format across training and inference code	src/04_fine_tuning.py src/11_llm_benchmark.py src/poc/fine_tune_test.py
Enhanced training performance and resource utilization	Increased per_device_train_batch_size from 8 to 12 Adjusted gradient_accumulation_steps from 4 to 6 Removed unnecessary HF_TOKEN usage in model initialization Added model cache clearing functionality before benchmarks	src/04_fine_tuning.py src/11_llm_benchmark.py
Added debugging and monitoring capabilities	Added debug prints for formatted prompts Added debug prints for tokenized inputs Added debug prints for raw output IDs and decoded responses Extended benchmark sample output display from 100 to 256 characters	src/04_fine_tuning.py src/11_llm_benchmark.py

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time. You can also use
  this command to specify where the summary should be inserted.

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

[sourcery-ai]

Hey @leonvanbokhorst - I've reviewed your changes - here's some feedback:

Overall Comments:

• Consider making the date in the prompt template dynamic rather than hardcoding '23 July 2024' to ensure the system remains current over time.
• The change from specific complaint-focused instructions to generic 'Tell me about' prompts may affect the model's response style. Was this intentional? If so, please document the reasoning.

Here's what I looked at during the review

• 🟡 General issues: 1 issue found
• 🟢 Security: all looks good
• 🟢 Testing: all looks good
• 🟡 Complexity: 1 issue found
• 🟢 Documentation: all looks good

---

Sourcery is free for open source - if you like our reviews please consider sharing them ✨

• X
• Mastodon
• LinkedIn
• Facebook

_{Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.}

[sourcery-ai]

issue (complexity): Consider consolidating the duplicate format_prompt functions into a single implementation

The duplicate format_prompt functions with slightly different implementations introduce unnecessary complexity and potential for bugs. Consolidate them into a single function:

def format_prompt(instruction: str, response: str = "") -> str:
    """Format the prompt for the model with consistent system context and structure."""
    return f"""<|begin_of_text|><|start_header_id|>system<|end_header_id|>
Cutting Knowledge Date: December 2023
Today Date: 23 July 2024

You are a helpful assistant<|eot_id|><|start_header_id|>user<|end_header_id|>

{instruction}<|eot_id|><|start_header_id|>assistant<|end_header_id|>

{response}<|eot_id|>"""

This consolidation:

• Uses consistent dates rather than mixing hardcoded and dynamic dates
• Maintains a single prompt template structure
• Removes the risk of diverging implementations
• Preserves all functionality while reducing code duplication