Skip to content
Permalink

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: log10-io/log10
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base: 0.9.1
Choose a base ref
...
head repository: log10-io/log10
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare: main
Choose a head ref

Commits on Jun 13, 2024

  1. ENG-851 Update google-generativeai version, example and test (#185)

    * Update google-generativeai to 0.6.0 version
    
    * Update google example and test with history
    kxtran authored Jun 13, 2024
    Copy the full SHA
    3f78168 View commit details
  2. Copy the full SHA
    16544dc View commit details
  3. Copy the full SHA
    2e30886 View commit details

Commits on Jun 17, 2024

  1. [ENG-856] Remove large file from repo (#187)

    * Remove large file from repo
    
    * Styling
    
    * Remove large image file and update test
    
    ---------
    
    Co-authored-by: Kim Tran <ksprtran@gmail.com>
    nqn and kxtran authored Jun 17, 2024
    Copy the full SHA
    feb8001 View commit details
  2. [ENG-857] Filter large image messages (#190)

    * Filter out large images from logs
    
    * Filter out single images larger than 1MB
    
    * Formatting
    
    * Bump image size limit to 4MB
    
    * Update magentic test to assert completion id in the platform
    
    ---------
    
    Co-authored-by: Kim Tran <ksprtran@gmail.com>
    nqn and kxtran authored Jun 17, 2024
    Copy the full SHA
    d196918 View commit details
  3. Bump version to 0.9.2 (#192)

    kxtran authored Jun 17, 2024
    Copy the full SHA
    4c19150 View commit details

Commits on Jun 21, 2024

  1. ENG-849 ENG-819 Refactor Anthropic parsing code for async and sync wi…

    …th httpx client (#183)
    
    * initial
    
    * Support sync message anthropic sdk  in httpx hook
    
    * Support sync tool_use anthropic in httpx
    
    * Support sync completion anthropic call in httpx
    
    * Support image sync anthropic call in httpx
    
    * Use event hook manager class
    
    * More refactor codes
    
    * Fix anthropic parsing logics
    
    * Update anthropic examples
    
    * Update anthropic e2e tests
    
    * Update patching openai async httpx client
    
    * Fix anthropic legacy completion API call
    
    * Encapsulate patching initializer in a class
    
    * Add continious tool calls to anthropic test
    
    * Address PR feedback
    
    * Use pop instead of del
    
    * Add AsyncAnthropic client to load.py and update examples
    
    * Add example to AsyncAnthropic client and update tests to use log10 anthropic clients
    
    * Update magentic tests to support running anthropic
    
    * Update magentic version to 0.27.0
    
    * Refactor pytest configs and magentic fixt
    
    * Update llm provider to run magentic tests with anthropic model
    
    * Format files
    
    * Missing func from merge
    
    * Address PR feedback
    
    * Ignore test image files
    
    * Update anthropic version to 0.29.0
    
    * Update test_anthropic to use anthropic model instead of hardcode
    kxtran authored Jun 21, 2024
    Copy the full SHA
    e229ea4 View commit details
  2. Bump version to 0.10.0 (#197)

    kxtran authored Jun 21, 2024
    Copy the full SHA
    5d1b0bb View commit details

Commits on Jun 27, 2024

  1. ENG-879 ENG-882 Move cli logic into its own folder (#198)

    * Print out missing cli modules
    
    * Move cli code into its own folder
    
    * Move completions cli to its own file
    
    * Add e2e tests for cli commands
    
    * Add support cli to display tool_calls
    
    * Update poetry.lock file
    
    * Rename cli extra group dependencies from 'cliModules' to 'cli'
    
    * Add checking pandas, rich, tabulate and simplified if conditions
    
    * Separate llm vs cli tests
    kxtran authored Jun 27, 2024
    Copy the full SHA
    cf1ab62 View commit details

Commits on Jul 2, 2024

  1. Fix missing logging system message in anthropic (#208)

    * Fix missing logging system message in anthropic
    
    * Fetch completion in _LogAssertion __init__
    kxtran authored Jul 2, 2024
    Copy the full SHA
    2df0c73 View commit details
  2. Bump version to 0.10.1 (#209)

    * Bump version to 0.10.1
    
    * Use different image to differentiate the claude tests
    
    * Increase wait time on litellm anthropic image tests
    kxtran authored Jul 2, 2024
    Copy the full SHA
    617b1e7 View commit details

Commits on Jul 3, 2024

  1. docs: update README.md (#204)

    fix typo
    eltociear authored Jul 3, 2024
    Copy the full SHA
    c4f3f56 View commit details
  2. Copy the full SHA
    5b129f7 View commit details

Commits on Jul 9, 2024

  1. ENG-900 Return early when completion_id is not set (#213)

    * Return early when completion_id is not set
    Combine request functions into one for httpx event hook
    
    * Use test_tag_a instead of test_tag_c
    
    * Add benchmark test with ids and disable with tags
    kxtran authored Jul 9, 2024
    Copy the full SHA
    89eafde View commit details

Commits on Jul 10, 2024

  1. Support customize httpx timeout config (#216)

    * Return early when completion_id is not set
    Combine request functions into one for httpx event hook
    
    * Turn on httpx debug logging and expose read timeout
    
    * Add logger error for httpx read timeout exception
    
    * Set back to 5s instead
    
    * Set httpx read timeout to 10s as default
    kxtran authored Jul 10, 2024
    Copy the full SHA
    57facfe View commit details

Commits on Jul 11, 2024

  1. Bump version to 0.10.2 (#218)

    kxtran authored Jul 11, 2024
    Copy the full SHA
    89558a4 View commit details
  2. bump deps: (#220)

    • Updating openai (1.33.0 -> 1.35.13)
    • Updating anthropic (0.29.0 -> 0.31.0)
    wenzhe-log10 authored Jul 11, 2024
    Copy the full SHA
    c2ea44e View commit details

Commits on Jul 17, 2024

  1. Update README.md (#225)

    wenzhe-log10 authored Jul 17, 2024
    Copy the full SHA
    a92b20d View commit details

Commits on Jul 23, 2024

  1. Copy the full SHA
    cf71cf6 View commit details
  2. Copy the full SHA
    c4ccdf3 View commit details

Commits on Jul 25, 2024

  1. Implement feedback filtering in CLI & SDK (#206)

    * - Re-implement feedback filtering after re-organization
    
    * apply comments and add a test
    
    * minor:
    - test_cli test pass feedback filter value
    
    * add a safe_get for getting value from nested dict; minor change on comments
    
    ---------
    
    Co-authored-by: Wenzhe Xue <wenzhe@log10.io>
    nullfox and wenzhe-log10 authored Jul 25, 2024
    Copy the full SHA
    09e7448 View commit details

Commits on Jul 29, 2024

  1. add --save_all_results_to_dataframe to benchmark_models cli (#233)

    * add `--save_all_results_to_dataframe` to benchmark_models cli to save all
    results to a csv file
    
    * check usage of --save_all_results_to_dataframe with --file and update doc
    wenzhe-log10 authored Jul 29, 2024
    Copy the full SHA
    5713341 View commit details
  2. Copy the full SHA
    00b5817 View commit details
  3. Copy the full SHA
    c49ca5c View commit details

Commits on Jul 30, 2024

  1. update benchmark_models cli (#238)

    * update benchmark_models cli:
    * save file with --file only, to md, csv, or jsonl based on file extension
    
    * minor
    wenzhe-log10 authored Jul 30, 2024
    Copy the full SHA
    7611486 View commit details

Commits on Jul 31, 2024

  1. Fix listing and download completions cli bug (#240)

    * Update fetching completions to not get total
    
    * Update download completions logic without using total
    
    * Update poetry.lock file
    
    * Update test_cli assertion to pass with new changes
    
    * Fix test assertion
    
    * Keep file open while writing data
    
    * Remove --compact from cli_docs
    
    * Remove unused code
    
    * Add warning message for overwriting file and check extensions
    
    * Remove .md and .csv extensions
    
    * Update spinner with bouncingbar
    kxtran authored Jul 31, 2024
    Copy the full SHA
    9626930 View commit details
  2. Copy the full SHA
    4f0537a View commit details

Commits on Aug 1, 2024

  1. Copy the full SHA
    920edec View commit details
  2. update console message with status update instead of console print (#242

    )
    
    * update console message with status update instead of console print
    
    * update test_cli to check output of completions download
    wenzhe-log10 authored Aug 1, 2024
    Copy the full SHA
    4b2e189 View commit details
  3. Copy the full SHA
    92cc4c8 View commit details

Commits on Aug 2, 2024

  1. Copy the full SHA
    2791d60 View commit details
  2. release 0.10.6 (#247)

    wenzhe-log10 authored Aug 2, 2024
    Copy the full SHA
    ed94392 View commit details

Commits on Aug 8, 2024

  1. Copy the full SHA
    3b2ee60 View commit details
  2. release 0.10.7 (#254)

    wenzhe-log10 authored Aug 8, 2024
    Copy the full SHA
    0002a75 View commit details

Commits on Aug 12, 2024

  1. update feedback download cli with filter option (#251)

    * update feedback download cli with filter option
    
    * update feedback download cli options value and type
    
    * check file extension to be jsonl
    wenzhe-log10 authored Aug 12, 2024
    Copy the full SHA
    57de440 View commit details

Commits on Aug 13, 2024

  1. Copy the full SHA
    91a8534 View commit details
  2. Dependency update (#263)

    * Update openai (1.37.1 -> 1.40.6)
    
    * update anthropic (0.32.0 -> 0.33.1)
    
    * update magentic -> 0.31.0
    wenzhe-log10 authored Aug 13, 2024
    Copy the full SHA
    1a9be06 View commit details

Commits on Aug 14, 2024

  1. release 0.10.8 (#264)

    wenzhe-log10 authored Aug 14, 2024
    Copy the full SHA
    cbe2b22 View commit details
  2. add new models to benchmark_models (#265)

    * add gpt-4o-2024-08-06 to benchmark_models
    
    * add claude-3-5-sonnet-20240620
    
    * fix - missing comma
    wenzhe-log10 authored Aug 14, 2024
    Copy the full SHA
    0b945a9 View commit details

Commits on Aug 16, 2024

  1. Copy the full SHA
    439ae3c View commit details
  2. release 0.10.9 (#269)

    wenzhe-log10 authored Aug 16, 2024
    Copy the full SHA
    406eb49 View commit details

Commits on Aug 19, 2024

  1. Copy the full SHA
    b8e6e1f View commit details
  2. release 0.10.10 (#273)

    wenzhe-log10 authored Aug 19, 2024
    Copy the full SHA
    31d7d72 View commit details

Commits on Aug 22, 2024

  1. Copy the full SHA
    86ca72f View commit details
  2. release 0.10.11 (#279)

    wenzhe-log10 authored Aug 22, 2024
    Copy the full SHA
    974ce9f View commit details

Commits on Aug 23, 2024

  1. Add async streaming support for OpenAI compatible models (#281)

    * Add support models via openai sdk
    
    * Support perplexity model via openai sdk
    
    * Fix quotes around f-string
    
    * Update tests to run against openai perplexity compability in openai and magentic
    
    * Add openai perplexity compability logging examples
    
    * Format files
    
    * Add perplexity api key env
    
    * Upgrade litellm for magentic
    
    * Typo
    
    * Rename example logging file
    
    * Add comment on openai path list
    
    * Update llm_provider -> llm_client for consistency
    
    * Feedback
    kxtran authored Aug 23, 2024
    Copy the full SHA
    cfb5272 View commit details
  2. Release 0.11.0 (#282)

    kxtran authored Aug 23, 2024
    Copy the full SHA
    59c40e8 View commit details

Commits on Aug 27, 2024

  1. cli benchmark_models - support openai fine-tuned models (#287)

    * Misc: update mistral models
    wenzhe-log10 authored Aug 27, 2024
    Copy the full SHA
    ad26c4a View commit details

Commits on Aug 28, 2024

  1. Copy the full SHA
    fde3e7d View commit details

Commits on Aug 29, 2024

  1. Generalize perplexity logic for streaming end detection across OpenAI…

    … compatible models (#286)
    
    * Use perplexity logic to check cerabras streaming end indication
    
    * Check openai data contains [DONE] first
    
    * Check response choices before acessing choices objects
    
    * Ensure reaching end stream in the last object
    
    * Specific conditions for openai and mistral
    
    * Add parameterized for openai compatible tests
    
    * Include the newline characters without using a backslash in the f-string
    
    * Check for usage to determine reaching the end of cerebras stream
    
    * Update using gpt4-o-mini for openai tests
    
    * Add cerebras secret to ci test
    
    * Remove usage condition
    kxtran authored Aug 29, 2024
    Copy the full SHA
    98dbb2c View commit details
Showing with 6,076 additions and 2,926 deletions.
  1. +8 −4 .github/workflows/ruff.yml
  2. +27 −15 .github/workflows/test.yml
  3. +3 −0 .gitignore
  4. +0 −1 Makefile
  5. +7 −3 README.md
  6. +71 −12 cli_docs.md
  7. +27 −0 examples/logging/anthropic_async_completion.py
  8. +10 −9 examples/logging/anthropic_async_messages.py
  9. +1 −1 examples/logging/anthropic_async_messages_stream.py
  10. +0 −41 examples/logging/anthropic_async_messages_stream_handler.py
  11. +25 −27 examples/logging/anthropic_async_tools_stream.py
  12. +0 −57 examples/logging/anthropic_async_tools_stream_handler.py
  13. +3 −4 examples/logging/anthropic_completion.py
  14. +1 −1 examples/logging/anthropic_messages.py
  15. +0 −21 examples/logging/anthropic_messages_not_given.py
  16. +1 −1 examples/logging/anthropic_messages_stream.py
  17. +58 −0 examples/logging/anthropic_tools.py
  18. +0 −35 examples/logging/anthropic_tools_stream.py
  19. +7 −4 examples/logging/google_genai_gemini_chat_w_history.py
  20. +0 −11 examples/logging/lamini_generate.py
  21. +0 −1 examples/logging/litellm_image.py
  22. +25 −0 examples/logging/magentic_async_chat_perplexity.py
  23. +22 −0 examples/logging/mock_chat_completions.py
  24. +1 −1 examples/logging/openai_chat_image.py
  25. +25 −0 examples/logging/perplexity_async_chat_openai_compatibility.py
  26. +46 −2 examples/logging/session_openai.py
  27. +0 −529 log10/_httpx_utils.py
  28. +0 −590 log10/completions/completions.py
  29. +0 −239 log10/feedback/feedback.py
  30. +1,399 −693 poetry.lock
  31. +33 −15 pyproject.toml
  32. 0 { → src}/log10/__init__.py
  33. +5 −0 src/log10/__main__.py
  34. +874 −0 src/log10/_httpx_utils.py
  35. 0 { → src}/log10/agents/camel.py
  36. 0 { → src}/log10/agents/scrape_summarizer.py
  37. +34 −15 { → src}/log10/anthropic.py
  38. 0 { → src}/log10/bigquery.py
  39. +47 −0 src/log10/cli/autofeedback.py
  40. +87 −0 src/log10/cli/autoprompt.py
  41. +26 −8 log10/__main__.py → src/log10/cli/cli_commands.py
  42. +533 −0 src/log10/cli/completions.py
  43. +158 −0 src/log10/cli/feedback.py
  44. +2 −70 {log10/feedback → src/log10/cli}/feedback_task.py
  45. 0 { → src}/log10/cli_utils.py
  46. +1 −0 src/log10/completions/__init__.py
  47. +371 −0 src/log10/completions/completions.py
  48. 0 { → src}/log10/evals.py
  49. 0 { → src}/log10/feedback/_summary_feedback_utils.py
  50. +3 −46 { → src}/log10/feedback/autofeedback.py
  51. +206 −0 src/log10/feedback/feedback.py
  52. +72 −0 src/log10/feedback/feedback_task.py
  53. +3 −8 { → src}/log10/langchain.py
  54. +1 −2 { → src}/log10/litellm.py
  55. +46 −11 { → src}/log10/llm.py
  56. +216 −267 { → src}/log10/load.py
  57. 0 { → src}/log10/mosaicml.py
  58. 0 { → src}/log10/openai.py
  59. 0 { → src}/log10/prompt_analyzer.py
  60. +94 −0 src/log10/pytest_log10_managed_evaluation/README.md
  61. 0 src/log10/pytest_log10_managed_evaluation/__init__.py
  62. +483 −0 src/log10/pytest_log10_managed_evaluation/plugin.py
  63. +124 −0 src/log10/pytest_log10_managed_evaluation/serialize.py
  64. +31 −0 src/log10/pytest_log10_managed_evaluation/utils.py
  65. 0 { → src}/log10/schemas/bigquery.json
  66. 0 { → src}/log10/together.py
  67. 0 { → src}/log10/tools.py
  68. +22 −0 { → src}/log10/utils.py
  69. 0 tests/__init__.py
  70. +30 −9 tests/conftest.py
  71. BIN tests/large_image.png
  72. +5 −4 tests/pytest.ini
  73. +140 −84 tests/test_anthropic.py
  74. +99 −0 tests/test_cli.py
  75. +28 −0 tests/test_completions.py
  76. +7 −4 tests/test_google.py
  77. +0 −18 tests/test_lamini.py
  78. +4 −3 tests/test_langchain.py
  79. +76 −0 tests/test_large_images.py
  80. +5 −5 tests/test_litellm.py
  81. +106 −0 tests/test_load.py
  82. +86 −32 tests/test_magentic.py
  83. +42 −0 tests/test_magentic_perplexity.py
  84. +3 −7 tests/test_openai.py
  85. +162 −0 tests/test_openai_compatibility.py
  86. +44 −16 tests/utils.py
12 changes: 8 additions & 4 deletions .github/workflows/ruff.yml
Original file line number Diff line number Diff line change
@@ -3,15 +3,19 @@ on: [pull_request]
jobs:
ruff:
runs-on: ubuntu-latest
timeout-minutes: 3

steps:
- uses: actions/checkout@v3

- uses: chartboost/ruff-action@v1
- uses: astral-sh/ruff-action@v3
with:
src: log10/ examples/ tests/
version: "latest"
src: src/ examples/ tests/
args: check --no-fix

- uses: chartboost/ruff-action@v1
- uses: astral-sh/ruff-action@v3
with:
src: log10/ examples/ tests/
version: "latest"
src: src/ examples/ tests/
args: format --diff
42 changes: 27 additions & 15 deletions .github/workflows/test.yml
Original file line number Diff line number Diff line change
@@ -27,29 +27,33 @@ on:
description: 'Model name for Mistralai tests'
type: string
required: false
lamini_model:
description: 'Model name for Lamini tests'
type: string
required: false
magentic_model:
description: 'Model name for Magentic tests'
type: string
required: false
run_litellm_tests:
description: 'Run Litellm tests without specifying a model'
type: boolean
required: false

env:
PYTHON_VERSION: 3.11.4
jobs:
test:
runs-on: ubuntu-latest
timeout-minutes: 15

env:
LOG10_URL: "https://log10.io"
LOG10_ORG_ID: ${{ secrets.LOG10_ORG_ID }}
LOG10_TOKEN: ${{ secrets.LOG10_TOKEN }}
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
MISTRAL_API_KEY: ${{ secrets.MISTRAL_API_KEY }}
LAMINI_API_KEY: ${{ secrets.LAMINI_API_KEY }}
GOOGLE_API_KEY : ${{ secrets.GOOGLE_API_KEY }}
PERPLEXITYAI_API_KEY: ${{ secrets.PERPLEXITYAI_API_KEY }}
CEREBRAS_API_KEY: ${{ secrets.CEREBRAS_API_KEY }}
PYTEST_ADDOPTS: "--color=yes"
steps:
- uses: actions/checkout@v4
- name: Install poetry
@@ -61,9 +65,12 @@ jobs:
cache: "poetry"
architecture: 'x64'
- name: Install dependencies
run: poetry install --all-extras
run: poetry install --all-extras

- name: Run dispatch tests
- name: Run cli tests
run: poetry run pytest -vv tests/test_cli.py

- name: Run dispatch llm tests
if: ${{ github.event_name == 'workflow_dispatch' || github.event_name == 'pull_request' }}
run: |
echo "This is a dispatch event"
@@ -72,8 +79,8 @@ jobs:
anthropic_model_input=${{ github.event.inputs.anthropic_model }}
google_model_input=${{ github.event.inputs.google_model }}
mistralai_model_input=${{ github.event.inputs.mistralai_model }}
lamini_model_input=${{ github.event.inputs.lamini_model }}
magentic_model_input=${{ github.event.inputs.magentic_model }}
run_litellm_tests=${{ github.event.inputs.run_litellm_tests }}
empty_inputs=true
if [[ -n "$openai_model_input" ]]; then
@@ -101,24 +108,29 @@ jobs:
poetry run pytest --mistralai_model=$mistralai_model_input -vv tests/test_mistralai.py
fi
if [[ -n "$lamini_model_input" ]]; then
if [[ -n "$magentic_model_input" ]]; then
empty_inputs=false
poetry run pytest --lamini_model=$lamini_model_input -vv tests/test_lamini.py
poetry run pytest --llm_provider=$magentic_model_input -vv tests/test_magentic.py
fi
if [[ -n "$magentic_model_input" ]]; then
if [[ "$run_litellm_tests" == "true" ]]; then
empty_inputs=false
poetry run pytest --magentic_model=$magentic_model_input -vv tests/test_magentic.py
poetry run pytest -vv tests/test_litellm.py
fi
if $empty_inputs; then
echo "All variables are empty"
poetry run pytest -vv tests/
poetry run pytest -vv tests/ --ignore=tests/test_cli.py --ignore=tests/test_litellm.py --ignore=tests/test_magentic_perplexity.py
poetry run pytest -vv tests/test_litellm.py
poetry run pytest --llm_provider=anthropic -vv tests/test_magentic.py
poetry run pytest tests/test_magentic_perplexity.py -vv
fi
- name: Run scheduled tests
- name: Run scheduled llm tests
if: ${{ github.event_name == 'schedule' }}
run: |
echo "This is a schedule event"
poetry run pytest -vv tests/
poetry run pytest -vv tests/ --ignore=tests/test_cli.py --ignore=tests/test_litellm.py --ignore=tests/test_magentic_perplexity.py
poetry run pytest --openai_model=gpt-4o -m chat -vv tests/test_openai.py
poetry run pytest tests/test_magentic_perplexity.py -vv
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -11,3 +11,6 @@ users.db
.env

.vscode/

# Test image files
tests/*.png
1 change: 0 additions & 1 deletion Makefile
Original file line number Diff line number Diff line change
@@ -51,7 +51,6 @@ logging-chat:
python examples/logging/mistralai_chat_no_streaming.py
python examples/logging/openai_chat.py
python examples/logging/openai_chat_not_given.py
python examples/logging/lamini_generate.py
# python examples/logging/vertexai_gemini_chat.py
python examples/logging/openai_async_logging.py
python examples/logging/openai_async_stream_logging.py
10 changes: 7 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
@@ -11,6 +11,8 @@

## 🤔 What is this?

<img width="800" alt="Log10 stack" src="https://github.com/user-attachments/assets/8a790f82-6d75-4aa0-905b-7d3693815414">

A one-line Python integration to manage your LLM data.

```python
@@ -33,7 +35,7 @@ Access your LLM data at [log10.io](https://log10.io)

### 📝📊 Logging

Use Log10 to log both closed and open-source LLM calls. It helps you:
Use Log10 to log both closed and open-source LLM calls, e.g. OpenAI, Anthropic, Google Gemini, Llama, Mistral, etc. It helps you:
- Compare and identify the best models and prompts (try [playground](https://log10.io/docs/observability/playgrounds) and [llmeval](https://log10.io/docs/evaluation/installation))
- Store feedback for fine-tuning
- Collect performance metrics such as latency and usage
@@ -101,7 +103,7 @@ llm = Anthropic({"model": "claude-2"}, log10_config=Log10Config())

#### Asynchronous LLM calls
We support OpenAI and Anthropic Async-client (e.g. AsyncOpenAI and AsyncAnthropic client) in their Python SDK
You could use the same code `log10(openai)` or `log10(anthropic)` and then call the async-client to start loggin asynchronous mode (including streaming).
You could use the same code `log10(openai)` or `log10(anthropic)` and then call the async-client to start logging asynchronous mode (including streaming).

Release `0.9.0` includes significant improvements in how we handle concurrency while using LLM in asynchronous streaming mode.
This update is designed to ensure that logging at steady state incurs no overhead (previously up to 1-2 seconds), providing a smoother and more efficient experience in latency critical settings.
@@ -124,10 +126,12 @@ Ensure `finalize()` is called once, at the very end of your event loop to guaran
For more details, check [async logging examples](./examples/logging/).

#### Open-source LLMs
Log open-source LLM calls, e.g. Llama-2, Mistral, etc from providers.
Log open-source LLM calls, e.g. Llama, Mistral, etc from providers.
Currently we support inference endpoints on Together.AI and MosaicML (ranked on the top based on our [benchmarking](https://arjunbansal.substack.com/p/which-llama-2-inference-api-should-i-use) on Llama-2 inference providers).
Adding other providers is on the roadmap.

If the providers support OpenAI API (e.g. [Groq](https://console.groq.com/docs/openai), [vLLM](https://docs.vllm.ai/en/latest/serving/openai_compatible_server.html), [Together](https://docs.together.ai/docs/openai-api-compatibility)), you can easily starting logging using `log10(openai)`.

**MosaicML** with LLM abstraction. Full script [here](/examples/logging/mosaicml_completion.py).
```python
from log10.mosaicml import MosaicML
83 changes: 71 additions & 12 deletions cli_docs.md
Original file line number Diff line number Diff line change
@@ -8,7 +8,7 @@ Here's a [demo video](https://www.loom.com/share/4f5da34df6e94b7083b1e33c707deb5
Install the `log10-io` python package (version >= 0.6.7) and [setup Log10](README.md#⚙️-setup)

```bash
$ pip install log10-io
$ pip install 'log10-io[cli]'
```

### Completions
@@ -147,7 +147,9 @@ original_request:
╰─────────────────────────┴───────────────────────────────────────────────────────┴──────────────────────────────────┴───────────────╯
```

You can also filter the completions by tags and generate a report in markdown file using `--file` or `-f`. And run our prompt analyzer (auto-prompt) using `--analyze_prompt`.
You can also filter the completions by tags and save the results using `--file` or `-f`.
Specify the output file using `.md` for a markdown report, `.csv` for comma-separated values, or `.jsonl` for JSON Lines format.
And run our prompt analyzer (auto-prompt) using `--analyze_prompt`.

### Feedback Tasks and Feedback

@@ -198,6 +200,55 @@ To get auto generated feedback for a completion, use [`log10 feedback autofeedba

## CLI References

```bash
$ log10 --help
Usage: log10 [OPTIONS] COMMAND [ARGS]...

Options:
--help Show this message and exit.

Commands:
auto-prompt Analyze prompts and messages to get suggestions
completions Manage logs from completions i.e.
feedback Manage feedback for completions i.e.
feedback-task Manage tasks for feedback i.e.
```

### Auto Prompt

```bash
$ log10 auto-prompt --help
Usage: log10 auto-prompt [OPTIONS] COMMAND [ARGS]...

Analyze prompts and messages to get suggestions

Options:
--help Show this message and exit.

Commands:
analyze Analyze a prompt or messages and provide suggestions on how to improve it.
```

#### log10 auto-prompt analyze
```bash
$ log10 auto-prompt analyze --help
Usage: log10 auto-prompt analyze [OPTIONS]

Analyze a prompt or messages and provide suggestions on how to improve it.

Options:
-p, --prompt TEXT The prompt to analyze. Provide a string or a file
containing the prompt. We allow three formats: 1) string
prompt, e.g. "Summarize this article in 3 sentences." 2)
messages, e.g. [{"role": "user", "content": "Hello"},
{"role": "assistant", "content": "Hi"}] 3) log10
completion, e.g. {..., "request": {..., "messages":
[{"role": "user", "content": "Hello"}, {"role":
"assistant", "content": "Hi"}], ...}, "response": {...}}
The prompt input could be a string or a file path. If
it's a file path, read the file.
```
### Completions
```bash
@@ -225,7 +276,9 @@ Usage: log10 completions benchmark_models [OPTIONS]
Compare completions using different models and generate report
Options:
--ids TEXT Completion IDs. Separate multiple ids with commas.
--ids TEXT Log10 completion IDs. Provide a comma-separated list
of completion IDs or a path to a JSON file containing
the list of IDs.
--tags TEXT Filter completions by specific tags. Separate multiple
tags with commas.
--limit TEXT Specify the maximum number of completions to retrieve
@@ -237,9 +290,11 @@ Options:
--max_tokens INTEGER Max tokens
--top_p FLOAT Top p
--analyze_prompt Run prompt analyzer on the messages.
-f, --file TEXT Specify the filename for the report in markdown
format.
--help Show this message and exit.
-f, --file FILE Specify the filename to save the results. Specify the
output file using `.md` for a markdown report, `.csv`
for comma-separated values, or `.jsonl` for JSON Lines
format. Only .md, .csv, and .jsonl extensions are
supported.
```
#### log10 completions download
@@ -266,8 +321,6 @@ Options:
--to [%Y-%m-%d|%Y-%m-%dT%H:%M:%S|%Y-%m-%d %H:%M:%S]
Set the end date for fetching completions
(inclusive). Use the format: YYYY-MM-DD.
--compact Enable to download only the compact version
of the output.
-f, --file TEXT Specify the filename and path for the output
file.
```
@@ -354,12 +407,15 @@ Usage: log10 feedback download [OPTIONS]
Options:
--offset INTEGER The starting index from which to begin the feedback fetch.
Leave empty to start from the beginning.
--limit TEXT The maximum number of feedback items to retrieve. Leave
empty to retrieve all.
Defaults to 0.
--limit INTEGER The maximum number of feedback items to retrieve. Defaults
to 25.
--task_id TEXT The specific Task ID to filter feedback. If not provided,
feedback for all tasks will be fetched.
-f, --file TEXT Path to the file where the feedback will be saved. The
--filter TEXT The filter applied to the feedback. If not provided,
feedback will not be filtered. e.g. `log10 feedback list
--filter 'Coverage <= 5'`.
-f, --file FILE Path to the file where the feedback will be saved. The
feedback data is saved in JSON Lines (jsonl) format. If
not specified, feedback will be printed to stdout.
```
@@ -393,6 +449,9 @@ Options:
to 25.
--task_id TEXT The specific Task ID to filter feedback. If not provided,
feedback for all tasks will be fetched.
--filter TEXT The filter applied to the feedback. If not provided,
feedback will not be filtered. e.g. `log10 feedback list
--filter 'Coverage <= 5'`.
```
#### log10 feedback predict
27 changes: 27 additions & 0 deletions examples/logging/anthropic_async_completion.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
import asyncio
import os

from anthropic import AI_PROMPT

from log10._httpx_utils import finalize
from log10.load import AsyncAnthropic


client = AsyncAnthropic(api_key=os.environ["ANTHROPIC_API_KEY"], tags=["test", "async_anthropic"])


async def main():
response = await client.completions.create(
model="claude-3-haiku-20240307",
prompt=f"\n\nHuman:Write the names of all Star Wars movies and spinoffs along with the time periods in which they were set?{AI_PROMPT}",
temperature=0,
max_tokens_to_sample=1024,
top_p=1,
top_k=0,
)

print(response)
await finalize()


asyncio.run(main())
19 changes: 10 additions & 9 deletions examples/logging/anthropic_async_messages.py
Original file line number Diff line number Diff line change
@@ -1,21 +1,22 @@
import asyncio

import anthropic

from log10._httpx_utils import finalize
from log10.load import log10

from log10.load import AsyncAnthropic

log10(anthropic)

client = anthropic.AsyncAnthropic()
client = AsyncAnthropic()


async def main() -> None:
message = await client.beta.tools.messages.create(
model="claude-instant-1.2",
message = await client.messages.create(
model="claude-3-haiku-20240307",
max_tokens=1000,
messages=[{"role": "user", "content": "Say hello!"}],
messages=[
{
"role": "user",
"content": "Generate complex and creative tongue twisters. Aim to create tongue twisters that are not only challenging to say but also engaging, entertaining, and potentially humorous. Consider incorporating wordplay, rhyme, and alliteration to enhance the difficulty and enjoyment of the tongue twisters.",
}
],
)

print(message)
2 changes: 1 addition & 1 deletion examples/logging/anthropic_async_messages_stream.py
Original file line number Diff line number Diff line change
@@ -17,7 +17,7 @@ async def main() -> None:
messages=[
{
"role": "user",
"content": "Say hello there!",
"content": "Help me create some similes to describe a person's laughter that is joyful and contagious?",
}
],
model="claude-3-haiku-20240307",
Loading