Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix llm-based evaluation metrics #572

Merged
merged 21 commits into from
May 24, 2024
Merged

Fix llm-based evaluation metrics #572

merged 21 commits into from
May 24, 2024

Conversation

kcortinas
Copy link
Collaborator

@kcortinas kcortinas commented May 23, 2024 •
edited
Loading

This PR fixes:

  • llm_context_precision: the metric considers the context used to generate the question-answer pair in step 2 (02_qa_generation.py), instead of the retrieved contexts. To assess the ability of the system to retrieve relevant chunks/contexts, we need to consider the relevancy of the retrieved contexts against the question. The computation of the metric is also using as input the generated answer (actual) instead of the question. The updated metric computes a simple average precision (proportion of relevant chunks without consideration of the ranking order).
  • llm_context_recall: this metric is also using the qna context instead of the retrieved contexts. To assess the ability of the system in retrieving contexts that are aligned with the ground truth answer we need to consider the retrieved contexts. The temperature parameter is removed as it's already part of the initialisation of ResponseGenerator.
  • llm_answer_relevance: the metric currently inputs the generated answer and the ground truth answer instead of the question and the generated answer.

kcortinas and others added 9 commits May 23, 2024 13:39
@kcortinas
Update evaluation documentation (#571)
91817d2 
closes #304 

This PR updates the following:

- **README.md**: improve clarity around the metrics included in the
accelerator, description of evaluation reports and common pitfalls.
- **evaluation-metrics.md**: make clear how cosine similarity is
computed
- **eval.py**: update cosine description. Fix experiment name bug.
@dependabot
Bump azure-ai-ml from 1.15.0 to 1.16.0 (#569)
07a4c2c 
Bumps [azure-ai-ml](https://github.com/Azure/azure-sdk-for-python) from
1.15.0 to 1.16.0.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/Azure/azure-sdk-for-python/releases">azure-ai-ml's
releases</a>.</em></p>
<blockquote>
<h2>azure-ai-ml_1.16.0</h2>
<h2>1.16.0 (2024-05-14)</h2>
<h3>Features Added</h3>
<ul>
<li>Add experimental support for working with Promptflow evaluators:
<code>ml_client.evaluators</code>.</li>
<li>Many changes to the Connection entity class and its associated
operations.</li>
<li>Workspace Connection <code>list</code>, <code>get</code>, and
<code>create_or_update</code> operations now include an optional
<code>populate_secrets</code> input, which causes the operations to try
making a secondary call to fill in the returned connections' credential
info if possible. Only works with api key-based credentials for
now.</li>
<li>Many workspace connection subtypes added. The full list of
subclasses is now:
<ul>
<li><code>AzureBlobStoreConnection</code></li>
<li><code>AzureBlobStoreConnection</code></li>
<li><code>MicrosoftOneLakeConnection</code></li>
<li><code>AzureOpenAIConnection</code></li>
<li><code>AzureAIServicesConnection</code></li>
<li><code>AzureAISearchConnection</code></li>
<li><code>AzureContentSafetyConnection</code></li>
<li><code>AzureSpeechServicesConnection</code></li>
<li><code>APIKeyConnection</code></li>
<li><code>OpenAIConnection</code></li>
<li><code>SerpConnection</code></li>
<li><code>ServerlessConnection</code></li>
</ul>
</li>
<li>Many workspace connections only accept api keys or entra ids for
credentials. Since Entra IDs require not inputs, these have been
refactored to not required a full credential object. Instead they only
accept an api_key as a top-level input, and default to an entra
credential otherwise. Their YAML schemas have been similarly
altered.</li>
<li>Client-side credential-type validation added for some workspace
connection types.</li>
<li>Added new credential type:
<code>AadCredentialConfiguration</code></li>
<li>Renamed WorkspaceHub class as Hub.</li>
<li>Added Project entity class and YAML support.</li>
<li>Project and Hub operations supported by workspace operations.</li>
<li>workspace list operation supports type filtering.</li>
<li>Add support for Microsoft Entra token (<code>aad_token</code>) auth
in <code>invoke</code> and <code>get-credentials</code> operations.</li>
<li>Add experimental support for working with indexes:
<code>ml_client.indexes</code></li>
</ul>
<h3>Breaking Changes</h3>
<ul>
<li>Removed WorkspaceHubConfig entity, and renamed WorkspaceHub to
Hub.</li>
<li>workspace_hub input of Workspace class hidden, renamed to hub_id,
and re-surfaced in child class Project.</li>
<li>Removed Workspace Hub Operations from ML Client.</li>
</ul>
<h2>azure-ai-ml_1.16.0b1</h2>
<h2>1.16.0b1 (2024-05-07)</h2>
<h3>Features Added</h3>
<ul>
<li>Add experimental support for working with Promptflow evaluators:
<code>ml_client.evaluators</code>.</li>
<li>Many changes to the Connection entity class and its associated
operations.</li>
<li>Workspace Connection <code>list</code>, <code>get</code>, and
<code>create_or_update</code> operations now include an optional
<code>populate_secrets</code> input, which causes the operations to try
making a secondary call to fill in the returned connections' credential
info if possible. Only works with api key-based credentials for
now.</li>
<li>Many workspace connection subtypes added. The full list of
subclasses is now:
<ul>
<li><code>AzureBlobStoreConnection</code></li>
<li><code>AzureBlobStoreConnection</code></li>
<li><code>MicrosoftOneLakeConnection</code></li>
<li><code>AzureOpenAIConnection</code></li>
<li><code>AzureAIServicesConnection</code></li>
<li><code>AzureAISearchConnection</code></li>
</ul>
</li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/Azure/azure-sdk-for-python/commit/d12e171e7d9badebd6d6fc63954f0750055db282"><code>d12e171</code></a>
Use esrp release task that supports federated auth (<a
href="https://redirect.github.com/Azure/azure-sdk-for-python/issues/35523">#35523</a>)</li>
<li><a
href="https://github.com/Azure/azure-sdk-for-python/commit/5e4c03e9c62ae784cc81da827d2b5bc8321d7962"><code>5e4c03e</code></a>
add release date</li>
<li><a
href="https://github.com/Azure/azure-sdk-for-python/commit/1cd8a1be39609cea1c42d0ebe5270b290f8fad23"><code>1cd8a1b</code></a>
revert connection name (<a
href="https://redirect.github.com/Azure/azure-sdk-for-python/issues/35578">#35578</a>)
(<a
href="https://redirect.github.com/Azure/azure-sdk-for-python/issues/35584">#35584</a>)</li>
<li><a
href="https://github.com/Azure/azure-sdk-for-python/commit/44fa956eb922e1f947ea0fa16b83eb62df7b6b52"><code>44fa956</code></a>
Fix the CHANGELOG.md</li>
<li><a
href="https://github.com/Azure/azure-sdk-for-python/commit/0e53f3e3ddb79d1a105b4b716d879cf1f6d08d88"><code>0e53f3e</code></a>
Merge branch 'release/azure-ai-ml/1.16.0' of <a
href="https://github.com/Azure/azure-s">https://github.com/Azure/azure-s</a>...</li>
<li><a
href="https://github.com/Azure/azure-sdk-for-python/commit/c7f0c13ac2044c2e6c0cfa56753d4cd22b1f439f"><code>c7f0c13</code></a>
Update CHANGELOG.md</li>
<li><a
href="https://github.com/Azure/azure-sdk-for-python/commit/a87a5f5b565ea890e5bdd2a3760de1297404629b"><code>a87a5f5</code></a>
feat: Add kwargs passthrough (<a
href="https://redirect.github.com/Azure/azure-sdk-for-python/issues/35553">#35553</a>)
(<a
href="https://redirect.github.com/Azure/azure-sdk-for-python/issues/35558">#35558</a>)</li>
<li><a
href="https://github.com/Azure/azure-sdk-for-python/commit/406de875875f68020e531e21d7793ea1b0021159"><code>406de87</code></a>
Cherry-pick the View API changes to the release branch (<a
href="https://redirect.github.com/Azure/azure-sdk-for-python/issues/35552">#35552</a>)</li>
<li><a
href="https://github.com/Azure/azure-sdk-for-python/commit/a2a593e48d2b897aaa070f2118ef1d2f1f057442"><code>a2a593e</code></a>
More may apiview fixes (<a
href="https://redirect.github.com/Azure/azure-sdk-for-python/issues/35541">#35541</a>)
(<a
href="https://redirect.github.com/Azure/azure-sdk-for-python/issues/35544">#35544</a>)</li>
<li><a
href="https://github.com/Azure/azure-sdk-for-python/commit/2d4c71afdb2a4c706410c772a01c7c876631071c"><code>2d4c71a</code></a>
fix no attribute (<a
href="https://redirect.github.com/Azure/azure-sdk-for-python/issues/35511">#35511</a>)</li>
<li>Additional commits viewable in <a
href="https://github.com/Azure/azure-sdk-for-python/compare/azure-ai-ml_1.15.0...azure-ai-ml_1.16.0">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=azure-ai-ml&package-manager=pip&previous-version=1.15.0&new-version=1.16.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)


</details>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
@dependabot
Bump matplotlib from 3.8.4 to 3.9.0 (#567)
71e457e 
Bumps [matplotlib](https://github.com/matplotlib/matplotlib) from 3.8.4
to 3.9.0.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/matplotlib/matplotlib/releases">matplotlib's
releases</a>.</em></p>
<blockquote>
<h2>REL: 3.9.0</h2>
<p>Highlights of this release include:</p>
<ul>
<li>Plotting and Annotation improvements
<ul>
<li>Axes.inset_axes is no longer experimental</li>
<li>Legend support for Boxplot</li>
<li>Percent sign in pie labels auto-escaped with usetex=True</li>
<li>hatch parameter for stackplot</li>
<li>Add option to plot only one half of violin plot</li>
<li>axhline and axhspan on polar axes</li>
<li>Subplot titles can now be automatically aligned</li>
<li>axisartist can now be used together with standard Formatters</li>
<li>Toggle minorticks on Axis</li>
<li>StrMethodFormatter now respects axes.unicode_minus</li>
</ul>
</li>
<li>Figure, Axes, and Legend Layout
<ul>
<li>Subfigures now have controllable zorders</li>
<li>Getters for xmargin, ymargin and zmargin</li>
</ul>
</li>
<li>Mathtext improvements
<ul>
<li>mathtext documentation improvements</li>
<li>mathtext spacing corrections</li>
</ul>
</li>
<li>Widget Improvements
<ul>
<li>Check and Radio Button widgets support clearing</li>
</ul>
</li>
<li>3D plotting improvements
<ul>
<li>Setting 3D axis limits now set the limits exactly</li>
</ul>
</li>
<li>Other improvements
<ul>
<li>New BackendRegistry for plotting backends</li>
<li>Add widths, heights and angles setter to EllipseCollection</li>
<li>image.interpolation_stage rcParam</li>
<li>Arrow patch position is now modifiable</li>
<li>NonUniformImage now has mouseover support</li>
</ul>
</li>
</ul>
<h2>REL: v3.9.0rc2</h2>
<p>This is the second release candidate for the meso release 3.9.0.</p>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/matplotlib/matplotlib/commit/be56634d682bed257cb941369d8d3600635ddadf"><code>be56634</code></a>
REL: v3.9.0</li>
<li><a
href="https://github.com/matplotlib/matplotlib/commit/846ce8a4889b7cc4e755de5af0d041a742be7282"><code>846ce8a</code></a>
DOC: Finish documentation for 3.9.0</li>
<li><a
href="https://github.com/matplotlib/matplotlib/commit/8604f671d3a6179ef47dfbc3efef562a7448c384"><code>8604f67</code></a>
Merge pull request <a
href="https://redirect.github.com/matplotlib/matplotlib/issues/28206">#28206</a>
from meeseeksmachine/auto-backport-of-pr-28205-on-v...</li>
<li><a
href="https://github.com/matplotlib/matplotlib/commit/196c8db2074c9a3d91a9e144c21f7ef204905988"><code>196c8db</code></a>
TST: Followup corrections to <a
href="https://redirect.github.com/matplotlib/matplotlib/issues/28205">#28205</a></li>
<li><a
href="https://github.com/matplotlib/matplotlib/commit/d8f301644fa839c0f0ae2a438d42dbc55b4dc223"><code>d8f3016</code></a>
Backport PR <a
href="https://redirect.github.com/matplotlib/matplotlib/issues/28205">#28205</a>:
TST: Fix tests with older versions of ipython</li>
<li><a
href="https://github.com/matplotlib/matplotlib/commit/4db5ac9c5719d7681157b7c2979e582e35baa8bc"><code>4db5ac9</code></a>
Merge pull request <a
href="https://redirect.github.com/matplotlib/matplotlib/issues/28203">#28203</a>
from QuLogic/auto-backport-of-pr-28164-on-v3.9.x</li>
<li><a
href="https://github.com/matplotlib/matplotlib/commit/ef1a2db3f7c7124b5091dd9f58ea254731dbeafc"><code>ef1a2db</code></a>
Merge pull request <a
href="https://redirect.github.com/matplotlib/matplotlib/issues/28204">#28204</a>
from meeseeksmachine/auto-backport-of-pr-28195-on-v...</li>
<li><a
href="https://github.com/matplotlib/matplotlib/commit/e4384b80c3e470c98eb7a5d04699ffff2f3a70ba"><code>e4384b8</code></a>
Merge pull request <a
href="https://redirect.github.com/matplotlib/matplotlib/issues/28191">#28191</a>
from QuLogic/no-dev-theme</li>
<li><a
href="https://github.com/matplotlib/matplotlib/commit/3b65546e92166b9ed5b4e026f6bec08234a38eb5"><code>3b65546</code></a>
Backport PR <a
href="https://redirect.github.com/matplotlib/matplotlib/issues/28195">#28195</a>:
TST: Prepare for pytest 9</li>
<li><a
href="https://github.com/matplotlib/matplotlib/commit/1b526c3286becf1e7fdfe9291d437aa19c8f3bb5"><code>1b526c3</code></a>
Backport PR <a
href="https://redirect.github.com/matplotlib/matplotlib/issues/28164">#28164</a>:
CI: Ensure code coverage is always uploaded</li>
<li>Additional commits viewable in <a
href="https://github.com/matplotlib/matplotlib/compare/v3.8.4...v3.9.0">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=matplotlib&package-manager=pip&previous-version=3.8.4&new-version=3.9.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)


</details>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
@dependabot
Bump tiktoken from 0.6.0 to 0.7.0 (#565)
6ce5cfe 
Bumps [tiktoken](https://github.com/openai/tiktoken) from 0.6.0 to
0.7.0.
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/openai/tiktoken/blob/main/CHANGELOG.md">tiktoken's
changelog</a>.</em></p>
<blockquote>
<h2>[v0.7.0]</h2>
<ul>
<li>Support for <code>gpt-4o</code></li>
<li>Performance improvements</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/openai/tiktoken/commit/bfe00ad1bf59fac47513b45fe5173672dcbbcbb4"><code>bfe00ad</code></a>
Bump cibuildwheel</li>
<li><a
href="https://github.com/openai/tiktoken/commit/9d01e5670ff50eb74cdb96406c7f3d9add0ae2f8"><code>9d01e56</code></a>
Sync codebase</li>
<li><a
href="https://github.com/openai/tiktoken/commit/1b9faf2779855124f05174adf1383e53689ed94b"><code>1b9faf2</code></a>
Simplify byte_pair_merge (<a
href="https://redirect.github.com/openai/tiktoken/issues/255">#255</a>)</li>
<li><a
href="https://github.com/openai/tiktoken/commit/6defed51291184e3de4cb3ac8329994d0cc1d721"><code>6defed5</code></a>
Inline custom mapping function in _byte_pair_merge</li>
<li><a
href="https://github.com/openai/tiktoken/commit/b4c687ef3625e1737fba4f6643d7bedb9d6d2b6d"><code>b4c687e</code></a>
Avoid calling byte_pair_encode for existing tokens</li>
<li><a
href="https://github.com/openai/tiktoken/commit/6e4851a76be22a4f9cc428de3ea39d50ca767c60"><code>6e4851a</code></a>
Add finer grained gratitude</li>
<li><a
href="https://github.com/openai/tiktoken/commit/c2960c16c4a0b3a6b1a760eaac50b6a2c89b45fa"><code>c2960c1</code></a>
Store tokens in u32 instead of usize</li>
<li>See full diff in <a
href="https://github.com/openai/tiktoken/compare/0.6.0...0.7.0">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=tiktoken&package-manager=pip&previous-version=0.6.0&new-version=0.7.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)


</details>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
@dependabot @julia-meshcheryakova
Bump pytest from 8.2.0 to 8.2.1 (#562)
336240e 
Bumps [pytest](https://github.com/pytest-dev/pytest) from 8.2.0 to
8.2.1.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/pytest-dev/pytest/releases">pytest's
releases</a>.</em></p>
<blockquote>
<h2>8.2.1</h2>
<h1>pytest 8.2.1 (2024-05-19)</h1>
<h2>Improvements</h2>
<ul>
<li><a
href="https://redirect.github.com/pytest-dev/pytest/issues/12334">#12334</a>:
Support for Python 3.13 (beta1 at the time of writing).</li>
</ul>
<h2>Bug Fixes</h2>
<ul>
<li><a
href="https://redirect.github.com/pytest-dev/pytest/issues/12120">#12120</a>:
Fix [PermissionError]{.title-ref} crashes arising from directories which
are not selected on the command-line.</li>
<li><a
href="https://redirect.github.com/pytest-dev/pytest/issues/12191">#12191</a>:
Keyboard interrupts and system exits are now properly handled during the
test collection.</li>
<li><a
href="https://redirect.github.com/pytest-dev/pytest/issues/12300">#12300</a>:
Fixed handling of 'Function not implemented' error under squashfuse_ll,
which is a different way to say that the mountpoint is read-only.</li>
<li><a
href="https://redirect.github.com/pytest-dev/pytest/issues/12308">#12308</a>:
Fix a regression in pytest 8.2.0 where the permissions of
automatically-created <code>.pytest_cache</code> directories became
<code>rwx------</code> instead of the expected
<code>rwxr-xr-x</code>.</li>
</ul>
<h2>Trivial/Internal Changes</h2>
<ul>
<li><a
href="https://redirect.github.com/pytest-dev/pytest/issues/12333">#12333</a>:
pytest releases are now attested using the recent <a
href="https://github.blog/2024-05-02-introducing-artifact-attestations-now-in-public-beta/">Artifact
Attestation</a> support from GitHub, allowing users to verify the
provenance of pytest's sdist and wheel artifacts.</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/pytest-dev/pytest/commit/66ff8dffdf9eee9b3dd6686de34542c49ff80dcd"><code>66ff8df</code></a>
Prepare release version 8.2.1</li>
<li><a
href="https://github.com/pytest-dev/pytest/commit/3ffcfd122cf4674ac45f6233d9b50be6c49abeea"><code>3ffcfd1</code></a>
Merge pull request <a
href="https://redirect.github.com/pytest-dev/pytest/issues/12340">#12340</a>
from pytest-dev/backport-12334-to-8.2.x</li>
<li><a
href="https://github.com/pytest-dev/pytest/commit/0b28313b46a04de08bddc18896b3e61312a0c5b3"><code>0b28313</code></a>
[8.2.x] Add Python 3.13 (beta) support</li>
<li><a
href="https://github.com/pytest-dev/pytest/commit/f3dd93ad8d62eb0a260d3090f31be82aafbcff13"><code>f3dd93a</code></a>
[8.2.x] Attest package provenance (<a
href="https://redirect.github.com/pytest-dev/pytest/issues/12335">#12335</a>)</li>
<li><a
href="https://github.com/pytest-dev/pytest/commit/bb5a1257b0aafe5932377fa8e9fd92ab39418ac7"><code>bb5a125</code></a>
[8.2.x] Spelling (<a
href="https://redirect.github.com/pytest-dev/pytest/issues/12331">#12331</a>)</li>
<li><a
href="https://github.com/pytest-dev/pytest/commit/f179bf252fe2c1d0afce64b4b4bab4449e366e84"><code>f179bf2</code></a>
Merge pull request <a
href="https://redirect.github.com/pytest-dev/pytest/issues/12327">#12327</a>
from pytest-dev/backport-12325-to-8.2.x</li>
<li><a
href="https://github.com/pytest-dev/pytest/commit/2b671b5f9208650e8e42e07782d95477cc41f42a"><code>2b671b5</code></a>
[8.2.x] cacheprovider: fix <code>.pytest_cache</code> not being
world-readable</li>
<li><a
href="https://github.com/pytest-dev/pytest/commit/65ab7cb96c95f83e922f21bb4a8a44eda2b79707"><code>65ab7cb</code></a>
Merge pull request <a
href="https://redirect.github.com/pytest-dev/pytest/issues/12324">#12324</a>
from pytest-dev/backport-12320-to-8.2.x</li>
<li><a
href="https://github.com/pytest-dev/pytest/commit/4d5fb7d71ccc069e2f882bee0e4253eaf484d2a9"><code>4d5fb7d</code></a>
Merge pull request <a
href="https://redirect.github.com/pytest-dev/pytest/issues/12319">#12319</a>
from pytest-dev/backport-12311-to-8.2.x</li>
<li><a
href="https://github.com/pytest-dev/pytest/commit/cbe5996cc684b00397494d9590f3179de232c3ee"><code>cbe5996</code></a>
[8.2.x] changelog: document unittest 8.2 change as breaking</li>
<li>Additional commits viewable in <a
href="https://github.com/pytest-dev/pytest/compare/8.2.0...8.2.1">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=pytest&package-manager=pip&previous-version=8.2.0&new-version=8.2.1)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)


</details>

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Julia Meshcheryakova <juliame@microsoft.com>
@dependabot @julia-meshcheryakova
Bump langchain-community from 0.0.38 to 0.2.0 (#568)
7617f7d 
Bumps [langchain-community](https://github.com/langchain-ai/langchain)
from 0.0.38 to 0.2.0.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/langchain-ai/langchain/releases">langchain-community's
releases</a>.</em></p>
<blockquote>
<h2>langchain-community==0.2.0</h2>
<h1>Release langchain-community==0.2.0</h1>
<p>Package-specific release note generation coming soon.</p>
<h2>langchain-community==0.2.0rc1</h2>
<h1>Release langchain-community==0.2.0rc1</h1>
<p>Package-specific release note generation coming soon.</p>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/langchain-ai/langchain/commit/169f525cfb309f774624e34a7cffd82932db010b"><code>169f525</code></a>
community: release 0.2.0 (<a
href="https://redirect.github.com/langchain-ai/langchain/issues/21834">#21834</a>)</li>
<li><a
href="https://github.com/langchain-ai/langchain/commit/2656bfe94178feca7ff69419e678d1cdbbc373b8"><code>2656bfe</code></a>
docs: how to guide tool calling using prompts (<a
href="https://redirect.github.com/langchain-ai/langchain/issues/21827">#21827</a>)</li>
<li><a
href="https://github.com/langchain-ai/langchain/commit/e5046cbd7226d7043535f87dd40973d83879786f"><code>e5046cb</code></a>
langchain: release 0.2.0, fix min deps (<a
href="https://redirect.github.com/langchain-ai/langchain/issues/21833">#21833</a>)</li>
<li><a
href="https://github.com/langchain-ai/langchain/commit/1b555021f7b1b7d5610ea32bc806fb57715ae4f7"><code>1b55502</code></a>
text-splitters: release 0.2.0 (<a
href="https://redirect.github.com/langchain-ai/langchain/issues/21832">#21832</a>)</li>
<li><a
href="https://github.com/langchain-ai/langchain/commit/0ad8de5eb7746192773c02bdec4be4a81a448ab5"><code>0ad8de5</code></a>
langchain: release 0.2.0 (<a
href="https://redirect.github.com/langchain-ai/langchain/issues/21831">#21831</a>)</li>
<li><a
href="https://github.com/langchain-ai/langchain/commit/33dbad02fe970a940cde69268c4bec7af61ba8a0"><code>33dbad0</code></a>
docs: update how-to for built in tools and toolkits (<a
href="https://redirect.github.com/langchain-ai/langchain/issues/21828">#21828</a>)</li>
<li><a
href="https://github.com/langchain-ai/langchain/commit/23310626b39dc0eed734c2ffb211dbc84377d0f8"><code>2331062</code></a>
core: release 0.2.0 (<a
href="https://redirect.github.com/langchain-ai/langchain/issues/21829">#21829</a>)</li>
<li><a
href="https://github.com/langchain-ai/langchain/commit/e3f30b4cdebf750175e1764c969e6d56921c865d"><code>e3f30b4</code></a>
docs: clean up link to bing search (<a
href="https://redirect.github.com/langchain-ai/langchain/issues/21825">#21825</a>)</li>
<li><a
href="https://github.com/langchain-ai/langchain/commit/22d9aed5085bec81abcd56532030d25ab202b48e"><code>22d9aed</code></a>
docs: how to tools, merge built in tools and toolkits (<a
href="https://redirect.github.com/langchain-ai/langchain/issues/21824">#21824</a>)</li>
<li><a
href="https://github.com/langchain-ai/langchain/commit/c4508ca7ef63dda4b7d30482ed7ffa3aa5acac68"><code>c4508ca</code></a>
docs: <code>arXiv</code> references page (<a
href="https://redirect.github.com/langchain-ai/langchain/issues/21450">#21450</a>)</li>
<li>Additional commits viewable in <a
href="https://github.com/langchain-ai/langchain/compare/langchain-community==0.0.38...langchain-community==0.2.0">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=langchain-community&package-manager=pip&previous-version=0.0.38&new-version=0.2.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)


</details>

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Julia Meshcheryakova <juliame@microsoft.com>
@kcortinas
Merge branch 'development' into karina/update-eval-metrics
4b9a02b
@dependabot @julia-meshcheryakova
Bump lxml from 5.2.1 to 5.2.2 (#566)
ec87f2b 
Bumps [lxml](https://github.com/lxml/lxml) from 5.2.1 to 5.2.2.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/lxml/lxml/releases">lxml's
releases</a>.</em></p>
<blockquote>
<h2>lxml-5.2.2</h2>
<h1>5.2.2 (2024-05-12)</h1>
<h2>Bugs fixed</h2>
<ul>
<li>
<p>GH#417: The <code>test_feed_parser</code> test could fail if
<code>lxml_html_clean</code> was not installed.
It is now skipped in that case.</p>
</li>
<li>
<p>LP#2059910: The minimum CPU architecture for the Linux x86 binary
wheels was set back to
&quot;core2&quot;, without SSE 4.2.</p>
</li>
<li>
<p>If libxml2 uses iconv, the compile time version is available as
<code>etree.ICONV_COMPILED_VERSION</code>.</p>
</li>
</ul>
</blockquote>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/lxml/lxml/blob/master/CHANGES.txt">lxml's
changelog</a>.</em></p>
<blockquote>
<h1>5.2.2 (2024-??-??)</h1>
<h2>Bugs fixed</h2>
<ul>
<li>
<p>GH#417: The <code>test_feed_parser</code> test could fail if
<code>lxml_html_clean</code> was not installed.
It is now skipped in that case.</p>
</li>
<li>
<p>LP#2059910: The minimum CPU architecture for the Linux x86 binary
wheels was set back to
&quot;core2&quot;, without SSE 4.2.</p>
</li>
<li>
<p>If libxml2 uses iconv, the compile time version is available as
<code>etree.ICONV_COMPILED_VERSION</code>.</p>
</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/lxml/lxml/commit/8e4b14cbee88e1b04d7c49d29b2e593a37cdf80e"><code>8e4b14c</code></a>
Prepare release of lxml 5.2.2.</li>
<li><a
href="https://github.com/lxml/lxml/commit/ddaa6ed727995a3fa56e97de7d2ae995c734adff"><code>ddaa6ed</code></a>
Provide compile time version of libiconv as
&quot;etree.ICONV_COMPILED_VERSION&quot;.</li>
<li><a
href="https://github.com/lxml/lxml/commit/1ab00bd9437e8ce584f8efc0a54207b2d4d78601"><code>1ab00bd</code></a>
CI: Do not install lxml as dependency of
&quot;lxml_html_clean&quot;.</li>
<li><a
href="https://github.com/lxml/lxml/commit/b3250545b542877788213ba72c8c4d02f43cf7e0"><code>b325054</code></a>
CI: Stop building LTO wheels to avoid messing with the library build
cache.</li>
<li><a
href="https://github.com/lxml/lxml/commit/81624c28e87cd4af6ff628af1410410c8a7b07ef"><code>81624c2</code></a>
CI: Exclude now-unsupported Python versions 3.6/7 from macOS jobs.</li>
<li><a
href="https://github.com/lxml/lxml/commit/069fa36dbde340b2ea2f6df31d6a90b0af7b50b7"><code>069fa36</code></a>
Fix typo.</li>
<li><a
href="https://github.com/lxml/lxml/commit/e0df2bace67e4bc914e7fd565446d979fa83607b"><code>e0df2ba</code></a>
Fix typo.</li>
<li><a
href="https://github.com/lxml/lxml/commit/df55eaf8a1883aaa2337284ad6800d547ffab3d4"><code>df55eaf</code></a>
Update changelog.</li>
<li><a
href="https://github.com/lxml/lxml/commit/f3e77fabd124062b04932a69c3c7e457c8bb9039"><code>f3e77fa</code></a>
Remove dependency on SSE4 instructions, reverting back to
&quot;core2&quot; as a target...</li>
<li><a
href="https://github.com/lxml/lxml/commit/a22e83caf637c8c68030da0278591df60c994126"><code>a22e83c</code></a>
Update changelog.</li>
<li>Additional commits viewable in <a
href="https://github.com/lxml/lxml/compare/lxml-5.2.1...lxml-5.2.2">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=lxml&package-manager=pip&previous-version=5.2.1&new-version=5.2.2)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)


</details>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Julia Meshcheryakova <juliame@microsoft.com>
@kcortinas
Merge branch 'development' into karina/update-eval-metrics
fda5e9d
joanassantos
joanassantos approved these changes May 23, 2024
View reviewed changes
Copy link
Collaborator

@joanassantos joanassantos left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @kcortinas for spotting and fixing the context error. Looks good to me!

guybartal
guybartal approved these changes May 24, 2024
View reviewed changes
Copy link
Collaborator

@guybartal guybartal left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great, thank you for taking this @kcortinas ,
You may merge this to prerelease.

We'll find someone from the team to take
#573

@kcortinas kcortinas changed the base branch from development to prerelease May 24, 2024 07:54
@kcortinas kcortinas force-pushed the karina/update-eval-metrics branch from 12a813b to ec6e2f0 Compare May 24, 2024 10:32
@kcortinas kcortinas merged commit dd3a933 into prerelease May 24, 2024
3 checks passed
@kcortinas kcortinas deleted the karina/update-eval-metrics branch May 24, 2024 11:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@joanassantos joanassantos joanassantos approved these changes

@guybartal guybartal guybartal approved these changes

@ritesh-modi ritesh-modi Awaiting requested review from ritesh-modi

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@kcortinas @guybartal @joanassantos