Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Python: Milvus: consider using static rather than dynamic fields for the Semantic Kernel metadata #2901

Closed
roji opened this issue Sep 20, 2023 · 3 comments
Closed

Python: Milvus: consider using static rather than dynamic fields for the Semantic Kernel metadata #2901

roji opened this issue Sep 20, 2023 · 3 comments
Labels
memory connector python Pull requests for the Python Semantic Kernel

Comments

@roji
Copy link
Member

roji commented Sep 20, 2023

In #1941, @filip-halt added a Python memory store implementation for the Milvus vector database; #2694 (pending) does the same for .NET. A Semantic Kernel memory store needs to support storing various metadata fields alongside the vector data: a timestamp, an optional external source name, additional metadata, and so on. The Python connector implements these metadata fields as Milvus dynamic schema fields; this means that the fields aren't defined up-front in the schema, but are added dynamically (and stored behind the scenes as JSON on the special $meta field).

Since the SK metadata fields are known in advance and do not change, I think it should be considered to switch representing them as regular, static schema fields instead. Dynamic fields aren't as well-supported in various ways; and importantly, AFAIK it's not possible to define an index over them (so efficiently filtering by metadata isn't possible).

As of now the .NET PR (#2694) was done via dynamic schema to align with the Python connector; but I believe both connectors should be changed to use static fields.

/cc @filip-halt @yhmo @weianweigan @stephentoub @awharrison-28 @ajcvickers
@shawncal shawncal added .NET Issue or Pull requests regarding .NET code python Pull requests for the Python Semantic Kernel triage labels Sep 20, 2023
@github-actions github-actions bot changed the title Milvus: consider using static rather than dynamic fields for the Semantic Kernel metadata .Net: Milvus: consider using static rather than dynamic fields for the Semantic Kernel metadata Sep 20, 2023
@github-actions github-actions bot changed the title .Net: Milvus: consider using static rather than dynamic fields for the Semantic Kernel metadata Python: Milvus: consider using static rather than dynamic fields for the Semantic Kernel metadata Sep 20, 2023
@nacharya1 nacharya1 removed the triage label Sep 21, 2023
@nacharya1 nacharya1 moved this to Backlog - New features in Semantic Kernel Sep 21, 2023
@nacharya1
Copy link
Contributor

@lemillermicrosoft will update as he working on this already.

@nacharya1 nacharya1 moved this from Backlog - New features to Sprint: In Progress in Semantic Kernel Sep 21, 2023
@nacharya1 nacharya1 moved this from Sprint: In Progress to Sprint: In Review in Semantic Kernel Sep 21, 2023
@nacharya1 nacharya1 added this to the R3: Cycle 3 milestone Sep 21, 2023
@lemillermicrosoft lemillermicrosoft moved this from Sprint: In Review to Backlog - New features in Semantic Kernel Sep 21, 2023
@lemillermicrosoft lemillermicrosoft removed this from the R3: Cycle 3 milestone Sep 21, 2023
@lemillermicrosoft
Copy link
Member

For now this is staying in the backlog, feedback from @filip-halt would be appreciaed.

@lemillermicrosoft lemillermicrosoft removed their assignment Sep 28, 2023
github-merge-queue bot pushed a commit that referenced this issue Oct 4, 2023
@dependabot @markwallace-microsoft
Python: Bump urllib3 from 1.26.14 to 1.26.17 in /samples/apps/hugging…
72ab743 
…-face-http-server (#3056)

Bumps [urllib3](https://github.com/urllib3/urllib3) from 1.26.14 to
1.26.17.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/urllib3/urllib3/releases">urllib3's
releases</a>.</em></p>
<blockquote>
<h2>1.26.17</h2>
<ul>
<li>Added the <code>Cookie</code> header to the list of headers to strip
from requests when redirecting to a different host. As before, different
headers can be set via <code>Retry.remove_headers_on_redirect</code>.
(GHSA-v845-jxx5-vc9f)</li>
</ul>
<h2>1.26.16</h2>
<ul>
<li>Fixed thread-safety issue where accessing a <code>PoolManager</code>
with many distinct origins would cause connection pools to be closed
while requests are in progress (<a
href="https://redirect.github.com/urllib3/urllib3/issues/2954">#2954</a>)</li>
</ul>
<h2>1.26.15</h2>
<ul>
<li>Fix socket timeout value when HTTPConnection is reused (<a
href="https://redirect.github.com/urllib3/urllib3/issues/2645">urllib3/urllib3#2645</a>)</li>
<li>Remove &quot;!&quot; character from the unreserved characters in
IPv6 Zone ID parsing (<a
href="https://redirect.github.com/urllib3/urllib3/issues/2899">urllib3/urllib3#2899</a>)</li>
<li>Fix IDNA handling of 'x80' byte (<a
href="https://redirect.github.com/urllib3/urllib3/issues/2901">urllib3/urllib3#2901</a>)</li>
</ul>
</blockquote>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/urllib3/urllib3/blob/main/CHANGES.rst">urllib3's
changelog</a>.</em></p>
<blockquote>
<h1>1.26.17 (2023-10-02)</h1>
<ul>
<li>Added the <code>Cookie</code> header to the list of headers to strip
from requests when redirecting to a different host. As before, different
headers can be set via <code>Retry.remove_headers_on_redirect</code>.
(<code>[#3139](urllib3/urllib3#3139)
&lt;https://github.com/urllib3/urllib3/pull/3139&gt;</code>_)</li>
</ul>
<h1>1.26.16 (2023-05-23)</h1>
<ul>
<li>Fixed thread-safety issue where accessing a <code>PoolManager</code>
with many distinct origins
would cause connection pools to be closed while requests are in progress
(<code>[#2954](urllib3/urllib3#2954)
&lt;https://github.com/urllib3/urllib3/pull/2954&gt;</code>_)</li>
</ul>
<h1>1.26.15 (2023-03-10)</h1>
<ul>
<li>Fix socket timeout value when <code>HTTPConnection</code> is reused
(<code>[#2645](urllib3/urllib3#2645)
&lt;https://github.com/urllib3/urllib3/issues/2645&gt;</code>__)</li>
<li>Remove &quot;!&quot; character from the unreserved characters in
IPv6 Zone ID parsing
(<code>[#2899](urllib3/urllib3#2899)
&lt;https://github.com/urllib3/urllib3/issues/2899&gt;</code>__)</li>
<li>Fix IDNA handling of '\x80' byte
(<code>[#2901](urllib3/urllib3#2901)
&lt;https://github.com/urllib3/urllib3/issues/2901&gt;</code>__)</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/urllib3/urllib3/commit/c9016bf464751a02b7e46f8b86504f47d4238784"><code>c9016bf</code></a>
Release 1.26.17</li>
<li><a
href="https://github.com/urllib3/urllib3/commit/01220354d389cd05474713f8c982d05c9b17aafb"><code>0122035</code></a>
Backport GHSA-v845-jxx5-vc9f (<a
href="https://redirect.github.com/urllib3/urllib3/issues/3139">#3139</a>)</li>
<li><a
href="https://github.com/urllib3/urllib3/commit/e63989f97d206e839ab9170c8a76e3e097cc60e8"><code>e63989f</code></a>
Fix installing <code>brotli</code> extra on Python 2.7</li>
<li><a
href="https://github.com/urllib3/urllib3/commit/2e7a24d08713a0131f0b3c7197889466d645cc49"><code>2e7a24d</code></a>
[1.26] Configure OS for RTD to fix building docs</li>
<li><a
href="https://github.com/urllib3/urllib3/commit/57181d6ea910ac7cb2ff83345d9e5e0eb816a0d0"><code>57181d6</code></a>
[1.26] Improve error message when calling urllib3.request() (<a
href="https://redirect.github.com/urllib3/urllib3/issues/3058">#3058</a>)</li>
<li><a
href="https://github.com/urllib3/urllib3/commit/3c0148048a523325819377b23fc67f8d46afc3aa"><code>3c01480</code></a>
[1.26] Run coverage even with failed jobs</li>
<li><a
href="https://github.com/urllib3/urllib3/commit/d94029b7e2193ff47b627906a70e06377a09aae8"><code>d94029b</code></a>
Release 1.26.16</li>
<li><a
href="https://github.com/urllib3/urllib3/commit/18e92145e9cddbabdf51c98f54202aa37fd5d4c8"><code>18e9214</code></a>
Use trusted publishing for PyPI</li>
<li><a
href="https://github.com/urllib3/urllib3/commit/d25cf83bbae850a290fe34ed1610ae55c0558b36"><code>d25cf83</code></a>
[1.26] Fix invalid test_ssl_failure_midway_through_conn</li>
<li><a
href="https://github.com/urllib3/urllib3/commit/25cca389496b86ee809c21e5b641aeaa74809263"><code>25cca38</code></a>
[1.26] Fix test_ssl_object_attributes</li>
<li>Additional commits viewable in <a
href="https://github.com/urllib3/urllib3/compare/1.26.14...1.26.17">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=urllib3&package-manager=pip&previous-version=1.26.14&new-version=1.26.17)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)
You can disable automated security fix PRs for this repo from the
[Security Alerts
page](https://github.com/microsoft/semantic-kernel/network/alerts).

</details>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Mark Wallace <127216156+markwallace-microsoft@users.noreply.github.com>
SOE-YoungS pushed a commit to SOE-YoungS/semantic-kernel that referenced this issue Nov 1, 2023
@dependabot @markwallace-microsoft @SOE-YoungS
Python: Bump urllib3 from 1.26.14 to 1.26.17 in /samples/apps/hugging…
40a87ac 
…-face-http-server (microsoft#3056)

Bumps [urllib3](https://github.com/urllib3/urllib3) from 1.26.14 to
1.26.17.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/urllib3/urllib3/releases">urllib3's
releases</a>.</em></p>
<blockquote>
<h2>1.26.17</h2>
<ul>
<li>Added the <code>Cookie</code> header to the list of headers to strip
from requests when redirecting to a different host. As before, different
headers can be set via <code>Retry.remove_headers_on_redirect</code>.
(GHSA-v845-jxx5-vc9f)</li>
</ul>
<h2>1.26.16</h2>
<ul>
<li>Fixed thread-safety issue where accessing a <code>PoolManager</code>
with many distinct origins would cause connection pools to be closed
while requests are in progress (<a
href="https://redirect.github.com/urllib3/urllib3/issues/2954">#2954</a>)</li>
</ul>
<h2>1.26.15</h2>
<ul>
<li>Fix socket timeout value when HTTPConnection is reused (<a
href="https://redirect.github.com/urllib3/urllib3/issues/2645">urllib3/urllib3#2645</a>)</li>
<li>Remove &quot;!&quot; character from the unreserved characters in
IPv6 Zone ID parsing (<a
href="https://redirect.github.com/urllib3/urllib3/issues/2899">urllib3/urllib3#2899</a>)</li>
<li>Fix IDNA handling of 'x80' byte (<a
href="https://redirect.github.com/urllib3/urllib3/issues/2901">urllib3/urllib3#2901</a>)</li>
</ul>
</blockquote>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/urllib3/urllib3/blob/main/CHANGES.rst">urllib3's
changelog</a>.</em></p>
<blockquote>
<h1>1.26.17 (2023-10-02)</h1>
<ul>
<li>Added the <code>Cookie</code> header to the list of headers to strip
from requests when redirecting to a different host. As before, different
headers can be set via <code>Retry.remove_headers_on_redirect</code>.
(<code>[microsoft#3139](urllib3/urllib3#3139)
&lt;https://github.com/urllib3/urllib3/pull/3139&gt;</code>_)</li>
</ul>
<h1>1.26.16 (2023-05-23)</h1>
<ul>
<li>Fixed thread-safety issue where accessing a <code>PoolManager</code>
with many distinct origins
would cause connection pools to be closed while requests are in progress
(<code>[microsoft#2954](urllib3/urllib3#2954)
&lt;https://github.com/urllib3/urllib3/pull/2954&gt;</code>_)</li>
</ul>
<h1>1.26.15 (2023-03-10)</h1>
<ul>
<li>Fix socket timeout value when <code>HTTPConnection</code> is reused
(<code>[microsoft#2645](urllib3/urllib3#2645)
&lt;https://github.com/urllib3/urllib3/issues/2645&gt;</code>__)</li>
<li>Remove &quot;!&quot; character from the unreserved characters in
IPv6 Zone ID parsing
(<code>[microsoft#2899](urllib3/urllib3#2899)
&lt;https://github.com/urllib3/urllib3/issues/2899&gt;</code>__)</li>
<li>Fix IDNA handling of '\x80' byte
(<code>[microsoft#2901](urllib3/urllib3#2901)
&lt;https://github.com/urllib3/urllib3/issues/2901&gt;</code>__)</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/urllib3/urllib3/commit/c9016bf464751a02b7e46f8b86504f47d4238784"><code>c9016bf</code></a>
Release 1.26.17</li>
<li><a
href="https://github.com/urllib3/urllib3/commit/01220354d389cd05474713f8c982d05c9b17aafb"><code>0122035</code></a>
Backport GHSA-v845-jxx5-vc9f (<a
href="https://redirect.github.com/urllib3/urllib3/issues/3139">#3139</a>)</li>
<li><a
href="https://github.com/urllib3/urllib3/commit/e63989f97d206e839ab9170c8a76e3e097cc60e8"><code>e63989f</code></a>
Fix installing <code>brotli</code> extra on Python 2.7</li>
<li><a
href="https://github.com/urllib3/urllib3/commit/2e7a24d08713a0131f0b3c7197889466d645cc49"><code>2e7a24d</code></a>
[1.26] Configure OS for RTD to fix building docs</li>
<li><a
href="https://github.com/urllib3/urllib3/commit/57181d6ea910ac7cb2ff83345d9e5e0eb816a0d0"><code>57181d6</code></a>
[1.26] Improve error message when calling urllib3.request() (<a
href="https://redirect.github.com/urllib3/urllib3/issues/3058">#3058</a>)</li>
<li><a
href="https://github.com/urllib3/urllib3/commit/3c0148048a523325819377b23fc67f8d46afc3aa"><code>3c01480</code></a>
[1.26] Run coverage even with failed jobs</li>
<li><a
href="https://github.com/urllib3/urllib3/commit/d94029b7e2193ff47b627906a70e06377a09aae8"><code>d94029b</code></a>
Release 1.26.16</li>
<li><a
href="https://github.com/urllib3/urllib3/commit/18e92145e9cddbabdf51c98f54202aa37fd5d4c8"><code>18e9214</code></a>
Use trusted publishing for PyPI</li>
<li><a
href="https://github.com/urllib3/urllib3/commit/d25cf83bbae850a290fe34ed1610ae55c0558b36"><code>d25cf83</code></a>
[1.26] Fix invalid test_ssl_failure_midway_through_conn</li>
<li><a
href="https://github.com/urllib3/urllib3/commit/25cca389496b86ee809c21e5b641aeaa74809263"><code>25cca38</code></a>
[1.26] Fix test_ssl_object_attributes</li>
<li>Additional commits viewable in <a
href="https://github.com/urllib3/urllib3/compare/1.26.14...1.26.17">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=urllib3&package-manager=pip&previous-version=1.26.14&new-version=1.26.17)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)
You can disable automated security fix PRs for this repo from the
[Security Alerts
page](https://github.com/microsoft/semantic-kernel/network/alerts).

</details>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Mark Wallace <127216156+markwallace-microsoft@users.noreply.github.com>
@matthewbolanos matthewbolanos moved this from Backlog - New features to Backlog in Semantic Kernel Nov 28, 2023
@matthewbolanos matthewbolanos removed the .NET Issue or Pull requests regarding .NET code label Jan 2, 2024
@eavanvalkenburg
Copy link
Member

This was actually done.

@github-project-automation github-project-automation bot moved this from Backlog to Sprint: Done in Semantic Kernel Mar 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
memory connector python Pull requests for the Python Semantic Kernel
Projects
Semantic Kernel
Archived in project
Milestone
No milestone
Development

No branches or pull requests
7 participants
@matthewbolanos @roji @eavanvalkenburg @lemillermicrosoft @shawncal @nacharya1 @awharrison-28