[Examples][Store] Implement indexing pipeline #465
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
carsonbot
changed the title
OskarStark
force-pushed
the
feature/document-indexing-pipeline
branch
from
643b560 to
d3a3a15
Compare
OskarStark
added
Store
carsonbot
changed the title
OskarStark
changed the title
OskarStark
force-pushed
the
feature/document-indexing-pipeline
branch
from
0827740 to
bbc3145
Compare
OskarStark
added a commit
that referenced
this pull request
Sep 5, 2025
…ark) This PR was squashed before being merged into the main branch. Discussion ---------- [AI Bundle][Demo] Make vectorizers configurable | Q | A | ------------- | --- | Bug fix? | no | New feature? | yes | Docs? | no | Issues | Refs #465 | License | MIT Add support for configuring vectorizers via ai.yaml configuration, allowing reuse across multiple indexers and centralized vectorizer management. Commits ------- 7acf871 [AI Bundle][Demo] Make vectorizers configurable
OskarStark
force-pushed
the
feature/document-indexing-pipeline
branch
from
53486dc to
a6b0ace
Compare
OskarStark
force-pushed
the
feature/document-indexing-pipeline
branch
from
a6b0ace to
242501e
Compare
chr-hertel
reviewed
Sep 6, 2025
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
OskarStark
changed the title
This comment was marked as outdated.
This comment was marked as outdated.
OskarStark
force-pushed
the
feature/document-indexing-pipeline
branch
from
0b5dc7e to
15236b3
Compare
OskarStark
added a commit
that referenced
this pull request
Sep 8, 2025
…arStark) This PR was merged into the main branch. Discussion ---------- [Store] Add with `TextDocument::withContent` method | Q | A | ------------- | --- | Bug fix? | no | New feature? | yes | Docs? | no | Issues | Helpful for #465 | License | MIT Commits ------- 4a3014f [Store] Add withContent method to TextDocument with test
OskarStark
force-pushed
the
feature/document-indexing-pipeline
branch
5 times, most recently
from
6d2dcf7 to
a49521a
Compare
OskarStark
force-pushed
the
feature/document-indexing-pipeline
branch
from
a49521a to
6bfb1ed
Compare
OskarStark
commented
Sep 8, 2025
OskarStark
commented
Sep 8, 2025
OskarStark
commented
Sep 8, 2025
OskarStark
force-pushed
the
feature/document-indexing-pipeline
branch
from
382151b to
b668771
Compare
chr-hertel
reviewed
Sep 8, 2025
| @@ -32,7 +34,7 @@ protected function execute(InputInterface $input, OutputInterface $output): int | |||
| $io = new SymfonyStyle($input, $output); | |||
| $io->title('Loading RSS of Symfony blog as embeddings into ChromaDB'); | |||
|
|
|||
| $this->embedder->embedBlog(); | |||
| $this->indexer->index('https://feeds.feedburner.com/symfony/blog'); | |||
There was a problem hiding this comment.
would be the source - as a next step
chr-hertel
reviewed
Sep 8, 2025
src/store/src/IndexerInterface.php
Outdated
| */ | ||
| public function index(TextDocument|iterable $documents, int $chunkSize = 50): void; | ||
| public function index(null|string|array $source = null, array $options = []): void; |
There was a problem hiding this comment.
would also argue for removing null here
OskarStark
commented
Sep 8, 2025
src/store/src/Indexer.php
Outdated
| if ($documents instanceof TextDocument) { | ||
| $documents = [$documents]; | ||
| // Prevent conflicting sources | ||
| if (null !== $source && null !== $this->source) { |
There was a problem hiding this comment.
this way we have both, we can set a default on config level, but cannot overwrite then, as one wants to overwrite, another one wants to merge. You can define/configure an indexer now with source = null, in this case you can provide it via the method.
OskarStark
force-pushed
the
feature/document-indexing-pipeline
branch
2 times, most recently
from
ef78629 to
659778d
Compare
chr-hertel
approved these changes
Sep 8, 2025
carsonbot
changed the title
chr-hertel
force-pushed
the
feature/document-indexing-pipeline
branch
from
d77a852 to
29349c1
Compare
|
Thank you @OskarStark. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Needs
./runner indexerbin/console app:blog:embed(removed)bin/console app:blog:querybin/console ai:store:index blog