Do not wrap filepath in Path to fix indexing markdown files on Windows #993
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Issue - Path with / are converted to \\ on Windows using the Path operator. - The markdown to entries method for some reason was doing this. This would store the file paths in DB entry differently than the file to entries map. Resulting in a KeyError when trying to look up the entry file path from file_to_text_map in the text_to_entries:update_embeddings() function. Fix - Removing the unnecessary OS dependendent Path normalization in markdown_to_entries should keep the file path storage consistent across file_to_text_map var, FileObjectAdaptor, Entry DB tables on Windows for Markdown files as well This issue would only affect users hosting Khoj server on Windows and attempting to index markdown files. Resolves #984
- Add type hints to improve maintainability of stabilzed indexing code - It shouldn't be necessary to wrap string variables in an f-string This change aims to improve code quality. It should not affect functionality.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Issue
Pathoperator.markdown_to_entriesmodule was trying to normalize file paths withPathfor some reason.This would store the file paths in DB Entry differently than the file to entries map if Khoj ran on Windows.
That'd result in a KeyError when trying to look up the entry file path from
file_to_text_mapin thetext_to_entries:update_embeddings()function.Fix
markdown_to_entriesshould keep the file path storage consistent acrossfile_to_text_mapvar,FileObjectAdaptor,EntryDB tables on Windows for Markdown files as well.This issue will affect users hosting Khoj server on Windows and attempting to index markdown files.
Resolves #984