Add additional tags when crawling files

[4islam]

Is your feature request related to a problem? Please describe.

I'm always frustrated when I can't categorize or add context to the files being indexed by FSCrawler. The current plugin lacks the ability to add custom tags during the crawling process, which limits the flexibility and searchability of the indexed files.

Describe the solution you'd like

I would like to have the capability to specify additional tags when crawling files with FSCrawler. These tags should be included in the metadata of each indexed file, allowing for better categorization, context, and enhanced search capabilities. This feature should be configurable through the fscrawler.yml file.

Describe alternatives you've considered

I've considered manually adding tags to the files after they are indexed, but this approach is time-consuming and inefficient. Another alternative is to use a different tool that supports tagging, but I prefer to continue using FSCrawler due to its other features and integration with Elasticsearch.

---

Feature Request: Add Additional Tags When Crawling Files

Summary:
Enhance the FSCrawler Elasticsearch plugin by adding the capability to include additional tags when crawling files. This feature will allow users to specify custom tags that can be associated with the files being indexed, providing more flexibility and improving searchability.

Description:
The current FSCrawler plugin for Elasticsearch efficiently indexes files and extracts metadata. However, it lacks the ability to add custom tags during the crawling process. By introducing a feature that allows users to specify additional tags, we can significantly enhance the plugin's functionality. These tags can be used to categorize files, add context, and improve the overall search experience.

Use Cases:

1. Categorization: Users can categorize files based on project, department, or any other custom criteria, making it easier to organize and retrieve relevant documents.
2. Contextual Information: Adding tags that provide context, such as "confidential," "urgent," or "archived," can help users quickly identify the nature of the files. In addition, when you are crawling folders, a meta.inf file can include metadata for each file's content. In the case of a book folder with pages to be indexed in it, a meta.inf file describes information like the book name, its author, its ISBN etc. which can be added to each indexed page as its metadata.
3. Enhanced Search: Custom tags can be used as search filters, allowing users to perform more precise and targeted searches within the indexed files.

Implementation:

• Configuration: Introduce a new configuration option in the fscrawler.yml file where users can define custom tags.
• Tagging Mechanism: Modify the crawling process to include the specified tags in the metadata of each indexed file.
• Search Integration: Ensure that the custom tags are indexed and searchable within Elasticsearch, allowing users to filter search results based on these tags.

Benefits:

• Improved file organization and retrieval.
• Enhanced search capabilities with custom filters.
• Greater flexibility in managing and categorizing indexed files.

Conclusion:
Adding the ability to specify additional tags when crawling files will greatly enhance the FSCrawler plugin's usability and functionality. This feature will provide users with more control over their indexed data and improve the overall search experience.

---

Let me know if you need any more help!

[dadoonet]

It's similar to #884 right? Note that FSCrawler supports tags when using the REST API.

[4islam]

Yes. The feature is available via REST API but it would be really valuable while crawling files.

[dadoonet]

Could you describe and example of fscrawler.yml as you would like to see it?

[4islam]

Following the example of REST API (see https://fscrawler.readthedocs.io/en/latest/admin/fs/rest.html#additional-tags), we should be able to not only specify additional global tags, but path to the file with tags for each folder branch.

In my case, where I am indexing a library of books under /library. The folder structure looks like this:

library
├── .meta.yml                        #potentially to define tags applicable to all books (good option to allow nesting)
├── book1
│   ├── .meta.yml
│   └── pages
│       ├── page1.txt
│       ├── page2.txt
│       └── page3.txt
├── book2
│   ├── .meta.yml
│   └── pages
│       ├── page1.txt
│       ├── page2.txt
│       └── page3.txt
└── book3
    ├── .meta.yml
    └── pages
        ├── page1.txt
        ├── page2.txt
        └── page3.txt

So fscrawler.yml file could look like this:

name: "my_fscrawler_job"
fs:
  url: "/library"
  update_rate: "15m"
  tags:                                           # Specify global tags
    - project: "ProjectName"
    - department: "DepartmentName"
  meta_file:
    path: "/.meta.yml"                            # Specify the path to the meta.yml file(s)
elasticsearch:
  nodes:
    - url: "http://localhost:9200"
  index: "my_fscrawler_index"
  bulk_size: 100
  flush_interval: "5s"

So if any folder had this .meta.yml file, fscrawler should fetch tag information from there for each folder and index it alongside each page from that folder. The file for each can look like:

author: "John Doe"
language: "English"
otherlanguages:
  - Arabic
  - Persian
isbn: "978-3-16-148410-0"
tags:
  - "fiction"
  - "adventure"
  - "bestseller"

[dadoonet]

I started a first implementation. It's not exactly what you asked for but it's a start at least.

Have a look at #2017 for the current status of the feature.

[4islam]

Looks great. Thanks a lot. Appreciated.