FSCrawler override my own index mapping fscrawler_mapping_content after started

[bobjiang1988]

The document said that If you want to define your own index settings and mapping to set analyzers for example, you can update the needed component template before starting the FSCrawler.

But, _component_template/fscrawler_mapping_content will be override after FSCrawler started.

Originally posted by @bobjiang1988 in #469 (comment)

[dadoonet]

I looked at this today and actually this is documented (see https://fscrawler.readthedocs.io/en/latest/admin/fs/elasticsearch.html#mappings):

You can stop FSCrawler creating/updating the index templates for you by setting push_templates to false:

name: "test"
elasticsearch:
  push_templates: false

Could you try this and confirm that this is fixing your issue?

[bobjiang1988]

push_templates will stop to push all _component_templates and _index_templates. Could it only stop to push the specified template, like _component_template/fscrawler_mapping_content?

[dadoonet]

No. But here's my take on this.

I think I should add a CLI option like --setup which would only send the templates for a given job but not start it. May be --loop 0 would do this.
Then you can overwrite the template you want and start fscrawler with push_templates to false.

Would that work for you in the meantime? I would like to implement a better user experience.

[bobjiang1988]

I think if user could put the customer templates in a specified directory would be a better user experience. Then FSCrawler should read the templates from both user's directory and the default directory. Is it?

[aaltonenp]

I used to edit the default file at ~./.fscrawler/_default/7/_settings.json but that seems to be ignored now. I only need one small modification to make filenames case-insensitive. It's a bit cumbersome to push all templates beforehand after such a small change.

The old way where local modifications would be combined with defaults would seem best to me.

[dadoonet]

@aaltonenp Would the solution posted here would work for you?

Basically, create an index template and disable fscrawler index templates?

[aaltonenp]

@dadoonet It might once I figure out how to add the normalizer. I let fscrawler run once to create the templates and then try to modify them.

In the _settings.json I used to define the normalizer where the analysis section is. That is now in the fscrawler_mapping_path template. That can be modified but then using that normalizer in fscrawler_mapping_file template causes a "normalizer not found" error.

I don't know much about Index & Component Templates and how they draw the settings together. Do I need to define the normalizer in the fscrawler_mapping_file a second time?

[dadoonet]

You need to define the normalizer in the same component template as you would use it. So probably indeed in fscrawler_mapping_file and in fscrawler_mapping_path if you are also using it there.

[aaltonenp]

Ok thanks. Although looks like I can get away using the built-in lowercase normalizer so I don't have to define a custom one at all.

[gsgleason]

I'm having a similar problem with the current docker build.

I use an ingest pipeline with a grok processor to match patterns in the filenames and use those to create custom fields. Some of those fields are date types with custom date formats.

I cannot figure out a streamlined way to do this that isn't totally hackish.

Perhaps have the ability to create mappings when the index is initially created would be helpful.