Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

"Limit of total fields [1000] in index [github2-pull_raw] has been exceeded" error for "github2:pull" #587

Open
ncsibra-lab49 opened this issue Apr 17, 2023 · 0 comments

Comments

@ncsibra-lab49
Copy link

I'm getting the error mentioned in the title when using "github2:pull" backend.
I'm using the "grimoirelab/grimoirelab:latest" docker image.
The projects.json file contains about 4000 repos and the backend configuration in setup.cfg looks like this:

[github2]
api-token = [...]

[github2:pull]
raw_index = github2-pull_raw
enriched_index = github2-pull_enriched
sleep-for-rate = true
category = pull_request
no-archive = true
studies = [enrich_geolocation:user, enrich_geolocation:assignee, enrich_feelings]

I changed the limit to 2000 with this:

PUT github2-pull_raw/_settings
{
  "index.mapping.total_fields.limit": 2000
}

Elasticsearch documentation mentions that maybe not a good idea: https://www.elastic.co/guide/en/elasticsearch/reference/6.8/mapping.html#mapping-limit-settings

Found a pull request with the same issue, where the structure was changed to avoid this error:
chaoss/grimoirelab-perceval#742

Something similar should be done for "github2" backend or changing the limit is the right solution?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant