Running a terms query on an IP datatype field under the filter context seems to create scorers and slow down searches - although no scoring should be applied in this case. We have confirmed this by running the query under profile: true and seeing "build_scorer": 37845185684, for said query clause.
Using 6.5.4 and confirmed with few other earlier versions as well.
See full details here: https://discuss.elastic.co/t/filter-context-and-build-scorers/160725