Allow to seal an index

[s1monw]

In a lot of use cases indices are used for indexing only for a limited amount of time. Ie. in the daily index use case indices are created with a highish number of shards to scale out indexing and then after a couple of days these indices are idle in terms of writing. Yet, we still keep all the resources open since we are accepting writes at any time. This not necessary in a lot of cases and would allow for a large amount of optimizations:

• we can close IndexWriter and all it's resources.
• primary promotions are very simple since no changes are coming in
• indices might be able to be merged ie. if you have 7 indices for per week we can allow people to
  merge them into a weekly index to reduce the number of indices.
• query caches can rely on the fact that the index doesn't change and we can pull out min/max values for numeric fields with more confidence that it's worth the effort.
• maybe most important, recoveries are instant. We would issue a special commit when we seal that ensures that all changes have been written containing some kind of identifier that we can use in recovery which will prevent the entire copying of files in situations of full cluster restarts etc. It will reduce the restart times dramatically with very low complexity for all the time-based indices cases which usually suffer from slow cluster restarts due to the large number of indices and shards

[kimchy]

++, I think the improved recovery part idea is a killer. I wonder if we can re-use one way or another the read only flag we have today on an index.

[bleskes]

nice

[s1monw]

@kimchy I think we can. To me it's a 2 staged process, first we switch the index read only using the flag we have and once we are there and the cluster state has been published we seal the index which causes the actual optimizations to happen. so I think we can just reuse it?

[kimchy]

@s1monw ++, exactly what I was thinking about, and we have the flag recorded in the cluster state, so might need to record it somewhere else as well to make sure nothing went in, ++.

[s1monw]

adding this is also going to make full-cluster restart as well as rolling restarts likely instant. Even for the non-timeseries data / logging case for full restarts we can seal all the indices, shutdown, restart & unseal. This also works for rolling restarts if folks can afford having read only indices which I think is reasonable im most cases since the restart will be pretty fast. very promising!

[runningman84]

+1

[nik9000]

This also works for rolling restarts if folks can afford having read only indices which I think is reasonable im most cases since the restart will be pretty fast. very promising!

This is great! Our use case wouldn't allow us to seal an index outside of a rolling restart window or some other temporary maintenance action but we can absolutely get away with sealing them all for an hour or so.

So this is a great solution for us!

[synhershko]

+1, especially if sealing is reversible (which #10032 (comment) implies)