Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Rewriting fields.idx is a bottleneck with many new fields and measurements #21577

Closed
davidby-influx opened this issue May 28, 2021 · 0 comments · Fixed by #21592 or #21610
Closed

Rewriting fields.idx is a bottleneck with many new fields and measurements #21577

davidby-influx opened this issue May 28, 2021 · 0 comments · Fixed by #21592 or #21610
Assignees
@davidby-influx
Labels
1.x area/performance area/writes area/2.x OSS 2.0 related issues and PRs kind/enhancement team/edge

Comments

@davidby-influx
Copy link
Contributor

When using schema designs that involve many (hundreds of thousands) of
measurements and fields, the non-incremental rewrites of fields.idx can
become a serious bottleneck. While the IOx storage engine will solve this
in the long run, as a temporary improvement, influxdb should abandon
rewrites that will be immediately overwritten by subsequent fields and
measurement creation, whenever possible.
@davidby-influx davidby-influx self-assigned this May 28, 2021
@davidby-influx davidby-influx mentioned this issue May 28, 2021
Closed
8 tasks
davidby-influx added a commit that referenced this issue Jun 3, 2021
@davidby-influx
fix: avoid rewriting fields.idx unnecessarily
e76530a 
Under heavy write load creating new fields and measurements
the rewrite of the fields.idx file is a bottleneck. This
enhancement combines multiple writes into a single one and
shares any error return value with all of the combined
invocations.

Closes #21577
@davidby-influx davidby-influx mentioned this issue Jun 3, 2021
Merged
4 tasks
davidby-influx added a commit that referenced this issue Jun 3, 2021
@davidby-influx
fix: avoid rewriting fields.idx unnecessarily
8976a8f 
Add syntactic sugar to wrap the error return
channel, with a writeRequest struct.  Add an explicit
Close() methods to MeasurementFieldSet and
MeasurementsFieldSetWriter. Update
tests and engine to call Close() methods.

Closes #21577
@davidby-influx davidby-influx linked a pull request Jun 3, 2021 that will close this issue
fix: avoid rewriting fields.idx unnecessarily #21592
Merged
4 tasks
This was referenced Jun 3, 2021
Closed
Closed
Closed
davidby-influx added a commit that referenced this issue Jun 4, 2021
@davidby-influx
fix: avoid rewriting fields.idx unnecessarily (#21592)
f64be28 
Under heavy write load creating new fields and measurements
the rewrite of the fields.idx file is a bottleneck. This
enhancement combines multiple writes into a single one and
shares any error return value with all of the combined
invocations. MeasurementFieldSet and the new 
MeasurementFieldSetWriter must both now be explicitly
closed.

Closes #21577
davidby-influx added a commit that referenced this issue Jun 4, 2021
@davidby-influx
fix: avoid rewriting fields.idx unnecessarily (#21592)
83a5377 
Under heavy write load creating new fields and measurements
the rewrite of the fields.idx file is a bottleneck. This
enhancement combines multiple writes into a single one and
shares any error return value with all of the combined
invocations. MeasurementFieldSet and the new
MeasurementFieldSetWriter must both now be explicitly
closed.

Closes #21577

(cherry picked from commit f64be28)
@davidby-influx davidby-influx mentioned this issue Jun 4, 2021
Merged
4 tasks
davidby-influx added a commit that referenced this issue Jun 4, 2021
@davidby-influx
fix: avoid rewriting fields.idx unnecessarily (#21592) [Port to 1.9] (#…
c8e8025 
…21609)

Under heavy write load creating new fields and measurements
the rewrite of the fields.idx file is a bottleneck. This
enhancement combines multiple writes into a single one and
shares any error return value with all of the combined
invocations. MeasurementFieldSet and the new
MeasurementFieldSetWriter must both now be explicitly
closed.

Closes #21577

(cherry picked from commit f64be28)

Closes #21597
davidby-influx added a commit that referenced this issue Jun 4, 2021
@davidby-influx
fix: avoid rewriting fields.idx unnecessarily (#21592)
fa0da4c 
Under heavy write load creating new fields and measurements
the rewrite of the fields.idx file is a bottleneck. This
enhancement combines multiple writes into a single one and
shares any error return value with all of the combined
invocations. MeasurementFieldSet and the new
MeasurementFieldSetWriter must both now be explicitly
closed.

Closes #21577

(cherry picked from commit f64be28)
@davidby-influx davidby-influx mentioned this issue Jun 4, 2021
Merged
4 tasks
davidby-influx added a commit that referenced this issue Jun 4, 2021
@davidby-influx
fix: avoid rewriting fields.idx unnecessarily (#21592) (#21610)
d10a727 
Under heavy write load creating new fields and measurements
the rewrite of the fields.idx file is a bottleneck. This
enhancement combines multiple writes into a single one and
shares any error return value with all of the combined
invocations. MeasurementFieldSet and the new
MeasurementFieldSetWriter must both now be explicitly
closed.

Closes #21577

(cherry picked from commit f64be28)

Closes #21598
lesam pushed a commit to lesam/influxdb that referenced this issue Jun 15, 2021
@davidby-influx @lesam
fix: avoid rewriting fields.idx unnecessarily (influxdata#21592) [Por…
f6286f0 
…t to 1.9] (influxdata#21609)

Under heavy write load creating new fields and measurements
the rewrite of the fields.idx file is a bottleneck. This
enhancement combines multiple writes into a single one and
shares any error return value with all of the combined
invocations. MeasurementFieldSet and the new
MeasurementFieldSetWriter must both now be explicitly
closed.

Closes influxdata#21577

(cherry picked from commit f64be28)

Closes influxdata#21597
lesam added a commit that referenced this issue Jun 15, 2021
@lesam @davidby-influx
fix: avoid rewriting fields.idx unnecessarily (#21692)
1661b6f 
* fix: avoid rewriting fields.idx unnecessarily (#21592) [Port to 1.9] (#21609)

Under heavy write load creating new fields and measurements
the rewrite of the fields.idx file is a bottleneck. This
enhancement combines multiple writes into a single one and
shares any error return value with all of the combined
invocations. MeasurementFieldSet and the new
MeasurementFieldSetWriter must both now be explicitly
closed.

Closes #21577

(cherry picked from commit f64be28)

Closes #21597

* chore: minor refactor suggested by go lint (#21614) (#21616)

(cherry picked from commit 7d10228)
(cherry picked from commit f820287)

Co-authored-by: davidby-influx <72418212+davidby-influx@users.noreply.github.com>
@davidby-influx davidby-influx mentioned this issue Jul 20, 2021
Closed
chengshiwen pushed a commit to chengshiwen/influxdb that referenced this issue Aug 27, 2024
@davidby-influx @chengshiwen
fix: avoid rewriting fields.idx unnecessarily (influxdata#21592)
a58ee5f 
Under heavy write load creating new fields and measurements
the rewrite of the fields.idx file is a bottleneck. This
enhancement combines multiple writes into a single one and
shares any error return value with all of the combined
invocations. MeasurementFieldSet and the new
MeasurementFieldSetWriter must both now be explicitly
closed.

Closes influxdata#21577
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@davidby-influx davidby-influx

Labels
1.x area/performance area/writes area/2.x OSS 2.0 related issues and PRs kind/enhancement team/edge
Projects
None yet
Milestone
No milestone
1 participant
@davidby-influx