Registry file fsync improvements #6988
Conversation
urso
force-pushed
the
improve-registry-sync
branch
2 times, most recently
from
6032970 to
6f5d96d
Compare
filebeat/registrar/registrar.go
Outdated
| @@ -41,10 +41,10 @@ var ( | |||
| registryWrites = monitoring.NewInt(nil, "registrar.writes") | |||
| ) | |||
|
|
|||
| func New(registryFile string, registryFilePermissions os.FileMode, flushTimeout time.Duration, out successLogger) (*Registrar, error) { | |||
| func New(registryFile string, fileMode os.FileMode, flushTimeout time.Duration, out successLogger) (*Registrar, error) { | |||
There was a problem hiding this comment.
exported function New should have comment or be unexported
- Return error if Sync fails - Execute fsync on new parent directory - improve metrics: - registrar.writes.total: total number of registry write attempts - registrar.writes.fail: total number of failed write attempts - registrar.writes.success: total number of successfull write attempts
urso
force-pushed
the
improve-registry-sync
branch
from
6f5d96d to
d570f9d
Compare
filebeat/registrar/registrar.go
Outdated
| ) | ||
|
|
||
| func New(registryFile string, registryFilePermissions os.FileMode, flushTimeout time.Duration, out successLogger) (*Registrar, error) { | ||
| func New(registryFile string, fileMode os.FileMode, flushTimeout time.Duration, out successLogger) (*Registrar, error) { |
There was a problem hiding this comment.
exported function New should have comment or be unexported
These metrics are internal/private only. No need for changelog entry. |
|
There is a clean_inactive failure on travis. Not sure if it is related: https://travis-ci.org/elastic/beats/jobs/374064172 |
|
The tests did succeed on jenkins. I wonder if the extra fsyncs change some timings, making some tests more flaky. |
|
@urso more flaky maybe, we will need a bit more iteration to find out. Did you restart the failings tests on travis? |
|
@ph yes, I did restart the test. Now it's green :) . Related beats-ci tests have been green. |
|
@urso I will merge this, lets keep an eye out if this test is still flaky, I've seen it fails before :) |
|
Should we backport this one? |
|
When will this pr release? in 6.3? We actually met this kind of problem in our production environment. Though we can solve by deleting the registrar file, but as a result, filebeat will read old files once again. This is terrible. |
urso
added
v6.3.1
and removed
needs_backport
* Registry file fsync improvements - Return error if Sync fails - Execute fsync on new parent directory - improve metrics: - registrar.writes.total: total number of registry write attempts - registrar.writes.fail: total number of failed write attempts - registrar.writes.success: total number of successfull write attempts (cherry picked from commit 5916636)
* Registry file fsync improvements - Return error if Sync fails - Execute fsync on new parent directory - improve metrics: - registrar.writes.total: total number of registry write attempts - registrar.writes.fail: total number of failed write attempts - registrar.writes.success: total number of successfull write attempts (cherry picked from commit 5916636)
* Registry file fsync improvements - Return error if Sync fails - Execute fsync on new parent directory - improve metrics: - registrar.writes.total: total number of registry write attempts - registrar.writes.fail: total number of failed write attempts - registrar.writes.success: total number of successfull write attempts (cherry picked from commit f5179f9)
Resolves: #6792 (followup for #6877)