engine: error syncing wal

[tinycolds]

System info:
Influxdb: 1.5.3
SySTEM: Linux 3.16.0-4-amd64 #1 SMP Debian 3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux

Steps to reproduce:
We build a single node influxdb server on SSD
Write requests rate: 10k/s
When we do a query request, found the error log bellow, and the error occurs only when do the query request, is the slow query the root cause?

httpd] 10.15.57.69 - - [05/Sep/2018:17:14:31 +0800] "POST /write?consistency=&db=default&precision=s&rp=default HTTP/1.1" 204 0 "-" "Go-http-client/1.1" 18f6eb6d-b0ec-11e8-8302-000000000000 1429509
ts=2018-09-05T09:14:32.865162Z lvl=error msg="[500] - \"engine: error syncing wal\"" log_id=0AMEUnG0000 service=httpd
[httpd] 10.15.57.69 - - [05/Sep/2018:17:14:31 +0800] "POST /write?consistency=&db=default&precision=s&rp=default HTTP/1.1" 204 0 "-" "Go-http-client/1.1" 19099a1d-b0ec-11e8-8b77-000000000000 1307072
ts=2018-09-05T09:14:32.865215Z lvl=info msg="Write failed" log_id=0AMEUnG0000 service=write shard=1675 error="engine: error syncing wal"
ts=2018-09-05T09:14:32.865223Z lvl=info msg="Write failed" log_id=0AMEUnG0000 service=write shard=1675 error="engine: error syncing wal"
[httpd] 10.15.57.69 - - [05/Sep/2018:17:14:31 +0800] "POST /write?consistency=&db=default&precision=s&rp=default HTTP/1.1" 204 0 "-" "Go-http-client/1.1" 19028b67-b0ec-11e8-844c-000000000000 1353330
ts=2018-09-05T09:14:32.865229Z lvl=error msg="[500] - \"engine: error syncing wal\"" log_id=0AMEUnG0000 service=httpd
[httpd] 10.15.57.69 - - [05/Sep/2018:17:14:31 +0800] "POST /write?consistency=&db=default&precision=s&rp=default HTTP/1.1" 204 0 "-" "Go-http-client/1.1" 19093df2-b0ec-11e8-8a3c-000000000000 1309439
ts=2018-09-05T09:14:32.865231Z lvl=info msg="Write failed" log_id=0AMEUnG0000 service=write shard=1675 error="engine: error syncing wal"
ts=2018-09-05T09:14:32.865241Z lvl=info msg="Write failed" log_id=0AMEUnG0000 service=write shard=1675 error="engine: error syncing wal"
ts=2018-09-05T09:14:32.865253Z lvl=info msg="Write failed" log_id=0AMEUnG0000 service=write shard=1675 error="engine: error syncing wal"
[httpd] 10.15.57.69 - - [05/Sep/2018:17:14:31 +0800] "POST /write?consistency=&db=default&precision=s&rp=default HTTP/1.1" 204 0 "-" "Go-http-client/1.1" 1902a5ac-b0ec-11e8-84a3-000000000000 1352660
ts=2018-09-05T09:14:32.865260Z lvl=info msg="Write failed" log_id=0AMEUnG0000 service=write shard=1675 error="engine: error syncing wal"
ts=2018-09-05T09:14:32.865267Z lvl=info msg="Write failed" log_id=0AMEUnG0000 service=write shard=1675 error="engine: error syncing wal"
ts=2018-09-05T09:14:32.865279Z lvl=info msg="Write failed" log_id=0AMEUnG0000 service=write shard=1675 error="engine: error syncing wal"
[httpd] 10.15.57.69 - - [05/Sep/2018:17:14:31 +0800] "POST /write?consistency=&db=default&precision=s&rp=default HTTP/1.1" 204 0 "-" "Go-http-client/1.1" 1909bc01-b0ec-11e8-8bed-000000000000 1306215
ts=2018-09-05T09:14:32.865289Z lvl=info msg="Write failed" log_id=0AMEUnG0000 service=write shard=1675 error="engine: error syncing wal"
ts=2018-09-05T09:14:32.865300Z lvl=info msg="Write failed" log_id=0AMEUnG0000 service=write shard=1675 error="engine: error syncing wal"
ts=2018-09-05T09:14:32.865310Z lvl=info msg="Write failed" log_id=0AMEUnG0000 service=write shard=1675 error="engine: error syncing wal"
[httpd] 10.15.57.69 - - [05/Sep/2018:17:14:32 +0800] "POST /write?consistency=&db=default&precision=s&rp=default HTTP/1.1" 204 0 "-" "Go-http-client/1.1" 19a55456-b0ec-11e8-8dcd-000000000000 286509
ts=2018-09-05T09:14:32.865320Z lvl=info msg="Write failed" log_id=0AMEUnG0000 service=write shard=1675 error="engine: error syncing wal"
ts=2018-09-05T09:14:32.865330Z lvl=info msg="Write failed" log_id=0AMEUnG0000 service=write shard=1675 error="engine: error syncing wal"
[httpd] 10.15.57.69 - - [05/Sep/2018:17:14:31 +0800] "POST /write?consistency=&db=default&precision=s&rp=default HTTP/1.1" 204 0 "-" "Go-http-client/1.1" 190c4873-b0ec-11e8-8f6e-000000000000 1289515
ts=2018-09-05T09:14:32.865341Z lvl=info msg="Write failed" log_id=0AMEUnG0000 service=write shard=1675 error="engine: error syncing wal"
[httpd] 10.15.57.69 - - [05/Sep/2018:17:14:31 +0800] "POST /write?consistency=&db=default&precision=s&rp=default HTTP/1.1" 204 0 "-" "Go-http-client/1.1" 190edc02-b0ec-11e8-9433-000000000000 1272635

The SSD IO stats bellow:

09/05/2018 05:14:27 PM
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          16.54    0.00    6.06    0.23    0.00   77.17

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sdb               0.00     3.00    0.00   27.00     0.00  7272.00   538.67     9.03   58.22    0.00   58.22   6.81  18.40
sdd               0.00   501.00    0.00  389.00     0.00  5544.00    28.50     0.06    0.17    0.00    0.17   0.14   5.60

09/05/2018 05:14:28 PM
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          39.81    0.00    5.39    0.05    0.00   54.75

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0.00     0.00    0.00    2.00     0.00     8.00     8.00     0.00    0.00    0.00    0.00   0.00   0.00
sdb               0.00     0.00    0.00   47.00     0.00 15180.00   645.96     7.74  323.40    0.00  323.40   6.47  30.40
sdd               0.00   649.00    0.00  509.00     0.00  6700.00    26.33     0.08    0.16    0.00    0.16   0.15   7.60

09/05/2018 05:14:29 PM
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          38.27    0.00    5.07    0.13    0.00   56.53

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sdd               0.00   408.00    0.00  358.00     0.00  4304.00    24.04     0.03    0.08    0.00    0.08   0.06   2.00

09/05/2018 05:14:30 PM
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          68.82    0.00    4.47    0.05    0.00   26.66

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sdd               0.00   279.00    0.00  245.00     0.00  2908.00    23.74     0.01    0.05    0.00    0.05   0.03   0.80

09/05/2018 05:14:31 PM
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          81.71    0.00    8.94    2.65    0.00    6.70

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sdb               0.00    18.00    0.00    6.00     0.00    96.00    32.00     0.13   22.00    0.00   22.00  10.00   6.00
sdd               0.00    15.00    0.00   28.00     0.00   688.00    49.14     0.01    0.29    0.00    0.29   0.29   0.80

09/05/2018 05:14:32 PM
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          33.27    0.00    9.12    4.30    0.00   53.30

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sdd               0.00    11.00    0.00    8.00     0.00  1028.00   257.00     0.01    1.50    0.00    1.50   0.50   0.40

09/05/2018 05:14:33 PM
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          56.49    0.00    9.56    0.49    0.00   33.46

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sdd               0.00  1072.00    0.00  910.00     0.00 18216.00    40.04     0.11    0.12    0.00    0.12   0.09   8.40

09/05/2018 05:14:34 PM
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          41.66    0.00    8.42    0.36    0.00   49.55

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0.00     5.00    0.00    6.00     0.00    44.00    14.67     0.00    0.00    0.00    0.00   0.00   0.00
sdc               0.00     0.00    0.00    2.00     0.00     8.00     8.00     0.00    0.00    0.00    0.00   0.00   0.00
sdd               0.00    79.00    0.00   72.00     0.00  1812.00    50.33     0.00    0.06    0.00    0.06   0.06   0.40

09/05/2018 05:14:35 PM
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          26.77    0.00    7.34    1.04    0.00   64.85

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sdc               0.00     7.00    0.00    2.00     0.00    36.00    36.00     0.00    0.00    0.00    0.00   0.00   0.00
sdd               0.00    38.00    0.00   33.00     0.00  3628.00   219.88     0.02    0.61    0.00    0.61   0.24   0.80

09/05/2018 05:14:36 PM
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          15.41    0.00    5.79    1.00    0.00   77.80

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sdb               0.00     8.00    0.00    2.00     0.00    40.00    40.00     0.02   12.00    0.00   12.00  12.00   2.40
sdd               0.00    51.00    0.00   31.00     0.00  3188.00   205.68     0.02    0.52    0.00    0.52   0.26   0.80

[tux-00]

Any news ?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically closed because it has not had recent activity. Please reopen if this issue is still important to you. Thank you for your contributions.