Bad bytes in generated config file prevent config file parsing

[meverett]

Bug report

System info:
InfluxDB 1.3.6 branch 1.3, commit a7c3d3c
Windows 10, 64 bit

Steps to reproduce:

1. Run: influxd config > influxdb.conf
2. Run: influxd --config ./influxdb.conf
3. Configuration file parsing fails with the following error:

 8888888           .d888 888                   8888888b.  888888b.
   888            d88P"  888                   888  "Y88b 888  "88b
   888            888    888                   888    888 888  .88P
   888   88888b.  888888 888 888  888 888  888 888    888 8888888K.
   888   888 "88b 888    888 888  888  Y8bd8P' 888    888 888  "Y88b
   888   888  888 888    888 888  888   X88K   888    888 888    888
   888   888  888 888    888 Y88b 888 .d8""8b. 888  .d88P 888   d88P
 8888888 888  888 888    888  "Y88888 888  888 8888888P"  8888888P"

[I] 2017-10-13T20:55:24Z InfluxDB starting, version 1.3.6, branch 1.3, commit a7c3d3c0f7b74f71919047adbc4933460967a576
[I] 2017-10-13T20:55:24Z Go version go1.8.3, GOMAXPROCS set to 8
[I] 2017-10-13T20:55:24Z Using configuration at: .\influxdb.conf
run: parse config: Near line 0 (last key parsed ''): Bare keys cannot contain '�'.

Expected behavior:
Generated configuration file should parse without errors.

Actual behavior:
Generated configuration file fails parsing with errors.

Additional info:
Looking at the file in a hex editor reveals what looks like a ton of unintended characters in the stream. The file starts with the hex characters: FF FE which are invisible to most text editors. Also there are NULL characters littered throughout the file, pretty much after every normal ASCII text character is a NULL character. The parser is tripping up on all of these bad characters written to the file stream.

I wrote a script that removes the first two characters (well anything over 7F basically) and removes all the NULL characters 00 from the generated output and when I run with that "cleaned" configuration file everything works as expected.

Are there not tests that attempt to run influxd with the default generated config file on multiple platforms?

I'm not sure if this bug affects systems outside of Windows.

[jsternberg]

Can you give me a bit more information about how you generated this file? It looks like it was generated on a Windows computer, but what program was it generated inside of?

[meverett]

I believe this is a Windows-only issue.

The file was generated using influxd.exe that was part of the 1.3.6, 64 bit release zip:
influxd.exe config > influxdb.conf

The executable was invoked from the Window's command line as well as powershell and always produced this output.

[jsternberg]

Do you have a hexdump of the generated configuration file?

[meverett]

This isn't the whole file, but you can see the issue by looking at it. This file was generated by the config command and then directly opened in a hex editor. You can see the first two characters FF and FE as well as all the NULL characters 00 after pretty much every other character (they appear as '.' on the ASCII column on the right). I had to clear out both FF and FE (well as I said I cleared everything above 7F just to be safe) as well as all of the 00's and then finally influxd.exe would parse it.

[meverett]

This is what the cleaned, working version looks like:

[jsternberg]

From some initial looking online, it appears that the first two bytes are a byte-order mark and it's likely possible that the file is being written out in UTF-16 for whatever reason (hence all of the zero bytes), but then being read back as UTF-8 with no byte order mark.

I'm not sure yet why it's doing that.

[meverett]

Wow, good catch. Yeah given the evidence that sounds like a perfectly acceptable explanation. I did end up testing this release on MacOS and did not run into the issue, so I guess the clue would be that the UTF16/UTF8 mismatch is somehow related to Windows-dependent code.