MSI: Cannot launch worker process when a key with non-ASCII character exists just under `SOFTWARE/Microsoft/Windows/CurrentVersion/Uninstall/` registry #616

tbs575 · 2023-06-14T08:44:34Z

Describe the bug

I am newbie. met error for log file with chinese character

To Reproduce

Installed td-agent 4.5
change config file
run command td-agent -c etc\td-agent\td-agent.conf
met error

sample log content:

[2023:06:13 00:00:00][INFO]请求:{"Parkcode":"510105037"}
[2023:06:13 00:00:00][INFO]返回:{"State":0,"Desc":"成功","Result":[{"num":"2032","garagecode":"1"}]}
[2023:06:13 00:00:00][INFO]开时校时...

Expected behavior

how to fixed?

Your Environment

- Fluentd version:
- TD Agent version: 4.5
- Operating system: Windows 10
- Kernel version:

Your Configuration

<source>
  @type tail
  path d:\test\logs\*.log
  tag apache.access
  encoding UTF-8
  from_encoding UTF-8
  <parse>
    @type apache2
  </parse>
  <storage>
    @type local
    persistent true
    path C:\opt\td-agent\winlog.json
  </storage>
</source>

<match winevt.raw>
  @type stdout
</match>

Your Error Log

2023-06-14 16:40:40 +0800 [info]: init supervisor logger path=nil rotate_age=nil rotate_size=nil
2023-06-14 16:40:40 +0800 [info]: parsing config file is succeeded path="etc\\td-agent\\td-agent.conf"
2023-06-14 16:40:40 +0800 [info]: gem 'fluent-plugin-calyptia-monitoring' version '0.1.3'
2023-06-14 16:40:40 +0800 [info]: gem 'fluent-plugin-elasticsearch' version '5.3.0'
2023-06-14 16:40:40 +0800 [info]: gem 'fluent-plugin-flowcounter-simple' version '0.1.0'
2023-06-14 16:40:40 +0800 [info]: gem 'fluent-plugin-kafka' version '0.19.0'
2023-06-14 16:40:40 +0800 [info]: gem 'fluent-plugin-metrics-cmetrics' version '0.1.2'
2023-06-14 16:40:40 +0800 [info]: gem 'fluent-plugin-opensearch' version '1.1.0'
2023-06-14 16:40:40 +0800 [info]: gem 'fluent-plugin-parser-winevt_xml' version '0.2.5'
2023-06-14 16:40:40 +0800 [info]: gem 'fluent-plugin-prometheus' version '2.0.3'
2023-06-14 16:40:40 +0800 [info]: gem 'fluent-plugin-prometheus_pushgateway' version '0.1.0'
2023-06-14 16:40:40 +0800 [info]: gem 'fluent-plugin-record-modifier' version '2.1.1'
2023-06-14 16:40:40 +0800 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '2.4.0'
2023-06-14 16:40:40 +0800 [info]: gem 'fluent-plugin-s3' version '1.7.2'
2023-06-14 16:40:40 +0800 [info]: gem 'fluent-plugin-sd-dns' version '0.1.0'
2023-06-14 16:40:40 +0800 [info]: gem 'fluent-plugin-td' version '1.2.0'
2023-06-14 16:40:40 +0800 [info]: gem 'fluent-plugin-webhdfs' version '1.5.0'
2023-06-14 16:40:40 +0800 [info]: gem 'fluent-plugin-windows-eventlog' version '0.8.3'
2023-06-14 16:40:40 +0800 [info]: gem 'fluent-plugin-windows-exporter' version '1.0.0'
2023-06-14 16:40:40 +0800 [info]: gem 'fluentd' version '1.16.1'
2023-06-14 16:40:40 +0800 [warn]: 'pos_file PATH' parameter is not set to a 'tail' source.
2023-06-14 16:40:40 +0800 [warn]: this parameter is highly recommended to save the position to resume tailing.
2023-06-14 16:40:40 +0800 [warn]: 'encoding' and 'from_encoding' are same encoding. No effect
2023-06-14 16:40:40 +0800 [info]: using configuration file: <ROOT>
  <source>
    @type tail
    path "d:\\test\\logs\\*.log"
    tag "apache.access"
    encoding "UTF-8"
    from_encoding "UTF-8"
    <parse>
      @type "apache2"
      unmatched_lines
    </parse>
    <storage>
      @type local
      persistent true
      path C:\opt\td-agent\winlog.json
    </storage>
  </source>
  <match winevt.raw>
    @type stdout
  </match>
</ROOT>
2023-06-14 16:40:40 +0800 [info]: starting fluentd-1.16.1 pid=6432 ruby="2.7.8"
2023-06-14 16:40:40 +0800 [info]: spawn command to main:  cmdline=["C:/opt/td-agent/bin/ruby.exe", "-Eascii-8bit:ascii-8bit", "C:/opt/td-agent/bin/fluentd", "-c", "etc\\td-agent\\td-agent.conf", "--under-supervisor"]
Traceback (most recent call last):
        17: from <internal:gem_prelude>:1:in `<internal:gem_prelude>'
        16: from <internal:gem_prelude>:1:in `require'
        15: from C:/opt/td-agent/lib/ruby/2.7.0/rubygems.rb:1427:in `<top (required)>'
        14: from C:/opt/td-agent/lib/ruby/2.7.0/rubygems.rb:1427:in `require'
        13: from C:/opt/td-agent/lib/ruby/2.7.0/rubygems/defaults/operating_system.rb:24:in `<top (required)>'
        12: from C:/opt/td-agent/lib/ruby/site_ruby/2.7.0/ruby_installer/runtime/singleton.rb:27:in `enable_dll_search_paths'
        11: from C:/opt/td-agent/lib/ruby/site_ruby/2.7.0/ruby_installer/runtime/msys2_installation.rb:125:in `enable_dll_search_paths'
        10: from C:/opt/td-agent/lib/ruby/site_ruby/2.7.0/ruby_installer/runtime/msys2_installation.rb:115:in `mingw_bin_path'
         9: from C:/opt/td-agent/lib/ruby/site_ruby/2.7.0/ruby_installer/runtime/msys2_installation.rb:102:in `msys_path'
         8: from C:/opt/td-agent/lib/ruby/site_ruby/2.7.0/ruby_installer/runtime/msys2_installation.rb:68:in `iterate_msys_paths'
         7: from C:/opt/td-agent/lib/ruby/site_ruby/2.7.0/ruby_installer/runtime/msys2_installation.rb:68:in `each'
         6: from C:/opt/td-agent/lib/ruby/site_ruby/2.7.0/ruby_installer/runtime/msys2_installation.rb:70:in `block in iterate_msys_paths'
         5: from C:/opt/td-agent/lib/ruby/2.7.0/win32/registry.rb:542:in `open'
         4: from C:/opt/td-agent/lib/ruby/2.7.0/win32/registry.rb:435:in `open'
         3: from C:/opt/td-agent/lib/ruby/site_ruby/2.7.0/ruby_installer/runtime/msys2_installation.rb:71:in `block (2 levels) in iterate_msys_paths'
         2: from C:/opt/td-agent/lib/ruby/2.7.0/win32/registry.rb:611:in `each_key'
         1: from C:/opt/td-agent/lib/ruby/2.7.0/win32/registry.rb:910:in `export_string'
C:/opt/td-agent/lib/ruby/2.7.0/win32/registry.rb:910:in `encode': U+767E to ASCII-8BIT in conversion from UTF-16LE to UTF-8 to ASCII-8BIT (Encoding::UndefinedConversionError)
2023-06-14 16:40:41 +0800 [error]: Worker 0 exited unexpectedly with status 1
Traceback (most recent call last):
        17: from <internal:gem_prelude>:1:in `<internal:gem_prelude>'
        16: from <internal:gem_prelude>:1:in `require'
        15: from C:/opt/td-agent/lib/ruby/2.7.0/rubygems.rb:1427:in `<top (required)>'
        14: from C:/opt/td-agent/lib/ruby/2.7.0/rubygems.rb:1427:in `require'
        13: from C:/opt/td-agent/lib/ruby/2.7.0/rubygems/defaults/operating_system.rb:24:in `<top (required)>'
        12: from C:/opt/td-agent/lib/ruby/site_ruby/2.7.0/ruby_installer/runtime/singleton.rb:27:in `enable_dll_search_paths'
        11: from C:/opt/td-agent/lib/ruby/site_ruby/2.7.0/ruby_installer/runtime/msys2_installation.rb:125:in `enable_dll_search_paths'
        10: from C:/opt/td-agent/lib/ruby/site_ruby/2.7.0/ruby_installer/runtime/msys2_installation.rb:115:in `mingw_bin_path'
         9: from C:/opt/td-agent/lib/ruby/site_ruby/2.7.0/ruby_installer/runtime/msys2_installation.rb:102:in `msys_path'
         8: from C:/opt/td-agent/lib/ruby/site_ruby/2.7.0/ruby_installer/runtime/msys2_installation.rb:68:in `iterate_msys_paths'
         7: from C:/opt/td-agent/lib/ruby/site_ruby/2.7.0/ruby_installer/runtime/msys2_installation.rb:68:in `each'
         6: from C:/opt/td-agent/lib/ruby/site_ruby/2.7.0/ruby_installer/runtime/msys2_installation.rb:70:in `block in iterate_msys_paths'
         5: from C:/opt/td-agent/lib/ruby/2.7.0/win32/registry.rb:542:in `open'
         4: from C:/opt/td-agent/lib/ruby/2.7.0/win32/registry.rb:435:in `open'
         3: from C:/opt/td-agent/lib/ruby/site_ruby/2.7.0/ruby_installer/runtime/msys2_installation.rb:71:in `block (2 levels) in iterate_msys_paths'
         2: from C:/opt/td-agent/lib/ruby/2.7.0/win32/registry.rb:611:in `each_key'
         1: from C:/opt/td-agent/lib/ruby/2.7.0/win32/registry.rb:910:in `export_string'
C:/opt/td-agent/lib/ruby/2.7.0/win32/registry.rb:910:in `encode': U+767E to ASCII-8BIT in conversion from UTF-16LE to UTF-8 to ASCII-8BIT (Encoding::UndefinedConversionError)
2023-06-14 16:40:41 +0800 [error]: Worker 0 exited unexpectedly with status 1

Additional context

No response

The text was updated successfully, but these errors were encountered:

daipom · 2023-06-14T09:01:00Z

Thanks for your report!

Hmm, does this always occur in a Chinese Windows environment?
From the error logs, it may be a problem that some dll have Chinese characters in their paths.

ashie · 2023-06-16T01:19:15Z

https://github.com/oneclick/rubyinstaller2/blob/1037499985fcaff2c471e88ed254a7b698e508f6/lib/ruby_installer/build/msys2_installation.rb#L61-L84
https://github.com/ruby/ruby/blob/364a6d56d776270da09604816d623047c66c5e32/ext/win32/lib/win32/registry.rb#L596C6-L616

Hmm, your registry includes a key with non-ASCII character under SOFTWARE/Microsoft/Windows/CurrentVersion/Uninstall/.
Typical keys under it is UUID but some applications seem register keys with it's application name (e.g. 7-zip).
So that it seems that we should assume non-ASCII characters are included under it.
But it doesn't seem work with current implementation because we use -Eascii-8bit:ascii-8bit at runtime.

ashie · 2023-06-16T02:00:02Z

Hmm, I've tried with a such sub key but I can't reproduce it.

tbs575 · 2023-06-21T01:56:47Z

4.5 bug on chinese windows

As tested, met this issue on version 4.5 without any logs operation.
In chinese windows, install version 4.5, and just startup td-agent directly, met error as first comment

met new issue

On version 4.4.2, can startup without any issue, but try to handle logs with chinese character, met error

2023-06-21 09:45:38 +0800 [warn]: #0 fluent/log.rb:351:warn: invalid line found file="D:/test/cars/2023-06-19-car.info.log" line="[2023:06:19 23:59:58][INFO]\xE5\xB1\x8F[172.16.69.10[\xE5\xB1\x8F]]\xE6\x95\xB0\xE6\x8D\xAE[16]\xE5\x8F\x91\xE9\x80\x81\xE6\x88\x90\xE5\x8A\x9F" error="incompatible encoding regexp match (UTF-8 regexp with ASCII-8BIT string)"
  2023-06-21 09:45:38 +0800 [debug]: #0 plugin/in_tail.rb:651:each: suppressed same stacktrace

log file encode

2023-06-19-car.info.log: Unicode text, UTF-8 (with BOM) text, with CRLF line terminators

my config

<source>
  @type tail
  path D:/test/cars/*.log
  tag yibo
  pos_file d:/test/1.log
  read_from_head true
  <parse>
    @type regexp
    expression /^\[(?<logtime>[^\]]*)\]\[INFO\]源数据上报\[(?<data_json>[^\]]*\]).*$/
  </parse>
</source>

<match yibo>
  @type stdout
</match>

how to fix it,thanks

hzilex · 2023-08-01T09:44:51Z

I faced a similar issue on the chinese Windows. It was resolved by manually installing the MSYS2 to the default path (c:/msys64 or c:/msys32)

batabatabatabata · 2023-12-18T06:16:19Z

I also met the same issue on Japanese Windows.
Since it is caused by a Microsoft product, it seems that the code needs to be fixed as a bug in multilingual support.

ashie · 2024-02-09T07:28:29Z

Hmm, I've tried with a such sub key but I can't reproduce it.

I've got the reason.
It can't be reproduced when another MSYS2 is installed in default path (c:\msys64).
https://github.com/oneclick/rubyinstaller2/blob/1037499985fcaff2c471e88ed254a7b698e508f6/lib/ruby_installer/build/msys2_installation.rb#L48-L108

Now I've confirmed this bug.

ashie · 2024-02-09T07:35:48Z

You can avoid this bug by creating a following empty file:

td-agent: c:\opt\td-agent\msys64\usr\bin\msys-2.0.dll
fluent-package: c:\opt\fluent\msys64\usr\bin\msys-2.0.dll

ashie · 2024-02-13T02:41:02Z

I confirmed that all td-agent v4 & fluent-package v5 are affected, because it's reproduced with vanilla RubyInstaller 2.7.0 & 3.2.2 (& 3.3.0) too:

C:\Users\aho\Downloads\rubyinstaller-2.7.0-1-x64\bin>echo "require 'rubygems'" | ruby -Eascii-8bit:ascii-8bit
Traceback (most recent call last):
        15: from <internal:gem_prelude>:1:in `<internal:gem_prelude>'
        14: from <internal:gem_prelude>:1:in `require'
        13: from C:/Users/aho/Downloads/rubyinstaller-2.7.0-1-x64/lib/ruby/2.7.0/rubygems.rb:1397:in `<top (required)>'
        12: from C:/Users/aho/Downloads/rubyinstaller-2.7.0-1-x64/lib/ruby/2.7.0/rubygems.rb:1397:in `require'
        11: from C:/Users/aho/Downloads/rubyinstaller-2.7.0-1-x64/lib/ruby/2.7.0/rubygems/defaults/operating_system.rb:3:in `<top (required)>'
        10: from C:/Users/aho/Downloads/rubyinstaller-2.7.0-1-x64/lib/ruby/site_ruby/2.7.0/ruby_installer/runtime/singleton.rb:27:in `enable_dll_search_paths'
         9: from C:/Users/aho/Downloads/rubyinstaller-2.7.0-1-x64/lib/ruby/site_ruby/2.7.0/ruby_installer/runtime/msys2_installation.rb:99:in `enable_dll_search_paths'
         8: from C:/Users/aho/Downloads/rubyinstaller-2.7.0-1-x64/lib/ruby/site_ruby/2.7.0/ruby_installer/runtime/msys2_installation.rb:89:in `mingw_bin_path'
         7: from C:/Users/aho/Downloads/rubyinstaller-2.7.0-1-x64/lib/ruby/site_ruby/2.7.0/ruby_installer/runtime/msys2_installation.rb:76:in `msys_path'
         6: from C:/Users/aho/Downloads/rubyinstaller-2.7.0-1-x64/lib/ruby/site_ruby/2.7.0/ruby_installer/runtime/msys2_installation.rb:48:in `iterate_msys_paths'
         5: from C:/Users/aho/Downloads/rubyinstaller-2.7.0-1-x64/lib/ruby/2.7.0/win32/registry.rb:542:in `open'
         4: from C:/Users/aho/Downloads/rubyinstaller-2.7.0-1-x64/lib/ruby/2.7.0/win32/registry.rb:435:in `open'
         3: from C:/Users/aho/Downloads/rubyinstaller-2.7.0-1-x64/lib/ruby/site_ruby/2.7.0/ruby_installer/runtime/msys2_installation.rb:49:in `block in iterate_msys_paths'
         2: from C:/Users/aho/Downloads/rubyinstaller-2.7.0-1-x64/lib/ruby/2.7.0/win32/registry.rb:611:in `each_key'
         1: from C:/Users/aho/Downloads/rubyinstaller-2.7.0-1-x64/lib/ruby/2.7.0/win32/registry.rb:910:in `export_string'
C:/Users/aho/Downloads/rubyinstaller-2.7.0-1-x64/lib/ruby/2.7.0/win32/registry.rb:910:in `encode': U+3042 to ASCII-8BIT in conversion from UTF-16LE to UTF-8 to ASCII-8BIT (Encoding::UndefinedConversionError)

ashie · 2024-02-13T02:42:39Z

This should be treated in fluent-pakckage-builder project.

ashie · 2024-02-13T03:22:05Z

I want to fix it in upstream, but I'm not sure where to report it yet.

The codes which causes this issue:

RubyInstaller2:

https://github.com/oneclick/rubyinstaller2/blob/898188c555acacbc81141adda73d361922fd1f72/lib/ruby_installer/build/msys2_installation.rb#L75

Ruby:

In addition, we should add work around for this issue until it's addressed in upstream.
Probably we can apply a patch to RubyInstaller by the following code:

fluent-package-builder/fluent-package/config.rb

Lines 38 to 44 in 7e637b3

    
           # Patch files are assumed to be for Ruby's source tree, then applied to 
        
           # lib/ruby/x.y.0 in RubyInstaller. So that "-p2" options will be passed 
        
           # to patch command. 
        
           BUNDLED_RUBY_INSTALLER_PATCHES = [ 
        
             # An example entry: 
        
             # ["ruby-3.0/0001-ruby-resolv-Fix-confusion-of-received-response-messa.patch", ["= 3.0.1"]], 
        
           ]

It would be better to treat only MSYS2 which is installed under C:\opt\fluent, it can be realized easily by adding return here.

When a non-ASCII key exists under the registry key `SOFTWARE/Microsoft/Windows/CurrentVersion/Uninstall/`, Fluentd fails to start workers due to `Encoding::UndefinedConversionError`. This patch avoid this issue. Fix #616 Signed-off-by: Takuro Ashie <ashie@clear-code.com>

ashie added the bug Something isn't working label Jun 16, 2023

ashie transferred this issue from fluent/fluentd Feb 13, 2024

ashie changed the title ~~met error with chinese char’s log file~~ MSI: Cannot run worker process when a key with non-ASCII character existi in SOFTWARE/Microsoft/Windows/CurrentVersion/Uninstall/ Registry Feb 13, 2024

ashie mentioned this issue Feb 13, 2024

Crash on require rubygems when running ruby with -Eascii-8bit:ascii-8bit and registry contains non-ASCII key oneclick/rubyinstaller2#372

Closed

ashie mentioned this issue Feb 16, 2024

Add a patch for RubyInstaller to avoid crash on start up #620

Merged

kenhys closed this as completed in #620 Feb 22, 2024

ashie mentioned this issue Feb 28, 2024

fluent/lib/ruby/3.2.0/win32/registry.rb:910:in `encode': U+767E to ASCII-8BIT in conversion from UTF-16LE to UTF-8 to ASCII-8BIT (Encoding::UndefinedConversionError) fluent/fluentd#4414

Closed

ashie mentioned this issue Feb 29, 2024

td-agent-v4: Backport #620 #625

Merged

jackcomtw mentioned this issue Jul 3, 2024

D:/opt/calyptia-fluentd/lib/ruby/3.2.0/rubygems.rb:1342:in `rescue in <top (required)>': U+795E to ASCII-8BIT in conversion from UTF-16LE to UTF-8 to ASCII-8BIT (Encoding::UndefinedConversionError) fluent/fluentd#4550

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

MSI: Cannot launch worker process when a key with non-ASCII character exists just under `SOFTWARE/Microsoft/Windows/CurrentVersion/Uninstall/` registry #616

MSI: Cannot launch worker process when a key with non-ASCII character exists just under `SOFTWARE/Microsoft/Windows/CurrentVersion/Uninstall/` registry #616

tbs575 commented Jun 14, 2023

daipom commented Jun 14, 2023

ashie commented Jun 16, 2023 •

edited

Loading

ashie commented Jun 16, 2023

tbs575 commented Jun 21, 2023 •

edited

Loading

hzilex commented Aug 1, 2023

batabatabatabata commented Dec 18, 2023

ashie commented Feb 9, 2024 •

edited

Loading

ashie commented Feb 9, 2024

ashie commented Feb 13, 2024 •

edited

Loading

ashie commented Feb 13, 2024

ashie commented Feb 13, 2024 •

edited

Loading

MSI: Cannot launch worker process when a key with non-ASCII character exists just under SOFTWARE/Microsoft/Windows/CurrentVersion/Uninstall/ registry #616

MSI: Cannot launch worker process when a key with non-ASCII character exists just under SOFTWARE/Microsoft/Windows/CurrentVersion/Uninstall/ registry #616

Comments

tbs575 commented Jun 14, 2023

Describe the bug

To Reproduce

Expected behavior

Your Environment

Your Configuration

Your Error Log

Additional context

daipom commented Jun 14, 2023

ashie commented Jun 16, 2023 • edited Loading

ashie commented Jun 16, 2023

tbs575 commented Jun 21, 2023 • edited Loading

4.5 bug on chinese windows

met new issue

hzilex commented Aug 1, 2023

batabatabatabata commented Dec 18, 2023

ashie commented Feb 9, 2024 • edited Loading

ashie commented Feb 9, 2024

ashie commented Feb 13, 2024 • edited Loading

ashie commented Feb 13, 2024

ashie commented Feb 13, 2024 • edited Loading

MSI: Cannot launch worker process when a key with non-ASCII character exists just under `SOFTWARE/Microsoft/Windows/CurrentVersion/Uninstall/` registry #616

MSI: Cannot launch worker process when a key with non-ASCII character exists just under `SOFTWARE/Microsoft/Windows/CurrentVersion/Uninstall/` registry #616

ashie commented Jun 16, 2023 •

edited

Loading

tbs575 commented Jun 21, 2023 •

edited

Loading

ashie commented Feb 9, 2024 •

edited

Loading

ashie commented Feb 13, 2024 •

edited

Loading

ashie commented Feb 13, 2024 •

edited

Loading