Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

add ignore_network_errors_at_startup #1399

Merged
merged 3 commits into from
Jan 10, 2017
Merged

add ignore_network_errors_at_startup #1399

merged 3 commits into from
Jan 10, 2017

Conversation

tagomoris
Copy link
Member

This change is revival of #735, for v0.14 out_forward.
This parameter is used for the case: deployment systems will create the DNS records of hosts in <server>.

@tagomoris tagomoris added enhancement Feature request or improve operations v0.14 labels Jan 4, 2017
@tagomoris tagomoris self-assigned this Jan 4, 2017
@tagomoris
Copy link
Member Author

@repeatedly Could you review this change?

@tagomoris tagomoris mentioned this pull request Jan 4, 2017
Closed
@tagomoris tagomoris force-pushed the ignore-network-error-at-startup-in-out_forward branch from 762b3f1 to c761785 Compare January 5, 2017 02:56
@tagomoris
Copy link
Member Author

@repeatedly ping

@repeatedly
Copy link
Member

This is incomplete.
We also need to care DNS error in heartbeat handler:

fluentd/lib/fluent/plugin/out_forward.rb

Line 321 in 9068e0f

def on_timer

@tagomoris
Copy link
Member Author

@repeatedly Heartbeat handler sends heartbeats by n.send_heartbeat method, and it should be done after n.validate_host_resolution!.
So, there are no need to add additional fix especially for heartbeat handlers.

@repeatedly
Copy link
Member

repeatedly commented Jan 5, 2017 •
edited
Loading

Really?
I got following error and timer is stopped.

2017-01-05 16:25:30 +0900 [info]: adding match pattern="forward.**" type="forward"
2017-01-05 16:25:30 +0900 [info]: adding forwarding server 'fooooobarrrr:24444' host="fooooobarrrr" port=24444 weight=60 plugin_id="object:3ff48a97a9dc"
2017-01-05 16:25:30 +0900 [warn]: failed to resolve node name when configured server="fooooobarrrr" error_class=SocketError error="getaddrinfo: nodename nor servname provided, or not known"
2017-01-05 16:25:30 +0900 [info]: adding match pattern="**" type="null"
2017-01-05 16:25:30 +0900 [info]: adding source type="forward"
2017-01-05 16:25:30 +0900 [info]: using configuration file: <ROOT>
  <source>
    @type forward
  </source>
  <match forward.**>
    @type forward
    flush_interval 5s
    ignore_network_errors_at_startup true
    retry_limit 4
    max_retry_wait 4
    <server>
      host "fooooobarrrr"
      port 24444
    </server>
    <buffer tag>
      flush_interval 5s
      retry_max_times 4
      retry_max_interval 4
    </buffer>
  </match>
  <match **>
    @type null
  </match>
</ROOT>
2017-01-05 16:25:30 +0900 [info]: starting fluentd worker pid=31724 ppid=31696 worker="0"
2017-01-05 16:25:30 +0900 [info]: delayed_commit_timeout is overwritten by ack_response_timeout
2017-01-05 16:25:30 +0900 [info]: fluentd worker is now running
2017-01-05 16:25:31 +0900 [error]: Unexpected error raised. Stopping the timer. title=:out_forward_heartbeat_request error_class=SocketError error="getaddrinfo: nodename nor servname provided, or not known"
  2017-01-05 16:25:31 +0900 [error]: /Users/repeatedly/dev/fluentd/fluentd/lib/fluent/plugin/out_forward.rb:607:in `getaddrinfo'
  2017-01-05 16:25:31 +0900 [error]: /Users/repeatedly/dev/fluentd/fluentd/lib/fluent/plugin/out_forward.rb:607:in `resolve_dns!'
  2017-01-05 16:25:31 +0900 [error]: /Users/repeatedly/dev/fluentd/fluentd/lib/fluent/plugin/out_forward.rb:593:in `resolved_host'
  2017-01-05 16:25:31 +0900 [error]: /Users/repeatedly/dev/fluentd/fluentd/lib/fluent/plugin/out_forward.rb:569:in `send_heartbeat'
  2017-01-05 16:25:31 +0900 [error]: /Users/repeatedly/dev/fluentd/fluentd/lib/fluent/plugin/out_forward.rb:340:in `block in on_timer'
  2017-01-05 16:25:31 +0900 [error]: /Users/repeatedly/dev/fluentd/fluentd/lib/fluent/plugin/out_forward.rb:333:in `each'
  2017-01-05 16:25:31 +0900 [error]: /Users/repeatedly/dev/fluentd/fluentd/lib/fluent/plugin/out_forward.rb:333:in `on_timer'
  2017-01-05 16:25:31 +0900 [error]: /Users/repeatedly/dev/fluentd/fluentd/lib/fluent/plugin_helper/timer.rb:77:in `on_timer'
  2017-01-05 16:25:31 +0900 [error]: /Users/repeatedly/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/cool.io-1.4.5/lib/cool.io/loop.rb:88:in `run_once'
  2017-01-05 16:25:31 +0900 [error]: /Users/repeatedly/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/cool.io-1.4.5/lib/cool.io/loop.rb:88:in `run'
  2017-01-05 16:25:31 +0900 [error]: /Users/repeatedly/dev/fluentd/fluentd/lib/fluent/plugin_helper/event_loop.rb:77:in `block in start'
  2017-01-05 16:25:31 +0900 [error]: /Users/repeatedly/dev/fluentd/fluentd/lib/fluent/plugin_helper/thread.rb:66:in `block in thread_create'
2017-01-05 16:25:31 +0900 [error]: Timer detached. title=:out_forward_heartbeat_request

@tagomoris
Copy link
Member Author

Oops, I misunderstood the problem... (and the former pull-request also doesn't solve this problem).

@tagomoris
Copy link
Member Author

@repeatedly I pushed a commit for that case.

@tagomoris
Copy link
Member Author

@repeatedly hey

repeatedly
repeatedly reviewed Jan 6, 2017
View reviewed changes
lib/fluent/plugin/out_forward.rb
dest_addr = resolved_host
@resolved_once = true
rescue ::SocketError
if !@resolved_once && @ignore_network_errors_at_startup
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ignore_network_errors_at_startup is not a Node's instance variable.

repeatedly
repeatedly reviewed Jan 6, 2017
View reviewed changes
lib/fluent/plugin/out_forward.rb
@resolved_once = true
rescue ::SocketError
if !@resolved_once && @ignore_network_errors_at_startup
log.warn "failed to resolve node name in heartbeating", server: (n.name || n.host), error: e
Copy link
Member

@repeatedly repeatedly Jan 6, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

log, n and e are not defined.

@tagomoris tagomoris force-pushed the ignore-network-error-at-startup-in-out_forward branch from b058cb2 to a3c233c Compare January 6, 2017 08:56
@tagomoris tagomoris force-pushed the ignore-network-error-at-startup-in-out_forward branch from a3c233c to d63e5e4 Compare January 6, 2017 08:57
@tagomoris
Copy link
Member Author

@repeatedly pushed updated commits.

@repeatedly
Copy link
Member

LGTM

@tagomoris tagomoris merged commit 2a53248 into master Jan 10, 2017
@tagomoris tagomoris deleted the ignore-network-error-at-startup-in-out_forward branch January 10, 2017 00:27
@gozer gozer mentioned this pull request Jan 29, 2018
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@repeatedly repeatedly repeatedly left review comments

Assignees

@tagomoris tagomoris

Labels
enhancement Feature request or improve operations v0.14
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@tagomoris @repeatedly