Metricbeat 6.0-alpha2 Vsphere module: not authenticated (auth not being reissued) #4673

alextxm · 2017-07-14T09:12:55Z

I've configured the Vsphere module in metricbeat 6.0.0-alpha2 (on a Windows 2008R2 machine) and let is run for a while. Please note than my VSphere configuration requires authentication (with insecure:true).
Metricbeat gathered data for about 12+ hours then Elasticseach started being filled with metric with the error.message field populated to NotAuthenticated (error.message:NotAuthenticated).
It happened when VSphere went "offline" for a scheduled backup activity; as it got back "online" the vsphere module seems not had performed auth again and as such ES started being populated with events with the NotAuthenticated error. I'm attaching a screenshot from kibana which details the described flow.
Please note also that no error messages can be found in the metricbeat log itself.
Is there a way to have the vsphere module to perform auth again ?

exekias · 2017-07-17T09:58:43Z

thank you for reporting @alextxm, I have some questions:

Did you change username/password for your beats user?
Did you restart VSphere server at any moment?

I'll have a look to our client library (govmomi), perhaps the session expired and it doesn't authenticate on reconnect

alextxm · 2017-07-20T13:03:42Z

Hi @exekias,
the username/password didn't change but the Vsphere server services had been restarted due to the nightly scheduled vshere DBbackup activity... this should be the cause of the session expiration.
Please note that this scenario could be pretty common IMHO since AFAIK vsphere backup requires stop/restart of the related services; as such, a session expiration/reconnect handling mechanism in the client lib (or the module itself) would be really useful.

exekias · 2017-07-21T09:53:26Z

So I think the client library is not issuing a new Login after reconnect, we will need to confirm that and fix it

alextxm · 2017-08-07T09:22:49Z

Hi, is there any news on this ? Can i help with further testing ?

amandahla · 2017-08-15T18:34:59Z

@exekias

First I tried to use iptables to block the access and it worked fine. Reconnects after I deleted the rule.
Then I try to terminate the metricbeat session using VSphere Web Client. This way I got the same error "NotAuthenticated" all the time.
I'm not sure if this is the best way to fix and reconnect but at least in this test that I made, it worked.

Update: I was getting a nil pointer but just needed to recreate the view after reauthenticate

exekias · 2017-08-16T09:08:04Z

@amandahla from what I see we could benefit from using Session instead of Client? It would handle keepalive and some more goodies, what do you think? https://github.com/vmware/vic/blob/master/pkg/vsphere/session/session.go#L50-L87

amandahla · 2017-08-16T14:44:24Z

@exekias I'm just afraid if it will work on both versions (6.0/6.5) because I see here that uses finder to populate. Same thing that I had to change because of the 'datastore (or host) '*' not found' error. But I believe that it's worth to give it a try. I'll try to follow this to make changes and test. What do you think?

exekias · 2017-08-16T15:43:21Z

Sounds good to me, I guess we can wait for that before merging #4883?

amandahla · 2017-08-16T17:05:07Z

Yes, I think it would be better.

Please, can you help me with something? I made a test and now I get this:

./metricbeat flag redefined: version
panic: ./metricbeat flag redefined: version

goroutine 1 [running]:
flag.(*FlagSet).Var(0xc42001c120, 0x38f87a0, 0x396f426, 0x29f1b1c, 0x7, 0x2a026f9, 0x11)
	/home/user/Documentos/CDGIN/2017/ESTALEIRO/metricbeat/go-1.8.3/go/src/flag/flag.go:793 +0x420
flag.BoolVar(0x396f426, 0x29f1b1c, 0x7, 0x0, 0x2a026f9, 0x11)
	/home/user/Documentos/CDGIN/2017/ESTALEIRO/metricbeat/go-1.8.3/go/src/flag/flag.go:572 +0x72
github.com/elastic/beats/metricbeat/module/vsphere/vendor/github.com/vmware/vic/pkg/version.init.1()
	/home/user/monvm1/src/github.com/elastic/beats/metricbeat/module/vsphere/vendor/github.com/vmware/vic/pkg/version/version.go:56 +0x5c
github.com/elastic/beats/metricbeat/module/vsphere/vendor/github.com/vmware/vic/pkg/version.init()
	/home/user/monvm1/src/github.com/elastic/beats/metricbeat/module/vsphere/vendor/github.com/vmware/vic/pkg/version/version.go:146 +0x5d
github.com/elastic/beats/metricbeat/module/vsphere/vendor/github.com/vmware/vic/lib/config/executor.init()
	/home/user/monvm1/src/github.com/elastic/beats/metricbeat/module/vsphere/vendor/github.com/vmware/vic/lib/config/executor/network_interface.go:85 +0x58
github.com/elastic/beats/metricbeat/module/vsphere/vendor/github.com/vmware/vic/lib/config.init()
	/home/user/monvm1/src/github.com/elastic/beats/metricbeat/module/vsphere/vendor/github.com/vmware/vic/lib/config/virtual_container_host.go:351 +0x67
github.com/elastic/beats/metricbeat/module/vsphere/vendor/github.com/vmware/vic/pkg/vsphere/session.init()
	/home/user/monvm1/src/github.com/elastic/beats/metricbeat/module/vsphere/vendor/github.com/vmware/vic/pkg/vsphere/session/session.go:330 +0x8e
github.com/elastic/beats/metricbeat/module/vsphere/host.init()
	/home/user/monvm1/src/github.com/elastic/beats/metricbeat/module/vsphere/host/host.go:128 +0x6c
github.com/elastic/beats/metricbeat/include.init()
	/home/user/monvm1/src/github.com/elastic/beats/metricbeat/include/list.go:114 +0x1ec
github.com/elastic/beats/metricbeat/cmd.init()
	/home/user/monvm1/src/github.com/elastic/beats/metricbeat/cmd/root.go:30 +0x71
main.init()
	/home/user/monvm1/src/github.com/elastic/beats/metricbeat/main.go:21 +0x49

I needed to import "github.com/vmware/vic/pkg/vsphere/session" and added to vsphere vendor. I'm not sure how to resolve this. :-(

exekias · 2017-08-16T17:31:41Z

You can patch https://github.com/vmware/vic/blob/master/pkg/version/version.go#L56 from the vendor folder temporarily and keep going, then we can treat that issue once/if it's working. Try by changing "version" to something else

amandahla · 2017-08-16T19:32:58Z

Thanks @exekias I tested with both versions and it was fine. Now, when the session is deactivacte, after the keepalive time, he re-authenticates.

For this commit, I changed 'version' to 'version1' in https://github.com/vmware/vic/blob/master/pkg/version/version.go#L56

Log:

WARN[0060] session keepalive error: ServerFaultCode: The session is not authenticated. 
INFO[0060] session keepalive re-authenticated

alextxm · 2017-10-23T10:25:46Z

Hi @amandahla,
i'm testing the vsphere module in 6.0.0-rc1 since last week and the bug seems to be still present:

alextxm · 2017-10-26T07:06:03Z

Hi all,
i can confirm it still happens regurarly due to vsphere daily backup activities: vsphere closes connections, metricbeats then starts logging "unable to connect" errors and after a while swiches to "not authenticated" and gets stuck in such condition even if vsphere is availlable again. The only way to get metricbeat collect data again is to restart it.

amandahla · 2017-10-26T11:06:36Z

Hi @alextxm . The fix was not merged yet. Still working.
#4883

exekias · 2017-10-27T12:21:19Z

I gave a try to #4883 without success :(

I have another idea, what about initializing a new client on every fetch? That would be moving https://github.com/elastic/beats/blob/master/metricbeat/module/vsphere/virtualmachine/virtualmachine.go#L58 to the Fetch method. Any opinion on this @amandahla ?

alextxm · 2017-11-08T08:28:26Z

Hi all,
i understand this is still a WIP but is there any change to get a fix in time for 6.0-GA ? It would be really nice !
Thank you

amandahla · 2017-11-08T16:47:13Z

@exekias I'll try that and see if logout like it's used here really works fine.

amandahla · 2017-11-08T18:07:07Z

Still using the PR #4883

exekias added bug Metricbeat Metricbeat labels Jul 17, 2017

tsg added the Pioneer Program label Jul 25, 2017

exekias mentioned this issue Aug 14, 2017

Fix Metricbeat/vsphere - Error "datastore '*' not found" #4879 #4883

Merged

exekias closed this as completed in 550b81e Nov 28, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Metricbeat 6.0-alpha2 Vsphere module: not authenticated (auth not being reissued) #4673

Metricbeat 6.0-alpha2 Vsphere module: not authenticated (auth not being reissued) #4673

alextxm commented Jul 14, 2017

exekias commented Jul 17, 2017

alextxm commented Jul 20, 2017

exekias commented Jul 21, 2017

alextxm commented Aug 7, 2017

amandahla commented Aug 15, 2017 •

edited

Loading

exekias commented Aug 16, 2017 •

edited

Loading

amandahla commented Aug 16, 2017

exekias commented Aug 16, 2017

amandahla commented Aug 16, 2017

exekias commented Aug 16, 2017

amandahla commented Aug 16, 2017

alextxm commented Oct 23, 2017

alextxm commented Oct 26, 2017 •

edited

Loading

amandahla commented Oct 26, 2017

exekias commented Oct 27, 2017

alextxm commented Nov 8, 2017

amandahla commented Nov 8, 2017 •

edited

Loading

amandahla commented Nov 8, 2017 •

edited

Loading

Metricbeat 6.0-alpha2 Vsphere module: not authenticated (auth not being reissued) #4673

Metricbeat 6.0-alpha2 Vsphere module: not authenticated (auth not being reissued) #4673

Comments

alextxm commented Jul 14, 2017

exekias commented Jul 17, 2017

alextxm commented Jul 20, 2017

exekias commented Jul 21, 2017

alextxm commented Aug 7, 2017

amandahla commented Aug 15, 2017 • edited Loading

exekias commented Aug 16, 2017 • edited Loading

amandahla commented Aug 16, 2017

exekias commented Aug 16, 2017

amandahla commented Aug 16, 2017

exekias commented Aug 16, 2017

amandahla commented Aug 16, 2017

alextxm commented Oct 23, 2017

alextxm commented Oct 26, 2017 • edited Loading

amandahla commented Oct 26, 2017

exekias commented Oct 27, 2017

alextxm commented Nov 8, 2017

amandahla commented Nov 8, 2017 • edited Loading

amandahla commented Nov 8, 2017 • edited Loading

amandahla commented Aug 15, 2017 •

edited

Loading

exekias commented Aug 16, 2017 •

edited

Loading

alextxm commented Oct 26, 2017 •

edited

Loading

amandahla commented Nov 8, 2017 •

edited

Loading

amandahla commented Nov 8, 2017 •

edited

Loading