Refresh during Spire SVID updating

[glazychev-art]

Current Behavior

If the SVID is updated during the refresh, the connection may be closed.
This happens due to a falling authorization policy in authorizeClient on the NSC side.
Due to the fact that refresh does not have a retry, we may lose the connection until the next scheduled refresh.

Failure Information (for bugs)

Nov 10 12:54:54.456�[37m [TRAC] [id:alpine-c6166013-d8a9-47ab-98ca-0b5deb24c98f-0] [type:networkService] �[0m(18)                  ⎆ sdk/pkg/networkservice/common/authorize/authorizeClient.Request()
Nov 10 12:54:54.456�[37m [TRAC] [id:alpine-c6166013-d8a9-47ab-98ca-0b5deb24c98f-0] [type:networkService] �[0m(19)                   ⎆ sdk/pkg/networkservice/common/trimpath/trimpathClient.Request()
Nov 10 12:54:54.457�[37m [TRAC] [id:alpine-c6166013-d8a9-47ab-98ca-0b5deb24c98f-0] [type:networkService] �[0m(20)                    ⎆ sdk/pkg/networkservice/common/connect/connectClient.Request()
Nov 10 12:54:56.960�[37m [TRAC] [id:alpine-c6166013-d8a9-47ab-98ca-0b5deb24c98f-0] [type:networkService] �[0m(20.1)                      request-response-diff={"mechanism":{"parameters":{"inodeURL":"inode://4/4026532886"}},"path":{"path_segments":{"0":{"expires":{"nanos":457186472,"seconds":1699621494},"token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJzcGlmZmU6Ly9rOHMubnNtL25zL25zLWtlcm5lbDJldGhlcm5ldDJrZXJuZWwvcG9kL2FscGluZSIsImF1ZCI6WyJzcGlmZmU6Ly9rOHMubnNtL25zL25zbS1zeXN0ZW0vcG9kL25zbWdyLTU1Zzg1Il0sImV4cCI6MTY5OTYyMTQ5NH0.XmedRgWDz0AqHDuxOac9sDGfGSLnKoldj5PuDUojpkB03PoyQu0wX62UG5jfS4wBjG_rwfHJxLGHBp2am9KTZA"},"1":{"expires":{"nanos":569124225,"seconds":1699621494},"token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJzcGlmZmU6Ly9rOHMubnNtL25zL25zbS1zeXN0ZW0vcG9kL25zbWdyLTU1Zzg1IiwiYXVkIjpbInNwaWZmZTovL2s4cy5uc20vbnMvbnNtLXN5c3RlbS9wb2QvZm9yd2FyZGVyLXZwcC04bXdoYyJdLCJleHAiOjE2OTk2MjE0OTR9.XD7kyPG-m29cqzb5CFb-YEGkmtJgbqGkV-WAw88JDdue_-9LUkHfKgBQK0fnH3dT9jbSe185IG5HJmx1NAUILg"},"2":{"expires":{"nanos":669673112,"seconds":1699621494},"metrics":{"client_rx_bytes":"17687584","client_rx_packets":"183376","client_tx_bytes":"24289628","client_tx_packets":"183380","server_rx_bytes":"17688144","server_rx_packets":"183382","server_tx_bytes":"17687530","server_tx_packets":"183375"},"token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJzcGlmZmU6Ly9rOHMubnNtL25zL25zbS1zeXN0ZW0vcG9kL2ZvcndhcmRlci12cHAtOG13aGMiLCJhdWQiOlsic3BpZmZlOi8vazhzLm5zbS9ucy9uc20tc3lzdGVtL3BvZC9uc21nci1zNjltbSJdLCJleHAiOjE2OTk2MjE0OTR9.TD8av1EHmYr3JANEVAYvDIeD5BtfoO0hvZdCwPAJFHF30poiG51YTGweL1Q88YBeZulFBZKZaPfqW2ABlzHyDQ"},"3":{"expires":{"nanos":736934076,"seconds":1699621494},"token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJzcGlmZmU6Ly9rOHMubnNtL25zL25zbS1zeXN0ZW0vcG9kL25zbWdyLXM2OW1tIiwiYXVkIjpbInNwaWZmZTovL2s4cy5uc20vbnMvbnNtLXN5c3RlbS9wb2QvZm9yd2FyZGVyLXZwcC1scXFuOSJdLCJleHAiOjE2OTk2MjE0OTR9.5qs0tIam8ayHShq2Qx7VO_bQXsAblJCO8LLRLse0p8VjY8J3n_Imqp4m_c5iHNJvXR9Y-QkRyR5T-ACUJDj_Tg"},"4":{"expires":{"nanos":825719294,"seconds":1699621494},"metrics":{"client_rx_bytes":"17687752","client_rx_packets":"183378","client_tx_bytes":"17687752","client_tx_packets":"183378","server_rx_bytes":"17687948","server_rx_packets":"183380","server_tx_bytes":"24289254","server_tx_packets":"183377"},"token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJzcGlmZmU6Ly9rOHMubnNtL25zL25zbS1zeXN0ZW0vcG9kL2ZvcndhcmRlci12cHAtbHFxbjkiLCJhdWQiOlsic3BpZmZlOi8vazhzLm5zbS9ucy9ucy1rZXJuZWwyZXRoZXJuZXQya2VybmVsL3BvZC9uc2Uta2VybmVsLTU1NjRmODdmNmQtbHpjbnQiXSwiZXhwIjoxNjk5NjIxNDk0fQ.eXEt52p3pnR6RSTACY3qyGhoRlIhIFcB8aUh9u0T2-XM453qvoHKgdovXK8MAa1ZlRivl7IFTKri8Q1EvnnXVA"},"5":{"expires":{"nanos":827889494,"seconds":1699621494},"token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJzcGlmZmU6Ly9rOHMubnNtL25zL25zLWtlcm5lbDJldGhlcm5ldDJrZXJuZWwvcG9kL25zZS1rZXJuZWwtNTU2NGY4N2Y2ZC1semNudCIsImF1ZCI6WyJzcGlmZmU6Ly9rOHMubnNtL25zL25zbS1zeXN0ZW0vcG9kL2ZvcndhcmRlci12cHAtbHFxbjkiXSwiZXhwIjoxNjk5NjIxNDk0fQ.CUEK7RkomCvT3IrCSHPEiTqICV_8_VA4X3qC6s5EEHNAWh04ZLPiVQGvOllrPYqbeVFttlORvKq1l_WLVLNn6g"}}}}
Nov 10 12:54:56.964�[31m [ERRO] [id:alpine-c6166013-d8a9-47ab-98ca-0b5deb24c98f-0] [type:networkService] �[0m(18.1)                    policy failed: policies/client/next_token_signed.rego
Nov 10 12:54:56.964�[37m [TRAC] [id:alpine-c6166013-d8a9-47ab-98ca-0b5deb24c98f-0] [type:networkService] �[0m(18.2)                    close={"id":"alpine-c6166013-d8a9-47ab-98ca-0b5deb24c98f-0","network_service":"kernel2ethernet2kernel","mechanism":{"cls":"LOCAL","type":"KERNEL","parameters":{"inodeURL":"inode://4/4026532886","name":"nsm-1"}},"context":{"ip_context":{"src_ip_addrs":["172.16.1.101/32"],"dst_ip_addrs":["172.16.1.100/32"],"src_routes":[{"prefix":"172.16.1.100/32"}],"dst_routes":[{"prefix":"172.16.1.101/32"}],"excluded_prefixes":["10.0.0.1/32","10.0.0.10/32","10.0.24.187/32","10.0.33.22/32","10.0.138.210/32","10.0.197.237/32","10.0.223.17/32","10.244.0.0/23"]},"dns_context":{},"MTU":1446},"labels":{"nodeName":"aks-nodepool1-31996482-vmss000001","podName":"alpine"},"path":{"path_segments":[{"name":"alpine-c6166013-d8a9-47ab-98ca-0b5deb24c98f","id":"alpine-c6166013-d8a9-47ab-98ca-0b5deb24c98f-0","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJzcGlmZmU6Ly9rOHMubnNtL25zL25zLWtlcm5lbDJldGhlcm5ldDJrZXJuZWwvcG9kL2FscGluZSIsImF1ZCI6WyJzcGlmZmU6Ly9rOHMubnNtL25zL25zbS1zeXN0ZW0vcG9kL25zbWdyLTU1Zzg1Il0sImV4cCI6MTY5OTYyMTQ5NH0.XmedRgWDz0AqHDuxOac9sDGfGSLnKoldj5PuDUojpkB03PoyQu0wX62UG5jfS4wBjG_rwfHJxLGHBp2am9KTZA","expires":{"seconds":1699621494,"nanos":457186472}},{"name":"nsmgr-55g85","id":"765b146a-4881-4e41-bd1a-8e78f0f848f1","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJzcGlmZmU6Ly9rOHMubnNtL25zL25zbS1zeXN0ZW0vcG9kL25zbWdyLTU1Zzg1IiwiYXVkIjpbInNwaWZmZTovL2s4cy5uc20vbnMvbnNtLXN5c3RlbS9wb2QvZm9yd2FyZGVyLXZwcC04bXdoYyJdLCJleHAiOjE2OTk2MjE0OTR9.XD7kyPG-m29cqzb5CFb-YEGkmtJgbqGkV-WAw88JDdue_-9LUkHfKgBQK0fnH3dT9jbSe185IG5HJmx1NAUILg","expires":{"seconds":1699621494,"nanos":569124225}},{"name":"forwarder-vpp-8mwhc","id":"c52614d3-a9d3-4d4c-abac-6d851d8ae7e9","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJzcGlmZmU6Ly9rOHMubnNtL25zL25zbS1zeXN0ZW0vcG9kL2ZvcndhcmRlci12cHAtOG13aGMiLCJhdWQiOlsic3BpZmZlOi8vazhzLm5zbS9ucy9uc20tc3lzdGVtL3BvZC9uc21nci1zNjltbSJdLCJleHAiOjE2OTk2MjE0OTR9.TD8av1EHmYr3JANEVAYvDIeD5BtfoO0hvZdCwPAJFHF30poiG51YTGweL1Q88YBeZulFBZKZaPfqW2ABlzHyDQ","expires":{"seconds":1699621494,"nanos":669673112},"metrics":{"client_drops":"3","client_rx_bytes":"17687584","client_rx_packets":"183376","client_tx_bytes":"24289628","client_tx_packets":"183380","server_drops":"0","server_rx_bytes":"17688144","server_rx_packets":"183382","server_tx_bytes":"17687530","server_tx_packets":"183375"}},{"name":"nsmgr-s69mm","id":"817fc706-952f-40bc-b2ff-e326ac34925f","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJzcGlmZmU6Ly9rOHMubnNtL25zL25zbS1zeXN0ZW0vcG9kL25zbWdyLXM2OW1tIiwiYXVkIjpbInNwaWZmZTovL2s4cy5uc20vbnMvbnNtLXN5c3RlbS9wb2QvZm9yd2FyZGVyLXZwcC1scXFuOSJdLCJleHAiOjE2OTk2MjE0OTR9.5qs0tIam8ayHShq2Qx7VO_bQXsAblJCO8LLRLse0p8VjY8J3n_Imqp4m_c5iHNJvXR9Y-QkRyR5T-ACUJDj_Tg","expires":{"seconds":1699621494,"nanos":736934076}},{"name":"forwarder-vpp-lqqn9","id":"1a8b2904-8ee8-4ce5-b070-015193283bd6","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJzcGlmZmU6Ly9rOHMubnNtL25zL25zbS1zeXN0ZW0vcG9kL2ZvcndhcmRlci12cHAtbHFxbjkiLCJhdWQiOlsic3BpZmZlOi8vazhzLm5zbS9ucy9ucy1rZXJuZWwyZXRoZXJuZXQya2VybmVsL3BvZC9uc2Uta2VybmVsLTU1NjRmODdmNmQtbHpjbnQiXSwiZXhwIjoxNjk5NjIxNDk0fQ.eXEt52p3pnR6RSTACY3qyGhoRlIhIFcB8aUh9u0T2-XM453qvoHKgdovXK8MAa1ZlRivl7IFTKri8Q1EvnnXVA","expires":{"seconds":1699621494,"nanos":825719294},"metrics":{"client_drops":"3","client_rx_bytes":"17687752","client_rx_packets":"183378","client_tx_bytes":"17687752","client_tx_packets":"183378","server_drops":"0","server_rx_bytes":"17687948","server_rx_packets":"183380","server_tx_bytes":"24289254","server_tx_packets":"183377"}},{"name":"nse-kernel-5564f87f6d-lzcnt","id":"8331a433-736c-4da5-a288-03ee920b10c7","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJzcGlmZmU6Ly9rOHMubnNtL25zL25zLWtlcm5lbDJldGhlcm5ldDJrZXJuZWwvcG9kL25zZS1rZXJuZWwtNTU2NGY4N2Y2ZC1semNudCIsImF1ZCI6WyJzcGlmZmU6Ly9rOHMubnNtL25zL25zbS1zeXN0ZW0vcG9kL2ZvcndhcmRlci12cHAtbHFxbjkiXSwiZXhwIjoxNjk5NjIxNDk0fQ.CUEK7RkomCvT3IrCSHPEiTqICV_8_VA4X3qC6s5EEHNAWh04ZLPiVQGvOllrPYqbeVFttlORvKq1l_WLVLNn6g","expires":{"seconds":1699621494,"nanos":827889494}}]},"network_service_endpoint_name":"nse-kernel-5564f87f6d-lzcnt","payload":"ETHERNET"}

Steps to Reproduce

1. Run kind cluster
2. Set default_x509_svid_ttl, default_jwt_svid_ttl equal to 1m30s
3. Set NSM_REQUEST_TIMEOUT for the cmd-nsc and cmd-nsc-init equals to 2m
4. Add a chain element to the nsmgr that waits X509Source update. For example:

func (t *testSvidServer) Request(ctx context.Context, request *networkservice.NetworkServiceRequest) (*networkservice.Connection, error) {
	err := t.source.WaitUntilUpdated(ctx)
	if err != nil {
		return nil, err
	}
	resp, err := next.Server(ctx).Request(ctx, request)
...
}

5. Deploy nsm-system, Kernel2Kernel
6. See cmd-nsc-init logs