Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[mgmt-framework]: service fails to start on 201911 #4291

Closed
nazariig opened this issue Mar 19, 2020 · 12 comments · Fixed by #4799
Closed

[mgmt-framework]: service fails to start on 201911 #4291

nazariig opened this issue Mar 19, 2020 · 12 comments · Fixed by #4799

Comments

@nazariig
Copy link
Collaborator

Description

Mgmt-framework service fails to start:

Mar 19 13:30:45.820590 sonic INFO mgmt-framework#supervisord: rest-server REST_SERVER_ARGS = -ui /rest_ui -logtostderr -cert /etc/sonic/telemetry/streamingtelemetryserver.cer -key /etc/sonic/telemetry/streamingtelemetryserver.key -cacert /etc/sonic/telemetry/dsmsroot.cer
Mar 19 13:30:45.830825 sonic INFO mgmt-framework#supervisord: rest-server ERROR: logging before flag.Parse: I0319 13:30:45.830099     423 util.go:161] Current Values of CVL Configuration File map[TRACE_CACHE:false TRACE_YPARSER:false __comment4__:Display log upto INFO level 8 TRACE_LIBYANG:false TRACE_SEMANTIC:false LOGTOSTDERR:false __comment3__:Display log upto INFO level TRACE_CREATE:false TRACE_UPDATE:false __comment1__:Log trace data when error occurs STDERRTHRESHOLD:ERROR SKIP_VALIDATION:false TRACE_DELETE:false TRACE_SYNTAX:false TRACE_ONERROR:true __comment2__:Set LOGTOSTDER to 'true' to log on standard error VERBOSITY:0 SKIP_SEMANTIC_VALIDATION:false]
Mar 19 13:30:45.831150 sonic INFO mgmt-framework#supervisord: rest-server ERROR: logging before flag.Parse: I0319 13:30:45.830414     423 util.go:161] Current Values of CVL Configuration File map[TRACE_CACHE:false TRACE_YPARSER:false __comment4__:Display log upto INFO level 8 TRACE_LIBYANG:false TRACE_SEMANTIC:false LOGTOSTDERR:false __comment3__:Display log upto INFO level TRACE_CREATE:false TRACE_UPDATE:false __comment1__:Log trace data when error occurs STDERRTHRESHOLD:ERROR SKIP_VALIDATION:false TRACE_DELETE:false TRACE_SYNTAX:false TRACE_ONERROR:true __comment2__:Set LOGTOSTDER to 'true' to log on standard error VERBOSITY:0 SKIP_SEMANTIC_VALIDATION:false]
Mar 19 13:30:45.889356 sonic INFO mgmt-framework#supervisord: rest-server libyang[0]: Failed to find "fec" as a sibling to "sonic-port:ifname".
Mar 19 13:30:46.250554 sonic INFO mgmt-framework#supervisord: rest-server Yang model List: [sonic-acl.yang sonic-common.yang sonic-extension.yang sonic-extensions.yang sonic-interface.yang sonic-port.yang openconfig-acl.yang openconfig-acl-annot.yang]
Mar 19 13:30:46.362128 sonic INFO mgmt-framework#supervisord: rest-server ERROR: logging before flag.Parse: I0319 13:30:46.361613     423 xspec.go:284] Module name(sonic-acl)
Mar 19 13:30:46.362222 sonic INFO mgmt-framework#supervisord: rest-server ERROR: logging before flag.Parse: I0319 13:30:46.361683     423 xspec.go:261] xDbSpecOrdTblMap after appending map[sonic-acl:[ACL_TABLE ACL_RULE]]
Mar 19 13:30:46.362561 sonic INFO mgmt-framework#supervisord: rest-server Xpath not found(/openconfig-acl:acl) #015
Mar 19 13:30:46.362630 sonic INFO mgmt-framework#supervisord: rest-server Xpath not found(/openconfig-acl:acl/acl-sets/acl-set) #015
Mar 19 13:30:46.362729 sonic INFO mgmt-framework#supervisord: rest-server Xpath not found(/openconfig-acl:acl/acl-sets/acl-set/config/name) #015
Mar 19 13:30:46.363039 sonic INFO mgmt-framework#supervisord: rest-server ERROR: logging before flag.Parse: I0319 13:30:46.361840     423 xspec.go:284] Module name(sonic-common)
Mar 19 13:30:46.363105 sonic INFO mgmt-framework#supervisord: rest-server ERROR: logging before flag.Parse: I0319 13:30:46.361872     423 xspec.go:261] xDbSpecOrdTblMap after appending map[sonic-acl:[ACL_TABLE ACL_RULE] sonic-common:[operation]]
Mar 19 13:30:46.363160 sonic INFO mgmt-framework#supervisord: rest-server ERROR: logging before flag.Parse: I0319 13:30:46.361913     423 xspec.go:284] Module name(sonic-port)
Mar 19 13:30:46.363206 sonic INFO mgmt-framework#supervisord: rest-server ERROR: logging before flag.Parse: I0319 13:30:46.361942     423 xspec.go:261] xDbSpecOrdTblMap after appending map[sonic-acl:[ACL_TABLE ACL_RULE] sonic-common:[operation] sonic-port:[PORT]]
Mar 19 13:30:46.363253 sonic INFO mgmt-framework#supervisord: rest-server ERROR: logging before flag.Parse: I0319 13:30:46.362016     423 xspec.go:284] Module name(sonic-interface)
Mar 19 13:30:46.363306 sonic INFO mgmt-framework#supervisord: rest-server ERROR: logging before flag.Parse: I0319 13:30:46.362046     423 xspec.go:261] xDbSpecOrdTblMap after appending map[sonic-acl:[ACL_TABLE ACL_RULE] sonic-common:[operation] sonic-port:[PORT] sonic-interface:[INTERFACE]]
Mar 19 13:30:46.363962 sonic INFO mgmt-framework#supervisord: rest-server Xpath not found(/openconfig-acl:acl/acl-sets/acl-set/config/type) #015
Mar 19 13:30:46.363962 sonic INFO mgmt-framework#supervisord: rest-server Xpath not found(/openconfig-acl:acl/acl-sets/acl-set/config/description) #015
Mar 19 13:30:46.363962 sonic INFO mgmt-framework#supervisord: rest-server Xpath not found(/openconfig-acl:acl/acl-sets/acl-set/state/name) #015
Mar 19 13:30:46.363962 sonic INFO mgmt-framework#supervisord: rest-server Xpath not found(/openconfig-acl:acl/acl-sets/acl-set/state/type) #015
Mar 19 13:30:46.363962 sonic INFO mgmt-framework#supervisord: rest-server Xpath not found(/openconfig-acl:acl/acl-sets/acl-set/state/description) #015
Mar 19 13:30:46.364005 sonic INFO mgmt-framework#supervisord: rest-server Xpath not found(/openconfig-acl:acl/acl-sets/acl-set/acl-entries/acl-entry) #015
Mar 19 13:30:46.364005 sonic INFO mgmt-framework#supervisord: rest-server Xpath not found(/openconfig-acl:acl/acl-sets/acl-set/acl-entries/acl-entry/ipv4) #015
Mar 19 13:30:46.364037 sonic INFO mgmt-framework#supervisord: rest-server Xpath not found(/openconfig-acl:acl/acl-sets/acl-set/acl-entries/acl-entry/ipv6) #015
Mar 19 13:30:46.364037 sonic INFO mgmt-framework#supervisord: rest-server Xpath not found(/openconfig-acl:acl/acl-sets/acl-set/acl-entries/acl-entry/config/sequence-id) #015
Mar 19 13:30:46.364063 sonic INFO mgmt-framework#supervisord: rest-server Xpath not found(/openconfig-acl:acl/acl-sets/acl-set/acl-entries/acl-entry/ipv4/config/source-address) #015
Mar 19 13:30:46.364063 sonic INFO mgmt-framework#supervisord: rest-server Xpath not found(/openconfig-acl:acl/acl-sets/acl-set/acl-entries/acl-entry/ipv4/config/destination-address) #015
Mar 19 13:30:46.364111 sonic INFO mgmt-framework#supervisord: rest-server Xpath not found(/openconfig-acl:acl/acl-sets/acl-set/acl-entries/acl-entry/ipv4/config/protocol) #015
Mar 19 13:30:46.364111 sonic INFO mgmt-framework#supervisord: rest-server Xpath not found(/openconfig-acl:acl/acl-sets/acl-set/acl-entries/acl-entry/ipv4/state/source-address) #015
Mar 19 13:30:46.364148 sonic INFO mgmt-framework#supervisord: rest-server Xpath not found(/openconfig-acl:acl/acl-sets/acl-set/acl-entries/acl-entry/ipv4/state/destination-address) #015
Mar 19 13:30:46.364148 sonic INFO mgmt-framework#supervisord: rest-server Xpath not found(/openconfig-acl:acl/acl-sets/acl-set/acl-entries/acl-entry/ipv4/state/protocol) #015
Mar 19 13:30:46.364175 sonic INFO mgmt-framework#supervisord: rest-server Xpath not found(/openconfig-acl:acl/acl-sets/acl-set/acl-entries/acl-entry/ipv6/config/source-address) #015
Mar 19 13:30:46.364193 sonic INFO mgmt-framework#supervisord: rest-server Xpath not found(/openconfig-acl:acl/acl-sets/acl-set/acl-entries/acl-entry/ipv6/config/destination-address) #015
Mar 19 13:30:46.364193 sonic INFO mgmt-framework#supervisord: rest-server Xpath not found(/openconfig-acl:acl/acl-sets/acl-set/acl-entries/acl-entry/ipv6/config/protocol) #015
Mar 19 13:30:46.364219 sonic INFO mgmt-framework#supervisord: rest-server Xpath not found(/openconfig-acl:acl/acl-sets/acl-set/acl-entries/acl-entry/transport/config/source-port) #015
Mar 19 13:30:46.364219 sonic INFO mgmt-framework#supervisord: rest-server Xpath not found(/openconfig-acl:acl/acl-sets/acl-set/acl-entries/acl-entry/transport/config/destination-port) #015
Mar 19 13:30:46.364259 sonic INFO mgmt-framework#supervisord: rest-server Xpath not found(/openconfig-acl:acl/acl-sets/acl-set/acl-entries/acl-entry/transport/state/source-port) #015
Mar 19 13:30:46.364259 sonic INFO mgmt-framework#supervisord: rest-server Xpath not found(/openconfig-acl:acl/acl-sets/acl-set/acl-entries/acl-entry/transport/state/destination-port) #015
Mar 19 13:30:46.364280 sonic INFO mgmt-framework#supervisord: rest-server Xpath not found(/openconfig-acl:acl/acl-sets/acl-set/acl-entries/acl-entry/transport/config/tcp-flags) #015
Mar 19 13:30:46.364303 sonic INFO mgmt-framework#supervisord: rest-server Xpath not found(/openconfig-acl:acl/acl-sets/acl-set/acl-entries/acl-entry/l2/config/ethertype) #015
Mar 19 13:30:46.364303 sonic INFO mgmt-framework#supervisord: rest-server Xpath not found(/openconfig-acl:acl/acl-sets/acl-set/acl-entries/acl-entry/actions/config/forwarding-action) #015
Mar 19 13:30:46.364326 sonic INFO mgmt-framework#supervisord: rest-server Xpath not found(/openconfig-acl:acl/acl-sets/acl-set/acl-entries/acl-entry/actions/state/forwarding-action) #015
Mar 19 13:30:46.364343 sonic INFO mgmt-framework#supervisord: rest-server Xpath not found(/openconfig-acl:acl/interfaces) #015
Mar 19 13:30:46.429063 sonic INFO mgmt-framework#supervisord: rest-server ERROR: logging before flag.Parse: I0319 13:30:46.428538     423 app_interface.go:82] Registering for path =/openconfig-acl:acl
Mar 19 13:30:46.429152 sonic INFO mgmt-framework#supervisord: rest-server ERROR: logging before flag.Parse: I0319 13:30:46.428586     423 app_interface.go:82] Registering for path =/sonic-
Mar 19 13:30:46.429200 sonic INFO mgmt-framework#supervisord: rest-server ERROR: logging before flag.Parse: I0319 13:30:46.428609     423 app_interface.go:82] Registering for path =*
Mar 19 13:30:46.429247 sonic INFO mgmt-framework#supervisord: rest-server ERROR: logging before flag.Parse: I0319 13:30:46.428639     423 intf_app.go:93] Init called for INTF module
Mar 19 13:30:46.429463 sonic INFO mgmt-framework#supervisord: rest-server ERROR: logging before flag.Parse: I0319 13:30:46.428662     423 app_interface.go:82] Registering for path =/openconfig-interfaces:interfaces
Mar 19 13:30:46.429525 sonic INFO mgmt-framework#supervisord: rest-server ERROR: logging before flag.Parse: I0319 13:30:46.428693     423 lldp_app.go:61] Init called for LLDP modules module
Mar 19 13:30:46.429573 sonic INFO mgmt-framework#supervisord: rest-server ERROR: logging before flag.Parse: I0319 13:30:46.428717     423 app_interface.go:82] Registering for path =/openconfig-lldp:lldp
Mar 19 13:30:46.429630 sonic INFO mgmt-framework#supervisord: rest-server ERROR: logging before flag.Parse: I0319 13:30:46.428741     423 pfm_app.go:44] Init called for Platform module
Mar 19 13:30:46.429676 sonic INFO mgmt-framework#supervisord: rest-server ERROR: logging before flag.Parse: I0319 13:30:46.428763     423 app_interface.go:82] Registering for path =/openconfig-platform:components
Mar 19 13:30:46.807895 sonic INFO mgmt-framework#supervisord: rest-server ERROR: logging before flag.Parse: I0319 13:30:46.795182     423 sys_app.go:47] SysApp: Init called for System module
Mar 19 13:30:46.807895 sonic INFO mgmt-framework#supervisord: rest-server ERROR: logging before flag.Parse: I0319 13:30:46.795266     423 app_interface.go:82] Registering for path =/openconfig-system:system
Mar 19 13:30:46.812676 sonic INFO mgmt-framework#supervisord: rest-server 2020/03/19 13:30:46 profile: cpu profiling enabled, /tmp/profile662473775/cpu.pprof
Mar 19 13:30:46.813676 sonic INFO mgmt-framework#supervisord: rest-server I0319 13:30:46.812561     423 router.go:79] Server has 2909 paths
Mar 19 13:30:47.829133 sonic INFO mgmt-framework#supervisord: rest-server I0319 13:30:47.828378     423 main.go:132] Server certificate file: /etc/sonic/telemetry/streamingtelemetryserver.cer
Mar 19 13:30:47.829133 sonic INFO mgmt-framework#supervisord: rest-server I0319 13:30:47.828421     423 main.go:133] Server private key file: /etc/sonic/telemetry/streamingtelemetryserver.key
Mar 19 13:30:47.829133 sonic INFO mgmt-framework#supervisord: rest-server F0319 13:30:47.828468     423 main.go:137] Failed to load server cert/key -- open /etc/sonic/telemetry/streamingtelemetryserver.cer: no such file or directory

Steps to reproduce the issue:

  1. Install SONiC 201911
  2. Deploy t0 topo
  3. service mgmt-framework restart

Describe the results you received:
mgmt-framework fails to start

Describe the results you expected:
mgmt-framework service should be operational or disabled

Additional information you deem important (e.g. issue happens only occasionally):

**Output of `show version`:**
```
(paste your output here)
```

**Attach debug file `sudo generate_dump`:**
```
(paste your output here)
```

The issue is observed with Removing explicit libyang plugin path settings #4144 cherry-picked

@nazariig
Copy link
Collaborator Author

@dutta-partha please have a look

@dutta-partha
Copy link
Contributor

@nazariig REST server is failing to load the certificate file. Is it present in mgmt-framework docker ?
@sachinholla Please provide your input.

"Failed to load server cert/key -- open /etc/sonic/telemetry/streamingtelemetryserver.cer: no such file or directory"

@sachinholla
Copy link
Contributor

Hi @nazariig , both telemetry and mgmt-framework containers read server certificate configurations from DEVICE_METADATA['x509'] config_db entry. So, both containers should include same certificate files at same path. Please copy /etc/sonic/telemetry/streamingtelemetryserver.cer and /etc/sonic/telemetry/streamingtelemetryserver.key to mgmt-framework repo as well.

@lguohan
Copy link
Collaborator

lguohan commented Apr 1, 2020

@sachinholla , can you check? it is strange that mgmt-framework container tries to read streaming telmetry cert. is this by design?

@sachinholla
Copy link
Contributor

Yes, it is by design. Is DEVICE_METADATA['x509'] entry intended only for telemetry container? I don't think it is practical to have different server & CA certificates for different management services on one box. This must be the reason for having certificate configurations in a generic DEVICE_METADATA['x509'] table entry; while other telemetry service specific configurations (like port) in the TELEMETRY['gnmi'] table.

Only caveat is that certificate files should be present in both containers at same path. This can be achieved by maintaining them under host /etc/sonic.

@lguohan
Copy link
Collaborator

lguohan commented Apr 2, 2020

in this pr, the telemetry is not using its own cert. a0d213c

we are deprecating DEVICE_METADATA['x509'] as a cert location.

mgmt-framework should have it's own certs location, even if it decides use the same cert as telemtry. it should still looking for its own location.

@sachinholla
Copy link
Contributor

sachinholla commented Apr 2, 2020

we are deprecating DEVICE_METADATA['x509'] as a cert location.

Not aware of this enhancement.. We can make similar changes in mgmt-framework startup script also.

hi @nazariig -- if you have pulled the commit a0d213c, please move certificate configurations from DEVICE_METADATA['x509'] to TELEMETRY['certs']. It should resolve the error in mgmt-framework for now.

@ben-gale
Copy link
Collaborator

ben-gale commented Apr 7, 2020

in this pr, the telemetry is not using its own cert. a0d213c

we are deprecating DEVICE_METADATA['x509'] as a cert location.

mgmt-framework should have it's own certs location, even if it decides use the same cert as telemtry. it should still looking for its own location.

Guohan,

I'm not sure about this change you made. Following reasons: -

  • You say (earlier) "it is strange that mgmt-framework container tries to read streaming telemetry cert." - I don't think this is the right conclusion. The DEVICE_METADATA['x509'] was a common path reference, not a common location - the location is within each container, so each container (Mgmt FW, Telemetry) has it's own certs. So they were already de-coupled.
  • Changing the name to "TELEMETRY" doesn't seem to accomplish anything in particular. In fact it kind of requires the Mgmt FW to now add a similar database object(s) for itself to use (to reduce naming confusion), which means that there are now 2 tables to configure.
  • The one benefit of all this is that it allows Telemetry and Mgmt FW to use different paths for their certs - do you see a good use case for this?

Thanks,

Ben

@lguohan
Copy link
Collaborator

lguohan commented Apr 7, 2020

mgmt fw should not use certs in TELEMETRY. it should use its own certs in config db, for example, MGMT_FRAMEWORK['certs'].

The DEVICE_METADATA['x509'] was a common path reference, not a common location.

I do not know how this can be achieved.

I think the right way is to have
TELEMTRY['certs'], MGMG_FRAMEWORK['certs'] defined in the config db. They can point to same certs file or different depending on user's needs.

The one benefit of all this is that it allows Telemetry and Mgmt FW to use different paths for their certs - do you see a good use case for this?

Yes. MGMG_FW and TELEMTRY have different users, we would like authenticate the differently.

@ben-gale
Copy link
Collaborator

mgmt fw should not use certs in TELEMETRY. it should use its own certs in config db, for example, MGMT_FRAMEWORK['certs'].

[Ben]: It isn't - each docker has it's own certs file

The DEVICE_METADATA['x509'] was a common path reference, not a common location.

I do not know how this can be achieved.

[Ben]: The path reference is local to the respective docker

I think the right way is to have
TELEMTRY['certs'], MGMG_FRAMEWORK['certs'] defined in the config db. They can point to same certs file or different depending on user's needs.

The one benefit of all this is that it allows Telemetry and Mgmt FW to use different paths for their certs - do you see a good use case for this?

Yes. MGMG_FW and TELEMTRY have different users, we would like authenticate the differently.

[Ben]: As I said, they were already using different certs files - the use case in question is whether they should different paths in configuration.

Anyway, no big deal - we can go with your change and have separate path configuration for the 2 dockers.

@ben-gale
Copy link
Collaborator

What do we have to do to close this one? Is there an associated PR that makes Guohan's requested change?

@sachinholla
Copy link
Contributor

sachinholla commented Jun 17, 2020

@ben, schema change was discussed as part of Mgmt framework phase2 HLD. Was waiting for that approval. Now we have it and I will open a PR shortly.

sachinholla added a commit to sachinholla/sonic-buildimage that referenced this issue Jun 17, 2020
REST and telemetry servers were using "DEVICE_METADATA|x509" table for
server certificate configurations. This table has been deprecated now.
Enhanced REST server startup script to read server certificate file
path configurations from REST_SERVER table. Three more attributes -
server_crt, server_key and ca_crt are introduced as described in
sonic-net/SONiC#550.

For backard compatibility, certificate configurations are read from
old "DEVICE_METADATA|x509" table if they (server_crt, server_key and
ca_crt) are not present in REST_SERVER table.

Fixes bug sonic-net#4291

Signed-off-by: Sachin Holla <sachin.holla@broadcom.com>
@lguohan lguohan linked a pull request Jun 19, 2020 that will close this issue
lguohan pushed a commit that referenced this issue Jun 19, 2020
REST and telemetry servers were using "DEVICE_METADATA|x509" table for
server certificate configurations. This table has been deprecated now.
Enhanced REST server startup script to read server certificate file
path configurations from REST_SERVER table. Three more attributes -
server_crt, server_key and ca_crt are introduced as described in
sonic-net/SONiC#550.

For backard compatibility, certificate configurations are read from
old "DEVICE_METADATA|x509" table if they (server_crt, server_key and
ca_crt) are not present in REST_SERVER table.

Fixes bug #4291

Signed-off-by: Sachin Holla <sachin.holla@broadcom.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

6 participants