CORTX-33564: Fixed panic in rpc_session_cancelled() due to session state

[yatin-mahajan]

Problem:
While session was getting canceled some other thread is trying to terminate
that session and moves session state to M0_RPC_SESSION_TERMINATING, which
lead to hit assert M0_POST(session->s_sm.sm_state == M0_RPC_SESSION_IDLE)
in rpc_session_cancel().

#5 in m0_arch_panic (c=c@entry=0x7f7276a91b40 <__pctx.14289>, ap=ap@entry=0x7f7268c05ce0) at lib/user_space/uassert.c:131
#6 in m0_panic (ctx=ctx@entry=0x7f7276a91b40 <__pctx.14289>) at lib/assert.c:52
#7 in m0_rpc_session_cancel (session=session@entry=0x56283c7c13d8) at rpc/session.c:863
#8 in m0_rpc_conn_sessions_cancel (conn=conn@entry=0x56283c7c1030) at rpc/conn.c:1333
#9 in rpc_conn__on_service_event_cb (clink=) at rpc/conn.c:1364
#10 in clink_signal (clink=clink@entry=0x56283c7c12c0) at lib/chan.c:135
#11 in chan_signal_nr (chan=chan@entry=0x56283c6a8768, nr=1) at lib/chan.c:154
#12 in m0_chan_broadcast (chan=chan@entry=0x56283c6a8768) at lib/chan.c:174
#13 in ha_state_accept (ignore_same_state=1, note=0x7f7268c06060, confc=0x56283816b028) at ha/note.c:18

Solution:
It is possible that some other thread is trying to terminate the same session
while session is getting cancelled, only the IDLE/BUSY sessions are allowed to
cancel. Updated pre check to return from m0_rpc_cancel instead of panic/assert.
Also replaced M0_POST()/assert with proper debug log.

Signed-off-by: Yatin Mahajan yatin.mahajan@seagate.com

Problem Statement

• Problem statement

Design

• For Bug, Describe the fix here.
• For Feature, Post the link for design

Coding

Checklist for Author

• Coding conventions are followed and code is consistent

Testing

Checklist for Author

• Unit and System Tests are added
• Test Cases cover Happy Path, Non-Happy Path and Scalability
• Testing was performed with RPM

Impact Analysis

Checklist for Author/Reviewer/GateKeeper

• Interface change (if any) are documented
• Side effects on other features (deployment/upgrade)
• Dependencies on other component(s)

Review Checklist

Checklist for Author

• JIRA number/GitHub Issue added to PR
• PR is self reviewed
• Jira and state/status is updated and JIRA is updated with PR link
• Check if the description is clear and explained

Documentation

Checklist for Author

• Changes done to WIKI / Confluence page / Quick Start Guide

[ivan-alekhin]

The patch does not address the root cause: unprotected access to sm state. The patch, essentially, disables the preconditions. It means the situation described in the PR description is still possible: session is getting canceled while another thread is trying to terminate it. However, with this patch we won't see it because preconditions were removed. We will have a hidden race condition instead.
Please take a look at the comments.

Also, please include the backtraces of the issue into the commit message.

[madhavemuri]

Triggered https://eos-jenkins.colo.seagate.com/job/Cortx-PR-Build/job/Motr/1495/

[rkothiya]

Due to rate limit the ci/cd results did not get updated in the pr, but I have checked it had know failures :