[chassis][202405]: orchagent crash in NotificationSwitchAsicSdkHealthEvent::executeCallback while handling SAI notification #19760

anamehra · 2024-08-01T06:08:00Z

Description

Intermittent orchagent crash seen on LCs during config reload while running sonic-mgmt nightly runs:

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/bin/orchagent -d /var/log/swss -b 1024 -s -f swss.asic1.rec -j sairedis.as'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x00007f3551c6e61b in ?? () from /lib/x86_64-linux-gnu/libc.so.6
[Current thread is 1 (Thread 0x7f3550c9e6c0 (LWP 181))]
(gdb) bt
#0 0x00007f3551c6e61b in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f3551c70908 in strftime_l () from /lib/x86_64-linux-gnu/libc.so.6
#2 0x00007f3551f4c90d in std::__timepunct::_M_put(char*, unsigned long, char const*, tm const*) const () from /lib/x86_64-linux-gnu/libstdc++.so.6
#3 0x00007f3551fa4fdf in std::time_put<char, std::ostreambuf_iterator<char, std::char_traits > >::do_put(std::ostreambuf_iterator<char, std::char_traits >, std::ios_base&, char, tm const*, char, char) const ()
from /lib/x86_64-linux-gnu/libstdc++.so.6
#4 0x00007f3551fa36fb in std::time_put<char, std::ostreambuf_iterator<char, std::char_traits > >::put(std::ostreambuf_iterator<char, std::char_traits >, std::ios_base&, char, tm const*, char const*, char const*) const ()
from /lib/x86_64-linux-gnu/libstdc++.so.6
#5 0x00005623dc9812de in ?? ()
#6 0x00005623dc7d4302 in ?? ()
#7 0x00007f35529902d2 in sairedis::NotificationSwitchAsicSdkHealthEvent::executeCallback(_sai_switch_notifications_t const&) const () from /lib/x86_64-linux-gnu/libsaimeta.so.0
#8 0x00007f3552aa34b2 in sairedis::RedisRemoteSaiInterface::handleNotification(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::vector<std::pair<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::__cxx11::basic_string<char, std::char_traits, std::allocator > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::__cxx11::basic_string<char, std::char_traits, std::allocator > > > > const&) () from /lib/x86_64-linux-gnu/libsairedis.so.0
#9 0x00007f3552af2878 in sairedis::RedisChannel::notificationThreadFunction() () from /lib/x86_64-linux-gnu/libsairedis.so.0
#10 0x00007f3551f584a3 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#11 0x00007f3551c2d134 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#12 0x00007f3551cad7dc in ?? () from /lib/x86_64-linux-gnu/libc.so.6

syslogs:
orchagent_segfault.log

The following SAI notification from SDK triggers the crash:

{"category":"SAI_SWITCH_ASIC_SDK_HEALTH_CATEGORY_ASIC_HW","data.data_type":"SAI_HEALTH_DATA_TYPE_GENERAL","description":"16:123,10,9,34,100,97,116,97,34,58,32,34,48,34,10,125","severity":"SAI_SWITCH_ASIC_SDK_HEALTH_SEVERITY_WARNING","switch_id":"oid:0x21000000000000","timestamp":"{\"tv_nsec\":\"12\",\"tv_sec\":\"172479515853275099\"}"}|

SAI redis recoreds before the above notification related to this feature:

2024-08-28.05:52:38.164808|q|attribute_capability|SAI_OBJECT_TYPE_SWITCH:oid:0x21000000000000|OBJECT_TYPE=SAI_OBJECT_TYPE_SWITCH|ATTR_ID=SAI_SWITCH_ATTR_REG_WARNING_SWITCH_ASIC_SDK_HEALTH_CATEGORY
2024-08-28.05:52:38.165174|Q|attribute_capability|SAI_STATUS_SUCCESS|OBJECT_TYPE=SAI_OBJECT_TYPE_SWITCH|ATTR_ID=SAI_SWITCH_ATTR_REG_WARNING_SWITCH_ASIC_SDK_HEALTH_CATEGORY|CREATE_IMP=true|SET_IMP=true|GET_IMP=true
2024-08-28.05:52:38.165239|s|SAI_OBJECT_TYPE_SWITCH:oid:0x21000000000000|SAI_SWITCH_ATTR_REG_WARNING_SWITCH_ASIC_SDK_HEALTH_CATEGORY=4:SAI_SWITCH_ASIC_SDK_HEALTH_CATEGORY_SW,SAI_SWITCH_ASIC_SDK_HEALTH_CATEGORY_FW,SAI_SWITCH_ASIC_SDK_HEALTH_CATEGORY_CPU_HW,SAI_SWITCH_ASIC_SDK_HEALTH_CATEGORY_ASIC_HW
2024-08-28.05:52:38.165651|q|attribute_capability|SAI_OBJECT_TYPE_SWITCH:oid:0x21000000000000|OBJECT_TYPE=SAI_OBJECT_TYPE_SWITCH|ATTR_ID=SAI_SWITCH_ATTR_REG_NOTICE_SWITCH_ASIC_SDK_HEALTH_CATEGORY
2024-08-28.05:52:38.165976|Q|attribute_capability|SAI_STATUS_SUCCESS|OBJECT_TYPE=SAI_OBJECT_TYPE_SWITCH|ATTR_ID=SAI_SWITCH_ATTR_REG_NOTICE_SWITCH_ASIC_SDK_HEALTH_CATEGORY|CREATE_IMP=true|SET_IMP=true|GET_IMP=true
2024-08-28.05:52:38.166036|s|SAI_OBJECT_TYPE_SWITCH:oid:0x21000000000000|SAI_SWITCH_ATTR_REG_NOTICE_SWITCH_ASIC_SDK_HEALTH_CATEGORY=4:SAI_SWITCH_ASIC_SDK_HEALTH_CATEGORY_SW,SAI_SWITCH_ASIC_SDK_HEALTH_CATEGORY_FW,SAI_SWITCH_ASIC_SDK_HEALTH_CATEGORY_CPU_HW,SAI_SWITCH_ASIC_SDK_HEALTH_CATEGORY_ASIC_HW

Steps to reproduce the issue:

Describe the results you received:

Describe the results you expected:

Output of `show version`:

commit sha branch 202405: e482a56148a8042638feb2073bd5810900711e24

Output of `show techsupport`:

(paste your output here or download and attach the file here )

Additional information you deem important (e.g. issue happens only occasionally):

The text was updated successfully, but these errors were encountered:

rlhui · 2024-08-14T17:28:43Z

seems to be triggerred from new feature - sairedis::NotificationSwitchAsicSdkHealthEvent::executeCallback

rlhui · 2024-08-14T17:29:09Z

@anamehra will debug more

arlakshm · 2024-09-11T17:48:24Z

@anamehra to triage more with @kcudnik to get more info.

arlakshm · 2024-11-13T18:41:37Z

lower priority for 202405. Need to debug for master.

kcudnik · 2024-11-14T08:43:51Z

Will investigate, in the backtrack number 5 and 6 are OA code? Symbols not loaded?

anamehra · 2024-11-14T18:34:48Z

If I remember correctly, 5 and 6 pointed somewhere in data segment pointing to, I think, notification function pointers. I loaded symbol files dbg debians for swss, libsairedis and meta but those did not help.

kcudnik · 2024-11-14T19:12:57Z

5 0x00005623dc9812de in ?? ()
6 0x00005623dc7d4302 in ?? ()
7 0x00007f35529902d2 in sairedis::NotificationSwitchAsicSdkHealthEvent::executeCallback(_sai_switch_notifications_t const&) const () from /lib/x86_64-linux-gnu/libsaimeta.so.0

callback execution on executeCallback suggests that deserialization succeeded correctly on the string, and later on crash happens on ::put, im invaestigating OA code

202405 which SAI is using ?

just tested on sairedis callback execution and it works fine

kcudnik · 2024-11-14T19:41:03Z

i reproduced that locally, problem is this line:
https://github.com/sonic-net/sonic-swss/blob/master/orchagent/switchorch.cpp#L1120

time_ss << std::put_time(std::localtime(&t), "%Y-%m-%d %H:%M:%S");

value tv_sec 172479515853275099 is too big and it crashes, thats about 5469289569 years, this is not correct
since this value is serialized, it could be due device returned it wrong, that's probably vendor issue with sai_timestamp_t and we should handle possible error in SWSS

anamehra · 2024-11-14T19:49:21Z

We hit the issue on 202405 which is using SAI 1.14
Our vendor SAI is at 1.13
Is there any compatibility issue between the two?

kcudnik · 2024-11-14T19:52:50Z

no, there should not be compatibility issue with v1.13 and v1.14, can you check how timestamp is populated ?

kcudnik · 2024-11-14T19:55:58Z

commit sonic-net/sonic-swss#3020 + @stephenxs can you take a look and handle this test case ? it's your code, and i made repro

static void on_switch_asic_sdk_health_event(
        _In_ sai_object_id_t switch_id,
        _In_ sai_switch_asic_sdk_health_severity_t severity,
        _In_ sai_timespec_t timestamp,
        _In_ sai_switch_asic_sdk_health_category_t category,
        _In_ sai_switch_health_data_t data,
        _In_ const sai_u8_list_t description)
{
    SWSS_LOG_ENTER();

    printf("swtch: %lx %ld %d\n", switch_id, timestamp.tv_sec,timestamp.tv_nsec);
    const std::time_t &t = (std::time_t)timestamp.tv_sec;
    std::stringstream time_ss;
    time_ss << std::put_time(std::localtime(&t), "%Y-%m-%d %H:%M:%S");
}

TEST(NotificationSwitchAsicSdkHealthEvent, executeCallback)
{
    sai_switch_notifications_t ntfs;
    auto str = "{\"category\":\"SAI_SWITCH_ASIC_SDK_HEALTH_CATEGORY_ASIC_HW\",\"data.data_type\":\"SAI_HEALTH_DATA_TYPE_GENERAL\",\"description\":\"16:123,10,9,34,100,97,116,97,34,58,32,34,48,34,10,125\",\"severity\":\"SAI_SWITCH_ASIC_SDK_HEALTH_SEVERITY_WARNING\",\"switch_id\":\"oid:0x21000000000000\",\"timestamp\":\"{\\\"tv_nsec\\\":\\\"12\\\",\\\"tv_sec\\\":\\\"172479515853275099\\\"}\"}";

    NotificationSwitchAsicSdkHealthEvent nn(str);

    ntfs.on_switch_asic_sdk_health_event = &on_switch_asic_sdk_health_event;

    nn.executeCallback(ntfs);
}

anamehra · 2024-11-14T19:58:55Z

Thanks @kcudnik ! I will check and fix the vendor side code.
This github issue could be used to handle the OA side crash.

stephenxs · 2024-11-14T22:58:59Z

I would like to add an exception handler here

stephenxs · 2024-11-15T02:18:45Z

172479515853275099

This is a segfault which is handled outside cpp try-catch scope.
I would like to handle this case specifically as follows
use the current date as the timestamp if the timestamp passed by vendor SAI is beyond 1 year from now.
@kcudnik @anamehra what do you think?

kcudnik · 2024-11-16T10:58:44Z

ok,i think what could be also problem that causes this time sec to be so big, which SAI headers are used in swss at 202405 ? since sairedis i using v1.14, if swss is using later headers then this was breaking change:
https://github.com/opencomputeproject/SAI/pull/1993/files#diff-444febffce442e4277a1ced22d1ab94df045dfaa10a1b671ee35a7e5c2d9f683R1950
(opencomputeproject/SAI#1993)

since it's added extra field sai_health_data_t data, which looks like that:

typedef struct _sai_switch_health_data_t
{
    /** Type of switch health data */
    sai_health_data_type_t data_type;
    /** @passparam data_type */
    sai_health_data_t data;
} sai_switch_health_data_t;

but that's after v1.14

on v1.14 it looks like this:

typedef struct _sai_switch_health_data_t
{
    /** Type of switch health data */
    sai_health_data_type_t data_type;
} sai_switch_health_data_t;

which changes struct size, and since this struct is used in param to callback:

3297 typedef void (*sai_switch_asic_sdk_health_event_notification_fn)(
3298         _In_ sai_object_id_t switch_id,
3299         _In_ sai_switch_asic_sdk_health_severity_t severity,
3300         _In_ sai_timespec_t timestamp,
3301         _In_ sai_switch_asic_sdk_health_category_t category,
3302         _In_ sai_switch_health_data_t data, /// <-here
3303         _In_ const sai_u8_list_t description);

since sizeof (data) have 2 different sizes on 2 different SAI versions, but since stacks is pushed in reverse (or in x64 is put in registers) this could lead to load wrong value to the specific register,i will investigate what could be changed

stephenxs · 2024-11-17T01:57:23Z

ok,i think what could be also problem that causes this time sec to be so big, which SAI headers are used in swss at 202405 ? since sairedis i using v1.14, if swss is using later headers then this was breaking change: https://github.com/opencomputeproject/SAI/pull/1993/files#diff-444febffce442e4277a1ced22d1ab94df045dfaa10a1b671ee35a7e5c2d9f683R1950 (opencomputeproject/SAI#1993)

since it's added extra field sai_health_data_t data, which looks like that:
typedef struct _sai_switch_health_data_t
{
    /** Type of switch health data */
    sai_health_data_type_t data_type;
    /** @passparam data_type */
    sai_health_data_t data;
} sai_switch_health_data_t;
but that's after v1.14

on v1.14 it looks like this:
typedef struct _sai_switch_health_data_t
{
    /** Type of switch health data */
    sai_health_data_type_t data_type;
} sai_switch_health_data_t;
which changes struct size, and since this struct is used in param to callback:
3297 typedef void (*sai_switch_asic_sdk_health_event_notification_fn)(
3298         _In_ sai_object_id_t switch_id,
3299         _In_ sai_switch_asic_sdk_health_severity_t severity,
3300         _In_ sai_timespec_t timestamp,
3301         _In_ sai_switch_asic_sdk_health_category_t category,
3302         _In_ sai_switch_health_data_t data, /// <-here
3303         _In_ const sai_u8_list_t description);
since sizeof (data) have 2 different sizes on 2 different SAI versions, but since stacks is pushed in reverse (or in x64 is put in registers) this could lead to load wrong value to the specific register,i will investigate what could be changed

swss does not specify SAI header. It references the SAI header specified in sairedis.
But vendor SAI can have its own SAI header. I think it can cause an error if SAI header differs between sairedis and vendor SAI.

kcudnik · 2024-11-17T13:32:42Z

@anamehra @stephenxs i performed such a test:

typedef struct _sai_switch_health_data_t_v14
{
    /** Type of switch health data */
    sai_health_data_type_t data_type;

    /** @passparam data_type */
    sai_health_data_t data;
} sai_switch_health_data_t_v14;

typedef struct _sai_switch_health_data_t_v13
{
    /** Type of switch health data */
    sai_health_data_type_t data_type;
} sai_switch_health_data_t_v13;

typedef void (*on_switch_asic_sdk_health_event_v13_fn)(
        _In_ sai_object_id_t switch_id,
        _In_ sai_switch_asic_sdk_health_severity_t severity,
        _In_ sai_timespec_t timestamp,
        _In_ sai_switch_asic_sdk_health_category_t category,
        _In_ sai_switch_health_data_t_v13 data,
        _In_ const sai_u8_list_t description);

static void on_switch_asic_sdk_health_event_v13(
        _In_ sai_object_id_t switch_id,
        _In_ sai_switch_asic_sdk_health_severity_t severity,
        _In_ sai_timespec_t timestamp,
        _In_ sai_switch_asic_sdk_health_category_t category,
        _In_ sai_switch_health_data_t_v13 data,
        _In_ const sai_u8_list_t description)
{
    SWSS_LOG_ENTER();

    printf("swtch v14: %lx %ld %d, count = %d\n", switch_id, timestamp.tv_sec,timestamp.tv_nsec,
            description.count);

    EXPECT_TRUE(timestamp.tv_sec == 1731848814);

    const std::time_t &t = (std::time_t)timestamp.tv_sec;
    std::stringstream time_ss;
    time_ss << std::put_time(std::localtime(&t), "%Y-%m-%d %H:%M:%S");
}

typedef void (*on_switch_asic_sdk_health_event_v14_fn)(
        _In_ sai_object_id_t switch_id,
        _In_ sai_switch_asic_sdk_health_severity_t severity,
        _In_ sai_timespec_t timestamp,
        _In_ sai_switch_asic_sdk_health_category_t category,
        _In_ sai_switch_health_data_t_v14 data,
        _In_ const sai_u8_list_t description);
static void on_switch_asic_sdk_health_event_v14(
        _In_ sai_object_id_t switch_id,
        _In_ sai_switch_asic_sdk_health_severity_t severity,
        _In_ sai_timespec_t timestamp,
        _In_ sai_switch_asic_sdk_health_category_t category,
        _In_ sai_switch_health_data_t_v14 data,
        _In_ const sai_u8_list_t description)
{
    SWSS_LOG_ENTER();

    printf("swtch v14: %lx %ld %d, count = %d\n", switch_id, timestamp.tv_sec,timestamp.tv_nsec,
            description.count);

    EXPECT_TRUE(timestamp.tv_sec == 1731848814);

    const std::time_t &t = (std::time_t)timestamp.tv_sec;
    std::stringstream time_ss;
    time_ss << std::put_time(std::localtime(&t), "%Y-%m-%d %H:%M:%S");
}

TEST(NotificationSwitchAsicSdkHealthEvent, version)
{
    SWSS_LOG_ENTER();

    sai_object_id_t switchId = 0x21000000000000;
    sai_switch_asic_sdk_health_severity_t severity = SAI_SWITCH_ASIC_SDK_HEALTH_SEVERITY_WARNING;
    sai_switch_asic_sdk_health_category_t category = SAI_SWITCH_ASIC_SDK_HEALTH_CATEGORY_ASIC_HW;
    sai_timespec_t timestamp = {1731848814, 123456};
    sai_switch_health_data_t_v13 data13 = { };
    sai_switch_health_data_t_v14 data14 = { };
    data13.data_type = SAI_HEALTH_DATA_TYPE_GENERAL;
    data14.data_type = SAI_HEALTH_DATA_TYPE_GENERAL;

    uint8_t list[] = {123,10,9,34,100,97,116,97,34,58,32,34,48,34,10,125};
    sai_u8_list_t desc;
    desc.count = 16;
    desc.list = list;

    {
        on_switch_asic_sdk_health_event_v13_fn v13 = &on_switch_asic_sdk_health_event_v13;
        v13(switchId, severity, timestamp, category, data13, desc); // v13 call v13

        on_switch_asic_sdk_health_event_v14_fn v14 = &on_switch_asic_sdk_health_event_v14;
        v14(switchId, severity, timestamp, category, data14, desc); // v14 call v14
    }

    {
        printf("call other\n");

        on_switch_asic_sdk_health_event_v13_fn v13 = reinterpret_cast<on_switch_asic_sdk_health_event_v13_fn>((void*)&on_switch_asic_sdk_health_event_v14);
        v13(switchId, severity, timestamp, category, data13, desc); // v13 call v14

        on_switch_asic_sdk_health_event_v14_fn v14 = reinterpret_cast<on_switch_asic_sdk_health_event_v14_fn>((void*)&on_switch_asic_sdk_health_event_v13);
        v14(switchId, severity, timestamp, category, data14, desc); // v14 call v13
    }
}

and results are like this:

swtch v13: 21000000000000 1731848814 123456, count = 16
swtch v14: 21000000000000 1731848814 123456, count = 16
call other
swtch v13: 21000000000000 1731848814 123456, count = -118218192
swtch v14: 21000000000000 1731848814 123456, count = 0
[       OK ] NotificationSwitchAsicSdkHealthEvent.version (4 ms)

so the timestamp is correct event when v13 calls v14 and viceversa, but the count is atully affected, because of sizeof(data) depending of version

for if in your case crash is in time/put and not deserialization of description, that means the headers are probably ok, but what's most likely is that device is returning wrong number of tv_sec in timestamp

anamehra · 2024-11-17T21:05:40Z

Hi @kcudnik , I figured out an issue with the timestamp in our platform SAI code for this API. I am working on fixing it. I think that should fix the timestamp issue. On Sonic side, we may add handling to prevent crash.

kcudnik · 2024-11-18T09:41:41Z

Hi @kcudnik , I figured out an issue with the timestamp in our platform SAI code for this API. I am working on fixing it. I think that should fix the timestamp issue. On Sonic side, we may add handling to prevent crash.

@stephenxs can you add exception handling for this in swss as you mentioned before?

stephenxs · 2024-11-18T10:10:48Z

Hi @kcudnik , I figured out an issue with the timestamp in our platform SAI code for this API. I am working on fixing it. I think that should fix the timestamp issue. On Sonic side, we may add handling to prevent crash.

@stephenxs can you add exception handling for this in swss as you mentioned before?

Sure. will do it.

kcudnik · 2024-11-18T10:58:37Z

@stephenxs i noticed taht put_time crashes after the tv_sec is around 67767710400000001, which causes to print Year over 2^31, which overlaps year and some internal crash, unfortunetly it does not throw any std exception and just segfault

#0  0x00007f017ae9b184 in __strftime_internal (s=0x7ffe0beb0e30 "\006", maxsize=128, format=0x7ffe0beb0e2c "%Y", tp=0x0, yr_spec=yr_spec@entry=0, tzset_called=tzset_called@entry=0x7ffe0beb0de7,
    loc=0x7f017afae960 <_nl_C_locobj>) at strftime_l.c:471
471     strftime_l.c: No such file or directory.
(gdb) bt
#0  0x00007f017ae9b184 in __strftime_internal (s=0x7ffe0beb0e30 "\006", maxsize=128, format=0x7ffe0beb0e2c "%Y", tp=0x0, yr_spec=yr_spec@entry=0, tzset_called=tzset_called@entry=0x7ffe0beb0de7,
    loc=0x7f017afae960 <_nl_C_locobj>) at strftime_l.c:471
#1  0x00007f017ae9d6ac in __GI___strftime_l (s=<optimized out>, maxsize=<optimized out>, format=<optimized out>, tp=<optimized out>, loc=<optimized out>) at strftime_l.c:460
#2  0x00007f017b09d11d in std::__timepunct<char>::_M_put(char*, unsigned long, char const*, tm const*) const () from /lib/x86_64-linux-gnu/libstdc++.so.6
#3  0x00007f017b0f5517 in std::time_put<char, std::ostreambuf_iterator<char, std::char_traits<char> > >::do_put(std::ostreambuf_iterator<char, std::char_traits<char> >, std::ios_base&, char, tm const*, char, char) const () from /lib/x86_64-linux-gnu/libstdc++.so.6
#4  0x00007f017b0f363c in std::time_put<char, std::ostreambuf_iterator<char, std::char_traits<char> > >::put(std::ostreambuf_iterator<char, std::char_traits<char> >, std::ios_base&, char, tm const*, char const*, char const*) const () from /lib/x86_64-linux-gnu/libstdc++.so.6
#5  0x000055844fad3138 in std::operator<< <char, std::char_traits<char> > (__os=..., __f=...) at /usr/include/c++/9/iomanip:379

i think you can assume that if tv_sec is more than 1k years, then this value is invalid or as you said, a year in future from now

Related to sonic-net/sonic-buildimage#19760. Tests added to rule out v13 and v14 headers health_data stack shift influencing timestamp.

kcudnik · 2024-11-18T11:03:09Z

I added this tests in sairedis: sonic-net/sonic-sairedis#1463 to test v13 and v14 missmatch

Related to sonic-net/sonic-buildimage#19760. Tests added to rule out v13 and v14 headers health_data stack shift influencing timestamp.

…ic-net#1463) Related to sonic-net/sonic-buildimage#19760. Tests added to rule out v13 and v14 headers health_data stack shift influencing timestamp.

zjswhhh added the Chassis 🤖 Modular chassis support label Aug 14, 2024

anamehra added this to SONiC Chassis Aug 14, 2024

rlhui added the Triaged this issue has been triaged label Aug 14, 2024

rlhui assigned anamehra Aug 14, 2024

anamehra changed the title ~~[chassis][202405]: orchagent crash during config reload while runnign sonic-mgmt nightly~~ [chassis][202405]: orchagent crash in NotificationSwitchAsicSdkHealthEvent::executeCallback while handling SAI notification Aug 28, 2024

kcudnik mentioned this issue Nov 18, 2024

[test] Add more testing for NotificationSwitchAsicSdkHealthEvent sonic-net/sonic-sairedis#1463

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[chassis][202405]: orchagent crash in NotificationSwitchAsicSdkHealthEvent::executeCallback while handling SAI notification #19760

[chassis][202405]: orchagent crash in NotificationSwitchAsicSdkHealthEvent::executeCallback while handling SAI notification #19760

anamehra commented Aug 1, 2024 •

edited

Loading

rlhui commented Aug 14, 2024

rlhui commented Aug 14, 2024

arlakshm commented Sep 11, 2024

arlakshm commented Nov 13, 2024 •

edited

Loading

kcudnik commented Nov 14, 2024

anamehra commented Nov 14, 2024 •

edited

Loading

kcudnik commented Nov 14, 2024 •

edited

Loading

kcudnik commented Nov 14, 2024 •

edited

Loading

anamehra commented Nov 14, 2024

kcudnik commented Nov 14, 2024

kcudnik commented Nov 14, 2024

anamehra commented Nov 14, 2024

stephenxs commented Nov 14, 2024

stephenxs commented Nov 15, 2024

kcudnik commented Nov 16, 2024

stephenxs commented Nov 17, 2024

kcudnik commented Nov 17, 2024 •

edited

Loading

anamehra commented Nov 17, 2024

kcudnik commented Nov 18, 2024

stephenxs commented Nov 18, 2024

kcudnik commented Nov 18, 2024 •

edited

Loading

kcudnik commented Nov 18, 2024

[chassis][202405]: orchagent crash in NotificationSwitchAsicSdkHealthEvent::executeCallback while handling SAI notification #19760

[chassis][202405]: orchagent crash in NotificationSwitchAsicSdkHealthEvent::executeCallback while handling SAI notification #19760

Comments

anamehra commented Aug 1, 2024 • edited Loading

Description

Steps to reproduce the issue:

Describe the results you received:

Describe the results you expected:

Output of show version:

Output of show techsupport:

Additional information you deem important (e.g. issue happens only occasionally):

rlhui commented Aug 14, 2024

rlhui commented Aug 14, 2024

arlakshm commented Sep 11, 2024

arlakshm commented Nov 13, 2024 • edited Loading

kcudnik commented Nov 14, 2024

anamehra commented Nov 14, 2024 • edited Loading

kcudnik commented Nov 14, 2024 • edited Loading

kcudnik commented Nov 14, 2024 • edited Loading

anamehra commented Nov 14, 2024

kcudnik commented Nov 14, 2024

kcudnik commented Nov 14, 2024

anamehra commented Nov 14, 2024

stephenxs commented Nov 14, 2024

stephenxs commented Nov 15, 2024

kcudnik commented Nov 16, 2024

stephenxs commented Nov 17, 2024

kcudnik commented Nov 17, 2024 • edited Loading

anamehra commented Nov 17, 2024

kcudnik commented Nov 18, 2024

stephenxs commented Nov 18, 2024

kcudnik commented Nov 18, 2024 • edited Loading

kcudnik commented Nov 18, 2024

anamehra commented Aug 1, 2024 •

edited

Loading

Output of `show version`:

Output of `show techsupport`:

arlakshm commented Nov 13, 2024 •

edited

Loading

anamehra commented Nov 14, 2024 •

edited

Loading

kcudnik commented Nov 14, 2024 •

edited

Loading

kcudnik commented Nov 14, 2024 •

edited

Loading

kcudnik commented Nov 17, 2024 •

edited

Loading

kcudnik commented Nov 18, 2024 •

edited

Loading