Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Orchagent crashed with SEGV when Flexcounter enabled for queue watermark, pg-drop #2935

Open
rajkumar38 opened this issue Oct 17, 2023 · 1 comment
Open

Orchagent crashed with SEGV when Flexcounter enabled for queue watermark, pg-drop #2935

rajkumar38 opened this issue Oct 17, 2023 · 1 comment
Assignees
@vadymhlushko-mlnx

Comments

@rajkumar38
Copy link
Contributor

Below crash is observed on platforms that are not supporting queue-watermark and pg-drop statistics.
Issue introduced with PR #2883

Back-trace:

Reading symbols from /usr/bin/orchagent...
Reading symbols from /usr/lib/debug/.build-id/48/c447dca41af351f964c7f81f2333d74eed6037.debug...
bt[New LWP 44]
[New LWP 82]
[New LWP 80]
[New LWP 83]
[New LWP 85]
[New LWP 86]
[New LWP 84]
[New LWP 148]
[New LWP 149]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/aarch64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/bin/orchagent -d /var/log/swss -b 1024 -s -m 00:51:82:11:22:00'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x0000aaaadb3d0e48 in std::_Bit_reference::operator= (__x=true, this=<optimized out>) at /usr/include/c++/10/bits/stl_bvector.h:93
93      /usr/include/c++/10/bits/stl_bvector.h: No such file or directory.
[Current thread is 1 (Thread 0xffffb2956690 (LWP 44))]
(gdb) bt
#0  0x0000aaaadb3d0e48 in std::_Bit_reference::operator= (__x=true, this=<optimized out>) at /usr/include/c++/10/bits/stl_bvector.h:93
#1  FlexCounterPgStates::enablePgCounter (this=this@entry=0xffffca3fe158, pgIndex=pgIndex@entry=0) at flexcounterorch.cpp:529
#2  0x0000aaaadb3d0eec in FlexCounterPgStates::enablePgCounters (this=this@entry=0xffffca3fe158, startIndex=startIndex@entry=0,
    endIndex=endIndex@entry=4294967295) at flexcounterorch.cpp:522
#3  0x0000aaaadb242504 in PortsOrch::generatePriorityGroupMap (this=this@entry=0xaaab0d8df6c0, pgsStateVector=std::map with 0 elements)
    at portsorch.cpp:6752
#4  0x0000aaaadb3d4f04 in FlexCounterOrch::doTask (this=0xaaab0d8c7ae0, consumer=...) at flexcounterorch.cpp:178
#5  0x0000aaaadb1bc400 in Orch::doTask (this=0xaaab0d8c7ae0) at orch.cpp:532
#6  0x0000aaaadb1b1358 in OrchDaemon::start (this=this@entry=0xaaab0d8b2250) at orchdaemon.cpp:833
#7  0x0000aaaadb1313e4 in main (argc=<optimized out>, argv=<optimized out>) at main.cpp:769
@rajkumar38 rajkumar38 mentioned this issue Oct 17, 2023
Merged
@rajkumar38 rajkumar38 changed the title Orchagent crashed with SEGV when Flexcounter enabled for queue watermark and pg-drop Orchagent crashed with SEGV when Flexcounter enabled for queue watermark, pg-drop Oct 17, 2023
@lizhijianrd lizhijianrd mentioned this issue Oct 18, 2023
Merged
11 tasks
@vadymhlushko-mlnx
Copy link
Contributor

vadymhlushko-mlnx commented Oct 19, 2023 •
edited
Loading

@rajkumar38 I've analyze an issue and found a bug, I will raise a PR with fixes

@vadymhlushko-mlnx vadymhlushko-mlnx self-assigned this Oct 19, 2023
@rajkumar38 rajkumar38 mentioned this issue Oct 20, 2023
Merged
prsunny pushed a commit that referenced this issue Oct 23, 2023
@rajkumar38
[FlexCounters] Fixed orchagent crash issue#2395 (#2936)
4f86f13 
Issue seen on platforms that are not supporting queue-watermark and pg counters.

What I did
Fixed orchagent crash reported in #2935
prsunny pushed a commit that referenced this issue Oct 23, 2023
@rajkumar38
[202205][FlexCounters] Fixed orchagent crash issue#2395 (#2939)
8b280d8 
Issue seen on platforms that are not supporting queue-watermark and pg counters.
*Cherry-pick fix for orchagent crash reported in #2935
yxieca pushed a commit to sonic-net/sonic-buildimage that referenced this issue Oct 26, 2023
@lizhijianrd
[minigraph-parser] Disable unsupported counters on management devices (…
1fd7746 
…#16937)

Why I did it
To avoid orchagent crash issue like sonic-net/sonic-swss#2935, disable unsupported counters on SONiC management devices.

Work item tracking
Microsoft ADO (number only): 25437720
How I did it
Update the minigraph parser to disable unsupported counters on management devices.

How to verify it
Verified by unittest.
Manually apply patch to DUT and do config load_minigraph
@mssonicbld mssonicbld mentioned this issue Oct 27, 2023
Merged
11 tasks
mssonicbld pushed a commit to mssonicbld/sonic-buildimage that referenced this issue Oct 27, 2023
@lizhijianrd @mssonicbld
[minigraph-parser] Disable unsupported counters on management devices (…
93e9557 
…sonic-net#16937)

Why I did it
To avoid orchagent crash issue like sonic-net/sonic-swss#2935, disable unsupported counters on SONiC management devices.

Work item tracking
Microsoft ADO (number only): 25437720
How I did it
Update the minigraph parser to disable unsupported counters on management devices.

How to verify it
Verified by unittest.
Manually apply patch to DUT and do config load_minigraph
yxieca pushed a commit to sonic-net/sonic-buildimage that referenced this issue Oct 27, 2023
@mssonicbld @lizhijianrd
[minigraph-parser] Disable unsupported counters on management devices (…
4ee39d1 
…#16937) (#17028)

Why I did it
To avoid orchagent crash issue like sonic-net/sonic-swss#2935, disable unsupported counters on SONiC management devices.

Work item tracking
Microsoft ADO (number only): 25437720
How I did it
Update the minigraph parser to disable unsupported counters on management devices.

How to verify it
Verified by unittest.
Manually apply patch to DUT and do config load_minigraph

Co-authored-by: Zhijian Li <zhijianli@microsoft.com>
StormLiangMS pushed a commit that referenced this issue Nov 2, 2023
@rajkumar38 @StormLiangMS
[FlexCounters] Fixed orchagent crash issue#2395 (#2936)
5d80f57 
Issue seen on platforms that are not supporting queue-watermark and pg counters.

What I did
Fixed orchagent crash reported in #2935
@mssonicbld mssonicbld mentioned this issue Nov 6, 2023
Merged
11 tasks
mssonicbld pushed a commit to mssonicbld/sonic-buildimage that referenced this issue Nov 6, 2023
@lizhijianrd @mssonicbld
[minigraph-parser] Disable unsupported counters on management devices (…
3dabbe4 
…sonic-net#16937)

Why I did it
To avoid orchagent crash issue like sonic-net/sonic-swss#2935, disable unsupported counters on SONiC management devices.

Work item tracking
Microsoft ADO (number only): 25437720
How I did it
Update the minigraph parser to disable unsupported counters on management devices.

How to verify it
Verified by unittest.
Manually apply patch to DUT and do config load_minigraph
StormLiangMS added a commit to sonic-net/sonic-buildimage that referenced this issue Nov 8, 2023
@mssonicbld @lizhijianrd @StormLiangMS
[minigraph-parser] Disable unsupported counters on management devices (…
f887f59 
…#16937) (#17103)

Why I did it
To avoid orchagent crash issue like sonic-net/sonic-swss#2935, disable unsupported counters on SONiC management devices.

Work item tracking
Microsoft ADO (number only): 25437720
How I did it
Update the minigraph parser to disable unsupported counters on management devices.

How to verify it
Verified by unittest.
Manually apply patch to DUT and do config load_minigraph

Co-authored-by: Zhijian Li <zhijianli@microsoft.com>
Co-authored-by: StormLiangMS <89824293+StormLiangMS@users.noreply.github.com>
@lizhijianrd lizhijianrd mentioned this issue Nov 30, 2023
Merged
11 tasks
lizhijianrd added a commit to lizhijianrd/sonic-buildimage that referenced this issue Nov 30, 2023
@lizhijianrd
[minigraph-parser] Disable unsupported counters on management devices (…
839c579 
…sonic-net#16937)

Why I did it
To avoid orchagent crash issue like sonic-net/sonic-swss#2935, disable unsupported counters on SONiC management devices.

Work item tracking
Microsoft ADO (number only): 25437720
How I did it
Update the minigraph parser to disable unsupported counters on management devices.

How to verify it
Verified by unittest.
Manually apply patch to DUT and do config load_minigraph
qiluo-msft pushed a commit to sonic-net/sonic-buildimage that referenced this issue Dec 4, 2023
@lizhijianrd
[minigraph-parser] Disable unsupported counters on management devices (…
2aa3298 
…#16937) (#17351)

Backport #16937

**Why I did it**
To avoid orchagent crash issue like sonic-net/sonic-swss#2935, disable unsupported counters on SONiC management devices.

**Work item tracking**
Microsoft ADO (number only): 25437720

**How I did it**
Update the minigraph parser to disable unsupported counters on management devices.

**How to verify it**
Verified by unittest.
Manually apply patch to DUT and do config load_minigraph

#### Why I did it

##### Work item tracking
- Microsoft ADO **(number only)**: 24802410

#### How I did it

#### How to verify it
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@vadymhlushko-mlnx vadymhlushko-mlnx

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@rajkumar38 @vadymhlushko-mlnx