Enable cavstool #897

keqiaozhang · 2022-04-29T08:09:56Z

No description provided.

marc-hb · 2022-05-03T00:01:00Z

test-case/check-sof-logger.sh

-        # several other places.
-        stdout_files=(data)
-        stderr_files=(error)
+	dlogi "Stop using sof-logger on Zephyr platforms."


cavstool.py has already been working at the same time than sof-logger for months. Let's start by using both first.

Change line 96 and simply call cavstool.py on the etrace.

marc-hb · 2022-05-03T00:02:06Z

case-lib/lib.sh

+    [[ "${OPT_VAL['z']}" -eq 0 ]] && return 0
+
+    [ "x$CAVSTOOL" == 'xnone' ] && {
+        dlogi 'CAVSTOOL logs collection globally disabled by CAVSTOOL=none'


Stick to SOF_LOGGING, no need for yet another parameter.

marc-hb · 2022-05-03T00:03:11Z

case-lib/lib.sh

+
+    test -x "$CAVSTOOL" || {
+        dlogw "$CAVSTOOL not found or not executable"
+        return 2


2 means SKIP. Did you test this?

marc-hb · 2022-05-03T00:05:24Z

case-lib/lib.sh

+        ZPH_LOG_COLLECT=1
+    else
+        >&2 dlogw "without sudo permission to run $CAVSTOOL command"
+        return 3


You're returning 3 different error codes but the callers never look at them. I don't think you ever tested all this error handling code; what problem does it solve? sudo, the shell and other system utilities already provide error messages for free, what was wrong with them?

Every line of code you submit should have been run at least once. Write the code you actually need, not the code you think you need.

marc-hb · 2022-05-03T00:08:12Z

test-case/check-alsabat.sh

@@ -42,6 +42,9 @@ OPT_HAS_ARG['n']=1             OPT_VAL['n']=240000
 OPT_NAME['s']='sof-logger'      OPT_DESC['s']="Open sof-logger trace the data will store at $LOG_ROOT"
 OPT_HAS_ARG['s']=0             OPT_VAL['s']=1

+OPT_NAME['z']='cavstool'       OPT_DESC['z']="Zephyr native logging tool, the data will store at $LOG_ROOT"
+OPT_HAS_ARG['z']=0             OPT_VAL['z']=1


Why do we need to make this configurable, do we have multiple names to support? Don't submit code that we don't need yet.

https://www.martinfowler.com/bliki/Yagni.html

marc-hb · 2022-05-03T00:10:18Z

test-case/check-alsabat.sh

+if is_zephyr; then
+    cavstool_disabled || func_cavstool_log_collect
+else
+    logger_disabled || func_lib_start_log_collect


is_zephyr must be kept _inside the logger_disabled and func_lib_start_log_collect abstractions to avoid so much copy/paste. Every time you do some repetitive copy/paste/diverge, an alarm should ring.

Don't pollute every single test with logging implementation details.

keqiaozhang · 2022-05-12T12:10:47Z

@marc-hb @aiChaoSONG
I have updated this PR and passed the on device test(sof-logger test failed on CML_RVP_SDW_ZEPHYR is not related to this PR). I also did some negative tests. Please help to review.

This is becoming required because Zephyr logging is adding another layer of complexity, See for instance thesofproject#897 work in progress. Signed-off-by: Marc Herbert <marc.herbert@intel.com>

marc-hb

I forgot how difficult this code was to understand. I'm adding some comments in #906, please help review.

This looks a good first step, thanks!

Actually, I think this is a good SECOND step: could you try to add the same feature to check-sof-logger.sh first in a separate PR? So if we find issues with cavstool.py then it will impact only the check-sof-logger.sh test whereas this PR is affecting all tests. If the new check-sof-logger.sh works fine for a couple days then we can merge this.

marc-hb · 2022-05-12T22:27:33Z

case-lib/hijack.sh

+	    dloge "cavstool.py was already dead"
+	        exit_status=1
+	    }
+	fi


This file does not use TABS, that's why the indentation looks wrong here on Github.

I'm really sorry that we use TABS in some projects and no TABS in other projects like sof-test. This means we must use editors that can quickly change from TABS mode to no TABS mode.

For Emacs I use this:

(defun tabs-toggle-myfunc() (interactive) (setq indent-tabs-mode (not indent-tabs-mode)))

case-lib/hijack.sh

marc-hb · 2022-05-12T22:43:00Z

case-lib/hijack.sh

-        fi
+	for logtool in "$cavstoolBin" "$loggerBin"
+	do
+            if pgrep -f -n "$logtool"; then


Why add -n?

For example, when running cavstool.py, it will trigger 2 processes in process list.

root 16635 0.0 0.0 13856 4748 pts/0 S+ 08:34 0:00 sudo cavstool.py -l root 16636 0.4 0.2 28744 16708 pts/0 S+ 08:34 0:01 python3 /usr/local/bin/cavstool.py -l

With -f option, it will list all 2 processes

pgrep -f cavstool.py 16635 16636

But it only list the child process with -n

pgrep -f -n cavstool.py 16636

If you want to be more selective then relying on the "newest" feels a bit brittle...

But why be more selective at all here? Both processes must be dead at this point in time. If any is left then something is wrong and the test should fail.

That makes sense, remove -n.

marc-hb · 2022-05-12T22:53:57Z

case-lib/lib.sh

@@ -175,8 +198,14 @@ func_lib_start_log_collect()
    if [ "X$is_etrace" == "X0" ];then
        logfile=$LOG_ROOT/slogger.txt
    else
-        logfile=$LOG_ROOT/etrace.txt
-        logopt=""
+	if is_zephyr; then


Same TABS problem.

outdated

This is becoming required because Zephyr logging is adding another layer of complexity, See for instance #897 work in progress. Signed-off-by: Marc Herbert <marc.herbert@intel.com>

keqiaozhang · 2022-05-31T14:06:37Z

Actually, I think this is a good SECOND step: could you try to add the same feature to check-sof-logger.sh first in a separate PR?

@marc-hb , sorry for late reply. I tried to enable the cavstool.py in check-sof-logger.sh to dump the etrace on zephyr platforms, but I have 2 issues to solve.

stderr is not empty.
The output of cavstool.py -l is:

INFO:cavs-fw:Existing driver "snd_sof_pci_intel_tgl" found
INFO:cavs-fw:Mapped PCI bar 0 of length 16384 bytes.
INFO:cavs-fw:Selected output stream 7 (GCAP = 0x1009701)
INFO:cavs-fw:Mapped PCI bar 4 of length 1048576 bytes.
INFO:cavs-fw:Detected cAVS 1.8+ hardware
INFO:cavs-fw:Waiting for firmware handoff, FW_STATUS = 0x5
[00000000] <inf> main: SOF on intel_adsp_cavs25
492724 zephyr_ll_scheduler_init(): unsupported domain 2
[00000000] <inf> main: SOF initialized
509231 zephyr_ll_scheduler_init(): unsupported domain 2

If only redirect stderr, the output file is not empty:

$ sudo timeout 2 cavstool.py -l  2> stderr  1> stdout ; cat stderr
INFO:cavs-fw:Existing driver "snd_sof_pci_intel_tgl" found
INFO:cavs-fw:Mapped PCI bar 0 of length 16384 bytes.
INFO:cavs-fw:Selected output stream 7 (GCAP = 0x1009701)
INFO:cavs-fw:Mapped PCI bar 4 of length 1048576 bytes.
INFO:cavs-fw:Detected cAVS 1.8+ hardware
INFO:cavs-fw:Waiting for firmware handoff, FW_STATUS = 0x5

@aiChaoSONG , can we change the error level for above logs?

How to redirect stderr+ stdout and stdout to separate files?
For sof logger, we can use -o to store the all etrace to $etrace_file and use 2> to redirect the stderr to etrace_stderr_file
sudo "$loggerBin" -f 3 -l "$ldcFile" 2> "$etrace_stderr_file" -o "$etrace_file"
But for cavstool.py, there's no -o option, so how to store stdout+stderr to $etrace_file after redirecting the stderr to etrace_stderr_file in one line of command?

aiChaoSONG · 2022-05-31T14:22:44Z

@aiChaoSONG , can we change the error level for above logs?

These I think these logs are from zephyr system, not the ones we can control

marc-hb · 2022-06-01T23:53:42Z

I just found by chance that cavstool.py disables power management (forcing /sys/devices/pci0000\:00/0000\:00\:0e.0/power/control). So we can't use it unconditionally yet. I'm looking into that right now.

keqiaozhang · 2022-06-09T08:44:42Z

@aiChaoSONG , can we change the error level for above logs?

These I think these logs are from zephyr system, not the ones we can control

@marc-hb , any comments on this?

marc-hb

I just found by chance that cavstool.py disables power management (forcing /sys/devices/pci0000:00/0000:00:0e.0/power/control). So we can't use it unconditionally yet. I'm looking into that right now.

Have you looked at this? This is very important. I was busy with the Zephyr conference, will look at this next.

marc-hb · 2022-06-10T01:24:32Z

case-lib/lib.sh

+
+    logfile=$LOG_ROOT/cavstool.txt
+
+    local cavstoolCmd=("$CAVSTOOL" "$logopt" '&>' "$logfile")


For here avoid this array and quoting complexity thanks to sudo tee:

sudo cavstool arguments |& sudo tee "$logfile" > /dev/null

I will see what can be done for the check-sof-logger test, thanks for highlighting this issue.

Sort of "fixed" for check-sof-logger in #929

marc-hb

Blocking this until cavstool is compatible with D3. Working on it.

keqiaozhang · 2022-06-13T08:01:38Z

I just found by chance that cavstool.py disables power management (forcing
/sys/devices/pci0000:00/0000:00:0e.0/power/control

@marc-hb please share more information about this, I tried cavstool on APL and TGL UP extreme board. Power control always on when running cavstool.py.

marc-hb · 2022-06-13T14:40:04Z

Power control always on when running cavstool.py.

Exactly.

marc-hb · 2022-06-27T23:03:12Z

cavstool.py has unrelated stability issues slowing or blocking progress: zephyrproject-rtos/zephyr#46864

There are tentative fixes.

marc-hb · 2022-07-01T02:56:07Z

cavstool support for D3 submitted in:

soc/intel_adsp: cavstool: add support for DSP power state: D3 zephyrproject-rtos/zephyr#47128

The immediate next step will be to use it in every test and collect Zephyr logs, see PR thesofproject#897 Signed-off-by: Marc Herbert <marc.herbert@intel.com>

keqiaozhang · 2022-07-19T09:28:02Z

Except for the TIMEOUT of suspend/resume on WHL, all other failures are related to the SOF issue thesofproject/sof#6020.

Also thanks to thesofproject/sof#6020, it helped to prove that this PR really works.
Please refer to multiple-pause-resume-5.sh on WHL_UPEXT_HDA_ZEPHYR.

The immediate next step will be to use it in every test and collect Zephyr logs, see PR #897 Signed-off-by: Marc Herbert <marc.herbert@intel.com>

keqiaozhang · 2022-07-20T13:38:52Z

@marc-hb , the PR testing results are not good on Zephyr platforms (EDIT: https://sof-ci.01.org/softestpr/PR897/build103/devicetest/). It seems that starting BOTH sof-logger DMA trace AND cavstool for the whole test duration will introduce new IPC errors/timeouts.
I can reproduce it manually(reproduction rate is about 1/5). For triggering the issue, we need to run one test case for several times or run different cases in sequence.

I've confirmed that cavstool will be terminated before the end of each case.

marc-hb · 2022-07-20T22:19:08Z

It seems that starting BOTH sof-logger DMA trace AND cavstool for the whole test duration will introduce new IPC errors/timeouts.

Wow, fascinating. So let's do this: please file a new bug, something "Long-running cavstool cause random failures". Then insert the bug number at the FIXME location below and let's merge this sof-test PR with the new feature commented out for now. This will help anyone trying to fix this new bug: now they will have a single line to change in sof-test.

marc-hb · 2022-07-20T22:14:26Z

case-lib/hijack.sh

@@ -54,9 +54,18 @@ function func_exit_handler()
        # shellcheck disable=SC2154
        logfile="$logfile"

-        local loggerBin wcLog; loggerBin=$(basename "$SOFLOGGER")
-        # We need this to avoid the confusion of a "Terminated" message
+        local loggerBin wcLog; loggerBin=$(basename "$SOFLOGGER");


Spurious edit

Suggested change

local loggerBin wcLog; loggerBin=$(basename "$SOFLOGGER");

local loggerBin wcLog; loggerBin=$(basename "$SOFLOGGER")

marc-hb · 2022-07-20T22:15:28Z

case-lib/lib.sh

@@ -293,6 +293,29 @@ find_ldc_file()
    printf '%s' "$ldcFile"
 }

+func_cavstool_etrace_collect()
+{
+    local logopt="-l"


marc-hb · 2022-07-20T22:16:16Z

case-lib/lib.sh

+        # start cavstool in the background to collect the etrace.
+        # Since cavstool can follow a ring buffer in etrace, so it should be
+        # started at the start of the test and we would not miss any Zephyr logs.
+        is_zephyr && func_cavstool_etrace_collect


Suggested change

is_zephyr && func_cavstool_etrace_collect

# FIXME (new) sof bug #1234

# is_zephyr && func_cavstool_etrace_collect

marc-hb · 2022-07-20T22:20:41Z

case-lib/lib.sh

+    local cavstoolCmd=("$CAVSTOOL" "$logopt")
+    dlogi "Starting ${cavstoolCmd[*]}"
+    # Cleaned up by func_exit_handler() in hijack.sh
+    sudo "${cavstoolCmd[@]}" > "$clogfile" &


Suggested change

sudo "${cavstoolCmd[@]}" > "$clogfile" &

# shellcheck disable=SC2024

sudo "${cavstoolCmd[@]}" >& "$clogfile" &

sudo "${cavstoolCmd[@]}" > "$clogfile" & ^-- SC2024: sudo doesn't affect redirects. Use ..| sudo tee file

You must use >& because we really want BOTH

marc-hb · 2022-07-20T22:30:23Z

case-lib/lib.sh

+func_cavstool_etrace_collect()
+{
+    local logopt="-l"
+    local clogfile=$LOG_ROOT/cavstool.txt


Suggested change

local clogfile=$LOG_ROOT/cavstool.txt

local clogfile=$LOG_ROOT/etrace.txt

Please keep the same name for consistency. I'm working on another PR that scans for ERRORs.

marc-hb · 2022-07-21T05:09:46Z

https://sof-ci.01.org/softestpr/PR897/build107/devicetest had a lot of thesofproject/sof#6039 failures

Looks like adding cavstool to the sof-logger test ONLY is already causing some trouble: thesofproject/sof#6042

cavstool.py is a Zephyr native logging tool, this tool will be used to dump the etrace from the shared memory. Signed-off-by: Keqiao Zhang <keqiao.zhang@intel.com>

we will use the Zephyr logging tool to dump the etrace form the shared memory for all Zephyr platforms. Signed-off-by: Keqiao Zhang <keqiao.zhang@intel.com>

we need to kill the cavstool.py process after the test to prevent the conflicts between each case. Signed-off-by: Keqiao Zhang <keqiao.zhang@intel.com>

keqiaozhang · 2022-07-22T07:14:47Z

I have filed a SOF issue:thesofproject/sof#6039

Since the latest cavstool.py supports verbose log, so I enabled it accordingly local clogopt="--log-only --verbose". But I don't see any obvious difference.

marc-hb · 2022-07-23T03:24:52Z

case-lib/lib.sh

+        }
+    fi
+
+    local cavstoolCmd=("$CAVSTOOL" "$clogopt")


Suggested change

local cavstoolCmd=("$CAVSTOOL" "$clogopt")

local cavstoolCmd=("$CAVSTOOL" --log-only --verbose)

Otherwise this tries to load a firmware file named '--log-only --verbose'

I just wasted a couple of hours on this... already asked to inline this pointless variable before (when it was a single option and working)

OK, I realize only I see this because I disable the sof-test sudo function, I'm using the regular sudo command.

Should still be fixed.

Please help review and merge this sudo() fix first:

Fix hijack.sh sudo() to use an array, support whitespace and behave #943

marc-hb · 2022-07-26T01:18:06Z

SOFCI TEST

EDIT: newer https://sof-ci.01.org/softestpr/PR897/build108/devicetest/ seems to still fail the same

Overriden commands should work exactly like real commands and support whitespace and special characters exactly like the real command. I usually disable the sudo() override locally when using `set -x` to reduce trace noise considerably. The previous difference between the sudo() override and the built-in sudo() cost me couple hours twice: - WIP thesofproject#897 (comment) - Merged commit 232d40a No more! Also fix the return value: 2 means SKIP. Signed-off-by: Marc Herbert <marc.herbert@intel.com>

Overriden commands should work exactly like real commands and support whitespace and special characters exactly like the real command. I usually disable the sudo() override locally when using `set -x` to reduce trace noise considerably. The previous difference between the sudo() override and the built-in sudo() cost me couple hours twice: - WIP #897 (comment) - Merged commit 232d40a No more! Also fix the return value: 2 means SKIP. Signed-off-by: Marc Herbert <marc.herbert@intel.com>

marc-hb

I just realized the $logfile is a complete mess that must be fixed before this PR in a different and earlier PR

marc-hb · 2022-07-29T01:38:42Z

case-lib/lib.sh

+        # it again. We only need to start sof-logger to collect the etrace logs
+        # only for the non-zephyr platforms at the end of a test.
+        is_zephyr && return 0
+


The $logfile variable in this function is a global variable passed to lib.sh and I just realized it is a complete mess that must be fixed before this PR

marc-hb · 2022-08-03T20:21:34Z

func_cavstool_etrace_collect 0 : starts BOTH sof-logger DMA trace AND cavstool for the whole test duration

I'm sorry for this wrong advice. @kv2019i explained me (and you?) that cavstool is "spamming" the SRAM which can have bad interactions with the firmware - as you found in test results above and filed in thesofproject/sof#6039. So we're back to having just a quick cavstool look at the end of the test, really sorry for the back and forth.

cc: @nklayman , @andyross , @jsarha , @enjiamai, @teburd

enjiamai · 2022-08-03T23:21:51Z

Excuse me, @marc-hb , is this caused by the modified cavstool.py (client-server-base)? I am working on another PR to split the pure cavstool.py (original one) and the client-server logic. Does this help? Thanks.

marc-hb · 2022-08-04T07:07:31Z

Excuse me, @marc-hb , is this caused by the modified cavstool.py (client-server-base)?

No, it's a warning that any castool.py is "spamming" the SRAM which is causing issues.

andyross · 2022-08-08T15:36:49Z

cavstool is "spamming" the SRAM which can have bad interactions with the firmware

It should only be polling 4 words every 100ms. Obviously if there's a ton of data to be copied out, that's going to have impact. But it's intended to be a reasonably good citizen. Are there existing interactions it's causing? We should look at those, I suspect.

kv2019i · 2022-08-10T08:27:25Z

@andyross wrote:

cavstool is "spamming" the SRAM which can have bad interactions with the firmware

It should only be polling 4 words every 100ms. Obviously if there's a ton of data to be copied out, that's going to have
impact. But it's intended to be a reasonably good citizen. Are there existing interactions it's causing? We should look at those,

Ack, thanks Andy. I actually assumed it was polling more frequently (and gave the above feedback). So yeah, this seems more like a bug (and probably not in cavstool.py). Given the problems only occur in large test campaigns, this is not so easy to debug. We do have other logging backends coming available very soon (zephyrproject-rtos/zephyr#47058 and thesofproject/sof#6072 ), I'd like to focus energy on getting these working first. Both mtrace (SRAM) and probes (DMA/DDR) backends provide capability to notify the host (with a IPC/DMA irq) when new data is available, eliminating the need for any polling of data by host. Given the possible people who could debug the rare failures with cavstool.py with SOF test campaigns are the same people working on the new backends, I'd prefer to complete the new backends first and prioritize enabling these in the SOF CI.

marc-hb · 2022-09-13T14:17:30Z

Ack, thanks Andy. I actually assumed it was polling more frequently (and gave the above feedback). So yeah, this seems more like a bug (and probably not in cavstool.py).

So may be the bug WAS in cavstool.py after all? Just fixed by @ujfalusi ?

soc: xtensa: tools: do not interfere with IPC in logs-only mode zephyrproject-rtos/zephyr#50178

marc-hb · 2022-09-21T04:30:33Z

Closed in the weekly CI meeting because IPC3+Zephyr will never be supported in production. Zephyr will be IPC4 only which means it will always have mtrace support.

marc-hb previously requested changes May 3, 2022

View reviewed changes

keqiaozhang force-pushed the enable-cavstool branch from 83d9a71 to 848b608 Compare May 12, 2022 11:26

keqiaozhang marked this pull request as ready for review May 12, 2022 12:08

keqiaozhang requested a review from a team as a code owner May 12, 2022 12:08

marc-hb mentioned this pull request May 13, 2022

lib.sh: add detailed description of func_lib_start_log_collect() #906

Merged

marc-hb reviewed May 13, 2022

View reviewed changes

keqiaozhang force-pushed the enable-cavstool branch from 848b608 to db0ca17 Compare June 9, 2022 08:40

marc-hb reviewed Jun 10, 2022

View reviewed changes

marc-hb mentioned this pull request Jun 10, 2022

[BUG][ADLP] ipc tx timed out for 0x60060000（TRIG_PAUSE）when testing multiple-pause-resume on ADLP Chrome device thesofproject/sof#5850

Closed

marc-hb requested changes Jun 10, 2022

View reviewed changes

mengdonglin added P1 Blocker bugs or important features type:enhancement New framework feature or request labels Jun 11, 2022

marc-hb mentioned this pull request Jun 28, 2022

soc: xtensa: partial fix of socket misuse and refine the code zephyrproject-rtos/zephyr#46880

Merged

keqiaozhang force-pushed the enable-cavstool branch 2 times, most recently from fd8d0b7 to ac0421e Compare July 19, 2022 08:24

marc-hb added a commit that referenced this pull request Jul 19, 2022

check-sof-logger: add cavstool support

d6c0e76

The immediate next step will be to use it in every test and collect Zephyr logs, see PR #897 Signed-off-by: Marc Herbert <marc.herbert@intel.com>

marc-hb requested changes Jul 20, 2022

View reviewed changes

marc-hb reviewed Jul 20, 2022

View reviewed changes

marc-hb mentioned this pull request Jul 21, 2022

[BUG] IPC timeout in sof-logger-check thesofproject/sof#6042

Closed

keqiaozhang mentioned this pull request Jul 22, 2022

[BUG]Long-running cavstool cause random failures thesofproject/sof#6039

Closed

keqiaozhang added 3 commits July 22, 2022 14:09

lib.sh: add a new logging tool casvtool.py support

69a9973

cavstool.py is a Zephyr native logging tool, this tool will be used to dump the etrace from the shared memory. Signed-off-by: Keqiao Zhang <keqiao.zhang@intel.com>

lib.sh: switch to use cavstool to dump the etrace

383747e

we will use the Zephyr logging tool to dump the etrace form the shared memory for all Zephyr platforms. Signed-off-by: Keqiao Zhang <keqiao.zhang@intel.com>

hijack.sh: kill the cavstool.py after the test

d11a6b5

we need to kill the cavstool.py process after the test to prevent the conflicts between each case. Signed-off-by: Keqiao Zhang <keqiao.zhang@intel.com>

keqiaozhang force-pushed the enable-cavstool branch from 17b9ec7 to d11a6b5 Compare July 22, 2022 07:02

marc-hb requested changes Jul 23, 2022

View reviewed changes

marc-hb mentioned this pull request Jul 26, 2022

Check for a valid trace for ALL tests #297

Closed

marc-hb mentioned this pull request Jul 26, 2022

Fix hijack.sh sudo() to use an array, support whitespace and behave #943

Merged

marc-hb requested changes Jul 29, 2022

View reviewed changes

marc-hb mentioned this pull request Sep 19, 2022

Enable mtrace #956

Merged

mengdonglin closed this Sep 21, 2022


		logfile=$LOG_ROOT/cavstool.txt

		local cavstoolCmd=("$CAVSTOOL" "$logopt" '&>' "$logfile")

	local loggerBin wcLog; loggerBin=$(basename "$SOFLOGGER");
	local loggerBin wcLog; loggerBin=$(basename "$SOFLOGGER")

	is_zephyr && func_cavstool_etrace_collect
	# FIXME (new) sof bug #1234
	# is_zephyr && func_cavstool_etrace_collect

	sudo "${cavstoolCmd[@]}" > "$clogfile" &
	# shellcheck disable=SC2024
	sudo "${cavstoolCmd[@]}" >& "$clogfile" &

	local clogfile=$LOG_ROOT/cavstool.txt
	local clogfile=$LOG_ROOT/etrace.txt

	local cavstoolCmd=("$CAVSTOOL" "$clogopt")
	local cavstoolCmd=("$CAVSTOOL" --log-only --verbose)

Enable cavstool #897

Enable cavstool #897

Conversation

keqiaozhang commented Apr 29, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

marc-hb May 3, 2022 • edited Loading

Choose a reason for hiding this comment

keqiaozhang commented May 12, 2022

marc-hb left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

keqiaozhang commented May 31, 2022

aiChaoSONG commented May 31, 2022

marc-hb commented Jun 1, 2022

keqiaozhang commented Jun 9, 2022

marc-hb left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

marc-hb left a comment

Choose a reason for hiding this comment

keqiaozhang commented Jun 13, 2022

marc-hb commented Jun 13, 2022

marc-hb commented Jun 27, 2022

marc-hb commented Jul 1, 2022

keqiaozhang commented Jul 19, 2022

keqiaozhang commented Jul 20, 2022 • edited by marc-hb Loading

marc-hb commented Jul 20, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

marc-hb Jul 20, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

marc-hb commented Jul 21, 2022 • edited Loading

keqiaozhang commented Jul 22, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

marc-hb commented Jul 26, 2022 • edited Loading

marc-hb left a comment

Choose a reason for hiding this comment

marc-hb Jul 29, 2022 • edited Loading

Choose a reason for hiding this comment

marc-hb commented Aug 3, 2022

enjiamai commented Aug 3, 2022

marc-hb commented Aug 4, 2022

andyross commented Aug 8, 2022

kv2019i commented Aug 10, 2022

marc-hb commented Sep 13, 2022

marc-hb commented Sep 21, 2022

marc-hb May 3, 2022 •

edited

Loading

keqiaozhang commented Jul 20, 2022 •

edited by marc-hb

Loading

marc-hb Jul 20, 2022 •

edited

Loading

marc-hb commented Jul 21, 2022 •

edited

Loading

keqiaozhang commented Jul 22, 2022 •

edited

Loading

marc-hb commented Jul 26, 2022 •

edited

Loading

marc-hb Jul 29, 2022 •

edited

Loading