Fix missing declaration of custom externs in the generated eBPF/uBPF header file. #4644

fruffy · 2024-04-29T15:01:00Z

Fixes a warning that occurs when compiling eBPF/uBPF programs with a custom, user-defined extern. The extern has not been declared, which causes GCC to spit out warning: implicit declaration of function.

Because there are also method declarations in core.p4 and the corresponding include file I need to add a filter based on source information. I do not know whether there is a better approach to this.

I also added the error type, which is the same as the enum type. I also could add Type_String, but it has unclear semantics on width.

@thomascalvert-xlnx

thomascalvert-xlnx · 2024-04-29T15:20:36Z

The documentation seems to suggest that this is by design, and -include must be supplied to clang with the extern definition. Maybe I've got the wrong end of the stick - otherwise this PR should update the README file accordingly.

Agree that the filename check used in this PR is brittle, but I don't know if there is a better way...

Is there any testdata for eBPF externs?

fruffy · 2024-04-29T15:35:51Z

The failing tests are here: https://github.com/p4lang/p4c/actions/runs/8862986990/job/24336526254

The problem is that, while the -include enables compilation. The function definition is missing when compiling individual objects. That leads to this unnecessary warning, which is treated as error for some systems.

My fix simply adds a declaration in the header file to prevent the warning from occurring. The definition is still in the .c file of the extern.

It would be great if we had golden files which show the change.

thomascalvert-xlnx

Thanks for clarifying @fruffy. I notice in that test output that only the ubpf extern tests are failing, not ebpf - do we know why?

I agree that emitting the declarations is a good step forwards, the suggested -include method feels like a bit of a hack (imho).

As mentioned in my previous comment, I don't think that discriminating based on the source filename is robust. We're already missing ebpf_model.p4 and xdp_model.p4 here. If we don't want to emit declarations for "standard" externs which are already defined, perhaps a better solution would be to filter based on a list of such standard externs? Any other suggestions?

fruffy · 2024-04-30T12:10:39Z

If we don't want to emit declarations for "standard" externs which are already defined, perhaps a better solution would be to filter based on a list of such standard externs? Any other suggestions?

I will give this a shot.

fruffy · 2024-04-30T12:36:08Z

I notice in that test output that only the ubpf extern tests are failing, not ebpf - do we know why?

Yeah, I wondered about that too. It must be related to the compile order, but I can't identify the difference.

…header file.

fruffy · 2024-04-30T15:34:28Z

@thomascalvert-xlnx I implemented a technique to filter based on method names instead. I also created #4645 because I ran into a bunch of issues. For example, progTarget is a member class of the EBPFProgram but is only really initialized by the XDP back end. In all other cases it is a nullptr.

thomascalvert-xlnx

LGTM thanks for looking at this Fabian!

Agree it's hard to work on the eBPF backend due to the tangled dependencies & inheritance with other backends (ubpf, tc, psa). Regarding the progTarget variable you refer to, as far as I can tell it is in fact only really useful to the PSA backend (sub-backend?).

fruffy · 2024-05-01T15:26:53Z

I do not understand the Ubuntu 20.04 failure. The tests passes on Ubuntu 18.04 and Ubuntu 22.04 but not here. The output doesn't help either. @thomascalvert-xlnx any intuition. Do we really need to declare every function static inline?

thomascalvert-xlnx · 2024-05-02T08:36:12Z

It looks like the verifier is rejecting the program with the following error:

BTF debug data section '.BTF' rejected: Invalid argument (22)!

There seems to be a similar issue reported here: xdp-project/xdp-tutorial#38

As far as I can tell it's an issue with specific tool versions - could be a mismatch between libbpf versions of the OS vs this repo? Or could be some problem with the debug info, try dumping it as suggested in the link above?

fruffy · 2024-05-02T14:09:09Z

#2407 could help here, but I never implemented it. Unfortunate.

fruffy · 2024-05-02T16:03:33Z

I give up. I do not know what else to try. I will just ignore this test on Ubuntu 20.04.

FYI this is the output of the failing test, I do not see anything suspicious:

INFO:Executing command: objdump -hj xdp /home/runner/work/p4c/p4c/build/tmp0gtym2bi/test.o
INFO:
INFO:/home/runner/work/p4c/p4c/build/tmp0gtym2bi/test.o:     file format elf64-little
INFO:
INFO:Sections:
INFO:Idx Name          Size      VMA               LMA               File off  Algn
WARNING:objdump: section 'xdp' mentioned in a -j option, but not found in any input file
ERROR:Error 1 when executing "objdump -hj xdp /home/runner/work/p4c/p4c/build/tmp0gtym2bi/test.o".
INFO:Executing command: llvm-objdump -d /home/runner/work/p4c/p4c/build/tmp0gtym2bi/test.o
INFO:
INFO:/home/runner/work/p4c/p4c/build/tmp0gtym2bi/test.o:	file format ELF64-BPF
INFO:
INFO:
INFO:Disassembly of section .text:
INFO:
INFO:0000000000000000 verify_ipv4_checksum:
INFO:       0:	71 12 00 00 00 00 00 00	r2 = *(u8 *)(r1 + 0)
INFO:       1:	64 02 00 00 0c 00 00 00	w2 <<= 12
INFO:       2:	71 13 01 00 00 00 00 00	r3 = *(u8 *)(r1 + 1)
INFO:       3:	64 03 00 00 08 00 00 00	w3 <<= 8
INFO:       4:	4c 23 00 00 00 00 00 00	w3 |= w2
INFO:       5:	71 12 02 00 00 00 00 00	r2 = *(u8 *)(r1 + 2)
INFO:       6:	4c 23 00 00 00 00 00 00	w3 |= w2
INFO:       7:	dc 03 00 00 10 00 00 00	r3 = be16 r3
INFO:       8:	69 12 04 00 00 00 00 00	r2 = *(u16 *)(r1 + 4)
INFO:       9:	dc 02 00 00 10 00 00 00	r2 = be16 r2
INFO:      10:	0c 32 00 00 00 00 00 00	w2 += w3
INFO:      11:	69 13 06 00 00 00 00 00	r3 = *(u16 *)(r1 + 6)
INFO:      12:	dc 03 00 00 10 00 00 00	r3 = be16 r3
INFO:      13:	0c 32 00 00 00 00 00 00	w2 += w3
INFO:      14:	71 13 08 00 00 00 00 00	r3 = *(u8 *)(r1 + 8)
INFO:      15:	64 03 00 00 0d 00 00 00	w3 <<= 13
INFO:      16:	69 14 0a 00 00 00 00 00	r4 = *(u16 *)(r1 + 10)
INFO:      17:	4c 43 00 00 00 00 00 00	w3 |= w4
INFO:      18:	dc 03 00 00 10 00 00 00	r3 = be16 r3
INFO:      19:	0c 32 00 00 00 00 00 00	w2 += w3
INFO:      20:	69 13 0e 00 00 00 00 00	r3 = *(u16 *)(r1 + 14)
INFO:      21:	dc 03 00 00 10 00 00 00	r3 = be16 r3
INFO:      22:	0c 32 00 00 00 00 00 00	w2 += w3
INFO:      23:	61 13 10 00 00 00 00 00	r3 = *(u32 *)(r1 + 16)
INFO:      24:	bf 34 00 00 00 00 00 00	r4 = r3
INFO:      25:	dc 04 00 00 20 00 00 00	r4 = be32 r4
INFO:      26:	54 04 00 00 ff ff 00 00	w4 &= 65535
INFO:      27:	0c 42 00 00 00 00 00 00	w2 += w4
INFO:      28:	dc 03 00 00 10 00 00 00	r3 = be16 r3
INFO:      29:	0c 32 00 00 00 00 00 00	w2 += w3
INFO:      30:	61 13 14 00 00 00 00 00	r3 = *(u32 *)(r1 + 20)
INFO:      31:	bf 34 00 00 00 00 00 00	r4 = r3
INFO:      32:	dc 04 00 00 20 00 00 00	r4 = be32 r4
INFO:      33:	54 04 00 00 ff ff 00 00	w4 &= 65535
INFO:      34:	0c 42 00 00 00 00 00 00	w2 += w4
INFO:      35:	dc 03 00 00 10 00 00 00	r3 = be16 r3
INFO:      36:	0c 32 00 00 00 00 00 00	w2 += w3
INFO:      37:	69 11 0c 00 00 00 00 00	r1 = *(u16 *)(r1 + 12)
INFO:      38:	0c 12 00 00 00 00 00 00	w2 += w1
INFO:      39:	bc 21 00 00 00 00 00 00	w1 = w2
INFO:      40:	74 01 00 00 10 00 00 00	w1 >>= 16
INFO:      41:	0c 21 00 00 00 00 00 00	w1 += w2
INFO:      42:	54 01 00 00 ff ff 00 00	w1 &= 65535
INFO:      43:	b4 00 00 00 01 00 00 00	w0 = 1
INFO:      44:	16 01 01 00 ff ff 00 00	if w1 == 65535 goto +1 <LBB0_2>
INFO:      45:	b4 00 00 00 00 00 00 00	w0 = 0
INFO:
INFO:0000000000000170 LBB0_2:
INFO:      46:	95 00 00 00 00 00 00 00	exit
INFO:
INFO:Disassembly of section prog:
INFO:
INFO:0000000000000000 ebpf_filter:
INFO:       0:	b4 00 00 00 02 00 00 00	w0 = 2
INFO:       1:	61 12 4c 00 00 00 00 00	r2 = *(u32 *)(r1 + 76)
INFO:       2:	61 11 50 00 00 00 00 00	r1 = *(u32 *)(r1 + 80)
INFO:       3:	bf 23 00 00 00 00 00 00	r3 = r2
INFO:       4:	07 03 00 00 0e 00 00 00	r3 += 14
INFO:       5:	2d 13 36 00 00 00 00 00	if r3 > r1 goto +54 <LBB1_5>
INFO:       6:	bf 23 00 00 00 00 00 00	r3 = r2
INFO:       7:	07 03 00 00 22 00 00 00	r3 += 34
INFO:       8:	2d 13 33 00 00 00 00 00	if r3 > r1 goto +51 <LBB1_5>
INFO:       9:	69 23 10 00 00 00 00 00	r3 = *(u16 *)(r2 + 16)
INFO:      10:	69 21 12 00 00 00 00 00	r1 = *(u16 *)(r2 + 18)
INFO:      11:	0c 31 00 00 00 00 00 00	w1 += w3
INFO:      12:	69 23 18 00 00 00 00 00	r3 = *(u16 *)(r2 + 24)
INFO:      13:	0c 31 00 00 00 00 00 00	w1 += w3
INFO:      14:	61 23 1a 00 00 00 00 00	r3 = *(u32 *)(r2 + 26)
INFO:      15:	bc 34 00 00 00 00 00 00	w4 = w3
INFO:      16:	54 04 00 00 ff ff 00 00	w4 &= 65535
INFO:      17:	0c 41 00 00 00 00 00 00	w1 += w4
INFO:      18:	74 03 00 00 10 00 00 00	w3 >>= 16
INFO:      19:	0c 31 00 00 00 00 00 00	w1 += w3
INFO:      20:	61 23 1e 00 00 00 00 00	r3 = *(u32 *)(r2 + 30)
INFO:      21:	bc 34 00 00 00 00 00 00	w4 = w3
INFO:      22:	54 04 00 00 ff ff 00 00	w4 &= 65535
INFO:      23:	0c 41 00 00 00 00 00 00	w1 += w4
INFO:      24:	74 03 00 00 10 00 00 00	w3 >>= 16
INFO:      25:	0c 31 00 00 00 00 00 00	w1 += w3
INFO:      26:	71 23 0e 00 00 00 00 00	r3 = *(u8 *)(r2 + 14)
INFO:      27:	bc 34 00 00 00 00 00 00	w4 = w3
INFO:      28:	64 04 00 00 08 00 00 00	w4 <<= 8
INFO:      29:	54 04 00 00 00 f0 00 00	w4 &= 61440
INFO:      30:	54 03 00 00 0f 00 00 00	w3 &= 15
INFO:      31:	64 03 00 00 08 00 00 00	w3 <<= 8
INFO:      32:	4c 43 00 00 00 00 00 00	w3 |= w4
INFO:      33:	71 24 0f 00 00 00 00 00	r4 = *(u8 *)(r2 + 15)
INFO:      34:	4c 43 00 00 00 00 00 00	w3 |= w4
INFO:      35:	dc 03 00 00 10 00 00 00	r3 = be16 r3
INFO:      36:	0c 31 00 00 00 00 00 00	w1 += w3
INFO:      37:	71 23 16 00 00 00 00 00	r3 = *(u8 *)(r2 + 22)
INFO:      38:	64 03 00 00 08 00 00 00	w3 <<= 8
INFO:      39:	71 24 17 00 00 00 00 00	r4 = *(u8 *)(r2 + 23)
INFO:      40:	4c 43 00 00 00 00 00 00	w3 |= w4
INFO:      41:	69 24 14 00 00 00 00 00	r4 = *(u16 *)(r2 + 20)
INFO:      42:	54 04 00 00 1f ff 00 00	w4 &= 65311
INFO:      43:	dc 04 00 00 10 00 00 00	r4 = be16 r4
INFO:      44:	71 22 14 00 00 00 00 00	r2 = *(u8 *)(r2 + 20)
INFO:      45:	64 02 00 00 08 00 00 00	w2 <<= 8
INFO:      46:	54 02 00 00 00 e0 00 00	w2 &= 57344
INFO:      47:	4c 42 00 00 00 00 00 00	w2 |= w4
INFO:      48:	dc 02 00 00 10 00 00 00	r2 = be16 r2
INFO:      49:	0c 21 00 00 00 00 00 00	w1 += w2
INFO:      50:	dc 03 00 00 10 00 00 00	r3 = be16 r3
INFO:      51:	0c 31 00 00 00 00 00 00	w1 += w3
INFO:      52:	bc 12 00 00 00 00 00 00	w2 = w1
INFO:      53:	74 02 00 00 10 00 00 00	w2 >>= 16
INFO:      54:	0c 12 00 00 00 00 00 00	w2 += w1
INFO:      55:	54 02 00 00 ff ff 00 00	w2 &= 65535
INFO:      56:	b4 00 00 00 01 00 00 00	w0 = 1
INFO:      57:	56 02 01 00 ff ff 00 00	if w2 != 65535 goto +1 <LBB1_4>
INFO:      58:	b4 00 00 00 00 00 00 00	w0 = 0
INFO:
INFO:00000000000001d8 LBB1_4:
INFO:      59:	64 00 00 00 01 00 00 00	w0 <<= 1
INFO:
INFO:00000000000001e0 LBB1_5:
INFO:      60:	95 00 00 00 00 00 00 00	exit
INFO:Executing command: ip netns exec 61631 tc qdisc add dev 0 clsact
INFO:Writing tc filter add dev 0 egress bpf da obj /home/runner/work/p4c/p4c/build/tmp0gtym2bi/test.o section prog verbose 
INFO:Writing  &&  
INFO:Writing ls -1 /sys/fs/bpf/tc/globals 
INFO:Input file: /home/runner/work/p4c/p4c/build/tmp0gtym2bi/pcap_in.pcap
INFO:Writing  && /home/runner/work/p4c/p4c/build/tmp0gtym2bi/test -f /home/runner/work/p4c/p4c/build/tmp0gtym2bi/pcap_in.pcap -n 1 -d 
INFO:Executing command: <subprocess.Popen object at 0x7f0d28144fd0>
ERROR:Error 1:

BTF debug data section '.BTF' rejected: Invalid argument (22)!
 - Length:       3734
Verifier analysis:

magic: 0xeb9f
version: 1
flags: 0x0
hdr_len: 24
type_off: 0
type_len: 1068
str_off: 1068
str_len: 2642
btf_total_size: 3734
[1] CONST (anon) type_id=2
[2] STRUCT IPv4_h size=28 vlen=13
	version type_id=3 bits_offset=0
	ihl type_id=3 bits_offset=8
	diffserv type_id=3 bits_offset=16
	totalLen type_id=5 bits_offset=32
	identification type_id=5 bits_offset=48
	flags type_id=3 bits_offset=64
	fragOffset type_id=5 bits_offset=80
	ttl type_id=3 bits_offset=96
	protocol type_id=3 bits_offset=104
	hdrChecksum type_id=5 bits_offset=112
	srcAddr type_id=7 bits_offset=128
	dstAddr type_id=7 bits_offset=160
	ebpf_valid type_id=3 bits_offset=192
[3] TYPEDEF u8 type_id=4
[4] INT unsigned char size=1 bits_offset=0 nr_bits=8 encoding=(none)
[5] TYPEDEF u16 type_id=6
[6] INT unsigned short size=2 bits_offset=0 nr_bits=16 encoding=(none)
[7] TYPEDEF u32 type_id=8
[8] INT unsigned int size=4 bits_offset=0 nr_bits=32 encoding=(none)
[9] FUNC_PROTO (anon) return=3 args=(1 iphdr)
[10] FUNC verify_ipv4_checksum type_id=9
[11] PTR (anon) type_id=12
[12] STRUCT __sk_buff size=192 vlen=34
	len type_id=13 bits_offset=0
	pkt_type type_id=13 bits_offset=32
	mark type_id=13 bits_offset=64
	queue_mapping type_id=13 bits_offset=96
	protocol type_id=13 bits_offset=128
	vlan_present type_id=13 bits_offset=160
	vlan_tci type_id=13 bits_offset=192
	vlan_proto type_id=13 bits_offset=224
	priority type_id=13 bits_offset=256
	ingress_ifindex type_id=13 bits_offset=288
	ifindex type_id=13 bits_offset=320
	tc_index type_id=13 bits_offset=352
	cb type_id=14 bits_offset=384
	hash type_id=13 bits_offset=544
	tc_classid type_id=13 bits_offset=576
	data type_id=13 bits_offset=608
	data_end type_id=13 bits_offset=640
	napi_id type_id=13 bits_offset=672
	family type_id=13 bits_offset=704
	remote_ip4 type_id=13 bits_offset=736
	local_ip4 type_id=13 bits_offset=768
	remote_ip6 type_id=16 bits_offset=800
	local_ip6 type_id=16 bits_offset=928
	remote_port type_id=13 bits_offset=1056
	local_port type_id=13 bits_offset=1088
	data_meta type_id=13 bits_offset=1120
	(anon) type_id=17 bits_offset=1152
	tstamp type_id=19 bits_offset=1216
	wire_len type_id=13 bits_offset=1280
	gso_segs type_id=13 bits_offset=1312
	(anon) type_id=21 bits_offset=1344
	gso_size type_id=13 bits_offset=1408
	tstamp_type type_id=23 bits_offset=1440
	hwtstamp type_id=19 bits_offset=1472
[13] TYPEDEF __u32 type_id=8
[14] ARRAY (anon) type_id=13 index_type_id=15 nr_elems=5
[15] INT __ARRAY_SIZE_TYPE__ size=4 bits_offset=0 nr_bits=32 encoding=(none)
[16] ARRAY (anon) type_id=13 index_type_id=15 nr_elems=4
[17] UNION (anon) size=8 vlen=1
	flow_keys type_id=18 bits_offset=0
[18] PTR (anon) type_id=31
[19] TYPEDEF __u64 type_id=20
[20] INT long long unsigned int size=8 bits_offset=0 nr_bits=64 encoding=(none)
[21] UNION (anon) size=8 vlen=1
	sk type_id=22 bits_offset=0
[22] PTR (anon) type_id=32
[23] TYPEDEF __u8 type_id=4
[24] FUNC_PROTO (anon) return=25 args=(11 skb)
[25] INT int size=4 bits_offset=0 nr_bits=32 encoding=SIGNED
[26] FUNC ebpf_filter type_id=24
[27] INT char size=1 bits_offset=0 nr_bits=8 encoding=SIGNED
[28] ARRAY (anon) type_id=27 index_type_id=15 nr_elems=4
[29] VAR _license type_id=28 linkage=1
[30] DATASEC license size=0 vlen=1 size == 0

Error fetching program/map!
Unable to load program

ERROR:Failed to execute the command sequence in namespace 61631
0 packets captured
0 packets received by filter
0 packets dropped by kernel
INFO:Executing command: ip netns pids 61631 | xargs -r kill; ip netns del 61631

fruffy · 2024-05-02T17:23:26Z

@thomascalvert-xlnx I disabled method signature generation for eBPF and only enabled it for uBPF. Until we fix the way we load eBPF programs (or move on from Ubuntu 20.04) I do not see a better way to do this. Please take a second look.

fruffy · 2024-05-07T23:21:13Z

@thomascalvert-xlnx I will merge this tomorrow if you have no further comments.

thomascalvert-xlnx

Sorry for the delay, we had a bank holiday here and this thread fell out of my cache!

LGTM now, thank you @fruffy :)

fruffy added the ebpf Topics related to the eBPF back end label Apr 29, 2024

fruffy requested review from osinstom and tatry April 29, 2024 15:01

fruffy mentioned this pull request Apr 29, 2024

Add -mbmi flag to Abseil for GCC versions greater than 14. #4638

Merged

thomascalvert-xlnx reviewed Apr 30, 2024

View reviewed changes

Fix missing declaration of custom externs in the generated eBPF/uBPF …

54d0981

…header file.

fruffy force-pushed the fruffy/ebpf_extern_definitions branch 2 times, most recently from 229af1a to cb0a5c2 Compare April 30, 2024 15:31

Add library functions instead of checking for file names.

4986811

fruffy force-pushed the fruffy/ebpf_extern_definitions branch from cb0a5c2 to 4986811 Compare April 30, 2024 15:37

Do not use static for extern definition if we generate a declaration.

cec9561

fruffy force-pushed the fruffy/ebpf_extern_definitions branch from a033738 to 782f630 Compare April 30, 2024 18:59

Use extern objects for the eBPF test framework.

333a083

fruffy force-pushed the fruffy/ebpf_extern_definitions branch from 782f630 to 333a083 Compare April 30, 2024 19:22

thomascalvert-xlnx approved these changes May 1, 2024

View reviewed changes

fruffy added the run-ubuntu18 Use this tag to trigger a Ubuntu-18 CI run. label May 1, 2024

fruffy force-pushed the fruffy/ebpf_extern_definitions branch from 88cac13 to 915199d Compare May 1, 2024 14:25

Try an inline declaration to fix Ubuntu 20.04 problems?

0378e15

fruffy force-pushed the fruffy/ebpf_extern_definitions branch from 915199d to 0378e15 Compare May 1, 2024 15:08

fruffy mentioned this pull request May 1, 2024

Do not throw an error when warnings are emit for uBPF program compilation. #4648

Closed

Use a different libbpf version?

95778d6

fruffy force-pushed the fruffy/ebpf_extern_definitions branch from 47ec5db to 95778d6 Compare May 2, 2024 13:35

fruffy force-pushed the fruffy/ebpf_extern_definitions branch 5 times, most recently from 01ff92f to 95778d6 Compare May 2, 2024 16:01

Revert to old techniques for eBPF code.

c030ea0

fruffy requested a review from thomascalvert-xlnx May 2, 2024 21:12

fruffy mentioned this pull request May 8, 2024

Link against the Boehm-Demers-Weiser Garbage Collector using FetchContent. #3930

Merged

thomascalvert-xlnx approved these changes May 8, 2024

View reviewed changes

fruffy added this pull request to the merge queue May 8, 2024

Merged via the queue into main with commit f26cd96 May 8, 2024
17 checks passed

fruffy deleted the fruffy/ebpf_extern_definitions branch May 8, 2024 13:54

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix missing declaration of custom externs in the generated eBPF/uBPF header file. #4644

Fix missing declaration of custom externs in the generated eBPF/uBPF header file. #4644

fruffy commented Apr 29, 2024

thomascalvert-xlnx commented Apr 29, 2024

fruffy commented Apr 29, 2024

thomascalvert-xlnx left a comment

fruffy commented Apr 30, 2024

fruffy commented Apr 30, 2024

fruffy commented Apr 30, 2024

thomascalvert-xlnx left a comment

fruffy commented May 1, 2024

thomascalvert-xlnx commented May 2, 2024

fruffy commented May 2, 2024

fruffy commented May 2, 2024 •

edited

Loading

fruffy commented May 2, 2024

fruffy commented May 7, 2024

thomascalvert-xlnx left a comment

Fix missing declaration of custom externs in the generated eBPF/uBPF header file. #4644

Fix missing declaration of custom externs in the generated eBPF/uBPF header file. #4644

Conversation

fruffy commented Apr 29, 2024

thomascalvert-xlnx commented Apr 29, 2024

fruffy commented Apr 29, 2024

thomascalvert-xlnx left a comment

Choose a reason for hiding this comment

fruffy commented Apr 30, 2024

fruffy commented Apr 30, 2024

fruffy commented Apr 30, 2024

thomascalvert-xlnx left a comment

Choose a reason for hiding this comment

fruffy commented May 1, 2024

thomascalvert-xlnx commented May 2, 2024

fruffy commented May 2, 2024

fruffy commented May 2, 2024 • edited Loading

fruffy commented May 2, 2024

fruffy commented May 7, 2024

thomascalvert-xlnx left a comment

Choose a reason for hiding this comment

fruffy commented May 2, 2024 •

edited

Loading