Unable to flash Jetson Orin NX

[NavSriv]

I am using Diamond Systems Jetson Orin NX 16GB board for development. It has 128GB SSD on it. I am building image through

poky -> langdale (because boards boots up with Jetpack BSP version -35.3.1, Kernel 5.10.104)
meta-tegra -> langdale

I have following change in my local.conf (file is attached for reference)
local.txt

MACHINE ??= "p3509-a02-p3767-0000"
TNSPEC_BOOTDEV:p3509-a02-p3767-0000 = "nvme0n1p1"

I had changed the jumpers to put it in recovery mode, extracted tegraflash file and executed following commands to replace only rootfs-

sudo ./initrd-flash --skip-bootloader

but it failes with below errors-

sudo ./initrd-flash --skip-bootloader
Starting at 2024-02-08T12:54:10-06:00
Machine:       p3509-a02-p3767-0000
Rootfs device: nvme0n1p1
Found Jetson device in recovery mode at USB 6-2
== Step 1: Signing binaries at 2024-02-08T12:54:10-06:00 ==
/home/nsrivastava/Image/tegra234-flash-helper.sh: line 322: warning: command substitution: ignored null byte in input
== Step 2: Boot Jetson via RCM at 2024-02-08T12:54:27-06:00 ==
Found Jetson device in recovery mode at USB 6-2
== Step 3: Sending flash sequence commands at 2024-02-08T12:54:30-06:00 ==
Waiting for USB storage device flashpkg from 6940020b......................

I also tried sudo ./doflash.sh which executed successfully but board didn't come up. I have kernel patches for this board which I have not applied yet. I am assuming, not applying patches might make interfaces not to work but shouldn't stop the booting of the board.

Appreciate the help.

Please find the similar issue in reference #1304

[dwalkes]

I suspect you will want the serial console output to help troubleshoot this, or at least that's where I would suggest starting.

[NavSriv]

I am trying to get an answer from Diamond systems on how to enable serial. Until I receive more information, I will pause this thread.
My worry was sudo ./initrd-flash --skip-bootloader should not have stopped. But indeed, let me get some more data to discuss efficiently.
Thank you

[joekale]

looking at the user manual for the board you linked. I believe you just need to connect some jumper wires to the J7 header to get debug serial. I didn't see any other place where they mentioned a serial connection so I assume one of these are the debug port.

[NavSriv]

yeah, that's not working somehow. Not sure why, I assumed it must be straight forward. I will be receiving new cable tomorrow so probably I will give it a new try. It's a newly purchased board and we are trying our hands on it for the first time.

[dwalkes]

My worry was sudo ./initrd-flash --skip-bootloader should not have stopped

At this step

Waiting for USB storage device flashpkg from 6940020b......................

The host is waiting for the init-flash.sh script, running from a dedicated initramfs. If there's something wrong with your boot or with the setup of the hardware in this initramfs, the initrd-flash step will hang up here. Hopefully boot logs will help you understand why but, if not, comparing boot logs with whatever NVMe flash logs you see when running successfully with the NVIDIA generated sequence and the manufacturer's instructions might help to explain the issue.

[NavSriv]

Finally I got the serial console up. Their custom board expected TTL to USB for serial.
I have analyzed the log wrt normal boot logs, langdale generated image hung at the below point

I> Task: Initialize SBSA UART CAR (0x50001f48)
I> Task: Initialize CPUBL Params (0x50016f88)
I> CPUBL-params @ 0x436000000
I> Task: Prepare eeprom data (0x50016d54)
E> I2C: slave not found in slaves.
E> I2C: Could not write 0 bytes to slave: 0x00ae with repeat start true.
E> I2C_DEV: Failed to send register address 0x00000000.
E> I2C_DEV: Could not read 256 registers of size 1 from slave 0xae at 0x00000000 via instance 0.
E> eeprom: Failed to read I2C slave device
C> Task 0x0 failed (err: 0x1f1e050d)
E> Top caller module: I2C_DEV, error module: I2C, reason: 0x0d, aux_info: 0x05
I> Busy Spin

Right now I am trying to get it fixed by following this link. I will apply Diamond systems patches and see if it can help. Will update the results here

[NavSriv]

The host is waiting for the init-flash.sh script, running from a dedicated initramfs. If there's something wrong with your boot or with the setup of the hardware in this initramfs, the initrd-flash step will hang up here. Hopefully boot logs will help you understand why but, if not, comparing boot logs with whatever NVMe flash logs you see when running successfully with the NVIDIA generated sequence and the manufacturer's instructions might help to explain the issue.

I got the bootlog and it hung at the below point -

[    6.233636] tegra_actmon d230000.actmon: initialization Completed for the device mc_all
[    6.241360] hvc_sysfs: hypervisor is not present
[    6.281802] ALSA device list:
[    6.281893]   No soundcards found.
[    6.282632] Freeing unused kernel memory: 3840K
[    6.282785] Run /bin/sh as init process
/bin/sh: can't access tty; job control turned off
/ # [    6.849385] mmc1: SDHCI controller on 3400000.sdhci [3400000.sdhci] using ADMA 64-bit
[    9.366490] usb usb2-port1: Cannot enable. Maybe the USB cable is bad?
[    9.366703] usb usb2-port1: config error
[   13.438502] usb usb2-port1: Cannot enable. Maybe the USB cable is bad?
[   13.438717] usb usb2-port1: config error

Though default image log shows

[   12.368553] Freeing unused kernel memory: 3968K
[   12.373285] Run /init as init process
[   12.389558] Root device found: PARTUUID=d528a3d9-5112-4b10-b733-43e46e749559
[   12.672350] EXT4-fs (nvme0n1p1): 5 orphan inodes deleted

I am assuming kernel is up and I have issues with mounting rootfs.

Thanks a lot for guiding me on how to debug this issue.

[joekale]

Just in case you haven't seen this yet. the Prerequisites section of the initrd flashing does describe that you need to disable your system's automount for the flashing to work. the wiki page uses dconf on ubuntu I believe but I use Pop!_OS and had to use gsettings to disable automounting of removable media.

[NavSriv]

I did that. Didn't help me here. Still it is stuck at

== Step 3: Sending flash sequence commands at 2024-02-29T13:23:20-06:00 ==
Waiting for USB storage device flashpkg from 6940020b......................

[joekale]

which command did you run on what operating system?

[NavSriv]

I checked the option "Never prompt or start programs on media insertion." in settings->Removable media. Then rebooted the system once and tried initrd-flash again.

[joekale]

what is the result of both

dconf read /org/gnome/desktop/media-handling/automount

and

gsettings get org.gnome.desktop.media-handling automount

?
For me the result of both is false. I'm not sure what that settings panel changes under the hood but I do know that I had to set at least the second setting to false for initrd flashing to work for me.

[NavSriv]

Just in case you haven't seen this yet. the Prerequisites section of the initrd flashing does describe that you need to disable your system's automount for the flashing to work. the wiki page uses dconf on ubuntu I believe but I use Pop!_OS and had to use gsettings to disable automounting of removable media.

Yes, I haven't done that. I will try and update it here. Thank you

[NavSriv]

what is the result of both

dconf read /org/gnome/desktop/media-handling/automount

and

gsettings get org.gnome.desktop.media-handling automount

Below is the result of the commands-

nstava@BTLinux:~$ dconf read /org/gnome/desktop/media-handling/automount
false
nstava@BTLinux:~$ gsettings get org.gnome.desktop.media-handling automount
false

Also I had set them explicitly again and rebooted the system then tried but no luck..

On serial console I see

[    6.282785] Run /bin/sh as init process
/bin/sh: can't access tty; job control turned off

Suspecting if mounting has something to do with it.

[NavSriv]

I had this line in local.conf..

KERNEL_ARGS:append = " rdinit=/bin/sh"

Commenting it out worked for me. Everything with initrd-flash logs looks fine but final status was failed. Though board is up so I will check why status is failed.

Thank you everyone for your help. Closing this ticket.