CM4 cannot boot from Intel M10 nvme optane drive

[chemwolf6922]

Describe the bug

The Intel M10 is an nvme optane drive running on a pcie3.0x2 bus. Please check Intel® Optane™ Memory M10 Series for details.

The CM4 can recognize and use the drive once booted. Meaning the kernel can use the drive.
However the bootloader cannot. It will show "Failed to open nvme" and cannot boot.

The M10 can be used as a boot drive on other platforms. I have tested the following:

• Ubuntu server running on a Intel xeon e3 system.
• Ubuntu running on the radax rock 5b+ (rk3588)
• Rpi os on CM4. M10 on a usb to nvme m.2 converter (JMS583). (Yes, this works... And no, I'll not accept that as a solution.)

I'm aware there are forum posts over the years. But all of those posts landed on a simple and irresponsible "incompatible" w/o any official response. Thus, I'm reporting this as a bug since it is not marked as by design. (If it is by design, please let people know why. And if we can help.)

Steps to reproduce the behaviour

1. Burn rpi os onto the M10 drive using the rpi imager.
2. Update the CM4's bootloader to the most recent stable version. (Supports nvme boot, tested with other nvme drives.)
3. Plug the CM4 and the M10 drive to the IO board. The M10 drive needs to use a pcie x1 to m.2 converter.
4. Plug the power cable.
5. Boot fails with error "Failed to open nvme"

Device (s)

Raspberry Pi CM4 Lite

Bootloader configuration.

[all]
BOOT_UART=0
WAKE_ON_GPIO=1
POWER_OFF_ON_HALT=0

# Try SD first (1), followed by, USB PCIe, NVMe PCIe, USB SoC XHCI then network
BOOT_ORDER=0xf25641

# Set to 0 to prevent bootloader self-updates (without recovery.bin)
# For remote units EEPROM hardware write protection should be used.
ENABLE_SELF_UPDATE=1

# Set to 1 to disable the bootloader HDMI diagnostics screen AND
# network install e.g. for kiosk devices.
DISABLE_HDMI=0

System

Since the system won't boot, this is from a sd card boot with the M10 pluged in.
Comment too long. Please see the attached file.
log.txt

Bootloader logs

(By the way, referencing the UART as 14 and 15 can be confusing. I'll suggest use gpio14 gpio15 or pin8 pin10)

RPi: BOOTLOADER release VERSION:e608a69d DATE: 2024/04/15 TIME: 14:12:14
BOOTMODE: 0x06 partition 0 build-ts BUILD_TIMESTAMP=1713186734 serial 72d52577 boardrev b03141 stc 476529
PM_RSTS: 0x00001000
part 00000000 reset_info 00000000
uSD voltage 3.3V
Initialising SDRAM 'Samsung' 16Gb x1 total-size: 16 Gbit 3200
DDR 3200 0 0 16 152

Boot mode: SD (01) order f2564
SD HOST: 200000000 CTL0: 0x00800000 BUS: 400000 Hz actual: 390625 HZ div: 512 (256) status: 0x1fff0000 delay: 276
SD HOST: 200000000 CTL0: 0x00800f00 BUS: 400000 Hz actual: 390625 HZ div: 512 (256) status: 0x1fff0000 delay: 276
EMMC
SD retry 1 oc 0
SD HOST: 200000000 CTL0: 0x00800000 BUS: 400000 Hz actual: 390625 HZ div: 512 (256) status: 0x1fff0000 delay: 276
SD retry 2 oc 0
SD HOST: 200000000 CTL0: 0x00800000 BUS: 400000 Hz actual: 390625 HZ div: 512 (256) status: 0x1fff0000 delay: 276
SDV1
SD CMD: 0x371a0010 (55) 0x0 0x1fff0001
Failed to open device: 'sdcard' (cmd 371a0010 status 1fff0001)
Retry SD 1
SD HOST: 200000000 CTL0: 0x00800000 BUS: 400000 Hz actual: 390625 HZ div: 512 (256) status: 0x1fff0000 delay: 276
SD HOST: 200000000 CTL0: 0x00800f00 BUS: 400000 Hz actual: 390625 HZ div: 512 (256) status: 0x1fff0000 delay: 276
EMMC
SD retry 1 oc 0
SD HOST: 200000000 CTL0: 0x00800000 BUS: 400000 Hz actual: 390625 HZ div: 512 (256) status: 0x1fff0000 delay: 276
SD retry 2 oc 0
SD HOST: 200000000 CTL0: 0x00800000 BUS: 400000 Hz actual: 390625 HZ div: 512 (256) status: 0x1fff0000 delay: 276
SDV1
SD CMD: 0x371a0010 (55) 0x0 0x1fff0001
Failed to open device: 'sdcard' (cmd 371a0010 status 1fff0001)
Boot mode: USB-MSD (04) order f256
USB xHC init failed
Boot mode: NVME (06) order f25
Failed to open device: 'nvme'
Retry NVME 1
Failed to open device: 'nvme'
Boot mode: BCM-USB-MSD (05) order f2
XHCI-STOP
xHC ver: 272 HCS: 01000140 0c0000f1 07ff000a HCC: 0220fe65
xHC ports 1 slots 64 intrs 1
USB2[1] 000206e1 connected
USB2[1] 00200e03 connected enabled
USB2 root HUB port 1 init
DEV [01:00] 2.00 000000:01 class 9 VID 0424 PID 2514
HUB init [01:00] 2.00 000000:01
HDMI0 edid block 0 offset 0
00ffffffffffff0004721107260b0094
281d0103803c22782e9325ad4f44a926
0d5054bfef80714f8140818081c08100
9500b300d1c0565e00a0a0a029503020
350055502100001a7ee500a0a0a03750
3034380055502100001a000000fd0030
901fde3c000a202020202020000000fc
0056473237325520500a202020200127
HDMI0 edid block 1 offset 128
020354f1559005040302011112131406
0715061f205a5d5e5f61230907078301
00006d030c001000383c200060030201
67d85dc401788003681a000001013090
ede40f000010e305c301e2002be60607
0161561c7ee500a0a0a0375030343800
55502100001a5aa000a0a0a046503038
350055502100001a00000000000000b5
HDMI0: best-mode 1 (limit 1) 1280x720 60 Hz CEA modes fe003f8001000000000000e402000000 extensions 1

Following logs are network boot and retries.

USB boot

No response

NVMe boot

Comment too long. Please see the attached file.
log.txt

Network (TFTP boot)

No response

[chemwolf6922]

One little thing since some posts mentioned this.
The clkreq line is not connected when using the CM4-IO board since the pcie port does not have that pin. But using a base board with clkreq pin connected also doesn't work. So, this pin does not seem to be the problem.

[pelwell]

How is the drive attached to the CM4? You don't say.

By the way, referencing the UART as 14 and 15 can be confusing

Where do you see this?

[chemwolf6922]

How is the drive attached to the CM4? You don't say.

By the way, referencing the UART as 14 and 15 can be confusing

Where do you see this?

In the issue template.

The drive is attached to the CM4 with a pcie to m.2 converter on the CM4 IO board.

[chemwolf6922]

Like this.
The converter works well with other drives. And the M10 can also be recognized as a second drive with this connection.

[timg236]

I'm pretty sure we don't support Optane

[chemwolf6922]

I'm pretty sure we don't support Optane

Is there any particular reason why? Since this is also a nvme drive.

[timg236]

I think they behave slightly differently - have you tried booting to RPi OS from SD or EMMC and verifying that it works from there?

[chemwolf6922]

I think they behave slightly differently - have you tried booting to Linux from SD or EMMC and verifying that it works from there?

Yes I have. As I stated in the issue description 😂

[chemwolf6922]

I can partition it, create file systems and read/write w/o any problem.

[timg236]

Ok, so PCIe is working. I think unfortunately, this is something that would require custom firmware support and therefore a fairly low priority.

[chemwolf6922]

Thanks for the reply and your work.
Is there any way that the community can help with the bootloader?

And feel free to close this one if it's not your priority.

[timg236]

It looks like it failed pretty early, so if anyone knows if these drive need a different init-sequence compared to generic NVMe or alternatively have firmware which take takes longer to respond after power on then that would be useful

[chemwolf6922]

Since the kernel and u-boot both supports Optane, I took a glance at both but there does not seem to be any specific treatment for these drivers. The kernel's code is more complicated and I did not check it thoroughly. The u-boot driver is simpler.
https://github.com/u-boot/u-boot/tree/master/drivers/nvme
If you are interested, maybe do a comparison on what might be different/missing on the rpi bootloader? Thanks.

(Proof of u-boot supporting Optane: I have a rock 5b+ (rk3588). Its bootloader is u-boot. And I can boot it off a Optane drive. )

[peterharperuk]

I think we'd have to get hold of one to see why it doesn't work. It might be that it doesn't like being used so soon after power up.

[timg236]

This sounds like a feature request, if someone gets time to look at this then I suggest we would initially release on Pi5 backport to CM4 etc. BCM2711 has to duplicate the NVMe driver updates in start.elf so adding features is more complicated for software releases.

[p4nda]

When testing the Optane P1600x 58GB with a CM4 via the Super S6C NVMe slot, it was possible to install Fedora 40 from the OS installer booted via EDK2 UEFI, but it wouldn't boot from the NVMe drive. Booting the same OS from an SD card allowed the NVMe drive to function normally. This might be due to the single PCIe 2.0 lane drawing less power than required by spec.

[chemwolf6922]

Some updates: It might be the refclk on and reset delay.

I draw a custom board and the M10 is not recognized even as a normal drive. (Probably related to the power rail timings). I have to set pcie_tperst_clk_ms to 500 to get it to work. I have not tested smaller numbers though.

@peterharperuk @timg236 Is it possible for you guys to maybe generate a special version of the boot loader where there is such a delay so I may test it out? Thanks.