HSS Flash load fail

[ken-hutchinson]

We're seeing a problem with flash load on boot. This is a custom board with MPFS250T and MT25Q flash. On most boards, we were seeing about 5-10% boot failure. The HSS is done, but application crashes on start. One board, once it warmed up, failed 90% of the time. We could see the HSS thought it was done (final debug msg), but the flash was still reading (scope probe on SPI bus). So when the application starts, it see uninitialized ram. Was able to trace the problem to the MSS_SYS_spi_copy returning before the flash copy was done. We are setting MSS_SYS_SERVICE_POLLING_MODE, so it should be blocking.

Added a check in getBootImageFromSpiFlash_, if the copy doesn't take as long as it should, add appropriate HSS_SpinDelay_MilliSecs
This has reduced the failures about 90%. The remaining failures can't read the header info (magic number is zero).

Anyone seen this or have any suggestions?

Thanks

[griffini]

Hi @ken-hutchinson, what version of the HSS are you using?

[ken-hutchinson]

I think I have the latest.
v2025.07 with two patches:
"spi-copy-error-manual-copy.patch"
"allow-flash-to-ddr-copy-to-finish"

[griffini]

Hmmm... I can't say I've seen this, and I did give a go at trying to reproduce, but failed... Reproduction could maybe be down to timing on specific boards.

If MSS_SYS_SERVICE_POLLING_MODE is set, the expectation is that MSS_SYS_spi_copy should block until completion. The fact that it’s returning early suggests either a race condition or an issue in the polling loop. It might be worth double-checking the service mode configuration and any recent changes in the HSS patches - for example git hashes 5e45b11196c809e8a23457af0ad7a3f1dff7a7e8 and ea5b6080d91cf9b67d5fd5cb922d4006848a641a...

I've been wondering about another angle: cache coherency and am wondering about cache flushing. If the flash-to-DDR copy completes but stale data remains in L2, the application could see uninitialized memory. I would suggest flushing before the SPI copy, to see if it makes a difference for you. The recommended way for deterministic eviction from L2 is:

• Ensure only one way is enabled during the sweep.
• After restoring the way mask, follow up with fence.i and sfence.vma
• Also consider flushing the data cache (dcache) explicitly before starting the application

For example:

    // flush L2 Cache, way-by-way
    // see https://forums.sifive.com/t/flush-invalidate-l1-l2-on-the-u54-mc/4483/9
    //
    // the thing to be wary of is the policy is random replacement by way,
    // so there must be only 1 way enabled at a time...
    // then for each way individually, still need to go through ~2MiB
    // (2MiB/16 per way) of reads to safely ensure the L2 is cleared...
    //

    for (int wayMaskN = 0; wayMaskN < LIBERO_SETTING_WAY_ENABLE; wayMaskN++) {
        // disable evictions from all but WayMaskN
        __atomic_store_8(&CACHE_CTRL->WAY_MASK_E51_DCACHE, wayMaskN, __ATOMIC_RELAXED);

        // read 2MiB/16 from L2 zero device
        for (int i = 0; i < 131*1024; i+=8) { (void)*(volatile uint64_t *)(ZERO_DEVICE_BOTTOM + i); };
    }

    // restore WayMask values...
    __atomic_store_8(&CACHE_CTRL->WAY_MASK_E51_DCACHE, LIBERO_SETTING_WAY_MASK_E51_DCACHE, __ATOMIC_RELAXED);
    __asm("fence.i"); // flush I$
    __asm("sfence.vma x0, x0"); // flush any local hardware caches related to address translation. (CIP-1200)

Your magic number reading zero -- this could indicate partial or corrupted reads. If the header fetch happens before the cache flush or before the copy truly completes, you’d probably see zeros. Adding the flush sequence before reading, and a memory barrier (__asm volatile("fence rw,rw");) after the copy and before validating the header might help?

[vfalanis]

Hi @ken-hutchinson,

Could you please create a support case in the Microchip Support website so that a member of our support team can provide further assistance?

Here are some instructions on how to create a ticket:

Thanks