SPI flash contains UEFI firmware and a small Linux kernel+initramfs+grub (you get these three after building image-spi image out of the BSP).
The SPI grub looks for boot/grub/grub.conf on the first active partition of the SD card or USB thumbdrive upon boot. If it finds it - that config is used instead of the default one. If it doesn't find it - the default one is used and that by default boots the SPI Linux image.
The symptoms you have IMHO suggest that your SD card (after you've restored the Debian on it) is not recognized as bootable by the UEFI firmware. Try checking if partitions look ok on the card, using e.g. fdisk.