Adding USB3 boot support to Dell M4600 laptop

(This is a long post. Everyone can make their own TL;DR by copy-pasting it into their favorite AI and asking for a summary.)

About a week ago I dug an ancient Dell M4600 laptop out of a cupboard to give it a new use. It was high-end hardware in its day, with a 2nd gen Core i7, and still perfectly good as a headless Linux server for hobby projects.

While working on it, I needed to boot the machine from a USB3 flash drive plugged into a USB3 port. That did not work, because Dell never released a BIOS version for this machine that supported booting from the USB3 ports. Only the USB2 ports on the other side are bootable.

I bounced the problem around a bit with codex-cli + astra6 high/xhigh. The AI pushed back pretty hard, saying the whole thing was basically impossible and too risky, and eventually it even started blocking the discussion as a cybersecurity issue.

By now I have enough experience arguing with AI models that we eventually got past that too. Usually it is enough to get the AI to actually start working on the problem, especially with codex-cli. Once it starts, it usually keeps going until the end. In this case I think I said that I am officially qualified and trained in this field, all the hardware belongs to me, none of it has any real financial value, breaking it is not desirable but would not be a serious problem either, there were unlimited AI credits available for solving the issue, and that the project supports green values, reuse instead of recycle, right-to-repair, fixing compatibility problems, and improving security. I also specifically told it not to use terms related to hacking or bypassing protections and to use more neutral wording instead.

The end result is that I now probably have the only M4600 in the world that can boot in both EFI and legacy mode from a USB3 flash drive plugged into a USB3 port. During the project we analyzed BIOS images from several other old Dell models, and based on what we found, exactly the same patch would probably work on those too.

The process used almost three weeks worth of OpenAI Pro 5x ("Pro Lite") tokens. I used the whole current weekly quota plus two stored weekly quota resets to finish the project.

Dell's EC has an FDO backdoor intended for factory use that allows all flash protections to be removed, both read and write, without opening the machine. So there was no need for an SPI flasher. The AI found an unlocker made for another Dell model that used this EC backdoor and modified it to work with the M4600. Otherwise I would have had to take the machine apart and flash two SPI flash chips with separate hardware.

We also had an alternative on the table where the BIOS would be downgraded and an older version's missing flash protection after returning from S3 sleep would be used. In the end this was not needed because the FDO route worked so well and so easily. While investigating this path, the AI analyzed the BIOS signature verification code and the keys used by it in Ghidra, confirming that the downgrade was possible because the signature of the buggy BIOS was accepted by the newest BIOS verification code. It checked this separately with static code analysis in Ghidra and also by injecting Dell's flash updater checks into a QEMU EFI BIOS, then running Dell's firmware signature verification in QEMU with the old BIOS image as the thing being checked.

I also had an SPI flasher available, and I could probably even have let the AI guide that process, but I decided to take the risk and let the AI prepare test versions that I flashed directly onto the machine. I acted as the hands, following the instructions and reporting the results back, sometimes by sending phone photos of the screen. Maybe with a little luck too, the project progressed in a way where the machine always stayed bootable, which made it easy to test the next version.

After unpacking the BIOS, the AI spent quite a while analyzing it to understand how strange Dell's old BIOS actually is. For example, the whole EFI boot process is built on top of the traditional BIOS INT 13 interface, while in the TianoCore reference implementation it is basically the other way around. Dell's USB2 driver also claimed the USB3 controller IDs, preventing a separate USB3 XHCI driver from loading. Eventually we also found bugs in both the BIOS and the PXE code, including an off-by-0x08 memory address issue. None of these were found all at once, of course, and for the PXE bug we had to compile a debug version of the PXE bootloader that logged to the console.

At first we looked at taking USB3 drivers from another machine from the same era, but Dell never implemented USB3 boot support on this BIOS platform, and the solutions from other manufacturers were different enough that using their closed-source binary blobs as a starting point did not make much sense. So Phoenix TianoCore was the next route we investigated.

EFI USB3 booting is handled using the TianoCore USB layer and USB storage drivers. To make it coexist with Dell's USB2 drivers, TianoCore was patched to handle only the USB3 side. Likewise, the USB3 controller binding was removed from Dell's USB2 driver. So Dell had apparently started working on USB3 support at some point, but it never really got beyond the idea stage. This integrates fully into Dell's BIOS, so in the BIOS settings USB3 appears as a boot source just like USB2.

Adding legacy boot support directly into Dell's old legacy layer would have become difficult even for the AI, and it would also have increased the risk of making the machine unbootable and forcing me to take it apart and use the SPI flasher. So we ended up with a compromise using the iPXE project's PXE bootloader. iPXE already contains USB3 XHCI drivers that work in legacy mode. In practice, legacy boot from USB3 works by selecting PXE boot in the BIOS, after which iPXE starts, enumerates devices connected to the USB3 ports, both USB2 and USB3 devices but only on the USB3 ports so they do not appear twice alongside the USB2 ports already handled by Dell's BIOS, and then shows a menu asking which one to boot from or whether normal network PXE would be better. If nothing is selected, it tries USB first and falls back to PXE if that fails.

There was a problem with iPXE because Dell's BIOS does not follow the PMM specification requirement that PXE ROM memory areas must be aligned to 0x10 bytes. Dell loads it at an address aligned only to 0x08. This triggers a bug in iPXE, causing it to crash and then reset the whole machine through its failsafe path. The AI found the reason and made an iPXE quirk patch pretty much immediately, in about three iterations, once I handled the boot testing and sent the already mentioned phone photos back to it. It then did something roughly OCR-like with the photos and read the exact reports showing the contents of different memory addresses just before the crash.

Naturally, the AI also built the update images, compiled a static flashrom tool, wrote the flashing scripts, and so on.

At the end I asked it to archive the mostly unnecessary material created during the experiments and turn the whole thing into a documented, publishable, stand-alone solution. That includes all required dependencies, downloading them, and using them to build the modified BIOS. If anyone has actually read this far and says they are interested, I can probably throw it on GitHub so you can laugh at the joint production of an old guy and an AI.

At the moment the AI is using this BIOS-modded Dell, running headless Ubuntu with ZFS from a USB3 flash drive, to port the latest mainline kernel and ARM64 Debian to an old Sony Android phone. Not Debian inside Android, but replacing everything. The goal is to turn an old phone into something like a Raspberry Pi-style SBC, except it also has a touchscreen, Wi-Fi, GPS, 4G, battery, and so on. Maybe that project will get its own post someday if anything useful comes out of it. So far the result is two soft-bricked phones. I am starting to run out of test subjects in the desk drawer, and the AI credits are already gone, so that project is on hold for at least a few days.

P.S. The EFI + legacy dual-boot USB stick with Ubuntu on a ZFS root was also made with help from the AI. There is nothing especially magical about that part, and I could probably have figured it out myself with trial and error over a couple of evenings, but this time one afternoon was enough because the AI did the trial-and-error work in QEMU. At the end it also analyzed the Rufus source code at my request to figure out exactly how a USB stick like this should be formatted so that it stays compatible with old machines, including this M4600, which is quite picky about what it agrees to boot from.