FreeBSD-10-STABLE hangs when booting from BeagleBone Black eMMC

Asked by 3 months ago
After some success with 11-CURRENT on the BBB/eMMC, I switched back to 10-STABLE but after building a crotchet-freebsd image (using Patrick's script), I can't get it to boot from the eMMC. The image works ok on the SD card, but if I either use the new copy-to-emmc.sh script, or build an eMMC specific image (using BEAGLEBONE_BOOT_EMMC=y), it hangs during the boot (below). It seems to fail when trying to access /boot/defaults/loader.conf Any ideas welcome! U-Boot SPL 2013.04 (Apr 18 2014 - 20:25:05) OMAP SD/MMC: 0 reading bb-uboot.img reading bb-uboot.img U-Boot 2013.04 (Apr 18 2014 - 20:25:05) I2C: ready DRAM: 512 MiB WARNING: Caches not enabled MMC: OMAP SD/MMC: 0, OMAP SD/MMC: 1 Using default environment musb-hdrc: ConfigData=0xde (UTMI-8, dyn FIFOs, HB-ISO Rx, HB-ISO Tx, SoftConn) musb-hdrc: MHDRC RTL version 2.0 musb-hdrc: setup fifo_mode 4 musb-hdrc: 28/31 max ep, 16384/16384 memory USB Peripheral mode controller at 47401000 using PIO, IRQ 0 musb-hdrc: ConfigData=0xde (UTMI-8, dyn FIFOs, HB-ISO Rx, HB-ISO Tx, SoftConn) musb-hdrc: MHDRC RTL version 2.0 musb-hdrc: setup fifo_mode 4 musb-hdrc: 28/31 max ep, 16384/16384 memory USB Host mode controller at 47401800 using PIO, IRQ 0 Net: <ethaddr> not set. Validating first E-fuse MAC cpsw, usb_ether Hit any key to stop autoboot: 0 mmc1(part 0) is current device SD/MMC found on device 1 reading bb-uEnv.txt reading bbubldr 247304 bytes read in 35 ms (6.7 MiB/s) reading bboneblk.dtb 15278 bytes read in 6 ms (2.4 MiB/s) Booting from mmc ... ## Starting application at 0x88000054 ... Consoles: U-Boot console Compatible API signature found @9f242240 MMC Device 2 not found MMC Device 3 not found MMC Device 2 not found Number of U-Boot devices: 3 FreeBSD/armv6 U-Boot loader, Revision 1.2 (root [ at ] freebsd, Fri Apr 25 20:27:41 EDT 2014) DRAM: 512MB Device: disk MMC Device 2 not found MMC Device 3 not found disk0: device open failed with error=2, handle=1 Device: net cpsw Waiting for PHY auto negotiation to complete. done link up on port 0, speed 100, full duplex

Your Answer

Name:
Reply:

All Answers

Answer by 3 months ago
I only have a BBW, no BBB to play with, so I can't help too much with this. I can say however that if you're having trouble with reading the eMMC in ubldr, the trouble is probably in u-boot. U-boot is still in memory when ubldr is running, and it serves as a kind of mini-bios, providing access to console, disk, and network IO. I heard a rumor a while back that Patrick Kelsey had some patches for u-boot to fix problems with probing for disk devices. It may be that they were only needed for older versions of u-boot, I'm not sure. -- Ian
Answer by 3 months ago
I do have Patrick's fixes for u-boot. Interestingly, the same u-boot (2013.4 + Patrick's patches) works with 11-CURRENT but not 10-STABLE -- that's booting from eMMC, there are no issues booting from SD; also once booted from SD, there is no issues mounting the eMMC and accessing it (well ... aside from lock reversal warnings and the occasional sdhc timeout). Is there a way to debug this? Thanks!
Answer by 3 months ago
Quoted message by Ian Lepore 3 months ago
I only have a BBW, no BBB to play with, so I can't help too much with this. I can say however that if you're having trouble with reading the eMMC in ubldr, the trouble is probably in u-boot. U-boot is still in memory when ubldr is running, and it serves as a kind of mini-bios, providing access to console, disk, and network IO. I heard a rumor a while back that Patrick Kelsey had some patches for u-boot to fix problems with probing for disk devices. It may be that they were only needed for older versions of u-boot, I'm not sure. -- Ian
Just ignore the LORs, or even better, turn off WITNESS. IMO, it has become useless, because nobody fixes them anymore, or even responds to reports of them other than to sometimes post a link to a site that tracks "known LORs", but that site hasn't been updated for like 6 years. Note that this is just my personal opinion, not the position of the freebsd project in general. When it comes to the eMMC timeouts that happen after you've booted, that's something in my arena, but hard for me to debug without eMMC hardware. We've recently had a few changes to both the core sdhci driver and to the ti_sdhci code that glues sdhci to the hardware. Do the timeouts happen often, or is this something you can do to trigger them? If so, it might be interesting to try to revert r264099 and see if the problems go away. Those changes were related to configuring the sd clock. I'm pretty sure the old code was wrong, but the replacement code could have errors too. :) -- Ian
Answer by 3 months ago
Quoted message by Winston Smith 3 months ago
I do have Patrick's fixes for u-boot. Interestingly, the same u-boot (2013.4 + Patrick's patches) works with 11-CURRENT but not 10-STABLE -- that's booting from eMMC, there are no issues booting from SD; also once booted from SD, there is no issues mounting the eMMC and accessing it (well ... aside from lock reversal warnings and the occasional sdhc timeout). Is there a way to debug this? Thanks!
That's kind of what I figured; I will turn off WITNESS. At this point: 1) Timeouts (11-CURRENT) - I have seen them twice under heavy eMMC write load, but they haven't seemed to cause a problem for me - Fabio did report the timeouts followed by a panic 2) I can't boot from eMMC with 10-STABLE (but I can with 11-CURRENT) (both using the *same* u-boot) So: A) Should I go back to 11-CURRENT? (although I need 10-something for Golang) B) How can I/we help debug this further? Thanks! -W
Answer by 3 months ago
Quoted message by Ian Lepore 3 months ago
Just ignore the LORs, or even better, turn off WITNESS. IMO, it has become useless, because nobody fixes them anymore, or even responds to reports of them other than to sometimes post a link to a site that tracks "known LORs", but that site hasn't been updated for like 6 years. Note that this is just my personal opinion, not the position of the freebsd project in general. When it comes to the eMMC timeouts that happen after you've booted, that's something in my arena, but hard for me to debug without eMMC hardware. We've recently had a few changes to both the core sdhci driver and to the ti_sdhci code that glues sdhci to the hardware. Do the timeouts happen often, or is this something you can do to trigger them? If so, it might be interesting to try to revert r264099 and see if the problems go away. Those changes were related to configuring the sd clock. I'm pretty sure the old code was wrong, but the replacement code could have errors too. :) -- Ian
If you need 10 we should probably figure out what the problem is there. If the same u-boot works on 11 and fails on 10, then the difference must be in ubldr, and there certainly have been changes there in 11. The quickest way to test that theory would be to build the image for 10 and then hand-copy the ubldr from an 11 build onto that sdcard and see if it works. If so, we can see about merging some ubldr stuff to 10. It may need to go into the msdos partition (I net-boot all my boards). -- Ian
Answer by 3 months ago
Quoted message by Winston Smith 3 months ago
That's kind of what I figured; I will turn off WITNESS. At this point: 1) Timeouts (11-CURRENT) - I have seen them twice under heavy eMMC write load, but they haven't seemed to cause a problem for me - Fabio did report the timeouts followed by a panic 2) I can't boot from eMMC with 10-STABLE (but I can with 11-CURRENT) (both using the *same* u-boot) So: A) Should I go back to 11-CURRENT? (although I need 10-something for Golang) B) How can I/we help debug this further? Thanks! -W
The issue with booting from eMMC on 10-STABLE is in ubldr. On 10-STABLE, ubldr is hardwired to boot from the first disk device, which will always be the SD card when using a u-boot that has my mmc device enumeration patches (or an equivalent). The changes that Ian and I made to ubldr would have to be MFC'd to 10-STABLE to get the behavior you want without resorting to copying ubldr built from an 11-CURRENT tree to your device. -Patrick
Answer by 3 months ago
Quoted message by Ian Lepore 3 months ago
If you need 10 we should probably figure out what the problem is there. If the same u-boot works on 11 and fails on 10, then the difference must be in ubldr, and there certainly have been changes there in 11. The quickest way to test that theory would be to build the image for 10 and then hand-copy the ubldr from an 11 build onto that sdcard and see if it works. If so, we can see about merging some ubldr stuff to 10. It may need to go into the msdos partition (I net-boot all my boards). -- Ian
That works!!! I'm not sure what MFC'd means, but yes, could we get this in 10-STABLE? It's my understanding that 11-CURRENT won't be released for quite some time, so presumably fixes of this nature would automatically get backported (MFC'd?) to 10-STABLE? Thanks!
Answer by 3 months ago
Quoted message by Patrick Kelsey 3 months ago
The issue with booting from eMMC on 10-STABLE is in ubldr. On 10-STABLE, ubldr is hardwired to boot from the first disk device, which will always be the SD card when using a u-boot that has my mmc device enumeration patches (or an equivalent). The changes that Ian and I made to ubldr would have to be MFC'd to 10-STABLE to get the behavior you want without resorting to copying ubldr built from an 11-CURRENT tree to your device. -Patrick
MFC is Merge From Current. Now that you Patrick have both confirmed that the recent ubldr changes are a good fix, I'll get them merged to 10. -- Ian
Answer by 3 months ago
Quoted message by Patrick Kelsey 3 months ago
The issue with booting from eMMC on 10-STABLE is in ubldr. On 10-STABLE, ubldr is hardwired to boot from the first disk device, which will always be the SD card when using a u-boot that has my mmc device enumeration patches (or an equivalent). The changes that Ian and I made to ubldr would have to be MFC'd to 10-STABLE to get the behavior you want without resorting to copying ubldr built from an 11-CURRENT tree to your device. -Patrick
Okay, all done. As of r265071 all the recent ubldr changes have been merged to the 10-stable branch and should work to boot eMMC on BBB. Unless I missed something, the ubldr in 10-stable should now be the same as the one in -current. -- Ian
Answer by 3 months ago
Quoted message by Winston Smith 3 months ago
That works!!! I'm not sure what MFC'd means, but yes, could we get this in 10-STABLE? It's my understanding that 11-CURRENT won't be released for quite some time, so presumably fixes of this nature would automatically get backported (MFC'd?) to 10-STABLE? Thanks!
Hmmm ... 10-STABLE builds and boots from eMMC, but I'm now running into a vm_fault when it tries (and fails) to set up the ethernet PHY: u-boot: Net: <ethaddr> not set. Validating first E-fuse MAC Phy not found PHY reset timed out kernel: ... cpsw0: <3-port Switch Ethernet Subsystem> mem 0x4a100000-0x4a103fff irq 40,41,42,43 on simplebus0 cpsw0: CPSW SS Version 1.12 (0) cpsw0: Initial queue size TX=128 RX=384 cpsw0: Ethernet address: 90:59:ef:4c:28:d7 cpsw0: Failed to read from PHY. cpsw0: attaching PHYs failed vm_fault(0xc07e2ad0, 0, 1, 0) -> 1 Fatal kernel mode data abort: 'Translation Fault (S)' trapframe: 0xc08e2af0 FSR=00000005, FAR=00000018, spsr=80000093 r0 =c27c8680, r1 =00000000, r2 =00000019, r3 =60000193 r4 =00000000, r5 =c27c8680, r6 =00000006, r7 =c053c8a8 r8 =c27c8680, r9 =c281b28c, r10=c28190c8, r11=c08e2b50 r12=00000000, ssp=c08e2b40, slr=c055a590, pc =c038e374 [ thread pid 0 tid 100000 ] Stopped at device_delete_child+0x14: ldr r1, [r4, #0x018] db> The full bootlog is below. I'm not entirely sure that this isn't something I've broken myself (I was messing with the beaglebone_black.dts file yesterday!), although I thought I had rebuilt everything properly -- and the same image boots from the SD card. Here's the full log: U-Boot SPL 2013.04 (Apr 29 2014 - 08:26:49) OMAP SD/MMC: 0 mmc_send_cmd : timeout: No status update reading bb-uboot.img reading bb-uboot.img U-Boot 2013.04 (Apr 29 2014 - 08:26:49) I2C: ready DRAM: 512 MiB WARNING: Caches not enabled MMC: OMAP SD/MMC: 0, OMAP SD/MMC: 1 Using default environment musb-hdrc: ConfigData=0xde (UTMI-8, dyn FIFOs, HB-ISO Rx, HB-ISO Tx, SoftConn) musb-hdrc: MHDRC RTL version 2.0 musb-hdrc: setup fifo_mode 4 musb-hdrc: 28/31 max ep, 16384/16384 memory USB Peripheral mode controller at 47401000 using PIO, IRQ 0 musb-hdrc: ConfigData=0xde (UTMI-8, dyn FIFOs, HB-ISO Rx, HB-ISO Tx, SoftConn) musb-hdrc: MHDRC RTL version 2.0 musb-hdrc: setup fifo_mode 4 musb-hdrc: 28/31 max ep, 16384/16384 memory USB Host mode controller at 47401800 using PIO, IRQ 0 Net: <ethaddr> not set. Validating first E-fuse MAC Phy not found PHY reset timed out cpsw, usb_ether Hit any key to stop autoboot: 0 mmc1(part 0) is current device SD/MMC found on device 1 reading bb-uEnv.txt reading bbubldr 251703 bytes read in 37 ms (6.5 MiB/s) reading bboneblk.dtb 15278 bytes read in 8 ms (1.8 MiB/s) Booting from mmc ... ## Starting application at 0x88000054 ... Consoles: U-Boot console Compatible U-Boot API signature found @9f242240 FreeBSD/armv6 U-Boot loader, Revision 1.2 (root [ at ] freebsd, Mon Apr 28 22:09:48 EDT 2014) DRAM: 512MB MMC Device 2 not found MMC Device 3 not found MMC Device 2 not found Number of U-Boot devices: 3 U-Boot env: loaderdev not set, will probe all devices. Found U-Boot device: disk Probing all disk devices... Checking unit=0 slice=<auto> partition=<auto>...MMC Device 2 not found MMC Device 3 not found disk0: device open failed with error=2, handle=1 Checking unit=1 slice=<auto> partition=<auto>... good. Loading /boot/defaults/loader.conf /boot/kernel/kernel data=0x468f08+0x17d8dc syms=[0x4+0x84b30+0x4+0x501f3] /boot/kernel/geom_label.ko text=0x50dc data=0x864+0x30 syms=[0x4+0x1020+0x4+0xfe2] Hit [Enter] to boot immediately, or any other key for command prompt. Booting [/boot/kernel/kernel]... Using DTB provided by U-Boot at address 0x0x80000100. Kernel entry at 0x80200100... Kernel args: (null) KDB: debugger backends: ddb KDB: current backend: ddb Copyright (c) 1992-2014 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 10.0-STABLE #0 r265072: Mon Apr 28 22:05:57 EDT 2014 root [ at ] freebsd:/root/Work/crochet-freebsd/work/obj/arm.armv6/usr/src/FreeBSD-stable-10/sys/BEAGLEBONE arm FreeBSD clang version 3.4 (tags/RELEASE_34/final 197956) 20140216 WARNING: WITNESS option enabled, expect reduced performance. CPU: Cortex A8-r3 rev 2 (Cortex-A core) Supported features: ARM_ISA THUMB2 JAZELLE THUMBEE ARMv4 Security_Ext WB disabled EABT branch prediction enabled LoUU:2 LoC:2 LoUIS:1 Cache level 1: 32KB/64B 4-way data cache WT WB Read-Alloc 32KB/64B 4-way instruction cache Read-Alloc Cache level 2: 256KB/64B 8-way unified cache WT WB Read-Alloc Write-Alloc real memory = 536870912 (512 MB) avail memory = 515538944 (491 MB) Texas Instruments AM3358 Processor, Revision ES1.1 random device not loaded; using insecure entropy random: <Software, Yarrow> initialized simplebus0: <Flattened device tree simple bus> on fdtbus0 aintc0: <TI AINTC Interrupt Controller> mem 0x48200000-0x48200fff on simplebus0 aintc0: Revision 5.0 ti_scm0: <TI Control Module> mem 0x44e10000-0x44e11fff on simplebus0 am335x_prcm0: <AM335x Power and Clock Management> mem 0x44e00000-0x44e012ff on simplebus0 am335x_prcm0: Clocks: System 24.0 MHz, CPU 550 MHz am335x_dmtimer0: <AM335x DMTimer> mem 0x44e05000-0x44e05fff,0x44e31000-0x44e31fff,0x48040000-0x48040fff,0x48042000-0x48042fff,0x48044000-0x48044fff,0x48046000-0x48046fff,0x48048000-0x48048fff,0x4804a000-0x4804afff irq 66,67,68,69,92,93,94,95 on simplebus0 Timecounter "AM335x Timecounter" frequency 24000000 Hz quality 1000 Event timer "AM335x Eventtimer0" frequency 24000000 Hz quality 1000 gpio0: <TI General Purpose I/O (GPIO)> mem 0x44e07000-0x44e07fff,0x4804c000-0x4804cfff,0x481ac000-0x481acfff,0x481ae000-0x481aefff irq 96,97,98,99,32,33,62,63 on simplebus0 gpioc0: <GPIO controller> on gpio0 gpiobus0: <GPIO bus> on gpio0 uart0: <TI UART (16550 compatible)> mem 0x44e09000-0x44e09fff irq 72 on simplebus0 uart0: console (115384,n,8,1) ti_edma30: <TI EDMA Controller> mem 0x49000000-0x490fffff,0x49800000-0x498fffff,0x49900000-0x499fffff,0x49a00000-0x49afffff irq 12,13,14 on simplebus0 ti_edma30: EDMA revision 40014c00 sdhci_ti0: <TI MMCHS (SDHCI 2.0)> mem 0x48060000-0x48060fff irq 64 on simplebus0 sdhci_ti1: <TI MMCHS (SDHCI 2.0)> mem 0x481d8000-0x481d8fff irq 28 on simplebus0 mmc0: <MMC/SD bus> on sdhci_ti1 cpsw0: <3-port Switch Ethernet Subsystem> mem 0x4a100000-0x4a103fff irq 40,41,42,43 on simplebus0 cpsw0: CPSW SS Version 1.12 (0) cpsw0: Initial queue size TX=128 RX=384 cpsw0: Ethernet address: 90:59:ef:4c:28:d7 cpsw0: Failed to read from PHY. cpsw0: attaching PHYs failed vm_fault(0xc07e2ad0, 0, 1, 0) -> 1 Fatal kernel mode data abort: 'Translation Fault (S)' trapframe: 0xc08e2af0 FSR=00000005, FAR=00000018, spsr=80000093 r0 =c27c8680, r1 =00000000, r2 =00000019, r3 =60000193 r4 =00000000, r5 =c27c8680, r6 =00000006, r7 =c053c8a8 r8 =c27c8680, r9 =c281b28c, r10=c28190c8, r11=c08e2b50 r12=00000000, ssp=c08e2b40, slr=c055a590, pc =c038e374 [ thread pid 0 tid 100000 ] Stopped at device_delete_child+0x14: ldr r1, [r4, #0x018] db>
Answer by 3 months ago
Quoted message by Winston Smith 3 months ago
That works!!! I'm not sure what MFC'd means, but yes, could we get this in 10-STABLE? It's my understanding that 11-CURRENT won't be released for quite some time, so presumably fixes of this nature would automatically get backported (MFC'd?) to 10-STABLE? Thanks!
This may be the problem -- ubldr's behavior for finding and using dtb files is more complex than it used to be. It may be that it used to claim to find and use the u-boot dtb, but it was really using a static dtb compiled into the kernel. Now if u-boot provides a valid dtb it'll actually get used, even if kernel has a static dtb compiled in. If you remove the u-boot env var 'fdtaddr' then it will fall back to using a compiled-in dtb file instead of a loaded one. What I've been doing these days is installing my dtb files in /boot/modules or /boot/kernel and then setting the dtb file name in a u-boot env var named fdtfile. That makes ubldr find and load the named file in the place it finds the kernel. -- Ian
Answer by 3 months ago
Quoted message by Winston Smith 3 months ago
That works!!! I'm not sure what MFC'd means, but yes, could we get this in 10-STABLE? It's my understanding that 11-CURRENT won't be released for quite some time, so presumably fixes of this nature would automatically get backported (MFC'd?) to 10-STABLE? Thanks!
After more testing, it appears the eMMC was somehow corrupted. I initially used Tim's copy-to-emmc.sh script to set up the eMMC -- I got a couple of lock reversal errors (still haven't figured out how to disable WITNESS via the make cmd line) and saw this issue. I then rewrote the img to /dev/mmcsd1 and it was able to bring up the eMMC. I tried Tim's script again and this time didn't get lock reversal errors and it's also able to boot from eMMC. So I don't know if there was some corruption ... I'll keep testing. But anyway, I can now boot 10-STABLE from eMMC. Thanks! -W
Answer by 3 months ago
Quoted message by Winston Smith 3 months ago
That works!!! I'm not sure what MFC'd means, but yes, could we get this in 10-STABLE? It's my understanding that 11-CURRENT won't be released for quite some time, so presumably fixes of this nature would automatically get backported (MFC'd?) to 10-STABLE? Thanks!
You can't disable WITNESS from the command line, but it should be off by default in the stable branches. In general if there's anything you need to change about a kernel config, you create your own config that includes the standard one and makes changes, like: include BEAGLEBONE nooption WITNESS nodevice usb # and so on -- Ian
Answer by 3 months ago
Quoted message by Winston Smith 3 months ago
That works!!! I'm not sure what MFC'd means, but yes, could we get this in 10-STABLE? It's my understanding that 11-CURRENT won't be released for quite some time, so presumably fixes of this nature would automatically get backported (MFC'd?) to 10-STABLE? Thanks!
Nice, good suggestion. Now I have this "hack" in bb-uenv.txt: uenvcmd=echo "[environment] delete fdtaddr"; env delete fdtaddr; setenv loadfdt ''; setenv fdtfile "/boot/kernel/${fdtfile}"; echo "[environment] loadfdt=${loadfdt}"; echo "[environment] fdtfile=${fdtfile}" Not sure where FDT file should go? First I though maybe /boot/fdt... then /boot/kernel seemed good place... Also, maybe we should use this way of loading FDT in every platform it's possible? No static FDT in kernel too. Ruins NFS boot, however...
Answer by 3 months ago
Quoted message by Winston Smith 3 months ago
That works!!! I'm not sure what MFC'd means, but yes, could we get this in 10-STABLE? It's my understanding that 11-CURRENT won't be released for quite some time, so presumably fixes of this nature would automatically get backported (MFC'd?) to 10-STABLE? Thanks!
Just use the plain filename, not /boot/kernel/filename. ubldr knows where to look for them. Right now it searches for dtb files the same way as for modules, so it will look in /boot/kernel and /boot/modules. There's been some talk of adding a /boot/dtb directory as the first place to search, and having the build system never automatically install anything to there. That gives users somewhere to put a customized dtb file that won't get wiped out every time a new kernel is installed. Yeah, when I started down this path my vision was a single IMX6 image that could boot on any imx6-based system because it comes with a collection of dtb files. The user just has to choose the right dtb file on the first boot by setting an env var. That would also require a pretty flexible u-boot, which may or may not be possible. -- Ian