What's the point of using Busybox in a low ram embedded system

4
votes

I'm working on bringing linux to a custom Cortex-M7 board with 16 Mb of SDRAM and 64 Mb of Flash. The platform has no-MMU, no shared libraries, FLAT executables.

I'm having problems booting a Busybox system with very simple init.d shell scripts. The system is running out of memory by executing simple shell commands like "[" or "printf". It turns out that everytime one of these commands are executed the system needs to load the FULL, one and only busybox executable (650 Kb on my system).

So the question is: if the system always have to load in memory a huge executable for each command implemented inside busybox, then how is this convenient? I don't get the point of saving some megabytes of cheap and abundant storage while going out of ram extremely fast, but maybe I'm overlooking something.

Is my platform an use case for Busybox? if not, is there anything to conveniently build linux system utilities each on their own executable?

Thanks in advance!

EDIT:

Busybox, according to themselves "has been written with size-optimization and limited resources in mind" and so became a sort of unquestioned de-facto standard in embedded systems. But how their statement relates to the forementioned issues on RAM (not storage) constrained systems? I believe this worths some clarification.

Follow up, system details:

The kernel is already compiled for XIP, executing from the 64 Mb external flash. The entire read/write ext3 root filesystem (including busybox binary) now resides on a micro SD card. Busybox executable is using the FLAT format ("bFLT") with load to RAM bit enabled, that bit seems to be causing a new load on a different memory block each time it runs a concurrent command until it exhausts the fitting blocks. The advice to put busybox (the entire /bin, /sbin) on a XIP filesytem is brilliant and it will surely improve the execution speed (of course, this new filesystem will need to reside on the 64 Mb external flash). I never tried a "bFLT" to be executed in place on such a filesystem (nor have an idea if it works) but I'll do my research/testing about that.

2
linuxarmembeddedcortex-mbusybox
Try to load the full [ or printf from bash. – KamilCuk
first question you are trying to run linux on a cortex-m7? second question busybox/u-boot are not required to load and start linux, its pretty easy to do that, what is it you want to get from a bootloader? these are quite bloated bootloaders, do you absolutely need a bootloader and if so what features do you need. Seems like the system engineering hasnt been done I suspect you either need to choose a different processor/family or choose a different software solution for the processor you have. but cant tell from your question. – old_timer
@old_timer yes, I'm running linux on a Cortex-M7, I believe it's a common MMU-less target. I'm using a custom, extremely simple bootloader similar to afboot-stm32 but with graphics subsystem initialization, but even if I was using u-boot, I'm afraid I don't get your point about the bootloader since I've plenty of flash storage. Surely I'll be happier with a MMU and plenty of memory, but I need to make this work as it is. The system is capable, It's just there is not much information available about running linux on MMU-less architectures. – conejoroy
thats part of the problem, running linux on anything non standard/common is a problem and the bootloader adds to the fun. Did arm port this or some hobbyist, etc? the m7 itself is almost irrelelvant most of the work is the non-arm portions of the chip. some arm specific stuff but the majority is not. is there a known working port of both a bootloader and a linux to that chip? is linux required, what about an rtos of which there are definitely some ported to mcus like that. is an os required at all, is the first question... – old_timer
I dont port operating systems I run bare metal but porting to a new platform is not a simple thing, for someone very experienced at it can take months. and thats for the easy ones with an mmu and basically a full blown linux. For a scaled down one or an ancient one that is still around for mmu-less platforms, may or may not be more or less work. If someone has already ported these things to your chip and possibly board then its a howto thing. might be worth finding a board/system/linux combination that has been ported. – old_timer

2 Answers

6
votes

TL-DR; Linux has a huge infrastructure and variety of rootfs or boot file systems available. The choice is due to accommodation of different system constraints and end user functions. Busybox is a good choice for the target system, but any software can be misused if a system engineer doesn't spend time to understand it.

Is my platform an use case for Busybox?

It is if you take time to minimize the kernel size and busybox itself. It is unlikely you need all features in your current busybox.

if not, is there anything to conveniently build linux system utilities each on their own executable?

See klibc information below. You can also build dash with musl, with buildroot and busybox. Many filesystem builders support shared libraries or static binaries. However, there are many goals such as package management, and live updates, that a filesystem builder may target.

More Details

You can configure features out of busybox. The idea is that all of the configured features are needed. Therefore you need them all in memory. With busybox, ls, mkdir, printf, etc are all the same binary. So if you run a shell script the one code load is all code loads. The other way, you have many separate binaries and each will take extra memory. You need to minimize Linux to get more RAM and you can take features out of busybox to make it smaller. Busybox is like a giant shared library; or more accurately a shared process. All code pages are the same.

a custom Cortex-M7 board with 16 Mb of SDRAM and 64 Mb of Flash

...

one and only busybox executable (650 Kb on my system)

Obviously 650KB is far less than 16MB. You don't say what the other RAM is used for. For another good alternative look at the klibc toolsuite. What is not clear is whether the FLASH is NAND/NOR and if you have XIP enabled. Generally, busybox would be better with XIP flash and klibc would be better (and more limited) for SDRAM only, with some filesystem in flash.

See: Memory used by relocatable code, PIC, and static linking in the Busybox FAQ. It is designed to run from Read-only memory which can be a big gain depending on system structure. It probably provides a more rich feature set than klibc as the goal with that project is just to boot some other mount device (a hard drive, SSD etc).

Klibc does not have as much documentation as busybox. It can be either a shared library or statically linked. Each binary will only use the RAM needed for that task with static linking, but this will take more flash space. The binary with klibc are,

 1. dash    2. chroot     3. dd      4.  dmesg  5.  mkdir  6.  mkfifo
 7. mknode  8. pivot_root 9. unmount 10. true   11. false  12. sleep
 13. ln    14. ls        15. mv      16. nuke   17. minips 18. cat
 19. uname 20. halt      21. kill    22. cpio   23. sync   24. readlink
 25. gzip  26. losetup

and that is IT! No networking, no media players, etc. You can write code to use klibc, but it is a very constrained library and may not have features that you require. Generally it would be limit to disk only tasks. It is great to probe USB for external device to boot from for example.

Busybox can do a lot more. Most klibc static binaries will be under 100kB; with 10-30kB typical. Dash and gzip are larger. However, I think you need to remove configuration items from your kernel as 650KB << 16MB and busybox would be a fine choice for this system even without XIP.

I should also be noted that Linux does 'demand page loading' for code with an MMU system. Even if you don't have swap, code can be kicked out of RAM and reloaded later with a page fault. Your system is no-MMU, so busybox will not perform as well in this case. With an mmu and 'demand page loading' it will do much better.

For severe constraints, you can always code a completely library free binary. This avoids libgcc startup and support infrastructure which you might not need. Generally, this is only good to test a kernel vs. initrd issue and for script/binary that must run in many different library environments.

See also:

  • AXFS - xip read-only file system.
  • CrafFs - another xip file system.
  • XIP kernel - the kernel can be huge. Get it out of RAM if possible. Configure with EMBEDDED option if not.
  • nommu.org - some information on github
  • elf2flt - Mike Frysingers updates to binutils 2.27-2.31.1
  • fdpic gcc - notes from 2016 by mickael-guene.

XIP can only work with ROM, NOR flash and possibly SPI-NOR MTDs.

1
votes

The point of BusyBox is that you need to have only one (shared) executable in memory.

You wrote:

It turns out that everytime one of these commands are executed the system needs to load the FULL, one and only busybox executable (650 Kb on my system).

That's not entirely true. For the first command the executable is loaded in memory indeed. But if you're running multiple commands (i.e. multiple BusyBox instances), the executable is not loaded into memory multiple times. A large part of the binary (simply said: all read-only data, such as the executable code and constant data) will be reused. Each additional process only requires some additional memory for its own tasks.

So if a single instance of BusyBox consumes 640 kB of memory, it's possible that 10 instances together only use 1 MB of memory, while 10 unique executables of 200 kB would use 2 MB.

I would recommend to do some realistic tests on your system and check the actual memory consumption. But note that tools such as ps or top can be a bit misleading or difficult to understand. A lot has been written about that already, and if you would like to know more about that, a good starting point would be here.