In one of my previous blog posts we went over how to make a minimal (sort of) root filesystem using buysbox.  This is great is you don’t need a package manager and want to built all our utilities and frameworks from source yourself. But if you would rather use a distribution to install packages and tools then then using a Ubuntu core distribution would be a good option.

Ubuntu base is basically a small Ubuntu root filesystem that only includes a command line interface.  It’s great starting point for any embedded system.  Even if you need a GUI X11 can be installed and configured.  Ubuntu base does not include a kernel, we need to provide that so it’s not as easy as the distributions you’d download and install on a laptop or desktop.

Before we get started please take a look at this page which basically already does what I’m about the explain.

Let’s download the ubuntu 16.04 Xenial for arm from here, we will need to download the following file ubuntu-base-16.04-core-armhf.tar.gz.

Make a directory where we will be creating our root filesystem, this is what I did on my system

mkdir -p zynq_xenial_rootfs

Now we need to uncompress the base system that we downloaded.  We can uncompress it into the directory we just made.

cd zynq_xenial_rootfs
sudo tar xf ubuntu-base-16.04-core-armhf.tar.gz

In our directory there should be the skeleton of the root filesystem with the correct permission since we uncompressed with sudo.  We still need to configure our serial port to have show our terminal output.  We’ll also create a chroot jail to test out our root filesystem and also install an utilities we may need.  To do this we will need to install qemu.

sudo apt-get install qemu-user-static

So you may be wondering why we want to create a chroot jail using qemu? I’ve used this method when I don’t have access to ethernet on my target board.  There may be situations where you can’t connect to wifi or there is no wired network that allows random devices to optain an IP address.  In these cases we can create our chroot jail and install any packages we need to get moving.

sudo cp $(which qemu-arm-static) zynq_xenial_rootfs/usr/bin/

Next, we are going to bind our hosts systems proc directory to our root filesystem.  This simply allows our chroot file system to use the hosts proc directory.  There is no harm in this and we can safely unmount it when we are done.

sudo mount -t proc proc zynq_xenial_rootfs/proc

We also need to set up the resolv.conf file, we will copy the one from our hosts system over.

sudo cp /etc/resolv.conf zynq_xenial_rootfs/etc/resolv.conf

We can now start our simulated chroot jail by executing the following command

 sudo chroot zynq_xenial_rootfs /bin/bash

We should now see the # sign to show we are logged in as root, we can use the exit command at any time to exit the chroot jail.

There are a couple things we will do using the chroot jail that will help when we first boot into our embedded Linux system.  We will set the root password, create a non-root user and install a couple of packages.

First let’s set the root password

passwd root

Enter the password that you’d like to use for root

Now we can create a non-root

adduser ubuntu

Then you’ll be asked to set a password for the new user.  Now that we’ve set up some users you are pretty much ready to use your system.  Since this system is Ubuntu (debian) based we can try using the package manager to install some utilities we will need.  Let’s try install python3,

apt-get install python3
apt-get install wireless-tools
apt-get install vim
apt-get isntall sudo

I’m assuming you are still logged in as root if not add sudo in front of this.  Install any other packages that your system may need.

One package we will need to install is the udev package.  For some reason it’s not included in the base image and will cause a fair amount of headaches when we are trying to spawn our serial console.  Let’s go ahead and install that, we will see some warnings in the output but we can ignore them.  The warnings are a result of us using a chroot jail.

apt-get -y install udev

In order to log into our system though the uart of the Zybo we need to configure the console login process for ttyPS0 which is UART0 on the ARM processor.  To do this we need to create a file called /etc/init/ttyPS0.conf. 

vi /etc/init/ttyPS0.conf

This file will spawn a console over our uart port on start up, the contents of the file should look like:

start on stopped rc or RUNLEVEL=[12345]
stop on runlevel [!12345]

respawn
exec /sbin/getty -L 115200 ttyPS0 vt102

Next we need to add ttyPS0 to the UART section in the file /etc/securetty.  We also need to edit the /etc/fstab file so that our root filesystem is mounted on start up.  Our /etc/fstab file should look like:

/dev/mmc.blk0p2 /   ext4    relatime,errors=remount-ro  0   1

I edited all my files with vi, which is why we installed it in the previous step.  Since we are done with our chroot environment we can type in exit on the command line and we should be back in our proper host system.

All that’s left to do now is to edit a couple of files on the linux and uboot side of things and we are good to go.

First we’ll need to edit the zynq-common.h file in u-boot so that we don’t try to load the initramfs filesystem anymore.

Back in our host environment we can switch into our u-boot source directory. We will need to edit the file include/configs/zynq-common.h

We will need to remove the following lines:

"load mmc 0 ${ramdisk_load_address} ${ramdisk_image} && " \
"bootm ${kernel_load_address} ${ramdisk_load_address} ${devicetree_load_address}; " 

Replace it with the following line :

"bootm ${kernel_load_address} - ${devicetree_load_address}; "  

Now u-boot should look for or try to load the ram disk when it starts up.  We will have to rebuild u-boot and replace it on our sdcard.

On the Linux side we’ll modify the device tree file to change where the rootfs is located.  In the zynq-zybo.dts file we will be changing the boot args.  Open zynq-zybo.dts which is located in the dts directory, and find the line that assigns the bootargs.  Change the boot args to the following.

bootargs = "console=ttyPS0,115200 root=/dev/mmcblk0p2 rw earlyprintk rootfstype=ext4 rootwait devtmpfs.mount=1";

Once we have those changes done we’ll need to recompile the device tree.  Since we’ve already built the kernel once (hopefully) we can run that command again and the devicetree files will be recompiled.

make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- O=<path_to_output_directory>UIMAGE_LOADADDR=0x8000 uImage modules dtbs

If you haven’t built the kernel yet now would be a good time to look at this tutorial.

Now we should copy our u-boot binary and new devicetree binary to the boot partition of our sdcard and we should have a fully working Ubuntu 16.04.  Remember to login using the new passwords we set above.  Now you can use the ubuntu package manager to install any tools that will be needed.

Throughout these tutorials I’ve been assuming you’ve been using the sdcard for all the files we need.  You should have two partitions on your sdcard.  One partition should be formated FAT32 and the other should be ext4.  These two partitions will hold our filesystem and boot files.  If you’ve looked at my previous posts you can see all of your boot files go into the FAT32 parition.  We are now going to populate our ext4 partition.  These patition will hold our root filesystem and give us a area of persistent storage.  We could do this with our busybox approach byt that’s a story for another post.  Let’s go ahead and populate that partition.

sudo rsync -aAXv <path_to_your_rootfs>/* /path/to/mount/point/

That should copy all of our files over to our sdcard and we are ready to boot into ubuntu.  

So far we’ve built u-boot from scratch, built the Linux kernel and built the u-boot SPL so we don’t have to use the Xilinx SDK if we don’t want to.  Our main goal here is to create a embedded Linux system on our Zybo.  Our secondary goal is going to be to add the Xenomai RT patches and create a real time Linux system.  One step that we haven’t gone over yet is creating a root filesystem.

We have a couple choices when it comes to root filesystems, depending on where our embedded system is going to be deployed.  Some smaller system will use a RAM disk as it’s root filesystem.  A ramdisk is a filesystem that is loaded into memory every time the system is started.  This type of file system is not persistent.  This means that any changes or modifications that are made do not survive a reboot.

There are two ramdisks that are commonly used in Linux systems.  The first is the initial ram disk (commonly called initrd).  This is an older method but it’s still supported in the Linux kernel.  When the kernel boots up it will decompress the ramdisk and use it as the root filesystem.  Some Linux systems (included embedded ones) may use this file system to perform some initialization and the pivot to the real root filesystem.  You can google “pivot_root” to see exactly how this is accomplished.  Some embedded systems will continue to use the initial ramdisk instead of loading a persistent one.  Any filesystem changes that we make will be lost on a reboot.  This can be good or bad depending on what we are trying trying to accomplish.  The initrd requires a synthetic block device of fixed size, which restricts the file system from growing without creating a new block device and starting from scratch again.  One draw back of using any RAM disk is that the more libraries and utilities we need in our file system the larger the file system will grow and hence the more RAM it will use.

The initramfs is the preferred (or the most recent) way of creating a ramdisk for your Linux system.  Traditionally initramfs can be built into the kernel itself which makes it very quick and compact.  We don’t need to create a block device to create it which makes it much easier to build.  One draw back is that we don’t want to include anything in the initramfs that can’t fall under the GPL license.  This is because we can include the initramfs into our kernel build therefore making it fall under the GPL.  One way around that is to use the initramfs filesystem but include it externally using the initrd hooks.

The more common approach in hobby based boards is to use the sdcard (or part of it) as the root filesystem and have persistent storage.  This method is much easier when add utilities, libraries and executables to our system.  For learning purposes I’ve chosen to use a ramdisk for my zybo system.   In a later blog post we will also go over how to use a Ubuntu (or Arch) based root filesystem which will be much bigger but give use more flexibility and ease when it comes to included third party libraries.

Moving on to creating or RAM based root filesystem.  Xilinx provided an example of building an initrd file system on there website here, which seems fairly old, so for our purposes we will use the initramfs method and use the initrd hooks to have it included outside of our kernel image.  I’ve taken a lot of information from the above page so it’s still worth a read.

First we will need the Busybox source.  We can download it here, at the time of writing the latest Busybox is 1.26.2.  We can uncompress the tar file into it’s own directory.

tar xvjf busybox-1.26.2.tar.bz2
cd busybox-1.26.2
make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- defconfig

So now we’ve uncompressed busybox, and configured it.  The next step is to add any custom configuration that we may need using menuconfig.  If your build environment hasn’t used menuconfig before make sure you have ncurses installed or we will see errors when running this next command.

make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- menuconfig

Next we need to set up where we are going to place the busybox executable and the symlinks that go along with it.  Once we are in the menuconfig screen go to busybox settings, installation options and specify a location for busybox installation prefix.  For me I placed this in a directory called zynq_ram_rootfs, and make sure to specify the full path.  Lastly exit menuconfig and save our changes.  Next let’s build our executable.

make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- install

We should see some build output, once the build is done we can cd into our install directory and see the the symlinks that were made by the build process.

mkdir dev 
mkdir etc etc/init.d 
mkdir mnt opt proc root 
mkdir sys tmp var var/log

Next we need to remove linuxrc, we are doing this because linux looks for an executable called init when the first process starts up.  We will need to link this to our busybox executable.  Remember for this to work we need to be in the install directory for our root filesystem.

rm ./linuxrc

ln –s ./bin/busybox ./init

If Linux can’t find the init script is should fall back to using an older method of starting up the first user process.  This should include calling linuxrc but I prefer to make the init symlink.  Next we need to create some configuration files that will help get Linux setup using our root filesystem. First we need to create a file named /etc/fstab.

LABEL=/     /           tmpfs   defaults        0 0
none        /dev/pts    devpts  gid=5,mode=620  0 0
none        /proc       proc    defaults        0 0
none        /sys        sysfs   defaults        0 0
none        /tmp        tmpfs   defaults        0 0

This file contains information about all the partitions, block devices and remote file systems.  Here we are mounting each of these directories at startup.

Next we need to create the /etc/inittab file, this file controls what happens whenever a system is booted or when a run level is changed.

::sysinit:/etc/init.d/rcS

# /bin/ash
# 
# Start an askfirst shell on the serial ports

ttyPS0::respawn:-/bin/ash

# What to do when restarting the init process

::restart:/sbin/init

# What to do before rebooting

::shutdown:/bin/umount -a -r

This file is from this Xilinx tutorial, it’s pretty straight forward.  We spawn and ash shell (busybox uses the ash shell) on the UART0 which is ttyPS0 and then we have actions to perform on shutdown and restart.

We also need to create the /etc/init.d/rcS file, this file is the second main boot script.  The rcS file is the run-level file for single user mode.  Because our system only has the root user we are a single user system.

#!/bin/sh

echo "Starting rcS..."

echo "++ Mounting filesystem"
mount -t proc none /proc
mount -t sysfs none /sys
mount -t tmpfs none /tmp

echo "++ Setting up mdev"

echo /sbin/mdev > /proc/sys/kernel/hotplug
mdev -s

mkdir -p /dev/pts
mkdir -p /dev/i2c
mount -t devpts devpts /dev/pts

echo "rcS Complete"

We also need to set this script as executable or we won’t be able to run this script and our linux system won’t be able to do anything useful.

chmod 755 <path_to_rootfs>/etc/init.d/rcS

One of the last steps here in creating our root file system is the create a password file for the root user.  We will need to create the etc/passwd file.

root:x:0:0:root:/root:/bin/sh

This file maintains the information about each user that can use the system.  If you want to know the meaning of the above line checkout this page.

We now have a basic root filesystem but we are missing libraries that our programs will need to run.  We could go download glibc and build it and install it in our root filesystem.  Or we can copy the libraries from our toolchain, the easiest way to do this (IMHO) is to use the sysroot that is provided by the toolchain vendor.  In our case we can download that from the Linaro site here.

I downloaded the file sysroot-glibc-linaro-2.21-2017.05-arm-linux-gnueabihf.tar.xz.  We can uncompress this file and then install it into our rootfs.

Let’s decompress, move the file into our development directory

tar xf sysroot-glibc-linaro-2.21-2017.05-arm-linux-gnueabihf.tar.xz

We now see a folder called sysroot-glibc-linaro-2.21-2017.05-arm-linux-gnueabihf which contains all of our libraries will need for our system.  Just as a warning, this is throwing the entire kitchen sink into your rootfs, so any lib that the compiler provides and it expects to be in a live system is here.  If you aren’t using some libraries you may want to remove them.  For example if you aren’t using fortran then you may want to consider removing those from the libraries you install in your rootfs.  If you are just writing C programs, libc, libm and a couple of gcc dependencies may be all you need.

When I do the following step I add about 256MB of files to my rootfs, which is great for prototyping, but it isn’t good for a live system that wants to use RAM for program data (remember how a RAM disk works).  By doing this next step we may defeat the purpose of using busybox to create a minimal rootfs.  There are some simple steps we can take to size down the libraries that we are deploying.  These steps may save us 200MB of space in our root filesystem.  Let’s first copy over everything.

cp -rf ./sysroot-glibc-linaro-2.21-2017.05-arm-linux-gnueabihf/* <path_to_busy_box_rootfs>

So if we check the size of the directory that contains our rootfs we should see something close to 256MB for it’s size.  This is very very large and in some cases it may be too large to fit into RAM.  So we are going to need to start reducing it’s size.  The first thing we can do is strip out all the debug symbols.  Using the following command will strip all the debug symbols in the libraries of our rootfs.

arm-linux-gnueabihf-strip <path_to_rootfs>/lib/*
arm-linux-gnueabihf-strip <path_to_rootfs>/usr/lib/*

Since I’m not using fortran I removed it from my lib directory and I also removed the debug directory from lib/ also.

rm <path_to_rootfs>/lib/libgfortran.*
rm -r <path_to_rootfs>/lib/debug

Now we should see out rootfs is about 65MB, these is much more manageable.  If you would like to slim it down even further look through the directories and see if there is anything else that you don’t need and remove it as needed.  Once we have that done we can move on with compressing it.

We need to compress our rootfs and get into the proper cpio format.

cd <path_to_rootfs>
find . | cpio -o --format=newc > <path_to_file>/rootfs.img

Almost done, the last step is to add the u-boot header that u-boot needs to load the rootfs image.  We will also be changing the name since xilinx u-boot will be looking to load a file called uramdisk.image.gz

mkimage -A arm -T ramdisk -C gzip -d rootfs.img uramdisk.image.gz

You’ll need to make sure that you have the mkimage utility installed.  Check with what package this is included in for your distribution.  For example if you are using Ubuntu you would install

sudo apt-get install u-boot-tools

One last thing we need to do is update our linux kernel config with the new size of the ramdisk.  So we’ll have to recompile the kernel once we do this but it’s pretty straight forward.  Open arch/arm/configs/xilinx_zynq_defconfig modify the following line

CONFIG_CMDLINE="console=ttyPS0,115200n8 root=/dev/ram rw initrd=0x00800000,65M earlyprintk mtdparts=physmap-flash.0:512K(nor-fsbl),512K(nor-u-boot),5M(nor-linux),9M(nor-user),1M(nor-scratch),-(nor-rootfs)"

I changed mine from 16M to 65M, there’s no reason that if you are just running C code that you’d need all the libs we included, so you could skip this step and slim down your rootfs.  If you’d like to continue with the larger rootfs with all it’s features then do the above modification (replacing 16M with the size of your rootfs in megabytes) and then follow my previous tutorial on compiling the linux kernel.

That’s it, you should now have a busy box based file system that we can add libraries and utilities to as needed.

Where to next?? If you’ve followed my blog posts you should now have all we need to create a custom embedded Linux distribution.  Next blog post we will put all of our steps together and then boot our system and finally get to building Xenomai 3 patched kernel for Zynq.