Recent Articles

When writing a freestanding application, it’s generally necessary for the firmware engineer to handle runtime initialization. Even when a library like newlib includes a rudimentary implementation of crt0.o, initialization is a very application-specific process owing to the need to initialize hardware, memory, and other loading tasks.

In this essay, we examine the current and historical implementation of executable initialization, finishing with a minimal implementation usable with firmware applications.

Note: Most firmware applications need to address the initialization of .data and .bss from nonvolatile memory. That is not addressed in this essay.

While self-hosting Git isn’t that hard (all you need is a shell accessible through SSH), some tools make it easier. One of them is Gitea, a nice, self-contained Go binary that provides you a GitHub clone without all the complexity and dependency hell of something like GitLab. One of the things it does not provide is an integrated continuous integration/delivery (CI/CD) platform. Instead, it implements the same basic patterns at GitHub allowing for pairing with a range of third-party services, cloud or hosted.

The most common CI paired with Gitea would be Drone. While there are many reasons I’m likely to discard it and try other options, I figure it’s worthwhile to at least share my experience trying to make it work for me, most notably how it handles runners.

Bazel is one of the least terrible build systems out there. It can handle large codebases, mixed languages, and cross-platform builds like a champ. Unfortunately, it suffers from rather poor documentation with an enterprise Java codebase that is a nightmare to decipher.

One of the features I’ve been trying to make use of are linkstamps. The idea behind linkstamps is to embed information such as the Git commit identifier into the resulting binary, providing direct traceability for deployed binaries. Unfortunately, regarding this feature, Bazel suffers from the common documentation anti-pattern where they describe what an option is, not what it does.

Note: These instructions were written at the time of Bazel 4.2.1.

When using FreeBSD, the most common method for virtualization and process isolation are jails. Introduced with FreeBSD 4.0 in March of 2000, they predate the closest Linux equivalent, cgroups (and, by extension, Docker), by nearly a decade.

A core part of any virtualization technology is its interaction with the networking infrastructure. In this regard, I’ve found much of the available documentation lacking, often deferring to third party tools which are no longer maintained. As such, I’ve had to scrape multiple sources and reverse engineer system programs to figure out how it’s put together.

In today’s article, I’ll describe the results of my foray into FreeBSD jail networking. It’s not the most cohesive piece, but I’ll refine it over time and hopefully it will assist someone else in their efforts to deploy FreeBSD jails.

Note: These instructions were written at the time of FreeBSD 13.0.

In the modern world, people tend to use massive frameworks to accomplish simple tasks. Nothing quite like swatting a fly with a nuclear missile when you load up 100 megs of runtime just to execute ping and post the result to a database. But sometimes, when you’re writing a utility, you want it to be quick and lightweight. And if you’re going through the effort, you might as well see about making it portable.

Every extent Unix system uses BSD sockets for their networking layer, which traces back to 1983 and the release of 4.2BSD. In this model, network connections (“sockets”) are full fledged kernel objects that built on top of the existing Unix API (e.g. read, write, and close). Additional system calls were introduced both to create new sockets (e.g. socket) as well as provide networking-related operations (e.g. connect, listen, and accept).

By comparison, the original version of WinSock was a completely user space affair with its own API, loosely modeled on BSD sockets, but distinct from the system’s native interface. As the interface transitioned into the 32-Bit era with WinSock 2, it began to integrate more closely into the Win32 API. While Microsoft has written some material on porting Unix applications to WinSock, it leaves unanswered the question as to how to portably target both platforms.

Inline functions are a notable feature of both C and C++. By exposing the source file to the implementation of a function, they allow a variety of optimization techniques that wouldn’t be possible if it had to call out to a subroutine in a different file (at least without link-time optimization).

However, despite the common syntax, the C and C++ languages implement them in very different ways. While C++ takes a “user friendly” approach and automatically manages the manipulation of multiple implementations, C requires a more manual approach. As a result, inline functions are less common in C and mixed language code, generally using the nuclear option of declaring them static.

/* test.h */
inline int test1(void) {
  return 12;
}

/* test.c or test.cpp */
int test2(void) {
  return test1();
}

Sometimes, the easiest way to understand a rule is to see it in action. Let’s analyze how modern compilers deal with C and C++ inline functions in different situations.

On my home server, I use FreeBSD. While FreeBSD beat Linux to the containers by nearly a decade (comparing jails to cgroups), I have to acknowledge that cgroups are the superior design. Whereas jails are a bunch of hacks piled on top of chroot, cgroups are a much cleaner abstraction of the kernel’s namespaces. But even beyond the elegance of the design, software like Docker makes it much easier to run your tools in containers, even if the offloading of sysadmin responsibilities it encourages triggers my OCD.

One of the things Docker does differently than most people’s usage of jails (at least from my limited understanding) is that a docker instance is ephemeral. The last time I touched iocage (years ago, granted), it was still focused on modeling jails like pets, not cattle. So I wanted my jails to go through the closest analog I could to Docker without porting over a massive ecosystem. That means if I’m going to write my own scripts, I need to understand how the FreeBSD jail system is put together.

Note: Jail functionality has improved a lot in the past two years. I had started this essay with a lot of frustration over things I had to work around only to find things much improved during my research.

Recently, I needed to hack together a quick server-side application for a test page. When it comes to quick and dirty, most people turn to Python. While Python is hardly my favorite language, it is certainly suited for the task.

One of my requirements was that I wanted to minimize the number of dependencies. Installing Python is bad enough (my nginx container is extremely bare bones), but installing a massive framework for what was effectively a wrapper around a system command would make things even worse. So I decided to make due with wsgiref.simple_server that ships with CPython.

Unfortunately, when it came time to installing the script on the server, I discovered that simple_server only supports IPv4 and that is baked in. For security purposes, I like to use unix domain sockets whenever possible. It guarantees the endpoint can only be access from the local host and I can even configure permissions based on the user. Fortunately, it isn’t that hard to modify simple_server to support other protocols.

On my home server, I use FreeBSD. Sadly, the BSD’s have been falling behind Linux in the past decade but they still appeal to many people and even have a few tricks left in them. Most notably, as Linux has been struggling with next generation file systems for a few years now, FreeBSD has integrated ZFS for over a decade.

While modern Linux distros become ever-more complex interplays of components, the BSD’s have remained relatively simple. Custom installation (by hand or script) is trivial when the system is distributed as a tarball or two versus a constellation of packages numbering in the thousands. Just boot into the live CD, format the file systems, unpack the tarballs, install the boot loader, and reboot into your new system.

A custom installation allows you to fulfill your OCD tendencies, but it requires you to have a strong understanding of how the system is constructed. Unfortunately, FreeBSD’s documentation has not been keeping up with the evolution with the system’s capabilities. For example, the page on efi(8) is extremely short and does not address modern concerns like ZFS and full disk encryption. In fact, much of the limited information on that page is downright misleading. [NOTE: This has been fixed as of FreeBSD 13.]

NOTE: These instructions were written at the time of FreeBSD 12.2. They are still applicable as of FreeBSD 13.0, but recent updates to the documentation have fixed many of the complaints.