An opinionated anthropology of the embedded programmer, its habits and habitat

Illustrations by Jonas Ekman

Registers! Oscilloscopes! Beards! Serial Ports! C! Cycle shaving! Beards! Interrupts! Assembly! Did I mention beards?

If I were to say the words “embedded programmer” most people in our industry would immediately conjure up an image of a heroic character. A magnificent developer with arcane skills, an encyclopedic knowledge of the occult, a dubious understanding of the concept of personal hygiene, and a truly epic amount of facial hair. A godlike figure.

But after studying the subject for several years, working undercover, I am here to tell you that your picture is incorrect. Most embedded programmers don’t have beards. You don’t have to dream in assembly language to become an embedded programmer. Some embedded programmers even shower.

I can also confirm that embedded programming is fun, rewarding, challenging and, if you are reading this, something you could probably do.

Nota bene. What follows is somewhat aspirational. It describes what embedded programs should be like, not what the majority of them are actually like. It is a possible and desirable state of affairs. Embedded code that you encounter in the wild can be exceedingly horrible, written by clever people being too clever. But I am trying to convince you that embedded programming at Spotify is something that you want to do – and excellent code is what we want. So if you are an embedded programmer, and what I describe is nothing like the code you work with daily, know that I feel you, that your reality is also true, and that life can be better.

We should start by making sure we understand the distinction between embedded system programming and embedded application programming.

Embedded system programming is what you probably think of when you imagine what embedded programming is like. It is all the work necessary to get an embedded hardware platform up and running: writing device drivers, bootloaders, porting or writing an operating system, fiddling with bits and worrying about cycles. Debugging missing interrupts. Staring for hours at an oscilloscope. Blaming the compiler. Starting to hate the interrupt controller like it’s a person. Waking up in the middle of the night in a cold sweat, convinced that your cat has stolen the positive flank of the interrupt signal. But I digress. Embedded system programming is the really low-level stuff.

Embedded application programming is the art of writing applications for resource constrained systems. This is much more easy-going. The development environment is nice. Code can be written, tested and debugged on a desktop computer. Some of the constraints can be challenging but you don’t have to think about assembly or GPIO pins or DMA descriptors. You do have to think about memory usage, execution contexts, code size and portability. But seriously, who doesn’t enjoy that?

This text is about embedded application programming – the type of embedded programming that we do most of at Spotify.

The beauty of simplicity

Constraints are what makes life interesting. A music album recorded using nothing but two pillows and a glass of water requires a lot more creativity than an album recorded using a full orchestra. I think. Creativity or a very unconventional understanding of the term “music”. My point here is that embedded programming is interesting. But why?

The overriding design constraint on most embedded programs is size. The code needs to be compact. The normal rules of good programming practice still applies – the code needs to be modular, maintainable, testable (and tested) – but also minimal and self-sufficient. In a word: elegant. Good embedded code is elegant.

So what is embedded programming like? What is the embedded programmer like? Let’s explore…

Memory usage – the hidden killer

The embedded programmer shies away from modern concepts of memory management. Garbage collection is almost impossible. The garbage collector may blow the code footprint limit all by itself. It would also need to run actual garbage collection from time to time, ruining the real-time aspects of some embedded programs. Even normal malloc() is preferably avoided. Calling malloc() can take a significant amount of time because the allocator might have to defragment the allocation arena in order to free up a chunk of memory large enough to satisfy the request. The embedded programmer will happily manage memory directly, writing custom allocators or even using statically allocated blocks of memory to guarantee that memory allocation failures cannot occur.

One of the big differences between most embedded systems and normal computers lie in the way memory is organized. Mainstream desktop and server processor architectures like the Intel x86 use a programming model where code and data is stored in the same address space. This means that if your machine has 64 MB of RAM (inconceivable!) and your program is 40 MB (mind boggling!) then you have 24 MB (endless!) of RAM left for your data. In my house we call this a von Neumann architecture, but I wouldn’t say that in a bar because that is how knife fights are started.

Because RAM is one of the most power hungry parts of a processor, and because RAM consumes a lot of chip surface, many embedded systems use a model where code and data live in separate memories. Code and static data is stored in ROM (usually a Flash memory or an EEPROM); dynamic data is stored in RAM. ROM is a lot cheaper than RAM so there is normally a lot more of it (typically 5-10 times as much). RAM and ROM may have separate address spaces (Harvard architecture) or be mapped into a single address space (modified Harvard architecture, sadly never referred to as the Clown architecture). The difference is not usually visible in code unless you are writing a bootloader or an in-system-upgrade feature (cases where you need to write to the code memory space). It does mean that code size and RAM usage need to be tallied separately. In other words: in embedded systems the constraint on code size is different from the constraint on RAM use. Remember that RAM usage is usually the most critical parameter.

What you see is what you get

Embedded programmers are truly the hipsters of the software world. They like artisanal code. Homegrown libraries. Languages from the 1960s. Buckling spring keyboards.

Rare is the ready-made library that will satisfy the peculiar constraints of an embedded system. While there are many JSON parsing libraries out there, not many of them support parsing a document that is larger than the amount of available RAM. The embedded programmer always tries to use existing libraries, because the embedded programmer is lazy. But when no suitable library exists the embedded programmer enjoys reinventing a smaller, faster wheel.

Because the embedded programmer puts a high value on understanding and controlling the execution and resource usage of code, that code is almost always written in C. Sometimes new languages try to break into the embedded world but the embedded programmer is sceptical. Does it have some fancy threading model? See Concurrency below. Is it garbage collected? See Performance below. Does the compiler support every computer architecture known to man? See Portability below. In short, the bar is set rather high and C is already awesome.

Embedded software systems tend to be pretty shallow. The lack of third-party libraries and fancy inheritance structures, and the presence of draconian code size limits, all conspire to keep the code small and understandable. In well-written embedded code, what you see on the page is what the code does. The embedded programmer doesn’t have to step through endless levels of indirection to figure out what the code is actually doing. This is not to say that the actual application logic can’t be horrendously complicated, but at least that fact is not hidden by twelve different abstraction patterns.

Concurrency

Concurrency. Prison money. The embedded developer is against it. Doing one thing at a time was good enough for Charles Babbage and it should be good enough for you. Most forms of concurrency support requires saving state and switching to a new task from time to time. This requires multiple stacks, one per task, which requires more RAM, sometimes much more RAM. A stack these days can easily be 1 kB or even larger. This, the embedded programmer cannot abide. He or she will hand code the hell out of that thing. Cooperative task switching. Non-blocking I/O. Polling. Callbacks. Hand-scheduled execution order. A main loop. These are just some of the arrows in the embedded programmer’s quiver. Don’t worry if you don’t know what some of those things mean, there is help to be found at the end of this literary ordeal.

An astute reader might reasonably object at this time: “Doesn’t any kind of task switching require saving the state of the tasks? Didn’t you just move that work from the OS to the programmer?”. To this the embedded programmer will reply: “Nu-huh!” or maybe: “Moving work from the OS to the programmer is what embedded programming is all about!”

Concurrency is a hard problem. However, once you remove preemptive multithreading from the picture, things are actually much simpler. You know that your code won’t be interrupted so execution order is much easier to reason about. The downside is that some things, like I/O, require more code since blocking is not an option.

The poor embedded systems programmer we abandoned earlier is not so lucky. For her, hardware interrupts throw all kinds of spanners into all the works.

Portability

If your program doesn’t run on every processor architecture known to man then your program is not an embedded program. Big endian. Little endian. Stack based. Register based. RISC. CISC. Scalar. Vector. DSP or PIC. These things do not matter. The code will run. Eight bits in a byte? Not so fast! The embedded programmer assumes nothing. Questions everything. And the day a customer demands that the program run on that Chewbacca 5000 12.5 bit stack based vector CPU with segmented write-only memory and a branch co-processor, the embedded programmer is ready.

What does this mean for you? A fanatical devotion to an old C standard.

Testability

Since many types of testing can be done “off the device”, test code is where the embedded programmer gets to play with all the trappings of modern software development.  Test code can be written in Node.JS, Python and even, *shudder*, C++. There is no more deliciously sinful feeling than allocating a large object on the stack in test code

Testing is absolutely crucial when you work with hardware. The code you write can end up in devices that will never be updated. Sometimes it is burned into a ROM chip. The cost of finding bugs after shipping is very high. For this reason, having a solid grasp of modern testing principles, a passion for writing testable code and an iron commitment to complete test coverage are highly valued traits in embedded programmers.

Performance

For most programmers, performance is about the simple question: “Is it fast enough?” If the code is faster than enough then we’re all good. For the embedded programmer things are slightly more complex: “Is it fast enough, real-time enough and does it hit its power budget?”

Apart from the constraints on RAM and code size, the code must be fast

but not faster. Cycles not used translates directly into reduced power consumption. Reduced power consumption means longer battery life, less heat generated and maybe, in the next hardware revision, a cheaper, slower CPU. It is always a race to the bottom with the hardware people. Good embedded code is architected to facilitate CPU sleep modes, for example by being explicitly polled so that the system can sleep between polls. This also means that the code should not block for any reason.

Some embedded systems have hard real-time requirements, a pacemaker for example. Any thread or task that gets to run in such a system must be guaranteed to return inside a fixed time period. If they run for too long the system can fail completely. In the case of a pacemaker, this can be classified as “bad”. Needless to say, programming pacemakers is not for the faint of heart.

Most embedded media applications instead have soft real time requirements – if a thread or task hogs the CPU for longer than it should then system performance will degrade, but not fail completely. Audio or video might glitch or stutter. The UI might feel sluggish. Certainly not good, but not catastrophic.

Writing code with good real time characteristics is very similar to writing code that facilitates CPU sleep modes – use short time slices, be very mindful of loop lengths and never ever block. Know your code paths. Avoid recursion at all costs.

Is it fun?

I think we can all agree that I have now conclusively proved that embedded programming is fun. Interested in finding out yourself? We’re hiring a Senior Engineer for our embedded team right now! And if embedded isn’t for you, we have plenty of other great positions!

What does it take?

So what characteristics are important for an aspiring embedded programmer?

You need a reasonably solid grasp of the basics of computer architecture. The stuff about memory hierarchies, throughput bottlenecks and concurrency at the hardware level. A strong mental model for how addressing works (pointers) is also required. There can be some domain-level skills that may be required depending on the type of software you will be working on. For example, at Spotify, having a good understanding of networking is helpful.

How would you get “a reasonably solid grasp” if you don’t feel that you have it? Some suggestions:

• Computer Architecture : A Quantitative Approach
  • This book can be intimidating. The sheer mass of it bends light and causes time dilation. However, it is one of the classics and is very well written. If you can digest the chapters on Memory Hierarchy Design and Thread-Level Parallelism then you should be excellently prepared to deal with any issues relating to memory management, concurrency or the throughput bottleneck.
• Modern Operating Systems
  • Another classic. It is true that the author wrote : “… 5 years from now everyone will be running free GNU on their 200 MIPS, 64 M SPARCstation-5.” in 1991, but the book he wrote is way greater than his precognitive abilities. The chapter on Memory Management is an excellent primer on… memory management.
• The C Programming Language (K&R)
  • The canonical C programming book. One of the top programming books of all time. Possibly one of the best examples of technical writing ever. Certainly the best selling book ever written in troff. Read it!
• C Interfaces and Implementations
  • This is a book on proper C usage – how to write maintainable and testable code with examples of the most useful data structures and algorithms.
• Expert C Programming: Deep Secrets
  • This is an in-depth look at the C language and its interaction with the system – memory management, pointer arithmetic, how the compiler works, how to work with interrupts.

Apart from that, the most important qualities are curiosity – the urge to understand how something works, and tenacity – the willingness to put in the work needed to gain that understanding.

The pictures by Jonas Ekman are licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.

Tags: engineering leadership