Setting up an OS development environment

You want to start working on your own operating system, but don't know where to start? This article describes the tools and settings you need to configure to prepare your system for operating system development.

A long time ago, writing an operating system was done by writing assembly code on paper, going through the CPU manual and translating every assembly instruction into binary by hand, and then punching tape using a special typewriter (citation needed). Luckily for us, things have gotten a lot better and we have much better tools available. We have smart editors to help us write code easier, advanced compilers and assemblers, and really good debugging tools.

While there are development tools available for pretty much any general purpose OS, not all existing OSs are well suited for creating your own operating system; some are too restrictive, and the available tools leave a lot to be desired.

For the best experience, the general recommendation is to use a Unix-like system such as Linux, MacOS or a BSD variant.

Windows is the most popular operating system for the desktop. There are multiple ways of setting it up for OS development:

• using WSL to create a Linux environment. Most OS developers choose this path, and is the one used in the tutorials on this site
• using a port of the Linux tools, such as Cygwin or MSYS2. The main disadvantages are the performance of the tools (e.g. the compiler) which means longer compile times, and the difficulty of creating disk images on Windows
• using native Windows and Microsoft tools

WSL is really easy to set up. All you have to do is open an elevated command prompt (i.e. run as administrator), and run:

wsl --install

This will install all the prerequisites and will set up an Ubuntu distribution. After installation, you only need to search for “Ubuntu” in the start menu to open the WSL instance. Many tools, such as Visual Studio Code and Windows Terminal, will automatically detect any installed distribution and make it very easy to interact with.

For more information, as well as instructions on how to install different distributions, check the official documentation.

Caveat: There are 2 versions of WSL. WSL 1 is built as a translation layer that translate linux system calls into Windows system calls. This has some limitations, the most notable is the lack of support for mounting devices. There are other limitations too that you will encounter when running certain Linux programs. WSL 2 works by virtualizing the Linux kernel and providing a “transparency” layer that has features like the integrated file system, a built-in X server for running GUI applications and audio pass-through. If you encounter problems running certain commands, you might be using WSL 1, in which case you will have to migrate your distribution to WSL 2.

If you are using Linux, you are already set up, all you need is to install the tools in the chapters below.

While MacOS is a Unix-like operating system, it doesn't come with all tools out of the box, so you will need to install homebrew.

There are some small differences between Linux and MacOS, such as the commands used to set up loopback devices, or the commands used to format a disk. It is a lot more similar to FreeBSD than Linux.

• BSD: you are already set up, you just need to install the tools below.
• Android: even though Android runs Linux under the hood, it is heavily locked down, making it a very poor choice for OS development. While most required tools can be found in Termux, there are many limitations such as not being able to mount disk images unless the device is rooted. Another issue is that most Android devices are built on the ARM architecture. If you're working on an x86 operating system, you will need to emulate x86 which will be very slow.
• ChromeOS: similar issues to Android. The best option is to create a virtual machine running a Linux distribution, and doing the development in that virtual machine.
• iOS: the operating system is too locked down to allow any kind of software development on the device. Even worse, Apple doesn't allow most emulators in the App Store, which means you have to sideload and/or jailbreak your device in order to use one.
• MS-DOS, FreeDOS: there are plenty of compilers and tools available (such as DjGPP, Turbo C, Microsoft C, Watcom etc). The main limitations are the fact that pretty much all the tools are outdated and unmaintained, the memory limits of MS-DOS (later software used DOS extenders which alleviated the issue), the lack of proper source control, and the lack of debugging tools. You also cannot virtualize, so you likely have to test on real hardware.