Building an OS - 2 - The disk

PAGE UNDER CONSTRUCTION

So far, we've been limited to the first sector of a floppy disk which is 512 bytes large. This is very little space; we haven't reached the limit yet, but after today's episode, we won't be far from it.

Our #1 priority is to implement some code which will load the rest of the operating system into memory. What this means is that we will have to split our operating system in 2 modules, where the first one will load the second one. All operating systems are actually split this way, because 512 bytes is not enough to fit even the most basic functions of an operating system. The first module is called a bootloader, and, generally speaking, it has several functions:

• it loads the most essential components of the operating system into memory
• it puts the computer in the state that the kernel expects it to be in
• it collects some basic information about the system.

Depending on the operating system, the bootloader can be very simple or very complex. Older operating systems, like MS-DOS, run in 16-bit real mode, the mode we're using right now, so the bootloader's job was quite simple: to just load some binary and run it. More modern operating systems typically expect the bootloader to make the switch to 32-bit or 64-bit protected mode for them, and also collect some system information. We haven't talked a lot about 32-bit/64-bit protected modes yet, but we will get there, I promise. However, one of its main limitations of 32/64-bit protected modes is that the BIOS functions that we talked about in Part 1 can no longer be used. Some of these functions are really important, as they provide us with critical information. For example, there is a function which shows us the memory layout: which parts of the memory are safe to
use, and which parts are reserved by hardware. Calling these functions is not possible once we are in 32/64-bit protected mode, so the responsibility falls upon the bootloader to collect all the required information before it starts the main kernel.

Note: technically, there are some ways of calling BIOS interrupts from protected mode, but they are quite advanced and require extensive setup (e.g. v8086 mode, instruction emulation). Since some of the information is needed during kernel initialization, it may not even be viable to use one of those methods; collecting this information from the bootloader is still the best option.

Now that we know what we're going to be working on, let's talk a bit about floppy disks.

Why in the world are we talking about floppy disks in <insert current year>? That's a very good question! When getting started working on an operating system, a floppy disk is the simplest form of disk storage we can work with; it is universally supported by all BIOSes, as well as all virtualization software. Creating and working with disk images is very easy, and the FAT12 file system is rudimentary simple. All of these make it ideal for making operating systems, at least until we learn the basics and we can move to other storage devices.

The simplest way in which we could use a disk would be to have the bootloader in the first sector (or the boot sector) and the rest of the operating system starting from sector 2. This would be quite easy to implement: our bootloader would read a number of sectors into memory, and then start executing them. The problem with this approach is that we wouldn't be able to use the disk for storing any files, which is not very useful. We could design our own file system around that, but it's probably a better idea to use an existing standard one like FAT or EXT or NTFS, so that we can easily exchange data between our operating system and other operating systems like Windows and Linux.

Let's get back to the code, and continue from where we left off in part 1. This time, I decided to use Visual Studio Code as the editor with the x86_64 Assembly extension installed.
since we want to split our code into two modules let's do that right now i created two different directories in our
source directory one for the bootloader and one for the kernel i put the same source file that we
worked on in part 1 in both folders next we need to make some changes to our
makefile to keep things organized i declare some phony targets
this way we can keep our makefile cleaner by referring to various modules using their names rather than their
output file names then i added a rule to tail make that the phony floppy image target depends on
the actual file main floppy.img in the floppy image dependencies i
replaced the main.pin with the bootloader and the kernel targets
next i added the rules for building the bootloader the always target will be used for creating the build directory if it
doesn't exist so we don't get compilation errors if the directory doesn't exist for the
build rules it's really simple we just call nasm like we did before
for now to build a bootloader in the kernel the steps are identical so i just added the same rules for the kernel
next i created the always target which simply creates the build directory if it doesn't exist
and the clean target will simply delete everything in the build folder let's give this a go and see what
happens looks like we got an error when creating the flop image ah yes i forgot to change the file names
in the main floppy dot img rules talking about the floppy image let's modify the way we create the image
so that we actually create a fat12 disk image first we need to generate an empty 1.44
megabyte file we can do that using the dd command with the block size set to 512 and the block
counts set to 2880 the next step is to create the file
system using the mkfs.fat command the dash f12 argument tells you to use
fat12 and dash n is used for the label which doesn't really matter since we will
overwrite it anyway next we need to put the bootloader in the first sector of the disk
the simplest way to do that is to use the dd command with the conf equals no trunk option
which tells the denotes to truncate the file otherwise we will lose the rest of the image
now that we have a file system we can copy the files to the image one option could be to mount the image
but i don't really like doing that because we would have to run the image generation with elevated privileges
fortunately there's a collection of tools called mtools which contains a bunch of utilities that we can use
to manipulate fat disk images directly without having to mount them to copy the kernel.bin file to the disk
we can use the m-copy command
our make file is now finished so let's give it another go and we're getting another error now and
copy is complaining that the disk image is not valid what happened the issue here is that we
have overwritten the first sector of the image with our bootloader this section contains some important
headers used by fat12 so by overwriting them we have broken the file system
can we fix this yes we just need to add these headers to our bootloader
going to the article about the fat file system on the os dev wiki there's this section which describes all
the fields that are required for a valid fat file system what we need to do is add all of them to
our bootloader to help us figure out what the values of these headers should be
i have created a test.img disk image using the same steps as in the make file but without overwriting the boot sector
by opening the file using a hex editor we can figure out what the value of each field should be
set to
let's begin working on our bootloader looking at the documentation there are
two sections that we need to add each containing a number of fields the first one is called the bios parameter
block according to the documentation the first three bytes must be a short jump
instruction followed by a no op so we can start with that
next we have the oem identifier which is an 8 byte string that is typically set
by the tool used to format the disk looking at the image we created previously this has been set to
mkfs.phat theoretically we can put anything here but for maximum compatibility
we will just set it to ms win 4.1 the next field is a word indicating the
number of bytes per sector which for a standard 1.4 megabyte floppy
is 512 bytes here it is in the disk image as well
remember that this is little endian so to read the numbers correctly you have to read the bytes from right to
left in our case the value is 0 2 0 0 in hexadecimal which
in decimal is equal to 512. the number of sectors per cluster is 1.
the number of reserved sectors is also one the fat or the file allocation table
count is 2. the directory entry count is e0 in
hexadecimal the total number of sectors is 2880
which multiplied by 512 bytes gives us the 1.44 megabytes
the media descriptor type indicates what type of disk this is the value f0 hexadecimal indicates a 3.5
inch floppy disk the number of sectors per fat is 9. the
number of sectors per track is 18. the head count is two the hidden
sector count and the large sector count are both zero the next section is called the extended
boot record and contains a few extra fields first we have the drive number which should be
set to 0. this value is pretty much useless because moving the disk to a different
drive would make its value incorrect next we have one reserved point that should be simply set to zero
the signature should be set to either two eight hexadecimal or two nine hexadecimal
the volume id is basically a four byte serial number you can put anything you want here
the volume label is an 11 byte string you can put anything here as long as
it's padded with spaces
the system id is an 8 byte string which should be set to fat 12 also padded with spaces
now that we added all the required headers we can test if make works
and it does we can also verify that the disk contains our kernel by running the
mdir command
before beginning to implement our disk reading operation it is useful to understand how data is laid out on these
disks this applies to all form of disks floppy cds dvds
and hard drives looking at the round disc if we divide it into rings each ring
represents a track or a cylinder another way of dividing the platter is into pizza slices
these are called sectors floppy disks as well as hard disks can store data on
both sides of the platter so we call each side ahead hard disks
can also have multiple platters in which case we count each side of each platter as a head
to read or write something we need a way to tell this controller where our data is to read or write something we need a way
to tell the disk controller where our data is so we can do that by giving it the cylinder number the head number and the
sector number this addressing scheme is called cylinder head sector or chs scheme while this scheme might
make sense when you need to determine physically where the data is located on the disk it is not very useful for us when
working with disks we don't really care what the data is physically located we only care if it's
at the beginning of the disk or the middle or the end for that we can use the logical block
addressing scheme or lba instead of a triplet of numbers you only need one single number to reference a
block on the disk unfortunately the bios function we will use only supports chs addressing so we will have to make
the conversion ourselves another thing i'd like to mention is that in most modern disks
the physical layout of the data has gotten a lot more complex and these controllers only pretend to have
cylinders heads and sectors maintain compatibility with this legacy addressing scheme
but they have their own methods of determining the physical location of the data in the chs scheme the cylinder and head
are indexed from 0 but the sector starts from 1. taking this into consideration we can
come up with the following formulas for making the conversion we have two constants the number of
sectors per track or the number of sectors per cylinder meaning how many sectors we can fit
in a single track on a single side of the platter and the number of heads per cylinder which is simply the number of faces the
entire disk has the sector is obtained by taking the remainder of the logical block
address divided by the number of sectors per track and then adding one for the head we
perform the same division and this time we take the result and divide it again by the number of heads
per cylinder from which we take the remainder the cylinder is calculated by taking the
result of the last division that is the logical block address divided by the number of sectors per
track and then divided by the number of heads per cylinder
let's write this into assembly
we will write a function which will take the lba address in the ax register
and to make things easier for us we will store the result exactly how the bios function expects us
to so the cylinder will be in cx in bits 6 to 15
the sector will be in cx bits 0 to 5 and the head number will be in the dh
register we can begin by dividing the logical block address stored in ax
by the number of sectors per track that number is a word so we need to clear dx because the div
instruction divides dx ax to the world operand
after this division we will have the result in ax and the remainder in dx to finish
calculating the sector we need to increment the remainder by 1 and then we will put it in cx
which is where the output should be next we performed a second division to
the number of heads per cylinder this will give us the cylinder in ax and
the head in dx
now we just need to shuffle the results so they are in the correct output registers
since dl is the lower 8 bits of dx we can simply move from dl to the h
so that the head number is now in the h the cylinder is a bit weird because it
is split this is what the cx register should look like
so we need to move the lower eight bits into ch which is the upper half of cx for the
upper two bits we can shift them to the left by six positions and then or the result
to the cl register which already contains the sector number
now to be nice we will save the registers that we modify and are not part of the output so we
save ax and dl by pushing them to the stack and when everything is done restore them but since we can't push
8-bit registers to the stack we push the whole dx and when we pop we only restore
dl finally we can return from this method
next we will write a method that reads from a disk
as parameters we will have the logical block address into ax cl will contain the number of
sectors to read dl will point to the drive number and
esbx will point to a memory location where we will store the data
the first thing we need to do is call our conversion function but since the function will overwrite
the contents of cx which contains the number of sectors to read we should save it first by pushing
it to the stack
let's quickly look at the function we want to call the read sectors from drive function and check all the parameters cylinder
sector head drive and memory destination should already be set all that's left to do is set the number
of sectors to read in a l and 0 to hexadecimal in a h
the sector count is saved to the stack so we pop it into ax and then we set a h but now we can call
the interrupt 13h in a virtual environment this should work perfectly but unfortunately in the
real world floppy disks tend to be pretty unreliable to address that the documentation
recommends us to retry the read operation at least three times so let's add that first let's set the
number of times we want to retry in a register that we haven't used yet di and then begin a loop we don't really
know what registers the bios interrupt will overwrite so we save all of them to the stack using push a
there is also another quirk of some bioses that they don't properly set the carry flag so we set it ourselves
this is how we can check the result of the operation if the carry flag is cleared that means
that the operation has succeeded so we can jump out of the loop
now we can restore the registers using pop a
if the operation failed we need to reset the floppy controller so we will write a method to do that
next we decrement di and check the loop condition if the i is not yet zero we jump back to
the beginning of the loop
if we exit the loop that means that all of our attempts have been exhausted and the operation still failed so we
will jump to another place which will simply display an error message and stop the brute process
to make it nicer i call this coded calls interrupt 16h with the function 0 which waits for a
key press after which i jump to the address ffff
which is where the bio starts effectively rebooting the system
as a last thing i saved the registers that were modified to the stack and restore them before
returning
the disk reset method is really simple it only has one parameter the drive number in dl
all we need to do is call interrupt 13h with the ah register set to 0 this will
reset the disk controller if the operation fails just like before we jump to the same floppy error label
that prints the error message after writing all this code let's give
it a try and see if it works let's go back to the main function and try to read some data from the disk
the bios should set the drive number from which it loaded our bootloader in the dl register
i used that useless field that we talked about earlier to store its value then i set up the call of the discrete
function to read the second sector with
lba1
now let's compile and run our code i kept forgetting the command line for running the vm so i decided to create a
run.sh shell script which simply contains the camo command
and it looks like we have a problem the hello world message doesn't appear anymore so there is a box somewhere
i think now would be a great time to introduce another extremely useful tool which is called box
this is basically an emulator and debugger for an x86 processor and we can use it to debug our bootloader
to get it running we need to create a configuration file
first i set it to emulate the computer with 128 megs of ram then i gave it the path to the rom and
the vgaram images then i configured the flowpa drive to
contain our disk image with the status inserted
right now we don't need any mouse support so i disabled it i set the display library to sdl with
the option of the gui debugger box also has a command line debugger but i prefer the gui
the run box i created another shell script debug.sh which calls box with the configuration
file we just created when i tried to run box i encountered
some issues first of all it wasn't installed on my machine in addition to the box package i also
needed to install box sdl for the ui box bios and vga bios which
contain the roms after that i encountered another error that the display library sdl wasn't
available the fix for the issue was to set the display library to sdl2 instead of sdl
and now we see the box interface it's not very pretty but we can work
with it and it's going to help us a lot
okay so let's get everything ready so i'm going to have the code here somewhere
so we can see it like this and the display window
and now the debugging window okay so now box has started and it has set a
breakpoint right at the beginning of the bios what we're gonna do is we're going to go to view disassemble
and in this window we are going to put 7c 0 0.
7c00 is the address where our bootloader will be loaded so we are going to double
click it this will create a breakpoint and box will stop when it gets here
so now let's continue okay so this doesn't look valid to me
so let's go ahead and disassemble again and now this is correct so this would be
the jump short start instruction now step so what happened here the current
instruction highlight has disappeared well that's not something to worry about
because we have to go back to view disassemble and the new address is also the same one
as in the ip register and let's go there
okay so now we have reached this jump instruction let's
scroll down and see what happens after this jump of ours so we should be at the start label
and at the jump main instruction so let's go one more step and now we are in the
main label okay let's scroll down to the main label and now we can recognize the code so
let's go step by step and see what is happening
first we just set up a few registers and we wrote this into the memory
and then we are calling the discrete method the parameters look okay now here you
can see all the registers we don't have the ax and bx registers but we have
eax and ebx and ecx this is nothing to worry about because
in modern processors these registers are actually extended and now they're 32-bit not 16-bit
in order to just see the value of the ax register for example we just need to look at the last four
digits over here okay so let's move on uh now we are
have reached this call to the discrete method so let's step and we can go to the
discrete method right now first we have pushed a few things to the
stack so let's skip over those and now we are calling this lba2chs
method so let's step into it and see what happens
okay so first we pushed some stuff to the stack we can also see the stack
by going to view linear memory dump and we have to add here the address
in our case the top of the stack is 7b ec so we can do that
7 c 7 b e c and press ok and now we see the value
of the stack so now this is the logic that
performs the lba to chs conversion first we set the dx register to zero and
then we want to divide the lba address by the number of sectors per track
in our case ax is one so one divided by the sectors per track
which is 18 that will give us the result zero and the remainder will be one so
that is the case here you can see dx is one ax is zero okay
now we are increasing dx to calculate the sector and now we have the sector
which is two and if we moved it to cx we don't care about these
first four digits just the last four so we have two set to cx
okay now we have the second division
the values the values are zero so we can see that dx and ax are zero
and we have the logic that puts everything into the right registers and
we can see the cx register is just two the h is zero and the cylinder is zero
now we are popping the registers that we have pushed so we are restoring dl to its previous value
okay so dl is now zero and we are returning back to where we
came from so now we are going back to the discrete method and
we have reached this pop x
now let's go on so now we are preparing to call the 13h interrupt so let's see what happens
there okay so we have all the parameters ready if we look
into the documentation all the parameters should match now we have this interrupt 13 instruction if i click
on step it will take me into the bios where the in 13 interrupt is actually handled
we don't really care about that we just want to see the result so what i'm going to do is i'm going to set a breakpoint just
after this interrupt and i'm gonna press continue
and now we have reached this place this was the jump if not carry to the
down label and it looks like it jumps so it means that the operation succeeded
and now we have reached this done label and now we are popping all the registers
that we have pushed and here i found the mistake so instead of popping the idx and so on we have pushed them
we can fix that really easily and let me show you what happens if you mess up the stack so let's go
and skip these instructions and now we have reached this return
and if we click on step now we are at address 00201 what is this
i mean this is not where we should be so what is happening here is that the
return instruction expects the return address to be at the top of the stack but because we pushed instead of popping
the top of the stack contains something else not the return address the return instruction is simply
interpreting whatever it finds as the return headers which is why we ended up at the address 200 hexadecimal
so let's fix it and see what happens
okay so now that we have learned what the problem was we can actually stop box
make and now we can run using the run command we have created
and we have hello world and then read from this field
now i think i know what is happening here so it's not stopping that's the issue here so
yeah so we are just calling halt without stopping the interrupts without
disabling interrupts so whenever something happens like the clock ticks or
we move the mouse or we press a key the processor is interrupted
if we just hold without disabling interrupts the processor can still get out of this
hold and it can still continue executing even though we have told it to
stop so that's why we need to disable these interrupts so that's what we're gonna do and that
should solve this issue okay so let's just make and
run and now we are seeing the hello world message unfortunately we cannot see if the read
operation has actually succeeded now let's go back and use the box again
okay now we can continue
and we have reached a whole instruction i'm not really sure why nothing is being displayed here maybe
something is wrong with my configuration so let's break now go to view
linear memory dump let's set the address to 7e0 which is where we read the data
and now let's open the hex editor
and i'm going to open the floppy image
and let's go to address 200 and see and this looks like it matches to what
we have read this means that the read is working properly success
with this we have reached the end of part 2. before you go let me show you the nanobyte github page where you can
find all the source code that we have worked on i will put the link in the description below
in part three we will talk about the festival file system and how to read the files of our disk
thank you for watching and if you enjoyed the video don't forget to like share and subscribe
bye [Music]
you