====== Building an OS - 2 - The disk ======

** PAGE UNDER CONSTRUCTION **

So far, we've been limited to the first sector of a floppy disk which is 512 bytes large. This is very little space; we haven't reached the limit yet, but after today's episode, we won't be far from it.

===== Bootloader? =====

Our #1 priority is to implement some code which will load the rest of the operating system into memory. What this means is that we will have to split our operating system in 2 modules, where the first one will load the second one. All operating systems are actually split this way, because 512 bytes is not enough to fit even the most basic functions of an operating system. The first module is called a **//bootloader,//** and, generally speaking, it has several functions:

  * it loads the most essential components of the operating system into memory
  * it puts the computer in the state that the kernel expects it to be in
  * it collects some basic information about the system.

Depending on the operating system, the bootloader can be very simple or very complex. Older operating systems, like MS-DOS, run in 16-bit real mode, the mode we're using right now, so the bootloader's job was quite simple: to just load some binary and run it. More modern operating systems typically expect the bootloader to make the switch to 32-bit or 64-bit protected mode for them, and also collect some system information. We haven't talked a lot about 32-bit/64-bit protected modes yet, but we will get there, I promise. However, one of its main limitations of 32/64-bit protected modes is that the BIOS functions that we talked about in Part 1 can no longer be used. Some of these functions are really important, as they provide us with critical information. For example, there is a function which shows us the memory layout: which parts of the memory are safe to\\ use, and which parts are reserved by hardware. Calling these functions is not possible once we are in 32/64-bit protected mode, so the responsibility falls upon the bootloader to collect all the required information before it starts the main kernel.

Note: technically, there are some ways of calling BIOS interrupts from protected mode, but they are quite advanced and require extensive setup (e.g. v8086 mode, instruction emulation). Since some of the information is needed during kernel initialization, it may not even be viable to use one of those methods; collecting this information from the bootloader is still the best option.

===== Floppy disks and file systems =====

Now that we know what we're going to be working on, let's talk a bit about floppy disks.

Why in the world are we talking about floppy disks in <insert current year>? That's a very good question! When getting started working on an operating system, a floppy disk is the simplest form of disk storage we can work with; it is universally supported by all BIOSes, as well as all virtualization software. Creating and working with disk images is very easy, and the FAT12 file system is rudimentary simple. All of these make it ideal for making operating systems, at least until we learn the basics and we can move to other storage devices.

The simplest way in which we could use a disk would be to have the bootloader in the first sector (or the boot sector) and the rest of the operating system starting from sector 2. This would be quite easy to implement: our bootloader would read a number of sectors into memory, and then start executing them. The problem with this approach is that we wouldn't be able to use the disk for storing any files, which is not very useful. We could design our own file system around that, but it's probably a better idea to use an existing standard one like FAT or EXT or NTFS, so that we can easily exchange data between our operating system and other operating systems like Windows and Linux.

Let's get back to the code, and continue from where we left off in part 1. This time, I decided to use Visual Studio Code as the editor with the [[https://marketplace.visualstudio.com/items?itemName=13xforever.language-x86-64-assembly|x86_64 Assembly extension]] installed.\\ since we want to split our code into two modules let's do that right now i created two different directories in our\\ source directory one for the bootloader and one for the kernel i put the same source file that we\\ worked on in part 1 in both folders next we need to make some changes to our\\ makefile to keep things organized i declare some phony targets\\ this way we can keep our makefile cleaner by referring to various modules using their names rather than their\\ output file names then i added a rule to tail make that the phony floppy image target depends on\\ the actual file main floppy.img in the floppy image dependencies i\\ replaced the main.pin with the bootloader and the kernel targets\\ next i added the rules for building the bootloader the always target will be used for creating the build directory if it\\ doesn't exist so we don't get compilation errors if the directory doesn't exist for the\\ build rules it's really simple we just call nasm like we did before\\ for now to build a bootloader in the kernel the steps are identical so i just added the same rules for the kernel\\ next i created the always target which simply creates the build directory if it doesn't exist\\ and the clean target will simply delete everything in the build folder let's give this a go and see what\\ happens looks like we got an error when creating the flop image ah yes i forgot to change the file names\\ in the main floppy dot img rules talking about the floppy image let's modify the way we create the image\\ so that we actually create a fat12 disk image first we need to generate an empty 1.44\\ megabyte file we can do that using the dd command with the block size set to 512 and the block\\ counts set to 2880 the next step is to create the file\\ system using the mkfs.fat command the dash f12 argument tells you to use\\ fat12 and dash n is used for the label which doesn't really matter since we will\\ overwrite it anyway next we need to put the bootloader in the first sector of the disk\\ the simplest way to do that is to use the dd command with the conf equals no trunk option\\ which tells the denotes to truncate the file otherwise we will lose the rest of the image\\ now that we have a file system we can copy the files to the image one option could be to mount the image\\ but i don't really like doing that because we would have to run the image generation with elevated privileges\\ fortunately there's a collection of tools called mtools which contains a bunch of utilities that we can use\\ to manipulate fat disk images directly without having to mount them to copy the kernel.bin file to the disk\\ we can use the m-copy command\\ our make file is now finished so let's give it another go and we're getting another error now and\\ copy is complaining that the disk image is not valid what happened the issue here is that we\\ have overwritten the first sector of the image with our bootloader this section contains some important\\ headers used by fat12 so by overwriting them we have broken the file system\\ can we fix this yes we just need to add these headers to our bootloader\\ going to the article about the fat file system on the os dev wiki there's this section which describes all\\ the fields that are required for a valid fat file system what we need to do is add all of them to\\ our bootloader to help us figure out what the values of these headers should be\\ i have created a test.img disk image using the same steps as in the make file but without overwriting the boot sector\\ by opening the file using a hex editor we can figure out what the value of each field should be\\ set to\\ let's begin working on our bootloader looking at the documentation there are\\ two sections that we need to add each containing a number of fields the first one is called the bios parameter\\ block according to the documentation the first three bytes must be a short jump\\ instruction followed by a no op so we can start with that\\ next we have the oem identifier which is an 8 byte string that is typically set\\ by the tool used to format the disk looking at the image we created previously this has been set to\\ mkfs.phat theoretically we can put anything here but for maximum compatibility\\ we will just set it to ms win 4.1 the next field is a word indicating the\\ number of bytes per sector which for a standard 1.4 megabyte floppy\\ is 512 bytes here it is in the disk image as well\\ remember that this is little endian so to read the numbers correctly you have to read the bytes from right to\\ left in our case the value is 0 2 0 0 in hexadecimal which\\ in decimal is equal to 512. the number of sectors per cluster is 1.\\ the number of reserved sectors is also one the fat or the file allocation table\\ count is 2. the directory entry count is e0 in\\ hexadecimal the total number of sectors is 2880\\ which multiplied by 512 bytes gives us the 1.44 megabytes\\ the media descriptor type indicates what type of disk this is the value f0 hexadecimal indicates a 3.5\\ inch floppy disk the number of sectors per fat is 9. the\\ number of sectors per track is 18. the head count is two the hidden\\ sector count and the large sector count are both zero the next section is called the extended\\ boot record and contains a few extra fields first we have the drive number which should be\\ set to 0. this value is pretty much useless because moving the disk to a different\\ drive would make its value incorrect next we have one reserved point that should be simply set to zero\\ the signature should be set to either two eight hexadecimal or two nine hexadecimal\\ the volume id is basically a four byte serial number you can put anything you want here\\ the volume label is an 11 byte string you can put anything here as long as\\ it's padded with spaces\\ the system id is an 8 byte string which should be set to fat 12 also padded with spaces\\ now that we added all the required headers we can test if make works\\ and it does we can also verify that the disk contains our kernel by running the\\ mdir command\\ before beginning to implement our disk reading operation it is useful to understand how data is laid out on these\\ disks this applies to all form of disks floppy cds dvds\\ and hard drives looking at the round disc if we divide it into rings each ring\\ represents a track or a cylinder another way of dividing the platter is into pizza slices\\ these are called sectors floppy disks as well as hard disks can store data on\\ both sides of the platter so we call each side ahead hard disks\\ can also have multiple platters in which case we count each side of each platter as a head\\ to read or write something we need a way to tell this controller where our data is to read or write something we need a way\\ to tell the disk controller where our data is so we can do that by giving it the cylinder number the head number and the\\ sector number this addressing scheme is called cylinder head sector or chs scheme while this scheme might\\ make sense when you need to determine physically where the data is located on the disk it is not very useful for us when\\ working with disks we don't really care what the data is physically located we only care if it's\\ at the beginning of the disk or the middle or the end for that we can use the logical block\\ addressing scheme or lba instead of a triplet of numbers you only need one single number to reference a\\ block on the disk unfortunately the bios function we will use only supports chs addressing so we will have to make\\ the conversion ourselves another thing i'd like to mention is that in most modern disks\\ the physical layout of the data has gotten a lot more complex and these controllers only pretend to have\\ cylinders heads and sectors maintain compatibility with this legacy addressing scheme\\ but they have their own methods of determining the physical location of the data in the chs scheme the cylinder and head\\ are indexed from 0 but the sector starts from 1. taking this into consideration we can\\ come up with the following formulas for making the conversion we have two constants the number of\\ sectors per track or the number of sectors per cylinder meaning how many sectors we can fit\\ in a single track on a single side of the platter and the number of heads per cylinder which is simply the number of faces the\\ entire disk has the sector is obtained by taking the remainder of the logical block\\ address divided by the number of sectors per track and then adding one for the head we\\ perform the same division and this time we take the result and divide it again by the number of heads\\ per cylinder from which we take the remainder the cylinder is calculated by taking the\\ result of the last division that is the logical block address divided by the number of sectors per\\ track and then divided by the number of heads per cylinder\\ let's write this into assembly\\ we will write a function which will take the lba address in the ax register\\ and to make things easier for us we will store the result exactly how the bios function expects us\\ to so the cylinder will be in cx in bits 6 to 15\\ the sector will be in cx bits 0 to 5 and the head number will be in the dh\\ register we can begin by dividing the logical block address stored in ax\\ by the number of sectors per track that number is a word so we need to clear dx because the div\\ instruction divides dx ax to the world operand\\ after this division we will have the result in ax and the remainder in dx to finish\\ calculating the sector we need to increment the remainder by 1 and then we will put it in cx\\ which is where the output should be next we performed a second division to\\ the number of heads per cylinder this will give us the cylinder in ax and\\ the head in dx\\ now we just need to shuffle the results so they are in the correct output registers\\ since dl is the lower 8 bits of dx we can simply move from dl to the h\\ so that the head number is now in the h the cylinder is a bit weird because it\\ is split this is what the cx register should look like\\ so we need to move the lower eight bits into ch which is the upper half of cx for the\\ upper two bits we can shift them to the left by six positions and then or the result\\ to the cl register which already contains the sector number\\ now to be nice we will save the registers that we modify and are not part of the output so we\\ save ax and dl by pushing them to the stack and when everything is done restore them but since we can't push\\ 8-bit registers to the stack we push the whole dx and when we pop we only restore\\ dl finally we can return from this method\\ next we will write a method that reads from a disk\\ as parameters we will have the logical block address into ax cl will contain the number of\\ sectors to read dl will point to the drive number and\\ esbx will point to a memory location where we will store the data\\ the first thing we need to do is call our conversion function but since the function will overwrite\\ the contents of cx which contains the number of sectors to read we should save it first by pushing\\ it to the stack\\ let's quickly look at the function we want to call the read sectors from drive function and check all the parameters cylinder\\ sector head drive and memory destination should already be set all that's left to do is set the number\\ of sectors to read in a l and 0 to hexadecimal in a h\\ the sector count is saved to the stack so we pop it into ax and then we set a h but now we can call\\ the interrupt 13h in a virtual environment this should work perfectly but unfortunately in the\\ real world floppy disks tend to be pretty unreliable to address that the documentation\\ recommends us to retry the read operation at least three times so let's add that first let's set the\\ number of times we want to retry in a register that we haven't used yet di and then begin a loop we don't really\\ know what registers the bios interrupt will overwrite so we save all of them to the stack using push a\\ there is also another quirk of some bioses that they don't properly set the carry flag so we set it ourselves\\ this is how we can check the result of the operation if the carry flag is cleared that means\\ that the operation has succeeded so we can jump out of the loop\\ now we can restore the registers using pop a\\ if the operation failed we need to reset the floppy controller so we will write a method to do that\\ next we decrement di and check the loop condition if the i is not yet zero we jump back to\\ the beginning of the loop\\ if we exit the loop that means that all of our attempts have been exhausted and the operation still failed so we\\ will jump to another place which will simply display an error message and stop the brute process\\ to make it nicer i call this coded calls interrupt 16h with the function 0 which waits for a\\ key press after which i jump to the address ffff\\ which is where the bio starts effectively rebooting the system\\ as a last thing i saved the registers that were modified to the stack and restore them before\\ returning\\ the disk reset method is really simple it only has one parameter the drive number in dl\\ all we need to do is call interrupt 13h with the ah register set to 0 this will\\ reset the disk controller if the operation fails just like before we jump to the same floppy error label\\ that prints the error message after writing all this code let's give\\ it a try and see if it works let's go back to the main function and try to read some data from the disk\\ the bios should set the drive number from which it loaded our bootloader in the dl register\\ i used that useless field that we talked about earlier to store its value then i set up the call of the discrete\\ function to read the second sector with\\ lba1\\ now let's compile and run our code i kept forgetting the command line for running the vm so i decided to create a\\ run.sh shell script which simply contains the camo command\\ and it looks like we have a problem the hello world message doesn't appear anymore so there is a box somewhere\\ i think now would be a great time to introduce another extremely useful tool which is called box\\ this is basically an emulator and debugger for an x86 processor and we can use it to debug our bootloader\\ to get it running we need to create a configuration file\\ first i set it to emulate the computer with 128 megs of ram then i gave it the path to the rom and\\ the vgaram images then i configured the flowpa drive to\\ contain our disk image with the status inserted\\ right now we don't need any mouse support so i disabled it i set the display library to sdl with\\ the option of the gui debugger box also has a command line debugger but i prefer the gui\\ the run box i created another shell script debug.sh which calls box with the configuration\\ file we just created when i tried to run box i encountered\\ some issues first of all it wasn't installed on my machine in addition to the box package i also\\ needed to install box sdl for the ui box bios and vga bios which\\ contain the roms after that i encountered another error that the display library sdl wasn't\\ available the fix for the issue was to set the display library to sdl2 instead of sdl\\ and now we see the box interface it's not very pretty but we can work\\ with it and it's going to help us a lot\\ okay so let's get everything ready so i'm going to have the code here somewhere\\ so we can see it like this and the display window\\ and now the debugging window okay so now box has started and it has set a\\ breakpoint right at the beginning of the bios what we're gonna do is we're going to go to view disassemble\\ and in this window we are going to put 7c 0 0.\\ 7c00 is the address where our bootloader will be loaded so we are going to double\\ click it this will create a breakpoint and box will stop when it gets here\\ so now let's continue okay so this doesn't look valid to me\\ so let's go ahead and disassemble again and now this is correct so this would be\\ the jump short start instruction now step so what happened here the current\\ instruction highlight has disappeared well that's not something to worry about\\ because we have to go back to view disassemble and the new address is also the same one\\ as in the ip register and let's go there\\ okay so now we have reached this jump instruction let's\\ scroll down and see what happens after this jump of ours so we should be at the start label\\ and at the jump main instruction so let's go one more step and now we are in the\\ main label okay let's scroll down to the main label and now we can recognize the code so\\ let's go step by step and see what is happening\\ first we just set up a few registers and we wrote this into the memory\\ and then we are calling the discrete method the parameters look okay now here you\\ can see all the registers we don't have the ax and bx registers but we have\\ eax and ebx and ecx this is nothing to worry about because\\ in modern processors these registers are actually extended and now they're 32-bit not 16-bit\\ in order to just see the value of the ax register for example we just need to look at the last four\\ digits over here okay so let's move on uh now we are\\ have reached this call to the discrete method so let's step and we can go to the\\ discrete method right now first we have pushed a few things to the\\ stack so let's skip over those and now we are calling this lba2chs\\ method so let's step into it and see what happens\\ okay so first we pushed some stuff to the stack we can also see the stack\\ by going to view linear memory dump and we have to add here the address\\ in our case the top of the stack is 7b ec so we can do that\\ 7 c 7 b e c and press ok and now we see the value\\ of the stack so now this is the logic that\\ performs the lba to chs conversion first we set the dx register to zero and\\ then we want to divide the lba address by the number of sectors per track\\ in our case ax is one so one divided by the sectors per track\\ which is 18 that will give us the result zero and the remainder will be one so\\ that is the case here you can see dx is one ax is zero okay\\ now we are increasing dx to calculate the sector and now we have the sector\\ which is two and if we moved it to cx we don't care about these\\ first four digits just the last four so we have two set to cx\\ okay now we have the second division\\ the values the values are zero so we can see that dx and ax are zero\\ and we have the logic that puts everything into the right registers and\\ we can see the cx register is just two the h is zero and the cylinder is zero\\ now we are popping the registers that we have pushed so we are restoring dl to its previous value\\ okay so dl is now zero and we are returning back to where we\\ came from so now we are going back to the discrete method and\\ we have reached this pop x\\ now let's go on so now we are preparing to call the 13h interrupt so let's see what happens\\ there okay so we have all the parameters ready if we look\\ into the documentation all the parameters should match now we have this interrupt 13 instruction if i click\\ on step it will take me into the bios where the in 13 interrupt is actually handled\\ we don't really care about that we just want to see the result so what i'm going to do is i'm going to set a breakpoint just\\ after this interrupt and i'm gonna press continue\\ and now we have reached this place this was the jump if not carry to the\\ down label and it looks like it jumps so it means that the operation succeeded\\ and now we have reached this done label and now we are popping all the registers\\ that we have pushed and here i found the mistake so instead of popping the idx and so on we have pushed them\\ we can fix that really easily and let me show you what happens if you mess up the stack so let's go\\ and skip these instructions and now we have reached this return\\ and if we click on step now we are at address 00201 what is this\\ i mean this is not where we should be so what is happening here is that the\\ return instruction expects the return address to be at the top of the stack but because we pushed instead of popping\\ the top of the stack contains something else not the return address the return instruction is simply\\ interpreting whatever it finds as the return headers which is why we ended up at the address 200 hexadecimal\\ so let's fix it and see what happens\\ okay so now that we have learned what the problem was we can actually stop box\\ make and now we can run using the run command we have created\\ and we have hello world and then read from this field\\ now i think i know what is happening here so it's not stopping that's the issue here so\\ yeah so we are just calling halt without stopping the interrupts without\\ disabling interrupts so whenever something happens like the clock ticks or\\ we move the mouse or we press a key the processor is interrupted\\ if we just hold without disabling interrupts the processor can still get out of this\\ hold and it can still continue executing even though we have told it to\\ stop so that's why we need to disable these interrupts so that's what we're gonna do and that\\ should solve this issue okay so let's just make and\\ run and now we are seeing the hello world message unfortunately we cannot see if the read\\ operation has actually succeeded now let's go back and use the box again\\ okay now we can continue\\ and we have reached a whole instruction i'm not really sure why nothing is being displayed here maybe\\ something is wrong with my configuration so let's break now go to view\\ linear memory dump let's set the address to 7e0 which is where we read the data\\ and now let's open the hex editor\\ and i'm going to open the floppy image\\ and let's go to address 200 and see and this looks like it matches to what\\ we have read this means that the read is working properly success\\ with this we have reached the end of part 2. before you go let me show you the nanobyte github page where you can\\ find all the source code that we have worked on i will put the link in the description below\\ in part three we will talk about the festival file system and how to read the files of our disk\\ thank you for watching and if you enjoyed the video don't forget to like share and subscribe\\ bye [Music]\\ you