Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

I'm really curious right now. I'm a Python programmer, and this question just boggled me: You write an OS. How do you run it? It has to be run somehow, and that way is within another OS?

How can an application run without being in an OS? How do you tell the computer to run, say, C, and execute these commands to the screen, if it doesn't have an OS to run in?

Does it have to do with a UNIX kernel? If so, what is a Unix kernel, or a kernel in general?

I'm sure OSes are more complicated than that, but how does it work?

share|improve this question
10  
I'm fairly sure that's what a BIOS is for - it's a really small OS that bootstraps running of the larger OS. –  sevenseacat Oct 24 '12 at 4:53
52  
An OS is convenient, but you do not need one in order to run programs on a computer. –  Andres F. Oct 24 '12 at 5:01
7  
It's perfectly possible to even write non-OS software without an OS. Many Forth interpreters traditionally ran without an OS (or you could say they were OSes). It's not even that difficult. If you know C, you might enjoy writing such a program (a small game, perhaps) as an learning exercise. –  Max Oct 24 '12 at 8:16
27  
This confusion is one of the costs of the wonderful, safe, highly abstracted computing systems that we use these days: people can be very good and competent programmers and not know even the fundamentals about how the computer works. How low do you want to go? For very low, but still above the physics see How were the first microprocessors programmed? on Electronics.SE. –  dmckee Oct 24 '12 at 15:23
1  
Programming was done before the invention of the current OS concept. Obviously something at that level is what kicks off the OS. OSs are bootstrapped. This is usually at least mentioned in a CS 4 year program at some point since most require a computer theory of operating systems course. –  Rig Oct 24 '12 at 15:59

11 Answers 11

up vote 215 down vote accepted

There are plenty of websites that go through the boot process (such as How Computers Boot Up). In a nutshell, its a multi-stage process that keeps building up the system a little bit at a time until it can finally start the OS processes.

It starts with the firmware on the motherboard which tries to get the CPU up and running. It then loads up the BIOS which is like a mini operating system that gets the other hardware up and running. Once that is done it looks for a boot device (disk, CD, etc) and, once found, it locates the MBR (master boot record) and loads it into memory and executes it. It's this little piece of code that then knows how to initialize and start the operating system (or other boot loaders as things have gotten more complicated). It's at this point that things like the kernel would be loaded and start running.

It's pretty incredible that it works at all!

share|improve this answer
89  
+1 for the final sentence. –  Michael Kjörling Oct 24 '12 at 9:23
29  
There's a reason it's called "booting"; the term is short for "bootstrapping", as in, "pulling yourself up by your bootstraps". –  KeithS Oct 24 '12 at 15:37
5  
There was a time when someone had to key or toggle in the bootstrap code. Sometimes it was a simple jump to the first instruction of a program in ROM. Other times it was code to read from a device and jump to the first program instruction in the data that was read. Things are much simpler now. –  BillThor Oct 25 '12 at 1:07
    
10  
@BillThor: Where by “much simpler” you, of course, mean “much more complicated”. They’re just simpler to use. –  Raphael Schweikert Oct 25 '12 at 7:56

A "bare metal" operating system doesn't run within anything. It runs the full instruction set on the physical machine, and has access to all of physical memory, all device registers and all privileged instructions, including those that control the virtual memory support hardware.

(If the operating system is running on a virtual machine, it may think it is in the same situation as above. The difference is that certain things are emulated or in some other way handled by the hypervisor; i.e. the level that runs the virtual machines.)

Anyway, while the OS might be implemented in (for example) C, it won't have all of the normal C libraries available to it. In particular, it won't have the normal 'stdio' libraries. Rather, it will implement (for example) a disk device driver that allows it to read and write disk blocks. It will implement a file system on top of the disk block layer, and on top of that it will implement the system calls that a user application's runtime libraries make to (for example) create, read and write files ... and so on.

How can an application run without being in an OS?

It needs to be a special kind of application (e.g. an operating system) that knows how to interact directly with the I/O hardware, etc.

How do you tell the computer to run, say, C, and execute these commands to the screen, if it doesn't have an OS to run in?

You don't.

The application (which was for the sake of argument written in C) is compiled and linked on some other machine to give a native code image. Then the image is written to the hard drive in a place that the BIOS can find it. The BIOS loads the image into memory, and executes an instruction to jump to the application's entry point.

There (typically) isn't any "running C and executing commands" in the application unless it is a full-blown operating system. And in that case, it is the operating system's responsibility to implement all of the required infrastructure to make it happen. No magic. Just lots of code.

Bill's answer covers bootstrapping which is the process in which you go from a powered-off machine to a machine in which the normal operating system is up and running. However, it is worth noting that when the BIOS completes its tasks, it (typically) gives over complete control of the hardware to the main operating system, and plays no further part — until the next system restart. The main OS is certainly not running "within" the BIOS in the conventional sense.

Does it have to do with a UNIX kernel? If so, what is a unix kernel, or a kernel in general?

Yes it does.

The UNIX kernel is the core of the UNIX operating system. It is the part of UNIX that does all of the "bare metal" stuff described above.

The idea of a "kernel" is that you try to separate the system software into core stuff (that requires physical device access, all of memory, etc), and non-core stuff. The kernel consists of the core stuff.

In reality, the distinction between kernel/core and non-kernel/non-core is more complicated than that. And there has been a lot of debate over what really belongs in a kernel, and what doesn't. (Look up micro-kernel for example.)

share|improve this answer
4  
Phenomenal answer. I'd give you several more upvotes if possible. –  weberc2 Oct 24 '12 at 12:24
5  
Lots of good answers here, so I'll added this as a comment, since no one's mentioned this so far: One of the key features in an OS is allowing multiple applications to execute "simultaneously" from a user's perspective. The ability to schedule processes, and protect processes from altering one another's behavior, are typically features only found only in an OS, and not firmware or BIOS. –  Sean Barbeau Oct 24 '12 at 14:14
1  
The idea of a "kernel" is that you try to separate the system software into core stuff Easy to remember by noting that the term kernel is from the German Kern, which means core/nucleus. –  deed02392 Oct 25 '12 at 9:36
1  
Love this answer here because it mentions that it is compiled and linked binary code running not C. –  Travis Pessetto Oct 25 '12 at 14:33
2  
"Getting away from that made PC users less smart." - Not less smart ... less computer literate. You could also say that it increased the number of PC users. –  Stephen C Oct 26 '12 at 4:20

In the beginning there was no power in the CPU.

And the Man said "let there be power", and the CPU started to read from a given address in memory and execute the instruction that was present there. Then the next one and so on until the end of the power.

This was the boot up. Its task was to load another piece of software to gain access to the environment, where the main software was, and load it.

Finally, a friendly screen invited you to log on.

share|improve this answer
49  
This answer should be moved to christianity.stackexchange.com –  Coomie Oct 24 '12 at 8:12
5  
what is "a given address" where does it come from. Sorry for playing Charles Darwin here. –  Midhat Oct 24 '12 at 9:58
6  
@Midhat - The first address to be fetched by the CPU is hardwired inside it. Usually it is 0. –  mouviciel Oct 24 '12 at 10:21
14  
... And on the 7th day, the Man rested with his games –  Canadian Luke Oct 24 '12 at 19:25
9  
@mouviciel The adress in memory is 0x7C00 for any x86 compatible architecture and first has to be filled by the BIOS which usually loads the first sector of whatever bootable device it prefers... Nice answer though :-7 –  Tobias Kienzler Oct 25 '12 at 6:10

Sorry to be late, but I'll describe it as such:

  • The motherboard gets power.

  • Timing circuits start and stabilize if necessary, based solely on their electrical characteristics. Some newer devices may actually use a very limited microprocessor or sequencer.

    It should be noted, alot[sic] of the things like "timing circuits start and stabilize if necessary" don't really happen in hardware anymore. A tremendous amount of that work is actually extremely specialized software running on very limited subprocessors/sequencers.

    -jkerian at 5:20 on October 25

  • Power is given to the CPU and the RAM.

  • The CPU loads(based on its internal wiring) data from BIOS. On some machines, the BIOS may be mirrored to RAM and then executed from there but that is rare IIRC.

    When turned on, x86-compatible CPUs start at the address 0xFFFFFFF0 in the address space...

    -Micheal Steil, 17 Mistakes Microsoft Made in the Xbox Security System(archive)

  • The BIOS makes calls to hardware ports and addresses used by the motherboard for disk and other hardware IO and spins up disks, gets the rest of RAM working, among other things.

  • The BIOS code(by way of the CMOS settings, stored in hardware) uses low-level IDE or SATA commands to read the boot sector of each disk, in an order specified by the CMOS or a user override with a menu.

  • The first disk with a boot sector gets its boot sector executed. This boot sector is Assembly that has instructions to load more data from the disk, load a larger NTLDR, later stages of GRUB, etc.

  • Finally, the OS machine code is executed by the bootloader, directly or indirectly via chainloading loading a boot sector from an alternate or offset location.

You then get a friendly kernel panic, a suffocated penguin, or your disk grinds to a halt due to a head crash. =) In the alternative scenario, your kernel sets up process tables, in-memory structures, and mounts disks, loading drivers, modules, and a GUI or set of services(if on a server). Then, programs are executed as their headers are read, and their assembly is brought into memory and mapped accordingly.

In a way, the BIOS is an OS. Its name even suggests this.

share|improve this answer
2  
It should be noted, alot of the things like "timing circuits start and stabilize if necessary" don't really happen in hardware anymore. A tremendous amount of that work is actually extremely specialized software running on very limited subprocessors/sequencers. - A friendly neighborhood firmware engineer –  jkerian Oct 25 '12 at 5:20
    
@jkerian Do you mind that I've quoted your comment into my post? –  hexafraction Nov 1 '12 at 11:26
    
heh, not at all. –  jkerian Nov 1 '12 at 14:53

There are many good answers but I wanted to add this: You mentioned you come from a Python background. Python is a ninterpreted (or "interpiled" or whatever, at least in typical CPython use cases) language. This means you have some other software (the Python interpreter) looking at the source and executing it in some way. This is a fine model and allows quite nice high level languages well abstracted from the actual hardware. Downside is you always need this interpreter software first.

Such interpreter software,typically, is written in a language which compiles to machine code, for instance C or C++. Machine code is what the CPU can handle. What a CPU can do is read some bytes from memory and depending on the byte values start a specific operation. So one byte sequence is a command to load some data from memory into a register, another sequence to add two values, another to store the value from a register back to the main memory and soon (a register is a special memory area which is part of the cpu where it can work best), most of these commands are quite low on that level. The human readable for of these machine code instructions is assembler code. This machine code, basically, is what what is stored in .exe or.com files on windows or inside Linux/Unix binaries.

Now if a computer is started up it is dumb, it has some wiring though which will read such machine code instructions. On a PC this usually (currently) is an EEPROM chip on the mainboard containing the BIOS (basicinput ouptput system), this system can't do much, it can ease access to some hardware etc and then do a key operation: go to the boot and copy the first few bytes (aka the master boot record, MBR) into memory and then tell the CPU "here, there's your program" the CPU will then treat those bytes there as machine code and execute it. Typically this is some operating system loader which will load the kernel with some parameters and then hand over the control to that kernel, which will then load all itsdriversto access all hardware, load some desktop or shell program or whatever and allow the user to login and use the system.

share|improve this answer
6  
"interpiled"? I've never heard that term before. –  Bryan Oakley Oct 25 '12 at 14:12
3  
That term was used a lot 5 or so yearts ago, to describe "modern" interpreters which have a distinct compilation phase which is separate from the execution. No idea if this term survived anywhere ;-) –  johannes Oct 26 '12 at 9:15
    
"ninterpreted"? I've never heard that term before. –  Cole Johnson Aug 22 at 21:55

You ask "How can an application run without being in an OS". The easy answer is "an OS is not an application". While an OS can be created with the same tools as an application, and made out of the same raw material, they are not the same thing. An OS doesn't have to play by the same rules as an application.

OTOH, you can think of the actual hardware and firmware as the "OS" in which the OS "application" runs. The hardware is a very simple OS -- it knows how to run instructions written in machine code, and it knows that when it starts up it should look at a very specific memory address for its first instruction. So, it starts up and then immediately runs that very first instruction, followed by the second, and so on.

So, the OS is simply machine code that exists at a known location, and which can directly interact with the hardware.

share|improve this answer
1  
+1 I think this is the best answer. In terms of abstractions I think your're nailing it at the right levels. –  Preet Sangha Oct 25 '12 at 19:43

Answer to your question requires the knowledge of how the native (for CPU) code looks like and how is it interpreted by CPU.

Usually whole compilation process is based of translating things you write in C, Pascal or even Python (using pypy) and C# into things CPU understands, i.e. simple instructions like "store something under [memory address]", "add numbers stored under registers eax and ebx", "call function foo", "compare eax to 10". Those instructions, executed one by one, do things you wanted to do with your code.

Now think about this: you don't really need an OS to execute this native code! All you need is to load this code into memory and tell the CPU it's there and you want it to be executed. Don't be concerned too much by that, though. That's the job BIOS should worry about - it loads your code (one and one sector only), just after the CPU starts, under physical address 0x7C00. Then CPU start to execute this one sector (512 B) of your code. And you can do whatever you imagine! Without, of course, any support from the OS. That's because YOU are the operating system. Cool, huh? No standard library, no boost, no python, no programs, no drivers! You have to write everything by yourself.

And how do you communicate with hardware? Well, you have two choices:

  1. You stay inside of "Real Mode" - CPU execution mode with only 1 MB of memory (and even less), no advanced CPU features like CPU extensions, memory protection, multitasking; 16-bit executable code, ancient addressing mode... But with some routines provided by BIOS, including simple screen output, keyboard support, disk I/O and power management. In one word, you're back in times of MS-DOS and 16-bit CPUs.
  2. You go under "Protected Mode" with all the features your CPU have, all the memory you have installed and so on. But in Protected Mode you ARE completely alone and you have to do EVERYTHING by yourself (and you communicate with hardware using "in" and "out" instructions to input/output data to I/O ports and using interrupts. And memory-mapped I/O). Do I have to say every OS since Windows 95 and very first Linux chooses this option?

Now you're asking what kernel is. Shortly, kernel is everything you don't see and experience directly. It manages, along with drivers, everything, beginning from your keyboard to almost every piece of hardware inside your PC. You communicate with it by graphical shell or terminal. Or by functions inside your code, now executed, luckily, with support of the OS.

For better understanding I can give you one advice: try to write your own OS. Even if it's gonna write "Hello world" on the screen.

share|improve this answer

There are some differences for how an operating system operates that are extremely system dependent. To be useful a system needs to have some predictable behavior on start-up, such as "start executing at address X". For systems that have non-volatile storage (such as Flash memory) mapped into their program space this is fairly easy since you just make sure that you put start-up code at the right location within the processor's program space. This is extremely common for microcontrollers. Some systems have to retrieve their start-up programs from some other location before executing it. These systems will have some operations hardwired (or almost hardwired) into them. There are some processors that retrieve their start-up code via i2c from another chip, so the processor does non-trivial operations without running any assembly instructions and then starts executing instructions at a predefined address.

Systems using the x86 family of processors typically use a multi-stage boot process that is fairly complex due to its evolution and backwards compatibility issues. The system executes some firmware (called BIOS - Basic Input/Output System, or similar) that is in some non-volatile memory on the motherboard. Sometimes some or all of this firmware is copied (relocated) into RAM to make it execute faster. This code was written with knowledge of what hardware would be present and usable for booting.

Start-up firmware is usually written with assumptions about what hardware will be present on the system. Years ago on a 286 machine there would probably be an assumption that there would be a floppy drive controller at I/O address X and would load sector 0 to a certain memory location if given a certain set of commands (and the code at sector 0 knows how to use the BIOS's own functions to load more code, and eventually enough code to be an OS is loaded). On a microcontroller there may be an assumption that there is a serial port operating with certain settings that it should wait for commands (to update more complex firmware) from for X amount of time before continuing with the boot process.

The exact start-up process of a given system isn't as important for you as knowing that it differs on different systems, but also that they all have things in common. Often within the start-up (bootstrapping) code when I/O needs to be done the I/O devices are polled rather than relying on interrupts. This is because interrupts are complex, use stack RAM (which may not be fully set up yet), and you don't need to worry about blocking other operations when you are the only operation.

Upon first being loaded the OS kernel (the kernel is the main part of most OSes) will initially act a lot like the firmware. It will need to either be programmed with knowledge of or discover hardware present, set up some RAM as stack-space, do various tests, set up various data structures, possibly discover and mount a filesystem, and then probably start some program that is more like the programs you are used to writing (a program that relies on an OS being present).

OS code is usually written in a mixture of C and assembly. The very first code for the OS kernel is probably always in assembly and does things like set up the stack, which C code relies on, and then calls a C function. Other hand written assembly will be in there too because some operations that an OS needs to do are often not expressible in C (like context switching/swapping stacks). Often special flags have to be passed to the C compiler to tell it not to rely on the standard libraries that most C programs use and to not expect that there is an int main(int argc, char *argv[]) in the program. Additionally special linker options that most application programmers never use have to be used. These may make the kernel program expect to be loaded at a certain address or set up things to look like there are external variables at certain locations even though those variables were never declared in any C code (this is useful for memory mapped I/O or other special memory locations).

The whole operation seems like magic at first, but after you look into it and understand parts of it the magic becomes just a set of programs that require a lot more planning and system knowledge to implement. Debugging them, though, takes magic.

share|improve this answer

To understand how operating systems work, it may be helpful to split them into two categories: those which simply provide services to applications upon request, and those which use hardware features in the CPU to prevent applications from doing things they shouldn't. MS-DOS was of the former style; all versions of Windows since 3.0 have been the latter style (at least when running anything more powerful than an 8086).

The original IBM PC running PC-DOS or MS-DOS would have been an example of the former style of "OS". If an application wished to display a character on the screen, there would have been a few ways of doing it. It could call the routine which would ask MS-DOS to send it to "standard output". If it did that, MS-DOS would check whether output was being redirected and, if not, it would call a routine stored in ROM (in a collection of routines IBM called the Basic Input/Output System) which would display a character at the cursor position and move the cursor ("write teletype"). That BIOS routine would then store a pair of bytes somewhere in the range 0xB800:0 to 0xB800:3999; hardware on the Color Graphics Adapter will repeatedly fetch pairs of bytes within that range, using the first byte of each pair to select a character shape and the second to select foreground and background colors. The bytes are fetched and processed into red, green, and blue signals, in a sequence which yields a legible text display.

Programs on the IBM PC could display text by using the DOS "standard output" routine, or by using the BIOS "write teletype" routine, or by storing it directly to display memory. Many programs which needed to display lots of text quickly opted for the latter approach, since it could be literally hundreds of times as fast as using the DOS routines. This wasn't because the DOS and BIOS routines were exceptionally inefficient; unless the display was blanked, it could only be written at certain times. The BIOS routine to output a character was designed so it could be called at any time; each request thus had to start afresh waiting for the right time to perform a write operation. By contrast, application code which knew what it needed to do could organize itself around the available opportunities to write the display.

A key point here is that while the DOS and BIOS provided a means of outputting text to the display, there wasn't anything particularly "magical" about such abilities. An application which wanted to write text to the display could do so just as effectively, at least if the display hardware worked the way the application expected (if someone had installed a Monochrome Display Adapter, which was similar to the CGA but had its character memory located at 0xB000:0000-0xB000:3999), the BIOS would automatically output characters there; an application which was programmed to work with either the MDA or CGA could do so as well, but an application which was programmed just for the CGA would be totally useless on the MDA).

On newer systems, things are a bit different. Processors have various "privilege" modes. They start out in the most privileged mode, where code is allowed to do anything it wants. They can then switch into a restricted mode, where only selected ranges of memory or I/O facilities are available. Code cannot switch directly from a restricted mode back to privilege mode, but the processor has defined privileged-mode entry points, and restricted-mode code can ask the processor to start running code at one of those entry points in privileged mode. Additionally, there are privileged-mode entry points associated with a number of operations which would be forbidden in restricted mode. Suppose, for example, that someone wanted to run multiple MS-DOS applications simultaneously, with each one having its own screen. If applications could write directly to the display controller at 0xB800:0, there would be no way to prevent one application from overwriting another application's screen. On the other hand, an OS could run the application in restricted mode and trap on any accesses to the display memory; if it discovered that an application which was supposed to be in the "background" was trying to write 0xB800:160, it could store the data to some memory which it had set aside as a background-application screen buffer. If that application is later switched to the foreground, the buffer could then be copied to the real screen.

The key things to note are (1) although it's often handy to have a standard set of routines to perform various standard services like displaying text, they don't do anything that an application which was running in "privileged mode" couldn't do if it was properly programmed to deal with the hardware that was installed; (2) although most applications running today would be prevented by their operating system from doing such I/O directly, a program which starts up in privileged mode gets to do whatever it wants, and can set up whatever rules it wants for restricted-mode programs.

share|improve this answer

As Stephen C. said, it's not just about starting up the Operating System, it's also about how it runs, interacts with the hardware and with the software on top of it.

I'll just add to his answer, that you may want to take a look at "The Elements of Computing Systems". It's a book and some tools, that explains how a computer, operating system and compilers interact. The unique thing about it is that it gives you the tools to very quickly develop your own operating system in a simulated environment, ignoring the many details required for a real one, so that you can grasp the concepts. It does a great job of letting you see the forest instead of the trees.

If you want to go into more details about how the Operating System interacts with the hardware, then check out Minix.

share|improve this answer

You write an OS. It has to be run somehow, and that way is within another OS?

Your application is running within an OS. This Operating System provides services to your application, things like opening a file and writing bytes to it. These services are usually provided via system calls.

The Operating System is running within the hardware. The hardware provides services to the Operating System, things like setting the baud rate of a serial port and writing bytes to it. These services are usually provided via memory mapped registers or I/O ports.


To give a very simplified example of how this works:

Your application tells the operating system to write something to a file. To your application, the operating system provides concepts like files and directories.

On the hardware, these concepts do not exist. The hardware provides concepts like disks divided in fixed blocks of 512 bytes. The operating system decides which blocks to use for your file, and some other blocks for metadata like the file name, size, and location on the disk. It then tells the hardware: write these 512 bytes to the sector with this number on the disk with that number; write these other 512 bytes to the sector with this different number on the disk with that same number; and so on.

The way the operating system tells the hardware to do that varies a lot. One of the functions of an operating system is to shield the applications from these differences. For the disk example, on one kind of hardware, the operating system would have to write the disk and sector number to an I/O port, and then write the bytes one by one to a separate I/O port. On another kind of hardware, the operating system would have to copy the whole 512 bytes of a sector to an area of memory, write the location of that area of memory to a special memory location, and write the disk and sector number to yet another special memory location.


Today's high-end hardware is extremely complicated. The manuals giving all their programming details are doorstoppers with thousands of pages; for instance, the latest Intel CPU manual is seven volumes, with a total of over 4000 pages—and that's only for the CPU. Most other components expose blocks of memory or I/O ports, which the operating system can tell the CPU to map to addresses within its address space. Several of these components expose even more things behind a few I/O ports or memory addresses; as an example, the RTC (Real Time Clock, the component which keeps the computer's time while it is powered off) exposes a few hundred bytes of memory behind a pair of I/O ports, and that is a very simple component dating back to the original PC/AT. Things like hard disks have whole separate processors, which the operating system talks to via standardized commands. GPUs are even more complicated.

Several people in the comments above suggested the Arduino. I agree with them, it is much simpler to understand—the ATmega328, which does everything on the Arduino Uno except exposing the USB connector as a serial port, has a manual with only a few hundred pages. On the Arduino, you run directly on the hardware, with no operating system in between; just a few small library routines, which you do not have to use if you do not want to.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.