CS162 Lecture Notes 2-7-2007 The professor is out of town at a conference on the East Coast. This lecture is given by GSI Thomas Kho. Today's Topics: Nachos Tutorial & Monitors Announcements: - Should have been able to sign up your groups on the course website. - Should have subversion account on the instructional machines. - If not, send Thomas an email, and he will help you get it set up. Nachos Tutorial: - Outline + What is Nachos? - Capabilities, purpose, and history. + How does it work? + What am I supposed to do? - The 4 phases of the project + How do I get started? - What is Nachos? + An instructional operating system + Includes many facets of a real operating system, including: - Threads - Interrupts - Virtual Memory - I/O driven by interrupts + In these 4 phases, you can and will modify and extend Nachos - What else is Nachos? + It also contains some hardware simulation. - It has a MIPS simulator that can handle MIPS code in the standard COFF format, except for floating point instructions + Thomas thinks it is the MIPS R4000 + You can and will write code in C, compile it to MIPS with the cross-compiler, and run it on Nachos. - We have cross-compilers for Linux, SPARC, and OS X - Console, which interfaces with standard in and standard out - Network Interface - Timer - Why Nachos? + What better way to learn how an OS works than by building one? + Much easier and more reasonable to build a simulated OS than a real one because compile-and-test cycles for real operating systems are orders of magnitude higher, and it is nice to work in Java because there is a great debugger and lots of other nice stuff. + Skeleton code allows us to work on, replace, or upgrade one piece at a time. - Why Java? + Java is much simpler than C++ + Java is typesafe -- can't write off the end of an array + Java is easier to debug + Much easier and more reasonable to machine-grade a Java project + Java is portable -- can work in Windows, Mac, Linux, whatever. - History of Nachos + Originally created here at Berkeley in 1992 + Written in C++ and a little assembly + Created by Wayne A. Christopher, Steven J. Proctor, and Thomas E. Anderson + Currently in use at a whole lot of schools + Rewritten in 1999 in Java by Daniel Hettena - Now it is simpler, easier to grade, type-safe, portable, and students nowadays are more comfortable with Java. - How does Nachos work? + Written entirely in Java, broken into the following packages: - nachos.ag -- the autograder - nachos.machine -- the machine simulation - nachos.network -- will modify this in phase 4 - nachos.security -- tracks priveleges - nachos.threads -- will modify this in phase 1 - nachos.userprog -- will modify this in phase 2 - nachos.vm -- will modify this in phase 3 - Nachos Architecture + Nachos is a simulator + Hardware and real OS on the bottom with Nachos running in the JVM on top of that + Nachos kernel runs side-by-side with the simulated hardware + The kernel is written in Java + The user programs are written in C - They get compiled into machine code which runs on the simulated CPU and interacts with the Nachos kernel Figure 1. Nachos Architecture .-------------. .---------------. .---------------. | .--------. | | User Program | | User Program | | | | | `---------------' `---------------' | | Nachos | `-------------------------------------. | | Kernel | .-----------------------------------. | | | | | Simulated Hardware | | | `--------' `-----------------------------------' | `---------------------------------------------------' .---------------------------------------------------. | JVM | `---------------------------------------------------' .---------------------------------------------------. | (Real) OS | `---------------------------------------------------' .---------------------------------------------------. | Hardware | `---------------------------------------------------' - Booting Nachos + When you start Nachos, it starts out in the nachos.machine.Machine.main method + Machine.main initializes devices: interrupt controller, timer, MIPS processor, console, file system + Then, it passes control to the autograder, which creates a new kernel and starts it -- this is what starts the operating system - The Machine + Represented by the Machine class + This is what kicks off the system + Provides access to various hardware devices: - Machine.interrupt() - Machine.timer() - Machine.console() - Machine.networkLink() - The Interrupt Controller + Kicks off hardware interrupts + nachos.machine.Interrupt class maintains an event queue and the clock + The clock ticks under 2 conditions: - One tick for executing a MIPS instruction - 10 ticks for enabling or re-enabling interrupts + When your Java code is running, simulating the Nachos kernel, time isn't going to elapse with every instruction - To make sure kernel code is thread-safe, you will probably want to change the clock-ticking mechanism for testing + After any tick, Interrupt checks for pending interrupts and runs them + Interrupt calls the device event handler, which is different from the software interrupt handler inside the kernel - The device event handler is part of the machine simulation - The interrupt controller calls a method in one of the devices inside the machine package, and the device event handler will call the software interrupt handler that you register with it + Important methods accessible to other hardware simulation devices: - schedule() takes a time and a handler - tick() takes a boolean representing either 1 or 10 ticks - checkIfDue() invokes interrupts that are due - enable() and disable() to allow for critical sections inside the kernel + All hardware devices depend on interrupts -- they don't get their own threads - The Timer (nachos.machine.Timer) + Part of the machine simulation + Hardware device that causes interrupts about every 500 ticks - Can't rely on it to tick exactly every 500 ticks + Important methods: - getTime() tells how many ticks have elapsed so far - setInterruptHandler() tells the timer what to do when it goes off + Use the timer to provide preemption for user programs - The Serial Console + Jave interface nachos.machine.SerialConsole + Contains methods: - readByte() -- returns 1 byte (or -1 if there are no bytes to be read) and waits to interrupts when it has more + if the user presses something at the console, readByte() will trigger an interrupt if there is something to be read, and the interrupt handler would call readByte() to actually get the data - writeByte() takes one byte and waits to interrupt when it is ready for more + When you are writing to the console, you write one byte at a time, and you don't write the next byte until getting a completion interrupt from this device - setInterruptHandlers() tells the console who to call when it receives data or finishes sending data + This interface is normally implemented by nachos.machine.StandardConsole, which is hooked up to standard in and standard out - Schedules read event every Stats.ConsoleTime ticks to poll stdin and invoke the interrupt handler - Other Hardware Devices + Disk - in the original C++ implementation, but not in our Java implementation, so we don't have to deal with it + Network Link - Similar to the console, but packet-based instead of byte-based - We will use this in phase 4 - should be able to figure it out by then - The Kernel + Abstract class nachos.machine.Kernel + Important methods: - initialize() -- initializes the kernel - selfTest() -- performs tests that students write + not used by the autograder + when working on projects, students are encouraged to test, and this is a good place to put tests - run() -- runs any user code (don't have to worry about this for phase one) - terminate() -- halts the kernel + Each phase has its own kernel subclass - Threading + Happens in the package nachos.threads + All Nachos threads are instances of nachos.thread.KThread or some subclass + Each KThread can be in 1 of 5 states: new, ready, running, blocked, finished + Every KThread has a nachos.machine.TCB (thread control block) + Internally, KThreads are implemented using Java Threads - Running Threads -- How do you run a thread? + Create a Runnable object (java.lang.Runnable), make a KThread, and call fork() on the KThread + Example: class Sprinter implements Runnable { public void run() { // run real fast } } Sprinter s = new Sprinter(); new KThread(s).fork(); + A lot of the time, you will see instances of anonymous classes passed to the KThread constructor (for small tasks). - Scheduler + Some subclass of nachos.machine.Scheduler + Creates ThreadQueue objects which decide what thread to run next + In the Nachos distribution given, the default is RoundRobinScheduler + For phase 1, will implement PriorityScheduler + For phase 2, will implement LotteryScheduler + Scheduler for Nachos to use is specified in Nachos configuration file - Nachos Configuration + In top-level Nachos directory, there is a directory for each phase - In each phase directory, there is a nachos.conf file for that phase + nachos.conf file lets you specify many options - which classes to use for Kernel, Scheduler - whether to be able to run user programs or not - whether the TLB is turned on or off - etc. - Creating the First Thread -- What Happens? + ThreadedKernel.initialize -- creates a new KThread, passing null as the argument + What does KThread perform? - Initialization, the first time it is called + What thread does it create? - The idle thread, which always runs in the background + When user threads have no work to do, the system will default to running the idle thread - Advanced Topics + Can learn about these on your own, if interested + The simulated MIPS processor + Address translation with the TLB + User level processes + System calls and exception handling + By the time this class is done, you will know all these facets of Nachos - How are we using Nachos? + 4 Phases + Phase 1 - Threading + Phase 2 - Multiprogramming - Running multiple user processes using the MIPS simulator + Phase 3 - Caching and Virtual Memory - Manipulating entries in the TLB - For virtual memory, lazy loading and swapping pages in and out + Phase 4 - Networks and Distributed Systems - Simplified version of TCP - Chat client and server - Extend and Embrace Nachos + Add features to Nachos inside the kernel code - Threading - File system calls + Implement user programs in C - Phase 1: Threading + KThread.join - 5% - when you call join in the parent thread, you block in the parent thread until the child thread finishes + Condition Variables - already given an implementation, but will implement them more efficiently + Alarm - one of the easier parts of this project + Communicator - allow two threads to communicate with each other safely + PriorityScheduler + Rowing Hawaiin Kids simulation - model Hawaiin adults and kids as threads and use safe synchronization to get them from one island to another - Row Boat Synchronization + Two islands: Oahu and Molokai + Want to get all the children and adults from Oahu to Molokai + Constraints - 1 boat that can fit 1 child, 2 children, or 1 adult - Must be a pilot to get the boat to the other side + Use locks and condition variables to ensure safety of simulation + Cannot have any global knowledge in the system - i.e. each child and adult must act independently toward the goal of getting everyone across - Phase 2 - Multiprogramming + File system calls - 30% - create, open, read, write, close, unlink + unlink is like delete - "decently simple" - given an interface to a disk, just need to pass these through + Multiprogramming - 25% - allows multiple programs to run at once + System calls - 30% - beef them up for multiprogramming - exec, join, exit + LotteryScheduler - 15% - Phase 3 - Caching and Virtual Memory + Implement TLB and an inverted page table - 30% + Paged virtual memory - 40% - Fit large programs into memory - Can run programs whose allocated memory is larger than physical memory + Lazy loading - 30% - When starting a program, only load up the pages of the executable that are needed, and don't load other pages until you reference them - Phase 4 - Networking + Networking system calls - 75% - connect, accept - behind this is the state machine for a reliable transport protocol + Chat program - 25% - Written in C using the 2 system calls connect and accept - Like IRC - Should be pretty straightforward - The workload percentages are given for each of the parts of the phases + Should divide the work fairly + After each phase, each group member fills out an evaluation, which will be taken into account when the grades are given - If group members think you didn't contribute, the staff will dock you - If group members think you did a great job, you will get more points + The different phases depend on each other - LotteryScheduler depends on PriorityScheduler - If you don't have Phase 1 working, you will have a tough time with the other phases - Phase 3 depends on phase 2, phase 2 depends on phase 1, and phase 4 depends on either phase 2 or phase 3 - How to get started + Whole section on Nachos on class webpage - a couple good walkthroughs + Links to the online javadoc + Download and install nachos package - one group member should import it into subversion + Read the README and make sure you can make proj1 - Unpack Nachos, go into proj1 directory and type 'make' + To run Nachos, there is a script inside the bin directory + First phase is posted, and initial design doc is due next Tuesday, the 13th at midnight - 4000 words, or 6-10 pages for the both initial and final design doc - should have pseudocode - shouldn't do the project, then copy and paste code into the design doc - sample design docs from other semesters are probably out there, so take a look - Advice + One step at a time - get a little working and make sure it works with your tests - Testing is important - in the design doc, you should provide a testing strategy + Find a good tool, including a debugger, and use it - Thomas recommends Eclipse - its debugger is great, and it can link up with Subversion - For more information + README file in the installation has lots of good stuff + The class webpage for introductions, background, and the code + Should read the code to see exactly what is going on - Subversion + Not everyone has a subversion account as of now + Subversion and CVS allow multiple people to work on the same code concurrently + 2 students editing the same file - first one commits, student 2 has to update and merge the changes, then can commit + SVN references are online at http://svnbook.red-bean.com + Windows shell plugin called TortoiseSVN is at http://tortoisesvn.tigris.org + Eclipse plugin called Subclipse is pretty good --- END OF NACHOS TUTORIAL --- - Shouldn't be a problem with Nachos assert() function in Java 1.5, even though assert is a keyword now in Java 1.5 Next Topic: Condition Variables - The professor left off on deadlocks, which he will pick back up on Wednesday - Process synchronization using condition variables + Processes or threads can cooperate using wait() and signal() along with condition variables + Can think of condition variables as queues and wait() and signal() as associated methods + The wait() operation means the current process waits until another process invokes signal() + The signal() operation resumes exactly one suspended process - If there aren't any suspended processes, then this has no effect - Can map P and V from semaphores onto wait() and signal() + difference is that signal() doesn't change any state, whereas with a semaphore, you would increment on a V - signal() and wait() are used to control synchronization within monitors + Only one process can be running within a monitor at a time + This maps onto Java as follows: - There is a monitor class, and on entry to any method in the class, you take a lock, and when you leave the class, you release the lock - One binary semaphore associated with each monitor (binary semaphore is a lock) + mutual exclusion is implicit - do a P on entry to any routine and a V on exit - Monitors are a higher level concept than P and V, and they are easier and safer to use - Monitors are a synchronization mechanism that combine 3 features: + Shared data + Operations on the data + Synchronization and scheduling between threads that operate on the data - Monitors are especially convenient for synchronizing involving a lot of state - Monitors need more facilities than just mutual exclusion - they need some way to wait - when you call wait inside a monitor, you can either busy wait or put the process to sleep inside the monitor - Condition variables are the things threads wait on + Can wait on a condition, which releases the monitor lock, puts the process to sleep + When a process wakes up again, it implicitly reacquires the monitor lock immediately + Can signal on a condition - wakes up one process waiting on the condition variable - can be FIFO or some other type of queue - If nobody waiting, doesn't do anything + Can also broadcast on a condition - signal all waiting processes - Several variations on wait and signal mechanism - vary in terms of who gets the monitor lock after the signal + We use Mesa-semantics - on a signal, the signaller keeps the monitor lock, and the awakened process waits for the monitor lock with no special priority - a new process could get it before the awakened one - this means that the lock the awakened process is waiting for could have come and gone, so it must check again as to whether the expected state is preserved, and it must be prepared to sleep again if it cannot acquire the lock + Contrasts with Hoare-semantics - on a signal, the signaller passes the lock to the awakened process - the single process that is signaled is guaranteed to be woken up - Readers/writers problem with monitors + Each synchronization operation gets encapsulated in a monitor's procedure - checkRead, checkWrite, doneRead, doneWrite + 2 conditions: OKToRead, OKToWrite + on a checkRead, if the number of active writers plus the number of waiting writers is greater than 0, then wait, increment the number of waiting readers, and wait on the condition OKToRead + if a thread calls checkRead, it will go to sleep - once a thread signals the condition OKToRead, the waiting reader will decrement the number of waiting readers, become an active reader (by incrementing the active reader count), and it will start the read operation + when a reader is done reading, it will decrement the active reader count, then check if the number of active readers is 0 and the number of waiting writers is > 0. if this is the case, then it will signal that it is OKToWrite + checkWrite - done in a loop - while there are any active readers or active writers, wait by: + incrementing the waiting writers count + waiting on the condition OKToWrite + going to sleep - get woken up when a reader is done and signals that it is OKToWrite + then, decrement waiting writers count and check if there are any active readers or active writers + if there are no active readers or active writers, then increment the active writers count and write - when done, decrement active writers count and check if there are any waiting writers - if there are any waiting writers, then signal the next one that it is OKToWrite - otherwise, signal all the waiting readers that it is OKToRead - Monitors Summary + The professor says that monitors are not present in very many languages yet, but are extremely useful - this is changing, though - Nowadays, Java and C# have built-in support for monitors - Also, the POSIX standard specifies monitors + Semaphores use a single structure for both exclusion and synchronization, which contrasts with monitors, which use different structures for each (mutex and condition variables) + Monitors enforce a style of programming in which complex synchronization code doesn't get mixed with other code + Next, we will talk about a mechanism similar to wait and signal that is used internally in UNIX for scheduling system processes + Best existing implementation of monitors is in the Mesa language at Xerox - UNIX has generalized semaphores, created in sets + Several operations on semaphores can be done simultaneously, and increments and decrements can be values greater than one + The kernel does these operations atomically - Associated with each semaphore is a queue of processes suspended on that semaphore + The semop system call takes a list of semaphore ops, each defined on a semaphore in the set and processes them one at a time + 3 cases: - semop is positive: kernel increments value of semaphore and awakens all processes waiting for the value of the semaphore to increase - semop is zero: kernel checks the semaphore's value, if it is zero, it continues with the list, otherwise it blocks the process and has it wait on the semaphore - semop is negative: if the semop's absolute value is less than or equal to the semaphore's value, then the kernel adds the semop to the semaphore value - if the result of that addition is 0, the kernel awakens all processes waiting for the value of the semaphore to equal 0 + otherwise, if the semop's absolute value is greater than the semaphore's value, the kernel suspends the process until the value of the semaphore increases + This is not that popular now in code, but was used in system 5 UNIX, and is still in all the UNIXes today - In UNIX, processes may send each other signals, which are software interrupts + A signal is processed when a process wakes up, or when it returns from a system call + There are about 20 defined signals, e.g. interrupt, quit, illegal instruction Next topic: Introduction to storage allocation and linkers - Object code is divided into 3 parts: + code - the text segment + data + the stack - Why distinguish between different segments of a program? + Because often times we want to share the code segment between processes + Generally the text segment is marked read-only so processes can't modify it + However, data and stack segments must not be shared between processes because they are modified + Might also want to make the distinction for dynamic linking and separate compilation - Code is generated by compilers from source code or from user assembly language + Code contains addresses, some of which are not known at compilation or assembly time + The address of the base of the text segment is not known at compile time + The address of the base of the code isn't known at assembly time + Addresses of other separately-compiled pieces of code aren't known - Why don't we know these addresses + We want the ability to combine pieces of code compiled at different times. + If we compile everything at the same time, there is no problem, but we lose the benefits of having separately-compiled modular components. - Division of responsibility between various portions of the system + Compiler - Generates 1 object file for each source code file, but this information is incomplete because each file generally references information contained in other source files - Provides symbol table and relocation table + Linker - AKA Linkage Editor - Combines all the object files for one or more programs into a single object file that is complete and self-sufficient + Loader - Adjusts addresses in the single object file to reflect the correct load address for the object file - Terms linker and loader often used interchangeably - Linker often includes functions of the loader + OS - Places object files into memory - Allows several different processes to share memory - Provides facilities for processes to get more memory after they have started running + Runtime Library - Works together with the OS to provide dynamic allocation routines such as calloc() and free() in C - What addresses are there in code? + There are absolute addresses within a code segment - e.g. a jump label + The values of these addresses need to be determined + When code is compiled or assembled, these addresses are set as if the code was loaded at address 0 + There are relative addresses - e.g. jump to PC + 20 - these are no problem + There are external addresses - e.g. calling a subroutine from somewhere else - need these addresses + There are addresses of data in the data segment - When compiling, what do we do? + Create a segment table + For each segment, need the segment name, size, and base address at which it was assumed to be loaded + Symbol table contains global definitions - table of labels that are needed in other segments + Relocation table - table of addresses within this segment that need to be fixed - contains internals (references to locations within this segment) and external references (references believed to be external, i.e. we could not find them in the current segment) + Compiler provides these tables along with the object code for each segment - 3 steps in linker/loader: + determine the location of each segment + calculate values of symbols and update the symbol table + scan the relocation table and relocate addresses - The linker + collects all pieces of the program - might involve finding pieces from file system and libraries + assign each segment a final location and build a segment table + resolve all addresses that can be fixed, in order to get new object file - result could be new object file with all addresses resolved or one with some addresses still unresolved + How do you resolve? - Take the symbol table for each segment and assign each symbol its new address - If the linker has been given an absolute address for loading, then the absolute address is calculated - If the linker is to produce a new relocatable object module, then the absolute address relative to 0 is calculated and the new symbol table is output - The relocation table is scanned and for every entry in the code, the value is replaced with the new absolute value - If the linker has been given an absolute value for loading, then the absolute address is calculated - If the linker is to produce a new relocatable object module, then the absolute address relative to 0 is calculated and the new relocation table is output -- END OF LECTURE -- We will pick up with dynamic storage allocation on Monday.