====== Lecture 14 Scribe Notes: More File Systems, Virtual Memory ======
By Nathaniel Dang, Konstantin Rogachyov, and Brian Wallerstein

===== File Systems =====

==== Invariants and Ordering ====
Last lecture, we discussed the scenario in which we couldn't move a file from one directory to another without potentially losing the file.  This was due to the fact that if the PG&E company were to cut the power during the //move// operation, this could result in the violation of one of the file system invariants, in particular one of the top three invariants (the fourth invariant is a "weak" invariant, resulting only in a space leak).  This is a major robustness problem!


**Reiterating the file system invariants again:**\\
1)  Every block must be used for a single purpose.\\
2)  Every referenced block is initialized.\\
3)  No referenced block is free.\\
4)  Every unreferenced block is free (weak)  (counts at most the number of links)\\
\\
**File System Ordering:**\\

^ 0 ^ 1 ^ 2 ^ 3  ...   9 ^ 10    ...    2047 ^ 2048 - 3191 Write Ahead Log ^^^^
| Boot Sector | Super Block | Free Block Bitmap | Inode Data | File System Data |  |\\


**Sample Inode Block:**\\
^ 1 ^^^^
| Type: Directory\\ Number of Links = 1\\ Direct[0] = 12\\ ... |\\


**Sample File System Setup:**\\
^ 0 ^ 1 ^ 2 ^ 3  ...   9 ^ 10...11 ^ 12 ^ 13 ^ .... ^ 20 ^ .... ^ 2047 ^ 2048 - 3191 Write Ahead Log ^^^^
| Boot Sector | Super Block | Free Block Bitmap | Inodes |  |  /  | :-) |  | /sub |  |  |  |\\

^ Block 2 ^   ^ Block 3 ^^^     ^ Block 12 ^^^ ^ Block 13 ^ ^ Block 20 ^^
| Free Data Blocks:\\ 10, 11, 14 - 19, 21+ |     | ** 1 **\\ Type: Directory \\ NumLinks = 1\\ Direct[0] = 12\\ ... | ** 2 ** \\ Type: Reg File\\ NumLinks = 1\\ Direct[0] = 13\\ ... | ** 3 ** \\ Type: Directory\\ NumLinks = 1\\ Direct[0] = 20\\ ... |     | ** Directory Entry 1 **\\ Name: Sub \\ Inode: 3 | ** Directory Entry 2 **\\ Name: hello.txt\\ Inode: 2 | ... | | ** Data Block **\\ Data:\\ :-) |      | ** Directory Entry 1 **\\ Name: \\ Inode: 0 | ... |

\\




==== Removing A File ====
\\
**Let's try removing a file, the proper operations are:**\\
A)  Clear directory entry\\
B)  Decrement link count\\
C)  If the link count is 0, then mark the blocks as free.\\
\\



**Example: Removing hello.txt**\\
^ 0 ^ 1 ^ 2 ^ 3  ...   9 ^ 10...11 ^ 12 ^ 13 ^ .... ^ 20 ^ .... ^ 2047 ^ 2048 - 3191 Write Ahead Log ^^^^
| Boot Sector | Super Block | Free Block Bitmap | Inodes |  |  /  | :-) |  | /sub |  |  | Clear Directory Entry | Decrement Link Count | Commit |\\

^ Block 2 ^   ^ Block 3 ^^^     ^ Block 12 ^^^ ^ Block 13 ^ ^ Block 20 ^^
| Free Data Blocks:\\ 10, 11, 14 - 19, 21+ |     | ** 1 **\\ Type: Directory \\ NumLinks = 1\\ Direct[0] = 12\\ ... | ** 2 ** \\ Type: Reg File\\ NumLinks = 1\\ Direct[0] = 13\\ ... | ** 3 ** \\ Type: Directory\\ NumLinks = 1\\ Direct[0] = 20\\ ... |     | ** Directory Entry 1 **\\ Name: Sub \\ Inode: 3 | ** Directory Entry 2 **\\ Name: hello.txt\\ Inode: 2 | ... | | ** Data Block **\\ Data:\\ :-) |      | ** Directory Entry 1 **\\ Name: \\ Inode: 0 | ... |
\\



**A) Clear Directory Entry**\\

**Example: Removing hello.txt:**\\
^ 0 ^ 1 ^ 2 ^ 3  ...   9 ^ 10...11 ^ 12 ^ 13 ^ .... ^ 20 ^ .... ^ 2047 ^ 2048 - 3191 Write Ahead Log ^^^^
| Boot Sector | Super Block | Free Block Bitmap | Inodes |  |  /  | :-) |  | /sub |  |  | <del>Clear Directory Entry</del> | Decrement Link Count | Commit |\\

^ Block 2 ^   ^ Block 3 ^^^     ^ Block 12 ^^^ ^ Block 13 ^ ^ Block 20 ^^
| Free Data Blocks:\\ 10, 11, 14 - 19, 21+ |     | ** 1 **\\ Type: Directory \\ NumLinks = 1\\ Direct[0] = 12\\ ... | ** 2 ** \\ Type: Reg File\\ NumLinks = 1\\ Direct[0] = 13\\ ... | ** 3 ** \\ Type: Directory\\ NumLinks = 1\\ Direct[0] = 20\\ ... |     | ** Directory Entry 1 **\\ Name: Sub \\ Inode: 3 | ** Directory Entry 2 **\\ Name: <del>hello.txt</del>\\ Inode: <del>2</del> **0** | ... | | ** Data Block **\\ Data:\\ :-) |      | ** Directory Entry 1 **\\ Name: \\ Inode: 0 | ... |
\\
\\
**B/C) Decrement Link Count and Free Blocks (if Number of Links = 0)**\\

**Example: Removing hello.txt:**\\
^ 0 ^ 1 ^ 2 ^ 3  ...   9 ^ 10...11 ^ 12 ^ 13 ^ .... ^ 20 ^ .... ^ 2047 ^ 2048 - 3191 Write Ahead Log ^^^^
| Boot Sector | Super Block | Free Block Bitmap | Inodes |  |  /  | :-) |  | /sub |  |  | <del>Clear Directory Entry</del> | <del>Decrement Link Count</del> | Commit |\\

^ Block 2 ^   ^ Block 3 ^^^     ^ Block 12 ^^^ ^ <del>Block 13</del> ^ ^ Block 20 ^^^
| Free Data Blocks:\\ 10, 11, **13**, 14 - 19, 21+ |     | ** 1 **\\ Type: Directory \\ NumLinks = 1\\ Direct[0] = 12\\ ... | ** 2 ** \\ Type: Reg File\\ NumLinks=<del>1</del> **0**\\ Direct[0]=<del>13</del> **0**\\ ... | ** 3 ** \\ Type: Directory\\ NumLinks = 1\\ Direct[0] = 20\\ ... |     | ** Directory Entry 1 **\\ Name: Sub \\ Inode: 3 | ** Directory Entry 2 **\\ Name: \\ Inode: 0 | ... | | **Data Block**\\ Data:\\ **JUNK** |      | ** Directory Entry 1 **\\ Name: \\ Inode: 0 | ... |
\\

After the commit is performed, the file is removed from the file system!\\

**Question:**
What would have happened if we had performed step C) first and then PG&E decided to cut the power?\\
**Answer:** We would have violated the third invariant! We can see that we must be careful in choosing the order of operations.\\
\\





==== Moving A File ====

**Now, our goals in moving a file from one directory to another are:**\\ 
 1) __Always__ preserve invariants 1-3.\\
 2) No matter when a crash might occur, we want the file to exist under //at least// the old or new file name.  \\
\\
\\
**Steps for properly moving a file from one directory to another**\\
A) Increment link count\\
B) Create new directory entry for the inode\\
C) Delete the old directory entry\\
D) Decrement the link count\\ 
\\
\\
**Example: Moving hello.txt**\\
^ 0 ^ 1 ^ 2 ^ 3  ...   9 ^ 10...11 ^ 12 ^ 13 ^ .... ^ 20 ^ .... ^ 2047 ^ 2048 - 3191 Write Ahead Log ^^^^^
| Boot Sector | Super Block | Free Block Bitmap | inodes |  |  /  | :-) |  | /sub |  |  | Increment Link Count | Create sub\hello.txt | Delete The Old Directory Entry | Decrement The Link Count | Commit |\\

^ Block 2 ^   ^ Block 3 ^^^     ^ Block 12 ^^^ ^ Block 13 ^ ^ Block 20 ^^
| Free Data Blocks:\\ 10, 11, 14 - 19, 21+ |     | ** 1 **\\ Type: Directory \\ NumLinks = 1\\ Direct[0] = 12\\ ... | ** 2 ** \\ Type: Reg File\\ NumLinks = 1\\ Direct[0] = 13\\ ... | ** 3 ** \\ Type: Directory\\ NumLinks = 1\\ Direct[0] = 20\\ ... |     | ** Directory Entry 1 **\\ Name: Sub \\ Inode: 3 | ** Directory Entry 2 **\\ Name: hello.txt\\ Inode: 2 | ... | | ** Data Block **\\ Data:\\ :-) |      | ** Directory Entry 1 **\\ Name: \\ Inode: 0 | ... |
\\
**A) Increment Link Count**\\

**Example: Moving hello.txt**\\
^ 0 ^ 1 ^ 2 ^ 3  ...   9 ^ 10...11 ^ 12 ^ 13 ^ .... ^ 20 ^ .... ^ 2047 ^ 2048 - 3191 Write Ahead Log ^^^^^
| Boot Sector | Super Block | Free Block Bitmap | inodes |  |  /  | :-) |  | /sub |  |  | <del>Increment Link Count</del> | Create sub\hello.txt | Delete The Old Directory Entry | Decrement The Link Count | Commit |\\

^ Block 2 ^   ^ Block 3 ^^^     ^ Block 12 ^^^ ^ Block 13 ^ ^ Block 20 ^^
| Free Data Blocks:\\ 10, 11, 14 - 19, 21+ |     | ** 1 **\\ Type: Directory \\ NumLinks = 1\\ Direct[0] = 12\\ ... | ** 2 ** \\ Type: Reg File\\ NumLinks = <del>1</del>**2**\\ Direct[0] = 13\\ ... | ** 3 ** \\ Type: Directory\\ NumLinks = 1\\ Direct[0] = 20\\ ... |     | ** Directory Entry 1 **\\ Name: Sub \\ Inode: 3 | ** Directory Entry 2 **\\ Name: hello.txt\\ Inode: 2 | ... | | ** Data Block **\\ Data:\\ :-) |      | ** Directory Entry 1 **\\ Name:  \\ Inode: 0 | ... |
\\

**B) Create New Directory Entry for the Inode**\\

**Example: Moving hello.txt**\\
^ 0 ^ 1 ^ 2 ^ 3  ...   9 ^ 10...11 ^ 12 ^ 13 ^ .... ^ 20 ^ .... ^ 2047 ^ 2048 - 3191 Write Ahead Log ^^^^^
| Boot Sector | Super Block | Free Block Bitmap | inodes |  |  /  | :-) |  | /sub |  |  | <del>Increment Link Count</del> | <del>Create sub\hello.txt</del> | Delete The Old Directory Entry | Decrement The Link Count | Commit |\\

^ Block 2 ^   ^ Block 3 ^^^     ^ Block 12 ^^^ ^ Block 13 ^ ^ Block 20 ^^
| Free Data Blocks:\\ 10, 11, 14 - 19, 21+ |     | ** 1 **\\ Type: Directory \\ NumLinks = 1\\ Direct[0] = 12\\ ... | ** 2 ** \\ Type: Reg File\\ NumLinks = 2\\ Direct[0] = 13\\ ... | ** 3 ** \\ Type: Directory\\ NumLinks = 1\\ Direct[0] = 20\\ ... |     | ** Directory Entry 1 **\\ Name: Sub \\ Inode: 3 | ** Directory Entry 2 **\\ Name: hello.txt\\ Inode: 2 | ... | | ** Data Block **\\ Data:\\ :-) |      | ** Directory Entry 1 **\\ Name: **hello.txt** \\ Inode: <del>0</del>**2**| ... |
\\

**C) Delete the Old Directory Entry**\\

**Example: Moving hello.txt**\\
^ 0 ^ 1 ^ 2 ^ 3  ...   9 ^ 10...11 ^ 12 ^ 13 ^ .... ^ 20 ^ .... ^ 2047 ^ 2048 - 3191 Write Ahead Log ^^^^^
| Boot Sector | Super Block | Free Block Bitmap | inodes |  |  /  | :-) |  | /sub |  |  | <del>Increment Link Count</del> | <del>Create sub\hello.txt</del> | <del>Delete The Old Directory Entry</del> | Decrement The Link Count | Commit |\\

^ Block 2 ^   ^ Block 3 ^^^     ^ Block 12 ^^^ ^ Block 13 ^ ^ Block 20 ^^
| Free Data Blocks:\\ 10, 11, 14 - 19, 21+ |     | ** 1 **\\ Type: Directory \\ NumLinks = 1\\ Direct[0] = 12\\ ... | ** 2 ** \\ Type: Reg File\\ NumLinks = 2\\ Direct[0] = 13\\ ... | ** 3 ** \\ Type: Directory\\ NumLinks = 1\\ Direct[0] = 20\\ ... |     | ** Directory Entry 1 **\\ Name: Sub \\ Inode: 3 | ** Directory Entry 2 **\\ Name: <del>hello.txt</del>\\ Inode: <del>2</del>**0** | ... | | ** Data Block **\\ Data:\\ :-) |      | ** Directory Entry 1 **\\ Name: hello.txt \\ Inode: 2| ... |
\\

**D) Decrement the Link Count**\\


**Example: Moving hello.txt**\\
^ 0 ^ 1 ^ 2 ^ 3  ...   9 ^ 10...11 ^ 12 ^ 13 ^ .... ^ 20 ^ .... ^ 2047 ^ 2048 - 3191 Write Ahead Log ^^^^^
| Boot Sector | Super Block | Free Block Bitmap | inodes |  |  /  | :-) |  | /sub |  |  | <del>Increment Link Count</del> | <del>Create sub\hello.txt</del> | <del>Delete The Old Directory Entry</del> | <del>Decrement The Link Count</del> | Commit |\\

^ Block 2 ^   ^ Block 3 ^^^     ^ Block 12 ^^^ ^ Block 13 ^ ^ Block 20 ^^
| Free Data Blocks:\\ 10, 11, 14 - 19, 21+ |     | ** 1 **\\ Type: Directory \\ NumLinks = 1\\ Direct[0] = 12\\ ... | ** 2 ** \\ Type: Reg File\\ NumLinks = <del>2</del>**1**\\ Direct[0] = 13\\ ... | ** 3 ** \\ Type: Directory\\ NumLinks = 1\\ Direct[0] = 20\\ ... |     | ** Directory Entry 1 **\\ Name: Sub \\ Inode: 3 | ** Directory Entry 2 **\\ Name: \\ Inode: 0 | ... | | ** Data Block **\\ Data:\\ :-) |      | ** Directory Entry 1 **\\ Name: hello.txt \\ Inode: 2| ... |
\\

After the commit is performed, hello.txt has been moved from the root directory to root/subdir!


==== Improving Robustness ====
One Idea: Recover lost space by running a file system checker program.
    - Examine the file system.
    - Search for any invariant violations.
    - Fix these violations.


Another Idea: Put the file systems operations in a __LOG__.

**GOLDEN RULE OF ATOMICITY**: Never modify the only copy of a file.
\\

To create atomicity for file system operations, MAKE A COPY! \\ \\
**Write Ahead Logging**: \\
  * Make a copy of a file system operation
    - Write results of operation to a __LOG__ (or __Journal__).
    - Write a __COMMIT RECORD__ to the log.
      * The preceding portion of the log is a faithful copy of the file system after the operation.
      * We are not allowed to write the __COMMIT RECORD__ until we know that the log is right.
    - Copy the results into the main file system.
      * We are not allowed to write into the main file system until we know that the commit record is right.
    - Cancel the commit record.
      * We are not allowed to commit the record until the results are copied into the main file system.
  * After we reboot from a crash:
    - Ignore all of the log entries that do not have a commit record.
    - Replay all of the log entries that do have a commit record.
  * Disadvantage:
    - Twice the number of writes are needed!
  * Advantage:
    - Most of the writes are contiguous!
  * Example:
    - An example of write ahead logging can be seen above in the Removing a File section and/or the Moving a File section.

\\



**IDEMPOTENT Operations:** The result is the same no matter how many times the operation is applied.
  * Example:  x = 1 will always have the same result no matter how many times it is applied, whereas x++ will have a different result each time.

\\

Example: \\

**Step One:**\\
^ 0 ^ 1 ^ 2 - 3 ^ 4 ^ 5 ^ 6 ... 2047 ^^^
| Boot Sector | Super Block | Block Bitmap | 1 | 2 | Decrement contents of Block 5 (A)| Increment contents of Block 4 (B)| Commit |

**Step Two:**\\
^ 0 ^ 1 ^ 2 - 3 ^ 4 ^ 5 ^ 6 ... 2047 ^^^
| Boot Sector | Super Block | Block Bitmap | 2 | <del>2</del> **1** | <del>Decrement contents of Block 5 (A)</del> **New Block #5 is 1** (A)| Increment contents of Block 4 (B) | Commit |

**Step Three:**\\
^ 0 ^ 1 ^ 2 - 3 ^ 4 ^ 5 ^ 6 ... 2047 ^^^
| Boot Sector | Super Block | Block Bitmap | <del>1</del>  **2** | 1 | New Block #5 is 1 (A) | <del>Increment contents of Block 4 (B)</del> **New block #4 is 2 (B)** | Commit |

Even if a crash occurs, it's okay because the correct results are in the log anyway!\\

\\

===== Virtual Memory =====

We want to ensure process memory isolation!\\
\\
**Robustness**\\
  - Segmentation
    * Help ensure isolation by telling architecture the BASE (low bound) and the SIZE (high bound) that the process is allowed access to.
    * The architecture has BASE and SIZE registers that delimit allowed memory accesses.
      * Dangerous instructions – changing the BASE or the SIZE!!
    * Disadvantage: external fragmentation
      * RT11 system solves this with compaction but compaction is difficult because of pointers.
        * After compaction all pointers become invalid.
  - Paged Virtual Memory 
    * Allocate memory to processes in fixed size blocks called pages.
    * Hardware introduces a translation from process address space to physical memory. 
      * That translation is controlled by the kernel!
      * It is called virtual memory because we are virtualizing the addresses.
    * Operation system defines a PAGE MAP FUNCTION
      * Physical Address = PageMap(Virtual Address)
      * Hardware uses this function whenever a process accesses memory
        * Example:
          * Process P wants to access address a
          * Physical Address(a) = PageMap(floor(a/PGSIZE)*PGSIZE) + (a mod PGSIZE)
      * On x86 – PGSIZE is 2^12 or 4KB.
      * Dangerous instruction – changing the page map function!
    * Example:
      * X86-like 2-level page table
        * Reg %cr3 holds the physical address of page directory.
        * The page directory (4096 bytes) is like the doubly indirect block in that it points to the page tables.
        * The page table is like the indirect block in that it directly points to 4096-byte pages.