Lecture 8 notes

Continue read/write coherence
Examples of critical section
Introduction to deadlock

by John Win, Jason Gregori, Parth Patel

Recall the mutex that was created at the end of Lecture 7: (Note: all code is pseudocode)

void acquire(mutex_t *m) {
   while (test_and_set(m, 1) == 1 )
       /* spin */;
}
 
void release(mutex_t *m) {
    *m = 0;
}

What if your computer implemented its memory using Professor Kohler's patented Moron Memory(c)? Moron Memory simply ignores 10% of writes to memory. Well, an acquire could possibly not be recorded, and another acquire could take place. That would not be good; more than one process would be accessing a lockable item. That's where Read/Write Coherence comes in.

The read of a memory location returns the value that was most recently written to that location.

Problematic Example: On the complex x86 architecture, a single instruction can change a lot of memory, for example copying 30 bytes from address %ebx to address %eax:

"memcpy" %ebx, %eax, 30

Problem: What if a read happens on address %eax while this code was executing elsewhere? It could get a combination of what was there before and after. Read/write coherence doesn't apply to such massive objects. What the processor implements is:

Read/Write Coherence for small objects (int, char).

• store 4 bytes atomically (if aligned)
• load 4 bytes atomically (if aligned)
• test_and_set [xchgl] atomic

Let's say we wanted to implement atomic increment: that is, to increment the value at address x by one.

movl x,%eax
incl %eax
movl %eax,x

This code moves x into a register, increments it, then moves it back. Unfortunately, this is not atomic.

incl x

While this is one instruction, it is actually implemented as the 3 instructions above; it is not atomic.

lock incl x

This is atomic because the machine locks down the bus while doing the 3 instructions above. No other command can read or write to x while this command is going through. The downside of this is that it takes a lot of time.

int c_a_s(unit32_t *addr, unit32_t oldv, unit32_t newv) {
   if (*addr == oldv) {
       *addr = newv;
       return 1;
   } else
       return 0;
}

This is another powerful atomic instruction implemented by many processors, including x86. c_a_s is a function that allows us to make other functions atomic. c_a_s itself is atomic; it takes an address, an old value, and a new value. If the value at the address equals the old value, c_a_s changes it to the new value. In the next section we will see how to use it.

We want to implement a function that finds the square root of a number, then adds 7 to it. How about:

sqrt_plus_7(uint32_t *addr)
{
   *addr = sqrt(*addr) + 7;
}

Lets try this code using two threads (s_p_7() is an abbrieviation of sqrt_plus_7(*addr)):

Problem: Even though we ran s_p_7 twice, from the results, it looks like it has only been run once. This implementation is not atomic. Lets try again:

sqrt_plus_7(uint32_t *addr) {
   uint32_t newv;
tryagain:
   newv = sqrt(*addr) + 7;
   if (c_a_s(addr, *addr, newv) == 0)
       goto tryagain;
}

This approach looks good, it uses the compare and swap, so we know it has atomicity -- right? Wrong: it does not actually change the problem from before. This is because the value of *addr used to compute newv might differ from the value passed to c_a_s. Let's try one more time:

sqrt_plus_7(uint32_t *addr) {
   uint32_t newv;
   uint32_t oldv;
tryagain:
   oldv = *addr;
   newv = sqrt(oldv) + 7;
   if (c_a_s(addr, oldv, newv) == 0)
       goto tryagain;
}

This in fact works!

Let's try this code like we did before:

Is this correct? *addr was changed many times during the s_p_7 of T1. Yes, this is correct because to the outside viewer, everything makes sense. 49 became 14 from a s_p_7 then 14 and 21 was added, then 49 became 14 again from another s_p_7. This makes sense. The order does not matter because two processes were acting on this memory. There were not two s_p_7s that did not seem to both affect the variable like in our first try.

Look for writes to shared state and expand to include dependent reads.

Also, look for dependent reads caused by reading a single object that is bigger than the maximum size of read/write coherence enforced by the processor.

The next example is an ATM. The ATM has two operations, deposit and withdraw. Both operations take a bank account and an amount of money as inputs. Two people can share a bank account (like Professor Kohler and his Mother) and use two different ATMs to access it at the same time. (ba_t is a bank account stuct).

typedef struct ba {
   uint32_t bal;
} ba_t;
 
void deposit(ba_t *b, uint32_t amt) {
   b->bal += amt;
}
 
void withdraw(ba_t *b, uint32_t amt) {
   b->bal -= amt;
}

Hold it. What if when we withdraw, b->bal is less than amt? Since we are using an unsigned int, b->bal would suddenly be a very large number (good for the onwer of the bank account but bad for bank :) ). How about this?

void withdraw(ba_t *b, uint32_t amt) {
   if (b->bal >= amt)
      b->bal -= amt;
}

Now that's more like it.

Shall we test this code? Yes, let's. In the test, Professor Kohler's Mother deposits his allowance while he withdraws some lunch money at the same time.

What went wrong here? The deposit and withdraw of $5 should have essentially cancelled each other out. The end result should be b->bal equals 5 but it's not, it's 10. What we have here is a critical section that is not being protected. Let's follow the first sentence from Finding Critical Sections. First, we need to find shared states (red quoted parts):

void deposit(ba_t *b, uint32_t amt) {
   "b->bal" += amt;
}
 
void withdraw(ba_t *b, uint32_t amt) {
   if (b->bal >= amt)
      "b->bal" -= amt;
}

The shared states are the writes to b->bal. Now, what are the dependent reads (red quoted parts)?

void deposit(ba_t *b, uint32_t amt) {
   "b->bal += amt;"
}

void withdraw(ba_t *b, uint32_t amt) {
   if ("b->bal" >= amt)
      "b->bal -= amt;"
}

These are the critical sections. All the red is part of one critical section. Do you see why these parts must be protected? Imagine a withdraw of $5 is happening from an account with $5 in one process. The if statement reads b->bal but then another process takes control and reads the same b->bal in another withdraw of $5. Then they both finish. b->bal now reads -5 or actually some very large number. ERROR! We need to fix this.

To fix this, let's use our old friend the mutex. Let's make a bank mutex. Here is the updated code with the new parts quoted and in red:

Here is the new mutex:

"mutex_t bank_m;"

ba stays the same:

typedef struct ba {
   uint32_t bal;
} ba_t;
 
void deposit(ba_t *b, uint32_t amt) {
   "acquire(&bank_m);"
   b->bal += amt;
   "release(&bank_m);"
}
 
void withdraw(ba_t *b, uint32_t amt) {
   "acquire(&bank_m);"
   if (b->bal >= amt)
       b->bal -= amt;
   "release(&bank_m);"
}

This code is correct, it will work and not have any problems with critical sections. However there is another problem. The bank mutex, bank_l, gets locked for every deposit and withdraw. This is a performance problem; it will make the whole system slow. What we need is finer-grained locking. Here goes (as always new stuff is in quotes and red):

typedef struct ba {
   uint32_t bal;
   "mutex_t m;"
} ba_t;
 
void deposit(ba_t *b, uint32_t amt) {
   acquire("&b->m");
   b->bal += amt;
   release("&b->m");
}
 
void withdraw(ba_t *b, uint32_t amt) {
   acquire("&b->m");
   if (b->bal >= amt)
       b->bal -= amt;
   release("&b->m");
}

Wow, that makes me happy. We are done...

So, what we just implemented was, first, a coarse grained lock (the bank lock), then a fine grained lock (b->m). The second lock is specific to the bank account. so when we do a lock that will be specific to one bank account and not all.

+ Easier to implement
- Many threads wait for one lock

+ Less contention
- Takes more memory for all those locks
- More locking operations

What if we wanted to add another function to our ATM? Like something to check the balance. Something like this:

void balance(ba_t *b) {
   return b->bal;
}

Would that work? Does it need a mutex lock? Um, no. It only reads and nothing is dependent on that read so it works fine. Reads happen in parallel.

For my final trick, what if b->bal was more than just an uint32_t? What if it was a uint256_t? It would need a mutex because if a write happened during in the middle of the read (from balance), it could read half of the value before the write and half of the value after. This is a case where the object read is so big that plain old read/write coherence is not enough, we need to prevent changes during the read. We need to protect this but we still want to allow multiple reads (since they don't hurt nobody and it would be faster)

But by using locks on the account, we would not be execute multiple balance operations at the same time. However, this should not be the case since we are only reading data, not changing it.

So, we can a new type of lock that has two types of acquire, a readlock and a writelock. The readlock operations are used when we are reading data, and the writelock operations should be used for writing data. The lock type is a read/write lock.

Readlocks allow any number of readers. Writelocks only 1 writer.

This is the structure of our new lock read/write lock. It has additional rcount value which stores the number of reader on a particular bank account.

typedef struct{
   mutex_t m;
   int rcount;
} rwlock_t;

This is the operation to acquire a lock for read operation.

void acquire_read(rwlock_t *l) {
   acquire(&l->m);
   l->rcount++;
   release(&l->m);
}

This is the operation to acquire a lock for write operation. It checks to makes sure that no one is reading data. If someone was reading, it will wait for the reading to end.

void acquire_write(rwlock_t *l) {
   while (1) {
      acquire(&l->m);
      if (l->rcount == 0)
         return;
      release(&l->m);
   }
}

This releases read lock.

void release_read(rwlock_t *l) {
   acquire(&l->m);
   l->rcount--;
   release(&l->m);
}

This releases write lock.

void release_write(rwlock_t *l) {
   release(&l->m);
}

There followed a bogus example that ended up demonstrating some problems with observability. A basic outline of the example follows, but it's best not too look too closely!

typedef struct
{
   int fdused[MAXFD];
   mutex_t fdused_l;
}proc_t;

open(proc_t *p)
{
   acquire(&p->fdused);
   for (i=0; i<MAXFD; i++)
   {
      if (p->fdused[i] == 0)
      {
         p->fdused[i] = 1;
         release(&p->fdused);
         return 1;
      }
   }
}

close(proc_t *p, int fd)
{
   acquire(&p->fdused);
   p->fdused[fd] == 0;
   release(&p->fdused);
}

typedef struct
{
   int fdused[MAXFD];
   mutex_t fdused_l;
   int close_count;
}proc_t;

open(proc_t *p)
{
   tryagain:
   int old = p->close_count;
   acquire(&p->fdused);
   for (i=0; i<MAXFD; i++)
   {
      if (p->fdused[i] == 0)
      {
         p->fdused[i] = 1;
         release(&p->fdused);
         if (close_count != old)
             goto tryagain;
         return 1;
      }
   }
}

close(proc_t *p, int fd)
{
   acquire(&p->fdused);
   p->fdused[fd] == 0;
   p->close_count++;
   release(&p->fdused);
}