CS111 Lecture 17: Distributed Systems II

Nicholas Brown, Steven Snyder, Rob Green, Huy Le, Timur Shevekhman

It is recommended that you review the concept of RPCs before reading on.

The server outline pseudocode below is reprinted from the previous lecture's introduction to distributed systems, and is an example of serial programming. This server is a remote procedure call server; it proccesses RPC requests from clients, performs the required actions, and returns a response message.

1   void serve(int fd)
2   {
3      char buf[1000];
4      // read client request
5      read(fd, buf, 1000);
6 
7      // process client request in buf
8
9      // send response to client
10     write(fd, ...);
11   }
 
12   int main(int c, char *v[])
13   {
14      int fd = socket(...);
15      bind(fd, &sockaddr, ...);
16      listen(fd, 5);
17      while (1) {
18           int conn = accept(fd, ...);
19           serve(conn);
20           close(conn);
21      } // while
22   } // main

The above implementation of the server has a critical weakness when it comes to utilization (and robustness, as we'll discuss later). It can only handle one request at a time! Once a connection is opened and being serviced, no other requests can be serviced until the current connection has been served. This provides for poor utilization of the server's resources because the server may spend a lot of time waiting for messages from the client, especially if the client is communicating through a heavily congested link. Utilization would be greatly improved if the server had something to do while waiting for a client's message.

This implementation also presents several robustness problems for the server. One problem is after the server receives a connection from a client, the client may never sends a request. The RPC packet may be dropped in the network, the client process may have crashed, or the client could be trying to exploit the server's resources by opening pointless connections. The server could block forever during the call to read(). This robustness problem leaves the server vulnerable to Denial of Service attacks.

Definition: Denial of Service - an attack whose goal is to deny service to legitimate users by monopolizing a server's resources. Often abbreviated DoS.

Parallel programming serves to not only solve the utilization problems of the serial server process, but to also reduce (though it cannot completely eliminate) the threat of denial of service attacks. With parallel architecture, the server can perform multiple tasks at once. This idea comes from computer architecture, where pipelining is used to improve uitilization of a processor.

The server pseudocode below improves utilization and robustness by allowing connections to clients to be serviced in parallel. Here, the server handles each client connection using separate process generated with fork():

1  int main(int c, char *v[])
2  {
3       int fd = socket(...);
4       bind(fd, &sockaddr, ...);
5       listen(fd, 5);
6       while (1) {
7            int conn = accept(fd, ...); // wait for a new connection
8            if (fork() == 0) { // fork a new process
9                serve(conn); // serve the connection
10               close(conn); // close the connection
11               exit(0); 
12           } // if
13           else // if the parent process
14               close(conn); // close the connection file descriptor
15      } // while
16 } // main

Note that the code here is a bit more tricky because we have to remember to exit out of the child process once the request is done being serviced. If the child processes never exit (we forgot to put line 11 of the code, for example), eventually, the kernel will run out of new process space. We also have more file descriptor bookkeeping to do, as we have to close the file descriptors corresponding to client connections where they exist in the parent's file descriptor table.

This server implementation achieves a much better utilization than the first since it can now service clients in parallel. While one server process is blocking on read(), other processes are handling clients and the parent process is generating a new working process whenever a new client creates a connection. However, this implementation still faces robustness problems. While an individual client cannot block every server process as before, clients control the server's process count and therefore can control the amount of server resources in use. In a type of Denial of Service attack, an attacker could open and hold numerous connections to the server. If an indefinite amount of processes are forked to service an indefinite amount of client connections, the server would eventually run out of kernel resources to hold process information, and cause the server to thrash as it is constantly swapping out process descriptor tables on the swap disk. This would bring server utilization to zero as even legitimate clients take forever to serve.

The potential problem of the server forking an arbitrarily large amount of process can be fixed by simply limiting the amount of child processes servicing client connections at one time. A revised version of the server pseudocode is below. Note lines 11, and 20-24 which have been added to the server code we last saw above.

1   int main(int c, char *v[])
2   {
3        int fd = socket(...);
4        bind(fd, &sockaddr, ...);
5        listen(fd, 5);
6        // initialize process pool count
7        int npool = 0;
8        while (1) {
9             int conn = accept(fd, ...);
10            // increment process pool count
11            npool++;
12            if (fork() == 0) {
13                 serve(conn);
14                 close(conn);
15                 exit(0);
16            }
17            else
18                 close(conn);
19            // if the pool has reached 100 forked processes
20            if (npool == 100) {
21                 // wait on a process that has finished servicing a client, then decrement pool count
22                 waitpid(-1, ...);
23                 npool--;
24            }
25       } // while
26  } // main

This implementation successfully prevents the server forking processes beyond its resources, and we achieve a higher level of robustness! Clever readers may notice that we have not solved the problem of a Denial of Service attack completely. In fact, we have made one form of attack even easier! An attacker needs only to open 100 dummy connections to the server to make it completely useless to any other clients. It no longer needs to open enough processes to use up the server's kernel resources.

The per-client fork implementation of the server fixed several utilization and robustness issues over the initial server implementation, but there are still improvements yet to be made. In the per-client fork, each time a new connection is opened with a client, there is an overhead associated with forking to create the new process to service the connection. This connection overhead can be rectified with a new implementation that uses preforking. With preforking, a set number of processes, the pool size, are forked prior to receiving connections. Therefore, when a connection is received, there is no overhead associated with creating a new process!

The pseudo-code below is a sample implementation of the server using preforking:

1   int main(int c, char *v[])
2   {
3        int fd = socket(...);
4        bind(fd, &sockaddr, ...);
5        listen(fd, 5);
6   
7        int i;
8        for(i = 0; i < 100; i++)
9             if(fork == 0)
10                 break;
11  
12       while (1) {
13            int conn = accept(fd, ...);
14            serve(conn);
15            close(conn);
16       }
17  }

An important thing to note is that accept() acts as an atomic operation. This prevents any synchronization issues where more than one process could receive and process the same connection. Therefore, only one process will service an incoming connection!

The preforking implementation of the server works very well with a large client base. However, if there are only a few clients being serviced at any given time, and we have 100 processes able to serve clients, we are essentially wasting resources with the other processes that are blocked waiting for an incoming connection. The per-client fork did not have the problem of wasted resources because for each individual client connection it created a process, serviced the client, and then exited the process; no resources were wasted with extraneous processes.

Naturally, there is a way to harness the advantages of both strategies with a hybrid approach! The server starts out with a set pool size (i.e. 20) and services clients normally like a preforking implementation. When the number of clients being served goes beyond the original pool size, the server will fork more processes to keep up with the growing client base, up to a specified maximum number of processes. The maximum prevents the robustness problem described at the beginning of this page, where a growing number of evil clients connect to the server until it runs out of resources.

If the number of clients drops and the processes that were forked to keep up with the growing client base become idle for some amount of time, the process pool will start to shrink to free up server resources not being used. The pool, given that the number of connections remains low, will eventually shrink down to the original allotted pool size. An adaptation of this hybrid strategy of per-client forking and preforking is present in the Apache server architecture. Apache web servers operate in different modes depending on the amount of traffic. If the system is very busy, the process pool will be kept large to service incoming clients immediately as they connect. After traffic dies down, the Apache server kills unneeded processes and returns to a state of lower resource use.

Huy Le

Forking isn't the only way to service client connections in parallel. Other venues available to programmers are the use of threads and event-driven programming. Each method has its own advantages and disadvantages.

If a server uses multiple threads rather than processes to service requests, one can save kernel resources since they share memory and other resources but are able to execute independently. Context switching between threads is also faster than context switching for processes. While this method is vulnerable to the same type of denial of service attack, where the attacker opens more connections than the server's memory or file descriptors can support, it can handle a larger amount of connections than processes can due to taking up less memory. Synchronization is also an issue with threads, since they share resources. Its data will need to be processed in the correct order to prevent data from being changed at the same time by different threads by using something like a mutex. If not done correctly, this may lead to failure by race conditions or deadlocks. If one of these threads fails, it affects all of the clients using the server.

In event-driven programming, the program will never block except when calling the poll() system call. This eliminates all synchronization issues associated with the usage of multiple processes or threads. All system calls such as read() or write() do not block and instead return an error if they would have blocked. This allows the server to skip any file descriptors that are not ready and go to the next connection. The problem with using event-driven programming is that concurrency is an application programmer's problem since a program might have partial data before it would have blocked and now must have a way of remembering such data. While it is difficult to code for it also uses very little resources while still performing well.

Steven Snyder

While at work an RPC server programmer should repeat the mantra: "evil clients will abuse my server". When a server is made available to the global internet, or even a local network of peers who are assumed to be trusted, it must be as robust as possible or it will eventually fall under the attack of evil peers. Even a fully trusted network can be infiltrated by ill-meaning users, so no network can be assumed to be safe. Additionally, the majority of robustness improvements against evil peers also improve a server's utilization of resources for serving good peers even when no evil peers are trying to take advantage of the system.

A server should reserve devote few resources as possible to any connection. This not only reduces the ability of attackers to perform Denial of Service attacks, but reduces the overhead of connections and can increase the amount of legitimate traffic that can be served at once.

The most dangerous connections are new connections. Before a client has made any requests, the server has no way of telling whether the client will ever do anything at all. Through clever programming, it is possible for the server to devote no resources to a new connection whatsoever. A classic version of the DoS attack floods the server with requests for new connections but never does anything with them. It is possible for the server to offload its new connection overhead to the client!

By clever use of a good keyed encryption algorithm, the server can store the information in a connection acknowledgement packet sent back to the client. This information would contain identifying information such as the client's IP address and the time of the connection, encoded using a key that only the server knows. By requiring clients to send this key along with their RPC messages, the server can restore the connection information from the key instead of having to store it locally. Evil clients are unable to spoof the information because they do not know the server's encryption key. This solution is similar to the addition of SYN cookies to servers using TCP for connections.

Robustness and utilization problems not only occur at the application level in distributed systems, but at the kernel level as well. The network interface creates a possible bottleneck in the system. Let's look at the series of events that occur when a packet shows up on the network card.

Upon receiving the packet, the network card generates an interrupt. The processor handles this interrupt by reading the packet, then processing it and passing it to the kernel. For an example, let us assume that the interrupt handler operation takes 5us, and 45us are required to process an average sized packet. This means each packet requires 50us of processing time, limiting us to 20,000 packets/second.

We can improve the performance of this system dramatically. Instead of the network card generating an interrupt whenever a packet is received, we could use a polling method in the kernel and a network interface with DMA (direct memory access). When the kernel has finished dealing with all of the packets it has already received, it checks the network card's memory for a new packet, requiring only 5us of time per packet. Now the kernel can process 200,000 packets/second, an improvement of a whole order of magnitude. This improves the ability of a server to not only serve many clients at once, but also reduces the overhead from dealing with packets that are part of a DoS attack.

Huy Le

Our goal is to allow computers to access data stored on another computer using the file system interface. We can achieve this by making distributed systems look like disks. Implementation:

Above is a diagram of how Network File System (NFS) works. On the client's side, the NFS module (/nfs/kohler/) is connected to the Virtual File System (VFS). Its job is to convert system calls such as read() or write() from varying file systems (ext3, FAT, ospfs, etc.) into Remote Procedure Calls (RPC) and then send the RPCs to the server. On the server's side, the NFS is in user space. The NFS at the server takes the data that the RPCs clients sent and converts the RPCs into system calls that can read or write files on the server's storage disk.

In synchronous RPCs, the client sends a single message to the server and blocks until it receives a response. The server then responds to the client and sends data back. When the client receives the response, the client unblocks to send the next message and then waits again for the server to respond.

This implementation of RPCs has very low utilization, as it is spending most of its time idling while the RPC makes a round trip to the server and then back to the client again.

In order to improve utilization, multiple RPCs are sent back to back to use the former idling time to send data instead. Although RPCs are sent in order, they do not always come back to the client in the same order which means there has to be a way to identify which RPC it receives to which RPC the client has previously sent.

There are other ways to improve the performance of RPCs other than using asynchronous RPCs. A few such methods are batching more than one character at a time, dallying, caching, or prefetching for reads. But these kinds of solutions bring a major issue: Cache Coherence Since moving data across the network is inherently slower than moving data inside the computer, the client cache can become outdated frequently. One solution is to invalidate the client's cache after a certain period of time, but this can be very resource intensive because of the latency of the network. An easier solution, and the one that NFS uses, is to simply Give Up; relax the consistency requirement on file data. That is, we allow the caches to get out of date.

At the beginning of the lecture we’ve looked the differences between serial and parallel form of communication between a client and a server. Using serial connections, a server serves one client at the time, and is forced to spend most of the time waiting on a client. Using the idea of pipelining from computer architecture, we can use parallel connections, where multiple processes handle each connection separately.

Two ways to implement parallel connections are per-client fork and process pool. Process pool solves a problem with per-client fork, where a single client can control how many processes are running on a server, by limiting the number of forked processes. This approach, however, makes certain DoS attacks easier. Preforking solves overhead associated with memory allocation that happens on every new connection in per-client fork model.

Other models that are used in parallel programming are threads and event-driven programming. These two models use less resources, but some problems, such as synchronization, can be difficult to address.

When we talk about distributed systems, we must talk about robustness of a system, and, in particular, server robustness. At the kernel level, it can be improved with a combination of polling and interrupts, which can be used for high-priority tasks.

One example of distributed systems applications is a Network File System. Associated with Network File Systems are Synchronous and Asynchronous RPC, and Cache Coherence.