Example Design Document: Insecure Sample Protocol

This document provides an example of a sample design document for the CS 161 Instant Message protocol. The design detailed within in is, as you may have guessed, not particularly secure. For the purposes of illustration, this design document makes many baseless claims and assumptions regarding the security of our sample solution.

You should not believe any of the claims it makes about the security properties of Java or other technologies. However, it should help to give you an idea of what the TAs want to see in you initial design document. For the impatient, here is a summary of the high points:

This document is in HTML format, but please submit your design documents in PDF format. You do not need to strictly follow the structure of our design document. However, you do need to touch upon the material presented in it.

For those of you that have taken CS 162 or another class that requires a design document, note that we want you to focus on the security properties of your design. In particular, we don't want to see a class hierarchy, method signatures, or other low-level details such as that. Instead, we want to be able to read your design document, and understand the security model of your system, and be convinced that you will produce a secure implementation of the protocol.

Protocol Overview

Or protocol, INSEC, uses a star topology to route messages. There is a single Chat Server through which all messages and control signals are routed. Users authenticate with the Chat Server by sending a a username/password pair that is compared to a cleartext version stored on the server. All communication occurs via a standard TCP/IP connection, and protocol data and control signals are sent as serialized Java objects.

There is one protocol message type per Hamsam API call. Like Hamsam, the protocol is mostly asynchronous and event driven. However, there are a number of Hamsam calls that return values. The protocol adds asynchronous messages and event handlers to handle these cases.

Security model

We assume that people interested in attacking our system will be unsophisticated users, and will not be able to launch complex attacks against the server software. Eavesdropping on network connections requires special expertise unless you are on the same local area network as one of the endpoints of a TCP/IP connection. We assume that users will be able to physically secure their own networks, and only use public network connections (such as WiFi hotspots) that are provided by reputable businesses.

The server is connected to a cable modem that is governed by a Terms of Service agreement that forbids packet sniffers, so despite the broadcast nature of cable modems, we assume that the adversary cannot eavesdrop on our packets. Furthermore, most people use some sort of hardware router these days, which would block packets before they reached the adversary's system. Our ISP will not let third parties change our server's DNS entry without a password. Since the password is only known to the people involved in administering the system an adversary could not redirect our DNS name to a malicious server, and launch a man-in-the middle attack. Our server is robust against conventional denial of service attacks because it is multithreaded, and only routes messages from users that it recognizes.

This server itself runs a commonly available operating system with default settings. Therefore, we can assume that the server OS has been hardened against most network based attacks. We've placed the server itself in an undisclosed location, protecting against physical tampering. While we could conceivably log the messages sent through our system, our users trust us not to do this.

Module Descriptions

The system consists of three basic modules:

Networking's primary security goal is to confirm that all data read from TCP/IP connections obeys Java's serialization conventions, and forms a stream of (int, Serializable[]) tuples. Furthermore, it is responsible for casting the contents of the Serializable array to the correct object types, and to verify that the arrays are of the correct length. Instead of explicitly checking for such errors, internally it throws unchecked exceptions, and then handles them in the logic that initiated the connection. This way, any deviation from the expected networking protocol will result in immediate termination of the connection.

Server is responsible for correct routing of messages to clients, and maintenance of state regarding each account. This state is managed as a few static Java Hashtable objects. These hash tables are Synchronized, alleviating any possibility of race conditions, while still allowing good server-side concurrency. Server also is responsible for authentication; when a connection is initiated, the client sends a "CONNECT" message and a username/password pair. The server then verifies the username/password, and either sends the buddy list and a "CONNECTED" message or sends a "CONNECT_FAILED" message to the client and terminates the connection if appropriate.

Client is responsible for translating Network events into Hamsam events. Client trusts the server, and therefore does not contain any security-related logic.

Summary of Algorithms

Note: The sample protocol doesn't make use of any interesting algorithms; we're looking for non-trivial, security sensitive algorithms that your project employs here. Something at the same level of detail as the description in the cellphone example on the midterm would be reasonable:

Alice wants to send a cellphone text message to Bob securely, over an insecure communication network. Alice's cellphone has a RSA public key KA and matching private key vA; likewise, Bob's cellphone has KB and vB.

Here is what Alice's cellphone will do to send the text message m:

  1. Alice's phone randomly picks a new AES session key k and computes c = RSA-Encrypt(KB,k), c' = AES-CBC-Encrypt(k,m), and t = RSA-Sign(vA, (c, c')).
  2. Alice's phone sends (c, c', t) to Bob's phone.

And her is what Bob's cellphone will do, upon receiving (c, c', t):

  1. Bob's phone checks that t is a valid RSA signature on (c, c') under public key KA. If not, abort.
  2. Bob's phone computes k' = RSA-Decrypt(vB, c) and m' = AES-CBC-Decrypt(k',c').
  3. Bob's phone informs Bob that Alice send message m'.

The example below would be appropriate if it did a good job of justifying the claim that INSEC uses Java's serialization mechanism in a secure way, and the justification required an understanding of the event loop and message formats.

Message Format and Dispatch Loop

INSEC relies upon Java's serialization mechanism to send messages over a TCP/IP connection. Each message consists of two parts; the message type and the payload. The message type is simply a Java int, while the payload is a Serializable[].

The message dispatch loop is simply a switch statement:

clientUsername = getUsernameForConnection();

while(!socket.EOF()) {
   int messageType = socket.readInt();

   switch(messageType) { 
      case foo:
         Serializable[] payload = socket.readInt();

         //Unpack appropriate arguments from payload
         arg1 = ...;
         arg2 = ...;
         argN = ...;

         foo(clientUserName, arg1, arg2, ..., argN);

      case bar: 

Note that there is one dispatch loop per TCP/IP connection, so the server can associate a username with each dispatch loop. The client dispatch loop is similar.

Software Development Practices

Implementation Strategy

The server-side code is short and therefore auditable. Many of the most common security problems in modern applications are not expressible in Java. Therefore, we do not need to worry about language-based attacks. Java's hashtable implementation is synchronized, saving us from any race conditions that may arise from multiple concurrent connections. Because our program does not attempt to access the server's hard drive, and Java is free from buffer overflow exploits, we can be sure that our server cannot be used as a stepping-stone for an attack against other software running on the server.

Similarly, ensuring client side security is trivial.


Whenever we add a new feature, we use the GUI to confirm that it works properly. Occasionally, we manually run through all of the GUI's functionality, checking to see if it behaves improperly, and watching for exceptions. If we notice erratic behavior, we attempt to reproduce it. We have found that most irreproducible bugs are the result of human error, and are generally confined to the client side code, and therefore do not pose a security threat. Because there is only one client for our IM protocol, we do not test the server on inputs not producible via our GUI.


We have described the system architecture of the insecure IM sample protocol that was developed for CS161. This protocol is available from the Hamsam UI. As you implement your project, feel free to look in hamsam.protocol.insecure in the code skeleton's tarball. Although there are many fundamental, and implementation level flaws in the sample protocol, it is designed to give you some idea of how we expect you to interact with the Hamsam API, and implements all of the features that we expect your protocol to support. Also, it may be of some use in differentiating between bugs in your protocol implementation, and bugs in the Hamsam clients.

Finally, most of the assumptions made throughout this design document are flawed; if you find yourself making them, it may be time to revisit your design, or at least come up with better justifications than those that were presented here. Good luck!