"Doors" in SolarisTM: Lightweight RPC using File Descriptors

Jim Voll
Senior Staff Engineer
Solaris Products Group
This new lightweight RPC mechanism, adapted from Sun's Spring O/S project, will be inforporated into a future release of the SolarisTM operating environment. Developers can refer to Solaris documentation for additional library and SPI information.

An Overview of Doors

A door is a "file" descriptor used to describe a procedure in a process and optionally some additional state associated with the procedure. Doors were initially designed as Spring object descriptors used to encapsulate the state and capabilities of C++ objects. A process which obtains a door is free to pass it, along with its capabilities, to other processes in the system.

A server normally creates a door for some service it plans on providing, and exports it to clients. Clients who obtain the door may then invoke the service associated with the door using the synchronous RPC semantics of a door call operation.

Conceptually, during a door invocation the client thread that issues the door procedure call migrates to the server process associated with the door, and starts executing the procedure while in the address space of the server. When the service procedure is finished, a door return operation is performed and the thread migrates back to the client's address space with the results, if any, from the procedure call.

A client may pass data, including other doors, as arguments during a door invocation, and the server procedure may return data, including other doors, as results from an invocation.

Normally each door invocation from a client results in an active thread executing in the server (a server may choose to limit the number of threads available for incoming door invocations). The server must therefore be a MT-safe application. The doors interface provides a convenient mechanism for enabling high performance multithreaded programs since the creation and dispatching of threads is handled by the interface based on load.

Doors as File Descriptors

Using file descriptors to encapsulate doors allows many existing UNIX(R) paradigms to be easily adopted for use with doors. Inventing a new name space for doors was widely viewed as poor design alternative (e.g SVR4 IPC) and would have resulted in a duplication of functionality already provided in UNIX.

File descriptors also provide a secure mechanism to encapsulate a door since the state associated with the door (procedure, process) is kept in the kernel. A user cannot construct a fake file descriptor to use as a door.

Some of the other important capabilities of file descriptors and how they related to doors include the naming facility available with fattach(3c).

Naming

The namefs facility in SVR4 allows a process to bind a STREAMS-based file descriptor to an object in filesystem name space using fattach(3). A logical extension to this facility allows a door to be named in a similar manner (see Figure 1).

The protection mode of the file associated with a named door does not provide strict access protection since a client may obtain access to the door through means other than open(). (Example: A client is free to pass a door to another process with weaker credentials.) This is also true of STREAMS which are named via fattach() (see Figure 1).


FIGURE 1. Named Doors

Doors Implementation

The doors interface is exported from a user level shared library. The library also controls the default server threads that are created in response to incoming door requests on the server.

A new synchronization object, called a shuttle, was engineered to encapsulate the state associated with a door invocation. A shuttle provides the necessary state for process control such as signals and procfs operations, while allowing a direct scheduler hand-off operation between two threads.

Rather than placing a client thread on a sleep queue, signaling the server and asking the scheduler to pick another thread to run, a shuttle marks the current thread as sleeping, marks the server thread as running, and passes control directly to the server thread.

Server Threads

Server threads are normally created on demand by the doors library. These server threads are created using the Solaris threads library with bound threads (each server thread runs on its own lightweight process). The first server thread is created automatically when the server issues a door create call. The server thread will inherit the scheduling class, and signal disposition of the thread issuing the door create. Door server threads use the Solaris thread library's default stack size. Once created, a server thread will place itself in a pool of available threads and wait for a door invocation. New server threads are created on demand when the available pool is depleted.

Benchmark Results

A simple benchmark was used to compare doors in Solaris 2.5 to other existing UNIX IPC mechanisms. The benchmark measures the round trip time associated with transferring control from a client thread in one process, to a server thread in another, and then back again. In the case of pipes and SVR4 messages, a single byte of data is passed.

TABLE 1. SPARCstation 10TM (dual processor, 40 MHz SuperSPARCTM processors)

IPC Mechanism
 usecs
Doors
66
SVR4 Semaphores
142
Pipes
175
SVR4 Messages
270
ONC-RPC
1020

Door Uses

A name service cache will be implemented in future Solaris versions, similar in nature to the DCE CDS clerk. A daemon process runs on the client machine and caches information from the configured name service such as files, NIS, or NIS+. The name service request interfaces, such as getpwnam(), communicate with the local name service cache daemon using a door named fattach(). Although most of the performance advantage of the name service cache is due to avoiding the setup and tear down cost associated with communicating with the name server (the connection to the name server is kept open by the daemon), doors do provide the fastest RPC mechanism for moving the results from daemon to the requestor. The automatic multithreaded dispatching provided by the doors interface makes a high performance implementation straightforward.

Future Work at SunSoft

To date, most of the performance work on Solaris doors concerns transferring control from one address space to another. Future work will optimize the transfer of data through such techniques as copy-on-write transfers. Plans are also underway to use doors as a general upcall mechanism in the kernel, enabling efficient implementations of work such as scheduler activations and user level file systems based on the vnode interface.

Initially developed as Spring object descriptors, doors are being used as a part of a future high performance CORBA compliant ORB implementation. As part of this work, a remote proxy service is being investigated to allow the doors programming model to extend over the network. Doors can also be used as a local transport mechanism for ONC-RPC.

For more information, see:
http://www.sun.com/tech/projects/spring/.