Process Sandboxing

A very dearly needed feature in Beelzebub is process sandboxing. The following scheme is planned to be implemented, with features separated in levels.

Important terminology:

• guest- the code running inside the sandbox;
• host- the code running outside of the sandbox, responsible with creating and sustaining the environment in which the guest runs.

Level 1 Sandboxing

In level 1 sandboxes, a thread will simply use a system call to transition from host to guest with a specific guest state. This system call will back up some of the host's own state required to transition out of the guest. Then it will load the guest state onto the current thread and resume execution.
The next system call issued by this thread (now in guest state) will trigger a transition to the host. The guest state will be backed up entirely and the host state that was saved before will be loaded again into the thread.
Thus, the host is free to implement its own system call ABI as needed.

Pros	Cons
Performance overhead manageable (slightly quicker than context switch)	No host memory protection
Host can access guest memory quickly	Address space prone to collisions
No fancy CPU features required
Architecture-agnostic

Level 2 Sandboxing

This level builds on top of level 1, adding memory protection through Intel Protection Keys (PKE).
Transitioning to guest will change the access rights so that the guest can only see its own memory, keeping the host protected.

Pros	Cons
Host memory is protected	Address space prone to collisions
Performance overhead manageable (virtually same as level 1)	Requires hardware support
Host can access guest memory quickly	Hard limits on number of guests managed by a host, if they need separation too
Host and guest can safely share memory

Level 3 Sandboxing

Level 3 sandboxes separate the host and guest processes entirely.
The host process is in charge of constructing the guest process and thread(s).
Guest threads doing system calls will cause context switches to host threads (if they're blocked) and the guest thread will block until the host returns.

Pros	Cons
Host memory is protected	Host accessing guest memory might be slow
Architecture-agnostic	All transitions are a full context switch
No address space collisions	Uses system resources like a full process
Host and guest can safely share memory	Setting up the shared memory will be slower than lower levels
Host can be slim; one host thread could serve system calls for every guest thread

Level 4 Sandboxing

Level 4 sandboxes are virtual machines, and their hypervisor is the host process.

Pros	Cons
Whole machine memory and hardware are protected	Host accessing guest memory will be slow
Wide support in hardware	Transitions will be most expensive
No address space collisions	Uses most system resources
Extensible to full hypervisor	Takes time and effort to implement
Fine grained control over virtually everything