Hypervisor and Storage

Hypervisor

Oxide’s hardware virtual machine solution is built on bhyve, an open source Virtual Machine Monitor (VMM) on illumos. The underlying technologies for the software stack also include:

  • Helios: Oxide’s illumos distribution, as the operating system for the host CPU in server sleds

  • Propolis: Oxide’s homegrown Rust-based userspace

Guest Workload Support

Oxide supports guest images that meet the following criteria:

  • Guest OS: major Linux distros, Windows

  • Boot mode: OS images enabled for UEFI booting

  • Device emulation: x86 images with VirtIO driver support

Network booting via the PXE protocol is also supported.

Note
Nested virtualization (i.e., VM within a VM) is unsupported. Windows versions that require the Trusted Platform Module (TPM), e.g., Windows 11, are also not supported.

Guest Facilities

The guest facilities available include idiomatic remote access systems such as SSH for Linux and Remote Desktop Protocol (RDP) for Windows. Serial console access is also available, allowing direct interaction with instances.

Live migration of virtual machine instances is currently limited to Oxide technicians and will be made available to operators in a future release.

Storage

Physical Layer

Each server sled in the Oxide rack includes SSDs of two different form factors:

  • U.2 devices (10x): store all user data and internal data (e.g., metadata, control plane data, software images).

  • M.2 devices (2x): store a limited amount of internal data (e.g., boot images, memory dump).

The physical disks form a common pool of resources distributed across the rack that backs the virtual block storage service.

Service Layer

The block storage service in the Oxide Rack is known as Crucible which comprises two components:

  • crucible-downstairs: reside with the target disk storage and provide access to it via the network for those upstairs

  • crucible-upstairs: reside with the server sled using the storage, making requests across the network to some number of downstairs replicas

The upstairs component is a virtual block device layer. Each virtual disk volume has 3 associated instances of downstairs to achieve data redundancy.

Crucible downstairs leverages ZFS and its many established facilities such as snapshots, compression, RAID, and encryption to support the level of persistence, integrity, scalability, and security required for data storage on Oxide.

Crucible upstairs and downstairs communicate over a network protocol for both block data and related metadata across all the server sleds within the rack.