Hypervisor and Storage

Hypervisor

Oxide’s hardware virtual machine solution is built on bhyve, an open source Virtual Machine Monitor (VMM) on illumos. The underlying technologies for the software stack also include:

  • Helios: Oxide’s illumos distribution, as the operating system for the host CPU in server sleds

  • Propolis: Oxide’s homegrown Rust-based userspace

Guest Workload Support

The initial version of Oxide Rack supports guest images that meet the following criteria:

  • Guest OS: Major Linux distros such as Ubuntu, Debian, CentOS, and OmniOS (Windows guests are expected to be fully functional in the near future)

  • Boot mode: OS images enabled for UEFI booting

  • Device emulation: x86 images with VirtIO driver support

Note
Nested virtualization (i.e., VM within a VM) is unsupported at this time.

Guest Facilities

The guest facilities available on the Oxide Rack include idiomatic remote access systems such as SSH for Linux and Remote Desktop Protocol (RDP) for Windows. Out-of-band VM console access is also supported to allow creating virtual machine templates from scratch using ISO files, or troubleshooting guest initialization issues.

Live migration of virtual machine instances is currently limited to Oxide technicians and will be made available to operators in a future patch release.

Storage

Physical Layer

Each server sled in the Oxide rack includes SSDs of two different form factors:

  • U.2 devices (10x): store all user data and internal data (e.g., metadata, control plane data, software images).

  • M.2 devices (2x): store a limited amount of internal data (e.g., boot images, memory dump).

The physical disks form a common pool of resources distributed across the rack that backs the virtual block storage service.

Service Layer

The block storage service in the Oxide Rack is known as Crucible which comprises two components:

  • crucible-downstairs: reside with the target disk storage and provide access to it via the network for those upstairs

  • crucible-upstairs: reside with the server sled using the storage, making requests across the network to some number of downstairs replicas

The upstairs component is a virtual block device layer. Each virtual disk volume has 3 associated instances of downstairs to achieve data redundancy.

Crucible downstairs leverages ZFS and its many established facilities such as snapshots, compression, RAID, and encryption to support the level of persistence, integrity, scalability, and security required for data storage in the Oxide rack.

Crucible upstairs and downstairs communicate over a network protocol for both block data and related metadata across all the server sleds within the rack.