This guide shows how to connect an Oxide rack to an upstream network via the Border Gateway Protocol (BGP). The BGP setup this guide will go over is depicted below.
The instructions are based on the assumption that the rack is already accessible through static networking and the goal is to have BGP run alongside it. The high-level steps for achieving that are:
Create an address lot for holding the interface addresses that will be used for peering
Configure BGP announcements for an address lot used for instance prefixes
Create a BGP configuration
Activate the BGP configuration by assigning addresses from the peering address lot to switch ports
The setup steps will make use of the oxide CLI.
Address Lot Creation
The first part of establishing a peer session with an upstream router is
assigning an address to the switch port over which peering will take place. On
the Oxide platform, addresses cannot be arbitrarily assigned, they must be
allocated from an address lot. The oxide CLI invocation and JSON specification
below set up an address lot for peering.
oxide system networking address-lot create \
--name peering \
--description 'a lot for bgp peering' \
--json-body peering-address-lot.json{
"name": "peering",
"description": "a lot for bgp peering",
"kind": "infra",
"blocks": [
{
"first_address": "192.168.22.1",
"last_address": "192.168.22.6"
}
]
}This configuration creates a single address lot of type infra. This means the
address lot is used for rack infrastructure. The alternative type is pool,
which means the lot is used for allocating instance IP addresses. These types
prevent address lots from being used for unintended purposes. The lot contains 6
addresses in the inclusive range 192.168.22.1 - 192.168.22.6.
Configure Announcements
The next thing to do is configure what network prefixes the rack will announce to upstream BGP peers.
Announce set prefixes are also tied to address lots. Here, we reference an
existing address lot named instances. If you do not already have an
address lot defined for instance prefixes, create one now using the same
process as the previous section before proceeding.
oxide system networking bgp announce-set update \
--name instances \
--description 'the instance block' \
--json-body announce-set.json{
"name": "instances",
"description": "the instance block",
"announcement": [{
"address_lot_block": "instances",
"network": "192.168.20.0/24"
}]
}This creates an object called an announce set, which is a collection of
announcements. The example above shows a single announcement for the prefix
192.168.20.0/24.
Create a BGP Configuration
Next, a BGP configuration must be created. BGP configurations are named objects that carry a BGP autonomous system number (ASN). BGP routers in the same ASN peer use the internal variant of the BGP protocol (iBGP). BGP routers in different ASNs peer use the external variant of the BGP protocol (eBGP). A common deployment configuration is one BGP configuration with a distinct ASN per switch. This ensures that the Oxide rack peers over eBGP with upstream routers.
oxide system networking bgp config create \
--name as47 \
--description 'autonomous system 47' \
--asn 47 \
--bgp-announce-set-id instancesBGP configurations are tied to an announce set. In the example above, the
instances announce set was created in the previous section.
Activating a BGP Configuration
After the BGP configuration is created, the switch ports will need to be configured to use it.
The Oxide network model is organized around switch ports and port settings. Port
settings are reusable configurations that are referenced by switch ports.
To illustrate how to update an existing port settings object with the Oxide
networking CLI, we’ll walk through the steps using port qsfp0 on switch0 in
an Oxide rack as an example. When you invoke the CLI for other switch ports,
you’ll modify your commands accordingly to match the switch and port target.
Before making any changes, let’s query the current switch port settings with
oxide system networking switch-port-settings show
switch0/qsfp0
=============
Autoneg Fec Speed
false None Speed100G
Address Lot VLAN
192.168.20.3/29 initial-infra None
BGP Peer Config Export Import Communities Connect Retry Delay Open Enforce First AS Hold Time Idle Hold Time Keepalive Local Pref Md5 Auth Min TTL MED Remote ASN VLAN
Destination Nexthop Vlan Preference
0.0.0.0/0 192.168.20.1/32 0 0The command above shows us that the port qsfp0 on switch0 is already set up
to use an address lot named initial-infra, configured during initial setup for
static routing.
To update the switch port settings to use BGP side-by-side with static routing, first identify your rack id
export rack=`oxide system hardware rack list | jq -r .[0].id`
echo $rackThen, add the source address the switch will use for peering to the port
oxide system networking addr add \
--rack $rack \
--switch switch0 \
--port qsfp0 \
--addr 192.168.22.3/29 \
--lot peeringNext, define the BGP peer that is expected to be reachable over the port.
The peer configuration references the overarching BGP configuration defined
above via the name as47.
The addr field is the address of the router to peer with.
oxide system networking bgp peer set \
--rack $rack \
--switch switch0 \
--port qsfp0 \
--addr 192.168.22.1 \
--bgp-config as47 \
--hold-time 6 \
--idle-hold-time 6 \
--delay-open 0 \
--connect-retry 3 \
--keepalive 2The example above covers a subset of the commonly used parameters such as timers from the BGP specification. Here is a complete list of the configurable options:
Hold Time | How long to hold peer connections between keepalives (seconds) [default: 6] |
Idle Hold Time | How long to hold a peer in idle before attempting a new session (seconds) [default: 0] |
Delay Open | How long to delay sending an open request after establishing a TCP session (seconds) [default: 0] |
Connect Retry | How long to wait between TCP connection retries (seconds) [default: 0] |
Keepalive | How often to send keepalive requests (seconds) [default: 2] |
Allowed Imports | Prefixes that may be imported from the peer. Empty list means all prefixes allowed |
Allowed Exports | Prefixes that may be exported to the peer. Empty list means all prefixes allowed |
Communities | Include the provided communities in updates sent to the peer |
Enforce First AS | Enforce that the first AS in paths received from this peer is the peer’s AS |
Local Preference | Apply this local preference to routes received from the peer |
Auth String | Use the given authorization string for TCP-MD5 authentication with the peer |
Minimum TTL | Require messages from a peer have a minimum IP time to live field |
Multi-exit Discriminator | Apply the provided multi-exit discriminator (MED) updates sent to the peer |
Remote ASN | Require that a peer has a specified ASN |
VLAN ID | Associate a VLAN ID with a peer |
After adding the address and BGP peer, you should see them in the updated switch port settings
oxide system networking switch-port-settings show
switch0/qsfp0
=============
Autoneg Fec Speed
false None Speed100G
Address Lot VLAN
192.168.20.3/29 initial-infra None
192.168.22.3/29 peering None
BGP Peer Config Export Import Communities Connect Retry Delay Open Enforce First AS Hold Time Idle Hold Time Keepalive Local Pref Md5 Auth Min TTL MED Remote ASN VLAN
192.168.22.1 as47 [no filtering] [no filtering] [] 3 0 false 6 6 2 None None None None None None
Destination Nexthop Vlan Preference
0.0.0.0/0 192.168.20.1/32 0 0Checking BGP Status
Once BGP has been configured on a port, its status is visible. It may take a few seconds for the peer to become established, so initial states might look like this:
oxide system networking bgp status
[
{
"addr": "192.168.22.1",
"local_asn": 47,
"remote_asn": 64601,
"state": "open_sent",
"state_duration_millis": 4573,
"switch": "switch0"
}
]This output indicates the first peer session is in the OpenSent state and has
been there for 4573 milliseconds.
After a few seconds the connection makes it to established.
oxide system networking bgp status
[
{
"addr": "192.168.22.1",
"local_asn": 47,
"remote_asn": 64601,
"state": "established",
"state_duration_millis": 24,
"switch": "switch0"
}
]Here is another way to query the BGP status
oxide system networking bgp show-status
switch0
=======
Peer Address Local ASN Remote ASN Session State State Duration
192.168.22.1 47 64601 Established 0day 0h 0m 0s 24msThe Oxide rack provides an API endpoint for retrieving the last 1024 BGP messages sent and received by each rack switch. Here is an example for a different bgp peer:
oxide system networking bgp history --asn 65002
{
"switch_histories": [
{
"history": {
"172.20.15.51": {
"received": [
{
"message": {
"type": "update",
"value": {
"nlri": [
{
"length": 0,
"value": []
}
],
"path_attributes": [
{
"typ": {
"flags": 64,
"type_code": "origin"
},
"value": {
"origin": "igp"
}
},
{
"typ": {
"flags": 64,
"type_code": "as_path"
},
"value": {
"as4_path": [
{
"typ": "as_sequence",
"value": [
64601
]
}
]
}
},
{
"typ": {
"flags": 64,
"type_code": "next_hop"
},
"value": {
"next_hop": "172.20.15.51"
}
}
],
"withdrawn": []
}
},
"timestamp": "2024-05-09T04:24:02.699067710Z"
},
...
],
"sent": [
{
"message": {
"type": "update",
"value": {
"nlri": [
{
"length": 24,
"value": [
172,
20,
26,
0
]
}
],
"path_attributes": [
{
"typ": {
"flags": 64,
"type_code": "origin"
},
"value": {
"origin": "igp"
}
},
{
"typ": {
"flags": 64,
"type_code": "as_path"
},
"value": {
"as4_path": [
{
"typ": "as_sequence",
"value": [
47
]
}
]
}
},
{
"typ": {
"flags": 64,
"type_code": "next_hop"
},
"value": {
"next_hop": "172.20.15.53"
}
}
],
"withdrawn": []
}
},
"timestamp": "2024-05-09T04:24:02.698739718Z"
},
...
]
}
},
"switch": "switch1"
},
...
]
}Related CLI References
Besides the commands covered in this guide, there are additional ones for managing BGP config, announcements, and related network settings. Here is a quick summary of the relevant CLI commands:
Modify/query BGP announce set and originated routes
Modify/query BGP configuration or peers
Modify/query BGP session filtering criteria
Modify route local preference
Modify switch port addresses
Modify/query address lots