This guide describes how to connect an Oxide rack to additional upstream networks through the API. It assumes that the rack is already accessible through a primary network, and the goal is to connect the rack to additional networks while maintaining primary network connectivity.
The example in this guide will walk through taking a rack that is connected to a transit provider in a data center, and expand connectivity to a direct connection with a cloud provider. In this example both forms of connectivity are using BGP. However, the concepts in this guide apply to static networking as well.
Initial Configuration
This guide will use the oxide CLI. To get started we’re going to look at the
current network configuration of the rack.
$ oxide system networking switch-port-settings show
switch1/qsfp0
=============
Autoneg Fec Speed
false None Speed100G
Address Lot VLAN
169.254.20.2/30 initial-infra None
BGP Peer Config Export Import Communities Connect Retry Delay Open Enforce First AS Hold Time Idle Hold Time Keepalive Local Pref Md5 Auth Min TTL MED Remote ASN VLAN
169.254.20.1 as65547 [no filtering] [no filtering] [] 3 3 false 6 0 2 None None None None None None
switch0/qsfp0
=============
Autoneg Fec Speed
false None Speed100G
Address Lot VLAN
169.254.10.2/30 initial-infra None
BGP Peer Config Export Import Communities Connect Retry Delay Open Enforce First AS Hold Time Idle Hold Time Keepalive Local Pref Md5 Auth Min TTL MED Remote ASN VLAN
169.254.10.1 as65547 [no filtering] [no filtering] [] 3 3 false 6 0 2 None None None None None None
This shows that each switch has a port configured with a BGP session. This configuration lines up with the diagram on the left side of the figure above. We can get the status of each BGP session as follows.
$ oxide system networking bgp show-status
switch0
=======
Peer Address Local ASN Remote ASN Session State State Duration
169.254.10.1 65547 64500 Established 15h 12m 56s 859ms
switch1
=======
Peer Address Local ASN Remote ASN Session State State Duration
169.254.20.1 65547 64500 Established 15h 12m 58s 669msExpanding Connectivity
Expanding connectivity in this example involves peering with BGP routers over a VLAN segmented network. To do that, an address lot needs to be created for holding the interface addresses that will be used for peering. Addresses from that lot will be assigned to switch ports. BGP sessions and announcements will be configured. And finally, an IP pool will be created for instances to utilize the address space being announced on the expanded portion of the network.
Address Lot Creation
Configuration of the expanded network starts with defining an address lot. This address lot will contain the range of addresses that will be used for peering with upstream routers.
oxide system networking address-lot create \
--json-body cloud-peering-lot.json{
"name": "cloud-peering",
"description": "a lot for cloud peering",
"kind": "infra",
"blocks": [
{
"first_address": "169.254.30.1",
"last_address": "169.254.40.255"
}
]
}We also create a lot for the address range that will be used later for IP pools announced to the cloud.
oxide system networking address-lot create \
--json-body cloud-pool-lot.json{
"name": "cloud-pool",
"description": "a lot for cloud communications",
"kind": "pool",
"blocks": [
{
"first_address": "203.0.113.1",
"last_address": "203.0.113.254"
}
]
}Address Assignment
Next, addresses are assigned to the switch ports. In this example we are re-using existing links for the connectivity expansion. The new connections are going to be made over a VLAN segment. It’s common when deploying to colocation environments to have network services such as cloud or ISP access provisioned over VLAN segments.
The commands below add the address 169.254.30.2/30 to switch0/qsfp0 on VLAN
300 and the address 169.254.40.2/30 to switch1/qsfp0 on VLAN 400.
To update addresses, the target rack needs to be identified. For single rack deployments this can be done as follows.
export rack=`oxide system hardware rack list | jq -r .[0].id`
echo $rackFor multi-rack deployments, run the oxide system hardware rack list command
and choose the appropriate rack UUID.
The following commands add the addresses we’ll need on the switch ports to peer with upstream cloud routers.
oxide system networking addr add \
--rack $rack \
--switch switch0 \
--port qsfp0 \
--addr 169.254.30.2/30 \
--lot cloud-peering \
--vlan 300oxide system networking addr add \
--rack $rack \
--switch switch1 \
--port qsfp0 \
--addr 169.254.40.2/30 \
--lot cloud-peering \
--vlan 400BGP Configuration
Next, BGP sessions are created to peer with routers in the new upstream network. Note that VLANs are also configured for the BGP sessions. This ensures that when the Oxide rack BGP daemons receive announcements from peers on these sessions, the routes that are added will have the corresponding VLAN association. This means that traffic egressing the rack for prefixes learned from these peers will have the specified VLAN tag.
oxide system networking bgp peer set \
--rack $rack \
--switch switch0 \
--port qsfp0 \
--addr 169.254.30.1 \
--bgp-config as65547 \
--vlan-id 300oxide system networking bgp peer set \
--rack $rack \
--switch switch1 \
--port qsfp0 \
--addr 169.254.40.1 \
--bgp-config as65547 \
--vlan-id 400The new sessions should eventually reach the Established state as observed
through the CLI.
oxide system networking bgp show-status
switch0
=======
Peer Address Local ASN Remote ASN Session State State Duration
169.254.10.1 65547 64500 Established 20s 83ms
169.254.30.1 65547 64502 Established 10s 811ms
switch1
=======
Peer Address Local ASN Remote ASN Session State State Duration
169.254.20.1 65547 64500 Established 12s 831ms
169.254.40.1 65547 64502 Established 3s 939msControlling BGP Announcements
In some situations, it’s necessary to limit what announcements go to what peers.
For the Oxide rack, this is configurable through the CLI. Consider the case where
the intent is to only export 198.51.100.0/24 to transit providers and
203.0.113.0/24 to cloud providers. For the example topology in this guide,
this filtering is accomplished as follows.
oxide system networking bgp filter \
--rack $rack \
--switch switch0 \
--port qsfp0 \
--peer 169.254.10.1 \
--direction 'export' \
--allowed 198.51.100.0/24
oxide system networking bgp filter \
--rack $rack \
--switch switch1 \
--port qsfp0 \
--peer 169.254.20.1 \
--direction 'export' \
--allowed 198.51.100.0/24
oxide system networking bgp filter \
--rack $rack \
--switch switch0 \
--port qsfp0 \
--peer 169.254.30.1 \
--direction 'export' \
--allowed 203.0.113.0/24
oxide system networking bgp filter \
--rack $rack \
--switch switch1 \
--port qsfp0 \
--peer 169.254.40.1 \
--direction 'export' \
--allowed 203.0.113.0/24--allowed flag may be specified multiple times to allow multiple
prefixes per-peer. The set of prefixes defined by the --allowed flag describe
the entire set of allowed prefixes for the peer. Any pre-existing allowed
prefixes that are omitted from the list will be removed. Prefixes are exact
matches and do not follow subnet inclusion rules. Thus, allowing 10.10.0.0/16
does not imply allowing 10.10.0.0/24.The new live configuration can be seen through the CLI.
oxide system networking switch-port-settings show
switch0/qsfp0
=============
Autoneg Fec Speed
false None Speed100G
Address Lot VLAN
169.254.10.2/30 initial-infra None
169.254.30.2/30 cloud-peering Some(300)
BGP Peer Config Export Import Communities Connect Retry Delay Open Enforce First AS Hold Time Idle Hold Time Keepalive Local Pref Md5 Auth Min TTL MED Remote ASN VLAN
169.254.10.1 as65547 [198.51.100.0/24] [no filtering] [] 3 3 false 6 3 2 None None None None None None
169.254.30.1 as65547 [203.0.113.0/24] [no filtering] [] 0 0 false 6 0 2 None None None None None Some(300)
switch1/qsfp0
=============
Autoneg Fec Speed
false None Speed100G
Address Lot VLAN
169.254.20.2/30 initial-infra None
169.254.40.2/30 cloud-peering Some(400)
BGP Peer Config Export Import Communities Connect Retry Delay Open Enforce First AS Hold Time Idle Hold Time Keepalive Local Pref Md5 Auth Min TTL MED Remote ASN VLAN
169.254.20.1 as65547 [198.51.100.0/24] [no filtering] [] 3 3 false 6 3 2 None None None None None None
169.254.40.1 as65547 [203.0.113.0/24] [no filtering] [] 0 0 false 6 0 2 None None None None None Some(400)
The prefixes that have been imported on a peer-by-beer basis can also be viewed through the CLI.
oxide system networking bgp imported ipv4 --asn 65547
[
{
"id": 3232235963,
"nexthop": "169.254.20.1",
"prefix": "0.0.0.0/0",
"switch": "switch1"
},
{
"id": 3232235882,
"nexthop": "169.254.40.1",
"prefix": "240.0.0.0/4",
"switch": "switch1"
},
{
"id": 3232235801,
"nexthop": "169.254.30.1",
"prefix": "240.0.0.0/4",
"switch": "switch0"
},
{
"id": 3232235887,
"nexthop": "169.254.10.1",
"prefix": "0.0.0.0/0",
"switch": "switch0"
}
]Here we can see that the transit providers are announcing a default route to the
rack, and the cloud providers are announcing the class-E prefix 240.0.0.0/4.
To round out the guide, the prefix 203.0.113.0/24 is announced to the cloud
providers.
oxide system networking bgp announce \
--announce-set as65547-announce \
--address-lot cloud-pool \
--prefix 203.0.113.0/24Then an IP pool is created for instances to be created that draw addresses from the pool being announced to the cloud.
cloud_pool_name='cloud-pool'
oxide ip-pool create \
--name $cloud_pool_name \
--description 'ip pool for talking to the cloud'
oxide ip-pool range add \
--pool $cloud_pool_name \
--first 203.0.113.1 \
--last 203.0.113.254