refactor2.md

Thunderbolt Mesh Cluster Setup Guide (Proxmox + FRR + BGP)

---

📁 Table of Contents

• 1. Network Overview
• 2. Interface Overview (All Nodes)
• 3. Stage 1 — Internal Mesh VM Routing Only
  • 3.1 Configure Network Interfaces
  • 3.2 Install FRRouting
  • 3.3 Configure OpenFabric (IS-IS)
  • 3.4 Verify Internal Mesh Connectivity
• 4. Stage 2 — LAN Access via Unifi Router
  • 4.1 Extend FRR for BGP
  • 4.2 Configure Unifi Router BGP
  • 4.3 Validate BGP Route Exchanges
• 5. Conclusion

---

1. Network Overview

• Three Proxmox nodes interconnected via Thunderbolt.
• Routed point-to-point interfaces, no bridging.
• FRRouting (FRR) with OpenFabric (IS-IS) for internal mesh routing.
• BGP to Unifi router for external LAN access.
• All services explicitly advertised via network statements (no redistribute connected).

---

2. Interface Overview (All Nodes)

Node	Interface	Address (CIDR)	MTU	Purpose
pve1	vmbr0	192.168.1.81/24	9182	Management IPv4
pve1	vmbr0	2001:db8:1000:1::81/64	9182	Management IPv6
pve1	en05	(no IP)	65520	Thunderbolt Fabric
pve1	en06	(no IP)	65520	Thunderbolt Fabric
pve1	vmbr100	10.0.81.1/24	65520	VM Routed IPv4
pve1	vmbr100	fc00:81::1/64	65520	VM Routed IPv6
pve1	lo	10.0.0.81/32	virtual	Loopback IPv4
pve1	lo	fc00::81/128	virtual	Loopback IPv6

(pve2 and pve3 follow similarly with .82/.83 addresses)

---

3. Stage 1 — Internal Mesh VM Routing Only

3.1 Configure Network Interfaces

Edit /etc/network/interfaces and /etc/network/interfaces.d/thunderbolt to match required MTUs and static assignments for Thunderbolt and loopbacks.

3.2 Install FRRouting - not needed on promox 8.4+

apt update
apt install frr
systemctl enable frr

3.3 Configure OpenFabric

Example /etc/frr/frr.conf:

frr version 8.5.2
frr defaults datacenter
hostname pve1
log syslog informational
service integrated-vtysh-config

interface en05
 ip router openfabric 1
 ipv6 router openfabric 1
exit

interface en06
 ip router openfabric 1
 ipv6 router openfabric 1
exit

interface lo
 ip router openfabric 1
 ipv6 router openfabric 1
 openfabric passive
exit

interface vmbr100
 ipv6 router openfabric 1
 openfabric passive
exit

router openfabric 1
 net 49.0000.0000.0081.00
 lsp-gen-interval 5
exit

3.4 Verify Internal Mesh Connectivity

vtysh -c "show openfabric topology"

---

4. Stage 2 — LAN Access via Unifi Router

4.1 Extend FRR for BGP

Example /etc/frr/frr.conf extension:

router bgp 65001
 bgp router-id 192.168.1.81
 no bgp ebgp-requires-policy

 neighbor 192.168.1.1 remote-as 65001
 neighbor 192.168.1.1 update-source 192.168.1.81
 neighbor 192.168.1.1 next-hop-self

 neighbor 2001:db8:1000:1::1 remote-as 65001
 neighbor 2001:db8:1000:1::1 update-source 2001:db8:1000:1::81
 neighbor 2001:db8:1000:1::1 next-hop-self

 address-family ipv6 unicast
  network 2001:db8:1000:1::81/128
  network fc00::81/128
  network fc00:81::/64
  neighbor 2001:db8:1000:1::1 activate
  neighbor 2001:db8:1000:1::1 next-hop-self
 exit-address-family

 address-family ipv4 unicast
  network 10.0.0.81/32
  network 10.0.81.0/24
  neighbor 192.168.1.1 activate
  neighbor 192.168.1.1 next-hop-self
 exit-address-family

4.2 Configure Unifi Router BGP

router bgp 65001
 bgp router-id 192.168.1.1
 no bgp ebgp-requires-policy

 neighbor 192.168.1.81 remote-as 65001
 neighbor 192.168.1.81 update-source 192.168.1.1
 neighbor 192.168.1.81 next-hop-self
 (repeat for .82, .83)

 neighbor 2001:db8:1000:1::81 remote-as 65001
 neighbor 2001:db8:1000:1::81 update-source 2001:db8:1000:1::1
 neighbor 2001:db8:1000:1::81 next-hop-self
 (repeat for ::82, ::83)

 address-family ipv6 unicast
  redistribute connected
  redistribute kernel
 exit-address-family

 address-family ipv4 unicast
  redistribute connected
  redistribute kernel
 exit-address-family

4.3 Validate BGP Route Exchanges

vtysh -c "show bgp summary"
vtysh -c "show bgp ipv6 unicast"
vtysh -c "show bgp ipv4 unicast"

Example IPv6 table:

*> 2001:db8:1000:1::81/128    fe80::dead:beef:1
*> 2001:db8:1000:1::82/128    fe80::dead:beef:2
*> 2001:db8:1000:1::83/128    fe80::dead:beef:3
*> fc00:81::/64               fe80::dead:beef:1
*> fc00:82::/64               fe80::dead:beef:2
*> fc00:83::/64               fe80::dead:beef:3

Example IPv4 table:

*>i10.0.0.81/32     192.168.1.81
*>i10.0.0.82/32     192.168.1.82
*>i10.0.0.83/32     192.168.1.83
*>i10.0.81.0/24     192.168.1.81
*>i10.0.82.0/24     192.168.1.82
*>i10.0.83.0/24     192.168.1.83

---

5. Conclusion

• ✅ Internal Thunderbolt Mesh operational
• ✅ External LAN reachability via Unifi router
• ✅ No redistribute connected on Proxmox nodes
• ✅ Safe for production with documentation-compliant IPv6 examples

---

🚀 Project Complete!

Hey Scyto,

Would you know anything about extending the mesh to include other cluster nodes using the additional 2.5GbE expansion slot on the nucs? I was advised to use ISIS in the frr slack about a year a half ago when I first tried to extend this setup. I am using the following frr config for my Nucs:

frr version 10.2.3
frr defaults traditional
hostname worker
log syslog informational
service integrated-vtysh-config
!
interface en05
 ipv6 router isis 1
 isis metric 2
 isis network point-to-point
exit
!
interface en06
 ipv6 router isis 1
 isis metric 2
 isis network point-to-point
exit
!
interface enp87s0
 ipv6 router isis 1
exit
!
interface lo
 ipv6 router isis 1
 isis passive
exit
!
router isis 1
 is-type level-2-only
 net 49.0000.0000.0002.00
exit
!

And this for my copper-only nodes:

frr version 10.2.3
frr defaults traditional
hostname zrouter
log syslog informational
no ip forwarding
service integrated-vtysh-config
!
interface net10g1b
 ipv6 router isis 1
exit
!
interface lo
 ipv6 router isis 1
 isis passive
exit
!
router isis 1
 is-type level-2-only
 net 49.0020.0000.0003.00
exit
!

The idea is that the lower metric for the TB links should win out using ISIS

It works pretty well but occasionally the topology will break down and the nucs will start to talk to each other using their copper interfaces. I am not sure why. Bringing the thunderbolt interfaces down and then up will fix the issue each time, however.

I have an MTU of 1400 across all interfaces on all nodes (I am not sure if this is necessary), and am using all of the latest scripts from the main gist. Just wondering if anything jumps out at you (besides this probably being a bad idea 😆 in the first place). Thanks again for documenting all of your efforts on this setup. It has been very rewarding to follow.