Skip to content

Network Architecture

Physical Infrastructure

Internet Connectivity

  • Service: 1Gb/s synchronous fiber
  • Router: Ubiquiti EdgeRouter-X (central routing)

Switching Infrastructure

Internet (1Gb/s fiber)
│
├── EdgeRouter-X (Central Router)
    │
    ├── Cisco Catalyst WS-C3850-12X48U (SW-AGGR-01 - Primary aggregation switch)
        │
        ├── Cisco Catalyst WS-C3850-12X48U (SW-RACK-01 - Server rack switch)
        │   │   40Gbps uplink (pending installation)
        │   │
        │   ├── emerald (R720XD)
        │   ├── fuji (R720XD)
        │   ├── apollo (R720XD + MD1200)
        │   └── bishop/castle/domino (R630s)
        │
        └── Multiple Cisco SG-300 series (House access switches)

Wireless Infrastructure

  • Access Points: 3x Cisco Catalyst 9120AXI
  • Standard: Wi-Fi 6 (802.11ax)
  • Controller: Mobility Express firmware (controller-less)
  • Coverage: Whole house wireless coverage
  • Upgraded: November 2025 (replaced Aironet 3802i)

VLAN Configuration

Network Segmentation Strategy

All networks use 172.16.x.0/24 addressing scheme:

VLAN 0 - Management Network

  • Subnet: 172.16.0.0/24
  • Purpose: Infrastructure management
  • Devices:
    • All network switches
    • Server iDRAC/IPMI ports
    • Network equipment management interfaces

VLAN 90 - Server Network

  • Subnet: 172.16.90.0/24
  • Purpose: Server management and host OS
  • Devices:
    • Proxmox web interfaces
    • unRAID management (apollo)
    • Server host OS interfaces
    • Administrative access to hypervisors

VLAN 100 - User Network

  • Subnet: 172.16.100.0/24
  • Purpose: Primary user devices
  • Devices:
    • Family computers and laptops
    • Phones and tablets
    • Trusted personal devices

VLAN 101 - Guest Network

  • Subnet: 172.16.101.0/24
  • Purpose: Guest device isolation
  • Status: Currently unused (guest WiFi configuration pending)
  • Planned use: Visitor device access with limited connectivity

VLAN 102 - IoT Network

  • Subnet: 172.16.102.0/24
  • Purpose: Internet of Things device isolation
  • Devices:
    • Apple TVs
    • Home automation devices
    • Amazon Echo devices
    • Smart home sensors and controllers
  • Security: Isolated from main user network

VLAN 103 - Lab Network

  • Subnet: 172.16.103.0/24
  • Purpose: Virtual machine and lab workloads
  • Devices:
    • All Proxmox VMs
    • Kubernetes cluster nodes (primary network interface)
    • Development and testing workloads
    • Lab experiments and projects

VLAN 104 - Data-Sync/Storage Network

  • Subnet: 172.16.104.0/24
  • Purpose: Dedicated 10GbE storage traffic
  • Status: Deployed and operational (December 2025)
  • MTU: 9000 (jumbo frames enabled)
  • Devices:
    • Apollo: 172.16.104.30 (Dell Y40PH 10GbE NIC)
    • Emerald: 172.16.104.34 (Built-in Broadcom BCM57810 10GbE)
    • Fuji: 172.16.104.35 (Dell Y40PH + SFP-10G-T-X transceiver)
    • Production cluster VMs (secondary network interface eth1)
  • Switch: SW-RACK-01 (Cisco 3850) ports Te1/0/37-42
  • Performance: Sub-millisecond latency, 10Gbps link speed
  • Use cases:
    • Ceph distributed storage cluster traffic
    • High-bandwidth VM storage operations
    • NFS storage traffic from apollo
    • Database replication and backups

VLAN 110 - LoadBalancer Network

  • Subnet: 172.16.110.0/24
  • Purpose: MetalLB IP pool for Kubernetes services
  • BGP peering: Advertised to EdgeRouter-X via BGP
  • Services:
    • Traefik ingress controller: 172.16.110.0
    • User-facing Kubernetes services

Kubernetes Network Integration

Cluster Networking Strategy

Management Access

  • Proxmox hosts: VLAN 90 (172.16.90.x)
  • Kubernetes API: Accessible from VLAN 90 and 100
  • Administrative access: VLAN 90 for management operations

VM Network Placement

Minimal Cluster VMs (VLAN 103 + VLAN 104):
├── minimal-cp-01: 172.16.103.10 (eth0) / 172.16.104.10 (eth1)
├── minimal-wk-01: 172.16.103.11 (eth0) / 172.16.104.11 (eth1)
└── minimal-wk-02: 172.16.103.12 (eth0) / 172.16.104.12 (eth1)

Production Cluster VMs (planned - VLAN 103 + VLAN 104):
├── prod-cp-01: 172.16.103.110 (eth0) / 172.16.104.110 (eth1)
├── prod-cp-02: 172.16.103.111 (eth0) / 172.16.104.111 (eth1)
├── prod-cp-03: 172.16.103.112 (eth0) / 172.16.104.112 (eth1)
├── prod-worker-01: 172.16.103.120 (eth0) / 172.16.104.120 (eth1)
├── prod-worker-02: 172.16.103.121 (eth0) / 172.16.104.121 (eth1)
├── prod-worker-03: 172.16.103.122 (eth0) / 172.16.104.122 (eth1)
├── prod-worker-04: 172.16.103.123 (eth0) / 172.16.104.123 (eth1)
├── prod-storage-01: 172.16.103.170 (eth0) / 172.16.104.170 (eth1)
└── prod-storage-02: 172.16.103.171 (eth0) / 172.16.104.171 (eth1)

Note: All VMs use dual network interfaces:

  • eth0 (VLAN 103): Kubernetes API, pod communication, service traffic
  • eth1 (VLAN 104): Ceph storage traffic, high-bandwidth I/O

Service Exposure Strategy

  • Internal services: Stay within VLAN 103
  • User-facing services: Expose to VLAN 110 via load balancer
  • Management interfaces: Accessible from VLAN 90

Load Balancing and Ingress

  • MetalLB: Kubernetes load balancer for service exposure
  • IP pools: Carved from VLAN 103 subnet
  • Ingress controller: Nginx or Traefik for HTTP/HTTPS routing

Security Considerations

Network Isolation

  • Management isolation: VLAN 0 separated from user traffic
  • IoT containment: VLAN 102 prevents lateral movement
  • Guest isolation: VLAN 101 limited connectivity when enabled
  • Lab sandboxing: VLAN 103 for experimental workloads

Firewall Rules (EdgeRouter-X)

VLAN 100 → VLAN 103: Allow HTTP/HTTPS to services
VLAN 90 → All: Management access
VLAN 0 → All: Infrastructure management
VLAN 102 → Internet: Allow, block inter-VLAN
VLAN 103 → Internet: Allow outbound

Access Control

  • Administrative access: Requires VLAN 90 connectivity
  • User services: Exposed selectively to VLAN 100
  • Inter-VLAN routing: Controlled by EdgeRouter-X policies

Storage Network Architecture

Deployed: 10GbE Storage Network (VLAN 104)

Status: Operational since December 2025

Physical Infrastructure

  • Network: VLAN 104 (172.16.104.0/24)
  • Switch: SW-RACK-01 (Cisco Catalyst 3850)
  • MTU: 9000 (jumbo frames)
  • Link speed: 10Gbps
  • Topology: Layer 2 switched network

Connected Servers (3 of 6)

Server Interface IP Address Switch Port Hardware
Apollo eth4 172.16.104.30 Te1/0/37 Dell Y40PH (Broadcom bnx2x)
Emerald enp68s0f0/vmbr2 172.16.104.34 Te1/0/41 Built-in Broadcom BCM57810 10GbE
Fuji enp68s0f0/vmbr2 172.16.104.35 Te1/0/42 Dell Y40PH + SFP-10G-T-X transceiver

Pre-configured Ports (Ready for expansion)

  • Te1/0/38: bishop-data-sync-10g (VLAN 104, MTU 9000)
  • Te1/0/39: castle-data-sync-10g (VLAN 104, MTU 9000)
  • Te1/0/40: domino-data-sync-10g (VLAN 104, MTU 9000)

Performance Validation

  • Link speed: 10Gbps confirmed on all interfaces
  • Jumbo frames: 8972-byte ICMP payload working
  • Latency: Sub-millisecond (~0.3-0.4ms between servers)
  • Persistence: All configurations survive reboots

Use Cases

  • Ceph distributed storage: High-bandwidth replication between OSDs
  • VM storage operations: Fast block device I/O
  • NFS traffic: High-performance NFS from apollo
  • Database replication: Low-latency, high-bandwidth DB clustering
  • Backup operations: Rapid large data transfers
  • Container registry: Fast image distribution

Dual-Network VM Architecture

Kubernetes cluster VMs use two network interfaces:

  • eth0 (VLAN 103): Control plane, API, pod networking
  • eth1 (VLAN 104): Storage traffic (Ceph, NFS, backups)

Benefits:

  • Isolates storage traffic from cluster management
  • Prevents storage operations from saturating cluster network
  • Enables 10x bandwidth increase for storage workloads
  • Simplifies network troubleshooting and monitoring

Future Expansion

Phase 1: Bishop, Castle, Domino (Planned)

  • Connect remaining 3 servers to 10GbE network
  • Switch ports already configured and ready
  • Requires 10GbE NICs or SFP+ transceivers
  • Hardware: C3850-NM-2-40G modules (installed)
  • Cable: QSFP+ DAC between SW-AGGR-01 and SW-RACK-01
  • Benefit: 40Gbps aggregate bandwidth for 6x 10GbE servers

Monitoring and Management

Network Monitoring

  • SNMP: Available on all Cisco equipment
  • Bandwidth monitoring: EdgeRouter-X and switch statistics
  • Connectivity monitoring: Ping and service checks

Integration with Kubernetes

  • Prometheus: Monitor cluster network performance
  • Grafana: Dashboard for network metrics
  • Alert manager: Network connectivity alerts

Hybrid Cloud Networking

Oracle Cloud Integration

  • VPN connectivity: Site-to-site VPN for hybrid workloads
  • Network segmentation: Separate subnets for cloud workloads
  • Security: Encrypted tunnels for sensitive data transfer

DNS and Service Discovery

  • Internal DNS: Resolve internal services
  • External DNS: Public service exposure
  • Service mesh: Future consideration for microservices

Performance Characteristics

Bandwidth Allocation

  • Internet: 1Gb/s shared across all VLANs
  • VLAN 103 (Cluster): 1Gb/s per VM
  • VLAN 104 (Storage): 10Gb/s per server/VM
  • Switching capacity: 176Gb/s (Cisco Catalyst 3850)
  • Inter-switch capacity: 40Gb/s QSFP+ uplink (pending installation)

Latency Expectations

  • Same VLAN: <1ms
  • Inter-VLAN: <2ms (routing overhead)
  • Internet: 10-20ms (fiber connection)

This network architecture provides robust segmentation, security, and scalability for the hybrid Kubernetes infrastructure while maintaining operational simplicity