Hybrid networking

Three sites, one VPN hub

Connect on-premise, AWS and Azure as if they were on the same LAN, with open-source SoftEther.

Kevin Keller · kevinkeller.org

Why it matters

Most enterprises end up with networks in three places

  • On-premise, one cloud, then another cloud.
  • The networks are isolated by default.
  • Cloud vendor VPN appliances cost money and lock you in.
  • SoftEther is open source and supports every major VPN protocol.

The architecture

Every site dials into a central hub with a public IP

SiteNetworkConnects via
On-premise192.168.1.0/24VPN client on a gateway
AWS10.1.0.0/16SoftEther client on EC2
Azure10.2.0.0/16SoftEther client on a VM
Hub192.168.30.0/24Ubuntu server, public IP

The idea

The hub forwards Ethernet frames, not subnets

SoftEther bridges at Layer 2, so all sites share one virtual switch. The hub doesn't need to know about individual subnets.

Building the hub

Four pieces turn one server into a router for all sites

  1. SoftEther server with a virtual hub, a TAP bridge and one user per site.
  2. IP forwarding enabled in the kernel.
  3. iptables rules, including TAP-to-TAP forwarding between clients.
  4. dnsmasq for DHCP and DNS on the VPN interface.

The key trick

DHCP pushes routes to every site automatically

  • dnsmasq advertises all three site subnets via the hub at 192.168.30.1.
  • Option 121 for Linux and Mac clients, option 249 for Windows.
  • Optional /etc/hosts entries give cross-site internal hostnames.

Connecting the clouds

Each cloud gateway needs one setting people forget

SiteGatewayMust-have
AWSSmall EC2 (t3.micro)Source/dest check disabled, VPC route table entry
AzureSmall VM (B1s)IP forwarding on the NIC, user-defined route
On-premiseAny Linux gatewayRouter sends cloud subnets to it

The routing flow

An on-premise machine reaching AWS takes a predictable path

  1. The packet hits the on-premise gateway.
  2. It enters the SoftEther tunnel to the hub.
  3. The hub forwards it on tap_soft to the AWS client.
  4. The AWS gateway delivers it inside the VPC.
  5. The response follows the same path back.

Security

A hub that connects everything needs tight controls

  • Per-site users: revoke one site without touching others
  • Management port 5555: trusted IPs only
  • Prefer certificates over passwords and pre-shared keys
  • Protect the config file: chmod 600
  • Audit sessions regularly with SessionList

The takeaway

A single SoftEther hub gives you a fully routed mesh across on-premise, AWS and Azure.

Read the full article

Kevin Keller · kevinkeller.org