UI Drafter's FreeBSD Jails Network Setup (vnet)

Added 6 months ago
Summary: UI Drafter runs on two FreeBSD servers, each one has three jails. The reverse proxy jails listen on the internet, while the applications and databases communicate over a private network (dotted lines).
Sentiment: negative
4 minute read
88 people have read this
Stocks: $YI$NET

UI Drafter runs on two FreeBSD servers, each one has three jails. The reverse proxy jails listen on the internet, while the applications and databases communicate over a private network (dotted lines).

Server A Server B Rev. Proxy App Server Database Rev. Proxy App Server Database

Each server has:

  • Two "virtual switches" (if_bridge)
    • ibridge: for the private traffic between servers.
    • xbridge: for passing incoming traffic to the reverse proxy jail, and outgoing traffic from all the jails.
  • An internal interface, inic, connected to ibridge.
  • An external interface, xnic, not directly connected to xbridge, but redirected through it.
  • Seven virtual cables with a VNIC at either end (epair)
    • Two jail-to-jail (blue links)
    • Five jail-to-bridge

xnic 107.155.81.155 107.155.81.165 xnic Server A Server B 192.168.56.110 inic 192.168.56.111 inic xbridge 10.0.2.1/24 ibridge ibridge nginx_j node_j pg_j .2.20 ngx_b .2.30 node_b .2.40 pg_b .3.20 ngx_node_a .3.30 ngx_node_b .4.30 node_pg_a .4.40 node_pg_b 192.168.56.30 inode_b 192.168.56.40 ipg_b 192.168.56.31 inode_b 192.168.56.41 ipg_b


Also, each server has four tunnels (spiped). The arrows point to the pipe's server-end.

spiped tunnels .56.31 .56.41 .56.30 .56.40 :5432 :5433 :5434 :5435 :5432 :5433 :5434 :5435


Configurations

The configuration files (*.conf) are exactly the same in both servers. For that, they read server-specific settings from other files.

The company-specific settings will be highlighted in green.


/etc/rc.conf

This rc.conf specifies the bridges and epairs to create at the hosts' boot time, their names, and how to connect the VNICs to the bridges.

/etc/inic has 192.168.56.110/24 in Server A.

About rc.conf's ifconfig syntax…

The following two lines rename a bridge. The first one shows how it's typed in the shell, which is non-persistent across reboots. While the second one shows how it's written in the rc.conf (persistent).

ifconfig bridge0 name xbridge
ifconfig_bridge0_name=xbridge

ifconfig_em0_name=xnic
ifconfig_em1_name=inic
ifconfig_xnic="`/bin/cat /etc/xnic`"
ifconfig_inic="`/bin/cat /etc/inic` vlanhwtag 2222"

defaultrouter=`/bin/cat /etc/gateway`


cloned_interfaces="
bridge0
bridge1
epair0
epair1
epair2
epair3
epair4
epair5
epair6
"
ifconfig_bridge0_name=xbridge ifconfig_bridge1_name=ibridge ifconfig_epair0a_name=ngx_a ifconfig_epair0b_name=ngx_b ifconfig_epair1a_name=node_a ifconfig_epair1b_name=node_b ifconfig_epair2a_name=pg_a ifconfig_epair2b_name=pg_b ifconfig_epair3a_name=ngx_node_a ifconfig_epair3b_name=ngx_node_b ifconfig_epair4a_name=node_pg_a ifconfig_epair4b_name=node_pg_b ifconfig_epair5a_name=inode_a ifconfig_epair5b_name=inode_b ifconfig_epair6a_name=ipg_a ifconfig_epair6b_name=ipg_b ifconfig_ngx_a=up ifconfig_node_a=up ifconfig_pg_a=up ifconfig_ngx_node_a=up ifconfig_node_pg_a=up ifconfig_inode_a=up ifconfig_ipg_a=up ifconfig_ibridge="
addm inic
addm inode_a
addm ipg_a
"
ifconfig_xbridge="
10.0.2.1/24
addm ngx_a
addm node_a
addm pg_a
"
jail_enable=YES jail_reverse_stop=YES gateway_enable=YES pf_enable=YES

/etc/jail.conf

The jails boot in the order they appear in this file. In addition, it specifies which VNICs the host will hand-over to the jails.

As PostgreSQL requires System V's shared memory, sysvshm provides and namespaces it to the database jail.

path = "/jails/$name";
host.hostname = "$name.uidrafter.lan";

devfs_ruleset = 4;
mount.devfs;
vnet;

exec.clean;
exec.start = "/bin/sh /etc/rc";
exec.stop  = "/bin/sh /etc/rc.shutdown";

pg_j {
  sysvshm = "new";
  vnet.interface = pg_b, node_pg_b, ipg_b;
}

node_j {
  vnet.interface = node_b, ngx_node_b, inode_b, node_pg_a;
}

nginx_j {
  vnet.interface = ngx_b, ngx_node_a;
}

Jails rc.conf's

This section has the jail-boot time configurations. They specify the VNICs' CIDR, gateway, and tunnels.

/jails/nginx_j/etc/rc.conf

ifconfig_ngx_b=10.0.2.20/24
ifconfig_ngx_node_a=10.0.3.20/24

defaultrouter=10.0.2.1

/jails/node_j/etc/rc.conf

/jails/node_j/etc/ip_inode_b has 192.168.56.30 in Server A.

ifconfig_node_b=10.0.2.30/24
ifconfig_ngx_node_b=10.0.3.30/24
ifconfig_inode_b=`/bin/cat /etc/ip_inode_b`/24
ifconfig_node_pg_a=10.0.4.30/24

defaultrouter=10.0.2.1


spiped_enable=YES
spiped_pipes=N2P

spiped_pipe_N2P_mode=encrypt
spiped_pipe_N2P_source="[`/bin/cat /etc/ip_inode_b`]:5432"
spiped_pipe_N2P_target="[`/bin/cat /etc/ip_peer_ipg_b`]:5433"
spiped_pipe_N2P_key=/etc/spiped.key

/jails/pg_j/etc/rc.conf

ifconfig_node_pg_b=10.0.4.40/24
ifconfig_pg_b=10.0.2.40/24
ifconfig_ipg_b=`/bin/cat /etc/ip_ipg_b`/24

defaultrouter=10.0.2.1


spiped_enable=YES
spiped_pipes="N2P PGS PGC"

spiped_pipe_N2P_mode=decrypt
spiped_pipe_N2P_source="[`/bin/cat /etc/ip_ipg_b`]:5433"
spiped_pipe_N2P_target="[`/bin/cat /etc/ip_ipg_b`]:5432"
spiped_pipe_N2P_key=/etc/spiped.key

spiped_pipe_PGS_mode=decrypt
spiped_pipe_PGS_source="[`/bin/cat /etc/ip_ipg_b`]:5434"
spiped_pipe_PGS_target="[`/bin/cat /etc/ip_ipg_b`]:5432"
spiped_pipe_PGS_key=/etc/spiped.key

spiped_pipe_PGC_mode=encrypt
spiped_pipe_PGC_source="[`/bin/cat /etc/ip_ipg_b`]:5435"
spiped_pipe_PGC_target="[`/bin/cat /etc/ip_peer_ipg_b`]:5434"
spiped_pipe_PGC_key=/etc/spiped.key

Firewall (pf)

This firewall configuration denies all traffic by default, and then allows what's needed. It assumes xnic is on a public IP. If not, remove its subnet from the <martians> table.

Incoming traffic

Only the Nginx jail is open to the internet. SSHing to the host or jails is only possible from IPs listed in the <xpeers> table.

Rate Limits…

If an IP connects 100 times within 10 seconds it'll be blocked. Not only from making new connections, but also from using established ones because flush global terminates them.

Effectively, that IP gets added to the <ratelimit> table. Therefore, you can use a cron to reallow those IPs. For example, to remove IPs that are at least three minutes old from that table:

/sbin/pfctl -t ratelimit -T expire 180

How to exclude a company from rate-limits? Create a table with their IPs. Then add a pass in quick rule before the block in quick one. For instance, if Cloudflare® is intercepting traffic, populate a <bypass> table with their IPs and:

pass in quick on xnic inet proto tcp from <bypass> to port 443
block in quick …

Outgoing traffic

The Node jail can reach Stripe® API and Fastmail®. As they need DNS services, Cloudflare® and Google® resolvers are allowed too.

NTP is allowed via inic to private NTP servers.

How to deploy to these servers? Copy over to them with rsync. The orchestration server IP must be in the <xpeers> table.

How to generate Let's Encrypt certificates? Like deploying. See my previous post: Isolated TLS Certificate Creation.

How to extract backups? rsync from the backup/orchestration server too.

The general answer to those questions is: initiate the connection from the orchestration server.


/etc/pf.conf

jailnet = "10.0.2/24"
nginx_j = "10.0.2.20"
node_j  = "10.0.2.30"
pg_j    = "10.0.2.40"

port_ssh_host    = "2200"
port_ssh_nginx_j = "2220"
port_ssh_node_j  = "2230"
port_ssh_pg_j    = "2240"

port_nginx = "{ 443 80 }"
port_email = "465"

resolvers = "{ 1.1.1.1 8.8.8.8 }"
priv_ntp = "{ 192.0.2.10 192.0.2.11 }"

table <ratelimit>
table <blocklist> file "/etc/ips_blocklist"  
table <martians>  file "/etc/ips_martians"   
table <payments>  file "/etc/ips_stripe"     
table <email>     file "/etc/ips_fastmail"   
table <xpeers>    file "/etc/ips_xnic_peers"
table <ipg_j>     file "/etc/ip_ipg_b"
table <inode_j>   file "/etc/ip_inode_b"
table <pgpeer>    file "/etc/ip_peer_ipg_b"
table <nodepeer>  file "/etc/ip_peer_inode_b"


scrub in all fragment reassemble no-df


nat on xnic inet from $jailnet -> (xnic:0)
rdr on xnic inet proto tcp from any      to port $port_nginx       -> $nginx_j
rdr on xnic inet proto tcp from <xpeers> to port $port_ssh_nginx_j -> $nginx_j
rdr on xnic inet proto tcp from <xpeers> to port $port_ssh_node_j  -> $node_j
rdr on xnic inet proto tcp from <xpeers> to port $port_ssh_pg_j    -> $pg_j


antispoof quick for xnic
block in quick on xnic \
      from { <blocklist> <ratelimit> <martians> no-route urpf-failed }
block all


pass in quick on xnic inet proto tcp from any \
     to port $port_nginx keep state \
     (max-src-conn-rate 100/10, overload <ratelimit> flush global)
pass out quick on xbridge inet proto tcp to $nginx_j port $port_nginx


pass in  quick on ipg_a   inet proto tcp from <ipg_j>    to <pgpeer> port 5434
pass in  quick on inic    inet proto tcp from <pgpeer>   to <ipg_j>  port 5434
pass in  quick on inode_a inet proto tcp from <inode_j>  to <pgpeer> port 5433
pass in  quick on inic    inet proto tcp from <nodepeer> to <ipg_j>  port 5433
pass out quick on ibridge inet proto tcp to port { 5434 5433 }


pass in on xbridge inet proto udp from $node_j to $resolvers port 53
pass in on xbridge inet proto tcp from $node_j to <payments> port 443
pass in on xbridge inet proto tcp from $node_j to <email>    port $port_email
pass out on xnic   inet proto udp from any     to $resolvers port 53
pass out on xnic   inet proto tcp from any     to <payments> port 443
pass out on xnic   inet proto tcp from any     to <email>    port $port_email


pass in on xnic inet proto tcp from <xpeers> \
     to port { $port_ssh_host $port_ssh_nginx_j $port_ssh_node_j $port_ssh_pg_j }
pass out on xbridge inet proto tcp from <xpeers> to $nginx_j port $port_ssh_nginx_j
pass out on xbridge inet proto tcp from <xpeers> to $node_j  port $port_ssh_node_j
pass out on xbridge inet proto tcp from <xpeers> to $pg_j    port $port_ssh_pg_j


pass out on inic inet proto udp to $priv_ntp port 123