/ SysAdmin

Setup a network bridge for your LXC containers with lxc-net

After having played around a bit with LXC and discovered its main features, you may want to have a proper network setup for your containers.

There are multiple network setups possible and multiple ways to implement them. In this post, we are going to setup a bridge, using lxc-net. It requires very little configuration and should be enough for a simple LXC architecture.

More details about this bridge setup:

  • Containers will have an IPv4 within their own subnet
  • Containers will be able to access each other within this subnet
  • The host will be able to access the containers trough this subnet
  • Containers will have access to the internet thanks to the bridge interface

Note that I'm using Debian 9 for this tutorial. Also, if you're using LXD to manage your LXC containers, this isn't necessary as it does everything automatically.

Table of content

Install lxc-net

That's pretty easy, as lxc-net is a part of LXC, it's already installed.

lxc-net uses dnsmasq to manage DHCP and DNS.

To install it, run:

apt install dnsmasq-base

Do not install the dnsmasq package. Indeed, dnsmasq-base contains the binary and the doc, whereas dnsmasq also contains the service.

However, lxc-net spawns its own dnsmasq process, so if you install dnsmasq, it will run on its own and cause a conflict with lxc-net, for example :

systemctl restart lxc-net
systemctl status lxc-net

....
failed to create listening socket for 10.0.3.1: Address already in use
Failed to setup lxc-net.
...

We need the dnsmasq binary and not the service, so we only install dnsmasq-base.

If you're using a distribution without dnsmasq-base, stop and disable the service:

systemctl stop dnsmasq
systemctl disable dnsmasq

(This is basically the same thing as uninstalling dnsmasq on Debian)

I'm adding this in the post in case someone else encounters this problem, which I did because I installed dnsmasq in the first place.

That being said, back to the tutorial!

Configure the bridge

The bridge interface will not be configured by default.

root@host ~ # cat /etc/lxc/default.conf 
lxc.network.type = empty

Replace the line by this:

lxc.network.type = veth
lxc.network.link = lxcbr0
lxc.network.flags = up
lxc.network.hwaddr = 00:16:3e:xx:xx:xx

lxcbr0 is the name of our bridge on the host.

Now, tell LXC to use the bridge. Create /etc/default/lxc-net and put this in it:

USE_LXC_BRIDGE="true"

Restart lxc-net and make sure it's running:

systemctl restart lxc-net
systemctl status lxc-net

The lxcbr0 interface should be up:

root@host ~ # ip -4 -o a show lxcbr0
3: lxcbr0    inet 10.0.3.1/24 scope global lxcbr0\       valid_lft forever preferred_lft forever

Our newly created containers will now be getting an address within the 10.0.3.1/24 subnet.

Test your bridge

Let's use our bridge!

lxc-create -t debian -n c1
lxc-create -t debian -n c2
lxc-create -t debian -n c3
lxc-start -n c1
lxc-start -n c2
lxc-start -n c3
root@host ~ # lxc-ls --fancy
NAME STATE   AUTOSTART GROUPS IPV4       IPV6 
c1   RUNNING 0         -      10.0.3.32  -    
c2   RUNNING 0         -      10.0.3.200 -    
c3   RUNNING 0         -      10.0.3.77  -

There you have it! Do some tests with your containers:

root@host ~ # ping -c2 10.0.3.32
PING 10.0.3.32 (10.0.3.32) 56(84) bytes of data.
64 bytes from 10.0.3.32: icmp_seq=1 ttl=64 time=0.043 ms
64 bytes from 10.0.3.32: icmp_seq=2 ttl=64 time=0.087 ms

--- 10.0.3.32 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1008ms
rtt min/avg/max/mdev = 0.043/0.065/0.087/0.022 ms
root@host ~ # ping -c2 10.0.3.200
PING 10.0.3.200 (10.0.3.200) 56(84) bytes of data.
64 bytes from 10.0.3.200: icmp_seq=1 ttl=64 time=0.071 ms
64 bytes from 10.0.3.200: icmp_seq=2 ttl=64 time=0.067 ms

--- 10.0.3.200 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1024ms
rtt min/avg/max/mdev = 0.067/0.069/0.071/0.002 ms
root@host ~ # lxc-attach -n c1

root@c1:~# ip -4 -o a
1: lo    inet 127.0.0.1/8 scope host lo\       valid_lft forever preferred_lft forever
15: eth0    inet 10.0.3.32/24 brd 10.0.42.255 scope global eth0\       valid_lft forever preferred_lft forever

root@c1:~# ping -c2 10.0.3.200
PING 10.0.3.200 (10.0.3.200): 56 data bytes
64 bytes from 10.0.3.200: icmp_seq=0 ttl=64 time=0.077 ms
64 bytes from 10.0.3.200: icmp_seq=1 ttl=64 time=0.088 ms
--- 10.0.3.200 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.077/0.082/0.088/0.000 ms
root@c1:~# curl -I https://angristan.xyz/
HTTP/2 200 
server: nginx
date: Thu, 15 Feb 2018 23:32:51 GMT
content-type: text/html; charset=utf-8
content-length: 17443
vary: Accept-Encoding
x-powered-by: Express
cache-control: public, max-age=0
etag: W/"4423-aJyAbkYavS/1P5xFFFgdnme4hCQ"
vary: Accept-Encoding
x-cache-status: EXPIRED
strict-transport-security: max-age=31536000; includeSubDomains; preload

Neat, right?

Use another subnet

Here is the default subnet and DHCP range used by lxc-net:

LXC_ADDR="10.0.3.1" #Address of lxcbr0 on the host
LXC_NETWORK="10.0.3.0/24"
LXC_DHCP_RANGE="10.0.3.2,10.0.3.254"

To use something else, define these options in /etc/default/lxc-net.

For example:

LXC_ADDR="10.0.42.42"
LXC_NETWORK="10.0.42.0/24"
LXC_DHCP_RANGE="10.0.42.100,10.0.42.200"

Restart lxc-net:

systemctl restart lxc-net

Then you have to restart the concerned containers for them to get their new IPs:

lxc-stop -n c1 && lxc-start -n c1
lxc-stop -n c2 && lxc-start -n c2
lxc-stop -n c3 && lxc-start -n c3

Check the result:

root@host ~ # ip -4 -o a show lxcbr0
3: lxcbr0    inet 10.0.42.42/24 scope global lxcbr0\       valid_lft forever preferred_lft forever

root@host ~ # lxc-ls --fancy
NAME STATE   AUTOSTART GROUPS IPV4        IPV6 
c1   RUNNING 0         -      10.0.42.185 -    
c2   RUNNING 0         -      10.0.42.153 -    
c3   RUNNING 0         -      10.0.42.117 -

Use static IPs

As you may have noticed, by default, new containers will get a random IP address within the subnet.

If you want to define your own IPs, it's simple.

Create /etc/lxc/dhcp.conf, and define you CTs as following:

dhcp-host=<ct-name>,<ip>

Example:

dhcp-host=c1,10.0.3.11
dhcp-host=c2,10.0.3.12
dhcp-host=c3,10.0.3.13

To tell lxc-net to load the configuration, add this line in /etc/default/lxc-net:

LXC_DHCP_CONFILE=/etc/lxc/dhcp.conf

Restart lxc-net:

systemctl restart lxc-net

Then you have to restart the concerned containers for them to get their new IPs:

lxc-stop -n c1 && lxc-start -n c1
lxc-stop -n c2 && lxc-start -n c2
lxc-stop -n c3 && lxc-start -n c3
root@host ~ # lxc-ls --fancy
NAME STATE   AUTOSTART GROUPS IPV4      IPV6 
c1   RUNNING 0         -      10.0.3.11 -    
c2   RUNNING 0         -      10.0.3.12 -    
c3   RUNNING 0         -      10.0.3.13 -

Success! 🎉

Define a domain

A quick tip: you can define a default domain for you containers using lxc-net.

Add this in /etc/defalt/lxc-net

LXC_DOMAIN="domain.tld"

For example:

LXC_DOMAIN="angristan.xyz"

If I create a new container, it now has a FQDN:

systemctl restart lxc-net
lxc-create -t debian -n ct
lxc-start -n ct
lxc-attach -n ct
root@ct:~# hostname -f
ct.angristan.xyz

Route a host port to a container

With our bridge, containers can access to the internet, but cannot be accessed.

We can use the iptables NAT routing table to map a host's port to a container's port, with the following command:

iptables -t nat -A PREROUTING -i <host_nic> -p tcp --dport <host_port> -j DNAT --to-destination <ct_ip>:<ct_port>

To be more specific, we're mapping a port from the host's public interface to the container's IP. Obviously, if you want your container to be accessible from the internet, use the interface (host_nic) where you public IPv4 is mounted.

As an exemple, I want the web server of my c1 container to be publicly accessible.

  • host_nic: eth0
  • ct_ip: 10.0.3.11
  • host_port: 80
  • ct_port: 8080

We'll use:

iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 80 -j DNAT --to-destination 10.0.3.11:8080

You can now reach your container's port 8080 trough your host's 80 port (via its public IP).

IPv6

Sadly, I have not succeeded in setting up IPv6 with lxc-net, with both my public and private IPv6 blocks. Even tough the feature has been added in 2015, there is not documentation whatsoever and I have not found any tutorial.
It seems another solution is to set up our own bridge manually.

Well, another subject for another tutorial?

In the meantime, I hope my lxc-net tutorial will be useful. The post is somewhat long, but in the end it's basic IPv4 addressing, so it's easy to understand.

I'm still learning networking though, so if you have any correction/suggestion to make, I'm all ears!

Sources:

Angristan

Angristan

I'm an 18 years old French sysadmin studying at a IT school and working for a web hosting company.

Read More