After having played around a bit with LXC and discovered its main features, you may want to have a proper network setup for your containers.
There are multiple network setups possible and multiple ways to implement them. In this post, we are going to setup a bridge, using lxc-net. It requires very little configuration and should be enough for a simple LXC architecture.
More details about this bridge setup:
- Containers will have an IPv4 within their own subnet
- Containers will be able to access each other within this subnet
- The host will be able to access the containers trough this subnet
- Containers will have access to the internet thanks to the bridge interface
Note that I’m using Debian 9 for this tutorial. Also, if you’re using LXD to manage your LXC containers, this isn’t necessary as it does everything automatically.
Install lxc-net
That’s pretty easy, as lxc-net is a part of LXC, it’s already installed.
lxc-net uses dnsmasq to manage DHCP and DNS.
To install it, run:
apt install dnsmasq-base
Info
Do not install the dnsmasq package. Indeed, dnsmasq-base contains the binary and the doc, whereas dnsmasq also contains the service.
However, lxc-net spawns its own dnsmasq process, so if you install dnsmasq, it will run on its own and cause a conflict with lxc-net, for example :
systemctl restart lxc-net
systemctl status lxc-net
....
failed to create listening socket for 10.0.3.1: Address already in use
Failed to setup lxc-net.
...
We need the dnsmasq binary and not the service, so we only install dnsmasq-base.
If you’re using a distribution without dnsmasq-base, stop and disable the service:
systemctl stop dnsmasq
systemctl disable dnsmasq
(This is basically the same thing as uninstalling dnsmasq on Debian)
I’m adding this in the post in case someone else encounters this problem, which I did because I installed dnsmasq in the first place.
That being said, back to the tutorial!
Configure the bridge
The bridge interface will not be configured by default.
root@host ~ # cat /etc/lxc/default.conf
lxc.network.type = empty
Replace the line by this:
lxc.network.type = veth
lxc.network.link = lxcbr0
lxc.network.flags = up
lxc.network.hwaddr = 00:16:3e:xx:xx:xx
lxcbr0 is the name of our bridge on the host.
Now, tell LXC to use the bridge. Create /etc/default/lxc-net and put this in it:
USE_LXC_BRIDGE="true"
Restart lxc-net and make sure it’s running:
systemctl restart lxc-net
systemctl status lxc-net
The lxcbr0 interface should be up:
root@host ~ # ip -4 -o a show lxcbr0
3: lxcbr0 inet 10.0.3.1/24 scope global lxcbr0\ valid_lft forever preferred_lft forever
Our newly created containers will now be getting an address within the 10.0.3.1/24 subnet.
Test your bridge
Let’s use our bridge!
lxc-create -t debian -n c1
lxc-create -t debian -n c2
lxc-create -t debian -n c3
lxc-start -n c1
lxc-start -n c2
lxc-start -n c3
root@host ~ # lxc-ls --fancy
NAME STATE AUTOSTART GROUPS IPV4 IPV6
c1 RUNNING 0 - 10.0.3.32 -
c2 RUNNING 0 - 10.0.3.200 -
c3 RUNNING 0 - 10.0.3.77 -
There you have it! Do some tests with your containers:
root@host ~ # ping -c2 10.0.3.32
PING 10.0.3.32 (10.0.3.32) 56(84) bytes of data.
64 bytes from 10.0.3.32: icmp_seq=1 ttl=64 time=0.043 ms
64 bytes from 10.0.3.32: icmp_seq=2 ttl=64 time=0.087 ms
--- 10.0.3.32 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1008ms
rtt min/avg/max/mdev = 0.043/0.065/0.087/0.022 ms
root@host ~ # ping -c2 10.0.3.200
PING 10.0.3.200 (10.0.3.200) 56(84) bytes of data.
64 bytes from 10.0.3.200: icmp_seq=1 ttl=64 time=0.071 ms
64 bytes from 10.0.3.200: icmp_seq=2 ttl=64 time=0.067 ms
--- 10.0.3.200 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1024ms
rtt min/avg/max/mdev = 0.067/0.069/0.071/0.002 ms
root@host ~ # lxc-attach -n c1
root@c1:~# ip -4 -o a
1: lo inet 127.0.0.1/8 scope host lo\ valid_lft forever preferred_lft forever
15: eth0 inet 10.0.3.32/24 brd 10.0.42.255 scope global eth0\ valid_lft forever preferred_lft forever
root@c1:~# ping -c2 10.0.3.200
PING 10.0.3.200 (10.0.3.200): 56 data bytes
64 bytes from 10.0.3.200: icmp_seq=0 ttl=64 time=0.077 ms
64 bytes from 10.0.3.200: icmp_seq=1 ttl=64 time=0.088 ms
--- 10.0.3.200 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.077/0.082/0.088/0.000 ms
root@c1:~# curl -I https://angristan.xyz/
HTTP/2 200
server: nginx
date = Thu, 15 Feb 2018 23:32:51 GMT
content-type: text/html; charset=utf-8
content-length: 17443
vary: Accept-Encoding
x-powered-by: Express
cache-control: public, max-age=0
etag: W/"4423-aJyAbkYavS/1P5xFFFgdnme4hCQ"
vary: Accept-Encoding
x-cache-status: EXPIRED
strict-transport-security: max-age=31536000; includeSubDomains; preload
Neat, right?
Use another subnet
Here is the default subnet and DHCP range used by lxc-net:
LXC_ADDR="10.0.3.1" #Address of lxcbr0 on the host
LXC_NETWORK="10.0.3.0/24"
LXC_DHCP_RANGE="10.0.3.2,10.0.3.254"
To use something else, define these options in /etc/default/lxc-net.
For example:
LXC_ADDR="10.0.42.42"
LXC_NETWORK="10.0.42.0/24"
LXC_DHCP_RANGE="10.0.42.100,10.0.42.200"
Restart lxc-net:
systemctl restart lxc-net
Then you have to restart the concerned containers for them to get their new IPs:
lxc-stop -n c1 && lxc-start -n c1
lxc-stop -n c2 && lxc-start -n c2
lxc-stop -n c3 && lxc-start -n c3
Check the result:
root@host ~ # ip -4 -o a show lxcbr0
3: lxcbr0 inet 10.0.42.42/24 scope global lxcbr0\ valid_lft forever preferred_lft forever
root@host ~ # lxc-ls --fancy
NAME STATE AUTOSTART GROUPS IPV4 IPV6
c1 RUNNING 0 - 10.0.42.185 -
c2 RUNNING 0 - 10.0.42.153 -
c3 RUNNING 0 - 10.0.42.117 -
Use static IPs
As you may have noticed, by default, new containers will get a random IP address within the subnet.
If you want to define your own IPs, it’s simple.
Create /etc/lxc/dhcp.conf, and define you CTs as following:
dhcp-host=<ct-name>,<ip>
Example:
dhcp-host=c1,10.0.3.11
dhcp-host=c2,10.0.3.12
dhcp-host=c3,10.0.3.13
To tell lxc-net to load the configuration, add this line in /etc/default/lxc-net:
LXC_DHCP_CONFILE=/etc/lxc/dhcp.conf
Restart lxc-net:
systemctl restart lxc-net
Then you have to restart the concerned containers for them to get their new IPs:
lxc-stop -n c1 && lxc-start -n c1
lxc-stop -n c2 && lxc-start -n c2
lxc-stop -n c3 && lxc-start -n c3
root@host ~ # lxc-ls --fancy
NAME STATE AUTOSTART GROUPS IPV4 IPV6
c1 RUNNING 0 - 10.0.3.11 -
c2 RUNNING 0 - 10.0.3.12 -
c3 RUNNING 0 - 10.0.3.13 -
Success! 🎉
Define a domain
A quick tip: you can define a default domain for you containers using lxc-net.
Add this in /etc/defalt/lxc-net
LXC_DOMAIN="domain.tld"
For example:
LXC_DOMAIN="angristan.xyz"
If I create a new container, it now has a FQDN:
systemctl restart lxc-net
lxc-create -t debian -n ct
lxc-start -n ct
lxc-attach -n ct
root@ct:~# hostname -f
ct.angristan.xyz
Route a host port to a container
With our bridge, containers can access to the internet, but cannot be accessed.
We can use the iptables NAT routing table to map a host’s port to a container’s port, with the following command:
iptables -t nat -A PREROUTING -i <host_nic> -p tcp --dport <host_port> -j DNAT --to-destination <ct_ip>:<ct_port>
To be more specific, we’re mapping a port from the host’s public interface to the container’s IP. Obviously, if you want your container to be accessible from the internet, use the interface (host_nic) where you public IPv4 is mounted.
As an exemple, I want the web server of my c1 container to be publicly accessible.
host_nic: eth0ct_ip: 10.0.3.11host_port: 80ct_port: 8080
We’ll use:
iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 80 -j DNAT --to-destination 10.0.3.11:8080
You can now reach your container’s port 8080 trough your host’s 80 port (via its public IP).
IPv6
Sadly, I have not succeeded in setting up IPv6 with lxc-net, with both my public and private IPv6 blocks. Even tough the feature has been added in 2015, there is not documentation whatsoever and I have not found any tutorial. It seems another solution is to set up our own bridge manually.
Sources: