Wednesday, January 31, 2007

QEMU TAP bridge network configuration

Well, it is interesting to see how many posts out there on the web deals with this topic "configure tun/tap bridge for QEMU". Yet, I have to write one more page about it.

OK, here is why: 1st while most of them are informative, there has been no SINGLE page I read (dozens, if not more) that really helped me to configure the thing easily. 2nd, once I figured it out, it is unbelievable EASY!!! - it's just the lack of documentation make people stump. Last but not least, there are some docs out there are incorrect or misleading. While following their instruction may lead to a working configuration, it will cause you trouble later, or lead you to a wrong route when troubleshoot.

Here is what would have saved my 4 hours:

First and foremost, bridge in this case is really the ISO term of bridge. It is not something specific for QEMU. Actually, the bridge functionality is provided by Linux kernel, not QEMU - some docs on the net may lead you to believe bridge is a QEMU feature (well, QEMU does have this feature - by utilizing Linux feature, instead of providing the functionality itself).

Back to QEMU network 101, there are several ways to configure QEMU network, the default which happens to be the easiest is "-net user" this is a very neat function implemented by QEMU - it creates a virtual router (with virtual DHCP server, virtual DNS, etc) it uses 10.0.2.x address.
Just think about a home high-speed internet router, it gives you NAT addresses to all your computers at home, the router then connects to the WAN (read internet here) and forward traffic back and forth between your computers and the internet. The user mode QEMU does the same thing, QEMU gives your "Virtual computer" a "virtual IP - i.e. 10.0.2.15" and QEMU talks to the real network.
This usually is good enough if you only surf the internet from your virtual computer. However, since not every application work with NAT, (and since most of the home network already use a NAT to share one real internet IP, the virtual computer is behind 2 NATs). You may want some "direct" connection for your virtual computer.

Now, here comes the confusing part. QEMU supports tap device ("-net tap" instead of "-net user"). And its official document says "... you can then configure it as if it was a real ethernet card." But how? if everyone is as familiar with TAP device as the QEMU author, probably I won't need to write this (at least it would have saved me 4 hours if someone has written something like this :-)


TUN/TAP device is "virtual ethernet driver", the official doc of tuntap.txt says: "TUN/TAP provides packet reception and transmission for user space programs. It can be seen as a simple Point-to-Point or Ethernet device, which, instead of receiving packets from physical media, receives them from user space program and instead of sending packets via physical media writes them to the user space program."

So, QEMU runs on the real computer as a program, can write packets to the TAP device. That's cool! QEMU can simply write all the virtual computer generated packets to the TAP device and forward packets to the virtual computer, right? Right! but it is only sufficient if your virtual computer wants to talk to the real computer it runs on ONLY. Because once the packets from the virtual computer reaches the TAP device, how does Linux know where to send the packets? That's not part of the TAP device functionality.

So here comes bridge.
The bridge is (sort of) the most reliable way of connecting your virtual computer to the real network. Unlike some docs on the internet suggested - you do NOT need to configure IP for the TAP device for the bridge to work, no IP forwarding, routing needed! That's why I said "1st, bridge is ISO bridge!" at the beginning. Because packets are forwarded at Ethernet level (layer 2 in terms of ISO standard), bridge does not even care about IP (sort of, you will see why you still need to manipulate your IP a little bit later.

For those who are not familiar with Ethernet bridging (this is not a scientific definition, rather it is an example tries to help people get a feeling of what bridge is): bridge is a device have 2 (or more) NIC (Network Interface Card), one NIC connected to one Ethernet, the other connected to a second Ethernet. The bridge then joins the 2 Ethernet together, to become one. Of course, a bridge can have more than 2 NIC and connects more than 2 networks. The good use of bridge for example is connecting a 1GBps Ethernet with a 10MBps Ethernet. Since you don't want to slow down the GB net with all the 10MB traffic, you keep them separate but connect them with a bridge. So they can talk to each other, but do not bother each other that much.

OK, back to how to bridge QEMU.
You must have already figured out you need to bridge your real NIC with the TAP device QEMU uses (once you start QEMU with -net tap, it creates a TAP device automatically).
One thing you need to know is once you start a bridge, you cannot manipulate your eth0 (or whatever real NIC device in your real Linux) anymore. "The reason?" I head. Well, since bridge is at level 2, (before IP level), it needs to be able to tell where a packet is going (by the MAC address in the packet). If you manipulate the eth0 device, your packet is going out to the real network directly, the bridge will not have access to the packet, hence it breaks the bridge. (think about a DHCP solution on our 1GBps network and 10MBps network. If you request a DHCP IP address on your eth0 - which is on your 1GBps network, your DHCP server has to be on the 1GBps network or it won't be able to get an IP address. In this situation, you will have to have 2 DHCP servers one on 1GBps, one on 10MBps. You will either create 2 segment of dynamic IPs or find a way to synchronize 2 DHCP servers leases).

So, to solve this problem (well, the problem is there only because you use your bridge-your Linux box-as a computer and a bridge; for a hardware bridge, it is transparent on the Ethernet.) you need to connect to the Ethernet though the bridge interface (once you configured it, you will see the bridge interface has the eth0 MAC address). So the bridge (read the Linux kernel bridge module) can then check the content of the packets and send the packets to the right network.

Since the IP layer is layer 3, which means it builds on top of the bridge which runs at layer 2, you will have to start your IP stack on top of the bridge instead of your real NIC (eth0 in most of the case).
Finally here comes the howto:
Be warned, your IP network will have to come down for a little while (for me this was about 5 seconds). This is because you need to start the IP address on the bridge interface later.
First, release your IP address assigned to your real NIC. I also rename my eth0 to reth0 (stands for Real ETH0) and create the bridge with name "eth0". The reason being it is simpler for your other programs (i.e. iptables which may use eth0 as interface name somewhere).
Then you create a bridge, and add 2 NIC (one is the real NIC the other is the TAP device).
You then re-acquire a DHCP lease with the bridge interface (in my case, is called eth0 now - hey I know it is confusing, but it is just a name, right?).
The following is tested on SLES 9, it should work on most of the linux distro with little change (i.e. the dhcp client tool may be different). You also need the bridge package which is not installed by default (at least on SLES 9 it is not)




#!/bin/bash
# 1st, release all DHCP address and remove all IP address associated with the original eth0
/sbin/dhcpcd -k
/sbin/ip addr flush eth0
# then take the interface down so we can rename it
/sbin/ip link set eth0 down
# now rename the original eth0 to reth0 (Real ETH0)
/sbin/nameif -r reth0 eth0
# OK, bring the same interface (with new name though) back up
/sbin/ip link set reth0 up
# 2nd let's create a bridge called eth0 so other programs think they are talking to the same old interface (actually they will talk to the bridge which is a clone of the original eth0 - with name MAC addr)
/sbin/brctl addbr eth0
# then add both origianl eth0 and tap1 device to the bridge
/sbin/brctl addif eth0 tap1
/sbin/brctl addif eth0 reth0
echo "showing bridge mac addresses"
/sbin/brctl showmacs eth0
# 3rd, we need to bring the newly created bridge UP
/sbin/ip link set eth0 up
# 4th, renew the DHCP address if possible
/sbin/dhcpcd -n
/sbin/ip addr show



Please note, there are other ways of connecting your virtual computer to the real network with TAP device: you can configure routing on your real computer (I don't see a reason why I want to do that - if you use NAT, why don't you use the "-net user" instead which is much simpler; if you use real world IP, why don't you use bridge? To configure routing for the TAP, you need to start IP stack on the TAP device and start ip forwarding on the real Linux - unnecessarily complex for my environment).

Another good option is VDE (Virtual Distributed Ethernet). It is a very good solution, in some case, it maybe better than bridge. The best feature is you don't need the brief down time as the bridge solution. I found the description of VDE is much better than bridge and hence I will skip this part.

21 comments:

Anonymous said...

Alan,

Thanks for your suggestion on bridging TAP interfaces; I was one of those guys trying to tweak multiple TAPs and routing between them.... bridging is far easier!

And you are right about the lack of quality documentation.

Mark.
lau.nam.mee[at]gmail.com

Mariano said...

Thanks a lot for your document !!!

Finally I understood the reason of being of bridge. I've read a lot of documents, saying to use iptables and others complex things, In layer 2 iptables?....

Only one comment to your doc, tap device must be created before to sucessfully attach it to the bridge.

In my distro, rename interface is:

nameif reth0 00:13:D4:3F:92:E0

where 00:.... is the MAC address of physical interface.

THANKS A LOT FOR YOUR HELP

Mark said...

Your a Genius!
One note: as mentioned in the comment above you have to create a tap device. To do this (on debian at least) you need uml-utilities and use tunctl -t tap0.
Cheers

ciphernaut said...

Hi,

how would one go about configuring a specific mac address to present to the guest on this tap/bridge configuration?

It appears that macaddr= can only be set in -net nic mode ....

Alan Tan said...

You should be able to specify "-net nic .... -net tap" like below:

qemu -boot c -hda /hd1 -m 1024 -localtime -net nic,macaddr1=0e:50:40:17:39:01
-net tap

V.I said...

Great explanations Alan!!

Unfortunately, it seems that there is a lack of (good) documentation about bridged networking in QEMU/KVM. I followed instructions from a couple of sources on the web, telling me to do A, then B, C, etc, without offering an adequate reasoning. I ended up totally confused, and it took me a whole day to figure out why I was supposed to connect things in the way I was being told. Finally I found this article of yours. Well done!

Thank you very much, indeed.

V.I

Anonymous said...

A minor nitpic: nameif -r doesn't work
You might want to use "ip link set eth0 name reth0" instead

Saurabh said...

I am still unable to get it to work. What ip/network address should I provide to tap device? What ip/network should be given to nic in guest os?

Saurabh said...

should we give the same mac address to nic in guest os as held by tap device?

Anonymous said...

Great article, thanks!

FYI, I put these three lines in my /etc/qemu-ifup:

sudo /usr/sbin/tunctl -t tap0
sudo /sbin/ifconfig $1 0.0.0.0 promisc up
sudo /usr/sbin/brctl addif br0 $1

Anonymous said...

Thanks for the simple instructions.

As another poster pointed out, "nameif -r" doesn't work, you can use "nameif reth0 00:aa:bb:cc:dd:ee" instead (replace with your MAC address), or specify this in /etc/mactab (see man nameif for more).

One thing that wasn't obvious for me is the actual qemu syntax: "qemu -net tap" won't work, instead you need to do "qemu -net nic -net tap".

Anonymous said...

I had to run qemu as root to get this to work - otherwise it complains about an ioctl on /dev/net/tun failing. I tried chmod and tunctl but couldn't get it to work as non-root.

John said...

Nice Information, I am seeking for this information I hope it will be help me in university assignment. Thanks again!

Regards,
John


P.S: Read This Candid Review About The Best Forex Signal Provider Online For Making Big Profits In Forex Market.

devi said...

Thank you very much for providing such a brief and self explanatory description of linux as well as qemu briding. It was very helpful

Devi

Knight Samar said...

Hi,

Thanks a lot for the detailed and precise info :)

But, in addition to what your wonderful tutorial told, I had to do this too :

* Creating a qemu-ifup script as described here http://compsoc.dur.ac.uk/~djw/qemu.html

* Running qemu with "-net nic -net tap"
to actually make the vm use the vlan

But, great thanks for the wonderful piece of information about this very obscure thing :)

Najib Ibrahim said...

Hi, I have a quick-howto that simplify the qemu networking using qemu-vde-tun/tap-bridge.

Take a look and please comment :)

http://selamatpagicikgu.wordpress.com/2011/06/08/quickhowto-qemu-networking-using-vde-tuntap-and-bridge/

Thank you.

Anonymous said...

how do you determine
what TAP device is associated with what VM once they have already been created?

s0rcerer said...

Can you say how to do that in the "/etc/network/interfaces" file?

I'm in a trouble with my beta-server and I can't solve it.

Thanks

Henrik said...

You've got a nice post here.
I had to figure that out too once, but it was good as a reminder as my memory keeps letting me down from time to time. ;-)

Also, in the unlikly event that You'd want to set up the bridge over a wireless device, like wlan0 (please don't ask why I keep doing these things), it won't work without doing some configuring on the link layer level of the wireless device.

You need to install the iw package and configure the device as follows, prior to configuring the bridge:
(sudo) iw dev wlan0 set 4addr on

Cheers,
Henrik.

Henrik said...

You've got a nice post here.
I had to figure that out too once, but it was good as a reminder as my memory keeps letting me down from time to time. ;-)

Also, in the unlikly event that You'd want to set up the bridge over a wireless device, like wlan0 (please don't ask why I keep doing these things), it won't work without doing some configuring on the link layer level of the wireless device.

You need to install the iw package and configure the device as follows, prior to configuring the bridge:
(sudo) iw dev wlan0 set 4addr on

Cheers,
Henrik.

Anonymous said...

Good and clear writeup!

One reason to configure routing on the host is:
- if you want to access your guest from the host (this excludes the default NAT) &&
- you do not want the guest to be visible to other hosts in the same broadcast domain (this excludes bridge, unless some iptables is possible betweeen eth0 and tap)
In this situation I'd rather go for routing, which I still have to think how.
Possibly with two virtual interfaces in eth0: eth0.0 in the physical segment and eth0.1 in the bridge.