Comprehensive xen Debian Wheezy PCI Passthrough Tutorial

Home Wiki Blog Lists User Voice Downloads Xen Planet

Comprehensive Xen Debian Wheezy PCI Passthrough TutorialFrom Xen

Contents1 Comprehensive Debian Wheezy Xen Tutorial

1.1 Introduction1.1.1 Objectives Overview

1.2 UEFI Configuration1.3 Wheezy Installation

1.3.1 References1.4 Wheezy Configuration

1.4.1 References1.5 Compiling a Custom Linux Kernel

1.5.1 References:1.6 Compiling and Installing Xen

1.6.1 References:1.7 Xen Configuration

1.7.1 References1.8 Installing Windows HVM

1.8.1 References1.9 Installing PFSense HVM1.10 Managerial Activities

1.10.1 Drive Partitioning1.10.2 LVM1.10.3 Using mount With LVM Virtual Partitions

1.10.4 dd

Comprehensive Debian Wheezy Xen TutorialIntroduction (http://www.youtube.com/watch?v=mnTj6_kDIeg)The following guide is comprised of my personal documentation and excessive filtering for human consumption. It details the series of steps I took to successfully install and compile acustom Linux Kernel, Xen 4.2 unstable, and run three virtual machines to perform unique tasks all on a single physical computer.To achieve the desired functionality, I used IOMMU for PCI Passthrough with a multimedia operating system, granting complete access to a graphics card for GPU Acceleration.I am a college student, not a trained professional, and I am sharing this documentation for educational purposes. Blind use of this document for a production environment in a businesssetting would be ill-advised.

To quickly summarize my experiences, I decided to try Xen and began researching in early January 2012.I purchased equipment in March 2012, and began what I thought would take "at most two weeks".By late April I finally had a (mostly) functional system.I have been fine tuning the system and my process since then to produce this guide for others.The purpose of this guide is to turn a 6 month project into a series of steps that can be reproduced inside the time frame of 1-2 days.

Objectives Overview

I wanted a single physical computer, that could handle three computers worth of separated activities at all times, including these four specifics:

Router/Firewall (PFSense)DNS & Web Development (Debian Squeeze)Application Development (Debian Wheezy)Multimedia & Gaming (Windows 7)

I investigated alternative software, including VMWare's ESXi and Citrix's XenServer.

I had come from a VMWare platform having used both VMware Server 2 and VMWare Workstation 8 for the same key objectives previously but with a Windows Host, subject toWindows Updates which let to my investigation.ESXi was easy to install, but missing numerous drivers for hardware components. I quickly ruled it out as I wanted a home-use system, and there was no convenient orwell-documented method to installing or even launching a virtual machine from their on-server interface.Citrix's XenServer was easily my favorite of the options, it's design made for a great user experience. Everything was easy to find and do. However it came with a minimum price tag of$1000, and if I wanted PCI passthrough for graphics cards $2500. This was beyond my reach as a college student.I chose Xen since it had support for my hardware, was a free open-source project, and had a great community with lots of activity.

Things you Will Need:Compatible HardwareATI Graphics CardMotherboard with UEFI & VT-d & Onboard GraphicsLatest Ubuntu Live DVDDebian Wheezy Beta1 (Or Newer) Installer

nVidia Cards can be made to work, with extensive patching, in Windows XP and supposedly Windows 8 Preview, for more information visit David Techer's blog (http://www.davidgis.fr/blog/) .If you plan to pass your graphics card to a virtual machine, you will need either a second graphics card, onboard graphics, or a second computer to manage your Dom0 system andinstall virtual machines over VNC.UEFI compatible boot DVD will save you an undocumented step for setting up a UEFI boot loader.

Xen is picky, different hardware may yield different results, both at compile time and runtime. If you want to save yourself some hassle, here is my hardware list and some suggestionsto avoid:

Motherboard:ASRock Z68 Extreme4 Gen3

CPU:Intel Core i2600

RAM:12GB 1333Mhz Corsair XMS (2x2G 2x4G)

Boot Disk:240GB OCZ Vertex 3

GPU:

ATI Radeon HD 6870LAN:

Onboard Broadcom BCM57781PCIe EXPI9301CTBLK

Hardware and Configurations to Avoid:

NF200 Chipsets are not IOMMU compatiblenVidia Graphics CardsRAID5 yields horrible performance

NF200 is a PCI Switch for motherboards sporting SLI and CrossFire, avoid it if you want be able to use those PCI slots for passthrough.

UEFI Configuration (http://www.youtube.com/watch?v=mnTj6_kDIeg)These instructions are for an Intel CPU and ASRock UEFI Motherboard, and may vary depending on your manufacturer as well as your choice of CPU. If you are unfamiliar withmotherboard configuration, you may want to watch the video for a visual walk through.Before starting reset your CMOS so you have a clean slate to work from.List of key settings:

Turn Legacy USB 3.0 offTurn on VT-x and VT-dSet drives to ACHI ModeChange default video to Onboard

Justification:Consider yourself warned, if you leave Legacy USB on and leave a backup USB drive connected, your system will fail to boot. To my surprise that is because it is trying to boot from thatUSB drive (even if that drive is USB 2.0), which threw me off so I recommend disabling that feature.Most motherboards will have Intel Virtualization enabled by default (VT-x), but that is not the same as IOMMU, be sure to look for VT-d or check the manual for details. For my systemVT-x was in the CPU configuration area, and VT-d was in the Northbridge configuration area.ACHI is a superior choice for performance with modern hard drives, and while RAID is an alternative that uses ACHI as its underlying type, it increases boot time by checking RAIDconfigurations, and most onboard RAID is software RAID that is built for Windows and rarely helpful for Linux.ASRock boards will use a PCI GPU by default if it is plugged into a PCI slot, regardless of if any video cables are connected, and if you plan to pass that device to a virtual machine, you

do not want to use it for video.

Wheezy Installation (http://www.youtube.com/watch?v=mnTj6_kDIeg&t=4m20s)Select Advanced Options from the menu, and then Advanced Install.To expedite this process we can select defaults for most options (such as for networking).Once we reach Hard Drive Partitioning options, select Manual at the bottom.Select the disk(s) we will be using by title not partition space and hit enter, we can now select gpt from the partition tables instead of msdos.We will then create three partitions:

256MB FAT32 Partition mounted to /boot/efi256MB Ext4 Partition mounted to /bootRemainder LVM Partition

Select Configure the Logical Volume Manager and create a volume group.Create three Logical Volumes:

8GB or More for Linux20GB or More for Home2GB or More for Swap

Complete LVM Configuration, and then back to our partition settings we can now select formats and mount points:

8GB LV Ext4 for root (/)20GB LV Ext4 for /home2GB LV swap

Optionally add noatime flags to all the Ext4 partitioned disks.Complete the Partition steps by selecting done and create the partitions.Continue through the options with defaults, until we get to the main Packages.De-select Debian Desktop Environment and continue

Do not install a boot loader, select the Continue without a bootloader option and be sure to jott down the information it provides us for later.Complete the installation and restart our computer.Insert an Ubuntu Live DVD (or live USB created from the DVD iso) and press whatever key opens your motherboard's boot menu.Select the UEFI boot option from the boot menu.When presented with the Grub menu screen, press c on the keyboard to access the Emergency Grub Console.Emergency Grub allows us to manually boot our Linux, since we did not install a boot loader.Type ls to list the disks, we should have an (hd0) and three gpt partitions. If booting off a Live USB these will appear under (hd1) instead as the USB drive is (hd0).Assuming the same partitioning as mine, in the below example tab indicates hit the tab key to auto-complete, and we can either use the data provided when we selected Continuewithout a bootloader or just substitute xen for the name we gave our Volume Group name, and linux for our root (/)Logical Volume:

grub> set root=(hd0,gpt2)grub> linux /vmlinu_tab_ ro root=/dev/mapper/xen-linuxgrub> initrd /initrd_tab_grub> boot

Gibberish may be displayed when tabbing to auto-complete, this is normal and can be ignored. After entering boot it will begin starting Linux.If done successfully a terminal login will be presented, which concludes this segment of the guide.In the next segment we will start by enter our credentials and we can start installing some key packages to get UEFI working.

References

EFI Grub on Wheezy (http://blog.garyhawkins.me.uk/?p=185)

Wheezy Configuration (http://www.youtube.com/watch?v=d3pFN2C10x0)Note:I have done my best to separate my crazier desires from the average persons needs.

Manoel AlvesNoteDeve ser executado o comando "ls" para listar as parties no disco

grub> ls

To do this I created bold headers with line breaks for special sub-sections, whose subsequent contents you can skip if said bold text does not apply to your situation.

Assuming you have just installed Debian Wheezy and are at the login prompt, you want to login as root so you can install packages (alternatively you can use su to change users if youprefer).

UEFI STEPSAssuming you came from my previous guide and have just manually booted the Debian system in EFI mode. Your first objective should be getting an efi bootloader installed:

aptitude install grub-efi-amd64

Once installed you will want to run the install and update scripts:

grub-installupdate-grub

Successful installation of the boot manager will result in a list of the boot order with an entry "debian" listed. If this didn't work then you may not have booted in EFI mode.END UEFI STEPS

Now let's install a couple more basic packages:

aptitude install ssh sudo

With ssh installed we can, optionlly, access remotely for the remainder of the guide. With sudo we can add our user to the sudoers group like so (cdelorme is my username):

usermod -aG sudo cdelorme

If you omit the a you will replace all existing groups your user belongs to, so be careful.I recommend rebooting now to get ssh working, and also so you can login as your user to use the sudo command (if you used su you may need to logout and log back in for sudo towork).Next we want to prime our server with some general utility packages:

Manoel AlvesNote#Repositrio do grub

deb http://ftp.de.debian.org/debian jessie main

Manoel AlvesNoteNome do usurio criado

partedsambantpscreenp7zip-full

Delicious Copy Pasta:

sudo aptitude install parted samba ntp screen p7zip-full

Note: I used to install ntfsprogs but the package is currently bugged and eats CPU cycles, so I would avoid it if possible.Configuration of samba and ntp will vary by system, I recommend using other guides online for this since as long as they work they will be fine when we start using Xen. If you want myconfigurations specifically, ask and I may consider adding them to a future edition of this guide.

Development Environment StepsSince I intend to use Debian Wheezy as a development environment, I also installed these packages:

openjdk-7-jdkantglibboost-devgnusteppythonpython-dev

Delicious Copy Pasta:

sudo aptitude install openjdk-7-jdk ant g libboost-dev gnustep python python-dev

End Development Environment Steps

GUI Environment StepsTo be honest, I don't care for the CD Installation of Gnome3, it's less work but the cost is 30 minutes of installation time, and a whole slew of packages I never use, so I went minimalist

and found exactly what I needed:

gnome-sessiongnome-terminalgnome-screensaverxserver-xorg-coregdm3gksuia32-libsia32-libs-gtklibc6-dev-i386binfmt-support

Delicious Copy Pasta:

sudo aptitude install gnome-session gnome-terminal gnome-screensaver xserver-xorg-core gdm3 gksu ia32-libs ia32-libs-gtk libc6-dev-i386 binfmt-support

If you intend to use Xen with SDL later on, I recommend replacing xserver-xorg-core with xorg and xorg-dev packages, as these are what we will install later, and they containthe xserver-xorg-core.As a bare-bones install of Gnome3, it should take roughly 5 minutes as opposed to 30.At this stage you can run startx to launch the GUI environment, but I generally reboot to test the login screen.End GUI Environment Steps

Optional GUI AdditionsOptional additions include enabling audio, installing some basic multimedia, utilities, and important software like Guake terminal, Google Chrome, and Sublime Text.Starting with Audio:

sudo aptitude install alsa-basesudo alsactl init

Now with GUI utilities:

python-xdg

font-managergnome-disk-utilitygnome-screenshotguake

Delicious Copy Pasta:

sudo aptitude install python-xdg font-manager gnome-disk-utility gnome-screenshot guake

Note: Guake has a bug with Gnome3 that prevents it from starting at boot, here is how you can fix it:This is actually a very simple edit, in this specific version of guake (located at /usr/bin/guake) line 653 contains a notification.show(), we just have to wrap it in a simpletry-catch so it doesn't crash when it is run before the GUI is ready to accept a pynotify event:

try:notification.show()except Exception:pass

Next we want to set Guake to start at login. You can make an auto-start file via:

mkdir ~/.config/autostartcd ~/.config/autostartecho "[Desktop Entry]" >> guake.desktopecho "Name=Guake Terminal" >> guake.desktopecho "Name[pt]=Guake Terminal" >> guake.desktopecho "Name[pt_BR]=Guake Terminal" >> guake.desktopecho "Comment=Use the command line in a Quake-like terminal" >> guake.desktopecho "TryExec=guake" >> guake.desktopecho "Exec=guake" >> guake.desktopecho "Icon=/usr/share/pixmaps/guake/guake.png" >> guake.desktopecho "Type=Application" >> guake.desktopecho "Categories=GNOME;GTK;Utility;TerminalEmulator;" >> guake.desktop

For google chrome we need to add a new debian source to /etc/apt/sources.list, and download their package key:

su root

echo "deb http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.listwget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add -aptitude updateexit

Next we can install these packages:

google-chrome-unstableeoggnashvlcgtk-recordmydesktop

Delicious Copy Pasta:

sudo aptitude install google-chrome-unstable eog gnash vlc gtk-recordmydesktop

Note: I am aware that google source is for Squeeze, I have had no problems running it in Wheezy.After google-chrome has been installed, it will create its own sources.list file entry, so we want to go back in and remove the line we added from /etc/apt/sources.list, and re-runsudo aptitude update to make sure it works.Finally, I am in love with Sublime Text 2, so I went to their website for the download, and went through these steps for a complete installation.Once downloaded, let's extract it and move it to /usr/local, and create a new command to run it from terminal:

tar xf sublime-text.tar.bz2sudo mv sublime-text /usr/local/cd /usr/local/sublime-textsudo ln -s /usr/local/sublime-text/sublime_text /usr/bin/subl

Next let's create a desktop file so we can run it as an application and set default mimetypes from the GUI. Let's navigate to /usr/share/applications/ and create a new filesubl.desktop with these contents:

[Desktop Entry]Name=Sublime Text 2Comment=The Best Text Editor in the World!TryExec=subl

Exec=sublIcon=/usr/local/sublime-text/Icon/256x256/sublime_text.pngType=Application

Next to set the mimetypes we add these lines to /usr/share/applications/defaults.list:

text/plain=subl.desktoptext/css=subl.desktoptext/htm=subl.desktoptext/javascript=subl.desktoptext/x-c=subl.desktoptext/csv=subl.desktoptext/x-java-source=subl.desktoptext/java=subl.desktop

There are user-account alternatives that may seem more appropriate, but I wanted these to apply system-wide.End Optional GUI Additions

Thus concludes our Debian Wheezy configuration, and we are now ready to move onto compiling a Custom Linux Kernel.

References

Missing ia32-libs-gtk for Gnome3 (http://lunduke.com/forum/discussion/126/ubuntu-12-04-64bit-error-while-loading-shared-libraries-libgtk-x11-2-0-so-0/p1)Samba Config (http://www.samba.org/samba/docs/man/manpages-3/smb.conf.5.html)

Compiling a Custom Linux Kernel (http://www.youtube.com/watch?v=xjcDL9X-2M8)Compiling a custom Linux Kernel will make PCI passthrough much easier, but it also allows us to improve performance in various areas for Dom0.For starters let's create a folder for compiling our packages in our home directory:

mkdir -p ~/src/linux

cd ~/src/linux

Next we will install the required packages:

wgettarbzip2build-essentiallibncurses-devkernel-packagefakeroot

Delicious copy pasta:

sudo aptitude install wget tar bzip2 build-essential libncurses-dev kernel-package fakeroot

To improve compiling speed we can set the concurrency to double the number of physical CPU cores by running these commands:

su rootecho "CONCURRENCY_LEVEL=8" >> /etc/kernel-pkg.confexit

IMPORTANT NOTE: Kernel 3.5 and 3.5.2 are bugged[1] (http://wiki.xenproject.org/wiki/Comprehensive_Xen_Debian_Wheezy_PCI_Passthrough_Tutorial#endnote_3.5-bug) and passthrough with USBControllers fails (though VGA Passthrough works for some odd reason). That is why this guide is using 3.4.9, however 3.3.8 has also been tested and works.Our next step is downloading, extracting, and inserting the current kernel configuration before adjusting the kernel we are about to compile:

wget http://www.kernel.org/pub/linux/kernel/v3.0/linux-3.4.9.tar.bz2tar jxf linux-3.4.9.tar.bz2 && rm linux-3.4.9.tar.bz2cd linux-3.4.9/cp /boot/config-3.2.0-3-amd64 .configmake menuconfig

Configuring a kernel is not as complex as it seems, the menu is just a hierarchy of options, and thankfully you can search for settings with / and get details on individual settings with h.Here is a tree-map list of all the important settings we need to change:

Processor Type and FeaturesPreemption Model (Voluntary Kernel Preemption (Desktop))

Preemptible Kernel (Low-Latency Desktop)Enable cleancache driver to cache clean pages if tmem is presentTimer frequency (250 HZ)

1000 HZBus options (PCI etc.)

PCI Stub driverXen PCI Frontend

Device DriversBlock devices

Xen virtual block device supportXen block-device backend driver

Network device supportXen network device frontend driverXen backend network deviceEthernet driver support

QLogic devicesNetXen Multi port (1/10) Gigabit Ethernet NIC

Xen driver supportXen memory balloon driverDynamically self-balloon kernel memory to targetMemory hotplug support for Xen balloon driverScrub pages before returning them to systemXen /dev/xen/evtchn deviceBackend driver supportXen filesystemCreate compatibility mount point /proc/xenCreate xen entries under /sys/hypervisoruserspace grant access device driverUser-space grant reference allocator driverXen PCI-device backend driver

Firmware DriversEFI Variable Support via sysfs

For SSH UsersFor users using ssh, you will want to use screen, this way if your machine goes to sleep or your connection is lost while compiling the process will not halt in the middle.Starting screen couldn't be easier:

screen

End For SSH Users

Now we can clean the system and then compile our kernel:

make-kpkg cleanfakeroot make-kpkg --initrd --revision=3.4.9.custom kernel_image

Note that in later versions of gcc the revision must begin with a number.On my machine it takes roughly 20 minutes to compile a kernel, so feel free to take a break.Once the compilation is complete, we can back up a directory where we should now find a portable .deb installation package. I highly recommend making a backup of this file if youwant to reinstall that kernel without all this effort in the future:

cd ..sudo dpkg -i linux-image-3.4.9_3.4.9.custom_amd64.deb

At the end of the installation, it should automatically run update-grub, at which point you can reboot your system.Once the machine has rebooted and you are logged in, type uname -r and if the value it spits out is 3.4.9 (or whatever kernel version you compiled) then you are successful!Strap in, because the next segment will cover compiling the latest Xen from source.

References:

^ Bug 46341 - Xen PCI passthrough does not work on kernel 3.5.2 (https://bugzilla.kernel.org/show_bug.cgi?id=46341)1.

Kernel Source (http://kernel.org/)Amazingly Simple Kernel Compiling Guide (http://vanilja.org/kernel/)Kernel Flag Database for Lookup (http://cateee.net/lkddb/)Performance Regarding Paravirt Spinlocks (http://lists.xen.org/archives/html/xen-devel/2010-12/msg01317.html)

Compiling and Installing Xen (http://www.youtube.com/watch?v=pxK2mVDmeVY)I have great news, the latest Xen source, RC3 currently, was the first time I have been able to compile the source without debugging make errors since I started back in March.Before we proceed I should run through the fact that Xen can be effected by:

HardwareEnvironment (Dom0 OS)SoftwareXen SourceDependent Packages

Xen is a spiderweb of dependencies simply due to the number of facets it has to fill, and we have control over barely half of these major factors.My point is primarily that you can use the same hardware, platform, and revision I use and still encounter brand new errors due to changes in dependent packages downloaded duringcompilation. Be prepared.

The following set of packages is a bare-minimum set to compile and run Xen successfully, and omits well over a gigabyte of the recommended dependencies:

sudo aptitude install bridge-utils build-essential libncurses-dev python-dev uuid uuid-dev libglib2.0-dev libyajl-dev bcc gcc-multilib iasl libpci-dev mercurial flex bis

Note: libpci-dev replaces pciutils-dev (which routes to said package), just an FYI.These packages appear to be on the requirements list, but were already on my system at this stage. If you have not been following from the previous segment you may want to run thisfor good measure;

sudo aptitude install bcc patch libncurses5-dev python python-dev libglib2.0-dev bin86 bzip2 module-init-tools make gcc libc6-dev libcurl3 iproute xz-utils

GUI Xen PackagesFor those of us using Gnome or a GUI environment, you may want to include these packages for vnc access and/or sdl:

sudo aptitude install libsdl-dev gvncviewer

Obviously you have the choice to use only one, you do not need both, but if you intend to use the GUI environment in Debian I highly recommend you give SDL a spin, it's much fasterthan VNC, though VNC does offer remote access.End GUI Xen Packages

Complete The DependenciesThe list I already provided is a minimalists list, containing only exactly what was required to compile and run Xen for my own purposes, which includes PVHVM systems of Windows,FreeBSD, and Linux, as well as PCI Passthrough, and both VNC and SDL. It has worked for me since I started tinkering back in March, with minor modifications since.For those who want the whole package and have a spare gigabyte of OS Disk space available, I have put together this sub-section.I should also note that some of these packages will change flags marked as no from ./configure to a yes, but that did not stop me from compiling and running the system:

zlib1g-devlibbz2-devliblzo2-deve2fslibs-devocamlocaml-noxocaml-findliblzmalzma-devliblzma-devmarkdownlibcurl4-openssl-devtransfigtgiflibvncserver-devpython-twistedlibjpeg62-dev

gawkgit-coretexinfotexlive-latex-basetexlive-latex-recommendedtexlive-fonts-extratexlive-fonts-recommended

If anyone knows exactly what any of these dependencies do or are used for and can shed some light on them I would greatly appreciate it.End Complete The Dependencies

Next we can visit xenbits to get the latest revision. In our case we will use revision 25824[2] (http://wiki.xenproject.org/wiki/Comprehensive_Xen_Debian_Wheezy_PCI_Passthrough_Tutorial#endnote_rev25824)[3] (http://wiki.xenproject.org/wiki/Comprehensive_Xen_Debian_Wheezy_PCI_Passthrough_Tutorial#endnote_rev25784) , and use mercurial to download the source:

mkdir -p ~/src/xen && cd ~/src/xenrev=25824hg clone -r $rev http://xenbits.xen.org/hg/xen-unstable.hg/ xen-unstable.hg-rev-${rev}cd xen-unstable.hg-rev-${rev}

^ Original guide suggests using Xen 4.2-rc4-pre, revision 25777. Xen 4.2-rc4, revision 25824 has also worked for another user, User:V2. ^ Xen 4.2-rc4, revision 25784 compiled with an error for a potentially uninstantiated variable, which may suggest a gcc compatibility bug3.

EFI Source ModificationGiven that I encountered no make errors, I was shocked to find that they still hadn't addressed the grub efi compatibility bug where it fails to recognize available system memory.The fix for this is a source modification, though they have supposedly added a build process for xen.efi, which would replace the debian.efi generated by grub. I have yet to findadequate instructions to make that work, and even if I did may break adding xen-pciback.hide to the grub configuration, which we need if we are planning to do the easy method ofPCI Passthrough.So, let's open up xen/arch/x86/setup.c with our favorite text editor, and find the line containing ( e820_raw_nr != 0 ), and make that area look like this:

#if 0 else if ( e820_raw_nr != 0 ) { memmap_type = "Xen-e820";

References:

Xenbits Repositories (http://xenbits.xensource.com/)Latest Instructions (http://wiki.xen.org/wiki/Xen_4.2_RC2_test_instructions)4.1 Compilation Instructions (http://wiki.xensource.com/xenwiki/Xen4.1)4.0 Compilation Instructions (http://wiki.xensource.com/xenwiki/Xen4.0)David Techer's Blog (http://www.davidgis.fr/blog/)EFI Memory Error (http://serverfault.com/questions/342109/xen-only-sees-512mb-of-system-ram-should-be-8gb-uefi-boot)

Xen Configuration (http://www.youtube.com/watch?v=ea3IY2CHBaM)In the previous segment we had just finished compiling Xen to the dist/ folder, which is where we will go now to install it!

cd dist/sudo dpkg -i xen-upstream-4.2.0-rc4.deb

Our work is not done yet, we have a lot of post-install configuration to take care of before we can reboot and verify that Xen is working.Let's start with a trip to /boot to delete extra symbolic links and related files:

cd /bootsudo rm xen.gz xen-4.gz xen-4.2.gz xen-syms-4.2.0-rc4

Next we want to have Xen boot before the Linux Dom0 kernel, Xen will load that after. To do this we need to move the grub configuration file forward and update grub via:

sudo mv /etc/grub.d/20_linux_xen /etc/grub.d/09_linux_xensudo update-grub

Finally, before we reboot the system we need to make sure we initialize the Xen toolstack and related features at boot time via:

sudo update-rc.d xencommons defaults 20 19sudo update-rc.d xendomains defaults 21 22sudo update-rc.d xen-watchdog defaults 23 22

At this stage we reboot our system, and if all goes well the first option in grub will be Xen, which will bring us to our linux login.Our next step is logging in and testing the xl toolstack with sudo xl dmesg, if the output you receive is a not an error, then you have successfully installed Xen.Next we need to setup a virtual bridge for our virtual machines going forward. Here is a simple recommended configuration you can use to replace the contents of /etc/network/interfaces (See my PFSense Segment for a more complex multi-bridge configuration):

auto lo xenbr0iface lo inet loopbackiface eth0 inet manualiface xenbr0 inet dhcpbridge_ports eth0

To make this change take effect you can restart the network service, ifconfig, or take my approach and reboot.

Making Things EasierAs many are aware, security comes at the cost of usability.So this section is a way to make using Xen easier at the slight cost of some security.Add one of the following segments of code to the end of /etc/sudoers to omit entering a password when running the xl command:Option 1:

# Add no-password xl access%sudo ALL=(ALL:ALL) ALL, !/usr/sbin/xl, NOPASSWD: /usr/sbin/xl

Option 2:

# Cmnd alias specificationCmnd_Alias XL = /usr/sbin/xl# Allow members of group sudo to execute any command%sudo ALL=(ALL:ALL) ALL, !XL, NOPASSWD: XL

If using vi you may have to use :wq! to force write quit when saving.Next let's add an alias so whenever we type xl it assumes sudo xl by adding an alias to /etc/bash.bashrc:

# XL Alias alias xl='sudo xl'

Now when we login next instead of using sudo xl and entering a password, we can just use xl and no password is necessary!End Making Things Easier

This concludes Xen Configuration, we are ready to move onto setting up HVM Virtual Machines!I recommend checking the table of contents for details on managerial activities, which includes backing up and restoring, LVM and partitioning, and a few other helpful related activities.

References

Xen Man Pages (http://wiki.xen.org/wiki/Xen_Man_Pages)Outdated Document Mentions Xen Syms File (http://xen.xensource.com/files/xen_user_manual.pdf)

Installing Windows HVM (http://www.youtube.com/watch?v=s1Ro0KxHooQ)While you are welcome to follow along, I highly recommend you spend a few hours reviewing the Xen Man Pages.Our first step is preparing a Logical Volume partition for Xen.With Xen, this partition will be treated as a whole hard drive, and will create a sub-partition-table and partitions inside of it, hence you cannot simply mount the logical volume without anoffset going forward. For more information on this, check out the managerial tasks section of my documentation.I recommended Samba because there are drive limitations with Xen, if you intend to share a partition between multiple machines mounting with write permissions can ruin your data.Xen will allow multiple mounts, but neither system will inform the other of changes, hence conflicting read and write operations.Other disk limitations include only up to 4 hdX drives, which are treated as Parallel ATA, and only up to 15 sdX drives which are treated as SCSI can be passed at any time.You will need a Windows 7 installation DVD, I recommend a disk image copy, which we can use to install Windows 7.

For now we are using xenbr0 for our network bring, but when we set the vif in our configuration it must have a mac address with a prefix of 00:16:3e. For this HVM I will be using theemulated Intel e1000 gigabit ethernet adapter.Here is my Windows 7 configuration file for installation:

name='windows'builder='hvm'vcpus=4memory=6144disk=['/dev/mapper/xen-win2,,hda,w','/media/samba/docs/win7.iso,,hdc,r,devtype=cdrom']vif=['bridge=xenbr0,model=e1000,mac=00:16:3e:14:b1:1c']boot='dc'pae=1nx=1videoram=16stdvga=1sdl=1vnc=0usb=1usbdevice="tablet"localtime=1

Once Windows has been installed we can change boot='dc' to boot='c', and remove the iso from the drives.Changes to the configuration will not take effect until we have shut down the HVM and run the xl create command again.

PassthroughPassthrough is an important and moderately complex topic.First we should cover the basics of PCI devices in Linux.Linux recognizes devices using a BDF code, which stands for Bus, Device, Function. You will commonly see it in the format of BB:DD.f, and it is how we point to devices when wewant Linux to do something with them.To check your systems PCI device identifiers you can use lspci. This will print them in BDF format. You can get more details by adding the verbosity flag -v, you can add up to tripleverbosity, which is several pages of in formation.

We usually just need basic names to identify our devices, here is an example lspci from my ASRock Z68 Extreme4 Gen3 Motherboard:

00:00.0 Host bridge: Intel Corporation 2nd Generation Core Processor Family DRAM Controller (rev 09)00:01.0 PCI bridge: Intel Corporation Xeon E3-1200/2nd Generation Core Processor Family PCI Express Root Port (rev 09)00:02.0 VGA compatible controller: Intel Corporation 2nd Generation Core Processor Family Integrated Graphics Controller (rev 09)00:16.0 Communication controller: Intel Corporation 6 Series/C200 Series Chipset Family MEI Controller #1 (rev 04)00:1a.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #2 (rev 05)00:1b.0 Audio device: Intel Corporation 6 Series/C200 Series Chipset Family High Definition Audio Controller (rev 05)00:1c.0 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 1 (rev b5)00:1c.4 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 5 (rev b5)00:1c.5 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 6 (rev b5)00:1c.6 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 7 (rev b5)00:1c.7 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 8 (rev b5)00:1d.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #1 (rev 05)00:1f.0 ISA bridge: Intel Corporation Z68 Express Chipset Family LPC Controller (rev 05)00:1f.2 SATA controller: Intel Corporation 6 Series/C200 Series Chipset Family SATA AHCI Controller (rev 05)00:1f.3 SMBus: Intel Corporation 6 Series/C200 Series Chipset Family SMBus Controller (rev 05)01:00.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI Barts XT [Radeon HD 6800 Series]01:00.1 Audio device: Advanced Micro Devices [AMD] nee ATI Barts HDMI Audio [Radeon HD 6800 Series]03:00.0 SATA controller: Marvell Technology Group Ltd. 88SE9120 SATA 6Gb/s Controller (rev 12)04:00.0 USB controller: Etron Technology, Inc. EJ168 USB 3.0 Host Controller (rev 01)05:00.0 USB controller: Etron Technology, Inc. EJ168 USB 3.0 Host Controller (rev 01)06:00.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba)07:01.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba)07:04.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba)07:05.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba)07:06.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba)07:07.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba)07:08.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba)07:09.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba)08:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection09:00.0 PCI bridge: ASMedia Technology Inc. ASM1083/1085 PCIe to PCI Bridge (rev 01)0b:00.0 FireWire (IEEE 1394): VIA Technologies, Inc. VT6315 Series Firewire Controller (rev 01)0c:00.0 Ethernet controller: Broadcom Corporation NetLink BCM57781 Gigabit Ethernet PCIe (rev 10)

Unfortunately USB devices are not labeled on the board as they are seen in the lspci printout, but I have 2 of each type of controller, and which items are connected to whichcontroller is unknown. I had to spend some time testing which controller was which by passing one at a time to a virtual machine. When passed I could not use them in Dom0, so besure you have some form of remote control or a PS/2 keyboard and mouse as a backup.I tested my USB 3.0 ports in linux and had zero gains in speed, so I decided to pass both USB 3.0 controllers and the larger USB 2.0 controller to the pciback module for passthrough.I ended up with these five items as shown in BDF format:

00:1d.0 (USB 2.0)01:00.0 (Graphics Card Video Function)01:00.1 (Graphics Card Audio Function)04:00.0 (USB 3.0)05:00.0 (USB 3.0)

Notice that the graphics card has two functions, video and audio. If I pass one I have to pass both, I cannot pass just Video and keep audio accessible in Dom0. This creates a seriousproblem when you have extremely dense motherboards that have shared PCI Bridges with multiple functions leading to several individual components. It may be important to keep thatin mind when purchasing a motherboard, this is not information you will have until the board is installed and you are running the lspci command.Passing these devices to pciback essentially tells the Linux OS not to assign drivers and use or activate them. However if you run lspci after hiding these devices, you will still see them.There are two methods to using pciback, one is as a module but I have had no luck with that approach hence why we compiled a custom kernel. With a custom kernel my pciback isbuilt-in and I can modify my grub bootloader to automatically hide the devices.Find this section of your grub configuration in the file /boot/grub/grub.cfg:

submenu "Xen 4.2.0-rc4-pre" {menuentry 'Debian GNU/Linux, with Xen 4.2.0-rc4-pre and Linux 3.4.9' --class debian --class gnu-linux --class gnu --class os --class xen {insmod part_gptinsmod ext2set root='(hd0,gpt2)'search --no-floppy --fs-uuid --set=root 9b2dde6d-5888-429c-a6dd-35e7fa11f632echo 'Loading Xen 4.2.0-rc4-pre ...'multiboot /xen-4.2.0-rc4-pre.gz placeholderecho 'Loading Linux 3.4.9 ...'module /vmlinuz-3.4.9 placeholder root=/dev/mapper/xen-dom0 ro quietecho 'Loading initial ramdisk ...'module /initrd.img-3.4.9}

The line we want to modify starts with module, and here is what we add:

module /vmlinuz-3.4.9 placeholder root=/dev/mapper/xen-dom0 ro quiet xen-pciback.hide=(00:1d.0)(01:00.0)(01:00.1)(04:00.0)(05:00.0)

We will have to force write-quit to save the file (:wq!), then when we reboot the system our devices will be available for passthrough!Next we add this to our HVM Configuration File:

pci = ['00:1d.0','04:00.0','05:00.0','01:00.0','01:00.1']

It is important to note that you may NOT want to pass through devices until after you have installed and updated the virtual machine. I have encountered many problems with passingdevices during installation.

Important Note for Graphics PassthroughnVidia cards do not work out of the box, but ATI does.However, most cards do not support FLR (Function Level Reset), which is important because it prevents the card from operating properly without the right steps.Essentially, unlike a physical computer, when the virtual machine is shut down or restarted, it does not change the power supplied to the graphics card. This means the graphics card isnever re-initialized, so FLR was invented. However, because most graphics cards do not support FLR you have to mimic this effect by ejecting the card after every reboot, otherwiseyou will see a severe performance degradation.In addition, if you do not reset the card, and it is not fresh when you attempt to install or uninstall drivers, the process may fail leaving your system crippled, plus a BSOD is likely toappear.So, my recommendation when dealing with GPU drivers and passthrough, always reboot the entire machine, or take extra care to reset the card before making any changes.After the installation, if you reboot the HVM you can use the Eject USB Media icon in the taskbar at the lower right hand corner of the screen to eject the card which will attempt amanual reset. You will loose video for a few seconds as the card re-initializes. This should fix performance on reboot.

PV on HVM Drivers are awesomeThey can improve network and hard drive performance by a substantial amount. In Windows these are referred to as GPLPV Drivers.To my understanding they bypass the emulated hardware layers reducing the latency between operations.My Windows Experience Index with a SSD went from 6.8 for hard drive to 7.8 by installing these drivers.Supposedly to enable PV on HVM you have to add this line to your HVM Configuration:

xen_platform_pci=1

The final configuration after my devices are passed and PVHVM Drivers are installed looks like this:

name='windows' builder='hvm' vcpus=4 memory=6144 disk=[ '/dev/mapper/xen-win2,,hda,w' ] vif=[ 'bridge=xenbr0,model=e1000,mac=00:16:3e:14:b1:1c' ] pci = [ '00:1d.0', '04:00.0', '05:00.0', '01:00.0', '01:00.1' ] boot='c' pae=1 nx=1 videoram=16 stdvga=1 #sdl=1 vnc=1 vnclisten="0.0.0.0:10" vncpasswd='password' usb=1 usbdevice="tablet" localtime=1 xen_platform_pci=1

If you do not have ntp setup on the host machine (Dom0) do so immediately. Setting localtime=1 should use the correct host time, without it Windows clock will drift at boot andduring runtime by a substantial amount.Note that as unimportant as clock time seems, if it drifts far enough your cookies, sessions and SSL certifications will disappear, become invalid, and the operating system itselfbecomes absolutely unstable. For example all of my taskbar icons disappeared and were repalced with blank spaces, including my eject USB Media icon. It was a scary experience.

Since I use this system for recreational purposes, I have posted a performance demonstration video that shows HDD, SSD, and USB data transfer speeds, as well as graphicsperformance with video games.

Windows Performance Demonstration Video (http://www.youtube.com/watch?v=Nz2c0Up2axk)

References

Xen Man Pages (http://wiki.xen.org/wiki/Xen_Man_Pages)

Installing PFSense HVM (http://www.youtube.com/watch?v=Nz2c0Up2axk)WORKING ON THIS STILLI have been re-using an imaged backup of a former configuration of PFSense for the last several installations, so I have not created updated documentation on installing and configuringPFSense yet.I hope to have this completed within the next few weeks, and a video to match shortly after.

Managerial ActivitiesThere are a few tricks I can share with regards to managing Xen virtual machines.However, having an understanding of the fundamentals is also helpful.Here are the key areas:

Drive PartitioningLVMUsing mount With LVM Virtual Partitionsdd

Drive Partitioning

This is a fundamental, you are welcome to skip ahead if you want the how and not the why.

I find a lot of emails floating around the xen-users email list due to misunderstandings in how Xen recognizes and managed partitions.It is important to understand how Xen Virtual Machines treat partitions.It sees them as entire hard drives, and will create a brand new partition table and partitions inside of it.The result is sub-partitioning, and this causes an offset between the start of the actual partition and the start of the sub-partition containing all of the data on the virtual machine.

In more detailComputers typically segment hard drives into three components.

Partition TablesPartitionsFile Systems

A partition table tells the disk where the beginning and end of a partition exists and usually the file system.The Partitions are simple blocks of storage.The File System tells the operating system how to access the partition.A normal installation of Windows for example, will look like:

Drive Block 0Partition Table:

Starts at Block 256 File System NTFSDrive Block 256

PartitionData

LVM

Linux has LVM handle all the tough parts of managing the positions of data blocks, and makes everything else easier for the user.Hence it is called the Logical Volume Manager, because it manages logical volumes.What this means is when we create a LVM Partition, we are actually using an LVM "File System", which looks like this:

Drive Block 0Partition Table:

Starts at Block 256 File System LVMDrive Block 256

PartitionData

We can then create partitions using LVM which automatically handles the location of the data inside the LVM Partition.For example, we can create two LVM Partitions for a Windows and Linux Virtual Machine, which would look like this:

Drive Block 0Partition Table:

Starts at Block 256 File System LVMDrive Block 256

PartitionData

LV WindowsLV Linux

Only an operating system that recognizes LVM can see "LV Windows" or "LV Linux" though, so if you plug that drive into a basic Windows machine, it won't be able to do anything withthe data.

Finally we have XenWith Xen we give one of those partitions to our virtual machine, but as stated it sees that as a hard drive.If we use the previous drive configuration and just add a Windows Virtual Machine to LV Windows our configuration looks like this:

Drive Block 0Partition Table:

Starts at Block 256 File System LVMDrive Block 256

PartitionData

LV WindowsLV Block 0

Partition TablePartition at Block 256 File System NTFS

LV Block 256Partition

DataLV Linux

At this point, we cannot simple access the data on our Xen partition because it has been sub-partitioned.If you try simply using mount it will fail, because it requires the offset to the start of the NTFS sub-partition!

Using mount With LVM Virtual Partitions

To get the offset of the partition, we will use a tool like fsdisk or parted:

sudo parted /dev/mapper/xen-windows unit B print

Using unit B we indicate bytes which will be used by the mount command, and we get this:

Number Start End Size Type File system Flags1 1048576B 105906175B 104857600B primary ntfs boot2 105906176B 171796594687B 171690688512B primary ntfs

Important to note that Windows 7 creates a 100MB Boot Partition, so the first item with the boot flag is actually the boot partition, we want the Start value of the second, which is105906176B.We can now mount the partition using:

sudo mount -o offset=105906176 /dev/mapper/xen-windows /path/to/mount/dir

That is how we can access the data inside our sub-partitions.Obviously it is ill-advised to access the data while the operating system is running, mounted disks will NOT inform the running platform of changes, so data can get overwritten andcause all kinds of problems.

dd

The utility dd is probably one of my favorites among the tools that come with most linux/unix systems.It can be used to create an exact copy of data, and is not subject to a number of limitations with higher level software.It is not the fastest utility, but supplemented with tools like gzip you can compress a complete backup into a size similar to that you might see with professional software.Here is a backup Example that compressed an 80GB Windows LV with 70GB of Data into a 43GB GZipped Image File:

sudo dd if=/dev/mapper/xen-windows of=/path/to/backup/windows.img bs=1Msudo gzip -9 windows.img

Alternative one-liner (Takes a lot longer since it has to compress "on the fly"):

sudo dd if=/dev/mapper/xen-windows bs=1M | sudo gzip -9 > /path/to/backup/windows.img.gz

I can also use this command to restore my system from that compressed image:

sudo gzip -d /path/to/backup/windows.img.gzsudo dd if=/path/to/backup/windows.img of=/dev/mapper/xen-windows bs=1M

Once again, one-liner alternative:

sudo gzip -dc /path/to/backup/windows.img.gz | sudo dd of=/dev/mapper/xen-windows bs=1M

There are plenty of other uses I have found for the dd command, but backups and restoration make Linux amazingly easy to restore after terrible failure, and believe me in writing thisguide I encountered that plenty of times.

You've reached the end of my guide!Thanks for taking your time to read through my documentation, if you found it at all helpful please feel free to shoot me an email on the Xen Users email list.Any feedback or corrections are also welcomed.I am still working on the PFSense segment and will have that up as soon as I can.

Retrieved from "http://wiki.xen.org/wiki/Comprehensive_Xen_Debian_Wheezy_PCI_Passthrough_Tutorial"

2013 Xen Project, A Linux Foundation Collaborative Project. All Rights Reserved.

Linux Foundation is a registered trademark of The Linux Foundation.

Xen Project is a trademark of The Linux Foundation.

Categories Xen Users Tutorial Debian VGA

This page was last modified on 28 May 2013, at 07:44.

Xen.org's servers are hosted with RackSpace, monitoring ourservers 24x7x365 and backed by RackSpace's Fanatical Support.