RDP VirtualBox without the proprietary Oracle extension pack

These days, virtualisation is all the rage. The various competing virtualisation products have reached a level of maturity where they can be reliably used for server consolidation. VirtualBox is one of the easiest to use, most featureful programs available in this space and with the ability to run on many different OSes on hardware with or without VM extensions, it is also one of the most popular. However, there is one wrinkle when it comes to using it for server consolidation: the proprietary RDP/USB2 extension pack.

The conventional wisdom when running a headless server with VirtualBox is that you need to install this proprietary extension pack from Oracle. This is fine until you want to use the server in production: as the PUEL only covers you for personal use and evaluation, you must purchase licenses. You can either pay £34 per user or £670 per “socket” (which has quite a convoluted definition). This gets you USB2 and RDP support.

However, there is another way, at least when it comes to RDP support. Chapter 7 of the VirtualBox manual covers running virtual machines on a remote host, either with VBoxManage command (which offeres similar functionality to the VirtualBox GUI) or the VBoxHeadless command, which appears to be the backend binary that VBoxManage calls out to to do the actual work. Now, chapter 7 covers in detail how to connect to remote VMs using the proprietary extensions, but there’s one feature that they conveniently failed to mention: the built-in VNC server.

man VBoxHeadless
VBOXHEADLESS(1)            User Commands             VBOXHEADLESS(1)

NAME
       VBoxHeadless - x86 virtualization solution

DESCRIPTION
    Oracle  VM  VirtualBox Headless Interface (C) 2008-2011 Oracle
    Corporation All rights reserved.

Usage:
    -s, -startvm, --startvm <name|uuid>
           Start given VM (required argument)
    -n, --vnc
           Enable the built in VNC server
    -m, --vncport <port>
           TCP port number to use for the VNC server
    -o, --vncpass <pw>
           Set the VNC server password
    -v, -vrde, --vrde on|off|config
           Enable (default) or disable the VRDE  server  or  don't
           change the setting
    -e,  -vrdeproperty,  --vrdeproperty  <name=[value]>
    Set a VRDE property:
           "TCP/Ports" - comma-separated list of  ports  the  VRDE
           server can bind to. Use a dash between two port numbers
           to specify a range "TCP/Address"  -  interface  IP  the
           VRDE server will bind to
    -c, -capture, --capture
           Record the VM screen output to a file
    -w, --width
           Frame width when recording
    -h, --height
           Frame height when recording
    -r, --bitrate
           Recording bit rate when recording
    -f, --filename
           File  name when recording.  The codec used will be cho‐
           sen based on the file extension

VBoxHeadless                January 2011             VBOXHEADLESS(1)

Right! So we can start a virtual machine and forward its root console (or main video or out-of-band console or whatever you want to call it) over the network with a VNC server. In my example I’ll use screen to keep my VMs running when I log out. I’ll start a couple of VMs as an example:

screen
VBoxHeadless --startvm 'Ubuntu' --vnc --vncport 5900
# hit "ctrl-a c" to open a new terminal within screen
VBoxHeadless --startvm 'XP1' --vnc --vncport 5901 --vncpass vnc2xrdp

Great! We don’t need the proprietary expansion pack to do this. However, there are a few drawbacks to this approach:

• VNC sucks over slow network connections, it’d be much nicer to use the more modern RDP protocol
• You have to keep the command running so you have to use screen or nohup or equivalent, which is a bit less convenient than using VBoxManage
• If the VNC server crashes (which I’ve had happen only once so far in testing, when changing screen resolution in a Windows XP guest), the whole VM goes down with it
• You need to make sure you only bind one VNC server to each port. As far as I can tell if you try to bind a second one to a port that’s already in use, the VM still starts up but you have no way of interacting with it!

We can actually work around the first limitation, by using the xrdp program to “translate” the VNC protocol into RDP. I’m on Ubuntu, so I have the luxury of installing xrdp the easy way. On the VirtualBox server machine:

sudo apt-get install xrdp

Now we configure it to use the existing VNC servers that we previously spawned with VBoxHeadless. One neat thing here is that xrdp uses a single RDP port to manage multiple VNC connections:

/etc/xrdp/xrdp.ini

[globals]
bitmap_cache=yes
bitmap_compression=yes
port=3389
crypt_level=low
channel_code=1

[xrdp1]
name=VBox-Ubuntu
lib=libvnc.so
ip=127.0.0.1
port=5900

[xrdp2]
name=VBox-XP1
lib=libvnc.so
username=
password=vnc2xrdp
ip=127.0.0.1
port=5901

So we have two VirtalBox VMs running, Ubuntu and XP1. In the example above, I started the ubuntu VNC without a password, so I’ve left out the username/password entries. The XP1 connection is protected with the password vnc2xrdp. You can also use the value ask and xrdp will prompt for a username/password for connecting to VNC. Note that VNC passwords are generally insecure, so it’d probably be best to protect the vnc ports using firewalling. It doesn’t appear to be possible to bind the VNC server to only the loopback device (at least from the man page above).

So now all that’s left to do is to connect to the RDP port using one of the myriad RDP clients for linux (I’m using Remmina, but there are heaps of options). You can then choose the VNC connection you want xrdp to connect to and you’re away!

xrdp login screen

Of course, SSH local port forwarding is your friend if you’re doing any of this through firewalls or over insecure connections. Remmina actually includes this functionality, or you can forward the remote port to your local machine with something like:

ssh -L 3389:localhost:3389 vboxservermachine

So there you are, RDP connections to remote VirtualBox VMs without the proprietary Oracle extensions; all free software!

Have fun. 🙂