I can never find the exact answer I’m looking for, so I try to write it down when I finally figure it out. This episode? How to get a fast VNC server on Ubuntu 6.10.
First, get the right packages

% sudo apt-get install vnc4server

This gets you a ‘vnc’ module for your xorg X server, which can export an X session as a vnc host, and also let you use it on your local display as well (unlike Xvnc and vncserver commands).
Once you’ve installed the packages, you need to edit your /etc/X11/xorg.conf. In your “Module” section, add the line:

Load "vnc"

In your “Screen” section, add the line:

Option "PasswordFile" "/root/.vnc/passwd"

Now you need to create the actual password file. You need to switch to root and run the realvncpasswd utility:

% sudo su
% realvncpasswd
(enter password)

You should be all set. Restart your X server and give it a shot.