I think you are using the same serial port on your host and guest OS's.
If yes, you can't.
Or you bind the guest OS serial port to the real serial port and use another serial port on your host (connected by a null modem cable), or you have to use sockets on the host. This socket is created by VirtualBox ( http://www.virtualbox.org/manual/ch03.html#serialports):
On a Mac, Linux or Solaris host, a local domain socket is used
instead. The socket filename must be chosen such that the user running
VirtualBox has sufficient privileges to create and write to it. The
/tmp directory is often a good candidate.
On Linux there are various tools which can connect to a local domain
socket or create one in server mode. The most flexible tool is socat
and is available as part of many distributions.
In this case, for on the Linux host, you won't be able to use PySerial.
But on the guest OS, PySerial will work fine, as to the guest OS the virtual serial port will look like a real one.