Sandrino is correct, it is not officially supported yet, and will not be until the framework is provided on the images used for the Cloud Services. If you ware looking for a clean solution only, then you'll need to wait; however, I don't think you'll wait too long.
I have seen no official release date announced; however, in a comment left on Jason Zander's Blog about the release of VS 2012 there is this response made by Scott Hanselman:
James, you should see Windows Server 2012 in Azure VM's shortly after
the RTM is released, followed by Cloud Services and then Web Sites. It
is coming soon.
In the past when we saw the new .NET Frameworks ship they were available on Azure within a few months (like less than 90 days if I recall correctly).