If you can encrypt a valid ticket in your Classic ASP code in such a way that the ASP.NET FormsAuthentication module can decrypt it, then a user with such a ticket (stored in a cookie or in the URL) will be considered authenticated in the ASP.NET request pipeline.
Here is the documentation on how to manually encrypt using the static FormsAuthentication.Encrypt
method:
http://msdn.microsoft.com/en-us/library/system.web.security.formsauthentication.encrypt.aspx
This illustrates that the ticket is really nothing special, basically the user name, expiration date. Note that you also store some extra user data in the ticket.
Of course, that still doesn't give us the actual encryption algorithm. This depends on the configuration in web.config, specifically the <machineKey>
-element. You should explicitly set the decryptionKey attribute, because you need to use the same key in your Classic ASP code.
MSDN: machineKey element
And you should also set the decryption attribute so that you can use the same algorithm in your Classic ASP code (like DES for example). If necessary you can create a custom decryption algorithm if you can's support the built-in ones in your Classic ASP code.
You would end up with a configuration like this:
<machineKey decryptionKey="your key here" decryption="DES" />
That is how I would implement it.