After examining the source code, I found there is no way to not have the issuer field, even though the standard (RFC7519) has it as optional. I resolved the issue by creating a subclass of JwtSecurityTokenHandler, and override the method ValidateToken() to fix up the issuer there. Next, I handed it over to base.ValidateToken().
I've also gotten the question from my peers why I want to change or remove the issuer from the jwt token, and how this doesn't impair the security aspect. To answer that question: I'm only planning on using public/private key certificates to sign the jwt tokens, who already have an issuer tied to them. Checking the issuer again would be doing some work twice.
UPDATED ANSWER
The System.Identitymodel.Tokens.Jwt has been updated to v5, and I strongly recommend to update to this version. It fixes the Issuer issue. A downside of this update is that it has some more dependencies (Microsoft.Identitymodel.Tokens, Microsoft.Identitymodel.Logging, Newtonsoft.Json.Linq) and having to retweak your code to a slighty different interface.