Issue resolved. In this case the site will eventually move to .NET and in the transition will be a mix of classic asp and C#.NET code, so the site is setup to handle both in IIS.
In IIS there's two sections of Error Pages configuration for custom error handling (in the ASP.NET area and the IIS area). Classic asp runtime errors (for instance, an array subscript error) will be handled by the Error Pages setup in the IIS section. But bad URL handling still gets handled by the .NET engine, so the example of www.mysitename.com/< gets handled by the ASP.NET Error Pages configuration.
So the fix was in the ASP.NET area, Error Pages -> Edit Feature Settings and setting a Default Page to a custom page did the trick (or alternately getting more granular by error type), and setting the Mode to Remote Only (or simply On, if desired).