Memory leak/constant high CPU usage when running Phoenix in IIS on Windows

Question

I am trying to prototype a small web application to see if I can use Phoenix and Elixir to build some intranet web applications for use by my co-workers at my company. We've been a typical Microsoft .NET shop forever, but I'm interested in playing with some other technologies, with Elixir and Phoenix being two of them.

I have a virtual machine running Windows 10 and IIS 8. I installed HttpPlatformHandler on the VM as well to run the Phoenix application and pass requests to it from IIS (there's a reason for this, which I'll get to in a moment).

When I run my Phoenix app standalone on Windows, it runs great. I don't see any issues. When I run my Phoenix app using HttpPlatformHandler and IIS, I see the erl.exe process constantly running and consuming about 35-40% of the CPU (I'm running with 2 cores and 8GB RAM on the VM). I also see the memory constantly growing without stopping. After a couple of minutes, it will blow past 1GB and move on its way to 2GB (I'm watching memory usage using Process Explorer). There's no load or active requests on the Phoenix app. This happens in both dev and prod environments.

I want to run Phoenix behind IIS because it's an intranet application and it will be running on a Windows domain and most of the users will be Windows users. I created a plug that will read the Windows user token from the X-IIS-WindowsAuthToken HTTP header that HttpPlatformHandler adds to the requests that it forwards to the Phoenix process, and I'm using a set of NIFS to call the Windows API to get the Windows SID and domain\username of the authenticated user in order to establish the session and authorize the user.

Has anyone tried this before and seeing the same issue? I'm relatively new to Elixir and Phoenix debugging and would love some hints on what I can do to try to figure out why the Erlang runtime is doing this, or how to determine if it's something that I'm doing in my code.

I'm not sure that it's me, because I see this happening on a plain Phoenix application that is freshly generated using mix phoenix.new.

For reference, here's my web.config for my Phoenix application:

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <system.webServer>
    <handlers>
      <add name="httpplatformhandler" path="*" verb="*" modules="httpPlatformHandler" resourceType="Unspecified"/>
    </handlers>
    <httpPlatform processPath="C:\Projects\neucode\run.bat"
        arguments=""
        stdoutLogEnabled="true"
        forwardWindowsAuthToken="true">
      <environmentVariables>
        <environmentVariable name="MIX_ENV" value="prod"/>
      </environmentVariables>
    </httpPlatform>
  </system.webServer>
</configuration>

My run.bat file:

mix phoenix.server

I greatly appreciate any assistance or suggestions that anyone can provide. Please let me know if there's anything else that I can provide that would be helpful in diagnosing the problem.

Thank you in advance.

UPDATE

I figured out how to connect to my Phoenix application running in IIS using Observer thanks to this gist. It looks like the problem process is the user process:

I started tracing on this process and the port that it is linked to and I see a continual stream of what I am guessing are receive events with empty data:

I'm guessing from looking around that this is the kernel user process. Does anyone know how I can figure out what that port is and why it's sending a continual stream of events?

I would be a bit surprised if that combination ever works since Phoenix is a web server just as is IIS. — Onorio Catenacci
@OnorioCatenacci Why shouldn't it work? It's just like using NGINX or Apache as a proxy server. It does work though. The only thing that I'm seeing is the weird CPU and memory issue. But the web application itself works without issues. — Michael Collins
I would just say that wrapping Phoenix with IIS so you can get AD authentication (which is what it sounds like you're trying to do) strikes me as rather a kludge--regardless of viability or the lack thereof. — Onorio Catenacci
@OnorioCatenacci It's one of the reasons, but certainly not the only one and I wouldn't go so far as to criticize it as a kludge. That's a little bit inappropriate, in my opinion. It's a fair use case, especially where Windows in the target platform, and Phoenix shouldn't have any issue running like this. I'd point out that they document using this configuration with NGINX and Apache in the guides here. In terms of Windows authentication, I'd be very open to any alternative approaches if you have any. — Michael Collins
I think the only non-hacky/non-kludgy solution in this case would be to write a NIF for AD authentication. IIS and Phoenix both seem to serve the same purpose (as does Nginx and Apache) hence it seems a bit kludgy to combine them. — Onorio Catenacci

Michael Collins Michael Collins · Accepted Answer · 2017-03-13T04:04:28

I figured out the solution to this. This is where my newbie status to Elixir and Erlang came into play. When you run an Elixir/Erlang application, the Erlang user server gets started to handle standard I/O for the application. On the second image in my question update, you can see that port <10014.664> controls "0/1". My understanding now is that basically means 0 == standard input and 1 == standard output.

The issue when running with HttpPlatformHandler is that the Phoenix application is running without a shell and there would never be anything coming in via standard input. So when you run using the default settings, standard input basically floods the user server with a bunch of empty data messages.

The solution is to tell the Erlang VM and the user server to ignore standard input. In my run.bat script, I was running mix phoenix.server, but based off the gist that I used to name the node so that I could attach Observer to it, I changed my run.bat script to this:

elixir --name [email protected] --cookie PUT_COOKIE_HERE --erl "-noinput" -S mix phoenix.server

The -noinput flag appears to tell the Erlang VM to never read from standard input. With -noinput passed to the command line, the memory issue went away and CPU usage is typically below 0.2% while idle. CPU and memory appear to be the same as if the Phoenix application were running from a console outside of IIS.

Memory leak/constant high CPU usage when running Phoenix in IIS on Windows

1 Answers