Elixir/Erlang file_server message backlog and unreliable throughput causing performance issues

2
votes

I'm running a production app that does a lot of I/O. Whenever the system gets flooded with new requests (with witch I do a ton of IO) I see the Erlang file_server backing up with messages. The backup/slowdown can last hours depending on our volume.

enter image description here

It's my understanding that a lot of File calls actually go through the Erlang file_server. Which appears to have limited throughput. Furthermore, When the message queue gets backed up the entire app is essentially frozen (locked up) and it cannot process new IO requests at all.

All of the IO calls are using the File module. I've specified the [:raw] option everywhere that will allow it. It's my understanding that passing in :raw will bypass the file_server.

This is a really big issue for us, and I imagine others have run into it at some point. I experimented with rewriting the IO logic in Ruby witch resulted in a massive gain in throughput (I don't have exact numbers, but it was a noticeable difference).

Anyone know what else I can look at doing to increase performance/throughput?

Sample Code:

defmodule MyModule.Ingestion.Insertion.Folder do
  use MyModule.Poller
  alias MyModule.Helpers

  def perform() do
    Logger.info("#{__MODULE__} starting check")

    for path <- paths() do
      files = Helpers.Path.list_files(path, ".json")

      Task.async_stream(
        files,
        fn file ->
          result =
            file
            |> File.read!()
            |> Jason.decode()

          case result do
            {:ok, data} ->
              file_created_at = Helpers.File.created_time(file)
              data = Map.put(data, :file_created_at, file_created_at)
              filename = Path.basename(file, ".json")
              :ok = MyModule.InsertWorker.enqueue(%{data: data, filename: filename})

              destination =
                Application.fetch_env!(:my_application, :backups) <> filename <> ".json"

              File.copy!(file, destination)
              File.rm!(file)

            _err ->
              nil
          end
        end,
        timeout: 60_000,
        max_concurrency: 10
      )
      |> Stream.run()
    end

    Logger.info("#{__MODULE__} check finished")
  end

  def paths() do
    path = Application.fetch_env!(:my_application, :lob_path)

    [
      path <> "postcards/",
      path <> "letters/"
    ]
  end
end
2
erlangelixirerlang-otp
I suspect you are encountering a bottleneck introduced by the configuration. All other things equal, I would expect better throughput and handling in Elixir, but you can still be hitting bottlenecks caused by things like the limits in the host OS (e.g. allowed number of file handles). Can you share some of the actual code? Preferably something that can paint a picture of your pipeline. – Everett
In the first place, one should not use synchronous calls in OTP for time-consuming operations in general. Offload IO to another process[es] and send asynchronous messages to it to unfreeze the main application. Check how logger backends writing to files use gen_event to offload IO, for instance. It’s impossible to tell something more specific without seeing the actual code, but you might want to offload IO to several different nodes, or like. – Aleksei Matiushkin
Thanks for the replys I edited the post with a sample of code, witch will 1. grab all files in a folder 2. read the file, and push it into a Rabbitmq queue 3. delete the file I have many other processing doing similar things (watching folders, and ingesting files that get written to them) – AndrewCottage

2 Answers

1
votes

Consider tunning the VM with async_threads

0
votes

For anyone finding this in the future. The root of the problem comes from using File.copy! with path names. When you do this, the copy will go through the Erlang file_server witch was the cause of a massive bottleneck that was extremely difficult to diagnose. Instead of using File.copy/1 with path names, use open files as the input. Like this

source = "path/to/some/file"
destination = "path/to/destination/"

with {:ok, source} <- File.open(source, [:raw, :read]),
     {:ok, destination} <- File.open(destination, [:raw, :write]),
     {:ok, bytes} <- File.copy(source, destination),
     :ok <- File.close(source),
     :ok <- File.close(destination) do
  {:ok, bytes}
end