I'm not sure this is practical to test headless. how could anything headless prompt the user? I'd think you'd have to tell it where to send the file. This might be more of a question for the webdriver folks, that's their underlying functionality. You might just have to call it directly.
does the site implement a REST interface or a way to just get the file directly? Maybe try cURL or REST-CLient?
If the main purpose is just to download the file, then your first challenge is to see if you can figure out what the URL of the file is from the webpage code. If you can determine that, then have a look at the cURL gem you could use it (or just cURL from the command line) to invoke a get against the HTTP address for the file download, then save it to a file or work with it in memory as needed