I'm aware that this has been asked and answered several months ago, but I was looking for an answer to the same problem today when I found this.
Although Sven's answers didn't help directly, option #2 gave the final hint to my solution. Maybe it can be of use for others, too:
First of all, my page uses a standard button (not a button of type "Submit" as I need to set some hidden fields along with the editable ones). So, before the final saving is done I added this script to my button code:
var numAtts = myDocDatasource.getAttachmentList("Body").size();
if(numAtts == 0){
var msg = new javax.faces.application.FacesMessage("You need to attach a file");
facesContext.addMessage("File validation error", msg);
return false;
}
//do some more stuff
...
myDocDatasource.save();
I had to realize that the content of the fileUpload control doesn't really matter when it comes to validation as at that stage of the process an uploaded file already is part of the datasource.
The "timing" of this validation step is a bit surprising, though: at least in my situation, validation of other fields is done before the file upload is validated:
in an errorMessages control first only the standard validation errors are listed. Only after all the other fields have been validated successfully my fileUpload validator is displaying its error.