To fit into the XPages & JSF validators and converters model, you would put your java code for format validation into a converter, and after doing the format validation, it should convert the submitted String to a java.util.Date. If the converter fails it should add a FacesMessage with the error message (there are blog posts with details - search FacesMessage).
[FYI, for date conversion there are many subtle gotchas around different date formats used in different countries, and around behavior when the browser and the server are in different timezones, and around handling daylight savings time in the different timezones.]
The converted Date object would then be made available to the validator (instead of the original string).
You can use a xp:validateExpression to call your Java date range validation. If you return false from the xp:validateExpression expression, it will use the error text provided in the message property. Else if you want to use different error messages use the FacesMessage mechanism instead of returning false.
Here's an example of a custom date date range converter:
<xp:validateExpression
message="The end date must be after the start date.">
<xp:this.expression><![CDATA[#{javascript:
var startDate = getComponent('inputText1').getValue();
if( null == startDate ){
return true;
}
if( value.before(startDate) ){
return false;
}
return true;}]]></xp:this.expression>
<xp:this.clientScript><![CDATA[
function validate(){
var startDate = dijit.byId('#{id:inputText1}').get('value');
var endDate = dijit.byId('#{id:inputText2}').get('value');
if( null == startDate || null == endDate){
return true;
}
if( endDate.getTime() < startDate.getTime() ){
return false;
}
return true;
}
validate()
]]></xp:this.clientScript>
</xp:validateExpression>
</xp:this.validators>