To fit into the XPages & JSF validators and converters model, you would put your java code for format validation into a converter, and after doing the format validation, it should convert the submitted String to a java.util.Date. If the converter fails it should add a FacesMessage with the error message (there are blog posts with details - search FacesMessage).
[FYI, for date conversion there are many subtle gotchas around different date formats used in different countries, and around behavior when the browser and the server are in different timezones, and around handling daylight savings time in the different timezones.]
The converted Date object would then be made available to the validator (instead of the original string).
You can use a xp:validateExpression to call your Java date range validation. If you return false from the xp:validateExpression expression, it will use the error text provided in the message property. Else if you want to use different error messages use the FacesMessage mechanism instead of returning false.
Here's an example of a custom date date range converter:
<xp:validateExpression
message="The end date must be after the start date.">
<xp:this.expression><![CDATA[#{javascript:// Server-side validation.
// value is the end date java.util.Date object
var startDate = getComponent('inputText1').getValue();
if( null == startDate ){
// no start date - this error message not applicable, validation pass.
return true;
}
if( value.before(startDate) ){
// failed validation
return false;
}
return true;}]]></xp:this.expression>
<xp:this.clientScript><![CDATA[// client-side validation (browser JavaScript)
function validate(){
var startDate = dijit.byId('#{id:inputText1}').get('value');
var endDate = dijit.byId('#{id:inputText2}').get('value');
if( null == startDate || null == endDate){
return true; // this error message not applicable, passed.
}
if( endDate.getTime() < startDate.getTime() ){
return false; // failed validation
}
return true;
}
validate()
]]></xp:this.clientScript>
</xp:validateExpression>
</xp:this.validators>