How about using fake field input for entering text and hiding datepicker/form field bellow it so that date picker opens in right place (it will open bellow field which has
[matDatepicker]).
You should listen for blur event on display input field and clear form field value if it's invalid.
Main issue is that new Date(number) is a valid date. Date picker will convert your input to date and emit it as new field value. Custom logic can be performed on blur event, probably with moment.js to Check if a string is a date value.
In case of it being invalid real form field value should be set to null.
Also display field value should be updated when date is picked via datepicker or to display initial values.
<mat-form-field [formGroup]="dateFormGroup">
<input matInput (blur)="onBlurEvent($event)" placeholder="Choose a date">
<input type="hidden" formControlName="date" [matDatepicker]="picker">
<mat-datepicker-toggle matSuffix [for]="picker"></mat-datepicker-toggle>
<mat-datepicker #picker></mat-datepicker>
</mat-form-field>
This logic should be wrapped in a custom form control or at least in a dedicated component because it's modular and having it in multiple places will cause many bugs.