You can split this into three components (fraction of first day, full workdays, fraction of last day)
Lets start with the middle part. Here you can use NETWORKDAYS and subtract the start and end date. I am assuming a start date in A1 and end date in B1. In order to exclude holidays you need to maintain a list of holidays in your sheet. The formula assumes that this list is in range C1:C10. The results is multiplied by 10 as there are 10 hours in your workday.
=MAX((NETWORKDAYS(A1,B1,C1:C10)-NETWORKDAYS(A1,A1,C1:C10)-NETWORKDAYS(B1,B1,C1:C10))*10,0)
For the fractions you will need to determine if the day itself is a holiday, we use the NETWORKDAYS function again as a factor which will be either 0 or 1. Now we only need to determine the hours to add for the day. Depending on the granularity you want you can consider hours, minutes or even seconds. I will use hours and minutes as a fraction (minutes/60 = hours).
For the first day you get
=MAX(17-MAX(HOUR(A1)+MINUTE(A1)/60,10),0)*NETWORKDAYS(A1,A1,C1:C10)
For the last day you get
=MAX(MIN(HOUR(B1)+MINUTE(B1)/60,17)-10,0)*NETWORKDAYS(B1,B1,C1:C10)
Putting it all together leaves us with:
=MAX(17-MAX(HOUR(A1)+MINUTE(A1)/60,10),0)*NETWORKDAYS(A1,A1,C1:C10)+MAX((NETWORKDAYS(A1,B1,C1:C10)-NETWORKDAYS(A1,A1,C1:C10)-NETWORKDAYS(B1,B1,C1:C10))*10,0)+MAX(MIN(HOUR(B1)+MINUTE(B1)/60,10)-10,0)*NETWORKDAYS(B1,B1,C1:C10)