You do not have to rename your metric. CloudWatch doesn't really care about your unit all that much. From CloudWatch Concepts doc:
Units help provide conceptual meaning to your data. Though CloudWatch attaches no significance to a unit internally, other applications can derive semantic information based on the unit.
Units are not tied to a metric, they are tied to each specific datapoint. In your case datapoints will have unit Count up to one point and Milliseconds from that point on. This is not a problem as long as you don't use GetMetricStatistics API with a specified unit. If you don't specify a unit, GetMetricStatistics will return data for both units, but if you set a unit, you'll only get the data for the specified unit (see the API docs for more detail).
CloudWatch Dashboards will display the last unit they see. In the example below, I was publishing the metric with unit Milliseconds up until the gap and with unit Count after the gap (opposite of your situation) and the dashboards are displaying the Count as unit:
![enter image description here](https://i.stack.imgur.com/9RHI0.png)
If I change the range of the graph to exclude the datapoints with Count unit, graph displays Milliseconds as unit:
![enter image description here](https://i.stack.imgur.com/0AQuY.png)