DependencyProperty has inheritance property, so if you don't set the UserControlDP's DataContext, the DataContext is inherited from the MainWindow's DataContext. In this case, the UserControlDP's DataContext in your code below is set as MainWindow_ViewModel. Thus, the binding is correctly executed.
<usercontrol:UserControlDP Width="200" Height="100"
TestValue="{Binding TestValueApp, Mode=TwoWay, NotifyOnSourceUpdated=True, NotifyOnTargetUpdated=True}"
Margin="152,54,151,157"></usercontrol:UserControlDP>
In the other case, UserControlDP's DataContext is set as UserControlDP_ViewModel, so the binding is broken. You can see the first exception message as the following at the debug window.
System.Windows.Data Error: 40 : BindingExpression path error: 'TestValueApp' property not found on 'object' ''UserControlDP_ViewModel' (HashCode=24672987)'. BindingExpression:Path=TestValueApp; DataItem='UserControlDP_ViewModel' (HashCode=24672987); target element is 'UserControlDP' (Name=''); target property is 'TestValue' (type 'Object')