Although I cannot say why the decision was made by the team, I absolutely agree that the DTO/VM-concept is not very clear.
I think, the basic idea was in analogy to the MVP/MVC pattern where often a new view-model is introduced to get a MVVM pattern. So the DTOs that do not affect any view were "pushed" into the actual model/controller layer. Which is ok, if you say that a DTO is a transfer-object between the services.
However, we also had "trouble" and a lot of discussions with the layers and the DTO, VM and so on in our older jhipster-app. We started at the same time when jhipster introduced the DTOs and Mapstruct stuff. In new projects, where we use Jhipster, we always remove all DTOs and VMs. First of all: the (Managed)UserVM/DTO stuff, which is very confusing. This has several reasons:
We never have a VM in the server-side of our applications. In a jhipster application, where the view is entirely based on javascript/angular, the view model must be there. We often have other applications that uses the REST-API and these applications are not views. So, especially in a microservice-architecture, I never would call something in a REST-API "view" or even "view-model".
Thus, in such applications where no view is written in Java (with JSP or so), you will never find the term "VM" in our Java code. Furthermore, there might be different kind of views (e.g. the angular web-app, an iOS app, a desktop client or other things) and each of them has other views, view parts, widgets and therefore needs other view-models. Finally, there were some people that say that the REST-API may not provide any stuff that is not a real resource/entity...
So, you are absolutely right that VMs in jhipster are in fact DTOs. But, as you also say, there is a problem with "DTO", because normally a DTO is just a stateless transfer object to encapsulate common values between two or more services (or systems) without any logic. We think that this is also a architectural problem between the top-down (REST) API-driven and bottom-up domain-driven design, which are somehow mixed in JHipster apps. Similar to your opinion, I think that JHipster uses DTOs where it should be a domain object (or an entity). E.g. a managed user is not a VM or a DTO, it is an entity. I think, the problem here is that some people think that only JPA-based entities are domain objects and there could be no other domain objects (entities).
Finally, we decide to introduce a new type called ADO (API Data Object or Access Data Object), because we found no term that fits. An ADO is compared to jhipster a VM, a DTO or even an entity without any logic. A common "API-Layer" only reads and writes ADOs. This layer is used for the REST API controllers as well as for a Java API that can be used by plugins. This gives us the possibility to ship all ADOs with the client-API-jar without adding any specific internal entities or domain objects. Since the REST-API and each plugin use the same ADOs (and therefore the same description and structure), the developers are not confused and the outer resource-oriented layers are properly separated from the internal layers that use another business logic than only resource-based.
Everything else that is part of the internal layer like entities, derived entities, subsets or supersets of entities are mostly domain objects. So we consequently removed all VMs and DTOs from this internal layer, the business logic and so on.
In sum: To encapsulate the internal (JPA) entities from the external access, we use the ADOs, since there might be others than just views. These ADOs are in terms of jhipster VMs as well as DTOs. But internally behind the common API layer, we never use any DTO or VM or even an ADO, because those entities are mostly domain objects.