As for now Network Loadbalancer doesn't support IPv6. Note that this is the only pass-through load balancer currently available on GCP. Compare it with Summary of Google Cloud load balancers table.
Only proxy loadbalancers such as HTTP/HTTPS Loadbalancers and TCP Proxy/SSL Proxy do support IPv6 (compare with IPv6 termination for HTTP(S), SSL Proxy, and TCP Proxy Load Balancing)
As you can read in docs:
Because the load balancer is a pass-through load balancer, your
backends terminate the load-balanced TCP connection or UDP packets
themselves.
As you may already know (and you can read about it here) for the time being GCP VPC and GCE instances support only IPv4 connectivity:
VPC networks only support IPv4
unicast traffic. They do not
support
broadcast,
multicast, or IPv6 traffic
within the network; VMs in the VPC network can only send to IPv4 destinations and only receive traffic from IPv4 sources. However, it
is possible to create an IPv6 address for a global load
balancer.
As you already mentioned TCP Proxy and SSL Proxy do not support arbitrary destination ports. Only well-know ports listed here are supported. Arbitrary ports can be used only with External Network TCP/UDP loadbalancer but because of lack of the support for IPv6 you cannot use it.
As to your particular use case the conclusion is that at the moment it cannot be done the way you want.
Does your backend really have to be exposed on this non standard port ? - I assume you have good reasons for using it so most probably changing it for some well-known port is not even an option.
The problem is that this is not HTTP service. - If it was http service but only exposed on some arbitrary port then you could use ingress. If it was the case you would simply create an ingress which forwards the request to one of your backend Services
like in this example. Ingress backends can really use arbitrary ports, but not arbitrary protocols - that's the point. The underlying application layer protocol has to be http. So if your service uses some completely different protocol you may forget about ingress right away.
You actually answered your question yourself. I see that you went through the docs and analyzed quite thoroughly all available solutions already, ruling out those which don't meet your requirements. Well... I can only confirm it with this summary. I hope it can be helpful even though it doesn't provide the solution that you are looking for.