So in the last few days, I tried to find a way to dynamically attach ingress names (like game-1.myapp.com) to solve TCP & UDP for Steam Dedicated Servers on Kubernetes. I have attached the following diagram on how I planned it, but there are some issues I encountered.
I can dynamically create Namespaces, Pods (controlled by Stateful Sets), PVCs, Services, and Ingresses for each individual game server using the Kubernetes API. Each game server lies in its own namespace, completely separated by the others. I assured that the server runs under the hood, the Pod is also Running and active, the logs are good.
I got locked out when I needed to assign the Stateful Set service to an Ingress that is able to continuously reply to TCP/UDP traffic by using a namespaced DNS, that routes to the cluster's Ingress Controller (in Minikube; for Production an ALB/NLB should be used, AFAIK).
Somehow, I need a way to ingress the game-xxxxx.myapp.com to the specific game-xxxxx namespace's pod. It doesn't really matter if they will have appended ports or not.
For this, I can simply just API-call the DNS solver for myapp.com and add or remove A Records when it's needed. This seems okay, but I have found out that I can use ExternalDNS (https://github.com/bitnami/charts/tree/master/bitnami/external-dns) to do this automatically for me, based on the existent services.
What I have tried, no luck yet:
NGINX
Setting up NGINX, but I had to define the exposed ports for each Service. Based on their documentation (https://kubernetes.github.io/ingress-nginx/user-guide/exposing-tcp-udp-services), it is OVERKILL to modify that ConfigMap and Recreate the NGINX pods each time, since there might be many changes and this does not seem viable. Plus, I highly doubt that NGINX will be a breeze under heavy load, I find it more suitable for web servers rather than game servers.
Also, I might need a way to make sure that I can have duplicated ports. For example, I cannot assign in NGINX the same 28015 port to many other servers, even when they are in different namespaces. If I use Agones (https://github.com/googleforgames/agones/blob/release-1.9.0/examples/gameserver.yaml) to assign random ports, at some point I might run out of them to assign.
Traefik
I have tried to use Traefik, but had no luck. The IngressRoute allows the TCP/UDP routing from a Router to and EntryPoint than then routes it to the service assigned. I am not really sure how this works, I tried setting annotations to services & defining entry points, but it still refuses to work: https://doc.traefik.io/traefik/routing/providers/kubernetes-crd/#kind-ingressroutetcp
Agones
Agones should be working for game servers and it supports TCPUDP protocol for service ports, but again, no luck with this.
Flow
I have posted below the diagram on how things should work. I also have this following YAML file that will create the Stateful Set, a PVC, and the Service. You can clearly see I tried ExternalName setup so maybe I can set the Minikube IP to that name and be able to connect, yet again, no luck:
Steam Dedicated Server workflow
apiVersion: v1
kind: Service
metadata:
name: rust-service
labels:
game: rust
spec:
# type: ExternalName
# externalName: rust-1.rust.coal.app
# clusterIP: ""
selector:
game: rust
ports:
- name: rust-server-tcp
protocol: TCP
port: 28015
targetPort: 28015
- name: rust-server-udp
protocol: UDP
port: 28015
targetPort: 28015
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: rust-server
spec:
selector:
matchLabels:
game: rust
replicas: 1
serviceName: rust-service
template:
metadata:
name: rust-server
labels:
game: rust
spec:
containers:
- name: rust
image: didstopia/rust-server:latest
ports:
- name: rust-server-tcp
protocol: TCP
containerPort: 28015
- name: rust-server-udp
protocol: UDP
containerPort: 28015
volumeClaimTemplates:
- metadata:
name: local-disk
spec:
resources:
requests:
storage: "10Gi"
accessModes: ["ReadWriteOnce"]
Edit: bump
