I have recently installed Istio 1.7.3 in Kubernetes Cluster (GKE) on Google Cloud and various other addons like Grafana Prometheus dashboards, etc. I have configured the dashbaords with Istio Service and other parameters, they started working when I resized the cluster size to 0(zero) and again after few time resize it to 2(or as per requirement). I found that the configured Grafana and Prometheus dashboard lost all configured dashboards after the resizing process of cluster. All the dashboards are not available also the users I have created also not there. What is the good practice to install and manage ISTIO with Prometheus, Grafana and other dashboards?
1 Answers
If you use addons provided by istio then it's just intended for new users to help them quickly getting started. However, it does not offer advanced customization, like persistence or authentication and as such should not be considered production ready. Probably that's why you lost all of your dashboards and configurations.
As mentioned here
Starting with Istio 1.6, we are introducing a new method for integration with telemetry addons, such as Grafana, Prometheus, Zipkin, Jaeger, and Kiali.
In previous releases, these addons were bundled as part of the Istio installation. This allowed users to quickly get started with Istio without any complicated configurations to install and integrate these addons. However, it came with some issues:
The Istio addon installations were not as up to date or feature rich as upstream installation methods. Users were left missing out on some of the great features provided by these applications, such as:
- Persistent storage
- Features like Alertmanager for Prometheus
- Advanced security settings
Integration with existing deployments that were using these features was more challenging than it should be.
What is the good practice to install and manage ISTIO with Prometheus, Grafana and other dashboards?
The good practise here would be to create your own addons and configure them with istio.
There are the addons yamls, you can use it as a reference to build your own setup.
Additionally I found a few sites and github issues which may help with configuration your own addons.