Two layers of autoscaling: pods via HPA, nodes via Cast AI

Now let's have a look what is going on the hub level. DHA is running on Kubernetes. Currently we are using horizontal pot autoscaler. When CPU or memory threshold go above target, Kubernetes adds new pot automatically to handle this spike and scale them back down when traffic drops. Our deployment hub deployment can scale from 10 to 500 bots depends of on the traffic. This is how we keep the hub healthy without manual interventions and without infrastructure overprovisioning. So this is also cost effective solution. However, what happens if horizontal pot autoscaler want to add new ports but kubernetes does not have free nodes anymore. This is where second layer comes in. We are using cast AI for Kubernetes note autoscaling when pods are pending because there is no capacity and Kubernetes scheduler is not able to schedule them. Cast AAI is adding new nodes and then scheduler is able to schedule those spots. So we have two layers of scaling. First one is at deployment level. Second one is at infrastructure level via castai.