How To Run The Cheapest Kubernetes Cluster at $1 Per Day (Tutorial)
This guide has been updated on September 2021.
Preamble
In Part 1 we went through the theory and price comparisons between all major cloud providers and how we achieve the magical number of $1 per day to run a fully managed Kubernetes Cluster, which powers my new app Fakes.io.
The following is a tutorial on how to create your very own cheapest managed Kubernetes cluster.
There will be two different AKS Clusters at different price points:
- AKS Cluster (Basic) — Cost: $1.25/day. Does not allow spot nodepool
- AKS Cluster (Standard) — Cost: $1.88/day. Allow spot nodepool, $0.36 per extra node
Assumptions
- I am assuming that you have basic knowledge of Azure. If you are new to Azure, feel free to have a look at the Azure Fundamentals Course here.
- You need to have an Azure account. You can sign up here if you don’t.
- You need to create a separate Pay-As-You-Go subscription in Azure. The Free Tier offering does not allow you to create a Spot Node Pool (took me days to figure this one out).
- I am using docker-desktop to run my container. You can download it from here.
By the end of this tutorial you will have a fully working web app that allow users to vote if they prefer a cat or dog.
Tutorial
We start by pulling down and running the azure-cli container. Install AKS CLI and AKS-PREVIEW to allow spot nodepool creation.
Set the global variables. You must use your Pay-As-You-Go subscription and not your Free Tier subscription (converting it does not work, you need a completely separate subscription). And create a SSH key pair too.
You need to authorise container services in your Azure subscription. You only need to do this once.
Instructions For AKS Cluster (Basic)
Run the following commands to login, create your resource group and AKS cluster.
Run the Kubernetes deployment manifest to start the application pods.
Give your Public IP address a DNS name, you should be able to reach it at http://ENTER-YOUR-DNS-NAME.westus.cloudapp.azure.com/
Instructions For AKS Cluster (Standard)
Run the following commands to login, create your resource group and AKS cluster. The very first default node pool of your AKS cluster uses on-demand virtual machine. We will leave one node running.
We will create the spot node pool by running the following commands. Once the node is running, we need to remove the taints to allow the core-dns services to run on a spot node.
Optional: Suppressing the System Node
What Azure says about System Nodepool:
System pools must contain at least one node, and user node pools may contain zero or more nodes.
What Azure says about Spot Nodepool:
A spot node pool can’t be the cluster’s default node pool. A spot node pool can only be used for a secondary pool.
So basically Azure stated that we must have one system node at all times.
There is a trick that we can do is to scale the system node to 0 nodes. This will cause Azure to “fight back” by scaling up to 1 node every few minutes, but your scaling rule will force that node to shutdown. You will ended up with the cluster starting/stopping node several hundred times a day.
You will end up paying for those few seconds that the node started up before dying again. But it will be a fraction of the full price.
We will scale down the default node pool to 0 instance using the following command:
Now we wait until all containers shut down in the default node and all the containers started running in the spot node.
You must set the scaling rule to use custom autoscale, set everything to 0.
Run the Kubernetes deployment manifest to start the application pods.
Give your Public IP address a DNS name, you should be able to reach it at http://ENTER-YOUR-DNS-NAME.westus.cloudapp.azure.com/
Scaling (Optional)
Set customer autoscale to CPU usage criteria
- Increase compute instance count by 1 if average percentage CPU is more than 70% for more than 10 minutes
- Decrease compute instance count by 1 if average percentage CPU is less than 20% for more than 10 minutes
Once you are done, you can tear everything down.
