Reduce Wasted GPU Dollars with Python-Powered Auto Scaling in Azure

Are you wasting money by running GPUs when you don’t need to?

We found a way to significantly reduce GPU costs with an auto-scaling solution for Kubernetes nodes. We are using Celery tasks, Redis as a broker, and KEDA (Kubernetes Event-driven Autoscaling) on Azure Kubernetes Service (AKS).

Overview

Powering one of our recent projects is a Django-based web application that automatically detects changes in satellite imagery using GPU-accelerated processing. Each month, the application automatically pulls up-to-date satellite images from a data provider. It then queues Celery tasks in Redis to process these images.

The analytics we run on these images are resource-intensive and require expensive GPU-enabled Kubernetes nodes.

To significantly reduce idle resource costs, we implemented an auto-scaling solution using KEDA. We set the GPU node’s initial replica count to zero and configured auto-scaling to spin up nodes only when tasks are queued.

KEDA is natively supported by Azure Kubernetes Service (AKS). It enables event-driven auto-scaling, allowing applications to scale dynamically based on external metrics such as queue length and message counts.

For detailed information on KEDA and AKS integration, take a look at the official Microsoft documentation.

Benefits of Scaling GPU Nodes from Zero

Maintaining GPU nodes in an idle state can be costly. Using KEDA to scale GPU nodes from zero offers significant advantages:

• Cost Efficiency: Scaling from zero ensures GPU resources are used only when necessary, drastically reducing overall infrastructure expenses.
• Resource Optimization: Dynamically allocating GPU resources ensures expensive computing resources are utilized efficiently, generating cost only when necessary.

Technical Background: KEDA

KEDA (Kubernetes Event-driven Autoscaling) is a lightweight, Kubernetes-native component designed to auto-scale deployments based on real-time metrics collected from external event sources.

It serves as a bridge between the Kubernetes Horizontal Pod Autoscaler (HPA) and external metrics providers (Redis, Azure Service Bus, and others), enabling event-driven scaling with minimal overhead.

Key features of KEDA:

• Event-driven: Scales workloads dynamically in response to real-time events.
• Support for various sources: Easily integrates with different event sources like Redis, Kafka, Azure Service Bus, RabbitMQ, and Prometheus.
• Customizable: Allows detailed customization through ScaledObject configurations.

Prerequisites

The following code snippets and steps assume you have the following prerequisites in place:

• Azure Kubernetes Service (AKS)
• Celery with Redis as the message broker
• KEDA installed in your AKS cluster

Enable KEDA on AKS

AKS supports KEDA natively, making setup straightforward.

Run the following Azure CLI command to enable KEDA on your cluster:

az aks update \
    --resource-group <your-resource-group> \
    --name <your-aks-name> \
    --enable-keda

Replace <your-resource-group> and <your-aks-name> with your actual Azure resource group and AKS cluster names.

Configuring a Redis Scaler with KEDA

KEDA uses ScaledObject resources to define auto-scaling behavior based on external metrics. Below, we’ll configure a Redis scaler that monitors the Celery task queue length.

Example ScaledObject for Redis

Here’s a detailed example of a KEDA ScaledObject configured to scale Celery worker deployments based on the length of a Redis queue:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: celery-worker-scaler
  namespace: default
spec:
  scaleTargetRef:
    name: celery-worker-deployment  # Target deployment for scaling
  minReplicaCount: 0                # Minimum pods to scale down to
  maxReplicaCount: 10               # Maximum pods to scale up to
  pollingInterval: 15               # Frequency (seconds) KEDA checks the queue
  cooldownPeriod: 60                # Time (seconds) before scaling down
  triggers:
  - type: redis
    metadata:
      address: redis://<redis-host>:6379  # Redis connection string
      listName: celery                   # Redis list for Celery tasks
      listLength: "5"                    # Scale threshold (tasks in queue)

Important Settings Explained

• scaleTargetRef.name: Specifies which Kubernetes deployment to scale.
• minReplicaCount: Minimum pod count, allowing scaling from zero.
• maxReplicaCount: Maximum allowed pods to prevent over-scaling.
• pollingInterval: Interval at which KEDA checks metrics from Redis.
• cooldownPeriod: Delay period before scaling down after the load decreases.
• listLength: The queue length threshold triggering scale-up events.

Deploying the ScaledObject

Deploy your ScaledObject to AKS using kubectl:

kubectl apply -f celery-worker-scaler.yaml

Verifying Auto-Scaling

Verify your auto-scaling setup with the following commands:

kubectl get scaledobjects
kubectl get hpa
kubectl describe scaledobject celery-worker-scaler

Review the scaling events in detail:

kubectl describe hpa celery-worker-deployment

Practical Example

The Django web framework layer powering our web application periodically downloads large volumes of new satellite images from a data provider.

As soon as these images are available, the web framework automatically generates Celery tasks to process these images with GPU-enabled Kubernetes nodes, making sure these intensive workloads are handled efficiently and promptly:

from celery import shared_task

@shared_task
def process_change_detection(basemap_id):
    # GPU-accelerated processing logic here
    pass

Tasks are dispatched as soon as our system detects new satellite images:

process_change_detection.delay(basemap_id)

When the queue size grows larger than the threshold in your ScaledObject, KEDA scales the Celery worker pods up or down. Once the queue is empty, KEDA scales the pods back down to zero, ensuring we only pay for the resources we need.

Other Available Scalers in KEDA

KEDA supports numerous other scalers for a variety different scenarios, including:

• Azure Service Bus Scaler: Scale based on Azure Service Bus queue length.
• RabbitMQ Scaler: Auto-scale based on RabbitMQ queue size.
• Kafka Scaler: Scale based on Kafka topic lag.
• HTTP Scaler: Scale applications based on incoming HTTP request rates.
• Prometheus Scaler: Scale based on custom Prometheus metrics.

A comprehensive list and detailed guidance are available in the official KEDA documentation.

Conclusion

Auto-scaling Kubernetes nodes using Celery, KEDA, and Redis provides a robust, efficient, and cost-effective method for managing dynamic workloads. Leveraging GPU-enabled nodes and scaling them from zero ensures optimized resource usage and reduced operational costs, significantly enhancing your cloud infrastructure’s effectiveness and performance.