AWS Cost Optimization: From $3800 to $1000 with EKS, KEDA and Karpenter
In the world of cloud computing, one of the biggest challenges for companies is keeping costs under control without sacrificing performance. In this article, I share how we’ve managed to reduce the computing costs of our AWS infrastructure by over 70% by migrating from traditional architectures to a Kubernetes-based solution with EKS, KEDA, and Karpenter.
The Starting Point: An Expensive Infrastructure
Our original architecture was based on two AWS services to run background processes (background workers):
AWS Elastic Beanstalk Workers
Our Elastic Beanstalk workers represented a significant portion of the monthly bill:
- Monthly cost: $2,500 - $2,800
- Operation: 24x7, always running
- Problem: Static resources, no adaptation to real load
- Inefficiency: We paid for resources we didn’t use during off-peak hours
AWS Lambda
For more specific tasks, we used Lambda:
- Monthly cost: $500 - $1,000
- Limitations: Maximum timeout of 15 minutes
- Static resources: No dynamic optimization
Total monthly cost: Approximately $3,000 - $3,800 to keep this infrastructure operational.
The Solution: EKS + KEDA + Karpenter
After analyzing different alternatives, we decided to migrate our infrastructure to an architecture based on:
- EKS (Elastic Kubernetes Service): AWS’s orchestrated Kubernetes platform
- KEDA (Kubernetes Event-driven Autoscaling): Event-based automatic scaling
- Karpenter: Automatic and elastic node provisioning
This combination allows us to:
- Dynamically adjust resources based on real load
- Scale to zero when there’s no pending work
- Optimize compute aggregation to reduce waste
- Fully control process timeouts and states
The Migration Process
The migration followed a structured process in several phases:
1. Identification and Analysis
First, we identified all processes running on both Beanstalk and Lambda, understanding:
- Resource consumption patterns
- Demand peaks
- Dependencies between services
2. Containerization
We created Docker images for each project and uploaded them to ECR (Elastic Container Registry), standardizing the format of all our services.
3. CI/CD Standardization
One of the major benefits has been creating a single CI/CD pipeline for all projects:
- Unification of the deployment process
- Reduced integration time for new services
- Improved consistency and quality of deployments
4. KEDA and Karpenter Configuration
We configured:
- KEDA: To automatically scale pods based on message queues (SQS and Redis depending on the case)
- Karpenter: To provision nodes only when needed, with optimal size based on load
Results: Over 70% Savings
After the complete migration, the results have been very significant:
Economic Impact
- Resulting monthly cost: ~$1,000
- Monthly savings: $2,000 - $2,800
- Cost reduction: >70%
Operational Benefits
Beyond economic savings, we’ve obtained significant operational improvements:
- Infrastructure simplification: Fewer managed services, more control
- Unified control: All processes follow the same patterns
- Flexibility: We can quickly add new background services in a standardized way
- Visibility: Better monitoring and understanding of service behavior
Conclusions
The migration from a Beanstalk and Lambda-based infrastructure to EKS with KEDA and Karpenter has been a resounding success. The benefits are not only economic, but we’ve also simplified our architecture and improved our ability to scale efficiently.
Key Learnings
- Not all managed services are the most economical: Sometimes, a initially more complex solution can be much more efficient in the long run
- Intelligent auto-scaling is essential: The ability to scale to zero and adapt to real load is key to optimizing costs
- Standardization is value: A single pipeline and consistent patterns reduce operational complexity
If you’re considering optimizing your AWS infrastructure costs, I recommend evaluating Kubernetes-based solutions with event-based auto-scaling. The return on investment can be very quick, as in our case.