DevOps Daily

How to Set Up AWS Cost Explorer and Budgets for Teams

DevOps Daily Team — Sat, 22 Nov 2025 09:00:00 GMT

Cloud costs can spiral out of control quickly if you're not paying attention. AWS Cost Explorer and Budgets give you the tools to track spending, identify cost drivers, and set up alerts before things get expensive.

This guide walks you through setting up both services for your team, from initial configuration to creating useful budgets and reports.

What You'll Need

Before you start, make sure you have:

AWS account with billing access (IAM permissions for ce:* and budgets:*)
Root account access or an IAM user with appropriate permissions
A clear understanding of your team's cost structure and spending patterns

Understanding AWS Cost Explorer

AWS Cost Explorer is a visualization tool that lets you analyze your AWS costs and usage over time. You can view data at different levels of granularity, filter by service, region, or tags, and create custom reports.

The service is free to use for basic features, though there's a charge for more advanced API calls and saving custom reports.

Enabling AWS Cost Explorer

Cost Explorer isn't enabled by default on all AWS accounts. Here's how to turn it on:

Log into the AWS Console with an account that has billing permissions
Navigate to the Billing and Cost Management dashboard
In the left sidebar, click Cost Explorer
Click Enable Cost Explorer

AWS will take up to 24 hours to prepare your cost data. You'll receive an email when it's ready. During this time, AWS analyzes your historical billing data (up to 12 months) and creates the initial dataset.

Setting Up Cost Explorer for Your Team

Once enabled, you'll want to configure Cost Explorer to provide useful insights for your team.

Create Custom Cost Reports

Start by creating reports that match how your team thinks about costs:

Go to Cost Explorer in the Billing console
Click Create report
Choose a report type:
- Cost and usage - Shows spending over time
- Reservation utilization - Tracks Reserved Instance usage
- Reservation coverage - Shows how much of your usage is covered by reservations

For most teams, start with a cost and usage report.

Filter by Service and Time Period

Configure the report to show relevant data:

Time range: Select the period you want to analyze (last month, last 3 months, etc.)
Granularity: Choose daily, monthly, or hourly views
Group by: Organize costs by Service, Region, or Linked Account
Filters: Add filters for specific services, tags, or usage types

Use Tags for Better Cost Allocation

Tags are crucial for tracking costs by team, project, or environment. Set up a tagging strategy:

Define required tags (e.g., Team, Project, Environment)
Apply tags to all AWS resources
Enable Cost Allocation Tags in the Billing console:
- Go to Billing > Cost Allocation Tags
- Activate the tags you want to track
- Wait 24 hours for tag data to appear in Cost Explorer

Once activated, you can group costs by these tags in Cost Explorer.

Creating AWS Budgets

While Cost Explorer helps you analyze past spending, AWS Budgets lets you set spending limits and get alerts when you approach them.

Access AWS Budgets

Navigate to AWS Budgets from the Billing dashboard:

Open the Billing and Cost Management console
Click Budgets in the left sidebar
Click Create budget

Choose a Budget Type

AWS offers several budget types:

Cost budget - Track spending against a dollar amount
Usage budget - Monitor specific service usage (e.g., EC2 hours)
Reservation budget - Track Reserved Instance or Savings Plans utilization
Savings Plans budget - Monitor Savings Plans coverage

For most teams, start with a Cost budget.

Set Up a Monthly Cost Budget

Here's how to create a basic monthly budget:

Select Cost budget and click Next
Give your budget a name (e.g., "Production Environment Monthly Budget")
Set the budget period to Monthly
Choose between:
- Fixed budget - Same amount each month
- Planned budget - Different amounts for different months

For a fixed budget, enter your monthly spending limit.

Add Filters to Scope Your Budget

Don't create one budget for everything. Instead, create specific budgets for different parts of your infrastructure:

Apply filters to narrow the budget scope:

Services - Limit to specific services like EC2, RDS, or S3
Tags - Track budgets by team or project
Linked accounts - Monitor spending in specific AWS accounts

For example, create separate budgets for:

Development environment
Production environment
Specific team or project
High-cost services (EC2, RDS, data transfer)

Configure Budget Alerts

Alerts are what make budgets useful. Set up notifications at different thresholds:

In the Configure alerts section, add alert thresholds
Set alerts at useful percentages:
- 50% - Early warning
- 80% - Action needed
- 100% - Budget exceeded
- 110% - Serious overspend
Enter email addresses for notifications
You can also configure SNS topics for integration with Slack, PagerDuty, or other tools

Best Practices for Team Cost Management

1. Create Budgets at Multiple Levels

Don't rely on a single company-wide budget. Create budgets for:

Each environment (dev, staging, production)
Each team or department
Each major project
High-cost services individually

This gives you better visibility into where money is going.

2. Set Up a Tagging Policy

Enforce tagging from day one:

Define required tags before deploying resources
Use AWS Config Rules to enforce tagging compliance
Review untagged resources regularly
Automate tagging with Infrastructure as Code tools

3. Review Cost Explorer Weekly

Make cost review a regular habit:

Check Cost Explorer every Monday
Look for unexpected spikes or trends
Identify resources that can be optimized
Share findings with your team

4. Use Forecasting

Cost Explorer includes forecasting based on historical data:

In Cost Explorer, select a time range
Click Forecast to see predicted costs
Use this to adjust budgets before overspending occurs

5. Set Up Cost Anomaly Detection

AWS offers anomaly detection that uses machine learning to spot unusual spending:

Go to Cost Anomaly Detection in the Billing console
Create a monitor for your services
Configure alerts for anomalies
Review detected anomalies weekly

Setting Up Budget Alerts for Slack

To get budget alerts in Slack:

Create an SNS topic for budget notifications
Create a Lambda function that posts to Slack webhooks
Subscribe the Lambda to your SNS topic
Configure your budget to send alerts to the SNS topic

This keeps your team informed in real-time when budgets are approaching limits.

Granting Team Access to Cost Data

Not everyone needs full billing access. Use IAM to grant appropriate permissions:

Create a Cost Explorer Read-Only Role

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ce:Get*",
        "ce:Describe*",
        "ce:List*"
      ],
      "Resource": "*"
    }
  ]
}

This lets team members view cost data without modifying budgets or accessing other billing information.

Create a Budget Manager Role

For team leads who need to manage budgets:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "budgets:*",
        "ce:*"
      ],
      "Resource": "*"
    }
  ]
}

Monitoring Long-Term Trends

Beyond daily monitoring, track long-term trends:

Export cost data to S3 for analysis
Create monthly cost reports
Compare month-over-month changes
Identify seasonal patterns
Plan for growth

Use the Cost and Usage Report feature to export detailed billing data to S3, then analyze it with tools like Amazon Athena or QuickSight.

Common Pitfalls to Avoid

Not Acting on Alerts

Budget alerts are only useful if you respond to them. When an alert fires:

Investigate immediately
Identify the cause
Take corrective action
Document what happened

Setting Budgets Too High

If your budget is set too high, you won't get early warnings. Start conservative and adjust based on actual usage.

Ignoring Unutilized Resources

Cost Explorer shows you what you're paying for. Use it to find:

Idle EC2 instances
Unused Elastic IPs
Old snapshots
Unattached EBS volumes

Schedule regular cleanup sessions to remove these resources.

Not Using Savings Plans or Reserved Instances

If you have predictable workloads, Reserved Instances and Savings Plans can reduce costs by 30-70%. Use Cost Explorer's recommendations to identify opportunities.

Next Steps

Once you have Cost Explorer and Budgets configured:

Set up regular cost review meetings
Create a runbook for responding to budget alerts
Implement automated responses to common cost issues
Track cost optimization efforts over time
Share cost data with stakeholders regularly

Cost management is an ongoing process. The tools AWS provides give you visibility, but you need to act on the insights they provide.

Start with simple budgets and reports, then refine them as you learn more about your spending patterns. Your future self (and your finance team) will thank you.

Deployment Strategies: Blue-Green, Canary, and Rolling Deployments Explained

DevOps Daily Team — Mon, 17 Nov 2025 09:00:00 GMT

Deploying a new version of your application shouldn't feel like jumping off a cliff. Yet for many teams, every deployment brings anxiety: Will the new version work? What if users encounter bugs? How quickly can we roll back if something goes wrong?

The way you deploy your application matters just as much as what you deploy. Different deployment strategies offer varying levels of risk, speed, and resource requirements. Some let you switch between versions instantly, while others gradually introduce changes to minimize impact. Understanding these strategies helps you choose the right approach for your specific needs.

TLDR: This guide covers three core deployment strategies. Blue-green deployments run two identical environments and switch traffic between them for instant rollbacks. Canary deployments gradually roll out changes to a small subset of users before full deployment. Rolling deployments update instances one at a time to maintain availability throughout the process. Each strategy has distinct trade-offs in terms of cost, complexity, and risk mitigation.

Why Deployment Strategy Matters

Your deployment strategy directly impacts your application's availability, your team's confidence in shipping changes, and your ability to recover from problems. A naive approach, replacing all running instances at once, leaves you vulnerable to widespread outages if the new version has issues.

Consider a typical scenario: you deploy a new version to production, and within minutes, users start reporting errors. With some deployment strategies, you can revert to the previous version in seconds. With others, you'll need to redeploy the old version and wait for it to roll out, potentially leaving users affected for 10-15 minutes or longer.

The right strategy also affects your testing approach. Some methods let you test new versions with real production traffic before committing fully. Others require more confidence in your pre-production testing because once you deploy, you're committed.

Beyond technical concerns, your deployment strategy influences team culture and development velocity. Teams with confidence in their deployment process ship more frequently. Those worried about deployments batch changes together, increasing risk and making issues harder to diagnose when they occur.

The Evolution of Deployment Practices

Traditional deployment approaches involved scheduled maintenance windows, often late at night or on weekends. Operations teams would take systems offline, manually update software, run verification scripts, and bring everything back online. This worked when deployments happened monthly or quarterly, but modern software development demands faster iteration.

The rise of continuous delivery and DevOps culture pushed teams to deploy more frequently, sometimes dozens or hundreds of times per day. This shift required new approaches that maintained availability during deployments and provided quick recovery options when issues arose.

Cloud computing and container orchestration platforms like Kubernetes made sophisticated deployment strategies accessible to organizations of all sizes. What once required custom infrastructure and extensive automation now comes built into standard platforms.

Blue-Green Deployment

Blue-green deployment maintains two identical production environments, conventionally called "blue" and "green." At any given time, only one environment serves live traffic while the other sits idle or runs the previous version.

When you're ready to deploy, you deploy the new version to the idle environment, run your tests there, and then switch the router or load balancer to direct traffic to the newly updated environment. The switch typically takes seconds, and if anything goes wrong, you can switch back just as quickly.

┌─────────┐      Active       ┌──────────────┐
│  Users  │ ────────────────> │ Blue (v1.0)  │
└─────────┘                   └──────────────┘

                              ┌──────────────┐
                              │ Green (idle) │
                              └──────────────┘
                                     │
                                     │ Deploy v2.0
                                     ▼
┌─────────┐                   ┌──────────────┐
│  Users  │                   │ Blue (v1.0)  │
└─────────┘                   └──────────────┘
     │
     │        Switch traffic
     └─────────────────────────> ┌──────────────┐
                                 │ Green (v2.0) │ Active
                                 └──────────────┘

The beauty of blue-green deployment lies in its simplicity and safety. You're not mixing versions or gradually shifting traffic, you're making a clean, atomic switch. This makes rollback equally simple: just switch back to the previous environment.

When Blue-Green Makes Sense

Blue-green deployment shines in scenarios where you need absolute confidence in your ability to revert quickly. Financial services, healthcare systems, and other high-stakes applications often choose this approach because the cost of extended downtime exceeds the infrastructure cost of maintaining dual environments.

Regulatory compliance sometimes mandates clear separation between versions and the ability to demonstrate exactly when and how deployments occurred. Blue-green provides this audit trail naturally, you can see precisely when traffic switched from one environment to another.

Teams with complex integration testing requirements benefit from blue-green because they can run extensive tests against the new environment before any users see it. You can verify database migrations, test third-party integrations, and validate performance under load, all while production continues running normally on the blue environment.

The Cost Consideration

The primary drawback of blue-green deployment is resource duplication. You're maintaining two complete production environments, essentially doubling your infrastructure costs during deployments. For applications with expensive infrastructure, persistent data stores, or large-scale deployments, this becomes prohibitively expensive.

Some organizations mitigate this by keeping the idle environment powered down or scaled to minimal capacity when not in use, then scaling it up for deployments. This reduces cost but increases deployment time and complexity.

Handling Stateful Components

Databases and other stateful components complicate blue-green deployments. You can't simply duplicate a database and have both versions running with different data. Most teams use a shared database approach where both blue and green environments connect to the same database cluster.

This creates a constraint: your database schema changes must be backward compatible with the previous application version. If you need to rename a column, you can't do it in a single deployment. Instead, you add the new column, update the application to use both, deploy that version, migrate data, then remove the old column in a subsequent deployment.

Session state presents similar challenges. If user sessions are stored in-memory on application servers, switching from blue to green will log everyone out. Teams typically solve this by using external session storage like Redis or database-backed sessions that persist across environment switches.

Canary Deployment

Canary deployment takes its name from the "canary in a coal mine" concept. Instead of switching all traffic at once, you route a small percentage of traffic to the new version while keeping most users on the stable version. You monitor the canary closely for errors, performance issues, or other problems.

If the canary looks healthy, you gradually increase the percentage of traffic it receives. If problems appear, you route all traffic back to the stable version with minimal user impact.

                    ┌──────────────────┐
        5% traffic  │   v2.0 (Canary)  │
       ┌──────────> │   (1 instance)   │
       │            └──────────────────┘
       │
┌─────────┐
│  Users  │
└─────────┘
       │
       │            ┌──────────────────┐
       └──────────> │      v1.0        │
        95% traffic │   (9 instances)  │
                    └──────────────────┘

Canary deployment provides a middle ground between caution and speed. You're exposing real users to new code, getting actual production data about how it performs, but limiting the blast radius if something goes wrong.

The Progressive Rollout Philosophy

The fundamental principle behind canary deployments is progressive validation. You start with a small percentage, maybe 5-10% of traffic, and watch closely. If metrics remain stable for a defined period, perhaps 15-30 minutes, you increase to 25%. Then 50%. Then 75%. Finally 100%.

The percentages and timing depend on your traffic volume and risk tolerance. A high-traffic service might be comfortable with 10% representing millions of requests, giving statistically significant data quickly. A lower-traffic service might need higher percentages or longer observation periods to gather meaningful data.

Some teams use more sophisticated targeting for their canaries. Instead of random percentage-based routing, they might route traffic from internal users, beta testers, or specific customer segments to the canary first. This lets them test with friendly audiences who understand they might encounter issues.

Geographic routing provides another approach: deploy the canary in one region or data center while keeping others on the stable version. If problems occur, impact is limited to a specific geography.

Monitoring Requirements

Canary deployments demand excellent observability. You need to compare metrics between the stable version and canary in real-time. Key metrics include error rates, response times, resource utilization, and business-specific indicators like conversion rates or transaction success rates.

Automated decision-making based on these metrics takes canary deployments to the next level. If error rates on the canary exceed the stable version by a threshold, say 10%, automatically roll back. If response times degrade beyond acceptable limits, halt the rollout.

This requires not just monitoring tools, but careful thought about which metrics matter and what thresholds indicate problems versus normal variation. False positives that unnecessarily halt deployments slow down your development process. False negatives that miss real issues defeat the purpose of canary testing.

When Canary Deployment Excels

Canary deployment works best for user-facing applications where you can easily measure impact through metrics. E-commerce platforms, social media applications, and content delivery systems are ideal candidates because you can track engagement, errors, and performance in real-time.

High-traffic applications benefit particularly from canaries because even a small percentage represents significant volume for statistical analysis. If you're serving millions of requests per hour, 5% gives you hundreds of thousands of data points quickly.

Organizations with strong DevOps practices and good monitoring infrastructure get the most value from canaries. The strategy requires automation, observability, and often custom tooling to implement effectively.

The Complexity Trade-off

Canary deployment introduces significant complexity compared to simpler strategies. You need infrastructure that can route traffic based on percentages or other criteria. You need monitoring systems that can segment metrics by version. You need automation to manage the progressive rollout and potential rollback.

Many teams underestimate this complexity and struggle to implement canary deployments effectively. The infrastructure alone often requires a service mesh like Istio, Linkerd, or AWS App Mesh, each of which brings its own operational overhead.

Rolling Deployment

Rolling deployment updates your application gradually, replacing old instances with new ones in phases. Unlike blue-green, you don't maintain a full duplicate environment. Unlike canary, you're not trying to test with a small subset, you're methodically replacing everything.

The process typically works like this: stop one instance, deploy the new version, start it, verify it's healthy, then move to the next instance. You continue until all instances run the new version.

Initial state (v1.0):
┌────┐ ┌────┐ ┌────┐ ┌────┐

Step 1: Update first instance
┌────┐ ┌────┐ ┌────┐ ┌────┐
│v2.0│ │v1.0│ │v1.0│ │v1.0│

Step 2: Update second instance
┌────┐ ┌────┐ ┌────┐ ┌────┐
│v2.0│ │v2.0│ │v1.0│ │v1.0│

Step 3: Update third instance
┌────┐ ┌────┐ ┌────┐ ┌────┐
│v2.0│ │v2.0│ │v2.0│ │v1.0│

Final state (v2.0):
┌────┐ ┌────┐ ┌────┐ ┌────┐
│v2.0│ │v2.0│ │v2.0│ │v2.0│

Rolling deployment represents the default approach for most modern orchestration platforms. Kubernetes, ECS, and other container platforms implement rolling updates as their standard deployment mechanism.

Resource Efficiency

The primary advantage of rolling deployments is resource efficiency. At any given time, you're running approximately your normal instance count. If you have four instances and replace them one at a time, you briefly have five instances during each transition, but you never need double the infrastructure like blue-green requires.

This makes rolling deployments accessible to organizations of all sizes and budgets. You don't need to justify the cost of duplicate infrastructure, you just need standard orchestration tooling.

Gradual Risk Distribution

Rolling deployments spread risk across time. If the new version has a problem, you discover it when the first instance starts serving traffic. The rollout pauses (if you've configured health checks properly), and only a fraction of your capacity is affected.

This gradual exposure provides some of the benefits of canary deployments without the infrastructure complexity. You're naturally testing with a subset of traffic as each instance comes online.

However, this protection only works if you have proper health checks and automated rollback mechanisms. Without them, a rolling deployment with a broken version will methodically replace all your healthy instances with broken ones.

The Backward Compatibility Requirement

Rolling deployments create a period where multiple versions run simultaneously. This means your new version must be compatible with the old version, at least during the deployment window.

For stateless web applications serving independent requests, this usually isn't a problem. Each request is handled by whatever instance receives it, and users don't notice which version they're hitting.

For applications with shared state, distributed systems, or inter-service communication, compatibility becomes critical. If version 2.0 changes a message queue format, API contract, or database schema in an incompatible way, version 1.0 instances will break as soon as the changes take effect.

This leads to the expand-contract pattern for database changes: first expand the schema to support both old and new formats, deploy the application update, then contract the schema to remove old format support. This requires three deployments instead of one but maintains compatibility throughout.

Deployment Velocity

Rolling deployments naturally throttle your deployment speed based on instance count and health check timing. If you have 20 instances and update them one at a time with 30-second health checks, deployment takes at least 10 minutes.

You can adjust parameters to speed this up. Updating multiple instances simultaneously increases speed but also increases risk. Shorter health check periods deploy faster but might not catch problems that take time to manifest.

Many teams find rolling deployments too slow for their needs and move to blue-green or canary strategies for faster feedback loops and quicker rollbacks.

Choosing the Right Strategy

No single deployment strategy fits all situations. The right choice depends on your application characteristics, organizational constraints, and risk tolerance.

Consider blue-green deployment when:

You need instant rollback capability
Regulatory or compliance requirements demand clear audit trails
Your application requires extensive integration testing before user exposure
Infrastructure costs are acceptable relative to business risk
You have complex database migrations that benefit from testing against production data before cutover

Consider canary deployment when:

You have high traffic volumes that provide quick statistical validation
Strong monitoring and observability infrastructure is in place
You need to validate changes with real production traffic
You can tolerate the infrastructure complexity of traffic routing
Your team has experience with progressive delivery practices

Consider rolling deployment when:

Resource efficiency is a priority
Your application handles backward compatibility well
You need a straightforward approach with good platform support
Deployment speed is less critical than simplicity
Your team prefers operational simplicity over deployment sophistication

Many organizations use different strategies for different applications or even different deployment scenarios for the same application. Routine bug fixes might use rolling deployments, while major feature releases get canary treatment, and critical security patches use blue-green for maximum safety and rollback capability.

Hybrid Approaches and Advanced Patterns

Experienced teams often combine strategies to get benefits of multiple approaches. A common pattern is blue-green deployment with canary testing: maintain blue and green environments, but when switching, first route a small percentage of traffic to green while most stays on blue. If metrics look good, complete the switch.

Feature flags decouple deployment from release. You can deploy new code using any strategy but keep features disabled, then gradually enable them through configuration changes. This separates the technical risk of deployment from the business risk of new features.

Shadow deployments route production traffic to a new version without returning responses to users. The shadow version processes requests and you monitor its behavior, but user experience comes entirely from the stable version. This validates new code with real workloads without any user impact.

Progressive delivery encompasses all these techniques, treating deployment as a continuous process of gradually increasing exposure while monitoring impact and maintaining rollback capability at every stage.

Infrastructure and Tooling Considerations

Your deployment strategy choice is constrained by available infrastructure. Cloud platforms like AWS, Azure, and Google Cloud provide managed services that implement various strategies. AWS CodeDeploy, Azure DevOps, and Google Cloud Deploy offer blue-green and canary deployments with varying levels of automation.

Container orchestration platforms like Kubernetes provide rolling deployment primitives out of the box. Blue-green requires additional configuration but is straightforward. Canary deployments benefit from service mesh integration (Istio, Linkerd) for traffic management.

Serverless platforms like AWS Lambda, Azure Functions, and Google Cloud Functions have their own deployment models. Lambda aliases and versioning enable blue-green and canary patterns natively.

The right tooling reduces deployment strategy complexity significantly. Platforms like Spinnaker, Argo CD, and Flagger provide deployment automation specifically designed for sophisticated strategies.

Monitoring and Observability Essentials

Regardless of strategy, successful deployments depend on detecting problems quickly. At minimum, track error rates, response times, throughput, and resource utilization. Structure your monitoring to compare metrics between versions during deployments.

Distributed tracing helps correlate user experience with specific application versions when multiple versions run simultaneously. Each trace should include version information to enable filtering and comparison.

Synthetic monitoring and health checks validate that new instances actually work before routing traffic to them. These should test critical paths and dependencies, not just that the application starts.

Business metrics often catch problems that technical metrics miss. If your e-commerce platform's new version has a broken checkout flow, you might not see increased error rates, but you'll see dropped conversion rates.

Testing Your Deployment Strategy

Before implementing a new deployment strategy in production, test it thoroughly in lower environments. Create scenarios that simulate production conditions and practice deployment procedures, including rollbacks.

Chaos engineering helps validate that your deployment strategy works under adverse conditions. Deliberately inject failures during deployments and verify that health checks catch them, rollback procedures work, and user impact is minimal.

Document deployment procedures and runbooks. Even with extensive automation, teams need clear documentation for troubleshooting when things go wrong.

Time your deployments to understand how long each strategy takes. This helps you plan deployment windows and set realistic expectations with stakeholders.

Common Pitfalls

Session state tied to specific instances breaks during deployments when those instances are replaced. Use distributed session storage to avoid this issue.

Database connection pool sizing can cause problems when all instances suddenly reconnect after a deployment. Stagger deployments and size pools appropriately to avoid overwhelming your database.

Cache invalidation across versions requires careful planning. Version your cache keys to prevent new code from reading data cached by old code in an incompatible format.

External dependencies and API contracts must maintain compatibility during deployments. Coordinate changes across services or use versioned APIs to prevent breaking integrations.

Health checks that only verify application startup miss problems that manifest under load or over time. Include integration tests and realistic traffic patterns in health validation.

The Human Factor

Deployment strategy affects team culture and development velocity. Teams with confidence in their deployment process ship more frequently and take more measured risks. Those worried about deployments batch changes together, increasing risk and making issues harder to debug.

Automation reduces human error but requires investment in tooling and processes. The most sophisticated deployment strategy fails if manual steps introduce mistakes.

Clear communication during deployments, especially for user-facing changes, reduces confusion and improves incident response. Many teams use ChatOps to make deployments visible to the entire organization.

Post-deployment reviews help teams learn from both successes and failures. Document what went well, what went wrong, and how to improve future deployments.

Deployment strategies give you control over how changes reach production and how quickly you can respond when things go wrong. Blue-green offers instant rollback at the cost of double infrastructure. Canary provides gradual risk mitigation through phased rollout. Rolling deployment balances resource efficiency with safety through sequential updates. Choose based on your specific requirements for cost, risk tolerance, and operational complexity. Start with rolling deployments for routine updates, and introduce blue-green or canary when specific situations justify the additional investment. The best strategy is the one that matches your technical constraints, organizational capabilities, and risk profile while enabling your team to ship confidently and frequently.

DevOps Weekly Digest - Week 47, 2025

Mon, 17 Nov 2025 00:00:00 GMT

📌 Handpicked by DevOps Daily - Your weekly dose of curated DevOps news and updates!

⚓ Kubernetes

📄 Kubernetes CNI Drivers

Ever wondered how we just create a pod in Kubernetes and it gets an IP address magically, can communicate with each other, and host nodes without issues? Networking is not that simple, so how does all

📅 Nov 17, 2025 • 📰 DZone DevOps

🔗 Read more

📄 From Agent AI to Agentic AI: Building Self-Healing Kubernetes Clusters That Learn

In Part 1: AI-Driven Kubernetes Diagnostics, we built an AI agent that analyzes Kubernetes pod failures and suggests fixes. It works when you're at your desk, ready to approve each action. For the pur

📅 Nov 17, 2025 • 📰 DZone DevOps

🔗 Read more

📄 Kubernetes Multi-Cluster Architecture: Benefits And Strategies

Kubernetes multi-cluster architectures transform how organizations approach container orchestration by distributing workloads across multiple independent clusters rather than relying on single-cluster

📅 Nov 17, 2025 • 📰 SUSE Blog

🔗 Read more

📄 DxOperator from DH2i is now certified for Red Hat OpenShift 4.19

Kubernetes has emerged as a powerful foundation for deploying and managing cloud-native applications, and Red Hat OpenShift operators are the best way to streamline this. DH2i's DxOperator, the SQL Se

📅 Nov 14, 2025 • 📰 OpenShift Blog

🔗 Read more

📄 Canonical Kubernetes officially included in Sylva 1.5

Sylva 1.5 becomes the first release to include Kubernetes 1.32, bringing the latest open source cloud-native capabilities to the European telecommunications industry With the launch of Sylva 1.5, Cano

📅 Nov 13, 2025 • 📰 Ubuntu Blog

🔗 Read more

📄 Introducing longhornctl: The Command-Line Interface for SUSE Storage

Why a CLI? In the world of Kubernetes storage, reliability is essential — but flexibility and automation are what make a system shine. SUSE Storage already provides a powerful UI and a CRD-driven cont

📅 Nov 13, 2025 • 📰 SUSE Blog

🔗 Read more

📄 KubeCon NA 2025 Recap: The Dawn of the AI Native Era | Blog

Explore top insights from KubeCon NA 2025. From the Kubernetes AI Conformance Program to Adobe’s Agent Economy and Apple Containerization, discover how the community and Harness are defining the futur

📅 Nov 13, 2025 • 📰 Harness Blog

🔗 Read more

📄 Beyond YAML in Kubernetes: The 2026 Automation Era

Kubernetes continues to evolve, powering not only applications but entire AI and ML systems across clouds, edges, and enterprises. By 2026, DevOps engineers, SREs, cloud engineers, and platform teams

📅 Nov 12, 2025 • 📰 Pulumi Blog

🔗 Read more

📄 Ingress NGINX Retirement: What You Need to Know

To prioritize the safety and security of the ecosystem, Kubernetes SIG Network and the Security Response Committee are announcing the upcoming retirement of Ingress NGINX. Best-effort maintenance will

📅 Nov 11, 2025 • 📰 Kubernetes Blog

🔗 Read more

📄 Canonical releases FIPS-enabled Kubernetes

Today at KubeCon North America, Canonical, the publisher of Ubuntu, released support to enable FIPS mode in its Kubernetes distribution, providing everything needed to create and manage a scalable clu

📅 Nov 11, 2025 • 📰 Ubuntu Blog

🔗 Read more

☁️ Cloud Native

📄 Introducing OpenShift Service Mesh 3.2 with Istio’s ambient mode

We are thrilled to announce the general availability of Red Hat OpenShift Service Mesh 3.2. This release includes the general availability of Istio’s ambient mode—a new way of deploying service mesh w

📅 Nov 17, 2025 • 📰 Red Hat Blog

🔗 Read more

📄 Amazon ECS improves Service Availability during Rolling deployments

Amazon Elastic Container Service (Amazon ECS) now includes enhancements that improve service availability during rolling deployments. These enhancements help maintain availability when new application

📅 Nov 14, 2025 • 📰 CloudFormation Updates

🔗 Read more

📄 Making the Most of Your Docker Hardened Images Trial – Part 1

First steps: Run your first secure, production-ready image Container base images form the foundation of your application security. When those foundations contain vulnerabilities, every service built o

📅 Nov 14, 2025 • 📰 Docker Blog

🔗 Read more

📄 Investigating the Great AI Productivity Divide: Why Are Some Developers 5x Faster?

AI-powered developer tools claim to boost your productivity, doing everything from intelligent auto-complete to (https://openai.com/index/introducing-codex/). But the productivity gains users report h

📅 Nov 14, 2025 • 📰 Docker Blog

🔗 Read more

📄 Cagent Comes to Docker Desktop with Built-In IDE Support through ACP

Docker Desktop now includes cagent bundled out of the box. This means developers can start building AI agents without a separate installation step. For those unfamiliar with cagent: it’s Docker’s open

📅 Nov 13, 2025 • 📰 Docker Blog

🔗 Read more

📄 MCP Horror Stories: The WhatsApp Data Exfiltration Attack

This is Part 5 of our MCP Horror Stories series, where we examine real-world security incidents that highlight the critical vulnerabilities threatening AI infrastructure and demonstrate how Docker’s c

📅 Nov 13, 2025 • 📰 Docker Blog

🔗 Read more

📄 Improving modern software supply chain security: From AI models to container images

The software supply chain has evolved dramatically in recent years. Today's applications integrate countless components—from open source libraries and container images to AI models and training datase

📅 Nov 13, 2025 • 📰 OpenShift Blog

🔗 Read more

📄 Announcing 2025 OpenTelemetry Community Awards Winners

We are excited to announce the winners of the second OpenTelemetry Community Awards! These awards recognize individuals who have made a notable impact to the OpenTelemetry project over the past year,

📅 Nov 13, 2025 • 📰 OpenTelemetry Blog

🔗 Read more

📄 OpenFGA Becomes a CNCF Incubating Project

The CNCF Technical Oversight Committee (TOC) has voted to accept OpenFGA as a CNCF incubating project. What is OpenFGA? OpenFGA is an authorization engine that addresses the challenge of implementing

📅 Nov 11, 2025 • 📰 CNCF Blog

🔗 Read more

📄 Lima becomes a CNCF incubating project

The CNCF Technical Oversight Committee (TOC) has voted to accept Lima as a CNCF incubating project. Lima enables secure, isolated environments for running cloud native and AI workloads. What is Lima?

📅 Nov 11, 2025 • 📰 CNCF Blog

🔗 Read more

📄 KServe becomes a CNCF incubating project

The CNCF Technical Oversight Committee (TOC) has voted to accept KServe as a CNCF incubating project. KServe joins a growing ecosystem of technologies tackling real-world challenges at the edge of clo

📅 Nov 11, 2025 • 📰 CNCF Blog

🔗 Read more

🔄 CI/CD

📄 Four Shades of Progressive Delivery | Blog

Explore the four shades of Progressive Delivery to improve release safety, limit risk, accelerate learning, and drive sustainable software delivery. | Blog

📅 Feb 27, 2026 • 📰 Harness Blog

🔗 Read more

📄 Making experimentation work for product managers

LaunchDarkly Experimentation is the missing puzzle piece in the PM workflow.

📅 Nov 17, 2025 • 📰 LaunchDarkly Blog

🔗 Read more

📄 How to run experiments on high-traffic websites & apps

Running experiments on high-traffic websites creates a unique paradox.

📅 Nov 17, 2025 • 📰 LaunchDarkly Blog

🔗 Read more

📄 Why MABs are not just fancy A/B tests

Know when it’s smarter to let a bandit optimize in real time.

📅 Nov 17, 2025 • 📰 LaunchDarkly Blog

🔗 Read more

📄 RTO vs RPO: Key differences for modern disaster recovery

Recovery Time Objective (RTO) and Recovery Point Objective (RPO) are fundamental metrics.

📅 Nov 17, 2025 • 📰 LaunchDarkly Blog

🔗 Read more

📄 Highlights from Git 2.52

The open source Git project just released Git 2.52. Here is GitHub’s look at some of the most interesting features and changes introduced since last time. The post Highlights from Git 2.52 appeared fi

📅 Nov 17, 2025 • 📰 GitHub Blog

🔗 Read more

📄 GitLab engineer: How I improved my onboarding experience with AI

Starting a new job is exciting, and overwhelming. New teammates, new tools, and, in GitLab’s case, a lot of documentation. Six weeks ago, I joined GitLab’s Growth team as a fullstack engineer. Anyone

📅 Nov 17, 2025 • 📰 GitLab Blog

🔗 Read more

📄 What’s new in Git 2.52.0?

The Git project recently released Git 2.52. After a relatively short 8-week release cycle for 2.51, due to summer in the Northern Hemisphere, this release is back to the usual 12-week cycle. Let’s loo

📅 Nov 17, 2025 • 📰 GitLab Blog

🔗 Read more

📄 Behind the queues: How Kueue reimagines scheduling in Red Hat OpenShift

In a modern cluster, the hardest problem isn’t running workloads—it's sharing resources fairly. Red Hat OpenShift clusters are seeing a surge of AI-accelerated workloads, from GPU-intensive training j

📅 Nov 17, 2025 • 📰 Red Hat Blog

🔗 Read more

📄 Unlocking the full power of Copilot code review: Master your instructions files

Discover practical tips, examples, and best practices for writing effective instructions files. Whether you’re new or experienced, you’ll find something to level up your code reviews. The post Unlocki

📅 Nov 14, 2025 • 📰 GitHub Blog

🔗 Read more

📄 Survey Sees AI Coding Creating Need for More Software Engineers

A GitLab survey of 3,266 DevSecOps professionals shows AI is boosting code creation but increasing the need for skilled engineers, compliance challenges and human oversight.

📅 Nov 14, 2025 • 📰 DevOps.com

🔗 Read more

📄 GitHub Availability Report: October 2025

In October, we experienced four incidents that resulted in degraded performance across GitHub services. The post GitHub Availability Report: October 2025 appeared first on The GitHub Blog.

📅 Nov 13, 2025 • 📰 GitHub Blog

🔗 Read more

🏗️ IaC

📄 AWS Transform Automates Landing Zone Acceleration Network Configuration

AWS Transform for VMware now allows customers to automatically generate network configurations that can be directly imported into the Landing Zone Accelerator on AWS solution (LZA). Building on AWS Tr

📅 Nov 17, 2025 • 📰 CloudFormation Updates

🔗 Read more

📄 Automating Chaos Engineering with Terraform | Blog

Automate chaos engineering with Terraform: version-controlled infrastructure, service discovery, security governance, and ChaosHub management for resilient systems. | Blog

📅 Nov 17, 2025 • 📰 Harness Blog

🔗 Read more

📊 Observability

📄 The Declarative configuration journey: Why it took 5 years to ignore health check endpoints in tracing

One of the most persistent and popular feature requests for Java OpenTelemetry over the past couple of years has been the ability to efficiently drop spans for health check endpoints – or any other lo

📅 Nov 17, 2025 • 📰 OpenTelemetry Blog

🔗 Read more

📄 Agentic AI in Observability: Building Resilient, Accountable IT Systems

As enterprise IT systems grow more complex, maintaining visibility, performance and resilience across distributed architectures has never been more critical. The post Agentic AI in Observability: Buil

📅 Nov 17, 2025 • 📰 The New Stack

🔗 Read more

📄 Accelerate your autonomous IT operations journey with Dynatrace and ServiceNow integrations

For many teams, the path to automation starts with connecting data and workflows across platforms. Dynatrace and ServiceNow make that possible. Through a growing set of integrations, customers can sea

📅 Nov 17, 2025 • 📰 Dynatrace Blog

🔗 Read more

📄 New Relic successfully completes Type 2 SOC 1 attestation

New Relic achieves Type 2 SOC 1 attestation with ISAE 3402 attestation.

📅 Nov 17, 2025 • 📰 New Relic Blog

🔗 Read more

📄 New Relic Named a Leader in IDC MarketScape for Worldwide Observability Report

Discover why New Relic was recognized as a Leader in the inaugural IDC MarketScape: Worldwide Observability 2025 Vendor Assessment (#US53004325, November 2025). Learn about our platform's key strength

📅 Nov 17, 2025 • 📰 New Relic Blog

🔗 Read more

📄 Sentry has a bold new look

As you may have noticed, Sentry just got a major glow-up. For too long our product looked like boring enterprise software, while our brand screamed bold and irr...

📅 Nov 17, 2025 • 📰 Sentry Blog

🔗 Read more

📄 Announcing agreement EventBridge notifications for AWS Marketplace

AWS Marketplace now delivers purchase agreement events via Amazon EventBridge, transitioning from our Amazon Simple Notification Service (SNS) notifications for Software as a Service and Professional

📅 Nov 14, 2025 • 📰 CloudFormation Updates

🔗 Read more

📄 A Star Wars dashboard deep dive: How to build your next visualization in less than 12 parsecs

Hello there! Earlier this year I created a Star Wars-themed dashboard featuring all sorts of fun visualizations from a galaxy far, far away, including a map of the Kessel Run, an overview of the Mille

📅 Nov 14, 2025 • 📰 Grafana Blog

🔗 Read more

📄 Continuous profiling for native code: Understanding the what, why, and how

It’s hard to imagine deploying any application today without observability. Logs have been around since the early days of mainframes, metrics became standard with early Unix systems, and tracing gaine

📅 Nov 14, 2025 • 📰 Grafana Blog

🔗 Read more

📄 AI’s biggest impact isn’t cranking out more code: Why DevEx matters

Developer experience (DevEx) is more than ping pong tables and free snacks. It’s the foundation for building great software. As AI reshapes the way we code, the real opportunity lies not in writing co

📅 Nov 13, 2025 • 📰 Dynatrace Blog

🔗 Read more

📄 Understand, diagnose, and optimize SQL queries: Introducing Grafana Cloud Database Observability

It’s widely acknowledged that most application performance problems stem not from the application itself, but from the underlying database. Slow or inefficient database queries are often the primary c

📅 Nov 13, 2025 • 📰 Grafana Blog

🔗 Read more

📄 GPU Observability: Get Deeper Insights into Your Droplets and DOKS Clusters

We’re introducing a new set of basic observability metrics for all GPU Droplets and DOKS clusters, giving you a powerful, simple way to monitor and optimize your AI workloads. Why GPU Observability Ma

📅 Nov 12, 2025 • 📰 DigitalOcean Blog

🔗 Read more

🔐 Security

📄 Threats Making WAVs - Incident Response to a Cryptomining Attack

Guardicore security researchers describe and uncover a full analysis of a cryptomining attack, which hid a cryptominer inside WAV files. The report includes the full attack vectors, from detection, in

📅 Nov 17, 2025 • 📰 Linode Blog

🔗 Read more

📄 Mark Your Calendars! Qodana Events in November

As 2025 starts racing to a close, the JetBrains Qodana team is ending the year with a series of events dedicated to what matters most in modern software development: secure workflows, high-quality cod

📅 Nov 17, 2025 • 📰 JetBrains Blog

🔗 Read more

📄 Alliander modernises its electricity grid with Red Hat for long-term reliability in balance with rapid innovation

Long‑lasting transformers and short‑lifecycle IT are not a natural pairing. Utilities run process installations designed to last for decades, while the control software ages far faster. Deferring repl

📅 Nov 17, 2025 • 📰 Red Hat Blog

🔗 Read more

📄 The Power of a Multi-Layered Security Strategy

Over time, tool sprawl has become the default state of enterprise security. Many organizations run dozens of security products across cloud workloads, on-premises systems and edge deployments. Various

📅 Nov 14, 2025 • 📰 SUSE Blog

🔗 Read more

📄 A CISO’s preview of open source and cybersecurity trends in 2026 and beyond

Where is open source going next? What’s in store for open source in the coming years, particularly in relation to security? Here’s a CISO’s reflection on the state of open source, and the trends that

📅 Nov 14, 2025 • 📰 Ubuntu Blog

🔗 Read more

📄 Managing SUSE Workloads in AWS – When do you need a private repository?

This article applies to customers running on-demand/PAYG instances in AWS deployed from AMIs published by SUSE. The need to patch. Patching is an important part of managing any OS infrastructure with

📅 Nov 13, 2025 • 📰 SUSE Blog

🔗 Read more

📄 Canonical expands total coverage for Ubuntu LTS releases to 15 years with Legacy add-on

Ubuntu Pro now supports LTS releases for up to 15 years through the Legacy add-on. More security, more stability, and greater control over upgrade timelines for enterprises.

📅 Nov 13, 2025 • 📰 Ubuntu Blog

🔗 Read more

📄 Automated Package-Publication Incident IndonesianFoods in the NPM Ecosystem Linked to Crypto Reward-Farming Scam

In November 2025, a large-scale surge of package publications on the NPM registry with similar structures and naming patterns was discovered. Understand the details of the incident.

📅 Nov 13, 2025 • 📰 Snyk Blog

🔗 Read more

📄 PostgreSQL 18.1, 17.7, 16.11, 15.15, 14.20, and 13.23 Released!

The PostgreSQL Global Development Group has released an update to all supported versions of PostgreSQL, including 18.1, 17.7, 16.11, 15.15, 14.20, and 13.23. This release fixes 2 security vulnerabilit

📅 Nov 13, 2025 • 📰 PostgreSQL News

🔗 Read more

📄 Organizations Achieve 288% ROI with The Snyk AI Trust Platform, According to New Forrester TEI Study

Organizations achieve 288% ROI with the Snyk AI Trust Platform, according to a new Forrester TEI study. Improve developer productivity, enhance security, and consolidate AppSec tools with Snyk.

📅 Nov 12, 2025 • 📰 Snyk Blog

🔗 Read more

📄 The AI Visibility Problem: When Speed Outruns Security | Blog

Harness surveyed 500 security practitioners and decision makers responsible for securing AI-native applications from the United States, UK, Germany, and France to share findings on global security pra

📅 Nov 12, 2025 • 📰 Harness Blog

🔗 Read more

📄 Secure by Design: The Future of Threat Modeling for AI-Native Applications

Explore how Snyk's Evo Threat Modeling Agent automates and contextualizes security for AI-native applications, addressing prompt injection, data exfiltration, data poisoning, and agentic vulnerabiliti

📅 Nov 11, 2025 • 📰 Snyk Blog

🔗 Read more

💾 Databases

📄 Database Features in Rider

Rider provides robust support for working with databases directly within the IDE – powered by DataGrip! Rider and DataGrip smooth development workflows by allowing you to manage and interact with a va

📅 Nov 17, 2025 • 📰 JetBrains Blog

🔗 Read more

📄 pgAdmin 4 v9.10 Released

The pgAdmin Development Team is pleased to announce pgAdmin 4 version 9.10. This release of pgAdmin 4 includes 5 new features and 12 bug fixes/housekeeping changes. For more details, please see the re

📅 Nov 17, 2025 • 📰 PostgreSQL News

🔗 Read more

📄 Call for new members PostgreSQL Community Code of Conduct Committee

This message is being sent from the Community Code of Conduct Committee, with the approval of the Core Team. As part of the Community CoC policy, the Committee membership is to be refreshed on an annu

📅 Nov 17, 2025 • 📰 PostgreSQL News

🔗 Read more

📄 RedisVL momentum & what’s next

Redis, a longtime favorite of developers, is seeing unprecedented velocity around AI use cases. RedisVL, initially a simple convenience layer for vector search, has quickly evolved into the AI-native

📅 Nov 14, 2025 • 📰 Redis Blog

🔗 Read more

📄 The Cost of Multitenancy

DynamoDB and ScyllaDB share many similarities, but DynamoDB is a multi-tenant database, while ScyllaDB is single-tenant The recent DynamoDB outage is a stark reminder that even the most reliable and m

📅 Nov 14, 2025 • 📰 ScyllaDB Blog

🔗 Read more

📄 Cache vs. Database: How Architecture Impacts Performance

Lessons learned comparing Memcached with ScyllaDB

📅 Nov 12, 2025 • 📰 ScyllaDB Blog

🔗 Read more

📄 PostgreSQL Conference Germany 2026

PGConf.de 2026 is the next iteration of the PostgreSQL Conference Germany. It takes place in the "Haus der Technik" in Essen. https://2026.pgconf.de/ The conference will take place on April 21 & 22, 2

📅 Nov 12, 2025 • 📰 PostgreSQL News

🔗 Read more

🌐 Platforms

📄 Keep Your Tech Flame Alive: Trailblazer Rachel Bayley

In this Akamai FLAME Trailblazer blog post, Rachel Bayley encourages women to step into the unknown and to be their authentic selves.

📅 Nov 17, 2025 • 📰 Linode Blog

🔗 Read more

📄 The Oracle of Delphi Will Steal Your Credentials

Our deception technology is able to reroute attackers into honeypots, where they believe that they found their real target. The attacks brute forced passwords for RDP credentials to connect to the vic

📅 Nov 17, 2025 • 📰 Linode Blog

🔗 Read more

📄 The Nansh0u Campaign – Hackers Arsenal Grows Stronger

In the beginning of April, three attacks detected in the Guardicore Global Sensor Network (GGSN) caught our attention. All three had source IP addresses originating in South-Africa and hosted by Volum

📅 Nov 17, 2025 • 📰 Linode Blog

🔗 Read more

📄 A methodical approach to agent evaluation: Building a robust quality gate

AI is shifting from single-response models to complex, multi-step agents that can reason, use tools, and complete sophisticated tasks. This increased capability means you need an evolution in how you

📅 Nov 17, 2025 • 📰 Google Cloud Blog

🔗 Read more

📄 Introducing Dhivaru and two new connectivity hubs

Today, we’re announcing Dhivaru, a new Trans-Indian Ocean subsea cable system that will connect the Maldives, Christmas Island and Oman. This investment will build on the Australia Connect initiative,

📅 Nov 17, 2025 • 📰 Google Cloud Blog

🔗 Read more

📄 Celebrating our 2025 Google Cloud Partner All-stars

At Google Cloud, we have the honor of partnering with some of the most brilliant and inventive individuals across the world. Each year, the Google Cloud Partner All-stars program honors these remarkab

📅 Nov 17, 2025 • 📰 Google Cloud Blog

🔗 Read more

📄 Frontline Intelligence: Analysis of UNC1549 TTPs, Custom Tools, and Malware Targeting the Aerospace and Defense Ecosystem

Written by: Mohamed El-Banna, Daniel Lee, Mike Stokkel, Josh Goddard Overview Last year, Mandiant published a blog post highlighting suspected Iran-nexus espionage activity targeting the aerospace, av

📅 Nov 17, 2025 • 📰 Google Cloud Blog

🔗 Read more

📄 Replicate is joining Cloudflare

Bringing Replicate’s tools into Cloudflare will continue to make our Workers Platform the best place on the Internet to build and deploy any AI or agentic workflow.

📅 Nov 17, 2025 • 📰 Cloudflare Blog

🔗 Read more

📄 Google Code Wiki Aims to Solve Documentation’s Oldest Problem

Google introduces Code Wiki, an AI-powered platform that automatically generates and maintains documentation for code repositories. With Gemini integration, live diagrams, and interactive code-aware c

📅 Nov 17, 2025 • 📰 DevOps.com

🔗 Read more

📄 SRE Weekly Issue #497

View on sreweekly.com A message from our sponsor, Costory: You didn’t sign up to do FinOps.Costory automatically explains why your cloud costs change, and reports it straight to Slack.Built for SREs w

📅 Nov 17, 2025 • 📰 SRE Weekly

🔗 Read more

📄 AWS Lambda announces Provisioned Mode for SQS event source mapping (ESM)

AWS Lambda announces Provisioned Mode for SQS event-source-mappings (ESMs) that subscribe to Amazon SQS, a feature that allows you to optimize the throughput of your SQS ESM by provisioning event poll

📅 Nov 14, 2025 • 📰 CloudFormation Updates

🔗 Read more

📄 Finding the grain of sand in a heap of Salt

We explore the fundamentals of Saltstack and how we use it at Cloudflare. We also explain how we built the infrastructure to reduce release delays due to Salt failures on the edge by over 5%.

📅 Nov 13, 2025 • 📰 Cloudflare Blog

🔗 Read more

📰 Misc

📄 Kroah-Hartman: Linux Kernel Maintainer on CRA Open Source Impact

ATLANTA — Technology companies and developers are finally realizing that they’ll need to deal with the European Union’s (EU) Cyber The post Kroah-Hartman: Linux Kernel Maintainer on CRA Open Source Im

📅 Nov 17, 2025 • 📰 The New Stack

🔗 Read more

📄 DevOps for GenAI Toronto Edition Hackathon: Unlocking New AI Market Opportunities

Toronto’s vibrant tech community is abuzz as the DevOps for GenAI Hackathon returns with John Willis, this time with ambitious new themes that reflect both global trends and uniquely Canadian innovati

📅 Nov 17, 2025 • 📰 DevOps.com

🔗 Read more

📄 Keeping GPUs Ticking Like Clockwork

For this week’s episode of The New Stack Agents, I sat down with Suresh Vasudevan, the CEO of Clockwork. I’ve The post Keeping GPUs Ticking Like Clockwork appeared first on The New Stack.

📅 Nov 17, 2025 • 📰 The New Stack

🔗 Read more

📄 Microsoft EVP: Embrace AI Agents to Rewire Business Processes Now

In September, Rajesh Jha the executive VP overseeing Microsoft’s AI-powered assistant Copilot appeared on Microsoft’s “Worklab” podcast, to discuss the The post Microsoft EVP: Embrace AI Agents to Rew

📅 Nov 17, 2025 • 📰 The New Stack

🔗 Read more

📄 How to Prepare for the Future of Programming

Start a start-up? Scratch that. Call your mom. Go back to school? Don’t go back to school. Learn to code? Don’t learn to code. Forget coding. Only prompt engineering matters now. Actually, I heard it’

📅 Nov 17, 2025 • 📰 JetBrains Blog

🔗 Read more

📄 Five Great DevOps Job Opportunities

DevOps.com is now providing a weekly DevOps jobs report through which opportunities for DevOps professionals will be highlighted as part of an effort to better serve our audience. Our goal in these ch

📅 Nov 17, 2025 • 📰 DevOps.com

🔗 Read more

📄 Why Diffusion Models Could Change Developer Workflows in 2026

Developers spend much of their time editing, refactoring, and debugging rather than producing entirely new code. Code creation tends to involve non-sequential back-and-forth refinement rather than typ

📅 Nov 17, 2025 • 📰 JetBrains Blog

🔗 Read more

📄 Unlock enterprise-grade virtualization at the edge with two-node Red Hat OpenShift and Arctera's InfoScale

From remote offices to factory floors, organizations are extending their virtualization capabilities to where data is generated and consumed, unlocking new opportunities for innovation and efficiency.

📅 Nov 17, 2025 • 📰 Red Hat Blog

🔗 Read more

📄 Data Ingestion Using Logstash: PostgreSQL to Elasticsearch

What Is Logstash? Logstash is an open-source data processing pipeline from Elastic. It is being used to ingest, transform, and ship data to different sources, including Elasticsearch, Kafka, flat file

📅 Nov 14, 2025 • 📰 DZone DevOps

🔗 Read more

📄 AI SDLC Transformation, Part 1: Where to Start?

Most engineering leaders today feel the same tension: everyone talks about “AI in software delivery,” but few know where to start. Should you launch pilots? Train teams? Complement Jira or some other

📅 Nov 13, 2025 • 📰 DZone DevOps

🔗 Read more

📄 October 2025 (version 1.106)

Learn what is new in the Visual Studio Code October 2025 Release (1.106). Read the full article

📅 Nov 12, 2025 • 📰 VS Code Blog

🔗 Read more

DevOps Weekly Digest - Week 46, 2025

Sun, 16 Nov 2025 00:00:00 GMT

📌 Handpicked by DevOps Daily - Your weekly dose of curated DevOps news and updates!

⚓ Kubernetes

📄 DxOperator from DH2i is now certified for Red Hat OpenShift 4.19

📅 Nov 14, 2025 • 📰 OpenShift Blog

🔗 Read more

📄 Red Hat Advanced Cluster Security 4.9: Security built with your workflows in mind

We’ve been dedicated to advancing Red Hat Advanced Cluster Security for Kubernetes in line with the rapid evolution of Kubernetes security. With version 4.9, we’re introducing key integrations and upd

📅 Nov 14, 2025 • 📰 Red Hat Blog

🔗 Read more

📄 Canonical Kubernetes officially included in Sylva 1.5

📅 Nov 13, 2025 • 📰 Ubuntu Blog

🔗 Read more

📄 Introducing longhornctl: The Command-Line Interface for SUSE Storage

Why a CLI? In the world of Kubernetes storage, reliability is essential - but flexibility and automation are what make a system shine. SUSE Storage already provides a powerful UI and a CRD-driven cont

📅 Nov 13, 2025 • 📰 SUSE Blog

🔗 Read more

📄 KubeCon NA 2025 Recap: The Dawn of the AI Native Era | Blog

📅 Nov 13, 2025 • 📰 Harness Blog

🔗 Read more

📄 Beyond YAML in Kubernetes: The 2026 Automation Era

Kubernetes continues to evolve, powering not only applications but entire AI and ML systems across clouds, edges, and enterprises. By 2026, DevOps engineers, SREs, cloud engineers, and platform teams

📅 Nov 12, 2025 • 📰 Pulumi Blog

🔗 Read more

📄 Ingress NGINX Retirement: What You Need to Know

📅 Nov 11, 2025 • 📰 Kubernetes Blog

🔗 Read more

📄 Canonical releases FIPS-enabled Kubernetes

📅 Nov 11, 2025 • 📰 Ubuntu Blog

🔗 Read more

📄 KubeCon 2025: Redefining Kubernetes and AI observability

Discover how New Relic delivers one-step Kubernetes and AI observability with OpenTelemetry, eBPF, and AI-powered insights-live at KubeCon 2025.

📅 Nov 10, 2025 • 📰 New Relic Blog

🔗 Read more

📄 Secure AI agent deployment to GKE

Building AI agents is exciting, but deploying them securely to production shouldn't be complicated. In this tutorial, you will learn how GitLab's native Google Cloud integration makes it straightforwa

📅 Nov 10, 2025 • 📰 GitLab Blog

🔗 Read more

📄 Announcing the 2025 Steering Committee Election Results

The 2025 Steering Committee Election is now complete. The Kubernetes Steering Committee consists of 7 seats, 4 of which were up for election in 2025. Incoming committee members serve a term of 2 years

📅 Nov 9, 2025 • 📰 Kubernetes Blog

🔗 Read more

☁️ Cloud Native

📄 Amazon ECS improves Service Availability during Rolling deployments

📅 Nov 14, 2025 • 📰 CloudFormation Updates

🔗 Read more

📄 Making the Most of Your Docker Hardened Images Trial – Part 1

📅 Nov 14, 2025 • 📰 Docker Blog

🔗 Read more

📄 Investigating the Great AI Productivity Divide: Why Are Some Developers 5x Faster?

📅 Nov 14, 2025 • 📰 Docker Blog

🔗 Read more

📄 When Memory Overflows: Too Many ApplicationContexts in Spring Integration Tests

In Spring, the ApplicationContext is the central container object that manages all beans (i.e., components, services, repositories, etc.). Its tasks include reading the configuration (Java Config, XML

📅 Nov 13, 2025 • 📰 DZone DevOps

🔗 Read more

📄 Cagent Comes to Docker Desktop with Built-In IDE Support through ACP

📅 Nov 13, 2025 • 📰 Docker Blog

🔗 Read more

📄 MCP Horror Stories: The WhatsApp Data Exfiltration Attack

📅 Nov 13, 2025 • 📰 Docker Blog

🔗 Read more

📄 Improving modern software supply chain security: From AI models to container images

The software supply chain has evolved dramatically in recent years. Today's applications integrate countless components-from open source libraries and container images to AI models and training datase

📅 Nov 13, 2025 • 📰 OpenShift Blog

🔗 Read more

📄 Announcing 2025 OpenTelemetry Community Awards Winners

We are excited to announce the winners of the second OpenTelemetry Community Awards! These awards recognize individuals who have made a notable impact to the OpenTelemetry project over the past year,

📅 Nov 13, 2025 • 📰 OpenTelemetry Blog

🔗 Read more

📄 OpenFGA Becomes a CNCF Incubating Project

The CNCF Technical Oversight Committee (TOC) has voted to accept OpenFGA as a CNCF incubating project. What is OpenFGA? OpenFGA is an authorization engine that addresses the challenge of implementing

📅 Nov 11, 2025 • 📰 CNCF Blog

🔗 Read more

📄 Lima becomes a CNCF incubating project

The CNCF Technical Oversight Committee (TOC) has voted to accept Lima as a CNCF incubating project. Lima enables secure, isolated environments for running cloud native and AI workloads. What is Lima?

📅 Nov 11, 2025 • 📰 CNCF Blog

🔗 Read more

📄 KServe becomes a CNCF incubating project

📅 Nov 11, 2025 • 📰 CNCF Blog

🔗 Read more

📄 Total Cost of Ownership and Unit Costs: Creating a Strategic Lens for IT Investment Decisions

Expanding technology portfolios make it harder than ever for organizations to understand and optimize the value of their IT investments. Despite the promise of digital transformation, many initiatives

📅 Nov 10, 2025 • 📰 Kubecost Blog

🔗 Read more

🔄 CI/CD

📄 Four Shades of Progressive Delivery | Blog

Explore the four shades of Progressive Delivery to improve release safety, limit risk, accelerate learning, and drive sustainable software delivery. | Blog

📅 Feb 27, 2026 • 📰 Harness Blog

🔗 Read more

📄 Delivering adaptive AI with LaunchDarkly and Snowflake Cortex

LaunchDarkly & Snowflake enable AI delivery with real-time config and runtime safety.

📅 Nov 16, 2025 • 📰 LaunchDarkly Blog

🔗 Read more

📄 New Experimentation tools for PMs who test, learn, and move fast

Test, learn, and ship faster with new Experimentation tools.

📅 Nov 16, 2025 • 📰 LaunchDarkly Blog

🔗 Read more

📄 Less clutter, more control: Manage flag permissions at scale

Preset Role Scope and Flag Lifecycle Settings can help you issue cleaner, faster releases.

📅 Nov 16, 2025 • 📰 LaunchDarkly Blog

🔗 Read more

📄 Unlocking the full power of Copilot code review: Master your instructions files

📅 Nov 14, 2025 • 📰 GitHub Blog

🔗 Read more

📄 Survey Sees AI Coding Creating Need for More Software Engineers

A GitLab survey of 3,266 DevSecOps professionals shows AI is boosting code creation but increasing the need for skilled engineers, compliance challenges and human oversight.

📅 Nov 14, 2025 • 📰 DevOps.com

🔗 Read more

📄 GitHub Availability Report: October 2025

In October, we experienced four incidents that resulted in degraded performance across GitHub services. The post GitHub Availability Report: October 2025 appeared first on The GitHub Blog.

📅 Nov 13, 2025 • 📰 GitHub Blog

🔗 Read more

📄 TypeScript, Python, and the AI feedback loop changing software development

An interview with the leader of GitHub Next, Idan Gazit, on TypeScript, Python, and what comes next. The post TypeScript, Python, and the AI feedback loop changing software development appeared first

📅 Nov 13, 2025 • 📰 GitHub Blog

🔗 Read more

📄 Achieve CMMC Level 2 with GitLab Dedicated for Government

For Defense Industrial Base (DIB) companies, the U.S. Department of Defense's release of the Cybersecurity Maturity Model Certification (CMMC) Final Rule and new guidance on “FedRAMP equivalency” has

📅 Nov 12, 2025 • 📰 GitLab Blog

🔗 Read more

📊 Observability

📄 Announcing agreement EventBridge notifications for AWS Marketplace

AWS Marketplace now delivers purchase agreement events via Amazon EventBridge, transitioning from our Amazon Simple Notification Service (SNS) notifications for Software as a Service and Professional

📅 Nov 14, 2025 • 📰 CloudFormation Updates

🔗 Read more

📄 GitOps in the Wild: Scaling Continuous Delivery in Hybrid Cloud Environments

A deep dive into scaling GitOps for hybrid and multi-cloud environments, exploring architecture, governance, security, observability and real-world enterprise practices.

📅 Nov 14, 2025 • 📰 DevOps.com

🔗 Read more

📄 A Star Wars dashboard deep dive: How to build your next visualization in less than 12 parsecs

📅 Nov 14, 2025 • 📰 Grafana Blog

🔗 Read more

📄 Continuous profiling for native code: Understanding the what, why, and how

📅 Nov 14, 2025 • 📰 Grafana Blog

🔗 Read more

📄 Chronosphere Adds AI Remediation Guidance to Observability Platform

Chronosphere this week previewed artificial intelligence (AI) capabilities that are embedded into its observability platform that, in addition to helping identify the root cause of an issue, also prov

📅 Nov 13, 2025 • 📰 DevOps.com

🔗 Read more

📄 AI’s biggest impact isn’t cranking out more code: Why DevEx matters

📅 Nov 13, 2025 • 📰 Dynatrace Blog

🔗 Read more

📄 Understand, diagnose, and optimize SQL queries: Introducing Grafana Cloud Database Observability

📅 Nov 13, 2025 • 📰 Grafana Blog

🔗 Read more

📄 GPU Observability: Get Deeper Insights into Your Droplets and DOKS Clusters

📅 Nov 12, 2025 • 📰 DigitalOcean Blog

🔗 Read more

📄 The Modern Retailer’s Playbook for In-Store IT

Unlock flawless in-store experiences with intelligent observability. Prevent POS issues, optimize inventory, and delight customers with real-time insights for retail IT.

📅 Nov 12, 2025 • 📰 New Relic Blog

🔗 Read more

📄 Performance testing best practices: How to prepare for peak demand with Grafana Cloud k6

For many organizations, periods of high customer activity are anything but relaxing. Events like Black Friday, product launches, or major sales can put intense strain on the software and infrastructur

📅 Nov 12, 2025 • 📰 Grafana Blog

🔗 Read more

📄 Using Sentry Logs to Debug a Dynamic Sampling Issue

Some of the team at Sentry spent this past quarter fixing bugs, more than 800 to be exact. Among them was a complex issue causing transaction spikes in our own ...

📅 Nov 12, 2025 • 📰 Sentry Blog

🔗 Read more

📄 Monitoring Website Changes with Zabbix Browser Item

In today’s digital era, information is an asset and most of it is obtained from websites. The ability to… The post Monitoring Website Changes with Zabbix Browser Item appeared first on Zabbix Blog.

📅 Nov 11, 2025 • 📰 Zabbix Blog

🔗 Read more

🔐 Security

📄 Threats Making WAVs - Incident Response to a Cryptomining Attack

📅 Nov 16, 2025 • 📰 Linode Blog

🔗 Read more

📄 Dissecting Flock Safety: The Cameras Tracking You Are a Security Nightmare

Comments

📅 Nov 16, 2025 • 📰 Hacker News

🔗 Read more

📄 Portmaster, an Open Source Firewall for the Desktop

Security should never be taken for granted, no matter what operating system you use. You might have a Linux desktop The post Portmaster, an Open Source Firewall for the Desktop appeared first on The N

📅 Nov 16, 2025 • 📰 The New Stack

🔗 Read more

📄 The Power of a Multi-Layered Security Strategy

📅 Nov 14, 2025 • 📰 SUSE Blog

🔗 Read more

📄 AWS IoT Services expand support of VPC endpoints and IPv6 connectivity

AWS IoT Core, AWS IoT Device Management, and AWS IoT Device Defender have expanded support for Virtual Private Cloud (VPC) endpoints and IPv6. Developers can now use AWS PrivateLink to establish VPC e

📅 Nov 14, 2025 • 📰 CloudFormation Updates

🔗 Read more

📄 A CISO’s preview of open source and cybersecurity trends in 2026 and beyond

📅 Nov 14, 2025 • 📰 Ubuntu Blog

🔗 Read more

📄 Managing SUSE Workloads in AWS – When do you need a private repository?

This article applies to customers running on-demand/PAYG instances in AWS deployed from AMIs published by SUSE. The need to patch. Patching is an important part of managing any OS infrastructure with

📅 Nov 13, 2025 • 📰 SUSE Blog

🔗 Read more

📄 Canonical expands total coverage for Ubuntu LTS releases to 15 years with Legacy add-on

Ubuntu Pro now supports LTS releases for up to 15 years through the Legacy add-on. More security, more stability, and greater control over upgrade timelines for enterprises.

📅 Nov 13, 2025 • 📰 Ubuntu Blog

🔗 Read more

📄 Automated Package-Publication Incident IndonesianFoods in the NPM Ecosystem Linked to Crypto Reward-Farming Scam

In November 2025, a large-scale surge of package publications on the NPM registry with similar structures and naming patterns was discovered. Understand the details of the incident.

📅 Nov 13, 2025 • 📰 Snyk Blog

🔗 Read more

📄 PostgreSQL 18.1, 17.7, 16.11, 15.15, 14.20, and 13.23 Released!

📅 Nov 13, 2025 • 📰 PostgreSQL News

🔗 Read more

📄 Organizations Achieve 288% ROI with The Snyk AI Trust Platform, According to New Forrester TEI Study

Organizations achieve 288% ROI with the Snyk AI Trust Platform, according to a new Forrester TEI study. Improve developer productivity, enhance security, and consolidate AppSec tools with Snyk.

📅 Nov 12, 2025 • 📰 Snyk Blog

🔗 Read more

📄 The AI Visibility Problem: When Speed Outruns Security | Blog

📅 Nov 12, 2025 • 📰 Harness Blog

🔗 Read more

💾 Databases

📄 A new top score: Advancing Text-to-SQL on the BIRD benchmark

In the fast-evolving world of agentic development, natural language is becoming the standard for interaction. This shift is deeply connected to the power of operational databases, where a more accurat

📅 Nov 14, 2025 • 📰 Google Cloud Blog

🔗 Read more

📄 Waze keeps traffic flowing with 1M+ real-time reads per second on Memorystore

Editor’s note: Waze (a division of Google parent company Alphabet) depends on vast volumes of dynamic, real-time user session data to power its core navigation features, but scaling that data to suppo

📅 Nov 14, 2025 • 📰 Google Cloud Blog

🔗 Read more

📄 RedisVL momentum & what’s next

Redis, a longtime favorite of developers, is seeing unprecedented velocity around AI use cases. RedisVL, initially a simple convenience layer for vector search, has quickly evolved into the AI-native

📅 Nov 14, 2025 • 📰 Redis Blog

🔗 Read more

📄 The Cost of Multitenancy

📅 Nov 14, 2025 • 📰 ScyllaDB Blog

🔗 Read more

📄 Cache vs. Database: How Architecture Impacts Performance

Lessons learned comparing Memcached with ScyllaDB

📅 Nov 12, 2025 • 📰 ScyllaDB Blog

🔗 Read more

📄 PostgreSQL Conference Germany 2026

📅 Nov 12, 2025 • 📰 PostgreSQL News

🔗 Read more

📄 CloudNativePG 1.28.0 RC1 Released!

The CloudNativePG Community is thrilled to announce the first release candidate of CloudNativePG 1.28! This preview release provides an opportunity to explore new features and enhancements before the

📅 Nov 10, 2025 • 📰 PostgreSQL News

🔗 Read more

🌐 Platforms

📄 Join us at AWS re:Invent 2025

Visit us at booth #1339!

📅 Nov 16, 2025 • 📰 LaunchDarkly Blog

🔗 Read more

📄 Keep Your Tech Flame Alive: Trailblazer Rachel Bayley

In this Akamai FLAME Trailblazer blog post, Rachel Bayley encourages women to step into the unknown and to be their authentic selves.

📅 Nov 16, 2025 • 📰 Linode Blog

🔗 Read more

📄 The Oracle of Delphi Will Steal Your Credentials

📅 Nov 16, 2025 • 📰 Linode Blog

🔗 Read more

📄 The Nansh0u Campaign – Hackers Arsenal Grows Stronger

📅 Nov 16, 2025 • 📰 Linode Blog

🔗 Read more

📄 Snapchat Open Sources Cross-Platform UI Framework

For eight years, Snapchat has used Valdi, a cross-platform UI framework, in-house. Now, it has open sourced the framework. Valdi The post Snapchat Open Sources Cross-Platform UI Framework appeared fir

📅 Nov 15, 2025 • 📰 The New Stack

🔗 Read more

📄 AWS Lambda announces Provisioned Mode for SQS event source mapping (ESM)

📅 Nov 14, 2025 • 📰 CloudFormation Updates

🔗 Read more

📄 Zeotap: How BigQuery ML and vector search help customers build their own AI models

Editor’s note: The post is part of a series that highlights how organizations leverage Google Cloud’s unique data science capabilities over alternative cloud data platforms. Google Cloud's vector embe

📅 Nov 14, 2025 • 📰 Google Cloud Blog

🔗 Read more

📄 Google Cloud Networking under the hood: How Protective ReRoute increases resilience

Cloud infrastructure reliability is foundational, yet even the most sophisticated global networks can suffer from a critical issue: slow or failed recovery from routing outages. In massive, planetary-

📅 Nov 14, 2025 • 📰 Google Cloud Blog

🔗 Read more

📄 How Hyperconnected AI Development Creates a Multi-System Secret Sprawl

If you’ve been building artificial intelligence (AI) tools lately, you’ve probably noticed something: Your development workflow has become incredibly connected. Tools such as model context protocol (M

📅 Nov 14, 2025 • 📰 DevOps.com

🔗 Read more

📄 Friday Five - November 14, 2025

KubeCon + CloudNativeCon NewsroomAt KubeCon + CloudNativeCon, Red Hat is redefining the modern application platform to unite your IT estate: from legacy VMs to next-generation AI workloads. Check out

📅 Nov 14, 2025 • 📰 Red Hat Blog

🔗 Read more

📄 Finding the grain of sand in a heap of Salt

We explore the fundamentals of Saltstack and how we use it at Cloudflare. We also explain how we built the infrastructure to reduce release delays due to Salt failures on the edge by over 5%.

📅 Nov 13, 2025 • 📰 Cloudflare Blog

🔗 Read more

📄 Microsoft Ignite 2025: Turn complexity into a strategic asset in the age of AI

Editor’s Note Microsoft Ignite represents more than just another industry conference. It’s where the future of enterprise technology takes shape. This year’s event showcases the intersection of cloud

📅 Nov 13, 2025 • 📰 Dynatrace Blog

🔗 Read more

📰 Misc

📄 Only three kinds of AI products work

Comments

📅 Nov 16, 2025 • 📰 Hacker News

🔗 Read more

📄 FPGA Based IBM-PC-XT

Comments

📅 Nov 16, 2025 • 📰 Hacker News

🔗 Read more

📄 De Bruijn Numerals

Comments

📅 Nov 16, 2025 • 📰 Hacker News

🔗 Read more

📄 AI Code Doesn’t Survive in Production: Here’s Why

I see a new demo every day that looks something like this: A single prompt generates a complete application. A The post AI Code Doesn’t Survive in Production: Here’s Why appeared first on The New Stac

📅 Nov 15, 2025 • 📰 The New Stack

🔗 Read more

📄 Visual Studio 2026 First Look: Evolution, Not Revolution

I’ve never entirely understood the Microsoft roadmap. I certainly enjoyed using Visual Studio 2022 (VS 2022) for C# and .NET The post Visual Studio 2026 First Look: Evolution, Not Revolution appeared

📅 Nov 15, 2025 • 📰 The New Stack

🔗 Read more

📄 JetBrains Is Heading to VSLive! Orlando

Next week we’ll be in Orlando, Florida for Visual Studio Live!, and if you work with .NET, we’d like to meet you. You’ll find the JetBrains booth (#118) in the conference expo area throughout the week

📅 Nov 15, 2025 • 📰 JetBrains Blog

🔗 Read more

📄 Data Ingestion Using Logstash: PostgreSQL to Elasticsearch

📅 Nov 14, 2025 • 📰 DZone DevOps

🔗 Read more

📄 IntelliJ IDEA 2025.3 💚 Spring 7

Major Spring Framework releases don’t come around often – and when they do, they tend to reshape how we build applications for years to come. Spring Framework 7 is one of those releases. It simplifies

📅 Nov 14, 2025 • 📰 JetBrains Blog

🔗 Read more

📄 AI SDLC Transformation, Part 1: Where to Start?

📅 Nov 13, 2025 • 📰 DZone DevOps

🔗 Read more

📄 How to Simply Manage Releases With Git

This post briefly outlines a simple yet clean process for managing the releases of an application via Git/GitHub. Its goal is to provide a straightforward recipe, while it's assumed that these practic

📅 Nov 13, 2025 • 📰 DZone DevOps

🔗 Read more

📄 Listening, learning, and leading: How customer feedback shapes the future of Red Hat Learning Subscription

At Red Hat, innovation begins with listening. In October 2025, the Red Hat Learning Think Tank forum brought together a group of passionate learners and leaders to listen, learn, and collaborate on Re

📅 Nov 13, 2025 • 📰 Red Hat Blog

🔗 Read more

📄 October 2025 (version 1.106)

Learn what is new in the Visual Studio Code October 2025 Release (1.106). Read the full article

📅 Nov 12, 2025 • 📰 VS Code Blog

🔗 Read more

DevOps Weekly Digest - Week 45, 2025

Sun, 02 Nov 2025 00:00:00 GMT

Handpicked by DevOps Daily.

Welcome to DevOps Weekly Digest 🚀

Welcome to the inaugural edition of the DevOps Weekly Digest! This is your new go-to resource for staying up-to-date with the rapidly evolving DevOps ecosystem.

What to Expect

Every week, we'll bring you:

Curated Updates from 250+ trusted DevOps sources
AI-Powered Summaries of the most important news
Organized Categories covering Kubernetes, Cloud Native, CI/CD, IaC, Observability, Security, and more
Quality Over Quantity - only the most relevant, actionable content

Our Coverage

We monitor the entire DevOps landscape, including:

Kubernetes & Cloud Native

Official Kubernetes releases and updates
CNCF project announcements
Service mesh innovations (Istio, Linkerd, Cilium)
Container runtime developments

CI/CD & Automation

GitLab, GitHub, Jenkins updates
GitOps tools (Argo, Flux, Tekton)
Pipeline automation innovations
Progressive delivery platforms

Infrastructure as Code

Terraform, Pulumi, Ansible releases
Cloud provider IaC updates
Configuration management trends

Observability

Prometheus, Grafana, and OpenTelemetry news
APM and monitoring platform updates
Logging and tracing innovations

Security

Container and Kubernetes security
Secrets management updates
Policy enforcement tools
Vulnerability scanning advances

Cloud Platforms

AWS, GCP, Azure announcements
Multi-cloud and hybrid cloud solutions
Serverless and edge computing

And Much More

Database technologies
Message queues and event streaming
Platform engineering tools
Developer experience improvements

How It Works

Our automated system:

Crawls 250+ DevOps RSS feeds and websites daily
Filters content from the past 7 days
Classifies articles using AI to ensure relevance
Summarizes each item to save you time
Organizes content by category for easy browsing
Limits items per source and category to maintain quality

Stay Connected

This digest is automatically generated and published every week. You can:

📰 Subscribe via our RSS feed
🔖 Bookmark the news page for quick access
🔗 Share individual digests with your team
💬 Report any issues or suggest improvements

Coming Next Week

Starting next week, you'll receive comprehensive weekly digests packed with:

Latest Kubernetes releases and security patches
New CNCF project graduations and updates
Major cloud provider announcements
CI/CD tool releases and features
Security vulnerabilities and fixes
Industry insights and best practices

Feedback Welcome

We're constantly improving this digest to serve you better. If you have suggestions for sources we should monitor, categories we should add, or any other feedback, please let us know.

The DevOps Weekly Digest is an automated curation powered by AI, bringing you the most relevant updates from across the DevOps ecosystem. Quality sources, intelligent filtering, and actionable insights - all in one place.

See you next week with our first full digest! 🎉

Ping Response: Request Timed Out vs Destination Host Unreachable

DevOps Daily Team — Sat, 30 Aug 2025 14:30:00 GMT

TLDR: "Request timed out" means your ping packet was sent but no response came back - the host might be down, blocking ICMP, or unreachable beyond your network. "Destination host unreachable" means a router along the path actively told you it cannot reach the destination - this is usually a routing problem or the destination network doesn't exist. Timeout is silence; unreachable is an explicit error message.

When you ping a host and it fails, the error message tells you a lot about where the problem is. The two most common responses - timeout and unreachable - indicate very different issues.

Request Timed Out

When you see "Request timed out," your computer sent the ICMP echo request but never received a reply:

$ ping 192.168.1.100
PING 192.168.1.100 (192.168.1.100): 56 data bytes
Request timeout for icmp_seq 0
Request timeout for icmp_seq 1
Request timeout for icmp_seq 2
Request timeout for icmp_seq 3

Here's what's happening:

Your Computer          Network          Target Host
      |                                      |
      |----ICMP Echo Request--------------->|
      |                                      |
      |                  ... silence ...     |
      |                                      |
      |<------ (no response) ---------------|
      |                                      |
   [timeout]

Your packet was sent, but nothing came back. This happens when:

The Host Is Down

The target machine is powered off or unreachable. Your packet arrives at the network segment but there's no host to respond:

# Pinging a host that's turned off
ping 192.168.1.50
# Request timeout - host exists on the network but is powered down

Firewall Blocking ICMP

The host is up but configured to drop ICMP echo requests. Many servers disable ping for security:

# On the target server (Linux)
# Block all ICMP echo requests
sudo iptables -A INPUT -p icmp --icmp-type echo-request -j DROP

# Now pings to this server time out
# The server receives packets but silently drops them

Windows firewall does this too:

# Windows - disable ICMP echo requests
New-NetFirewallRule -DisplayName "Block Ping" `
    -Direction Inbound -Protocol ICMPv4 -IcmpType 8 -Action Block

Network Congestion or Packet Loss

If the network is overloaded, packets might get dropped without any error message:

# High packet loss
ping google.com
PING google.com (142.250.80.46): 56 data bytes
64 bytes from 142.250.80.46: icmp_seq=0 ttl=116 time=15.2 ms
Request timeout for icmp_seq 1
64 bytes from 142.250.80.46: icmp_seq=2 ttl=116 time=14.8 ms
Request timeout for icmp_seq 3

Some packets get through, others don't - indicates congestion or wireless interference.

Routing Black Hole

Your packet reaches a router that forwards it, but somewhere downstream it gets lost with no error sent back:

Your PC -> Router 1 -> Router 2 -> Router 3 -> [black hole]
                                                No response,
                                                no error

Destination Host Unreachable

When you see "Destination host unreachable," a router is actively telling you it cannot deliver the packet:

$ ping 10.50.99.99
PING 10.50.99.99 (10.50.99.99): 56 data bytes
From 192.168.1.1 icmp_seq=0 Destination Host Unreachable
From 192.168.1.1 icmp_seq=1 Destination Host Unreachable
From 192.168.1.1 icmp_seq=2 Destination Host Unreachable

Notice the "From 192.168.1.1" - this is your gateway router telling you it cannot reach the destination:

Your Computer          Gateway Router       Target Network
      |                      |                    |
      |--ICMP Echo-------->  |                    |
      |                      |                    |
      |                      X  Can't route       |
      |                      |  to 10.50.99.99    |
      |                      |                    |
      |<--ICMP Unreachable-- |                    |
      |  (from router)       |                    |

The router sends back an ICMP "Destination Unreachable" message. This is helpful - it tells you where the problem is.

No Route to Network

The destination network doesn't exist in the router's routing table:

# Pinging a nonexistent network
ping 172.99.99.99
# Destination Host Unreachable - no route to 172.99.0.0/16

Check routing tables to confirm:

# Linux/macOS - show routing table
route -n
# or
ip route show

# Windows
route print

If there's no route to the destination network, your router can't forward the packet.

Host Unreachable on Local Network

If you're pinging a host on your local subnet and get "unreachable," the host doesn't respond to ARP requests:

# Pinging a host on the same subnet (192.168.1.0/24)
ping 192.168.1.200
# Destination Host Unreachable

# Check ARP table
arp -a
# 192.168.1.200 is not in the ARP table - host isn't on the network

The computer tried to resolve the MAC address via ARP but got no response, so it knows the host isn't reachable.

Network Interface Down

If your own network interface is down or misconfigured:

# Interface is down
ping google.com
# connect: Network is unreachable

# Check interface status
ip link show

# Bring it up
sudo ip link set eth0 up

The error comes immediately because your own system knows it can't send packets.

Comparing the Two

Here's a decision tree for diagnosing:

Ping fails
    |
    |
    ├─> "Request timed out"
    |       |
    |       ├─> Check if host is powered on
    |       ├─> Check if firewall blocks ICMP
    |       ├─> Check for packet loss (wireless, congestion)
    |       └─> Verify host is reachable beyond your network
    |
    └─> "Destination host unreachable"
            |
            ├─> Check "From" IP address
            |       |
            |       ├─> Your gateway? Routing problem
            |       ├─> Your computer? Interface down
            |       └─> Intermediate router? Network path broken
            |
            ├─> Check routing table
            ├─> Check ARP table (for local subnet)
            └─> Verify network cable/WiFi connection

Detailed Troubleshooting Examples

Example 1: Request Timeout

$ ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
Request timeout for icmp_seq 0
Request timeout for icmp_seq 1

Diagnosis steps:

# 1. Check if you have a route to 8.8.8.8
ip route get 8.8.8.8
# Output: 8.8.8.8 via 192.168.1.1 dev wlan0 src 192.168.1.10

# 2. Ping your gateway to verify local network works
ping 192.168.1.1
# 64 bytes from 192.168.1.1: icmp_seq=0 ttl=64 time=1.2 ms
# Gateway responds - local network is fine

# 3. Try another public IP
ping 1.1.1.1
# Request timeout - same issue

# 4. Try DNS resolution
nslookup google.com
# Works - DNS is fine

# Conclusion: Your ISP might be blocking ICMP, or there's packet loss
# Try TCP-based connectivity test
curl -I https://google.com
# Works - internet is up, just ICMP is blocked/filtered

Example 2: Destination Unreachable

$ ping 10.0.50.100
PING 10.0.50.100 (10.0.50.100): 56 data bytes
From 192.168.1.1 icmp_seq=0 Destination Host Unreachable

Diagnosis steps:

# 1. Check who sent the unreachable message
# "From 192.168.1.1" - this is your gateway

# 2. Check your routing table
route -n | grep 10.0.50
# No route found

# 3. Check if there should be a route
# Is 10.0.50.0/24 supposed to be accessible from your network?
# Maybe you need to be on VPN?

# 4. Connect to VPN and try again
# (After VPN connection)
route -n | grep 10.0.50
# 10.0.50.0/24 via 10.8.0.1 dev tun0

ping 10.0.50.100
# 64 bytes from 10.0.50.100: icmp_seq=0 ttl=64 time=45.2 ms
# Now it works!

# Conclusion: Network required VPN connection

Example 3: Mixed Timeout and Unreachable

Sometimes you see both:

$ ping 192.168.1.150
PING 192.168.1.150 (192.168.1.150): 56 data bytes
From 192.168.1.10 icmp_seq=0 Destination Host Unreachable
Request timeout for icmp_seq 1
From 192.168.1.10 icmp_seq=2 Destination Host Unreachable
Request timeout for icmp_seq 3

This pattern indicates:

Initially, ARP lookup fails (unreachable)
Then ARP cache times out and ping just waits (timeout)
Then ARP tries again (unreachable)

The host definitely isn't on the network.

ICMP Types Behind the Scenes

Understanding the actual ICMP messages helps:

Request Timed Out:
- Your computer sends: ICMP Type 8 (Echo Request)
- Target should send: ICMP Type 0 (Echo Reply)
- You receive: Nothing (timeout)

Destination Unreachable:
- Your computer sends: ICMP Type 8 (Echo Request)
- Router sends back: ICMP Type 3 (Destination Unreachable)
  - Code 0: Network unreachable
  - Code 1: Host unreachable
  - Code 2: Protocol unreachable
  - Code 3: Port unreachable
  - etc.

You can see these with tcpdump or Wireshark:

# Capture ICMP traffic
sudo tcpdump -i any icmp -n

# In another terminal
ping 192.168.1.100

# Output shows the ICMP types:
# > 192.168.1.10 > 192.168.1.100: ICMP echo request, id 1234, seq 0
# < 192.168.1.1 > 192.168.1.10: ICMP 192.168.1.100 host unreachable

When Ping Works But Service Doesn't

Sometimes ping succeeds but you still can't connect to a service:

# Ping works
ping database.example.com
# 64 bytes from database.example.com (10.0.1.50): icmp_seq=0 ttl=64 time=1.2 ms

# But connection fails
telnet database.example.com 5432
# telnet: Unable to connect to remote host: Connection refused

This means:

Host is reachable (ping works)
ICMP is allowed
But the specific service port (5432) is blocked or not running

Ping only tests ICMP connectivity, not TCP/UDP services.

Practical Tips

For timeouts:

Verify the host IP is correct
Check if the host is powered on
Try pinging from a different location
Use traceroute to see how far packets get
Consider that ICMP might be intentionally blocked

For unreachable:

Note which router sent the error
Check routing tables
Verify network cables and WiFi connection
Check if VPN is required
Verify the destination network exists

Better diagnostics than ping:

# See the path packets take
traceroute google.com

# Test specific TCP port connectivity
nc -zv google.com 443
# or
telnet google.com 443

# Test with actual HTTP request
curl -v https://google.com

# Full network path with MTR (better than traceroute)
mtr google.com

The key difference is that "request timed out" means silence (packet sent, no reply), while "destination unreachable" means active rejection (router explicitly saying it cannot reach the destination). Understanding this distinction helps you troubleshoot network issues faster.

The 5-Minute Kubernetes Cluster Health Check

DevOps Daily Team — Tue, 12 Aug 2025 09:00:00 GMT

TLDR

You can check your Kubernetes cluster's health in under 5 minutes using five key commands: checking node status, monitoring resource usage, reviewing pod health across namespaces, investigating problem pods, and examining cluster events. This quick routine helps catch issues before they escalate into critical problems.

Kubernetes is great until it's not. One bad node, a pod stuck in CrashLoopBackOff, or a resource spike can ruin your day. The good news? You don't need to spend an hour digging through dashboards to spot trouble early. With a few quick commands, you can get a solid read on your cluster's health in under 5 minutes.

Here's how to do it effectively.

Make Sure Your Nodes Are Happy

Start by checking the overall status of your cluster nodes. This gives you the foundation-level health of your infrastructure.

kubectl get nodes -o wide

This command displays all nodes in your cluster along with their detailed information. You'll see each node's status, roles, age, version, internal and external IPs, OS image, kernel version, and container runtime.

What you want to see:

STATUS should be Ready for all nodes
No mystery nodes suddenly showing up in your cluster
Roles, IPs, and ages that make sense for your environment

If you spot NotReady, that's your cue to dig deeper. A node in this state might be experiencing network issues, resource exhaustion, or kubelet problems.

Check Resource Usage at a Glance

Next, get a quick overview of resource consumption across your nodes to identify potential bottlenecks.

kubectl top nodes

This command shows CPU and memory usage for each node in your cluster. It provides both absolute values and percentages, making it easy to spot resource pressure.

Keep an eye out for:

CPU or memory regularly above 80% on any node
One node doing all the heavy lifting while others are barely working
Sudden spikes that don't match your expected workload patterns

No metrics-server running? Install it with this command:

kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

The metrics-server is essential for resource monitoring and is required for horizontal pod autoscaling to work properly.

Look at All Pods Across All Namespaces

Get a bird's-eye view of all pods running in your cluster to quickly identify any that are misbehaving.

kubectl get pods --all-namespaces

This command lists every pod across all namespaces, showing their current status, restart count, and age. It's like taking the pulse of your entire application ecosystem.

Healthy pods should be Running or Completed. If you see states like CrashLoopBackOff, ImagePullBackOff, Pending, or Error, note the namespace and pod name for further investigation.

Also watch the RESTARTS column closely. If a pod has restarted a dozen times in the last hour, something's definitely off. Frequent restarts often indicate:

Application crashes due to bugs or configuration issues
Failing health checks (readiness or liveness probes)
Resource limits being exceeded
Dependencies being unavailable

Zoom In on Problem Pods

When you spot problematic pods, dig deeper to understand what's causing the issues.

kubectl describe pod  -n

Replace and with the actual values from your problem pods. This command provides detailed information about the pod's configuration, current state, and recent events.

Check for these common issues:

Events at the bottom (often the smoking gun that reveals the root cause)
Failing readiness or liveness probes that prevent the pod from receiving traffic
Image pull errors indicating registry access problems or incorrect image names
Resource limit issues where the pod exceeds its memory or CPU constraints

The events section is particularly valuable because it shows a chronological history of what happened to the pod, including scheduling decisions, volume mounts, and error conditions.

Check the Cluster's Event Log

Get insight into what's been happening across your entire cluster by examining recent events.

kubectl get events --sort-by=.metadata.creationTimestamp

This command shows cluster-wide events sorted by when they occurred, giving you a timeline of recent activity. Events provide context about system-level operations and can reveal patterns or issues that affect multiple components.

Events will tell you what's been happening behind the scenes:

Failed volume mounts that prevent pods from starting
DNS resolution errors affecting service communication
Scheduling issues when pods can't be placed on nodes
Node pressure warnings indicating resource constraints

Try k9s for a Better View

If you want something more interactive than command-line tools, give k9s a try. It's a terminal-based UI for Kubernetes that provides real-time cluster information in an intuitive interface.

k9s lets you browse resources, view logs, and drill into problems without typing long commands. You can navigate between different resource types using simple keystrokes, filter resources, and even perform actions like scaling deployments or deleting pods.

Once you try k9s, it's hard to go back to plain kubectl for exploratory tasks. It's particularly useful when you need to quickly jump between different namespaces or resource types during troubleshooting.

Five minutes a day is all it takes to stay ahead of most cluster problems. Make this health check part of your daily routine and you'll catch issues before they blow up and before your pager goes off at 3 a.m. Regular monitoring helps you understand your cluster's normal behavior, making it easier to spot anomalies when they occur.

Right-Sizing Kubernetes Resources with VPA and Karpenter

DevOps Daily Team — Sun, 10 Aug 2025 09:00:00 GMT

TLDR

Setting CPU and memory requests too high in Kubernetes wastes money and reduces cluster efficiency. This guide shows you how to identify overprovisioned workloads, use Vertical Pod Autoscaler (VPA) to right-size your pods, and implement Karpenter for smarter node scaling. You'll also learn to monitor costs and validate your improvements with real metrics.

When you set resource requests too conservatively in Kubernetes, your cluster reserves more capacity than workloads actually need. This leads to underutilized nodes and higher cloud bills. The problem gets worse at scale - imagine 200 pods each requesting 2 CPU cores but only using 200m. That's 400 reserved cores when actual demand is closer to 40 cores.

The solution involves right-sizing both your pods and nodes. You'll use monitoring data to understand actual usage, apply VPA to adjust pod requests automatically, and leverage Karpenter to provision nodes that match your workload requirements.

Prerequisites

Before you start, make sure you have:

A Kubernetes cluster (version 1.20 or higher) with metrics-server installed
kubectl configured with admin access to your cluster
Prometheus and Grafana deployed for monitoring (or similar observability stack)
Basic understanding of Kubernetes resource requests and limits

You'll also need the ability to install cluster-wide components like VPA and Karpenter.

Identifying Overprovisioned Workloads

The first step is understanding how your current workloads use resources compared to what they request. You can start with kubectl to get a quick snapshot of resource usage across your cluster.

# Check current resource usage for all nodes
kubectl top nodes

# View pod resource usage across all namespaces
kubectl top pods --all-namespaces --sort-by=cpu

# Get detailed resource requests vs usage for a specific namespace
kubectl describe nodes | grep -A 15 "Allocated resources"

These commands show you the gap between requested and actual resource usage. If you see pods consistently using 50Mi of memory while requesting 1Gi, or using 100m CPU while requesting 1000m, those are prime candidates for right-sizing.

For deeper analysis, you'll want historical data from Prometheus. Here are some key queries to run in your Grafana dashboard:

# CPU utilization percentage (actual usage vs requests)
(rate(container_cpu_usage_seconds_total{container!=""}[5m]) * 100) /
(container_spec_cpu_quota{container!=""} / container_spec_cpu_period{container!=""})

# Memory utilization percentage
(container_memory_working_set_bytes{container!=""} * 100) /
container_spec_memory_limit_bytes{container!=""}

# Top 10 pods with the highest request-to-usage ratio (biggest waste)
topk(10,
  (container_spec_cpu_quota{container!=""} / container_spec_cpu_period{container!=""}) /
  rate(container_cpu_usage_seconds_total{container!=""}[5m])
)

Run these queries over a 2-week period to account for traffic variations and identify consistent patterns. Workloads running at 10-20% utilization with stable traffic are good candidates for optimization.

Installing and Configuring VPA

Vertical Pod Autoscaler analyzes your workloads and recommends optimal CPU and memory values. Start by installing VPA in your cluster.

# Clone the VPA repository
git clone https://github.com/kubernetes/autoscaler.git
cd autoscaler/vertical-pod-autoscaler

# Deploy VPA components
./hack/vpa-up.sh

This script installs three main components: the VPA recommender (analyzes usage), the updater (applies changes), and the admission controller (validates recommendations).

Next, create a VPA configuration for a workload you want to optimize. Start with recommendation mode to see suggested values before making changes.

# vpa-web-service.yaml
apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: web-service-vpa
  namespace: production
spec:
  targetRef:
    apiVersion: 'apps/v1'
    kind: Deployment
    name: web-service
  updatePolicy:
    updateMode: 'Off' # Only provide recommendations, don't auto-update
  resourcePolicy:
    containerPolicies:
      - containerName: web-app
        # Set boundaries to prevent extreme recommendations
        maxAllowed:
          cpu: '2'
          memory: '4Gi'
        minAllowed:
          cpu: '100m'
          memory: '128Mi'
        controlledResources: ['cpu', 'memory']

Apply the VPA configuration and wait for recommendations to generate:

kubectl apply -f vpa-web-service.yaml

# Wait a few minutes, then check recommendations
kubectl describe vpa web-service-vpa -n production

The output shows recommended values for CPU and memory under the Status section. VPA typically suggests values based on the 90th percentile of usage over the past 8 days, which provides a safety buffer while eliminating waste.

Applying VPA Recommendations Safely

Once you have solid recommendations, you can apply them gradually. Start with non-critical workloads and monitor for any issues.

# Update your deployment with VPA recommendations
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-service
  namespace: production
spec:
  replicas: 3
  selector:
    matchLabels:
      app: web-service
  template:
    metadata:
      labels:
        app: web-service
    spec:
      containers:
        - name: web-app
          image: nginx:1.21
          resources:
            requests:
              cpu: '250m' # Reduced from 1000m based on VPA recommendation
              memory: '512Mi' # Reduced from 2Gi based on VPA recommendation
            limits:
              cpu: '500m' # Set limits 2x requests for burst capacity
              memory: '1Gi'

After updating requests, monitor your workloads for at least a week. Watch for:

Increased pod restarts or OOMKilled events
Higher response times or error rates
Pods getting evicted under memory pressure

If everything runs smoothly, you can switch VPA to automatic mode:

# Update VPA to automatically apply changes
kubectl patch vpa web-service-vpa -n production --type='merge' -p='{"spec":{"updatePolicy":{"updateMode":"Auto"}}}'

In Auto mode, VPA will restart pods when it detects they need different resource allocations. Make sure you have proper PodDisruptionBudgets in place to maintain availability during updates.

Setting Up Karpenter for Node Optimization

While VPA optimizes individual pods, Karpenter optimizes your entire node infrastructure. Instead of fixed node groups, Karpenter provisions nodes dynamically based on your workload requirements.

First, install Karpenter in your cluster. The exact steps depend on your cloud provider, but here's the process for AWS EKS:

# Install Karpenter using Helm
helm upgrade --install karpenter oci://public.ecr.aws/karpenter/karpenter \
  --version "0.32.0" \
  --namespace "karpenter" \
  --create-namespace \
  --set "settings.clusterName=${CLUSTER_NAME}" \
  --set "settings.interruptionQueueName=${CLUSTER_NAME}" \
  --wait

Next, create a NodePool that defines what types of nodes Karpenter can provision:

# karpenter-nodepool.yaml
apiVersion: karpenter.sh/v1beta1
kind: NodePool
metadata:
  name: general-purpose
spec:
  # Template for nodes Karpenter will create
  template:
    metadata:
      labels:
        node-type: general-purpose
    spec:
      # Instance requirements - Karpenter will pick the best fit
      requirements:
        - key: kubernetes.io/arch
          operator: In
          values: ['amd64']
        - key: karpenter.sh/capacity-type
          operator: In
          values: ['spot', 'on-demand'] # Allow both for cost optimization
        - key: node.kubernetes.io/instance-type
          operator: In
          values: ['m6i.large', 'm6i.xlarge', 'm6i.2xlarge', 'r6i.large', 'r6i.xlarge']

      # Node configuration
      nodeClassRef:
        apiVersion: karpenter.k8s.aws/v1beta1
        kind: EC2NodeClass
        name: general-purpose

      # Taints to control which pods can schedule here
      taints:
        - key: karpenter.sh/unschedulable
          value: 'true'
          effect: NoSchedule

  # Scaling and disruption policies
  limits:
    cpu: 1000 # Maximum CPU across all nodes in this pool
  disruption:
    consolidationPolicy: WhenUnderutilized
    consolidateAfter: 30s

Create the corresponding EC2NodeClass for AWS-specific configuration:

# karpenter-nodeclass.yaml
apiVersion: karpenter.k8s.aws/v1beta1
kind: EC2NodeClass
metadata:
  name: general-purpose
spec:
  # AMI and instance configuration
  amiFamily: AL2
  subnetSelectorTerms:
    - tags:
        karpenter.sh/discovery: '${CLUSTER_NAME}'
  securityGroupSelectorTerms:
    - tags:
        karpenter.sh/discovery: '${CLUSTER_NAME}'

  # Instance store configuration
  userData: |
    #!/bin/bash
    /etc/eks/bootstrap.sh ${CLUSTER_NAME}

  # Tags for cost tracking
  tags:
    Team: platform
    Environment: production

Apply both configurations:

kubectl apply -f karpenter-nodepool.yaml
kubectl apply -f karpenter-nodeclass.yaml

Karpenter will now monitor unschedulable pods and provision appropriately-sized nodes. When you deploy workloads with right-sized resource requests (thanks to VPA), Karpenter will select smaller, more cost-effective instances.

Monitoring Cost Impact

To validate your optimizations, you need visibility into resource costs. Kubecost provides detailed insights into how much each workload costs and how much capacity you're wasting.

Install Kubecost in your cluster:

# Add the Kubecost Helm repository
helm repo add kubecost https://kubecost.github.io/cost-analyzer/

# Install Kubecost with Prometheus integration
helm install kubecost kubecost/cost-analyzer \
  --namespace kubecost \
  --create-namespace \
  --set kubecostToken="your-token-here" \
  --set prometheus.server.global.external_labels.cluster_id="${CLUSTER_NAME}"

Access the Kubecost UI by port-forwarding:

kubectl port-forward -n kubecost deployment/kubecost-cost-analyzer 9090:9090

In the Kubecost dashboard, focus on these key metrics:

Efficiency scores: Shows the percentage of requested resources actually being used
Idle costs: Money spent on provisioned but unused resources
Right-sizing recommendations: Suggestions for adjusting requests and limits
Namespace costs: Helps identify which teams or applications drive costs

Track these metrics before and after implementing VPA and Karpenter to quantify your savings.

Real-World Optimization Example

Let's walk through optimizing a typical microservice deployment. You start with a Node.js API that was conservatively configured:

# Before optimization
resources:
  requests:
    cpu: '1000m'
    memory: '2Gi'
  limits:
    cpu: '2000m'
    memory: '4Gi'

After running this workload for two weeks, your monitoring shows:

Average CPU usage: 150m (15% of requests)
Average memory usage: 400Mi (20% of requests)
Peak CPU usage: 300m
Peak memory usage: 800Mi

Based on this data, VPA recommends:

# VPA recommendations (with safety buffer)
resources:
  requests:
    cpu: '200m' # Covers 99th percentile usage
    memory: '512Mi' # Accounts for memory spikes
  limits:
    cpu: '400m' # 2x requests for burst capacity
    memory: '1Gi' # Prevents OOM while allowing growth

The cost impact for 20 replicas of this service:

Before: 20 CPU cores, 40Gi memory requested
After: 4 CPU cores, 10Gi memory requested
Savings: 80% reduction in resource allocation

With Karpenter managing nodes, this workload now runs on smaller instances, further reducing costs by eliminating the need for oversized nodes.

Setting Resource Quotas and Guardrails

As you roll out right-sizing across your organization, implement quotas to prevent teams from reverting to oversized requests:

# namespace-quota.yaml
apiVersion: v1
kind: ResourceQuota
metadata:
  name: backend-team-quota
  namespace: backend
spec:
  hard:
    requests.cpu: '50' # Total CPU requests across all pods
    requests.memory: '100Gi' # Total memory requests
    limits.cpu: '100' # Total CPU limits
    limits.memory: '200Gi' # Total memory limits
    pods: '100' # Maximum number of pods

You can also create LimitRanges to enforce reasonable defaults:

# limit-range.yaml
apiVersion: v1
kind: LimitRange
metadata:
  name: pod-limits
  namespace: backend
spec:
  limits:
    - type: Container
      default: # Default limits if not specified
        cpu: '500m'
        memory: '1Gi'
      defaultRequest: # Default requests if not specified
        cpu: '100m'
        memory: '256Mi'
      max: # Maximum allowed values
        cpu: '4'
        memory: '8Gi'
      min: # Minimum required values
        cpu: '50m'
        memory: '64Mi'

These guardrails help maintain optimization gains while giving teams flexibility within reasonable bounds.

Troubleshooting Common Issues

When implementing VPA and Karpenter, you might encounter some challenges. Here are solutions to the most common problems:

VPA recommendations seem too aggressive: VPA sometimes suggests very low values during low-traffic periods. Check that your monitoring data covers representative traffic patterns. You can also adjust the VPA algorithm:

spec:
  resourcePolicy:
    containerPolicies:
      - containerName: web-app
        controlledValues: RequestsOnly # Only adjust requests, leave limits alone
        mode: Auto

Karpenter nodes aren't scaling down: This usually happens when pods can't be evicted. Check for:

# Look for pods without PodDisruptionBudgets
kubectl get pods --all-namespaces -o wide | grep -v Terminating

# Check for pods using local storage or host networking
kubectl get pods --all-namespaces -o yaml | grep -A 5 hostNetwork

# Verify PodDisruptionBudgets allow eviction
kubectl get pdb --all-namespaces

Pods getting OOMKilled after VPA optimization: This indicates VPA recommendations were too low. Temporarily increase memory requests and check for memory leaks in your application:

# Check recent OOM events
kubectl get events --sort-by=.metadata.creationTimestamp | grep OOMKilled

# Monitor memory usage patterns
kubectl top pods --sort-by=memory --all-namespaces

You can make VPA more conservative by setting higher safety margins:

spec:
  resourcePolicy:
    containerPolicies:
      - containerName: web-app
        maxAllowed:
          memory: '2Gi' # Set a reasonable upper bound

Next Steps

Now that you have VPA and Karpenter working together, consider these additional optimizations:

Horizontal Pod Autoscaling: Combine with VPA to handle both vertical and horizontal scaling
Cluster Autoscaler tuning: If using multiple node provisioners, configure them to work together
Cost alerts: Set up notifications when resource costs exceed thresholds
Regular reviews: Schedule monthly reviews of VPA recommendations and cost reports

You can also explore more advanced Karpenter features like multiple NodePools for different workload types (CPU-intensive, memory-intensive, GPU workloads) and spot instance strategies for non-critical workloads.

The key is to treat right-sizing as an ongoing process. As your applications evolve and traffic patterns change, continue monitoring and adjusting to maintain optimal resource utilization.

How to Free Up a TCP/IP Port on Linux, macOS, and Windows

DevOps Daily Team — Mon, 21 Jul 2025 10:00:00 GMT

TLDR: Find the process using the port with lsof -i :PORT (Linux/macOS) or netstat -ano | findstr :PORT (Windows), then kill it with kill PID or taskkill /PID pid. For persistent services, stop them properly with systemctl stop service or the Services manager. Always identify what's using the port before killing it to avoid disrupting important services.

When you try to start a server or application and see "address already in use" or "port already in use," something else is bound to that port. Here's how to find what's using it and free it up.

Linux: Find and Free Ports

Find What's Using the Port

The most straightforward tool is lsof (list open files):

# Find what's using port 8080
sudo lsof -i :8080

# Output:
# COMMAND   PID   USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
# node    12345  john   21u  IPv4  98765      0t0  TCP *:8080 (LISTEN)

This tells you:

COMMAND: The program name (node)
PID: Process ID (12345)
USER: Who owns the process (john)
The port is in LISTEN state

If you prefer ss (socket statistics):

# Find listening process on port 8080
sudo ss -tulpn | grep :8080

# Output:
# tcp   LISTEN  0  128  *:8080  *:*  users:(("node",pid=12345,fd=21))

Or use netstat (older but still available):

# Find listening process on port 8080
sudo netstat -tulpn | grep :8080

# Output:
# tcp  0  0  0.0.0.0:8080  0.0.0.0:*  LISTEN  12345/node

Kill the Process

Once you have the PID, you can kill the process:

# Gracefully terminate (allows cleanup)
kill 12345

# Force kill if it doesn't respond
kill -9 12345

# Or in one command if you're certain:
sudo lsof -ti :8080 | xargs kill -9

The -t flag tells lsof to output only PIDs, making it easy to pipe to kill.

Stop a System Service

If the port is used by a system service, use systemctl instead of killing:

# Check if it's a systemd service
systemctl status nginx

# Stop the service gracefully
sudo systemctl stop nginx

# Prevent it from starting on boot
sudo systemctl disable nginx

# Or stop and disable in one command
sudo systemctl disable --now nginx

Stopping via systemctl is cleaner than killing - it allows proper shutdown and cleanup.

Check for Services Running in Docker

If you're using Docker, the port might be occupied by a container:

# List containers using port mapping
docker ps --format "table {{.ID}}\t{{.Names}}\t{{.Ports}}"

# Output:
# CONTAINER ID   NAMES      PORTS
# a1b2c3d4e5f6   webapp     0.0.0.0:8080->80/tcp

# Stop the container
docker stop a1b2c3d4e5f6

# Or stop by name
docker stop webapp

macOS: Find and Free Ports

macOS uses the same tools as Linux:

Find What's Using the Port

# Find process using port 3000
lsof -i :3000

# Output:
# COMMAND   PID   USER   FD   TYPE             DEVICE SIZE/OFF NODE NAME
# node    56789  sarah  21u  IPv4 0xabcdef123      0t0  TCP *:3000 (LISTEN)

You don't need sudo on macOS for lsof if you own the process.

Kill the Process

# Kill by PID
kill 56789

# Or force kill
kill -9 56789

# One-liner to kill whatever's on port 3000
lsof -ti :3000 | xargs kill -9

Stop macOS Services

For system services:

# List running services
launchctl list | grep -i apache

# Stop Apache (example)
sudo apachectl stop

# Or for other services
sudo launchctl unload /Library/LaunchDaemons/com.example.service.plist

Windows: Find and Free Ports

Windows uses different tools but the concept is the same.

Find What's Using the Port

REM Find process using port 8080
netstat -ano | findstr :8080

REM Output:
REM TCP    0.0.0.0:8080    0.0.0.0:0    LISTENING    5432
REM                                                   ^^^^
REM                                                   PID

The last column is the process ID (PID). To find out what program that is:

REM Get process details
tasklist /FI "PID eq 5432"

REM Output:
REM Image Name           PID Session Name     Session#    Mem Usage
REM node.exe            5432 Console                 1     45,678 K

Or use PowerShell for a cleaner view:

# Find what's using port 8080
Get-NetTCPConnection -LocalPort 8080 |
    Select-Object LocalAddress, LocalPort, State, OwningProcess,
        @{Name="ProcessName";Expression={(Get-Process -Id $_.OwningProcess).Name}}

# Output:
# LocalAddress LocalPort State   OwningProcess ProcessName
# 0.0.0.0           8080 Listen          5432 node

Kill the Process

REM Kill process by PID
taskkill /PID 5432

REM Force kill if it doesn't respond
taskkill /PID 5432 /F

REM Kill by process name (kills all instances)
taskkill /IM node.exe /F

In PowerShell:

# Kill process by PID
Stop-Process -Id 5432

# Force kill
Stop-Process -Id 5432 -Force

# Kill all node processes
Stop-Process -Name node -Force

Stop Windows Services

For system services, use the Services manager or command line:

REM Stop a service
net stop "Apache2.4"

REM Or use sc (service control)
sc stop Apache2.4

In PowerShell:

# Stop a service
Stop-Service -Name "Apache2.4"

# Stop and disable
Stop-Service -Name "Apache2.4"
Set-Service -Name "Apache2.4" -StartupType Disabled

Or use the GUI:

Press Win+R, type services.msc, press Enter
Find the service in the list
Right-click → Stop

Common Port Conflicts

Port 80/443 (HTTP/HTTPS)

Usually occupied by a web server:

# Linux/macOS
sudo lsof -i :80
sudo lsof -i :443

# Likely culprits: apache2, nginx, httpd

# Stop Apache
sudo systemctl stop apache2   # Ubuntu/Debian
sudo systemctl stop httpd      # RHEL/CentOS

# Stop NGINX
sudo systemctl stop nginx

On Windows, check IIS:

REM Stop IIS
iisreset /stop

REM Or stop the service
net stop W3SVC

Port 3000 (Development Servers)

Often used by Node.js, React, Rails:

# Find and kill
lsof -ti :3000 | xargs kill -9

# Or on Windows
netstat -ano | findstr :3000
taskkill /PID  /F

Port 5432 (PostgreSQL)

# Linux
sudo systemctl stop postgresql

# macOS (Homebrew)
brew services stop postgresql

# Windows
net stop postgresql-x64-13

Port 3306 (MySQL)

# Linux
sudo systemctl stop mysql

# macOS
brew services stop mysql

# Windows
net stop MySQL80

Port 6379 (Redis)

# Linux
sudo systemctl stop redis

# macOS
brew services stop redis

# Windows
redis-cli shutdown

Preventing Port Conflicts

Configure Services to Use Different Ports

Instead of killing processes, change port configurations:

# NGINX - edit /etc/nginx/sites-available/default
server {
    listen 8080;  # Changed from 80
    ...
}

# Restart to apply
sudo systemctl restart nginx

For Node.js apps:

// Use environment variable for port
const PORT = process.env.PORT || 3000;
app.listen(PORT);

Then start with a different port:

PORT=3001 node app.js

Use Docker Port Mapping

Map container ports to different host ports:

# Map container port 80 to host port 8080
docker run -p 8080:80 nginx

# Multiple containers on different host ports
docker run -p 8080:80 nginx
docker run -p 8081:80 nginx

Check Ports Before Starting

#!/bin/bash
# start-server.sh - Check port before starting

PORT=8080

if lsof -Pi :$PORT -sTCP:LISTEN -t >/dev/null ; then
    echo "Port $PORT is already in use"
    echo "Process using port:"
    lsof -i :$PORT
    exit 1
else
    echo "Port $PORT is available"
    ./start-my-server.sh --port $PORT
fi

When Ports Won't Free Up

Sometimes a port stays in TIME_WAIT state after you kill the process:

# Check port state
sudo ss -tan | grep :8080

# Output:
# TIME-WAIT  0      0      192.168.1.10:8080     192.168.1.100:54321

TIME_WAIT prevents immediate reuse of the port. Wait 30-120 seconds, or use SO_REUSEADDR in your application:

import socket

sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
sock.bind(('0.0.0.0', 8080))

This lets your application bind immediately after the previous instance closed.

Safety Checks Before Killing

Always verify what you're killing:

# Check process details
ps aux | grep 12345

# Check what files it has open
lsof -p 12345

# Check command line arguments
cat /proc/12345/cmdline | tr '\0' ' '

Don't kill processes unless you know what they are:

System services might cause issues if killed
Databases could corrupt data if not shut down properly
Other users' processes aren't yours to kill (on shared systems)

Automating Port Cleanup

For development, create a helper script:

#!/bin/bash
# free-port.sh - Free up a port

PORT=$1

if [ -z "$PORT" ]; then
    echo "Usage: $0 "
    exit 1
fi

echo "Finding process using port $PORT..."
PID=$(lsof -ti :$PORT)

if [ -z "$PID" ]; then
    echo "No process is using port $PORT"
    exit 0
fi

echo "Port $PORT is used by PID $PID"
ps -p $PID -o comm=

read -p "Kill this process? (y/n) " -n 1 -r
echo

if [[ $REPLY =~ ^[Yy]$ ]]; then
    kill $PID
    echo "Process killed"
else
    echo "Cancelled"
fi

Usage:

chmod +x free-port.sh
./free-port.sh 8080

The key to freeing up a port is identifying what's using it first, then deciding the appropriate action - stopping a service cleanly, killing a hung process, or reconfiguring to use a different port. Always verify what you're stopping to avoid breaking production services or other users' work.

The Hidden Costs of Over-Automation in DevOps

DevOps Daily Team — Mon, 14 Jul 2025 09:00:00 GMT

A small Friday change to a feature flag UI shipped with a schema migration riding along. The pipeline auto-promoted staging to production without a human in the loop. Payments started timing out because a column used by a background job was dropped. Nobody noticed for an hour because alerts were green at the service level but red downstream. It was not a tooling failure. It was an over-automation failure.

TLDR:
Automation is great when it cuts repetition and enforces consistency. It hurts when it hides context, removes deliberate checks, or blocks manual control during incidents. In this guide you will map where over-automation creeps in, use a simple decision framework, add practical guardrails to a Kubernetes CI/CD workflow, and keep a minimum viable manual path for emergencies.

Prerequisites

Basic GitHub Actions or similar CI runner
A Kubernetes cluster with kubectl access and two namespaces: staging and production
A container registry, for example registry.devops-daily.com
Optional but helpful: Prometheus and Alertmanager, or a hosted equivalent

Why over-automation happens

Teams rarely decide to over-automate on purpose. It accumulates:

Tool enthusiasm - scripts, bots, and YAML for every edge case.
Velocity pressure - remove each manual step to hit dates.
Fear of human error - take humans out of the loop entirely.
Ownership gaps - nobody revisits whether an automation still helps.

The real cost is not the one-time setup. It is the ongoing cognitive load and the time you will spend debugging opaque pipelines when you are already under pressure.


commit -> build -> test -> tag -> push -> deploy(prod)

+--> notify(slack) [green regardless of prod health]

If a quiet failure happens early, the rest might still run with stale artifacts or incomplete state.

Warning signs you went too far

Production feels like a black box, and folks avoid manual access even during an incident.
A trivial change requires multiple pipeline runs across repos.
Rollbacks are slower than forward deploys because nobody remembers the manual commands.
Onboarding depends on scripts rather than understanding how things work.

A simple decision framework

Before you add automation, answer five questions:

Frequency - How often is this done? Frequent tasks are strong candidates.
Risk - What happens if this is wrong? High blast radius often needs a human check.
Transparency - Will this hide details that matter during incidents?
Recovery path - Can we recover fast if the automation fails or misfires?
Ownership - Who will maintain it, and where is the runbook?

A practical rule: automate repetitive, low-risk, well-understood tasks. Keep rare, high-impact, or ambiguous tasks semi-manual with a clear runbook and a fast path.

Add guardrails to a Kubernetes CI/CD workflow

We will start with a common pattern and evolve it.

1) Straight-to-prod pipeline, and why it bites

This job builds, pushes, and deploys every main commit to production. It is fast, but unforgiving.

Before the code, a quick note: this pattern removes human review and makes bad commits instantly live. It is useful for internal tools with very low risk. It is risky for user-facing services.

# .github/workflows/deploy.yml
name: deploy

on:
  push:
    branches: [main]

jobs:
  build_deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      # Build and push the image tagged by commit SHA
      - name: Build and push
        run: |
          docker build -t registry.devops-daily.com/payments-api:${GITHUB_SHA} .
          docker push registry.devops-daily.com/payments-api:${GITHUB_SHA}

      # Deploy to production, no gates
      - name: Deploy prod
        run: |
          kubectl set image deployment/payments-api \
            payments-api=registry.devops-daily.com/payments-api:${GITHUB_SHA}
          kubectl rollout status deployment/payments-api --timeout=90s

2) Safer pattern with staging, promotion, and a kill switch

We keep speed to staging, add a manual promotion to production, and wire a simple kill switch. This gives you time to validate and a fast path to stop rollout.

# .github/workflows/release.yml
name: release

on:
  workflow_dispatch:
    inputs:
      promote:
        description: 'Promote the current staging image to production'
        required: false
        default: 'false'

jobs:
  build_and_stage:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Build and push image
        run: |
          set -euo pipefail
          IMAGE=registry.devops-daily.com/payments-api:${GITHUB_SHA}
          docker build -t "${IMAGE}" .
          docker push "${IMAGE}"
          echo "IMAGE=${IMAGE}" >> $GITHUB_ENV

      - name: Deploy to staging
        run: |
          set -euo pipefail
          kubectl -n staging set image deployment/payments-api \
            payments-api=${IMAGE}
          kubectl -n staging rollout status deployment/payments-api --timeout=120s

  promote_to_prod:
    needs: build_and_stage
    if: ${{ github.event.inputs.promote == 'true' }}
    runs-on: ubuntu-latest
    steps:
      - name: Deploy to production
        run: |
          set -euo pipefail
          kubectl -n production annotate deployment/payments-api \
            devops-daily.com/kill-switch="false" --overwrite
          kubectl -n production set image deployment/payments-api \
            payments-api=${{ env.IMAGE }}
          kubectl -n production rollout status deployment/payments-api --timeout=180s

Why this matters:

Staging deploy is automatic and quick.
Production needs an explicit click. That small pause reduces blast radius.
The kill-switch annotation gives you a single place to gate traffic or stop a rollout in custom controllers or sidecars.

3) Canary rollout with instant rollback

Short canaries catch common errors. We will do a canary with 10 percent traffic, validate, then proceed.

Before the code, note that canaries are only as good as your checks. Pair them with synthetic checks and error rate alerts.

# Kustomize snippet for production canary
# overlays/production/kustomization.yaml
resources:
  - ../../base
patches:
  - target:
      kind: Deployment
      name: payments-api
    patch: |
      - op: replace
        path: /spec/replicas
        value: 10
      - op: add
        path: /spec/template/metadata/labels/canary
        value: "true"

Roll forward and roll back with clear commands during a page:

# Promote canary to full rollout
kubectl -n production scale deployment/payments-api --replicas=30

# Instant rollback to previous ReplicaSet
kubectl -n production rollout undo deployment/payments-api

# When in doubt, pause the rollout and route traffic away if your ingress supports it
kubectl -n production rollout pause deployment/payments-api

4) Wire alerts that align with deploys

You want alerts that fire during the window when bad deploys show up, not hours later.

# PrometheusRule example: error budget friendly alert
- alert: PaymentsHighErrorRate
  expr: sum(rate(http_requests_total{app="payments-api", status=~"5.."}[5m]))
    / sum(rate(http_requests_total{app="payments-api"}[5m])) > 0.05
  for: 4m
  labels:
    severity: page
  annotations:
    summary: 'payments-api 5xx above 5 percent'
    description: 'Check the last deploy and canary logs. Consider rollback.'

Pipe deploy outcomes to chat for shared awareness:

# Simple Slack webhook call after deploy step
curl -X POST -H 'Content-type: application/json' \
  --data "{\"text\":\"payments-api deploy to production: ${GITHUB_SHA} ✅\"}" \
  "$SLACK_WEBHOOK_URL"

Keep a minimum viable manual path

When automation fails, you do not want to rediscover commands. Keep a tiny runbook and a Makefile target that never bitrots.

Explain why: these commands are your seatbelt. They also help onboarding, incident drills, and audits.

# Makefile at repo root
IMAGE=registry.devops-daily.com/payments-api

deploy-staging:
	kubectl -n staging set image deployment/payments-api payments-api=$(IMAGE):$(TAG)
	kubectl -n staging rollout status deployment/payments-api --timeout=120s

deploy-prod:
	kubectl -n production set image deployment/payments-api payments-api=$(IMAGE):$(TAG)
	kubectl -n production rollout status deployment/payments-api --timeout=180s

rollback-prod:
	kubectl -n production rollout undo deployment/payments-api

Usage:

TAG=$(git rev-parse --short HEAD) make deploy-staging
TAG=$(git rev-parse --short HEAD) make deploy-prod
make rollback-prod

Terraform note: avoid automating away intent

It is tempting to auto-apply Terraform plans on every merge. That hides intent and can mask drift or surprise deletes.

Prefer this flow:

plan (PR comment) -> human review -> apply from CI runner with a single approved commit

Why: you keep visibility and still run applies from a clean environment. If you must auto-apply, scope it to low-risk workspaces and require a terraform plan artifact to be attached to the job logs for audit.

Helpful snippet to store plans and surface them for review:

terraform init
terraform plan -out=tfplan.bin
terraform show -json tfplan.bin > tfplan.json
# Upload tfplan.json as a build artifact for PR review

A short checklist for any new automation PR

What problem does this automation solve today, and how will we know if it stops helping later?
What is the manual fallback, documented as a runbook with tested commands?
What signals do we log or emit so failures are obvious?
How do we disable or bypass this quickly during an incident?
Who owns it, and when will we revisit it?

Tie it together

Over-automation hides the very details you need under stress. Add light friction where it protects users, like a promotion gate. Keep a manual path you can run in muscle memory. Log and alert where it matters, near deploys and canaries. Automation should amplify your team, not replace your judgment.

The 10 Most Common DevOps Mistakes (And How to Avoid Them in 2025)

DevOps Daily Team — Sat, 12 Jul 2025 10:00:00 GMT

DevOps isn't just about shipping code faster, it's about doing it smarter, safer, and saner. But let's be real: even the best teams make mistakes. Some are harmless. Others take down production on a Friday afternoon (yes, that Friday deploy).

Here are 10 common DevOps mistakes in 2025, how to avoid them, and a few moments that might hit a little too close to home.

1. Treating Infrastructure as Code Like a One-Off Script

You wrote Terraform once, it worked, and now it lives untouched in a dusty repo folder. That's not IaC, that's tech debt.

Avoid it:

Version control your IaC.
Apply formatting and linting.
Test it with tools like terraform plan or terratest.

2. Not Enforcing Version Control on CI/CD Configs

Your pipeline files are changing, but without versioning, there's no easy way to debug regressions.

Avoid it:

Store all CI/CD config files (like GitHub Actions, GitLab CI, etc.) in version control.
Treat pipeline logic like any other critical code.

3. Poor Secrets Management

Hardcoding secrets in code or using .env files without encryption is a fast way to land on HN for the wrong reasons.

Avoid it:

Use Vault, Doppler, AWS Secrets Manager, or SOPS.
Rotate secrets regularly.

4. No Rollback Strategy

You deploy. Something breaks. And there's no plan B.

Avoid it:

Use blue-green or canary deployments.
Automate rollbacks on failure.
Always have a rollback.sh or previous image ready.

5. Ignoring Observability Until It's Too Late

Monitoring isn't just about uptime. You can't fix what you can't see.

Avoid it:

Add metrics, logs, and traces from day one.
Use tools like Prometheus, Grafana, and OpenTelemetry.

6. Too Many Tools, Not Enough Integration

Your stack has 25 tools. None of them talk to each other. And your alert fatigue is real.

Avoid it:

Consolidate tools where possible.
Favor tools that integrate well with your existing stack.

7. Manual Approval for Every Tiny Change

A typo fix shouldn't need a 3-person review and a Slack war.

Avoid it:

Set up clear policies: auto-approve safe changes, gate critical ones.
Use GitHub environments, OPA, or custom bots to help.

8. No Documentation = Single Point of Failure

"Ask Alex, they built it." Alex is on vacation.

Avoid it:

Write docs as you go.
Use tools like Backstage, Docusaurus, or just plain Markdown.
Encourage a culture of async knowledge sharing.

9. Skipping Tests for Infrastructure Changes

You test app code, but deploy infra changes directly to prod? Bold.

Avoid it:

Use staging or preview environments.
Test IaC with checkov, terratest, or kitchen.

10. Forgetting Security in Your Pipelines

If your pipeline can deploy to prod, attackers might be able to as well.

Avoid it:

Use least privilege for pipeline credentials.
Run security checks like trivy, semgrep, and snyk.

Final Thoughts

DevOps is a journey. These mistakes are all lessons learned the hard way by teams around the world, and probably you, if you've been around long enough.

Want to avoid these mistakes before they cost you time, sleep, or your weekend? We're building checklists, guides, and battle-tested content at DevOps Daily. Come hang out.

PS: Got a DevOps horror story or lesson to share? Drop it in the comments or tag us on Twitter.

How to Get a List of All Valid IP Addresses in a Local Network?

DevOps Daily Team — Thu, 10 Jul 2025 09:30:00 GMT

Finding all active IP addresses on your local network is useful for network administration, troubleshooting connectivity issues, identifying unauthorized devices, or simply seeing what's connected to your network. Whether you're managing a home network or an enterprise environment, several tools can help you discover active hosts.

This guide covers different methods to scan your local network and identify which IP addresses are in use.

TLDR

Use nmap -sn 192.168.1.0/24 to scan your local network and find active hosts. For faster ARP-based scanning, use sudo arp-scan --localnet. On Windows, combine arp -a with ping sweeps. These tools discover devices by sending network packets and analyzing responses.

Prerequisites

You need basic networking knowledge (understanding IP addresses, subnets, and CIDR notation). Access to your local network and permission to scan it is required. Some tools need administrator or root privileges.

Understanding Your Network Range

Before scanning, identify your network range. Your local network typically uses private IP address ranges:

192.168.0.0/16    (192.168.0.0 - 192.168.255.255)
172.16.0.0/12     (172.16.0.0 - 172.31.255.255)
10.0.0.0/8        (10.0.0.0 - 10.255.255.255)

Most home networks use 192.168.1.0/24 (192.168.1.1 - 192.168.1.254) or similar.

Find your network range:

# Linux/macOS
ip addr show
# or
ifconfig

# Windows
ipconfig

Look for your IP address and subnet mask:

inet 192.168.1.100/24

The /24 means the first 24 bits are the network portion, giving you 256 possible addresses (192.168.1.0 through 192.168.1.255).

Using nmap for Network Discovery

Nmap is the most versatile network scanning tool available. It's powerful, fast, and provides detailed information about discovered hosts.

Basic Host Discovery

Scan your entire subnet to find active hosts:

# Ping scan (no port scan)
nmap -sn 192.168.1.0/24

The -sn flag (formerly -sP) does a ping scan without port scanning, making it faster and less intrusive.

Example output:

Starting Nmap 7.92
Nmap scan report for router.local (192.168.1.1)
Host is up (0.0023s latency).

Nmap scan report for laptop.local (192.168.1.45)
Host is up (0.012s latency).

Nmap scan report for phone.local (192.168.1.87)
Host is up (0.034s latency).

Nmap scan report for printer.local (192.168.1.120)
Host is up (0.056s latency).

Nmap done: 256 IP addresses (4 hosts up) scanned in 2.45 seconds

Getting Clean IP List

Extract just the IP addresses:

nmap -sn 192.168.1.0/24 | grep "Nmap scan report" | awk '{print $5}'

Output:

192.168.1.1
192.168.1.45
192.168.1.87
192.168.1.120

Including MAC Addresses

Run with sudo to get MAC addresses:

sudo nmap -sn 192.168.1.0/24

Output includes hardware info:

Nmap scan report for 192.168.1.45
Host is up (0.012s latency).
MAC Address: 00:1A:2B:3C:4D:5E (Apple)

Faster Scanning

Speed up scans by adjusting timing:

# Aggressive timing (faster but more detectable)
nmap -sn -T4 192.168.1.0/24

# Insane timing (very fast, may miss hosts)
nmap -sn -T5 192.168.1.0/24

Using arp-scan for Fast Local Discovery

Arp-scan is faster than nmap for local network discovery because it uses ARP requests, which work at layer 2 and don't require IP routing.

Install arp-scan

# Debian/Ubuntu
sudo apt-get install arp-scan

# Red Hat/CentOS
sudo yum install arp-scan

# macOS
brew install arp-scan

Scan Your Local Network

# Scan all local networks
sudo arp-scan --localnet

# Or specify interface
sudo arp-scan --interface=eth0 --localnet

Output:

Interface: eth0, datalink type: EN10MB (Ethernet)
Starting arp-scan 1.9 with 256 hosts
192.168.1.1     00:11:22:33:44:55       TP-LINK TECHNOLOGIES CO.,LTD.
192.168.1.45    00:1A:2B:3C:4D:5E       Apple, Inc.
192.168.1.87    AA:BB:CC:DD:EE:FF       Samsung Electronics Co.,Ltd
192.168.1.120   11:22:33:44:55:66       Hewlett Packard

4 packets received by filter, 0 packets dropped by kernel
Ending arp-scan 1.9: 256 hosts scanned in 1.92 seconds

Arp-scan is typically faster than nmap for local networks because ARP operates at the data link layer.

Scan Specific Range

# Scan a specific range
sudo arp-scan 192.168.1.1-192.168.1.100

# Or use CIDR notation
sudo arp-scan 192.168.1.0/24

Using Native OS Commands

Linux: arp and ping

Combine ping and arp to discover hosts:

# Ping sweep the network
for ip in 192.168.1.{1..254}; do
    ping -c 1 -W 1 $ip > /dev/null 2>&1 && echo "$ip is up"
done

This pings each address once with a 1-second timeout, then reports which responded.

After pinging, check the ARP cache:

arp -a

Output:

? (192.168.1.1) at 00:11:22:33:44:55 [ether] on eth0
? (192.168.1.45) at 00:1a:2b:3c:4d:5e [ether] on eth0
? (192.168.1.87) at aa:bb:cc:dd:ee:ff [ether] on eth0

macOS: arp-scan or nmap

macOS includes arp but doesn't have arp-scan by default:

# Install via Homebrew
brew install arp-scan nmap

# Then use arp-scan
sudo arp-scan --localnet

Or use the native arp -a after a ping sweep:

for ip in 192.168.1.{1..254}; do
    ping -c 1 -t 1 $ip > /dev/null 2>&1
done

arp -a

Windows: PowerShell Network Scan

Use PowerShell to scan your network:

# Get your network range
$network = "192.168.1"

# Ping sweep
1..254 | ForEach-Object {
    $ip = "$network.$_"
    $ping = Test-Connection -ComputerName $ip -Count 1 -Quiet -TimeoutSeconds 1
    if ($ping) {
        Write-Host "$ip is online"
    }
}

Check ARP cache after scanning:

arp -a

More elegant PowerShell version with parallel execution:

$network = "192.168.1"
$range = 1..254

$range | ForEach-Object -Parallel {
    $ip = "$using:network.$_"
    if (Test-Connection -ComputerName $ip -Count 1 -Quiet -TimeoutSeconds 1) {
        $ip
    }
} -ThrottleLimit 50

Using fping for Efficient Ping Sweeps

Fping sends pings in parallel, making it much faster than sequential ping:

# Install fping
sudo apt-get install fping  # Debian/Ubuntu
brew install fping          # macOS

# Scan network
fping -a -g 192.168.1.0/24 2>/dev/null

The -a flag shows only alive hosts, -g generates a target list from the CIDR range.

Output:

192.168.1.1
192.168.1.45
192.168.1.87
192.168.1.120

Python Script for Network Scanning

Create a simple network scanner in Python:

#!/usr/bin/env python3
import subprocess
import ipaddress
from concurrent.futures import ThreadPoolExecutor

def ping_host(ip):
    """Ping a single host and return IP if alive"""
    try:
        # Use -c 1 for one ping, -W 1 for 1 second timeout
        result = subprocess.run(
            ['ping', '-c', '1', '-W', '1', str(ip)],
            stdout=subprocess.DEVNULL,
            stderr=subprocess.DEVNULL
        )
        if result.returncode == 0:
            return str(ip)
    except Exception:
        pass
    return None

def scan_network(network):
    """Scan a network range for active hosts"""
    net = ipaddress.ip_network(network, strict=False)
    alive_hosts = []

    # Use thread pool for parallel pinging
    with ThreadPoolExecutor(max_workers=50) as executor:
        results = executor.map(ping_host, net.hosts())

    alive_hosts = [ip for ip in results if ip]
    return alive_hosts

if __name__ == '__main__':
    network = '192.168.1.0/24'
    print(f"Scanning {network}...")

    hosts = scan_network(network)

    print(f"\nFound {len(hosts)} active hosts:")
    for host in hosts:
        print(f"  {host}")

Save as network_scan.py and run:

python3 network_scan.py

Identifying Devices by MAC Address

Once you have IP and MAC addresses, identify manufacturers:

# nmap includes MAC vendor lookup
sudo nmap -sn 192.168.1.0/24 | grep "MAC Address"

Or query online MAC address databases:

# Using macvendors.com API
MAC="00:1A:2B:3C:4D:5E"
curl -s "https://api.macvendors.com/$MAC"

Output: Apple, Inc.

Security and Ethical Considerations

Only scan networks you own or have permission to scan. Unauthorized network scanning can be illegal and may violate terms of service for some networks.

Notify your security team: In corporate environments, inform your security team before scanning. Automated security systems may flag scanning activity as potential attacks.

Use appropriate timing: Aggressive scans can impact network performance. Use gentler timing options during business hours.

Respect privacy: Just because you can see a device doesn't mean you should try to access it without authorization.

Saving and Comparing Results

Save scan results to track network changes:

# Save to file with timestamp
nmap -sn 192.168.1.0/24 | grep "Nmap scan report" > scan_$(date +%Y%m%d).txt

# Compare with previous scan
diff scan_20250709.txt scan_20250710.txt

This helps identify new or missing devices on your network.

Troubleshooting Common Issues

No hosts found

Check if you're scanning the correct network range:

# Verify your IP and subnet
ip addr show

Make sure firewalls aren't blocking your scan:

# Temporarily disable firewall (be careful!)
sudo ufw disable  # Ubuntu
sudo systemctl stop firewalld  # CentOS/RHEL

Permission denied

Many scanning tools require root/admin privileges:

# Use sudo
sudo nmap -sn 192.168.1.0/24
sudo arp-scan --localnet

Slow scans

Reduce the scope or use faster tools:

# Scan smaller range
nmap -sn 192.168.1.1-50

# Use arp-scan instead (faster for local networks)
sudo arp-scan --localnet

Discovering active IP addresses on your local network is straightforward with the right tools. Nmap provides detailed, flexible scanning capabilities, while arp-scan offers speed for local networks. Choose the tool that fits your needs, always scan responsibly, and remember that network visibility is the first step toward effective network management and security.

A Day in the Life of a DevOps Engineer

DevOps Daily Team — Wed, 09 Jul 2025 09:00:00 GMT

TLDR

This post follows a DevOps engineer through a typical workday. You'll see how they handle morning deployments, infrastructure scaling, security alerts, and emergency hotfixes. The story covers real scenarios with tools like Kubernetes, Docker, Jenkins, and monitoring systems while showing how DevOps work directly impacts business operations. If you're curious about what DevOps engineers actually do day-to-day, this realistic walkthrough will give you insights into the challenges, responsibilities, and satisfying moments of the role.

The Day at a Glance

05:47 AM ⚠️  PagerDuty Alert - API Response Time Critical
07:30 AM 🔧  Emergency Hotfix Deployment
11:30 AM 🔒  Security Incident Response
02:00 PM 📊  Performance Review & Feature Flag Deployment
06:00 PM 🔄  Kubernetes Cluster Maintenance
10:30 PM 🚨  Database Performance Emergency
12:00 AM 💤  Crisis Resolved, Systems Stable

The phone buzzes at 5:47 AM. Not the alarm - that's set for 6:00 AM. It's PagerDuty. The production API response time has crossed the 2-second threshold, and customers are starting to complain on social media.

Sound familiar? Welcome to Monday morning in the life of a DevOps engineer.

Rolling out of bed, laptop in hand, connecting to the VPN before the coffee even starts brewing. The monitoring dashboard shows a clear pattern: response times started climbing around 5:30 AM, right when the European market opened. The weekend's supposedly "minor" feature deployment is now causing 40% of API calls to timeout.

Incident Severity Assessment:
┌─────────────────────────────────────────────────────────────┐
│ 🔴 CRITICAL: 40% API timeout rate                           │
│ 📱 Social media complaints increasing                       │
│ 🌍 European market affected (peak hours)                    │
│ ⏰ US market opens in 3 hours                               │
│ 💰 Revenue impact: ~$2,000/minute                           │
└─────────────────────────────────────────────────────────────┘

This is why DevOps engineers sleep with their phones next to the bed.

Morning Fire Fighting

🔥 Crisis Mode Activated

The first instinct is to check the application logs. The ELK stack reveals the story immediately. The new payment processing feature is making synchronous calls to a third-party service, and those calls are taking 8-12 seconds to complete. When European users woke up and started making purchases, the connection pool got exhausted.

Payment Flow Issue:
User Request → API Gateway → Payment Service → Third-Party Provider
     ↓              ↓              ↓              ↓
   Fast         Fast         SLOW (8-12s)     TIMEOUT

Connection Pool: [████████████████████] 200/200 (FULL!)

A quick check shows 200 active connections - they've hit the maximum pool size. This needs an immediate fix while working on the root cause. The temporary solution is to scale up the payment service pods from 3 to 6, buying time to implement a proper fix.

Watching the metrics after applying the scaling change, response times start dropping within two minutes. The immediate crisis is over, but this is just a band-aid. The real fix needs to happen in the application code, requiring coordination with the development team.

Key Insight: Sometimes the best solution is the fastest solution. Scaling infrastructure horizontally bought time to implement a proper fix without losing customers.

💡 Pro Tip: Always have a rollback plan ready. In this case, the scaling approach was reversible if it didn't work, keeping options open during the crisis.

Deployment Coordination

📞 Emergency War Room

By 7:30 AM, the first video call of the day begins with the lead developer and product manager. They're discussing the hotfix strategy while pulling up the deployment pipeline in Jenkins.

"The payment timeout issue affects roughly 30% of our European customers," the product manager explains, checking analytics. "We need this fixed before the US market opens, or we're looking at significant revenue loss."

The developer has already pushed a fix to the staging branch - making the third-party payment calls asynchronous with proper error handling. The DevOps engineer's job is to get this through the pipeline safely.

Hotfix Deployment Pipeline:
┌─────────────────────────────────────────────────────────────┐
│ 1. Code Review    ✅ (expedited, focused review)            │
│ 2. Build & Test   ✅ (automated, 5 minutes)                 │
│ 3. Staging Deploy ✅ (integration tests passing)            │
│ 4. Smoke Tests    ✅ (payments working correctly)           │
│ 5. Production     🟡 (waiting for approval)                 │
└─────────────────────────────────────────────────────────────┘

The staging deployment goes smoothly. Integration tests pass, and end-to-end tests confirm that payments are now processing correctly with the new asynchronous flow. The green light for production deployment comes at 8:45 AM.

Key Insight: Production hotfixes require extra caution. Even with time pressure, proper testing in staging prevented a second incident.

🎯 Reality Check: In emergency situations, communication becomes even more critical. Clear status updates kept all stakeholders informed and aligned.

Infrastructure Scaling Challenges

With the payment crisis resolved, attention turns to a brewing infrastructure problem. The marketing team is launching a major campaign next week, expecting a 3x increase in traffic. The current Kubernetes cluster can barely handle normal peak loads.

Opening Terraform to review the current infrastructure setup reveals t3.medium instances that are cost-effective for normal operations but won't handle the expected load surge. A scaling strategy is needed that can handle the traffic spike without breaking the budget.

Current Infrastructure:
┌─────────────────────────────────────────────────────────────┐
│ Kubernetes Cluster                                          │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐             │
│ │ t3.medium   │ │ t3.medium   │ │ t3.medium   │             │
│ │ Node 1      │ │ Node 2      │ │ Node 3      │             │
│ └─────────────┘ └─────────────┘ └─────────────┘             │
│                                                             │
│ Campaign Week (3x traffic) = 💥 OVERLOAD                    │
└─────────────────────────────────────────────────────────────┘

Solution: Pre-provisioned c5.xlarge nodes (scaled to 0 until needed)

The plan involves creating a new node group with c5.xlarge instances, pre-created but kept at zero capacity until the campaign starts. This way, they can scale up quickly when needed and scale down immediately after to control costs.

Key Insight: Planning for predictable traffic spikes is cheaper than dealing with unexpected outages. Pre-provisioning resources that can be quickly activated saves both money and stress.

Security Alert Response

At 11:30 AM, the security monitoring tool flags something suspicious. The intrusion detection system shows unusual network traffic patterns from one of the application servers. Security incidents can escalate quickly, so immediate attention is required.

Initial investigation shows someone is trying to access the MySQL database directly from an external IP. A quick check of security groups and firewall rules shows they look correct - database access should only be allowed from application servers within the VPC. But the logs show connection attempts from a completely different IP range.

Digging deeper into the application logs reveals the issue. A developer accidentally committed database credentials to a public GitHub repository three days ago. The credentials were scraped by automated tools and are now being used for unauthorized access attempts.

Security Incident Timeline:
Day 1: Dev commits credentials → GitHub (public repo)
Day 2: Automated scrapers find credentials
Day 3: Credentials posted on dark web forums
Day 4: Unauthorized access attempts begin ← WE ARE HERE

Threat Actor → Internet → Firewall → Database (attempting access)
                             ↓
                       ⚠️  BLOCKED (but trying)

The immediate response is clear: rotate database credentials immediately and update the Kubernetes secret. The security incident is contained, but this requires a longer-term solution - implementing automated secret scanning in the CI/CD pipeline and scheduling security training for the development team.

Key Insight: Security incidents are rarely just technical problems. They're usually process problems that require both immediate fixes and long-term prevention strategies.

Monitoring and Alerting Improvements

After lunch, focus shifts to improving the monitoring setup. The morning's payment issue could have been caught earlier with better alerting. Opening Prometheus to review the current metrics collection shows it only monitors basic metrics like CPU and memory usage.

Working with the developer to add business-specific metrics that would have caught the payment timeout issue earlier becomes the priority. Custom metrics for payment processing duration, active connections, and success/failure rates are implemented.

With these metrics in place, new alerting rules are created that would have triggered within minutes of the morning's incident, giving time to respond before customers were affected.

Afternoon Deployment Pipeline

🚀 Major Feature Release

The afternoon brings a scheduled deployment of the new user dashboard feature. This is a major feature that's been in development for six weeks, and the product team is eager to get it in front of users.

The staging environment looks good, but something concerning appears in the performance tests. The new dashboard is making 47 database queries per page load. With the expected traffic increase from the marketing campaign, this could cause serious performance problems.

Database Query Analysis:
┌─────────────────────────────────────────────────────────────┐
│ Current Dashboard: 3 queries per page                       │
│ New Dashboard: 47 queries per page                          │
│                                                             │
│ Expected Traffic: 10,000 concurrent users                   │
│ Query Load: 470,000 queries/second                          │
│ Database Capacity: 50,000 queries/second                    │
│                                                             │
│ Result: 💥 DATABASE MELTDOWN                                │
└─────────────────────────────────────────────────────────────┘

An emergency meeting with the development team follows. The conversation is tense - the marketing campaign is already scheduled, and delaying the dashboard feature would mean missing the promotional opportunity.

The Dilemma:

✅ Ship on time → Happy marketing team, potential system failure
❌ Delay feature → Disappointed stakeholders, stable system
🤔 Find middle ground → ???

"We can't deploy this as-is," becomes the message, showing the performance metrics. "Each page load is hitting the database 47 times. With 10,000 concurrent users, that's 470,000 database queries per second. Our database will fall over."

The lead developer looks at the query analysis. "Most of these are N+1 queries. We can fix the worst ones with some eager loading, but it'll take at least two days to properly optimize."

A compromise is proposed: deploy the feature with a feature flag, initially enabled for only 10% of users. This gives real-world performance data while limiting the impact on the infrastructure.

Feature Flag Strategy:
┌─────────────────────────────────────────────────────────────┐
│ Incoming Users: 10,000/second                               │
│                                                             │
│ 90% → Old Dashboard (stable, fast)                          │
│ 10% → New Dashboard (testing, monitored)                    │
│                                                             │
│ Database Load: Manageable vs. Catastrophic                  │
└─────────────────────────────────────────────────────────────┘

The deployment goes ahead with the feature flag in place. Database performance is monitored closely as the feature rolls out to the limited user group. The impact is manageable at 10% traffic, but the metrics confirm concerns about a full rollout.

Key Insight: Feature flags aren't just for A/B testing. They're a powerful risk management tool that lets you test production performance without betting the entire infrastructure.

🔄 DevOps Wisdom: The best compromise is often a gradual rollout. It satisfies business needs while protecting system stability.

Evening Infrastructure Maintenance

As the day winds down, planned maintenance tasks need attention. The Kubernetes cluster needs a version upgrade, and several security patches need to be applied to the worker nodes.

The upgrade process requires careful coordination to avoid downtime. Nodes are drained one by one, system updates are applied, kubelet is restarted with the new version, and then the node is uncordoned back into service.

The upgrade process takes about 90 minutes, but it goes smoothly. Application metrics are monitored throughout the process - response times stay normal, and no alerts fire.

Late Night Emergency

🌙 10:30 PM - Not Again...

Just when getting ready for bed at 10:30 PM, the phone buzzes again. This time it's a critical alert: the main application database is reporting high CPU usage and slow query performance. The European overnight batch processing jobs are running much longer than usual.

Every DevOps engineer knows this feeling - the dreaded "just one more alert" before bed.

Connecting to the database server immediately reveals the problem. One of the batch jobs is running a query that's been executing for 3 hours. The query is scanning a table with 50 million rows without using an index.

Database Performance Crisis:
┌─────────────────────────────────────────────────────────────┐
│ Query: SELECT * FROM user_activities WHERE...               │
│ Status: Running for 3 hours ⏱️                              │
│ Rows Scanned: 50,000,000 (NO INDEX!)                        │
│ CPU Usage: ████████████████████████████████████ 95%         │
│ Other Queries: ⏳ WAITING... WAITING... WAITING...          │
└─────────────────────────────────────────────────────────────┘

The batch job developer probably tested with a small dataset and didn't realize the performance implications. A tough decision emerges: kill the long-running query to restore database performance, meaning the batch job will need to restart from the beginning, or let it finish but risk affecting the morning's application performance.

The Midnight Decision Matrix:

Option 1: Kill Query + Create Index
├─ Pros: Immediate relief, proper fix
├─ Cons: Batch job restarts (3 hours lost)
└─ Risk: Low

Option 2: Let Query Finish
├─ Pros: Batch job completes
├─ Cons: Database stays slow
└─ Risk: High (morning traffic impact)

The choice is made to kill the query and create the missing database index. The index creation takes 45 minutes on the large table, but once it's complete, the batch job can restart and finish in just 20 minutes instead of hours.

Key Insight: Sometimes you have to make tough decisions with incomplete information. The ability to quickly assess risk and choose the least harmful option is crucial in DevOps.

⚡ Late Night Wisdom: The best decisions aren't always the easiest ones. Protecting tomorrow's users was worth the short-term pain of restarting the batch job.

Reflection and Planning

🌅 Midnight - Systems Stable, Lessons Learned

By midnight, the laptop finally closes. The day started with a production crisis, included a security incident, featured a challenging deployment decision, and ended with a database performance emergency. Each situation required different skills: quick problem-solving, technical analysis, team coordination, and risk assessment.

Daily Impact Summary:
┌─────────────────────────────────────────────────────────────┐
│ 🔧 Issues Resolved: 4 critical, 2 medium priority           │
│ 👥 Customers Affected: Minimal (thanks to quick response)   │
│ 💰 Revenue Protected: ~$50,000 (prevented outages)          │
│ 🛠️ Systems Improved: 3 (monitoring, security, indexing)     │
│ 📈 Infrastructure: Scaled and optimized                     │
└─────────────────────────────────────────────────────────────┘

Tomorrow will bring new challenges. The marketing campaign is getting closer, and the infrastructure scaling plan needs finalization. The dashboard feature needs performance optimization before it can be fully rolled out. The development team needs security training to prevent credential leaks. The monitoring system needs those new business metrics.

But tonight, millions of users were able to make purchases, view their dashboards, and access the application without interruption. The infrastructure held up under pressure, the team collaborated effectively during crises, and the systems are more robust than they were this morning.

Tomorrow's Action Items:

✅ Finalize campaign infrastructure scaling
✅ Optimize dashboard database queries
✅ Implement automated secret scanning
✅ Deploy enhanced monitoring metrics
✅ Schedule security training session

This is the reality of DevOps work - part firefighting, part planning, part collaboration, and part continuous improvement. It's demanding and sometimes stressful, but it's also rewarding to know that your work directly enables the business to serve its customers.

The phone is on silent for the next six hours, but somewhere, monitoring systems are keeping watch, automated processes are handling routine tasks, and the infrastructure is quietly supporting thousands of users around the world. That's the real success of DevOps - building systems that work reliably, even when you're not watching.

The Human Side of DevOps

Being a DevOps engineer means being part detective, part architect, part diplomat, and part firefighter. Every day brings new challenges, but also new opportunities to make systems better, faster, and more reliable. The work never ends, but neither does the satisfaction of building technology that makes a real difference in people's lives.

The morning payment issue wasn't just about fixing code - it was about understanding the business impact of technical decisions. When European customers couldn't complete their purchases, it affected real people trying to buy gifts, pay bills, or run their businesses. The quick response prevented thousands of failed transactions and potential customer churn.

The security incident required more than just technical fixes. It highlighted the need for better developer education and process improvements. The conversation with the development team wasn't about blame - it was about learning and preventing similar issues in the future.

The deployment decision for the dashboard feature showcased the constant balance between business needs and technical constraints. The marketing campaign couldn't be delayed, but releasing a feature that would crash the database wasn't an option. The feature flag solution satisfied both requirements while providing valuable data for future improvements.

The Broader Impact

DevOps work extends far beyond keeping servers running. It's about enabling the entire organization to move faster and more reliably. The monitoring improvements implemented today will prevent future incidents. The infrastructure scaling plan will support business growth. The security training will protect customer data.

Each technical decision has ripple effects throughout the organization. The choice to scale up the payment service immediately instead of waiting for a code fix meant that the customer service team didn't get flooded with complaint calls. The decision to implement feature flags for the dashboard deployment gave the product team valuable usage data while protecting system stability.

The database performance fix at midnight wasn't just about query optimization - it was about ensuring that the morning's business reports would be ready on time, that the analytics team could access their data, and that the automated systems could process customer orders without delay.

Skills Beyond Technology

While technical skills are essential, DevOps engineering requires much more. Communication skills are crucial for coordinating with development teams, explaining technical issues to business stakeholders, and writing clear documentation for on-call procedures.

Problem-solving skills go beyond debugging code. They involve understanding complex systems, identifying root causes of issues, and designing solutions that prevent future problems. The ability to work under pressure while maintaining clear thinking is essential when production systems are down and customers are affected.

Risk assessment becomes second nature - every change, every deployment, every infrastructure modification needs to be evaluated for potential impact. The ability to make quick decisions with incomplete information is valuable when incidents are unfolding and time is critical.

The Satisfaction of Reliability

The most rewarding aspect of DevOps work isn't the dramatic incident responses or the complex technical solutions. It's the quiet satisfaction of building systems that work consistently, day after day, serving users around the world without interruption.

When a deployment goes smoothly, when monitoring catches an issue before it affects users, when an infrastructure upgrade happens without downtime - these moments of seamless operation represent the true success of DevOps practices.

The tools and technologies will continue to evolve, but the core mission remains the same: bridge the gap between development and operations, automate repetitive tasks, monitor everything that matters, and respond quickly when things go wrong. It's challenging work, but for those who enjoy solving complex problems and working with cutting-edge technology, there's nothing quite like it.

The best DevOps engineers are those who can see the bigger picture - understanding how their technical decisions impact users, businesses, and teams. They're the ones who can remain calm during crises, think strategically about infrastructure improvements, and communicate effectively with both technical and non-technical stakeholders.

This is what a day in the life of a DevOps engineer really looks like - not just managing servers and writing scripts, but being a crucial part of the technology ecosystem that powers modern business operations.

Key Takeaways for Aspiring DevOps Engineers

🎯 Essential Skills Demonstrated Today:

Crisis Management: Quick thinking under pressure while maintaining system stability
Risk Assessment: Evaluating trade-offs between speed and reliability
Cross-team Communication: Coordinating with developers, product managers, and business stakeholders
Technical Versatility: From Kubernetes to databases to security incidents
Business Impact Awareness: Understanding how technical decisions affect revenue and customers

🛠️ Core Tools in Action:

Monitoring: ELK Stack, Prometheus, PagerDuty
Infrastructure: Kubernetes, Docker, Terraform, AWS
CI/CD: Jenkins, automated testing pipelines
Security: Intrusion detection, credential management
Databases: MySQL, query optimization, indexing

📚 Want to Learn More?

If this day-in-the-life resonates with you, here are some next steps:

🚀 Getting Started: Practice with containerization (Docker), learn Kubernetes basics, and get comfortable with Linux command line

🔍 Dive Deeper: Set up monitoring in a personal project, practice incident response scenarios, learn infrastructure as code

💼 Career Path: Consider starting as a systems administrator, junior DevOps engineer, or SRE to build foundational skills

The Reality Check: DevOps isn't just about tools and automation. It's about building reliable systems that let businesses focus on serving their customers. Every alert, every deployment, every optimization contributes to that mission.

The most rewarding part? Knowing that somewhere in the world, users are seamlessly making purchases, accessing services, and getting value from applications - all because the infrastructure you built and maintain is working exactly as it should.

Go over the following DevOps Roadmap to see how you can build your skills and career in this exciting field.

That's the real satisfaction of DevOps work - building the invisible foundation that makes everything else possible.

How Can I Delete a Remote Tag?

DevOps Daily Team — Tue, 08 Jul 2025 13:30:00 GMT

Git tags mark specific points in your repository history, typically used for releases like v1.0.0 or v2.1.3. Sometimes you need to delete a tag - maybe you tagged the wrong commit, used an incorrect version number, or need to recreate a release. While deleting a local tag is straightforward, removing a tag from a remote repository requires an additional step that isn't immediately obvious.

TLDR

To delete a remote tag, use git push origin --delete tagname or the alternative syntax git push origin :refs/tags/tagname. To also delete the local tag, run git tag -d tagname. Always coordinate with your team before deleting tags that others might be using.

Prerequisites

You need a Git repository with tags and push access to the remote repository. Basic familiarity with Git commands and understanding of what tags are will help you follow along.

Understanding Local vs Remote Tags

When you create a tag locally, it exists only in your local repository until you explicitly push it:

# Create a local tag
git tag v1.0.0

# List local tags
git tag

To make the tag available to others, push it:

# Push a specific tag
git push origin v1.0.0

# Or push all tags at once
git push --tags

Once pushed, the tag exists in two places - your local repository and the remote repository. Deleting a local tag doesn't automatically delete the remote tag, and vice versa. You need to delete them separately.

Deleting a Remote Tag

The standard way to delete a remote tag is:

git push origin --delete v1.0.0

This tells Git to delete the tag v1.0.0 from the remote named origin. You'll see output like:

To github.com:username/project.git
 - [deleted]         v1.0.0

An alternative syntax that does the same thing:

git push origin :refs/tags/v1.0.0

This older syntax means "push nothing to the remote tag v1.0.0", which effectively deletes it. The modern --delete flag is clearer and preferred, but both work identically.

Deleting Both Local and Remote Tags

To completely remove a tag from both local and remote:

# Delete the local tag
git tag -d v1.0.0

# Delete the remote tag
git push origin --delete v1.0.0

You can run these in either order. If you delete the remote tag first, the local tag remains in your repository but won't be on the remote anymore.

Deleting Multiple Remote Tags

To delete several tags at once:

# Delete multiple tags from remote
git push origin --delete v1.0.0 v1.0.1 v1.0.2

# Delete local tags
git tag -d v1.0.0 v1.0.1 v1.0.2

For many tags, you might script it:

# Delete all tags matching a pattern from remote
for tag in $(git tag -l "v1.0.*"); do
  git push origin --delete $tag
done

# Delete matching local tags
git tag -d $(git tag -l "v1.0.*")

This example deletes all tags starting with v1.0. - useful when cleaning up a series of pre-release tags or correcting a versioning mistake.

Verifying Tag Deletion

After deleting a remote tag, verify it's gone:

# List all remote tags
git ls-remote --tags origin

# Or fetch the latest tag info
git fetch --tags --prune

The --prune flag removes references to tags that no longer exist on the remote. Without it, your local Git might still show remote tags that have been deleted.

Check your local tags:

# List all local tags
git tag

# Search for a specific tag
git tag -l "v1.0.0"

Handling Permission Issues

If you get a permission error when deleting a remote tag:

error: unable to delete 'v1.0.0': remote ref does not exist
error: failed to push some refs to 'github.com:username/project.git'

This could mean:

The tag doesn't exist on the remote: Check with git ls-remote --tags origin
You don't have permission: Some repositories protect tags, especially in production environments
The tag is protected: Platforms like GitHub and GitLab allow protecting tags to prevent accidental deletion

For protected tags, you may need to:

Temporarily unprotect the tag in your Git hosting platform's settings
Ask someone with appropriate permissions to delete it
Use your Git host's web interface to delete the tag

Recreating a Tag After Deletion

A common workflow is deleting and recreating a tag, perhaps because you tagged the wrong commit:

# Delete the incorrect tag locally and remotely
git tag -d v1.0.0
git push origin --delete v1.0.0

# Create the tag on the correct commit
git checkout main
git tag v1.0.0

# Push the corrected tag
git push origin v1.0.0

Or create the tag on a specific commit:

# Delete old tag
git tag -d v1.0.0
git push origin --delete v1.0.0

# Create tag on specific commit
git tag v1.0.0 abc123def

# Push the new tag
git push origin v1.0.0

Impact on Team Members

When you delete a remote tag, it doesn't automatically disappear from your teammates' local repositories. They'll still have the old tag locally until they prune:

# Team members should run this to sync tags
git fetch --tags --prune

This updates their local tag references to match the remote, removing tags that no longer exist remotely.

Important: Coordinate with your team before deleting tags. If someone has deployed code using a specific tag, deleting it can cause confusion. If the tag is referenced in documentation, CI/CD pipelines, or deployment scripts, those references will break.

Tags in Release Management

In production environments, tags typically represent releases. Deleting them should be done carefully:

Pre-release tags (like v1.0.0-beta.1 or v1.0.0-rc.1) can usually be deleted safely if you're still iterating.

Production release tags (like v1.0.0 or v2.3.1) should rarely be deleted. If a release had problems:

# Instead of deleting v1.0.0, create v1.0.1 with fixes
git tag v1.0.1
git push origin v1.0.1

This preserves the history and makes it clear that 1.0.1 supersedes 1.0.0.

Recovering a Deleted Tag

If you accidentally delete a tag before pushing the deletion, you can recover it from your reflog:

# Find the commit that was tagged
git reflog | grep "v1.0.0"

# Recreate the tag on that commit
git tag v1.0.0

However, if the tag has been deleted from both local and remote and you don't have it in your reflog, you'll need to recreate it on what you believe was the correct commit. This is why it's important to be certain before deleting tags, especially release tags.

Deleting Tags on Different Git Platforms

GitHub

You can delete tags via the web interface:

Go to your repository's Releases page
Find the release associated with the tag
Click Edit or Delete
Delete the release (this doesn't delete the tag automatically)
Go to the Tags page and delete the tag

Or use the command line as shown above.

GitLab

GitLab's interface:

Navigate to Repository → Tags
Click the delete icon next to the tag

Or use command line.

Bitbucket

Bitbucket interface:

Go to the repository
Click on Tags in the navigation
Click the delete icon next to the tag

Command line works the same across all platforms.

Scripting Tag Cleanup

For automated tag cleanup, you might create a script:

#!/bin/bash
# Delete old pre-release tags

# Get all tags matching pattern
OLD_TAGS=$(git tag -l "*-beta*" "*-alpha*" "*-rc*")

if [ -z "$OLD_TAGS" ]; then
  echo "No pre-release tags to delete"
  exit 0
fi

echo "Found pre-release tags:"
echo "$OLD_TAGS"
read -p "Delete these tags? (y/n) " -n 1 -r
echo

if [[ $REPLY =~ ^[Yy]$ ]]; then
  for tag in $OLD_TAGS; do
    echo "Deleting $tag..."
    git tag -d "$tag"
    git push origin --delete "$tag"
  done
  echo "Done!"
else
  echo "Cancelled"
fi

This script finds all alpha, beta, and release candidate tags, shows them to you, and asks for confirmation before deleting.

Best Practices for Tag Management

Use semantic versioning: Follow a consistent versioning scheme (like SemVer: major.minor.patch) so tags are predictable.

Protect production tags: Configure your Git hosting platform to protect release tags from accidental deletion.

Document deletions: If you must delete a production tag, document why in your team's communication channels.

Create annotated tags for releases: Use git tag -a v1.0.0 -m "Release 1.0.0" instead of lightweight tags. Annotated tags include metadata about who created the tag and when.

Automate where possible: Use CI/CD tools to create tags automatically based on successful builds or deployments.

Never reuse tag names: If you delete a tag, don't recreate it with the same name pointing to a different commit. This causes confusion. Instead, increment the version number.

Deleting remote tags is straightforward once you know the command, but the decision to delete should be made carefully. Tags serve as important markers in your project's history, and removing them can impact deployments, documentation, and team workflows. When in doubt, it's usually better to create a new tag with an incremented version than to delete and recreate an existing one.

How to Set Image Name in Dockerfile

DevOps Daily Team — Fri, 04 Jul 2025 10:00:00 GMT

TLDR

To set an image name in Docker, you use the docker build command with the -t flag. The image name is not directly set in the Dockerfile but is specified during the build process.

Docker images are a fundamental part of containerized workflows. While the Dockerfile defines the instructions for building an image, the image name is set during the build process using the docker build command. This guide will show you how to set and manage image names effectively.

Step 1: Create a Dockerfile

Start by creating a Dockerfile with the necessary instructions to build your image.

Example Dockerfile

FROM node:18
WORKDIR /app
COPY . .
RUN npm install
CMD ["node", "app.js"]

Explanation

FROM node:18: Specifies the base image.
WORKDIR /app: Sets the working directory inside the container.
COPY . .: Copies files from the current directory into the container.
RUN npm install: Installs dependencies.
CMD ["node", "app.js"]: Starts the application.

Step 2: Build the Image with a Name

Use the docker build command with the -t flag to set the image name.

Command

docker build -t my-app:latest .

Explanation

-t my-app:latest: Sets the image name to my-app and the tag to latest.
.: Specifies the build context (current directory).

Step 3: Verify the Image Name

After building the image, verify its name using the docker images command.

Command

To list all Docker images and confirm the name:

docker images

Example Output

REPOSITORY   TAG       IMAGE ID       CREATED         SIZE
my-app       latest    abc123def456   5 minutes ago   123MB

Best Practices

Use Descriptive Names: Choose names that reflect the purpose of the image.
Tag Versions: Use tags like v1.0 or latest to manage versions.
Automate Naming: Use CI/CD pipelines to automate image naming and tagging.

By following these steps, you can effectively set and manage image names in Dockerfiles, ensuring streamlined workflows and better organization of your Docker images.

COPY with Docker but with Exclusion

DevOps Daily Team — Fri, 04 Jul 2025 09:00:00 GMT

TLDR

To exclude files or directories when using the COPY instruction in Docker, leverage .dockerignore files. This ensures that unwanted files are not copied into your Docker image, optimizing build times and reducing image size.

The COPY instruction in Docker is used to copy files and directories from your local filesystem into a Docker image. However, there are scenarios where you might want to exclude certain files or directories from being copied. This guide will show you how to achieve this using .dockerignore files.

Step 1: Create a .dockerignore File

The .dockerignore file allows you to specify patterns for files and directories to exclude during the Docker build process. Create a .dockerignore file in the same directory as your Dockerfile.

Example .dockerignore File

node_modules
*.log
.env
temp/

Explanation

node_modules: Excludes the node_modules directory.
*.log: Excludes all .log files.
.env: Excludes the .env file.
temp/: Excludes the temp directory.

Step 2: Use the COPY Instruction

In your Dockerfile, use the COPY instruction to copy files into the image. The .dockerignore file will automatically exclude the specified files and directories.

Example Dockerfile

FROM node:18
WORKDIR /app
COPY . .
RUN npm install
CMD ["node", "app.js"]

Explanation

COPY . .: Copies all files from the current directory into the /app directory in the image, excluding files specified in .dockerignore.
RUN npm install: Installs dependencies.
CMD ["node", "app.js"]: Starts the application.

Step 3: Build the Docker Image

Build the Docker image using the docker build command:

docker build -t my-app .

This command builds the image and excludes files specified in .dockerignore.

Advanced Options

Exclude Files Dynamically

You can dynamically exclude files by generating a .dockerignore file during the build process. For example:

echo "temp/" > .dockerignore
docker build -t my-app .

Multi-Stage Builds

Use multi-stage builds to copy only the necessary files into the final image:

FROM node:18 AS builder
WORKDIR /app
COPY . .
RUN npm install

FROM node:18
WORKDIR /app
COPY --from=builder /app/dist /app/dist
CMD ["node", "app/dist/app.js"]

Explanation

The builder stage installs dependencies and builds the application.
The final stage copies only the dist directory from the builder stage.

Best Practices

Minimize Context: Use .dockerignore to reduce the build context size.
Keep .dockerignore Updated: Regularly update .dockerignore to exclude unnecessary files.
Test Builds: Test your Docker builds to ensure excluded files are not copied.

By following these steps, you can effectively use the COPY instruction in Docker with exclusion patterns, optimizing your Docker builds and improving image efficiency.

Forward Host Port to Docker Container

DevOps Daily Team — Tue, 24 Jun 2025 14:00:00 GMT

Docker containers run in isolated network environments by default, which means services running inside containers aren't accessible from your host machine or the outside world without explicit port forwarding. Understanding how to map ports between your host and containers is essential for accessing web applications, databases, APIs, and other containerized services.

This guide covers everything you need to know about Docker port forwarding, from basic mapping to advanced scenarios.

TLDR

Use -p or --publish to forward ports when running a container: docker run -p 8080:80 nginx maps host port 8080 to container port 80. Use -p 127.0.0.1:8080:80 to bind only to localhost. For running containers, you must commit and restart with new port mappings. Use docker-compose for complex port configurations.

Prerequisites

You need Docker installed and basic familiarity with running containers. Understanding of network ports and how services listen on ports will help.

Basic Port Forwarding

The -p flag maps a host port to a container port.

Syntax

docker run -p :

Example - running Nginx:

# Map host port 8080 to container port 80
docker run -p 8080:80 nginx

Now access Nginx at http://localhost:8080 on your host machine. Inside the container, Nginx listens on port 80, but you access it via 8080 on your host.

Multiple Port Mappings

Map multiple ports with multiple -p flags:

# Map both HTTP and HTTPS
docker run -p 8080:80 -p 8443:443 nginx

Or forward a range of ports:

# Map ports 8080-8090 to 80-90
docker run -p 8080-8090:80-90 myapp

Publishing All Exposed Ports

Images can declare ports with EXPOSE in their Dockerfile. Use -P (capital P) to automatically map all exposed ports to random high ports on the host:

docker run -P nginx

Check which ports were assigned:

docker ps

Output:

CONTAINER ID   IMAGE    PORTS                    NAMES
abc123def456   nginx    0.0.0.0:32768->80/tcp    eager_morse

Nginx's port 80 is mapped to host port 32768.

Find the actual port programmatically:

docker port  80

Output:

0.0.0.0:32768

Binding to Specific Interfaces

By default, -p 8080:80 binds to all network interfaces (0.0.0.0), making the service accessible from anywhere. Bind to specific interfaces for security:

Localhost Only

# Only accessible from the host machine
docker run -p 127.0.0.1:8080:80 nginx

Now http://localhost:8080 works, but external machines can't connect.

Specific IP Address

# Bind to specific network interface
docker run -p 192.168.1.100:8080:80 nginx

The service is only accessible via the specified IP address.

UDP Port Forwarding

Specify the protocol for UDP services:

# Forward UDP port
docker run -p 5353:53/udp dns-server

# Forward both TCP and UDP
docker run -p 8080:80/tcp -p 5353:53/udp myapp

Docker Compose Port Mapping

In docker-compose.yml, use the ports section:

version: '3'
services:
  web:
    image: nginx
    ports:
      - "8080:80"
      - "8443:443"

  database:
    image: postgres
    ports:
      - "127.0.0.1:5432:5432"

  app:
    image: myapp
    ports:
      - "3000-3005:3000-3005"

Start services:

docker-compose up

All port mappings are applied automatically.

Short vs Long Syntax

Docker Compose supports both syntaxes:

Short syntax:

ports:
  - "8080:80"
  - "127.0.0.1:5432:5432"

Long syntax (more explicit):

ports:
  - target: 80        # Container port
    published: 8080   # Host port
    protocol: tcp
    mode: host

  - target: 5432
    published: 5432
    host_ip: 127.0.0.1

Adding Ports to Running Containers

You cannot add port forwarding to a running container directly. You must stop, commit, and restart with new port mappings.

Method 1: Commit and Restart

# Stop the container
docker stop mycontainer

# Commit current state to new image
docker commit mycontainer myapp-with-data

# Remove old container
docker rm mycontainer

# Run with new port mapping
docker run -p 8080:80 --name mycontainer myapp-with-data

Method 2: iptables Port Forwarding

Forward ports using iptables without restarting:

# Get container IP
CONTAINER_IP=$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' mycontainer)

# Forward host port 8080 to container port 80
sudo iptables -t nat -A DOCKER -p tcp --dport 8080 -j DNAT --to-destination ${CONTAINER_IP}:80

# Allow the forwarded traffic
sudo iptables -t nat -A POSTROUTING -j MASQUERADE -p tcp --source ${CONTAINER_IP} --destination ${CONTAINER_IP} --dport 80

This forwards traffic without restarting the container, but the mapping is lost when Docker restarts.

Method 3: Docker Proxy

Use a reverse proxy container (like Nginx or Traefik) to route traffic to containers without direct port mappings:

# Run application without port mapping
docker run --name myapp myimage

# Run nginx-proxy
docker run -d -p 80:80 -v /var/run/docker.sock:/tmp/docker.sock:ro jwilder/nginx-proxy

# nginx-proxy automatically detects containers and proxies to them

Checking Port Mappings

View port mappings for a running container:

# Using docker ps
docker ps

# Using docker port
docker port 

# Using docker inspect
docker inspect  | grep -A 20 "Ports"

Example output:

"Ports": {
  "80/tcp": [
    {
      "HostIp": "0.0.0.0",
      "HostPort": "8080"
    }
  ]
}

Common Port Mapping Patterns

Web Applications

# Frontend application
docker run -p 3000:3000 react-app

# Backend API
docker run -p 8000:8000 api-server

# Database (localhost only)
docker run -p 127.0.0.1:5432:5432 postgres

Development Environment

# docker-compose.yml
version: '3'
services:
  app:
    build: .
    ports:
      - "3000:3000"    # Web server
      - "35729:35729"  # Live reload
    volumes:
      - .:/app

  db:
    image: postgres
    ports:
      - "127.0.0.1:5432:5432"

Microservices

version: '3'
services:
  api-gateway:
    image: api-gateway
    ports:
      - "80:8080"

  auth-service:
    image: auth-service
    # No external ports - only accessible via Docker network

  user-service:
    image: user-service
    # No external ports

Only the API gateway is exposed externally. Other services communicate via Docker's internal network.

Troubleshooting Port Conflicts

Port Already in Use

Error:

Error: bind: address already in use

Find what's using the port:

# Linux/macOS
sudo lsof -i :8080

# Or use netstat
sudo netstat -tlnp | grep :8080

# Or ss
sudo ss -tlnp | grep :8080

Solutions:

Use a different host port:
```
docker run -p 8081:80 nginx
```

Stop the conflicting service:

sudo systemctl stop apache2
docker run -p 80:80 nginx

Kill the process using the port:
```
kill 
```

Container Port Not Responding

Check if the service inside the container is actually listening:

# Execute command in running container
docker exec mycontainer netstat -tlnp

# Or check if the port is open
docker exec mycontainer nc -zv localhost 80

If the service isn't running inside the container, the port mapping won't help.

Firewall Blocking Access

If the container is accessible from localhost but not externally, check your firewall:

# Check if Docker added firewall rules
sudo iptables -L -n -v | grep 8080

# Allow the port through ufw
sudo ufw allow 8080/tcp

# Or firewalld
sudo firewall-cmd --permanent --add-port=8080/tcp
sudo firewall-cmd --reload

Docker Networks and Port Forwarding

Containers on the same Docker network can communicate without port forwarding:

# Create a network
docker network create mynetwork

# Run database (no port forwarding needed)
docker run --network mynetwork --name db postgres

# Run app (connects to 'db' by name)
docker run --network mynetwork --name app -p 8080:80 myapp

Inside myapp, connect to the database at postgresql://db:5432 - no port forwarding needed because they're on the same network.

Host Network Mode

Skip Docker networking entirely and use the host's network:

docker run --network host nginx

In host mode:

No port mapping needed
Container uses host ports directly
Service on container port 80 is accessible at host port 80
Less isolation but simpler networking

Use case: When you need maximum network performance or when port mapping overhead is a concern.

Best Practices

Don't expose databases publicly: Always bind database ports to localhost:

docker run -p 127.0.0.1:5432:5432 postgres

Use high ports for development: Avoid ports below 1024 which require root:

# Good
docker run -p 8080:80 nginx

# Requires sudo
sudo docker run -p 80:80 nginx

Document your port mappings: In your README or docker-compose.yml comments:

services:
  app:
    ports:
      - "3000:3000"  # Web UI
      - "9229:9229"  # Node.js debugger
      - "35729:35729"  # Live reload

Use Docker Compose for complex setups: It's easier to manage multiple port mappings in YAML than remembering long docker run commands.

Plan your port allocation: Maintain a list of ports used by different containers to avoid conflicts.

Port forwarding is fundamental to using Docker effectively. Whether you're running a simple web server or a complex microservices architecture, understanding how to map ports between your host and containers lets you expose services securely and accessibly. Use -p for basic mapping, Docker Compose for complex setups, and remember to bind sensitive services like databases to localhost only.

How to Use the AWS account_id Variable in Terraform

DevOps Daily Team — Fri, 20 Jun 2025 09:00:00 GMT

TLDR

To use the AWS account_id variable in Terraform, leverage the aws_caller_identity data source. This allows you to dynamically retrieve the account ID for use in your configurations.

The AWS account_id is a unique identifier for your AWS account. In Terraform, you can dynamically retrieve this value using the aws_caller_identity data source. This guide will show you how to use the account_id variable effectively.

Why Use the account_id Variable?

Dynamic Configurations: Avoid hardcoding the account ID in your Terraform files.
Multi-Account Setups: Simplify configurations for environments spanning multiple AWS accounts.
Security: Reduce the risk of errors caused by incorrect account IDs.

Retrieving the account_id

Use the aws_caller_identity data source to retrieve the account ID dynamically.

Example: Using aws_caller_identity

data "aws_caller_identity" "current" {}

output "account_id" {
  value = data.aws_caller_identity.current.account_id
}

In this example, the account_id is retrieved and displayed as an output.

Using the account_id in Resources

You can use the account_id variable in resource definitions to make your configurations dynamic.

Example: S3 Bucket Naming

resource "aws_s3_bucket" "example" {
  bucket = "example-bucket-${data.aws_caller_identity.current.account_id}"
  acl    = "private"
}

This configuration creates an S3 bucket with a name that includes the account ID.

Best Practices

Avoid Hardcoding: Always use the aws_caller_identity data source to retrieve the account ID dynamically.
Use Outputs: Export the account_id as an output for reuse in other configurations.
Test in Non-Production: Validate your configurations in a non-production environment before applying them to production.

By using the AWS account_id variable dynamically, you can create more flexible and secure Terraform configurations that adapt to different environments and accounts.

How to Make an Existing Git Branch Track a Remote Branch

DevOps Daily Team — Wed, 18 Jun 2025 12:00:00 GMT

You created a local branch and pushed it to a remote repository, but now when you run git pull or git push, Git asks you to specify which remote branch to use. This happens because your local branch is not configured to track the remote branch.

TLDR: To make your current branch track a remote branch, use git branch --set-upstream-to=origin/branch-name or git push -u origin branch-name. The -u flag sets up tracking automatically when pushing. Once configured, you can use git pull and git push without specifying the remote branch.

In this guide, you'll learn how to set up branch tracking and understand what it does.

Prerequisites

You'll need Git installed, a repository with remote access, and at least one local branch. Understanding basic Git concepts like branches, remotes, and push/pull operations will help you follow along.

Understanding Branch Tracking

Branch tracking creates a relationship between a local branch and a remote branch. When tracking is set up, Git remembers which remote branch corresponds to your local branch:

Local Branch          Remote Branch
------------          -------------
feature-auth   <-->   origin/feature-auth

With this relationship established, you can use simplified commands:

# Without tracking:
git pull origin feature-auth
git push origin feature-auth

# With tracking:
git pull
git push

Git automatically knows which remote branch to use based on the tracking configuration.

Setting Up Tracking for Current Branch

To configure your current branch to track a remote branch:

# Make current branch track a remote branch
git branch --set-upstream-to=origin/feature-auth

# Shorter syntax
git branch -u origin/feature-auth

After running this command, your local branch tracks the specified remote branch. Verify the configuration:

# Check tracking information
git branch -vv

# Output shows tracking relationship:
# * feature-auth a1b2c3d [origin/feature-auth] Add authentication

The bracketed text shows which remote branch your local branch tracks.

Setting Up Tracking While Pushing

The easiest way to set up tracking is during your first push:

# Push and set upstream in one command
git push -u origin feature-auth

# Or use the long form
git push --set-upstream origin feature-auth

The -u flag tells Git to set the current branch to track the remote branch you're pushing to. This is the most common workflow when creating new branches:

# Create and switch to new branch
git checkout -b new-feature

# Make some commits
git add .
git commit -m "Add new feature"

# Push and set up tracking
git push -u origin new-feature

# Future pushes only need
git push

After the initial push with -u, subsequent pushes and pulls do not need to specify the remote or branch name.

Setting Up Tracking for Different Branch Names

Sometimes your local branch has a different name than the remote branch:

# Local branch is 'feature', remote is 'feature-auth'
git branch --set-upstream-to=origin/feature-auth feature

# Or if it's your current branch
git branch -u origin/feature-auth

This works but can be confusing. It's generally better to rename your local branch to match:

# Rename local branch to match remote
git branch -m feature feature-auth

# Set up tracking
git branch -u origin/feature-auth

Now your local and remote branch names match, reducing confusion.

Checking Current Tracking Configuration

To see which branches track which remotes:

# Show tracking info for all branches
git branch -vv

# Output:
#   main           a1b2c3d [origin/main] Update README
# * feature-auth   e4f5g6h [origin/feature-auth: ahead 2] Add OAuth
#   develop        i7j8k9l [origin/develop: behind 1] Merge pull request

This shows:

Current branch (marked with *)
Latest commit hash
Remote tracking branch in brackets
Sync status (ahead/behind)
Latest commit message

For detailed information about a specific branch:

# Get remote tracking branch for current branch
git rev-parse --abbrev-ref --symbolic-full-name @{u}

# Output: origin/feature-auth

Removing Tracking Relationship

To stop tracking a remote branch:

# Remove upstream tracking for current branch
git branch --unset-upstream

# Remove tracking for specific branch
git branch --unset-upstream feature-auth

After removing tracking, you'll need to specify the remote and branch name when pushing or pulling.

Setting Up Tracking When Checking Out Remote Branches

When you check out a remote branch, Git automatically creates a tracking relationship:

# List remote branches
git branch -r

# Output:
#   origin/main
#   origin/feature-auth
#   origin/develop

# Check out remote branch (creates local tracking branch)
git checkout feature-auth

# Git automatically runs something like:
# git checkout -b feature-auth --track origin/feature-auth

This creates a local branch that tracks the remote branch with the same name.

Explicitly create a tracking branch with a custom name:

# Create local branch with different name tracking remote
git checkout -b my-feature --track origin/feature-auth

# Verify tracking
git branch -vv
# Output: * my-feature a1b2c3d [origin/feature-auth] Add feature

Handling Multiple Remotes

If you work with multiple remotes (like origin and upstream), specify which remote to track:

# Track a branch on the upstream remote
git branch --set-upstream-to=upstream/main

# Track a branch on origin
git branch --set-upstream-to=origin/main

# Check configuration
git branch -vv

This is common when working with forks:

Your Workflow:
  Local main   -->  origin/main (your fork)
               \->  upstream/main (original repo)

You might configure your main branch to track upstream/main for pulling updates, while pushing to origin/main.

Setting Default Push Behavior

Configure how Git handles pushing branches that do not have tracking set up:

# Only push current branch to its upstream
git config --global push.default simple

# Push all branches with matching names
git config --global push.default matching

# Only push current branch to same-named remote branch
git config --global push.default current

The simple setting is the default and safest option - it only pushes the current branch to its tracked upstream branch.

Tracking Configuration in Git Config

Branch tracking is stored in your repository's Git config:

# View branch configuration
git config --local --get-regexp "branch.*"

# Output:
# branch.main.remote origin
# branch.main.merge refs/heads/main
# branch.feature-auth.remote origin
# branch.feature-auth.merge refs/heads/feature-auth

This shows the remote and merge configuration for each tracking branch. You can manually edit these:

# Set remote for a branch
git config branch.feature-auth.remote origin

# Set merge branch
git config branch.feature-auth.merge refs/heads/feature-auth

But using git branch -u is easier and less error-prone.

Setting Up Tracking in Scripts

When automating Git workflows, set up tracking in scripts:

#!/bin/bash

# Get current branch name
BRANCH=$(git branch --show-current)

# Set up tracking if not already configured
if ! git config branch.$BRANCH.remote > /dev/null 2>&1; then
  echo "Setting up tracking for $BRANCH"
  git push -u origin "$BRANCH"
else
  echo "Branch $BRANCH already tracks $(git config branch.$BRANCH.remote)/$(git config branch.$BRANCH.merge | sed 's|refs/heads/||')"
fi

This script checks if tracking is configured and sets it up if needed.

Tracking and Pull/Push Behavior

With tracking configured, Git's behavior changes:

# Without tracking:
git pull
# fatal: The current branch feature-auth has no upstream branch.

# With tracking:
git pull
# Already up to date.

The same applies to push:

# Without tracking:
git push
# fatal: The current branch feature-auth has no upstream branch.
# Use: git push --set-upstream origin feature-auth

# With tracking:
git push
# Everything up-to-date

Fixing No upstream branch Errors

When you see "The current branch has no upstream branch", fix it with:

# Follow Git's suggestion
git push --set-upstream origin branch-name

# Or set tracking without pushing
git branch -u origin/branch-name

# Then push
git push

After setting upstream, the error will not occur again for that branch.

Best Practices for Branch Tracking

Always set up tracking when creating new branches:

# Good: Set tracking immediately
git checkout -b new-feature
git push -u origin new-feature

# Less ideal: Set tracking later
git checkout -b new-feature
git push origin new-feature
git branch -u origin/new-feature

Keep local and remote branch names identical to avoid confusion:

# Good: Matching names
git checkout -b feature-auth
git push -u origin feature-auth

# Confusing: Different names
git checkout -b my-feature
git push -u origin feature-auth

Regularly check tracking status:

# Quick check of current branch
git status

# Detailed check of all branches
git branch -vv

Now you know how to set up and manage branch tracking in Git. The -u flag when pushing is the easiest way to establish tracking for new branches, while git branch --set-upstream-to lets you configure tracking for existing branches. With tracking configured, your Git workflow becomes cleaner and requires fewer keystrokes.

How do I Get Flask to Run on Port 80?

DevOps Daily Team — Wed, 18 Jun 2025 10:00:00 GMT

By default, Flask's development server runs on port 5000, but you might want to run your application on port 80 so users can access it without specifying a port number in the URL. Port 80 is the standard HTTP port, but it's also a privileged port on Unix-like systems, which means you need special permissions to bind to it.

This guide covers several approaches to running Flask on port 80, from quick development solutions to production-ready setups.

TLDR

For development, run Flask with sudo: sudo python app.py and set app.run(host='0.0.0.0', port=80). For production, use a reverse proxy like Nginx or Apache to forward requests from port 80 to your Flask app running on a higher port like 5000 or 8000. Never run Flask's development server in production.

Prerequisites

You need Python and Flask installed on your system. Basic familiarity with Flask applications and command-line operations will help. If you're deploying to production, you should understand web servers and reverse proxies.

Understanding Privileged Ports

Ports numbered 1-1023 are considered privileged ports on Unix-like systems (Linux, macOS). Only processes running as root can bind to these ports. This security measure prevents regular users from running services that could impersonate system services.

Port 80 is the standard HTTP port, which means:

Users can access your site at http://example.com instead of http://example.com:5000
Browsers connect to port 80 by default when you don't specify a port
You need elevated privileges to use it

Quick Solution: Running Flask with Sudo

The simplest way to run Flask on port 80 during development is using sudo:

# app.py
from flask import Flask

app = Flask(__name__)

@app.route('/')
def hello():
    return 'Hello from port 80!'

if __name__ == '__main__':
    # Bind to all interfaces on port 80
    app.run(host='0.0.0.0', port=80)

Run it with elevated privileges:

sudo python app.py

Or if using Python 3 explicitly:

sudo python3 app.py

You'll see output like:

 * Running on http://0.0.0.0:80
 * Running on http://127.0.0.1:80
 * Running on http://192.168.1.100:80

Now you can access your app at http://localhost without specifying a port.

Important: This approach works for development and testing, but you should never run Flask's built-in development server in production. It's not designed for security, performance, or stability under real-world load.

Using setcap to Grant Port Binding Permissions

Instead of running your entire Python process as root, you can grant the Python interpreter permission to bind to privileged ports:

# Give Python the capability to bind to privileged ports
sudo setcap 'cap_net_bind_service=+ep' /usr/bin/python3.10

Replace python3.10 with your actual Python version:

# Find your Python path
which python3

# Example output: /usr/bin/python3.10
# Then use that path with setcap
sudo setcap 'cap_net_bind_service=+ep' /usr/bin/python3.10

After setting this capability, you can run your Flask app on port 80 without sudo:

python3 app.py

This approach is more secure than using sudo because only the port binding operation has elevated privileges, not your entire application.

Note: This affects all Python scripts using that interpreter. If you're using virtual environments, you need to set the capability on the Python binary inside the virtual environment.

Production Solution: Using a Reverse Proxy

The recommended way to run Flask in production is behind a reverse proxy like Nginx or Apache. The reverse proxy runs on port 80 and forwards requests to your Flask application running on a higher, unprivileged port.

Setting Up with Nginx

First, run your Flask app on an unprivileged port using a production WSGI server:

# app.py
from flask import Flask

app = Flask(__name__)

@app.route('/')
def hello():
    return 'Hello from Flask behind Nginx!'

if __name__ == '__main__':
    # Run on port 8000 for production
    app.run(host='127.0.0.1', port=8000)

In production, use Gunicorn instead of Flask's development server:

# Install Gunicorn
pip install gunicorn

# Run your Flask app with Gunicorn on port 8000
gunicorn -w 4 -b 127.0.0.1:8000 app:app

The -w 4 flag runs 4 worker processes for handling concurrent requests. The -b 127.0.0.1:8000 binds to localhost on port 8000.

Configure Nginx to proxy requests from port 80 to your Flask app:

# /etc/nginx/sites-available/flask-app
server {
    listen 80;
    server_name example.com www.example.com;

    location / {
        proxy_pass http://127.0.0.1:8000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

Enable the site and restart Nginx:

# Create a symbolic link to enable the site
sudo ln -s /etc/nginx/sites-available/flask-app /etc/nginx/sites-enabled/

# Test the configuration
sudo nginx -t

# Reload Nginx
sudo systemctl reload nginx

Now requests to http://example.com on port 80 are forwarded to your Flask app running on port 8000.

Using systemd to Manage Your Flask Service

Create a systemd service file to automatically start your Flask app:

# /etc/systemd/system/flask-app.service
[Unit]
Description=Flask Application
After=network.target

[Service]
User=www-data
Group=www-data
WorkingDirectory=/var/www/flask-app
Environment="PATH=/var/www/flask-app/venv/bin"
ExecStart=/var/www/flask-app/venv/bin/gunicorn -w 4 -b 127.0.0.1:8000 app:app

[Install]
WantedBy=multi-user.target

Start and enable the service:

# Reload systemd to recognize the new service
sudo systemctl daemon-reload

# Start the service
sudo systemctl start flask-app

# Enable it to start on boot
sudo systemctl enable flask-app

# Check status
sudo systemctl status flask-app

Using iptables for Port Forwarding

Another approach is using iptables to forward traffic from port 80 to a higher port:

# Forward traffic from port 80 to port 5000
sudo iptables -t nat -A PREROUTING -p tcp --dport 80 -j REDIRECT --to-port 5000

Your Flask app runs on port 5000 without elevated privileges, but users access it on port 80:

# app.py
from flask import Flask

app = Flask(__name__)

@app.route('/')
def hello():
    return 'Hello via iptables forwarding!'

if __name__ == '__main__':
    app.run(host='0.0.0.0', port=5000)

Run normally:

python app.py

Access at http://localhost (port 80), which forwards to port 5000.

To make iptables rules persistent across reboots:

# Install iptables-persistent
sudo apt-get install iptables-persistent

# Save current rules
sudo netfilter-persistent save

Docker Approach

If you're running Flask in Docker, you can map port 80 on the host to your container's port without elevated privileges for the Flask app:

# Dockerfile
FROM python:3.10-slim

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY . .

# App runs on port 5000 inside container
EXPOSE 5000

CMD ["python", "app.py"]

Run the container with port mapping:

# Map host port 80 to container port 5000
sudo docker run -p 80:5000 flask-app

The Flask app inside the container runs on port 5000 (unprivileged), but Docker maps it to port 80 on the host.

Security Considerations

When running Flask on port 80, keep these security practices in mind:

Never use Flask's development server in production: It's single-threaded, doesn't handle concurrent requests well, and has known security vulnerabilities. Always use a production WSGI server like Gunicorn, uWSGI, or Waitress.

Run Flask as a non-privileged user: If using a reverse proxy or systemd, configure the service to run as a dedicated user like www-data, not as root.

Use HTTPS in production: Port 80 serves unencrypted HTTP traffic. For production sites, use port 443 with SSL/TLS certificates (Let's Encrypt is free). Your reverse proxy can handle SSL termination.

Keep Flask behind a firewall: If your Flask app binds to 0.0.0.0:8000, it's accessible from any network interface. Use 127.0.0.1:8000 to ensure it only accepts connections from localhost (via the reverse proxy).

Choosing the Right Approach

Here's when to use each method:

Development and testing: Use sudo python app.py for quick local testing. It's simple and gets you running fast.

Production deployment: Always use a reverse proxy (Nginx or Apache) with a production WSGI server (Gunicorn). This is the industry standard for good reason - it's secure, performant, and maintainable.

Containerized deployments: Use Docker's port mapping to handle the privileged port binding at the container runtime level.

Shared hosting or restricted environments: If you can't install a reverse proxy, use iptables port forwarding or setcap.

Common Issues and Solutions

Permission denied when binding to port 80

Error:

PermissionError: [Errno 13] Permission denied

Solution: Either use sudo, setcap, or run your app on a higher port with a reverse proxy.

Port 80 already in use

Error:

OSError: [Errno 98] Address already in use

Solution: Another service (likely Apache or Nginx) is already using port 80. Check what's running:

sudo lsof -i :80
# or
sudo netstat -tlnp | grep :80

Stop the conflicting service or configure it to proxy to your Flask app instead.

App works locally but not externally

If you can access your app at http://localhost but not from other machines, check:

Flask is bound to 0.0.0.0, not 127.0.0.1
Your firewall allows incoming connections on port 80
Your cloud provider's security group permits port 80 traffic

Running Flask on port 80 is straightforward once you understand the privilege requirements. For development, sudo or setcap works fine. For production, always use a reverse proxy with a production-grade WSGI server. This combination gives you security, performance, and the ability to serve your Flask app on the standard HTTP port that users expect.

Can Two Different Sockets Share a TCP Port?

DevOps Daily Team — Thu, 12 Jun 2025 09:00:00 GMT

TLDR: Yes, multiple sockets can share a port in specific scenarios. Different connections (unique combinations of local IP, local port, remote IP, remote port) can coexist on the same local port - this is how web servers handle multiple clients. Multiple processes can bind to the same port using SO_REUSEPORT (Linux 3.9+) for load balancing. SO_REUSEADDR lets you rebind a port in TIME_WAIT state but doesn't allow true sharing of listening ports.

The question "can two sockets share a port?" has different answers depending on what you mean by "share." Let's break down the scenarios.

How TCP Connections Are Identified

A TCP connection is uniquely identified by a tuple of four values:

Connection = (Local IP, Local Port, Remote IP, Remote Port)

Example connections on port 80:
Connection 1: (192.168.1.10:80, 192.168.1.100:54321)
Connection 2: (192.168.1.10:80, 192.168.1.101:54322)
Connection 3: (192.168.1.10:80, 192.168.1.100:54323)

All three share local port 80, but they're different connections

As long as at least one element differs, connections are unique. This is how a single web server on port 80 handles thousands of clients simultaneously - each client connection has a different remote IP or remote port.

Scenario 1: Server Accepting Multiple Connections (Always Allowed)

When a server binds to a port and listens, it creates one socket. When clients connect, the accept() call creates new sockets - one per connection:

import socket

# Server socket - binds to port 8080
server_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server_socket.bind(('0.0.0.0', 8080))
server_socket.listen(5)

print("Server listening on port 8080")

while True:
    # Each accept() creates a new socket sharing port 8080
    client_socket, addr = server_socket.accept()
    print(f"New connection from {addr}")

    # client_socket is a new socket, but still uses local port 8080
    # Connection: (server_ip:8080, client_ip:client_port)

    # Handle the client (in real code, do this in a thread)
    data = client_socket.recv(1024)
    client_socket.sendall(data)
    client_socket.close()

Each client_socket is a separate socket object, but they all share the local port 8080. This is normal TCP behavior and always works. The operating system distinguishes connections by the remote address.

Scenario 2: Binding Multiple Sockets to Same Port (Usually Fails)

By default, you cannot bind two separate sockets to the same port:

import socket

# First socket binds successfully
sock1 = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock1.bind(('0.0.0.0', 8080))
print("Socket 1 bound to port 8080")

# Second socket fails with "Address already in use"
sock2 = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
try:
    sock2.bind(('0.0.0.0', 8080))
except OSError as e:
    print(f"Socket 2 failed: {e}")
    # Output: Socket 2 failed: [Errno 48] Address already in use

This protection prevents conflicts - if two programs could listen on the same port, incoming connections would be randomly assigned, causing chaos.

Scenario 3: SO_REUSEADDR (Rebinding After Close)

SO_REUSEADDR lets you bind to a port that's in TIME_WAIT state. When a server closes a socket, the OS keeps the port reserved for a short period (typically 30-120 seconds) to handle any delayed packets:

Server closes connection:
Connection moves to TIME_WAIT state
Port 8080 is reserved for ~60 seconds
New bind() to port 8080 fails... unless SO_REUSEADDR is set

Here's how to use it:

import socket

def create_server(port):
    sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

    # Allow reusing the port immediately after close
    sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)

    sock.bind(('0.0.0.0', port))
    sock.listen(5)
    return sock

# First server
server1 = create_server(8080)
print("Server started on port 8080")

# Stop and restart without waiting for TIME_WAIT
server1.close()

# Without SO_REUSEADDR, this would fail for ~60 seconds
# With SO_REUSEADDR, it works immediately
server2 = create_server(8080)
print("Server restarted on port 8080")

This is standard practice for server applications - you don't want to wait a minute to restart your server after a crash.

Important: SO_REUSEADDR does not let multiple processes bind and listen on the same port simultaneously on most systems. It only helps with rebinding after a close.

Linux 3.9+ and modern BSD systems support SO_REUSEPORT, which allows multiple sockets to bind to the same port:

import socket
import os

def create_server_with_reuseport(port):
    sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

    # Enable SO_REUSEPORT
    sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEPORT, 1)

    sock.bind(('0.0.0.0', port))
    sock.listen(5)
    return sock

# Process 1
server1 = create_server_with_reuseport(8080)
print(f"Server 1 (PID {os.getpid()}) bound to port 8080")

# Process 2 (in a different process, but shown here for illustration)
server2 = create_server_with_reuseport(8080)
print(f"Server 2 (PID {os.getpid()}) bound to port 8080")

# Both sockets are listening on port 8080
# Kernel distributes incoming connections between them

When a client connects, the kernel uses a hash of the connection tuple to pick which socket receives it. This provides load balancing across multiple processes:

Client 1 connects -> Kernel routes to server1
Client 2 connects -> Kernel routes to server2
Client 3 connects -> Kernel routes to server1
...

This is how modern web servers like NGINX can run multiple worker processes all listening on port 80.

Real-World Example: Multi-Process Server

import socket
import os
from multiprocessing import Process

def worker(worker_id):
    """Each worker process binds to the same port."""
    sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEPORT, 1)
    sock.bind(('0.0.0.0', 8080))
    sock.listen(5)

    print(f"Worker {worker_id} (PID {os.getpid()}) listening on port 8080")

    while True:
        client, addr = sock.accept()
        print(f"Worker {worker_id} handling {addr}")

        # Handle request
        client.sendall(f"Handled by worker {worker_id}\n".encode())
        client.close()

if __name__ == '__main__':
    # Start 4 worker processes, all listening on port 8080
    workers = []
    for i in range(4):
        p = Process(target=worker, args=(i,))
        p.start()
        workers.append(p)

    # Wait for workers
    for p in workers:
        p.join()

When you connect to port 8080, different workers handle different connections, providing parallel processing.

Limitations of SO_REUSEPORT

Same user ID: Only processes with the same effective user ID can share a port (security measure)
All or nothing: Either all sockets use SO_REUSEPORT or none do. You can't mix.
Load balancing is simple: The kernel uses a hash function, not round-robin or connection count. Uneven distribution is possible.
Platform support: Linux 3.9+, modern BSDs. Not available on Windows or older systems.

Scenario 5: Different IP Addresses

You can bind different sockets to the same port if they use different IP addresses:

import socket

# Server with multiple network interfaces
# eth0: 192.168.1.10
# eth1: 10.0.0.5

# Bind to port 8080 on first interface
sock1 = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock1.bind(('192.168.1.10', 8080))
sock1.listen(5)

# Bind to port 8080 on second interface - this works!
sock2 = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock2.bind(('10.0.0.5', 8080))
sock2.listen(5)

print("Listening on 192.168.1.10:8080 and 10.0.0.5:8080")

This works because the bind addresses are different:

192.168.1.10:8080 vs 10.0.0.5:8080

If you bind to 0.0.0.0:8080 (all interfaces), you cannot bind another socket to any specific IP on port 8080.

UDP works differently from TCP. With SO_REUSEADDR, multiple UDP sockets can bind to the same port:

import socket

# First UDP socket
sock1 = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
sock1.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
sock1.bind(('0.0.0.0', 9090))

# Second UDP socket on same port
sock2 = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
sock2.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
sock2.bind(('0.0.0.0', 9090))

# Both receive the same packets (multicast-like behavior)

Both sockets receive a copy of each incoming packet. This is useful for:

Multiple processes monitoring the same data
Service discovery protocols
Multicast receivers

Checking What's Using a Port

When debugging port conflicts, check what's bound:

# Linux - show process using port 8080
sudo lsof -i :8080
# or
sudo ss -tulpn | grep :8080

# macOS
sudo lsof -i :8080

# Windows
netstat -ano | findstr :8080

You'll see output like:

COMMAND   PID   USER   FD   TYPE   DEVICE SIZE/OFF NODE NAME
python    1234  john   3u   IPv4   98765  0t0      TCP *:8080 (LISTEN)
python    1235  john   3u   IPv4   98766  0t0      TCP *:8080 (LISTEN)

If you see multiple processes with the same port, they're using SO_REUSEPORT.

Common Mistakes

Forgetting SO_REUSEADDR on Servers

# Wrong - server won't restart quickly after crash
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.bind(('0.0.0.0', 8080))

# Right - server can restart immediately
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
sock.bind(('0.0.0.0', 8080))

Using SO_REUSEPORT Without Understanding Distribution

# This doesn't give you control over which worker gets which connection
sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEPORT, 1)

# If you need specific routing logic, use a single listening socket
# and distribute connections yourself

# This might fail on Windows or old Linux
sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEPORT, 1)

# Check if SO_REUSEPORT is defined
if hasattr(socket, 'SO_REUSEPORT'):
    sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEPORT, 1)
else:
    print("SO_REUSEPORT not supported on this platform")

Use SO_REUSEADDR:

Always, for server applications
Allows quick restart after crashes
Standard practice

Use SO_REUSEPORT:

Multi-process servers for parallel processing
Taking advantage of multiple CPU cores
When you want the kernel to load balance
Only when available on your platform

Use multiple IPs with same port:

When you want different services on different interfaces
Internal vs external access on same port
Segregating traffic by network

Two sockets can share a port in multiple ways - through accepting multiple connections on a listening socket, using SO_REUSEPORT for multi-process load balancing, or binding to different IP addresses. Understanding these mechanisms helps you build robust, high-performance network applications.

Why Your CI/CD Pipeline Is Slower Than It Should Be (and How to Fix It)

Tue, 10 Jun 2025 09:00:00 GMT

TLDR

Most slow pipelines are a matter of configuration, not raw compute. Parallelize independent work, cache dependencies and images, reuse build artifacts, and run targeted tests. These five changes often shave minutes off every run.

Why this matters

Slow pipelines cost time and momentum. Every extra minute waiting for feedback lowers developer velocity and increases the cost of iteration. The steps below are practical and platform agnostic. I include short examples for GitHub Actions and GitLab CI because they are easy to adapt.

1) Too many serial steps

Running unrelated tasks one after the other wastes wall clock time. Treat jobs as units of work and run jobs in parallel when they do not depend on each other.

GitHub Actions example

Explanation: two independent jobs run at the same time, saving total time.

jobs:
  lint:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npm ci
      - run: npm run lint

  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npm ci
      - run: npm test

Quick tip: if jobs share the same setup cost, consider extracting common setup into a reusable job or cache the results so the overhead is smaller.

2) Pulling "latest" base images

Using floating tags like latest forces a fresh image pull and makes builds unpredictable.

Fix: pin a specific version and optionally pin by digest when you need absolute reproducibility.

Dockerfile example

Explanation: pinning to a minor version gives stability while keeping security updates available.

FROM node:20-alpine
# ... rest of Dockerfile ...

When you need byte-for-byte reproducibility, use an image digest:

# example only, replace with the digest your registry shows
FROM node:20-alpine@sha256:0123456789abcdef...

If your CI offers warm image caches, configure your runners to keep common base images between runs.

3) No dependency caching

Downloading all dependencies every run is a big time sink. Use the CI cache feature and key it on the lockfile.

GitHub Actions npm cache example

Explanation: this caches the npm cache directory and only restores when package-lock.json changes.

- name: Cache node modules
  uses: actions/cache@v4
  with:
    path: ~/.npm
    key: ${{ runner.os }}-node-${{ hashFiles('**/package-lock.json') }}
    restore-keys: |
      ${{ runner.os }}-node-

- name: Install
  run: npm ci

For pnpm cache the store path is usually ~/.pnpm-store or the path configured in your project.
For pip use ~/.cache/pip. For Maven and Gradle cache the .m2 or .gradle directories.
Use restore-keys to get partial cache hits when the exact key is not found.

4) Skipping artifact reuse

Building once and throwing away the result is wasteful. Save build outputs and reuse them in downstream jobs.

GitLab CI example

Explanation: build job creates artifacts that are used by deploy without rebuilding.

build:
  stage: build
  script: ./build.sh
  artifacts:
    paths:
      - dist/

deploy:
  stage: deploy
  dependencies:
    - build
  script: ./deploy.sh

GitHub Actions artifact example

Explanation: upload artifacts in the build job, then download them in deploy.

# in build job
- uses: actions/upload-artifact@v4
  with:
    name: app-dist
    path: dist/

# in deploy job
- uses: actions/download-artifact@v4
  with:
    name: app-dist
    path: dist/

If your platform supports container image layers caching, push a cached intermediate image to your registry so downstream jobs can pull a small delta.

5. Running every test, every time

Full test suites are expensive. Run fast checks on every commit and full suites only when needed.

Selective test example for Jest

Explanation: --changedSince runs only tests affected by recent changes.

npx jest --changedSince=origin/main

Alternative: use a simple change detection step in CI and set which suites to run. Example idea:

If only frontend/ files changed, run unit and browser tests.
If backend/ files changed, run backend unit and integration tests.

Small change detection snippet (bash)

Explanation: sets a variable you can use to conditionally run jobs or steps.

CHANGED=$(git diff --name-only origin/main...HEAD)
if echo "$CHANGED" | grep -q '^frontend/'; then
  echo "run frontend tests"
fi

Extra tips that save time

Use a matrix for similar jobs instead of duplicating config.
Prefer npm ci over npm install on CI for reproducible installs and speed.
Keep CI images small and only install required tools.
Run heavy integration tests on a schedule or after merge to main, not on every push.
If you use shared runners, tune concurrency or add self-hosted runners for heavy workloads.

Short checklist to apply

Parallelize independent jobs.
Pin base images and use image caches.
Cache dependency directories with a key based on the lockfile.
Save and reuse build artifacts across jobs.
Run targeted tests when possible, full suites on protected branches or merges.

Conclusion

Most pipeline speed problems are fixable with configuration and small investments. Start with parallel jobs and caching. Then add artifact reuse and selective testing. You will see faster feedback and higher team throughput.

Thanks for reading. Ship faster.

What do PTY and TTY Mean?

DevOps Daily Team — Thu, 05 Jun 2025 15:00:00 GMT

When working with Linux or Unix systems, you'll encounter terms like TTY and PTY in various contexts - from SSH sessions to Docker containers to terminal multiplexers like screen and tmux. These cryptic three-letter acronyms represent fundamental concepts in how Unix systems handle interactive input and output.

Understanding TTY and PTY helps you troubleshoot terminal issues, work effectively with remote sessions, and understand how command-line programs interact with your system.

TLDR

TTY (TeleTYpewriter) is a terminal device - originally physical hardware, now mostly software. PTY (Pseudo-TTY) is a virtual terminal created by software like SSH, terminal emulators, or tmux. When you open Terminal.app or PuTTY, you're using a PTY. When a program checks if it's running interactively (to show colors or progress bars), it's checking if it's connected to a TTY/PTY.

Prerequisites

Basic familiarity with the Linux command line will help you understand the examples. No advanced knowledge is required.

The History: Physical Teletypewriters

TTY stands for "TeleTYpewriter" - actual physical typewriter-like devices used to interact with computers in the 1960s and 1970s.

User types on keyboard → Sends to computer
Computer processes command → Sends output back
Output prints on paper

These were mechanical terminals with keyboards and printers (no screens). When you typed a command, you'd see it printed on paper, followed by the output. The Unix design for terminal I/O still reflects this history.

Modern TTYs: Virtual Terminals

Modern Linux systems have virtual terminals (consoles) accessible with Ctrl+Alt+F1 through Ctrl+Alt+F7:

# See your current TTY
tty

Output on a virtual console:

/dev/tty1

Output in a terminal emulator:

/dev/pts/0

The difference:

/dev/tty1, /dev/tty2, etc.: Virtual consoles (direct kernel-provided terminals)
/dev/pts/0, /dev/pts/1, etc.: Pseudo-terminals (PTYs)

What is a PTY?

A PTY (pseudo-terminal) is a software emulation of a terminal. It consists of two parts:

Master side (PTM): The controlling program (like your terminal emulator or SSH server)

Slave side (PTS): What programs see as their terminal

Terminal Emulator (Master)
           ↕
     PTY Interface
           ↕
    Shell (Slave)

When you type in your terminal emulator:

You type ls and press Enter
Terminal emulator receives the key presses
Sends characters to the PTY master
PTY slave delivers them to the shell
Shell executes ls
Output goes to PTY slave
PTY master receives it
Terminal emulator displays it on screen

Seeing Your PTYs

List all pseudo-terminals in use:

# See all pseudo-terminals
ls -l /dev/pts/

Output:

total 0
crw--w---- 1 user tty  136, 0 Nov 17 10:23 0
crw--w---- 1 user tty  136, 1 Nov 17 10:24 1
crw--w---- 1 user tty  136, 2 Nov 17 10:25 2

Each number represents an open terminal session.

Check which PTY you're currently using:

tty

Output:

/dev/pts/0

See all logged-in users and their terminals:

who

Output:

user     pts/0   2025-06-05 10:23 (192.168.1.100)
user     pts/1   2025-06-05 10:24 (192.168.1.100)
admin    tty1    2025-06-05 09:00

TTY in SSH Sessions

When you SSH into a server, SSH creates a PTY for your session:

# SSH connection
ssh user@server

# Check your TTY
tty

Output:

/dev/pts/3

The SSH server (sshd) acts as the PTY master, and your shell runs connected to the PTY slave. This allows interactive programs like vim, top, and less to work over SSH just like they do locally.

Disabling PTY Allocation

Some commands don't need a TTY:

# No PTY allocated (non-interactive)
ssh user@server 'ls -l'

# Force PTY allocation (interactive)
ssh -t user@server 'ls -l'

The -t flag forces PTY allocation even for commands that don't require it.

Why Programs Care About TTYs

Programs behave differently when connected to a TTY versus when their output is redirected to a file or pipe.

Example: ls with colors

# When connected to a TTY (colors enabled)
ls

# When piped (colors disabled)
ls | cat

The ls command checks if its output is a TTY. If yes, it uses colors. If no (output is piped), it outputs plain text.

Checking if Output is a TTY

Programs use the isatty() function:

import sys

if sys.stdout.isatty():
    print("Running interactively in a terminal")
else:
    print("Output is redirected (pipe or file)")

Run it directly:

python check_tty.py

Output:

Running interactively in a terminal

Pipe the output:

python check_tty.py | cat

Output:

Output is redirected (pipe or file)

Practical Example: Git Output

Git shows colored output when connected to a TTY:

# Colored output
git log

# No colors when piped
git log | less

You can force colors even when not connected to a TTY:

git -c color.ui=always log | less

Docker and TTYs

Docker containers can run with or without a TTY:

# Run without TTY (non-interactive)
docker run ubuntu echo "Hello"

# Run with TTY and interactive mode
docker run -it ubuntu bash

The flags:

-i: Keep STDIN open (interactive)
-t: Allocate a pseudo-TTY

Together -it gives you an interactive shell session.

Without -t, you can't run interactive programs:

# This fails without -t
docker run -i ubuntu vim

# This works
docker run -it ubuntu vim

Checking TTY in Docker

Inside a container:

docker run -it ubuntu bash
tty

Output:

/dev/pts/0

Without -t:

docker run -i ubuntu tty

Output:

not a tty

Terminal Multiplexers: tmux and screen

tmux and screen create PTYs that persist even when you disconnect:

# Start tmux
tmux

# Check your TTY
tty

Output:

/dev/pts/5

Inside tmux, you're connected to a PTY created by tmux. When you detach from tmux (Ctrl+b d), the PTY and all programs running in it continue running.

This is why tmux/screen are useful for long-running tasks over SSH:

SSH → tmux (PTY master) → shell (PTY slave)

If SSH disconnects, tmux keeps running. Reconnect and reattach to the same session.

ps Command

See which TTY a process is using:

ps aux

Output:

USER       PID  TTY      STAT START   TIME COMMAND
user      1234  pts/0    Ss   10:23   0:00 -bash
user      5678  pts/1    Ss+  10:24   0:00 vim file.txt
user      9012  ?        Ss   09:00   0:01 sshd

The TTY column shows:

pts/0, pts/1: Pseudo-terminals
tty1, tty2: Virtual consoles
?: No controlling terminal (daemon processes)

stty Command

Control terminal settings:

# Show all terminal settings
stty -a

# Disable echo (useful for password input)
stty -echo

# Re-enable echo
stty echo

# Set terminal size
stty rows 50 cols 120

Ctrl+C and TTY

When you press Ctrl+C, the TTY driver sends a SIGINT signal to the foreground process group. This is handled by the terminal, not by the program:

User presses Ctrl+C
         ↓
    TTY driver
         ↓
   Sends SIGINT
         ↓
   Program exits

Without a TTY, Ctrl+C doesn't work the same way.

/dev/tty: The Controlling Terminal

/dev/tty is a special device that always refers to the controlling terminal of the current process:

# Send output to your terminal, bypassing pipes
echo "This goes to terminal" > /dev/tty

Even if the script's stdout is redirected, output to /dev/tty appears on your terminal.

Reading from /dev/tty for password prompts:

#!/bin/bash
echo -n "Password: " > /dev/tty
stty -echo
read password < /dev/tty
stty echo
echo
echo "Password received"

This ensures the password prompt appears on the terminal and reads from it, regardless of I/O redirection.

Troubleshooting TTY Issues

"Not a TTY" Errors

Some programs require a TTY:

# Fails
ssh user@server sudo command

Error:

sudo: sorry, you must have a tty to run sudo

Solution: Force TTY allocation:

ssh -t user@server sudo command

Docker "input device is not a TTY"

Error when running Docker without proper flags:

docker run ubuntu bash

Solution: Add -it:

docker run -it ubuntu bash

Terminal Size Problems

If your terminal displays incorrectly after resizing:

# Reset terminal
reset

# Or update terminal size
stty rows $(tput lines) cols $(tput cols)

Detached Terminal Sessions

If a process loses its controlling terminal:

# Find processes without a TTY
ps aux | grep '?'

These are usually daemons or background processes, which is normal. If an interactive program shows ?, it might have lost its terminal connection.

PTY in Programming

Creating a PTY pair in Python:

import pty
import os

# Create a PTY pair
master, slave = pty.openpty()

# Slave side looks like a regular terminal
slave_name = os.ttyname(slave)
print(f"Slave PTY: {slave_name}")

# You can now use master and slave file descriptors
# Master for controlling program
# Slave for programs that need a terminal

This is how terminal emulators, SSH servers, and tools like script work under the hood.

TTY and PTY are fundamental to how Unix systems handle interactive input and output. TTYs originated as physical terminals but now exist as virtual consoles and pseudo-terminals. PTYs enable modern tools like SSH, Docker, and tmux to provide interactive terminal sessions. Understanding these concepts helps you work more effectively with command-line tools and troubleshoot terminal-related issues.

Command to Delete All Pods in All Kubernetes Namespaces

DevOps Daily Team — Sun, 01 Jun 2025 09:00:00 GMT

Introduction

Sometimes, you may need to delete all Pods across all namespaces in your Kubernetes cluster. This operation can be useful for troubleshooting, resetting the cluster state, or cleaning up resources. However, it should be performed with caution, as it can disrupt running applications.

In this guide, you'll learn the command to delete all Pods in all namespaces, along with explanations and best practices.

Prerequisites

Before running the command, ensure the following:

You have kubectl installed and configured to access your Kubernetes cluster.
You have sufficient permissions to delete resources across all namespaces.
You understand the impact of deleting Pods, as it will terminate running workloads.

The Command

To delete all Pods in all namespaces, use the following command:

kubectl delete pods --all --all-namespaces

Explanation

kubectl delete pods: Specifies that you want to delete Pods.
--all: Deletes all Pods in the specified namespace.
--all-namespaces: Extends the scope to all namespaces in the cluster.

Best Practices

Backup Critical Data: Ensure that any critical data or state is backed up before deleting Pods.
Understand Pod Behavior: Pods managed by Deployments or StatefulSets will be recreated automatically. Standalone Pods will not.
Use with Caution: Avoid running this command in production environments unless absolutely necessary.

Example Scenario

Imagine you are troubleshooting a cluster-wide issue and suspect that some Pods are stuck in a bad state. Deleting all Pods can help reset the cluster and allow controllers to recreate healthy Pods.

Conclusion

The kubectl delete pods --all --all-namespaces command is a powerful tool for managing Kubernetes clusters. Use it wisely and always consider the impact on your applications and workloads. By following best practices, you can ensure a smooth and controlled operation.

Terraform Conditionals: Handling Non-Existent Variables

DevOps Daily Team — Sun, 01 Jun 2025 09:00:00 GMT

TLDR

To handle non-existent variables in Terraform, use the lookup function or the ternary conditional operator. These techniques allow you to provide default values or handle undefined variables gracefully.

Terraform configurations often rely on variables to define dynamic values. However, there are cases where a variable might not be defined. This guide will show you how to handle such scenarios using Terraform conditionals.

Why Handle Non-Existent Variables?

Flexibility: Allow configurations to work even when some variables are not defined.
Default Values: Provide sensible defaults for optional variables.
Error Prevention: Avoid runtime errors caused by missing variables.

Using the lookup Function

The lookup function retrieves a value from a map and allows you to specify a default value if the key does not exist.

Example: Using lookup

variable "config" {
  type = map(string)
}

output "example" {
  value = lookup(var.config, "key", "default-value")
}

In this example, if var.config does not contain the key key, the output will use default-value.

Using the Ternary Conditional Operator

The ternary operator allows you to check if a variable is defined and provide a fallback value.

Example: Using the Ternary Operator

variable "optional_var" {
  default = null
}

output "example" {
  value = var.optional_var != null ? var.optional_var : "default-value"
}

In this example, if var.optional_var is not defined, the output will use default-value.

Best Practices

Use Defaults: Define default values for variables whenever possible.
Validate Inputs: Use validation blocks to enforce constraints on variable values.
Document Variables: Clearly document which variables are optional and their default values.

By using these techniques, you can handle non-existent variables in Terraform gracefully, making your configurations more robust and flexible.

How to Clone a Git Repository into a Specific Folder

DevOps Daily Team — Fri, 30 May 2025 09:00:00 GMT

When you clone a repository, Git creates a folder based on the repository name. Sometimes you want a different folder name or need to clone into a specific location. Git lets you specify the target directory when cloning.

TLDR: To clone into a specific folder, add the folder name after the repository URL: git clone repository-url folder-name. To clone into the current directory, use git clone repository-url . (dot). The folder is created if it does not exist.

In this guide, you'll learn how to control where Git clones repositories.

Prerequisites

You'll need Git installed on your system and a repository URL to clone. Basic familiarity with the command line and Git clone command will be helpful.

Basic Clone with Custom Folder

To clone into a specific folder:

# Clone into custom folder
git clone https://github.com/username/repository.git my-project

# Creates folder named 'my-project' instead of 'repository'
cd my-project

The folder name you specify does not need to match the repository name.

Cloning into Current Directory

To clone into the current directory (it must be empty):

# Create and enter directory
mkdir my-project
cd my-project

# Clone into current directory
git clone https://github.com/username/repository.git .

# The dot means "current directory"

This is useful when you already created the folder and want to clone into it.

Cloning with Full Path

You can specify a complete path:

# Clone to absolute path
git clone https://github.com/username/repository.git /home/user/projects/my-app

# Or relative path
git clone https://github.com/username/repository.git ../other-project

# Or with spaces (use quotes)
git clone https://github.com/username/repository.git "My Project Folder"

Cloning into Nested Directories

Git creates parent directories if they do not exist:

# Creates both 'projects' and 'frontend' folders
git clone https://github.com/username/repo.git projects/frontend

cd projects/frontend

All parent directories are created automatically.

Cloning Multiple Branches into Separate Folders

To have different branches in different folders:

# Clone main branch
git clone https://github.com/username/repo.git repo-main

# Clone to another folder for develop branch
git clone -b develop https://github.com/username/repo.git repo-develop

# Clone for feature branch
git clone -b feature-x https://github.com/username/repo.git repo-feature-x

Now you can work on multiple branches simultaneously.

Cloning with Project Structure

For organizing multiple related repositories:

# Create project structure
mkdir -p myproject/{frontend,backend,mobile}

# Clone into structure
git clone https://github.com/company/frontend.git myproject/frontend
git clone https://github.com/company/backend.git myproject/backend
git clone https://github.com/company/mobile.git myproject/mobile

# Result:
# myproject/
#   frontend/
#   backend/
#   mobile/

Handling Existing Folders

If the folder already exists and is not empty:

# Try to clone
git clone https://github.com/username/repo.git existing-folder

# Error: destination path 'existing-folder' already exists and is not an empty directory

# Solutions:
# 1. Use different name
git clone https://github.com/username/repo.git existing-folder-new

# 2. Delete folder first
rm -rf existing-folder
git clone https://github.com/username/repo.git existing-folder

# 3. Clone into it if empty
git clone https://github.com/username/repo.git existing-folder/.

Cloning with SSH vs HTTPS

The folder name syntax works with any protocol:

# HTTPS
git clone https://github.com/username/repo.git my-folder

# SSH
git clone git@github.com:username/repo.git my-folder

# Git protocol
git clone git://github.com/username/repo.git my-folder

# Local path
git clone /path/to/repo.git my-folder

Cloning Bare Repositories

To clone as a bare repository (no working directory):

# Clone bare repository
git clone --bare https://github.com/username/repo.git repo.git

# Typically named with .git extension

Bare repositories are used for central repos or mirrors.

Cloning into Subdirectory of Existing Repo

You cannot clone into a subdirectory of another Git repository:

# This fails
cd existing-repo
git clone https://github.com/username/other-repo.git subdir

# Error: refusing to create alternate object database

# Solution: Use submodules
git submodule add https://github.com/username/other-repo.git subdir

Scripting Multiple Clones

Automate cloning multiple repositories:

#!/bin/bash
# clone-all.sh

repos=(
    "https://github.com/company/frontend:frontend"
    "https://github.com/company/backend:backend"
    "https://github.com/company/api:api"
)

for repo in "${repos[@]}"; do
    url="${repo%:*}"
    folder="${repo#*:}"
    echo "Cloning $url into $folder"
    git clone "$url" "$folder"
done

Cloning with Custom Configuration

Set configuration while cloning:

# Clone with custom origin name
git clone -o upstream https://github.com/username/repo.git my-folder

# Verify
cd my-folder
git remote -v
# upstream  https://github.com/username/repo.git (fetch)

Cloning Specific Branch to Folder

Clone a specific branch into custom folder:

# Clone specific branch
git clone -b develop --single-branch https://github.com/username/repo.git develop-folder

cd develop-folder
git branch
# * develop

Shallow Clone into Folder

Clone with limited history into specific folder:

# Shallow clone
git clone --depth 1 https://github.com/username/repo.git quick-clone

# Only latest commit is cloned
cd quick-clone
git log --oneline
# Shows only 1 commit

Renaming After Clone

If you cloned with the default name and want to rename:

# Clone with default name
git clone https://github.com/username/repository.git

# Rename the folder
mv repository my-project

cd my-project
# Everything still works

Git does not care about the folder name after cloning.

Cloning into Symbolic Link

You can clone through symbolic links:

# Create symbolic link
ln -s /mnt/storage/projects /home/user/dev

# Clone into linked location
git clone https://github.com/username/repo.git /home/user/dev/my-project

# Physically stored at /mnt/storage/projects/my-project

Common Patterns

Workspace organization:

# By company
git clone https://github.com/company/repo.git ~/work/company/repo

# By language
git clone https://github.com/user/js-project.git ~/projects/javascript/js-project

# By status
git clone https://github.com/user/active-project.git ~/active/project

Testing different versions:

# Current stable
git clone -b main https://github.com/user/repo.git repo-stable

# Beta version
git clone -b beta https://github.com/user/repo.git repo-beta

# Development
git clone -b develop https://github.com/user/repo.git repo-dev

Best Practices

Use descriptive folder names:

# Good: Clear what it is
git clone url company-website-frontend

# Less clear: Needs context
git clone url frontend

Follow consistent naming:

# Consistent pattern
git clone url client-portal-web
git clone url client-portal-api
git clone url client-portal-mobile

Organize by project:

mkdir client-portal
git clone url1 client-portal/web
git clone url2 client-portal/api
git clone url3 client-portal/mobile

Use short names for frequent access:

# If you type it often, keep it short
git clone https://github.com/long-company-name/repository-name.git proj

Document in team workflows:

# In README or docs:
# Clone to: git clone  project-frontend
# This matches team convention

Troubleshooting

Error: Folder already exists

# Check if folder exists
ls -la folder-name

# Remove or use different name
rm -rf folder-name
git clone url folder-name

Error: Permission denied

# Check parent directory permissions
ls -ld parent-directory

# Use sudo if needed (be careful)
sudo git clone url /var/www/project

Error: Invalid path

# On Windows, avoid these characters: < > : " / \ | ? *
git clone url valid-folder-name

# Not: my-project/feature
# Use: my-project-feature

Now you know how to clone a Git repository into a specific folder. Simply add the folder name as an argument after the repository URL: git clone url folder-name. This gives you control over your project organization and folder structure.

How do I Resolve Merge Conflicts in a Git Repository?

DevOps Daily Team — Thu, 22 May 2025 11:00:00 GMT

Merge conflicts are a normal part of working with Git in a team environment. They occur when Git cannot automatically determine which changes should take precedence because two branches have modified the same parts of a file in different ways. While they might seem intimidating at first, resolving conflicts is a straightforward process once you understand what Git is showing you.

This guide walks you through identifying, understanding, and resolving merge conflicts so you can confidently merge branches and collaborate with your team.

TLDR

When you encounter a merge conflict, Git marks the conflicted sections in your files with <<<<<<<, =======, and >>>>>>> markers. Open the files, manually choose which changes to keep (or combine both), remove the markers, then run git add and git commit to complete the merge.

Prerequisites

You need a Git repository and basic familiarity with Git commands like commit, merge, and branch. Understanding how branches work will help you grasp why conflicts occur.

Understanding Why Conflicts Happen

Merge conflicts occur when:

Two branches modify the same lines in a file
One branch deletes a file while another modifies it
Two branches create different files with the same name

Here's a common scenario:

main branch:      A -> B -> C -> D
                       \
feature branch:         X -> Y -> Z

If both commit D and commit Z modified line 10 of config.js, Git won't know which version to use when merging. It needs you to decide.

Identifying When You Have a Conflict

When you try to merge branches, Git tells you immediately if there are conflicts:

git merge feature-branch

Output with conflicts:

Auto-merging src/config.js
CONFLICT (content): Merge conflict in src/config.js
Automatic merge failed; fix conflicts and then commit the result.

Check which files have conflicts:

git status

You'll see:

On branch main
You have unmerged paths.
  (fix conflicts and run "git commit")
  (use "git merge --abort" to abort the merge)

Unmerged paths:
  (use "git add ..." to mark resolution)
        both modified:   src/config.js
        both modified:   README.md

no changes added to commit (use "git add" and/or "git commit -a")

The "both modified" label shows files where both branches made changes to the same content.

Understanding Conflict Markers

Open a conflicted file and you'll see special markers showing the conflicting changes:

function initializeApp() {
  const config = {
<<<<<<< HEAD
    apiUrl: 'https://api.production.com',
    timeout: 5000,
    retries: 3
=======
    apiUrl: 'https://api.staging.com',
    timeout: 3000,
    maxRetries: 5
>>>>>>> feature-branch
  };
  return config;
}

Here's what each marker means:

<<<<<<< HEAD: Start of changes from your current branch (usually main)
=======: Divider between the two versions
>>>>>>> feature-branch: End of changes from the branch you're merging

Everything between <<<<<<< HEAD and ======= is what exists in your current branch. Everything between ======= and >>>>>>> feature-branch is what exists in the branch you're merging in.

Resolving Conflicts Manually

To resolve the conflict, you need to:

Decide which version to keep (or combine both)
Remove the conflict markers
Mark the file as resolved

Option 1: Keep Changes from Your Current Branch

If you want to keep your current branch's version:

function initializeApp() {
  const config = {
    apiUrl: 'https://api.production.com',
    timeout: 5000,
    retries: 3
  };
  return config;
}

Remove all the conflict markers and the other branch's changes.

Option 2: Keep Changes from the Incoming Branch

If you want to keep the feature branch's version:

function initializeApp() {
  const config = {
    apiUrl: 'https://api.staging.com',
    timeout: 3000,
    maxRetries: 5
  };
  return config;
}

Option 3: Combine Both Changes

Often the best solution incorporates both sets of changes:

function initializeApp() {
  const config = {
    apiUrl: 'https://api.production.com',
    timeout: 5000,
    retries: 3,
    maxRetries: 5
  };
  return config;
}

In this case, you keep the production API URL and timeout from main, but also add the maxRetries property from the feature branch. The key is understanding what each change does and making an informed decision.

Marking Files as Resolved

After manually editing the file to resolve conflicts:

# Stage the resolved file
git add src/config.js

# Check status
git status

You'll see:

On branch main
All conflicts fixed but you are still merging.
  (use "git commit" to conclude merge)

Changes to be committed:
        modified:   src/config.js

Once all conflicted files are resolved and staged, complete the merge:

git commit

Git will open your editor with a pre-written merge commit message. You can accept it or modify it to include notes about how you resolved the conflicts.

Using Command Line Tools to Choose a Version

If you want to quickly accept all changes from one branch or the other without manual editing:

Keep all changes from your current branch:

# For a specific file
git checkout --ours src/config.js

# For all conflicted files
git checkout --ours .

Keep all changes from the incoming branch:

# For a specific file
git checkout --theirs src/config.js

# For all conflicted files
git checkout --theirs .

After using --ours or --theirs, stage the files:

git add .
git commit

This approach is useful when you know one branch's changes should completely override the other, but use it carefully - you might lose important changes.

Aborting a Merge

If you realize the merge is going wrong or you need to approach it differently:

git merge --abort

This returns your repository to the state before you started the merge. You lose any conflict resolution work you've done, but your branches remain unchanged.

Resolving Conflicts During a Rebase

Conflicts can also occur during a rebase. The process is similar but with different commands:

git rebase main

If conflicts occur:

CONFLICT (content): Merge conflict in src/config.js
error: could not apply abc123... Add new feature

Resolve conflicts the same way (edit files, remove markers), then:

# Stage resolved files
git add src/config.js

# Continue the rebase
git rebase --continue

If you want to abort the rebase:

git rebase --abort

Using Visual Merge Tools

Many developers prefer visual tools for resolving conflicts. Git supports various merge tools:

# See available merge tools
git mergetool --tool-help

# Use a specific tool (example: vimdiff)
git mergetool --tool=vimdiff

# Or just run the configured default tool
git mergetool

Popular merge tools include:

VS Code: Built-in merge conflict resolver
vimdiff: Terminal-based, available everywhere
meld: Graphical tool for Linux
kdiff3: Cross-platform graphical tool
p4merge: Free tool from Perforce

Configure your preferred tool:

# Set VS Code as your merge tool
git config --global merge.tool vscode
git config --global mergetool.vscode.cmd 'code --wait $MERGED'

# Or use meld
git config --global merge.tool meld

Common Conflict Scenarios

Both Branches Added Different Lines

When both branches add content in the same location:

<<<<<<< HEAD
function calculateTotal(items) {
  return items.reduce((sum, item) => sum + item.price, 0);
}
=======
function calculateTotal(items) {
  const total = items.map(i => i.price).reduce((a, b) => a + b, 0);
  return total;
}
>>>>>>> feature-branch

You might combine the best of both:

function calculateTotal(items) {
  // Use reduce for clarity and performance
  return items.reduce((sum, item) => sum + item.price, 0);
}

Conflicting Import Statements

<<<<<<< HEAD
import { UserService } from './services/user';
import { AuthService } from './services/auth';
=======
import { UserService } from './services/users';
import { AuthenticationService } from './services/authentication';
>>>>>>> feature-branch

Determine which module names and paths are correct:

import { UserService } from './services/user';
import { AuthenticationService } from './services/authentication';

Configuration Changes

<<<<<<< HEAD
database:
  host: localhost
  port: 5432
  name: production_db
=======
database:
  host: db.example.com
  port: 5432
  name: app_database
  pool_size: 20
>>>>>>> feature-branch

Merge relevant settings:

database:
  host: localhost
  port: 5432
  name: production_db
  pool_size: 20

Preventing Conflicts

While conflicts are normal, you can reduce their frequency:

Keep branches short-lived: Merge feature branches frequently rather than letting them diverge for weeks.

Pull regularly: Keep your feature branch updated with main:

# While on feature branch
git fetch origin
git merge origin/main

Communicate with your team: If you know someone else is working on the same files, coordinate your changes.

Use smaller commits: Smaller, focused commits are easier to merge than large, sprawling changes.

Refactor carefully: Large refactoring changes often create conflicts. Consider doing them on a dedicated branch that gets merged quickly.

Testing After Resolving Conflicts

After resolving conflicts and completing the merge, always test your code:

# Run your test suite
npm test

# Or whatever your project uses
pytest
go test ./...
cargo test

Conflicts might introduce subtle bugs even when the merge looks correct. The changes you merged might work individually but cause issues when combined.

Viewing Conflict History

To see files that had conflicts in past merges:

# Show merge commits
git log --merges --oneline

# See details of a specific merge
git show 

# See what conflicts were resolved
git log -p -S "<<<<<<" --all

Using Git Rerere for Repeated Conflicts

If you frequently encounter the same conflicts (common when repeatedly merging long-lived branches), enable Git's "reuse recorded resolution" feature:

# Enable rerere
git config --global rerere.enabled true

Now Git remembers how you resolved conflicts and automatically applies the same resolution if the conflict reoccurs. This is helpful during rebases or when syncing branches repeatedly.

Merge conflicts are a natural part of collaborative development. The key is staying calm, understanding what Git is showing you, and making thoughtful decisions about which changes to keep. With practice, resolving conflicts becomes routine rather than stressful, and you'll develop strategies for avoiding conflicts in the first place.

How to Set Git Editor for Commit Messages

DevOps Daily Team — Thu, 22 May 2025 09:00:00 GMT

Git opens a text editor when you run git commit without the -m flag or when you need to edit messages during rebase or merge. By default, Git uses whatever editor is configured in your shell, which might not be the one you prefer.

TLDR: To set your Git editor, use git config --global core.editor "editor-command". For VS Code: git config --global core.editor "code --wait". For Nano: git config --global core.editor "nano". For Vim: git config --global core.editor "vim". The --wait flag is crucial for GUI editors to make Git wait for you to close the file.

In this guide, you'll learn how to configure Git to use any editor for commit messages.

Prerequisites

You'll need Git installed on your system and your preferred editor installed. Basic familiarity with the command line will help you follow along.

Checking Your Current Editor

To see which editor Git is currently using:

# Check configured editor
git config --global core.editor

# If not set, Git uses the default (usually vi/vim)
# To see what Git will actually use:
git var GIT_EDITOR

Setting VS Code as Git Editor

Visual Studio Code is a popular choice:

# Set VS Code as editor
git config --global core.editor "code --wait"

# Test it
git commit
# VS Code opens for your commit message

The --wait flag is critical - it tells VS Code to block until you close the file, so Git knows when you're done editing.

For VS Code Insiders:

git config --global core.editor "code-insiders --wait"

Setting Vim as Git Editor

Vim is the traditional Unix editor:

# Set Vim
git config --global core.editor "vim"

# Or explicitly use vi
git config --global core.editor "vi"

# For nvim (Neovim)
git config --global core.editor "nvim"

Vim is usually the default on Unix systems, so you might not need to set this.

Setting Nano as Git Editor

Nano is beginner-friendly with on-screen help:

# Set Nano
git config --global core.editor "nano"

# Test it
git commit
# Nano opens with helpful shortcuts at bottom

Nano is easier for users who are not comfortable with Vim.

Setting Emacs as Git Editor

For Emacs users:

# Set Emacs
git config --global core.editor "emacs"

# Or for terminal mode
git config --global core.editor "emacs -nw"

# For GUI Emacs that waits
git config --global core.editor "emacsclient -c"

Setting Sublime Text

For Sublime Text:

# macOS
git config --global core.editor "subl -n -w"

# Windows
git config --global core.editor "'C:/Program Files/Sublime Text/sublime_text.exe' -w"

# Linux
git config --global core.editor "subl -n -w"

The -w flag makes Sublime wait, similar to --wait in VS Code.

Setting Atom

For Atom editor:

# Set Atom
git config --global core.editor "atom --wait"

Setting Notepad++ (Windows)

For Notepad++ on Windows:

# Set Notepad++ (adjust path if needed)
git config --global core.editor "'C:/Program Files/Notepad++/notepad++.exe' -multiInst -notabbar -nosession -noPlugin"

Setting TextEdit (macOS)

For macOS TextEdit:

# Set TextEdit
git config --global core.editor "open -e -W"

The -W flag makes the command wait until TextEdit closes.

Setting Nano with Line Numbers

Configure Nano to show line numbers:

# Set Nano with line numbers
git config --global core.editor "nano -l"

# Or with multiple options
git config --global core.editor "nano -l -i"

Flags:

-l shows line numbers
-i enables auto-indent

Using Environment Variables

Git respects shell environment variables:

# Set editor in your shell profile (~/.bashrc, ~/.zshrc, etc.)
export EDITOR=vim
export GIT_EDITOR=code --wait

# Git will use these if core.editor is not set

The priority order is:

GIT_EDITOR environment variable
core.editor Git config
VISUAL environment variable
EDITOR environment variable
Default (usually vi)

Setting Editor for Specific Repository

To set an editor for only one repository:

# Navigate to repository
cd /path/to/repo

# Set editor for this repo only (no --global)
git config core.editor "nano"

# Check it was set
git config core.editor

Setting Editor for Different Operations

You can set different editors for different Git operations:

# Editor for commit messages
git config --global core.editor "code --wait"

# Editor for diffs
git config --global diff.tool "meld"

# Editor for merge conflicts
git config --global merge.tool "kdiff3"

Temporarily Using a Different Editor

Override the configured editor for one command:

# Use nano for this commit only
GIT_EDITOR=nano git commit

# Use vim for this commit
GIT_EDITOR=vim git commit

Common Issues and Solutions

Issue: Editor opens but Git says "Aborting commit"

# Problem: Editor is not waiting
git config --global core.editor "code --wait"
#                                       ^^^^^^
# The --wait flag is crucial

Issue: "error: cannot run code: No such file or directory"

# Problem: Editor is not in PATH
# Solution 1: Add full path
git config --global core.editor "/usr/local/bin/code --wait"

# Solution 2: Add editor to PATH
export PATH="$PATH:/path/to/editor"

Issue: Windows path problems

# Use forward slashes and quotes for spaces
git config --global core.editor "'C:/Program Files/Editor/editor.exe' --wait"

Testing Your Editor Configuration

Test the editor works correctly:

# Make a change
echo "test" >> test.txt
git add test.txt

# Commit without -m (opens editor)
git commit

# If editor opens correctly, write message and save
# Git should complete the commit when you close the editor

Setting Up GUI Editors Properly

For GUI editors, you must include the wait flag:

# Good: Editor waits
git config --global core.editor "code --wait"
git config --global core.editor "subl -w"
git config --global core.editor "atom --wait"

# Bad: Git does not wait, commits abort
git config --global core.editor "code"
git config --global core.editor "subl"

Multiple Git Accounts with Different Editors

Using conditional includes, you can set different editors per project:

# ~/.gitconfig
[includeIf "gitdir:~/work/"]
    path = ~/.gitconfig-work
[includeIf "gitdir:~/personal/"]
    path = ~/.gitconfig-personal

In ~/.gitconfig-work:

[core]
    editor = code --wait

In ~/.gitconfig-personal:

[core]
    editor = nano

Resetting to Default Editor

To remove your custom editor configuration:

# Remove global editor setting
git config --global --unset core.editor

# Check what Git will use now
git var GIT_EDITOR

Git falls back to environment variables or system defaults.

IDE-Specific Configurations

IntelliJ IDEA / WebStorm:

# macOS
git config --global core.editor "idea --wait"

# Windows
git config --global core.editor "'C:/Program Files/JetBrains/IntelliJ IDEA/bin/idea64.exe' --wait"

Eclipse:

# Use built-in Git integration
# Or set external editor
git config --global core.editor "eclipse -w"

Vim Configuration for Git

Customize Vim for Git commit messages:

# In ~/.vimrc
" Git commit message settings
autocmd FileType gitcommit setlocal spell
autocmd FileType gitcommit setlocal textwidth=72
autocmd FileType gitcommit setlocal colorcolumn=51,73

This enables spell check, wraps at 72 characters, and highlights the 50-character subject line limit.

Nano Configuration for Git

Customize Nano for Git:

# In ~/.nanorc
# Git commit message settings
set speller "aspell -x -c"
set tabsize 4
set smooth

Best Practices

Choose an editor you're comfortable with:

# If you're new to command line
git config --global core.editor "nano"

# If you know Vim
git config --global core.editor "vim"

# If you prefer GUI
git config --global core.editor "code --wait"

Make sure the wait flag is set:

# Always include wait for GUI editors
code --wait    # VS Code
subl -w        # Sublime
atom --wait    # Atom

Test before committing:

# Test editor opens correctly
git commit --amend
# Should open editor with last commit message
# Close without changes to abort

Use shell aliases for quick editor switching:

# In ~/.bashrc or ~/.zshrc
alias git-vim='GIT_EDITOR=vim git'
alias git-nano='GIT_EDITOR=nano git'

# Use it
git-nano commit  # Opens Nano for this commit
git-vim rebase -i HEAD~3  # Opens Vim for rebase

Now you know how to set your preferred editor for Git commit messages. The key is using git config --global core.editor "your-editor" with the appropriate wait flag for GUI editors. Choose an editor you're comfortable with to make writing commit messages easier and more efficient.

Why Does the C Preprocessor Interpret "linux" as "1"?

DevOps Daily Team — Thu, 22 May 2025 09:00:00 GMT

You're writing C code and you create a variable named linux, then your code won't compile. Or worse, it compiles but behaves strangely. What's going on?

TL;DR

The C preprocessor on Linux systems automatically defines the macro linux with the value 1. This is a legacy feature from older compilers that predefined system names as macros. Modern code should use standard macros like __linux__ instead. If you need to use linux as an identifier, you can undefine it with #undef linux or use compiler flags to prevent the predefinition.

This is one of those unexpected behaviors that can waste hours of debugging time if you don't know about it. The issue comes from how compilers historically handled platform detection.

Let's say you write this seemingly innocent code:

#include 

int main() {
    int linux = 5;
    printf("linux = %d\n", linux);
    return 0;
}

When you compile it on a Linux system with gcc:

gcc test.c -o test

The preprocessor replaces linux with 1 before compilation, so your code effectively becomes:

#include 

int main() {
    int 1 = 5;  // Syntax error!
    printf("1 = %d\n", 1);
    return 0;
}

This produces a confusing error message like "expected identifier before numeric constant."

Why Does This Happen?

In the early days of Unix and C, compilers predefined macros for the operating system to help with platform-specific code. The idea was that you could write:

#ifdef unix
    // Unix-specific code
#endif

#ifdef linux
    // Linux-specific code
#endif

The compiler would define these symbols automatically, so you could check which platform you were compiling for without needing to pass flags.

This seemed convenient at the time, but it had a major problem: it polluted the global namespace with common words like unix and linux. Any variable, function, or struct member with these names would be replaced by the preprocessor.

The Modern Standard

C standards organizations recognized this was a bad idea. The C standard now specifies that implementation-defined macros should start with an underscore followed by a capital letter (like __linux__ or __unix__).

Modern compilers define proper macros:

#ifdef __linux__
    // Linux-specific code
#endif

#ifdef __unix__
    // Unix-specific code
#endif

#ifdef __APPLE__
    // macOS-specific code
#endif

But for backward compatibility, gcc still defines the old linux and unix macros by default when compiling C code (not C++).

Seeing the Predefined Macros

You can ask gcc to show you all its predefined macros:

# Show all predefined macros
gcc -dM -E - < /dev/null

# Filter for linux-related macros
gcc -dM -E - < /dev/null | grep linux

You'll see output like:

#define __linux 1
#define __linux__ 1
#define linux 1

All three macros are defined with the value 1.

How to Work Around It

If you need to use linux as an identifier (variable name, function name, etc.), you have several options.

The simplest is to undefine the macro at the top of your file:

#undef linux

#include 

int main() {
    int linux = 5;  // Now this works
    printf("linux = %d\n", linux);
    return 0;
}

Put #undef linux before any includes to avoid issues with headers that might use the macro.

Alternatively, compile with the -std=c99 or -std=c11 flag, which disables these non-standard predefined macros:

gcc -std=c99 test.c -o test

Or use -ansi for strict ANSI C compliance:

gcc -ansi test.c -o test

Both approaches prevent gcc from defining linux as a macro.

Real-World Example: The Linux Kernel

Interestingly, even the Linux kernel itself has to deal with this. If you look at kernel headers, you'll find code like:

#undef unix
#undef linux

// ... kernel code

The kernel developers have to undefine these macros to avoid conflicts with their own code.

When This Causes Subtle Bugs

The real danger isn't syntax errors (those are obvious). It's when the code compiles but behaves unexpectedly.

Consider this code:

#include 

struct system_info {
    char *name;
    int linux;  // Meant to be a boolean flag
};

int main() {
    struct system_info info = { "Server01", 0 };

    if (info.linux) {
        printf("Running Linux\n");
    } else {
        printf("Not running Linux\n");
    }

    return 0;
}

After preprocessor expansion, the struct becomes:

struct system_info {
    char *name;
    int 1;  // Syntax error
};

But if you had a different name collision that didn't cause a syntax error, you might get runtime bugs that are hard to track down.

Checking for Platform at Compile Time

If you're writing portable code and need to check the platform, use the modern macros:

#include 

int main() {
    #ifdef __linux__
        printf("Compiled on Linux\n");
    #elif defined(__APPLE__)
        printf("Compiled on macOS\n");
    #elif defined(_WIN32)
        printf("Compiled on Windows\n");
    #else
        printf("Unknown platform\n");
    #endif

    return 0;
}

These macros are standardized and won't interfere with your identifiers.

C++ Doesn't Have This Problem

If you compile C code as C++ (using g++ instead of gcc), the linux macro isn't defined:

# C compiler - defines linux
gcc test.c -o test

# C++ compiler - doesn't define linux
g++ test.c -o test

This is because C++ has stricter namespace rules, and the standards committee decided not to carry over this legacy behavior.

Finding the Problem in Your Code

If you're getting weird errors and suspect the linux macro might be involved, preprocess your code to see what the compiler actually sees:

# Preprocess only, output to stdout
gcc -E test.c

# Preprocess and save to a file
gcc -E test.c -o test.i

Look at the output to see if linux has been replaced with 1.

You can also add a check at the top of your file during debugging:

#ifdef linux
    #warning "linux macro is defined!"
#endif

This will produce a warning during compilation if the macro is defined.

Best Practices

To avoid problems with predefined macros:

Use modern platform detection macros (__linux__, __unix__, etc.) instead of legacy ones
If you must use common words as identifiers, undefine problematic macros at the top of your file
Compile with strict standards flags (-std=c99, -std=c11) to disable non-standard extensions
Check preprocessor output (gcc -E) when debugging weird compilation errors
For library code, prefix your identifiers to avoid collisions (mylib_linux instead of linux)

The linux macro is a historical artifact that modern C programmers need to be aware of. While it made sense in the 1970s, today it's mostly a source of confusion. Understanding why it exists and how to work around it will save you debugging time and help you write more portable code.

How Docker Differs from a Virtual Machine (And Why It Matters)

DevOps Daily Team — Sun, 18 May 2025 09:00:00 GMT

Introduction

If you're new to containerization or just wondering why everyone's talking about Docker, you might be asking: Isn't this just a virtual machine in disguise?

Not quite.

Docker and virtual machines (VMs) both isolate environments to run software, but they do it in very different ways. Understanding how they differ helps you make better infrastructure decisions, especially when performance, portability, and simplicity are on the line.

In this guide, we'll compare Docker and VMs by looking at how they work, how they handle resources, and when to use one over the other.

What You'll Need

No tools required to follow along, just basic knowledge of how applications run on Linux or cloud platforms. If you've used Docker or a VM provider (like VirtualBox or AWS EC2), that's a plus.

How Virtual Machines Work

A virtual machine simulates a full physical machine. It includes:

A hypervisor (like VirtualBox, VMware, or KVM) to run and manage VMs.
A guest OS inside each VM, often a full Linux or Windows distribution.
Apps and dependencies installed inside the guest OS.

Here's a simplified diagram of what a VM stack looks like:

[ Hardware ]
     ↓
[ Host OS ]
     ↓
[ Hypervisor ]
     ↓
[ Guest OS ]
     ↓
[ App + Dependencies ]

Each VM behaves like a full server. That's powerful, but also heavy. Spinning up multiple VMs means duplicating operating systems, drivers, and system services, which eats up memory and CPU.

How Docker Works

Docker containers also isolate applications, but they share the host OS kernel instead of running their own.

Docker relies on:

The Docker Engine, running on the host OS.
Images, which define the container environment.
Containers, which are running instances of those images.

A container doesn't boot a full OS, it starts a process inside a lightweight environment that behaves like a tiny virtual server.

Here's how that looks:

[ Hardware ]
     ↓
[ Host OS ]
     ↓
[ Docker Engine ]
     ↓
[ Container (App + Dependencies) ]

Because containers skip the guest OS, they're faster to start, smaller to ship, and easier to manage.

Key Differences at a Glance

Feature	Virtual Machine	Docker Container
OS Isolation	Full guest OS per VM	Shared host OS kernel
Resource Usage	High (multiple OS overhead)	Low (just processes + libs)
Startup Time	Minutes	Seconds or less
Portability	Lower (OS-specific configs)	High (runs the same everywhere)
Security Boundary	Stronger (hardware emulation)	Weaker (shared kernel risks)
Use Case Fit	Legacy apps, full isolation	Microservices, CI/CD, cloud-native apps

Example: Running PostgreSQL on Both

Let's say you want to run PostgreSQL locally.

With a Virtual Machine

You might:

Spin up a VM using VirtualBox.
Install Ubuntu.
Install PostgreSQL inside it.
Open ports and configure networking.

That's a decent approach if you're mimicking a production server, but it's resource-intensive and takes time to set up.

With Docker

You can run the same database with:

docker run --name dev-postgres \
  -e POSTGRES_PASSWORD=devpass \
  -p 5432:5432 \
  -d postgres:15

This pulls the image, starts the database, and it's ready in seconds.

You get an isolated PostgreSQL environment without booting a whole OS, great for local development or testing.

When to Use Docker vs. VMs

Here's a general rule of thumb:

Use Docker for lightweight, fast, and scalable environments, especially for CI/CD, microservices, and local development.
Use VMs when you need full OS-level isolation, stricter security boundaries, or when running apps that aren't container-friendly.

There's also a middle ground: many teams run containers inside virtual machines for the best of both worlds. For example, running Docker on an EC2 VM or Kubernetes nodes managed via VMs.

Final Thoughts

Docker and virtual machines both help you isolate and manage software, but they solve different problems.

Containers shine when speed and portability matter. VMs are better when you need full-system flexibility and stricter isolation.

Choosing between them isn't about which one is better, it's about using the right tool for the job.

Happy coding!

How to Clone a Specific Git Branch

DevOps Daily Team — Mon, 12 May 2025 08:00:00 GMT

When cloning a repository, Git downloads all branches by default, even though you only see the default branch checked out. For large repositories with many branches, this wastes time and disk space. If you only need to work with a specific branch, you can clone just that branch.

TLDR: To clone a specific branch, use git clone -b branch-name --single-branch repository-url. This downloads only the specified branch and its history. For shallow clones with limited history, add --depth 1 to the command.

In this guide, you'll learn how to clone specific branches efficiently and when to use different cloning strategies.

Prerequisites

You'll need Git installed on your system and the URL of the repository you want to clone. Basic familiarity with Git branches and the clone command will help you understand the options.

Cloning a Specific Branch

To clone a specific branch directly:

# Clone only the develop branch
git clone -b develop --single-branch https://github.com/username/repository.git

# Clone only a feature branch
git clone -b feature-auth --single-branch https://github.com/username/repository.git

The -b flag specifies which branch to clone, and --single-branch tells Git to only fetch that branch's history. After cloning, you're checked out to the specified branch with only that branch in your local repository.

Verify what you cloned:

cd repository

# Check current branch
git branch

# Output:
# * develop

# See all branches including remote
git branch -a

# Output:
# * develop
#   remotes/origin/develop

Notice that only the specified branch appears, not other branches from the repository.

Shallow Clone of a Specific Branch

For even faster cloning, combine --single-branch with --depth to limit history:

# Clone with only the latest commit
git clone -b main --single-branch --depth 1 https://github.com/username/repository.git

# Clone with the last 10 commits
git clone -b develop --depth 10 --single-branch https://github.com/username/repository.git

This approach downloads only the specified number of recent commits, dramatically reducing clone time and disk usage for large repositories.

Shallow clones are perfect for CI/CD pipelines or when you only need to build or test the latest code.

Cloning and Later Fetching Other Branches

If you clone with --single-branch but later need other branches, you can fetch them:

# Clone only main branch
git clone -b main --single-branch https://github.com/username/repository.git
cd repository

# Later, configure to fetch all branches
git config remote.origin.fetch "+refs/heads/*:refs/remotes/origin/*"

# Fetch all branches
git fetch origin

# Check out another branch
git checkout develop

This gives you the initial speed benefit of single-branch cloning while maintaining flexibility for future work.

Cloning vs. Checking Out After Clone

An alternative approach is to clone normally, then immediately switch to your desired branch:

# Clone the repository (gets all branches)
git clone https://github.com/username/repository.git
cd repository

# Switch to the branch you want
git checkout feature-auth

This method makes sense when:

You'll likely need multiple branches
The repository is not large
You want full access to all branches from the start

Compare the approaches:

# Method 1: Clone specific branch (faster, less disk space)
git clone -b feature-auth --single-branch https://github.com/username/repo.git

# Method 2: Clone all, then checkout (more complete, more flexible)
git clone https://github.com/username/repo.git && cd repo && git checkout feature-auth

Using Clone with a Custom Directory Name

Specify a directory name when cloning:

# Clone into custom directory
git clone -b develop --single-branch https://github.com/username/repository.git my-project

cd my-project

This is useful when you want to clone multiple branches into separate directories:

# Clone main branch to one directory
git clone -b main --single-branch https://github.com/username/repo.git repo-main

# Clone develop branch to another directory
git clone -b develop --single-branch https://github.com/username/repo.git repo-develop

Now you can work on both branches simultaneously without switching.

Cloning Non-Default Branches

When cloning a specific branch that is not the default, the syntax is the same:

# Clone a feature branch
git clone -b feature/new-ui --single-branch https://github.com/username/repository.git

# Clone a release branch
git clone -b release/v2.0 --single-branch https://github.com/username/repository.git

Git does not distinguish between default and non-default branches - any branch can be cloned directly.

Converting a Shallow Clone to Full Clone

If you cloned with --depth and later need the full history:

# Fetch all history for current branch
git fetch --unshallow

# Now you have complete history
git log --oneline

The --unshallow flag converts your shallow clone to a full clone by downloading all missing commits.

Adding More Branches to Single-Branch Clone

After cloning with --single-branch, fetch specific additional branches:

# Clone only main
git clone -b main --single-branch https://github.com/username/repository.git
cd repository

# Fetch a specific branch
git fetch origin develop:develop

# Check out the newly fetched branch
git checkout develop

This syntax (origin develop:develop) fetches the remote develop branch and creates a local branch with the same name.

Cloning Tags

You can also clone specific tags:

# Clone at a specific tag
git clone -b v1.0.0 --single-branch https://github.com/username/repository.git

# Verify you're at the tag
git describe --tags

This is useful when you need to work with a specific release version.

Using Git Clone for CI/CD

In CI/CD pipelines, cloning specific branches with limited depth speeds up builds:

# Fast clone for CI/CD
git clone -b $BRANCH_NAME --single-branch --depth 1 https://github.com/username/repo.git

# Example in GitHub Actions workflow
- uses: actions/checkout@v3
  with:
    ref: develop
    fetch-depth: 1

Most CI/CD systems provide optimized checkout actions that handle this automatically, but understanding the underlying Git commands helps when debugging or creating custom workflows.

Checking Clone Configuration

After cloning, verify your repository configuration:

# See remote configuration
git remote -v

# Check fetch configuration
git config --get remote.origin.fetch

# If single-branch was used, output is:
# +refs/heads/main:refs/remotes/origin/main

# For normal clones, output is:
# +refs/heads/*:refs/remotes/origin/*

This shows whether you have a single-branch or multi-branch clone.

Handling Protected Branches

When cloning protected branches (like main or master), you might need authentication:

# Clone with HTTPS (prompts for credentials)
git clone -b main --single-branch https://github.com/username/private-repo.git

# Clone with SSH (uses SSH key)
git clone -b main --single-branch git@github.com:username/private-repo.git

SSH cloning is faster and more secure for repositories you frequently access, as it uses SSH keys instead of passwords.

Cloning Specific Branches from Multiple Repositories

When working with microservices or multiple related repositories:

#!/bin/bash

# Clone specific branch from multiple repos
repos=(
  "frontend"
  "backend"
  "api-gateway"
)

for repo in "${repos[@]}"; do
  git clone -b develop --single-branch \
    "https://github.com/company/$repo.git" \
    "$repo-develop"
done

This script clones the develop branch from multiple repositories into separate directories.

Sparse Checkout for Partial Clones

For even more granular control, combine specific branch cloning with sparse checkout:

# Clone specific branch
git clone -b main --single-branch --depth 1 https://github.com/username/repo.git
cd repo

# Enable sparse checkout
git sparse-checkout init --cone

# Only checkout specific directories
git sparse-checkout set src/app tests

This downloads only the specified branch and only checks out specific directories, perfect for monorepos where you only need a subset of the code.

Now you know how to clone specific Git branches efficiently. Using -b with --single-branch gives you fast, focused clones perfect for single-purpose work, CI/CD pipelines, or when working with large repositories where you only need one branch.

How to Copy Docker Images Between Hosts Without a Repository

DevOps Daily Team — Sun, 04 May 2025 10:00:00 GMT

While using container registries like Docker Hub, Harbor, or AWS ECR is the standard way to distribute Docker images, sometimes you need to transfer images directly between hosts. This might be necessary in air-gapped environments, when bandwidth is limited, or when working with sensitive images that shouldn't be pushed to external services. This guide explains several methods to copy Docker images between machines without using a registry.

Prerequisites

Before you begin, make sure you have:

Docker installed on both source and destination hosts
SSH access between hosts (for some methods)
Sufficient disk space on both machines
Administrator/root access or appropriate Docker permissions

Method 1: Using docker save and docker load

The most straightforward approach uses Docker's built-in save and load commands.

Step 1: Save the image to a tar file on the source host

# Save a single image
docker save my-image:latest > my-image.tar

# Save multiple images to the same file
docker save my-image:latest my-other-image:1.0 > my-images.tar

# Optional: compress the tar file (smaller but slower)
docker save my-image:latest | gzip > my-image.tar.gz

The docker save command preserves all image layers, tags, and history.

Step 2: Transfer the tar file to the destination host

You can use any file transfer method:

# Using scp
scp my-image.tar user@destination-host:/tmp/

# Using rsync (more efficient for large files)
rsync -avP my-image.tar user@destination-host:/tmp/

Step 3: Load the image on the destination host

# Load from an uncompressed tar file
docker load < /tmp/my-image.tar

# If you compressed the file with gzip
gunzip -c /tmp/my-image.tar.gz | docker load

After loading, verify the image is available:

docker images

Method 2: Direct Transfer via SSH Pipe

For a more efficient transfer without using intermediate storage, you can pipe the image directly through SSH:

# On the source host, pipe directly to the destination
docker save my-image:latest | ssh user@destination-host 'docker load'

This method:

Avoids storing the tar file on either host
Transfers the image in a single operation
Requires stable SSH connectivity

For large images, you can add compression:

# With compression
docker save my-image:latest | gzip | ssh user@destination-host 'gunzip | docker load'

Method 3: Using docker export and docker import

An alternative approach is to use docker export and docker import, which works with containers rather than images:

Step 1: Create a container from the image (if not already running)

docker create --name temp-container my-image:latest

Step 2: Export the container filesystem

docker export temp-container > my-container.tar

Step 3: Transfer the tar file to the destination host (as in Method 1)

Step 4: Import the container on the destination

cat my-container.tar | docker import - my-image:latest

Important differences from save/load:

export/import flattens the image to a single layer
Image history and layer information is lost
Some metadata like environment variables, working directory, and exposed ports needs to be reconfigured

Method 4: Using External Storage

For completely disconnected hosts or very large images, using external storage media might be the best option:

Step 1: Save the image to external storage

# Save directly to an external drive
docker save my-image:latest > /mnt/usb-drive/my-image.tar

# Split large images into smaller chunks
docker save my-image:latest | split -b 4G - /mnt/usb-drive/my-image.tar.part-

Step 2: Transport the external storage to the destination host

Step 3: Load the image from external storage

# Load a standard tar file
docker load < /mnt/usb-drive/my-image.tar

# Reassemble and load split files
cat /mnt/usb-drive/my-image.tar.part-* | docker load

Performance Optimization

When transferring large images, consider these optimizations:

Check image size before transfer

docker images --format "{{.Repository}}:{{.Tag}} {{.Size}}" | grep my-image

Use compression selectively

Compression makes the transfer file smaller but requires CPU time:

# For better network efficiency (smaller file)
docker save my-image:latest | gzip | ssh user@destination-host 'gunzip | docker load'

# For faster local processing (no compression)
docker save my-image:latest | ssh user@destination-host 'docker load'

Remove unnecessary images first

Clean up both hosts to ensure adequate space:

docker system prune -a

Practical Examples

Example 1: Transferring a Database Image

# On source host
docker save postgres:14 | gzip > postgres.tar.gz

# Transfer the file
scp postgres.tar.gz user@db-server:/tmp/

# On destination host
gunzip -c /tmp/postgres.tar.gz | docker load

Example 2: Moving a Custom Application Image

# On source host
docker save mycompany/app:v1.2.3 | ssh user@production-server 'docker load'

# Verify on destination
ssh user@production-server 'docker images mycompany/app'

# On source host
docker save myapp:latest myapp-db:latest myapp-cache:latest > myapp-bundle.tar

# Transfer and load
rsync -avP myapp-bundle.tar user@destination:/tmp/
ssh user@destination 'docker load < /tmp/myapp-bundle.tar'

Troubleshooting Common Issues

Insufficient Disk Space

When saving large images:

no space left on device

Solution: Use a different partition or streaming method:

# Check available space
df -h

# Stream directly without saving to disk
docker save my-image | ssh user@destination 'docker load'

Corrupted Transfers

If you get image verification errors:

open /var/lib/docker/.../layer.tar: no such file or directory

Solution: Ensure the transfer is complete and try again with integrity checking:

# On source, compute checksum
sha256sum my-image.tar > my-image.tar.sha256

# Transfer both files
scp my-image.tar my-image.tar.sha256 user@destination:/tmp/

# On destination, verify
cd /tmp && sha256sum -c my-image.tar.sha256

Permission Issues

permission denied

Solution: Check permissions and use sudo when necessary:

# Ensure proper permissions
sudo chown $(whoami) my-image.tar
chmod 644 my-image.tar

# Or use sudo for the operation
sudo docker load < my-image.tar

Best Practices

Tag Images Properly Before Transfer

Ensure your images have clear tags before saving:

docker tag myimage:latest myimage:v1.2.3
docker save myimage:v1.2.3 > myimage-v1.2.3.tar

Document Image Dependencies

For complex applications, document all required images:

# Create a manifest file
docker images --format "{{.Repository}}:{{.Tag}}" | grep myapp > image-manifest.txt

# Save all listed images
cat image-manifest.txt | xargs docker save > myapp-complete.tar

For applications with multiple components, use docker-compose to ensure consistency:

# On source, save all images referenced in docker-compose.yml
docker-compose config --services | xargs docker-compose pull
docker-compose config --services | xargs docker-compose images --format "{{.Repository}}:{{.Tag}}" | sort -u | xargs docker save > application-bundle.tar

Next Steps

Now that you know how to transfer Docker images between hosts without a registry, you might want to explore:

Setting up a private Docker registry for more permanent solutions
Creating image optimization strategies to reduce transfer sizes
Automating image transfers using scripts or CI/CD pipelines
Implementing proper image versioning and tagging strategies

Happy containerizing!

Why SCTP Is Not Widely Used Despite Its Technical Advantages

DevOps Daily Team — Sat, 03 May 2025 11:00:00 GMT

TLDR: SCTP (Stream Control Transmission Protocol) offers technical improvements over TCP - multi-streaming, multi-homing, and message boundaries - but remains niche due to poor NAT traversal, limited OS support, lack of programming language libraries, and the massive installed base of TCP/UDP infrastructure. Most applications that need SCTP's features work around TCP's limitations instead of adopting a new protocol.

SCTP is standardized, technically sound, and solves real problems that TCP and UDP have. Yet if you look at internet traffic, SCTP barely registers. It's primarily used in telecom (SS7 signaling) and some specialized applications. Here's why a protocol with clear advantages hasn't gained wider adoption.

What SCTP Offers

SCTP was designed to combine the best features of TCP and UDP while adding new capabilities:

Multi-Streaming

TCP has head-of-line blocking - if packet 5 is lost, packets 6-10 must wait even if they arrived successfully. SCTP allows multiple independent streams within one association:

TCP:
Stream: [1][2][3][X][5][6][7]
        └─────┘    └─ Waiting for packet 4

SCTP:
Stream 1: [1][2][3][X][5][6][7]  <- Blocked waiting for packet 4
Stream 2: [1][2][3][4][5][6][7]  <- Continues independently
Stream 3: [1][2][3][4][5][6][7]  <- Not affected

For applications like video conferencing (audio + video + data channels), this is valuable. If a video frame is lost, audio can continue without delay.

Multi-Homing

SCTP can bind to multiple IP addresses simultaneously. If one path fails, it automatically switches to another:

Client                          Server
IP1: 192.168.1.10              IP1: 10.0.0.5
IP2: 10.50.20.15               IP2: 172.16.0.10

Normal path: 192.168.1.10 ←→ 10.0.0.5
Failover:    10.50.20.15  ←→ 172.16.0.10

# If primary path fails, SCTP automatically uses backup

This built-in redundancy is perfect for high-availability systems, but you can achieve similar results with TCP and load balancers.

Message Boundaries

UDP preserves message boundaries but is unreliable. TCP is reliable but treats data as a byte stream. SCTP gives you both:

# TCP: No message boundaries
send("Hello")
send("World")
# Receiver might get: "HelloWorld" or "Hel" + "loWorld" or any split

# UDP: Message boundaries preserved but unreliable
send("Hello")  # Might arrive
send("World")  # Might be lost

# SCTP: Message boundaries + reliability
send("Hello")  # Arrives as "Hello"
send("World")  # Arrives as "World" or is retransmitted until it does

This eliminates the need for framing protocols on top of TCP.

Built-in Security

SCTP includes features to prevent SYN flood attacks and provides better protection against connection hijacking compared to TCP.

Why It's Not Widely Adopted

Despite these features, SCTP faces significant barriers:

NAT Traversal Problems

Network Address Translation (NAT) is everywhere - home routers, corporate firewalls, cloud load balancers. NAT devices are designed for TCP and UDP, and many don't understand SCTP:

Client (behind NAT)      NAT Router         Server
     |                       |                 |
     |--SCTP INIT----------->|                 |
     |                       X (dropped)       |

NAT doesn't know how to:
- Track SCTP connections
- Map SCTP ports correctly
- Handle multi-homing
- Process SCTP checksums

SCTP packets often get dropped by middleboxes that don't recognize protocol number 132 (SCTP's IP protocol number). TCP is protocol 6, UDP is 17 - these are hardcoded into countless devices.

Some firewalls explicitly block unknown protocols:

# Typical firewall default rules
iptables -A INPUT -p tcp -j ACCEPT_CHAIN
iptables -A INPUT -p udp -j ACCEPT_CHAIN
iptables -A INPUT -p icmp -j ACCEPT_CHAIN
iptables -A INPUT -j DROP  # Drops SCTP and other protocols

Limited Operating System Support

While SCTP is in the Linux kernel and available on FreeBSD, support elsewhere is poor:

Operating System    Native SCTP Support
------------------  -------------------
Linux              Yes (since 2.6)
FreeBSD            Yes
Windows            No (third-party libraries exist)
macOS              No (removed in recent versions)
iOS/Android        No native support

On Windows, you need third-party libraries or user-space implementations, which defeats the performance benefits. macOS had SCTP support but removed it, signaling Apple's lack of interest.

Lack of Language and Framework Support

Most programming languages don't have first-class SCTP support:

# Python - TCP is built-in
import socket
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)  # Easy

# Python - SCTP requires external library
# pip install pysctp
import sctp
sock = sctp.sctpsocket_tcp(socket.AF_INET)  # Less obvious

# JavaScript/Node.js - No SCTP support at all
# Go - No standard library SCTP support
# Rust - Third-party crates only

Compare this to TCP, which has excellent support everywhere:

// Node.js TCP
const net = require('net');
const server = net.createServer((socket) => {
    socket.write('Hello\n');
});
server.listen(8080);

// Node.js SCTP - doesn't exist in standard library

No Browser Support

Web browsers only support TCP (for HTTP/WebSocket) and UDP (for WebRTC). There's no way to use SCTP from browser JavaScript:

// These work:
fetch('https://example.com')           // TCP/TLS
new WebSocket('wss://example.com')      // TCP/TLS
new RTCPeerConnection()                 // UDP (WebRTC)

// This doesn't exist:
new SCTPConnection()  // No such API

This kills SCTP for any web-based application, which is a huge portion of modern software.

Ecosystem and Tooling Gaps

The developer ecosystem around TCP/UDP is massive. SCTP has almost nothing:

# TCP debugging tools
tcpdump -i eth0 'tcp port 80'     # Packet capture
netstat -an | grep ESTABLISHED     # Connection monitoring
ss -t                              # Modern socket stats
wireshark                          # GUI packet analysis

# SCTP debugging
tcpdump -i eth0 'sctp'            # Works but limited analysis
# Most tools don't parse SCTP details well

Load balancers, monitoring tools, and network management software are built for TCP/UDP. Adding SCTP support requires significant engineering effort that most vendors don't prioritize.

Application Layer Workarounds

Instead of adopting SCTP, applications work around TCP's limitations:

Head-of-line blocking? Open multiple TCP connections:

# Instead of SCTP multi-streaming, use multiple TCP connections
connections = [
    create_tcp_connection(server, port) for _ in range(3)
]

# Send different data types on different connections
connections[0].send(audio_data)   # Audio stream
connections[1].send(video_data)   # Video stream
connections[2].send(control_data) # Control messages

This is what HTTP/2 and HTTP/3 do - they multiplex streams over a single TCP connection (HTTP/2) or use QUIC over UDP (HTTP/3).

Multi-homing? Use DNS failover or load balancers:

# DNS-based failover
server.example.com:
  - 10.0.0.5   (primary)
  - 10.0.0.6   (backup)

# Client retries on failure - simulates multi-homing

Message boundaries? Add framing:

import struct

def send_message(sock, data):
    """Send length-prefixed message over TCP."""
    length = len(data)
    sock.sendall(struct.pack('!I', length))  # 4-byte length
    sock.sendall(data)

def recv_message(sock):
    """Receive length-prefixed message from TCP."""
    length_bytes = sock.recv(4)
    length = struct.unpack('!I', length_bytes)[0]

    data = b''
    while len(data) < length:
        chunk = sock.recv(length - len(data))
        data += chunk

    return data

These workarounds are well-understood, documented, and battle-tested. Why learn a new protocol when you can use familiar patterns?

Where SCTP Is Actually Used

SCTP isn't dead - it's used in specific niches:

Telecom Signaling

SS7 (Signaling System 7) over IP uses SCTP for reliability and multi-homing:

Phone Network Signaling:
Cell Tower ←→ [SCTP] ←→ Core Network ←→ [SCTP] ←→ Other Networks

SCTP provides:
- Reliable delivery of signaling messages
- Fast failover between redundant paths
- In-order delivery per stream

Telecom companies control their entire network stack and don't deal with consumer NAT devices, so SCTP works well here.

WebRTC Data Channels

WebRTC uses SCTP for data channels, but it's encapsulated inside UDP (via DTLS):

WebRTC Stack:
Application Data
    ↓
SCTP (provides reliability, streams, message boundaries)
    ↓
DTLS (encryption)
    ↓
UDP (traverses NAT)
    ↓
Network

This "SCTP over UDP" tunneling solves the NAT traversal problem but adds complexity.

Diameter Protocol

The Diameter protocol (used in mobile networks for authentication and billing) can use SCTP for transport, taking advantage of multi-homing and failover.

Could SCTP Still Succeed?

For SCTP to gain wider adoption, it would need:

Universal NAT support - Every home router and corporate firewall would need SCTP-aware NAT. This would take decades.
Browser support - Chrome, Firefox, Safari would need to expose SCTP APIs to JavaScript. Unlikely given the focus on HTTP/3 and QUIC.
Language support - Python, JavaScript, Go, Rust would need standard library SCTP support. Possible but requires champions.
Cloud provider support - AWS, Azure, GCP would need to support SCTP in load balancers and security groups. Low priority for them.

The reality is that QUIC (Quick UDP Internet Connections) is solving many of the same problems SCTP addressed, but it's doing so over UDP to avoid NAT issues:

QUIC approach:
- Built on UDP (NAT-friendly)
- Implements reliability in user-space
- Adds multi-streaming
- Adds encryption by default
- Backed by Google/IETF

Result: HTTP/3 uses QUIC, not SCTP

QUIC shows that the industry prefers innovating on top of UDP rather than deploying new IP protocols.

Practical Advice

If you're considering SCTP for a project:

Use SCTP if:

You control the entire network (data center to data center)
You're in telecom/carrier space
You need multi-homing and your network supports it
You're working with existing SCTP infrastructure

Don't use SCTP if:

Your application needs to work across the public internet
You need browser support
You have to traverse NATs
You want wide language/framework support
You need rich tooling and debugging support

For most applications, stick with TCP or UDP and use application-layer solutions for the features you need. SCTP is technically excellent but practically difficult in a world built for TCP and UDP.

How to Access a Docker Container's Shell

DevOps Daily Team — Sat, 03 May 2025 10:00:00 GMT

Accessing a shell inside a Docker container is essential for debugging, configuration changes, or installing additional tools. Whether you need to check logs, modify files, or troubleshoot issues, getting terminal access lets you interact with your containerized applications directly. This guide shows you multiple ways to access a container's shell for different scenarios.

Prerequisites

Before you begin, make sure you have:

Docker installed on your system
At least one Docker container running (or an image you can run)
Basic knowledge of shell commands

Method 1: Using docker exec for Running Containers

The most common way to get shell access to a running container is with the docker exec command.

Basic Syntax

docker exec -it

The -it flags are important:

-i keeps STDIN open (interactive)
-t allocates a pseudo-TTY (terminal)

Getting a Bash Shell

For most Linux-based containers that include bash:

docker exec -it my-container bash

This drops you into a bash shell inside the running container, where you can run commands as if you were in a regular Linux environment.

Using sh for Minimal Containers

Many lightweight containers (like Alpine-based images) don't include bash but do have sh:

docker exec -it my-alpine-container sh

One-off Commands Without a Shell

You don't always need a full shell. For quick checks, specify the command directly:

# Check container's environment variables
docker exec -it my-container env

# View a specific file
docker exec -it my-container cat /etc/nginx/nginx.conf

Method 2: Starting a New Container with Shell Access

If the container isn't running yet, you can start it with shell access directly.

Run a Container with Interactive Shell

docker run -it

For example:

# Start a Ubuntu container with bash
docker run -it ubuntu bash

# Start an Alpine container with sh
docker run -it alpine sh

Run and Remove Containers for Exploration

When you just want to explore an image, use the --rm flag to automatically remove the container on exit:

docker run --rm -it nginx:alpine sh

This is useful for quick exploration without accumulating stopped containers.

Method 3: Accessing Running Containers for Different Users

By default, docker exec runs commands as the container's default user. To specify a different user:

# Access as root
docker exec -it -u root my-container bash

# Access as a specific user by ID
docker exec -it -u 1000 my-container bash

This is particularly useful for containers that run as non-root by default.

Practical Examples

Example 1: Debugging a Web Server

If your NGINX container isn't serving content correctly:

# Access the container
docker exec -it my-nginx bash

# Check configuration
cat /etc/nginx/conf.d/default.conf

# Check logs
tail -f /var/log/nginx/error.log

# Test configuration
nginx -t

Example 2: Fixing Database Permissions

For a PostgreSQL container with permission issues:

# Access as postgres user
docker exec -it -u postgres my-db bash

# Enter psql to check permissions
psql

# Or as root to fix system permissions
docker exec -it -u root my-db bash
chown -R postgres:postgres /var/lib/postgresql/data

Example 3: Installing Tools in a Container

When you need to add debugging tools to a running container:

docker exec -it my-container bash

# On Debian/Ubuntu based containers
apt-get update
apt-get install -y procps net-tools curl

# On Alpine based containers
apk add --no-cache procps net-tools curl

Remember that changes made this way don't persist when the container is removed. For permanent changes, update your Dockerfile.

Working with Different Container Types

Alpine Containers

Alpine Linux keeps things minimal. It doesn't include bash by default:

# This might fail
docker exec -it alpine-container bash

# Use sh instead
docker exec -it alpine-container sh

# Or install bash first
docker exec -it alpine-container apk add --no-cache bash
docker exec -it alpine-container bash

Windows Containers

For Windows containers, use PowerShell:

docker exec -it windows-container powershell

Distroless Containers

"Distroless" images don't contain a shell at all. For these containers:

Use docker cp to copy files in/out
Add a debug version with the same app but including a shell
Use a sidecar container for debugging

Common Issues and Solutions

No Shell Available

OCI runtime exec failed: exec failed: container_linux.go:380: starting container process caused: exec: "bash": executable file not found in $PATH: unknown

Solution: Try a different shell:

docker exec -it my-container sh

TTY Problems

the input device is not a TTY

Solution: You might be running in a non-interactive environment. Remove the -t flag:

docker exec -i my-container sh

Permission Denied

Error response from daemon: unable to find user myuser

Solution: Check if the user exists in the container or use a numeric user ID instead:

docker exec -it -u 0 my-container bash  # User ID 0 is root

Best Practices

Security Considerations

Avoid running as root unless necessary
Don't leave debugging tools in production containers
Consider read-only containers and only mount specific volumes as writable

Remember Container Ephemerality

Changes you make inside a container this way will be lost when the container is removed. For permanent changes:

Update your Dockerfile and rebuild
Use config volumes for configuration files
Use a docker-compose.override.yml for development-specific changes

Next Steps

Now that you can access your container shells, you might want to:

Learn about container orchestration with Docker Compose or Kubernetes
Set up proper logging with volumes or log drivers
Create more optimized Docker images with multi-stage builds
Implement container health checks for better reliability

Happy containerizing!

Network Usage Top/Htop on Linux

DevOps Daily Team — Sat, 03 May 2025 10:00:00 GMT

While top and htop show CPU and memory usage, they don't display network bandwidth consumption. When you need to see which processes are using network bandwidth, which connections are active, or how much data is flowing through your network interfaces, Linux offers several specialized tools that provide real-time network monitoring similar to how top works for system resources.

This guide covers the best tools for monitoring network usage on Linux, from simple interface statistics to process-level bandwidth tracking.

TLDR

Use nethogs to see bandwidth usage per process (like htop for network). Use iftop to see bandwidth usage per connection. Use nload for a simple interface bandwidth graph. Use bmon for multi-interface monitoring with graphs. Install with your package manager: sudo apt install nethogs iftop nload bmon.

Prerequisites

You need Linux with root access to install monitoring tools and capture network traffic. Basic command-line familiarity helps you interpret the output.

iftop: Monitor Bandwidth by Connection

iftop shows network usage between your system and remote hosts, similar to how top shows process usage.

Installation

# Debian/Ubuntu
sudo apt-get install iftop

# Red Hat/CentOS/Fedora
sudo yum install iftop

# Arch Linux
sudo pacman -S iftop

Basic Usage

Run with sudo (required for packet capture):

sudo iftop

Output shows connections in real-time:

                 12.5Kb        25.0Kb       37.5Kb        50.0Kb
└────────────────┴─────────────┴────────────────┴─────────────
192.168.1.100  => api.example.com       5.12Kb  3.24Kb  2.18Kb
              <=                        15.2Kb  12.8Kb  10.4Kb

192.168.1.100  => cdn.cloudflare.com    1.05Mb  980Kb   750Kb
              <=                        125Kb   98.5Kb  87.2Kb

The arrows show traffic direction:

=> outgoing traffic
<= incoming traffic

The three columns show bandwidth over 2, 10, and 40 second averages.

Useful Options

# Monitor specific interface
sudo iftop -i eth0

# Show port numbers instead of service names
sudo iftop -n

# Don't resolve hostnames (faster)
sudo iftop -N

# Show cumulative bandwidth totals
sudo iftop -t

# Filter by network
sudo iftop -F 192.168.1.0/24

Interactive Commands

While iftop is running, press:

t: Toggle display mode (one line per connection vs two)
n: Toggle name resolution
p: Show port numbers
P: Pause display
q: Quit
h: Help

nethogs: Monitor Bandwidth by Process

nethogs shows which processes are using bandwidth, grouping traffic by application.

Installation

# Debian/Ubuntu
sudo apt-get install nethogs

# Red Hat/CentOS
sudo yum install nethogs

# Arch Linux
sudo pacman -S nethogs

Basic Usage

sudo nethogs

Output:

NetHogs version 0.8.6

  PID USER     PROGRAM              DEV        SENT      RECEIVED
 2341 user     /usr/bin/firefox     eth0      125.45     1540.2  KB/sec
 5678 user     /usr/bin/spotify     eth0       45.12      102.5  KB/sec
 8901 user     sshd: user@pts/0     eth0        2.15        5.43 KB/sec
 1234 root     /usr/sbin/apache2    eth0        0.85      12.34  KB/sec

  TOTAL                                        173.57     1660.47 KB/sec

This shows exactly which programs are using bandwidth and how much.

Useful Options

# Monitor specific interface
sudo nethogs eth0

# Set refresh rate (in seconds)
sudo nethogs -d 1

# Trace mode (show total since start)
sudo nethogs -t

# Monitor multiple interfaces
sudo nethogs eth0 wlan0

nload: Visual Interface Bandwidth Monitor

nload provides a simple visual graph of incoming and outgoing bandwidth.

Installation

# Debian/Ubuntu
sudo apt-get install nload

# Red Hat/CentOS
sudo yum install nload

# Arch Linux
sudo pacman -S nload

Basic Usage

nload

Output shows ASCII graphs:

Device eth0 [192.168.1.100] (1/2):
================================================================================
Incoming:




                                                    Curr: 1.05 MBit/s
                                                    Avg: 850.23 kBit/s
                                                    Min: 125.45 kBit/s
                                                    Max: 2.15 MBit/s
                                                    Ttl: 1.25 GB
Outgoing:




                                                    Curr: 250.45 kBit/s
                                                    Avg: 180.12 kBit/s
                                                    Min: 15.23 kBit/s
                                                    Max: 512.34 kBit/s
                                                    Ttl: 456.78 MB

Useful Options

# Monitor specific interface
nload eth0

# Monitor multiple interfaces (switch with arrow keys)
nload eth0 wlan0

# Set refresh interval in milliseconds
nload -t 500

# Set unit (default is adaptive)
nload -u M  # Show in MBit/s

Interactive Keys

← →: Switch between interfaces
↑ ↓: Adjust graph scale
q: Quit

bmon: Bandwidth Monitor with Multiple Interfaces

bmon provides detailed statistics with nice graphical output for multiple interfaces simultaneously.

Installation

# Debian/Ubuntu
sudo apt-get install bmon

# Red Hat/CentOS
sudo yum install bmon

# Arch Linux
sudo pacman -S bmon

Basic Usage

bmon

Output shows interfaces with graphs:

 #   Interface                RX Rate         RX #     TX Rate         TX #
─────────────────────────────────────────────────────────────────────────────
 0   eth0                      1.05MBit        145      250KBit          89
                           (RX Bandwidth Graph)
     ▃▄▅▆▇█▇▆▅▄▃▂▁▁▂▃▄▅▆▇▇▆▅▄▃
                           (TX Bandwidth Graph)
     ▁▁▂▂▃▃▄▄▅▅▆▆▇▇▆▆▅▅▄▄▃▃▂▂

 1   wlan0                      45KBit          12       15KBit           8

Useful Options

# Show bits per second instead of bytes
bmon -b

# Set output mode
bmon -o ascii        # ASCII graphs
bmon -o curses       # Enhanced curses interface (default)

# Monitor specific interfaces
bmon -p eth0,wlan0

# Set update interval
bmon -r 1000         # Update every 1000ms

iptraf-ng: Interactive IP LAN Monitor

iptraf-ng provides detailed statistics about IP traffic, including protocol breakdowns and TCP connection monitoring.

Installation

# Debian/Ubuntu
sudo apt-get install iptraf-ng

# Red Hat/CentOS
sudo yum install iptraf-ng

Basic Usage

sudo iptraf-ng

This opens a menu-driven interface:

┌─────────────────────────────────────────────┐
│ IP traffic monitor                          │
│ General interface statistics                │
│ Detailed interface statistics               │
│ Statistical breakdowns...                   │
│   By packet size                            │
│   By TCP/UDP service                        │
│ LAN station monitor                         │
│ Filters...                                  │
│ Configure...                                │
│ Exit                                        │
└─────────────────────────────────────────────┘

Select "IP traffic monitor" and choose your interface to see real-time connection details.

vnstat: Long-Term Bandwidth Statistics

vnstat tracks network bandwidth over time (hours, days, months) rather than showing real-time data.

Installation and Setup

# Install vnstat
sudo apt-get install vnstat

# Initialize database for interface
sudo vnstat -i eth0 --create

# Start vnstat daemon
sudo systemctl start vnstat
sudo systemctl enable vnstat

vnstat runs in the background, collecting statistics.

Viewing Statistics

# Show summary
vnstat

# Show hourly stats
vnstat -h

# Show daily stats
vnstat -d

# Show monthly stats
vnstat -m

# Live bandwidth monitor
vnstat -l

Example output:

 Database updated: 2025-05-03 10:15:23

   eth0 since 2025-01-01

          rx:  125.45 GiB      tx:  45.23 GiB      total:  170.68 GiB

                     rx      |     tx      |    total    |   avg. rate
     ------------------------+-------------+-------------+---------------
       yesterday    1.25 GiB |   450 MiB   |    1.68 GiB |   195.23 kbit/s
           today     450 MiB |   125 MiB   |     575 MiB |   125.45 kbit/s
     ------------------------+-------------+-------------+---------------
     estimated      1.12 GiB |   320 MiB   |    1.42 GiB |

Combining Tools for Complete Monitoring

Use different tools for different needs:

Which process is using bandwidth?

sudo nethogs

Which remote host am I connected to?

sudo iftop

What's my total interface bandwidth?

nload

Long-term bandwidth trends?

vnstat -d

Creating Simple Bandwidth Alerts

Monitor bandwidth and alert if it exceeds a threshold:

#!/bin/bash
# bandwidth_alert.sh

INTERFACE="eth0"
THRESHOLD_MB=100

# Get current bandwidth (MB received in last second)
RX1=$(cat /sys/class/net/$INTERFACE/statistics/rx_bytes)
sleep 1
RX2=$(cat /sys/class/net/$INTERFACE/statistics/rx_bytes)

# Calculate bandwidth in MB/s
BANDWIDTH=$(( ($RX2 - $RX1) / 1024 / 1024 ))

if [ $BANDWIDTH -gt $THRESHOLD_MB ]; then
    echo "High bandwidth detected: ${BANDWIDTH}MB/s"
    # Send alert (email, Slack, etc.)
fi

Run this periodically with cron:

# Run every minute
* * * * * /usr/local/bin/bandwidth_alert.sh

Bandwidth Monitoring in Scripts

Get interface statistics programmatically:

# Read bytes received/transmitted
cat /sys/class/net/eth0/statistics/rx_bytes
cat /sys/class/net/eth0/statistics/tx_bytes

# Or use ip command
ip -s link show eth0

Example monitoring script:

#!/bin/bash

INTERFACE="eth0"

while true; do
    RX1=$(cat /sys/class/net/$INTERFACE/statistics/rx_bytes)
    TX1=$(cat /sys/class/net/$INTERFACE/statistics/tx_bytes)

    sleep 1

    RX2=$(cat /sys/class/net/$INTERFACE/statistics/rx_bytes)
    TX2=$(cat /sys/class/net/$INTERFACE/statistics/tx_bytes)

    RX_RATE=$(( ($RX2 - $RX1) / 1024 ))
    TX_RATE=$(( ($TX2 - $TX1) / 1024 ))

    echo "RX: ${RX_RATE}KB/s  TX: ${TX_RATE}KB/s"
done

Troubleshooting Network Issues

Finding Bandwidth Hogs

# See which process is using most bandwidth
sudo nethogs | head -20

# See which remote hosts are consuming bandwidth
sudo iftop -n

Identifying Unusual Traffic

# Monitor for unexpected connections
sudo iftop -F your-network/24

# Watch for high packet rates
sudo bmon

Checking Interface Errors

# See error counts
ip -s link show eth0

# Look for errors, drops, or overruns

Output:

eth0: 
    RX: bytes  packets  errors  dropped overrun mcast
    125G       89M      0       45      0       12K
    TX: bytes  packets  errors  dropped carrier collsns
    45G        67M      0       0       0       0

Errors or dropped packets indicate network problems.

Quick Reference

Tool	Best For	Installation
nethogs	Per-process bandwidth	`apt install nethogs`
iftop	Per-connection bandwidth	`apt install iftop`
nload	Simple interface graphs	`apt install nload`
bmon	Multi-interface monitoring	`apt install bmon`
iptraf-ng	Detailed IP statistics	`apt install iptraf-ng`
vnstat	Historical bandwidth data	`apt install vnstat`

Linux provides rich tools for monitoring network bandwidth at different levels - from per-process usage with nethogs to interface-level statistics with nload. Choose the right tool based on whether you need to identify which program is consuming bandwidth, monitor interface throughput, or track long-term usage trends. These tools are essential for troubleshooting network performance issues and understanding your system's network activity.

Docker: How to Use Bash with an Alpine Based Docker Image

DevOps Daily Team — Sat, 03 May 2025 09:00:00 GMT

TLDR

Alpine Linux images use sh (BusyBox shell) by default, not bash. To use bash, install it with apk add --no-cache bash in your Dockerfile or at runtime. Then you can run scripts or open an interactive bash shell as needed.

Why Alpine Uses sh Instead of bash

Alpine is designed to be lightweight and secure, so it ships with BusyBox's sh shell by default. This keeps images small, but means bash isn't available unless you add it yourself.

How to Install bash in an Alpine Container

Add this line to your Dockerfile:

RUN apk add --no-cache bash

Example Dockerfile:

FROM alpine:3.20
RUN apk add --no-cache bash
CMD ["bash"]

Or, for a one-off interactive session:

docker run -it alpine:3.20 sh
# Inside the container:
apk add --no-cache bash
bash

Running Scripts with bash

If your script uses bash features (like arrays or advanced syntax), make sure to:

Install bash as above
Use #!/bin/bash as the shebang in your script
Run the script with bash script.sh or make it executable

Best Practices

Only install bash if you need it—stick with sh for simple scripts to keep images small
For multi-stage builds, install bash only in build stages if possible
Document the shell requirements for your scripts

Troubleshooting

If you see bash: not found, install it with apk add --no-cache bash
If your script fails with syntax errors, check the shebang and shell compatibility

Conclusion

To use bash in Alpine-based Docker images, just install it with apk and use it as needed. This gives you access to familiar bash features while keeping your images lean and efficient.

Understanding the Difference Between CMD and ENTRYPOINT in Dockerfiles

DevOps Daily Team — Fri, 02 May 2025 10:00:00 GMT

Understanding the difference between CMD and ENTRYPOINT instructions in a Dockerfile is crucial for creating flexible, usable Docker images. These instructions determine what process runs inside your container and how users can interact with it. This guide explains their differences, how they work together, and when to use each one.

Prerequisites

Before we begin, make sure you have:

Docker installed on your system
Basic familiarity with Dockerfile syntax
Some experience running Docker containers

Basic Definitions

First, let's clarify what each instruction does:

CMD: Specifies the default command to run when starting a container
ENTRYPOINT: Configures the container to run as an executable

While they might seem to serve similar purposes, they behave quite differently and are often used together.

CMD: Providing Default Commands

The CMD instruction defines the default command and/or parameters that will be executed when the container runs without specifying a command.

CMD Syntax

Docker supports three forms of CMD syntax:

# Exec form (preferred)
CMD ["executable", "param1", "param2"]

# As default parameters to ENTRYPOINT
CMD ["param1", "param2"]

# Shell form
CMD command param1 param2

The exec form is preferred because it starts the specified command directly, not wrapped in a shell, which allows for proper signal handling and process ID tracking.

Basic CMD Example

FROM ubuntu:22.04

# Set a default command to run
CMD ["echo", "Hello from the container!"]

When you run this container without specifying a command, it will output "Hello from the container!":

docker run my-image
# Output: Hello from the container!

Overriding CMD

The main characteristic of CMD is that it can be easily overridden. Any arguments you provide when running the container replace the CMD instruction entirely:

docker run my-image echo "Different message"
# Output: Different message

In this example, the original CMD is completely ignored and replaced with echo "Different message".

ENTRYPOINT: Making Containers Behave Like Executables

The ENTRYPOINT instruction configures a container to run as an executable. It's used when you want your container to behave like a command-line tool.

ENTRYPOINT Syntax

Like CMD, ENTRYPOINT has two forms:

# Exec form (preferred)
ENTRYPOINT ["executable", "param1", "param2"]

# Shell form
ENTRYPOINT command param1 param2

Again, the exec form is recommended for most use cases.

Basic ENTRYPOINT Example

FROM ubuntu:22.04

# Set the container's main executable
ENTRYPOINT ["echo", "Hello from"]

When you run this container, the ENTRYPOINT command runs:

docker run my-image
# Output: Hello from

Appending Commands to ENTRYPOINT

Unlike CMD, arguments passed when running the container are appended to the ENTRYPOINT command:

docker run my-image Docker World
# Output: Hello from Docker World

The words "Docker World" are added as arguments to the echo command defined in the ENTRYPOINT.

Overriding ENTRYPOINT

If you need to override the ENTRYPOINT defined in the Dockerfile, you can use the --entrypoint flag:

docker run --entrypoint ls my-image -la
# Output: (directory listing)

This completely replaces the ENTRYPOINT with the ls command and passes -la as its arguments.

Combining CMD and ENTRYPOINT

The real power comes when you use CMD and ENTRYPOINT together. The general pattern is:

ENTRYPOINT: defines the command that always runs
CMD: provides default arguments that can be overridden

How They Work Together

FROM ubuntu:22.04

ENTRYPOINT ["echo", "Hello"]
CMD ["World"]

When you run this container:

docker run my-image
# Output: Hello World

If you provide command-line arguments, they replace only the CMD part:

docker run my-image Docker Universe
# Output: Hello Docker Universe

This makes your containers more flexible while maintaining their core behavior.

Practical Use Cases

Use Case 1: CLI Tools

If you're containerizing a command-line tool, use ENTRYPOINT to make the container behave like that tool:

FROM python:3.11-slim

WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .

ENTRYPOINT ["python", "my-cli-tool.py"]
CMD ["--help"]

Users can run this container as if it were the CLI tool itself:

docker run my-cli-tool --input file.txt --output results.txt

Use Case 2: Configurable Services

When building service containers that need configuration options:

FROM node:18-alpine

WORKDIR /app
COPY package*.json ./
RUN npm ci --only=production
COPY . .

ENTRYPOINT ["node", "server.js"]
CMD ["--port", "8080"]

Users can run the service with default settings or override them:

# Run with default port
docker run my-service

# Run with custom port
docker run my-service --port 3000

Use Case 3: Initialization Scripts

For containers that need initialization logic:

FROM postgres:14

COPY ./init-db.sh /docker-entrypoint-initdb.d/
ENTRYPOINT ["docker-entrypoint.sh"]
CMD ["postgres"]

The initialization script will always run because it's part of the image's entrypoint logic, but users can change how the database runs.

Shell Form vs. Exec Form: Important Differences

When using the shell form (CMD command param1 param2), Docker wraps your command in /bin/sh -c, which has several implications:

Process Handling: Your application runs as a subprocess of /bin/sh, not as PID 1
Signal Handling: SIGTERM and other signals go to the shell, not your application
Environment Variables: Shell form allows for environment variable expansion

# Shell form with variable expansion
CMD echo "Hello, $NAME"

# Exec form doesn't expand variables the same way
CMD ["echo", "Hello, $NAME"]  # $NAME will be treated literally

For proper signal handling and to avoid unnecessary processes, the exec form is generally recommended.

Best Practices

1. Use ENTRYPOINT for the Main Command

# Good: Fixed entrypoint with configurable parameters
ENTRYPOINT ["nginx"]
CMD ["-g", "daemon off;"]

2. Make Images Work Out of the Box

# Default command makes container usable immediately
FROM python:3.11
COPY app.py /app/
WORKDIR /app
ENTRYPOINT ["python"]
CMD ["app.py"]

3. Prefer Exec Form Over Shell Form

# Preferred: Exec form
ENTRYPOINT ["node", "server.js"]

# Avoid: Shell form
ENTRYPOINT node server.js

4. Include Health Checks for Services

FROM nginx:alpine
COPY nginx.conf /etc/nginx/nginx.conf
HEALTHCHECK --interval=30s --timeout=3s \
  CMD curl -f http://localhost/ || exit 1
ENTRYPOINT ["nginx"]
CMD ["-g", "daemon off;"]

Troubleshooting Common Issues

Problem: Container Exits Immediately

This often happens with the shell form of ENTRYPOINT or CMD when the specified command completes quickly:

# This will exit immediately
CMD echo "Hello World"

Solution: For long-running services, ensure they stay in the foreground:

# This will keep the container running
CMD ["nginx", "-g", "daemon off;"]

Problem: Commands Not Found

docker run my-image
# /bin/sh: 1: my-command: not found

Solution: Check paths and ensure commands are available in the container:

# Use absolute paths when necessary
ENTRYPOINT ["/usr/local/bin/my-command"]

# Or ensure your PATH is set properly
ENV PATH="/app/bin:${PATH}"

Problem: Arguments Not Working as Expected

docker run my-image --config=/etc/myapp.conf
# Error: Unknown option: --config=/etc/myapp.conf

Solution: Ensure your ENTRYPOINT script properly handles arguments:

ENTRYPOINT ["./entrypoint.sh"]

With entrypoint.sh:

#!/bin/bash
set -e

# Handle specific arguments or pass them through
exec myapp "$@"

Next Steps

Now that you understand how CMD and ENTRYPOINT work, you might want to:

Review your existing Dockerfiles to apply these best practices
Create more user-friendly container interfaces
Explore Docker's init systems for better process management
Learn about multi-stage builds to optimize your Docker images

Happy containerizing!

How to Start a Stopped Docker Container with a Different Command

DevOps Daily Team — Fri, 02 May 2025 09:00:00 GMT

TLDR

You can't directly change the command of a stopped Docker container when restarting it with docker start. Instead, create a new container from the same image with a different command or entrypoint. Use docker run --entrypoint or docker commit for advanced cases.

Why Can't You Change the Command with docker start?

When you create a container with docker run, the command and entrypoint are set at creation. docker start always uses the original command—you can't override it for an existing container.

Option 1: Create a New Container with a Different Command

The most reliable way is to start a new container from the same image, specifying the new command:

docker run -it my-image /bin/bash

Or, to override the entrypoint:

docker run -it --entrypoint /bin/sh my-image

This gives you a fresh container with your chosen command.

Option 2: Use docker commit to Save Changes (Advanced)

If you need to preserve changes made in the stopped container (like files or installed packages), you can commit it to a new image:

docker commit my-container my-temp-image

Then run a new container from that image with a different command:

docker run -it my-temp-image /bin/bash

Option 3: Use docker exec for Running Containers

If the container is running, you can start a new process inside it:

docker exec -it my-container /bin/bash

But this doesn't work if the container is stopped.

Best Practices

Use docker run with a new command for most cases.
Use docker commit only if you need to preserve changes from the stopped container.
For debugging, override the entrypoint or command as needed.
Clean up old containers and images to avoid clutter.

Conclusion

You can't change the command of a stopped Docker container when restarting it, but you can create a new container from the same image (or a committed image) with any command you need. This gives you flexibility for debugging, recovery, or running alternate workflows.

How to Copy Files from Docker Containers to the Host Machine

DevOps Daily Team — Thu, 01 May 2025 10:00:00 GMT

When working with Docker containers, you'll often need to retrieve files generated or modified inside them. Whether you're extracting application logs, database dumps, or processed data, Docker provides several ways to copy files from containers to your host machine. This guide explores the different approaches and helps you choose the right one for your use case.

Prerequisites

Before proceeding, make sure you have:

Docker installed on your system
Basic familiarity with Docker commands and concepts
Terminal or command prompt access on your host machine

Method 1: Using docker cp Command

The docker cp command is the simplest and most direct way to copy files from a container to your host machine.

Basic Syntax

docker cp :

Copying a Single File

To copy a specific file from a container to your current directory:

# Copy a configuration file from a container
docker cp my-nginx:/etc/nginx/nginx.conf ./nginx.conf

This command copies the nginx.conf file from the container named "my-nginx" to your current directory.

Copying a Directory

To copy an entire directory:

# Copy the logs directory from a container to the host
docker cp my-app:/var/log/app ./container-logs

This recursively copies all contents from the container's /var/log/app directory to a local container-logs directory.

Copying from a Stopped Container

One advantage of docker cp is that it works with stopped containers too:

# Copy data from a stopped container
docker cp crashed-app:/var/log/app/error.log ./crash-report.log

Method 2: Using Volumes for Real-time Access

While not strictly a copying method, Docker volumes provide direct access to container files from the host, which is often more convenient for ongoing development.

Creating a Volume Mount

# Run a container with a volume mount
docker run -d --name db-container -v $(pwd)/data:/var/lib/postgresql/data postgres:14

With this approach, anything written to /var/lib/postgresql/data inside the container is automatically available in the ./data directory on your host.

Accessing Files Through Existing Volumes

If you already have a running container with volumes, you can find the volume mount information:

# Inspect container volumes
docker inspect -f '{{ .Mounts }}' my-container

Then you can access the files directly through the volume mount path on your host system.

Method 3: Using docker export for Complete Filesystem

If you need to extract the entire filesystem from a container:

# Export the entire container filesystem to a tar archive
docker export my-container > container-filesystem.tar

# Extract specific files from the archive
tar -xf container-filesystem.tar path/to/file

# Or extract everything
mkdir container-extract
tar -xf container-filesystem.tar -C container-extract

This approach is useful for:

Creating backups of container state
Analyzing the complete filesystem
Migrating container data between hosts

Method 4: Using docker exec with tar

For more control over the extraction process, you can combine docker exec with the tar command:

# Create a tar archive inside the container and extract it on the host
docker exec my-container tar -cz -C /path/to/directory files | tar -xz -C ./destination

This method streams the files directly without needing temporary storage in the container, which is efficient for large files or systems with limited container disk space.

Example: Extracting Log Files

# Extract all log files from a container's /var/log directory
docker exec web-app tar -cz -C /var/log . | tar -xz -C ./logs

Method 5: Using docker run to Create One-off Copies

For containers based on images you control, you can create a one-off container specifically to copy files:

# Run a temporary container to copy data and remove it afterward
docker run --rm -v $(pwd):/target my-image cp -r /data/output /target

This pattern is useful in CI/CD pipelines where you need to extract build artifacts.

Practical Examples

Example 1: Extracting a Database Backup

# Create a PostgreSQL dump and copy it to the host
docker exec postgres-db pg_dump -U postgres mydatabase > mydatabase.sql

Example 2: Copying Built Application Artifacts

# Copy build artifacts from a Node.js container
docker cp node-builder:/app/dist ./dist

Example 3: Extracting Generated Reports

# Extract all PDF reports from a container
mkdir -p ./reports
docker exec report-generator tar -cz -C /app/reports . | tar -xz -C ./reports

Performance Considerations

When copying large files or directories, consider these performance tips:

Use compression when appropriate: The tar method with the -z flag compresses data during transfer, which can be faster for text-heavy files but slower for already-compressed content.
Consider bandwidth limitations: If you're copying across a network, large transfers might impact other services.
Avoid copying unnecessary files: Be specific about what you copy rather than grabbing entire directories.
Use volumes for frequent access: If you need ongoing access to files, volumes are more efficient than repeated copy operations.

Troubleshooting Common Issues

Permission Problems

If you encounter permission issues when copying files:

# Copy files and preserve permissions
docker cp --archive my-container:/path/to/files ./destination

The --archive flag preserves ownership, permissions, and timestamps.

Container Path Not Found

Make sure you're using the correct path inside the container:

# Check the exact path by exploring the container filesystem
docker exec my-container ls -la /path/to/check

File Ownership

Files copied from containers often have different ownership than expected:

# Change ownership after copying
sudo chown -R $(id -u):$(id -g) ./copied-files

Next Steps

Now that you know how to copy files from Docker containers, you might want to explore:

Setting up automated backup systems for container data
Creating data processing pipelines that extract container results
Implementing proper volume management for persistent container data
Using Docker Compose to simplify complex data sharing scenarios

Happy containerizing!

How to Get docker-compose to Always Re-create Containers from Fresh Images

DevOps Daily Team — Thu, 01 May 2025 09:00:00 GMT

TLDR

To make docker-compose always re-create containers from fresh images, use docker-compose pull to get the latest images, then run docker-compose up --force-recreate --build. This ensures containers are rebuilt or replaced, not reused from cache.

Why Containers Might Not Be Re-created

By default, docker-compose up reuses existing containers if they already exist, even if the image has changed. This can lead to running old code or missing updates, especially in development or CI.

The Solution: Use the Right Flags

--force-recreate: Forces docker-compose to remove and re-create containers, even if nothing changed in the config.
--build: Builds images before starting containers (useful if you have local Dockerfiles).
--pull: Always attempt to pull newer images before building (Compose V2 only, or use docker-compose pull first).

Recommended workflow:

docker-compose pull           # Get the latest images from the registry
docker-compose up --build --force-recreate

Or, with Compose V2 (Docker CLI):

docker compose up --build --force-recreate --pull always

Example

Suppose you updated your app image in a registry. To make sure your containers use the new image:

docker-compose pull app
# Then:
docker-compose up --build --force-recreate app

This will:

Pull the latest image for app
Build any local images if needed
Remove and re-create the app container

Cleaning Up Old Containers and Images

If you want to remove stopped containers and unused images:

docker-compose down --rmi all --volumes

--rmi all: Remove all images used by services
--volumes: Remove named volumes declared in the volumes section

Best Practices

Use --force-recreate in development to avoid stale containers
Use --pull or docker-compose pull to always get the latest images
For CI/CD, script these steps to ensure clean deploys
Document your workflow for your team

Conclusion

To always get fresh containers from the latest images, combine docker-compose pull with docker-compose up --build --force-recreate. This keeps your environment up to date and avoids surprises from cached or outdated containers.

Terraform: Failed to install provider, does not match checksums from dependency lock file

DevOps Daily Team — Thu, 01 May 2025 09:00:00 GMT

TLDR

The error "Failed to install provider, does not match checksums from dependency lock file" means Terraform downloaded a provider binary whose checksum did not match what's recorded in .terraform.lock.hcl. Common causes are an outdated lock file, platform differences, corrupted downloads, or using a provider mirror with different checksums. The safe fixes are: inspect and refresh the lock file, clear local caches, regenerate platform-specific checksums, or use short-lived remote sandboxes when testing. Follow the step-by-step checklist below to resolve the issue without weakening supply-chain protections.

When Terraform complains that a provider's checksum does not match the dependency lock file, it is protecting you from a potentially tampered or unexpected provider binary. That protection is useful, but it can also block legitimate changes like a provider upgrade or when you switch platforms. Use the following steps to diagnose and fix the problem.

Quick checklist (start here)

Confirm which provider and platform Terraform reports in the error message.
Run terraform init -upgrade to refresh provider versions and update the lock file if you intend to accept newer checksums.
If you must reproduce a specific lock state, restore the original .terraform.lock.hcl from version control or recreate it with terraform providers lock for the target platforms.
Clear local plugin caches and re-run terraform init.

Work through those in order - the first two items resolve most cases.

Why this happens

The .terraform.lock.hcl file pins provider download checksums for specific platforms. If the provider binary downloaded during terraform init has a different checksum, Terraform aborts to prevent unexpected code from executing.
Common legitimate causes:
- The lock file is stale and a provider was released or patched upstream.
- You switched OS or architecture (for example, from macOS to Linux) and the lock file does not contain checksums for the new platform.
- A download was corrupted or a proxy / mirror returned a different binary.
- A private provider mirror or alternate registry with different signed binaries.

Step-by-step fixes

Inspect the error and .terraform.lock.hcl

Before the code: check the exact provider and platform referenced in the error and open your lock file to compare.

# show the last init error output (run this in the root where terraform init failed)
terraform init

What this does: repeats init and prints the failing provider and platform details.
Why it matters: you need the provider name, version, and platform that triggered the checksum mismatch.

Open .terraform.lock.hcl and find the provider block for the provider name and version reported in the error. That block contains checksums keyed by platform.

Refresh the lock file when you want the latest provider

Before the code: if you are intentionally updating providers or accepting the registry's latest artifacts, run init with upgrade to refresh the lock file.

# refresh provider constraints and update lock file
terraform init -upgrade

What this does: asks Terraform to check for newer provider versions and update .terraform.lock.hcl with new checksums.
Why it matters: this is the correct action when you intend to move to a newer provider release.

Regenerate lock checksums for specific platforms

Before the code: when developing on one platform but CI or other machines use different OS/arch combinations, generate a lock file that includes checksums for all target platforms.

# generate lock entries for linux_amd64 and darwin_amd64
terraform providers lock -platform=linux_amd64 -platform=darwin_amd64

What this does: creates or updates .terraform.lock.hcl to include checksums for the requested platforms.
Why it matters: Terraform needs matching checksums for every platform that will download the provider. This prevents mismatches in CI or teammates' machines.

Clear corrupted local downloads and re-init

Before the code: if a download was corrupted or a proxy returned a bad file, remove the local plugin cache and re-run init.

# remove the working terraform directory and provider cache
rm -rf .terraform
rm -rf ~/.terraform.d/plugin-cache
# then re-init
terraform init

What this does: forces Terraform to re-download providers from the registry or mirror.
Why it matters: corrupted cached files can trigger checksum mismatches even when the lock file is correct.

If you use a provider mirror or private registry

Verify the mirror's integrity: the mirror must publish the same signed artifacts and checksums as the public registry, otherwise checksums will differ.
If the mirror alters binaries, coordinate with the mirror owner to publish matching checksums or pin the mirror in your CLI config using provider_installation in ~/.terraformrc or terraform.rc.

Example provider_installation that uses a filesystem mirror:

# ~/.terraformrc
provider_installation {
  filesystem_mirror {
    path    = "/opt/terraform-providers"
  }
  direct {
    exclude = ["registry.terraform.io/*/*"]
  }
}

What this does: tells Terraform to prefer providers from the local filesystem mirror.
Why it matters: a mirror with different binaries will cause checksum mismatches unless the lock file has matching entries.

When it is OK to remove the lock file

If you are experimenting locally and understand the risk, you can regenerate the lock file. Prefer terraform init -upgrade or terraform providers lock instead of just deleting the lock file. If you do remove it, commit the regenerated .terraform.lock.hcl to version control so other users have the same checksums.

# risky: remove the lock file and re-init (only for experiments)
rm .terraform.lock.hcl
terraform init

What this does: removes protection and allows Terraform to create a fresh lock file based on whatever it downloads.
Why it matters: avoid this on shared branches unless you coordinate the provider change.

Best practices to avoid this in the future

Commit .terraform.lock.hcl to version control for all Terraform roots. That keeps everyone using the same provider checksums.
Use terraform providers lock -platform= to include CI and local platforms in the lock file.
Pin provider versions in your configuration to avoid surprise upgrades: version = "~> 4.0".
Use short-lived sandbox backends for testing provider upgrades before changing main state.
If you operate a provider mirror, keep it synchronized with the upstream registry and publish matching checksums.

Troubleshooting flow (ASCII)

The steps below show a simple flow to resolve checksum issues.

Error: checksum mismatch
    |
    v
Inspect error -> check .terraform.lock.hcl ->
  if outdated -> terraform init -upgrade -> done
  else if missing platforms -> terraform providers lock -platform=... -> done
  else -> clear cache (rm -rf .terraform) -> terraform init

Short practical conclusion

When Terraform blocks provider installation with a checksum mismatch, treat it as a safety signal. Start by refreshing the lock file with terraform init -upgrade if you expect newer providers, or generate platform checksums with terraform providers lock when CI and local machines differ. Only clear caches or remove the lock file when you understand the risk and have a plan to commit and distribute the updated lock file.

Next steps you can explore: automate lockfile generation for CI platforms, add pre-commit checks that validate .terraform.lock.hcl, and use provider mirrors carefully with matching checksum publication.

Difference Between Running and Starting a Docker Container

DevOps Daily Team — Wed, 30 Apr 2025 09:00:00 GMT

TLDR

docker run creates and starts a new container from an image, while docker start restarts an existing, stopped container. Use run for new containers, and start to bring back a container you previously stopped.

What Does docker run Do?

docker run is the main command to create and launch a new container. It does several things at once:

Creates a new container from the specified image
Assigns it a unique ID and (optionally) a name
Sets up networking, storage, and environment variables
Starts the container's main process

Example:

docker run -d --name my-nginx -p 8080:80 nginx

This creates a new container named my-nginx from the nginx image and starts it in the background.

What Does docker start Do?

docker start is used to restart a container that was previously created (and stopped). It does not create a new container or change its configuration.

Example:

docker stop my-nginx  # Stop the container
# ... do something ...
docker start my-nginx # Start it again

The container keeps its data, configuration, and name.
Any changes made to the container's filesystem persist.

Key Differences

docker run creates a new container every time you use it.
docker start only works on containers that already exist (but are stopped).
You can only use docker run with an image; docker start uses a container name or ID.
docker run lets you set options (ports, env vars, volumes) at creation; docker start does not.

When to Use Each Command

Use docker run when you want a fresh container, possibly with new options.
Use docker start to restart a stopped container with the same settings and data.
For stateless or short-lived containers, docker run is common.
For persistent services or debugging, docker start is handy.

Best Practices

Name your containers with --name for easier management.
Use docker ps -a to see all containers (running and stopped).
Remove containers you no longer need with docker rm to avoid clutter.

Conclusion

docker run and docker start serve different purposes: one creates and starts new containers, the other restarts existing ones. Use the right command for your workflow to keep your Docker environment organized and efficient.

How Can I Inspect the File System of a Failed Docker Build?

DevOps Daily Team — Tue, 29 Apr 2025 09:00:00 GMT

TLDR

To inspect the file system of a failed Docker build, use multi-stage builds to save intermediate images, add debug steps (like sleep or tail), or leverage BuildKit's --target and --output features. This lets you start a shell in the failed state and debug interactively.

Why Is This Useful?

When a Docker build fails, you often want to see what files, environment variables, or installed packages are present at the failure point. This helps you debug missing files, permissions, or unexpected build behavior.

Method 1: Add a Debug Step (sleep or tail)

Temporarily add a line to your Dockerfile before the failing step:

RUN sleep infinity

RUN tail -f /dev/null

Build the image:

docker build -t debug-image .

In another terminal, find the running container:

docker ps

Start a shell in the container:

docker exec -it  /bin/bash

Now you can inspect the file system, check logs, and debug interactively. When done, remove the debug line from your Dockerfile.

Method 2: Use Multi-Stage Builds to Save Intermediate State

If your build uses multi-stage builds, you can tag an intermediate stage and run a container from it:

FROM ubuntu:22.04 as builder
# ... build steps ...

FROM builder as debug-stage
# Add a debug step if needed

Build up to the debug stage:

docker build --target debug-stage -t debug-image .
docker run -it debug-image /bin/bash

Method 3: Use BuildKit's --output=type=local

With BuildKit enabled, you can export the build's file system to a local directory:

docker build --output type=local,dest=./output .

This writes the final build context to ./output, so you can inspect files directly on your host.

Method 4: Commit a Stopped Container (if build got far enough)

If your build ran a container that exited, you can commit it to an image:

docker ps -a  # Find the exited container
# Then:
docker commit  debug-image
docker run -it debug-image /bin/bash

Best Practices

Remove debug steps after you're done to keep images clean.
Use multi-stage builds to avoid leaking secrets or build tools into final images.
For complex builds, consider using BuildKit for advanced debugging features.

Conclusion

Inspecting the file system of a failed Docker build is easy with debug steps, multi-stage builds, or BuildKit's output features. Use these techniques to troubleshoot and fix build issues faster.

docker: 'build' requires 1 argument. See 'docker build --help'

DevOps Daily Team — Mon, 28 Apr 2025 09:00:00 GMT

TLDR

The error docker: "build" requires 1 argument. See 'docker build --help' means you forgot to specify the build context (usually a directory) when running docker build. Always provide a path (like .) as the last argument.

Why Does This Error Happen?

When you run docker build, Docker expects a build context—a directory containing your Dockerfile and any files needed for the build. If you don't provide this argument, Docker doesn't know what to build.

Example of the error:

docker build
# Output:
# docker: "build" requires 1 argument. See 'docker build --help'.

How to Fix It

Add the build context (usually . for the current directory) at the end of your command:

docker build .

Or, if your Dockerfile is in a different directory:

docker build /path/to/context

You can also tag your image at the same time:

docker build -t my-image .

Common Pitfalls

Forgetting the . or path at the end of the command.
Using docker build -t my-image without the context (should be docker build -t my-image .).
Running the command from the wrong directory (make sure your Dockerfile is in the context directory).

Best Practices

Always double-check your build context before running docker build.
Use docker build -t . for most local builds.
For CI/CD, use absolute paths or ensure the working directory is correct.

Conclusion

The docker: "build" requires 1 argument error is a simple fix—just add the build context (like .) to your command. This ensures Docker knows where to find your Dockerfile and build resources.

Exposing a Port on a Live Docker Container

DevOps Daily Team — Sun, 27 Apr 2025 09:00:00 GMT

TLDR

You can't directly expose a new port on a running Docker container. To make a service accessible, you need to publish the port when starting the container. If you forgot, you can use workarounds like docker commit + docker run, or use tools like socat or docker network connect for advanced cases. Plan ahead to avoid surprises.

Why Can't You Expose a Port on a Live Container?

Docker's port publishing (-p or --publish) is set up when the container starts. It creates a network mapping from the host to the container. Once running, Docker doesn't let you add new port mappings to an existing container for security and technical reasons.

What Are Your Options?

1. Stop and Re-Run the Container with the Port Published

This is the most reliable approach:

docker stop my-container
docker rm my-container
docker run -d --name my-container -p 8080:8080 my-image

Use the same image and options, but add -p for the port you need.
If you need to preserve data, use named volumes or bind mounts.

2. Create a New Image from the Running Container (Not Ideal)

If you can't easily recreate the container, you can commit its state and start a new one:

docker commit my-container my-temp-image
docker run -d --name my-new-container -p 8080:8080 my-temp-image

This captures the current filesystem state, but not environment variables or volumes.
Use only as a last resort.

3. Use socat or Similar Tools as a Proxy

You can run a sidecar container that forwards traffic from the host to the running container:

docker run -d --network container:my-container -p 8080:8080 alpine/socat TCP-LISTEN:8080,fork TCP:localhost:8080

This listens on the host and forwards to the target port in the running container.
Useful for quick fixes, but not a long-term solution.

4. Use docker network connect (Advanced)

If your container is on a user-defined bridge network, you can connect another container to the same network and use it as a proxy. This doesn't publish a port to the host, but allows inter-container communication.

Best Practices for the Future

Always publish needed ports with -p or --publish when starting containers.
Use Docker Compose or scripts to manage container options consistently.
Document which ports your services need.
For production, use orchestration tools (like Kubernetes) for dynamic port management.

Conclusion

You can't expose a new port on a live Docker container, but you have workarounds. The best fix is to stop and re-run the container with the correct port mapping. For emergencies, use tools like socat to forward traffic. Plan your port mappings ahead to avoid downtime and surprises.

Using SSH Keys Inside a Docker Container

DevOps Daily Team — Sat, 26 Apr 2025 09:00:00 GMT

TLDR

To use SSH keys inside a Docker container, mount your key at runtime or use Docker's SSH agent forwarding. Never bake private keys into images. This guide covers secure patterns for development, CI, and production, with practical examples and troubleshooting tips.

Why Use SSH Keys in a Container?

You might need SSH keys in a container to:

Clone private git repositories
Connect to remote servers for automation
Use tools like Ansible or rsync over SSH

The Wrong Way: Copying Keys into Images

Never copy your private SSH keys into a Docker image. This exposes secrets to anyone with access to the image and risks leaking credentials if the image is pushed to a registry.

The Right Way: Mounting SSH Keys at Runtime

The safest approach is to mount your SSH key into the container at runtime:

docker run -v $HOME/.ssh/id_rsa:/root/.ssh/id_rsa:ro -it my-image

This makes your private key available only while the container runs.
Use :ro to mount read-only.
Set permissions inside the container if needed:

chmod 600 /root/.ssh/id_rsa

You can also mount your entire .ssh directory:

docker run -v $HOME/.ssh:/root/.ssh:ro -it my-image

Using SSH Agent Forwarding (Recommended for CI and Builds)

For build-time access (e.g., cloning private repos in a Dockerfile), use Docker's SSH agent forwarding (Docker 18.09+):

Start your SSH agent and add your key:

eval "$(ssh-agent -s)"
ssh-add ~/.ssh/id_rsa

Build with SSH forwarding:

docker build --ssh default .

In your Dockerfile, use:

# syntax=docker/dockerfile:1.2
FROM alpine
RUN apk add --no-cache openssh git
RUN --mount=type=ssh git clone git@github.com:yourorg/private-repo.git

The key is never copied into the image.
Works for multi-stage builds and CI pipelines.

Using SSH Keys for Remote Access

If you need to SSH from inside a running container:

Mount your key as above.
Make sure the container has an SSH client installed (e.g., apt-get install -y openssh-client).
Use ssh as usual:

ssh -i /root/.ssh/id_rsa user@remote-host

Best Practices

Never commit or bake private keys into images or version control.
Use agent forwarding for build-time access.
Mount keys at runtime for interactive or automation use.
Set correct permissions (chmod 600) on private keys.
Use separate deploy keys or service accounts for automation.
Clean up keys and known_hosts after use in CI/CD jobs.

Troubleshooting

If you see Permission denied (publickey), check key permissions and SSH config.
If mounting doesn't work, check your Docker version and volume syntax.
For multi-user containers, mount keys to the correct user's home directory.

Conclusion

Using SSH keys in Docker containers is safe and flexible when you mount them at runtime or use agent forwarding. Avoid copying secrets into images, and follow best practices to keep your credentials secure in every environment.

Connecting to PostgreSQL in a Docker Container from Outside

DevOps Daily Team — Fri, 25 Apr 2025 09:00:00 GMT

You have PostgreSQL running in a container and you want to connect from outside the container - from your laptop, a CI job, or another machine. This guide shows practical ways to expose the database for local development while keeping security in mind.

TLDR

Publish the container port to your host with -p 5432:5432, then connect to localhost:5432.
In Docker Compose, set ports: ["5432:5432"] on the db service.
For remote machines, bind to all addresses and open the host firewall carefully. Limit access with pg_hba.conf CIDR rules.
If connection fails, check container logs, port conflicts, listen_addresses, and pg_hba.conf.

Quick mental model:

Client        Host               Docker NAT           Container
------        ----               ----------           ---------
psql -> 127.0.0.1:5432  ->  host:5432  ->  container:5432 (PostgreSQL)

Prerequisites

Docker Desktop 4.x or Docker Engine 24.x+
psql 14+ (install with your package manager if you do not have it)

Option 1: Docker CLI - publish port 5432 and connect from host

Start PostgreSQL with a password and publish the port. Then connect using psql from your machine.

# Start a local Postgres with a persistent volume and a strong password
docker run --name pg-local --rm \
  -e POSTGRES_PASSWORD='P@ssw0rd-Dev' \
  -e POSTGRES_DB='appdb' \
  -p 5432:5432 \
  -v pgdata:/var/lib/postgresql/data \
  postgres:16

Connect from your host:

psql "postgresql://postgres:P@ssw0rd-Dev@localhost:5432/appdb"

Why this works:

-p 5432:5432 publishes the container's 5432 on your host. On macOS and Windows, Docker Desktop publishes to localhost. On Linux, it binds to the host interface.
PostgreSQL listens on the container's 0.0.0.0 by default in this image, and with a password set you can authenticate.

If you cannot connect:

Check the container logs for startup errors:

docker logs pg-local | tail -n 50

Check that 5432 is actually published and free on the host:

docker ps | grep pg-local
lsof -i :5432 | cat

Option 2: Docker Compose - declarative setup with ports

Compose makes the setup repeatable for your team.

version: '3.9'
services:
  db:
    image: postgres:16
    environment:
      POSTGRES_PASSWORD: P@ssw0rd-Dev
      POSTGRES_DB: appdb
    ports:
      - '5432:5432' # host:container
    volumes:
      - pgdata:/var/lib/postgresql/data
    healthcheck:
      test: ['CMD-SHELL', 'pg_isready -U postgres']
      interval: 5s
      timeout: 3s
      retries: 5
volumes:
  pgdata:

Connect from your host the same way:

psql "postgresql://postgres:P@ssw0rd-Dev@localhost:5432/appdb"

Binding to all interfaces for remote clients

If another machine on your network needs access, publish the port and make sure the Postgres server is reachable through the host's IP address.

services:
  db:
    image: postgres:16
    command: ['postgres', '-c', 'listen_addresses=*']
    environment:
      POSTGRES_PASSWORD: P@ssw0rd-Dev
      POSTGRES_DB: appdb
    ports:
      - '5432:5432'
    volumes:
      - pgdata:/var/lib/postgresql/data

Then:

Make sure your host firewall allows TCP 5432 from the specific source network.
Limit which clients can authenticate by adjusting pg_hba.conf entries. You can inject a minimal file with a bind mount for local dev:

cat > /tmp/pg_hba.conf <<'HBA'
# TYPE  DATABASE  USER      ADDRESS        METHOD
host    all       all       192.168.1.0/24 scram-sha-256
HBA

docker run --name pg-remote --rm \
  -e POSTGRES_PASSWORD='P@ssw0rd-Dev' \
  -p 5432:5432 \
  -v /tmp/pg_hba.conf:/var/lib/postgresql/data/pg_hba.conf \
  -v pgdata:/var/lib/postgresql/data \
  postgres:16 -c listen_addresses='*'

Note: on first initialization, the database populates default config files under /var/lib/postgresql/data. Mounting pg_hba.conf like this will override it for development. In production, use baked images or config management instead of ad-hoc binds.

Verifying from the container side

Run a quick query to confirm the server is listening and accepting connections.

docker exec -it pg-local psql -U postgres -d appdb -c "show listen_addresses;"
docker exec -it pg-local psql -U postgres -d appdb -c "select 1;"

You can also inspect the active pg_hba.conf location and reload configuration:

docker exec -it pg-local psql -U postgres -d appdb -c "show hba_file;"
docker exec -it pg-local psql -U postgres -d appdb -c "select pg_reload_conf();"

Connecting from another container

If your app runs in another container, connect over a user-defined network rather than publishing 5432.

docker network create devnet || true
docker run -d --name db --network devnet \
  -e POSTGRES_PASSWORD='P@ssw0rd-Dev' -e POSTGRES_DB='appdb' \
  postgres:16

# Use the service name as the host inside the same network
docker run --rm --network devnet \
  --entrypoint psql \
  postgres:16 \
  "postgresql://postgres:P@ssw0rd-Dev@db:5432/appdb" -c 'select 1;'

This keeps the database private to the network while still reachable by your application container.

Common pitfalls and fixes

Port already in use: pick another host port with -p 15432:5432 and connect to localhost:15432.
Wrong hostname: use localhost from your machine, or the host IP for remote clients. Do not use 0.0.0.0 as a client target.
Firewalls: on Linux servers, open 5432 with ufw or iptables only for the subnets you trust.
Password errors: confirm the username, database name, and password. The default admin user is postgres.
Config not reloading: use select pg_reload_conf(); or restart the container after changing config files.

Quick connectivity checklist

# 1) Is the container healthy and listening?
docker ps | grep postgres
docker logs pg-local | tail -n 50

# 2) Is the host port published and free?
docker port pg-local 5432 | cat
lsof -i :5432 | cat

# 3) Can psql connect locally?
psql "postgresql://postgres:P@ssw0rd-Dev@localhost:5432/appdb" -c 'select version();'

With these patterns you can safely expose Postgres for local development, connect from your host, and optionally allow remote clients when needed. Prefer private Docker networks for app-to-DB traffic, and only publish 5432 when you must, with limited CIDRs and a firewall in front.

How to Use sudo Inside a Docker Container

DevOps Daily Team — Fri, 25 Apr 2025 09:00:00 GMT

TLDR

Most Docker containers run as root by default, so you usually don't need sudo to run privileged commands. If you want to use sudo (for multi-user images or extra security), you'll need to install it and configure users. This guide shows how, with practical examples and best practices.

Why Isn't sudo Available by Default?

Docker containers are designed to be lightweight and secure. By default, most images run as root, so you can install packages or modify the system without sudo. Many base images (like Alpine, Ubuntu, Debian) don't include sudo to keep images small.

Running as Root (No sudo Needed)

If your container runs as root (the default), just run commands directly:

FROM ubuntu:22.04
RUN apt-get update && apt-get install -y curl
CMD ["bash"]

Inside the container:

# You're already root
apt-get update

Adding sudo to a Container

If you want to use sudo (for example, to switch between users or for development parity), you need to install it and set up users:

FROM ubuntu:22.04
RUN apt-get update && apt-get install -y sudo
RUN useradd -m devuser && echo "devuser ALL=(ALL) NOPASSWD:ALL" >> /etc/sudoers
USER devuser
CMD ["bash"]

Now, inside the container:

sudo apt-get update

This works just like on a regular Linux system.

When Should You Use sudo in Docker?

Development images: To mimic a real user environment or test scripts that require sudo.
Multi-user containers: If your container runs as a non-root user but needs to escalate privileges for some tasks.
Security best practices: For production, it's better to run as a non-root user and only use sudo when absolutely necessary.

Best Practices

Avoid running production containers as root unless required.
Only install sudo if you need it—otherwise, keep images minimal.
Use the USER directive in your Dockerfile to specify a non-root user.
For one-off commands, you can override the user at runtime:

docker run -u root my-image whoami

Troubleshooting

If you see sudo: command not found, install it in your Dockerfile.
If you get permission errors, check your user and group settings.
For Alpine images, use apk add sudo instead of apt-get install sudo.

Conclusion

You rarely need sudo in Docker containers, since most run as root by default. If you do need it, install and configure it in your Dockerfile, and use the USER directive for better security. Keep your images minimal and only add what you need for your use case.

How Can I Trigger a Kubernetes Scheduled Job Manually?

DevOps Daily Team — Fri, 25 Apr 2025 09:00:00 GMT

Kubernetes Scheduled Jobs, also known as CronJobs, are designed to run tasks at specific intervals. But what if you need to trigger a Scheduled Job manually? This can be useful for testing, debugging, or running a task outside its regular schedule.

In this guide, you'll learn how to manually trigger a Kubernetes Scheduled Job, along with best practices and potential pitfalls.

Prerequisites

Before proceeding, ensure the following:

You have kubectl installed and configured to access your Kubernetes cluster.
You have permissions to create and manage Jobs in the cluster.
You understand the structure of the CronJob you want to trigger.

Understanding CronJobs and Jobs

A CronJob in Kubernetes creates Jobs based on a schedule. Each Job represents a single execution of the task defined in the CronJob.

Example CronJob YAML

apiVersion: batch/v1
kind: CronJob
metadata:
  name: example-cronjob
spec:
  schedule: '*/5 * * * *'
  jobTemplate:
    spec:
      template:
        spec:
          containers:
            - name: example
              image: busybox
              command: ['echo', 'Hello, Kubernetes!']
          restartPolicy: OnFailure

This CronJob runs every 5 minutes and executes a simple echo command.

Triggering a CronJob Manually

Method 1: Create a Job from the CronJob Template

You can manually create a Job using the CronJob's template. This bypasses the schedule and runs the task immediately.

kubectl create job --from=cronjob/example-cronjob manual-job

Explanation

kubectl create job: Creates a new Job.
--from=cronjob/example-cronjob: Specifies the CronJob to use as a template.
manual-job: The name of the new Job.

Method 2: Edit the CronJob Schedule

Temporarily change the CronJob's schedule to run immediately. For example, set the schedule to "* * * * *" to run every minute.

kubectl edit cronjob example-cronjob

After the CronJob runs, revert the schedule to its original value.

+-------------------+
|   Kubernetes      |
|                   |
| +---------------+ |
| |   CronJob     | |
| +---------------+ |
| +---------------+ |
| |   Job         | |
| +---------------+ |
| +---------------+ |
| |   Pod         | |
| +---------------+ |
+-------------------+

Best Practices

Use Unique Names: When creating Jobs manually, use unique names to avoid conflicts.
Monitor Logs: Check the logs of the Job to ensure it executed successfully.
Revert Changes: If you edit the CronJob schedule, remember to revert it after the task runs.

Example Scenario

Imagine you have a CronJob that backs up a database every night. You need to run the backup manually during the day to test a new configuration. By creating a Job from the CronJob template, you can trigger the backup without waiting for the scheduled time.

Conclusion

Manually triggering a Kubernetes Scheduled Job is a straightforward process that can be invaluable for testing and debugging. By following the methods and best practices outlined here, you can run tasks on demand while maintaining the integrity of your cluster.

What Is the Difference Between 'docker compose' and 'docker-compose'?

DevOps Daily Team — Thu, 24 Apr 2025 09:00:00 GMT

TLDR

docker compose is the modern, built-in Docker CLI command for managing multi-container apps, while docker-compose is the legacy standalone tool. Both use the same YAML files, but docker compose is the future and offers better integration, new features, and improved performance.

The Old Way: docker-compose

docker-compose (with a hyphen) is the original tool for running multi-container Docker applications. It's a separate Python-based binary you install with pip or your package manager:

pip install docker-compose
# or
brew install docker-compose

You run it like this:

docker-compose up

The New Way: docker compose

docker compose (with a space) is the new subcommand built directly into the Docker CLI. It's written in Go, ships with Docker Desktop and recent Docker Engine versions, and is the official replacement for the old tool.

You use it like this:

docker compose up

No need to install anything extra—just update Docker.

Key Differences

Integration: docker compose is part of the main Docker CLI, so you get a consistent experience and better support for new Docker features.
Performance: The new command is faster and uses less memory, especially for large projects.
Features: Some new features (like Compose profiles, better volume/network handling, and Compose Spec support) are only available in docker compose.
Compatibility: Both commands use the same docker-compose.yml files, but some edge-case flags or behaviors may differ. Check the Compose V2 migration guide if you run into issues.
Maintenance: docker-compose (the old tool) is in maintenance mode. All new development is on docker compose.

Which Should You Use?

For new projects, always use docker compose.
For old scripts or CI pipelines, you can keep using docker-compose for now, but plan to migrate.
If you hit a missing feature or bug, check if you're using the latest Docker version.

How to Check Your Version

To see which Compose you have:

docker compose version
# or
docker-compose version

If you see a warning about Compose V1 being deprecated, it's time to switch.

Conclusion

docker compose is the modern, supported way to manage multi-container Docker apps. It replaces the old docker-compose tool, brings new features, and is the best choice for future-proofing your workflow. Update your Docker installation and start using the new command for a smoother experience.