Sometimes you need a lightweight, dependency-free solution that works across any Unix-like system. That's where sed and awk shine:

  • They're available on virtually every Unix-like system
  • No need for additional installations or dependencies
  • Blazing fast execution for on-the-fly changes
  • Powerful enough to handle complex transformations

The Dynamic Duo in Action

Let's start with a real-world scenario: You need to generate a Nginx configuration file dynamically based on environment variables and a template. Here's a template to work with:


server {
    listen 80;
    server_name {{SERVER_NAME}};
    
    location / {
        proxy_pass {{UPSTREAM_URL}};
    }
}

Now, let's use sed to replace those placeholders with actual values:


#!/bin/bash

SERVER_NAME="example.com"
UPSTREAM_URL="http://backend:8080"

sed -e "s/{{SERVER_NAME}}/$SERVER_NAME/g" \
    -e "s|{{UPSTREAM_URL}}|$UPSTREAM_URL|g" \
    nginx.template > nginx.conf

This script reads the template, replaces the placeholders, and outputs a new configuration file. Simple, right? But what if we need something more complex?

Leveling Up: Multi-Server Configuration

Let's say you have a list of backend servers and want to generate an upstream block dynamically. This is where awk really shines:


#!/bin/bash

SERVERS="server1:8080 server2:8081 server3:8082"

echo "upstream backend {"
echo "$SERVERS" | awk '{
    split($0, servers, " ")
    for (i in servers) {
        split(servers[i], parts, ":")
        printf "    server %s:%s;\n", parts[1], parts[2]
    }
}'
echo "}"

This script will generate an upstream block like this:


upstream backend {
    server server1:8080;
    server server2:8081;
    server server3:8082;
}

The Plot Thickens: Conditional Config Generation

Now, let's combine sed and awk for some truly dynamic configuration. We'll create a script that generates different configs based on the environment:


#!/bin/bash

ENV=${1:-production}
TEMPLATE="nginx.template"

# Generate server list based on environment
if [ "$ENV" = "production" ]; then
    SERVERS="prod1:8080 prod2:8080 prod3:8080"
else
    SERVERS="dev1:8080"
fi

# Generate upstream block
UPSTREAM=$(echo "$SERVERS" | awk '{
    print "upstream backend {"
    split($0, servers, " ")
    for (i in servers) {
        split(servers[i], parts, ":")
        printf "    server %s:%s;\n", parts[1], parts[2]
    }
    print "}"
}')

# Replace placeholders in template
sed -e "s/{{ENV}}/$ENV/g" \
    -e "s|{{UPSTREAM_BLOCK}}|$UPSTREAM|g" \
    "$TEMPLATE" > "nginx_$ENV.conf"

This script takes an environment as an argument, generates an appropriate upstream block, and then uses sed to insert it into the final configuration.

Pitfalls to Watch Out For

While sed and awk are powerful, they're not without their quirks:

  • Beware of special characters in your replacements. Use different delimiters (like | instead of /) when necessary.
  • Awk's default field separator is whitespace. Use -F to change it if needed.
  • Remember that sed processes the input line by line, which can limit some operations.

Taking It Further

Once you've mastered these basics, you can do some truly impressive things:

  • Generate complex HAProxy configurations based on service discovery data
  • Dynamically update database connection strings across multiple config files
  • Create environment-specific application configurations on container startup

Wrapping Up

Sed and awk might seem like relics from a bygone era, but they're still incredibly relevant for modern DevOps practices. Their ability to perform complex text transformations quickly and without dependencies makes them invaluable tools for dynamic configuration management.

Remember, the next time you're faced with a config nightmare, don't reach for that heavy-duty configuration management system just yet. Sometimes, a bit of sed and awk magic is all you need to save the day (and your sanity).

"Give me sed, awk, and a place to stand, and I will move the world." - Archimedes (if he were a DevOps engineer)

Now go forth and conquer those configs!