Home
Available on AWS Marketplace

Apache Airflow

Amazon Linux 2023 | Production-Ready | Workflow Automation

Complete AMI guide for Apache Airflow workflow automation platform. Learn how to access the web UI, create DAGs, manage connections, and configure your instance for production use.

Airflow 2.10 Python 3.11 PostgreSQL Backend Amazon Linux 2023

Quick Start Guide

Follow these steps to access your Apache Airflow instance and start creating workflows.

1

SSH to Your Instance

Connect to your EC2 instance using SSH with your key pair.

ssh -i your-key.pem ec2-user@<your-instance-ip>
2

Find Your Credentials

View the auto-generated Airflow admin password.

sudo cat /opt/airflow/credentials.txt
3

Access the Web UI

Open Airflow in your browser using your instance's public IP on port 8080.

http://<your-instance-ip>:8080

Login with:
Username: admin
Password: (from credentials file)

4

Check Service Status

Verify that Airflow services are running properly.

sudo systemctl status airflow-webserver airflow-scheduler

Credentials & Connection Details

All credentials are automatically generated and stored securely on your instance.

Airflow Web UI

  • URL: http://<your-ip>:8080
  • Username: admin
  • Password: See credentials file
  • Port: 8080

PostgreSQL Backend

  • Database: airflow
  • Username: airflow
  • Executor: LocalExecutor
  • Encryption: Fernet key enabled

Credentials File Location

sudo cat /opt/airflow/credentials.txt

Important File Locations

File Location
Airflow Home /opt/airflow
DAGs Directory /opt/airflow/dags
Configuration /opt/airflow/airflow.cfg
Logs Directory /opt/airflow/logs
Plugins Directory /opt/airflow/plugins

Using the Airflow Web UI

The Airflow web interface provides a powerful dashboard for managing your workflows.

1

Login to Airflow

Open your browser and navigate to the Airflow Web UI.

http://<your-instance-ip>:8080

Username: admin
Password: (from credentials file)

2

View DAGs Dashboard

The main dashboard shows all your DAGs with their status.

  • 1. View DAG list with on/off toggles
  • 2. See last run status and schedule
  • 3. Click DAG name for detailed view
  • 4. Monitor task success/failure rates
3

Trigger DAG Runs

Manually trigger DAG execution from the UI.

  • 1. Click the Play button on any DAG
  • 2. Optionally add configuration JSON
  • 3. Click Trigger DAG
  • 4. Monitor progress in Graph or Tree view
4

View Task Logs

Access detailed logs for debugging tasks.

  • 1. Click on any task in the DAG view
  • 2. Select Log from the popup
  • 3. View stdout, stderr, and execution details
  • 4. Download logs for offline analysis

Key Web UI Features

Graph View

Visualize DAG structure and task dependencies with interactive graph.

Grid View

See historical runs in a grid format with color-coded status indicators.

Calendar View

View DAG runs over time on a calendar with success/failure indicators.

Task Duration

Analyze task execution times to identify bottlenecks.

Gantt Chart

View task execution timeline with parallel task visualization.

Code View

View the actual Python code of your DAG directly in the UI.

Creating and Managing DAGs

DAGs (Directed Acyclic Graphs) define your workflows in Airflow.

Create a New DAG

Create a Python file in the DAGs directory:

sudo nano /opt/airflow/dags/my_dag.py

Example DAG Structure

Basic DAG template:

from airflow import DAG from airflow.operators.bash import BashOperator from datetime import datetime with DAG( 'my_dag', start_date=datetime(2024, 1, 1), schedule_interval='@daily', catchup=False ) as dag: task1 = BashOperator( task_id='hello', bash_command='echo Hello!' )

DAG Management Commands

# List all DAGs airflow dags list # Trigger a DAG run airflow dags trigger my_dag # Pause a DAG airflow dags pause my_dag # Unpause a DAG airflow dags unpause my_dag # Test a DAG (dry run) airflow dags test my_dag 2024-01-01

DAG Development Tips

  • DAGs are automatically detected from /opt/airflow/dags - no restart needed.
  • Use catchup=False to prevent backfilling historical runs.
  • Set max_active_runs=1 to prevent overlapping DAG runs.
  • Use Airflow Variables and Connections for sensitive configuration.

Using the Airflow CLI

The Airflow CLI provides powerful commands for managing your workflows.

DAG Commands

# List all DAGs airflow dags list # Show DAG details airflow dags show my_dag # List DAG runs airflow dags list-runs -d my_dag

Task Commands

# List tasks in a DAG airflow tasks list my_dag # Test a specific task airflow tasks test my_dag task_id 2024-01-01 # Run a task airflow tasks run my_dag task_id 2024-01-01

Connection Commands

# List connections airflow connections list # Add a connection airflow connections add 'my_conn' \ --conn-type 'postgres' \ --conn-host 'localhost' \ --conn-login 'user' \ --conn-password 'pass'

Variable Commands

# List variables airflow variables list # Set a variable airflow variables set my_var 'my_value' # Get a variable airflow variables get my_var # Import variables from JSON airflow variables import variables.json

Connections & Variables

Configure external connections and environment variables for your workflows.

Adding Connections via UI

  • Go to Admin > Connections
  • Click + to add a new connection
  • Select connection type (postgres, mysql, aws, etc.)
  • Fill in host, port, credentials
  • Test connection before saving

Managing Variables via UI

  • Go to Admin > Variables
  • Click + to add a new variable
  • Enter key and value (supports JSON)
  • Access in DAGs: Variable.get('key')
  • Import/export variables as JSON

AWS Connection Example

# Add AWS connection via CLI airflow connections add 'aws_default' \ --conn-type 'aws' \ --conn-login 'YOUR_ACCESS_KEY' \ --conn-password 'YOUR_SECRET_KEY' \ --conn-extra '{"region_name": "us-east-1"}'

Log Management

Access and manage Airflow logs for debugging and monitoring.

View Service Logs

Check Airflow service logs using journalctl.

# Webserver logs sudo journalctl -u airflow-webserver -f # Scheduler logs sudo journalctl -u airflow-scheduler -f

View Task Logs

Access task execution logs from the logs directory.

# List DAG logs ls -la /opt/airflow/logs/dag_id=my_dag/ # View specific task log cat /opt/airflow/logs/dag_id=my_dag/run_id=manual__2024-01-01/task_id=my_task/attempt=1.log

Log Configuration

Configure log levels and retention in airflow.cfg.

# Edit Airflow configuration sudo nano /opt/airflow/airflow.cfg # Key logging settings: [logging] base_log_folder = /opt/airflow/logs logging_level = INFO

Clean Old Logs

Remove old log files to free up disk space.

# Remove logs older than 30 days find /opt/airflow/logs -type f -mtime +30 -delete # Check log disk usage du -sh /opt/airflow/logs

Security Recommendations

Follow these steps to secure your Airflow deployment.

Important: Change the default admin password immediately after your first login. The pre-generated password should only be used for initial access.

Change Password

Update admin password via Security > List Users in the Web UI.

Security Groups

Restrict port 8080 access to only your IP addresses or VPN CIDR range.

Fernet Key

Fernet encryption is enabled for connection passwords and variables.

Change Admin Password

# Reset admin password via CLI airflow users reset-password -u admin -p 'your-new-secure-password'

Service Management

Manage Airflow services using systemd.

Start/Stop Services

# Start services sudo systemctl start airflow-webserver sudo systemctl start airflow-scheduler # Stop services sudo systemctl stop airflow-webserver sudo systemctl stop airflow-scheduler # Restart services sudo systemctl restart airflow-webserver airflow-scheduler

Check Service Status

# Check all Airflow services sudo systemctl status airflow-webserver airflow-scheduler # Check if services are enabled sudo systemctl is-enabled airflow-webserver airflow-scheduler # Enable services on boot sudo systemctl enable airflow-webserver airflow-scheduler

Troubleshooting

Common issues and how to resolve them.

Can't Access Web UI

Symptom: Connection refused or timeout when accessing port 8080.

Solution:

  • 1. Check Security Group allows inbound on port 8080
  • 2. Verify webserver is running: sudo systemctl status airflow-webserver
  • 3. Check logs: sudo journalctl -u airflow-webserver -n 50

DAGs Not Appearing

Symptom: New DAG files not showing in the Web UI.

Solution:

  • 1. Check DAG syntax: python3 /opt/airflow/dags/my_dag.py
  • 2. Verify file location is /opt/airflow/dags/
  • 3. Wait for scheduler to parse (up to 30 seconds)
  • 4. Check scheduler logs for import errors

Tasks Stuck in Queued

Symptom: Tasks remain in "queued" state and never run.

Solution:

  • 1. Check scheduler is running: sudo systemctl status airflow-scheduler
  • 2. Verify DAG is unpaused in Web UI
  • 3. Check parallelism settings in airflow.cfg
  • 4. Restart scheduler: sudo systemctl restart airflow-scheduler

Database Connection Failed

Symptom: Airflow fails to start with database errors.

Solution:

  • 1. Check PostgreSQL is running: sudo systemctl status postgresql
  • 2. Verify database connection in airflow.cfg
  • 3. Restart PostgreSQL: sudo systemctl restart postgresql
  • 4. Re-initialize database if needed: airflow db init

Need Help?

Our team is available 24/7 to assist you with any questions or issues.

support@flowopsconsulting.co.uk
24/7 Support Available