Amazon Linux 2023 | Production-Ready | Workflow Automation
Complete AMI guide for Apache Airflow workflow automation platform. Learn how to access the web UI, create DAGs, manage connections, and configure your instance for production use.
Follow these steps to access your Apache Airflow instance and start creating workflows.
Connect to your EC2 instance using SSH with your key pair.
ssh -i your-key.pem ec2-user@<your-instance-ip>
View the auto-generated Airflow admin password.
sudo cat /opt/airflow/credentials.txt
Open Airflow in your browser using your instance's public IP on port 8080.
http://<your-instance-ip>:8080
Login with:
Username: admin
Password: (from credentials file)
Verify that Airflow services are running properly.
sudo systemctl status airflow-webserver airflow-scheduler
All credentials are automatically generated and stored securely on your instance.
sudo cat /opt/airflow/credentials.txt
| File | Location |
|---|---|
| Airflow Home | /opt/airflow |
| DAGs Directory | /opt/airflow/dags |
| Configuration | /opt/airflow/airflow.cfg |
| Logs Directory | /opt/airflow/logs |
| Plugins Directory | /opt/airflow/plugins |
The Airflow web interface provides a powerful dashboard for managing your workflows.
Open your browser and navigate to the Airflow Web UI.
http://<your-instance-ip>:8080
Username: admin
Password: (from credentials file)
The main dashboard shows all your DAGs with their status.
Manually trigger DAG execution from the UI.
Access detailed logs for debugging tasks.
Visualize DAG structure and task dependencies with interactive graph.
See historical runs in a grid format with color-coded status indicators.
View DAG runs over time on a calendar with success/failure indicators.
Analyze task execution times to identify bottlenecks.
View task execution timeline with parallel task visualization.
View the actual Python code of your DAG directly in the UI.
DAGs (Directed Acyclic Graphs) define your workflows in Airflow.
Create a Python file in the DAGs directory:
sudo nano /opt/airflow/dags/my_dag.py
Basic DAG template:
from airflow import DAG
from airflow.operators.bash import BashOperator
from datetime import datetime
with DAG(
'my_dag',
start_date=datetime(2024, 1, 1),
schedule_interval='@daily',
catchup=False
) as dag:
task1 = BashOperator(
task_id='hello',
bash_command='echo Hello!'
)
# List all DAGs
airflow dags list
# Trigger a DAG run
airflow dags trigger my_dag
# Pause a DAG
airflow dags pause my_dag
# Unpause a DAG
airflow dags unpause my_dag
# Test a DAG (dry run)
airflow dags test my_dag 2024-01-01
/opt/airflow/dags - no restart needed.
catchup=False to prevent backfilling historical runs.
max_active_runs=1 to prevent overlapping DAG runs.
The Airflow CLI provides powerful commands for managing your workflows.
# List all DAGs
airflow dags list
# Show DAG details
airflow dags show my_dag
# List DAG runs
airflow dags list-runs -d my_dag
# List tasks in a DAG
airflow tasks list my_dag
# Test a specific task
airflow tasks test my_dag task_id 2024-01-01
# Run a task
airflow tasks run my_dag task_id 2024-01-01
# List connections
airflow connections list
# Add a connection
airflow connections add 'my_conn' \
--conn-type 'postgres' \
--conn-host 'localhost' \
--conn-login 'user' \
--conn-password 'pass'
# List variables
airflow variables list
# Set a variable
airflow variables set my_var 'my_value'
# Get a variable
airflow variables get my_var
# Import variables from JSON
airflow variables import variables.json
Configure external connections and environment variables for your workflows.
Variable.get('key')
# Add AWS connection via CLI
airflow connections add 'aws_default' \
--conn-type 'aws' \
--conn-login 'YOUR_ACCESS_KEY' \
--conn-password 'YOUR_SECRET_KEY' \
--conn-extra '{"region_name": "us-east-1"}'
Access and manage Airflow logs for debugging and monitoring.
Check Airflow service logs using journalctl.
# Webserver logs
sudo journalctl -u airflow-webserver -f
# Scheduler logs
sudo journalctl -u airflow-scheduler -f
Access task execution logs from the logs directory.
# List DAG logs
ls -la /opt/airflow/logs/dag_id=my_dag/
# View specific task log
cat /opt/airflow/logs/dag_id=my_dag/run_id=manual__2024-01-01/task_id=my_task/attempt=1.log
Configure log levels and retention in airflow.cfg.
# Edit Airflow configuration
sudo nano /opt/airflow/airflow.cfg
# Key logging settings:
[logging]
base_log_folder = /opt/airflow/logs
logging_level = INFO
Remove old log files to free up disk space.
# Remove logs older than 30 days
find /opt/airflow/logs -type f -mtime +30 -delete
# Check log disk usage
du -sh /opt/airflow/logs
Follow these steps to secure your Airflow deployment.
Important: Change the default admin password immediately after your first login. The pre-generated password should only be used for initial access.
Update admin password via Security > List Users in the Web UI.
Restrict port 8080 access to only your IP addresses or VPN CIDR range.
Fernet encryption is enabled for connection passwords and variables.
# Reset admin password via CLI
airflow users reset-password -u admin -p 'your-new-secure-password'
Manage Airflow services using systemd.
# Start services
sudo systemctl start airflow-webserver
sudo systemctl start airflow-scheduler
# Stop services
sudo systemctl stop airflow-webserver
sudo systemctl stop airflow-scheduler
# Restart services
sudo systemctl restart airflow-webserver airflow-scheduler
# Check all Airflow services
sudo systemctl status airflow-webserver airflow-scheduler
# Check if services are enabled
sudo systemctl is-enabled airflow-webserver airflow-scheduler
# Enable services on boot
sudo systemctl enable airflow-webserver airflow-scheduler
Common issues and how to resolve them.
Symptom: Connection refused or timeout when accessing port 8080.
Solution:
8080sudo systemctl status airflow-webserversudo journalctl -u airflow-webserver -n 50Symptom: New DAG files not showing in the Web UI.
Solution:
python3 /opt/airflow/dags/my_dag.py/opt/airflow/dags/Symptom: Tasks remain in "queued" state and never run.
Solution:
sudo systemctl status airflow-schedulersudo systemctl restart airflow-schedulerSymptom: Airflow fails to start with database errors.
Solution:
sudo systemctl status postgresqlsudo systemctl restart postgresqlairflow db initOur team is available 24/7 to assist you with any questions or issues.
support@flowopsconsulting.co.uk