Home
Data Engineering

Deploy Apache Airflow on AWS in 5 Minutes

Apache Airflow is the standard for workflow orchestration. Whether you're building ETL pipelines, ML workflows, or complex data processing jobs, Airflow lets you define, schedule, and monitor everything in Python. Our AMI gets you running in minutes.

8 min read FlowOps Consulting

What's Included

  • Apache Airflow latest stable
  • Web UI for DAG management
  • PostgreSQL metadata backend
  • Celery executor ready
  • Example DAGs included
  • systemd service management
  • Separate data volume for logs

Step-by-Step Guide

1

Launch the AMI

Find "Airflow FlowOps" on AWS Marketplace and launch.

# Recommended instance types: # t3.large (8GB) - Development, small workloads # m5.xlarge (16GB) - Production # m5.2xlarge (32GB) - Heavy workloads # Airflow needs memory for workers
2

Access the Web UI

Log into the Airflow dashboard to manage your DAGs.

ssh -i your-key.pem ubuntu@your-instance-ip sudo cat /opt/flowops/credentials.txt # Airflow UI: http://your-instance-ip:8080 # Login with admin credentials from credentials.txt
3

Create Your First DAG

Write a Python DAG and deploy it to Airflow.

# Save to /opt/airflow/dags/my_first_dag.py from airflow import DAG from airflow.operators.python import PythonOperator from airflow.operators.bash import BashOperator from datetime import datetime, timedelta default_args = { 'owner': 'airflow', 'depends_on_past': False, 'start_date': datetime(2024, 1, 1), 'retries': 1, 'retry_delay': timedelta(minutes=5), } with DAG( 'my_first_dag', default_args=default_args, schedule_interval='@daily', catchup=False, ) as dag: def extract(): print("Extracting data...") return {"data": [1, 2, 3]} def transform(data): print(f"Transforming: {data}") return [x * 2 for x in data["data"]] extract_task = PythonOperator( task_id='extract', python_callable=extract, ) transform_task = PythonOperator( task_id='transform', python_callable=transform, op_kwargs={'data': '{{ ti.xcom_pull(task_ids="extract") }}'}, ) load_task = BashOperator( task_id='load', bash_command='echo "Loading data to destination..."', ) extract_task >> transform_task >> load_task
4

Monitor and Trigger

Use the UI to trigger runs and monitor execution.

# Trigger DAG via CLI airflow dags trigger my_first_dag # List running DAGs airflow dags list-runs -d my_first_dag # Check task logs airflow tasks logs my_first_dag extract 2024-01-01 # Pause/unpause DAG airflow dags pause my_first_dag airflow dags unpause my_first_dag

Comparison

FeatureThis AMIMWAASelf-Install
Setup Time5 minutes15 minutes2+ hours
Cost (similar spec)~$70/month~$300/month~$70/month
Custom PluginsYesLimitedYes
Version ControlYou chooseAWS managedYou choose
ScalingManualAutoManual

Use Cases

  • ETL/ELT pipelines
  • ML model training workflows
  • Data warehouse loading
  • Report generation
  • API data synchronization
  • DevOps automation

Pricing

Our AMIs are available on AWS Marketplace with simple hourly pricing:

Need Help?

Our UK-based support team is here to help you get the most out of your deployment. Email us at support@flowopsconsulting.co.uk for any questions.

Ready to Get Started?

Deploy in 5 minutes on AWS Marketplace

View on AWS Marketplace

Comments