Overview
What's Included
- Apache Airflow latest stable
- Web UI for DAG management
- PostgreSQL metadata backend
- Celery executor ready
- Example DAGs included
- systemd service management
- Separate data volume for logs
Step-by-Step Guide
1
Launch the AMI
Find "Airflow FlowOps" on AWS Marketplace and launch.
# Recommended instance types:
# t3.large (8GB) - Development, small workloads
# m5.xlarge (16GB) - Production
# m5.2xlarge (32GB) - Heavy workloads
# Airflow needs memory for workers
2
Access the Web UI
Log into the Airflow dashboard to manage your DAGs.
ssh -i your-key.pem ubuntu@your-instance-ip
sudo cat /opt/flowops/credentials.txt
# Airflow UI:
http://your-instance-ip:8080
# Login with admin credentials from credentials.txt
3
Create Your First DAG
Write a Python DAG and deploy it to Airflow.
# Save to /opt/airflow/dags/my_first_dag.py
from airflow import DAG
from airflow.operators.python import PythonOperator
from airflow.operators.bash import BashOperator
from datetime import datetime, timedelta
default_args = {
'owner': 'airflow',
'depends_on_past': False,
'start_date': datetime(2024, 1, 1),
'retries': 1,
'retry_delay': timedelta(minutes=5),
}
with DAG(
'my_first_dag',
default_args=default_args,
schedule_interval='@daily',
catchup=False,
) as dag:
def extract():
print("Extracting data...")
return {"data": [1, 2, 3]}
def transform(data):
print(f"Transforming: {data}")
return [x * 2 for x in data["data"]]
extract_task = PythonOperator(
task_id='extract',
python_callable=extract,
)
transform_task = PythonOperator(
task_id='transform',
python_callable=transform,
op_kwargs={'data': '{{ ti.xcom_pull(task_ids="extract") }}'},
)
load_task = BashOperator(
task_id='load',
bash_command='echo "Loading data to destination..."',
)
extract_task >> transform_task >> load_task
4
Monitor and Trigger
Use the UI to trigger runs and monitor execution.
# Trigger DAG via CLI
airflow dags trigger my_first_dag
# List running DAGs
airflow dags list-runs -d my_first_dag
# Check task logs
airflow tasks logs my_first_dag extract 2024-01-01
# Pause/unpause DAG
airflow dags pause my_first_dag
airflow dags unpause my_first_dag
Comparison
| Feature | This AMI | MWAA | Self-Install |
|---|---|---|---|
| Setup Time | 5 minutes | 15 minutes | 2+ hours |
| Cost (similar spec) | ~$70/month | ~$300/month | ~$70/month |
| Custom Plugins | Yes | Limited | Yes |
| Version Control | You choose | AWS managed | You choose |
| Scaling | Manual | Auto | Manual |
Use Cases
- ETL/ELT pipelines
- ML model training workflows
- Data warehouse loading
- Report generation
- API data synchronization
- DevOps automation
Pricing
Our AMIs are available on AWS Marketplace with simple hourly pricing:
- Software: $0.10-0.15/hour (varies by product)
- EC2: Standard AWS pricing for your chosen instance
- Support: 24/7 included with guaranteed 24-hour response
Need Help?
Our UK-based support team is here to help you get the most out of your deployment. Email us at support@flowopsconsulting.co.uk for any questions.
Comments