# airflow

## Learning Path Diagrams Book Mode

[View on GitHub](https://github.com/apache/airflow "View on GitHub")

### 15 units • 5 levels

#### Change Guide

Welcome to airflow

Your guide to understanding the codebase

### Start Reading

5 levels

15 learning units

### Orchestration & APIs
- Pipelines, workflows, public interfaces • 3 units

### DAGs and Task Fundamentals
- Data Model·12

### Data Persistence Layer
- Data Model

### Configuration and CLI Basics
- Infrastructure·11

### Core Logic & Data
- Business rules, schemas, models • 3 units

### REST API and Service Layer
- API

### Scheduling and Execution
- Workflow·5

### Task Communication and Dependencies
- Workflow·7

### Interaction & Integration
- UI components, external connectors • 3 units

### Core Architecture and Executors
- Architecture·8

### UI Architecture and Visualization
- Architecture·6

### Security and Integration Systems
- Security·29

### Cross-Cutting Concerns
- Auth, logging, config, testing • 2 units

### Concurrency and Resource Management
- Concurrency·4

### Observability and Plugin Systems
- Infrastructure·3

### Edge Cases & Resilience
- Error handling, fault tolerance • 1 units

### Additional Data Model Patterns
- Data Model·5

### Test Your Knowledge

Test your deep understanding of the codebase.

### Start Quiz

Progress 0/26 answered

#### Question Tiers - ordered for deep understanding
- Why 8
- Purpose & Problem  Architecture 10
- Design & Patterns  Code  8
- Implementation

### Hands-On Assignment

2-3 hours

### Start Assignment

### Your Challenge

Build a custom executor that implements a "Priority Queue" execution strategy, where tasks can be assigned priority levels and higher-priority tasks are executed before lower-priority ones when resources are constrained. Your executor should integrate with Airflow's existing executor framework, respect concurrency limits, and allow priority to be set via task configuration. This will require understanding how executors interact with the scheduler, task queuing mechanisms, and the core execution loop.

### Starting Points

- `airflow-core/docs/img/diagram_basic_airflow_architecture.py:1-103`  
Study the architecture diagram to understand how executors fit into the overall system - note the interaction between Scheduler, Executor, and Workers

- `airflow/executors/base_executor.py`  
Examine the BaseExecutor class that all executors inherit from - understand the core methods like execute_async, sync, heartbeat, and the queued_tasks structure

- `airflow/executors/local_executor.py`  
Study this as a reference implementation - see how it manages task queuing, worker processes, and result handling

- `airflow/executors/sequential_executor.py`  
Look at the simplest executor implementation to understand the minimal contract you need to fulfill

- `airflow/executors/`  
Create your new PriorityQueueExecutor class here, following the naming and structure conventions of existing executors

- `airflow/models/taskinstance.py`  
Explore how task instances store metadata - you may need to understand task attributes to extract priority information

- `airflow/config_templates/default_airflow.cfg`  
See how executor configuration is handled - you'll need to add configuration options for your executor

### Success Criteria

- Executor successfully inherits from BaseExecutor and implements all required methods
- Tasks with higher priority values execute before lower priority tasks when parallelism limit is reached
- Executor respects the configured parallelism limit (doesn't exceed max concurrent tasks)
- Tasks complete successfully and state changes are properly reported back to the scheduler
- A test DAG with 10 tasks (5 high priority, 5 low priority) and parallelism=2 shows high-priority tasks completing first
- Executor can be configured via airflow.cfg and selected as the active executor
- Existing Airflow functionality remains unaffected when using standard executors

### Hints
**(click to reveal)**
- Hint 1: Understanding the executor contract conceptual
- Hint 2: Priority queue data structure implementation
- Hint 3: Respecting concurrency limits code location
- Hint 4: Task execution mechanism implementation
- Hint 5: Testing your executor implementation

**Prerequisites:**
- Python multiprocessing or subprocess
- Priority queue data structures
- Airflow executor architecture
- Task lifecycle and state management

# airflow - Learning Path