Learning Path

Welcome to arrow

Your guide to understanding the codebase.

Overview

15 units • 5 levels

Levels

Orchestration & APIs

  • Pipelines, workflows, public interfaces • 3 units
    • Apache Arrow Columnar Format and Type System (Architecture·7)
    • Error Handling and Utility Functions (Error Handling·2)
    • Memory Management and Array Builders (Infrastructure·3)

Core Logic & Data

  • Business rules, schemas, models • 3 units
    • Data Serialization and IPC (API·6)
    • File Formats and I/O (Data Model·9)
    • Compute Kernels and SIMD Optimizations (Workflow·18)

Interaction & Integration

  • UI components, external connectors • 3 units
    • Arrow Flight RPC and Flight SQL (API·7)
    • Extension Types and Advanced Features (Patterns·2)
    • Acero Query Engine and Dataset API (Workflow·9)

Cross-Cutting Concerns

  • Auth, logging, config, testing • 2 units
    • Build Systems, Testing, and CI/CD (Tooling·7)
    • Gandiva Expression Compiler and GPU Integration (Workflow·9)

Edge Cases & Resilience

  • Error handling, fault tolerance • 1 unit
  • Additional Data Model Patterns (Data Model·9)

Test Your Knowledge

Progress

0/26 answered

Question Tiers

  • Why: 8
  • Purpose & Problem: Architecture 10
  • Design & Patterns: Code 8
  • Implementation: Hands-On Assignment 2-3 hours

Your Challenge

Implement a custom extension type for storing geographic coordinates (latitude/longitude pairs) with validation in Apache Arrow. Your extension type should store coordinates as a FixedSizeList of two doubles, validate that latitude is between -90 and 90 and longitude is between -180 and 180, and provide serialization support so the type can be transmitted via IPC. This will require understanding Arrow's type system, array builders, and extension type registration mechanism.

Starting Points

  • c_glib/arrow-cuda-glib/cuda.cpp:1-105 Examine how C++ Arrow types are wrapped in the GLib bindings, particularly the pattern of creating wrapper classes and handling memory management.

  • c_glib/arrow-cuda-glib/arrow-cuda-glib.h:1-27 Study the header structure and include patterns used for exposing Arrow functionality through GLib.

  • c_glib/test/file-system-tests.rb:1-134 Review the testing patterns used in the codebase - you'll need similar tests for your extension type.

Success Criteria

  • Extension type correctly stores coordinate pairs as FixedSizeList<double>[2]
  • Builder validates latitude range [-90, 90] and longitude range [-180, 180], returning appropriate errors for invalid values
  • Type can be serialized to IPC format and deserialized while preserving metadata
  • At least 5 unit tests covering valid data, invalid data, serialization, and integration with Arrow arrays
  • Type can be registered globally and retrieved by name
  • Documentation comments explain the coordinate system and validation rules.

Hints

  • Hint 1: Understanding Extension Types conceptual
  • Hint 2: Array Builder Pattern code location
  • Hint 3: Registration and Serialization implementation
  • Hint 4: Testing Your Implementation implementation

Prerequisites

  • C++ classes and inheritance
  • Arrow type system basics
  • Memory management with shared_ptr
  • Basic understanding of coordinate systems