Welcome to transformers

Your guide to understanding the codebase

Learning Path

5 levels
15 learning units

Orchestration & APIs

  • Pipelines, workflows, public interfaces • 3 units
    • Pipeline Abstraction and Basic Usage
    • Model Configuration and Auto Classes
    • Tokenization and Data Processing

Core Logic & Data

  • Business rules, schemas, models • 3 units
    • Transformer Architecture Fundamentals
    • Feature Extraction and Generation
    • Training Framework and Callbacks

Interaction & Integration

  • UI components, external connectors • 3 units
    • Multimodal and Specialized Architectures
    • Distributed Training and Model Sharding
    • Model Hub and Integration Ecosystem

Cross-Cutting Concerns

  • Auth, logging, config, testing • 2 units
    • Model Optimization and Quantization
    • Testing, Benchmarking, and Deployment

Edge Cases & Resilience

  • Error handling, fault tolerance • 1 unit
    • Additional Data Model Patterns

Test Your Knowledge

24 questions across all levels

Start Quiz

Progress
0/24 answered

Hands-On Assignment

2-3 hours

Your Challenge

Implement a custom tokenizer that supports a new special token type called 'ENTITY' tokens for named entity recognition tasks. Your tokenizer should extend the existing PreTrainedTokenizer class, handle entity markers (e.g., <ENTITY:PERSON>, <ENTITY:ORG>) during encoding/decoding, and properly integrate with the AutoTokenizer registry so it can be loaded via AutoTokenizer.from_pretrained(). The implementation should maintain compatibility with existing tokenization pipelines while adding entity-aware vocabulary management.

Starting Points

  • src/transformers/tokenization_utils_base.py
    Explore the PreTrainedTokenizerBase class to understand the core tokenization interface, special token handling, and encoding/decoding methods

  • src/transformers/tokenization_utils.py
    Study PreTrainedTokenizer implementation - this is the class you'll extend for your custom tokenizer

  • src/transformers/models/bert/tokenization_bert.py
    Reference this as a concrete example of how tokenizers are implemented, including __init__, vocab loading, and special methods

  • src/transformers/models/auto/tokenization_auto.py
    Understand how AutoTokenizer registry works - you'll need to register your custom tokenizer here

  • src/transformers/models/
    Create a new directory here for your custom tokenizer implementation (e.g., 'entity_aware/')

  • tests/models/bert/test_tokenization_bert.py
    Review tokenizer testing patterns - you'll create similar tests for your implementation

Success Criteria

  • Your tokenizer can encode text with entity markers like 'John ENTITY:PERSON works at Google ENTITY:ORG' and preserve entity information
  • Decoding returns the original text with entity markers intact
  • The tokenizer can be saved with save_pretrained() and loaded with AutoTokenizer.from_pretrained()
  • Entity type vocabulary persists across save/load cycles
  • At least 5 unit tests pass covering: basic tokenization, special token handling, save/load, entity type management, and edge cases
  • The tokenizer works with a simple text classification pipeline without errors

Hints

  • (click to reveal)
    • Hint 1: Understanding special tokens architecture conceptual
    • Hint 2: Key methods to override code location
    • Hint 3: Registry integration pattern implementation
    • Hint 4: Handling dynamic entity types implementation
    • Hint 5: Testing your implementation implementation

Prerequisites

  • Python class inheritance
  • Tokenization concepts (vocab, encoding, decoding)
  • JSON serialization
  • Basic understanding of transformer model inputs