acwa-trainer

Adaptive Class Weight Adjustment (ACWA) - Automated Class Balancing for Deep Learning

📖 Table of Contents

• Overview
• Key Features
• Algorithm Design
• When to Use ACWA
• Implementation Guide
• Usage for Different Datasets
• FAQ
• Best Practices
• Benchmark Results
• Contributing

🌟 Overview

ACWA is an advanced optimization algorithm designed to automatically adjust class weights during neural network training, particularly effective for imbalanced datasets. Unlike traditional approaches, ACWA dynamically adapts based on real-time performance metrics.

Traditional Methods Limitations:

• Static class weighting based on frequency
• Manual oversampling/undersampling
• Fixed cost-sensitive learning

ACWA Advantages:

• 🚀 Real-time performance monitoring
• ⚖️ Dynamic weight adjustment
• 🎯 Focus on underperforming classes
• 🤖 No manual intervention needed

✨ Key Features

• Adaptive Learning: Adjusts weights based on validation performance
• Smoothing Mechanism: Prevents drastic weight fluctuations
• Multi-class Support: Works with any number of classes
• Framework Agnostic: Compatible with PyTorch, TensorFlow, etc.
• Plug-and-Play: Easy integration into existing pipelines
• TorchMetrics Integration: Efficient F1-score calculation
• Dynamic Weight Initialization: Supports inverse class frequency
• Early Stopping: Prevents overfitting by monitoring validation performance
• Numerical Stability: Epsilon added to class frequency for robust weight initialization

🧠 Algorithm Design

Core Concept

ACWA operates through a feedback loop:

1. Monitor class-wise performance
2. Calculate performance gaps
3. Adjust weights dynamically

Mathematical Formulation

Performance Error:

error_c = target\_metric - current\_metric_c

Weight Update:

weight_c^{(t+1)} = clip(\beta \cdot weight_c^{(t)} + (1-\beta) \cdot (weight_c^{(t)} + \alpha \cdot error_c), 0.5, 2.0)

Loss Modification:

\mathcal{L} = \sum_{c=1}^C weight_c \cdot \mathcal{L}_c

Hyperparameters

🏆 When to Use ACWA

Ideal Scenarios

• 🏥 Medical diagnosis (rare disease detection)
• 💳 Fraud detection
• ⚠️ Rare event prediction
• 🛡️ Anomaly detection
• 📊 Highly imbalanced datasets

Comparison with Alternatives

💻 Implementation Guide

Installation

pip install acwa-torch

Basic Usage

from acwa import ACWATrainer

# Initialize
trainer = ACWATrainer(
    num_classes=10,
    target_metric=0.85,  # Target F1-score
    alpha=0.02,
    beta=0.9,
    update_freq=100
)

# Training loop
for batch in dataloader:
    # Forward pass
    outputs = model(inputs)
    
    # ACWA-weighted loss
    loss = trainer.get_weighted_loss(outputs, labels)
    
    # Backward pass
    loss.backward()
    optimizer.step()
    
    # Update metrics
    trainer.update_metrics(outputs, labels)

Advanced Features

# Custom metrics
trainer = ACWATrainer(
    metric_fn=custom_f1_function,
    metric_mode='max'  # or 'min'
)

# Combined with Focal Loss
trainer = ACWATrainer(
    loss_fn=FocalLoss(gamma=2.0),
    ...
)

# Initialize weights using inverse class frequency
class_counts = torch.bincount(torch.tensor(trainset.targets))
class_frequencies = class_counts.float() / (class_counts.sum() + 1e-6)

trainer = ACWATrainer(
    model=model,
    num_classes=10,
    class_frequencies=class_frequencies
)

# Early stopping example
best_f1 = 0
early_stop_counter = 0
patience = 5

for epoch in range(num_epochs):
    # ...training logic...
    if val_f1 > best_f1:
        best_f1 = val_f1
        torch.save(model.state_dict(), 'best_model.pth')
        early_stop_counter = 0
    else:
        early_stop_counter += 1

    if early_stop_counter >= patience:
        print("Early stopping triggered.")
        break

📚 Usage for Different Datasets

Vision Task (CIFAR-100)

1. Prepare the dataset:

   from demo_cifar100 import main
   main()
   

NLP Task (IMDB with Hugging Face)

1. Use Hugging Face's datasets library to load the IMDB dataset:

   from demo_huggingface import main
   main()
   

Time-Series Task (UCR)

1. Load UCR dataset using tslearn or a similar library.

❓ FAQ

1. How to integrate Hugging Face models?

   • Use the transformers library and wrap the model with ACWATrainer.

2. What if F1-score does not improve?

   • Ensure update_freq is set appropriately (20-50).
   • Check if the dataset is properly preprocessed.

📝 Best Practices

1. Validation Set: Ensure representative distribution
2. Initial Weights: Start with uniform weights (1.0)
3. Hyperparameter Tuning:
   • Start with α=0.01, β=0.9
   • Adjust based on convergence
4. Monitoring: Track weight evolution during training
5. Combination Strategies:
   • Works well with data augmentation
   • Can be combined with focal loss

# Example weight evolution plot
plt.plot(weight_history)
plt.title('ACWA Weight Adjustment')
plt.xlabel('Update Steps')
plt.ylabel('Class Weight')
plt.show()

🏅 Benchmark Results

CIFAR-10 (Imbalanced)

🤝 Contributing

We welcome contributions! Please see our:

• Contribution Guidelines
• Code of Conduct

Future Improvements

1. Unit Testing:

   • Add test cases for edge scenarios (e.g., empty classes, small batch sizes).
   • Ensure compatibility with various datasets and imbalance ratios.

2. Distributed Training:

   • Implement support for multi-GPU setups using torch.nn.parallel.DistributedDataParallel.
   • Synchronize metrics across GPUs for consistent weight updates.

3. Additional Frameworks:

   • Extend support to TensorFlow/Keras for broader adoption.

📜 License

MIT License - Free for academic and commercial use