activations-plus

Activations Plus

Activations Plus is a Python package designed to provide a collection of advanced activation functions for machine learning and deep learning models. These activation functions are implemented to enhance the performance of neural networks by addressing specific challenges such as sparsity, non-linearity, and gradient flow.

Features

• Entmax: Sparse activation function for probabilistic models.
• Sparsemax: Sparse alternative to softmax.
• Bent Identity: Smooth approximation of the identity function. (Experimental feature require review)
• ELiSH (Exponential Linear Squared Hyperbolic): Combines exponential and linear properties. (Experimental feature require review)
• Maxout: Learns piecewise linear functions. (Experimental feature require review)
• Soft Clipping: Smoothly clips values to a range. (Experimental feature require review)
• SReLU (S-shaped Rectified Linear Unit): Combines linear and non-linear properties. (Experimental feature require review)

Installation

To install the package, use pip:

pip install activations-plus

Usage

Import and use any activation function in your PyTorch models:

import torch
from activations_plus.sparsemax import Sparsemax
from activations_plus.entmax import Entmax

# Example with Sparsemax
sparsemax = Sparsemax()
x = torch.tensor([[1.0, 2.0, 3.0], [1.0, 2.0, -1.0]])
output_sparsemax = sparsemax(x)
print("Sparsemax Output:", output_sparsemax)

# Example with Entmax
entmax = Entmax(alpha=1.5)
output_entmax = entmax(x)
print("Entmax Output:", output_entmax)

These examples demonstrate how to use Sparsemax and Entmax activation functions in PyTorch models.

Documentation

Comprehensive documentation is available documentation.

Supported Activation Functions

1. Entmax: Sparse activation function for probabilistic models. Reference Paper
2. Sparsemax: Sparse alternative to softmax for probabilistic outputs. Reference Paper
3. Bent Identity: A smooth approximation of the identity function. (Experimental feature require review) reference missing
4. ELiSH: Combines exponential and linear properties for better gradient flow. (Experimental feature require review) Reference Paper
5. Maxout: Learns piecewise linear functions for better expressiveness. (Experimental feature require review) Reference Paper
6. Soft Clipping: Smoothly clips values to a range to avoid extreme outputs. (Experimental feature require review) Reference Paper
7. SReLU: Combines linear and non-linear properties for better flexibility. (Experimental feature require review) Reference Paper

Contributing

Contributions are welcome! Please read the CONTRIBUTING.md file for guidelines.

Testing

To run the tests, use the following command:

pytest tests/

License

This project is licensed under the MIT License. See the LICENSE file for details.

Acknowledgments

Special thanks to the contributors and the open-source community for their support.