Image by Editor | ChatGPT# Introduction
Hugging Face has become the standard for many AI developers and data scientists because it drastically lowers the barrier to working with advanced AI. Rather than working with AI models from scratch, developers can access a wide range of pretrained models without hassle. Users can also adapt these models with custom datasets and deploy them quickly.
One of the Hugging Face framework API wrappers is the Transformers Pipelines, a series of packages that consists of the pretrained model, its tokenizer, pre- and post-processing, and related components to make an AI use case work. These pipelines abstract complex code and provide a simple, seamless API.
However, working with Transformers Pipelines can get messy and may not yield an optimal pipeline. That is why we will explore five different ways you can optimize your Transformers Pipelines.
Let’s get into it.
# 1. Batch Inference Requests
Often, when using Transformers Pipelines, we do not fully utilize the graphics processing unit (GPU). Batch processing of multiple inputs can significantly boost GPU utilization and enhance inference efficiency.
Instead of processing one sample at a time, you can use the pipeline’s batch_size parameter or pass a list of inputs so the model processes several inputs in one forward pass. Here is a code example:
from transformers import pipeline
pipe = pipeline(
task="text-classification",
model="distilbert-base-uncased-finetuned-sst-2-english",
device_map="auto"
)
texts = [
"Great product and fast delivery!",
"The UI is confusing and slow.",
"Support resolved my issue quickly.",
"Not worth the price."
]
results = pipe(texts, batch_size=16, truncation=True, padding=True)
for r in results:
print(r)
By batching requests, you can achieve higher throughput with only a minimal impact on latency.
# 2. Use Lower Precision And Quantization
Many pretrained models fail at inference because development and production environments do not have enough memory. Lower numerical precision helps reduce memory usage and speeds up inference without sacrificing much accuracy.
For example, here is how to use half precision on the GPU in a Transformers Pipeline:
import torch
from transformers import AutoModelForSequenceClassification
model = AutoModelForSequenceClassification.from_pretrained(
model_id,
torch_dtype=torch.float16
)
Similarly, quantization techniques can compress model weights without noticeably degrading performance:
# Requires bitsandbytes for 8-bit quantization
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained(
model_id,
load_in_8bit=True,
device_map="auto"
)
Using lower precision and quantization in production usually speeds up pipelines and reduces memory use without significantly impacting model accuracy.
# 3. Select Efficient Model Architectures
In many applications, you do not need the largest model to solve the task. Selecting a lighter transformer architecture, such as a distilled model, often yields better latency and throughput with an acceptable accuracy trade-off.
Compact models or distilled versions, such as DistilBERT, retain most of the original model’s accuracy but with far fewer parameters, resulting in faster inference.
Choose a model whose architecture is optimized for inference and suits your task’s accuracy requirements.
# 4. Leverage Caching
Many systems waste compute by repeating expensive work. Caching can significantly enhance performance by reusing the results of costly computations.
with torch.inference_mode():
output_ids = model.generate(
**inputs,
max_new_tokens=120,
do_sample=False,
use_cache=True
)
Efficient caching reduces computation time and improves response times, lowering latency in production systems.
# 5. Use An Accelerated Runtime Via Optimum (ONNX Runtime)
Many pipelines run in a PyTorch not-so-optimal mode, which adds Python overhead and extra memory copies. Using Optimum with Open Neural Network Exchange (ONNX) Runtime — via ONNX Runtime — converts the model to a static graph and fuses operations, so the runtime can use faster kernels on a central processing unit (CPU) or GPU with less overhead. The result is usually faster inference, especially on CPU or mixed hardware, without changing how you call the pipeline.
Install the required packages with:
pip install -U transformers optimum[onnxruntime] onnxruntime
Then, convert the model with code like this:
from optimum.onnxruntime import ORTModelForSequenceClassification
ort_model = ORTModelForSequenceClassification.from_pretrained(
model_id,
from_transformers=True
)
By converting the pipeline to ONNX Runtime through Optimum, you can keep your existing pipeline code while getting lower latency and more efficient inference.
# Wrapping Up
Transformers Pipelines is an API wrapper in the Hugging Face framework that facilitates AI application development by condensing complex code into simpler interfaces. In this article, we explored five tips to optimize Hugging Face Transformers Pipelines, from batch inference requests, to selecting efficient model architectures, to leveraging caching and beyond.
I hope this has helped!
Cornellius Yudha Wijaya is a data science assistant manager and data writer. While working full-time at Allianz Indonesia, he loves to share Python and data tips via social media and writing media. Cornellius writes on a variety of AI and machine learning topics.
