Fine-Tuning a Text Classifier: When a Few Examples Beat Zero-Shot

Zero-shot classification sorts text into labels it has never seen examples of, using only the labels’ names. It’s a great start, and it hits a ceiling when two labels sound alike. The fix is to show the model what you mean: label a few real examples and train a small classifier on them. A few labelled examples per label tell a model what your labels mean better than any label name can. Here you’ll train one in your browser, on real bank customer queries, and watch how few examples it takes.

The task: eight bank intents#

Banking77 is a public dataset of about 13,000 real customer messages to a bank, each labelled with one of 77 intents (what the customer wants). We took eight that come in confusable pairs: a card that hasn’t arrived and a question about when it will; a lost card and a compromised one; a declined payment and an unexpected fee; a top-up still pending and one that failed.

As a starting point, zero-shot: turn each query and each intent name into an embedding with MiniLM, a small free embedding model, and pick the intent whose name is most similar. On 200 held-out test queries it gets 79% right. Its worst confusion is the first pair: 11 queries about a card that hasn’t arrived are taken for questions about delivery times. The names alone can’t tell those apart.

The classifier here is the simplest one there is, logistic regression: for each intent it learns one weight per number in the embedding, and adds them up to score the query. The embedding model itself doesn’t change; the classifier learns which directions in its embedding space separate your labels. Averaged over three draws, accuracy climbs fast, from 70% with one example per intent to 88% with 8, and then slowly: 94% with 48. The biggest gain is on exactly the pair zero-shot got wrong. With 48 examples per intent (first draw), “card hasn’t arrived” and “when will my card arrive?” are mixed up only 5 times, and the most common mistake becomes a genuinely hard one: a top-up that’s pending taken for one that failed.

Press “Draw different examples” at low numbers and the score jumps around. With one example per intent, which example you happened to label matters a lot. With 48, all three draws score 94%.

Fine-tuning the whole model#

Training only a classifier on top leaves the embedding model as it was. Fine-tuning goes further: it also adjusts the embedding model’s own weights, about 22 million of them for MiniLM, so that the embeddings themselves start to separate your labels. It takes longer, needs a GPU for bigger models, and can overfit when there are only a few examples.

Fine-tuning the whole model helped a little once there were enough examples: at 16 per intent it averaged 93% against 92% for the classifier alone, and 95% against 94% at 32 and 48. With one or two examples per intent it did slightly worse, 69% and 79% against 70% and 81%: 22 million weights are a lot to adjust from 8 or 16 sentences. Each fine-tuning run took 15 to 30 seconds on a 4-core CPU, against well under a second for the classifier in your browser. Start with a simple classifier on good embeddings; fine-tune when the last point or two matters.

How many examples do you need?#

• Start with a few dozen per label and plot a curve like the one above. If it’s still climbing, label more; if it has flattened, more of the same won’t help much.
• Label where the model is confused. The mix-ups list tells you which pairs need more examples, or clearer definitions.
• Keep a test set the model never trains on, labelled with the same care, or every number above is wishful thinking.
• Use an LLM to draft labels, and a person to check them. It’s much faster than labelling from scratch, as long as someone catches its mistakes.

In code#

The same idea with sentence-transformers and scikit-learn, free and local (pip install sentence-transformers scikit-learn pandas). It uses a different random draw of examples and all 320 test queries for these intents, so the numbers differ a little from the figure:

import pandas as pd
from sentence_transformers import SentenceTransformer
from sklearn.linear_model import LogisticRegression

base = "https://raw.githubusercontent.com/PolyAI-LDN/task-specific-datasets/master/banking_data/"
train, test = pd.read_csv(base + "train.csv"), pd.read_csv(base + "test.csv")
intents = ["card_arrival", "card_delivery_estimate", "lost_or_stolen_card", "compromised_card",
           "declined_card_payment", "card_payment_fee_charged", "pending_top_up", "top_up_failed"]
train, test = train[train.category.isin(intents)], test[test.category.isin(intents)]

model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
X_test = model.encode(test.text.tolist(), normalize_embeddings=True)
for n in [2, 8, 32]:
    sample = train.groupby("category").sample(n, random_state=0)
    X = model.encode(sample.text.tolist(), normalize_embeddings=True)
    clf = LogisticRegression(max_iter=1000).fit(X, sample.category)
    print(f"{n:>2} per intent: {clf.score(X_test, test.category):.0%} of {len(test)} test queries right")
#  2 per intent: 75% of 320 test queries right
#  8 per intent: 85% of 320 test queries right
# 32 per intent: 90% of 320 test queries right