Prompt Compression Techniques — Reduce Token Count Without Losing Quality

1. Remove filler language

"I want you to act as a helpful assistant who specializes in..." → "You are a helpful assistant specializing in...". "Please make sure to" → "Always". Every filler phrase adds tokens with no instruction signal.

2. Convert prose instructions to bullets

Prose: "When the user asks a question, you should first think about what they are trying to accomplish, and then provide a concise answer..." (40 tokens). Bullets: "Think about user intent. Answer concisely." (8 tokens). 5× compression.

3. Replace few-shot examples with function signatures

Instead of 3 full input/output examples (1,500 tokens), use a JSON schema showing the expected output structure (200 tokens) + 1 example (500 tokens). Total: 700 tokens vs 1,500.

4. LLMLingua and similar compression models

LLMLingua, LLMLingua-2, and LongLLMLingua are small models that remove low-information tokens from prompts. Achievable compression: 2–5× on verbose documents with <5% quality loss. Works best on retrieved document contexts, not system prompts.

5. Abbreviate your terminology consistently

Define abbreviations in the system prompt: "Use 'usr' for user, 'req' for request, 'resp' for response." Then use abbreviations throughout. On prompts sent millions of times, even small compressions compound.