📊 Information Theory • Shannon Entropy • Lexical Density • Boilerplate Elimination
Prompt Token Entropy & Lexical Diversity Studio
Quantify the information density of your system prompts. Detect repetitive padding words, compute character-level Shannon entropy, and measure Type-Token Ratio (TTR).
System Prompt Text
💡 Shannon Entropy Interpretation
English prose typically measures between 3.8 and 4.4 bits per character. Values below 3.5 indicate severe keyword redundancy, while values above 4.5 signify dense technical instructions or code.
Information Density Scorecard
Low Density (Repetitive)Shannon Entropy
3.42 bits
Lexical TTR
52%
Unique Tokens
21 / 40
Top Repeated Token Candidates for Pruning
Engineering Recommendation: