# High-quality generated text has negative-log probability close to average human text entropy Language models generate text according to a probabilistic distribution obtained through training. However, the most probable text is usually not the best; while probability and quality are at first positively correlated, there is a threshold after which the correlation breaks. If we developed language for the efficient transfer of information, natural language strings should contain a typical amount of information. Therefore, **the average amount of information per string should be relatively constant**: text with significantly more or less information would be respectively hard to process or boring. Meister et al.[^1] show that this is indeed the case: ==**highly-rated generated strings and human reference strings have an entropy that is within 1 standard-deviation of the model's entropy**== (they use a fine-tuned GPT-2 and assume it has entropy close to natural language), while strings with outlier entropy score significantly worse. --- ## 📚 References [^1]: Meister, Clara, et al. [On the Probability-Quality Paradox in Language Generation.](https://doi.org/10.48550/arXiv.2203.17217) arXiv:2203.17217, arXiv, 31 Mar. 2022. arXiv.org.