> Behrooz, Morteza, et al. [“Story Quality as a Matter of Perception: Using Word Embeddings to Estimate Cognitive Interest.”](https://ojs.aaai.org/index.php/AIIDE/article/view/5217) _Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment_, vol. 15, no. 1, 1, Oct. 2019, pp. 3–9. # Story Quality as a Matter of Perception: Using Word Embeddings to Estimate Cognitive Interest > Storytelling is a capable tool for interactive agents and better stories can enable better interactions. Many existing automated evaluation techniques are either focused on textual features that are not necessarily reflective of perceived interestingness (e.g. coherence), or are domain-specific, relying on a priori semantics models (e.g. in a game). However, the effectiveness of storytelling depends both on its versatility to adapt to new domains and the perceived interestingness of its generated stories. In this paper, drawing from cognitive science literature, we propose and evaluate a method for estimating cognitive interest in stories based on the level of predictive inference they cause during perception. ## Method Given a short story with stop words and named entities removed, they extract BERT embeddings for each word occurrence. To simulate the linear perception of the narrative, they compute _moving cosine similarity_ for each word $w_i$: $ \mathrm{sim}(w_i) = \cos \left( \mathrm{mean}\left( \left[w_b, \dots, w_{i-1} \right] \right), w_i \right) $ where $w_b$ is the first word of the second sentence of the story. The assumption is that foreshadowing consists of two parts, contained in two sentences: a hint and a twist. The words which are central to those parts should show anormally low movine cosine similarity and can be detected with outlier detection algorithms. They move the threshold of detection until the outlier set consists only in words from two sentences in the story. The metric $M$ that is computed is supposed to reflect this anomaly threshold: 1. If no such threshold is found, $M=0$; 2. Otherwise, $ M = \frac{A + |B|}{3} \in [0,1] $ where $ A = 1 - \cos(\mathrm{mean}_\mathrm{all words}, \mathrm{mean}_\mathrm{outliers}) \in [0,2] $ and $ B = 1 - \underset{i \in \mathrm{outliers}}{\mathrm{mean}}(\cos(w_i, \mathrm{mean}(\mathrm{\{outliers\}} - w_i)) \in [0,1] $ $A$ represents a measure of how anomalous the outliers (hint and twist) are compared to the rest of the text, and $B$ represents how related the outliers are with one another. $M$ is not a probability so $M=1$ does not bear any special significance.