Don't we all hate them? Why do they still happen? What makes me hate them all the more is knowing how simple it is to prevent the repetitions within any reasonable time frame. (The quality and quantity of the phrases, we programmers can't help at all.)
There is more than one way to approach randomness. I don't mean the RNG (random-number generator), which is the truly involved element to develop properly. I mean there are multiple ways to apply a hopefully competent RNG. The obvious method is to select something from a list at random. And that's what even the best games seem to do. That's the problem. That's wrong, because it can lead to irritatingly frequent repetition of the same phrase, or a few phrases, and skipping over others for long periods of time.
The correct method is akin to a deck of cards. After shuffling the deck, dealing from the top produces a random series of values, but each value will appear exactly once until the deck is exhausted. Shuffling a list of items requires an algorithm, and it can arguably produce a lengthy or resource-intensive calculating process. With today's hardware, I doubt it, but let's say for the sake of argument that it is. Fortunately, there is a great shortcut.
Instead of shuffling and dealing from the top, we can deal from random places in an unshuffled deck. The only trick to this approach is removing each dealt card from the deck, and reducing the range of the RNG-generated index by one, after each card is dealt. This can be handled by a 1.79 MEGA-Hz 8-bit 6502 CPU without breaking stride. I know that for a fact, because I did it in a Cribbage game I programmed in the early 80s.
To this, I'd add a timestamp to each utterance of a phrase. If a certain amount of time hasn't elapsed since it was last heard, I'd discard it altogether. Silence is preferable to hearing the same crap over and over.
Rant comes courtesy of my increasing irritation with The Witcher 3 and Fallout 4, because of this wholly unnecessary issue. If I was responsible for the ambient dialog in these games, I'd hang my head in shame.