What 640 London shopping streets, 4,000 Billboard hits, a million films and synthetic AI personas tell us about why everything is flattening into sameness and how to regulate the algorithms behind it.
This is a more sophisticated version of a model I made for why older people are less imaginative. (At 78, I am admitting this is a real thing.) It uses the image of searching for car keys. Think of the places you search for the keys as being on a search tree that you keep optimizing. At some depth in the tree it pays to go back and search places already visited; as the tree gets better that depth gets shallower and shallower. At some point you find yourself searching the same few places over and over. And that often does work ...
I love this and I think that the car keys version is really great, and it captures the mechanism exactly. As the search tree gets “better,” the depth at which you revisit known spots keeps getting shallower, until you’re just circling the same few locations. That’s precisely the flattening: that a well-optimised system keeps telling you not to bother rather than exploration stopping to work.
This is great! I’ve always had a dim sense that evolutionary dynamics shape a lot of the world, and this was the sort of thing I tended to have in mind when I thought that.
This might be a dense thing to say, but: I don’t see why there’s any reason to expect this dynamic to be stable. It seems to me that it would collapse with no intervention at all, although intervention would be a happier way for it to disappear. If there’s an algorithmic optimiser feeding on no new information that’s stuck in a local optimum, then I don’t see how it adapts to any dramatic change. A monoculture collapse is obviously bad when the monoculture is the food which we need to eat, but I don’t know that it’s as bad a thing in this case? I might be missing an obvious way that it is.
Also, this is unrelated, but: I suspected that this dynamic was the real reason AI writing is flattened. It can’t take risks in case it produces some wildly unfortunate style, hedges on generic “good style,” then finds “good style” is no longer that good because its overuse has affected how it’s judged. But there, again, homogeneity doesn’t really resolve anything because the homogeneity undermines its own conditions? You are clearly very smart and I’m concerned that these thoughts might be stupid
That it collapses on its own is certainly true but slow, and the damage is also done in the meantime. The high-street or Billboard analogues don’t wait for a dramatic exogenous shock to reveal the fragility, so diversity is lost long before the monoculture visibly fails, and it doesn’t automatically regenerate afterwards.
On AI writing, yes, I think you have an excellent point, risk-averse optimisation toward generic “good style,” then good style decays because ubiquity changes how it’s judged. The self-undermining part is exactly the point, homogeneity is a process that eats its own reference point. Which is why I don’t think it ever actually resolves, it just keeps chasing a moving target it’s degrading by chasing.
The randomness in the Generative AI models ('temperature') can be set, but it is a single 'randomness' (on selecting the next token). If you turn the 'temperature' up, everything starts to 'variate' concurrently, and that includes for instance grammar. So OpenAI carefully capped temperature at a level that the grammar would remain good, because people's 'pattern/mental shortcut'-based psychology take 'good language' as a shortcut to trustworthiness. In 2023 it was still possible possible to illustrate this clearly (https://youtu.be/9Q3R8G_W0Wc?t=1004 about 2.5 minutes. Note: I am still using the big 'Lie to Children' about LLMs at that point in the presentation). I expect that these days the models have separate 'experts' running on grammar/linguistics alone, which means the effect is hidden (and it is of course a hideously inefficient trial-and-error setup, but that is another story).
Really great piece, beautifully written and I learned a lot. Great point also in your comment that diversity is lost long before monoculture collapses and the we may see catastrophic consequences. I need to think about what this means for Software Engineering.
Excellent column. As someone who is involved in retail and market research, across different markets , you hit the nail on the head. We find we look at large amounts of data, in an attempt to ' boil' this down to answer questions we have - whether in catchment demographics or store location or even assortment planning.
Great article, and I agree with the central thesis that "personalisation" algorithms are in fact acting as the precise opposite of what they promise. The tricky thing is the policy response. Recommenders not based on profiling are even more powerful mean-reversion machines, and if you mandate an industry to recommend e.g. 20% diversity, the next obvious (and non-trivial) question is how that 20% is selected. If purely random, its feed will get clogged with poor quality stuff, if not, then some form of judgement will need to take place and we're back to square one.
Someone else mentioned incentive based interventions and I thing they are more promising. You could measure the variety that a company or service introduces into the system (with e.g. an entropy-based metric) and reward it maybe? The article made e think of Walmart vs independent shops. Walmart got to define its range and many suppliers lived or died by their inclusion. However small independent retailers survive - people know and appreciate quirky, alternative or specialised things. Is the problem with the digitial sphere that it is so much more concentrated, and the biggest players have even more power than Walmart? Just some thoughts - thanks again for a great article and well done for getting it picked up so widely!
Thanks, Jack and really appreciate your comment! The policy response is difficult, and the diversity-mandate route runs straight into the selection problem you describe: any metric you pick to fill that 20% is itself a judgement and once it’s mandated it becomes a market-shaping instrument. Whoever defines “diversity” or “quality” is now setting the terms competitors have to optimise against, which is a purely political decision that reshapes the market.
The entropy-based reward is a clever one, and I think it’s more promising than quotas precisely because it targets the behaviour (introducing variety) rather than dictating the content. But it doesn’t escape the underlying issue: you still have to decide what counts as variety, at what granularity, and gaming an entropy metric is trivial. So the judgement doesn’t disappear.
I think your Walmart vs independents point is nice. The physical retail world sustained quirky specialists because concentration was bounded and thus there was room in the geography for both. The digital problem is that they operate at a concentration Walmart never had, so the same dynamic produces a far more totalising outcome. Which makes me now thinking about it again lean toward interventions that constrain concentration and power rather than ones that try to legislate the feed’s contents directly.
Thanks Lauren – yes concentration- and market-abuse-limiting regulation makes sense to me too. I do think appropriately regulated but still relatively "free" markets should reward companies or services that can offer variety and quirkiness, as many of us like seeking out the different and unique. The problem arises when the big flatteners are so dominant that they starve the oxygen from the quirk shops!
I think another factor is the advertising model of the internet. In the physical world increasing the volume of a signal requires exponentially more energy. For example if a bird wanted to double the range of its making call it would need 8 times the energy to produce that. This limits the ability of a single organism to saturate its environment. The inhibitory feedback limits one organism and allows room for variability.
In the digital world as it is constructed now it takes 0 cost to increase the volume of a signal. And because of the advertising model of clicks leading to revenue, the digital organism (web site for example) gains energy the greater the range of its signal. Once a digital organism starts to gain attention, it gains more energy and will saturate its environment. This closes out other digital organisms.
Until we find a way to place inhibitory feedback into the growth of digital organisms I think we are going to be stuck with a few organisms that fill the environment and leave little space for variations to arise.
Great article! We are trapping ourselves in algorithmic Skinner Boxes, not sandboxes. We are begging the question: What happens when the Skinner Box doesn’t just train you to respond according to certain rules in order to get pellets you already consider to be rewards, but in doing so, and because of the manner in which it is done, ALSO manages to train you over time on what pellet attributes to consider rewarding?
Unwanted flattening of culture is part of the reason we decided to build an open, controllable recommender system. Now available for BlueSky. You can have any amount of diversity you want.
Thank you for such a smart exposition that causes one to think about these issues. I reflect on my own behavior when looking for new restaurants. The 5 star rating system reinforces early positive reviews that create a positive feed back loop and who then looks at restaurants rated below 4? I have found that when i make a comment on a Substack article that is somewhat intelligent and I do it right away, i get likes that build on themselves and have gotten over 100 likes. I think people dont scroll down and see others. Like being on page 1 in a google search. Who goes beyond page 2 at best? I have found that the search for the interesting and the excellent lies in the tails whether it be food, music, investments but whether it is society in general or LLM's, which seem to be merging into a post human hybrid, how do we create a positive rate of return on the new, different, innovative. It is ironic that while there has been much innovation in the creation of LLM's, which have their limitations, such innovation then creates such sameness. your suggestion that answer lied in regulation does not encourage me. Can Europe remain an island? Do not see it in the USA as a possibility today. While there is a clear rate of return on as you postulate having 20% on new stuff, the pure short term profit maximizers or would be monopolizers have a cost or a yield advantage over those that invest in the new. People ignore the risks of monoculture until its too late. Look at bananas and avocadoes eg. anyway you are a delight to read.
Thanks a lot Bruce, appreciate it! You might also be interested in the Google Maps piece I wrote a while back where I try to identify and correct for this rating system loop created by google maps. Europeans just love their regulation:) but you are right it’s difficult to remain an island, but that having said, this great book the Brussels effect shows how EU is quite a bit with its regulations shaping the world.
Got here via Cory Doctorow. Great read. The Big Data Analytics variant of AI carries this risk strongly. Generative AI has some randomness built in its fundamental engine, but it is far too weak to escape this trap, it is foam on a current.
I was reminded of two things. 1. The brain seems to have parts operating at 'the edge of chaos'. Two things probably require this: efficiency and variability of its operation. 2. Large organisations with complex IT have become aware of that only building directly functional results gets you trapped in a mess. Reserving 20% for the work that has no direct functional effect ('architecture') is important for th medium to long term. Of course, it depends on the quality of management to keep the discipline of taking that space.
I've been calling this phenomenon "the tyranny of the market."
When our city had a population of 125,000 there were at least a dozen small auto parts stores in addition to a couple of major chains. Now with more than double the population, small stores no longer exist and the chains have mostly converted to shop services backstopped by housewares. As far as I can tell, there are still a good number of people trying to do their own work, but it's just not feasible for most people to take their vehicle out of commission for a week or two waiting for a parts order.
You might enjoy Patrick Deneen’s book, Why Liberalism Failed, especially the part on anti-culture.
Maybe I’m naive, but are algorithmic recommenders really to blame for cultural sameness? That does seem to assume that decision-makers really follow verbatim what algorithms suggest. Hasn’t culture been converging long before algorithmically curated content and decision tools? Now just accelerated?
I think a culture that values comfort and is risk-avoidant might prefer the algorithms. They get spared the uncertainty of exploration. Not that they’re necessarily better off, but it is easy, and heaven forbid I have to engage in the messy art of choosing.
Finally, instead of regulating (which I thought was a strong take), what do you think about subsidizing entrepreneurs (your exploration) instead of “taxing” exploiters?
Looking at your results, it seems like the algorithms are still not that well optimized. Like, if I watch one Scandinavian crime drama, that doesn't mean I want that genre forever, but the algorithm often treats it as a stable signal. It is likely that I would rewatch something similar, but this probably varies a lot from user to user, and it wouldn't be hard to check who's actually a same genre viewer and who isn't. Similar logic probably applies to restaurants and music too. The most likely bet is the safest choice for the algorithm, but that doesn't mean it's the choice that actually makes the user happy.
This is a more sophisticated version of a model I made for why older people are less imaginative. (At 78, I am admitting this is a real thing.) It uses the image of searching for car keys. Think of the places you search for the keys as being on a search tree that you keep optimizing. At some depth in the tree it pays to go back and search places already visited; as the tree gets better that depth gets shallower and shallower. At some point you find yourself searching the same few places over and over. And that often does work ...
I love this and I think that the car keys version is really great, and it captures the mechanism exactly. As the search tree gets “better,” the depth at which you revisit known spots keeps getting shallower, until you’re just circling the same few locations. That’s precisely the flattening: that a well-optimised system keeps telling you not to bother rather than exploration stopping to work.
This is great! I’ve always had a dim sense that evolutionary dynamics shape a lot of the world, and this was the sort of thing I tended to have in mind when I thought that.
This might be a dense thing to say, but: I don’t see why there’s any reason to expect this dynamic to be stable. It seems to me that it would collapse with no intervention at all, although intervention would be a happier way for it to disappear. If there’s an algorithmic optimiser feeding on no new information that’s stuck in a local optimum, then I don’t see how it adapts to any dramatic change. A monoculture collapse is obviously bad when the monoculture is the food which we need to eat, but I don’t know that it’s as bad a thing in this case? I might be missing an obvious way that it is.
Also, this is unrelated, but: I suspected that this dynamic was the real reason AI writing is flattened. It can’t take risks in case it produces some wildly unfortunate style, hedges on generic “good style,” then finds “good style” is no longer that good because its overuse has affected how it’s judged. But there, again, homogeneity doesn’t really resolve anything because the homogeneity undermines its own conditions? You are clearly very smart and I’m concerned that these thoughts might be stupid
These are really great points!
That it collapses on its own is certainly true but slow, and the damage is also done in the meantime. The high-street or Billboard analogues don’t wait for a dramatic exogenous shock to reveal the fragility, so diversity is lost long before the monoculture visibly fails, and it doesn’t automatically regenerate afterwards.
On AI writing, yes, I think you have an excellent point, risk-averse optimisation toward generic “good style,” then good style decays because ubiquity changes how it’s judged. The self-undermining part is exactly the point, homogeneity is a process that eats its own reference point. Which is why I don’t think it ever actually resolves, it just keeps chasing a moving target it’s degrading by chasing.
The randomness in the Generative AI models ('temperature') can be set, but it is a single 'randomness' (on selecting the next token). If you turn the 'temperature' up, everything starts to 'variate' concurrently, and that includes for instance grammar. So OpenAI carefully capped temperature at a level that the grammar would remain good, because people's 'pattern/mental shortcut'-based psychology take 'good language' as a shortcut to trustworthiness. In 2023 it was still possible possible to illustrate this clearly (https://youtu.be/9Q3R8G_W0Wc?t=1004 about 2.5 minutes. Note: I am still using the big 'Lie to Children' about LLMs at that point in the presentation). I expect that these days the models have separate 'experts' running on grammar/linguistics alone, which means the effect is hidden (and it is of course a hideously inefficient trial-and-error setup, but that is another story).
Really great piece, beautifully written and I learned a lot. Great point also in your comment that diversity is lost long before monoculture collapses and the we may see catastrophic consequences. I need to think about what this means for Software Engineering.
Thanks Robert, appreciate it!
Excellent column. As someone who is involved in retail and market research, across different markets , you hit the nail on the head. We find we look at large amounts of data, in an attempt to ' boil' this down to answer questions we have - whether in catchment demographics or store location or even assortment planning.
Great article, and I agree with the central thesis that "personalisation" algorithms are in fact acting as the precise opposite of what they promise. The tricky thing is the policy response. Recommenders not based on profiling are even more powerful mean-reversion machines, and if you mandate an industry to recommend e.g. 20% diversity, the next obvious (and non-trivial) question is how that 20% is selected. If purely random, its feed will get clogged with poor quality stuff, if not, then some form of judgement will need to take place and we're back to square one.
Someone else mentioned incentive based interventions and I thing they are more promising. You could measure the variety that a company or service introduces into the system (with e.g. an entropy-based metric) and reward it maybe? The article made e think of Walmart vs independent shops. Walmart got to define its range and many suppliers lived or died by their inclusion. However small independent retailers survive - people know and appreciate quirky, alternative or specialised things. Is the problem with the digitial sphere that it is so much more concentrated, and the biggest players have even more power than Walmart? Just some thoughts - thanks again for a great article and well done for getting it picked up so widely!
Thanks, Jack and really appreciate your comment! The policy response is difficult, and the diversity-mandate route runs straight into the selection problem you describe: any metric you pick to fill that 20% is itself a judgement and once it’s mandated it becomes a market-shaping instrument. Whoever defines “diversity” or “quality” is now setting the terms competitors have to optimise against, which is a purely political decision that reshapes the market.
The entropy-based reward is a clever one, and I think it’s more promising than quotas precisely because it targets the behaviour (introducing variety) rather than dictating the content. But it doesn’t escape the underlying issue: you still have to decide what counts as variety, at what granularity, and gaming an entropy metric is trivial. So the judgement doesn’t disappear.
I think your Walmart vs independents point is nice. The physical retail world sustained quirky specialists because concentration was bounded and thus there was room in the geography for both. The digital problem is that they operate at a concentration Walmart never had, so the same dynamic produces a far more totalising outcome. Which makes me now thinking about it again lean toward interventions that constrain concentration and power rather than ones that try to legislate the feed’s contents directly.
Thanks Lauren – yes concentration- and market-abuse-limiting regulation makes sense to me too. I do think appropriately regulated but still relatively "free" markets should reward companies or services that can offer variety and quirkiness, as many of us like seeking out the different and unique. The problem arises when the big flatteners are so dominant that they starve the oxygen from the quirk shops!
I think another factor is the advertising model of the internet. In the physical world increasing the volume of a signal requires exponentially more energy. For example if a bird wanted to double the range of its making call it would need 8 times the energy to produce that. This limits the ability of a single organism to saturate its environment. The inhibitory feedback limits one organism and allows room for variability.
In the digital world as it is constructed now it takes 0 cost to increase the volume of a signal. And because of the advertising model of clicks leading to revenue, the digital organism (web site for example) gains energy the greater the range of its signal. Once a digital organism starts to gain attention, it gains more energy and will saturate its environment. This closes out other digital organisms.
Until we find a way to place inhibitory feedback into the growth of digital organisms I think we are going to be stuck with a few organisms that fill the environment and leave little space for variations to arise.
Great article! We are trapping ourselves in algorithmic Skinner Boxes, not sandboxes. We are begging the question: What happens when the Skinner Box doesn’t just train you to respond according to certain rules in order to get pellets you already consider to be rewards, but in doing so, and because of the manner in which it is done, ALSO manages to train you over time on what pellet attributes to consider rewarding?
The curious case of transatlantic FOMO
"The grass is always greener. Social media has given us a very strange neurosis indeed." — Kate Bevan
Speaking to Kyle Chayka author of FILTERWORLD and how algorithms flattened culture. open.substack.com/pub/katebevan/…
Unwanted flattening of culture is part of the reason we decided to build an open, controllable recommender system. Now available for BlueSky. You can have any amount of diversity you want.
go.greenearth.social/r/ptryit
I need to thank Cory Doctorow for pointing me to your Substack
Thank you for such a smart exposition that causes one to think about these issues. I reflect on my own behavior when looking for new restaurants. The 5 star rating system reinforces early positive reviews that create a positive feed back loop and who then looks at restaurants rated below 4? I have found that when i make a comment on a Substack article that is somewhat intelligent and I do it right away, i get likes that build on themselves and have gotten over 100 likes. I think people dont scroll down and see others. Like being on page 1 in a google search. Who goes beyond page 2 at best? I have found that the search for the interesting and the excellent lies in the tails whether it be food, music, investments but whether it is society in general or LLM's, which seem to be merging into a post human hybrid, how do we create a positive rate of return on the new, different, innovative. It is ironic that while there has been much innovation in the creation of LLM's, which have their limitations, such innovation then creates such sameness. your suggestion that answer lied in regulation does not encourage me. Can Europe remain an island? Do not see it in the USA as a possibility today. While there is a clear rate of return on as you postulate having 20% on new stuff, the pure short term profit maximizers or would be monopolizers have a cost or a yield advantage over those that invest in the new. People ignore the risks of monoculture until its too late. Look at bananas and avocadoes eg. anyway you are a delight to read.
Thanks a lot Bruce, appreciate it! You might also be interested in the Google Maps piece I wrote a while back where I try to identify and correct for this rating system loop created by google maps. Europeans just love their regulation:) but you are right it’s difficult to remain an island, but that having said, this great book the Brussels effect shows how EU is quite a bit with its regulations shaping the world.
Got here via Cory Doctorow. Great read. The Big Data Analytics variant of AI carries this risk strongly. Generative AI has some randomness built in its fundamental engine, but it is far too weak to escape this trap, it is foam on a current.
I was reminded of two things. 1. The brain seems to have parts operating at 'the edge of chaos'. Two things probably require this: efficiency and variability of its operation. 2. Large organisations with complex IT have become aware of that only building directly functional results gets you trapped in a mess. Reserving 20% for the work that has no direct functional effect ('architecture') is important for th medium to long term. Of course, it depends on the quality of management to keep the discipline of taking that space.
I've been calling this phenomenon "the tyranny of the market."
When our city had a population of 125,000 there were at least a dozen small auto parts stores in addition to a couple of major chains. Now with more than double the population, small stores no longer exist and the chains have mostly converted to shop services backstopped by housewares. As far as I can tell, there are still a good number of people trying to do their own work, but it's just not feasible for most people to take their vehicle out of commission for a week or two waiting for a parts order.
This was interesting, thanks for writing.
You might enjoy Patrick Deneen’s book, Why Liberalism Failed, especially the part on anti-culture.
Maybe I’m naive, but are algorithmic recommenders really to blame for cultural sameness? That does seem to assume that decision-makers really follow verbatim what algorithms suggest. Hasn’t culture been converging long before algorithmically curated content and decision tools? Now just accelerated?
I think a culture that values comfort and is risk-avoidant might prefer the algorithms. They get spared the uncertainty of exploration. Not that they’re necessarily better off, but it is easy, and heaven forbid I have to engage in the messy art of choosing.
Finally, instead of regulating (which I thought was a strong take), what do you think about subsidizing entrepreneurs (your exploration) instead of “taxing” exploiters?
Interesting piece, thanks!
Looking at your results, it seems like the algorithms are still not that well optimized. Like, if I watch one Scandinavian crime drama, that doesn't mean I want that genre forever, but the algorithm often treats it as a stable signal. It is likely that I would rewatch something similar, but this probably varies a lot from user to user, and it wouldn't be hard to check who's actually a same genre viewer and who isn't. Similar logic probably applies to restaurants and music too. The most likely bet is the safest choice for the algorithm, but that doesn't mean it's the choice that actually makes the user happy.
What do you think?
Thanks! You are totally right and explaining exactly why exploration is needed before settling
Another super column, gripping to the end, great work.