I have known Emma for over a month. She has contacted me four times and follows me after I post an article. Although she claims to live in Shoreditch each time, she seems to have multiple personalities. Her end goal appears to be to generate revenue from her OnlyFans page, which she advertises.
To draw in customers, she shares photos of herself in revealing clothing. She is attractive and has a notably large backside, if that’s your preference. The source of these images is uncertain, but they are likely generated with highly realistic synthetic faces, bodies, or bios created by language models to mimic real users. These accounts are primarily involved in coordinated influence campaigns, crypto scams, spam, and political boosting. Research shows that such accounts often operate in groups and tend to have fewer followers.
To understand how these accounts operate, particularly given the advanced AI technology involved, visual and behavioural cues are crucial. Emma’s fake accounts seldom post original or varied content. Other fake profiles often act as reply-guys, spam affiliate links, promote schemes like “get-rich-quick,’ cryptocurrency scams, or use generic language similar to ChatGPT. These profiles usually follow thousands of users but have very few followers.
Although Emma the bot dismisses her work as trivial and insists she’s real, the rise of AI-generated fake identities and synthetic bot networks poses more than just a technical challenge. It exposes a deeper problem rooted in capitalism’s social dynamics. This issue significantly affects humans as aware, social beings and could be harmful. To fully understand this, we must link it to capitalism’s long history of using technology to benefit the ruling class, as well as the wider social crisis capitalism has induced in human awareness and community.
I identified Emm’s game early, but for others less aware, a collapse of Shared Reality can threaten mental health. One of the most damaging effects of AI-created fake profiles is what can be called an epistemic crisis—a systematic breakdown in an individual’s capacity to tell reality from falsehood. This problem isn’t novel under capitalism; historically, the ruling class has sustained control via ideological mystification.
However, AI bots operating on an industrial scale mark a significant advancement. When someone cannot be sure if their online interlocutor is human or machine, if the consensus they see reflects genuine public opinion or a synthetic effort, and if the emotional connection they feel is with a real person or an algorithm the very basis of rational social discourse begins to break down..
This has profoundly corrosive effects on individuals. The natural human response to an environment saturated with deception and manipulation is a generalized suspicion, not merely of bots but of everyone. When you cannot reliably distinguish the genuine from the fake, you begin to distrust all online interactions. This cynicism is, in many ways, a rational adaptation to an irrational environment, but it carries an enormous psychological cost. It deepens social atomisation, makes solidarity harder to build, and breeds a pervasive sense of isolation and powerlessness. People retreat from engagement, or are drawn into filter bubbles where algorithmic amplification — often driven by bot networks — creates false communities built around manufactured outrage.
In this already fragmented social landscape, the emergence of AI-generated fake social environments often results in predictable harms. Young people, still forming their social identities and seeking validation through peer interactions, are particularly vulnerable. When online communities are dominated by artificial personas created to provoke engagement, outrage, or emotional reliance, authentic developmental progress is hindered. The fundamental human ability to form genuine relationships—based on mutual vulnerability, genuine uncertainty, and meaningful stakes—is jeopardised when these interactions occur mainly with machines rather than real individuals.
A Trumpet of Sedition 