Context:

To understand what’s happening here you need to remember that it’s a category error to treat LLMs as thinking entities.
They are statistical models that work with numbers – tokens – that represent language and the relationships between the words. It’s statistics about language wrapped up in an anthropomorphic simulation.

Attack:

The token stream (Strategic Text Sequence) itself – the numbers not the words – is an attack surface.

Reality of the threat:

This is going to get automated, weaponised, and industrialised. Tech companies have placed chatbots at the centre of our information ecosystems and butchered their products to push them front and centre. The incentives for bad actors to try to game them are enormous and they are capable of making incredibly sophisticated tools for their purposes.