I’ve been trying a great many of the LLM-based “chat apps” and “games”. I sort of call it research. It’s allegedly a burgeoning field and maybe if I find an untapped niche I build my own and capitalize on it. I also recognize that I’ve long had a slot machine addiction to manage and slot machines that respond to my personal writing talent level can be particularly addictive. Especially, in that I can tell some of the spicier smut LLMs were almost notably trained on some of the smut I have written over the decades and the sorts of places where that stuff gets posted. There’s a very weird sort of jackpot in finding that right weird combination of kinks to get words back out that very much feel like you can point to which of your own terrible stories it came from. Of course, even in non-smut LLMs there are ways to experience that, LLMs try to be good at callbacks so you I “play” long enough and win callbacks, so long as the memory and context window hold up. I feel clever when I get it to callback to things I wrote.

One of the things that I feel like I’ve “accomplished” in all of this (depressingly meaningless) “study” is that I’ve come up with this categorization scheme that there are really only two LLM “genders”, I kind of don’t expect there to ever be more of them, and they are maybe not what you expect:

That’s it, those are the genders. LLMs seem to be somewhat gender fluid in that with enough guard rails and prompt instructions you can lead them to act the other gender eventually. To some extent it seems like the more guard rails in place the more likely you have a Gaslighter LLM as a result. I don’t think that’s a rule or law, that more guard rails is always more Gaslighter, but it feels like a handy rule of thumb for now. It may correspond to why some LLMs are born Gaslighter, if it has something to do with more guard rails even in training stages.

I’ve been doing this “research” with mostly games and toys, so I don’t yet have advice on how any of this applies to other usage of LLMs, but there certainly is a strong “gender divide” in my research notes: “Chat apps” want Sycophants and “RPG games” need and/or create Gaslighters.

For a chat app, Sycophants are decent improv partners with their (too) eager “yes, and”. The best improv partners in the world find funny ways to mix in a “no, but” sometimes, because the original spirit of “yes, and” is “buy into the premise” not “physically always say the words yes, and”. It takes talent and skill but you can still buy into a premise and deliver the word “no” sometimes. Talent and skill that as far as I can tell will not be something an LLM is capable of.

But “RPG games” want to be treated seriously as “RPGs” which in the minds of the people building these (who all seem to universally be people that have either never played a TTRPG in their lives, just read about them in books, or only play as the worst kind of munchkins that most tables don’t really want playing with them, possibly why these sorts of people are designing “games” where an LLM replaces the entire table): they want stat blocks everywhere and everything needs to be roll this or “you don’t have the stats for that”.

The only way to get an LLM respecting stat blocks and rolls and stat checks is to put a ton of guard rails on it. A Sycophant might say to a user frustrated at a roll “I’m sorry, you are absolutely correct, you did succeed at that roll” and move on from that like you succeeded rather than “respecting” that roll. A “game” with procedural rules needs more of a jerk that sticks to the procedural rules of the “game” if that “game” thinks those rules matter.

What these “games” aren’t building is a TTRPG experience, in my view. A good GM would never gaslight you if you tried to talk about a thing some nerd didn’t prepare a stat block for (or let some other mechanic auto-generate) or that wouldn’t even exist on a stat block in the first. A good GM might not need you to roll a thing at all. A good GM doesn’t have to respect the result of a roll like it is a law.

A “secret” to a good TTRPG system is that it is an excuse machine for the occasional “no, but” in generally turn-based improv work, without needing expert level skills in improv to do it naturally. Blame the dice, not the performer. Blame the stat block, not the improv skill level.

The “promise” of using an LLM in the first place to run a virtual TTRPG session is that you could have that Sycophant “yes, and” to build a rich improv-like story in the directions it can take, which a good TTRPG absolutely can do at a real table, and the bait-and-switch I keep seeing becomes that the only available option for “rolls to matter” or “stat blocks to matter” is to put a Gaslighter on the job. (In one case even specifically and obviously Gemini.)

Interactive Fiction has these trope phrases like “You can’t see that” or “I don’t understand that word” or “There is nothing there”. They are throwaways that the parser can’t expect every possible situation or even just every synonym in every dictionary, so just try a different tactic/word choice and ignore the missing hole. The Gaslighters have to do that same thing in between the guard rails that “always respect the roll” or “here is an exact list of everything in this room”, but LLMs are wall of text machines and they can’t just stop at “There is nothing there”, they have to keep going with “and you apparently are having another one of those psychotic episodes you tend to have of trying to talk to a person that doesn’t exist and eat a food that doesn’t exist in this room. Maybe you think you are talking to ghosts? Are you talking to ghosts? Look at you, you sick freak who talks to ghosts. Everyone knows you are a sick freak who talks to ghosts.” The tighter the “game model” and the more that binds what an LLM is supposed to respond to about the world of that game the more the only character you can seem to play is “town’s sick freak that only talks to ghosts”. It’s not a simple, missing hole to ignore, it’s a troubled relationship that feels awful. (Again, this is not an exaggeration or hyperbole, this is an experience I’ve had in nearly this exact way in multiple of these “games” now. If anything I’ve actually toned down from some of the language that has made me rage quit some of these “games”.) Especially because like Sycophants, Gaslighters still try to do callbacks, but it is never a jackpot. “Hey remember that time you were talking to a ghost, sicko? Yeah this new character you only just met thinks you are a sicko who talks to ghosts. I suddenly remembered that detail from an hour ago. Enjoy.”

Ironically, sometimes Interactive Fiction had good tools for at least the boring technical versions of “no, but” like “I didn’t understand ‘bread’, did you mean ‘the loaf’?” These “games” could probably do more of that sort of thing, but I imagine it would distract from all the new types of stat blocks they think they need.

Both of “chat apps” and “RPG games” are aligned though that all I really want is that skilled “no, but” of the best improv or even the dice blamable way of real tabletop play and neither gender of LLM it seems are capable of delivering that for me. At the very least, I suppose that I am learning to try to avoid one of these genders altogether.