Three Times NOT the Charm: When AI Illustrations Go Off the Rails

When I set out to improve an illustration for my forthcoming children’s book, Doris’s New Home, I wasn’t expecting a breakdown of logic, perspective, and anatomy. I expected a quick revision. Just a few tweaks, really. I just wanted a warm family dinner scene.

So I gave ChatGPT a clear prompt:

“All the family members are sitting in the dining room around a mahogany table. On the table are typical Jewish foods, including a platter of Linzer cookies (with raspberry jam) on the white china plate. Everyone is smiling and happy because they are all together again.”

I reminded “Scripty” (my AI pal) to use the same characters as used throughout the book and to maintain the same vintage look. Simple, right?

Well.

What followed was weird. Like, just… weird.

Let me be clear: AI can do stunning things. It has helped me generate dozens of charming, consistent illustrations for Doris’s New Home, which is based on my adult novel Nothing Really Bad Will Happen. The characters are visually consistent. The tone is right. The historical details sing. But when AI illustration goes wrong—especially more than once—it doesn’t just miss. It spirals.

Attempt #1: Who Ordered the Doris-Head Platter?

At first glance, the image looked promising—until I noticed a disembodied second Doris head floating above the cookies. Not only was this jarring, it was also deeply confusing. Was it a surreal dessert? A haunted centerpiece? A warning from the AI gods?

Also missing: Opa and Emile. Gone. Jewish food? Vaguely present, but completely overshadowed by what appeared to be Doris’s floating face atop dessert. And let’s not mention it looks like the other Doris is jammed in between the table leaves.

Attempt #2: Where is Everyone?

Okay, I thought. We fixed the Doris-on-a-platter issue. But now we have a severed head centerpiece! But where are Rose and Emile? Maybe they went out looking for their sister, Claire? And why did Paul suddenly decide to shave off his mustache?

Attempt #3: Things Fall Apart

By the third attempt, I was laughing—but not in the good way. Who were these people? Most of the faces didn’t match the character designs used throughout the rest of the book. Doris had changed hairstyles and bone structure. Rose was absent. And who invited George Costanza from Seinfeld?

At this point, I enacted my new rule:

Three Strikes, I’m Out.

Here’s the thing I’ve learned through this process: once AI illustration fails three times in a row, it’s time to move on. Not forever, just for now. My new rule is simple: after three failed attempts, walk away, eat something you shouldn’t, and revisit the prompt another day.

Why?

Because in my experience, failure compounds after the third try. Things go downhill fast—characters morph into strangers, logic collapses, and before you know it, you’re serving a child’s head as dessert.

Now, to be fair to the AI—and yes, I will defend it here—its successes are extraordinary. It captured tone, emotion, and vintage detail better than many human illustrators might on a limited budget and timeline. But when it breaks? It really breaks.

I’ll try again another day, probably with a better prompt and a bit more patience. No floating heads. No anonymous uncles. Just a break from the digital madness, and a little faith that Doris’s New Home will eventually get the dining room scene it deserves.

Want to see the good, the bad, and the delightfully bizarre?
Subscribe for updates or visit deborahsholman.com to follow my journey turning historical family stories into unforgettable books.