I’ve been a professional content writer and editor for more than a decade. When AI tools began reshaping the writing landscape, I chose to lean in rather than resist — taking on Human-in-the-Loop editing work, learning the strengths and limitations of LLMs firsthand, and studying conversational UX design. I’m not an engineer. I’m a writer and human-centered designer in training — and I believe that’s exactly the kind of perspective needed in this evolving space.
My HITL experience has taught me how to guide AI outputs toward clarity, tone, and usefulness — exactly the same skills needed to design conversational experiences that feel natural, helpful, and human.
The Hype vs. the Reality
I recently read an article about the gap between AI productivity hype and reality — and it resonated with what I’ve seen firsthand in Human-in-the-Loop (HITL) work. Many companies underestimated the human skills required to guide these tools effectively. In my own experience editing and evaluating AI-generated content, I’ve seen how essential human input still is. Left on its own, AI tends to produce shallow or inconsistent content. Skilled HITL work is what turns those rough outputs into something clear, trustworthy, and genuinely useful for audiences.
One of my first up-close encounters with the gap between AI promise and reality came when I was writing for a hospitality startup — one I was a natural fit for, having written extensively for some of the same properties under a prior brand. I’d been fully onboarded, trained on their brand voice, and had turned in several blogs when the team suddenly paused content production.
A few months later, I noticed new blogs going up on their site — but not mine. Curious, I read them. At first glance, they looked polished. But it quickly became clear they were fully AI-generated. The writing was repetitive, the voice flat and brandless, and — most concerning — there were hallucinated details that could have misled guests.
At the time, many companies — especially in travel and hospitality — were under enormous financial pressure and looking for ways to cut costs. Using AI to generate “good enough” SEO content at the push of a button was becoming an appealing option. But from a guest experience perspective, this content was a shoddy welcome mat.
That experience was a turning point for me. I realized that while AI could produce fast, surface-smooth sentences, it couldn’t replace the depth, trustworthiness, and brand alignment that experienced writers bring — at least not without human guidance. If every company used it this way, every company would soon sound the same. That’s when I began leaning into Human-in-the-Loop work, to better understand how AI and writers could collaborate.
Decision to Lean In
Around the same time, I was also writing podcast reviews for a major media platform — and that client strictly forbade the use of AI in any way beyond basic Grammarly and Hemingway checks. We had to run our own work through Clearscope before submission, editing until we hit acceptable scores. Ironically, even using podcast titles or host names would trigger penalties, forcing us to rewrite accurate content. It was a tedious balancing act — but it revealed how limited AI was at that stage.
Out of curiosity, I experimented with AI for research. At the time (early GPT-3), the model hallucinated heavily — inventing entire fake podcasts or misrepresenting details. But now and then, it produced content surprisingly close to publishable quality. It was clear the technology wasn’t ready — but it was also clear it was improving fast.
A friend of mine — a philosophy professor with 20 years of experience — saw this shift firsthand. Under pressure to design a new Critical Theory course, he watched as I prompted GPT to draft a full course outline as a demonstration. In awe that it took only minutes he said, “I think we’re all going to be unemployed very soon.” Neither of us fully believed that, though what I took away was this: AI could be a useful tool, but one that still required thoughtful human guidance.
With concern that AI might upend my own writing career, I pursued an entry-level Human-in-the-Loop content role with an AI services firm. There, I worked hands-on with LLM-generated content — editing drafts, evaluating tone and factuality, writing fine-grained evaluation criteria, and helping optimize outputs for enterprise clients.
LLM editing gave me a peek behind the Wizard of Oz curtain — a chance to see how these models actually work, where they break down, and how human input shapes what users eventually see.
In HITL roles, there’s a key distinction: some focus on improving the language model itself (LLM editing — evaluating and training how the model responds to different inputs), while others focus on refining the outputs it generates for actual users and audiences (content HITL — editing for voice, tone, and clarity before publication). I’ve worked across both — evaluating LLM outputs for tone and accuracy and shaping AI drafts into publishable content.
At first, I focused on open-ended projects where I could experiment. I prompted the models on everything from Shakespeare’s Hamlet to telemark skiing techniques and Subaru engine repairs, just to understand what they could and couldn’t do. I was tasked with rating and editing the better of two LLM responses, often over multi-turn dialogues.
Later projects became more focused: testing niche academic prompts, uploading open-source documents and interacting with the model on those texts, refining responses, and tagging subtle errors.
Projects soon became more technical and nuanced with tasks like jailbreak prompts — crafting inputs to test whether the model would violate defined guardrails — and on system prompts, which shape the model’s persistent behavior during a session. I also explored retrieval-augmented generation (RAG), where the model uses external documents to improve accuracy, and contributed to fine-grained evaluations of multi-turn dialogue — testing how well the model maintained context and coherence over extended exchanges.
Over time, I watched the models improve. Hallucinations became less frequent. Responses became more sophisticated than what I’d seen a year earlier when writing podcast reviews. I even experimented with one of my own creative projects: about two years ago, I plugged in a short fiction piece I was working on, and the model’s attempt to continue it was laughable — full of junior high clichés and flat, predictable prose. But when I tried the same exercise again a couple of months ago, the response was surprisingly decent. Phrases like “the door groaned open” added texture, and the suggested plot turns actually gave the story forward momentum. It was a reminder of both how far these tools have come — and how much creative discernment is still needed to guide them.
Through all of this, I learned to collaborate with AI, not fear it. And that mindset continues to shape my work.
Expanding into Conversational UX
Building on my HITL work, I’m now expanding into conversational UX — studying flow design, tone shaping, and human-centered conversation through the Conversation Design Institute. As part of my coursework, I’m developing a sample chatbot — a “Guest Ready” bot designed to help travelers prepare for short-term rental stays by answering questions and providing useful local context. Designing that bot has deepened my understanding of how conversational flows need to anticipate user concerns, guide interactions gracefully, and adapt to varied needs.
In Conversational UX, voice consistency is crucial. Chatbots and AI interfaces need to sound like “one voice” — and I already know how to spot tone drift and shape responses to brand voice from my HITL editing.
Conversational design is about designing for human expectations — understanding what feels trustworthy versus robotic, where phrasing breaks down, and what frustrates users. My HITL experience gives me an edge here: I’ve seen firsthand how users interact with AI outputs, what succeeds, and what falls flat.
Conversational design isn’t just flowcharts — it involves writing good prompts, designing fallback flows, and thinking about system behavior. It’s a human-centered discipline — and that’s what drew me to it.
Personal Dimension
Caring for my mother, who has dementia, has made this work personal. I often hear her talking in circles with Alexa — asking the date over and over, worried she’s forgotten to pay her bills, and eventually tiring herself out. I’ve started thinking about how a more thoughtfully designed chatbot — one that could detect worry patterns, remind her that her bills are paid, suggest she check the weather, or gently encourage her to go outside and water her beloved azaleas — could offer real support and comfort.
Working on my Guest Ready project has made me realize that these same conversational design principles — clarity, tone, responsiveness — are exactly what would make AI more helpful for vulnerable users like my mom. It’s a vision I’d like to keep exploring.
Conclusion
This work has shown me again and again that human skills — clarity, empathy, judgment — remain at the core of meaningful AI collaboration. There are few experienced writers who can articulate why HITL roles matter, what human judgment brings to AI outputs, and how conversational UX principles apply to content strategy. That’s where I want to keep contributing: helping shape AI outputs and interactions that truly serve human needs — with depth, care, and understanding.
Brian Olson 