Resume

Brian Olson
Whitefish, MT • Remote (U.S.)
406-210-5480 • brian.olsonwriter@gmail.com
LinkedIn: linkedin.com/in/brian-olson-b527408
Portfolio: brianolsonwriter.com

AI Evaluation Specialist | LLM Training | Adversarial Testing | Content Quality | AI Safety

Professional Summary

AI Evaluation Specialist with 1,000+ hours of experience reviewing and improving large language model (LLM) outputs for clarity, factual accuracy, grounding, safety, tone, and real-world usefulness.

Experienced in response ranking, rubric-based quality review, prompt refinement, adversarial testing (red teaming), and human-in-the-loop editing workflows.

Background spans AI evaluation, technical writing, UX content systems, and behavioral assessment—bringing a structured, safety-focused approach to evaluating model behavior in real-world contexts.

Core Skills

  • LLM Evaluation & Human Feedback (RLHF-style workflows)
  • Response Ranking / Preference Selection
  • Prompt Testing & Refinement
  • Adversarial Testing (Red Teaming)
  • Rubric-Based QA & Quality Assurance
  • RAG Grounding & Source Fidelity
  • Content Editing / Rewriting for Alignment
  • Trust & Safety Review
  • UX Writing & Technical Documentation
  • Cross-Functional Communication

Technical & Tools

  • AI Lab & Infrastructure: Oracle VirtualBox, Ubuntu CLI, Ollama (local LLM orchestration)
  • Testing Tools: Garak (LLM vulnerability scanning), familiarity with PyRIT
  • Technical Stack: VS Code (Jupyter), GitHub, Postman; working knowledge of Python, JavaScript, Swift/Xcode
  • Productivity: Google Workspace (Docs, Sheets, Drive, Meet), Microsoft 365 (Excel, Word, PowerPoint, Teams)
  • Content & Ops: Airtable, Asana, SEO workflows, Grammarly, Hemingway

Professional Experience

AI Evaluation & Content Specialist (Freelance Contract)

Remote | Jul 2023 – Present

  • Evaluate LLM outputs for clarity, factual accuracy, neutrality, tone, safety, and real-world usefulness
  • Compare model responses and select preferred outputs using structured evaluation criteria
  • Assess Retrieval-Augmented Generation (RAG) outputs for grounding and source fidelity
  • Rewrite outputs to improve quality, alignment, and usability
  • Perform adversarial testing (red teaming) to identify jailbreaks, hallucinations, and multi-turn reasoning failures
  • Contribute to rubric-based evaluation, ranking workflows, and quality calibration

Technical Lab & Testing Work:

  • Maintain a sandboxed Linux environment (Ubuntu via VirtualBox) for controlled LLM testing
  • Run local models via Ollama to explore failure modes and behavior under varied prompts
  • Use Garak to identify model vulnerabilities; conduct hands-on experimentation with PyRIT

Product Writer & Content Strategist (Contractor)

Amazon Projects | Remote | Aug 2020 – 2022

  • Audited high-volume podcast and editorial content using AI-assisted research and QA workflows
  • Performed fact-checking, editing, and HITL content validation
  • Maintained clarity, neutrality, and consistency across large-scale content systems
  • Managed workflows using Airtable and editorial QA tools

Writer & Content Project Lead (UX Content)

Stay Alfred | Remote | Nov 2017 – Jan 2020

  • Led guest-facing content initiatives across 33 U.S. cities
  • Produced multi-channel content (Airbnb, Expedia, Booking.com, TripAdvisor)
  • Led production of 40+ property and city guides
  • Coordinated workflows across design, ops, and content teams in Asana
  • Translated operational processes into clear, user-facing documentation

Technical Content Consultant & Editorial Lead

Citrix and Other Clients | Remote | 2014 – 2017

  • Produced technical documentation and B2B content for enterprise clients
  • Translated complex cloud/networking systems into structured, readable documentation
  • Conducted SME interviews and delivered rapid-turnaround technical content

Residential Counselor

Montana Academy | Kalispell, MT | Jun 2011 – Sep 2014

  • Supported adolescents in therapeutic settings (mental health, behavioral challenges)
  • Collaborated on treatment and safety planning
  • Managed de-escalation and high-stakes communication scenarios

Additional Experience

Amazon Home Stores (Product Writer) • CenturyLink (Help Systems Writer) • ESL Instructor (U.S., Mexico, Japan)

Education

  • Gonzaga University — B.A. Spanish Language & Literature
  • Graduate Coursework in Teaching (M.A.T. track, 21 credits)

Additional Training

  • Conversation Design Institute — Intro to Conversation Design
  • UX Foundations Coursework
  • American Avalanche Association Level 1
  • Swiftwater Rescue / Raft Guide Training
  • PSIA Fully Certified Ski Instructor