LLM Training

→ Content strategy for training an LLM to deliver fitness guidance

🏆 Outcome: Launched feature → Increased activation rate

• Client/Team: Freeletics | Data Scientists, Developers, QA Engineer, UX/Product Team


• Timeline: 9 months


• Role: Project Lead | Content Strategy + Product Design


• Tools: Open AI, Google Suite, Miro, Figma

 🧪 Approach

The product team strategically placed Coach+ within the app to maximize impact. With these use cases in mind, I mapped out the content needs to establish priorities, grouping information into two main categories: general fitness information and product-specific content.

I then broke down these broad categories into manageable deliverables to maintain focus and meet deadlines. Simultaneously, I developed a quality framework to set clear response standards, ensuring users would receive the best content possible.

To support scalability, I established a clear information hierarchy and created a product concept map that organized content for easy updates and seamless integration of future features.

• 🗓️ Strategic Content Planning
  
  
  • Identified key topics, content gaps, and mapped them to use cases
  • Grouped content into 2 main categories: general fitness and brand-specific info
  • Prioritized deliverables and outlined a QA framework for response standards
  • Collaborated with developers, QA engineers, designers and data scientists to define technical needs

Training an LLM involves two key components: the preprompt—instructions defining the assistant’s behavior—and the custom corpus—a curated knowledge base the assistant draws from to answer queries.

For the preprompt, we drafted a directive to ensure responses were positive and informative. For the custom corpus, I defined the scope of fitness knowledge required, starting by collecting common fitness questions users ask online to identify trends and build a list of high-priority topics.

Using these insights, I curated relevant articles from the Freeletics blog, selecting content aligned with our goals and feeding it into the LLM. For the most frequently asked topics, I wrote dedicated articles providing clear, actionable advice in our brand voice.

On the product side, I conducted a comprehensive audit of the app, breaking down its features and sections to identify areas where users might need guidance. I then categorized and prioritized these topics based on their importance to new users. This structured approach allowed me to set clear milestones for the initial release, leaving time for quality testing and refinement.

• 🔍 Content Research & Curation
  
  
  • Collected common user questions and concerns from multiple online platforms
  • Audited the app to identify content gaps and prioritize topics relevant to new users
  • Selected and created content for top fitness topics, including curated blog articles

📝 Improvements

Refining the LLM was an iterative process that required continuous testing and optimization. To guide this, I collaborated with our QA engineer to develop a custom rubric—a clear evaluation framework for assessing response quality. I then ran an initial round of internal testing, using this rubric to pinpoint weak areas and improve the content based on user feedback and question frequency trends.

Once we reached a stable baseline, I expanded testing to a group of internal users for further refinement, before releasing Coach+ to 100% of English-speaking users.

• 🧪 Internal Testing
  
  
  • Collaborated with QA engineer to create an evaluation rubric
  • Test responses against a list of frequently asked questions
  • Conducted internal testing before staged feature release

To prepare for multilingual deployment, I adapted both the preprompt and documentation—breaking down responses into smaller tokens and adding language-specific instructions to handle local variations and names. During this phase, I onboarded and mentored an intern, training him in QA best practices, data reporting, and documentation, which allowed me to focus on trend analysis and stakeholder reporting.

• 🔄 Continuous Refinement
  
  
  • Curated content based on testing feedback
  • Prepared and adapted content for multilingual deployment

I designed the initial chat screen concepts and mapped the flow guiding users to the feature entry point. I also reviewed the final designs implemented by the UX team to ensure consistency with our original vision. Additionally, I wrote the preselected question options users could choose from—grounded in the data we had collected during research—to ensure relevant and helpful suggestions aligned with real user needs.

• 📊 Data-Driven Decisions
  
  
  • Analyzed trends and reporting insights
  • Identified key user pain points to inform future product decisions
  • Selected pre-filled questions for the chat interface