Overview
We wanted to explore the capabilities of using LLMs in our application, but we wanted to make sure that such a feature would deliver real value to customers and not lean on the growing trends at the time. It was important that this assistant could provide more value than ChatGPT or Anthropic for example, was safe, secure, and easily available to customers. Our overall vision was that managers would benefit from in the moment in-app assistance.
Our final result: writing assistant in action. This feature includes a beta label, prompt limit, and ability to leave qualitative feedback.
The Challenge
Managers have the historical problem of not feeling comfortable with delivering feedback to their team. One of the areas of opportunity identified in our customer research was managers needing to deliver valuable feedback, without feeling blocked and in situations where they do not necessarily have feedback training or an HR team to support such activities. This is a common friction point especially at small or growing organizations where talented ICs are promoted into leadership positions without training.
Challenge 1: lack of training
Managers at small companies do not have access to the training needed to deliver valuable feedback. This can create mental blocks for the manager to actually create and deliver feedback, ultimately slowing down feedback cycles at the organization.
Challenge 2: be better than alternatives
Customers, especially managers, could easily "hire" ChatGPT or alternatives to write feedback without us. This creates the challenge for us to be better than these alternatives, with a focus of leveraging our unique value propositions.
Challenge 3: make it secure and have solid messaging
HR teams and organizations especially value privacy and security for their employee and company data. AI model training has inherent risks involved if data is used to train models on an ongoing basis. As we were serving B2B customers, the benefits of AI have to outweigh the compliance and security risks.
My Approach
After countless interviews with senior HR admins and senior managers, I saw a particular gap in feedback writing. We learned that managers were already actively leveraging AI services like ChatGPT to improve their feedback. We wanted to further explore this opportunity and deliver a valuable feature within scope and on time.
Research & Discovery
Customer interviews were a key piece of understanding the problem. Patterns uncovered included that while managers could quickly identify "problems" they weren't necessarily trained on how to approach feedback in a clear and meaningful way, making feedback actionable and avoiding bias. We looked at the current resources available for managers including online training, guides, books, coaching and existing GPTs geared at improving feedback. As the market is quite noisy, and we already had a good idea from our own expert customers and audiences on gaps, we focused on understanding how we could measure quality.
Analysis & Strategy
I prototyped "feedback buddy" a writing assistant that would allow managers to adapt different feedback frameworks (Radical Candor, NVC, STAR methodology) to help adapt feedback accordingly. While this tool was more advanced in that it incorporated different feedback methodologies, we wanted to focus on something more lightweight and geared to any audience. We wrote and tested hundreds of prompts while testing this feature and our prompt frameworks.
Implementation
Implementation included building the UI for such a feedback to be included directly in the rich text editor while writing feedback. This in the moment approach separated us from alternatives that would require separate tabs or browsers to analyze feedback. The assistant did not use inputs to improve the model, delivering on our focus to be a secure alternative to public products managers were already using. We prepared public documentation about our approaches to AI, and included feedback mechanisms to improve the feature itself on a continous basis.
Measurement & Iteration
Since this feature was released as beta, we were able to measure and learn how effective the actual responses were with the in-app feedback. We could also count the number of times re-generation occurred, or feedback was directly applied by the user. Iterative and incremental improvements were necessitated based on security upgrades, UI enhancements, and prompt refinement.
The Solution
Our solution included a secure model that did not leverage training data, but instead helped transform feedback based on expert prompts.
Early time to value
Value delivered in the moment, while actually writing feedback.
Velocity
Reducing friction for managers improved overall completion deadlines for feedback within cycles.
ICP focused
The solution is geared towards managers who don't have specific training, and would benefit from this assistant.
Results & Impact
This solution resulted in faster time to value for managers, reducing cycle lengths for HR Admins.
Reduction in overall cycle completion rates
At least 30% of managers applied provided feedback improvements
Managers engaged in the feature request and feedback loops to measure quality.
Lessons Learned
AI integration is both more simple and more complex than initially envisioned.
Prompt engineering
Learning by doing: prompt engineering moves more quickly if you can rapidly test different approaches, and simplify the prompt to match more audiences.
Integrated
It's important that the feature delivers real value and isn't gimmicky - as well as being directly integrated and contextual for the user.
Positioning
HR teams in particular want to ensure compliance and security standards when interacting with AI features. Additionally, we learned that many organizations were wanting to strictly balance the use of these types of tools over human interaction.