Using AI for Product Feedback Management

In a world chronically suffering from too much information and too little understanding, user feedback is paramount for any product’s success. But with the sheer volume of feedback that companies receive, sifting through it manually is neither efficient nor effective. AI has the potential to revolutionize how we handle, analyze, and act upon user feedback.

We’re diving into some cool AI applications and ideas that’ll change how you see feedback. For every AI application, I will try to evaluate usefulness vs. hype.

Summarizing feedback

Hype: 6/10
Value: 4/10

The trivial use of AI is to summarize long feedback into shorter, digestible points, allowing product managers to quickly grasp the essence without reading through lengthy paragraphs. Many articles and vendors praise summarization as a very useful tool, but I find it relatively useless. It works only for the very specific use cases when you really need to grasp the essence of the conversation to decide whether to dig into details or not.

Tools

Summarization is quite easy to implement, so it is likely that it might work already in your favorite tool. Fibery was among the first to introduce it. For example, here is how summaries of customer support chats work. You can quickly review recent chats, spot some interesting topics, and dig into details if you want.

Automatic video/audio interview transcription

Hype: 4/10
Value: 7/10

Automatic transcripts are not new, but they are affordable now and embedded into many tools by default. With the advancement in AI, the accuracy and efficiency of these transcriptions are continuously improving, reducing the need for manual corrections. Moreover, with automatic text translation, you can quickly transform a German interview into a quite good English text version. Language unification for product feedback processing is important.

Text is better than audio/video in many ways:

• You can search it, so you can find some specific keywords or passages fast.
• You can feed text into your feedback processing tool and augment it with links to ideas, tag some phrases, etc.
• You can quote it and insert it into feature specs, problem definitions, and other docs.

Tools

From what I saw, Dovetail provides the best embedded transcription tool for product feedback. You can upload the video, generate the transcript, and tag some parts. The transcript is synced with the video, so you can quickly jump from the text part to the video segments. Overall, it is a very pleasant experience.

If you need something custom, you can use Open AI Whisper to build your own tools fast.

Sentiment analysis

Hype: 6/10
Value: 3/10

A sentiment score can provide a quick pulse check on whether users are generally happy, frustrated, or neutral about a particular feature or the product as a whole. In theory, you can use sentiment analysis to assign priority levels, allowing urgent matters to be addressed first. For example, when a customer is really angry you can give this request higher priority.

Example

Sentiment analysis demands some good prompts to work well. I’ve had good results with a prompt like this:

Sentiment scores are a metric for measuring customer sentiment. Scores can range from 0-100, where 100 is the most positive possible outcome and 0 is the least. Generate a sentiment score for the text below and return result as a number in a format [sentiment score] without brackets 
"""
{{Conversation text}}
"""

Tools

In Fibery, we do sentiment analysis for incoming Intercom conversations. For example, this report shows sentiment scores by a person in the Fibery team. You see that average sentiment is good, which means that discussions are usually polite, constructive, and decent.

Feedback clustering & categorization

Hype: 6/10
Value: 6/10

Now you can use AI to automatically categorize feedback into various buckets such as bugs, feature requests, usability issues, compliments, tag feedback, or find common themes. Manually sorting through and categorizing feedback can be time-consuming. AI can quickly process large volumes of feedback, allowing product managers to focus on analysis and decision-making.

Tools

In Dovetail, you can clusterize highlights automatically with the help of AI. It works neat on a Canvas View, so you can find common themes really fast and spot patterns more easily. However, all these highlights should be created manually, so in reality it saves time, but not so much.

Automatic insights extraction from feedback

Hype: 3/10
Value: 8/10

When you have a lot of feedback, it is cool to extract some useful insights from it automatically. Imagine you feed all your intercom conversations, all survey results, all customer interviews, and all forum requests into some tool, push a button, and get dozens of problems, observations, and requests ranked by frequency and pain.

Example

Let’s say, here the real conversation starts with our user. Note that the user reply is totally irrelevant to the first message, it was just a trigger to ask for something new:

Michael: 
Hi, We've just released Replies in entity comments. Replies are nested inside a top comment, so you can organise discussions better. To add the first reply, find Reply to Comment icon in a comment and click it.

User: 
How about summing numerical contents of a column? When are you releasing that?

The ideal tool can analyze this reply, create new insight “Summarize numerical values in Table View” and link this request to the insight. Right now we are doing it manually in Fibery, but product managers have better ways to spend their time…

Tools

Viable can process a lot of feedback and generate insights automatically. I watched a few videos but didn’t try it myself. Anyway, according to these videos, it works very well, so maybe it is worth a try.

It seems Cardinal does something like that semi-automatically. You can setup Intercom sync and all conversations will be processed to find relevant feedback. Then Cardinal suggests potential features to link and you can do the linking manually then.

Automatic feedback linking to problems and features

Hype: 2/10
Value: 7/10

Imagine you have a backlog of features and problems in some tool. Then you push a button and hundreds of Intercom chats are analyzed, and relevant parts of text are extracted and linked to these features and problems. What value can it bring to you as a product manager? Why connect feedback to product hierarchy? Here are the top two things in my opinion:

• Data-driven decision making. This system would provide concrete data on which features or problems users most frequently discuss, allowing for more informed decisions on prioritization. And prioritization is hard. Very hard.
• Richer context: Directly linking chat excerpts to features or problems provides context. This context can offer deeper insights into user sentiments or specific pain points they face. As a result, you will write better feature specs and miss fewer problems.

In a nutshell, prioritize problems better and invent better solutions, since you will have more context attached to a problem. Automation will save you time and an AI-powered system potentially can do that.

Example

Let’s say, you have a backlog that contains Product Areas and Features. You also have intercom feedback accumulated in some tools. Then some parts of conversations are linked to Product Areas and Features and you can see all linked feedback inside a Feature.

Tools

There are no tools that can do it so far. Interestingly enough, there are not so many tools that do it even manually. What an opportunity! In Fibery you can use semantic search to link feedback faster, but still manually. You select some text and Fibery suggests to link it to existing Feature or Insight.

Cycle also suggests some semi-manual linking. It finds relevant highlights and you can link the to features or insights via some suggestions.

Feedback Chatbots

Hype: 5/10
Value: 5/10

Feedback chatbots are automated conversational agents designed to interact with users for the primary purpose of collecting feedback. Instead of having users fill out a static form, a chatbot can guide them through a dynamic conversation, making the process more engaging and potentially capturing richer insights.

Example

Imagine wanting to collect feedback for some new feature. You can use intercom announcements and start conversations, but if you have 10K+ users it can be challenging. With an AI chatbot you can have decent conversations around new features, and the bot can ask relevant questions to deepen the feedback and improve its quality.

Bot: Hey, we released the new comments feature a week ago and it seems you tried it already. What is the most annoying in it for you?
User: Nothing, really, I like it. 
Bot: OK, maybe you miss something then?
User: Oh yeah, it would be handy to have replies nested to comments.
Bot: Noted! How do you want these replies to work? How many nested levels?
User: Well, usual replies with one level are good enough. 

Note that without AI it is impossible to script such questions, you will stop after a second answer for sure, but with AI in theory you can get very annoying bots that will fetch useful feedback from a clever user.

Tools

While we have many chatbots to answer questions and help with support, I’m not aware of chatbots that can create dynamic surveys with the help of AI. If you know any, ping me on twitter and I will update the article.

Conclusion

I think some AI tools are overhyped, while others are less known but more useful. Here is my subjective Hype vs. Value scatterplot diagram to help you navigate feedback management AI tools.

As you see, my personal top 3 valuable tools are:

• Automatic insights extraction from feedback

• Automatic feedback linking to problems and features (but it does not exist yet…)

• Automatic video/audio interview transcription

As AI continues to evolve, we can anticipate more seamless integration with existing tools and new, even more powerful ideas. Existing vendors might catch up, but we will also see new feedback management tools built with AI core. Interesting times!