Lithosphere The Lithium Community

Highlighted
Frequent Advisor
Frequent Advisor

CSAT and helpfulness scores at more mature communities

Hi all,

 

Question to those in a more mature community (Although new CMs are also welcome to join in). I'm looking for some benchmark in avg CSAT scores and helpfulness scores ("Did you find what you're looking for - yes/no") across other communities.

 

We're just a bit over 4 months with Lithium and I'm still not seeing much increase from the initial launch. If anyone has some metrics to share that'd be great!

 

Thanks

0 Kudos
16 Replies
Frequent Advisor
Frequent Advisor

Re: CSAT and helpfulness scores at more mature communities

it looks like I'm alone here =/

0 Kudos
Reply
Loading...
Honored Contributor Honored Contributor
Honored Contributor

Re: CSAT and helpfulness scores at more mature communities

Sorry about the slow response @edgiansante.

 

I used to manage a support-based community. I would suggest that CSAT scores are not something that most businesses share widely. It's probably not also that useful to compare businesses. Much better to set your own goals based on your own circumstances and business objectives. 

 

It's good that you are being ambitious and want to lift your CSAT but I would be cautious about reading too much into results over just four months. It can take a while to change customer behaviour in regards to using community as a support channel and even longer to change average customer satisfaction levels - particularly in the upwards direction!

 

If you'd like to share some of the ways in which you are hoping to improve CSAT I'm sure you would get some helpful feedback from community members here. There is lots of wisdom, expertise and experience in the group. 

 

Good luck. 

 

Jason 

Generic1.jpg

Reply
Loading...
Honored Contributor Honored Contributor
Honored Contributor

Re: CSAT and helpfulness scores at more mature communities

I agree with Jason. Comparing CSAT scores across businesses may not be the best. It depens so much on the overall customer experience and support offering/strategy of the company. Even the way the survey designed and the questions being asked may result in different scores.

We find that customer usually provide feedback around their experience with the brand as a whole, meaning, the csat score is often not based on the experience with the forum but other interactions or problems with the product features etc. It makes it difficult to pinpoint the areas we can influence but we are doing our analysis, we learn, we try new things either by changing survey questions to get more actionable data, or changing the methodology, we make process improvements and provide feedback to other internal organizations.

Personally I believe a score by itself is of less value if you don't have the context. I prefer to ask more specific questions to understand where we might have failed, what the customer expectation was, what were they hoping the forum would for them and did we live up to that expectation and if not..why not.

I think it is important to continue learning and find your way in this. Help is available here so as Jason said. feel free to share more around what you are trying to achieve.

Cheers!
Wendy
Learning from others and helping where I can!
Community Passionista!
Reply
Loading...
Frequent Advisor
Frequent Advisor

Re: CSAT and helpfulness scores at more mature communities

Thanks both and sorry for my delay in getting back here.

So behavioral/context is great for me and our team but at the end of the day, you need some sort of metric to understand the outcome of these behavioral changes.

For a support driven community, the 2 common questions are around findability and satisfaction but I ran some tests and found a massive difference in them.

1st test was done on the regular 'pop up' survey (OOB). Tried diff combinations of time / sign ins /etc and results were overall the same. The feedback was mostly about the brand as you said Wendy. Not really about the experience using the community.

I then ran a 2nd test: inviting people to fill out a survey (using a qualtrics survey) with the exact same questions as the OOB one but after they received a response from either a member / super user or a moderator.

The results were extremely different as they were more transactional to that specific experience.

When looking at the community alone, you may find that result skewed. However, when comparing to an overall 'support experience' (which tends to be -> contact support staff -> receive information -> ticket is closed -> receive feedback survey) that's a much closer comparable (where I find the 1st one on pop up is more comparable to an NPS / pulse of the community).

Again - there's no one formula but Im finding that in a support community you can't just open a broad survey to any visitor and expect that to be comparable to anything (again - for you as a community manager yes, but not to report as a 'support' experience).

When asking about others' experiences, I was more curious about methods / response rates / overall increase/ decrease. Of course each business has its own results that are not necessarily a close comparative.

Hope this helps.
Reply
Loading...
Honored Contributor Honored Contributor
Honored Contributor

Re: CSAT and helpfulness scores at more mature communities

Hi @edgiansante


Thanks for sharing more information. It was interesting to read as we've also discussed surveying users after they get a response to be more comparable to other channels and the survey method, to get more insights on areas we can influence thus more actionable.

 

re:

When looking at the community alone, you may find that result skewed. However, when comparing to an overall 'support experience' (which tends to be -> contact support staff -> receive information -> ticket is closed -> receive feedback survey) that's a much closer comparable (where I find the 1st one on pop up is more comparable to an NPS / pulse of the community).

 

I am not sure if I follow you there. Perhaps you are saying the same. A broad survey will measure overall experience and not specific to the interaction and experience on the forum.

On the other hand, people looking for solutions, browsing, searching and navigating, are also users that have had an experience with the forum. They should not be neglected in that sense but I think there is ways to work around that and perhaps have surveys designed for those who browse (and it could be periodically) and for those who post and receive a response from the community.

 

We use Qualtrics to receive customer feedback and have an algorithm and rules for when and where the survey pops.

The response rate is quite good, especially for our largest language English.

re: Overall increase/decrease..do you mean the actual result or response rate? We see numbers don't really fluctuate unless we make drastic changes in methodology. Since the feedback is very broad and mostly about the brand & products that's in line with my expectation.

 

Cheers

Wendy

Learning from others and helping where I can!
Community Passionista!
0 Kudos
Reply
Loading...
Honored Contributor Honored Contributor
Honored Contributor

Re: CSAT and helpfulness scores at more mature communities

Morning @edgiansante

 

Hope all is well. Quick question to you since you're using Qualtrics too 

 

Re:

I then ran a 2nd test: inviting people to fill out a survey (using a qualtrics survey) with the exact same questions as the OOB one but after they received a response from either a member / super user or a moderator.

 

How did you actually do this? Did you send an email to users after they received a response with a link to the survey? Are you making it part of the "reply received" notification. If you were able to pop up the survey ONLY for users that got a reply I am very interested to learn about this and promise to bring you Stroopwafels should we ever meet (the stroopwafel bribe never failed me Smiley Wink)

 

Cheers

Wendy

 

Learning from others and helping where I can!
Community Passionista!
Tags (1)
0 Kudos
Reply
Loading...
Frequent Advisor
Frequent Advisor

Re: CSAT and helpfulness scores at more mature communities

Love stroopwafel Smiley Very Happy

No, what I'm doing (manually thus far but planning to automate in our next round of iteration) is to invite people to fill out the survey when it's marked as solved on the thread itself, rather than email.

That means that I'm not only surveying the OP or people subscribed to the thread but rather anyone who ever sees that 'solution'.

That's my theory: we're serving multiple users, not only OPs (otherwise all posts would be private and not public!). That means that I should survey everyone who's benefiting from that answer.

Hope it makes sense?
Reply
Loading...
Honored Contributor Honored Contributor
Honored Contributor

Re: CSAT and helpfulness scores at more mature communities

Makes a lot of sense! Love to try it out. I guess my question then is the how....what process do you then use to invite all those who benefitted from that solution? Can you tell a little more about that?

Thanks Ed.
Who doesn't love stroopwafels Smiley Happy


Learning from others and helping where I can!
Community Passionista!
0 Kudos
Reply
Loading...
Frequent Advisor
Frequent Advisor

Re: CSAT and helpfulness scores at more mature communities

I manually invite them (through a post after the solution) to fill out the survey. I make it in a generic way so anyone who reads the thread will see the 'invitation' to fill out a survey at the end of the conversation.

It's not beautiful and the fact that I have to manually post (so it comes from 'Ed') might be skewing things (for good or bad). My plan is to add this in a more programatic way so when a thread is marked as solved, there's a link added at the end for people to fill out the survey.

This is an example of how it looks now: https://www.dropboxforum.com/t5/Syncing-and-uploads/how-do-I-kill-a-backup-job/m-p/214262#M25885
Reply
Loading...