Deeper Facebook Engagement: Dissecting Interactivity

[#1]

Michael Wu, Ph.D. is Lithium's Principal Scientist of Analytics, digging into the complex dynamics of social interaction and group behavior in online communities and social networks.

Michael was voted a 2010 Influential Leader by CRM Magazine for his work on predictive social analytics and its application to Social CRM.He's a regular blogger on the Lithosphere's Building Community blog and previously wrote in the Analytic Science blog. You can follow him on Twitter at mich8elwu.

---

It’s been about 3 weeks since I last blogged on Lithosphere as I’ve been busy traveling around UK and Italy for both work and play. The play part involved my wife and I traveling around UK and Italy, sightseeing and enjoying delicious food and fine wine. In UK, we visited Oxford, Stratford-upon-Avon, and Bath. And before we left for Italy we even stopped by Salisbury to see Stonehenge. Then, in Italy, we toured Milan, Florence, and Rome. And of course we couldn’t miss visiting the Vatican.

The work component of the trip involved me speaking at a number of conferences, business meetings, social media events, workshops, and interviews. I talked about a wide range of topics from social CRM, psychology of gamification, cyber anthropology, science of influence, the Facebook Engagement Index, as well as some of the more technical topics, such as machine learning, social network analysis and predictive social analytics.

Among the topics I presented, gamification was by far the most popular topic. It was heavily tweeted and several excellent blog articles resulted from my presentation at Digital Surrey (a very engaging not-for-profit community of digital professionals).

1. The Science of Gamification (@mich8elwu at #digitalsurrey) by Mark Wilson
2. The Science of Gamification on the gamification blog by Gabe Zichermann
3. The Science of Gamification at DigitalSurrey on GameTuned by James Monjack
4. Want to change behaviour? Pull the trigger on Strange Fascination by Jane Franklin
5. Benjamin Ellis also took some very nice photos at the event

Now that you know where I’ve been, let me return to the topic of Facebook engagement. Last time I showed you the structural similarity between a Facebook fan page and a community. By treating fan pages as communities, we can develop a whole spectrum of engagement metric from the very shallow (level 0) fan count to something that is eight levels deep. And I talked about the first two levels in my last post.

• Level 0: Total fan counts
• Level 1: Active fans
• Level 2: Interactivity (through comments) – Commented post fraction.

Today we will examine several deeper level engagement metrics.

Disentangling Interactivities

Since the Level 2 engagement metric looks at what fractions of the posts were interactive (i.e. commented), Level 3 hones in on the interactive posts and tries to quantify how much interaction took place in those posts. This is traditionally characterized by a metric called thread depth: the number of comments a post receives. I computed the average thread depth across all posts within a fan page and plotted the distribution on a log scale in Figure 3. The median level average thread depth is about 12.5, meaning that posts on fan pages receive about 12 comments on average.

Although the average thread depth is simple way to estimate the amount of interaction, it doesn’t go deep enough to distinguish who you are interacting with. So it cannot tell you whether it is the same fan posting 100 comments or 100 different fans posting a comment each. The latter is clearly more desirable because it means more of your fans are interacting with each other. So for the next level of engagement (Level 4), I computed the average number of unique fans per conversation (see Figure 4).

Here, the median level for the average number of unique fans per conversation is about 11.7. This means that posts on fan pages typically receive comments from 11 other fans (not counting the initiator of the conversation). Notice that this value is very close to the median level of the average thread depth. This observation suggests that most of the fans only post once within any conversation. This, as we shall see in the next post, will have significant implications in terms of the conduciveness of fan pages as a suitable environment for building relationships.

The Dynamics of Interactions

If your fans are engaged enough to post, comment, and interact with other fans, that is great. It is already quite an achievement already, but we are not done yet. The full spectrum of engagement can be very deep. The next level of engagement (Level 5) goes a step further and looks at the dynamics of the interaction between fans. That is, the timing and velocity of how fans interact on your fan page.

Having a fan who interacts with 10 other fans through 100 comments may sound excellent, but you might change your mind when you found out that it took them over a month to respond to each other. I computed the average response time between every posting and show the result in figure 5a. The median level average-response-time is about 9.8 hours. Since the response time data has large variance, I also computed the distribution of median response time between every message (figure 5b). Although the shape of this distribution is similar to that of average response time, the distribution is shifted. Now the median level of the median-response-time is only about 2.2 hours.

Do you know where your fan page stands among these distributions? Are you beating the median level? And if so, by how much?

Conclusion

Besides giving you a little coverage of my recent European speaking tour, we returned to the subject of Facebook engagement. Today, we dug deeper to understand how fans engage within a fan page by looking at thread depth, the unique number of other fans they interact with, and the dynamics of these interactions. By doing so, we covered three more levels of engagement metrics. So our spectrum of engagement is five levels deep now.

• Level 0: Total fan counts
• Level 1: Active fans
• Level 2: Interactivity through comments
• Level 3: Thread Depth – amount of interaction
• Level 4: Unique fans per conversation – with how many other fans?
• Level 5: Average/median response time – dynamics of interaction

The Level 3 (thread depth) and Level 4 (unique fans) data suggest that most of the fans only post once within any conversation. We will see if this is indeed the case next time. This is a very important point, and we will discuss some of its consequence in subsequent posts.

Although we are already at Level 5, we are not yet at the bottom. There are three more levels to go! In subsequent posts, I will reveal more data on the deeper levels of engagement. But for now, let me know what you think about this spectrum of engagement metrics. As usual, kudos, comments, suggestions, critiques, and discussions are always welcome. Stay tuned for even deeper level of engagement...

And by the way, I will be participating in the Big Data Analytics for Social Media panel at the Enterprise 2.0 Conference in Boston next week. So if you happen to be in Boston from June 21--23, then we are probably destined to meet.  :-)  Please stop by and say hello. See you next week in person, or till next time on Lithosphere.

[#2]

Hi Mike!

Thx for great next step in clearing FEI! 

I still have 1 q: you suggest to measure "dynamics of the interaction between fans" by measuring response time. I meet same problem by analysis comments of different fans. I mark simply the fact of conversation in the thread as attribut of whole thread.

But how do you define that their comments are inter-actions between fans (conversation fan-fan) and not simply reactions on wall post (brand-fan or fan-author -- fan)? Or do I understand something wrong? 

[#3]

Hello Andrei,

Glad to see you back. And thanks for the question.

It is true that when a fan post a comment in a thread, he can be responding to the initiator of the thread, or the fan who commented the thread right before him, or any of the other fan's comment in the thread. However, for the purpose of measuring dynamics (velocity of participation), this is irrelevant.

The dynamics of the interaction is simply measures the rate at which fans participate within a thread of conversation. And since it is all within a thread there is interaction. Regardless of whether it is interaction with the initiator or other commenting fans, it is interaction. So that is what is what this metric intent to measure. In later post we will dig deeper to determine whether there are really interactivity between fans.

Alright, I hope this clarify the Level 5 engagement metric that is presented here. Thanks again for asking the question. See you next time.

[#4]

Hi Mike!

Thx for explanation. 

I don't know what next levels of engagement you will measure, but for me is very relevant next question. Does primary (it means - the earliest comments in the thread) amount and velocity of interactions between (given unique fans - as next logical step for measuring social engagement drivers) fans influence more unique fans engagement in the discussion later.

I think fans, when they see in the feed on the given post a high amount of comments and high velocity of these reactions on the post - these fans are more motivated to participate on discussion too. 

And I think you can measure it correlation on the data. Do you? 

[#5]

Hello Andrei,

Thank you for coming back and continuing the conversation.

Well, as I said before, you will just have to wait until next time before I reveal the deeper metrics.

However, I like to emphasize that the Facebook Engagement Index (FEI) is purely for measuring engagement, and only engagement. It has no component that measures influence. That is a compltely different metric I implemented based on Social Network Analysis (SNA).

At a really high level, definition of "Engagement" is the ability to capture someone's attention so they spend time and effort on whatever that is engaging him. And "Influence" is the ability to change someone's thoughts or behaviors. They mean something totally different, so we want to keep the metric for each one separate from the other. We need to keep these concepts strict and distinct so people don't abuse the terms and talk around it with inprecision. This is a danger and a mistake that many people make that result in bad analytics. They conflated too many concepts into the metric. As a result, it just become something very fuzzy and don't tell you anything precise. So it will end up being less useful than a super strict, single-purpose metric for quantifying a single concept.

Your speculation is probably true, and there are correlation between high rate of post and the probability that another fan will post. But keep in mind that this is correlation, not causation. It is very hard to prove causation with statistical data analysis. It could very well be that because people are motivated about the topic, that is why they participate in such high velocity. And it is not because of the velocity of post that motivated the subsequent user to post. You can go into a whole course in statistics just on method that establishes casuality. So I don't want to bore the other readers here with that...

Alright, thanks again for the question. And see you again next time. More data and metrics coming...  ;-)

[#6]

Mike!

Thx for explanation of differencies between Indluence and Engagement. I will use these concepts like ypu propose. 

I understand what are correlation and causation in statisticall sense - have some expirience in analysis of sociological data with SPSS. And I suggest some set of attributs can engage user to participate on conversation. Millward-Brown has made analysis like this for video views hier. They investigate set of 4 drivers.

What do you think about this approach (using a set of drivers that have benn statistically meassured with correlation)?

[#7]

Hi, Michael -

Enjoyed your Gamification talk at Digital Surrey - before the event I was a bit sceptical that it would be over-enthusiastic, as there's a lot of that about, but I found it to be balanced and insightful.  It also incorporated a perspective from behavioural economics which in retrospect is 'obvious' but isn't often included.

Is there any way you can make the talk downloadable?   I would like to cite some elements of it in a presentation I am preparing.   

regards,

Heather

[#8]

Hello Andrei,

Apologize for the late reply. There are two urgent production issues that I've been assigned to last week, and that delayed my interaction on the blogosphere.

I've look at the paper you linked. So are you suggesting that we use the 4 attributes mentioned there to drive engagement? I think those 4 attributes are just the start, and there are likely many different ways to engage different groups of people in different ways. That is the whole idea behind gamification, which is not limited to a fixed set of attributes. The game mechanics and dynamics can be string together in creative ways to make new ones.

If there is anything we know about human engagement, that would be people are adaptive. So people get tired and bored quickly if they are engaged the same way over long periods of time. There are probably infinite number of attributes that drive engagement. I wouldn't limit to just the 4 they mention. Those 4 would work for a while until people adapt, then you will have to think of new ones. So it is better to understand why and how people engage, which will guide your engagement strategy in the long term, rather than just focus on tactics of engagement, which may only work for short term.

Alright, thanks for the comment, and see you next time.

[#9]

Hi Michael,

Enjoying your posts on FB engagement.  As I read through your post, a couple of things jumped to my mind.

One is of course the unique fans per conversation. I know you said you will be talking about it more in your future posts but wanted to share some of my own thoughts on this.  Ideally, I like an interaction where a fan has posted more than once as that to me signals a true exchange of thoughts. If that is not the case, the second tier would be the 'quantity' of interaction (a proxy could be # of words). My logic here is that if someone responds with a 1 or 2 word post (e.g 'Wow', 'Thumbs down','Great' etc.) - it is an interaction  but really does not convey much in terms of why/why not etc.  Of course, one could debate the semantic/sentiment value.

The other area is the 'response time' between posting. I agree with your thoughts here and the analysis you did. However, I did have a question - Wouldn't  the 'validity' of the response times depend on your fan page objectives and what the response is for - e.g product feedback,  crowdsourcing ideas, brand awareness initiatives etc.. I was thinking that in some cases you want a faster respons time but in some cases a response is good enough and the time is secondary.  Thoughts?

Again, enjoyed the read.

Ned 

[#10]

Hello Heather,

Thank you for stopping by and commenting.

I'm glad to hear that my presentation on the "Science of Gamification" has been insightful to you. Coming from academia, it is rather surprising to me that many new ideas/concepts in the industry are driven by hype without much validation. That was what motivated me to do some deeper research into this topic.

By popular request, I've made my deck downloadable as a pdf. That is as much as I can do for now. Please feel free to share the pdf. However, I like to reserve the right to the original powerpoint file.

The deck is available on SlideShare:

http://www.slideshare.net/mich8elwu/2011-0526-digtali-surrey-science-of-gamificationv03

Alright. Thank you for your interest. Hope to see you again on Lithosphere in the future.

[#11]

Hello Ned,

Nice seeing you here again.

You are absolutely right that repeated comments are more desirable than just a single comment. And that is precisely what I will be talking about in the next few posts. I guess you been following my blogs long enough to be able read a bit of what's in my mind.  ;-)   Let me just whet your appetite by saying that the data there is quite surprising. At least it was a surprise to me.

Concerning the use of word counts as a metric, I have some bad experience with them. And the reason is that it is too easy to game. People can start to ramble about junks that may or not not be related to the topic of discussion. I tend to stay away from word counts. It may seem like a good idea if you look at existing data before it become a metric. But once you make that a measurable quantity, and make that visible to others, you change the psychology of the user. And even perfectly honest who have no desire to game the system, maybe motivated to game the system. So I would just watch out for using word count as a metric in open environment like the fan page where people can do whatever they want.

It is true that response time depends on the goal of your fan page. But you must also remember that the Facebook Engagement Index (FEI) is meant to measure engagement regardless of what your goal is. So even though for some fan pages, having any response is good enough, those fan pages have lower level of fan engagement than pages that have high rate of response.

I hope this address your question. And hope to see you again. Stay tuned till next time.

[#12]

Michael,

I agree with you about the 'risk of gaming' the metric. But you also bring out another important point - "But once you make that a measurable quantity, and make that visible to others, you change the psychology of the user".  

The key  point here is that folks might get affected and change behavior once they KNOW what the metrics beings used are. And while I am not at all encouraging the use of word count, I think it helps to have multiple levels of metrics - some transparent to the end-users and some not,  with each being cross-corroborated to make the overall engagement metric more robust.

Looking forward to your next post.

Ned

[#13]

Hello Ned,

Thank you for continuing the conversation. This repeated engagement is definitely desirable.  ;-)

What to measure is a tricky subject. It is like quantum mechanic, the act of measuring, changes the value that you try to measure. Being an ex-physicist, this is not new to me, but many people find this hard to believe. So it is great that we are in agreement on this point.

One must be careful when measuring several correlated quantities, because sometimes these metrics work in conjunction (like a logical AND), yet other times in disjunction (like a logical OR). The challenge really comes down to how you combine these metrics to give you a meaningful result at the end. Simply taking the average is not sufficient, because there is no particular reason to use the arithmetic mean over the geometric mean, or other more generalized mean.

As you might know, couple years ago, I've developed the Community Health Index (CHI), and this is precisely the kind of techniques I've used when computing CHI. It is, however, a nonlinear model and many people in industry are having hard time understanding how to interpret it.

Anyway, Thanks for the suggestion though. Great observation as always. See you next time.

[#14]

My view is too much time is wasted in talking about deeper interactions monitoring and analysis

Where is the real value to business ,In todays economy business are looking for true ROI.

we are questioned every day by customers.

[#15]

Hello Ram,

Thank you for the great question.

That is a very valid point. But without accurate measurement of engagement, there is no way to infer ROI. Measurement is always the first step. An accurate measurement can then be model as predictors with the transactional data on the right hand side of the equation. Then, using pretty standard machine learning techniques, we can see whether there are any engagement variable that predicts transactional outcome or any business KPI.

Many people do a sloppy job in measurement, then even if there is really a correlation between engagement and ROI, they would not be able to discover that because the effect is masked by the noise in the sloppy measurement. So developing a very precisely defined and accurate measurement of engagement is to enable subsequent analysis and quantification of ROI. That is how predictive science works. Everything comes in steps.

If you can get me some interesting transactional data, sales, click through, etc. I'd be happy to do the analysis for you and show you how well engagement correlate with these transaction, whether it is sales or other business KPIs. It should be possible to show whether there is correlation, how much, how predictable with pretty standard statistical methods.

Alright, thank you for bringing up this important point. I hope to see you on Lithosphere again.