Conventional wisdom tells us Community Owners should rely on two key metrics to track the success of an online community: Membership (number of registered users) and participation (number of active users in a given time period). That’s well and good but what about measuring the health of the community, not just its size?
Back in 2010, our team at the University of Phoenix created PhoenixConnect, one of the largest academically-focused communities in the world with over half a million registered members, 40% of which are active participants every month. Moreover, students and faculty members who participate in the community spend an incredible 25% more time-on-site than other their peers who are not. This additional time-on-site is incremental, meaning that it does not cannibalize time spent in the classroom. As satisfied as we were with these numbers and their impact on our student’s engagement, we realized early on that we were only measuring the impact of the community on overall student behavior but not the behavior inside the community itself. The first step towards closing this gap was understanding who were these different types of users and what key indicators should we use to track their behavior within the community?
The Contribution Framework
Back during the “dawn” years of social media, people like Bradley Horowitz (then with Yahoo!) and Jakob Nielsen were already tackling this question and pointing out the participation inequalities between community participants, eventually coining the term “participation pyramid” to denote a 1:10:100 ratio between those creating content (creators), those engaging with content created by others (synthesizers) and those simply consuming said content (consumers). The major takeaway both Horowitz and Nielsen left us with back in 2006 was that as social media matured, it was imperative to remove barriers to participation in order to drive growth across all three user segments.
Figures 1 & 2: Horowitz’s Contribution Pyramid and Nielsen’s Pyramid
Our team liked the contribution model outlined by Horowitz and Nielsen so we used it as the basis for our Community Health reporting framework, iterating along the way. The result is encapsulated in the three maturity stages described below.
Stage 1 – Creating the Pyramid KPI and Segmenting Communities
First off, we decided to segment out our KPI reporting into creators, synthesizers and consumers – a simple enough task given the fact that our Jive SBS platform clearly reports the number of discussions, replies and views for each community. During the first iteration of our community we did not have the right instrumentation in place to measure unique contributors so we settled on the number of discussions, messages and views for a community as good enough proxies for creators, synthesizers and consumers respectively. We plotted the data (messages/replies per discussion threads and views per discussion thread) for our four major communities and came up with the following visualization aimed at allowing us to understand the distribution and behavior of users within each community:
Figure 3: November 2010 Contribution “Pyramid” for 4 major PhoenixConnect communities
Which community do you think is the most balanced? This is our interpretation and how we use this data to drive decision making and investment:
- Support Communities seem to be under performing relative to its peers but we quickly abstracted that this community is clearly transactional in nature (question-and-answer) so we would expect a much shallower degree of engagement evidenced by low replies/views per discussion numbers.
- Academic Communities have the highest synthesizer ratio (messages per discussion thread), which is great since these communities are aimed at encouraging constructive academic discourse between our students and faculty. However the consumer ratio is the highest of the bunch (126 views per discussion), which implies people are viewing the discussions but are not as incented to participate or engage. This insight helps us drill deeper into the behavior inside these communities to understand whether or not we have a content quality problem.
Stage 2 – Removing Barriers to Contribution and Consumption
As we discussed earlier, Horowitz and Nielsen both argued in favor of “removing barriers to participation.” In early 2011 we made two major changes to our community platform that addressed this specific problem:
- We upgraded from Jive 4.0 to Jive 4.5 in order to take advantage of the improved contribution features, particularly the revamped activity feed and the “Like” button (which improves ultra lightweight participation)
- We surfaced the activity feed to the home page of our student portal in order to increase awareness and consumption of community content.
After releasing these changes, we adjusted our KPI metric slightly (to account for “likes” as indicators of synthesizer contribution). The results can be seen below:
Figure 4: August 2011 Extended Contribution “Pyramid” for 4 major PhoenixConnect communities
As you’ll notice at first glance there was a dramatic shift in the ratios: contributions from synthesizers and consumers tripled thanks to these two key enhancements. In short, we were able to see the return on investment in both a Jive upgrade as well as a custom activity feed widget on our student portal homepage.
But aren’t these ratios misaligned? The quick answer is yes. Like many first generation communities out there, PhoenixConnect heavily relied on user-generated content (UGC) to seed conversations. UGC is great to get a community off the ground but it’s far less effective when it comes to creating a steady seeding of relevant, on-topic and insightful conversations at scale. Our ability to generate fresh, engaging content could not keep up with the new found demand from our community. This data-driven insight was used to inform and adjust our community strategy particularly around content creation through a two-pronged approach:
- Create a community content team responsible for generating monthly editorial calendars that will generate a consistent source of new content our community can congregate around
- Break down barriers to content creation by introducing new objects beyond discussions. These include capabilities such as blogs, ideation and most importantly, the upcoming Jive 5.0 What Matters stream to bring improved discoverability and relevance to status updates
Stage 3 – Disseminating Insights and Keeping your Team Accountable
Constantly pivoting and course-correcting is a must for any Community Owner; however it can be a dizzying prospect for other people in your organization who don’t care or don’t need to care about the details. In order to keep a consistent KPI metric for community health that can be disseminated inside of our organization and easily interpreted, we created a simple “score” based on a contribution-pyramid inspired algorithm:
We made sure we weighed this score against our all-time highest activity day to give us both an easy-to-read percentage as well as a good frame of reference for how active our community is at any given point in time. The graphic below gives you an example of how we report “Health” for a particular PhoenixConnect community:
Figure 5: August 2011 Community Activity Score for a PhoenixConnect community.
Note that the huge jump in activity in March 11 was a result of the enhancements described in Stage 2 above and the ensuing “shiny penny” bump consistent with new features.
Each and every community out there will have its own particular intricacies and you organization will surely require you to adhere to its own KPIs and reporting frameworks. In our case, the content contribution pyramid-inspired reporting model was a very valuable addition to our reporting toolbox. This KPI enabled us to understand variations in context, purpose and participants within each of our communities while keeping an eye on overall growth trends.