A Review of the Elearnity 9-Grid
In the first of a series looking at published research findings and statistics in the e-learning space, Peter Phillips takes a look at the Elearnity 9-Grid™ True Grid? Elearnity are a long established and well-known learning analysts and consultants, who provide independent research in the UK and European Learning and Talent market, “drawing upon the insights and experience of our Corporate Research Network”. You may have seen their recently published research results in the form of the Elearnity 9-Grid™. Here for example is their Grid for bespoke e-learning vendors. Follow this link for the full report. The report originally came out in July 2013, when Epic were in the process of an AIM listing. Here is a quote from their official Admission Document. “Epic was rated first for ‘‘Potential and Performance’’ by Elearnity in July 2013.” From a glance at the chart above, I suspect that is what most of us would have said if our bespoke e-learning services had appeared in the top right corner. Elearnity retweeted the Epic statement without any qualification. The 9-Grid™ is a serious and considered piece of research. But how meaningful is the information in the way it is presented? Does it really support Epic’s statement? Or is the medium obscuring the message? As a statistician I take a particular interest in the use and misuse of statistics, so let’s take a closer look. “When I use a word”, Humpty Dumpty said, “it means just what I choose it to mean” My first query is with the terminology. Here is another 9-Grid™, this time for LMS vendors. As you can see, CornerStone is the only vendor in the “Strategic Leader” section, high in both Performance and Potential, while only Cornerstone and NetDimensions are rated as strong performers. Naturally Cornerstone reported this accolade on their web site and on Twitter. Why wouldn’t they? What does ‘Performance’ mean to you? Perhaps the build quality, range of features or ease of use of the LMS, or some measure of fitness for purpose? In Elearnity’s model, ‘Performance’ is defined as a combined measure of: a) “How often do vendors get shortlisted and how often do they win?” and b) “Corporate customer advocacy” My own view is that (a) is a rather sweeping assumption that the amount of new business chased and won by competitive tender is a valid performance measure. Doesn’t this statement more reflect business strategy and marketing spend, not necessarily performance? Many LMS providers will say only a small proportion of their annual revenue comes from new business won through competitive tender, and some avoid that route altogether. Point (b), I believe, is more appropriate in principle. There should be some correlation between performance and customer satisfaction, but this is more difficult to measure, and the data are unlikely to be statistically valid. The Elearnity model combines these two elements to arrive at a simple overall score of 1, 2 or 3 for Performance. Only Cornerstone and NetDimensions achieve a 3 in the latest published 9-Grid™. Does that make them the best performing LMSs in Europe, or does it suggest they have the biggest marketing budgets? Equally, in terms of investing in bespoke e-learning, many clients might be more interested in a provider’s instructional design skills, subject matter expertise, creativity, and delivery within budget and on time, than in whether they get on lots of shortlists. Potential The other major axis on the 9-Grid™ is ‘Potential’. Again, what does that mean to you? Future growth prospects of the vendor perhaps, or their product development plans? Or is it the potential of the product or service to deliver added value features? For bespoke e-learning, the Elearnity measure of ‘Potential’ is a combination of:
- “the breadth of sector focus such as Finance, Retail, Telecoms etc.”
- experience “developing for different learning contexts”
- “sophistication” of learning created.