The truth - shoddy statistics, dressed up as analyst insight.
If you follow the Low-code platform market, then you have probably seen this supposed league table, recently promoted with the incendiary title - "Lots of Low-code vendors have already been rejected" ...
At first glance the report seems sensational. Would Forrester really ‘spill the beans’ if they acquired these metrics through their vendor research? I think not – there’s no sense in 'poisoning the waterhole'.
So, who’s behind this story and what’s their motive?
The original report was commissioned by Appian, a long established BPM vendor, that started calling themselves a Low-code platform in early 2015. That was about the time Forrester said …
Little if anything of substance
Having read all 24 pages of their report, I didn’t see anything in it that would have required a bespoke survey to glean. It’s pretty much all analysis that you could read in previous Forrester reports.
Personally, I'd be unhappy to commission an expensive, bespoke survey report, and have so little that’s new, or insightful to show for it.
Put yourself in Appian’s CMO’s shoes for a moment ... would you gladly pay for these insights?
- Lack of funding, skills, and stakeholder buy-in can derail low-code implementation efforts.
- Unstable and insecure software are a risk factor you’ll want to take into account
- Skills to build apps are a prerequisite for success.
And who’d have thought...
“Digital transformation involves streamlining and modernizing processes.”
And in case you didn’t know; “Digital customer engagement is about improving customer satisfaction”.
What do we really learn from this report?
The one thing I deduced was how generous Appian’s marketing budget must be! This report probably absorbed over $30,000 of their marketing budget. Don’t get me wrong, Appian do some great marketing - the Barclays case study video that you’ll find alongside their Forrester report is certainly impressive. But this report is not Appian’s (or Forrester’s) finest hour.
Trying to make a silk purse from a sow's ear
Appian have probably generated a shedload of marketing leads, from muppets (like me), who parted with their Email address for the privilege of "reading the bleeding obvious”.
But, perhaps realising the frailty of the report, Appian decided to create a spin-off article, based on the survey responses that Forrester gave them. The result, the aforementioned sensational adoption statistics, published as a news article on The News Stack.
I wonder how Forrester feel about having such nonsense derived from their survey?
So, what’s my beef?
Until you read the small print you’re likely to assume that this adoption chart reveals how the various vendors are performing in the market. But, Forrester hasn’t got insight into these vendors’ sales pipelines. Neither do they know what evaluations are really underway, and which were lost.
So, all this chart shows us, is the response of the 209 North American end user companies, who we can assume were asked a question rather like this:
The experience of just 209 companies tells us little of how these vendors are really progressing. The picture will have been skewed according to which companies responded to the survey.
Many more customers of the larger Low-code vendors would have been on Forrester’s contact list, as a result of recent Forrester Wave and Market Landscape research surveys.
- Is it likely that Forrester invited these customers to participate in this boutique survey? Yes.
- Is it likely that these customers made up a large proportion of the responses? Yes.
- Would this have skewed the adoption league table? Of course.
In evidence – 49% of respondents had installed either Outsystems (22%), Salesforce App Cloud (19%) or Appian (8%).
This sample bias it’s bound to detriment the position of the alternative vendors that were evaluated, but not chosen.
Are most Low-code vendors “in for a world of hurt”?
In his article, Lawrence Hecht at TheNewStack says:
“Most of the other 13 vendors asked about are in for a world of hurt. Except for one vendor, all had been evaluated by at least 40 percent of respondents. Yet, a majority of people who knew about one of these vendors had already evaluated and rejected them.”
Lawrence who has apparently been producing research reports about information technology markets for the last fifteen years, seems to have developed statistical myopia while writing for* Appian.
(* The only reason he got these survey responses to concoct something from, is that Appian gave them to him and commissioned his article.)
The league table is horse shit. Here’s why.
Had the 49% of customer survey responses come from smaller vendors like MatsSoft, (unlikely I grant you, as they’re tiny compared to Appian), then the “world of hurt” would rain down on whichever vendors they’d beaten in the past few years.
On the subject of the recently acquired MatsSoft, they’re to be heartily congratulated for being under evaluation by 18% of these North American enterprises. Well done! That’s on a par with Salesforce and Outsystems, which if you know anything about the relative size of these companies makes no sense at all.
The league table tells you little, if anything, about these companies’ prospects. Take Bizagi for example. They’re several times bigger, than many of the firms ranked above them, in this league table. The low number of implementations and active evaluations is a factor of the survey sample, rather than anything more meaningful.
Bizagi’s recently announced $48M investment from Invus is a more reliable indicator of their merits. Expect Bizagi to be a formidable contender in the Low-code market in the next couple of years.
Are Outsystems having the last laugh about this report; or embarrassing themselves?
The league table may be nonsense, but it has obviously delighting folks at Outsystems, who have enthusiastically acclaimed their top position for combined implementations and active evaluations.
They probably can’t believe their luck. It’s not often one of your main competitors pays for your marketing. Thanks Appian!
In fact, seeing the 600+ shares Outsystems’ article has had, they’re probably outgunning Appian when it comes to peddling the horse shit statistics that Appian paid to produce. (Just-deserts?)
But, in truth, it’s all a bit embarrassing, as Outsystems’ article bloats the veracity of the league table, as though the underlying statistics had merit…
“… the article showcases the huge variance in vendor adoption across the low-code market.”
No it doesn’t you muppet! It just shows the variance amongst 209 respondents, skewed as you had 22% of the installs.
For the league table to have any veracity, you’d need insight into each of these companies sales pipelines, or need a randomised survey of a few thousand companies, not 209.
“One of the most important take-aways can be seen in the purple bars, evaluated and chose not to implement.”
Nope, I’m not convinced. Neither you, or I know what is meant by “Evaluated and chose not to implement”.
- Was the evaluation a cursory glance, or a detailed trial?
- Did the vendor even track these as opportunities in their sales pipeline?
What strikes me from the chart is that Outsystems had the highest percent of installs (22%), and therefore it’s no surprise that they had a lower percent of evaluation failures. Every other vendor (except Bizagi and FileMaker) suffered 30% (+-3%) evaluation failures, which is unremarkable.
Bizagi’s poor performance is either a factor of the small sample size, or the fact that their Process Modeller and Studio products are freely available for download. In recent years that generated many more evaluations than they could properly convert into managed sales opportunities – particularly in North America. Expect that to change following their recent investment.
Leaders or lemmings?
There’s much to admire in Outsystems marketing. Frankly they’ve been pretty much “knocking it out of the park” compared to most of the other vendors’ content marketing efforts.
But in relation to this article, it strikes me as full of cheap shots:
- Calling your competitors “laggards and lemmings” is beneath you.
- Dressing up Appian’s / Forrester’s report as having some kind of statistical veracity is naïve.
The lemming-like behaviour here, may be leaping onto Appian’s horse shit league table, and dignifying it as Forrester’s research. Although I have a slight admiration of your chutzpah, neither Outsystems, Appian or for that matter Forrester look good over this.
Subscribe to low-code-daily.com and never miss a piece of Low-code market news.
Want to comment? See linkedin.com/pulse/most-low-code-vendors-world-hurt-says-who