Oftentimes, it’s what you don’t know that really contributes to poor decisions.

If you’ve read my previous blogs you know by know that I am a avid reader. I often like to read about things unrelated to my work. The latest book is Gulp: Adventures on the Alimentary Canal by Mary Roach .

It’s about how the digestive system works, not about diets or food itself, just the mechanics of the system. There are tons of interesting and truly funny chapters (really!) but what caught my attention was the possible origin of the Fire-Breathing Dragon (Spoiler alert.)

Often hunters would bring back reptiles that had just eaten a large meal. This meant that the reptile was sluggish and unaggressive, easier to catch. Hunters also got the bonus of extra meat, both the reptile and whatever it had eaten. They would lay the reptile near the fire, and the contents of its stomach would let off flammable gas that ignited, making it appear that the reptile was breathing fire. It was logical to conclude that it was a fire-breathing dragon, but the assumption was wrong.

How does that relate to our work at Mobile Pulse? We offer services that allow organizations to measure Mobile Carrier performance and coverage. Today, many IT and procurement professionals are under the perception that they already have the tools to understand their carrier’s coverage and performance, when actually what they are looking at is not performance data, it’s really fire-breathing marketing.

Let me explain. I’ll cover three common tools organizations use to select coverage for their employees; bars on devices, coverage maps from the carriers and NTIA data on cellular coverage from the FCC and US government. If you’ve looked at all three you might think that you’d done an in-depth analysis for your company and your mobile employees. The truth is, you might as well tell everyone you saw a fire-breathing dragon, because the facts are quite different.

Let’s start with the signal bars on cellular devices.

There are no standards for this feature, therefore each OS developer decides what scale to use on their devices. Carriers and manufactures want their customers to feel like their device is performing well and has great reception, so there is a bias towards making it seem that the device has a great signal.

Think of it as a teacher who doesn’t want contentious parent meetings so they give everyone with 50% or higher on a test an A. Parents think their kids are doing great and everyone is happy. But the kids are failing.

Fire-Breathing Dragons And Mobile Carrier Performance

As you can see by the graphic, even within a device OS, the bars mean very different things. Below we are looking at Apple iOS 4 and iOS 4.1 the signal strength indicators are set at different levels. As you can see, 5 bars on an iOS 4 device now displays as 3 bars in iOS 4.1. Bars are not reliable indicators of signal strength.

Next, coverage maps from the wireless carriers.

I think the best this is to show the disclaimer you can click on when you go to the AT&T coverage map. On their disclaimer it says, “These maps are not intended to show performance on their network”. It’s on their website. Enough said.

Fire-Breathing Dragons And Mobile Carrier Performance

Finally, what about the FCC and the NTIA?

The NTIA collects data from the states and the states receive their data from the wireless provides (yes, that means the carriers) biannually.

Fire-Breathing Dragons And Mobile Carrier Performance

The wireless carriers report only the maximum “advertised” speeds their network is capable of supporting. Think about reporting the speeds of a highway if there is no traffic or weather, its 75 all the time. This is also supplemented occasionally by drive tests and reports from citizens. Again, this is advertising.

Where to find independent ratings of mobile performance?

So, while you have really taken the time and looked at several different sources to select the best carrier, the data is just not correct, it marketing. As dependence on mobile networks continues to grow, performance becomes more important to organizations. If you are looking for real-time coverage maps, signal strength ratings that you can set yourself, and the data from your employees devices is critical in selecting the best carriers and negotiating SLA’s for your organization, Mobile Pulse can give you the data you need. Accurate data is critical because you don’t have time to chase fire-breathing dragons.