Lack of Evidence in Medical Apps: The Solution

In our initial post reviewing the recent article in Evidence Based Medicine (BMJ Group), we provided an overview of the problem as explained by the authors of this paper.  In this post, we summarize their recommended solution for establishing quality control, safety, and clinician confidence in the utilization of medical apps.


For reference, here is the article again:   “Medical apps for smartphones: lack of evidence undermines quality and safety” that was published in Evidence Based Medicine in August 2012 [ePub ahead of print - not open-access]:

PUBMed Link:


Article Review Part II: The Solution 

(See Part I here)


We discussed some of the concerns that were raised regarding authenticity (the credentials of the author, reliability of the information, lack of references, security, etc.)  I also wanted to reiterate some of what was mentioned in the comments and discussion following this first post.  One of the points being that there is (and has been) a proliferation of textbooks and medical handbooks available for sale in bookstores and on the Internet.  This is important to remember, because  although the author’s name is always transparent, there is still no guarantee as to the nature of the content, how well it has been referenced, or whether it represents best-evidence content for its subject.  There is still significant controversy over various aspects of medicine, and we certainly have not arrived at a place in medicine where consensus on everything is universal.  Additionally I mentioned the list of online calculators and resources that many clinicians access, which currently have very little if any regulation or validation process.  As we jump forward to make a case for the need to validate/regulate apps, let us remember the current state of affairs.


The authors of the EBM paper discuss possible solutions around various issues, but overall seem to make the case for a multidisciplinary approach, involving various stakeholders, to include……..


American Food and Drug Administration (FDA): We’ve written in the past about the draft guidance for regulating apps that the FDA has published (you can read more here), and this has been a target of much press and debate lately so I won’t revisit this in detail.  While these authors agree that some regulation is important, they reiterate what I think many of us feel as well, that “government health authorities should not over-regulate medical apps so as to retain their open nature.”  And, what we believe to some extent as well, is that “the regulation process should be managed primarily by the healthcare community itself.”


Where I think the authors over-simplify the problem is in this:

“However it would be beneficial for government health authorities to provide official certification marks guaranteeing the quality of apps so that physicians can make an informed choice as to whether an app has evidence-based reliability”


I believe certification will be a challenging process, as in many cases new versions of the same app come out on almost a weekly basis.  There are other issues such as determining which apps require certification and what features of the app need certification. I think those closest to making this a reality are at Happtique (a non-federal entity), who is currently rolling out such a process this fall.  It will be interesting to see how they’ve managed to progress through some of these issues, and I think we’ll be able to learn a lot from the processes they will have implemented.


However, where I think the statement from the authors a bit naïve, is in the thought that certifying apps will allow physician to make an informed choice about “whether an app has evidence-based reliability”.  We’re learning through the establishment of our medical app peer-review process that this is often challenging to implement.  How many references do you need for an anatomy app for example?  (Although, one of our reviewers who is the head anatomy instructor at the University of Colorado School of Medicine, has found a couple different variances in terminology in some of the apps he has reviewed).  What about the variety of clinical practice guidelines out there, and the fact that there is poor adherence by clinicians to many guidelines (for example in the management of low back pain where diagnostic imaging is over-utilized, narcotics are over-prescribed, and providers continue to ignore the evidence in this area)?  Guidelines may vary slightly from different countries and new evidence comes out constantly.  And maybe even most importantly, there is often disagreement within the medical community regarding what is best-evidence practice.  Ensuring that information in apps are always representative of the most current evidence, even if there were universal consensus on all recommendations, would be an enormous and daunting challenge. 


Publishers:  Another recommendation the authors of this study make is to get medical publishers involved.   However, publishing companies are often counting on the “subject-matter-expert” author they’ve signed a contract with to provide them the information to publish.  That’s historically how the relationship works, and if a given author is unable to validate the nature of their content, or the publisher in some way has to verify it’s usefulness based on their own clinical resources, you could argue that the author may not even be needed in the first place.  It is most certainly a business-relationship, just as it has always been in the traditional textbook publishing model.  Many traditional book publishers have already entered the app scene and are delivering much of their content either directly (Elsevier) or indirectly with app development partnerships (Unbound Medicine, Medhand International, Skyscape, etc).  In addition, and most important to keep in mind, is that having publishers regulate their own content would result in a significant conflict of interest.


Hospitals & Healthcare Systems:  The other solution they present is to have hospitals and healthcare systems involved in the curation process, ultimately providing their own “app stores” where clinicians in their network would in theory only be able to access apps that have been approved by the hospital.  In theory I think this may sound like a valid solution, but I see several loopholes in this proposed solution from a pragmatic perspective.  First of all, each hospital will still have the need for a curation process that will need to address all of the issues that have been discussed so far.  The cost as related to time, personnel, and resources required to maintain an established and ongoing approval system will not be trivial.  But, the other challenge will be that of the individual provider.  A clinician is taught to be an independent thinker and problem-solver.  The skill of appraising the literature and being open to new developments in medicine is a welcome one.  Whereas it is nice to have resources to help with this process, I’m not sure that every clinician wants complete autonomy taken away from them.  I would anticipate that if a hospital does not do a good job of curating the apps in their store, then they will begin to lose credibility with clinicians.  For example, if I hear about a new app that sounds interesting to me, but then I don’t  see in my hospital’s approved app store, probably out of curiosity I would be inclined to go and find it elsewhere.  I think I can trust my own clinical intuition and appraisal skills to determine if there is some merit or potential use for the app.  Will there be a “grass is greener on the other side” sentiment about this if I’m tied to only using apps within my app store?  Will clinicians wonder what “other apps” are in the regular app store that are not found in their hospital’s approved app store?  While I’m not saying there is no value with this solution, I do think that the implementation process will have to be extremely sound for there to be success with this model.


Finally the authors in this paper show a figure representing the various stakeholders that should be involved in the regulation process of medical apps:

-       Medical Device Companies

-       Hospitals and Healthcare Institutions

-       Government (Policymakers and Regulators)

-       Physician accrediting and Organizing Bodies

-       Medical Publishers

-       Healthcare Professionals and Executives

-       Pharmaceutical Industry


Whatever the best solution is, it will likely require input from various stakeholders


In my opinion, one crucial component is missing from their list: the consumer (in our case the clinician, but also the patient when health apps are concerned) – should they not have a voice in this as well?


Just like many other things, a balance will be important.  Too much power in the hands of one entity can cause problems such as stifling innovation (FDA) or introduce bias and/or conflicts of interest (Pharmaceutical and Medical Publishers).


Ultimately, while it will be nice to have some sort of filter and some guidance for making decisions, what it comes down to is that clinicians need to continue to learn and master the art of being wise consumers of medical literature and information.  There will be no valid answer for this problem that completely removes thought, clinical reasoning, and experience from the equation (and I don’t think there ever should be). 



This entry was posted in Editorial. Bookmark the permalink.
  • Dlschermd

    Happtique has incorporated all of these stakeholders in the development of its App Certification program.

  • Pingback: Mobile Medical Apps - Where is the Evidence?

  • Jeremy Chatfield

    I think there’s another group that isn’t represented, but it critical for both consumer facing apps and for healthcare professionals’ apps. And that’s the usability/user experience designer. This is is still an emerging role within IT, but we do know that some user interface designs are less likely to be easily understood and less likely to elicit useful responses.

    It’s not sufficient to present the right information. It has to be solicited, analysed and expressed in ways that the user finds most helpful. Examples? I’ve used a Diabetes monitoring app that silently discards all data over 30 days old – how do you monitor exercise, weight control, hypertension, blood sugar levels, etc over reasonable periods, when apps are designed to discard data a month or more old? And the user interface is designed to collect one blood pressure reading, incorrectly visualises the results of multiple BP readings, and a single digit change is sufficient to trigger an alert that BP is too high (at 130/80 you get a warning; at 129/79 you don’t) – just not useful for the user, even if the right data is being collected.

    I have other examples of poor design in user interfaces for health apps. :)

  • Marc-Emile Plourde

    Good post Dan. I’m a little late to the party, but here we go.

    Why is it that everyone keeps saying that we need medical apps to be regulated, but no one ever defines what exactly is a medical app?

    I worked on a definition for medical app, taking different ideas into consideration

    I really don’t see why we need evidence to support the use of an app such as 5minute clinical consults (for example). One group has access to the app and not the other? Is that ethical? I don’t think there’s equipoise there.

    Also, I noticed the interesting fact that people tend to believe what is published on paper more than on an electronic format. I agree that it’s easier to publish electronically so there’s more cr@$*&, but one must remember that publishers are not into the business of ensuring the information contained in their book is 100% correct; that’s the author’s job. Even then, medicine is such that it is practiced differently everywhere despite some available guidelines (look at PSA screening for example), so no one will ever have THE truth.

    On the other hand, I can see why there should be regulation of apps such as EMR systems, or remote patient monitoring apps, to name a few.

    As for Happtique, I’m also following what they’re doing. I just went through their draft specs for the certification program, and I doubt any current app meet all of their criteria. Interestingly, I was also not able to find a definition of a “medical app” on their website. How can you try to certify something without clearly defining it? Maybe I just missed it… At least someone is looking into this difficult problem.

  • Ben

    You might find the NHS Apps Library of interest
    Dentist /App Developer

The terms “bugs” and “drugs” certainly aren’t anything new, even used together as a common moniker in medical speak for prescription medications to combat infectious diseases. With hundreds of pathogens and equal numbers of anti-infective agents, a handy reference that can provide this type of up-to-date information to clinicians at the point of care would certainly be welcome.  And, it’s to that end that several developers have created such a...
We performed a systematic review of goniometer apps – presented below.  But first, a real quick recap on the value of goniometers in clinical practice. Goniometry is a method of measuring joint range of motion used by clinicians such as physical therapists, orthopaedists, physical medicine clinicians and others.   Goniometric measurements can be used as outcome measures (after a course of treatment), as an exam finding to aid in the diagnosis...
It’s true that Apple, Inc has grown from somewhat menial to Titan over the last decade.  We discussed our thoughts on the battle between Apple and Google specifically in the healthcare sector previously.  It generated some good debate in the LinkedIn Digital Health Group, and interesting to see how emotional folks can get over this topic.  There is definitely a range from hard core fans on both sides of the...
This article was originally published as invited commentary in the 12th Edition of iBusiness Magazine, focused on “Business Apps & Healthcare Solutions”.  Copies of this edition are available HERE. AUTHORS: Dan Rhon Ben Hando Robert Spence   WHY THERE IS SUCH A FRENZY ABOUT MEDICAL APPS Roll back the calendar just six short years and you encounter a pivotal period in world history, the “pre-iPhone era”.  Perhaps statements such as...