Friday 12 April 2013

Autism and Evidence 6. What is wrong with the draft NICE guidance on “management” for children with autism?

What is wrong with the draft NICE guidance on “management” for children with autism?

I promised to post something on this blog about my thoughts on the NICE clinical guideline for autism in children currently out for consultation (http://www.nice.org.uk/guidance/index.jsp?action=folder&o=63399). This blog isn’t in the form of the response that I might submit to NICE partly because they ask for a response using a standard form rather than a text story. I’m also not going to write about details here – more ideas and broader issues.

As a starting point, I’d like to acknowledge how tough it must have been for the experts and other members of the committee who worked on this NICE guideline. It is pretty clear in the document that NICE procedures were rigorously (and rigidly) applied. This includes the use of the GRADE system that surely was not designed for the purpose of understanding the evidence for complex interventions in contexts that will not be the same as short primary or secondary healthcare consultation. For example, the rating of quality of evidence from any intervention included in this NICE autism review has been downgraded if the meta-analysis sample is under 400 children (because any smaller sample may not provide a good estimate of the size of effect of an intervention). This is despite some well-designed individual trials that passed strict peer review for competitive funding and were published in some of the highest impact scientific journals. Such studies will have included a sample size calculation by a suitably expert statistician to decide the number of children needed for the study. This is important since there is little point recruiting more children than needed as it would waste money and (if the intervention has no positive effect) could subject more children than necessary to a poor intervention. It may be rather harsh in areas where we are dealing with low incidence problems such as autism to downgrade the quality of evidence for an intervention using this (almost) arbitrary 400 children requirement.

It is also quite possible that many of the committee members (and I’m sure they’re not allowed to say!) may agree with some or all of the points that I make below. It is hard to understand though what latitude the guidelines development group has in making recommendations. Much of what they may have wanted to drawn upon isn’t available to be used as the basis for recommendations – because of the NICE review process. So, it will be interesting to see what happens with the final version of the autism NICE guidance.

So, to be clear – pretty much all of what I have to say below is not directed to the people who gave their time and expertise (including the lay members) to the committee. It is more about the overall process.

Autism as a medical disorder

One thing that stands out from the NICE document is that the report comes primarily from a perspective of autism as a medical disorder. I’ve written about this in a blog before (see http://profhastings.blogspot.co.uk/2013/03/recovery-from-autism-false-hope-or.html). There is a long and detailed chapter on interventions that aim to deal with the core symptoms of autism. Interesting in two respects. The first is the “aim” phrasing. Because the theoretical sensibleness of reviewed interventions is not included in the NICE format for reviewing (i.e., whether the intervention makes sense is not an inclusion criterion), then the use of “aim” is wrong. In practice, it instead means studies that happen to have included a measure of autism symptoms as an outcome measure. Forgive the example, but this means that a horseriding RCT was seriously evaluated despite no discussion of whether this made sense. Basically, the “how” question is not considered – how exactly is horseriding for 12 weeks once a week meant to be an intervention for the core symptoms of autism?

The second thing about interventions aimed at the core symptoms of autism is that this is rarely the point of many autism interventions. Especially those interventions used in educational contexts often focus on making a difference to children with autism in key domains that have an impact on the quality of life of them and their families. To be fair, one of the later chapters of the draft guidance includes interventions that “aim” to address associated features (including adaptive behaviour, IQ and the like) – or more properly, interventions that include these variables as outcome measures. However, it still makes me uncomfortable that curing symptoms of autism (although “cure” is of course not used as a term) gets such prominence.

Missing evidence

The data included to inform the guideline are based on outcomes of RCTs, but also qualitative research studies (a huge number of them in this draft guideline) and economic evidence. Although it is clear at several points that systematic reviews and meta-analyses would be within scope, I assume that this means only such analyses that are focused on RCTs alone. Otherwise, it is puzzling why a number of recent well-cited reviews are not discussed.

Economic evidence seems to be discussed with much less of an interest initially in its quality (although clearly quality of evidence weighs heavily in the conclusions). This does lead to the perverse situation on occasion of the discussion of economic analysis evidence for interventions that are not actually included in the outcome evidence review.

Qualitative research studies do not seem to have been identified for their quality, but for their relevance to review questions. There is also no requirement for qualitative studies to include a minimum number of participants. Despite this, the qualitative research evidence is used as the basis for a series of recommendations.

I am not necessarily criticising the inclusion of lower quality evidence or evidence where quality is not assessed in detail (although this does seem a bit weird!), but I do want to point out the perversity of this when a whole domain of evidence relating to autism is missing from the reviews informing the draft guidance. Specifically, Single Case Experimental Design (SCED, sometimes called Single Subject Experimental Design) studies have not been reviewed. These are not descriptions of single clinical cases, nor are they case series descriptions. Rather, they are a method used to bring a controlled analysis to studies involving a small number of participants. Strategies include multiple baselines (introducing an intervention to one child at a time, and showing that behaviour changes only when the intervention is introduced each time) and reversal designs (apply the intervention, remove it, and then reinstate to show the behaviour change is related to the intervention being in place).

Several other evidence review bodies throughout the world have recognised that SCED evidence is of value and needs to be reviewed when the effectiveness of an intervention is being addressed. Several bodies (including the What Works Clearing House) have established methods and criteria for reviewing SCED evidence and the level of evidence and replication needed to recommend an intervention as evidence-based.

Why does this matter? Well, a huge amount of the intervention research in the field of autism uses behavioural methods to teach skills or to reduce problematic behaviours. Single and focused behavioural interventions are often evaluated using SCED approaches. Thus, a whole bunch of relevant evidence is being excluded. Unless this issue is addressed by NICE it will have a serious impact on other upcoming reviews. Specifically, challenging behaviour in adults with learning disability is a topic in the early stages of the guideline development process. Again, many of the interventions use SCED evaluation methods. With the expansion of NICE to social care, there are also likely to be other reviews affected in future (for example, the same problem would occur if NICE reviewed supported employment interventions for adults with learning disability).

The current exclusion of SCED evidence is likely to be detrimental to people with disabilities. One might even argue that it is potentially discriminatory - since SCED evaluation methods have been primarily (though not exclusively) applied to interventions used with people with disabilities and not to (or less so with) other groups.

However, my main point for now is that given qualitative research evidence is accepted as a part of NICE evidence reviews then SCED evidence really ought to be too.

The development of evidence for complex interventions

The NICE draft autism guideline clearly states that interventions for autism are complex in nature. It is then surprising that guidelines about the development of evidence for complex interventions based on Medical Research Council and other frameworks (see my blog http://profhastings.blogspot.co.uk/2012/12/who-do-you-believe-model-for.html) do not seem to feature in the NICE review process. It is clear that various stages are needed to build evidence for complex interventions. At the very start, theoretical development is important (see earlier comment about horses!) – interventions have to make sense theoretically speaking rather than just being tried because they seem like a nice idea. This is especially a problem in autism it seems, and it is certainly good to have (in the draft NICE guidance) some “do not use” recommendations for interventions in autism that few scientists ever thought made proper sense.

Other stages of the process make it clear that different questions are asked at different stages. So, studies designed to test efficacy will look very different to those designed to test effectiveness. Similarly, studies designed to test large scale implementation may look different. RCT methods can be used at all of these stages but may not be, and it is probably reasonable to rely less on RCT designs as one gets towards the latter stages of building evidence for a complex intervention. There are several large scale implementation studies using controlled (but not RCT designs) that will have been excluded from the NICE review just because they were not RCTs.

The problem is that the inclusion criteria for evidence about intervention for autism is focused specifically on the use of RCT designs. No evaluation has been made about whether the study using a RCT design was actually designed to answer the question about an intervention’s effectiveness. I assume that this is a general issue for NICE reviews – perhaps the methodology has not yet caught up with the fact that NICE review a range of complex interventions and now also in a variety of settings including social care.

The problem of behavioural interventions

Many people are going to be disappointed or angry that the evidence for behavioural interventions for children with autism does not appear to have been reviewed as a part of the NICE process. I am going to offer a few thoughts on this since I know that: (a) some people might be interested, and (b) I can imagine the debate may get quite unpleasant.

First point comes back to the SCED evidence for autism interventions (see above). At present, this is clearly not evidence that can be included in the NICE process. It should be. However, it is difficult at this point to complain that it isn’t there. An implication for the NICE guideline though is that some statements will need to be re-worded. For example, it is unlikely to be true that there are no interventions specifically for addressing repetitive behaviours or rituals in children with autism. I suspect that there will be at least some SCED studies. Thus, it needs to be clearer that there are no studies meeting the restrictive NICE inclusion criteria. Also, when there is apparently no other evidence, you would think that taking a look at SCED evaluations would actually be rather useful as a way of developing recommendations.

A second point is that there could be more consistency in how behavioural approaches are described throughout the text of the guideline. The committee tentatively recommend the LEAP intervention model, for example. They are also clear that many interventions (including the Early Start Denver Model) use ABA methods. PECS is also discussed positively, and PECS is an ABA intervention of course. In fact, my reading of the LEAP evaluation paper that the committee liked so much is that it is also a behavioural intervention. Granted, discrete trial teaching methods are not the core of the approach, but all of the core elements of the intervention described by the study authors are ABA teaching technologies. There is also evaluation research from the UK where a variety of behavioural methods were used together as an educational approach in the early years and in a mainstream setting (like the LEAP model) [see Grindle, C., Hastings, R. P., Saville, M., Hughes, J. C., Kovshoff, H., & Huxley, K. (2009). Integrating evidence-based behavioural teaching methods into education for children with autism. Educational and Child Psychology, 26(4), 65-81. And Grindle, C. F., Hastings, R. P., Saville, M., Hughes, J. C., Huxley, K., Kovshoff, H., Griffith, G. M., Walker-Jones, E., Devonshire, K., & Remington, B. (2012). Outcomes of a behavioral education model for children with autism in a mainstream school setting. Behavior Modification, 36, 298-319.]

It would be good to see the guideline include a specific section that draws together the guideline development group’s perspective on behavioural methods. It is important to point out how many interventions that don’t even call themselves “ABA” or behavioural actually rely almost exclusively or in large part on ABA teaching technologies. The guideline group could help a rapprochement and future collaboration between autism professionals/experts to the benefit of children with autism and their families. I almost expected some discussion of this sort near the beginning of the draft guideline document where intervention “conceptual frameworks” are mentioned. However, this was instead a rather disappointing section.

So, this is a plea directly to the members of the guideline group – please, please, do something positive about behavioural interventions/methods along the lines I suggest. Otherwise the guidance is in serious danger of just reinforcing the fault lines that are already doing our field (and children with autism and their families) no good.

A final point on the behavioural stuff is that it is encouraging to see recommendations about the use of functional analysis and behavioural interventions based on functional analysis when dealing with challenging behaviours. The language used could do with a thorough edit though. For example, experts in behaviour analysis might reserve the term “functional analysis” for experimental functional analysis, which is a technique requiring considerable expertise and also ethical review. The term “functional assessment” may be what is meant. A small group of experts in ABA and Positive Behaviour Support would be able to take on an editing task. In addition, the guideline probably should clarify the competencies that may be needed to carry out some interventions (the example of experimental functional analysis is an example). The guideline could helpfully recommend referral to expert challenging behaviour teams or to a clearly (behaviourally) trained challenging behaviour expert within a multi-disciplinary team.

7 comments:

  1. Congratulations Professor Hastings in eloquently putting what so many parents feel.

    It is unfair for NICE to require 400 subjects in RCT trials before taking them seriously, when no one in the DOH or NHS (or even in USA) would be willing to fund research into an autism intervention on this scale. We need to get some realism into our evidence requirements or if not, we need a significant increase in the amount of UK public money being invested in autism intervention research.

    Unreasonable requirements for evidence is not the only problem with this guideline. Experts in behavioural interventions/ABA, of which there are several in the UK, do not seem to have been invited to join the group, and therefore the wealth of "evidence" that ABA therapists and parents see day to day with a behavioural approach, was not available in the room when judgements were being made.

    Overall, the guideline will be a huge disappointment for parents, and will undermine the level of trust between families who have autistic children and the DOH/NICE/NHS

    ReplyDelete
    Replies
    1. I think several sources of evidence were not use in a way that informed the recommendations which is a real shame but does seem to be about the NICE process primarily.

      Delete
    2. God bless Dr Akhigbe for his marvelous work in my life, I was diagnosed of Herpes Virus since over a month now and I was taking my medications, I wasn't satisfied i needed to get the Herpes virus out of my system, I searched about some possible cure for Herpes i saw a comment about Dr Akhigbe, how he cured Herpes with his herbal medicine, I contacted him and he guided me. I asked for solutions, he started the remedy for my health, he sent me the medicine through UPS SPEED POST. I took the medicine as prescribed by him and 7 days later i was cured from Herpes ,thank you for anyone that may need the medication you can contact him for your urgent cure
      Drakhigbeherbalhome5@gmail.com
      Whats App +2349021374574..

      Delete
  2. SPECIAL THANKS TO DR WILLIAMS FOR THE AUTISM CURED HE
    RENDER TO MY SON.
    I'm giving a testimony about Dr. WILLIAMS for the great deed he have done for my family, he has
    the natural cure to autism, he cured my son from autism ,
    though I went through different website I saw a testimonies about this
    Dr Williams, I was like: 'Dr Williams have the autism cure why are
    people still suffering from it?' I thought of it, then I contact Dr Williams, I didn't believe him that much, I just wanted to give him a try, he replied my mail and Needed some Information about my son, then I sent them to him, he prepared the (CURE) and sent it to me through Courier Service he gave my details to the Courier Office, they told me that 1-3 days I will receive the package and my son took the medicine as prescribed by Dr Williams and i took my son for check-up 1 week after finishing the medicine, then the autism was not there anymore, if you are suffering from autism you can also contact him on his via email address: drwilliams098675@gmail.com for his help.

    ReplyDelete
  3. Dr Williams,his medicine was extremely helpful to my daughter. I have had a daughter with autism for 3 years now , and my doctor have prescribe me with different kinds of drugs without no improvement in my daughter symptoms . I am so glad I have found Dr Williams medicine (as I was surfing the net for a solution and saw a recommendation for it on a women's health forum early this year). I have felt so empowered by the knowledge you have given me and I quickly self diagnosed the individual cause of my daughter autism and armed with your clear instructions, I was finally able to beat this monster, in less than 4 weeks of using dr Williams medication on my daughter. I went for a checkup and it was clear that my daughter autism were gone,you can also email him on drwilliams098675@gmail.com for help.

    Thank you, thank you , thank you. dr williams "

    ReplyDelete
  4. Just wanted to say a huge thank you to Dr Williams for everything he have done in my life. I have been in tears for so many years now, having found that my only son is suffering from autism, lesion after some years of being under extreme stress., . I spent 24 hours freaking out and finally calmed down after reading some testimony on here about Dr Williams herbal medicine to autism. The moment came and I mumbled out to my husband about Dr Williams product (with a few tears) and he was AMAZE – . He was just worried about our son been free from this disorder that was how will order for Dr Williams product on line , after my son finish taking Dr Williams medicine for just within a period of one month. Will went to the doc together who confirmed that he was autism free.
    Thank you again for being upfront, truthful and taking this pain away from my family for more information about his product you can email him on drwilliams098675@gmail.com i hope you can also help other terrified people in a similar situation.

    ReplyDelete
  5. ALL THANKS TO DR WILLIAMS FOR THE GREAT DEED HE HAVE DONE FOR MY DAUGHTER?
    My daughter suffered from autism for more than 2 years which we started experiencing in her when she turned 1 year and 5 months we all thought it will end but got even worse as days went by. We tried all several treatments and therapy prescribed by various doctors we met but to no avail, she lost total concentration and always complain of sensitivity to sound . She usually tells me she haves poor eyes contact. This were steady disorder that disrupted her entire life, even at night she slept less because of this.It was during a casual conversation with a friend that i learned about Dr Williams herbal medicine I was able to contact him on his email address. and give him all the necessary information that he needed,few day later he sent me the herbal portion and his medicine was able to restore her back to normal and she is very okay now without any side effects whatsoever. If you have autism, do not hesitate to contact him on drwilliams098675@gmail.com for advice and for his product. I hope this also helps someone out there

    ReplyDelete