BRAIN'S IDEA

Ideas for brainy people by someone who minds.

canine confirmation confound – lessons from poorly performing drug detection dogs September 27, 2012

Intuitively, the use of police dogs as drug detectors makes sense. Dogs are known to have a better sense of smell than their human handlers. Furthermore, they cooperate easily. Still, compared to the generally good picture sniffer dogs have in the public eye, their performance as drug detectors in real life is terrible. The reason why scent dogs get used anyway holds important lessons for behavioural researchers working with animals or humans.

Survey data coming out of Australia paints an appalling picture of sniffer dog abilities. Their noses hardly ever detect drugs that they are trained on. For example, only about 6% of regular ecstasy users in possession of drugs reported that they were found out by a sniffer dog they saw (Hickey et al., 2012). But once they bark, you can be pretty sure that a drug was found, right? No, you can’t be sure at all. An Australian review by the ombudsman for New South Wales found that nearly three quarters of dog alerts did not result in any drugs being found. It’s clear: using sniffer dogs to detect drugs just does not work very well.
drug detection, military, dog

Both looking in the same direction. Who is following whom?

This raises the question why scent dogs are actually used at all. My guess is that they perform a lot better in ability demonstrations compared to real life. This is because in demonstration scenarios their handlers know the right answer. This answer can then be read off unconscious behavioural cues and thus guide the dog. This is exactly what a Californian research team led by Lit et al. (2011) found. When an area was marked so as to make the handler believe that it was containing an illicit substance, more than 80% of the time the handler reported that his/her dog had found the substance. However, the researchers in this study misled the dog handlers and in fact never hid any illicit substances, i.e. every alarm was a false alarm. Interestingly, when an area was not marked, significantly fewer dog alerts were reported. This suggests that the dog owners control to a large extent when their own dog responds. Apparently, sniffer dogs game the system by trusting not just their nose but also their handler when it comes to looking for drugs. This trick won’t work, though, if the handler himself doesn’t have a clue either, as in real life scenarios.
The deeper issue is that good test design has to exclude the possibility that the participant can game it. The most famous case where this went wrong was a horse called Clever Hans. Early last century this horse made waves because it could allegedly count and do all sorts of computations. Hans, however, was clever in a different way than people realised. He only knew the answer if the person asking the question and recording the response also knew the answer. Clearly, Hans gamed the system by reading off the right answers from behavioural cues sent out by the experimenter.
Whether reading research papers or designing studies, remember Hans! Remember that the person handling the participant during a test should never know the right answer. If s/he does, the research is more likely to produce the intended result for unintended reasons. This can happen with scent dogs (Lit et al., 2011), with horses but also with adult humans (see the Bargh controversy elicited by Doyen et al., 2012). Unfortunately, after 100 years of living with this knowledge, reviewers start noticing that the lesson has been forgotten (see Beran, 2012). Drug detection dogs show where this loss leads us.

——————————————————————————————————–

Beran, M.J. (2012). Did you ever hear the one about the horse that could count? Front. Psychology, 3 DOI: 10.3389/fpsyg.2012.00357

Doyen S, Klein O, Pichon CL, & Cleeremans A (2012). Behavioral priming: it’s all in the mind, but whose mind? PloS one, 7 (1) PMID: 22279526

Hickey S, McIlwraith F, Bruno R, Matthews A, & Alati R (2012). Drug detection dogs in Australia: More bark than bite? Drug and alcohol review, 31 (6), 778-83 PMID: 22404555

Lit L, Schweitzer JB, & Oberbauer AM (2011). Handler beliefs affect scent detection dog outcomes. Animal cognition, 14 (3), 387-94 PMID: 21225441

NSW Ombudsman (2006). Review of the Police Powers (Drug Detection Dogs) Act 2001 Sydney: Office of the New SouthWales Ombudsman
—————————————————————————————————————————-

ResearchBlogging.orgIf you liked this post you may also like:
Correcting for Human Researchers – the Rediscovery of Replication

images:

1) By U.S. Navy photo by Photographer’s Mate 3rd Class Douglas G. Morrison [Public domain], via Wikimedia Commons

—————————————————————————————————————————-

If you were not entirely indifferent to this post, please leave a comment.

 

4 Responses to “canine confirmation confound – lessons from poorly performing drug detection dogs”

  1. Jeff Emtman Says:

    Great article! Experimenter effect is one of those really tricky realms where good intentions come ’round to bite us in the rear. Essentially, Hans and Bargh are the reasons for double-blind studies, right?

    I recently became enamored by Clever Hans and his owner, Willhelm Von Osten. If you’re also interested, you might enjoy this: http://is.gd/HBM006/

    You should consider writing an article on Facilitated Communication. FC is, in many ways, analogous to what Hans did, except with the more heavy handed experimenters (quite literally)

    Keep up the good work!

    • Thanks a lot Jeff.

      Not sure whether I would call Hans the reason for double-blind studies. In any case the episode is a good anecdote to keep in mind. Bargh on the other hand, is a controversial case. The jury is out whether his experiments really suffered from confirmation bias. He himself denies it.

      Will have a look into FC.

      • Jeff Emtman Says:

        Maybe less of a reason for double-blinds and more of an early case study. Thanks for listening to the show.

        Bargh is an interesting case, to be certain.

        Another related psychological rabbit-hole is the ideomotor effect, which ties into all manner of strange phenomenon.

        I’m thinking about doing a show about FC–fascinating stuff. I’d really love to talk to a therapist who practiced it before it fell out of vogue.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

 
Follow

Get every new post delivered to your Inbox.

Join 1,240 other followers