Commentary: Junk Horse Science and the New York Times

In the interest of presenting different perspectives on issues of importance to the Thoroughbred industry, the following commentary was submitted by Mel Moser, who examined the recent article in the New York Times on horse racing (“Mangled Horses, Maimed Jockeys: Death and Disarray at America's Racetracks”) and the statistics supporting the article.

 

Moser is a retired attorney who has been an avid fan and horseplayer for more than 40 years. He is president of the Horseplayers' Coalition and member of the NHC Players' Committee. – Ray Paulick

In the early summer of 1872, West Coast photographer Eadward Muybridge undertook the complex task of proving that for centuries, artists' and sculptors' depictions of horses were all wrong. Five years and $50,000 later, Muybridge had overcome technical obstacles to produce a photograph he believed proved that a galloping horse's legs are gathered together, not splayed, at the moment when it lifts all of its hooves off the ground and becomes airborne.

Members of the press were skeptical and some refused publication on the ground that Muybridge had engaged in the almost universal pre-photoshop method of manually retouching the negative to eliminate minor defects.

Undaunted, in less than a year Muybridge figured out a way to improve upon his photographic evidence by arranging a series of trip wires across a racetrack so that the racemare Sallie Gardner would trigger 24 separate cameras placed 27 inches apart as she ran by at a controlled speed of 36 miles per hour. Leaving nothing to chance, this time Muybridge invited members of the press to witness both the taking and development of the photographs, and in short order made the original negatives available for onsite inspection.

Muybridge was thus able to prove his original contention conclusively, and his fast motion photographs played a pivotal role in the eventual development of movies. And to this day, “Sallie Gardner at a Gallop” is a shorthand description of the well known phenomenon of observation bias, which includes both observing what isn't there, and observing only what one expects to observe, instead of letting data, and conclusions based on data, speak for themselves.

The science pendulum has swung so far in the opposite direction over the last 134 years that many scientists are convinced that some reporters are either ignorant of basic scientific concepts or contemptuous of science itself. Both shortcomings are on full display in the New York Times article “Mangled Horses, Maimed Jockeys.”

Turf writer Andy Beyer was the first to point out, in an article in the Washington Post, that it was fundamentally dishonest for the Times to lump together statistics from the entirely different sports of Quarter Horse and Thoroughbred racing in order to produce “a carnage-laden front page story.” Or, as reporters say, if it bleeds, it leads.

Beyer's charge must have struck a chord, as his piece elicited a published letter response from Times' Investigations Editor Matt Purdy. Purdy's letter justifies the decision on the grounds that: (1) Races for both breeds are “almost always governed by the same state racing commissions and operate under the same rules”; and (2) The Jockey Club's injury database includes Thoroughbreds and Quarter Horses.

Of course, the fact that the FDA regulates and maintains databases for bottled water and heart valves has no bearing on the question of whether it makes sense to combine statistics for such disparate items.

More to the point, the Times' authors took the exact opposite approach to fatality statistics in England where, the authors claim, “breakdown rates are half of what they are in the United States…”

Far fewer races take place in England each year, and all are conducted on turf or artificial surfaces, whereas the majority of races in the U.S. are run on dirt. In contrast to the state-by state system in the U.S., English racing is regulated by a single entity, the British Horseracing Authority, which dictates medication rules, maintains an all-inclusive database, and publishes annual reports on horse fatalities. What the BHA annual reports show is that over the last 15 years, the number of fatalities has fluctuated about 33%, from a high of more than 3 per thousand races in 1997 to a low of 2 per thousand races in 2011.

More importantly, if one compares the number of breakdowns reported in the admittedly incomplete Jockey Club Equine Injury Database to the BHA's annual reports for the 2009 to 2011 period covered in the Times' article, the fatality rate in England was actually higher than it was in the U.S.

It's not disclosed in the article, but the only possible way for the Times' reporters to have reached the conclusion that breakdown rates are lower in England was to exclude all fatality statistics for English jump races, which make up close to a quarter of the annual total and, like Quarter Horse racing in the United States, result in a much higher percentage of injuries and fatalities. And the most logical reason for excluding jump racing is that it is an entirely different sport, even though, like Quarter Horse racing in the U.S., English jump racing is subject to the same medication and other rules and regulations as flat racing, and is governed by the same regulatory authority as flat racing.

In short, the Times' article used one standard to reach its decision to include Quarter Horse statistics in the U.S., which inflated the number of fatalities and injuries, and used an entirely different and contradictory standard to exclude jump race statistics, which reduced the numbers in England, and then compared the two as if they were equivalent. Standing alone, that in and of itself is misleading enough to raise serious questions about the Times' approach and conclusions.

But it is now clear beyond argument that there's an even bigger and more fundamental problem with the database that Times' co-author Joe Drape, who has covered racing for 13 years, recently described on NPR as “the foundation” for the article. The main body of the article does not make it clear that the accuracy of the Times' analysis depends on a brand new statistical measurement that was created by the Times and making its debut in the article.

Rather, the only way for an online reader to discern the critical importance of the Times' newly created statistic was to follow a link that does not appear to have been included in most reprints of the article, and which explains that what the article refers to and measures isn't actual breakdowns, but “incidents,” which were defined to include all instances where a chart caller used the words broke down, lame, or vanned off.

As even casual handicappers know, one often sees such comments in the past performances of horses which are entered to race in the future, so it is simply factually wrong to equate them with a breakdown or injury. In fact, based on an exhaustive search for horses with such comments which returned to race, Frank Angst of the Thoroughbred Times has demonstrated that the actual incident rate is: (a) 34.8% lower for Thoroughbreds alone; and (b) 14.6% lower even if Thoroughbreds and Quarter Horses are combined.

Faced with such data, the Times is no longer even trying to defend its inflated and indefensible numbers, arguing instead that they knew the Jockey Club's numbers were incomplete so, in the words of Assistant Editor Walt Bogdanich:  “We were working in an area where there wasn't a good database and we tried to come up with one…”

An understandable, and perhaps even commendable goal, but one of the things that has always been thought to distinguish the Times is scrupulous fact-checking and editing, and a good motive is no excuse for the publication of grossly inaccurate data.

The article's use of anecdotal evidence is every bit as bad, particularly given that their basic premise is fundamentally flawed. Without citing or referencing any scientific evidence to support their claim, the authors assert that “legal therapeutic (emphasis added) drugs-pain medicine in particular-pose the greatest threat to horse and rider.” However, the only relevant statistic for the only pain medication mentioned in the article directly contradicts their assertion. If what they're claiming is true, then Virginia's fatality rate would have been expected to have gone down instead of staying the same over the more than two-year period since regulators reduced the permissible level of race day phenlybutazone, i.e., bute, from five to two grams.

There have been countless scientific and peer reviewed studies of bute which are readily available. As leading veterinarian Dr. Lawrence Soma noted in his comprehensive review of the scientific literature: (1) It is the opinion “of many veterinarians that [bute] would allow a horse to compete with mild chronic arthritic changes, but did not possess sufficient anti-inflammatory activity to allow a horse with a serious injury to compete.” and (2) “Results from performance studies suggested that [bute] had no clear effect on the performance of normal, healthy horses.”

The Times' authors also cite an unidentified California study and their own examination of an unspecified number of Pennsylvania necropsies to support the proposition that horses that broke down during racing had existing “problems.” However, as Dr. Rick Arthur explained, some percentage of unidentified problems in California was microscopic in nature. As a result, in addition to not knowing what was being studied or what the actual numbers show, the reader is left to speculate on whether and how it would even be possible to identify microscopic problems in advance and if so, how microscopic problems could have contributed, if at all, to fatal breakdowns.

A fair article would have made clear that there is no relationship between the microscopic and other problems found in the unidentified California study and what one or more of the authors learned when some of the underlying data for Pennsylvania was actually reviewed. The article does not even claim that the Pennsylvania review led to the conclusion that a majority of horses that suffered fatal injuries there had existing problems, but instead reports on a small number of anecdotal situations, which for the most part have nothing to do with horse fatalities.

There have been a number of horses with one eye, like Casseleria and Pollard's Vision, which retired to stud after successful racing careers. Nor, as many scientific studies demonstrate, is it uncommon for horses to compete for a long time and retire sound after a successful operation which includes the use of screws to fix a broken bone, such as undefeated champion Personal Ensign, who competed with five screws in her left hind pastern. And it is even harder to imagine how stomach ulcers, which are common in horses and ironically can be treated therapeutically, could possibly contribute to a fatal breakdown.

Since the Times' article, a number of prominent owners and regulatory groups have either called for a complete race day ban on all therapeutic medications, or are considering limited bans on specific medications for certain classes of horses. For the most part, their not entirely unreasonable thinking is based on the idea that scientific truth doesn't matter, because public perception is the only reality that counts and, according to a recent study by McKinsey & Co., medication issues are among the reasons the public has an unfavorable view of racing. In contrast, leading trainer Dale Romans has taken the position that in light of the attendance and handle records set at Gulfstream's recent meet, and record crowds at Keeneland, the public perception argument doesn't hold up.

To the extent that public perception is being influenced by misleading and inaccurate information, it's no doubt frustrating to the trainers, veterinarians and backstretch workers who have devoted their lives to the safety and welfare of the horses in their care. However, when all is said and done, even after considering what science is able to tell us, a fair interpretation is that many of the questions surrounding the issue of race day medication are still largely a matter of opinion.

Mine is that as a result of the many mistakes and misstatements in the article, the New York Times has forfeited any claim it might have had to be an objective and honest voice in the debate.