Do the Doctors Manipulate Numbers?

Definition of failure: There are many definitions of failure used by different clinics and different definitions on each of their studies. It appears that with computers they can run their numbers with one definition and if the results do not make them happy - try another definition. This goes on until they find something they like and makes their studies look better than what they really are. (Back to our golf analogy: This is like saying that any ball within six inches, or a foot, or three feet, of the cup is in the hole! Most of us could improve our golf scores by changing the definition of "in the cup.")

Simply put, less stringent definitions of failure tend to result in a higher reported "cure" rate. As you read journal articles be aware of what definition of failure is being used, and how that is likely to affect the reported results.

During the past 10 years as I have read hundreds of studies. I decided that there had to be a way that we could find a common denominator in which we could weigh the various treatments and doctors. I researched every paper I could find that had anything to do with definition of failure for prostate cancer. In doing so I found the necessary data for me to come to some logical conclusions. Now I can look at any study and reduce it down, based on a PSA nadir of 0.2 at 10 years, and compare each of them side by side.

One of the more interesting developments is the use of the ASTRO definition (3). This definition was developed by the Radiation Oncology Society from data of patients who had prostate cancer and who were treated with radiation. However now other modalities of treatments outside of radiation, having realized the great benefit it makes to their figures, have started using the ASTRO definition. Using it in places where it was never designed for and has never been designed for or even tested for. It is in the process of changing yet today as I write this. I would guess they are looking for a better way yet to give them high scores.

Now we are finding that some studies are coming out with a bastardized manipulation of the ASTRO definition that adds things that were never there - and they still refer to it as the ASTRO definition. Undoubtedly this is being done to make results look better.

Who is included and who is not included in a study: Another way to "manipulate" the results of a study is through the definition of which patients are included in a study. What is an average golf score? If we include only touring professional golfers, we will come up with one answer. If we include All golfers who have ever played a round of golf, we will see quite a different answer. Which is the truth? They are both the truth, but based on a very different group of people. If we publish a prostate cancer paper that is based on men with stage T1c disease and Gleason scores of 5 or below, we will get a very different result than if we include all men that present with prostate cancer. Which is the truth? Which group do you fit into?

Look carefully at who is included in the study. Common things to look for: Including men (or at least too many men) in the study who have not been treated long enough ago to experience failure! Including men that can’t yet have failed is bound to increase the "cure" rate. (This is the main reason that studies of less than five years duration are not very useful, and may be just plain misleading.) Another is including only men who have the highest probability of success: Men with low Gleason scores for instance. That, too, will increase the "cure" rate reported.

There are many games to play, just be aware of who is included, and think critically about how this might affect the reported results. The ideal is a study that includes all comers: That is, all men treated at a facility irrespective of stage, Gleason score, etc. (No study is likely to have all comer’s because some refuse to participate by providing on-going information. Yet, a high percentage of those treated should be included.) Yet, there is even a caution here: Some clinics weed out men who are not likely to benefit from their treatment. That, too, will increase the reported "cure" rate.