Who should be the judge?

I have, more than once, faced criticism from peers in Computer Science that my research might have a better home… well, elsewhere. To be fair, the sciences have long wrestled with the question of whether the study of pedagogical issues within the domain is a valid avenue of research and reflection, or whether this kind of inquiry belongs “outside” the domain.

Bouncing through a few weblogs after finishing a movie, I found a brilliant bit of navel-gazing-avec-criticism in the writings of Glenn Fleishman. I want to take a moment here to address this, simply because it was an excellent example of how easy it is for people to misunderstand what it means to do research in a humanistic paradigm. Much like some of my close-minded colleagues in Computer Science.

The navel gazing:

First, a high Google rank does not make one right. And secondly, Glenn could not have carried out an analysis unless he

1. understood the research question being asked,
2. had access to the original research data, and
3. had human subjects approval to view that data.

Since none of these things are the case, Glenn has actually gone about asking a completely different set of questions than the researchers were attempting to address. From these questions (which he does not actually explicate), he proceeds to proclaim that the research was inappropriate and inadequate (to answer his questions). This is an easy mistake to make when one is not accustomed to actually dealing with real research data and reports on it.

Glenn’s analysis is less than brilliant. I’ve provided some of his bullet points to save you having to open another browser window:

Point by point, Glenn stumbles into classic misunderstandings of qualitative and quantitative research, as well as a general misunderstanding of the role of statistics in evaluating real-world data:

1. Glenn wants the results of this work to be
   1. significant,
   2. objective, and
   3. quantitative.

   First, it is nearly impossible to obtain objective data in any humanistic regime; clearly, Glenn has a notion that “good” research involves putting people in the equivalent of a Skinner box (rats in cages), where we can control the entire environment around them. In the real world, with real people, we cannot provide objective; instead, research in the social sciences attempts to triangulate one or more qualitative observations to paint a full and robust picture of what was going on.

2. Glenn’s (desired) hard, numerical data should not to be confused with “subjective surveys … useful as a touchpoint”, which only address “subjective experiences”, which “must be correlated.”

   Glenn seems to feel we are dealing with a difference between quantitative data and subjective data; most researchers I know would call this qualitative data, but I have encountered many people who feel the way Glenn does about research that does not involve numbers. Now, while I’m sure 3 out of 4 doctors would agree with Glenn, the truth is that we use different research methods to address different questions. Inappropriately used, numbers are no better than words in attempting to answer a research question.

   The word must is also inappropriate; it is not true that we must correlate surveys and interviews with grades (or any other metric) to “prove” or “demonstrate” success. This is simply the easiest way for someone who has been fed bar graphs and pie charts their entire life by FOX News to understand things; it does not make it the right way to analyze the research, no matter what Glenn might wish was the case.

3. “Obvious correlation apepars to be missing.”

   First, there might not be any “obvious correlation.” I’ve spent months digging through my research data; an “obvious correlation” would be like pennies from heaven; I believe there is no such thing. Real data, in the real world, is real messy. Nothing in Glenn’s background indicates he has ever done any (academic) research of any sort, from his undergraduate degree in graphic design on through his career writing columns and books for end-users about how to use drawing packages and desktop publishing software. Real research is not a matter of taking a few numbers and tossing them into Excel; properly done, studying education is a time-consuming, expensive, and tiring process that typically requires hundreds of person-hours to be done well.

I don’t know what this man wants; below is a figure summarizing one response on a survey of middle-school special-ed teachers. It accounts for responses from over two-hundred and ninety-three (293) teachers.

There is no lie in this report; it is a report on the perceptions of the teachers. In this report, 269 (of 293) teachers reported the use of the laptops improved their students engagement and interest in the material and instruction they were receiving. Does that necessarily turn up in a student’s final grade? No. We should not expect a student’s grade to necessarily indicate how engaged or interested they were. Furthermore, are we to distrust the judgment of all 269 of these teachers? I’m sorry—they’re all professionals, probably having spent many, many more years being trained to teach and teaching students with special needs than Mr. Fleishman has ever spent in a classroom or doing research of any kind.

Real research is real hard. And it is a sad world where people think that numbers tell the story better than the story itself. Given the scope of the work, any numerical results would be difficult (at best) to trust, given the huge number of schools and diversity of environments involved.

Honestly. I think I’m growing tired of this qualitative vs. quantitative crap.