11:37 AM, HomeHome
This post represents the first of an experiment. While travelling in the US, I’m working at getting through a number of papers from the Empirical Studies of Programmers conferences. As this weblog partially represents a continuing experiment in the use of a weblog as part of my Ph.D. work, I’m going to post comments and notes regarding these papers here as I work through them. Caveat researchor applies to all of these posts, as they represent my working notes as I develop a proper review of the literature.
Spohrer, J., and Soloway, E. Analyzing the high-frequency bugs in novice programs . In Empirical Studies of Programmers, E. Soloway and S. Iyengar, Eds. Ablex, New York, 1986.
Overview
The researchers classified “bugs” in student programs (semantic, not syntactic errors) by examining the first syntactically correct version of programs generated for a sequence of homework assignments in a (typical?) CS1 course at Yale.
Discussion
The role of folklore in CS education and the role of context in CS-ED research both struck me as important as I read this paper. What follows then are some additional thoughts and comments.
Folklore
The authors were looking to probe two pieces of “folk wisdom” in programming education at the time:
- Just a few types bugs are made by many students, and
- Most bugs come from misunderstandings of language constructs.
The second piece of folklore comes from the linguistic debates of the time (“LISP vs.” of one form or another), most of which were syntactic at some level, and semantic on others. Today, the debates are cast in new language (we speak of language paradigms instead of debating syntactic issues, but let’s be honest… it’s the same thing.)
We are religious about our languages. Period. Rarely are people truly open-minded about the programming languages they use and teach. Our pedagogy is infused with folklore and religion that stems from linguistic differences.
Cast in that light, my thesis challenges a popular piece of folklore:
| STUDENTS WHO COMPILE EARLY AND OFTEN ARE NOT THINKING ABOUT THE PROBLEM. |
Test-driven methodologies (as endorsed/implied/imposed by eXtreme Programming, for example) challenge this notion. However, my suspicion is that top-down thinking and design is an idea that runs swift and deep in the mind of many computer science faculty, whether they know it or not.
Context in CS-ED research
The students were programming in Pascal on a VAX 750. This implies (I believe) a text terminal in a controlled environment. Students today can program anywhere on their laptops, in their dorm room on a machine often more powerful than anything the school can offer (with music, TV, hallmates interrupting), or in public computing labs with anywhere from 5-50 machines stuffed in a room.
The text editor used (‘ED’, I would guess) would need to be exited to compile and run the program. I think. Really, I don’t know. I don’t know which Pascal compiler was used. I don’t know how the compiler would be invoked, I don’t know how long the programs took to compile, and I don’t know how much time students spent on their programs (workgin on paper, I would assume) before sitting down in front of the terminal. In short, there is a huge amount of context regarding student programmers at Yale in the mid-1980′s that I simply don’t have access to from this paper.
Before any (fair/educated/informed) comparison or evaluation of the research presented can be made, a more thorough investication of the paper’s context is necessary.
Additional comments
Limitations of target population
The study examines the first syntactically correct program generated by students for a sequence of assignments over the course of a semester. The authors argue that these programs contain the most semantic (logical) errors (“bugs”), and are therefore the most interesting.
As a result of this premise, 1/8th of all programs were thrown out of the study “because the students had not tried to solve enough of the problem in their first attempt”. Therefore, this analysis seems to exclude the kind of programmer I’m (implicitly) most interested in: the programmer who compiles early and often. However, considering my lack of understanding regarding the context these students were working in, it may be that compile early/often programming behavior was “incorrect” (that is, there was neither the time nor resources available for students to program in an exploratory manner). If it was possible to be an eXtreme Programmer in Pascal on the VAX 750, then the Spohrer/Soloway study ignores one of my (potentially) most interesting populations entirely based on the assumption that early and incomplete compiles are incorrect and/or uninteresting.
Theory and bias, food for thought
Their analysis is not atheoretic; this is not a problem as such, but is a clear source of potential bias in the research. Soloway et. al. (large number of self-citations in the paper) were devloping a “theory of the knowledge that programmers use in generating and interpreting programs.” These involved programming goals and programming plans; from the paper, “goals are what must be accomplished to solve a problem, and plans are how the goals can be achieved.” This, again, implies a top-down programming methodology; I might argue that if you go looking for one, you’ll find one. Their analysis (roughly) involved doing a decomposition (based on their theory of goals and plans in programming) of the instructor’s solution, and then comparing that to the student solutions. While the authors acknowledge there are many solutions to any one problem, the analysis implies a fundamentally top-down approach to programming. This, again, may be contextual.
The implication Spohrer/Soloway claim their research supports:
“Educators can most effectively improve their students’ performance by changing instruction to address and to eliminate the high-frequency bugs.”
does not address a very wide variety of issues regarding programming instruction. This thesis implies a focus on “bugs” like “division by zero,” and bad boundary conditions. Does the data support the hypothesis? I would even begin by asking what Spohrer and Soloway mean by “performance” in this context. Giving them the benefit of the doubt on either of these questions, and acknowledging that multi-million dollar mistakes have been made (in recent memory) on these bugs, will focusing on them improve students as programmers?
Put simply, (as CS educators) are we trying to improve the programming process our students engage in, or the end product they produce? Spohrer and Soloway examined the end product, and claim that we can improve the end product by focusing on where the most mistakes were found in that product. My thesis involves examining the process, and ideally being able to support, inform, and guide students engaged in the process of programming.
