research » Always Where

again: qualitative research is good research

Greg Wilson recently posted about student learning, specifically with respect to studying from their own lecture notes. He received some advice from Brock MacDonald at the Writing Centre at Woodsworth College. Brock gave Greg some very bad advice about qualitative research, based on what Greg passed on in his blog (emphasis mine). Here’s some of Brock’s advice:

Empirical evidence for the value of students writing their own assignment specs is harder to come by, because it’s less amenable to direct testing (like most aspects of teaching writing)–the support is more qualitative than quantitative, hence more open to question.

…

both Elbow and Bean refer to quite a bit of supporting research, though as I said it’s mainly qualitative.

Let me set the record straight here: qualitative methods are excellent research tools. They can be applied correctly or incorrectly in any given situation, and the analysis of a qualitative study can be carried out poorly or it can be performed with skill. Qualitative methods give us access to the why of social settings things in a way that statistical methods are hard pressed to do with any rigor whatsoever. Put simply, qualitative and quantitative methods both have a place in research, and we should no more dismiss one than the other.

For Brock to say that a piece of research is “open to question” simply because of the methodology used demonstrates a lack of understanding of the methods involved. We do not evaluate research simply based on the methods used, unless the method employed is inappropriate to the question being asked. We do not consider either qualitative or quantitative research to be flawed unless the method was inappropriately applied or analyzed. And that is not what Brock said in his note to Greg. He just implied (strongly) that the research was questionable simply because it employed qualitative methods.

Peer reviewed qualitative research is not “open to question” as a general rule. (Or, it is always to be challenged… but that is ideally a challenge issued by experts through another research study.) Qualitative research has rigor, and clearly addresses the questions asked in the context they were asked. Unless you’re going to develop expertise enough to evaluate the research in question, accept it as having been executed by people with expertise, undergone peer review by experts, and see for yourself what the research says to you about your specific educational context.

programming less visually

Operation: Stick Figure Army is a project that will provide tools for converting 2D imagery (as typically found in many introductory computing texts) into 2.5D representations: physical tiles with raised and textured surfaces that represent one or more diagrams from a text. This lets us explore, as a group, image processing, 3D printing, and the process of creating usable software for a target class of user.

The project is being carried out by two students at Allegheny and one student at TCNJ. Sara and Stephanie (Allegheny) have spent this semester learning Python and basics of image manipulation, recently posted their experiences while exploring Blender’s Python API (so we can automate the process of 3D object generation), and are beginning construction of their Cupcake CNC 3D printer. Cory (TCNJ) has also begun looking at Python, and is (at the moment) providing critical reflections on what it is like to be a blind student of computing.

His most recent post on the project blog, How to cope with the design phase?, explores the notion of visual representations in computing. I found his reflections interesting for two reasons. First, it reminded me of a paper by Marian Petre and Alan Blackwell titled Mental Imagery in Program Design and Visual Programming (PDF) (International Journal of Human-Computer Studies, 1999 (51) p7-30). They explore the role of visualization and visual language in expert programmers, and while the paper is too rich to summarize concisely, I’ll pull most of a paragraph from the conclusion here:

There has been a great deal of confusion in the visual languages community about the relationship between mental representations of a design and the programming notation. As noted by Green, Petre & Bellamy (1991), the superlativist position (claiming that visual languages always improve performance in programming tasks) is not justified by experimental evidence. Blackwell (1996a) has analysed the variety of beliefs among visual language researchers about the cognitive effects of using different notations. These tend to range far beyond the practical benefits of the notation or the programming paradigm, but their statements about mental representations bear little resemblance to our findings in the present studies.

Put simply, we tend to get all hot and bothered about visual tools, but research doesn’t imply that this is always a Good Thing.
The second aspect of Cory’s post that I thought was important to reflect back was the importance of verbal communication in software design and development. Whether it is aural or written, being able to express ideas about the design and implementation of software is critical to students of computing today. In some circles, this comes out in the context effective communication and collaboration in teams (Begel/Simon, Struggles of new college graduates in their first software development job (PDF), ACM SIGCSE 2008). Regardless of whether it is a software team at Microsoft or a distributed, community-driven effort surrounding open source software, the overwhelming majority of our communication and collaboration regarding software developing is written/verbal, not visual. That is, we’re not shipping pictures back-and-forth 24/7—we’re chatting on mailing lists, IRC, and blogs to get things done.
I think these ideas say a lot about where we are failing in terms of computing education today, but that will have to be the subject of another post. For now, I found Cory’s words thought-provoking, and wanted to reflect a little bit on what he had to say.

the open group @ allegheny college

How is a research group at a small, liberal arts college born? I think you get some students working on related things, and declare them a group. That, and have them over for dinner at least once.

Background (not so important)

My current research focuses on the behavior of novice programmers and the development of tools for parallel programming on small embedded systems. Long term, I am interested in the usability of programming languages—which lies at the intersection of my current lines of research inquiry.

One reason I take a long view on my research because I am at the start of my career. Another is that I am committed to involving students at Allegheny College with diverse backgrounds in my research. For these reasons, I paint my research with a broad brush to better support more students with more interests. In doing so, I can productively move my research forward regardless of what interests my students might have in any given year. It also makes it easier to collaborate across disciplinary boundaries when you keep an open mind regarding where your work can be applied.

The Students (very important)

This semester, I am excited to be working with four students. Stephanie Cost and Sara Doan (both Allegheny ’11) have been learning the foundations of image manipulation using Python, with an eye towards image analysis and transformation. Their work is funded by the Collaborative Research Experiences for Undergraduates programme, and is titled Operation: Stick Figure Army. Their project explores the problem of transforming 2D images into 3D models suitable for printing on the Makerbot Cupcake CNC 3D printer (project overview). The goal is to produce an open source set of tools for supporting blind readers—images are the bane of a blind student’s existence, as they translate poorly using existing printing technologies. Our hope is that low-cost, personal fabrication technologies like the Cupcake CNC (along with usable, open software tools) will create new opportunities for blind readers in this space. Both Sara and Stephanie are doing excellent work, and I’m excited to have the entire year (both semesters plus the summer) to work with them on this project.

Although this is our Fall Break, I spent two hours today catching up with Radu Creanga (Allegheny ’11), who is helping our distributed group bring parallel programming to the Arduino platform. Using the occam-pi programming language and executing on the Transterpreter virtual machine, Radu has quickly learned the joys of checking out an open source project, cross-compiling it from source, and doing hardware-level additions to our library for safe, process-based parallel programming called Plumbing. You can follow his work online (project overview, more posts coming soon), and both his software and documentation will be part of our pending release at concurrency.cc.

Last, but not least, Drew Pirrone-Brusse (Allegheny ’10) is working on Verbing Stories. Drew is pursuing a self-designed major at the intersection of Communication Arts and Computer Science, and this semester we are exploring Kodu (and, time permitting, Scratch) as platforms for interactive storyboarding in the context of game design and creation. While Kodu is not an open platform, we hope that our work will provide inspiration for others interested in the use of novice programming environments for interactively exploring the role of story in game development.

In later posts, I’ll point out where some of our infrastructure (repositories, etc.) live. In the meantime, I wanted to introduce the exciting projects that these students are engaged in. If you’re so inclined, you can also follow their work at planet.baseplate.org; for lack of a better name, we’ve named ourselves the Open Group @ Allegheny College, because we’re open to working with students on a wide range of projects, and we strive to make our work open source wherever and whenever possible.

Blogs: Stephanie/Sara (Operation: Stick Figure Army), Radu (Plumbing for the Arduino), Drew (Verbing Stories)

Planet: planet.baseplate.org

documentation and its role in open projects

Mel was musing on documentation in open projects recently.

In our recent code sprint on the Transterpreter project, we realized that for years we have had far too little documentation. Specifically, we’ve been lacking documentation on not just one front, but two.

First, we’ve lacked developer documentation. That didn’t change during our sprint. Someday, but not during this sprint.

Instead, we focused on a new (user-centered) platform and end-user documentation. Given that we are working on a language and a runtime for that language, it becomes difficult to document things for people. “I downloaded it… now what do I do?!” Invariably, we say “write some programs!” That’s… less than ideal. We realized that if we want more people to explore parallel programming for embedded systems, we’re going to have to provide an easy path for engaged beginners to dive in and be successful from the start.

So, we brought our language and runtime to the Arduino. (There’s some cool videos over on the Transterpreter blog; check it out.)

The Arduino is a piece of “open hardware,” meaning the design of the device is free and open. It is powered by the Atmel Atmega328, a 16MHz processor with 32K of flash and 2K of RAM. To many people, those are small numbers, but to us, they’re enough to write interesting parallel programs for controlling embedded microcontroller projects.

We have drafted the first 50 pages of a book that supports programming these affordable ($25 or less), open devices in one of the few parallel programming languages around. Our support library is called Plumbing, and we are excited about completing the first ten chapters and sharing them with a group of “alpha testers” who can help make sure our writing is appropriate for our target audience.

Who is our target audience? We have chosen the Student Artist as our target persona for the book. This is a notional college-age student who is studying art, has no significant programming experience, but wants to add interactivity and computation to their work. They don’t want to learn to program (per se), but instead want to achieve some end: a piece of art, an installation, or similar. The electronics and programming component is fascinating to them (and perhaps a touch daunting), but they’re willing and ready to learn what is necessary to make their art as successful as possible.

This persona, we believe, overlaps a great deal with the MAKE community, and even with many students of engineering and computing. However, we have explicitly forced ourselves to consider a target audience that is not studying programming. This forces us to think about what concepts we introduce when, focus on outcomes and results (as opposed to belaboring theory up-front), and do our best to support the artist/maker in fun and thought-provoking explorations from the start.

If you are interested in being part of the group who gets a look at the book early, please either drop me an email (matt at transterpreter dot org) or leave a note with contact information in the comments (a URL to your contact info, an email address, etc.). We’ll be finishing the drafting and revising that material over the coming weeks, and hope to have an initial release of the software (a simple IDE with Arduino support built-in) soon.

hacking

For the next three weeks, I have to friends and colleagues here in town. We’re busy hacking away, laying some foundations for the “next steps” in the work that we do surrounding concurrent and parallel programming languages.

It does mean that we have a very full house, but on the flip side, we’re eating well (everyone likes to cook or bake) and getting good work done so far. That, I suspect, will continue.

Especially the eating well part.

Speaking of which, I’m no longer frying, but instead burning, the mushrooms I have in the pan…

manufacturing for the little guy

Twenty years ago, CNC machining was a black art. CAD-driven solutions were expensive, and program-at-the-machine was just becoming available. My first experiences with this technology were with my father’s 1986 Hitachi Seiki CNC lathe with Fanuc 5T control. It had a punched tape reader, and editing (once a program was loaded into memory) was on a per-line basis using a 9-key numeric keypad.

What looks like a Hitachi Seiki 3NE CNC lathe

Today is a different world. It is possible to draw, using free/open-source software, a 2D part and have it sent out to be laser cut by companies like Ponoko. You can take 3D designs and have them printed, or you can have them produced (on-demand, one-off) by CNC. You can even have them turned into molds through the same process for short-run (10-1000) injection-molding. For that matter, you can build your own 3D printer from open-source designs, or buy a kit that gets you moving in the world of personal fabrication.

I am fascinated by this space, in part, because of my experiences in my father’s shop. It is also as a computer scientist, and in particular someone who is fascinated with languages and robotics, that I see a great deal of potential. As these tools become more affordable (for play and exploration), they become not just tools of manufacture, but of art and creativity. Likewise, I can begin to think about how I would create a mid-sized, affordable robotics platform for use in and out of the classroom—without being entirely constrained by what I can purchase off-the-shelf. That is, I can actually design, and have produced, low-cost bracketing specific to my needs, if necessary.

So many toys, so little time…

invited talk

In two weeks, I’ll be giving a talk (along with Christian Jacbosen) at USENIX ’09 entitled Towards Designing Usable Languages. We’re excited about the talk, and it should be very, very exciting.

I believe the talk will be on-line when we’re done, so I’ll make sure to point to it. If you’re in the San Diego area, and want to try and get together, drop a note—the schedule will be tight, however. We’re in-and-out like a burger.

paris bound

Actually, I’m not. But a colleague is.

http://didithrodrigo.blogspot.com/2009/03/woohoo.html

This is really great. Several years ago, I had the good fortune of meeting Didith Rodrigo… the details of which are a story for another time. For the moment, it is nice to see that the work that she and her students have been doing will be shared with our colleagues in CS-ED(R). And I’m very glad that I’ve been able to play a part in this ongoing work.

I don’t think I can get over to Paris this July, but it would be nice.

For those following along at home, no… still no baby.

We built this city…

Sunday I take off for San Fran, and am pretty psyched. Christian, Jon, and I are heading to the AAAI Spring Symposium to talk robotics and concurrency with a bunch of other robotically-minded people; that fills our time between Sunday and Wednesday. Thursday, we’ll be giving a Google TechTalk, which should be very fun—at least, we’re planning on enjoying the talk immensely. Then, we’re thinking (but aren’t committed) to doing a little bit of light camping, as this time of year can be soooo nice in the Bay area.

We don’t have housing nailed down for Palo Alto Sunday/Monday/Tuesday, but I think we might just grab a hotel near Stanford. It will be a bit pricey, but otherwise, we’re dealing with cars, and parking, and commuting, and… that’s not so much fun. We’ll see.

I’ve touched base with a few friends in the area (dbort and Tzu), but I’m wondering if there’s anyone I missed. Is there anyone who reads this blog who is in the Bay area I should try and say ‘hi’ to? Is there anyone who reads this blog who knows anyone in the Bay area I should say ‘hi’ to? (I think that’s one degree of separation; more degrees of separation than that, and I suspect I only want to know them if they’re going to give me money for my ideas on fixing email, or if they want to give us a place to sleep.)

This is very, very exciting. I mean… very. Exciting. Very.

OK, so I just like talking about and playing with robots. What can I say?

The cost of backups

Yojimbo-Crumb

I recently started using Yojimbo for managing bookmarks, PDFs, and other things that I find around the net. It seemed like a good choice of tool for the quick archival of websites and data that I’d like to be able to find later.

Unfortunately, my Yojimbo database has already grown to 50MB in size. Now, this is not such a problem—I have a 160GB disk in my MacBook. I could have 3000 of those files on my hard drive, and still have space for… well, a few things. However, my commitment to backup makes me realize that 50MB is really quite substantial when your backups run over a slow broadband connection. By “quite substantial,” I mean it takes 30 minutes and costs 1 US cent.

The problem is, if I add one bookmark to Yojimbo on a given day, it will force the upload of the entire database. This means that adding one bookmark requires me to copy 50MB to Amazon’s servers, and they’ll charge me a penny. If I add one bookmark a day, it will cost me roughly $4 in upload costs over the year (assuming the database doesn’t grow substantially in size, which is a bad assumption).

I dropped a note to Bare Bones Software to see if they would consider storing data in a different way to accommodate this kind of backup pattern. More likely, I’ll just have to create a weekly backup schedule for things like the Yojimbo DB, and accept that if something happens to my machine, I’ll loose a few days worth of data ninjaing.

again: qualitative research is good research

programming less visually

the open group @ allegheny college

Background (not so important)

The Students (very important)

documentation and its role in open projects

hacking

manufacturing for the little guy

invited talk

paris bound

We built this city…

The cost of backups

Pages

Archives

Admin

Background (not so important)

The Students (very important)

Pages

Tags

Archives

Admin