Usability ROI and e-learning

Boxes and Arrows just published this scathing review of one of Nielsen/Norman Group's $122 reports about the return on investment for usability work. The wonderful Peter Merholz covered most of this stuff at the last Adaptive Path workshop we attended in San Francisco, but I was mostly interested in the last half of the review, where the authors link to a better study, share a simple matrix of some of the ways usability gives value, and offer some advice for ensuring that usability is measurable and integrated into every design project. And I thought the conclusion was really solid:

User experience practitioners have long known in their guts that their efforts truly add value when developing products or systems. And we've been long frustrated to see our abilities relegated to the tactical end of the product development process, where we're given poorly considered ideas and told to make them into useful, usable, and desirable products. By concretely demonstrating our impact on the success of our works, we will find ourselves involved earlier in the process, helping determine what will be made, not just how to make it.

This is mostly referring to corporate web-design projects, but I think it applies equally well to the design of learning experiences (especially e-learning, of course). It sounds as though professional instructional designers are often asked to turn bad learning materials into mediocre e-learning modules, rather than being given the chance (and resources) to create better ways to help students build knowledge.

When we study the usability of a web site, we focus on how difficult it is for users to complete tasks and find information. Substitute "learners" for "users", and you see why e-learning could benefit from these efforts to justify user-experience design in the usability community. It's easy to see the "soft" benefits of making an online course more usable, and you could measure the benefits of a redesign -- satisfaction, information retention, amount of support required, but shouldn't it go beyond that? How do you calculate the real ROI of online-course usability/design when in most cases the students represent a captive market?

There seems to be an essential feedback loop missing, perhaps related to the quasi-business status of the current post-secondary education system. I've felt that lack of accountability in the worst online courses I've taken so far -- the ones with bad organization, boring materials, ghost-in-the-machine instruction and limited feedback. But those courses are requred for the program I'm taking, so I suck it up and jump through the hoops, since I've already paid for the term and there are no other courses available. If I complain to the instructor, I'll be viewed as a troublemaker and be punished with poor marks, and even if the administration finds out that the course is terrible, and they initiate an expensive redesign of the course and remedial intervention for the instructor...how would that affect their bottom line? Students might be more engaged, but most of them are stuck in the program for the duration anyway, and it won't necessarily affect their purchasing decisions.

I suppose in the corporate training sector, the ROI focus may be more on how training results are manifested in business metrics: improved sales, reduced costs, etc. In general, how and what is learned is less important to a corporation than how much the training costs and how it will improve the bottom line of the company. When you get into these secondary cause/effect relationships, it gets difficult to attribute much value to the actually usability of the online course. So even in this case, although everyone knows usability is important, how do you prove it?