User Pilot Tests of Technologies

Electronic Textbooks — Were they Ready to be Adopted?

Background

In 2012, electronic textbooks (e-texts) had emerged as a potentially viable option for universities. The rising cost of printed textbooks was too much of a financial burden for many students. At the same time, the ubiquity of electronic devices, such as phones, tablets, and laptops on campuses enabled most students to be able to read and collaboratively annotate e-texts. Publishers began to see e-texts as critical to their future survival. So, the University of Colorado Boulder (UCB), decided to participate in an e-text pilot to identify issues that might arise if it adopted e-texts as an option for students.

Action

I worked with Professor Caroline Sinkinson in University Libraries to conduct a study of how the McGraw Hill Courseload system might be used by professors and students to read and interact with e-texts. Nine sections of eight courses in Business, Chemistry, Computer Science, Theater, and Psychology participated in the pilot. This included nine faculty members and 729 students. In order to have a natural setting for the uses of e-texts, we let the classes unfold as the professors saw fit. We sought feedback from all people affiliated with the courses. We provided support for the faculty members in the pilot, and recorded the issues they reported as we helped them resolve the issues. We also conducted structured interviews with five faculty members and four information technology support staff. Students completed an online survey (n=86) and three of them participated in informal interviews. We analyzed our support notes as well as interview and survey results to compile our report.

Results

While we found that students and faculty members reported the usability of this system was acceptable, we recommended UCB wait to adopt e-texts. Some faculty members had disengaged with the pilot during the study. Students reported little impact on their study and learning habits. Most did not purchase a print copy of the texts. We surmised that the technology was not sufficiently mature to be adopted on our campus. We recommended that any future e-text service should provide lower costs to the students, be portable, and easy to read. We further suggested:

• Students should be given the option to obtain a color, printed version if they desire.

• Any e-text service must be accessible to people with visual challenges.

• Before an e-text service is widely deployed, a plan should be created for using it effectively in teaching and helping faculty and students see the advantages of adopting e-texts. 

The full internal report of our findings is available here. Our work was part of a 23-university study of E-Texts that was coordinated by Internet2 and EDUCAUSE. Their broader report is here.

Google Apps for Education — Should It be a Service for Students and Faculty Members?

Background

In 2012, the student email system at the University of Colorado Boulder (UCB) needed to be replaced. It had become outdated and was not meeting requirements. Faculty and staff members used Microsoft Exchange for their email, so one (albeit expensive) option was to simply make Exchange available to students. However, another option was to explore Google Applications for Education (GAFE), which would give every student email, calendars, Documents, Sheets, Slides, Hangouts, and web sites with no license cost. This piqued the interest of UCB administrators because it would be less expensive than the current system or Microsoft Exchange while substantially enhancing the current suite of tools available to students. Also, students were pressuring the University to adopt GAFE because many of them had already used GAFE in their high schools, and expected it to be available to them at the University.

Action

I formed a cross-functional team with Professor Diane Sieber, and Professor Caroline Sinkinson to conduct a pilot test of GAFE in four classes in the spring of 2012. Two classes were in Engineering, one in Education, and one in Architecture and Planning. We focused our pilot on how professors might use GAFE to teach. We conducted a heuristic review of the interface and of each tool within it. We conducted a competitive analysis of other universities and compiled a report of how they used GAFE for teaching. We also observed the ways the classes used GAFE, and interviewed the professors teaching the classes.

Results

In our full report we reported that instructors and students voiced a desire for tools that encourage participation, sharing, and collaborative knowledge building, and they demonstrated a need for online spaces that could engage audiences outside the university, and that may be used to model student participation in authentic online communities. Our study documented ways that GAFE could be used to stimulate teaching and learning. We found that GAFE could facilitate sharing artifacts among faculty and students, encourage conversations at a distance, and provide tools for course management. So we recommended UCB adopt it for all faculty and students. UCB, however, ultimately decided to adopt GAFE for students only. They continued with Microsoft Exchange for faculty and staff.

GoingOn as a Social Network — Should it be a Service for Students and Faculty Members?

Background

In 2010, the popular social network, Ning, announced they would begin to charge a fee for their service. At that time there was a significant cohort of professors at the University of Colorado Boulder (UCB) using Ning as part of their classes. Ning gave them the ability to communicate in a more media rich and flat social network, rather than the hierarchical, structured, and media-scarce threaded discussions boards or direct email. However, with the new fee structure, these professors were not able to afford Ning. A competitor emerged that we wanted to explore: GoingOn. Their product aimed to provide a learning system that was built social from the ground up.

Action

I formed a cross-functional team with Professor Diane Sieber, and Professor Caroline Sinkinson to conduct a pilot test of GoingOn in 2011. We studied four classes (two in humanities, one in social sciences, and one in engineering) and three non-course communities for one semester. We surveyed the communities that participated in the pilot. We examined Google Analytics reports for the system, which gave us information on the popularity of tools within GoingOn. We also conducted a heuristic review of the interface and of each tool within it, and observed how each class used the tools within GoingOn.

Results

In our full report we confirmed the desire among our pilot test subjects for a social network to be used in their classes and to support their communities. We found a number of compelling tools in GoingOn, such as a wiki and a blog, community sharing, and a drag-and-drop interface. We recommended GoingOn make changes in terminology, their user interface, system navigation, account creation, accessibility, and performance. We were concerned with the stability of the platform and the maturity of the company, which at the time was still in start-up mode. We ultimately recommended that the university wait to adopt GoingOn and to continue looking for another option. We recommended that we should also review Google Apps for Education and two other platforms to see if they were viable. Eventually GoingOn was not able to continue operating, and it is no longer in business.