When a new learning technology enters a formal pilot phase at UBC, it is released and supported in specific courses for a trial run, typically lasting one to two terms. After this trial run completes, the LT Hub gathers feedback from users to understand how the pilot went. Details of this evaluation process and outcome are provided below.
Evaluation Process
A formal pilot typically includes a limited number of courses and three groups of users: instructors, teaching assistants, and students. For each course, the evaluation team looks at how these groups perceived the technology and its usefulness for teaching and learning in their context.
Most often, feedback is gathered in the following ways:
- Instructors:
- Provide context and intended use case before or soon after the start of the pilot
- Participate in a reflective interview after having sufficient experience with the tool
- TAs:
- Participate in a reflective interview or focus group session after having sufficient experience with the tool
- Students:
- Participate in a class-wide survey after having sufficient experience with the tool
- When relevant, optionally participate in a focus group session
Instructor time commitments
When an instructor’s course(s) are included in a formal pilot, the instructor will be asked to assist the evaluation team in five small ways:
- Provide details of their context(s) and use case(s) for the technology
- Distribute an evaluation student survey (online or paper-based) in each course at an appropriate time
- Actively encourage student participation in sharing feedback
- Connect any TAs for the course(s) with the evaluation team
- Set aside time for a reflective interview near the conclusion of the pilot
Instructors are not expected to perform any data analysis or reporting.
Secondary use of data
Instructional teams can use the raw data collected from evaluation student surveys and focus groups in their course(s). However, LT Hub will not normally seekBREBapproval, so anyone wishing to apply the data to wider research goals may need to consider a BREB application, ideally prior to data collection. Feel free to contactAdriana Briseno-Garzonat CTLT for additional information and advice.
Final Outcome
All information the LT Hub gathers for a formal pilot is summarized by the evaluation team in a report. This report is shared with learning technology governance groups and forms one piece of a larger assessment they undertake in deciding on campus-wide support for the piloted tool.
Reports do not make recommendations regarding the adoption of the technology at UBC nor make claims regarding the tool’s effectiveness. The scope is limited to summarizing how people perceived the technology during the pilot—focusing on strengths, weaknesses, and best ways to implement.
Once governance groups review the report and decide on an outcome for the technology, the decision is reported through theLT Hub website.