A typical production pilot goes like this. The institution sends three lectures. The vendor returns three polished videos. Everybody agrees they look much better. Then somebody senior asks whether it is worth twenty lakh across the programme, and nobody can answer, because looking better was never in dispute.
The pilot proved the wrong thing. Here is how to design one that does not.
Decide the question first
Write down, before any footage moves, the sentence you want to be able to say at the end. It is usually one of these:
- “Produced modules were completed by X% of the cohort against Y% for unproduced ones.” The retention question. The strongest one, and the hardest to run.
- “We can produce N hours a week at a consistent standard.” The throughput question — the right one if a filing deadline is what is driving you.
- “Our faculty approved 20 of 20 videos without requesting changes to content.” The acceptance question, and the one to run first if faculty resistance is your real blocker.
- “Production cost less per hour than our current captioning supplier.” The substitution question. Easiest to answer, smallest prize.
Pick one. A pilot designed to answer four questions answers none of them well, and the reporting becomes an argument about which number mattered.
If you chose the retention question
This is the one worth doing properly, so it gets the detail.
Split within a course, not between courses. Produce half the modules of one course and leave the other half as recorded. Comparing two different courses compares two different subjects, two different lecturers and two different difficulty curves — and proves nothing about production.
Interleave, do not front-load. If all the produced modules come first, you have measured the well-known fact that engagement decays across a term. Alternate them.
Fix the metrics before you look. Completion rate, average view duration, the timestamp of median drop-off, rewatch rate. Write them down. Deciding what to measure after seeing the data is how a pilot becomes a press release.
Do not tell students which is which. Not a deception — simply do not announce it. An announced upgrade measures the announcement.
Run a full module, not three lectures. Three videos is a demo. A module is a behaviour.
Agree in advance what a negative result means. The most important step and the one always skipped. If the delta is small, what will you do? Answer it before you know.
What your LMS already gives you
Almost every institution has this data and almost nobody looks at it. Canvas, Moodle and Blackboard all report per-video completion and view duration; capture platforms like Panopto and Kaltura report in-video drop-off timestamps. You do not need new instrumentation, you need somebody with access and two hours.
One term of your own data is worth more than every study we have cited, because it is about your students, your subject and your programme.
Being honest about what a pilot cannot tell you
Read this before you present the result
- Your sample is small. One course in one cohort will not produce a statistically robust effect, and presenting it as one invites a statistician in the room to take the whole thing apart. Report it as what it is: an observation from your own programme.
- Novelty is real. The first produced module in a course of plain ones may perform well because it is different, not because it is better. Interleaving helps; it does not eliminate this.
- Completion is not learning. A student who finishes a video has not necessarily understood it, and the published research is clear that production quality moves engagement far more reliably than it moves test scores.
- Term-to-term comparisons are contaminated. Different cohort, different admissions cycle, different everything. Compare within a cohort or do not compare.
What we commit to on our side
- A flat fee, agreed before the work, with no per-revision charges — so the cost of the pilot is not a function of how much you ask for.
- The engagement report written up whatever it says, including a null result.
- The method stated in enough detail that your own analytics team can reproduce it.
- No requirement to proceed, and no claim on the produced videos if you do not. They are yours either way.
The version we would run if you asked us to choose
One twenty-hour course. Ten modules produced, ten left as recorded, interleaved. Four metrics, fixed in writing beforehand. One term. A written report at the end, and a decision taken on it.
That is ₹75,000 and eight weeks, and at the end of it you own something no vendor can give you: a number about your own students that you can take into any meeting for the next three years.