New AI system turns science videos into interactive virtual experiments

How it works

An AI system that turns watched science videos into interactive virtual lab pages you can click through

Watches a science demo video and lists each step
Draws a map of every step and what actions are allowed
Writes code for an interactive lab page on your screen
Runs the code and fixes any problems found

Science experiment videos are everywhere online, from classroom demonstrations to YouTube tutorials. They show exactly how to set up an experiment and what happens when you change something. But there is one big problem: you can only watch. You cannot touch, move, or try the experiment yourself.

Researchers at The Hong Kong Polytechnic University, including Xusen Hei, Jiali Chen, Jinyu Yang and Yi Cai, set out to fix that. They published their work on November 2, 2026, describing a new benchmark called Lab2Sim and a method called EduVideo2Code that together aim to turn watched videos into interactive virtual experiments ¹.

What you could do with it

Students could click through real science demo videos and run the experiments themselves on their screens instead of just watching ¹.
Researchers and educators could turn a library of watched science videos into interactive virtual lab pages that enforce the correct order of operations ¹.
Learners could try changing conditions in a virtual experiment after seeing it demonstrated in a video, learning through doing rather than just looking ¹.

How it works

The team started by gathering 218 science experiment videos and building detailed descriptions for each one, along with 11,881 rubric items that serve as checkpoints for quality. A rubric is simply a scoring guide — a list of specific things to check to see whether the virtual experiment matches the real one.

The core of the system is EduVideo2Code. It works in three stages. First, it watches a video and pulls out the experimental procedures step by step, from high-level goals down to tiny actions. Second, it turns those steps into a state machine — a diagram that maps every possible state the experiment can be in and what operations are allowed in each state. This is important because in a real lab, certain actions can only happen under certain conditions: you cannot heat a liquid before you pour it into the beaker. The state machine enforces those rules automatically. Third, the system generates the actual code for the interactive page and then tests that code by running it, fixing any problems that come up along the way ¹.

Guess

By how many points did the method improve equipment accuracy?

What the tests showed

The researchers tested their method using existing multimodal AI models as the starting point. They found that these models could usually produce pages that run without errors, but they often got the details of the lab equipment wrong — mixing up beakers, misplacing tubes, or drawing instruments that looked nothing like what appeared in the video.

EduVideo2Code fixed most of those problems. Across two different backbone models, it improved scores on all four evaluation dimensions the team measured. The biggest jump was in apparatus fidelity — how closely the virtual lab equipment matched what was shown in the video — where the method added 24.15 points ¹.

What this means for students

Right now, this is a research benchmark, not a product you can download. But the idea behind it is straightforward: take any good science demonstration video and give it a body. Students could pause a video, click a button, and explore the experiment themselves — changing variables, watching what happens, and learning by doing rather than just watching.

This story reports on a trend for information only. It is not advice or encouragement to do any of it.

Key facts

Lab2Sim features 218 videos of science experiments, complete with evidence-backed specifications and 11,881 rubric items

EduVideo2Code uses a state machine to enforce preconditions for each experimental operation
Existing multimodal models can generate runnable pages but struggle to faithfully reproduce lab equipment
EduVideo2Code improved all four evaluation dimensions across two backbone models, with a 24.15-point gain in apparatus fidelity
The research comes from The Hong Kong Polytechnic University and was published on November 2, 2026