Several Perspectives

Experience Perspective: Since there are repeatable ways to measure the amount of information or the number of frames being written out from a PC video frame buffer, measuring video playback was an interesting yet challenging design. Challenging because previous methods rely upon expensive measurements devices whose price tags routinely top $20,000. We wanted DHCAT to be self-contained, so we had to find another way to take this measurement from within the system.

Experience can take many different hits, for the case shown in Figure 2, a stream was being broadcast from a host system over a local wireless network to a remote corner of a house and decoded and displayed on a client system. From the time-code stamp you can see that the video was stalled for over seven seconds. This would mean that the viewer saw a frozen frame during that time period, which the viewer would rate poorly. Now to measure this objectively, we collect program statistics (like the test video’s actual run-time of the video its expected run-time), and correlate that to end-users experience.

Figure 4 – Video Stall for WMV playback

Using the instantaneous frame rates, we calculate the aggregate deviation (Root Mean Square Error, or RMSE) from the expected frame rate of the video. In English, if we were expecting 30 fps stream but only measure 26 at times and then an overflow of 31 at other times these are the values that are captured and summed. The Media and Acoustics Perception Lab conducted a suite of end-user research and experimentation with fixed FPS RMSE values, and mapped these to users’ mean opinion score, the average of all participants’ rating of each condition. From these data points, we developed a perceptual model, and integrated it into DHCAT so that mapping of the measured data would automatically report the subjective experience. This was the first research in the field of video quality measurement to map directly the measured RMSE to real user experiences. What’s powerful about this technique is that we found that RMSE had high predictive power about how people rate video quality. In other words, an objective measurement, RMSE, can predict real people’s subjective opinions about video quality.

We used a similar approach for measuring streaming video quality in DHCAT. First, DHCAT installs a virtual digital media adapter (DMA) on the system under test. Think of a DMA as a media “thin client,” that lives in your entertainment center, and can play back video, audio and display photos from PCs on your home network. As with the playback measurement methodology, we wanted DHCAT to be self-contained, so we created a virtual DMA that installs in the system being evaluated.

DHCAT streams video from the system being testing to the virtual DMA on the same system. The statistics of the video frames received into the DMA with the time stamps are collected and DHCAT calculates what’s called FPS Error. A 30-second video clip should take 30 seconds to play. Not 25. Not 35. 30 seconds. If a video takes longer to play then the actual run-length of the clip, that means you would have likely been seeing a frozen video frame on the DMA while the server and DMA tried to synch back up and restore playback. Our end-user testing showed that, not surprisingly, viewers don’t like to see a lot of freeze-frames in their video. Similar to the Playback usage, our team did end-user research and experimentation with fixed FPS Error values, and created a mapping of the mean opinion score (MOS) - an average over all participants for each condition. From these data points, we developed a perceptual model, which we integrated into DHCAT such we can automatically map the measured data to end-user subjective experience, and report the system capabilities in terms of the quality of experience.

When you look across these three modules it is a very nice way to package up and report out the end-user experience.

In my next blog entry I will dive into how we actually determined and implemented the perceptual models discussed here.