Detail

Practical content search over millions of videos

Even though the accuracy of content based video search systems (CBVS) has drastically improved, high accuracy systems tend to be too inefficient for interactive search. Therefore, to strive for real-time web-scale CBVS, we perform a comprehensive study on the different components in a CBVS system to understand the tradeoffs between accuracy and speed of each component. Directions investigated include exploring different low-level and semantics based features, testing different compression factors and approximations during video search, and understanding the time v.s. accuracy trade-of. Extensive experiments on data sets consisting of more than 1,000 hours of video showed that through a combination of effective features, highly compressed representations, and one iteration of reranking, our proposed system can achieve an 10,000-fold speedup while retaining 80% accuracy of a state-of-the-art CBVS system. We further performed search over 1 million videos and demonstrated that our system can complete the search in 0.975 seconds with a single core.

Semantic search in video is a novel and challenging problem in information and multimedia retrieval. Existing solutions are mainly limited to text matching, in which the query words are matched against the textual metadata generated by users. This talk will also discuss an approach for event search both with and without example videos, but without text metadata. The system relies on substantial video content analysis and allows for both low-level and semantic search over a large collection of videos. We share our observations and lessons in building such a system, which may be instrumental in guiding the design of future systems for search in video.