Illuminating the Path of Video Visualization

Illuminating the Path of Video Visualization

The notion of video visualization was coined by the PI and his postgraduate student in their 2003 IEEE VIS paper. It is a technology drawing the concepts and methodologies from volume and flow visualization, image and video processing, and vision science. It extracts meaningful information from a video data set and conveys the extracted information to users in appropriate visual representations. It is not intended to provide fully automatic solutions to the traditional problems in video processing, but involves human in the loop of intelligent reasoning while reducing the burden of viewing videos.

PI: Professor Min Chen, University of Oxford (formerly Swansea University)

CI: Professor Ian Thornton, Swansea University

Period: February 2009 - September 2013

In the subsequent work in collaboration with Stuttgart, the PI and CI introduced the concept of visual signatures in video visualization, and reported a major user study conducted at Swansea involving some 92 subjects [IEEE TVCG 2006]. This work offered an important scientific insight as to how human observers may learn to recognize visual signatures of events depicted in an abstract visual representation of a video. A bounded visual search (e.g., looking for all moving pixel clusters with 20-60 pixels) can be achieved in linear time, whist an unbounded visual search (e.g., looking for something abnormal in a video) is NP-complete.

For most practical problems in video processing and computer vision, we rarely have perfectly bounded visual search. We often search simultaneously for entities (e.g., objects, motions or events) in different classes. The models that are used to guide a search are usually incomplete and may lead to uncertainty or errors in detection, segmentation and classification. The dynamic and unpredictable nature of the input videos instigates mechanisms for heuristic reasoning and iterative decision optimization, which further depart from linear or polynomial performance. In contrast, the human eye-brain system is undeniable more powerful than any current vision system in performing visual searches, especially unbounded visual searches. Even we suppose that the human eye-brain system is a Turing machine, its 100 billion neurons and 100-500 trillion synaptic connections between neurons will unlikely to be matched by computers in the near future. Hence this raises the possibility that using video visualization to aid unbound visual search may provide a more scalable means for dealing with large volumes of video datasets.

Video visualization can be deployed in many application areas, such as scientific experimentation and computation, security industry and media and entertainment industry. However, in traditional visualization (e.g., medical visualization), the users are normally familiar with the 3D objects (e.g., bones or organs) depicted in a visual representation. In contrast, human observers are not familiar with the 3D objects depicted in a visual representation of a video because one spatial dimension of these objects shows the temporal dimension. The problem is further complicated by the fact that, in most videos, each 2D frame is the projective view of a 3D scene. Hence, a visual representation of a video on a computer display is, in effect, a 2D projective view of a 4D spatiotemporal domain. In order to for us to see 'time' without using 'time', we need to address a range of challenges in science, technology, visual perception and applications. This project is intended to continue the UK's leadership in tackling these challenges by building on the existing expertise and excellence in video visualization.

Researchers Who Worked on the Project:

  • Dr. Alfie Abdul-Rahman, currently a research officer at University of Oxford, UK
  • Dr. Kai Berger, currently a research officer at INRIA Rennes, France
  • Dr. Rita Borgo, currently a lecturer at Swansea University, UK
  • Mr. Edi Grundy
  • Dr. Richard M. Jiang, currently a research officer at Bath University, UK
  • Dr. Heike Leitte (née Jänicke), currently a Juniorprofessor at Heidelberg University, Germany
  • Dr. Karl Proctor
  • Dr. Jeyan Thiyagalingam, currently a researcher at MathWorks Cambridge, UK

Main Collaboration Partners and Colleagues

  • Dr. Suzanne I. Bevan, Swansea University, UK
  • Mr. David H. S. Chung, Swansea University, UK
  • Dr. Ben Daubney, Swansea University, UK
  • Mr. Yoann Drocourt, Swansea University, UK
  • Dr. Brian Duffy, University of Oxford, UK
  • Professor David Ebert, Perdue University, USA
  • Dr. Hui Fang, Swansea University, UK
  • Dr. Eamonn A. Gaffney, University of Oxford, UK
  • Dr. Phil Grant, Swansea University, UK
  • Dr. Iwan W. Griffiffths, Swansea University, UK
  • Professor Gunther Heidemann, University of Stuttgart, Germany
  • Mr. Benjamin Hoeferlin, University of Stuttgart, Germany
  • Dr. Marcus Hoeferlin, University of Stuttgart, Germany
  • Dr. Mark W. Jones, Swansea University, UK
  • Dr. Jackson C. Kirkman-Brown, the Birmingham Women’s Hospital and University of Birmingham, UK
  • Dr. Bob S. Laramee, Swansea University, UK
  • Dr. Phil Legg, University of Oxford (formerly Swansea University)
  • Dr. John S. D. Mason, Swansea University, UK
  • Mr. Adrian Morris, Swansea University, UK
  • Professor Tavi Murray, Swansea University, UK
  • Mr. Matthew L. Parry, Swansea University, UK
  • Dr. Kilian Scharrer, Swansea University, UK
  • Dr. David J. Smith, University of Birmingham, UK
  • Professor Peter Townsend, Swansea University, UK
  • Professor Anne E. Trefethen, University of Oxford, UK
  • Professor Daniel Weiskopf, University of Stuttgart, Germany
  • Dr. Jason X. Xie, Swansea University, UK
  • Other partners and colleagues in two spinoff projects, including Matchroom Ltd., UK and Welsh Rugby Union, UK

Publications

  • H. Jaenicke, R. Borgo, J. S. D. Mason and M. Chen, SoundRiver: Semantically-Rich Sound Illustration , Computer Graphics Forum , 29(2):357-366, 2010.
  • H. Jaenicke and M. Chen, A Salience-based Quality Metric for Visualization , Computer Graphics Forum , 29(3):1183-1192, 2010.
  • M. Hoeferlin, E. Grundy, R. Borgo, D. Weiskopf, M. Chen, I. W. Griffiths and W. Griffiths, Video Visualization for Snooker Skill Training , Computer Graphics Forum , 29(3):1053-1062, 2010.
  • R. Borgo, K. Proctor, M. Chen, H. Jaenicke, T. Murray and I. M. Thornton, Evaluating the impact of task demands and block resolution on the effectiveness of pixel-based visualization , IEEE Transactions on Visualization and Computer Graphics , 16(6):963-972, 2010.
  • M. Chen and H. Jaenicke, An Information-theoretic Framework for Visualization IEEE Transactions on Visualization and Computer Graphics , 16(6):1206-1215, 2010.
  • S. Walton, M. Chen and D. Ebert, LiveLayer -- Live Traffic Projection onto Maps , Poster Proc. of Eurographics , 2011.
  • P. A. Legg, M. L. Parry, D. H. S. Chung, M. R. Jiang, A. Morris, I. W. Griffiths, D. Marshall and M. Chen, From video to animated 3D reconstruction: A computer graphics application for snooker skills training , Poster Proc. of Eurographics , 2011.
  • R. Borgo., M. Chen, B. Daubney, E. Grundy, H. Jaenicke, G. Heidemann, B. Hoeferlin, M. Hoeferlin, D. Weiskopf and X. Xie, A survey on video-based graphics and video visualization. Eurographics 2011 STAR , 2011.
  • P. A. Legg, M. L. Parry, D. H. S. Chung, R. M. Jiang, A. Morris, I. W. Griffiths, D. Marshall and M. Chen. Intelligent filtering by semantic importance for single-view 3D reconstruction from snooker video . Proc. IEEE International Conference on Image Processing (ICIP) , 2433-2436, 2011.
  • G. K. L. Tam, H. Fang, A. J. Aubrey, P. W. Grant, P. L. Rosin, D. Marshall and M. Chen, Visualization of time-series data in parameter space for understanding facial dynamics , Computer Graphics Forum , 30(3):901-910, 2011.
  • H. Jaenicke, T. Weidner, D. Chung, R. S. Laramee, P. Townsend and M. Chen, Visual reconstructability as a quality metric for flow visualization , Computer Graphics Forum , 30(3):781-790, 2011.
  • Y. Drocourt, R. Borgo, K. Scharrer, T. Murray, S. I. Bevan and M. Chen, Temporal visualization of boundary-based geo-information using radial projection , Computer Graphics Forum , 30(3):981-990, 2011.
  • M. L. Parry, P. A. Legg, D. H. S. Chung, I. W. Griffiths, M. Chen, Hierarchical event selection for video storyboards with a case study on snooker video visualization , IEEE Transactions on Visualization and Computer Graphics , 17(12):1747-1756, 2011.
  • P. Legg, D. Chung, M. Parry, M. Jones, R. Long, I. Griffiths and M. Chen, MatchPad: Interactive glyph-Based visualization for real-time sports performance analysis , Computer Graphics Forum , 31(3):1255-1264, 2012.
  • R. Borgo, A. Abdul-Rahman, F. Mohamed, P. W. Grant, I. Reppa, L. Floridi, and M. Chen, An Empirical Study on Using Visual Embellishments in Visualization , IEEE Transactions on Visualization and Computer Graphics , 18(12):2759-2768, 2012.
  • R. Borgo., M. Chen, B. Daubney, E. Grundy, H. Jaenicke, G. Heidemann, B. Hoeferlin, M. Hoeferlin, D. Weiskopf and X. Xie, State of the art report on video-based graphics and video visualization , Computer Graphics Forum , 31(8):2450-2477, 2012.
  • J. Thiyagalingam, S. Walton, B. Duffy, A. Trefethen, and M. Chen, Complexity Plots , Computer Graphics Forum , 32(3pt1):111-120, 2013.
  • P. A. Legg, D. H. S. Chung, M. L. Parry, R. Bown, M. W. Jones, I. W. Griffiths, M. Chen, Transformation of an uncertain video search pipeline to a sketch-based visual analytics loop , IEEE Transactions on Visualization and Computer Graphics , 19(12):2109-2118, 2013.
  • R. Borgo, M. Chen, M. Hoeferlin, K. Kurzhals, P. Legg, S. Walton and D. Weiskopf, EG2013 Tutorial on Video Visualization , Eurographics Tutorial Programme , 2013.
  • R. Borgo, M. Chen, K. Kurzhals, P. Legg, S. Walton and D. Weiskopf, IEEE VIS Tutorial on Video Visualization , 2013.
  • S. Walton, K. Berger, D. Ebert and M. Chen, Vehicle object retargeting from dynamic
    traffic videos for real-time visualisation
    , to appear in The Visual Computer , Springer. Oonline version is available from September 2013.
  • B. Duffy, J. Thiyagalingam, S. Walton, D. J. Smith, A. Trefethen, J. C. Kirkman-Brown, E. A. Gaffney and M. Chen, Glyph-Based Video Visualization for Semen Analysis , to appear in IEEE Transactions on Visualization and Computer Graphics .
  • D. H.S. Chung, M. L. Parry, P. A. Legg, I. W. Griffiths, R. S. Laramee, and M. Chen, Visualizing Multiple Error-Sensitivity Fields for Single Camera Positioning , to appear in Computing and Visualization in Science .