About

The European Training Network on Full Parallax Imaging organizes an intensive workshop on Multi-camera Image Processing for Media Production. The workshop instructor is Dr. Frederik Zilly, Head of Group Computational Imaging at Fraunhofer IIS, Erlangen, Germany. The workshop is intended for doctoral students dealing with imaging though master students at their final stage of graduation can also attend it. Details about workshop content, timetable and registration are given below.

Summary

Advanced applications in the field of visual effects such as synthetic aperture rendering, virtual camera movements, depth-map and normal map generation for live action footage or virtual reality applications such as 360° degree panorama videos require the acquisition and processing of multi-camera data. In this workshop, relevant underlying algorithms from the field of computational imaging such as multi-camera calibration, rectification, depth estimation and virtual view synthesis and presented and elaborated.

Different data-sets with pre-calibrated multi-camera video data will be used within the workshop along with Matlab files demonstrating the basic algorithmic steps within the image processing framework.

Within the workshop, the students will learn to operate the framework. Subsequently, they can extend the basic processing chain with own implementations of alternative algorithms and can design visual effect algorithms based on the available multi-view multi-depth data.

The workshop is intended for advanced master and doctoral students at TUT. Demo data is made available according to the Consortium Agreement of the Marie-Curie action ETN-FPI (WP1 providing data to WP2).

Students attending the workshop sessions receive one ECTS credit. Students implementing a project based on the workshop materials and data can receive up to 5 ECTS credits depending on the scope and difficulty of the project, which should be agreed with the project instructor.

Organization

The workshop will be held at the premises of the Centre of Immersive Visual Technologies (CIVIT), Tietotalo, 4th floor. First session is on Wednesday, 17 August at 14:15 in TC442.

Timetable

Wednesday, 17 August, 14:15 – 15:45, TC442

• Overview of Multi-Camera Image Processing for Light-Field Applications (Big Picture)
• Presentation of the Test-Data
• Overview of the Matlab Sample Files for Multi-Camera Image Processing

Thursday, 18 August, 10:15 – 11:45, TC442

• Stereo-Block-Matcher and Merging of Disparity Maps
• Post-Processing Disparity Maps using Cross-Bilateral Filter
• View-Rendering using Depth-Image Based Rendering
• Relighting Using Blender
• Preparation of Exercises

Tuesday, 23 August, 10:15 – 11:45, TC442

• Review of Exercises, Q&A
• Discussion of Algorithmic Improvements
  • Advanced Disparity Estimation
  • Advanced Merging Strategies
  • Occlusion-Handling Strategies
• Extension towards non-planar camera arrays

Wednesday, 24 August, 10:15 – 11:45, TC442

• Discussion of target Applications
  • Lightfield-Video Refocussing
  • HDR-Video, dynamic exposure series
  • Depth-Based Compositing
  • Chroma-Keying

Thursday, 25 August, 10:15 – 11:45, TC442

• Inserting Content Captured at TTY into Matlab-Chain
• Rendering for Virtual Reality devices

Registration and further info

For registration, interested students must send an email to Robert Bregovic at (robert.bregovic (at) tut.fi).

For more information, please contact the workshop instructor Frederik Zilly at (frederik.zilly (at) adminc.org) or the workshop organizer Atanas Gotchev (atanas.gotchev (at) tut.fi)

Instructor

Frederik Zilly is head of the Computational Imaging and Algorithms group in the Moving Picture Department of Fraunhofer IIS in Erlangen, Germany. His research focus lies in multi-camera image processing algorithms. He received a diploma degree in physics from Free University of Berlin, and a Ph.D. degree in Computer Science from Technical University Berlin. Before joining Fraunhofer IIS, he was scientific project manager in the Immersive Media and 3D Video group in the Image Processing Department at Fraunhofer HHI in Berlin where he was involved in several German and European research projects related to 3DTV. In this function he was mainly responsible for the development of the stereoscopic analyzer (STAN) and coordinated the activities of the Fraunhofer HHI concerning the European research project MUSCADE project. Zilly has been honored for his work on the assistance system STAN (Stereoscopic Analyzer) with the Award for Outstanding Merit in Young Scientists, the Rudolf Urtel Prize 2011.

Further Reading

Over-View Multi-Camera Content Creation:

• [Bartczak11] , [Tanimoto06], [Zilly13], [Zilly14]

Geometry & Rectification

• [Faugeras93b], [Georgiev13], [Hartley04], [Zilly12c]

Bilateral Filtering

• [MüllerM10], [Riemens09], [Tomasi98]

Stereo-Disparity Estimation

• [Bleyer07], [Boykov01], [Riechert12c]

Multi-Camera Disparity Estimation

• [Hirschmüller08], [Kim13], [Okutomi93], [Zilly14]

View Synthesis

• [Balogh07], [Köppel10], [MüllerK08], [Stefanoski13], [Zitnick04]

 

Literature

[Balogh07] T. Balogh, P. Kovacs, and A. Barsi. Holovizio 3D display system. In 3DTV-Conference (3DTV-CON), Kos Island, Greece, May 2007.

[Bartczak11] B. Bartczak, P. Vandewalle, O. Grau, G. Briand, J. Fournier, P. Kerbiriou, M. Murdoch, M. Müller, R. Goris, R. Koch, and R. van der Vleuten. Display-Independent 3D-TV Production and Delivery Using the Layered Depth Video Format. IEEE Trans. on Broadcasting, vol. 57, issue 2, part 2, pp. 477-490, June

[Bleyer07] M. Bleyer and M. Gelautz. Graph-cut-based stereo matching using image segmentation with symmetrical treatment of occlusions. Signal Processing: Image Communication, vol. 22, issue 2, pp. 127-143, Feb. 2007.

[Boykov01] Y. Boykov, O. Veksler, and R. Zabih. Fast Approximate Energy Minimization via Graph Cuts. IEEE Trans. on Pattern Analysis and Machine Intelligence (PAMI), vol. 23, issue 11, pp. 1222-1239, Nov. 2001.

[Faugeras93b] O. Faugeras. Three-Dimensional Computer Vision (Artificial Intelligence). The MIT Press, Nov. 1993.

[Georgiev13] M. Georgiev, A. Gotchev, and M. Hannuksela. A fast and accurate re-calibration technique for misaligned stereo cameras. In International Conference on Image Processing (ICIP), pp. 24-28, Melbourne, Australia, Sept. 2013.

[Hartley04] R. I. Hartley and A. Zisserman, Multiple View Geometry in Computer Vision, Cambridge University Press, ISBN: 0521540518, second edition, 2004.

[Hirschmüller08] H. Hirschmüller. Stereo Processing by Semiglobal Matching and Mutual Information. IEEE Trans. on Pattern Analysis and Machine Intelligence (PAMI), vol. 30, issue 2, pp. 328-341, Feb. 2008.

[Kim13] C. Kim, H. Zimmer, Y. Pritch, A. Sorkine-Hornung, and M. Gross. Scene reconstruction from high spatioangular resolution light fields. ACM Trans. on Graphics (TOG) – Proc. of ACM SIGGRAPH 2013, vol. 32, issue 4, no. 73, pp. 1-12, Anaheim, USA, July 2013.

[Köppel10] M. Köppel, P. Ndjiki-Nya, D. Doshkov, H. Lakshman, P. Merkle, K. Müller, T. Wiegand. Temporally Consistent Handling of Disocclusions with Texture Synthesis for Depth-Image-based Rendering. In International Conference on Image Processing (ICIP), pp. 1809-1812, Hong Kong, Sept. 2010

[MüllerK08] K. Müller, A. Smolic, K. Dix, P. Merkle, P. Kauff, and T. Wiegand. View Synthesis for Advanced 3D Video Systems. EURASIP J. on Image and Video Proc., 2008.

[MüllerM10] M. Müller, F. Zilly, and P. Kauff. Adaptive cross-trilateral depth map filtering. In 3DTV-Conference (3DTV-CON), Tampere, Finland, June 2010.

[Okutomi93] M. Okutomi and T. Kanade. A Multiple-Baseline Stereo. IEEE Trans. on Pattern Analysis and Machine Intelligence (PAMI), vol. 15, issue 4, pp. 353-363, April 1993.

[Riechert12c] C. Riechert, F. Zilly, M. Müller, and P. Kauff. Real-Time Disparity Estimation Using Line-Wise Hybrid Recursive Matching and Cross-Bilateral Median Up-Sampling. In International Conference on Pattern Recognition (ICPR), Tsukuba, Japan, Nov. 2012.

[Riemens09] A. K. Riemens, O. P. Gangwal, B. Barenbrug, and R.-P. M. Berretty. Multi-step joint bilateral depth upsampling. Proc. SPIE, Visual Communications and Image Processing, vol. 7257, pp. 1-12, 2009.

[Stefanoski13] N. Stefanoski, O. Wang, M. Lang, P. Greisen, S. Heinzle, and A. Smolic. Automatic View Synthesis by Image-Domain-Warping. IEEE Trans. on Image Processing, vol. 22, no. 9, pp. 3329-3341, Sept. 2013.

[Tanimoto06] M. Tanimoto. Overview of free viewpoint television. Signal Processing: Image Communication, 21 (2006) 454–461, 2006.

[Tomasi98] C. Tomasi and R. Manduchi. Bilateral Filtering for Gray and Color Images. In International Conference on Computer Vision (ICCV), pp. 839-846, Jan. 1998.

[Zilly12c] F. Zilly, C. Riechert, M. Müller, W. Waizenegger, T. Sikora, and P. Kauff. Multi-Camera Rectification using Linearized Trifocal Tensor. In International Conference on Pattern Recognition (ICPR), pp. 2727-2731, Tsukuba, Japan, Nov. 2012.

[Zilly13] F. Zilly, M. Müller, and P. Kauff. Generic Content Creation for 3D Displays. In 3D-TV System with Depth-Image-Based Rendering, Springer New York. Ce Zhu, Yin Zhao, Lu Yu, Masayuki Tanimoto (Editors), pp. 39-68, Jan. 2013.

[Zilly14] F. Zilly, C. Riechert, M. Müller, P. Eisert, T. Sikora, P. Kauff. Real-time generation of multi-view video plus depth content using mixed narrow and wide baseline. Journal of Visual Communication and Image Representation, Special Issue on 3D Video Processing, vol. 25, issue 4, pp. 632-648, 2014.

[Zitnick04] C.L. Zitnick, S.B. Kang, M. Uyttendaele, S. Winder, and R. Szeliski. High-quality video view interpolation using a layered representation. ACM Trans. on Graphics (TOG) – Proc. of ACM SIGGRAPH 2004, vol. 23, issue 3, pp. 600-608, Los Angeles, USA, Aug. 2004.