Table of Contents(I) [Fold]
Introduction †
This page provides an implementation of our image-based visual frustum intersection used in
T. Yano, S. Nobuhara, T. Matsuyama: "3D Shape from Silhouettes in Water for Online Novel-View Synthesis", IPSJ Transactions on Computer Vision and Applications, 2013.
- paper: coming soon
- movie: coming soon
- bibtex:
@article{yano13shape, title = {3D Shape from Silhouettes in Water for Online Novel-View Synthesis}, author = {Tomohiko Yano and Shohei Nobuhara and Takashi Matsuyama}, journal = {IPSJ Transactions on Computer Vision and Applications}, year = {2013}, }
The code here provides an implementation of "image-based visual frustum intersection". It computes appearance of intersections by multiple 3D objects, and can be used for image-based visual hull rendering by computing the intersection of visual cones or frusta. Unlike standard IBVH in literature, this DOES NOT exploit the epipolar geometry, and hence is not strictly optimized best for perspective case. Instead, it can be used for non-perspective environment such as underwater photography.
Please notice that the CSG part is completely done by OpenCSG. In this sense, this program is just a wrapper of this library.
Source code †
License †
This source code is provided under the the terms of the GNU General Public License (GPL), Version 2. See http://www.gnu.org/licenses/gpl-2.0.html for more information.
Copyright (C) 2013 Tomohiko Yano, Shohei Nobuhara and Takashi Matsuyama This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License, Version 2, as published by the Free Software Foundation. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
Download links †
If you use this software, please consider citing the aforementioned paper in any resulting publication.
- Source code:
ibvfi-20130804.zip (Runs on both Windows and Linux. Requires Qt4, libQGLViewer, and OpenCSG installed. Tested with Intel HD Graphics 3000, AMD Radeon HD 6650M, nVidia GeForce GT640 and Quadro 4000) - Win32 binary: ibvfi-x86-20130804.zip (precompiled binary with dlls)
How to use the program †
Usage †
After unzipping the downloaded file, you will have the following files in a single directory named "ibvfi".
- ibvfi.exe : The main binary
- Qt{Core4,Gui4,OpenGL4,Xml4}.dll : Runtime libs
- data/cone/main.csg (+ cam??.vc) : A sample data of visual cones
- data/frustum/main.csg (+ cam??.vc) : A sample data of visual frusta
Run ibvfi.exe, then drag-and-drop either of data/cones/main.csg or data/frusta/main.csg. Then you will see the intersection result, and can change the viewing directions by mouse (left-dragging).
Unchecking the "Enabled" checkbox in the "CSG" tab disables the intersection process by CSG. This lets you see the visual cones or frusta to be intersected as is.
Trouble shooting †
If you find no 3D objects are shown in the GUI, then your GPU driver is not likely to support OpenGL frame-buffer-object. Notice that even if the GPU chip itself is capable of fbo, your driver may kill it. For example, Sony Vaio-Z with dual GPU (Intel + nVidia or AMD) may do this trick when the Intel GPU is turned on.
To verify this quickly, you can use "glewinfo" provided by GLEW. With my Vaio-Z (VPCZ2) using the factory default driver, it reports that all the modern OpenGL functions are missing when the Intel chip is turned on.
GLEW version 1.9.0 Reporting capabilities of pixelformat 2 Running on a GDI Generic from Microsoft Corporation OpenGL version 1.1.0 is supported ... GL_ARB_framebuffer_object: MISSING ... GL_EXT_framebuffer_object: MISSING
You can install the driver by Intel manually, and it enables them as follows.
GLEW version 1.9.0 Reporting capabilities of pixelformat 3 Running on a Intel(R) HD Graphics 3000 from Intel OpenGL version 3.1.0 - Build 9.17.10.2875 is supported ... GL_ARB_framebuffer_object: OK ... GL_EXT_framebuffer_object: OK
Please notice that installing the driver by Intel will let your Vaio get BSOD immediately when you connect "Power Media Dock". You have been warned.
Compile †
The source zip file contains "main.pro", the project file for qmake. So you can simply do
$ qmake $ make $ ./ibvfi
Please note that you need to install the following libraries in advance.
- Qt4
- libqglviewer
- GLEW
- OpenCSG
For Debian/wheezy, you can install them by
$ sudo apt-get -f install libqt4-dev libqt4-opengl-dev qt4-qmake libqglviewer-qt4-dev libglew-dev libopencsg-dev
File format †
This program uses two types of files, .csg and .vc formats. The .vc format is used to define a 3D object, say visual cones for 3D video, to be used for CSG. The .csg file lists .vc files to be used for intersection computation.
Visual cone format (.vc) †
This is a text file start by a magic keyword “VisualCone”. The file looks like this.
VisualCone 50 49 1 # vertices -451.238 362.921 279.081 -29.5315 -33.0406 -9.62701 ... # side triangles 0 1 2 0 2 3 ... # base polygons 0.619908 -0.564087 -0.545454 1 1 49
- The 1st line is the magic keyword indicating the file format.
- The 2nd line has 3 fields, from left to right, \(N_v\), \(N_f\), and \(N_b\).
- \(N_v\) is the total number of vertices used in this 3D object.
- \(N_f\) is the total number of triangles in this object.
- This is mainly to define the side surface of the visual cone / frustum.
- \(N_b\) is the total number of polygons (quads, …).
- This is mainly to define the base polygon of the visual cone (or top and base planes of visual frustum)
- The lines start by '#' are simply ignored.
- The next \(N_v\) lines define the 3D positions of \(N_v\) vertices, say \(v_0,\dots,v_{Nv−1}\).
- Following the vertex definitions, the next \(N_f\) lines define the triangles by specifying three vertex IDs [0:\(N_v\)−1] for each of them.
- The order of the three vertices defines the surface direction (normal vector) of the polygon by the right hand rule. For example, the line "0 1 2" indicates that the triangle is define by connecting \(v_0\), \(v_1\), and \(v_2\), and its normal vector is given by \((v_1-v_0)\times(v_2-v_0)\).
- The normal direction is one of the most crucial points in CSG. Be careful about the order.
- The last Nb lines define the base polygons.
- Each line starts by 3 floating point numbers indicating the normal vector of the plane \((n_x,n_y,n_z)\).
- The next 4th field indicates the number of contours in this polygon (say \(n\)). If this is a simple polygon, it should be 1. If it has a hole, or it consists of two isolated polygons, this number can be 2.
- Then 5th to 2\(n\)+4th fields form \(n\) pairs of two integers each of which indicating the start and end of the vertex indices to define a contour. For example, a pair of “1 49” above means that the contour connects \(v_1,v_2,v_3,\dots,v_{49}\). These vertices should appear in CCW order (or right-hand rule).
- To define a hole, use CW-order. Google "OpenGL winding rule" (Non-zero rule), "OpenGL tessellation", for more info.
CSG format (.csg) †
The .csg file looks as follows.
BBOX -200 -200 0 200 200 200 VC 1 cam01.vc VC 2 cam02.vc ... VC 15 cam15.vc
- The 1st line indicates the bounding box of the scene; xmin, ymin, zmin, xmax, ymax and zmax.
- This is mainly used to define the camera positions.
- The rest of the lines define the visual cones.
- The 1st field "VC" is a magic string.
- The 2nd field is the ID of the cone. This should be a unique non-negative number among the all visual cones.
- The 3rd field is the filename of the .vc file. This should be the relative path from the .csg file directory; meaning that if the program is reading "a/b/c.csg" and a VC line specifies "d/e.vc", then the program opens "a/b/c/d/e.vc".
Live View Synthesis †
Since the "live" part implementation should heavily depend on your hardware to maximize the performance, we leave this part for you and just suggest how to make it.
To realize "live view synthesis", you need to implement your "capture" program which implements the following functions.
- Capture
- Silhouette extraction
- Visual-frustum generation
In our case, we used Pointgrey CMLN-13S2C-CS cameras with external trigger for synchronized capture and a simple pixel-wise background subtraction for the silhouette extraction.
And to connect your capture program with this app, we strongly recommend you to do it first via .vc files on a shared memory with lock; meaning that your capture program writes .vc files on tmpfs with (advisory) file-write-locks, and you will modify this app to load the files as fast as possible (with file-read-locks, of course). This will simplify the debugging of your capture program by paying the additional cost for the file I/O (on the shared memory). Then, once your capture implementation reaches "production" quality, you can integrate the both codes to maximize the performance.
Also, you can optimize the following points to achieve a better performance.
- binalize the .vc file format,
- use named-pipes instead of regular files on the ramdisk,
- parallelize the .vc file loading,
- enable double-buffering (or pipelining) to .vc file loading and rendering.