, terenceh@sfu.ca
Stereoscopic Image Generation for Arbitrary Viewing Rotations
Naoki Chiba SANYO Electric Co., Ltd.
Terence T. Huang Simon Frar University
, terenceh@sfu.ca
Abstract We propo a method to generate stereoscopic images for arbitrary viewing rotations from a collection of images. Virtual reality systems, which can provide photo-realistic omni-directional views, are collecting a lot of attention. Existing methods for creating stereoscopic images, however, have a problem that limits viewing rotations to be only horizontal panning. To solve this problem, we have developed a method that can generate images not just for panning but also for other rotation orientations such as tilting and twisting. The method us a single digital-camera mounted on a computer-driven pan-tilt unit who rotating center has a fixed offt to the projection center. We prent how to generate stereoscopic images from the images acquired with such camera configuration. Our method is completely image-bad without recovering 3D depth. Experiments using
real images show that our method is very effective to generate images for arbitrary viewing rotations. Our method can generate images supporting six degrees of freedom in 3D space; in other words, it reconstructs a 4D plenoptic function.
驴打滚是哪里的特产
e rea r ch r e ct s ca r ri e r n e rence T Hu n
f s i n e rnsh SA e ct r c C
1INTRODUCTION
Virtual reality systems, which can provide photo-realistic views, are collecting a lot of attention. One of the first systems is QuickTime VR [CHEN95]. After constructing a panoramic image from a collection of images acquired by a panning camera, the system can interactively playback ctions of the 360-degree horizontal panorama. Methods extending the work to construct spherical mosaics have been propod. They u images taken with a hand-held camera [SH97] or a camera mounted on a computer-controlled pan-tilt unit [WM96].
Methods for generating an arbitrary view can be categorized into two types. One is to build a 3D model by using computer vision techniques. The other is to generate a view by using acquired images, which is called image-bad rendering (IBR).
Among IBR techniques for providing arbitrary views of an environment model, two approaches have been propod. One is to u an omni-directional camera that can capture a horizontal 360-degree view at once by using a convex mirror [YY01]. The other approach is to stitch a collection of images captured with a standard camera, which is called image mosaics.
One of the rearch trends in image mosaics is to extend the number of degrees of freedom for viewing. In the following subction, we review previous methods for generating stereoscopic views. The approach has a strong connection with the plenoptic function which is well-known in computer graphics rearch. After reviewing related work in this area, we prent their problems to which we will show a solution with our method.
1.1Stereo Panorama
By using a single rotating camera, methods for creating panoramic stereoscopic views have been propod in [IYT92], [PBE99] and [NK00]. Ishiguro et al. ud two vertical stripes of a camera swivelling around a rotation center with the optical axis remaining perpendicular to the tangent of the camera path in [IYT92]. Two horizontal panoramas were built by stitching the vertical stripes. They ud rotation angles measured by the computer-driven turntable. Stereo pairs were obtained from the panoramas to estimate depth information for building depth maps by a mobile robot.
Peleg et al. extended the work to u a video camera on a levelled tripod in [PBE99]. After estimating camera motion between video frames, two stereo panoramas were constructed again by stacking dual stripes from each image. Instead of estimating depth, they generated stereoscopic views for a visualization purpo.
Nayar et al. created stereo spherical panoramas called 360 x 360 Mosaics [NK00]. A “slice camera” employing a parabolic mirror and a telecentric lens enlarged the vertical field of view of each slit to 360 degrees. Each 180-degree half of the slit was ud in constructing a spherical panorama for one eye viewpoint.
1.2The Plenoptic Function and Novel View Generation
Adelson and Bergen have introduced the concept of the plenoptic function [AB91] that can describe the intensity of light en at every viewpoint (V x, V y, V z) at every viewing angle (θ, φ) for every wavelength (λ) at every time (t). Fig. 1 illustrates the function for one single viewpoint in one direction. The 7D function can be written as р = P(θ, φ, V x, V y, V z, λ, t). If we can record a real scene as a smooth plenoptic function, then any arbitrary view of the scene can be “played back” at any time. If the scene is static without light changes and captured with a RGB camera, the last two dimensions can be omitted.
Fig. 1 The plenoptic function for one single
viewpoint in direction (θ, φ)
McMillan and Bishop defined the 5D plenoptic function in their system bad on a t of cylindrical panoramas imaged at various points in space [MB95].
If we assume that radiance along any ray remains constant within a clod space, we can reduce the number of parameters in the space. The Light Field Rendering [LH96] and The Lumigraph [GGSC96] prented a 4D parameterization of the plenoptic function if it can be constrained within a bounding box. Their reprentation us a t of coordinates on two planes when a light ray cross them. The function is р = P(s, t, u, v), where (s, t) and (u, v) are coordinates on the planes.
Naemura et al. also showed reduction of the function parameterization to 4D if the space is divided by a 2D surface into an obrving area and an obrved area. When we capture images with camer
as on the surface, we can reduce the number of dimensions by projecting light rays onto a planar screen, described by 2D parameters P and Q, facing each direction of rays. The function is described by р= P(P, Q, θ, φ). Since the shape of the surface is not restricted to be planar, it can also be spherical.
Shum and He’s Concentric Mosaics [SH99] showed that the plenoptic function can be further scaled down to three dimensions by confining viewpoints within a planar circular region. One of their suggested implementations in fact coincides with tho in [IYT92, PBE99].
Recent upgraded systems include Unstructured Lumigraph Rendering by Buehler et al. [BBMGC01], Dynamically Reparameterized Light Fields by Isakn et al. [IMG00], and Plenoptic Stitching by Aliaga et al. [AC01].
1.3 Limitations
Fig. 2 contrasts the camera configurations ud in the previous method [PBE99] and in our method. We rotate a camera along the surface of a sphere by using a computer-driven pan-tilt unit (PTU). The offt between the path of camera’s projecting center and the center of rotation is fixed. The PTU provides vertical rotation in addition to horizontal rotation as illustrated in Fig. 2 (b).
The stereo panoramas are limited to reproducing only horizontal parallax. The lack of vertical parallax prevents the viewer from gaining stereo nsation if, for example, the virtual camera is twisted around the optical axis or is oriented to the polar regions. Furthermore, in the ca of spherical panoramas [NK00], the output image quality tends to degrade significantly towards the poles. With current omnidirectional imaging apparatus such as parabolic mirrors, it is difficult to obtain a 360-degree image slice with uniform resolution.
2.2 apture Routine
he capture routine is performed by a software prog h by subd RENDERING
nce we have discretely sampled images, we can gene .1 Rendering Routine
租房合同协议书ig. 3 (b) shows a top-down view of a virtual image plan
Previous IBR systems bad on the plenoptic function have some problems as follows. In Light Field Rendering and The Lumigraph , the allowable viewing space is affected by the geometric constraints of the acquisition system configured to work with the chon parameterization for the plenoptic function. As Shum et al. have noted in [SH99], it is difficult to capture an inside-looking-out Lightfield or Lumigraph of a real scene, which has also been examined in [TS03]. While the above issue is looned with Concentric Mosaics , which do permit an omnidirectional viewing range of a surrounding scene on a flat plane, the method again lacks both vertical field of view and vertical parallax. (a) Previous method [PBE99] (b) Our method
Fig. 2Camera configuration.
C
T ram that nds rotation commands to the PTU to orient the camera at desired angles and shutter commands to the digital camera to capture images. The routine relies on an internal reprentation of the viewing sphere which provides pairs of preci spherical angles for each capture point. The sphere model can be a collection of vertices and polygons defining its geometric surface. Becau w
e want to capture at uniformly spaced angles, we have opted a surface model bad on the geodesic dome structure.
A geodesic dome can be constructed from scratc
1.4 The Propod method
We propo a method to solve the above problems that can allow the ur to explore a virtual environment with stereoscopic views. Unlike previous stereo panoramas which restrict viewing rotations to panning, it enables the ur to rotate their heads in all orientations by panning, tilting and twisting. The propod method also permits lateral motion within the viewing space.
ividing a tetrahedron, an octahedron, or an icosahedron. One that is subdivided from an icosahedron distributes the most equal weight for each child face and thus gives the most uniform spherical surface structure. Another advantage of subdivided geodesic domes is that we can control the number of subdivisions to adjust the sampling frequency at discrete steps.
Our method is an extension of [IYT92, PBE99, SH99] and prerves the four-dimensionality of [NTKH97] by exploiting a special spherical parameterization of the plenoptic function and a novel ac
quisition technique with a computer driven pan-tilt-unit. We also show a method of rendering novel images that deals with approximation errors in the plenoptic function during resampling. Experiments using real images demonstrate that we can generate very high quality images for six degrees of freedom and stereo disparity regardless of view orientation.
3
O rate an arbitrary view of the scene. We describe a rendering routine that achieves this resampling procedure and how we deal with the involved approximation error.
The remainder of the paper is organized as follows. We prent the configuration for image capturing and the procedure for novel view rendering. After showing the performance of our method with experiments, we conclude with a discussion on performance issues and future work.
3
F
e w and a sphere O which defines the path o
f the capturin
g camera’s optical center. The coordinate pair (u, v) indicates a pixel position on w . The rendering routine is basically the mapping from captured image pixels to the virtual image pixels. One way to implement this mapping
2 ACQUISITION
2.1 Camera Configuration
function is to follow a geometrical approach instead of solving the algebraic relationships which might take a lengthy process. A simple implementation is through ray-tracing.
The output of the ray-tracer will directly tell us where the
source pixel for each virtual image element should
come from in terms of the capture angle of the source
image and its location in local source image space. First, a
ray is shot from the projection center V L towards (u, v) in
world coordinate space. Its interction with O is returned
as a spherical coordinate pair (θ
u,v
, φu,v) using Cr as the
world origin. A cond ray leaves the world origin towards
the previous interction point and finds where it cross
the corresponding source image. We can render the entire
virtual image by following the same procedure for all (u, v).
In the ca where we do not have an image for (θu,v,
φu,v)
.2Handling Approximation Error
鬓毛衰的正确读音here is a source of approximation error due to the
fact
source image for ray R
actually corresponds to ob
point A. One method fo
pixel Pc which is on
position of pixel Pc
广东产假
obtained by scaling tho of
the ratio of (f / (f
formed by PcIcC and
Fig. 5Ray adjustment
少年闰土小练笔
4
no view rotation and the other经济分析报告
showing a 90-degree twist. Next, we include experiments
show
speed, exposure,
nd white balance to manual ttings. Table 1 shows the
specification of the came ffective focal length and
the dial lens distortion were measured by using a
varia
, we end up getting the pixel P b that
ject point B instead of object
r reducing this error is to choo
a ray clor to the object point A. The爱护眼睛
in local im ge co nates can
P b with a factor described as
+ d)) if we note the similar triangles
PbIcX.
a ordi be
, we apply the nearest-neighbour strategy and u the
source image having the clost capture angle in the cond
step.
3
T
that we only have a finite number of source images.
This approximation error is vere becau it leads us to
u a sample from a different ray direction during the
ray-tracing procedure. [Fig. 4]
Fig ed with approximation error.
(b)n error re technique
that when jacent
(a) (b)
. 4(a) shows an image render
shows the approximatio duced with our
In Fig. 5, we can e we u an ad
EXPERIMENTS
4.1Overview
First, we have obtained two ts of novel images
rendered using the method described in [PBE99] and our
own method. In each t, there are two stereo pairs of
images, one showing
ing the effect of insufficient sampling rate for image
acquisition and the effect of inaccurate alignment of the
camera’s optical axis.
For image capturing, we ud an Olympus C-3040
digital camera with its Software Development Kit. We can
control various functions of the camera (such as shutter,
zoom, focus and etc.) and transfer captured image data
through a USB cable. We fixed the shutter
a
ra. The e
ra
tion of Zhang’s calibration method.
(a) Previous method
Fig. 3 Rendering a novel image. In (a), only two stripes from each
source image are ud to create the stereo cylindrical panoramas. In
our method in (b), a ray-tracer determines corresponding source
pixels for each virtual image element.
Table 1
Focal length
36
For the capture stage, we utilized a pan-tilt unit (model no. PTU-46-70 with nodal-gimbal upgrade op
tion) manufactured by Directed Perception, Inc. The spe z Pan Range: 159˚ le and 159˚ right Resolution: 0.771 arc minutes (0.012857˚)
ig
4.2 previous work with our method, we have o for crea ]. The In the first pair of images without view rotation, we cond pair does not show any parallax effect after the virtual view is rotated 90 degrees clockwi. [Fig. 7]
images demonstrate clear tereo disparity in the ca where there is no view rotation, w ere the viewing direction is rotated [Fig. 8] Motion parallax along ertical translation is also demonstrated in Fig 11.
errors will increa and ig. 9, Fig. 10]
age quality. Since the PTU has 0.001 degrees, we can add more ages easily. This is our future work.
mm cification of the unit is as follows:
ft z Tilt Range: 31˚ up and 47˚ down
z We have also made a special jig that offts the camera from the rotation center with adjustable le
ngth from 0cm to 18cm, and interests the camera’s optical axis with the rotation center. Fig. 6 shows a photo of the PTU and the jig.
Fig. 6 A photo showing the PTU and the camera j Comparison with a previous method
For comparison of ch n the method described by Peleg et al. ting stereo cylindrical panoramic images [PBE99camera is offt 15cm from the rotation center of the PTU which moves in 0.5˚ steps. To create the stereo pair, we ud two 22-pixel-wide vertical stripes from each image. The stripes are located 150 pixels to the left and 150 pixels to the right of the image center, respectively.
can obrve significant horizontal parallax. However, the For the experiment of our method, we have captured about 5000 images of a conference room with the following capture configuration:
z Sampling step: approx. 0.7˚ in each direction z Camera offt from rotation center: 15cm
饭局狼人
With our propod method, the s as ell as in the ca wh 90 degrees clockwi. v 4.3 Insufficient Sampling Rate for Acquisition &
Misaligned Camera
If we capture images less frequently by reducing the subdivision level of the spherical surface model for guiding the acquisition process, approximation (35mm-film equivalent) the aliasing effect will become apparent. Similarly, if the camera’s optical axis is not aligned well to interct the rotation center, additional errors will be introduced in the rendered result. [F Image Size (pixels) 1024 x 768 Im ge Format
JPEG
a 5 DISCUSSION
We have propod a method that can generate stereoscopic views for arbitrary rotations. Unlike cylindrical stereo panorama, we do not restrict the ur’s head to horizontal rotation, panning. The experimental result of the 90-degree rotation in Fig. 8 showed that our method can recover a significant parallax which cannot be generated with the previous methods.
Our method is not restricted to stereoscopic image generation. We also showed the result of vertical translation of a single camera. Our method can provide motion parallax as well.
We can generate very high-quality images due to the large number of captured images. We ud 84
captured images to render a 1024x768 virtual image who focal length is 1000 pixels. The processing time to render the image was 42 conds on a Pentium 4 2.3GHz with 4GB memory. We have also implemented a caching function to reduce the time of disk access. Without the disk caching, the processing time was 864 conds.
As en in Fig. 12, visible am-lines are obrvable on clo-ups becau of finite number of images. To increa the quality of the generated images, we can think in two directions. One is to recover 3D depth for interpolating views. This is an active area of rearch in computer vision. One example is to apply multi-baline techniques to recover 3D depth in cylindrical stereo panoramas [JOS04]. The other direction is to increa the number of source images. The experiment in Fig. 9 showed that the number of source images significantly affect the image quality. If we capture a greater number of images, we can easily increa the im its rotation resolution up to im The drawback is its capturing time and rendering time. If we increa the number of capturing images, the capturing time increas linearly. Since it takes much time to capture, even a light change coming through windows can cau a problem. The scene that we can recover is limited to pure indoor scenes. Using a video camera might solve the problem partially becau we can rotate the camera during capturing. This, however, affects the image quality, becau video camcorders have li
mited resolution such as 640x480 pixels, which is much lower than that of a digital still camera. We need to wait for high-resolution video camcorders in future.
6 views for arbitrary orientations. Unlike revious methods, the allowable rotations are not restricted tation. By using a collection of images single digital still camera rotated by a ing.
es and generate a virtual view from them. We howed the performance against low sampling rate and structing a ystem to cover complete spherical viewing range is our B91] E.H. Adelson, J.R. Bergen. The Plenoptic Function and the ble hs. roceedings of the 28th annual conference on Computer graphics ured Lumigraph Rendering . Proceedings of the 8th annual conference on Computer graphics and interactive Image Bad pproach to Virtual Environment Navigation. Computer Graphics: Reco H96] M. Levoy and P. Hanrahan. Light Field rendering. ACM odeling: An age-Bad Rendering System. Computer Graphics: Proc. of 0 x 360 Mosaics. Proc. of EE Conference on Computer Vision and Pattern Recognition, 97] T. Naemura, T. Takano, M. Kaneko, H. Harashima. ay-Bad Creation of Photo-Realistic Virtual World. Proc. of , pages 59-68 1997. gle of IEEE Conference on Computer Vision and attern Recognition, pages 395–401, June 1999. a Approach. International Journal of Computer ision, vol. 47, pages 149-160, 2002. Concentric s 299-306, August 999. 1-38, 2002. Computer Vision, vol. 48, no. 3, pages 59-172, 2002. ic 7, pages 251–258, August
1997. CONCLUSION
We have propod a method that can generate stereoscopic p to horizontal ro captured by a computer-driven pan-tilt unit, we can generate high quality views for any rotations by panning, tilting, and twist We have described a system for capturing imag a procedure to s misalignment of the optical axis to the center of rotation.
In our examples, we did not generate views for complete omni-directions. The method can be applicable for that purpo without any modification. Con s future work.
References
[A Elements of Early Vision. Computation Models of Visual Processing, M. Landy and J.A. Movshon, eds., MIT Press, Cambridge, 1991.
[AC01] D. Aliaga, I. Carlbom. Plenoptic stitching: a scala method for reconstructing 3D interactive walk throug P and interactive techniques, pages 443-450, 2001.
[BBMGC01] C. Buehler, M. Bos, L. McMillan, S. Gortler, M. Cohen. Unstruct 2techniques, pages 425 – 432, 2001.
[CHEN95] S. E. Chen. QuickTime VR - An A Proc. Of SIGGRAPH 95, pages 29–38, August 1995.
[GGSC96] S. J. Gortler, R. Grzeszczuk, R. Szeliski, and M. F. Cohen. The Lumigraph. ACM SIGGRAPH, pages 43-54, 1996. nstruction Using Stereo Multi-Perspective Panoramas, n Japane)
[IMG00] A. Isaken, L. McMillan, S. Gortler. Dynamicall Reparameterized Light Fields. P y roceedings of the 27th annual onference on Computer graphics and interactive techniques, pages suji. Omnidirectional tereo. IEEE Transactions on Pattern Analysis and Machine i and S. Sugimoto. Omnidirecional S03] T. Todoroki and H.Saito. Arbitrary Viewpoint Image ynthesis via Light Field Rendering with Omni-directional Camera. Technical Report 2003-CVIM-138, pp. 45-50, 2003. M96] T. Wada and T. Matsuyama. Appearance Sphere: ackground Model for Pan-tilt-zoom Camera. Proc. Of ICPR, ages 718-722, 1996.
Y01] T. Yagi and N.Yokoya. Omnidirectional Vision: A survey n Sensors and Applications. IPSJ Transactions on Computer ision and Im c 297-306, 2000.
[IYT92] H. Ishiguro, H. Yamamoto, and S. T S Intelligence, vol. 14, pages 257–262, 1992.
[JOS04] W. Jiang, M. Okutom 3D Technical Report of IEICE, PRMU2003-211, pp. 77-82, 2004. (In
Japane)
[L SIGGRAPH, pages 31-42, 1996.
[MB95] L. McMillan and G . Bishop. Plenoptic M Im SIGGRAPH 95, pages 39–46, August 1995.
[NK00] S. Nayar and A. Karmarkar. 36IE vol. 2, pages 388-395, 2000.
[NTKH R VSMM '97
[PBE99] S. Peleg and M. Ben-Ezra. Stereo Panorama with a Sin Camera. Proc.P
[PS02] P. Peer and F. Solina. Panoramic Depth Imaging: Single Standard Camer V
[SH99] H.Y . Shum and L.W. He. Rendering with Mosaics. Proceedings of SIGGRAPH 99, page 1[SH99b]
[SK02] S. Seitz and J. Kim. The Space of All Stereo Images. International Journal of Computer Vision, vol. 48, no. 1, pages 2
[SKS02] S. Seitz, A. Kalai, H.Y. Shum. Omnivergent Stereo. nternational Journal of I 1
[SS97] R. Szeliski and H. Y. Shum. Creating Full View Panoram Image Mosaics and Environment Maps. Proc. of ACM IGGRAPH’9S
[T S IPSJ SIG
[W B p
[Y o V age Media No. SIG13(CVIM3), pages 1-17, 2001. (I
