ACM Reference Format
超级演说家第二季Kopf, J., Cohen, M., Lischinski, D., Uyttendaele, M. 2007. Joint Bilateral Upsampling. ACM T rans. Graph. 26, 3, Article 96 (July 2007), 5 pages. DOI = 10.1145/1239451.1239547 doi.acm/10.1145/1239451.1239547.
Copyright Notice
Permission to make digital or hard copies of part or all of this work for personal or classroom u is granted without fee provided that copies are not made or distributed for profi t or direct commercial advantage and that copies show this notice on the fi rst page or initial screen of a display along with the full citation. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwi, to republish, to post on rvers, to redistribute to lists, or to u any component of this work in other works requires prior specifi c permission and/or a fee. Permissions may be requested from Publications Dept., ACM, Inc., 2 Penn Plaza, Suite 701, New Y ork, NY 10121-0701, fax +1 (212) 869-0481, or permissions@acm.
© 2007 ACM 0730-0301/2007/03-ART96 $5.00 DOI 10.1145/1239451.1239547 doi.acm/10.1145/1239451.1239547
Joint Bilateral Upsampling
Johannes Kopf University of Konstanz
Michael F.Cohen Microsoft Rearch
Dani Lischinski The Hebrew University
Matt Uyttendaele Microsoft Rearch
生活中的点点滴滴
Abstract
Image analysis and enhancement tasks such as tone mapping,col-orization,stereo depth,and photomontage,often require computing a solution (e.g.,for exposure,chromaticity,disparity,labels)over the pixel grid.Computational and memory costs often require that a smaller solution be run over a downsampled image.Although general purpo upsampling methods can be ud to interpolate the low resolution solution to the full resolution,the methods gener-ally assume a smoothness prior for the interpolation.
We demonstrate that in cas,such as tho above,the available high resolution input image may be
leveraged as a prior in the con-text of a joint bilateral upsampling procedure to produce a better high resolution solution.We show results for each of the applica-tions above and compare them to traditional upsampling methods.CR Categories:I.3.7[Computer Graphics]:Three-Dimensional Graphics and Realism—Color,shading,shadowing,and texture Keywords:bilateral filter,upsampling
1Introduction
A variety of new image analysis and image processing methods,both automatic and ur guided,have recently been demonstrated in the computer graphics and computer vision literature.The in-clude stereo depth computations [Scharstein and Szeliski 2002],image colorization [Levin et al.2004;Yatziv and Sapiro 2006],tone mapping of high dynamic range (HDR)images [Reinhard et al.2005],and applications of minimal graph cuts to image composi-tion [Agarwala et al.2004].All of the methodologies share a common problem of finding a global solution :a piecewi smooth function describing some value of interest (depth,chromaticity,ex-posure,label,etc.)over the pixel grid of the input image.Digital images continue to grow in size from one quarter million pixel video frames to multi-Megapixel digital photos,to recent Gi-gapixel images arising from specialized cameras [Flint 2007]and from stitching multiple images into a panorama [Kopf et al.2007].Such high resolutions po a difficult challenge for the methods cited above,which t
ypically require at least linear time and,more importantly,linear space to compute a global solution.Thus,in order to operate on such high resolution images,they must first be downsampled to a lower resolution to make the computation tractable.This is particularly esntial for interactive applications.Once a solution is available for the smaller downsampled image,the question then becomes how to upsample the solution to the full
original resolution of the input image.Upsampling is a fundamen-tal image processing operation,typically achieved by convolving the low-resolution image with an interpolation kernel,and resam-pling the result on a new (high-resolution)grid.Wolberg [1990]provides a good survey of common interpolation kernels.Images upsampled in this manner typically suffer from blurring of sharp edges,becau of the smoothness prior inherent in the linear inter-polation filters.
However,for the applications cited above,additional information is available in the form of the original high-resolution input image.Ignoring this information and relying on the smoothness prior alone is clearly not the best strategy.We propo to leverage the fact that we have a high-resolution image in addition to the low-resolution solution.In particular,we demonstrate that a joint bilateral upsam-pling (JBU)operation can produce very good full resolution results from solutions computed at very low resolutions.We show results for stereo depth,image colorization,adaptive tone mapping,and graph-c
ut bad image composition.
2Bilateral Filters
The bilateral filter is an edge-prerving filter,originally introduced by Tomasi and Manduchi [1998].It is related to broader class of non-linear filters such as anisotropic diffusion and robust estimation [Barash 2002;Durand and Dory 2002;Elad 2002].The bilateral filter us both a spatial (or domain )filter kernel and a range filter kernel evaluated on the data values themlves.More formally,for some position p ,the filtered result is:
J p =
1k p
∑
q ∈Ω
I q f (||p −q ||)g (||I p −I q ||),(1)
where f is the spatial filter kernel,such as a Gaussian centered over p ,and g is the range filter kernel,centered at the image value at p .Ωis the spatial support of the kernel f ,and k p is a normalizing factor,the sum of the f ·g filter weights.Edges are prerved since the bilateral filter f ·g takes on smaller values as the range distance and/or the spatial distance increa.
Recently we have en the introduction of joint (or cross )bilateral filters in which the range filter is applied to a cond guidance im-age ,˜I
,for example,when trying to combine the high frequencies from one image and the low frequencies from another [Petschnigg et al.2004;Eimann and Durand 2004].Thus,
J p =
1k p
∑
q ∈Ω
I q f (||p −q ||)g (||˜I p −˜I q ||).(2)
The only difference to (1)is that the range filter us ˜I
instead of I .2.1
Previous Work
The bilateral filter has been ud previously for various image pro-cessing tasks.Durand and Dory [2002]applied the bilateral fil-ter to HDR tone mapping and also described a fast approximation,which was recently improved upon [Paris and Durand 2006;Weiss 2006].
Ramanath and Snyder [2003]ud the bilateral filter in the context of demosaicking to improve edge nsitivity.Their method is re-stricted to Bayer patterns with a fixed small upsampling factor,and does not u a guidance image as we do.
ACM Transactions on Graphics, Vol. 26, No. 3, Article 96, Publication date: July 2007.
Durand et al.[2005]mention using a bilateralfilter to up-sample the shading results of a ray tracer.However,no details are given in the paper and no other applications are explored.
Sawhney et al.[2001]upsample stereoscopic images where one view has higher resolution than the
other.Their method estimates an alignment mapping,and then us warping andfill-in from neighboring movie frames to upsample the low-resolution image.
3Joint Bilateral Upsampling
In contrast to general purpo image upsampling,in the problems that we are interested in,additional information is available to us in the form of the original high-resolution input image.Given a high resolution image,˜I,and a low resolution solution,S,computed for a downsampled version of the image,we propo a simple method that applies a joint bilateralfilter to upsample the solution.
The idea is to apply a spatialfilter(typically a truncated Gaussian) to the low resolution solution S,while a similar rangefilter is jointly applied on the full resolution image˜I.Let p and q denote(integer) coordinates of pixels in˜I,and p↓and q↓denote the corresponding (possibly fractional)coordinates in the low resolution solution S. The upsampled solution˜S is then obtained as:
˜S
p=1
k p
∑
q↓∈Ω
S q
↓
f(||p↓−q↓||)g(||˜I p−˜I q||)(3)
This is almost identical to eq.(2)with the exceptions that we are constructing a high resolution solution as oppod to an image,and operate at two different resolutions simultaneously.
Note,that q↓takes only integer coordinates in the low resolution solution.Therefore the guidance image is only sparly sampled, and the performance does not depend on the upsampling factor(e Section5).
4Applications
In this ction we demonstrate the ufulness of the joint bilateral upsampling operation for a variety of applications.
Tone Mapping:With the increasing popularity and utility of High Dynamic Range(HDR)imaging[Reinhard et al.2005],there is a need for tone mapping methods to display HDR images on ordinary devices.A variety of such methods have been propod over the years(e[Reinhard et al.2005]for an extensive survey).Some of the methods produce high-quality results,but require solving a very large system of linear equations[Fattal et al.2002;Lischinski et al.2006].Although the systems are spar and may be solved efficiently using multi-resolution solvers[Szeliski2006],handling today’s multi-megapixel images remains a challenge:once the data exceeds the available physical memory,iteratively sweeping over the data results in thrashing.
We apply the joint bilateral upsamplingfilter as follows.Let I be the low-resolution HDR image,and T(I)the tone mapped im-age produced by some tone mapping operator.The correspond-ing low-resolution solution is then defined as the pixelwi quotient S=T(I)/I.In other words,the solution is an exposure map,which states the amount of exposure correction to be applied at each pixel. Such exposure maps are generally smooth but may have disconti-nuities along significant image edges[Lischinski et al.2006].Thus, they are ideal candidates for our upsampling technique.Note that the exposure map may have a single channel(if only the luminance has been adjusted),or multiple channels(to support arbitrary tonal manipulations).Figure2shows how applying an exposure map up-
sampled using our technique compares with a number of standard upsampling methods.The joint bilateral upsampling yields results that are visually and numerically clor to the ground truth. Colorization:A similar linear system to tho in the tone map-ping methods cited above aris in the colorization and recoloring method of Levin et al.[2004].Thus,again,processing of very large images is not tractable due to thrashing.This is also true for the more recent colorization method of Yatziv and Sapiro[2006], which does not solve a linear system,but nevertheless iteratively sweeps over the data.
To upsample a low-resolution colorization result,wefirst convert it into the YIQ color space(or to any other color space parating lu-minance from chrominance),and then apply our upsampling tech-nique to each of the two chrominance channels.Figure3shows the result.As in the tone mapping example,one can e that the JBU avoids having the chromaticity spill over edges in the image. Stereo Depth:Stereo matching is a fundamental task in image anal-ysis,who goal is to determine the disparities between pairs of corresponding pixels in two or more images.Many different ap-proaches to stereo matching have been explored over the years(for a comprehensive overview e[Scharstein and Szeliski2002]).In many of the methods an optimization problem of some sort is solved,yielding a piecewi continuous disparityfield over the en-tire image.
Our technique can be ud to upsample low resolution depth maps with guidance from the high resolution photos.Depth maps also have ideal properties for our technique.They are rather smooth,and the discontinuities typically correspond with edges in the image. Figure4shows the advantages of our technique in action.
Graph-cut bad image operations:Several recent interactive im-age editing techniques involvefinding minimal cuts in graphs. For example,the interactive digital photomontage[Agarwala et al. 2004]system us graph-cut optimization[Boykov et al.2001]to compute the least objectionable ams when fusing together veral pre-aligned photographs.The result of the optimization is a label map,indicating for each pixel in the composite which photograph it originates from.
We tested our joint bilateral upsampling technique with an image stitching application.Here,the ur constrains a number of pix-els to come from a certain input image.The stitching algorithm then computes a label map,which assigns a label to each of the remaining unconstrained pixels,such that the resulting ams are least conspicuous.
This application differs fundamentally from the previous ones,be-cau here we have a quantized solution(a discrete number of la-bels),rather than a continuous one.Furthermore,in this ca there are multiple full resolution images.
We apply our technique in the following way:suppo we want to compute the label for a pixel.Each low-resolution solution pixel with a non-zero bilateral weight votes for it’s label.The winning label is the one that has aggregated the highest total weight.Figure 5demonstrates our technique for this application.
5Performance and Accuracy
The complexity of the joint bilateral upsampling operation is O(Nr2)where N is the output image size and r is the domainfilter radius.The performance is proportional to the output size and not to the upsampling factor,becau the domainfilter is always applied to the low resolution solution.For all results we have ud a5×5 Gaussian,which is very fast but still has enough spatial support to pull solution values from some distance.Our implementation takes approximately2conds per megapixel of output.
96-2 • Kopf et al.
ACM Transactions on Graphics, Vol. 26, No. 3, Article 96, Publication date: July 2007.
2x2
4x4
8x8
16x16
32x32
Tone Mapping M S
E
2x2
4x4
8x8
16x16
u盘格式怎么转换32x32
Colorization
M S
E
2x2
自然风景图片壁纸4x4
8x8
16x16
32x32
Depth from Stereo
M S E
Figure 1:MSE error profiles for various applications and upsampling methods.
This is significantly faster than running the original algorithms on the full resolution images.For exam
ple,the tone mapper took 80conds for a 3.1megapixel image,while our upsampling took only 6conds to upsample a smaller solution which was computed much faster.The colorization solver of Levin et al.[2004]was even slower,and needed veral minutes for a megapixel sized im-age.As noted above,due to the memory issue we cannot run a very high resolution solution so upsampling a low resolution solution is our only way to approach such large images.
The JBU is strictly local with a very small memory footprint.Large images can be computed in a single sweep,where only parts are paged in at any time.We have successfully applied our method to upsample tone mapping solutions for multi-gigapixel images [Kopf et al.2007].
In our experiments,we generally t the domain filter’s Gaussian σd to 0.5with 5×5support.The range filter Gaussian σr is strongly application dependent.The following default values worked well for the images we tried:colorization,stereo depth,and graph-cut labelings ud images with color values normalized to the [0,1]in-terval.σr =0.1worked well on most images.The tone mapping application works with unbounded luminance values.We found that tting σr to the standard deviation of the values has always given good results.
Figure 1shows MSE error profiles for the JBU compared to other upsampling methods.To compute t
he errors,we computed a full resolution solution (or simply ud the full resolution color image or depth map for colorization and stereo depth)as ground truth.We then downsampled by factors of 2,4,8,16,and 32in each direction.Then,we performed upsampling using various methods and plotted the difference from the ground truth.Our filter performed well at all downsampling levels,and,as expected,the relative improvement incread with each additional level of down sampling.
Not surprisingly,the MSE error increas with the upsampling fac-tor.But in practice it often turns out that the application limits how much one can downsample the problem.The results we show are for solutions on quite tiny downsampled images.Since some of the applications require some UI,you need enough image left to,for example,scribble on the hints for tone mapping or colorization.
6Conclusion
We have demonstrated the benefits of a joint bilateral upsampling strategy when a high resolution prior is available to guide the inter-polation from low to high resolution.The four applications we have shown all improve relative to previous “blind”upsampling meth-ods.We believe this strategy is applicable to a number of other do-mains within and beyond image processing.For example,a global illumination solution computed over a coar simplified mesh can be upsampled to a finer mesh.The domain filter’s kernel might
be measured in geodesic distance,while the range kernel would be over the Gaussian Sphere (differences in normal).We look forward to trying the joint bilateral upsampling on this and other problems of interest in computer graphics.
References
A GARWALA ,A.,D ONTCHEVA ,M.,A GRAWALA ,M.,D RUCKER ,S.,C OLBURN ,A.,C URLESS ,B.,S ALESIN ,D.,AND C OHEN ,M.2004.Interactive digital photomontage.ACM Transactions on Graphics 23,3(Proc.SIGGRAPH 2004),294–302.
B ARASH ,D.2002.A fundamental relationship between bilateral filtering,adaptive smoothing,and the nonlinear diffusion equa-tion.IEEE Transactions on Pattern Analysis and Machine Intel-ligence 24,6,844–847.B OYKOV ,Y.,V EKSLER ,O.,AND Z ABIH ,R.2001.Fast approxi-mate energy minimization via graph cuts.IEEE Transactions on Pattern Analysis and Machine Intelligence 23,11,1222–1239.D URAND ,F.,AND D ORSEY ,J.2002.Fast bilateral filtering for the display of high-dynamic-range images.ACM Transactions on Graphics 21,3(Proc.SIGGRAPH 2002),257–266.D URAND ,F.,H OLZSCHUCH ,N.,S OLER ,C.,
C HAN ,E.,AN
D S ILLION ,F.X.2005.A frequency analysis of light transport.ACM Transactions on Graphics 24,3(Proc.SIGGRAPH 2005),1115–1126.
E ISEMANN ,E.,AND D URAND ,F.2004.Flash photography en-hancement via intrinsic relighting.ACM Transactions on Graph-ics 23,3(Proc.SIGGRAPH 2004),673–678.E LAD ,M.2002.On the bilateral filter and ways to improve it.IEEE Transactions on Image Processing 11,10,1141–1151.
F ATTAL ,R.,L ISCHINSKI ,D.,AND W ERMAN ,M.2002.Gradient domain high dynamic range compression.ACM Transactions on Graphics 21,3(Proc.SIGGRAPH 2002),249–256.F LINT ,G.2007.The Gigapxl Project .
K OPF ,J.,U YTTENDAELE ,M.,D EUSSEN ,O.,AND C OHEN ,M.F.2007.Capturing and viewing gigapixel images.ACM Transactions on Graphics 26,3(Proc.SIGGRAPH 2007).L EVIN ,A.,L ISCHINSKI ,D.,AND W EISS ,Y.2004.Colorization using optimization.ACM Transactions on Graphics 23,3(Proc.SIGGRAPH 2004),689–694.L ISCHINSKI , D.,F ARBMAN ,Z.,U YTTENDAELE ,M.,AND S ZELISKI ,R.2006.Interactive local adjustment of tonal val-ues.ACM Transactions on Graphics 25,3(Proc.SIGGRAPH 2006),646–653.
Joint Bilateral Upsampling • 96-3
宝泉旅游ACM Transactions on Graphics, Vol. 26, No. 3, Article 96, Publication date: July 2007.
Upsampled
Result Nearest
Neighbor
Bicubic
Gaussian Joint
Bilateral Ground Truth
Figure2:Tone Mapping:The low resolution exposure map solution at left is at scale relative to thefinal upsampled result next to it.Detail ints to the right show results from Nearest Neighbor,Gaussian Interpolation,Bicubic Interpolation,Joint Bilateral Upsampling,and ground truth bad on a full resolution solution.Note that JBU does not exhibit the blocking and halo artifacts of the other upsampling
methods.
Figure3:Colorization:The low resolution pudo-colored chrominance solution is at left.Detail ints corresponding to four different upsampling methods are shown on the right(with incread saturation).Note that the JBU result does not exhibit color spills.
P ARIS,S.,AND D URAND,F.2006.A fast approximation of the
bilateralfilter using a signal processing approach.Proceedings
of ECCV2006).
P ETSCHNIGG,G.,S ZELISKI,R.,A GRAWALA,M.,C OHEN,M.,
H OPPE,H.,AND T OYAMA,K.2004.Digital photography with
flash and no-flash image pairs.ACM Transactions on Graphics
23,3(Proc.SIGGRAPH2004),664–672.
R AMANATH,R.,AND S NYDER,W.2003.Adaptive demosaicking.
Journal of Electronic Imaging12,4,633–642.
R EINHARD,E.,W ARD,G.,P ATTANAIK,S.,AND D EBEVEC,P.
2005.High Dynamic Range Imaging.Morgan Kaufmann.
S AWHNEY,H.S.,G UO,Y.,H ANNA,K.,K UMAR,R.,A DKINS,
S.,AND Z HOU,S.2001.Hybrid stereo camera:an ibr ap-
形容温柔的诗句
男士护肤品十大排行proach for synthesis of very high resolution stereoscopic image
quences.Proceedings of SIGGRAPH2001,451–460.
S CHARSTEIN,D.,AND S ZELISKI,R.2002.A taxonomy and eval-
uation of den two-frame stereo correspondence algorithms.In-
ternational Journal of Computer Vision47,1–3,7–42.
S ZELISKI,R.2006.Locally adapted hierarchical basis precon-
ditioning.ACM Transactions on Graphics25,3(Proc.SIG-
GRAPH2006),1135–1143.
T OMASI,C.,AND M ANDUCHI,R.1998.Bilateralfiltering for
gray and color images.Proceedings of ICCV1998,839–846.
W EISS,B.2006.Fast median and bilateralfiltering.ACM Trans-
actions on Graphics25,3(Proc.SIGGRAPH2006),519–526.
W OLBERG,G.1990.Digital Image Warping.IEEE Computer
Society Press,Los Alamitos,CA.
Y ATZIV,L.,AND S APIRO,G.2006.Fast image and video col-
orization using chrominance blending.IEEE Transactions on
Image Processing15,5,1120–1129.
96-4 • Kopf et al.
ACM Transactions on Graphics, Vol. 26, No. 3, Article 96, Publication date: July 2007.
Figure4:Stereo Depth:The low resolution depth map is shown at left.The top right row shows details from the upsampled maps using different methods.Below each detail image is a corresponding3d view from an offt camera using the upsampled depth
map.
商务皮包Nearest
Neighbor
Bicubic
Gaussian Joint Bilateral
Figure5:Graph Cut bad Photomontage.Upper left:two original aligned images.Upper center:the downsampled images and the resulting labeling solution.Upper right:a composite generated using the JBU-upsampled labeling solution.The label colors are superimpod over the image.The bottom row shows a detail that highlights the advantages of using JBU over standard upsampling methods.
Joint Bilateral Upsampling • 96-5
ACM Transactions on Graphics, Vol. 26, No. 3, Article 96, Publication date: July 2007.
