Crisp Boundary Detection Using Pointwi
心理暗示法Mutual Information
爱情其实很简单Phillip Isola,Daniel Zoran,Dilip Krishnan,and Edward H.Adelson
Massachutts Institute of Technology
{phillipi,danielz,dilipkay,eadelson}@mit.edu
Abstract.Detecting boundaries between mantically meaningful ob-
jects in visual scenes is an important component of many vision algo-
rithms.In this paper,we propo a novel method for detecting such
boundaries bad on a simple underlying principle:pixels belonging to
the same object exhibit higher statistical dependencies than pixels be-
longing to different objects.We show how to derive an affinity measure
bad on this principle using pointwi mutual information,and we show
that this measure is indeed a good predictor of whether or not two pixels
reside on the same object.Using this affinity with spectral clustering,we
canfind object boundaries in the image–achieving state-of-the-art re-
sults on the BSDS500datat.Our method produces pixel-level accurate
boundaries while requiring minimal feature engineering.
Keywords:Edge/Contour Detection,Segmentation
1Introduction
Semantically meaningful contour extraction has long been a central goal of com-puter vision.Such contours mark the boundary between physically parate ob-jects and provide important cues for low-and high-level understanding of scene content.Object boundary cues have been ud to aid in gmentation[1–3], object detection and recognition[4,5],and recovery of intrinsic scene properties s
uch as shape,reflectance,and illumination[6].While there is no exact definition of the“objectness”of entities in a scene,datats such as the BSDS500gmen-tation datat[1]provide a number of examples of human drawn contours,which rve as a good objective guide for the development of boundary detection algo-rithms.In light of the ill-pod nature of this problem,many different approaches to boundary detection have been developed[1,7–9].
As a motivation for our approach,first consider the photo on the left in Figure1.In this image,the coral in the foreground exhibits a repeating pattern of white and gray stripes.We would like to group this entire pattern as part of a single object.One way to do so is to notice that white-next-to-gray co-occurs suspiciously often.If the colors were part of distinct objects,it would be quite unlikely to e them appear right next to each other so often.On the other hand, examine the blue coral in the background.Here,the coral’s color is similar to the color of the water behind the coral.While the change in color is subtle along岁月长留
红色部队2Phillip Isola,Daniel Zoran,Dilip Krishnan,Edward H.Adelson Sobel & Feldman 1968Arbeláez et al. 2011 (gPb) Dollár & Zitnick 2013 (SE)Our method Human labelers
侧脸漫画
Fig.1.Our method suppress edges in highly textured regions such as the coral in the foreground.Here,white and gray pixels repeatedly occur next to each other.This pattern shows up as a suspicious coincidence in the image’s statistics,and our method infers that the colors must therefore be part of the same object.Converly,pixel pairs that straddle the coral/background edges are relatively rare and our model assigns the pairs low affinity.From left to right:Input image;Contours recovered by the Sobel operator [10];Contours recovered by Doll´a r &Zitnick 2013[8];Contours recovered by Arbel´a ez et al.(gPb)[1];Our recovered contours;Contours labeled by humans [1].Sobel boundaries are crisp but poorly match human drawn contours.More recent detectors are more accurate but blurry.Our method recovers boundaries that are both crisp and accurate.
this border,it is in fact a rather unusual sort of change –it only occurs on the narrow border where cor
al pixels abut with background water pixels.Pixel pairs that straddle an object border tend to have a rare combination of colors.
The obrvations motivate the basic assumption underlying our method,which is that the statistical association between pixels within objects is high,whereas for pixels residing on different objects the statistical association is low.We will u this property to detect boundaries in natural images.
One of the challenges in accurate boundary detection is the emingly inher-ent contradiction between the “correctness”of an edge (distinguishing between boundary and non-boundary edges)and “crispness”of the boundary (precily localizing the boundary).The leading boundary detectors tend to u relatively large neighborhoods when building their features,even the most local ones.This results in edges which,correct as they may be,are inherently blurry.Becau our method works on surprisingly simple features (namely pixel color values and very local variance information)we can achieve both accurate and crisp con-tours.Figure 1shows this appealing properties of contours extracted using our method.The contours we get are highly detailed (as along the top of the coral in the foreground)and at the same time we are able to learn the local statistical regularities and suppress textural regions (such as the interior of the coral).
It may appear that there is a chicken and egg problem.To gather statistics within objects,we need to already have the object gmentation.This problem can be bypasd,however.We find that natural objects produce probability density functions (PDFs)that are well clustered.We can discover tho clusters,and fit them by kernel density estimation,without explicitly identifying objects.This lets us distinguish common pixel pairs (arising within objects)from rare ones (arising at boundaries).
Crisp Boundary Detection Using Pointwi Mutual Information3 In this paper,we only look at highly localized features–pixel colors and color variance in3x3windows.It is clear,then,that we cannot derive long feature vectors with sophisticated spatial and chromatic computations.How can we hope to get good performance?It turns out that there is much more information in the PDFs than one might atfirst imagine.By exploiting this information we can succeed.
Our main contribution is a simple,principled and unsupervid approach to contour detection.Our algorithm is competitive with other,heavily engineered methods.Unlike the previous methods,we u extremely local features,mostly at the pixel level,which allow us tofind crisp and highly localized edges,thus outperforming other methods significantly when more exact edge localization is required.Finally,our method is unsupervid and is able to adapt to each given image independently.The resulting algorithm achieves state-of-the-art results on the BSDS500gmentatio
n datat.
The rest of this paper is organized as follows:we start by prenting related work,followed by a detailed description of our model.We then proceed to model validation,showing that the assumptions we make truly hold for natural images and ground truth contours.Then,we compare our method to current state-of-the-art boundary detection methods.Finally,we will discuss the implications and future directions for this work.
2Related Work
Contour/boundary detection and edge detection are classical problems in com-puter vision,and there is an immen literature on the topics.It is out of scope for this paper to give a full survey on the topic,so only a small relevant subt of works will be reviewed here.
The early approaches to contour detection relied on local measurements with linearfilters.Classical examples are the Sobel[11],Roberts[12],Prewitt[13] and Canny[14]edge detectors,which all u local derivativefilters offixed scale and only a few orientations.Such detectors tend to overemphasize small, unimportant edges and lead to noisy contour maps which are hard to u for subquent higher-level processing.The key challenge is to reduce gradients due to repeated or stochastic textur
es,without losing edges due to object boundaries.
As a result,over the years,larger(non-local)neighborhoods,multiple scales and orientations,and multiple feature types have been incorporated into con-tour detectors.In fact,all top-performing methods in recent years fall into this category.Martin et al.[15]define linear operators for a number of cues such as intensity,color and texture.The resulting features are fed into a regression classifier that predicts edge strength;this is the popular Pb metric which gives, for each pixel in the image the probability of a contour at that point.Doll´a r et al.[16]u supervid learning,along with a large number of features and mul-tiple scales to learn edge prediction.The features are collected in local patches in the image.
4Phillip Isola,Daniel Zoran,Dilip Krishnan,Edward H.Adelson Recently,Lim et al.[7]have ud random forest bad learning on image patches to achieve state-of-the-art results.Their key idea is to u a dictionary of human generated contours,called Sketch Tokens,as features for contours within a patch.The u of random forests makes inference fast.Doll´a r and Zitnick[8] also u random forests,but they further combine it with structured prediction to provide real-time edge detection.Ren and Bo[17]u spar coding and oriented gradients to learn dictionaries of contour patches.They achieve excellent contour detection results on BSDS500.
The above methods all u patch-level measurements to create contour maps, with non-overlapping patches making independent decisions.This often leads to noisy and broken contours which are less likely to be uful for further processing for object recognition or image gmentation.Global methods utilize local mea-surements and embed them into a a framework which minimizes a global cost over all disjoint pairs of patches.Early methods in this line of work include that of Shashua and Ullman[18]and Elder and Zucker[19].The paper of Shashua and Ullman ud a simple dynamic programming approach to compute clod, smooth contours from local,disjoint edge fragments.
The globalization approaches tend to be fragile.More modern methods include a Conditional Random Field(CRF)prented in[20],which builds a probabilistic model for the completion problem,and us loopy belief propaga-tion to infer the clod contours.The highly successful gPb method of Arbel´a ez et al.[1]embeds the local Pb measure into a spectral clustering framework[21, 22].The resulting algorithm gives long,connected contours higher probability than short,disjoint contours.
The rarity of boundary patches has been studied in the literature
[23].We measure rarity bad on pointwi mutual information[24](PMI).PMI gives us a value per pat
ch that allows us to build a pixel-level affinity matrix. This local affinity matrix is then embedded in a spectral clustering framework [1]to provide global contour information.PMI underlies many experiments in computational linguistics[25,26]to learn word associations(pairs of words that are likely to occur together),and recently has been ud for improving image categorization[27].Other information-theoretic takes on gmentation have been previously ,[28].However,to the best of our knowledge,PMI has never been ud for contour extraction or image gmentation.
3Information theoretic affinity
Consider the zebra in Figure2.In this image,black stripes repeatedly occur next to white stripes.To a human eye,the stripes are grouped as a coherent object–the zebra.As discusd above,this intuitive grouping shows up in the image statistics:black and white pixels commonly co-occur next to one another, while white-green combinations are rarer,suggesting a possible object boundary where a white stripe meets the green background.
In this ction,we describe a formal measure of the affinity between neigh-boring image features,bad on statistical association.We denote a generic pair
Crisp Luminance A 00.5100.2
0.4
0.6
4.654.74.75
Luminance A
00.51
00.05Luminance pairs chon from within each circle are plotted where they fall distribution and PMI functions.over multiple distances:P (A,B )=1Z 1X d =d 0w (d )p (A,B ;d ),(1)where w is a weighting function which decays monotonically with distance d,and Z is a normalization constant.We take the marginals of this distribution to get P (A )and P (B ):P (A )=Z B P (A,B ),(2)
介绍一本好书
小巴鱼and correspondingly for P(B).In order to pick out object boundaries,a first guess might be that a nity should be measured with joint probability P (A,B ).After all,features that al-ways occur together probably should be grouped together.For the zebra image in Figure 2,the joint distribution over luminance values of nearby pixels is shown in the middle column.Overlaid on the zebra image are three ts of pixel pairs in the colored circles.The pairs correspond to pairs {A,B }in our model.The
pair of pixels in the blue circle are both on the same object and the joint proba-bility of their colors –green next to green –is high.The pair in the bright green circle straddles an object boundary and the joint probability of the colors of this pair –black next to green –is correspondingly low.Now consider the pair in the red circle.There is no physical object boundary on the edge of this zebra stripe.However,the joint probability is actually lower for this pair than for the pair in the green circle,where an object boundary did in fact exist.This demonstrates a shortcoming of using joint probability as L u m i n a n c e B distribution and PMI functions.over multiple distances:P (A,B )=1Z 1X d =d 0w (d )p (A,B ;d ),(1)where w is a weighting function which decays monotonically with distance d,and Z is a normalization constant.We take the marginals of this distribution to get P (A )and P (B ):P (A )=Z B P (A,B ),(2)and correspondingly for P(B).In order to pick out object boundaries,a first guess might be that a nity should be measured with joint probability P (A,B ).After all,features that al-ways occur together probably should be grouped together.For the zebra image in Figure 2,the joint distribution over luminance values of nearby pixels is shown in the middle column.Overlaid on the zebra image are three ts of pixel pairs in the colored circles.The pairs correspond to pairs {A,B }in our model.The pair of pixels in the blue circle are both on the same object and the joint proba-bility of their colors –green next to green –is high.The pair in the bright green circle straddles an object boundary and the joint probability of the colors of this pair –black nex
t to green –is correspondingly low.Now consider the pair in the red circle.There is no physical object boundary on the edge of this zebra stripe.However,the joint probability is actually lower for this pair than for the pair in the green circle,where an object boundary did in fact exist.This demonstrates a shortcoming of using joint probability as log P(A,B)Luminance pairs chon from within each circle are plotted where in the joint distribution and PMI functions.
of neighboring features by random variables A and B ,and investigate the joint distribution over pairings {A,B }.Let p (A,B ;d )be the joint probability of features A and B occurring at a Eu-clidean distance of d pixels apart.We define P (A,B )by computing probabilities over multiple distances:P (A,B )=1Z ∞ d =d 0w (d )p (A,B ;d ),(1)where w is a weighting function which decays monotonically with distance d (Gaussian in our implementation),and Z is a normalization constant.We take the marginals of this distribution to get P (A )and P (B ).In order to pick out object boundaries,a first guess might be that affinity should be measured with joint probability P (A,B ).After all,features that al-ways occur together probably should be grouped together.For the zebra image in Figure 2,the joint distribution over luminance values of nearby pixels is shown in the middle column.Overlaid on the zebra image are three ts of pixel pairs in the colored circles.The pairs correspond to pairs {A,B }in our model.The pair of pixels in the blue circle are both on the same object and the joint prob挂靠合同
a-bility of their colors –green next to green –is high.The pair in the bright green circle straddles an object boundary and the joint probability of the colors of this pair –black next to green –is correspondingly low.Now consider the pair in the red circle.There is no physical object boundary on the edge of this zebra stripe.However,the joint probability is actually lower for this pair than for the pair in the green circle,where an object boundary did in fact exist.This demonstrates a shortcoming of using joint probability as a measure of affinity.Becau there are simply more green pixels in the image
