Hey,You,Get Off of My Cloud:
Exploring Information Leakage in
Third-Party Compute Clouds
Thomas Ristenpart∗Eran Tromer†Hovav Shacham∗Stefan Savage∗
∗Dept.of Computer Science and Engineering†Computer Science and Artificial Intelligence Laboratory University of California,San Diego,USA Massachutts Institute of T echnology,Cambridge,USA
{tristenp,hovav,savage}@cs.ucsd.edu tromer@csail.mit.edu
ABSTRACT
Third-party cloud computing reprents the promi of out-sourcing as applied to computation.Services,such as Mi-crosoft’s Azure and Amazon’s EC2,allow urs to instanti-ate virtual machines(VMs)on demand and thus purcha precily the capacity they require when they require it. In turn,the u of virtualization allows third-party cloud providers to maximize the utilization of their sunk capital costs by multiplexing many customer VMs across a shared physical infrastructure.Howeve
r,in this paper,we show that this approach can also introduce new vulnerabilities. Using the Amazon EC2rvice as a ca study,we show that it is possible to map the internal cloud infrastructure,iden-tify where a particular target VM is likely to reside,and then instantiate new VMs until one is placed co-resident with the target.We explore how such placement can then be ud to mount cross-VM side-channel attacks to extract information from a target VM on the same machine.
Categories and Subject Descriptors
不变的承诺
K.6.5[Security and Protection]:UNAUTHORIZED AC-CESS
General Terms
Security,Measurement,Experimentation
Keywords
Cloud computing,Virtual machine curity,Side channels 1.INTRODUCTION
It has become increasingly popular to talk of“cloud com-puting”as the next infrastructure for hosting data and de-ploying software and rvices.In addition to the plethora of technical approaches associ
ated with the term,cloud com-puting is also ud to refer to a new business model in which Permission to make digital or hard copies of all or part of this work for personal or classroom u is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on thefirst page.To copy otherwi,to republish,to post on rvers or to redistribute to lists,requires prior specific permission and/or a fee.
CCS’09,November9–13,2009,Chicago,Illinois,USA.
交安设施Copyright2009ACM978-1-60558-352-5/$ computing and software capabilities are outsourced on demand to shared third-party infrastructure.While this model,exemplified by Amazon’s Elastic Compute Cloud (EC2)[5],Microsoft’s Azure Service Platform[20],and Rack-space’s Mosso[27]provides a number of advantages—in-cluding economies of scale,dynamic provisioning,and low capital expenditures—it also introduces a range of new risks. Some of the risks are lf-evident and relate to the new trust relationship between customer and cloud provider.For example,customers must trust their cloud providers to re-spect the privacy of their data and the integrity of their computations.However,cloud infrastructures can also in-troduce non-obvious threats from other customers due to the subtleties of how physical resources can be transparently shared between virtual machines(VMs).
In particular,to maximize efficiency multiple VMs may be simultaneously assigned to execute on the same physi-cal rver.Moreover,many cloud providers allow“multi-tenancy”—multiplexing the virtual machines of disjoint customers upon the same physical hardware.Thus it is con-ceivable that a customer’s VM could be assigned to the same physical rver as their adversary.This in turn,engenders a new threat—that the adversary might penetrate the iso-lation between ,via a vulnerability that allows an“escape”to the hypervisor or via side-channels between VMs)and violate customer confidentiality.This paper ex-plores the practicality of mounting such cross-VM attacks in existing third-party compute clouds.
The attacks we consider require two main steps:place-ment and extraction.Placement refers to the adversary ar-ranging to place their malicious VM on the same physical machine as that of a target customer.Using Amazon’s EC2 as a ca study,we demonstrate that careful empirical“map-ping”can reveal how to launch VMs in a way that maximizes the likelihood of an advantageous placement.Wefind that in some natural attack scenarios,just a few dollars invested in launching VMs can produce a40%chance of placing a malicious VM on the same physical rver as a target cus-tomer.Using the same platform we also demonstrate the existence of simple,low-overhead,“co-residence”checks to determine when such an advantageous placement has taken place.While we fo
cus on EC2,we believe that variants of our techniques are likely to generalize to other rvices, such as Microsoft’s Azure[20]or Rackspace’s Mosso[27],as we only utilize standard customer capabilities and do not require that cloud providers disclo details of their infras-tructure or assignment policies.
Having managed to place a VM co-resident with the tar-get,the next step is to extract confidential information via a cross-VM attack.While there are a number of avenues for such an attack,in this paper we focus on side-channels: cross-VM information leakage due to the sharing of physical ,the CPU’s data caches).In the multi-process environment,such attacks have been shown to enable ex-traction of RSA[26]and AES[22]cret keys.However,we are unaware of published extensions of the attacks to the virtual machine environment;indeed,there are significant practical challenges in doing so.
We show preliminary results on cross-VM side channel at-tacks,including a range of building ,cache load measurements in EC2)and coar-grained attacks such as measuring activity burst ,for cross-VM keystroke monitoring).The point to the practicality of side-channel attacks in cloud-computing environments.
Overall,our results indicate that there exist tangible dan-gers when deploying nsitive tasks to third-
party compute clouds.In the remainder of this paper,we explain the findings in more detail and then discuss means to mitigate the problem.We argue that the best solution is for cloud providers to expo this risk explicitly and give some place-ment control directly to customers.
2.THREAT MODEL
As more and more applications become exported to third-party compute clouds,it becomes increasingly important to quantify any threats to confidentiality that exist in this t-ting.For example,cloud computing rvices are already ud for e-commerce applications,medical record rvices[7, 11],and back-office business applications[29],all of which require strong confidentiality guarantees.An obvious threat to the consumers of cloud computing is malicious behav-ior by the cloud provider,who is certainly in a position to violate customer confidentiality or integrity.However,this is a known risk with obvious analogs in virtually any in-dustry practicing outsourcing.In this work,we consider the provider and its infrastructure to be trusted.This also means we do not consider attacks that rely upon subverting a cloud’s administrative functions,via insider abu or vul-nerabilities in the cloud management ,virtual machine monitors).
In our threat model,adversaries are non-provider-affiliated malicious parties.Victims are urs runnin
法学名词解释g confidentiality-requiring rvices in the cloud.A traditional threat in such a tting is direct compromi,where an attacker attempts re-mote exploitation of vulnerabilities in the software running on the system.Of cour,this threat exists for cloud appli-cations as well.The kinds of attacks(while important)are a known threat and the risks they prent are understood. We instead focus on where third-party cloud computing gives attackers novel abilities;implicitly expanding the at-tack surface of the victim.We assume that,like any cus-tomer,a malicious party can run and control many instances in the cloud,simply by contracting for them.Further,since the economies offered by third-party compute clouds derive from multiplexing physical infrastructure,we assume(and later validate)that an attacker’s instances might even run on the same physical hardware as potential victims.From this vantage,an attacker might manipulate shared physical ,CPU caches,branch target buffers,network queues,etc.)to learn otherwi confidential information.
In this tting,we consider two kinds of attackers:tho who cast a wide net and are interested in being able to attack some known hosted rvice and tho focud on attacking a particular victim rvice.The latter’s task is more expensive and time-consuming than the former’s,but both rely on the same fundamental attack.
In this work,we initiate a rigorous rearch program aimed at exploring the risk of such attacks,using
a concrete cloud rvice provider(Amazon EC2)as a ca study.We address the concrete questions in subquent ctions:
•Can one determine where in the cloud infrastructure an instance is located?(Section5)
•Can one easily determine if two instances are co-resident on the same physical machine?(Section6)
•Can an adversary launch instances that will be co-resident with other ur’s instances?(Section7)
•Can an adversary exploit cross-VM information leakage once co-resident?(Section8)
Throughout we offer discussions of defens a cloud provider might try in order to prevent the success of the various at-tack steps.
3.THE EC2SERVICE
By far the best known example of a third-party compute cloud is Amazon’s Elastic Compute Cloud(EC2)rvice, which enables urs toflexibly rent computational resources for u by their applications[5].EC2provides the ability to run Linux,FreeBSD,OpenSolaris and Windows as guest op
erating systems within a virtual machine(VM)provided by a version of the Xen hypervisor[9].1The hypervisor plays the role of a virtual machine monitor and provides isolation between VMs,intermediating access to physical memory and devices.A privileged virtual machine,called Domain0(Dom0)in the Xen vernacular,is ud to manage guest images,their physical resource provisioning,and any access control rights.In EC2the Dom0VM is configured to route packets for its guest images and reports itlf as a hop in traceroutes.
Whenfirst registering with EC2,each ur creates an ac-count—uniquely specified by its contact e-mail address—and provides credit card information for billing compute and I/O charges.With a valid account,a ur creates one or more VM images,bad on a supplied Xen-compatible ker-nel,but with an otherwi arbitrary configuration.He can run one or more copies of the images on Amazon’s network of machines.One such running image is called an instance, and when the instance is launched,it is assigned to a single physical machine within the EC2network for its lifetime; EC2does not appear to currently support live migration of instances,although this should be technically feasible.By default,each ur account is limited to20concurrently run-ning instances.
In addition,there are three degrees of freedom in specify-ing the physical infrastructure upon which instances should run.At the time of this writing,Amazon provides two “regions”,one located in the Uni
ted States and the more recently established one in Europe.Each region contains three“availability zones”which are meant to specify in-frastructures with distinct and independent failure modes 1We will limit our subquent discussion to the Linux ker-nel.The same issues should apply for other guest operating systems.
(e.g.,with parate power and network connectivity).When requesting launch of an instance,a ur specifies the re-gion and may choo a specific availability zone(otherwi one is assigned on the ur’s behalf).As well,the ur can specify an“instance type”,indicating a particular com-bination of computational power,memory and persistent storage space available to the virtual machine.There are five Linux instance types documented at prent,referred to as‘m1.small’,‘c1.medium’,‘m1.large’,‘m1.xlarge’,and ‘c1.xlarge’.Thefirst two are32-bit architectures,the latter three are64-bit.To give some n of relative scale,the “small compute slot”(m1.small)is described as a single vir-tual core providing one ECU(EC2Compute Unit,claimed to be equivalent to a1.0–1.2GHz2007Opteron or2007Xeon processor)combined with1.7GB of memory and160GB of local storage,while the“large compute slot”(m1.large) provides2virtual cores each with2ECUs,7.5GB of mem-ory and850GB of local storage.As expected,instances with more resources incur greater hourly ,‘m1.small’in the United States region is currently$0.10pe扦插生根最快方法
r hour,while ‘m1.large’is currently$0.40per hour).When launching an instance,the ur specifies the instance type along with a compatible virtual machine image.
Given the constraints,virtual machines are placed on available physical rvers shared among multiple instances. Each instance is given Internet connectivity via both an external IPv4address and domain name and an internal RFC1918private address and domain name.For example, an instance might be assigned external IP75.101.210.100, external pute-1.amazonaws .com,internal IP10.252.146.52,and internal pute-1.internal.Within the cloud, both domain names resolve to the internal IP address;out-side the cloud the external name is mapped to the external IP address.
Note that we focus on the United States region—in the rest of the paper EC2implicitly means this region of EC2.
4WORK PROBING
In the next veral ctions,we describe an empirical mea-surement study focud on understanding VM placement in the EC2system and achieving co-resident placement for an adversary.To do this,we make u of network probing both to identify public rvices hosted on EC2
and to pro-vide evidence of co-residence(that two instances share the same physical rver).In particular,we utilize nmap,hping, and wget to perform network probes to determine liveness of EC2instances.We u nmap to perform TCP connect probes,which attempt to complete a3-way hand-shake be-tween a source and target.We u hping to perform TCP SYN traceroutes,which iteratively nds TCP SYN pack-ets with increasing time-to-lives(TTLs)until no ACK is received.Both TCP connect probes and SYN traceroutes require a target port;we only targeted ports80or443.We ud wget to retrieve web pages,but capped so that at most 1024bytes are retrieved from any individual web rver.
圣母玛利亚We distinguish between two types of probes:external probes and internal probes.A probe is external when it originates from a system outside EC2and has destination an EC2instance.A probe is internal if it originates from an EC2instance(under our control)and has destination another EC2instance.This dichotomy is of relevance par-ticularly becau internal probing is subject to Amazon’s acceptable u policy,whereas external probing is not(we discuss the legal,ethical and contractual issues around such probing in Appendix A).
We u DNS resolution queries to determine the external name of an instance and also to determine the internal IP address of an instance associated with some public IP ad-dress.The latter queries are
always performed from an EC2 instance.
5.CLOUD CARTOGRAPHY
In this ction we‘map’the EC2rvice to understand where potential targets are located in the cloud and the instance creation parameters needed to attempt establish-ing co-residence of an adversarial instance.This will speed up significantly adversarial strategies for placing a malicious VM on the same machine as a target.In the next ction we will treat the task of confirming when successful co-residence is achieved.
To map EC2,we begin with the hypothesis that different availability zones are likely to correspond to different inter-nal IP address ranges and the same may be true for instance types as well.Thus,mapping the u of the EC2internal address space allows an adversary to determine which IP ad-dress correspond to which creation parameters.Moreover, since EC2’s DNS rvice provides a means to map public IP address to private IP address,an adversary might u such a map to infer the instance type and availability zone of a tar-get rvice—thereby dramatically reducing the number of instances needed before a co-resident placement is achieved. We evaluate this theory using two data ts:one created by enumerating public EC2-bad web rvers using ext
ernal probes and translating responsive public IPs to internal IPs (via DNS queries within the cloud),and another created by launching a number of EC2instances of varying types and surveying the resulting IP address assigned.
To fully leverage the latter data,we prent a heuristic algorithm that helps label/24prefixes with an estimate of the availability zone and instance type of the included Inter-nal IPs.The heuristics utilize veral beneficial features of EC2’s addressing regime.The output of this process is a map of the internal EC2address space which allows one to estimate the availability zone and instance type of any tar-get public EC2rver.Next,we enumerate a t of public EC2-bad Web rvers
Surveying public rvers on EC2.Utilizing WHOIS queries,we identified four distinct IP address prefixes,a/16, /17,/18,and/19,as being associated with EC2.The last three contained public IPs obrved as assigned to EC2in-stances.We had not yet obrved EC2instances with public IPs in the/16,and therefore did not include it in our sur-vey.For the remaining IP address(57344IP address), we performed a TCP connect probe on port80.This re-sulted in11315responsive IPs.Of the9558responded (with some HTTP respon)to a follow-up wget on port 80.We also performed a TCP port443scan of all57344IP address,which resulted in8375responsive IPs.Via an ap-propriate DNS lookup from within EC2,we translated each public IP address that responded to either
the port80or port443scan into an internal EC2address.This resulted in a list of14054unique internal IPs.One of the goals of this ction is to enable identification of the instance type and availability zone of one or more of the potential targets.
326410.249.0.0
10.250.0.0
10.251.0.010.252.0.010.253.0.010.254.0.010.255.0.0
I P a d d r e s s m o d 64
Internal IP
不绝如缕
address
32640326410.252.0.0
10.253.0.010.254.0.0
A c c o u n t
B A c c o u n t A
Internal IP
address
Figure 1:(Top)A plot of the internal IP address assigned to instances launched during the initial mapping exper-iment using Account A.(Bottom)A plot of the internal IP address of instances launched in Zone 3by Account A and,39hours later,by Account B.Fifty-five of the Account B IPs were repeats of tho assigned to instances for Account A.
老月饼Instance placement parameters.Recall that there are three availability zones and five instance types
in the prent EC2system.While the parameters could be assigned in-dependently from the underlying infrastructure,in practice this is not so.In particular,the Amazon EC2internal IP ad-dress space is cleanly partitioned between availability zones (likely to make it easy to manage parate network con-nectivity for the zones)and instance types within the zones also show considerable regularity.Moreover,different accounts exhibit similar placement.
To establish the facts,we iteratively launched 20in-stances for each of the 15availability zone/instance type pairs.We ud a single account,call it “Account A”.The top graph in Figure 1depicts a plot of the internal IP address assigned to each of the 300instances,partitioned according to availability zone.It can be readily en that the sam-ples from each zone are assigned IP address from disjoint portions of the obrved internal address space.For ex-ample,samples from Zone 3were assigned address within 10.252.0.0/16and from discrete prefixes within 10.253.0.0/16.If we make the assumption that internal IP address are statically assigned to physical machines (doing otherwi would make IP routing far more difficult to implement),this data supports the asssment that availability zones u p-arate physical infrastructure.Indeed,none of the data gath-ered in the rest of the paper’s described experiments have cast doubt on this conclusion.
While it is perhaps not surprising that availability zones enjoy disjoint IP assignment,what about instance type and accounts?We launched 100instances (20of each type,39hours after terminating the Account A instances)in Zone 3from a cond account,“Account B”.The bottom graph in Figure 1plots the Zone 3instances from Account A and Account B,here using distinct labels for instance type.Of the 100Account A Zone 3instances,92had unique /24prefixes,while eight /24prefixes each had two instances,though of the same type.Of the 100Account B instances,
88had unique /24prefixes,while six of the /24prefixes had two instances each.A single /24had both an m1.large and an m1.xlarge instance.No IP address were ever obrved being assigned to more than one instance type.Of the 100Acount B IP’s,55were repeats of IP address assigned to instances for Acount A.
A fuller map of EC2.We would like to infer the instance type and availability zone of any public EC2instance,but our sampling data is relatively spar.We could sample more (and did),but to take full advantage of the sampling data at hand we should take advantage of the significant regularity of the EC2addressing regime.For example,the above data suggests that /24prefixes rarely have IPs as-signed to distinct instance types.We utilized data from 4499instances launched under veral accounts under our control;the instances were also ud in many of the exper-iments described in
the rest of the paper.The included 977unique internal IPs and 611unique Dom0IPs associated with the instances.
Using manual inspection of the resultant data,we derived a t of heuristics to label /24prefixes with both availability zone and instance type:
•All IPs from a /16are from the same availability zone.•A /24inherits any included sampled instance type.If there are multiple instances with distinct types,then we label the /24with each distinct type (i.e.,it is ambiguous).•A /24containing a Dom0IP address only contains Dom0IP address.We associate to this /24the type of the Dom0’s associated instance.
•All /24’s between two concutive Dom0/24’s inherit the former’s associated type.
The last heuristic,which enables us to label /24’s that have no included instance,is derived from the obrvation that Dom0IPs are consistently assigned a prefix that immedi-ately precedes the instance IPs they are associated with.(For example,10.250.8.0/24contained Dom0IPs associated
with m1.small instances in prefixes10.254.9.0/24and 10.254.10.0/24.)There were869/24’s in the data,and ap-plying the heuristics resulted in assigning a unique zone and unique type to723of the;
a unique zone and two types to 23of the;and left123unlabeled.The last were due to areas(such as the lower portion of10.253.0.0/16)for which we had no sampling data at all.
While the map might contain errors(for example,in areas of low instance sample numbers),we have yet to encounter an instance that contradicts the/24labeling and we ud the map for many of the future experiments.For instance, we applied it to a subt of the public rvers derived from our survey,tho that responded to wget requests with an HTTP200or206.The resulting6057rvers were ud as stand-ins for targets in some of the experiments in Section7. Figure7in the appendix graphs the result of mapping the rvers.
Preventing cloud cartography.Providers likely have in-centive to prevent cloud cartography for veral reasons,be-yond the u we outline here(that of exploiting placement vulnerabilities).Namely,they might wish to hide their in-frastructure and the amount of u it is enjoying by cus-tomers.Several features of EC2made cartography signif-icantly easier.Paramount is that local IP address are statically(at least over the obrved period of time)as-sociated to availability zone and instance type.Changing this would likely make administration tasks more challeng-ing(and costly)for providers.Also,using the map requires translating a victim instance’s external IP to an internal IP,and the provider might inhibit this by isolating each account’s view of the
internal IP address via VLANs and bridging).Even so,this would only appear to slow down our particular technique for locating an instance in the LAN—one might instead u ping timing measure-ments or traceroutes(both discuss more in the next ction) to help“triangulate”on a victim.
6.DETERMINING CO-RESIDENCE
Given a t of targets,the EC2map from the previous ction educates choice of instance launch parameters for attempting to achieve placement on the same physical ma-chine.Recall that we refer to instances that are running on the same physical machine as being co-resident.In this ction we describe veral easy-to-implement co-residence checks.Looking ahead,our eventual check of choice will be to compare instances’Dom0IP address.We confirm the accuracy of this(and other)co-residence checks by exploit-ing a hard-disk-bad covert channel between EC2instances. Network-bad co-residence checks.Using our expe-rience running instances while mapping EC2and inspect-ing data collected about them,we identify veral poten-tial methods for checking if two instances are co-resident. Namely,instances are likely co-resident if they have
(1)matching Dom0IP address,
(2)small packet round-trip times,or
(3)numerically clo internal IP within7). As mentioned,an instance’s network traffic’sfirst hop is the Dom0privileged VM.An instance owner can determine its Dom0IP from thefirst hop on any route out from the in-stance.One can determine an uncontrolled instance’s Dom0 IP by performing a TCP SYN traceroute to it(on some open port)from another instance and inspecting the last hop.For the cond test,we noticed that round-trip times(RTTs)re-quired a“warm-up”:thefirst reported RTT in any quence of probes was almost always an order of magnitude slower than subquent probes.Thus for this method we perform 10probes and just discard thefirst.The third check makes u of the manner in which internal IP address appear to be assigned by EC2.The same Dom0IP will be shared by in-stances with a contiguous quence of internal IP address. (Note that m1.small instances are reported by CPUID as having two CPUs each with two cores and the EC2in-stance types are limited to50%core usage,implying that one such machine could handle eight instances.)
Veracity of the co-residence checks.We verify the cor-rectness of our network-bad co-residence checks using as ground truth the ability to nd messages over a cross-VM covert channel.That is,if two instances(under our control) can successfully transmit via the covert channel then they are co-resident,otherwi not.If the checks above(which do not require both instances to be under our contr
ol)have suf-ficiently low fal positive rates relative to this check,then we can u them for inferring co-residence against arbitrary victims.We utilized for this experiment a hard-disk-bad covert channel.At a very high level,the channel works as follows.To nd a one bit,the nder instance reads from random locations on a shared disk volume.To nd a zero bit,the nder does nothing.The receiver times reading from afixed location on the disk volume.Longer read times mean a1is being t,shorter read times give a0.
We performed the following experiment.Three EC2ac-counts were utilized:a control,a victim,and a probe.(The “victim”and“probe”are arbitrary labels,since they were both under our control.)All instances launched were of type m1.small.Two instances were launched by the control account in each of the three availability zones.Then20in-stances on the victim account and20instances on the probe account were launched,all in Zone3.We determined the Dom0IPs of each instance.For each(ordered)pair(A,B) of the40instances,if the Dom0IPs pasd(check1)then we had A probe B and each control to determine packet RTTs and we also nt a5-bit message from A to B over the hard-drive covert channel.
We performed three independent trials.The generated, in total,31pairs of instances for which the Dom0IPs were equal.The internal IP address of each pair were within7 of each other.Of the31(pot
entially)co-resident instance pairs,12were‘repeats’(a pair from a later round had the same Dom0as a pair from an earlier round).
The31pairs give62ordered pairs.The hard-drive covert channel successfully nt a5-bit message for60of the pairs.The last two failed due to a single bit error each,and we point out that the two failures were not for the same pair of ding a message in the rever direc-tion succeeded).The results of the RTT probes are shown in Figure2.The median RTT for co-resident instances was significantly smaller than tho to any of the controls.The RTTs to the controls in the same availability zone as the probe(Zone3)and victim instances were also noticeably smaller than tho to other zones.
Discussion.From this experiment we conclude an effec-tive fal positive rate of zero for the Dom0IP co-residence check.In the rest of the paper we will therefore utilize the公休假是什么意思
