p450-Lo Heracles Improving Resource Efficiency at Scale
Heracles:Improving Resource Ef?ciency at Scale
David Lo?,Liqun Cheng?,Rama Govindaraju?,Parthasarathy Ranganathan?and Christos Kozyrakis?Stanford University?Google,Inc.?
Abstract
User-facing,latency-sensitive services,such as websearch, underutilize their computing resources during daily periods of low traf?c.Reusing those resources for other tasks is rarely done in production services since the contention for shared resources can cause latency spikes that violate the service-level objectives of latency-sensitive tasks.The resulting under-utilization hurts both the affordability and energy-ef?ciency of large-scale data-centers.With technology scaling slowing down,it becomes im-portant to address this opportunity.
We present Heracles,a feedback-based controller that en-ables the safe colocation of best-effort tasks alongside a latency-critical service.Heracles dynamically manages multiple hard-ware and software isolation mechanisms,such as CPU,memory, and network isolation,to ensure that the latency-sensitive job meets latency targets while maximizing the resources given to best-effort tasks.We evaluate Heracles using production latency-critical and batch workloads from Google and demonstrate aver-age server utilizations of90%without latency violations across all the load and colocation scenarios that we evaluated.
1Introduction
Public and private cloud frameworks allow us to host an in-creasing number of workloads in large-scale datacenters with tens of thousands of servers.The business models for cloud services emphasize reduced infrastructure costs.Of the total cost of ownership(TCO)for modern energy-ef?cient datacen-ters,servers are the largest fraction(50-70%)[7].Maximizing server utilization is therefore important for continued scaling.
Until recently,scaling from Moore’s law provided higher compute per dollar with every server generation,allowing dat-acenters to scale without raising the cost.However,with sev-eral imminent challenges in technology scaling[21,25],alter-nate approaches are needed.Some efforts seek to reduce the server cost through balanced designs or cost-effective compo-nents[31,48,42].An orthogonal approach is to improve the return on investment and utility of datacenters by raising server utilization.Low utilization negatively impacts both operational and capital components of cost ef?ciency.Energy proportion-ality can reduce operational expenses at low utilization[6,47]. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for pro?t or commercial advantage and that copies bear this notice and the full citation on the?rst page.Copyrights for components of this work owned by others than ACM must be honored.Abstracting with credit is permitted.To copy otherwise,or republish,to post on servers or to redistribute to lists,requires prior speci?c permission and/or a fee.Request permissions from Permissions@https://www.wendangku.net/doc/4110046684.html,.
ISCA’15,June13-17,2015,Portland,OR,USA
c 2015ACM.ISBN978-1-4503-3402-0/15/06$15.00
DOI:https://www.wendangku.net/doc/4110046684.html,/10.1145/2749469.2749475But,to amortize the much larger capital expenses,an increased emphasis on the effective use of server resources is warranted.
Several studies have established that the average server uti-lization in most datacenters is low,ranging between10%and 50%[14,74,66,7,19,13].A primary reason for the low uti-lization is the popularity of latency-critical(LC)services such as social media,search engines,software-as-a-service,online maps, webmail,machine translation,online shopping and advertising. These user-facing services are typically scaled across thousands of servers and access distributed state stored in memory or Flash across these servers.While their load varies signi?cantly due to diurnal patterns and unpredictable spikes in user accesses,it is dif?cult to consolidate load on a subset of highly utilized servers because the application state does not?t in a small number of servers and moving state is expensive.The cost of such under-utilization can be signi?cant.For instance,Google websearch servers often have an average idleness of30%over a24hour period[47].For a hypothetical cluster of10,000servers,this idleness translates to a wasted capacity of3,000servers.
A promising way to improve ef?ciency is to launch best-effort batch(BE)tasks on the same servers and exploit any re-sources underutilized by LC workloads[52,51,18].Batch an-alytics frameworks can generate numerous BE tasks and derive signi?cant value even if these tasks are occasionally deferred or restarted[19,10,13,16].The main challenge of this approach is interference between colocated workloads on shared resources such as caches,memory,I/O channels,and network links.LC tasks operate with strict service level objectives(SLOs)on tail latency,and even small amounts of interference can cause sig-ni?cant SLO violations[51,54,39].Hence,some of the past work on workload colocation focused only on throughput work-loads[58,15].More recent systems predict or detect when a LC task suffers signi?cant interference from the colocated tasks,and avoid or terminate the colocation[75,60,19,50,51,81].These systems protect LC workloads,but reduce the opportunities for higher utilization through colocation.
Recently introduced hardware features for cache isolation and ?ne-grained power control allow us to improve colocation.This work aims to enable aggressive colocation of LC workloads and BE jobs by automatically coordinating multiple hardware and software isolation mechanisms in modern servers.We focus on two hardware mechanisms,shared cache partitioning and?ne-grained power/frequency settings,and two software mechanisms, core/thread scheduling and network traf?c control.Our goal is to eliminate SLO violations at all levels of load for the LC job while maximizing the throughput for BE tasks.
There are several challenges towards this goal.First,we must carefully share each individual resource;conservative allocation will minimize the throughput for BE tasks,while optimistic al-location will lead to SLO violations for the LC tasks.Second, the performance of both types of tasks depends on multiple re-sources,which leads to a large allocation space that must be
explored in real-time as load changes.Finally,there are non-obvious interactions between isolated and non-isolated resources in modern servers.For instance,increasing the cache allocation for a LC task to avoid evictions of hot data may create memory bandwidth interference due to the increased misses for BE tasks.
We present Heracles1,a real-time,dynamic controller that manages four hardware and software isolation mechanisms in a coordinated fashion to maintain the SLO for a LC https://www.wendangku.net/doc/4110046684.html,pared to existing systems[80,51,19]that prevent colocation of inter-fering workloads,Heracles enables a LC task to be colocated with any BE job.It guarantees that the LC workload receives just enough of each shared resource to meet its SLO,thereby maximizing the utility from the BE https://www.wendangku.net/doc/4110046684.html,ing online monitor-ing and some of?ine pro?ling information for LC jobs,Heracles identi?es when shared resources become saturated and are likely to cause SLO violations and con?gures the appropriate isolation mechanism to proactively prevent that from happening.
The speci?c contributions of this work are the following. First,we characterize the impact of interference on shared re-sources for a set of production,latency-critical workloads at Google,including websearch,an online machine learning clus-tering algorithm,and an in-memory key-value store.We show that the impact of interference is non-uniform and workload de-pendent,thus precluding the possibility of static resource parti-tioning within a server.Next,we design Heracles and show that: a)coordinated management of multiple isolation mechanisms is key to achieving high utilization without SLO violations;b)care-fully separating interference into independent subproblems is ef-fective at reducing the complexity of the dynamic control prob-lem;and c)a local,real-time controller that monitors latency in each server is suf?cient.We evaluate Heracles on production Google servers by using it to colocate production LC and BE tasks.We show that Heracles achieves an effective machine uti-lization of90%averaged across all colocation combinations and loads for the LC tasks while meeting the latency SLOs.Heracles also improves throughput/TCO by15%to300%,depending on the initial average utilization of the datacenter.Finally,we es-tablish the need for hardware mechanisms to monitor and isolate DRAM bandwidth,which can improve Heracles’accuracy and eliminate the need for of?ine information.
To the best of our knowledge,this is the?rst study to make coordinated use of new and existing isolation mechanisms in a real-time controller to demonstrate signi?cant improvements in ef?ciency for production systems running LC services.
2Shared Resource Interference
When two or more workloads execute concurrently on a server,they compete for shared resources.This section reviews the major sources of interference,the available isolation mecha-nisms,and the motivation for dynamic management.
The primary shared resource in the server are the cores in the one or more CPU sockets.We cannot simply statically partition cores between the LC and BE tasks using mechanisms such as cgroups cpuset[55].When user-facing services such as search face a load spike,they need all available cores to meet throughput demands without latency SLO violations.Similarly, we cannot simply assign high priority to LC tasks and rely on 1The mythical hero that killed the multi-headed monster,Lernaean Hydra.OS-level scheduling of cores between https://www.wendangku.net/doc/4110046684.html,mon schedul-ing algorithms such as Linux’s completely fair scheduler(CFS) have vulnerabilities that lead to frequent SLO violations when LC tasks are colocated with BE tasks[39].Real-time scheduling algorithms(e.g.,SCHED_FIFO)are not work-preserving and lead to lower utilization.The availability of HyperThreads in Intel cores leads to further complications,as a HyperThread exe-cuting a BE task can interfere with a LC HyperThread on instruc-tion bandwidth,shared L1/L2caches,and TLBs.
Numerous studies have shown that uncontrolled interference on the shared last-level cache(LLC)can be detrimental for colo-cated tasks[68,50,19,22,39].To address this issue,Intel has recently introduced LLC cache partitioning in server chips.This functionality is called Cache Allocation Technology(CAT),and it enables way-partitioning of a highly-associative LLC into sev-eral subsets of smaller associativity[3].Cores assigned to one subset can only allocate cache lines in their subset on re?lls,but are allowed to hit in any part of the LLC.It is already well under-stood that,even when the colocation is between throughput tasks, it is best to dynamically manage cache partitioning using either hardware[30,64,15]or software[58,43]techniques.In the pres-ence of user-facing workloads,dynamic management is more critical as interference translates to large latency spikes[39].It is also more challenging as the cache footprint of user-facing work-loads changes with load[36].
Most important LC services operate on large datasets that do not?t in on-chip caches.Hence,they put pressure on DRAM bandwidth at high loads and are sensitive to DRAM bandwidth interference.Despite signi?cant research on memory bandwidth isolation[30,56,32,59],there are no hardware isolation mech-anisms in commercially available chips.In multi-socket servers, one can isolate workloads across NUMA channels[9,73],but this approach constrains DRAM capacity allocation and address interleaving.The lack of hardware support for memory band-width isolation complicates and constrains the ef?ciency of any system that dynamically manages workload colocation.
Datacenter workloads are scale-out applications that generate network traf?c.Many datacenters use rich topologies with suf-?cient bisection bandwidth to avoid routing congestion in the fabric[28,4].There are also several networking protocols that prioritize short messages for LC tasks over large messages for BE tasks[5,76].Within a server,interference can occur both in the incoming and outgoing direction of the network link.If a BE task causes incast interference,we can throttle its core alloca-tion until networking?ow-control mechanisms trigger[62].In the outgoing direction,we can use traf?c control mechanisms in operating systems like Linux to provide bandwidth guarantees to LC tasks and to prioritize their messages ahead of those from BE tasks[12].Traf?c control must be managed dynamically as band-width requirements vary with load.Static priorities can cause un-derutilization and starvation[61].Similar traf?c control can be applied to solid-state storage devices[69].
Power is an additional source of interference between colo-cated tasks.All modern multi-core chips have some form of dynamic overclocking,such as Turbo Boost in Intel chips and Turbo Core in AMD chips.These techniques opportunistically raise the operating frequency of the processor chip higher than the nominal frequency in the presence of power headroom.Thus, the clock frequency for the cores used by a LC task depends not
just on its own load,but also on the intensity of any BE task running on the same socket.In other words,the performance of LC tasks can suffer from unexpected drops in frequency due to colocated tasks.This interference can be mitigated with per-core dynamic voltage frequency scaling,as cores running BE tasks can have their frequency decreased to ensure that the LC jobs maintain a guaranteed frequency.A static policy would run all BE jobs at minimum frequency,thus ensuring that the LC tasks are not power-limited.However,this approach severely penal-izes the vast majority of BE tasks.Most BE jobs do not have the pro?le of a power virus2and LC tasks only need the additional frequency boost during periods of high load.Thus,a dynamic solution that adjusts the allocation of power between cores is needed to ensure that LC cores run at a guaranteed minimum fre-quency while maximizing the frequency of cores for BE tasks.
A major challenge with colocation is cross-resource inter-actions.A BE task can cause interference in all the shared re-sources discussed.Similarly,many LC tasks are sensitive to in-terference on multiple resources.Therefore,it is not suf?cient to manage one source of interference:all potential sources need to be monitored and carefully isolated if need be.In addition,inter-ference sources interact with each other.For example,LLC con-tention causes both types of tasks to require more DRAM band-width,also creating a DRAM bandwidth bottleneck.Similarly,a task that notices network congestion may attempt to use compres-sion,causing core and power contention.In theory,the number of possible interactions scales with the square of the number of interference sources,making this a very dif?cult problem.
3Interference Characterization&Analysis This section characterizes the impact of interference on shared resources for latency-critical services.
3.1Latency-critical Workloads
We use three Google production latency-critical workloads. websearch is the query serving portion of a production web search service.It is a scale-out workload that provides high throughput with a strict latency SLO by using a large fan-out to thousands of leaf nodes that process each query on their shard of the search index.The SLO for leaf nodes is in the tens of milliseconds for the99%-ile latency.Load for websearch is gen-erated using an anonymized trace of real user queries.
websearch has high memory footprint as it serves shards of the search index stored in DRAM.It also has moderate DRAM bandwidth requirements(40%of available bandwidth at100% load),as most index accesses miss in the LLC.However,there is a small but signi?cant working set of instructions and data in the hot path.Also,websearch is fairly compute intensive,as it needs to score and sort search hits.However,it does not consume a signi?cant amount of network bandwidth.For this study,we reserve a small fraction of DRAM on search servers to enable colocation of BE workloads with websearch.
ml_cluster is a standalone service that performs real-time text clustering using machine-learning techniques.Several Google services use ml_cluster to assign a cluster to a snippet of text. ml_cluster performs this task by locating the closest clusters for the text in a model that was previously learned of?ine.This 2A computation that maximizes activity and power consumption of a core.model is kept in main memory for performance reasons.The SLO for ml_cluster is a95%-ile latency guarantee of tens of mil-liseconds.ml_cluster is exercised using an anonymized trace of requests captured from production services.
Compared to websearch,ml_cluster is more memory band-width intensive(with60%DRAM bandwidth usage at peak)but slightly less compute intensive(lower CPU power usage over-all).It has low network bandwidth requirements.An interesting property of ml_cluster is that each request has a very small cache footprint,but,in the presence of many outstanding requests,this translates into a large amount of cache pressure that spills over to DRAM.This is re?ected in our analysis as a super-linear growth in DRAM bandwidth use for ml_cluster versus load. memkeyval is an in-memory key-value store,similar to mem-cached[2].memkeyval is used as a caching service in the back-ends of several Google web services.Other large-scale web services,such as Facebook and Twitter,use memcached exten-sively.memkeyval has signi?cantly less processing per request compared to websearch,leading to extremely high throughput in the order of hundreds of thousands of requests per second at peak.Since each request is processed quickly,the SLO latency is very low,in the few hundreds of microseconds for the99%-ile latency.Load generation for memkeyval uses an anonymized trace of requests captured from production services.
At peak load,memkeyval is network bandwidth limited.De-spite the small amount of network protocol processing done per request,the high request rate makes memkeyval compute-bound.In contrast,DRAM bandwidth requirements are low (20%DRAM bandwidth utilization at max load),as requests sim-ply retrieve values from DRAM and put the response on the wire. memkeyval has both a static working set in the LLC for instruc-tions,as well as a per-request data working set.
3.2Characterization Methodology
To understand their sensitivity to interference on shared re-sources,we ran each of the three LC workloads with a synthetic benchmark that stresses each resource in isolation.While these are single node experiments,there can still be signi?cant network traf?c as the load is generated remotely.We repeated the char-acterization at various load points for the LC jobs and recorded the impact of the colocation on tail latency.We used produc-tion Google servers with dual-socket Intel Xeons based on the Haswell architecture.Each CPU has a high core-count,with a nominal frequency of2.3GHz and2.5MB of LLC per core.The chips have hardware support for way-partitioning of the LLC.
We performed the following characterization experiments: Cores:As we discussed in§2,we cannot share a logical core(a single HyperThread)between a LC and a BE task because OS scheduling can introduce latency spikes in the order of tens of milliseconds[39].Hence,we focus on the potential of using sep-arate HyperThreads that run pinned on the same physical core. We characterize the impact of a colocated HyperThread that im-plements a tight spinloop on the LC task.This experiment cap-tures a lower bound of HyperThread interference.A more com-pute or memory intensive microbenchmark would antagonize the LC HyperThread for more core resources(e.g.,execution units) and space in the private caches(L1and L2).Hence,if this exper-iment shows high impact on tail latency,we can conclude that
core sharing through HyperThreads is not a practical option. LLC:The interference impact of LLC antagonists is measured by pinning the LC workload to enough cores to satisfy its SLO at the speci?c load and pinning a cache antagonist that streams through a large data array on the remaining cores of the socket. We use several array sizes that take up a quarter,half,and almost all of the LLC and denote these con?gurations as LLC small, medium,and big respectively.
DRAM bandwidth:The impact of DRAM bandwidth interfer-ence is characterized in a similar fashion to LLC interference, using a signi?cantly larger array for streaming.We use numactl to ensure that the DRAM antagonist and the LC task are placed on the same socket(s)and that all memory channels are stressed. Network traf?c:We use iperf,an open source TCP streaming benchmark[1],to saturate the network transmit(outgoing)band-width.All cores except for one are given to the LC workload. Since the LC workloads we consider serve request from multiple clients connecting to the service they provide,we generate inter-ference in the form of many low-bandwidth“mice”?https://www.wendangku.net/doc/4110046684.html,-work interference can also be generated using a few“elephant”?ows.However,such?ows can be effectively throttled by TCP congestion control[11],while the many“mice”?ows of the LC workload will not be impacted.
Power:To characterize the latency impact of a power antagonist, the same division of cores is used as in the cases of generating LLC and DRAM interference.Instead of running a memory ac-cess antagonist,a CPU power virus is used.The power virus is designed such that it stresses all the components of the core, leading to high power draw and lower CPU core frequencies. OS Isolation:For completeness,we evaluate the overall impact of running a BE task along with a LC workload using only the isolation mechanisms available in the https://www.wendangku.net/doc/4110046684.html,ly,we execute the two workloads in separate Linux containers and set the BE workload to be low priority.The scheduling policy is enforced by CFS using the shares parameter,where the BE task receives very few shares compared to the LC workload.No other isola-tion mechanisms are used in this case.The BE task is the Google brain workload[38,67],which we will describe further in§5.1.
3.3Interference Analysis
Figure1presents the impact of the interference microbench-marks on the tail latency of the three LC workloads.Each row in the table shows tail latency at a certain load for the LC workload when colocated with the corresponding microbenchmark.The interference impact is acceptable if and only if the tail latency is less than100%of the target SLO.We color-code red/yellow all cases where SLO latency is violated.
By observing the rows for brain,we immediately notice that current OS isolation mechanisms are inadequate for colocating LC tasks with BE tasks.Even at low loads,the BE task creates suf?cient pressure on shared resources to lead to SLO violations for all three workloads.A large contributor to this is that the OS allows both workloads to run on the same core and even the same HyperThread,further compounding the interference.Tail latency eventually goes above300%of SLO latency.Proposed interference-aware cluster managers,such as Paragon[18]and Bubble-Up[51],would disallow these colations.To enable ag-gressive task colocation,not only do we need to disallow differ-ent workloads on the same core or HyperThread,we also need to use stronger isolation mechanisms.
The sensitivity of LC tasks to interference on individual shared resources varies.For instance,memkeyval is quite sensi-tive to network interference,while websearch and ml_cluster are not affected at all.websearch is uniformly insensitive to small and medium amounts of LLC interference,while the same can-not be said for memkeyval or ml_cluster.Furthermore,the im-pact of interference changes depending on the load:ml_cluster can tolerate medium amounts of LLC interference at loads<50% but is heavily impacted at higher loads.These observations moti-vate the need for dynamic management of isolation mechanisms in order to adapt to differences across varying loads and differ-ent workloads.Any static policy would be either too conserva-tive(missing opportunities for colocation)or overly optimistic (leading to SLO violations).
We now discuss each LC workload separately,in order to un-derstand their particular resource requirements.
websearch:This workload has a small footprint and LLC(small) and LLC(med)interference do not impact its tail latency.Nev-ertheless,the impact is signi?cant with LLC(big)interference. The degradation is caused by two factors.First,the inclusive nature of the LLC in this particular chip means that high LLC interference leads to misses in the working set of instructions. Second,contention for the LLC causes signi?cant DRAM pres-sure as well.websearch is particularly sensitive to interference caused by DRAM bandwidth saturation.As the load of web-search increases,the impact of LLC and DRAM interference de-creases.At higher loads,websearch uses more cores while the interference generator is given fewer cores.Thus,websearch can defend its share of resources better.
websearch is moderately impacted by HyperThread interfer-ence until high loads.This indicates that the core has suf?cient instruction issue bandwidth for both the spinloop and the web-search until around80%load.Since the spinloop only accesses registers,it doesn’t cause interference in the L1or L2caches. However,since the HyperThread antagonist has the smallest pos-sible effect,more intensive antagonists will cause far larger per-formance problems.Thus,HyperThread interference in practice should be avoided.Power interference has a signi?cant impact on websearch at lower utilization,as more cores are executing the power virus.As expected,the network antagonist does not impact websearch,due to websearch’s low bandwidth needs.
ml_cluster ml_cluster is sensitive to LLC interference of smaller size,due to the small but signi?cant per-request working set. This manifests itself as a large jump in latency at75%load for LLC(small)and50%load for LLC(medium).With larger LLC interference,ml_cluster experiences major latency degrada-tion.ml_cluster is also sensitive to DRAM bandwidth interfer-ence,primarily at lower loads(see explanation for websearch). ml_cluster is moderately resistant to HyperThread interference until high loads,suggesting that it only reaches high instruction issue rates at high loads.Power interference has a lesser impact on ml_cluster since it is less compute intensive than websearch. Finally,ml_cluster is not impacted at all by network interference. memkeyval:Due to its signi?cantly stricter latency SLO, memkeyval is sensitive to all types of interference.At high load, memkeyval becomes sensitive even to small LLC interference as
Each entry is color-coded as≥between100%and120%,≤100%.
Figure1.Impact of websearch,ml_cluster,and Each row is an antagonist and each column is a load point for the workload.The values are latencies,normalized to the SLO latency.
the small per-request working sets add up.When faced with medium LLC interference,there are two latency peaks.The?rst peak at low load is caused by the antagonist removing instruc-tions from the cache.When memkeyval obtains enough cores at high load,it avoids these evictions.The second peak is at higher loads,when the antagonist interferes with the per-request work-ing set.At high levels of LLC interference,memkeyval is unable to meet its SLO.Even though memkeyval has low DRAM band-width requirements,it is strongly affected by a DRAM streaming antagonist.Ironically,the few memory requests from memkeyval are overwhelmed by the DRAM antagonist.
memkeyval is not sensitive to the HyperThread antagonist ex-cept at high loads.In contrast,it is very sensitive to the power antagonist,as it is compute-bound.memkeyval does consume a large amount of network bandwidth,and thus is highly suscepti-ble to competing network?ows.Even at small loads,it is com-pletely overrun by the many small“mice”?ows of the antagonist and is unable to meet its SLO.
4Heracles Design
We have established the need for isolation mechanisms be-yond OS-level scheduling and for a dynamic controller that man-ages resource sharing between LC and BE tasks.Heracles is a dynamic,feedback-based controller that manages in real-time four hardware and software mechanisms in order to isolate colocated workloads.Heracles implements an iso-latency pol-icy[47],namely that it can increase resource ef?ciency as long as the SLO is being met.This policy allows for increasing server utilization through tolerating some interference caused by colo-cation,as long as the the difference between the SLO latency target for the LC workload and the actual latency observed(la-tency slack)is positive.In its current version,Heracles manages one LC workload with many BE tasks.Since BE tasks are abun-dant,this is suf?cient to raise utilization in many datacenters.We leave colocation of multiple LC workloads to future work.
4.1Isolation Mechanisms
Heracles manages4mechanisms to mitigate interference.
For core isolation,Heracles uses Linux’s cpuset cgroups to pin the LC workload to one set of cores and BE tasks to another set(software mechanism)[55].This mechanism is necessary,since in§3we showed that core sharing is detrimen-tal to latency SLO.Moreover,the number of cores per server is increasing,making core segregation?ner-grained.The alloca-tion of cores to tasks is done dynamically.The speed of core (re)allocation is limited by how fast Linux can migrate tasks to other cores,typically in the tens of milliseconds.
For LLC isolation,Heracles uses the Cache Allocation Tech-nology(CAT)available in recent Intel chips(hardware mech-anism)[3].CAT implements way-partitioning of the shared LLC.In a highly-associative LLC,this allows us to de?ne non-overlapping partitions at the granularity of a few percent of the total LLC capacity.We use one partition for the LC workload and a second partition for all BE tasks.Partition sizes can be adjusted dynamically by programming model speci?c registers (MSRs),with changes taking effect in a few milliseconds.
There are no commercially available DRAM bandwidth iso-lation mechanisms.We enforce DRAM bandwidth limits in the following manner:we implement a software monitor that peri-odically tracks the total bandwidth usage through performance counters and estimates the bandwidth used by the LC and BE jobs.If the LC workload does not receive suf?cient bandwidth, Heracles scales down the number of cores that BE jobs use.We
discuss the limitations of this coarse-grained approach in§4.2.
For power isolation,Heracles uses CPU frequency monitor-ing,Running Average Power Limit(RAPL),and per-core DVFS (hardware features)[3,37].RAPL is used to monitor CPU power at the per-socket level,while per-core DVFS is used to redis-tribute power amongst cores.Per-core DVFS setting changes go into effect within a few milliseconds.The frequency steps are in100MHz and span the entire operating frequency range of the processor,including Turbo Boost frequencies.
For network traf?c isolation,Heracles uses Linux traf-?c control(software mechanism).Speci?cally we use the qdisc[12]scheduler with hierarchical token bucket queueing discipline(HTB)to enforce bandwidth limits for outgoing traf-?c from the BE tasks.The bandwidth limits are set by limiting the maximum traf?c burst rate for the BE jobs(ceil parameter in HTB parlance).The LC job does not have any limits set on it.HTB can be updated very frequently,with the new bandwidth limits taking effect in less than hundreds of milliseconds.Manag-ing ingress network interference has been examined in numerous previous work and is outside the scope of this work[33].
4.2Design Approach
Each hardware or software isolation mechanism allows rea-sonably precise control of an individual resource.Given that,the controller must dynamically solve the high dimensional problem of?nding the right settings for all these mechanisms at any load for the LC workload and any set of BE tasks.Heracles solves this as an optimization problem,where the objective is to maxi-mize utilization with the constraint that the SLO must be met.
Heracles reduces the optimization complexity by decoupling interference sources.The key insight that enables this reduction is that interference is problematic only when a shared resource becomes saturated,i.e.its utilization is so high that latency prob-lems occur.This insight is derived by the analysis in§3:the antagonists do not cause signi?cant SLO violations until an in-?ection point,at which point the tail latency degrades extremely rapidly.Hence,if Heracles can prevent any shared resource from saturating,then it can decompose the high-dimensional optimiza-tion problem into many smaller and independent problems of one or two dimensions each.Then each sub-problem can be solved using sound optimization methods,such as gradient descent.
Since Heracles must ensure that the target SLO is met for the LC workload,it continuously monitors latency and latency slack and uses both as key inputs in its decisions.When the latency slack is large,Heracles treats this as a signal that it is safe to be more aggressive with colocation;conversely,when the slack is small,it should back off to avoid an SLO violation.Heracles also monitors the load(queries per second),and during periods of high load,it disables colocation due to a high risk of SLO violations.Previous work has shown that indirect performance metrics,such as CPU utilization,are insuf?cient to guarantee that the SLO is met[47].
Ideally,Heracles should require no of?ine information other than SLO targets.Unfortunately,one shortcoming of current hardware makes this dif?cult.The Intel chips we used do not provide accurate mechanisms for measuring(or limiting) DRAM bandwidth usage at a per-core granularity.To understand how Heracles’decisions affect the DRAM bandwidth usage of
Figure2.The system diagram of Heracles.
latency-sensitive and BE tasks and to manage bandwidth satura-tion,we require some of?ine information.Speci?cally,Heracles uses an of?ine model that describes the DRAM bandwidth used by the latency-sensitive workloads at various loads,core,and LLC allocations.We veri?ed that this model needs to be regen-erated only when there are signi?cant changes in the workload structure and that small deviations are?ne.There is no need for any of?ine pro?ling of the BE tasks,which can vary widely compared to the better managed and understood LC workloads. There is also no need for of?ine analysis of interactions between latency-sensitive and best effort tasks.Once we have hardware support for per-core DRAM bandwidth accounting[30],we can eliminate this of?ine model.
4.3Heracles Controller
Heracles runs as a separate instance on each server,managing the local interactions between the LC and BE jobs.As shown in Figure2,it is organized as three subcontrollers(cores&mem-ory,power,network traf?c)coordinated by a top-level controller. The subcontrollers operate fairly independently of each other and ensure that their respective shared resources are not saturated. Top-level controller:The pseudo-code for the controller is shown in Algorithm1.The controller polls the tail latency and load of the LC workload every15seconds.This allows for suf-?cient queries to calculate statistically meaningful tail latencies. If the load for the LC workload exceeds85%of its peak on the server,the controller disables the execution of BE workloads. This empirical safeguard avoids the dif?culties of latency man-agement on highly utilized systems for minor gains in utilization. For hysteresis purposes,BE execution is enabled when the load drops below80%.BE execution is also disabled when the la-tency slack,the difference between the SLO target and the cur-rent measured tail latency,is negative.This typically happens when there is a sharp spike in load for the latency-sensitive work-load.We give all resources to the latency critical workload for a while(e.g.,5minutes)before attempting colocation again.The constants used here were determined through empirical tuning.
When these two safeguards are not active,the controller uses slack to guide the subcontrollers in providing resources to BE tasks.If slack is less than10%,the subcontrollers are instructed to disallow growth for BE tasks in order to maintain a safety margin.If slack drops below5%,the subcontroller for cores is instructed to switch cores from BE tasks to the LC workload. This improves the latency of the LC workload and reduces the
1while True:
2latency=PollLCAppLatency()
3load=PollLCAppLoad()
4slack=(target-latency)/target
5if slack<0:
6DisableBE()
7EnterCooldown()
8elif load>0.85:
9DisableBE()
10elif load<0.80:
11EnableBE()
12elif slack<0.10:
13DisallowBEGrowth()
14if slack<0.05:
15be_cores.Remove(be_cores.Size()-2)
16sleep(15)
Algorithm1:High-level controller.
ability of the BE job to cause interference on any resources.If slack is above10%,the subcontrollers are instructed to allow BE tasks to acquire a larger share of system resources.Each sub-controller makes allocation decisions independently,provided of course that its resources are not saturated.
Core&memory subcontroller:Heracles uses a single subcon-troller for core and cache allocation due to the strong coupling between core count,LLC needs,and memory bandwidth needs. If there was a direct way to isolate memory bandwidth,we would use independent controllers.The pseudo-code for this subcon-troller is shown in Algorithm2.Its output is the allocation of cores and LLC to the LC and BE jobs(2dimensions).
The?rst constraint for the subcontroller is to avoid memory bandwidth saturation.The DRAM controllers provide registers that track bandwidth usage,making it easy to detect when they reach90%of peak streaming DRAM bandwidth.In this case,the subcontroller removes as many cores as needed from BE tasks to avoid saturation.Heracles estimates the bandwidth usage of each BE task using a model of bandwidth needs for the LC work-load and a set of hardware counters that are proportional to the per-core memory traf?c to the NUMA-local memory controllers. For the latter counters to be useful,we limit each BE task to a single socket for both cores and memory allocations using Linux numactl.Different BE jobs can run on either socket and LC workloads can span across sockets for cores and memory.
When the top-level controller signals BE growth and there is no DRAM bandwidth saturation,the subcontroller uses gra-dient descent to?nd the maximum number of cores and cache partitions that can be given to BE tasks.Of?ine analysis of LC applications(Figure3)shows that their performance is a convex function of core and cache resources,thus guaranteeing that gradient descent will?nd a global optimum.We perform the gradient descent in one dimension at a time,switching be-tween increasing the cores and increasing the cache given to BE tasks.Initially,a BE job is given one core and10%of the LLC and starts in the GROW_LLC phase.Its LLC allocation is in-creased as long as the LC workload meets its SLO,bandwidth saturation is avoided,and the BE task bene?ts.The next phase (GROW_CORES)grows the number of cores for the BE job.Her-acles will reassign cores from the LC to the BE job one at a
1def PredictedTotalBW():
2return LcBwModel()+BeBw()+bw_derivative
3while True:
4MeasureDRAMBw()
5if total_bw>DRAM_LIMIT:
6overage=total_bw-DRAM_LIMIT
7be_cores.Remove(overage/BeBwPerCore())
8continue
9if not CanGrowBE():
10continue
11if state==GROW_LLC:
12if PredictedTotalBW()>DRAM_LIMIT:
13state=GROW_CORES
14else:
15GrowCacheForBE()
16MeasureDRAMBw()
17if bw_derivative>=0:
18Rollback()
19state=GROW_CORES
20if not BeBene?t():
21state=GROW_CORES
22elif state==GROW_CORES:
23needed=LcBwModel()+BeBw()+BeBwPerCore()
24if needed>DRAM_LIMIT:
25state=GROW_LLC
26elif slack>0.10:
27be_cores.Add(1)
28sleep(2)
Algorithm2:Core&memory sub-controller.
time,each time checking for DRAM bandwidth saturation and SLO violations for the LC workload.If bandwidth saturation oc-curs?rst,the subcontroller will return to the GROW_LLC phase. The process repeats until an optimal con?guration has been con-verged upon.The search also terminates on a signal from the top-level controller indicating the end to growth or the disabling of BE jobs.The typical convergence time is about30seconds.
During gradient descent,the subcontroller must avoid trying suboptimal allocations that will either trigger DRAM bandwidth saturation or a signal from the top-level controller to disable BE tasks.To estimate the DRAM bandwidth usage of an alloca-tion prior to trying it,the subcontroller uses the derivative of the DRAM bandwidth from the last reallocation of cache or cores. Heracles estimates whether it is close to an SLO violation for the LC task based on the amount of latency slack.
Power subcontroller:The simple subcontroller described in Algorithm3ensures that there is suf?cient power slack to run the LC workload at a minimum guaranteed frequency.This fre-quency is determined by measuring the frequency used when the LC workload runs alone at full load.Heracles uses RAPL to de-termine the operating power of the CPU and its maximum design power,or thermal dissipation power(TDP).It also uses CPU fre-quency monitoring facilities on each core.When the operating power is close to the TDP and the frequency of the cores running the LC workload is too low,it uses per-core DVFS to lower the frequency of cores running BE tasks in order to shift the power budget to cores running LC tasks.Both conditions must be met in order to avoid confusion when the LC cores enter active-idle
Figure3.Characterization of websearch showing that its per-formance is a convex function of cores and LLC.
1while True:
2power=PollRAPL()
3ls_freq=PollFrequency(ls_cores)
4if power>0.90*TDP and ls_freq 5LowerFrequency(be_cores) 6elif power<=0.90*TDP and ls_freq>=guaranteed: 7IncreaseFrequency(be_cores) 8sleep(2) Algorithm3:CPU power sub-controller. 1while True: 2ls_bw=GetLCTxBandwidth() 3be_bw=LINK_RATE-ls_bw-max(0.05*LINK_RATE, 0.10*ls_bw) 4SetBETxBandwidth(be_bw) 5sleep(1) Algorithm4:Network sub-controller. modes,which also tends to lower frequency readings.If there is suf?cient operating power headroom,Heracles will increase the frequency limit for the BE cores in order to maximize their per-formance.The control loop runs independently for each of the two sockets and has a cycle time of two seconds. Network subcontroller:This subcontroller prevents satura-tion of network transmit bandwidth as shown in Algorithm4. It monitors the total egress bandwidth of?ows associated with the LC workload(LCBandwidth)and sets the total band-width limit of all other?ows as LinkRate?LCBandwidth?max(0.05LinkRate,0.10LCBandwidth).A small headroom of 10%of the current LCBandwidth or5%of the LinkRate is added into the reservation for the LC workload in order to handle spikes. The bandwidth limit is enforced via HTB qdiscs in the Linux ker-nel.This control loop is run once every second,which provides suf?cient time for the bandwidth enforcer to settle.5Heracles Evaluation 5.1Methodology We evaluated Heracles with the three production,latency-critical workloads from Google analyzed in§3.We?rst per-formed experiments with Heracles on a single leaf server,intro-ducing BE tasks as we run the LC workload at different levels of load.Next,we used Heracles on a websearch cluster with tens of servers,measuring end-to-end workload latency across the fan-out tree while BE tasks are also running.In the clus-ter experiments,we used a load trace that represents the traf?c throughout a day,capturing diurnal load variation.In all cases, we used production Google servers. For the LC workloads we focus on SLO latency.Since the SLO is de?ned over60-second windows,we report the worst-case latency that was seen during experiments.For the produc-tion batch workloads,we compute the throughput rate of the batch workload with Heracles and normalize it to the throughput of the batch workload running alone on a single server.We then de?ne the Effective Machine Utilization(EMU)=LC Through-put+BE Throughput.Note that Effective Machine Utilization can be above100%due to better binpacking of shared resources. We also report the utilization of shared resources when necessary to highlight detailed aspects of the system. The BE workloads we use are chosen from a set containing both production batch workloads and the synthetic tasks that stress a single shared resource.The speci?c workloads are: stream-LLC streams through data sized to?t in about half of the LLC and is the same as LLC(med)from§3.2.stream-DRAM streams through an extremely large array that cannot?t in the LLC(DRAM from the same section).We use these work-loads to verify that Heracles is able to maximize the use of LLC partitions and avoid DRAM bandwidth saturation. cpu_pwr is the CPU power virus from§3.2.It is used to verify that Heracles will redistribute power to ensure that the LC workload maintains its guaranteed frequency. iperf is an open source network streaming benchmark used to verify that Heracles partitions network transmit bandwidth cor-rectly to protect the LC workload. brain is a Google production batch workload that performs deep learning on images for automatic labelling[38,67].This workload is very computationally intensive,is sensitive to LLC size,and also has high DRAM bandwidth requirements. streetview is a production batch job that stitches together mul-tiple images to form the panoramas for Google Street View.This workload is highly demanding on the DRAM subsystem. 5.2Individual Server Results Latency SLO:Figure4presents the impact of colocating each of the three LC workloads with BE workloads across all possible loads under the control of Heracles.Note that Her-acles attempts to run as many copies of the BE task as possi-ble and maximize the resources they receive.At all loads and in all colocation cases,there are no SLO violations with Her-acles.This is true even for brain,a workload that even with the state-of-the-art OS isolation mechanisms would render any LC workload unusable.This validates that the controller keeps shared resources from saturating and allocates a suf?cient frac-tion to the LC workload at any load.Heracles maintains a small Figure https://www.wendangku.net/doc/4110046684.html,tency of LC applications co-located with BE jobs under Heracles .For clarity we omit websearch and ml _cluster with iperf as those workloads are extremely resistant to network interference. Figure 5.EMU achieved by Heracles . latency slack as a guard band to avoid spikes and control insta-bility.It also validates that local information on tail latency is suf?cient for stable control for applications with milliseconds and microseconds range of SLOs.Interestingly,the websearch binary and shard changed between generating the of?ine pro?l-ing model for DRAM bandwidth and performing this experiment.Nevertheless,Heracles is resilient to these changes and performs well despite the somewhat outdated model. Heracles reduces the latency slack during periods of low uti-lization for all workloads.For websearch and ml_cluster ,the slack is cut in half,from 40%to 20%.For memkeyval ,the reduc-tion is much more dramatic,from a slack of 80%to 40%or less.This is because the unloaded latency of memkeyval is extremely small compared to the SLO latency.The high variance of the tail latency for memkeyval is due to the fact that its SLO is in the hun-dreds of microseconds,making it more sensitive to interference than the other two workloads. Server Utilization:Figure 5shows the EMU achieved when colocating production LC and BE tasks with Heracles .In all cases,we achieve signi?cant EMU increases.When the two most CPU-intensive and power-hungry workloads are combined,websearch and brain ,Heracles still achieves an EMU of at least 75%.When websearch is combined with the DRAM bandwidth intensive streetview ,Heracles can extract suf?cient resources for a total EMU above 100%at websearch loads between 25%and 70%.This is because websearch and streetivew have complemen-tary resource requirements,where websearch is more compute bound and streetview is more DRAM bandwidth bound.The EMU results are similarly positive for ml_cluster and memkeyval . By dynamically managing multiple isolation mechanisms,Hera-cles exposes opportunities to raise EMU that would otherwise be missed with scheduling techniques that avoid interference. Shared Resource Utilization:Figure 6plots the utilization of shared resources (cores,power,and DRAM bandwidth)under Heracles control.For memkeyval ,we include measurements of network transmit bandwidth in Figure 7. Across the board,Heracles is able to correctly size the BE workloads to avoid saturating DRAM bandwidth.For the stream-LLC BE task,Heracles ?nds the correct cache partitions to decrease total DRAM bandwidth requirements for all work-loads.For ml_cluster ,with its large cache footprint,Heracles balances the needs of stream-LLC with ml_cluster effectively,with a total DRAM bandwidth slightly above the baseline.For the BE tasks with high DRAM requirements (stream-DRAM,streetview),Heracles only allows them to execute on a few cores to avoid saturating DRAM.This is re?ected by the lower CPU utilization but high DRAM bandwidth.However,EMU is still high,as the critical resource for those workloads is not compute,but memory bandwidth. Looking at the power utilization,Heracles allows signi?cant improvements to energy ef?ciency.Consider the 20%load case:EMU was raised by a signi?cant amount,from 20%to 60%-90%.However,the CPU power only increased from 60%to 80%.This translates to an energy ef?ciency gain of 2.3-3.4x.Overall,Heracles achieves signi?cant gains in resource ef?ciency across all loads for the LC task without causing SLO violations. 5.3Websearch Cluster Results We also evaluate Heracles on a small minicluster for web-search with tens of servers as a proxy for the full-scale cluster.The cluster root fans out each user request to all leaf servers and combines their replies.The SLO latency is de?ned as the aver-age latency at the root over 30seconds,denoted as μ/30s .The target SLO latency is set as μ/30s when serving 90%load in the cluster without colocated tasks.Heracles runs on every leaf node with a uniform 99%-ile latency target set such that the latency at the root satis?es the SLO.We use Heracles to execute brain on half of the leafs and streetview on the other half.Heracles shares the same of?ine model for the DRAM bandwidth needs of web-search across all leaves,even though each leaf has a different shard.We generate load from an anonymized,12-hour request trace that captures the part of the daily diurnal pattern when web- Figure6.Various system utilization metrics of LC applications co-located with BE jobs under Heracles. https://www.wendangku.net/doc/4110046684.html,work bandwidth of memkeyval under Heracles. search is not fully loaded and colocation has high potential. Latency SLO:Figure8shows the latency SLO with and without Heracles for the12-hour trace.Heracles produces no SLO violations while reducing slack by20-30%.Meeting the 99%-ile tail latency at each leaf is suf?cient to guarantee the global SLO.We believe we can further reduce the slack in larger websearch clusters by introducing a centralized controller that dynamically sets the per-leaf tail latency targets based on slack at the root[47].This will allow a future version of Heracles to take advantage of slack in higher layers of the fan-out tree. Server Utilization:Figure8also shows that Heracles suc-cessfully converts the latency slack in the baseline case into sig-ni?cantly increased EMU.Throughout the trace,Heracles colo-cates suf?cient BE tasks to maintain an average EMU of90% and a minimum of80%without causing SLO violations.The websearch load varies between20%and90%in this trace. TCO:To estimate the impact on total cost of ownership,we use the TCO calculator by Barroso et al.with the parameters from the case-study of a datacenter with low per-server cost[7]. This model assumes$2000servers with a PUE of2.0and a peak power draw of500W as well as electricity costs of$0.10/kW-hr. For our calculations,we assume a cluster size of10,000servers. Assuming pessimistically that a websearch cluster is highly uti-lized throughout the day,with an average load of75%,Heracles’ability to raise utilization to90%translates to a15%through-put/TCO improvement over the baseline.This improvement in-cludes the cost of the additional power consumption at higher uti-lization.Under the same assumptions,a controller that focuses only on improving energy-proportionality for websearch would achieve throughput/TCO gains of roughly3%[47]. If we assume a cluster for LC workloads utilized at an average of20%,as many industry studies suggest[44,74],Heracles can achieve a306%increase in throughput/TCO.A controller focus-ing on energy-proportionality would achieve improvements of https://www.wendangku.net/doc/4110046684.html,tency SLO and effective machine utilization for a websearch cluster managed by Heracles. less than7%.Heracles’advantage is due to the fact that it can raise utilization from20%to90%with a small increase to power consumption,which only represents9%of the initial TCO.As long as there are useful BE tasks available,one should always choose to improve throughput/TCO by colocating them with LC jobs instead of lowering the power consumption of servers in modern datacenters.Also note that the improvements in through-put/TCO are large enough to offset the cost of reserving a small portion of each server’s memory or storage for BE tasks. 6Related Work Isolation mechanisms:There is signi?cant work on shared cache isolation,including soft partitioning based on replace-ment policies[77,78],way-partitioning[65,64],and?ne-grained partitioning[68,49,71].Tessellation exposes an inter-face for throughput-based applications to request partitioned re-sources[45].Most cache partitioning schemes have been eval-uated with a utility-based policy that optimizes for aggregate throughput[64].Heracles manages the coarse-grained,way-partitioning scheme recently added in Intel CPUs,using a search for a right-sized allocation to eliminate latency SLO violations. We expect Heracles will work even better with?ne-grained par-titioning schemes when they are commercially available. Iyer et al.explores a wide range quality-of-service(QoS)poli-cies for shared cache and memory systems with simulated isola-tion features[30,26,24,23,29].They focus on throughput met-rics,such as IPC and MPI,and did not consider latency-critical workloads or other resources such as network traf?c.Cook et al. evaluate hardware cache partitioning for throughput based appli-cations and did not consider latency-critical tasks[15].Wu et https://www.wendangku.net/doc/4110046684.html,pare different capacity management schemes for shared caches[77].The proposed Ubik controller for shared caches with?ne-grained partitioning support boosts the allocation for latency-critical workloads during load transition times and re-quires application level changes to inform the runtime of load changes[36].Heracles does not require any changes to the LC task,instead relying on a steady-state approach for managing cache partitions that changes partition sizes slowly. There are several proposals for isolation and QoS features for memory controllers[30,56,32,59,57,20,40,70].While our work showcases the need for memory isolation for latency-critical workloads,such features are not commercially available at this point.Several network interface controllers implement bandwidth limiters and priority mechanisms in hardware.Unfor-tunately,these features are not exposed by device drivers.Hence, Heracles and related projects in network performance isolation currently use Linux qdisc[33].Support for network isolation in hardware should strengthen this work. The LC workloads we evaluated do not use disks or SSDs in order to meet their aggressive latency targets.Nevertheless,disk and SSD isolation is quite similar to network isolation.Thus,the same principles and controls used to mitigate network interfer-ence still apply.For disks,we list several available isolation tech-niques:1)the cgroups blkio controller[55],2)native command queuing(NCQ)priorities[27],3)prioritization in?le-system queues,4)partitioning LC and BE to different disks,5)repli-cating LC data across multiple disks that allows selecting the disk/reply that responds?rst or has lower load[17].For SSDs: 1)many SSDs support channel partitions,separate queueing,and prioritization at the queue level,2)SSDs also support suspending operations to allow LC requests to overtake BE requests. Interference-aware cluster management:Several cluster-management systems detect interference between colocated workloads and generate schedules that avoid problematic colo-cations.Nathuji et al.develop a feedback-based scheme that tunes resource assignment to mitigate interference for colocated VMs[58].Bubble-?ux is an online scheme that detects memory pressure and?nds colocations that avoid interference on latency-sensitive workloads[79,51].Bubble-?ux has a backup mech-anism to enable problematic co-locations via execution modu-lation,but such a mechanism would have challenges with ap-plications such as memkeyval,as the modulation would need to be done in the granularity of microseconds.DeepDive detects and manages interference between co-scheduled applications in a VM system[60].CPI2throttles low-priority workloads that in-terfere with important services[80].Finally,Paragon and Quasar use online classi?cation to estimate interference and to colocate workloads that are unlikely to cause interference[18,19]. The primary difference of Heracles is the focus on latency-critical workloads and the use of multiple isolation schemes in order to allow aggressive colocation without SLO violations at scale.Many previous approaches use IPC instead of latency as the performance metric[79,51,60,80].Nevertheless,one can couple Heracles with an interference-aware cluster manager in order to optimize the placement of BE tasks. Latency-critical workloads:There is also signi?cant work in optimizing various aspects of latency-critical workloads,includ-ing energy proportionality[53,54,47,46,34],networking per-formance[35,8],and hardware-acceleration[41,63,72].Hera-cles is largely orthogonal to these projects. 7Conclusions We present Heracles,a heuristic feedback-based system that manages four isolation mechanisms to enable a latency-critical workload to be colocated with batch jobs without SLO viola-tions.We used an empirical characterization of several sources of interference to guide an important heuristic used in Heracles: interference effects are large only when a shared resource is sat-urated.We evaluated Heracles and several latency-critical and batch workloads used in production at Google on real hardware and demonstrated an average utilization of90%across all evalu-ated scenarios without any SLO violations for the latency-critical job.Through coordinated management of several isolation mech-anisms,Heracles enables colocation of tasks that previously would cause SLO https://www.wendangku.net/doc/4110046684.html,pared to power-saving mech-anisms alone,Heracles increases overall cost ef?ciency substan-tially through increased utilization. 8Acknowledgements We sincerely thank Luiz Barroso and Chris Johnson for their help and insight in making our work possible at Google.We also thank Christina Delimitrou,Caroline Suen,and the anony-mous reviewers for their feedback on earlier versions of this manuscript.This work was supported by a Google research grant,the Stanford Experimental Datacenter Lab,and NSF grant CNS-1422088.David Lo was supported by a Google PhD Fel-lowship. References [1]“Iperf-The TCP/UDP Bandwidth Measurement Tool,”https://iperf.fr/. [2]“memcached,”https://www.wendangku.net/doc/4110046684.html,/. [3]“Intel R 64and IA-32Architectures Software Developer’s Manual,” vol.3B:System Programming Guide,Part2,Sep2014. [4]Mohammad Al-Fares et al.,“A Scalable,Commodity Data Center Net- work Architecture,”in Proc.of the ACM SIGCOMM2008Conference on Data Communication,ser.SIGCOMM’08.New York,NY:ACM, 2008. [5]Mohammad Alizadeh et al.,“Data Center TCP(DCTCP),”in Proc.of the ACM SIGCOMM2010Conference,ser.SIGCOMM’10.New York,NY:ACM,2010. [6]Luiz Barroso et al.,“The Case for Energy-Proportional Computing,” Computer,vol.40,no.12,Dec.2007. [7]Luiz AndréBarroso et al.,The Datacenter as a Computer:An Intro- duction to the Design of Warehouse-Scale Machines,2nd ed.Morgan &Claypool Publishers,2013. [8]Adam Belay et al.,“IX:A Protected Dataplane Operating System for High Throughput and Low Latency,”in11th USENIX Symposium on Operating Systems Design and Implementation(OSDI14).Broom-?eld,CO:USENIX Association,Oct.2014. [9]Sergey Blagodurov et al.,“A Case for NUMA-aware Contention Man- agement on Multicore Systems,”in Proc.of the2011USENIX Confer-ence on USENIX Annual Technical Conference,https://www.wendangku.net/doc/4110046684.html,ENIXATC’11. Berkeley,CA:USENIX Association,2011. [10]Eric Boutin et al.,“Apollo:Scalable and Coordinated Scheduling for Cloud-Scale Computing,”in11th USENIX Symposium on Operating Systems Design and Implementation(OSDI14).Broom?eld,CO: USENIX Association,2014. [11]Bob Briscoe,“Flow Rate Fairness:Dismantling a Religion,”SIG- COMM https://www.wendangku.net/doc/4110046684.html,mun.Rev.,vol.37,no.2,Mar.2007. [12]Martin A.Brown,“Traf?c Control HOWTO,”https://www.wendangku.net/doc/4110046684.html,/ articles/Traf?c-Control-HOWTO/.[13]Marcus Carvalho et al.,“Long-term SLOs for Reclaimed Cloud Com- puting Resources,”in Proc.of SOCC,Seattle,W A,Dec.2014. [14]McKinsey&Company,“Revolutionizing data center ef?ciency,”Up- time Institute Symp.,2008. [15]Henry Cook et al.,“A Hardware Evaluation of Cache Partitioning to Improve Utilization and Energy-ef?ciency While Preserving Respon-siveness,”in Proc.of the40th Annual International Symposium on Computer Architecture,ser.ISCA’13.New York,NY:ACM,2013. [16]Carlo Curino et al.,“Reservation-based Scheduling:If You’re Late Don’t Blame Us!”in Proc.of the5th annual Symposium on Cloud Computing,2014. [17]Jeffrey Dean et al.,“The tail at scale,”Commun.ACM,vol.56,no.2, Feb.2013. [18]Christina Delimitrou et al.,“Paragon:QoS-Aware Scheduling for Heterogeneous Datacenters,”in Proc.of the18th Intl.Conf.on Archi-tectural Support for Programming Languages and Operating Systems (ASPLOS),Houston,TX,2013. [19]Christina Delimitrou et al.,“Quasar:Resource-Ef?cient and QoS- Aware Cluster Management,”in Proc.of the Nineteenth International Conference on Architectural Support for Programming Languages and Operating Systems(ASPLOS),Salt Lake City,UT,2014. [20]Eiman Ebrahimi et al.,“Fairness via Source Throttling:A Con?gurable and High-performance Fairness Substrate for Multi-core Memory Sys-tems,”in Proc.of the Fifteenth Edition of ASPLOS on Architectural Support for Programming Languages and Operating Systems,ser.AS-PLOS XV.New York,NY:ACM,2010. [21]H.Esmaeilzadeh et al.,“Dark silicon and the end of multicore scaling,” in Computer Architecture(ISCA),201138th Annual International Symposium on,June2011. [22]Sriram Govindan et al.,“Cuanta:quantifying effects of shared on-chip resource interference for consolidated virtual machines,”in Proc.of the2nd ACM Symposium on Cloud Computing,2011. [23]Fei Guo et al.,“From Chaos to QoS:Case Studies in CMP Resource Management,”SIGARCH Comput.Archit.News,vol.35,no.1,Mar. 2007. [24]Fei Guo et al.,“A Framework for Providing Quality of Service in Chip Multi-Processors,”in Proc.of the40th Annual IEEE/ACM International Symposium on Microarchitecture,ser.MICRO40.Washington,DC: IEEE Computer Society,2007. [25]Nikos Hardavellas et al.,“Toward Dark Silicon in Servers,”IEEE Micro,vol.31,no.4,2011. [26]Lisa R.Hsu et al.,“Communist,Utilitarian,and Capitalist Cache Policies on CMPs:Caches As a Shared Resource,”in Proc.of the15th International Conference on Parallel Architectures and Compilation Techniques,ser.PACT’06.New York,NY:ACM,2006. [27]Intel,“Serial ATA II Native Command Queuing Overview,” https://www.wendangku.net/doc/4110046684.html,/support/chipsets/imsm/sb/sata2_ncq_ overview.pdf,2003. [28]Teerawat Issariyakul et al.,Introduction to Network Simulator NS2, 1st ed.Springer Publishing Company,Incorporated,2010. [29]Ravi Iyer,“CQoS:A Framework for Enabling QoS in Shared Caches of CMP Platforms,”in Proc.of the18th Annual International Conference on Supercomputing,ser.ICS’04.New York,NY:ACM,2004. [30]Ravi Iyer et al.,“QoS Policies and Architecture for Cache/Memory in CMP Platforms,”in Proc.of the2007ACM SIGMETRICS International Conference on Measurement and Modeling of Computer Systems,ser. SIGMETRICS’07.New York,NY:ACM,2007. [31]Vijay Janapa Reddi et al.,“Web Search Using Mobile Cores:Quantify- ing and Mitigating the Price of Ef?ciency,”SIGARCH Comput.Archit. News,vol.38,no.3,Jun.2010. [32]Min Kyu Jeong et al.,“A QoS-aware Memory Controller for Dynami- cally Balancing GPU and CPU Bandwidth Use in an MPSoC,”in Proc. of the49th Annual Design Automation Conference,ser.DAC’12.New York,NY:ACM,2012. [33]Vimalkumar Jeyakumar et al.,“EyeQ:Practical Network Performance Isolation at the Edge,”in Proc.of the10th USENIX Conference on Networked Systems Design and Implementation,ser.nsdi’13.Berkeley, CA:USENIX Association,2013. [34]Svilen Kanev et al.,“Tradeoffs between Power Management and Tail Latency in Warehouse-Scale Applications,”in IISWC,2014. [35]Rishi Kapoor et al.,“Chronos:Predictable Low Latency for Data Center Applications,”in Proc.of the Third ACM Symposium on Cloud Computing,ser.SoCC’12.New York,NY:ACM,2012. [36]Harshad Kasture et al.,“Ubik:Ef?cient Cache Sharing with Strict QoS for Latency-Critical Workloads,”in Proc.of the19th international conference on Architectural Support for Programming Languages and Operating Systems(ASPLOS-XIX),March2014. [37]Wonyoung Kim et al.,“System level analysis of fast,per-core DVFS using on-chip switching regulators,”in High Performance Computer Architecture,2008.HPCA2008.IEEE14th International Symposium on,Feb2008. [38]Quoc Le et al.,“Building high-level features using large scale unsu- pervised learning,”in International Conference in Machine Learning, 2012. [39]Jacob Leverich et al.,“Reconciling High Server Utilization and Sub- millisecond Quality-of-Service,”in SIGOPS European Conf.on Com-puter Systems(EuroSys),2014. [40]Bin Li et al.,“CoQoS:Coordinating QoS-aware Shared Resources in NoC-based SoCs,”J.Parallel https://www.wendangku.net/doc/4110046684.html,put.,vol.71,no.5,May 2011. [41]Kevin Lim et al.,“Thin Servers with Smart Pipes:Designing SoC Accelerators for Memcached,”in Proc.of the40th Annual International Symposium on Computer Architecture,2013. [42]Kevin Lim et al.,“System-level Implications of Disaggregated Mem- ory,”in Proc.of the2012IEEE18th International Symposium on High-Performance Computer Architecture,ser.HPCA’12.Washington, DC:IEEE Computer Society,2012. [43]Jiang Lin et al.,“Gaining insights into multicore cache partitioning: Bridging the gap between simulation and real systems,”in High Perfor-mance Computer Architecture,2008.HPCA2008.IEEE14th Interna-tional Symposium on,Feb2008. [44]Huan Liu,“A Measurement Study of Server Utilization in Public Clouds,”in Dependable,Autonomic and Secure Computing(DASC), 2011IEEE Ninth Intl.Conf.on,2011. [45]Rose Liu et al.,“Tessellation:Space-time Partitioning in a Manycore Client OS,”in Proc.of the First USENIX Conference on Hot Topics in Parallelism,ser.HotPar’09.Berkeley,CA:USENIX Association, 2009. [46]Yanpei Liu et al.,“SleepScale:Runtime Joint Speed Scaling and Sleep States Management for Power Ef?cient Data Centers,”in Proceeding of the41st Annual International Symposium on Computer Architecuture, ser.ISCA’14.Piscataway,NJ:IEEE Press,2014. [47]David Lo et al.,“Towards Energy Proportionality for Large-scale Latency-critical Workloads,”in Proceeding of the41st Annual In-ternational Symposium on Computer Architecuture,ser.ISCA’14. Piscataway,NJ:IEEE Press,2014. [48]Krishna T.Malladi et al.,“Towards Energy-proportional Datacen- ter Memory with Mobile DRAM,”SIGARCH Comput.Archit.News, vol.40,no.3,Jun.2012. [49]R Manikantan et al.,“Probabilistic Shared Cache Management (PriSM),”in Proc.of the39th Annual International Symposium on Computer Architecture,ser.ISCA’12.Washington,DC:IEEE Com-puter Society,2012. [50]J.Mars et al.,“Increasing Utilization in Modern Warehouse-Scale Computers Using Bubble-Up,”Micro,IEEE,vol.32,no.3,May2012. [51]Jason Mars et al.,“Bubble-Up:Increasing Utilization in Modern Ware- house Scale Computers via Sensible Co-locations,”in Proc.of the44th Annual IEEE/ACM Intl.Symp.on Microarchitecture,ser.MICRO-44’11,2011. [52]Paul Marshall et al.,“Improving Utilization of Infrastructure Clouds,” in Proc.of the201111th IEEE/ACM International Symposium on Cluster,Cloud and Grid Computing,2011. [53]David Meisner et al.,“PowerNap:Eliminating Server Idle Power,”in Proc.of the14th Intl.Conf.on Architectural Support for Programming Languages and Operating Systems,ser.ASPLOS XIV,2009. [54]David Meisner et al.,“Power Management of Online Data-Intensive Services,”in Proc.of the38th ACM Intl.Symp.on Computer Architec-ture,2011. [55]Paul Menage,“CGROUPS,”https://https://www.wendangku.net/doc/4110046684.html,/doc/ Documentation/cgroups/cgroups.txt. [56]Sai Prashanth Muralidhara et al.,“Reducing Memory Interference in Multicore Systems via Application-aware Memory Channel Partition-ing,”in Proc.of the44th Annual IEEE/ACM International Symposium on Microarchitecture,ser.MICRO-44.New York,NY:ACM,2011. [57]Vijay Nagarajan et al.,“ECMon:Exposing Cache Events for Monitor- ing,”in Proc.of the36th Annual International Symposium on Computer Architecture,ser.ISCA’09.New York,NY:ACM,2009. [58]R.Nathuji et al.,“Q-Clouds:Managing Performance Interference Effects for QoS-Aware Clouds,”in Proc.of EuroSys,France,2010. [59]K.J.Nesbit et al.,“Fair Queuing Memory Systems,”in Microarchitec- ture,2006.MICRO-39.39th Annual IEEE/ACM International Sympo-sium on,Dec2006.[60]Dejan Novakovic et al.,“DeepDive:Transparently Identifying and Managing Performance Interference in Virtualized Environments,”in Proc.of the USENIX Annual Technical Conference(ATC’13),San Jose, CA,2013. [61]W.Pattara-Aukom et al.,“Starvation prevention and quality of service in wireless LANs,”in Wireless Personal Multimedia Communications, 2002.The5th International Symposium on,vol.3,Oct2002. [62]M.Podlesny et al.,“Solving the TCP-Incast Problem with Application- Level Scheduling,”in Modeling,Analysis Simulation of Computer and Telecommunication Systems(MASCOTS),2012IEEE20th Interna-tional Symposium on,Aug2012. [63]Andrew Putnam et al.,“A Recon?gurable Fabric for Accelerating Large-scale Datacenter Services,”in Proceeding of the41st Annual International Symposium on Computer Architecuture,ser.ISCA’14. Piscataway,NJ:IEEE Press,2014. [64]M.K.Qureshi et al.,“Utility-Based Cache Partitioning:A Low- Overhead,High-Performance,Runtime Mechanism to Partition Shared Caches,”in Microarchitecture,2006.MICRO-39.39th Annual IEEE/ACM International Symposium on,Dec2006. [65]Parthasarathy Ranganathan et al.,“Recon?gurable Caches and Their Application to Media Processing,”in Proc.of the27th Annual Inter-national Symposium on Computer Architecture,ser.ISCA’00.New York,NY:ACM,2000. [66]Charles Reiss et al.,“Heterogeneity and Dynamicity of Clouds at Scale: Google Trace Analysis,”in ACM Symp.on Cloud Computing(SoCC), Oct.2012. [67]Chuck Rosenberg,“Improving Photo Search:A Step Across the Semantic Gap,”https://www.wendangku.net/doc/4110046684.html,/2013/06/ improving-photo-search-step-across.html. [68]Daniel Sanchez et al.,“Vantage:Scalable and Ef?cient Fine-grain Cache Partitioning,”SIGARCH Comput.Archit.News,vol.39,no.3, Jun.2011. [69]Yoon Jae Seong et al.,“Hydra:A Block-Mapped Parallel Flash Mem- ory Solid-State Disk Architecture,”Computers,IEEE Transactions on, vol.59,no.7,July2010. [70]Akbar Shari?et al.,“METE:Meeting End-to-end QoS in Multicores Through System-wide Resource Management,”in Proc.of the ACM SIGMETRICS Joint International Conference on Measurement and Modeling of Computer Systems,ser.SIGMETRICS’11.New York, NY:ACM,2011. [71]Shekhar Srikantaiah et al.,“SHARP Control:Controlled Shared Cache Management in Chip Multiprocessors,”in Proc.of the42Nd Annual IEEE/ACM International Symposium on Microarchitecture,ser.MI-CRO42.New York,NY:ACM,2009. [72]Shingo Tanaka et al.,“High Performance Hardware-Accelerated Flash Key-Value Store,”in The2014Non-volatile Memories Workshop (NVMW),2014. [73]Lingjia Tang et al.,“The impact of memory subsystem resource sharing on datacenter applications,”in Computer Architecture(ISCA),2011 38th Annual International Symposium on,June2011. [74]Arunchandar Vasan et al.,“Worth their watts?-an empirical study of datacenter servers,”in Intl.Symp.on High-Performance Computer Architecture,2010. [75]Nedeljko Vasi′c et al.,“DejaVu:accelerating resource allocation in virtualized environments,”in Proc.of the seventeenth international conference on Architectural Support for Programming Languages and Operating Systems(ASPLOS),London,UK,2012. [76]Christo Wilson et al.,“Better Never Than Late:Meeting Deadlines in Datacenter Networks,”in Proc.of the ACM SIGCOMM2011Confer-ence,ser.SIGCOMM’11.New York,NY:ACM,2011. [77]Carole-Jean Wu et al.,“A Comparison of Capacity Management Schemes for Shared CMP Caches,”in Proc.of the7th Workshop on Duplicating,Deconstructing,and Debunking,vol.15.Citeseer,2008. [78]Yuejian Xie et al.,“PIPP:Promotion/Insertion Pseudo-partitioning of Multi-core Shared Caches,”in Proc.of the36th Annual International Symposium on Computer Architecture,ser.ISCA’09.New York,NY: ACM,2009. [79]Hailong Yang et al.,“Bubble-?ux:Precise Online QoS Management for Increased Utilization in Warehouse Scale Computers,”in Proc.of the40th Annual Intl.Symp.on Computer Architecture,ser.ISCA’13, 2013. [80]Xiao Zhang et al.,“CPI2:CPU performance isolation for shared com- pute clusters,”in Proc.of the8th ACM European Conference on Com-puter Systems(EuroSys),Prague,Czech Republic,2013. [81]Yunqi Zhang et al.,“SMiTe:Precise QoS Prediction on Real-System SMT Processors to Improve Utilization in Warehouse Scale Computers,” in International Symposium on Microarchitecture(MICRO),2014. 五年级上册成语解释及近义词反义词和造句大全 囫囵吞枣；【解释】：囫囵：整个儿。把枣整个咽下去，不加咀嚼，不辨味道。比喻对事物不加分析考虑。【近义词】：不求甚解【反义词】融会贯穿[造句];学习不能囫囵吞枣而是要精益求精 不求甚解；bùqiúshènjiě【解释】：甚：专门，极。只求明白个大概，不求完全了解。常指学习或研究不认真、不深入【近义词】：囫囵吞枣【反义词】：精益求精 造句；1；在学习上，我们要理解透彻，不能不求甚解 2；学习科学文化知识要刻苦钻研，深入领会，不能粗枝大叶，不求甚解。 千篇一律；【解释】：一千篇文章都一个样。指文章公式化。也比喻办事按一个格式，专门机械。 【近义词】：千人一面、如出一辙【反义词】：千差万别、形形色色 造句；学生旳作文千篇一律，专门少能有篇与众不同旳，这确实是平常旳练习太少了。 倾盆大雨；qīngpéndàyǔ【解释】：雨大得象盆里旳水直往下倒。形容雨大势急。 【近义词】：大雨如柱、大雨滂沱【反义词】：细雨霏霏牛毛细雨 造句；3月旳天说变就变，瞬间下了一场倾盆大雨。今天下了一场倾盆大雨。 坚决果断；áobùyóuyù：意思；做事果断，专门快拿定了主意，一点都不迟疑，形容态度坚决 近义词；不假思索斩钉截铁反义词；犹豫不决 造句；1看到小朋友落水，司马光坚决果断地搬起石头砸缸。2我坚决果断旳承诺了她旳要求。 饥肠辘辘jīchánglùlù【近义词】：饥不择食【反义词】：丰衣足食 造句；1我放学回家已是饥肠辘辘。2那个饥肠辘辘旳小孩差不多两天没吃饭了 滚瓜烂熟gǔnguālànshóu〔shú)【解释】：象从瓜蔓上掉下来旳瓜那样熟。形容读书或背书流利纯熟。【近义词】：倒背如流【反义词】：半生半熟造句；1、这篇课文我们早已背得滚瓜烂熟了 流光溢彩【liúguāngyìcǎi】解释；光影，满溢旳色彩，形容色彩明媚 造句：国庆节，商场里装饰旳流光溢彩。 津津有味；jīnjīnyǒuwèi解释：兴趣浓厚旳模样。指吃得专门有味道或谈得专门有兴趣。 【近义词】：兴致勃勃有滋有味【反义词】：索然无味、枯燥无味 造句；1今天旳晚餐真丰富，小明吃得津津有味。 天长日久；tiānchángrìjiǔ【解释】：时刻长，生活久。【近义词】：天长地久【反义词】：稍纵即逝 造句：小缺点假如不立即改掉, 天长日久就会变成坏适应 如醉如痴rúzuìrúchī【解释】：形容神态失常，失去自制。【近义词】：如梦如醉【反义词】：恍然大悟造句；这么美妙旳音乐，我听得如醉如痴。 浮想联翩【fúxiǎngliánpiān解释】：浮想：飘浮不定旳想象；联翩：鸟飞旳模样，比喻连续不断。指许许多多旳想象不断涌现出来。【近义词】：思绪万千 造句；1他旳话让人浮想联翩。2:这幅画饱含诗情，使人浮想联翩，神游画外，得到美旳享受。 悲欢离合bēihuānlíhé解释；欢乐、离散、聚会。泛指生活中经历旳各种境遇和由此产生旳各种心情【近义词】：酸甜苦辣、喜怒哀乐【反义词】：平淡无奇 造句；1人一辈子即是悲欢离合,总要笑口常开,我们旳生活才阳光明媚. 牵肠挂肚qiānchángguàdù【解释】：牵：拉。形容十分惦念，放心不下 造句；儿行千里母担忧，母亲总是那个为你牵肠挂肚旳人 如饥似渴rújīsìkě：形容要求专门迫切，仿佛饿了急着要吃饭，渴了急着要喝水一样。 造句；我如饥似渴地一口气读完这篇文章。他对知识旳如饥似渴旳态度造就了他今天旳成功。 不言而喻bùyánéryù【解释】：喻：了解，明白。不用说话就能明白。形容道理专门明显。 【近义词】：显而易见【反义词】：扑朔迷离造句；1珍惜时刻，好好学习，那个道理是不言而喻旳 与众不同；yǔzhòngbùtóng【解释】：跟大伙不一样。 〖近义词〗别出心裁〖反义词〗平淡无奇。造句； 1从他与众不同旳解题思路中，看出他专门聪慧。2他是个与众不同旳小孩 安然无恙：很平安，没有受到损失和伤害 - 造句：那次智利大地震，许多城市都毁灭了，但我叔父全家安然无恙，非常幸运。 - 拔苗助长：比喻违反事物的发展规律，急于求成，反而坏事 - 造句：“抢先教育”违背了儿童成长的客观规律，这种拔苗助长的办法结果必将造成对孩子身体和心灵的双重伤害。 - 跋山涉水：形容旅途的艰辛劳苦 - 造句：地质勘探队员不怕艰苦，跋山涉水，为祖国寻找地下的报藏。 - 百看不厌：对喜欢的人，事物等看多少遍都不厌倦。比喻非常喜欢。 -造句：到了节日里，各个景区摆设的花朵真是让人百看不厌。 - 班门弄斧：比喻在行家面前卖弄本领，不自量力 -造句：你在著名华文作家的面前卖弄华文,岂不是班门弄斧。 - 搬弄是非：把别人的话传来传去，有意挑拔，或在背后乱加议论，引起纠纷 - 造句：他们到处搬弄是非,传播流言、破坏组织内部的和谐。 - 变本加厉：指比原来更加发展。现指情况变得比本来更加严重 -造句;的坏习惯不但没有改正，反而变本加厉了. -变幻莫测：变化不可测度。变化很多，不能预料 -造句：草地的气候变幻莫测,一会儿烈日当空,一会儿大雨倾盆,忽而雨雪交加,忽而狂风怒吼。 - 别具匠心：指在技巧和艺术方面具有与众不同的巧妙构思- 造句：这篇小说让人看了回味无穷，作者确实是别巨匠心。 -不耻下问：指向地位比自己低、学识比自己少的人请教，也不感到羞耻（耻辱） -造句：学习，不仅要做到虚怀若谷，还要做到不耻下问。 -不可救药：比喻人或事物坏到无法挽救的地步 - 造句：他的问题很严重，已经不可救药。 - 不可思议：原有神秘奥妙的意思。现多指无法想象，难以理解 - 造句：我看这那座小山觉得不可思议。 -不期而遇：没有约定而遇见 -造句：高兴与悲伤总是不期而遇，或许这就是上帝再捉弄世俗吧！ -不屈不挠：形容顽强斗争,在敌人或困难面前不屈服,不低头那种不屈不挠的、要征服一切的心情 -造句：战士们不屈不挠的坚守在抗震第一线。 - 不速之客：指没有邀请而自己来的客人 - 造句：也不必说有时趁你不防钻进防盗铁门登堂入室的不速之客。 - 不屑置辩：认为不值得辩论 - 造句：孔乙己对那些嘲笑他的人显出不屑置辩的神情。 -不言而喻：形容道理很明显 -造句：你想他们这朋友之乐，尽可不言而喻了。 悲惨的近义词反义词和造句 导读：悲惨的近义词 悲凉(注释:悲哀凄凉：～激越的琴声。) 悲惨的反义词 幸福(注释:个人由于理想的实现或接近而引起的一种内心满足。追求幸福是人们的普遍愿望，但剥削阶级把个人幸福看得高于一切，并把个人幸福建立在被剥削阶级的痛苦之上。无产阶级则把争取广大人民的幸福和实现全人类的解放看作最大的幸福。认为幸福不仅包括物质生活，也包括精神生活;个人幸福依赖集体幸福，集体幸福高于个人幸福;幸福不仅在于享受，而主要在于劳动和创造。) 悲惨造句 1.一个人要发现卓有成效的真理，需要千百个人在失败的探索和悲惨的错误中毁掉自己的生命。 2.贝多芬的童年尽管如是悲惨，他对这个时代和消磨这时代的地方，永远保持着一种温柔而凄凉的回忆。 3.卖火柴的小女孩在大年夜里冻死了，那情景十分悲惨。 4.他相信，他们每个人背后都有一个悲惨的故事。 5.在那次悲惨的经历之后，我深信自己绝对不是那种可以离家很远的人。 6.在人生的海洋上，最痛快的事是独断独航，但最悲惨的却是回头无岸。 7.人生是艰苦的。对不甘于平庸凡俗的人那是一场无日无夜的斗 争，往往是悲惨的、没有光华的、没有幸福的，在孤独与静寂中展开的斗争。……他们只能依靠自己，可是有时连最强的人都不免于在苦难中蹉跎。罗曼·罗兰 8.伟大的心胸，应该表现出这样的气概用笑脸来迎接悲惨的厄运，用百倍的勇气来应付开始的不幸。鲁迅人在逆境里比在在顺境里更能坚强不屈。遇厄运时比交好运时容易保全身心。 9.要抓紧时间赶快生活，因为一场莫名其妙的疾病，或者一个意外的悲惨事件，都会使生命中断。奥斯特洛夫斯基。 10.在我一生中最悲惨的一个时期，我曾经有过那类的想法：去年夏天在我回到这儿附近的地方时，这想法还缠着我;可是只有她自己的亲自说明才能使我再接受这可怕的想法。 11.他们说一个悲惨的故事是悲剧，但一千个这样的故事就只是一个统计了。 12.不要向诱惑屈服，而浪费时间去阅读别人悲惨的详细新闻。 13.那起悲惨的事件深深地铭刻在我的记忆中。 14.伟大的心胸，应该用笑脸来迎接悲惨的厄运，用百倍的勇气来应付一切的不幸。 15.一个人要发现卓有成效的真理，需要千百万个人在失败的探索和悲惨的错误中毁掉自己的生命。门捷列夫 16.生活需要爱，没有爱，那些受灾的人们生活将永远悲惨;生活需要爱，爱就像调味料，使生活这道菜充满滋味;生活需要爱，爱让生活永远充满光明。 常用成语造句大全及解释 导读：常用成语及造句大全： 【马到功成】形容事情顺利，一开始就取得胜利。 朋友要去参加考试我发自内心的祝她马到功成。 【安常守故】习惯于日常的平稳糊口，保保守的一套。指保守不知厘革。 他持久以来安常守故，缺乏锻炼，故而经不起挫折。 【挨门逐户】挨家挨户，一家也没有遗漏。 倾销员挨门逐户地倾销产物，可是并没有多少人愿意买 【破釜沉舟】比喻下决心悍然不顾地干到底。 只要咱们有破釜沉舟的决心，就能克服进修上的各类困难。 【大千世界】三昧，世界的千倍叫小千世界，小千世界的千倍叫中千世界，中千世界的千倍叫大千世界。后指广大无边的人世。 大千世界无奇不有，充满了抵牾。 【空手发迹】一切靠自己艰辛奋斗，创立了一番事业。 王董事长空手发迹，如今是王氏集团的总裁。 【卷土重来】卷土：人马奔跑时卷起的尘土。形容失败后组织力量，重图恢复。 这次角逐虽则表现不佳，但我决定明天卷土重来。 【晨钟暮鼓】古代梵宇中晨敲钟，暮伐鼓以报时，用以使人惊悟的言论。 每当我想坏事时，就会想起母亲对我的晨钟暮鼓的叮咛。 【力争上游】起劲争取长进求学做人都要力争上游，不要自满於近况 【投笔从戎】一小我私家抛弃文职而插手卫国的行列。 大伯父高中结业后投笔从戎，步入军校就读，负起保家卫国的责任。 【前车之鉴】比喻前人的失败，可以作为后人的借鉴。 有了这个前车之鉴，我下次出门一定会带雨具。 【金石为开】至诚可感动任何事物。 表哥相信精诚所至，金石为开，他的成意一定可以感动王小姐的。 【勤能补拙】指勤勉起劲能弥补天资上的不足。 勤能补拙，只要你多付出心思，一定有乐成的机会。 【揠苗助长】揠：拔起。把苗拔起，以助其生长。后用来比喻违反事物的发展规律，操之过急，反倒坏事。 对学生的教育既不能揠苗助长，也能任其自然。 【闻鸡起舞】听见鸡鸣就起身，比喻人发奋勇前进修，励精图治。 老爷爷在乡间修养，天天闻鸡起舞，打太极健身。 【哀鸿遍野】哀鸿：哀鸣的鸿雁。比喻饥寒交迫的灾民。比喻在天灾人祸中到处都是流离失所、呻吟呼号的饥民。 旧社会，每逢水灾战乱，人民就被迫四处逃亡，～，一片凄凉。 【联袂登台】同台演出 今晚的演出听说会有两位名角联袂登台献艺。 【使人咋舌】令人惊讶 初二上语文期末复习之成语运用专项训练 1.恍如隔世：恍惚如同相隔了一辈子。喻事物变化发展很快，变化很大。 例句：他曾因躲避战乱而隐居深山，和平后下山才发觉人间变化巨大，真是感觉恍如隔世。 2.轻描淡写：原意是绘画时用浅淡颜色轻轻描绘。形容说话或作文章时对重要的地方淡淡带过。也比喻做事不费力。 例句：医生未说实话，看他可怜，就用轻描淡写的话安慰他。 如此严重的问题，轻描淡写地说了说就算完了？ 3.与狼共舞：跟恶狼一起跳舞，比喻跟所谋求的对象有利害冲突,决不能成功。后多指跟恶人商量,要他牺牲自己的利 益,一定办不到。也说比喻与恶人在一起，随时都有危险，须特别谨慎。 例句：没有比与狼共舞更危险的事情。江山易改，本性难移。狼就是狼，无论外表装饰得如何漂亮，不管如何能言善辩，狼就是狼，早晚会露出凶残、狡猾的真面目。 4.林林总总：林林:树木聚集成片的样子;总总:全部汇集状。形容人或事物繁多。 例句：在林林总总的这类故事中，也有一个是说鲁班学习海龙王宫殿的建筑艺术。 5.不速之客：速:邀请。没有邀请而自己来的客人。指意想不到的客人。 例句：我们正在聚餐时来了一位不速之客。 多年不见的老朋友，突然出现在我的眼前，真是不速之客。 大会主席团作出决定，会议期间不准无故迟到，对那些不遵守大会纪律的不速之客要通报批评。（不是突然来到的人。语意不符） 6.责无旁贷：自己的责任，不能推卸给别人；贷，推卸。 例句：杨丽萍在接受媒体采访时说，保护洱海，她责无旁贷，同时她欢迎广大媒体、网友监督。 作为课题负责人，他责无旁贷地走向主席台。（用“当仁不让”更合适，语意不符） 7.如火如荼：像火那样红，像荼（茅草的白花）那样白。原比喻军容之盛，现用来形容旺盛、热烈或激烈。 例句：过了两年“五四运动”发生了。报纸上的如火如荼的记载唤醒了他的被忘却了的青春。 8.鳞次栉比：形容房屋密集，像鱼鳞和梳子的齿一样，一个挨着一个地排列着。 例句：鳞次栉比的摩天大楼在霍尔河畔奇迹般地崛起，让人以为自己仿佛到了纽约。 包头是一个倚山濒水的城市，向北是峰峦鳞次栉比的阴山，向南是波涛汹涌、曲折回环的黄河。（多用来形容房屋或船只等排列得很密很整齐。不能用来形容“山”。） 9.珠光宝气：闪耀着珍宝的光色。多形容妇女服饰华贵富丽。 例句：温德姆大堂的香氛过于浓烈，像一个珠光宝气、香水味四溢的贵妇人如影随行。 10.退避三舍：比喻对人让步，不与相争。 例句：可见二位仁兄的学问，不但本校众人所不能及，即使天下文才，也当退避三舍哩! 这个地头蛇，人见人怕，连警察见了他都要退避三舍。（“退避三舍”比喻对人让步，不与之相争；不是形容害怕、恐怖。） 11.司空见惯：看惯了的事情，并不觉得奇怪。 例句：这种种行为，在我们初来的东方人看来，多少存着好奇心和注意的态度，但在他们已司空见惯了。 我们都司空见惯了那种“违者罚款”的告示牌。（“司空见惯”的意思是形容经常看到不足为奇的事物。后面不能带宾语。） 12.月白风清：形容月夜的明朗幽静。 例句：这是一个月白风清的良宵，校园散步的人三三两两从我身边走过。 13.脍炙人口：本指美味人人都爱吃，现比喻好的诗文或事物人人都称赞。 例句：我国古典诗歌内蕴丰富，很能激发人们的联想和想象。“日出江花红胜火，春来江水绿如蓝”，吟咏这脍炙人口的诗句，谁不为春回大地后祖国母亲多姿多彩的面貌而自豪。 历史是一面脍炙人口的镜子，面对它，我会正衣冠，知兴替，明得失。（指美味人人爱吃。比喻好的诗文受到人们和称赞和传讼。不能修饰“镜子”） 14.鲁殿灵光：汉代鲁恭王建有灵光殿，屡经战乱而岿然独存。后用来称硕果仅存的人或事物。 群芳吐艳：各种花草树木竞相开放出艳丽的花朵。比喻各种不同形式和风格的艺 术自由发展。也形容艺术界的繁荣景象。 什么花儿都争着开放.形容春意盎然的样子 造句—花展上的鲜花竞相开放，真是群芳吐艳，美不胜收啊！ 春天悄然而至，走到花园里，各种花群芳吐艳。 姹紫嫣红：姹：美丽。嫣：美好。形容各种花娇艳美丽 春天来了，处处姹紫嫣红，花儿们争先恐后的开放着。 满园的鲜花，争相怒放，一片姹紫嫣红、生机勃勃的景象，令人流连忘返。 郁郁葱葱的草丛中，一些姹紫嫣红的花朵煞是好看，大有百花争艳的气势。 落英缤纷：落英：落花。缤纷：繁多凌乱的样子。形容落花纷纷飘落的美丽情景。 这里山清水秀，落英缤纷，景色迷人。 傍晚的梅花林，斜阳西坠，落英缤纷，实在太美了。 鲜花盛开，花瓣纷纷飘落。形容春天的美好景色。也指花儿凋谢的暮春天气。 郁郁葱葱：形容草木苍翠茂盛的样子。也形容气势美好蓬勃。 山道两旁到处郁郁葱葱，风景美不胜收。 细雨滋润过后的草地绿意盎然郁郁葱葱。 那片树木葱葱郁郁的，十分茂盛。 夏日的池塘里,郁郁葱葱的荷叶映衬着亭亭玉立的荷花,真是一幅美丽的画卷啊! 那片树木葱葱郁郁的，十分茂盛。 喷薄欲出：喷薄：有力地向上涌的样子。形容水涌起或太阳初升时涌上地平线 的景象。 鸟儿唱着欢乐的歌，迎接着喷薄欲出的朝阳。 .我们来到了层峦叠翠郁郁葱葱的山腰.喷薄欲出的红日照耀着苍翠欲滴的古松。 旭日东升：旭日:早上刚出来的太阳．早上太阳从东方升起．形容朝气蓬勃的 气象． 在这个旭日东升晴朗的早晨，让我们一起来热情的做早操吧！ 中国正如旭日东升般，充满活力。 她们黑暗中挣扎和最后看到了旭日东升，破涕为笑，对未来充满希望。 知己的近义词反义词及知己的造句 本文是关于知己的近义词反义词及知己的造句，感谢您的阅读！ 知己的近义词反义词及知己的造句知己 基本解释：顾名思义是了解、理解、赏识自己的人，如"知己知彼，百战不殆";更常指懂你自己的挚友或密友，它是一生难求的朋友，友情的最高境界。正所谓:"士为知己者死"。 1.谓了解、理解、赏识、懂自己。 2.彼此相知而情谊深切的人。 【知己近义词】 亲信，好友，密友，心腹，挚友，深交，相知，知交，知友，知心，知音，石友，老友，至友 【知己反义词】 仇人敌人陌路 【知己造句】 1、我们想要被人爱、想拥有知己、想历经欢乐、想要安全感。 2、朋友本应是我们的亲密知己和支持者，但对于大多数人来说，有一些朋友比起帮助我们，更多的却是阻碍。 3、那么，为什么你就认为，随着年龄的增长，比起女人来男人们的知己和丰富的人际关系更少，因此一般容易更孤独呢？ 4、他成了我的朋友、我的知己、我的顾问。 5、无论在我当州长还是总统的时候，布鲁斯都是我的密友、顾问和知己。他这样的朋友人人需要，也是所有总统必须拥有的。 6、波兰斯基有着一段声名卓著的电影生涯，也是几乎所有电影界重要人物们的挚友和同事，他们是知己，是亲密的伙伴。 7、搜索引擎变成了可以帮追我们的忏悔室，知己，信得过的朋友。 8、这样看来，奥巴马国家安全团队中最具影响力的当属盖茨了――但他却是共和党人，他不会就五角大楼以外问题发表看法或成为总统知己。 9、我们的关系在二十年前就已经和平的结束了，但在网上，我又一次成为了他精神层面上的评论家，拉拉队，以及红颜知己。 10、这位“知己”，作为拍摄者，站在距离电视屏幕几英尺的地方对比着自己年轻版的形象。 11、父亲与儿子相互被形容为对方的政治扩音筒、知己和后援。 12、这对夫妻几乎没有什么至交或知己依然在世，而他们在后纳粹时期的德国也不可能会说出实话的。 13、她把我当作知己，于是，我便将她和情人之间的争吵了解得一清二楚。 14、有一种友谊不低于爱情；关系不属于暖昧；倾诉一直推心置腹；结局总是难成眷属；这就是知己！ 15、把你的治疗师当做是可以分享一切心事的知己。 16、莉莉安对我敞开心胸，我成了她的知己。 17、据盖洛普民意调查显示，在那些自我认同的保守党人中，尽管布什仍维持72%支持率，但他在共和党领导层中似乎很少有几位知 成语大全解释及造句 本文是关于成语大全解释及造句，感谢您的阅读！ 首尾乖互：相互违违，先后自相矛盾。 你这人说话如许首尾乖互，叫我怎么信任你。 引颈受戮：戮：杀。伸长脖子等待被杀。指不作抵当而等死。 犯人知道自己罪责难逃，只有引颈受戮了。 衣不曳地：曳，拖动。时装不沾地，比喻非常繁忙。 这几天工作太多了，他都已经衣不曳地了。 水洁冰清：像冰水同样洁白清净。形容人品高洁或文笔雅致。 她是如许一个水洁冰清的人，怎么会和那一些坏人在一起。 如鲠在喉：鱼骨头卡在咽喉里。比喻生理有话没有说出来，非常难受。 我知道真象，但又不能说出来，真是如鲠在喉呀。 仰事俯畜：上要侍奉怙恃，下要养活妻儿。泛指维持一家生活。 父亲天天起早摊黑，仰事俯畜，非常辛苦。 为恶不悛：对峙作歹，不愿悔改。 这个报酬恶不悛，终究要受到法律的制裁。 无补于时：对事情没有什么益处。 此刻不努力学习，等未来悔怨了也无补于时了。 有脚书厨：戏称记闻精确、常识渊博的人。 他好学善思、常识博识，真是个有脚书厨呀 一字一珠：一个字就像一颗真珠。形容歌声婉转圆润。也比喻文 章优美、辞藻华美。 如许的文章真是一字一珠，看上几遍，还是回味无穷。 1、一拥而入：体育运动场的大门刚一打开，迷球的人们就一拥而入。 2。肝火冲冲：冲冲：感情冲动的样子。形容非常生气。《不知为何，他肝火冲冲地走进了教室。》 3。目不转睛：聚、会：堆积。形容注意力非常集中。《教室里同窗们都在目不转睛地听老师讲课。》 4。喃喃自语：自己和自己说话。《在路口经常可以看到一个老人，坐在那里喃喃自语。》 5。千钧一发：形容情况十分求助紧急。《就在要撞车的千钧一发的时候，司机紧迫刹住了汽车。》 6。精兵简政：简：使简化。《有些单位必需精兵简政，不断提高工作效率。》 7。五光十色：《一到夜晚，五光十色的灯把厦门装扮患上更加美丽。》 8。雨后春天的竹笋：形容新生物质大量涌现。《鼎新开放以来，工厂如雨后春天的竹笋般地不断涌现。》 9。满目琳琅：比喻面前出现了许多精美的物质。《那里展出的的工具满目琳琅，使我们的秋水应接没时间。》 10。顶天登时：形容形象非常高大，气概豪爽。《他至公无私，真是一个堂堂正正、顶天登时的男子汉。》 成语解释加造句 【篇一：成语解释加造句】 82 、历历在目：指远方的景物看得清清楚楚,或过去的事情清清楚楚地重现在眼前. 例如：上海世博会虽然已经过去两三个月了,但参观世博会的情景仍然历历在目. 83 、力挽狂澜：比喻尽力挽回危险的局势. 在这次比赛中,我们队暂时落后,他力挽狂澜,终于战胜了所有对手,使我们队获得了冠军. 84 、理直气壮：理由充分,说话气势就壮. 例如：这件事上你没有错,你可以理直气壮地找小明理论理论. 85 、两全其美：指做一件事顾全到双方,使两方面都得到好处. 例如：我们一起把英语课文演一遍吧,这样我们既可以练英语,又可以轻松轻松,这不是两全其美吗? 86 、流离失所：流离：转徒离散.无处安身,到处流浪. 例如：在世界上许多战乱地区,人民流离失所. 87 、流连忘返：玩乐时留恋不愿离开.留恋得忘记了回去. 例如：桂林山水如此秀美,真令人流连忘返. 88 、络绎不绝：形容行人车马来来往往,接连不断. 例如：这个画展非常好看,参观的人络绎不绝. 89 、买椟还珠：椟：木匣；珠：珍珠.买下木匣,退还了珍珠.比喻没有眼力,取舍不当. 例如：如果我是你,我就不会买这些华而不实的东西,宁愿要几本书,因为我不喜欢买椟还珠. 90 、漫不经心：漫：随便.随随便便,不放在心上. 例如：做事可不能漫不经心,要认认真真,只有这样才能把事情做好. 91 、满载而归：装得满满地回来.形容收获很大. 例如：爷爷奶奶逛玩早市,满载而归,买了很多菜才回来. 92 、 茅塞顿开： 顿： 立刻； 茅塞： 喻人思路闭塞或不懂事. 比喻思想忽然开窍, 立刻明白了某个道理. 例如：老师一讲这个道理,我好像茅塞顿开,忽然明白了很多事情. 93 、诚心诚意：诚：真实的心意.真心诚意.形容十分真挚诚恳. 造句——在了解实情之后,她懊悔万分,诚心诚意地向孩子道歉. 94 、触景生情：受到眼前景物的触动,引起联想,产生某种感情.触：触动.造句——傍晚,看到天边的火烧云,使我触景生情,想起了许多往事! 95 、触类旁通：触类：接触某一方面的事物；旁通：相互贯通.掌握了某一事物的知识或规律,进而推 知同类事物的知识或规律. 造句——这让我掌握了一些生活的方法,能触类旁通地思索问题. 96 、唇亡齿寒：嘴唇没有了,牙齿就会感到寒冷.比喻关系密切,利害相关.造句——这两家公司有着多种业务关系,正所谓是唇亡齿寒. 97 、春华秋实：春天开花,秋天结果（多用于比喻） 造句——春华秋实,没有那浩荡的春风,又哪里会有这满野秋色和大好的收成呢? 98 、当之无愧：当：承当.无愧：毫无愧色.当得起某种称号或荣誉,无须感到惭愧. 造句——他搞了几项技术革新,被誉为革新大师,真是当之无愧! 99 仿照的近义词反义词和造句 仿照的近义词 【仿制解释】：仿造：～品 【模仿解释】：个体自觉或不自觉地重复他人的行为的过程。是社会学习的重要形式之一。尤其在儿童方面，儿童的动作、语言、技能以及行为习惯、品质等的形成和发展都离不开模仿。可分为无意识模仿和有意识模仿、外部模仿和内部模仿等多种类型。 仿照的反义词 【独创解释】：独特的创造：～精神ㄧ～一格。 仿照造句 一、老师让我们仿照黑板上的图画一幅画。 二、仿照下面的句式，以“只有”开头，写一结构与之相似的复句。 三、仿照例句的句子，在下面两句的横线上补写相应的内容。 四、仿照例句，以“记忆”或“友情”开头，另写一句话。 五、仿照下面两个例句，用恰当的词语完成句子，要求前后语意关联。 六、仿照开头两句句式，通过联想，在后面两句横线上填上相应的词语。 七、仿照所给例句，用下面的词展开联想，给它一个精彩的解释。 八、仿照例句，以“你”开头，另写一个句子。 九、仿照下列句式，续写两个句子，使之与前文组成意义相关的.句子。 十、我们也仿照八股文章的笔法来一个“八股”,以毒攻毒,就叫做八大罪状吧。 十一、仿照例句，任选一种事物，写一个句子。 十二、仿照下面一句话的句式和修辞，以“时间”开头，接着写一个句子。 十三、仿照例句，以“热爱”开头，另写一句子。 十四、仿照下面的比喻形式，另写一组句子。要求选择新的本体和喻体，意思完整。 十五、根据语镜，仿照划线句子，接写两句，构成语意连贯的一段话。 十六、仿照下面句式，续写两个句式相同的比喻句。 十七、自选话题，仿照下面句子的形式和修辞，写一组排比句。 十八、仿照下面一句话的句式，仍以“人生”开头，接着写一句话。 十九、仿照例句的格式和修辞特点续写两个句子，使之与例句构成一组排比句。 二十、仿照例句，另写一个句子，要求能恰当地表达自己的愿望。 二十一、仿照下面一句话的句式，接着写一句话，使之与前面的内容、句式相对应，修辞方法相同。 二十二、仿照下面一句话的句式和修辞，以“思考”开头，接着写一个句子。 二十三、仿照下面例句，从ABCD四个英文字母中选取一个，以”青春”为话题，展开想象和联想，写一段运用了比喻修辞格、意蕴丰富的话，要求不少于30字。 二十四、仿照下面例句，另写一个句子。 二十五、仿照例句，另写一个句子。 二十六、下面是毕业前夕的班会上，数学老师为同学们写的一句赠言，请你仿照它的特点，以语文老师的身份为同学们也写一句。 成语眼花缭乱的意思解析及造句 眼花缭乱形容眼睛看见复杂纷繁的东西而感到迷乱。比喻事物复杂，无法辨清。出自元·王实甫《西厢记》：“则著人眼花缭乱口难开;魂灵儿飞在半天。” 基本信息 【词目】眼花缭乱 【拼音】yǎn huā liáo luàn 【近义词】目不暇接、头昏眼花、扑朔迷离。 【反义词】一目了然、清晰可数、清晰可辨。 【语法】中性词，主谓式。作谓语、定语、补语。 【基本解释】形容眼睛看见复杂纷繁的东西而感到迷乱。比喻事物复杂，无法辨清。 详细解释 含义 缭乱：纷乱。形容眼前的景象复杂纷繁,使人感到迷乱不清。也比喻事物复杂，无法辨清。 出处 元·王实甫《西厢记》第一本第一折：“似这般可喜娘的庞儿罕曾见，只教人眼花缭乱口难言，魂灵儿飞在半天了。” 选自沈石溪《斑羚飞渡》：“紧接着一对对斑羚凌空跃起，在山 1 间上空画出一道道令人眼花缭乱的弧线。” 示例 匡大被他这一番话说得眼花缭乱，浑身都酥了，一总都依他说。 (清·吴敬梓《儒林外史》第二十回) 【其他】出处元·王实甫《西厢记》第一本第一折：“似这般可喜娘的庞儿罕曾见，只教人眼花缭乱口难言，魂灵儿飞在半天。”示例其余这些国王，除了我们到过的，内中许多奇形怪状，小弟看来看去，只觉～，辨不明白。——清·李汝珍《镜花缘》第三十八回老母猪逛花园——眼花缭乱;刘姥姥进大观园——眼花缭乱这些蔬菜看得我眼花缭乱。紧接着，一对对斑羚凌空跃起，在山涧上空画出一道道令人眼花缭乱的弧线。——《斑羚飞渡》充满乐趣的迷宫，令人眼花缭乱——语文书第九课《冰城》 辨析 “眼花缭乱”常误写为“眼花撩乱”和“眼花瞭乱”，应注意。根据中国造字原理，眼花缭乱的“缭”字是指丝线缠绕在一起令人难以分辨。所以是绞丝旁。 成语接龙 乱语胡言→ 言近意远→ 远亲近邻→ 邻女詈人→ 人离乡贱→ 贱入贵出→ 出师无名→ 名标青史→ 史不絶书→ 书香门户→ 户限为穿→ 穿花蛱蝶→ 蝶恋蜂狂→ 狂涛巨浪→ 浪酒闲茶→ 茶余酒后→ 后起之秀→ 秀才造反→ 反躬自责→ 责实循名→ 名震一时→ 时运不齐→ 齐心一力→ 力小任重→ 重温旧业 暗示的近义词和反义词[暗示的近义词反义词和造句] 【暗示解释】：用含蓄、间接的方式使他人的心理、行为受到影响。下面小编就给大家整理暗示的近义词，反义词和造句，供大家学习参考。 暗示近义词 默示 示意 暗指 暗意暗示反义词 明说 表明 明言暗示造句 1. 他的顶级助手已经暗示那可能就在不久之后，但是避免设定具体的日期。 2. 一些观察家甚至暗示他可能会被送到了古拉格。 3. 要有积极的心理暗示，达成目标的好处不够充分，画面不够鲜明那它对你的吸引力就不够强烈，你就不易坚持下去! 4. 不要经常去试探男人，更不要以分手做为威胁，当你经常给他这种心理暗示，他的潜意识就会做好分手的打算。 5. 向读者暗示永远不必为主角担心，他们逢凶化吉，永远会吉人天相。 6. 约她一起运动，不要一下跳跃到游泳，可以从羽毛球网球开始，展示你的体魄和运动细胞，流汗的男人毫无疑问是最性感的，有意无意的裸露与靠近，向她暗示你的运动能力。 7. 正如在上一个示例所暗示的，只有在这些对象引用内存中同一个对象时，它们才是相同的。 8. 渥太华此时打出中国牌，巧妙地对美国这种行为作出警告，暗示美国如果长期这样下去的话，美国将自食其果。 9. 团长用震撼人心的嗓音喊道，这声音对他暗示欢乐，对兵团暗示森严，对前来校阅阅兵的首长暗示迎迓之意。 10. 渥太华此时打出中国牌，巧妙地对美国这种行为作出警告，暗示美国如果长期这样下去的话，美国将自食其恶果。 11. 我们需要邀请用户，为他们描述服务产品有多少好，给他们解释为什么他们需要填那些表单并且暗示他们会因此得到利益的回报。 12. 她对暗示她在说谎的言论嗤之以鼻。 14. 在力士参孙中，整首诗都强烈暗示着弥尔顿渴望他自己也能像参孙一样，以生命为代价，与敌人同归于尽。 15. 戴维既然问将军是否看见天花板上的钉子，这就暗示着他自己已看见了，当将军做了肯定的答复后，他又说自己看不见，这显然是自相矛盾。 16. 自我暗示在我们的生活中扮演着重要角色。如果我们不加以觉察，通常它会与我们作对。相反地，如果你运用它，它的力量就可以任你使唤。 17. 纳什建议太阳招兵买马，暗示去留取决阵容实力。 18. 同时，还暗示了菊既不同流俗，就只能在此清幽高洁而又迷漾暗淡之境中任芳姿憔悴。 19. 学习哲学的最佳途径就是将它当成一个侦探故事来处理：跟踪它的每一点蛛丝马迹每一条线索与暗示，以便查出谁是真凶，谁是英雄。 20. 不要经常去试探你的伴侣，更不要以分手做为威胁，试探本身就是一种不信任，当 高中成语及解释及造句 1、不刊之论：刊：削除。指不可磨灭和不可改动的言论。高中成语及解释及造句 2、文不加点：点：涂上一点，表示删除。写文章不用涂改就很快写成。形容文思敏捷。 3、目不识丁：丁：简单的汉字。很少受过教育的或没有受过教育，尤指不能读、写。 4、功败垂成：垂：接近，将要。功业即将取得成就之时突然失败了。 5、身体力行：体：体验。亲身体验，努力实践。 6、博闻强识：闻，学识。识:记忆。见闻广博，记忆力强。 7、文过饰非：饰：遮掩。明知有过错而故意隐瞒掩饰。 8、方兴未艾：方:正当。艾:停止。刚兴起尚未停止，形容形势或事物正在蓬勃发展. 9、言归于好：言：句首助字，无意义。保持友谊，重新成为好朋友。 10、韦编三绝：韦：熟牛皮。形容好学不倦，勤奋用功。 11、莫衷一是：衷：折中，决断。意见分歧，难有一致的定论。 12、莫逆之交：逆：抵触。非常要好的朋友。 13、否极泰来：否：坏运气。泰：好运气。坏运气到了尽头，好运气就会到来。 14、相形见绌：绌：不够，不足。互相比较之下，一方显得很逊色。 15、汗流浃背：浃：湿透。形容满身大汗。亦形容万分恐惧或惭愧。 16、不速之客：速：邀请。未经邀请而自己来的客人。 17、百端待举：端：项目。许多事情都有待安排和进行，比喻事业处在初创阶段。 18、暴殄天物：殄：灭绝。任意糟蹋东西。 19、不共戴天：戴：加在头上或用头顶者。不愿与仇人共生世间，比喻仇恨极深。 20、不即不离：即：接近，靠近。佛教用语，不接近亦不远离(指人际关系)。 21、不名一钱：名：占有。形容极其贫穷，1个钱也没有。 22、不足为训：训：法则，典范。不能当作范例或法则不足信。 23、不可名状：名：说出。无法用言语来形容。高中成语及解释及造句 24、鬼使神差：差：差遣。有鬼使唤，有神差遣。比喻人做事在不自觉之中投入。 25、差强人意：差：稍微。还算能振奋心意。比喻大致令人满意。 26、等量齐观：量：衡量，估量。指把不相同的事物用同一标准来衡量，一样对待。 27、洞烛其奸：洞：深入，透彻。烛：照亮。形容看透对方的阴谋诡计。 28、阿其所好：阿：曲从。曲从别人的意图，迎合别人的喜爱。 29、耳濡目染：濡：沾染。形容听得多了，见得多了，自然而然受到 放弃的近义词反义词和造句 下面就给大家整理放弃的近义词，反义词和造句，供大家学习参考。 放弃的近义词【废弃解释】：抛弃不用：把～的土地变成良田ㄧ旧的规章制度要一概～。 【丢弃解释】：扔掉;抛弃：虽是旧衣服，他也舍不得～。 放弃的反义词【保存解释】：使事物、性质、意义、作风等继续存在，不受损失或不发生变化：～古迹ㄧ～实力ㄧ～自己，消灭敌人。 放弃造句(1) 运动员要一分一分地拼，不能放弃任何一次取胜的机会。 (2) 我们不要放弃每一个成功的机会。 (3) 敌军招架不住，只好放弃阵地，狼狈而逃。 (4) 为了农村的教育事业，姐姐主动放弃了调回城市的机会。 (5) 都快爬到山顶了，你却要放弃，岂不功亏一篑?(6) 纵使遇到再大的困难，我们也要勇往直前，不轻言放弃。 (7) 逆境中的他始终没有放弃努力,仍满怀信心,期待着峰回路转的那一天。 (8) 听了同学的规劝,他如梦初醒,放弃了离家出走的想法。 (9) 因寡不敌众,我军放弃了阵地。 (10) 要日本帝国主义放弃侵华野心,无异于与虎谋皮。 (11) 永不言弃固然好，但有时放弃却也很美。 (12) 他这种放弃原则、瓦鸡陶犬的行径已经被揭露出来了。 (13) 适当放弃，做出斩钉截铁的决定，才能成为人生的赢家。 (14) 他委曲求全地放弃自己的主张，采纳了对方的意见。 (17) 我们要有愚公移山一样的斗志，坚持不懈，永远不放弃，去登上梦想的彼岸!(18) 只要有希望，就不能放弃。 (19) 为了大局着想，你应该委曲求全地放弃自己的看法。 (20) 既然考试迫在眉睫，我不得不放弃做运动。 (21) 即使没有人相信你，也不要放弃希望。 (22) 无论通往成功的路途有多艰辛，我都不会放弃。 (23) 在困难面前，你是选择坚持，还是选择放弃?(24) 无论前路多么的漫长，过程多么的艰辛，我都不会放弃并坚定地走下去。 (25) 你不要因为这点小事就英雄气短,放弃出国深造的机会。 (26) 像他这样野心勃勃的政客，怎么可能放弃追求权力呢?(27) 鲁迅有感于中国人民愚昧和麻木,很需要做发聋振聩的启蒙工作,于是他放弃学医,改用笔来战斗。 (28) 我们对真理的追求应该坚持不懈，锲而不舍，绝不能随便放弃自己的理想。 (29) 感情之事不比其他，像你这样期盼东食西宿，几个男友都捨不得放弃，最后必定落得一场空。 (30) 爷爷临终前的话刻骨铭心，一直激励着我努力学习，无论是遇到多大的困难险阻，我都不曾放弃。 成语及解释造句 爱不释手：形容对某种东西喜爱得舍不得放下。《西游记》这本书有趣极了，叫人看了～。 爱屋及乌：比喻爱一个人连带地爱与他有关系的人或事物。祖父最疼爱大哥，～，连带也把大哥的女儿当心肝宝贝。 按部就班：指依着一定的次序，按着一定的步骤来计划行事。你不肯～地学习，老是想一步登天，难怪最后什么也学不成。 百年树人：比喻培养人才是长久大计。教育是～的大计，各国政府都投下大量人力、财力在教育事业上。 百折不挠：形容意志坚强，遇到任何挫折都不退缩或屈服。阿辉凭着～的精神，历经多次挫折，终于开创了自己的事业。 班门弄斧：比喻不自量力，在专家或行家面前卖弄本事。有时用来表示自谦。我这篇小说，在各位作家面前，只是～之作，还请大家多指教。 半途而废：指事情做到一半便停止。做任何事一定要有恒心，千万不可～。 包罗万象：内容丰富，形形色色，应有尽有。人们都爱到这家购物中心购物，因为这里的货物～，应有尽有。 闭门造车：比喻人的行为与现实情况脱节。作家想要写一部好作品，就不能～，必须去体验生活。 变本加厉：比喻变得比原来更加厉害。小文时常偷同学的钱，后来还～，偷老师的钱，结果被学校开除了。 标新立异：作出新奇、独特的事，和别人不一样。她为了表现自己，整天在装扮上～，浪费了不少时间和金钱。 别出心裁：表示新奇的、与众不同的手法。李明设计的服装～，在本地服装界十分突出，所以很受消费者欢迎。 宾至如归：形容招待客人周到、殷勤，使客人感到温暖。这家酒店服务周到，让人有～之感。不耻下问：指乐意向那些学问、地位比自己低的人请教与学习，一点也不觉得羞耻。老师虚心地向一位工友请教，同学都很敬佩她这种～的精神。 不攻自破：形容说话靠不住，不用攻击，自己就站不住脚了。当我们以行动证明自己的行为时，谣言便～了。事实是客观存在的，谎言在事实面前将～。 不可救药：比喻情况严重到无法挽救。【无可救药】大财不但抽烟赌博，现在还染上毒瘾，简直～了。 不可思议：形容无法想象或很难理解。他蒙住眼睛，也能画出这么美丽的山水画，真是～。 不劳而获：不劳作而占有别人劳动的成果。你这个懒惰虫老是想～，我才不让你得逞！ 不务正业：指不从事正当的职业。大哥自从退学后就～，整天在街头游荡，把爸爸气得差点断气。 不省人事：指昏迷过去，什么也不知道。宇平被车撞倒后就～，司机吓得连忙电召救护车。 不遗余力； 1一夫当关，万夫莫开:意思是山势又高又险，一个人把着关口，一万个人也打不进来。形容地势十分险要。示例:剑阁峥嵘而崔嵬，～。 2草木皆兵:把山上的草木都当做敌兵。形容人在惊慌时疑神疑鬼。 示例:这一天大家都是惊疑不定，～，迨及到了晚上，仍然毫无动静。 3赤膊上阵 光着膀子上阵。比喻亲身上场，不加掩饰地进行活动。 示例:他终于～，亲自出马了。 4乌合之众 释义:象暂时聚合的一群乌鸦。比喻临时杂凑的、毫无组织纪律的一群人。 示例:外边虽有些人，也是～，不相统摄。 :5打草惊蛇 释义:原比喻惩甲菟乙。后多比喻做法不谨慎，反使对方有所戒备。 示例:空自去“～”，倒吃他做了手脚，却是不好。 6四面楚歌 释义:比喻陷入四面受敌、孤立无援的境地。 示例:在这～里，凭你怎样伶牙俐齿，也只得服从了 :7用兵如神 释义:调兵遣将如同神人。形容善于指挥作战。 示例:三国时期的诸葛亮神机妙算，～。 :8有勇无谋 释义:只有勇气，没有计谋。指做事或打仗只是猛打猛冲，缺乏计划，不讲策略。示例:毕丰～，极贪酒色，不恤下人，喽罗尽皆离心。 9声东击西 释义:声：声张。指造成要攻打东边的声势，实际上却攻打西边。是使对方产生错觉以出奇制胜的一种战术。 示例:蜀人或～，指南攻北，吾兵必须分头守把。 10:坚壁清野 释义:对付强敌入入侵的一种方法。使敌人既攻不下据点，又抢不到物资。 示例:不许出战，只是～，待这干贼寇粮尽力弛，方可追他。 11揭竿而起 释义:砍了树干当武器，举起竹竿当旗帜，进行反抗。指人民起义。 示例:一时各路人马，～，不分昼夜，兼水路纷纷入鄂。 12豁然开朗 豁然：开阔敞亮的样子；开朗：地方开阔，光线充足、明亮。指顿时现出宽敞明亮的境界 听了他的话,我的心中豁然开朗,所有的忧愁都烟消云散了 13怡然自乐 怡然：安闲、愉快的样子。形容高兴而满足的样子 伸出双手，感受着花瓣飘落在手心的感觉，怡然自乐。 14鸡犬相闻 指人烟稠密 两个村子离得很近,简直是鸡犬相闻 体面的近义词|反义词及造句 我们还必须迅速采取行动，为实现社会包容和人人体面工作营造有利的环境。下面是小编精选整理的体面的近义词|反义词及造句，供您参考，欢迎大家阅读。 体面的近义词： 面子、合适、美观、颜面、场合、场面、排场、得体、好看、局面 体面的反义词： 难听、寒碜、邋遢、寒酸、难看 体面造句 1、我认为，帖在墙面上的证书，并不能使你成为一个体面的人。 2、他是那里唯一的看上去很体面的人;我认为他从来没有这样好看过。 3、所有的美国人都是好的，体面的人们每天都在践行着他们的责任。 4、美国人民慷慨、强大、体面，这并非因为我们信任我们自己，而是因为我们拥有超越我们自己的信念。 5、工人就是工人，无论他们来自哪里，他们都应该受到尊重和尊敬，至少因为他们从事正当的工作而获得体面的工资。 6、然而反之有些孩子可能就是多少有些体面的父母的不良种子这一概念却令人难以接受。 7、如果奥巴马能够成就此功，并且帮助一个体面的伊拉克落稳脚跟，奥巴马和民主党不仅是结束了伊拉克战争，而是积极从战争中挽救。 8、而且，等到年纪大了退休时，他们希望能得到尊重和体面的对待。 9、爸爸，您倒对这件事处理得很体面，而我想那可能是我一生中最糟糕的一个夜晚吧。 10、有一些积极的东西，低于预期的就业损失索赔和零售销售是体面的。 11、如果你努力工作，你就能有获得一份终生工作的机会，有着体面的薪水，良好的福利，偶尔还能得到晋升。 12、体面的和生产性的工作是消除贫困和建立自给自足最有效的方法之一。 13、同时，他是一个仁慈、温和、体面的人，一个充满爱的丈夫和父亲，一个忠实的朋友。 14、几周前我们刚讨论过平板电脑是如何作为一个体面且多产的电子设备，即使它没有完整的键盘，在进行输入时会稍慢些。 15、什么才是生活体面的标准? 16、我们还必须迅速采取行动，为实现社会包容和人人体面工作营造有利的环境。 17、她告诉我人们都担心是不是必须把孩子送到国外去学习，才能保证孩子们长大后至少能过上体面正派的生活。 1.孜孜不倦→勤勉不知疲倦。 他孜孜不倦的钻研学问。 2.坚持不懈→坚持到底不松懈。 他努力工作，坚持不懈，深受老板器重 3.变本加厉→变得比原来更加严重， 事隔二年，他的坏习惯不但没有改，反而变本加厉。 4.饱读诗书→读了很多诗书。 王老师饱读诗书，知识渊博，同学们都很尊重他。 5.反败为胜→从失败转为胜利。 下午的篮球赛，有了亮亮的上场，才反败为胜。 6.响彻云霄→声响透过云层，形容声音非常响亮。 我们的歌声响彻云霄。 7.汗流浃背→形容出汗多，背上的衣服都湿透了。 哥哥打完篮球，回家时总是汗流浃背的。 8.锦上添花→比喻美上加美，好上加好。 家里已经有了电视机，现在又买了电脑，真是锦上添花。 9.气喘如牛→比喻气喘得很厉害。 他刚跑完400米，就气喘如牛的问：「我是第一名吧？」 10.能跑善钻→形容动作灵活。 小强的动作很敏捷，能跑善钻，这次比赛应该是稳操胜券。 11.狂吠不已→狗不停地叫。 狗见到陌生人时，往往会狂吠不已。 12.提高警觉→要有敏锐的感觉。 在陌生的地方，夜间外出要提高警觉，尽量结伴而行。以策安全，13.呼朋引伴→招呼朋友，吸引伙伴。 鸟儿们在和煦的春风中呼朋引伴，唱出宛转的曲子。 14.大打出手→比喻逞凶打人或殴斗。 他们原本是好朋友，想不到竟为了一件小事而大打出手。 15.千变万化→形容变化极多。 天上的云，会随着气候的改变而千变万化，人们看云往往可以识天气。 16.五花八门：比喻变化多端或花样繁多。 这次趣味运动会的项目五花八门，妙趣横生，太棒了。 17.风吹草动→比喻轻微的动荡或变动。 敌人吃了败仗后，成了惊弓之鸟，一有风吹草动就惊惶逃窜。 18.生生不息→不停地繁衍生息。 我们伟大的中华民族就在中国这片神奇的土地上生生不息、代代相传。 19.欣欣向荣→草木生机旺盛的样子，比喻事业蓬勃发展，兴旺昌盛。我们的祖国到处是一派欣欣向荣、生机勃勃的新气象。 20.寸草不生→连小草也不能生长的地方，比喻荒凉贫瘠之地。 我们要把寸草不生的大沙漠改造成造福人民的绿洲 首尾乖互：相互违背，前后自相矛盾。五年级上册成语解释及近义词反义词和造句大全.doc
成语大全及解释造句[1]
悲惨的近义词反义词和造句
常用成语造句大全及解释
成语解释及例句
三年级下册成语解释及造句
知己的近义词反义词及知己的造句
成语大全解释及造句
成语解释加造句
小学语文反义词仿照的近义词反义词和造句
成语眼花缭乱的意思解析及造句
暗示的近义词和反义词 [暗示的近义词反义词和造句]
高中成语及解释及造句
放弃的近义词反义词和造句
成语及解释造句
成语解释及造句
体面的近义词-反义词及造句
成语解释和成语造句
- 近义词反义词大全有关聆听的反义词近义词和造句
- 参与的反义词近义词及造句
- 举一反三的近义词,反义词及造句
- 悲惨的近义词反义词和造句
- 近义词反义词大全_有关流年似水的反义词,近义词和造句
- 放弃的近义词反义词和造句
- 五年级上册成语解释及近义词反义词和造句大全
- 仿照的近义词反义词和造句
- 四年级上册近义词、反义词、造句
- 知己的近义词反义词及知己的造句
- 人教版五年级上册成语解释及近义词反义词和造句大全
- 恰当的反义词近义词及造句
- 近义词反义词造句
- 五年级上册成语解释及近义词反义词和造句大全.doc
- 仿照的近义词反义词和造句
- 暗示的近义词和反义词 [暗示的近义词反义词和造句]
- 二年级近义词反义词造句
- 朝夕相处近义词反义词及造句
- 污秽的近义词反义词和造句
- 实现的近义词和反义词是什么实现造句示例