|By Andreas Grabner||
|October 29, 2009 04:35 PM EDT||
Because virtual machines work by time-sharing host physical hardware, a virtual machine cannot exactly duplicate the timing behaviour of a physical machine. This leads to the timekeeping problems explained in the VMWare White Paper about Timekeeping in Virtual Machines that results in inaccurate time measurements within the virtual machine. This affects ALL performance metrics that rely on the operating system clock time to keep track of time which includes system counters like CPU or I/O Utilization. Performance Management solutions therefore run into the problem that the monitored metrics are inaccurate and can lead to incorrect enforcement of SLAs or wrong assumptions about application performance.
This blog explains the time keeping problem, how it impacts Application Performance Management in virtualized environments and what can be done to solve this problem.
Time keeping problem explained
Operating Systems that use a Tick Counting approach to keep track of time use hardware interrupts to count how many ticks have occurred since the system started. In a virtualized environment these interrupts are consumed by the virtualization infrastructure which keeps track of what we call the “Real Time”. The interrupts are then forwarded to the hosted virtual machines which itself keep track of the time that we call “Apparent Time“. In the best case scenario Real Time and Apparent Time are the same:
A virtual machine is not “always on” as it gets descheduled by the virtual server because of time-sharing with other virtual machines. In that time the hardware interrupts cannot be handled by the virtual machine and are therefore put into a queue for later consumption.
At the time the Virtual Machine gets scheduled again the operating system’s Apparent Time is still the time it was before it got descheduled as it has not yet received the timer interrupts that happened in the meantime. In that case the Apparent Time has drifted from the Real Time. Impact: Any performance metric taken at this time only shows the time that the Virtual Machine believes had passed which is not the time that really passed.
Over time the Virtual Machine catches up with the interrupts it missed while descheduled.
There are several other techniques that virtualization environments use to bring the Apparent Time back to Real Time as fast as possible. For more details have a look at the VMWare White Paper as mentioned on the top of this blog.
Impacts of the Time Keeping Problem
Any time based metrics captured from within the Virtual Machine are subject to the timekeeping problem including CPU Utilization, Memory Allocations per time interval, I/O access per time interval,… and any custom time tracking that can be used for e.g.: response time or transaction time monitoring. Operating system counters like % CPU per process also runs into another interesting problem where individual processes might get charged incorrectly with time that they never consumed. After a Virtual Machine is resumed the queued timer interrupts are processed. These interrupts come in a faster rate than normal. The currently active processes are charged with all these timing events although they have not done any work in that time because they were actually descheduled.
You can see that the timekeeping issue can really mess up your performance counters. The more load you have on a virtual server, the more virtual machines there are to schedule and de-schedule – the higher the impact on accurate timing will be. Other side effects like over-provisioning of CPU or Memory have an impact as well.
Using performance metrics from within the Virtual Machine for application performance management and enforcement of Service Level Agreements is therefore very questionable as the results are not accurate and not predictable.
Accurate Time Keeping with Pseudo Performance Counters
VMWare is aware of this problem and explains in great detail the reasons and the effects in their White Paper. As a solution for performance management solutions VMWare provides a way to query the actual Real Time at any time from within the Virtual Machine. Pseudo Performance Counters are made available via virtual processor registers that can be accessed from any application within the Virtual Machine.
dynaTrace is using these new counters for accurate time measurement when Managing Application Performance in virtualized VMWare environments. This allows accurate SLA enforcement and application performance management down to individual transactions or even methods. The following illustration shows a single captured transaction with accurate timings. dynaTrace captures the Real and the Apparent Time on method and transaction level:
Capturing the Real Time values and also showing the Apparent Time Drift enables Application Performance Management with accurate timing values. Accurate timings are the basics for accurate SLA Enforcement in Production as well as Application Performance Monitoring.
Is timekeeping a real issue in your environment?
The timekeeping problem is well known within the VMWare community and brings challenges to accurate application performance management in virtualized environments. I am interested in your experience with this problem. Have you been aware of it? Do you live with the inaccuracy or do you have other approaches for accurate measuring? Please share your thoughts on this topic.
WebRTC services have already permeated corporate communications in the form of videoconferencing solutions. However, WebRTC has the potential of going beyond and catalyzing a new class of services providing more than calls with capabilities such as mass-scale real-time media broadcasting, enriched and augmented video, person-to-machine and machine-to-machine communications. In his session at @ThingsExpo, Luis Lopez, CEO of Kurento, introduced the technologies required for implementing these idea...
Jan. 16, 2017 04:30 AM EST Reads: 4,463
The WebRTC Summit New York, to be held June 6-8, 2017, at the Javits Center in New York City, NY, announces that its Call for Papers is now open. Topics include all aspects of improving IT delivery by eliminating waste through automated business models leveraging cloud technologies. WebRTC Summit is co-located with 20th International Cloud Expo and @ThingsExpo. WebRTC is the future of browser-to-browser communications, and continues to make inroads into the traditional, difficult, plug-in web co...
Jan. 16, 2017 03:30 AM EST Reads: 2,825
The Internet of Things will challenge the status quo of how IT and development organizations operate. Or will it? Certainly the fog layer of IoT requires special insights about data ontology, security and transactional integrity. But the developmental challenges are the same: People, Process and Platform. In his session at @ThingsExpo, Craig Sproule, CEO of Metavine, demonstrated how to move beyond today's coding paradigm and shared the must-have mindsets for removing complexity from the develop...
Jan. 16, 2017 03:30 AM EST Reads: 5,202
While not quite mainstream yet, WebRTC is starting to gain ground with Carriers, Enterprises and Independent Software Vendors (ISV’s) alike. WebRTC makes it easy for developers to add audio and video communications into their applications by using Web browsers as their platform. But like any market, every customer engagement has unique requirements, as well as constraints. And of course, one size does not fit all. In her session at WebRTC Summit, Dr. Natasha Tamaskar, Vice President, Head of C...
Jan. 16, 2017 03:15 AM EST Reads: 5,707
Who are you? How do you introduce yourself? Do you use a name, or do you greet a friend by the last four digits of his social security number? Assuming you don’t, why are we content to associate our identity with 10 random digits assigned by our phone company? Identity is an issue that affects everyone, but as individuals we don’t spend a lot of time thinking about it. In his session at @ThingsExpo, Ben Klang, Founder & President of Mojo Lingo, discussed the impact of technology on identity. Sho...
Jan. 16, 2017 02:30 AM EST Reads: 3,891
DevOps is being widely accepted (if not fully adopted) as essential in enterprise IT. But as Enterprise DevOps gains maturity, expands scope, and increases velocity, the need for data-driven decisions across teams becomes more acute. DevOps teams in any modern business must wrangle the ‘digital exhaust’ from the delivery toolchain, "pervasive" and "cognitive" computing, APIs and services, mobile devices and applications, the Internet of Things, and now even blockchain. In this power panel at @...
Jan. 16, 2017 01:45 AM EST Reads: 2,646
With all the incredible momentum behind the Internet of Things (IoT) industry, it is easy to forget that not a single CEO wakes up and wonders if “my IoT is broken.” What they wonder is if they are making the right decisions to do all they can to increase revenue, decrease costs, and improve customer experience – effectively the same challenges they have always had in growing their business. The exciting thing about the IoT industry is now these decisions can be better, faster, and smarter. Now ...
Jan. 16, 2017 01:15 AM EST Reads: 4,142
WebRTC is about the data channel as much as about video and audio conferencing. However, basically all commercial WebRTC applications have been built with a focus on audio and video. The handling of “data” has been limited to text chat and file download – all other data sharing seems to end with screensharing. What is holding back a more intensive use of peer-to-peer data? In her session at @ThingsExpo, Dr Silvia Pfeiffer, WebRTC Applications Team Lead at National ICT Australia, looked at differ...
Jan. 16, 2017 12:00 AM EST Reads: 4,745
Internet of @ThingsExpo, taking place June 6-8, 2017 at the Javits Center in New York City, New York, is co-located with the 20th International Cloud Expo and will feature technical sessions from a rock star conference faculty and the leading industry players in the world. @ThingsExpo New York Call for Papers is now open.
Jan. 15, 2017 10:15 PM EST Reads: 3,482
WebRTC has had a real tough three or four years, and so have those working with it. Only a few short years ago, the development world were excited about WebRTC and proclaiming how awesome it was. You might have played with the technology a couple of years ago, only to find the extra infrastructure requirements were painful to implement and poorly documented. This probably left a bitter taste in your mouth, especially when things went wrong.
Jan. 15, 2017 08:45 PM EST Reads: 7,376
A critical component of any IoT project is what to do with all the data being generated. This data needs to be captured, processed, structured, and stored in a way to facilitate different kinds of queries. Traditional data warehouse and analytical systems are mature technologies that can be used to handle certain kinds of queries, but they are not always well suited to many problems, particularly when there is a need for real-time insights.
Jan. 15, 2017 07:00 PM EST Reads: 6,125
Big Data engines are powering a lot of service businesses right now. Data is collected from users from wearable technologies, web behaviors, purchase behavior as well as several arbitrary data points we’d never think of. The demand for faster and bigger engines to crunch and serve up the data to services is growing exponentially. You see a LOT of correlation between “Cloud” and “Big Data” but on Big Data and “Hybrid,” where hybrid hosting is the sanest approach to the Big Data Infrastructure pro...
Jan. 15, 2017 05:15 PM EST Reads: 4,775
Businesses are struggling to manage the information flow and interactions between all of these new devices and things jumping on their network, and the apps and IT systems they control. The data businesses gather is only helpful if they can do something with it. In his session at @ThingsExpo, Chris Witeck, Principal Technology Strategist at Citrix, discussed how different the impact of IoT will be for large businesses, expanding how IoT will allow large organizations to make their legacy applica...
Jan. 15, 2017 04:15 PM EST Reads: 1,920
SYS-CON Events announced today that Catchpoint, a leading digital experience intelligence company, has been named “Silver Sponsor” of SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Catchpoint Systems is a leading Digital Performance Analytics company that provides unparalleled insight into your customer-critical services to help you consistently deliver an amazing customer experience. Designed for digital business, C...
Jan. 15, 2017 01:15 PM EST Reads: 1,690
With major technology companies and startups seriously embracing IoT strategies, now is the perfect time to attend @ThingsExpo 2016 in New York. Learn what is going on, contribute to the discussions, and ensure that your enterprise is as "IoT-Ready" as it can be! Internet of @ThingsExpo, taking place June 6-8, 2017, at the Javits Center in New York City, New York, is co-located with 20th Cloud Expo and will feature technical sessions from a rock star conference faculty and the leading industry p...
Jan. 15, 2017 12:30 PM EST Reads: 3,564
"There's a growing demand from users for things to be faster. When you think about all the transactions or interactions users will have with your product and everything that is between those transactions and interactions - what drives us at Catchpoint Systems is the idea to measure that and to analyze it," explained Leo Vasiliou, Director of Web Performance Engineering at Catchpoint Systems, in this SYS-CON.tv interview at 18th Cloud Expo, held June 7-9, 2016, at the Javits Center in New York Ci...
Jan. 15, 2017 12:15 PM EST Reads: 5,380
The 20th International Cloud Expo has announced that its Call for Papers is open. Cloud Expo, to be held June 6-8, 2017, at the Javits Center in New York City, brings together Cloud Computing, Big Data, Internet of Things, DevOps, Containers, Microservices and WebRTC to one location. With cloud computing driving a higher percentage of enterprise IT budgets every year, it becomes increasingly important to plant your flag in this fast-expanding business opportunity. Submit your speaking proposal ...
Jan. 15, 2017 12:15 PM EST Reads: 4,960
20th Cloud Expo, taking place June 6-8, 2017, at the Javits Center in New York City, NY, will feature technical sessions from a rock star conference faculty and the leading industry players in the world. Cloud computing is now being embraced by a majority of enterprises of all sizes. Yesterday's debate about public vs. private has transformed into the reality of hybrid cloud: a recent survey shows that 74% of enterprises have a hybrid cloud strategy.
Jan. 15, 2017 11:15 AM EST Reads: 4,093
SYS-CON Events announced today that Super Micro Computer, Inc., a global leader in Embedded and IoT solutions, will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 7-9, 2017, at the Javits Center in New York City, NY. Supermicro (NASDAQ: SMCI), the leading innovator in high-performance, high-efficiency server technology, is a premier provider of advanced server Building Block Solutions® for Data Center, Cloud Computing, Enterprise IT, Hadoop/Big Data, HPC and E...
Jan. 15, 2017 11:15 AM EST Reads: 5,645
SYS-CON Events announced today that Linux Academy, the foremost online Linux and cloud training platform and community, will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Linux Academy was founded on the belief that providing high-quality, in-depth training should be available at an affordable price. Industry leaders in quality training, provided services, and student certification passes, its goal is to c...
Jan. 15, 2017 11:15 AM EST Reads: 1,834