Welcome!

Containers Expo Blog Authors: John Rauser, Liz McMillan, Don MacVittie, Automic Blog, Elizabeth White

Related Topics: @DevOpsSummit, Containers Expo Blog, @CloudExpo

@DevOpsSummit: Article

Site Reliability Engineering: DevOps 2.0 | @DevOpsSummit #Cloud #DevOps #DataCenter #Monitoring

The dream that brought DevOps together is for someone who can be half dev and half ops

Site Reliability Engineering: DevOps 2.0
By Saba Anees

Has there ever been a better time to be in DevOps? TV shows like “Person of Interest” and “Mr. Robot” are getting better at showing what developers actually do, using chunks of working code. Movies like Michael Mann’s “Blackhat” (2015) won praise from Google’s security team for its DevOps accuracy in a few scenes. Look around and you’ll discover elements of DevOps culture filtering out into wider society, such as people in all walks of life discussing their uptime or fast approaching code lock.

On the other hand, perhaps the biggest thorn in the side of DevOps is that developers and operations teams don’t normally get along well. Developers want to rush ahead and compile some groundbreaking code under extremely tight schedules, while operations teams try to slow everyone down to identify systemic risks from accidents or malicious actors. Both teams want to end up with a better user experience, but getting there becomes a power struggle over what that experience truly means.

The dream that brought DevOps together is for someone who can be half dev and half ops. That split desire is exactly the point of the SRE (site reliability engineer).

Defining the SRE
In
introducing the term SRE, Google’s VP of Engineering, Ben Treynor, stated,

“It’s what happens when you ask a software engineer to design an operations function…. The SRE is fundamentally doing work that has historically been done by an operations team, but using engineers with software expertise, and banking on the fact that these engineers are inherently both predisposed to, and have the ability to, substitute automation for human labor.”

Way back in 2010, Facebook SRE Mark Schonbach explained what he did this way:

“I’m part of a small team of Site Reliability Engineers (SRE) that works day and night to ensure that you and the other 400+ million users around the world are able to access Facebook, that the site loads quickly, and all of the features are working…. We regularly hack tools on the fly that help us manage and perform complex maintenance procedures on one of the largest, if not the largest memcached footprints in the world. We develop automated tools to provision new servers, reallocate existing ones, and detect and repair applications or servers that are misbehaving.”

Where Did SREs Come From?
Reliability engineering is a concept that grew out of the operations world and has been around for more than 100 years. It became more closely connected with electronic systems after World War II, when the IEEE created the
Reliability Society. In the past 10 years, five 9s (99.999) became the golden standard for application performance management. That standard led to the creation of a class of operations experts who knew enough code to recover the site and put the last stable release back into production as fast as possible.

Treynor explained the impetus for creating this new category at Google with his typical deadpan humor: “One of the things you normally see in operations roles as opposed to engineering roles is that there’s a chasm not only with respect to duty, but also of background and of vocabulary, and eventually, of respect. This, to me, is a pathology.”

Which Toolsets Do SREs Use?
For SREs, stability and uptime top priorities. However, they should be able to take responsibility and code their own way out of hazards, instead of adding to the to-do lists of the development team. In the case of Google, SREs are often software engineers with a layer of network training on top. Typically,
Google software engineers must demonstrate proficiency in:

  1. Google’s own Golang and OO languages such as C++, Python or Java

  2. A secondary language like JavaScript, CSS & HTML, PHP, Ruby, Scheme, Perl, etc

  3. Advanced fields like AI research, cryptography, compilers, UX design, etc

  4. Getting along with other coders

On top of those proficiencies, Google’s SREs must have experience in network engineering, Unix sys admin or more general networking/ops skills such as LDAP and DNS.

The Critical Role of SRE
Downtime is costing businesses around
$300,000 per hour, according to a report from Emerson Network Power. The most obvious impact is when traffic spikes bring down e-commerce sites, which was covered in a recent AppDynamics white paper. However, Treynor also pointed out how standard dev vs. ops friction can be costly to businesses in other ways. The classic conflict starts with the support checklist that ops presents to dev before feature updates are released. Developers win when users like newly developed features, the sooner the better. Meanwhile, operations wins when there are the maximum 9s in their uptime reports. All change brings instability; how do you align their interests?

Treynor’s answer is a relief for those with compensation tied to user satisfaction metrics, but not so much for those with heart conditions. He said,

“100% is the wrong reliability target for basically everything. Perhaps a pacemaker is a good exception! But, in general, for any software service or system you can think of, 100% is not the right reliability target because no user can tell the difference between a system being 100% available and, let’s say, 99.999% available. Because typically there are so many other things that sit in between the user and the software service that you’re running that the marginal difference is lost in the noise of everything else that can go wrong.”

This response shifts the focus from specific uptime metrics, which may not act as accurate proxies for user expectations, to a reliability index based on market realities. Treynor explained,

“If 100% is the wrong reliability target for a system, what, then, is the right reliability target for the system? I propose that’s a product question. It’s not a technical question at all. It’s a question of what will the users be happy with, given how much they’re paying, whether it’s direct or indirect, and what their alternatives are.”

Who Is Hiring SREs?
The simple answer is “Everyone.” From software/hardware giants like
Apple to financial portals like Morningstar to non-profit institutions like the Lawrence Berkeley National Laboratory. Berkeley is a great example of an organization that’s both at the cutting edge of energy research, yet also maintains some very old legacy systems. Assuring reliability across several generations of technologies can be an enormous challenge. Here’s a look into what their SREs at Berkeley Labs are responsible for:

  • Linux system administration skills to monitor and manage the reliability of the systems under the responsibility of the Control Room Bridge.

  • Develop and maintain monitoring tools used to support the HPC community within NERSC using programming languages like C, C++, Python, Java or Perl.

  • Provide input in the design of software, workflows and processes that improve the monitoring capability of the group to ensure the high availability of the HPC services provided by NERSC and ESnet.

  • Support in the testing and implementation of new monitoring tools, workflows and new capabilities for providing high availability for the systems in production.

  • Assist in direct hardware support of data clusters through managing component upgrades and replacements (dimms, hard drives, cards, cables, etc) to ensure the efficient return of nodes to production service.

  • Help in investigating and evaluating new technologies and solutions to push the group’s capabilities forward, getting ahead of users’ needs, and convincing staff who are incentivized to transform, innovate and continually improve.

Contrast that skill profile with an online company like Wikipedia, where an SRE assignment tends to be less technical and more diplomatic:

  • Improve automation, tooling and processes to support development and deployment

  • Form deep partnership with engineering teams to work on improving user site experience

  • Participate in sprint planning meetings, and support intra-department coordination

  • Troubleshoot site outages and performance issues, including on-call response

  • Help with the provisioning of systems and services, including configuration management

  • Support capacity planning, profiling of site performance, and other analysis

  • Help with general ops issues, including tickets and other ongoing maintenance tasks

Within the past year, there has been a marked shift to a more strategic level of decision-making that reflects the increasing automation of customer requests and failover procedures. Even at traditional companies like IBM, SREs work with some of the newest platforms available due to the advance of IoT agendas. For example, one opening for an SRE at IBM in Ireland requires experience in OpenStack Heat, Urban Code Deploy, Chef, Jenkins, ELK, Splunk, Collect D and Graphite.

How SREs Are Changing
The online world is quite different now than when SREs entered the scene nearly a decade ago. Since then, mobile has redefined development cycles, and easy access to cloud-based data centers has brought
microservices into the mainstream IT infrastructure. Startups regularly come out of the gates using Rest and JSON as their preferred protocol for mobile apps. In accordance with the principles of Lean Startups, DevOps are often smaller, more focused teams that function as collective SREs.

You’ll find there’s a great deal more collaboration and less conflict between development and operations, simply because the continuous delivery model has collapsed the responsibilities of development and operations into a single cycle. The term DevOps is likely to disappear as the two distinct divisions merge in the new world, where UX is everything and updates may be pushed out weekly. Regardless of how many 9s are in any given SREs job description, this career path appears to offer you maximum reliability with job security.

The post Site Reliability Engineering: DevOps 2.0 appeared first on Application Performance Monitoring Blog | AppDynamics.

More Stories By AppDynamics Blog

In high-production environments where release cycles are measured in hours or minutes — not days or weeks — there's little room for mistakes and no room for confusion. Everyone has to understand what's happening, in real time, and have the means to do whatever is necessary to keep applications up and running optimally.

DevOps is a high-stakes world, but done well, it delivers the agility and performance to significantly impact business competitiveness.

@ThingsExpo Stories
"There's plenty of bandwidth out there but it's never in the right place. So what Cedexis does is uses data to work out the best pathways to get data from the origin to the person who wants to get it," explained Simon Jones, Evangelist and Head of Marketing at Cedexis, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.
"Space Monkey by Vivent Smart Home is a product that is a distributed cloud-based edge storage network. Vivent Smart Home, our parent company, is a smart home provider that places a lot of hard drives across homes in North America," explained JT Olds, Director of Engineering, and Brandon Crowfeather, Product Manager, at Vivint Smart Home, in this SYS-CON.tv interview at @ThingsExpo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.
"IBM is really all in on blockchain. We take a look at sort of the history of blockchain ledger technologies. It started out with bitcoin, Ethereum, and IBM evaluated these particular blockchain technologies and found they were anonymous and permissionless and that many companies were looking for permissioned blockchain," stated René Bostic, Technical VP of the IBM Cloud Unit in North America, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Conventi...
WebRTC is great technology to build your own communication tools. It will be even more exciting experience it with advanced devices, such as a 360 Camera, 360 microphone, and a depth sensor camera. In his session at @ThingsExpo, Masashi Ganeko, a manager at INFOCOM Corporation, introduced two experimental projects from his team and what they learned from them. "Shotoku Tamago" uses the robot audition software HARK to track speakers in 360 video of a remote party. "Virtual Teleport" uses a multip...
SYS-CON Events announced today that Telecom Reseller has been named “Media Sponsor” of SYS-CON's 22nd International Cloud Expo, which will take place on June 5-7, 2018, at the Javits Center in New York, NY. Telecom Reseller reports on Unified Communications, UCaaS, BPaaS for enterprise and SMBs. They report extensively on both customer premises based solutions such as IP-PBX as well as cloud based and hosted platforms.
Coca-Cola’s Google powered digital signage system lays the groundwork for a more valuable connection between Coke and its customers. Digital signs pair software with high-resolution displays so that a message can be changed instantly based on what the operator wants to communicate or sell. In their Day 3 Keynote at 21st Cloud Expo, Greg Chambers, Global Group Director, Digital Innovation, Coca-Cola, and Vidya Nagarajan, a Senior Product Manager at Google, discussed how from store operations and ...
In his session at 21st Cloud Expo, Carl J. Levine, Senior Technical Evangelist for NS1, will objectively discuss how DNS is used to solve Digital Transformation challenges in large SaaS applications, CDNs, AdTech platforms, and other demanding use cases. Carl J. Levine is the Senior Technical Evangelist for NS1. A veteran of the Internet Infrastructure space, he has over a decade of experience with startups, networking protocols and Internet infrastructure, combined with the unique ability to it...
Gemini is Yahoo’s native and search advertising platform. To ensure the quality of a complex distributed system that spans multiple products and components and across various desktop websites and mobile app and web experiences – both Yahoo owned and operated and third-party syndication (supply), with complex interaction with more than a billion users and numerous advertisers globally (demand) – it becomes imperative to automate a set of end-to-end tests 24x7 to detect bugs and regression. In th...
"Cloud Academy is an enterprise training platform for the cloud, specifically public clouds. We offer guided learning experiences on AWS, Azure, Google Cloud and all the surrounding methodologies and technologies that you need to know and your teams need to know in order to leverage the full benefits of the cloud," explained Alex Brower, VP of Marketing at Cloud Academy, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clar...
SYS-CON Events announced today that Evatronix will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Evatronix SA offers comprehensive solutions in the design and implementation of electronic systems, in CAD / CAM deployment, and also is a designer and manufacturer of advanced 3D scanners for professional applications.
"MobiDev is a software development company and we do complex, custom software development for everybody from entrepreneurs to large enterprises," explained Alan Winters, U.S. Head of Business Development at MobiDev, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.
SYS-CON Events announced today that CrowdReviews.com has been named “Media Sponsor” of SYS-CON's 22nd International Cloud Expo, which will take place on June 5–7, 2018, at the Javits Center in New York City, NY. CrowdReviews.com is a transparent online platform for determining which products and services are the best based on the opinion of the crowd. The crowd consists of Internet users that have experienced products and services first-hand and have an interest in letting other potential buye...
It is of utmost importance for the future success of WebRTC to ensure that interoperability is operational between web browsers and any WebRTC-compliant client. To be guaranteed as operational and effective, interoperability must be tested extensively by establishing WebRTC data and media connections between different web browsers running on different devices and operating systems. In his session at WebRTC Summit at @ThingsExpo, Dr. Alex Gouaillard, CEO and Founder of CoSMo Software, presented ...
Leading companies, from the Global Fortune 500 to the smallest companies, are adopting hybrid cloud as the path to business advantage. Hybrid cloud depends on cloud services and on-premises infrastructure working in unison. Successful implementations require new levels of data mobility, enabled by an automated and seamless flow across on-premises and cloud resources. In his general session at 21st Cloud Expo, Greg Tevis, an IBM Storage Software Technical Strategist and Customer Solution Architec...
A strange thing is happening along the way to the Internet of Things, namely far too many devices to work with and manage. It has become clear that we'll need much higher efficiency user experiences that can allow us to more easily and scalably work with the thousands of devices that will soon be in each of our lives. Enter the conversational interface revolution, combining bots we can literally talk with, gesture to, and even direct with our thoughts, with embedded artificial intelligence, whic...
To get the most out of their data, successful companies are not focusing on queries and data lakes, they are actively integrating analytics into their operations with a data-first application development approach. Real-time adjustments to improve revenues, reduce costs, or mitigate risk rely on applications that minimize latency on a variety of data sources. In his session at @BigDataExpo, Jack Norris, Senior Vice President, Data and Applications at MapR Technologies, reviewed best practices to ...
An increasing number of companies are creating products that combine data with analytical capabilities. Running interactive queries on Big Data requires complex architectures to store and query data effectively, typically involving data streams, an choosing efficient file format/database and multiple independent systems that are tied together through custom-engineered pipelines. In his session at @BigDataExpo at @ThingsExpo, Tomer Levi, a senior software engineer at Intel’s Advanced Analytics gr...
When talking IoT we often focus on the devices, the sensors, the hardware itself. The new smart appliances, the new smart or self-driving cars (which are amalgamations of many ‘things’). When we are looking at the world of IoT, we should take a step back, look at the big picture. What value are these devices providing? IoT is not about the devices, it’s about the data consumed and generated. The devices are tools, mechanisms, conduits. In his session at Internet of Things at Cloud Expo | DXWor...
Everything run by electricity will eventually be connected to the Internet. Get ahead of the Internet of Things revolution. In his session at @ThingsExpo, Akvelon expert and IoT industry leader Sergey Grebnov provided an educational dive into the world of managing your home, workplace and all the devices they contain with the power of machine-based AI and intelligent Bot services for a completely streamlined experience.
SYS-CON Events announced today that Synametrics Technologies will exhibit at SYS-CON's 22nd International Cloud Expo®, which will take place on June 5-7, 2018, at the Javits Center in New York, NY. Synametrics Technologies is a privately held company based in Plainsboro, New Jersey that has been providing solutions for the developer community since 1997. Based on the success of its initial product offerings such as WinSQL, Xeams, SynaMan and Syncrify, Synametrics continues to create and hone inn...