Welcome!

Containers Expo Blog Authors: Pat Romanski, Rishi Bhargava, Liz McMillan, Elizabeth White, Steven Lamb

Related Topics: IBM Cloud, @CloudExpo

IBM Cloud: Blog Post

Databases in the Cloud

Cloud computing is such a fascinating topic because ...

Cloud computing is such a fascinating topic because, to anybody who can get enough distance from the marketing hype, it really represents a significant departure from the way the software and the hardware industry have been operating in the last decades. In fact, cloud computing is to the traditional IT industry what the Internet has been to the music and film industry: an unwelcome and very threatening development dictating completely different business models.

The basic technical premises behind cloud computing are known since many years. Already in the 80's, as the number of computers around constantly increased, it did not take long until people realized many of them were idle for most of time. The result was the first cluster management tools capable of moving simple jobs between machines. From there, the advances in networking during the 90's brought Grid Computing and then the large computer farms behind the Web brought what we now call Cloud Computing. On the software side, a similar development took place over the years going from monolithic designs to multi-tier architectures and from tightly coupled systems to service oriented architectures. Add virtualization, and all the basic tools for building computing clouds are in place.

Whether Software as a Service (SaaS) or Platform as a Service (PaaS), there are many examples out there of services that work well; make economic sense to all parties involved; and are destined to grow both in the number of users as well as in the functionality they provide. This, however, does not seem to be the case for databases. At least not for databases containing important data.

This is an interesting observation since, technically, there is nothing that prevents databases from residing in the cloud. To understand the complex relation between databases and the cloud, one needs to understand the complex chain of problems that need to be solved before a database with important data resides in the cloud. These problems are: 

- legal aspects of where the data resides

- long term custody warranties

- trust in the cloud

If the data residing in a database is of any real value to anybody except a small group of individuals, it is likely that there are many regulations imposing a wide variety of constraints on where the data can be, who can look at it, and whether it can be moved anywhere. For instance, in federal countries, local governments often have legislation imposing that the data must be stored within the region. At larger scale, it seems unlikely that a country would agree to have government data stored in a different country. In the private sector, many software development outsourcing efforts have failed because of the difficulty to provide realistic data for testing without giving any confidential information away. And if the database stays where it is, it is unlikely that the software stack built on top of it will move to the cloud.

Assuming there is a cloud in the vicinity that fulfills all the locality requirements, the next hurdle is the legal custodian warranties imposed on data. Important and relevant data must be by law available and searchable for long periods of time, often many decades. Clouds cannot provide such guarantees today and, in this matter, the IT industry has never dealt with such time horizons before.

Finally, even there is a cloud in the proper place that guarantees that it will stay there for the nest 50 years, the question that remains is whether it can be trusted to do so. What happens to the data if the cloud simply disappears? Replication makes the location problem even more difficult and it certainly does not help to reduce the cost of the cloud. It also does not solve the problem of a company simply shutting down the service. Without very strong, enforceable guarantees -as it happens in other branches of industry that are critical to the economy - there will be not enough trust to move databases with important data into a commercial cloud.

Does this imply that we will never see databases in the cloud? Not at all. However, the clouds were important databases will reside might be different from the commercial ones that are attracting so much attention these days.

First, the clouds where databases may live very comfortably will be private clouds. Governments, for instance, are likely to own (or contract) such clouds to offer cloud services to the public sector. Second, community clouds linking the private clouds of partner companies are also likely to be common since they spread the costs among several participants while still giving access to more resources that anyone of them directly owns. Being a federation of private clouds, they are easier to protect, organize under well defined contractual agreements, and to tailor to the particular application by using, e.g., application aware networks. Third, public clouds will be used not necessarily for storing the data but for scalability and processing purposes in all those cases where parts of the data can be safely brought into the open. For instance, a company can keep the confidential data within the private cloud but place databases with copies of the publicly available data on a public cloud. By keeping the master copy of the data, the company takes advantage of the public cloud for scalability but can make sure all regulations are followed in house using conventional solutions.

The challenges to bring databases into the cloud are both technical and regulatory. Until the regulatory problems are solved -and that may take a long time- the key to putting databases into the cloud will be to have infrastructures that give users flexibility and complete control over the databases and the data inside, regardless of the type of cloud used. If users can easily create copies of their databases or part of their databases and place those into the cloud; can guarantee that they have within their premises a consistent copy of the data at all times; and can take advantage of the cloud to reduce the costs of provisioning, scaling out, and adding functionality to their data management systems, then databases will move to the cloud. If all these chores are not provided through automatic tools, the overhead, costs, and risks involved will be too high to justify moving enterprise class databases to the cloud.

More Stories By Maximilian Ahrens

Ahrens is an expert and frequent speaker on international conferences for service oriented architecture and virtualization. Before co-founding Zimory, he served as a project manager and research scientist at the innovation development entity of Deutsche Telekom Laboratories. Responsible for infrastructure and enterprise IT projects spanning multiple divisions of the Deutsche Telekom group -- Ahrens is an expert on enterprise IT and business processes. Before Deutsche Telekom, he led several business process reengineering projects for major German companies. Ahrens received his degree in computer science and business administration from Technische Universität Berlin.

@ThingsExpo Stories
The Internet of Things will challenge the status quo of how IT and development organizations operate. Or will it? Certainly the fog layer of IoT requires special insights about data ontology, security and transactional integrity. But the developmental challenges are the same: People, Process and Platform. In his session at @ThingsExpo, Craig Sproule, CEO of Metavine, demonstrated how to move beyond today's coding paradigm and shared the must-have mindsets for removing complexity from the develo...
SYS-CON Events announced today that MangoApps will exhibit at the 19th International Cloud Expo, which will take place on November 1–3, 2016, at the Santa Clara Convention Center in Santa Clara, CA. MangoApps provides modern company intranets and team collaboration software, allowing workers to stay connected and productive from anywhere in the world and from any device.
The IETF draft standard for M2M certificates is a security solution specifically designed for the demanding needs of IoT/M2M applications. In his session at @ThingsExpo, Brian Romansky, VP of Strategic Technology at TrustPoint Innovation, explained how M2M certificates can efficiently enable confidentiality, integrity, and authenticity on highly constrained devices.
The 19th International Cloud Expo has announced that its Call for Papers is open. Cloud Expo, to be held November 1-3, 2016, at the Santa Clara Convention Center in Santa Clara, CA, brings together Cloud Computing, Big Data, Internet of Things, DevOps, Digital Transformation, Microservices and WebRTC to one location. With cloud computing driving a higher percentage of enterprise IT budgets every year, it becomes increasingly important to plant your flag in this fast-expanding business opportuni...
In today's uber-connected, consumer-centric, cloud-enabled, insights-driven, multi-device, global world, the focus of solutions has shifted from the product that is sold to the person who is buying the product or service. Enterprises have rebranded their business around the consumers of their products. The buyer is the person and the focus is not on the offering. The person is connected through multiple devices, wearables, at home, on the road, and in multiple locations, sometimes simultaneously...
“delaPlex Software provides software outsourcing services. We have a hybrid model where we have onshore developers and project managers that we can place anywhere in the U.S. or in Europe,” explained Manish Sachdeva, CEO at delaPlex Software, in this SYS-CON.tv interview at @ThingsExpo, held June 7-9, 2016, at the Javits Center in New York City, NY.
From wearable activity trackers to fantasy e-sports, data and technology are transforming the way athletes train for the game and fans engage with their teams. In his session at @ThingsExpo, will present key data findings from leading sports organizations San Francisco 49ers, Orlando Magic NBA team. By utilizing data analytics these sports orgs have recognized new revenue streams, doubled its fan base and streamlined costs at its stadiums. John Paul is the CEO and Founder of VenueNext. Prior ...
"We've discovered that after shows 80% if leads that people get, 80% of the conversations end up on the show floor, meaning people forget about it, people forget who they talk to, people forget that there are actual business opportunities to be had here so we try to help out and keep the conversations going," explained Jeff Mesnik, Founder and President of ContentMX, in this SYS-CON.tv interview at 18th Cloud Expo, held June 7-9, 2016, at the Javits Center in New York City, NY.
Internet of @ThingsExpo, taking place November 1-3, 2016, at the Santa Clara Convention Center in Santa Clara, CA, is co-located with the 19th International Cloud Expo and will feature technical sessions from a rock star conference faculty and the leading industry players in the world and ThingsExpo Silicon Valley Call for Papers is now open.
The IoT is changing the way enterprises conduct business. In his session at @ThingsExpo, Eric Hoffman, Vice President at EastBanc Technologies, discussed how businesses can gain an edge over competitors by empowering consumers to take control through IoT. He cited examples such as a Washington, D.C.-based sports club that leveraged IoT and the cloud to develop a comprehensive booking system. He also highlighted how IoT can revitalize and restore outdated business models, making them profitable ...
With 15% of enterprises adopting a hybrid IT strategy, you need to set a plan to integrate hybrid cloud throughout your infrastructure. In his session at 18th Cloud Expo, Steven Dreher, Director of Solutions Architecture at Green House Data, discussed how to plan for shifting resource requirements, overcome challenges, and implement hybrid IT alongside your existing data center assets. Highlights included anticipating workload, cost and resource calculations, integrating services on both sides...
Big Data engines are powering a lot of service businesses right now. Data is collected from users from wearable technologies, web behaviors, purchase behavior as well as several arbitrary data points we’d never think of. The demand for faster and bigger engines to crunch and serve up the data to services is growing exponentially. You see a LOT of correlation between “Cloud” and “Big Data” but on Big Data and “Hybrid,” where hybrid hosting is the sanest approach to the Big Data Infrastructure pro...
"We are a well-established player in the application life cycle management market and we also have a very strong version control product," stated Flint Brenton, CEO of CollabNet,, in this SYS-CON.tv interview at 18th Cloud Expo, held June 7-9, 2016, at the Javits Center in New York City, NY.
We all know the latest numbers: Gartner, Inc. forecasts that 6.4 billion connected things will be in use worldwide in 2016, up 30 percent from last year, and will reach 20.8 billion by 2020. We're rapidly approaching a data production of 40 zettabytes a day – more than we can every physically store, and exabytes and yottabytes are just around the corner. For many that’s a good sign, as data has been proven to equal money – IF it’s ingested, integrated, and analyzed fast enough. Without real-ti...
I wanted to gather all of my Internet of Things (IOT) blogs into a single blog (that I could later use with my University of San Francisco (USF) Big Data “MBA” course). However as I started to pull these blogs together, I realized that my IOT discussion lacked a vision; it lacked an end point towards which an organization could drive their IOT envisioning, proof of value, app dev, data engineering and data science efforts. And I think that the IOT end point is really quite simple…
A critical component of any IoT project is what to do with all the data being generated. This data needs to be captured, processed, structured, and stored in a way to facilitate different kinds of queries. Traditional data warehouse and analytical systems are mature technologies that can be used to handle certain kinds of queries, but they are not always well suited to many problems, particularly when there is a need for real-time insights.
We're entering the post-smartphone era, where wearable gadgets from watches and fitness bands to glasses and health aids will power the next technological revolution. With mass adoption of wearable devices comes a new data ecosystem that must be protected. Wearables open new pathways that facilitate the tracking, sharing and storing of consumers’ personal health, location and daily activity data. Consumers have some idea of the data these devices capture, but most don’t realize how revealing and...
Unless your company can spend a lot of money on new technology, re-engineering your environment and hiring a comprehensive cybersecurity team, you will most likely move to the cloud or seek external service partnerships. In his session at 18th Cloud Expo, Darren Guccione, CEO of Keeper Security, revealed what you need to know when it comes to encryption in the cloud.
You think you know what’s in your data. But do you? Most organizations are now aware of the business intelligence represented by their data. Data science stands to take this to a level you never thought of – literally. The techniques of data science, when used with the capabilities of Big Data technologies, can make connections you had not yet imagined, helping you discover new insights and ask new questions of your data. In his session at @ThingsExpo, Sarbjit Sarkaria, data science team lead ...
Extracting business value from Internet of Things (IoT) data doesn’t happen overnight. There are several requirements that must be satisfied, including IoT device enablement, data analysis, real-time detection of complex events and automated orchestration of actions. Unfortunately, too many companies fall short in achieving their business goals by implementing incomplete solutions or not focusing on tangible use cases. In his general session at @ThingsExpo, Dave McCarthy, Director of Products...