Welcome!

Containers Expo Blog Authors: Liz McMillan, Elizabeth White, Pat Romanski, Yeshim Deniz, Moshe Kranc

Related Topics: IBM Cloud, Containers Expo Blog, @CloudExpo

IBM Cloud: Article

The Intersection of Data Virtualization and Enterprise Data Warehouses

Eight Best Practices - Part 1

Virtualization Magazine on Ulitzer

Large enterprises and government agencies are drowning in data. IT teams deploy a myriad of data warehouse-centric solutions - BI, predictive analytics, data and content mining, portals and dashboards - to harness and deliver data for intelligent decision-making.

Yet, large enterprises are also expected to act like start-ups: nimble, agile and flexible to adapt to ever-changing market conditions. Impossible? By examining the best practices of their successful peers and adapting these to their own enterprises, data teams and enterprise architects can contribute to their corporate initiatives.

The intersection of data virtualization and enterprise data warehouses represents corporate best practices for delivering the rich data assets available in the enterprise data warehouse with the myriad sources of data now available outside the data warehouse. In Part One of this two-part series, I examine ways to use data virtualization to improve data warehouse effectiveness. By extending the warehouse schema to include additional data, data virtualization delivers greater business value from existing warehouse and non-warehouse data assets. Part Two will target improving data warehouse efficiency by showing four best practices where data virtualization, used alongside data warehouses, save time and money.

1. Data Warehouse Extension
Despite proliferating data silos accommodating exponentially growing data volumes, valuable data still remains outside the warehouse. Enterprise IT teams responsible for providing users with complete data in support of revenue, cost and risk management goals typically require the following combinations of data from both the data warehouse plus external sources:

  • Historical data as well as current data from transactional systems or operational data stores.
  • Summarized data and drill-down detail from transactional systems or operational data stores.
  • Proprietary data and external data from outside sources including cloud computing

Data virtualization federates data-warehouse contents with additional information sources. These complementary views are conducive to adding current data to historical warehouse data, detailed data to summarized warehouse data, and external data to internal warehouse data.

In Figure 1, data virtualization middleware hosts new views that integrate additional RDBMS and web service data sources as well as extend existing sources such as packaged applications.

Industry Examples
A global energy company uses data virtualization to federate its real-time data about crew, equipment and production status from its wells with its SAP maintenance management system of historical surface, subsurface and business data.  This information enables the optimal deployment of repair crews and equipment across 10,000-plus production oil wells, resulting in faster repairs for greater uptime and more profitability.

To analyze the effectiveness of its pharmaceutical sales and marketing programs, a life sciences leader uses data virtualization to federate externally sourced competitor sales data from an industry data services provider with internal prescription sales data from its sales data warehouse. This total market view provides the sales team with the intelligence required for more effective sales and marketing programs that increase revenues.

2. MDM Hub Extension
Master Data Management (MDM) hubs maintain and control critical master data attributes, but not the detailed transaction histories and other related data maintained and controlled in dozens of other systems across the extended enterprise. Data virtualization acts as the foreign key to quickly and easily federate master data within an MDM hub with additional transactional and historical data, delivering a comprehensive view of customer, product and employee information.

In the integration pattern shown in Figure 2, the data virtualization middleware hosts new complementary views that integrate additional RDBMS and Web service data sources as well as extend existing sources, such as packaged applications.

Industry Examples
A European mobile phone operator, known for its customer service leadership, maximizes revenue per customer and customer service levels by using data virtualization to combine customer service data from its customer reporting data warehouse, billing data from its financial systems, and call and configuration data from its operational support systems to deliver a 360o view of customers to customer service representatives. Faster issue resolution and more productive up-sell programs reduce churn, increase customer satisfaction, and ultimately contribute to revenues.

A global investment bank federates its HR master data with myriad internal benefits and compensation systems as well as external payroll services to provide its employees with a 360o view of their total compensation through a self-service employee benefits portal. Securely exposing this information improves retention and lessens HR staff workload.

3. Data Warehouse Federation
A fundamental reason enterprises implement data warehouses is to overcome the disparity of transactional and analytical system silos typical in today's large enterprise and government agencies. However, the single "enterprise" data warehouse remains elusive. Instead, multiple data warehouses and data marts have been developed and deployed, perpetuating, rather than overcoming, data silo proliferation.

Optimizing business performance requires data from across these various warehouses and marts. The effort of physically combining multiple marts and warehouses into a complete enterprise-wide data warehouse is simply too costly and time-consuming.

Data virtualization federates multiple physical warehouses. Two examples include combining data from sales and financial data warehouses, or combining two sales data warehouses after a corporate merger or acquisition. This approach creates an integrated view by using abstraction to rationalize the different schema designs, and thereby achieves a logical consolidation of multiple warehouses.

In the federation pattern shown in Figure 3, the data virtualization middleware hosts federated warehouse views that logically integrate both data warehouses.

Industry Examples
To enable more flexible customer self-service reporting and meet SEC compliance reporting mandates, a leading prime brokerage federates equity, fixed income and other investment positions and trades information from siloed trading data warehouses into a data virtualization layer. The net result is higher customer satisfaction and lower reporting costs.

One of the world's leading pharmaceutical companies uses data virtualization to enable research scientists to access and analyze data from research, clinical trial, FDA submission and other data warehouses. Scientists use this data to accelerate time-to-market for new compounds and drugs, thereby increasing revenues in an otherwise lengthy and costly development process.

4. Data Warehouse in Enterprise Architectures
Increasingly enterprises are seeking unified ways to integrate warehouse and other data in an enterprise-wide information architecture. According to Cambridge, Mass.-based industry analyst firm Forrester Research, "New architectural approaches such as Information-as-a-Service (IaaS) have emerged to provide flexible, real-time, service-oriented data integration and data-quality capabilities that support both structured data and unstructured content, delivering a true information integration platform." (1)

Data virtualization, which is included in IaaS, integrates data warehouses into an unified enterprise information architecture, as shown in Figure 4. IT teams use data virtualization middleware to form an enterprise data virtualization layer that is home to a consistent and complete logical schema covering multiple consolidated and virtual sources. When designing the enterprise information architecture, developers use data virtualization design tools to develop semantic abstractions in the form of web services or relational views. At runtime, end user-level applications, reports or mash-ups are created to call web data services, on demand, to query, federate, abstract and deliver the requested data to information consumers.

Industry Examples
To meet the information needs of its diverse group of technical and business users, an energy company has deployed enterprise information architecture to deliver disparate warehouse and operational data from more than 12 refineries located around the world. As a result, the company's refinery yields are increasing, and it proactively maintains its equipment, reducing downtime. In addition, the company complies with a myriad of regulations more consistently and at a reduced cost.

Several government agencies are using data virtualization to create a common information layer that spans agency information databases and enables intelligence analysts to better control threats. Agencies involved include the Drug Enforcement Administration (DEA) and the Immigration and Naturalization Service (INS). The common information layer delivers access to passenger, crew and manifest data from a U.S. Coast Guard port arrivals data warehouse, for example.

Conclusion
As data sources proliferate, including many web-based and cloud computing sources outside traditional enterprise data warehouses, enterprises and government agencies are deploying solutions that represent the intersection of enterprise data warehouses and data virtualization. These hybrid solutions deliver the most comprehensive information to decision-makers and positively impact the bottom line. Additional results include extended life for existing information system investments, greater agility for adding new BI and other analytic technologies, and less disruption from corporate activities including mergers and acquisitions.

Resources
1. 2009 Update: Evaluating Integration Alternatives, Scenario-Based Guidance For Choosing Products That Provide Application, Process, And Data Integration Features, Ken Vollmer Copyright (r) 2009, Forrester Research, Inc.

More Stories By Robert Eve

Robert "Bob" Eve is vice president of marketing at Composite Software. Prior to joining Composite, he held executive-level marketing and business development roles at several other enterprise software companies. At Informatica and Mercury Interactive, he helped penetrate new segments in his role as the vice president of Market Development. Bob ran Marketing and Alliances at Kintana (acquired by Mercury Interactive in 2003) where he defined the IT Governance category. As vice president of Alliances at PeopleSoft, Bob was responsible for more than 300 partners and 100 staff members. Bob has an MS in management from MIT and a BS in business administration with honors from University of California, Berkeley. He is a frequent contributor to publications including SYS-CON's SOA World Magazine and Virtualization Journal.

Comments (0)

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.


@ThingsExpo Stories
SYS-CON Events announced today that DXWorldExpo has been named “Global Sponsor” of SYS-CON's 21st International Cloud Expo, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Digital Transformation is the key issue driving the global enterprise IT business. Digital Transformation is most prominent among Global 2000 enterprises and government institutions.
SYS-CON Events announced today that Datera, that offers a radically new data management architecture, has been named "Exhibitor" of SYS-CON's 21st International Cloud Expo ®, which will take place on Oct 31 - Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Datera is transforming the traditional datacenter model through modern cloud simplicity. The technology industry is at another major inflection point. The rise of mobile, the Internet of Things, data storage and Big...
"We've been engaging with a lot of customers including Panasonic, we've been involved with Cisco and now we're working with the U.S. government - the Department of Homeland Security," explained Peter Jung, Chief Product Officer at Pulzze Systems, in this SYS-CON.tv interview at @ThingsExpo, held June 6-8, 2017, at the Javits Center in New York City, NY.
SYS-CON Events announced today that Calligo, an innovative cloud service provider offering mid-sized companies the highest levels of data privacy and security, has been named "Bronze Sponsor" of SYS-CON's 21st International Cloud Expo ®, which will take place on Oct 31 - Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Calligo offers unparalleled application performance guarantees, commercial flexibility and a personalised support service from its globally located cloud plat...
"We are focused on SAP running in the clouds, to make this super easy because we believe in the tremendous value of those powerful worlds - SAP and the cloud," explained Frank Stienhans, CTO of Ocean9, Inc., in this SYS-CON.tv interview at 20th Cloud Expo, held June 6-8, 2017, at the Javits Center in New York City, NY.
Internet of @ThingsExpo, taking place October 31 - November 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA, is co-located with 21st Cloud Expo and will feature technical sessions from a rock star conference faculty and the leading industry players in the world. The Internet of Things (IoT) is the most profound change in personal and enterprise IT since the creation of the Worldwide Web more than 20 years ago. All major researchers estimate there will be tens of billions devic...
"The Striim platform is a full end-to-end streaming integration and analytics platform that is middleware that covers a lot of different use cases," explained Steve Wilkes, Founder and CTO at Striim, in this SYS-CON.tv interview at 20th Cloud Expo, held June 6-8, 2017, at the Javits Center in New York City, NY.
"We provide IoT solutions. We provide the most compatible solutions for many applications. Our solutions are industry agnostic and also protocol agnostic," explained Richard Han, Head of Sales and Marketing and Engineering at Systena America, in this SYS-CON.tv interview at @ThingsExpo, held June 6-8, 2017, at the Javits Center in New York City, NY.
DX World EXPO, LLC., a Lighthouse Point, Florida-based startup trade show producer and the creator of "DXWorldEXPO® - Digital Transformation Conference & Expo" has announced its executive management team. The team is headed by Levent Selamoglu, who has been named CEO. "Now is the time for a truly global DX event, to bring together the leading minds from the technology world in a conversation about Digital Transformation," he said in making the announcement.
"MobiDev is a Ukraine-based software development company. We do mobile development, and we're specialists in that. But we do full stack software development for entrepreneurs, for emerging companies, and for enterprise ventures," explained Alan Winters, U.S. Head of Business Development at MobiDev, in this SYS-CON.tv interview at 20th Cloud Expo, held June 6-8, 2017, at the Javits Center in New York City, NY.
While the focus and objectives of IoT initiatives are many and diverse, they all share a few common attributes, and one of those is the network. Commonly, that network includes the Internet, over which there isn't any real control for performance and availability. Or is there? The current state of the art for Big Data analytics, as applied to network telemetry, offers new opportunities for improving and assuring operational integrity. In his session at @ThingsExpo, Jim Frey, Vice President of S...
SYS-CON Events announced today that DXWorldExpo has been named “Global Sponsor” of SYS-CON's 21st International Cloud Expo, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Digital Transformation is the key issue driving the global enterprise IT business. Digital Transformation is most prominent among Global 2000 enterprises and government institutions.
In his opening keynote at 20th Cloud Expo, Michael Maximilien, Research Scientist, Architect, and Engineer at IBM, discussed the full potential of the cloud and social data requires artificial intelligence. By mixing Cloud Foundry and the rich set of Watson services, IBM's Bluemix is the best cloud operating system for enterprises today, providing rapid development and deployment of applications that can take advantage of the rich catalog of Watson services to help drive insights from the vast t...
SYS-CON Events announced today that EnterpriseTech has been named “Media Sponsor” of SYS-CON's 21st International Cloud Expo, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. EnterpriseTech is a professional resource for news and intelligence covering the migration of high-end technologies into the enterprise and business-IT industry, with a special focus on high-tech solutions in new product development, workload management, increased effic...
SYS-CON Events announced today that Massive Networks, that helps your business operate seamlessly with fast, reliable, and secure internet and network solutions, has been named "Exhibitor" of SYS-CON's 21st International Cloud Expo ®, which will take place on Oct 31 - Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. As a premier telecommunications provider, Massive Networks is headquartered out of Louisville, Colorado. With years of experience under their belt, their team of...
SYS-CON Events announced today that Cloud Academy named "Bronze Sponsor" of 21st International Cloud Expo which will take place October 31 - November 2, 2017 at the Santa Clara Convention Center in Santa Clara, CA. Cloud Academy is the industry’s most innovative, vendor-neutral cloud technology training platform. Cloud Academy provides continuous learning solutions for individuals and enterprise teams for Amazon Web Services, Microsoft Azure, Google Cloud Platform, and the most popular cloud com...
SYS-CON Events announced today that Cloudistics, an on-premises cloud computing company, has been named “Bronze Sponsor” of SYS-CON's 21st International Cloud Expo, which will take place on Oct 31 - Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Cloudistics delivers a complete public cloud experience with composable on-premises infrastructures to medium and large enterprises. Its software-defined technology natively converges network, storage, compute, virtualization, and ...
SYS-CON Events announced today that CHEETAH Training & Innovation will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place on Oct. 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. CHEETAH Training & Innovation is a cloud consulting and IT training firm specializing in improving clients cloud strategies and infrastructures for medium to large companies.
SYS-CON Events announced today that Datanami has been named “Media Sponsor” of SYS-CON's 21st International Cloud Expo, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Datanami is a communication channel dedicated to providing insight, analysis and up-to-the-minute information about emerging trends and solutions in Big Data. The publication sheds light on all cutting-edge technologies including networking, storage and applications, and thei...
The current age of digital transformation means that IT organizations must adapt their toolset to cover all digital experiences, beyond just the end users’. Today’s businesses can no longer focus solely on the digital interactions they manage with employees or customers; they must now contend with non-traditional factors. Whether it's the power of brand to make or break a company, the need to monitor across all locations 24/7, or the ability to proactively resolve issues, companies must adapt to...