Click here to close now.

Welcome!

Virtualization Authors: Elizabeth White, Carmen Gonzalez, Robin Kent, John Treadway, Liz McMillan

Related Topics: Virtualization

Virtualization: Article

NoSQL and Data Virtualization - Soon to Be Best Friends

Big Data is increasing the popularity of NoSQL; the challenge - data integration

According to a recent McKinsey Big Data report , "The amount of data in our world has been exploding. Companies capture trillions of bytes of information about their customers, suppliers, and operations, and millions of networked sensors are being embedding in the physical world in devices such as mobile phones and automobiles, sensing, creating and communicating data."

NoSQL is increasingly being adopted with the expansion of “big data” use cases. The challenge to enterprises is to integrate disparate NoSQL systems, each with its unique and non-standard API, with their traditional enterprise data sources.

Today’s evolved data virtualization platforms provide access to information from almost any type of data store and therefore are inherently well-suited to integrate NoSQL data stores into the enterprise.

NoSQL and Data Virtualization Defined
NoSQL data stores manage data that is not strictly tabular and relational. Beyond being non-relational, NoSQL data stores are typically distributed, open-source, and horizontally scalable, although there are exceptions for specific NoSQL data stores.

Data virtualization, has over the past eight years expanded its adoption among enterprises and government agencies due to its ability to evolve rapidly and incorporate the latest IT innovations. Originally limited to relational sources and business intelligence (BI) consumers, data virtualization today supports a wide range of sources including multi-dimensional stores, Web and data services, XML documents, analytic appliances, on- and off-premises applications and more. NoSQL data stores are the newest source type supported by data virtualization.

The NoSQL Data Stores Landscape
Although the original emergence of NoSQL data stores was motivated by Web-scale data, the movement has grown to encompass a wide variety of data stores that do not use SQL as their primary processing language. NoSQL data stores can be categorized as:

-- Tabular/Columnar Data Stores: Storing sparse tabular data, these stores look most like traditional tabular databases. Examples include Hadoop/HBase (Yahoo!), BigTable (Google), Hypertable, and VoltDB.

-- Document Stores: These sources store unstructured (e.g., text) or semi-structured (e.g., XML) documents. Examples include MongoDB, Mark Logic, and CouchDB.

-- Graph Databases: These NoSQL sources store graph-oriented data with nodes, edges and properties, and are commonly used to store associations in social networks. Examples include Neo4J, AllegroGraph, and FlockDB.

-- Key/Value Stores: These sources store simple key/value pairs like a traditional hash table. They are further subdivided into in-memory and disk-based solutions. This category of NoSQL systems probably has the largest number of members, each embodying slightly different characteristics. Examples include Memcached, Cassandra (Facebook), SimpleDB, Dynamo (Amazon), Voldemort (LinkedIn), and Kyoto Cabinet.

-- Object and Multi-value Databases: These types of stores preceded the NoSQL movement but they have found new life as part of the movement. Object databases store objects (as in object-oriented programming). Multi-value databases store tabular data, but individual cells can store multiple values. Examples include Objectivity, GemStone, and Unidata.

-- Miscellaneous Sources: Several other data stores can be classified as NoSQL stores, although they don’t fit into any of the categories above. Examples include GT.M, IBM Lotus/Domino, and the ISIS family.

Virtualizing NoSQL Data Store Sources
Data virtualization platforms provide a complete toolset for accessing, federating, abstracting, and delivering information from diverse sources. Access is typically done via standards-based protocols and APIs; for example, JDBC and ODBC for SQL-based sources, HTTP and SOAP for Web services, JMS for messages, and APIs for enterprise and cloud-based applications. Through these methods, source data is securely exposed from a single virtual location, regardless of how and where it is physically stored.

Although NoSQL access standards have yet to develop fully, each implementation provides a Java-based development API appropriate for accessing that type of NoSQL data. Data virtualization platforms typically use these APIs to access and integrate data. Three kinds of NoSQL systems are particularly suited for the data virtualization platform: tabular/columnar data, XML documents, and key-value stores.

How to Integrate Tabular/Columnar Data Stores
Because data virtualization platforms were originally designed for tabular data, retrieving and processing data from this category is a natural fit. The preferred data retrieval paradigm for tabular/columnar data stores leverages “table functions” in the FROM clause of a SQL statement. That is, a procedure resource that returns a cursor can be dropped into the data virtualization development environment as a table, where it will show up in the FROM clause of the SQL statement.

Tabular/columnar NoSQL data sources typically store very large data sets. Table function implementations should ensure sufficient data reduction within the source by leveraging input parameters. Also, the processing of large data sets can take a long time, so some form of caching may be prudent to retain the results for reuse.

This approach provides full access to the data in the underlying NoSQL source, and it will likely be sufficient for most near-term needs. However, more generic filtering and aggregation might be possible with the underlying NoSQL source, and purpose-built table functions provide only a limited interface to the data virtualization platform. If a particular NoSQL tabular data store becomes quite popular, expect data virtualization platform providers to develop a custom adapter that more fully integrates and leverages that data source’s specific capabilities.

How to Integrate XML Document Stores
Because XML document store sources leverage XQuery as their preferred data retrieval paradigm, data virtualization platforms with embedded XQuery engines (and XML as a native data type) can easily retrieve and further process documents from this category of NoSQL data store.

For a specific NoSQL XML document store, a minimum of two custom procedures can be implemented that leverage the NoSQL system’s Java API. Both procedures would return an XML document that can be further manipulated by any of the upstream XML manipulation functionality (e.g., XSLT transformations). The first procedure takes a document handle (unique identifier) as its input argument and leverages the API to retrieve and return that document. The second procedure takes an XQuery specification as its input argument and leverages the API to execute the query and return the results as a single document. Of course, additional procedures accepting more specific parameters could also be implemented, making integration into multiple views easier.

How to Integrate Key/Value Stores
Data virtualization platforms can integrate key/value stores in two ways. The first is as a simple custom SQL function. This function can be created so that it takes the key as a parameter and returns the value. This common function can then be used in SQL statements throughout the data virtualization platform.

The second leverages an in-memory key/value store as a cache target. This approach is best for small data sets or procedure results; it doesn’t work very well for large tabular data sets. Further, this form of cache integration is often challenged by the impedance mismatch between cached tabular data and cached key/value data (the cached data is opaque inside the key/value store), so the entire set must be retrieved for processing.

Key Takeaways
Web analytics, predictive analytics, voice-of-the-customer, churn, fraud, sensor-tracking and other “big data” use cases are accelerating demand  for NoSQL data stores as well as for the integration of NoSQL data with enterprise data.  Data virtualization, a more modern and versatile approach to data integration, is proving a successful solution to this fast growing problem.

More Stories By Robert Eve

Robert Eve is the EVP of Marketing at Composite Software, the data virtualization gold standard and co-author of Data Virtualization: Going Beyond Traditional Data Integration to Achieve Business Agility. Bob's experience includes executive level roles at leading enterprise software companies such as Mercury Interactive, PeopleSoft, and Oracle. Bob holds a Masters of Science from the Massachusetts Institute of Technology and a Bachelor of Science from the University of California at Berkeley.

@ThingsExpo Stories
So I guess we’ve officially entered a new era of lean and mean. I say this with the announcement of Ubuntu Snappy Core, “designed for lightweight cloud container hosts running Docker and for smart devices,” according to Canonical. “Snappy Ubuntu Core is the smallest Ubuntu available, designed for security and efficiency in devices or on the cloud.” This first version of Snappy Ubuntu Core features secure app containment and Docker 1.6 (1.5 in main release), is available on public clouds, and for ARM and x86 devices on several IoT boards. It’s a Trend! This announcement comes just as...
In this session we look at creating interactive communications via the web by adding messaging, file transfer, and group communication (group chat and audio/video conferencing) into the web experience. We will also discuss potential applications of this technology in areas including B2B, B2C, P2P, and gaming. Peter is Technical Director at Acision. He graduated from The University of Edinburgh in 2000 with a BSc (Hons) in Computer Science. After graduation Peter worked on a PSTN switch developing signalling stacks for SS7, ISDN and similar protocols and creating advanced routing and serv...
SYS-CON Events announced today the IoT Bootcamp – Jumpstart Your IoT Strategy, being held June 9–10, 2015, in conjunction with 16th Cloud Expo and Internet of @ThingsExpo at the Javits Center in New York City. This is your chance to jumpstart your IoT strategy. Combined with real-world scenarios and use cases, the IoT Bootcamp is not just based on presentations but includes hands-on demos and walkthroughs. We will introduce you to a variety of Do-It-Yourself IoT platforms including Arduino, Raspberry Pi, BeagleBone, Spark and Intel Edison. You will also get an overview of cloud technologies s...
Health care systems across the globe are under enormous strain, as facilities reach capacity and costs continue to rise. M2M and the Internet of Things have the potential to transform the industry through connected health solutions that can make care more efficient while reducing costs. In fact, Vodafone's annual M2M Barometer Report forecasts M2M applications rising to 57 percent in health care and life sciences by 2016. Lively is one of Vodafone's health care partners, whose solutions enable older adults to live independent lives while staying connected to loved ones. M2M will continue to gr...
SYS-CON Events announced today that Vicom Computer Services, Inc., a provider of technology and service solutions, will exhibit at SYS-CON's 16th International Cloud Expo®, which will take place on June 9-11, 2015, at the Javits Center in New York City, NY. They are located at booth #427. Vicom Computer Services, Inc. is a progressive leader in the technology industry for over 30 years. Headquartered in the NY Metropolitan area. Vicom provides products and services based on today’s requirements around Unified Networks, Cloud Computing strategies, Virtualization around Software defined Data Ce...
Dave will share his insights on how Internet of Things for Enterprises are transforming and making more productive and efficient operations and maintenance (O&M) procedures in the cleantech industry and beyond. Speaker Bio: Dave Landa is chief operating officer of Cybozu Corp (kintone US). Based in the San Francisco Bay Area, Dave has been on the forefront of the Cloud revolution driving strategic business development on the executive teams of multiple leading Software as a Services (SaaS) application providers dating back to 2004. Cybozu's kintone.com is a leading global BYOA (Build Your O...
Recent technology advances in miniaturization has positioned the wearables as the pinnacle of technology convergence with the human body. We inquire if wearables are mere standard miniaturized devices extended with the connectivity and present our views on considerations like design, applications, performance, efficiency, interoperability, usage scenarios, human device interaction and consequent trade-offs enabling wearables to impart optimal value.
“In the past year we've seen a lot of stabilization of WebRTC. You can now use it in production with a far greater degree of certainty. A lot of the real developments in the past year have been in things like the data channel, which will enable a whole new type of application," explained Peter Dunkley, Technical Director at Acision, in this SYS-CON.tv interview at @ThingsExpo, held Nov 4–6, 2014, at the Santa Clara Convention Center in Santa Clara, CA.
What exactly is a cognitive application? In her session at 16th Cloud Expo, Ashley Hathaway, Product Manager at IBM Watson, will look at the services being offered by the IBM Watson Developer Cloud and what that means for developers and Big Data. She'll explore how IBM Watson and its partnerships will continue to grow and help define what it means to be a cognitive service, as well as take a look at the offerings on Bluemix. She will also check out how Watson and the Alchemy API team up to offer disruptive APIs to developers.
The IoT Bootcamp is coming to Cloud Expo | @ThingsExpo on June 9-10 at the Javits Center in New York. Instructor. Registration is now available at http://iotbootcamp.sys-con.com/ Instructor Janakiram MSV previously taught the famously successful Multi-Cloud Bootcamp at Cloud Expo | @ThingsExpo in November in Santa Clara. Now he is expanding the focus to Janakiram is the founder and CTO of Get Cloud Ready Consulting, a niche Cloud Migration and Cloud Operations firm that recently got acquired by Aditi Technologies. He is a Microsoft Regional Director for Hyderabad, India, and one of the f...
The 17th International Cloud Expo has announced that its Call for Papers is open. 17th International Cloud Expo, to be held November 3-5, 2015, at the Santa Clara Convention Center in Santa Clara, CA, brings together Cloud Computing, APM, APIs, Microservices, Security, Big Data, Internet of Things, DevOps and WebRTC to one location. With cloud computing driving a higher percentage of enterprise IT budgets every year, it becomes increasingly important to plant your flag in this fast-expanding business opportunity. Submit your speaking proposal today!
In 2015, 4.9 billion connected "things" will be in use. By 2020, Gartner forecasts this amount to be 25 billion, a 410 percent increase in just five years. How will businesses handle this rapid growth of data? Hadoop will continue to improve its technology to meet business demands, by enabling businesses to access/analyze data in real time, when and where they need it. Cloudera's Chief Technologist, Eli Collins, will discuss how Big Data is keeping up with today's data demands and how in the future, data and analytics will be pervasive, embedded into every workflow, application and infra...
The best mobile applications are augmented by dedicated servers, the Internet and Cloud services. Mobile developers should focus on one thing: writing the next socially disruptive viral app. Thanks to the cloud, they can focus on the overall solution, not the underlying plumbing. From iOS to Android and Windows, developers can leverage cloud services to create a common cross-platform backend to persist user settings, app data, broadcast notifications, run jobs, etc. This session provides a high level technical overview of many cloud services available to mobile app developers, includi...
SYS-CON Media announced today that @WebRTCSummit Blog, the largest WebRTC resource in the world, has been launched. @WebRTCSummit Blog offers top articles, news stories, and blog posts from the world's well-known experts and guarantees better exposure for its authors than any other publication. @WebRTCSummit Blog can be bookmarked ▸ Here @WebRTCSummit conference site can be bookmarked ▸ Here
SYS-CON Events announced today that Ciqada will exhibit at SYS-CON's @ThingsExpo, which will take place on June 9-11, 2015, at the Javits Center in New York City, NY. Ciqada™ makes it easy to connect your products to the Internet. By integrating key components - hardware, servers, dashboards, and mobile apps - into an easy-to-use, configurable system, your products can quickly and securely join the internet of things. With remote monitoring, control, and alert messaging capability, you will meet your customers' needs of tomorrow - today! Ciqada. Let your products take flight. For more inform...
Can call centers hang up the phones for good? Intuitive Solutions did. WebRTC enabled this contact center provider to eliminate antiquated telephony and desktop phone infrastructure with a pure web-based solution, allowing them to expand beyond brick-and-mortar confines to a home-based agent model. It also ensured scalability and better service for customers, including MUY! Companies, one of the country's largest franchise restaurant companies with 232 Pizza Hut locations. This is one example of WebRTC adoption today, but the potential is limitless when powered by IoT.
GENBAND introduced its Real Time Communications (RTC) Client for Lync* to seamlessly combine real-time communications with Lync Instant Messaging (IM) and Presence. “We’re shaking up the economics of delivering Unified Communications (UC) and offering a compelling way to integrate previously bespoke communications technologies,” said Carl Baptiste, GENBAND’s Senior Vice President, Enterprise Solutions. “We’re offering enterprises the best of both worlds by combining our own high availability voice, video and collaboration with Lync’s IM and Presence; creating a single, web centric, client. O...
SYS-CON Events announced today that GENBAND, a leading developer of real time communications software solutions, has been named “Silver Sponsor” of SYS-CON's WebRTC Summit, which will take place on June 9-11, 2015, at the Javits Center in New York City, NY. The GENBAND team will be on hand to demonstrate their newest product, Kandy. Kandy is a communications Platform-as-a-Service (PaaS) that enables companies to seamlessly integrate more human communications into their Web and mobile applications - creating more engaging experiences for their customers and boosting collaboration and productiv...
SYS-CON Events announced today that SoftLayer, an IBM company, has been named “Gold Sponsor” of SYS-CON's 16th International Cloud Expo®, which will take place June 9-11, 2015 at the Javits Center in New York City, NY, and the 17th International Cloud Expo®, which will take place November 3–5, 2015 at the Santa Clara Convention Center in Santa Clara, CA. SoftLayer operates a global cloud infrastructure platform built for Internet scale. With a global footprint of data centers and network points of presence, SoftLayer provides infrastructure as a service to leading-edge customers ranging from ...
SYS-CON Events announced today that BroadSoft, the leading global provider of Unified Communications and Collaboration (UCC) services to operators worldwide, has been named “Gold Sponsor” of SYS-CON's WebRTC Summit, which will take place on June 9-11, 2015, at the Javits Center in New York City, NY. BroadSoft is the leading provider of software and services that enable mobile, fixed-line and cable service providers to offer Unified Communications over their Internet Protocol networks. The Company’s core communications platform enables the delivery of a range of enterprise and consumer calling...