Welcome!

Containers Expo Blog Authors: Eric Robertson, XebiaLabs Blog, Elizabeth White, Stefan Bernbo, Liz McMillan

Related Topics: @CloudExpo, Java IoT, Microservices Expo, Linux Containers, Containers Expo Blog, @BigDataExpo

@CloudExpo: Article

Take Control of Your Schemalessness with Dynamic Schemas

Addressing the inflexibility of structured data by enabling schemaless data to be dynamically and logically structured

Static data structures have been at the heart of data processing tools since the dawn of computing, but they have always limited the flexibility of the organization leveraging the data. Recently, the rise of flexible formats like JSON have led to schemaless data as an attempt to increase agility. However, schemaless data have proven difficult to work with, because of hidden rigid structure in the form of implied schemas.

EnterpriseWeb addresses the problems of both the inflexibility of structured data as well as the impracticality of schemaless data, by enabling schemaless data to be dynamically and logically structured.

From the fixed-length fields of the 1950s, to the relational structures of modern database management systems, to the semistructured data formats XML and JSON, the structure of our data has always informed code about how it should be processed. Data are defined by their relationships, and we used to hard-code those relationships into rigid structures. That approach allows only one static view, which is difficult to work with, and even more difficult to change. Nevertheless, such rigid data structures - and the models that represent them - are an integral part of enterprise information management.

Traditional relational database management systems (RDBMSs) exemplify this point with their static entity-relationship models (ERMs) and tightly interconnected data structures. XML improves this situation slightly, allowing semi-structured information, but schemas still constrain flexibility and performance. With both approaches, fixed definitions, views, and reports limit the ability for businesses to freely transform information into insight and become obstacles to systemwide change.

The Rise of Schemalessness
This challenge of inflexible data structures has given rise to schemaless data. With JSON in particular, we can create whatever data structure we like when we author data. We don't have to shoehorn data into rigid data structures, thus allowing every record to have its own structure.

But there is a problem with schemaless data. Consider this simple task: how do you create a query for all the addresses in a particular Zip Code if every record has a different name or format for Zip Code? Schemalessness, after all, isn't magic - even schemaless data require some kind of metadata so the code will know how to process such information, what software development guru Martin Fowler calls an implied schema.

Implied schemas represent the structure inherent in any data record. If each address record has its own format, then that format provides the implied schema for that record. Dealing with implied schemas thus falls to the developer, who must figure out how to code software to process these implied schemas, which are different for each and every record.

In Fowler's tutorial on schemalessness, he explains the pros and cons of implied schemas. Despite acknowledging the power of schemalessness to support more flexible and responsive user experiences, he recommends avoiding it and implied schemas for developer convenience. Good advice with respect to traditional software, but the world of data is changing. Today we live in an increasingly schemaless world, where more often than not, the structure of our data is fluid or nonexistent.

Raising the Discussion to Dynamic Schemas
Fowler makes it clear that in the past it has been impractical from the developer's perspective to work systematically with schemaless data, because implied schemas are difficult to deal with. After all, structure is itself useful, and isn't the problem per se. Rather, how to avoid the limitations of static structure without falling into the trap of unmanageable schemaless data that is the real challenge.

EnterpriseWeb's unique approach to modeling solves this critically important challenge by leveraging dynamic schemas that have flexible, metadata-driven relationships with underlying information. Using metadata this way separates concerns, letting people consider relationships from multiple perspectives, rather than from a single static point of view. In addition, it's now possible to change and extend metadata to meet diverse business needs without disruption.

Instead of settling for complex ERMs with their inflexible, tightly coupled data structures or dealing with the coding complexities of implied schemas, developers can project dynamic schemas from the metadata simply by writing different transformations. As a result, dynamic schemas are developer friendly and dynamic - a welcome change from the difficult problem of schemalessness.

Add an Agent for Performance
So far so good, but how do we build software to process all such data in a general way, freeing ourselves from custom coding for implicit schemas? The solution is an intelligent agent.

EnterpriseWeb's intelligent agent, SmartAlex™, is a distributable transaction manager that resolves dynamic schemas for each interaction. Every human or system client interaction is a request for SmartAlex to interpret dynamic schemas (as well as other models and additional metadata) and translate them to a context-specific set of resources in order to construct a custom response.

This Agent-Oriented approach maximizes performance for such dynamic computing. In the background, SmartAlex handles all run time connection and transformation details, sparing programmers from manually integrating resources for varied and unanticipated uses, greatly improving IT productivity while enabling business agility.

SmartAlex logs all system events, indexes all new and updated resources, and tags all changes in relationships for detailed and navigable audit history. This practice creates a feedback loop as SmartAlex leverages the same indexed logs to guide its execution. Data, code, and user interface components, as well as connectors for federated services, systems, databases, and devices, can be updated or replaced without breaking related apps and processes - as SmartAlex is ‘aware' of the changes. In this way EnterpriseWeb supports real time exception and change management for resilient solutions that can evolve naturally.

The EnterpriseWeb Take
Schemalessness was a reaction to the limitations of structured data. People struggled with the constraints of static structure, and figured that if they simply got rid of structure, then the problem would go away. But this move was merely a shell game, as the limitations of fixed schemas shifted to implied schemas, now without the benefits of structure to inform the code responsible for their processing.

The solution is to raise the level of abstraction, and instead of arguing over fixed vs. implied schemas, to work at the dynamic schema level. Such an approach is model-driven, allowing application designers to build models that capture their data structures, and allowing an intelligent agent to use the metadata each model represents to meet the specific needs of each interaction. The real lesson here is that the solution to resolving the challenge of schemalessness combines both dynamic schemas and the action of the agent. Stay tuned to my next newsletter for more information.

More Stories By Jason Bloomberg

Jason Bloomberg is the leading expert on architecting agility for the enterprise. As president of Intellyx, Mr. Bloomberg brings his years of thought leadership in the areas of Cloud Computing, Enterprise Architecture, and Service-Oriented Architecture to a global clientele of business executives, architects, software vendors, and Cloud service providers looking to achieve technology-enabled business agility across their organizations and for their customers. His latest book, The Agile Architecture Revolution (John Wiley & Sons, 2013), sets the stage for Mr. Bloomberg’s groundbreaking Agile Architecture vision.

Mr. Bloomberg is perhaps best known for his twelve years at ZapThink, where he created and delivered the Licensed ZapThink Architect (LZA) SOA course and associated credential, certifying over 1,700 professionals worldwide. He is one of the original Managing Partners of ZapThink LLC, the leading SOA advisory and analysis firm, which was acquired by Dovel Technologies in 2011. He now runs the successor to the LZA program, the Bloomberg Agile Architecture Course, around the world.

Mr. Bloomberg is a frequent conference speaker and prolific writer. He has published over 500 articles, spoken at over 300 conferences, Webinars, and other events, and has been quoted in the press over 1,400 times as the leading expert on agile approaches to architecture in the enterprise.

Mr. Bloomberg’s previous book, Service Orient or Be Doomed! How Service Orientation Will Change Your Business (John Wiley & Sons, 2006, coauthored with Ron Schmelzer), is recognized as the leading business book on Service Orientation. He also co-authored the books XML and Web Services Unleashed (SAMS Publishing, 2002), and Web Page Scripting Techniques (Hayden Books, 1996).

Prior to ZapThink, Mr. Bloomberg built a diverse background in eBusiness technology management and industry analysis, including serving as a senior analyst in IDC’s eBusiness Advisory group, as well as holding eBusiness management positions at USWeb/CKS (later marchFIRST) and WaveBend Solutions (now Hitachi Consulting).

@ThingsExpo Stories
Successful digital transformation requires new organizational competencies and capabilities. Research tells us that the biggest impediment to successful transformation is human; consequently, the biggest enabler is a properly skilled and empowered workforce. In the digital age, new individual and collective competencies are required. In his session at 19th Cloud Expo, Bob Newhouse, CEO and founder of Agilitiv, drew together recent research and lessons learned from emerging and established compa...
"IoT is going to be a huge industry with a lot of value for end users, for industries, for consumers, for manufacturers. How can we use cloud to effectively manage IoT applications," stated Ian Khan, Innovation & Marketing Manager at Solgeniakhela, in this SYS-CON.tv interview at @ThingsExpo, held November 3-5, 2015, at the Santa Clara Convention Center in Santa Clara, CA.
Financial Technology has become a topic of intense interest throughout the cloud developer and enterprise IT communities. Accordingly, attendees at the upcoming 20th Cloud Expo at the Javits Center in New York, June 6-8, 2017, will find fresh new content in a new track called FinTech.
Information technology is an industry that has always experienced change, and the dramatic change sweeping across the industry today could not be truthfully described as the first time we've seen such widespread change impacting customer investments. However, the rate of the change, and the potential outcomes from today's digital transformation has the distinct potential to separate the industry into two camps: Organizations that see the change coming, embrace it, and successful leverage it; and...
Bert Loomis was a visionary. This general session will highlight how Bert Loomis and people like him inspire us to build great things with small inventions. In their general session at 19th Cloud Expo, Harold Hannon, Architect at IBM Bluemix, and Michael O'Neill, Strategic Business Development at Nvidia, discussed the accelerating pace of AI development and how IBM Cloud and NVIDIA are partnering to bring AI capabilities to "every day," on-demand. They also reviewed two "free infrastructure" pr...
"Dice has been around for the last 20 years. We have been helping tech professionals find new jobs and career opportunities," explained Manish Dixit, VP of Product and Engineering at Dice, in this SYS-CON.tv interview at 19th Cloud Expo, held November 1-3, 2016, at the Santa Clara Convention Center in Santa Clara, CA.
Extracting business value from Internet of Things (IoT) data doesn’t happen overnight. There are several requirements that must be satisfied, including IoT device enablement, data analysis, real-time detection of complex events and automated orchestration of actions. Unfortunately, too many companies fall short in achieving their business goals by implementing incomplete solutions or not focusing on tangible use cases. In his general session at @ThingsExpo, Dave McCarthy, Director of Products...
"At ROHA we develop an app called Catcha. It was developed after we spent a year meeting with, talking to, interacting with senior citizens watching them use their smartphones and talking to them about how they use their smartphones so we could get to know their smartphone behavior," explained Dave Woods, Chief Innovation Officer at ROHA, in this SYS-CON.tv interview at 19th Cloud Expo, held November 1-3, 2016, at the Santa Clara Convention Center in Santa Clara, CA.
SYS-CON Events has announced today that Roger Strukhoff has been named conference chair of Cloud Expo and @ThingsExpo 2017 New York. The 20th Cloud Expo and 7th @ThingsExpo will take place on June 6-8, 2017, at the Javits Center in New York City, NY. "The Internet of Things brings trillions of dollars of opportunity to developers and enterprise IT, no matter how you measure it," stated Roger Strukhoff. "More importantly, it leverages the power of devices and the Internet to enable us all to im...
We are always online. We access our data, our finances, work, and various services on the Internet. But we live in a congested world of information in which the roads were built two decades ago. The quest for better, faster Internet routing has been around for a decade, but nobody solved this problem. We’ve seen band-aid approaches like CDNs that attack a niche's slice of static content part of the Internet, but that’s it. It does not address the dynamic services-based Internet of today. It does...
The many IoT deployments around the world are busy integrating smart devices and sensors into their enterprise IT infrastructures. Yet all of this technology – and there are an amazing number of choices – is of no use without the software to gather, communicate, and analyze the new data flows. Without software, there is no IT. In this power panel at @ThingsExpo, moderated by Conference Chair Roger Strukhoff, Dave McCarthy, Director of Products at Bsquare Corporation; Alan Williamson, Principal...
"ReadyTalk is an audio and web video conferencing provider. We've really come to embrace WebRTC as the platform for our future of technology," explained Dan Cunningham, CTO of ReadyTalk, in this SYS-CON.tv interview at WebRTC Summit at 19th Cloud Expo, held November 1-3, 2016, at the Santa Clara Convention Center in Santa Clara, CA.
20th Cloud Expo, taking place June 6-8, 2017, at the Javits Center in New York City, NY, will feature technical sessions from a rock star conference faculty and the leading industry players in the world. Cloud computing is now being embraced by a majority of enterprises of all sizes. Yesterday's debate about public vs. private has transformed into the reality of hybrid cloud: a recent survey shows that 74% of enterprises have a hybrid cloud strategy.
IoT solutions exploit operational data generated by Internet-connected smart “things” for the purpose of gaining operational insight and producing “better outcomes” (for example, create new business models, eliminate unscheduled maintenance, etc.). The explosive proliferation of IoT solutions will result in an exponential growth in the volume of IoT data, precipitating significant Information Governance issues: who owns the IoT data, what are the rights/duties of IoT solutions adopters towards t...
WebRTC is the future of browser-to-browser communications, and continues to make inroads into the traditional, difficult, plug-in web communications world. The 6th WebRTC Summit continues our tradition of delivering the latest and greatest presentations within the world of WebRTC. Topics include voice calling, video chat, P2P file sharing, and use cases that have already leveraged the power and convenience of WebRTC.
Unsecured IoT devices were used to launch crippling DDOS attacks in October 2016, targeting services such as Twitter, Spotify, and GitHub. Subsequent testimony to Congress about potential attacks on office buildings, schools, and hospitals raised the possibility for the IoT to harm and even kill people. What should be done? Does the government need to intervene? This panel at @ThingExpo New York brings together leading IoT and security experts to discuss this very serious topic.
In his keynote at 18th Cloud Expo, Andrew Keys, Co-Founder of ConsenSys Enterprise, provided an overview of the evolution of the Internet and the Database and the future of their combination – the Blockchain. Andrew Keys is Co-Founder of ConsenSys Enterprise. He comes to ConsenSys Enterprise with capital markets, technology and entrepreneurial experience. Previously, he worked for UBS investment bank in equities analysis. Later, he was responsible for the creation and distribution of life sett...
20th Cloud Expo, taking place June 6-8, 2017, at the Javits Center in New York City, NY, will feature technical sessions from a rock star conference faculty and the leading industry players in the world. Cloud computing is now being embraced by a majority of enterprises of all sizes. Yesterday's debate about public vs. private has transformed into the reality of hybrid cloud: a recent survey shows that 74% of enterprises have a hybrid cloud strategy.
Whether your IoT service is connecting cars, homes, appliances, wearable, cameras or other devices, one question hangs in the balance – how do you actually make money from this service? The ability to turn your IoT service into profit requires the ability to create a monetization strategy that is flexible, scalable and working for you in real-time. It must be a transparent, smoothly implemented strategy that all stakeholders – from customers to the board – will be able to understand and comprehe...
An IoT product’s log files speak volumes about what’s happening with your products in the field, pinpointing current and potential issues, and enabling you to predict failures and save millions of dollars in inventory. But until recently, no one knew how to listen. In his session at @ThingsExpo, Dan Gettens, Chief Research Officer at OnProcess, discussed recent research by Massachusetts Institute of Technology and OnProcess Technology, where MIT created a new, breakthrough analytics model for ...