The Cube - Strata Conference 2012 - Paul Doscher, Lucid Imagination, with John Furrier and Dave Vellante Big Data makes every employee smarter . What we’re seeing as it relates to Big Data is a requirement for an app to be smarter, with better awareness. With the overwhelming amount of information, apps need to be smarter and aware, looking at user usage patterns and putting that feedback into the app ranking to understand [end user] roles and provide access to the right information. After a while, the system can anticipate what the user’s going to need to make more effective decisions. Talking about the role of IT and the role of end users, it’s going to be about the ability to integrate content contextually, and make it more valuable. The current paradigm is more poll-based. This has been true for some time. You have to know the questions the user’s going to ask before building an effective app. With the next generation of apps, there’s going to be too much information and data. That [poll-based] paradigm is going to break. The system needs to be smarter and more context-aware to provide more answers as it culls through all the data. Consumer trends impact the CIO . So the relationship then, between the CIO/IT and the end user, is the ability to build that platform where the information is consolidated, and the user interface is easier to use than ever before. The original threshold will be what the user experience currently is on the web. We’re moving to a space where information is displayed on all sorts of devices, through Business Intelligence dashboards and different types of user interfaces. Information access is built in phases, based on patterns you’ve been researching, etc. The ultimate phase is when the system starts to provide answers, almost using Artificial Intelligence in a way where it can aggregate content, look at different variables and provide different possible outcomes. If you think about it, based on everything we’ve been able to read and understand, is similar to what IBM’s trying to do with Watson in the healthcare space. A doctor can type in the specifics around a particular patient and Watson can go through the clinical data and come back with a statistical probability of outcomes the patient might have. Watson can go further and say, “here are the best treatments for this particular individual” based on genetics, location, gender, age and other data points. LucidWorks builds the plumbing to bring data full circle . Right now we’re working on the plumbing. It’s not sexy, but someone’s gotta do it. Unless you have the underpinnings to contextualize this for the user, there’s UI developers for that particular user community. We don’t have any preconceived ideas around that. In more cases, the enterprise developing the app understands the best opportunity to display that information properly. The way we look at is is sort of the three-level architectural tech stack. If you consider Hadoop to be the new generation data stack, the best Hadoop implementation will be limited by people’s ability to get content out of it. This requires workflow, machine learning technology, search as a fundamental component, security, cluster awareness, understanding how data is moving, etc. When you break down the plumbing, it’s all about providing the parts. What we’ve done with our big data product is to pull together search technology and other Apache products that provide those capabilities and put it inside a REST API, and let clients take advantage of those function calls. They don’t have to worry about if the data’s in Hadoop, or somewhere else. . Hear more from Doscher on #theCube . Doscher has plenty to say on the topic, his company and what we can expect from the Big Data space this year. See his latest interview with #theCube from O’Reilly Media’s Strata Conference, held last Spring.