Is GPU technology giving Spark a flame? | #BigDataNYC by Marlene Den Bleyker | Sep 26, 2016 Gearing up for three days of coverage of BigDataNYC 2016 at 37 Pillars in New York City, the SiliconANGLE Media team and NVIDIA Corp. hosted The Future: AI-Driven Analytics, An Evening of Deep Learning. This event kicked off the conversation about deriving benefits from Big Data to advanced Artificial Intelligence (AI) and Machine Learning (ML). An event panel met to talk about deep learning, what it means, where it’s headed and implications for next-gen apps. Panelists Jim McHugh, VP and GM of NVIDIA Corp.; Randy Swanberg, distinguished engineer at IBM; Ram Sriharsha, product manager, Apache Spark, at DataBricks, Inc.; and Josh Patterson, director of Field Engineering at Skymind joined host George Gilbert, (@ggilbert41), Big Data analyst at Wikibon and theCUBE cohost (from the SiliconANGLE Media team), to talk about deep learning and where is going in the future. GPU and Spark: Getting better answers Gilbert began the panel discussion by saying that the real advance that is impending right now is the magnitude of cores that use GPUs (Graphics Processing Unit) as auxiliary processing units, which he feels is going to change the future of where computation will go. He also noted that in ML reality many different levels of good answers and when there is more compute power coming from GPUs we can get better answers. Rethinking we apply machine learning technology “In reality, if you think about what’s going on in ML and deep learning and AI, it’s really changing rapidly. It’s really why we are all up here together talking about Spark and deep learning,” McHugh began. He stated that many companies are considering GPU technology for ML, and the growth of this technology is bringing companies together. McHugh also said that Spark is what most customers are using for their data management. “Spark and GPU are two technologies that need to work together,” he pointed out. McHugh was asked to pick a panelist and ask about where their company is in the ecosystem. He did mention that all the panelists have a great deal in common and the all view the ecosystem together. Narrowing the skills gap McHugh turned to IBM’s Swanberg to get his take on Spark. “The value proposition of Spark is really about simplification, unifying data sources, time-to-value, productivity … you can actually focus on the workload that your wanting to distribute at scale,” Swanberg responded. He also revealed that his company is working on accelerating Spark ML APIs without the Spark programmer even knowing it’s happening. Therefore, the skill gap is reduced because you don’t have to be a GPU programmer if acceleration happens behind the scenes, he explained. The best of both worlds Gilbert asked Sriharsha from Databricks, what qualitative changes the company is seeing. Sriharsha replied by saying Spark has a good deal of support on ML, and much of it is distributed learning that when running on GPUs becomes faster. He also referenced frameworks and how all these technologies will bring together the best of both worlds. Big Data and the Fortune 500 Patterson talked about his days of running Hadoop clusters and looking at what the Fortune 500 was searching for in this technology. He noted that with a cleaner interface, now Fortune 500 companies can build apps on Hadoop and run Spark faster, and with plug-in GPUs the technology is better. Moderated by George Gilbert, Big Data Analyst, SiliconANGLE Media Inc. Jim McHugh (@JimMcHugh), VP and GM, NVIDIA Randy Swanberg, Distinguished Engineer, IBM Power Systems Ram Sriharsha (@halfabrane), Product Manager, Apache Spark, DataBricks Josh Patterson (@jpatanooga), Director of Field Engineering, Skymind on the #theCUBE at #BigDataNYC 2016 New York, New York