01. Matthew Hunt, Bloomberg, Visits theCUBE . (00:20)
02. Enthusiasm Surrounding Spark. (00:55)
03. Understanding the Origins of Hadoop and It's Pieces. (02:04)
04. The Secrets of Spark SQL. (06:27)
05. Making Streaming and Optimizers Work. (11:38)
06. Solutions that Bloomberg Is Providing. (16:48)
07. People Matter to Everything. (20:13)
#theCUBE #Bloomberg #Spark #SparkSummit #SparkSummitEast
--- ---
Demystifying the complexities of Spark and Hadoop | #SparkSummit
by Betsy Amy-Vogt | Feb 18, 2016
There is a lot of confusion around about Spark and Hadoop, but Matthew Hunt, head of Big Data at Bloomberg LP, thinks this is only due to a lack of understanding.
In an interview at Spark Summit East 2016 at the New York Hilton Midtown in NYC, Hunt talked with Jeff Frick and George Gilbert, cohosts of theCUBE from the SiliconANGLE Media team, to demystify the complexities of working with Spark and Hadoop, as well as provide an analysis of how Spark and Hadoop fit within the market.
Hadoop developed to answer a practical problem
In a conversation that delved deep into the mechanics of Big Data frameworks, Hunt explained the differences between Spark and Hadoop, answering where they come from, where they are going, what they do today and how they fit together.
For example, Hadoop was created to resolve a practical problem — how to download and index the web economically — by engineers rolling up their sleeves to solve real issues. As the platform grew, layers were added on top and the complexity and number of tools grew, yet the instruction set remained basic. Spark was developed with a more complicated instruction set that increased its speed yet simplified the many tools of Hadoop into one.
Spark takes language constructs and makes them performant
Hunt gave a practical example to help viewers create a mental model of Spark: When you compile a program, you write code, hit a button and the computer turns it into machine-level instructions. The same thing happens in Spark — it has an instruction set under the hood where whatever you are writing in is transformed.
Gilbert summarized this as Spark “taking language constructs and making it performant.”
Mental model shift
People assume what will be in a fast computation engine, but they are often not accurate and this causes confusion, said Hunt. “There is a mental model shift, and there are pieces that haven’t come together yet to make that happen.”
@theCUBE
#SparkSummit
Forgot Password
Almost there!
We just sent you a verification email. Please verify your account to gain access to
Spark Summit East 2016 | New York. If you don’t think you received an email check your
spam folder.
In order to sign in, enter the email address you used to registered for the event. Once completed, you will receive an email with a verification link. Open this link to automatically sign into the site.
Register For Spark Summit East 2016 | New York
Please fill out the information below. You will recieve an email with a verification link confirming your registration. Click the link to automatically sign into the site.
You’re almost there!
We just sent you a verification email. Please click the verification button in the email. Once your email address is verified, you will have full access to all event content for Spark Summit East 2016 | New York.
I want my badge and interests to be visible to all attendees.
Checking this box will display your presense on the attendees list, view your profile and allow other attendees to contact you via 1-1 chat. Read the Privacy Policy. At any time, you can choose to disable this preference.
Select your Interests!
add
Upload your photo
Uploading..
OR
Connect via Twitter
Connect via Linkedin
EDIT PASSWORD
Share
Forgot Password
Almost there!
We just sent you a verification email. Please verify your account to gain access to
Spark Summit East 2016 | New York. If you don’t think you received an email check your
spam folder.
In order to sign in, enter the email address you used to registered for the event. Once completed, you will receive an email with a verification link. Open this link to automatically sign into the site.
Sign in to gain access to Spark Summit East 2016 | New York
Please sign in with LinkedIn to continue to Spark Summit East 2016 | New York. Signing in with LinkedIn ensures a professional environment.
Are you sure you want to remove access rights for this user?
Details
Manage Access
email address
Community Invitation
Matthew Hunt, Bloomberg | Spark Summit East 2016
01. Matthew Hunt, Bloomberg, Visits theCUBE . (00:20)
02. Enthusiasm Surrounding Spark. (00:55)
03. Understanding the Origins of Hadoop and It's Pieces. (02:04)
04. The Secrets of Spark SQL. (06:27)
05. Making Streaming and Optimizers Work. (11:38)
06. Solutions that Bloomberg Is Providing. (16:48)
07. People Matter to Everything. (20:13)
#theCUBE #Bloomberg #Spark #SparkSummit #SparkSummitEast
--- ---
Demystifying the complexities of Spark and Hadoop | #SparkSummit
by Betsy Amy-Vogt | Feb 18, 2016
There is a lot of confusion around about Spark and Hadoop, but Matthew Hunt, head of Big Data at Bloomberg LP, thinks this is only due to a lack of understanding.
In an interview at Spark Summit East 2016 at the New York Hilton Midtown in NYC, Hunt talked with Jeff Frick and George Gilbert, cohosts of theCUBE from the SiliconANGLE Media team, to demystify the complexities of working with Spark and Hadoop, as well as provide an analysis of how Spark and Hadoop fit within the market.
Hadoop developed to answer a practical problem
In a conversation that delved deep into the mechanics of Big Data frameworks, Hunt explained the differences between Spark and Hadoop, answering where they come from, where they are going, what they do today and how they fit together.
For example, Hadoop was created to resolve a practical problem — how to download and index the web economically — by engineers rolling up their sleeves to solve real issues. As the platform grew, layers were added on top and the complexity and number of tools grew, yet the instruction set remained basic. Spark was developed with a more complicated instruction set that increased its speed yet simplified the many tools of Hadoop into one.
Spark takes language constructs and makes them performant
Hunt gave a practical example to help viewers create a mental model of Spark: When you compile a program, you write code, hit a button and the computer turns it into machine-level instructions. The same thing happens in Spark — it has an instruction set under the hood where whatever you are writing in is transformed.
Gilbert summarized this as Spark “taking language constructs and making it performant.”
Mental model shift
People assume what will be in a fast computation engine, but they are often not accurate and this causes confusion, said Hunt. “There is a mental model shift, and there are pieces that haven’t come together yet to make that happen.”
@theCUBE
#SparkSummit
