Brett Rudenstein, WANdisco, at Big Data NYC 2013 with John Furrier and Jeff Kelly
The three discuss WANdisco's unique patented technology that allows Hadoop to run across a wide area network despite severe network failures.
Rudenstein performs a demonstration of an active-active environment involving two data centers that are about 3,000 miles away. He shows how one can drill down on a wan-link that gives an exposed view of the data center. Rudenstein explains, "....across a wide area network we stand up a single HDFS." The demonstration involves two data centers and running MapReduce to illustrate data replication across the two.
In his typical direct fashion, Kelly asks, "What does this allow an enterprise to do that it couldn't do before?" Rudenstein explains that the new advantage is for disaster recovery, failover is virtually zero because the other site is already active. In his demonstration, both sites are active, allowing users to ingest and run data into both sites. He contrasts this with the problems typically encountered with distCp and parallel data ingest methods, stating that typically, a problem arises when a file is not copied if it has the same size and name as its predecessor. Rudenstein further explains, "After a couple of months you see clusters diverge. And, then it's a constant manual effort to figure out the differences between the data centers."
Vellante enquires, "How does a sys-admin get behind this?" Rudenstein says their task would be to check and run sums against each cluster individually to make sure the files line up; when they don't, they have to ask the users about which files are correct.
Continuing with the demonstration, Rudenstein reviews what happens when a local namenode fails by rebooting it. Despite the failure, the job continues uninterrupted. Now, as the namenode comes back online the program begins its "self-healing" process. Rudenstein says, "It will learn from the other namenodes that it's behind in the global sequence...it will get caught up and then become an active participant in the cluster." He throws one more failure at it, causing a complete wan separation. At the end of the demonstration, Rudenstein shows how the technology resumes replicating the data despite the failures.
Vellante explains that the demonstration was not trivial. He asks if there is any other technology that can provide resiliency like this. Rudenstein responds that most technologies are typically limited because latency is a big challenge. WANdisco's technology is time independent, wan independent, and does not utilize hardware; it is also a share nothing architecture.
@thecube
#BigDataNYC
Forgot Password
Almost there!
We just sent you a verification email. Please verify your account to gain access to
BigData NYC 2013 | New York. If you don’t think you received an email check your
spam folder.
In order to sign in, enter the email address you used to registered for the event. Once completed, you will receive an email with a verification link. Open this link to automatically sign into the site.
Register For BigData NYC 2013 | New York
Please fill out the information below. You will recieve an email with a verification link confirming your registration. Click the link to automatically sign into the site.
You’re almost there!
We just sent you a verification email. Please click the verification button in the email. Once your email address is verified, you will have full access to all event content for BigData NYC 2013 | New York.
I want my badge and interests to be visible to all attendees.
Checking this box will display your presense on the attendees list, view your profile and allow other attendees to contact you via 1-1 chat. Read the Privacy Policy. At any time, you can choose to disable this preference.
Select your Interests!
add
Upload your photo
Uploading..
OR
Connect via Twitter
Connect via Linkedin
EDIT PASSWORD
Share
Forgot Password
Almost there!
We just sent you a verification email. Please verify your account to gain access to
BigData NYC 2013 | New York. If you don’t think you received an email check your
spam folder.
In order to sign in, enter the email address you used to registered for the event. Once completed, you will receive an email with a verification link. Open this link to automatically sign into the site.
Sign in to gain access to BigData NYC 2013 | New York
Please sign in with LinkedIn to continue to BigData NYC 2013 | New York. Signing in with LinkedIn ensures a professional environment.
Brett Rudenstein, WANdisco, at Big Data NYC 2013 with John Furrier and Jeff Kelly
The three discuss WANdisco's unique patented technology that allows Hadoop to run across a wide area network despite severe network failures.
Rudenstein performs a demonstration of an active-active environment involving two data centers that are about 3,000 miles away. He shows how one can drill down on a wan-link that gives an exposed view of the data center. Rudenstein explains, "....across a wide area network we stand up a single HDFS." The demonstration involves two data centers and running MapReduce to illustrate data replication across the two.
In his typical direct fashion, Kelly asks, "What does this allow an enterprise to do that it couldn't do before?" Rudenstein explains that the new advantage is for disaster recovery, failover is virtually zero because the other site is already active. In his demonstration, both sites are active, allowing users to ingest and run data into both sites. He contrasts this with the problems typically encountered with distCp and parallel data ingest methods, stating that typically, a problem arises when a file is not copied if it has the same size and name as its predecessor. Rudenstein further explains, "After a couple of months you see clusters diverge. And, then it's a constant manual effort to figure out the differences between the data centers."
Vellante enquires, "How does a sys-admin get behind this?" Rudenstein says their task would be to check and run sums against each cluster individually to make sure the files line up; when they don't, they have to ask the users about which files are correct.
Continuing with the demonstration, Rudenstein reviews what happens when a local namenode fails by rebooting it. Despite the failure, the job continues uninterrupted. Now, as the namenode comes back online the program begins its "self-healing" process. Rudenstein says, "It will learn from the other namenodes that it's behind in the global sequence...it will get caught up and then become an active participant in the cluster." He throws one more failure at it, causing a complete wan separation. At the end of the demonstration, Rudenstein shows how the technology resumes replicating the data despite the failures.
Vellante explains that the demonstration was not trivial. He asks if there is any other technology that can provide resiliency like this. Rudenstein responds that most technologies are typically limited because latency is a big challenge. WANdisco's technology is time independent, wan independent, and does not utilize hardware; it is also a share nothing architecture.
@thecube
#BigDataNYC
