In the excitement about Big Data, what is being missed is that large enterprises are actually drowning in Petabytes of historical data, both structured and unstructured, generated by internal systems from ERP to e-mail, says consultant, lawyer and records management expert Randolph Kahn. And despite the commonly held myth that "storage is cheap", this data is costing those companies millions of dollars annually in direct storage costs plus huge amounts in "soft costs" ranging from the 20%+ of executives' time wasted searching for the significant information buried in that largely useless data to the huge cost of finding relevant information in the case of a legal action.
"I started my career as a trial lawyer," Kahn told Wikibon Co-Founder and Chief Analyst David Vellante in an interview in the Cube in Boston (full video below). "Litigation response tells us over & over & over again that companies don't have their act together. So I would tell you that harnessing their data in a Big Data context is aspirational for most organizations. I would say ubiquitous mismanagement is much more the theme of the day."
This data is supposed to be managed through records management, but that has failed completely. The rules are too complex, they are not applied evenly or at all to much company data. Nobody has clear ownership of any of the data, and while IT owns the storage systems, corporate counsel constantly warns them to be very careful about what is deleted, which means that all the data is saved.
"Most IT professionals don't think of information as having a life-cycle," he says. "Today everything is electronic, and it is all generating data. At the end of the information life-cycle, the idea of parking it somewhere in a repository, a shared drive, it's an expense. It's a liability. And I think IT professionals need to understand what's in that life-cycle."
Much of that old data is actually valueless, unneeded, and a source of legal risk to the organization. And with Petabytes sitting in a wide variety of systems, in multiple formats, the physical act of reviewing all that data is too massive and expensive for companies to undertake manually. "What is needed," Kahn says, "is a technological solution."
Kahn has devoted much of his career to this problem. Last year he founded Delve to provide large organizations with the technology and support they need to bring this data monster under control. This year he is completing a new book, Chucking the Daisies: How Companies Deal with Big Data that presents a set of specific rules for managing data over its lifetime.
Despite the idea that "storage is cheap", Kahn's company, Delve, cost-justifies itself to large corporate clients purely on the hard savings they will realize by eliminating unneed data that is past its sell-by date. Many of those companies have not even bothered to dedupe, compress, and archive that data out of their production databases. And it is growing at a rate of 20% to 50% per year. "When we tell the CIO that we can save you $20 million, $30 million, $40 million per year it is an easy sell," he says.
However it is not as easy as just deleting every piece of data created before a specific date. Different kinds of data have different life-spans. Therefore, everything has to be reviewed to determine, for instance, whether it is still valuable to some process, whether it is covered by compliance or civil procedure requirements, whether is it still in use. This is complex and difficult given the huge amount of data involved, and the number of systems, different formats, and physical and logical locations in which it resides.
And the selection of data to be deleted has to be made using a formal, rules-based process that makes corporate legal, compliance, and business executives comfortable "or in the end they will not pull the trigger and delete the data," Kahn says.
Delve does this using a powerful machine-learning system. First it simplifies the organization's retention schedule and rules and procedures and teaches that and the various formats the data is in -- including unusual and custom formats associated with custom software -- to the computer system. Then Delve's automated records management systems works its way through the petabytes of data in the enterprise, seeking out all the places where data hides. It reads all the records, both structured and unstructured, and identifies all the data that is no longer required for legal or business purposes and can be safely deleted from the system. Once that is done the company can apply those rules to its data going forward to keep the expensive internal databases under control.
Forgot Password
Almost there!
We just sent you a verification email. Please verify your account to gain access to
Hack-Reduce Launch 2012 | Cambridge. If you don’t think you received an email check your
spam folder.
In order to sign in, enter the email address you used to registered for the event. Once completed, you will receive an email with a verification link. Open this link to automatically sign into the site.
Register For Hack-Reduce Launch 2012 | Cambridge
Please fill out the information below. You will recieve an email with a verification link confirming your registration. Click the link to automatically sign into the site.
You��re almost there!
We just sent you a verification email. Please click the verification button in the email. Once your email address is verified, you will have full access to all event content for Hack-Reduce Launch 2012 | Cambridge.
I want my badge and interests to be visible to all attendees.
Checking this box will display your presense on the attendees list, view your profile and allow other attendees to contact you via 1-1 chat. Read the Privacy Policy. At any time, you can choose to disable this preference.
Select your Interests!
add
Upload your photo
Uploading..
OR
Connect via Twitter
Connect via Linkedin
EDIT PASSWORD
Share
Forgot Password
Almost there!
We just sent you a verification email. Please verify your account to gain access to
Hack-Reduce Launch 2012 | Cambridge. If you don’t think you received an email check your
spam folder.
In order to sign in, enter the email address you used to registered for the event. Once completed, you will receive an email with a verification link. Open this link to automatically sign into the site.
Sign in to gain access to Hack-Reduce Launch 2012 | Cambridge
Please sign in with LinkedIn to continue to Hack-Reduce Launch 2012 | Cambridge. Signing in with LinkedIn ensures a professional environment.
Are you sure you want to remove access rights for this user?
Details
Manage Access
email address
Community Invitation
Ely Kahn - hack-reduce Launch - theCUBE
In the excitement about Big Data, what is being missed is that large enterprises are actually drowning in Petabytes of historical data, both structured and unstructured, generated by internal systems from ERP to e-mail, says consultant, lawyer and records management expert Randolph Kahn. And despite the commonly held myth that "storage is cheap", this data is costing those companies millions of dollars annually in direct storage costs plus huge amounts in "soft costs" ranging from the 20%+ of executives' time wasted searching for the significant information buried in that largely useless data to the huge cost of finding relevant information in the case of a legal action.
"I started my career as a trial lawyer," Kahn told Wikibon Co-Founder and Chief Analyst David Vellante in an interview in the Cube in Boston (full video below). "Litigation response tells us over & over & over again that companies don't have their act together. So I would tell you that harnessing their data in a Big Data context is aspirational for most organizations. I would say ubiquitous mismanagement is much more the theme of the day."
This data is supposed to be managed through records management, but that has failed completely. The rules are too complex, they are not applied evenly or at all to much company data. Nobody has clear ownership of any of the data, and while IT owns the storage systems, corporate counsel constantly warns them to be very careful about what is deleted, which means that all the data is saved.
"Most IT professionals don't think of information as having a life-cycle," he says. "Today everything is electronic, and it is all generating data. At the end of the information life-cycle, the idea of parking it somewhere in a repository, a shared drive, it's an expense. It's a liability. And I think IT professionals need to understand what's in that life-cycle."
Much of that old data is actually valueless, unneeded, and a source of legal risk to the organization. And with Petabytes sitting in a wide variety of systems, in multiple formats, the physical act of reviewing all that data is too massive and expensive for companies to undertake manually. "What is needed," Kahn says, "is a technological solution."
Kahn has devoted much of his career to this problem. Last year he founded Delve to provide large organizations with the technology and support they need to bring this data monster under control. This year he is completing a new book, Chucking the Daisies: How Companies Deal with Big Data that presents a set of specific rules for managing data over its lifetime.
Despite the idea that "storage is cheap", Kahn's company, Delve, cost-justifies itself to large corporate clients purely on the hard savings they will realize by eliminating unneed data that is past its sell-by date. Many of those companies have not even bothered to dedupe, compress, and archive that data out of their production databases. And it is growing at a rate of 20% to 50% per year. "When we tell the CIO that we can save you $20 million, $30 million, $40 million per year it is an easy sell," he says.
However it is not as easy as just deleting every piece of data created before a specific date. Different kinds of data have different life-spans. Therefore, everything has to be reviewed to determine, for instance, whether it is still valuable to some process, whether it is covered by compliance or civil procedure requirements, whether is it still in use. This is complex and difficult given the huge amount of data involved, and the number of systems, different formats, and physical and logical locations in which it resides.
And the selection of data to be deleted has to be made using a formal, rules-based process that makes corporate legal, compliance, and business executives comfortable "or in the end they will not pull the trigger and delete the data," Kahn says.
Delve does this using a powerful machine-learning system. First it simplifies the organization's retention schedule and rules and procedures and teaches that and the various formats the data is in -- including unusual and custom formats associated with custom software -- to the computer system. Then Delve's automated records management systems works its way through the petabytes of data in the enterprise, seeking out all the places where data hides. It reads all the records, both structured and unstructured, and identifies all the data that is no longer required for legal or business purposes and can be safely deleted from the system. Once that is done the company can apply those rules to its data going forward to keep the expensive internal databases under control.
