theCUBEd Awards | WEKA® Data Platform (4th Generation) by WEKA (Most Innovative Data Platform)

Clips
More from theCUBEd Awards

Shimon Ben-David

Director of Product Management

Google

play_circle_outline WEKA's accelerated data platform for GPU workloads and HPC environments

play_circle_outline WEKA as an extension to GPU memory, reducing processing time by 48 times

play_circle_outline Customer examples of solving exabyte scale challenges and training environment acceleration

play_circle_outline Reduced deployment time and operational efficiency in inference environments using WEKA

Info
Transcript

WEKA® Data Platform (4th Generation) by WEKA (Most Innovative Data Platform)

Shimon Ben-David

Director of Product Management Google

In this discussion, Shimon Ben-David, CTO of WEKA, joins Rob Strechay of theCUBE to explore WEKA's groundbreaking innovations in data platforms. They talk about WEKA's success in the “Most Innovative Data Platform” category at theCUBEd Awards, highlighting the company's drive to advance the data platform industry amid stiff competition.

Ben-David speaks about WEKA's approach to creating a no-compromise file system optimized for demanding AI and HPC environments. The discussion also covers how WEKA's accelerated data platform uniquely supports complex w... Read more

explore Keep Exploring

What is WEKA and how does it aim to optimize high performance compute GPU environments for new AI workloads? add

What were the results of the experiment involving the distributed KV cache feature working with vLLM and WEKA? add

What are some examples of customers running at exabyte scale and the challenges they face in managing such vast amounts of data while maintaining low latency? add

What were some of the reasons for an environment to choose WEKA for running at scale across thousands of GPUs? add

bolt Powered by CUBE AI

WEKA® Data Platform (4th Generation) by WEKA (Most Innovative Data Platform)

search

Rob Strechay

>> Hello and welcome to theCUBE's Tech Innovation CUBEd Awards. I'm Rob Strechay, managing director with theCUBE Research, and today we're celebrating another winner of our inaugural program that just completed. The competition was fierce, especially in this category, to put it mildly with everything that's going on in AI. And we received a number of exceptional high caliber entries that showcased the incredible innovation happening across the tech landscape. Today we're joined by Shimon Ben-David, CTO from WEKA, to discuss their award-winning entry, the innovation behind it, and how they're driving data platform industry forward. We look at this is WEKA took top honors in most innovative data platform category, and it was really, like I said, a really highly contested set of entries in there. A lot of really, really good stuff, but WEKA came out on top. So Shimon, welcome on board and let's dive in here.

Shimon Ben-David

>> Hey, thank you, Rob. It's great being here. I'm really excited to talk about WEKA with you.

Rob Strechay

>> So why don't we do that? Why don't we start for the people who aren't familiar with WEKA, not exactly a household name yet, but like I was saying, if you've been to the sphere, there's some magic going on behind the scenes over there with you guys. But for those who don't know WEKA, why don't you give them an idea of who you are?

Shimon Ben-David

>> Sure. So WEKA is an accelerated data platform for accelerated workloads, which basically means GPU acceleration, HPC environments, and definitely with when we're seeing the confluence of HPC and AI combining. We built WEKA to create an environment, a file system at the core that has no compromises. Looking back, we saw that there were multiple storage environments where you had to, it was a game of compromises. Do I want high volume of files? Do I want a high amount of small files at work latency? And then we created WEKA actually to service the new workloads, the new AI workloads in a very efficient manner to make sure that you can get the most utilization out of your high performance compute GPU environments. Definitely looking ahead for the new AI environment. I think technically the best analogy I have to describe WEKA because it's such a differentiated environment, unlike any other storage environment, the best analogy would be imagine your GPU server with the GPU server having its local NVMes, and that's usually considered the fastest in terms of throughput, ARP latency and metadata. So now imagine, and we see customers copying data to the local GPU servers NVMes and working with it. Imagine WEKA is that local NVMe, only faster and distributed and shared across all of your GPU environments. So now all of your GPU servers are working with a shared, distributed, protected WEKA environment that's faster than their local NVMe.

Rob Strechay

>> I think that's a good way to put it. I think that will help people understand, especially with the fact that I ran into you guys for the first time, it was with a customer in the biotech area. And again, AI is not new, not new to WEKA, like you said, with HPC, High Performance Computing, that's been AIs what was traditional AI. Now, we have the explosion of this gen AI stuff. But I think that was key to these awards where innovation really was at this central core of these awards. Can you share any specific example on how your company really approached data platforms innovatively? You gone into it a little bit there, but what is the impact for customers so that... What do they gain out of this innovation?

Shimon Ben-David

>> Yeah, definitely. I actually, I'll be glad to share multiple innovations and definitely deep dive into, I'll pick one of them. So we innovated on many things. First, in terms of performance, we innovated, as I mentioned, with a no compromise environment. So running on throughput apps, latency and metadata performance, we innovated by being able to run on all cloud environments, providing multi protocols access to all of the data. We saw that we were accelerating AI environments, training environments. And then if I'll pick a most recent innovation that we did, we actually wanted to look at customer outcomes. We saw that a lot of customers are nowadays focusing on inference environments, definitely with gen AI, LLMs focusing on massive inference environments. And we took a look at how can WEKA accelerate that? Also, what are the challenges that customers are facing in these massive scale out inferencing environments? And we saw that there's eventually GPUs. Inferencing is a memory limited challenge, and GPUs are limited in the amount of memory and therefore in the amount of tokens per second that they can create and the value that they can provide. And we saw that. Then we approached the community and we saw that that's being handled differently by different community open source environments trying to create a KV cache environment on local NVMes, which alleviated the problem somehow, but it didn't solve it in a shared and scale out fashion. So what we did is we hypothesized of what if we can use WEKA as an extension to the GPU memory as the next stage, the next tier of the GPU memory, which is something that we imagined that we could do because of the WEKA low latency, high performance environment capabilities. We then went out and proved this hypothesis where we actually worked with the open source community and created a feature which we call distributed KV cache, where now we created an inference server environment in this case specifically with vLLM that actually utilized WEKA as an extension to the KV cache from the GPU memory. And we proved that hypothesis. I'll just mention a number, imagine GPU loading a hundred thousand tokens, converting them to a data structure that's called KV cache, which is a mandatory state in the LLM activity. And so these a hundred thousand tokens took approximately 24 seconds to be converted. That's a stage that's called the pre-fill. And then we said, okay, for the next GPU, if we now load that same data from WEKA, what will the performance be? And actually we were able to show that loading from WEKA took half a second. So we're able to take an activity that takes 24 seconds and shrink it into half a second. That's 48 times faster. Eventually, the value for these inference customers is a massive acceleration in inferencing in the amount of tokens per second, and therefore also the cost per tokens that they can generate and provide massive value. And also, some of these customers will be able to benefit from just maybe not buying as many GPU servers to just meet what they can already do with the amount that they already have.

Rob Strechay

>> That makes total sense. I think when you look at it, people are looking at these as AI factories and they want to keep the factory busy because that's where you can get your margin from. And really, I think that is key to it. And vLLM, we started talking about that a little bit over eight months ago with some of the things that have been going on and some of the open source community specifically in the Kubernetes community and cloud native community and CNCF where there's some really good activity going on as part of the Linux Foundation. But looking back at the journey for WEKA, what was the key turning point or decision that really led you you think to the recognition that being innovative and getting this innovation award in data platforms?

Shimon Ben-David

>> Brilliant, thank you. So I think looking back, and I'm in the company for the last 11 and a bit years. Looking back, the decisions that the founders made were, we saw the gap in the market, we saw the need for high performance, low latency file systems, and the decision was, let's take everything to the design board. Let's not take anything for granted. Let's not assume that any data structure or algorithm that currently existed for distributed shared environment is the optimal one. Because we saw that the inflection of technology going towards high speed networking, high speed random access SSDs and NVMes and accelerated compute. So essentially shrinking the amount of compute from multiple racks to a single server. So we saw of that, and the decision was, let's design everything anew. And as a result, what we did is we created WEKA, which has the capabilities that are designed for these modern environments, and these are not just words. Eventually when our customers are working with WEKA, they're experiencing faster than local NVMes on a shared distributed environment. So that ability, that initial design still holds, and that's where we are getting our massive performance. And then we took additional calculated decisions, for example, being completely software defined. So we never use any hardware acceleration. We do everything in our own algorithms and code. As a result of that, we're running the same code that WEKA runs on-premise servers. We're actually running that on cloud environments. It's the same code, same binary, same orchestration, same look and feel, same performance that our customers benefit. So they don't need to adapt to different environments and use WEKA on one environment. And another thing, it's WEKA all the way around. Another approach was the multi-protocol environment decision. We understood that in a modern AI environment, we saw a lot of different data silos. So we understood that in order to improve our customer's experience and eventually provide more value, we would like them to be able to access all of the data in all of the protocols. So as a result, another key part of the innovation is the ability to access all of the data in all protocols, which when you think about it in a modern pipeline, customers actually are shoving data into the system in one protocol mostly, for example, SRE. And then they're working on that data with SRE or with our POSIX driver or GPU direct storage. Some customers are even doing it with SMB and NFS that we have. And I would say that we're seeing Kubernetes as the operating system of AI environments, and that's actually something that we integrated into the product as well. I'd be glad to talk for hours about that as well.

Rob Strechay

>> Oh, I'm sure. I think, again, we both get very excited about this stuff, especially when you're diving into the tech. I mean, you and I have been on together previously, and I can tell you that and even off-stage for that matter, had some really great deep conversations. I think one of the things getting to our final question here, your product has been recognized for its innovation and its innovative approach to data platforms. Going down that same path, could you give us a specific example of how customers are using your solution to solve problems that really weren't possible or practical before WEKA and this platform came about?

Shimon Ben-David

>> Yep, definitely. I'll give a few examples. So we have customers that are running at exabyte scale, which is massive. And we see that there's something that's growing on more and more towards the upcoming, even in this year. And definitely next year we'll have more and more of these customers. And I would already pounce on that as something that's very challenging to multiple environments. To manage more than an exabyte of data on a single cluster is challenging and impressive, especially while still providing the same low latency. So that's one benefit. I'll give another massive example. An environment that's running WEKA at scale across thousands of GPUs basically chose WEKA because when we started working with them, they were running their training environments on another all flash NAS appliance, and the repo time took 12 days. Moving to WEKA, it went down to four hours. So literally 12 days to four hours. Usually, this is where the part where it's hard to believe, but the reason is if you imagine all of the things that I mentioned before, there's no need to copy data between different storage environments. There's no need to copy data between environments because of protocol also. And when you are working on WEKA, it's shared, distributed and faster than your local NVMe that actually accelerates the GPUs. The GPUs, basically, were making sure that the GPUs are constantly fed with data. So 12 days to four hours in a large training environments. As I mentioned, we see also customers running inferencing environments. In an inference environment we see customers loading and unloading models. So imagine an environment that took, that was running a fleet of inference servers that needed to expand and shrink. The expansion took anywhere between four to 10 minutes per server to expand because they wanted to accommodate for more prompts and to provide that very low prompt latency. So four minutes to 12 minutes. After WEKA, went down to half a minute to one minute. So the ability to expand and shrink on demand while providing that low latency prompt is massive. The ability to load the models in seconds is massive. The ability to mobilize the data. Okay. We also have the ability to mobilize the data, and some of these customers are also, I'll give an example. We have customers SLAC, Stanford Linear Accelerator, that are gathering data from multiple location, and they're using our data movement capability to centralize that data into a single location where they're running their AI environment on that. So that's also massive. These are just some of the values I'll mentioned.

Rob Strechay

>> Yeah, no, I think, again, it's great to hear those customer stories about how people are really getting their value out of WEKA and your data platform. So Shimon, thank you for coming on board. Congratulations again to you and to WEKA and to your 11 years of hard work getting to this point. We really appreciate you being on board here today.

Shimon Ben-David

>> Thank you very much for having me.

Rob Strechay

>> And thank you for watching theCUBE Tech Innovation CUBEd Awards, Rob Strechay signing off. See you soon.