HPE Discover 2024 | The need for Private AI

Clips
News
More from HPE Discover 2024

Jordan Nanos

Machine Learning Architect

HPE

Enayet Shabon

Executive, Emerging Technologies, Corporate Strategic Partnerships & Alliances

Peraton

HPE AI solutions enhance generative AI for modern enterprises

Hewlett Packard Enterprise Co. is at the forefront of helping organizations leverage HPE AI solutions to maximize the potential of AI and cloud technologies.Companies such as Peraton Inc. emphasize the importance of utilizing confidential, private and regulated data sources to power AI applications. This approach results in a hybrid model of public and private AI, essential for addressing data sovereignty, security and performance needs

play_circle_outline Maximizing AI Quality Through Private Data: Focusing on Dedicated Datasets for Improved Results

play_circle_outline Importance of GPUs in AI development and inference

play_circle_outline GPUs needed for inference in enterprise, supply chain challenges

play_circle_outline AI models optimized for specific processors, CPU for general processing

play_circle_outline Internal productivity use cases for generative AI success

Info

The need for Private AI

Jordan Nanos

Machine Learning Architect HPE

Enayet Shabon

Executive, Emerging Technologies, Corporate Strategic Partnerships & Alliances Peraton

search

Rebecca Knight

>> Hello everyone and welcome back to day two of theCUBE's live wall-to-wall coverage of HPE Discover here in Las Vegas. I'm your host, Rebecca Knight, sitting alongside my co-host, co-analyst, Dave Vellante. This show is really getting going. We're feeling the buzz down on the show floor.

>> Fun to have Antonio on today.

Rebecca Knight

>> Exactly, it was a great interview.

>> He's a very calm CEO, love it.

Rebecca Knight

>> Calm, but charismatic.

>> Yep, for sure.

Rebecca Knight

>> Indeed. Well, I'd like to welcome our next two guests to the show. We have Jordan Nanos. He is machine learning architect at HPE. Welcome, Jordan. And Enayet Shabon, Executive Emerging Technologies, Corporate Strategic Partnerships and Alliance at Peraton Inc. Welcome, Enayet.

Enayet Shabon

>> Thank you.

Rebecca Knight

>> Enayet, I want to start with you. Why don't you tell our viewers a little bit about Peraton Inc. and what you do there?

Enayet Shabon

>> At Peraton, I run emerging technology. What that include is running all the research and development across the corporation. I work for a strategic innovation group where we are actually looking at research and development, the hard problems that we are solving for our customers. That includes AI, cloud, cyber, on those areas. Actually, I run that across the corporation. We build solution, and then take it to our customers. That's one part of my job. The other part of my job, as you explained on my title, I also run partnership alliance, which makes a lot of sense. I'll go and discuss meetings with partners such as having right now in here in Discover, find solution, AIs and other emerging technology and take that and build it as part of partner alliance that we do across the company.

>> Has AI always been part of the scope? Was it relatively new? How have things changed since the AI heard around the world?

Enayet Shabon

>> I would say AI has always been part of the scope. We have changed a little bit of how we implement. When we are looking at AI as ours is more of our restricted customers. Our AI is more on dedicated environment, whereas public AI has been around long, but the private AI is where we focus on that. I wouldn't say new, but it's new for our customers. That's what we are taking it.

>> Jordan, machine learning architect, not GenAI architect. Okay. Thank you for just not jumping on the bandwagon, but we all know there's AI before GenAI, but do you think about AI in separate domains for separate use cases or is that actually part of your scope?

Jordan Nanos

>> Yes. It's separate use cases, for sure. Over time we've seen an evolution in terms of what data people are using to power their AI applications, and I think generative AI is just an evolution of things that we've already been seeing. When I started working on AI we were focused on things like recommendation systems, computer vision, and now people want to do things like chat with their data. They want productivity focused use cases in language, and that's where generative AI really comes in.

>> When you think about the private AI announcements you made this week, tell us more about those, how they fit into what you just said. What's different around between private AI and public AI? Let's get into that.

Jordan Nanos

>> Really it breaks down in terms of quality. The quality of an AI application is really powered by the dataset that you're using. <any of the organizations that we work with, like Peraton, are really focused on confidential or private or regulated data sources to power their AI applications. We believe, just like in the space of cloud, AI is just going to be hybrid. There's going to be use cases that are public, there's going to be use cases that are private, and that's really based on the data set that's powering the application itself.

>> Is the decision point a function of physical locality of the data or are there other factors involved?

Enayet Shabon

>> Oh, data gravity, locality definitely is important aspect of it. You talked about public AI, we do want to use public AI, but data sovereignty comes in place, data gravity, locality of how we want to do. A lot of implementation has happened in public data. We want to bring that to it in AI or a hybrid implementation, but we need to look at the data, how we can implement that in a private sense. Locale is important, not just from GenAI, but also massive amount of data that are existing that we need to consider that.

>> I wonder if we can draw comparisons between clouds in your title and what's happening with AI. I have said the companies like HPE, you were HPE at the time, you're not alone, and others, they were way late to cloud. They just focused on private cloud, the experience wasn't this. They finally, when they embraced hybrid, and then when GreenLake came out, that set a new standard. I was like, okay, it's about time. But you gave the cloud guys a long, long time to get a leap. Not happening with AI. My question is what do you need from the companies like HPE to be comfortable that you don't have to go to the cloud. That actually, if you have data locally with data gravity, you can keep the data where it is, you can bring the AI to the data. What do you need from them? What's that experience and what's the portfolio and the partnerships need to look like for you as a technologist to be comfortable?

Enayet Shabon

>> We have actually worked long with HP hybrid cloud. Many of my conversation has been with HPE executive is the private cloud is the need. That's what my customer is looking for. Again, I said hybrid cloud, but private cloud is where we are looking. What we are looking for is as that AI as part of ... If you say GreenLake or Flex that we have, that HP has, that needs to be part of it. That is a package. I don't think you can separate the infrastructure from AI. You need the data, you need the infrastructure, you need the GPUs. All of those comes in and combined. What we are looking from HPE a comprehensive solution starting with the cloud, whether it's hybrid, private with the data, and then taking that to a dedicated environment, to achieve that environment to a local or to a more restricted data.

>> Is it fair to say then that the security, the privacy, the performance of your private cloud is more important than the consistency of experience between that private cloud and the public cloud, the so-called hybrid? Is that latter less important or are they both really important -

Enayet Shabon

>> We're a cyber company, so the answer is clear. That is important. Most of our customers are cyber customers, cyber and space. The data sovereignty is important to us, data privacy. There's a lot of data that not publicly available. We have access to data, and those are data that we need to look at. So yeah, -

>> You prioritize that first over-

Enayet Shabon

>> Absolutely -...

>> over some interface or developer experience.

Enayet Shabon

>> Absolutely. Absolutely.

Rebecca Knight

>> Last year was about training, and we saw a huge build out in GPU capacity as the success of NVIDIA can demonstrate vividly to all of us. What do you see as in store for this year and even in the future?

Jordan Nanos

>> I think this year is the, really, the year of inference in the enterprise, which means that we've seen really impactful releases of public free open source models that many organizations can download and use for their business applications. That means that there's now a requirement for GPUs and accelerated compute to be all over the world where the data is being created. Our CEO, Antonio, talks about bringing the compute to the data rather than bringing the data to the compute. I think this is particularly important when we look at AI applications because there's data sets that you don't want to share that are not public. This could be your code repository. This could be product documentation about things you haven't released yet. This could be confidential communications with third parties. You don't want to release that data into the public domain, and really bringing GPUs into the data center onto a mobile device, onto a ship, onto a armored vehicle, onto a drone. There's all sorts of different use cases where when you bring compute to the data, you're able to unlock a lot of new experiences for people.

Enayet Shabon

>> I want to add on to inference, I think that is important. I mean, I think we build the data, we train them, we fine-tune them. We will continue to do some fine-tuning. But the inference is the most important right now. There are many examples. AV cars, well, you can train them on a stop sign, but they will stop somewhere. What happens if there is no stop sign? Still the reference implementation will help it to stop. It understands that. On fraud prevention, we do a lot of fraud prevention. We do a lot of cybersecurity mal detection. Well, I cannot teach, I cannot train about all the malware that comes in. Well, inference comes in play. Absolutely agree with Jordan, that is the next thing. That is what everybody's going to focus on, how we can fine tune it so we get more advanced on inferences.

>> Do you need GPUs to do that inference?

Enayet Shabon

>> We need GPUs in everywhere. Yes. -

>> Because a lot of talk about whether or not you need GPUs to do inference. I'd love your expert opinion on this. Somebody recently said to me, "Look, it's matrix math, but there's a lot of it, and so you're actually going to need GPUs at the edge doing those." Others have said, "Not necessarily." I'm curious from your perspective.

Enayet Shabon

>> I ... go ahead.

Jordan Nanos

>> It depends what type of experience you're trying to create. I think if you look at some of these applications, people expect that there's just one model powering it. In fact, there's applications where there's going to be multiple models. Some are going to be a good fit for accelerated compute and others potentially not. Let's give an example where you're running a model on a ship that's disconnected. It doesn't have access to the internet, but you still want users on the ship to effectively have a personal assistant where they can chat with a manual. Instead of 500 pages of details, they can quickly get a response and learn about some detail of the machine they're working on. If a user is trying to speak their native language, we could involve a model to transcribe the audio-to-text, to translate the text, to then do a retrieval of the relevant documents, then form a chat response, translate it back and potentially take the text-to-speech back to the user to give them an audio experience. That's six models. Some of them are potentially required to run on a GPU because they're so big. Others, you can quickly short circuit it, run it on the CPU that you already have. But I think more and more we're going to see these models be optimized for specific processors. CPU is good for general processing. The GPU is very good for specific stuff.

Enayet Shabon

>> I was in another panel, Northern Virginia Technology Council, and one of the questions regarding the GPU was, "What is the challenge?" I know we haven't talked about this, the supply chain on GPU. There is impossible amount of requests that we are getting right now that the industry can provide amount of GPU. Again, it's not just restricted public. I know there was an announcement that we have done enough GPU, I don't think we have. I think that would be the biggest thing. Your question is do we need GPU? I think we need more of it. How are we going to meet that supply demand or supply chain? I'd like to hear from HPE and others, or NVIDIA, what they're providing. I'm sure you have heard from Jensen, but that's not enough amount of AI development we are doing.

>> Just to put a finer point on it. You're saying you need GPUs even for inference?

Enayet Shabon

>> That's right.

>> That's consistent with what the hardcore geeks are telling me.

Enayet Shabon

>> Well, I got a geek right here.

>> The second thing is ... right, confirms Jordan as well. The second thing is, I mean supply at some point's going to loosen up. I mean, I think Antonio on the call said HPE's supply is going to come under, I think he said six to 10 weeks. I can't remember the exact ... Don't hold me to that. My information suggests that we're closer to four to five weeks now, but that's for H100s. Now everybody's going to want Blackwell.

Enayet Shabon

>> That's right. Or -

>> It's like we need bigger GPUs as he says. We'll see.

Rebecca Knight

>> I'm interested in both of your take on the implications for the workforce, and not just the hardcore geeks. Thinking about the employees that are chatting with the data.

>> That's why we make such a good team here.

Rebecca Knight

>> That's right. Exactly. Exactly. The person you talked about on the ship who is trying to figure out a problem that is obviously imminent. That person needs to be able to interact with the data in meaningful ways and in urgent ways. I'm wondering if you could talk a little bit about the kinds of skills that you foresee being the most important skills even for the non-hardcore geeks, for the people whose job it is to chat with the data and interact with the data.

Enayet Shabon

>> Well, I don't know, maybe you have insight to our research and development that we do. One of the things that we are looking at is what we call is debaters and judges. We prompt engineers. You're talking about prompting, how are you going to make use of all these LLMs and GenAI that we have? What we build is a system. This is in prototype, and I can talk a little bit about that. We basically build debaters, debaters asking question from ChatGPT or whatever your favorite tool is. You ask the same questions and you get response. Then you have the judge take the response and send it back. Guess what? The response that the judge get is closer to that number than what the prompters, each debaters are getting. There are ways, I think. When you're talking about workforce, I think we work very closely with a lot of universities. Data science in every university is increasing amount of data, so we want to bring in a lot of young generation into our company. I think that is where we're talking about how we can make better use of ... I don't want to be the one person saying responsible AI. That's the challenge of how we can address that.

Jordan Nanos

>> For the example of chat with the data, I have yet to see an example of an LLM powered application where you're just replacing an absolute expert in the domain, but what we're seeing is the ability to create new really productive people really quickly. I think training and learning is a really powerful way in which these models can be used. I particularly focus with customers on internal use cases versus external. If you're focused on internal productivity things, you remove a lot of the risks of AI or generative AI. A hallucination means a confused developer or a confused new employee who's trying to learn. It's not really a risk to the reputation of the business. But if you put this on your website and it's public, there's a lot of differences in terms of that experience. I think most organizations that I'm seeing be successful with generative AI are starting with internal productivity focused use cases, and they're getting a lot of value from that.

Rebecca Knight

>> Where there's a real human in the loop all the time.

Jordan Nanos

>> Yeah.

>> Well, bringing together Rebecca's point about the human impact and the technology impact, we often say, "Well, we're in the early days of AI." It's like the dial-up days of the internet. You can only imagine what it's going to be like in the future. But I love using LLMs because it saves a lot of time, makes you more productive. But what I find myself doing now, to the exact point you were saying, because of the hallucinations, I'll have ChatGPT open. I'll have Perplexity open. I'll go to Meta AI for Llama. I'll do Gemini. I literally have four tabs open.

Rebecca Knight

>> You're the best product engineer.

>> Exact same question of each. Then I will use my human intuition to say, "I think that's the right answer." I might even Google it.

Rebecca Knight

>> Or take a little bit of both and really inspect it.

>> I might even Google it just to fact check. It has made my fact checking, I guess, more productive. I think probably more accurate. My question is, are we going to get to the point where I can interact with these multiple models simultaneously and get a higher quality answer?

Jordan Nanos

>> I think absolutely, yeah. Relying on your intuition I think is really important. The big thing I've seen when rolling out these generative AI applications is that you actually need to train users to operate in this world of uncertainty where it's not a deterministic response. You don't know it's going to be exactly right. You need to check the sources, you need to verify things. That's a big change in how people operate and interact with technology. But I'm really surprised by how easy it seems for people to work this way because it's a more human experience. It's actually more strange to get an exact response that's deterministic from a Google search or you're checking a webpage than it is to basically hear from a friend. You know sometimes they're just making stuff up.

>> Okay. My last question is, I wrote a piece recently on hybrid AI and I got a lot of grief for it. Well, that's hybrid AI, sounds like a marketing term. Maybe it is.

Enayet Shabon

>> I hope you didn't say, "Hey, hybrid AI is dead."

>> No. Gosh no.

Enayet Shabon

>> Because there was an article is dead about 10 years ago.

>> The opposite, actually.

Enayet Shabon

>> Okay.

>> I said, "Hybrid AI is actually a thing." I also said Supercloud is a thing, and I got a lot of grief for that, but we defined it. Anyway, what's your point of view on hybrid AI? How do you maybe define it or think about what it actually is? Is it real? Is hybrid AI a thing?

Enayet Shabon

>> It is, and you can take advantage of that. Again, in my case, and that's true for a lot of things, you could use the public AI to get a lot of information, gather that information, and then go to the private side of it where I'm focusing on data sovereignty. I cannot handle data sovereignty in public data, so we're locale or data gravity when I'm going to private and I want to have that local data for me. I think there's definitely a big play for hybrid. Hopefully we are not going to do the same thing that we talked about. Hybrid cloud, there's only a hybrid cloud. There's only public cloud. You hear all these, "Everything will go public cloud." I have a lot of customers coming back too. It's like, "What happened to that private cloud you're talking to me about 10 years ago?" I don't want to be in that position. I think there's going to be combination of both.

>> Is that mixture of data sources that you're saying.

Enayet Shabon

>> Yeah, but it depends on the customers, who are looking. Most of my customers are looking on private AI. But definitely I need to take advantage of public AI because there's so much I can learn from that and take that and use it on a private setting.

>> Anything you'd add?

Jordan Nanos

>> I think just a focus on locality like you were saying. The distinction between public and private I think is important, but there's also real examples of a model that can run on your phone or your laptop compared to a model that can run in a data center, which could be in your metro region or it could be quite far away from you. We've seen this with autonomous vehicles. There's models that run on the car. There's models that run in a data center. I think that's going to be the biggest driver for the reason people adopt a hybrid approach. Generally, when people bring up hybrid cloud, hybrid AI, it seems to me that they're making the point that the world is complicated. Cloud is complicated, AI is complicated. You're going to need to contend with that. There's going to be things that need to run locally and remotely.

>> Jensen at GTC, we had a private meeting with him with a bunch of analysts, and he said something that really resonated me. Because were so used to getting instantaneous responses, and sometimes those instantaneous responses are just, man ... He said, "What if you could tell the AI, 'Go solve this problem. I got $5,000 to spend. Go reason, take the time, take two weeks and go reason, and then come back with an answer in two weeks.'" That is the future that he laid out, which I thought was quite interesting you would do that. That's different thinking than the instantaneous response, the instant gratification. Yeah, it is early days.

Jordan Nanos

>> Well, we talked about this year being the year of inference, certainly for the models that are already trained, but there are large organizations training absolutely massive models. If you think about a 10 trillion parameter dense model, I mean, how long is it going to take to get just one token out of that thing? Clearly there's a need for batch inference use cases where this is going to be really high quality, but it's going to take a long time to get a response. Then there's also going to be a need for smaller models to give you a near instantaneous response.

>> Lots of training, even more inference.

Rebecca Knight

>> And it is complicated.

Enayet Shabon

>> I think you asked regarding the human going to be in the loop, I think in a lot of cases it will be in the loop. It has to be in the loop. We cannot do rely-

Rebecca Knight

>> Our jobs are secure.

Enayet Shabon

>> Thank you, that's great. Our jobs are secure.

Rebecca Knight

>> Enayet, thank you both so much for coming on the show.

Enayet Shabon

>> I appreciate it.

Rebecca Knight

>> A great conversation.

Enayet Shabon

>> Maybe it is.

Rebecca Knight

>> I'm Rebecca Knight. For Dave Vellante, stay tuned for more of theCUBE's live coverage of HPE Discover. You're watching theCUBE, the leader in enterprise tech news and analysis.

The need for Private AI

search

Rebecca Knight

>> Hello everyone and welcome back to day two of theCUBE's live wall-to-wall coverage of HPE Discover here in Las Vegas. I'm your host, Rebecca Knight, sitting alongside my co-host, co-analyst, Dave Vellante. This show is really getting going. We're feeling the buzz down on the show floor.

>> Fun to have Antonio on today.

Rebecca Knight

>> Exactly, it was a great interview.

>> He's a very calm CEO, love it.

Rebecca Knight

>> Calm, but charismatic.

>> Yep, for sure.

Rebecca Knight

>> Indeed. Well, I'd like to welcome our next two guests to the show. We have Jordan Nanos. He is machine learning architect at HPE. Welcome, Jordan. And Enayet Shabon, Executive Emerging Technologies, Corporate Strategic Partnerships and Alliance at Peraton Inc. Welcome, Enayet.

Enayet Shabon

>> Thank you.

Rebecca Knight

>> Enayet, I want to start with you. Why don't you tell our viewers a little bit about Peraton Inc. and what you do there?

Enayet Shabon

>> At Peraton, I run emerging technology. What that include is running all the research and development across the corporation. I work for a strategic innovation group where we are actually looking at research and development, the hard problems that we are solving for our customers. That includes AI, cloud, cyber, on those areas. Actually, I run that across the corporation. We build solution, and then take it to our customers. That's one part of my job. The other part of my job, as you explained on my title, I also run partnership alliance, which makes a lot of sense. I'll go and discuss meetings with partners such as having right now in here in Discover, find solution, AIs and other emerging technology and take that and build it as part of partner alliance that we do across the company.

>> Has AI always been part of the scope? Was it relatively new? How have things changed since the AI heard around the world?

Enayet Shabon

>> I would say AI has always been part of the scope. We have changed a little bit of how we implement. When we are looking at AI as ours is more of our restricted customers. Our AI is more on dedicated environment, whereas public AI has been around long, but the private AI is where we focus on that. I wouldn't say new, but it's new for our customers. That's what we are taking it.

>> Jordan, machine learning architect, not GenAI architect. Okay. Thank you for just not jumping on the bandwagon, but we all know there's AI before GenAI, but do you think about AI in separate domains for separate use cases or is that actually part of your scope?

Jordan Nanos

>> Yes. It's separate use cases, for sure. Over time we've seen an evolution in terms of what data people are using to power their AI applications, and I think generative AI is just an evolution of things that we've already been seeing. When I started working on AI we were focused on things like recommendation systems, computer vision, and now people want to do things like chat with their data. They want productivity focused use cases in language, and that's where generative AI really comes in.

>> When you think about the private AI announcements you made this week, tell us more about those, how they fit into what you just said. What's different around between private AI and public AI? Let's get into that.

Jordan Nanos

>> Really it breaks down in terms of quality. The quality of an AI application is really powered by the dataset that you're using. <any of the organizations that we work with, like Peraton, are really focused on confidential or private or regulated data sources to power their AI applications. We believe, just like in the space of cloud, AI is just going to be hybrid. There's going to be use cases that are public, there's going to be use cases that are private, and that's really based on the data set that's powering the application itself.

>> Is the decision point a function of physical locality of the data or are there other factors involved?

Enayet Shabon

>> Oh, data gravity, locality definitely is important aspect of it. You talked about public AI, we do want to use public AI, but data sovereignty comes in place, data gravity, locality of how we want to do. A lot of implementation has happened in public data. We want to bring that to it in AI or a hybrid implementation, but we need to look at the data, how we can implement that in a private sense. Locale is important, not just from GenAI, but also massive amount of data that are existing that we need to consider that.

>> I wonder if we can draw comparisons between clouds in your title and what's happening with AI. I have said the companies like HPE, you were HPE at the time, you're not alone, and others, they were way late to cloud. They just focused on private cloud, the experience wasn't this. They finally, when they embraced hybrid, and then when GreenLake came out, that set a new standard. I was like, okay, it's about time. But you gave the cloud guys a long, long time to get a leap. Not happening with AI. My question is what do you need from the companies like HPE to be comfortable that you don't have to go to the cloud. That actually, if you have data locally with data gravity, you can keep the data where it is, you can bring the AI to the data. What do you need from them? What's that experience and what's the portfolio and the partnerships need to look like for you as a technologist to be comfortable?

Enayet Shabon

>> We have actually worked long with HP hybrid cloud. Many of my conversation has been with HPE executive is the private cloud is the need. That's what my customer is looking for. Again, I said hybrid cloud, but private cloud is where we are looking. What we are looking for is as that AI as part of ... If you say GreenLake or Flex that we have, that HP has, that needs to be part of it. That is a package. I don't think you can separate the infrastructure from AI. You need the data, you need the infrastructure, you need the GPUs. All of those comes in and combined. What we are looking from HPE a comprehensive solution starting with the cloud, whether it's hybrid, private with the data, and then taking that to a dedicated environment, to achieve that environment to a local or to a more restricted data.

>> Is it fair to say then that the security, the privacy, the performance of your private cloud is more important than the consistency of experience between that private cloud and the public cloud, the so-called hybrid? Is that latter less important or are they both really important -

Enayet Shabon

>> We're a cyber company, so the answer is clear. That is important. Most of our customers are cyber customers, cyber and space. The data sovereignty is important to us, data privacy. There's a lot of data that not publicly available. We have access to data, and those are data that we need to look at. So yeah, -

>> You prioritize that first over-

Enayet Shabon

>> Absolutely -...

>> over some interface or developer experience.

Enayet Shabon

>> Absolutely. Absolutely.

Rebecca Knight

>> Last year was about training, and we saw a huge build out in GPU capacity as the success of NVIDIA can demonstrate vividly to all of us. What do you see as in store for this year and even in the future?

Jordan Nanos

>> I think this year is the, really, the year of inference in the enterprise, which means that we've seen really impactful releases of public free open source models that many organizations can download and use for their business applications. That means that there's now a requirement for GPUs and accelerated compute to be all over the world where the data is being created. Our CEO, Antonio, talks about bringing the compute to the data rather than bringing the data to the compute. I think this is particularly important when we look at AI applications because there's data sets that you don't want to share that are not public. This could be your code repository. This could be product documentation about things you haven't released yet. This could be confidential communications with third parties. You don't want to release that data into the public domain, and really bringing GPUs into the data center onto a mobile device, onto a ship, onto a armored vehicle, onto a drone. There's all sorts of different use cases where when you bring compute to the data, you're able to unlock a lot of new experiences for people.

Enayet Shabon

>> I want to add on to inference, I think that is important. I mean, I think we build the data, we train them, we fine-tune them. We will continue to do some fine-tuning. But the inference is the most important right now. There are many examples. AV cars, well, you can train them on a stop sign, but they will stop somewhere. What happens if there is no stop sign? Still the reference implementation will help it to stop. It understands that. On fraud prevention, we do a lot of fraud prevention. We do a lot of cybersecurity mal detection. Well, I cannot teach, I cannot train about all the malware that comes in. Well, inference comes in play. Absolutely agree with Jordan, that is the next thing. That is what everybody's going to focus on, how we can fine tune it so we get more advanced on inferences.

>> Do you need GPUs to do that inference?

Enayet Shabon

>> We need GPUs in everywhere. Yes. -

>> Because a lot of talk about whether or not you need GPUs to do inference. I'd love your expert opinion on this. Somebody recently said to me, "Look, it's matrix math, but there's a lot of it, and so you're actually going to need GPUs at the edge doing those." Others have said, "Not necessarily." I'm curious from your perspective.

Enayet Shabon

>> I ... go ahead.

Jordan Nanos

>> It depends what type of experience you're trying to create. I think if you look at some of these applications, people expect that there's just one model powering it. In fact, there's applications where there's going to be multiple models. Some are going to be a good fit for accelerated compute and others potentially not. Let's give an example where you're running a model on a ship that's disconnected. It doesn't have access to the internet, but you still want users on the ship to effectively have a personal assistant where they can chat with a manual. Instead of 500 pages of details, they can quickly get a response and learn about some detail of the machine they're working on. If a user is trying to speak their native language, we could involve a model to transcribe the audio-to-text, to translate the text, to then do a retrieval of the relevant documents, then form a chat response, translate it back and potentially take the text-to-speech back to the user to give them an audio experience. That's six models. Some of them are potentially required to run on a GPU because they're so big. Others, you can quickly short circuit it, run it on the CPU that you already have. But I think more and more we're going to see these models be optimized for specific processors. CPU is good for general processing. The GPU is very good for specific stuff.

Enayet Shabon

>> I was in another panel, Northern Virginia Technology Council, and one of the questions regarding the GPU was, "What is the challenge?" I know we haven't talked about this, the supply chain on GPU. There is impossible amount of requests that we are getting right now that the industry can provide amount of GPU. Again, it's not just restricted public. I know there was an announcement that we have done enough GPU, I don't think we have. I think that would be the biggest thing. Your question is do we need GPU? I think we need more of it. How are we going to meet that supply demand or supply chain? I'd like to hear from HPE and others, or NVIDIA, what they're providing. I'm sure you have heard from Jensen, but that's not enough amount of AI development we are doing.

>> Just to put a finer point on it. You're saying you need GPUs even for inference?

Enayet Shabon

>> That's right.

>> That's consistent with what the hardcore geeks are telling me.

Enayet Shabon

>> Well, I got a geek right here.

>> The second thing is ... right, confirms Jordan as well. The second thing is, I mean supply at some point's going to loosen up. I mean, I think Antonio on the call said HPE's supply is going to come under, I think he said six to 10 weeks. I can't remember the exact ... Don't hold me to that. My information suggests that we're closer to four to five weeks now, but that's for H100s. Now everybody's going to want Blackwell.

Enayet Shabon

>> That's right. Or -

>> It's like we need bigger GPUs as he says. We'll see.

Rebecca Knight

>> I'm interested in both of your take on the implications for the workforce, and not just the hardcore geeks. Thinking about the employees that are chatting with the data.

>> That's why we make such a good team here.

Rebecca Knight

>> That's right. Exactly. Exactly. The person you talked about on the ship who is trying to figure out a problem that is obviously imminent. That person needs to be able to interact with the data in meaningful ways and in urgent ways. I'm wondering if you could talk a little bit about the kinds of skills that you foresee being the most important skills even for the non-hardcore geeks, for the people whose job it is to chat with the data and interact with the data.

Enayet Shabon

>> Well, I don't know, maybe you have insight to our research and development that we do. One of the things that we are looking at is what we call is debaters and judges. We prompt engineers. You're talking about prompting, how are you going to make use of all these LLMs and GenAI that we have? What we build is a system. This is in prototype, and I can talk a little bit about that. We basically build debaters, debaters asking question from ChatGPT or whatever your favorite tool is. You ask the same questions and you get response. Then you have the judge take the response and send it back. Guess what? The response that the judge get is closer to that number than what the prompters, each debaters are getting. There are ways, I think. When you're talking about workforce, I think we work very closely with a lot of universities. Data science in every university is increasing amount of data, so we want to bring in a lot of young generation into our company. I think that is where we're talking about how we can make better use of ... I don't want to be the one person saying responsible AI. That's the challenge of how we can address that.

Jordan Nanos

>> For the example of chat with the data, I have yet to see an example of an LLM powered application where you're just replacing an absolute expert in the domain, but what we're seeing is the ability to create new really productive people really quickly. I think training and learning is a really powerful way in which these models can be used. I particularly focus with customers on internal use cases versus external. If you're focused on internal productivity things, you remove a lot of the risks of AI or generative AI. A hallucination means a confused developer or a confused new employee who's trying to learn. It's not really a risk to the reputation of the business. But if you put this on your website and it's public, there's a lot of differences in terms of that experience. I think most organizations that I'm seeing be successful with generative AI are starting with internal productivity focused use cases, and they're getting a lot of value from that.

Rebecca Knight

>> Where there's a real human in the loop all the time.

Jordan Nanos

>> Yeah.

>> Well, bringing together Rebecca's point about the human impact and the technology impact, we often say, "Well, we're in the early days of AI." It's like the dial-up days of the internet. You can only imagine what it's going to be like in the future. But I love using LLMs because it saves a lot of time, makes you more productive. But what I find myself doing now, to the exact point you were saying, because of the hallucinations, I'll have ChatGPT open. I'll have Perplexity open. I'll go to Meta AI for Llama. I'll do Gemini. I literally have four tabs open.

Rebecca Knight

>> You're the best product engineer.

>> Exact same question of each. Then I will use my human intuition to say, "I think that's the right answer." I might even Google it.

Rebecca Knight

>> Or take a little bit of both and really inspect it.

>> I might even Google it just to fact check. It has made my fact checking, I guess, more productive. I think probably more accurate. My question is, are we going to get to the point where I can interact with these multiple models simultaneously and get a higher quality answer?

Jordan Nanos

>> I think absolutely, yeah. Relying on your intuition I think is really important. The big thing I've seen when rolling out these generative AI applications is that you actually need to train users to operate in this world of uncertainty where it's not a deterministic response. You don't know it's going to be exactly right. You need to check the sources, you need to verify things. That's a big change in how people operate and interact with technology. But I'm really surprised by how easy it seems for people to work this way because it's a more human experience. It's actually more strange to get an exact response that's deterministic from a Google search or you're checking a webpage than it is to basically hear from a friend. You know sometimes they're just making stuff up.

>> Okay. My last question is, I wrote a piece recently on hybrid AI and I got a lot of grief for it. Well, that's hybrid AI, sounds like a marketing term. Maybe it is.

Enayet Shabon

>> I hope you didn't say, "Hey, hybrid AI is dead."

>> No. Gosh no.

Enayet Shabon

>> Because there was an article is dead about 10 years ago.

>> The opposite, actually.

Enayet Shabon

>> Okay.

>> I said, "Hybrid AI is actually a thing." I also said Supercloud is a thing, and I got a lot of grief for that, but we defined it. Anyway, what's your point of view on hybrid AI? How do you maybe define it or think about what it actually is? Is it real? Is hybrid AI a thing?

Enayet Shabon

>> It is, and you can take advantage of that. Again, in my case, and that's true for a lot of things, you could use the public AI to get a lot of information, gather that information, and then go to the private side of it where I'm focusing on data sovereignty. I cannot handle data sovereignty in public data, so we're locale or data gravity when I'm going to private and I want to have that local data for me. I think there's definitely a big play for hybrid. Hopefully we are not going to do the same thing that we talked about. Hybrid cloud, there's only a hybrid cloud. There's only public cloud. You hear all these, "Everything will go public cloud." I have a lot of customers coming back too. It's like, "What happened to that private cloud you're talking to me about 10 years ago?" I don't want to be in that position. I think there's going to be combination of both.

>> Is that mixture of data sources that you're saying.

Enayet Shabon

>> Yeah, but it depends on the customers, who are looking. Most of my customers are looking on private AI. But definitely I need to take advantage of public AI because there's so much I can learn from that and take that and use it on a private setting.

>> Anything you'd add?

Jordan Nanos

>> I think just a focus on locality like you were saying. The distinction between public and private I think is important, but there's also real examples of a model that can run on your phone or your laptop compared to a model that can run in a data center, which could be in your metro region or it could be quite far away from you. We've seen this with autonomous vehicles. There's models that run on the car. There's models that run in a data center. I think that's going to be the biggest driver for the reason people adopt a hybrid approach. Generally, when people bring up hybrid cloud, hybrid AI, it seems to me that they're making the point that the world is complicated. Cloud is complicated, AI is complicated. You're going to need to contend with that. There's going to be things that need to run locally and remotely.

>> Jensen at GTC, we had a private meeting with him with a bunch of analysts, and he said something that really resonated me. Because were so used to getting instantaneous responses, and sometimes those instantaneous responses are just, man ... He said, "What if you could tell the AI, 'Go solve this problem. I got $5,000 to spend. Go reason, take the time, take two weeks and go reason, and then come back with an answer in two weeks.'" That is the future that he laid out, which I thought was quite interesting you would do that. That's different thinking than the instantaneous response, the instant gratification. Yeah, it is early days.

Jordan Nanos

>> Well, we talked about this year being the year of inference, certainly for the models that are already trained, but there are large organizations training absolutely massive models. If you think about a 10 trillion parameter dense model, I mean, how long is it going to take to get just one token out of that thing? Clearly there's a need for batch inference use cases where this is going to be really high quality, but it's going to take a long time to get a response. Then there's also going to be a need for smaller models to give you a near instantaneous response.

>> Lots of training, even more inference.

Rebecca Knight

>> And it is complicated.

Enayet Shabon

>> I think you asked regarding the human going to be in the loop, I think in a lot of cases it will be in the loop. It has to be in the loop. We cannot do rely-

Rebecca Knight

>> Our jobs are secure.

Enayet Shabon

>> Thank you, that's great. Our jobs are secure.

Rebecca Knight

>> Enayet, thank you both so much for coming on the show.

Enayet Shabon

>> I appreciate it.

Rebecca Knight

>> A great conversation.

Enayet Shabon

>> Maybe it is.

Rebecca Knight

>> I'm Rebecca Knight. For Dave Vellante, stay tuned for more of theCUBE's live coverage of HPE Discover. You're watching theCUBE, the leader in enterprise tech news and analysis.