Howie Mandel is reflecting on the versatility of his career.

The comedian, 70, sat down with Billy Corgan on a recent episode of his The Magnificent Others podcast, where they had a candid conversation about the ups and downs of life in the spotlight. Although Mandel is now well-known for hosting Deal or No Deal, he revealed he almost didn’t take the gig 20 years ago.

“I went into Deal or No Deal kicking and screaming,” he shared. “And not only because of what I thought being a game show host would be perceived [as], as far as 2005 with what the landscape was for people who [hosted], but also because I didn’t understand how you could make an hour of television of having people pick a case and then me, say ‘open the case.’”

“I didn’t have a trivia question to ask somebody I didn’t have a skill that they were going to comment on,” he continued.

Joking that he didn’t know “how many times” he could say, “open the case,” before it got boring, Mandel noted that he was also 50 years old and “treading water, trying not to drown.”

“I always say yes to everything in life because I have fear of missing out, and I also jump at things impulsively,” he explained. “It’s the first thing that I ever said no to.”

“I was in the midst of retiring, and when I say retiring, it was like, I wasn’t going to go out on the road anymore. I wasn’t selling tickets,” he added. “I was getting kicked in the nuts, my soul was being crushed every day. It was just embarrassing.”

After years in the comedy world, Mandel admitted he “wasn’t incredibly successful,” but he settled on doing real estate and “dropping in” at comedy clubs a few times a week to keep his passion alive. 

“When they came to me with [Deal or No Deal], I thought, ‘This is the nail in the coffin of my career,’” he said, but his wife Terry changed his mind. “She knows me better than anybody and protects me more than anybody and loves me more than anybody, and she was the one that said, ‘You better go do it.’”

At first, he pushed back. “I go, ‘But it’s nothing.’ She goes, ‘You better go do it. You just need to do something.’”

At the time, Mandel had been married to his wife for 25 years (they will celebrate their 45th anniversary this month), and he said she knew that waiting around for late-night stand-up sets wasn’t going to fulfill him.

“She said, ‘Just go do it, and it doesn’t matter if it’s embarrassing,’” he recalled. “‘Just go do it. Just be busy, be amongst people, try to be creative.’” 

So, he went for it. Now, he admits: “I love Deal or No Deal. I loved it, and [it’s] probably the biggest success to date in my career in [terms] of notoriety.”

Never miss a story — sign up for PEOPLE’s free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer, from juicy celebrity news to compelling human interest stories.

Still, Mandel said he grapples with his decision to host two decades later.

“The point that I’m making is, not a day goes by when I’m in public where somebody says to me, ‘You should do comedy. I used to love when you were a comedian,’” he explained. “I love that you know me as the Deal or No Deal guy. I love that people watch me on [America’s Got Talent].”

“I love all those other things. It’s just that you forget my other art, or the art that is closest to who I am and what I need.”