Fallout Star Walton Goggins Says He Would Return to Justified Under 1 Condition (Exclusive)

Walton Goggins is suddenly a very busy man.

The actor, 52, is currently starring as The Ghoul in Prime Video's Fallout, which is based on the popular video game.

He's also in the midst of filming the third season of The White Lotus in Thailand, though sadly he can't reveal anything about his character or the plot. "We're filming in Thailand, that's all I can say," he tells PEOPLE.

However, Goggins knows that a lot of his fans are wondering if he'd ever return to the cult favorite, Justified. The neo-Western, which ran on FX from 2010 to 2015, featured Goggins starring as criminal mastermind Boyd Crowder, who was a longtime foe to Timothy Olyphant's Deputy US Marshall Raylan Givens.

"Yeah, everyone is primed and ready to do it," Goggins says of being down to reprise the role. "It's a matter of FX. We're waiting for them to kind of pull the trigger, and then it'll be a matter of working out schedules."

Still, he says just because there are interested parties, it's not a guarantee it will happen.

"Everyone behind the scenes, behind the camera and in front of the camera ,really want to do this last lap," he says. "But I think that for all of us, it has to make sense. Like it isn't just gratuitous, right?"

"It's not just to do another season of this for the sake of it. But I do think there is one last chapter of this story to be told," he adds.

Goggins recently opened up to PEOPLE about his interesting rise to fame, from growing up in a tiny house in rural Georgia, raised mostly by women and his mother's gaggle of fun, free-thinking and spirited friends.

"It was so colorful," Goggins said of his childhood. He says it was back then that he watched his aunt and uncle perform in regional theater, and realized that acting was something that he could possibly do for a living. When he was 13, he says he asked his mother to drive him to a casting agency in downtown Atlanta.

Even though he didn't have an appointment, he says he stayed in the waiting room until the agent would see him. "Eventually she was like, ‘F---, this kid isn’t going anywhere,’" he says with a laugh. He was right to stay put — he says the casting director became a mentor and lifelong friend.

In 1991, after two years of college, Goggins decided to pursue acting full-time and moved to L.A. He immediately landed small jobs on shows like Beverly Hills 90210 and JAG before finding work on films like Shanghai Noon and Major League: Back to the Minors. His film credits eventually grew to include Django Unchained, Lincoln and The Hateful Eight.

Now, Goggins says he's pretty much recognized everywhere he goes, and although he has fans that span generations from Gen-Z to Boomers, they still have a hard time remembering his name “Even if they recognize me, they still get my name wrong," he admits. "It’s usually ‘I love your work, Walter Scriggins!’”

Never miss a story — sign up for PEOPLE's free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer, from juicy celebrity news to compelling human interest stories.

Fallout is now streaming on Prime Video.