If you thought you’d heard the last of the It Ends with Us drama—which included rumors of fat shaming, reports of fights in the editing room, and a crisis PR manager—you thought wrong. With the controversial film hitting Netflix Dec. 9, rumored Blake Lively feud partner Justin Baldoni is back in promotion mode, appearing on the Dec. 4 episode of the How to Fail podcast. And while he didn’t address the reports of on-set discord, he did admit filming It Ends with Us was not exactly an enjoyable experience.

ICYMI, Justin wasn’t just a director on the film, but he also starred in the Colleen Hoover adaptation as Ryle, an abusive partner, opposite Blake’s character Lily, the protagonist and florist just trying to survive in her floral world. It turns out that playing a domestic abuser and running a movie set are not entirely compatible.

“Directing is a very lonely job,” Justin told How to Fail host Elizabeth Day. “In your moments of quiet, everybody has a thousand questions for you and also nobody wants to disturb you. And you don’t really have many people to talk to and you can’t necessarily share your anxiety or your nervousness about something because you’re also the leader.”

it ends with us
Sony Pictures Releasing

“It’s a very strange place to be,” he continued. “Let alone directing while trying to play a character who does the things that Ryle does in the movie.” Sometimes it would get so intense that he would have to step away between takes. “There were moments in the filming of this where I would just have to leave. I’d have to remove myself and go shake it out.” He recalled one instance in particular, while filming a scene where his character finds a phone number and gets extremely jealous, which was especially difficult. “He’s angry and he doesn’t harm her, but you can see in his eyes how dangerous he is. After that scene, I had a near breakdown,” he said, “I had to leave and just cry and shake because there was so much pain.”

Justin added that it took him months after production before he was able to really let go of the character. Having said that, it probably won’t surprise you to hear an It Ends with Us sequel isn’t top of mind for him. “We’re not even thinking about a sequel right now,” he said. “Just enjoying the fact that this movie that’s taken years and years to come to fruition is actually having the true impact and is now going to be able to be seen by so many people on Netflix.” Fair enough!

Of course, even if they do end up making an It Ends with Us sequel, likely based on the book sequel It Starts with Us, there’s no guarantee Justin would return. The man himself dodged questions about whether or not he’d be back in the director’s chair for a follow-up, saying he’d like Blake to direct. And while his character Ryle is in the story, based on the reports of behind-the-scenes drama, I wouldn’t be surprised if Justin chose to leave the potential franchise behind entirely.