Sean Penn still remembers when he heard the rumor that accused him of hitting Madonna with a baseball bat.

Allegations of domestic violence, which Madonna has since denied, were in the news during and after the couple’s rocky marriage from 1985 to 1989. However, Penn said he first heard the particular story about the bat over a morning cigarette.

A woman he was dating confronted him about it after “a lovely evening” together, he told The New York Times in an interview that published Saturday.

“She’s looking at me like I killed her dog,” the “Mystic River” star said.

“I didn’t know what the hell she was talking about,” the Oscar winner added. “Now I think it’s fair to say that I’m not the biggest guy in the world. But if I hit Mike Tyson in the head with a baseball bat, he’s going to the hospital.”

The rumor that Penn hit the singer with a bat in 1987 fit in with the narrative around the pair’s relationship that flared up when Madonna filed an assault complaint against Penn in December 1988. While she reportedly withdrew the complaint a week later, police flooded the couple’s Malibu, California, home and ordered Penn outside with his hands up.

“I had a freaking SWAT team come into my house,” the 63-year-old actor told the Times. “I said: ‘I’m not coming out. I’m going to finish my breakfast.’ The next thing I knew, windows were being broken all around the house and they came in.”

The allegations of abuse drew attention again in 2015 when director Lee Daniels defended his “Empire” star Terrence Howard by comparing the actor’s own admitted domestic violence to Penn’s alleged actions with Madonna.

Penn, who was also rumored in the tabloids to have tied Madonna up and left her “trussed up like a turkey” for nine hours, sued Daniels for defamation in the sum of $10 million in damages. Madonna provided an affidavit in support of Penn.

“While we certainly had more than one heated argument during our marriage, Sean has never struck me, ‘tied me up,’ or physically assaulted me, and any report to the contrary is completely outrageous, malicious, reckless and false,” she said in her statement at the time.

Penn told the Times that Daniels had to issue an apology and contribute to his nonprofit charity efforts as part of the settlement in 2016. He also said he’s long repaired his friendship with Madonna, as has been evident with their various in-person reunions.

“She’s someone I love,” he told the outlet.

Need help? In the U.S., call 1-800-799-SAFE (7233) for the National Domestic Violence Hotline.