Even though he was the one who brought his large family's lifestyle into the spotlight with their popular show Sister Wives back in 2010, Brown family patriarch came very close this year to quitting and putting an end to the reality series once and for all.

In a sneak peek clip for the Tell-All special scheduled to air after the March 1 season finale, Brown admits he nearly quit the series after some negative backlash at the beginning of the season, when his on-air fight with third wife Christine aired.

During the episode and the argument, Brown made a comment to Christine about her PMS-ing-which led to a flurry of negative comments and attention. And though Brown is used to some negative attention from critics of his lifestyle, the attacks he faced following the episode were enough to make him seriously want to give up.

"I literally started talking to the family the week after that aired," he says in the clip. "I started talking about, how can we get out of this show? I was dead serious, I had had enough."

The incident which set off the firestorm involved Brown saying PMS made his wives "insane," and also saw him accusing Christine of not being responsible with her behavior-statements he now admits he regrets making.

"I'm sorry that I didn't understand it well enough . And I'm sorry that it becomes a very offensive and polarizing thing," he says. "I actually apologized to Christine about this whole experience, but the PMS subject-in spite of being married for so many years-was a new problem to me. And so, when it became a real thing to me, it was a place where I was very insensitive about it."

The Sister Wives season 5 finale airs Sunday, March 1 at 8 p.m., with the tell-all airing directly after at 9 p.m.

Watch the preview clip here.