Jessica Simpson is angry that her ex-husband Nick Lachey made a joke about her father being gay.

Lachey and his younger brother, Drew, appeared on Bravo's Watch What Happens Live with Andy Cohen on May 7. When the host asked what was the best part of no longer having Joe Simpson as a father-in-law, Drew wanted to answer first and replied, "everything."

Nick took the answer a bit farther.

"The best thing about not having Joe Simpson anymore as a father-in-law is I don't have to play grab-ass under the table on Easter Sunday," Nick said.

After he made his comment, Cohen was not sure how to react and said he was blushing. Nick joked that his publicist would not be happy but that he did not care in the moment.

"It may not be [funny] tomorrow but it is tonight," Nick said.

The remark touched a nerve with his ex-wife since Joe has been accused of being secretly gay. In October 2012, allegations surfaced in the National Enquirer that the elder Simpson was carrying on an affair with a 21-year old model Bryce Chandler Hill.

"Jessica was shocked and horrified that Nick said what he said. She's upset. Nick might have thought it was funny and he got a cheap laugh, but to Jessica, it wasn't - at all," a source said.

Jessica allegedly felt that her ex-husband should have shown more restraint and respect. The fashion maven did not understand why he would bring up her father when they have been divorced for years.

The two ended their marriage in 2005 after three years together.

"It's been eight years since they were married and six years since they last spoke," the source said. "Jessica doesn't feel that Nick has a right to speak about her business anymore."

Nick told Cohen during his interview that he had not spoken to Jessica in six years. Nick also shared that he no longer watches episodes of Newlyweds: Nick and Jessica, the MTV reality TV show that starred him and his former wife.

"I have not. As you can imagine, that's not popular viewing in my house. We don't sit around as a family and watch the old DVDs," he said.

Nick married Vanessa Minnillo in 2011 and they welcomed their son, Camden, in 2012.