Holly Madison chronicled her life at the Playboy mansion in her 2015 memoir, Down the Rabbit Hole: Curious Adventures and Cautionary Tales of a Former Playboy Bunny, and next week, her latest publication hits bookstores.

Ahead of the release of her second book, The Vegas Diaries: Romance, Rolling the Dice, and the Road to Reinvention, Madison spoke to press about Playboy and her seven miserable years at the Mansion.

"It was a constant struggle," Madison told PEOPLE Magazine of her time at the Playboy Mansion with former boyfriend Hugh Hefner. "I was trying to sell this image of 'Oh everything is so great here,' but I was miserable inside. The Mansion is all about conformity and judgment and following the rules."

"I didn't like being a clone," she explained. "That was definitely something that bothered me the whole time. But the Mansion wasn't a place where confidence was built. It was a place where manipulation was used to keep everyone scrambling to keep her spot in the house."

In her latest book, Madison will tell readers about her venture to Las Vegas, where she embarked on a reality show, Holly's World, before meeting and marrying Pasquale Rotella, the father of her 3-year-old daughter, Rainbow.

"There was so much competition among the other women. I learned not to confide in certain people. There was a lot of fear. If I could go back and do things differently, I wouldn't be scared of people," she continued. "And I'd push the boundaries a lot more. The number of times I stuck up for myself at the Mansion I could probably count on one hand."

Although Madison doesn't have many good memories from her time with Hefner at his iconic Los Angeles home, she doesn't have any regrets either.

"It's a part of my life," she said. "I can't look back and wipe it away. It happened and I learned from it. [Now], I'm not trying to fit the mold of what anybody else thinks I should be. And I'm not running away from my past anymore."

The Vegas Diaries: Romance, Rolling the Dice, and the Road to Reinvention hits stores on May 17.