Queen Elizabeth II has been the stabilizing force of the monarch over the years, a strong foundation that has weathered some major controversies that could have destroyed the reputation of the royal family.

Recently, however, the Queen has been tested once again after three key senior members decided to step back from their royal duties.  

Royal Family's Huge Blow Caused By Prince Andrew, Sussexes

Last year, Prince Andrew announced that he would withdraw from public duties after his dsastrous BBC interview linking him to the deceased American financier and sex offender Jeffrey Epstein. 

It came just a few months before Prince Harry and wife Meghan Markle announced their plans to quit the firm and seek independence outside Britain. 

Unfortunately, after the Duke and Duchess of Sussex officially stepped down from their roles last April, it was reported that the Queen banned the two from using royal as their branding and stripped them off of their HRH titles.

Aside from this, the controversial couple was asked to repay $3 million worth of taxpayers' funds spent on renovating their Frogmore Cottage home.

The Duke of York, on the other hand, was able to keep his HRH title as well as his security despite not being a working member of the royal family. 

In addition, the 60-year-old royal is still living at his 30-room residence in Royal Lodge in Windsor Great Park in Berkshire with his ex-wife Sarah Ferguson, their daughter Princess Eugenie and her husband, Jack Brooksbank.

Is Queen Elizabeth II Unfair To Meghan Markle?

With this, royal expert and writer Daniela Elser cited in News.com.au that the 94-year-old monarch has ignored Prince Andrew's "shortcomings."

Moreover, Elser pointed out that the Queen's decision not to  punish Prince Andrew appears to be "unjust" and "racist" towards Prince Harry and Meghan.

"If Her Majesty does not, in some fashion or another, address this contradictory situation, it will start to make her decision regarding the Sussexes' look not only petty and deeply unjust but it could take on a racist hue," Elser wrote.

Unfortunately for the Queen's favorite son, the Duke of York's ongoing word war with the U.S. authorities regarding the Epstein probe has affected the support of Her Majesty, given that she "is acutely aware of the public mood on the matter."

"The idea that the Queen will simply indulge Prince Andrew ... is wide of the mark. Her patience has been wearing thin for a long time. She had resisted this slimming down of the monarchy but it's fair to say she is not now standing in the way of that in her lifetime," a source told Sunday Times. 

To recall, the U.S. Department of Justice has formally requested to speak to Prince Andrew as part of their criminal investigation of Epstein. 

Following this, the Duke of York's legal representative released a statement saying that their client has offered his assistance in at least three occassions, but to no avail.

READ MORE: Meghan Markle Too Ambitious, Slammed For Being A 'Fame Addict'