Although "Downton Abbey" was given the green light for the fourth season by U.K.'s ITV network, there have been a slew of rumors swirling about one of the show's main characters not returning.

Production of the show's upcoming installment is scheduled to begin February, 2013.  However, a close source previously informed the The Sunday Express that actor Dan Stevens (Matthew Crawley) won't be finishing out the upcoming season.

"Dan Stevens isn't returning to the series. He will probably do the first episode of the fourth series, but that will be it," stated the source.

According to the Daily Mail, the 30-year-old actor will make his very last appearance in the opening episode of season 4 before leaving to pursue other acting opportunities.

"Sometimes actors feel they want to move on.  Initially, for the first two or three [seasons] you can contract them, but after that, every season, you have to re-contract them," stated writer Julian Fellowes.

"If they don't want to come back, there's nothing we can do," he added.

Fellowes was asked by Entertainment Weekly whether he would consider re-casting the Matthew Crawley character and gave the following response:

"I can't see us doing that. I think we'll go on jogging quite happily through the 20s. You know there's always a little time jump," he said.

The "Downton Abbey" series takes place in the fictional Yorkshire country and follows the lives of the aristocratic Crawley family following the Edwardian era. The show's premise revolves around historic events that occurred during that respective time period.

With Season three now wrapped, "Downton Abbey" will next air its two-hour Christmas special, which was actually shot this past August.  The featured piece is set to air in the U.K. on Christmas Day and is said to involve the Crawley family taking a trip to unfamiliar territory, as they head to Scotland to visit their relatives.

The show's third season is set to premiere Jan. 6 in the U.S. on the PBS network.