Rumors have been spreading that Robert Downey, Jr. will be done with the Iron Man franchise before Iron Man 4 can even be made. Although Downey, Jr. hasn't confirmed giving up the suit, could the iconic actor really be done with Tony Stark?

Why Robert Downey, Jr. Says He May Be Done With Film

Downey, Jr. admitted his age could come into play if he decides not to do Iron Man 4. "The smart money is you have to look at everybody's age," Downey, Jr. said. "I'll put myself at the top of the list. Sooner or later, they've got to start over and get somebody young. I'm not there with them yet."

Colin Farrell's name has even been thrown around as the new Tony Stark ever since fans voted him their first choice to replace Downey, Jr. in an online poll.

Was Robert Downey, Jr. Not First Choice For 'Iron Man'?

Now, Jon Favreau who directed Iron Man and Iron Man 2 is saying that he'd love to see Downey, Jr. continue the role, despite Downey, Jr. obviously getting older. "I think people would love to see Robert Downey continue the Iron Man character," Favreau said, "but he's done it quite a bit now. I've had a few movies off, so it's a lot easier for me to say that it should be done. I hope he continues doing it and, as long as he fits in that suit, there's no reason he shouldn't." Downey, Jr. currently has a role in Favreau's new film Chef.

Even if Downey, Jr. hangs up his suit for Iron Man 4, the actor will be in Avengers: Age of Ultron, which is supposed to open May 2015.