Simon Cowell appeared on Chelsea Lately Tuesday night to talk about what it's like being a judge on The X Factor and the upcoming One Direction documentary.

When Chelsea Handler asked Cowell, "I think it would be so annoying to have to sit and listen to untalented people all day long...Is this why you have such a drinking and drug and cigarette problem?" Cowell joked, "Probably yes." 

He then explained, "It is total utter torture sitting there for 10 hours a day trying to find one good person and you've got all these awful singers coming out."

When he sees "awful" singers who are 17 years old come onto the stage, he asks them why they would even compete on the show and they respond, "I'm blessed."

"It's like, you're not," the 53-year-old British judge told Handler.

While auditioning contestants for this season's X Factor, the judges - including Demi Lovato, Kelly Rowland and Paulina Rubio - did come across a talented middle-aged singer.

"We had this girl who was 54 who auditioned this year," Cowell recalled. "She, I would say, is probably the best singer we've ever had on this show...Her name's Lily."

Even if Lily is the best singer, the most successful group of singers from X Factor came from the U.K. edition. Most famously known as One Direction, the five boys originally auditioned as solo artists. Following the foot steps of Justin Bieber and Katy Perry, the British band filmed their own documentary titled This Is Us, which hits theaters Aug. 30. 

"With these five, you can't tell them what to do. No choreography, no dancing. They make all their own decisions," Cowell stated. "You see everything in this film. It's really really good."

Season 3 of The X Factor returns to FOX on Wednesday, Sept. 11 at 8 p.m. EST. Watch a clip from the Denver auditions below. 

Tags: Simon Cowell The X Factor Chelsea Lately Chelsea Handler