The rock matriarch, the wife of Black Sabbath star Ozzy Osbourne, was due to join music boss Simon along with Robbie Williams, Ayda Field and Louis Tomlinson to help judge the live shows on the competition when they kick off later this month.
However, Sharon told fans in a statement that she has decided not to take part in the programme because she doesn't see where she fits in.
"After watching the new season X Factor unfold, I've seen the new judges finding their rhythm and are doing brilliantly," she wrote. "Simon was so gracious to find me a spot on the live shows but at this point I really don't think I'm needed and I would honestly feel odd coming in at this point. So I have decided to pass on the series this year.
"I wish the show continued success and as much as I love to get my face on TV you will have to do without me this season. Loves ya, Mrs. O."
Sharon's statement comes amid rumours that Simon decided to kick her off the programme following her recent interview with shock jock Howard Stern, in which she labelled Simon a "fat faced mother f**ker" and joked about drowning him. She also labelled the show's auditionees as "little s**ts" and likened their performances to "karaoke."
And while Sharon said in her statement that if her departure "reads differently in the papers then it's been altered", a source told British newspaper The Sun that she was always going to try and make it seem as though it was her decision to leave.
"Sharon is going to try and spin this as her choice to leave, but after that disgusting interview there was no way Simon Cowell or the executives would have her back on the panel," the insider said. "Not only did she make shocking attacks on him, she slagged off the show itself and the lovely members of the public who decide to audition. Good riddance to her."