So that's a no then...
The 34-year-old admitted on Thursday’s edition of This Morning that he turned down Simon’s offer because he wanted to remain loyal to his new show.
He told hosts Holly Willoughby and Phillip Schofield: ‘Last year he tried to tease me back and I said to him “I just love The Voice too much.
‘When I was presenting [X Factor] The Voice came out, I wasn’t considered as a judge on the X Factor and then it came up that I could be a judge on The Voice so I went for it.’
Olly also appeared to throw shade at the show that made him a star, saying he thinks it’s about time it took a break.
He added: ‘If they are going to take a break with the show I think it’s a good thing to do, I think they need that.
‘It does need a little break from what it’s doing but I think Simon has seen that.
‘I love the show, I always have, but I felt that something wasn’t clicking on the show, it needs something different and hopefully this does that.’
Olly’s opinion comes after Simon revealed that he is resting the usual format this year in favour of celebrity and all-stars versions.
‘The time feels right, and it will be a huge change. We haven’t exactly confirmed the dates or the order of the shows, but essentially there are going to be two versions of X Factor running this year,’ Simon told The Sun.
‘I think and hope the panel will be the same. I’m literally speaking to Robbie tonight because he’s been busy, I’ve been filming, so I’ll have more news again on that soon.’