Taylor revealed last week that he had not discussed rumours linking former Sheffield United boss Neil Warnock with his job following a disappointing start to the new season.
But although the pair have recently spoken, Taylor remains unconvinced of his future at the helm of the Eagles.
"We had a discussion, but to be fair it wasn't so much football-related," he told the South London Press.
"We didn't talk about my future. Simon's not the type to go into that and neither am I.
"We need to win more matches to take any cloud away. Results take care of things and there will be less rumours about Neil Warnock if we stop giving away stupid goals, because we'll keep winning matches."