Although Jarvis is out of contract, Gillingham will receive a transfer fee - which has been agreed between the two clubs - as he is under 24.
"The manager Mick McCarthy seems like a man who's very keen on encouraging you," Jarvis said.
"I delayed my holiday to complete the signing, and I'm looking forward to coming back for pre-season training."
Charlton, Nottingham Forest and Plymouth had all tried to sign the youngster, who was voted into last season's League One team of the year.