Having declared on 575-9 at Lord's on Friday midway through the second day of the opening Test, England reduced the tourists to 140-1 by close of play, with Dimuth Karunaratne the man to go for 38.
Karunaratne departed in the 13th over and England thought they had a second wicket when Kaushal Silva nicked a delivery to Prior that he took low down to his right.
Both the bowler Stuart Broad and Prior thought that the catch had carried, but the opening batsman was given a reprieve after television replays suggested otherwise.
Silva used that to his advantage to finish the final session 62 not out and leave Prior frustrated.
"You saw my reaction, I had no doubt about the catch. I thought it carried," he told Sky Sports. "I thought I got my hand, certainly my finger, under the ball.
"There's a rubber tip on keeping gloves and you can feel when it hits that.
"I've gone forward at the ball and certainly thought it carried but it's always tricky when it goes upstairs with the camera angles.
"We're disappointed but we get on with it."
It capped a frustrating day for the wicketkeeper after he was only able to add 10 to his overnight score before being dismissed by Shaminda Eranga for 86.