Bangladesh captain Mushfiqur Rahim added a defiant 87-run partnership with Sabbir Rahman for the sixth wicket to pull things back for the hosts.
Debutant Sabbir Rahman scored a valiant half-century and kept Bangladesh alive as they finished on 253/8. On Day 5, home side need 33 runs while England are two wickets away from a victory.
Kayes fell for 43 off 61 balls as Bangladesh reached 86 for 2 at Lunch.
Bangladesh lost Mominul Haque (above) and Mahmudullah in quick succession to Gareth Batty, giving visitors the control of the game again.
Batty broke the stand and Stuart Broad rattled the tail to leave Bangladesh in a pickle.
After England folded for 240 and set Bangladesh a target of 286, Imrul Kayes set the tone for Bangladesh with a breezy start.