why did you let them roll in the first place?

Because I don't have everyone's modifier for every skill, ability, saving throw, and attack memorized off the top of my head

Why the hell not? You're the DM. Why do you not have copies of your player's character sheets?