Travelling in history.

I have a fascination with humans measuring and calculating time. We all measure a day as 24 hours. An hour is 60 minutes and a minute is 60 seconds. Didn't need a genius to tell you that. Anyway, the Earth actually spins on its axis once every 23 hours 56 minutes and 4.1 seconds. So I was wondering, what happened to those 4 minutes, approximately. we are adding an extra 4 minutes to each day so we are 4 minutes ahead of the new Earth day, each time we measure 24 hours. So how do we account for the lost time?