Inconcievable internal clock drift during deep sleep

My setup function:

1) do some quick stuff 2) check the time online 3) compute x = how many seconds until the next "o'clock" 4) go in deep sleep for x seconds

So, the idea is to have the esp32 to wake up every hour. I don't care about precision so I can live with possible drifts of the internal clock.

The problem is, these drifts seem to be cumulative: the first hours I have a drift of 10-15". The second hour I've got 30-40". The third 1'... and so on. How is this possible??