Do you always take riding first as Imperius small,1v1 drylands?

I do this basically every game no matter what to prepare to contest the first median. Any time you shouldnt?