ATC clearance question

I’ve got an ATC clearance question for you guys.
While on an Instrument flight plan, ATC tells you to “climb and maintain 5000 feet” and you comply. As you begin to get closer to your destination, you ask ATC if you can fly the published hold. ATC clears you to fly the hold as published.
The altitude for the hold is at or above 2000 feet. Are you cleared to descend to 2000 feet or must you maintain the prior clearance of 5000 feet?
My CFII had me start descending and at around 4400 feet, ATC gave us a stern talking to and got into a back and forth with my CFII.
ATC says we weren’t authorized to leave 5000 feet, my CFII says being cleared to fly the hold as published is authorization. Nothing further came from this. We climbed back to 5000 feet and then continued with the approach.

Upon landing, we asked some other CFI’s at the school and they all agreed with my CFI, we were cleared to descend to the published altitude.

I’ve googled and asked some AI’s and they say we were in the wrong, ATC was correct. As I’m a student who is learning, I’d like your guys opinions.

https://redd.it/1x2999v
@r_aviation