Drill-Down using time has a huge problem

When I start drilling into a year, and move into a quarter drill down, the drill down is reversed and shows me data for each quarter for every year again. That's completely illogical in my opinion. The function behind a drill down method should remember your filter/drill down step and keep filtering data in that manner. In order to avoid that problem I need to put slicers for the year and quarter in the report page just to make sure the data I see is not wrong.