Measures in Dashboard should not refresh if neither of DAX formula and data changes

Observed:
There is a slicer and a matrix on a dashboard, the slicer has nothing to do with the matrix. Now if I remove the slicer, everything in the matrix refreshes, which takes about 10 minutes.

Expected:
Any element in Dashboard, should NOT be refreshed if neither of its formula or underlying data changes.

I'm not a professional programmer but what I observed is: Power BI tries to refresh as MANY times as possible to make sure that it doesn't lose any change. It cannot tell if the change is related or not, so basically it refreshes everything on each change.