I did the same thing in my environment as well. We have over 900 monitored nodes and I setup the alert threshold based on 1 year average. I wanted to group them into pools of 20% range, so as to simplify the effective thresholds. I used the following formula in Excel:
=CEILING(CPU_AVG/20,1)*20
This of course was not without faults. I found two cases where manual interception was needed:
1. Some devices 1 year AVG CPU turned out to be higher than their normal operating conditions. This can be easily fixed by using Percentile value instead of AVG value.
2. Some devices kept on having spikes above their average value even with significant head room allowed by the 20% pools. They were exhibiting periodic fluctuations based on user activity. Finally we manually increased it to a higher value, because you shouldn't be alerted to something if there is no follow up action to it, in this case periodic fluctuations didn't seem to indicate any trouble.