Creating custom metrics is powerful - but it's also easy to create metrics that look impressive while hiding the truth. Vanity formulas are custom KPIs that make performance look better than it is, can be gamed, or don't drive action.
Warning Signs of Vanity Formulas
- Always goes up: A metric that never declines regardless of business performance is measuring the wrong thing
- Too many inputs: Metrics with 5+ inputs often average out meaningful signals. Simplify
- Cherry-picked timeframes: A metric that only looks good with specific date ranges is misleading
- Not actionable: If no one can change behavior based on the metric, it's reporting theater
- Easily gamed: If teams can inflate the metric without improving actual performance, the formula incentivizes the wrong behavior
Common Custom Metric Pitfalls
The Denominator Trap
"Revenue per Active User" sounds great - until you define "active" as anyone who logged in once. Tight definitions inflate the denominator; loose definitions inflate the ratio. Be rigorous about definitions and document them clearly.
Survivorship Bias
Measuring only customers who stayed ignores those who churned. "Average customer satisfaction among active users" hides that dissatisfied users already left. Include churn and drop-off in your metrics.
Double Counting
Combining correlated metrics amplifies noise. If your composite includes both "page views" and "time on site," you're partially double-counting engagement. Use independent signals.
Principles for Honest Custom Metrics
- Can someone take action on it? If the metric changes, does anyone know what to do differently?
- Can it go down? A metric that can't decline is a celebration metric, not a management metric
- Is it comparable over time? Changing the formula or definition breaks comparability. Version your metrics
- Does it resist gaming? If teams can hit the target without improving the underlying outcome, redesign the formula
- Is it simple enough to explain? If you can't explain it in one sentence, it's too complex for decision-making
Metric Governance
Establish a metric governance process: document every custom KPI with its formula, data sources, owner, and review date. Review and validate custom metrics quarterly. Retire metrics that no longer drive action. See our marketing data governance guide for frameworks.
How Xplainr Helps
Xplainr's AI validates your custom metrics by checking for common pitfalls - correlated inputs, gaming risks, and definitional inconsistencies - before you roll them out.
See AI Diagnosis in Action
2-minute setup • No credit card required
Related Resources
- Correlation vs Causation
Avoid misleading metrics caused by correlated data.
- Data Quality Frameworks
Governance for reliable custom metrics.