RFP QUESTION 3.12
How we measure success
"Please explain how you would measure that the communications are effective."
Measurement starts by agreeing what 'effective' looks like — with the Trustee and members, before any communications go out. We use a five-step approach: define outcomes, test and learn, monitor short-term engagement, track long-term understanding and trust, then close the loop with clear reporting.
1 | Define what 'effective' looks like
Before any measurement takes place, we would work with the Trustee Executive, Member Experience Committee, and a representative member reference group to agree what success looks like for the Collective Plan. This might include:
- Members understand that benefits are targets, not guarantees
- Members understand why pensions can go up or down
- Members have confidence that Trustee communications are clear, honest and independent
- Members know where to find reliable information and use it appropriately
- Sensitive communications, such as adjustment letters, reduce, rather than increase, anxiety
These agreed outcomes become the measurement framework for reporting and review.
2 | Test messages and formats
A test-and-learn approach helps us understand what resonates before and after key communications. For example, testing different ways of explaining benefit variability, comparing text-led and visual formats and analysing open-text feedback and common member questions. This can be informed by member reference groups, pulse surveys and engagement through union or member forums. Insights from testing are shared with the Trustee and used to refine messaging over time.
3 | Monitor short-term engagement
Following key communication touchpoints, we would track short-term indicators including email open and click-through rates, website and portal usage, FAQ and video engagement, changes in portal registrations following communications and the volume and nature of member queries. These help identify whether communications are being noticed and accessed, and where clarification may be needed.
4 | Track long-term outcomes
Short-term engagement alone does not demonstrate effectiveness, particularly for a CDC scheme where understanding and trust are built over years. Long-term indicators would include:
- Trends in member understanding of CDC concepts — target benefits, variability, Trustee governance
- Trust and sentiment measures, particularly following valuations or adjustment communications
- Stability or improvement in trust scores year-on-year
- Reduced reactive contact and repeat queries following sensitive communications
- Consistent understanding across member groups — active, deferred and pensioner
5 | Report and improve
Measurement is only valuable if it drives improvement. We would provide the Trustee with clear, regular reporting, focused on interpretation and recommendations, not raw metrics. This creates a continuous improvement loop: communications are delivered, insight is gathered and reviewed, and future communications build on what has been learned.
The Collective Plan already has strong foundations in place — quarterly pulse surveys, engagement analytics and member feedback mechanisms. Our role is to help the Trustee make the most of that data and connect it to communication decisions.
Jump to a response
Each question from the RFP has its own dedicated page. Use the links below to jump straight to any response.