With over 300 locations across the US and 10,000 employees, Bowlmor AMF retained Studio28 to assess the customer experience (CX) practice at the corporate and local levels. We analyzed the performance of their enterprise feedback management vendor, the validity of solicited NPS surveys (100,000 yearly) and the reliability of the algorithm used to classify unsolicited feedback (online comments) from Yelp and Facebook.

Initial Discovery

Evaluated the current EFM vendor in order to establish current state vs future optimal capabilities.

• Performed a detailed assessment of how CX survey results, secret shopper reports and online sentiment data were integrated into dashboards and email alerts to inform operational decision-making.

• Observed operations at several locations where we interviewed managers and line staff on how their operations and internal systems impact their ability to utilize CX feedback in their daily work.
• Thoroughly assessed the statistical validity and methodology behind each individual CX survey, secret shopper reports and the reliability of the artificial intelligence template classifies online sentiment.

Findings

• Corporate managers and staff found the CX feedback interesting but often of questionable value. Users disliked the interface, system dashboards were irrelevant to them and the majority never logged in to view reports.
• Methodological errors in the survey questions made the results and their impact on key performance indicators (KPI’s) unreliable.
• The artificial intelligence algorithm used to analyze the sentiment of the natural language in survey responses and online comments often misclassified the results and generated faulty recommendations.
• When asked, the vendor was unaware of these issues, agreed that they were important and committed to working with the client to fix them.

Results

• The client asked Studio28 to evaluate alternative vendors and present a shortlist of potential replacements.
• Studio28 gathered and documented detailed requirements for the enterprise CX vendor candidates, researched alternative vendors and narrowed the candidates down to four.
• We then prepared a request for information (RFI) for vendor candidates and arranged for the presentation of system demos.