Financial institutions have a double burden when facing the digital era: keeping up with changing technology and keeping up with regulators’ attempts to respond to and encourage those changes. In response to consumer demands, financial institutions are developing mobile apps and mobile-responsive websites of their own. Their consumers want access to the goods and services they once got from brick-and-mortar stores (or their PCs) on their mobile devices. To provide the convenience they have come to expect, financial institutions need to utilize methods for creating the most attractive, responsive and effective mobile products possible. At the same time, regulators have focused significant attention on how banks and other entities they oversee can use Internet-based tools to impact consumers.
A/B Testing is one standard protocol for developing and improving web products. It calls for presenting two versions of a product to different groups of users and measuring their reactions against a metric of success. Despite being “industry standard,” this approach is receiving increased criticism from consumers and may be risky for financial institutions.
A/B testing recently dominated conversations regarding Facebook’s treatment of its users after it presented upbeat or depressing stories to different groups of users and then collected and shared the resulting data about the users’ behavior. This move caught tremendous flack in the media, including claims that Facebook may have manipulated consumers. Facebook and its defenders noted that this sort of testing is routine at technology companies. Presenting different versions of a product to different sets of users and comparing the result is an effective, common tool for evaluating potential development strategies. Such testing allows developers to treat qualitative factors quantitatively, providing clear evidence of what seemingly personal preference will actually have the greatest usability impact.
As mobile products reach into ever more sensitive corners of consumers’ lives, this approach is gaining increased scrutiny.
A/B testing may be a foundation of good development for the larger world of mobile, but financial institutions entering the mobile market should consider some potential liabilities:
>> TILA/RESPA: Regulators are beginning to recognize that disclosures cannot be created for mobile devices under the same restrictive font and other requirements that apply to paper disclosures. However, even those disclosures have been loosened and remain ambiguous, without concrete agency action or comment to describe what is and is not, acceptable in terms of delivery and appearance. The A/B testing context presents a particularly risky situation, as regulators may see two versions of a set of disclosures as favoring one set of consumers over another.
>> Gramm-Leach-Bliley Act (GLBA): Using an outside app or web development vendor may streamline product creation, particularly for smaller companies, but the introduction of a third party creates GLBA complications for financial institutions. It may be unclear which party is actually doing the A/B testing or collecting the related data, increasing chances of accidentally ignoring a data-sharing opt-out request required under the Privacy Rule. Involving third parties also creates additional burdens on the information security programs required by the Safeguards Rule.
>> Future Regulation: The CFPB has zeroed in on mobile financial services, requesting information from the public, hosting a field hearing on the topic and promising to release new rules for mobile by the end of the year. CFPB Director Richard Cordray specifically noted that the agency is investigating the information banks are collecting from their consumers and whether they are “using their data to target [low-income consumers] for high-cost products.” An A/B test designed to evaluate different pitches for a new loan product could be perceived as demonstrating this behavior if presented to the wrong users. Bad press on testing gone awry would draw negative attention at a crucial moment and may lead to enforcement action or stricter rules.
>> Consumer Trust/Reputational Damage: Facebook’s recent matter demonstrates that regardless of the legal implications, getting on the wrong side of public sentiment regarding consumer manipulation is at best a distraction and waste of company time and resources. For financial institutions, organizations with which consumers trust with their most sensitive information, this is even truer. Imagine the backlash from users misinformed about their access to credit or account due dates, or who find out that data about how they handle their money was shared with researchers.
Avoiding mobile is not an option for financial services companies that want to remain competitive, satisfy customers and ensure they are engaged with their products and services. Mobile financial services are here to stay, and providers must focus on creating the best mobile experience while reducing the risks associated with app and website development. Financial companies must think beyond the standard A/B testing box: focus groups, opt-in programs and surveys – all of which may require additional planning and incentives, but will help avoid many of the legal and reputational obstacles that quietly testing a user base raises. With thoughtful planning and a little creativity, A/B testing will produce desired results without collateral damage or undue risk of negatively impacting their customers.
About The Author