Using A/B Design Methodologies to Test user EXPERIENCES
Designing with Data
The DoorDashers' Stories
DoorDash has been a lifeline for both restaurants and drivers since the start of the COVID-19 pandemic. The increase in online orders has been helping restaurants keep their doors open, while the compensation and tips received from fulfilling these orders have been helping millions of Dashers facing economic hardships.
This proposal focuses on determining whether users of food delivery applications are more likely to tip or increase their tip amounts if there’s a personalized message from a delivery driver.
We predict that…
If users are shown their Dasher’s personal story
users will have an empathetic viewpoint of Dashers’ hardships during the COVID-19 pandemic
which will increase the likelihood of tipping BEHAVIORS and increased tip amounts.
A/B Testing
A/B USER INTERFACES
-
Test Cell A - Checkout Screen (No Changes)
Test Cell A will receive no changes in the DoorDash Checkout screen. This is where users select their tip amounts.
-
Test Cell A - Order Status Screen (No Changes)
Test Cell A will receive no changes in the DoorDash Order Status Screen.
-
Test Cell B - Order Status Screen (View Story Added)
Test Cell B will receive an updated UI with a “View Story” button in their DoorDash Order Status Screen.
-
Test Cell B - Dasher Story Screen (New UI Element)
Test Cell B will receive a DoorDasher Story UI pop-up when tapping the View Story button. This is where users can read their Dasher’s story and decide if and how much they would like to tip.
Data Analysis
DATA VALUE TYPES
Categorical values:
If the user viewed a Dasher’s Story or not (Test Cell B)
Tips left for Dasher (Test Cell A & B)
Continuous Numerical Value:
Tipping mounts (Test Cell A & B)
T-test - With a .05 alpha to compare the tip amounts of users tipped in Test Cell B to the amounts of users tipped in Test Cell A.
DATA Testing Methods
Proportion Test - To analyze the proportion of users who tipped after reading the story in Test Cell B to the number of users who tipped in Test Cell A.
CONSIDERATIONS & Conclusion
How will we determine success? If the analysis of the results of A/B testing proves our hypothesis and the experiment succeeds, a secondary experiment should be conducted to verify the first experiment's results per our budgetary limits.
Interpreting Results
How do we ensure this feature does not become another dark pattern? While not considered illegal, is playing on people's emotions for tips morally unethical? Could delivery people utilize the platform in nefarious ways?
Design Implications
Should this feature be permanent? Or periodical? For instance, many people took delivery jobs as a form of employment during the global pandemic. Would it have been beneficial to include this feature only during a worldwide time of hardship?
Potential Pitfalls
What does empathy look like across cultures? Would this feature even matter outside of countries like the United States?
Further Exploration
While there is some research on Online Food Delivery, the factors determining tipping behavior in this industry are yet to be determined. While our study aims to determine if sharing personal stories affects tipping behavior, what this suggests about empathy toward the delivery persons/agents remains unquantified. However, we hope our hypothesis will help quantify this behavior to some extent and help establish a relationship between tipping behavior and emotional incentives.