Generating Quantitative and Qualitative UX Research to Drive Company Strategy
PRODUCT
-
Data Subscription Services
-
Wholesale and Retail Customers
-
B2B and B2C
MY ROLE
-
Primary UX Researcher
-
Conducted interviews from coast to coast
-
Created extensive survey of findings
-
Gathered and analyzed data
-
DEVELOP & DELIVER
-
Identify and Recruit Users
-
In Depth Interviews
-
Write and Administer Survey
-
Extensive Data Analysis
-
Interpretation of Data
-
Create New Company Strategy
DISCOVER & DEFINE
-
Determine Strategy
-
Analyze Methods
-
Gain Company Buy-In
-
Map Gameplane
The problem
Create a company-wide culture of innovation by gathering quantitative and qualitative user research, analyzing the findings and determining the future strategy for a company providing data subscription services to real estate agents.
Assumption Mapping
We began by mapping our assumptions to help define the high level goals and expectations. Key points from this process:
-
What is the company able to dependably deliver to the consumer?
-
What is the current company strategy for growth and improvement?
-
How accurate is the company database tracking ownership data from 75 million residences?
-
Can Realtors rely on the data in order to find new clients?
-
Can we create an effective strategy of innovation?
Goals
-
Conduct generative research in order to deeply understand what the user is trying to accomplish.​​​​
-
Understand all customer-defined metrics for success.
-
Align all departments including product, R & D, M & A, sales, customer support, marketing and training with new strategies.
-
Build products and provide experiences to reflect this new understanding.
​overview​
​
-
We conducted a case study of Landvoice to understand its approach to innovation and the challenges it faces. We analyzed Landvoice's approach to addressing its needs and deficiencies simultaneously, and the results of this effort.
-
We also reviewed the literature on innovation and strategy to provide a framework for understanding these concepts.
-
Finally, we conducted interviews with key stakeholders at Landvoice to gain insights into the organization's approach to innovation.
​
​
define innovation Strategy
​
-
​Innovation is critical to the success of any organization, and Landvoice is no exception.
-
Landvoice has been facing many challenges despite having no shortage of reasonable and logical ideas to effect the largest changes.
-
The lack of data to support one course of action over another has been a significant hurdle for Landvoice to overcome.
​
​​
Jobs to be done (JTBD) Theory​
​
-
After an extensive literature review, a JTBD approach was selected.
-
JTBD is a theory of consumer action that describes the mechanism that causes a consumer to adopt an innovation.
-
When we purchase a product or solution, we essentially “hire” it to help us do a job.
-
If it does the job well, we hire that product again the next time we’re faced with the same job.
-
If it does a poor job of helping us, we “fire” it and look for an alternative.*
-
People buy products and services to get their jobs done. While products come and go, the underlying job-to-be-done will not go away.
-
-
​The focus is not on the customer or the product, but on the underlying process the customer is trying to execute, or the "job" the customer is trying to complete.​
methodology
Qualitative Research
-
Stakeholder interviews​
-
Focus groups
-
Ethnographic field studies
-
In depth interviews
​
Quantitative Research
-
100+ question survey
-
Preprocess data in Excel
-
Analyze with Tableau
-
Calculate net opportunity scores
​
Companywide Alignment
Align all departments with research findings and rewrite the overall company strategy.
​​
qualitative Research process
Conduct extensive stakeholder interviews to understand the company's history and approach to providing lead services for real estate agents. Document business goals and potential future strategies for success.
​
Identify, screen and conduct 40+ user interviews and field studies; code and analyze the data.
Stage 1: Facilitate three focus groups with residential realtors to map out all of the major steps a realtor would have to complete in order to sell a home.
​​
Create a job map that visually depicts the core functional job, deconstructed into its discrete processes or job steps. This will detail what users are trying to get done and how they define success. The job map is solution agnostic. The data gathered is stable, eliminating the need to gather research for five years or more. Most products only get part of a job done; the key is to discover the entire job the customer is trying to accomplish.
​
Completed real estate agent job map from start to finish:
Complete
Actual portion of the job map that Landvoice currently addresses:
It is quite evident that the company could improve upon a number of jobs by creating new products that would address additional needs of clients.
Stage 2: Interview realtors and elucidate the steps of each category of the job map. Due to time and budget constraints, this was limited to eight categories:
​
-
Obtain seller contact information
-
Prepare for the conversation with the seller
-
Reach out to seller to obtain a listing appointment
-
Update the lead followup system
-
Define the emotional and social jobs.
-
Define related jobs to help complete transactions.
-
Define consumption chain jobs to improve user experience.
-
Define financial outcomes to enhance the business model.
Beginning with a blank framework, all the steps required to accomplish each category were discussed. Interviews were typically 2 hours and only a finite amount of steps could be surfaced at a time. The interviewer shared a computer screen and processes were carefully discussed and documented using wording directly from interviewees. Each subsequent realtor would review the steps elucidated in prior interviews and offer adjustments and corrections. In this methodical way, all of the job steps were discovered and were "peer-reviewed" by other real estate agents. This greatly improved the accuracy of the data and created a coast to coast consensus agreed upon by other professionals. The process was also solution agnostic to prevent future limitations when designing new solutions.
Stage 3: Extract 150+ statements in two specific formats that will be used later to create a survey. Statements were framed as either “Minimize the time…” or “Minimize the likelihood…”
-
Example: Minimize the likelihood of calling a lead that is on the Do Not Call List.
-
Example: Minimize the time it takes to import contact information into a lead followup system.
​
Primary Results:
​
​
​
​
​
Secondary Results:
Quantitative Research process
Goal
Create a quantitative survey drawn from UX interviews and observation. Administer to at least 90 customers. Their answers will reveal how important it is that they achieve each outcome and how well the solution they use today satisfies each outcome.
Methodology
Several different survey platforms were tested and determined that QuestionPro was the best equipped to accommodate side-by-side questions. One hundred questions were carefully written and checked by several realtors. Four versions of the survey were created with questions in different sequences to account for decision fatigue and acquiescence bias. Several trap questions were included in the survey to account for unqualified respondents.
​​
Question categories:
-
Demographics
-
Primary jobs
-
Related, social and emotional jobs
-
Supply, consumption and financial jobs
​
For each question, respondents provided 2 answers graded on the 5 point Likert scale. Realtors would rate each question for importance and satisfaction.
​
Sample of survey questions and possible responses:
​
​
​
​
​
​
​
​
​​
​
Gathering and Vetting Participants
Agents from coast to coast were invited to participate in the survey. Agents responded by email and were sent a short prescreening survey. Eligible agents were asked to read and sign a research participation agreement. Applicants were sent a link to the survey site when the agreement was signed. Completed surveys were examined and only qualified responses were used.
​
Data Collection Summary
-
580 agents enrolled in the survey
-
135 surveys were completed
-
29 survey responses were disqualified
-
Total qualified surveys = 106
​
Survey Geographic Response Dispersal
​
Completed Surveys by Location (Left) vs Density of Multiple Listing Services by Location (Right)
The survey response coverage was excellent, showing great representation of real estate agents from coast to coast. The survey results match North American MLS densities extremely well, including Canada.​
Pre-Processing of Data with Excel
-
Align and combine valid responses from different versions
-
Calculate importance and satisfaction scores
-
​Outcomes ripe for innovation
-
High Importance​
-
Low satisfaction
-
​
Net Opportunity Scores and Visualization With Tableau
We are most interested in the gap between the importance that customers assign to the expected outcome of a task and how satisfied they are with their current solutions.
-
Opportunity = Importance + (Importance – Satisfaction)​
-
Example: If importance = 8 and satisfaction = 2, then
-
Opportunity = 8 + (8 - 2)
-
Opportunity = 14
-
Scores higher than 13 or greater are highly actionable and will profoundly impact customers
-
​
Results:
data analysis
Highly Underserved Opportunities
Lowest Areas of Opportunities
The highly overserved opportunities indicate areas of improvement that will make a profound difference to our users. These are the areas that we pursued; for example, we improved lead accuracy and provided more qualified leads. The lowest scores also informed strategy in an important way. It demonstrate what is not important to realtors and that these needs were well met. This lead to significant defunding of certain programs; for example, eliminating the high cost of providing motivational speakers for customers. ​​
​
The Opportunity Landscape
​
All Opportunity Scores
This is a plot of all the points combined. What’s remarkable about this synopsis of the data is that 96% of the outcomes gleaned from the interviews have medium to high opportunity scores (any scores to the right of the center line).
This indicated two very important points:
-
The interviews were highly successful at determining what is most important to our customers.
-
Agents are largely unsatisfied with their lead generation efforts and may indicate a market primed and ready for a new and improved solutions that Landvoice can provide.
Scores above 13 are highly actionable
Strategy and companywide alignment
Outcome Based Value Propositions and Positioning Statements
Overall Strategy Goals
-
Reword value propositions to show how the product directly addresses underserved needs.
-
Reword positioning statements to show how the product addresses underserved needs better than competitors.
-
Select a target segment and communicate the strengths of a product that matter to them.
-
Replace product claims that don’t really matter to users with those that do, or even with previously un-messaged strengths that actually matter.
​
Formulate Product Portfolio Strategy
-
Borrow features from other company offerings.
-
Accelerate offerings in the product pipeline and R&D.
-
Partner with or license from other firms.
-
Acquire another firm to fill a gap.
-
Devise a new feature set.
-
Create new ancillary services.
-
Conceptualize the ultimate solution to get the whole job done better.
​
Marketing Campaigns
-
Enable potential customers search the web using JTBD keywords/phrases.
-
Knowing the job outcomes, create highly targeted key word campaigns.
-
Increase the chance of a potential customer who is searching for answers to their unmet needs seeing our marketing message and learning how our products can help them get the job done better, faster, and more cheaply.
​
Sales Messaging
-
Train sales teams to identify what outcome-based segments a customer belongs to and guide the conversation accordingly.
-
Approach customers with an improved value proposition and a clear understanding of their situation and unmet needs to build credibility.
​
Customer Support
-
Hire and train additional support staff to onboard and train new customers.
-
Create monthly webinars to showcase current platform features.
-
Reach out to existing customers that have not used Landvoice in the last 30 days and walk through important features and how to use them effectively.
​
Product
-
Create a product roadmap of new features.
-
Begin UX designs of each new feature.
-
Plan out release versions and dates.
​
Conclusion
This was a long, complex project with many interviews, observations and discussions. Creating survey questions directly from interview responses was a significant part of this project. Often a great deal of time and expense is spent on qualitative research and an unrelated survey is created to collect psychographic data, almost as an afterthought. This process of utilizing extensive generative research to write the survey produced excellent results and lead to a very accurate and effective strategy that pivoted the company in a new, productive direction.