2018
Case study: Validating Elon Musk’s news website idea
Using Lean UX Design methodology to determine if Elon Musk’s news rating site is a valid product idea
Musk’s Big Idea
In May of 2018 Elon Musk, Founder and CEO of SpaceX and Tesla announced on Twitter that he was:
“Going to create a site where the public can rate the core truth of any article & track the credibility score over time of each journalist, editor & publication.”
Musk created a twitter poll asking people to vote on the idea. The results were overwhelmingly in his favor. However, this hardly counts for good market research. To understand if Musk’s idea is truly a good one, I had to dig a little deeper.
Following Lean UX Design methodology, there are three fundamental questions that need to be answered before it can be determined whether or not a product idea is stellar or bunk:
What is being defined as the problem and is it actually a problem?
Is there a market or audience that wants or requires a solution to this problem?
What is the solution being put forward and does it solve the problem as defined?
This case study will answer the first two questions.
Validating the Problem
Musk’s product idea rests on the assumption that people no longer have enough trust in mainstream media and that that is a problem. While these assumptions might be true, it’s not good enough to simply rely on anecdotal evidence. There are a number of organizations that have performed polling research on this very topic.
Gallup
Starting in 1997 Gallup started polling Americans annually about the trust they place in mass media. Based on their polling data, confidence in mass media has fallen from 53% in 1997 down to 32% in 2016.
According to this data, the most severe drops in media trust have occurred among Republicans.
Both Democrats and Independents also show a reduction in trust in “mass media” going into 2016.
According to more recent Gallup polling research done in 2017, a reverse in the trend seems to have occurred. Confidence in newspapers grew from 20% in 2016 up to 27% in 2017. It should be noted that this research is specifically in relation to “newspapers”, whereas the previous research relates to “mass media”.
Pew Research Center
Pew Research Center performed similar polling research in 2016 and 2017, which shows results that are similar to Gallup’s.
In both 2016 and 2017 only one-fifth of adults said that national news organizations do a “very good job”.
Similar to Gallup’s data, a widening gap between Democrats and Republicans with regard to trust in national media can be observed.
NYU & Stanford University Research
In the spring of 2017, a joint research paper was published in the Journal of Economic Perspectives. The research paper is mostly concerned with the proliferation of “fake news” on social media and how it may have impacted voting patterns. The paper cites Gallup’s polling statistics, speculating that “The declining trust in mainstream media could be both a cause and a consequence of fake news gaining more traction”.
So what is the problem?
Based on the data from Pew Research Center and Gallup, it would seem that public trust in media has been in rapid decline since the mid-2000s. As a result, people increasingly rely on social media websites like Facebook and Twitter for news.
Is it actually a problem?
Yes. Alternative sources of information like social media are quickly becoming the predominant medium by which information is shared. Platforms like Facebook and Twitter lack accountability and journalistic standards, which means the dissemination of false or faulty information is becoming increasingly likely. This has a direct impact on the public’s perception of the truth.
Validating the Market: Landing Page Test
I needed to put the product in front of real people to determine whether or not they would be truly interested in using it. To accomplish this, I pulled a page out of Laura Klein’s book UX for Lean Startups by creating a real landing page to sell a fake product.
Designing a Fake Product
I put together several screens of a fake product, based on Musk’s tweeted specifications:
Rate the core truth — I interpreted this as communitarian fact-checking.
Credibility score — I interpreted this as a rating system for each news story, which impacts each author’s overall score.
Creating a Landing Page
I put together a decent looking landing page for the Trüf News App, which contains copy supporting the idea that this app is for people interested in reading truthful, fact-checked news from an app that allows them to rate each news story.
When users clicked one of the download buttons, a modal opened explaining that the app has not launched yet and that they could sign up via email to be notified when it is launched.
I set up event tracking in Google Analytics to determine how many visitors clicked “download”, which I consider to be an active desire to download the product. Each unique “download” click counts as a conversion.
It’s possible that some users might click “download” in order to be taken to the App Store to read more about the app before deciding to download it.
Sending Relevant Traffic
I created a basic pay-per-click campaign in Google Ads, targeting keywords like “unbiased newspaper”, “fake news stories”, “true world news”, “true news stories”, and “unbiased news sites”.
I set my ad budget to $100 and let the campaign run for one week.
The Results
The ad campaign ran for six days. By the end of it, I had garnered 90 total clicks from 2,221 total impressions, which puts the click-through rate of these ads at around 4% — well above average.
Of those 90 unique visitors to the site, 20 unique “download” clicks occurred. This puts the conversion rate of these visitors at 22%. This is surprisingly high.
Three users even signed up to receive email updates when the app is launched.
The sample size of this landing page experiment was very small. If this were more than a theoretical project, I would:
Increase the ad spend in order to generate a much larger sample size
Perform A/B testing of various concepts and messaging in order to gain a clearer picture of where user interests lie
Encourage more email signups in order to garner a much larger base of potential users for interviewing and prototype testing
Is there a market or audience that wants or requires a solution to this problem?
Yes. Even if the sample size for this test was very small, a 22% conversion rate is very promising. 1 out of every 5 people searching for the keywords specified in the ad campaign attempted to download the app or was interested in learning more about it by visiting its page in the App Store.
Next Steps: Validating the Solution
Elon Musk is one of the most successful tech entrepreneurs of our time, so it’s no surprise that he’s got the pulse on a really provocative and disruptive product. But this doesn’t necessarily mean the solution he’s putting forward will work.
The problem and the market are valid, and this research helps zero in on what that solution should be, but this is where the real design work begins.
A pathway forward with this project should include lots of user research and interviews, conversations with experts, a more clearly defined hypothesis, and lots of prototyping and testing.