Closed Bug 832547 Opened 11 years ago Closed 11 years ago

FHR user-facing page design

Categories

(Firefox Health Report Graveyard :: Client: Desktop, defect, P1)

defect

Tracking

(firefox21+ fixed)

VERIFIED FIXED
Tracking Status
firefox21 + fixed

People

(Reporter: lco, Assigned: laura)

References

()

Details

Attachments

(3 files, 11 obsolete files)

Let's start discussing the contents of the FHR user-facing page here. 

See http://people.mozilla.com/~lco/FHR/130118%20FHR%20Concepts.pdf for some strawman wireframes and general explanation of who I think our users and features are.

My goal here is to get us to agree first on the broad needs we have to address, then talk specifically about what features and data we can use to address those needs.

The kind of feedback I'm looking for:

1. What do you think about the v1 user needs I've identified on pg. 1 (and hence, created broad feature groups for on pg. 2)?
2. Which v1 user needs have I overlooked?


The kind of feedback I am NOT looking for at this time (which we can talk about in our 1:1's):
1. What specific data we should/can show
2. What specific graphs we should/can show
3. What specific yardstick (e.g. should we show a health-o-meter, images of healthy vs sick firefoxes, have a rating system) we should be using to convey the health of the user's machine
Attached image Healthy Firefox (obsolete) —
Attaching some screenshots to this bug for quick reference, but please see the document for a more in-depth explanation
Attached image Sick Firefox (obsolete) —
Blocks beta?
Blocks: 829887
OS: Mac OS X → All
Hardware: x86 → All
Discussed the current wireframes with the Metrics team today. I'll be doing another revision to the wireframes based on some tweaks to the feature set. See https://etherpad.mozilla.org/About-FHR for details.

Please still add your comments to what I currently have though!

Proposed Changes for v1:
* Add links to Raw Data and Glossary (useful for those who want to audit the system)
* Defer comparison graphs till v.next (because we don't have enough data yet), and use a herringbone graph about the user's daily speed and crash data instead. The Metrics team will create this graph.
* Add/Remove certain fields from the Vital Stats area (see https://etherpad.mozilla.org/About-FHR)

Things which need more thinking but that I still want to push for:
* A summary of the user's performance (e.g. average load time, % of crashes) -- we need to figure out what parameters would make this data meaningful to the user.
OS: All → Mac OS X
Hardware: All → x86
Update to the wireframe concepts: http://people.mozilla.com/~lco/FHR/130125%20FHR%20Concepts.pdf

Still needs a little work:
* The Notifications Pane contents -- need to add back the button for allowing the user to update their Firefox
* Need to add more detail to the graph, and discuss what the best graph to show is

Next steps:
* I'll work on the interactions above, but if people are generally happy with the features and layout, I'll discuss with Visual Design about creating high-fidelity mockups.
Attached image Dashboard (obsolete) —
For the full set of wireframes, please check: http://people.mozilla.com/~lco/FHR/130125%20FHR%20Concepts.pdf
Attachment #704128 - Attachment is obsolete: true
Attachment #704129 - Attachment is obsolete: true
Attached image Cover Page (obsolete) —
Attached image Data Page (obsolete) —
I realized that I should point out why I changed some things from the previous design:

1.  Removed the graphic indicator of Firefox's health (e.g. "Firefox is Healthy!", "Firefox is Sick!")

I think it's still important to have some kind of overview indicator of how the browser's doing, but after talking with the metrics team, I realized that having such a composite measure is difficult to create without having comparison data first. Also, Alex pointed out that a benchmark for health has a lot of nuance: for example, if the user hasn't updated his browser because he's on an old OS which we don't update, he'll always see a "sick firefox" image, which would be disheartening.

Instead, I'd still like to get a couple of bigger picture hints to help users, such as: # of crashes this month, average start-up time this month, or others that you can think of.


2. Updated the graph type

In the wireframes, there are a couple of different mocks of graphs. Initially, the metrics team proposed a herringbone-style chart with start-up time/day, # of crashes/day, and any events that might have occurred (such as updates, enabling/disabling addons etc.) I think these are useful data for the user, but I'd like to propose a simpler graph (in my main dashboard wireframe).

Basically, I eliminated the # of crashes/day component because I think this will look rather empty for many people. Instead, I rolled it into the "events" section of the graph, distinguishing between neutral events (updates, addons, plugins) and crashes. I think that any time the browser crashes, it's a significant event for the user, regardless of the # of times it crashed per day.


3. Changed the "Share with Mozilla" ON/OFF switch to a Tab

In the previous wireframe, the user didn't see much of a difference in the dashboard when he flips the switch. This is because turning sharing on or off stops Mozilla from receiving the data, but doesn't affect any of the graphs that the user sees.

By changing it to a tab, the user is less likely to assume that there will be a direct impact to the page itself. Also, On/Off is less precise than "Enable" or "Disable" in describing the data sharing. It also gives us the opportunity, in the form of a popover (or dialog, if we wanted), to explain to the user what enabling/disabling does.


4. Where to put SUMO, About: Support, and Clear Profile?

I'm not yet sure I like my solution to this, but what I decided was to show links to support only when there seems to be something wrong (e.g. in the Tips Panel, or in the Events Log). I was worried that if I made these links visible always, I may send the user on a wild goose chase. However, I'm not sure if users might like having a reliable link to these pages.
Here is some general feedback on v.3 of the wireframes.

1. Regarding "Sick Firefox" -- Maybe we could have a neutral icon to display most of them time and come up with some sort of indicator that we have a suggestion for a possible improvement to stability or performance?  This plays well into the Tips section of your mockup.  It might help relieve some of the problems with dead end-updates showing a sick icon and such but still allow us to let the user know when there are potential improvements.

2. Regarding rolling up events with crashes -- This might be useful, but I really think we need to see how frequent crashes are in the data sample to get a handle on it.  This is on the Metrics team to put together some stats and visualizations based on real data.

3. Data Page -- An important note about the Health Report is that there is no concept of a history of sent reports.  The browser always has two things:  A local copy of the last report submitted and its report identifier; The current data that has been collected since the last submission but not yet sent in.  Submitted reports are always cumulative.  The first report sent will contain Day 1.  The second report sent will contain Day 1 and Day 2 (up to 180 days).  When a follow-up report is sent in, the id of the previous report is also sent with an instruction to the server to delete the old document.  That way, Mozilla only ever has one copy of a health report for a given installation, and it matches up with the last submitted report on the client.  If the user opts out, then the client sends a command to the server to delete the last submitted document.  For the Data Page, this means that the only thing that can easily be shown is the last submitted report and the glossary.

4. When sharing is disabled, the tool-tip could also mention to the user that any previously submitted data has been (or will be if they disable it while off-line) deleted.  The client knows when it has managed to successfully send the delete command.
(In reply to Daniel Einspanjer :dre [:deinspanjer] from comment #10)
> 2. Regarding rolling up events with crashes -- This might be useful, but I
> really think we need to see how frequent crashes are in the data sample to
> get a handle on it.  This is on the Metrics team to put together some stats
> and visualizations based on real data.

As someone who works with crashes and crash data daily, I can tell you right away that "below 2 crashes per 100 ADI" is roughly the current measure of we call "healthy" for a Firefox release or channel. We only have summary data in that form right now and not per-installation metrics, but I think we can model things roughly based on that experience until we have actual data from FHR that we can base better metrics on.
So, if we see 2 crashes per 100 ADI as a rather typical value (which it is), that means .02 crashes per active installation per day, so also 2 crashes per 100 days in which an installation is active. So, people who are using Firefox every day should get less than one crash per month (at least in terms of what they report to us), for the whole 180-day window, that would be 3-4 crashes on average.
My Feedback: 

Cover Page

General: A big +1 for having this page. It will really help onboard users to this new feature. 

1. Is there a reason we include "Firefox Health Report" twice in the same small screen? Consider listing it once to make it more reader friendly. Perhaps change "Getting Started with Firefox Health Report" to simply "Getting Started"

2. To save words why not change "Go to my report" to "View My Report"

3. For final copy on both screens please continue to work with Product Marketing - I also suggest we bring Matej in for his input and to make it sparkle/add the right Firefoxy tone. 

4. When is final copy needed? When will the first round of the graphic design be ready for review? What chart will we be showing on this panel? Let's make this look beautiful. 

Dashboard View

1. These numbers don't add much value to the user. The "Tips" add value. 

What will the User Experience be for those users that don't trigger a Tip? Can we add a "Tip" for them that says something valuable such as "Good news, your Firefox is running smoothly. Come on back if you run into problems!" This will hopefully be the majority of users who see the Health Report. 

2. Same comment as above on the descriptive copy. When is that needed by?

3. In general I'd prefer to not "air dirty laundry" if it's not needed. For example, what about removing the crash count, and only showing a tip on how to fix your version of Firefox to those users who do have a crash *problem*? Why show what could be a rather alarming number if it's still within reason?

4. Users care about "Speed" - they don't necessarily think about "Startup Time." I understand that "Speed" is complicated but it seems like we're featuring Startup Time because we happen to be able to show that data. IF there's a correlation between longer startup time to slower browser in general, let's make that crystal clear. Let's include copy like "Longer Startup times correlate to slower browser performance in general. See how your customizations affect XYZ..." (something like that) 

5. What exactly is the logic behind when each of the three tips will show? If they're one version behind? Two versions? How many crashes within the last few days trigger that Tip?

6. Where within SUMO do the Tips link? In general the more exact, the more helpful they will be. 

7. This feels like a confusing place to ask for even more data from the user. Aren't they already sharing data with Mozilla? They're looking at a report of it right that moment. I think that I just don't understand the UX here, why someone who isn't sharing data would use FHR. And if they're not sharing, what will they actually see within this FHR? Is there a mockup for that user segment?
Attached image Dashboard 130214 (obsolete) —
http://cl.ly/2q3V1n1c0k0n 

Here are updated wireframes with suggested text strings. Mostly, I cleaned up my work based on previous feedback and added more concrete details. 

I forgot to add a couple of suggestions from Laura (such as some way to tell users that their machine is running smoothly, rather than just negative messages all the time). I'll continue to update that part of the document.

(I need to move the link to my people file directory, but haven't had the time to do so yet)
Attachment #706637 - Attachment is obsolete: true
Attached image Cover Page 130214 (obsolete) —
Attachment #706638 - Attachment is obsolete: true
Attached image Data Page 130214 (obsolete) —
Attachment #706639 - Attachment is obsolete: true
Depends on: 844170
Attached image FHR Visual Design - i02
First visual design iteration. I also built an HTML mockup here to test some of the interactions and the layout: http://people.mozilla.com/~shorlander/files/firefox-health-report/firefox-health-report-01.html

Some things left to do:
- Cover page graphic
- Graph is placeholder since I don't know what this is actually going to look like
- Raw Data view needs a sidebar navigation treatment (not sure what the content is going to look like here either)
It certainly looks like things like the "Update Now" button may require postMessage listeners in chrome and thus changes to the about: page to be uplifted to Aurora. Or, is there a way to do everything in the mockup from content?
(In reply to Gregory Szorc [:gps] from comment #17)
> It certainly looks like things like the "Update Now" button may require
> postMessage listeners in chrome and thus changes to the about: page to be
> uplifted to Aurora. Or, is there a way to do everything in the mockup from
> content?

I am not sure what our capabilities are from about: pages. I pulled these Tips from the requested default set of Tips in Larissa's latest wireframes in comment 13.
(In reply to Stephen Horlander from comment #18)
> (In reply to Gregory Szorc [:gps] from comment #17)
> > It certainly looks like things like the "Update Now" button may require
> > postMessage listeners in chrome and thus changes to the about: page to be
> > uplifted to Aurora. Or, is there a way to do everything in the mockup from
> > content?
> 
> I am not sure what our capabilities are from about: pages. I pulled these
> Tips from the requested default set of Tips in Larissa's latest wireframes
> in comment 13.

No worries. My original question was mostly directed at the engineers. I'm just trying to figure out what we're looking at and whether we need to push back on the design to meet deadlines :)
I saw the mockup here : http://people.mozilla.com/~shorlander/files/firefox-health-report/firefox-health-report-01.html

Looks great. And i know it's a placeholder, but please no no histograms for measurements. Also,  dzeber has created prelim mockups for the startup and will speak to Larissa next week.
> Looks great. And i know it's a placeholder, but please no no histograms for
> measurements. Also,  dzeber has created prelim mockups for the startup and
> will speak to Larissa next week.

Okay, now that i dont over react, i see it's not a histogram, but a bar chart. Sorry.
(In reply to Gregory Szorc [:gps] from comment #19)
> (In reply to Stephen Horlander from comment #18)
> > (In reply to Gregory Szorc [:gps] from comment #17)
> > > It certainly looks like things like the "Update Now" button may require
> > > postMessage listeners in chrome and thus changes to the about: page to be
> > > uplifted to Aurora. Or, is there a way to do everything in the mockup from
> > > content?
> > 
> > I am not sure what our capabilities are from about: pages. I pulled these
> > Tips from the requested default set of Tips in Larissa's latest wireframes
> > in comment 13.
> 
> No worries. My original question was mostly directed at the engineers. I'm
> just trying to figure out what we're looking at and whether we need to push
> back on the design to meet deadlines :)

I'm open to removing the "Update Now" button on v1, but let's talk about what we can or can't do easily. I still think if there's a place that makes updating easy for the user, it would have a big impact on their browser's performance.
Assets for the widget. Not sure how this will be implemented. I can create different resources if needed.

Styles from the mockups: http://www.pastebin.mozilla.org/2196666

Close-up of widget: http://cl.ly/image/392n2F3W3w1Q
Attached file FHR Wireframes v5 (obsolete) —
Updated the wireframes with Matej's copy. See bug 844170 for more information on what's still in progress.
Attachment #715671 - Attachment is obsolete: true
Attachment #715672 - Attachment is obsolete: true
Attachment #715673 - Attachment is obsolete: true
Not sure if this is the right place to comment from a l10n point of view, but when extracting the strings, based on a quick look to the wireframe:
* tooltips on graphs should be really flexible, considering that some languages have really long localizations for the term "add-on"
* enabled/disabled: "add-ons" and "plug-ins" could have different genres
I am working on getting the repo and server set up that will deliver the static assets to the client.

From what I see in shorlander's mocks:
* The "Tips" will not be in V1 from what daniel tells me, so will remove those
* The HTML we send will include the following:
** Headers
** Sidebar with no data
** Graph placeholder with no data
The client will need to fill in that data.  I don't see how we can do more than that. 
* The pref toggle in the top right hand corner: how is that supposed to work?  Do we send plain HTML and just not worry about what the client does with it? 
* Is someone going to provide JS to deal with the DOM manipulation, or does that all happen in the client?

I'm just having a little trouble understanding the detail of what comes from where and how it hooks together.  I understand the big picture, but I still don't know things like what particular page elements should be named so that the client can inject data appropriately and so on.
(In reply to Laura Thomson :laura from comment #27)
> I am working on getting the repo and server set up that will deliver the
> static assets to the client.
> 
> From what I see in shorlander's mocks:
> * The "Tips" will not be in V1 from what daniel tells me, so will remove
> those

I'm not sure why that's true.  We should be able to do some of them, i.e. "you're out of data by N versions, click here to download a new copy"

> * The HTML we send will include the following:
> ** Headers
> ** Sidebar with no data
> ** Graph placeholder with no data
> The client will need to fill in that data.  I don't see how we can do more
> than that.

We'll provide the data in JSON format (in the same way as we submit to Bagheera: http://docs.services.mozilla.com/healthreport/index.html#payload-format ) , it was expected that the web piece would use that data to generate any graphs/visualizations/recommendations.  I thought that was generally clear, but it's been a while so perhaps that expectation is lost in the mix.

> * The pref toggle in the top right hand corner: how is that supposed to
> work?  Do we send plain HTML and just not worry about what the client does
> with it?

Everything in the header will be client-side for now, it's below that which will be remote.

> * Is someone going to provide JS to deal with the DOM manipulation, or does
> that all happen in the client?

Which DOM manipulation do you mean?
Note that we could, in theory, do 100% remote UI and just use a defined postMessage API to handle toggles/data requests.  We would need to make that call basically today if so.
> > From what I see in shorlander's mocks:
> > * The "Tips" will not be in V1 from what daniel tells me, so will remove
> > those
> 
> I'm not sure why that's true.  We should be able to do some of them, i.e.
> "you're out of data by N versions, click here to download a new copy"
> 

The tips UI was designed to be flexible enough to accommodate different contents. It's very important to have this kind of component to help the user interpret the graphs and other contents of FHR. We can discuss which "tips" we will launch with v1, but I think it's important to have at least a few to support this page. I agree with Mike for example, that we could have a tip to display a message about an outdated browser.

Additionally, Shorlander and I have been discussing ways to make the tips a little more vibrant. For example: http://cl.ly/image/0D3E0w1b2c2w . However, depending on timing, I think we are ok with starting with the visuals that Shorlander has already created.
(In reply to Mike Connor [:mconnor] from comment #29)
> Note that we could, in theory, do 100% remote UI and just use a defined
> postMessage API to handle toggles/data requests.  We would need to make that
> call basically today if so.

That is what I recall the plan was!



(In reply to Mike Connor [:mconnor] from comment #28)

> it was expected that the web piece would use that data to generate any
> graphs/visualizations/recommendations.  I thought that was generally clear,
> but it's been a while so perhaps that expectation is lost in the mix.

Just to reiterate this point: one of the goals I remember is to allow Metrics to ship new visualizations, advice, etc. etc. without waiting three months for a new Firefox version to reach the market.

Having the client generate anything apart from raw data adds some really tight coupling to particular shipping Firefox versions, which will make that much more difficult.

So my understanding of this was (with apologies for not following too closely in the past couple of months; this might have drifted without me noticing!):

* about:healthreport is an empty skeleton that grabs data from HealthReporter and uses postmessage calls to:
* foo.html, a chunk of HTML served by *.mozilla.com that does *absolutely everything else*, including analysis of the payload, generation of graphs, production of recommendations, etc.

The former will rarely change. The latter can change all the time, as new logic is added to make recommendations based on correlations.


As a sidenote: anything we do in chrome JS won't be jitted, and will be slow. Anything we do in content will be faster. Yeah, I know.
I've only be following this from a high-level, so forgive me if I glanced over details.

I've been wondering what thought has been given to the long-term vision of FHR-derived (and Telemetry too, I suppose) user feedback and product improvement. From what I've seen in this bug, it appears most (all?) of our user feedback comes via about:healthreport. This seems like it would be a reactive design: the user would have to explicit visit about:healthreport in order for recommended improvements to be noticed and/or acted upon. If the user never visits about:healthreport, their Firefox will likely remain slow.

I've heard Asa say he wants Firefox to be more proactive about fixing problems. This implies more automatic background fixes and/or user notification/consent to try things (e.g. disabling a known slow add-on). How will this be done? Will we funnel people to about:healthreport somehow? Should we measure if people actually visit about:healthreport so we know if it is working?

I worry about solving this problem from a technical perspective because unless we have a long-term vision, I foresee us implementing a number of one-offs or implementing things at the application layer (as opposed to the Gecko platform layer) and thus digging ourselves into engineering debt when it comes time to roll out these improvements to different apps (e.g. Metro, Android, Thunderbird) or when we start considering the interactions between all of the notifications. Bug 836010 is a perfect example of this. Great feature - but it is only implemented for the desktop browser - not Metro, Android, Thunderbird, etc. Furthermore, it is using the notification bar. Do we want N independent feature recommendations competing for notification bar real estate or do we want a unified mechanism?

Perhaps this discussion is not appropriate for this bug. If not, perhaps we should talk about it at the next FHR status meeting?
Recommendations for the features of the graph (as discussed at yesterday's meeting with Larissa and shorlander) are collected on this etherpad: https://metrics.etherpad.mozilla.org/98.
The current scope of the first release doesn't make any particular provisions about proactive notification, but we have discussed this.  Once we have a really good reactive design inside about:healthreport that can guide the users to improvements, we can work on ways within the client to funnel them there.  Things that were discussed were:

* Start page notifications that provide a link to the health report page
* Internal client self checks that direct the user to the health report page
* Responses from the document submission that would instruct the client to direct the user to the health report page
We are way out of scope for this bug.  Please move it to email or IRC if we want to talk about long-term evolution.  That said, comment 34 matches my view of the world, but we're early in the process.
So back on scope, re:
"foo.html, a chunk of HTML served by *.mozilla.com that does *absolutely everything else*, including analysis of the payload, generation of graphs, production of recommendations, etc"

Up until now that was not my understanding, but it does make a lot more sense, so good.

Things we still need:
- if tips/recommendations are to be in V1, need logic, visuals, and strings for those.   The latter two are part of this bug.  What's the best way to figure out what our recommendation rules look like?  Another bug? An etherpad? Something else?  
- the incoming data will come, mconnor tells me, from a postMessage.  Is the data we will have identical to http://docs.services.mozilla.com/healthreport/index.html#payload-format?  (I'm not sure everything needed for the visualization is available in that payload, but I'll check that next, and get back to you.)
(In reply to Laura Thomson :laura from comment #36)
> Is the data we will have identical to
> http://docs.services.mozilla.com/healthreport/index.html#payload-format? 
No, this doc is out of date.
Depends on: 849868
OS: Mac OS X → All
Hardware: x86 → All
Blocks: 849879
(In reply to Laura Thomson :laura from comment #36)
> - if tips/recommendations are to be in V1, need logic, visuals, and strings
> for those.   The latter two are part of this bug.  What's the best way to
> figure out what our recommendation rules look like?  Another bug? An
> etherpad? Something else?  

I just created bug 849879 to discuss the contents of the Tips Pane. There's a link to the working etherpad https://etherpad.mozilla.org/FHR-Tips (which doesn't currently contain anything, but that I'll start filling out)
This is now in a repo at https://github.com/mozilla/fhr-jelly
Looks like Laura is working on this, so assigning.
Assignee: nobody → laura
Priority: -- → P1
Attached file FHR wireframes v6 (obsolete) —
Updated the wireframes with the following changes:
Daily Startup Time Graph 
• Updated with graphs from the Metrics team
• Replaced the strings which explain the graph (still needs copy review)
• Updated the Graph Information Pane -- the brief explanation of the graph now lives below it rather than in a separate info box.
• Add details about the different graph views

Tips
• Added tip related to not having enough data yet (still needs copy review)
Attachment #721950 - Attachment is obsolete: true
Here are the links to the contents of the Raw Data Pages of the Health Report:

Raw Data: Go to about:healthreport in Nightly, and click the button that says "Show Details" to see what this looks like.

Glossary: https://services.mozilla.com/healthreport/glossary.html
Attached file FHR v1 MVP r1 (obsolete) —
Uploaded a set of high-level wireframes that show the MVP screens for v1.
Attached file r7 FHR MVP-v1
Attached the design spec for v1 of FHR. All strings and features finalized.

(Sorry for the confusing document naming scheme. I'll try to be more consistent from now on)
Attachment #724548 - Attachment is obsolete: true
Attachment #725482 - Attachment is obsolete: true
Blocks: 853120
Component: Metrics and Firefox Health Report → Client: Desktop
Product: Mozilla Services → Firefox Health Report
So v1 as documented here is implemented, and this bug can be closed.  Open new bugs for v2 features.
Status: NEW → RESOLVED
Closed: 11 years ago
Resolution: --- → FIXED
Verified - v1 successfully shipped.
Status: RESOLVED → VERIFIED
Product: Firefox Health Report → Firefox Health Report Graveyard
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: