Changes in Web Usability since 1994

I have been running Web usability studies since 1994 and the most striking conclusion from looking back is that most findings about Web usability are the same now as they were in 1994. This may be surprising, but usability is about basic human capabilities and users' needs which do not change nearly as rapidly as technology.

The dominant Web look in 1994 was a mixture of gray text and large images, as shown to the right (screens from a comparative study of sites in 1994). This has certainly changed: We have better layout capabilities and many designers have learned to minimize their use of graphics and save download time (though still not enough). Despite the change in looks, many of the findings from 1994 continue to hold:

There are also a few cases where early findings have to be modified, as described in the following.

Scrolling Now Allowed

In early studies, It was found that only 10% of Web users would scroll a navigation page to see any links that were not visible in the initial display. The vast majority of users would make their selection from those links they could see without scrolling. In retrospect, we believe this was due to people treating a set of Web options like they would treat a dialog box: You always design dialog boxes so that all choices are visible (except for tabbed dialogs which are known to have severe usability problems; and the tabs do indicate the amount and nature of the hidden options).

In more recent studies, we have seen that most users have started scrolling when they visit a long home page or a long navigation screen. This change in behavior is probably due to users getting more experience with scrolling Web pages.

There are still a few users who rarely scroll. Those users who are willing to scroll may be tempted to chose one of the initially visible options when it seems to match their goals. Such users will never see an even better, but invisible, choice that would have required scrolling. Therefore, I still recommend trying to design navigation pages to make all major choices visible without scrolling on the monitors used by the average visitor to a site. Also, the likelihood of making the best choice from a navigation page is maximized if the user can see and compare all the options at the same time without having to scroll and remember the hidden choices.

The change from 1994 is that scrolling is no longer a usability disaster for navigation pages. Scrolling still reduces usability, but all design involves trade-offs, and the argument against scrolling is no longer as strong as it used to be. Thus, pages that can be markedly improved with a scrolling design may be made as long as necessary, though it should be a rare exception to go beyond three screenfulls on an average monitor.

Imagemaps Less of a Problem

Imagemaps caused endless usability problems in 1994, with users overlooking clickable areas and expressing frustration that they didn't know what they could do on the page. Imagemaps have caused very few problems in recent studies, for several reasons:

Comprehensiveness Needed

Users have always been intolerant of downtime and crashes. As one of my very first Web test users said, "if a site doesn't work, I may give it a second chance; if it still doesn't work, I am never going back." Users also felt that "under construction" signs were disrespectful of their time.

Even though users wanted sites to work, they were willing to accept sites that had very limited functionality. In the early years, the Web was more of an experimental environment and users understood that they would be limited to a sample of a company's services. Now, users expect comprehensive service from sites. Not only do sites need to be up and available at all times, they also need to have all the information and services users want in a certain category. For example, in 1994 a publishing company could get away with featuring a few recent books on its home page and not providing access to its full backlist. Now, if a company doesn't have all its products online, users will complain.

The Web is no longer an experiment: it is mainstream. You have to rely on the ability to do business on the Web, and people get very annoyed when they can't: Limited sites are seen as a sign of corporate incompetence.

Additional reading: Perry Hewitt (editor of www.lotus.com) describes changes since 1994 in a company's editorial approach to the Web

Report From a Web Usability Study

Introduction

Users were observed as they browsed the Web sites of Hewlett-Packard, IBM, Microsoft, Sun Microsystems, and Time Warner. The report has only been very lightly edited and thus represents my thinking about Web usability. In fact, the report was originally written for distribution to the rest of the Web team on paper since we were not heavy intranet users, despite having designed SunWeb a few months before this study.

The report is of some historical interest, both because it includes screen captures of several famous early websites and because it is one of the first formal usability studies of the Web. In fact, it is remarkable how well the findings and conclusions in this report hold up in the light of the current state of the Web. People often talk about how the Web changes on "Internet time", but usability issues seem to change much more slowly since they stem from human capabilities and interests.

Method

Three external participants were tested: an MIS director, a programmer, and a systems administrator. All were employed with technically oriented companies in the Silicon Valley area, all had extensive Unix experience, and all were highly technically competent. One participant was a current Sun user, one used another version of Unix on the Intel platform, and one was currently a Windows user but had been a Sun user for seven years. All participants had used the WWW extensively before the test. Thus, the participants were very advanced users and one would expect that less sophisticated users would have many more problems in using the WWW than described here. In other words, this test investigated a best-case situation.

Each participant was tested for 60-90 minutes. During the test, the participants visited 2-4 WWW sites from a list of sites prepared by the experimenter. For each site, the participants were first asked to give their initial impression of the home page, after which they were allowed to explore the site freely. After about ten minutes of exploratory browsing, the participants were given a directed task that asked them to find some specific information that was available on the site. The sites and the directed tasks were:

Site	Directed Task
Hewlett-Packard	You want to see the photo of the garage in which the company was founded
IBM	You want to know how many hard drives you can put into a PC Server 50
Microsoft	You are interested in knowing how Windows'95 will support access to the Internet
Sun Microsystems	You are interested in knowing more about the Spring Distributed Operating System Project (a project trying out alternatives to Unix)
Time-Warner	You want to read the profile of the co-creator of Mosaic. His first name is Marc and his last name starts with an A, but you can't remember the exact spelling

The users were tested using NCSA Mosaic version 2.4 for X Windows running on a SPARC 10 workstation with a megapixel 256-color display. The workstation had an indirect connection to the Internet through a firewall with transfer rates of approximately 10 kilobytes per second. This is about five times faster than a modem and about the same as ISDN and thus represents best-case performance for home users, though on the slow side of what one would experience from an office computer that was directly connected to the Internet. When using the Sun WWW pages, the users accessed an internal mirror server to which the workstation had a direct network connection, and transfer rates were much faster for the Sun pages (around 65 kB/s) than for the pages retrieved from the Internet through the firewall. The users commented several times that access to the Sun pages was faster than their normal experience and that access to the non-Sun pages seemed a little slow. Thus, we can assume that the network speeds used in this study were approximately (though not exactly) the same as those normally experienced by the users.

The Sites

We obviously wanted to test Sun's own existing web pages since the previous release of a product is the first prototype of its own replacement. Four additional sites were selected for the study based on discussions in the www.sun.com redesign project. Project participants were asked to recommend good web sites that had interesting design with an emphasis on sites with a magazine or news-oriented approach to WWW information. We believed that frequent information updates are an emerging trend and would be important for attracting repeat traffic to our WWW site. IBM was selected because it used a magazine format for its home page with a monthly cover story. Time Warner was selected because of its highly news-oriented design. HP was chosen because of its clean and highly structured design, and Microsoft was chosen due to its status in the computer industry.

Many other sites were considered and were rejected either because they were similar to the ones chosen or because we were sure that we did not want to emulate their designs. For example, the home page for HotWired used icons and labels that were essentially impossible to understand. Such a design may be appropriate if "hipness" is the essential quality to be. One possibility would be to follow the example set by eWorld and design the corporate Web page to mirror the graphical user interface of our desktop software, but we were not aware of the eWorld Web site when the comparative usability test was planned, so this option was not tested, except for the observation that users liked a similar design feature at Microsoft's site.

Page Complexity

On the Sun home page, all users clicked on the Santa Claus icon first due to its prominent position at the top left corner of the page. Also, several users expressed a liking for the untraditional icon design with the button graphically protruding from the image. All users overlooked the row of flat buttons on top of the Santa button. Several users were actively looking for a "new at Sun" button and a search feature but did not find them. The design used for the Sun home page makes the button bar look too much like captions or headers for the top row of much more prominent icons.

All users complained when they retrieved pages with a screen or more of unstructured text. They plainly did not want to read much, and at best scanned the text for important or relevant paragraphs. The users liked information that was presented in lists that were easily scannable, especially when icons were used to indicate the different parts of the list (see the figure from Hewlett-Packard's site). Also, users liked the use of a "New" indicator to highlight new information in a list and the use of horizontal rules (<HR>) to partition a page. Long pages (more than a screen) were only deemed acceptable when users could quickly decide to ignore most of them and focus on the relevant parts (e.g., a list of Microsoft networking products organized by operating system).

In one case, a user retrieved a page that was very similar to one he had seen before and wondered whether the two pages were in fact identical. This incident indicates a potential problem with representing the same information multiple times with slight variations.

Observations regarding search in this study confirmed the results from more quantitative studies conducted as part of the AnswerBook redesign project: users typed very short search strings (normally one or two words) and sometimes overlooked the item they were trying to find even when it was included on a list of search results. The main conclusion from these observations is that one cannot rely on search as a mechanism for leading users to the information they need.

Users also sometimes scoped their search incorrectly: in one case, a user was looking for information on the Sun server and came across the Catalyst project page, which included a search form for finding information in these pages. The user happily used the Catalyst search in the belief that it searched the entire Sun server and not just the Catalyst database. The conclusion from this observation is not to offer search features that are limited to a subset of the information on a sever. If scoped search is needed anyway for some reason, it should be made very clear to the users what information is being searched and what it not being searched and there should be a direct hyperlink to the global search page. Also, users sometimes identified the need for search after they had traversed several links and were deep in the hierarchy. They expressed a desire to be able to start a search from their current location and not have to navigate back to the home page first.

Navigation and Metaphors

During this study, users were mostly very good at knowing what site they were currently visiting, so they did not seem to get lost in hyperspace at the macro level of knowing what company's information they were browsing. One user complained about the inclusion of information about third party products on Sun's server. He felt that he was browsing Sun's web to get information about Sun's products and that he would access those other companies' sites if he wanted information about them. A possible solution to this problem would be to make third-party pages more visibly different from Sun's own pages or to eliminate the information itself from Sun's server and only provide links to the other companies servers.

On the Time Warner page, one user was surprised when the "Virtual Garden" button pointed to information for home gardeners. The user had expected it to be a list of links to other interesting Web sites to visit. Users may have become so accustomed to the heavy use of metaphors in many WWW designs (and systems like Bob and Magic Link) that they expect design elements to be metaphorical rather than literal. The lesson for user interface design is to consider not just the first-level interpretation of proposed design elements but also whether they could be misconstrued as inappropriate metaphors.

In a few cases, users retrieved information that caused a PostScript viewer to be launched without warning. The users complained that the WWW page had not warned them that the link would download a PostScript document rather than jumping to another hypertext screen and they also did not want to read the PostScript document (as mentioned above, the users disliked long texts). Even though the users disliked links to PostScript files that were displayed in a separate viewer, they did like the possibility of using FTP to retrieve additional information, executables, patches, or other non-hypertext files.

Overview Diagrams

Even though users understood what sites they were visiting, they often got lost within sites, and they several times expressed a wish for overview diagrams. Hewlett-Packard's design was particularly good at informing users about their current location in the information space.

Users consistently praised screens that provided overviews of large information spaces. For example, they liked the HP navigation screens, Microsoft's home page with its extensive listing of server content, and the overview page for Windows'95. Some users did complain that the text in the Microsoft home page overview was too small.

Several users who were getting lost coped by analyzing the file names with full directory paths given for HTML files to understand the server hierarchy and their current location. Unix experts may be capable of doing so, but general users would need a simpler mechanism for understanding the information space.

Thumbnails

Users several times complained about small thumbnail pictures where too much photographic detail was shown in too little space to be clearly visible. The figure shows a page with several overly complex thumbnails from IBM. Another example from the IBM server, below, shows how a small picture can be perfectly acceptable provided it shows a fairly simple object without much detail. In general, users appreciated when the system was giving them advance notice before they decided to retrieve a large file (e.g., a big image).

Broadly speaking, it seemed that users disliked 1x1 inch thumbnails but liked 1x2 or 2x2 inch thumbnails. To communicate graphical information in a one-square-inch space one should thus use icons and not thumbnails. The general conclusion regarding thumbnails, though, is that they should communicate some information to the user and have a clean and uncluttered appearance. Thus, the exact size of a thumbnail should be a function of the complexity of the original image. If it is impossible to represent an image clearly with a thumbnail then it will be better to use a textual description.

Incomplete Sites

Users distinctly disliked seeing "under construction" markers. As one user put it, "either the information is there or it is not; don't waste my time with information you are not giving me." Users were particularly aggravated when they had linked to a page only to find that it was under construction. One user said "At least give me something for going to the page; don't put it out there if it is not working." Users were less upset when currently inactive hypertext markers had construction signs that could be seen before activating the link.

Users had little patience for server error messages. One user said that he might try retrieving the page once more, knowing that servers sometimes got overloaded, but if he got a second error message from the same server then he would never visit it again: he would assume that it was too unreliable and/or too poorly maintained to be worth his time.

Users were also very disappointed when they came across evidence that a server was not being kept up to date. For example, several sites listed talks or conferences that had already taken place as if they were still upcoming. In related comments, users praised the way some information on the Sun and Microsoft servers was labeled with dates to indicate how current it was. Users also liked the use of the month name to indicate the currency of the information on IBM's home page.

The Human Touch

All users liked Microsoft's picture of their Webmaster in front of the server. They felt that it was nice to know that they were communicating with a service supported by an actual person and not a faceless entity. The users also liked being able to read the technical specifications for Microsoft's server, again from the perspective that the information was not just coming in over the wire, but was coming from somewhere. Of course, server specs may be less interesting to less technically oriented users than the ones tested in this study.

Two-Way Communication

The users liked the ability to request further information about products by filling in a form. One user mentioned that he would often use a Web site to get a general idea about a company's products but that he would then fill in a form asking to get contacted with more information about the specific products he was interested in buying. He was less motivated for having to search the Web site himself to find this detailed information. This user was particularly interested in finding lists of contact telephone numbers and email addresses for the various products.

The users also liked the forms that were provided at several sites to allow them to give feedback to the Webmaster. This observation may also be related to the "human touch" phenomenon mentioned above since having the form makes the users feel more in contact with the actual people who maintain the server as opposed to being just consumers of a flow of information.

Users did not mind online surveys and questionnaires as long as they were at most one page long. Users frequently complained when they came across pages with questionnaire forms that did not fit on the screen. Users expressed a wish to have as much information as possible filled in for them. For example, a user who had already filled in his name and address on one questionnaire wanted the computer to automatically insert that information in another questionnaire he retrieved from the same server a few minutes later. One user wanted to be able to enter his email address to subscribe to notices about updates for a certain page that he found particularly interesting.

Mirror Servers

One problem related to mirror servers was encountered during the test: one user had initially accessed Sun's WWW site through the internal mirror. From there, he had linked to an outside site which happened to have a link back to Sun. The user followed this link, but since the outside link pointed to the outside version of Sun's Web site (the "original" copy of the site), all further cross-reference links to other Sun pages now pointed to the outside copies. This change in addressing caused a usability problem because the Web viewer did not display appropriate breadcrumbs on the link anchors. Normally, Mosaic shows anchors for links that the user has already followed in a different color, but since even those links now pointed to new URLs, they were displayed in the color reserved for unexplored links. This confused the user who remembered having followed some of the links earlier in the session.

This usability problem cannot be solved in the design of any individual server but must await a change to URNs or some other addressing scheme that takes mirror servers into account.

Parallel Sessions

All users opened two Mosaic windows and kept separate sessions going in parallel. Most of the time, these two sessions visited the same Web site, but explored different parts of its information space. The users would typically start a retrieval operation in one window and, while waiting, turn their attention to the other to study the information it contained.

Two conclusions can be drawn from the users' extensive user of parallel sessions: WWW users are very impatient and do not like waiting for information to be retrieved over the Internet. As far as possible, one should avoid having them wait (since their attention will wander).

One cannot assume a one-to-one correspondence between the stream of WWW requests received from a user and that user's hypertext browsing, since two or more browsing sessions may be interleaved.

Integration with Rest of World

One user who had read a page about the speed of PhotoShop on Sun computers, wanted to get a demo of PhotoShop from Sun's WWW server (or have a link to Adobe's server with a demo). Another user mentioned that he normally requested demo CD-ROMs for products before buying them.

One user had read in the newspaper that Sun had information about the Rolling Stones online and searched for that information during the study. Knowing about certain information from paper-based sources is one way that users can get prompted to look for it, and the URL should be given in those printed sources as far as possible. Some users mentioned that they would not want to spend time during the day to read extensive information but that they might print it out or email it to themselves to read later. With current technology, most printouts will be restricted to grayscale graphics and much email will be restricted to ASCII text, so the information should be designed to be useful in those formats.

Users sometimes had problems with flashing colormaps, leading to a recommendation to minimize the number of colors used in bitmaps. Also, several users expressed an appreciation for graphics with a small number of colors (e.g., the HP home page) because they looked clean and felt like they did not waste a lot of bandwidth (and user time) when they were transmitted. It is possible that the move to screens with larger color spaces and the use of faster networks will change this preference for less-colorful images, but for now, it seemed prevalent.

Conclusions

Users had low tolerance for anything that did not work, was too complicated, or that they did not like. With non-WWW user interfaces, the technically oriented users in this study would normally persist for some time in trying to figure out how to use the system, but with the WWW, there are so many sites out there that users have zero patience. Thus, the demands for good usability are probably higher for WWW user interfaces than for normal user interfaces, even though the designers' options are fewer. "Under construction" signs should be avoided and the server should always provide a response within a few seconds.

If the requested information cannot be provided, a meaningful error message should be given instead.

We found that users wanted search and that global search mechanisms should be globally available. Even so, users were poor at specifying search strings and they often overlooked relevant hits. Thus, we cannot rely on search as the main navigation feature. Navigational structure and overviews are necessary to avoid user confusion and should be provided both in the large (server structure and location) and in the small (structure for the individual pages with iconic markers for the various types of information). Users liked the feeling of being part of a two-way communication with a site staffed by real humans and not just the recipients of a stream of bytes coming in over the net. Care should be taken to provide a "high-touch" feeling in addition to the "high-tech" image of a WWW server.