Findings

Overall experience

From the pre-test interviews, we learned the expert users had a neutral attitude toward the previous Cognos, and they expected the new version would be more user-friendly. However, in the post-test interview, the expert users stated they were not happy with the changes. As one tester remarked “I thought it is going to be easier [in the new version], but it wasn’t.” The inclusion of the SUS survey material allows us to compare the evaluated system against the national benchmark of usability. The average SUS score from 500 studies is a 68³. The post-test survey yielded an average score of 33 for Cognos 11, well below the national average and falling into the SUS category of the worst tier. Figure 1 shows how the percentile ranks are associated with SUS scores and equivalent letter grades. The blue line indicates the national usability average of a given tool, and the red line indicates where the Cognos 11 stands compared to the national benchmark.

Finding 1. Information Architecture is unnecessarily complex.

Good Information Architecture (IA) design organizes contents so users can easily find everything they need without significant effort. Unfortunately, the current Cognos constructs its contents and navigation system in a way that is inconsistent with user perception. The testers gave an average of 4.3 out of 5, in which 5 represents “strongly agree,” to the survey item, “I found the system unnecessarily complex.”

In our study, Task 2 (clear Task 1's result and prepare to run another report) examines if the user can successfully navigate across pages. Task 2 captured the usability challenges regarding Cognos 11’s IA design. After running the first report, users had extreme difficulty navigating back to the home page (the report inquiry page). Two users were able to return to the home page by trial and error. However, one user decided to give up after spending three minutes and 18 seconds trying. Overall, the completion rate of Task 2 was 67% (two out of three) and the average time spent on the task was two minutes and 11 seconds (during which 14 errors were made, on average).

While completing the task, the expert users heavily relied on their experience with the last system version, whereas the new user spent more time reading the explanatory text of the buttons before clicking them. The expert users first looked at the right upper corner of the page, where the “return” button used to be in the previous interface version. When they realized the button was no longer available, they looked at the left panel, hovered over the available buttons one-by-one, and read the explanatory text. One user stated she was looking for buttons that say “clear,” “return,” ”back,” or another action suggesting removing the on-screen results or redoing the report. All testers repeatedly clicked on the buttons such as “refresh,” “cancel,” and the home icon even after several previous unsuccessful attempts.

Upon accidentally discovering that he could clear the report by going to the “team content” folder, the new user expressed frustration with such folder organization and commented that the information structure made no sense to him.

A heuristic evaluation of the current interface suggests a severe violation of the visibility of system status. The site did not provide any visible navigational aids—such as menu maps, place markers to indicate where the user is in the system. The lack of navigational aids created a significant barrier when users navigate between pages.

Recommendation

• Reconstruct the site IA. Our analysis reveals an urgent need to regroup site content in a way that is understandable to both expert and new users.
• Provide feedback to let users know where they are in the website by offering navigational options. Feedback provides users with the information they need to understand where they are within the system, and for proceeding to the next activity.

Finding 2. The staffing report does not address user needs directly.

The goal is to use the system is primarily to monitor users’ staffing coverage. To do so, users would either pick “Actual” or “Forecast,” depending on whether they want to see their charged hours in the past or budgeted hours for coming months. They need to specify the date range on “Staff Hours” page (as seen in Figure 2) and submit the request to get a report.

While the experienced users remarked that they have gotten used to this report format (seen in Figure 3) and did not have a problem understanding the report, some improvements could be made to improve the efficient use of the report.

A better report design should take a step further and provide information that directly suits user needs.

Informed by the post-test interviews, to truly make use of the report, there are several additional things users would have to do after getting it. The following describes the common user goals along with the procedures some users following along after receiving the report:

1. Identify the coverage gap
• Users take a note of their % covered for each month, then have 100% minus the percent covered to calculate the % not covered;
• Since not every month has the same number of working hours, users would have to check how many hours the 100% represents in that month;
• Users translate the % not covered into hours and identify how many hours they need to be staffed for the month they are not sufficiently staffed;
• Look for additional projects/tasks if needed.
2. Plan work for the month and track charged hours against their billable time
• Users take note of their allocated project hours for the month;
• They divide the monthly total hours by 4 or 2 to calculate how many hours they can charge to a specific project each or every two weeks (such as one pay period) respectively;
• Users notify the project director if they see a discrepancy between their budgeted hours and workload requirements during the month.
3. Share the staffing report with the staff manager
• The staffing report is commonly used in the staff meeting where staff managers review the report with subordinates. Task 2 involved a component of assessing the ease of report sharing. All users were able to complete the task. However, one user completed the task but remarked that she was unsure if the action was a success. In average, users spent two minutes and six seconds on the task, suggesting a significant usability issue given the simple nature of the task. As it intentionally did not specify what format of the report to share, users were free to choose whichever format was more accessible for them to complete the task. Using the ‘think aloud’ technique, we observed a user preference to share the report in the Microsoft Excel format as it allows the users to work with the report numbers and perform additional calculation if needed. However, none of the users were able to share the report in the Excel format successfully.
4. Keep informed of the coverage change
• In the post-test interviews, users expressed their wishes to have some features added to the report to keep them informed of any changes and stay in control of their time. The features they would like to see included are:
• a. An activity alert if any project directors update their budgeted hours;
• b. A notification if a team member’s estimated hours exceed a preset threshold;
• c. Include a mechanism to prevent project director from budgeting more user’s available hours, so that user’s coverage does not end up being more than 100%.

A good redesign of the report ought to address the user goals by presenting additional statistics about users’ coverage gaps, helping users to plan for future work against budgeted hours, allowing for easy report sharing, and providing tools to keep users informed of their staffing status.

Also, we compared the report interface against Jakob Nielsen's ten general design principles, and two usability issues stood out. One is a severe violation of Nielsen’s sixth principle (recognition rather than recall). The report is generated in a table format where each row represents a project, and each column represents a month. An issue would occur if the report includes many projects (many rows) over several months (many columns), that is the table becomes very large to view and reading requires scrolling.

The row and column headings are not frozen, so when users scroll down or scroll to the right, they have a hard time knowing which project or month the value is referring to (see Figure 4). A good report interface design minimizes the user’s memory load by making the referring months and projects visible so that the user does not need to scroll back and forth to figure out what the values refer to.

Another violation of the design principles is Nielsen’s second (match between system and the real world). The report uses internal terminologies and presents staffing coverage in percentages rather than hours, both of which do not effectively communicate to users using familiar language. During the testing, the new user stated he did not know what “sold %” and “covered %” means, nor did he understand how these terms are distinguished from each other. Therefore, a better report should use plain language and convey the information in a manner that is easily understandable to both expert users and those new to the system.

Recommendation

Redesign the report by incorporating information or features that directly address user needs.

1. The information should help users to answer the following questions:
• How are the user’s hours allocated across projects?
• What is the user’s coverage gap in both percentage and hours?
• What are the outstanding months in which the staff is either under or over necessary levels?
2. The new features should allow users to:
• Know how they should create their monthly work plan based on the budgeted coverage;
• Share their staffing report easily with a staff manager or project directors;
• Keep updated of their coverage changes if there is any.

Finding 3. Several design choices prevent efficient use of the system

The system should enable users to finish a task in about one minute to create a good user experience⁴. Task 1, which asks users to check how they charged to the projects last year, is a considerably simple task. User needs to select “Actual” from the menu to check their charged hours in the past, then specifies the month range and click “Finish” button to generate a staffing report. The average time testers spent on the task is three minutes 12 seconds, comparing to the one-minute benchmark. Also, Task 2 (clearing the report result to reset the search) took users an average of two minutes 11 seconds—too long to produce a positive user experience.

The are several factors that contribute to the slow down.

Issue 1: Non-standard visual representation confused users

We observed some errors made during the usability tests were due to participants’ misinterpretation of icons and buttons. The case is especially true for the new user as he spent time exploring the interface, trying to make sense of the visible buttons and menu before working on the task. The new user hovered over the icons, paused, and thought if those were relevant to the task. If the system had used more standard visual representations, the new user could have glanced over, knowing they are not relevant and then move on to the task.

The non-standard icons created additional obstacles when users had to find solutions from the system. In the scenario of Task 3 (create a staffing report and share with the manager), none of the testers were able to share the report on their first attempt. All users’ first reaction was to click on the “Save” button located in the top left area of the window after the report was generated, but the system pops up an error message when the button is hit. Users had to find another way to save and send the report. The non-standard icons became a usability issue, and users had to do some exploration within the system.

As some of the icons used on the site are not commonly seen elsewhere in other applications, and users reported feeling uncertain about the icons, they spent more time reading the label text and yet interpreted the same icons differently. Moreover, the familiar icons used on the site do not convey the information users typically associate with them, missing the point of using the icon in the first place.

Recommendation

Further usability tests should be done to test the icons for recognizability and gauge what users expect the icons to stand for. Research the commonly used icons and replace the ambiguous ones with those which accurately communicate their function.

Issue 2: System response time is too long

It was reported that the website’s response time is beyond acceptable. The think-aloud method allowed us to tap into the user’s thinking process when they waited for the system to respond. An expert user waited patiently and commented that “I will wait because I know it'll just take a minute or two for the report to pop up... I don’t expect an instant report.” The other experienced user said, “It usually takes about ten minutes, maybe is too long, (because I) want it to be done in two minutes.” Table 1 shows the average system response time of each task.

While it didn’t take ten minutes to receive a report as perceived by the user, the system certainly took longer than expected to produce a result as users may normally expect from a simple task like this.

Staff at client organization have no alternative options but to commit to using Cognos. The extended system response time creates unpleasant user experience after ten seconds, “the average attention span is maxed out, and the user does not retain information in short-term memory to easily resume the interaction once the computer finally loads the next screen.”⁵

Also, the current wait message (a mostly empty pop-up window with a spinning circle in the middle and a cancel button under the spinning icon; see Figure 6) does not deliver enough information about the report status to users. Users were unclear how long they needed to wait and, when they were waiting longer than expected, they were not sure if the system was still running or simply not responding.

Recommendation

If possible, the system should strive to reduce its processing time to less than ten seconds. If that is hard to achieve, use progress indicators or estimates to keep users informed of their waiting time or even allow the system to send a report to the user’s email once the report is complete. Users then can choose to involve in other activities if they know they must wait for long periods and let the computer complete in the background.

Issue 3: Many visible irrelevant options on the page

There are many action buttons on the page that users are not authorized to use, yet they are made visible and clickable. These irrelevant options on the screen compete with and diminish the visibility of other important information. When users must explore the website and experiment with the available options, as a user commented, they would expect everything they see on the site is relevant to them. One example is that testers were confused to see an error message read that “You are currently not authorized to access this object” (see Figure 7) when they clicked on some buttons as they explored the system.

On the other hand, critical actions are not obvious. A user explained that when she first used the site, she did not know what to do after the month selection because the critical action button, “Finish,” is small and located at the very bottom of the page—out of visual range (and possibly requiring scrolling).

Recommendation

Customize interface based on user need and authorization level. Simplify the site by eliminating irrelevant options and making the critical selections, such as “Finish,” more prominent.

Issue 4: The date-picking is a “pain.”

There are three select menus to let the user submit a staffing report within a specific period.

The first select menu asks if the inquiry is about the hours in the past or hours in the future. If the user wants to know the actual hours that have been charged in the past, then “Actual” should be selected. If a user is interested in knowing their estimated hours in the future, then “Forecast” should be selected. While the options seem self-explanatory, a user was confused about why such selection is necessary if the past months’ data would show the actual hours charged and future months data would show the budgeted hours regardless of either “Actual” or “Forecast” is selected.

The other two select menus ask the user to specify the beginning and ending months of the staffing report. Users need to select the months of interest from a list of every single month from January 2018 to December 2022 (seen in Figure 9). It is difficult for users to read the options due to small text font and lack of contrast. Small text font and lack of visual cues lead to poor readability. It is especially true for new users not familiar with this format, as the new user who first saw the select menus and complained immediately saying that “It is crazy to look at the options that look almost the same [audible sigh]... it is hard to target what I want.”

The current date picker format is not commonly used and is easy to cause user error. An expert user who participated in the study also commented that she paid extra attention to pick the right month because she “messed up” before and tried to avoid the same error again. The current month picking menu is problematic in several ways.

A user shared in the pre-test interview that she once helped a colleague to troubleshoot an issue of receiving an empty staffing report. The problem turned out to be a result of user error, as they picked an earlier month in the “To” field than the selected month in the “From” field. Users expected that if a wrong month were selected, the system would display an error message and recommend a change in the inquiry without proceeding to the next step. This issue can be easily fixed through disabling unacceptable entries. For example, when January 2019 is selected in the “From” field, any other earlier months become disabled in the “To” field. It is also a good practice that when a user error occurs, the system should help users to recognize, diagnose, and recover from the error by suggesting a solution⁶. In our case, the system should not run the report if it detects problematic entries, and the system should highlight the problematic fields and suggest modifying the month range.

Recommendation

• Eliminate the select menu for “Actual” and “Forecast” as it does not affect the report.
• Redesign the date picker using a more familiar format.
• Include practical constraints to limit users’ choices, provide immediate feedback when an error is detected, present users with a confirmation option before committing to the action, and suggest ways for the user to recover from an error. When selecting the month, if it is not possible to include constraints in Cognos, users should be reminded and become aware of the limitation.

Issue 5: Bad defaults

All testers did not like the current default months. The first month in the “From” field is January 2017 and the first month in the “To” field is also January 2017. Users remarked they rarely need to see what their coverage was two years back; they must scroll down in the list to make the correct selection. A user said that “I already know I don't need to go here” while pointing at months in 2017, “because I've messed up before” and then she scrolled down to January 2018 to continue Task 1. Another user suggested a better default is to have the current month in the “From” field and the third month after in the “To” field because this is how she usually specifies the month range when requesting a staffing report.

Recommendation

Utilize the existing user data to prioritize the options and set the most popular choices as the default selections. For example, verify if the current month is the most frequently selected in the “From” field and identify how many months earlier or later users commonly specify in the “To” field.

Issue 6: System version inconsistencies

Users reported that the current Cognos interface is different from the previous version. As a user stated, “It changed enough that [I] need to stop and figure it out.” Another user commented that “It took me a while to learn.” When users had to use the new system for a familiar task (for instance, Task 2, clearing the current staffing report to run a new one), and the solution was not obvious on the interface (in this case, no button indicating the action of clearing data or returning to the previous page is visible), the expert users heavily relied on their knowledge of prior system version (that is, looking for a button at the upper right corner of the page where the “back” button used to be in the last version). Failure to leverage a user’s knowledge of prior systems requires users to learn the new UI, distracts them from completing the central task, and undermine users’ confidence in using the system. In contrast, when consistency is present among different versions, users can easily transfer knowledge from the previous version to the new one, focus on executing the task, and enjoy a more satisfying experience.

Recommendation

Ensure consistency among versions if possible. User onboarding can be a powerful tool for both expert and new users. If users do not know the system, provide user onboarding focused on how to use the system. For expert users, providing onboarding which concentrates on showing the new features or explain the new changes to facilitate a smoother transition.

Conclusion

Cognos is an important internal tool for client organization staff to plan and monitor their staffing hours. A thoughtful user interface design of such an important tool not only provides a satisfying user experience to staff but also can reduce training and support cost and increase staff productivity. The evaluation of the upgraded Cognos 11 suggests several areas for interface design improvements. Some outstanding issues include the system’s overly complicated IA, unfriendly report layout, and other usability barriers preventing efficient use of the system. The recommendations embedded in this report are designed to address the issues discovered. We hope that this report not only serves as a UI guide for the next Cognos Staff Hours application development but also could start a dialogue for the company leadership to become aware of the staff’s experience with internal tools.