Statistical Analysis & Data Visualisation
App Evaluation
& User Testing
To assess app usability, a two-phase evaluation was employed. First, user-independent methods were used to analyze the app's design. Concurrent user trials then had participants complete pre-defined tasks, revealing factors like task success, time taken, etc. This combined approach helped gain data enabling judgement on the app’s usability.
Hierarchical Task Analysis
NASA-TLX
System
Usability Scale
Heuristic
Evaluation
Cognitive
Walkthrough
Hick’s Law
Fitt’s Law
Methods Used:
Tasks Chosen:
01
Check most read
news on the app
02
Change the language
03
Check comments of a news article on the app
The data collected during user testing was analyzed using statistical methods to assess usability. Completion rates, time spent on tasks, and error rates were likely evaluated. Visualizations like charts and graphs aided in interpreting the data and identify usability strengths and weaknesses within the app.
Task 1
Task 2
Task 3
Sub-task 1: Click on TOI+
Will users try to achieve the right result
Will users notice that the correct action is available?
Will users associate the correct action with the result they’re trying to achieve?
After the action is performed, will users see that progress is made toward the goal?
Sub-task 2: Scroll right on local navigation bar and click on ‘Most Read’
Sub-task 1: Click on Profile
Sub-task 2: Click on “Change Language”
Sub-task 3: Select Kannada
Sub-task 4: Click on “Save My Preferences”
Sub-task 3: Select an article to read
Sub-task 1: Click on any article
Sub-task 2: Click on the comment icon on the header.
Cognitive Walkthrough
Hierarchical Task Analysis (HTA)
Key Findings
Task: Revisit saved articles and photos
A lack of clear hierarchy and structure within the application's tasks.
Multiple menus contained duplicated tasks under various names, leading to confusion and inefficiency.
Sequential activities were not logically arranged, hindering smooth navigation through the app.
Quick tasks were unnecessarily extended due to the presence of excessive alternatives at each stage, slowing down user interactions.
Heuristic Evaluation
H2. Match between system & real world
-
unfamiliar icons + illogical menu order create a mismatch between what users expect and what the app offers.
-
no speech-to-text option might not align with real-world communication methods.
-
screen space has not been utilized properly.
-
missing crucial visual elements and elements disappearing on some screens. this disrupts user flow and makes it difficult to find what they need.
H1. Visibility of system status
H3. User control & freedom
-
disappearing elements take control away from users, interrupting their interaction flow.
H5. Error Prevention
-
the app suffers from poor error prevention as it lacks clear feedback mechanisms.
-
this is compounded by: unclear menus, buttons too small or close together, and a lack of visual distinction between interactive elements and simply displayed information.
!
H4. Consistency & standard
-
inconsistent layout and lack of a responsive grid lead to a confusing interface. users struggle to differentiate between clickable elements and simple information due to mismatched styles and poorly positioned buttons.
H6. Recognition Rather Than Recall
-
no onboarding screens are provided.
-
important elements are difficult to find, forcing users to rely on memory instead of clear app guidance.
H7. Flexibility & Efficiency of Use
-
splash screens waste time, frequently used menus are hard to reach.
-
app doesn't leverage device features like orientation changes, date/time, or location data and speech-to-text for easier interaction.
Help Users Recognise, Diagnose & Recover from Errors
H9.
-
no error feedback makes it difficult for users to identify and fix their mistakes.
?
-
text readability issues hinder communication between the app and the user.
H8. Aesthetic & Minimalist Design
-
inconsistent layout and excessive animations create a cluttered interface with misaligned elements and hard-to-reach buttons, hindering user focus and interaction.
H10. Help & Documentation
-
missing onboarding screens deprive users of essential guidance.
H12. Privacy
*no issues found*
H11. Pleasurable & Respectful Interaction
?
T
(Key Findings)
After conducting User-Independent Methods to analyse the apps navigation and design, data about the cognitive load TOI causes the users was gathered by conducting following User Testing Research Methods and the data was analysed using statistical methods to assess usability.
Usability Metrics of Efficiency
Data Type: Continuous (Ratio)
“How easily can users navigate through task flows of simple basic tasks on TOI?”
Null Hypothesis: 75% of users can complete
a simple basic task in 25 secs
Alternative Hypothesis: 75% of the users cannot complete a simple basic task in 25 secs
Results and inferences:
Among these tasks, two of them (Task 1 and Task 3) resulted in the rejection of the null hypothesis. Conversely, for Task 2, we accept the null hypothesis. This suggests that users, on average, could not navigate effortlessly for basic tasks.
Variables: Time taken to complete the task: independent variable.
Test: Two Tailed Binomial Test
Usability Metrics of Effectiveness
Data Type: Continuous (Ratio)
“Does user expertise reduce errors and enhance the TOI app's usability?”
Null Hypothesis: There is no significant difference in the distribution of the percentage of mistakes between novice and the expert users.
Alternative Hypothesis: There is a significant difference in the distribution of the percentage of mistakes between novice and the expert users.
Results and inferences:
Upon conducting the T test, T value of for task A and for task B , was less than the critical t value. Hence the hypothesis gets accepted. Conversely, for task C, the t value was more so the hypothesis got rejected. The data suggests that user expertise doesn’t play a role in influencing the error rates. However, for Task C, there is significant difference in the distribution of mistakes between novice and the experts.
Variables: Percentage of mistakes made dependent variable.
Test: Unpaired T-Test
System Usability Scale
Data Type: Continuous (Ratio)
“What are the 95% confidence limits for the SUS scores? What does that say about the usability of the app?”
Results and inferences:
The data says that we can be 95% confident that the true population average SUS score falls within 24.56 to 28.53.
The results indicate that the average SUS score for the TOI application, based on this sample of 30 users, is likely to be somewhere within these intervals. The true SUS score falls below a 68, that means the application falls short of meeting the benchmark, indicating the potential need for improvements in the app's design and it usability.
Variables: SUS Scores make the dependent variable
Test: Confidence Interval
NASA-TLX
Data Type: Continuous (Ratio)
“Does user expertise reduce errors and enhance the TOI app's usability?”
Null Hypothesis: Novice and expert users do not have significantly different ratings due to the variance in cognitive framework where the expert is well versed with the app.
Alternative Hypothesis: Novice and expert users have significantly different ratings due to the variance in cognitive framework where the expert is well versed with the app.
Results and inferences:
Upon conducting the T test on NASA TLX data, the test yielded a T value less than the critical T value and hence, the null hypothesis (H0) is accepted.
In simpler terms, regardless of whether users are novices or experts, both groups seem to experience a substantial cognitive load while interacting with the app. This outcome emphasizes the need to improve both the user experience and aiming to lessen the cognitive load for all users.
Variables: App design is the independent variable and movement time is a dependent variable.
Test: Unpaired T-Test
Through the analytical methods employed we understood that the key problems with the software included an incorrect task hierarchy, inconsistent representation of features, difficult navigation, excessive text, too many options and repetitive features, clustered screen layouts and cognitive overload.
Complex TOI app benefits from an onboarding to familiarize users, showcase features, and personalize content.
Reader is made aware of the progress while the interface is loading
Splash Screen
Onboarding Screens
They’re made more readable, has better placement and tells the user what to do next using a very polite and friendly language.
Login Screens
Home Page
Articles
Redesigned Screens
Home screen is less cluttered.
More hierarchically efficient hamburger menu for enhanced accessibility and visual flow.
The categories from the local navigation bar are integrated into the filter feature added to the home screen.
A calendar is added so the users can access old news.
Since in the app the icons were overlapping with the imagery making it look cluttered, users experienced some strain while trying to locate them.
In the redesign the placement of icons, the hierarchy of text and visuals are changed for better readability and optical appeal.
Briefs
To make the briefs more interesting and interactive whilst making it look visually more appealing, filter with all the categories from the local nav, comments, save and share are added.
Games
To make the briefs more interesting and interactive whilst making it look visually more appealing, filter with all the categories from the local nav, comments, save and share are added.
Profile
Previously, the settings page had the user profile. But that is not what users expected. They expected a separate page for profile. The new interface is less cluttered and still gives you the options you look for in the settings.
Saved Articles
A separate page for the games that is more easily accessible for the users. The imagery and font is changed so as to match the aesthetic of the app and retain the experience the newspaper provides till date.
After re-designing the apps’ screens, data about the screens’ (old and new) visual hierarchy and navigation was collected by running user tests employing fitts’ and hick’s law to make a thorough comparison between movement and reaction times of both designs. This helped infer if the redesigned screens succeeded in reducing the users’ cognitive load
Fitts’ Law
Data Type: Continuous (Ratio)
“Does the redesign of the app have an impact on users' movement times & cognitive workload?”
Null Hypothesis: There is no significant difference in the mean movement times between the two app designs (Old Design and Re-design).
Alternative Hypothesis: There’s a significant difference in the mean movement times between the two app designs (Old Design and Re-design).
Results and inferences:
The paired t-test rejected the null hypothesis (H0) by observing a substantial difference in mean movement times between the two app designs (Old design and Re design). The findings of the paired t-test show that there was a beneficial influence on movement time since there was a significant difference in readings between the old and redesigned interface of TOI. This means the redesign helped in making the experience and navigation seamless and convenient for users.
Variables: App design is the independent variable and movement time is a dependent variable
Test: Paired T-Test
Data Type: Continuous (Ratio)
“Does the redesign of the app have an impact on users' reaction times & cognitive workload?”
Null Hypothesis: There is no significant difference in the mean reaction times between the two app designs (Old Design and Re-design).
Alternative Hypothesis: There’s a significant difference in the mean reaction times between the two app designs (Old Design and Re-design).
Results and inferences:
The paired t-test rejected the null hypothesis (H0) by observing a substantial difference in mean reaction times between the two app designs (Old design and Re design). These findings show that there was a beneficial influence on reaction times since there was a significant difference in reaction times between the old and redesigned interface of TOI which means the redesign helped reduce cognitive workload for users.
Variables: App design is the independent variable and reaction time is a dependent variable
Test: Paired T-Test
User Feedback: the redesigned screens seemed cleaner, made navigation easier and efficient for them.
)
:
