Background

Electronic learning refers to the learning methods of electronic educational content that is provided through the Internet (1-5) and provides an opportunity for students and teachers to interact. Several studies have highlighted the benefits of using the Internet in education, and it seems that virtual education has been welcomed by student (6). The growth of e-learning has affected the way learners learn and communicate in the educational environment, and learning management systems are being developed. These systems are designed to track registration status, attendance, grades, class schedule, test results, and other university and classroom management requirements and are special platforms that make it possible to present content through digital tools (7).

With the covid-19 pandemic, universities have been forced to conduct many of their educational activities using video conferencing software and online learning platforms to maintain social distancing, and online and electronic learning have remained an important part of education. Although professors and trainers have several training options, this type of training has presented new challenges regarding the usability of these systems (8).

Usability is considered a key concept in learning management systems. When designing and developing electronic learning systems, the principles of usability are applied in designing web-based user interfaces and application software so that users can easily access educational content and tools and react effectively by using them (9). The use of modern technologies can improve the usability of electronic learning systems. Using features, such as videos, interactive images, simulations, and interactive tools, can make the learning experience more attractive and meaningful. In addition, providing facilities such as language translation, paying attention to the needs of students with visual or hearing problems, and providing clear and complete explanations about how the system should be used can also help increase the usability of electronic learning systems (10).

The interface of electronic learning systems and their design can make it easy for learners to access the content of courses and be effective in encouraging learners or professors to use these systems and expand their use (6); however, if the electronic learning systems are not well and sufficiently equipped, it causes discouragement, frustration, and anxiety in users and reduces their interest in using the systems (8). Therefore, the low usability of electronic learning systems is an important factor in users non-acceptance (11).

Various studies have evaluated academic electronic management systems, and the results have demonstrated that although these systems were accepted in universities, there are significant challenges related to their usability (5). Penha et al. suggested that there were many problems using e-learning systems, which caused inappropriate and sometimes unpleasant user interactions and made it difficult for users to use these systems. Owing to the lack of basic principles of design and usability in these systems, there was overload and, in some cases, unnecessary content, inconsistency of messages, and inappropriate language were also observed. In addition, the design is not properly standard (12). Evaluating the usability of electronic management systems, Mtebe and Kissaka showed that some of these systems have usability problems that prevent their effective use (13). The research conducted by Fatma Tansu and Akbari Samani concluded that the electronic management system used in the Eastern Mediterranean University should be developed in terms of uploading new content and adding new features (14). Emiroğlu's study found that there were some usability problems such as system loading speed, site structure, grading mechanism, homework management, special language support, content abuse, and filtering (15). Melton's study showed that the success rate of sending homework using a learning system was 50%, and it was necessary to improve the system in this area. Another usability problem is that the visited links are not distinguished from the non-visited links (16). In Alshehri et al.'s research, information quality, system navigation and interactivity, system learning, and visual design were important aspects of electronic learning systems (17).

The research showed that if the usability of a learning management system is not at the desired level, learners would focus on learning the system itself instead of learning the content (18). On the other hand, despite the growing trend of using electronic learning management systems in higher education all around the world, a few studies have been conducted in the field of evaluating the usability and examining the possibilities of these systems in the higher education setting in Iran. The lack of sufficient information about the usability and possible obstacles of these systems can prevent their optimal design and effective use. In addition, it is necessary to understand the strengths and weaknesses of different learning management systems to improve their performance.

Objectives

The aim of this study was to evaluate the usability of electronic learning management systems at Kerman University of Medical Sciences.

Methods

This cross-sectional and descriptive study evaluated the usability of electronic learning management systems (ELMS) used at Kerman University of Medical Sciences, including the Navid system (https://kmunavid.vums.ac.ir), Skyroom (https://sr.kmu.ac.ir/), and Adobe Connect (https://oc.kmu.ac.ir/) by using heuristic evaluation. Three evaluators independently evaluated these systems using the Nielsen heuristics (19) and prepared a list of usability problems at the interfaces. The evaluators had academic education in the field of medical library and information sciences, specialized in the field of interfaces, and conducted research in this field. In addition, they had experience working with ELMS and had used these systems in their teaching.

The set of usability principles, Nielsen, is as
follows (19):

1- Visibility of system status

2- Match between system and the real world

3- User control and freedom

4- Consistency and standards

5- Error prevention

6- Recognition rather than recall

7- Flexibility and efficiency of use

8- Aesthetic and minimalist design

9- Help users recognize, diagnose, and recover
from errors

10- Help and documentation

First, meetings to unify concepts were held in the presence of evaluators and experts in the field of human-computer interaction. Ten items on how to extract problems were reviewed at these meetings. The evaluators then individually performed the evaluation based on the "severity rating" criterion. The severity rating of a usability problem is a combination of
three factors: 1) Frequency of the problem: is the problem common or rare? 2. Impact of the problem: Is it easy or difficult for users to overcome a problem?
3- Persistence of the problem: Does the problem repeat itself once, and if the users are aware of it, can they overcome it, or are they constantly bothered by it?

The evaluators rated each usability problem using 5-point rating scale. The severity rating scale is shown in following:

1. 0= I don’t agree that this is a usability problem at all
2. 1= Cosmetic problem only: need not be fixed unless extra time is available on project
3. 2= Minor usability problem: fixing this should
   be given low priority
4. 3= Major usability problem: important to fix,
   so should be given high priority
5. 4= Usability catastrophe: It is imperative to fix this before the product can be released (20). The final list of usability problems was provided to the evaluators to determine the severity rating of the problems independently. The average severity rating assigned by the evaluators was considered the final severity of the problems, and the final ratings were calculated and declared in %. In this study, a standard usability checklist was used to collect data, and its validity was confirmed (20).

All items were discussed by the evaluators in meetings to ensure the reliability of the checklist, and agreement was reached regarding their perceptions. Previous studies have also measured the reliability of this checklist in the same way (21). Three researcher-made forms were used to enter the problems in a categorized manner. Each evaluator recorded the problems using Form 1. This form consists of a table including the problem name, problem description, problem location, and the corresponding "heuristic.” Form No. 2 included the final problem, description of the final problem, location of the final problem, relevant heuristic, and relevant system. The researcher completed the study. Form 3 was used to determine the severity rating of the final problems, which was completed by the evaluators and included the name of the problem and the severity rating scale.

Descriptive statistics (frequency, percentage, and mean) were used to analyze the research findings. The information related to the problems (problem, its related heuristic, severity rating of the problem, and its occurrence location) is presented in tables. Finally, the obtained data were analyzed using SPSS software version 20 (version 20, IBM Corporation, Armonk, NY, USA) and Excel.

Results

Based on Nielsen heuristics, 106 usability problems were found in ELMS of Kerman University of Medical Sciences.

Note that Skyroom showed the largest number of problems detected (52 cases) and the lowest number of issues were observed in Navid system (22 cases) and Adobe Connect (39 cases) (Figure 1).

According to Table 1, the most problems in the Navid system were classified in ‘flexibility and efficiency of use’ (45.45%). No issues related to ‘helping users recognize, diagnose, and recover from errors’ were observed in the Navid system. The high percentage of problems in Skyroom were also classified as ‘flexibility and efficiency of use’ (23.07%).

The largest issues in Adobe Connect were related to ‘helping users recognize, diagnose, and recover from errors’ (38.46%). In this system, the problems related to ‘visibility of system status’ and ‘match between system and the real world’ were not found. Problems related to ‘error prevention’ and ‘help and documentation’ were not observed in any of the electronic learning management systems (Table 1).

Table 1. Number of problems in terms of Nielsen’s usability heuristics

Table 2 shows that Adobe Connect had no problems in terms of ‘visibility of system status’. In the Navid system and Skyroom, the high percentage of problems identified in ‘visibility of system status’ were mainly due to ‘failure to provide (new) information or system feedback following a user's action’. In Adobe Connect, there were no problems related to ‘match between system and the real world’. All the problems identified in ‘match between system and the real world’ in the Navid system and 60% of these problems in Skyroom were due to the ‘ambiguity in commands, information, signs, and options’. ‘lack of appropriate functional options (buttons) to start system activities’ was observed in all the systems. The highest number of this problem was observed in Skyroom (50%) and the lowest number of problems in this field was related to the Navid system (33.33%).

Half of the problems related to ‘consistency and standards’ in Skyroom were due to the ‘using of heterogeneous layout and design to display information and buttons’. All the problems in the Navid system and Adobe Connect were due to the ‘absence of search button on all pages’ and ‘non-use of uniform items that users are used to’, respectively. In all systems, there was no exit button in the main menu (being in the submenu). The highest number of this problem was observed in the Navid system (100%). In Skyroom, the most problems observed in ‘recognition rather than recall’ were related to the ‘impossibility of recognizing the use of available buttons according to their labels’. Twenty percent of problems identified in ‘flexibility and efficiency of use’ in the Navid system were related to system shutdown while conducting online tests/updating the system. Among the significant problems of ‘flexibility and efficiency of use’ in Adobe Connect, we can mention the impossibility of writing Farsi correctly in this system.

Half of the problems classified in ‘aesthetic and minimalist design’ in Navid system were due to ‘existence of too much information on the pages’ and the other half were due to ‘the lack of suitable visual icons in the design’. The most aesthetic problems in Skyroom were related to ‘repetition of buttons and icons’ and ‘inappropriate board design’. In Adobe Connect, aesthetic problems such as ‘inappropriate color of pages’, ‘inappropriate board design’, ‘crowded pages’ and ‘complexity of menus’ were observed. In Navid system, there was no problem in terms of ‘help users recognize, diagnose, and recover from errors’.

The most problems in terms of ‘help users recognize, diagnose, and recover from errors’ in Skyroom and Adobe Connect were due to ‘use of inappropriate error message in response to user action’ (Table 2).

Table 2. Distribution of problems identified in Nielsen’s usability heuristics

Additionally, evaluators rated 50% of the problems in the Navid system as ‘major’ and 50% of them as ‘catastrophic’. The most problems in Skyroom were ‘Major’ (57.69%). More than half of the problems in Adobe Connect were ‘catastrophic’ (51.28%) (Table 3).

In the Navid system and Adobe Connect, most usability problems were related to the main page
(the whole system) (45.45% and 51.28%, respectively). In Skyroom, more than 30% of the problems were related to using the board (34.61%) (Table 4).

Table 3. Severity ratings of problems identified in electronic learning management system

Table 4. Number of problems identified in each part of the system

Discussion

Today, with the increase in the need for online education and the use of electronic learning management systems, the learning management systems used in universities have also increased and they are equipped with the ability to provide content and online educational facilities (22).

In our study, a total of 106 usability problems were identified in different parts of these systems. Some of the problems overlapped with the results of previous studies in this field (12, 13, 23). For example, in Mtebe and Kissaka's study, 9 usability problems were found in the learning management system of the University of Dar es Salaam (13). In the evaluation of Info3Net in the Ssemugabi study, 75 usability problems were identified (23).

The results of our study showed that in the Navid system, 50% of the problems were ‘major’ and 50% were ‘catastrophic’. The most problems in Skyroom were reported as ‘major’ (57.69%). More than half of problems in Adobe Connect were ‘catastrophic’ (51.28%).

In this study, all of Nielsen’s heuristics were violated and there was no problem with the ‘error prevention’ and ‘help and documentation’ heuristics. While Penha et al. evaluated the interface of e-learning management systems used in the Instituto Federal de Pernambuco (IFPE), they identified 54 usability problems. All Nielsen’s heuristics were violated in Moodle, Blackboard, Teleduc, Aulanet, and Amadeus systems, and the lowest number of problems was related to ‘help and documentation’ (12).

One of the important factors in the usability of electronic learning systems is their simplicity and ease of use. The interface of electronic learning systems should be simple, understandable, and familiar to users so that they can easily move in the educational environment and take advantage of the available tools and resources. Also, they should be adaptable to the needs and preferences of users so that they can fully respond to their demands and expectations (10).

While the results of our study showed that the most usability problems in the Navid system and Skyroom were identified in ‘flexibility and efficiency of use’, in Adobe Connect, the most problems were related to ‘help users recognize, diagnose, and recover from errors’. In such a way that sometimes the system could not display the desired page and there was no error message to show what happened and suggest a solution to the problem. An efficient system should provide an appropriate error message at appropriate times in response to the user's action.

In the Penha et al. study, after ‘consistency and standards’, ‘flexibility and efficiency of use’ showed the highest number of problems in electronic learning management systems (12). In the Eastern Mediterranean University (EMU), the investigated system (Moodle) had good flexibility and according to the evaluators, it provided the possibility of efficient use for experienced and inexperienced users. Also, this system was successful in identifying and improving errors; because the errors were expressed in simple language (without codes) (14). To improve usability, providing appropriate and timely feedback to users is also very important (24, 25).

In the present study, the most flexibility problems identified in the Navid system were due to ‘system shutdown while conducting online tests/updating the system’. Among the significant problems identified in ‘flexibility and efficiency of use’ in Adobe Connect, we can mention the ‘Impossibility of writing Farsi correctly’.

In Emiroğlu's study, the learning management system did not support the Turkish language in some parts of the program (15), which shows that learning system designers should pay attention to this issue and consider flexible options to help users achieve their goals and increase the efficiency of e-learning management systems.

In the study of Eltahir et al., most evaluators of e-learning courses at Ajman University agreed that there is not enough capacity in the used e-learning system and to achieve a more flexible system, the capacity of the system should be increased in such a way that it is possible to load more data in the form of sound, image, video, and recording of lectures (26).

In the evaluation of Adobe Connect, no problems were found in the ‘visibility of system status’. In the Navid system and Skyroom, the most problems identified in ‘visibility of system status’ were due to ‘failure to provide (new) information or system feedback following a user's action’; while the system should always inform the user about the current situation at the right time through appropriate feedback (27). In the Penha et al. study, 11% of usability problems were related to ‘visibility of system status’ (12), but in the study of Fatma Tansu and Akbari Samani, the system used in the Eastern Mediterranean University had appropriate visibility and at the right time, it provided appropriate feedback to the evaluators (14).

In this research, there was no ‘match between system and real-world’ problem in Adobe Connect. In the study of Fatma Tansu and Akbari Samani, the heuristic of ‘match between system and real world’ was well observed and instead of system-oriented terms, words, phrases, and concepts familiar to the user were used (14).

All the problems identified in ‘match between system and real world’ in the Navid system and 60% of these problems in Skyroom were due to ‘ambiguity in commands, information, signs, and options’.

In the Skyroom, half of the problems identified in ‘Consistency and standards’ were due to the ‘using of heterogeneous layout and design to display information and buttons’. All the Consistency problems of the Navid system and Adobe Connect were due to the ‘absence of search button on all pages’ and ‘non-use of uniform items that users are used to’, respectively. In Penha et al. research, most of the problems identified in consistency and standards were due to the lack of uniformity in the environment design (12). In the learning management system of the Eastern Mediterranean University, ‘consistency and standards’ was observed on average (14). It is suggested to use a specific policy and uniform items that users are used to in the design of systems and avoid using words, terms, or functions that have different meanings.

In all three systems examined in our study, there was the problem of ‘lack of exit button in the main menu’. Most of this problem was observed in the Navid system.

In Skyroom, the most ‘Recognition rather than recall’ problems were related to the ‘Impossibility of recognizing the use of available buttons according to their labels’. The system should reduce the user's mental pressure by creating visible objects, operations, and options. In other words, the system should help identify operations and options so that the user needs less mental and physical activity and the instructions for working with the system should be visible and accessible (26). For example, users recognize images very quickly. Therefore, the instructions for working with the system can be explained with images. These images are a good way to convey functionality to users (28). In Fatma Tansu and Akbari Samani's study, designers minimized the user's mental pressure by making objects, actions, options, and instructions visible (14).

Half of the problems identified in ‘aesthetic and minimalist design’ in the Navid system were due to the ‘existence of too much information on the pages’ and the other half were due to ‘the lack of suitable visual icons in the design’. The problems classified in ‘aesthetic and minimalist design’ in Skyroom were due to ‘repetition of buttons and icons’ and ‘inappropriate board design’. In Adobe Connect, ‘aesthetic and minimalist design’ problems such as ‘inappropriate color of pages’, ‘inappropriate board design’, ‘crowded pages’, and ‘complexity of menus’ were observed.

Additional and repetitive information on pages causes distraction and reduces performance. Systems designers should avoid duplicate information to increase readability so that users can focus on the main topic and without getting confused by irrelevant images or texts, they can easily find the information they need (28, 29).

In Penha et al. study, most of the ‘aesthetic and minimalist design’ problems were related to excessive content of information on some pages, inappropriate use of items, and confusing banners (12).

Limitations: Conducting evaluation only in one university, conducting evaluation only from the point of view of evaluators, failure to consider environmental factors such as hardware and software facilities related to the usability of the systems, which sometimes led to the lack of access to the systems in some cases.

Conclusion

The results of our study showed that many problems were observed on the home page of the electronic learning management systems. Some heuristics, such as ‘error prevention’ and ‘Help and documentation’, had more suitable conditions. But in terms of heuristics such as ‘flexibility and efficiency of use’, there was no suitable situation. It is recommended that the designers of these systems pay attention to the use of usability principles in designing the interface and improving their usability to create a more favorable user experience. In addition, the need to train users in using the systems and increase their information about the benefits and capabilities of these systems is felt to reduce the obstacles to using these systems for users.