Usability Measurement for Student Grading Information System Based Web

Usability Measurement for Student Grading Information System Based Web

Hendra, S.Kom.,M.T.a, Yulyani Arifin, S.Kom., M.M.b

a,b Computer Science Department, School of Computer Science, Bina Nusantara University,  Jl. K.H. Syahdan No. 9 Kemanggisan Palmerah Jakarta 11480, Indonesia

Abstract

This paper demonstrates the usability measurement of web-based student grade processing information system. The instrument used is a Use Questionnaire to obtain user satisfaction data. Atisa Dipamkara High School has been using an online application system to process the Students Grade. The application system is a web-based information system to assist the process of making the student reports every semester. This paper discusses the method of measuring the level of use of the application of student grade processing in Atisa Dipamkara High School. One factor in measuring the quality of an information system is usability. Usability level refers to the ease of use of such information systems or software. The higher the usability value means the higher the benefits of the information system in helping the users. This measurement of usability is using Use Questionnaire consisting of 3 parameters namely benefits (usefulness), ease to use and ease to learn. Data collection involves 25 teachers as user respondents of this information system. The result of usability measurement will have the value of “feasibility” and proof that there is significant influence between usefulness variable, ease to use and ease to learn to user satisfaction variable.

* Corresponding author. Tel.: +62-21-534-5830.

E-mail address: hendras@binus.ac.id

 

Introduction

Atisa Dipamkara High School has implemented online data processing information system for student grade reports. This web-based information system has been running for about a year. Utilization of information technology at Atisa Dipamkara school is needed to improve efficiency and productivity in education management. The information system that will be examined in this paper is the student grading system in Atisa Dipamkara high school that is used to assist the process of calculating the grade of student reports. This student grade processing system has been used by all high school teachers in Atisa Dipamkara to enter the score of the student assignment, midterm exam and final exam. This information system has been running for about a year but that has not been able to indicate that the information system has been well built because there has been no appropriate quality standard software measurement for the web-based system. In the development of a good system, one important part is good usability. Usability is closely related to user experience. User experience plays an important role in the development of a system, especially in this web-based information system, because the user experience can show users perceived ease and efficiency through the user experience in using the system.

Based on the background mentioned above, it can be seen the main problem that occurs is the absence of a standard measurement quality web-based online information system. The web-based information system has been widely used in everyday life, but not all information systems have passed through the testing phase of usability. This is due to the focus of the problem that is still centered on managing the needs, schedules, and resources available for the implementation of the system. While measuring the usability of the system from the user’s point of view is still considered not a major requirement in the development of a system, so very rarely done. Measurement of the success of the implementation of a system is done to determine the extent to which the user experience in using the system so that the purpose of the use of the system can be achieved. The results of the measurements will empirically describe user satisfaction of the system.

Measurement of usability is done by using a series of questionnaires that can process data related to effectiveness, efficiency, and satisfaction in the use of the information system. One of the questionnaire packages that can be used to measure usability is Use Questionnaire, because it can include three dimensions of ISO usability measurement that are efficiency, effectiveness, and satisfaction (Hevner, AC, March, S., Park, J., and Ram, S. 2004). 1

The objective to be achieved is to know the level or rate of usability in the user experience on that information system in Atisa Dipamkara high school by using Use Questionnaire. Research also aims to determine the relationship of independent variables to the dependent variable either simultaneously or partially. Given the results of Use Questionnaire measurements can be known “Feasibility” of this web-based student grading information system and also obtained a proof of the relationship between variables usefulness, easy to use and easy to learn to the variable of user satisfaction. The instrument used for this research is a Use Questionnaire to obtain user satisfaction data. Questionnaires were distributed to Atisa Dipamkara high school teachers that use web-based grade processing information system. This research uses Green and Pearson (2004) 2 theory to measure user satisfaction with usefulness, ease of use, ease of learning and one dependent variable of user satisfaction. The research object is a web-based information system for student grading.

The benefits of this research contribute to the scientific use of usability measurement standards in the user experience, specifically in web-based applications. This usability measurement standard can be useful for measuring user experience in web-based information system.

Methodology

The research methodology used adapted by IS Research methodology framework proposed by RH Von Alan, ST March, J Park and S Ram (2004) 1 where the information system research must have two sides that are relevant to the knowledge of the environment and adhere to existing knowledge base. The stages of research methodology begin with study of literatures and then the requirement analysis with research variables, hypothesis, population and sample and also research instruments. Then doing the questionnaire feasibility test and finally get the usability measurement result.

Experiment Design and Discussion

The research variable is an attribute or the nature or value of a person, object, or activity that has certain variations set by the researcher to study and then draw the conclusion. There are two variables that will be examined here, namely the independent variable and the dependent variable. The independent variables are usefulness variable, ease of use variables, and ease of learning variables, while the satisfaction variable is categorized as dependent variable.

This paper use one of the research design that is causal design that aims to analyze the relationship between one variable with another variables. The causal design is used to empirically analyze the variables that affect the user satisfaction, that is the usefulness variable, the ease of use variable, and the ease of learning variables based on the Use Questionnaire. The simple design of the causal design refers to the Green and Pearson (2004) theories in the conceptual model with several hypotheses:

H0: There is no significant influence between free variables, that is usefulness variables, ease of use variables, and ease of learning variables against dependent variables, that is satisfaction variables performed simultaneously or partially.

H1: There is a significant influence between the independent variables that is usefulness variables, ease of use variables, and ease of learning variables against the dependent variable that is the satisfaction variable performed simultaneously.

H2: There is a significant influence between variable usefulness to dependent variable that is variable satisfaction.

H3: There is a significant influence between ease of use variables on the dependent variable is the satisfaction variable.

H4: There is a significant influence between the ease of learning variable on the dependent variable is the variable satisfaction.

The population in this research is all 25 teachers who use the information system. From that population, multiple samples were taken to be respondents. The sampling technique is using simple random sampling method so that all members of the population have the same opportunity to be sampled. By using simple random sampling method, the entitled to be respondent is all active teachers. Determination of sample size was done by using Slovin formula:

Where: n = sample, N = population, e = the error rate is 10%       (1)

With population = 25 and the fault tolerance limit of 10%, the required number of samples is n = N / (1 + N e²) = 25 / (1 + 25x 0,1²) = 20 people.

The research instrument used is the Use Questionnaire. The questionnaire used in usability measurement is a series of questionnaires that can process data related to effectiveness, efficiency, and satisfaction in the use of an information system. Underlying the use of questionnaires is a questionnaire can provide convenience for respondents to understand and answer the questions properly. In addition, the questionnaire made the respondents more comfortable and freely in answering questions. The usability measurement adopts Use Questionnaire because according to the ISO 9241-11:1998 documentation the measurement of user satisfaction as part of Usability parameter includes three measures parameter, that is efficiency, effectiveness and satisfaction.

The feasibility test of the Use Questionnaire needs to be done to ensure that the results of the questionnaire data collection are feasible to be used for analysis. An instrument to be used in research must be valid and reliable so it is appropriate to be used as a research instrument. The feasibility test of the questionnaire was conducted using two methods, namely Validity Test and Reliability Test.

Validity test is used to determine the eligibility of the items in a question in defining the variables. The validity test used is correlate bivariate pearson (product moment correlation) and R table with degree of freedom, df = n-2 (n = number of respondents) and significance, α = 10%.

In conducting the validity test, the number of respondents is 20 people so it is known that the value of R table of 0.3783, which means the measuring tool can be declared valid if r count is greater than r table, and vice versa if the value of r countless than r table then the measuring tool declared invalid. In Table 1 there are 8 questionnaire points that are invalid. The cause of invalid questions is the lack of understanding of the users with the questions given, the different perceptions on the essence of the questions asked, the answers given by the respondents are inconsistent, or the respondents wan not serious in providing answers. Invalid questions cannot be used as a questionnaire (Matondang Z, p. 93, 2009), or should be replaced with another questionnaire. In response to this, it is chosen to eliminate invalid questions because they can be represented by another questions.

Table 1. Questionnaire Validity Test Results

Question No. R Value Validity Question No. R Value Validity Question No. R Value Validity
1 0,765 Valid 11 0,451 Valid 21 0,424 Valid
2 0,768 Valid 12 0,597 Valid 22 0,574 Valid
3 0,688 Valid 13 0,423 Valid 23 0,174 Invalid
4 0,582 Valid 14 0,361 Invalid 24 0,566 Valid
5 0,394 Valid 15 0,474 Valid 25 0,165 Invalid
6 0,727 Valid 16 0,312 Invalid 26 0,606 Valid
7 0,484 Valid 17 0,578 Valid 27 0,66 Valid
8 0,322 Invalid 18 0,627 Valid 28 0,29 Invalid
9 0,72 Valid 19 0,353 Invalid 29 0,792 Valid
10 0,627 Valid 20 0,352 Invalid 30 0,594 Valid

The reliability test then carried out to determine the consistency of measuring tools. An instrument is considered reliable if the instrument can be trusted as a measurement of research data. In this study, reliability test was performed using Cronbach’s Alpha size. To know the high reliability of the instrument used the category shown in Table 2. From result of calculation by using SPSS, got result of coefficient Cronbach’s Alpha as presented in Table 3. This reliability test is performed by entering the answers of all 22 valid questions and yielding Cronbach’s Alpha value is 0.915. Based on the reliability level of Cronbach’s Alpha described in Table 2, the value of 0.915 is in the range of 0.80 <α ≤ 1.00 so the results of the tests show that the reliability of the questionnaire is very high. So, the components of questions and answers can be said reliable so that further data processing questionnaire can be done.

Table 2. Cronbach’s Alpha Reliability Level.

Interval Reliability Category
0,80 < α ≤ 1,00 Reliability is very high
0,60 < α ≤ 0,80 Reliability is high
0,40 < α ≤ 0,60 Reliability is middle
0,20 < α ≤ 0,40 Reliability is low
0 < α ≤ 0,20 Reliability is very low

Table 3. Questionnaire Reliability Test Result.

Cronbach’s Alpha Total Item
0,915 22

The Analysis of questionnaire results is done after doing the data processing first. Data processing is done after obtaining validity test results and reliability accordingly under the condition. This data processing aims to measure the value of feasibility percentage and to know the relationship between research variables that exist in the Use questionnaire. The full form of Use questionnaire package used based on measuring usability with the Use Questionnaire (Lund, A. M., 2001).

For the purposes of quantitative analysis of the research, respondents will be given five alternative answers using the Likert measurement scale, as shown in Table 5.

Table 5. Likert Scale Measurement Criteria.

Score Criteria
1 Strongly Disagree
2 Disagree
3 Neutral
4 Agree
5 Strongly Agree

Quantitative analysis methods used to analyze the primary data obtained from the sample is to use statistical calculation method, namely multiple linear regression method, which consists of simultaneous correlation test (F test) and individual regression coefficient test (t test). Before doing multiple regression analysis, it is necessary to test the prerequisite of multiple regression analysis that is classical assumption test consisting of Normality test, Multicollinearity test, Heteroscedasticity test, and Autocorrelation test.

Normality test conducted to determine each research variable has a normal distribution or not. The data is said to be normally distributed when the data in the form of dots spread around the diagonal line and its distribution follows the direction of the histogram graph.

Multicollinearity test is done to determine the correlation between independent variables in a regression model. The non-occurrence requirement of multicollinearity is if the tolerance value is greater than 0.10 and the Variance Inflation Factor (VIF) value is less than 10. Based on the output in Table 6 it is known that the tolerance value of all independent variables is greater than 0.1 and the VIF value is smaller of 10, so it is concluded that there is no multicollinearity.

Table 6. Multicollinearity Test Result.

Model Tolerance VIF
Usefulness 0,700 1,429
Ease of Use 0,723 1,382
Ease of Learning 0,770 1,317

Heteroscedasticity test aims to find out in a regression model there is a variant or not inequality. The result of this test shows that there is no heteroscedasticity in the regression model.

The autocorrelation test is performed to find out whether in linear regression model there is correlation between the confounding error in period t with the error in the previous period or not. If the value of Durbin Watson is between -2 to +2 then there is no autocorrelation. From the test results using Durbin Watson, obtained the value of d-count as big as 1.825. This indicates that there is no autocorrelation, since the value is in the range -2 to +2.

Measurement of usability is done by calculating the percentage of respondents’ answers using the following formula: feasibility percentage = (observation score / expected score) x 100% .

From the result of observation score obtained and the expected score obtained the percentage of feasibility of the questionnaire result is equal to 75,23%.

The feasibility percentage of the results of this questionnaire is then converted according to the feasibility category table in Table 7. The result of the percentage of eligibility value of 75.23% is in the interval of 61% to 80%, which indicates that the measurement result of the usability of the web-based student grade processing information processing system has a “decent” value.

Table 7. Feasibility Category.

Interval Category
0 – 20% Very unfeasible
21% – 40% Not decent
41% – 60 % Quite decent
61% – 80% Decent
81% – 100% Very decent

F test or simultaneous test is used to see the relationship between independent variables (usefulness variables, ease of use variables, and ease of learn variables) on the dependent variable (satisfaction variable) together. The result of F test using SPSS is 20,807 with degrees of freedom df = 3 obtained from the number of free variables used and the df denominator of 96 obtained from the number of samples minus the number of variables. Then it is known that the value of F table is 2.14. While the significance value is known at 0.000. Since F count is greater than F table (20,807> 2,14) and its significance value is less than 0,1 (0,000 <0,1), it is concluded that simultaneously, usefulness, ease of use, and ease of learning variables influence significant to the variable satisfaction in terms of the use of this web-based student grade processing information systems.

The t test or partial test is used to find out the relationship between usefulness variables, ease of use variables, and ease of learning variables on partial satisfaction variables. The result of t test using SPSS can be seen in Table 9.

Table 9. Partial Test Result.

Model B t Significant
Constant 2,784 1,390 0,168
Usefulness 0,224 3,111 0,002
Ease of use 0,198 4,257 0,000
Ease of learn 0,095 0,620 0,537

Based on Table 9, the regression model can be analyzed based on the coefficients are:

𝑌 = 2,784 + 0,224 𝑋1 + 0,198 𝑋2 + 0,095 𝑋3                        (3)

Where: Y = Satisfaction, X1 = Usefulness, X2 = Ease of Use, X3 = Ease of Learning

From Table 9, we get t count for usefulness variable equal to 3,111, ease of use variable equal to 4,257 and ease of learning variable equal to 0,620. While the t value of the table is obtained from the t distribution table by looking at the df value and the level of significance divided by two. The value of df is obtained from the number of samples 20 minus the number of independent variables 3 then minus one, that is equal to 16, and the level of significance divided into two, to 0,05, so it is known t value table equal to 1,664. While the significance value of the variable usefulness of 0.002, ease of use variable of 0.000 and ease of learning variables of 0.537.

To test the truth of hypothesis 1, F test (simultaneous test) is to compare the value of F arithmetic with F table. If the calculated F value is greater than F table and the significance level is less than 0.1, then the regression equation and the correlation coefficient are significant so that 𝐻0 is rejected and 𝐻1 is accepted.

From the F test that has been done, obtained the fact that the value of F arithmetic is equal to 20.807 and F table of 2.14, so it is known that the value of F arithmetic is greater than F table. While the significance level of 0.000 which means less than 0.1. This proves that 𝐻0 is rejected and 𝐻1 is accepted, which means there is significant influence between usefulness variables, ease of use variables, and ease of learning variables against satisfaction variables performed simultaneously (in conjunction) in terms of the use of this web-based student grade processing information system.

From the t test that has been done to prove hypothesis 2, it is found that the variable 𝑋1 (usefulness) significantly influence partially satisfaction variable, because the t value obtained is 3.111, bigger than t table, that is 1,664, and the level of significance of 0.002 less than 0.1. This shows that 𝐻0 is rejected and 𝐻2 is accepted.

From the results of partial tests that have been done to prove hypothesis 3, it is known that the value of t arithmetic of 4.257 and t table of 1.664. While the level of significance is 0.000. It shows that 𝐻0 is rejected and 𝐻3 is accepted which means partially variable 𝑋2 (ease of use) significantly influence to variable Y (satisfaction).

From the test results partially, it is known that the value of t arithmetic of 0.620 and t table of 1.664. While the level of significance is 0.537. It shows that 𝐻0 is accepted and 𝐻4 is rejected which means partially variable 𝑋3 (ease of learning) not have a significant and positive effect on the satisfaction variable, so the regression equation becomes:

Y = 2.784 + 0.224X1 + 0.198X2                                            (4)

The regression equation shows that ease of learning variables cannot stand alone such as variable usefulness and ease of use. The average value of the lowest variable lies in the ease of learning variable (EOL), whereas the average of the highest variable value is the ease of use (EOU) variable. EOL variable with the lowest average value of 11.65 is caused by the difference of respondent ability in studying the web-based student grading information system. While the EOU variable that has the highest values of other variables with an average value of 33.92 is caused by the experience of respondents in perceiving the ease when using the system.

Conclusions

Based on the results of outcomes achieved from the analysis of this research data, can be drawn conclusion as follows. Measurement usability resulted 75.23% feasibility percentage indicating that the measurement result usability of web-based student grade processing information system has the value “Feasible”. Then, there is a significant influence between the independent variables, the usefulness variables, the ease of use variables, and the ease of learning variables against the dependent variable, the satisfaction variable, which is done simultaneously. It can be concluded also that partially, usefulness variables and ease of use variables significantly influence the satisfaction variable. While the ease of learning variable does not significantly affect the satisfaction variable.

References

[1] D.T. Green & J. M. Pearson. (2004). Confirmatory factor analysis of two website Usability, In proceeding of the third Annual Workshop on HCI Research, PP. 10-11, 2004

[2] Hevner, A. C., March, S., Park, J., dan Ram, S. (2004). Design Science in Information Systems Research, Management Information Systems Quarterly, 28(1), PP.77-105, 2004.

[3] Tullis, Tom, and Albert, Bill.(2008). “Measuring The User Experience”. Morgan Kaufmann Publisher

[4] ISO 9241-11:1998 –  The ISO website. [Online]. (2017). https://www.iso.org/obp/ui/#iso:std:iso:9241:-11:ed-1:v1:en, accessed: January 15th, 2017.

[5] Lund, Arnold M. (2017). “Measuring Usability with the Use Questionnaire”, Usability and User Experience, STC Community. Accessed from: http://www.stcsig.ord/usability/newsletter/0110 measuring with use.html , Dated : January, 10th  2017

[6] Spencer E.HarpePharmD, PhD, MPH. (2010). “Analyzing and Interpreting Data From Likert-Type Scales”. Currents in Pharmacy Teaching and Learning Volume 7, Issue 6, November–December 2015, Pages 836-850

[7] Matondang Z. (2009). “Validitas dan Reliabilitas Instrumen  Penelitian,”. Tabularasa Journal, PPS UNIMED, Medan.

[8] Nielsen, Jakob.(2017). “Usability 101: Introduction”, accessed from http://www.useit.com/alertbox/20030825.html , on January, 10th 2017

[9] Perlman, Gary. (2017). “User Interface Usability Evaluation with Web-Based Questionnaires”, Access from http://hcibib.org/perlman/question.html, on January, 10th 2017

[10] Sauro, Jeff. (2017). “8 Advantages of Standardized Usability Questionnaires”, Accessed from http://www.measuringusability.com/blog/standardizied-usability.php , on January, 10th 2017

 

Bapak Hendra, S.Kom., M.T. adalah pendiri sekaligus penulis artikel dalam website ini. Saat ini Pak Hendra bekerja sebagai developer advisor di PT. Visionet Data Internasional. Beliau juga mengajar di School of Computer Science BiNus University Jakarta.

Leave a Reply

Your email address will not be published. Required fields are marked *