Determining the Best Answers for Balinese Language Problems using Latent Semantic Analysis

- In Balinese, descriptions or essays are formed in an interrogative format using question words such as “akuda”, “apa”, “dija”, “kenken”, “kuda”, dan “nyen”. The assessment process on description questions tends to be more difficult and complex than multiple choice questions, this is because the description questions are described in sentence form. The solution to facilitate the assessment process on description questions can be done using automated essay scoring. Based on the results of previous studies, the Latent Semantic Analysis (LSA) method provides a better level of accuracy, because the LSA method uses the Singular Value Decomposition (SVD) method to obtain a new pattern of relationships between terms and reference terms. The data used in this study are five questions and their answer keys and there are five candidate answers for each question in Balinese. Based on the tests that have been carried out, the method used obtained an overall average accuracy of 70.26%, this shows that the LSA method can be used well in the assessment process or automatic essay assessment.


INTRODUCTION
Essay questions are used to measure the level of understanding of "someone" towards something (Contreras et al., 2018). In Balinese, essay questions are formed in an interrogative format using question words such as akuda, apa, dija, kenken, kuda, and nyen (Granoka et al., 1996). Description or essay questions are described in the form of sentences, this makes the assessment process more difficult and complex compared to multiple choice questions (Chen et al., 2014). Based on previous research, the use of the automated essay scoring method can facilitate the assessment process on description questions and under certain conditions can obtain good accuracy (McNamara et al., 2015). Previous research related to automated essay scoring has been done by Fauzi, et al. in the e-learning system using the cosine similarity method and the n-gram method, the accuracy obtained is 67% (Fauzi et al., 2017). Citawan, et al. on the e-learning system using the latent semantic analysis, cosine similarity, and ngram methods, the accuracy obtained is 78.65% (Citawan et al., 2017). Based on the results of previous studies, the LSA method provides a better level of accuracy, because the LSA method uses the Singular Value Decomposition (SVD) method to obtain a new pattern of relationships between terms and reference terms (Citawan et al., 2017). Considering the process of assessing description questions is more complex than multiple choice questions, we propose a system that helps the process of assessing description questions as well as sorting the answers that are most relevant to the answer key using the LSA method. Figure 1 is the proposed method.

RESEARCH METHODOLOGY
Source: (Subali & Suniantara, 2022) Figure 1. Question Answering System Method 1. Answer Key and User Answer At this stage, two inputs will be given, namely the answer key and the user answer. In the answer key and user answer, the process of changing each character to the lowercase form and removing punctuation marks is carried out.

Preprocess
In the pre-processing stage, tokenize, stopwords removal, and stemming processes are carried out. The determination of stopwords in Balinese has been studied by Putra, et al. which includes the words anggen, sane, ring, miwah, puniki, and olih (Putra et al., 2016), and for the Balinese stemming process, we use the stemmer method that has been done by Subali, et al., where the stemmer method uses the rule based method and n-gram string similarity (Subali & Fatichah, 2019).
3. LSA LSA is a method for analyzing the semantic structure of the text by utilizing the statistical computing (Citawan et al., 2017). The following are each step in the LSA method: a. Form a matrix , where row of the matrix contains unique words in each document and column contains document labels, while the cell contains the frequency of occurrence of words , .
b. Applying Singular Value Decomposition (SVD) on matrix , where the matrix is decomposed into three forms, , , and matrix. c. Reduces the matrix by storing all the rows in the first columns and and the first rows and columns .

= × ×
(2) Information: is the number of matrix reduction parameters.
d. To determine the similarity of each text, the matrix obtained by the LSA method will be measured using the cosine similarity method.

Cosine Similarity
The cosine similarity method is used to measure the level of similarity between the keywords obtained and the document (Fauzi et al., 2014;Subali & Wijaya, 2021) in equation (3) is a way of measuring the level of similarity using the cosine similarity method.
(3) Information: , is the weight of word in document . , is the weight of the word in the question . Term weight is calculated using a bag of words.

Data
The research data used were five questions and their answer keys in Balinese. These five questions are topics related to basic computer science. In Table 1 are the five questions used. Meanwhile, data related to the list of answers will be collected using a questionnaire method, where each question contains five candidate answers. At the time of data collection will also involve five respondents who work as active students. Source: (Subali & Suniantara, 2022) 2. Answer Key and User Answer The data answer key is the answer key for each question, while the user answer is the answer to the five respondents for each question. Figure 2 shows the answer key (AK) and user answer (UA) initialization model for each question.

Preprocess
In the preprocessing stage, the answer key and user answer data are carried out in several stages, starting from tokenize, stop word removal, and finally stemming. In Table 2 is the number of features generated at the data preprocessing stage.  (Subali & Suniantara, 2022) 4. LSA In the early stages of the LSA method, it is done by forming a term matrix from the answer key and user answer data. Where the term matrix is formed with conditions, where the row contains the features, the column contains the question number, while the cell contains the number of words or features that appear in the question number. In Table 3, the term matrix in the answer key and user answer in question number one is shown, while the term matrix for other question numbers can be seen at the link https://intip.in/5SoalBesertaRespondenFitur.  Source: (Subali & Suniantara, 2022) Information: is answer key. is user answer.
There were 33 features obtained from the answer key and the five user answers to question number one. After the term matrix is obtained, then the matrix decomposition process is carried out using the SVD method which produces three different matrices, namely the , , and matrices using equation (1). From the three matrices, the matrix reduction process is then carried out by storing all rows in the first columns and and the first rows and columns using equation (2), where the value of = 2. In Table 4, the decomposition matrix of in question number one is obtained.  (Subali & Suniantara, 2022) Paradigma, Vol. 24, No. 2, September 2022P-ISSN 1410-5063, E-ISSN: 2579 In Table 4 it can be seen that the matrix only takes 2 columns from the decomposition matrix. In Table 5 it is a decomposition matrix of , while in Table 6 it is a decomposition matrix of in question number one. Source: (Subali & Suniantara, 2022)   Source: (Subali & Suniantara, 2022) In Table 6 : is user answer.

Cosine Similarity
Before the similarity measurement process is carried out, the vector value for the answer key must first be calculated using equation (4), as follows: is answer key. is answer key transpose. is the decomposition matrix . −1 is the decomposition matrix power -1.
In Table 7 is the vector value of the answer key in question number one. Source: (Subali & Suniantara, 2022) So that the vector value in the answer key to question number one is obtained, namely: : (-0.141781402, 0.015503256) Information: is answer key.
The last step is to calculate the level of similarity between the answer keys and each respondent's answer using the cosine similarity in equation (3).
The following is the process of measuring the level of similarity in question number one for respondent number one,

Results
Details of the results of the level of similarity in all question numbers can be seen in Table 8.  Based on the results of the level of similarity obtained and a manual examination of the answers of each respondent and the answer key showed very good accuracy (close to the maximum value of one) for the five questions. It can be seen from the average value obtained in each question, that the majority obtained an average value of > 0.5, or the average accuracy on all questions was 0.70263403 or 70.26%. This proves that the LSA and cosine similarity methods can be applied well in the process of automatically scoring essay questions. If you look at Figure 4, it is only in question number three that the majority get an average value of < 0.3, this is due to the lack of similarity in word structure in respondent's number 2, 3, and 5 when compared to the answer key, on the other hand, respondents with numbers 1 and 4 get accuracy. which is very good. In addition, in another case in question number 5 respondent 3 obtained a very small accuracy of 0.18960860 compared to other respondents on the same question, this is because the number of dimensions or features in respondent 3 is very large compared to the number of features in the answer key, this is an advantage of the LSA method because the LSA method in addition to paying attention to the structure of the occurrence of similar words, the LSA method also pays attention to the number of data features being compared.

CONCLUSION
The application of the LSA and cosine similarity methods to the automated essay scoring that has been carried out has obtained a good average accuracy of 70.26% of tests that have been carried out. Based on the test results, the LSA method is also able to overcome the difference in the number of features in the words being compared, this is because the LSA method does not only focus on paying attention to the structure of the similarity of words but also pays attention to the number of features of the words being compared. The LSA and cosine similarity methods have a weakness when the compared word conditions have the same word similarity structure but have different word orders, then the LSA and cosine similarity methods will provide a high level of similarity value even though the word order has differences or does not have meaning. In future research, the LSA and cosine similarity methods will use the n-gram method in the formation of each feature to be able to focus on paying attention to the word order when two words are compared.