Psychometric Evaluation of Culturally Responsive Numeracy and Problem-Solving Assessment: A Rasch Modeling Approach
), Daud Karel Walanda(2), Mustamin Idris(3), Mery Napitupulu(4), Anggraini Anggraini(5), Rita Lefrida(6), Richard Albert Walanda(7)
| 
Country:
(1) Doctoral Program in Science Education, Universitas Tadulako, Department of Mathematics Education, Universitas Alkhairaat, Indonesia
(2) Department of Chemical Education, Universitas Tadulako, Indonesia
(3) Department of Mathematics Education, Universitas Tadulako, Indonesia
(4) Department of Chemical Education, Universitas Tadulako, Indonesia
(5) Department of Mathematics Education, Universitas Tadulako, Indonesia
(6) Department of Mathematics Education, Universitas Tadulako, Indonesia
(7) Department of Computer Science, University of Newcastle, Australia
Corresponding Author
The quality of measurement in educational research is highly dependent on the validity and reliability of the instruments used. However, in practice, many numeracy literacy and problem-solving instruments have not undergone adequate psychometric testing, particularly those that use cultural contexts as question stimuli. This study aims to develop and evaluate instruments for numeracy literacy and problem-solving in sequences and series, grounded in cultural context/local wisdom, using a research and development approach and Rasch modeling. The instrument's content validity was assessed through expert review by learning evaluation experts, mathematics education lecturers, and cultural experts (traditional councils). The analysis was conducted using the average validator score, Aiken's V coefficient, and percentage agreement to ensure the appropriateness of the substance, clarity of language, instrument appearance, and the appropriate integration of cultural context. Empirical trials were conducted in small groups with 71 high school students in grade X with 15-17 years in three regions in Indonesia: Palu City, Donggala Regency, and Sigi Regency, who were selected purposively. The test data were analyzed using Rasch modeling to evaluate item suitability, difficulty level, reliability, separation index, unidimensionality, and potential bias through DIF analysis. The analysis results showed that the developed instrument had very high content validity, strong person and item reliability, and that most items fit the Rasch model. However, the distribution of item difficulty did not fully cover the extreme range of respondents' abilities, and indications of DIF were found in some items in one area, indicating the need for further refinement. Thus, the developed numeracy literacy and problem-solving instrument was deemed suitable based on initial findings at the development stage; however, it still requires further revision and testing before being widely used in a more diverse population.
Keywords: numeracy literacy, problem-solving, rasch model, reliability, validity.
Adams, E. K., Hancock, K. J., & Taylor, C. L. (2020). Student achievement against national minimum standards for reading and numeracy in years 3, 5, 7, and 9: A regression discontinuity analysis. Australian Journal of Social Issues, 55(3), 275–301. https://doi.org/10.1002/ajs4.124
Amalina, I. K., & Vidákovich, T. (2022). An integrated STEM-based mathematical problem-solving test: Developing and reporting psychometric evidence. Journal on Mathematics Education, 13(4), 587–604. https://doi.org/http://doi.org/10.22342/jme.v13i4.pp587-604
Amalina, I. K., & Vidákovich, T. (2023). Assessment of domain-specific prior knowledge : A development and validation of mathematical problem-solving test. International Journal of Evaluation and Research in Education, 12(1), 468–476. https://doi.org/10.11591/ijere.v12i1.23831
Andrian, D., Nofriyandi, Wahyuni, R., Loska, F., Nurhalimah, S., Mz, Z. A., Risnawati, Maisaroh, S., Dacara, E., & Hidayat, R. (2025). Numeracy skills students' profile of Riau Province: content and construct instrument-based Riau culture and evaluation results. International Journal of Evaluation and Research in Education, 14(6), 4638–4654. https://doi.org/10.11591/ijere.v14i6.33426
Anriana, R., Witri, G., Putra, Z. H., Fendrik, M., Dahnilsyah, & Aljarrah, A. (2023). Ethnomathematics study in measurement of Bengkalis Malay community as mathematics resources for elementary school. Ethnography and Education, 18(3), 299–322. https://doi.org/10.1080/17457823.2023.2232500
Ariffin, S. R., Omar, B., Isa, A., & Sharif, S. (2010). Validity and reliability multiple intelligent item using rasch measurement model. Procedia: Social and Behavioral Sciences, 9, 729–733. https://doi.org/10.1016/j.sbspro.2010.12.225
Asy’ari, M., Ikhsan, M., & Muhali, M. (2018). Validitas instrumen karakterisasi kemampuan metakognisi mahasiswa calon guru fisika. Prisma Sains : Jurnal Pengkajian Ilmu dan Pembelajaran Matematika dan IPA IKIP Mataram, 6(1), 18. https://doi.org/10.33394/j-ps.v6i1.955
Avinç, E., & Doğan, F. (2024). Digital literacy scale : Validity and reliability study with the rasch model. In Education and Information Technologies (Vol. 29, Issue 17). Springer US. https://doi.org/10.1007/s10639-024-12662-7
Boone, W. J., & Staver, J. R. (2020). Advances in Rasch analyses in the human sciences. Springer Nature Switzerland.
Charoentham, M., Kantathanawat, T., Pimdee, P., & Teerasoonthontai, K. (2025). Assessing creative problem-solving competencies in primary school educators : A confirmatory factor analysis approach. Edelweiss Applied Science and Technology, 9(1), 262–277. https://doi.org/10.55214/25768484.v9i1.4111
Chen, Y., Guo, C. J., Lim, K. M., Mun, K. J., Otsuji, H., Park, Y. S., Sorrell, D., & Winnie So, W. M. (2020). The influence of school entry skills in literacy and numeracy on the science achievement of fourth grade students and schools in Asian regions. Eurasia Journal of Mathematics, Science and Technology Education, 16(9). https://doi.org/10.29333/EJMSTE/8385
Debelak, R., Strobl, C., & Zeigenfuse, M. D. (2022). An introduction to the rasch model with examples in r. Boca Raton: CRC Press.
Directorate of Senior High School. (2021). Panduan penguatan literasi dan numerasi [Guide to strengthening literacy and numeracy]. Ministry of Education and Culture.
Evans, C. M., & Taylor, C. S. (2025). Culturally responsive assessment in classrooms and large-scale contexts. New York: Routledge.
Gusmanida, G., Sujati, H., & Herwin. (2024). Testing the construct validity and reliability of the student learning motivation scale using confirmatory factor analysis (cfa). Jurnal Penelitian Pendidikan IPA, 10(7), 4227–4234. https://doi.org/https://doi.org/10.29303/jppipa.v10i7.7518
Hardiyanti, T. A., Syaf, A. H., & Widiastuti, T. (2022). Pengembangan modul berbasis etnomatematika pada materi barisan dan deret. Seminar Nasional Pendidikan, FKIP UNMA, 285–300.
Hellstrand, H., Korhonen, J., Räsänen, P., Linnanmäki, K., & Aunio, P. (2020). Reliability and validity evidence of the early numeracy test for identifying children at risk for mathematical learning difficulties. International Journal of Educational Research, 102, 101580. https://doi.org/https://doi.org/10.1016/j.ijer.2020.101580
Hu, L., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6(1), 1–55.
Husain, H., & Aziz, H. (2022). Exploratory factor analysis (EFA) and confirmatory factor analysis (CFA) were used to assess the construct validity and reliability of historical thinking skills, TPACK, and the application of historical thinking skills. International Journal of Education, Psychology and Counselling, 7(46), 608–623. https://doi.org/10.35631/IJEPC.746046
Jackson, C. J. (2022). The utility of NAPLAN data: issues of access, use and expertise for teaching and learning. Australian Journal of Language and Literacy, 45(2), 141–157. https://doi.org/10.1007/s44020-022-00009-z
Kline, R. B. (2016). Principles and practice of Structural Equation Modeling. New York: The Guilford Press.
Ladson-billings, G. (1995). Toward a theory of culturally relevant pedagogy. American Educational Research Journal, 32(3), 465–491.
Lam, T. T., Seng, Q. K., Hoong, L. Y., Jaguthsing, D., & Guan, T. E. (2011). Making mathematics practical: An approach to problem solving. Making Mathematics Practical: An Approach to Problem Solving, 1–156. https://doi.org/10.1142/8171
Lin, Y., Yu, Y., Zeng, J., Zhao, X., & Wan, C. (2020). Comparing the reliability and validity of the SF-36 and SF-12 in measuring quality of life among adolescents in China: a large sample cross-sectional study. Health and Quality of Life Outcomes, 18(1), 1–14. https://doi.org/10.1186/s12955-020-01605-8
Mayer, R. E. (1998). Cognitive, metacognitive, and motivational aspects of problem solving. Instructional Science, 26(1–2), 49–63. https://doi.org/10.1023/a:1003088013286
Ministry of Education, Culture, Research, and Technology. (2023). Framework asesmen kompetensi minimum (AKM) [Minimum competency assessment (AKM) framework]. Center for Educational Assessment, Agency for Standardization, Curriculum, and Assessment in Education.
Mohamad, M. M., Sulaiman, N. L., Sern, L. C., & Salleh, K. M. (2015). Measuring the validity and reliability of research instruments. Procedia: Social and Behavioral SciencesSocial and Behavioral Sciences, 204, 164–171. https://doi.org/10.1016/j.sbspro.2015.08.129
Nahdi, D. S., Jatisunda, M. G., Cahyaningsih, U., & Suciawati, V. (2020). Pre-service teacher’s ability in solving mathematics problem viewed from numeracy literacy skills. Elementary Education Online, 19(4), 1902–1910. https://doi.org/10.17051/ilkonline.2020.762541
Nisa, K., Suprapto, N., Amiruddin, M. Z., Sari, E. P. D. N., & Athiah, B. D. (2024). Ethnoscience-quizizz test to measure problem-solving skills : a Rasch analysis. International Journal of Evaluation and Research in Education, 13(6), 4247–4255. https://doi.org/10.11591/ijere.v13i6.28075
OECD. (2012). Literacy, numeracy and problem solving in technology-rich environments: Framework for the OECD survey of adults skills. OECD Publishing. https://doi.org/10.1787/9789264128859-en
OECD. (2023). Pisa 2022 results (volume I): The state of learning and equity in education. PISA, OECD Publishing, Paris. https://doi.org/10.1787/53f23881-en
Olkun, S., Altun, A., Şahin, S. G., & Kaya, G. (2016). Psychometric properties of a screening tool for elementary school students’ math learning disorder risk. International Journal of Learning, Teaching and Educational Research, 15(12), 48–66.
Platas, L. M., Ketterlin-Gellar, L., Brombacher, A., & Sitabkhan, Y. (2014). Early grade mathematics assessment (EGMA) toolkit (Issue March). RTI International.
Prasetyo, Z. K., Sesa, E., & Lembah, G. (2020). Psychometric and Structural Evaluation of the Physics Metacognition Inventory Instrument. European Journal of Educational Research, 9(1), 215–225.
Polya, G. (1957). How to solve it: a new aspect of mathematical method, second edition. In Princeton University Press: United States of America (Vol. 2, p. 253). http://www.jstor.org/stable/3609122?origin=crossref
Pratama, R. A., & Yelken, T. Y. (2024). Effectiveness of ethnomathematics-based learning on students’ mathematical literacy: a meta-analysis study. Discover Education, 3(1). https://doi.org/10.1007/s44217-024-00309-1
Rahman, N. A., Masuwai, A., Tajudin, N. M., Tek, O. E., & Adnan, M. (2016). Validation of teaching and learning guiding principles instrument for Malaysian higher learning institutions. Malaysian Journal of Learning and Instruction, 13(2), 125–146. https://doi.org/https://doi.org/10.32890/mjli2016.13.2.5
Rahmawati, D. E., & Trimulyono, G. (2021). Validitas instrumen penilaian higher order thinking skills (hots) pada materi keanekaragaman hayati. Berkala Ilmiah Pendidikan Biologi (BioEdu), 11(1), 138–147. https://doi.org/10.26740/bioedu.v11n1.p138-147
Ramadhani, R., Soeharto, S., Arifiyanti, F., Prahmana, R. C. I., Saleh, A., & Lavicza, Z. (2025). Assessing quality and biases in ethnomathematics-based numeracy worksheets: A many-facet Rasch model analysis. Social Sciences & Humanities Open, 12, 101736. https://doi.org/https://doi.org/10.1016/j.ssaho.2025.101736
Raof, S. A., Musta’amal, A. H., Zamzuri, F. K., & Salleh, M. H. (2021). Validity and reliability of students' perceptions on obe approach in Malaysian VC using rasch model. Journal of Innovation in Educational and Cultural Research, 2(2), 44–50. https://doi.org/10.46843/jiecr.v2i2.30
Reffiane, F., Sudarmin, Wiyanto, & Saptono, S. (2021). Developing an instrument to assess students’ problem-solving ability on hybrid learning model using ethno-stem approach through quest program. Pegem Journal of Education and Instruction, 11(4), 1–8. https://doi.org/10.47750/pegegog.11.04.01
Rosa, M., D’Ambrosio, U., Orey, D. C., Shirley, L., Alangui, W. V., Palhares, P., & Gavarrete, M. E. (2016). Current and future perspectives of ethnomathematics as a program. In ICME-13 Topical Surveys. Switzerland: Springer Nature. http://www.springer.com/series/14352
Sari, C. M., Ridwan, A., & Sastrawijaya, Y. (2024). Learning environment of numeracy: development and validation with confirmatory factor analysis and rasch model. Pedagogika, 153(1), 185–198. https://doi.org/https://doi.org/10.15823/p.2024.153.9 Pedagogika
Schouten, H. (1985). Statistical measurement of interobserver agreement. Erasmus Rotterdam.
Setiyani, Waluya, S. B., Sukestiyarno, Y. L., Cahyono, A. N., & Santi, D. P. D. (2024). Assessing numeracy skills on flat shapes and scaffolding forms in junior high school. International Journal of Evaluation and Research in Education , 13(1), 422–432. https://doi.org/10.11591/ijere.v13i1.25186
Suciati, Munadi, S., Sugiman, & Ratna Febriyanti, W. D. (2020). Design and validation of mathematical literacy instruments for assessment for learning in Indonesia. European Journal of Educational Research, 9(2), 865–875. https://doi.org/10.12973/eu-jer.9.2.865
Swarni, A., Herwin, & Sujati. (2024). Testing the construct validity and reliability of the student learning interest scale using confirmatory factor analysis (cfa). Jurnal Penelitian Pendidikan IPA, 10(9), 6322–6330. https://doi.org/https://doi.org/10.29303/jppipa.v10i9.8794
Syifa, A., Azizah, A. N., Fardanti, A. C., Rahayu, F., Indratno, T. K., Sukarelawan, M. I., & Abdullah, N. S. Y. (2024). Problem-solving ability of high school students : Preliminary study analysis using Rasch modeling. Journal of Environment and Sustainability Education, 2(2), 135–145. https://doi.org/10.62672/joease.v2i2.37
Umbara, U., & Suryadi, D. (2019). Re-interpretation of mathematical literacy based on the teacher’s perspective. International Journal of Instruction, 12(4), 789–806. https://doi.org/10.29333/iji.2019.12450a
Utami, N., Setiawan, A., Hamidah, I., & Koehler, T. (2025). Investigating problem-based worksheets ( PBWs ) to improve understanding in logic gates topic : stacking and racking analyses in rasch model. Educational Process International Journal, 16, e2025243. https://doi.org/https://doi.org/10.22521/edupij.2025.16.243
Walker, M. E., Olivera-aguilar, M., Lehman, B., Laitusis, C., Guzman-orth, D., & Gholson, M. (2023). Culturally Responsive Assessment: Provisional Principles (Issue December). https://doi.org/10.1002/ets2.12374
Winckel, A. van de, Kozlowski, A. J., Johnston, M. V., Weaver, J., Grampurohit, N., Terhorst, L., Juengst, S., Ehrlich-jones, L., Heinemann, A. W., Melvin, J., Sood, P., & Mallinson, T. (2022). Reporting guideline for RULER: Rasch reporting guideline for rehabilitation research: Explanation and elaboration. Archives of Physical Medicine and Rehabilitation, 103(7), 1487–1498. https://doi.org/10.1016/j.apmr.2022.03.019
Zainil, M., Putra, A. P., Sa’ud, U. S., Helsa, Y., Desmaiyanti, Ariani, Y., & Rusdinal. (2024). Development of learning management system model and numerical literacy-based message content. International Journal of Instruction, 16(4), 1041–1060. https://doi.org/https://doi.org/10.29333/iji.2023.16457a
Zebua, V., Rahmi, & Yusri, R. (2020). Analisis kesalahan siswa dalam menyelesaikan soal barisan dan deret ditinjau dari kemampuan pemahaman konsep matematis [Analysis of students' errors in solving sequence and series problems viewed from the perspective of their ability to understand mathematical concepts.]. Jurnal LEMMA, 6(2), 122–133. https://doi.org/10.22202/jl.2020.v6i2.4088
Refbacks
- There are currently no refbacks.
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.