EXPERIENCES WITH THE METACOGNITIVE SKILLS INVENTORY
Untuk memenuhi tugas :
Mata kuliah : Aplikiasi Komputer / TIK (AMDK 322)
Dosen Pembimbing : Drs. Syahmani, M. Si
Drs. Mahdian, M. Si
Disusun oleh :
Progam Studi :Pendidikan Kimia REG B
FAKULTAS KEGURUAN DAN ILMU PENDIDIKAN
JURUSAN MATEMATIKA DAN ILMU PENGETAHUAN ALAM
PROGRAM STUDI PENDIDIKAN KIMIA
UNIVERSITAS LAMBUNG MANGKURAT
EXPERIENCES WITH THE METACOGNITIVE SKILLS INVENTORY
Don Miles1, Toni Blum2, Wayne J. Staats3, David Dean4
Despite the national surge in interest in technology fields, student performance measures continue to reveal that many students lack the analytical skills to complete college level courses in math and computer science. The development of tools for the measurement and enhancement of metacognitive skills will be examined. The primary measuring tool, the Metacognitive Skills Inventory, was designed to measure metacognitive abilities. The inventory consists of two subscales: Decomposition and Confidence. The Decomposition subscale measures the subjects’ awareness and reported use of the critical problem solving steps. Examples are problem identification, planning of solution strategies, and comparison of these strategies. The Confidence subscale, measures the extent to which subjects are confident in their own problem solving ability. The inventory has been used in studies comparing student grades in college courses, problem solving skills at different levels in various college programs, and student performance on various tests. These tests include the SAT (both verbal and math), the MARS-R math anxiety scales, and the General Expectancy of Success Scale-Revised. The MSI is currently being used as an integral component of current research dealing with the development of training and education programs to increase metacognitive skills.
Due to the high attrition rate of computer science (CS) majors in the freshman year, the principle investigators have begun to develop educational exercises for the improvement of specific cognitive skills. A set of assessment tools for analyzing the students’ cognitive abilities as they progress through the program is also being developed [3,4, 20,21]. The development and distribution of the Metacognitive Skills Inventory (MSI) was the first step of this inclusive plan to incorporate the study and enhancement of metacognitive skills for CS students. These important cognitive and metacognitive skills must be clearly identified and then actively taught to students. While many educators assume that the requisite cognitive skills develop as a by-product of the CS curriculum, research has shown that many of these skills need to be taught . The MSI has been developed to measure people’s awareness of the cognitive processes they use in solving problems as well as the level of planning and organization they believe they engage in when solving a problem. The principle investigators have also begun to develop educational exercises, based on work being done by Deanna Kuhn et al.  at Columbia University, that are designed to improve specific cognitive skills. The MSI is used in conjunction with other assessment tools for analyzing students’ cognitive abilities as they progress through the educational exercises.
THE METACOGNITIVE SKILLS INVENTORY
CS students historically have successfully acquired a basic understanding of the grammar and features of a programming language, yet they often fail to achieve the desired level of performance in introductory programming courses [1,13]. Researchers argue that students must move beyond this syntactic level of knowledge to achieve conceptual knowledge, including basic cognitive skills similar to those used in program design. They also stress the importance of acquiring strategic level knowledge, which includes metacognitive skills that are used in solving novel programming problems, testing programs, and debugging logic errors . Formulating a mental model of problem structure and identifying a hierarchical structure of the relationships between the involved objects are conceptual cognitive skills that are crucial for CS education. In addition, CS students need to develop an awareness of their own mental models as they access and evaluate multiple design
- strategies during design development. The general argument among CS educators [10,11,13,22] is that most introductory programming courses do not place enough emphasis upon awareness and understanding of conceptual and strategic level knowledge, focusing instead upon syntactical issues. Thus, some key education/training issues to be resolved are: achieving a comprehensive, flexible, mental model of the problem structure.
- decomposing the model into a hierarchical structure that represents the relations among objects.
- evaluating the mental model as design development progresses
Cognitive and Metacognitive Skills
Each of the above key education/training issues can be addressed by introducing one or more metacognitive skills into the CS curriculum. One traditional approach to the realmof problem solving in human cognition is the notion that humans learn about new systems by comparing them to mental models of known (called base) systems . These mental models include objects, the attributes of those objects, and the relations among objects. The problem solver with extensive knowledge of the domain to be modeled would seem to have a distinct advantage in this area. Yet, it is important to note that the system to be modeled need not be literally similar to the base system; it must simply be analogous to it . That is to say that objects in the two domains need not have similar features, only similar relations. Two key metacognitive skills are necessary to develop appropriate and complete mental models. These skills are awareness and analogical transfer, respectively.
Researchers seem to agree that the first step to attaining insight into ones own mental models is simply getting individuals to become aware of their own thinking processes [2,14]. Attempts to induce such awareness have included having students “think aloud” (i.e. verbalize their thoughts) while solving a problem , asking students questions aloud about their actions and thoughts while solving a problem , and requiring students to answer a checklist of questions during the completion of some task.
Even when we have successfully induced students to pay attention to their mental models, they may not be able to discern whether or not the model they have activated is indeed the appropriate one for representing the problem domain. It is intuitively appealing to accept the view that people think by analogs (comparing problems with similar structures, but not necessarily the same features or story line). A key obstacle to this process is the failure of subjects to abstract relevant principles from the problem at hand . Finding the correspondence between objects and relations in one problem to objects and relations in another is referred to as “mapping” . Unfortunately, people are frequently distracted by “surface” features in the content of the problems and rarely see similarities in the underlying
structure or principles involved in the problems [6, 12].
In order to successfully use analogy to create a mental model, students must be able to extract the relevant facts (objects) from the problem, compare it to their own knowledge base in the problem domain, recognize relevant similarities between the new problem and previous ones they have encountered, and abstract out the relevant entities and principles [6, 12,17]. Research suggests that these skills can be somewhat improved by simply offering numerous opportunities to design models across a variety of domains, using the same underlying principles each time . Many researchers caution that individuals frequently fail to see the underlying principles unless they are explicitly pointed out [8, 17]. Even when provided with multiple examples, people often focus upon the procedural similarities of the problems, rather than the abstract principles that tie the problem structures together. Regardless of the programming paradigm to be used,
most researchers have found that good programmers begin by decomposing the larger problem into several smaller, subproblems implied by the larger design concerns. In addition, they must impose some structure upon the overall problem, in order to successfully track their progress through to completion. Standard decomposition includes such steps as algorithm development, the conversion of the algorithm into a flowchart or some sort of pseudocode representation, the coding from that representation into a specific programming language, and the execution and debugging of the code . Problem decomposition in the object-oriented paradigm is approached somewhat differently. The focus of design then, is upon what, rather than how. While this type of decomposition process should be inherently more natural than the decomposition required in traditional data processing approaches, research shows that without some inducement, people often ignore this step . People who fail to set appropriate goals and subgoals often use inefficient trial-and error methods to solve problems. This lack of planning can result in a poor understanding of the problem itself, even if it is successfully solved . While a solution arrived at via serendipity may be viable, it is unlikely to be useful in aiding the solver in future endeavors. Explicit decomposition and planning of the steps in a problemsolving task allows the solver to retrace those steps at a later time when a similar problem is encountered. Surprisingly, simply notifying students of the need to plan ahead has been shown to improve performance in problem-solving tasks .
The most common approach to enhance planning is the employment of active questioning during or prior to the task e.g. “What will you need to do to begin this task?”).
Even when students design object-oriented applications, the structure of the object space in the software design will be analogous to the programmers’ perceived structure of the problem space in the real world. To the extent that this assumption is true, the creation of a
sufficiently robust mental model (as defined previously) should imply the appropriate structure for problem decomposition. The mechanism by which object-oriented design supposedly captures this natural structure is termed “inheritance”, whereby objects in a class inherit the properties and abilities of their own superclass and lend their properties to their subclasses . The structure is essentially a hierarchical matrix, permitting programmers to
organize their objects much in the same way as concepts are organized in the natural world. Hierarchical organization, however, is another “natural” notion that is not so natural. Countless studies of memory and problem solving have shown that subjects who explicitly organized the to-beremembered material or the to-be-implemented steps into larger categories and subcategories perform better than those who do not. Yet, those same studies show that most subjects do not do this spontaneously. This lack of organization is especially prominent in novices (of virtually field).
Assessment of Metacognitive Skills
The MSI was designed as an elaboration upon these key issues dealing with metacognitive awareness. The inventory presented questions designed to measure peoples’ awareness of the levels of planning, organization, and evaluation that they implement while solving problems. The questions were also designed to assess the extent to which people have confidence in their problem-solving ability.
The MSI was derived from two scales that were designed for entirely different purposes. The State Metacognitive Inventory  was created as a device to be given immediately after the administration of an achievement test, for the purpose of assessing the extent to which students were aware of the thinking skills they used in the completion of that test. The Problem Solving Inventory  was created for the purpose of assessing people’s awareness of their own style of solving personal problems (such as relationship conflicts, career choices, emotional difficulties). Items were selected from each of these scales and then
reworded to refer to the more general problem solving tasks encountered by CS students. The initial scale consisted of a 55-item inventory. This scale was administered to 157 undergraduate students in psychology, general business, and CS courses to perform item analysis. Students rated each of the 55 items in the inventory using Likert-scale response options, ranging from 1 (strongly agree) to 4 (strongly disagree). Sample items included “I use multiple thinking techniques or strategies to solve the problems I encounter” and “I ask myself how new concepts relate to what I already know.” There were also items dealing with participants’ confidence in problem solving abilities. Sample items dealing with confidence included “I have the ability to solve most problems even though initially no solution is immediately apparent” and “When faced with a novel situation I have confidence that I can handle problems that may arise.”
Based upon item analysis, including item-to-total correlations, ten items that had an alpha level less than .285 were excluded. Therefore, the current version of the MSI consists of 45 items. The raw scores on each item of the MSI (following appropriate recoding) were added to obtain a summated score for the inventory. High scores on the inventory indicated lower levels of metacognitive skill while lower scores indicated high levels of metacognitive skill.
The Decomposition subscale, which measured the subjects’ awareness and reported use of the critical problem solving steps, such as problem identification, planning of
solution strategies, and comparison of these strategies, included 24 questions. The Confidence subscale, which measured the extent to which subjects are confident in their own problem solving ability, included eight questions. The other 13 items were used only for calculating the overall MSI scale score.
Standardization and Interpretation
After item analysis, 210 students from three separate courses were used for standardization purposes. The students consisted of students in a lower-level survey of technology course, a sophomore level statistics in psychology course, and a third group in an upper level CS course. As a preliminary step toward evaluating discriminant validity, the MSI scores from students in each of these classes were compared. A one-way analysis of variance revealed significant differences among the three groups of students, F(2, 207) = 9.22, p < .01. The MSI scores of upper-level CS students (M=84.06) were significantly better than lower level psychology students (M=95.13), and general survey-oftechnology students (M=94.26). A similar pattern was detected for the Decomposition subscale, with F(2, 207) =
6.91, p < .01, and for the Confidence subscale, with F(2, 207) = 3.69, p < .05.
In a second component of the test development sample, responses from 54 students within three different levels of courses in the CS department were also compared. A oneway
analysis of variance indicated a significant difference between student course levels on responses to the MSI, F(2,51)=6.34, p< .01. Post-hoc analyses (using Tukey’s Honestly Significant Difference) revealed that the first semester (fall) students had significantly higher (i.e. worse) MSI scores (M=92.79) than either the second (spring) (M=81.00) or third semester (summer) (M=80.92) students. However, no such differences were found between student scores on mathematics evaluation anxiety (F<1) or mathematics learning anxiety (F<1), as measured by the Mathematics Anxiety Rating Scale . These results
Note 1. N = 20
Note 2. Correlations significant at a <.05 indicated with *
Note 3. Correlations significant at a <.01 indicated with **
suggested that metacognitive ability is a better indicator of success in CS than mathematics anxiety.
The next step, after standardization and test development studies was completed, was to establish the validity of using the MSI to predict performance in the CS field. The perceived metacognitive skills of twenty professionals in the software industry were measured with the MSI. The levels of experience of these professionals were also recorded. Table 1 provides the Pearson correlation coefficients for these measures, and notable relationships are identified. From these results, metacognition, as measured by the MSI, was significantly related to the number of years in software development, with r = -.53, p < .05. subscale scores were significantly related to software development experience, with r = -.57, p < .05. Years of experience in object-oriented technology appeared to be similarly related to the MSI scores, but because of the small sample size, were not significantly related to any MSI measure (p = .08). Interestingly, Confidence subscale scores were positively related to number of years in CS education, indicating that confidence in problem solving ability actually decreases with teaching experience. Possible reasons for this finding could include that educators may be considering their ability to solve pedagogical problems rather than CS
problems, or perhaps educators are indoctrinated to believe that they always have more to learn.
During the summer and fall of 2000, a total of 88 participants, including undergraduates from many different areas and levels of study (science, humanities, social sciences, business), were measured with the MSI as well as several additional measures for the purpose of establishing convergent and discriminant validity. These other measures included the MARS-R math anxiety scales previously discussed, the General Expectancy of Success Scale-Revised (GESS-R) and Quinto’s Visual-Spatial Self-Report Inventory. The GESS-R is an established measure of optimism about success in life, including items such as “In the future I expect that I will succeed at most things I try”. This measure was included in order to rule out the possibility that the MSI is simply tapping into subjects’ optimism about their abilities. The Quinto scale was used to assess the extent to which subjects can identify what types of problems they are best at solving. Correlation between the Quinto scale and MSI was expected because the Quinto scale measures a person’s perceived style of problem solving (spatial versus other) one subset skill of metacognition. Finally, SAT mathematical and verbal component scores were included, where possible, for comparison purposes.
The Pearson correlation coefficients between these measures are presented in Table 2. As expected, neither the MSI nor the Decomposition subscale correlated significantly with either of the math anxiety scales. However, the Confidence subscale did correlate reliably with both the math learning anxiety score (r = .35, p < .01) and the math evaluation anxiety score (r = .29, p < .01), providing support for the contention that confidence in problem-solving decreases as anxiety about mathematics increases. These data provide additional support for the discriminant validity of the subscales.
The significant correlation between the MSI and GESS-R (r = -.39, p < .01) brings its’ discriminant validity into question. However, closer examination reveals that this relationship was largely accounted for by the GESS-R’s strong correlation to the Confidence subscale (r = -.55, p < .01). The Decomposition subscale was also correlated to the
Correlations, students during fall and summer 2000 semesters
Note 1. N = 88
Note 2. Correlations significant at a <.05 indicated with *
Note 3. Correlations significant at a <.01 indicated with **
GESS-R (r = -.27, p < .01), but at a much lower magnitude. It is not surprising to find a relationship between confidence in one’s problem solving skills and expectations of future success, but it is somewhat more intriguing that knowledge of planning and strategy selection would coincide with optimism about the future. It may be possible that those who are aware of their utilization of metacognitve abilities and strategies also have developed optimism about their future success.
As expected, Quinto’s Visual-Spatial Self-Report Inventory and the MSI are also related (r = .43, p < .01), providing some additional support for the convergent validity of these measures. A similar relationship was revealed between the Quinto inventory and the Decomposition subscale (r = .41, p < .01), but the relationship between the Confidence subscale and the Quinto inventory is of lower magnitude, with r = .24, p < .05. Again, these data support the assertion that the two subscales were measuring different components of metacognition. One curious finding was the significant correlation between the Quinto inventory and the GESS-R, suggesting that increased knowledge of problem-solving style corresponds to increased expectations for future success. This finding seemed to confirm the
previous suggestion that developing metacognitive awareness leads to optimism about future success.
SAT scores were available for only twenty-six participants from the original 88. There was a significant correlation between the Confidence subscale and SAT-V scores (r = -.44, p < .05), suggesting that confidence in problem solving accompanies proficiency in verbal skills. The correlation coefficients for the MSI and SAT-V, and the Decomposition subscale and the SAT-V were in the expected direction, but were not statistically significant.
As predicted, the MSI correlated positively with grades in CS courses, programming performance evaluations, and years of expertise in software development. It also successfully
differentiated between CS and non-CS students, first-year and upper-level CS students, and CS students and CS professionals. It accounted for more of the variability in CS grades than did math anxiety levels, SAT and ACT scores, or general expectations of success in school performance. The two subscales, Confidence and Decomposition, tapped into different components of metacognitive awareness in the individuals being assessed. Moreover, each component seemed to be related to students’ success in early CS courses. We suggest, therefore, that the MSI, in combination with other more traditional measures, can be a useful tool for predicting level of CS expertise.
The Metacognitive Skills Inventory is the first step in enhancing an undergraduate CS curriculum to explicitly address metacognitive skills. CS curricula assume these skills develop naturally as a student progress through the curriculum. It seems more likely, however, that the better MSI scores for more advanced students are a result of students with minimal metacognitive skills either opting out or dropping out of CS programs as they reach perceived barriers in their education. To address this question, the authors are currently working on the development of enhancement tools; these are exercises modeled after the work being done by Deanna Kuhn et al.  at Columbia University. Kuhn’s work concentrates on teaching metacognitive skills to middle school students. Using their exercises as models, the authors have developed exercises for both undergraduate CS and psychology students. During the spring 2003 semester, a small pilot study was completed to determine if the approach being taken at the middle school level is applicable to college students .
The pilot study showed that the MSI was a good predictor of the results on the sample exercises. Scores on the exercises, which may serve as behavioral measures (rather than self-report measures) of metacognitive skill reveal a pattern familiar to the MSI results: CS students performed better than non-CS students. This difference, however, was confounded with a gender effect. Future studies will expand the sample to balance gender more effectively and include gender in a regression model and to identify the amount of variance due to that factor. Follow-up studies will use the MSI to explore changes over time as the exercises are implemented within the curriculum.
We would like to thank Dana Bodner (Stetson University), Tameika Jackson (College Of Charleston), Phil Balem (College Of Charleston), and Ssu-yu (Tim) Lu (College Of Charleston) for their work in data collection and in the research process.
 Bayman, P. & Mayer, R. Using Conceptual Models to Teach BASIC Computer Programming. Journal of Educational Psychology, Vol. 80, 1988.
 Berardi-Coletta, B., Buyer, L. S., Dominowski, R. L., & Rellinger,
E. R. Metacognition and problem solving: A process-oriented
approach. Journal of Experimental Psychology: Learning,
Memory, and Cognition, Vol. 21, 1995.
 Blum, T. L., Staats, W. J., & Cochran, C.D. Reliability and
Validity of a Metacognitive Skills Inventory. Presented at the
45th Annual Southeastern Psychological Association Convention
in Savannah, GA, March 19-21, 1999.
 Blum, T. L. & Staats, W. J. The Assessment of Cognitive Skills
for Computer Science Education. IEEE Frontiers in Education
Conference, November 1999.
 Gentner, D. The mechanisms of analogical learning. In S.
Vosniadou & A. Ortony (Eds.), Similarity and analogical
reasoning. Cambridge, England: Cambridge University Press,
 Gick, M. L., & McGarry, S. J. Learning from mistakes: Inducing
analogous solution failures to a source problem produces later
successes in analogical transfer. Journal of Experimental
Psychology: Learning, Memory, and Cognition, Vol.18, 1992.
 Heppner, P. P. Problem-Solving Inventory (PSI). In Joel Fisher
and Kevin Corcoran (Eds) Measures for Clinical Practice, Vol.2,
New York: The Free Press, 1994.
 Hsu, C. , & Wedman, J. F. Content emphasis, practice, and
cognitive style in analogical problem solving. Journal of
Research and Development in Education, Vol. 28, 1994.
 Kuhn, D., Garcia-Mila, M., Zohar, A., & Andersen, C. (1995).
Strategies of knowledge acquisition, Monographs of the Society
for Research in Child Development (Vol. 60).
 Linn, M. C. & Clancy, M. J. The Case for Case Studies of
Programming Problems. Communications of the ACM, Vol.
 Linn, M.C. The Cognitive Consequences of Programming
Instruction in Classrooms. Educational Researcher, Vol. 14,
 Mayer, R. E. Thinking, Problem Solving, Cognition (2nd ed.).
New York: W.H. Freeman and Company, 1992.
 McGill, T. J. & Volet, S. E. A Conceptual Framework for
Analyzing Students’ Knowledge of Programming. Journal of
Research on Computing in Education, Vol. 29, 1997.
 O’Neil, H. F., Jr. & Abedi, J.. Reliability and validity of a state
metacognitive inventory: Potential for alternative assessment.
The Journal of Educational Research, Vol. 89, 1996.
 Plake, B. S. & Parker, C. S. (1994). Mathematics Anxiety Rating
Scale-Revised (MARS-R). In Joel Fisher and Kevin Corcoran
(Eds) Measures for Clinical Practice, Vol.2, New York: The Free
 Reif, F. Transcending prevailing approaches to science
education. In M. Gardner (Ed.), Toward a Scientific Practice of
Science Education. New Jersey: Lawrence Erlbaum Associates,
 Ross, B. H., & Kilbane, M. C. Effects of principle explanation
and superficial similarity on analogical mapping in problem
solving. Journal of Experimental Psychology: Learning,
Memory, and Cognition, Vol. 23, 1997.
 Rosson, M. B., & Alpert, S. R. The cognitive consequences of
object-oriented design. Human Computer Interaction, 5, 1990.
 Staats, W.J.,Dean, D., Miles, D., Blum, T. Developing
Techniques for Measuring and Enhancing Students’ Cognitive
and Metacognitive Skills, IEEE Frontiers In Education,
 Staats, W. J. & Blum, T. L. Enhancing an Object-Oriented
Curriculum: Metacognitive Assessment and Training. IEEE
Frontiers in Education, November, 1999.
 Staats, Wayne J. A Three-Pronged Approach to an Object
Oriented Computer Information Systems Curriculum.
Proceeding of the AIS 1998 Americas Conference, Baltimore,
Maryland, Aug 1998.
 Volet, S. E., & Lund, C. P. Metacognitive instruction in
introductory computer programming: A better explanatory
construct for performance than traditional factors. Journal of
Educational Computing Research, Vol. 10, 1994.