Study with the several resources on Docsity
Earn points by helping other students or get them with a premium plan
Prepare for your exams
Study with the several resources on Docsity
Earn points to download
Earn points by helping other students or get them with a premium plan
Community
Ask the community for help and clear up your study doubts
Discover the best universities in your country according to Docsity users
Free resources
Download our free guides on studying techniques, anxiety management strategies, and thesis advice from Docsity tutors
A technical report from teachers college, columbia university's department of human development, program in measurement, evaluation, and applied statistics. It provides a revised coding manual for identifying the involvement of content, context, and process subskills for the timss and timss-r 8th grade mathematics tests. The report includes detailed descriptions of the tc attributes and acknowledgements of research team members.
Typology: Study notes
1 / 21
This page cannot be seen from the preview
Don't miss anything!
th
James E. Corter Kikumi Tatsuoka Anabelle Guerrero Michael Dean Enis Dogan Department of Human Development Teachers College, Columbia University
Acknowledgements
We wish to acknowledge the contributions of all the members of our research
team who assisted in this project by solving TIMSS items and proposing attributes.
These team members included (in addition to the authors): Jennifer Grossman, Toshihiko
Matsuka, Eun Kyung Um, Tao Xin, Seongah Im, and Tomoko Yamada.
This Technical Report is based upon work supported by the National Science
Foundation, Award # REC-0126064. Any opinions, findings, and conclusions or
recommendations expressed in this material are those of the authors and do not
necessarily reflect the views of the National Science Foundation.
The TIMSS (1995) and TIMSS-R (1999) tests
The Third International Math and Science Study, or TIMSS (1995) was an ambitious project designed to measure educational achievement in math and science, across grade levels from 4 to 12 and in more than 50 countries. The follow-up TIMSS-R (1999) study focused on Grade 8 math and science achievement, and gathered more detailed data on curriculum, school, teacher, and student background variables, plus benchmarking data from 27 jurisdictions in the U.S. Thirty-eight countries participated in the TIMSS-R (Mullis, Martin, Gonzales, Gregory, Garden, O’Connor, Chrostowski, & Smith, 2000). The data collected is many-faceted, including not only achievement data from the actual math and science test, but also background questionnaires aimed at measuring various aspects of students, teachers and schools, and even including videotaped observations of actual math and science lessons in the various countries.
The TIMSS-R (1999) Test Framework
The TIMSS-R test items were classified according to a framework that assigned each item to one of five content areas and one of five “performance expectation” classes. The five content areas of the test were: 1) Fractions and number sense, 2) Measurement,
Development of Attributes for the Present Study
For the present study, we identified attributes believed to be involved in successful solution of the TIMSS mathematics items by the following method. First, we asked a team of domain experts (faculty and graduate students in measurement and statistics, including several former high school mathematics teachers) to solve the complete pool (163 items) of TIMSS-R (1999) “population 2” (eighth grade) mathematics test items. These experts individually generated proposed attributes corresponding to important facets of content knowledge, and process subskills. As an additional source of information on the processes by which students solve these mathematics items, we also collected and analyzed written protocols from four high school students who were asked to solve all the items. The attributes generated by this process were then jointly discussed, combined and refined. The attributes were classified as being one of three general types. The first group of attributes consists of content knowledge attributes (see Section II), that is, clusters of content knowledge. The second group consists of attributes describing special-purpose test-taking skills, some associated with a particular type of test item (for example, items with “open-ended” answer formats). This group is called “skill” or item type” attributes. Finally, the third group of
stakes tests such as the SAT, test prep instruction often teaches “non-mathematical” strategies for finding the correct alternative in multiple-choice tests.
4) “Component coding” versus “Focus coding” of content knowledge. The published test design frameworks for the TIMSS and TIMSS-R tests specify that the Content categories of items are defined to be mutually exclusive. For example, an item is described in the TIMSS test design framework as dealing with Geometry or Algebra, but not both. We refer to this type of coding as focus coding of Content, because enforcing the requirement of mutual exclusivity requires that the coder make a decision as to the primary content (or the primary purpose of the item from an educational measurement perspective). In contrast, a complete specification of the cognitive and mathematical skills required in solving an item would requires that all necessary content knowledge be recognized in the coding ( component coding of Content), as long as the requirement for that category of skill is non-trivial. Consider item B08: “If there are 300 calories in 100 g of a certain food, how many calories are there in a 30 g portion of this food?” A student who chose to solve this problem algebraically might set up this problem as an equation with one unknown: 300 /100 = x / 30, then rearrange it to solve: x = (300)(30)/100 = 90. Under focus coding, such an algebraic solution for this item might be coded as focusing on attribute C3 (Algebra). However, under component coding, we might code C1 and C2 in addition to C3, because we recognize that integer arithmetic skills (C1) and fraction/ratio manipulation skills (C2) are required for simplifying the ratio expression to find the exact answer.
Depending on the researchers’ purposes, one or the other of these coding conventions might be followed. However, componential coding of content seems more in line with prominent theories underlying diagnostic testing. In general, we recommend component coding of content and have used that type of coding in the majority of our reanalyses of student performance in TIMSS and TIMSS-R.
C1: Basic concepts and operations in whole numbers and integers C2: Basic concepts and operations in fractions, decimals, and ratios C3: Basic concepts and operations in elementary algebra C4: Basic concepts and properties in geometry C5: Data techniques; probability and basic statistics C6 Measurement and estimation
C 1 (Whole Numbers and Integers) A student who has mastered this attribute should be able to understand and apply basic concepts and operations in whole numbers and integers, including addition, subtraction, multiplication, division, exponentiation, sign, absolute value, place value, etc. The attribute does not include simple comparisons of the magnitudes of integers or digits (which of two numbers is larger or smaller). It does include rounding of 2- or 3- digit integer values.
C 2 (Fractions, Decimals, Ratios) A student who has mastered this attribute should be able to understand and apply basic concepts and operations in fractions and decimals, such as setting up ratios, using them in calculations and equations, use of mixed fractions, decimals, etc. Note that setting up ratios of integers and evaluating them could be considered to be merely a basic operation involving integers (C1), but we argue that the skills involved are somewhat distinct and more closely allied with other skills in C2, thus should be coded as C2. Rounding and ordering of decimal quantities and fractions is included.
C 3 (Algebra) A student who has mastered this attribute should be able to understand and apply basic concepts and operations in elementary algebra, such as representing an unknown by a symbol, simplifying expressions and solving equations involving an unknown, etc.
C 4 (Geometry) A student who has mastered this attribute should be able to understand and apply basic concepts and operations in geometry, such as knowing basic geometric concepts, shapes, property of polygons, Cartesian coordinates, etc. The concept of distance as applied to a 1-dimensional space (number line) is included, but not the use of a number line to represent simple magnitudes.
C 5 (Data, Probability, and Statistics) A student who has mastered this attribute should be able to interpret and manipulate data in simple ways, and to use data already represented in graphs, charts, or tables. Also, the student should understand basic probability concepts, and be able to
compute and interpret basic statistics (for example, understand the concept of the mean and calculate it).
C 6 (Measurement and Estimation) A student who has mastered this attribute should be able to measure (or estimate) length, time, angle, coordinates, temperature, etc., using appropriate tools and appropriate units. The knowledge necessary to convert units of measurement is also included (e.g., the number of milliliters in a liter; the number of seconds in a minute, etc.) Simple rounding of numbers and numeric place value does not count as this attribute (knowledge of place value would be C1 or C2), nor does simple visual estimation.
P1 Translate words by formulating algebraic equations or algebraic expressions P2 Computational applications of knowledge in arithmetic, algebra and geometry P3 Judgmental applications of knowledge in arithmetic, algebra and geometry P4 Applying complex rules in algebra P5 Logical reasoning (including case reasoning, deductive thinking skills, necessary and sufficient, generalization skills) P6 Problem-space search: analytic thinking; problem restructuring; search for novel strategies P7 Generating, visualizing or transforming figures and graphs P8 Recognizing and evaluating mathematical correctness; Proof skills P9 Managing complex data, procedures, subgoals or conditions P10 Reading and understanding comparative or logical terms
P 1 (Translate Words by Formulating Algebraic Equations or Algebraic Expressions): A student who has mastered this attribute (at the 8th^ grade level) should be able to translate a word problem into an algebraic expression or equation where the designation of the variable term and constant(s) is not necessary apparent, there are no more than two unknowns, and the language in the word problem that describes the appropriate operations (e.g., subtraction) is simple (e.g., "less than"). Translation is sometimes straightforward, but most of the time it requires identification of variables and relationships buried implicitly in verbal expressions. This attribute usually refers to descriptions of “real world” problems and experiences, rather than simple verbal expression of mathematical content. This attribute does not include using given geometrical figures or spatial information.
holds true in the general case as opposed to simply a specific case (i.e., check necessary and sufficient conditions); understand the conditions for solving systems or problems.
P 6 (Problem-Space Search: Analytic thinking, Problem Restructuring, Search for Novel Strategies): The student should able to analyze the problem into component parts or constituent elements which are not obvious, or are latent or implicit; structure the component parts cognitively so that a problem become solvable; synthesize two or more concepts and theorems into a solvable form. A student who has mastered this attribute should be able to reject a "wrong structure" which is already given, and find a new one, where the new structure is in a more solvable form. When two or more strategies exist, the student can choose the better, simpler, or quicker strategy. When two or more rules, properties, or theorems are available, the student can choose a better, simpler, or quicker one.
P 7 (Generating, Visualizing or Transforming Figures and Graphs): A student who has mastered this attribute should be able to generate or manipulate (in explicit written form, or mentally) diagrams and figures to facilitate problem solving activities.
P 8 (Recognizing and Evaluating Mathematical Correctness; Proof Skills): A student who has mastered this attribute should be able to work backward from the options in order to reach a solution. Such a student may solve a task by: working backwards from the multiple-choice options, producing one or two examples and from these inferring the correct option; solving the problem "intuitively" in which case the reasoning may be unconventional; substituting numbers to select the correct option; eliminating one or more of the options and guessing from among the remaining options.
P 9 (Managing Complex Data, Procedures, Subgoals or Conditions ) : A student who has mastered this attribute should be able to: deal with problems that involve two or more steps/subgoals by paying explicit attention to record keeping, management of information or data, or management of a complicated chain of reasoning. Also, be able to do calculations involving management of intermediate results. These steps can include explicit or implicit subgoals, and are not necessarily latent. In other words, the student should be able to: establish the subgoals of the problem; order and prioritize the subgoals; execute the subgoals in the proper order in a step-by-step fashion. A student who has mastered this attribute should be able to keep track of what the question is asking, pay attention to detail, ignore irrelevant information, and identify constraints and use them in problem solving activities.
P 10 (Quantitative and Logical Reading): A student who has mastered this attribute should be able to: read complex, long sentences, and/or sentences with negation; use of terms such as “at least”, “at most”, “equivalent”, “must be”, “could be”, etc.; and should be able to deal with relations of
increasing and decreasing quantities and comparisons. Also, understand and use logical quantifiers like “for every”, “for any”, “for a given” or “there exists”. Note that relatively easy and everyday terms (for 8 th^ graders) should be viewed as trivial and not coded, for example: “more”, “smallest”, “greater than”, ”equal”, “add”, “subtract”, “multiply”. Also, terms that are specialized to one particular content area, such as “isosceles” (geometry), “fair” (probability), “rounded to the nearest..” (measurement), would not be coded, because the terminology is considered inherent in the content knowledge.
S1 Unit conversion S2 Apply number properties and relationships; number sense/number line S3 Using provided figures, tables, charts and graphs S4 Approximation/estimation S5 Evaluating/verifying/checking solutions S6 Recognizing serial patterns S7 Proportional reasoning S10 Open-ended items S11 Reading/understanding problems with complex text or real-world context
S 1 (Unit Conversion): A student who has mastered this attribute should be able to convert units to another scale or unit, and compare quantities expressed in different units.
S 2 (Apply Number Properties and Relationships/Number Sense): This attribute is coded in the presence of two somewhat distinct subskills. The first subskill involves procedural manipulations and reasoning using knowledge of formal number properties (e.g. that a negative number squared gives a positive result). The second subskill involves using “number sense”; i.e., make experience-based judgments about relationships among numbers and quantities. The prototypical skill here is reasoning about numbers, comparing magnitudes, and other skills that require use of a mental “number line”, or equivalent reasoning (including working with an explicitly given number line).
S 3 (Using Provided Figures, Tables, Charts and Graphs) A student who has mastered this attribute should be able to: follow written instructions that involve tables, figures, and graphs; comprehend the information contained in tables, figures and graphs, and use such information in answering questions.
S 4 (Approximation/Estimation): These items require the students to solve problems by estimating or approximating numeric values (involving integers, reals, or fractions), or approximating
In this section, several of the publicly released TIMSS-R (1999) 8th^ -grade (“population 2”) math items are displayed, along with their coding in terms of the TCD attributes described in this manual. The approximate difficulty of each item is (inversely) indicated by its observed mean proportion correct in the international sample (reported in parentheses). Brief explanations are also given as to why each attribute was coded for an item.
Example 1
Item T01. A club has 86 members, and there are 14 more girls than boys. How many boys and how many girls are members of the club? Answer: _______________
Attributes involved: C1, C3, S10, S11, P1, P2, P4, P9, P10 (p=.33 )
Question and answer involve integer values, operations ------------------------> C Most common strategy is to set up an equation involving x --------------------> C Item type is open-ended (short answer) --------------------------------------------> S Traditional word problem (w/ moderate complexity) ----------------------------> S Algebraic equation must be developed to represent situation -------------------> P Computations on integer expressions are needed to solve expressions --------> P Must solve equation for x ------------------------------------------------------------> P Need subgoal to represent both quantities with a single variable, or use two equations and two unknowns -------------------------------------------------> P Quantitative reading: phrases such as “more”, “less”, etc. ----------------------> P
Example 2 Item D12.
What is the best estimate of the number corresponding to P?
Attributes involved: C2, S2, S3, S4, S5 (p= .68)
Explanation for coding: Answers are real numbers expressed as mixed decimals ---------------------------> C Visual understanding of real number line is necessary ------------------------------> S Problem is presented via a graph of the real number line ---------------------------> S Problem explicitly asks for estimation of a real quantity ----------------------------> S Examinee is asked to examine multiple options to pick the best one --------------> S
Example 3
Item J11. Of the following, which is NOT true for all rectangles? A. The opposite sides are parallel. B. The opposite sides are equal. C. All angles are right angles. D. The diagonals are equal. E. The diagonals are perpendicular.
Attributes involved: C4, S5, P3, P10 (p= .54)
Explanation for coding: Question requires knowledge of geometry concepts --------------------------------> C Finding best answer requires checking all options ----------------------------------> S Judgment is required to find which statements are generally true -----------------> P Negation in statement of problem adds complexity ---------------------------------> P
Example 4
Item F10.
Using a centimeter ruler like this one, you can measure accurately to the nearest A. millimeter B. half-millimeter C. centimeter D. half-centimeter
Example 6
Item R15. John sold 60 magazines and Mark sold 80 magazines. The magazines were all sold for the same price. The total amount of money received for the magazines was $700. How much money did Mark receive? Answer: _______________
Strategy A: ARITHMETIC SOLUTION METHOD Attributes involved: C1, S10, S11, P1, P2, P9 (p=.44)
Explanation for coding: Stated quantities in problem and goal are integers --------------------------------> C Open-ended item -----------------------------------------------------------------------> S Problem cover story involves real-world story, fairly complex ------------------> S Solver must formulate math expressions to solve ----------------------------------> P Solver must evaluate expressions and do basic computations to solve ---------> P Problem has two subgoals: find cost per magazine, then find Mark’s amt -----> P
Strategy B: ALGEBRAIC SOLUTION METHOD Attributes involved: C1, C3, S10, S11, P1, P4, P5 (p=.44)
Explanation for coding: Stated quantities in problem and goal are integers --------------------------------> C Algebra: Set up 2 equations w/ 2 unknowns (= # books sold by each) ---------> C Open-ended item -----------------------------------------------------------------------> S Problem cover story involves real-world story, fairly complex ------------------> S Solver must formulate equations and expressions to solve -----------------------> P Student must know how to solve 2 equations in 2 unknowns --------------------> P Some logical inference is required to turn given info into equations ------------> P
Example 7
Item B08. If there are 300 calories in 100 g of a certain food, how many calories are there in a 30 g portion of this food? A. 90 B. 100 C. 900 D. 1000 E. 9000
