Welcome to Step 5: Analyzing Evidence of Learning
|
Teams keeping project logs should record the following information:
|
|
Teams keeping project logs should record the following information:
|
Carmen Wilson taught the construct validity research lesson Thursday, October 14, 2004 in Psychology 451/551: Psychological Measurement. Rob Dixon observed and two guest observers from the Psychology Department were Tracie Blumentritt and Susan Wabaunsee.
Summarize the evidence, identifying major patterns and tendencies in student performance.
The evidence of student learning we collected consisted of:
• Students’ written work: 1) definitions of depression, 2) test items to measure depression, 3) and descriptions of studies to determine test validity.
• Observations of students during the lesson. Three instructors observed students, and a videographer filmed the lesson.
• Minute paper. Students completed a three question “minute paper” at the end of class (what was most difficult; what was the most important thing you learned; what is still confusing).
The comments below identify patterns for some students and some groups and “problems” observed in one or more groups. The problems may not typify the entire class but it is important to make note of them when we revise the lesson.
In preparation for the class, each student developed a definition of depression. Students who were familiar with the concept from other classes wrote well developed definitions.
At the start of the lesson, students were in small groups and asked to compare their ideas and formulate a single group definition. Students were familiar with the concept of depression from other classes and were able to produce a lot of characteristics.
• Problem: Familiarity with the topic of depression had a dual effect. It was easy for groups to generate a list of characteristics about the concept, but some got bogged down in minute details about depression. (e.g., talking about etiology and whether it is always triggered by a traumatic event).
Based on their group definition, students created several test items to measure depression.
• Problem: Some groups got bogged down in framing/posing questions effectively.
• Problem: Some groups got bogged down on what statistical test to use.
• Problem: Little or no discussion about the nature of a “construct.” Were students aware that they had created a construct?
Groups then designed two studies to determine whether the test they had just created actually works.
• Problem: Some groups got caught up in technical details of doing a study (e.g., how to word the questions properly, how to get subjects to participate) rather than the logic of the research design. Were students aware of the real point of the lesson—to test validity?
• Problem: In some cases, there was no evidence from the group discussions that students understood the logic of determining test validity. It’s not clear whether students understood it and therefore had no need to discuss it or whether they really did not understand it clearly.
Based on their worksheets:
• 6 of 9 groups described one study that was acceptable
• 6 of 9 groups wrote a tangled description of at least one study. In these cases we could not determine what the students understood about validity.
Last segment of the lesson. Groups were given a handout that described two tests (one of depression and one of academic skills), and asked to use these to determine the validity of their test. Groups were perplexed about whether the academic skills test was related to depression or not.
• 5 groups wrote an acceptable description of a divergent validity study and two others seemed to be on the right track but their answers were tangled.
Describe major findings and conclusions about what, how and why students met or did not meet learning goals.
The concept of “construct” was invisible during the lesson. There is no evidence that students are aware of the idea and how it relates to the test they developed.
Based on observations of groups and their written descriptions:
1. Most groups designed at least one acceptable study to determine test validity, which suggests they “get” the logic at some level.
2. Many group written responses were tangled and incomplete. We can’t tell whether they didn’t get it or just can’t talk about it or are just careless.
3. Groups got sidetracked in the details of research design and data analysis
4. There were multiple examples in which the term “correlation” was used or invoked incorrectly. In some cases they used it generically (e.g., “correlate scores”) without indicating which scores and in some cases they were just wrong (e.g., “correlate the scores of two groups;” or they used negative correlation when they meant no or low correlation).
5. There is a group dynamics issue. Some students are not engaged, some groups are not effective.
6. The results are actually pretty good considering students have not read about or been taught about construct validity.
Based on your analysis how will you change the lesson?
1. Stay with the overall structure of the lesson
2. Have students define depression and develop test items outside of class
3. Pick a different type of test for the divergent study (e.g., math facts) so students are less likely to anticipate a connection to depression
4. More analysis at the end of class to compare, analyze and discuss their studies and connect them to the major concepts of the lesson
5. Try to de-emphasize statistics and focus more on “evidence” and general descriptions of the studies.
Contact: Bill Cerbin
Previous Logs: Step 1 | Step 2 | Step 3 | Step 4
For Logic Proofs:
We noticed that many of the students were alert even for an 8:50 class. We noticed that many of the student pairs worked well together, and seem geniunely committed to the tasks.
There was perhaps too much material to cover in a 50 minute course. This lesson might work better if steps 1-3 were already mastered.
We might change the name of one component of the lesson--to pragmatic questions as opposed to kinds...The space of the clssroom might be more fully utilized for the small group work.
Overall the class was very successful. Nearly all of the students were engaged in active learning.
On Doing Phenomenology:
All of the students participated in the process. Professor Maly commented on how the camera disrupted the natural flow of the class more so than in the logic class. We covered too much material for one class period. So, one change would be in the amount of material or the time it takes to cover it. We noticed a gap between the students being able to experience the phenemena and their ability to articulate it using the technical jargon. We wondered how important being able to use the jargon is as compared to noticing what is going on. We decided that although using the jargon is important for further discussions in philosophy at this point it is even more important to focus on the phenomena and noticing that.
The aspects that could be changed were: more use of board, less material, and no camera.
The strategies with the jug and the experiential experiences were quite effective.
Contacts: kramer.eric@uwlax.edu, maly.kenn@uwlax.edu, ross.sher@uwlax.edu
Previous Logs: 1 | 2 | 3 | 4
1. Summarize the evidence, identifying major patterns and tendencies in student performance.
Students were very engaged in the trade experiment and identified the main ideas of the lesson.
2. Describe major findings and conclusions about what, how and why students met or did not meet learning goals.
The trade experiment was a great success. Students got involved and understood the gains from trade as well as the costs to subsidies and taxes. Students realized that they could trade just the subsidized goods, and make a profit while generating no real gains to trade. Furthermore, one student recognized that everyone pays for subsidies. What the students did not get, however, from the experiment were the losses associated with free trade.
Students were more likely to participate in the experiment and in the case studies. However, during the case study, only a few students made comments to the entire class. In other words, the case studies were not conducted in a manner that got the entire class involved. Each group seemed to have a “leader” who would initiate dialogue and free riding occurred in larger groups.
As the instructor “roamed” about the room, students were more likely to focus.
3. Based on your analysis how will you change the lesson?
Experiment
The instructors ad hoc promptings during the trade experiment were good, but need to be formalized so that they can be reproduced.
The order of the trades should be changed: do most restrictive first, free trade last so trading options are not exhausted. When they heard what others offered as a price, it influenced their answers. It would be interesting to not only query students who traded about the value of their post-trade basket but also students who chose not to trade.
Publicly recording values influences the outcome. Students might not feel comfortable or are less honest about the value of their basket.
There was too much lecturing during the experiment. For example, after the first round of trade there was a brief lecture on why there was no increase in value for some, etc. Students seemed confused and that was a big leap in the lesson. Limit lecturing and have students see what lessons they learned in the end. Between the lecturing and the students copying each other, we could conclude that the students weren’t figuring out things for themselves.
Case Studies
During the case studies, the groups of students need to be smaller (4 students maximum) in order to encourage more participation by more students.
The reading was too difficult and should have been performed outside of class in order to not waste time in class. There needs to be a reading assignment prior to the case study day that holds the students accountable for the readings. Additionally, on the case day, there should be some sort of questions or assignments to keep students accountable for the material for the day.
The Wal-Mart case generated the most discussion.
Pre and Post-Test
Exhibit 5: Potatoes and wheat output (tons per day)
Country Potatoes Wheat
United States 4 2
Ireland 3 1
1. In Exhibit 5, the United States has an absolute advantage in producing:
a. potatoes.
b. wheat.
c. both wheat and potatoes.
d. neither wheat nor potatoes.
2. In Exhibit 5, the opportunity cost of wheat is:
a. 1/2 ton of potatoes in the United States and 1/3 ton of potatoes in Ireland.
b. 2 tons of potatoes in the United States and 3 tons of potatoes in Ireland.
c. 2 tons of potatoes in the United States and 1 ton of potatoes in Ireland.
d. 4 tons of potatoes in the United States and 3 tons of potatoes in Ireland.
3. If the countries in Exhibit 5 follow the principle of comparative advantage, the United States should
a. buy all of its potatoes from Ireland.
b. buy all of its wheat from Ireland.
c. buy all of its potatoes and wheat from Ireland.
d. produce both potatoes and wheat and not trade with Ireland.
4. What terms of trade would be beneficial to both countries in Exhibit 5?
a. 1 ton of wheat trades for 3 tons of potatoes.
b. 1 ton of wheat trades for 2 tons of potatoes.
c. 1 ton of wheat trades for 1½ tons of potatoes.
d. 1 ton of wheat trades for 2⅓ tons of potatoes.
5. Which of the following is true when countries specialize according to their comparative advantage, and then trade?
a. It is possible to increase world output of all goods traded.
b. No one will be harmed by international trade.
c. One country is likely to gain from trade, while others will lose.
d. It is possible to increase output of all products in each county.
6. When economists illustrate gains from specialization and trade using a production possibilities model, what assumptions are made?
a. Opportunity costs are different between producers.
b. All land, labor and capital is fully employed when producers specialize and trade.
c. Producers are able to negotiate terms of trade that are beneficial to both.
d. All of the above
e. Only a. and c. are correct.
Contact: Lisa Giddings
Previous Logs: 1 | 2 | 3 | 4
Recent Comments