Overall objectives: Development of a naturalistic learning assessment method that:

1. gives feedback on a dimension of learning of agreed professional importance to module developers and adapters, funders, and all interested colleagues
2. modular teachers can continue to use as a matter of good professional practice beyond the end of the coalitions' formal evaluation process.
3. opens up a dialogue about learning objectives between modular chemistry developers and their non-participating colleagues--whether in chemistry or in other disciplines--and that acts, thereby, as a catalyst for change among individual faculty and their departments.

Student Learning Retention: The issue of what students carry forward with them from earlier classes into subsequent classes--whether in the same discipline or others--is a commonly-agreed professional aim. However, it is rarely carried out or evaluated. Indeed, on many campuses, the ongoing professional conversations necessary for its achievement are not a normal feature of the institutional structure and culture, whether within or between departments.

From our ongoing interviews with faculty, it is clear that many of those who have taught at least one module are now looking more critically than at the outset at the fit between their learning objectives, their teaching, and their assessment practices. Some of this group are interested in collaborating in the development of campus-appropriate experiments in student learning retention, both for the purposes of the consortia evaluation, and for their own longer-term professional use. The last three years of our joint evaluation offers a unique opportunity to design, and to refine in working practice, a number of models that can be adapted by others.

Experimental Structure: During the present year we will aim to set up eight experiments (four in each consortium) at participating institutions of different type. Their design will vary according to the particular configuration of classes in each department. However, the central features of the design will be constant. A "giver" modular teacher will solicit the active collaboration of a colleague who is a "receiver" of students from one of their modular classes. They will meet with the evaluators to negotiate the following:

1. The timing and sequence of classes to be used in the experiment
2. The specific learning gains that the modular teacher wishes to see the students carry forward into the subsequent class (e.g., knowledge, understanding, skills, approach to learning etc.)
3. What specific or general learning attributes the receiver would like to see students bring with them from the modular class (and from other comparable preparatory classes).
4. Agreement on a limited set of specific learning attributes that one teacher wishes to give and the other to receive, and that are amenable to student assessment in both classes. In each case, assessments will be experienced by the students as a normal part of their class assessment pattern. The evaluators will encourage a balanced set of items to be tested that includes gains in understanding and skills. The number of items to be jointly tested should be approximately four to six.
5. Clarification of the points in time in the receiver class when the teacher begins to draw upon each of these items of student learning. Testing of the degree to which students have remembered and can use each agreed aspect of learning will be undertaken a few days after the receiver teacher begins to draw upon the students' presumed remembrance of each agreed item in class. (Testing for all agreed items at the start of the class is rejected because this could advantage students who had recently completed the "giver" class, compared with those who had taken it some time earlier.)
6. Development of student assessment methods for both the giver and receiver classes
7. A set of criteria by which to grade the performance of students in these assessments in both classes.
8. Agreement on a format for the comparison of the performance data from the two classes.

These matters will be negotiated over time, but the discussion will begin in a meeting with one of the evaluators who will help to clarify aspects of the process, the form, content, and conduct of the assessments to be used, and the analysis of the data generated. The evaluators will continue to act as consultants throughout the set-up and implementation of each experiment and will collate and analyze the data from each participating campus group.

The simplest, and perhaps most common, format for these experiments is likely to be giver-receiver pairs. However, there may be more than one giver teacher (e.g. where two or three faculty are teaching sections of the same modular class), and more than one receiver. Receiver teachers may be in chemistry, another science discipline, engineering, or a non-science discipline. Both giver and receiver may be modular teachers. Chains of three teachers may be formed, such that the first receiver (also a modular teacher) becomes a giver for a third level receiver. What matters is that variations arise solely from the expressed need of the campus group to know how effectively students carry forward particular aspects of their learning into other classes. Variations will also reflect the particular configuration and timing of classes in any department. By allowing these variations, the researchers expect to learn about their relative efficacy. One important outcome of the overall experiment will be the production of a set of models which can be shared with other participants who wish to adapt an experimental design for their own use.

The student performance data generated will be of immediate value to each campus group. They will also offer a collective overview of the efficacy of the modular approach in terms of student learning retention.  

We will begin the work with eight campus groups in the present year as part of the overall evaluation plan for the double consortia. As new experiments are set up, those already in existence will be continued, and may be elaborated to include new pairs, additional members, or additional classes, and faculty in other disciplines. By these means, the evaluators and participants expect to learn much about the merits and limitations of particular forms of the experiments. This information will be passed on to other faculty through our consortium network. 

The performance data generated by each campus will be used to compare the learning retention of students in prior modular classes with those who have experienced preparations of similar content and level in more traditional classes. The naturalistic experimental design has a number of advantages for this purpose: 

1. The design is campus-appropriate. The variation as what constitutes "traditional teaching" across the wide array of institutions in the consortia is not a problem: the comparison is always between modular teaching (which is referenced by a shared set of pedagogical ideas and strategies) and whatever is regarded as "traditional teaching" on each campus. In some departments in our consortia, the modular form is but one of a number of non-traditional teaching strategies, but the need to establish the efficacy of modular teaching is no less. 
2. Students will experience the assessments required for the experiment as part of the normal testing process for their class  
3. Faculty are not being compared--with the consequent possibility of damage to professional reputations and relationships. The focus is on the modular teachers' work, and the co-experimenter is offering the modular developer the opportunity to discover whether their new pedagogy allows students to do at least as well as students who have not been exposed to it. 
4. The method also has an important secondary function in encouraging collegial dialogue in an area of agreed professional value. Even if the experiments of some campus groups work less well than others, or the results are inconclusive, the method encourages the beginnings of collegial conversations about pedagogical issues which are absent in many departments. 
5. The method also has the potential to extend these conversations to faculty of other disciplines, to explore the transfer of learning gains from one discipline to another, and to improve the quality of that transfer. The evaluators also see its potential as a focus for the agenda of the Campus Advisory Groups, whose formation is required by the NSF, but which have, thus far, failed to flourish for want of activities of agreed common value.