- Traditional individualized standardized measures are specifically designed to discriminate among students and tend to blame poor achievement on intrinsic characteristics of the child. - Research also suggests that these methods are not effective. They posit that competence and performance are also context bound and that different situations and environments influence performance. - After school settings offer an alternative to typical school settings and have the potential to support and enhance school-based knowledge as well. However, they need to change the environment completely and not just replicate school type interactions. - Education researchers have been tasked with accountability and they have been trying to record events, activities and participation in substantive ways but traditional methods don't work well in alternative settings. Unfortunately there is an emphasis on more traditional experiment designs. - One major issue with traditional approaches to documenting student achievement is that the outcomes are thought to reside exclusively within an individual. - The other issue is that is looks at achievement in the form of product scores and not the circumstances that surround these outcomes. - Finally, traditional methods of documentation assume that all participation is equal. - This paper describes the method used in one after school project called the Fifth Dimension. They specifically look at four locations in Southern California. - Fifth Dimension is pretty loosely organised and usually involves a university partner, a community partner and the larger volunteer community called DLC. They work towards cognitive and social development among participants while also providing undergraduate students of psychology and education to connect theory and practice. - Activity is a mix of play and education and children must come voluntarily. Children are also given a substantial amount of agency and there is no connection to their traditional school. Adults work with children as co-participants and children are encouraged to describe, through written and oral language how they're doing things. - Activity logs of children is maintained and their progress is typically indicated by their position within the maze. - Fifth Dimension was deliberately chosen to take place outside of school because this period of time for children is unsupervised and unproductive time. - Each centre in the Fifth Dimension was different and reflected the varied research interests of the facilitator. - Data sources for the study - facilitator applications, facilitator surveys before joining, facilitator field notes, interviews, research team field notes language assessment scales and site profiles. The language assessment scales was embedded in a computer based game - Ace Reporter Reading and Writing. - Methodological Issues - Framework - Finding a framework that takes all levels of analysis into account not just individual or a particular group - Insider vs Outside - Though evaluation is often scene as an outsider job that perspective is often inadequate - Product vs Process - There is a push towards product based approaches and quantitative data - Comparing Sites - There's no way to compare sites in such a study since each community and site has significant differences - Evaluation Design - Traditional approaches did not really work at all since every space was so different - Intrusiveness - Data collection and requiring students to complete tasks was often disruptive. It also took away from agency of the children. - To counter these issues we must use a multi-method, multilevel analysis approach based on sociocultural theory. - To effectively document and study we need to consider different planes - individual, interpersonal and community. Only looking at data and sources that capture all planes will give a clear picture. - The perspectives of insiders, outsiders and those that don't neatly fit into each bucket must all be considered in settings like this. Just looking at one type of perspective doesn't really help. - The site, the facilitator and the culture itself is constantly changing and dynamic. Any documentation or reflection needs to keep this in mind. Only by collecting various types of data can effective understanding occur. - This paper opted for a case study approach to understand each learning site. There was no way for them to compare different sites at all and they felt that generating a site profile and understanding each site separately would give them flexibility while still using common data collection strategies. - Data collection tasks were weaved into daily activities and routines of facilitator and learner. This made it much less intrusive. For example they embedded the testing task in a maze that students would already use. --- Gallego, Margaret A., Robert Rueda, and Luis C. Moll. "Multilevel approaches to documenting change: Challenges in community-based educational research." _Teachers College Record_ 107.10 (2005): 2299-2325. - [Link](https://lchcautobio.ucsd.edu/wp-content/uploads/2015/10/Gallego-Rueda-Moll-2005-Multilevel-Approaches-to-Documenting-Change.pdf)