05 Nov Project update: University of California, Irvine
OLOS Observation System
Over the last two years, the Early Learning Network’s assessment team at the University of California, Irvine (UCI), has been developing and testing a web-based observational tool for practitioners in pre-K through third grade classrooms to help improve student outcomes and strengthen student-teacher relationships.
Using electronic assessments and observations, the Optimizing Learning Opportunities for Students Observation System, known as OLOS, provides real-time feedback on children’s academic progress and teachers’ instructional practices. It also offers recommendations on how to best meet students’ unique learning needs.
The tool plays an important role in the Early Learning Network’s nationwide research studies because it addresses the need for a practitioner-friendly, comprehensive measure tapping classroom structures and processes, instructional quality and child responses in areas associated with early learning outcomes.
Below is a summary of the UCI team’s recent efforts.
Piloting the OLOS Observation System
After first developing and using OLOS in southern California pre-K and elementary schools, primarily serving ethnic/racial minorities, English Language Learners and children living in poverty, the team began working with other Early Learning Network teams to pilot-test and validate the tool.
Thus far, the UCI team has helped deploy OLOS in selected school districts in Ohio and Nebraska. They have been actively collecting data related to the tool’s reliability and training time, the feasibility of using it in classrooms in real-time and its predictive validity across the early grades.
The screenshot below shows how teachers can gather OLOS data simultaneously on three different students.
OLOS screenshot of student observations.
Ensuring Reliable Data
In June, the UCI team hosted a three-day training in partnership with other Early Learning Network teams. The assessment team trained project representatives from the University of Nebraska-Lincoln and Ohio State University on the OLOS tool, who will train new users on their research teams to reliably code teacher practices and student behaviors in their respective states.
A comprehensive reliability system has been developed that gives users the chance to practice and test their use of the measure with immediate feedback on their accuracy.
Collecting Data in Real-time
In the spring of 2018, 58 children were observed in 12 pre-K classrooms to determine the feasibility of using OLOS in real-time. Among these classrooms, multiple program types were represented, including Head Start, the California State Preschool Program, special education classrooms and tuition-based programs. Results demonstrated the potential for OLOS to capture differences in learning opportunities across various program types.
Currently, the UCI team is collecting data in approximately 40 classrooms, which includes assessment of language, literacy, math and self-regulation skills at three time points. Additionally, the team is conducting three live and video-recorded classroom observations across the school year.
Assessing Dual-Language Learners
In California, where Spanish is spoken by many students, the UCI team has been thoughtful about its protocol for measuring language and literacy abilities in children’s native language, as well as in English.
All children will receive a direct screening of their grammatical skills (BESOS screener, Peña et al., 2015) in both languages. Children who demonstrate either more than 50 percent accuracy on the Spanish screener, or a stronger performance in Spanish than in English, will receive a battery of Spanish language and literacy assessments, in addition to the English assessments every child will receive. Additionally, detailed information about children’s exposure to, use and proficiency in both languages will be collected via a parent questionnaire.
Next Steps
Although still in the pilot phase, OLOS shows promise of becoming the observation system conceptualized by the Institute of Education Sciences and presented in the UCI proposal.
In the coming year, the UCI team will begin assessing predictive validity and developing more practitioner-friendly ways to present OLOS data to help them better meet students’ learning needs.
They also plan to launch OLOS in the remaining research studies led by MDRC, the University of Virginia and the University of North Carolina at Chapel Hill.
Learn more about the UCI assessment team.