Course Redesign
The Problem
A 20 year old interdisciplinary science online course core earns 50% completion rate. This means that half of the users, after being given 6 months, are not successful in passing the courses.
Many users take 18 months (three 6-month terms) to be successful, although some users have taken years to be successful.
Users become disheartened and exhausted.
The old course core causes overall university drops as users fail and lose the goal of graduating.
Content is split into 2 courses: a lab course (4 units) and lecture course (4 units). The 8 total units however, is deceiving because the two halves do not take the same amount of user time. Users regularly spent 10-14 weeks on the lecture content (physics, chemistry, earth science, biology, & the scientific method) and then 1 week on the lab (biology-leaning content).
Screen capture above (dated 2011) shows the original online course landing page on a supplementary Blackboard-like platform from Pearson. The resource contained an outdated PDF version of the 600+ page textbook and low level quizzes. The overall design aesthetic of the platform is Internet circa 1998.
This case study page focuses on the successful redevelopment that occurred between December 2016 and July 2017.
This was a work project. As such, the internal course details are proprietary. This summary explains my role, my decisions, and the project results.
User Analysis
The average users are:
Age range: 16- 85, average user age is 41.
Married with 3 kids.
Works full time in addition to attending college full time.
Both male and female.
70% first-generation college.
Last formal educational experience was more than 10 years prior.
Wide range of cultures from across the United States.
Freshmen/sophomore-level taking General Education/Liberal Arts science courses to meet core requirements.
Users can type and click on a computer and can complete simple web searches.
Time management problems: Users were often squeezing studying into an already full life with job & family.
Lack of adequate study skills because students often utilized only one way to learn, by reading (no note taking, no self-quizzing).
Additional user characteristics as redevelopment time arrived:
Strong demand for mobile access.
Americans with Disabilities Act requirements to make resources accessible (alt choices for click & drag or drop down menus, and closed-captioned videos.).
Stakeholders
Standards:
At this time of redevelopment, NCATE was ceding standards to CAPE but the science subcommittee had not yet provided a report. Therefore, we used the older NCATE standards but left out interstellar space science as it was a place of low student scores previously: we determined to cover solar system space but stopped at planets and seasons and did not explain extra-solar space (galaxies & black holes).
Intercollege requests:
Teachers College needs basic biology as prereq for psychology.
Nursing College needs biology as prereq for anatomy & physiology.
Business College needs physical science to foster an understanding of the business applications of global warming.
IT college needs physical science basics of electricity and magnetism to understand electrical currents, solid state hard drives, and quantum computing.
Constraints
Development was a concurrent goal to all other institutional goals. This means that designers and researchers' primary support is dedicated to our users and development fit into spare work time.
Development total time was 7 months (December to July) for 3 courses. The last comparable development took 12 months.
Vendors who can work with competency-based content are rare.
Content standards are in flux.
In order to both speed development and assure a quality result, all stakeholders needed to work from the same set of expectations. A solid scoping document was prepared over 6 weeks which proved invaluable at keeping all departments aligned.
My Role
In addition to having conducted thousands of hours of user interviews to both understand the product failings, I was also the leader of the design & research team that was in charge of the course content. I worked collaboratively with multiple teams representing user interests.
The Team
A course redevelopment of this size required the input of many departments across the organization.
Primarily, the Product Development Department takes lead and assigns a Project Manager role.
Supporting roles include vendor-relations, instructional design, editing, and quality review.
The Assessment Department covers the high-stakes measurement of final competency, including all item-creation, assessment administration, scoring, and any psychometric analysis.
The Mentoring Department works directly with the users and contributes at all stages with Designers, Researchers, Content Experts.
In this triad model, each department is a check on the other two departments; able to question, collaborate, design, and problem-solve at any point in the development. The more the departments work together, the more successful the product.
1st Decision: Break the content into smaller chunks.
Rather than separating the work by type of work (lab versus lecture), the first innovative decision came from hours of user interviews: split the content, in this case biology from physical science. This would allow users to engage in the content in two separate attempts that could be spaced sequentially over time. This also reduced the number of items on the final assessment from one exam of 188 questions to two exams of 70 questions. The mental stress and burden on the user is lightened because they do not need to feel like they need to know it all in one four-hour long assessment event.
In order to accomplish this, the biology content could go deeper to make more conceptual connections but the physical science content had to be decreased in coverage from the older design.
My specific research request was to liaison with the Registrar's office to determine what problems exist with the current design. The interview showed that their problems centered on the lab course and applying equivalent incoming transfer credit. Often when a student was transferring credits in, they would receive any passing science course credit applied to the lab first (which was easier) and would still have to complete the more laborious lecture portion, regardless of what that content actually was. Therefore, it was common for a user to have passed a biology lab and get lab transfer credit but still have to complete the larger combination physics, chemistry, earth science, and biology lecture course. This was demoralizing for the user before they even started engaging with the institution. It created a bitter user outlook.
This solidified our decision to split the content and then break out the lab as a further seperated third course:
Original 2 course design: lab and lecture covering all sciences.
New 3 course design: biology, physical science, general lab.
We found a publishing vendor, Acrobatiq, that was willing to work with our competency-based model which means we needed courses that could open and run without end. Updates would always be completed in the middle of user academic work because the course does not shut down for maintenance. In addition, Acrobatiq is one of the first vendors to offer data analytics and could constantly provide feedback to the development team in the form of heat maps.
The Biology course already existed from our vendor, Acrobatiq.
This Physical Science course did not exist! We needed to make it from scratch.
Biology
Focus on 3 main concepts:
The Cell
Evolution
The Environment
Cross-connections between these concepts (for example, the driving role that genetics plays in evolution) could be emphasized.
Integrated Physical Science
Spend only 2 weeks worth of content (30 hours) in each:
Physics
Cover gravity, magnitism, electricity, and forces
Chemistry
Cover atoms and basic reactions
Earth Science
Cover common cycles, dynamic earth processes, cause of seasons & climate change.
Because more content was covered versus the biology course, the content was brief but built to be relevant (i.e. the impact of global warming for IT network users).
2nd Decision: Work with 3 experts at once.
The initial biology development proceeded smoothly because the course already existed. Our research and design team only had to edit and customize the content to our objectives. Development took about 4 months, December 2016 to March 2017.
The physical science course development started in April 2017 with a goal to launch by July 1, 2017. Each of my designer/researchers was an expert in one course area (physics, chemistry, or earth science).
Because the publishing vendor Acrobatiq was able to move quickly, we needed to review multiple modules at the same time.
I proposed and we conducted a round robin-style review where the primary expert did the most amount of review in the most amount of time. Then a secondary review was conducted at a lighter level with less time. Then a final review was completed with very little time just checking the first two designer/researchers work.
Review #1: Expert. Every detail checked. 9 days allotted. Extreme high quality expected. Designer/researcher prioritized this work over other job duties.
Review #2: Intermediate. Review of all content, but not expected to catch every detail. 4 days allotted. Work fit into existing schedule.
Review #3: Check Review. Not checking content directly, but checking the work of Review #1 and 2 for logic and validity. 2 days allotted. Work fit into existing schedule.
Moving through 3 content areas in this rotating way meant that the expert working on review #1 for physics was free to do review #2 for chemistry immediately after finishing their review #1. This meant that instead of waiting for each content area to be addressed separately, we could reduce the total time by switching the burden to match the expert's strength.
Because of this approach, we were able to spread the workload so that no one designer was carrying a heavy load during this three month project. Designer/researchers were also able to complete other projects as assigned. Finally, the vendor was able to turn around edits for us very quickly and we could finish multiple parts of the course during the same time period. This became the fastest course redevelopment project that I was part of in 14 years.
3rd Decision: Add interactive simulations to convey abstract concepts.
PhET Resources
Content in this development was heavy on the abstract side. This means that users cannot easily see with their eyes about what they were learning; they had to imagine the unseen world of forces and atoms.
I advocated for the inclusion of a large amount of online PhET simulations. Unlike biological concepts which can often be seen by the eye (birds in trees for an ecosystem, bones in the hand for form follows function, and children looking like their parents for genetics), physical science concepts dominate the unseen world. Abstract physical concepts like forces, motion, inertia, atoms, and waves are hard to imagine with any visual cues. Simulations can show concepts, mathematically, graphically, and with the manipulation of time that were previously difficult to visualize. PhET simulations free to include in educational resources with minimal contracting burden.
Prior to launch, I demonstrated a few simulations to C-Suite level stakeholders. Feedback from this group was very positive. Comments were that the material was made accessible and even delightful. Getting internal stakeholders excited about a project helps the product gain a positive reputation even before launch. Department heads particularly spurred our sales and marketing agents to push the new courses as positive choice to our users.
Soon after launch, early user feedback highlighted the use of simulations (only present in the physical science course, not in the biology course) as a positive change from just reading text or watching videos.
Results
All three courses launched on time with a total of 7 months development, while maintaining service standards on other user-focused projects.
IT student users in particular earned a 95% completion rate in physical science, an increase of 45% over the prior course core.
The physical science course surged in popularity among users. The biology course struggled due to less interaction and relevance; stayed in maintenance cycles. Further planned fixes included adding in a greater amount of simulations.
The following are screen captures of the material from these courses that is publicaly available.
