It's been about a year since we started conceptualising Ira Project. I find myself at an interesting headspace at this time, and I'm writing this to honestly and sincerely reflect on whether we're making progress or going around in circles. I've documented fairly extensively what we've been up to the last year and I feel like it's a good time to look back and document learnings and see where we are. The YC applications in particular are quite helpful here since they are a window into our thinking and a snapshot of where we are at that particular point in time. ### The Beginnings When we started, we had an idea that we could leverage the Feynman technique and help students learn better by building a teachable AI. The first version of the product started with this narrow focus, with a single probability activity and an AI that was simply a creative prompt to ChatGPT. ![[Ira v1.png]] We did a round of user testing and conducted teacher interviews before showing them the product. The user testing was quite interesting and though the AI didn't work as intended, we saw the students actually exercising metacognition. #### Learnings - We needed to improve the AI so it doesn't frustrate students. Simple GPT wrapper isn't working well enough. - Particularly, it was really hard to get the AI to compute wrong answers. The AI would often correct the user or simply just return correct. - The UI was incredibly confusing for students. They tended towards typing the answer and didn't focus on conceptual understanding. - For us to start looking for customers, we realised that we had to build it in a way for teachers to share assignments with their students and some mechanism to handle classroom management. ### V2 The second version of Ira Project involved cleaning up the UI and building a classroom management system. To allow us to robustly test our AI for every topic we gave teachers a form to upload a lesson plan post which we could take 48 hours and give them an assignment to try out. ![[Ira v2 Classroom.png]] ![[Ira v2.png]] To improve the AI engine, we built an underlying concept graph and had the AI go through the list of concepts. We showed users the concept graph as a way to measure progress. From a sales standpoint, we attempted to sell this version of the product to schools, particularly those teachers teaching Algebra I. We also reached out to Technology Coordinators and School Administrators predominantly on Email and LinkedIn. We also got into this whole cycle of selling to other tools as an API integration. The thesis was that we can predict much better than other platforms how well students understand concepts, and this could help other tools deliver personalised content. Think something like recommender as a service. #### Learnings - Everyone we talked to found the idea compelling and pretty cool, but they didn't actually end up trying the platform out. - There were some issues with having students login to the platform and some privacy concerns that one teacher had. - The concept graph allowed us to catch when students had or had not included some concepts. But computing wrong answers was still a shot in the dark. Particularly for a wrong formula, the AI would often fail quite poorly. - The concept graph was good as a progress indicator, but students still focused on putting in specific answers. - UI was still a concern. Many teachers and students were confused about answering all questions at the same time. - Note: We were and still are particularly stubborn about this, since we want to have students focus on conceptual understanding rather than procedures. - Two types of sales processes - Bottoms up - Teachers using the product and then recommending it to their schools - Top down - Technology coordinators are responsible for deciding the technology, more prevalent at the beginning of the school year - Teachers hesitated to assign a product that wasn't fully fleshed out. The time to 'aha' moment was too long and required a lot of work and trust to upload a lesson plan. - Having only one math topic meant we are only able to go after one type of teacher. To handle other subjects, we need to process images as well. ### V3 When we got a hit from a couple of Physics teachers who wanted to use the platform, we decided that we would spend time building the application for Physics. We worked on making our AI handle images and also move towards a reasoning engine approach where we would use a combination of a decision tree, LLM and a wolfram alpha-esque calculator. We also moved to a formula bar so the conceptual part and the calculation part could be separated out. The idea was that we would be able to handle wrong calculations in this manner. ![[Ira v3 Classroom.png]] ![[Ira v3.png]] With a more fleshed out platform that was aligned to the curriculum, we were able to get more meaningful engagement on the platform. Teachers were able to experience the 'aha' moment a little faster and were able to try out activities that were aligned to their curriculum. The first pilot we had completely ghosted us after we built out the activity for him, and we're still not sure why. The second pilot classroom was the first time we had a set of students using our tool. The students quickly got frustrated with the AI, and they uploaded entire wikipedia pages. This AI took far too long to respond and it completely jammed the system. Students experienced a lot of errors and became annoyed. There were a few positives though, and we saw some students go from very rudimentary explanations to more complex ones as they got feedback from the AI. #### Learnings - Curriculum alignment is critical and to build an EdTech product for teachers we must be aware and in sync with what they are teaching in the classrooms. - Once the school year starts, teachers don't really have a lot of time. Response times are really slow, and it takes a while before teachers are able to act on trying out something. - Administrators and technology coordinators don't respond at all once the school year starts. Perhaps the buying cycle is over for them. The only responses are from teachers. - IB curriculum might be the most conducive for trying out the Learn by Teaching activity. There's an emphasis on conceptual understanding. - Lots of teachers are worried that their students might use AI in their assessments, but these teachers aren't necessarily the same ones adopting new technology. - LinkedIn has been somewhat helpful to reach out to teachers, particularly those in international schools. - Emails are not effective at all for public schools. Maybe deliverability is an issue. - In terms of email sequences, it's the free trial and immediate usage email that gets teachers. Sequences absolutely destroy deliverability, though, so perhaps best to just leading with try it out is better. - Learn by teaching is too removed from the curriculum and their school work. Students and teachers are less inclined to use a platform that doesn't help them do better in exams and their school work. - Adding one more platform for teachers is a lot of work for the marginal utility that "Learn by Teaching" gives. Teachers already have a lot of platforms they work with. - The UI is still bare bones. Teachers and friends were hesitant to share it with their students and colleagues, since the app still felt like a prototype. ### V4 We did something very atypical of the startup advice out there. We spent time building a full UI and design overhaul. With the ShadCn component library and cursor, this happened quite quickly, and we were able to build a polished looking product. We took sometime to break down the kind of skills we want to build in students and how we can create well rounded learning. We studied other apps in the market to see the activities they were offering. We added four different activity types - Knowledge Zap, Step Solve, Reason Trace and Learn by Teaching. ![[Ira v4 Classroom.png]] ![[Ira v4 Knowledge Zap.png]] ![[Ira v4 Learn by Teaching.png]] We built out about 10 activities across 5 topics and gave teachers the ability to request topics like before. We did make sure that we had a sense of most of the curriculums though to ensure that the topics teachers were currently teaching were catered to. Our LinkedIn and Emails started getting a lot more hits. We had some semblance of a sales motion where teachers would go from the landing page to the application and try out an activity immediately. IB schools in particular seemed to be interested in using the platform, and we started to get initial traction. We hit the 100 student landmark on our platform. #### Learnings - Though we were offering four activity types, it was the more traditional assignments that teachers started with. Teachers wanted to try the others and were intrigued by the idea but only a few ended up assigning them. - Students were much happier to do the traditional activities. They often found the new activity types too challenging but teachers mentioned that this challenge was good. - There was some friction moving away from existing platforms. Many teachers immediately checked out the traditional activities but didn't find them superior. The main competitors seemed to be textbook style platforms that had embedded learning content. - The school calendar is really hard to predict and work around. We ended up being behind the eight-ball quite often with respect to creating activities for topics teachers wanted. When an activity wasn't there for the topic they needed, we tended to lose the lead. - Teachers tended to be quite forgiving of bugs. Though we had shipped bugs a few times, teachers were willing to work with us as we fixed the issues. - It's very easy to build UI and basic components so there's no reason for us to give rudimentary versions of those. Iteration and evolution is more applicable to the learning activities themselves. ### V5 While we had a thesis on why students need to engage with the topic in different ways and different activities we had not shared that with teachers and students. The first thing we did was to make that more visible in the form of a skills graph where students and teachers could see how different activities build different skills. We spent time to build out an AI engine that can build activities for any topic. We were able to successfully generate assignments for two activity types and a clear path on how to do the others as well. We generated a library of activities for the entire IB Physics curriculum so teachers could use them as they pleased. We added two more activity types. Concept Mapping came from a teacher we worked with. Read and Relay was conceptualised to build information literacy and to include some content as well on the platform that other platforms were offering. We also quickly spun up a Leaderboard feature for one teacher that we were working with. ![[Ira v5.png]] ![[Ira v5 Read and Relay.png]] For our AI engine, we had built out a concept graph that captured all the topics in Physics. We made this visible to students and teachers as well so they could visually see their progress. This was our attempt at finding the balance between gamification and not. Given the exam season was coming up we added a revision button for students to use and practice concepts. These would leverage the concept graph to show them question variants based on their previous performance. #### 5.1 We realised that with our underlying concept graph, we could build a lot more than just learning activities. We experimented with generating presentations, worksheets and found that our approach gives us quite reasonable results with not much effort. We decided to build these out so we can save the teachers some time. We have heard of many teacher tool platforms find success. Given our focus on a particular subject we are able to produce better quality results than these other tools. We have moved back to sales and talking more to administrators and technology coordinators. We also started to check in on Facebook as a channel to get users apart from LinkedIn and Email. #### Learnings - Students preferred more and tried and tested approaches for their exams. There was a fear of the unknown with our platform and the stakes were too high. We found little to no usage on the revision modules. It did not help that their progress throughout the year was captured in other platforms. - The behavioural nudge that we expected to get from the skills and concepts never had the chance to play out at all. The end of the year is a hectic time with a lot of pressure. - The end of the school year is an odd time to add new tools or to do sales. Response times have been quite slow from teachers. Perhaps it's time to go back to administrators. - Teachers are extremely swamped. If we want them to invest time and effort into adding a new type of learning activity and skill then we need to find a way to save them some time. - Somewhere along the way we have lost the novelty and excitement and the application feels more typical and similar to other applications. ### The Future We need to find a way to get students to want to do the harder things. Rather than minor nudges, we want to make all the learning activities combine to something meaningful. Moving students from consumption to creation and giving them a project to show off could make things interesting. One of these projects could be writing a paper. The "Learn by Teaching" activity would serve as the background, the "Read and Relay" would serve as citations and we could include an "Experiment Design" and "Data Analysis" activity. Another option would be to create a game. The "Step Solve" problems could serve as the underlying Physics models for the game.