Year 2 Project Reflection
Reflections on, and lessons learnt, from the various project milestones and phases from September to December 2017.
Milestone 1: Finalising your proposal and schedule (Deadline: 26/09/17)
A lot of time and effort went into my research proposal during Year 1 of the MSc, and the initial proposal submitted on 29/06/17 was very positively received as evidenced in the ‘Research Proposal Marksheet’. Only minor changes were suggested, so I took the decision to submit an unchanged document as my final proposal. My plan was to incorporate these minor changes with any other changes required later in the Applied Project. This turned out to be a good plan, as the there were important changes required later in the year. Changes were necessitated by a new Junior Cycle Mathematics syllabus that was unexpectedly published in November 2017, the addition of three new key informants (NCCA, Project Maths, Adaptive Learning Expert) to improve the quality and variety of interviews, and the use of three different domain model authoring tools (NCCA, 2017). These changes could easily be incorporated into the initial proposal. I learnt that a solid research proposal, combined with a well thought out and realistic timetable (Table 1), were key components in the success of the overall Applied Project.
Milestone 2: Ethics Submission/Approval (Deadline: 31/10/17)
I began work on my Ethics Submission on 04/11/17 and made an online submission to DIT’s Research Ethics and Integrity Committee (REIC) on 14/11/17. I discovered that I had to carry out work on my project that I had planned for later in the year for this submission. Detailed information was required by REIC in relation to the survey and interview questions, and this is something I had not anticipated. In preparing for a future Ethics Submission, I would schedule two weeks for it in my Research Proposal. Approval was granted for this research to be undertaken on 06/02/18 (Table 2).
Milestone 3: Literature Review and Interim Report (Deadline: 17/11/17)
I submitted an interim report to my supervisor on 19/11/17 as well as a literature review on 10/12/17 (Table 2). A very comprehensive literature review pertaining to domain model and adaptive learning had already been carried out for the Research Proposal. This review dealt with conceptual understanding of a function, and referenced the “function concept” as “one of the most important topics in high school mathematics” (Dubinsky & Wilson, 2013). It also alluded to an experimental study suggesting that conceptual and procedural knowledge in mathematics should be taught iteratively (Rittle-Johnson, Siegler, & Alibali, 2001). However, I could not discover literature that demonstrated a specific link between functions and sequences. Part of my domain model used this link and I was somewhat frustrated and disappointed not to find references in the literature.
Milestone 4: WIP Presentation 1 (28/11/17)
The formative feedback that I received the day after my presentation was very positive. It was great to receive the feedback so quickly, as I find formative assessment is important for my own motivation. The link between formative assessment and the motivation and self-esteem of students was brought into sharp focus with the publishing of a seminal paper by Paul Black and Dylan Wiliam at the end of the twentieth century (Black & Wiliam, 1998). I learnt that I included too much technical detail, and I didn’t tell the overall ‘story’ of my project as well as I could (Table 2). In future, I will try to make my presentations shorter, make the narrative a ‘story’, and not assume that my audience has prior knowledge of my field of research.
Phase 1 – Design and Develop Digital Artefacts (01/09/17 to 31/12/17)
In May 2017, I envisaged using a single domain model authoring tool (GAT) from the GRAPPLE project (GRAPPLE, 2011). However, in June 2017 Professor Paul de Bra informed me by email that a new authoring tool (GAM AT) to replace GAT was being developed by a Masters student of his in Eindhoven. I made the decision to use the newer GAT AT, and this delayed the start of my domain model development work from early September to early October 2017. When it finally become available for me to use, I discovered that it contained bugs and no user manual. Getting the software to work at all was extremely challenging. The developer (Marc Craenen) was very helpful though and I spent a number of hours talking to him on WhatsApp. He remotely logged into my Mac using TeamViewer, and there were many emails exchanged between us. I don’t think there are any lessons to learn from this other than using early versions of software is riskier than using mature applications. I didn’t have much choice in the matter as there aren’t many domain model authoring tools to choose from. Paul de Bra advised me not to use the older GAT tool.
Ultimately, I ended up using three different authoring tools: GAM AT, Mindomo, and Rhumbl Maps (Craenen, 2017; Mindomo, 2018; Rhumbl, 2018). This is not something I envisaged or anticipated at the start of the project. I used Mindomo because the user interface of GAM AT did not allow me to draw a domain model. I ended up sketching models with pen and paper, and I discovered that these sketches looked like mind maps. This is why I decided to use a mind mapping application, and since I had a positive experience using Mindomo during the TELTA module in Year 1, I opted to use it again in Year 2. Rhumbl is something I almost ‘accidentally’ discovered and I decided to use it as it was described as a concept-mapping tool. However, I later discovered that it simply maps learning outcomes and topics, whereas domain models link concepts to other concepts. I am still happy that I used Rhumbl in my evaluation because it allowed me to further explore the relationships between concepts, topics and learning outcomes.
I feel I learnt more about different design and development approaches to domain model authoring by using different tools. The different design paradigms utilised by these tools led to different domain model designs that provided me with a greater insight into domain modelling. Microsoft Access and Excel were the two software applications that were of great assistance in the learning outcome decomposition process. I hadn’t anticipated how valuable Excel would be before the project commenced. I discovered myself working interdependently, rather than chronologically, with these five applications. Therefore, to a large extent the domain models were developed concurrently. It would be great if a single domain model authoring tool that has a mind mapping interface as well as some basic spreadsheet functionality became available in the future. The five domain model artefacts (three Mindomo, one GAM AT, one Rhumbl Maps) were developed between 07/10/17 and 15/02/18 (Tables 1, 3). According to my schedule, the domain model artefacts should have been completed by 31/12/17. This meant that my project was behind schedule by approximately six weeks. However, I had spent two weeks working on other aspects of my project during the ethics submission process, so this meant that I was really about four weeks behind schedule.
Phase 2 – Design online survey, interview format, topics/questions (01/09/17 to 31/12/17)
The early versions of the survey questionnaires and interview themes were created during the ethics submission preparation period between 04/11/17 and 14/11/17. It was invaluable to have draft versions of these created early in the research project. This made it easier to improve their quality later in the project. If I am to embark upon a similar project in the future, I will try to schedule the work so that draft and pilot versions of work, for different components of the project, are carried out earlier rather than later. The final versions of the questionnaires were ready by 27/02/18 (Mk 2 – School) and 05/03/18 (Mk 3 – IMTA) (Tables 1, 3).
Reflections on, and lessons learnt, from the various project milestones and phases from January to June 2018.
Milestone 5: Implementing and evaluating the project with your target user group (16/03/18)
This involved completing Phases 3, 4, 5 (see below) (Table 2).
Phase 3 – Implement online survey of IMTA members (01/01/18 to 31/01/18)
As detailed in my research proposal, I suggested that I would invite teachers from the Irish Mathematics Teachers’ Association (IMTA) to participate in my survey. I had hoped “to get upwards of fifty responses” using this method. Ultimately, I only got 26 responses and I think this was due to the onerous time demands on the survey participants. They were required to view a screencast that was 41 minutes long, as well as complete a Google Forms survey that would have taken approximately 20 minutes to complete. If I carry out surveys in future research, I will significantly reduce the time demands on participants.
Prior to implementing the IMTA survey, I decided to carry out a ‘test survey’ using Mathematics teachers from the post-primary school where I worked. I received 10 responses between 27/02/18 to 03/03/18. Having reflected on the ‘test questionnaire’ and responses, I decided to add six extra questions for the IMTA survey. It took from 05/03/18 to 06/04/18 to get 26 responses from IMTA participants.
Since the number of participants was lower than the anticipated 50 responses, I decided to combine the two surveys so that I had 36 responses to the common 30 questions in both questionnaires, as well as 26 responses to the extra six questions. It was also necessary to ‘manually’ merge the two sets of data that had each been imported from Google Forms to Microsoft Excel. I consider this to be a minor methodological limitation to my Applied Project.
Before the online survey and interviews could take place, one further artefact needed to be created. I wasn’t aware that I was required to create this artefact, as well as the domain model artefacts, until a meeting with my project supervisor. Originally, I tried to create an eLearning tutorial using Adobe Captivate 9 but I discovered that this wasn’t working out. The tutorial courseware I created during the instructional design module was successful. However, this time around I was concerned that the stage ‘actors’ and general methodology I was using in the tutorial was in danger of treating the adult participants as children. I then remembered the screencast I had created during the TELTA module, and after some experimentation, this is the design paradigm I ultimately decided to use.
The screencast I produced explicated the learning outcome decomposition process and domain model artefacts for the participants. The screencast turned out to be both useful and necessary in relation to educating them so that they could answer the survey and interview questions.
I used the 10-day mid-term break in February 2018 to complete a significant amount of work on the screencast. This was cognitively and technically challenging as well as very time consuming. It was difficult to tell the complex story of domain models and adaptive learning to a non-expert audience. The graphic output produced in the Adobe Captivate 9 output was originally of poor quality. Eventually, I surmounted this technical challenge by using Adobe Reader, Microsoft PowerPoint and alternative ways of processing the graphics in Captivate. The development of the screencast seemed to go on forever! The screencasts were developed using Adobe Captivate 9 between 23/01/18 and 04/03/18 (Tables 1, 3).
A total of 134 emails were exchanged between the IMTA and me between 19/11/17 and 05/04/18 with the two surveys completed between 27/02/18 and 06/04/18 (Table 3). These emails were to and from eight different branches of the IMTA (Clare, Donegal, Dublin, Galway, Kerry, Midlands, Waterford, Kerry). This exercise was required to maximise the response and to aim for nationwide participation. This was challenging, but there wasn’t any easier method of accessing these teachers. It might have been a little easier if IMTA used an online forum with instant messaging.
Phase 4 – Implement interviews with key informants (01/01/18 to 31/01/18)
Most of the interviews took place between 06/03/18 and 21/03/18. Nine key informants agreed to be interviewed, six of whom were referenced in my initial research proposal. I decided to use two extra Mathematics Educations Professionals (MEPs), sourced from Project Maths Development Group (PMDG) and National Council for Curriculum and Assessment (NCCA), instead of two Mathematics teachers. These two additional MEPs had a lot of experience with designing and implementing new Mathematics syllabuses in Ireland (Tables 1, 3). These were important additions as my domain model was based on the new syllabus. Both MEPs provided me with important insights into curriculum development during the interviews.
The most interesting thing to discover from the MEPs was that they all believed that making connections between the various parts of the mathematics courses was the most important thing to be achieved for the successful teaching and learning of mathematics. Creating these connections is one of the key components of a domain model. This is consistent with my philosophical standpoint that the interconnectedness of the various fields of mathematics requires a constructivist approach to teaching and learning.
Phase 5 – Data Analysis (01/01/18 to 31/01/18)
I personally carried out all of the transcribing, and this took an enormous amount of time. It required in the region of 12 to 15 hours to transcribe one hour of interview audio. The transcribing process took the entire month of April to complete. This included working full time on this activity for the two week Easter holiday period in April 2018. I had been made aware in advance that transcribing would be time consuming but I didn’t really appreciate the considerable and tedious work involved until later.
I had not transcribed interviews before, so it was interesting to learn Google Voice and use it, in conjunction with ‘parroting’, for some of the transcribing process. However, I found that using headphones and listening to the audio played via VLC Player on a computer at half speed while simultaneously word processing the text in Google Docs was the most effective method.
The advantage of personally transcribing the interviews was that it allowed me to take notes and do some initial data analysis. If I were to carry out interviews for a future research study, I would definitely try to save time by either outsourcing the transcribing process, or preferably find a suitable software application to automatically transcribe the interviews.
The actual analysis of the interview data by theme and by question mainly took place during May 2018 (Tables 1, 3). The use of semi-structured questions meant that most of the themes were pre-ordained and this was very helpful.
The data analysis of the Google Forms survey was greatly assisted by Microsoft Excel. It still took a considerable amount of time, especially having to merge two surveys and triangulate with the interview data. For a future survey, I would try to ensure that I only have one survey to deal with.
Milestone 6: WIP Presentation 2 (15/05/18)
The formative feedback that I received the day after my presentation was fairly positive. However, it appeared that I could have provided more information about the findings and responses of the participants, and provide less information on the technical details of the project (Table 2). I don’t think that I fully learned from WIP Presentation 1 (28/11/17) that I need to make the narrative a ‘story’. I went in with the idea that I could just bring my audience on a whistle-stop tour of my ePortfolio.
Milestone 7: Applied e-learning Project Submission (24/05/18)
This was submitted to my Supervisor and First Reader well in advance of the deadline date 4th June 2018 (Table 2).
Milestone 8: Journal Paper Submission (09/07/18)
This was emailed to the Supervisor and First Reader a week after the deadline date of 2nd July 2018, having been granted an extension of one week (Table 2). I worked on the journal paper full time for just over five weeks, from 1st June until 9th July 2018. I learnt that I should have provided my supervisor with a first full draft of my paper a little bit earlier. This might have avoided a certain amount of restructuring, and the one-week extension. The formative feedback from both the Supervisor and First Reader was excellent and helped to produce a better quality final paper. The Microsoft Word markup tool that they both used is a great way to receive and utilise formative feedback. However, interactive conversation via telephone was also very useful during the paper restructuring process.
Milestone 9: e-Portfolio Submission (09/07/18)
The URL was submitted to the Supervisor and First Reader a week after the deadline date of 2nd July 2018 having been granted an extension of one week (Table 2). I received an email on 20th August 2018 from them suggesting a number of changes to my ePortfolio. I learnt that my Project Reflection and Reflection on Practice were not properly following a reflective model. Rolfe, Freshwater, & Jasper, 2001 suggested a model based upon three simple questions: What? So what? Now what? After an exchange of a small number of emails, and a 40-minute telephone conversation with my Supervisor and First Reader on 22nd August 2018, I felt that I understood what changes I needed to make to the two reflective pieces, my eLearning Philosophy statement, and also what revisions were required to make my ePortfolio easier to navigate and look more professional.
Black, P., & Wiliam, D. (1998). Inside the Black Box: Raising Standards through Classroom Assessment. The Phi Delta Kappan, 80(2), 139-148. https://www.rdc.udel.edu/wp-content/uploads/2015/04/InsideBlackBox.pdf
Craenen, M. (2017). GAM Authoring Tool: An authoring tool for GALE. (Master’s thesis), Eindhoven University of Technology, Eindhoven, Netherland. Retrieved from https://pure.tue.nl/ws/files/91106245/Craenen_Marc_Thesis_CSE.pdf
Dubinsky, E., & Wilson, R. T. (2013). High school students’ understanding of the function concept. The Journal of Mathematical Behavior, 32(1), 83-101. https://doi.org/10.1016/j.jmathb.2012.12.001
GRAPPLE. (2011). Welcome to the GRAPPLE project Website. Retrieved from http://grapple.win.tue.nl/home.html
Mindomo. (2018). Collaborative mind mapping, concept mapping, outlining. Retrieved from https://www.mindomo.com/
NCCA. (2017). Draft Specification for Junior Cycle Mathematics. Dublin: NCCA Retrieved from https://www.ncca.ie/media/3163/jcmathematics_draft_specification.pdf
Rhumbl. (2018). Interactive data visualization for exploration, engagement and analytics. Retrieved from https://rhumbl.com/
Rittle-Johnson, B., Siegler, R. S., & Alibali, M. W. (2001). Developing conceptual understanding and procedural skill in mathematics: An iterative process. Journal of Educational Psychology, 93(2), 346-362. https://doi.org/10.1037/0022-06188.8.131.526
Rolfe, G., Freshwater, D., & Jasper, M. (2001). Critical reflection in nursing and the helping professions: a user’s guide. Basingstoke: Palgrave Macmillan.