M4-Week 7-Class

Educational Research Design Module (Week 7)

The following is a reflection on the Tuesday morning class that took place on 30th May 2017 from 10:00 to 13:00 using Gibbs Reflective Cycle.

Description

THIS WEEK:  Ethics

NOTES:  from today’s class…

Today’s Class
Introduction
Role of Research and Role of the Researcher

Introducing the Module
10 ECTS
8 weeks
Research Proposal for Assessment
The Handbook…

Role of Research

What is Research?
US
– Body of material
– Hypothesis/Thesis
– Noun or Verb
– Literature
– Methodology
– Discovery
– Ethical
THE BOOKS
…the systematic, controlled, empirical and critical investigation of hypothetical propositions about the presumed relations among natural phenomena.
Cohen & Mannion (2007)
What does Research Do?
– Asks questions
– Examines Policy/Practice
– Provides Solutions
– Creates Questions
-Tests Hypotheses

What Makes Good Research
– Relevant
– Unbiased
– Clear Objectives
– Ethical
– Theoretical Background
– Verifiable
– Organised
– Concise
– Can be built on

Who Does Research
– Everybody
– In education, undergraduates and postgraduates
– Professional Researchers

What is Educational Research?

…the collection and analysis of information on the world of education so as to understand it and explain it better.
Opine, 2007 p.3

Role of Your Research

What’s my goal?

Ans. To bring about change in my area of teaching.

What can I get from the process?

How can other’s potentially benefit from my research?

Role of Researcher during the Research Process

– Research Proposal
– Introduction
– Literature Review
– Research Design and Methodology
– Data/Findings
– Conclusions

– Interpreter
– Interviewer
– Listener Proofreader
– Creator
– Administrator
– Decision Maker
– Writer
– Authority
– Reviewer
– Ethics Guardian
– Reflector
– Observer

The
Paper (Say on effects/supports provided by the artefact)
Artefact (say website for teachers)
EPortfolio

Ontology and Epistemology

Ontology – knowledge that’s out there to be known (all the stars)
Epistemology – how we are going to come to know about it (use a telescope)

Ontology – Definition…..
Epistemology –

O’Leary, 2010, p.5
Ontology – What’s out there to know?
Epistemology – How can we know about it
Methodology – How can we go about acquiring that knowledge?
Methods – Which precise procedures can we use to acquire it
Sources – Which data can we collect?

Figure 1: The interrelationship between the building blocks of research
Source: Figure adapted from Hay, 2002, p.64

Subjective-Objective Dimension

A scheme for analysing assumptions about the nature of social science
The Subjectivist
– Nominalism
– Interptretivism-Anti-Positivisn
– Voluntarism
– Idiographic

The Objectivist



Cohen & Mannion quoting Burrell
Four “Worldviews”

Postpositivism
– Determination
– Reductionism
– Empirical observation and measurement
– theory verification

Constructivism
– Understanding
– Multiple participant meanings

Transformative

Pragmatism
– Consequences of actions
– Problem- centred
– Pluralistic

Creswell 2014, p.6
Framework for Research

Postpostivist => Quantitative
Constructivist => Qualitative

Research Methods
Questions
Data Collection
…plus 2 others?

Paradigms and Paradigm Wars

Qualitative v Quantitative

Quantitative Purists: We can observe objectively, measure, analyse and determine reliable and valid outcomes
Qualitative Purists: It is not possible nor desirable to research objectively, multiple realities constructed by individuals
Theoretical Perspectives
As an educational researcher you need to be aware of your perspective…

– What assumptions might you be making?
– What is your “home” discipline telling you and is it different to what you are investigating here?
What might be the connections with learning theories?
Researching with people?

Your Research in Practice

RESEARCH IS THINKING WORK
Focus shifts from design/disciplinary tradition to the research question
What is the best (mix) of methods we can use to answer that question?

The Research and Writing Process
Research is not necessarily a linear process

Step 1 => Step 2 => Step 3 => Step 4

Ideas, Drafting

Your Approach will Depend on your Preference

– Linear
– Mindmap
– Coming up with an idea, writing, showing it to others, rewriting
– Weaving

M4-Week 6-Home

Educational Research Design Module (Week 6)

The following is a reflection on the week immediately after the class that took place on 23rd May 2017 using Gibbs Reflective Cycle.

Description

THIS WEEK:  Data Collection (Home)

The broad area / theme of my research topic is adaptive learning / personalisation.

My current research question is:

“How can a domain model for adaptive learning in secondary school mathematics be evaluated?

My current research title is:

“The Design and Evaluation of a Domain Model for Adaptive Learning in Secondary School Mathematics (Functions at Leaving Certificate Ordinary Level )”

In relation to methodology, I reduced the number of choices available to me by eliminating the following methodologies: experimental and quasi-experimental research, phenomenological research, action research, heuristic inquiry, grounded theory and ethnography. (Gray, 2013).

I propose to take two different methodological approaches:  analytical survey and case study.  Epistemologically speaking, the former is underpinned by an objectivist approach and the latter by a constructivist approach.  (Gray, 2013).  (Note:  After Padraig McDonagh’s talk, perhaps I will call my methodology post-positivist).  Methodologies can be classified according to their purpose as well as by their approach.  According to Robson (2002), there are three possible forms of study:  exploratory, descriptive and explanatory with Maxwell (1996) adding interpretive to this list.  The case study component of my research will be an interpretive study seeking to explore teachers’ and text book authors’ views on why learning topics should be sequenced in a particular way.  The analytical survey will seek to find out how these topics are actually sequenced by teachers in class (scheme of work/lesson plans) and by authors in textbooks (chapters/topics).

I plan to take a mixed methods approach by doing an analytical survey and by conducting interviews.  The analytical survey will be cross-sectional rather than longitudinal.

How do you plan to find the answer? i.e. What methods will you use?

What areas of your research proposal do you still need to develop?

M4-Week 6-Class

Educational Research Design Module (Week 6)

The following is a reflection on the Tuesday morning class that took place on 23rd May 2017 from 10:00 to 13:00 using Gibbs Reflective Cycle.

Description

THIS WEEK:  Data Collection

NOTES:  from today’s class…

Qualitative Data

Characteristics
– Depth rather than breadth
– Rich thick descriptions
– Experience the phenomenon through the eyes of others
– Social and historical context
– Flexible research designs

Concerns/Criticisms and Questions about Qualitative Data
(As a reader/consumer)
– it’s soft
– not credible / not scientific /

MORE STUFF (look at file in BB)

Interviews as a Method
– MORE STUFF (look at file in BB)

Interviewer Metaphors (What’s your epistemology?)
– Miner (More structured)
– Traveller (Less structured)
(Kvale, 1996)

Interview Considerations (Planning)
– Formality
– Structure(Structured, Semi-structured, Unstructured)
– Single or Multiple
– In Person or Distanced

Types of Questions
– Closed, e.g. Do you get on well with your colleagues in your office?
– Open, e.g. Can you tell me about your relationship with your colleagues in your office?

Interviews and Ethical Considerations
– Putting people under pressure
– Not safeguarding anonymity

Interviews Resources
-Uwe Flick
– Graham Gibbs (YouTube channels)

Focus Groups (instead of interviews)
– Suitable for studying interaction and for comparing viewpoints
– Less suitable for narrative studies or addressing individual attitudes
– Need careful planning and design
– May know each other already or be unfamiliar to each other
– Pragmatic in MA and MSc projects
* a means to gather rich cars in a tight timescale
* may be more comfortable in a group
* can be conducted online if necessary
– Get help!
– If possible ask a colleague to share the focus group

Purposeful Sampling
– Researchers intentionally select individuals and sites

Gray – Doing Research in the real world – Sampling Strategies

M4-Week 5-Home

Educational Research Design Module (Week 5)

The following is a reflection on the week immediately after the class that took place on 16th May 2017 using Gibbs Reflective Cycle.

Description

THIS WEEK:  Data Collection (Home)

Lesson Planner Tool for Maths (Vincent Wade et al, 2016).
ADAPT Centre, TCD
Wednesday 17 May 2017
Keyword search instructional design and domain model and mathematics
Note:  Doesn’t have user model layer for adaptivity

Homework:

Critiquing your chosen journal article
(This involved reading the article by Jones & Lea in Week 5).

The article you have been assigned is listed on the next page and can be accessed in the ‘Content’ folder in Webcourses. Please read the article with the following questions in mind, record your responses and bring them to the next session on May 23rd.
We will ask you to discuss your research design critique with those who read the same paper during the class on May 23rd and to prepare a collaboratively written critique for sharing on Webcourses.

Some questions to think about…

1.     Is there a clear research question?
Not exactly.  However, the research project sought to focus on the textual engagements in digital environments by third level students.

2.     Is the methodology clearly stated by the author(s)?

Yes.  Fine-grained, ethnographic-style (Green & Bloome 1997) research of literacies and
technologies in higher education.

3.     What methods have the author(s) used?
Qualitative text based methods.  There were three sets of interviews over six months.
Interviews carried out in small groups (3 to 4) as well as some individual interviews.  These were followed up by email, chat, text.  This follow-up is what the authors call ‘shadowing’.

4.     What issues – if any – arose during the data collection phases?
Not being insider researchers resulted in limitations in data collection.  Students did not spend much time in the institutions outside of lectures.  Many students worked part-time.

5.     What are the main strengths of the research design?

(a) Good stratified sampling by using 45 students from three very different third level
institutions.  This yielded a broad spectrum of students. (b) Direct observation of students
by outside researchers (c) A rich data-base was assembled (transcripts, field notes (both electronic), curriculum sources, personal development plans (PDPs).

6.     What are some weaknesses of the research design?
No triangulation.

M4-Week 5-Class

Educational Research Design Module (Week 5)

The following is a reflection on the Tuesday morning class that took place on 16th May 2017 from 10:00 to 13:00 using Gibbs Reflective Cycle.

Description

THIS WEEK:  Data Collection

NOTES:  from today’s class…

Note:  Check out Envivo software.
Note:  Ethics:  if you offer an incentive to complete your survey, this should be stated in your ethics section.
Note:  You should let people know what you plan to do about the data that you collect.
Note:  Informed consent.

Quantitative Data

Nominal (Male, Female???!!!), Ordinal, Scale

Andy Field – Statistics and SPSS and YouTube

Statistical Data
– Census

Survey Data
– End of module feedback sheets
Scores of Results
– exams, tests, quizzes
Systems Analytics
– Webcourses/Blackboard/Moodle

Primary Research
– Questionnaires
– Structured Observations

Secondary Research
-CSO Data
– Institutional Data, records
– Web hits, Google analytics
– System analytics

What is Qualitative Data

Qualitative data analysis explores themes, patterns, stories, narrative structure, and language within research texts (interview transcripts, field notes, documents, visual data, etc) in order to interpret meanings…

Cousin, 2009

Some Examples of Qualitative Data

Interview, focus group
Questionnaire Data
Observation Notes
Reflective Diaries and Logs
Written Texts and Documents
– Government policies or reports
– Institutional documents
Images
Video

Common Data Collection Issues
– Sample Size
– Triangulation (see p.37 in Gray) (can be from one quantitative method to a qualitative method or to another quantitative method)

Something to think about…
– Does the analysis of qualitative data involve some quantitative methods?
– A second question…

Validity
“Good” or “bad” research
(Opie, 2004)

Validity relates to findings and results
But it also relates to research design and process

Qualitative Research
In qualitative data, validity might be addressed through honesty, depth, richness and scope of the data achieved, the participants approached, the extent of triangulation and the disinterestedness of the researcher.

Quantitative Research
Validity of instruments
– importance of ……
Construct validity
– what your interested in has actually been measured
Internal validity
– can a causal relationship be shown?
External validity

M4-Week 4-Home

Educational Research Design Module (Week 4)

The following is a reflection on the week immediately after the class that took place on 9th May 2017 using Gibbs Reflective Cycle.

Description

THIS WEEK:  Methodology (Home)

Saturday 13/05/17

MyEndNote
Signed up for a myendnoteweb.com account
Installed Cite While You Write EndNote plug-in for Microsoft Word for Mac 2011
Installed EndNote Capture Reference tool in the bookmarks of (a) Safari (b) Firefox (both for Mac)
Installed EndNote Capture Reference app for Chrome

Sunday 14/05/17

Homework

(1) Email from Claire McAvinia on 11/05/17
The prompt questions were:

·         What is the broad area/theme of your research topic? (e.g. curriculum design, student experience)
Answer:  Adaptive/Personalised Learning
·         What are you trying to find out/what is your research question?
Answer:  How can a domain model for adaptive learning in mathematics be evaluated?
·         How do you plan to find the answer? So what methodologies might help you do this?
Answer:  Literature review, design and evaluation of a domain model for adaptive learning in secondary school mathematics.
·         What methodology/methodologies do you feel you can rule out?
Answer:  (1) Experimental and quasi-experimental research (testing hypotheses, control and experimental groups, etc.)
(2) Phenomenological research
(3) Heuristic inquiry
(4) Grounded theory (methodology or method)
What’s left?
(1) Analytical Surveys (A deductive approach, the identification of the research population, the drawing of a representative sample from the population, control of variables,
the generation of both qualitative and quantitative data, generalizability of results)
(2) Action Research (Involves both researchers and practitioners (or practitioners as researchers within their own organisation), can be highly structured and involve the use of
experimental and control groups used to test a hypothesis, can also be quite unstructured and used inductively (and qualitatively)

·         Plan to write about a page over the week on the methodology you are most interested in. Note any questions for the session next week once you have drafted your page.

(2) Email from Claire McAvinia on 12/05/17
Between now and Tuesday morning, can we ask you to add a note to our Padlet wall here:
https://padlet.com/wall/1hpwjlmjmis9
Simply double-click on the screen to get started.

This very short exercises is leading out of the work and the reflective activity you had last week and leading us into our next topic.

On Tuesday, we will be starting to think about Methods and looking briefly at quantitative methods before getting into questionnaire design. The session will be very participative and we look forward to hearing more of your ideas.

Padlet

Research Question and Methodology – Gerard Kilkenny

Research Question – How can a domain model for adaptive learning in Junior Certificate Mathematics (Strand 5 – Functions) be evaluated?

Methodology – I am assessing the suitability of a Case Study where experienced Mathematics teachers/textbook authors will evaluate a domain model (digital artefact) designed by the researcher (me).

Literature Support for Case Study –
O’Donnell et al (2015, p.26) assert that “the case study methodology is possibly the most appropriate approach to apply to the complex research question of the evaluation of personalised e-learning.”

Šimko (2012, p.2) states that “particularly challenging is development of a proper methodology for domain model evaluation.  A direct “quantitative” evaluation is difficult as it is hard to define exact measures that are suitable to assess quality of a domain model for adaptive web-based learning.”

O’Donnell, E., Lawless, S., Sharp, M., & Wade, V. (2015). A review of personalised e-learning: Towards supporting learner diversity. International Journal of Distance Education Technologies, 13(1), 22-47.

Šimko, M. (2012). Automated acquisition of domain model for adaptive collaborative web-based learning. Information Sciences and Technologies Bulletin of the ACM Slovakia, 4(2), 1-9.

M4-Week 4-Class

Educational Research Design Module (Week 4)

The following is a reflection on the Tuesday morning class that took place on 9th May 2017 from 10:00 to 13:00 using Gibbs Reflective Cycle.

Description

THIS WEEK:  Methodology

NOTES:  from today’s class…

Case Study (Pauline Rooney)

Harnessing Serious Games in Higher Education

Validity
Insider Research
Generalisability

Note:  She didn’t use a hypothesis.

What is a Case Study

Focuses on one or more instances of a particular phenomenon
The study of an instance in action
Boundaries may be temporal, geographic…
Appropriate when the desire is to…

Research study and questions

“What are the implications of adopting an interdisciplinary in-house approach to the design of serious games in HE?  What processes are involved in, and what issues may arise, with such an approach?”

Features of my methodology

* In-depth
* Phenomenological – aiming to identify the “essence” of human experiences concerning a phenomenon as described by participants (Cresswell, 1998)
* Participant observation case study
* Naturalistic (participants carefully observed in their natural setting without interference by researchers)

Data Collection Methods

Personal emails
Game design specification documents and storyboards
Two semi-structured interviews
Team email correspondences
Participant blog

Data Analysis

* Combined all data sources (excluding spec docs & storyboards)into a primary case document. (Triangulating multiple sources).  Used this to generate a detailed chronological narrative.
* Categorical aggregation (“the aggregation of instances…
* Direct interpretation (drawing meaning from individual instances of events)
* Winnowing (Wolcott, 1990)
* AIM

Let’s talk about validity…
(…in qualitative)

No such thing as one objective reality to be uncovered by researcher (Cohen et al. 2000)
We create truth or meaning

Valid data = authentic, confirmable, credible
(Denizen and Lincoln, 2000?)

How did I strive for validity?
Triangulation which is…
* Constant comparison method: trace correspondence / contradictions across multiple data sources (Glaser & Strauss)
* Challenge patterns/themes by deliberately searching for negative instances of pattern
* Shared the reduce case narrative with another team member to verify accuracy of narrative (Yin 1981)

My role as an insider researcher/participant observer…
Positives:
* Gain deeper insight into relationships/processes/events
* Knowledge of context
Negatives
* Interpreter bias?

Claire

ROLE OF METHODOLOGY

Methods
What you do to collect and analyse the data

Methodology

(1) Case study

Characteristics
Depth rather than breadth

Criticisms
* May be difficult to generalise
* Perceives as lacking rigour
* Observer effect – presence of researcher may result in participants behaving differently
* Data can be unwieldy and cases become too long

(2) Action Research

* Coined by social psychologist Kurt Lewin c. 1944
* Describes a form of research that could incorporate an experimental approach of social science

Characteristics

* Practical nature – real world problems (so, responding to a problem)
* Change – integral part of the research
* Cyclical/iterative process – feedback loop
* Participation is active not passive (so, the researcher is involved in the process)
*’Takes place on situ (e.g. classroom)
* Research in action rather than about action
* Change – improving education by changing the situation
* Cyclical

Action Research Models
Identify  -> Solution -> Try it out -> Evaluate it -> Change your practice

(3) Activity Theory

M4-Week 3-Class

Educational Research Design Module (Week 3)

The following is a reflection on the Tuesday morning class that took place on 2nd May 2017 from 10:00 to 13:00 using Gibbs Reflective Cycle.

Description

THIS WEEK:  Literature Review

NOTES:  from today’s class…

4 Groups

1-What are the purposes of a Literature Review? (GROUP 1)
2-What academic criteria should be used to assess a Review? (GROUP 2)
3-In your experience, what makes for a weak Review? (GROUP 3)
4-Draw up a set of self-assessment criteria (Have I) (GROUP 4)

GROUP 2 (ME)
Relevance
Coherence
Sequential – reviewing the arguments/analysing in sequence, e.g. under themes
Evolution – historical context
Referenced
Up to date
Accuracy

LECTURER (JOHN)
Collect sources using an established methodology: what is included and what is not (boundaries)
Identify the most seminal of sources
Organise these sources into themes
Compare and contrast findings
Identify conflicts between and within
Assess the significance /status of a source relevant to your study and in its own right (e.g. it’s methodology)
Build on and apply sources used
Demonstrate own contribution to knowledge/professional practice.  Make sure you bring your point of view.

GROUP 1
To find evidence to support your work
To evidence opposing ideas
To show context and background
To highlight a gap in knowledge

LECTURER (JOHN)
To set topic in its academic context – origins, significance and (historical) development of same
To identify what is known as well as gaps
To identify seminal and other studies
To identify opposing views
To compare, classify and summarise
To show how personally influencing certain resources have been: best practices
To assess status/significance of sources
To identify perspectives on research methods
To help assess own knowledge contribution
Is it a small scale or a large scale study?  (Nonetheless, however…words to be used)

“Two major studies have informed my thinking”
Classify the literature and then present it that way in your review.
Status of journal (ejournal versus paper journal)
What are the key journals?
High impact journals.
– Studies in Higher Education
– Assessment for (and?) Learning

GROUP 2 (ME)
Relevance
Coherence
Sequential – reviewing the arguments/analysing in sequence, e.g. under themes
Evolution – historical context
Referenced
Up to date
Accuracy

LECTURER (JOHN)
Collect sources using an established methodology: what is included and what is not (boundaries)
Identify the most seminal of sources
Organise these sources into themes
Compare and contrast findings
Identify conflicts between and within
Assess the significance /status of a source relevant to your study and in its own right (e.g. it’s methodology)
Build on and apply sources used
Demonstrate own contribution to knowledge/professional practice.  Make sure you bring your point of view.

GROUP 3
Irrelevance
Sources not reliable
Scope – should be balanced, not biased, demonstrate both arguments
Up-to-date, current
Values of writers, context of their work, acknowledged (could be an ideological framing on something); if it’s historical especially. Also, funding source.
Bad literature review will sit outside on its own from the rest of the work

LECTURER (JOHN)
The review is descriptive: ‘he says, she says’: no synthesis (bring the 3 people saying the same thing together).
Methods and findings are accepted rather than critiqued: lacks evaluation
Findings are mostly randomly discussed/presented:  lacks thematic classification
N.B. Findings are not internalised/applied (this is for the end of the review…which will be a bridge to the next section ‘Methodology’)
It is unclear how the review was constructed
Referencing is inconsistent and or inaccurate

GROUP 4
Have I clarified the objectives and goals of the literature review (in a particular context)
Has it defined the context of the research question?
Is the state of the art displayed clearly?  (“Current thinking is exemplified by…..”)
Reviewed all the relevant research?
Demonstrated knowledge of the area?
Contextualised the research question?
Found supporting and opposing literature?
Linked the literature to my purpose in research?  Linked to the findings of my research (Year 2!)?
Tested assumptions/intended/unintended outcomes?

LECTURER (JOHN)
Explain how and where literature was found?
Establish boundaries?
Clarify purposes of review?
Identify themes as well as shared and opposing views within?
Synthesise these resources? ‘All leading writers in the field have identified….. (Taylor, 2014; Jones, 2015; Yin, 2016).
Reveal source as influencing?
Extrapolate only what is most salient to own study?

Note:  Good research is based on testing assumptions or hypotheses.  My intended outcomes were….. and my unintended outcomes were…..
Note:  Chronology may an idea.
Note:  Scope – last 5 years or last 10 years etc excluding seminal works

Book:  How to get a  PhD – Estelle Phillips and Pew?
All good research questions have words like ‘fewer’, ‘different’, ‘more slowly’