(Research checklist: Version 6.6.1)
- The best thesis is grounded in theory.
- Does your study pass the "So what?" test! Is it adding anything new to what is already know. Replication is needed yes but you need to expand on the existing body of research.
- Download the ethics application and read through it. Fill in as you go along so that you have most of it ready before you need to submit it. (Check out my ethics blog post)
- Are you sampling from more than one population and if rightly so is that dealt with in your analysis and text?
- Have you got reliable and valid measures for all your concepts? You can start writing your thesis now, reporting on the validity and reliability as well of the measures then later add the internal reliability (Cronbach's alpha) from your study.
- How easy will it be to get participants?
- What will your return rate (response rate) be?
- What statistical analyses will you use?
- What is your time frame?
- Run a pilot. Yes even if you are doing a survey (e.g., internet, paper-and-pencil) type study. At least then you know if there are any problems with any of the questions and how long it will take to complete them.
- Running an experiment?
- Make sure that you test your manipulation in the pilot.
- Have you got manipulation checks in place?
- Calculate power for all hypotheses. Try using G*Power http://www.psycho.uni-duesseldorf.de/abteilungen/aap/gpower3/
- Start formatting your thesis document using the latest APA manual as your guide.
- Use the heading 1 etc. to format your headings but modify the default in line with APA style.
- Install EndNote to organise your references and citations.
- Make sure that the IV (the manipulation) is not “contaminated”
- For example if you measure depression and fatigue then both concepts may share similar questions thus the correlation between the two will be inflated.
- Different levels of different IVs should not overlap.
- When you create tables use the insert table function rather than tabs.
- Please do NOT use spaces to create tables.
- If you table needs to be in landscape then insert a section break before and after the table before you apply the new orientation (landscape) to the table.
- Make sure you refer to every table in text before the table appears and make sure that the table appears on the page you refer to it or the following if it cannot fit on the page where it is being referred to.
- Check out table templates at http://einarbthorsteinsson.blogspot.com.au/2013/11/table-templates.html
- Consider using a figure to capture the allocation of participants to different conditions and any drop out and treatment.
- Download one document from apastyle.org covering the following:
- "Journal Article Reporting Standards (JARS)".
- "Meta-Analysis Reporting Standards (MARS)".
- "Flow of Participants Through Each Stage of an Experiment or Quasi-Experiment".
This is now covered in a separate blog
Ethics (UNE related)This is now covered in a separate blog
While gathering data
- Are you working on your literature review?
- Are you getting enough data. If not, take steps to get more data.
- If you are conducting an experiment remember to check your manipulation checks to see if it is working.
- If it is not working at all then back to the drawing board – you should have found this out in your pilot.
- If it is only working on some participants then back to the drawing board or get more participants to meet your power needs.
- Remember to use sensible:
- file names based on your surname, three words to capture your thesis project and a version number (e.g., “Thorsteinsson Socials support and depression v01.sav”), when you make changes to the file use the “save as” function to update the version number to v02 etc. making sure that you keep a copy of older versions. It is not a bad idea to do the same for your thesis word processing document.
- variables names
- variable labels
- value labels.
- When you reverse score create a new variable (e.g., “ffs01” reverse scored into “ffs01r”).
- When you transform then transform into a new variable (e.g., “ffs01r” into “ffs01rt”).
- Don’t forget to give new variables clear variable label (including information such as transformed square root, transformed log 10, reverse scored).
- Use syntax if you can so that you know what steps you have already taken.
- Make sure that you have individual items from every scale used in the data set, how else will you calculate internal reliability and summarise individual scales.
- When you are aggregating (e.g., mean, sum) item scores into a sub-scale/scale score make sure to use check how many item scores are needed for you to calculate a scale score. Usually you can have a few items missing before you start thinking that you don't have enough items to calculate a scale score. Having scores missing for 3 items does not seem much if there are another 7 items in the scale all 7 with scores.
- If you need the sum of scores for the scale in question make sure to calculate the mean and then multiply with the number of items on the scale as that will give you a better indication of the sum for that scale than if you sum maybe 7 item scores while 3 item scores are missing.
- Do you define key concepts as soon as they appear in text – don’t leave the reader wondering what you are talking about; use examples and/or definitions to explain your concepts.
- Do you draw conclusions about cause and effect when you have only done a correlational or a quasi-experimental study? Try not to, some makers are dead against it while other markers are more pragmatic.
- Do you use concepts consistently; don’t start by talking about psychological problems then later talk about psychological issues if you do then the reader will assume that psychological issues and problems are not the same or that you are confused and don’t know what your are writing about.
- Does your introduction support your hypotheses and the questionnaires employed?
- Are there any variables/concepts in your hypotheses not mentioned in the introduction?
- Are your hypotheses non-directional or directional as per evidence provided in the introduction?
- Results must test all of your hypotheses (test them in the correct order and make it clear to the reader when you are testing them).
- Report effect sizes for all statistical analyses reported (e.g., ANOVAs, t tests, Chi-square).
- Is it easy to follow hypothesis testing in the result section?
- If you have replaced missing values, adjusted outliers, and/or transformed the data the reader will wonder if the raw data supports the findings you end up reporting. Why not run the same analyses that you have already run using the raw data to see if the results are similar (e.g., effect size you get and conclusions you would draw) to the ‘transformed’ data, if these are the same then consider reporting the raw data as it has values that make ‘sense”.
- If you have not enough power you need to acknowledge this in your discussion and be careful what you say as your study has not proved the absence of effects if it got no statistically significant effects.
- If you are looking at sub-samples are they confounded? For example if you compare males and females and there are only internal students in the male sample but external and internal student in the female sample then you have a problem because your interpretation cannot say if there is a sex difference or if there is study mode difference.
- Do you treat your r-values correctly? Remember that an r=.20 is the same as r=.20. However, one of these may have a large N and thus be statistically significant. You have more confidence in the statistically significant r while the other may only be based on a small N thus you have less confidence in the finding. Just remember they the rs are the same size (same effect size) but they differ in the how reliable they are (i.e. can you trust the results to be close to the real association or are they off the mark).
- In your Discussion section make sure you discuss your findings in relation to previous findings and theories as covered in the introduction.
- Write the abstract after finishing other parts of your thesis.
- Do you have to run a MANOVA before running multiple ANOVAs? Well, I would say no if the ANOVAs are testing your hypothesis/es. Make sure you read and cite Huberty, C. J., & Morris, J. D. (1989). Multivariate analysis versus multiple univariate analyses. Psychological Bulletin, 105, 302-308. doi:10.1037/0033-2909.105.2.302
- Which analysis to use? Try http://www.ats.ucla.edu/stat/mult_pkg/whatstat/ Note that this table has links to the relevant sections for SAS, Stata, SPSS, and R with instructions/information for the relevant analysis.
- Be consistent when it comes to the number of decimal places. Stick with two decimal places in most cases as any more implies a level of measurement accuracy that you simply not have, depending on your area obviously. You can make an exception for the p value as so many editors hold it in such high esteem thus it may require three decimal places.
- Put a leading zero such as Hedges' g = 0.24 for values where the index in question can take a value lower than -1 or higher than +1. Thus you don't have to put a leading zero for r, or p values but you should for F, t, b, and beta.
- Do you need SPSS or will JASP https://jasp-stats.org/ do the job?
Materials sectionFor each scale (and sub-scale) that you use report
- alpha level(s) from your study;
- number of items;
- scoring range and anchors;
- reverse scoring of what items;
- give an item(s) example(s);
- citation for the overall scale (not each sub-scale assuming the citation is the same for them all).
Partial example: “The _____ scale is a 10-item scale, answered on a 4-point Likert scale ranging from 0 (Did not apply to me) to 3 (Applied to me very much, or most of the time) to answer questions such as “I felt that life was meaningless”. In the present study internal reliability was .78.”
For more examples and coverage see the APA manual.
- Report ethics approval and number.
Effect sizesThere are other guidelines for what is a small, medium or large effect size but at least these match small, medium, and large for the d effect size when you convert d to r squared etc. These d value ranges here are based on Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, New Jersey: Lawrence Erlbaum.
For Cohen's d or Hedges' g or d (be the values positive or negative) we haveSmall: 0.00 to 0.20
Medium >0.20 and <0.80
Large 0.80 and higher
For r (be the values positive or negative) we haveSmall: .00 to .10
Medium >.10 and <.37
Large .37 and higher
For partial eta squared, eta squared, r squared use:
Small .000 to .010
Medium >.010 and <.138Large .138 and higher
Question 1: When reporting effect sizes, is it necessary to be consistent in the type of effect size reported?
It is fine to report different types of effect sizes (ES) for different results such as
Partial eta squared for ANOVA.
Hedges' g, Cohen's d, or d to capture differences between two groups.
Pearson's r is an ES thus no need for additional ES when reporting correlation coefficients.
When you report Hedges' g, Cohen's d, or d you also need to report the CI and usually you'd report the 95% CI maybe like
Question 2: Where can I find an ES calculator?
Here is an Excel spreadsheet calculator.
Google the question and you'll find plenty of other options such as spreadsheets, software, and web pages.
- Use a reference manager such as Mendeley http://www.mendeley.com/ or EndNote http://blog.une.edu.au/studentexperience/2012/03/09/endnote-free-for-hdr-postgrad-ons-students-details/ to organise your references and citations.
- Remember to italicise volume numbers, journal titles (not the name of the article), and book names.
- Remember NOT to put the issue number unless the journal is paginated by issue (i.e. each issue starts on page 1), the latter would be unusual for psychology journals.
That is all for now.