Study with the several resources on Docsity
Earn points by helping other students or get them with a premium plan
Prepare for your exams
Study with the several resources on Docsity
Earn points to download
Earn points by helping other students or get them with a premium plan
Community
Ask the community for help and clear up your study doubts
Discover the best universities in your country according to Docsity users
Free resources
Download our free guides on studying techniques, anxiety management strategies, and thesis advice from Docsity tutors
As of May of 2016, Quantic lessons had a 96- percent learner approval rating. The Quantic MBA curriculum consists of nine core subject areas.
Typology: Schemes and Mind Maps
1 / 55
This page cannot be seen from the preview
Don't miss anything!
July 2016
Mean test score Course Number of participants Pre-test Post-test Score Improvement Accounting 34 57% 86% 29 points Finance 43 49% 82% 33 points Though Quantic participants took a similar test twice within two weeks, it is unlikely that this improvement is due to practice or memory effects. After the pre-test, Quantic participants did not receive feedback on their performance, could not view correct answers, and could not view the test again. Post-test performance: Quantic versus MBA Quantic groups performed as well as, in the case of Finance, or better than, in the case of Accounting, the MBA groups on assessments of the topics studied. Quantic MBA Course Mean Post- test score Number of participants Mean Post-test score Number of participants Accounting 86% 34 75% 50 Finance 82% 43 81% 54 Time Spent in Quantic Courses On average, Quantic participants spent 147 minutes completing accounting coursework and 118 minutes completing finance coursework. These times were in line with course estimates of 128 minutes for Accounting and 111 minutes for Finance. Participant evaluation of Quantic custom assessment At the end of the post-test, all participants were asked, “Do you think this was a fair assessment of your [accounting / finance] knowledge? (Yes / No)”, and were also asked to explain their response. A majority of participants said that the custom Quantic Accounting assessment was fair (85 percent Quantic, 86 percent MBA); the Finance assessment was also considered fair (84 percent Quantic, 69 percent MBA).^1 Representative comments include: MBA (Accounting) “It asked the fundamentals that I learned in Accounting I” Quantic (Finance) “I was pleasantly surprised that the answers I had zero clue about in the pre-assessment, I now understood.” (^1) Only 26 of 54 the MBA Finance participants answered this question. The question for this group was sent out via email a few weeks after the post-test was taken, so the lower response rate may have just been circumstantial and should not be taken as an intentional unwillingness among this group to answer this question.
Quantic participant satisfaction (Net Promoter Score) Quantic participants were asked to respond (at the conclusion of the second test) to a standard satisfaction question, “How likely are you to recommend Quantic to a friend?”, on a scale of 0 (Would not recommend) to 10 (Definitely would recommend). The Net Promoter Score, NPS, is calculated as the percentage of respondents answering 9 or 10 (promoters), less the percentage of respondents answering 0 to 6 (detractors), with scores ranging from - 100 to 100. The Quantic accounting course had 58 percent promoters and 6 percent detractors, receiving an NPS score of 52. The Quantic finance course had 49 percent promoters and 9 percent detractors, receiving an NPS score of 40. These scores are in line with recent NPS scores for the Harvard MBA program and Wharton MBA program of 41 and 51, respectively. 2 ,^3 Representative comments include: Accounting “I actually really loved the courses. I felt like I learned a lot and was consistently engaged - the layout was great and constantly answering questions and clicking through was super entertaining/helpful. I would definitely take more courses in the future if offered.” Finance “Excellent. I feel I learned an extraordinary amount in a short period of time. The challenges and questions were the perfect balance between challenging and fun.” Conclusions Many companies are looking for the skills developed while pursuing an MBA. However, the pursuit of an MBA may not always be the right choice for all learners, either due to financial, geographic or time constraints. This study supports the assertion that some of the foundational accounting and financial concepts taught in traditional brick-and-mortar MBA program can be learned independently, online through Pedago’s targeted Quantic active-learning courses. Significant improvement in students’ knowledge can be gained in as little as two hours of engagement with these courses. In addition to being taken on their own as part of a comprehensive independent study MBA program, these courses could be used as supplements to existing MBA programs. First, they could be used as remedial materials for incoming MBA students who need to enhance their understanding of basic concepts in certain subject matter areas. Second, they could be used by MBA instructors as pre-requisite work for blended-learning classes. By having students learn basic concepts and complete structured practice exercises on their own, instructors could devote more classroom time to deeper learning through case studies and group work. (^2) Steven Hind, “What is HBS’s NPS?” The Harbus , February 5, 2015, (www.harbus.org/2015/the-harbus-survey- says-hbs-mba-program-more-walgreens-than-apple/) (^3) Natalie Kitroeff, “MBAs are more Satisfied at Wharton than at Harvard Business School”, Bloomberg , March 31, 2015 (www.bloomberg.com/news/articles/2015- 03 - 31/mbas-are-more-satisfied-at-wharton-than-at-harvard- business-school).
Quantic groups Participants in the Quantic groups could not have completed any MBA program nor completed an undergraduate degree in business, finance, or accounting. Additionally, participants must not have completed related courses outside of an MBA program or held a job in that field. For Quantic Accounting participants, this included courses in accounting or a job as an accountant. For Quantic Finance participants, this included courses in finance or a job with “finance” in the title. From those screened, 93 participants were selected to take the Quantic Accounting pre-test and 119 participants were selected to take the Quantic Finance pre-test. Of those invited, 62 participants in each group completed the pre-test. For Accounting, 49 participants started the coursework and 40 participants both finished the coursework requirements and took the post-test. For Finance, 47 participants started the coursework and 43 participants both finished the coursework requirements and took the post-test. Participants in both MBA and Quantic groups were assigned either to a Finance or Accounting group, and could not be included in both even if they completed tests and/or coursework for both subject areas. Neither MBA nor Quantic participants had previously completed any Quantic courses prior to the study. Test development Both the Accounting and Finance tests were created by Pedago’s content development team, which includes full-time employees and contracted professors. Existing MBA course exams in these fields were used as a guideline for development. Evaluation and pilot testing The tests were evaluated by Pamela Drake, PhD, CFA, Chandler/Universal Eminent Professor of Finance at James Madison University and Patrick Gosselin, MBA, CFP. A Spanish-language version of the test was piloted with students from Universidad Politécnica de Madrid in Spain (an institutional partner of Pedago) to evaluate the time required to complete each test. Pilot tests were conducted with groups of current MBA students recruited through ten online social media MBA groups to ensure that questions were a good test of MBA-level knowledge. One pilot test with eight participants was conducted for the Accounting test, while two pilots were conducted for the Finance test with 15 and 31 participants. Participants were paid $25 each and met the same eligibility requirements as MBA participants in the main study. One question on the Accounting test was removed and ten questions on the Finance test were either removed or changed based on this testing. Final tests The final pre-test and post-test in Accounting contained fourteen questions and the final pre-test and post-test in Finance contained sixteen questions. Many questions in the Accounting tests contained multiple parts. Therefore, participants were required to provide forty-three answers for the Accounting tests. The Finance tests included seventeen answers. Pre-tests included a question again asking students about prior coursework in the respective subject matter areas. Post-tests also contained one question asking whether the test was a fair assessment of subject matter knowledge (either MBA course knowledge or Quantic course knowledge, respectively) and a follow up question where participants could explain their answer. Post-tests also included a Net Promoter Score (NPS) question for Quantic participants to indicate whether they would recommend the Quantic course to a friend, and why or why not.
Participation MBA group Once selected, participants in the MBA group were given a few days from when they were notified of being accepted into the study to complete the MBA post-test. Quantic students were given up to two weeks to complete their pre-test, Quantic coursework, and post-test. All participants communicated with a research lead who monitored their progress, answered any questions about the research study procedures, and sent reminders to complete the study. MBA group participants were paid $50 to complete the one-hour post-test. Quantic participants were paid $50 to complete the one-hour pre-test, $100 to complete all necessary coursework (approximately 2 - 2.5 hours), and $50 to complete the one-hour post-test. 107 MBA participants were sent a link to complete the Accounting post-test and 95 MBA participants were sent the link to complete the Finance post-test. From this group, 51 participants completed the Accounting post-test and 54 participants completed the Finance post-test. Quantic group Each Quantic participant received an account for the respective courses and was emailed a username and a temporary password. Quantic lessons provide detailed feedback on both correct and incorrect answers and allow students to retake SMARTCASES, short quizzes at the end of each chapter. Quantic participants were required to receive at least 80 percent on all course quizzes to be eligible to take the post-test. All participants that met these requirements were emailed the post-test to complete by the end of the initial allotted time for the entire study period. Quantic coursework Accounting participants completed the Quantic courses Accounting I: Fundamentals and Accounting II: Revenue and Expenses. These courses are intended to help students discover financial statement preparation principles, learn bookkeeping mechanics for basic transactions, and take a deeper look at how to account for revenues and expenses, including the cost of goods sold and depreciation. Finance participants completed the Quantic courses Finance: Time Value of Money , Cost of Capital I: Capital Structure , and Valuation I: Discounted Free Cash Flow. These courses help students apply the principles related to a company’s balance between debt and equity, assess the profitability of proposed projects, acquire basic mathematical ability with the time value of money, gain familiarity with financial terminology, forecast a company’s future free cash flows, and value a company by discounting those cash flows back to the present. Based on course estimates, students should have been able to complete all lessons within 128 minutes for Accounting and 111 minutes for Finance. Detailed course syllabi including course structure, as well as concepts about and time requirements for each lesson within the required courses, are available in the Appendices. Testing Once participants qualified, instructions for participating in the study were emailed to each candidate. The pre- and post-tests were administered over Google Forms. Participants were told that each test should take less than an hour to complete. Though strict time limits were not enforced, students were encouraged to spend no more than 3 minutes on each question.
Table 1. Participation by gender Accounting Finance MBA Quantic MBA Quantic Gender Proportion Number of participants Proportion Number of participants Proportion Number of participants Proportion Number of participants Women 39%^20 44%^15 43%^23 57%^24 Men 44%^31 53%^18 57%^31 43%^18 Total 51 33 54 100%^43 Table 2. Participation by Age Accounting Finance MBA Quantic MBA Quantic Age range Proportion Number of participants Proportion Number of participants Proportion Number of participants Proportion Number of participants 18 - 24 4% 2 44% 15 4% 2 37% 16 25 - 34 92% 47 53% 18 93% 50 58% 24 35 - 44 2% 1 0% 0 4% 2 4% 2 Not disclosed 2% 1 3% 1 0% 0 2% 1 Total 51 34 54 43
Findings Quantic pre-course performance vs. MBA performance All Quantic participants had at least a bachelor’s degree, so we expect that most of the participants in the Quantic group would have had a lay knowledge of at least some of the concepts in each of the subject areas being tested, but not as much knowledge as MBA students who had completed formal coursework in these subjects. We compared pre-test scores for Quantic participants with post-test scores for MBA participants to examine this issue. As we show in Table 3, the mean score for the 34 Quantic Accounting participants is 5 6.7 percent and the median is 5 3.7 percent, with a low score of 26.8 percent and a high score of 80.5 percent. For the 43 Quantic participants in finance, the mean score for Quantic participants is 49.2 percent and the median is 5 2.9 percent, with a low score of 11.8 percent and a high score of 82.4 percent. We calculated the t-statistic for the test of the means of the Quantic participants versus the MBA participants to ensure that the initial knowledge of the Quantic participants was different from that of the MBA participants. We provide these results in Table 3. Our test of difference in means indicates that there is a statistically significant difference between the mean pre-test scores for Quantic participants and the mean post-test scores for MBA. Table 3. Comparison of Quantic pre-test results with MBA group Panel A: Accounting Number of participants Mean Score (Mean) Median Lowest score Highest Score Quantic pre-test 34 57% 54% 27% 80% MBA post-test 51 75% 76% 41% 93% t-statistic for test of difference in means, p-value in ()
(<0.0001) Panel B: Finance Number of s Mean score Median Lowest score Highest Score Quantic pre-test 43 49.2% 53% 12% 82% MBA post-test 54 82.4% 8% 29% 100% t-statistic for test of difference in means, p-value in ()
(<0.0001)
showing the most improvement in scores. For accounting participants, the greater the time spent on the material, the greater the change in the test scores from pre-to-post-test.^5 Table 5. Explaining score improvement The dependent variable is the change in the test score. Estimated coefficients, with p-values in parentheses below coefficients Finance Accounting Intercept 0. (<0.0001)
(<0.0001) Pre-test score 0. (0.0241)
(0.0164) Time 0. (0.0383)
(0.8335) R-squared 0.2041 0. F-statistic 5. (0.0104)
(0.0390) Quantic Post-test performance vs. MBA performance We provide the results of comparing the Quantic post-test scores with the MBA group scores for the Accounting and Finance courses, respectively, in Table 6. As we indicate with a t-statistic to test the difference in means scores between the Quantic group and the MBA group, the Quantic Accounting group outperformed the MBA group, with the difference in mean scores significant at most reasonable levels of significance. The Quantic Finance course participants score no differently than those in the MBA group, as indicated by a p - value of the test statistic that is above reasonable levels of significance. (^5) The pre-test score and time together explain 20 percent of the variation in test scores for Finance participants, and almost 19 percent for Accounting participants.
Table 6. Comparison of post-test performance between Quantic and MBA groups Quantic MBA t-statistic (with p - value in parentheses) for test of difference in mean scores Course Mean Number of participants Mean Number of participants Assuming equal variances Assuming unequal variances Accounting 86% 34 75% 51 3. (0.0002)
(0.0002) Finance 82% 43 81% 54 0. (0.6594)
(0.6473) Feedback from participants All participants completing the post-test were asked if they thought the test was a fair assessment of their subject matter knowledge in the respective areas. For Accounting, 85 percent of Quantic participants and 86 percent of MBA participants indicated that the test was a fair assessment. For Finance, 84 percent of Quantic participants and 69 percent of MBA participants indicated that the test was a fair assessment of their subject matter knowledge.^6 We provide representative comments in Table 7 for Accounting groups, and in Table 8 for Finance groups. (^6) Only 26 of 54 the MBA Finance participants answered this question. The question for this group was sent out via email a few weeks after the post-test was taken, so the lower response rate may have just been circumstantial and should not be taken as an intentional unwillingness among this group to answer this question.
Table 8. Representative Comments: Test was fair assessment of finance knowledge?
● It assessed basic building concepts of finance that should be learnt during introductory courses. ● The question itself was simple but require certain knowledge about finance. ● It captured the essentials any finance professional should know.. Of course, this only scratches the surface, but you managed to capture the essentials. ● I was a blank slate for finance knowledge. So anything I have or have not learned came directly from this platform! ● I was pleasantly surprised that the answers I had zero clue about in the pre-assessment, I now understood. ● The questions pertained to the material learned through the online modules. These questions were not too difficult, but also did not give away the answers.
● It looked more at specifics instead of general principles ● Finance is more about conceptual knowledge than just numbers.. This assessment concentrated more on quantitative stuff. ● It was too basic ● Because the questions seems very simple and similar to Quantic content. It felt like the course taught to the test. ● Simply taking a course online without any extra study time does not accurately reflect my ability to take in and learn information. The questions in the final assessment were very detailed. It is hard to remember these details from just taking an online course once. ● There should be case based questions. Quantic participant satisfaction (Net Promoter Score) Quantic participants were asked (at the conclusion of the second test) to respond to a standard satisfaction question, the Net Promoter Score (NPS) question, “How likely are you to recommend Quantic to a friend?”, on a scale of 0 (Would not recommend) to 10 (Definitely would recommend). The NPS is calculated as the percentage of respondents answering 9 or 10 (promoters), less the percentage of respondents answering 0 to 6 (detractors), with scores ranging from - 100 to 100. The Quantic Accounting courses had 58 percent promoters and 6 percent detractors, receiving an NPS score of 52. The Quantic Finance courses had 49 percent promoters and 9 percent detractors, receiving an NPS score of 40. These scores are in line with recent NPS scores for the Harvard MBA program and Wharton MBA program, with NPS of 41 and 51, respectively.^7 ,^8 (^7) Steven Hind, “What is HBS’s NPS?” The Harbus , February 5, 2015, (www.harbus.org/2015/the-harbus-survey- says-hbs-mba-program-more-walgreens-than-apple/). (^8) Natalie Kitroeff, “MBAs are more Satisfied at Wharton than at Harvard Business School”, Bloomberg , March 31, 2015 (www.bloomberg.com/news/articles/2015- 03 - 31/mbas-are-more-satisfied-at-wharton-than-at-harvard- business-school).
Conclusion Many companies are looking for the skills developed while pursuing an MBA. However, the pursuit of an MBA may not always be the right choice for all learners, due to financial, geographic, or time constraints. This study supports the assertion that some of the foundational accounting and financial concepts taught in traditional brick-and-mortar MBA programs can be learned independently, online through Pedago’s targeted Quantic active-learning courses. Significant improvement in students’ knowledge is acquired in as little as two hours of engagement with these courses. In addition to being taken on their own as part of a comprehensive independent study MBA program, these courses could be used as supplements to existing MBA programs. First, they could be used as remedial materials for incoming MBA students who need to enhance their understanding of basic concepts in certain subject matter areas. Second, they could be used by MBA instructors as pre-work for blended-learning classes. By having students learn basic concepts and complete structured practice exercises on their own, instructors could devote more classroom time to deeper learning through case studies and group work. Limitations of the study The Quantic courses are still under development and few students have taken all of the courses that would be equivalent to all of the coursework in a given MBA subject matter area. Therefore, it is not currently possible to test comprehensive subject-matter knowledge. The tests used were developed in-house and intended specifically to cover the concepts taught in the current courses. Whereas the tests used were deemed by students to be a fair assessment of their knowledge, many MBA students mentioned that the tests covered basic accounting principles, but did not cover other concepts they had learned in their accounting courses. For future studies with students who have completed an entire course series, it would be helpful to use a more-widely accepted test. First, such a test would allow for the assessment of a broader set of knowledge. Additionally, this sort of assessment would allow for comparison across a wider set of participants in traditional MBA programs. A more-widely accepted test would permit more cross-sections of the group, as well as a deeper analysis. Additionally, the timeframe for the study was compact. Therefore, the time between starting a course and taking the post-test was relatively short, and may not represent the time that most students would take to finish a given course. It would be useful to test this same group again in 3-6 months to see how much of the course knowledge is retained over time. Possible future studies Comparisons to other online, self-study platforms There are many free, online self-guided learning systems now available for students who are trying to improve their knowledge skills at a time and place that is most convenient for them. Pedago has designed Quantic’s learning system to provide a different experience from these other systems by focusing on “bite- sized” lessons that incorporate active learning. It would be interesting to conduct pre-/post- evaluations with students taking specific subject matter courses on other platforms with those completing the courses in Quantic. This comparison would help to normalize some conditions that were not available in this study, including independent vs. classroom-based learning, and digital vs. analog learning.
Appendices Appendix A: Eligibility funnel Screen participants (810) ACCOUNTING QUANTIC Pre-test taken (62) MBA invited to participate (107) QUANTIC Coursework started (49) QUANTIC Analysis group (34) MBA Analysis group (51) FINANCE MBA Take post- test (54) QUANTIC take pre-test (62) QUANTIC Coursework completed and post-test taken (43) QUANTIC Start coursework started (47) QUANTIC Analysis group (43) MBA Analysis group (54) QUANTIC Coursework completed and post-test taken (40) MBA Invited to participate (95) MBA Take post-test (51) QUANTIC Invite to participate (93) QUANTIC Invite to participate (119)
Appendix B: Accounting - Detailed results Quantic Accounting group results (N = 34) Mean Median Mode Minimum Maximum Pre-test scores 57% 54% 54% 27% 80% Post-test scores 86% 89% 98% 49% 100% % Change 59% 46% 38% - 9% 191% Point Change 29 27 27 - 5 61 Quantic Accounting individual participant results (N = 34) Student Pre-test Post-test % change Point change Minutes to complete Quantic course Retook Quizzes Quantic Acctg Student 1 34% 61% 79% 27 135 Yes Quantic Acctg Student 2 61% 88% 44% 27 173 Yes Quantic Acctg Student 3 78% 98% 25% 20 126 No Quantic Acctg Student 4 46% 90% 95% 44 99 No Quantic Acctg Student 5 54% 95% 77% 41 179 No Quantic Acctg Student 6 54% 90% 68% 37 176 No Quantic Acctg Student 7 63% 93% 46% 29 81 No Quantic Acctg Student 8 49% 93% 90% 44 101 No Quantic Acctg Student 9 49% 71% 45% 22 233 Yes Quantic Acctg Student 10 68% 95% 39% 27 342 No Quantic Acctg Student 11 27% 78% 191% 51 155 Yes Quantic Acctg Student 12 68% 85% 25% 17 157 Yes Quantic Acctg Student 13 71% 98% 38% 27 134 No Quantic Acctg Student 14 71% 98% 38% 27 122 Yes Quantic Acctg Student 15 66% 90% 37% 24 141 No Quantic Acctg Student 16 51% 78% 52% 27 95 Yes Quantic Acctg Student 17 66% 98% 48% 32 123 Yes
