Perceived Differences Between Classroom and Distance Education: Seeking Instructional Strategies for Learning Applications
- Doo H. Lim, University of Tennessee
Abstract
In an effort to compare the issues in learning, application, and instructional design factors between different instructional delivery formats in Internet, classroom, and satellite-based system, an HRD course of a university was studied. Data analysis was conducted to compare the learning and application difference and the reasons for high or low learning and application were identified and categorized. The effectiveness of several instructional design factors was identified. From the findings, issues in learning and application and instructional design strategies to enhance learning and application were discussed.
Introduction
In comparing learning difference between classroom instruction and distance education method, several studies have revealed that there is no significant difference between the two (Relan, A. & Gillani, 1997: Russell, 1999; Wentling & Johnson, 1999). Even though this research finding has been widely accepted and cited by other studies, some researchers argue this kind of media comparison studies are not valid because each study's attributes, characteristics, and learner needs are not reflected to measure the learning outcomes (Lockee, Moore, & Burton, 2001). Claiming the validity of the studies of the 'no significant learning difference' between the different delivery media might be arguable, many comparative studies have been considered valuable because they have identified instructional conditions and variables enhancing technology mediated learning and instruction. While most of these comparative studies have focused on the learning aspect between the different delivery formats, seldom has been conducted to identify the difference in learning application between classroom and distance instruction (Ravitz, 1997; Wentling & Johnson, 1999). Here, the term 'application of learning' refers to the degree to which learners use and apply learned knowledge and skills to their current studies or to current and future jobs and tasks. One problematic situation in higher education is that most classroom and distance instructions have focused on learning of the subject content of a course but not on applying and transferring the learned content to learner's personal career. Considering the nationwide initiative of "school-to-work transition" in education, the lack of learning application becomes a critical issue to overcome in higher education.
Along with the learning application issue, promoting a high degree of learning and learning application in a virtual learning environment has become another important issue in higher education as more universities and colleges try to develop and deliver quality distance instruction. While we can apply many of the learning application strategies for classroom instruction to designing an applicable distance instruction, it is still a fact that distance instruction requires unique strategies and methodologies to meet the learning and application needs of distance learners in the virtual environment. Different delivery formats such as satellite-based and web-based instruction within distance education may even need to apply different instructional approaches to satisfy the different learning needs. To address these issues of learning and application in classroom and distance instruction, the findings of this kind of study are expected to reveal the reasons why certain delivery formats (i.e. web-based, satellite-based system) incur a high or low learning and learning application and seek for appropriate instructional strategies to enhance students’ learning that highly applicable to their jobs, tasks, and studies. The findings are also expected to identify instructional conditions and design factors promoting students’ learning and application of learning in online learning environment.
Theoretical Framework
Numerous instructional programs have owed much to behavioral learning theories for decades. Diverse learning strategies were stranded from the realm of behaviorism such as learning by doing, engagement in trial and error, and giving repetitive response (Cho, Golshani, & Park, 1999). Behavioral learning design was also proved to be effective in multimedia learning environment where the programmed instruction of stimuli-response-reinforcement process is applied (Jonassen, 1991). In cognitive tradition, learning is viewed as a process of perceiving and thinking. Cognitive learning strategies incorporate association and analogies for better learning (Cho et al., 1999). As a relatively recent learning theory, the central tenet of constructivism is constructing knowledge in complex learning environment by utilizing multiple metaphor, collaboration, and reflection (Boyle, 1997). To study interactivity in instructional events, many constructivists have tested the effect of diverse collaborative learning strategies in distance learning environment (Suthers & Jones, 1997; Collins, 1997).
One question arising from traditional learning theories is whether focusing only on learning rather than actual performance outcomes is meaningful for the client of any instructional programs including schools, employers, and learners. Some researchers even assert that learning without application is of no use since it lacks rich understanding of the learned concepts through meaningful application (Dolence, 1995; Laurillard, 1993). Imposing significant challenges on traditional learning theories, the shift of instructional focus on learning application has been a big thrust in higher education. Responding to this trend, several learning theories were evolved and studied by numerous researchers in distance as well as classroom learning environment. Reflective learning theory is one example adopting action oriented approaches (Kemmis, 1985) and stretching learning experiences into new appreciations of novel situation (Baud, Keogh, & Walker, 1985) and working knowledge (Schön, 1987). Situated learning theories press the critical role of real-world practice during learning (Lave & Wenger, 1991) by engaging learners in authentic, problem-solving situations (Lave, 1988; Vygotsky, 1978). In problem-based learning, realistic problems are believed to provide motivation to apply learning to the real situation (Sfard, 1998). Similar to these perspectives, collaborative learning theorists emphasize the co-construction process of student learning and idea sharing to solve real life problems (Crook, 1994). The collaborative learning process is characterized by diverse learning activities such as observation, interpretation, construction, contextualization, multiple manifestation, ownership of knowledge, and self-awareness of learning process (Black & McClintock, 1996; Honebein, 1996).
In recent studies, it was identified that a high degree of learning does not always result in a high degree of application because there are many factors inhibiting application of learning during and after an instruction (Foxon, 1997). To enhance learning application for an instruction, Baldwin and Ford (1988) recommend three instructional strategies: identical elements between the instruction and application settings, stimulus variability in the instruction, and teaching of general principles and rules. Other researchers also identified several strategies that positively influence application of learning. Those are overlearning (Hagman & Rose, 1983), goal setting (Wexley & Nemeroff, 1975), action plans (Foxon, 1997), and tutoring and coaching (Huczynski, 1989).
Methodology
Study Purpose
The purpose of this study is to compare the perceived degree of learning and application of learning made by different groups of students who have taken a college course either in classroom, web-based, or satellite-based delivery format respectively and identify the reasons for high or low learning and application. The study also expects to find out what instructional strategies and design factors affect students’ higher learning and application.
Subjects
To explore learning and learning application made by the different groups of students, an instructional systems design course at a mid-western university has been studied. The subjects for the study included nineteen HRD (Human Resource Development) major students who took the course in the three different delivery formats (web-based instruction, classroom, satellite-based instruction) respectively. All students were undergraduate students and sixteen students among them had part- or full-time occupations during the instruction. The subjects were composed of ten male and nine female students. Eight students had taken the course through web-based instruction, six students through satellite-based instruction, and five students through classroom instruction respectively. All students were given an option to select one of the three delivery formats voluntarily during the course registration period.
The instructional systems design course was designed for the three delivery formats and delivered by an instructor. Regarding the content development of the course, the instructor developed thirteen learning modules for web-based instruction and prepared classroom instruction based on the web-based instructional content. The classroom instruction was delivered in a multimedia classroom equipped with satellite-based delivery capability where the classroom and the satellite-based instructions were implemented simultaneously. The satellite-based system utilized one-way video and two-way audio communication system for instruction.
Data Collection
An online questionnaire was developed to obtain the students’ perceived degree of learning and application from the class. Students were asked to participate in the online survey conducted at the end of the semester. The questionnaire included question items composed of the thirty-four learning objectives extracted from the instructor’s lesson plans that were used during the semester. The question items used a five point Likert-type scale to measure the perceived degree of the learning and application. Regarding the reliability of the question items, Cronbach’s alpha test showed both question items (learning and application) are reliable (.98). The questionnaire asked reasons for high or low learning and learning application made by the students during the class. Some question items used fill-in blank boxes to ask students' opinions and qualitative information about the instruction. The content and construct validity of the instrument was evaluated and considered valid by the review of a researcher who had extensive research experience in the university. A pilot test was conducted and minor changes such as corrections of typographic errors had been made to the questionnaire.
Data Analysis
The data analysis of the study utilized both quantitative and qualitative analysis. The units of analysis were: (1) the students’ perceived degree of learning, (2) their perceived degree of learning application, (3) the reasons why learning and application of their learning do or do not occur, and (4) the quality of instructional design factors promoting their learning and application. The differences in the degree of learning and application were analyzed using Kruskal-Wallis Test for the three student groups of different delivery formats. The reasons for high or low learning and application were identified and classified into categories. Students' course evaluation results analyzed by the university’s evaluation center were compared with the data analysis results of the study.
Limitations
Since the study results were analyzed from a data set acquired from a small population, the findings may not be generalized into a broad scope. The subjects also consist of those who have jobs during the instruction. Because of this, generalizing this study’s findings to traditional college undergraduate students may not be appropriate.
Findings
Learning Difference
In order to identify what was learned or not learned from the course, the students were asked to rate their perceived degree of learning for each of the thirty-four learning objectives of the course. As a group, the nineteen students indicated a fairly high perceived degree of learning. The population mean score for the perceived degree of learning was 4.04 on a 5 point scale (scale range was 1 to 5), which indicates “mostly understood” in the rating scale of the questionnaire. Further analysis was conducted to examine the differences in the degree of perceived learning between the student groups in terms of three different delivery formats. In calculating the perceived degree of learning difference, the mean difference was not found to be a significant one (p=.995).
Table 1 Learning Difference
N M SD pa
Web 8 4.07 0.60 .995
Classroom 5 4.01 0.91
Satellite 6 4.16 0.85
a Exact significant value (1-tailed significance).
From the qualitative analysis, several reasons have been identified as the factors to promote high learning. Among them, instructional effectiveness, instructor effectiveness, and learner motivation were found as the major categories for the reasons to enhance students’ learning (accounted for 70% of the reasons for high learning). Some example reasons of the instructional effectiveness are: instructional design, interactive interface, and learning activities. According to the different delivery formats certain reasons were more frequently replied from a certain group. For example, reasons in instructional effectiveness were mostly replied from the web-base instruction group while reasons in instructor effectiveness were mostly replied from the satellite-based instruction group.
Table 2 Reasons for High Learning
Reasons Web Classroom Satellite Sum %
Instructional effectiveness 12 2 14 28
Instructor effectiveness 1 1 9 11 22
Instructor preparedness 2 2
Instructor's subject expertise 2 2
Clear presentation 1 4 5
Good facilitation 1 1
Enthusiasm and high involvement 1 1
Motivation to learn 5 1 4 10 20
Applicable to my jobs and tasks 6 1 7 14
Previous understanding of the topics 2 2 2 6 12
Association to other classes 2 2 4
Total response 26 4 20 50 100
As shown in Table 3, several reasons were found to inhibit the student learning. Some instructional design factors and personal reasons seemed to negatively influence the student learning.
Table 3 Reasons for Low Learning
Reasons Web Classroom Satellite Sum %
Instructional design issues 6 1 3 10 56
Content was confusing or unclear 3 2 5
Non applicable content 3 1 4
Too much information for a class 1 1
Personal reasons 3 3 6 33
Miss the classes 2 2
Lack of interest 2 2
Short of time to study 1 1
Short attention span 1 1
Lack of opportunity to use 1 1 2 11
Total response 10 4 4 18 100
Learning Application
In calculating the perceived degree of learning application, the mean scores of all students’ perceived degree of learning application was 3.89, which can be interpreted as "frequently applied". In comparing the perceived degree of learning and learning application made by the students, there was a high relationship between the two. The Pearson’s correlation between the mean scores of learning and learning application showed a high degree of relationship (.896) at the 0.01 level (2-tailed). The perceived application difference made by the three groups of different delivery formats was not a significant one (p=.828).
Table 4 Learning Application
N M SD pa
Web 8 3.97 0.58 .828
Classroom 5 4.06 0.92
Satellite 6 3.80 1.09
a Exact significant value (1-tailed significance).
The reasons for high application of the learned content varied. First, the most frequently replied reason was that the learning content was constructed in such a way that it could be applied to students’ studies and current job tasks. Opportunity to use the learning on the jobs and during the instruction was another major category of reasons for high application. High understanding and interest toward the learning content were also found as another category of reasons promoting high application of the learning.
Table 5 Reasons for High Application
Reasons Web Classroom Satellite Sum %
Applicable content 2 2 3 7 23
Opportunity to use on the job or else where 3 2 1 6 19
Opportunity to use during the class 4 2 6 19
High understanding 4 2 6 19
High interest 1 4 5 16
Content was easy to apply 1 1 4
Total response 14 7 10 31 100
Several reasons were found to negatively affect the application of student learning. Those were low degree of learning, not applicable to the students’ jobs and tasks, lack of opportunity to use, and lack of interest.
Table 6 Reasons for Low Application
Reasons Web Classroom Satellite Sum %
Low learning 3 1 1 5 38
Not applicable to my job 5 5 38
Lack of opportunity to use 1 1 2 16
Lack of interest 1 1 8
Total response 10 1 2 13 100
Instructional Design Components
When the students were asked to describe their specific learning experience from the instruction, diverse opinions and feelings could be collected and categorized. Of the total fifty-six qualitative comments made by the students, twenty-eight responses were directly related to instructional design issues. The other twenty-eight responses indicated students’ satisfaction with the current status or no need to change. From the content analysis of the 28 comments in instructional design issues, the most expressed concern was group related learning issue (7) followed by communication and interaction issue (6), learning content issue (3), and learning practice activities (2). In this section, the number of each parenthesis indicates the frequency of students’ responses for each category. In group learning issue, various opinions were expressed: need of more group works (4), solutions for schedule conflicts between group members to complete group project (3), and more opportunities for group communication (2). Need for more frequent interaction and learning activities (6) was another instructional issue perceived important for the students to actively engage in learning and application. One distinct difference in the student learning experience between the distance and classroom delivery format was that the students in distance delivery methods (web and satellite-based) had expressed more concerns on group work, communication, and interaction than the students who studied the course through classroom format. Students in classroom group seemed satisfied with the current instructional method which heavily depended on lecture format.
Regarding the instructional design components of web-based delivery format, all eight students seemed to be satisfied with the general web design. The text design of the web instructional modules was indicated to be clear and easy to understand the learning content. Except a few graphics that were not directly related to the learning content, most graphics used for the learning modules were said to be attractive and represent main concepts of the learning content. Use of multimedia in the learning module seemed mostly satisfactory (6), but a student wanted to have more frequent multimedia interactions for improved learning. Regarding the use of online form for instructional interaction between the students and the instructor, seven students replied the form interaction supported learning feedback to a great degree. For the general degree of interaction among the students and between the instructor and the students, four students felt the level of interaction was appropriate while three students did not. Group work was considered as the most effective motivator for student interaction and communication. In asking the benefits of taking the online course, five students replied flexible study schedule, two students indicated the distance feature of studying anywhere, and one student indicated the self-paced mode of learning as their benefits respectively. Disadvantages expressed were procrastination (2), lack of face-to-face interaction (2), inappropriate weekly module update (1), ineffective chat communication (1), and confusion of the online learning process (1).
Discussion
Several themes emerge from the data analysis of the study. These themes and their resultant implications are organized around two areas: learning/application and instructional design issues.
Learning and Application
From the data analysis, it was identified that there was not a significant difference in the perceived degree of learning between the different delivery groups. This finding supports Russell’s (1999) research study claiming there is “no significant difference” in learning between the classroom and distance instruction in general. One meaningful finding from this study is the fact that the “no significance” symptom occurs even in the application of learning between the comparison groups. From this finding, it can be argued that attaining certain level of learning and application of learning from a course may not be severely influenced by the different delivery formats. Rather, it may be influenced more often by the instructional design factors and strategies that decide the quality of the instruction.
Applicability of learning content seems to be an important issue to enhance student’s learning experience regardless the types of delivery format from the study. To make a learning experience meaningful for students in higher education, the learning content needs to be “applicable.” Designing an instruction with a focus on learning or learning application uses quite different approaches from the needs assessment to designing learning activities and evaluation (Boyd, Boll, Brawner, & Villaumer, 1998). When focusing on learning only, instructional designers tend to use learning activities that sustain the memory of facts, concepts, procedure, and skills based on competency-based criteria, which is considered inappropriate for teaching today’s changing and growing body of new knowledge (Herrington, Herrington, & Oliver, 1999). When focusing on learning application, instructional designers can provide generic and reflective skills that assist learners to apply those skills in novel situations, which will result in far transfer (Clark & Taylor, 1992). To create meaningful learning experience through applicable moments of student learning, several strategies for instructional design are advised. First, instructional content need to be constructed plain and simple enough to be applied to students’ studies and current tasks involved. Second, as Baldwin and Ford (1988) recommend, the learning content needs to be identical or at least similar to the actual application settings. Third, to promote near and far transfer of student learning, diverse learning activities are deemed effective. A step-by-step guided practice after a segment of instruction followed by an individual practice is one effective way for near transfer of learning. Developing independent practice through class assignments that have similar construct but different application content is another strategy to promote application of the learned content not only to similar but also to different context (far transfer).
Instructional Design Issues
Promoting higher interaction during learning has become a major instructional design issue in distance education (Jones & Jo, 1998). The need for meaningful interaction in student learning was not an exception from this study’s findings. Several communication problems seemed to negatively affect the meaningful interaction in web-based delivery system. First, one conflicting issue of group communication of the study was the different perceptions about the group communication methods between the students. Some students needed more communications for group works while other students were dissatisfied with the group communications tools due to the inefficiency in the communication process and the poor quality of the conversation. The expressed concerns in synchronous chat sessions were scheduling conflicts between group members and lack of meaningful learning interaction between the students. The problems identified in asynchronous communication tools were lack of immediacy and procrastination tendency experienced during threaded discussion sessions. To address these problems, different solutions can be applied to synchronous or asynchronous communication tools. For chat sessions, providing predetermined schedules for group chat sessions at the beginning of a semester is one way to solve the problem of schedule conflicts. Providing students with specific guidelines for participation and clear expected outcomes is another solution to assure the quality of synchronous chat session. For asynchronous discussion sessions, specifying the purpose, due dates, and grading policies for online discussion sessions is an effective way to encourage students’ participation in group communication. From the researcher’s experience, no credit for group discussion activity tends to lead to low student participation.
To sustain higher degree of learning and application during online instruction, various learning principles from cognitive, behaviorists’ and constructivists’ theories can be adapted and applied. First, from constructivists’ viewpoints, embedding learning activities in actual problem situation can be achieved by using virtual case studies and scenario analysis. Using multiple modes of representation of information through text, graphics, audio, and video is also an effective strategy to provide rich understanding of the learning content. Completion of group and individual projects in learning related environment is another method applying situated learning principles for learning application. Giving opportunities for reflection through review questions that ask applicable situations and examples of learned content is an effective technique adapting constructivists’ reflective strategies. Asking short questions checking the understanding of major learning content at frequent intervals is an example learning activity borrowing from behavioral orientation. Sending immediate feedback on the students’ questions, learning activities, and learning progresses is a good reward practice enhancing students’ motivation. One instructional design example representing cognitive learning theory, used by the researcher, was asking students to take a learning review quiz at the beginning of each week’s learning module in an intention to promote longer memory of the learned content. Evaluation of peer students’ class assignments is another method utilizing meta-cognitive learning strategy.
Conclusion
This study has identified several facts about college students’ learning and application experience occurred in an HRD course from a mid-western university. Even though the size of the population limits the generalization of the study results, several issues in instructional design were stemming from the study findings and possible solutions and recommendations were discussed to address these issues. One major contribution of this study to the related study fields is that the symptom of no significant difference between face to face and distance instruction is found not only for the learning but also for the learning application. The generalization of this finding, however, may need another set of studies using a broader population. Another distinct benefit acquired from this study is that the findings provide basis to choose various instructional strategies to enhance application of learning in distance learning environment. As instructors or instructional designers, our major concern is then to test and wisely apply these instructional strategies to design an effective instruction. The effectiveness of the instruction, however, will be dominantly affected by the level of our experience, insight toward these instructional strategies, and concerns for more meaningful application of these strategies in our instruction.
References
Baldwin, T. T., & Ford, J. K. (1988). Transfer of training: A review and directions for future research. Personnel Psychology, 4(1), 63-105.
Baud, D., Keogh, R., & Walker, D. (1985). Promoting reflection in learning: A model. In D. Boud, R. Keogh, & Y. D. Walker (Eds.), Reflection: Turning Experience into Learning (pp. 18-40). London: Kogan Page.
Black, J. B., & McClintock, R. O. (1996). An interpretation construction approach to constructivist design. In B. G. Wilson &
D. N. Perkins (Eds.), Constructivist Learning Environments: case studies in instructional design (pp. 25-32). Englewood Cliffs, NJ: Educational Technology Publications.
Boyd, P., Boll, M., Brawner, L., & Villaumer, S.K. (1998). Becoming reflective professionals: An exploration of preservice teachers’ struggles as they translate language and literacy theory into practice. Action in Teacher Education, 19(4), 61-75.
Cho, S. H., Golshani, F.,& Park, Y. C. (1999). Multimedia technologies in education of mathematics: An experiment with Pythagorean numbers. In B. Collis & R. Oliver (Eds), Proceedings Ed-Media ’99 (pp. 540-545). Seattle, WA: Association for the Advancement of Computing in Education.
Clark, R., & Taylor, D. (1992). Training problem solving skills using cognitive strategies: Novice versus expert problem solvers. Performance and Instruction, 31(3), 32-37.
Collis, B. (1997). Pedagogical re-engineering: A new approach to course enrichment and re-design with the WWW. Educational Technology Review, 8, 11-15.
Crook, C. (1994). Computers and the Collaborative Experience of Learning. London: Routledge.
Dolence, M. G. (1995). Transforming Higher Education: A Vision for Learning in the 21st Century. New York: Society for Colleges and Universities.
Foxon, M. J. (1997). The influence of motivation to transfer, action planning, and manager support on the transfer process. Performance Improvement Quarterly, 10(2), 42-63.
Hagman, J. D., & Rose, A. M. (1983). Retention of military tasks: A review. Human Factors, 25(2), 199-213.
Herrington, T., Herrington, J., & Oliver, R. (1999). Providing reflective online support for preservice teachers on professional practice in schools. In B. Collis & R. Oliver (Eds), Proceedings Ed-Media ’99 (pp. 166-171). Seattle, WA: Association for the Advancement of Computing in Education.
Honebein, P. C. (1996). Seven goals for the design of constructivist learning environments. In B. G. Wilson & D. N. Perkins (Eds.), Constructivist Learning Environments: case studies in instructional design (pp. 11-24). Englewood Cliffs, NJ: Educational Technology Publications.
Huczynski, A. A. (1989). Training designs for organizational change. Management Decision, 27(4), 27-35.
Jonassen, D. (1991). Objectivism versus constructivism: Do we need a new philosophical paradigm? Educational Technology Research and Development, 39(3), 5-14.
Jones, V., & Jo. J. H. (1998). Interactive multimedia based on learning theories to enhance tertiary education, Proceedings of ICCIMA ’98, Australia.
Kemmis, S. (1985). Action research and the politics of reflection. In D. Boud, R. Keogh, & D. Walker (Eds.), Reflection: Turning experience into learning (pp. 139-163). London: Kogan Page.
Lave, J. (1988). Cognition in Practice: Mind, mathematics, and culture in everyday life. Cambridge, UK: Cambridge University Press.
Lave, J., & Wenger, E. (1991). Situated learning: Legitimate peripheral participation. Cambridge: Cambridge University Press.
Laurillard, D. (1993) Rethinking university teaching: A framework for the effective use of educational technology. London and New York: Routledge.
Ravitz, J. (1997). Evaluating learning networks: A special challenge for web-based instruction. In B. H. Khan (Ed.), Web-based instruction (pp. 361-368). Englewood Cliffs, JN: Educational Technology Publications.
Relan, A., & Gillani, B. (1997). Web-based instruction and the traditional classroom: Similarities and differences. In B. Khan (Ed.), Web-based instruction (pp. 41-47). Englewood Cliffs, JN: Educational Technology Publications.
Russell, T. L. (1999, September 15). No significant difference phenomenon [WWW document]. URL http://cuda.teleeducation.nb.ca/nosignificantdifference
Schön, D. A. (1987). Educating the reflective practitioner. San Francisco: Jossey-Bass.
Sfard, A. (1998). On two metaphors for learning and the dangers of choosing just one. Educational Research, 27(2), 4-12.
Suthers, D., & Jones, D. (1997). An architecture for intelligent collaborative educational systems, In D. Bovlay & Mizoguchi (Eds.), Proceedings of AI-ED '97 (pp. 55-62).
Tielemans, G., & Collis, B. (1999). Strategic requirements for a system to generate and support WWW-based environments for a faculty. In B. Collis & R. Oliver (Eds.), Proceedings of Ed-Media ’99 (pp. 346-351). Seattle, WA: Association for the Advancement of Computing in Education.
Vygotsky, L. S. (1978). Mind in society: The development of higher psychological processes. (M.Cole, V. John-Steiner, S. Scribner, & E. Souberman, Editors and Translators). Cambridge, MA: Harvard University Press.
Wentling, T., & Johnson, S. (1999). The design and development of an evaluation system for online instruction. In P. Kuchinke (Ed.), Proceedings of the 1999 Academy Human Resource Development Annual Conference (pp. 548-553). Washington, D.C.: Academy of HRD.
Wexley, K. N., & Nemeroff, W. (1975). Effectiveness of positive reinforcement and goal setting as methods of management development. Journal of Applied Psychology, 64, 239-246.
________________________________________
IJET Homepage | Article Submissions | Editors | Issues
Copyright © 2002. All rights reserved.
Last Updated on 20 June 2002
For additional information, please us at ijet@ncsa.uiuc.edu
Abstract
In an effort to compare the issues in learning, application, and instructional design factors between different instructional delivery formats in Internet, classroom, and satellite-based system, an HRD course of a university was studied. Data analysis was conducted to compare the learning and application difference and the reasons for high or low learning and application were identified and categorized. The effectiveness of several instructional design factors was identified. From the findings, issues in learning and application and instructional design strategies to enhance learning and application were discussed.
Introduction
In comparing learning difference between classroom instruction and distance education method, several studies have revealed that there is no significant difference between the two (Relan, A. & Gillani, 1997: Russell, 1999; Wentling & Johnson, 1999). Even though this research finding has been widely accepted and cited by other studies, some researchers argue this kind of media comparison studies are not valid because each study's attributes, characteristics, and learner needs are not reflected to measure the learning outcomes (Lockee, Moore, & Burton, 2001). Claiming the validity of the studies of the 'no significant learning difference' between the different delivery media might be arguable, many comparative studies have been considered valuable because they have identified instructional conditions and variables enhancing technology mediated learning and instruction. While most of these comparative studies have focused on the learning aspect between the different delivery formats, seldom has been conducted to identify the difference in learning application between classroom and distance instruction (Ravitz, 1997; Wentling & Johnson, 1999). Here, the term 'application of learning' refers to the degree to which learners use and apply learned knowledge and skills to their current studies or to current and future jobs and tasks. One problematic situation in higher education is that most classroom and distance instructions have focused on learning of the subject content of a course but not on applying and transferring the learned content to learner's personal career. Considering the nationwide initiative of "school-to-work transition" in education, the lack of learning application becomes a critical issue to overcome in higher education.
Along with the learning application issue, promoting a high degree of learning and learning application in a virtual learning environment has become another important issue in higher education as more universities and colleges try to develop and deliver quality distance instruction. While we can apply many of the learning application strategies for classroom instruction to designing an applicable distance instruction, it is still a fact that distance instruction requires unique strategies and methodologies to meet the learning and application needs of distance learners in the virtual environment. Different delivery formats such as satellite-based and web-based instruction within distance education may even need to apply different instructional approaches to satisfy the different learning needs. To address these issues of learning and application in classroom and distance instruction, the findings of this kind of study are expected to reveal the reasons why certain delivery formats (i.e. web-based, satellite-based system) incur a high or low learning and learning application and seek for appropriate instructional strategies to enhance students’ learning that highly applicable to their jobs, tasks, and studies. The findings are also expected to identify instructional conditions and design factors promoting students’ learning and application of learning in online learning environment.
Theoretical Framework
Numerous instructional programs have owed much to behavioral learning theories for decades. Diverse learning strategies were stranded from the realm of behaviorism such as learning by doing, engagement in trial and error, and giving repetitive response (Cho, Golshani, & Park, 1999). Behavioral learning design was also proved to be effective in multimedia learning environment where the programmed instruction of stimuli-response-reinforcement process is applied (Jonassen, 1991). In cognitive tradition, learning is viewed as a process of perceiving and thinking. Cognitive learning strategies incorporate association and analogies for better learning (Cho et al., 1999). As a relatively recent learning theory, the central tenet of constructivism is constructing knowledge in complex learning environment by utilizing multiple metaphor, collaboration, and reflection (Boyle, 1997). To study interactivity in instructional events, many constructivists have tested the effect of diverse collaborative learning strategies in distance learning environment (Suthers & Jones, 1997; Collins, 1997).
One question arising from traditional learning theories is whether focusing only on learning rather than actual performance outcomes is meaningful for the client of any instructional programs including schools, employers, and learners. Some researchers even assert that learning without application is of no use since it lacks rich understanding of the learned concepts through meaningful application (Dolence, 1995; Laurillard, 1993). Imposing significant challenges on traditional learning theories, the shift of instructional focus on learning application has been a big thrust in higher education. Responding to this trend, several learning theories were evolved and studied by numerous researchers in distance as well as classroom learning environment. Reflective learning theory is one example adopting action oriented approaches (Kemmis, 1985) and stretching learning experiences into new appreciations of novel situation (Baud, Keogh, & Walker, 1985) and working knowledge (Schön, 1987). Situated learning theories press the critical role of real-world practice during learning (Lave & Wenger, 1991) by engaging learners in authentic, problem-solving situations (Lave, 1988; Vygotsky, 1978). In problem-based learning, realistic problems are believed to provide motivation to apply learning to the real situation (Sfard, 1998). Similar to these perspectives, collaborative learning theorists emphasize the co-construction process of student learning and idea sharing to solve real life problems (Crook, 1994). The collaborative learning process is characterized by diverse learning activities such as observation, interpretation, construction, contextualization, multiple manifestation, ownership of knowledge, and self-awareness of learning process (Black & McClintock, 1996; Honebein, 1996).
In recent studies, it was identified that a high degree of learning does not always result in a high degree of application because there are many factors inhibiting application of learning during and after an instruction (Foxon, 1997). To enhance learning application for an instruction, Baldwin and Ford (1988) recommend three instructional strategies: identical elements between the instruction and application settings, stimulus variability in the instruction, and teaching of general principles and rules. Other researchers also identified several strategies that positively influence application of learning. Those are overlearning (Hagman & Rose, 1983), goal setting (Wexley & Nemeroff, 1975), action plans (Foxon, 1997), and tutoring and coaching (Huczynski, 1989).
Methodology
Study Purpose
The purpose of this study is to compare the perceived degree of learning and application of learning made by different groups of students who have taken a college course either in classroom, web-based, or satellite-based delivery format respectively and identify the reasons for high or low learning and application. The study also expects to find out what instructional strategies and design factors affect students’ higher learning and application.
Subjects
To explore learning and learning application made by the different groups of students, an instructional systems design course at a mid-western university has been studied. The subjects for the study included nineteen HRD (Human Resource Development) major students who took the course in the three different delivery formats (web-based instruction, classroom, satellite-based instruction) respectively. All students were undergraduate students and sixteen students among them had part- or full-time occupations during the instruction. The subjects were composed of ten male and nine female students. Eight students had taken the course through web-based instruction, six students through satellite-based instruction, and five students through classroom instruction respectively. All students were given an option to select one of the three delivery formats voluntarily during the course registration period.
The instructional systems design course was designed for the three delivery formats and delivered by an instructor. Regarding the content development of the course, the instructor developed thirteen learning modules for web-based instruction and prepared classroom instruction based on the web-based instructional content. The classroom instruction was delivered in a multimedia classroom equipped with satellite-based delivery capability where the classroom and the satellite-based instructions were implemented simultaneously. The satellite-based system utilized one-way video and two-way audio communication system for instruction.
Data Collection
An online questionnaire was developed to obtain the students’ perceived degree of learning and application from the class. Students were asked to participate in the online survey conducted at the end of the semester. The questionnaire included question items composed of the thirty-four learning objectives extracted from the instructor’s lesson plans that were used during the semester. The question items used a five point Likert-type scale to measure the perceived degree of the learning and application. Regarding the reliability of the question items, Cronbach’s alpha test showed both question items (learning and application) are reliable (.98). The questionnaire asked reasons for high or low learning and learning application made by the students during the class. Some question items used fill-in blank boxes to ask students' opinions and qualitative information about the instruction. The content and construct validity of the instrument was evaluated and considered valid by the review of a researcher who had extensive research experience in the university. A pilot test was conducted and minor changes such as corrections of typographic errors had been made to the questionnaire.
Data Analysis
The data analysis of the study utilized both quantitative and qualitative analysis. The units of analysis were: (1) the students’ perceived degree of learning, (2) their perceived degree of learning application, (3) the reasons why learning and application of their learning do or do not occur, and (4) the quality of instructional design factors promoting their learning and application. The differences in the degree of learning and application were analyzed using Kruskal-Wallis Test for the three student groups of different delivery formats. The reasons for high or low learning and application were identified and classified into categories. Students' course evaluation results analyzed by the university’s evaluation center were compared with the data analysis results of the study.
Limitations
Since the study results were analyzed from a data set acquired from a small population, the findings may not be generalized into a broad scope. The subjects also consist of those who have jobs during the instruction. Because of this, generalizing this study’s findings to traditional college undergraduate students may not be appropriate.
Findings
Learning Difference
In order to identify what was learned or not learned from the course, the students were asked to rate their perceived degree of learning for each of the thirty-four learning objectives of the course. As a group, the nineteen students indicated a fairly high perceived degree of learning. The population mean score for the perceived degree of learning was 4.04 on a 5 point scale (scale range was 1 to 5), which indicates “mostly understood” in the rating scale of the questionnaire. Further analysis was conducted to examine the differences in the degree of perceived learning between the student groups in terms of three different delivery formats. In calculating the perceived degree of learning difference, the mean difference was not found to be a significant one (p=.995).
Table 1 Learning Difference
N M SD pa
Web 8 4.07 0.60 .995
Classroom 5 4.01 0.91
Satellite 6 4.16 0.85
a Exact significant value (1-tailed significance).
From the qualitative analysis, several reasons have been identified as the factors to promote high learning. Among them, instructional effectiveness, instructor effectiveness, and learner motivation were found as the major categories for the reasons to enhance students’ learning (accounted for 70% of the reasons for high learning). Some example reasons of the instructional effectiveness are: instructional design, interactive interface, and learning activities. According to the different delivery formats certain reasons were more frequently replied from a certain group. For example, reasons in instructional effectiveness were mostly replied from the web-base instruction group while reasons in instructor effectiveness were mostly replied from the satellite-based instruction group.
Table 2 Reasons for High Learning
Reasons Web Classroom Satellite Sum %
Instructional effectiveness 12 2 14 28
Instructor effectiveness 1 1 9 11 22
Instructor preparedness 2 2
Instructor's subject expertise 2 2
Clear presentation 1 4 5
Good facilitation 1 1
Enthusiasm and high involvement 1 1
Motivation to learn 5 1 4 10 20
Applicable to my jobs and tasks 6 1 7 14
Previous understanding of the topics 2 2 2 6 12
Association to other classes 2 2 4
Total response 26 4 20 50 100
As shown in Table 3, several reasons were found to inhibit the student learning. Some instructional design factors and personal reasons seemed to negatively influence the student learning.
Table 3 Reasons for Low Learning
Reasons Web Classroom Satellite Sum %
Instructional design issues 6 1 3 10 56
Content was confusing or unclear 3 2 5
Non applicable content 3 1 4
Too much information for a class 1 1
Personal reasons 3 3 6 33
Miss the classes 2 2
Lack of interest 2 2
Short of time to study 1 1
Short attention span 1 1
Lack of opportunity to use 1 1 2 11
Total response 10 4 4 18 100
Learning Application
In calculating the perceived degree of learning application, the mean scores of all students’ perceived degree of learning application was 3.89, which can be interpreted as "frequently applied". In comparing the perceived degree of learning and learning application made by the students, there was a high relationship between the two. The Pearson’s correlation between the mean scores of learning and learning application showed a high degree of relationship (.896) at the 0.01 level (2-tailed). The perceived application difference made by the three groups of different delivery formats was not a significant one (p=.828).
Table 4 Learning Application
N M SD pa
Web 8 3.97 0.58 .828
Classroom 5 4.06 0.92
Satellite 6 3.80 1.09
a Exact significant value (1-tailed significance).
The reasons for high application of the learned content varied. First, the most frequently replied reason was that the learning content was constructed in such a way that it could be applied to students’ studies and current job tasks. Opportunity to use the learning on the jobs and during the instruction was another major category of reasons for high application. High understanding and interest toward the learning content were also found as another category of reasons promoting high application of the learning.
Table 5 Reasons for High Application
Reasons Web Classroom Satellite Sum %
Applicable content 2 2 3 7 23
Opportunity to use on the job or else where 3 2 1 6 19
Opportunity to use during the class 4 2 6 19
High understanding 4 2 6 19
High interest 1 4 5 16
Content was easy to apply 1 1 4
Total response 14 7 10 31 100
Several reasons were found to negatively affect the application of student learning. Those were low degree of learning, not applicable to the students’ jobs and tasks, lack of opportunity to use, and lack of interest.
Table 6 Reasons for Low Application
Reasons Web Classroom Satellite Sum %
Low learning 3 1 1 5 38
Not applicable to my job 5 5 38
Lack of opportunity to use 1 1 2 16
Lack of interest 1 1 8
Total response 10 1 2 13 100
Instructional Design Components
When the students were asked to describe their specific learning experience from the instruction, diverse opinions and feelings could be collected and categorized. Of the total fifty-six qualitative comments made by the students, twenty-eight responses were directly related to instructional design issues. The other twenty-eight responses indicated students’ satisfaction with the current status or no need to change. From the content analysis of the 28 comments in instructional design issues, the most expressed concern was group related learning issue (7) followed by communication and interaction issue (6), learning content issue (3), and learning practice activities (2). In this section, the number of each parenthesis indicates the frequency of students’ responses for each category. In group learning issue, various opinions were expressed: need of more group works (4), solutions for schedule conflicts between group members to complete group project (3), and more opportunities for group communication (2). Need for more frequent interaction and learning activities (6) was another instructional issue perceived important for the students to actively engage in learning and application. One distinct difference in the student learning experience between the distance and classroom delivery format was that the students in distance delivery methods (web and satellite-based) had expressed more concerns on group work, communication, and interaction than the students who studied the course through classroom format. Students in classroom group seemed satisfied with the current instructional method which heavily depended on lecture format.
Regarding the instructional design components of web-based delivery format, all eight students seemed to be satisfied with the general web design. The text design of the web instructional modules was indicated to be clear and easy to understand the learning content. Except a few graphics that were not directly related to the learning content, most graphics used for the learning modules were said to be attractive and represent main concepts of the learning content. Use of multimedia in the learning module seemed mostly satisfactory (6), but a student wanted to have more frequent multimedia interactions for improved learning. Regarding the use of online form for instructional interaction between the students and the instructor, seven students replied the form interaction supported learning feedback to a great degree. For the general degree of interaction among the students and between the instructor and the students, four students felt the level of interaction was appropriate while three students did not. Group work was considered as the most effective motivator for student interaction and communication. In asking the benefits of taking the online course, five students replied flexible study schedule, two students indicated the distance feature of studying anywhere, and one student indicated the self-paced mode of learning as their benefits respectively. Disadvantages expressed were procrastination (2), lack of face-to-face interaction (2), inappropriate weekly module update (1), ineffective chat communication (1), and confusion of the online learning process (1).
Discussion
Several themes emerge from the data analysis of the study. These themes and their resultant implications are organized around two areas: learning/application and instructional design issues.
Learning and Application
From the data analysis, it was identified that there was not a significant difference in the perceived degree of learning between the different delivery groups. This finding supports Russell’s (1999) research study claiming there is “no significant difference” in learning between the classroom and distance instruction in general. One meaningful finding from this study is the fact that the “no significance” symptom occurs even in the application of learning between the comparison groups. From this finding, it can be argued that attaining certain level of learning and application of learning from a course may not be severely influenced by the different delivery formats. Rather, it may be influenced more often by the instructional design factors and strategies that decide the quality of the instruction.
Applicability of learning content seems to be an important issue to enhance student’s learning experience regardless the types of delivery format from the study. To make a learning experience meaningful for students in higher education, the learning content needs to be “applicable.” Designing an instruction with a focus on learning or learning application uses quite different approaches from the needs assessment to designing learning activities and evaluation (Boyd, Boll, Brawner, & Villaumer, 1998). When focusing on learning only, instructional designers tend to use learning activities that sustain the memory of facts, concepts, procedure, and skills based on competency-based criteria, which is considered inappropriate for teaching today’s changing and growing body of new knowledge (Herrington, Herrington, & Oliver, 1999). When focusing on learning application, instructional designers can provide generic and reflective skills that assist learners to apply those skills in novel situations, which will result in far transfer (Clark & Taylor, 1992). To create meaningful learning experience through applicable moments of student learning, several strategies for instructional design are advised. First, instructional content need to be constructed plain and simple enough to be applied to students’ studies and current tasks involved. Second, as Baldwin and Ford (1988) recommend, the learning content needs to be identical or at least similar to the actual application settings. Third, to promote near and far transfer of student learning, diverse learning activities are deemed effective. A step-by-step guided practice after a segment of instruction followed by an individual practice is one effective way for near transfer of learning. Developing independent practice through class assignments that have similar construct but different application content is another strategy to promote application of the learned content not only to similar but also to different context (far transfer).
Instructional Design Issues
Promoting higher interaction during learning has become a major instructional design issue in distance education (Jones & Jo, 1998). The need for meaningful interaction in student learning was not an exception from this study’s findings. Several communication problems seemed to negatively affect the meaningful interaction in web-based delivery system. First, one conflicting issue of group communication of the study was the different perceptions about the group communication methods between the students. Some students needed more communications for group works while other students were dissatisfied with the group communications tools due to the inefficiency in the communication process and the poor quality of the conversation. The expressed concerns in synchronous chat sessions were scheduling conflicts between group members and lack of meaningful learning interaction between the students. The problems identified in asynchronous communication tools were lack of immediacy and procrastination tendency experienced during threaded discussion sessions. To address these problems, different solutions can be applied to synchronous or asynchronous communication tools. For chat sessions, providing predetermined schedules for group chat sessions at the beginning of a semester is one way to solve the problem of schedule conflicts. Providing students with specific guidelines for participation and clear expected outcomes is another solution to assure the quality of synchronous chat session. For asynchronous discussion sessions, specifying the purpose, due dates, and grading policies for online discussion sessions is an effective way to encourage students’ participation in group communication. From the researcher’s experience, no credit for group discussion activity tends to lead to low student participation.
To sustain higher degree of learning and application during online instruction, various learning principles from cognitive, behaviorists’ and constructivists’ theories can be adapted and applied. First, from constructivists’ viewpoints, embedding learning activities in actual problem situation can be achieved by using virtual case studies and scenario analysis. Using multiple modes of representation of information through text, graphics, audio, and video is also an effective strategy to provide rich understanding of the learning content. Completion of group and individual projects in learning related environment is another method applying situated learning principles for learning application. Giving opportunities for reflection through review questions that ask applicable situations and examples of learned content is an effective technique adapting constructivists’ reflective strategies. Asking short questions checking the understanding of major learning content at frequent intervals is an example learning activity borrowing from behavioral orientation. Sending immediate feedback on the students’ questions, learning activities, and learning progresses is a good reward practice enhancing students’ motivation. One instructional design example representing cognitive learning theory, used by the researcher, was asking students to take a learning review quiz at the beginning of each week’s learning module in an intention to promote longer memory of the learned content. Evaluation of peer students’ class assignments is another method utilizing meta-cognitive learning strategy.
Conclusion
This study has identified several facts about college students’ learning and application experience occurred in an HRD course from a mid-western university. Even though the size of the population limits the generalization of the study results, several issues in instructional design were stemming from the study findings and possible solutions and recommendations were discussed to address these issues. One major contribution of this study to the related study fields is that the symptom of no significant difference between face to face and distance instruction is found not only for the learning but also for the learning application. The generalization of this finding, however, may need another set of studies using a broader population. Another distinct benefit acquired from this study is that the findings provide basis to choose various instructional strategies to enhance application of learning in distance learning environment. As instructors or instructional designers, our major concern is then to test and wisely apply these instructional strategies to design an effective instruction. The effectiveness of the instruction, however, will be dominantly affected by the level of our experience, insight toward these instructional strategies, and concerns for more meaningful application of these strategies in our instruction.
References
Baldwin, T. T., & Ford, J. K. (1988). Transfer of training: A review and directions for future research. Personnel Psychology, 4(1), 63-105.
Baud, D., Keogh, R., & Walker, D. (1985). Promoting reflection in learning: A model. In D. Boud, R. Keogh, & Y. D. Walker (Eds.), Reflection: Turning Experience into Learning (pp. 18-40). London: Kogan Page.
Black, J. B., & McClintock, R. O. (1996). An interpretation construction approach to constructivist design. In B. G. Wilson &
D. N. Perkins (Eds.), Constructivist Learning Environments: case studies in instructional design (pp. 25-32). Englewood Cliffs, NJ: Educational Technology Publications.
Boyd, P., Boll, M., Brawner, L., & Villaumer, S.K. (1998). Becoming reflective professionals: An exploration of preservice teachers’ struggles as they translate language and literacy theory into practice. Action in Teacher Education, 19(4), 61-75.
Cho, S. H., Golshani, F.,& Park, Y. C. (1999). Multimedia technologies in education of mathematics: An experiment with Pythagorean numbers. In B. Collis & R. Oliver (Eds), Proceedings Ed-Media ’99 (pp. 540-545). Seattle, WA: Association for the Advancement of Computing in Education.
Clark, R., & Taylor, D. (1992). Training problem solving skills using cognitive strategies: Novice versus expert problem solvers. Performance and Instruction, 31(3), 32-37.
Collis, B. (1997). Pedagogical re-engineering: A new approach to course enrichment and re-design with the WWW. Educational Technology Review, 8, 11-15.
Crook, C. (1994). Computers and the Collaborative Experience of Learning. London: Routledge.
Dolence, M. G. (1995). Transforming Higher Education: A Vision for Learning in the 21st Century. New York: Society for Colleges and Universities.
Foxon, M. J. (1997). The influence of motivation to transfer, action planning, and manager support on the transfer process. Performance Improvement Quarterly, 10(2), 42-63.
Hagman, J. D., & Rose, A. M. (1983). Retention of military tasks: A review. Human Factors, 25(2), 199-213.
Herrington, T., Herrington, J., & Oliver, R. (1999). Providing reflective online support for preservice teachers on professional practice in schools. In B. Collis & R. Oliver (Eds), Proceedings Ed-Media ’99 (pp. 166-171). Seattle, WA: Association for the Advancement of Computing in Education.
Honebein, P. C. (1996). Seven goals for the design of constructivist learning environments. In B. G. Wilson & D. N. Perkins (Eds.), Constructivist Learning Environments: case studies in instructional design (pp. 11-24). Englewood Cliffs, NJ: Educational Technology Publications.
Huczynski, A. A. (1989). Training designs for organizational change. Management Decision, 27(4), 27-35.
Jonassen, D. (1991). Objectivism versus constructivism: Do we need a new philosophical paradigm? Educational Technology Research and Development, 39(3), 5-14.
Jones, V., & Jo. J. H. (1998). Interactive multimedia based on learning theories to enhance tertiary education, Proceedings of ICCIMA ’98, Australia.
Kemmis, S. (1985). Action research and the politics of reflection. In D. Boud, R. Keogh, & D. Walker (Eds.), Reflection: Turning experience into learning (pp. 139-163). London: Kogan Page.
Lave, J. (1988). Cognition in Practice: Mind, mathematics, and culture in everyday life. Cambridge, UK: Cambridge University Press.
Lave, J., & Wenger, E. (1991). Situated learning: Legitimate peripheral participation. Cambridge: Cambridge University Press.
Laurillard, D. (1993) Rethinking university teaching: A framework for the effective use of educational technology. London and New York: Routledge.
Ravitz, J. (1997). Evaluating learning networks: A special challenge for web-based instruction. In B. H. Khan (Ed.), Web-based instruction (pp. 361-368). Englewood Cliffs, JN: Educational Technology Publications.
Relan, A., & Gillani, B. (1997). Web-based instruction and the traditional classroom: Similarities and differences. In B. Khan (Ed.), Web-based instruction (pp. 41-47). Englewood Cliffs, JN: Educational Technology Publications.
Russell, T. L. (1999, September 15). No significant difference phenomenon [WWW document]. URL http://cuda.teleeducation.nb.ca/nosignificantdifference
Schön, D. A. (1987). Educating the reflective practitioner. San Francisco: Jossey-Bass.
Sfard, A. (1998). On two metaphors for learning and the dangers of choosing just one. Educational Research, 27(2), 4-12.
Suthers, D., & Jones, D. (1997). An architecture for intelligent collaborative educational systems, In D. Bovlay & Mizoguchi (Eds.), Proceedings of AI-ED '97 (pp. 55-62).
Tielemans, G., & Collis, B. (1999). Strategic requirements for a system to generate and support WWW-based environments for a faculty. In B. Collis & R. Oliver (Eds.), Proceedings of Ed-Media ’99 (pp. 346-351). Seattle, WA: Association for the Advancement of Computing in Education.
Vygotsky, L. S. (1978). Mind in society: The development of higher psychological processes. (M.Cole, V. John-Steiner, S. Scribner, & E. Souberman, Editors and Translators). Cambridge, MA: Harvard University Press.
Wentling, T., & Johnson, S. (1999). The design and development of an evaluation system for online instruction. In P. Kuchinke (Ed.), Proceedings of the 1999 Academy Human Resource Development Annual Conference (pp. 548-553). Washington, D.C.: Academy of HRD.
Wexley, K. N., & Nemeroff, W. (1975). Effectiveness of positive reinforcement and goal setting as methods of management development. Journal of Applied Psychology, 64, 239-246.
________________________________________
IJET Homepage | Article Submissions | Editors | Issues
Copyright © 2002. All rights reserved.
Last Updated on 20 June 2002
For additional information, please us at ijet@ncsa.uiuc.edu
Komentar