Profiles of Accomplished Projects at the Trial Course.
High-level information systems (IS) professionals such as project managers or system architects are needed in IT industries all over the world. They are required an adaptive expertise instead of a routine one to solve an ill-structured problem in the interdisciplinary field. However, people have no effective instructional method to develop the expertise yet.
In this chapter, we discuss how to develop such the expertise by proposing our trial model of PBL (Project-Based Learning). In the proposed model, an undergraduate students' project is managed by an IT company's engineer, who is also expected to acquire project management skills. She/he collaboratively learns with students how to develop a software product for real customers and users. We named this model as "Collaborative Management" approach.
In the following section two, we discuss the current issues to develop the IS engineers and managers with focusing in the Japanese situation. In section three and four, we introduce our trial model, a research scheme, and the results of the trial education. In section five and six, design principles and a conceptual model of our PBL model is proposed based on the results of our research and experiences.
2. Current Issues
Japanese IT industries are worried about a shortage of project managers and IS-engineers now. Although both information systems engineering and its management involve many human factors (Demarco, 1987, Weinberg, 1972), still main stream of the education in Japanese universities is a traditional CS approach. Our objective to improve this situation is to bring up those human factors for both university's students and engineers in industries.
To develop skills described above, PBL (Problem Based Learning/Project Based Learning) approach gets a lot of attention in ICT education domains in Japan (Inoue, 2007). PBL is an educational strategy, a method to organize the learning process in such a manner that the students are actively engaged in finding answers by themselves (Graff and Kolmos, 2007). PBL approach has been adopted for a long time in engineering education at Alborg (Kolmos, 2007). Recently, a PBL model for IS engineers (Leitch & Warren, 2007) or a model to develop managers (Stoyan 2007) has also been proposed.
However, PBL approach is not a best model to develop the expertise today. There are too many kind of PBL models all over the world, these models have not been verified well, because it is difficult to evaluate the achievements of the learning goal. Therefore, the formalized PBL model is undefined in the current situation.
The situation is considered as same as in Japan. MEXT (Ministry of Education in Japan) has been budgeted to several ICT engineers educational program from early 2000s, where PBL based curriculum for higher education has been developed. However, a lot of the model of PBL developed there was not verified and the model was not formalized at all.
Additionally, a lot of Japanese PBL model is scenario-based where students are given a particular scenario by teachers. That is as similar as the traditional educational style. We think that the PBL model to bring up an adaptive expertise have to be real situated (Lave & Wenger, 1991) and project managed by members themselves. However, facilitators well know that it is hard for undergraduate students to learn both subject-matter domain skills and project management skills at the same time (Batatia 2001).
Because of these described above, we have developed a trial course using project-based learning in which a project manager comes from an actual IT company. The purpose of this course is not only to develop the engineers for university students, but also to develop project managers through the experience of managing a real project. This is also one possible solution for the problem of scarce project managers in Japanese IT industries.
3. Proposed PBL Model and Research Scheme
3.1. The Model and Objectives of a Proposed PBL
We have developed a university's course where both undergraduate students and engineers in industry can learn the information systems development collaboratively through project-based learning. The model of a project in this course is as shown in Figure 1.
The project is composed of two to four students who have only learned basic programming skills and a project manager working in an IT company. They try to develop a small information system for real customers and users. The evaluation of the development is carried out by customers' satisfaction.
This course has two learning objectives as follows. 1) For students as members: to acquire IS-engineering competencies or adaptive expertise on it. 2) For project managers: to acquire project management competencies or adaptive expertise on it.
Managers visit the university once a week for the project meeting. During other days, they work at their companies. E-mail or other ICT tools are used for communication while the project managers are working in their company.
All projects are required to satisfy real customers and users. That means they will develop a system that described as "real users prefer to use", not just "real users may use". Popular customers projects selected were the owners of small shops/restaurants in the local area, or a professor who is a non-IT professional. These amateur customers often have ambiguous requirements and students must clarify into a coherent plan. Each project must choose their software engineering process so it is appropriate to meet a customer's satisfaction.
3.2. Learning Environment of the Course
The learning environment of the course is explained in Figure 2. Learning activities are mainly composed of reporting and discussing the processes that are carried out by learners to accomplish the project goals. Students can take the course repeatedly. We think it is important that students try the course with right feedback and reflection of the former project.
In the course, multiple projects run at the same time. Students and project managers can compare various kinds of methodologies and philosophies that are brought from project managers that belong to different companies.
A facilitator of the course gives hints and advice to the students and the manager for the obstacles in the project, without giving direct solutions to them. He let them be conscious to the purpose of each activity in the project without noticing whether learners' trial is successful or not. He also advises them to be careful in applying the technologies and methodologies.
An evaluation committee composed of leaders of the IT industry and academia is organized external of the course. They evaluate projects and the course from the viewpoint of experts in this field.
3.3. Research Method and Research Question
It costs long time to evaluate the educational model. Additionally, learning objectives of the proposed model is not simple knowledge or skills but adaptive competencies that it is difficult to evaluate achievements of the educational goal. Therefore, we adapted an action research (Mansell 1991) to evaluate and improve learners' achievement and the proposed PBL model comprehensively.
Field structure of our action research is explained in Figure 3. The model is composed of two parts. A top side of the model (Field A, B and Procedure E, F) is the general model for developing the information systems proposed by Kaminuma (Kaminuma 2001). Field B is a university. Students had been collected a variety of knowledge concerning information technology through activities including learning in classes. In Field A, the students observe domain specific world. Then students acquired the view for the world as analyzers, whereas the people, members and managers share the view for the world as users. Procedure E is an interface between the manager's view and the analyzer's one.
A bottom side of the model (Field C,B and Procedure G,H) has been extended by the authors to explain the proposed PBL model environment. Field C is the project management office (PMO) that established in the course, where manager's meeting takes place by once a week. The scheme for an analysis procedure (G) is as same as top side's one. In case of the project emergencies, managers and facilitators produce some solutions collaboratively. The knowledge analyzed reflects the management of the project by managers, and new curriculum or PBL model by facilitators. Finally, the analysis and design are repeated as
shown in the arrow mark (a) and (b). Research question of our action research described above is 1) How the proposed model works, 2) How and what managers and students bring up in this course, 3) What are the design principles or good practice for this model or general PBL environment.
4.1. Summary of the Results
We have examined the learning environment discussed above as a trial course in our university for the three semesters from 2005 to 2006. After 2007 until now (2009) the course is continued, however this article focuses the early three trials for clarifying the results. Seven companies participated in this program, and fifteen projects were carried out. Various kinds of projects with different process were accomplished. Profiles of these projects are shown in Table 1.
|Name||bers||Client||Users||Kind of Software||/Enhance||Language||(Iter Count)||L(Line)||Hour)|
|2005fA||3||Company||Employees||Web Application||New||Java||WaterFall||1363 S||225|
|2005fB||3||Company||Employees||Web Application||New||Java,VBA||WaterFall||1206 S||-|
|2005fC||3||Local Shop||Client's customer||Web Application||New||Java||WaterFall||440 S||-|
|2005fD||4||Professor||Students||Web Application||Enhance||PHP||Iterative(2)||1541 S||-|
|2005fE||3||Themselves||Students||Standalone Application||Enhance||Java||RUP(3)||20985 S||420|
|2006sA||3||Themselves||Students||Web Application||New||Java||Iterative(1)||5258 S||-|
|2006sB||3||Professor||Researcher||Standalone Application||New||Java||Iterative(2)||1329 S||314|
|2006sC||3||Professor||Teachers and Students||Web Application||New||PHP||Iterative(2)||1251 L||294|
|2006sD||4||Company||Handicapped people||Standalone Game||New||C++||Iterative(2)||6500 L||535|
|2006fA||3||Researcher||Researchers||A part of Web App lication||New||Java||WaterFall||1105 S||591|
|2006fB||3||Professor||Students||Web Application||Enhance||PHP||WaterFall||3500 S||496|
|2006fC||3||Professor||Researchers||Mobile Web Application||New||PHP||WaterFall||1133 S||491|
|2006fD||2||Company||Handicapped people||Standalone Game||Enhance||C++||GameProcess||2404 S||491|
|2006fE||3||Local Shop||Client's customer||Mobile Web Application||New||PHP,Java||RUP(3)||1723 S||374|
Steps of the programs are around thousand, and the time each project spent is a range of two hundreds to six hundreds hours, which is considered small in scale. As no real money was treated in this course, costs were measured only by time. Traditional waterfall process and iterative processes like RUP (Rational Unified Process; Jacobson et al., 1999) were adopted as the engineering processes under the influence of the project manager's company. One manager who qualified by PMP (Project Management Professional) applied PMBOK (Project Management Body of Knowledge; Project Management Institute, 2004) for a management process.
Final products were evaluated by the satisfaction of the end users. The results for customers' satisfaction are shown in Table 2. In Table 2, we have been appeared grades and its rubric to evaluate products where (A) is the highest grade, and (F) is the lowest. Numbers of the projects they achieved to the grade was written in the table.
|Rank||Rubric of the Rank||05f(Trial 1)||06s(Trial 2)||06f(Trial 3)|
|A||A product have been used until now in the real situation||1|
|B||A product was used limited situation by the real user||2|
|C||A product was appreciated by user-testing||2||2|
|D||A product was not appreciated by user-testing||2||3|
|E||A product didn't have user-testing (Demonstration only)||1||2|
|F||A product couldn't be developed|
As a result, almost every product was examined by an end users' testing. Until the 2005 spring semester, some of them were appreciated and one product had deployed for real situation and satisfied users. The others could have been released, if they had had quality enough to use for end users or there had been no misunderstanding of the client's requirement.
4.2. Qualitative Results Observed in the Trial
The project in the proposed PBL, resources are tightly limited. Managers have to manage their project while working in their company. Students have to follow the project while taking other courses in their university. Our tiny projects have as similar difficulty as an actual large-scale project with consideration from a view of resource scarcity. An evaluator with rich experience for developing information systems remarked that similar to large scale projects, common problems were found out in our tiny projects such as "Failure to acquire requirements", "Lack of member's motivation when members are not interested in the product", "Communication gap among the members", "One of the member is hospitalized", "Huge difference of member's skills", "Schedule delaying", "Poor quality", "Specification changing", "Deadline effect"...etc.
Projects must set the project scope as small as possible for poor students' development skills. However, setting the project scope too small causes the problem of decreasing motivation for members. Managers must make up the project scope with balance between the customer's requirement and members skills and motivation. To clarify the project scope is one of the most important works for the managers.
Difficult works for managers is balance between quality and delivery. Project members wish to spend more time to complete the system analysis to clarify their goals. However, to waste the time that should have been spent for the design and implementation makes the quality of the product poor. Such conflicts between students and managers were often observed.
Both skilled and unskilled managers participated this course gave us positive feedback for their learning. Especially, they appreciated that they can try a new management theory that is risky to try in the actual projects in industries.
On the other hand, Managers claimed about their role is not clearly defined whether mentor for students or manager for a project in the Trial 1. That makes managers confused to do decision making for their projects. Because the good mentors wait for solutions until students solve the problem themselves, but it might be caused the project delaying.
We have found that real managers make scaffolding students who have no experience of developing systems, by proposing engineering processes to them and leading project as a project manager. Figure 4 shows an example of a project schedule/results and costs they spent. RUP was applied to the project and the project have successfully finished.
We can show another example. Figure 5 shows an example of an unsuccessful project. All processes are delayed that causes reducing both quantities and qualities of their products. However, user testing was examined and finished. Therefore, final quality of their products was extremely higher than that produced by the project composed of students only. Additionally, the differences of the contributed time among students were small. That is considered effect of the project manager for the students.
In this trial, we have found that common communication-errors between developers and customers occurred in projects such as "Product was not accepted by user because of lack of needs analysis and the products are developed by their own impression", "A function that was quite important for the customers was missed", "Some functions had high priority for the customers, but developers fixed those as out of the scope", "Customer's requirements had too many details but those purposes were ambiguous"
Students learned the importance of requirement engineering from the failure of their first trial. Their processes were enhanced and refined from their trial and error. We have found that students had found good practices through the trial and errors described above. Students' descriptions in their project reports that are written after finishing their project were changed as is shown in Table 3.
|Lv. 1 Self-satisfaction||We had thought that we could develop the products only through perfect programming. (But, they often wrongly fix the specification and product wasn't used).|
|Lv. 2 Listen customer's requirements||We must not develop based on our wrong impression. We must listen the customer's requirements carefully.|
|Lv. 3 Propose the solution to customers||Customer's requirements are usually ambiguous. We should propose the solution through analyzing the customer's requirements.|
|Lv. 4 Collaborate with customers||We should discuss the specification deeply with customers and analyze user's requirements together.|
At the last in Trial 3, the reservation status viewing system for the therapeutic service shop has been developed by a project. The system has been used for several months and it is still on the service.
We can analyze the reason why they succeeded as follows. First, it was a good decision to take out the "make reservation" function from their project scope, because customers and many users are both amateurs for using computers. Therefore, customers have felt there was too much risk to make reservations by a system. Additionally, their business hours are not fixed, so customers wish to fix reservations by themselves. Secondly, it was a good decision to fix to architecture as cell phone based, because customers and many users have no computers, but almost all of them have cell phones. Their success is based on their survey from both developers and user's point of view and research.
4.3. Results of Questionnaire by Evaluators
The results of a questionnaire by evaluators for this course are shown in Table 4. Values in the table are average of evaluation by number five levels (One means poor, five means excellent). According to the results, this course has been appreciated by both Industry and academia. The result was almost same for each trial.
|Leaders from industries||Leaders from academia||Others||Total|
|Total value of this education||4.35||4.31||4.45||4.36|
|Value of education for managers||4.11||4.38||4.25||4.18|
|Value of education for students||4.61||4.46||4.58||4.58|
|Number of evaluators||46||13||12||71|
Values for project managers' education from Industry is a little lower than from academia. Values for IS-engineers education from academia is a little lower than from industries.
5. Summarize the Result of Action Research and Discussion
5.1. Summarize for the Result of Action Research
The author has done an action research on it and some improvements of the system have been made each time as described in Section.3. Summary of the result of that is shown in Table 5.
In the first trial, we have found that the experience of a managing students' project is effective for developing managers' competencies because the problems occurring in the projects are similar to those of actual large-scale projects. However, roles of the manager whether mentor for students or manager for a project is not clearly defined, which led managers confusing to do decision making for their projects. Additionally, "customer" was not clearly defined for students. One of the reasons why to do was considered that a customer and a manager were the same person in several projects. Another one was that project evaluation policy was not clearly indicated to students.
In the second trial, communication between a manager and students has been improved by actions A) project scope has shared between facilitators and managers with extending the manager meeting time twice B) clarifying the criterion for the evaluation of a project, that is, whether a product is accepted by real customers (not the same person as manager) or not. In this process, the project managers' role was clearly defined. Managers have no responsibility for education of students but must take a responsibility for a project success. However, customers' satisfaction of the products developed by projects' was not good, because many common customer-developer communication errors were occurred.
In the third trial, collaboration between a customer and a developing team has been encouraged, that led to a tiny successful system for the customer. The last serious problem is that large difference of the effects of education for students that depends on managers' quality. It is not fixed yet.
|Trial No||Actions||Result and Found Facts|
|1||(F ir st Trial)||Found Facts : + Common management problems occurred in projects + Positive feedback from evaluators for the basically PBL system. Problems: + M anager claims about their role is not defined. Whether ment o r for students or manager for a project + 'customer" was not clearly defined for students. Because customer and manager are the same person , and evaluation policy was not clearly defined .|
|2||+ The e valuation policy of the project was clarified: p rojects are evaluated whether a product is accepted by real customers (not the same person as manager) or not. + M anager meeting time was extended from 45 min to 90 min +Managers responsibility for education was reduced. ( Almost students were second t rial of this course )||Fixed Problems: + Communication between a manager and students has been improved. + P roject scope has shared between facilitators and managers by extending the manager meeting . + Managers confusing and claiming whether manager or mentor was reduced. New Problems: + C ustomers' satisfaction of the products developed by projects' was a not good, because many common customer-developer communication errors were occurred.|
|3||+ collaboration between a customer and a developing team has been encouraged + +Facilitator told customers the responsibilities of them. + +Facilitator give some a dvice s to customers ( Almost students were third trial of this course )||Fixed Problems: +A tiny successful system for the customer was produced. New Problems: +Large difference of the effects of the education for students that depends on managers' quality.|
5.2. Analysis for the Effects by Improvements of the environment
To analyze the effects of improvements discussed above, we have diagrammed the situations and results of projects that carried out the proposed PBL. The diagram model is shown in Figure 6 where vertical axis indicates a customer satisfaction of the project, and another axis do a project goal difficulty.
The evaluation for vertical axis is almost the same as the customers' satisfaction that has been discussed before and shown in Table 2. The evaluations for horizontal axis have been done by the authors with taking the arguments from evaluators and customers. The evaluations are relative for among projects, and we has been appreciated the specification downed project if that is with a contract of customers.
We can discuss about the relevant of our action research and its results by using Figure 8. The first trial, the proposed model did not deliver the evaluation policies to students and managers. As a result, project members set their project goal and scope for themselves. That led to projects result to low customer satisfaction in spite of non-challenged goal had been set.
The second trial, project scope has shared between facilitators, managers and students. Project members could not project scope for themselves any more, which means to start negotiation between customers and developers in the proposed PBL. Although they could not meet the customers' satisfactions, they challenging project goal had been created.
The third trial, entire range of customers' satisfaction has been improved because collaboration between a customer and a developing team has been encouraged by facilitators. We believe the challenging project goal and its successful experience bring up the good engineers, managers and customers.
5.3. Design Principles Led by Our Experience
Through the process of this research, knowledge of the proposed approach has been accumulated and summarized into the following design principles.
(1) The customer and the evaluation policy should be clearly shown to learners.
It is better that a customer is neither the same person as a manager nor a person from organization which a manager belongs to. It is better an evaluation policy to be described as, "Real users prefer to use", not just "Real users may use".
(2) Project scope should be shared between facilitators and managers though a PMO.
Additionally, it needs plenty time to do it at least ninety minutes per week for five projects.
(3) Managers should not have any responsibility for education.
Responsibility for education makes managers be confused to make decision for their project.
Managers should focus the management of the customers' satisfaction. (It does not mean that managers should not do mentoring for students.)
(4) Customers should take a responsibility as well as a real project.
Customers-developers communication errors in this course were occurred because that customer's requirement was not clarified as well as a real project situation. Students also have remarked, "We should discuss the specification deeply with customers and analyze user's requirements together."
(5) Facilitation for customers is needed.
In common case, customers is a beginner to order an information system. Facilitators should give advices about development procedures and customer's responsibility on it.
(6) It is important for students to participate in various projects repeatedly.
The one of the reason why many projects met a customer's satisfaction is that the students had been improved by repeated study. Effects of the repeated study are not only to improve their competencies to develop information systems, but to generalize some concepts on it.
Principle (1), (2) and (3) is led by Trial 2, and (4), (5) is led by Trial 3 as discussed in the early phase of this section. These are not only design principles for the proposed PBL environment, but also definition of the responsibilities for managers, customers, students and facilitators.
6. Construction of the Conceptual Model of "Collaborative Management" Approach
We have constructed the conceptual model of "Collaborative Management" approach based on our research described until this section. The model is shown in Fig 9.
The model represents the collaborative learning system by students, managers, facilitators and customers. Their goal is to be satisfied with people in social system with receiving feedback by them. Ellipse represents the entities in the system. Rectangle indicates the products produced by activities in the project.
A project is composed of students and managers. They produce the information system products with customers and facilitators. Through this process, educational products are produced by the project. Information systems are evaluated by end users, and learning results are evaluated by evaluators.
The university-industry collaboration in engineering education through PBL is common all over the world. The importance of stakeholders and real world settings for engineering education has been remarked by Robert (Robert, 2005), and Inoue (Inoue, 2007). However, the model regulated roles and responsibilities of the stakeholders around the PBL in a university as a learning system for all stakeholders. In our model, a manager is not mentor but a learner any more. A customer is not a stakeholder outside a project but a creator to produce a product and a member of the learning system. Developed and educational products are distinguished in this model. It means not only to clarify the responsibilities of a manager and facilitators but also to clarify responsibilities of evaluators and end-users.
In this paper, we have described the building procedure of the conceptual model, with some examples of what is happened and how students and managers learn in a microcosm implemented by the model. We believe that the competencies and knowledge found by learners in the project would be needed for further adaptive expertise for information systems.
We have developed a learning environment in which a microcosm is created for the development of information systems. Several developing teams composed of an industry manager and several undergraduate students carry out small but real projects for customers. Communication skills are required for identifying essential requirements and the final products are evaluated by the satisfaction of the end users. Although projects are very small in scale, the problems occurring in the project are similar to those of actual large-scale projects. Project members are confronted with and overcame these problems. We believe that for all participating members, the experience of developing tiny information systems has been educational.
Batatia H. 2001A model for an innovative project-based learning management system for engineering education. CALIE’2001 - Computer Aided Learning in Engineering Education.
Demarco T. Lister T. 1987
Graff E. Kolmos A. 2007Management of Change-Implementation of Problem-Based and Project-Based Learning in Engineering. Sense Publishers
Jacobson I. Booch G. Rumbaugh J. 1999 The Unified Software Development Process. Addison-Wesley Professional.
Kaminuma Y. 1995Development of Human-oriented Information Systems- Learning with mentally handicapped people. Symbiosis of Human and Artifact, 935 940.
Kolmos A. 2007PBL at Aalborg University. Working Paper at the International PBL Conference
Lave J. Wenger E. 1991Situated learning:legitimate peripheral participation. Cambridge University Press.
Leitch S. Warren M. 2007Teaching Future Australian Information Systems Professionals, IFIP WG3.4 Proceeding(Education, Training and Life long Learning), 63 70
Mansell G. 1991Action Research in information systems development. Journal of Information Systems, 29 40
Inc Project Management Institute. 2004PMBOK Guide- Third Edition. Project Management Institute, Inc..
Inoue A. Kaneda S. 2007A PBL Approach using Real World Application Development between University and Local Government, Transactions of Information Processing Society of Japan, 49 2 930 943(In Japanese).
Robert H. T. Spencer P. M. 2005Elements of a succesful capstone course considering the needs of stakeholders. European Journal of Engineering Education, 30 2 203 214.
Schummer T. Lukosch S. Haake J. M. 2005Teaching Distributed Software Development with the Project Method. Proceedings of the International Conference on Computer Support for Collaborative Learning (CSCL), 577 586
Stoyan R. 2007More Successful IT Projects by Low-Cost High-Interactivity Project Leardership Education, IFIP WG3.4 Proceeding(Education, Training and Life long Learning), 81 92
Weinberg G. M. 1972Psychology of Computer Programming. Van Nost. Rainhold.