A BVA Telephone Facilities Project: Lessons Learned

Natasja Logemann, Municipality of Eemsmond, Groningen, The Netherlands

Isaac Kashiwagi, Delft University of Technology, Delft, The Netherlands

The Municipality of Eemsmond has been searching for a way to improve their method of delivering Telephone facilities in order to improve program requirements. To achieve this goal, the Best Value Approach was identified as the optimal system. The Best Value Approach was successfully implemented in the contracting and execution phases of a Telephone Facilities project. An expert was found with a quality solution that delivered the project two months earlier than required and below the expected budget. The advantage of the Best Value Approach was that it identified the client satisfaction and outcome as the most important aspect of the project instead of focusing on the specific technical requirements and method used to accomplish it. This paper will review the results of the Telephone Facilities project and the lessons learned in regard to the Best Value philosophy and methodology.

Keywords: Best Value Approach, Information Technology, Netherlands, City government.

Introduction

Eemsmond (English: Ems Mouth) is a municipality located in the north-east of the province of Groningen, The Netherlands (Figure 1). A municipality is usually an urban administrative division having corporate status and powers of self-government or jurisdiction. Eemsmond, in geographical size, is one of the largest municipalities in the Netherlands. The municipality of Eemsmond has roughly 16,000 residents of which 185 are employed at the municipality. The municipality’s mission and vision include:

Maintaining order while moving with the style, trends and character of the community.
Responding to new developments in society through interactive policy development that align with the local community.

Best Value at Eemsmond

In 2015 the ICT department of Eemsmond was preparing a Request for Proposal (RFP) to procure new Telephone Facilities. Due to their experience in the past with traditional ICT and telephone contracts, which seemed to produce non-performing vendors, the municipality saw the need to find a different method to identify and select an expert vendor.

In 2015 the purchasing department of Eemsmond organized a presentation to introduce the ICT department’s management and project managers to the Best Value Approach. The BVA provided an alternative way to procure and manage services by emphasizing the need to increase the utilization of expertise and minimize the buyer’s use of management, direction and control (Kashiwagi, 2016; Kashiwagi, 2013; Rivera et al, 2016). After the presentation, a project manager of the ICT department decided to run a pilot project utilizing the Best Value Approach.

Telephone Facilities

In January 2016, the project manager proposed to use the Best Value Approach to run the Telephone Facilities project. The Telephone Facilities project consisted of the delivery of a fully integrated “in the cloud” communications platform, including delivery of hardware and software needed for communications (computers, wires and computer programs). The Vendor would be responsible for hosting the (new) facilities and the maintenance, which includes updates and upgrades during the contract.

The proposal to use the Best Value Approach was approved by the management of Eemsmond. One of the key benefits that interested the municipality was the claim that through the Best Value Approach an expert vendor could be identified without requiring the client to know exactly what they want (Kashiwagi, 2015; Sullivan, 2007; Kashiwagi, 2013). Due to the many different options and technical requirements of the project, it was one of the determining factors in deciding to apply the Best Value Approach to the Telephone Facilities project. The project team and the project manager were both pleasantly surprised by the Best Value Approach (BVA) which is based on the concepts of listening, understanding, aligning, accepting reality, and utilizing logic and common sense (Kashiwagi, 2016b). This approach was perceived as contrary to the traditional approach they were used to which focused on managing, directing, and controlling (MDC) (Lepatner, 2007; Kashiwagi, 2014; Kashiwagi et al, 2013; Kashiwagi et al, 2009). The philosophy of the Best Value Approach motivated the entire team to start their first BVA project: Telephone Facilities.

This paper will review the process of implementing the Best Value Approach on the Telephone Facilities project. The following topics are covered:

Necessary preparations for Best Value.
Evaluation and selection of the BV vendor.
Execution and project results.
Client and vendor evaluation and lessons learned from the BVA/PIPS Process.
Reflections and recommendations.

Necessary Preparations

In November 2015, the project group and the selection committee for the Telephone Facilities project was formed. The project group consisted of the project manager, an ICT network specialist and a Best Value purchaser/advisor. Except for the Best Value advisor (author of the paper), no one had experience with the Best Value Approach. The selection committee consisted of five people with different backgrounds and disciplines, however, none of the participants in the selection committee were technical specialists. The Best Value advisor conducted the interviews and facilitated the meetings where the selection committee had to reach a consensus.

In order to become a member of the project group or the selection committee, all members were selected and trained in the necessary paradigm shift required, from “MDC” to “listen, understand and align” (Kashiwagi, 2016; Kashiwagi, 2013; Rivera et al, 2016). The project manager and the ICT specialists were trained in Best Value by two A+ certified individuals in BV, Susan van Hes and Steven Bookelmann. The other members of the team were trained in the Best Value Approach throughout the project.

In the BVA, the client is not intended to instruct the vendors how to deliver the project but rather to identify what outcome they expect (Kashiwagi, 2016; Kashiwagi, 2013; Rivera et al, 2016). Vendors are then expected to utilize their expertise to create an optimal scope to fulfill the client’s expectation. This change in paradigm requires the client to refocus their efforts in simplifying and communicating to the vendors what they want as an end product through a requirement (Kashiwagi, 2015; Sullivan, 2007; Kashiwagi, 2013). The municipality required several comprehensive sessions to create a good vision and project requirement. The formulation of accurate project objectives is essential to the preparation phase of a Best Value project (Kashiwagi, 2016).

Implementation

A lot of effort was put into formulating the project objectives. The first questions were: What is the purpose of the project? Which project results do we want? After several meetings and discussions, the main objective was decided to be: Mobile coverage (accessibility) in the entire area (North-West Groningen) for optimum functionality and safety. After the goal of the project was clear; the objectives and sub-objectives were formulated so that they challenged the market to show their expertise.

Ability to provide consistent coverage, availability and functioning of the telecom.
Functionality and stability. Achieve an accessibility of at least 98% during regular opening hours; this accessibility will be realized in the first year of the contract.
Achieve a user satisfaction of 80% within the first year of the contract.
Reach simple management activities while achieving manageable costs, hence the new situation costs should be predictable and transparent.
Step by step innovation. Continuation of the current functionality and stability is an important goal. However, the municipality of Eemsmond wants a hosted solution on a step by step functionality renewal (Unified Communications).

Budget Estimation and Planning

The budget was estimated on what was paid in the current situation, but the client expected the costs should to be less in the new situation. After the publication of the Tender, the potential Vendors, including the project team and the selection committee, were educated once on the philosophy and methodology of the Best Value Approach [Performance Information Procurement System (PIPS) and Information Measurement Theory (IMT)]. The main purpose of this meeting was to educate the vendors thoroughly on the Best Value Approach and the emphasis to the vendors of the client’s project goals. Seven different vendors attended this educational meeting. The response to the information was diverse, some of the vendors were very positive while some were very negative. It was clear that the vendors would have to adjust to the new paradigm and learn about the approach. The complete process schedule provides more details into the project’s activities as seen in table 1.

Table 1: Tender Process Schedule
Stage of the tender process	Date
Publication Request for Proposal	April 27, 2016
Information meeting, Best Value	May 10, 2016
Deadline asking questions for the information notice	May 24, 2016
Sending information notice	May 31, 2016
Registration and opening of the Proposals	June 7, 2016
Interviews	June 21/22, 2016
Start Pre-Award phase	July 5, 2016
Final Award	September 15, 2016
Start contract	January 9, 2016

Evaluation and Selection of the Best Value Vendor

The submittals that were requested for the project included: project capability (including schedule), risk assessment and value added. Interviews were conducted with two key employees of the vendors’ team. To find the Best Value vendor, the rating system 0, 5, and 10 was used. For this project, the price and quality was divided into a ratio of 25%/75% respectively (Kashiwagi, 2016).

The maximum quality ratios per criterion are as follows:

Project Capability (including schedule) 20%
Risk Assessment 15%
Value Added 10%
Interviews (2 * 15%) 30%

Phase 1: Proposals Rating

On June 7, four Proposals were received for the project. The Best Value advisor reviewed the Proposals to ensure regulations of validity, anonymity and budget were followed. Then the Proposals, excluding price, were forwarded anonymously to the members of the selection committee. These members rated the Proposals individually based on “dominant information”. By following the Best Value philosophy “dominant information” brings consensus, decreases decision making and minimize risks (Kashiwagi, 2015; Sullivan, 2007; Kashiwagi, 2013). Knowledge expressed by non-technical performance metrics is a good example of dominant information. Prior to rating the Proposals, a separate training was held on May 24, 2016. The training reviewed the Best Value methodology, the guidelines to be followed when rating the vendors’ submittals and practice cases.

After the review meeting, under supervision of the Best Value advisor, the selection committee reached a consensus rating for each of the vendors’ submittals with the following conclusions (see Table 2):

Submittals that scored “0” were unanimously scored “0” in the individual ratings from each selection committee member.
Similarly, submittals that scored “10” were unanimously scored “10” in the individual ratings from each selection committee member.
The best and the worst submittals were dominant.
There was a discussion about the two vendors (A and B) who received ratings that were neutral, “5.” The multidisciplinary composition of the selection committee worked well, especially for grading the project capability submittal. Through the conversation of the multidisciplinary team in which different points of view were exposed, the consensus scores were established (Table 2).

Table 2: Submittal Scores
Submittals	%	A	B	C	D
Project capability	20%	5	5	5	0
Risk Assessment	15%	5	5	5	0
Value Added	10%	5	5	5	0
Interview 1	15%	5	10	5	–
Interview 2	15%	5	10	5	–

Phase 2: Interviews

The key personnel at the interviews were the project manager and technical superintendent. The client determined these two individuals play a crucial role in the success of the project. The vendors were required to have scored a minimum average of “5” on their submittals to be invited to the interviews. This meant that Vendor “D” was not allowed to proceed to the interviews. All vendors were asked the same set of general questions. In addition to these general questions specific questions about each vendor’s individual proposal was also used.

The interviews were held at a quiet location in the Municipality of Eemsmond. The Best Value advisor conducted the interviews, which were all recorded. At the end of the interview, the selection committee was given the opportunity to ask one last question in order to reach a correct rating. All three vendors were interviewed in one day. Most individuals felt nervous at the start of the interview, but felt comfortable within five minutes.

The interviews were rated individually by the members of the selection committee after which the process in establishing a consensus score per interview was performed. Similar to the submittals, dominant information was used as the basis for each scoring and the supporting reasoning for all the scores was written down.

Vendor Ranking

After the consensus scores of the submittals and the interviews were finalized, the Best Value advisor revealed the pricing information. The prioritization was filled in and checked, and vendor B (RSE) was identified as the prioritized Best Value vendor. RSE had received the highest scores in the submittals and the interviews and had the lowest price (Figure 2). An example of the ambition and commitment of RSE is that they finished the project two months earlier than requested.

Pre-Award

The Municipality of Eemsmond organized the Pre-Award kick-off meeting in which the project teams of RSE and the Municipality Eemsmond were invited. The kick-off started with a Best Value PIPS presentation by the client’s Best Value advisor and indicated what the Pre-Award documents should contain:

Performance metrics (key performance indicators).
Detailed financial breakout of the project.
Detailed project planning.
List of all identified risks by the other vendors and solutions conceived by the vendor.
Response to any possible technical concerns or risks.
List of all uncontrollable risks/activities (not controlled by the vendor).
List of all assumptions.
List of roles and responsibilities.
List of accepted/rejected value added items.

After this, RSE took control of the meeting and presented a schedule (Figure 3) for the Pre-Award phase. Dates were made for when RSE would deliver certain submittals and time was scheduled for the project team of Eemsmond to get a thorough understanding of these submittals. There were two meetings scheduled with RSE and the project team of Eemsmond. The author of this article put a lot of effort into educating the project team of RSE and Eemsmond about the mindset of Best Value.

During the Pre-Award, it was a challenge to determine which of the value-added options (four) admitted by RSE, was worth accepting. In the end, the value-added options did not show enough dominant information to financially justify the client’s approval. The Pre-Award period finished according to the schedule within six weeks after the kickoff meeting.

Execution and Project Results

After the signing of the contract, RSE started work immediately, fully utilizing the BV process by sending in the weekly risk report (WRR). The WRR is a weekly report which measures the project deviations from the initial vendor’s plan outlined in the Pre-Award phase in terms of time, cost or quality and the source of the deviation (Kashiwagi, 2016). The vendor maintains and distributes the weekly to the client for review and approval. The project manager of Eemsmond would then be responsible to assess each deviation mentioned in the WRR and provide feedback through a rating of 1 (not satisfied), 5 (neutral) or 10 (satisfied). In total, four deviations were reported, all of which have received a client rating of 10.

The implementation phase has since finished (on time and on budget) with a customer satisfaction of 9 out of 10. The project is currently in the service/maintenance phase of the contract of which performance metrics are not available at the moment.

Client and Vendor Evaluation of the BV Process

The project group evaluated the Telephone Facilities project internally after the project was awarded to RSE. The overall picture of the tender process was positive and can be summarized as follows:

The tender was very successful in terms of time and cost.
The expert was found.
The high quality was received for a competitive price (lowest bidder).
Compared to the traditional system, the vendor is more committed to the project.
The degree of chemistry between people, both on the client side as well as between client and vendor in the Pre-Award period, can be described as HIGH.

Client Lessons Learned

Provide all relevant information. In case of doubt, if the information is relevant, provide it to vendors anyway.
If, during the tender process, new information is provided, mention the reason why it is added.
Give vendors time and space to learn.
Provide guidance for the submittals with information in your request of proposal.
Provide concrete information about the dates of the interviews.
Arrange an interview setting that is as relaxed and quiet as possible.
Mention, in the education session of Best Value PIPS, that it is not possible “to train someone on the content of the interview”, but that it does make sense to prepare someone for the interview.
Do not shortlist when moving into the interviews. Allow all vendors to go to the interviews.
Adjust the interview questions to the different key personnel.

Vendor Lessons Learned

Know your client: is this a “real” Best Value project? Who is in the client project team?
Prepare the key personnel for the interview, for example: extensive discussion about the project and submittals, thinking in advance about possible interview questions and training the key personnel on interview techniques. It is important to prepare the interviewee.
Involve key personnel throughout the tender process so they are fully informed, and do not send an interviewee who does not understand the project.
Involve colleagues with Best Value experience who are not directly involved in the tender to review and help with the tender.

Best Value Observations

Philosophy

The Best Value philosophy teaches to see from the beginning to the end (Kashiwagi, 2016b), by doing this, potential risks are mitigated (Kashiwagi, 2016). Looking back on the process of the Telephone Facilities project not all risks were identified in advance.

One of the important lessons learned from this project is that better training is needed in the philosophy of Best Value for the personnel of Eemsmond who are outside of the immediate project organization. The shift from ‘management, direction, and control’ to ‘listen, utilize and align’ is especially important and difficult to understand for some. Education helps stakeholders to listen to and understand the project manager in cases of unforeseen risk. Thus, stakeholders will be focused on how they can help the project manager, rather than managing, directing and controlling the project manager. This will ensure that unforeseen risks are mitigated as soon as possible. Therefore, proper use of the weekly risk report requires it to be sent to the correct people (stakeholders and managers) and those people need to be trained in the Best Value philosophy.

Project Team Composition

It requires the necessary attention to compose the right project team and the right selection committee. This is a critical step for the client because not everyone in the organization will understand and participate in the paradigm shift of “manage, direct and control” to “listen, utilize and align,” (Kashiwagi, 2016). It is important to have a mixed selection committee, this means having a team that contains experts with different backgrounds. A multidisciplinary and balanced selection committee with a minority of technical specialists and preferably people who can think in terms of processes, contributes to an objective assessment.

Pre-Award

The Pre-Award period of the Telephone Facilities project was successfully completed in six weeks. It is important in the Pre-Award stage to manage the expectations between vendor and buyer. The key is that the vendor pre-plans this period. Staying in the right role as a client or a vendor is an important point of attention. The different roles for the client and the vendor are very clearly defined in the Best Value process: the vendor is in the lead, it is their plan, and the client supports the vendor to make their plan to be as complete and efficient as possible (Kashiwagi, 2016). Education on the concepts of Best Value is necessary in this period. The technically educated people on the client side have to be trained to ask questions instead of trying to re-make some components of the vendors plan.

Pre-planning of the Pre-Award period helped the project members; it was clear from the beginning what the purpose of this period was and what input was needed from the client’s representatives. It worked very well to organize an internal meeting before the meeting with the vendor. In this meeting MDC was turned into asking questions and naming concerns.

Conclusion

To achieve the municipality’s goals, the Best Value Approach was identified as the optimal system. The BVA approach achieved the following results:

An expert was found with a high-quality solution.
Highest quality and lowest bid amongst competitors and below the existing costs of the municipality by 27%.
Implemented solution on time and on budget with a customer satisfaction of 9 out of 10.
An estimated time savings of 50% by the team (6 employees) over the traditional tenders.

The conclusion of this review details how the results of the Telephone Facilities project provides several valuable lessons in regards to the Best Value philosophy and methodology. The author of this paper strongly recommended that the BVA was a major improvement to the way the municipality selects their Best Value supplier.

References

[1]	Kashiwagi, D. (2014). The Best Value Standard. Performance Based Studies Research Group, Tempe, AZ. Publisher: KSM Inc., 201.
[2]	Kashiwagi, D. (2016). 2016 Best Value Approach, Tempe, AZ: KSM Inc., 2016.
[3]	Kashiwagi, D. (2016b) Information Measurement Theory, ISBN # 978-0-9850496-8-3.
[4]	Kashiwagi, D., and Kashiwagi, J. (2013) “Dutch Best Value Effort.” RICS COBRA Conference 2013, New Delhi, India, pp. 349-356 (September 10-12, 2013).
[5]	Kashiwagi, J. (2013). Dissertation. “Factors of Success in Performance Information Procurement System / Performance Information Risk Management System.” Delft University, Netherlands.
[6]	Kashiwagi, J., Kashiwagi, D. T., and Sullivan, K. (2009) “Graduate Education Research Model of the Future.” 2nd Construction Industry Research Achievement International Conference, Kuala Lumpur, Malaysia, CD-Day 2, Session E-4 (November 3-5, 2009).
[7]	Lepatner, B.B. (2007), Broken Buildings, Busted Budgets. The University of Chicago Press, Chicago.
[8]	Rivera, A., Le, N., Kashiwagi, J., Kashiwagi, D. (2016). Identifying the Global Performance of the Construction Industry. Journal for the Advancement of Performance Information and Value, 8(2), 7-19.
[9]	Sullivan, K., Kashiwagi, J., Sullivan, M., Kashiwagi, D. (2007). Leadership Logic Replaces Technical Knowledge in Best Value Structure/Process. Associated Schools of Construction.