Communication quality and added value: a measurement instrument for municipalities

Purpose – This study aims at a better understanding of communication quality and how it can be measured in the municipal context. A previously developed instrument for measuring communication in municipalities was tested and evaluated. Design/methodology/approach - The instrument draws on the balanced scorecard of Kaplan and Norton and quality control procedures as utilised by the European Foundation of Quality Measurement. For municipalities, communication quality can be defined as the degree to which communication contributes towards the effectiveness of municipal policy and how it strengthens the relationship between citizens and municipal organisations. Three communication functions are given, and for each function seven quality criteria, for example responsiveness, are defined. The latter serve as an umbrella for several indicators that are assessed on a Likert scale. The results for four municipalities in the Netherlands are presented and compared, and the instrument is evaluated. Findings – The Corporate communication scores were relatively high while the Policy communication scores were low. Of the quality criteria, Accessibility and Publicity scored high and Responsiveness low. The instrument has mainly been developed on the basis of auditor and self-assessment, as municipalities have, as yet, few facts and figures with which to support the assessment. The measurement instrument needs to be integrated in the organisation’s planning cycle, as reflection on the results can help in implementing improvements in quality. The measurement process stimulates dialogue on communication quality and the priorities to be set for communication policy. Research limitations – The study is based on four cases. The instrument needs to be tested across a range of governmental-level organisations. Practical


Introduction
The purpose of this study was to better understand communication quality and to find a way of measuring this at the municipal level. Measurement can clarify the added value that communication has for governmental organisations, support accountability and help set priorities for the organisation's communication policy. In the Netherlands an instrument has been developed to measure the communication quality of municipalities. This was tested and evaluated in four cases. The instrument is draws both on the idea of performance indicators as developed by Kaplan and Norton (1997, 2004a, 2006) and on the auditor and self-assessment procedures as developed by the European Foundation of Quality Measurement (Instituut Nederlandse Kwaliteit, 1999;Ten Have, 2003).
The instrument does not focus solely on the activities of communication departments, but on all municipal communication. The quality of three communication functions was studied: namely corporate communication, policy communication and organisation-related communication, each of which is measured using 12 indicators. In order to obtain a clear view of the improvement options available, the results are itemised according to seven quality criteria such as the transparency of governmental information and policy, the responsiveness of the organisation, interactive policy and the effectiveness and efficiency of communication. The end results can be compared with, for instance, targets set previously or the results for a previous year. Here, the results of applying the instrument are presented and discussed.

Quality control
Many organisations have adopted a method of quality control for the organisation as a whole. The balanced scorecard of Kaplan and Norton is a measurement and improvement system designed to translate strategies into concrete actions (Buytendijk and Brinkhuis-Slaghuis, 2000). It provides managers with an overview of performance indicators, and thus may resemble a dashboard. Originally, quality control had a technical orientation, but later it became a management perspective in which quality is defined for all organisational policy areas (Boomsma & Van Borrendam, 1990). Brown (2007) emphasises the importance of gathering information not only about finance and productivity but also about employees and customers, for instance to predict customer loyalty. Parmentier (2007) later added environment and community.
Because communication departments often have difficulties in applying general quality control methods, Hering et al. (2004) propose the use of a customised quality assessment based on the balanced scorecard. While intangibles are acknowledged to be difficult to measure, doing so is nevertheless seen as feasible (Hubbard, 2007). Customised scorecards have been produced for the 'Human Resource' area of expertise (e.g. Becker, 2001), marketing (e.g. Peelen et al., 2000) and IT (e.g. Keynes, 2005), and also for specialist areas such as disaster management (Moe et al., 2007) and city management (Weig, 2004). For the communication area no research data on the results of applying complete instruments have to date been published, hence this paper.
Fleisher and Burton formulated the following criteria for measuring performance in the communication area (1995):  the management must support the assessment;  the assessment must be an integral part of the communication function;  the assessment process must be a team activity and those responsible for implementing improvements must be part of this team;  the assessment must be well-prepared and well-organised;  before commencing with the assessment, those concerned should have clear insight into the organisation's communication activities.
Measuring performance should not be seen as extra work but rather as part of normal business operations (Fleischer and Burton, 1995). Assessment must be more than a snapshot of a given moment in time. It should be seen as opportunities for continuous assessment and improvement. In this sense, it can be seen as a strategic feedback system (Fleischer and Burton, 1995). Performance measurement leads to transparency, which in turn stimulates innovation (De Bruijn, 2001).
As far as the actual assessment structure is concerned, Fleisher and Mahaffy (1997) stress that this must relate to organisational policy. The indicators that assess performance must focus on core processes so that opportunities for improvement can be identified (Fleischer and Mahaffy, 1997). These indicators should clarify the added value of communication for the organisation (Hering et al., 2004). Rolke and Koss (2005) suggest structuring the performance indicators according to key objectives of communication that may serve as an umbrella for several communication activities.
A balanced scorecard for the communication area can be implemented in phases. The following is proposed by Vos and Schoemaker (2004). The first phase is an orientation phase with an initial meeting of the team. This is followed by a second phase, of preparing and implementing the quality instrument. The third phase includes discussion of the results and the planning of improvement projects. This is followed by a fourth action phase in which improvements are implemented. Some time later, after the fifth phase of evaluation, a new cycle starts.
By tailoring it to the organisational strategy the instrument can become better tuned to the characteristics and requirements of the target organisation with each cycle 2004b). The process of assessment and improvement is cyclical (Assegaff, 2002), in that as the instrument is refined, improvement teams can be put in place to implement new working procedures (Rampersad, 2000). The measurement instrument tested here was developed following the above criteria. It could not simply be taken from the literature and applied to governmental communication, as an appropriate instrument for this sector had not yet been published.
A Swedish study suggests the use of indicators such as market share, share price, ranking and awareness among stakeholder groups (Sverges Informations förening, 1996). Ritter (2003) mentions indicators used by large companies, such as the number of invoices sent out. Zerfass (2008) proposes a framework for a corporate communication scorecard, including elements C. Organisation-related communication -This supports the internal processes of a municipality and focuses upon its continuity. Middel (2002) mentions the following elements:  internal communication;  labour market communication;  crisis communication.
The above three functions of governmental communication served as the basis for the structure of the instrument used in this study, as they were well known in practice at the time. Each formed a section for which performance indicators were defined and measured.

Research questions
An advisory board was set up for the study and consisted of representatives of various governmental organisations that, as a voluntary group of experts, also supervised research into the trends of governmental communication in the Netherlands (Vos, 2006;Vos and Westerhoudt, 2008). The members of the board were interested in finding out if the results of the instrument would clarify the state of the art of communication in municipalities. Would the results be comparable? They also suggested an evaluation of the instrument itself. Is communication quality actually measurable and is its measurement not too time-consuming? Does measurement really help to improve quality? They addressed the difficult area of discussing the added value of communication for the organisation, that is, the fit between the expectations of general managers and what communication professionals offer. Would the method stimulate dialogue on communication quality with top managers? This brings us to the following research questions.
RQ 1: What is the quality of communication in the four municipalities? RQ 2: Is the measurement instrument useful in improving communication quality and stimulating dialogue about communication priorities with top managers?

Methodology
The research was implemented in close co-operation with practitioners in governmental organisations. After the instrument was developed, communication departments were encouraged to use it and to report their experiences. When municipalities asked for assistance, the measurements were undertaken by the researchers. In some cases the measurement was adapted according to needs of the municipality which unfortunately rendered the results incomparable. In the present study, four cases are discussed in which the instrument was used in a similar way enabling the results to be compared. The procedure used is reported below.

The respondents in the four municipalities
The measurement was conducted in four municipalities in the Netherlands, one small, one big and two middle-sized; the population of the four municipalities ranged from 25 000 to 280 000 inhabitants. In each municipality applying the instrument was prefaced by gathering relevant available documents and by briefly discussing it in a meeting. All the respondents received a brochure explaining the purpose of the instrument and the questionnaire itself. They answered the scale questions in the survey individually. The researchers reported their findings. In each municipality this was followed by a seminar in which the interviewees reflected on the survey results, taking other available facts and figures into consideration, discussing their implications and also evaluating the instrument. The heads of the four communication departments were asked to list the key personnel involved in communication; all of those so listed participated in the study. The 35 respondents comprised 33 communication professionals, a human resource manager and a controller. The results for these municipalities are presented below, the purpose being to discuss the potential of the instrument and to indicate directions for further research. The instrument was originally developed in connection with an earlier research project (see Vos, 2003 and2006).

Constructing and testing the instrument
The indicators used to measure communication quality were derived from interviews with the participating communication professionals and general managers. The interviewees were given the three communication functions, described earlier, and asked to suggest indicators that show quality in these areas and add value to the organisation. Also, they were asked to cite examples, from their own experience, of good and bad communication and explain why they considered these good or bad. In this way, suggestions found in the literature suggesting that the indicators should show opportunities for improvement (Fleischer and Mahaffy, 1997) and clarify the added value of communication for the organisation were confirmed (Hering et al., 2004).
The final indicators were phrased as statements that could be measured on a 5-point scale, after which they were clustered to form quality criteria, such as responsiveness. This conforms to the suggestion, mentioned earlier in this paper, by Rolke and Koss (2005) that the indicators be grouped according to the key objectives of communication. The labels for the quality criteria were derived from the interviews, in which the interviewees often referred to topics of current interest, many of which were also mentioned in professional journals. Later, the interviewees were asked to give feedback on the performance indicators and to specify which elements they found more or less important; thus, the respective weights of the indicators and communication functions were defined, and the instrument was made ready for a pilot study. The pilot was implemented in one municipality and the instrument refined and adapted on the basis of the outcome (see Vos, 2003). All the municipalities in the Netherlands were encouraged to use the instrument and to report on their experiences with it.
Several measurements were conducted. In two cases the measurement was undertaken by auditors, who used in-depth interviews and searched for facts and figures to back up their findings. In these cases the instrument was adapted to the needs of the municipality and the process was more time-consuming. The results were very useful to the organisation but not comparable enough for this study. Some attempts were also made to add the perspectives of stakeholder groups, e.g. by a survey with a similar content but with statements adapted for civilians, in this case representatives of residential areas. This survey was also made available to the municipalities. Furthermore, the list of indicators was used on numerous occasions in departmental strategy meetings in which participants discussed and marked the indicators that needed more attention in the following year. The auditor mode of measurement showed a better validity than measurement solely on the basis of the documents, survey and team meeting. Where additional measurements were conducted to add the stakeholder perspective, this also resulted in a broader-based measurement. The 'lightest' version of working with the listed indicators was used in the departmental strategy meetings, where it gave focus to the discussion, but did not function as a measurement. In the latter cases interest in a later measurement was sometimes aroused.

The measurement instrument
The prototype developed is a type of audit, an instrument that can be used to investigate and assess situations (Hargie and Tourish, 2000;Downs and Adrian, 2004). This instrument can be used to obtain an overview of communication policies and assess the strategic contribution of communication to organisational objectives, including checking on whether research is used in communication projects. Thus the instrument not only encourages communication research but is itself also a research method. The structure of the instrument was based on the three functions of communication for municipalities described above. For each function the indicators were itemised according to seven quality criteria (based on interviews as described in the previous section), as follows. The seven quality criteria re-appear in each of the three communication functions. When interpreting the final results, the organisation's communication functions are examined to see which of these have a relatively high or low score The same is also done for the quality criteria. This clarifies what priorities are to be set in order to improve the quality of the communication delivered. An overview of the composition of the quality criteria by communication function is given in

Results
The findings for the four municipalities are presented in table 2. As the table shows, few of the indicators score above 60. The respondents were not satisfied with their municipality's communication performance because, as stated in the meetings, they themselves were ambitious and saw room for improvement. The overall scores for this self assessment by the municipalities vary from 47 to 62, with an average of 55. Of the communication functions A shows the highest average score and B the lowest. Of the seven quality criteria Accessibility and Publicity scored relatively high, whereas Responsiveness scored low.  In the meetings with the respondents the results were presented in the form of a snake diagram (semantic differential) for each communication function. For example, figure 1 shows the results of Corporate Communication for the four municipalities. In Corporate Communication Transparency scored lowest, indicating that municipal priorities are not clear to citizens and that the municipalities could be more result-focused. Some respondents suggested that to Vos, M. (2009), Communication quality and added value: a measurement instrument for municipalities, Journal of Communication Management, Volume 13 -Issue 4, pp.362-377.
____________________________________________________________________________ 11 achieve these goals political consensus and consistency over time were needed. Accessibility and Publicity scored high. In the past considerable energy was invested in information centres and media relations. This was not the case for Responsiveness, which includes the more recent requirements of demand-focused working within the municipality and the monitoring of public perceptions. Interactive policy, communication policy and the effectiveness and efficiency of communication, referring to the organisation as a whole, scored relatively high in this area.  Of all the quality criteria Accessibility and Publicity score relatively high, followed by Interactive policy. Transparency is next (especially low for Corporate communication), followed by low scores for Communication policy (connected to policy projects) and Effectiveness and efficiency of communication (especially researching effectiveness). The scores for Responsiveness are lowest. Figure 2 shows an aggregate of all the results for the four municipalities. During the meetings some respondents suggested transferring manpower and attention from strong areas to weak areas, in an effort to preserve good results where expertise was high, as in Publicity, while investing in new priorities like Responsiveness of the organisation. Often, respondents referred to situations typically encountered in governmental organisations and to persons managerially and politically responsible, such as when discussing result-focused working methods and the transparency of municipal policies.

Discussion
Here, we discuss the findings by returning to the research questions. Vos, M. (2009), Communication quality and added value: a measurement instrument for municipalities, Journal of Communication Management, Volume 13 -Issue 4, pp.362-377.

RQ 1: What is the quality of communication in the four municipalities?
The measurement of communication quality clearly indicated both strong and to weak areas in each municipality. As for generalising across all four municipalities, only a few tentative observations can be made. Accessibility and Publicity were relatively high probably because these are more established quality criteria than, for instance, Responsiveness. Transparency could be improved by clarifying municipal policy priorities and making the results measurable (Rotterdam, for instance, has implemented this). Corporate communication received higher scores than Policy communication and Organisation-related communication. In Policy communication it seems that integrating communication in policy-making is not an easy task. It requires in-depth involvement by communication experts in the various policy areas and commitment to communication by other policymakers.
As well as similarities, the results showed many differences between the municipalities. For the most part, it is the specific pattern of the results that shows the municipality how to improve its communication quality. By summing the results for several municipalities many of these patterns are lost. Big differences in scores were found for Organisation-related communication; it seems that the quality of internal communication would appear to vary widely. A comparison was useful in one case, where two municipalities wanted to know how advantage could be taken of the strong points of each after a merger.

RQ 2: Is the measurement instrument useful in improving communication quality and stimulating dialogue about communication priorities with top managers?
In all cases the respondents reported that the measurement process was less time-consuming than they had expected. It is timing that matters most. The instrument is most useful when it fits into the policy cycle, or is initiated, for instance, by a new head of communication. When more time is invested in the process, such as by using auditors to do interviews, the outcome is more reliable. Respondents commented that they would have liked more facts and figures to back up the assessment, but that this could only be done gradually. In a next cycle the validity of the instrument can be strengthened by involving outside auditors and adding more facts and figures based on measurements among public groups.
In each municipality the interpretation of the survey outcomes was facilitated by a seminar. The reflection during these meetings was appreciated as it helped to create a common understanding. Discussion led quite naturally to improvement plans and supported the early stages of reaching a consensus. The effect on improvement projects also depends on other factors, such as attitudes within the organisation and the reason for conducting the measurement. In one case, for example, a simplified measurement led to a big improvement plan.
The respondents agreed that to contribute structurally to quality improvement the measurement should be done (bi)annually and the results combined with monitoring of the internal and external environment. One respondent suggested that in sensitive situations, the quality process could begin simply with dialogue inside the communication department. In other cases the measurement might better be conducted by including the input of top managers, for example by making a comparison with the results obtained by the communication staff. In all the situations investigated the measurement results led to a dialogue with the top managers.
They gave their opinion of the quality of the municipal communication, and discussed priorities for communication policies with staff members. Managers reported that the process gave them a better insight into what communication can contribute. Communication experts reported that it helped to discuss their expert role on a more strategic level.

Conclusions
This study on the first test of an instrument designed to measure the quality of communication in municipalities has some limitations that might be addressed in future research. The study was implemented in four municipalities and, although these were one small, one big and two middle-sized, ranging from 25 000 to 280 000 inhabitants, the results cannot simply be generalised to all such organisations. On a more general level only a few observations could be made. Alongside similarities the results also showed a different pattern for each individual municipality, and it was the individual pattern that made the results useful in identifying areas for improvement.
The applicability of the research method for other municipalities is promising. The method worked well in cases where the motivation to learn from the findings was high. The respondents were positive about the research method and were prepared to use the findings to further improve communication quality. Any structural improvement in communication quality will, however, only appear if the quality cycle is implemented more often. This is a direction for further research.
What are the implications for practice? The results indicate that performance measurement can enhance communication quality by supporting dialogue about the priorities to be set. With increasing organisational awareness of the power of communication, communication experts are receiving many requests for support from within organisations. However, such internal requests for support or advice on communication might not relate to the most pressing issues, for example a possible decline in public support for current municipal priorities. Furthermore, problems urgently requiring communication advice are not always addressed to the organisation's communication department. Accordingly, communication experts need to proactively monitor the situation. The instrument tested here clearly helps in maintaining an overview and setting priorities. It gives managers a broad insight into how communication can contribute to the organisation, as the various objectives of communication are mentioned and made concrete, such as enhancing responsiveness to citizen views and supporting internal change processes. In this way the instrument supports dialogue between communication experts and the top managers of governmental organisations.
Municipalities do not, as yet, according to the interviewees and the advisory board of the study, possess many continuously measured facts and figures about their communication performance. An assessment instrument seems one of the few ways in which at this stage communication performance can be measured, while backing it up with at least the facts and figures available at the time. Adding independent auditor assessment to self-assessment and dialogue between communication experts and top managers may increase reliability.
A separate goal should be gradually to increase the number of facts and figures to support the assessment by regularly measuring the reputation of the organisation among citizens and organisations in the area, measuring the satisfaction of stakeholders with communication including employee satisfaction with internal communication. Future research might usefully focus on further customising such instruments to meet the needs of individual governmental organisations. Furthermore, experiences with this instrument can be compared with performance measurement in the private sector and in other countries. These would be interesting directions for further research.