MISREPRESENTATION OR REFUTATION?
A QUALITATIVE META-ANALYSIS ON SIX METHODOLOGICAL ISSUES 
IN ECONOMIC IMPACT STUDIES OF ARTS FESTIVALS

A Thesis
Presented to the Faculty of Erasmus School of History, Culture and Communication
In Requirement for the Degree of 
Master of Cultural Economics and Cultural Entrepreneurship 

By
Charalampos Mallidis 
August 2011

The question of the thesis title is inspired by a phrase quoted by Christopher Madden (2001) as written by John Stewart Mill (1826): 
                                                    “Misrepresentation is always beautifully brief; 
                                                    refutation always tediously long.”

Date:                                30 August 2011
Student:                           Charalampos Mallidis
Student Nr.:                     350954
E-mail:                             cmallidis@yahoo.com
Thesis Supervisor:          Dr. Erwin Dekker
Second Supervisor:         Dr. Hans Abbing
TABLE OF CONTENTS
List of tables                                                                                                                vi
Abstract                                                                                                                      vii                                     
Acknowledgements                                                                                                   viii 
Preface                                                                                                                         ix
                    Chapter 1
                            Introduction                                                                                       1
                            I. Economic Impact studies: definition                                             2
                            II. Economic Impact studies: main characteristics                           2
                    Chapter 2
                            Research Methodology                                                                     5
                            I. Qualitative Meta-analysis                                                             5                                   
                                     i. Definition and Methodology                                                5
                                    ii. Strengths and Weaknesses                                                   7
                            II. Methodological Process of Meta-analysis                                  9
                                    i. Selection of method                                                              9
                                    ii. Aims and Objectives                                                          10
                                   iii. Limitations of research                                                       11
                                   iv. Methodology Stages                                                           12
                            III. Six elements of Meta-analysis                                                  18
                    Chapter 3
                            Element 1: Collecting Data Method                                               23
                            I. Research Findings                                                                       25                                                                                                                   
                                   i. The former Edinburgh study                                                25
                                   ii. The National Arts Festival study                                        25
                                   iii. The recent Edinburgh study                                              27
                                   iv. The Ithaca study                                                                28 
                                   v. The Glastonbury study                                                       29
                            II. Interpretation of Research Results                                           31
                   Chapter 4
                            Element 2: Defining the Area of study                                          34
                            I. Research Findings                                                                      34                                                                                                                   
                                   i. The former Edinburgh study                                               34
                                   ii. The National Arts Festival study                                       35
                                   iii. The recent Edinburgh study                                              35
                                   iv. The Ithaca study                                                                35 
                                   v. The Glastonbury study                                                       36
                            II. Interpretation of Research Results                                            37                              
                  Chapter 5                                                                                             
                            Element 3: Determining Visitors’ number                                    39 
                            I. Research Findings                                                                      40                                                                                                                   
                                   i. The former Edinburgh study                                               40
                                   ii. The National Arts Festival study                                       42
                                   iii. The recent Edinburgh study                                              43
                                   iv. The Ithaca study                                                                45 
                                   v. The Glastonbury study                                                       46
                            II. Interpretation of Research Results                                           48                           
                  Chapter 6
                            Element 4: Evaluating Supply Side Constraints                            50
                            I. Research Findings                                                                      52                                                                                                                   
                                   i. The former Edinburgh study                                                52
                                   ii. The National Arts Festival study                                        53
                                   iii. The recent Edinburgh study                                              53
                                   iv. The Ithaca study                                                                55 
                                   v. The Glastonbury study                                                       56
                            II. Interpretation of Research Results                                           60                                        
                  Chapter 7
                            Element 5: Estimating Multipliers’ Effects                                   61
                            I. Research Findings                                                                      63                                                                                                                   
                                   i. The former Edinburgh study                                               63
                                   ii. The National Arts Festival study                                       64
                                   iii. The recent Edinburgh study                                              64
                                   iv. The Ithaca study                                                                65 
                                   v. The Glastonbury study                                                       66
                            II. Interpretation of Research Results                                            69
                   Chapter 8                     
                            Element 6: Estimating Total Impact figures                                 72
                            I. Research Findings                                                                      73                                                                                                                   
                                   i. The former Edinburgh study                                               73
                                   ii. The National Arts Festival study                                       75
                                   iii. The recent Edinburgh study                                              77
                                   iv. The Ithaca study                                                                79 
                                   v. The Glastonbury study                                                       81
                            II. Interpretation of Research Results                                            86
                   Chapter 9                                                   
                            Summary and Conclusions                                                            90
                            I. Summary of Research Results                                                   90
                            II. Misrepresentation or refutation?                                               91
References                                                                                                                 95
Appendix                                                                                              
          List 1: Sources of Data                                                                                 101
          List 2: List of all Economic Impact Studies initially scanned                      102

                  

LIST OF TABLES
Table 1: Research Guide Tools for E.I. Assessment                                               14
Table 2:  Methodology Stages                                                                                 17
Table 3: Meta-analysis criteria                                                                                19
Table 4: Examined E.I. Studies                                                                               20
Table 5: Collecting Data Method                                                                            31
Table 6: Defining the Area if study                                                                        37
Table 7: Determining Visitors’ number                                                                  47
Table 8: Supply Side Constraints                                                                            59
Table 9: Estimating Multipliers’ Effects                                                                 68
Table 10: Estimating Total Impact Figures                                                             85

ABSTRACT
	Economic Impact studies have absorbed considerably important attention of the academic world not only for their validity as a research method but also their general contribution in measuring the value of culture. This paper extends on the existing knowledge of the validity of such studies as far as their employed research tools are concerned. The question primarily raised addresses the relative effectiveness of researchers to present the pursued economic figures, and the overall constraints of the foundations of this largely discredited method. The methodology of qualitative meta-analysis was adapted in order to synthesize and review a selected number of Economic Impact studies of arts festivals upon six main research issues practitioners are challenged with. For each element respectively, the most effective practice was indicated along with considerations for further improvement. Results of the comparative evaluation demonstrated that each study case applies significantly different methodology and research assumptions, both of which crucially affect the final figures. Furthermore, none of the analyzed impact assessment papers was proved to deal accurately with the examined elements. These findings suggested substantial issues of inconsistency of the general methodology as well as broader limitations regarding the actual usage of such studies by diverse stakeholders. From the perspective of academic research, this thesis precisely delineates the procedural steps required to conduct a meta-analysis evaluating methodology diversities. 

KEY-WORDS
Economic Impact studies, arts festivals, meta-analysis, research methods.
ACKNOWLEDGEMENTS
	Firstly, I would like to thank my supervisor, Dr. Dekker, for his patient and generous coaching. You have taught me the importance of “calibration” and made me feel proud for being an economist. I have learnt tons through the process of writing this paper thanks to you.
	This thesis would not have been made without the research tools and support received by Dr. Snowball. Her contribution to the topic of Economic Impact studies has been immensely useful for the academic community. I would also like to mention all Economic Impact practitioners who kindly allowed me to use their studies for the purposes of this paper or guided me towards the right direction for this research.
	Appreciation should also be attributed to the second reader of this thesis, Dr. Abbing, for his time and attention, as well as Dr. Handke for directing me towards the right selection of topic.
	Finally, my colleague and good friend, Zuzanna, has provided me a great deal of support with her poignant remarks and precious advice. I shall not leave out my twin sister, Joanna, for helping me with the format of this paper and listening to me whenever I needed it. I will always be grateful to you both, Zuza and Jo.

PREFACE
	An alternative title for this thesis would be “the hours”, meaning the long duration demanded for this research to be conducted and written down. I have spent a considerable amount of time delving into the Economic Impact method through various studies, which also required a laborious effort to retrieve. Even before the actual research was undertaken, it took even more time along with my supervisor to focus on the methodology of meta-analysis in order to discuss the economic assessment of arts festivals. 
	This has been an introductory encounter for me with meta-studies, secondary studies dealing with reviewing and synthesizing other studies upon their methodology and other research issues. The idea for doing such a thesis, based on relevant impact studies, came as the most appropriate decision following an initial intention to undertake myself a whole Economic Impact study as part of this paper.
	My love to arts festivals dates back my childhood years, where I was anticipating with immense enthusiasm every year the Athens International Film Festival. It was the combination of the artistic content and the vibrant cultural atmosphere of the events rendering it a particular reality within my own world. Later on, I became part of this festival by participating as a volunteer and contributing to its organization, a stage which led me to a position in the press office of the festival for two consecutive years dealing with the participating artists as well as the visitors. There I learnt that festivals are a vital component of the communal life of a city providing a window for the international export and recognition of local cultural amenities.
	By being part of the International Film Festival Rotterdam earlier this year, probably the greatest annual international event of the Netherlands, the idea was generated that I conduct a study on the Economic Impacts of the event in the regional community, as my thesis plan. Very soon I realized that such task would be unfeasible, as it would require a whole group of researchers and, undoubtedly, a commission behind it to finance the entire project. By delving into the method, I comprehended the implications involved and the accurate development design that such research requires.
	Apart from the feasibility constraints, while examining impact assessment attempts, validity as well as accuracy issues of the methodology itself emerged. Each research appeared to be faulted with a range of pitfalls and, also consistency was a general lack of most reports. That is how I thought that repeating a method that is already filled with criticism and potential flaws would be of no actual use to spend my limited time with. At that time, I had already obtained an immense amount of information and knowledge on this specific type of research. Therefore, writing a thesis on the methodological issues of this method, and investigating research challenges and implications involved in such assessment would be of real interest primarily for me to explore and, secondly, of actual usefulness in terms of producing a valuable academic paper. Furthermore, practitioners of Economic Impact research could be provided with an explicit report on typical hindrances of the methodology and further suggestions to overcome them.
	 These have been my initial motivations as well as aims for this research. By allowing myself to plunge deeply into the Economic Impact research, the topic emerged by itself and appeared as the right choice for an effort that had already begun its course. With the appropriate direction I received from my helpful supervisor as well as several guidelines indicated by Economic Impact practitioners, I placed the pieces together, in the right spot I hope, and were led to this meta-analysis attempt. 
	During the writing process of this paper, a structural question on the validity of the Economic Impact method accompanied my meta-reviewing disposition. It was, also, the actual outcome of my research results, gradually shaped. Hence, this is reflected on my thesis title: is it the studies individually to be charged with misrepresenting the Economic Impact assessment of arts festivals or is it the method itself weak, limited and responsible for its own discredit? Maybe a definite statement cannot be determined by this study. However, I hope I have raised some outstanding issues of the general feasibility of the Economic Impact approach within my research.
	This thesis has been written step by step focusing greatly on the methodological process applied, since it is an indispensable part of a meta-analysis. Knowing that I am producing a study that can be of actual use to an already established economic development research framework deserving considerable academic attention has been the most satisfying and challenging part of my concerted effort. This has kept me motivated and engrossed during the complete project.
Chapter 1
Introduction
	Whilst the effects of the late ‘00s global economic crisis are evident in multiple aspects of national activities, changes in the structure as well as function are being processed in most bodies of public nature. Cuts on the funding of the arts have been witnessed in most countries of the western world, as a result of a general economic, political and social recession. Especially for European countries, where arts and cultural organizations are primarily financed by governmental subsidies, the not-for-profit sector is facing a foundational threat as far as its existence and longevity are concerned. Unless organic adjustments are made immediately, issues on the survival of many arts companies, either that is a museum or a theater group, will soon arise. 
	In such bleak times for the arts, tools and arguments on behalf of culture are needed in order to support and persuade on the value of these essential goods. Indeed, the arts and culture have a marked effect on the economic profile of countries, notably reflected on the GDP as well as the employment rate, while cultural tourism is evolving into a respectable area of investment worldwide. Apart from their financial rewards, they are also responsible for the generation of non-market social values enhancing the educational, spiritual and emotional level of a nation. For the latter elements it appears nearly impossible to put a monetary value on, due to their particular intangible features.
	Scientific models have been developed in favor of the arts by academic actors seeking to advocate for the undeniable effects of cultural goods. A variety of valuation techniques have been established for measuring different characteristics of the contribution of arts. Economic Impact studies, Contingent Valuation method and Choice experiments are the most notable modeling frameworks applied to capture the value of cultural goods. These instruments are implemented to diverse cultural organizations such as museums, orchestras, community arts centers, festivals, fairs, theaters as well as various creative industries. 
	Each method enables the evaluation of such organizations on a different scale. This paper will analyze the particular case of Economic Impact studies and examine how this economic tool measures the value of culture. A description of the method will be provided along with its substantial features. 

I. Economic Impact studies: definition
	The Economic Impact studies represent a research method to evaluate the short-term contribution of arts and culture on local and regional level. Basically, they “attempt to calculate all the additional economic activity that takes place in the impact area as a result of the event or facility being studied” (Snowball, 2008: 34). They are focused, according to Snowball (2008), primarily on measuring the market value generated in the reference area rather than the non-use benefits. Therefore, they provide, mainly, economic figures with regards to the expenditure made directly due to the production of a cultural good or service. Such studies are conducted not only for the support of arts events such as festivals, arts organizations such as museums and generally non-profit arts industries, but also non-arts related events such as sports games, conferences or trade shows.

II. Economic Impact studies: main characteristics
	Bille and Schulze (2006) argue that the Economic Impact assessment approach is aimed at analyzing, within a specific geographic area and over an accurate period of time, the effects on income, consumption and employment induced by a particular cultural event or institution or, even, the whole cultural scene within a city. More precisely, they evaluate the economic impact of arts by measuring explicitly with economic tools the direct, indirect and induced effects of the arts on consumption. 
The total impact is suggested as followed (Bille and Schulze, 2006: 1058):
                     Total Impact = Direct Effects +Induced Effects + Indirect Effects,  
                    while Indirect Effects = (Direct Effects + Induced Effects) × Multiplier
According to the Bille and Schulze configuration (2006), the direct effects comprise the income and employment induced by the cultural event itself, while the induced effects estimate visitor spending on ancillary activities stemming from the specific event, such as accommodation, food and beverages, transportation and shopping. The Indirect Effects constitute the so-called multiplier effects, which are the secondary rounds of spending. All initial injections of new money into an area’s economy stimulate further economic activity and, hence, generate “additional business turnover, income and employment” (Hendon and Costa, 1988). The multiplier is merely a figure in which “each of the successive rounds of economic activity is summed” (Hendon and Costa, 1988: 6).
	The most typical approach of Economic Impact research involves the estimation of visitor spending data so as to assess the economic effects of a cultural asset within the local community of the study area (Langen and Garcia, 2009). However, no consistent model is adapted by such studies (Madden, 2001). Most research attempts have developed an extensive variety of methodologies to present the necessary economic figures, as will be further examined in the main body of this analysis.
The advantage of the Economic Impact approach is that it provides useful figures indicating that specific cultural events and the related tourism produce essential economic flows for the regional and national economies (Herrero et all, 2006). Nevertheless, these figures should be measured meticulously and analyzed with ample consideration since there can be various errors in miscounting the direct, induced or multiplier effects (Bille and Schulze, 2006), as will be analyzed. 
Apart from hindrances in calculating precise numerical data and the probability of errors, the Economic Impact scope lacks in evaluating the total impact of cultural activities on regional economies, since they only focus on a narrow perspective of short-term economic growth and ignore the positive externalities induced by the social and educational role of the arts. As Throsby (2001) suggests, we should take into account the public good character of the arts and cultural sector, which cannot be properly operated through the economic impact analysis. Additional to the educational spillovers of arts, the non-market values cultivated within the residents of a specific community should be also highly considered. In other words, the sense of pride, cultural identity, option value and the intangible cultural value of the arts should be included as significant poles of positive long-term externalities.	The reasons driving the commissioning of Economic Impact research are numerous and diverse, since the assessment reports are of great interest to a number of different stakeholders reading them: academic actors, arts managers, festival organizers, corporate sponsors, policy makers, urban planners, government officials as well as other decision makers with regards to the allocation of public funds. Comparing similar event projects, evaluating the progress of a cultural good in terms of visitor amount and financial rewards, and identifying the market value of an arts event are among the most significant motivations for commissioning such studies.
 Moreover, it can be said that Economic Impact reports are increasingly appointed, nowadays, as a tool for cultural advocacy in order to indicate that a cultural asset is responsible for the generation of broader pecuniary gains within the regional economy – an area that has caused a respective part of the academic world’s discredit to the method due to its mistakenly expanded usage. Deliberately abusive application of the method will not be further discussed here as it exceeds the scope of this thesis. 
The main body of analysis of this paper will follow starting from the methodology of this study to the research results and their interpretation.

	

Chapter 2
Research Methodology
	Defining the type of research as well as describing the steps that have led to the outcome of this process is possibly the most essential component of a thesis writing attempt. As proclaimed by the title, this is a qualitative meta-analysis of the Economic Impact Studies on arts festivals based on six criteria-challenges of the methodology, that researchers encounter and are called to resolve. Firstly, I am going to discuss this particular method, which is relatively new in the scientific field and I will explain why I chose it. Then, I will delineate the stages of this specific effort, although not in ample details due to the space and time constraints of my research.

I. Qualitative Meta-analysis
i. Definition and Methodology
	“Meta-analysis” has evolved into an independent branch of research the last thirty years having its roots in social sciences. While Russell (2002) and Bryman (2008) affiliate this term with quantitative research defining it as a review of all quantitative studies conducted on a particular area aiming at confirming the interdependence of two variables, they both identify qualitative meta-analysis in the term “meta-ethnography”. The latter is a distinct type of qualitative research analyzed in depth by Noblit and Hare (1988) as applying the metaphors and concepts of one study to interpret other studies. Hence, a distinction should be made here between qualitative and quantitative meta-analysis. 
	The meta-analysis method, also referred to as “meta-study”, “meta-assessment”, “meta-synthesis” and “cross-case research or evaluation”, has been a well established strategy for quantitative research (Stall-Meadows and Hyle, 2010), whereas in qualitative studies it is still on an insufficient development stage (McCormick, Rodney and Varcoe, 2003). The main difference between a quantitative and a qualitative meta-study is that the former is adopted by researchers focused on demonstrating the extent of a cause and effect relationship for a specific phenomenon, while the latter method is purely “hermeneutic” aiming to interpret patterns and phenomena (Walsh and Downe, 2004). Both, however, allow for “new insights and understandings to emerge” (Walsh and Downe. 2004: 205). 
	Since this research attempt is a qualitative meta-analysis, more space and thought will be given to this specific design. Before providing an accurate definition for this term, a special note ought to be made on the prefix “meta” that constitutes the most vital component of this method (Zhao, 1991); it literally means “after, about and beyond” thus entailing the nature of this study. Zhao (1991), stemming from a sociological background, characterizes it as a study subsequent to a number of similar studies, involving these studies and taking them to a further depth. Certainly, a qualitative meta-study is not a sheer literature review summarizing the data and results of previous research papers over the same topic, but a creative synthesis of them aiming to generate some fruitful conclusions.
	The qualitative research literature refers to this conceptual framework with several different terms –meta-ethnography, systematic review, aggregation-, all of which include the selection of similar empirical studies, that are thoroughly examined and accordingly amalgamated (Reis, Hermoni, Van-Raalte, Dahan and Borkan, 2007). The concept of qualitative meta-analytical method has been initially featured, according to Walsh and Downe (2004), in Stern and Harris (1985) research on women’s health care, a scientific area that has largely applied and developed this particular research design. Additionally to clinical studies in various sectors of health research –psychotherapy, nursing, medical statistics-, it is also used in consumer analysis involving the collection of a vast number of case studies. 
	For the needs of this specific thesis, the definition provided by Neuman (2011) in his “Social Research Methods” comprehensive approach markedly depicts the methodology that is going to be attempted by this paper. Neuman (2011) driven by the fundamental that contemporary studies should be built on the foundations of previous studies thus serving the notion of knowledge accumulation, explains meta-study as a particular technique leading to “methodological review”. He defines the latter as a special type of review where “the author compares and evaluates the relative methodological strength of various studies and shows how different methodologies account for different results”. He also identifies five steps in the process of this synthesis, as the following (Neuman, 2011: 125): 
1.	Locate all potential studies on a specific topic or research question
2.	Develop consistent criteria and screen studies for relevance and/or quality
3.	Identify and record relevant information for each study
4.	Synthesize and analyze the information into broad findings
5.	Draw summary conclusions based on the findings
Apart from aggregating research results, researchers also examine how relevant studies utilize essential concepts and the definitions they provide. 
	Equally important ingredient of this methodology remains the analytical description of all stages of the procedure the researchers undertake to reach their conclusions (Stall-Meadows and Hyle, 2010), as delineated above by Neuman. “Transparency” is commonly recognized as a necessary element since readers should be able to comprehend the research strategy and how it has been gradually formed with proper argumentation for justifying each decision (Walsh and Downe, 2004). It has additionally been stated that the ideal methodological process should engage in an exhaustive account of detecting, selecting and carefully examining all relevant studies on a specific field rather than a representative sample, including as well the criteria of exclusion and inclusion of them (Barroso, Gollop, Sandelowski, Meynell, Pearce and Collins, 2003).

ii. Strengths and Weaknesses
	As concluding from the definitions provided in the previous part of this chapter, the aim of meta-studies is interpretive, seeking to analyze data, results and research methods in a remarkable depth. Synthesizing relevant qualitative studies enables the exposure of “nuances, taken-for-granted assumptions and textured milieu of varying accounts” (Walsh and Downe, 2004). In this way, it provides space for the generation of a novel thought process, which can lead to significant insights and further fruitful conclusions. This creative synthesis does not settle with a brief summary of all collected information; on the contrary, it greatly succeeds in outlining the gaps and lacks of research attempts, as Russell (2000) explicitly points out: what is missing so far and what has been erroneously performed. An accurate meta- assessment can allow a sound judgment of how legitimate a method appears to be in measuring a particular phenomenon (Richman, 1983).
	Gilbert (2001) and Zhao (1991) highlight as the strongest point of meta-analysis its power to produce conclusions of higher value than the individual studies it employs. Certainly, there lies the most important reason motivating researchers to initiate a study about other relevant studies. By evaluating the potential strengths and weaknesses of a method itself based on previous study-cases through a process of examination, comparison and, eventually, evaluation, it offers a real opportunity to reflect upon the scientific evolution of the specific method: which stage we are now and where we are heading. Therefore, one could argue that a method drawing on the peripheral methodologies, applied so far by researchers, can lead to further progress of science. 
	Last but not least, Stall-Meadows and Hyle (2010: 413) appraise “objectivity and generalizability…while preserving the subtle nuances of real life context” and “thereby increasing the degrees of freedom and offering greater explanatory power” as the most beneficial tools for meta-researchers in bridging individual case studies into a whole consistent analysis.
	
	However, despite its positive contribution to certain areas, the method has been often described as “poorly developed”, “time consuming” and “complex” (McCormick et al, 2003) while it has created an increasing amount of controversy by dealing with “theoretical and practical barriers” (Reis et al, 2007). These hindrances cover various aspects, from technicalities such as feasibility of conducting such research to issues of ethics and interpretive power. Sandelowski, Docherty and Emden (1997: 365), on the other hand, indicate as the greatest challenge to be untangled “developing usable and communicable systematic approaches to conducting meta-synthesis projects that maintain the integrity of the individual studies”. Synthesizing single case studies, which very often use similar but not the same methodology and are conducted on very different contexts, runs the risk of leading to spurious and irrelevant conclusions. Zhao (1991) has included the view a number of sociologists share that meta-studies seem to be of no real worth in most areas by producing results inapplicable on a broader extent.
	Indeed, this method as relatively new has not had the time to develop a scientifically established technique and this can also be affirmed by the limited amount of the researches that have been published to date. The most significant limitation, hindering potential researchers to adopt it, is the difficulty in accessing the raw data on which the primary studies are based, such as questionnaires, sampling methods, survey results, as well as information on the original circumstances under which the primary studies have been conducted (McCormick et al, 2003). Meta-aggregating attempts require a broad and thorough knowledge of the field studied and the above mentioned obstacles may further obstruct researchers with the various assumptions they would have to make, thus crucially affecting the final conclusions.
	
	Despite all barriers this method has to overcome, it still remains vital to the needs of scientific research. All realms of research necessitate “integrated analyses of bodies of research literature that portray the state of the art in their specialty and a sense of the trends, problems and continuing issues in the field to envision directions for their work” (McCormick et al, 2003: 943).

II. Methodological Process of Meta-analysis
 i. Selection of method
	This research is a study on the methodology of Economic Impact studies of arts and cultural festivals. It will attempt to examine the methodological issues as dealt by five individual Economic Impact studies that differ regarding the time and location they have been produced, and compare them in a qualitative research strategy. Hence, it is a secondary study analyzing the methodology of relevant primary studies aiming mainly at a constructive synthesis of those.
	This thesis has selected the methodology of qualitative meta-analysis on a modified version due to the several feasibility constraints that my research had to deal with, thus limiting its extent and potential explanatory power, and to the particular nature of Economic Impact studies. As explained in the previous chapter, there is no single consistent methodology Economic Impact practitioners have used so far, and the methodology applied by most researchers is mainly quantitative but containing a number of qualitative remarks. Hence, these two factors will crucially determine the quality of this research as well as its outcome. Although some qualitative researchers tend to think that combined methods are not advised to be subject of a review, especially when there is no common methodology applied, other researchers tend to view this meta-synthesis of assorted approaches as a fresh and fertile injection to the diversity of analysis (McCormick et al, 2003).
	As it appears, there is no other method more appropriate to deal with the content of this study than that of a meta-analysis on a qualitative ground. There has not even been a matter of selection since the topic necessitated on its own this type of research design immediately. Additionally, the significance of selecting this particular method lies in its power to draw conclusions on the potential evolution or stagnation of the primary studies it analyzes. Meta-synthesizing Economic Impact studies allows for an assessment of their particular methodology, which is the ultimate goal of this thesis, as well as an indication of the repeated errors. It should be mentioned that the present paper does not intend to create a consistent model for the Economic Impact studies methodology but to serve as an aid to all diverse stakeholders interested in this topic.
	Five relevant studies on the same topic of the Economic Impacts of arts and cultural festivals are the subject of analysis here through the scope of six different criteria – methodological issues. More specifically, the procedural steps of the methodology are delineated in the following section, including the location, selection and final review of the studies along with the reasons that have led to specific decisions.

ii. Aims and Objectives
	This research basically aims to compare the different methodologies researchers have employed, so far, to perform the same type of primary study that demonstrate the financial gains of arts festivals. It is going to illustrate the best practices as well as most common pitfalls that Economic Impact practitioners are challenged with. The method of Economic Impact studies will be overly evaluated as an integrated body, in terms of its strengths and weaknesses. This paper will, also, indicate the significant underlying assumptions made by researchers in the data-analysis process and how these have affected the outcome of each study. Last but not least, it will investigate how the various methodologies of Economic Impact studies have been developed lately and suggest how they should be potentially improved.
	The main objectives of this specific attempt are related to the key stakeholders, to whom Economic Impact studies are of primary interest. The most essential objective is the provision of tools in filtering data and figures contained in these studies so as to ensure their proper interpretation and identification of potential biases or mistakes. It shall facilitate, instrumentally, Economic Impact researchers in avoiding pitfalls and overcoming obstacles in order to deliver more accurate and useful reports. Furthermore, this research pursues to contribute as a guide to policy makers, academic actors as well as other stakeholders interested in employing this type of studies as a tool for cultural advocacy.

iii. Limitations of research
	Before proceeding with depicting the procedural stages of the methodology, it is considered of high significance to the value of this research to note the crucial limitations that should be taken into account. Time and space feasibility factors are the initial constraints imposed by the nature of an academic thesis. The research had to be undertaken within a specific period of time and the report, as well, had to be restricted to a specific extent considering again time pressure. Therefore, the information contained in this thesis should be thought of as part of the overall research findings generated in this limited amount of time. 
	Then, access to the original primary studies has not been easy as a large number of these are either not published or not available through the Internet. Also, for a few of the studies obtained through an attentive individual effort, the details of which will be outlined in the following section, commissioners did not permit their usage for the purposes of this analysis. Additionally, several of the papers I encountered were lacking sufficient parts, such as the raw data themselves –questionnaires, survey findings, essential sampling data- but also substantial components of the original studies. This suggests that for some studies I had only access to a mere report of the original study, thus entailing an increasing amount of insufficient data for my analysis; by data meaning the number of primary economic impact studies. It also demonstrates the degree of difficulty in assessing the accuracy of these studies since I was not able to crosscheck the data and figures implemented by the researchers.
	It has to be mentioned that eventually after a concerted effort to retrieve the raw data of the primary studies used in my analysis, I was able to obtain the necessary figures only for two of them. Apart from the lack of the initial raw data, important details with regards to the context under which the original studies were performed are missing. As stated already, the absence or insufficient provision of these signify the greatest limitation of the qualitative meta-analysis method, thus “encouraging” researchers towards a number of creative assumptions in order to conduct their study. 
	All these constraints have urged me to decisions in the research with likely marked effects on the validity of the final conclusions. Therefore, I will try as thoroughly as possible to indicate these “dark” choices in order to assure the quality and transparency of the study. In addition, these constraints have also affected the description of the stages as will follow subsequently.

iv. Methodology Stages
	This thesis had initially been assigned to explore commonalities and differences in the methods Economic Impact practitioners apply in their research. However, during the research design it was made clear that it would suggest a whole scientific process of a systematic meta-evaluation of relevant Economic Impact studies, employing similar procedural steps.
	Hence, after the description of the stages undertaken during the research procedure there will be a table outlining them in a clear form so as to facilitate readers and potential future researchers who would wish to repeat such task - Table 2.
	The steps involved in this specific attempt are nearly identical with those outlined by Neuman (2011); however there are few additions and adjustments made for the needs of this particular meta-analysis, as will be seen further on.
	Before locating the Economic Impact studies on the arts and cultural sector, it was considered necessary to become acquainted with the method itself: how it began, how it is defined and how it has been viewed throughout the last three decades since its development by academic actors, policy makers and media stakeholders. This step was intended to provide a more complete perspective on the theoretical and practical implementation of this empirical type of studies. Therefore, information was scanned through multiple sources such as academic books, journal articles, government reports on policies, arts management documents, websites of cultural organizations, papers presented in relevant conferences on the economic sustainability of the arts sector as well as several articles and critiques on established newspapers. These sources will not be analytically presented, as this would extend the purpose this thesis, and they have been used predominantly for the enlightenment of the researcher. It has to be noted, though, that while becoming acquainted with the Economic Impact study method, each new bit of information provided by the above mentioned sources was leading this research to a different direction, new conceptions and insights, and hence demanded a strong sense of disciplined orientation.
	The next step was the review of a number of methodology guides providing the basic tools for conducting Economic Impact studies. These are targeted to assist potential researchers as well as arts and cultural practitioners in dealing with Economic Impact analysis. Table 1, as included below, briefly presents the most robust instruments I have investigated that proved of beneficial value to analyzing the primary studies of my research. They stem from different countries and researchers so as to ensure a diversity of views. An analytical report on these guide tools will not presented within this paper though.

Table 1: Research Guide Tools for E.I. Assessment

 	TITLE	YEAR	COUNTRY	PREPARED BY	AIM	
1	Measuring the Impact of Festivals: Guidelines for conducting an economic impact study	1997	Australia	National Centre for Culture ad Recreation Statistics, Australian Bureau of Statistics	- To indicate how to generate useful data and figures for an economic impact study                                                          - To provide a beneficial marketing aid to festival managers                                       - To contribute to the creation of a model of Economic Impact research	
2	A Guide for Undertaking Economic Impct Studies: The Springfest Example	2001	USA	Department of Recreation, Park and Tourism Science at Texas A&M University / Crompton, J. L., Lee, S. and Shuster, T. J.	- To offer a general model for conducting Economic Impact study research                                                        - To contribute with four integral principles to the consistency of the Economic Impact study methodology	
3	Estimating the Economic Impacts of festivals and Events: a Research Guide	2002	Australia	Cooperative Research Centre for Sustainable Tourism / Janeczko, B., Mules, T. and Ritchie, B.	- To provide a assistant tool to festival managers, tourism agencies and enterprises                                       - To contribute a basic comprehension of the methodological procedure of Economic Impact studies	
4	Measuring the Value of Culture	2008	South Africa	Department of Economics, Rhodes University / Snowball, J. D.	- To provide practical tools to researcher in conducting Economic Impact studies                                         - To contribute as a guide to policy makers for reviewing the results of such studies                                                  - To highlight the various methodologies as well as pitfalls applied by Economic Impact practitioners 	

	The following most crucial stage of locating Economic Impact studies of arts and cultural organizations proved to be largely time consuming and fostered a number of barriers to the research project due to the induced constraints, as already mentioned. Research reports of Economic Impact studies were detected in various formats: scholarly books, journal articles, master theses and dissertations, government files, papers presented in conferences and cultural organizations’ archive documents. The sources used, initially, were online as well as paper archive databases, mainly in libraries of the Netherlands (UB Erasmus University Rotterdam, Boekman foundation, UBA University of Amsterdam, European Festivals Association database and Impact Database maintained by the Centre of Cultural Policy Research of the University of Glasgow). Search engines on the Internet were greatly helpful, such as Google, Google Scholar and Yahoo. The searching process involved the following precise key words: “Economic Impacts”, “arts and cultural sector” and “festivals”. A detailed list of all sources used will be contained in Appendix – List 1.
	The main disadvantage encountered was that the studies retrieved by these sources were either not precisely relevant or lacking important information. Most of them were brief reports while the original primary studies with analytical methodological details could not be traced. Hence, a more direct method was required to obtain these studies. Academic stakeholders through universities as well as consultants through development agencies conducting Economic Impact research were contacted via emails to contribute by providing either available studies or sections that were missing from those already found, or information in order to locate them. Even though many of them appeared reluctant to help and some others did not permit the usage of the studies for the needs of this review, the outcome was fruitful. A large number of Economic Impact studies were retrieved in this way, all of which were scanned in order to make an initial selection of these, which were to be studied carefully in the following step, with regards to the quality of their content as well as their relevance. Almost thirty primary studies were chosen to be thoroughly reviewed as part of the research project and they are listed together in Appendix – List 2. 
	The next stage was to determine appropriate inclusion and exclusion criteria for the selection of the studies that would be further discussed in the main analysis of the paper. Thirty Economic Impact studies, primarily selected, were carefully inspected in order to check the amount and quality of details they included concerning the specific elements examined in the meta-synthesis. Studies accounting for the economic impacts of the whole cultural sector of a region were excluded. Also, studies on events or arts organizations other than festivals were excluded as it was decided that only the impacts of arts and cultural festivals would be examined in this paper. By meticulously selecting this specific type of Economic Impact study, which is on arts festivals, the research was led to a more consistent review and synthesis. It gave this paper a more specialized research methodology to meta-analyze, thus serving the validity and translation power of this attempt.
	The most essential inclusion criterion was the origin of the study. Special attention was drawn so that the analyzed studies stem from a variety of countries and researchers in order to achieve diversity in the analysis “sample”. In addition, a study written in 1977 was particularly selected in order to examine the evolution of the methodology. The criteria of selection aimed to lead to specific conclusions on the Economic Impact methodology: whether and how it has changed over time, and whether researchers tend to repeat similar mistakes.
	The result of this step was the final selection of five Economic Impact studies on arts and cultural festivals intended to be evaluated concerning a number of standards that will be accurately described. The examined studies are listed by chronological order, in table 4 at the end of this chapter, accompanied by their basic characteristics.
	The following stages are going to be presented in the rest of the chapters of this paper and constitute the main body of analysis of the research. The five selected Economic Impact Studies were thoroughly screened and analyzed according to six vital methodological elements described in the following section. The researcher identified these six elements on each of these studies, recorded them and synthesized the findings for the needs of the meta-analysis. These findings were then analyzed and expanded into precise methodological results, separately on each element. The most effective practice for each of the issues according to the studies investigation, as well as considerations for further improvements were considered. 
	The last step of the methodology process comprises the formation of conclusive remarks on the methodology of Economic Impact research. The main analysis of this meta-evaluation has led to a synthesized view about the usage of such type of research as a cultural advocacy tool and the progress of this method: where we are now and whether there has been any progress noted, as had been originally aimed. Based on the interpretation of the research findings, the final conclusions also extended over the implementation of potential improvements on the methodology. Before proceeding with the six elements there will be a summary of the procedural stages in the following table.

Table 2: Methodology Stages
Steps	Methodological Process	
1	Become familiar with Economic Impact studies literature by multiple sources	
2	Review research guides for conducting Economic Impact studies	
3	Locate potential studies in: 
a. Online & paper archive catalogues and Internet Search Engines 
b. Personal enquiries towards researchers and practitioners	
4	Make first selection of relevant studies	
5	Determine inclusion & exclusion criteria to select studies for meta-analysis	
6	Examine the selected seven studies according to five elements	
7	Synthesize the findings of meta-analysis	
8	Interpret the research results and consider potential improvements on each criterion	
9	Present final conclusions on the research methodology and its implementation	

	

III. Six elements of Meta-analysis
	This section will define the six selected issues-challenges of Economic Impact studies on arts and cultural festivals that each researcher is faced with when conducting such study. These will be used as criteria for the main analysis-synthesis of the chosen studies. Before further expanding on them, a special note will be made with regards to the reasons justifying this decision.
	There are indeed various aspects to review the Economic Impact studies upon, since the validity of the data and figures presented by them relies on a number of different methodological aspects (Yuan and Yuan, 1996; Tyrrell and Johnston, 2001; Snowball and Antrobus, 2002), all of which focus on a single issue. As already stated, Economic Impact studies are interested in depicting the injection of new money flowing into the local economy, hence a distinction should be made between gross and net impact. Net Economic Impact refers to the expenditure made solely as a direct result of the event studied, while gross impact represents the total amount of spending in the area of study irrespective of its origin (Seaman, 2003; Snowball, 2008). A large number of researchers, though, tend to confound gross with net impacts, which has caused a great deal of discredit towards the method itself. 
	Tyrrell and Johston (2001) suggest that the Economic Impact framework should account on a number of factors while presenting their results so that there is no confusion between gross and net figures of the true impacts of an event. These include the “source”, “geographic starting point”, “destination” as well as the “reason” for spending (Tyrrel and Johnston, 2001). They are also identified in Snowball’s (2008) distinct issues in conducting Economic Impact studies, as well as Crompton, Lee and Shuster’s (2001) assessment guide. This means that the main methodological research challenges derive from the crucial distinction between gross and net impact: it has to be consistently proved that the specific event has generated an additional flow of spending in the community.
	Driven by the scholarly literature review, the selected guide tools as well as the examination of the Economic Impact reports, it was decided for the needs of this paper that the basic issues drawn by Snowball (2008) will comprise, with some minor alterations, the elements of the meta-analysis. All six criteria, depicted in Table 3, are focused around the implications of distinguishing gross and net economic impact, as will be examined.
Table 3: Meta-analysis criteria
 	6 Elements for Meta-analysis	
1	Collecting data method	
2	Defining the area of study	
3	Determining visitors’ number	
4	Evaluating supply side constraints	
5	Estimating multipliers’ effects	
6	Estimating the total impact figures 	

	It should be said that these six criteria are all tightly woven into each other and are not precisely distinct features of the methodology. Indeed, collecting data methods, for instance, determine the estimation of visitor number, while the latter is crucial for the calculation of final impact figures, which is importantly influenced by the definition of multiplier’s effects, the area of study and supply side constraints. All elements are linked like a rigid chain: if one of these breaks, then all the rest become fragile. Certainly, they all aim at indicating the total net impact figures of a festival event, which is the ultimate stage of the assessment process. The distinction in this research has been made to facilitate the overall analysis. As will be observed, each one results in another so as to eventually be linked all together. 
	For our analysis, we can observe that each researcher deals effectively with a different criterion according to the data which is at the their disposal. However, the lack of access to the raw data of the studies should be also taken into account as far as the evaluation of their methodology is concerned. Additionally, the particular circumstances and technical difficulties of each research should be also considered.
	All above mentioned criteria will serve as a baseline to examine and synthesize the selected Economic Impact studies –listed subsequently in Table 4-, which are the main part of the meta-analysis. Before though proceeding with the meta-review, a brief outline of its structure will be provided.

Table 4: Examined E.I. Studies

 	TITLE 	YEAR	REGION / COUNTRY	PREPARED BY	COMMISIONED BY	TYPE OF EVENTS	
1	The Economic Impact of the Edinburgh Festival 1976	Oct. 1977	Edinburgh, Scotland / UK	R. Vaughan / Department of Economics, Heriot-Watt University 	Lothian Regional Council and Scottish Tourist Board	- 2 Performing Arts Festivals of diverse events from opera to dance and poetry                      - 1 Film Festival                       - 1 Cultural Festival based on British military traditions	
2	The National Arts Festival 2004: Festino Survey and Impact Study	Nov. 2004	Grahamstown / South Africa	G. G. Antrobus and J. D. Snowball / Department of Economics, Rhodes University	Nationa Arts Festival	- 1 Arts Festival including a wide range of performances (e.g. drama, music) and other events (films, crafts)	
3	Edinburgh's Year Round Festivals 2004-2005 Economic Impact Study	Sept. 2005	Edinburgh, Scotland / UK	SQW Economic Development Consultants and TNS Travel and Tourism	City Council of Edinburgh, Scottish Entreprise Edinburgh and Lothian, EventScotland and VisitScotland	- 9 Arts and Media Festivals                         - 4 Cultural Festivals on local traditions                   - 1 Science Festival                 - 1 Sports Festival           - 1 Christmas Festival                       - 1 Easter Festival	
4	An analysis of the Economic Impacts of Ithaca's Light In Winter Festival 2007	Aug. 2007	Ithaca, Tompkins County / USA	J. C. Daniels (Master Thesis) / Department of City and Regional Planning, Cornell University	Not specified	- 1 Arts and Science Festival consisting of performing arts events, gallery exhibitions as well as a series of local cultural and science events	
5	Glastonbury Festivals 2007: Economic Impact Assessment	2007	Glastonburt, Mendip District / UK	Bakers Associates	Mendip District Council and Glastonbury Festivals Ltd	- 1 Arts Festival comprising of a series of music and other arts performances held in the summer and two music events during the rest of the year	
	This research will not make an evaluation of each of the studies and indicate the most accurate one, even though the analysis provided may allow it. It attempts to evaluate the methodological issues of the selected Economic Impact studies and explore how each researcher deals with them. After having analyzed how these elements are presented within each paper, it will demonstrate which is the best practice for each of these criteria and make some useful suggestions for their further improvement.
	The following six chapters, being the primary part of the body of this research analysis, will record the synthesis of the five case-papers for the methodological evaluation of the Economic Impact studies. It should be emphasized, though, that it is not claimed that those selected comprise the most representative sample of the Economic Impact methodology. As stated, they were chosen out of the ones that were made available for this thesis due to a number of constraints considered. 
	They will be reviewed according to six elements previously stated. However, some of them may not provide data on one or more of the criteria due to several reasons. Primarily, the raw data could not be retrieved for most of them and, also, a lack of important information may be due to the limited version of the report accessible to me. Some of the assumptions the researchers had to take and analytical details regarding the calculation of the figures are left out of the final reports, thus entailing a number of implications for their comprehension. These limitations will be seriously taken into account and further assumptions that the meta-review may require will be clearly stated. 
	As far as all of the five studies are concerned, the festivals reported are either fully or partially funded by the government, and some of them specifically mention their non-profit state. We can assume that for the rest of the studied festivals as well, because of their budget outlay. This characteristic will also determine the size of the impacts and their structure.
	The paper will proceed with the analysis of each criterion separately, consisting of six respective chapters and following the order of elements as previously delineated. For each element - chapter, an analysis of the scholarly views will be initially made, followed by the report of results of the five selected studies as well as their interpretation serving as a conclusion particularly. By combining the research findings with their interpretation, a thorough comprehension of the issues involved within Economic Impact practices is pursued. Each element’s review will be accompanied by a summarizing table, serving further as a tool for the reader’s coherent overview.

Chapter 3
Element 1: Collecting Data Method
	This is listed, according to Bragge and Snowball (2006), as the most significant element accounting for the accuracy of the results of an Economic impact report. In this type of research, the required data consists of the visitors and other stakeholders’ spending patterns. Most common method applied by researchers is the survey: either interviews or self-completion questionnaires. While the former results in more reliable and suitable data since the quality of sample can be well controlled, it is considerably more costly and time consuming to perform (Snowball, 2008). Janeczko, Mules, and Ritchie (2002) state an additionally impractical requirement: professionally trained interviewers. Self-completion questionnaires, on the other hand, represent the most typical method used by researchers due to its much lower cost. However, it is generally not advised due to the number of biases it involves (Frechtling, 2006; Crompton, Lee and Shuster, 2001).
	First and foremost, the content of the questionnaires can be often confusing to the respondents and, thus, be liable to several different biases. An example illustrated by Snowball (2008) is the number of expenditure categories:  if too many they can lead to exaggerated spending amounts and be exhausting for the respondents, if too less, though, they may cause an underestimation of the expenditure, thus in both cases generating false estimations of the true economic impact. Another example is reporting on “group spending”; it can be really useful in specific types of expenditure such as accommodation while it can be totally inappropriate in others like food and souvenir purchases (Snowball, 2008). Crompton, Lee and Shuster (2001), on the contrary, markedly defend the inclusion of the “group spending” component in questionnaires. 
	The time the questionnaires should be completed has drawn an important attention by researchers (Heaney and Heaney, 2003; Breen, Bull and Walo, 2001; Zhou, 2000; Frechtling, 1994). When conducted in the middle of the event taking place, asking from respondents to predict and report on expenditure occurring in the following days, it can lead to “prediction bias”, whereas questionnaires completed at the end of the trip, demanding an accurate recall of past spending, may cause a “recall bias” (Cultural Ministers Council, 1997). Therefore, the time questionnaires are filled out can induce further complications for the estimations.
	Biases can also be generated by the choice of sample of respondents, since according to Frechtling (2006) a randomly selected large sample cannot always guarantee representativeness. Loomis (2007) identifies two crucial issues as far as the sampling process is concerned: “avidity bias” can occur when random people who happen to be present in the area of study are included in the sample, and “length of stay bias”, which is argued as very common, when visitors who spend more time in the sampling area are those most likely to be sampled. Additionally, day trippers are less likely to be included in the sample as they tend to be unwilling to participate due to their time constraint (Wilton and Nickerson, 2006).
	Snowball (2008) adds another, less noticeable but equally essential, bias caused by the language of the questionnaires. Indeed, researchers appear to neglect the inclusion of questionnaires in several languages so as to encourage non-native visitors participating in the survey. Apart from visitors’ survey, supplementary data should be collected by other festival stakeholders generating significant expenditure for the community. Frechtling (2006) makes special note to surveys conducted for “producers”, “sponsors”, “media representatives”, “vendors”, and “future visitors”, meaning those reporting they would repeat their visit in the region. He, also, argues on “direct observation” as an additional method to report on spending data. However simple this method may seem, following and reporting people’s expenditure patterns can be impertinent for the participants and has not been a broadly established method yet. 
	To sum up, there can be various biases generated by the data collection method, only a number of which are stated here. Taking them into consideration in the final Economic Impact report is of vital significance for the readers’ accurate comprehension and further transparency of the research (Wilton and Nickerson, 2006).

I. Research Findings
i. The former Edinburgh study 
	The Edinburgh study (1977) applies a number of diverse methods to gather the required data. Attenders’ interviews were conducted with a sample of 660 groups of people which led to 346 group accounts on spending data as well as visitors’ profile and origin. Even though it could be considered a relatively small sample, the report confirms that it is fairly representative. Additional sets of interviews were made at the entry points of numerous events of each festival, specific details of locations and times are not provided though.  Spending data was collected, also, by performing groups of two of the festivals with surveys to 39 and 20 non-local deriving people respectively. Other interviews were conducted with 89 business establishments located within Edinburgh and the Lothian region and, finally, data was retrieved by the Scottish Tourism and Recreation Study of 1973, updated to correspond the year’s visitor number.
	Although the methods applied were effectively diverse to capture and, potentially, cross-check the estimated figures, no information is provided with regards to the special conditions of the interviews, the way they were conducted as well as details of time and specific location of the surveys. Regarding the content of the questionnaire, although there was no access to it, the report included information on the types of spending expenditure, where an implication may be observed: food expenditure may indeed be attributed by the researchers to the category “café/pub”, even though it is not explicitly stated. It can, hence, be concluded that a proportion of respondents may have accounted only on drinks, beverages and snacks for this category, unless precise instructions were given by the interviewers, for which we can only make an assumption. Furthermore, no remark is mentioned on group spending or length of staying related questions.

ii. The National Arts Festival study 
	The National Arts Festival study (2004) applied both most essential methods highlighted by the scholars: direct face-to-face interviews and self-completion questionnaires. According to its researchers, Antrobus and Snowball (2004), each tool was used for different reasons. While the “direct person-to-person interviews are considered more reliable and likely to be more fully completed than questionnaires completed voluntarily”, they also enable an actual interaction thus providing the interviewer the opportunity to ask for additional clarifications when necessary (p.7). On the other hand, “self-completion surveys have the great benefit of being less costly, but suffer the disadvantage of introducing bias in terms of the willingness to complete” (p. 7). Furthermore, longer-staying visitors tend to be more willing to participate in such surveys than shorter-staying and local attendees may also refuse due to “interview fatigue”.
	Antobus and Snowball (2004) report on all potential biases stemming from the data collection methods and are extremely meticulous, as well, with the description of their application in the festival. They provide analytical details on interviewers’ selection, recruitment and monitoring, and they also conducted a pilot session of interviews on the first day of the events, in order to assure the correctness of the data collecting procedure. They mention a daily supervision of the interviewers by the researchers themselves to establish a safeguard against potential biases. 
	Regarding the sample selected for the interviews, instead of having a random sample, the researchers decided on the construction of a sample base formed by their experience from previous similar cases. As far as the sample selection of the other method is concerned, questionnaires were provided on “a comprehensive range of theatre seats and venues…and accommodation establishments as well as at the Village Green craft market” (p. 9), so as to obtain the sampling of diverse tastes. They acknowledge potential biases generated from the absence of day-trippers in the sample, and, additionally, argue that “the differences between the interviewed population and the self-completion population remains one of judgment” (p. 9).
	As specifically stated, the process resulted in a number of “562 face-to-face successfully completed interviews and a further 407 self-completion questionnaires” (p. 8) and their results led to the construction of visitors’ profile extensive report. Even though the actual questionnaire was not contained in the version of the study available, from the analysis of both methods’ results it was shown that information was asked about attenders’ expenditure on a broad range of categories (“accommodation”, “shows”, “food and drinks” and “shopping”), apart from details on their particular profile (“sex”, “home language”, “origin”, “age group”, “occupation” and “earnings”). A distinction was, also, made for group spending and the size of the group was considered for the impact estimations.
	A very significant remark, Antobus and Snowball (2004) make, is that each data collection method may have importantly influenced the expenditure figures, as they observe a difference in the spending patterns provided by each method respectively. They attribute this to sampling differences between the two methods: while the interviewed sample is certainly more representative due to its careful selection, questionnaires were filled mostly by longer-staying visitors as well as those able to read and write in English since the forms were only available in this language.
	In addition, data regarding earnings made by traders and performers was retrieved by a 1996 producers’ survey undertaken by one of the study’s researchers, which can be considered undoubtedly outdated. The researchers themselves, though, acknowledge this constraint in their final estimations.
	Overall, the National Arts Festival study (2004) devotes an analytical account on the methods allowing for a transparent data collection procedure. The researchers thoroughly designed and performed both processes with deserving efforts on various issues from the interviewers’ training to the sampling selection. The most significant aspect here is the fact that they acknowledge potential biases and limitations generated by each method applied, even by the considerably outdated producers’ survey.

iii. The recent Edinburgh study
	The Edinburgh study (2005) has successfully applied a number of diverse sources in order to gather all necessary data for the impact assessment. However, no explicit details are provided with regards to these methods or conditions under which they where performed. A survey of the festival visitors was conducted that led to 4,129 on-site interviews during the festival dates – no particular account on the location, time and events where the survey took place are reported. There is, though, a mere reference to the representative sample interviewed, the well-trained interviewers and the thorough design of the questionnaire suited to the needs of the required for the research information.  
	In addition, web surveys were accomplished by 115 performers and delegates and 25 media representatives – these were conducted during the Summer Festival period. Supplementary secondary data was retrieved by the Edinburgh Principal Hotels Association, the Edinburgh Hotel and Guesthouse Association and the Edinburgh City Center Management Company. Previous studies on Edinburgh Festivals’ impact assessment as well as accommodation sector were also implemented by the present research.  Last but not least, important data was obtained by the festival organizers concerning events’ expenditure and festival revenues generated by ticket sales and its funding sources.

iv. The Ithaca study
	The Ithaca study (2007) has analytically described the survey methodology applied for its data collection. It provides a full detailed account on the conditions under which the 3-day festival survey questionnaire took place. The researcher selected this on-site during the festival period method, so as to avoid implications and induced memory bias from seeking the visitors after the occurrence of the festival. Daniels (2007) refers to overcoming the “recall bias” as defined by Frechtling (2006) and recognizes that collecting data in the middle of the event may have generated an additional prediction bias based on the expectations of respondents on their future spending. Volunteer interviewers were recruited specifically for the needs of that survey and were supervised daily to ensure the effectiveness of the data collection. A relatively large sample was gathered from all nine events of the festival, involving in total 737 respondents. The researcher considers potential constraints imposed by on-site intercept survey methods but assumes that the sample selected was representative despite not entirely random. 
	The report, very extensively, provides information on the precise locations and times of the survey, as well as the actual questionnaire used. Questions referred to various aspects of the visitors: demographics, length of staying, motivation for attending and accommodation data for those residing outside the local area, sources of information promoting the festival and questions on the particular events they attended. 
	A careful scanning of the questionnaire enables, though, a marked observation, which directly affects the impact methodological process. While the majority of the questions result in useful data on the marketing strategies of the festival that can be of great significance to arts managers, they do not really assist the researchers in the Economic Impact assessment. Essential questions are left out such as spending on retail, group spending, the local residents’ primary reason for visiting the festival and their opportunity cost of attending it. An illustrative example is the food expenditure question, which may be misleading to respondents and generate additional prediction bias to the survey, as it is literally based on respondents’ future expectations.
	Apart from the survey, constraints imposed by the produced primary data urged the researcher to retrieve information from other secondary sources, even though these are not extensively discussed in the paper. Visitor spending figures regarding accommodation, food and drinks were obtained by previous recent tourist studies in Tomkins County, and ticket expenditure data were fetched by the festival’s ticket office.
	Overall, Daniels (2007) should be praised for the analytical design of the festival’s survey and for considering a number of diverse parameters in its conducting process from the quality of interviewers to the specific conditions of the survey. However, the content of the questionnaire itself is doubtfully consistent with the study’s pursued aims and this raises questions for the assumed quality of the paper. The content of a questionnaire is primarily evaluated on the effective usage of data produced. This aspect will be further examined at the targeted estimation of the total impact figure.

v. The Glastonbury study
	The Glastonbury study (2007) has applied various methods to collect all required data for the impact assessment of the events; unfortunately, no sufficient details are reported on the specific conditions and characteristics of these methods. The methodology, mainly, consists of surveys to three different types of festival stakeholders: visitors, traders and local enterprises. 
	The visitor survey resulted in 2500 sampled individuals, stated as face-to-face and on-site interviews that took place during the festival period. Even though there was no access to the questionnaire designed, it can be concluded that information was drawn by the respondents regarding their demographic profile, origin, basic reason for visiting the area, length of staying and spending patterns. An essential advantage is that there is a distinction of visitor expenditure made on-site and off-site, which is seldom accounted in Economic Impact surveys. An equally significant disadvantage, though, is that the research has left out spending on tickets on the festivals with no adequate justification on this intention.
	The trader survey was also an on-site and face-to-face method with a sample of 12% of on-site traders, suppliers and promoters covering diverse issues including their “location, staffing, turnover and supply chains”. The third survey involved a web or post self-completion questionnaire of local businesses located in Mendip District - accommodation, food and drink, and retail establishments mostly. The questionnaire concerned the overall impacts of the festival to them, both positive and negative. 
	Apart from the surveys, data was gathered via “case-study interviews” conducted with a number of fundamental stakeholders of the festival including festival organizers, sponsors, tourism centers and other local associations. The most important sample of them is incorporated in the impact report indicating positive as well as negative externalities the festival generated for the local community. In addition, information was collected by the festival organization with regards to its expenditure pattern.
	The Glastonbury report (2007) successfully combines a number of diverse methods to ensure the accuracy of collected data, although no analytical account is provided concerning the validity of the surveys – details on specific time, location, sample selection and interviewers’ recruitment are left out.

Table 5: Collecting Data Method

Studies	Collecting Data Method	
The Economic Impact of the Edinburgh Festival 1976 	A number of surveys to diverse stakeholders:                                                                                                                  festival visitors, participating performing groups, off-site business establishments and data retrieved by the Scottish Tourism and Recreation Study 1973                                                      No details regarding the surveys conducted	
The National Arts Festival 2004: Festino Survey and Impact Study	Individual face-to-face interviews and self-completion questionnaires - both designed and performed very thoroughly, numerous biases considered and ample details provided to ensure the transparency of procedures                                                                                             Data retrieved by a 1996 producers' survey - acknowledged update requirement	
Edinburgh's Year Round Festivals 2004-2005 Economic Impact Study	Various sources applied to gather diverse required data:                                                                                  on-site visitor survey during the festival period, web surveys of performers, delegates and media representatives, previous research on Edinburgh festivals' impact assessment and accommodation sector, contact with festival organizers and other Edinburgh Associations                                                                   No specific details on the surveys conducted                                     	
An analysis of the Economic Impacts of Ithaca's Light In Winter Festival 2007	A survey questionnaire to festival visitors conducted by recruited interviewers on-site during the festival period - fairly representative and considerably large sample, thorough design of the survey, potential biases accounted but the content of the questionnaire significantly weak and inefficient                                                                                                                                                     Additional data obtained by previous tourist studies as well as the festival's ticket office - not extensively presented                                                	
Glastonbury Festivals 2007: Economic Impact Assessment	A number of diverse collecting data methods applied:                                                                                                                                                                                                               on-site face-to-face visitor survey, on-site face-to-face trader survey, web or post self-completion questionnaire to local business establishments, case-study interviews with important festival stakeholders and data retrieved by the festival organization                                                                                      No further details on methods used	

II. Interpretation of Research Results
	All studies reviewed selected a number of methods in order to collect the required data for the impact assessment. Certainly this criterion particularly should be examined in combination with the estimation of the final figure, where its aimed result can be viewed. Another consideration one should keep in mind is the lack of access to the original tools used here, either the questionnaires or the raw festival data that is, which certainly limits the scope of this analysis.
	The National Arts Festival method (2004) should be acknowledged for containing both face-to-face interviews and self-completion questionnaires thus minimizing potential biases that derive from each method by aggregating the figures. Both Edinburgh studies (1977 and 2005) applied surveys to a wide range of stakeholders from visitors to business establishments and participating performing groups, with which they were able to obtain a pluralism of spending figures. The Glastonbury report (2007) made a crucial distinction between off-site and on-site expenditure, which is of great importance for tracing the destination of spending, hence enabling an account on traders’ origin. Last but not least, the Ithaca study (2007) included extensive information on the conditions assuring the quality of the survey: fairly representative sample, well recruited interviewers, potential biases of survey highly considered and thorough design of the survey conducted despite its weak content. Hence, each study dealt effectively with a different aspect of the criterion but none was overall successful.
	All of the above mentioned characteristics should be part of an Economic Impact study so as to assure its quality of data. Furthermore, a study should be very analytical in describing the methods used and the way they are designed and conducted in order to establish the transparency of the research. A combination of methods not only as far as the population surveyed is concerned but also the type of method itself –implementation of the costly direct interviews and the more practical self-completion questionnaires- will result in triangulation of the data gathered. Triangulation of data sources allows researchers a comparison and cross-check of the information assembled “at different times and by different means” (Patton, 2002). By obtaining data from diverse sources, a research can examine how potential biases of each method may distort the data and lead to more proper estimates of figures by combining them. This will definitely be a very useful tool in determining visitor number, expenditure as well as other substantial figures for the impact assessment. 
	With regards to the time of the survey, it can be concluded that the middle of the festival period may be the most accurate in terms of reducing the effects of emerging recall and prediction biases of the respondents. This enables the selection of a more representative sample by the recruited interviewers, as well as the inclusion of day-trippers in the sample. The researchers should, though, account for the occurrence of such biases and consider them while analyzing the data in order to enhance the reliability of the study. 
	Another method suggested to diminish memory and prediction biases of the respondents is the “on-site diary” (Wilton and Nickerson, 2006). Visitors would be provided at the beginning of the festival such questionnaire - forms where they can directly report on their daily spending patterns and return it by postage paid mail to the researchers by having some promotional offers as a motivation. On one hand, such diary questionnaires may produce more reliable and accurate spending data. On the other hand, this method would entail a great cost for the researchers and, also, an additional difficulty as far as the sampling procedure is concerned.
	Overall, collecting data from various stakeholders and diverse sources is necessary to achieve triangulation. The most essential feature here, though, is that the research acknowledges all limitations imposed by the data collecting methods and refers to them in specific details, since they appear inevitable. This will enhance the validity as well as the accuracy of the study. 

Chapter 4
Element 2: Defining the Area of study
	A precise definition of the impact area of an event may seem of lower importance to a casual researcher. However, determining the area of study is crucial for an accurately estimated economic impact. This will also signify the area where the survey will take place as well as the on-site and off-site expenditure of visitors to the community.
	Snowball (2008) argues that a mistaken expansion of the research area could lead to an inflation of the true impacts of an event, while such act might be intended by the commissioners themselves, as a means to create a more impressive number. Crompton (2006), on the other hand, emphasizes that a false overextension of the area could alternatively underestimate the total impact, as a greater amount of its visitors would, then, be considered as local residents and, thus, be subtracted from the visitors number, since local visitors are not usually included in the analysis. 
	In addition, Vrettos (2006) points out the strong correlation between the definition of reference area and the multipliers’ effects as the latter figures are determined by the relative size of the specified study area. The geographical area for estimating the economic impacts of an event should be accurately defined as “to include the most important aspects of the impact, and yet not too much area or the effects will be masked by extraneous economic activity” (Daniels, Norman and Henry, 2004: p.184). Hence, special attention should be drawn so as “deliberate” mistakes are considered and avoided.

I. Research Findings
i. The former Edinburgh study
	The Edinburgh paper (1977) depicts the geographic area markedly with the graphical aid of a map of the region: the city of Edinburgh along with the rest broad area of the Lothian region of Scotland, UK. The map includes the whole Lothian region in the presentation of the study area. However, multiplier’s effects are estimated exclusively for the City of Edinburgh, which implies the main impact area. The study explicitly states, “the boundaries of the City of Edinburgh adopted for this study correspond approximately to those of the old City of Edinburgh Corporation and nit to the present local government district” (p. 11).

ii. The National Arts Festival study
	The National Arts Festival study (2004) defines as area of impact Grahamstown region in South Africa, without though precisely depicting the boundaries of the geographical area. They accurately mention, though, that this decision affects the size of the multiplier, because if the study area included the broader Easter Cape the multiplier would be significantly larger due to less leakage.

iii. The recent Edinburgh study
	The Edinburgh study (2005) defines as area of study mainly the City of Edinburgh Council Area but also the Lothian Region and the rest of Scotland. The geographic boundaries of these areas are explicitly depicted with a map, thus markedly accounting on the impact areas analyzed. The delineation of these three specific geographic areas of study will also be illustrated in the estimation of three different multipliers, one for each area.

iv. The Ithaca study
	The Ithaca study (2007) does not accurately define the geographic area of impact. It initially states that the festival takes place in Ithaca, which is part of Tompkins County, and the survey questions regarding the visitor expenditure, such as on accommodation or food, refer to it specifically. Further on, however, when referring the local economy, it includes the whole Tompkins County and, also, based on the content of the questionnaire, we can conclude that local residents are those living in Tompkins County and non-locals are the non-residents of that area. Hence, confusion is generated hereby, but still we can assume that the whole Tomkins County is considered as the impact area since leakages outside the local economy are those outside the County. The researcher should have defined more clearly the actual boundaries so as to avoid implications, which may have affected the estimation of multipliers’ size as well.

v. The Glastonbury study
	The Glastonbury study (2007) clearly determines the area of impact comprising of the following four geographic areas: Mendip District (including Glastonbury), Bristol and Bath, the rest of the South West Region (excluding the above areas), and the UK and international. Even though the researchers consider Mendip District as the local area of study, they actually estimate the net impacts of the whole South West Region, which may be accounted as an overly expanded assessment area for the specific festival causing an overinflated final figure. The research could have been more careful in defining the precise geographic boundaries the festival had an actual effect on.

Table 6: Defining the Area of study

Studies	Defining the Area of study	
The Economic Impact of the Edinburgh Festival 1976 	Clear description of the Lothian impact area as well as graphical presentation of the geographic boundaries with a map: City of Edinburgh and the Lothian Region, Scotland/UK	
The National Arts Festival 2004: Festino Survey and Impact Study	Well defined as the Grahamstown region in South Africa, thus determining the relative size of the multiplier	
Edinburgh's Year Round Festivals 2004-2005 Economic Impact Study	Markedly defined as the City of Edinburgh Council Area, the Lothian Region and the rest of Scotland - precise geographic boundaries depicted with a map	
An analysis of the Economic Impacts of Ithaca's Light In Winter Festival 2007	Not accurately delineated                                                                                                                          Confusion is generated between the city the festival and visitor expenditure take place (Ithaca) and the broader county area considered as the local economy (Tompkins County)	
Glastonbury Festivals 2007: Economic Impact Assessment	Clearly depicted as comprising of four distinct geographic areas: Mendip District, Bath & Bristol, Rest of South West Region, UK & International                                                                                              Certainly exaggerated study area potentially causing overinflated impact figures	

II. Interpretation of Research Results
	This element is directly linked with the estimation of the impact figures and the multipliers’ size, since the assessment area determines the geographic boundaries both amounts are studied within. When researchers fail to accurately define the specific boundaries and present an exaggerated area, implications are incurred leading to overinflated spending figures. Also, when the festival takes place in a large-sized city, it appears difficult to isolate the impacts and indicate a narrow geography of study.
	The Edinburgh studies (1977 and 2005) both depict the impact area with the graphical presentation of a map, thus precisely demonstrating the geography and do not leave room for inconsistency to emerge. On the other hand, the Ithaca study creates confusion between the local area of impact and that of the broader region, thus extending the area of visitor spending with further implications in the final estimations.
	The assisting usage of a map is highly recommended to ensure a clear indication of the study area along with an accurate description of it.

Chapter 5
Element 3: Determining Visitors’ number
	This element is considered by Bragge and Snowball (2006) together with the data collection method as the most influential for the estimation of the total impact figure.	 It basically consists of two research challenges: determining what type of visitors will be included in the spending impact and estimating their total number. 
	Regarding the inclusion of certain types of visitors, it should be always borne in mind that Economic Impact studies report the net injection of new money into the local economy, the monetary flow that is directly related to the studied event. Therefore, it is emphasized that spending by visitors who are also local residents is disregarded, since it would have been spent in the area irrespective of the event taking place and, thus, it is considered as a recycling of already existing regional money (Crompton, 1995; Janeczko, Mules, and Ritchie, 2002; Cultural Ministers Council, 1997). However, Snowball (2008) argues that expenditure by local spectators that would have otherwise occurred outside the local area should be added as net impact. For this reason, it is suggested that local resident respondents are asked to account on the supplementary expenses to their usual monthly spending (Snowball, 2008). It is also stated that local residents expenditure can represent a positive net impact in the local economy as long as it is proved that this injection would have occurred elsewhere outside the community (Getz, 1991 and Seaman, 2004).
	Apart from the local origin of spending, Crompton (1995) makes a significant distinction as far as the motivation of visitors attending is concerned. “Time switchers” are visitors who planned their trip in the area so that it coincides with the event but they would have visited regardless of its presence, while “casuals” are already visitors of the area who happened to decide randomly on attending the event rather than doing something else in the area (Crompton, 1995). Since spending made by “time switchers” and “casuals” would have taken place in the impact area irrespective of the event, it should, hence, not be attributed as net economic effect (Crompton, Lee and Shuster, 2001; Tyrrell and Johnston, 2001; Snowball and Antrobus, 2002). Frechtling (2006), though, denounces this exclusion of expenditure viewing it as speculative, and as such it may defect the research project’s validity.
	The second challenge involved in the visitors’ number is the actual estimation of their total size. Determining this figure is of high significance in estimating the final amount of visitors’ expenditure, which is the main component of the total impact, as well as in drawing visitors’ capacity of the festival and thus shaping its prestige (Snowball, 2008). The latter could entail a deliberately overestimated number of visitors enforced by the commissioners to enhance the event’s profile.
	There are several different methods recorded by researchers to measure this figure. Janeczko, Mules, and Ritchie  (2002) suggest a combination of the figures provided by the ticket service, which are liable to errors of double counting, and aerial photographs taken by helicopters during the event period counting individuals. The latter method could be vastly costly and, hence, impractical for the majority of researchers. Wilton and Nickerson (2006), in their study, report a model of “thirty nine highway entry points” and “the state’s eight main airports” constructed for a specific impact area to estimate the number of tourist inflow by counting heads. Another method was applied by Antrobus et al. (1996) combining ticket sales figures and accommodation data drawn by visitors’ survey. However, both latter methods largely depend on the representative sampling of attendees and, hence, they can be highly biased, as Snowball (2008) points out.
	Furthermore, the “selection bias” generated in the sampling process and based on the length of staying of the attendees, can also affect the estimation of visitors’ number (Wilton and Nickerson, 2006). Since longer staying visitors are more likely to be surveyed than shorter staying and day trippers, this can have a marked effect on the calculation of the size of total visitor days, thus importing a substantial bias in the estimation of the net direct impact.

I. Research Findings
i. The former Edinburgh study
	The Edinburgh report (1977) makes an initial strong distinction between local and non-local resident visitors of the festivals. It provides an accurate audience composition, according to the origin of residence, for each festival respectively as well as each week of the three weekly festival period. However, there is no account on the motivation for attending the festival. The research does not distinguish motivated visitors to casual attenders or those who had already planned to visit the area and coincided with the event period. Also, no distinction is made between longer and shorter staying visitors, but day-trippers, for whom there is no extensive information with regards to their particular spending patterns, are included.
	As far as the visitors’ number is concerned, the study makes an estimation for “staying festival-goers”, those deriving from outside the Lothian region and stayed in Edinburgh for one night or more, providing solely the figure of 341,000 visitor nights. This figure was drawn approximately calculated by the 60% of the total amount of tourist nights spent in the city’s accommodation establishments during the festival period. The researcher was not able to make a more accurate estimation due to the constraints enforced by the data collected from the visitors’ survey. Furthermore, he does not elaborate on the reasons justifying how this number was calculated accordingly. However, a composition of these 321,000 bed nights is reported based on visitors’ origin as well as the type of accommodation used.
	He, also, provides a figure of 100,000 day-visitors attending the festival, warning readers, though, to consider this estimation cautiously as it was not made based on the survey but on a rather unconventional measurement. From the total amount of non-local visitors expenditure on festival tickets, the amount spent by “staying festival-goers” on this spending type was subtracted, thus shaping the amount of spending on tickets by day-trippers. Then, this figure was divided by the average ticket prices and this estimation indicated the number of day visitors. 
	Vaughan (1977), very appropriately, argues on the opportunity cost for local residents’ spending on festival tickets. He excludes residents from the city of Edinburgh and the whole Lothian region from the visitors’ number because “to treat local resident expenditure in the same way as visitor expenditure on the festivals it must be assumed that the festivals induced “extra” spending on behalf of local residents” (p. 29). Vaughan remains skeptical, highlights this implication and eventually considers it in the final assessment.
	Overall, the estimation of visitor number is quite weak and based on many research assumptions due to the lack of primary data. In addition, the study solely provides the number of bed nights spent by visitors for attending the festival, without distinguishing their original motivation nor including the total days spent in the festival. This percentage of holiday overnight visitors to Edinburgh may very likely include non-festival attenders. The researcher, though, has registered considerable caveats for the usage of these estimates.

ii. The National Arts Festival study
	The National Arts Festival study (2004) makes an initial segmentation of attendees according to origin –being local residents or not-, age, home language –Afrikaans, English, Xhosa, Zulu and other- and the method their data has been retrieved by –interviews or self-completion. Numerous tables are provided on the “festinos”, the so-called visitors and local residents, with regards to the number of festival visits, return visits, length of staying distinguishing number of days and nights, type of accommodation, number of performances attended according to type of event and expenditure by category. 
	As far as the estimation of the amount of “festinos” is concerned, Antrobus and Snowball (2004) adopt the ticket sales method: “dividing the number of tickets sold by the average number of ticketed events attended by respondents”. The researchers used data only from the interviews method due to their reliable quality of sample. In this section, there is a single reference to the “festinos” motivation for attending the festival, stating that only for 8% of the respondents the festival had not been the primary reason for visiting the area. Hence, regarded as relatively unimportant percentage, they did not account for it in the impact calculations. Some commentators could suggest that such proportion seems considerably small to be true and, thus, a weakness to the research method. Another substantial observation is that local residents, comprising the 17% of the total number of “festinos”, are included in the visitor number, thus potentially generating implications in the calculation of the total net impact.
	In addition, the researchers attempted a double cross-check of the estimated number with the visitor number provided by the Festival Office, which appeared similar, and with the gate count at two important entry points of the festival, which demonstrated an anomaly. The latter estimation could be liable to traders’ entering this point as well as double-counted visitors entering more than once per day. The researchers took this inconsistency into account, made a fairer rounding for this estimate and considered it in the determination of the number of “festinos”. Furthermore, they incorporated suggestions for future research in examining the occurred peculiarity of the entry point counting. 
	Overall, determining visitors’ number has been thoroughly delineated by Antrobus and Snowball (2004). The application of two collecting data methods gives them the opportunity of more representative data drawn by the respondents, and the cross-checking of the actual amount with three available methods enhances the accuracy of the estimation. Even though the report lacks an account on day-trippers, the researchers, also, acknowledge it and view it as an area to be further explored by future practitioners. However, the greatest implication occurred here is the inclusion of local residents to visitors’ number without considering whether their spending is a direct result of the specific festival.

iii. The recent Edinburgh study
	The Edinburgh study (2005) has initially distinguished the festival attenders to visitors, performers, delegates and media representatives, and has identified for each type of attender the origin of residence as well as the primary reason for attending in order to make the right estimations in the net impact assessment. As far as their total number is concerned, the report does not provide an analytical account. The research estimated the number of visitors by subtracting the number of performers, delegates and media representatives, as measured by the festival organizers, from the total amount of festival attendances. However, the latter number has not been elaborated by the researchers. We can merely assume that the total number of festival attendees is, also, provided by the festival organizers.
	Then, specifically for the visitors of the festival, an audience profile was constructed according to demographics, origin, length of staying, motivation for attending, source of information on the festival and opinion on the event’s experience. It is precisely stated that the issue of whether the festival comprises the main reason for visiting the area has been highly considered by the researchers. Local residents visiting the festival are entirely excluded since it is assumed that their spending would have taken place in the city despite the event. 
	Moreover, since the report is accounting on a series of seventeen festivals occurring for an annual period of time in Edinburgh, some of the visitors may have attended more than one festival, which may have involved implications in the counting process. Questions such as allocating the spending amounts of each individual across different festivals emerge. The research clearly indicates that, “the survey was used to identify the Festival that was the main motivation for the trip” (p.7). The research has, thus, attributed each individual’s spending to the most crucial festival event for one’s decision to visit Edinburgh in the first place, so as to avoid potential implications of double counting. The festival visitors have been allocated, according to the main festival they attended, to three categories: “Summer Festivals”, “Winter Festivals”, and “Autumn and Spring”.
	Determining visitor number has been a relatively complex process for this specific study, mainly due to the festivals’ particular nature (large number of them and extensive annual duration). Even though the researchers have very properly defined visitors’ origin and primary motivation for attending, which are of high significance for the impact estimations, no details are provided with regards to the method or the source denoting the total number. An attenders’ estimate is reported without any justification behind it, which seriously affects the transparency of the research. Furthermore, no account at all has been made to day visitors. More essentially, the decision to allocate visitors according to the main festival they attended may have generated additional assumptions in the final estimates, particularly when visitors attended several festivals throughout the year – and the research seems to have disregarded this issue in the final analysis.

iv. The Ithaca study
	The Ithaca study (2007) makes an initial distinction between the 2,291 local residents and 696 deriving from outside the local area attenders, which makes almost 3,000 over the 3-day festival. These figures were drawn from the combination of three different sources, so as to obtain a “triangulation” of methods to enhance the accuracy of the estimate, as stated in the report. Daniels (2007) used ticket sales information, figures extracted from the survey and scholarly literature to estimate the visitors’ number. She has taken into account a downward bias caused by “a combination of inaccurate reporting of onsite ticket sales at the door and figures reported by the ticket center” (p. 67) from the data received by festival managers, so as to make an estimate of the total number of people visiting the festival. Then, Snowball’s method is applied in measuring visitors’ number based on dividing total amount of counted visitors by their average length of staying. Finally, the proportion of foreign visitors to the area is determined by the survey as 23.3%, thus entailing a total of 696 non-locals out of the 2,987 festival attenders.
	One can argue that the method used here does not really constitute triangulation of sources, since each of them were not applied to cross-check the final figure but were respectively combined to shape the final estimate of the figure. Overall, visitors’ number is determined considerably accurately in this study by this combination of sources but, certainly, triangulation has not taken place to assure the correctness of the estimate.
	Apart from distinguishing local residents to non-local, the study does not further examine primary motivation for attending the festival. Even though the questionnaire includes a question on the reason for visiting the area by the non-local residents, the result is not included in the main body of analysis. On the contrary, it is assumed that there were no “casual” or “time-switcher” visitors to the festival in spite of 21% of non-locals stating that the festival was not the main reason for visiting Ithaca. The researcher has intentionally chosen to disregard this finding, possibly driven by the fact that non-local attenders’ number is already relatively too small, compared to the local, thus undermining the non-local impact of the festival. Furthermore, no account has been provided towards shorter-staying or day-tripper visitors.
v. The Glastonbury study
	The Glastonbury study (2007) makes an initial visitor segmentation, seemingly according to origin or ticket purchased, to the following types: standard ticket, traders’ pass, local residents and others (performers, staff etc). Apart from this categorization, the report, also, distinguishes visitors according to their postcode of origin from different parts of the UK and abroad, and their length of their trip.
	Regarding the motivation for visiting the area, 94% indicated the festival as the only reason for their trip, while trivial percentages of visitors stated that they had extended their visit to Mendip District because of the festival. It appears reasonable, as the area does not seem to be a touristic resort attracting people irrespective of the festival. The main component of this element, the estimation of actual visitor number, has not been accurately described in this report. The researchers provide a total number of 177,500 estimated festival visitors including local residents, performers and festival crew without, though, stating how this estimate was drawn. 
	All in all, the Glastonbury paper (2007) manages an accurate visitor segmentation indicating origin of home residence, length of staying and motivation for attending. However, the information provided seems overly summarized thus limiting the transparency of the study. Similar observation ought to be made for the visitor number estimate, which may have caused further implications in measuring the final impact.

Table 7: Determining Visitors’ number
 	Determining Visitors' number	
Studies	Distinguishing Locals & Non-locals	Considering primary motivation for attending	Distinguishing Longer-staying & Shorter-staying visitors	Estimating the amount of visitors	
The Economic Impact of the Edinburgh Festival 1976 	YES                                    Local residents not included in visitors' number	NO                                                                     All non-local festival attenders included irrespective of their motivation for attending	NO                           Limitations of data did not allow the research for such account but day-trippers included	Weak method:              raw data constraints led to an estimate based on the 60% of total holiday bed nights in Edinburgh (to be treated with caution)	
The National Arts Festival 2004: Festino Survey and Impact Study	YES                                    Local residents distinguished from non-local but still included in visitors' number	YES                                All festival attenders included, though, due to the considered low percentage of those visiting not primarily driven by the festival	YES                                     Both data collecting methods provided estimations of average number of days and nights of visitors, day-trippers not accounted though	Ticket sales method originally applied but also cross-checked and adjusted with other two available estimates, analytical description of process and future considerations included	
Edinburgh's Year Round Festivals 2004-2005 Economic Impact Study	YES                                                Local festival attenders distinguished and excluded from all categories (visitors, performers and delegates, media representatives)	YES                                      Identified for all festival attenders	YES                                       Length of staying identified for all festival attenders, day visitors not accounted though	Total number of attendances provided where the total number of performers, delegates and media representatives were subtracted - no explicit account made	
An analysis of the Economic Impacts of Ithaca's Light In Winter Festival 2007	YES                                            Local residents accurately distinguished from non-local but included as festival visitors together with non-local	NO                                       Even though the questionnaire contains relevant question the research ignores its results	NO                                               Not even considered by the research	Ticket sales method as identified by Snowball has been applied with data drawn by the festival organizers and the survey	
Glastonbury Festivals 2007: Economic Impact Assessment	YES                                                       Local visitors accurately distinguished from other visitors although included in total number	YES                                     Majority of survey respondents indicating the festival as main reason for visiting	YES                                 Length of visitor trips determined by the survey, day trippers not accounted though	No details provided regarding the stated estimate of visitors' number	

II. Interpretation of Research Results
	This criterion, particularly, constitutes a critical component for the derivation of the final figure since the estimate of the number of visitors shall be multiplied with the spending patterns to eventually result in the total spending. It is highly important that the individuals included are all directly linked with the event rather than random people who happened to be in the area during the studied period. Apart from their motivation for attending, all visitors included should be stemming from outside the local area, otherwise the research is merely reporting on a recycling of local residents’ money. These two features will determine the actual net figure and this, also, tends to be the most typical pitfall of the Economic Impact practice.
	None of the analyzed studies manages successfully with all aspects of determining visitors’ number. While all reports distinguish local residents from the visitors, only the two Edinburgh studies (1977 and 2005) exclude them from the final number while the other three have integrated them. As far as the motivation for attendance is concerned, the National Arts Festival (2004), the Edinburgh (2005) and the Glastonbury (2007) reports properly estimated only those visiting the area primarily driven by the festival, even though they did not account precisely on “casual” or “time-switcher” visitors. The Edinburgh (1977) and the Ithaca (2007) papers should be pointed out as examples to avoid, as they both include all festival attenders regardless of whether the event has been the main reason to attend. Length of staying, on the other hand, remains a problematic area for all the five reports. While only the former Edinburgh study (1977) incorporates day-trippers in the estimation, limitations of data did not allow the researcher to further account on shorter and longer-staying attenders. The National Arts Festival report (2004), however, should be recognized for combining both data collecting methods, self-completion questionnaires and interviews, for estimating length of staying thus enhancing the quality of findings.
	With regards to the actual method of estimating visitors’ number, the majority of studies provide figures without elaborating on how these estimates were shaped. The former Edinburgh report (1977) should be definitely discredited for its utterly inappropriate method, as it is purely speculative, despite the notable research caveat. Nonetheless, the National Arts Festival report ought to be acclaimed for applying triangulation in cross-checking the estimated figures with all available data sources. In addition, the originally applied “ticket sales method” appears accurate and rather practical for most researchers as it makes use of data accessible to them by the visitor survey and the festival organizers.
	This element appears challenging for most researchers, as multiple data sources are required in order to lead to a safe estimate. Since some festivals contain events that coincide or visitors attending more than one event during a day, the implications in measuring the number are increasing. It is highly important, therefore, that researchers explicitly analyze the method applied as well as all underlying assumptions to ensure the validity of the estimate. Triangulation of data sources is recommended so as to reach a more accurate estimate of the visitors’ number. Moreover, researchers should use all available data they have at their disposal in determining visitors’ origin, primary motivation and length of staying, with a view to obtain a net result, which is the targeted aim.

Chapter 6
Element 4: Evaluating Supply Side Constraints
	This criterion has been adopted from the issue Seaman (2004) has identified as “Supply Constraints”. They are greatly distinguished from the first three challenges collected as the “demand side errors”, and consist of three major, but rather understated, research issues: displacement issue (or crowding-out), opportunity cost of expenditure allocated to the festival (or deadweight) and negative externalities.
	Seaman (1987, 2003 and 2004) has strongly argued on local infrastructure capacity limitations generating the so called “displacement issue” or also known as “crowding-out” effect. Constraints induced by the tourism capacity potential as well as local infrastructure to accommodate the inflow of visitors can result in the community’s inefficiency to absorb all incoming tourists (Seaman, 2003). The displacement issue is, hence, viewed as space and hospitality limitations to foster the additional injection of visitors due to an event. Snowball (2008) thinks this crowding-out of visitors who would normally visit the region by visitors arriving particularly for the event can even lead to a substitution of the expenditure in the impact area. For this reason, not all spending by visitors in the region should be included in the impact figures because if the event had not occurred, then the displaced visitors would have spent it there anyways (Snowball, 2008). 
	Seaman (2004), however, thinks that there might not be a significant spending displacement, since supplementary expenditure, generated by local residents who would otherwise spend it outside the impact area as well as additional incoming visitors, overweighs the loss of income. Also, it can be assumed that most regional economies can manage to absorb both normal and event visitors due to their accommodation and infrastructure sufficiency (Seaman, 2004); not to mention the fact that most popular events’ dates are established well in advance in order to facilitate the general organization. Regardless of these assumptions, it would be extremely difficult to measure the degree of crowding-out in order to subtract the portion of displaced income from the net direct impact (Snowball, 2008; Seaman, 2003, 2004).
	The displacement issue can, additionally, be attributed to local residents abandoning deliberately their homes during the festival period in order to avoid potential inconvenience from congestion or for other reasons (Cultural Ministers Council, 1997; Snowball and Antrobus, 2002). Whatever the causes of this phenomenon are, Economic Impact practitioners tend to neglect its potential effects in the overall impact (Phynthian-Adams, Sapsford and Southern, 2008). 
	A recent research (Litvin and Fetter, 2006), on the Spoleto Festival in the USA, demonstrated that the festival itself had been detrimental in attracting a greater amount of visitors. The researchers proved that the hotels had been less occupied on average during the festival days, thus identifying this as displacement phenomenon “due to concerns of crowding, congestion, and/or price inflation during the event” (Litvin and Fetter, 2006: 46).	
	Other researchers view displacement as a diversion of local income from potential regional spending on other sectors of the economy (Johnson and Thomas, 2001; Long and Owen, 2006). “Especially the opportunity costs of not using the land and other resources for another beneficial project” (Johnson and Sack, 1996: 374) comprise the second aspect of the supply side constraints. Seaman (1987) argues that it is essential to examine the source as well as the alternative spending plan of the allocated resources to the festival organization, which would have been otherwise spent in other regional projects potentially more financially sustainable. Snowball (2008) agrees by questioning on other possible uses of the public funds within the impact area, while Madden (2001) attributes these states grants to enforced expenditures of residents through increased taxation.
	Local government funding to the festival organization budget should be subtracted from the final net figures, as it would represent a recycling of local money deriving from citizens’ taxes (Crompton, Lee and Shuster, 2001). Crompton (2006) adds to the alternative costs of public subsidies those of private funds, which could also result in a negative net economic effect of the actual event. The latter possible effect appears to be disregarded in the scholarly literature as well as the actual economic impact reports, despite the great attention given to indicate estimates of the net figures rather than the gross.
	The third issue of supply constraints consists of the negative externalities generated by a festival event. Shaw and Rogers (2005) indicate negative impacts induced to the communities by the occurrence of festival events, such as congestion, increased noise pollution, litter and criminality, without attempting to measure them though, because of their non-market value nature. Crompton et all (2001) add to those the frustration of local residents, due to their burden with potentially higher taxes while sharing the economic benefits of the event with foreign stakeholders. Most Economic Impact practitioners tend to ignore these issues because of the difficulty to identify and measure them. However, they should be reported qualitatively at least (Snowball, 2008), as they entail additional expenses for local residents indirectly through their taxes –increased public cleaning and police service costs.

I. Research Findings
i. The former Edinburgh study
	The Edinburgh study (1977) contains an extensive qualitative report on the opportunity costs generated by the festival to the local community resources: “payments for goods and services from outside the area”, “dividend remittances”, “tax payments”, “employment benefits foregone”, “wage benefits foregone” and “trade diversion”. By the latter term “trade diversion”, Vaughan (1977) means the displacement issue caused by tourists’ reluctance to visit the area during the festival period due to the likelihood of the city being over congested by event attenders. He does not attempt to measure it since, as stated, the survey provided no proof of the occurrence of crowding-out effect and, hence, he assumes that the creation of additional trade due to the event counterbalanced any potential displacement issue. 
	With regards to the opportunity cost of the local resources allocated to the festival, the study devotes a respective account on them without, though, evaluating them qualitatively due to the implications involved. As far as local residents’ spending on the festival event is concerned, the research acknowledge its potential substitution effect but does not measure it either, as it is assumed that the supplementary income induced to local establishments would have outweighed it. Furthermore, Vaughan (1977) assumes that this local expenditure may possibly stem from savings, which provides an additional flow of local income. However considerate this remark may appear, it remains a pure assumption.
	As for negative externalities, Vaughan (1977) states that the typical social costs such as additional expenses on police or medical services were not covered by the impact assessment. Despite the absence of a further analysis on this topic, it is certainly of high importance that such a remark is made thus accounting on this aspect as well. Overall, this study does not measure supply side constraints so as to consider them in the final impact figure, though it refers fairly on a qualitative basis to them serving the readers’ accurate comprehension.

ii. The National Arts Festival study
	The National Arts Festival study (2004) merely outlines some of the generated supply side constraints stating clearly that they are excluded from the total impact calculation. Their qualitative remarks on “costs”, as the researchers call them, are listed as following: “increased competition for local businesses”, “increased costs to local municipality in terms of extra policing, traffic control and refuse removal”, “inconvenience to local residents and festival refugees”, “possible crowding out of non-festival tourists” and “possible use of political capital and the opportunity cost of festival sponsorship” (p.36).
	Although Antrobus and Snowball (2004) identify various issues of the supply side constraints, they do not further examine their possible effects on the Economic Impact figures.

iii. The recent Edinburgh study
	The Edinburgh study (2005) has made a considerably extensive account on the supply side constraint issues. The researchers have identified the displacement issue caused by capacity limitations of hotels and other accommodation establishments particularly during the summer period of the events’ occurrence. Apart from accommodation occupancy constraints, they recognize a potential loss of tourism business induced to local enterprises at that specific period. A quantitative estimation of the substitution effect is not attempted here, but a grounded qualitative presentation of the phenomenon is made, based on the information collected by the accommodation and retailers’ associations of Edinburgh.
	“Nearly all (90%) of the hotels had turned potential bookings away and of these 40% had turned away more than 200 potential bookings…over the Festival period” (p. 58) while “the licensed trade and restaurant sectors are huge beneficiaries during the Festivals, more so than retailers” (p. 60). The research has highlighted the specific festival events, which were most beneficial for the hotel industry sector of the region, and, also, supplemented the report with various positive as well as negative views of the business sector stakeholders with regards to the festival impact.
	Even though the researchers acknowledge that a level of displacement may have taken place due to the festival in Edinburgh during the summer period, they highly praise its role on behalf of the elevation of the whole tourism business sector of the region. They think that “the evidence of excess demand is largely from people hoping to attend the Festival and not being displaced by it” (p. 62), hence it is difficult to evaluate tourism in the region if the Festival would not take place. They strongly believe that potential displacement issue generated by the festival visitors is clearly counterbalanced by the undoubtedly “positive effect that the Festivals have on attracting other trips to the city throughout the rest of the year” (p. 62).
	Irrespective of capacity constraints resulting in the displacement issue, the research, also, considers the overlap of some festival events during the summer period causing an additional source of potential loss. While the report does not provide a definite response to whether this overlap obstructs or contributes to the festival impacts, the majority of survey respondents appear to support it, as they state it “adds to the Festival experience” (p. 65). The researchers appear to think that the size of the actual overlap of festival events is considerably limited.
	As far as the opportunity cost of the festival resources is concerned, the research has accounted on the origin of all grants and subsidies comprising the budget of the festival organization in order to examine potential substitution effects generated. As will be viewed in the final estimation of the impact figure, the researchers have excluded all funds deriving from public sources as well as those that would have been allocated locally to other projects regardless of the event taking place. 
	Furthermore, the study uses the term “additionality” to delineate potential leakage effects, deadweight and displacement issues that visitors’ and other attenders’ expenditure may have generated due to the festival. All these issues, dealing with the opportunity cost of spending directly because of the festival, will be further analyzed and evaluated in the estimation of the total net impact figure. The research has considered all potential substitution effects caused either by leakages from outside the local economy, or alternative choices of spending allocated to other sectors of the city, and has added a quantitative account on them according to the geographic origin of stakeholders. Local residents’ expenditure as well as tourist spending that would have occurred in the area anyways are both excluded. Certainly all this data is retrieved by the conducted survey for the specific needs of the research.
	Last but not least, negative externalities are mentioned in the study as referenced by tourism establishments affected adversely by the festival. While those included focus on the decrease of profit margins for a number of retail enterprises, no other reference to the local community’s externalities is made.
	Overall, the recent Edinburgh study analyzes broadly supply side constraint issues and incorporates their evaluation at the aimed estimation of the total net figures, as will be further analyzed in the relevant section. Not only have the researchers consulted the actual stakeholders being directly affected by these issues, but also have attempted to thoroughly evaluate most of them beyond the scope of a qualitative analysis. Furthermore, this consistent attempt highly reflects their effort to demonstrate the actual net effect of the festival.

iv. The Ithaca study
	The Ithaca study (2007) has not made any particular reference to supply side constraints. As far as the displacement issue is concerned, we may assume that since the festival takes place during the winter period in a non-high touristic season, such issue does not emerge. Neither has the researcher accounted on negative externalities generated by the festival’s events. However, a qualitative remark on both issues should have been made.
	As far as the opportunity cost of the festival resources are concerned, Daniels (2007) merely mentions that public funds would alternatively be allocated to other projects outside the local economy, thus entailing no benefit at all for the local economy. Similarly, local residents’ expenses due to the festival would have occurred alternatively outside the local economy, hence they are not displaced. This section of the report is somewhat vague and includes a number of underlying assumptions by Daniels so as to resolve the lack of sufficient data dealing with supply constraint issues. No careful consideration is given here while the research has not efficiently demonstrated the absence of potential losses generated by the event.

v. The Glastonbury study
	The Glastonbury report (2007) has identified the displacement issue caused by the excess demand of visitors flooding the impact area because of the events during the summer period. The researchers driven by the interviews held with local business establishments, as well as the accounts of festival organizers, have estimated a small speculative percentage of displacement caused by potential non-festival tourists canceling or postponing their trip due to either fully booked accommodation or fear of congestion. They assume that a relatively small displacement issue takes place since the festival period is established in advance and is considerably short. 
	The research actually estimates a different percentage of displacement for each impact area: 10% for Mendip District, 5% for Bath and Bristol and 5% for the rest South West Region. These percentages are underlying assumptions of the researchers, as they clearly state, based solely on the interviews of stakeholders. Therefore, these percentages should be treated with significant caution while no available specific evidence can support them.
	With regards to the opportunity cost of expenditure allocated to the festival, the report accounts on deadweight of spending amounts that would have taken place in the study area regardless of the festival. Again, the researchers have quantitatively evaluated this amount for each impact area separately (Mendip District, Bath and Bristol, and South West Region), as it is mainly generated by local attenders from each area. Thy have, also, accounted on the festival expenditure that is sustained to leakages from outside the local economy due to production, performers and staff fees. The research has provided a budget of the festival resources and their distribution to a broad range of categories attributed to suppliers, contractors and wages according to the area they have been leaked. Hence, the study reports on the organization’s expenditure spent on each segment of the impact area individually (Mendip, Bristol and Bath, South West Region) in order to allocate final expenditure to each area separately. 
	However, what the researchers have ignored is the origin of the festival resources, whether they stem from local grants or corporate sponsorships. Indeed, the study should be highly praised for providing an extensive report on the budget allocated to expenses according to area of assessment, but neglecting the initial origin of these resources constitutes a major pitfall in the final net impact estimation.
	As far as the last issue of supply side constraints is concerned, the researchers include specific aspects of negative externalities as extracted by the stakeholders through the interviews. “Traffic congestion”, that according to some establishments is presented as exaggerated by the local media due to the events, and “confusion of the actual location of the festival”, both may have prevented additional tourists from visiting during that period. The study focuses only on externalities harming local businesses, while it neglects the potential negative effects caused to the local community as a whole.
	To sum up, the Glastonbury report (2007) devotes a fair analysis on various supply constraint issues from potential displacement to the opportunity cost of visitors’ spending as well as the destination of all expenditure made by the festival organization. Moreover, including a quantitative evaluation of displacement and deadweight issues in the final net assessment should be substantially highlighted, even though based on numerous research assumptions,. On the other hand, the researchers have ignored to account on the origin of the resources allocated to the organization budget, which can prove to be an important drawback of the research. Last but not least, negative externalities generated to the local residents are not accounted at all.

Table 8: Supply Side Constraints

 	Evaluating Supply Side Constraints	
Studies	Displacement issue	Opportunity Cost of allocated resources	Negative Externalities	
The Economic Impact of the Edinburgh Festival 1976 	YES                             Identified but no attempt to measure it as it is counterbalanced by additional trade creation	YES                        Extensive but mere qualitative report on it as it is outweighed by additional locally created income	NO                                  No proper account on them but a significant reference to the lack of evaluating them	
The National Arts Festival 2004: Festino Survey and Impact Study	YES                                      Mere reference but no attempt to evaluate it	YES                                    Mere reference but no further examination	YES                                              Identification of them but no attempt to examine their effect	
Edinburgh's Year Round Festivals 2004-2005 Economic Impact Study	YES                                                Recognized and thoroughly analyzed but no attempt to measure it as assumed to be offset by additional visitor spending induced by the festival	YES                                            Extensive report not only based on a qualitative ground but also quantitative evaluation of these, accurately accounted 	YES                                        Identification only of those mentioned by local business establishments but no further examination	
An analysis of the Economic Impacts of Ithaca's Light In Winter Festival 2007	NO                                       Not mentioned at all	NO                                                 Only mentioned as an assumptive statement on its absence	NO                                              Not mentioned at all	
Glastonbury Festivals 2007: Economic Impact Assessment	YES                                                 Identified and quantitatively evaluated despite being based on research assumptions	YES                               Identification and quantitative evaluation of deadweight of visitor spending, identification of festival resources' destination but initial origin ignored	YES                                      Account solely to those mentioned by local business establishments but no examinations of those induced to local community	

II. Interpretation of Research Results
	Supply constraints seem to be an area of the festival impacts assessment most practitioners overlook, mainly because of the difficulties in evaluating them. In the specific studies synthesized here, researchers have at least attempted to indicate a number of them qualitatively in order to serve the consistency of the overall evaluation. 
	While the Ithaca report (2007) is the sole one disregarding all supply side issues, the rest of the studies identify the displacement phenomenon as well as the creation of opportunity cost of the festivals’ allocated resources. The Edinburgh paper (2005), especially, makes an extensive and quantitative account on these issues by considering them in the total estimation of net impact. The study should be credited for its accurate method in measuring “additionality” issues caused either by leakages of visitors’ spending and the deadweight of local residents’ expenditure or the alternative cost of the public funds from the festivals’ budget. The origin as well as destination of spending of numerous stakeholders are accounted here in order to reach a net impact.
	As far as negative externalities are concerned, though, none of the studies really captures the loss of income induced to the local community from the occurrence of a festival. The National Arts Festival (2004), the Edinburgh (2005) and the Glastonbury (2007) studies identify only a limited number of these issues but do not further attempt to examine their adverse effects and their potential cost for local residents. No reference was made with regards to the extra expenses borne by the local community for the supplementary services, cleaning or security enforcement for instance.
	Economic impact practitioners should thoroughly investigate the start and end point of all types of expenditure, so that they can effectively evaluate supply side constraint issues and consider them in the net calculations. Moreover, despite the implications in evaluating negative externalities, they could extract information on their effects by interviewing local stakeholders, both residents and municipality actors directly dealing with relevant issues, such as the local police department or the cleaning service sector. Accounting on these issues even on a qualitative ground is of high significance for the overall analysis of impacts of an event.
Chapter 7
Element 5: Estimating Multipliers’ Effects
	Apart from the first round of spending generated mainly by visitors and other festival stakeholders to the local economy forming the direct impact, Economic Impact studies are also interested in estimating the flow-on effects in the region (Cultural Ministers Council, 1997). As already stated, multiplier effects measure the secondary rounds of expenditure in a community by indicating the indirect and induced effects of the initial festival spending on income, output and employment (Janeczko, Mules, and Ritchie, 2002). This requires the calculation of three distinct types of multipliers –income, output and employment-, although a number of researchers appear to ignore or misrepresent the latter two. 
	While Keynes introduced and developed the concept of multiplier effects on the on-going recycling of money through an economy, Leontieff created the first input-output model used for calculating the size of multipliers, on which subsequent researchers have been based to create analogous models tailored to a region’s particular characteristics (Phynthian-Adams, Sapsford and Southern, 2008). It should be noted, though, that multipliers do not necessarily derive from input-output modeling, as there are also other, less common, research methods.
	The size of the multipliers mainly depends on the “leakages” and “linkages” of the economy studied, which represent the business interrelationships –“the level of imports and exports”- of the impact area with the outside area (Snowball, 2008). A larger and more diversified local economic base means a more self-sufficient region, thus entailing greater expected multiplier effects (Cwi and Lyall, 1977).
	It is highly recommended that each study applies its own estimated multipliers, custom made specifically on the characteristics of the considered area, as the structure of leakages and linkages seems to differ within each region (Crompton, 1995; Cultural Ministers Council, 1997; Phynthian-Adams, Sapsford and Southern, 2008). Seaman (2003), though, suggests alternatively the usage of previous multipliers measured for the specific area or simply an approximate estimation of the multiplier’s size due to the high cost and time required to construct a new model. Certain countries, such as the USA or Canada, have formulated their own regional input-output modeling systems –IMPLAN, REMI and RIMMS II are some marked examples- to help communities measure their own adjusted multipliers (Schwer and Rickman, 1995). 
	Baaijens and Nijkamp (2000) conducted a meta-analysis of tourism income multipliers of eleven different case Economic Impact studies and concluded to a model estimating the size of a multiplier for any geographic area based on a set of geographic and tourism variables. An additional method in determining multiplier size has been applied by Snowball and Antrobus (2002): comparing the size as well as the derivation of multipliers of similar studies to have a rough estimate of the studied multiplier (Snowball, 2008). 
	With regards to the employment multiplier effects, which represent the additional direct and indirect employment generated by visitors’ expenditure, Crompton, Lee and Shuster’s guide (2001) identifies three typical research pitfalls. Most commonly, the estimated figures appear to ignore any distinction between full and part-time jobs and focus, solely, on induced full-time occupation, which is less likely for the majority of extra occupational needs of events. Most importantly, an abusive assumption is made that all additional jobs will be filled by new employees rather than by the existing ones who may extend their working hours for the specific event period. Also, the supplementary working staff is assumed to be deriving from the local area, while a considerable percentage of them tend to be covered by people who commute from outside the region. All these assumptions may significantly cause an exaggeration of the actual employment impacts on the local economy.
	While Seaman (2003) denotes this crucial procedural step of the Economic Impact practice as the most commonly erroneous due to researchers’ limited experience, it has to be mentioned that the majority of final reports lack an explicit presentation on it. Therefore, their analysis is subjected to misrepresentation and discredit. 

I. Research Findings
i. The former Edinburgh study
	The Edinburgh study (1977) applies the traditional Keynesian income multiplier to explore the circulation of new money injected as a result of the festival within the local economy. It, also, provides a graphic presentation of this flow, which illustrates the circulation and leakages of the studied economy to indicate that the secondary rounds of spending are smaller than the initial expenditure. Vaughan (1977) explains it as “the inability of any local economy to retain more than a fraction of any expenditure owing to leakages such as taxation, the purchase of goods and services from outside the local area and the remittance of profits” (p.25).
	The report provides a number of “local income coefficients”, as hereby the multipliers are called, without though explicitly demonstrating the actual estimations.  Local income coefficients, as well as their effects, are calculated for expenditures on festival tickets by non-locals for each festival separately and, also, for ancillary spending made in the city of Edinburgh by non-locals. The latter figures are provided for each type of expenditure (accommodation, retail, souvenir, café/pub, transport, other), for each type of tourist by area of residence (Scotland, rest of UK, W. Europe, N. America, elsewhere) and for each type of tourist by the accommodation used (hotel, guesthouse, bed & breakfast, tent, caravan, friends & relatives, others) respectively. The size of all income multipliers, indicated within the report, are in the range of 0.1-0.5. The “weighted average” of every set of multipliers is 0.285, as given by the researcher. 
	Even though the additional local income generated to the festival by non-local spending on tickets of events and off-site spending is, indeed, calculated by the researcher, no explicit information is provided on the estimation of the size of the multipliers used for this analysis, hence the calculations are not transparent to the reader. 
	The research excludes local residents’ spending from the multiplier’s analysis, as it would constitute a recycling of already existing money, as very rightly stated. However, no account is made on the employment multiplier’s effects on the regional economy. 
ii. The National Arts Festival study
	The National Arts Festival report (2004) applies only a single income multiplier based on a previous study estimate, the 1996 National Arts Festival study, which a previous paper of the researchers (Snowball and Antrobus, 2002) had considered as the most valid, by comparing it to other relevant figures. The estimate is sized as 0.18, according to that research. Antrobus and Snowball (2004), though, suggest future research should conduct an original calculation of the size of multipliers for the region. 
	The researchers apply the previously estimated multiplier for the direct impact as a whole, without estimating the indirect and induced impact of each amount of expenditure separately. No further analysis has been provided to the multiplier’s effects and no reference has been made to the employment multiplier.

iii. The recent Edinburgh study
	The Edinburgh study (2005) initially identifies two different types of multiplier effects. “Supplier effect” is described as the beneficial impact on local suppliers due to sales increase, while “income effect” is the rise in employment or income of local residents already employed, generating successive rounds of spending within the local economy. The research applies the relatively outdated Scottish Tourism Multiplier Study (STMS) 1991, which measures income, output and employment multipliers for the tourism and non-tourism sector of Edinburgh and Scotland. The study has accurately adjusted only employment multipliers to the inflation rate, so that their size is updated, considering that income and output multipliers are not influenced by inflation. Also, it is worth mentioning that the researchers are using different multipliers for the festival organization expenditure, non-tourism sectorial multipliers, while for visitor expenditure tourism multipliers are applied. All income multipliers cover the range of 0.3-0.6, while output multipliers are within 1.5-1.9. For the output multipliers it has to be additionally stated that the direct impact is included in the actual size, thus being 1.5-1.9 rather than 0.5-0.9, as they refer to the total output of each sector of local economy. As far as the employment multiplier is concerned, the research has estimated approximately 3,200 full-time jobs for Edinburgh and 3,900 for Scotland in total, as generated by the festival for a year. It seems though a rather exaggerated figure considering the temporary nature of festivals.
	The study includes an analytical estimation of the income, output and employment multiplier effects for tourism and non-tourism sector by combining supplier and income effect for each area individually: Edinburgh, Lothian Region and Scotland. It should be acknowledged that they also indicate the different multipliers’ size for each expenditure stream: accommodation, food and drink, entertainment, shopping and transport.
	As will be examined in the following criterion, “the various multipliers are applied to the net expenditure after allowing for additionality and displacement effects to produce the final results in the form of output or expenditure and income to each of the geographical areas and employment” (p. 10). The study has meticulously estimated all multiplier effects and devoted an analytical explanation of the calculations for each category of expenditure. However, with regards to the employment multiplier effects no additional consideration has been given to the particular nature of the new working placements created. 

iv. The Ithaca study
	The Ithaca study (2007) used the IMPLAN Input – Output model to create its own customized model based on Tompkins County leakages and linkages with the outside economy, according to data from all sectors available from relevant figures produced in 2000, as vaguely stated. The research devotes a fair analysis on the estimation of various multipliers, output and employment, for both types of visitors, local and tourists, and three different types of expenditure, eating and drinking, accommodation and tickets. It, also, estimates the output multiplier’s size for the festival’s organization expenditure separately. All output multipliers applied for this assessment - on local, non-local and festival’s expenditure - are in the range of 1.2-1.8; as previously stated the direct impacts are also included in the actual multiplier size, hence 1.2-1.8 rather than 0.2-0.8.
	With regards to local residents’ spending, the researcher recognizes the view certain economists share on limiting the use of multipliers due to the recycling of existing money. For this reason, in this specific study, Daniels (2007) has not included induced impacts for locals’ expenditure, hence “the results provide a conservative estimate of the economic impact of expenditure” at the festival. Therefore, only the direct and indirect effects of local spending have been estimated here.
	Daniels (2007) calculates multipliers’ effects for total tourist expenditure, local spending –only the indirect impact-, external revenue sources of the festival as well as the number of new jobs generated by the festival. As observed, local residents have spent the greatest amount of expenditure owing to the festival, although this can raise questions as the research has incorporated all local spending without proving its direct correlation to the event. As far as the employment multiplier’s effects are concerned, the report concluded that the festival is responsible for creating 5.5 full-time jobs – a seemingly reasonable figure due to the festival’s small size. The nature of these jobs should be treated with certain caution due to the temporary character of the festival, even though not mentioned by Daniels.
	Overall, the Ithaca’s Festival paper has successfully adopted the IMPLAN model to the special sectorial characteristics of Tompkins County and has accurately described the calculation of output and employment multipliers’ size as well as their effects. However, a deeper analysis and consideration of the creation of new jobs should have been accounted. 

v. The Glastonbury study
	The Glastonbury study (2007) has estimated a set of output and employment multipliers for each of the impact areas, based on the Scottish Tourism Multiplier Study (STMS) 1992 adapted by the research to fit the festival’s specific characteristics. With regards to output multipliers, the research has estimated the size of multipliers for on-site and off-site expenditure on a range of spending categories – “accommodation”, “food and drink”, “shopping”, “transport” and “entertainment” -where eligible for each assessment area – Mendip District, Bath and Bristol, rest of South West region, whole South West region. The multipliers’ effects are calculated for each area individually and are added to the net direct impact of each of the districts for the final impact estimation. The specific size of the output multipliers is in the range of 1.2-1.6 – including the direct impacts in the actual estimation, hence 1.2-1.6 rather than 0.2-0.6.
	Concerning the employment multipliers, the research similarly estimates the multipliers for each type of expenditure for on-site and off-site spending as allocated to each segment of the assessment areas. They have, also, estimated the employment effect generated by the festival organization spending. For these estimations, the STMS 1992 was adjusted to the updated inflation rate, as in the case of the recent Edinburgh paper (2004). The employment multiplier’s effects are reported in full-time equivalent jobs for each of the impact areas in 2007 values, indicating totally 1,100 newly created jobs in the whole South West Region. It should be noted here, though, that no account has been made on the nature of the additionally created jobs despite the feedback provided by the business establishments’ as well as the key stakeholders’ interviews. The research could have examined this aspect through the already collected data.
	We may assume that the Glastonbury study (2007) probably adopted the STMS 1992 model due to lack of other multipliers’ modeling available. However, it should be stated that even though the model was adapted to the festival’s particular conditions, the South West Region of the UK is markedly different to Scotland regarding its leakages and linkages featured. Moreover, the model appears relatively outdated. Therefore, the researchers should have adopted an updated or more recent input-output model tailored to the area’s particular characteristics.

Table 9: Estimating Multipliers’ Effects

 	Estimating Multipliers' Effects	
Studies	Income and Output multiplier measure	Employment multiplier measure	
The Economic Impact of the Edinburgh Festival 1976 	YES                                                                                      Application of traditional Keynesian income multiplier model and estimation of the effects but no explicit details on the specific calculations
Income multipliers: 0.1-0.5	NO                                                                                             No account	
The National Arts Festival 2004: Festino Survey and Impact Study	YES                                                       The income multiplier estimate applied is based on a previous study on the festival, no more details provided
Income multiplier: 0.18	NO                                                     No reference	
Edinburgh's Year Round Festivals 2004-2005 Economic Impact Study	YES                                                                     The Scottish Tourism Multiplier Study 1991 applied with necessary updates adjusted, detailed analysis on the calculation of various multiplier effects: income and output multipliers for tourism and non-tourism sector for each area of impact separately
Income multipliers: 0.3-0.6
Output multipliers: 1.5-1.9	YES                                                 Estimation of number of newly created jobs as a result of the expenditure made by visitors, performers, journalists and the festival organizers, based on STMS 1991 employment multipliers updated to inflation, no other reference on the cautious usage of this number
Estimated full-time jobs in Scotland: 3,900	
An analysis of the Economic Impacts of Ithaca's Light In Winter Festival 2007	YES                                                          IMPLAN Input-Output model applied to sectorial characteristics of Tompkins County, customized model for this specific study, output multipliers' calculation for locals, tourists and festival organization expenditure estimated separately, analytical estimation of indirect and induced effects
Output multipliers: 1.2-1.8	YES                                             Estimation of number of new jobs created based on customized IMPLAN model but no account on the particular nature of these jobs
Estimated full-time jobs locally: 5.5	
Glastonbury Festivals 2007: Economic Impact Assessment	YES                                                                        The Scottish Tourism Multiplier Study 1992 applied to the festival's particular characteristics to estimate output multipliers for on-site and off-site spending for each impact assessment area separately
Output multipliers: 1.2-1.6	YES                                           Estimation of number of new jobs generated for each of the impact areas as a result of visitor and festival organization expenditure, based on the estimation of employment multipliers adapted by the STMS 1992 updated for inflation, no details on characteristics of these new jobs
Estimated full-time jobs in whole South West Region: 1,100	

II. Interpretation of Research Results
	This remains a laborious task for the majority of researchers, as it ideally entails the formation of an input-output model specifically produced for the studied area. Such effort exceeds the resources as well as the scope of research for most impact assessment attempts. Therefore, Economic Impact practitioners adopt an already made relevant leakages and linkages’ framework and adjust it to the particular characteristics of the impact area. The importance of this element should be emphasized due to its being a crucial component of the expenditure, comprising the indirect and induced impact leading to the estimation of the final net figure.
	An attempted comparison of the size of multipliers applied by the researchers would be rather ineffective since all selected studies account on local economies of different size and geographic area, thus entailing different leakages and linkages and, consequently, diverse multipliers. Apart from that, two studies have solely estimated income multipliers, while other two have accounted on output multipliers, which would further hinder the comparison. It can be said, overall, that the income multipliers used by the meta-analyzed studies are all in the range of 0.1-0.6, and the output multipliers within 1.2-1.9, both of which indicate a similar size for the calculation of secondary impacts. No significant diversions are, hence, noted with regards to the relative size of the different multipliers applied.
	Another issue is the estimation of multipliers’ effects on expenditure made by local residents, most scholars agreeing on its total exclusion. The National Arts Festival (2004) and, partially, the Ithaca (2007) papers calculate secondary impacts for local spending, even though the latter report estimates only the indirect local effect. Both should be highly faulted, as such estimations are liable to account on the recycling of local already existing money and should be avoided.
	The former Edinburgh (1977) and the National Arts Festival (2004) reports apply previous models but do not provide sufficient details on the estimation of the effects, rather than stating the calculated figures, thus harming their validity. These two, also, do not include employment multiplier effects in their analysis, which is a further lack.
	Out of the rest studies, all of which borrow previous models to estimate the secondary spending, the Ithaca report (2007) manages successfully as it adapts the IMPLAN input-output modeling and customizes it to the specific sectorial characteristics of the assessment area. It continues estimating the amounts of indirect and induced spending separately for each type of expenditure. The recent Edinburgh (2005) and the Glastonbury (2007) papers, on the other hand, incorporate outdated models and, particularly, the latter study should be faulted, as it applies a model tailored for a different area without making any further adjustments. 
	Regarding the employment multiplier effects, the three above stated studies provide an estimation of the additionally created jobs due to the festival, either directly or indirectly, in full time equivalent. However, all of them fail to consider the specific characteristics and nature of these jobs. They all stumble on the common pitfalls identified in Crompton, Lee and Shuster’s guide (2001), previously mentioned within this chapter, thus resulting in overinflated employment impacts on the local economy. Especially exaggerated figures, as in the case of the recent Edinburgh (2005) report ought to be cautiously treated.
	All in all, it is recognized and advised that Economic Impact practitioners adopt already made input-output models to estimate multipliers’ effects in order to avoid supplementary implications. However, they should consider that the borrowed modeling should be specifically tailored to the characteristics of the studied area, and be relatively updated, as linkages and leakages effects change over time. Furthermore, the studies ought to provide explicit account on the second-round impact measurements, and report on the cautious usage of the employment figures as well as all additional assumptions undertaken for the estimations. 

Chapter 8
Element 6: Estimating Total Impact figures
	The last criterion derives accordingly as an aggregation of the previous five elements and is the principal goal of the Economic Impact research. The final step of such research attempt is the addition of all necessary figures in the most appropriate and transparent way to lead to the total net impact figure generated by the event. Before further continuing with the process of the total measurement, it has to be reiterated that the ultimate figure is aimed to represent the net value of the overall expenditures spent. 
	Each study provides a final result based on the different methodology applied by the researchers according the availability of data and particular characteristics of each case. As already stated in the first chapter, a general framework followed by most practitioners is the addition of the net Direct to the Induced and Indirect Impact as outlined by the definition of the method itself (Bille and Schulze, 2006). It seems that the execution of the method is relatively simple. However, there are many implications involved in the actual assessment as viewed in the meta-synthesis: methodological errors, double-counted expenditures and abusive assumptions.
	Tyrell and Johnston (2001), in their model of net Direct Impact assessment, supplement expenditures made by visitors with those made by performers, volunteers, media representatives, sponsors and vendors. They also note that, to counterbalance the risk of double counting certain amounts or including leakages from outside the local economy, it is necessary to examine the origin, destination and the reason behind each expenditure. Seaman (1987), on the other hand, strongly suggests that the earnings of traders deriving from outside the region, signifying a competition for regional enterprises, should be subtracted from the direct impact as they take away potential earned income from the local economy. Additionally, as far as public funds are concerned, Crompton (1995) argues that those deriving from local government sources ought to be, also, subtracted as comprising recycling of local citizens’ taxes.
	Therefore, estimating the true net impact by considering substantial factors such as which part of the contribution is made from outside the impact area and which actually represents a recycling of local money, can lead to a negative net figure (Tyrell and Johnston, 2001). However, most studies appear to neglect these factors as will be investigated in the following chapter, and hence present an overinflated total net impact. 
	In the estimation of the net Economic Impacts of an event, Seaman (2003) points out the existence of long-term externalities generating benefits for the “productivity and economic development” of a region as well as its general cultural profile and educational level, despite the implications involved in evaluating them. Shaw and Rogers (2005) provide an extensive analysis of the non-market values induced by an event, such as the cultivation of pride, cultural identity, social unity and other intangible spillovers disseminated throughout a region, all of which are seldom depicted by Economic Impact practitioners. Snowball (2008), also, elaborates on the long-run impacts induced by the media coverage spending; while non-measurable, they are typically not reported in studies.
	All in all, under this criterion, it will be investigated how diverse researchers apply their tailored methodology in the final estimation, what types of expenditure they have included or excluded, and which are the most common errors repeated in the process. Since there is no consistent method exercised by practitioners, various patterns as well as underlying assumptions, adjusted to the constraints of each assessment, will be explored.

I. Research Findings
i. The former Edinburgh study
	The Edinburgh study (1977) provides estimates on numerous amounts on peripheral festival impacts and emphasizes an essential limitation for the explanatory power of the impact assessment. Due to the data limitations of this specific research, all estimates should be treated with caution and, very importantly, due to rounding, total figures may not precisely correspond to each other. 
	The overall estimates this report provides are the following: expenditure of non-local residents on festival tickets, off-site spending made by overnight visitors to the city of Edinburgh, expenditures made by members of visiting companies for two of the festivals, expenditure made by day-trippers, spending by local residents on festival tickets, grants, donations and sponsorships by local and non-local sources as well as all their multiplier effects for the city of Edinburgh, the Lothian region and the rest of Scotland.
	Vaughan (1977), apart from the above estimates of all the festivals’ impacts, concludes to a figure of 3,7 million pounds “flowing to the City of Edinburgh, from outside the Lothian Region, by way of visitor expenditure and grants” (p. 34). As he precisely mentions, this figure does not represent the net economic impact for the local economy as it includes both “private and social profit”; this is a gross figure including leakages and services purchased or provided from outside the impact area. Taking these features into account, the report results in a final net figure 960,00 pounds as a “positive benefit to the residents of the Lothian Region”.
	Both total gross and net figures attributed as the final impact figures are not explicitly measured, hence the reader cannot cross-check them from the peripheral impact estimates and this, essentially, does not establish transparency for the research. It is though of high credit to the research that special attention has been drawn to estimate a net figure excluding any leakages, which would overinflate the economic impact of the event.
	An additional observation ought to be made regarding the estimation of the grants and sponsorships of the festival. A calculation of these along with the off-site expenditure made by the performing groups of the festival may be considered as double counting of the same money, since we can assume that their spending derived from performing fees, which stem from the festival budget. Another consideration should be given to the origin of off-site traders where non-local expenditure was allocated: some of these establishments may be foreign temporary vendors due to the event occurrence. Therefore, their earnings should not be regarded as income generated for the local community.
	Special acknowledgment should be made to the study for accounting on numerous issues that even recent researchers, often, neglect. Vaughan (1977), additionally, refers to a number of long-term benefits the festivals have generated on behalf of the local community: socio-cultural spillovers, trade creation opportunities, advertising expansion for the local area and the foundation of a festival industry for the region.
	All supplementary qualitative references on non-measurable impacts as well as the research accuracy and consideration on several methodological issues demonstrate a relatively proper report. However, the project is entitled to several limitations due to the lack of sufficient data and other essential details on the final estimations, thus diminishing its overall value.

ii. The National Arts Festival study
	The National Arts Festival study (2004) before estimating the final impact figures, it provides an account on the limitations imposed by the Economic Impact methodology itself. The researchers recognize that their methodology is based on various underlying assumptions crucially influencing the final figure, which they regard as a “rough estimate rather than an exact calculation” or even a “best guess” (p.32). This type of studies is utterly inappropriate to be used in arts advocacy, according to Antrobus and Snowball (2004). They, hence, consider a more qualitative method for measuring the value of art based on its contribution to society. 
	With regards to calculations of the Economic Impact figures, the researchers provide an analytical account on the final estimate.  Combining the two main collecting data methods, they estimated total visitor expenditure considering the “festinos” number estimate, which included local residents, and the average visitor spending, which included “additional spending by local residents”. By the latter statement, it is possibly implied that they have included local residents’ supplementary expenses to their normal patterns, which would indicate a direct result of the event. However, the estimated number they provide has been calculated on a previous part of the report as the average spending per person including local residents, with no distinction to their normal spending. Therefore, it can be assumed that they actually included all local residents’ expenditures, neglecting to prove that they have been generated directly owing to the festival. 
	Apart from this pitfall, the report has included consideration on group spending by leaving out spending data stemming from large groups, on the reason that “one person could not possibly know with any accuracy what the spending of each person in each category was”. Except for total visitor expenditure, Antrobus and Snowball (2004) have added “sponsorship” and “craft market spending” to the direct impact and have subtracted a 30% percentage of the whole initial impact as comprising of “immediate outflows”. This percentage is based on the 1996 producers’ survey, pointed out as outdated. The “immediate outflows” the researchers have subtracted, refer to all induced leakages stemming mainly from earnings of traders and performers from outside the local economy. The net direct impact is then supplemented with the income multiplier’s effect so as to result in the total economic impact.
	Acknowledging that the final figure should be seen as an estimate, they view the usage of such studies as “a quick quantitative way to evaluate arts projects” and compare similar arts events (p.33). Furthermore, they delineate long-term benefits generated by the festival, since as argued such studies appear to ignore non-market values that are the main driver of arts festivals. They identify as the festival’s positive spillovers the following: “effects of media coverage on local businesses and other stakeholders”, “long term economic growth and productivity increases” and “value of externalities to local population” (p. 36).
	Additionally, the National Arts Festival report (2004) accounts on suggestions for future improvements of the methodology applied, as well as a comparison of the 2004 results with those of previous studies on the festival. Overall, Antrobus and Snowball (2004) provide a simple but analytical calculation of the impact figure considering a number of important parameters and constraints in their attempt. The implication though is that their total figure is not entirely net as it incorporates all local residents’ expenditure, which should have been proved to be resulting directly from the festival.

iii. The recent Edinburgh study
	The Edinburgh study (2005) primarily emphasizes the constraints imposed by the Economic Impact methodology regarding various assumptions that the researchers had to make in order to lead to a final estimate. They have, therefore, recognized and denoted them to assure the transparency of the study. A special note ought to be made concerning their concerted effort to delineate the net final impact by accounting, particularly, for all the economic activity stemming directly from the festival. In the final calculations, spending incurred irrespective of the festival events has been removed.
	According to the methodology of the present paper, impact data stem from four different sources of festival attenders: spectators, performers, media representatives and festival organizers. Also, special attention has been drawn to separate the expenditure generated by each of the festivals; spending figures are allocated to the festival that was most influential for the attenders visiting the city. This particular approach has been adopted to avoid double counting of spending amounts. For each stream of impact the research, firstly, derives the direct expenditure by subtracting all potential leakages from outside the local economy and displaced amounts that would have occurred in the city regardless of the event. Local residents, for instance, who would have spent money in the area anyway, foreigners who would have visited Scotland even if the festivals had not taken place, local journalists or performers who would have been in the city, they are all excluded. “Displacement therefore varies, between Edinburgh, the Lothians and Scotland in terms of the origin of the visitor” (p. 14). 
	The research highly positions the factor that the event is the primary reason for tourists visiting Edinburgh and spending money in the city. Particularly for estimating the festival organizers’ net impact, the research has made some additional assumptions. All public funds deriving from local as well as national sources are entirely subtracted, as it is assumed that they would have been allocated to alternative projects within these geographical areas. Also, sponsorships stemming from small regional bodies are excluded, while those generated by major corporate sponsors are included, as they are more likely to have been alternatively allocated to projects outside Scotland. Next step is adding multipliers’ effects to the net first round of expenditure so as to reach the final impact figure for each category of stakeholders.
	The research should, certainly, be highly praised for indicating all these assumptions as far as the substitutions effects’ account is concerned. Furthermore, it should be mentioned that the net impact of festival organizers’ expenditure attributed to some of the festivals is estimated as negative. The negative effect is explained by the assorted leakages of the locally raised revenues allocated mostly to performers’ fees outside the local economy, and is further intensified by the resulted lost multipliers’ effects. Certainly, this negative effect should be viewed collectively with the positive effect generated by the festivals’ additional income induced by other stakeholders’ expenditure. However, recognizing the major effect that the “additionality” issues may have caused to the net figures appears of vital significance for the estimation of a true net impact.
	The report provides, finally, total sums of net economic impact figures for each geography and festival individually. An implication ought to be stated with regards to the problematic method applied for allocating visitor spending figures on the festivals, in the case of visitors attending more than one. Indeed, assigning each spending pattern to the single festival most crucial to one’s decision to visit Edinburgh, may lead to misguiding conclusions on behalf of the festivals, whereas aggregating the amount equally on all events a visitor attended would seem more appropriate.
	In addition, the study contains various qualitative references to long-term benefits generated by the festivals, such as “social inclusion”, “education”, “cultural diversity”, “city promotion”, “civic pride”, “quality of life” and “creativity”. There is a special report on the value of media coverage of the Edinburgh Festivals resulting in the promotion of the city worldwide. Even though a quantitative account of the media value is demonstrated, the research does not attempt to incorporate this measurement in the total Economic Impact figure of the festivals due to further stated implications. 
	To sum up, the recent Edinburgh paper (2005) is fairly analytical and accurate in the net impact assessment despite certain implications. The researchers have taken a number of parameters into consideration for delivering a proper net impact figure and suggest a systematic method for measuring the economic contribution of a festival to the local community. By acknowledging the assumptions as well as further limitations of such studies, they enhance the validity and explanatory power of this specific attempt. Furthermore, they conclude that there is a growing need for a more holistic model accounting on the primary targeted impacts of cultural events related to socio-cultural values, going beyond the pure economic framework that views festivals as a touristic product.
	
iv. The Ithaca study
	The Ithaca report (2007) has delivered a detailed but somewhat complicated analysis of the calculation of the final impact figure. It provides a rather problematic calculation of the total net figure based on the estimation of three main expenditures: local, non-local and festival’s external revenue. Before further proceeding with the actual estimation of each, it should be emphasized that the research has adapted a relatively common methodological pitfall despite its effort to avoid it.
	The paper, very proudly, argues upon typical errors of the Economic Impact study practitioners and highlights its own specific methodology for allowing distinction between net and gross figures. As precisely stated, “by further accounting for the source of expenditures whether from within or outside the county, by tourists versus locals, and the reason for expenditure, we obtain the net economic impact or the actual net increase in regional expenditures directly related to the event” (p. 71). 
	However, Daniels (2007) has incorporated all expenditure made by local residents attending the festival without examining the substitution effect, whether this money would have been spent in the local area irrespective of the event. The research assumes that the festival event “keeps residents at home who would otherwise leave as well as encourages more local spending for entertainment, food and beverages, and merchandise” (p. 69) and, hence, it includes all local spending without considering additional expenses to their normal local spending pattern. This, apparently, results in a major research issue of recognizing the final estimate as a net figure, thus undermining the validity and accuracy of the whole paper.
	Irrespective of this foundational error turning this concerted effort into a failure, the actual calculations of the impacts appear somewhat vague. Daniels (2007), indeed, provides analytical calculations of the estimates of local direct and indirect spending as well as tourist direct, indirect and induced spending, both measured according to type of expenditure, with ample qualitative remarks on spending patterns. However, most spending patterns’ estimations are based on research assumptions due to lack of original data. For example, hotel expenditure patterns are outlined according to total holiday spending on hotels during the festival period, with no specific account on visitors of the festivals though. The researcher recognizes that this figure might have been, therefore, exaggerated. Another example is the food and drink expenditures, which are based on the figures produced by a previous survey due to data constraints, for which the same amount of spending has been abusively assumed for all three types of visitors: tourists staying in hotels, tourists not staying in hotels and local residents. Similar assumption has been made concerning the ticket expenditures. These comprise three marked cases of underlying assumptions the research required to make due to data limitations that have importantly influenced the spending figures. A note should be made, as well, with regards to neglecting the retail visitor expenditure, because of further limitations of data drawn by the survey. 
	Furthermore, festival organization’s sources of revenues are added, both those deriving from outside the local economy and local funds. For the latter injections, the research precisely mentions the particular reason why they are accounted: “they are funded by the Tompkins County Room Occupancy Tax”, which is applied to non-local visitors to the area. Festival leakages due to performers’ fees and production costs are, then, subtracted from the external revenue sources in order to have an estimate of the net direct impact of the expenditures made by the festival organization. Another special category of additional value to the festival is considered by the research the contribution of volunteers. Although no attempt has been made to quantify the monetary value of their employment to the festival, their assistance is recognized as of high significance.
	Very properly, the research considers leakages from outside the local economy and subtracts them so as to estimate the net direct impact. Indirect and induced impacts are, then, calculated by the aid of multipliers in order to reach the total net impact figure. Daniels (2007) has additionally calculated the “Total Return on Investment” of the festival by dividing the total net impact to the festival expenditures figure. The rate indicates “a net gain in local revenues more than one and half times the costs of the festival” (p. 86).
	Overall, one should acknowledge the thorough research effort to accurately make all calculations of impacts by considering multiplier effects and leakages. However, the report has stumbled on a basic methodological pitfall by including all local spending, thus eventually accounting on a gross impact rather than a net figure. It has included possibly displaced local spending that would have occurred elsewhere in the region. We could assume that this may have been deliberately chosen in order to produce a highly inflated final figure, considering the fact that most of the festival’s impact is attributed to local residents. 
	Apart for this methodological error, a series of underlying assumptions, only a number of which are identified here, may have generated a total figure that is far from the one actually occurred. Therefore, Daniels has not managed to demonstrate that if the event had not taken place there would be a loss for the local economy. Since the greatest part of the gains stemmed from local residents, it is undoubtedly hard to measure the net contribution. All these hindrances derive initially from a poorly efficient survey design, considering all assumptions the researchers were required to make due to primary data insufficiencies. A last area to consider would be the lack of any reference to long-term benefits of the festival generated for the local community.

v. The Glastonbury study
	The Glastonbury assessment study (2007) emphasizes from the beginning that the final estimated figures provided within the report are solely assessing rather than definitive figures of the Economic Impact of the festival. This statement enhances the efficacy as well as quality of the whole research. The study makes a substantial distinction between gross and net impact.
	Gross direct impact is estimated by the addition of expenditure made by the festival organization, -to suppliers, contractors and working staff- to the sum of on-site and off-site visitor spending as distributed to the following impact areas: Mendip District, Bristol and Bath, rest of South West Region, whole South West Region, UK and abroad. Then, Net Direct Impact is calculated by subtracting the estimated displacement and deadweight amounts for each of the following impact areas: Mendip District, Bristol and Bath, rest of South West Region and whole South West Region. Finally, the output multiplier effects per region are supplemented to the Net Direct Impact in order to derive the Total Net Impact for each of the four areas respectively.
	Certainly, the method for estimating the impact figures is simple and quite accurate as it considers various parameters leading to a net final figure. However, the source of the festival organization’s expenditure has not been accounted as far as the locality factor is concerned. On one hand, local grants and subsidies should have been excluded from the net figures, as they constitute recycling of already existing money. On the other hand, the research appears to have stumbled on a typical pitfall in the estimation by calculating both festival’s resources, and performers and staff’s expenditure, which are included in the visitor expenditure and stem mainly from the festival expenses. The issue emerged in the study is double-counting amounts, thus resulting in an overestimated impact. 
	With regards to on-site expenditure of visitors, spending on festival tickets is not included and no justification is provided whatsoever. Additionally, the fact remains that the majority of on-site vendors derive from outside the South West Region, according to the report, but the total amount of that expenditure is allocated to Mendip District. The on-site vendors, markedly, induce a type of competition for local businesses and take away potential earnings from them. The research has assumed that these vendors temporarily belong to the local area. However, their earnings are, mostly, liable to leakages that are not accounted for. This certainly results in misleading conclusions for the overall impact of the festival, in the local area at least.
	Despite these implications, the report should be acknowledged for distributing the actual impact according to the assessment area, and including estimating figures for the supply constraint issues in the final net calculations. Also, the research has considered leaked spending amounts from the relevant impact areas. Accounting for substitution and leakage effects in the final figure is undoubtedly of high significance for reaching a net impact.
	Furthermore, the Glastonbury study contains an extensive qualitative analysis on long-term benefits generated to the local community. The researchers have indicated a number of economic impacts, which could not be measured due to their non-quantifiable nature: trading opportunities for both local and international organizations, non-profit and corporate, being associated with the events - enhancement of the image of Glastonbury town and the wider region due to repeated visits subsequent to attending the Festival as well as to the broad media coverage of the festival events internationally - increased awareness on ecological issues towards the environmental impacts of companies through the Festival’s “strong green ethic” – encouragement of local entrepreneurship by providing multiple opportunities to local people and businesses – support of the economic viability of Pilton village through the Festival’s annual donations. 
	The above mentioned features comprise merely a reference of the overall long-term contributions of the Glastonbury Festivals to the local economy, highlighted by the research, since a greater number of benefits emerge from the interviews with the key stakeholders. All of these remain in the sphere of economic gains, while there is no reference on wider socio-cultural spillovers to the local community.
	The Glastonbury study (2007) demonstrates that Economic Impacts are generated by the Festivals not only within the local economy but also a wider geographic area. The researchers, very appropriately, do not determine a total net figure but four different figures allocated to each of the assessment areas, thus dealing partially with the leakage issue of the effects. They further include a number of future considerations for the economic vitality of the festivals on a long-term basis. They, also, incorporate an account on “monitoring and updating” the Economic Impact research of the Glastonbury Festival events with potential future adjustments and improvements.
	To sum up, a number of implications and assumptions of the research methodology may have led to net figures that do not reflect the real effects. Although several parameters are considered as far as the net estimation is concerned, various pitfalls occurred and significant assumptions were undertaken, which have significantly infected the validity of this study.

Table 10: Estimating Total Impact Figures

 	Estimating Total Impact figures	
Studies	Distinguishing Gross & Net Impact	Types of spending categories included in total figure	Research pitfalls in the measurement	Including positive long-term benefits	
The Economic Impact of the Edinburgh Festival 1976 	YES                       Subtracting leakages from outside the local economy led to a positive net total E.I.
No explicit analysis provided	Visitor expenditure, funds deriving from outside the local community, income multiplier's effect	Estimating both off-site spending made by performing groups and grants allocated to the festival may have led to double-counting error	YES                                Qualitative report on socio-cultural spillovers, trade creation opportunities, advertising expansion of the region and development of cultural tourism 	
The National Arts Festival 2004: Festino Survey and Impact Study	NO                                                                 Even though leakages from outside the local economy have been subtracted, the total impact includes all local residents' expenditure 	Visitor expenditure including local residents', sponsorship and craft market spending, income multiplier's effect 	Including local residents' spending in the total impact figure without proving that it is a direct result of the festival - however researchers are aware of general methodology constraints	YES                                     Qualitative reference on media coverage effects, long term economic growth, local externalities	
Edinburgh's Year Round Festivals 2004-2005 Economic Impact Study	YES                                 Various parameters accounted such as leakages, displacement effects and any type of additionality according to the attenders' origin, aiming to produce a fairly net final figure Negative net effect of some festivals considered	Expenditure made by spectators, performers, journalists and the festival organizers (the latter including external resources), income and output multipliers' effects	Special attention drawn to avoid double-counting of spending amounts but minor implication incurred by the allocation of visitor expenditure to the festivals	YES                                           Particular analysis devoted to the media value generated on behalf of the festivals and wide qualitative examination of long-term benefits for the local community regarding socio-cultural and educational values                                             Special note on the need to develop an impact framework accounting on these values	

An analysis of the Economic Impacts of Ithaca's Light In Winter Festival 2007	NO                                   
Despite accounting on leakages from outside the local economy, the largest part of the total impact figure comprises of local residents' spending 	Tourist expenditure, local residents' spending, festival's external revenues, income multiplier's effects	Including total expenditure made by local residents without considering for substitution effect, various assumptions imposed by data limitations	NO                                                 No reference at all	
Glastonbury Festivals 2007: Economic Impact Assessment	YES                                                          Extensive account on leakages and various substitution effects (deadweight, displacement) estimated thus excluding local residents’ spending in each relevant impact area, despite a number of pitfalls by ignoring the origin of on-site vendors and that of festival resources	Visitor expenditure including performers, staff, volunteers, festival's overall spending, output multiplier effects                       All allocated according to area of impact 	Ignoring the source of festival's budget amounts                    Double-counting of expenditure made by performers & staff and festival organization expenses                                       Neglecting to account on visitor spending on festival tickets                                       Attributing intentionally to local impact area on-site expenditure made to non-local vendors	YES                                 Broad qualitative analysis on the long-term economic non-quantifiable effects on the local community (trading opportunities, corporate ecological politic, promotion of regional image, economic vitality of local area)                                  No reference regarding socio-cultural spillovers	

II. Interpretation of Research Results
	The last element comprises the ultimate and most essential aim of an Economic Impact research, assembling all previous criteria to serve the estimation of net impact figures. It is, also, the area most practitioners are faulted at, since the most common pitfall charged upon them is assuming that all expenditure made in the impact area is new and induced by the festival. Ignoring factors such as leakage effects or the opportunity cost of spending in the study area leads the research to abusive and overinflated results that do not account on the true net effects. As already stated, the Economic Impact studies are interested in depicting the overall spending occurring because of an event, excluding the amounts that would have taken place in the absence of it.
	The National Arts Festival (2004) and the Ithaca (2007) reports fail to distinguish the accounted gross impact with the real net effect, while both incorporate local residents’ spending without proving that it is directly linked with the event. The Glastonbury study (2007), on the other hand, estimates a number of potential external leakages and substitution effects in order to reach a net figure, although it neglects several other parameters of amounts that should have been considered in the net estimation, such as the origin of festival resources and that of traders. All three above mentioned papers fail to deliver an accurate net impact eventually. The former Edinburgh study (1976), also, cannot be adequately evaluated due to its lacking sufficient details, even though it appears to distinguish the net impact by subtracting leakages.
	The newer Edinburgh study (2005), however, manages to successfully consider various factors leading to an appropriate net total figure –leakages, displacement effects, deadweight and substitution effects, each of which according to the origin of attender. The report includes expenditure made by diverse attenders’ categories: spectators, performers, journalists and the festival’s organizers by meticulously estimating the net spending for each. Particular attention has been drawn so that double-counting spending amounts does not occur, whereas other studies have failed to consider this by including both expenditure made by performers and the festivals’ resources. Undoubtedly, the Edinburgh methodology is not flawless, as implications may have occurred in the measurements while allocating visitors to the festivals attended. The researchers have acknowledged that numerous underlying assumptions may have affected the results and have, even, considered the case that some of the accounted festivals generate a fairly negative net effect. It remains the only report making such a valuable contribution to the methodology.
	With regards to positive long-term benefits, most of the analyzed studies make a qualitative reference to socio-cultural spillovers, media coverage effects and economic opportunities. None of them, though, manages to make a thorough evaluation of the extent of these effects on the local community, but merely recognize their non-quantifiable nature. A special note ought to be made to the recent Edinburgh study (2005), for attempting a quantitative account on the media coverage value of the festivals and acknowledging the lack of a suitable framework for assessing such non-tangible effects. 
	Overall, a totally consistent and accurate methodology for estimating the net impact of a festival was not identified from any of the meta-reviewed studies. Practitioners should certainly follow the path the Edinburgh study (2005) has taken by accounting on expenditure by attenders’ category and evaluate all diverse “additionality” issues so as to approach the net figure as much as possible. The origin as well as destination of each spending amount should be thoroughly examined, and considered in the final estimation.
	A parameter that the majority of reports have overlooked is the origin of traders and vendors, where much of visitors’ spending results in. Therefore, they erroneously include spending on traders or vendors, who remove their earnings from the local economy after the event has finished. Apart from that, these foreign businesses take away potential earnings from local establishments, thus increasing the trading competition within the impact area. In addition, negative feelings of injustice are generated among local residents, since they are burdened with the taxes financing the festival events, while sharing the benefits with others. A rational methodology would entail the subtraction of their earnings from the net impact since they are part of the actual leakages, as Seaman (1987) argues. 
	As far as the non-market impacts of a festival is concerned, a suggestion for evaluating them substantially would be to conduct interviews with a number of various stakeholders, including festival organizers, arts consultants, local officials and local residents, so as to obtain a more specific judgment on their expanded effects. This would allow a definitely qualitative and analytical evaluation on the role of these factors to the local community. 
	Vrettos (2006) has stated two supplementary areas commonly ignored by Economic Impact practitioners in the final estimation: evaluating volunteers’ contribution and accounting on the opportunity cost of free passes provided to numerous visitors, the VIPs for instance. To these I would also like to add the clarity with which the final estimations should be reported. As observed, researchers sometimes present impact figures without specifying how they have been estimated, hence limiting the transparency and harming the validity of the study. Particular attention should be drawn in order to include essential details explaining the size of the final net effects as well as caveats for the cautious usage of all impact figures.

Chapter 9
Summary and Conclusions
I. Summary of Research Results
	To sum up the outcome of the research results, none of the meta-analyzed Economic Impact studies deals thoroughly with any of the methodological elements. All reports examined within this thesis proved to be somewhat defective; certainly some attempts were more accurate while some other much less. The estimates provided by them could, also, be characterized as sheer “guesses” because the original studies deriving them contain a number of underlying assumptions and other biases that have crucially affected the results, in spite of some careful research efforts. This leads to the conclusion that the researchers, probably, do not study the methodologies applied by other research attempts, since common pitfalls are repeated over time. One can observe that methodological issues considered in the oldest study reviewed – the Edinburgh report (1976) – have been overlooked in subsequent studies. 
	Furthermore, investigating the economic impacts of a festival in the local community necessitates the isolation of the effects directly associated with the event. Hence, such task would be much more effective to perform in a small region that would enable a relatively accurate identification of the leakages and other substitution effects. This would entail that the methodology itself can be applied only to specific cases. But even the consistency of the methodology is under question, since each research adapts different pattern of estimations according to the availability of data. 
	Another parameter to account is the deliberate presentation of exaggerated figures, driven by commissioners using precarious procedures. Some of these inaccurate methods are, already, viewed in the studied cases, such as inclusion of spending made by local residents, “casuals” and “time switchers”, extension of the impact area, overestimation of the visitor number, neglecting opportunity costs, displacement issues and leakages’ effects, abusing multipliers’ effects and indicating the “gross” impact instead of the actual “net”. Such an observation would further exacerbate the discredit of this economic instrument for measuring the value of cultural assets.
	There are, indeed, various areas of the Economic Impact assessment methodology to consider. Then, is it the method’s foundational failure to achieve its targeted aims? A question of consistency of the methodology would clearly lead to an issue of its greater validity as a research method. While it is hard to provide a definite answer, it is certainly essential to reflect on the overall benefits as well as limitations imposed by this cultural tool.

II. Misrepresentation or refutation?
	This paper has attempted a meta-analysis of Economic Impact studies by synthesizing and reviewing six main methodological issues of this particular type of research, which measures the monetary value of cultural assets. According to the results of the evaluation, all studied cases implement a different methodology so as to assess arts festivals, selected by each researcher to correspond a number of parameters respectively –availability of data, conditions of research, nature of the festival and size of local economy. It has been proved that the overall methodology applied by all five studies is inconsistent and problematic. Overinflated, mostly, impact figures were derived due to inefficient data collecting methods, abusive usage of the raw data, mistaken estimation of the multipliers’ effects and a general tendency to account on gross figures that have not been a direct result of the studied event.
	It can be also argued, by the overview of the selected studies covering a considerable period of time over the last four decades, that there have not been substantial positive changes in the methodology researchers apply. On the contrary, challenges, meticulously considered in the earliest study (1977), have been neglected or ignored by subsequent practitioners. A poignant conclusive remark would be that despite the proliferation of Economic Impact studies the last years, the general methodology has not been improved significantly. Indeed, one would expect that practitioners by observing, comparing and reflecting upon previous Economic Impact studies would be led towards more accurate and consistent studies. However, researchers appear to follow neither the guidelines constructed by academic actors nor the steps achieved by previous empirical studies. 
	Weaknesses characterizing practitioners may be attributed partially to constraints they are faced with when conducting their research, such as limitations of resources, time and data availability, and to deliberations induced specifically to serve targeted policy goals. It is certainly hard to examine the motivations driving researchers’ abusive applications, while random errors generated by inexperienced and careless practitioners ought to be recognized as well.
	Apart from the pursued research results, this study has delineated the process of a meta-analytical methodology on the Economic Impact assessment research by explicitly indicating the specific stages involved. It demonstrated that this methodological procedure can be undertaken by subsequent researchers applying relevant studies so as to extract additional conclusions regarding the method’s accuracy. From the beginning of this research, transparency has been a foundational characteristic to ensure its academic value.
	The meta-synthesis of the selected study-cases has resulted in the emergence of considerable issues of the overall validity of the method. A generally inconsistent research design combined with a vague framework of its achieved aims and underlying limitations have led undoubtedly to a crucial question: whether it is not just a misrepresentation of the method caused by inexperienced or deliberately faulted researchers, but an inherent failure of the Economic Impact research to reach an accurate outcome. The latter argument is favored by an increasing number of academic actors seeking to refute the foundations of this method on behalf of the intrinsic value of culture.
	Providing a definite answer would clearly result in additional implications, since this paper has not accounted on the total amount of such studies but an accessible sample, and is liable to diverse constraints markedly limiting its scope. This thesis does not intend to reject the Economic Impact assessment method despite its numerous drawbacks. Therefore, inaccuracies and inconsiderate assumptions induced by the Economic Impact practitioners are acknowledged together with substantial limitations of this type of research. Both sides should be recognized as partially faulted for the problematic estimation of the economic gains generated by an arts event.
	On one hand, practitioners are called to conduct more thorough and analytical research design in order to ensure an accurate and transparent approach. They should undertake valid and representative methods, and implement the final reports with additional qualitative details and remarks so as to enhance the translation power of the study. Research methodology issues, as examined by this paper, should be meticulously dealt with and overcome by the suggested ways depicted in previous chapters. On the other hand, the Economic Impact method should be accepted for its important contribution and its specific but rather narrow scope. It should become a widely agreed upon fact that Economic Impact research is limited in assessing the short-term market value of a festival for the local and regional economy by providing approximate and not definite figures. The considerably weak and limited outcome of such studies ought to be realized with a view to a constructive and not abusive usage of the results.
	Along with these acknowledgements for both sides, we should consider that even if the economic value of arts festivals is proved, this does not necessarily justify their existence and support. Illustrating the market sustainability of the arts industries, which seems to be a feeble characteristic of this sector, does not importantly contribute to its further development.  Hence, a more holistic framework accounting on the value of culture is required. Since Economic Impact studies are constrained in estimating short-term net financial gains, a model evaluating a broader range of impacts, viewing beyond the pure economic scope, needs to be developed for the sake of culture. Long-term benefits with regards to the socio-cultural, educational, artistic as well as economic elevation of a community, but also environmental and other negative externalities, constitute a much more important set of impacts that have been rather insufficiently represented. A new impact assessment tool is demanded to effectively evaluate, not necessarily on a quantitative basis though, the intrinsic value of cultural assets. Furthermore, part of its baseline should be consistency; a valid framework eligible to every kind of arts events would certainly comply with academic standards along with the specific needs of the cultural environment.
	We shall bear in mind that the main goal of organizing such festivals does not lie in pecuniary gains and market values but, primarily, in the socio-cultural spillovers of the arts aiming at social cohesion, expression of cultural identity, encouragement of creative expression, enhancement of urban image, expansion of various industries and provision of cultural amenities to the regional community. By offering opportunities to people engaging in cultural experiences, the general communal good is being promoted. Apart from social, cultural, economic and environmental aspects, another crucial effect should be highlighted: the learning impact. Individual personal development can be achieved by participating in arts activities.		
	The overall gains for communities and nations are multiple. Particularly in such hollow times of social turbulence accompanying the global economic crisis, a broader impact assessment model of cultural assets is urgently needed. Until such a framework is created, the Economic Impact studies should become a valid assisting instrument for policy makers and arts practitioners. A simple refutation of the method is not a solution. They are, certainly, of beneficial value for a range of stakeholders and applications, as long as both their strengths and weaknesses are taken into consideration, and a consistent research pattern is established.

REFERENCES
Antrobus, G., Williams, V., Fryer, D. Khumalo, B., Streak, J. and Webb, A. (1996), The Economic Impact of the 1996 Standard Bank National arts Festival, Department of Economics, Rhodes University, Grahamstown: South Africa.
Antrobus, G. and Snowball, J. (2004), The National Arts Festival 2004: Festino Survey and Impact Study, Department of Economics, Rhodes University, Grahamstown: South Africa.
Baaijens, S. and Nijkamp, P. (2000), “Meta-Analytic Methods for Comparative and Exploratory Policy Research: An Application to the Assessment of Regional Tourist Multipliers”, Journal of Policy Modeling, Vol. 22, No, 7, pp. 821-858. 
Baker Associates (2007), Glastonbury Festivals 2007 Economic Impact Assessment, a Research commissioned by Mendip District District Council and Glastonbury Festivals Ltd., UK.   
Barroso, J., Gollop, C., Sandelowski, M., Meynell, J., Pearce, P. and Collins, L. (2003), “The challenge of searching for and retrieving qualitative studies”, Western Journal of Nursing Research, Vol. 25, No. 2, pp. 153-178. 
Bille, T. and Schulze, G. (2006), “Culture in Urban and Regional Development”, in Victor A. Ginsburgh and David Throsby (eds.) Handbook of the Economics of Art and Culture, North Holland, pp. 1052-1093.
Bragge, B. and Snowball, J. (2006), The National Arts Festival 2006: Economic Impact Study, Department of Economics, Rhodes University, Grahamstown: South Africa. 
Breen, H., Bull, A. and Walo, M. (2001), “A Comparison of Survey Methods to Estimate Visitor Expenditure at a Local Event”, Tourism Management, Vol. 22, No. 5, pp. 473-479.
Bryman, A. (2008), Social Research Methods, Third Edition, Oxford University Press, Oxford: UK pp. 85-95.
Crompton, J. (1995), “Economic impact analysis of sports facilities and events: Eleven sources of misapplications”, Journal of Sports Management, Vol. 9, No. 1, pp. 14-35.
Crompton, J. (2006), “Economic Impact studies: Instruments for Political Shenanigans?”, Journal of Travel Research, Vol. 45, No. 1, pp. 67-82.
Crompton, J., Lee, S. and Shuster T. (2001), “A Guide for Undertaking Economic Impact Studies: The Springfest Example”.  Journal of Travel Research, Vol. 40, No. 1, pp. 79-87.
Cultural Ministers Council (1997), “Measuring the Impact of Festivals: Guidelines for conducting an economic impact study”, Culture and Recreation Statistics, Australian Bureau of Statistics.
Daniels, J. (2007), An analysis of the Economic Impacts of Ithaca’s Light in Winter Festival”, a Thesis presented to the faculty of Graduate School of Cornell University, Ithaca: USA.
Daniels, M., Norman, W. and Henry M. (2004), “Estimating Income Effects of a Sport Tourism Event”, Annals of Tourism Research, Vol. 31, No. 1, pp. 180-199.
Frechtling, D. (1994), “Assessing the Impacts of Travel and Tourism – Measuring Economic Benefits”, In Travel, Tourism and Hospitality Research, Ed. Ritchie, B. and Goeldner, C., John Wiley and Sons: New York, USA, pp. 437-450.
Frechtling, D. (2006), “ An Assessment of Visitor Expenditure Methods and Models”, Journal of Travel Research, Vol. 45, No. 1, pp.26-15.
Getz, D. (1991), Festivals, Special Events and Tourism, Van Nostrand Reinhold: New York, USA.
Gilbert, N. (2001), Researching social life, Second Edition, Sage: London, UK, pp. 509-510.

Heaney, J. and Heaney, M. (2003), “Using economic impact analysis for arts management: An empirical application to a music institute in the USA”, International Journal of Nonprofit and Voluntary Sector Marketing, Vol. 8, No. 3, pp. 251-266.
Hendon, W. and Costa, F. (1988), Economics and The Akron Art Museum: an economic impact and benefit cost analysis, Center for Urban Studies, University of Akron, Ohio: USA.

Herrero, L., Sanz, J., Devesa, M., Bedate, A. and Barrio, M. (2006), “The Economic Impact of Cultural Events: A Case Study of Salamanca 2002”, European Urban and Regional Studies, Vol. 13, No.1, pp. 41-57.  

Janeczko, B., Mules, T. and Ritchie, B. (2002), “Estimating the economic impacts of festivals and events: a research guide”, Research report, CRC for Sustainable Tourism Pty Ltd.: Queensland: Australia.

Johnson, A. and Sack A. (1996), “Assessing the value of sports facilities: the importance of non-economic factors”, Economic Development Quarterly, Vol. 10, No. 4, pp. 369-382.

Johnson, P. and Thomas, R. (2001), The economic impact of the arts: a critique”, The UK Cultural Sector, Ed. Selwood, S., Policy Studies Institute, London, pp. 202-215.

Langen, F. and Garcia, B. (2009), “Measuring the Impacts of Large Scale Cultural Events: A Literature Review”, Impacts 08: The Liverpool Model, European Capital of Culture Research Programme.
Litvin, S. and Fetter, E. (2006), “Can a festival be too successful? A review of Spoleto, USA”, International Journal of Contemporary Hospitality Management, Vol. 18, No. 1, pp. 41-49.
London Development Agency (2003), The Economic Impact of the Notting Hill Carnival, a Research undertaken by Man Weaver Drew and De Montfort University, London: UK.
Long, P. and Owen, E. (2006), “The arts festival sector in Yorkshire; economic, social and cultural benefits, benchmarks and developments”, ACE, Dewsbury: UK.
Loomis, J. (2007), “Correcting for on-site visitor sampling bias when estimating the regional effects of tourism”, Tourism Economics, Vol. 12, No. 1, pp. 41-47.
Madden, C. (2001), “Using economic impact studies in arts and cultural advocacy: a cautionary note”, Media International Australia Incorporating Culture and Policy, Vol. 98, pp. 161-178.
McCormick, J., Rodney, P. and Varcoe, C. (2003), “Reinterpretations Across Studies: An approach to meta-analysis”, Qualitative Health Research, Vol. 13, No. 7, pp. 933-944.
Neuman, W. L. (2011), Social Research Methods: Qualitative and Quantitative Approaches, Seventh Edition, Pearson: Boston, USA, pp. 124-126.
Noblit, G. W., and Hare, R. D. (1988), “Meta-ethnography: Synthesizing qualitative studies”, Sage: California, USA.
Patton, M. (2002), “Qualitative research and evaluation methods”, 3rd Edition, Sage: California, USA, pp. 555-563.
Phynthian-Adams, S., Sapsford, D. and Southern, A. (2008), “Considering the Economic Impacts of 2008 European Capital of Culture”, Impacts 08: The Liverpool Model, European Capital of Culture Research Programme.
Quinn, B. (2005), “Arts Festivals and the City”, Urban Studies, Vol. 42, Nos 5/6, pp. 927-943. 
Reis, S., Hermoni, D., Van-Raalte, R., Dahan R. and Borkan, J. M. (2007), “Aggregation of qualitative studies - From theory to practice: Patient priorities and family medicine/general practice evaluations”, Patient Education and Counseling, Vol. 65, No. 2, pp. 214-222.
Richman, H. P. (1983), The adventure of Reason, Greenwood Press: Westport, USA, p. 136.
Russell, B. (2000), Social research methods: Quantitative and qualitative approaches, Sage Publications: California, USA, pp. 92-94.
Sandelowski, M., Docherty, S. and Emden, C. (1997), “Qualitative metasynthesis: Issues and techniques”, Research in Nursing and Health. Vol. 20, No. 4, pp. 365-371.
Schwer, R. K. and Rick D. S. (1995), “A comparison of the multipliers of IMPLAN, REMI, and RIMS II: Benchmarking ready-made models for comparison”, The Annals of Regional Science, Vol. 29, No. 4, pp. 363-374.
Seaman, B. (1987), “Arts Impact Studies: A Fashionable Excess”, in Economic Impacts of the Arts: A Sourcebook, National Conference of the State Legislatures, Denver: USA, pp. 43-75.
Seaman, B. (2003), The Economic Impact of the Arts, in Handbook of Cultural economics, Ed. Towse, R., Edward Elgar, Cheltenham: UK, pp. 224-231.
Seaman, B. (2004), “ The Supply Constraint Problem in Economic Impact Analysis: An Arts/Sports Disparity”, Prepared for presentation at Conference, University of Chicago, USA.
Shaw, W.D., and Rogers, J. (2005), “Review of Non-market Value Estimation for Festivals and Events: A Discussion Paper”, Draft report submitted to the Ontario Tourism Board, Research Resolutions Ltd.: Toronto, Canada.

Snowball, J. (2008), Measuring the Value of Culture, Springer–Verlag: Berlin Heidelberg, Germany.
Snowball, J. and Antrobus, G. (2002), “Valuing the arts: Pitfalls in economic impact studies of arts festivals”, South African Journal of Economics, Vol. 70, No. 8, pp. 1297-1319.
Southern, R., Meethan, K, Fisher, A. and Butts, S. (2007), Economic Impact Assessment of the Summer Festival in Plymouth, Final Report, Research undertaken by the Socio-economic Research and Intelligence Observatory, Plymouth: UK.
Spoleto Festival USA (2005), The Economic Impact of the 2005 Spoleto Festival USA on the Charleston economy, a Report prepared by the Division of Research, Moore School of Business, South Carolina: USA.
Stall-Meadows, C. and Hyle, M. (2010), “Procedural methodology for a grounded meta-analysis of qualitative case studies”, International Journal of Consumer Studies, Vol. 34, No. 4, pp. 412-418.
SQW Limited and TNS Travel and Tourism (2005), Edinburgh’s Year Round Festivals 2004-2005, Economic Impact Study, pp. 1-90. 
Throsby, D. (2001), Economics and Culture, Cambridge University Press, Cambridge UK.
Tyrrell, T. and Johnston, R. (2001), “A framework for assessing direct economic impacts of tourist events: Distinguishing origins, destinations and causes of expenditure”, Journal of Travel research, Vol. 40, No. 1, pp. 94-100.
Vaughan, R. (1977), The Economic Impact of the Edinburgh Festival 1976, Summary Report.
Vrettos, A. (2006), The Economic Value of Arts & Culture festivals/ A Comparison of four European Economic Impact Studies, a MA Thesis, University of Maastricht, Maastricht: The Netherlands.
Walsh, D. and Downe, S. (2005), “Meta-synthesis method for qualitative research: a literature review”, Journal of Advanced Nursing, Vol. 50, No. 2, pp. 204-211. 
Wilton, J. and Nickerson, N. (2006), “Collecting and Using Visitor Spending Data, Journal of Travel Research, Vol. 45, No. 1, pp. 17-25.
Yuan, M. and Yuan, S. (1996), “Sixteen versus Nine Expenditure Categories in Tourism Surveys: Is there a Difference”, Journal of Travel Research, Vol. 34, No. 4, pp. 59-62.
Zhao, S. (1991), “Metatheory, Metamethod, Meta-data-analysis: What, Why, and How?”, Sociological Perspectives, Vol. 34, No. 3, pp. 377-390. 
Zhou, Z. (2000), “The Impact of Memory on Expenditures Recall in Tourism Conversion Studies”, Journal of Travel Research, Vol. 38, No. 3, pp. 303-307.

APPENDIX 
List 1: Sources of Data
	Sources of Economic Impact Studies 	
1	Online and Paper Archive Databases of Libraries 	
	i. UB Erasmus University Rotterdam	
	ii. Boekman Foundation, Amsterdam	
	iii. UBA University of Amsterdam	
	iv. European Festivals Association database 
(http://www.efa-aef.eu/en/association/home/archives/)	
	v. Impact Database - Centre of Cultural Policy Research of University of Glasgow 
(http://www.impact.arts.gla.ac.uk/) 	
2	Internet Search Engines	
	i. Google 
(http://www.google.com/)	
	ii. Google Scholar 
(http://scholar.google.com/)	
	iii. Yahoo
 (http://www.yahoo.com/)	
3	Personal enquiries to consultants and academic actors via the Internet	

List 2: List of all Economic Impact Studies initially scanned (by chronological order):
•	“The Economic Impact of the Edinburgh Festival 1976” (October 1977), prepared by R. Vaughan, Department of Economics, Heriot-Watt University, UK.
•	“Economic Impacts of Arts and Cultural Institutions: A Model for Assessment and a Case Study in Baltimore” (November 1977), prepared by D. Cwi and K. Lyall, John Hopkins University, Baltimore, USA.
•	“The arts as an industry: their economic importance to the New York – New Jersey Metropolitan Region” (1984), prepared by R. Scanlon and R. Longley, University of Akron, USA. 
•	“Economics and the Akron Art Museum: an Economic Impact and Benefit Cost Analysis” (1988), prepared by W. S. Hendon and F. J. Costa, University of Akron, USA.
•	“The Economic and Social Contribution of the Wexford Opera Festival” (1989), prepared by J. O’Hagan, A. Barret and M. Purdy, Trinity College Dublin.
•	 “Monitoring Glasgow 1990” (December 1991), prepared by J. Myerscough, System three Scotland and Ecotec Reseach & Consulting, UK.
•	“Economic Impact of the Arts: Theatre Festivals in small Ontario communities” (1993), prepared by C. J. A. Mitchell, Canada Social Sciences and Humanity Research, Waterloo University, Southern Ontario, Canada.
•	“A report into the impact of folk festivals on cultural tourism” (January 2003), prepared by M. H. McIntyre and the Association of Festival Organizers, UK.
•	“Economic impact study of UK theatre” (April 2004), prepared by D. Shellard, University of Sheffield, UK.
•	“Brighton Festival 2004: Everyone benefits – the Economic and Cultural Impact of Brighton Festival upon Brighton and Hove” (September 2004), prepared by Sussex Arts Marketing, UK.
•	“The Chichester Festival Theatre: Economic Impact Study 2003/04” (October 2004), prepared by D. Clark, J. Grainger and S. Jaffry, Centre For Local and Regional Economic Analysis, University of Portsmouth, UK.
•	“The National Arts Festival 2004: Festino Survey and Impact Study” (November 2004), prepared by G. G. Antrobus and J. D. Snowball, Department of Economics, Rhodes University: Grahamstown, South Africa.
•	“The Economic and Social Impact of Cultural Festivals in the East Midlands of England” (December 2004), prepared by C. Maughan and F. Bianchini, De Montfort University, Leicester UK.
•	“Edinburgh’s Year Round Festivals 2004-2005 Economic Impact Study” (September 2005), prepared by SQW Limited and TNS Travel & Tourism, UK.
•	“The Economic Impact of the 2005 Spoleto Festival USA in the Charleston Economy” (December 2005), prepared by the Division of Research Moore School of Business, University of South Carolina, USA.
•	“The Economic Impact of Cultural Events: A Case-Study of Salamanca 2002” (January 2006), prepared by L. C. Herrero, J. A. Sanz, M. Devesa, A. Bedate and M. J. del Barrio, University of Valladolid, Spain. 
•	“The arts festival sector in Yorkshire: economic, social and cultural benefits, benchmarks and development” (February 2006), prepared by P. Long and E. Owen, Sheffield Hallam University, UK.
•	“The National Arts Festival 2006: Economic Impact Study” (October 2006), prepared by B. R. Bragge and J. D. Snowball, Department of Economics, Rhodes University: Grahamstown, South Africa.
•	“An analysis of the Economic Impacts of Ithaca’s Light In Winter Festival” (August 2007), prepared by J. C. Daniels, Cornell University, USA.
•	“Economic Impact Assessment of the Summer Festival in Plymouth” (September 2007), prepared by R. Southern, K. Meethan, A. Fisher and S. Butts, Socio-economic Research Intelligence Observatory, University of Plymouth, UK.
•	“Glastonbury Festivals 2007: Economic Impact Assessment” (2007), prepared by Baker Associates, Mendip District Council, UK.
•	“Festivals Mean Business: A Survey of Arts Festivals in the UK” (March 2008), prepared by SAM and the University of Brighton, UK.
•	“Culture & Prosperity: The Economic Role of Culture in the west Midlands” (March 2009), prepared by the West Midlands Cultural Observatory in association with the West Midlands Cultural Research & Intelligence Group, UK.
•	“Sundance Film Festival economic Impact” (June 2009), prepared by B. Addicott and A. McGee, University of Utah’s Bureau of Economic and Business Research, USA. 
•	“The Economic Impact of New Jersey’s Nonprofit Arts and Associated Audience Spending on the State’s Economy – 2009” (June 2009), prepared by C. Lanier and E. Spruck, Association for Cultural Economics, University of Akron, USA.
•	“The Economic Impact of the Nonprofit Arts on Orange County” (Fall 2010), prepared by A. G. Anderson, Centre for Economic Research, Chapman University and The Orange County, Business Committee for the Arts Inc., USA.
•	“Creating an impact: Liverpool’s experience as European Capital of Culture” (2010), prepared by B. Garcia, R. Melville and T. Cox, University of Liverpool, UK.
•	“2010 South By Southwest: Economic Impact Analysis” (2010), prepared by Greyhill Advisors, Texas, USA.
•	“Economic Value of Arts Activity in the district of Lancaster – Economic Impact study” (January 2011), prepared by Genecon LLP, Information by Design and Helen Corkery Research & Marketing, Lancaster, UK.
•	“Edinburgh Festivals Impact Study Final Report” (May 2011), prepared by U. Chouguley, R. Naylor and C. Rosemberg Montes, BOP Consulting, UK.