Culture Metrics @ Museums and the Web 2016

We've just come back from the Museums and the Web 2016 conference (Los Angeles, CA), where we presented some of the initial results of our research. You can read the full paper here: Data Culture and Organisational Practice. We received great feedback from conference participants and had the opportunity to discuss the role, value and use of audience data in policy and practice in museums, libraries and archives. It was also useful to share views on the extent in which current and emerging systems and practices of data aggregation and analysis relate to the rhetoric and/or reality of the so-called data-driven decision making.

Our research so far has shown that the discussion about cultural performance and quality measurement is less about audit and reporting and more about cultural and creative practice. Cultural partners are interested in understanding what the generation and use of this audience data can do for their daily practice, rather than looking for the “easy wins” of collating evaluation data for their annual reports. This suggests that before data-driven decision making becomes an option that creates real change, the generation and analysis of data needs to become part of an organisation’s culture, with the issues and barriers that this brings with it. So, at this stage, our assessment is that if the discussion about data is a discussion about cultural practice and policy, then this might suggest that a data culture is being formed in a cultural organisation.

Next step for us is continuing analysing our research data and planning some additional research that would follow the adoption and use of this system by cultural organisations and examining how the metrics system impacts on their organisational practice and vice versa. 

Policy Week event: Using Digital Technology to Assess Quality in the Arts

Policy week event

This event followed on from research as part of the NESTA R&D Culture Metrics project which involved researchers from the Institute for Cultural Practices, The University of Manchester, in collaboration with a partnership of arts and cultural organisations and a technology partner, Culture Counts. The research considers how digital technologies, social media and big data might help arts organisations benchmark and demonstrate the quality of their work, and assess and evaluate its impact on audiences and for funders.

Arts and cultural policy increasingly requires arts organisations to demonstrate the public value and outcomes of publicly funded work, through evidence and evaluation. At the same time arts organisations want to be able to demonstrate the quality of this work in ways they understand and control. To this end, the Culture Metrics project has attempted to develop measures for the quality of arts and cultural experiences, which have been co-produced and tested by the sector, and bring together data from different stakeholders in the process, using digital technologies. As the research partner, we have been looking at some of the claims of this project, along with the motivations and challenges for cultural partners and policy stakeholders when adopting the metrics system.

This Policy Week event aimed to further explore the challenges for arts organisations and policy makers in this area of interest – and to consider more broadly the ways in which the arts use digital technology, social media and big data to demonstrate their public value.

Presentation of the Culture Metrics research

In the project, we explored how organisations already use social media data with increasing sophistication to generate and continue conversations with their audiences and to promote their brand values and missions, as well as arts experiences and events. We found that organisations regularly bring together and triangulate data to understand their audiences and inform programming and producing decisions. We found less evidence of a consistent and structured use of social media data as a source for measuring quality or as an integrated component of their performance management regimes.  Technically, however, the data arts organisations use is unlikely to be ‘big’ in the ways understood by the growing literature that enthuses about the potential value of big data and its distinctiveness from other more routine data sources, such as box office and occasional audience questionnaire surveys. 

We also found that Culture Metrics “ticks a lot of the boxes” found by recent research on performance measurement and quality in arts organisations - of so-called ‘artistic vibrancy’ (the Australian term for healthy cultural organisations). For example, a literature review by Bailey and Richardson (2010) found a number of models with had the following recommendations for performance measurement in common:

  • include external and internal views of the organization’s performance (e.g., audience, funders, artists, peers, staff);
  • ensure organizational ownership of the measurement process for it to be meaningful;
  • engage employees and management in the measurement process;
  • match measures to the organization’s mission.

(Bailey and Richardson 2010: 294)

The event heard an introduction to the NESTA R&D project from Abigail Gilmore and John Knell, with some insights into the process for the research and some early findings. This preceded the live testing of the Culture Counts evaluation system by delegates representing both ‘public’ and ‘peers’ in an evaluation of the Matthew Darbyshire exhibition

The results of this test event, combined with data already collected in the Gallery by University of Manchester Arts Management students the week before, were then displayed and discussed.

Questions this exercise raised included: What are the differences between verbal and text responses and ‘hard’ quantitative metrics? How should we understand them in terms of the implications for what the data can be used for? What provisions are made for responses and effects of arts and culture which don’t happen in the short window between experiencing events and exhibitions and undertaking the survey, and that can be delayed by some time? These ‘longitudinal’ impacts are discussed within the methodologies reviewed by the Arts Council England’s recent literature review on understanding the value of cultural experiences as one of the critical drawbacks in post-event surveys. However, John Knell explained how Culture Counts can be used retrospectively and, as other audience members contributed, the discussion considered how the methodologies for understanding audience impact should not only rely on post-event surveying, but also include other mixed methods that allow for conversations with audiences and publics.

Another question raised was about the relation between the use of social media data as object of further interpretation and the taxonomy of self, peer, public? Is this the ‘right’ taxonomy? Interestingly there was little discussion about the sampling strategies or data collection techniques used by Culture Counts, although there were some concerns about the representativeness of respondents, particularly in relation to ‘peers’ of arts organisations. As the project testing also found, recruiting suitable peers to act as critical friends for particular bits of programming takes effort and care is needed to avoid accusations of bias selection. 

In their presentation Kostas Arvanitis and Chiara Zuanni introduced follow-on research from the Culture Metrics project, which explores and combines social media data with analysis of audience experiences. Focusing on current methodologies for the collection and analysis of social media data, they discussed the relation between this data and the data collected by the Culture Counts system, highlighting the organisational challenges of a data-rich cultural professional practice. They highlighted broader issues including the impact that the rhetoric of data, especially big data, has on producing preconceptions of validity and value, and considered the gaps in the data and how these gaps are accounted for in organizational practice. Overall, Arvanitis and Zuanni raised questions about the data cultures that are being formed in cultural organisations, and about what data-driven decision-making might actually mean, how it manifests itself in organisational life and how the collection and analysis of social media data might fit into organisational data culture and practice. In addressing what we need to take into consideration in planning, carrying out, and evaluating social media metrics they talked about:

  • Understanding the context and motivation of audiences’ social media activity
  • Value and usefulness of unprompted/unstructured reactions (as opposed to structured surveys)
  • Accuracy of data
  • Representativeness of audiences
  • Different platforms, different users, different uses?
  • Methodological and ethical issues on capturing and using social media data

Panel discussion

This was followed by the panel discussion, which focused on the following questions:

  • How can arts organisations use social media, digital technologies and big data more strategically?
  • What implications for cultural policy derive from the use of this data?
  • What one improvement would help?
Roundtable: from left, Nick Merriman, Alison Clark, chair Abi Gilmore, Cimeon Ellerton, Hasan Bakshi.

Roundtable: from left, Nick Merriman, Alison Clark, chair Abi Gilmore, Cimeon Ellerton, Hasan Bakshi.

Hasan Bakshi, NESTA , discussed the questions in relation to his experience as an economist attempting to develop robust measures for innovation, value creation and, recently, cultural or ‘intrinsic value’ through methods such as contingent valuation. With the proliferation of big data, from diverse sources, his concerns include the question of data standards and how we might understand these in relation to bigger data. One recommendation for the arts might be to involve more data scientists, who arts organisations can work with to ensure the quality of the data and its analysis. He gave an example from recent research of ‘machine learning’ through exploring massive datasets, which have helped to identify video games as a growing form of cultural participation – in contrast to other forms of participation research that miss changes in behaviour, particularly associated with new or emergent cultural forms. One particular method which needs updating is the survey model: instead, Hasan argues, big data use cases can reveal new patterns of behaviour and experience, and even redefine what we understand as culture.

Cimeon Ellerton, The Audience Agency, talked of his experience in developing audience data through services such as Audience Finder. For him, the priorities remain the standardisation of data and also the need to aggregate data whilst encouraging arts organisations to be open. This needs to be a collective effort, involving all organisations, so that the smaller arts companies with less capacity for audience evaluation and research could benefit from scalable economies. Leadership by the sector in using data and evidence is also key: “whoever owns the story gets the funding”.

Alison Clark, Arts Council England North, said she feels arts organisations continue to have a fear of social media and of sharing. There isa sense that the more organisations value data, the less they are likely to share data. Arts Council England has embraced these issues, by developing a new data strategy and encouraging the use of data scientists to bring new skills and practices to enrich big data analysis and evaluation. Although this may bring more opportunities for ‘data-driven decision making’, at the same time it brings other anxieties. For example, evidence-based funding schemes, such as Creative People and Places, which is only for places with identified low levels of engagement established by Active People survey data, mean places and programmes, which other types of knowledge and analysis would reward, miss out. There is a concern that policy for arts and culture becomes somehow un-human – that there is an automation of creativity, and policy formation can be algorithmic. A solution to this lies to some extent in a collective embrace of big data, where everyone joins in: for Alison, this means leadership within the sector to create peer pressure and encourage sharing of resources and practice.

Nick Merriman, Manchester Museum,  speaking from the ‘user’ perspective, discussed how initiatives like Culture Counts can provide better accountability for public funding, since by aggregating stakeholder perspectives they provide a credible means of understanding quality, not just of arts and cultural experiences, but also other activities which involve exchange between publicly funded services and the public, for example, science engagement. He spoke of the added value of the process of developing shared metrics through collaboration, although he also cautioned it is early days still for the embedding of this approach in the sector, particularly in terms of bringing together the data from Culture Counts with social media data.

Discussion focused on the barriers to data sharing: unlike the commercial sector, for the publicly funded arts there is more opportunity – and need – for policy to intervene in this space and to encourage organisations to work collaboratively to increase their analytical capacity. The discussion suggested that concerns for technologically-determined funding decisions are balanced by the opportunities to create better transparency and accountability. This, it was felt, is facilitated viaa richer conversation based on a range of knowledge which includes the generative potential of big data rather than relying on single source or set of measures; to achieve thisfurther scientific endeavour is needed to develop the data standards, metrics and methodologies to measure and demonstrate public value. 

The arts may feel they are playing catch-up with other sectors in this agenda, and whilst one wonders if the rhetoric of evidence-based policy is simply replaced by the rhetoric of big data within data-driven decision-making, with the added ‘wow factor’ of algorithmic policy-making, there is clearly an appetite to work together to understand and harness the potential of big data, to adapt and use its social properties, especially when led by the sector rather than imposed top down by policy.

 

References

Bailey, J and Richardson, L (2010) Meaningful measurement: a literature review and Australian and British case studies of arts organizations conducting “artistic self-assessment” in Cultural Trends Vol. 19, No. 4, December 2010, 291–306.

Interviewing our cultural partners

In June and July 2015, our Research Associate Franzi Florack interviewed a range of cultural partners who are involved in the development of Culture Metrics. She spoke to representatives of the Manchester Jazz Festival, Octagon Theatre Bolton, Manchester Museum, Hallé and the Royal Opera House. 

The interviews centred around the organisations' collection and use of data and their application of the Culture Metrics program. They give us a unique insight into cultural organisations' interact with data and serve as a useful case study as the project enters its final phase.

To find out more, please click on the report picture to the left. 

The Culture Metrics workshop 2

The second workshop of the Culture Metrics program was held on Monday the 16th March 2015 at the Cornerhouse in Manchester. Speakers included Julian and Sarah Hartley and Chiara Zuanni, who outlined different uses of data in the cultural sector. This workshop was entitled ‘Bigger Data- Better Data?’ and more than twenty academics, cultural partners and arts representatives from all over the UK had travelled to Manchester to attend.

IMG_20150316_133921.jpg

Initially, the workshop was supposed to revolve around Big Data (as discussed in Sarah’s blog post below), but the research team had found during the first workshop that most cultural organisations interacted with ‘just enough’ rather than ‘big’ data. In response to this discovery, we agreed with Anthony Lilley that 'whether the data is technically defined as “big” is of comparatively little importance in some ways.  It is the use of data-driven approaches to drive insight and change behaviour which matters.’

Here are the questions we focused on throughout the afternoon:

* What kinds of data collection does your organisation undertake?
* How is this data discussed/ analysed/ stored/ visualised?
* What forms of data presentation/visualisation do you/your organisation use/value most?
* (How) does the data affect the management of your organisation?
* Do you collect data via facebook or twitter or other social media?
* What data channels do you think your audiences/stakeholders use most to express their opinions?
* What kinds of data do funders and policy makers value most and do these include ‘big data’?

Several interesting observations were made during the afternoon. Many workshop attendees felt that their organisations were obliged to gather audience data in order to advocate or ‘justify’ their practice, but there was little understanding of what happened to the data once it was submitted to the Arts Council or similar organisations. The gathering of data, they felt, should be used to create a ‘response to current world’ rather than remain a purely artificial exercise.

Although many organisations outlined a range of ways in which they collected data, only few knew how this data was then used within their organisations and whether it had an impact on the overall culture of their institution. It seemed that attendees mostly had an understanding of how to gather data, but that the actual analysis and visualisation was still a difficult endeavour. A skills gap in this area was particularly lamented.

Especially smaller organisations were reluctant to employ external help to increase their data gathering and analysis capacity. It was unclear to them how an extra investment might lead to a bigger or more diverse audience. This reluctance to ‘up-skill’ due to its financial implication was felt across the room (though not everyone agreed) and the group discussed whether audience data rather than creative, artist-led practice actually should drive artistic curation.

Some organisations were actively in touch with their audiences via social media and tried to gather feedback online. Whether this type of response was actually objective or useful remained difficult to judge for most. It was mentioned that some visitors might for example tweet something positive, so that the cultural organisation would re-post their feedback. Overall, many organisations felt overwhelmed by the amount of data which was available to them and/or which could be gathered. Data itself was perceived as meaningless- it is the analysis which counts.

A part of the discussion revolved around the notion of ‘open’ data which could be accessed by all via the internet. Although cultural organisations supported the democratisation of data in principle, they were worried in which ways their data would be (ab)used by others, suspecting that the narrative and culture or their organisation might not be visible via sterile numbers. Although many organisations welcomed to idea of benchmarking their success with others, the thought emerged that this could be done via the sharing of analysis rather than the raw data itself.

Overall, both the attendees and the research team enjoyed the lively discussion in a very stimulating environment. Thank you again to all those who made the afternoon possible. Although this was the last ‘Culture Metrics’ discussion group, further activity on the website will follow, so please check back soon.  

Getting in a spin with some cultural data

By Julian and Sarah Hartley of Dim Sum Digital

Remember the story of the amazing spinning statuette from Manchester Museum? If your memory needs a jog, this is our favourite piece of coverage on the matter from 2013 to set this blogpost in context.

he story became a viral phenomena in the way that only curiously quirky summer stories can and at the last meeting of the cultural metrics project group, we heard more about the data which researcher Chiara Zuanni has collated on the topic:

  • 14927 tweets.

  • 622 media articles, blogs and forums.

  • around 70 youTube videos.

  • 2868 of comments below online articles.

  • observations and interviews in the Ancients Worlds Galleries.

It was a fascinating insight into the data Chiara had collected and analysed as part of her phd.

But was it big data?

During the session at Cornerhouse, the participants discussed the big data question without too much attempt to define what we mean by the term. In our last blog post here, we hoped to provoke some of that debate further and introduced ideas around the ‘open data’ movement into the equation too.

This was intentional because it’s hard to consider the two things in isolation with them being so intrinsically linked when it comes to practical work in this field.

There’s always a danger of falling into a purely theoretical process to work on a big data definition so, for the purposes of this blog post, we are taking ‘big data’ to mean anything taken from varied and multiple data sources. The classic definition is that big data is a broad term for data sets so large or complex that traditional data processing applications are inadequate. In other words, it’s data that’s too big for an excel spreadsheet!

Using that definition, the spinning statuette data on its own, wouldn’t necessarily fall into a ‘big data’ area BUT there were many other organisations and individuals in the room that day with access to additional data which could expand that and, if it was incorporated into data from data sets outside of the immediate cultural sector then - there’s some very big data potential indeed.

The workshop discussions (which will be summed up more fully in a blog post to come this week) considered the data channels and sources held by the different cultural organisations. These revealed that ‘boring’ data, ie. everyday stuff such as attendance levels, can be as useful as the rarer forms of analysis available through organic or social media data, such as the data associated with the spinning statue.

We’d suggest there’s an interesting space to be developed here where the cultural institutional data and the many bigger data sets out there in the world could be interrogated together.

What difference would it make to your institution if your internal data could be overlaid with shopping habits from your locality? With transport trends for your area? With census data? With benefits data? etc. etc.

How would your organisation cope with that? How would it address the supply and demand elements of the findings? Would it change the very structure of your organisation and its activities?

And finally…….

As we leave this project, hopefully with some provocative questions to be considered, we’d like to share an amusing big data set which would be easy to replicate with a whole variety of topics and locations - the autocomplete from Google searches.

It was created by Twitter user @TechnicallyRon and shows a map of England according to the results you get when you type into Google '[Name of county] is’. Does this data set give us a unique perspective on how people feel about the place they live in?


* If you’d like to hear more about data in the cultural sector please do stay in touch via the directors’ blog or on Twitter @dimsumdigital.

Big, open and demanding: The data challenge for cultural organisations

By Julian and Sarah Hartley, Dim Sum Digital Ltd

Kicking off data conversations in cultural organisations can be framed in terms of the demand and the supply of data. So as a first point of discussion, it’s worth thinking about data in the following terms:

  1. Who, if anyone, was calling for the data to be released before it was open? (Demand-side)

  2. How, specifically, it was released – including stakeholders involved, those     supportive and against, specific leverage points, etc. (Supply-side)

Both are valid points of discussion which we will touch on here. Looking at those two aspects will typically revolve around questions of whether data is ‘open’, whether it is ‘big data’, what benefits can be gained, how to go about gathering it ? etc.etc.

As we dig deeper to consider Monday’s research proposition this blogpost seeks to put forward a couple of additional thought provocations to the above framework:

  1. Is there ‘accidental’ data which cultural organisations could use in order to better understand, and therefore serve, their audiences?

  2. What are the implications of big data for cultural funding organisations?

Accidental data gathering?

In all our interactions online, data trails follow our activities. Whether that’s the location our tweets are sent from or the pathway a website link travels as it’s shared across Facebook, these data trails are powerful and visible leads for a cultural detective.

It means the real-time data we accidently give out as we shop online can be queried about the tastes and habits of those communities’ neighbouring arts and cultural organisations.

As Juan Mateos-Garcia posted in The art of analytics: using bigger data to create value in the arts and cultural sector (February 2014):

“The web is like a vast mirror that reflects our actions, and it can provide insights into our behaviour, and even our desires.”

This data can guide cultural organisations to consider who, why, when and for what purposes local communities might be interested in becoming audiences. Therefore this data has potential to be harnessed to help organisations find ways to broaden their audiences through new initiatives and programmes.

Yet demand for social data from the cultural sector would require data literacy on their part and it is interesting to consider where those skills would sit within the existing organisational structures. Another aspect to this ‘social data’ is what gaps in their cultural metrics will be revealed when it is queried for cultural insights into the communities the arts organisation serve.

Might funders begin to use this data to evaluate organisations independently of any cultural metrics offered to them?

How are cultural funders considering big data?

The organisations which fund cultural activities produce data with every grant, award, activity review, funding round and evaluation. Much of that data is now being made publicly available for the first time - sometime not necessarily with the participation of the funder itself.

We spoke to one of those behind the open data initiative, 360 Giving, William Perrin. As well as being a leading adviser on technology (member of the Digital Government Review 2015 and a former technology adviser to Tony Blair), Perrin co-founded the transparency tool to inform different funding bodies about each others’ activities. The database now includes information from the Heritage Lottery Fund, Arts Council England and Big Lottery.

In this short video interview (recorded over Skype) he talks about reasons for starting 360 Giving, and the way data being included at the service impacts on cultural organisations.

The pressure to improve the information being shared about those cultural organisations who receive funding through public bodies is increasing all the time and coming from individual campaigners as well as organisations.

Chris Unitt runs a website on digital metrics and has recently taken the Arts Council to task about the way it collects data.

In a post titled, Improving the digital metrics Arts Council England collects from funded organisations, he outlines the need to re-think the questions being posed to fundees: “Arts Council England asks their National Portfolio Organisations to submit an annual report with all sort of information and figures. As part of that they’re required to provide some digital metrics.

Here’s what they ask for (photo via @SamScottWood)”:

Unitt concludes that the questions themselves are a big part of the problem in this exercise as they can’t result in useful and valuable data to share with the sector..

Whether you consider intervention like Unitt’s to be pesky interference or a helpful starting point for a more in-depth conversation might reflect your own organisation’s attitudes towards the value of data openness, visibility and sharing. But there’s no doubting that the demand end of the equation here, coming as it does from an engaged member of the cultural community, reflects a move to catalyse change in the sector.

Going back to the supply end of things to wrap up this provocation, the organisational benefit of producing more, better and bigger data is something researchers have been attempting to measure too.

Nesta ‘s report (March 2014) Inside the Datavores, claimed to quantify, for the first time, ‘the link between higher levels of online data use in UK businesses, and their economic performance in terms of productivity and profitability’.

It concludes: “Our analysis makes it clear that, despite the hype, managers ignore the potential of data at their peril. At the same time, building up their IT infrastructure to collect and process more data on its own will do little. The data needs to be probed and analysed, and their workforce needs to be empowered to act on what is learned.”

As Perrin also pointed out: “People often in the public sector - despite now seven years of the open data movement and a remarkable turnaround in this attitude to open publication - people intrinsically don’t think about publishing things by default and then they also wonder ‘who on earth might want it’.

“The only way to find out what this stuff can do is in fact to see what people do with it. It’s a discovery-led process, an innovation-led process and should no longer be a bureaucratic process. The bureaucrats need to put aside their traditional reserve and secrecy and say ‘there is a greater public good here, we don’t know what it is, but we won’t find out until we put the data out there for people to manipulate’.”

 

***

Further data related links:

http://www.leedsdatamill.org - owned and managed by Leeds City Council as a Civic Enterprise in partnership with Leeds-based digital content, data insight and storytelling specialists, Hebe Works.

http://datagm.org.uk - created by public sector organisations in Greater Manchester, to release and bring together in one place as much data as possible.

http://www.beehivegiving.org - Beehive is our attempt to make the work of small non-profits visible to those most likely to support them.

https://blog.scraperwiki.com/2013/12/big-lottery-data - Big Lottery Data explored.

A summary of the first part of the literature review

So far, a literature review of 11,000 words (26 pages), has been produced which focuses on the cultural sector’s metrics for success and value. A range of definitions has been proposed for ‘cultural value’ which also include economic and social value. The public, artists and organisations have all described the meaning of ‘cultural value’ in different ways. The public, for example, has indicated that the primary value of culture is quality of the artistic experience, which could ‘be judged by understanding the emotional response of the audience to a piece of work and the impact it has in terms of challenging perceptions and broadening horizons’ (Bunting, 2007, p. 16). The Arts Council, in contrast, has only offered a limited view of how it understands value.

Previously, the introduction of metrics for value in the arts has had predominantly economical and political reasons. As funders are trying to save money and arts budgets are cut, it becomes ever more important to create transparent guidelines which judge the success of cultural projects and organisations. This ‘need for justification’ and its resulting cross-cultural quantitative research has been accused of promotion dualism and a market-driven economy- catching the arts organisations between a rock and a hard place.

Despite a lack of rigorous methodology and framework, researchers and practitioners have used a range of tools to measure the impact of arts such as biometric research, post-event surveying, qualitative post-event research, and longitudinal or retrospective studies. One surprisingly large field has reviewed the use of arts projects in prison, primarily relying on observations and interviews. Currently there are several research projects which are trying to understand and create cross-cultural metrics, but none have been accepted by the cultural community (yet). Additionally, several longitudinal studies are measuring long term impact and extended value.

There is little documented co-production of metrics in the cultural sector and the Arts Council itself has lamented that there are ‘too few examples of collaboration across backgrounds, organisations, disciplines and perspectives’ (Arts Council England, 2013, p. 25). It could be argued that more collaboration between practitioners and funders is needed in order to produce coherent funding guidelines. Collaboration between practitioners are more common, but have been documented in a similarly limited fashion. Apart from the Digital R & D fund , few organisations explicitly encourage collaboration with a particular emphasis on co-created success criteria. As a third category, collaboration between the practitioners and the public is more common. However, the majority of this interaction is often restricted to the organisations asking the audience for feedback, rather than involving visitors in the creation of the success metrics themselves.

The following part of the literature review will try to understand how cultural organisations engage with their audience through social media and what kind of impact this feedback (and information gathering process) has on the culture of the organisations themselves. 

The Culture Metrics workshop 1

A big thank you to all the brilliant people who joined us for our first workshop this week! We had a very stimulating afternoon and it was great to meet you all.

The workshop consisted of four parts: First, Franzi gave an introduction to the literature review, followed by breakout discussions and a group feedback session. Finally, Sarah and Julian provided us with a very exciting preview of the next workshop, which will take place on the 16th March. Sarah has posted a comment on her impressions of the day below. If you are interested in the lit review, please feel free to download the sheet with our sources and the slides of the presentation.

Over the course of the afternoon, we discussed the following questions:
•Does your organisation use different kinds of data in order to judge ‘quality’? What are the advantages or disadvantages of working with different sets of data?
•Does the public like to give your organisation feedback? Are there ways in which you get more feedback than in others (eg online survey vs paper survey, etc)?
•Have you changed your data gathering approach in the past 12 months at all? Do you think this might happen in the near future? Why?

It became clear very quickly that due to the great diversity of our workshop attendees their personal experiences in the above areas differed considerably, contributing to a wealth of rich discussions. However, it also surprised some of the participants to find similar practices in culture forms which they had felt were very different to their own. Questionnaires seemed to be the most common way in which organisations gathered audience feedback, but there was some discussion about the nature of the included questions.

Popular questions, to which representatives of a wide range of culture were able to relate, included: 'What does the audience experience feel like?' and 'Have we achieved what we wanted to achieve?'. Spontaneous audience response (such as clapping and body language) were also mentioned repeatedly. Many attendees mentioned social media as a good medium to engage with the audience and several people hoped to increase their organisation's understanding and use of twitter in particular. There seemed to be a sense that 'voluntary' feedback, which the audience produced in their own time, might provide the most reliable feedback. 

Overall, the group discussion was lively and critical and many attendees felt that they had encountered ideas and practices which might benefit their own organisation.

If you are interested in joining us for our second workshop, which will take place on the 16th March in Manchester, please don't hesitate to get in touch via franziska.florack@manchester.ac.uk. Our next topic is going to be the gathering and use of data in cultural organisations.

Blog post

This week, our guest blogger Sarah from Dim Sum Digital gives us an introduction to Big Data.

'Gathering a group of arts organisations together to talk about '*measuring cultural value*' was never going to be an easy task. As researcher Franzi Florack pointed out in her opening remarks, every word in that phrase can be contested.

In the first of the two workshops looking at the sort of measures and metrics which could be useful when concerning cultural value (however that's eventually defined!) participants were faced with a series of questions seeking to assess areas including, but by no means limited to, economic, cultural and social impact.

This blog post contains some notes the day from myself and Julian as we start to focus on the issues. We both attended to help formulate the provocations for the next workshop which looks more at data aspects and would appreciate any input you might have to the debate.

Some of the issues raised yesterday:

  • is a framework to assess cultural value even necessary/relevant/desirable?

  • when co-producing metrics, (how) could participatory events be used for the activity?

  • how can evaluation be longitudinal enough to include community?

Working in groups, participants considered their own organisation's methods of data collection and evaluation. These included feedback surveys left in venues, interviews with visitors, random telephone cold call research interviews, social media monitoring and collation of newspaper reviews.

Some interesting points emerged including:

  • was collection and evaluation steered by financial imperatives?

  • notable that traditional marketing segmentation still seems widespread use across organisations.

  • changing role of front of house staff mentioned as venue 'hosts'.

  • the friction between rewarding loyal engaged audiences and developing new ones through outreach to non-audiences and non-visitors.

  • discussion about the extent to which data collection was driven with funders in mind.

The two of us were asked to finish the session with a very brief introduction into the big data session which will come next.

Julian spoke about the need to identify gaps in the data currently being collected, and also referred to some of the rhetoric surrounding the 'big data' agenda which, in itself, can sometimes put up barriers to finding new, collaborative ways of working.

I used two case studies from the media sector to illustrate different ways in which data is being harvested, visualised and analysed. The first was this example from ReFramed.TV and the second, this from www.detectiv.io

Before the next session on March 16, we will post some provocations into the internal critical friends forum and further debate via the comments here also most welcome.'

First thoughts on the literature review

The literature review currently explores two overarching questions which concern the UK arts and cultural sector: how do UK cultural organisations, their audiences, funders and peers define the measures for ‘quality’, ‘value’ and ‘success’ in the context of arts experiences? And, in which way do art organisations employ data-driven decision making to reflect on and implement these metrics within strategic management? 

Whilst quantifiable measures and indicators have become increasingly common in the arts and cultural sector of the UK over the past decade, their use has been criticised as the promotion of dualism and a market-driven economy which is unrepresentative of the complexity of arts experiences and their potential benefits to audiences and participants. Quantifying the effects of the arts is not a simple matter and the lack of bespoke, standardised metrics for quality which works cross-sector is understood to be detrimental to the sector as a whole. The problem of quality measures for the arts has previously been addressed by imposing other forms of alternative metrics to measure impact, for example, in the areas of performance studies, Sociology and Leisure Studies and education. There is a range of so-called intrinsic or individual benefits associated with arts experiences, both public and private, including increased civic pride, improved self-efficacy, learning skills and health, increased socialization and ‘learning new things. 

In order to understand these effects further qualitative approaches have explored audience responses to theatre, revealing visual and oral evidence of pleasure and captivation, a growth in communal feeling, emotional growth and personal resonance. In addition, both qualitative and quantitative research and evaluation of these individual benefits have recently been reviewed in an attempt to develop frameworks of measures which can be applied by arts organisations and pilot work has been carried out to test standardised metrics for use in self-evaluation as well as advocacy, public value statements and investment decision-making. Furthermore, the case has recently been made for an increased use of big data as part of an informed decision making process as part of advertising, user engagement and revenue measure. 

The ongoing literature review will continue to explore the above topics.

Welcome!

This is where we are going to post updates on the project. For now, have a look at this interesting video: