Action Research: Measurement and the Lifeworld

What follows is the next part of my action research project that scrutinises how useful outcomes and measurements bureaucracies are for those whom they are meant to be serving – i.e. by making services more effective.

Waves

You can read the previous section of this project ‘Action Research: Deadweight Cost and Meaningful Work’ by clicking HERE.

The previous section examined the rise of, what sociologist David Graeber called, ‘bullshit jobs’ and how they are related to bureaucracies.  It raises questions about the erosion of meaning from lives through the imposition of paperworks and working from the abstract rather than from the concrete and present.  It finishes up examining tangled hierarchies looking at natural phenomena in ant populations.

 

This section of the study examines the suitability of measurement for understanding people’s lives.  It walks through some of the conceptual problems of using metrics and systems of ‘scientific management’ to measure improvements in people’s lives or the performance of support workers.  Threading some of the key principles of participatory action research methodologies through the narrative, it is a critique of the use of the Outcomes Star to organise support for individuals who need it and funding for those who give it.

 

Acknowledging the limitations of bureaucratic metrics is brought together around the fact that the social sciences are complex and non homogenious phenomena.  Thinkers like John Maynard Keynes recognized this and organisations like NASA understood this warning against the imposing of natural science methodologies on subjects of the social sciences.

 

The trend for what gets touted as efficiency has permeated the workings of our world and as a consequence smothered many people not equipped to deal with such onerous paperworks.  Metrics have become a sort of idealised, policy laden representation of the life of a person which parses the individual for funding structures and institutional needs.

 

measuring the unmeasurable

The metrics culture is a political ideology in action which is rooted in the imaginings of structuring forces which implement them rather than a scientific approach identifying and responding to the inconvenient, organic life circumstances of the individual.

 

The lives of people have become product that put the potatos on the tables of a class of people who are interjecting themselves as middle-people in processes which have functioned well without them.

 

As a part of the ritual culture it is evident that the outcomes star perpetuates problems by leaving out of its measurements the structural problems which are bearing down on the individual; it is as false as measuring the economies of African countries without quantifying the exorbitant trade barriers imposed on them.

 

There is no proportionality in the representation of the most vulnerable and most exploited – all detail has been rendered from their lives in the paperworks which govern them.  Burying them – and those who are trying to help them – under reams of administration simply perpetuates the problems through abstraction and the creation of work.  At worst it fosters the dehumanisation of people and the development of professional psychoses.


 

real numbers

Part 6: Measurement and the Lifeworld

The development of empirical measures runs in conflict with some of the complexities of human and organisational development. For example, how do you measure the achievements of an individual or service when the process of change often takes longer than the funding period or beyond the contracted service; or when it only manifests in a subtle meta-observable way (maybe the individual smiles more often) or outside of discrete contact (manifests outside of proximity with the service provider)?

 

A key consideration in the discussion of measuring human development outcomes is the fact that the process of change is often characterised by ‘two steps forward, one step back’ [1]. In a sea of changing social factors, how can failure or success be neatly ascribed to one environmental factor (i.e. support service) when support needs change often and independently of each other.

 

Observation and evidencing are distinct in various ways; not least that one is passive and one is active, respectively, in their influence on the situation. This asks the question of whether it is desireable that the documentation process or the structured support service strategy should be that influence ? The difference between tacitly observing and actively pulling evidence from the situation can often be unbridgeable when dealing with the ‘soft’ variables in the sphere of human action.

 

If we imagine introducing such active measurement behaviours into other interpersonal relationships we might get a sense of the feelings and changes of behaviour which come out of impositions of bureaucracies in intensely personal spaces. As an exercise, one could discuss the idea of measuring and documenting outcomes with your partner or friend to understand part of the imposition of the bureaucratic process.

 

The quest for ways of fostering “change on the inside” is part of the discourse we find in the Outcomes Star literature. The perspective of examining and changing the relationship of the individual to the challenges they face is put forward as the primary focus.

 

In and of itself, the Outcomes Star bureaucracies do not seem to have the ability to do this; much more, this ideal is contained in the theories cited in generating the bureaucracies and most primarily in the ‘skilled helper’. However in practice, what is a static paperwork is far from the reality of Action Research, which as a methodology defines its starting points but not its outcomes.

 

As Bradbury and Reason put it: “Action Research is emancipatory, it leads not just to new practical knowledge, but to new abilities to create knowledge. In Action Research, knowledge is a living, evolving process of ‘coming to know’ rooted in everyday experience; it is a verb rather than a noun. This means Action Research cannot be programmatic and cannot be defined in terms of hard and fast methods, but is, in Lyotard’s (1979) sense, a work of art.” [10]

 

Jean Francois Lyotard
Jean Francois Lyotard

Lyotard provided an important description to the post modern paradigm which enables understandings in terms of working methodologies in the human sphere:

 

“A postmodern artist or writer is in the position of a philosopher: the text he writes, the work he produces are not in principle governed by pre-established rules, and they cannot be judged according to a determining Judgment, by applying familiar categories to the text or to the work. Those rules and categories are what the work of art itself is looking for. The artist and the writer, then, are working without rules in order to formulate the rules of what will have been done. Hence the fact that work and text have the characters of an event” [27].

 

Assessment and outcomes measurement tools often focus on the severity of problems (the number of units of alcohol consumed) or the external circumstances (does the person have a job or a home). The focus on severity negates the person in the process and often amounts to deterministic policy laden triage systems which are not reflexive for the client but for the worker side of the relationship and sector budget. A primary focus on logistics and resources similarly reduces the scenario to a situation which neglects the person or the capabilities/opportunities (or lack of them) inherent in the situation [13].

 

Indeed, we find the problems of measurements engendered on a macro scale with the approximation of the welfare of a nation by use of Gross Domestic Product (GDP) metrics. GDP is criticized in its use for this purpose. Martha Nussbaum explains the inherent problem:

 

“Dominant models asserted that the quality of life in a nation was improving when, and only when, Gross Domestic Product (GDP) per capita was increasing. This crude measure gave high marks to countries that contained alarming inequalities, countries in which a large proportion of people were not enjoying the fruit of a nation’s overall economic improvement. Because countries respond to public rankings that affect their international reputation, the crude approach encouraged them to work for economic growth alone, without attending to the living standard of their poorer inhabitants, and without addressing issues such as health and education, which typically do not improve with economic growth.” [28]

 

The ‘Health of the Nation Outcome Scale’ (HONOS) is widely used in the mental health sector in the UK and measures behaviour, impairment symptoms and social functioning in service users; it is clinician led and each item of measurement is rated on a five-point item of severity (0 – no problem to 4; – severe to very severe problem; 5 unknown or not applicable). The following fields are used as assessment criteria [14]:

 

  • Overactive, aggressive, disruptive or agitated behaviour
  • Non-accidental self-injury
  • Problem drinking or drug-taking
  • Cognitive problems
  • Physical illness or disability problems
  • Problems associated with hallucinations and delusions
  • Problems with depressed mood
  • Other mental and behavioural problems
  • Problems with relationships
  • Problems with activities of daily living
  • Problems with living conditions
  • Problems with occupation and activities

 

Another popular metric is the ‘Treatment Outcomes Profile’ (TOP) which is used in the UK in the field of substance misuse. It measures a range of indicators but focuses on the extent of drug use and of offending. The Treatment Outcomes Profile provides a standardised method for monitoring client outcomes within the drug treatment system in England. Outcome measurements are based on changes in substance use, injecting risk behaviour, crime, health and social functioning during the course of treatment [15].

 

tickbox culture

 

Participatory assessment and measurement takes the position that whilst measures are important, they only give a partial picture. Both service users and frontline workers report that it is important for people to be active participants rather than objects of assessment. The role that agency plays and the validation of people’s situated experience is critical to the changes people are trying to make [16].

 

Acknowledging people’s experience and investing power as locating agency in these accounts requires a very different kind of process and research methodology from the conventional natural science methods for the testing of hypotheses. The process of natural science is commonly given primacy as producing the strongest truth claims in the acquisition of knowing. The research process of natural science is largely based on Logical Postivism and testing hypotheses until failure [18].

 

The reduction of information to a number plotted against a list of areas of a human being’s life extricates the representation from the reality and is arguably reduction to the absurd without an appending body of work which restores context and irreplacable features of the persons life to these figures. The further we abstract people from circumstance, the more tragedies can befall them by assumption, ommission, automation or simplification.
The power of scientific method lies in the ability to replicate and reproduce results.

 

The independent replication over time refines its findings on a scale of “increasingly reliable knowledge”. This replicability of experimental results stems from the fact that the phenomena investigated must be, according to John Maynard Keynes “homogenous through time” – for example, the inverse square law of magnetism is always, demonstrably, an inverse square law [31].

 

Keynes made a point that economics should avoid attempts to turn it into a pseudo-natural science because unlike the typical natural science the phenomena to which economics is applied is, in too many respects, not ‘homogenous through time’.

 

The application of the natural science process to researching phenomena other than those for which they were developed (the regularities of the physical universe) is problematic. Social phenomena are not, in Keynes’ phrase, “homogeneous through time”.

 

Thus for the purpose of research into social phenomena there is increasing engagement in the field of the ‘action methodologies’ of Action Research, Participatory Action Research, Action Learning and Action Science [18].

 

There are problems with scales and measurements in a world where there are few geometrically regular social realities, and in which, societies operate through relative expressions of value. The unaccounted for or unmeasureable in a scale might hold great significance to an individuals circumstance.

 

All or nothing measurements are problematic in exploring fuzzy systems and forced compliance might end up in distorted realities and disturbed priorities.

Fuzzy Logic Example

Zadeh prepared a report on the nature of fuzzy systems for NASA: “Many of the systems encountered in the real world are too complex and/or too ill-defined to be susceptible to exacting analysis. We lack methods for dealing with systems which are too complex or too ill-defined to admit of precise analysis. Such systems pervade life sciences, social sciences, philosophy, economics, psychology and many other “soft” fields.

 

Perhaps the major reason for the ineffectiveness of classical mathematical techniques in dealing with systems of high order of complexity lies in their failure to come to grips with the issue of fuzziness, that is, with imprecision which stems not from randomness but from a lack of sharp transition from membership in a class to non-membership in it” [91].

 

The “extractive” natural science approaches to assessment and measurement create a position of abstraction where in rarefied conditions the “expert” decides what course of action is most appropriate and is then tasked, at best, with convincing (or at worst coercing) the service user that the course decided is the best one for them.

 

The traditional research approach is one in which research aims to increase knowledge and is kept independent from implementation. In this operational perspective, research yields knowledge which is documented and later applied separately in practical situations. This approach contravenes a key concept in any attempt to reach a solution to a problem through scientific method, that of remaining connected to the first principles of the thing itself.

 

Participatory Action Research has the focus of real world, nuanced situations and aims to approach complex problems through a systematic ongoing method of planning, taking action, observing and evaluating. O’Brien describes it as a process: “a spiral of steps, each of which is composed of a circle of planning, action, and fact-finding about the result of the action” [36].

 

If we take Richard Feynman’s definition of science as ‘a system we have devised to keep us from fooling ourselves’, and understand measurements and comparison to be a part of this system, then it logically follows that falsification and increasingly reliable knowledge can only be reached towards if the phenomena under research remain as an integral part of the iterative process.

 

If the researcher attempts to operate in an extractive vacuum in a mythical hunt for pure objectivity they disembody their ability to sense which direction on the spectrum of increasingly reliable knowledge their work is moving. Karl Popper is most famous for articulating these principles [109]. This act of abstraction dismembers the ability to discern the ‘facts’ by separating the scientist from the thing they study.

 

Working in such abstracts is thus an anomalous practice when we inspect the idea of rejecting subjectivity in this light. It is as flawed as bringing a dogmatic article of ideology to a study or investigation. The philosophy of science still must attempt to reconcile the stances of positivism with that of post structuralism, and culturally those subjects which fall into the area of humanities and social sciences are greatly suffering.

 

With the introduction of the metrics and specialised bureaucracies comes a kind of professional demarcation of people from being experts in their own lives.  One account of knowledge is held above another and in the shadow of the dominant professional orthodoxy a silence is created in which experience is discounted.  This is the birth of a structural violence.

 

An ‘expert’ only becomes one through effort of trying and investing time in gaining experience of the subject typified in formally accreditriced training and study. If one layperson who has graduated to having expertise through experience (i.e. formal training) seeks to denigh another layperson the possibility of becoming knowledgeable through experience and sharing that expertise which they have then it becomes a situation associated with status and hierarchy which is removed from science and discovery. It becomes a teleology with the dangers of the protectionism of the middle aged guild system, cut off from new practices emerging from a wider pool of knowledge [60].

 

With the advance of professionalism, Payment By Results (PBR) and social impact bonds the pressure to present evidence of measured change has never been greater, and it continues to grow. This is pressure and workload not simply for commissioners and service providers, but for service users and communities as well.  A struggle for representation and resources embodies itself in the ruling paperwork deracinating (uprooting) people from their own lives and experience.

 

 

YouTube player

 

 

It sets up a cabal of assets in nested hierarchies around phenomena of the natural world, such as Vesalius encountered in his lifetime in regard to medicine [61]. Vesalius, although making great contributions to the advance of medical knowledge, was rejected by the orthodoxy of his day – thus being denied agency in his culture.

 

In contrast, participatory practices open and extend the authentic possibilities of any given area of study, enlightening the co-operants through peer interaction setting the scene for the negotiation of shared values and contribution to knowledge as a transparent commons.

 

Participatory Action Research has an operational perspective where the process of assessing the individual and measuring change is integrally bound up with the process of working with the service user.

 

It is false to imagine the service user out of this shared workload, externalisation of administration, and bureaucratic burden; it is very much an institutionalizing reality particularly for those with no option but to ‘play the game’ as they are in a massively disadvantaged position without access to a route of social mobility or to resources needed to take on such tasks.

 

A primary question is to ask if these bureaucracies emancipate or entrench the individuals dependence on support in a culture of asymmetric and specialized knowledge.

 

Outcomes Star literature itself suggests that simplistic approaches to measurement and payment formulae distort service delivery and incentivises service providers to focus on targets for payment rather than taking a more sophisticated humane approach to measuring and learning from outcomes [1].

 

YouTube player

 

Measuring outcomes is quite different to learning from them. Measuring, metrics and empiricism are not equipped with the language or the expressive capability to deal with such complex ‘systems’ as are represented in the lives of individuals requiring support.

 

Measuring such systems must necessarily involve a human driven observation process, such as the older inspector system in schools, to supplement and override the rote mechanics of a policy driven data array when required. Machines cannot run themselves and skilled management is necessary for reaching ends effectively.  Equally there needs to be a means of input from the service user to the policy level which is autonomous of the operational systems which act to facilitate and/or govern the lives of individuals.

References:

[1] MacKeith, Joy (2011) “The development of the Outcomes Star: a participatory approach to assessment and outcome measurement”, Housing, Care and Support, Vol. 14 Iss: 3, pp.98 – 106

[10] Reason, P., & Bradbury, H. (Eds.). (2008). Sage Handbook of Action Research: Participative inquiry and practice (2nd ed.). London: Sage Publications, pp 1 – 19

[13] Sen, Amartya. (2000). Development as freedom. New York, Anchor Books. Page 87

[14] Eagar, K. Buckingham, B. Coombs, T. Trauer T, Graham, C. Eagar, L. and Callay, T (2000) Health of the Nation Outcome Scales (HoNOS), Australian Mental Health Outcomes and Classification Network, Victorian outcome measurement strategy resource manual. Victorian Department of Human Services

[15] Treatment Outcomes Profile (TOP), The protocol for reporting TOP, A keyworkers guide, 2010, National Treatment Agency for Substance Misuse. Accessed Online 20/06/2015: http://www.nta.nhs.uk/uploads/treatment_outcomes_profile_keyworkers_ guide_final_110110.pdf

[16] Burns, S., MacKeith, J. and Graham, K. (2008) Using the Outcomes Star: Impact and Good Practice, London: Homeless Link

[18] Peter Checkland, Sue Holwell, ‘Action Research: Its Nature and Validity’, Systemic Practice and Action Research February 1998, Volume 11, Issue 1, pp 9-21

[27] Lyotard, Answering the Question: What Is Postmodernism? Translated by Regis Durand, From Jean-Francois Lyotard, The Postmodern Condition: A Report on Knowledge, Minneapolis; University of Minnesota Press, 1984, pp. 71-82.

[28] Martha Nussbaum (2011). Creating capabilities: the human development approach. Cambridge, Mass, Belknap Press of Harvard University Press, ISBN: 978-0-674-05054-9, Page ix

[31] Argyris, C., Putnam, R., and MacLain-Smith, D. (1987). Action science: [concepts, methods, and skills for research and intervention]. San Francisco, Jossey-Bass, Page xi

[36] O’Brien, R. (2001). Um exame da abordagem metodológica da pesquisa ação [An Overview of the Methodological Approach of Action Research]. In Roberto Richardson (Ed.), Teoria e Prática da Pesquisa Ação [Theory and Practice of Action Research]. João Pessoa, Brazil: Universidade Federal da Paraíba. (English version) Available: http://www.web.ca/~robrien/papers/arfinal.html (Accessed 20/1/2002)

[60] Roberta Dessi, Sheilagh Ogilvie, Social Capital and Collusion: The Case of Merchant Guilds, CES IFO Working Paper No. 1037, Category 10: Empirical and Theoretical Methods, Sept 2003

[61] Prof. Howard W. Haggard, Vesalius and the Struggle for Intellectual Freedom, Nature 153, 707-708 (10 June 1944), doi:10.1038/153707c0

[91] Zadeh, L. (1969). “Toward a theory of fuzzy systems” prepared by University of California, Berkeley, California for National Aeronautics and Space Administration Washington D.C

[109] Popper, K. R. (1959). The Logic of Scientific Discovery. New York : Basic Books, Inc., ISBN 0-203-99462-0, Page 22 – 26