Assessing the Payback from AHFMR-funded research

TABLE OF CONTENTS

EXECUTIVE SUMMARY

SECTION 1:

• INTRODUCTION AND TERMS OF REFERENCE OF THE STUDY
  
  • Background

  • Terms of Reference

SECTION 2:

• METHODS

SECTION 3:

• IS THE WORK ON METHODS FOR ASSESSMENT OF RESEARCH BENEFITS INTERNATIONALLY GENERALIZABLE OR IS AHFMR OR ITS CONTEXT DIFFERENT?
  • Introduction

  • Is the Alberta Health Care System So Different?

  • The local objectives for research

  • The Implications of the Alberta Context

  • The Range of AHFMR Funding

SECTION 4:

• APPLYING THE BUXTON/HANNEY MODEL TO AHFMR-FUNDED HSR RESULTS FROM THE CASE STUDIES
  • Introduction

  • Case-study 1: Diagnostic methods for obstructive sleep apnea (OSA)

  • Case-study 2: Insight into back x-rays

  • HTA Unit Products

  • Case-study 3(a): Stereotactic radiosurgery: options for Albertans

  • Case-study 3(b): Telephone nurse triage services

• CONCLUSIONS FROM HSR CASE STUDIES

SECTION 5:

• EXTENDING THE MODEL TOWARDS CLINICAL AND BASIC BIOMEDICAL RESEARCH
  • Introduction

  • Justifying basic science

  • Experience with the case studies

  • Case-study 4: Transplantation of pancreatic islets of Langerhans

  • Case-study 5: Ligament healing

  • Case-study 6: Electrophysiology of the heart

  • Case-study 7: Lipid metabolism

• CONCLUSIONS FROM THE CASE STUDIES

SECTION 6:

• THE NEXT STEPS
  • Immediate use by the AHFMR
• FURTHER DEVELOPMENTAL WORK BY THE AHFMR

• THE BROADER RESEARCH ISSUES

• REFERENCE LIST

We would like to thank our colleagues at Brunel University, particularly Dr. Steve Hanney, and at the AHFMR, particularly Dr. David Hailey and Dr. Matt Spence for their help, advice and encouragement; the latter additionally for providing us with opportunity to work on these fascinating issues. Most importantly, we would like to thank those researchers who agreed to act as 'guinea pigs' in our testing of the model and for the stimulating ideas they helped produce.

Executive Summary

SECTION 1: INTRODUCTION AND TERMS OF REFERENCE OF THE STUDY
Health services internationally are emphasizing the need to ensure that medicine and health care delivery are evidence-based. For those involved with health related research, this development focuses attention on the importance of a strong research foundation and it enhances the potential contribution that relevant research can make. Equally, it places an onus on the research community to provide evidence of the value of its own activities, and it is one of the factors leading to a growing interest in methods for assessing, either retrospectively or prospectively, the value of research undertaken or proposed.

A small team of researchers, at the Health Economics Research Group (HERG) at Brunel University, have been considering how best the 'payback' or benefits of funded research projects might be conceptualized, described and measured. This work, initially for the UK Department of Health, proposed a conceptual categorisation of the benefits arising from health services research and introduced a model of 'payback'. These are referred to as the Buxton/Hanney payback categories and the Buxton/Hanney payback model, and they are summarized in Figures 1 and 2. They have been applied, by the HERG Team, in a number of ex post case studies of applied health services research projects in the UK.

There has been a considerable interest in the wider applicability of this work both to HSR undertaken in other countries, and to basic science or clinical research that may not yet be immediately applicable to health policy or practice. Professor Martin Buxton was invited by the Alberta Heritage Foundation for Medical Research to test and develop the Buxton/Hanney model in the context of its own programs of research. The objective was not to assess the overall payback from AHFMR-funded research, nor to 'audit' individual projects. Rather, the work focused on testing the approach to see if it might prove useful to the AHFMR in reviewing the payback achieved, or achievable, from the research it funds. This involved both confirming that the approach was transferable from the UK context, and the additional challenge of establishing whether the approach, or an adaptation of it, could be applied to other areas of health research such as basic biomedical and clinical science.

SECTION 2: METHODS
A selective case study approach was adopted in which the studies would be developed only as far as was necessary to begin to test the applicability of the model. This was consistent with the objective to test the approach, rather than the research, or the researchers, chosen as test cases.

In order to test whether the Buxton/Hanney approach could readily be applied to HSR in this different context, two case studies were chosen that represented good examples of applied HSR funded by the AHFMR. Interviews with the researchers were used to verify conclusions drawn from existing written materials and to identify any payback in other categories that might have been missed. In addition, two case studies were undertaken on Health Technology Assessment (HTA) projects from the AHFMR HTA program.

To test the model or a variant of it, on both clinical research and basic biomedical research, the work of four distinguished researchers, who had all received extensive AHFMR funding over a number of years, was chosen. In each case, the mode of working was to undertake a desk review of materials on file followed by interviews with the researchers themselves.

SECTION 3: IS THE WORK ON METHODS FOR ASSESSMENT OF RESEARCH BENEFITS INTERNATIONALLY GENERALISABLE, OR IS AHFMR OR ITS CONTEXT DIFFERENT?
The Buxton/Hanney model was developed specifically in the UK context, but drew on lessons from studies undertaken elsewhere, including work undertaken in Canada. Given the general concern that research findings are not necessarily geographically generalizable from one country or setting to another, it was a necessary first step to test whether the Buxton/Hanney approach, would appear to be valid and appropriate in this different context.

The key characteristics of the local context were examined, focusing mainly on an analysis of the explicit and implicit objectives of health research in Alberta. While there are many similarities to the UK context, there are a number of ways in which the work of the AHFMR does differ from that in the UK for which the model was originally created.

A brief look at the characteristics and context of health care systems in the UK and Alberta suggest that there are many relevant similarities, including the importance of public provision and financing of much health care, and concerns about increasing the effectiveness and efficiency of health care provision. Similar public, political and professional pressures provide the context to research. Specifically, the debate about the need to justify spending on research appears to be very similar in Alberta as in the UK and elsewhere.

If performance is to be evaluated it is essential that the evaluation criteria reflect as closely as possible the objectives of the activity in question. The various statements of mission, goals and objectives of the AHFMR, the Alberta Health Ministry, and the Alberta Science and Research Authority are in no way exceptional and the Buxton/Hanney categories or payback are quite consistent with them.

However, there are some important differences in emphasis in Alberta that need to be reflected in work on payback. Most of the development of substantial research interests in health sciences in Alberta is relatively recent. The AHFMR has focussed on developing a capacity and building an infrastructure for health research. This implies an emphasis on payback in terms of 'development of research skills, personnel and research capacity'. Much of the early funding has aimed to build research teams, to encourage and develop younger research staff and to equip them to begin to be competitive at a national and international level. It has been recognized that the health paybacks would occur in the longer term, from subsequent phases in this strategic development. It has also meant that research funding has been seen as having a broader educational role in helping to educate future researchers.

Moreover there is recognition, within the Government of Alberta, of the economic importance of developing science and R&D capabilities, not least in health related areas, as part of broad long-term provincial strategy. This suggests that the category of broader economic benefits from research may be viewed as important for this strategic investment as the benefits to the health sector itself.

A further important contextual factor is that the AHFMR funds a comprehensive range of activities, including fundamental biology through clinical investigation to studies on population health in the community. The single largest component of funding is the personnel support programs, which contribute to the salary costs of faculty-level investigators in universities and affiliated institutions. In addition, it provides funding for training programs, major equipment and facilities, visiting lecturer and travel awards, visiting professorships, infrastructure grants, and conferences and workshops. Further, through collaboration with Alberta Health, the AHFMR administers a Health Research Fund for individual project support and operates a Health Technology Assessment Unit.

SECTION 4: APPLYING THE BUXTON/HANNEY MODEL TO AHFMR-FUNDED HSR - RESULTS FROM THE CASE STUDIES
The Buxton/Hanney payback model was tested on two specific areas of HSR funded by the AHFMR, and these two case studies have been summarized using the framework of the model. The case studies relate to the evaluation of a novel diagnostic and treatment method for sleep apnea, and to the value of lumbar spine radiographs in the management of low back pain. As expected, no major difficulties were encountered in applying the model to either of these cases and the research sequence model provided a useful framework for reviewing, ex post, what had been the benefits from the research.

In that neither of the streams of research studied had been commissioned, an aspect of particular interest was to see how the process of 'research needs assessment' had occurred. It was possible to reconstruct how the researchers had identified the original topics and how they had continued to identify key research sub-topics and objectives at each stage of their work.

The other two HSR case studies were examples of health technology assessments undertaken by the HTA Unit of the AHFMR, relating to stereotactic radiosurgery and telephone nurse triage services. These commissioned studies were also readily amenable to analysis using the Buxton/Hanney approach.

Together they provide clear examples that suggest that the approach is applicable to HSR in the Alberta context. They also show that an analyst, who has no previous experience of the model can use the approach, and that it is possible to go a considerable way in analyzing payback from a project using existing on-file sources of information. The interviews with the external researchers tended to be mainly confirmatory, rather than essential. Certainly it would be possible to undertake such analysis routinely, if an after-project review report were provided by researchers some time, say two years, after the end of funding.

SECTION 5: EXTENDING THE MODEL TOWARDS CLINICAL AND BASIC BIOMEDICAL RESEARCH
Previously, the Buxton/Hanney approach has been applied principally to examples of applied health services research. Where the research has been methodological, it has been observed that the payback will typically arise through the subsequent use of the methods in applied research. By extension, the same argument might be applicable to more basic research in health sciences, where health service or economic benefits (and other benefits) arise as a result of the incorporation of the results of the basic biomedical research into subsequent more applied research.

The scientific literature highlighted studies that had retrospectively traced the path from application back to basic science. To understand the eventual payback from particular investments in basic science, it would be necessary to trace the linkages forward from a single piece or program of research, as results and concepts are taken up, or influence, an increasingly disparate range of subsequent research and eventual application. In principle, this forward mapping could identify complex linkages between basic research and future health benefits. Although some basic research ideas may be taken up and applied rapidly, others may only be applied after a significant time has elapsed. Because basic science may have very disparate and unforeseeable impacts, attempts to identity and describe them may involve much wider searches than might typically be needed to follow-up on a specific piece of applied research.

Four case studies were chosen as important examples of long-term research support provided by the AHFMR to respected researchers: two clinician investigators and two basic scientists. The research concerned transplantation of pancreatic islets of Langerhans; ligament healing; electrophysiology of the heart, and lipid metabolism. Each was studied to assess the usefulness and applicability of the existing payback model, or possible variants of it.

It was concluded that the research sequence model provided a useful framework for reviewing what had been the benefits from the research, and that there was particular merit in focussing on the two interfaces. The first interface is between the process of defining research 'needs' and the undertaking of a specific piece of research. None of the four studies had been commissioned: it was the researchers themselves who identified the original topics and key research sub-topics and objectives at each stage of their work. Typically, the researchers' original interest in the broad topic or theme arose through happenstance. What also came to light was that some researchers moved their original curiosity-driven, or health needs-led positions. For the clinicians, moving towards basic science may have been a recognition of the need to understand mechanisms by trying to determine the molecular basis of disease. For basic scientists the focus on clinical application may have reflected not only a real interest in improving health, but also a recognition of the access it provides to alternative sources of funding.

There were important differences in the subsequent process of defining and refining research strategy and objectives, with differing degrees of involvement of the broader research community. In none of the cases studied did there seem to have been a significant research program specific interaction between the researcher and the AHFMR, despite the latter's use of international scientific reviewers for proposals, and the availability of these reviews to the researcher. Although researchers welcomed this lack of interference, an opportunity to productively influence research direction or design may have been lost.

The second interface relates to the way the research is disseminated, or fed into the international reservoir of knowledge. The researchers interviewed expressed very different attitudes to this process. For some, publications were the sole measure of communication and success: others emphasised additional mechanisms and channels. These seemed to be particularly important for those who were personally concerned to maximise the impact of the research in terms of downstream applications. This has important implications for the value and relevance of bibliometric indicators.

These case studies confirmed the importance of understanding better the feedback loops, through which research feeds into other research. If the model were to be used for basic research, then it would seem important to begin to conceptualize how these feedback loops operate, and to categorize what is being fed back. Focusing on the 'dissemination' interface through which these feedbacks occur could perhaps develop these ideas. This is particularly important when the feedback is into research being undertaken by quite separate groups, possibly from a different discipline. In order for research ideas to migrate from the basic science bench to application in the clinic, it may need to be disseminated out of the narrow specialty into the thinking of a more disparate scientific audience.

Thus, while the case studies indicated that there are significant difficulties in attempting to analyze the payback from more basic research, they showed that the 'payback' model provides a framework for posing some key questions and for structuring ideas and indicators of payback.

SECTION 6: THE NEXT STEPS
From a short period of study of a very complex question, it is possible to draw a number of tentative conclusions and proposals for further work.

Immediate use by the AHFMR

• The Buxton/Hanney approach could be applied to all, or at least a large sample of the projects undertaken with the HTA Unit's program of work. The objectives of this would be:
  • to provide a monitoring device and a process of accountability,
  • to begin to indicate which circumstances or types of projects, have the best 'payback', as an aid to future prioritization.
• For AHFMR-funded HSR, selective case studies could be undertaken to see whether opportunities exist for the AHFMR to help researchers achieve a better payback on their research.
• The AHFMR could review the routine data collected on HSR projects to see whether it could easily be adjusted to provide better routine indications of payback.
• It might consider whether its dissemination efforts could focus more on achieving the key 'secondary outputs' from projects.
• For basic biomedical and clinical science research, it might review whether the full potential of the interaction between researchers and scientific reviewers (interface a) is being realized.

Further development work by the AHFMR

• It would be valuable, using citations analysis, to try to map the pathways through which specific research results have been taken up in other research, particularly attempting to link through to specific applications.
• Additional case studies should be undertaken relating to the use of the model for analyzing clinical and basic science. It would be particularly useful to explore some projects that have been perceived as relative failures, to see whether this analytical approach could help identify whether they have generated little payback, and if so why that may have been.
• The AHFMR may wish to set up a seminar to encourage a wider local debate about, and commitment to the importance of, these issues.

The broader research issues
It remains unclear how best the bibliometric approaches and the more qualitative Buxton/Hanney approach can most usefully complement each other. Further systematic work is needed.

There is a key on-going disagreement, between those who assert that research is unarguably beneficial in the long term and should not be assessed in terms of short-term benefits, and those who argue that research needs to be justified in terms of its contribution to specific shorter-term objectives. This argument has been played out in a number of political contexts at least since the 1960's, amid concerns from the research community that fiscal restraint and government accountability may disadvantage basic research through failing to recognize its long-term benefits.

What is needed is a vigorous international debate about the best methods for ensuring accountability of research, for ensuring value for money from research programs, and for deciding whether or nor to prioritise research, be it applied or basic. In the absence of approaches that are tested, evidence-based and acceptable to the broad research community, political and bureaucratic pressures will see the introduction of methods that have not been tested, are not evidence-based, and may well not command the support of the research community.

SECTION 1:

INTRODUCTION AND TERMS OF REFERENCE OF THE STUDY

Background
Health services internationally are placing increasing emphasis on ensuring that medicine in particular and health care delivery more generally, is evidence-based. For those concerned with promoting health related research, this movement is very positive in focussing attention on the need for a research foundation, and it enhances the potential contribution that relevant research can make. However, this same evidence-based movement places an onus on the research community to provide evidence of the value of its own activities, and highlights the reality that the mere existence of evidence is not sufficient to ensure changes in policy and practice. In aggregate, it points to growing interest in methods for assessing, retrospectively or prospectively, the value of research undertaken or proposed.

Following an initial approach by the Research and Development Directorate of the UK Department of Health (DH) in 1993, a small team at the Health Economics Research Group, Brunel UniversIty, began working to consider how best the 'payback' or benefits of DH funded research projects might be conceptualized, described and measured. This work has now led to a series of reports (6,7,9,10), and shorter conference presentations and journal publications (8,11). It provides in particular a conceptualization of the benefits arising from health services research in terms of a number of categories of payback (see Figure1), and a model of payback which places the research itself within a longer sequence beginning with the processes of research needs assessment and going on to the use of research and the benefits that arise from its application (Figure 2). These are referred to in this paper as the Buxton/Hanney payback categories and the Buxton/Hanney payback model. Using these as frameworks for analysis, the Brunel team have undertaken and published some 18 ex post case studies, analyzing research projects or programs undertaken in the last few years (6,10). Currently the team is undertaking work for North Thames NHSE R&D Directorate reviewing research and related educational activities that it has commissioned in the last few years with a view to establishing a routine system for monitoring 'payback' achieved.

To date this work by the HERG Team has all been undertaken within the context of the UK and the published case studies have all been applied health services research, or methodological development for use in applied health services research. However, there has been a considerable amount of interest in the wider applicability of this work both to HSR undertaken in other countries or settings, and to basic science or clinical research that is some way from being of immediately applicable to health policy and or practice. Following interest expressed by the AHFMR in this work, it was agreed that Martin Buxton should spend a short period working at the AHFMR, to test whether the model might be useful to the AHFMR in reviewing the payback achieved, or achievable, from the research it funds.

The aim of this particular project was not, at this stage, to assess the overall payback from AHFMR-funded research, nor to 'audit' individual projects. Either of these would require that thorough and systematic case studies be undertaken on all, or a formal sample of, funded research. Neither would have been feasible within the time available, nor appropriate without some initial testing of the model in this context. Instead the work focused on this initial testing.

Terms of Reference

There were two broad objectives to this phase of model testing:

• The first was to confirm that an approach developed in the UK for the Department of Health to help them look at the benefits gained from health services research projects could usefully and meaningfully be applied to similar applied research funded through the AHFMR.

• The second objective, more speculative and challenging, was to see whether the approach, or an adaptation of it, could be applied to funding focused on less applied areas of health research such as basic and clinical science.

The additional factor, and in essence a third test to the model, was that in both instances the approach had to work in the context of where most of the AHFMR's funding is focused. That is primarily to support individuals, or teams, as part of a process of capacity building, rather than simply short-term funding of specific individual projects.

In the light of this testing the agreed remit was to make some initial recommendations as to how most usefully the AHFMR's interest in this broad issue might be pursued and how the techniques and methods might best be applied and developed.

Figure 2: Buxton/Hanney Payback Model, a model for assessing payback

SECTION 2:

METHODS

Most of the work was undertaken in a two-month period during mid-June to mid-August, 1998, and completed in October 1998. Professor Martin Buxton (of the Health Economics Research Group, Brunel University) who had led the UK studies on which this project was based, and Wendy Schneider from the AHFMR undertook the work. Within the limited time available, it was agreed to use a selective case study approach in which the case studies would be developed only as far as was necessary to begin to test the applicability of the model. It was not feasible, and was never the intention, to make the case studies exhaustive accounts of the research in question. It was the approach that was being tested, neither the research nor the researchers, chosen as test cases.

The first step was to attempt to get a reasonable understanding of the context of the work of the AHFMR and of the health system in Alberta, with a concern to establish whether there seemed to be any particular characteristics which might cause problems in applying the existing Buxton/Hanney approach. This was achieved through review of earlier studies undertaken in Alberta pertaining to this broad question, and background information on the AHFMR, the Alberta Health Care system, through informal discussions with the AHFMR's staff and with a number of researchers involved in health research. Additional insights came through discussion with others involved in the health service and analysis of the AHFMR's funding processes relating to HSR. In addition Wendy Schneider brought first-hand knowledge and experience of the local system.

To test whether the Buxton/Hanney approach could readily be applied, two case studies were chosen initially as good examples of applied HSR funded by the AHFMR. As such, they had already been fairly well documented. Indeed, the work by one researcher had been written up as a case study in the report by Kerry Toll (35). It was largely possible to undertake both case studies to the extent necessary to test the model using materials on file: research applications, reviewers comments, annual reports and copies of main publications and secondary accounts of the research to the extent necessary to test the model. Interviews with the researchers were done to verify the conclusions drawn from the file materials and to identify other categories of payback. In addition to these case studies of funded HSR, two case studies (3a and 3b) were undertaken on Health Technology Assessment (HTA) projects, part of the work of the AHFMR's HTA program. These, too, were undertaken principally from on-file documentation, and with the first hand knowledge of, and access to, the staff researchers concerned.

The second and more innovative part of the work involved establishing whether the Buxton/Hanney approach, or a variant of it, could usefully be applied to biomedical and clinical research. For this it was agreed, in this first phase to use a number of case studies to test the model and focus thinking about the problems posed by more basic science. On the advice of the AHFMR's staff, four distinguished researchers, from the Universities of Alberta and Calgary, were chosen each of whom clearly was acknowledged as a leader in their respective fields, and whose research is of the highest standard. The first two were Clinical Investigators: Dr. "A", and Dr. "B". The second two were basic biomedical scientists: Dr. "C" and Dr. "D". All four had received extensive AHFMR funding over a number of years. In each case the mode of working was, again, initially to undertake a desk review of materials on file and then to interview the researchers themselves and to explore the way that they perceived and conceptualized the 'payback' on their own research and to consider how this payback might be assessed. For these four interviews, a common structure of broad questions was used, although general questions were individualized somewhat as a result of the initial desk review of their research. The general questions are set out in Figure 3. Due to time constraints, it was not possible to follow-up issues raised in these interviews with others, such as those who may have used the research findings, and hence, once again, the case studies should not be seen as definitive analyzes of the payback from these streams of research. They simply provide an initial indication of how the model might be used in such contexts.

In parallel with the case studies, a continuing review of relevant literature on the evaluation of the benefits of research, particularly basic science research was undertaken. While at this stage this does not represent a systematic review of that diffuse, and difficult to access, existing knowledge base, our work does draw on some of this research, by way of ideas or contrasting approaches.

Figure 3: Interview questions for researchers

SECTION 3:

IS THE WORK ON METHODS FOR ASSESSMENT OF RESEARCH BENEFITS INTERNATIONALLY GENERALIZABLE OR IS AHFMR OR ITS CONTEXT DIFFERENT?

Introduction
There appears to be a cycle of interest in establishing the benefits of research. At one point in time those who assert that research is unarguably beneficial in the long-term, and an undeniable public good, may be in the ascendant. At another point of time those who argue that spending on research needs to be justified in terms of its contribution to specific shorter-term benefits have a stronger pull on the public purse-strings. In reviewing some of the international literature, it is interesting to see how the concerns of the US academics in the late 1960's about the stance of President Lyndon Johnson. His statement that basic science needed to be more targeted and the time from laboratory discovery to useful application reduced (14) are mirrored in the recent concern expressed by the Federation of American Societies for Experimental Biology (17). Their concern was that the 1993 Government Performance and Results Act might disadvantage basic research through failing to recognize its long-term benefits.

In drawing on this international literature, it is tempting to presume that method for the assessment of benefits or value derived from research might be internationally generalizable. While the earlier Buxton/Hanney work was developed specifically in the UK context, it certainly drew on lessons from studies undertaken elsewhere. For example, its consideration of the policy impact was informed by a stream of work undertaken elsewhere in Canada reviewing the policy impact of the Quebec Council on Health Care Technology Assessment (22-25). This work appears to reflect very similar policy concerns to those in the UK. It comes to rather similar conclusions about the difficulty of precisely estimating impact but the systematic descriptive analysis of impact is both feasible and worthwhile.

But such presumptions need to be checked for in Health Services Research there is a frequent concern that research findings may not be geographically generalizable. For example, a new technology may be cost-effective in one country or context but not in another, or a particular form of incentive may be powerful in one health care system but ineffective in another. Generally, however, methods of analysis are transferable, although even these may be culturally influenced. For example, willingness-to-pay as a way of obtaining individuals' valuations of health states has tended to be much more readily accepted and practically valid in the USA. There, patients are more used to making direct payments for health care, than in the UK, where patients have different cultural expectations and experience.

Thus, it was a reasonable precaution to consider as a first step whether there was evidently any reason why the nature and objectives of the AHFMR, or the health care and health research systems in Alberta, should make the Buxton/Hanney approach invalid or inappropriate in this new context.

This section begins by very briefly reviewing a number of characteristics of the local context but focuses principally on analysis of the explicit and implicit objectives of health research in Alberta. It highlights a number of ways in which the work of the AHFMR does differ from that in the UK for which the model was originally created.

Is the Alberta Health Care System So Different?
In this brief period of study, no attempt was made systematically to compare the health care systems of the UK and Alberta. However, a cursory examination suggests that for the most part the important characteristics, from the perspective of how one might consider the payback on health services research, are common to both. Other writers have looked at the use of research in the development of health policy in Canada and the UK, and while the pictures they present differ in detail many common themes emerge (26,28).

This is not surprising, for the two systems share much in common: not least the importance of public provision and financing of much health care. They share the same current concerns about increasing the efficiency of health care provision, of ensuring equity of access and of striving towards a stronger evidential basis for health policy and practice. Similar barriers exist: democratically elected politicians in both systems appear to work to equally short time-scales. Health professionals resent external intrusion and monitoring, and prefer self-regulation. Past practice, convention and recent experience are as strong moderators of behaviour as the most powerful of meta-analyses. The public in both countries are fascinated by what science can offer and expect new developments, often heralded by the media well in advance of their being ready for public use, to be immediately available as an addition to the range of health care already on offer to them. More specifically the debate about the need to justify spending on research appears to be very similar in Alberta as in the UK and elsewhere.

It is important, however, to recognize the difference in population sizes, with Alberta having only 1/20th of the population of the UK. Thus, Alberta has a population considerably smaller than any of the UK health regions, albeit in a land mass nearly three times the size of the UK. This has some important implications for the research debate. In terms of excellence in research itself, it would require a much larger investment per capita to achieve the same absolute levels of international excellence as that of a much larger country. Thus, there is likely to be an even stronger debate than in the UK as to whether it is justified to undertake particular research locally or whether it is better to free-ride on international research findings. And, more like the debate at a Regional level in the UK, there may be arguments about the value of the involvement of health professionals locally in non-internationally competitive research, as a way of enhancing their receptivity to research findings generated elsewhere (7).

But, while recognizing many important differences of details, an informal review of the system indicated that there were no major structural or cultural differences that might invalidate the underpinnings of the Buxton/Hanney approach. There seems every reason to expect the framework to be relevant.

That is not to say that there may not be differences in what enhances, or detracts from, the payback on particular research. For example, the smaller scale may make it easier for researchers to make the necessary, on going linkages to policy-makers (33). On the other hand, there may not be local researchers who can pick-up more basic research and take it a step further towards application. Instead, it may be necessary to ensure the international dissemination of research ideas and then to be able to bring them back into the province when others have developed them further. Thus, the empirical findings using the model might indeed differ somewhat.

The local objectives for research
As with any system of evaluation or performance measurement, it is essential that the evaluation criteria or performance measures reflect as closely as possible the objectives of the activity in question. Thus, one factor that might lead to different conclusions about research might be if there were different objectives, or if within the same overall set of objectives, priorities and relative emphases differed significantly. Sometimes those objectives are implicit rather than explicit and frequently they may be complex and multifaceted. While explicit mission statements may not adequately articulate the true complexity of objectives, they are the obvious and appropriate starting point.

The Alberta Heritage Foundation for Medical Research (AHFMR) has the following mission statement:
We support a community of researchers who generate knowledge that improves the health and quality of life of Albertans and people throughout the world. Our long-term commitment is to fund basic, patient and health research based on international standards of excellence and carried out by new and established investigators and researchers in training.

Created by Legislation in 1979, the purpose of the AHFMR was to establish and support a balanced, long-term program of medical research based in Alberta. The AHFMR stands at arms-length from the Provincial Government and is supported by an endowment fund that generates interest revenue for operating expenses.

In addition, under the Health Research Collaboration, Alberta Health contracted its health research roles and responsibilities to the AHFMR in three main areas. It gave a granting agency role for the Health Research Fund, a research institute role for Health Technology Assessment, and a research policy role to develop a Health Research Agenda and health research priorities.

Alberta Health's mission is: to enhance the public and health providers to protect, maintain, restore and enhance the health of Albertans.

The goals of their health research mandate have been expressed as follows:

• competent researchers work in Alberta, undertaking research in support of health system goals;
• effectiveness and efficiency of the health system is evaluated;
• broad awareness of research results.

In an Alberta Health communication to the Alberta Science and Research Authority (ASRA), the expected outcomes are phrased in terms of impact as follows:

• the application of research results improves the health status of individuals and the population as a whole;
• health system services become more efficient and effective as a result of the evaluation of interventions, programs and services;
• evidence based decision making contributes to the effective management of the health system;
• expertise in health services is enhanced and applied in support of health system goals;
• research results are broadly disseminated, increasing awareness of research results.

In turn the mission of ASRA is:
to enhance the contribution of science and research to the sustainable prosperity and quality of life of all Albertans.

<The Implications of the Alberta Context
These various mission statements and descriptions of objectives and goals are in no way exceptional and could apply to similar institutions in many countries. Each one of the objectives stated above points to an aspect of payback already included in the Buxton/Hanney categories. Indeed, the broader context of a major commitment to health research, but with an increasing need to justify that commitment, reflects a common situation in many parts of the world, and one certainly similar to that in the UK where the model was developed.

However, there may be some important differences in emphasis in Alberta that need to be reflected in work on payback. The first is the recognition that the development of substantial research interests in health sciences is relatively recent in Alberta. Much of what the AHFMR has been doing in recent years has been developing a capacity and building an infrastructure for health research based principally on the two main and competing universities (University of Calgary and the University of Alberta, in Edmonton). This means that there has been an emphasis on payback in terms of Category bii): 'development of research skills, personnel and overall research capacity'. Thus, much of the early research has aimed to build research teams to encourage and develop younger research staff and to equip them to begin to be competitive at a national and international level. The implication is that it has been recognized that the health paybacks would occur in the longer term from subsequent phases in this strategic development. It has also meant that research funding has been seen as having a broader educational role, in helping educate future researchers. These together have meant that the AHFMR has an indirect but very significant objective to foster the success of the two Universities in relation to health research.

Another factor informing research investment is the AHFMR's own periodic review processes. Under the provisions of the AHFMR Act, an International Board of Review (IBR) is required to appraise the operation of the AHFMR at intervals of six years, and report its recommendations to the Trustees of the AHFMR. The AHFMR has undergone three of these reviews, in 1987, 1992, and 1998. The review process assembles information bearing on some of the aspects of payback in the Buxton-Hanney model (categories a, b (i & ii), and e, (i)). The information informs the forward planning processes of the AHFMR.

The Universities in turn may be rather more sensitive than many other Universities to the need to show the broader value of the investment in them. For example the Vice President for Research at the University of Calgary (Dr Cooper Langford) commissioned internal studies of the economic benefits to the Province of Alberta, and of the influence on policy, provided by the Faculty of the University (12,37). Both these reports provide evidence of the impact on policy-making by members of the Faculty of Medicine, which was judged as being the most productive in the University with respect to activities leading to economic development.

An important part of the strategic context for the major investment by the AHFMR in health research is the Provincial concern that, as Alberta moves into the 21st century, there cannot be continued reliance on commodity-based prosperity. A high level of innovation is required based upon investments in science and R&D, and there is concern that in terms of innovative capacity Alberta appears to be falling behind the rest of Canada and the world (14). More recently a task force was set up in the province which concluded:

Alberta has a solid base for health research and for a health research industry. Development of a robust health research industry represents a strategic economic opportunity for this province. This industry has significant potential to capitalize on a strong foundation to advance the creation of knowledge, to improve quality of life and to increase prosperity for Albertans by creating high quality jobs and a more productive and skilled workforce (1).

This suggests that the broad economic benefits from research [Buxton/Hanney Category ei)] may be of equal or greater importance for this strategic investment than the benefits to the health sector itself. Thus, within the standard tensions between what may be best for the health care system (for example, the cautious acceptance of new technology) and what may be best for the economy (for example, a strong home market base to facilitate the export of an indigenously developed technology) the balance in Alberta might be perceived as lying towards concerns about the economy.

Biopharmaceuticals, agri-food, information technology and telecommunications are seen as key R&D intensive areas for development in which Alberta wishes to recruit companies of sufficient size to create critical mass in R&D (2). The Technology Commercialization Program within the AHFMR exemplifies the concern to encourage, if commercially feasible, the commercial development and marketing of technologies that arise from the basic and applied research that the AHFMR and others have supported in the Province (3).

Finally, there seems to be a broad acceptance of the considerable time-scales such investment entails. The Report "Sustaining the Alberta Advantage" recognizes that the benefits from investment in R&D occur over a long time period (see Figure 4). But already, the biomedical disciplines (cell and developmental biology; molecular biology and genetics; immunology, and biochemistry and biophysics) are seen as shining exceptions to the declining research status of Alberta. In these fields, Alberta researchers have publication impacts 2-3 times the world average (for the same fields) and this success is seen as demonstrating the effectiveness of the provincial strategy of investing in the AHFMR (5).

The Range of AHFMR Funding
What may also be unusual is the comprehensive range of activities that the AHFMR supports. By scientific field, the areas supported range from fundamental biology through clinical investigation to studies on population health in the community. The description that follows is the breakdown of expenditures on a total base of $36.6 million in fiscal 97/98. The single largest component is the Personnel Support programs, which contribute to the salary costs of faculty-level investigators in universities and affiliated institutions (56%). The second largest component is the Training programs at 17%. These include Summer Studentship awards, Graduate Studentships, and Postdoctoral Fellowships. Both the Training and Personnel awards are available to qualified researchers across the entire spectrum of research investigation from basic biomedical research to health services and population health research. In addition, the AHFMR has a variety of other programs (totaling 12% of expenditures) which includes funding for major equipment and facilities, visiting lecturer and travel awards, visiting professorships, infrastructure grants, conference and workshop support, and the support for special projects.

A Technology Commercialization program accounts for some 3% to 4% of expenditures and funds Alberta innovators in the transferring of new ideas in scientific findings into commercial health care products and processes.

The AHFMR provides direct support for training programs for the Regional Health Authorities to promote the uptake of evidence, and the development of research projects at the community level (approximately 2% of expenditures). The program involves both the funding of training, and the institutional support services for consultation and methodology support.

In addition, through collaboration with Alberta Health, the AHFMR administers a Health Research Fund for individual project support, a Health Technology Assessment Unit, and provides support for dissemination and health research agenda setting activities (6% of expenditures).

SECTION 4:

APPLYING THE BUXTON/HANNEY MODEL TO AHFMR-FUNDED HSR RESULTS FROM THE CASE STUDIES

Introduction
Given that the Buxton/Hanney approach to assessing the payback from research was developed in the context of HSR projects the first test was simply to see whether it could readily, and usefully, be applied to similar HSR funded by AHFMR in Alberta. The expectation was that this would be feasible, particularly as the earlier review of the context of AHFMR funding (Section 3) had not indicated any major local characteristic that might invalidate the approach.

In order to demonstrate the application of the model two pieces of research were initially analyzed as case studies of externally funded HSR:

• Case-Study 1: Evaluation of a novel diagnostic and treatment method for sleep apnea. Principal Investigator - Dr. HSR-1

• Case-Study 2: The value of lumbar spine radiographs in the management of low back pain. Principal Investigator - Dr. HSR-2

  In addition, two shorter examples of the work of the AHFMR's own internal HTA Unit was analyzed using the model:

• Case-Study 3: HTA Projects:
  • Stereotactic radiosurgery
  • Telephone nurse triage services.

Each of these is summarized briefly below using the framework of the model: the full accounts of the case studies are available upon request. A final section then draws together a number of conclusions from this test of the model with these case studies.

Case-study 1: Diagnostic methods for obstructive sleep apnea (OSA)

This particular research work was conducted under the leadership of Dr. HSR-1 who had been supported since 1984 with an AHFMR 'Scientist' award. During this period he developed a world-class sleep laboratory. Thus, while it is viewed here as a separately funded phase of research, it constitutes part of a longer and ongoing program of research.

Research Needs Assessment The impetus for Dr. HSR-1 to begin this research project came out of a feeling of an inability to treat patients with OSA in a timely manner. Approximately 80,000 Albertans and six to nine million people in North America suffer from OSA. At the time this study was undertaken there were only 2 facilities in Alberta (Edmonton and Calgary) that were able to diagnose sleep apnea. The standard approach, the polysomnogram, to diagnose and to establish a treatment for these individuals typically requires one to two nights of monitoring in a sleep laboratory at a cost of approximately $1,000 to $2,000 in a special hospital based facility. Dr. HSR-1 thought that this technology was extremely expensive and inadequate in terms of its ready availability. A simpler device for outpatient and home diagnosis called SNORESAT® was developed.

The researcher himself identified the need for the research, but patients dissatisfied with services available supported this need locally. The Alberta Solicitor General's Office restricts Albertans suspected of having sleep apnea from driving until they have been diagnosed and treated. The costs of accidents (in terms of mortality, injuries and financial costs) due to sleep apnea are thought to be substantial.

Interface (a) As is normal for AHFMR-funded research, expert scientific review informs the decision to fund the research. Dr. HSR-1 collaborated with the departments of Cardiology, Engineering and Industrial Design which helped to shape the research design. This research was part of an ongoing stream of work and it is obvious that Dr. HSR-1 knew well, and was known well by, the key local stakeholders.

Inputs The major specific 'input' was a 4-year grant totaling $231,544 awarded in 1993 from the Health Services Research and Innovation Fund. The work also drew on the existing facilities of the sleep laboratory.

Processes The main element of the study was a randomized clinical trial involving 300 participants. In the trial, the SNORESAT" was compared with the gold standard: a polysomnogram. The purpose of the research was to change medical practice, specifically by developing a device that efficiently diagnoses sleep apnea without the need of a hospital stay.

Primary Outputs Three papers documenting results for different aspects of this study were submitted to international journals. Examples of journals that have published Dr. HSR-1's work are the American Review of Respiratory Disease. 148(4 Pt 1):1023-9, 1993 Oct., Journal of Applied Physiology 83(3):851-9, 1997 Sept.; and Sleep 19(10 Suppl):S243-7, 1996 Dec. The results have also fed directly into further research by the same team.

Interface (b) The main form of dissemination took place in the form of presentations at international conferences. As well, a post-doctoral fellow from Dr. HSR-1's lab took what he learned with him to Mexico where he is involved in training pulmonary specialists in Mexico and South America. Through the University where Dr. HSR-1 is a faculty member there is an office of 'technology transfer' that provides business advice and strategy for the commercialization of SNORESAT".

Secondary Outputs The post-doctoral fellow continues to look into the relationship between childhood disorders and obstructive sleep apnea. In addition, researchers in the Cardiology field are looking further into the relationship between sleep apnea and heart disease. A commercial company, after becoming familiar with Dr. HSR-1's research on sleep apnea, commissioned him to do a health surveillance study. The study addresses the relationship between sleep apnea and fatigue-related accidents.

Applications It is reported that there has since been some adoption of the technology and there is likely to be a future shift toward diagnosis being performed by the general practitioner that may preclude a visit to a sleep specialist. (A quantification of the extent of usage could be obtained from billing records).

Impacts or Final Outcomes The main benefits to the health service will be in terms of easier patient access to diagnosis and treatment and hence, more widespread treatment of patients than before the introduction of this technology. The cost per patient will be considerably lower, with diagnosis that can be undertaken by general practitioners rather than specialists. There is pending commercial development of the SNORESAT" device. (In future, sales figures could be obtained to provide an indication of the commercial benefits in terms of royalties and as an indication of how widely accepted the device is as a diagnostic tool for sleep apnea).

Case-study 2: Insight into back x-rays
The principal investigator for this study was Dr. HSR-2. The researcher is an epidemiologist, associate professor in the University of Alberta's Faculty of Medicine and Dentistry, an Arthritis Society Scholar and a Heritage Population Health Investigator. The project was a collaborative effort of the U of A's Healthcare Quality and Outcomes Research Centre and investigators in the Alberta Family Practice Research Network.

Research Needs Assessment Practice variations, where physicians adopt different approaches to the treatment of particular patient groups, are commonly found in medicine. Dr. HSR-2 felt that such variations in the use of X-rays for patients with lower back pain was a cause for concern. Therefore, it was appropriate usage of radiographs for this indication that was investigated.

The concern was radiation load. For women, back x-rays have a risk of affecting the reproductive organs. The radiation in a set of lower lumbar x-rays is equivalent to a woman having a daily chest x-ray for six years. As part of the research plan Dr. HSR-2 retrospectively determined patterns of practice among family physicians in relation to the use of lumbar spine radiography in patients with low back pain. In addition, the sensitivity, specificity and predictive value of lumbar spine radiographs in the diagnosis of low back pain disorders were assessed.

At the time of this study the U.S. Agency for Health Care Policy and Research (AHCPR) had recently published guidelines for management of patients with acute low back pain, which included recommendations for the use of lumbar radiographs. The purpose of this retrospective study was to evaluate the potential impact of these guidelines in patients with new episodes of low back pain seen by primary care physicians.

Interface (a) As stated in Case Study 1, expert scientific review is part of the process used by the AHFMR in deciding which research to fund. The research proposal was developed as a collaborative effort between the principal investigator (PI), the Healthcare Quality and Outcomes Research Centre at the U of A, and the Alberta Family Practice Research Network. The PI's involvement with the Cochrane Collaboration and the Arthritis Society provide opportunities for both extramural collaboration and further planning of the research agenda.

Inputs The funding for this study was received from Alberta Health's Health Services Research and Innovation Fund administrated by the AHFMR. The amount was $114,800 over two years (94/95 and 95/96). Expertise of the researchers included a clinical epidemiologist MD, MSc, PhD (the PI) and two co-investigators; a family practitioner representing the Alberta Family Practice Research Network and the Head of the Division of Rheumatology.

Processes The study was 3-staged: 1) a retrospective study of medical records of 963 patients who presented with LBP at a family practice, 2) a prospective study to examine the outcome of patients with LBP, and 3) compliance with AHCPR guidelines, and the guidelines published by Deyo, were assessed. As a result of reviewing the AHCPR guidelines for low back pain it was suggested that a more restricted and cost-efficient set of guidelines should be proposed. It was intended that this research would a) influence clinical practice, b) inform the researchers regarding methodological principles, and 3) inform and influence policy. The ultimate goal was to improve the health and quality of care of patients.

Primary outputs The review of patient records showed that 13% had been given lumbar radiographs. If the recently released clinical guidelines had been applied to this group, 44% of patients would have undergone radiography. This, in turn, would have increased utilization by 238%, and with it the radiation load. Abstracts were presented at the American College of Rheumatology National Meeting, two in 1995, one in 1996 and two in 1998. A journal article describing the results of part of this research was published in the Journal of the American Medical Association (JAMA), June 11, 1997 Vol. 277, No. 22. Additionally, the JAMA abstract appeared in the American Journal of Managed Care.

Interface (b) In addition to the presentations at the American College of Rheumatology National Meetings, these research findings were disseminated in; clinicians' and investigators' newsletters, and managed care journals. The PI mentioned that, typically, after the research data has been compiled, there is discussion with the key-players to decide on how best to disseminate the results. A good quality, peer-reviewed journal may be the best way to influence both policy and clinical behaviour.

Secondary outputs Arthritis/rheumatology researchers from other centres in North America and overseas have expanded on the PI's research by replicating some of these studies and produced similar results. There has been a positive feedback to the PI about this research project. Although there may not be a rewriting of the AHCPR's clinical practice guideline (they are no longer writing CPG's) there has been a change in clinical practice which is the action that rewriting a CPG would, hopefully, effect. Worker's Compensation organizations have shown interest in the kinds of research the PI is involved with which, in turn, may result in another form of policy change.

Applications Family practitioners involved in the study were informed early about the radiation load involved if AHCPR guidelines were used in the diagnosis of low back disorders and may, as a result, be more selective in the use of lumbar radiographs than the AHCPR guidelines would have recommended.

Impacts or final outcomes In the context of this research project, the impacts and final outcomes would be identified as cost savings, increased efficiency and improved quality of patient care. If the AHCPR guidelines are to be used in the treatment of patients with low back pain, there may be unnecessary increases in utilization and economic costs.

HTA Unit Products
The HTA Unit of the AHFMR produces HTA Reports and HTA Briefs. Reports in these series are peer-reviewed prior to publication. As well, the Unit provides rapid assessments referred to as Technotes, which are not subject to external review. The HTA products are intended to be unbiased, transparent documents.

Alberta Health, RHAs and other organizations use information in the HTA products as an adjunct to other information in order to more effectively set policy. Requests for assessments are prioritized by the Unit taking account of their origin, urgency and timelines involved to undertake the assessment, and the level of analysis required. The time required to prepare an externally peer-reviewed report varies, but is typically of the order of 6 months. Non-externally reviewed reports (Technotes) can take 2 months.

Once the type of report and the question to be answered is defined, a researcher from the HTA Unit is assigned to the project and a project outline agreed upon. A literature review on the subject is undertaken, and, if pertinent to the report, additional information such as administrative data will be obtained. Depending on the type of document being prepared, interviews with experts and other pertinent parties will be arranged. A synthesis is then made of available data.

Case-study 3(a): Stereotactic radiosurgery: options for Albertans
Research Needs Assessment The request for an assessment of stereotactic radiosurgery (SRS) came from policy makers within Alberta Health. Patients were being sent outside the province for SRS therapy. The principal question was whether there were any differences in efficacy of two approaches to SRS; the Gamma Knife"(GK) and the modified linear accelerator (LINAC). There was also interest in the efficacy, effectiveness and safety of SRS in the management of several conditions.

Interface (a) The HTA Unit, in consultation with the policy makers decided that a peer-reviewed report would be prepared. The researcher assigned to the project developed the shape and content of the assessment. During the preparation of this report there was continued interaction between the HTA Unit and Alberta Health. The scope of the literature review was refined as a result of these interactions. As well, the report would include a determination of the costs of different options for SRS and some consideration of quality assurance issues.

Inputs The project was supported financially through the HTA program. Input for the assessment included advice from experts in the fields of radiation oncology, medical physics, neurosurgery, and health economics. The time invested in compiling the report spanned 5 months. The number of full-time equivalents would be 3.5 months. The persons involved in preparing the report were the Research Associate (BSc, MSc, major time commitment), the AHFMR librarian, (BSc, MLS), Director of the HTA Unit, (PhD), a Health Economist, (PhD), from the University of Alberta and the HTA Administrative Assistant.

Processes The information for the SRS report built on information in previous HTAs undertaken by agencies in several countries. The literature review was updated from the time of the last of these assessments. As well, interviews with experts in the field of SRS, were undertaken. During the preparation of HTA 9 two further requests for information on SRS were received from Alberta Health. A Technote and an Information Paper were prepared in response to these requests reflecting an ongoing interaction between the process of research needs assessment by the policy-makers and the research process itself. The draft report was reviewed and modified internally and then sent to external reviewers (a Radiation Oncologist from the Cross Cancer Institute, a Medical Physicist from the Tom Baker Cancer Centre and a Radiation Oncologist from McGill University, Montreal). The draft was also delivered in person to Alberta Health so that some discussion could occur. Final editing of the report took account of comments received.

Primary outputs The final report, HTA 9, was published by the AHFMR in March, 1998 (32). The report confirmed findings from other assessments that:

• the quality of the available evidence on SRS effectiveness is limited.

• there is insufficient information to determine the comparative effectiveness of GK versus LINAC.

• the GK approach is more expensive than the LINAC.

• surgery for the indications considered remains the treatment of choice in most cases. SRS has a role in selected cases when surgery is not an option.

• as there is no known difference in efficacy between the GK and LINAC, and the former is more expensive, referral of patients for treatment by GK is not appropriate.

The potential caseload for Alberta is uncertain but estimated to be 30 to 50 per year. An Alberta-based SRS facility might cost about $4,000 per case, at a caseload of 30 per year. This would be substantially lower than referral of patients to other provinces or the USA.

Interface (b) The dissemination of HTA reports takes place by providing copies to key-players. In the case of HTA 9 the key-players were policy-makers within Alberta Health, medical physicists, neurosurgeons, radiation oncologists, Regional Health Authorities, the College of Physicians and Surgeons and the Alberta Medical Association. A structured abstract appears on the AHFMR website, which allows the option of downloading the full report. Reports were also sent out to all that contributed information and in response to requests. The report is also included in the registry maintained by the International Network of Agencies for Health Technology Assessment (INAHTA) Secretariat and the DARE database.

Secondary outputs Alberta Health was contacted to determine the utilization of the SRS report. They reported that the Out-of-Country Health Services Committee has used it extensively. Since March 1998 they have received several requests for Albertans to travel to the USA to obtain SRS using the Gamma Knife" technique. These have been denied and patients have gone to Ontario for treatment or will be going in the near future. The committee has sent a letter advising the Minister of Health that based on this report, they will not be approving requests for patients to attend the USA facilities for SRS and have asked for his support in this decision.

Applications The report has (possibly) provided background and support for subsequent decisions by the Alberta Cancer Board to acquire SRS technology in Calgary.

Impact/Final Outcome Impacts and final outcomes are identified as cost-savings to Alberta Health Care, more appropriate referral and allocation of certain patients to treatment options.

Case-study 3(b): Telephone nurse triage services
Research Needs Assessment The request for this Technote came from a Regional Health Authority (RHA). They asked that an evaluation be undertaken of the literature regarding the implementation of a telephone nurse triage service that answers calls and carefully assesses the caller's symptoms. This typically involves the use of either computerized or written protocols that incorporate an algorithm or 'decision tree'. Callers are given advice based on their symptoms. This may include self-care at home or being directed to the most appropriate health care setting (4). The nurse triage service might provide an answer to the growing demands being placed on emergency medical services in Alberta. The triage service would handle non-urgent health-care issues, which would, in theory, reduce the number of emergency department visits.

Interface (a) During the preparation of the Technote there was contact with the RHA. A preliminary outline of the report was sent to them for their comments. They, in turn, suggested additional information to include in the report and the outline was revised accordingly.

Inputs The time invested in compiling the Technote was spread over 2 months and equivalent to about 2 weeks full time research. The persons involved in preparing the report were the HTA researcher, the AHFMR librarian, Director of the HTA Unit, and the HTA Administrative Assistant.

Processes Preparation of a such a Technote involves a expeditious survey of available literature. Additional information is sought through interviews, in this case, with a Canadian service that provides telephone nurse triage. After a review of available literature a draft Technote was prepared, reviewed by the Director of the HTA Unit, edited and sent to the RHA.

Primary outputs The Technote was published by the AHFMR in April, 1998. It stated that if a telephone nurse triage service were to be implemented in Alberta there would be a need to look at: the population to be serviced, protocols to be used, staffing needs/training, and any direct links to other services.

It was concluded that:

• there are several telephone nurse triage services in the U.S., one in Canada;

• there is evidence that telephone nurse triage services are useful, but this is based on poor quality studies;

• data on outcomes are lacking in terms of use of services and health status of callers;

• legal, organizational, and economic matters need to be resolved. Good quality studies are required to more firmly establish the health-care and monetary benefits of a telephone nurse triage service.

Interface (b) The dissemination of HTA Technotes takes place by supplying copies to key-players in Alberta. In the case of TN14 these included the Regional Health Authority that requested the information, other RHA's, Alberta Health, the Alberta Assocation of Registered Nurses and others. Titles of Technotes are listed on the AHFMR website, and can easily be requested on-line or by phone. Technotes are provided free of charge. Several requests have been received from out-of-province.

Secondary outputs Calgary RHA has used the report as input to further appraisal of a proposed service. Alberta Health has reviewed the report and followed up by asking the HTA unit for more information about telephone triage nursing services. AH plans to conduct a feasibility study regarding the implementation of such a service in the Province.

Applications The Calgary Regional Health Authority (CRHA) was contacted with respect to their interest in a telephone nurse triage service. The final report of a feasibility study conducted by the CRHA was completed late this year (1998). They addressed accessibility to the service, training of personnel and costs. The report has now gone to their Management Committee, along with the Technote, as part of an information package to be used for further decision-making.

Impact/Final Outcome Information on impact and final outcome of telephone nurse triage services is not available at this time.

Conclusions from HSR Case Studies
Even from the limited amount of work put into these case studies, it is quite clear that, as expected, the Buxton/Hanney approach works in a comparable way in relation to these AHFMR HSR projects as to the UK DH projects for which it was developed. The case studies raised no new problems or specific difficulties. Much of the work on these case studies was undertaken by WLS, who was thus far unfamiliar with the approach. This acted as a reminder that, while others can readily use the approach, there is a learning process to gain familiarity with the categorization and stages involved. The case studies also remind us that the framework is not rigid, but needs to be used flexibly to accommodate the circumstances of particular projects. For example, as previously noted, interaction with funders and stakeholders may continue throughout the project, and the process of project specification may, as in some of the HTA examples, be an ongoing process throughout the project.

These case studies show that it is possible to go a considerable way to analyze a project in terms of the model using existing 'on file' sources of information, but that these typically leave some important gaps. The implication of this is that, if the AHFMR were to decide to undertake more routinely such ex post analyses of its funded research this could be facilitated by making researchers aware of this and by requesting that certain relevant information be clearly provided in annual or end of project reports. In the case of the latter, however, some of the information would not be available immediately at the end of the project, and there may be an argument for an 'after project' review report say two years after the end of the relevant funding. While this might be seen as an unnecessary and undesirable additional burden by some researchers, for those receiving, and expecting to receive, ongoing support from the AHFMR it might be recognized as a reasonable request. Attitudes may depend on the general climate as to whether it seems that such evidence is important in the ongoing process of justifying and funding research, and the observed policy of the AHFMR will be the main determinant of this climate of opinion locally.

Conclusions from HSR Case Studies
Even from the limited amount of work put into these case studies, it is quite clear that, as expected, the Buxton/Hanney approach works in a comparable way in relation to these AHFMR HSR projects as to the UK DH projects for which it was developed. The case studies raised no new problems or specific difficulties. Much of the work on these case studies was undertaken by WLS, who was thus far unfamiliar with the approach. This acted as a reminder that, while others can readily use the approach, there is a learning process to gain familiarity with the categorization and stages involved. The case studies also remind us that the framework is not rigid, but needs to be used flexibly to accommodate the circumstances of particular projects. For example, as previously noted, interaction with funders and stakeholders may continue throughout the project, and the process of project specification may, as in some of the HTA examples, be an ongoing process throughout the project.

These case studies show that it is possible to go a considerable way to analyze a project in terms of the model using existing 'on file' sources of information, but that these typically leave some important gaps. The implication of this is that, if the AHFMR were to decide to undertake more routinely such ex post analyses of its funded research this could be facilitated by making researchers aware of this and by requesting that certain relevant information be clearly provided in annual or end of project reports. In the case of the latter, however, some of the information would not be available immediately at the end of the project, and there may be an argument for an 'after project' review report say two years after the end of the relevant funding. While this might be seen as an unnecessary and undesirable additional burden by some researchers, for those receiving, and expecting to receive, ongoing support from the AHFMR it might be recognized as a reasonable request. Attitudes may depend on the general climate as to whether it seems that such evidence is important in the ongoing process of justifying and funding research, and the observed policy of the AHFMR will be the main determinant of this climate of opinion locally.

SECTION 5:

EXTENDING THE MODEL TOWARDS CLINICAL AND BASIC BIOMEDICAL RESEARCH

Introduction
For the most part, research to date has applied the Buxton/Hanney approach to examples of applied health services research, particularly evaluation of particular forms of health care. This work has already noted that in the case of more methodologically orientated research, the health service benefit, if any, will typically arise through subsequent use of the methods rather than directly from the methodological research project itself. By implication, the same argument might apply to more basic research in health sciences, where health service (and similar benefits) arise as a result of the incorporation of the results of the basic research into more applied research. Indeed there might be a series of 'feeds' into progressively more applied and service focused research, before health service benefits arise.

This section considers some of these possibilities in the light of previous attempts to analyze the role and value of basic science. It then uses four case studies to begin to see how the existing model, or extensions of it, might be used to analyze the research of clinical investigators and basic scientists. It then draws some tentative conclusions, recognizing that the work here represents no more than a first foray into a very complex field of study.

Justifying basic science
The need for researchers to justify spending on basic science seems to arise in regular cycles. Most of the best known analyses of the 'value' of basic science have been carried out as responses to perceived threats to future funding or to scientists' freedom to pursue pure curiosity-led rather than objective-focused research.

One of the first studies tracing the role of basic research in contributing to advances of social and economic importance looked at oral contraception (29). It pointed to the role of the convergence of a number of different streams of scientific knowledge - the physiology of reproduction, hormones and their functioning and steroid chemistry that was necessary for the development of an oral contraceptive. The study importantly established the tracing methodology as a way of portraying research advance.

A more recently cited study using the same basic approach is that of Comroe and Dripps (13). This major study was undertaken as a response to the Johnson Administration in the USA in 1966 that expressed concern that too much basic science remained 'locked up in the laboratory', and that the lag between initial discovery and final application was over long. It was believed that this situation could be improved if basic science were made more 'mission-orientated' or in the case of medical research more 'disease-orientated'. The result was an upsurge in contract research and commission-initiated research supported by the National Institutes of Health.

This position was justified in part by the conclusions of a preliminary report of a study, Project Hindsight, commissioned by the Department of Defense and published in 1966 (34). This study analyzed ex post how twenty important military weapons came to be developed. The conclusions were that: the contributions of university research were minimal; scientists contributed more effectively when their effort was mission-orientated, and the lag between initial discovery and final application was shortest when the scientists worked in areas targeted by their sponsor.

Comroe and Dripps argued that scientists had to counter these arguments with more than illustrative, highly selective, and largely anecdotal case studies. Therefore, they set up a study of a whole field - cardiovascular and pulmonary diseases, and, used 40 physicians to identify the top ten clinical advances in the previous 30 years. Then with 140 consultants they identified the essential bodies of knowledge necessary to achieve the advances. Thus for example, they identified the sequence of 37 key scientific events that necessarily preceded the development of electrocardiography: a chronology stretching back to the scientific advances of ancient civilizations, through to a number of final key steps in the post-war period. For each step, where possible, they identified a key article that was most significant in encapsulating and reflecting the element of scientific advance.

Analysis of these 529 key articles led them to conclude that:

• 41% of all work judged to be essential for clinical advance was not clinically orientated at the time it was done - the scientists responsible for these articles sought knowledge for the sake of knowledge;

• 62% of these articles described basic rather than applied research;

• clinical advances require a combination of different types of research and development, and not one to the exclusion of another.

This classic study emphasizes the complexity of the science base for many current advances, the diversity of component elements on which they drew, and the long time lags between some scientific advances and their useful application.

More recently in 1993, seemingly in response to more recent threats to basic science, Raiten and Berman published an analysis of the development of monoclonal antibodies that traced the seminal scientific events from the time of Pasteur and Koch in 1876, onwards (31). They went on to estimate the very favourable returns on investment involved in this case. However, their analysis suffers from the fatal logical flaw of only allowing for the costs of the science research that subsequently proved useful and not all the research that was funded but may not have had any subsequent importance. (It is 'epidemiology' without a population base!)

In each of these studies cited, the path was traced ex post from application back to basic science. It should, in principle, be possible ex post to trace the path in the opposite direction from a single major piece of basic science research to the various applications that have directly or indirectly drawn upon it; what has been referred to as the 'knowledge cycle' (30). To understand the eventual payback from particular investments in basic science it would be necessary to trace the linkages forward from a single piece or program of research as they are perhaps taken up, or influenced, by an increasingly disparate range of subsequent research and eventual applications. In principle this could link basic science to the specific health benefits which have later resulted at least in part from it. This work has been facilitated by the development of citation indices, pioneered by Eugene Garfield at the Institute for Scientific Information (18). However the data from them needs to be interpreted with caution (19).

There are a number of implications of these complex linkages between basic science and application. The first is that the time-scales can be very long indeed and important discoveries may lie dormant for many years until, with the juxtaposition of another advance, they are taken up and applied (14). Although some basic research ideas may be taken up and applied rapidly, generally such assessments of research would need to take place after a significant time has elapsed. The second implication is that basic science may have very disparate and unforeseeable impacts. This means that the process of attempting to identify and describe such impacts may involve a process of search that goes very wide indeed, as distinct from the very focused search that might follow-up on a specific piece of applied research.

An added complexity is that frequently knowledge develops through accretion and rather than the results of a particular study leading directly to an application, the results accumulate in a reservoir of knowledge (38,39).

An alternative approach may be simply to accept that for much basic science the knowledge generation is the only measurable payback. But this begs a question as to how one would value that knowledge. Ex post, and with 20/20 hindsight, many would probably value a particular piece of scientific knowledge in terms of the importance of that knowledge when subsequently applied. Certainly, if the objectives in funding that research are the pursuit of human health rather than simply knowledge, then relevance as identified by valuable application would appear to be a more appropriate measure of payback than simply some measure of 'volume' of knowledge generated.

Experience with the case studies
With such issues in mind, four case studies of important examples of long-term research support provided by the AHFMR were used as vehicles to explore the relevance of the model, and how it might be extended or elaborated to deal with more basic research. Two case studies related to researchers who were clinician investigators and two related to researchers who were basic biomedical scientists. Below is a summary of each case study using the model's format.

Case-study 4: Transplantation of pancreatic islets of Langerhans
Research Needs Assessment Diabetes mellitus is a medical, financial and social 'time bomb' to thousands of Canadians each year. It causes blindness, kidney failure, stroke, gangrene and impotence. Normalizing circulating glucose throughout the day can prevent these serious complications. An attractive approach is islet cell transplantation that has been shown to succeed in several Type I diabetic patients throughout the world. The aims of this research are to develop transplantation of insulin-producing islets by reducing antirejection drugs for recipients, enhancing the supply of islets for transplantation, and storage of the islets in an ultra-low frozen state (cryopreservation).

As a clinician, Dr. A thought this would be his opportunity to bring science to the bedside. As well, he wanted to satisfy his curiosity, liked the complexity and wanted to contribute to the solution of the problem of Type I diabetes as well as extend the availability of this treatment to Type II diabetic patients. Ideally, this would be before advanced complications set in. [The bigger 'need', of course, would be coming from the patients who are unable to produce sufficient insulin.]

Interface As previously stated, expert scientific review is part of the process used by the AHFMR in deciding which research to fund. An 'interface' would be the exchange of ideas from the reviewers to Dr. A, which, in turn, would have seen him revise his proposal. This particular project builds on Dr. A's successes demonstrated in 1990 for islet cell transplantation. Reviewers of this proposal were strongly in favour of the project. One reviewer suggested that Dr. A look into ways of developing a technique for islet transplant that does not require any form of immunosuppression. The interface of Dr. A with other researchers is mainly through DRINET (Diabetes Research Institute Network). It was set up 4 years ago with the goal of, among other things, steering around competitiveness, promoting more collaborative efforts, sharing protocols and realizing that it takes a global perspective to solve the problem of treating/curing Type I diabetes. There are 8 other groups in DRINET: two in the UK, one in Germany, one in Sweden, and four in USA.

Inputs Dr. A and his team received grant support totaling around $380,000 in 1996/97. Sources included The Alberta Foundation for Diabetes Research, Capital Health Authority, Health Services Research and Innova, Jdf International, Medical Research Council of Canada, and Universities Hospitals Foundation. Dr. A is an AHFMR Senior Medical Scholar (1996-2001). He is recognized nationally and internationally for his outstanding contributions to the field of islet transplantation. He and his associates were the first to demonstrate long term function of transplanted islets into a Type 1 diabetic patient. In addition, they demonstrated that it was possible to use cryopreserved islets for the transplants and thus utilize human islets from several different pancreases. He holds a medical degree (MD) from the University of Alberta, and an MSc (experimental surgery) from the University of Alberta. He is now a Professor and Director of Surgery, in the Division of Surgical Research.

Processes The two major research objectives were 1) to determine the minimum quantity of human islet tissue needed to induce euglycemia after transplantation; and 2) to determine whether immunomodulation of islets using tissue culture, cryopreservation, or gene therapy will reduce their susceptibility to immune injury.

Primary outputs Journals that have published Dr. A's work (1996) include: American Journal of Surgery (1 article), Transplantation (3 articles), Xenotransplantation (1 article), Journal of Clinical Investigation (1 article), and Cell Transplantation (1 article).

Key findings include the development of ex vivo gene therapy of intact islets that protects islets from rejection after transplantation. Improved recovery of human islets for transplantation has been possible through the identification of variables that influence the isolation process. A new bulk preservation method has been developed to preserve islets in the ultra-low frozen state.

Dr. A also supervises a number of students in summer research programs, undergraduate degree projects and graduate programs. He is a major supervisor for an MD and a PhD student. As well, he is a committee member for 4 MSc and 4 PhD students.

Interface (b) Dissemination took place at conferences that Dr. A and his collaborators have attended. These include the 7th Ottawa International Conference on Medical Education and Assessment (Netherlands), 3rd International Congress of the Cell Transplant Society (Florida), 65th Annual Meeting of the RCPS of Canada (Nova Scotia), International Symposium on Endocrine Cell Transplantation and Genetic Engineering (Germany). Other presentations include: city-wide endocrine rounds (University of Calgary), CDA 43rd National Conference (Jasper), Toronto Diabetes Association Annual Meeting and CDA "75 Years of Progress in Diabetes Care, Management and Treatment" (Toronto).

Dr. A is also Chairman of the "Research Development Committee" of the Canadian Association of Surgical Chairmen. It is in this environment that results of his research would be further disseminated.

Secondary outputs Dr. A states that the immediate results of his research are being used by practitioners treating Type I diabetics. However, we were unable to determine the extent of this. Dr. A states that some of his research areas are 'generalizable', that it to say the methods are being picked up by others. An example is researchers involved in cell replacement. The areas of cell transplant research include muscular dystrophy, advanced myopathies, neural cells for Parkinson's Disease and parathyroid replacements.

Other secondary outputs would include the rewriting of guidelines in the treatment of Type I diabetes. Especially important, would be the adoption of these guidelines by a professional body. This research provides an opportunity for intellectual property rights.

Applications This information is not available. However, an example of 'application' of this work would be if practitioners were using the results of Dr. A's research to change their practice patterns.

Impact/Final Outcome The timelines involved in measuring outcomes of this research are indeterminate. However, what one would look for is improved quality patient care. Is this treatment better than anything offered currently? How is it better? Is it more or less costly? Is there an increase in survival and/or quality of life of patients?

Case-study 5: Ligament healing
Research Needs Assessment The impetus for Dr. B to begin research in the area of orthopaedics came from a personal experience with knee surgery. As part of a research project he conducted a retrospective review of persons having knee surgery, specifically ligament surgery. Many were developing arthritis. He realized the need for further research into the area of ligament healing.

Currently the state-of-the-art replacement of damaged ligaments is to use autogenous (host) tendon tissue. Another alternative is the use of allograft tissue (various tendons or ligaments). There is a potential for all of these tissues, both autografts and allografts, to be successful. However, they clearly require optimization. Variables that affect their rates and endpoints of healing are the subject of this research.

Interface As stated previously, expert scientific review is part of the process used by the AHFMR in deciding which research to fund. Any changes to Dr. B's proposal that had taken place as a result of reviewer's comments would be an example of 'interface' mentioned in the model. Dr. B collaborates with a biochemist/molecular biologist and an engineer. There is sharing of ideas that comes mostly at conferences/meetings. A group that Dr. B is part of is the "Rocky Mountain Ligament Group". This group, mostly American, work mainly on ligaments. The ideas that are shared amongst this group work well in planning further research.

Inputs Grants and awards given to Dr. B include the MRC of Canada who has given an operating grant (94-97) and for the "Quantification of loading within normal and healing ligaments" project (97-99). Another award from the MRC was an operating grant for 95 to 98 for "ligament transplantation". Awards from the Arthritis Society (94-2000) have supported work on "The effects of ligament contact on ligament healing". An operating grant was received from the Arthritis Society (97-2000) for "Targeting of collagens by gene therapy to enhance ligament healing". The London Life Award was given as an operating grant for 3 years for "Gene therapy to optimize ligament healing".

Dr. B is an extremely well trained scientist and orthopaedic surgeon. He has a MD degree and did surgical and orthopaedic training. He then went on to do significant research at the University of California, San Diego. His multidisciplinary team consists of researchers from Bioengineering, Biochemistry, Morphology as well as a collagen specialist.

Processes The experimental design will build on prior work in this area. Specifically, the researchers propose to continue studying the potential for longer term incorporation or replacement of grafts at insertions by examining bone-ligament-bone graft properties at the longest practical (2 and 3 year) healing intervals. They plan to look at alternate grafting approaches. Further, they propose to use the tendon graft as an experimental and theoretical model for the study of graft differentiation under different stress conditions. Finally, the application of existing methods and development of new methods to begin complimentary investigations of anterior cruciate ligaments (ACLs) in a larger animal model (sheep) will be attempted. Methods include:

• Multidisciplinary assessments of both soft tissue and insertional portions of each type of graft at each healing interval.

• Biomechanical measurements of both prefailure and failure properties will continue to be the main measures of tissue function.

• Morphologic, histomorphometric (light and electron microscopic) and biochemical measures will be used to understand the processes of healing, reasons for graft differences and help to explain the mechanical outcomes.

• Finite element methods will be used for correlative studies of tissue stresses versus cellular, mechanical and biochemical differentiation along graft lengths.

Primary outputs There have been 16 publications, 1 patent, 16 abstracts and 4 book chapters coming out of this research. The London Life Research Award (1997) was given to Dr. B for his work entitled: "Gene Therapy for Optimizing Ligament Healing". As well, Dr. B's (and colleagues) work in expanding the research team at a Centre that has attracted researchers from all over the world.

Key findings of the research include the development and implementation of a model and methods for studying a liposome-mediated gene transfer technique in ligament scars. As well, new models and methods for the investigation of ACL healing, transplantation and repair were developed. A test system and database for the investigation of mechanisms by which ligaments and ligament scars 'creep' has been developed. The researchers succeeded in altering a number of cells in ligament scars with some exciting effects on ligament organization and mechanical properties.

The number of supervised trainees included: 1 MSc/PDF ('96 to present), 3 additional PDF's ('96 to present, '97 to present, '97 to '98) and one Summer Student ('97).

Interface (b) There were 16 presentations in the 97/98 period. (More than one presentation was given at each conference). They included: the Combined Meeting of the Orthopaedic Association, Keystone Symposia, Fifteenth Annual Surgeons Day, 24th Annual Alberta Orthopaedic Resident Research Day, Canadian Orthopaedic Association and the 31st Annual Meeting of the Canadian Orthopaedic Research Society, and, Transactions of the 43rd Annual Meeting of the Orthopaedic Research Society.

Secondary outputs A measuring device was developed in Dr. B's lab. Some graduate students took this development one step further and formed a company called TENET. They engineer and design surgical equipment and measuring instruments. Dr. B feels that this kind of technology commercialization may be a way of funding further research. Dr. B has been cited at the hand/joint wound-healing research meetings, probably because of the spill over of the two research areas.

Applications This research will not go forward into application without all the proper testing. Dr. B estimated it would be 5-10 years before gene therapy would be ready for trials in human patients.

Impact/Final Outcome This information will not be available for many years. Potential outcomes would include better patient care, possibly a safer mechanism for ligament repair. Cost implications have not been addressed. There is also potential for commercial interest in this particular gene therapy.

Case-study 6: Electrophysiology of the heart
Research Needs Assessment Cardiac arrhythmias remain a major source of morbidity and mortality in western societies. A thorough understanding of the membrane currents that control excitation forms the basis for an approach to arrhythmia prevention and treatment. The studies proposed by the principal investigator would contribute a significant advancement to this field. Dr. C's own interest in this work began as a student more than 20 years ago when he saw nerve/heart cells generate impulses spontaneously.

From a clinical point of view, Dr. C's work may also prove to be relevant in understanding some of the dysrhythmias that occur associated with cardiac inflammation. His work in characterizing mouse heart cell electrophysiology is likely to come into its own as increasing numbers of transgenic animals are used to study human disease. Dr. C uses classical electrophysiology in combination with a range of modern techniques to understand better the events associated with cardiac pacemaker activity and conduction. As one reviewer put it, supporting the downstream benefits of this research: "The connection between understanding SA and AV node function and the development of pharmacologic interventions is obvious".

The principal investigator states that his collaborative work is in part 'applied' in that it is directed toward developing animal models of human disease or mathematical models of human atrium. Both of these topics are relevant to forefront clinical research throughout North America.

Interface As is normal for AHFMR funded research, expert scientific review results in the decision to fund the research. If the reviewers' comments influenced changes in the research proposal this was not mentioned when Dr. C was interviewed. Comments from reviewers were generally positive and supported the research plan.

Inputs Main grant support comes from MRC of Canada Group Grant, (1996 to 2001) covering six independent projects and four Core Facilities. In addition, support is provided by The Heart & Stroke Foundation of Canada (1995 to 1998), an endowed Chair sponsored by the Alberta affiliate of The Heart & Stroke Foundation of Canada (1995 to 2000, renewable to 2005, pending review in 1999). The Wellcome Trust has given a Foreign Starter grant over 5 years.

Dr. C continues to Head a Cardiac Electrophysiology laboratory. His education includes a BSc and MSc (both in Physiology). His PhD was completed at Yale University. He was a PDF at Oxford University. A second PDF was completed at the University of Washington in Bioengineering. He was Assistant Professor in Physiology at the University of Texas, then Associate Professor of Physiology at his present institution. Currently he holds the position of Professor of Physiology and Biophysics. Dr. C was awarded an American Heart Association Established Investigators Award for 1980-83. From 1995 to the present he has had the Heart and Stroke Foundation of Alberta Endowed Research Chair.

There is an ongoing collaboration and consultation with the pharmaceutical industry. Wyeth-Ayerst has provided an unrestricted Educational Grant to the appropriate core of our MRC Group. This involves free exchange of information mainly concerning priorities and directions for development of genetically altered mouse models of cardiovascular disease. A second somewhat similar initiative is ongoing with Otsuka Pharmaceuticals. Within a month additional meetings will be held with Proctor and Gamble Pharmaceuticals and one other international company.

Processes The model systems used to characterize membrane currents and transporters in cardiac pacemakers are myocytes isolated from sinoatrial and atrioventricular nodes and the Purkinje cells of the conduction systems. Conventional step voltage clamp and an action potential clamp is used to determine the time and voltage dependence of the membrane currents that control repolarization and the hyperpolarization-activated current. The coupling between pairs of cardiac cells and the conduction between groups of cardiac cells will be looked at with classical voltage clamp studies and with voltage sensitive dyes in a collaborating laboratory. The membrane currents that underlie the action potential in the mouse heart will be studied using both normal and transgenic mice. These preparations may provide useful models of a number of human diseases.

Primary outputs There were 11 publications and 6 abstracts that came out of this research during 1996-1997. The journals include Journal of Physiology, American Journal of Physiology, Circulation Research, Canadian Journal of Cardiology, FEBS Letters, Chaos, and Nature.

Key findings include the completion of a mathematical model of the electrophysiological responses in human atrium, publication of the first full-length description of the mechanisms of repolarization in subsidiary cardiac pacemaker/conduction tissue, the Purkinje cell from rabbit heart. Collaborative work done with scientists at the University of Utah resulted in improved understanding of electrontonic coupling between cells within the atrio-ventricular node, as well as myocytes from atrium and ventricle. There was further characterization of the repolarization process in the atria and ventricles of control mouse hearts, and hearts rendered arrhythmic or exhibiting congestive failure as a result of selected genetic engineering. The cloning of some of the potassium channels from mouse heart has been carried out in the Molecular Biology and Educational Cores by two trainees. The PI has collaborated with a researcher at the University of Alberta who is an expert with electronics and together with a colleague's computer skills they have developed a high speed, high-resolution ultrasensitive camera for use in the context of studying normal cardiac rhythms, and rhythm disturbances.

Dr. C supervises postdoctoral fellows (4-5/yr) and PhD candidates 1-2 at his University; and 1-2 at Rice University. In addition he sits on 4 graduate student advisory committees and does 3 external exams per year. Teaching duties include graduate students, the nursing class and the first year medical class. Dr. C is participating in the preparation of the first 'electronic textbook' for physiology and biophysics as part of an initiative of the Biophysical Society.

Interface (b) Dr. C participated in the establishment of a lap top computer familiarity/education program at a local Children's Hospital. As well, he gave public presentations for the Heart and Stroke Fund during annual Heart & Stroke Week. Dr. C has a consistent track record of productive interactions with clinical scientists. This has involved supporting their junior colleagues (fellows) and writing and rewriting their applications and papers. An ongoing collaboration exists between Dr. C and Rice University.

In July of 1997 Dr. C and his colleagues presented research at the International Union of Physiological Sciences in St. Petersburg, Russia. They contributed as invitees to 3 sessions.

Secondary outputs The genetically altered mice that Dr. C has collaborated in designing, with congestive heart failure in mind, may be picked up by the pharmaceutical industries. The biotechnology area may pick up the use of the camera, designed and built by Dr. Frank Witkowski at the University of Alberta, to use in experiments that track the ventricular rhythms, i.e. the genesis of fibrillation. From this they can determine where to concentrate further research areas. As well, the neurology researchers are picking up the tracking dye/camera ideas. It is a technology that works even better in brain and gut. For example, the high-speed camera, with a tracking dye, may show how a drug designed for gut disorders may have a direct, negative impact on the heart. The mathematical model that Dr. C has created is the first and only model of human atrium. Both the device industry and the pharmaceutical industry are interested in it.

Applications There is no application of this research at this time.

Impact/Final Outcome As there has been no application, there would be no way to measure impact or final outcomes.

Case-study 7: Lipid metabolism

Research Needs Assessment Dr. D stated that the impetus to start him on this course of research was born out of curiosity. He wanted to understand lipid metabolism at the biochemical level and to understand the basis of atherosclerosis. Phospholipids are components of all cellular membranes. These researchers have shown that biosynthesis of one of these phospholipids, phosphatidylcholine (PC), is necessary for cells to grow and divide. The major pathway for PC biosynthesis in all animal cells is the choline pathway. Liver cells are different since they also have a second pathway, the PEMT pathway. It was not obvious why this PEMT pathway exists and only in the liver.

Other work in this lab includes characterizing the enzyme, triacylglycerol hydrolase (TGH). It is involved in the secretion of fat from the liver into the blood stream. The DNA that codes for TGH was cloned and expressed in immortalized liver cells in culture. The cells that had TGH used fat stored in these cells for secretion much better than cells that did not have TGH. These and other results suggest that TGH may have an important role in regulating the mobilization and secretion of fat from liver. The enzyme is a potential target for therapeutic intervention to lower serum fats and other lipids.

In the past research also concentrated on investigating the role of apolipoprotein E in neurons in relation to the putative linkage between an accumulation of an isoform of apo E and Alzheimer's disease. Since neurons do not make apo E, a fundamental question is: how does apo E enter neurons and escape proteolytic degradation? The fate of cholesterol and phospholipids that are associated with apo E when added to the medium of cultured neurons will be characterized.

Interface As is normal for AHFMR-funded research, expert scientific review informs the decision to fund the research. If the reviewers' comments influenced changes in the research proposal this was not mentioned when Dr. D was interviewed. Comments from reviewers were generally positive and supported the research plan. Dr. D mentioned that his research is strictly investigator-driven with the MRC and the Heart Foundation. The area of lipid metabolism is extremely competitive. There are no major collaborations outside the University. There are 4 different faculty members that interact in planning future research goals.

Inputs Dr. D holds a 'Scientist' award from the AHFMR. In addition he has an operating grant from the Heart & Stroke Foundation of Canada, a shared operating grant from the Medical Research Council of Canada, an operating grant from Medical Research Council of Canada, and an operating/contract grant from Glaxo-Wellcome. The names of the projects for each of these grants are: Phosphatidylcholine and lipoprotein metabolism, Metabolism and trafficking of apoE and lipids in neurons, Regulation of phosphatidylcholine metabolism, and Triacylglycerol hydrolase, respectively.

Dr. D obtained his BSc (Chemistry) from Dickinson College, his PhD in biochemistry from the University of Pittsburgh. He did a PDF at the University of Pittsburgh, a PDF at Harvard University and a final PDF at the University of Warwick. He began as Assistant Professor at a Canadian Institution and left there a full Professor. He also held the positions of Associate Dean of Medicine and then Head of the Department of Biochemistry. He started as a Professor of Biochemistry at his current university in 1986. He is also the Director of the Lipid and Lipoprotein Research Group. Dr. D is a Scientific Officer for the Medical Research Council, a reviewer for Heart and Stroke Foundation of Canada, a reviewer for the Medical Research Council of Canada, a reviewer for the Biochemical Journal Advisory Board, a reviewer for Progress in Lipid Research, a reviewer for Journal of Lipid Research Advisory Board and an Executive Editor for Biochimica Biophysica Acta.

Processes It was not obvious why the PEMT pathway exists and only in the liver. The gene that codes for PEMT was isolated, altered genetically in a test tube then cloned into mice. With this modified gene, mice were produced in which PEMT was no longer present. Further experiments were done to determine the function of PEMT. Other work in this lab included characterizing the enzyme, triacylglycerol hydrolase (TGH). It is involved in the secretion of fat from the liver into the blood stream. The DNA that codes for TGH was cloned and expressed in immortalized liver cells in culture. The cells that had TGH used fat stored in these cells for secretion much better than cells that did not have TGH. These and other results suggest that TGH may have an important role in regulating the mobilization and secretion of fat from liver. The enzyme is a potential target for therapeutic intervention to lower serum fats and other lipids.

In the past research also concentrated on investigating the role of apolipoprotein E in neurons in relation to the putative linkage between an accumulation of an isoform of apo E and Alzheimer's disease. Since neurons do not make apo E, a fundamental question is: how does apo E enter neurons and escape proteolytic degradation? The fate of cholesterol and phospholipids that are associated with apo E when added to the medium of cultured neurons will be characterized.

Primary outputs In 1997, Dr. D has had 8 publications and 5 book chapters. The journals are: Biochemistry Journal, Journal of Biological Chemistry, Journal of Lipid Research, Biochimica Biophysica Acta, and Proceedings of the National Academy of Science. He currently has 4 graduate students and 3 PDFs.

Interface (b) Dr. D has participated in scientific conferences of the Dept of Biochemistry and Molecular Biology, Uniformed Services University, Washington, D.C.; International Conference on Biochemistry of Lipids, Italy; Kern Aspen Lipid Conference, Colorado; Dept of Veterinary Biochemistry, Utrecht University, The Netherlands; Dept of Biological Sciences, Carnegie-Mellon University of Pittsburgh; Glaxo-Wellcome Research Centre, Paris; Dept of Biochemistry and Molecular Biology, University of Manitoba.

Secondary outputs Other labs have made use of the PEMT genetically altered mice. There was an application of Dr. D's and other's earlier research that resulted in a phospholipid being developed that is extremely important in lung surfactant/infant respiratory distress syndrome.

Applications There have been no applications of Dr. D's current research at this time.

Impact/Final Outcome Without application of the research the impact and final outcomes cannot be measured.

Conclusions from the Case Studies
In all four cases the research sequence model seemed to provide a useful framework for reviewing, ex post, what had been the benefits from the research. There seemed a priori to be particular merit in focusing on the two interfaces, and analysis of these identified some interesting issues.

None of these streams of research had in any sense been commissioned: there were no formal prior processes of needs assessment by the AHFMR or other independent parties. It was, therefore, interesting to see how the researchers had identified the original topics and how they continued to identify key research sub-topics and objectives at each stage of their work. What emerged was that, typically, the original interest in the body of research, the broad topic or theme, arose through happenstance. For example, one of the clinical researchers whole interest had been triggered by a personal medical problem: one of the scientists had been stimulated by one particular lecture attended as a student. Subsequently, their interests and focus may have moved to being more or less curiosity-driven or problem-led. It appeared that Dr. A had been initially, and remained, essentially problem-led. His aim was to apply intellectual rigour to finding a better clinical therapy for diabetic patients. Dr. D had been and remained curiosity driven: his intellectual challenge was to achieve a better understanding of lipid metabolism at the molecular level. Dr. C had initially been entirely curiosity driven, but increasingly recognized or accepted the importance of ensuring the clinical relevance of his work. Dr. B's interest had been initiated from an applied concern to improve on available therapy but increasingly he saw the need to go back to very basic science to begin to provide a scientific underpinning, and was increasingly fascinated by that basic science.

While this no doubt represents both an oversimplification and perhaps an over-reading of brief informal interviews, it raises some interesting questions. Why have some of these researchers moved their original curiosity-driven, or health needs-led positions? For the clinicians, moving towards basic science, it may on the one hand be recognition of the need to understand mechanisms by trying to determine the molecular basis of a disease state before you can properly treat it. It may also be the result of experience that has found bench science easier to deal with than persuading subjects and clinical colleagues to participate in clinical trials. For some basic scientists the need to focus on clinical application may reflect not only a real interest in improving health, but also a recognition of the access it provides to alternative sources of funding.

Overall it suggests that the differences between basic scientists and clinical investigators may not be as stark as might be expected, and that as careers develop the balance of motivation may change.

The subsequent process of defining and refining research strategy and specific research objectives also differed. The relative importance of personal views on direction, of the ideas of an immediate group of colleagues within the university and of influences from a wider international research community varied. In some cases, the strategy was a personal one, devised in the context of what was seen as an essentially competitive international research community. In other cases what emerged were quite significant attempts to define an agenda with, and to some extent to agree to a division of the research amongst, groups of researchers working internationally on a basis of friendly cooperative rivalry.

In none of the cases studied did there seem to have been a significant interaction between the researcher and the AHFMR (in terms of its staff or its scientific advisers and reviewers) in refining or focusing a research program or research strategy. The researchers welcomed this lack of interference, but it may represent a lost opportunity to get the greatest benefit from the AHFMR's international scientific review process.

In considering inputs, what emerged unprompted was recognition of the very privileged circumstances they have enjoyed through the long-term support of the AHFMR. All noted that this was a major advantage to working in Alberta, and one that gave them a competitive advantage over colleagues elsewhere.

The second interface in the model relates to the way in which results are disseminated, or fed into the international reservoir of knowledge ready to be picked up and used by others. One point of important difference that showed itself in the interviews was the weight different individuals placed on publications as a measure of research output and as the main means of communicating or disseminating results. For some, publications were the key measure of success and the key means of communication. For others, other indicators of success were as important, for example influencing colleagues clinical behaviour and a key means of dissemination was through more informal communications within research 'networks' that brought together the key research players working on a particular topic. Tentatively, it seems to be the case that there is a relationship between degree of concern about the extent to which research results influence others, particularly in terms of the downstream application of research results, and the perceived importance of outputs other than publications. This, in turn, suggests that the value and relevance of bibliometric measures may vary between areas. Certainly the differences in opinion expressed in this study suggest that such measures cannot be ignored but should not be used alone, incautiously or crudely. Further evidence is needed on the correlation between bibliometric measures and other more subjective measures of research 'value' and impact (40).

Until now, accounts of the Buxton/Hanney model have stressed that research, even at the applied end of the spectrum, feeds back into future research. Research paybacks are the second main Buxton/Hanney category of payback. From the earliest account of the linear model, it was recognized that feedback loops existed from the primary output of research back to further research. Subsequent iterations of the model have emphasized the complexity and importance of such feedback loops, but have never attempted to analyze them specifically. If the model is to be used for basic research then it would seem important to begin to conceptualize how these feedback loops operate, and to categorize what is being fed back.

This is a step that needs more analysis but at first sight the following categories emerge:

• specific pieces of knowledge that fill gaps in understanding and so enable a further line of research to be pursued

• development of broad research models, or techniques or methods

• development of possible therapeutic interventions that need proper evaluation and testing

• transfer of ideas from one therapeutic area, or organ, or disease system, to another outside the range of the original research.

Alternatively, it might be conceptualized in terms of two types of influence that research might have on subsequent research. It may produce incremental knowledge: bit by bit filling knowledge gaps or providing building blocks for subsequent research, or identifying dead ends. Alternatively, it may produce a dramatic conceptual change: a breakthrough in how a problematic issue is perceived, refuting or challenging previous research or current perceptions and requiring a major readjustment of how to proceed. It has been argued more generally that the "impact of research information depends upon its ability to change beliefs or assumptions within the relevant policy audience" (28).

Additionally, the research process itself, rather than its specific outcomes, may feed into future research through the development of trained personnel or tools and techniques that can be used by others that were not the formal research objective.

It would appear that this aspect of the model - research feedbacks - could beneficially be developed, focusing logically on the 'dissemination' interface through which these feedbacks occur, particularly when the feedback is to research being undertaken by quite separate groups. For example, in analyzing publications and dissemination via conferences etc, it may be useful to try to distinguish the types of audience to which they are directed. The type of journal may be a proxy indicator for the type of audience: generalist, specialist or sub-specialist. In order for research ideas to migrate from the basic science bench to application in the clinic, it may need to be disseminated out of the narrow specialty into the thinking of a wider scientific audience.

SECTION 6:

THE NEXT STEPS

The aims of this project have been essentially methodological: to test the Buxton/Hanney approach in the context of the work of the AHFMR. In the short period involved some of the conclusions from this are necessarily somewhat tentative, and need further confirmatory testing. This final section attempts to summarize what has been found and to propose a series of possible next steps. These fall into three broad categories:

• Steps that the AHFMR might reasonably take to apply this approach as it stands;

• Further work that the AHFMR might undertake or commission to further develop and test more formally approaches for its future use;

• A number of key issues that need to form part of a broader research agenda in which the AHFMR might wish to be involved but which is not specific to it.

In each case the suggestions or recommendations are offered here for discussion and would benefit from further thought, development and refinement.

Immediate use by the AHFMR It would appear that the approach could readily be used for all, or a larger sample of the projects undertaken with the HTA Unit's program of work. The objectives of this would be two-fold:

• to provide a monitoring device and a process of accountability, hopefully, to justify to Alberta Health the continuation and possible expansion of this program;

• to begin to indicate which circumstances, or types of projects, have the greater or clearer 'payback', as an aid to future prioritization and to find ways of improving future payback.

A number of in depth assessments of payback from AHFMR-funded HSR could be undertaken, to serve as illustrative examples to use as evidence of the value of this type of work. These case studies would be more formalized versions of the types of analyses already undertaken for the AHFMR (35). Those presented here could be used but they would probably require further work to complete and validate them. The AHFMR might undertake some formal or informal experiments to structure the advice and 'steering' provided to some projects with an aim to maximizing potential payback. It might, for example, help researchers obtain the 'buy in' to the research of key individuals eventually likely to be concerned with policy development from the research.

It would be useful to review the routine data collected on HSR projects, to see whether this could be adapted better to provide indications of payback. For example, the following might be helpful:

• applicants might be asked to more clearly structure parts of the application to set out how the basis of their needs assessment for the project, the form of the interfaces, how they see secondary outputs arising, and what they can do to encourage these;

• scientific reviewers might also comment on these aspects;

• some system of post project end monitoring, for example a report two years after the end of the project might be required.

The AHFMR already undertakes a large amount of general dissemination about projects. However, it might wish to see whether it could actively focus on attempting to ensure that appropriate secondary outputs are achieved. For instance, it might bring together professional leaders to develop and endorse guidelines reflecting and encompassing specific research results, or support other fora at which research-based health policies might be developed.

While firm recommendations relating to basic science research need for the most part to await further developmental work (such as that set out in the next sub-section), it might nevertheless be worth considering whether the full potential of 'interface (a)' could be realized. This could be accomplished by experimenting with a process of feeding back to applicants the reviewers' comments on their applications and encouraging them to provide their response to these comments prior to their proposal being considered. This approach, used by the UK MRC, actively encourages applicants to take on board, or present an argument to counter, suggestions made about focusing research, about improving methods, etc. At present, it appears that these comments are only provided 'for information' after a funding decision has been made and when there may be little incentive to consider changes.

Further developmental work by the AHFMR

It has not been feasible to undertake detailed citations analysis alongside the clinical/basic science case studies. It would be very valuable, using citations analysis, to try to trace and map the pathways through which some of the research results have been taken up in other research. It may be possible to trace these ideas through citations to specific applications, for which associated health benefits might be assessed. This analysis, if it is to involve review of citations (not simply counting them), will be time-consuming and would probably benefit from the skills of an experienced librarian familiar with the ISI database and someone reasonably familiar with the substantive topic.

Such work might then form the basis for beginning to form a view as to whether the AHFMR should invest further in using citations analysis to better understand the significance of the publication outputs from projects. Such an initiative might possibly be made in collaboration with other Canadian funding agencies, and (or) the Universities of Alberta and Calgary.

The conclusions and suggestions relating to the use of the model for analyzing clinical and basic science are necessarily still quite tentative at this stage. If, nevertheless, they are seen as promising, then some additional case studies should be undertaken.

It would be useful to find a way of exploring some projects that have been, or are perceived as being relative failures, to see whether this analytical approach can help identify whether they have indeed generated little payback, and if so why that may have been. In particular, with hindsight, could anything have been done that might have made them more successful?

The AHFMR might wish to set up a seminar to advertise its interest, to encourage a wider local debate about these issues, and to obtain additional views as to how best to take the issues forward. In particular, it would be good to involve some of the basic scientists who perhaps see this approach as rather threatening to curiosity-led research.

The broader research issues

It remains unclear as to how different are the results from bibliometric approaches and these more qualitative approaches such as that of Buxton/Hanney, and how the two approaches can most usefully complement each other. Other work being undertaken at present by Buxton and Hanney for North Thames NHS Executive R&D Program, and similar studies in one other UK Region, may shed some light on this but further systematic work is needed.

There is clearly an important sub-text running through this work as to whether more focusing, priority setting, mission-orientation (or whatever) by the AHFMR would increase or reduce the payback from research. There appears to be very little comparative analysis of this. Currently the AHFMR represents the case of non-direction. Are there other health research funding programs in Canada that reflect a more directed approach and could be analyzed comparatively? (The Quebec HTA program may be an example (22-25)).

These proposals do not, of course, constitute a complete agenda for possible work on 'payback'. This report has not formally considered the value to the AHFMR of using analysis of potential payback to inform itself of the possible magnitudes of benefit that might be achieved from specific projects (27,36). It may be even more ambitious to use such information in the process of prioritizing research spending (15). Such approaches have been considered in the USA by the Institute of Medicine (16,21) and are again under serious consideration in the UK (20). While there is evidence that they can inform individual decisions, particularly those involving major research expenditure, as yet their value in prioritization is largely unproven.

With such ex ante studies, as with the ex post analysis on which this report focuses it is clear that there is a growing international interest. What is needed is a vigorous international debate about the best methods for ensuring accountability of research, for ensuring value for money from research programs, and for deciding whether or not, and, if so, how, to prioritize research, applied or basic. The issues will not go away; they have been around largely unresolved at least since the 1960's. In the absence of approaches that are tested, evidence-based and acceptable to the broad research community, political and bureaucratic pressures will see the introduction of methods that have not been tested, are not evidence-based and may well not command the support of the research community. The AHFMR is in a strong position to take a lead role in pursuing this important international debate.

Reference List

1. Alberta Health Research Industry Task Force. Alberta Health
   Research Industry Task Force, editor. Health Research: A
   Strategic Opportunity for Albertans. 1996.

2. Alberta Heritage Foundation for Medical Research. Crossing
   the Gap, Triennial Report 1996. Edmonton, Alberta, Canada.
   AHFMR. 1996.

3. Alberta Heritage Foundation for Medical Research. From
   Innovation to Commercialization. Edmonton, Alberta, Canada,
   AHFMR. 1997

4. Alberta Heritage Foundation for Medical Research. Telephone
   nurse triage services. Edmonton, Alberta, Canada. AHFMR.
   1998. TN14.

5. Alberta Science and Research Authority. Sustaining the
   Alberta Advantage. http://www.gov.ab.ca/sra/publicdocs/sustain/
   1998.
   Ref Type: Internet Communication

6. Buxton M, Elliott R, Hanney S et al. Health Economics
   Research Group, editor. Assessing Payback from Department of
   Health Research and Development: Preliminary Report.
   Uxbridge, Middlesex UK Brunel University. 1994; No.19.
   Volume 2: Eight Case Studies.

7. Buxton M and Hanney S. Health Economics Research Group,
   editor. Assessing Payback from Department of Health Research
   and Development: Preliminary Report. Uxbridge, Middlesex UK
   Brunel University. 1994; No.19. Volume 1: The Main Report.

8. Buxton M, Hanney S. How can payback from health services
   research be assessed? Journal of Health Services Research
   Policy 1996a;1(1):35-43.

9. Buxton M and Hanney S. Health Economics Research Group,
   editor. Assessing Payback from Department of Health Research
   and Development: Second Report. Uxbridge, Middlesex UK
   Brunel University. 1997b; No.24. Volume 1: The Main Report.

10. Buxton M and Hanney S. Health Economics Research Group,
    editor. Assessing Payback from Department of Health Research
    and Development: Second Report. Uxbridge, Middlesex UK
    Brunel University. 1997c; No.24. Volume 2: Ten Further Case
    Studies.

11. Buxton M, Hanney S. Evaluating the NHS research and
    development program: will the program give value for money?
    Journal of the Royal Society of Medicine 1998;91(suppl.
    35):2-6.

12. Chrisman JJ. University of Calgary, editor. Economic benefits
    provided to the Province of Alberta by the Faculty of the
    University of Calgary. Calgary, AB. University of Calgary. 1994.
    Report prepared for Vice-President (Research).

13. Comroe JH, Dripps RD. Scientific basis for the support of
    biomedical science. Science 1976;192(4235):105-11.

14. Comroe JH Jr., Dripps RD. Ben Franklin and open heart
    surgery. Circulation Research 1974;35:661-9.

15. Davies L, Drummond M, and Papanikoloau P. Informing
    research funding decisions: Payback analysis. Paper presented
    to the UK Health Economists' Study Group. July, 1998.

16. Eddy D. Selecting technologies for assessment. International
    Journal of Technology Assessment in Health Care
    1989;5:485-501.

17. Federation of American Societies for Experimental Biology.
    FASEB News, FASEB adopts policy on the Government
    Performance and Results Act. FASEB . 9-7-1998.
    Ref Type: Internet Communication

18. Garfield E. Citation indexes for science. Science
    1955;122:108-11.

19. Garfield E. How can impact factors be improved? British
    Medical Journal 1996;313:
    411-3.

20. Harper G, Townsend J, Buxton MJ. Preliminary economic
    evaluation of health technologies - a discussion. [In Press]
    International Journal of Technology Assessment in Health Care.
    1998.

21. Institute of Medicine. Institute of Medicine, editor. Setting
    priorities for health technology assessment: a model process.
    Washington, D.C. National Academic Press. 1992.

22. Jabob R. Evaluation de l'influence et de l'impact des rapports
    du Conseil d'Evaluation des Technolgies de la Sante (CETS).
    Montreal, Quebec. 1994.

23. Jacob R. Evaluation de l'influence et de l'impact des rapports
    du Conseil d'Evaluation des Technologies de la Sante. Montreal,
    Quebec. 1995.

24. Jacob R, Battista RN. Assessing technology assessment.
    International Journal of Technology Assessment in Health Care 1993;9(4):564-72.

25. Jacob R, McGregor M. Assessing the impact of health
    technology assessment. International Journal of Technology
    Assessment in Health Care 1997;13(1):68-80.

26. Klein R. Research, policy, and the National Health Service.
    Journal of Health Politics, Policy and Law 1990;15(3):501-23.

27. Lilford R, Royston G. Decision analysis in the selection,
    design and application of clinical and health services research.
    Journal of Health Services Research Policy 1998;3(3):159-66.

28. Lomas J. Finding audiences, changing beliefs: the structure
    of research use in Canadian health policy. Journal of Health
    Politics, Policy and Law 1997;15(3):525-42.

29. Narin F. National Science Foundation, editor. TRACES -
    Technology in retrospect and critical events in science.
    Washington, D.C. USA. National Science Foundation. 1968; ITT
    Research institute report prepared for the US National Science
    Foundation under contract NSF C-535.

30. PRISM. The Knowledge Cycle. 1997; Q4 Ed. London, England: Wellcome Trust.
    

31. Raiten DJ and Berman SM. Bethesda Life Sciences
    Research Office, editor. Can the impact of basic biomedical
    research be measured? A case study approach. Bethesda, MD.
    1993.

32. Schneider WL and Hailey D. AHFMR, editor. Stereotactic
    radiosurgery: options for Albertans. Edmonton, Alberta, Canada.
    Alberta Heritage Foundation for Medical Research. 1998; HTA9.

33. Selby Smith C. Research in VET Decision-making: Chapter
    2. Australia. National Centre for Vocational Education Research.
    1998. Volume 1.

34. Sherwin, C. and Isenson, R. U.S.Director of Defense,
    R.&.E., editor. First interim report on Project Hindsight
    (Summary). Washington, D.C. Director of Defense, Research &
    Engineering. 1966.

35. Toll, K. Kerry Toll & Associates, editor. Making a Difference.
    Calgary, Alberta, Canada. Kerry Toll & Associates. 1998. Case
    Studies on the Socio-Economic Benefits of Research Funded by
    Alberta Health.

36. Townsend J, Buxton MJ. Cost-effectiveness scenario
    analysis for a proposed trial of hormone replacement therapy.
    Health Policy 1997;39:181-94.

37. Unrau Y and McDonald J. University of Calgary, editor. The
    frequency, nature, and impact of faculty influence on policy
    external to the University of Calgary. Calgary, Alberta. University
    of Calgary. 1995. Prepared for Vice-President (Research).

38. van der Meulen B and Rip A. Enschede: University of
    Twente, editor. Assessing societal quality of research in
    environment sciences. Enschede: University of Twente. 1995.
    Indicatoren en indicaties voor de beoordeling vans
    maatschappelijke kwaliteit van onderzoek.

39. Weiss C. Knowledge creep and decision accretion.
    Knowledge: Creation, Diffusion, Utilisation 1980;1:381-404.

40. Wellcome Trust, P. Wellcome Trust, editor. Mapping the
    Landscape. London, England. Wellcome Trust. 1998.
    National biomedical research outputs 1988-1995.