Breaking News – NSW Parliament Adopts our Proposal, go to our Campaign page.

Evidence Based Policy Research Project

(a not-for-profit incorporated association)

As a result, the Institute of Public Affairs (IPA), a self-described ‘free-market’ think tank identified with the Right of politics, and Per Capita Australia, a self-labelled ‘progressive’ think tank identified with the Left, were commissioned in 2018, 2019, 2020, and 2021 to undertake a ‘double blind’ assessment each year of 20 legislated policies drawn from around the country: eight from the Commonwealth and four from each of the three states. In 2022, Blueprint (BP), an ‘economically conservative’ think tank, replaced IPA because the latter could no longer spare researchers for the task.

 

Though the two think tanks used each year often have opposing policy outlooks, their benchmarking results against the Wiltshire criteria were remarkably similar across the 100 legislative initiatives analysed in total over the five years. In each of these years, the think tanks found that basic standards of evidence and consultation-based policy making were inadequately met by Australian federal and state governments.

Our Story


In November 2017 a Symposium convened by the newDemocracy Foundation agreed to challenge opinion leaders by asking them what practical, measurable steps could be taken to improve public trust in government decisions. The objective was to move from complaining about problems to offering potential solutions. The Evidence Based Policy Review Project was one of the major initiatives to emerge from the 2017 Symposium event.

Many participants – company directors, the advocacy sector, journalists, and even former MPs – lamented that “evidenced based policy making” had become an empty phrase which everyone claimed to pursue but no one knew how to quantify

Former Secretary of the NSW Treasury, Percy Allan, championed a proposal to draw on the “good practice” policy making criteria developed by Prof. Kenneth Wiltshire AO from the University of Queensland which had been used in assessing the quality of federal legislation by the Institute of Public Administration Australia (IPAA) in 2012 when Percy was its national president.

With the support of newDemocracy, it was agreed to undertake an independent benchmarking assessment of legislative initiatives from the Commonwealth and three states – New South Wales, Victoria and Queensland – rating them against the “Wiltshire criteria”.

As a result, the Institute of Public Affairs (IPA), a self-described ‘free-market’ think tank identified with the Right of politics, and Per Capita Australia, a self-labelled ‘progressive’ think tank identified with the Left, were commissioned in 2018, 2019, 2020 and 2021, in a ‘double blind’ exercise which each year saw them assess 20 policies drawn from around the country: eight from the Commonwealth and four from each of the three states.

Though these think tanks often have opposing policy outlooks, their benchmarking results against the Wiltshire criteria were remarkably similar, across what is now a total of 80 legislative initiatives analysed. For four years running, the think tanks have found that basic standards of evidence and consultation-based policy making are inadequately met by Australian federal and state governments.

Our Findings

The results of the 100 case studies undertaken so far over the last five years suggest a solid process was followed in thirty-two of them by the governments involved. In twenty-five cases the ratings were well below par. In the balance of cases the process quality was mediocre. See table below

 

Overall Findings, 2018-2022

 

Policy Decision-Making Process

Think Tanks’

Average Score out of 10 Test Criteria

2018
Case               Studies
Number

2019
Case                Studies
Number

2020
Case              Studies
Number

2021
Case                   Studies
Number

2022

Case       Studies   Number

2018-22 Total Case Studies No. & % Share

Solid (Acceptable, 
Sound or Excellent)

7 – 10 criteria satisfied

6

6

9

6

 

5

32 (32%)

Mediocre

5 – 6.5 criteria
satisfied

10

6

9

9

9

 

43 (43%)

Unacceptable

Under 5 criteria
satisfied

4

8

2

5

 

6

25 (25%)

Total

20

20

20

20

20

100 (100%)

 

The think tanks’ total scores on the ten Wiltshire criteria for the eighty case studies to date were remarkably similar in sixty cases (either identical or only one-point difference). Of the remaining twenty cases, seventeen had differences in scoring of two-points. See table below.

Total Score               Differences

2018
Case                         Studies 
Number

2019
Case                    Studies 
Number

2020
Case

Studies

Number

2021
Case                 Studies 
Number

2022

Case                 Studies Number

2018-22               Total Case Studies No.& % Share

None

7

8

6

3

4

28 (28%)

1 point

9

7

10

9

15

50 (50%)

2 point

4

5

4

5

1

19 (19%)

3 point

0

0

0

3

0

3 (3%)

Total

20

20

20

20

20

100(100%)

  

Findings by Jurisdictions

Averaging the two think tanks’ total scores (out of 10) for each of the 100 cases done over the last five years (20 each in 2018, 2019, 2020, 2021, and 2022) gives the following average score for each jurisdiction

Jurisdiction

2018-2022                   

Number of Case Studies

2018-2022              

Average Score for all Case Studies

Federal

40

5.4

NSW

20

5.6

Vic

20

6.0

Qld

20

6.4

Weighted Mean Average

100

5.8

 

Table Legend:

  • Excellent: 9.0 – 10.0
  • Sound: 8.0 – 8.5
  • Acceptable: 7.0 – 7.5
  • Mediocre: 5.0 – 6.5
  • Unacceptable: below 5.0

Our Proposal for Australian Parliaments

The remedy is for a more rigorous process of explanation, justification and consultation (such as the traditional but now-rarely used, two-stage Green and White paper approach) to apply to any legislation that deals with a contentious public policy issue. What that involves is explained here.

At the very least, all bills introduced in a parliament should be accompanied by a Statement of Public Interest (SPI) that answers six fundamental questions at the core of the Wiltshire criteria, namely:

  1. Need
    Why is the policy needed based on factual evidence and stakeholder input?
  2. Objectives
    What is the policy’s objective couched in terms of the public interest?
  3. Options
    What alternative policies and mechanisms were considered in advance of the bill?
  4. Analysis
    What were the pros/cons and benefits/costs of each option considered?
  5. Pathway
    What are the timetable and steps for the policy’s rollout and who will administer it?
  6. Consultation
    Were the views of affected stakeholders sought and considered in making the policy?

An SPI questionnaire would take only a few pages for a bill’s proponent to answer in the affirmative or negative so would not be onerous to prepare. Every member of Parliament and interested citizen is entitled to have this information before a bill is considered.

An SPI would encourage public servants, ministers and private members who develop and submit bills to address the fundamental steps of good policy making.

Also, it would assist a parliament to decide whether a bill should be debated and voted upon without further action or referred to a standing committee of the parliament for closer scrutiny.