This is an extract from Human Rights and Technology Final Report, the culmination of a three-year national initiative led by Edward Santow, Australian Human Rights Commissioner.
Until recently, public debate about AI and human rights focused almost exclusively on the right to privacy. However, the use of AI can affect a much broader range of civil and political rights, and economic, social and cultural rights.[i]
For example, in June 2020 the UN Special Rapporteur on contemporary forms of racism, racial discrimination, xenophobia and related intolerance noted the growing use of digital technologies, including AI, to ‘determin[e] everyday outcomes in employment, education, health care and criminal justice, which introduces the risk of systemised discrimination on an unprecedented scale.[ii]
The UN Special Rapporteur on the right to freedom of opinion and expression,[iii] and the UN Special Rapporteur on extreme poverty and human rights,[iv] have made similar reflections in their respective areas of focus. There is also growing academic and civil society work in this area.[v]
In this section, the Commission considers three areas of government decision making where the use of AI poses particular human rights risks.
(i) Automated government services
Governments in Australia and overseas are increasingly considering or using AI in service delivery. In September 2020, for example, the NSW Government released an AI strategy to guide its expansion of digital government service delivery.[vi] However, there remain risks. As a Council of Europe study observed, automation by government can reduce transparency and accountability, and increase the risk of arbitrary decisions.[vii]
The use of AI, and especially automation, in delivering government services can engage human rights including the right to social security and an adequate standard of living,[viii] the right to non-discrimination and equality,[ix] and the right to an effective remedy.[x]
The UN Committee on Economic, Social and Cultural Rights has concluded the right to social security imposes an obligation on governments to ensure that eligibility criteria for social security benefits are ‘reasonable, proportionate and transparent’.[xi] Further, any ‘withdrawal, reduction or suspension’ of social security benefits should be circumscribed and ‘based on grounds that are reasonable, subject to due process, and provided for in national law’.[xii]
| Case Study 1: AI used in social services Many stakeholders expressed strong views about the human rights implications of automation in the social security area. Particular concern was directed towards an Australian Government program of automated debt recovery in relation to social security payments. Some have called this program ‘Robodebt’.[xiii] The automated debt recovery system used an algorithm to identify any discrepancies between an individual’s declared income to the Australian Taxation Office, and the individual’s income reported to Centrelink. Where a discrepancy was identified, this was treated as evidence of undeclared or under-reported income, and a debt notice was automatically generated and sent to the individual.[xiv] A parliamentary inquiry concluded that this process resulted in some inaccurate debt notices.[xv] As this debt recovery program related to welfare or social security payments, these errors disproportionately affected people who were already disadvantaged or vulnerable.[xvi] Drawing on its casework experience, Victoria Legal Aid submitted that the scheme was poorly designed, unfair, and undermined public confidence in government decision making.[xvii] On 29 May 2020, Services Australia announced all debts raised wholly or partly under the automated debt recovery system would be refunded; a total of $721 million across 470,000 debts would be refunded, affecting 373,000 people.[xviii] In November 2020, the Prime Minister announced the Australian Government would pay a further $112 million in compensation payments as part of an agreement to settle a class action lodged in November 2019. The Government also agreed, as part of the settlement, to not pursue the payment of an additional $398 million in debts that had been wrongly raised.[xix] |
[i] The breadth of the human rights impact of AI was noted by stakeholders to the consultation. See IP submissions: Law Council of Australia, 16; I Law, 8; Commonwealth Ombudsman; Australian Privacy Foundation, Queensland Council for Civil Liberties, Electronic Frontiers Australia, 8; M Paterson, 4; P Henman, 6; M O’Sullivan, 3; Kingsford Legal Centre, 7; National Association of Community Legal Centres, 5. DP submissions: Access Now, 1; Financial Rights Legal Centre, 4; Law Council of Australia, 5; Castan Centre for Human Rights Law, 4.
[ii] E Tendayi Achiume, Report of the Special Rapporteur on Contemporary Forms of Racism, Racial Discrimination, Xenophobia and Related Intolerance, UN Doc A/HRC/44/57 (18 June 2020) [6].
[iii] The UN Special Rapporteur on Freedom of Opinion and Expression, the Organization for Security and Cooperation in Europe Representative on Freedom of the Media, the Organization of American States Special Rapporteur on Freedom of Expression and the African Commission on Human and People’s Rights Special Rapporteur on Freedom of Expression and Access to Information, Twentieth Anniversary Joint Declaration: Challenges to Freedom of Expression in the Next Decade (10 July 2019).
[iv] UN Office of the High Commissioner for Human Rights, ‘World Stumbling Zombie-like Into a Digital Welfare Dystopia, Warns UN Human Rights Expert’ (Media Release, 17 October 2019).
[v] See, for example, Mark Latonero, Governing Artificial Intelligence: Upholding Human Rights & Dignity (Data & Society, October 2018); Mark Hodge and Dunstan Allison-Hope, Artificial Intelligence: A Rights-Based Blueprint for Business (Working Paper No 2, BSR, August 2018); Filippo A Raso, Hannah Hilligoss, Vivek Krishnamurthy, Christopher Bavitz and Levin Kim, Artificial Intelligence & Human Rights: Opportunities & Risks (Berkman Klein Center Research Publication No 2018-6, September 2018).
[vi] NSW Government, ‘First NSW AI Strategy to Make Life Easier for Citizens’ (Media Release, 4 September 2020).
[vii] Committee of Experts on Internet Intermediaries (MSI-NET), Algorithms and Human Rights: Study on the Human Rights Dimensions of Automated Data Processing Techniques and Possible Regulatory Implications (Council of Europe Study, DGI/2017/12, March 2018) 28, 29. See also Crofton Black and Cansu Safak, Government Data Systems: The Bureau Investigates (The Bureau of Investigative Journalism, 8 May 2019).
[viii] ICESCR art 9; UDHR art 22. Article 25 of the UDHR provides for the right to an adequate standard of living, that includes ‘necessary social services, and the right to security in the event of unemployment, sickness, disability, widowhood, old age or other lack of livelihood in circumstances beyond his control’. The right to security has been incorporated into other human rights treaties, including ICESCR art 9; ICERD art 5(e)(iv); CEDAW art 11(1)(e), 14(2)(c); CRC art 26.
[ix] It has been argued that the use of AI in welfare systems can reinforce racially discriminatory structures: see E Tendayi Achiume, Report of the Special Rapporteur on Contemporary Forms of Racism, Racial Discrimination, Xenophobia and Related Intolerance, UN Doc A/HRC/44/57 (18 June 2020) [41]-[43].
[x] ICCPR art 2(3) requires each State Party to ensure a person whose Covenant rights have been violated has an effective remedy, and that this remedy will be enforced. See also UN Human Rights Committee, General Comment No 31 [80]: The Nature of the General Legal Obligation Imposed on States Parties to the Covenant, 80th sess, UN Doc CCPR/C/21/Rev/1/Add/13 (26 May 2004, adopted 29 March 2004).
[xi] UN Committee on Economic, Social and Cultural Rights, General Comment No 19: The Right to Social Security (Art. 9), 39th sess, UN Doc E/C.12/GC/19 (4 February 2008, adopted 3 November 2007) para 24.
[xii] UN Committee on Economic, Social and Cultural Rights, General Comment No 19: The Right to Social Security (Art. 9), 39th sess, UN Doc E/C.12/GC/19 (4 February 2008, adopted 3 November 2007) para 24.
[xiii] DP submissions: T Aulich, 2; Ethics Matters Pty Ltd, 2; Community and Public Sector Union, 2; Victoria Legal Aid, 1; The Australia Institute’s Centre for Responsible Technology, 4, 5; WiseLaw, 4; J Thornton; Carers Australia, 4; Digital Rights Watch, 4-5. IP submission: Commonwealth Ombudsman, 4. Senate Community Affairs References Committee, Parliament of Australia, Design, Scope, Cost-benefit Analysis, Contracts Awarded and Implementation Associated with the Better Management of the Social Welfare System Initiative (June 2017).
[xiv] Commonwealth Ombudsman, Centrelink’s Automated Debt Raising and Recovery System: A Report About the Department of Human Services’ Online Compliance Intervention System for Debt Raising and Recovery (April 2017).
[xv] Errors and discrepancies arose when an assumption was made about income, and, consequently, incorrect information was included in the OCI’s calculation: see Senate Community Affairs References Committee, Parliament of Australia, Design, Scope, Cost-benefit Analysis, Contracts Awarded and Implementation Associated with the Better Management of the Social Welfare System Initiative (June 2017) ch 2.85 – 2.101. See also Senate Community Affairs References Committee, Parliament of Australia, Centrelink’s Compliance Program (Second Interim Report, September 2020).
[xvi] See, for example, Australian Council of Social Service, Submission No 31 to the Senate Community Affairs References Committee, Inquiry into the Design, Scope, Cost-benefit Analysis, Contracts Awarded and Implementation Associated with the Better Management of the Social Welfare System Initiative (21 March 2017).
[xvii] DP submission: Victoria Legal Aid, 2.
[xviii] Luke Henriques-Gomes, ‘Robodebt: Government to Refund 470,000 Unlawful Centrelink Debts Worth $721m’, The Guardian (online, 29 May 2020) <https://www.theguardian.com/australia-news/2020/may/29/robodebt-government-to-repay-470000-unlawful-centrelink-debts-worth-721m>; Jordan Hayne and Matthew Doran, ‘Government to Pay Back $721m as it Scraps Robodebt for Centrelink Welfare Recipients’, ABC News (online, 29 May 2020) <https://www.abc.net.au/news/2020-05-29/federal-government-refund-robodebt-scheme-repay-debts/12299410>.
[xix] Matthew Doran, ‘Federal Government Ends Robodebt Class Action With Settlement Worth $1.2 Billion’, ABC News (online, 16 November 2020) <https://www.abc.net.au/news/2020-11-16/government-response-robodebt-class-action/12886784>; Paul Karp, ‘Robodebt Class Action to go Ahead Despite Overhaul of Centrelink Debt Recovery’, The Guardian (online, 20 November 2019) <https://www.theguardian.com/australia-news/2019/nov/20/robodebt-class-action-to-go-ahead-despite-overhaul-of-centrelink-debt-recovery>. See also: Gordon Legal, Robodebt Class Action <https://gordonlegal.com.au/robodebt-class-action/>; ‘Centrelink Robodebt Class Action Lawsuit to be Brought Against Federal Government’, ABC News (online, 17 September 2019) <https://www.abc.net.au/news/2019-09-17/centrelink-robodebt-class-action-lawsuit-announced/11520338>.