European Union Law and Mitigation of Artificial Intelligence-Related Discrimination Risks in the Private Sector: With Special Focus on the Proposed Artificial Intelligence Act

Abstract

Integrating AI systems into decision-making processes in the private sector may place the right to non-discrimination in danger. In order to illustrate this threat, risky fields in the private sector, namely employment, banking, advertising, pricing and insurance, were investigated in this paper with authentic examples of AI-related discrimination. Then, the current EU non-discrimination laws and data protection laws were examined, and it was found out that these EU laws do not have the necessary tools to tackle specific risks arising from AI-related discrimination in the private sector. Therefore, there is an immediate need for new EU legislation equipped with tools which explicitly target AI-related discrimination risks in the private sector. The proposed AI Act may provide new tools against AI-related discrimination in the private sector. Thus, this paper analyzes the proposed AI Act in terms of its potential impacts on mitigating AI-related discrimination risks. Due to the cradle-to-grave approach adopted by the proposed AI Act, providers and users of high-risk AI systems are required to comply with various specific ex-ante and ex-post obligations. It is found out that these obligations can contribute to the mitigation of AI-related discrimination by providing new legal tools. However, these tools are not sufficient in the face of AI-related discrimination risks. Therefore, it is concluded that the crucial need for specific legislation to mitigate AI-related discrimination risks in the private sector is still present.

Keywords

• Artificial Intelligence
• discrimination
• AI-related discrimination
• European Union law
• non-discrimination laws
• data protection law
• European Union Artificial Intelligence Act Proposal

References

1. Accesswire, ‘72% of Resumes are Never Seen by Employers’ (16 February 2016) accessed 20 June 2021. google scholar
2. Ali M et al., ‘Discrimination Through Optimization: How Facebook’s ad Delivery Can Lead to Skewed Outcomes’ (2019) Proceedings of the ACM on Human-Computer Interaction. accessed 20 June 2021. google scholar
3. Angelopoulos C et al., ‘Study Of Fundamental Rights Limitations For Online Enforcement Through Self-Regulation’ (Study Report of Institute for Information Law, University of Amsterdam) (2016) accessed 20 June 2021. google scholar
4. Angwin J, Mattu S and Larson J, ‘The tiger mom tax: Asians are nearly twice as likely to get a higher price from Princeton Review’. ProPublica (1 September 2015) accessed 20 June 2021. google scholar
5. Angwin J, Scheiber N and Tobin A, ‘Dozens of companies are using Facebook to exclude older workers from job ads’ ProPublica (20 December 2017) accessed 20 June 2021. google scholar
6. Angwin J, Tobin A and Varner M, ‘Facebook (still) letting housing advertisers exclude users by race’ ProPublica (21 November 2017) accessed 20 June 2021. google scholar
7. Avraham R, ‘Discrimination and Insurance’ in Lippert-Rasmussen K. (ed.), The Routledge Handbook of the Ethics of Discrimination, (Routledge, UK 2018), pp. 335-347. google scholar
8. Bartlett R et al., ‘Consumer-Lending Discrimination in the FinTech Era’ (2019) (National Bureau of Economic Research Working Papers 25943). google scholar

9. Barocas S and Selbst A D, ‘Big Data’s Disparate Impact’ (2016) 104 California Law Review, 671-732. google scholar
10. Bathaee Y, ‘The Artificial Intelligence Black Box And The Failure Of Intent And Causation’ (2018) 31:2 Harvard Journal of Law & Technology, 890-938. google scholar
11. Beck S et al., ‘Künstliche Intelligenz und Diskriminierung Herausforderungen und Lösungsansatze’ (2019) (Whitepaper from Plattform Lernende Systeme) accessed 20 June 2021. google scholar
12. Biao v. Denmark (Grand Chamber) (App no 38590/10) ECHR 24 May 2016. google scholar
13. Bodo B et al., ‘Tackling the algorithmic control crisis-the technical, legal, and ethical challenges of research into algorithmic agents’ (2017) 19 Yale Journal of Law & Technology, 133-180. google scholar
14. Borgesius F Z, ‘Discrimination, Artificial Intelligence and Algorithmic Decision-making’ (Study Report) (Council of Europe, Strasbourg 2018) accessed 20 June 2021. google scholar
15. Borgesius F J Z, ‘Strengthening legal protection against discrimination by algorithms and artificial intelligence’ (2020) 24 10 The International Journal of Human Rights, 1572- 1593. google scholar
16. Bundesanstalt für Finanzdienstleistungsaufsicht, ‘Big data meets artificial intelligence: Challenges and implications for the supervision and regulation of financial services’ (2018) (Study) www.bafin.de/SharedDocs/Downloads/EN/dl_bdai_studie_en.html> accessed 20 June 2021. google scholar
17. Burrell J, ‘How the machine ‘thinks’: understanding opacity in machine learning algorithms’ (2016) Big Data & Society Doi:10.1177/2053951715622512. google scholar
18. Case 170/84, Bilka-Kaufhaus GmbH and Karin Weber von Hartz, (Judgment of the Court of 13 May 1986.) google scholar
19. Case C-83/14, “CHEZ Razpredelenie Bulgaria” AD v Komisia za zashtita ot diskriminatsia, (Judgment of the Court (Grand Chamber) of 16 July 2015). google scholar
20. Case C443/15, David L. Parris v Trinity College Dublin and Others (Judgment of the Court (First Chamber) of 24 November 2016). google scholar
21. Case C-414/16, Vera Egenberger v Evangelisches Werk für Diakonie und Entwicklung e.V. (Judgment of the Court (Grand Chamber) of 17 April 2018). google scholar
22. Chander S and Jakubowska E, ‘EU’s AI law needs major changes to prevent discrimination and mass surveillance’ EDRI (2021) accessed 20 June 2021. google scholar
23. Charter of Fundamental Rights of the European Union, 2012/C 326/02 OJ C 326, [2012]. google scholar
24. Commission (EC) ‘Proposal for a Regulation Of The European Parliament And Of The Council Laying Down Harmonised Rules On Artificial Intelligence (Artificial Intelligence Act) And Amending Certain Union Legislative Acts’ COM(2021) 206 final, 21 April 2021. google scholar
25. Corner N, ‘Revealed: How your name, your ethnicity, and even your e-mail address could be adding hundreds of pounds to your car insurance premium’ Dailymail (29 June 2018) www.dailymail.co.uk/femail/article-5896561/How-email-address-add-hundreds-car-insurance-premium.html> accessed 20 June 2021. google scholar
26. Council Directive (EC) 2000/43 of 29 June 2000 implementing the principle of equal treatment between persons irrespective of racial or ethnic origin [2000] OJ L 180. google scholar
27. Council Directive (EC) 2000/78 of 27 November 2000 establishing a general framework for equal treatment in employment and occupation [2000] OJ L 303. google scholar
28. Council Directive (EC) 2001/95 of the European Parliament and of the Council of 3 December 2001 on general product safety [2001] OJ L 11. google scholar
29. Council of Europe, ‘Explanatory Report to the Protocol No. 12 to the Convention for the Protection of Human Rights and Fundamental Freedoms’ (2000) accessed 20 June 2021. google scholar
30. Council of Europe, ‘Algorithms and Human Rights: Study On The Human Rights Dimensions Of Automated Data Processing Techniques (In Particular Algorithms) And Possible Regulatory Implications’ (Study Report) (DGI(2017)12) (2018). google scholar
31. Council Regulation (EC) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing Directive 95/46/EC (General Data Protection Regulation) [2016] OJ L 119. google scholar
32. Craglia M et al., ‘Artificial Intelligence: A European Perspective’ (Joint Research Centre Working Papers) (2018). google scholar
33. Danks D and London A J, ‘Algorithmic Bias in Autonomous Systems’ (2017) IJCAI’17: Proceedings of the 26th International Joint Conference on Artificial Intelligence, 4691-4697. google scholar
34. Dastin J, ‘Amazon scraps secret AI recruiting tool that showed bias against women’ Reuters (11 October 2018) accessed 20 June 2021. google scholar
35. Data Protection Working Party Guidelines on Automated individual decision-making and Profiling for the purposes of Regulation 2016/679 Wp251rev.01. [2018]. google scholar
36. Datta A, Tschantz M C and Datta A, ‘Automated Experiments on Ad Privacy Settings A Tale of Opacity, Choice, and Discrimination’ (2015) Proceedings on Privacy Enhancing Technologies accessed 20 June 2021. google scholar
37. Delipetrev B, Tsinarakii C and Kostic U, ‘Historical Evolution of Artificial Intelligence’ (Technical Report) (Publications Office of the European Union, Luxembourg 2020) https://publications.jrc. ec.europa.eu/repository/handle/JRC120469 accessed 20 June 2021. google scholar
38. Dwork C et al., ‘Fairness Through Awareness, Proceedings of the 3rd Innovations in Theoretical Computer Science Conference’ (2012) ITCS ‘12 (Association for Computing Machinery), 214226. google scholar
39. Edwards, L and Veale M, ‘Slave to the algorithm: Why a right to an explanation is probably not the remedy you are looking for’ (2017) 16 Duke Law & Technology Review, 18-84. google scholar
40. Espinoza J and Murgia M, ‘Europe attempts to take leading role in regulating uses of AI’ Financial Times (24 April 2021) accessed 20 June 2021. google scholar
41. European Commission, Proposal for a Council Directive on implementing the principle of equal treatment between persons irrespective of religion or belief, disability, age or sexual orientation [2008] {SEC(2008) 2180} {SEC(2008) 2181} /* COM/2008/0426 final - CNS 2008/0140 */. google scholar
42. European Commission, ‘Special Eurobarometer Discrimination in the EU in 2015’ (Report) (2015). google scholar
43. European Commission’s High-Level Expert Group on Artificial Intelligence, ‘A Definition of AI: Main Capabilities and Disciplines’ (2019) (Report) accessed 20 June 2021. google scholar
44. European Commission, ‘A Union of Equality: Gender Equality Strategy 2020-2025’ [2020] COM (2020) 152. google scholar
45. European Commission, ‘Internal Market, Industry, Entrepreneurship and SMEs’ accessed 20 June 2021. google scholar
46. European Commission, ‘Harmonised Standards’ accessed 20 June 2021. google scholar
47. European Convention for the Protection of Human Rights and Fundamental Freedoms (1950) accessed 20 June 2021. google scholar
48. European Union Agency for Fundamental Rights, ‘#BigData: Discrimination in data-supported decision making’ (Report) (2018) Doi:10.2811/343905. google scholar
49. European Union Agency for Fundamental Rights, ‘Getting the Future Right-Artificial Intelligence and Fundamental Rights’ (2020) Doi:10.2811/774118. google scholar
50. Fairness, Accountability, and Transparency in Machine Learning accessed 20 June 2021. google scholar
51. Favaretto M, De Clercq E and Elger B S, ‘Big Data and discrimination: perils, promises and solutions. A systematic review’ (2019) 6 Journal of Big Data accessed 20 June 2021. google scholar
52. Financial Conduct Authority, ‘Pricing practices in the retail general insurance sector: Household insurance’ (2018) accessed 20 June 2021. google scholar
53. Stan F, ‘History, Motivations, and Core Themes’ in Keith Frankish and William W. Ramsey, (eds.), The Cambridge Handbook Of Artificial Intelligence (Cambridge University Press, UK 2014). google scholar
54. Fuster A et al., ‘Predictably Unequal? The Effects of Machine Learning on Credit Markets’ (2020). SSRN accessed 20 June 2021. google scholar
55. Future of Life Institute, ‘The Asilomar AI principles’ (2017) accessed 20 June 2021. google scholar
56. Gillis T B and Spiess J L, ‘Big Data and Discrimination’ (2019) 86 The University of Chicago Law Review, pp. 459-487. google scholar
57. Hacker P., ‘Teaching Fairness to Artificial Intelligence: Existing and Novel Strategies against Algorithmic Discrimination under EU Law’ (Pre-print version) (2019) accessed 20 June 2021. google scholar
58. Heinrichs B, ‘Discrimination in the age of artificial intelligence’ (2021) AI & Society Springer. accessed 20 June 2021. google scholar
59. Hildebrandt M and Gutwirth S, ‘Concise Conclusions: Citizens Out of Control’ in Hildebrandt M and Gutwirth S (eds.) Profiling the European Citizen: Cross-Disciplinary Perspectives (Springer, 2008). google scholar
60. Hill K, ‘How Target Figured Out A Teen Girl Was Pregnant Before Her Father Did’ Forbes (16 February 2012) accessed 20 June 2021. google scholar
61. Humphreys P, ‘The philosophical novelty of computer simulation methods’ (2008) 169 Synthese, pp. 615-626 accessed 20 June 2021. google scholar
62. International Association of Insurance Supervisors, ‘Issues Paper on the Use of Big Data Analytics in Insurance’ (2020) accessed 20 June 2021. google scholar
63. International Covenant on Civil and Political Rights adopted and opened for signature, ratification and accession by General Assembly resolution 2200A (XXI) of 16 December 1966 entry into force 23 March 1976. google scholar
64. Izzidien R, ‘Higher insurance if you’re called Mohammed? That’s just the start of institutionalized Islamophobia’ NewStatesman (23 January 2018) accessed 20 June 2021. google scholar
65. Johnson K, Pasquale F and Chapman J, ‘Artificial Intelligence, Machine Learning, and Bias in Finance: Toward Responsible Innovation’ (2019) 88 2 Fordham Law Review 499-529. google scholar
66. Jones H, ‘When the wrong postcode puts insurance out of your reach’ The Guardian (30 October 2016) accessed 20 June 2021. google scholar
67. Kelion L, ‘Apple’s ‘sexist’ credit card investigated by US regulator’ BBC (11 November 2019) accessed 20 June 2021. google scholar
68. Kim P T, ‘Data-Driven Discrimination at Work’ (2017) 58 3 William & Mary Law Review, pp. 857-936. google scholar
69. Kleinberg J et al., ‘Discrimination in the Age of Algorithms’ (2018) 10 Journal of Legal Analysis pp. 113-174. google scholar
70. Kloza D et al., ‘Data protection impact assessments in the European Union: complementing the new legal framework towards a more robust protection of individuals’ (d.pia.lab Policy Brief) (2017) accessed 20 June 2021. google scholar
71. Knutson T, ‘AI Lending Discrimination Needs To Be Tackled With Legislation Says House Financial Services Chair’ Forbes (7 May 2021) accessed 20 June 2021. google scholar
72. Kretchmer H, ‘Insurers ‘risk breaking racism laws’’ BBC (9 February 2018) accessed 20 June 2021. google scholar
73. Kroll J A et al., ‘Accountable Algorithms’ (2017) 165 University of Pennsylvania Law Review, 633-705. google scholar
74. Lacroix C, ‘Preventing discrimination caused by the use of artificial intelligence’ (Report of Committee on Equality and Non-Discrimination at Council of Europe) (2020). google scholar
75. Lambrecht A and Tucker C E, ‘Algorithmic Bias? An Empirical Study of Apparent Gender-Based Discrimination in the Display of STEM Career Ads’ (2019) 65 7 Management Science, 2966-2981. google scholar
76. Liboreiro J, ‘The higher the risk, the stricter the rule’: Brussels’ new draft rules on artificial intelligence’ Euronews (21.4.2021) accessed 20 June 2021. google scholar
77. Loi M and Christen M, ‘Choosing how to discriminate: navigating ethical trade-offs in fair algorithmic design for the insurance sector’ (2021) Philosophy & Technology accessed 20 June 2021. google scholar
78. Lucci S and Kopec D, Artificial Intelligence in the 21st Century: A Living Introduction (Mercury Learning and Information, USA 2016). google scholar
79. MacCarthy M and Propp K, ‘Machines Learn That Brussels Writes the Rules: The EU’s New AI Regulation’ Lawfare Blog (28 April 2021) accessed 20 June 2021. google scholar
80. Makinen J, ‘The EU’s AI Regulation Proposal - How to Fix It?’ (2021) accessed 20 June 2021. google scholar
81. Meneceur Y, ‘European Commission’s AI regulation proposal: between too much and too little?’ LinkedIn (23 April 2021) accessed 20 June 2021. google scholar
82. Modernised Convention for the Protection of Individuals with Regard to the Processing of Personal Data CM/Inf(2018)15-final [2018]. google scholar
83. Möslein F, ‘Robots in the Boardroom: Artificial Intelligence and Corporate Law’ in Barfield W and Pagallo U (eds.), Research Handbook on the Law of Artificial Intelligence (Edward Elgar, UK 2018) 649-669. google scholar
84. Newell S and Marabelli M, ‘Strategic opportunities (and challenges) of algorithmic decision-making: A call for action on the long-term societal effects of ‘datification’’ (2015) 24 Journal of Strategic Information Systems, 3-15. google scholar
85. Peck D, ‘They’re Watching You at Work’ Atlantic (December 2013) accessed 20 June 2021. google scholar
86. Peppet S R, ‘Regulating the Internet of Things: first steps toward managing discrimination, privacy, security, and consent’ (2014) 93 Texas Law Review, 85-176. google scholar
87. Pettinger-Kearney N et al., ‘The EU’s proposed AI Regulation’ (2021) accessed 20 June 2021. google scholar
88. Pope D G and Sydnor J R, ‘Implementing Anti-Discrimination Policies in Statistical Profiling Models’ (2011) 3 American Economic Journal: Economic Policy, 206-231. google scholar
89. Prince A E R and Schwarcz D, ‘Proxy Discrimination in the Age of Artificial Intelligence and Big Data’ (2020) 105 Iowa Law Review, 1257-1318. google scholar
90. Probasco J, ‘The Insurance Industry Confronts Its Own Racism’ Investopedia (1 September 2020) accessed 20 June 2021. google scholar
91. Reinhold F and Müller A, ‘Algorithm Watch’s response to the European Commission’s proposed regulation on Artificial Intelligence - a major step with major gaps’ (Report) (22 April 2021) accessed 20 June 2021. google scholar
92. Rieke A, Bogen M and Robinson D G, ‘Public scrutiny of automated decisions: Early lessons and emerging methods’ (Upturn and Omidyar Network Report) (2018) accessed 20 June 2021. google scholar
93. Rodrigues R, ‘Legal and human rights issues of AI: Gaps, challenges and vulnerabilities’ (2020) 4 Journal of Responsible Technology accessed 20 June 2021. google scholar
94. Samoili S et al., ‘AI Watch. Defining Artificial Intelligence. Towards An Operational Definition And Taxonomy Of Artificial Intelligence’ (Joint Research Centre Technical Reports) (Luxembourg 2020). google scholar
95. Schauer F, ‘Statistical (and non-statistical) discrimination’ in Kasper Lippert-Rasmussen (ed.), The Routledge Handbook of the Ethics of Discrimination, (Routledge, UK 2018), 42-53. google scholar
96. Swedloff R, ‘Risk classification’s Big Data (r)evolution’ (2014) 21 Connecticut Insurance Law Journal, 339-373. google scholar
97. The European Consumer Organization, ‘Automated Decision Making And Artificial Intelligence - A Consumer Perspective’ (Report) (2018) accessed 20 June 2021. google scholar
98. The European Consumer Organization, ‘AI Rights for Consumers’ (Report) (2019) accessed 20 June 2021. google scholar
99. The European Consumer Organization, ‘The Use Of Big Data And Artificial Intelligence In Insurance’ (Report) (2020) accessed 20 June 2021. google scholar
100. The European Consumer Organization, ‘EU proposal for artificial intelligence law is weak on consumer protection’ (Report) (2021) accessed 20 June 2021. google scholar
101. The Federal Government of Germany, ‘Artificial Intelligence Strategy’ (2018) accessed 20 June 2021. google scholar
102. The Montreal Declaration for Responsible AI (2017) accessed 20 June 2021. google scholar
103. The Partnership on AI to Benefit People and Society accessed 20 June 2021. google scholar
104. The Royal Society, ‘Machine Learning: The Power And Promise Of Computers That Learn By Example’ (Report) (April 2017) accessed 20 June 2021. google scholar
105. The Royal Society, ‘Explainable AI: the basics’ (Policy briefing) (2019) accessed 20 June 2021. google scholar
106. The White House, ‘Big Data: Seizing Opportunities, Preserving Values’ (Report) (2014). google scholar
107. Universal Declaration of Human Rights, UNGA Res 217/A (10 December 1948). google scholar
108. Valentino-Devries J, Singer-Vine J, and Soltani A, ‘Websites vary prices, deals based on users’ information’ Wall Street Journal (24 December 2012) accessed June 2021. google scholar
109. Vigdor N, ‘Apple Card Investigated After Gender Discrimination Complaints’ The New York Times (10 November 2019) accessed June May 2021. google scholar
110. Wachter S, Mittelstadt B and Russell C, ‘Counterfactual explanations without opening the black box: automated decisions and the GDPR’ (2018) 31 2 Harvard Journal of Law & Technology, 842-887. google scholar
111. Wachter S, Mittelstadt B and Russell C, ‘Why Fairness Cannot Be Automated: Bridging The Gap Between EU Non-Discrimination Law And AI’ (Draft) (2020) accessed 20 June 2021. google scholar
112. Wagner B et al., ‘Algorithms and human rights. Study on the human rights dimensions of automated data processing techniques and possible regulatory implications’ (Study Report) (DGI(2017)12) (2018) accessed 20 June 2021. google scholar
113. Wahlster W and Winterhalter C, ‘German Standardization Roadmap On Artificial Intelligence’ (German Institute for Standardization and DKE German Commission for Electrical, Electronic & Information Technologies of DIN and VDE November 2020) accessed 20 June 2021. google scholar
114. Walch K, ‘Is There A Difference Between Assisted Intelligence Vs. Augmented Intelligence?’ Forbes. (12 January 2020) accessed 20 June 2021. google scholar
115. Wang P, ‘On the Working Definition of Intelligence’ (1995) accessed 20 May 2021. google scholar
116. Weisbaum H, ‘Data Mining Is Now Used to Set Insurance Rates; Critics Cry Foul’ CNBC (16 April 2014) accessed 20 June 2021. google scholar
117. Whitehead D, ‘AI & Algorithms (Part 3): Why the EU’s AI regulation is a ground breaking proposal’ Hogan Lovells Engage (3 May 2021) accessed 20 June 2021. google scholar
118. Williams B et al., ‘How Algorithms Discriminate Based on Data they Lack: Challenges, Solutions, and Policy Implications’ (2018) 8 Journal of Information Policy, 78-115. google scholar
119. World Economic Forum, ‘Transforming Paradigms A Global AI in Financial Services Survey’ (2020) (Report) accessed 20 June 2021. google scholar
120. Xenidis R, ‘Tuning EU equality law to algorithmic discrimination: Three pathways to resilience’ (2020) 27(6) Maastricht Journal of European and Comparative Law, 736-758. google scholar
121. Xenidis R and Senden L, ‘EU non-discrimination law in the era of artificial intelligence: Mapping the challenges of algorithmic discrimination’ in Ulf Bernitz et al. (eds.), General principles of EU law and the EU digital order (Kluwer Law International 2020), 151-182. google scholar