Deepfake Dilemmas: Imagine Tomorrow’s Surveillance Society through Three Scenarios

Abstract

As deepfakes—hyperrealistic, manipulated media content becomes increasingly sophisticated, they raise critical questions about the future of social interaction and trust. Drawing on sociological insights enriched by the literature on virtual reality surveillance, this study explores the social landscapes that may emerge with the widespread adoption of deepfake avatars in the near future. Moving beyond technical considerations, the research employs social prediction methods to explore three potential future scenarios involving deepfake avatars: seamless integration that facilitates self-expression but complicates digital identity control; pervasive algorithmic surveillance that exacerbates social control and privacy issues; and the concept of ghost avatars, that is, digital representations that function as commodities or tools of resistance. This analysis highlights the ethical dilemmas surrounding deepfakes, particularly in relation to mass surveillance, digital self-control, and resistance movements. Contributing to scholarly discourse, the study underscores the need for proactive social and ethical frameworks to navigate the evolving deepfake landscape and highlights the importance of ongoing research for the responsible development and use of this powerful technology.

Keywords

• deepfake technology
• Artificial Intelligence (AI)
• virtual surveillance
• digital society
• future scenarios

References

1. Adams, S. (2013). Post-panoptic surveillance through healthcare rating sites: Who’s watching whom?. Information, Communication & Society, 16(2), 215-235. google scholar
2. Albrechtslund, A. (2008). Online social networking as participatory surveillance. First Monday, 13(3). https://doi.org/10.5210/fm.v13i3.2142 google scholar
3. Aldridge, A. (1999). Prediction in sociology: Prospects for a devalued activity. Sociological Research Online, 4(3), 57-62. Algorithmic Justice League (n.d). https://www.ajl.org/ google scholar
4. Angell, M. (2020, Feb 9). How real is the deepfake threat? American Banker. Retrieved from https://www.americanbanker.com/creditunions/news/how-real-is-the-deepfake-threat google scholar
5. Attestiv (2021, May) Deepfakes: The Business threat: Inagural deepfakes in business survey results. Retrieved from https://f.hubspotusercontent30.net/hubfs/7603739/deepfake_report2021.pdf?__hstc=143725418.1879e4f87f3af9d371de4bc1807298ac. 1724673569402.1724673569402.1724673569402.1&__hssc=143725418.1.1724673569402&__hsfp=1308513962&hsCtaTracking= 2eea443f-cbd5-4229-a7c2-5237a703046d%7Cdab0da4f-7ddd-4586-b257-5fce836345bd google scholar
6. Bardzell, J. (2014). Social media and the making of global imaginaries. Information, Communication & Society, 17(3), 329-347. google scholar
7. Barfield, W. (2015). Cyber-Humans: Our Future with Machines. Copernicus Cham Springer. google scholar
8. Bassett, D. J. (2018). Digital afterlifes: From social media platforms to thanabots and beyond. In C. Tandy (Ed.), Death and anti-death, Volume 16: Two hundred years after Frankenstein. (pp. 27-38). Ria University Press. google scholar

9. Bentham, J., (1995). The Panopticon Writings. Bozovic, Miran (Ed.),. London, UK: Verso. google scholar
10. Benzinga (2022, May 14). Apple Vs. Google: Which one is winning the augmented reality race? Business Insider. Retrieved from https://markets.businessinsider.com/news/stocks/apple-vs-google-which-one-is-winning-the-augmented-reality-race-1031460242 google scholar
11. Bigo, D. (2006). Security, exception, ban and surveillance. In D. Lyon (Ed.), Theorising surveillance: The panopticon and beyond (pp. 46-68). Portland: Willan Publishing. google scholar
12. Björklund, F. (2021). Trust and surveillance: An odd couple or a perfect pair?. (Eds)cViola, L. A., & Laidler, P. (2022). In Trust and Transparency in an Age of Surveillance (pp. 183-200). Routledge. google scholar
13. Blodgett, B., & Tapia, A. (2011). Do avatars dream of electronic picket lines?: The blurring of work and play in virtual environments, Information Technology & People, 24 (1), 26-45. google scholar
14. Bostrom, N. (2014). Superintelligence: Paths, dangers, strategies. Oxford University Press. google scholar
15. Boyd, D., (2014). It’s complicated: The social lives of networked teens. Yale University Press. google scholar
16. Brynjolfsson, E., & McAfee, A. (2011). Race against the machine: How the Digital Revolution is Accelerating Innovation, Driving Productivity, and Irreversibly Transforming Employment and the Economy. Lexington, VA: Beard Books. google scholar
17. Busch, E., Ware, J., & Brief, I. P. (2023, December). The weaponisation of deepfakes. International center for counter-terrorism. Retrieved from https://www.icct.nl/sites/default/files/2023-12/The%20Weaponisation%20of%20Deepfakes.pdf google scholar
18. Chen, Y., Wu, X., Hu, A. He, G., & Ju, G. (2021). Social prediction: a new research paradigm based on machine learning. Journal of Chinese Sociology. 8 (15). google scholar
19. Cheney-Lippold, J. (2011). A new algorithmic identity: Soft biopolitics and the modulation of control. Theory, Culture and Society, 28(6), pp. 164-181. google scholar
20. Clarke, R. (1988). Information technology and dataveillance. Communications of the ACM, 31(5), 498-512. google scholar
21. Crawford, K. (2021). Atlas of AI: Power, politics, and the planetary costs of artificial intelligence. Yale University Press. google scholar
22. Dastin, J. (2018, Oct 11). Amazon scraps a secret AI recruiting tool that showed bias against women. Retrieved from https://www.reuters.com/article/us-amazon-comjobs-automation-insight-idUSKCN1MK08G/ google scholar
23. De Rancourt-Raymond, A. and Smaili, N. (2023). The unethical use of deepfakes. Journal of Financial Crime, 30(4), 1066-1077. google scholar
24. Defleur, L. B. (1982). Technology, social change, and the future of sociology. The Pacific Sociological Review, 25 (4), 403-417. google scholar
25. Deleuze, G., and Guattari, F. (1987). A thousand plateaus: capitalism and schizophrenia. Minneapolis: University of Minnesota Press. google scholar
26. Deleuze, G. (1992). Postscript on the societies of control. October 59, 3-7. http://www.jstor.org/stable/778828 google scholar
27. Dufva, T., & Dufva, M. (2019). Grasping the future of the digital society. Futures, 107, 17-28. google scholar
28. Fagni, T., Falchi, F., Gambini, M., Martella, A., & Tesconi, M. (2021). TweepFake: About detecting deepfake tweets. Plos One, 16(5). Retrieved from https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0251415 google scholar
29. Fourcade, M., & Healy, K. (2017). Seeing like a market. Socio-economic Review, 15(1), 9-29. google scholar
30. Foucault, M., (1977). Discipline and punish: The birth of the prison. Vintage, New York. google scholar
31. Galic, M., Timan, T., & Koops, B.-J. (2017). Bentham, Deleuze and Beyond: An overview of surveillance theories from the panopticon to participation. Philosophy & Technology, 30(1), 9-37. google scholar
32. Gamage, D., Chen, J., & Sasahara, K. (2021, Oct 7). The emergence of deepfakes and their societal implications: A systematic review. Paper presented at Truth and Trust International Conference. Retrieved from https://truthandtrustonline.com/wp-content/uploads/2021/10/TTO2021_paper_20.pdf google scholar
33. Gibson, A. (2021, May 17). Keeping It Real: Historians in the Deepfake Era. Perspectives on History. Retrieved from https://www.historians.org/perspectives-article/keeping-it-real-historians-in-the-deepfake-era-may-2021/ google scholar
34. Gissen, L. (2022, Nov 21). Talented impressionist morphs into twenty different celebrities using jaw-dropping deepfake technology... So, can you work out who they all are of them? Mail Online. Retrieved from https://www.dailymail.co.uk/femail/article-11445427/Impressionist-transforms-celebrities-using-wild-deepfake-technology.html google scholar
35. Greengard, S. (2019). Will deepfakes do deep damage?. Communications oftheACM, 63(1), 17-19. google scholar
36. Giusti, S., & Piras, E. (Eds.). (2020). Democracy and fake news: information manipulation and post-truth politics. Routledge. google scholar
37. Gonzalez, S. M., & Deckard, F. M. (2022). We Got Witnesses: Black women’s counter-surveillance for navigating police violence and legal estrangement. Social Problems. 71 (3), 894-911. google scholar
38. Hadan, H., Choong, L., Zhang-Kennedy, L., & Nacke, L. E. (2024). Deceived by immersion: A systematic analysis of deceptive design in extended reality. ACM Computing Surveys, 56(10), 1-25. google scholar
39. Haggerty, K. D., & Ericson, R. V. (1999). Governing the young. In R. Smandych (ed.) Governable Places: Readings on Governmentality and Crime Control, Dartmouth, UK: Ashgate. google scholar
40. Haggerty, K.D, & Ericson, R.V. (2000) The surveillant assemblage. British Journal of Sociology. 51(4), 605-622. google scholar
41. Haggerty, K. (2006). Tear down the walls: on demolishing the panopticon. In D. Lyon (Ed.), Theorising surveillance: The panopticon and beyond (pp. 23-45). Portland: Willan Publishing. google scholar
42. Hancock, J. T, Naaman, M., & Levy, K. (2020) AI-mediated communication: Definition, research agenda, and ethical considerations. Journal of Computer-Mediated Communication. 25(1). 89-10. https://doi.org/10.1093/jcmc/zmz022 google scholar
43. Hancock, J. T., & Bailenson, J. N. (2021). The social impact of deepfakes. Cyberpsychology, Behavior, and Social Networking. 24(3).149-152. http://doi.org/10.1089/cyber.2021.29208.jth google scholar Henshel, R. L. (1982). Sociology and social forecasting. Annual Review of Sociology, 8(1), 57-79. google scholar
44. Hermida, A., & Hernandez-Santaolalla, V. (2018). Twitter and video activism as tools for counter-surveillance: The case of social protests in Spain. Information, Communication & Society, 21(3), 416-433 google scholar
45. Hofman, J.M., A. Sharma, &D.J. Watts. (2017). Prediction and explanation in social systems. Science. 355(6324). 486-488. google scholar
46. Horowitz, M. C., Allen, G. C., Kania, E. B., & Scharre, P. (2022, Jul 25). Strategic competition in the era of artificial intelligence. Center for a New American Security. Retrieved from https://www.cnas.org/publications/reports/strategic-competition-in-an-era-of-artificial-intelligence google scholar
47. Hwang, S. (2023, Sep 26). In South Korea, AI-driven virtual humans are becoming mainstream. The Japan Times. Retrieved from https://www.japantimes.co.jp/business/2023/09/26/tech/ai-humans-south-korea/ google scholar
48. iProov.com. (2022, Aug 26). Protecting against deepfakes: statistics and solutions. Retrieved from http://www.iproov.com/blog/deepfakes-statistics-solutions-biometric-protection google scholar
49. Jungherr, A., and Schlarb, D. B. (2022). The extended reach of game engine companies: Epic Games and Unity Technologies provide platforms for extended reality and metaverse applications. Social Media + Society, 8(2), doi:10.1177/20563051221107641 google scholar
50. Kang, J., Shilton, K., Estrin, D., & Burke, J. (2011). Self-Surveillance and privacy. Iowa Law Review, 97, 809. google scholar
51. Kaplan, O. (1940). Prediction in the social sciences. Philosophy of Science. 7(4): 492-498. google scholar
52. Kelly, J. (2022, Jan 21). The metaverse set off a battle between Google, Apple, Microsoft, and Meta to build virtual and augmented reality (VR) headsets. Forbes. Retrieved from https://www.forbes.com/sites/jackkelly/2022/01/21/the-metaverse-set-off-a-battle-between-tech-giants-google-apple-microsoft-and-meta-to-build-virtual-and-augmented-reality-headsets/?sh=6b4516c239c6. google scholar
53. Kietzmann, J., Lee, L. W., McCarthy, I. P., & Kietzmann, T. C. (2020). Deepfakes: Trick or treat?. Business Horizons, 63(2), 135-146. google scholar
54. Koskela, H. (2004). Webcams, TV shows, and mobile phones: empowering exhibitionism. Surveillance and Society, 2(2/3), 199-215. google scholar
55. Köbis, N. C., Dolezalova, B., & Soraperra, I. (2021). Fooled twice: People cannot detect deepfakes but think they can. Iscience, 24(11). https://doi.org/10.1016/j.isci.2021.103364. google scholar
56. Lacroix, C. (2020, Sep). Preventing discrimination caused by the use of artificial intelligence [Report 1]. Committee on Equality and Non-Discrimination. Council of Europe. Retrieved from https://assembly.coe.int/LifeRay/EGA/Pdf/TextesProvisoires/2020/20200915-PreventingDiscriminationAI-EN.pdf google scholar
57. Langa, J. (2021). Deepfakes, real consequences: Crafting legislation to combat threats posed by deepfakes. Boston University Law Review,101, 761-801. google scholar
58. Lyon, D. (Ed.). (2003). Surveillance as social sorting: privacy, risk, and digital discrimination. London: Routledge. google scholar
59. Lyon, D. (2006). The search for surveillance theories. In D. Lyon (Ed.), Theorising surveillance: The panopticon and beyond (pp. 3-20). Portland: Willan Publishing. google scholar
60. Lyon, D., (2007). Surveillance studies: An overview. Polity Press, UK: Cambridge. google scholar
61. Lyon, D., (2003). Surveillance after September 11. Polity Press, UK: Cambridge. google scholar
62. Mager, A., & Katzenbach, C. (2021). Future imaginaries in the making and governing of digital technology: Multiple, contested, commodified. New Media and Society, 23(2), 223-236. google scholar
63. Mann, S., Nolan, J., & Wellman, B. (2003) Sousveillance: inventing and using wearable computing devices for data collection in surveillance environments. Surveillance and Society, 1(3), 331-355. google scholar
64. Mann, S. (2004). "Sousveillance": inverse surveillance in multimedia imaging. In Proceedings of the 12th annual ACM international conference on Multimedia. Association for Computing Machinery. New York, USA, 620-627. https://doi.org/10.1145/1027527.1027673 google scholar
65. Manovich, L. (2014). The language of new media. MIT Press. google scholar
66. Marcellino, W., Beauchamp-Mustafaga, N., Kerrigan, A., Navarre Chao, L., & Smith, J. (2023, Sep 7). The rise of generative AI and the coming era of social media manipulation 3.0: Next-Generation Chinese Astroturfing and Coping with Ubiquitous AI. RAND Commentary. Retrieved from https://www.rand.org/pubs/perspectives/PEA2679-1.html google scholar
67. Martin, T., Hofman, J. M., Sharma, A., Anderson, A., & Watts, D. J. (2016, April). Exploring limits to prediction in complex social systems. In Proceedings of the 25th international conference on world wide web (pp. 683-694). Retrieved from https://www.cs.toronto.edu/ ashton/pubs/limpred.pdf google scholar
68. Marx, G. (2003) ‘A tack in the shoe: neutralizing and resisting the new surveillance’. Journal of Social Issues, 59(2),. 369-390. google scholar
69. Mearns, G. W., Richardson, R. and Robson, L. (2015). Enact the internet and social media on the public sector’s frontline. New Technology, Work and Employment, 30(3), 190-208. google scholar
70. Murphy, G. & Flynn, E. (2021). Deepfake false memories. Memory, 30(4), 480-492. google scholar
71. Murray, J. (2021). Deepfakes and digital avatars: the new celebrity brand ambassadors. Retrieved from https://www.campaignlive.co.uk/article/deepfakes-digital-avatars-new-celebrity-brand-ambassadors/1713070. google scholar
72. MIT Open Learning (2020, Jul 20): Tackling the misinformation epidemic with “In Event of Moon Disaster”. Retrieved from https://news.mit.edu/2020/mit-tackles-misinformation-in-event-of-moon-disaster-0720. google scholar
73. Murphy, G.; Ching, D.; Twomey, J., & Linehan C. (2023) Face/Off: Changing the face of movies with deepfakes. Plos One. 18(7). https://doi.org/10.1371/journal.pone.0287503 google scholar
74. Negroponte, N. (1995). Being digital. Alfred A. Knopf. google scholar
75. Nellis, S. & Dave, P. (2021, Apr 1). Microsoft wins $21.9 billion contract with U.S. Army to supply augmented reality headsets. Reuters. Retrieved from https://www.reuters.com/article/us-microsoft-army/microsoft-wins-21-9-billion-contract-with-u-s-army-to-supply-augmented-reality-headsets-idUSKBN2BN36B/ google scholar
76. Nojeim, G. ,& Laperruque, J. (2023, Dec 8). Government surveillance: A trojan horse in a house Intel google scholar
77. committee bill massively expands FISA 702 surveillance. Center for Democracy & Technology. Retrieved from https://cdt.org/insights/a-trojan-horse-in-a-house-intel-committee-bill-massively-expands-fisa-702-surveillance/ google scholar
78. Nowak, K. L., & Fox, J. (2018). Avatars and computer-mediated communication: a review of the definitions, uses, and effects of digital representations. Review of Communication Research. 6. 30-53. google scholar
79. Poster, M. (1995). The second machine age. Polity. google scholar
80. M. S. Rana, M. N. Nobi, and B. Murali & Sung H. (2022) .Deepfake detection: A systematic literature review. IEEE Access 10, 25494-25513. doi: 10.1109/ACCESS.2022.3154404. google scholar
81. Rodriguez Resendiz, H., & Rodriguez Resendiz, J. (2024). Digital resurrection: Challenging the boundary between life and death with artificial intelligence. Philosophies, 9(3), 71. google scholar
82. Savin-Baden, M., & Mason-Robbie, V. (Eds.). (2020). Digital afterlife: Death matters in a digital age. CRC Press. google scholar
83. Sherlock, A. (2013). Larger than life: Digital resurrection and the re-enchantment of society. The Information Society, 29(3), pp. 164-176. google scholar
84. Schmidt, E., & Cohen, J. (2013). The new digital age: Reshaping the future of people, nations and business. John Murray. google scholar
85. Schwab, K. (2016). The fourth industrial revolution. Crown publishing. google scholar
86. Sentinel (2020, Oct). Deepfaked 2020: The tipping point: The current threat landscape, its impact on the U.S. 2020 elections, and the coming of AI-generated events at scale. [Report]. Retrieved from https://thesentinel.ai/media/Deepfakes%202020:%20Tipping%20Point,%20Sentinel.pdf google scholar
87. Somers, M. (2020, Jul 21) Deepfakes, explained. MIT Management Sloan School. Retrieved from https://mitsloan.mit.edu/ideas-made-to-matter/deepfakes-explained google scholar
88. Suchman, L. (2012, online publication), (2006, print publication). Human-machine reconfigurations: Plans and situated actions. Cambridge University Press. google scholar
89. Talvitie-Lamberg, K. (2018). Video streaming and internalized surveillance. Surveillance and Society, 16(2). google scholar
90. Tariq, S., Abuadbba, A. and Moore, K. (2023, July). Deepfake in the metaverse: Security implications for virtual gaming, meetings, and offices. Proceedings of the 2nd Workshop on Security Implications of Deepfakes and Cheapfakes (pp. 16-19). Retrieved from https://doi.org/10.1145/3595353.3595880 google scholar
91. Yu, P., Xia, Z., Fei, J. & Lu, Y. (2021). A survey on deepfake video detection. IET Biometrics: The Institue of Engineering and Technology. 10(6). 607-624. google scholar
92. Watts, D. J. (2014). Common Sense and Sociological Explanations. American Journal of Sociology. 120(2): 313-351. google scholar
93. Westling, J. (2019). Are Deep Fakes a Shallow Concern? A critical analysis of the likely societal reaction to deep fakes. TPRC47: The 47th Research Conference on Communication, Information, and Internet Policy; 2019. Retrieved from https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3426174. google scholar
94. Whitaker, R. (1999). The end of privacy: How total surveillance is becoming a reality. New York: The New Press. google scholar
95. Vaccari, C., & Chadwick, A. (2020). Deepfakes and disinformation: Exploring the impact of synthetic political video on deception, uncertainty, and trust in news. Social media+ society, 6(1), 2056305120903408. google scholar
96. Zuboff, S. (2015). Big other: surveillance capitalism and the prospects of an information civilization. Journal of Information Technology. 30. 75-89. google scholar