HOW TO USE GENAI AS A RATER IN MARKETING: A COMPREHENSIVE GENAI-BASED CONTENT CODING FRAMEWORK

Öz

With the recent developments in Generative AI (GenAI) applications, popular tools such as ChatGPT have gained potential not only for individual or commercial purposes but also for the marketing discipline. As an important process for the qualitative data analysis in marketing research, the task of coding texts is mostly performed by human experts, and this can cause loss of time and increase costs. These tools may also have the potential to perform the task of coding texts, which is mostly performed by human experts for the qualitative data analysis process in marketing research. Although there are techniques applied for coding qualitative data, no study has yet presented a systematic framework to use GenAI tools in coding tasks, especially for marketing research. To fill this research gap, the current study proposes a 6-step GenAI-based content coding framework. In the framework, firstly brand messages are collected, and the themes needed for classification are determined. Then, appropriate Gen-AI models are selected to separate brand messages according to the desired themes, prompts are prepared, classes with AI outputs are coded and finally, compatibility between coders is checked. In this respect, an application is carried out to test the proposed framework. Informational, entertaining and remunerative content strategies in consumer engagement literature were used as themes and the coding agreements between 3 Large Language Models (LLMs) and human experts were determined with Kappa statistics. According to the results, the level of agreement between Human and ChatGPT gave the best Inter-Rater Reliability (IRR) for informational and remunerative contents in the comparison of Human and AI coders, while Gemini performed better for entertaining messages. As a most important practical contributions, the framework of this study offers a useful and faster coding processes for marketing practitioners.

Anahtar Kelimeler

• GenAI
• large language model
• inter-rater reliability
• marketing
• framework

PAZARLAMADA YAPAY ZEKAYI DEĞERLENDİRİCİ OLARAK NASIL KULLANIRIM? ÜRETKEN AI İLE KAPSAMLI BİR İÇERİK KODLAMA ÇERÇEVESİ

Öz

Yapay zekâ (AI) alanındaki son gelişmelerle birlikte, üretken-AI (GenAI) uygulamaları arasında öne çıkan ChatGPT gibi popüler araçlar, bireysel veya ticari amaçların yanı sıra pazarlama disiplini için de potansiyel vadetmektedir. Pazarlama araştırmasında metin kodlama görevi, nitel veri analizi için önemli bir süreç olmakla birlikte çoğunlukla insanlar tarafından gerçekleştirilmektedir ve bu durum vakit kaybına ve maliyet artışına neden olabilmektedir. GenAI araçları, pazarlama araştırmasında nitel veri analizi süreci için çoğunlukla insan uzmanlar tarafından gerçekleştirilen metin kodlama görevini de gerçekleştirme yeteneğine sahip olabilir. Nitel verileri kodlamak için uygulanan teknikler bulunmasına rağmen, şimdiye kadar hiçbir çalışma, özellikle pazarlama araştırması için kodlama görevlerinde GenAI araçlarını kullanmak için sistematik bir çerçeve sunmamıştır. Bu araştırma boşluğunu doldurmak için, mevcut çalışma 6 adımlı GenAI tabanlı bir içerik kodlama çerçevesi önermektedir. Çerçevede öncelikle marka mesajları toplanmakta ve sınıflandırma için ihtiyaç duyulan temalar tespit edilmektedir. Ardından marka mesajlarını arzu edilen temalara göre ayırmak için uygun Gen-AI modelleri seçilmekte, prompt hazırlanmakta, AI çıktıları olan sınıflar kodlanmakta ve son olarak kodlayıcılar arası uyuma bakılmaktadır. Bu doğrultuda, önerilen çerçeveyi test etmek için bir uygulama gerçekleştirilmiştir. Tüketici katılımı literatüründeki bilgilendirici, eğlendirici ve ödüllü içerik stratejileri tema olarak kullanılmış ve 3 Büyük Dil Modeli (LLM) ile insan uzmanlar arasındaki kodlama uyumları Kappa istatistikleri ile belirlenmiştir. Ulaşılan sonuçlara göre, İnsan ve ChatGPT arasındaki uyuşma düzeyi, bilgilendirici ve ödüllü içerikler için en iyi Kodlayıcılar Arası Uyum (IRR) vermiş olup Gemini, eğlenceli mesajlar için daha iyi performans göstermiştir. Varılan sonuçların önemli pratik katkıları arasında, pazarlama uygulayıcıları için yararlı ve daha hızlı kodlama süreçleri sunan mevcut çalışmanın önerilen çerçevesi öne çıkmaktadır.

Anahtar Kelimeler

• GenAI
• büyük dil modeli
• kodlayıcılar arası uyum
• pazarlama
• çerçeve

Kaynakça

1. Anthropic. (2024). Claude 3.5 Sonnet. Accessed: 7.30.2024. https://www.anthropic.com/news/claude-3-5-sonnet
2. Ashley, C., and Tuten, T. (2015). Creative strategies in social media marketing: An exploratory study of branded social content and consumer engagement. Psychology & Marketing, 32(1), 15-27.
3. Basit, T. (2003). Manual or electronic? The role of coding in qualitative data analysis. Educational research, 45(2), 143-154.
4. Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., ... & Amodei, D. (2020). Language models are few-shot learners. Advances in neural information processing systems, 33, 1877-1901.
5. Buono, D., Felecan, M., and Tessitore, C. (2024). An introduction to Large Language Models and their relevance for statistical offices. Eurostat: Luxembourg. Accessed: 27.08.2024 https://ec.europa.eu/eurostat/documents/3888793/18771440/KS-TC-24-001-EN-N.pdf/fbdbcc5b-7b93-39af-5980-944112feaff6?version=1.0&t=1711031922718
6. Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., ... & Xie, X. (2024). A survey on evaluation of large language models. ACM Transactions on Intelligent Systems and Technology, 15(3), 1-45.
7. Chen, B., Zhang, Z., Langrené, N., and Zhu, S. (2023). Unleashing the potential of prompt engineering in Large Language Models: a comprehensive review. arXiv preprint arXiv:2310.14735.
8. Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37-46. https://doi.org/10.1177/001316446002000104

9. Cvijikj, I. P., and Michahelles, F. (2013). Online engagement factors on Facebook brand pages. Social Network Analysis and Mining, 3(4), 843–861. https://doi.org/10.1007/s13278-013-0098-8
10. Demir, S. (2023). Investigation of ChatGPT and Real Raters in Scoring Open-Ended Items in Terms of Inter-Rater Reliability. Uluslararası Türk Eğitim Bilimleri Dergisi, 11(21), 1072-1099. https://doi.org/10.46778/goputeb.1345752
11. Dolan, R., Conduit, J., Frethey-Bentham, C., Fahy, J., and Goodman, S. (2019). Social media engagement behavior: A framework for engaging customers through social media content. European journal of marketing, 53(10), 2213-2243.
12. Faul, F., Erdfelder, E., Lang, A. G., & Buchner, A. (2007). G* Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior research methods, 39(2), 175-191.
13. Fui-Hoon Nah, F., Zheng, R., Cai, J., Siau, K., and Chen, L. (2023). Generative AI and ChatGPT: Applications, challenges, and AI-human collaboration. Journal of Information Technology Case and Application Research, 25(3), 277-304.
14. Giray, L. (2023). Prompt engineering with ChatGPT: a guide for academic writers. Annals of biomedical engineering, 51(12), 2629-2633.
15. Gisev, N., Bell, J. S., and Chen, T. F. (2013). Interrater agreement and interrater reliability: key concepts, approaches, and applications. Research in Social and Administrative Pharmacy, 9(3), 330-338.
16. Gupta, H., Singh, S., and Sinha, P. (2017). Multimedia tool as a predictor for social media advertising-a YouTube way. Multimedia tools and applications, 76(18), 18557-18568.
17. Hadi, M. U., Al Tashi, Q., Shah, A., Qureshi, R., Muneer, A., Irfan, M., ... & Shah, M. (2024). Large language models: a comprehensive survey of its applications, challenges, limitations, and future prospects. Authorea Preprints. https://doi.org/10.36227/techrxiv.23589741.v4.
18. Hallgren, K. A. (2012). Computing inter-rater reliability for observational data: an overview and tutorial. Tutorials in quantitative methods for psychology, 8(1), 23-34. doi: 10.20982/tqmp.08.1.p023
19. Huang, M. H., and Rust, R. T. (2021). A strategic framework for artificial intelligence in marketing. Journal of the Academy of Marketing Science, 49, 30-50.
20. Jin, Z., and Lu, W. (2023). Tab-cot: Zero-shot tabular chain of thought. arXiv preprint arXiv:2305.17812.
21. Koçak, B. B. (2021). Fly “With us”! Impact of consumer-brand relationship on consumer engagement: An empirical investigation on Turkish airline instagram pages. Tüketici ve Tüketim Araştırmaları Dergisi, 13(2), 253–282. https://doi.org/10.15659/ttad.13.2.139
22. Koçak, B. B. (2023). Impact of Brand Linguistic Characteristics on Consumer Engagement: A Psycholinguistics Approach for Airline Facebook Pages. M. Dalkılıç (Ed.), INSAC 2023 New Trends in Social and Education Sciences içinde, (13-30). Ankara: Duvar.
23. Koçak, C. B., and Atalık, Ö. (2024). Figurative language effect on consumer engagement: an empirical investigation for Turkish airline industry. Aviation, 28(2), 128-140.
24. Koçak, C. B., Atalık, Ö., and Koçak, B. B. (2024). Mecazi dil unsurlarının tüketici katılımı üzerindeki etkisi: Türk havayolu Instagram sayfaları örneği. Pazarlama ve Pazarlama Araştırmaları Dergisi, 17(1), 1-38.
25. Labrecque, L. I., Swani, K., and Stephen, A. T. (2020). The impact of pronoun choices on consumer engagement actions: Exploring top global brands' social media communications. Psychology & Marketing, 37(6), 796-814.
26. Lacy, S., Watson, B. R., Riffe, D., & Lovejoy, J. (2015). Issues and best practices in content analysis. Journalism & mass communication quarterly, 92(4), 791-811.
27. Landis, J. R., and Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33(1), 159-174.
28. Lee, Dokyun, Kartik Hosanagar, and Harikesh Nair (2018), “Advertising Content and Consumer Engagement on Social Media: Evidence from Facebook,” Management Science, 64 (11), 5105–31.
29. Lewis, S. C., Zamith, R., and Hermida, A. (2013). Content analysis in an era of big data: A hybrid approach to computational and manual methods. Journal of broadcasting & electronic media, 57(1), 34-52.
30. Marvin, G., Hellen, N., Jjingo, D., and Nakatumba-Nabende, J. (2023). Prompt engineering in large language models. In International conference on data intelligence and cognitive informatics (pp. 387-402). Singapore: Springer Nature Singapore.
31. Menon, R. V., Sigurdsson, V., Larsen, N. M., Fagerstrøm, A., Sørensen, H., Marteinsdottir, H. G., and Foxall, G. R. (2019). How to grow brand post engagement on Facebook and Twitter for airlines? An empirical investigation of design and content factors. Journal of Air Transport Management, 79, Article 101678. https://doi.org/10.1016/j.jairtraman.2019.05.002
32. Minaee, S., Mikolov, T., Nikzad, N., Chenaghlu, M., Socher, R., Amatriain, X., and Gao, J. (2024). Large language models: A survey. arXiv preprint arXiv:2402.06196.
33. Ollion, E., Shen, R., Macanovic, A., & Chatelain, A. (2023). ChatGPT for Text Annotation? Mind the Hype!. https://fles.osf.io/v1/resources/x58kn/providers/osfstorage/651d60731bc8650a79f376cf?direct=&mode=render.
34. Paul, J., Ueno, A., and Dennis, C. (2023). ChatGPT and consumers: Benefits, pitfalls and future research agenda. International Journal of Consumer Studies, 47(4), 1213-1225.
35. Prasad, B. D. (2008). Content analysis. Research methods for social work, 5(le20).
36. Pennebaker, J. W., Boyd, R. L., Jordan, K., and Blackburn, K. (2015). The Development and Psychometric Properties of LIWC2015. Austin, TX: University of Texas at Austin.
37. Pezzuti, T., Leonhardt, J. M., and Warren, C. (2021). Certainty in language increases consumer engagement on social media. Journal of Interactive Marketing, 53(1), 32-46.
38. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. (2019). Language models are unsupervised multitask learners. OpenAI blog, 1(8), 9.
39. Sahoo, P., Singh, A. K., Saha, S., Jain, V., Mondal, S., and Chadha, A. (2024). A systematic survey of prompt engineering in large language models: Techniques and applications. arXiv preprint arXiv:2402.07927.
40. Siedlecki, S. L. (2020). Understanding descriptive research designs and methods. Clinical Nurse Specialist, 34(1), 8-12.
41. Stemler, S., (2000) “An overview of content analysis”, Practical Assessment, Research, and Evaluation 7(1): 17. doi: https://doi.org/10.7275/z6fm-2e34
42. Team, G., Anil, R., Borgeaud, S., Wu, Y., Alayrac, J. B., Yu, J., ... & Ahn, J. (2023). Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805.
43. Theelen, H., Vreuls, J., and Rutten, J. (2024). Doing Research with Help from ChatGPT: Promising Examples for Coding and Inter-Rater Reliability. International Journal of Technology in Education, 7(1), 1-18.
44. Wang, Z., Zhang, H., Li, C. L., Eisenschlos, J. M., Perot, V., Wang, Z., ... & Pfister, T. (2024). Chain-of-table: Evolving tables in the reasoning chain for table understanding. arXiv preprint arXiv:2401.04398.
45. Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., ... & Zhou, D. (2022). Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171.
46. Weber, R. P. (1990). Basic Content Analysis, 2nd ed. Newbury Park, CA.
47. Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., ... & Zhou, D. (2022). Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35, 24824-24837.
48. White, J., Fu, Q., Hays, S., Sandborn, M., Olea, C., Gilbert, H., ... & Schmidt, D. C. (2023). A prompt pattern catalog to enhance prompt engineering with chatgpt. arXiv preprint arXiv:2302.11382.
49. Zhou, D., Schärli, N., Hou, L., Wei, J., Scales, N., Wang, X., ... & Chi, E. (2022). Least-to-most prompting enables complex reasoning in large language models. arXiv preprint arXiv:2205.10625.