A HYBRID STATISTICAL APPROACH TO STEMMING IN TURKISH: AN AGGLUTINATIVE LANGUAGE
Abstract
Finding Stem is a complicated and important issue for agglutinative languages like Turkish where theoretically infinite number of surface forms can be obtained from a single lexeme. Both analytical and statistical approaches have been tried for stemming Turkish words. Two main problems apparent with these approaches are the involvement of a dictionary which enforces the assumption of closed vocabulary and the disambiguation of the actual stem among the numerous candidates. Here, we present a method that exploits the simple fact that nouns and verbs have different suffix patterns. Statistical methods are used for stripping off the suffixes. Based on the suffix pattern PoS is determined which then enables the decision for the stem boundary. Thus, the major contribution of the study is the avoiding the disambiguation problem and not using a regular dictionary for stemming. The performance rate for proposed method on golden standard PoS tagged Turkish corpus is 93.83%.
Keywords
Stemming, Natural Language Processing, Turkish, Agglutinative Language
References
- Solak A, Can F. Effects of stemming on Turkish text retrieval, in Proceedings of the Ninth Int. Symp. on Computer and Information Sciences (ISCIS’94), 1994, pp. 49-56.
- Ekmekçioglu FC. Lynch MF, Willett P. Stemming and n-gram matching for term conflation in Turkish texts, Information Research News 1996: 7; 2-6.
- Ekmekçioglu FC, Willett P. Effectiveness of stemming for Turkish text retrieval, PROGRAM- LONDON-ASLIB-2000: 34: 195-200.
- Sever H, Bitirim Y. FindStem: Analysis and evaluation of a Turkish stemming algorithm, in String Processing and Information Retrieval, 2003: 238-251.
- Pembe FC, Say ACC. A linguistically motivated information retrieval system for Turkish, in Computer and Information Sciences-ISCIS 2004, ed: Springer, pp. 741-750.
- Can F, Koçberber S, Balcık E, Kaynak C, Öcalan HÇ, Vursavaş OM. First large-scale information retrieval experiments on Turkish texts, in Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, 2006, pp. 627-628.
- Can F, Koçberber S, Balcık E, Kaynak C, Öcalan HÇ, Vursavaş OM. Information retrieval on Turkish texts, Journal of the American Society for Information Science and Technology 2008: 59: 407
- Lovins JB. Development of a stemming algorithm: MIT Information Processing Group, Electronic Systems Laboratory, 1968.
- Porter MF. An algorithm for suffix stripping, Program, vol. 14, pp. 130-137, 1980.
- Dolamic L, Savoy J. Stemming approaches for east european languages, in Advances in Multilingual and Multimodal Information Retrieval, ed: Springer, 2008, pp. 37-44.