Murdoch University Research Repository

Welcome to the Murdoch University Research Repository

The Murdoch University Research Repository is an open access digital collection of research
created by Murdoch University staff, researchers and postgraduate students.

Learn more

Efficient token based clone detection with flexible tokenization

Basit, H.A., Puglisi, S.J., Smyth, W.F., Turpin, A. and Jarzabek, S. (2007) Efficient token based clone detection with flexible tokenization. In: 6th Joint Meeting on European software engineering conference and the ACM SIGSOFT symposium 2007, 3 - 7 September 2007, Dubrovnik, Croatia p. 513.

Link to Published Version:
*Subscription may be required


Code clones are similar code fragments that occur at multiple locations in a software system. Detection of code clones provides useful information for maintenance, reengineering, program understanding and reuse. Several techniques have been proposed to detect code clones. These techniques differ in the code representation used for analysis of clones, ranging from plain text to parse trees and program dependence graphs. Clone detection based on lexical tokens involves minimal code transformation and gives good results, but is computationally expensive because of the large number of tokens that need to be compared. We explored string algorithms to find suitable data structures and algorithms for efficient token based clone detection and implemented them in our tool Repeated Tokens Finder (RTF). Instead of using suffix tree for string matching, we use more memory efficient suffix array. RTF incorporates a suffix array based linear time algorithm to detect string matches. It also provides a simple and customizable tokenization mechanism. Initial analysis and experiments show that our clone detection is simple, scalable, and performs better than the previous well-known tools.

Item Type: Conference Paper
Item Control Page Item Control Page