Downloads - Software - FReSET.KOM

An Evaluation Framework for Folksonomy-based Recommender Systems


FReSET.KOM is a Java framework for the evaluation of folksonomy-based recommender systems. It provides interfaces for the implementation of folksonomy-based recommender systems and supports the consistent and reproducible offline evaluation using historical data.

Unlike other recommender system framework projects, the emphasis here is on providing a flexible framework allowing users to implement their own folksonomy-based recommender algorithms and pre-processing fi ltering methods rather than just providing a collection of collaborative filtering implementations.

FReSET.KOM comprises a graphical interface for result visualization and diff erent cross-validation implementations to complement the basic functionality.

Folksonomy Data Model and File Format

In FReSET.KOM, folksonomies are designed as a graph consisting of vertices and edges connecting them. The package folksonomy contains classes for the definitions of folksonomies. Vertex is the central data structure.  It simply has a set of edges (represented by the class Edge) and through them knows about its neighbours. The classes Resource, Tag, and User extend it. The class Folksonomy is additionally used to represent the whole folksonomy. It manages the addition, connection and removal of vertices and edges.

FReSET.KOM input fi les follow a simple plain-text format. Each line in the file represents one post. A post consists of a resource r, the user u who added the resource and all tags t that the user has assigned to r. The elements (resource, tag(s), user) are delimited using TAB. The fi rst string represents the URL of a resource (e.g., the second the tag(s) separated by comma (e.g. bookmarking,social) and the third the user who has tagged the resource (e.g. matthiasb). Optionally, a fourth string representing the date the resource was added can be speci fied as well. The following table shows an example of a folksonomy:


Furthermore, folksonomy graphs can be extended using edges between any two vertices, for example to represent friendship relationships between users, similarity measures between resources or semantic relations between tags. The input format for these relations consists of a capital letter representing the vertex type, i.e. resource (R), tag (T) and user (U) followed by its name. An example for additional edges is shown in the following table:


Recommender System Interfaces and Evaluation Methods

FReSET.KOM supports three types of recommendation tasks which are given as interfaces in the package recommender:

  1. The tag recommendation task, which finds the best fitting tags for a given user. It is represented by the interface TagRecommender.
  2. The resource recommendation task, which calculates a list of resources for a given user described by the interface ResourceRecommender.
  3. The user recommendation task that returns users similar to a given user. The interface UserRecommender defines this recommendation task.

All interfaces implement the interface Recommender which consists of the method recommend which takes a folksonomy and a user as input and returns a list of tags, resources or users depending on the recommendation task.

An evaluator is de fined by the abstract class Evaluator. This class provides the basic structure to implement diff erent types of evaluations on a single recommender. The most often used technique to evaluate recommender systems is Cross-validation, which is represented by the class CrossValidation. It is the generic and abstract implementation of a per-user cross-validation. The validation algorithm calculates for every user in the folksonomy, a list of folds. A fold consists of two parts: the list of edges and vertices removed from the folksonomy and the list of vertices to validate the recommendation against. For any of these folds, a single evaluation of a recommender is now applied. Currently the following variations of Cross-validation are already implemented:

  • Leave-Post-Out: This method removes in each fold for a user one of his resources as well as the concurrent edges of the resource.
  • Leave-Tag-Out: This validation method can be applied to resource recommenders. Instead of iterating over all resources, it leaves each tag connected to a user out and validates against the resources connecting the user and tag.
  • N-Folds-per-User: This approach ensures that there are a maximum of N folds used for each user. Again there are two variations: one using posts for tag recommenders NFoldsPostCrossValidation and another that leaves tags out for resource recommenders NFoldTagCrossValidation.

Further evaluator algorithms can be implemented.

Pre-Processing and Parametrization

FreSET.KOM allows the preprocessing of data. This is done using filters. The interface Filter can be found in the package preprocess. It declares a method filterFolksonomy which gets a folksonomy and returns the modifi ed object. Implemanting this interface allows the definition of own filters and preprocessing methods. Currently there is only one preprocessing method implemented, a p-core at level k that can be parametrized using the package parameterization. This package can not only be used to parametrize filters, but also to parametrize recommenders and evaluations. It adds a class Parameter and an immutable list ParameterList that also serves as a type of map by enabling the direct search for a parameter given its name. A parameter has a type, a name, and a default value. Currently there are four types available: String, Integer, Float and Boolean. Additionally, an event based system exists that can be used to ensure a parameter ful fills certain constraints.


Graphical User Interface

Even though the framework can directly be used as a library in Java or to implement own experiments, there is an additional package containing a graphical user interface. It provides an environment that leads step by step through the evaluation process. This user interface consists of three parts as shown here in detail:

(1) The first part when starting the program is the pre-processing panel Preprocess shown in the following screenshot:

This panel gives access to the functionality needed to load folksonomies into the program, to modify them and to write them back to a file. After opening a fi le, some statistics about the current folksonomy will be shown at the bottom of the tab. The middle part allows modi cations through the fi lters described earlier.

Having prepared a folksonomy, the recommender selection panel Recommendation can now be opened. This panel allows to perform the evaluation experiments and to display textual results. At fi rst the recommender has to be selected and confi gured. After that the evaluation method can be selected. This is shown in the following screenshot:

Whenever an evaluation method produces numerical results, these are displayed in the last panel called Visualization. Its sole purpose is the visualization of the evaluation results which can exported as PNG-files.



Getting Started

The following program code gives an example of how to use the FReSET.KOM framework for evaluating an implemented recommender on a given folksonomy:

The first step is to instantiate the recommender. In our case we create an instance of a simple tag recommender. Since every evaluation has to be based upon a dataset, the second step is to load a folksonomy from a fi le. After that, we need to choose and instantiate an evaluation method. This evaluation method has to fit the type of recommender and thus has to be an evaluator working on tag recommenders. Finally, the evaluate method can be called which will return a result object. In the given example, only the resulting object is used to print out a textual summary of the results. But obviously it is also possible to directly access the resulting values.


The FReSET.KOM framework is published under the terms of the GNU GENERAL PUBLIC LICENSE version 3 ( If you plan on using the library in your project we would be happy if you could drop us a line.


Renato Domínguez García, Matthias Bender, Mojisola Anjorin, Christoph Rensing, Ralf Steinmetz: FReSET - An Evaluation Framework for Folksonomy-based Recommender Systems. In: Bamshad Mobasher, Dietmar Jannach, Werner Geyer, Andreas Hotho: Proceedings of the 4th ACM RecSys workshop on Recommender systems and the social web, p. 25-28, ACM, September 2012.


The FReSET.KOM library is currently maintained by Mojisola Anjorin and Matthias Bender. For questions, comments, or suggestions regarding FReSET.KOM you can contact us at {anjorin,matthias.bender}[at]

FReSET.KOM was originally designed together with Renato Domínguez García in 2011.