Documentation

Table of Contents Architecture/Implementation Dataflow XML or JSON dump Dump Parser Feature CSV Data Analyzer Suggester Data Example Usages Suggester Engine Detailed Ideas regarding Implementation: Algorithms UI Integration

Architecture/Implementation

There are any number of concrete Suggesters that implement the Suggester Interface which provides a function to obtain fitting attributes for an entity or a certain set of attributes (optionally values can be suggested). This interface may be used at different places. E.g. for adding attribute suggestions to the edit item page.

Dataflow

It is essential to define on what data a suggester-engine is basing its suggestions and how it can access this data/how the data is generated - this can vary depending on the suggester that is used - for the SimplePHPSuggester we have the following approach in mind:

|XML or JSON dump| -> (dump parser) -> |feature CSV| -> (data analyzer) -> |suggester data| -> (suggester engine)

XML or JSON dump

|XML or JSON dump| is a dump of the full entity data of all entities, provided as a compressed file. The XML also wraps JSON, but that "internal" JSON should be avoided as soon as "real" JSON dumps are available.

Dump Parser

(dump parser) is a script that reads the dump and extracts the information("features") needed by the data analyzer, and outputs them to a CSV file (which can then be loaded into MySQL, or processed with Hadoop, or whatever).

Feature CSV

|feature CSV| is a CSV file containing the features needed by the data analyzer. In the most basic form, the file consists of two columns: |subject| and |feature|. For suggesting properties based on existing properties, that would mean |itemid| |propertyid| (for suggesting products to customers, like on amazon, it would correspond to |customer| |product|). For some use cases like finding classifiers (see here) or suggesting values it might also be useful to have additional columns for storing the value and type of a property.

Data Analyzer

(data analyzer) generates the information used by the suggester (the data could either be stored into files to be loaded into a database, or directly into the database). This includes finding correlations between features by counting co-occurences of properties and possibly property/value pairs as well as computing their usage frequencies. Filtering and perhaps clustering could also well be integral parts in this process.

Suggester Data

|suggester data| is the information used by the suggester engine. It resides in a database that can be accessed from the application servers. The goal is to structure and index the data in such a way that the suggester engine can generate suggestions instantly.

For example the Data that is accessed by the SimplePHPSuggester is stored according to the following schema:

wbs_PropertyPairs(pid1 INT, pid2 INT, count INT, correlation FLOAT, primary key(pid1, pid2))

• Explanation: One entry for each occuring binary combination of properties, count (how often does the combination occur), correlation(how like is it to find pid1 in combination with pid2)

wbs_Properties(pid INT, count INT, type VARCHAR(20), primary key(pid))

Example Usages

1. Use Case:

Assumptions:

• we are given an entity E described by a list of Properties L
• we want the top X property-suggestions for the given set of properties
• we only wish to suggest properties that correlate well with the properties in L(their rating has to pass a certain threshold T) and that are not already contained in L of course

In order to achieve all this we could use the following query:

SELECT pid2, sum(correlation) AS cor
FROM wbs_PropertyPairs
WHERE pid1 IN L AND pid2 NOT IN L
GROUP BY pid2
HAVING sum(correlation) > T
ORDER BY cor DESC
LIMIT X

(see here for further information on this use case)

2. Use Case:

A simple algorithm for finding classifying properties could make use of the query below:

SELECT pid, count
FROM wbs_Properties
WHERE type = "item"
ORDER BY count
LIMIT 5

(see here for further information on the described use case)

Suggester Engine

(suggester engine) generates suggestions based on the provided data set. It has information about how the data is structured, and how it should be interpreted. It's written in PHP and is called directly from (or is part of) a MediaWiki extension.

Detailed Ideas regarding Implementation:

Algorithms

• Nilesh's idea
• Ziawasch Abedjan, Felix Naumann Improving RDF Data Through Association Rule Mining

UI Integration

While the input field is still empty, the best property suggestions are displayed based on existing attributes of the current item. When the user starts typing, the input is used to filter the suggested properties: Properties that do not match the current input are replaced by traditional search results. Another approach (which we are not working on currently) is to rank all properties that match the input and display the top n of them.

There are basically two ideas to integrate the suggester results with the "add statement" functionality.
1. combine suggestions with former search results on client side.

or
2. combine suggestions with former search results on server side.