Redesign suggestions / explore panel

[BeritJanssen]

• multiple suggestion panels for sources / annotations / categories (?)
• explanation to user about what "explore" means and does

[jgonggrijp]

Should be done in discussion with the French team, in particular Brigitte and Guillaume (LNB).

[BeritJanssen]

I now have an idea that might consolidate this issue with #163 : the wish was that instead of 3 sources / annotations / categories, one should be able to browse all sources / annotations / categories here. What about we added dropdown menu items to "Explore" which would allow users to navigate to an Explorer starting with a source list, with an annotation list, or with a category list? (My suggestion would also be to rename "Explore" to "Browse", as that's perhaps a more intuitive term.)

For every list (except for categories, see below), the panel contains a footer with pagination. On top, there should be a control which lets the user determine whether they want alphabetical sorting, by creation date, or in random order, and if it's a logged in user, they can set if they want to restrict the view to sources or annotations of which they are the creator (not applicable for categories). Categories would show a full list of all categories (arguably, seeing three random categories is indeed not super useful...).

After logging in, a user would be taken to a version of this view which sets the "own" boolean to True and sorts by creation date. Non-logged in users would be taken to this Browse view as well, with the SourceList restricted to public domain sources, and with AnnotationList restricted to annotations in these sources.

[jgonggrijp]

I would strongly prefer to list items (in the narrow sense) rather than annotations. Browsing annotations is inefficient because there is so much redundancy. Every annotation can efficiently be reached through a source, an item or a category; the converse is not true. (This was already described in my original specification of the suggestions panel. I somehow missed that "items" became "annotations" when reviewing #374.)

I don't think a full list of sources or even (especially) annotations/items should be available by the click of a button. If people are going to be fetching these full lists all the time, it's going to be far too taxing on the system (even if paginated; @JeltevanBoheemen and I discovered at the 0.11.0 release that pagination was actually slowing things down, so we removed it again from the annotations list in a hotfix). I much prefer the current situation, where people can either search specifically for what they expect to exist or get a limited random sample.

However, I don't mean to say that the suggestions panel should stay as-is. I can agree to all of the following elements of your proposal:

• Renaming "Explore" to "Browse".
• Splitting sources, items and categories to separate panels.
• Merging Personal landing page #163 into this by adding a toggle to these panels to switch between all own creations and a limited sample of everything.
• Adding sorting options for own sources and items.
• Showing all categories instead of just three.
• Showing more sources and items than just three at a time.
• Showing own creations by default for authenticated users.
• Restricting the samples for unauthenticated users to public sources.

Some suggestions for refinement:

• Instead of adding a separate categories panel, put an OntologyCategoryPicker at the top of the items panel, so users can optionally filter by category. This addresses the wish to list all categories as well.
• Instead of full-blown pagination, add a button to simply fetch a fresh random sample. Pagination doesn't seem necessary when listing own creations.
• The items panel with own creations toggled on should probably not only list items that the user created, but also items that the user has created annotations with.
• Likewise, list not only sources that the user uploaded but also sources that the user annotated.

[jgonggrijp]

While working on finalising #501, I came up with an idea that enables us to get a limited, truly random uniform sample of resources from the triple store in a single SPARQL query. After the weekend I'll implement this, because it shaves off a request-response cycle and doesn't require the frontend to fetch a complete list of all resources that exist. However, I'm sharing the query now already so there is a "backup" of it.

I have tried the query below in the Fuseki web interface on the production triplestore and confirmed that it yields different results on each request, in no predictable order. I confirmed this with limit 10, limit 50 and limit 5. On the Fuseki web interface, it takes a bit over 3.5 seconds before the response arrives. Both old and recent items are included.

construct {
    ?item ?prop ?val
} from item: where {
    ?item ?prop ?val
    {
        select distinct ?item where {
            ?anno oa:hasBody ?item.
            ?item a ?x.
            bind (substr(str(?item), 32) as ?serial)
            bind (strlen(?serial) - 1 as ?cap)
            bind (substr(?serial, ?cap) as ?suffix)
            bind (substr(?serial, 1, ?cap) as ?prefix)
        }
        order by md5(concat(?suffix, substr(str(rand()), 2), ?prefix))
        limit 5
    }
}

The inner select subquery takes care of the random sampling while the outer construct ensures that the fetched items are complete and in RDF format. The ?anno oa:hasBody ?item and ?item a ?x constraints together ensure that only ontology class instances (aka items in the narrow sense) are included.

The magic happens in the complex order by expression. A string is composed out of the final two digits of the serial number, a random number and then the remaining initial digits of the serial number, in that order, then hashed. While rand() is evaluated only once, it ensures that the resulting expression is different between consecutive executions of the query. The md5 hash ensures that there is no structural relationship between the item serial and the resulting order. The last two digits of the serial are moved to the front in order to further boost sampling entropy. The first two characters of str(rand()) are removed because they are always 0..