Skip to content

Commit

Permalink
improved evaluation docs
Browse files Browse the repository at this point in the history
  • Loading branch information
maniospas committed Jun 12, 2024
1 parent 611a445 commit 0164660
Showing 1 changed file with 22 additions and 11 deletions.
33 changes: 22 additions & 11 deletions docs/userguide/evaluation.md
Original file line number Diff line number Diff line change
Expand Up @@ -67,14 +67,10 @@ In addition to base measures, there also exist c) variants
that combine them, and d) evaluation that runs on multiple communities.
To begin with, multiple measures can be aggregated through the `pg.AM` and
`pg.GM` classes, which respectively perform arithmetic and geometric
averaging of measure outcomes. These can be constructed and measures
can be added to them with a function chain of `.add` calls.
They also accept optional weights and saturation
thresholds. When such thresholds are provided,
the constructor of the measure combination mechanism
can be annotated with `differentiable=True` to indicate that
a differentiable relaxation of saturation should be
applied. Here is an example:
averaging of assessment values. After instantiateing these
classes, measures can be added to them with a chain of `.add` method calls
that also accept optional weights and saturation
thresholds. Here is an example:

```python
import pygrank as pg
Expand All @@ -85,12 +81,19 @@ prule = pg.pRule(sensitivity_scores, exclude=personalization)
measure = pg.AM(differentiable=True).add(auc).add(prule, weight=10., max_val=0.8)
```

!!! info
When working on autodifferntiation backends,
the constructor of the measure combination mechanism
can be annotated with `differentiable=True`. This indicates that,
when thresholds are provided, a differentiable relaxation of saturation
should be applied instead of a simple min or max.

Extension of base measures to multiple communities can be obtained from the
`pg.MultiSupervised` and `pg.MultiUnsupervised` classes. These take as arguments
the *classes* of base measures and any keyword arguments
needed to construct them. In case of the *exclude* attribute of
supervised measures a dictionary from community
identifiers (e.g., strings with community names) to
identifiers (like strings with community names) to
graph signals/signal data should be provided to the `pg.MultiSupervised`.
When called, these measures process and output dictionaries from community
identifiers to create an assessment value for each of those communities.
Expand All @@ -109,8 +112,7 @@ measure = pg.MultiUnsupervised(pg.Conductance)
print(measure(comm_scores))
```

In case of unsupervised evaluation that is needed
when there are too few example members,
In case of unsupervised evaluation
you may also be interested in computing link-based
unsupervised measures. This is done with the
`pg.LinkAssessmen` class; this needs the graph
Expand All @@ -129,6 +131,15 @@ measure = pg.LinkAssessment(graph, progress=tqdm.tqdm) # graph argument mandato
print(measure(comm_scores))
```

!!! info
Link-based evaluation or, in general,
some kind of unsupervised evaluation is needed
when there are only a few community
members that serve as examples towards which to
score proximity. In this case, splitting example members to
training and test sets risks degrading supervised
evaluation quality.


## Datasets

Expand Down

0 comments on commit 0164660

Please sign in to comment.