Are SSSOM tables too verbose for some use cases?

[matentzn]

A colleague from the KG domain reviewed SSSOM this morning and noted that it is much too verbose for sharing large edge sets. An interesting debate ensued about the tradeoffs of encoding nodes as 16 bit numbers and having a simple 2 (or 3)-column tables and the very verbose model SSSOM has become. It would be interesting to think about an encoded, much terser serialisation for sssom as an alternative format.

[nichtich]

That's what standard compression exist for. Processing gz'ed TSV files should be no big deal. To reduce the unpacked TSV there might be better ways than inventing your own compression, e.g.:

• omit columns subject_label and object_label optional (are they mandatory?)
• omit column mapping_justification when all mappings have same value anyway
• omit column predicate_id when all mappings have same value anyway
• replace curie-syntax with prefix-less identifiers in subject_id and object_is when same prefix is used for each anyway

So this example

subject_id	predicate_id	object_id	mapping_justification	subject_label	object_label
HP:0009124	skos:exactMatch	MP:0000003	semapv:LexicalMatching	Abnormal adipose tissue morphology	abnormal adipose tissue morphology
HP:0008551	skos:exactMatch	MP:0000018	semapv:LexicalMatching	Microtia	small ears
HP:0000411	skos:exactMatch	MP:0000021	semapv:LexicalMatching	Protruding ear	prominent ears

in minimal form could be reduced to

subject_id	object_id
0009124	0000003
0008551	0000018
0000411	0000021

[matentzn]

I guess we could discuss compression this way of there is a very very strong use case. I personally think that a published mapping set should always be decompressed to make it easily readable - the use case of file size is not really anything I usually care about. However, even when doing what you are suggesting (which is interpreting globally defined metadata pertaining to mappings as applicable for the whole mapping sets), the absolute minimum would be:

subject_id	object_id
a:0009124	b:0000003
a:0008551	b:0000018
a:0000411	b:0000021

Because we have no nothing of default namespace in SSSOM (not really that useful, and annoying for endusers without RDF knowledge).

Note that the only two properties that are required (aside from the above) are predicate_id and mapping_justification.

[nichtich]

Is there a default for mapping_justification such as "unknown"? Most of our mappings have been created or decided on intellectually so there is no specific justification for each individual mapping (justification as property of MappingSet). Having a default value could reduce the TSV format (but might encourage to not use this field at all).

The default for predicate_id in our applications is http://www.w3.org/2004/02/skos/core#mappingRelation (somehow related but unknown semantics).

[matentzn]

I agree with your suggestion and tried to argue for this here: #117, but I got a hard no from other core team members.. Maybe we need to table that suggestion again and build some reasonable arguments. There are a number of people that agree with you, but there are two camps. The second one is afraid that if we make mapping_justification optional or have it revert to something meaningless, bad practices will creep in. That said, there is something you can do:

https://github.com/mapping-commons/sssom/blob/master/src/sssom_schema/schema/sssom_schema.yaml#L226