Where to store ZedToken if the resources are in a tree structure?

[henkosch]

We have a hierarchical resource structure similar to a folder tree. Let's say we have a similar setup to google drive, where resources are files and they are in folders. Folders could have folders in them. You can assign permissions for users on folders which will give them access to any of the files in that folder or any of its subfolders transitively. You can assign a single user or a group of users to a folder where a group of users could also contain other groups of users.

The question is: Where should I store my ZedTokens after I write a relationship?
(Create a file, create a user, assign someone permission, add someone to a group, etc)

The documentation says that I should store the returned ZedToken in the parent resource. But there is a full chain of parents in this case. Which parent? Should I store in with the top most folder? That I think would be the safest option, because that would mean any change in any of the relationships would result in a new ZedToken for the whole tree. Although it would also be a bottleneck because I would have to store it with proper locking in place. Also it would not add much performance gain compared to a fully consistent read when performing lookup resources.

But if I consider storing it with any other folder down the tree, I could potentially miss a permission change that was happening upper in the tree.

What would be the best practice in this case?

[josephschorr]

The general rule of thumb is the immediate resource for the relationship being changed: the document itself if its permission is being modified, the folder if a document is added/removed, the group if a member is being added/removed.

You can certainly do so at the top level, but as you mentioned you'd need some (basic) form of locking or to have some slight staleness.

Ultimately, a ZedToken is most important for preventing the New Enemy Problem (https://authzed.com/blog/new-enemies/), and in that case, the most important place is the document itself. Otherwise, some staleness is typically a valid outcome for permission checks.

[henkosch]

But let's say I add a user to a group, so I save the returned ZedToken to the group in my database. Which would result in a transitive permission change to a lot of files as permissions are being inherited through the folder hierarchy downwards. Then I want to issue let's say a LookupSubjects request to find out who has access to a certain file, so I grab the ZedToken from the file itself or from its immediate parent folder, but that was saved way before I added that user to a group, so I would not see the new user having permission to this file unless I use the ZedToken saved in the group. But how should I know which ZedToken I should use that would be the newest?

[josephschorr]

so I would not see the new user having permission to this file unless I use the ZedToken saved in the group

You only won't see the new user having permission to the file until the caches expire; this is by default ~5s on SpiceDB.

ZedTokens/Zookies are intended to address the case where you explicitly need to guarantee freshness because, if you do not, someone who may have had access to the document's contents may be able to see the new contents added after their permission was revoked.

In your case, unless you need to guarantee against the New Enemy problem, its not necessary to have a "transitive" ZedToken. You can, of course, do so if you want to guarantee freshness at all times, but as stated above that does mean keeping an "overall" ZedToken at, say, the organization level, which we've seen users do also.

[henkosch]

Oh, interesting. I did not know about the 5s cache.
But that raises an other question for the new enemy problem: it means every time I change just the content of the file, I have to "touch" its relation to the folder just to have a new ZedToken to avoid the new enemy problem? Even though non of the relations should have to be changed...

[josephschorr]

I have to "touch" its relation to the folder just to have a new ZedToken to avoid the new enemy problem

Just issue a check for a permission to edit the document (which I imagine you'd want to do anyway to verify the caller can change the document), and then store the returned ZedToken next to the Document, not its folder

[pdewilde]

I think it is an interesting problem that affects the Zanzibar zookie concept as a whole for transitive permissions.

Imagine you have some document D that group G is allowed to edit.

If user U is in G, and then gets removed, the logical place to store the zookie is in G, but if you do a permissions check for U on D, it would pass, as the zookie you would provide would be for D.

Its not obvious how to guard against the new enemy problem if you have transitive permissions. I'm not sure there is a good solution, but would love to hear thoughts on it.

[josephschorr]

If user U is in G, and then gets removed, the logical place to store the zookie is in G, but if you do a permissions check for U on D, it would pass, as the zookie you would provide would be for D.

Yes, but that permission check result is fine unless the contents of the document have changed since the permission was removed, in which case, you'll (hopefully) have stored the updated Zookie/ZedToken on the document itself.

If the contents of the document have not changed, the user no longer having access (within the cache latency window) should be fine: they previously had access, so the contents have already been "seen" by that user.

Consider the following scenario:

1. I create a document "somedocument" and grant access to all users in group G
2. I add some new contents to the document (ZedToken gets updated here)
3. All users in group G have permission to read that document currently, so they have access to those updated contents
4. I remove user tom from group G; if he immediately tries to open somedocument, he might still be granted access because of a cached entry; however, this is fine: he previously had access to the document's contents, so nothing has been made insecure
5. Within the cache consistency window (5-30s), tom loses the ability to gain access to the document
6. I remove user sarah from group G; same behavior as above
7. Now, however, I make a change to the content of somedocument that sarah should not be able to see: this would update the ZedToken as well, preventing sarah from seeing the new, sensitive, contents

ZedTokens/Zookies are designed to handle this explicit case: the expectation that its the content of the document that matters: if it was available to the user before, leaving them access within the cache window is fine, because they could have already downloaded that information previously. In other words, they are a new enemy, not a previous one.

[pdewilde]

That makes sense. If the document content was edited, there should be a new zedtoken written to it. The zanzibar paper even says as much, guess I need some more caffeine this morning ☕

[henkosch]

These are great explanations! You helped me a lot to wrap my head around ZedTokens! Makes a lot of sense. Somehow I thought I could potentially get very old permission results if I don't use the proper ZedTokens, and that made me a bit nervous. Thank you so much @josephschorr !

[josephschorr]

@henkosch My pleasure :)

Do you think it would be worth having another document or even a YouTube video describing some of this?

[gmiklos-ltg]

Thank you for the explanation @josephschorr! A document or video would be wonderful, I think this is quite an important concept and would help newcomers immensely.

[imclem]

Thank you for the explanation.

As far as I understand, your advice is to store the ZedToken with the new version of the document this way within the 5~30s window the user which is removed from a parent folder or a group will stil have access to that particular revision but not to new versions of the document ?

What's your advice for the case where you store only the newest version of a document (no versionning), in this case the user will still be able to access the document even if the document is modified ?

One more question, I'd really like to understand what's the performance impact of using fully_consistent mode for everything ? We're not running at a scale where we will have 100 000's permission checks/write per seconds

[josephschorr]

As far as I understand, your advice is to store the ZedToken with the new version of the document this way within the 5~30s window the user which is removed from a parent folder or a group will stil have access to that particular revision but not to new versions of the document ?

They will have access to the document so long as its content (and therefore ZedToken) has not changed, which is okay, since they previously had access to it

What's your advice for the case where you store only the newest version of a document (no versionning), in this case the user will still be able to access the document even if the document is modified ?

Update the ZedToken when the document's contents change; there isn't a need to store different versions of the document

One more question, I'd really like to understand what's the performance impact of using fully_consistent mode for everything ? We're not running at a scale where we will have 100 000's permission checks/write per seconds

No caching will be used at all, which means all load will hit the datastore at all time

[imclem]

It's clear now, thanks a lot !

[svenefftinge]

How would the inverse scenario work out where I add a user ben to the group G, which should get them access to somedocument.
Since there was no change on somedocument a permission check would use the outdated token and therefore falsely deny read access, no?

[vroldanbet]

Same reasoning, zedtokens are in place to address the new enemy problem. Denying access to ben does not make it a new enemy.

What you are bringing up, however, is the read-your-writes problem, which can lead to a jarring user experience. For this kind of scenarios you could store the most recent zedtoken in a singleton resource (e.g. the organization?) so that each most recent write is there. The challenge is that you have no way to know which zedtoken is the most recent, so for this we are discussing adding support for providing multiple zedtokens in an API call, letting the server side use the most recent. You could workaround this in application for now by storing the last_updated_at column and using the most recent zedtoken attached to the top level resource.

Of course adding zedtokens at a top level resource means contention on that resource. Alternatively you can look for opportunities to call fully_consistent and then snap the zedtoken returned.

We are also in early explorations on how to make read-your-writes easier with new APIs

[svenefftinge]

Thanks! My current thinking is that we would store the token per resource on after every write and in case of writes to parent resources (which are relatively rare) we lookup all relationships from sub resources (we only have two levels) and update their token as well.
Would that make sense?

[vroldanbet]

sure! as long as you think you can scale those writes, keeping the most recent zedtoken will always work