project scope and other questions

[sido420]

Thanks for the hard work on this project.

I wanted to know the scope of this project. Specifically, with regards to support for Safe Browsing API.

Does it support all features that Safe Browsing API provides? Also, are database updates and any maintenance etc taken care of by sbserver automatically? In other words, is there anything that I must take care of in application/client side myself?

I was not sure what is meant by "The server also has a lightweight implementation of the API v4 threatMatches endpoint."

Also, how do I configure the server and where are local database files stored. I guess, more details in documentation might be helpful for new users.

[sido420]

Just to clarify, I'll most likely be using it as a Docker service. Hence, its important for me to know where files are stored and which ports etc to open/redirect traffic to.

[colonelxc]

The code does automatically manage requesting DB updates.

By default, the local DB is not persisted. If you pass in a configuration value for that, it will save (and load from) the given path. If you are running the sbserver binary, the flag is called 'db'. If you are using it as a library, you can pass it in the Config struct when calling NewSafebrowser

Additionally, if you're using it as a library, you can call the function WaitUntilReady to block your code until the db is in a good state (not stale, initialized).

The line about "lightweight implementation of the APIv4" refers to sbserver actually serving its own lightweight API. So you could use it like a 'caching proxy' almost for the real API. This could potentially help reduce the number of API requests made (as opposed to every client calling the SB api directly). If you're using sbserver as a redirector, or using the code as a library, you don't need to care about that.

[colonelxc]

As for the question, "does it support all features that the Safe Browsing API provides", the answer is yes and no.

The Safe Browsing API offers two different ways to interact with it. One way downloads lists of hash prefix (the "database"), and uses those as a first filtering step (most urls will not match a hash prefix, so no further action is taken). When a match is made, this client (and other clients implementing this protocol, like Firefox, Chrome, and Safari), it just sends the matching hash prefix. The API returns the full hash to match against and the client makes the comparison. This is privacy preserving, as you only expose the first 4 bytes of your matching hash (not the real url visited), which wont necessarily match up with the url we originally hashed. This is called the Update API

The other part is the Lookup API. It is much simpler, as you just send the urls you want to check, along with categories of threat types, etc. It is obviously not privacy preserving.

This client only integrates with the privacy-preserving Update API. As far as being a fully-featured client, it is pretty good. It is a known issue that this client doesn't respect the request backoff rules for url requests (just with DB update requests). This doesn't have any practical detriment on the client, though it would be nice if it followed our own rules :-).