Decentralized pallet-staking audit tool

[taqtiqa-mark]

Project Abstract

Grant level

• Level 3: Unlimited, 5 approvals

Application Checklist

• The application template has been copied and aptly renamed (project_name.md).
• I have read the application guidelines.
• Payment details have been provided (bank details via email or BTC, Ethereum (USDC/DAI) or Polkadot/Kusama (USDT) address in the application).
• The software delivered for this grant will be released under an open-source license specified in the application.
• The initial PR contains only one commit (squash and force-push if needed).
• The grant will only be announced once the first milestone has been accepted (see the announcement guidelines).
  ).
• I prefer the discussion of this application to take place in a private Element/Matrix channel. My username is: @_______:matrix.org (change the homeserver if you use a different one)

[Noc2]

Thanks a lot for the application, and I'm sorry for the delay here. I will mark it as ready for review and share it with the rest of the team. But I have one initial question: Why was the price suddenly increased from 16k to 75k? I also find the estimated workload to develop an Excel/OpenOffice workbook, as well as the daily rate, very high to be honest.

[laboon]

This information seems already available pretty easily for me - for user accounts, they can use the Staking Rewards Collector which makes a CSV file. Or systemically, you can get the data from Subscan. At this point it's a pretty simple calculation to calculate the rest of the parameters.

[taqtiqa-mark]

I think a few points are relevant.

• This deals with the aggregate figures. This is because that is where I proved there is a problem.
• I don't dispute this is trivial for you. However it does not appear to be the case generally.
• If my analysis in the previous comment is correct, then we can say that even for experts within the W3F this is not as simple as it seems. If my prior analysis is wrong - disregard this point.

I suppose the review process will tease out how representative that view is etc.

[taqtiqa-mark]

But I have one initial question: Why was the price suddenly increased from 16k to 75k?

The scope is different.

I also find the estimated workload to develop an Excel/OpenOffice workbook, as well as the daily rate, very high to be honest.

Fair enough. Can you reveal the budget you consider reasonable for this?

[taqtiqa-mark]

I also find the estimated workload to develop an Excel/OpenOffice workbook, as well as the daily rate, very high to be honest.

I'll push back here and point to a couple of pieces of evidence that suggest you are underestimating the care that needs to be taken (this is in addition to the evidence provided in the application):

• @laboon makes an assertion that some data is trivially inferred.
• Then he has to edits this claim, showing he thought the current ideal staking rate was 45%,
• now thinks it is 52.5%.
• In both cases he is eyeballing off an imaginary chart.

Not ideal for a system that invites the unsophisticated to commit in the order of USD 600-700 million.

I was the data lead on the Opal project rolling out a smart card on, rail, bus, ferry and light rail, across the whole of NSW, Australia. Throughout that roll out the system was in development. Nonetheless, there was only myself and one other analyst to support the design team, the internal reporting, paliamentary reporting, privacy commissioner, and auditor general. As well as all the transport operators.

I don't believe I have mis-estimated the work required. It is possible you may not understand what is required to build a model of a system such as this - that would also explain the current state of affairs.

My applications have been a good faith attempt to help remedy some serious defects and deficiencies.

It is getting a little tiresome to be constantly met with push-back that is either wholly ignorant of basic information (Jonas3w on the state of token economic research) or reveals in the comment the precise problem that this project seeks to address (in the case of laboon in this thread)

[taqtiqa-mark]

I also find the estimated workload to develop an Excel/OpenOffice workbook, as well as the daily rate, very high to be honest.

Fair enough. Can you reveal the budget you consider reasonable for this?

@Noc2, perhaps you could point to one or more projects of similar difficulty and applicant of similar skill and experience, where you found the budget acceptable?

[semuelle]

Thank you for the application, @taqtiqa-mark. I appreciate the level of detail you put into your analyses and argumentation. I tend to agree with my colleagues, though, that this price tag is extraordinary. We occasionally get applications with monthly rates of 12,000-13,000 USD. You are asking for almost double that.

I'm also concerned about the use of spreadsheets. Can you version control all aspects of it, i.e. the spreadsheet, scripts and tests? Is this data structured in a way that new users could pick changes apart? Please correct me if I'm wrong, but an open source data analysis/scripting tool like R's Notebook/Markdown, Shiny or Jupyter might work better in this regard. It could also be hosted on a server for those who don't mind, which would help with verifying results when, inevitably, there will be iterations and multiple versions of it floating around.

[taqtiqa-mark]

Thank you for the application, @taqtiqa-mark.

No worries.

I tend to agree with my colleagues, though, that this price tag is extraordinary.

Sure, as I asked @noc, please point to applications you consider to be comparable and applicants that have comparable skill and expereince.

To save an iteration: I believe the comparable project would need to address an issue that if unaddressed posed a threat to project similar to the issue this application addresses.
Specifically,

1. Investor rewards were altered by 25% with no announcement or warning. There is no reason to believe further such changes aren't possible. If users loose confidence in this aspect of the project they will seek alternatives. This tool proposes a tool where the most naive user can use on almost any desktop without installing additional software (Python, Jupyter, etc.) and confirm the rewards/dilution they are engaging with.
2. While this proposal is and will remain pitched at the LCD end user, it could (should) lead to a radical change in the development work flow of a central element of any relay chain (since it will reside in the pallet-staking crate) - no change should be made to the reward/dilution code without first being made to the audit workbook. Once such a change is made, and socialized from there, it can be implemented and the implementation validated against the audit tool results.

I'm also concerned about the use of spreadsheets. Can you version control all aspects of it, i.e. the spreadsheet, scripts and tests?

I could store the XML version for the workbook in the repository. Workbook tests are mocked in the proposal. The script tests are addressed in the application (bats).

Is this data structured in a way that new users could pick changes apart?

As a workbook they could, ideally pick every calculation apart. The data storage is an issue - mainly because there aren't (currently) clean ways to ingest blockchain data (see future/extension section). Ideally, a chain would provide an audit record containing all the on-chain data relevant to validating/auditing the chain operation. But this is a chicken and egg problem - there is no widely distributed audit tool that needs and audit record. If this tool proves popular maybe a chain will step forward to meet the need. Who knows - right now it is proving very difficult to persuade people one would expect to need the least persuasion :)

Please correct me if I'm wrong, but an open source data analysis/scripting tool like R's Notebook/Markdown, Shiny or Jupyter might work better in this regard.

No. Again you're targeting the tech-bro ghetto. I'm targeting the simplest user setup that is feasible to put the actual calculations in the hands of, independent of any mediating party (server). Think an SME inside a regulator monitoring the conduct of chain developers/promoters - concerned about them diluting the holdings of one group less involved in the code base/governance for the benefit of another group more involved in the code base/governance ;)

To be clear, on my experience a company making an unannounced change of 25% in their 'stakholders' rewards/dilution, would lead to a Wells notice or some such equivalent.

It could also be hosted on a server for those who don't mind, ...

This is the usual argument for centralization - developer convenience. IIUC, it buries the problem. Personally I would have my chain audit tool front and centre. Changes to their rewards/dilution exposure are a critical feature to alert users about, not a nuisance to hide behind a server.

... which would help with verifying results when, inevitably, there will be iterations and multiple versions of it floating around.

As above these are critical details to be exposed and discussed. If your rewards/dilution changes so frequently that it challenges an end users ability to understand and keep up with then you really need to address that, rather then hide those changes from them behind a centralized server they will likely never visit.

[taqtiqa-mark]

I will mark it as ready for review and share it with the rest of the team.

@Noc2 could you clarify one thing for me. Does approval require reviewer unanimity or just a majority approval?

In the case of unanimity, addressing reviewer B's objections becomes moot if you haven't persuaded Reviewer A.

[semuelle]

Does approval require reviewer unanimity or just a majority approval?

Neither. Five committee members must approve a Level 3 application, regardless of what other committee members think.

[taqtiqa-mark]

Does approval require reviewer unanimity or just a majority approval?

Neither. Five committee members must approve a Level 3 application, regardless of what other committee members think.

Hmmm, okay now I'm confused by the neither statement. So there are committee members and reviewers? What is the relation between them, and who is providing feedback here?

[laboon]

Does approval require reviewer unanimity or just a majority approval?

Neither. Five committee members must approve a Level 3 application, regardless of what other committee members think.

Hmmm, okay now I'm confused by the neither statement. So there are committee members and reviewers? What is the relation between them, and who is providing feedback here?

By "neither", he means that neither of the cases you mentioned are correct - neither unanimity nor majority approval is required. Rather, a certain number of committee members must approve a grant, based on the level of the grant.

Anyone can "review" a grant, so the term "reviewer" you used is ambiguous, but correcting it earlier probably seemed a bit pedantic and people assumed you meant "Grants Committee member". But to clarify, the term "reviewer" is not terminology in the W3F Grants Program used except in the informal sense of "one reviewing an application".

You can see who is on the Grants Committee in the README of this repo: https://github.com/w3f/grants-Program/#w3f-grants-committee

Web3 Foundation also has Grant Evaluators who have been hired to focus specifically on evaluating grants. You can see a list of them here: https://github.com/w3f/grants-Program/#w3f-grants-evaluators External evaluations can also be done, but these must be reviewed by W3F Grant Evaluators.

Currently, all Grant Evaluators are also on the Grants Committee, but this is not necessarily the case. See the links above for more details on how it works.

[takahser]

@taqtiqa-mark thanks for your interest in our grants program. However, I have some concerns about the proposal:

• An Excel implementation would not meet the open source requirements of the Grants Program. Here's a quote from the guidelines:

  All code produced as part of a grant must be open-sourced, and it must also not rely on closed-source software for full functionality.

• An OpenOffice implementation is also not ideal since OpenOffice is a discontinued project.

• I would recommend exploring a web interface tailored to this use case rather than a spreadsheet program. Binary workbook files make version control and decentralization more challenging compared to source code. Also, a web UI has the potential to be much more user-friendly and accessible, also w.r.t. device compatibility.

• A web app hosted on IPFS could provide many of the desired decentralization benefits while being easier to version and update.

• The requested budget of $75k USD seems excessive compared to grants of similar scope. For reference of what other teams charged in the past, I would encourage reviewing the list of previously funded projects and rates here.

• Finally, there are some unresolved concerns related to the necessity and sustainability of a tool like this - Existing solutions can already provide the required data, and major upcoming changes to the network also risk quickly rendering a new tool outdated.

For these reasons, I'm going to abstain from approving this proposal.

[semuelle]

Hi again, @taqtiqa-mark. Thanks again for your application. After careful deliberation, the committee has decided not to support your application at this point. I want to reiterate that everyone appreciates the time and effort you put into your research and application. I hope you'll still be able to make this project a reality. In any case, best of luck and feel free to apply again in the future.