Create data_platform_with_deep_indexed_data_and_staking_reports_milestone_1.md

[bigslimdogg]

Create data_platform_with_deep_indexed_data_and_staking_reports_milestone_1

Final solution with 1st (and the last) milestone

Milestone Delivery Checklist

• The milestone-delivery-template.md has been copied and updated.
• This pull request is being made by the same account as the accepted application.
• I have disclosed any and all sources of reused code in the submitted repositories and have done my due diligence to meet its license requirements.
• In case of acceptance, an invoice must be submitted and the payment will be transferred to the BTC/ETH/fiat account provided in the application.
• The delivery is according to the Guidelines for Milestone Deliverables.

Link to the application pull request: w3f/Grants-Program#1866 < please fill this in with the PR number of your application.

[keeganquigley]

Updated to include application PR.

[keeganquigley]

Thanks for the delivery @bigslimdogg in the meantime can you please change the filename so that it is in .md format? Thanks.

[keeganquigley]

Also, the infrastructure repo doesn't appear to exist. Did this move?

In addition, I'm realizing that the documentation link doesn't work. Did you mean to link to your personal GH account?

[bigslimdogg]

Also, the infrastructure repo doesn't appear to exist. Did this move?

In addition, I'm realizing that the documentation link doesn't work. Did you mean to link to your personal GH account?

Hey, sorry for that - I've published the repo.

Could you please help what is wrong with the name of the file?
Currently it named as "data_platform_with_deep_indexed_data_and_staking_reports-milestone_1.md"
So it is <name_of_the_project>-<milestone_1>.md
I guess it is correct but the checks didn't pass

[keeganquigley]

Thanks @bigslimdogg looks good now, don't worry about the checks.

[keeganquigley]

Thanks for the changes @bigslimdogg please see my further comments below.

• Can you fix the Milestone Number near the top, so that it says Milestone Number: 1 instead of e.g.
• 0b. Documentation link still gives a 404 error.
• Deliverable 2: Publish DWH Data also gives a 404 error. Can you fix these?

[keeganquigley]

I'm also not seeing any instructions for how to use the Apache Superset dashboard. For example, what is guser dash? How can I create a test dashboard and test chart?

Also are there any unit tests and or e2e tests you can point me to?

[bigslimdogg]

• Can you fix the Milestone Number near the top, so that it says Milestone Number: 1 instead of e.g.

Done!

• 0b. Documentation link still gives a 404 error.

Done!

• Deliverable 2: Publish DWH Data also gives a 404 error. Can you fix these?

Done!

how to use the Apache Superset dashboard

I've done video instruction: https://www.youtube.com/watch?v=kJ5E2WIbXL4. Is it work for you?

Also are there any unit tests and or e2e tests you can point me to?

We could do it fast with docker but we've chosen Terraform and end2end would take a lot more time (+1 month of 100% of devOps). So it could be tested though launching on any infra with configs we provided. Is it work as well?

[keeganquigley]

Thanks for the changes @bigslimdogg much appreciated.

• Is there a reason the repo is called "Polkadot Grant" instead of "Polkadot Monitoring System"? Are you planning to maintain this after the grant?
• Changes to the grant need to be made via an amendment. Technically, I could reject the delivery due to the lack of a Docker deliverable. We expect an easy way to spin up the application and run tests, and Docker provides a way to do that without needing to install a bunch of stuff. Docker does work with GKE as well. That being said, I will continue to work through the given instructions for now, but it has already proven to be rather time consuming.

[keeganquigley]

As an example, I'm not sure what settings I need here to set up the Kubernetes engine. Should the application type be Web Application?

[keeganquigley]

I was able get to the part where I'm setting up the Google service accounts. I'm assuming that I need to replace the boilerplate text here:

gcloud iam service-accounts create <SERVICE_ACCOUNT_NAME> system --description "Polkadot monitoring system's service account"
gcloud projects add-iam-policy-binding <PROJECT_ID> --member "serviceAccount:<SERVICE_ACCOUNT_NAME>@YOUR_PROJECT_ID.iam.gserviceaccount.com" --role "roles/bigquery.dataEditor"
gcloud projects add-iam-policy-binding <PROJECT_ID> --member "serviceAccount:<SERVICE_ACCOUNT_NAME>@YOUR_PROJECT_ID.iam.gserviceaccount.com" --role "roles/storage.objectUser"
gcloud iam service-accounts keys create key.json --iam-account <SERVICE_ACCOUNT_NAME>@<PROJECT_ID>.iam.gserviceaccount.com
zsh: no such file or directory: SERVICE_ACCOUNT_NAME
zsh: no such file or directory: PROJECT_ID
zsh: no such file or directory: PROJECT_ID
zsh: no such file or directory: SERVICE_ACCOUNT_NAME

But when I try to create a test it gives me this:

gcloud iam service-accounts create my-service-account system --description "Polkadot monitoring system's service account"

ERROR: (gcloud.iam.service-accounts.create) unrecognized arguments: system

To search the help text of gcloud commands, run:
  gcloud help -- SEARCH_TERMS

Should I be entering an email address for the service account? I'm not sure how these key types work. Thanks for any assistance you can provide!

[bigslimdogg]

• Is there a reason the repo is called "Polkadot Grant" instead of "Polkadot Monitoring System"? Are you planning to maintain this after the grant?

We are continue to maintain it but we thought about apply on the next grant to build rated.network dashboard but for Polkadot. These next grant supposed to cover some infra costs (1-2k $)

• Changes to the grant need to be made via an amendment. Technically, I could reject the delivery due to the lack of a Docker deliverable. We expect an easy way to spin up the application and run tests, and Docker provides a way to do that without needing to install a bunch of stuff. Docker does work with GKE as well. That being said, I will continue to work through the given instructions for now, but it has already proven to be rather time consuming.

Yeah, I'm agree but on the other side we didn't want to provide solution which actually impossible to use in production. Polkadot has a lot of data which make it more sense to build ETL in Kuber.

As an example, I'm not sure what settings I need here to set up the Kubernetes engine. Should the application type be Web Application?

Everything is managed by terraform, you don't need to create anything with your hands. Can you specify how you get there so I can provide more info?

Should I be entering an email address for the service account? I'm not sure how these key types work. Thanks for any assistance you can provide!

Our bad, we fixed it. Check this out!

Sorry for delay in response!

[bigslimdogg]

@keeganquigley If it is hard to use we can provide instructions with video if it works for you. Because it doesn't supposed to be so hard - everything is automated with Terraform

[keeganquigley]

Thanks for your answers @bigslimdogg yes if you could create a tutorial video that would be great; will hopefully answer some of my config questions. Thank you!

• Everything is managed by terraform, you don't need to create anything with your hands. Can you specify how you get there so I can provide more info?

Understood, thanks. This was just in getting the prerequisites ready, as I had to manually enable all the necessary APIs for my Google account. I'm basically just trying to get through steps 1. and 2. here in order to get the point where I can run terraform init.

That's great if Terraform can do some of this initial setup automatically but I'm not seeing any instructions for how to set it up with Terraform. Does that mean Terraform will update the values.yaml file automatically as well? Or do these setup instructions still apply? Thanks!

[bigslimdogg]

Hey @keeganquigley

1. we updated the project and now it runs by only one line
2. we published the video to launch solution https://www.youtube.com/watch?v=paZ6utMdRHw

Is it works for you?

[bigslimdogg]

@keeganquigley what are the next steps?

[keeganquigley]

Thanks @bigslimdogg looking at this now.

[keeganquigley]

Thanks much for the video @bigslimdogg it is very helpful. I made it down to step 5, but am getting an error when running terraform init:

Initializing the backend...
Initializing modules...
╷
│ Error: Failed to get existing workspaces: querying Cloud Storage failed: storage: bucket doesn't exist

Which is strange because it does exist:

I tried terraform init -reconfigure but no luck. Also tried these solutions to no avail. My permissions are set the same as yours:

Any ideas? Apparently maybe the bucket has to pre-exist? In the meantime I will keep troubleshooting. Thanks for any guidance you can provide!

[bigslimdogg]

@keeganquigley can you share the code you've tried to run? It may be some modifications that lead to that

[p2paltvnv]

Thanks much for the video @bigslimdogg it is very helpful. I made it down to step 5, but am getting an error when running terraform init:

Initializing the backend...
Initializing modules...
╷
│ Error: Failed to get existing workspaces: querying Cloud Storage failed: storage: bucket doesn't exist

Which is strange because it does exist:

I tried `terraform init -reconfigure` but no luck. Also tried [these solutions](https://stackoverflow.com/questions/59053993/failed-to-get-existing-workspaces-querying-cloud-storage-failed-storage-bucke) to no avail. My permissions are set the same as yours: Any ideas? Apparently maybe the bucket has to pre-exist? In the meantime I will keep troubleshooting. Thanks for any guidance you can provide!

It may indicate that you do not have a bucket for state-file created. Basically, it's an optional step (4) and can be skipped. Can you try to proceed with clean repository and update values.yaml file only?

[keeganquigley]

Thanks @p2paltvnv much appreciated, I figured it might be something small. I will try that!

[keeganquigley]

Thanks for your help @p2paltvnv I was able to get it running by only changing values.yaml and also removing the Public Access Protection from the bucket:

Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

I will update the eval accordingly.

[keeganquigley]

Thanks again for your help @bigslimdogg I was able to get through the guides and run everything as expected. Here is my final evaluation. I'm good to go ahead and pass the milestone, congrats!

[github-actions]

🪙 Please fill out the invoice form in order to initiate the payment process. Thank you!

[github-actions]

Congratulations on completing the first milestone of this grant! As part of the Grants Program, we want to help grant recipients acknowledge their grants publicly. To that end, we've created a badge for projects that successfully deliver their first milestone. Please use the badge only in reference to the work that has been completed as part of this grant, so please do not display it on your team or project's homepage unless accompanied by a short description of the grant. Furthermore, you're now welcome to announce the grant publicly. Please remember to observe the foundation's guidelines in doing so. If you haven't already, reach out to [email protected] for feedback on your announcement and cross-promotion.

Thank you for your contribution, and good luck! If you have any remaining milestone, let us know if you encounter any delays by leaving a comment on the application PR or submitting an amendment.

[github-actions]

We noticed that this is the last milestone of your project. Congratulations on completing your grant! 🎊

So, where to from here? First of all, you should join our Grants Community chat, if you haven't already, so we can stay in touch.
If you are looking for continuative support for your project, there are quite a few options. The main goal of the W3F grants program is to support research as well as early-stage technical projects. If your project still falls under one of those categories, you might want to apply for a follow-up grant. However, depending on your goals and project status, there are other support programs in our ecosystem that might be better suited as the next step. For example, projects with a Business Case/Token should look into the Substrate Builders Program or VC Funding and Common Good projects have a good chance of receiving Treasury Funding. If you are looking for guidance, the team at https://substrate.io/ecosystem/square-one/ has compiled a list of ecosystem support sources and are happy to help you navigate it.

For a more comprehensive list, see our Alternative Funding page. Let us know if you have any questions regarding the above. We are more than happy to point you to additional resources and help you determine the best course of action.
Lastly, we hope your W3F grant was a success and we want to thank you for being part of the journey!

[meldien]

@bigslimdogg the payment was executed yesterday