-
Notifications
You must be signed in to change notification settings - Fork 5.7k
Team Members
As a team member, you'll find in this section all the guides that relate to the general workflow as well as the taxonomy of tasks you can undertake.
Generally, ongoing communication-oriented tasks are given higher priority over the individual one-time tasks. The best way to work is to prioritize getting up to speed with the ongoing tasks first, and then work on one-time tasks once you are caught up with the ongoing tasks. This approach ensures that people in the community are never left without a response for too long. Consequently, you should follow the below list when giving priorities to your tasks, by order of most urgent to least:
- Review assigned PRs
- Address assigned issues
- Reply on Discord
- Respond to mails
- Individual Tasks
- Fix Tests
- Implement supersets
- Fix-up functional submodules
- Fix-up stateful submodules
- Implement front-ends
To make it easier to find issues of interest to work on, you can use the "Projects" tab in the repository which divides the currently open issues into the following groups:
- Individual Tasks: Includes all issues that were opened that relate to individual tasks requiring the creation of PRs at a certain point. This section is mostly useful for reviewers who want to keep track of the progress in the task they opened.
- Shared Tasks: Groups all the shared tasks lists and their instructions, except for the reviewing tasks as well as front-ends and extensions implementations, for which you can find more details below
- Jax (resp Numpy, Tensorflow, Torch) Frontends : Includes all the Jax (resp Numpy, Tensorflow, Torch) Frontend To-do lists
When working on functions implementations, you can refer to the different ToDo list issues which are used to group high-level tasks into a list of sub-tasks that you can work on individually. Applicants must comment on the ToDo list issue in order to get their issue added to the todo list, as they do not have write privileges on the main ToDo list comment. However, as members of the Ivy team, we do all have access to modify the ToDo list comment. Therefore, when self-assigning a sub-task to work on, you should simply modify this list directly, replacing the sub-task text with the issue reference directly, rather than adding a new comment at the bottom.
When finding unit tests to fix, the best approach is to check out the CI dashboards located here, on the “dashboard” branch of the repo. The ReadMe provides links to each of the dashboards, for Functional Core, Functional NN and Stateful. It is recommended to add a bookmark folder in your browser, which contains the individual bookmarks to each dashboard inside. This allows you to check the status of all tests within seconds. But again, if you have a different way of organizing yourself feel free to work whichever way you want !
Also, because of the use of GitHub action artifacts, when tests fail the failing Check on failures tab will very often simply show the unhelpful message “Run exit 1, Error: Process completed with exit code 1”
. Instead, you will need to look in the “Run <Name> Tests”
tab, and there you will find a breakdown of the tests that failed.
When you create a pull request, your point of contact will automatically be assigned to it. You don’t need to request a review, you are totally entitled to merge your own PRs without a second opinion. If you would like for it to be reviewed though, then you are equally very welcome to request one. If the PR is for a shared task, then you should request the review from the PR assignee (your point of contact). If the task is an individual task, you should instead request a review from the author of the task, instead of your point of contact. In either case, you should not request a review from anyone else in the team, such as myself (unless I am your point of contact or task author).
If your original reviewer is unable to review your PR, or they have questions themselves, then they will then be permitted to request a review from their own point of contact, and so on.
Firstly, you should never create branches on the main repository. You should always work from your own fork, and create any new branches on your fork directly. If you are working with others, then you will simply need to give them write access to your own fork or branch.
Aside from this, there are a few other things to check locally before you commit to master or merge a PR, and after, on the CI tests, in order to avoid introducing new failures.
Before Committing/Merging
- Check for formatting errors: lint and flake8
The lint test is one of the most frequently failing tests on the CI, from both PRs and direct pushes to master. Do you have the pre-commit hook installed to help you catch these before pushing ? If not:
- Go to the terminal/command line in PyCharm/VSCode
- Run pip install pre-commit
- Run pre-commit install Then it should work ! The next time you try to commit, a message will pop up stating whether these two formatting checks have passed. If one of the tests fails, some reformatting is required. Sometimes the re-formatting will be automatic, and in that case, you just need to re-stage the changed file and commit again (add [file] -> commit -m "[message]"). If reformatting is required by you, simply make the changes, add the changed file again and make the commit again. You can see whether or not the commit has successfully been made by checking git log. This is also detailed in the docs.
- Check for failing unit tests
It can be quite laborious to run every single possible test that could be affected by your changes locally, so this is not expected. However, if you have made changes to a backend function, it makes sense to run both the ivy unit test and the array-api test (if it exists) for that function before pushing to master. If you're working on a PR, you can easily check for newly failing tests on the CI by looking for Xs beside the test runs. Sometimes one or more of the tests will be failing on master too. This does not mean that you should not check if your changes have added to the failures.
- Pull the most recent changes from master frequently.
Resolving merge conflicts on pushing to master can be tiresome. It is possible to avoid this most of the time by pulling from master frequently and using stashing:
- Before making any changes to your repo when you sit down to work, run git pull origin master (or similar, make sure origin is set to https://github.com/unifyai/ivy.git by running git remote -v). This ensures that you're working on the latest version of master and helps to reduce possible merge conflicts.
- You've worked on an issue for a while, have made sure it doesn't cause obvious test failures, and you now want to commit these changes.
- Run git pull origin master again. Your local repo may update seamlessly, or there may be a conflict warning. Do not fear! You do not necessarily need to resolve the conflict.
- Simply run git stash , then git pull origin master again.
- You may notice that your recent changes have disappeared. Do not fear! To get them back, run git stash apply.
- You can now add + commit your changes on top of the most recent pull from master, and then push to master without issue.
- You can also use the convenient shell script made by Dan for this procedure stash_pull.sh. More details on using git stash can be found in the docs.
- Check what changes you’re committing
Sometimes files you have changed locally but do not want pushed to master can be accidentally staged for committing. This can easily happen if you use git add -a without first checking what files have been edited with git status. A good way to avoid this is adding files one by one after you’re sure you want them to be committed. If you realise that you have staged files you do not want committed, simply run git restore --staged [filepath] to unstage the file.
You should always check at least what files are included in your commit before pushing with git log --stat. You can check the details of your commit with git log -p. If there are files in the commit which you do not want to push, undo the change in a subsequent commit and then squash the two commits.
After Committing + Pushing to Master / Merging PR
Once you’ve pushed to master, the CI tests will run. Some may take 1hr+, so it is important to check back on them. At the time of writing, there are ~14 tests running on each commit. We’ll touch on some of the most important ones to check here.
The fastest ones to run are the lint, test-docstrings and test-numpy-style-docstrings.
- These ones are both easy to get passing and important for reviewing PRs, so we should try and keep them passing always.
- If you notice that one of these tests is failing on a recent commit you’ve made, try to address the issue immediately.
- You can view the error messages generated by the test by going into Details -> Run [x] Tests. Simply search the output for the function(s)/ file(s) you edited in your commit to check if you’ve caused any new failures.
- If it’s failing but the issue does not originate with your commit, try to track down where it started. You can either a) fix it yourself or b) ask the committer to fix it or c) post the commit link on the general internal chat and state that you’ve noticed a failure here. Slower running tests include the four array-api tests and test-ivy.
- Due to the nature of hypothesis testing, failures may pop up for no obvious reason (i.e. seemingly unrelated to your commit or the commit on which the failure originated). That is OK. Simply raise the issue of a newly failing CI check on one of the google threads (array-api thread or ivy tests thread). NB: some of these tests have been failing for quite some time, but it is still necessary to check the output of the test to see if your commits are causing any new failures. This is especially important if you are working on backend functions or on the submodule fix-up task.
- You can view the error messages generated by the test by going into Details -> Run [x] Tests. Simply search the output for the function(s)/ file(s) you edited in your commit to check if you’ve caused any new failures.
- If you are unsure how to fix the new failure, reach out to the rest of the team on the appropriate google chat thread.
Small Fixes If at any point you spot a small bug in Ivy which has nothing to do with your individual task or your shared task, you are more than welcome to quickly push a fix for this! If you’re not 100% sure if the fix is correct, then you can ask your point-of-contact about it, or create a PR. The main point being: you should not feel limited to the realm of your individual or shared tasks when it comes to small quick and easy fixes. While it’s on your radar, you might as well quickly push a fix !
For dealing with individual tasks, we've decided to use Trello boards as it makes managing and attributing tasks easier. This section will explain how to interact with the board. Firstly, do not create or delete any custom fields even if you aren't using them, as this would delete the field from all the cards and dashboards. Instructions specific to trello:
- When a new task is created , the author of the task needs to create a card in the unassigned list, filling up all the required details and adding the appropriate labels.
- If someone is interested in a task, they should enter their name in the designated custom field "interested".
- When the task's author assigns an individual to a task, they should add them as a member to the card. This would automatically add the card to the adjacent list "Todo".
- If a task is present in either the Todo or "WIP" list, you are encouraged to select one of the options from the status fields indicating your progress on that particular task.
- Once a task is completed, you can just drag and drop your card to the done list, which would also update the card's status to "done."
- If you would like to see all the cards that are assigned to you, you could use the filter and select the option "Cards assigned to me", which would display all the cards that have been assigned to you.If you have any cards assigned to you in the “Awaiting Review” list then most likely you have a PR assigned to you awaiting your review. Once you have reviewed the PR assigned to you , you could change the Status to either “Approved” or “Reviewed”.
- If anyone has been assigned an individual task for which you are required to create a PR and would want to get it reviewed by your reviewer, you would need to first change the status field to “In Review” and add your reviewer as a member to your card.
Creation
Everyone on the team is totally entitled and is encouraged to author their own tasks in the individual task sheet. This applies both to senior members of the team and also to fresh interns. With so many pairs of eyes working on different areas of the code, general problems with parts of the code, new things to fix, or better design decisions can reveal themselves to anyone within the team. If/when this happens, you do not need to message me or your point of contact about it. Just explain it in a new individual task. You can then either assign yourself and get to work, or leave it for someone else if you are in the middle of something else or if your plate is full in general. Worst case scenario the task is not necessary or there is a flaw in your logic, but this is very easy to undo if that’s the case. It is much better that you proactively add new tasks (which understandably might include some mistakes) than wait for Dan to give the all-clear on every task before it gets added 🙂 If you’re not sure, you are welcome to add the task already, and then also ask either Dan or your point of contact if they think anything should be improved with the description etc.
Interest
You should regularly check this spreadsheet for new tasks, and when you find a task that you like the sound of, you should add your name in the “Interested” Custom Field to flag that you are interested in working on this task. You could still add your name beside other’s names in the interested field, you are not restricted to add your name to tasks which have no one in the “Interested” field.
You are encouraged to continue adding your name under Interested even after the task has been assigned. Sometimes individual tasks are changed to paired tasks halfway through, based on the complexity of the task and the amount of requested tasks among the team. Therefore, you might still be assigned to help out with the task eventually.
Assignment
Once a task has been on the dashboard for at least a few hours, If you are the only person in the “Interested” field and you have no other individual task that is either a Work in Progress or a Todo task, feel free to add yourself as a member to the relevant card.
Status
If you have been assigned to an Individual Task and it’s either in the “ToDo” Or “Work In Progress” List you are encouraged to update the Status of your task.(Select a Status from the drop-down which is closest to the current status of your task) Available Statuses include:
- Todo
- In Progress
- Done
- In Review (Only for PRs and Issues which are awaiting review from the assignee). Once you add this status to your card your card will be moved to the “Awaiting Review” List, after which you are also supposed to add your assignee as a member to the card, which would notify them)
- Approved/Reviewed
- Not Sure
- Started
- On Hold
- Almost Done
Outcome
Once the task is complete, you should add an explanation of the outcome under the “Outcome” field. This should ideally include links to the relevant commits and/or files which show the completion of the task. You should only add content into the “Outcome” field once the task is completed. Please make sure to share the file with the task author so they can view the file.
Comments
If you are having difficulties, or would like to explain why things are taking longer than expected, you can add this information under the “Comments” field. You should not add it under the “Outcome” field.
Filter Tasks Assigned to You
To navigate to the tasks which have been assigned to you, you could use Trello’s in built filter feature to view the cards that have been assigned to you .
Offering Help
Even though these tasks are generally assigned on an individual basis, everyone is both welcome and encouraged to send messages to anyone who is working on a task which you are also interested in working on. Simply directly message them something along the lines of “Hey, I saw that you were working on <task_name>, and I was wondering if you could use some help?”. If they don’t need any help then you can find another task and message that person instead, or simply continue with shared tasks. It’s always at the discretion of the original task assignee to decide if they’d like to open the task up and collaborate on it, and in many cases it might simply not make sense if the task is near completion or involves a lot of prior explaining which might make collaboration and unproductive choice for the task.
Requesting Help
It’s very easy to hit dead-ends or roadblocks when working on your tasks, and it’s easy to spend many hours banging your head against the wall trying to get things working! This is not a good outcome for anybody. You should always be able to ask for and receive help on the tasks you’re working on 🙂
If you are having issues resolving some particular aspect of either your shared tasks or your individual task, then you should add a description of the problems you are facing to the “Help Requested” List in the. Anyone is welcome to write their name under the “Helpers” column, and then reach out to the author of the request directly, to help them with their issue !
Besides the individual tasks, a number of “shared tasks” are available for you to work on. Some of them are recurring tasks that pertain to your day-to-day work as a member of the team, and others are created to answer a specific need that may arise as Ivy grows. These latter tasks are intended to be worked on as small groups and will have you engage and share with other team members as you progress towards the tasks’ completion.
When reviewing pull requests (PRs), there are a couple of things to watch out for:
- When a PR is first created, someone from our Research Engineer team will be randomly assigned to the PR, but the author of the PR cannot directly request a review from the PR assignee. Therefore, you should always first filter for PRs with:
- Reviews -> “Not reviewed by you”
- Assignee -> your account These are the most important PRs to give attention, as they have received no feedback
- You should always start on the oldest PR first, you can sort for this on GitHub
- If there are no PRs which meet these two criteria, then you should filter for:
- Reviews -> “Awaiting review from you”
- Assignee -> your account If there are also no PRs meeting this criteria, then you have no PRs to review!
- If the PR is for a bullet point in a ToDo list issue, then check if the PR author has commented on the issue with “Close #Issue_number”. If they have not added this comment, then you should comment “Hey <@their_github_id> 🙂, I will link this to the corresponding issue, but in the future you should do this yourself by adding a comment ‘Close #Issue_number’, as explained here”
- Link the associated issue to the PR manually if the comment has not been added by the author. This can be done on the right-hand pane at the bottom, under “Development”. This ensures that the issue is closed upon the PR being merged, which in turn will mark the item as complete in the ToDo list issue. If the author has commented “Close #Issue_number”, then the PR should have already been correctly linked, but you should still verify that this is the case.
- Add the correct labels to the PR from the selection of: Array API, Function Reformatting, NumPy Frontend, TensorFlow Frontend, PyTorch Frontend, JAX Frontend, Ivy Functional API. This is very important, and is needed in order for CI to run on the PR.
- Make sure the commit does not include any changes in the IDE configuration, for example changes to .idea folder. The changes should all be relevant only to the function being worked on.
- Make sure the formatting is consistent with the formatting in the contributor guide.
- It is often necessary to resolve conflicts. If these cannot be resolved in the browser then you will need to clone the fork, resolve in your own terminal, and then push to the fork. The easiest way is to just run
./merge_with_upstream.sh name_of_branch
. This will either complete the merge and then push, or it will fail with conflicts for you to resolve manually. In the latter case, after resolving manually, you’ll just need to do:
git add -A
git commit -m “some message”
git push
- When the PR is ready to be merged, you should choose the option “Squash and merge”. Please do not select “Create a merge commit” or “Rebase and merge”.
- If the tests are failing, but this is not the fault of the contributor for this PR, then we should still accept this PR.
- If the PR overlaps with some work we have been doing internally, this is not the fault of the PR author, and so we should still merge the PR, and then we can simply incorporate our own changes on top of theirs if needed. We should not ask them to choose another task, and we should not close the PR because of this overlap.
- You should not need to hand-hold at all when reviewing PRs. If the author of the PR asks questions like “what should I do now?”, then you can simply reply “please read the contributor guide and Deep Dive sections to see what else is needed”, with links provided in your response. If their PR contains many errors, then feel free to provide a response with one or two sentences quickly explaining the main issues, and then direct them to the docs. You should not spend time painstakingly going through a poor PR, and request a ton of required changes for each line! It shouldn’t demand too much of your own valuable time !
- The bullet-point style commit history in the body should all be deleted. For each person involved in reviewing or commenting on the PR, you should then add "Co-authored-by: your_github_name <your_github_email>" for this person on a new line. The exception to this rule is Dan's GitHub account (djl11), who doen't want to be listed as co-author on PRs regardless of his involvement (He has enough commits to his name already 😅). These lines should be added into the now empty body (following the deletion of all content). Once merged, all names will then appear as co- authors of the commit. In cases where multiple people review and/or contribute commits towards the same PR, then you should list each person side-by-side in the Co-authored by message, like so "Co-authored-by: your_github_name <your_github_email>, their_github_name <their_github_email>, … etc".
- In order to avoid the PR appearing in the PR Dashboard, you should make sure to always respond to the authors messages in some form, so that they are never the last person to message on the PR (which will always cause it to appear in the dashboard). If they have said something which doesn’t warrant a response such as “okay, I will work on it”, then you can simply reply with a 👍 to their message, and this will register as a response. This will keep your PRs away from the dashboard, and keep you in the clear so none of us need to chase after you !
When reviewing reformatting pull requests (PRs), on top of the standard steps to follow, we should also pay particular attention to the contributor guide. The sections Type Hints, Docstrings and Docstring Examples are particularly important.
We should ensure that the applicant has produced the reformatting checklist successfully. Then, the applicant will mark items of the checklist according to the LEGEND section.
As reviewers, for items marked:
- ❌ : We do nothing until they are completed.
- ✅ : We check the item’s implementation in the ‘files changed’ tab.
- 🆘 : We address the applicant’s point of struggle/check their implementation for errors.
- ⏩ : We check whether the checklist item should be skipped.
- 🆗 : We check the implementation of the checklist item in the original ivy repo.
When a check item is confirmed as complete by the reviewer, we should mark the check box next to it indicating that this item was integrated successfully. We should also make sure that the docstring tests are passing for the particular function being modified in the PR. This is very important.
Adding the reformatting checklist to a PR
The PR author should add a comment on their PR with the format "add_reformatting_checklist_<category_name>", where <category_name> is the category name of the function addressed in the PR. E.g. for the function ‘add’ under the category elementwise, the applicant will comment “add_reformatting_checklist_elementwise”. This comment will then automatically be edited with the reformatting checklist including links to the relevant files for the category. For more information please view this subsection here. NOTE: To ensure that the PR author has permissions to update the checklist with their progress, they must be the ones to comment this checklist generating text and not us.
When reviewing ivy API extension pull requests (PRs), on top of the standard steps to follow, we should also pay particular attention to a few things:
- The function is genuinely a superset of the associated backend functions
- The argument names and behavior seem sensible according to your best judgment
- The decision of whether to implement the function as a primary, compositional or mixed function is well thought through and sensible
- Their are backend implementations if it is being implemented as a primary function
- A unit test has been added for the function, and it is passing.
- The unit test tests for all arguments with an adequate span of values.
- The function and test have both been added to the extension sections, and not to the corresponding submodules. We want to flag that these functions will not form part of the next official release of Ivy.
When reviewing frontend API pull requests (PRs), on top of the standard steps to follow, we should also pay particular attention to the frontend API unit tests. The most important thing above all else is to check that the frontend test is passing for the particular function that PR has added. In some cases, the frontend test will be failing as a result of problems with the Ivy functional API. In such cases, we should still merge the PR. We don’t want to keep applicants waiting because of unit tests that are still failing for Ivy’s functional API (which is our responsibility, not theirs).
Sometimes, a function is clearly missing from Ivy’s functional API, which would make the frontend function much simpler to implement. The process for such cases is explained in the Frontend APIs explanation in the Open Tasks section of the docs. In a nutshell, the contributor can either implement the function as a composition, with a “#ToDo” comment added in the implementation, or they can add the “Next Release” label to the “Sub-Task” issue, and then choose another frontend function to work on.
Many applicants simply use another function as a starting point and then copy this over during their implementation. Also, the old documentation was incorrect, and this was up for a long time which may have caused confusion. Combined, these can lead to common mistakes such as:
-
Adding
unsupported_dtypes
attributes which are not correct (these should only be added at the frontend level if the frontend function in the original framework does not support the data type.) -
Removing some data types from the unit test when they don’t work, when in reality these data types should be tested for and there is just a bug
-
Not testing for all optional arguments present in the frontend function.
You should be vigilant of points such as these when reviewing the PRs.
You will be assigned an issue if it:
- was not created by a member of our team
- does not contain either of the labels “Sub Task” or “ToDo”
- is not linked directly to a Pull Request
If this issue is a genuine issue raised by a member of the community, then you should take ownership of this issue, and do your best to address it, responding to the author of the issue in a timely manner etc. and making use of the internal discussions to find answers when you’re not sure yourself.
If this issue does not have the label “Sub Task”, but it is clearly a sub task which is stemming from a ToDo list issue, then you should modify the title such that it contains no spaces, and is simply the subtask name as it should appear in the ToDo lists (usually just a function name). Once you have modified the issue title, you should then manually add the label “Sub Task”, and you can then un-assign yourself from the issue.
In order to avoid the Issue appearing in the Issue Dashboard, you should make sure to always respond to the author's messages in some form, so that they are never the last person to message on the Issue (which will always cause it to appear in the dashboard). If they have said something which doesn’t warrant a response such as “okay, I will work on it”, then you can simply reply with a 👍 to their message, and this will register as a response. This will keep your Issues away from the dashboard, and keep you in the clear so none of us need to chase after you !
Overview
The task is simple: implement frontend functions listed in these GitHub ToDo lists that correspond to the Jax, Numpy, Tensorflow, and Torch frameworks, so that they adhere to the implementation, formatting and testing requirements outlined in the guide.
Task details
The open task section in the contributor guide was recently updated with many more details, outlining how each frontend function should be implemented, formatted and tested.
Once you allocate yourself to a desired function, it is your responsibility to determine which aspects to focus on, and how to track your own progress as you work through.
A non-exhaustive list of things to consider for each function are:
- Implement the frontend function according to its native arguments and behaviour
- Implement its hypothesis test
- Get the unit test passing
- Docstrings and type hints are not required
There are a series of guides with examples for the frontend task as your ground-truth reference for what needs to be added.
Special Cases
As we won’t be adding new features until the next release, if a function has a native implementation in most/all frameworks but is absent in Ivy, please open an issue (or convert your existing issue) requesting the function to be added in the backends. You should add the label “future release” to the issue.
If a function is not present in Ivy and is only unique to one or two frontends, composition is required.
Hypothesis
The hypothesis module is one of the steepest learning curves, and so it’s useful to go through some general tutorials beforehand. Aarsh Chaube has helpfully found these tutorials and this workshop, which are both very useful starting points. The answers to the exercises for the tutorials are in the instructor branch of the repository.
We should not skip data examples (a, b) or generate examples (c, d) in the test body, this should all be handled in the hypothesis generation itself. The hypothesis tests are explained in more detail here.
Requesting Help
If you hit a brick wall, and you’re not exactly sure how to proceed with the task, then just create a new card in the "Need Help" section of the Trello board dedicated to individual tasks, and then Dan or someone else in the team will also take a look ! If a PR has already been created, please also add this, as well as any extra info which might be helpful.
Overview Your task is to implement functions listed in this GitHub ToDo list so that they adhere to the implementation, formatting and testing requirements outlined in the guide.
Taks Details The Contributor guide was updated with the Ivy API Extensions section providing the details for adding new functions to Ivy’s functional API.
Please ensure that the following steps are carried out:
- Decide what features and arguments each function should have taking into consideration the Superset Behaviour section.
- Implement the function in Ivy API, also adding Array and Container instances if they can be supported.
- Implement the function in each backend API.
- Implement the test for the new function.
All the implementations have to be placed in the corresponding extensions directories.
Please refer to the contributor guide or post a question in the internal discussion thread. Feel free to create a PR to be reviewed by your POC (point of contact) and reference the PR/commit once completed in the PR/Commit column.
On top of the recurring tasks listed above, you will find other shared tasks on which you can work on in the Shared Tasks section of the project tab, along with their individual instructions.