-
Notifications
You must be signed in to change notification settings - Fork 125
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Reusing declarations in the reverse pass #659
Milestone
Comments
vgvassilev
pushed a commit
to PetroZarytskyi/clad
that referenced
this issue
Jan 22, 2024
vgvassilev
pushed a commit
to PetroZarytskyi/clad
that referenced
this issue
Feb 8, 2024
When we produce a gradient function we generally have a forward and reverse sweep. In the forward sweep we accumulate the state and in the reverse sweep we use that state to invert the program execution. The forward sweep generally follows the sematics of the primal function and when neccessary stores the state which would be needed but lost for the reverse sweep. However, to minimize the stores onto the tape we need to reuse some of the variables between the forward and the reverse sweeps which requires some variables to be promoted to the enclosing lexical scope of both sweeps. Fixes vgvassilev#659, fixes vgvassilev#681.
vgvassilev
pushed a commit
that referenced
this issue
Feb 8, 2024
When we produce a gradient function we generally have a forward and reverse sweep. In the forward sweep we accumulate the state and in the reverse sweep we use that state to invert the program execution. The forward sweep generally follows the sematics of the primal function and when neccessary stores the state which would be needed but lost for the reverse sweep. However, to minimize the stores onto the tape we need to reuse some of the variables between the forward and the reverse sweeps which requires some variables to be promoted to the enclosing lexical scope of both sweeps. Fixes #659, fixes #681.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
This issue is going to arise after landing #655.
Reusing the same declarations for the reverse pass as the ones used in the forward pass does not cause any errors since scopes are currently reused as well. For example, if we differentiate this statement
We will get this statement for the forward pass
And this one for the reverse pass
Both statements have the same scope and, therefore, using
n
in the reverse pass does not cause an error. However, compiling the dumped code by hand obviously doesn't work.Ideally, this should be solved by placing a separate declaration statement in the reverse pass. This would cause a couple of complications:
The declaration should be inserted before the variable is used and that might be tricky because the order of statements in the reverse pass is reversed.
We should store the last value of a variable in the forward pass before it goes out of scope and then assign the corresponding variable in the reverse pass with that value. Also, it would be better to do this only when the value is useful (use TBR analysis).
There is also an easier temporary solution to move the declaration to function globals and place an assignment instead of an initialization. For example this code
Can be replaced with
Obviously, this will harm performance and decrease readability.
The text was updated successfully, but these errors were encountered: