Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reusing declarations in the reverse pass #659

Closed
PetroZarytskyi opened this issue Nov 24, 2023 · 0 comments · Fixed by #738
Closed

Reusing declarations in the reverse pass #659

PetroZarytskyi opened this issue Nov 24, 2023 · 0 comments · Fixed by #738
Assignees
Milestone

Comments

@PetroZarytskyi
Copy link
Collaborator

This issue is going to arise after landing #655.
Reusing the same declarations for the reverse pass as the ones used in the forward pass does not cause any errors since scopes are currently reused as well. For example, if we differentiate this statement

if (1) {
    double n;
    n *= x;
}

We will get this statement for the forward pass

if (1) {
    double n;
    _t0 = n;
    n *= x;
}

And this one for the reverse pass

if (1) {
    {
        n = _t0;
        double _r_d0 = _d_n;
        _d_n += _r_d0 * x;
        double _r0 = n * _r_d0;
        * _d_x += _r0;
        _d_n -= _r_d0;
        _d_n;
    }
}

Both statements have the same scope and, therefore, using n in the reverse pass does not cause an error. However, compiling the dumped code by hand obviously doesn't work.
Ideally, this should be solved by placing a separate declaration statement in the reverse pass. This would cause a couple of complications:

  1. The declaration should be inserted before the variable is used and that might be tricky because the order of statements in the reverse pass is reversed.

  2. We should store the last value of a variable in the forward pass before it goes out of scope and then assign the corresponding variable in the reverse pass with that value. Also, it would be better to do this only when the value is useful (use TBR analysis).

    There is also an easier temporary solution to move the declaration to function globals and place an assignment instead of an initialization. For example this code

if (int k = 1) {
...

Can be replaced with

int k;
if (k = 1) {
...

Obviously, this will harm performance and decrease readability.

@vgvassilev vgvassilev added this to the v1.3 milestone Dec 6, 2023
vgvassilev pushed a commit to PetroZarytskyi/clad that referenced this issue Jan 22, 2024
vgvassilev pushed a commit to PetroZarytskyi/clad that referenced this issue Feb 8, 2024
When we produce a gradient function we generally have a forward and reverse
sweep. In the forward sweep we accumulate the state and in the reverse sweep
we use that state to invert the program execution. The forward sweep generally
follows the sematics of the primal function and when neccessary stores the state
which would be needed but lost for the reverse sweep.

However, to minimize the stores onto the tape we need to reuse some of the
variables between the forward and the reverse sweeps which requires some
variables to be promoted to the enclosing lexical scope of both sweeps.

Fixes vgvassilev#659, fixes vgvassilev#681.
vgvassilev pushed a commit that referenced this issue Feb 8, 2024
When we produce a gradient function we generally have a forward and reverse
sweep. In the forward sweep we accumulate the state and in the reverse sweep
we use that state to invert the program execution. The forward sweep generally
follows the sematics of the primal function and when neccessary stores the state
which would be needed but lost for the reverse sweep.

However, to minimize the stores onto the tape we need to reuse some of the
variables between the forward and the reverse sweeps which requires some
variables to be promoted to the enclosing lexical scope of both sweeps.

Fixes #659, fixes #681.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
2 participants