Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

VariantDir breaks dependency tree #2908

Open
bdbaddog opened this issue Jan 2, 2018 · 6 comments · May be fixed by #4153
Open

VariantDir breaks dependency tree #2908

bdbaddog opened this issue Jan 2, 2018 · 6 comments · May be fixed by #4153

Comments

@bdbaddog
Copy link
Contributor

bdbaddog commented Jan 2, 2018

This issue was originally created at: 2013-05-14 09:21:18.
This issue was reported by: unclezeiv.

unclezeiv said at 2013-05-14 09:21:18

Simple case:

env = Environment()
env.VariantDir('build/variant1/', 'src', duplicate=0)
env.Depends(['src/test.cc'], ['src/input.strings'])
env.Program('bin/Linux64/test', ['build/variant1/test.cc'])

This is the resulting --tree=all dependency tree:

+-.
  +-SConstruct
  +-bin
  | +-bin/Linux64
  |   +-bin/Linux64/test
  |     +-build/variant1/test.o
  |     | +-src/test.cc          <------------ NO DEPENDENCY
  |     | +-/usr/bin/g++
  |     +-/usr/bin/g++
  +-build
  | +-build/variant1
  |   +-src/test.cc
  |   +-build/variant1/test.o
  |     +-src/test.cc            <------------ NO DEPENDENCY
  |     +-/usr/bin/g++
  +-src
    +-src/input.strings
    +-src/test.cc                <------------ CORRECT DEPENDENCY
      +-src/input.strings

Notice how src/test.cc appears several times, but only once the dependency on input.strings is correctly identified. When build/variant1/test.o correctly resolves to src/test.cc via VariantDir, it doesn't seem to remember the dependency - and, as a matter of fact, changing input.strings does indeed NOT trigger a recompilation :(

Contrast this with the tree that I get when NOT using VariantDir:

+-.
  +-SConstruct
  +-bin
  | +-bin/Linux64
  |   +-bin/Linux64/test
  |     +-src/test.o
  |     | +-src/test.cc
  |     | | +-src/input.strings  <-------------- CORRECT DEPENDENCY
  |     | +-/usr/bin/g++
  |     +-/usr/bin/g++
  +-src
    +-src/input.strings
    +-src/test.cc
    | +-src/input.strings        <-------------- CORRECT DEPENDENCY
    +-src/test.o
      +-src/test.cc
      | +-src/input.strings      <-------------- CORRECT DEPENDENCY
      +-/usr/bin/g++

This does not only happen when using Depends() explicitly. Also adding a custom builder to a node seems to be lost in the process.

Any help or workaround would be appreciated. Thank you.

(I'm testing this under Python 2.6 and Linux, but it doesn't look platform dependent).

bdbaddog said at 2013-05-14 11:41:52

If you change it to:

env.VariantDir('build/variant1/', 'src', duplicate=0)
env.Depends(['build/variant1/test.cc'], ['src/input.strings'])

Does it work?

In general, if you have a problem, you are better off first bringing it to the attention of the scons users mailing list. And then if it cannot be resolved and/or someone asks you to file it as a bug, then to proceed to do so.

unclezeiv said at 2013-05-14 16:44:03

Ok, will do, but let me just clarify that I'm not trying to make this toy example work. It took me a long time to distill the problem to what I think it's its essence. I thought the dependency tree dump was evidence enough of a misbehaviour. But I'll seek for advice on the mailing list first, thanks.

garyo said at 2013-05-15 05:23:28

This line is incorrect:

env.Depends(['src/test.cc'], ['src/input.strings'])

Saying that a source file depends on some other file doesn't cause anything to happen; source files have no builders. It's basically meaningless. Perhaps you mean to say that src/test.o (the object file) depends on src/input.strings, i.e. SCons should rebuild test.o whenever input.strings has changed?

unclezeiv said at 2013-05-15 05:43:58

Hi Gary, that's actually a good point. I should point out though that:

  1. nonetheless it does indeed work in the non-VariantDir case; so at the very least there's a behaviour that is different between the two cases
  2. this is a toy example. In my real case I do actually generate src/test.cc with a builder. I removed that part as I thought it was not fundamental for describing the problem. And again, even in that case, it works when not using VariantDir, it doesn't otherwise.

I poked around the Scons source and it looks like at some stage Scons is aware of src/test.cc having a builder and some dependencies, while at some later stage this info is lost. I can give you more details on this if you want, but I don't really understand the code base.

Thank you guys for your help.

Votes for this issue: 1.

@dmoody256
Copy link
Contributor

I ran into this in the latest version 3.0.4 with python 3.6 on Ubuntu 14.

I am generating source code with a Builder, then using the generated source later in a variant dir path I pass to the Program builder, but scons doesn't seem to make the connection that the generated source is the output of another builder, and just thinks the file doesn't exist without calling the builder that generates the source.

The answer here solved it for me by explicitly telling scons that dependency, it would then call my source code builder to generate the file before trying to build with the Program builder.

@bdbaddog
Copy link
Contributor Author

bdbaddog commented Mar 7, 2019

@dmoody256 - Can you attache taskmastertrace file on a simple example both with and without the explicit Depends()?

@dmoody256
Copy link
Contributor

Example:

env = Environment()
env.VariantDir('build/variant1', 'repo/src', duplicate=0)
env.Command('repo/src/foobar.cc', 'repo/src/foobar.input', action="cp repo/src/foobar.input repo/src/foobar.cc" )
env.Depends("build/variant1/foobar.cc", "repo/src/foobar.cc")
env.Program('bin/test', ['build/variant1/foobar.cc'])

brokentrace.txt
workingtrace.txt

@Cygon
Copy link

Cygon commented Sep 29, 2019

I believe I'm observing this bug with SCons 3.1.1

Here's a minimal reproduction SConstruct file:

#!/usr/bin/env python

def make_main_c(target, source, env):
    source_file = open(str(target[0]), 'w')
    source_file.write('#include <stdio.h>\n')
    source_file.write('int main() { printf("Hello World"); }')
    source_file.close()

env = Environment()

env.Command(source = None, action = make_main_c, target = 'src/main.c')
env.VariantDir('obj', 'src', duplicate = 0)
env.Program('bin/hello', 'obj/main.c')

What happens: SCons forgets that main.c needs to be produced by calling make_main_c() before it can be compiled.

Build output:

$ scons --tree=prune
scons: Reading SConscript files ...
scons: done reading SConscript files.
scons: Building targets ...
scons: *** [obj/main.o] Source `src/main.c' not found, needed by target `obj/main.o'.
+-.
  +-SConstruct
  +-bin
  | +-bin/hello
  |   +-obj/main.o
  |   | +-src/main.c
  |   | +-/usr/bin/gcc
  |   +-/usr/bin/gcc
  +-obj
  | +-src/main.c
  | +-[obj/main.o]
  +-src
    +-src/main.c
scons: building terminated because of errors.

A more complete example that attempts to download and compile the googletest library can be found here: https://pastebin.com/HVGnuSQE

@garyo
Copy link
Contributor

garyo commented Sep 29, 2019

Here's the taskmaster trace for @Cygon's nice simple repro case (which I confirm). It looks like it thinks src/main.c is up to date incorrectly. It's pending, then immediately marked up to date.

% scons --tree=prune --taskmastertrace=-
scons: Reading SConscript files ...
scons: done reading SConscript files.
scons: Building targets ...

Taskmaster: Looking for a node to evaluate
Taskmaster:     Considering node <no_state   0   '.'> and its children:
Taskmaster:        <no_state   0   'SConstruct'>
Taskmaster:        <no_state   0   'bin'>
Taskmaster:        <no_state   0   'obj'>
Taskmaster:        <no_state   0   'src'>
Taskmaster:      adjusted ref count: <pending    1   '.'>, child 'SConstruct'
Taskmaster:      adjusted ref count: <pending    2   '.'>, child 'bin'
Taskmaster:      adjusted ref count: <pending    3   '.'>, child 'obj'
Taskmaster:      adjusted ref count: <pending    4   '.'>, child 'src'
Taskmaster:     Considering node <no_state   0   'SConstruct'> and its children:
Taskmaster: Evaluating <pending    0   'SConstruct'>

Task.make_ready_current(): node <pending    0   'SConstruct'>
Task.prepare():      node <up_to_date 0   'SConstruct'>
Task.executed_with_callbacks(): node <up_to_date 0   'SConstruct'>
Task.postprocess():  node <up_to_date 0   'SConstruct'>
Task.postprocess():  removing <up_to_date 0   'SConstruct'>
Task.postprocess():  adjusted parent ref count <pending    3   '.'>

Taskmaster: Looking for a node to evaluate
Taskmaster:     Considering node <no_state   0   'bin'> and its children:
Taskmaster:        <no_state   0   'bin/hello'>
Taskmaster:      adjusted ref count: <pending    1   'bin'>, child 'bin/hello'
Taskmaster:     Considering node <no_state   0   'bin/hello'> and its children:
Taskmaster:        <no_state   0   'obj/main.o'>
Taskmaster:        <no_state   0   '/usr/bin/gcc'>
Taskmaster:      adjusted ref count: <pending    1   'bin/hello'>, child 'obj/main.o'
Taskmaster:      adjusted ref count: <pending    2   'bin/hello'>, child '/usr/bin/gcc'
Taskmaster:     Considering node <no_state   0   'obj/main.o'> and its children:
Taskmaster:        <no_state   0   'src/main.c'>
Taskmaster:        <no_state   0   '/usr/bin/gcc'>
Taskmaster:      adjusted ref count: <pending    1   'obj/main.o'>, child 'src/main.c'
Taskmaster:      adjusted ref count: <pending    2   'obj/main.o'>, child '/usr/bin/gcc'
Taskmaster:     Considering node <no_state   0   'src/main.c'> and its children:
Taskmaster: Evaluating <pending    0   'src/main.c'>

Task.make_ready_current(): node <pending    0   'src/main.c'>
Task.prepare():      node <up_to_date 0   'src/main.c'>
Task.executed_with_callbacks(): node <up_to_date 0   'src/main.c'>
Task.postprocess():  node <up_to_date 0   'src/main.c'>
Task.postprocess():  removing <up_to_date 0   'src/main.c'>
Task.postprocess():  adjusted parent ref count <pending    1   'obj/main.o'>

Taskmaster: Looking for a node to evaluate
Taskmaster:     Considering node <no_state   0   '/usr/bin/gcc'> and its children:
Taskmaster: Evaluating <pending    0   '/usr/bin/gcc'>

Task.make_ready_current(): node <pending    0   '/usr/bin/gcc'>
Task.prepare():      node <up_to_date 0   '/usr/bin/gcc'>
Task.executed_with_callbacks(): node <up_to_date 0   '/usr/bin/gcc'>
Task.postprocess():  node <up_to_date 0   '/usr/bin/gcc'>
Task.postprocess():  removing <up_to_date 0   '/usr/bin/gcc'>
Task.postprocess():  adjusted parent ref count <pending    0   'obj/main.o'>
Task.postprocess():  adjusted parent ref count <pending    1   'bin/hello'>

Taskmaster: Looking for a node to evaluate
Taskmaster:     Considering node <pending    0   'obj/main.o'> and its children:
Taskmaster:        <up_to_date 0   'src/main.c'>
Taskmaster:        <up_to_date 0   '/usr/bin/gcc'>
Taskmaster: Evaluating <pending    0   'obj/main.o'>

Task.make_ready_current(): node <pending    0   'obj/main.o'>
Task.prepare():      node <executing  0   'obj/main.o'>
scons: *** [obj/main.o] Source `src/main.c' not found, needed by target `obj/main.o'.
Task.failed_stop():  node <executing  0   'obj/main.o'>
Taskmaster:        removing node <executing  0   'obj/main.o'> from the pending children set
Taskmaster:        removing parent <pending    0   'bin/hello'> from the pending children set
Taskmaster:        removing parent <pending    0   'bin'> from the pending children set
Taskmaster:        removing parent <pending    2   '.'> from the pending children set

+-.
  +-SConstruct
  +-bin
  | +-bin/hello
  |   +-obj/main.o
  |   | +-src/main.c
  |   | +-/usr/bin/gcc
  |   +-/usr/bin/gcc
  +-obj
  | +-src/main.c
  | +-[obj/main.o]
  +-src
    +-src/main.c
Task.postprocess():  node <failed     0   'obj/main.o'>

Taskmaster: Looking for a node to evaluate
Taskmaster:        removing node <no_state   0   'src'> from the pending children set
Taskmaster:        removing node <no_state   0   'obj'> from the pending children set
Taskmaster:        removing node <up_to_date 0   '/usr/bin/gcc'> from the pending children set
Taskmaster:        removing parent <failed     1   '.'> from the pending children set
Taskmaster:        removing parent <failed     0   '.'> from the pending children set
Taskmaster: No candidate anymore.

scons: building terminated because of errors.

@bdbaddog
Copy link
Contributor Author

I have a work in progress branch for this. It doesn't work yet.
https://github.com/bdbaddog/scons/tree/fix_2908_files_change_in_source_of_variantdir

Also a bigger example from @Cygon
https://github.com/bdbaddog/zikare-issue/tree/no_sconscript

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants