Headless build ignores project dependencies and builds in random order

[trancexpress]

This change adjusts HeadlessBuilder.buildConfigurations() to sort the input projects, based on their references. Projects that are referenced are built first, so that their dependencies are available to referring projects.

Fixes: #528

[trancexpress]

Something like this I guess. I'll see if we can add testing, especially for the new sorting.

[jonahgraham]

Failed test is known flaky #194

[jonahgraham]

This code seems to be handling this type of references:

But the CDT has its own project references model (which I hope works for headless):

See the documentation about CDT's more advanced references here.

[trancexpress]

This code seems to be handling this type of references:

But the CDT has its own project references model (which I hope works for headless):

See the documentation about CDT's more advanced references here.

I tried this after trying project references, neither helps with the headless build. From what I see in the headless builder code, the workspace root is asked for all workspace projects, then those are put in a hash map. The hash map order determines the build order. I.e. there is no actual order. Unlike the build in the IDE.

I did see that setting up the CDT references results also in project references.

[jonahgraham]

I.e. there is no actual order. Unlike the build in the IDE.

That makes it sound like the entry point is too low. Without the original authors around it will be interesting to understand what was done on purpose and what wasn't.

[umairsair]

I tried this after trying project references, neither helps with the headless build.

AFAICT, each project build makes sure that its referenced config of other project is built first. However, there can be a case where project is built twice, like B references project A then first A is built and then build is again invoked on A because B references it. Can you please share the console logs after adding project references in C/C++ General > Paths and Symbols > References and building headless? Also make sure after headless build that references still exist in C/C++ General > Paths and Symbols > References (just want to make sure that nothing is messed up).

I did see that setting up the CDT references results also in project references.

Yes, thats correct. But for build order, it relies on CDT references which takes configuration level references in account.

[trancexpress]

AFAICT, each project build makes sure that its referenced config of other project is built first. However, there can be a case where project is built twice, like B references project A then first A is built and then build is again invoked on A because B references it. Can you please share the console logs after adding project references in C/C++ General > Paths and Symbols > References and building headless? Also make sure after headless build that references still exist in C/C++ General > Paths and Symbols > References (just want to make sure that nothing is messed up).

Output with projects that have no references:

WARNING: Using incubator modules: jdk.incubator.vector, jdk.incubator.foreign
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Create.
Opening 'Test1'.
Create.
Opening 'Test2'.
Cleaning All Projects...
14:56:10 **** Clean-only build of configuration Debug for project Test1 ****
make clean 
rm -rf ./src/Test1.d ./src/Test1.o
rm -rf Test1
 

14:56:10 Build Finished. 0 errors, 0 warnings. (took 75ms)

14:56:11 **** Clean-only build of configuration Debug for project Test2 ****
make clean 
rm -rf ./src/Test2.d ./src/Test2.o
rm -rf Test2
 

14:56:11 Build Finished. 0 errors, 0 warnings. (took 59ms)

Building All Projects...
14:56:11 **** Clean-only build of configuration Debug for project Test2 ****
make clean 
rm -rf ./src/Test2.d ./src/Test2.o
rm -rf Test2
 

14:56:11 Build Finished. 0 errors, 0 warnings. (took 59ms)

14:56:11 **** Build of configuration Debug for project Test2 ****
make all 
Building file: ../src/Test2.cpp
Invoking: GCC C++ Compiler
g++ -O0 -g3 -Wall -c -fmessage-length=0 -MMD -MP -MF"src/Test2.d" -MT"src/Test2.o" -o "src/Test2.o" "../src/Test2.cpp"
Finished building: ../src/Test2.cpp
 
Building target: Test2
Invoking: GCC C++ Linker
g++  -o "Test2"  ./src/Test2.o   
Finished building target: Test2
 

14:56:11 Build Finished. 0 errors, 0 warnings. (took 160ms)

14:56:11 **** Build of configuration Debug for project Test2 ****
make all 
make: Nothing to be done for `all'.

14:56:11 Build Finished. 0 errors, 0 warnings. (took 58ms)

14:56:11 **** Clean-only build of configuration Release for project Test2 ****
make clean 
rm -rf ./src/Test2.d ./src/Test2.o
rm -rf Test2
 

14:56:11 Build Finished. 0 errors, 0 warnings. (took 62ms)

14:56:11 **** Build of configuration Release for project Test2 ****
make all 
Building file: ../src/Test2.cpp
Invoking: GCC C++ Compiler
g++ -O3 -Wall -c -fmessage-length=0 -MMD -MP -MF"src/Test2.d" -MT"src/Test2.o" -o "src/Test2.o" "../src/Test2.cpp"
Finished building: ../src/Test2.cpp
 
Building target: Test2
Invoking: GCC C++ Linker
g++  -o "Test2"  ./src/Test2.o   
Finished building target: Test2
 

14:56:12 Build Finished. 0 errors, 0 warnings. (took 160ms)

14:56:12 **** Build of configuration Release for project Test2 ****
make all 
make: Nothing to be done for `all'.

14:56:12 Build Finished. 0 errors, 0 warnings. (took 60ms)

14:56:12 **** Clean-only build of configuration Debug for project Test1 ****
make clean 
rm -rf ./src/Test1.d ./src/Test1.o
rm -rf Test1
 

14:56:12 Build Finished. 0 errors, 0 warnings. (took 60ms)

14:56:12 **** Build of configuration Debug for project Test1 ****
make all 
Building file: ../src/Test1.cpp
Invoking: GCC C++ Compiler
g++ -O0 -g3 -Wall -c -fmessage-length=0 -MMD -MP -MF"src/Test1.d" -MT"src/Test1.o" -o "src/Test1.o" "../src/Test1.cpp"
Finished building: ../src/Test1.cpp
 
Building target: Test1
Invoking: GCC C++ Linker
g++  -o "Test1"  ./src/Test1.o   
Finished building target: Test1
 

14:56:12 Build Finished. 0 errors, 0 warnings. (took 210ms)

14:56:12 **** Build of configuration Debug for project Test1 ****
make all 
make: Nothing to be done for `all'.

14:56:12 Build Finished. 0 errors, 0 warnings. (took 60ms)

14:56:12 **** Clean-only build of configuration Release for project Test1 ****
make clean 
rm -rf ./src/Test1.d ./src/Test1.o
rm -rf Test1
 

14:56:12 Build Finished. 0 errors, 0 warnings. (took 60ms)

14:56:12 **** Build of configuration Release for project Test1 ****
make all 
Building file: ../src/Test1.cpp
Invoking: GCC C++ Compiler
g++ -O3 -Wall -c -fmessage-length=0 -MMD -MP -MF"src/Test1.d" -MT"src/Test1.o" -o "src/Test1.o" "../src/Test1.cpp"
Finished building: ../src/Test1.cpp
 
Building target: Test1
Invoking: GCC C++ Linker
g++  -o "Test1"  ./src/Test1.o   
Finished building target: Test1
 

14:56:12 Build Finished. 0 errors, 0 warnings. (took 160ms)

14:56:13 **** Build of configuration Release for project Test1 ****
make all 
make: Nothing to be done for `all'.

14:56:13 Build Finished. 0 errors, 0 warnings. (took 61ms)

Console output after adding a reference in the paths and symbols tab, for Test2, referring to Test1 (referring tothe active build configuration):

WARNING: Using incubator modules: jdk.incubator.foreign, jdk.incubator.vector
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Create.
Opening 'Test1'.
Create.
Opening 'Test2'.
Cleaning All Projects...
14:58:48 **** Clean-only build of configuration Debug for project Test1 ****
make clean 
rm -rf ./src/Test1.d ./src/Test1.o
rm -rf Test1
 

14:58:48 Build Finished. 0 errors, 0 warnings. (took 69ms)

14:58:48 **** Clean-only build of configuration Debug for project Test2 ****
make clean 
rm -rf ./src/Test2.d ./src/Test2.o
rm -rf Test2
 

14:58:48 Build Finished. 0 errors, 0 warnings. (took 58ms)

Building All Projects...
14:58:49 **** Clean-only build of configuration Debug for project Test1 ****
make clean 
rm -rf ./src/Test1.d ./src/Test1.o
rm -rf Test1
 

14:58:49 Build Finished. 0 errors, 0 warnings. (took 59ms)

14:58:49 **** Incremental Build of configuration Debug for project Test1 ****
make all 
Building file: ../src/Test1.cpp
Invoking: GCC C++ Compiler
g++ -O0 -g3 -Wall -c -fmessage-length=0 -MMD -MP -MF"src/Test1.d" -MT"src/Test1.o" -o "src/Test1.o" "../src/Test1.cpp"
Finished building: ../src/Test1.cpp
 
Building target: Test1
Invoking: GCC C++ Linker
g++  -o "Test1"  ./src/Test1.o   
Finished building target: Test1
 

14:58:49 Build Finished. 0 errors, 0 warnings. (took 209ms)

14:58:49 **** Clean-only build of configuration Debug for project Test2 ****
make clean 
rm -rf ./src/Test2.d ./src/Test2.o
rm -rf Test2
 

14:58:49 Build Finished. 0 errors, 0 warnings. (took 58ms)

14:58:49 **** Build of configuration Debug for project Test2 ****
make all 
Building file: ../src/Test2.cpp
Invoking: GCC C++ Compiler
g++ -O0 -g3 -Wall -c -fmessage-length=0 -MMD -MP -MF"src/Test2.d" -MT"src/Test2.o" -o "src/Test2.o" "../src/Test2.cpp"
Finished building: ../src/Test2.cpp
 
Building target: Test2
Invoking: GCC C++ Linker
g++  -o "Test2"  ./src/Test2.o   
Finished building target: Test2
 

14:58:49 Build Finished. 0 errors, 0 warnings. (took 209ms)

14:58:50 **** Build of configuration Debug for project Test2 ****
make all 
make: Nothing to be done for `all'.

14:58:50 Build Finished. 0 errors, 0 warnings. (took 59ms)

14:58:50 **** Clean-only build of configuration Release for project Test2 ****
make clean 
rm -rf ./src/Test2.d ./src/Test2.o
rm -rf Test2
 

14:58:50 Build Finished. 0 errors, 0 warnings. (took 58ms)

14:58:50 **** Build of configuration Release for project Test2 ****
make all 
Building file: ../src/Test2.cpp
Invoking: GCC C++ Compiler
g++ -O3 -Wall -c -fmessage-length=0 -MMD -MP -MF"src/Test2.d" -MT"src/Test2.o" -o "src/Test2.o" "../src/Test2.cpp"
Finished building: ../src/Test2.cpp
 
Building target: Test2
Invoking: GCC C++ Linker
g++  -o "Test2"  ./src/Test2.o   
Finished building target: Test2
 

14:58:50 Build Finished. 0 errors, 0 warnings. (took 165ms)

14:58:50 **** Build of configuration Release for project Test2 ****
make all 
make: Nothing to be done for `all'.

14:58:50 Build Finished. 0 errors, 0 warnings. (took 59ms)

14:58:50 **** Build of configuration Debug for project Test1 ****
make all 
make: Nothing to be done for `all'.

14:58:50 Build Finished. 0 errors, 0 warnings. (took 60ms)

14:58:50 **** Build of configuration Debug for project Test1 ****
make all 
make: Nothing to be done for `all'.

14:58:50 Build Finished. 0 errors, 0 warnings. (took 60ms)

14:58:50 **** Clean-only build of configuration Release for project Test1 ****
make clean 
rm -rf ./src/Test1.d ./src/Test1.o
rm -rf Test1
 

14:58:50 Build Finished. 0 errors, 0 warnings. (took 60ms)

14:58:50 **** Build of configuration Release for project Test1 ****
make all 
Building file: ../src/Test1.cpp
Invoking: GCC C++ Compiler
g++ -O3 -Wall -c -fmessage-length=0 -MMD -MP -MF"src/Test1.d" -MT"src/Test1.o" -o "src/Test1.o" "../src/Test1.cpp"
Finished building: ../src/Test1.cpp
 
Building target: Test1
Invoking: GCC C++ Linker
g++  -o "Test1"  ./src/Test1.o   
Finished building target: Test1
 

14:58:51 Build Finished. 0 errors, 0 warnings. (took 210ms)

14:58:51 **** Build of configuration Release for project Test1 ****
make all 
make: Nothing to be done for `all'.

14:58:51 Build Finished. 0 errors, 0 warnings. (took 60ms)

After the headless build, if I check the project, the reference is still there.

Yes, thats correct. But for build order, it relies on CDT references which takes configuration level references in account.

Where is that supposed to be done during a headless build?

[trancexpress]

However, there can be a case where project is built twice, like B references project A then first A is built and then build is again invoked on A because B references it.

Ah, I think I see what you mean, looking at the output.

So the referenced config will be built as the project gets built, and then the project gets built again (later on, clean build)? I think that works for our users, I'll have to double check with them though.

Thanks!

[umairsair]

It seems that your problem is fixed for Test2/Debug; Test1/Debug is built first as Test2/Debug depends on it. The problem is with Test2/Release. I think you haven't added the reference for Test2/Release. Use Configuration combo in References tab to select Release config and add appropriate config reference.

[jonahgraham]

Closing for now - see discussion at #528 (comment)