Built-in linker in Linux #299

data-man · 2021-09-29T06:03:15Z

I'm currently using this patch in my OS.
It's bad, but it works.

diff --git a/src/compiler/compiler.c b/src/compiler/compiler.c
index 62e2cbf..9ba427b 100644
--- a/src/compiler/compiler.c
+++ b/src/compiler/compiler.c
@@ -414,14 +414,16 @@ void compiler_compile(void)
        {
                if (active_target.arch_os_target == ARCH_OS_TARGET_DEFAULT)
                {
+                       printf("Use platform linker.\n");
                        platform_linker(active_target.name, obj_files, output_file_count);
                }
                else
                {
+                       printf("Use built-in linker.\n");
                        if (!obj_format_linking_supported(platform_target.object_format) || !linker(active_target.name, obj_files,
                                                                                                    output_file_count))
                        {
-                               printf("No linking is performed due to missing linker support.");
+                               printf("No linking is performed due to missing linker support.\n");
                                active_target.run_after_compile = false;
                        }
                }
diff --git a/src/compiler/linker.c b/src/compiler/linker.c
index 4def40f..a9dd077 100644
--- a/src/compiler/linker.c
+++ b/src/compiler/linker.c
@@ -124,26 +124,26 @@ static bool link_exe(const char *output_file, const char **files_to_link, unsign
                                        if (platform_target.pie || platform_target.pic)
                                        {
                                                vec_add(args, "--eh-frame-hdr");
-                                               vec_add(args, "/usr/lib/x86_64-linux-gnu/crt1.o");
-                                               vec_add(args, "/usr/lib/gcc/x86_64-linux-gnu/10/crtbeginS.o");
+                                               vec_add(args, "/usr/lib/crt1.o");
+                                               vec_add(args, "/usr/lib/gcc/x86_64-pc-linux-gnu/11.1.0/crtbeginS.o");
                                                add_files(&args, files_to_link, file_count);
-                                               vec_add(args, "/usr/lib/x86_64-linux-gnu/crti.o");
-                                               vec_add(args, "/usr/lib/gcc/x86_64-linux-gnu/10/crtendS.o");
+                                               vec_add(args, "/usr/lib/crti.o");
+                                               vec_add(args, "/usr/lib/gcc/x86_64-pc-linux-gnu/11.1.0/crtendS.o");
                                                vec_add(args, "-pie");
                                        }
                                        else
                                        {
-                                               vec_add(args, "/usr/lib/x86_64-linux-gnu/Scrt1.o");
-                                               vec_add(args, "/usr/lib/gcc/x86_64-linux-gnu/10/crtbegin.o");
+                                               vec_add(args, "/usr/lib/Scrt1.o");
+                                               vec_add(args, "/usr/lib/gcc/x86_64-pc-linux-gnu/11.1.0/crtbegin.o");
                                                add_files(&args, files_to_link, file_count);
                                                vec_add(args, "-lc");
                                                vec_add(args, "-lm");
-                                               vec_add(args, "/usr/lib/x86_64-linux-gnu/crti.o");
-                                               vec_add(args, "/usr/lib/gcc/x86_64-linux-gnu/10/crtend.o");
+                                               vec_add(args, "/usr/lib/crti.o");
+                                               vec_add(args, "/usr/lib/gcc/x86_64-pc-linux-gnu/11.1.0/crtend.o");
                                                vec_add(args, "-no-pie");
                                        }
-                                       vec_add(args, "/usr/lib/x86_64-linux-gnu/crtn.o");
-                                       vec_add(args, "-L/usr/lib/x86_64-linux-gnu/");
+                                       vec_add(args, "/usr/lib/crtn.o");
+                                       vec_add(args, "-L/usr/lib/");
                                        vec_add(args, "--dynamic-linker=/lib64/ld-linux-x86-64.so.2");
                                        break;
                                case ARCH_TYPE_X86:

data-man · 2021-09-29T06:23:07Z

Just the idea: a configuration file (c3c.toml?) in some dir (~, ~/.config/c3c, etc.) with paths to system libs/files.

lerno · 2021-09-29T07:40:05Z

Zig actually provides its own musl-based linker libraries for cross compilation. I think this must be the goal, since making a linker work like this is not adding any extra capabilities: cross compilation still breaks.

data-man · 2022-01-08T06:33:41Z

I propose to integrate pkgconf into the compiler.
muon is the great example.

lerno · 2022-01-09T18:59:02Z

What would pkgconf do?

data-man · 2022-04-20T06:38:20Z

I checked dev branch now.
Is built-in lld completely removed? :(

lerno · 2022-04-20T07:46:13Z

No, don't mind what's there for now, I need to experiment a bit with getting the right arguments. lld will return.

lerno · 2022-04-21T22:13:39Z

You can start looking at dev now: it should have what's necessary to cross compile to MSVC and MacOS. Using LLD also works fine.

data-man · 2022-04-22T07:02:12Z

Yes, thank you, I've already tested it.
Maybe use LLD by default and rename --forcelinker to --externallinker?

lerno · 2022-04-22T07:04:51Z

It will all get cleaned up.

data-man · 2022-04-22T07:06:53Z

--wincrt=<optoion>
->
--wincrt=<option>

lerno · 2022-04-22T07:07:50Z

Great, I'll fix that right away.

data-man · 2022-04-22T07:18:27Z

Perhaps static, dynamic can be common options for all platforms?

lerno · 2022-04-22T16:07:43Z

Unfortunately not. MacOS for example does not support static linking of libc

data-man · 2022-05-04T14:58:15Z

Issue: the linking time in the --debug-stats output still empty with --forcelinker switch.
Feature request: use LLD by default and rename --forcelinker to --externallinker.

lerno · 2022-08-13T09:18:32Z

How does this work now? Is it better?

data-man · 2022-08-13T12:43:50Z

$ c3c compile --debug-stats something.c3

-- AST/EXPR/TYPE INFO --
 * Ast size: 56 bytes
 * Decl size: 136 bytes
 * Expr size: 56 bytes
 * TypeInfo size: 40 bytes
 * Ast memory use: 203kb (3719 elements)
 * Decl memory use: 349kb (2630 elements)
 * Expr memory use: 1062kb (19437 elements)
 * TypeInfo memory use: 123kb (3170 elements)
-- ARENA INFO --
 * Memory used:  113614 Kb
 * Allocations: 13118
 * String memory used:  48 Kb
Program linked to executable 'sysclock3'.
Timings
-------
Initialization took: 2.7250 ms
Parsing took:        9.8670 ms
Analysis took:       3.8890 ms
Ir gen took:         16.9390 ms
Codegen took:        1222.0340 ms
Linking took:        0.117000 ms

$ c3c compile --debug-stats --forcelinker something.c3

-- AST/EXPR/TYPE INFO --
 * Ast size: 56 bytes
 * Decl size: 136 bytes
 * Expr size: 56 bytes
 * TypeInfo size: 40 bytes
 * Ast memory use: 203kb (3719 elements)
 * Decl memory use: 349kb (2630 elements)
 * Expr memory use: 1062kb (19437 elements)
 * TypeInfo memory use: 123kb (3170 elements)
-- ARENA INFO --
 * Memory used:  113614 Kb
 * Allocations: 13118
 * String memory used:  48 Kb
Timings
-------
Initialization took: 1.3180 ms
Parsing took:        5.7270 ms
Analysis took:       2.3850 ms
Ir gen took:         19.6230 ms
Codegen took:        1186.6120 ms
Failed to find the C runtime at link time. <---

lerno · 2022-08-13T17:36:52Z

Do you know where libc is found for you?

lerno · 2022-08-13T17:52:22Z

@data-man can you see what possible paths there are to crt1.o and crtbegin.o? The simpler the better.

data-man · 2022-08-13T18:02:40Z

Sure:
/usr/lib/x86_64-linux-gnu/crt1.o
/usr/local/musl/lib/crt1.o
/usr/lib/gcc/x86_64-linux-gnu/11/crtbegin.o
/usr/lib/gcc/x86_64-linux-gnu/12/crtbegin.o

lerno · 2022-08-13T20:21:48Z

Please try the very latest version.

data-man · 2022-08-14T04:57:12Z

Sorry, In my previous message I omitted wasm and mingw paths as irrelevant, but ...

-- AST/EXPR/TYPE INFO --
 * Ast size: 56 bytes
 * Decl size: 136 bytes
 * Expr size: 56 bytes
 * TypeInfo size: 40 bytes
 * Ast memory use: 205kb (3749 elements)
 * Decl memory use: 350kb (2638 elements)
 * Expr memory use: 1083kb (19810 elements)
 * TypeInfo memory use: 124kb (3183 elements)
-- ARENA INFO --
 * Memory used:  113622 Kb
 * Allocations: 13272
 * String memory used:  48 Kb
Timings
-------
Initialization took: 0.8010 ms
Parsing took:        4.8600 ms
Analysis took:       2.6860 ms
Ir gen took:         16.2730 ms
Codegen took:        1205.5990 ms
ld.lld: error: /usr/lib/wasm32-wasi/crt1.o: unknown file type
ld.lld: error: /usr/lib/gcc/i686-w64-mingw32/10-posix/crtbegin.o: unknown file type
ld.lld: error: cannot open /usr/lib/wasm32-wasi/crti.o: No such file or directory
ld.lld: error: /usr/lib/gcc/i686-w64-mingw32/10-posix/crtend.o: unknown file type
ld.lld: error: cannot open /usr/lib/wasm32-wasi/crtn.o: No such file or directory
ld.lld: warning: /usr/lib/wasm32-wasi/libc.a: archive member 'dlmalloc.o' is neither ET_REL nor LLVM bitcode
ld.lld: warning: /usr/lib/wasm32-wasi/libc.a: archive member 'write.o' is neither ET_REL nor LLVM bitcode
...
Failed to create an executable: (null)

lerno · 2023-05-30T19:24:35Z

Does this work now @data-man

data-man · 2023-05-31T00:04:20Z

$ c3c compile --target linux-x86 --forcelinker hello_world_many.c3

WARNING! This architecture is not supported.
ld.lld: error: /usr/lib/wasm32-wasi/crt1.o: unknown file type
ld.lld: error: /usr/lib/gcc/i686-w64-mingw32/12-posix/crtbegin.o: unknown file type
ld.lld: error: cannot open /usr/lib/wasm32-wasi/crti.o: No such file or directory
ld.lld: error: /usr/lib/gcc/i686-w64-mingw32/12-posix/crtend.o: unknown file type
ld.lld: error: cannot open /usr/lib/wasm32-wasi/crtn.o: No such file or directory
ld.lld: warning: /usr/lib/wasm32-wasi/libc.a: archive member 'dlmalloc.o' is neither ET_REL nor LLVM bitcode
ld.lld: warning: /usr/lib/wasm32-wasi/libc.a: archive member 'write.o' is neither ET_REL nor LLVM bitcode
ld.lld: warning: /usr/lib/wasm32-wasi/libc.a: archive member 'usleep.o' is neither ET_REL nor LLVM bitcode
ld.lld: warning: /usr/lib/wasm32-wasi/libc.a: archive member 'unlinkat.o' is neither ET_REL nor LLVM bitcode
...skipped
Failed to create an executable: (null)

lerno · 2023-05-31T08:30:51Z

Oh wait, it's x86 and not x64?

lerno · 2023-05-31T10:41:10Z

Ok so it accidentally grabs your wasm32 and mingw installs

data-man · 2023-05-31T13:22:53Z

Also musl library can be located in /usr/include/x86_64-linux-musl directory if it installed by some system's package manager, or in /usr/local/musl/lib if it installed manually with default make options.

lerno · 2023-05-31T15:01:57Z

It's really difficult to add a good heuristic. You can look at "find_linux_crt" and "find_linux_crtbegin" on how this is done.

If we know that some particular variant of linux uses some fixed paths, then we could check for those first.

data-man · 2023-05-31T18:34:43Z

Some time ago I proposed use libpkgconf. It's small dependency (~256K). Now c3c contains miniz's sources.
Can we discuss my proposal again? :)

Maybe make it optional, like mimalloc?

lerno · 2023-05-31T19:40:14Z

Ok, a few things:

I just added --linux-crt and --linux-crtbegin options. With those you can set the correct paths and override anything detected by default.
To me the endgame is that the compiler either (1) is able to create a single binary for Linux, or (2) to have compile options like "ubuntu20-x64" "ubuntu22-x64" "debian11-x64". pkgconf helps with native builds but that's about it.

lerno · 2023-06-01T07:11:02Z

@data-man Can you test if --linux-crt and --linux-crtbegin resolves your issues?

data-man · 2023-06-01T08:01:33Z

I tried this (with musl libc in the same directory):

~~$ c3c compile hello_world_many.c3 --forcelinker --nolibc -L . -l libc (or libc.a)~~

~~> ld.lld: error: unable to find library -llibc~~

or

~~> ld.lld: error: unable to find library -llibc.a~~

lerno · 2023-06-01T08:02:58Z

But you're not setting --linux-crt and --linux-crtbegin?

data-man · 2023-06-01T08:03:51Z

NVM, sorry. -l c should be used.

lerno · 2023-06-01T08:05:37Z

Can you try without --nolibc and just pointing to --linux-crt and --linux-crtbegin?

data-man · 2023-06-01T08:09:14Z

Package musl-dev has this:

musl-dev: /usr/lib/x86_64-linux-musl/Scrt1.o
musl-dev: /usr/lib/x86_64-linux-musl/crt1.o
musl-dev: /usr/lib/x86_64-linux-musl/crti.o
musl-dev: /usr/lib/x86_64-linux-musl/crtn.o
musl-dev: /usr/lib/x86_64-linux-musl/libc.a
musl-dev: /usr/lib/x86_64-linux-musl/libdl.a
musl-dev: /usr/lib/x86_64-linux-musl/libm.a
musl-dev: /usr/lib/x86_64-linux-musl/libpthread.a
musl-dev: /usr/lib/x86_64-linux-musl/libresolv.a
musl-dev: /usr/lib/x86_64-linux-musl/librt.a
musl-dev: /usr/lib/x86_64-linux-musl/libutil.a
musl-dev: /usr/lib/x86_64-linux-musl/libxnet.a
musl-dev: /usr/lib/x86_64-linux-musl/musl-gcc.specs
musl-dev: /usr/lib/x86_64-linux-musl/rcrt1.o

lerno · 2024-08-11T17:16:44Z

Is this working now?

lerno added the Enhancement Request New feature or request label Jan 25, 2022

data-man mentioned this issue Apr 14, 2022

Fix linking logic #433

Closed

lerno added this to the First release (v1.0) milestone Sep 14, 2024

Built-in linker in Linux #299

Built-in linker in Linux #299

Comments

data-man commented Sep 29, 2021

data-man commented Sep 29, 2021

lerno commented Sep 29, 2021

data-man commented Jan 8, 2022

lerno commented Jan 9, 2022

data-man commented Apr 20, 2022

lerno commented Apr 20, 2022

lerno commented Apr 21, 2022

data-man commented Apr 22, 2022

lerno commented Apr 22, 2022

data-man commented Apr 22, 2022

lerno commented Apr 22, 2022

data-man commented Apr 22, 2022

lerno commented Apr 22, 2022

data-man commented May 4, 2022

lerno commented Aug 13, 2022

data-man commented Aug 13, 2022

lerno commented Aug 13, 2022

lerno commented Aug 13, 2022

data-man commented Aug 13, 2022

lerno commented Aug 13, 2022

data-man commented Aug 14, 2022

lerno commented May 30, 2023

data-man commented May 31, 2023

lerno commented May 31, 2023

lerno commented May 31, 2023

data-man commented May 31, 2023

lerno commented May 31, 2023

data-man commented May 31, 2023

lerno commented May 31, 2023

lerno commented Jun 1, 2023

data-man commented Jun 1, 2023 • edited Loading

lerno commented Jun 1, 2023

data-man commented Jun 1, 2023

lerno commented Jun 1, 2023

data-man commented Jun 1, 2023 • edited Loading

lerno commented Aug 11, 2024

data-man commented Jun 1, 2023 •

edited

Loading

data-man commented Jun 1, 2023 •

edited

Loading