Static compilations with musl standard library – great alternative to glibc.

 

 

Building statically linked tools with glibc is real pain, compiled executables are big but there’s no room for complaints, you’re lucky if it successfully compiled.

I’ve been following the musl project for some time already – musl manages to address all the issues with static linking, where Glibc is lacking. It’s superior in every way.

 

For now, musl is the clear winner compared to known alternatives. Bloat comparison[2], dealing with resource exhaustion, security, and more. Musl is doing it right but isn’t yet popular enough, fingers crossed!

 

Building fully static binaries is another feature musl is good with and that’s the part I’ll explain below.

First of – the difference – executables:

• Dynamic – binaries that require external libraries to work, smaller size.

• Static – binaries with all libraries built-in no dependencies needed to run it, heavy.

Under normal circumstances there’s no reason to use static programs, however, in some cases, it’s the only way, for example:

•  a compromised system where shared libraries can’t be trusted;

•  the target system is too old;

•  damaged system recovery;

•  building a tool that will run on many different systems;

Static compilation can be done with glibc as well but often it’s pain to deal with, in some cases, it will keep failing and when it’s going smoothly – the resulting static executable is really big.

---

1. Building musl cross compiler.

Note: instructions below were executed on fresh Debian 10 minimal install with testing repositories.

Actually, before we start with a cross compiler, musl dynamic linker must be present, which is shipped with musl-gcc wrapper:

 

now the dynamic linker is present at /lib/ld-musl-x86_64.so.1 so cross compiler can be built:

 

It’s recommended to adjust make’s number of concurrent jobs (-j), as it’s going to take some time on older hardware. Optimal value = num_cores+1

(optional step) activate musl ‘ldd’ tool via symlink to musl dynamic linker:

 

[2]. Testing on sample code.

musl dynamic linker and cross compiler is now installed – let’s compile some sample code using glibc and musl to check the differences.

Save the above to test.c file.

Compilation – gcc uses glibc, second is musl:

Verify it’s static:

Compare sizes:

Removing symbols via strip to shrink it further returns even better results:

Detailed view – comparison of ELF and program headers between glibc and musl generated executables:

The LOAD segments define parts of the executable meant to be opened in the memory on runtime – long story short – more segments means more mmap() operations the kernel deals with (see ELF specification for details).

However, the difference in the size of the executables is plain crazy – musl needed 1% of what glibc allocated, no point to compare further.

 

3. Practical examples – static compilation with musl – using two popular projects.

Note: For Nginx I’ve used ‘configure’ script to define musl compiler and cflag options. Then with util-linux I’ve set musl details via environmental variables (CC, (CPP/CXX), CFLAGS). Generally I’m using environmental variables unless the software I build provided equivalent options to use.

A) Nginx webserver with all components

NOTE: at the time of writing this there’s an issue with the OpenSSL part. Nginx build will fail while processing OpenSSL, the error is:
crypto/blake2/m_blake2s.c:53:1: error: missing initializer for field ‘md_ctrl’ of ‘EVP_MD’

It might be confusing since the error is real, OpenSSL could ignore missing initializers and successfully finish compiling but Nginx isn’t passing needed flag so it needs to be manually added. Simply pass ‘-Wno-missing-field-initializers‘ flag to Nginx via ‘–with-cc-opt‘ option.

Done, Nginx is now ready to use.

 

B) Toolset – latest util-linux

 

That’s it. Both Nginx and util-linux tools are now fully static: