Static compilations with musl standard library – great alternative to glibc.
Building statically linked tools with glibc is real pain, compiled executables are big but there’s no room for complaints, you’re lucky if it successfully compiled.
I’ve been following the musl project for some time already – musl manages to address all the issues with static linking, where Glibc is lacking. It’s superior in every way.
For now, musl is the clear winner compared to known alternatives. Bloat comparison[2], dealing with resource exhaustion, security, and more. Musl is doing it right but isn’t yet popular enough, fingers crossed!
Building fully static binaries is another feature musl is good with and that’s the part I’ll explain below.
First of – the difference – executables:
-
Dynamic – binaries that require external libraries to work, smaller size.
-
Static – binaries with all libraries built-in no dependencies needed to run it, heavy.
Under normal circumstances there’s no reason to use static programs, however, in some cases, it’s the only way, for example:
-
a compromised system where shared libraries can’t be trusted;
-
the target system is too old;
-
damaged system recovery;
-
building a tool that will run on many different systems;
Static compilation can be done with glibc as well but often it’s pain to deal with, in some cases, it will keep failing and when it’s going smoothly – the resulting static executable is really big.
1. Building musl cross compiler.
Note: instructions below were executed on fresh Debian 10 minimal install with testing repositories.
Actually, before we start with a cross compiler, musl dynamic linker must be present, which is shipped with musl-gcc wrapper:
1 2 3 4 |
~# apt -y install wget gcc g++ make patch unzip cd /usr/src ; wget https://musl.libc.org/releases/musl-latest.tar.gz ~# tar -zxf musl-latest.tar.gz ; cd musl-[1-9]* ./configure ; make ; make install |
now the dynamic linker is present at /lib/ld-musl-x86_64.so.1 so cross compiler can be built:
1 2 3 |
~# cd /usr/src wget https://github.com/richfelker/musl-cross-make/archive/master.zip unzip master.zip ; cd musl-cross* |
It’s recommended to adjust make’s number of concurrent jobs (-j), as it’s going to take some time on older hardware. Optimal value = num_cores+1
1 2 |
~# time make -j8 TARGET=x86_64-linux-musl OUTPUT=/usr/local/musl-cross-compiler install |
(optional step) activate musl ‘ldd’ tool via symlink to musl dynamic linker:
1 2 3 4 5 |
~# ln -s /lib/ld-musl-x86_64.so.1 /usr/bin/musl-ldd ~# /usr/bin/musl-ldd /usr/local/musl-cross-compiler/bin/x86_64-linux-musl-ld /lib64/ld-linux-x86-64.so.2 (0x7f6b9106b000) libdl.so.2 => /lib64/ld-linux-x86-64.so.2 (0x7f6b9106b000) libc.so.6 => /lib64/ld-linux-x86-64.so.2 (0x7f6b9106b000) |
[2]. Testing on sample code.
musl dynamic linker and cross compiler is now installed – let’s compile some sample code using glibc and musl to check the differences.
1 2 3 4 5 |
#include <stdio.h> int main(void) { fprintf(stdout, "plonk\n"); return 0; } |
Save the above to test.c file.
Compilation – gcc uses glibc, second is musl:
1 2 |
~# gcc -static -o glibc-test-static test.c ~# /usr/local/musl-cross-compiler/bin/x86_64-linux-musl-gcc -static -o musl-test-static test.c |
Verify it’s static:
1 2 |
~# echo $(ldd ./glibc-test-static ./musl-test-static) ~# ./glibc-test-static: not a dynamic executable ./musl-test-static: not a dynamic executable |
Compare sizes:
1 2 3 |
~# du -b glibc-test-static musl-test-static 768168 glibc-test-static 10112 musl-test-static (98.6% smaller) |
Removing symbols via strip to shrink it further returns even better results:
1 2 3 |
~# strip glibc-test-static musl-test-static 694824 glibc-test-static 5240 musl-test-static (99.2% smaller) |
Detailed view – comparison of ELF and program headers between glibc and musl generated executables:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 |
glibc-c ELF Header: Magic: 7f 45 4c 46 02 01 01 03 00 00 00 00 00 00 00 00 Class: ELF64 Data: 2's complement, little endian Version: 1 (current) OS/ABI: UNIX - GNU ABI Version: 0 Type: EXEC (Executable file) Machine: Advanced Micro Devices X86-64 Version: 0x1 Entry point address: 0x401ac0 Start of program headers: 64 (bytes into file) Start of section headers: 693096 (bytes into file) Flags: 0x0 Size of this header: 64 (bytes) Size of program headers: 56 (bytes) Number of program headers: 8 Size of section headers: 64 (bytes) Number of section headers: 27 Section header string table index: 26 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 |
musl-c ELF Header: Magic: 7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00 Class: ELF64 Data: 2's complement, little endian Version: 1 (current) OS/ABI: UNIX - System V ABI Version: 0 Type: EXEC (Executable file) Machine: Advanced Micro Devices X86-64 Version: 0x1 Entry point address: 0x400152 Start of program headers: 64 (bytes into file) Start of section headers: 4472 (bytes into file) Flags: 0x0 Size of this header: 64 (bytes) Size of program headers: 56 (bytes) Number of program headers: 4 Size of section headers: 64 (bytes) Number of section headers: 12 Section header string table index: 11 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 |
glibc-c: file format elf64-x86-64 Program Header: LOAD off 0x0000000000000000 vaddr 0x0000000000400000 paddr 0x0000000000400000 align 2**12 filesz 0x0000000000000488 memsz 0x0000000000000488 flags r-- LOAD off 0x0000000000001000 vaddr 0x0000000000401000 paddr 0x0000000000401000 align 2**12 filesz 0x000000000007d601 memsz 0x000000000007d601 flags r-x LOAD off 0x000000000007f000 vaddr 0x000000000047f000 paddr 0x000000000047f000 align 2**12 filesz 0x0000000000024c1c memsz 0x0000000000024c1c flags r-- LOAD off 0x00000000000a4120 vaddr 0x00000000004a5120 paddr 0x00000000004a5120 align 2**12 filesz 0x0000000000005110 memsz 0x00000000000068a0 flags rw- NOTE off 0x0000000000000200 vaddr 0x0000000000400200 paddr 0x0000000000400200 align 2**2 filesz 0x0000000000000044 memsz 0x0000000000000044 flags r-- TLS off 0x00000000000a4120 vaddr 0x00000000004a5120 paddr 0x00000000004a5120 align 2**3 filesz 0x0000000000000020 memsz 0x0000000000000060 flags r-- STACK off 0x0000000000000000 vaddr 0x0000000000000000 paddr 0x0000000000000000 align 2**4 filesz 0x0000000000000000 memsz 0x0000000000000000 flags rw- RELRO off 0x00000000000a4120 vaddr 0x00000000004a5120 paddr 0x00000000004a5120 align 2**0 filesz 0x0000000000002ee0 memsz 0x0000000000002ee0 flags r-- |
1 2 3 4 5 6 7 8 9 10 |
musl-c: file format elf64-x86-64 Program Header: LOAD off 0x0000000000000000 vaddr 0x0000000000400000 paddr 0x0000000000400000 align 2**21 filesz 0x0000000000000e1c memsz 0x0000000000000e1c flags r-x LOAD off 0x0000000000000fe0 vaddr 0x0000000000600fe0 paddr 0x0000000000600fe0 align 2**21 filesz 0x0000000000000130 memsz 0x0000000000000838 flags rw- STACK off 0x0000000000000000 vaddr 0x0000000000000000 paddr 0x0000000000000000 align 2**4 filesz 0x0000000000000000 memsz 0x0000000000000000 flags rw- RELRO off 0x0000000000000fe0 vaddr 0x0000000000600fe0 paddr 0x0000000000600fe0 align 2**0 filesz 0x0000000000000020 memsz 0x0000000000000020 flags r-- |
The LOAD segments define parts of the executable meant to be opened in the memory on runtime – long story short – more segments means more mmap() operations the kernel deals with (see ELF specification for details).
However, the difference in the size of the executables is plain crazy – musl needed 1% of what glibc allocated, no point to compare further.
3. Practical examples – static compilation with musl – using two popular projects.
Note: For Nginx I’ve used ‘configure’ script to define musl compiler and cflag options. Then with util-linux I’ve set musl details via environmental variables (CC, (CPP/CXX), CFLAGS). Generally I’m using environmental variables unless the software I build provided equivalent options to use.
A) Nginx webserver with all components
1 2 3 4 5 6 7 8 9 10 11 12 |
~# apt -y install perl-modules ~# groupadd nginx ; useradd -g nginx nginx ; passwd -l nginx ~# cd /usr/src // download needed deps ~# wget https://ftp.pcre.org/pub/pcre/pcre-8.44.tar.gz \ https://www.openssl.org/source/openssl-1.1.1g.tar.gz \ http://www.zlib.net/zlib-1.2.11.tar.gz \ https://nginx.org/download/nginx-1.18.0.tar.gz // unpack: ~# cat pcre-*.tar.gz zlib-*.tar.gz openssl-*.tar.gz nginx-*.tar.gz|tar -zxif - ~# cd nginx-1.18.0 |
NOTE: at the time of writing this there’s an issue with the OpenSSL part. Nginx build will fail while processing OpenSSL, the error is:
crypto/blake2/m_blake2s.c:53:1: error: missing initializer for field ‘md_ctrl’ of ‘EVP_MD’
It might be confusing since the error is real, OpenSSL could ignore missing initializers and successfully finish compiling but Nginx isn’t passing needed flag so it needs to be manually added. Simply pass ‘-Wno-missing-field-initializers‘ flag to Nginx via ‘–with-cc-opt‘ option.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 |
~# ./configure --prefix=/usr/local/nginx-1.18.0-static \ --with-cc=/usr/local/musl-cross-compiler/bin/x86_64-linux-musl-gcc \ --with-ld-opt="-static" --user=nginx --group=nginx \ --with-cc-opt="-static -static-libgcc -Wno-missing-field-initializers" \ --with-cpp=/usr/local/musl-cross-compiler/bin/x86_64-linux-musl-g++ \ --with-pcre=/usr/src/pcre-8.44 --with-zlib=/usr/src/zlib-1.2.11 \ --with-openssl=/usr/src/openssl-1.1.1g --with-file-aio --with-mail \ --with-poll_module --with-select_module --with-stream \ --with-select_module --with-poll_module --with-http_ssl_module \ --with-http_realip_module --with-http_sub_module \ --with-http_addition_module --with-http_dav_module --with-http_flv_module \ --with-http_mp4_module --with-http_gunzip_module --with-http_gzip_static_module \ --with-http_auth_request_module --with-http_random_index_module --with-http_secure_link_module \ --with-http_degradation_module --with-http_stub_status_module --with-http_v2_module ~# make -j7 ~# make install |
Done, Nginx is now ready to use.
B) Toolset – latest util-linux
1 2 3 4 5 6 7 |
~# cd /usr/src ~# wget https://mirrors.edge.kernel.org/pub/linux/utils/util-linux/v2.35/util-linux-2.35.tar.xz ~# tar xJf util-linux-2.35.tar.xz ;cd util-linux-2.35 ~# ./configure CC="/usr/local/musl-cross-compiler/bin/x86_64-linux-musl-cc" \ CFLAGS="-static --static" --prefix=/usr/local/util-linux-2.35-static ~# make ~# make install |
That’s it. Both Nginx and util-linux tools are now fully static:
1 2 3 4 5 6 7 8 9 10 11 12 |
~# ldd /usr/local/nginx-1.18.0-static/sbin/nginx /usr/local/util-linux-2.35-static/sbin/fdisk /usr/local/nginx-1.18.0-static/sbin/nginx: not a dynamic executable /usr/local/util-linux-2.35-static/sbin/fdisk: not a dynamic executable ~# du -h /usr/local/nginx-1.18.0-static/sbin/nginx /usr/local/util-linux-2.35-static/sbin/fdisk 12M /usr/local/nginx-1.18.0-static/sbin/nginx 812K /usr/local/util-linux-2.35-static/sbin/fdisk ~# strip /usr/local/nginx-1.18.0-static/sbin/nginx /usr/local/util-linux-2.35-static/sbin/fdisk ~# du -h /usr/local/nginx-1.18.0-static/sbin/nginx /usr/local/util-linux-2.35-static/sbin/fdisk 3.9M /usr/local/nginx-1.18.0-static/sbin/nginx 688K /usr/local/util-linux-2.35-static/sbin/fdisk |