Gcc-ICE-reporting-guide

Resources

Bugs (upstream)

If you have gcc crashing when building some gentoo package internal compiler error: Segmentation fault this page is for you!

To extract useful information from the crash to report a bug requires a few steps. This article lists the steps and provides a few real-world debugging sessions.

Steps to track an ICE (internal compiler errors) down

Check if an ICE is reproducible

Sometimes gcc crashes due to external reasons not directly related to gcc:

• out-of-memory condition (check dmesg)
• bad RAM modules (sys-apps/memtest86+)

Make sure the above is not your case.

Look at build.log with ICE and try to resume the build to see if the same command causes the same ICE. Usually command looks something like make --jobs=8 --load-average=8 CPPFLAGS= CFLAGS= LDFLAGS=.

Go to Working directory (specified in build.log as well) and rerun build command. You should see exact command and the failure.

Our running example will be ICE on dev-lang/python-3.6.5:

user $cd '/dev/shm/portage/dev-lang/python-3.6.5/work/Python-3.6.5'

user $LANG=C make

running build
running build_ext
INFO: Can't locate Tcl/Tk libs and/or headers
building 'cmath' extension
gcc-7.3.0 -pthread -fPIC -Wno-unused-result -Wsign-compare -DNDEBUG -march=prescott -mfpmath=sse -O2 -pipe -fomit-frame-pointer -fwrapv -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -I./Include -I. -I/dev/shm/portage/dev-lang/python-3.6.5/work/Python-3.6.5/Include -I/dev/shm/portage/dev-lang/python-3.6.5/work/Python-3.6.5 -c /dev/shm/portage/dev-lang/python-3.6.5/work/Python-3.6.5/Modules/cmathmodule.c -o build/temp.linux-i686-3.6/dev/shm/portage/dev-lang/python-3.6.5/work/Python-3.6.5/Modules/cmathmodule.o
In file included from /dev/shm/portage/dev-lang/python-3.6.5/work/Python-3.6.5/Modules/cmathmodule.c:11:0:
/dev/shm/portage/dev-lang/python-3.6.5/work/Python-3.6.5/Modules/clinic/cmathmodule.c.h: In function 'cmath_acos':
/dev/shm/portage/dev-lang/python-3.6.5/work/Python-3.6.5/Modules/clinic/cmathmodule.c.h:45:1: internal compiler error: Segmentation fault
 }
 ^
Please submit a full bug report,
with preprocessed source if appropriate.
See <https://bugs.gentoo.org/> for instructions.

The build fails every time LANG=C make is reran. Yay!

Extract exact command

If build system already prints exact command just re-execute it (make sure you are in correct directory). If build system does not print the command you will have to use other methods of getting command like running strace or passing verbose flags manually.

In case of python command is printed as-is:

user $LANG=C gcc-7.3.0 -pthread -fPIC -Wno-unused-result -Wsign-compare -DNDEBUG -march=prescott -mfpmath=sse -O2 -pipe -fomit-frame-pointer -fwrapv -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -I./Include -I. -I/dev/shm/portage/dev-lang/python-3.6.5/work/Python-3.6.5/Include -I/dev/shm/portage/dev-lang/python-3.6.5/work/Python-3.6.5 -c /dev/shm/portage/dev-lang/python-3.6.5/work/Python-3.6.5/Modules/cmathmodule.c -o build/temp.linux-i686-3.6/dev/shm/portage/dev-lang/python-3.6.5/work/Python-3.6.5/Modules/cmathmodule.o

In file included from /dev/shm/portage/dev-lang/python-3.6.5/work/Python-3.6.5/Modules/cmathmodule.c:11:0:
/dev/shm/portage/dev-lang/python-3.6.5/work/Python-3.6.5/Modules/clinic/cmathmodule.c.h: In function 'cmath_acos':
/dev/shm/portage/dev-lang/python-3.6.5/work/Python-3.6.5/Modules/clinic/cmathmodule.c.h:45:1: internal compiler error: Segmentation fault
 }
 ^
Please submit a full bug report,
with preprocessed source if appropriate.
See <https://bugs.gentoo.org/> for instructions.

Extract self-contained source (use -save-temps)

Add -save-temps to gcc command on a previous step to extract preprocessed sample. foo.c will create foo.i (or foo.ii for c++).

Obligatory python example:

user $LANG=C gcc-7.3.0 -pthread -fPIC -Wno-unused-result -Wsign-compare -DNDEBUG -march=prescott -mfpmath=sse -O2 -pipe -fomit-frame-pointer -fwrapv -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -I./Include -I. -I/dev/shm/portage/dev-lang/python-3.6.5/work/Python-3.6.5/Include -I/dev/shm/portage/dev-lang/python-3.6.5/work/Python-3.6.5 -c /dev/shm/portage/dev-lang/python-3.6.5/work/Python-3.6.5/Modules/cmathmodule.c -o build/temp.linux-i686-3.6/dev/shm/portage/dev-lang/python-3.6.5/work/Python-3.6.5/Modules/cmathmodule.o -save-temps

... same failure ...

The preprocessed file is in cmathmodule.i now. You can get more tips at https://gcc.gnu.org/bugs/

Expand -march=native, exact gcc version and other system-specific options

Some compiler options like -march=native are depend on the environment. You need to resolve them into exact options. One way to do it is to query gcc for expansion:

user $LANG=C gcc -Q --help=target > default.opts

user $LANG=C gcc -march=native -Q --help=target > native.opts

user $diff -U0 default.opts native.opts

--- default.opts        2018-08-05 00:27:33.038834012 +0100
+++ native.opts 2018-08-05 00:27:33.052834195 +0100
@@ -16 +16 @@
-  -maes                                [disabled]
+  -maes                                [enabled]
@@ -24 +24 @@
-  -march=                              x86-64
+  -march=                              sandybridge
@@ -26 +26 @@
-  -mavx                                [disabled]
+  -mavx                                [enabled]
...

Here -march=native is substituted for -march=sandybridge -maes -mavx .... Make sure bug is reproducible after substitution as well.

Report bug on bugs.gentoo.org

File a bug at https://bugs.gentoo.org/ against toolchain@gentoo.org and provide a few details:

• gcc -v output
• attach .i reproducer

[bonus] minimize needed flags to reproduce failure

You can do similar expansion for optimizer options as well: LANG=C gcc -O2 -Q --help=optimizers

Try to find minimal set of flags needed to trigger the crash by removing some expanded options.

[bonus] minimize self-contained source using creduce

Not to the fun part! Let's shrink .i down to a manageable size with help of https://embed.cs.utah.edu/creduce/.

1. emerge creduce

2. write a tester:

$ cat test.sh 
LANG=C gcc-7.3.0 -pthread -fPIC -Wno-unused-result -Wsign-compare -DNDEBUG -march=prescott -mfpmath=sse -O2 -pipe -fomit-frame-pointer -fwrapv -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -I./Include -I. -I/dev/shm/portage/dev-lang/python-3.6.5/work/Python-3.6.5/Include -I/dev/shm/portage/dev-lang/python-3.6.5/work/Python-3.6.5 -c cmathmodule.i -o cmathmodule.o \
>gcc_out.txt 2>&1
grep "internal compiler error" gcc_out.txt >/dev/null 2>&1

3. validate tester

$ ./test.sh
$ echo $?
0

4. Reduce!

$ creduce test.sh cmathmodule.i

===< 2430 >===
running 4 interestingness tests in parallel
===< pass_includes :: 0 >===
===< pass_unifdef :: 0 >===
===< pass_comments :: 0 >===
===< pass_blank :: 0 >===
(1.2 %, 733906 bytes)
(27.4 %, 539234 bytes)
===< pass_clang_binsrch :: replace-function-def-with-decl >===
(29.5 %, 523834 bytes)
(30.2 %, 518466 bytes)
(32.0 %, 505350 bytes)
...
          ******** cmathmodule.i ********

typedef struct {
  double a
} b;
b c;
d, e, f;
g() {
  _setjmp();
  b h;
  if (e)
    h.a = copysign(f, d);
  c = h;
}

Done!

cmathmodule.i contains the same shrunk example.

[bonus] Check if bug still exists on vanilla gcc master

Building gcc locally could look like that:

$ git clone git://gcc.gnu.org/git/gcc.git
$ mkdir -p gcc-build && cd gcc-build
$ ../gcc/configure --enable-languages=c,c++ --disable-bootstrap --prefix="$(pwd)/../gcc-installed" --disable-nls CFLAGS="-O1 -ggdb" CXXFLAGS="-O1 -ggdb"
$ make && make install

Look at configure options of gentoo's gcc -v to get more options to add to defaults. Be careful: some configure options like --enable-default-pie --enable-default-ssp can change code generation enough to trigger or hide a bug.

[bonus] Report bug on gcc.gnu.org/bugzilla

File the bug at https://gcc.gnu.org/bugzilla/ with attached minimal reproducer and gcc -v detals.

[bonus] Extract gcc backtrace

Example session would be:

    $ gdb --quiet --args gcc-7.3.0 -fPIC -march=prescott -mfpmath=sse -O1 -c cmathmodule.c -o cmathmodule.o
    (gdb) set follow-fork-mode child 
    (gdb) run
    ...
    Thread 2.1 "cc1" received signal SIGSEGV, Segmentation fault.
    [Switching to process 17286]
    0x085375d4 in lra_eliminate_reg_if_possible(rtx_def**) ()
    (gdb) bt
    #0  0x085375d4 in lra_eliminate_reg_if_possible(rtx_def**) ()
    ...

[bonus] Fix gcc bug

By now you have a source tree and command how to build it!

A few starting points for you:

• https://gcc.gnu.org/contribute.html for general notes on coding tips and guidelines
• https://gcc.gnu.org/onlinedocs/

Good luck!

See also

Bugzilla/Bug_report_guide —

Bugzilla/Guide — covers the recommended method of reporting bugs for Gentoo.