This is Gentoo's testing wiki. It is a non-operational environment and its textual content is outdated.
Please visit our production wiki at https://wiki.gentoo.org
Talk:Distcc
This is a talk page. Please add newer comments below older ones, and sign your comments using four tildes (
~~~~).
When adding a new section (at the bottom of the page), please mark it as "open for discussion" by using {{talk|open}} so it will show up in the list of open discussions.DistCC Needs a Tuning Section
If I'm not mistaken, DistCC pretty much ignores the "/etc/make.conf" or "/etc/portage/make.conf" MAKEOPTS variable, and spawns the jobs as needed until the jobs fallback to the local box (not via network) for compiling. (So tinkering with the MAKEOPTS variable is likely not needed, as this variable is usually already setup on most Gentoo boxes.)
Think I figured how to properly tune DistCC. Add the following "--jobs" option to the server or helper box performing the compiling. Typically the "--jobs" option will mirror the MAKEOPTS variable within the "/etc/portage/make.conf" file or CPU's+1. Notice that the DistCC server (or helper box) now has spawned ten distccd threads. (ie. ps ax |grep distccd)
/etc/conf.d/distccd:
DISTCCD_OPTS="--port 3632 --log-level warning --log-file /var/log/distccd.log -N 15 --allow 127.0.0.1 --allow 192.168.1.0/24 --jobs 10"
On the client box, the "/etc/distcc/hosts" for designating the DistCC server (or helper box); the jobs can be limited, but not designated more jobs? /etc/distcc/hosts
192.168.1.5/5
Per DistCC manual; by defaults, DistCC spawns four threads or four jobs (plus one) for per each host listed within the /etc/distcc/hosts file.
I'm still slightly confused here. And likely the "--jobs 10" option needs to be integrated into both the client and server/helper boxes. And then the /etc/distcc/hosts limits the job number as necessary by using the /num suffix option to the server address?
--Roger (talk) 22:26, 1 May 2014 (UTC)
DistCC Pump Mode
Article jumps right into pump mode. Might be nice to have a very very brief one sentence explaining pump mode. ie. "Distcc's pump mode accelerates remote compilation with distcc by also distributing preprocessing to the servers. (ie. man pump)" --Roger (talk) 19:05, 21 January 2015 (UTC)
- I missed to read the discussions page before changing the article and added quick note about pump mode in the configuration section. --Totktonada (talk) 03:24, 19 February 2016 (UTC)
The only other noteworthy item within the pump manual is; "Note that distcc's pump-mode assumes that sources files will not be modified during the lifetime of the include server, so modifying source files during a build may cause inconsistent results." Of which, I don't think I've ever personally seen source files modified during compilation, unless somebody remotely logged into a version control system while the sources were being recompiled. Being a source Linux based distribution, I still don't think the later is worth much concern? --Roger (talk) 18:15, 21 January 2015 (UTC)
- As I understand it’s about situations when some different macros defined before including a header in different compilation units. Or distcc able to check such things and it’s out of topic? Maybe your understanding right. Anyway I think it’s unlikely to be common and concerned user surely looks into man page, briefly at least. The article I guess shouldn’t describe all kind of uncommon weird things that may lead to breaking build process or broken executable. --Totktonada (talk) 03:24, 19 February 2016 (UTC)
Also, no mention of requiring to add ",cpp,lzo" (ie. 192.168.1.2/10,cpp,lzo) to the hosts lines within the /etc/distcc/hosts file! This is a requisite for DistCC pump mode! The example 192.168.1.2/10,cpp,lzo first specifies a remote compiler host, then limit sending 10 jobs max to the remote host compiler, including sending cpp jobs with lzo compression for which the later two cpp/lzo options are required for pump mode to work. Else, the user will see warnings and errors upon pump emerge! --Roger (talk) 19:05, 21 January 2015 (UTC)
- Thanks! I’m going to try pump mode on the week and will return with some thought here. --Totktonada (talk) 03:24, 19 February 2016 (UTC)
- You right. I updated the article: expanded configuration section and usage example. --Totktonada (talk) 01:03, 24 February 2016 (UTC)
The user should be warned that distcc pump mode is really buggy and is causing « random » build failures. See the following bugs :
--Netfab (talk) 09:20, 8 December 2016 (UTC)
Command for obtaining -march=native options
Previously, this command was listed as the mechanism for obtaining the CFLAGS equivent to -march=native:
user $gcc -Q -march=native --help=targetHowever, it has some flaws. Something valuable that the -march=native option obtains is information about the processor's cache (including the L1 line size). However, the one-liner I've replaced it with I can't promise will be 100% accurate on all archs, since I'm filtering out the hordes of -mno-* flags that gcc puts in there since 4.7 or so. I'm betting on gcc using these flags to add features to a base march, but if it uses a -mno-* flag to remove a feature from any arch, then this will be broken.
Maybe what we really need is a mechanism for the distcc client to insert these flags and relieve the user from having to fish them out.
Daniel Santos (talk) 22:42, 26 January 2015 (UTC)
- I don’t understand why concerned about 'all archs'? C[XX]FLAGS should fit particular machine and be equivalent to '-march=native' on *that* machine (with some additions/modifications/optimization flags according to you). There is a good article about filtering redundant flags from gcc’s output—'Inlining -march=native for distcc'—it’s mentioned in the configuration section and can be found in 'External resources' too. But maybe scripting around gcc’s output is good idea. --Totktonada (talk) 03:41, 19 February 2016 (UTC)
- Well there are a great many problems with the approach laid out in that article. I went through those sources (for the i386 back end) to see if I could implement a
gcc -Q --help=target-distccfunctionality and I gave up and filed a report (PR 81519) instead. Another developer who knows gcc better came along and tried and said "no way", but he has fixed PR 39851, so this will now make it possible to script it without having to decypher the gcc sources. This does still have flaws related to brevity, since some -m<isa> flags implicitly include others, so it still won't remove all redundant flags, but it can be fully automated, will never remove any -mno-<isa> flags that should not be removed (some of them shouldn't), be very close to as brief as possible and continue to work on newer architectures as they emerge in the market and support is added to gcc.
- Well there are a great many problems with the approach laid out in that article. I went through those sources (for the i386 back end) to see if I could implement a
- Honestly, I think that the entire mechanism gcc is currently using for machine flags needs to be completely redone so that all data structures used are defined in the middle-end and populated by each back-end so that arch-generic processing can be done on them. But at least when this patch is backported, we'll have a better way to get C(XX)FLAGS. Daniel Santos (talk) 23:44, 29 August 2017 (UTC)
Update: I've added a new section for the CFLAGS and CXXFLAGS with yet another script. This one isn't too useful until the next release of the GCC 6, 7 or 8 branch, but it boils down the machine flags quite dramatically. As it turns out, the answer for my Phenom machine was just -march=amdfam10. Also --param l1-cache-size et. al. aren't needed because that information is embedded in struct processor_costs which is dictated by the specific value of -march. The script doesn't extrapolate which ISA flags are implied by other flags however, so for example:
$ ./distccflags -march=native -mavx CFLAGS="-march=amdfam10 -mavx -msse4 -msse4.1 -msse4.2 -mssse3 -mxsave"
Here, everything in the output after -mavx is implied and not actually needed. It's also fun because you can pass it different flags and it can tell you what it translates into, but I'm only processing machine flags. Also, this IS broken if you try to use both Windows and non-windows machines in your farm because of the -mabi defaults and a lot of other things (SEH, etc.). Daniel Santos (talk) 08:28, 16 September 2017 (UTC)
No Real Mention of DistCC Required Packages with Versions Synced
I always thought there were a few packages with versions requiring to be synchronized across servers and clients for DistCC to work flawlessly. The packages I thought requiring versions to be synchronized across servers and clients were: distcc, glibc, gcc, binutils, ... ? Or is the only package requiring to have a similar major version is GCC? No mention or strong noting or warning for this version synchronization seems to be currently mentioned within the WIKI page. There were strong warnings in the past elsewhere for such package version synchronization, else users would risk major system breaks. I should also mention, the previous fore mentioned built with similar USE flags (ie. configure --help) as well? --Roger (talk) 11:00, 4 October 2015 (UTC)
Testing
When testing main.o with command './main.o' I get message:
-bash: ./main.o: Permission denied
After I chmod main.o with command 'chmod +x main.o' I get message:
-bash: ./main.o: cannot execute binary file: Exec format error
I need an explanation why can't I run main.o?
--Best, Pál (talk) 07:00, 6 February 2016 (UTC)
- The example was particuary broken. It missed linking and suggested to run an object file instead of an executable. Now it should be fine. --Totktonada (talk) 04:06, 19 February 2016 (UTC)
Another SSH usage example
Suppose that :
- you are on a powerful system (system A) and you want to help a tiny device (system B) to compile (or cross-compile).
- you already can connect from A to B with ssh.
On the system A
- install the distcc server
- configure it to allow connections only from localhost :
/etc/conf.d/distccdDISTCCD_OPTS="${DISTCCD_OPTS} --allow 127.0.0.1"
- run the distcc server
On the system B
- install distcc
- set participating hosts, here 16 jobs for the helper, 2 jobs for locale cores :
root #distcc-config --set-hosts "127.0.0.1/16 localhost/2"From the system A
- Open a reversed SSH tunnel to the system B :
user $ssh -R 3632:localhost:3632 systemBYou can now update your tiny system, jobs will be sent to the distcc server through the reversed SSH tunnel. --Netfab (talk) 08:00, 4 September 2016 (UTC)
Bootstrapping Error
When doing Step 2 in the Bootstrapping section:
root #USE='-*' emerge --nodeps sys-devel/distccI got:
!!! Problem resolving dependencies for sys-devel/distcc
!!! The ebuild selected to satisfy "sys-devel/distcc" has unmet requirements.
- sys-devel/distcc-3.2_rc1-r4::gentoo USE="-crossdev -gnome -gssapi -gtk -hardened -ipv6 (-selinux) -xinetd -zeroconf" ABI_X86="(64)" PYTHON_TARGETS="-python2_7"
The following REQUIRED_USE flag constraints are unsatisfied:
python_targets_python2_7
Fixed by doing:
root #USE='-* python_targets_python2_7' emerge --nodeps sys-devel/distccsince the -* was blowing away the PYTHON_TARGETS expanded flag
Huntersan9 (talk) 05:03, 22 July 2017 (UTC)
Why we cannot strip -mno-* flags
When I originally added the snippet for extracting machine flags from gcc, I stripped out all of the -mno-* flags because they seemed like garbage. I was later told (in IRC) that GCC represents some CPUs with an -march value combined with -mno-<isa> flags, but that this was for very old CPUs. Out of caution I modified the snippet to not strip them, but then another editor reverted that change citing breakage in gimp ebuild where it enables ISAs for specific .c files for functions that test the ISA availability before executing it. This is of course the most common case in the world -- most people don't custom compile all of software specifically for their CPU.
But it is now known that at least one recent CPU in the westmere family requires -mno-* flags. As such, it is simply wrong to remove them.
Thankfully, as of GCC 7.3, my distccflags script will work to determine the most succinct and correct machine flags. Distcc is interested in this upstream if anybody wants to convert this to C. This script will also work on GCC 6.5 if that is ever released as well as the upcoming GCC 8. Daniel Santos (talk) 18:14, 1 March 2018 (UTC)