This is Gentoo's testing wiki. It is a non-operational environment and its textual content is outdated.
Please visit our production wiki at https://wiki.gentoo.org
Portage TMPDIR on tmpfs
When emerging packages it is possible to build them in tmpfs (RAM) space instead of having build files pushed and pulled to Hard Disk Drive (HDD) or Solid State Drive (SSD) space. Building in tmpfs both speeds up emerge times and reduces HDD/SSD wear.
For systems running on a SSD, it is generally a good idea to have Portage compile using tmpfs (RAM) instead burning up precious SSD write cycles (especially on something like compiling software).
Generally speaking, unless the system has a large amount of RAM, it may be more of a hassle to setup Portage in tmpfs. Larger packages such as sys-devel/gcc (see below) will fail on only 2 GBs of tmpfs. Keep this in mind when proceeding!
Configuration
Considering tmpfs' size
The system's tmpfs space should be large enough to handle the largest packages to be compiled on the system. If the tmpfs space were to ever become completely full then the emerge will fail. Most packages would not need more than 1 GB for compilation, but there are a few that are very large and would need more. Those still wanting to compile these packages on tmpfs should verify enough free tmpfs space exists. The following list are estimates on how much space would be allocated on each package. Some of these are based on the minimum space requirements specified on the ebuilds themselves. Note that the actual allocated size may vary depending on the features included when building the package. It should also vary on every version update.
| Package | Memory usage |
|---|---|
| www-client/chromium | 10 GBs or so with 3 GBs of extra system memory. |
| dev-java/icedtea | 8.5 GBs |
| www-client/firefox | Around 4 GBs; or 8 GBs if the pgo, debug, or test USE flag has been enabled.
|
| sys-devel/gcc | Around 4 GBs, but should be much less if Java and Objective C is not included. |
| app-office/libreoffice | 6 GBs or so with 512 MBs extra system memory. |
| dev-lang/ghc | More than 4 GBs (will fail with <=4 GBs) |
When using ccache to assist in resuming compiles, it should be noted an equal size of the /var/tmp and /var/tmp/portage directories is necessary. Alternatively the per-package choices as shown below for large packages requiring large amounts of space can be implemented.
ccache creates a directory in /var/tmp/ccache to store compiled elements for resuming. This is why if /var/tmp is a tmpfs it must be of size similar to that of /var/tmp/portage if a amount of system RAM is to be used in this way.
fstab
Mount Portage's TMPDIR to tmpfs by adding the following to the system's /etc/fstab config file:
/etc/fstabtmpfs fstab exampletmpfs /var/tmp/portage tmpfs size=4G,uid=portage,gid=portage,mode=775,noatime 0 0
Adjust the size parameter /etc/fstab to the desired amount of RAM. Systems with large amounts of RAM can increase the number quite significantly.
After /etc/fstab has been modified, mount Portage's TMPDIR to RAM by running the mount command followed by the directory location outline in fstab:
root #mount /var/tmp/portageIn the unlikely event that the entire /var/tmp/ directory is already mounted as tmpfs, it can be worked around by the special x-mount.mkdir mount option:
/etc/fstabtmpfs fstab exampletmpfs /var/tmp tmpfs rw,nosuid,noatime,nodev,size=4G,mode=1777 0 0 tmpfs /var/tmp/portage tmpfs rw,nosuid,noatime,nodev,size=4G,mode=775,uid=portage,gid=portage,x-mount.mkdir=775 0 0
Per-package choices at compile time
Portage can be configured to build large packages outside of the tmpfs space on a per-package basis.
Create a file to tell Portage where to place the temporary files directory:
/etc/portage/env/notmpfs.confPORTAGE_TMPDIR="/var/tmp/notmpfs"
Create a separate temporary file directory outside of the tmpfs mount location:
root #mkdir /var/tmp/notmpfs
root #chown portage:portage /var/tmp/notmpfs
root #chmod 775 /var/tmp/notmpfs
Create a special Portage file called package.env in /etc/portage/ and list all the packages that are too large to be compiled using tmpfs:
app-office/libreoffice notmpfs.conf mail-client/thunderbird notmpfs.conf sys-devel/gcc notmpfs.conf www-client/chromium notmpfs.conf www-client/firefox notmpfs.conf
Tips
Resizing tmpfs
To resize the current tmpfs instance in /var/tmp/portage, run:
root #mount -o remount,size=N /var/tmp/portage
Where N is in the form of bytes. It can also be suffixed with k, m, or g to respectively have the form of (k)ilobytes, (m)egabytes or (g)igabytes. It can also be suffixed with a % to limit the tmpfs instance to the percentage of current physical RAM, the default being 50% when the parameter is not specified.
The resized tmpfs will not persist to the next boot unless the size parameter is modified in /etc/fstab. This is not necessary since a larger tmpfs is only needed during large package compilations.
It is recommended to leave-out at least 1 GB of space for the system to prevent out-of-memory problems. Using swap-disks for some heavy compile-time and link-time instances which are unexpected may also be helpful. Now even if swap-disks are used, reads and writes to it would only be minimal compared to having a physical filesystem behind /var/tmp/portage.
Here is a note about the size parameter in Linux kernel's documentation which can be found in /usr/src/linux/Documentation/filesystems/tmpfs.txt as long as a kernel has been emerged:
tmpfs.txtTMPFS information'"`UNIQ--pre-00000000-QINU`"'
Besides the obvious danger of choking the system by allocating too much memory for tmpfs space, it should be generally safe to enlarge the tmpfs during an emerge as this would only increase the size limit of the tmpfs without destroying any data from the emerge process.
For example, if a system has 12 GB of RAM and 3 disks with 2 GB of swap space working in parallel on each disk, then it would be pretty safe to choose size limit equal to 16G. 16 GB size is usually enough to compile Libreoffice and Chromium in parallel (usual emerge -1uDN @world) while reading Internet in a web browser.
It's not often that you'll ever have to do it and emerge would tell you that tmpfs is too small however there are instances that the package's ebuild would be not accurate at estimating the amount of disk space necessary for building the package. Newer packages may end up allocating more space, whereas using lesser USE flags would make it allocate less.
The solution for this is to either enlarge tmpfs, or add the exception to /etc/portage/package.env, and then run emerge again.
Save an emerge and resume later
This is experimental. ebuild experts should be queried about how reliable this command is; when to use it and when not to.
Example: emerging webkit-gtk can take a long time. I want to reboot into another OS and resume this ebuild later.
Optional: I use app-portage/genlop to inspect the current emerge session. I like using it to remind me of the ebuild version number or hopefully to get an estimated time remaining.
user $genlop -c
Currently merging 1 out of 2
* net-libs/webkit-gtk-2.4.8
current merge time: 4 hours, 27 minutes and 35 seconds.
ETA: unknown.
Press Ctrl+C to quit the current emerge session.
Since I am rebooting, I'll have to use cp -a or tar -cpf to save /var/tmp/portage/* while preserving permissions. Otherwise the tmpfs contents will be lost; You may want to inspect the memory size of /var/tmp/portage by using du:
root #du -sh /var/tmp/portage/251M /var/tmp/portage/
Reboot, do other stuff, come back later.
Restore /var/tmp/portage/*.
Resume the ebuild with ebuild <repository_directory>/<category>/<package_name>-<version>.ebuild merge:
root #ebuild /usr/portage/net-libs/webkit-gtk/webkit-gtk-2.6.5.ebuild merge
>>> Existing ${T}/environment for 'webkit-gtk-2.6.5' will be sourced. Run
>>> 'clean' to start with a fresh environment.
>>> Checking webkitgtk-2.6.5.tar.xz's mtime...
>>> WORKDIR is up-to-date, keeping...
* checking ebuild checksums ;-) ... [ ok ]
* checking auxfile checksums ;-) ... [ ok ]
* checking miscfile checksums ;-) ... [ ok ]
>>> It appears that 'setup' has already executed for 'webkit-gtk-2.6.5'; skipping.
>>> Remove '/var/tmp/portage/net-libs/webkit-gtk-2.6.5/.setuped' to force setup.
>>> It appears that 'unpack' has already executed for 'webkit-gtk-2.6.5'; skipping.
>>> Remove '/var/tmp/portage/net-libs/webkit-gtk-2.6.5/.unpacked' to force unpack.
>>> It appears that 'prepare' has already executed for 'webkit-gtk-2.6.5'; skipping.
>>> Remove '/var/tmp/portage/net-libs/webkit-gtk-2.6.5/.prepared' to force prepare.
>>> Configuring source in /var/tmp/portage/net-libs/webkit-gtk-2.6.5/work/webkitgtk-2.6.5 ...
>>> Working in BUILD_DIR: "/var/tmp/portage/net-libs/webkit-gtk-2.6.5/work/webkit-gtk-2.6.5_build"
..
..
.If you're using other repository sources besides gentoo like layman overlays, make sure that you're using the correct repository directory of the ebuild as one package can also belong to other repositories and be chosen to be installed over the one in gentoo. You can get the repository name of the current package by reading the last action entry in /var/log/emerge.log or reading the build.log file in the package's build directory with a command like fgrep Repository: /var/tmp/portage/net-libs/webkit-gtk-2.6.5/temp/build.log.
Do not use the .ebuild file found in /var/tmp/portage/<category>/<package_name>-<version>/build-info/<package_name>-<version>.ebuild as it seems to be only a reference. Perhaps there's a way to use it, but one would have to thoroughly understand how ebuild and ebuild.sh work.
Happy hacking!
Troubleshooting
No space left on device
If you encounter a not-enough space error or anything similar, there are basically two things to do:
- Check the /var/tmp/portage directory for old package directories from previously failed compiles. Any packages found therein should be deleted; with exceptions made for any failed packages the user would like to resume compiling later.
- Resize the tmpfs.
- In case you still get messages related to exhausted disk space during emerge, even though the allocated tmpfs size is not nearly exceeded (check with du -h during emerge), you may have stumbled upon an inodes shortage. So far it definitely may be a problem for the www-client/chromium package, for it's grand storage requirements, but can be expected for other large packages as well. To workaround - append
nr_inodes=0to the list of your options for the tmpfs mount in the /etc/fstab file. For additional information refer to 'tmpfs' section in man mount.
As last instance, add a (temporary) swap file somewhere on your system with enough capacity:
root #touch /swap.img
root #chmod 600 /swap.img
root #dd if=/dev/zero bs=1024M of=/swap.img count=8
This will create the file /swap.img with 8GB filled with zeroes.
Set up the swap area using:
root #mkswap /swap.imgEnable the file for swapping:
root #swapon /swap.imgCheck, if the swap file is active:
root #swapon --summaryFilename Type Size Used Priority /swap.img file 8388604 3876352 -2
Compile the packages which would deadlock the computer because of high RAM usage (e.g. chromium):
root #emerge --ask www-client/chromiumAlternatively, disable and remove the swap file when finished:
root #swapoff /swap.img
root #rm /swap.img