Benchmark Paludis 0.38 and Portage 2.1.6.9 #2

As it seems unclear to some, I have redone my benchmarks today with extra comments.

peper is using different configuration files for portage and paludis. So the comparison is quite ... uhm ... difficult.
I did include -E portage timinings too, didn't I? But let's look again.

# time paludis -ip sys-apps/portage -E portage
real    1m39.186s
user    1m2.442s
sys     0m38.543s

<bonsaikitten> dleverton: since I don't have any paludis config files my results are what should happen, -E seems to not do things the same (or there are some artifacts from the previous config left)
Still not enough? Ok. (Yes, I know these are silly.)

# mv /etc/paludis /etc/paludis.hidden
# time paludis -ip sys-apps/portage
real    1m39.407s
user    1m2.099s
sys     0m38.633s
# time paludis -ip sys-apps/portage -E portage
real    1m39.948s
user    1m2.133s
sys     0m38.873s

And portage didn't get any faster since yesterday either:

# time emerge -puD sys-apps/portage
real    5m30.694s
user    3m30.119s
sys     1m56.093s

What am I missing this time?

And to get things back to normal:

# mv /var/lib/gentoo/repositories/gentoo/metadata.hidden /var/lib/gentoo/repositories/gentoo/metadata
# time emerge --metadata
real    2m58.175s
user    0m16.025s
sys     0m8.808s

# time paludis --metadata
Usage error: Error handling command line: Bad argument '--metadata'

real    0m0.022s
user    0m0.007s
sys     0m0.004s

Oh, right, paludis doesn't do that.
And now:

# time emerge -puD sys-apps/portage
real    0m4.196s
user    0m3.924s
sys     0m0.177s

# time paludis -ip sys-apps/portage -E portage
real    0m3.082s
user    0m2.554s
sys     0m0.518s

My /etc/make.conf for the record:

CFLAGS="-march=athlon64 -O2 -pipe"
CXXFLAGS="$CFLAGS"
CHOST="x86_64-pc-linux-gnu"
USE="mmx sse sse2 -gnome hal -cups bash-completion -ldap vim-syntax laptop"

PORTDIR="/var/lib/gentoo/repositories/gentoo"

MAKEOPTS="-j2"

FEATURES="distcc"

KBUILD_OUTPUT=/usr/src/build-current
VIDEO_CARDS="nvidia"
INPUT="evdev"
DISTDIR="/home/data/distfiles"

ACCEPT_KEYWORDS=~amd64

P.S. How do you call having comments blocked completly if it's "braindamaged" to allow comments from multiple popular services?

Benchmark Paludis 0.38 and Portage 2.1.6.9

I didn't quite believe the benchmark done by bonsaikitten so I did my own and wasn't able to reproduce the results.

Env (same as bonsai used)

• Portage 2.1.6.9
• Paludis 0.36.0
• hot kernel cache
• no metadata cache in gentoo repo
• no package manager on-disk cache (nuked /var/cache/edb/dep and /var/cache/paludis/metadata/gentoo)

# time emerge -puD sys-apps/portage
real    5m32.896s
user    3m36.263s
sys     1m53.655s

# time paludis -ip sys-apps/portage
real    1m31.509s
user    1m0.432s
sys     0m32.432s

# time paludis -ip sys-apps/portage -E portage (bonsai doesn't have a paludis config)
real    1m36.575s
user    1m2.969s
sys     0m36.501s

And with cold kernel cache (echo 3 > /proc/sys/vm/drop_caches)

# time emerge -puD sys-apps/portage
real    6m25.499s
user    3m34.228s
sys     1m53.624s

# time paludis -ip sys-apps/portage
real    2m38.976s
user    1m3.312s
sys     0m38.318

So reading the necessary ebuilds (and friends) takes roughly a minute on my box.

A little curiosity as a bonus:

# strace -e file paludis -ip sys-apps/portage 2>&1 | wc -l
14219
# strace -e file emerge -puD sys-apps/portage 2>&1 | wc -l
32240

Interesting difference, let's look closer at what portage does:

access("/var/lib/gentoo/repositories/gentoo/sys-devel/autoconf/autoconf-2.63.ebuild",
stat("/var/lib/gentoo/repositories/gentoo/sys-devel/autoconf/autoconf-2.63.ebuild",
open("/var/cache/edb/dep/var/lib/gentoo/repositories/gentoo/sys-devel/autoconf-2.63",
stat("/var/lib/gentoo/repositories/gentoo/sys-devel/autoconf/autoconf-2.63.ebuild",
getcwd("/etc"...,
lstat("/var",
lstat("/var/lib",
lstat("/var/lib/gentoo",
lstat("/var/lib/gentoo/repositories",
lstat("/var/lib/gentoo/repositories/gentoo",
lstat("/home",
lstat("/home/data",
lstat("/home/data/distfiles",
lstat("/usr",
lstat("/usr/portage",
lstat("/usr/portage/rpm",
lstat("/var",
lstat("/var/tmp",
stat("/var/tmp/portage/sys-devel/autoconf-2.63/temp/environment",
stat("/usr/bin/sandbox",
access("/usr/bin/sandbox",

And the same for each ebuild read from the repo. Highlighted lines are interesting...

Mounting a raw kvm/qemu/... image

I have been doing some kernel hacking recently (mostly uni tasks) and the ability to mount a kvm image turned out very useful. All you need is losetup (and loop device support in kernel of course) and kpartx. On gentoo they are in sys-apps/util-linux and sys-fs/multipath-tools accordingly.

Create a new image:

# kvm-img create -f raw gentoo.raw.img 4G
# losetup -f
/dev/loop/0
# losetup /dev/loop/0 gentoo.raw.img
# cfdisk /dev/loop/0
# kpartx -av /dev/loop/0
add map 0p1 (253:1): 0 8385867 linear /dev/loop0 63
# mkfs /dev/mapper/0p1
# kpartx -dv /dev/loop/0
del devmap : 0p1
# losetup -d /dev/loop/0

Mount an already formatted image:

# kpartx -av gentoo.raw.img
add map loop0p1 (253:1): 0 8385867 linear /dev/loop0 63
# mount /dev/mapper/loop0p1 /mnt/guests/gentoo-kvm

Note that it's dangerous to have the image mounted like that and use it with kvm/qemu/etc at the same time, so here is the reverse process:

# umount /mnt/guests/gentoo-kvm
# kpartx -dv gentoo.raw.img
del devmap : loop0p1
loop deleted : /dev/loop0