window.name hack taken a step further: full XHR proxying

My current GWT project requires a secure cross-site rpc solution so I started digging:

• JSONP aka script tag inclusion - someone even implemented it in GWT, but it's insecure and limited by the url lenght size limit.
• IFrames, Fragment Identifiers and XHR Proxying - secure but the communication via #hash is far from perfect.
• window.name hack with from submission - that's really nice, but limited to form submission and requires server-side changes (special response format)

I liked the window.name hack the most and started implementing it in GWT. And while doing so I asked myself a question - why is it only limited to form submission? Well, it's not! You can use window.name for communication like in the #hash communication and do full XHR proxying. Look at this new (at least I didn't see it anywhere else) cross-site communication scheme:

• Create an iframe
• Encode XHR params and a dummy localUrl in the iframe's window.name
• Change the iframe's location to the server's proxy script (i.e. if you want to send a request to example.org, example.org needs to provide a proxy script at e.g. example.org/cross_site_proxy.html)
• The proxy script reads params from window.name and creates the real XHR
• Fire the XHR and encode the response (all of it) in window.name
• Change the location back to localUrl
• Read the response from the iframe's window.name

It is important to set proper caching headers for both localUrl and server's proxy script so that they will be loaded from browser's cache w/o any additional requests.

Pros:

• As secure as Fragment Identifiers XHR proxying and the original window.name hack
• Full XHR proxying like in the Fragment Identifiers XHR proxying
• No server changes needed other than providing the proxy script

I am currently finishing my proof of concept implementation in GWT and I will do a follow-up on it shortly. In the meantime it can probably be easily implemented in js libraries like dojo as they have most of the required bits already done.

What do you think?

paludis.org

Just a quick note: I have taken over paludis.org and it finally points to where it should!

Benchmark Paludis 0.38 and Portage 2.1.6.9 #2

As it seems unclear to some, I have redone my benchmarks today with extra comments.

peper is using different configuration files for portage and paludis. So the comparison is quite ... uhm ... difficult.
I did include -E portage timinings too, didn't I? But let's look again.

# time paludis -ip sys-apps/portage -E portage
real    1m39.186s
user    1m2.442s
sys     0m38.543s

<bonsaikitten> dleverton: since I don't have any paludis config files my results are what should happen, -E seems to not do things the same (or there are some artifacts from the previous config left)
Still not enough? Ok. (Yes, I know these are silly.)

# mv /etc/paludis /etc/paludis.hidden
# time paludis -ip sys-apps/portage
real    1m39.407s
user    1m2.099s
sys     0m38.633s
# time paludis -ip sys-apps/portage -E portage
real    1m39.948s
user    1m2.133s
sys     0m38.873s

And portage didn't get any faster since yesterday either:

# time emerge -puD sys-apps/portage
real    5m30.694s
user    3m30.119s
sys     1m56.093s

What am I missing this time?

And to get things back to normal:

# mv /var/lib/gentoo/repositories/gentoo/metadata.hidden /var/lib/gentoo/repositories/gentoo/metadata
# time emerge --metadata
real    2m58.175s
user    0m16.025s
sys     0m8.808s

# time paludis --metadata
Usage error: Error handling command line: Bad argument '--metadata'

real    0m0.022s
user    0m0.007s
sys     0m0.004s

Oh, right, paludis doesn't do that.
And now:

# time emerge -puD sys-apps/portage
real    0m4.196s
user    0m3.924s
sys     0m0.177s

# time paludis -ip sys-apps/portage -E portage
real    0m3.082s
user    0m2.554s
sys     0m0.518s

My /etc/make.conf for the record:

CFLAGS="-march=athlon64 -O2 -pipe"
CXXFLAGS="$CFLAGS"
CHOST="x86_64-pc-linux-gnu"
USE="mmx sse sse2 -gnome hal -cups bash-completion -ldap vim-syntax laptop"

PORTDIR="/var/lib/gentoo/repositories/gentoo"

MAKEOPTS="-j2"

FEATURES="distcc"

KBUILD_OUTPUT=/usr/src/build-current
VIDEO_CARDS="nvidia"
INPUT="evdev"
DISTDIR="/home/data/distfiles"

ACCEPT_KEYWORDS=~amd64

P.S. How do you call having comments blocked completly if it's "braindamaged" to allow comments from multiple popular services?

Benchmark Paludis 0.38 and Portage 2.1.6.9

I didn't quite believe the benchmark done by bonsaikitten so I did my own and wasn't able to reproduce the results.

Env (same as bonsai used)

• Portage 2.1.6.9
• Paludis 0.36.0
• hot kernel cache
• no metadata cache in gentoo repo
• no package manager on-disk cache (nuked /var/cache/edb/dep and /var/cache/paludis/metadata/gentoo)

# time emerge -puD sys-apps/portage
real    5m32.896s
user    3m36.263s
sys     1m53.655s

# time paludis -ip sys-apps/portage
real    1m31.509s
user    1m0.432s
sys     0m32.432s

# time paludis -ip sys-apps/portage -E portage (bonsai doesn't have a paludis config)
real    1m36.575s
user    1m2.969s
sys     0m36.501s

And with cold kernel cache (echo 3 > /proc/sys/vm/drop_caches)

# time emerge -puD sys-apps/portage
real    6m25.499s
user    3m34.228s
sys     1m53.624s

# time paludis -ip sys-apps/portage
real    2m38.976s
user    1m3.312s
sys     0m38.318

So reading the necessary ebuilds (and friends) takes roughly a minute on my box.

A little curiosity as a bonus:

# strace -e file paludis -ip sys-apps/portage 2>&1 | wc -l
14219
# strace -e file emerge -puD sys-apps/portage 2>&1 | wc -l
32240

Interesting difference, let's look closer at what portage does:

access("/var/lib/gentoo/repositories/gentoo/sys-devel/autoconf/autoconf-2.63.ebuild",
stat("/var/lib/gentoo/repositories/gentoo/sys-devel/autoconf/autoconf-2.63.ebuild",
open("/var/cache/edb/dep/var/lib/gentoo/repositories/gentoo/sys-devel/autoconf-2.63",
stat("/var/lib/gentoo/repositories/gentoo/sys-devel/autoconf/autoconf-2.63.ebuild",
getcwd("/etc"...,
lstat("/var",
lstat("/var/lib",
lstat("/var/lib/gentoo",
lstat("/var/lib/gentoo/repositories",
lstat("/var/lib/gentoo/repositories/gentoo",
lstat("/home",
lstat("/home/data",
lstat("/home/data/distfiles",
lstat("/usr",
lstat("/usr/portage",
lstat("/usr/portage/rpm",
lstat("/var",
lstat("/var/tmp",
stat("/var/tmp/portage/sys-devel/autoconf-2.63/temp/environment",
stat("/usr/bin/sandbox",
access("/usr/bin/sandbox",

And the same for each ebuild read from the repo. Highlighted lines are interesting...

Mounting a raw kvm/qemu/... image

I have been doing some kernel hacking recently (mostly uni tasks) and the ability to mount a kvm image turned out very useful. All you need is losetup (and loop device support in kernel of course) and kpartx. On gentoo they are in sys-apps/util-linux and sys-fs/multipath-tools accordingly.

Create a new image:

# kvm-img create -f raw gentoo.raw.img 4G
# losetup -f
/dev/loop/0
# losetup /dev/loop/0 gentoo.raw.img
# cfdisk /dev/loop/0
# kpartx -av /dev/loop/0
add map 0p1 (253:1): 0 8385867 linear /dev/loop0 63
# mkfs /dev/mapper/0p1
# kpartx -dv /dev/loop/0
del devmap : 0p1
# losetup -d /dev/loop/0

Mount an already formatted image:

# kpartx -av gentoo.raw.img
add map loop0p1 (253:1): 0 8385867 linear /dev/loop0 63
# mount /dev/mapper/loop0p1 /mnt/guests/gentoo-kvm

Note that it's dangerous to have the image mounted like that and use it with kvm/qemu/etc at the same time, so here is the reverse process:

# umount /mnt/guests/gentoo-kvm
# kpartx -dv gentoo.raw.img
del devmap : loop0p1
loop deleted : /dev/loop0