tag:blogger.com,1999:blog-24412775486131532912024-03-08T07:28:39.032+01:00Piotr Jaroszyński's blogaka peper's blogPiotr Jaroszyński's bloghttp://www.blogger.com/profile/03869937582891371047noreply@blogger.comBlogger17125tag:blogger.com,1999:blog-2441277548613153291.post-8740088634450375442011-03-01T23:40:00.004+01:002011-08-23T19:35:03.924+02:00cudaram - a block device exposing NVIDIA GPUs' RAM implemented with CUDAI have been looking for a <strike>silly</strike>learning project for Linux kernel for some time and I think I have finally come up with something suitable.<br />
Why not use the extra free RAM on your GPU for something useful while not hindering your normal GPU use (vdpau, desktop effects etc.)? I started looking and I didn't really find anything of the sort, the closest was an entry in the Gentoo Wiki - <a href="http://en.gentoo-wiki.com/wiki/TIP_Use_memory_on_video_card_as_swap">Use memory on video card as swap</a> - but that approach forces you to use the vesa driver and you can't map the whole GPU RAM like that at least on my GTX 260 anyway.<br />
<br />
I came to the conclusion that the most generic way would be to expose the extra resources via a block device.<br />
<br />
Next up was actually figuring out how to do that. I have immediately thought about CUDA as I have had some contact with it before and I knew you can easily manage GPU RAM with it. And it's also possible to use it without disrupting the normal chores of your GPU - like actually displaying something on your monitor.<br />
<br />
Sounds perfect, right? The only problem is that both CUDA toolkit and the NVIDIA drivers are closed-source and their interactions aren't documented anywhere. The only API they provide is in userspace and hence accessing it from kernel isn't easily doable. One could try and reverse-engineer the internal API, but I didn't want to go there with my first project especially as both the toolkit and drivers are constantly evolving and surely changing the API along the way.<br />
I ended up deciding to be nice and use the CUDA userspace API. It complicates the design, but that actually might be a plus given that it's supposed to be a learning project. The final design follows:<br />
<br />
<b>cudaram</b> kernel module <-> <b>cudaramd</b> userspace daemon <-> CUDA toolkit <-> nvidia kernel module<br />
<br />
Basically, it is a block device with its storage implemented in userspace. There are similar things out there - like <a href="http://nbd.sourceforge.net/">NBD</a> and <a href="https://patchwork.kernel.org/patch/37461/">ABUSE</a>. There is also <a href="http://fuse.sourceforge.net/">FUSE</a>, but that's at different level.<br />
I decided to write my own module for two reasons, firstly I wanted to learn as much as possible and secondly it gives me the most flexibility should I need it later.<br />
<br />
And so I did. I have pushed the code to <a href="https://github.com/peper/cudaram">https://github.com/peper/cudaram</a>. There is a basic README included in the repo too if you are brave enough to try it out :) I wouldn't necessarily recommend that as at this point I would like to mostly gather feedback on my implementation.<br />
<br />
Nevertheless it does seem to work and it's pretty fast at least for some loads:<br />
<pre class="brush: plain; gutter: false;">$ mkfs.ext2 /dev/cudaram0
...
$ mount /dev/cudaram0 /mnt/cuda
# /mnt/tmpfs/foo is a 250MB file in tmpfs
# copy from tmpfs to cudaram
$ dd if=/mnt/tmpfs/foo of=/mnt/cuda/foo bs=$((1000*1000)) count=250 conv=fdatasync
250000000 bytes (250 MB) copied, 0.296378 s, 844 MB/s
# copy from cudaram to tmpfs
$ echo 3 > /proc/sys/vm/drop_caches
$ dd if=/mnt/cuda/foo of=/mnt/tmpfs/foo bs=$((1000*1000)) count=250 conv=fdatasync
250000000 bytes (250 MB) copied, 0.275168 s, 909 MB/s
# copy from tmpfs to tmpfs
$ dd if=/mnt/tmpfs/foo of=/mnt/tmpfs/foo2 bs=$((1000*1000)) count=250 conv=fdatasync
250000000 bytes (250 MB) copied, 0.13663 s, 1.8 GB/s
</pre>So <b>cudaram</b> is about 2 times slower than tmpfs at copying one big file. Doesn't seem too bad at all for the first version. What helps it here is that in this load it's getting pretty big I/O requests. Where it might hurt is a lot of small requests - that should be obvious after reading the following overview of the implementation.<br />
<br />
Currently the <b>cudaram</b> module creates a few cudaramX block devices with matching cudaramctlX control devices. The <b>cudaramd</b> daemon allocates the GPU RAM and a transfer buffer and starts communicating with the <b>cudaram</b> module via ioctl()s on the control device.<br />
<br />
After initialization the flow is as follows:<br />
<ul><li>ioctl() call start</li>
<li>Submit the last completed I/O request</li>
<li>If the I/O request direction was READ then the module copies the data from the transfer buffer</li>
<li>The module marks the request as completed</li>
<li>Sleep waiting for more requests</li>
<li>If there are pending I/O requests, take the first one from the queue</li>
<li>If the I/O request direction is WRITE then copy the data to the transfer buffer</li>
<li>Return the data required to complete the request (sector number etc.)</li>
<li>ioctl() call end</li>
<li>Perform the request, i.e. copy data between the GPU RAM and the transfer buffer</li>
<li>Start over</li>
</ul>The I/O requests are queued asynchronously by the block device susbsytem.<br />
<br />
For any more details I will have to redirect you to the source code.<br />
<br />
TODO:<br />
<ul><li>Figure out whether making swap to cudaram work is possible - currently it can deadlock! Might be especially tricky given that the nvidia driver is closed-source</li>
<li>Allocating GPU RAM in smaller chunks - to avoid fragmentation problems</li>
<li>Allocating GPU RAM on demand</li>
<li>Test different userspace-kernel communication schemes - e.g. vmapping the userspace buffer, adding separate read/write buffers, etc.</li>
<li>Make it more user-friendly</li>
</ul>That's it for now. I would really appreciate all kinds of feedback, but especially a code review from kernel hackers :)Piotr Jaroszyński's bloghttp://www.blogger.com/profile/03869937582891371047noreply@blogger.com8tag:blogger.com,1999:blog-2441277548613153291.post-81357084199789985082010-07-10T16:00:00.000+02:002010-07-10T16:00:36.603+02:00Paludis 0.48.2 ReleasedI don't normally post about new <a href="http://paludis.org/">Paludis</a> releases, but this one brings a new client that many has been asking about.<br />
<br />
From NEWS:<br />
<br />
<ul><li>The ‘cave’ client is now enabled by default. ‘cave’ is a modular console client that will eventually replace ‘paludis’. It is currently reasonably functional and well tested, but does not yet have all the features present in ‘paludis’, and is thus not yet considered a complete replacement.</li>
</ul><br />
<br />
See <a href="http://ciaranm.wordpress.com/2010/03/31/paludis-is-going-into-a-cave/">this post</a> for more on cave.Piotr Jaroszyński's bloghttp://www.blogger.com/profile/03869937582891371047noreply@blogger.com1tag:blogger.com,1999:blog-2441277548613153291.post-20701110653074991282009-04-28T17:58:00.002+02:002009-04-28T18:22:14.711+02:00window.name hack taken a step further: full XHR proxyingMy current <a href="http://code.google.com/webtoolkit/">GWT</a> project requires a secure cross-site rpc solution so I started digging:<br />
<ul><li>JSONP aka script tag inclusion - <a href="http://groups.google.com/group/Google-Web-Toolkit-Contributors/browse_thread/thread/94c18c4ec158070c">someone even implemented it in GWT</a>, but it's insecure and limited by the url lenght size limit.</li>
<li><a href="http://www.dojotoolkit.org/node/87">IFrames, Fragment Identifiers and XHR Proxying</a> - secure but the communication via #hash is far from perfect.</li>
<li><a href="http://development.lombardi.com/?p=611">window.name hack with from submission</a> - that's really nice, but limited to form submission and requires server-side changes (special response format)</li>
</ul><br />
I liked the window.name hack the most and started implementing it in GWT. And while doing so I asked myself a question - why is it only limited to form submission? Well, it's not! You can use window.name for communication like in the #hash communication and do full XHR proxying. Look at this new (at least I didn't see it anywhere else) cross-site communication scheme:<br />
<ul><li>Create an iframe</li>
<li>Encode XHR params and a dummy <code>localUrl</code> in the iframe's <code>window.name</code></li>
<li>Change the iframe's location to the server's proxy script (i.e. if you want to send a request to example.org, example.org needs to provide a proxy script at e.g. example.org/cross_site_proxy.html)</li>
<li>The proxy script reads params from <code>window.name</code> and creates the real XHR</li>
<li>Fire the XHR and encode the response (all of it) in <code>window.name</code></li>
<li>Change the location back to <code>localUrl</code></li>
<li>Read the response from the iframe's <code>window.name</code></li>
</ul><br />
It is important to set proper caching headers for both localUrl and server's proxy script so that they will be loaded from browser's cache w/o any additional requests.<br />
<br />
Pros:<br />
<ul><li>As secure as Fragment Identifiers XHR proxying and the original window.name hack</li>
<li>Full XHR proxying like in the Fragment Identifiers XHR proxying</li>
<li>No server changes needed other than providing the proxy script</li>
</ul><br />
I am currently finishing my proof of concept implementation in GWT and I will do a follow-up on it shortly. In the meantime it can probably be easily implemented in js libraries like <a href="http://www.dojotoolkit.org/">dojo</a> as they have most of the required bits already done.<br />
<br />
What do you think?Piotr Jaroszyński's bloghttp://www.blogger.com/profile/03869937582891371047noreply@blogger.com1tag:blogger.com,1999:blog-2441277548613153291.post-63604284514785840402009-04-07T12:45:00.004+02:002009-04-07T18:20:54.193+02:00paludis.orgJust a quick note: I have taken over <a href="http://paludis.org" target="_blank">paludis.org</a> and it finally points to where it should!Piotr Jaroszyński's bloghttp://www.blogger.com/profile/03869937582891371047noreply@blogger.com0tag:blogger.com,1999:blog-2441277548613153291.post-32219396730308088492009-03-22T15:55:00.000+01:002009-03-22T15:59:27.231+01:00Benchmark Paludis 0.38 and Portage 2.1.6.9 #2As it seems <a href="http://gentooexperimental.org/~patrick/weblog/archives/2009-03.html#e2009-03-21T21_49_51.txt">unclear</a> to some, I have redone my <a href="http://blog.piotrj.org/2009/03/benchmark-paludis-038-and-portage-2169.html">benchmarks</a> today with extra comments.<br />
<br />
<i>peper is using different configuration files for portage and paludis. So the comparison is quite ... uhm ... difficult.</i><br />
I did include <code>-E portage</code> timinings too, didn't I? But let's look again.<br />
<pre class="brush: plain"># time paludis -ip sys-apps/portage -E portage
real 1m39.186s
user 1m2.442s
sys 0m38.543s
</pre><i><bonsaikitten> dleverton: since I don't have any paludis config files my results are what should happen, -E seems to not do things the same (or there are some artifacts from the previous config left)</i><br />
Still not enough? Ok. (Yes, I know these are silly.)<br />
<pre class="brush: plain"># mv /etc/paludis /etc/paludis.hidden
# time paludis -ip sys-apps/portage
real 1m39.407s
user 1m2.099s
sys 0m38.633s
# time paludis -ip sys-apps/portage -E portage
real 1m39.948s
user 1m2.133s
sys 0m38.873s
</pre><br />
And portage didn't get any faster since yesterday either:<br />
<pre class="brush: plain"># time emerge -puD sys-apps/portage
real 5m30.694s
user 3m30.119s
sys 1m56.093s
</pre><br />
What am I missing this time?<br />
<br />
And to get things back to normal:<br />
<pre class="brush: plain"># mv /var/lib/gentoo/repositories/gentoo/metadata.hidden /var/lib/gentoo/repositories/gentoo/metadata
# time emerge --metadata
real 2m58.175s
user 0m16.025s
sys 0m8.808s
# time paludis --metadata
Usage error: Error handling command line: Bad argument '--metadata'
real 0m0.022s
user 0m0.007s
sys 0m0.004s
</pre>Oh, right, paludis doesn't do that.<br />
And now:<br />
<pre class="brush: plain"># time emerge -puD sys-apps/portage
real 0m4.196s
user 0m3.924s
sys 0m0.177s
# time paludis -ip sys-apps/portage -E portage
real 0m3.082s
user 0m2.554s
sys 0m0.518s
</pre><br />
My /etc/make.conf for the record:<br />
<pre class="brush: plain">CFLAGS="-march=athlon64 -O2 -pipe"
CXXFLAGS="$CFLAGS"
CHOST="x86_64-pc-linux-gnu"
USE="mmx sse sse2 -gnome hal -cups bash-completion -ldap vim-syntax laptop"
PORTDIR="/var/lib/gentoo/repositories/gentoo"
MAKEOPTS="-j2"
FEATURES="distcc"
KBUILD_OUTPUT=/usr/src/build-current
VIDEO_CARDS="nvidia"
INPUT="evdev"
DISTDIR="/home/data/distfiles"
ACCEPT_KEYWORDS=~amd64
</pre><br />
P.S. How do you call having comments blocked completly if it's "braindamaged" to allow comments from multiple popular services?Piotr Jaroszyński's bloghttp://www.blogger.com/profile/03869937582891371047noreply@blogger.com22tag:blogger.com,1999:blog-2441277548613153291.post-32846366975155722942009-03-21T21:10:00.004+01:002009-03-22T04:14:07.480+01:00Benchmark Paludis 0.38 and Portage 2.1.6.9I didn't quite believe <a href="http://gentooexperimental.org/~patrick/weblog/archives/2009-03.html#e2009-03-21T16_43_58.txt">the benchmark done by bonsaikitten</a> so I did my own and wasn't able to reproduce the results.<br />
<br />
Env (same as bonsai used)<br />
<ul><li>Portage 2.1.6.9</li>
<li>Paludis 0.36.0</li>
<li>hot kernel cache</li>
<li>no metadata cache in gentoo repo</li>
<li>no package manager on-disk cache (nuked <code>/var/cache/edb/dep</code> and <code>/var/cache/paludis/metadata/gentoo</code>)</li>
</ul><br />
<pre class="brush: plain"># time emerge -puD sys-apps/portage
real 5m32.896s
user 3m36.263s
sys 1m53.655s
# time paludis -ip sys-apps/portage
real 1m31.509s
user 1m0.432s
sys 0m32.432s
# time paludis -ip sys-apps/portage -E portage (bonsai doesn't have a paludis config)
real 1m36.575s
user 1m2.969s
sys 0m36.501s
</pre><br />
And with cold kernel cache (<code>echo 3 > /proc/sys/vm/drop_caches</code>)<br />
<pre class="brush: plain"># time emerge -puD sys-apps/portage
real 6m25.499s
user 3m34.228s
sys 1m53.624s
# time paludis -ip sys-apps/portage
real 2m38.976s
user 1m3.312s
sys 0m38.318
</pre><br />
So reading the necessary ebuilds (and friends) takes roughly a minute on my box.<br />
<br />
A little curiosity as a bonus:<br />
<pre class="brush: plain"># strace -e file paludis -ip sys-apps/portage 2>&1 | wc -l
14219
# strace -e file emerge -puD sys-apps/portage 2>&1 | wc -l
32240
</pre><br />
Interesting difference, let's look closer at what portage does:<br />
<pre class="brush: plain; highlight: [10,13,16,19,20]">access("/var/lib/gentoo/repositories/gentoo/sys-devel/autoconf/autoconf-2.63.ebuild",
stat("/var/lib/gentoo/repositories/gentoo/sys-devel/autoconf/autoconf-2.63.ebuild",
open("/var/cache/edb/dep/var/lib/gentoo/repositories/gentoo/sys-devel/autoconf-2.63",
stat("/var/lib/gentoo/repositories/gentoo/sys-devel/autoconf/autoconf-2.63.ebuild",
getcwd("/etc"...,
lstat("/var",
lstat("/var/lib",
lstat("/var/lib/gentoo",
lstat("/var/lib/gentoo/repositories",
lstat("/var/lib/gentoo/repositories/gentoo",
lstat("/home",
lstat("/home/data",
lstat("/home/data/distfiles",
lstat("/usr",
lstat("/usr/portage",
lstat("/usr/portage/rpm",
lstat("/var",
lstat("/var/tmp",
stat("/var/tmp/portage/sys-devel/autoconf-2.63/temp/environment",
stat("/usr/bin/sandbox",
access("/usr/bin/sandbox",
</pre>And the same for each ebuild read from the repo. Highlighted lines are interesting...Piotr Jaroszyński's bloghttp://www.blogger.com/profile/03869937582891371047noreply@blogger.com6tag:blogger.com,1999:blog-2441277548613153291.post-33162570479188339672009-03-03T23:48:00.002+01:002009-03-07T23:29:26.733+01:00Mounting a raw kvm/qemu/... imageI have been doing some kernel hacking recently (mostly uni tasks) and the ability to mount a kvm image turned out very useful. All you need is <code>losetup</code> (and loop device support in kernel of course) and <code>kpartx</code>. On gentoo they are in <code>sys-apps/util-linux</code> and <code>sys-fs/multipath-tools</code> accordingly.<br />
<br />
Create a new image:<br />
<br />
<pre class="brush: plain"># kvm-img create -f raw gentoo.raw.img 4G
# losetup -f
/dev/loop/0
# losetup /dev/loop/0 gentoo.raw.img
# cfdisk /dev/loop/0
# kpartx -av /dev/loop/0
add map 0p1 (253:1): 0 8385867 linear /dev/loop0 63
# mkfs /dev/mapper/0p1
# kpartx -dv /dev/loop/0
del devmap : 0p1
# losetup -d /dev/loop/0
</pre><br />
Mount an already formatted image:<br />
<br />
<pre class="brush: plain"># kpartx -av gentoo.raw.img
add map loop0p1 (253:1): 0 8385867 linear /dev/loop0 63
# mount /dev/mapper/loop0p1 /mnt/guests/gentoo-kvm
</pre><br />
Note that it's dangerous to have the image mounted like that and use it with kvm/qemu/etc at the same time, so here is the reverse process:<br />
<br />
<pre class="brush: plain"># umount /mnt/guests/gentoo-kvm
# kpartx -dv gentoo.raw.img
del devmap : loop0p1
loop deleted : /dev/loop0
</pre>Piotr Jaroszyński's bloghttp://www.blogger.com/profile/03869937582891371047noreply@blogger.com5tag:blogger.com,1999:blog-2441277548613153291.post-30399010893559278312008-01-14T17:44:00.006+01:002009-03-07T23:02:25.399+01:00UNINSTALL_PROTECT in Paludis #3After a fair time of using the hook described in the <a href="http://blog.piotrj.org/2007/05/uninstallprotect-in-paludis-2.html">previous post</a> I realized that it noticeably slows down the uninstalls of packages with lots of files like <code>sys-kernel/gentoo-sources</code> and hence I decided to rewrite it as a <a href="http://paludis.pioto.org/configuration/hooks.html#py-hooks">Python hook</a> to see how it performs:<br />
<pre class="brush: py">UNINSTALL_PROTECT=["/lib64/modules/"]
def hook_run_unmerger_unlink_file_override(env, hook_env):
for path in UNINSTALL_PROTECT:
if hook_env["UNLINK_TARGET"].startswith(path):
return "skip"
else:
return ""
</pre>I was expecting a pretty big difference as for Python hooks an embedded interpreter is used and <a href="http://paludis.pioto.org/configuration/hooks.html#hook-hooks">.hook hooks</a> need a separate bash interpreter each time they are run. Nonetheless I was still surprised to see how big the difference really was for <code>sys-kernel/gentoo-sources-2.6.23-rX</code> uninstallation:<br />
<pre>.hook hook ~ 8m
python hook ~ 1m 15s
no hook ~ 1m
</pre>Python hook is not much slower than no hook at all and hence I didn't care to try and write it as a <a href="http://paludis.pioto.org/configuration/hooks.html#so-hooks">.so hook</a>, which probably would be even faster.Piotr Jaroszyński's bloghttp://www.blogger.com/profile/03869937582891371047noreply@blogger.com0tag:blogger.com,1999:blog-2441277548613153291.post-16325000477848253382007-08-24T16:41:00.002+02:002009-03-04T05:33:31.969+01:00Python 2.5 and Boost.PythonAs <a href="http://www.python.org/" title="Python">Python</a> 2.5 is now unmasked in <a href="http://www.gentoo.org/" title="Gentoo">Gentoo</a> I decided to try it with Boost.Python, more specifically with Python bindings for <a href="http://paludis.pioto.org/" title="Paludis">Paludis</a>. Everything seems to work, but, of course, boost needs to be rebuilt first.Piotr Jaroszyński's bloghttp://www.blogger.com/profile/03869937582891371047noreply@blogger.com5tag:blogger.com,1999:blog-2441277548613153291.post-89279241506086413022007-07-09T00:50:00.008+02:002009-03-07T23:07:36.906+01:00Boost.Python: docstrings in enumsFor quite some time I wanted to add docstrings to enums exposed with Boost.Python as my project's API docs seemed incomplete. Let's see how I found the solution eventually:<br />
<br />
Enum we want to expose:<br />
<pre class="brush: cpp">// Nice enum!
enum Foo
{
blah
};
</pre><br />
While exposing the enum itself couldn't be any simpler:<br />
<pre class="brush: cpp">BOOST_PYTHON_MODULE(enum_test)
{
bp::enum_<Foo> enum_foo("Foo");
enum_foo.value("blah", blah);
...
</pre>... adding <code>__doc__</code> is completly another story.<br />
<br />
First (silly) attempt:<br />
<pre class="brush: cpp">...
PyObject_SetAttrString(enum_foo.ptr(), "__doc__",
PyString_FromString("Nice enum!"));
};
</pre>"It must work" I thought, but what I got on import then was far from satisfying:<br />
<br />
<code>TypeError: attribute '__doc__' of 'type' objects is not writable</code><br />
<br />
After digging in <a href="http://boost.cvs.sourceforge.net/boost/boost/libs/python/src/object/" title="Boost.Python">Boost.Python</a>, <a href="http://docs.python.org/api/" title="Python C API docs">Python C API docs</a> and harassing people at #python:<br />
<pre class="brush: cpp">...
PyTypeObject * pto = reinterpret_cast<pytypeobject *>(enum_foo.ptr());
pto->tp_doc = "Nice enum!";
};
</pre>Thils looked really promising... and what? Nothing at all! Changing <code>tp_doc</code> had no effect whatsoever, and only after some more digging, it turned out it was too late to change it and the right way is:<br />
<pre class="brush: cpp">...
PyTypeObject * pto = reinterpret_cast<pytypeobject *>(enum_foo.ptr());
PyDict_SetItemString(pto->tp_dict, "__doc__",
PyString_FromString("Nice enum!"));
};
</pre>Boost.Python and Python C API are fun ;)Piotr Jaroszyński's bloghttp://www.blogger.com/profile/03869937582891371047noreply@blogger.com0tag:blogger.com,1999:blog-2441277548613153291.post-85118503948945827672007-06-20T10:43:00.002+02:002009-02-28T20:24:23.353+01:00Python bindings now in the scm ebuild!Python bindings are now available in the paludis-scm.ebuild from the <a href="http://paludis.pioto.org/trac/browser/overlay/" title="Paludis overlay">Paludis overlay</a>. Just set the python use-flag and reinstall. Comments are welcome, but keep in mind it's far from the final version :]<br />
<br />
Additional links: <a href="http://dev.gentooexperimental.org/%7Epeper/paludis/python/doc/" title="API docs">API docs</a>, <a href="http://paludis.pioto.org/trac/browser/trunk/python" title="repository">repository</a>Piotr Jaroszyński's bloghttp://www.blogger.com/profile/03869937582891371047noreply@blogger.com1tag:blogger.com,1999:blog-2441277548613153291.post-10304202230934555542007-06-16T17:45:00.002+02:002009-02-28T20:15:22.668+01:00Python bindings for Paludis #1I was supposed to give some updates about <a href="http://code.google.com/soc/gentoo/appinfo.html?csaid=EF1427E80304229E" title="my SoC project">my SoC project</a>, so here is the much delayed first one. <br />
<br />
I have been working on the bindings since the moment I thought about the project and had some code ready even for the initial project proposal. After it had been accepted I was continuing the work with bigger and smaller breaks. Meanwhile I was doing some contributions to Paludis in other areas, learning its internals and getting better at C++.<br />
<br />
I believe the first big date for my project was 5 April 2007 when bindings code was initially imported to the Paludis repository (<a href="http://paludis.pioto.org/trac/changeset/2881" title="r2881">r2881</a>). The second big date is still coming, which will be the release of Paludis 0.26 with the Python bindings included.<br />
<br />
The exact status of the bindings is best seen in the <a href="http://paludis.pioto.org/trac/browser/trunk/python" title="repository">repository</a> or in the <a href="http://dev.gentooexperimental.org/%7Epeper/paludis/python/doc/" title="API docs">API docs</a>, which I update every now and then.<br />
<br />
Currently I am working on bringing *DepSpec up to date and preparing for yet another Paludis API change. The bigger plan is to finish bindings for all the core classes sometime soon...Piotr Jaroszyński's bloghttp://www.blogger.com/profile/03869937582891371047noreply@blogger.com0tag:blogger.com,1999:blog-2441277548613153291.post-66386316645252150402007-05-25T09:20:00.008+02:002009-05-06T23:46:56.907+02:00UNINSTALL_PROTECT in Paludis #2<b>Update:</b> Check <a href="http://blog.piotrj.org/2008/01/uninstallprotect-in-paludis-3.html">this</a> for a better performing python version of this hook.<br />
<br />
I need to make a follow-up again as the hack described in my <a href="http://blog.piotrj.org/2007/05/uninstallprotect-in-paludis.html" title="UNINSTALL_PROTECT in Paludis">previous post</a> is now obsolote as <a href="http://paludis.pioto.org/" title="Paludis">Paludis</a>' trunk has now _override hooks for merger and unmerger actions. They are a little different than all the hooks till now as their effect is determined from their output, more specifically unmerger_unlink_*_override has two options: <br />
<ul><li>"skip" - skips the action</li>
<li>"force" - force it (without the usual tests like type or mtime check)</li>
</ul>And merger_install_*_override has only the "skip" option. Also no output means the default.<br />
<br />
Let me show an example:<br />
<br />
/etc/paludis/hooks/unmerger_unlink_file_override/uninstall_protect.hook:<br />
<pre class="brush: shell">#!/bin/bash
UNINSTALL_PROTECT="/lib64/modules/"
hook_run_unmerger_unlink_file_override() {
for PROTECT in ${UNINSTALL_PROTECT}; do
if [[ "${UNLINK_TARGET}" == "${PROTECT}"* ]]; then
echo "skip"
fi
done
}
</pre>As easily you can make INSTALL_MASK hook and others...<br />
<br />
This and <b>more</b> in forthcoming 0.26.Piotr Jaroszyński's bloghttp://www.blogger.com/profile/03869937582891371047noreply@blogger.com0tag:blogger.com,1999:blog-2441277548613153291.post-73689920635121271752007-05-11T18:37:00.005+02:002009-05-06T23:45:45.472+02:00UNINSTALL_PROTECT in Paludis<b>Update:</b> Check <a href="http://blog.piotrj.org/2007/05/uninstallprotect-in-paludis-2.html">this</a> for the current bash version of this hook or <a href="http://blog.piotrj.org/2008/01/uninstallprotect-in-paludis-3.html">this</a> for a better performing python one.<br />
<br />
This is somehow related to my previous posts about the moduledb as the idea also arose when playing with kernel modules.<br />
<br />
When installing modules for a new kernel I noticed that <a ="" href="http://paludis.pioto.org/" title="Paludis">Paludis</a> treated them as any other package and removed the old installs leaving my old kernel without the modules. Nothing really surprising here, just different than the portage behaviour. Wanting to avoid this in the future my first thought was just to add <code>/lib64/modules</code> to the <code>CONFIG_PROTECT</code>, but after a second I realised it wasn't the best idea as I didn't want to review modules' changes when upgrading, I just wanted them to be not removed when uninstalling. Hence that it wasn't long until making an <a ="" href="http://paludis.pioto.org/hooks.html#merger-hooks" title="Unmerger Hook">Unmerger Hook</a> came to my mind.<br />
<br />
/etc/paludis/hooks/uninstall_protect.hook:<br />
<pre class="brush: shell">#!/bin/bash
UNINSTALL_PROTECT="/lib64/modules/"
hook_run_unmerger_unlink_file_pre() {
for PROTECT in ${UNINSTALL_PROTECT}; do
if [[ "${UNLINK_TARGET}" == "${PROTECT}"* ]]; then
mv "${UNLINK_TARGET}" "${UNLINK_TARGET}.protect"
fi
done
}
hook_run_unmerger_unlink_file_post() {
if [[ -e "${UNLINK_TARGET}.protect" ]]; then
mv "${UNLINK_TARGET}.protect" "${UNLINK_TARGET}"
echo "protected '${UNLINK_TARGET}'"
fi
}
</pre><br />
And two symlinks pointing at it:<br />
<br />
<code>/etc/paludis/hooks/unmerger_unlink_file_pre/uninstall_protect.hook -> ../uninstall_protect.hook</code><br />
<br />
<code>/etc/paludis/hooks/unmerger_unlink_file_post/uninstall_protect.hook -> ../uninstall_protect.hook</code>Piotr Jaroszyński's bloghttp://www.blogger.com/profile/03869937582891371047noreply@blogger.com5tag:blogger.com,1999:blog-2441277548613153291.post-33085676375710794102007-04-23T10:31:00.005+02:002009-10-04T23:17:35.188+02:00moduledb in Paludis #2This is only a short follow-up to my <a href="http://blog.piotrj.org/2009/02/moduledb-in-paludis.html" title="previous post">previous post</a>.<br />
Support of <a href="http://ciaranm.org/show_post/113" title="dynamic configuration files">dynamic configuration files</a> has been implemented a few days ago and for a start I have made two simple dynamic sets:<br />
<br />
kernel-modules-ver.bash (set of kernel-modules with exact versions):<br />
<pre class="brush: shell">#!/bin/bash
sed -e 's/.*:/* =/' /var/lib/module-rebuild/moduledb
</pre><br />
kernel-modules.bash (set of kernel-modules - unversioned):<br />
<pre class="brush: shell">#!/bin/bash
shopt -s extglob
while read PKG; do
PKG=${PKG##*:}
PKG=${PKG%%-scm*([[:digit:]])}
PKG=${PKG%%-[[:digit:]]*([^-]|-[^[:digit:]])}
echo "* ${PKG}"
done < /var/lib/module-rebuild/moduledb
</pre>
<br />
And that's really only the beginning...Piotr Jaroszyński's bloghttp://www.blogger.com/profile/03869937582891371047noreply@blogger.com0tag:blogger.com,1999:blog-2441277548613153291.post-42784588631374313792007-04-14T19:47:00.001+02:002009-10-04T23:16:22.826+02:00moduledb in Paludis<b>Update</b>: Check <a href="http://blog.piotrj.org/2007/04/moduledb-in-paludis-2.html">this</a> for the current solution.<br />
<br />
With portage I was quite often using module-rebuild, more or less after every kernel update and thus I thought it would be nice to make something similar for Paludis, especially as it seemed really simple with Paludis' <a href="http://paludis.pioto.org/hooks.html" title="Hooks">Hooks</a> and <a href="http://paludis.pioto.org/sets.html" title="Sets">Sets</a>.<br />
<br />
So I made two simple hooks:<br />
<ul><li>${paludis_confdir}/hooks/ebuild_postinst_post/<a href="http://dev.gentooexperimental.org/%7Epeper/paludis/hooks/ebuild_postinst_post/modules_set_add.bash" title="modules_set_add.bash">modules_set_add.bash</a></li>
<li>${paludis_confdir}/hooks/ebuild_postrm_post/<a href="http://dev.gentooexperimental.org/%7Epeper/paludis/hooks/ebuild_postrm_post/modules_set_rm.bash" title="modules_set_rm.bash">modules_set_rm.bash</a></li>
</ul>And a simple one-liner to import entries from the moduledb:<br />
<br />
<code>sed -e 's/.*:/* =/' /var/lib/module-rebuild/moduledb > ${PALUDIS_CONFDIR}/sets/modules.conf</code><br />
<br />
But then I realized that it's not so perfect:<br />
<ul><li>Won't work when using Environment != Paludis ( ${PALUDIS_CONFDIR} won't be reliable then )</li>
<li>To actually rebuild the modules one has to use: <code>paludis --dl-reinstall always --dl-deps-default discard -i modules</code> and hope that the deps are satisfied :]</li>
<li>Doubles the work of the linux-mod.eclass</li>
</ul>So I decided to talk with <a href="http://www.ciaranm.org/" title="ciaran">Ciaran</a> about the above issues. Yet while we were talking he implemented a new paludis option (--dl-reinstall-targets auto|always|never), which affects the way paludis treats targets. Up till now the default "auto" was always used(="never" for sets and "always" for ordinary packages). When it is released the module rebuild will be simply done with:<br />
<br />
<code>paludis --dl-reinstall-targets always -i modules</code> and there won't be any risk of missing deps.<br />
<br />
Going further ciaran thought about allowing bash scripts in sets/, in short paludis runs a foo.bash script from sets/ and its output is interpreted as foo set. The moment it is implemented (probably today) and released we can forget about the two hooks above and make a modules.bash similar to the import one-liner.<br />
<br />
So don't forget to update Paludis when it's out ;]Piotr Jaroszyński's bloghttp://www.blogger.com/profile/03869937582891371047noreply@blogger.com0tag:blogger.com,1999:blog-2441277548613153291.post-45856530471527240032007-04-14T00:05:00.002+02:002009-02-28T19:58:06.737+01:00My brand new blog and Google Summer of Code<span style="font-weight: bold;">Welcome to my brand new blog!</span><br />
<br />
Thanks to <a href="http://www.s9y.org/" title="serendipity">seredipity</a> setting it up was quite easy :] I've been thinking about making one for quite some time and today I have eventually decided to do so. Why today? Well, <a href="http://code.google.com/soc/">GSoC</a>!<br />
<a href="http://code.google.com/soc/gentoo/appinfo.html?csaid=EF1427E80304229E" title="My proposal">My proposal</a> was accepted as one of the <a href="http://code.google.com/soc/gentoo/about.html" title="Gentoo projects">Gentoo projects</a>. In short, my task is to make python bindings for <a href="http://paludis.pioto.org/" title="Paludis">Paludis</a>, the Other Package Mangler for <a href="http://www.gentoo.org/" title="Gentoo">Gentoo</a>. If you want more details, check out <a href="http://dev.gentoo.org/%7Epeper/soc/application.txt" title="my application">my application</a> and, if you are even more desperate, <a href="http://dev.gentoo.org/%7Epeper/soc/paludis-python/" title="code">code</a>, which I have already written. Also <a href="http://www.ciaranm.org/" title="ciaran">ciaran</a> was nice enough to write <a href="http://www.ciaranm.org/show_post/111" title="a few words">a few words</a> about it, but don't forget to ignore the rubbish about Python :]<br />
So... I am planing to blog about progress of my project, some Gentoo stuff, and... time will tell.<br />
<br />
P.S. 1) My name is Piotr Jaroszyński (hence the domain), I live in <a href="http://www.gentoo.org/proj/en/devrel/roll-call/devmap.xml?dev=Peper" title="Warsaw">Warsaw</a> and am first year Computer Science student at Warsaw University.<br />
P.S. 2) I hope my pathetic 256 kbit/s upload will manage ;]Piotr Jaroszyński's bloghttp://www.blogger.com/profile/03869937582891371047noreply@blogger.com1