HTTP/1.1 200 OK Date: Sat, 27 Dec 2025 09:06:44 GMT Server: Apache Last-Modified: Fri, 21 Mar 2014 14:43:47 GMT ETag: "c758-4f51ee9ca0ec0-gzip" Accept-Ranges: bytes Vary: Accept-Encoding Content-Encoding: gzip X-Frame-Options: SAMEORIGIN X-XSS-Protection: 1; mode=block Referrer-Policy: same-origin X-Content-Type-Options: nosniff Content-Length: 14423 Content-Type: application/xml https://teythoon.cryptobitch.de/This is a demo site for Nikola.enFri, 21 Mar 2014 15:43:47 GMTNikola <https://getnikola.com/>https://blogs.law.harvard.edu/tech/rsshttps://teythoon.cryptobitch.de/posts/debianhurd-switches-to-sysvinit/<div><p>Previously, Debian/Hurd used a home-grown init system. Last year, I participated in the gsoc and set out to make it boot using sysvinit instead.</p> <p>On Debian/Hurd, one can switch between the available init systems using <cite>update-alternatives(8)</cite>. With the latest Debian/Hurd packages uploaded earlier today, the priority of the old init system was lowered so that sysvinit is preferred. So as of now, Debian/Hurd is using sysvinit by default.</p> <p>(If you are upgrading your Debian/Hurd installation now, please remember that you must use <cite>reboot-hurd</cite> or <cite>halt-hurd</cite> to shut it down whenever you switch to a different init system.)</p> <p>And because screenshots of booting operating systems are just awesome, here is a current one. Note how smooth it is:</p> <pre class="literal-block"> start ext2fs: Hurd server bootstrap: ext2fs[device:hd0s1] exec init proc auth INIT: version 2.88 booting Using makefile-style concurrent boot in runlevel S. Activating swap...done. Checking root file system...fsck from util-linux 2.20.1 /dev/hd0s1: clean, 29799/181056 files, 206131/723200 blocks done. Creating compatibility symlink from /etc/mtab to /proc/mounts. ... (warning). mount: cannot remount /proc: Invalid argument Activating lvm and md swap...done. Checking file systems...fsck from util-linux 2.20.1 done. Cleaning up temporary files... /tmp. Mounting local filesystems...done. Activating swapfile swap...done. Cleaning up temporary files.... Configuring network interfaces...Internet Systems Consortium DHCP Client 4.2.2 Copyright 2004-2011 Internet Systems Consortium. All rights reserved. For info, please visit https://www.isc.org/software/dhcp/ Listening on Socket//dev/eth0 Sending on Socket//dev/eth0 DHCPDISCOVER on /dev/eth0 to 255.255.255.255 port 67 interval 7 DHCPREQUEST on /dev/eth0 to 255.255.255.255 port 67 DHCPOFFER from 10.0.2.2 DHCPACK from 10.0.2.2 bound to 10.0.2.15 -- renewal in 40021 seconds. done. Cleaning up temporary files.... INIT: Entering runlevel: 2 Using makefile-style concurrent boot in runlevel 2. Starting enhanced syslogd: rsyslogd. Starting deferred execution scheduler: atd. Starting periodic command scheduler: cron. Starting OpenBSD Secure Shell server: sshd. Debian GNU/Hurd jessie/sid debian console login: </pre></div>debianhurdinit systemssysvinithttps://teythoon.cryptobitch.de/posts/debianhurd-switches-to-sysvinit/Fri, 21 Mar 2014 14:35:37 GMThttps://teythoon.cryptobitch.de/posts/on-portability-of-init-systems/<div><p>There is one thing in the current init system debate that irritates me. It is about the portability of init systems. The new init systems are evaluated by how portable they are. This is one of the arguments that is most often brought against systemd, which is understandable given the polarizing attitude of one of systemd's authors. In this context, the currently used sysvinit is considered portable. But in my book, it is not portable at all.</p> <ul class="simple"> <li>There is no standard for the proc filesystem.</li> <li>Any program using the proc filesystem is therefore not portable.</li> <li>sysvinit uses the proc filesystem (in src/{bootlog,hddown,killall5}.c).</li> <li>Therefore, sysvinit is not portable.</li> </ul> <p>But why can we use sysvinit to boot Debian/{kFreeBSD,Hurd}?</p> <p>Debian/kFreeBSD uses <a class="reference external" href="https://www.freebsd.org/cgi/man.cgi?query=linprocfs&amp;sektion=5">linprocfs</a> for /proc to provide a familiar Linux-like environment for the userspace tools available in Debian (say, pgrep). linprocfs was originally written by the FreeBSD folks to support running Linux binaries using the <a class="reference external" href="https://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/linuxemu.html">Linux Binary Compatibility</a> layer.</p> <p>Debian/Hurd uses <a class="reference external" href="https://git.sceen.net/hurd/procfs.git/">procfs</a> to provide /poc. This procfs translator is written mainly to provide a Linux-compatible /proc filesystem for the same reason linprocfs is used by Debian/kFreeBSD.</p> <p>So sysvinit works on those system not because it is portable, but because the environment has been made Linux-like enough for sysvinit. We (most likely anyone not using Linux) often do this, because it is the easiest way to run popular software developed (mainly) for Linux. This is often the path of the least resistance, as opposed to getting the upstream project to support the native way of doing things on platform X.</p> <p>During gsoc last year I had to patch our procfs to finally be able to safely shut down Debian/Hurd systems using sysvinit. The problem was, that sysvinit at certain runlevel transitions (like shutting down, or I guess, switching to single user mode), sysvinit assumes that it is okay to stop and kill (almost) all processes on the system (that's what killall5 does). This might be okay on monolithic systems, but on (multiserver) microkernel systems like the Hurd, where your root filesystem and your network driver and stack are running as userspace processes, it is clearly not. I wonder how Linux systems using a FUSE-based root filesystem get away with this.</p> <p>This highlights that not only sysvinit depends on a Linux-specific kernel interface (/proc), but it also hard-codes assumptions about the system architecture.</p> <p>Amusingly, systemd get's this right (ok, I'm not sure if it does, but it could get this right...). systemd organizes processes in cgroups, one for each service it starts and one for each login session or something like that. It can (could?) kill only those processes in it was responsible for, leaving all essential translators (system servers) alone. In fact, even my tiny cgroupfs prototype can <a class="reference external" href="https://teythoon.cryptobitch.de/posts/cgroupfs-keeps-track-of-processes">keep track</a> of translators that are started by the root filesystem translator.</p></div>debianhurdinit systemssysvinithttps://teythoon.cryptobitch.de/posts/on-portability-of-init-systems/Sun, 26 Jan 2014 14:08:05 GMThttps://teythoon.cryptobitch.de/posts/final-gsoc-report/<div><p>This is my final report :) the GSoC was great, I learned a lot about the Hurd and Mach programming in general. I am also very pleased to announce that I reached my initial goal and almost all of my patches already made it upstream and into Debian :)</p> <p>I spent my last week <a class="reference external" href="https://lists.gnu.org/archive/html/bug-hurd/2013-09/msg00210.html">fixing issues I introduced</a> and splitting up <cite>/hurd/init</cite> into <a class="reference external" href="https://lists.gnu.org/archive/html/bug-hurd/2013-09/msg00245.html">two programs</a>. This would make it possible to integrate the patch that frees PID 1 for <cite>sysvinit</cite> into the Hurd upstream sources. I didn't quite finish the separation, but my proof of concept works and I will finish this as my next Hurd project.</p> <p>Looking back at the last fourteen weeks, I accomplished the following:</p> <p>I implemented /proc/mounts, umount, freed up PID 1 for sysvinit, fixed ifupdown, sysvinit and initscripts on Hurd, implemented a proof-of-concept cgroupfs and fixed many small issues along the way. Almost all of my patches are already upstream and in Debian, a Debian/Hurd booting using sysvinit is just a few uploads away.</p> <p>It has been a lot of fun and I will definitively see you around :)</p> <p>Justus</p></div>gsocdebianhurdhttps://teythoon.cryptobitch.de/posts/final-gsoc-report/Mon, 23 Sep 2013 13:04:26 GMThttps://teythoon.cryptobitch.de/posts/cgroupfs-is-as-cgroupy-as-it-gets/<div><p>... at least until the cgroup interface is fixed. So, what can it do?</p> <ul class="simple"> <li>There is <cite>tasks</cite> and <cite>cgroup.procs</cite>. There are no thread IDs on Hurd, so cgroupfs works only on a per-process basis, not per-threads. Consequently <cite>tasks</cite> has the same semantic as <cite>cgroup.procs</cite>. Seeing that PIDs and TIDs can be used (mostly) interchangeably on Linux I think this is okay to do.</li> <li>You can create and destroy cgroups, child processes are properly tracked.</li> <li>You can register an <cite>release_agent</cite> and it is executed whenever the last process in a cgroup dies.</li> <li>There is <cite>notify_on_release</cite> to enable or disable the use of <cite>release_agent</cite>.</li> <li>There is <cite>cgroup.clone_children</cite>, one can toggle this bit but it is ignored.</li> </ul> <p>So, what's missing?</p> <ul class="simple"> <li>There are no controllers. I haven't looked into this and resource accounting is one of Hurds weakest points, but it is fathomable that one could e.g. advise the scheduler inside the Mach kernel based upon the state of the cgroups if the cgroupfs process is sufficiently privileged (did I mention that any user can use cgroupfs?).</li> <li>The notification API aka <cite>cgroup.event_control</cite>. The Hurd lacks <cite>eventfd(2)</cite>, but even if that was implemented, this interface would still be impossible to implement. Rant below.</li> <li>A patch for gnumach to make this bulletproof. I made some encouraging progress with that one this week, but there's nothing presentable yet.</li> </ul> <p>So, what's wrong with Linux cgroup API?</p> <p>Well for one thing the whole API is underspecified. Yes, there is <a class="reference external" href="https://www.kernel.org/doc/Documentation/cgroups/cgroups.txt">Documentation/cgroups/cgroups.txt</a>, but that is not a specification, that's a howto at best. Second, the notification API is not particularly nice:</p> <pre class="literal-block"> To register a new notification handler you need to: - create a file descriptor for event notification using eventfd(2); - open a control file to be monitored (e.g. memory.usage_in_bytes); - write "&lt;event_fd&gt; &lt;control_fd&gt; &lt;args&gt;" to cgroup.event_control. Interpretation of args is defined by control file implementation; </pre> <p>Seriously? There is a POSIXly way to pass file-descriptors around, but smashing the decimal representation of it into a string is not the way to do that. Linux gets away with this hack because the kernel knows the process who wrote(2) that string in the first place, parse the string into an integer and look it up in the table of file descriptors for that process.</p> <p>Now the trouble for cgroupfs is, that it is not the kernel and even if it were, it wouldn't solve the problem because on Hurd there are no file descriptors (well there are, but that's only to appease all the POSIX programs out there). Instead Hurd has ports, and you can send messages to ports, and this is pretty much everything that you can do on a Mach system. Reading a file works roughly like this:</p> <ol class="arabic simple"> <li>You open a file and get a port X.</li> <li>You send a message like "I'm like really interested in the first Y bytes of that file" to X.</li> <li>Whoever has the receiving end of X (probably the one who gave you X in the first place) answers your request.</li> </ol> <p>Ports look pretty much like file descriptors, they are (usually small) integers, you can make them, destroy them, pass them around easily (yes, ports are first class objects in the Mach messaging system). Everything is implemented atop of this mechanism. It is transport-agnostic, the other end could be on another machine and you wouldn't even know. You can create proxies or filters (in fact, that is exactly how the firewall <a class="reference external" href="https://darnassus.sceen.net/gitweb/savannah_mirror/incubator.git/tree/907b010dd4158d8f838e46bd666d0f61caf432a6:/eth-filter">eth-filter</a> is implemented). It's beautiful and extensible at it's heart, like Lego bricks.</p> <p>So if X were a port to e.g. <cite>memory.usage_in_bytes</cite> and the cgroups interface would be less braindead^W^Wmore carefully designed so that on Hurd it could be transported like ports usually are, then cgroupfs could in fact use port X' to look up which file the caller is interested in (this is possible because cgroupfs was the one handing out the port in the first place) and generate notifications for that file. This is not possible when X is "serialized for transport" using sprintf because port names are specific for each process, so X != X'. The kernel would do the translation while sending the message, but it obviously cannot do that if the number is carried in a character array.</p> <p>I'm not sure what I'm going to do next week. The gsoc timeline suggests a soft-pencils-down, time to scrub code and write documentation, not sure that this is applicable to me as I have pushed most of my work upstream as early as possible. I guess I will nag Samuel so that he merges the outstanding patches and continue working on my gnumach patch.</p></div>gsocdebianhurdhttps://teythoon.cryptobitch.de/posts/cgroupfs-is-as-cgroupy-as-it-gets/Fri, 13 Sep 2013 17:59:49 GMThttps://teythoon.cryptobitch.de/posts/cgroupfs-keeps-track-of-processes/<div><p>Tl;dr!!elfel1 Screenshot (slightly edited and annotated shell trace):</p> <pre class="literal-block"> + settrans -ca /cgroup /hurd/cgroupfs + mkdir /cgroup/init /cgroup/rootfs + echo $$ &gt;&gt; /cgroup/init/tasks # $$ is 6 + echo 3 &gt;&gt; /cgroup/rootfs/tasks # pid 3 is the root filesystem + sleep 1m &amp; echo sleep has pid $! sleep has pid 16 + cat /proc/cmdline &gt; /dev/null + tail /cgroup/init/tasks /cgroup/rootfs/tasks ==&gt; /cgroup/init/tasks &lt;== 6 16 20 ==&gt; /cgroup/rootfs/tasks &lt;== 3 19 17 + pstree -p init(1)-+-auth(5) |-cgroupfs(14) |-ext2fs(3)-+-exec(4) | |-null(17) | |-pflocal(8) | |-procfs(19) | `-term(7) |-mach-defpager(10) |-root=device:hd0s1(2) `-sh(6)-+-pstree(21) `-sleep(16) </pre> <p>Isn't she a beauty?</p> <p>So we bind the <cite>cgroupfs</cite> translator to <cite>/cgroup</cite>, create two cgroups, <cite>init</cite> and <cite>rootfs</cite>, move the currently executing shell script (that later execs sysvinit) into the former and the root filesystem translator into the latter cgroup. We then spawn a <cite>sleep</cite> process and cat the content of <cite>/proc/cmdline</cite> into <cite>/dev/null</cite> which will make the root filesystem start the <cite>/hurd/procfs</cite> and the <cite>/hurd/null</cite> translator. We then inspect <cite>/cgroup/{init,rootfs}/tasks</cite> and find indeed all the newly spawned processes in the cgroup their parent process was in.</p> <p>This is accomplished by:</p> <ul class="simple"> <li>The cgroupfs translator: <a class="reference external" href="https://darnassus.sceen.net/gitweb/teythoon/cgroupfs.git/">https://darnassus.sceen.net/gitweb/teythoon/cgroupfs.git/</a></li> <li>A way to request notifications about process changes from the <cite>/hurd/proc</cite> server: <a class="reference external" href="https://lists.gnu.org/archive/html/bug-hurd/2013-09/msg00029.html">https://lists.gnu.org/archive/html/bug-hurd/2013-09/msg00029.html</a></li> </ul> <p>I also filed a bug report containing my patches for the <cite>sysvinit</cite> package (<a class="reference external" href="https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=721917">#721917</a>). This is the second bug report I filed during my gsoc, the first one was for the <cite>ifupdown</cite> package (<a class="reference external" href="https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=720531">#720531</a>) which Andrew Shadura improved and merged on the very next day, thanks Andrew!</p> <p>Next week I'll continue to improve the <cite>cgroupfs</cite> translator, work on the notification prototype (hopefully fixing non-root subhurds in the process, this requires a similar notification mechanism for newly created tasks and making <cite>/hurd/proc</cite> just a little subhurd aware) and trying to get my <a class="reference external" href="https://lists.gnu.org/archive/html/bug-hurd/2013-09/msg00023.html">gnumach patch</a> into a working shape (currently the parental relation of processes is a Hurd-only concept and relies upon processes telling the <cite>/hurd/proc</cite> server that a newly created process is their child. This is automatically done if the process uses <cite>fork(2)</cite> of course, but not if it uses <a class="reference external" href="https://www.gnu.org/software/hurd/gnumach-doc/Task-Creation.html">task_create</a> to start a new Mach task).</p></div>gsocdebianhurdhttps://teythoon.cryptobitch.de/posts/cgroupfs-keeps-track-of-processes/Fri, 06 Sep 2013 12:51:05 GMThttps://teythoon.cryptobitch.de/posts/what-will-i-do-next-cgroupfs-o/<div><p>With the <cite>ifupdown</cite> fixes that I published last week I actually reached my initial goal, that is to make Debian/Hurd boot using <cite>sysvinit</cite> and the <cite>initscripts</cite> provided by Debian. So on Monday we were discussing in #hurd what I could do next. Michael Banck suggested that I should port Upstart, but we agreed to do something different instead for two reasons:</p> <ol class="arabic simple"> <li>Upstart and systemd are somewhat competing to be the default init system for Debian, and we felt it might be inappropriate to get involved with this question as porting Upstart to Hurd would probably also enable it to be used on FreeBSD. The Upstart folks could then point out that Upstart is more portable because it runs on all kernels used by Debian.</li> <li>Upstart uses ptrace(2) to track child processes of servers it monitors. Obviously this is kind of a hack, and it was conjectured that Upstart would eventually use cgroups to do that. Also, the Hurd lacks support for ptrace(2) (that is most likely by choice by the way, ptrace(2) is not a nice interface and the Hurd (Mach actually) has much nicer interfaces to implement a debugger).</li> </ol> <p>So we decided that no matter how the struggle between Upstart and systemd turns out, the Hurd would eventually need to support cgroups. So I started to write a cgroupfs translator, it is in its early stages but it already looks and acts a lot like Linux' cgroups:</p> <pre class="literal-block"> % settrans -ac cg ./cgroupfs --release-agent=foobar % ls cg release_agent tasks % tail -n3 cg/tasks 11395 12869 1266 % mkdir cg/foo % echo 1266 &gt;&gt; cg/foo/tasks % tail -n3 cg/tasks cg/foo/tasks ==&gt; cg/tasks &lt;== 215 11395 12869 ==&gt; cg/foo/tasks &lt;== 1266 </pre> <p>To make this fully functional I will have to modify <cite>/hurd/proc</cite> and most likely also GNU Mach, but on the bright side this will help make <a class="reference external" href="https://www.gnu.org/software/hurd/hurd/subhurd.html">subhurds</a> (Hurds native, by-design-for-free-and-without-overhead container like functionality) work better and more securely (among other things this could enable non-root users to start subhurds). I will also look into porting <a class="reference external" href="https://libcg.sourceforge.net/">libcg</a> (I have a hacky patch series ready) so that we can actually test the cgroupfs translator. All current users of the cgroup interface are very Linux specific (surprise!), and libcg looks like the easiest one to port. And they do have a test suite that could help me improve the cgroupfs translator.</p></div>gsocdebianhurdhttps://teythoon.cryptobitch.de/posts/what-will-i-do-next-cgroupfs-o/Fri, 30 Aug 2013 16:56:18 GMThttps://teythoon.cryptobitch.de/posts/no-noweb-anymore/<div><p>... which is probably a good thing. But here is the boot log you all have been waiting for:</p> <pre class="literal-block"> start ext2fs: Hurd server bootstrap: ext2fs[device:hd0s1] exec init proc auth INIT: version 2.88 booting Using makefile-style concurrent boot in runlevel S. Activating swap...done. Checking root file system...fsck from util-linux 2.20.1 hd2 : tray open or drive not ready hd2 : tray open or drive not ready hd2 : tray open or drive not ready hd2 : tray open or drive not ready end_request: I/O error, dev 02:00, sector 0 /dev/hd0s1: clean, 44693/181056 files, 291766/723200 blocks done. Activating lvm and md swap...(default pager): Already paging to partition hd0s5! done. Checking file systems...fsck from util-linux 2.20.1 hd2 : tray open or drive not ready hd2 : tray open or drive not ready end_request: I/O error, dev 02:00, sector 0 done. Cleaning up temporary files... /tmp. Mounting local filesystems...done. Activating swapfile swap...(default pager): Already paging to partition hd0s5! done. df: Warning: cannot read table of mounted file systems: No such file or directory Cleaning up temporary files.... Configuring network interfaces...Internet Systems Consortium DHCP Client 4.2.2 Copyright 2004-2011 Internet Systems Consortium. All rights reserved. For info, please visit https://www.isc.org/software/dhcp/ Listening on Socket//dev/eth0 Sending on Socket//dev/eth0 *** stack smashing detected ***: dhclient terminated Aborted Failed to bring up /dev/eth0. done. Cleaning up temporary files.... Setting up X socket directories... /tmp/.X11-unix /tmp/.ICE-unix. INIT: Entering runlevel: 2 Using makefile-style concurrent boot in runlevel 2. Starting enhanced syslogd: rsyslogd. Starting deferred execution scheduler: atd. Starting periodic command scheduler: cron. Starting system message bus: dbusFailed to set socket option"/var/run/dbus/system_bus_socket": Protocol not available. Starting OpenBSD Secure Shell server: sshd. unexpected ACK from keyboard GNU 0.3 (debian) (console) login: root [...] root@debian:~# ifup /dev/eth0 Internet Systems Consortium DHCP Client 4.2.2 Copyright 2004-2011 Internet Systems Consortium. All rights reserved. For info, please visit https://www.isc.org/software/dhcp/ Listening on Socket//dev/eth0 Sending on Socket//dev/eth0 *** stack smashing detected ***: dhclient terminated Aborted Failed to bring up /dev/eth0. root@debian:~# dhclient -v -pf /run/dhclient.-dev-eth0.pid -lf /var/lib/dhcp/dhclient.-dev-eth0.leases /dev/eth0 Internet Systems Consortium DHCP Client 4.2.2 Copyright 2004-2011 Internet Systems Consortium. All rights reserved. For info, please visit https://www.isc.org/software/dhcp/ Listening on Socket//dev/eth0 Sending on Socket//dev/eth0 *** stack smashing detected ***: dhclient terminated Aborted root@debian:~# dhclient -pf /run/dhclient.-dev-eth0.pid -lf /var/lib/dhcp/dhclient.-dev-eth0.leases /dev/eth0 root@debian:~# ifup /dev/eth0 Internet Systems Consortium DHCP Client 4.2.2 Copyright 2004-2011 Internet Systems Consortium. All rights reserved. For info, please visit https://www.isc.org/software/dhcp/ Listening on Socket//dev/eth0 Sending on Socket//dev/eth0 DHCPREQUEST on /dev/eth0 to 255.255.255.255 port 67 DHCPACK from 10.0.2.2 bound to 10.0.2.15 -- renewal in 34108 seconds. ps: comm: Unknown format spec root@debian:~# halt Broadcast message from root@debian (console) (Fri Aug 23 19:42:19 2013): The system is going down for system halt NOW! INIT: Switching to runlevel: 0root@debian:~# INIT: Sending processes the TERM signal INIT: Sending processes the KILL signal Using makefile-style concurrent boot in runlevel 0. Stopping deferred execution scheduler: atd. task c10f53f8 deallocating an invalid port 2098928, most probably a bug. Asking all remaining processes to terminate...done. All processes ended within 1 seconds...done. Stopping enhanced syslogd: rsyslogd. Deconfiguring network interfaces...Internet Systems Consortium DHCP Client 4.2.2 Copyright 2004-2011 Internet Systems Consortium. All rights reserved. For info, please visit https://www.isc.org/software/dhcp/ Listening on Socket//dev/eth0 Sending on Socket//dev/eth0 DHCPRELEASE on /dev/eth0 to 10.0.2.2 port 67 /dev/eth0 (2): inet address 0.0.0.0 netmask 255.255.255.0 broadcast 10.0.2.255 flags BROADCAST ALLMULTI MULTICAST mtu 1500 done. Deactivating swap...swapoff: /dev/hd0s5: 177152k swap space done. Unmounting weak filesystems...umount: /etc/mtab: Warning: duplicate entry for device /dev/hd0s1 (/servers/socket/26) umount: /etc/mtab: Warning: duplicate entry for device /dev/hd0s1 (/dev/cons) umount: could not find entry for: /dev/cons umount: could not find entry for: /servers/socket/26 done. mount: cannot remount /: Device or resource busy Will now halt. store a new irq 11init: notifying pfinet of shutdown...init: notifying tmpfs swap of shutdown...init: notifying tmpfs swap of shutdown...init: notifying tmpfs swap of shutdown...init: notifying ext2fs device:hd0s1 of shutdown...init: halting Mach (flags 0x8)... In tight loop: hit ctl-alt-del to reboot </pre> <p>With some tiny patches for <cite>ifupdown</cite> I've been able to resolve <a class="reference external" href="https://teythoon.cryptobitch.de/gsoc/#networkrelatedissues">network related issues</a>. All of them? Of course not, funny thing about developing for the Hurd is that once you fix one thing, then some other thing or code path is executed that has never been run on Hurd before, and therefore something else breaks. In this case I fixed <cite>ifupdown</cite> to generate valid names for the pid file and leases file and all of the sudden <cite>dhclient</cite> starts dying.</p> <p>Funny thing about that is, if one drops the <cite>-v</cite> flag from the <cite>dhclient</cite> invocation as I did it above, the crash isn't triggered and once the lease file has been successfully written, it is safe to add the <cite>-v</cite> flag again. Not yet sure what goes on there, then again, looking at the <a class="reference external" href="https://sources.debian.net/src/isc-dhcp/4.2.4-7/client/dhclient.c">source of isc-dhcp-client</a> it is not so surprising that it crashes :/</p> <p>When I first looked at <cite>ifupdown</cite> it was written in <a class="reference external" href="https://en.wikipedia.org/wiki/Noweb">noweb</a>, a literate programming tool. It is an interesting idea, even more so since (classic) c can be very verbose and cryptic. But it decouples the control flow from the structure of the program, which makes patching it quite a challenge since it is not as obvious where the changes have to go in. This is how <cite>ifupdown</cite> looked some weeks ago:</p> <pre class="literal-block"> % wc --lines ifupdown.nw 6123 ifupdown.nw % pdftk ifupdown.pdf dump_data | grep NumberOfPages NumberOfPages: 113 </pre> <p>The <cite>ifupdown.nw</cite> is the noweb source, from which seven .c, four .h, two .pl and one Makefile are generated. It also contains a redicioulus amount of documentation, to the point that the authors at several points did not now what to write and just drop some nonsensical lines into the file. The source also compiles to a 113 page pdf file, that contains all of the documentation and all of the code, not at all in the order that one would expect a program to be written, but in the order the authors chose to structure the documentation. Fortunately for me the maintainer decided to drop the noweb source and to add the generated files to the source control system. This made my job much easier :)</p> <p>So here are the patches I published this week:</p> <ul class="simple"> <li>Fixes for <cite>ifupdown</cite>: <a class="reference external" href="https://lists.gnu.org/archive/html/bug-hurd/2013-08/msg00063.html">https://lists.gnu.org/archive/html/bug-hurd/2013-08/msg00063.html</a></li> <li>Fixes for <cite>fakeroot-hurd</cite> and <cite>remap</cite>: <a class="reference external" href="https://lists.gnu.org/archive/html/bug-hurd/2013-08/msg00064.html">https://lists.gnu.org/archive/html/bug-hurd/2013-08/msg00064.html</a></li> </ul> <p>I must admit that I do not know exactly what I will do next week. Obviously fixing the <cite>dhclient</cite> crash would be nice, I'll look into that. But I'm surely find some useful thing to do.</p></div>gsocdebianhurdhttps://teythoon.cryptobitch.de/posts/no-noweb-anymore/Fri, 23 Aug 2013 21:30:48 GMThttps://teythoon.cryptobitch.de/posts/all-the-important-bits-are-there-please-test-and-review/<div><p>Finally, more bootlog-pr0n:</p> <pre class="literal-block"> start ext2fs: Hurd server bootstrap: ext2fs[device:hd0s1] exec init proc auth INIT: version 2.88 booting Using makefile-style concurrent boot in runlevel S. Activating swap...done. Checking root file system...fsck from util-linux 2.20.1 hd2 : tray open or drive not ready hd2 : tray open or drive not ready hd2 : tray open or drive not ready hd2 : tray open or drive not ready end_request: I/O error, dev 02:00, sector 0 /dev/hd0s1: clean, 44680/181056 files, 292234/723200 blocks done. Activating lvm and md swap...(default pager): Already paging to partition hd0s5! done. Checking file systems...fsck from util-linux 2.20.1 hd2 : tray open or drive not ready hd2 : tray open or drive not ready end_request: I/O error, dev 02:00, sector 0 done. Cleaning up temporary files... /tmp. Mounting local filesystems...done. Activating swapfile swap...(default pager): Already paging to partition hd0s5! done. df: Warning: cannot read table of mounted file systems: No such file or directory Cleaning up temporary files.... Configuring network interfaces...inetutils-ifconfig: invalid arguments ifup: failed to open pid file /run/network/ifup-/dev/eth0.pid: No such file or directory Internet Systems Consortium DHCP Client 4.2.2 Copyright 2004-2011 Internet Systems Consortium. All rights reserved. For info, please visit https://www.isc.org/software/dhcp/ can't create /var/lib/dhcp/dhclient./dev/eth0.leases: No such file or directory Listening on Socket//dev/eth0 Sending on Socket//dev/eth0 DHCPDISCOVER on /dev/eth0 to 255.255.255.255 port 67 interval 8 DHCPREQUEST on /dev/eth0 to 255.255.255.255 port 67 DHCPOFFER from 10.0.2.2 DHCPACK from 10.0.2.2 can't create /var/lib/dhcp/dhclient./dev/eth0.leases: No such file or directory bound to 10.0.2.15 -- renewal in 34744 seconds. done. Cleaning up temporary files.... Setting up X socket directories... /tmp/.X11-unix /tmp/.ICE-unix. INIT: Entering runlevel: 2 Using makefile-style concurrent boot in runlevel 2. Starting enhanced syslogd: rsyslogd. Starting deferred execution scheduler: atd. Starting periodic command scheduler: cron. Starting system message bus: dbusFailed to set socket option"/var/run/dbus/system_bus_socket": Protocol not available. Starting OpenBSD Secure Shell server: sshd. unexpected ACK from keyboard GNU 0.3 (debian) (console) login: root root@debian:~# shutdown -h now Broadcast message from root@debian (console) (Fri Aug 16 20:02:47 2013): The system is going down for system halt NOW! INIT: Switching to runlevel: 0root@debian:~# INIT: Sending processes the TERM signal INIT: Sending processes the KILL signal Using makefile-style concurrent boot in runlevel 0. Stopping deferred execution scheduler: atd. task c10f72a8 deallocating an invalid port 2098928, most probably a bug. Asking all remaining processes to terminate...done. All processes ended within 1 seconds...done. Stopping enhanced syslogd: rsyslogd. Deconfiguring network interfaces...ifdown: failed to open pid file /run/network/ifdown-/dev/eth0.pid: No such file or directory Internet Systems Consortium DHCP Client 4.2.2 Copyright 2004-2011 Internet Systems Consortium. All rights reserved. For info, please visit https://www.isc.org/software/dhcp/ can't create /var/lib/dhcp/dhclient./dev/eth0.leases: No such file or directory Listening on Socket//dev/eth0 Sending on Socket//dev/eth0 /bin/sh: 1: ifconfig: not found done. Deactivating swap...swapoff: /dev/hd0s5: 177152k swap space done. Unmounting weak filesystems...umount: /etc/mtab: Warning: duplicate entry for device /dev/hd0s1 (/dev/cons) umount: could not find entry for: /dev/cons done. Unmounting local filesystems...done. mount: cannot remount /: Device or resource busy Will now halt. init: notifying pfinet of shutdown...init: notifying tmpfs swap of shutdown...init: notifying tmpfs swap of shutdown...init: notifying tmpfs swap of shutdown...init: notifying ext2fs device:hd0s1 of shutdown...init: halting Mach (flags 0x8)... In tight loop: hit ctl-alt-del to reboot </pre> <p>(You might note that df complains about not being able to read the mtab file. That is because it has been built with <cite>_PATH_MOUNTED</cite> being <cite>/var/run/mtab</cite>. This will correct itself when the <cite>coreutils</cite> package is being rebuilt against a patched libc.)</p> <p>I spent my last two weeks with polishing my patch series, that meant a lot of package rebuilds and that means a lot of waiting (even more so on the Hurd, short version: fakeroot-hurd could be fast but is not yet working properly, fakeroot-tcp <a class="reference external" href="https://www.gnu.org/software/hurd/open_issues/pfinet_timers.html">is slow</a>) and thus some frustration ;)</p> <p>Also I had to pay special attention so that the upgraded packages could be installed without accidentally breaking anything in the process. Making sysvinit pid 1 is surprisingly tricky in this regard since it breaks the ABI and requires a libc fix that also worked with the current Hurd servers.</p> <p>Here are the patches:</p> <ul class="simple"> <li>The mtab translator (v4) provides <cite>/proc/mounts</cite>: <a class="reference external" href="https://lists.gnu.org/archive/html/bug-hurd/2013-07/msg00259.html">https://lists.gnu.org/archive/html/bug-hurd/2013-07/msg00259.html</a></li> <li>When I wrote <cite>umount</cite> I thought I implemented all the flags used by the <cite>initscripts</cite> package. Turns out I forgot <cite>-d</cite>. umount.patch (v2): <a class="reference external" href="https://lists.gnu.org/archive/html/bug-hurd/2013-08/msg00018.html">https://lists.gnu.org/archive/html/bug-hurd/2013-08/msg00018.html</a></li> <li>Make sysvinit pid 1 (sysvinit kinda requires that), fix killall5 (that was freezing the system before) (v2): <a class="reference external" href="https://lists.gnu.org/archive/html/bug-hurd/2013-08/msg00020.html">https://lists.gnu.org/archive/html/bug-hurd/2013-08/msg00020.html</a></li> <li>Making sysvinit pid 1 breaks the ABI, since the glibc assumes <cite>/hurd/init</cite> to have pid 1: <a class="reference external" href="https://lists.gnu.org/archive/html/bug-hurd/2013-08/msg00046.html">https://lists.gnu.org/archive/html/bug-hurd/2013-08/msg00046.html</a> <a class="reference external" href="https://sourceware.org/ml/libc-alpha/2013-08/msg00205.html">https://sourceware.org/ml/libc-alpha/2013-08/msg00205.html</a></li> <li>So now killall5 is not freezing the system anymore (actually it froze all processes but most importantly the procfs translator providing <cite>/proc</cite> and it then tried to walk over <cite>/proc</cite> which obviously blocked killall5 itself). Cool. But then it looks into <cite>/proc/*/stat</cite> for the fields <cite>start_code</cite> and <cite>end_code</cite>. So I dug into the exec server where the ELF decoding happens and handed this to the proc server, but the procfs translator has to make use of this: <a class="reference external" href="https://lists.gnu.org/archive/html/bug-hurd/2013-08/msg00017.html">https://lists.gnu.org/archive/html/bug-hurd/2013-08/msg00017.html</a></li> <li>Patch series for the sysvinit package (v2). This contains a <cite>runsystem.sysv</cite> that can be enabled using <cite>update-alternatives --config runsystem</cite> and several small fixes: <a class="reference external" href="https://lists.gnu.org/archive/html/bug-hurd/2013-08/msg00038.html">https://lists.gnu.org/archive/html/bug-hurd/2013-08/msg00038.html</a></li> </ul> <p>So I had some spare time on my hand while waiting for numerous package rebuilds and I took this as an opportunity to read papers about Mach and to familiarize myself with mig, the <em>Mach Interface Generator</em>. While I have used it in the past, I had not yet looked at its implementation. And I had to patch the exec server, and there was both code implementing a questionable feature (on-demand unzipping of binaries) and code that was not even compiled (courtesy of the preprocessor) and had probably bit-rot by now. So I figured I could spend my time doing some cleanups:</p> <ul class="simple"> <li>Cleanups for the exec server: <a class="reference external" href="https://lists.gnu.org/archive/html/bug-hurd/2013-08/msg00049.html">https://lists.gnu.org/archive/html/bug-hurd/2013-08/msg00049.html</a></li> <li>Cleanups for mig: <a class="reference external" href="https://lists.gnu.org/archive/html/bug-hurd/2013-08/msg00055.html">https://lists.gnu.org/archive/html/bug-hurd/2013-08/msg00055.html</a></li> </ul> <p>I have rebuild all the necessary packages and uploaded them into an apt repository:</p> <pre class="literal-block"> deb https://teythoon.cryptobitch.de/gsoc/heap/debian unstable main </pre> <p>Please use unstable for now. Also make sure that you have a recovery plan for your Debian/Hurd installation if anything goes wrong. For your convenience there's a seed tarball containing packages with the appropriate sources.list.d snippets and the repository key:</p> <p><a class="reference external" href="https://teythoon.cryptobitch.de/gsoc/heap/debian/seed.tar">https://teythoon.cryptobitch.de/gsoc/heap/debian/seed.tar</a></p> <p>If you want to switch to the new <cite>runsystem.sysv</cite>, do:</p> <pre class="literal-block"> # update-alternatives --config runsystem </pre> <p>Whenever you switch runsystems, please use <cite>reboot-hurd</cite> to reboot the system. This is the most robust way.</p> <p>Known issues:</p> <ul> <li><p class="first">procfs hardcodes the default kernel pid to 2. This breaks <cite>/proc/uptime</cite> and any program relying on it, most notably top and friends. Until this is properly fixed, you can do:</p> <pre class="literal-block"> # settrans -apfg /proc /hurd/procfs -c -k 3 </pre> </li> <li><p class="first">The mtab translator should probably try to filter out non-filesystem translators. <cite>df</cite> complains loudly about <cite>/dev/cons</cite> for example.</p> </li> </ul> <p>Next week I will address the <a class="reference external" href="https://teythoon.cryptobitch.de/gsoc/#networkrelatedissues">network related issues</a>. By now they are the source of most of the error messages in the bootlog.</p></div>gsocdebianhurdhttps://teythoon.cryptobitch.de/posts/all-the-important-bits-are-there-please-test-and-review/Fri, 16 Aug 2013 17:07:22 GMThttps://teythoon.cryptobitch.de/posts/my-worst-week-yet/<div><p>This hasn't been my week. I had to fix two of my Hurd installations I use for development and testing. I spent countless hours waiting for some package to be rebuilt, reading scientific papers about adding stuff to the Mach kernel. Fascinating stuff like <a class="reference external" href="https://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.19.7348">thread migration</a> that would (among other things) allow for proper resource accounting on Mach kernels.</p> <p>Oh, and gcc is a sadistic bastard:</p> <pre class="literal-block"> In file included from printf-parsewc.c:2:0: printf-parsemb.c: In function ‘__parse_one_specwc’: printf-parsemb.c:407:1: internal compiler error: Segmentation fault Please submit a full bug report, with preprocessed source if appropriate. See &lt;file:///usr/share/doc/gcc-4.7/README.Bugs&gt; for instructions. The bug is not reproducible, so it is likely a hardware or OS problem. </pre> <p>A similar error came up like five hours into my libc rebuild. How nice of gcc to retry the build to figure out whether it is to blame or the environment only to throw away the successful build and abort my build process :/</p> <ul class="simple"> <li>I published my patch series addressing all <cite>sysvinit</cite>-related issues: <a class="reference external" href="https://lists.gnu.org/archive/html/bug-hurd/2013-08/msg00000.html">https://lists.gnu.org/archive/html/bug-hurd/2013-08/msg00000.html</a></li> <li>I since found three minor issues, so I'll roll another series shortly. But since this somewhat breaks the ABI (<cite>/hurd/init</cite> is no longer PID 1) this is waiting for a libc rebuild that I haven't managed to get done so far.</li> <li>I updated my patch series for the <cite>sysvinit</cite> package. I'm quite pleased with it as it is and I'll propose it for inclusion with the <cite>sysvinit</cite> package shortly. Unfortunately this depends on the patched <cite>hurd</cite> package with is blocked by the libc package :/</li> </ul> <p>Next week I'll finally propose my <cite>sysvinit</cite> patch series. Oh, and implement the <cite>-d</cite> flag in our <cite>umount</cite>. I thought I had implemented all the flags used by the <cite>initscripts</cite>, but somehow I missed <cite>-d</cite>. If I get bored, I'll take a look at the <a class="reference external" href="https://teythoon.cryptobitch.de/gsoc/#networkrelatedissues">network related issues</a>, but I bet I won't get there...</p></div>gsocdebianhurdhttps://teythoon.cryptobitch.de/posts/my-worst-week-yet/Fri, 09 Aug 2013 21:18:57 GMThttps://teythoon.cryptobitch.de/posts/there-is-no-spoon/<div><p>This is the Hurd shutting down:</p> <pre class="literal-block"> root@debian:~# init 0 INIT: Switching to runlevel: 0 INIT: Sending processes the TERM signal INIT: Sending processes the KILL signal Using makefile-style concurrent boot in runlevel 0. Stopping deferred execution scheduler: atd. Asking all remaining processes to terminate...done. All processes ended within 1 seconds...done. Stopping enhanced syslogd: rsyslogd. Deconfiguring network interfaces...ifdown: failed to open pid file /run/network/ifdown-/dev/eth0.pid: No such file or directory Internet Systems Consortium DHCP Client 4.2.2 Copyright 2004-2011 Internet Systems Consortium. All rights reserved. For info, please visit https://www.isc.org/software/dhcp/ can't create /var/lib/dhcp/dhclient./dev/eth0.leases: No such file or directory Listening on Socket//dev/eth0 Sending on Socket//dev/eth0 /bin/sh: 1: ifconfig: not found done. Deactivating swap...swapoff: /dev/hd0s5: 177152k swap space done. mount: cannot remount /: Device or resource busy Will now halt. INIT: no more processes left in this runlevel </pre> <p>The important line is the one saying <cite>Asking all remaining processes to terminate...done.</cite>. What happens there is that <cite>/sbin/killall5</cite> is run, actually does its job and the system survives that. What was needed to fix <cite>killall5</cite> is to mark some processes (all translators being started as <cite>root</cite> as well as any essential processes) as important and to exempt them from being frozen and killed by <cite>killall5</cite>. Furthermore it was necessary to fill in the correct values for the <cite>start_code</cite> and <cite>end_code</cite> fields of the <cite>/proc/*/stat</cite> records (this used to be an issue for Debian/kFreeBSD <a class="reference external" href="https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=344546">as well</a>).</p> <p>I am still in the process of cleaning up the patch series, I will finish that first thing Monday morning. I also spent my time polishing my mtab translator, the patch series is in its fourth revision and I consider it ready for inclusion:</p> <p><a class="reference external" href="https://lists.gnu.org/archive/html/bug-hurd/2013-07/msg00259.html">https://lists.gnu.org/archive/html/bug-hurd/2013-07/msg00259.html</a></p> <p>I also had this moment of clarity and enlightenment. For the first time I thought I finally understood what the Hurd really was. It is just a bunch of Mach programs that talk to each other in this really strange language and as a result they behave very much like the equivalent programs would on a different POSIX-like system. This is of course hidden away from the application programmer in our libc. This language is a mere convention, Richard calls this <cite>system personality</cite>. On some other level I have known this for a long time, but I never grasped the profound implications for users and developers, and for composability and the security of the system as a whole. I will try to show some of the cool stuff that can be done with such a system, stuff that is hard to do on monolithic kernels.</p> <p>And I had to patch the exec server. You know what they say, once you looked into <cite>/hurd/exec</cite>, there is no going back.</p> <p>I am very happy to report that I addressed all three major issues I identified in my <a class="reference external" href="https://teythoon.cryptobitch.de/posts/getting-into-the-mood-my-second-week">second week</a>. Next week I will clean up my <cite>initscripts</cite> patch series and submit it for inclusion. I will also rebuild the <cite>hurd</cite>, <cite>sysvinit</cite> and <cite>libc</cite> packages with all my patches included for broader testing. And if I ever run out of stuff to do first, I will have to look at the <a class="reference external" href="https://teythoon.cryptobitch.de/gsoc/#networkrelatedissues">network related issues</a>.</p></div>gsocdebianhurdhttps://teythoon.cryptobitch.de/posts/there-is-no-spoon/Fri, 02 Aug 2013 10:34:23 GMT