Carview!

CARVIEW

MOTORHOMES

Select Language

HTTP/1.1 200 OK Date: Fri, 16 Jan 2026 09:39:49 GMT Server: Apache Last-Modified: Mon, 05 May 2025 14:53:43 GMT ETag: "25531-63464aa95bcb8-gzip" Accept-Ranges: bytes Vary: Accept-Encoding Content-Encoding: gzip Content-Type: application/rss+xml Content-Length: 30890 Well, quite. https://wellquite.org/ Recent content on Well, quite. Matthew's Static Site Generator en-us Copyright © 2025, Matthew Sackman Mon, 05 May 2025 15:53:55 +0100 https://wellquite.org/android-chrome-512x512.png Well, quite. https://wellquite.org/ Payslips and tax: calculating your own https://wellquite.org/posts/payslips/ Mon, 05 May 2025 14:30:08 +0000 https://wellquite.org/posts/payslips/ In the UK, it’s very common that your employer pays you once a month. When this happens, they give you a document called a payslip, that has some numbers on it, such as how much your salary is, how much they paid you this month, how much went to HMRC in tax, how much went to your pension, and a few other numbers. But they never show any workings, so you really have no way to check whether any of these numbers are correct. There are plenty of online take-home-pay calculators, but these all focus on the full year; they have no facility to calculate your next payslip. About half way through April 2024, I stopped working for one company. Everything was wrapped up – I received my final payslip from them, along with my P45. I then had a few months off, and started a new job in July 2024. When you start a new job it always takes a while for money things to get sorted out, for example pension enrolment and sorting out pension contributions, so it’s really worthwhile to keep a close eye on your payslips particularly for these first few months. Mine were arriving and some numbers looked right, but other numbers, such as the amount of tax I was paying, were changing dramatically, month to month. I had no idea why; whether they should be changing like that; whether they were going to keep changing or would eventually settle down. I had no way to check any of these numbers. Was I going to get in trouble with HMRC and get investigated? I was also a little on edge because this was the first job where my pension contributions were using a thing called <a href="https://www.thepensionsregulator.gov.uk/en/employers/new-employers/im-an-employer-who-has-to-provide-a-pension/declare-your-compliance/ongoing-duties-for-employers-/earnings-thresholds">Qualifying Earnings</a>. In all my previous jobs, if I chose for 10% of my salary to go into my pension, then that’s what would happen. But now there was this thing called Qualifying Earnings, which is (numbers correct at time of writing) a band from £6240 to £50,270. If you’re earning, say £30k, then your x% contribution is actually x% of £30,000-£6240. If you’re earning above £50,270, then any further increase to your salary will not result in any extra contributions to your pension because you’re above the band. The 2008 Pensions Act, which created the legal requirement for all employees to have workplace pensions and for automatic enrolment (with a minimum 8% combined contribution from the employer and employee), also created this concept of Qualifying Earnings. I consider this is a pretty scummy way of reducing employer pension contributions for large firms. It complicates the maths and no doubt adds confusion for people trying to check their own payslips. Given that <a href="https://pensionsage.com/pa/Almost-three-quarters-of-Brits-not-on-track-to-retire-with-sufficient-funds.php">74% of the population have pensions that are too small to retire on</a>, this whole concept of Qualifying Earnings seems amoral at best. These days, a lot of smaller companies outsource their payroll processing. In my case, I was officially working for an international <a href="https://en.wikipedia.org/wiki/Employer_of_Record">Employer of Record</a> and they were then outsourcing payroll processing to local firms with country-specific expertise. So when I started asking questions, there was no ability to go and sit with someone and work through it. Or have a call. It was all messages passed across multiple different systems, and partial answers at best would come back several days later. Even if your payroll is done in-house, I strongly suspect that a lot of the time, some software package will be being used that does all the calculations and quite likely no one will actually understand or be able to explain the maths that’s going on. After a while of getting no-where, and after uncovering some substantial mistakes that had been made that affected me, I decided to spend some weekends actually figuring out how <a href="https://www.which.co.uk/money/tax/income-tax/tax-codes-paye/what-is-paye-aTECt5c4IVA0">PAYE</a> works, and writing some code that can calculate my next payslip. This <a href="https://pkg.go.dev/wellquite.org/tax">library is available</a> for anyone to use. There’s a <a href="https://pkg.go.dev/wellquite.org/tax#section-readme">README</a> that hopefully explains the basic principles of how the calculations are done. This only works if your <a href="https://www.gov.uk/tax-codes/what-your-tax-code-means">tax-code</a> ends in an <code>L</code>, and it only works if you’re in <a href="https://www.gov.uk/national-insurance-rates-letters/category-letters">National Insurance category</a> <code>A</code>. All the code can do is use some details you provide to predict your next payslips. Also, I’m not a trained accountant or financial adviser, and even for my own payslips, every month, the numbers don’t quite match up (but they’re within £1). So please treat this as a toy, rather than the basis for building a payroll processor! <h2 id="getting-started">Getting started</h2> The library is written in <a href="https://go.dev">Go</a> so you’ll need Go installed. Then, in a terminal do: <pre tabindex="0" class="chroma"><code>$ mkdir payslips $ cd payslips $ go mod init mypayslips $ go get wellquite.org/tax@latest </code></pre> Now we need to write a tiny amount of code. In your new <code>payslips</code> directory, create a <code>main.go</code> file, and open it in your editor. You want something like this: <pre tabindex="0" class="chroma"><code>package main import ( "fmt" "wellquite.org/tax" ) func main() { payslips := tax.Payslips{ { Year: 2024, TaxCode: "1257L", Salary: tax.Yearly(50000), PensionType: tax.Salary, EmployeePensionContributionRate: 0.05, EmployerPensionContributionRate: 0.03, }, { Salary: tax.Yearly(50000), PensionType: tax.Salary, EmployeePensionContributionRate: 0.05, EmployerPensionContributionRate: 0.03, }, {}, { Salary: tax.Yearly(60000), PensionType: tax.QualifyingEarnings, EmployeePensionContributionRate: 0.05, EmployerPensionContributionRate: 0.03, }, { Salary: tax.Yearly(60000), PensionType: tax.QualifyingEarnings, EmployeePensionContributionRate: 0.15, EmployerPensionContributionRate: 0.03, }, { Salary: tax.Yearly(60000), PensionType: tax.QualifyingEarnings, Expenses: 116.08, EmployeePensionContributionRate: 0.15, EmployerPensionContributionRate: 0.03, }, } payslips.Complete() fmt.Println(payslips) } </code></pre> We create a list of <a href="https://pkg.go.dev/wellquite.org/tax#Payslip">Payslips</a>. The first payslip must specify a year, and your tax-code. These details are automatically applied to the payslips that follow, if not explicitly provided. Many of the calculations rely on year-to-date totals, and so we must have a complete record of your payslips from the start of the tax year. So that means the first payslip is month 1 (in this example, April 2024), then month 2 (May 2024) and so on. If you have no income for a month then you can just put in an empty payslip (<code>{}</code>). The above example describes being paid in April and May 2024, then nothing in June, and then being paid (with a higher salary) in July, August and September. Save this <code>main.go</code> file. Then, back in your terminal, in your <code>payslips</code> directory, just do: <pre tabindex="0" class="chroma"><code>go run main.go </code></pre> You should get some output showing all sorts of calculations, including <a href="https://www.gov.uk/income-tax-rates">income tax, and personal allowance</a>. With a little luck, if you change the numbers to match your own salary and other details, the numbers produced should match quite closely your own payslips, provided nothing you’re doing is too exotic. There is documentation for all the <a href="https://pkg.go.dev/wellquite.org/tax#Payslip">different fields</a> that you can provide in each payslip. In general, the code will try to fill in missing values. It should be able to cope with things like salary-sacrifice, or, if you change job within a month and have several payslips for the same month, this should work too. Everything is run locally on your computer: please feel free to <a href="https://fossil.wellquite.org/repo/tax/dir">check the source</a> – there are no 3rd party libraries at all, and nothing imports the <code>net</code> package. It’ll work just the same if you yank out your network cable or disable your WiFi. Note however, this code is lightly tested. Whilst it works for me (and one or two friends), I make no claims that it correctly models the entirety of PAYE, so it may very well not work for you. Feedback, contributions, corrections, and patches are all very welcome! Rallentando, and the awfulness of anti-virus software https://wellquite.org/posts/rallentando/ Sat, 26 Apr 2025 17:30:08 +0000 https://wellquite.org/posts/rallentando/ Since I was a child, I’ve been playing the <a href="https://en.wikipedia.org/wiki/French_horn">French Horn</a>. I still play, and I take it quite seriously. I’m lucky enough to play with some good ensembles, and I perform many concerts each year. When learning difficult music, I often practise with a <a href="https://en.wikipedia.org/wiki/Metronome">metronome</a>. A metronome is a device that clicks or beeps regularly. You can set how often it clicks; for example you might set it to click 80 times a minute. The tempo (or speed) of a piece of music is often specified by the composer telling you how many beats per minute they want. This is guidance and not sacrosanct: you don’t normally have to play at exactly this tempo, and frequently music requires some implicit variation of tempo for it to be successful. But it certainly is an important piece of information from the composer, telling you at what basic speed the piece should be going. The problem with metronomes is that they can’t change their speed by themselves. For complex music, the speed could be changing a lot, or, maybe the number of beats per bar changes. This is annoying because it means for some pieces of music you have to keep stopping playing, readjust the metronome, and then continue on. There are also gradual changes in tempo in music: a part of a piece might gently get faster or slower. No metronome can cope with this: essentially, metronomes know nothing about the piece of music you’re playing. So in some spare time this year, I <a href="https://rallentando.me/">built one that does understand music</a>. It does nearly all the musical things you would want it to. As this is mainly a technical blog though, here I’ll focus on those aspects. It could have been an app; I’ve built apps before. But the faff of getting onto the play store, or the app store is just not worth it. The development tools are heavyweight and annoying. Sending out new versions requires approval processes, and you have to convince people to install something before they can use it. So I wanted this to be browser based. Also, modern web browsers are pretty amazing – great features and well designed APIs. Yep, all the old APIs are terrible and awful to work with, but everything that’s at all recent is pretty great. Drawing music in a browser is quite a challenge though. The way I’m doing it is I’m building an SVG, client-side. This was the first thing I started on: trying to figure out how to draw music in a browser, how to be able to detect clicks, and make it all interactive. The client side code is all generated from <a href="https://typescriptlang.org/">TypeScript</a> using the plain <code>tsc</code> to do the translation to JavaScript. I can’t stand complex tool-chains, and modern browsers are absolutely fine with loading modules (and you can do some really nice things with <a href="https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script/type/importmap">import maps</a> as we’ll see). I’m not even minimising the JavaScript: I’ve written the server myself; the modules are sent over the wire gzipped and I have correct cache-control settings using <a href="https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/Cache-Control#immutable">immutable and “cache-busting”</a>, so minimising the source just makes debugging life harder for no real gain. A score is essentially a list of blocks. I’m using a <a href="https://en.wikipedia.org/wiki/Conflict-free_replicated_data_type">CRDT</a> (the <a href="https://mattweidner.com/2022/10/21/basic-list-crdt.html">fugue list CRDT</a>) to allow local-first editing (and even offline editing). Dirty blocks get sent over a websocket and stored on the server, using <a href="https://wellquite.org/posts/golmdb/">LMDB</a> which is all very normal for me. The server has a neat part of its design: when you compile the server, all the static assets are embedded into the binary, thus making it a single self-contained executable. Now those assets (HTML, CSS, images, JavaScript etc) are just normally named files, but they can also be Go templates. When the server starts up, it works through these static assets, building HTTP routes for them. The HTTP routes contain in their paths the hashcode of the file – this is necessary for the <a href="https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/Cache-Control#immutable">cache busting</a>. If the asset is a template, the server knows how to run the template, and critically, I provide a <code>url</code> function in the template engine so that templates can get the URL of some other asset including its hashcode. So this means that if some HTML file needs to link to some CSS file, the HTML file as built into the server can be a template. At start up, this template gets run, it can invoke this <code>url</code> function, and it can find out the final URL of the CSS file. And of course this URL now influences the hashcode of the HTML file itself. This also plays very nicely with the <a href="https://developer.mozilla.org/en-US/docs/Web/Security/Subresource_Integrity">integrity</a> attribute you can put on all sorts of things these days. So it all works out rather nicely: if you consider the tree of file A importing files B and C, and file B imports file D, then if I make some change to file D, then it means its hashcode changes, and so its URL changes. This propagates up to file B, and from there to file A (but no change to file C). So it’s safe to serve all these static assets with immutable cache-control headers and rely on this lovely hashcode chaining. All of this work is done once, each time the server starts-up. And it’ll all explode and stop working if there’s ever a cycle in the graph of file imports. Now in practice, it seems that references between HTML, CSS, images, or JavaScript don’t seem to create cycles – at least I’ve not had a problem so far. But between JavaScript modules, it’s much more common, as you’d likely expect. But here, <a href="https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script/type/importmap">import maps</a> come to the rescue: in my TypeScript/JavaScript, I just import modules normally. I have a function in the template engine which knows how to generate an import-map of all my JavaScript modules, which gets injected into the top HTML page. This import-map provides both the rewriting of paths (to add hashcodes onto the paths), and also provides the <a href="https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script/type/importmap#integrity_metadata_map">integrity section</a>. This solves the problem of circular imports because it means the JavaScript itself never needs to contain the hashcode of any module it imports. Yet, if I change some JavaScript module, then its hashcode changes, which means the import-map changes, and so again, the browser is forced into correctly fetching the updated resource. <hr> A couple of weekends ago, I was up visiting my parents and I wanted to demonstrate this thing to them (they’re also musicians). They have computers running Windows. I tried loading up a demo score, and it just didn’t work. Their browsers were up to date. Nothing of note in the server logs, so I opened up the browser console and found errors from deserialisation of data coming over the websocket: it was claiming the data was corrupted. I’d never seen this in my own development and use. Checking a few other things, and I spotted that the source HTML for the page had had some additional <code><script></code> elements added to it: something was injecting some JavaScript. And then the penny dropped: this is <a href="https://en.wikipedia.org/wiki/Man-in-the-middle_attack">MITM</a> behaviour by some shitty anti-virus software – in this case, AVG. Some quick web searching, and yep, those products are also known for dicking around with websocket traffic: if you’re sending binary messages and you’re compressing the stream, it’s apparently quite common that the anti-virus software intercepts the traffic, and then screws up the framing leading your own code to face corrupted data. Completely ridiculous. In my case, disabling compression on the websocket was enough to prevent the corruption, and I then established that even for big scores, the initial load would be maybe 25kB of data over the websocket, so not compressing it isn’t terrible. What made me laugh though was this: the browser console was telling me both about the corrupted data, and also about the fact the browser was refusing to run some script, due to it violating <a href="https://developer.mozilla.org/en-US/docs/Web/HTTP/Guides/CSP">CSP</a> settings. It took me a moment to realise that the script that wasn’t being run, was the script that the anti-virus software was injecting into my pages! Now, import-maps can’t be external files, they have to be inline in the HTML. But there’s no way I’m serving HTML pages with a CSP header with <code>script-src 'unsafe-inline'</code>. Instead, I’m issuing a <code>script-src</code> CSP header with <code>'self'</code> and also the hashcode of import-map itself. What this says to the browser is that it can trust the import-map (because hashing it will give a hashcode that matches the CSP header), the import-map itself has its integrity for every JavaScript module it needs to load, and the CSP headers also tell the browser that it’s OK to load JavaScript modules from the same domain (this is the <code>'self'</code>). But, this does not give permission for the browser to run arbitrary bits of crap JavaScript that some awful anti-virus thing has injected! So, by making use of CSP and import-maps, you can defeat attackers from tampering with your website and code! Let's build! Boolean operations of polygons: Part 2 - Intersections https://wellquite.org/posts/lets_build/polygons_intersections/ Sat, 13 Jul 2024 18:00:00 +0000 https://wellquite.org/posts/lets_build/polygons_intersections/ In this series: <ul> <li><a href="https://wellquite.org/posts/lets_build/polygons_intro/">Let‘s build! Boolean operations of polygons: Part 1 - Introduction</a></li> <li><a href="https://wellquite.org/posts/lets_build/polygons_intersections/">Let’s build! Boolean operations of polygons: Part 2 - Intersections</a></li> </ul> <hr> In <a href="https://wellquite.org/posts/lets_build/polygons_intro/">Part 1</a> I explained what the boolean operations on polygons are, and I gave a motivating example by way of an algorithm that traces out the intersections of a set of overlapping polygons. That algorithm uses intersection (∩), union (∪), and subtraction (−) operations between polygons. It’s also necessary to identify duplicate polygons, which can be harder that you might first think: a polygon is typically defined by giving the coördinates of its vertices, in some order (for example, in <a href="https://en.wikipedia.org/wiki/GeoJSON">GeoJSON</a>, vertices (of an exterior ring) should be listed in anti-clockwise order). But the list of vertices can start at any of the vertices. More problematic is that the same polygon, calculated in different ways, may have tiny differences in its coördinates due to the rounding of floating point numbers. I’ll be spending a lot of time talking about floating point numbers very soon! One algorithm that is widely used for these boolean operations is given by the papers: <ul> <li><a href="https://wellquite.org/posts/lets_build/polygons/new_algorithm_paper.pdf">A new algorithm for computing Boolean operations on polygons</a> by Francisco Martínez, Antonio Jesús Rueda, and Francisco Ramón Feito</li> <li><a href="https://wellquite.org/posts/lets_build/polygons/simple_algorithm_paper.pdf">A simple algorithm for Boolean operations on polygons</a> by Francisco Martínez, Carlos Ogayar, Juan R. Jiménez, and Antonio Jesús Rueda</li> </ul> The algorithm described by these papers will be the focus of the blog posts in this series. These papers are not quite enough on their own to understand the algorithm. You will also want to refer to bits of these books: <ul> <li>“Computational Geometry - An Introduction” by Franco P. Preparata and Michael Ian Shamos</li> <li>“Geometric Tools for Computer Graphics” by Philip J. Schneider and David H. Eberly</li> </ul> Both of these you can find after a quick web search. Francisco Martínez has a <a href="https://www4.ujaen.es/~fmartin/bool_op.html">page for these papers</a> and if that’s disappeared then the <a href="https://web.archive.org/web/20231224051632/https://www4.ujaen.es/~fmartin/bool_op.html">Wayback Machine has a copy</a>. That page provides a link to a <a href="https://www4.ujaen.es/~fmartin/bop12.zip">C++ implementation</a> which also appears in various Github repositories, such as <a href="https://github.com/JimBobSquarePants/BPol/tree/master/reference">this one</a>. This algorithm has been quite widely implemented (sometimes with a few changes), and <a href="https://github.com/engelsjk/polygol?tab=readme-ov-file#implementation-survey-of-the-mart%C3%ADnez-rueda-feito-algorithm">this Github repo</a> provides a list of implementations, which I reproduce verbatim here: <table> <thead> <tr> <th>Language</th> <th>URL</th> <th>Notes</th> </tr> </thead> <tbody> <tr> <td>C++</td> <td><a href="https://www4.ujaen.es/~fmartin/bool_op.html">fmartin/bool_op</a></td> <td>original implementation</td> </tr> <tr> <td>JavaScript</td> <td><a href="https://github.com/w8r/martinez">w8r/martinez</a></td> <td>an extension of the original algorithm</td> </tr> <tr> <td>JavaScript</td> <td><a href="https://github.com/mfogel/polygon-clipping">mfogel/polygon-clipping</a></td> <td>originally forked from <a href="https://github.com/w8r/martinez">w8r/martinez</a>, notably used <a href="https://github.com/Turfjs/turf">here</a> and <a href="https://github.com/openstreetmap/iD">here</a></td> </tr> <tr> <td>JavaScript</td> <td><a href="https://github.com/mapbox/polyclip">mapbox/polyclip</a></td> <td>by <a href="https://github.com/mourner">mourner</a>! archived though</td> </tr> <tr> <td>JavaScript</td> <td><a href="https://github.com/velipso/polybooljs">velipso/polybooljs</a></td> <td>“based somewhat” on the original</td> </tr> <tr> <td>Go</td> <td><a href="https://github.com/engelsjk/polygol">engelsjk/polygol</a></td> <td>this one, a port of <a href="https://github.com/mfogel/polygon-clipping">mfogel/polygon-clipping</a></td> </tr> <tr> <td>Go</td> <td><a href="https://github.com/toanqng/martinez-rueda">toanqng/martinez-rueda</a></td> <td>based on <a href="https://github.com/kudm761/martinez-rueda-php">kudm761/martinez-rueda-php</a></td> </tr> <tr> <td>Go</td> <td><a href="https://github.com/akavel/polyclip-go">akavel/polyclip-go</a></td> <td>“known to have bugs”</td> </tr> <tr> <td>Rust</td> <td><a href="https://github.com/21re/rust-geo-booleanop">21re/rust-geo-booleanop</a></td> <td>closely follows <a href="https://github.com/w8r/martinez">w8r/martinez</a></td> </tr> <tr> <td>Python</td> <td><a href="https://github.com/lycantropos/martinez">lycantropos/martinez</a></td> <td>port of the original</td> </tr> <tr> <td>Python</td> <td><a href="https://github.com/lycantropos/clipping">lycantropos/clipping</a></td> <td></td> </tr> <tr> <td>PHP</td> <td><a href="https://github.com/kudm761/martinez-rueda-php">kudm761/martinez-rueda-php</a></td> <td>based on the original</td> </tr> <tr> <td>C/ActionScript3</td> <td><a href="https://github.com/akavel/martinez-src">akavel/martinez-src</a></td> <td>copy of the original with an AS3 port</td> </tr> <tr> <td>Scala</td> <td><a href="https://github.com/JonMcPherson/martinez-polygon-clipper">JonMcPherson/martinez-polygon-clipper</a></td> <td>ported from <a href="https://github.com/w8r/martinez">w8r/martinez</a></td> </tr> </tbody> </table> I have looked at some, but not all, of these implementations. Often, they seem to be quite mechanical translations of the original C++ code, so bugs may also have been translated across. Working closely and in detail on this algorithm has turned out to be very interesting, and has completely changed the way I think about floating point maths. That alone has been worth the effort to me. Please do not interpret any of what follows as criticisms of the original authors or their work; for the following reasons (amongst many others): <ol> <li>Most importantly, this algorithm seems to work well for a large number of people and applications. There’s a good chance that most reasonable people would consider the issues I’ve found to be “nit-picking”.</li> <li>Many years ago I attempted to do a PhD. I abandoned it after three years or so, but during that time, I attempted to write a few papers. I learnt one of the most frustrating things about writing papers is the page limit, which forces huge compromise on how you describe and explain your own work.</li> <li>They’ve provided some source code, which helps to explain many of the details elided from the papers (though in some places also suggests the details in the papers may not be quite right). This is vastly better than most academic papers.</li> <li>This is far from the first time I’ve come across papers that turn out to be missing certain key details. In fact nearly every paper describing an algorithm that I’ve wanted to implement turns out to be missing certain details.</li> <li>Without their work I couldn’t do this work.</li> </ol> <h1 id="some-constraints-on-the-input-polygons">Some constraints on the input polygons</h1> I need to define some terminology, and what is considered to be valid input to this algorithm. A polygon is defined by a list of (x,y) coördinates. Each (x,y) coördinate defines a vertex of the polygon. The polygon is <a href="https://en.wikipedia.org/wiki/Polygonal_chain#Closed">closed</a>: there is an edge between its final vertex back to its first vertex (some standards, such as GeoJSON, require that you restate the first vertex as the last. This typically makes iterating through the vertices and constructing the edges easier. I’m not specifying an input format here – anything will do). The set of vertices that define the outside perimeter of the polygon is called the “exterior ring”. A polygon may have zero or more <a href="https://en.wikipedia.org/wiki/Polygon_with_holes">“holes”</a>. A “hole” cuts out a region inside of the polygon. These are sometimes called “interior rings”. Both exterior and interior rings are sometimes collectively called “contours”. The “edge interior” is the set of all the points that lie on an edge, excluding the two vertices that define the ends of the edge. The “polygon interior” is the set of all points that lie inside the exterior ring and outside all interior rings. I rule as invalid any input polygon that: <ul> <li>is self-intersecting. It’s perfectly OK for a polygon to revisit the same vertex many times. But it’s not OK for a polygon to either: <ul> <li>have two edges that intersect each other away from their vertices (the “edge interior”),</li> <li>have a vertex of some edge A to lie on the edge interior of some other edge B.</li> </ul> In both these scenarios, adding additional vertices can solve these problems.</li> <li>has a hole that intersects with itself (i.e. the hole is self-intersecting in the same way as described above).</li> <li>has a hole that intersects any other hole (except by the sharing of vertices).</li> <li>has a hole that intersects with the exterior ring (except by the sharing of vertices).</li> <li>has a hole within a hole.</li> </ul> <figure> <img src="https://wellquite.org/images/polygons/polygons.svg" alt="The diagram shows two valid polygons, and three invalid polygons, illustrating the requirements just stated"> <figcaption> <h4>Figure 1: valid and invalid polygons</h4> </figcaption> </figure> Figure 1 shows two valid polygons, a and b, and three invalid polygons: <ul> <li>c is invalid because it is self-intersecting;</li> <li>d is invalid because its hole intersects with its exterior ring;</li> <li>e is invalid because its hole has a hole.</li> </ul> Note that b could be a single exterior ring, or an exterior ring with one or two interior rings. All would be valid because the only intersections possible between these different rings are at vertices. <h1 id="it-s-all-about-intersections">It’s all about intersections</h1> No matter which boolean operation you wish to perform between polygons A and B, you need to find the coördinates of the intersections between the edges of A and the edges of B. To find the coördinates of an intersection between two edges (or “line segments”), refer to page 244 (as printed on the page; page 281 of the PDF) of “Geometric Tools for Computer Graphics”. There are more results possible than you might like though: <ul> <li>0 points of intersection found: the two edges have no points in common;</li> <li>1 point of intersection found: the two edges cross over each other somewhere;</li> <li>several points of intersection found: the two edges overlap (there are several points which are <a href="https://en.wikipedia.org/wiki/Collinearity">collinear</a> to both edges). Typically, you want to know the coördinates of the two vertices which mark the ends of this overlapping region.</li> </ul> <h1 id="starting-down-the-floating-point-rabbit-hole">Starting down the floating-point rabbit hole</h1> A refrain that I’m going to keep coming back to is: <blockquote> Floats are not much different from ints, and if an algorithm isn’t correct for ints, it’s definitely not going to be correct for floats. </blockquote> By “correct”, I don’t necessarily mean “calculates the exact right number”. I more mean that from a logical point of view, it “does the right thing”. If the coördinates of the vertices of our line segments were expressed in ints and not floats, then we could draw this sort of diagram: <figure> <img src="https://wellquite.org/images/polygons/intersection_ints.svg" alt="The diagram shows two valid line segments that intersect, and illustrates how the point of intersection cannot be expressed as integer coördinates"> <figcaption> <h4>Figure 2: Intersection of two line segments with integer coördinates</h4> </figcaption> </figure> <ul> <li>The coördinates of our original two line segments are valid integer coördinates: the orange line is (20,65) to (24,60), and the blue line is (20,61) to (25,64).</li> <li>The point of intersection cannot be expressed as integer coördinates. And once you ponder this for a moment, you realise the probability of any point of intersection being expressible as integer coördinates is close to 0. So we have no choice but to move the point of intersection to the nearest coördinate that we can express with integers. I think of this as “snap to grid”.</li> <li>The result is four line segments, none of which exactly match up to the originals, but are as close as they can be. Note that this “snapping to grid” has changed the angle of the orange and blue lines. Foreshadowing for the distant future!</li> </ul> This “snapping to grid” is logically correct: it’s impossible to avoid, but it does also mean that the result you get back won’t be quite perfect, simply because we cannot represent the point of intersection exactly. Well, the situation with floating point coördinates is no different to that of integer coördinates (in fact, it’s a bit worse). <h2 id="what-is-a-floating-point-number-anyway">What is a floating point number anyway?</h2> A <a href="https://en.wikipedia.org/wiki/Double-precision_floating-point_format">64-bit float</a> consists of: <ul> <li>a sign bit. This is the 63rd bit. 0 indicates a positive number; 1 means negative.</li> <li>Then 11 bits which represent the exponent. This is a biased unsigned integer: to get the real exponent, decode these 11 bits as an unsigned int, cast to a signed int, and then subtract 1023.</li> <li>Then 52 bits which represent the fraction (or mantissa). Again, this encoded as an unsigned int.</li> </ul> <figure> <img src="https://wellquite.org/images/polygons/IEEE_754_Double_Floating_Point_Format.svg" alt="Diagram showing the bit layout of a 64-bit float."> <figcaption> <h4>Figure 3: Bit layout of a 64-bit float. <a href='https://commons.wikimedia.org/w/index.php?curid=3595583'>Diagram by Codekaizen</a>, CC BY-SA 4.0</h4> </figcaption> </figure> We put these parts together as follows: <ul> <li>If the exponent is all 0s then we have a subnormal number: (−1)sign × 2−1022 × 0.fraction</li> <li>If the exponent is all 1s (a.k.a 0x7FF), then we have either NaN (not-a-number), or infinity. We won’t worry about these any further here.</li> <li>Otherwise, we have a normal number: (−1)sign × 2(exponent − 1023) × 1.fraction</li> </ul> For the rest of this discussion when I write exponent, I mean the fully decoded and unbiased number, i.e. the result of the −1023 step. Here’s some Go code I wrote that decodes a float64. Note the fraction gets printed in binary (base-2): <pre tabindex="0" class="chroma"><code>func decodeFloat64(num float64) string { bits := math.Float64bits(num) sign := bits >> 63 // 0x7FF is 2^11 - 1 exponent := int64(((bits >> 52) & 0x7FF)) - 1023 isSubnormal := ((bits >> 52) & 0x7FF) == 0 // 0xFFFFFFFFFFFFF is 2^52 -1 fraction := bits & 0xFFFFFFFFFFFFF signStr := "+" if sign == 1 { signStr = "-" } if isSubnormal { return fmt.Sprintf("%s0.%052b * 2^%d (subnormal)\t%g", signStr, fraction, -1022, num) } else { return fmt.Sprintf("%s1.%052b * 2^%d\t%g", signStr, fraction, exponent, num) } } </code></pre> Let’s try it out: <pre tabindex="0" class="chroma"><code>func main() { for i := float64(1); i > 0; i /= 2 { fmt.Println(decodeFloat64(i)) } } </code></pre> We start with: <pre tabindex="0" class="chroma"><code>+1.0000000000000000000000000000000000000000000000000000 * 2^0 1 +1.0000000000000000000000000000000000000000000000000000 * 2^-1 0.5 +1.0000000000000000000000000000000000000000000000000000 * 2^-2 0.25 +1.0000000000000000000000000000000000000000000000000000 * 2^-3 0.125 +1.0000000000000000000000000000000000000000000000000000 * 2^-4 0.0625 </code></pre> <ul> <li>The fraction is all 0s (these are normal numbers, so there’s an implicit <code>1.</code> prefix to the fraction).</li> <li>Each halving changes only the exponent, which is just slowly decreasing.</li> </ul> The lowest exponent we can represent in these 11-bits, biased, is −1022. So let’s jump ahead to that section: <pre tabindex="0" class="chroma"><code>+1.0000000000000000000000000000000000000000000000000000 * 2^-1020 8.900295434028806e-308 +1.0000000000000000000000000000000000000000000000000000 * 2^-1021 4.450147717014403e-308 +1.0000000000000000000000000000000000000000000000000000 * 2^-1022 2.2250738585072014e-308 +0.1000000000000000000000000000000000000000000000000000 * 2^-1022 (subnormal) 1.1125369292536007e-308 +0.0100000000000000000000000000000000000000000000000000 * 2^-1022 (subnormal) 5.562684646268003e-309 +0.0010000000000000000000000000000000000000000000000000 * 2^-1022 (subnormal) 2.781342323134e-309 +0.0001000000000000000000000000000000000000000000000000 * 2^-1022 (subnormal) 1.390671161567e-309 +0.0000100000000000000000000000000000000000000000000000 * 2^-1022 (subnormal) 6.953355807835e-310 </code></pre> 2−1022 is the smallest possible normal number we can represent: the fraction is all 0s, and the exponent is as low as it can get. If we want to go even smaller, we need to start using the subnormal encoding. For subnormals, the exponent is fixed at −1022 (the exponent bits are all 0s), and now the implicit <code>1.</code> prefix of the fraction becomes a <code>0.</code> prefix, allowing us to represent even smaller numbers. So as we continue to halve the number, the exponent is staying the same, but the fraction is now changing. This sequence ends as: <pre tabindex="0" class="chroma"><code>+0.0000000000000000000000000000000000000000000000100000 * 2^-1022 (subnormal) 1.6e-322 +0.0000000000000000000000000000000000000000000000010000 * 2^-1022 (subnormal) 8e-323 +0.0000000000000000000000000000000000000000000000001000 * 2^-1022 (subnormal) 4e-323 +0.0000000000000000000000000000000000000000000000000100 * 2^-1022 (subnormal) 2e-323 +0.0000000000000000000000000000000000000000000000000010 * 2^-1022 (subnormal) 1e-323 +0.0000000000000000000000000000000000000000000000000001 * 2^-1022 (subnormal) 5e-324 </code></pre> 0.00000000000000000000000000000000000000000000000000012 × 2−1022 (a.k.a. 5e−324) is the smallest possible number greater than 0 that we can represent with a 64-bit float. No number between 0 and 5e−324 can be represented. <h2 id="thinking-about-floats">Thinking about floats</h2> What does this all mean though, and how can we think about it a bit more intuitively? For any given exponent we have all 52 bits of the fraction available to us, so 252 possible values, which are all evenly distributed. So we can think of the fraction as a simple integer count (multiple) of the smallest possible value for that exponent. For any normal float, we can find that smallest value by: (1.00000000000000000000000000000000000000000000000000012 × 2exponent) − (1.00000000000000000000000000000000000000000000000000002 × 2exponent) Or in other words, it’s 0.00000000000000000000000000000000000000000000000000012 × 2exponent Let’s imagine our exponent is 48. So that means we have 252 evenly-distributed values available to us from 248 (inclusive) to 249 (exclusive). We can think of the fraction as a simple counter of the number of smallest-values, which for this exponent is: <ul> <li>0.00000000000000000000000000000000000000000000000000012 × 248</li> <li>which is the same as 2−52 × 248</li> <li>which is the same as 2−4</li> <li>which is 0.0625</li> </ul> So for this exponent, the only values available are multiples of 0.0625. This smallest-value is known as <a href="https://en.wikipedia.org/wiki/Unit_in_the_last_place">unit in the last place</a> (ULP). Just to use a computer to double check this, here’s some Go code: <pre tabindex="0" class="chroma"><code>func main() { a := math.Pow(2,48) b := math.Nextafter(a, math.Pow(2,49)) fmt.Println(b - a) } </code></pre><pre tabindex="0" class="chroma"><code>0.0625 </code></pre> <a href="https://pkg.go.dev/math#Nextafter"><code>math.Nextafter(a, b)</code></a> finds the smallest possible value after <code>a</code>, in the direction of <code>b</code> What’s the ULP if we go up to the next exponent: 49, instead of 48? Well we’d have: 2−52 × 249 = 2−3 = 0.125 So the ULP has just doubled. I.e. the distance between neighbouring values when the exponent is 49, is twice as big, as when the exponent was 48. If our exponent is 52, then we have a ULP of 1; i.e. when the exponent is 52 a 64-bit float can only represent integers. <pre tabindex="0" class="chroma"><code>func main() { a := math.Pow(2,52) b := math.Nextafter(a, math.Pow(2,53)) fmt.Println(b == a + 1) fmt.Println(decodeFloat64(a)) fmt.Println(decodeFloat64(b)) } </code></pre><pre tabindex="0" class="chroma"><code>true +1.0000000000000000000000000000000000000000000000000000 * 2^52 4.503599627370496e+15 +1.0000000000000000000000000000000000000000000000000001 * 2^52 4.503599627370497e+15 </code></pre> So if we treat the fraction as an integer (ignoring the <code>1.</code> prefix), then we can think of a number as 2exponent + fraction × ULP-for-this-exponent For subnormals the exponent is fixed at −1022, and the <code>1.</code> prefix of the fraction is gone, so the ULP for subnormals is the same as the smallest possible non-0 value: <ul> <li>0.00000000000000000000000000000000000000000000000000012 × 2−1022</li> <li>which is the same as 2−52 × 2−1022</li> <li>which is the same as 2−1074</li> <li>a.k.a 5e−324</li> </ul> So between 0 and the first normal number (2−1022), there are once again 252 possible values, which are evenly spaced, and we can think of them as multiples of 2−1074. Interestingly, the ULP is the same for subnormals as for the first normal range: the first normal range has an exponent of −1022, it’s just the fraction now has the <code>1.</code> prefix. Putting it all together, the ULP is smallest when we’re closest to 0. Every time the exponent increases by 1, the ULP doubles. <h1 id="back-to-the-grid">Back to the grid</h1> <figure> <img src="https://wellquite.org/images/polygons/intersection_ints.svg" alt="The diagram shows two valid line segments that intersect, and illustrates how the point of intersection cannot be expressed as integer coördinates"> <figcaption> <h4>Figure 3 (again): Intersection of two line segments with integer coördinates</h4> </figcaption> </figure> Coming back to this grid, now armed with our understanding of floats, it’s suddenly apparent that the numbers on this grid could just be the integer interpretation of the fraction from floating point numbers. I.e. this is just the multiple of the ULP for whatever exponent we’re currently using. So the intuition about “snapping to grid” that made so much sense earlier when talking about integer coördinates applies to floats too. Earlier, I said things were a bit worse with floats. This is because of having to deal with the fact the grid might may not be regular. Consider: <figure> <img src="https://wellquite.org/images/polygons/intersection_floats.svg" alt="The diagram shows two valid line segments that intersect, and illustrates how the point of intersection cannot be expressed as float coördinates either"> <figcaption> <h4>Figure 4: Intersection of two line segments with float coördinates</h4> </figcaption> </figure> Here, the x-coördinates go beyond one exponent (52) and into the next (53). This means that the resolution of the grid halves (or the spacing doubles). But the y-coördinates are all within the range of a single exponent, which ends up creating a rectangular grid. Somewhat unusual (but again, when you ponder this for a moment, it’s clearly the common case: the square grid only exists when the exponents for the x- and y-coördinates are the same). Furthermore, the intersection of the same two line segments, translated around to different exponents, can be wildly different as the resolution of the grid changes. For example, if these polygons are geo-spatial data (i.e. shapes on the ground in some way), then intersecting them using different <a href="https://en.wikipedia.org/wiki/Spatial_reference_system">coördinate reference systems</a> (CRS) could result in very different shapes, as the same polygons on the ground could end up using different exponents when expressed in different CRSs. But I think the most challenging problem with non-regular grids is that it means subtraction might not work, and that means that when we calculate vectors by subtracting the coördinates of two vertices, that vector might be wrong. With all values spaced evenly apart (such as with integers), we have: a + (b − a) = b. But with a non-regular grid, if a and b have different exponents, then this may no longer hold, because it may be impossible to represent b − a. For example: <ul> <li>Let’s say a is the value just before you get to 1.0, and b is 2.0</li> <li>b − a should be a value just greater than 1.0, so we need to use the exponent for the range 1.0-to-2.0 to represent this value.</li> <li>But the exponent for the range 1.0-to-2.0 is 1 greater than for the range 0.5-to-1.0. So therefore the ULP for the range from 1.0-to-2.0 is twice the ULP for the range 0.5-to-1.0, and we know that a makes use of this finer-grained resolution, because we defined it to be the value immediately before 1.0, i.e. the greatest value possible using the 0.5-to-1.0 exponent.</li> <li>So the smallest value greater than 1.0 is too big to represent b − a.</li> </ul> <pre tabindex="0" class="chroma"><code>func main() { a := math.Nextafter(1, 0) b := float64(2) difference := b - a fmt.Println(" a =", decodeFloat64(a)) fmt.Println(" b =", decodeFloat64(b)) fmt.Println("(b-a) =", decodeFloat64(difference)) fmt.Println("a + (b-a) == b ?", a+difference == b) fmt.Println("a == b - (b-a) ?", a == b-difference) fmt.Println() a = math.Nextafter(a, 0) difference = b - a fmt.Println(" a' =", decodeFloat64(a)) fmt.Println(" b =", decodeFloat64(b)) fmt.Println("(b-a') =", decodeFloat64(difference)) fmt.Println("a' + (b-a') == b ?", a+difference == b) fmt.Println("a' == b - (b-a') ?", a == b-difference) } </code></pre><pre tabindex="0" class="chroma"><code> a = +1.1111111111111111111111111111111111111111111111111111 * 2^-1 0.9999999999999999 b = +1.0000000000000000000000000000000000000000000000000000 * 2^1 2 (b-a) = +1.0000000000000000000000000000000000000000000000000000 * 2^0 1 a + (b-a) == b ? true a == b - (b-a) ? false a' = +1.1111111111111111111111111111111111111111111111111110 * 2^-1 0.9999999999999998 b = +1.0000000000000000000000000000000000000000000000000000 * 2^1 2 (b-a') = +1.0000000000000000000000000000000000000000000000000001 * 2^0 1.0000000000000002 a' + (b-a') == b ? true a' == b - (b-a') ? true </code></pre> So we can see the computer decides that b − a is 1.0 exactly – i.e. a value that’s too small. If we define aʹ to be one value smaller than a, then we have that aʹ is on an even multiple of its ULP, which means it can be expressed correctly in the ULP of the next exponent up, so b − aʹ can be expressed exactly as a 64-bit float. Given that finding the point of intersection between two line segments will require calculating the vector of each line segment, we need to tread carefully, to say the least! <h1 id="wrapping-up">Wrapping up</h1> Hopefully I’ve illustrated that whether or not your coördinates are ints or floats, it’s unavoidable that line segment intersections will change the lines of your polygons: the intersection will always “snap to grid”. A floating point number can be thought of as 2exponent + fraction × ULP-for-this-exponent, and so within a given exponent, floats are really no different than integers. But for line segments that have coördinates that use different exponents, trouble lies ahead because we can’t expect the vector to be correct: we will likely have to do further, and coarser, “snapping to grid” if we want to be able to trust our vectors. Thinking back to <a href="https://wellquite.org/posts/lets_build/polygons_intro/">Part 1</a>, the algorithm there needed to repeatedly consider whether one polygon covered another, to union polygons together and then subtract or intersect other polygons. It now seems that we actually need to do quite a lot of preparation work for each polygon based on what it intersects with, in order to have a logically correct outcome. This suggests an algorithm that allows you to prepare many polygons at a time, before then querying and calculating boolean operations between any pair of the prepared polygons. This is not what the Martínez algorithm does: the Martínez algorithm loads only the two polygons you want to operate on, and so this is further motivation for really digging into the algorithm so we can see where it can be extended. Stay tuned for part 3! Let's build! Boolean operations of polygons: Part 1 - Introduction https://wellquite.org/posts/lets_build/polygons_intro/ Sat, 06 Jul 2024 13:00:00 +0000 https://wellquite.org/posts/lets_build/polygons_intro/ In this series: <ul> <li><a href="https://wellquite.org/posts/lets_build/polygons_intro/">Let‘s build! Boolean operations of polygons: Part 1 - Introduction</a></li> <li><a href="https://wellquite.org/posts/lets_build/polygons_intersections/">Let’s build! Boolean operations of polygons: Part 2 - Intersections</a></li> </ul> <hr> There are a small set of common boolean operations that you can perform on two polygons: union (a∪b), intersection (a∩b), subtraction (a−b and b−a), and exclusive-or (a xor b) (also called symmetric-difference). All apart from subtraction are commutative, e.g. a∪b gives the same result as b∪a. <figure> <img src="https://wellquite.org/images/polygons/operations.svg" alt="The diagram shows two overlapping polygons, a and b, and the result of their union, intersection, subtraction (in both orders), and exclusive-or (xor)."> <figcaption> <h4>Boolean operations between two overlapping polygons, a and b</h4> </figcaption> </figure> I found myself in a situation where I needed to overlay a few dozen datasets of polygons, and trace the intersecting shapes. When you’re dealing with over 100 million polygons, this becomes non-trivial, and I ended up writing my own batch processing solution, which was highly parallelised. My solution nevertheless used an existing implementation of these boolean operations from <a href="https://libgeos.org">libgeos</a>. Although not the focus of this series, it’s probably worth documenting the algorithm I came up with. The algorithm proceeds in rounds: processing each round may produce some polygons that go into the next round, and it may produce some polygons that go into the output. When you’ve processed all the polygons in the current round, you start work on the next round. Eventually you reach a point where the next round is empty, and your result is then all the polygons you’ve output. <ol> <li>For each polygon A in the current round: <ol> <li>If there is some other polygon B in the current round that completely covers A, then ignore A and move on to the next polygon in the current round.</li> <li>For all other polygons (B, C, …) in the current round that intersect with A, output A − ⋃{B, C, …}.</li> <li>In addition, for each polygon B from the current round that intersects with A, add to the next round A ∩ B.</li> </ol></li> <li>If the next round is empty, we’re done, otherwise start processing the next round after removing duplicates.</li> </ol> Let’s walk through an example: <h2 id="round-1">Round 1</h2> Here are the original polygons: <figure> <img src="https://wellquite.org/images/polygons/overlay/round_1_input.svg" alt="The original polygons are labelled A, B, C and D."> <figcaption> <h4>Round 1 input</h4> </figcaption> </figure> <ul> <li>A, B and C are all covered by D (D does not have any holes in it), so A, B and C are completely ignored.</li> <li>But for D, we: <ul> <li>output D − ⋃{A, B, C}</li> <li>add to next-round D ∩ A, D ∩ B, and D ∩ C (which is equal to the set A, B, C because the ∩ D changes nothing).</li> </ul></li> </ul> <figure> <img src="https://wellquite.org/images/polygons/overlay/round_1_results.svg" alt=""> <figcaption> <h4>Round 1 results</h4> </figcaption> </figure> <h2 id="round-2">Round 2</h2> Round 2 now starts, processing the next-round results from round 1: A, B, and C. <ul> <li>A is covered by C, so A gets completely ignored.</li> <li>For B we: <ul> <li>output B − ⋃{A, C}</li> <li>add to next-round B ∩ A and B ∩ C.</li> </ul></li> <li>For C we: <ul> <li>output C − ⋃{A, B}</li> <li>add to next-round C ∩ A (which is equal to A).</li> </ul></li> </ul> <figure> <img src="https://wellquite.org/images/polygons/overlay/round_2_results.svg" alt=""> <figcaption> <h4>Round 2 results</h4> </figcaption> </figure> <h2 id="round-3">Round 3</h2> Round 3 now starts, processing the next-round results from round 2: A, B ∩ A, and B ∩ C <ul> <li>For A, we <ul> <li>output A − ⋃{(B ∩ A), (B ∩ C)} (which is the same as A − (B ∩ A), or A − (B ∩ C))</li> <li>add to next-round A ∩ (B ∩ A), which is the same as B ∩ A.</li> <li>add to next-round A ∩ (B ∩ C), which is the same as B ∩ A.</li> </ul></li> <li>B ∩ A is covered by both A and B ∩ C, so B ∩ A gets completely ignored.</li> <li>For B ∩ C, we: <ul> <li>output (B ∩ C) − ⋃{A, (B ∩ A)} (which is the same as (B ∩ C) − A, or (B ∩ C) − (B ∩ A))</li> <li>add to next-round: <ul> <li>(B ∩ C) ∩ A, which is the same as B ∩ A</li> <li>(B ∩ C) ∩ (B ∩ A), which is the same as B ∩ A</li> </ul></li> </ul></li> <li>After removing duplicates, the next-round contains only one polygon, equal to B ∩ A.</li> </ul> <figure> <img src="https://wellquite.org/images/polygons/overlay/round_3_results.svg" alt=""> <figcaption> <h4>Round 3 results</h4> </figcaption> </figure> <h2 id="round-4">Round 4</h2> Round 4 now starts, processing the next-round results from round 3: B ∩ A <ul> <li>For B ∩ A we output (B ∩ A) − ⋃{}. I.e. (B ∩ A)</li> <li>There’s nothing to go to the next round, so we are finished.</li> </ul> The full set of output polygons is: <ul> <li>D − ⋃{A, B, C}</li> <li>B − ⋃{A, C}</li> <li>C − ⋃{A, B}</li> <li>A − (B ∩ A)</li> <li>(B ∩ C) − A</li> <li>B ∩ A</li> </ul> <h1 id="making-it-go-fast">Making it go fast</h1> In each round, for each polygon A, we need to be able to find the set of polygons that intersect with A. So that suggests using an <a href="https://en.wikipedia.org/wiki/R-tree">R-tree</a> or <a href="https://en.wikipedia.org/wiki/Quadtree">quad-tree</a> and building some sort of index of intersections. In my particular case, I had so many polygons that they couldn’t be held in RAM, so these data structures had to be designed to work off disk. As usual for me, I used my <a href="https://wellquite.org/posts/golmdb/">LMDB bindings</a> and built a quad-tree on top of that. After some effort, my quad-tree could add multiple non-intersecting polygon segments in parallel. Within each round, you can process every polygon in parallel. With a modern CPU with 16 cores or so, this makes a big difference. This is a major point of difference to other approaches I attempted such as using <a href="https://postgis.net">PostGIS</a>: in my experience, PostGIS would often start off using multiple cores, and then fall back to a single core far too quickly. Even with my custom approach and a powerful desktop machine, processing these datasets would take multiple days. PostGIS would have taken months or more. Additionally, the more I used <a href="https://libgeos.org">libgeos</a> to do the polygon intersection, union, and difference operations, the more I felt libgeos itself might be some way from optimal. I started wondering whether I could reimplement these boolean algorithms using <a href="https://www.khronos.org/opencl/">OpenCL</a> or <a href="https://developer.nvidia.com/cuda-zone">CUDA</a>. That required learning how these boolean operations actually work, and so I started down that rabbit hole. <hr> The next post in this series will start looking at a widely used algorithm for boolean operations on polygons, and will start to uncover the challenges and difficulties in this space. Golang Bebop serialization codec https://wellquite.org/posts/bebop/ Mon, 04 Dec 2023 16:01:09 +0000 https://wellquite.org/posts/bebop/ I’ve been paying some attention to serialization formats since 2014 or so. I used <a href="https://capnproto.org/">Cap’n Proto</a> when I was writing <a href="https://goshawkdb.io/">GoshawkDB</a> which dates from around 2015, and just before that I think I’d been using <a href="https://protobuf.dev/">Protobuf</a>. Two years ago, when working on <a href="https://walktogether.online/">WalkTogether</a>, was the first time I’d used <a href="https://github.com/betwixt-labs/bebop/">Bebop</a>. There are dozens of different serialization formats, and a very wide range of trade-offs to each one. I tend to favour formats where you define the schema in a separate language and file, and from there use tools to generate suitable code for different languages. I think I was very resistant to this approach when I first came across it, but without it, it means you have no single source of truth as to what the protocol or its types are, and manually updating different implementations in different languages quickly becomes a disaster. JSON has its place: it benefits from being mildly human-readable on the wire, and if you really need something vaguely self-describing then it’s probably the obvious choice. Tooling support for JSON is great, especially in browsers, though <a href="https://json-schema.org/">JSON Schema</a> is pretty awful. But JSON doesn’t scale, and the awful efficiency of the encoding is frustrating. About a year ago I found myself dealing with 100GB+ JSON files. That wasn’t a lot of fun, though I would be surprised if even the most efficient serialization format would get that below 70GB and so you’d still be writing your own tooling to deal with streaming and processing files at that size. <a href="https://github.com/betwixt-labs/bebop/">Bebop</a> brings nothing new to the table. It doesn’t say anything about memory allocation like Cap’n Proto, and it doesn’t have built-in mechanisms to deal with versioning and schema evolution (though you can build support for that yourself by using its unions). It doesn’t have <a href="https://github.com/betwixt-labs/bebop/wiki/Writing-Bops:-The-Bebop-Schema-Language">good documentation</a> – certainly nothing that I would consider acceptable as a specification. There are also some statements there that at best are questionable. For example when talking about <code>opcodes</code>, it says: <blockquote> All the compiler does is check that no opcode is used twice… </blockquote> Well that’s impossible unless you’re doing compile-the-world. But there’s no specification of how compilation should proceed. So in my implementation it’s a runtime check (via Go’s <code>init()</code> funcs) that no <code>opcode</code> is used twice. There is an existing <a href="https://github.com/200sc/bebop">Go binding</a> which I’ve used before. It’s fast and it seems to work just fine. Nevertheless, I wasn’t super keen on a few aspects: the parser is hand rolled; the code generation is done by appending strings together; and the generated code isn’t all that nice. None of these are critical issues really, but nevertheless I decided to see whether I could address these points, and <a href="https://wellquite.org/bebop">here is the result</a>. For the parser, I’m using <a href="https://github.com/mna/pigeon/tree/master">Pigeon</a> which I’ve used before. Given that upstream doesn’t have a specification of the Bebop language, I propose <a href="https://fossil.wellquite.org/repo/bebop/file?name=generator/parser.peg&ci=tip">my PEG grammar</a> as a specification (of sorts). That grammar is fairly relaxed about where you need semicolons, new lines, that sort of thing. I’m using Go’s stdlib templates to drive the code generation. In truth the code to <a href="https://fossil.wellquite.org/repo/bebop/file?name=generator/templates.go&ci=tip">drive it</a> is not super pretty: it would probably benefit from a bunch of refactoring and tidying. The generated code though is fairly nice: I don’t think it’s terribly far off the sort of code I’d write if I was rolling it by hand. There’s a <code>[]byte</code>-based APIs (<code>Marshal</code>/<code>Unmarshal</code>) which makes sense when you have something else doing framing (for example maybe you’re sending over websocket and so your transport is message oriented already, or you’re reading in from a key-value store so the value’s length is already known), and there’s an <code>io.Reader</code>/<code>io.Writer</code>-based API too for when your transport is a stream (e.g. plain TCP). Using rsync for backups https://wellquite.org/posts/backups/ Sat, 30 Sep 2023 16:01:09 +0000 https://wellquite.org/posts/backups/ Blimey, almost exactly a year since my last post. I guess that’s what happens when you have a full-time job at a start-up. I’ve run my own servers for 15 years or so. Regardless of the technologies used, servers have mutable state, and that needs backing up, ideally regularly. Whenever I have to do a lot of sysadmin work, e.g. major OS upgrades, or changing hosting provider, I always look at what backup services are provided, but they’re never super great. Even if they are offsite, accessing them can be tricky, and it seems unwise to trust that they would be available if the hosting company goes under. So I’ve basically just winged it, for roughly 15 years. And it’s been fine. A few stories in the news recently, and having a spare weekend, made me consider whether maybe I should finally try to improve this situation. I have a machine here at home which would be a suitable backup destination. What follows here is more or less what I’ve done. From a security point-of-view, if anyone gets into any of my servers, or my home machines, it’s game-over, so provided nothing I’ve done here weakens the security of any of those machines, then that’s good enough for me. Creating a backup needs to occur as <code>root</code> on the servers: it needs to access all sorts of files and directories all over the machine, regardless of their user. The question to answer is: should the backup be initiated on the backup machine, and pull data from the server; or should it be initiated on the server, and push data to the backup machine. I don’t want to add <code>setuid</code> binaries to the servers, and I don’t want to allow <code>root</code> to log-in via <code>ssh</code> on the servers. So really that means the backup must be initiated on the server, and push data out. I have <a href="https://www.wireguard.com/">WireGuard</a> connections between the backup machine and my servers, so connectivity is simple: there’s nothing fancy to do to get the servers to be able to connect to my backup machine. So, initial setup: <ol> <li>On the backup machine, create a new user, <code>backup</code>.</li> <li>Make sure <code>root</code> on the server has an ssh key-pair (<code>ssh-keygen -t ed25519</code> etc).</li> <li>Make sure the public key side of that is in the <code>authorized_keys</code> for the <code>backup</code> user on the backup machine.</li> </ol> <h2 id="authorized-keys">Authorized keys</h2> A while ago, I learnt that you can put a lot of extra stuff in <code>~/.ssh/authorized_keys</code> (<a href="https://man.openbsd.org/sshd#AUTHORIZED_KEYS_FILE_FORMAT">man sshd</a>). So for the <code>backup</code> user on the backup machine, the <code>~/.ssh/authorized_keys</code> file looks rather like this: <pre tabindex="0" class="chroma"><code>command="/home/backup/bin/checkssh.sh",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty ssh-ed25519 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA comment </code></pre> This ensures that the server cannot request various network forwarding, nor a terminal. Yes, earlier I said I’m not overly concerned with security provided I don’t make things worse, so these restrictions are something of a belt-and-braces approach. At the end of the day, if the server is compromised, then there’s probably nothing I can do to guarantee the integrity of the backups either. Now because I’m going to be using <code>rsync</code>, I want to stop the server from being able to overwrite <code>~backup/.ssh/authorized_keys</code>, as that would also be bad. So on the backup machine: <pre tabindex="0" class="chroma"><code>chown -R root:backup ~backup/.ssh chmod u=rx,g=rx,o= ~backup/.ssh chmod u=r,g=r,o= ~backup/.ssh/authorized_keys </code></pre> But, going back to the <code>authorized_keys</code> line, what’s that <code>command="/home/backup/bin/checkssh.sh"</code> bit? Well, that file started out as this: <pre tabindex="0" class="chroma"><code>#!/bin/bash if [ -n "$SSH_ORIGINAL_COMMAND" ]; then if [[ "$SSH_ORIGINAL_COMMAND" =~ ^rsync\ ]]; then echo "`/bin/date`: $SSH_ORIGINAL_COMMAND" >> $HOME/ssh-command-log exec $SSH_ORIGINAL_COMMAND else echo "`/bin/date`: DENIED $SSH_ORIGINAL_COMMAND" >> $HOME/ssh-command-log fi fi </code></pre> As the <a href="https://man.openbsd.org/sshd#AUTHORIZED_KEYS_FILE_FORMAT">man-page</a> says, <code>command="command"</code> specifies that the command is executed whenever this key is used for authentication. And also, the command originally supplied by the client is available in the <code>SSH_ORIGINAL_COMMAND</code> environment variable. So, this script is always run whenever the key is used for authentication, and we can check that the original command supplied is <code>rsync</code>. That’s a command that the server is requesting be run on the backup machine. I.e. it’s a binary under the control of the backup machine, it needs to be in the <code>PATH</code> etc. With all of this in place, from the server: <ul> <li>trying to do a plain <code>ssh backup@backup.wireguard.local</code> should fail (due to no terminal being allowed).</li> <li>doing something like <code>ssh backup@backup.wireguard.local ls</code> should fail silently because <code>ls</code> is not <code>rsync</code> (though we should have an entry in the <code>~backup/ssh-command-log</code> file now saying <code>DENIED ls</code>).</li> <li>but <code>ssh backup@backup.wireguard.local rsync -h</code> should work - you should get the help text from <code>rsync</code> back.</li> </ul> Again, I want to make sure <code>rsync</code> can’t overwrite the <code>checkssh.sh</code> script, so: <pre tabindex="0" class="chroma"><code>chown -R root:backup ~backup/bin chmod u=rx,g=rx,o= ~backup/bin chmod u=r,g=rx,o= ~backup/bin/checkssh.sh </code></pre> <h2 id="running-rsync">Running rsync</h2> With all that done, from the server I should be able to do: <pre tabindex="0" class="chroma"><code>rsync -az --stats /important/data/ backup@backup.wireguard.local:/home/backup/server/important/data/ </code></pre> It does work. However, because on the backup machine, <code>rsync</code> isn’t running as <code>root</code> (it’s running as our <code>backup</code> user), all the ownership data of all the files and directories gets lost. This is rubbish. I want to keep using <code>rsync</code>, because of its ability to do incremental backups and general efficiency. But I don’t want to lose file ownership or permissions. It’s just data after all - it can’t be that hard to get it kept! The problem is that <code>rsync</code> is using the normal file-system on the backup machine, and its ability to set arbitrary owners and groups is limited as a consequence of running as the <code>backup</code> user. So I started thinking about can I get <code>rsync</code> to write into a loop-back device? Something where the host OS isn’t going to interfere? After a bit of searching, it turns out loop-back is wrong, but a user namespace is right. <h2 id="namespaces-and-unshare">Namespaces, and unshare</h2> These days, because of the rise and rise of containers, Linux supports a lot of different namespaces. E.g. from within one <code>pid</code> namespace, you can’t see the processes of another <code>pid</code> namespace. Similarly, the users and groups of one <code>user</code> namespace are isolated from the users and groups of a different <code>user</code> namespace. As a normal, unprivileged user, I can create (and enter) a new <code>user</code> namespace. I can even be <code>root</code> in that new user namespace! Thankfully, the <code>root</code> in this new namespace is very different to the <code>root</code> in the actual host, so I can’t do terrifying damage. But it is enough to be able to set arbitrary users, groups and permissions on files that I create within this namespace. Let’s play around with this a bit. <pre tabindex="0" class="chroma"><code>backup@~/> touch foo backup@~/> ls -l foo -rw------- 1 backup backup 0 Sep 30 10:16 foo backup@~/> unshare --user --map-auto --map-root-user root@~/> ls -l foo -rw------- 1 root root 0 Sep 30 10:16 foo root@~/> touch bar root@~/> ls -l bar -rw------- 1 root root 0 Sep 30 10:17 foo root@~/> ls -l /root ls: cannot open directory '/root': Permission denied root@~/> exit backup@~/> ls -l foo bar -rw------- 1 backup backup 0 Sep 30 10:16 foo -rw------- 1 backup backup 0 Sep 30 10:17 bar </code></pre> So, I created a file as the simple <code>backup</code> user. I used the <code>unshare</code> command to create and enter a new <code>user</code> namespace, and I became <code>root</code> within it. Files that I previously owned were still owned by me; but it seems like I am <code>root</code>! If I create new files as <code>root</code>, then when I exit the namespace, they’re owned by the normal <code>backup</code> user. Thankfully, even as this new <code>root</code>, it looks like I can’t access things that are restricted to the real host <code>root</code>, like <code>/root</code>! Do I gain any new super-powers as this new <code>root</code>? <pre tabindex="0" class="chroma"><code>backup@~/> ls -l foo -rw------- 1 backup backup 0 Sep 30 10:16 foo backup@~/> chown 1111 foo chown: changing ownership of 'foo': Operation not permitted backup@~/> unshare --user --map-auto --map-root-user root@~/> ls -l foo -rw------- 1 root root 0 Sep 30 10:16 foo root@~/> chown 1111 foo root@~/> ls -l foo -rw------- 1 1111 root 0 Sep 30 10:16 foo root@~/> exit backup@~/> ls -l foo -rw------- 1 297718 backup 0 Sep 30 10:16 foo </code></pre> Woah, yes I do! So the numeric <code>1111</code> user-id inside the namespace gets mapped to some other user-id outside of the namespace (you can configure this: see the man-pages for <code>subuid</code> and <code>subgid</code>). But yes, as <code>root</code> inside my new namespace, I have permission to use arbitrary user-ids! If I go back in, it’s still all good – the mapping is stable and persistent: <pre tabindex="0" class="chroma"><code>backup@~/> ls -l foo -rw------- 1 297718 backup 0 Sep 30 10:16 foo backup@~/> unshare --user --map-auto --map-root-user root@~/> ls -l foo -rw------- 1 1111 root 0 Sep 30 10:16 foo root@~/> exit </code></pre> This is exactly what I need then. I just need to edit that <code>checkssh.sh</code> script so that the <code>rsync</code> command is run within its own user namespace: <pre tabindex="0" class="chroma"><code>#!/bin/bash if [ -n "$SSH_ORIGINAL_COMMAND" ]; then if [[ "$SSH_ORIGINAL_COMMAND" =~ ^rsync\ ]]; then echo "`/bin/date`: $SSH_ORIGINAL_COMMAND" >> $HOME/ssh-command-log exec unshare --user --map-auto --map-root-user $SSH_ORIGINAL_COMMAND else echo "`/bin/date`: DENIED $SSH_ORIGINAL_COMMAND" >> $HOME/ssh-command-log fi fi </code></pre> So now, from the server, the full <code>rsync</code> command looks like: <pre tabindex="0" class="chroma"><code>rsync -az --stats --numeric-ids --delete --chmod=o= \ -M --log-file="/home/backup/rsync-$(date --rfc-3339=ns | sed -e 's/ /_/g').log" \ /important/data/ backup@backup.wireguard.local:/home/backup/server/important/data/ </code></pre> A few other options have appeared here; the man-page for <code>rsync</code> is your best guide, but in brief: <ul> <li><code>--delete</code>: if a file exists on the backup machine that doesn’t on the server then delete it. This means that the backup should be a snapshot of the server (more or less).</li> <li><code>--chmod=o=</code>: no files on the backup machine should be accessible in any way to other. Yes, this is throwing away a little of the data I’ve just worked hard to preserve. But still, I generally don’t want any of these files globally accessible.</li> <li><code>-M --log-file="/home/backup/rsync-$(date --rfc-3339=ns | sed -e 's/ /_/g').log"</code> This creates a log file on the backup machine of what it did. Could be handy one day. Doesn’t exactly hurt.</li> </ul> All done: using good old <code>rsync</code> for its fairly efficient incremental backups, with the reasonably new namespace capabilities and <code>unshare</code>, to ensure ownership and permissions are not lost, without having to allow <code>root</code>-logins anywhere. Complexity and software engineering https://wellquite.org/posts/complexity/ Mon, 26 Sep 2022 16:01:09 +0000 https://wellquite.org/posts/complexity/ The software industry seems to fetishize complexity. It's extremely harmful, yet often appealing, and I don't know how we can avoid it. Follow the link to read the full post. Programming with async / await https://wellquite.org/posts/async_await/ Mon, 07 Mar 2022 18:35:09 +0000 https://wellquite.org/posts/async_await/ I've recently been doing some phone app development using Dart and Flutter. Although I've used languages with async/await before, the full consequences of that concurrency design had clearly never sunk in properly. It's much more subtle and complex than I'd realised: arguably the same as infinite threads on a single CPU core. Locking is most definitely still needed! Follow the link to read the full post. Let's build! A distributed, concurrent editor: Part 6 - Testing https://wellquite.org/posts/lets_build/edist_testing/ Thu, 13 Jan 2022 11:01:09 +0000 https://wellquite.org/posts/lets_build/edist_testing/ After a week off, I return to the distributed editor to write some tests for the server. How to test actors? How to write randomised and deterministic soak tests? And ultimately, how to build confidence that this system works like I claim it does. Follow the link to read the full post. Another Go binding for LMDB https://wellquite.org/posts/golmdb/ Fri, 07 Jan 2022 16:01:09 +0000 https://wellquite.org/posts/golmdb/ I spent this week writing a new high-level binding in Go for LMDB. Follow the link to read the full post. Let's build! A distributed, concurrent editor: Part 5 - Actors https://wellquite.org/posts/lets_build/edist_actors/ Thu, 30 Dec 2021 11:01:09 +0000 https://wellquite.org/posts/lets_build/edist_actors/ I have chosen to use actors to structure the server. I believe this presents a simple model to understand, in terms of concurrency and state management, whilst allowing the server to scale and make use of multiple CPU cores. This week's instalment introduces the actors model, my actors library, and explains how I have used it for the distributed editor server. Follow the link to read the full post. Let's build! A distributed, concurrent editor: Part 4 - Writing to disk https://wellquite.org/posts/lets_build/edist_disk/ Thu, 23 Dec 2021 11:01:09 +0000 https://wellquite.org/posts/lets_build/edist_disk/ Back to the fun stuff: how shall I store the document and its editing history on disk? How do I calculate what the document should look like after receiving an undo or redo message? How do I get the server to make reasonable use of its CPU and disk resources so that it can scale well? These are the questions I'm exploring this week. Follow the link to read the full post. Static Site Generator https://wellquite.org/posts/ssg/ Sun, 19 Dec 2021 16:01:09 +0000 https://wellquite.org/posts/ssg/ Hugo started to annoy me, so I wrote my own static site generator. Follow the link to read the full post. Let's build! A distributed, concurrent editor: Part 3 - Client https://wellquite.org/posts/lets_build/edist_client/ Thu, 16 Dec 2021 11:01:09 +0000 https://wellquite.org/posts/lets_build/edist_client/ This week I focus on the client: how does it send changes to the server; how does it process updates it receives from the server; how does it synchronise with the server when it first connects and then reconnects? It turns out that most of the problems the server faces, in terms of concurrent modifications to words, can also occur at the client because of concurrent changes received from the server, and received from the user typing. Follow the link to read the full post. Let's build! A distributed, concurrent editor: Part 2 - Protocols https://wellquite.org/posts/lets_build/edist_protocol/ Thu, 09 Dec 2021 11:01:09 +0000 https://wellquite.org/posts/lets_build/edist_protocol/ In Part 1 I set the scene. Alice and Bob haven't fully specified the system, but they've certainly given me some requirements: The exact behaviour of concurrent edits is left for me to define, but I'm not allowed to solve it trivially (i.e. just do nothing), and the key requirement is that all users should end up seeing the exact same document fairly quickly, once they're connected to the server; They want the editing history to be stored so that undo/redo will work even if they disconnect and reconnect; They want to be able to edit the document locally even when disconnected from the network. Some sort of synchronisation will need to happen when they reconnect to the network. Follow the link to read the full post. Let's build! A distributed, concurrent editor: Part 1 - Introduction https://wellquite.org/posts/lets_build/edist_intro/ Thu, 02 Dec 2021 11:01:09 +0000 https://wellquite.org/posts/lets_build/edist_intro/ The sun is bright and the sky's blue when I get to the office in London. As the lift doors ping open and I step out, a pair of heads near my desk swivel and look towards me. A sense of shame and embarrassment immediately forms in the pit of my stomach as I spot Alice and Bob sat waiting for me. I'd forgotten the meeting with them this morning that I'm late for. Follow the link to read the full post. NixOS, again! https://wellquite.org/posts/nixos_again/ Wed, 03 Nov 2021 20:10:56 +0000 https://wellquite.org/posts/nixos_again/ Now I promise this isn't just a NixOS advocacy site; I do have some other content planned, honest! I just wanted to give a couple of examples of the using Nix as part of your daily workflow of building stuff. Follow the link to read the full post. LaTeX, fonts, & NixOS https://wellquite.org/posts/latex_fonts_and_nixos/ Sat, 30 Oct 2021 17:54:53 +0100 https://wellquite.org/posts/latex_fonts_and_nixos/ Back in 2006, for reasons that made sense to me at the time, and make no sense to anyone at all now, I decided to buy some fonts and use them when writing my final-year thesis for university. I was writing my thesis in LaTeX. What followed was a week in which I read some pretty thorough documentation and slowly figured out how to convert fonts in normal TrueType and OpenType formats into the variety of formats that LaTeX needs, and how to install them. Follow the link to read the full post. Moving to NixOS https://wellquite.org/posts/moving-to-nixos/ Sat, 30 Oct 2021 13:20:15 +0100 https://wellquite.org/posts/moving-to-nixos/ I started using Linux as my main desktop OS in about the year 2000. I bought the Debian 2.1 "slink" CD-ROM set. At the time I also dabbled with other OSes too: Windows, BeOS, and also Minix. Over the last 20 years, through University and my career so far as a software engineer, Debian has been incredibly reliable, and has been my go-to choice whenever setting up a new machine. About five years ago I came across NixOS. Follow the link to read the full post. Pushing Back https://wellquite.org/posts/lshift/pushing_back/ Thu, 05 May 2016 12:00:00 +0000 https://wellquite.org/posts/lshift/pushing_back/ Over the last year I’ve become more and more convinced that possibly the most important feature of any queuing system is the ability to take action immediately upon enqueuing of a new item, where the action can modify the queue, and is based on state of the queue itself. Most commonly, this is referred to as back-pressure. But back-pressure can have several different forms, suited to different scenarios. Follow the link to read the full post.

Original Source | Taken Source