CARVIEW

MOTORHOMES

Select Language

HTTP/1.1 200 OK Date: Fri, 16 Jan 2026 03:27:26 GMT Server: Apache/2.4.58 (Ubuntu) Strict-Transport-Security: max-age=15768000 Last-Modified: Mon, 12 Jan 2026 12:25:46 GMT ETag: "1869c2-6482ff981034c-gzip" Accept-Ranges: bytes Vary: Accept-Encoding Content-Encoding: gzip Transfer-Encoding: chunked Content-Type: application/xml An infrequently updated site about Haskell, and sometimes other stuff 2026-01-12T12:25:45.652107798Z https://hbtvl.banquise.net/ Solving a gamebook : Lone Wolf book 05, Shadow on the Sand mega solution (ng) https://hbtvl.banquise.net/posts/2026-01-16-GBS-chapter5bis.html 2026-01-16T00:00:00Z

This is a rebuild of the previous post of the same name. I had some ideas for optimizations, so that it can finally be vizualised online, and in the process fixed some bugs, and made the choice selection a bit less restrictive. The description is almost identical:

Missing disc: if you have done all the previous books, you should have 9 disciplines, which means you lack one. This column shows the discipline that is missing. It means that the character has all the other remaining disciplines available! Note that clicking on that column will let you show a visualisation of the related solution!.
SS: did you start with the Sommerswerd or with a Sword?
SH: did you start with the Silver Helm?
+4: did you start with the Potent Strength Potion?
BA: did you start with the +4HP body armor?
EX: have you fought an Elix previously?

The the following data is available:

Win rate: what is the probability of winning this book?
Imprisoned: probability of going through a route that leads to prison.
Offer Oede: probability of offering Oede to the poor waxeler (nice!).
Limbdeath: does the route leads to Limbdeath?
Prism: probability of following a path that leads to the acquisition of the Prism.
c27 +2HP: probability of buying the +2Hp potion at section 27.
c27 LS: probability of buying the +4Hp laumspur potion at section 27.
c27 +6HP: probability of buying the +6Hp potion at section 27.
Dhorgaan: final fight against Dhorgaan (instead of the Crypt Spawn and Dark Lord).
Keep SS: probability of finishing the book WITH the Sommerswerd, as you can lose it in the end in order to get the easier fight.

Observations:

the sash is useless, there are no routes where it is acquired
handling of the jewelled mace is a bit bad, it is currently handled as a flag + checking if the player owns the mace. I would have to make sure that it is not possible to lose the mace and pick a random other mace afterwards, but it is probably good enough for now
the game is sensitive to starting money, but this book is horrible to compute, so I do not know if I am going to work on that ^^

Missing disc #states	SS	SH	+4	BA	EX	Win rate	Imprisoned	Offer Oede	Limbdeath	Prism	c27 +2HP	c27 +LS	c27 +6HP	Dhorgaan	Keep SS
SixthSense 75.13M states	yes	yes	yes	yes	yes	88.612%	-	63.000%	100.000%	100.000%	yes	yes	no	63.000%	28.910%
Camouflage 116.80M states	yes	yes	yes	yes	yes	88.612%	-	63.000%	100.000%	100.000%	yes	yes	no	63.000%	28.910%
Hunting 105.61M states	yes	yes	yes	yes	yes	88.612%	-	63.000%	100.000%	100.000%	yes	yes	no	63.000%	28.910%
Tracking 101.62M states	yes	yes	yes	yes	yes	88.612%	-	63.000%	100.000%	100.000%	yes	yes	no	63.000%	28.910%
SixthSense 75.13M states	yes	yes	yes	yes	no	88.611%	-	63.000%	100.000%	100.000%	yes	yes	no	63.000%	28.910%
Camouflage 116.80M states	yes	yes	yes	yes	no	88.611%	-	63.000%	100.000%	100.000%	yes	yes	no	63.000%	28.910%
Hunting 105.61M states	yes	yes	yes	yes	no	88.611%	-	63.000%	100.000%	100.000%	yes	yes	no	63.000%	28.910%
Tracking 101.62M states	yes	yes	yes	yes	no	88.611%	-	63.000%	100.000%	100.000%	yes	yes	no	63.000%	28.910%
SixthSense 75.15M states	yes	no	yes	yes	yes	85.888%	-	63.000%	100.000%	100.000%	yes	yes	no	63.000%	26.776%
Camouflage 116.89M states	yes	no	yes	yes	yes	85.888%	-	63.000%	100.000%	100.000%	yes	yes	no	63.000%	26.776%
Hunting 105.69M states	yes	no	yes	yes	yes	85.888%	-	63.000%	100.000%	100.000%	yes	yes	no	63.000%	26.776%
Tracking 101.66M states	yes	no	yes	yes	yes	85.888%	-	63.000%	100.000%	100.000%	yes	yes	no	63.000%	26.776%
SixthSense 75.15M states	yes	no	yes	yes	no	85.667%	-	62.837%	100.000%	100.000%	yes	yes	no	62.837%	26.776%
Camouflage 116.89M states	yes	no	yes	yes	no	85.667%	-	62.837%	100.000%	100.000%	yes	yes	no	62.837%	26.776%
Hunting 105.69M states	yes	no	yes	yes	no	85.667%	-	62.837%	100.000%	100.000%	yes	yes	no	62.837%	26.776%
Tracking 101.66M states	yes	no	yes	yes	no	85.667%	-	62.837%	100.000%	100.000%	yes	yes	no	62.837%	26.776%
SixthSense 73.09M states	yes	yes	yes	no	yes	84.812%	-	62.997%	100.000%	100.000%	yes	yes	no	62.997%	26.186%
Camouflage 113.78M states	yes	yes	yes	no	yes	84.812%	-	62.997%	100.000%	100.000%	yes	yes	no	62.997%	26.186%
Hunting 102.58M states	yes	yes	yes	no	yes	84.812%	-	62.997%	100.000%	100.000%	yes	yes	no	62.997%	26.186%
Tracking 99.61M states	yes	yes	yes	no	yes	84.812%	-	62.997%	100.000%	100.000%	yes	yes	no	62.997%	26.186%
SixthSense 73.09M states	yes	yes	yes	no	no	84.719%	-	62.928%	100.000%	100.000%	yes	yes	no	62.928%	26.186%
Camouflage 113.78M states	yes	yes	yes	no	no	84.719%	-	62.928%	100.000%	100.000%	yes	yes	no	62.928%	26.186%
Hunting 102.58M states	yes	yes	yes	no	no	84.719%	-	62.928%	100.000%	100.000%	yes	yes	no	62.928%	26.186%
Tracking 99.61M states	yes	yes	yes	no	no	84.719%	-	62.928%	100.000%	100.000%	yes	yes	no	62.928%	26.186%
MindShield 93.97M states	yes	yes	yes	yes	yes	83.852%	-	63.000%	100.000%	100.000%	no	no	no	90.000%	-
MindShield 93.97M states	yes	yes	yes	yes	no	83.851%	-	63.000%	100.000%	100.000%	no	no	no	90.000%	-
SixthSense 65.99M states	no	yes	yes	yes	yes	82.963%	-	62.430%	100.000%	100.000%	yes	yes	yes	89.186%	-
Camouflage 101.69M states	no	yes	yes	yes	yes	82.963%	-	62.430%	100.000%	100.000%	yes	yes	yes	89.186%	-
Hunting 92.07M states	no	yes	yes	yes	yes	82.963%	-	62.430%	100.000%	100.000%	yes	yes	yes	89.186%	-
Tracking 87.71M states	no	yes	yes	yes	yes	82.955%	-	62.424%	100.000%	100.000%	yes	yes	yes	89.177%	-
SixthSense 65.99M states	no	yes	yes	yes	no	81.822%	-	61.571%	100.000%	100.000%	yes	yes	yes	87.959%	-
Camouflage 101.69M states	no	yes	yes	yes	no	81.822%	-	61.571%	100.000%	100.000%	yes	yes	yes	87.959%	-
Hunting 92.07M states	no	yes	yes	yes	no	81.822%	-	61.571%	100.000%	100.000%	yes	yes	yes	87.959%	-
Tracking 87.71M states	no	yes	yes	yes	no	81.805%	-	61.559%	100.000%	100.000%	yes	yes	yes	87.941%	-
SixthSense 73.11M states	yes	no	yes	no	yes	80.672%	-	62.927%	100.000%	100.000%	yes	yes	no	62.927%	22.956%
Camouflage 113.87M states	yes	no	yes	no	yes	80.672%	-	62.927%	100.000%	100.000%	yes	yes	no	62.927%	22.956%
Hunting 102.68M states	yes	no	yes	no	yes	80.672%	-	62.927%	100.000%	100.000%	yes	yes	no	62.927%	22.956%
Tracking 99.65M states	yes	no	yes	no	yes	80.672%	-	62.927%	100.000%	100.000%	yes	yes	no	62.927%	22.956%
SixthSense 67.42M states	yes	yes	no	yes	yes	79.519%	-	63.000%	100.000%	100.000%	yes	yes	no	63.000%	23.127%
Camouflage 100.07M states	yes	yes	no	yes	yes	79.519%	-	63.000%	100.000%	100.000%	yes	yes	no	63.000%	23.127%
Hunting 94.43M states	yes	yes	no	yes	yes	79.519%	-	63.000%	100.000%	100.000%	yes	yes	no	63.000%	23.127%
Tracking 92.37M states	yes	yes	no	yes	yes	79.519%	-	63.000%	100.000%	100.000%	yes	yes	no	63.000%	23.127%
SixthSense 67.42M states	yes	yes	no	yes	no	79.519%	-	63.000%	100.000%	100.000%	yes	yes	no	63.000%	23.127%
Camouflage 100.07M states	yes	yes	no	yes	no	79.519%	-	63.000%	100.000%	100.000%	yes	yes	no	63.000%	23.127%
Hunting 94.43M states	yes	yes	no	yes	no	79.519%	-	63.000%	100.000%	100.000%	yes	yes	no	63.000%	23.127%
Tracking 92.37M states	yes	yes	no	yes	no	79.519%	-	63.000%	100.000%	100.000%	yes	yes	no	63.000%	23.127%
SixthSense 73.11M states	yes	no	yes	no	no	78.930%	-	61.568%	100.000%	100.000%	yes	yes	no	61.568%	22.956%
Camouflage 113.87M states	yes	no	yes	no	no	78.930%	-	61.568%	100.000%	100.000%	yes	yes	no	61.568%	22.956%
Hunting 102.68M states	yes	no	yes	no	no	78.930%	-	61.568%	100.000%	100.000%	yes	yes	no	61.568%	22.956%
Tracking 99.65M states	yes	no	yes	no	no	78.930%	-	61.568%	100.000%	100.000%	yes	yes	no	61.568%	22.956%
MindShield 81.91M states	no	yes	yes	yes	yes	78.560%	-	56.159%	100.000%	100.000%	yes	yes	yes	80.227%	-
MindShield 93.98M states	yes	no	yes	yes	yes	74.784%	-	63.000%	100.000%	100.000%	no	no	no	90.000%	-
SixthSense 59.09M states	no	yes	no	yes	yes	74.629%	-	56.159%	100.000%	100.000%	yes	yes	yes	80.227%	-
Camouflage 86.63M states	no	yes	no	yes	yes	74.629%	-	56.159%	100.000%	100.000%	yes	yes	yes	80.227%	-
Hunting 81.79M states	no	yes	no	yes	yes	74.629%	-	56.159%	100.000%	100.000%	yes	yes	yes	80.227%	-
MindShield 93.98M states	yes	no	yes	yes	no	74.592%	-	62.837%	100.000%	100.000%	no	no	no	89.768%	-
Tracking 79.54M states	no	yes	no	yes	yes	74.586%	-	56.127%	100.000%	100.000%	yes	yes	yes	80.181%	-
MindShield 91.93M states	yes	yes	yes	no	yes	73.663%	-	62.997%	100.000%	100.000%	no	no	no	89.996%	-
MindShield 91.93M states	yes	yes	yes	no	no	73.582%	-	62.928%	100.000%	100.000%	no	no	no	89.898%	-
SixthSense 64.08M states	no	yes	yes	no	yes	73.535%	-	60.224%	100.000%	100.000%	yes	yes	yes	86.034%	-
Camouflage 98.76M states	no	yes	yes	no	yes	73.535%	-	60.224%	100.000%	100.000%	yes	yes	yes	86.034%	-
Hunting 89.14M states	no	yes	yes	no	yes	73.535%	-	60.224%	100.000%	100.000%	yes	yes	yes	86.034%	-
Tracking 85.82M states	no	yes	yes	no	yes	73.526%	-	60.217%	100.000%	100.000%	yes	yes	yes	86.024%	-
MindOverMatter 78.91M states	yes	yes	yes	yes	yes	73.072%	20.000%	50.400%	80.000%	100.000%	yes	yes	no	56.050%	28.047%
MindOverMatter 78.91M states	yes	yes	yes	yes	no	73.071%	20.000%	50.400%	80.000%	100.000%	yes	yes	no	56.050%	28.047%
MindShield 81.91M states	no	yes	yes	yes	no	72.776%	-	61.571%	100.000%	100.000%	yes	yes	yes	87.959%	-
SixthSense 64.08M states	no	yes	yes	no	no	71.018%	-	58.162%	100.000%	100.000%	yes	yes	yes	83.089%	-
Camouflage 98.76M states	no	yes	yes	no	no	71.018%	-	58.162%	100.000%	100.000%	yes	yes	yes	83.089%	-
Hunting 89.14M states	no	yes	yes	no	no	71.018%	-	58.162%	100.000%	100.000%	yes	yes	yes	83.089%	-
Tracking 85.82M states	no	yes	yes	no	no	70.992%	-	58.141%	100.000%	100.000%	yes	yes	yes	83.058%	-
MindOverMatter 78.93M states	yes	no	yes	yes	yes	70.892%	20.000%	50.400%	80.000%	100.000%	yes	yes	no	56.050%	25.951%
SixthSense 66.40M states	yes	yes	no	no	yes	70.743%	-	62.997%	100.000%	100.000%	yes	yes	no	62.997%	19.156%
Camouflage 98.55M states	yes	yes	no	no	yes	70.743%	-	62.997%	100.000%	100.000%	yes	yes	no	62.997%	19.156%
Hunting 92.92M states	yes	yes	no	no	yes	70.743%	-	62.997%	100.000%	100.000%	yes	yes	no	62.997%	19.156%
Tracking 91.36M states	yes	yes	no	no	yes	70.743%	-	62.997%	100.000%	100.000%	yes	yes	no	62.997%	19.156%
MindOverMatter 78.93M states	yes	no	yes	yes	no	70.715%	20.000%	50.270%	80.000%	100.000%	yes	yes	no	55.920%	25.949%
SixthSense 66.40M states	yes	yes	no	no	no	70.666%	-	62.928%	100.000%	100.000%	yes	yes	no	62.928%	19.156%
Camouflage 98.55M states	yes	yes	no	no	no	70.666%	-	62.928%	100.000%	100.000%	yes	yes	no	62.928%	19.156%
Hunting 92.92M states	yes	yes	no	no	no	70.666%	-	62.928%	100.000%	100.000%	yes	yes	no	62.928%	19.156%
Tracking 91.36M states	yes	yes	no	no	no	70.666%	-	62.928%	100.000%	100.000%	yes	yes	no	62.928%	19.156%
MindOverMatter 67.36M states	no	yes	yes	yes	yes	70.267%	20.000%	49.936%	80.000%	100.000%	yes	yes	yes	81.455%	-
MindOverMatter 76.85M states	yes	yes	yes	no	yes	70.032%	20.000%	50.398%	80.000%	100.000%	yes	yes	no	56.048%	25.370%
MindOverMatter 76.85M states	yes	yes	yes	no	no	69.958%	20.000%	50.343%	80.000%	100.000%	yes	yes	no	55.993%	25.369%
MindOverMatter 67.36M states	no	yes	yes	yes	no	69.356%	20.000%	49.250%	80.000%	100.000%	yes	yes	yes	80.475%	-
MindOverMatter 76.88M states	yes	no	yes	no	yes	66.720%	20.000%	50.342%	80.000%	100.000%	yes	yes	no	55.992%	22.206%
MindShield 74.14M states	no	yes	no	yes	yes	66.379%	-	56.159%	100.000%	100.000%	yes	yes	yes	80.227%	-
MindOverMatter 70.83M states	yes	yes	no	yes	yes	65.797%	20.000%	50.400%	80.000%	100.000%	yes	yes	no	56.050%	22.360%
MindOverMatter 70.83M states	yes	yes	no	yes	no	65.797%	20.000%	50.400%	80.000%	100.000%	yes	yes	no	56.050%	22.360%
MindShield 80.01M states	no	yes	yes	no	yes	65.386%	-	49.580%	100.000%	100.000%	yes	yes	yes	70.829%	-
MindOverMatter 76.88M states	yes	no	yes	no	no	65.326%	20.000%	49.255%	80.000%	100.000%	yes	yes	no	54.904%	22.189%
MindOverMatter 60.24M states	no	yes	no	yes	yes	63.605%	20.000%	44.922%	80.000%	100.000%	yes	yes	yes	74.293%	-
MindShield 91.96M states	yes	no	yes	no	yes	63.286%	-	62.928%	100.000%	100.000%	yes	yes	yes	89.897%	-
MindOverMatter 65.42M states	no	yes	yes	no	yes	62.711%	20.000%	48.158%	80.000%	100.000%	yes	yes	yes	78.916%	-
MindShield 91.96M states	yes	no	yes	no	no	61.919%	-	61.569%	100.000%	100.000%	yes	yes	yes	87.955%	-
MindShield 85.31M states	yes	yes	no	yes	yes	61.290%	-	63.000%	100.000%	100.000%	no	no	no	90.000%	-
MindShield 85.31M states	yes	yes	no	yes	no	61.289%	-	63.000%	100.000%	100.000%	no	no	no	90.000%	-
MindOverMatter 65.42M states	no	yes	yes	no	no	60.691%	20.000%	46.504%	80.000%	100.000%	yes	yes	yes	76.552%	-
SixthSense 58.13M states	no	yes	no	no	yes	60.539%	-	49.580%	100.000%	100.000%	yes	yes	yes	70.829%	-
Camouflage 85.16M states	no	yes	no	no	yes	60.539%	-	49.580%	100.000%	100.000%	yes	yes	yes	70.829%	-
Hunting 80.32M states	no	yes	no	no	yes	60.539%	-	49.580%	100.000%	100.000%	yes	yes	yes	70.829%	-
Tracking 78.60M states	no	yes	no	no	yes	60.478%	-	49.530%	100.000%	100.000%	yes	yes	yes	70.758%	-
SixthSense 65.98M states	no	no	yes	yes	yes	60.310%	-	58.785%	100.000%	100.000%	yes	yes	yes	83.978%	-
Hunting 92.04M states	no	no	yes	yes	yes	60.310%	-	58.785%	100.000%	100.000%	yes	yes	yes	83.978%	-
Camouflage 101.66M states	no	no	yes	yes	yes	60.278%	-	58.754%	100.000%	100.000%	yes	yes	yes	83.934%	-
Tracking 87.70M states	no	no	yes	yes	yes	60.259%	-	58.735%	100.000%	100.000%	yes	yes	yes	83.907%	-
MindOverMatter 69.80M states	yes	yes	no	no	yes	58.777%	20.000%	50.398%	80.000%	100.000%	yes	yes	no	56.048%	18.445%
MindOverMatter 69.80M states	yes	yes	no	no	no	58.715%	20.000%	50.343%	80.000%	100.000%	yes	yes	no	55.993%	18.444%
SixthSense 67.42M states	yes	no	no	yes	yes	57.972%	-	63.000%	100.000%	100.000%	yes	yes	no	63.000%	19.056%
Camouflage 100.11M states	yes	no	no	yes	yes	57.972%	-	63.000%	100.000%	100.000%	yes	yes	no	63.000%	19.056%
Hunting 94.47M states	yes	no	no	yes	yes	57.972%	-	63.000%	100.000%	100.000%	yes	yes	no	63.000%	19.056%
Tracking 92.38M states	yes	no	no	yes	yes	57.972%	-	63.000%	100.000%	100.000%	yes	yes	no	63.000%	19.056%
SixthSense 59.09M states	no	yes	no	yes	no	57.870%	-	43.548%	100.000%	100.000%	yes	yes	yes	62.211%	-
Camouflage 86.63M states	no	yes	no	yes	no	57.870%	-	43.548%	100.000%	100.000%	yes	yes	yes	62.211%	-
Hunting 81.79M states	no	yes	no	yes	no	57.870%	-	43.548%	100.000%	100.000%	yes	yes	yes	62.211%	-
SixthSense 67.42M states	yes	no	no	yes	no	57.822%	-	62.837%	100.000%	100.000%	yes	yes	no	62.837%	19.056%
Camouflage 100.11M states	yes	no	no	yes	no	57.822%	-	62.837%	100.000%	100.000%	yes	yes	no	62.837%	19.056%
Hunting 94.47M states	yes	no	no	yes	no	57.822%	-	62.837%	100.000%	100.000%	yes	yes	no	62.837%	19.056%
Tracking 92.38M states	yes	no	no	yes	no	57.822%	-	62.837%	100.000%	100.000%	yes	yes	no	62.837%	19.056%
Tracking 79.54M states	no	yes	no	yes	no	57.812%	-	43.504%	100.000%	100.000%	yes	yes	yes	62.149%	-
MindShield 81.90M states	no	no	yes	yes	yes	55.936%	-	41.246%	100.000%	100.000%	yes	yes	yes	58.923%	-
MindShield 80.01M states	no	yes	yes	no	no	55.651%	-	58.162%	100.000%	100.000%	yes	yes	yes	83.089%	-
SixthSense 65.98M states	no	no	yes	yes	no	54.884%	-	53.495%	100.000%	100.000%	yes	yes	yes	76.422%	-
Hunting 92.04M states	no	no	yes	yes	no	54.884%	-	53.495%	100.000%	100.000%	yes	yes	yes	76.422%	-
Camouflage 101.66M states	no	no	yes	yes	no	54.795%	-	53.409%	100.000%	100.000%	yes	yes	yes	76.299%	-
Tracking 87.70M states	no	no	yes	yes	no	54.758%	-	53.373%	100.000%	100.000%	yes	yes	yes	76.247%	-
MindOverMatter 59.27M states	no	yes	no	no	yes	52.295%	20.000%	39.628%	80.000%	100.000%	yes	yes	yes	66.729%	-
MindOverMatter 67.34M states	no	no	yes	yes	yes	52.089%	20.000%	46.962%	80.000%	100.000%	yes	yes	yes	77.207%	-
MindShield 74.14M states	no	yes	no	yes	no	51.473%	-	43.548%	100.000%	100.000%	yes	yes	yes	62.211%	-
MindOverMatter 60.24M states	no	yes	no	yes	no	50.199%	20.000%	34.835%	80.000%	100.000%	yes	yes	yes	59.882%	-
MindShield 84.29M states	yes	yes	no	no	yes	48.627%	-	62.997%	100.000%	100.000%	no	no	no	89.996%	-
MindShield 84.29M states	yes	yes	no	no	no	48.573%	-	62.928%	100.000%	100.000%	no	no	no	89.898%	-
MindOverMatter 70.84M states	yes	no	no	yes	yes	48.559%	20.000%	50.400%	80.000%	100.000%	yes	yes	no	56.050%	18.200%
MindOverMatter 70.84M states	yes	no	no	yes	no	48.440%	20.000%	50.270%	80.000%	100.000%	yes	yes	no	55.920%	18.197%
MindOverMatter 67.34M states	no	no	yes	yes	no	47.699%	20.000%	42.684%	80.000%	100.000%	yes	yes	yes	71.095%	-
MindShield 73.19M states	no	yes	no	no	yes	47.440%	-	49.580%	100.000%	100.000%	yes	yes	yes	70.829%	-
SixthSense 64.07M states	no	no	yes	no	yes	43.718%	-	31.930%	100.000%	100.000%	yes	yes	yes	45.614%	-
Camouflage 98.73M states	no	no	yes	no	yes	43.718%	-	31.930%	100.000%	100.000%	yes	yes	yes	45.614%	-
Hunting 89.11M states	no	no	yes	no	yes	43.718%	-	31.930%	100.000%	100.000%	yes	yes	yes	45.614%	-
Tracking 85.81M states	no	no	yes	no	yes	43.459%	-	31.741%	100.000%	100.000%	yes	yes	yes	45.344%	-
SixthSense 58.13M states	no	yes	no	no	no	43.440%	-	35.577%	100.000%	100.000%	yes	yes	yes	50.824%	-
Camouflage 85.16M states	no	yes	no	no	no	43.440%	-	35.577%	100.000%	100.000%	yes	yes	yes	50.824%	-
Hunting 80.32M states	no	yes	no	no	no	43.440%	-	35.577%	100.000%	100.000%	yes	yes	yes	50.824%	-
Tracking 78.60M states	no	yes	no	no	no	43.378%	-	35.526%	100.000%	100.000%	yes	yes	yes	50.751%	-
SixthSense 59.07M states	no	no	no	yes	yes	42.459%	-	41.385%	100.000%	100.000%	yes	yes	yes	59.122%	-
Hunting 81.75M states	no	no	no	yes	yes	42.459%	-	41.385%	100.000%	100.000%	yes	yes	yes	59.122%	-
Camouflage 86.59M states	no	no	no	yes	yes	42.316%	-	41.246%	100.000%	100.000%	yes	yes	yes	58.923%	-
Tracking 79.52M states	no	no	no	yes	yes	42.295%	-	41.225%	100.000%	100.000%	yes	yes	yes	58.893%	-
MindShield 80.00M states	no	no	yes	no	yes	39.158%	-	31.930%	100.000%	100.000%	yes	yes	yes	45.614%	-
SixthSense 66.41M states	yes	no	no	no	yes	39.082%	-	62.927%	100.000%	100.000%	yes	yes	no	62.927%	15.159%
Camouflage 98.60M states	yes	no	no	no	yes	39.082%	-	62.927%	100.000%	100.000%	yes	yes	no	62.927%	15.159%
Hunting 92.97M states	yes	no	no	no	yes	39.082%	-	62.927%	100.000%	100.000%	yes	yes	no	62.927%	15.159%
Tracking 91.39M states	yes	no	no	no	yes	39.082%	-	62.927%	100.000%	100.000%	yes	yes	no	62.927%	15.159%
MindOverMatter 59.27M states	no	yes	no	no	no	38.621%	20.000%	28.429%	80.000%	100.000%	yes	yes	yes	50.731%	-
MindOverMatter 65.41M states	no	no	yes	no	yes	38.549%	20.000%	25.171%	80.000%	100.000%	yes	yes	yes	46.280%	-
SixthSense 66.41M states	yes	no	no	no	no	38.238%	-	61.568%	100.000%	100.000%	yes	yes	no	61.568%	15.159%
Camouflage 98.60M states	yes	no	no	no	no	38.238%	-	61.568%	100.000%	100.000%	yes	yes	no	61.568%	15.159%
Hunting 92.97M states	yes	no	no	no	no	38.238%	-	61.568%	100.000%	100.000%	yes	yes	no	61.568%	15.159%
Tracking 91.39M states	yes	no	no	no	no	38.238%	-	61.568%	100.000%	100.000%	yes	yes	no	61.568%	15.159%
MindShield 81.90M states	no	no	yes	yes	no	37.912%	-	53.409%	100.000%	100.000%	yes	yes	yes	76.299%	-
MindOverMatter 60.22M states	no	no	no	yes	yes	37.719%	20.000%	32.956%	80.000%	100.000%	yes	yes	yes	57.198%	-
SixthSense 64.07M states	no	no	yes	no	no	35.015%	-	44.542%	100.000%	100.000%	yes	yes	yes	63.632%	-
Camouflage 98.73M states	no	no	yes	no	no	35.015%	-	44.542%	100.000%	100.000%	yes	yes	yes	63.632%	-
Hunting 89.11M states	no	no	yes	no	no	35.015%	-	44.542%	100.000%	100.000%	yes	yes	yes	63.632%	-
Tracking 85.81M states	no	no	yes	no	no	34.877%	-	44.367%	100.000%	100.000%	yes	yes	yes	63.382%	-
MindShield 73.19M states	no	yes	no	no	no	34.041%	-	35.577%	100.000%	100.000%	yes	yes	yes	50.824%	-
MindShield 85.31M states	yes	no	no	yes	yes	33.578%	-	63.000%	100.000%	100.000%	no	no	no	90.000%	-
MindShield 85.31M states	yes	no	no	yes	no	33.491%	-	62.837%	100.000%	100.000%	no	no	no	89.768%	-
MindOverMatter 69.81M states	yes	no	no	no	yes	33.448%	20.000%	50.342%	80.000%	100.000%	yes	yes	no	55.992%	14.170%
MindOverMatter 69.81M states	yes	no	no	no	no	32.772%	20.000%	49.255%	80.000%	100.000%	yes	yes	no	54.904%	14.150%
MindOverMatter 65.41M states	no	no	yes	no	no	31.697%	20.000%	35.351%	80.000%	100.000%	yes	yes	yes	60.619%	-
MindShield 74.13M states	no	no	no	yes	yes	29.278%	-	41.246%	100.000%	100.000%	yes	yes	yes	58.923%	-
SixthSense 59.07M states	no	no	no	yes	no	25.509%	-	24.863%	100.000%	100.000%	yes	yes	yes	35.519%	-
Hunting 81.75M states	no	no	no	yes	no	25.509%	-	24.863%	100.000%	100.000%	yes	yes	yes	35.519%	-
Tracking 79.52M states	no	no	no	yes	no	25.363%	-	24.721%	100.000%	100.000%	yes	yes	yes	35.316%	-
Camouflage 86.59M states	no	no	no	yes	no	25.344%	-	24.702%	100.000%	100.000%	yes	yes	yes	35.289%	-
SixthSense 58.12M states	no	no	no	no	yes	25.100%	-	31.930%	100.000%	100.000%	yes	yes	yes	45.614%	-
Camouflage 85.13M states	no	no	no	no	yes	25.100%	-	31.930%	100.000%	100.000%	yes	yes	yes	45.614%	-
Hunting 80.29M states	no	no	no	no	yes	25.100%	-	31.930%	100.000%	100.000%	yes	yes	yes	45.614%	-
Tracking 78.58M states	no	no	no	no	yes	24.952%	-	31.741%	100.000%	100.000%	yes	yes	yes	45.344%	-
MindOverMatter 60.22M states	no	no	no	yes	no	24.163%	20.000%	19.742%	80.000%	100.000%	yes	yes	yes	38.321%	-
MindOverMatter 59.25M states	no	no	no	no	yes	23.792%	20.000%	25.295%	80.000%	100.000%	yes	yes	yes	46.253%	-
Hunting 80.29M states	no	no	no	no	no	22.146%	100.000%	-	-	100.000%	no	no	no	57.337%	-
Tracking 78.58M states	no	no	no	no	no	22.146%	100.000%	-	-	100.000%	no	no	no	57.337%	-
SixthSense 58.12M states	no	no	no	no	no	22.121%	100.000%	-	-	100.000%	no	no	no	57.273%	-
MindShield 80.00M states	no	no	yes	no	no	21.526%	-	44.542%	100.000%	100.000%	yes	yes	yes	63.632%	-
MindOverMatter 59.25M states	no	no	no	no	no	19.541%	100.000%	-	-	100.000%	no	no	no	50.591%	-
MindShield 74.13M states	no	no	no	yes	no	17.535%	-	24.702%	100.000%	100.000%	yes	yes	yes	35.289%	-
MindShield 84.30M states	yes	no	no	no	yes	17.326%	-	62.928%	100.000%	100.000%	yes	yes	yes	89.897%	-
Camouflage 85.13M states	no	no	no	no	no	16.953%	100.000%	-	-	100.000%	no	no	no	45.965%	-
MindShield 84.30M states	yes	no	no	no	no	16.952%	-	61.569%	100.000%	100.000%	yes	yes	yes	87.955%	-
MindShield 73.18M states	no	no	no	no	yes	15.431%	-	31.930%	100.000%	100.000%	yes	yes	yes	45.614%	-
MindShield 73.18M states	no	no	no	no	no	7.683%	-	15.898%	100.000%	100.000%	yes	yes	yes	22.711%	-

Missing disc #states	Dagger	Spear	Mace	Sword	Quarterstaff	Sommerswerd	Backpack	Shield	BodyArmor	Potion2Hp	Potion4Hp	Potion6Hp	Blowpipe and Sleep Dart	Copper Key	Gaoler's Keys	Prism	Silver Helmet	Oede herb	Meal	Gold	Laumspur	FoughtElix	Scroll	Jewelled Mace	HadCombat
SixthSense 75.13M states	0.00	0.00	0.71	0.00	0.29	0.29	1.00	1.00	1.00	0.81	0.02	0.00	0.00	1.00	0.00	1.00	1.00	0.00	0.00	2.00	1.59	100.000%	-	71.090%	28.910%
Camouflage 116.80M states	0.00	0.00	0.71	0.00	0.29	0.29	1.00	1.00	1.00	0.81	0.02	0.00	0.00	1.00	0.00	1.00	1.00	0.00	0.00	2.00	1.59	100.000%	-	71.090%	28.910%
Hunting 105.61M states	0.00	0.00	0.71	0.00	0.29	0.29	1.00	1.00	1.00	0.81	0.02	0.00	0.00	1.00	0.00	1.00	1.00	0.00	0.00	2.00	1.59	100.000%	-	71.090%	28.910%
Tracking 101.62M states	0.00	0.00	0.71	0.00	0.29	0.29	1.00	1.00	1.00	0.81	0.02	0.00	0.00	1.00	0.00	1.00	1.00	0.00	0.00	2.00	1.59	100.000%	-	71.090%	28.910%
SixthSense 75.13M states	0.00	0.00	0.71	0.00	0.29	0.29	1.00	1.00	1.00	0.81	0.02	0.00	0.00	1.00	0.00	1.00	1.00	0.00	0.00	2.00	1.59	-	-	71.090%	28.910%
Camouflage 116.80M states	0.00	0.00	0.71	0.00	0.29	0.29	1.00	1.00	1.00	0.81	0.02	0.00	0.00	1.00	0.00	1.00	1.00	0.00	0.00	2.00	1.59	-	-	71.090%	28.910%
Hunting 105.61M states	0.00	0.00	0.71	0.00	0.29	0.29	1.00	1.00	1.00	0.81	0.02	0.00	0.00	1.00	0.00	1.00	1.00	0.00	0.00	2.00	1.59	-	-	71.090%	28.910%
Tracking 101.62M states	0.00	0.00	0.71	0.00	0.29	0.29	1.00	1.00	1.00	0.81	0.02	0.00	0.00	1.00	0.00	1.00	1.00	0.00	0.00	2.00	1.59	-	-	71.090%	28.910%
SixthSense 75.15M states	0.00	0.00	0.73	0.00	0.27	0.27	1.00	1.00	1.00	0.80	0.01	0.00	0.00	1.00	0.00	1.00	0.00	0.00	0.00	2.00	1.57	100.000%	-	73.224%	26.776%
Camouflage 116.89M states	0.00	0.00	0.73	0.00	0.27	0.27	1.00	1.00	1.00	0.80	0.01	0.00	0.00	1.00	0.00	1.00	0.00	0.00	0.00	2.00	1.57	100.000%	-	73.224%	26.776%
Hunting 105.69M states	0.00	0.00	0.73	0.00	0.27	0.27	1.00	1.00	1.00	0.80	0.01	0.00	0.00	1.00	0.00	1.00	0.00	0.00	0.00	2.00	1.57	100.000%	-	73.224%	26.776%
Tracking 101.66M states	0.00	0.00	0.73	0.00	0.27	0.27	1.00	1.00	1.00	0.80	0.01	0.00	0.00	1.00	0.00	1.00	0.00	0.00	0.00	2.00	1.57	100.000%	-	73.224%	26.776%
SixthSense 75.15M states	0.00	0.00	0.73	0.00	0.27	0.27	1.00	1.00	1.00	0.80	0.01	0.00	0.00	1.00	0.00	1.00	0.00	0.00	0.00	2.00	1.57	-	-	73.224%	26.776%
Camouflage 116.89M states	0.00	0.00	0.73	0.00	0.27	0.27	1.00	1.00	1.00	0.80	0.01	0.00	0.00	1.00	0.00	1.00	0.00	0.00	0.00	2.00	1.57	-	-	73.224%	26.776%
Hunting 105.69M states	0.00	0.00	0.73	0.00	0.27	0.27	1.00	1.00	1.00	0.80	0.01	0.00	0.00	1.00	0.00	1.00	0.00	0.00	0.00	2.00	1.57	-	-	73.224%	26.776%
Tracking 101.66M states	0.00	0.00	0.73	0.00	0.27	0.27	1.00	1.00	1.00	0.80	0.01	0.00	0.00	1.00	0.00	1.00	0.00	0.00	0.00	2.00	1.57	-	-	73.224%	26.776%
SixthSense 73.09M states	0.00	0.00	0.74	0.00	0.26	0.26	1.00	1.00	0.00	0.84	0.02	0.00	0.00	1.00	0.00	1.00	1.00	0.00	0.00	2.00	1.65	100.000%	-	73.814%	26.186%
Camouflage 113.78M states	0.00	0.00	0.74	0.00	0.26	0.26	1.00	1.00	0.00	0.84	0.02	0.00	0.00	1.00	0.00	1.00	1.00	0.00	0.00	2.00	1.65	100.000%	-	73.814%	26.186%
Hunting 102.58M states	0.00	0.00	0.74	0.00	0.26	0.26	1.00	1.00	0.00	0.84	0.02	0.00	0.00	1.00	0.00	1.00	1.00	0.00	0.00	2.00	1.65	100.000%	-	73.814%	26.186%
Tracking 99.61M states	0.00	0.00	0.74	0.00	0.26	0.26	1.00	1.00	0.00	0.84	0.02	0.00	0.00	1.00	0.00	1.00	1.00	0.00	0.00	2.00	1.65	100.000%	-	73.814%	26.186%
SixthSense 73.09M states	0.00	0.00	0.74	0.00	0.26	0.26	1.00	1.00	0.00	0.84	0.02	0.00	0.00	1.00	0.00	1.00	1.00	0.00	0.00	2.00	1.65	-	-	73.814%	26.186%
Camouflage 113.78M states	0.00	0.00	0.74	0.00	0.26	0.26	1.00	1.00	0.00	0.84	0.02	0.00	0.00	1.00	0.00	1.00	1.00	0.00	0.00	2.00	1.65	-	-	73.814%	26.186%
Hunting 102.58M states	0.00	0.00	0.74	0.00	0.26	0.26	1.00	1.00	0.00	0.84	0.02	0.00	0.00	1.00	0.00	1.00	1.00	0.00	0.00	2.00	1.65	-	-	73.814%	26.186%
Tracking 99.61M states	0.00	0.00	0.74	0.00	0.26	0.26	1.00	1.00	0.00	0.84	0.02	0.00	0.00	1.00	0.00	1.00	1.00	0.00	0.00	2.00	1.65	-	-	73.814%	26.186%
MindShield 93.97M states	0.00	0.00	0.74	0.00	0.26	0.00	1.00	1.00	1.00	0.00	0.00	0.00	0.00	1.00	0.00	1.00	1.00	0.26	0.00	10.00	0.64	100.000%	-	74.413%	-
MindShield 93.97M states	0.00	0.00	0.74	0.00	0.26	0.00	1.00	1.00	1.00	0.00	0.00	0.00	0.00	1.00	0.00	1.00	1.00	0.26	0.00	10.00	0.64	-	-	74.413%	-
SixthSense 65.99M states	0.00	0.00	0.74	1.00	0.26	0.00	1.00	1.00	1.00	0.65	0.00	0.70	0.00	1.00	0.00	1.00	1.00	0.26	0.00	0.00	1.61	100.000%	-	74.111%	-
Camouflage 101.69M states	0.00	0.00	0.74	1.00	0.26	0.00	1.00	1.00	1.00	0.65	0.00	0.70	0.00	1.00	0.00	1.00	1.00	0.26	0.00	0.00	1.61	100.000%	-	74.111%	-
Hunting 92.07M states	0.00	0.00	0.74	1.00	0.26	0.00	1.00	1.00	1.00	0.65	0.00	0.70	0.00	1.00	0.00	1.00	1.00	0.26	0.00	0.00	1.61	100.000%	-	74.111%	-
Tracking 87.71M states	0.00	0.00	0.74	1.00	0.26	0.00	1.00	1.00	1.00	0.65	0.00	0.69	0.00	1.00	0.00	1.00	1.00	0.26	0.00	0.00	1.61	100.000%	-	74.111%	-
SixthSense 65.99M states	0.00	0.00	0.74	1.00	0.26	0.00	1.00	1.00	1.00	0.64	0.00	0.69	0.00	1.00	0.00	1.00	1.00	0.26	0.00	0.00	1.60	-	-	74.111%	-
Camouflage 101.69M states	0.00	0.00	0.74	1.00	0.26	0.00	1.00	1.00	1.00	0.64	0.00	0.69	0.00	1.00	0.00	1.00	1.00	0.26	0.00	0.00	1.60	-	-	74.111%	-
Hunting 92.07M states	0.00	0.00	0.74	1.00	0.26	0.00	1.00	1.00	1.00	0.64	0.00	0.69	0.00	1.00	0.00	1.00	1.00	0.26	0.00	0.00	1.60	-	-	74.111%	-
Tracking 87.71M states	0.00	0.00	0.74	1.00	0.26	0.00	1.00	1.00	1.00	0.64	0.00	0.68	0.00	1.00	0.00	1.00	1.00	0.26	0.00	0.00	1.59	-	-	74.111%	-
SixthSense 73.11M states	0.00	0.00	0.77	0.00	0.23	0.23	1.00	1.00	0.00	0.84	0.01	0.00	0.00	1.00	0.00	1.00	0.00	0.00	0.00	2.00	1.65	100.000%	-	77.044%	22.956%
Camouflage 113.87M states	0.00	0.00	0.77	0.00	0.23	0.23	1.00	1.00	0.00	0.84	0.01	0.00	0.00	1.00	0.00	1.00	0.00	0.00	0.00	2.00	1.65	100.000%	-	77.044%	22.956%
Hunting 102.68M states	0.00	0.00	0.77	0.00	0.23	0.23	1.00	1.00	0.00	0.84	0.01	0.00	0.00	1.00	0.00	1.00	0.00	0.00	0.00	2.00	1.65	100.000%	-	77.044%	22.956%
Tracking 99.65M states	0.00	0.00	0.77	0.00	0.23	0.23	1.00	1.00	0.00	0.84	0.01	0.00	0.00	1.00	0.00	1.00	0.00	0.00	0.00	2.00	1.65	100.000%	-	77.044%	22.956%
SixthSense 67.42M states	0.00	0.00	0.77	0.00	0.23	0.23	1.00	1.00	1.00	0.81	0.00	0.00	0.00	1.00	0.00	1.00	1.00	0.00	0.00	2.00	1.59	100.000%	-	76.873%	23.127%
Camouflage 100.07M states	0.00	0.00	0.77	0.00	0.23	0.23	1.00	1.00	1.00	0.81	0.00	0.00	0.00	1.00	0.00	1.00	1.00	0.00	0.00	2.00	1.59	100.000%	-	76.873%	23.127%
Hunting 94.43M states	0.00	0.00	0.77	0.00	0.23	0.23	1.00	1.00	1.00	0.81	0.00	0.00	0.00	1.00	0.00	1.00	1.00	0.00	0.00	2.00	1.59	100.000%	-	76.873%	23.127%
Tracking 92.37M states	0.00	0.00	0.77	0.00	0.23	0.23	1.00	1.00	1.00	0.81	0.00	0.00	0.00	1.00	0.00	1.00	1.00	0.00	0.00	2.00	1.59	100.000%	-	76.873%	23.127%
SixthSense 67.42M states	0.00	0.00	0.77	0.00	0.23	0.23	1.00	1.00	1.00	0.81	0.00	0.00	0.00	1.00	0.00	1.00	1.00	0.00	0.00	2.00	1.59	-	-	76.873%	23.127%
Camouflage 100.07M states	0.00	0.00	0.77	0.00	0.23	0.23	1.00	1.00	1.00	0.81	0.00	0.00	0.00	1.00	0.00	1.00	1.00	0.00	0.00	2.00	1.59	-	-	76.873%	23.127%
Hunting 94.43M states	0.00	0.00	0.77	0.00	0.23	0.23	1.00	1.00	1.00	0.81	0.00	0.00	0.00	1.00	0.00	1.00	1.00	0.00	0.00	2.00	1.59	-	-	76.873%	23.127%
Tracking 92.37M states	0.00	0.00	0.77	0.00	0.23	0.23	1.00	1.00	1.00	0.81	0.00	0.00	0.00	1.00	0.00	1.00	1.00	0.00	0.00	2.00	1.59	-	-	76.873%	23.127%
SixthSense 73.11M states	0.00	0.00	0.77	0.00	0.23	0.23	1.00	1.00	0.00	0.84	0.01	0.00	0.00	1.00	0.00	1.00	0.00	0.00	0.00	2.00	1.65	-	-	77.044%	22.956%
Camouflage 113.87M states	0.00	0.00	0.77	0.00	0.23	0.23	1.00	1.00	0.00	0.84	0.01	0.00	0.00	1.00	0.00	1.00	0.00	0.00	0.00	2.00	1.65	-	-	77.044%	22.956%
Hunting 102.68M states	0.00	0.00	0.77	0.00	0.23	0.23	1.00	1.00	0.00	0.84	0.01	0.00	0.00	1.00	0.00	1.00	0.00	0.00	0.00	2.00	1.65	-	-	77.044%	22.956%
Tracking 99.65M states	0.00	0.00	0.77	0.00	0.23	0.23	1.00	1.00	0.00	0.84	0.01	0.00	0.00	1.00	0.00	1.00	0.00	0.00	0.00	2.00	1.65	-	-	77.044%	22.956%
MindShield 81.91M states	0.00	0.00	0.71	1.00	0.29	0.00	1.00	1.00	1.00	0.63	0.00	0.65	0.00	1.00	0.00	1.00	1.00	0.29	0.00	0.00	1.57	100.000%	-	71.256%	-
MindShield 93.98M states	0.00	0.00	0.82	0.00	0.18	0.00	1.00	1.00	1.00	0.00	0.00	0.00	0.00	1.00	0.00	1.00	0.00	0.18	0.00	10.00	0.60	100.000%	-	82.082%	-
SixthSense 59.09M states	0.00	0.00	0.74	1.00	0.26	0.00	1.00	1.00	1.00	0.63	0.00	0.65	0.00	1.00	0.00	1.00	1.00	0.26	0.00	0.00	1.57	100.000%	-	74.111%	-
Camouflage 86.63M states	0.00	0.00	0.74	1.00	0.26	0.00	1.00	1.00	1.00	0.63	0.00	0.65	0.00	1.00	0.00	1.00	1.00	0.26	0.00	0.00	1.57	100.000%	-	74.111%	-
Hunting 81.79M states	0.00	0.00	0.74	1.00	0.26	0.00	1.00	1.00	1.00	0.63	0.00	0.65	0.00	1.00	0.00	1.00	1.00	0.26	0.00	0.00	1.57	100.000%	-	74.111%	-
MindShield 93.98M states	0.00	0.00	0.82	0.00	0.18	0.00	1.00	1.00	1.00	0.00	0.00	0.00	0.00	1.00	0.00	1.00	0.00	0.18	0.00	10.00	0.60	-	-	82.082%	-
Tracking 79.54M states	0.00	0.00	0.74	1.00	0.26	0.00	1.00	1.00	1.00	0.62	0.00	0.63	0.00	1.00	0.00	1.00	1.00	0.26	0.00	0.00	1.54	100.000%	-	74.111%	-
MindShield 91.93M states	0.00	0.00	0.80	0.00	0.20	0.00	1.00	1.00	0.00	0.00	0.00	0.00	0.00	1.00	0.00	1.00	1.00	0.20	0.00	10.00	0.64	100.000%	-	80.260%	-
MindShield 91.93M states	0.00	0.00	0.80	0.00	0.20	0.00	1.00	1.00	0.00	0.00	0.00	0.00	0.00	1.00	0.00	1.00	1.00	0.20	0.00	10.00	0.64	-	-	80.260%	-
SixthSense 64.08M states	0.00	0.00	0.78	1.00	0.22	0.00	1.00	1.00	0.00	0.78	0.00	0.88	0.00	1.00	0.00	1.00	1.00	0.22	0.00	0.00	1.77	100.000%	-	78.218%	-
Camouflage 98.76M states	0.00	0.00	0.78	1.00	0.22	0.00	1.00	1.00	0.00	0.78	0.00	0.88	0.00	1.00	0.00	1.00	1.00	0.22	0.00	0.00	1.77	100.000%	-	78.218%	-
Hunting 89.14M states	0.00	0.00	0.78	1.00	0.22	0.00	1.00	1.00	0.00	0.78	0.00	0.88	0.00	1.00	0.00	1.00	1.00	0.22	0.00	0.00	1.77	100.000%	-	78.218%	-
Tracking 85.82M states	0.00	0.00	0.78	1.00	0.22	0.00	1.00	1.00	0.00	0.78	0.00	0.88	0.00	1.00	0.00	1.00	1.00	0.22	0.00	0.00	1.76	100.000%	-	78.218%	-
MindOverMatter 78.91M states	0.00	0.03	0.69	0.03	0.28	0.28	0.97	0.97	0.97	0.79	0.02	0.00	0.03	0.97	0.00	1.00	0.97	0.00	0.06	2.06	1.57	100.000%	2.986%	68.967%	28.047%
MindOverMatter 78.91M states	0.00	0.03	0.69	0.03	0.28	0.28	0.97	0.97	0.97	0.79	0.02	0.00	0.03	0.97	0.00	1.00	0.97	0.00	0.06	2.06	1.57	-	2.986%	68.967%	28.047%
MindShield 81.91M states	0.00	0.00	0.79	1.00	0.21	0.00	1.00	1.00	1.00	0.64	0.00	0.69	0.00	1.00	0.00	1.00	1.00	0.21	0.00	0.00	1.60	-	-	79.376%	-
SixthSense 64.08M states	0.00	0.00	0.78	1.00	0.22	0.00	1.00	1.00	0.00	0.78	0.00	0.88	0.00	1.00	0.00	1.00	1.00	0.22	0.00	0.00	1.76	-	-	78.218%	-
Camouflage 98.76M states	0.00	0.00	0.78	1.00	0.22	0.00	1.00	1.00	0.00	0.78	0.00	0.88	0.00	1.00	0.00	1.00	1.00	0.22	0.00	0.00	1.76	-	-	78.218%	-
Hunting 89.14M states	0.00	0.00	0.78	1.00	0.22	0.00	1.00	1.00	0.00	0.78	0.00	0.88	0.00	1.00	0.00	1.00	1.00	0.22	0.00	0.00	1.76	-	-	78.218%	-
Tracking 85.82M states	0.00	0.00	0.78	1.00	0.22	0.00	1.00	1.00	0.00	0.77	0.00	0.87	0.00	1.00	0.00	1.00	1.00	0.22	0.00	0.00	1.75	-	-	78.218%	-
MindOverMatter 78.93M states	0.00	0.03	0.71	0.03	0.26	0.26	0.97	0.97	0.97	0.77	0.01	0.00	0.03	0.97	0.00	1.00	0.00	0.00	0.06	2.06	1.55	100.000%	3.078%	70.970%	25.951%
SixthSense 66.40M states	0.00	0.00	0.81	0.00	0.19	0.19	1.00	1.00	0.00	0.85	0.00	0.00	0.00	1.00	0.00	1.00	1.00	0.00	0.00	2.00	1.67	100.000%	-	80.844%	19.156%
Camouflage 98.55M states	0.00	0.00	0.81	0.00	0.19	0.19	1.00	1.00	0.00	0.85	0.00	0.00	0.00	1.00	0.00	1.00	1.00	0.00	0.00	2.00	1.67	100.000%	-	80.844%	19.156%
Hunting 92.92M states	0.00	0.00	0.81	0.00	0.19	0.19	1.00	1.00	0.00	0.85	0.00	0.00	0.00	1.00	0.00	1.00	1.00	0.00	0.00	2.00	1.67	100.000%	-	80.844%	19.156%
Tracking 91.36M states	0.00	0.00	0.81	0.00	0.19	0.19	1.00	1.00	0.00	0.85	0.00	0.00	0.00	1.00	0.00	1.00	1.00	0.00	0.00	2.00	1.67	100.000%	-	80.844%	19.156%
MindOverMatter 78.93M states	0.00	0.03	0.71	0.03	0.26	0.26	0.97	0.97	0.97	0.77	0.01	0.00	0.03	0.97	0.00	1.00	0.00	0.00	0.06	2.06	1.55	-	3.086%	70.965%	25.949%
SixthSense 66.40M states	0.00	0.00	0.81	0.00	0.19	0.19	1.00	1.00	0.00	0.85	0.00	0.00	0.00	1.00	0.00	1.00	1.00	0.00	0.00	2.00	1.67	-	-	80.844%	19.156%
Camouflage 98.55M states	0.00	0.00	0.81	0.00	0.19	0.19	1.00	1.00	0.00	0.85	0.00	0.00	0.00	1.00	0.00	1.00	1.00	0.00	0.00	2.00	1.67	-	-	80.844%	19.156%
Hunting 92.92M states	0.00	0.00	0.81	0.00	0.19	0.19	1.00	1.00	0.00	0.85	0.00	0.00	0.00	1.00	0.00	1.00	1.00	0.00	0.00	2.00	1.67	-	-	80.844%	19.156%
Tracking 91.36M states	0.00	0.00	0.81	0.00	0.19	0.19	1.00	1.00	0.00	0.85	0.00	0.00	0.00	1.00	0.00	1.00	1.00	0.00	0.00	2.00	1.67	-	-	80.844%	19.156%
MindOverMatter 67.36M states	0.00	0.00	0.76	1.00	0.24	0.00	0.94	0.94	0.94	0.61	0.00	0.64	0.00	1.00	0.06	1.00	0.94	0.24	0.11	0.80	1.36	100.000%	5.562%	75.551%	-
MindOverMatter 76.85M states	0.00	0.03	0.72	0.03	0.25	0.25	0.97	0.97	0.00	0.81	0.02	0.00	0.03	0.97	0.00	1.00	0.97	0.00	0.06	2.06	1.62	100.000%	3.116%	71.514%	25.370%
MindOverMatter 76.85M states	0.00	0.03	0.72	0.03	0.25	0.25	0.97	0.97	0.00	0.81	0.02	0.00	0.03	0.97	0.00	1.00	0.97	0.00	0.06	2.06	1.62	-	3.119%	71.512%	25.369%
MindOverMatter 67.36M states	0.00	0.00	0.76	1.00	0.24	0.00	0.94	0.94	0.94	0.60	0.00	0.62	0.00	1.00	0.06	1.00	0.94	0.24	0.11	0.81	1.34	-	5.635%	75.570%	-
MindOverMatter 76.88M states	0.00	0.03	0.75	0.03	0.22	0.22	0.97	0.97	0.00	0.81	0.01	0.00	0.03	0.97	0.00	1.00	0.00	0.00	0.07	2.07	1.63	100.000%	3.271%	74.524%	22.206%
MindShield 74.14M states	0.00	0.00	0.79	1.00	0.21	0.00	1.00	1.00	1.00	0.63	0.00	0.65	0.00	1.00	0.00	1.00	1.00	0.21	0.00	0.00	1.57	100.000%	-	79.376%	-
MindOverMatter 70.83M states	0.00	0.03	0.74	0.03	0.22	0.22	0.97	0.97	0.97	0.78	0.00	0.00	0.03	0.97	0.00	1.00	0.97	0.00	0.07	2.07	1.57	100.000%	3.317%	74.323%	22.360%
MindOverMatter 70.83M states	0.00	0.03	0.74	0.03	0.22	0.22	0.97	0.97	0.97	0.78	0.00	0.00	0.03	0.97	0.00	1.00	0.97	0.00	0.07	2.07	1.57	-	3.317%	74.323%	22.360%
MindShield 80.01M states	0.00	0.00	0.74	1.00	0.26	0.00	1.00	1.00	0.00	0.77	0.00	0.86	0.00	1.00	0.00	1.00	1.00	0.26	0.00	0.00	1.75	100.000%	-	74.467%	-
MindOverMatter 76.88M states	0.00	0.03	0.74	0.03	0.22	0.22	0.97	0.97	0.00	0.81	0.01	0.00	0.03	0.97	0.00	1.00	0.00	0.00	0.07	2.07	1.63	-	3.340%	74.470%	22.189%
MindOverMatter 60.24M states	0.00	0.00	0.76	1.00	0.24	0.00	0.94	0.94	0.94	0.57	0.00	0.55	0.00	1.00	0.06	1.00	0.94	0.24	0.12	0.88	1.28	100.000%	6.144%	75.702%	-
MindShield 91.96M states	0.00	0.00	0.89	0.00	0.11	0.00	1.00	1.00	0.00	0.67	0.00	0.80	0.00	1.00	0.00	1.00	0.00	0.11	0.00	0.00	1.68	100.000%	-	88.992%	-
MindOverMatter 65.42M states	0.00	0.00	0.80	1.00	0.20	0.00	0.94	0.94	0.00	0.72	0.00	0.77	0.00	1.00	0.06	1.00	0.94	0.20	0.12	0.89	1.46	100.000%	6.232%	79.576%	-
MindShield 91.96M states	0.00	0.00	0.89	0.00	0.11	0.00	1.00	1.00	0.00	0.67	0.00	0.80	0.00	1.00	0.00	1.00	0.00	0.11	0.00	0.00	1.68	-	-	88.992%	-
MindShield 85.31M states	0.00	0.00	0.89	0.00	0.11	0.00	1.00	1.00	1.00	0.00	0.00	0.00	0.00	1.00	0.00	1.00	1.00	0.11	0.00	10.00	0.64	100.000%	-	89.269%	-
MindShield 85.31M states	0.00	0.00	0.89	0.00	0.11	0.00	1.00	1.00	1.00	0.00	0.00	0.00	0.00	1.00	0.00	1.00	1.00	0.11	0.00	10.00	0.64	-	-	89.269%	-
MindOverMatter 65.42M states	0.00	0.00	0.80	1.00	0.20	0.00	0.94	0.94	0.00	0.72	0.00	0.75	0.00	1.00	0.06	1.00	0.94	0.20	0.13	0.92	1.44	-	6.439%	79.621%	-
SixthSense 58.13M states	0.00	0.00	0.78	1.00	0.22	0.00	1.00	1.00	0.00	0.77	0.00	0.86	0.00	1.00	0.00	1.00	1.00	0.22	0.00	0.00	1.75	100.000%	-	78.218%	-
Camouflage 85.16M states	0.00	0.00	0.78	1.00	0.22	0.00	1.00	1.00	0.00	0.77	0.00	0.86	0.00	1.00	0.00	1.00	1.00	0.22	0.00	0.00	1.75	100.000%	-	78.218%	-
Hunting 80.32M states	0.00	0.00	0.78	1.00	0.22	0.00	1.00	1.00	0.00	0.77	0.00	0.86	0.00	1.00	0.00	1.00	1.00	0.22	0.00	0.00	1.75	100.000%	-	78.218%	-
Tracking 78.60M states	0.00	0.00	0.78	1.00	0.22	0.00	1.00	1.00	0.00	0.76	0.00	0.84	0.00	1.00	0.00	1.00	1.00	0.22	0.00	0.00	1.73	100.000%	-	78.218%	-
SixthSense 65.98M states	0.00	0.00	0.78	1.00	0.22	0.00	1.00	1.00	1.00	0.79	0.00	0.77	0.00	1.00	0.00	1.00	0.00	0.22	0.00	0.00	1.71	100.000%	-	78.243%	-
Hunting 92.04M states	0.00	0.00	0.78	1.00	0.22	0.00	1.00	1.00	1.00	0.79	0.00	0.77	0.00	1.00	0.00	1.00	0.00	0.22	0.00	0.00	1.71	100.000%	-	78.243%	-
Camouflage 101.66M states	0.00	0.00	0.78	1.00	0.22	0.00	1.00	1.00	1.00	0.77	0.00	0.75	0.00	1.00	0.00	1.00	0.00	0.22	0.00	0.00	1.69	100.000%	-	78.243%	-
Tracking 87.70M states	0.00	0.00	0.78	1.00	0.22	0.00	1.00	1.00	1.00	0.78	0.00	0.77	0.00	1.00	0.00	1.00	0.00	0.22	0.00	0.00	1.71	100.000%	-	78.243%	-
MindOverMatter 69.80M states	0.00	0.04	0.78	0.04	0.18	0.18	0.96	0.96	0.00	0.82	0.00	0.00	0.04	0.96	0.00	1.00	0.96	0.00	0.07	2.07	1.64	100.000%	3.713%	77.842%	18.445%
MindOverMatter 69.80M states	0.00	0.04	0.78	0.04	0.18	0.18	0.96	0.96	0.00	0.82	0.00	0.00	0.04	0.96	0.00	1.00	0.96	0.00	0.07	2.07	1.64	-	3.717%	77.839%	18.444%
SixthSense 67.42M states	0.00	0.00	0.81	0.00	0.19	0.19	1.00	1.00	1.00	0.83	0.00	0.00	0.00	1.00	0.00	1.00	0.00	0.00	0.00	2.00	1.64	100.000%	-	80.944%	19.056%
Camouflage 100.11M states	0.00	0.00	0.81	0.00	0.19	0.19	1.00	1.00	1.00	0.83	0.00	0.00	0.00	1.00	0.00	1.00	0.00	0.00	0.00	2.00	1.64	100.000%	-	80.944%	19.056%
Hunting 94.47M states	0.00	0.00	0.81	0.00	0.19	0.19	1.00	1.00	1.00	0.83	0.00	0.00	0.00	1.00	0.00	1.00	0.00	0.00	0.00	2.00	1.64	100.000%	-	80.944%	19.056%
Tracking 92.38M states	0.00	0.00	0.81	0.00	0.19	0.19	1.00	1.00	1.00	0.83	0.00	0.00	0.00	1.00	0.00	1.00	0.00	0.00	0.00	2.00	1.64	100.000%	-	80.944%	19.056%
SixthSense 59.09M states	0.00	0.00	0.74	1.00	0.26	0.00	1.00	1.00	1.00	0.61	0.00	0.61	0.00	1.00	0.00	1.00	1.00	0.26	0.00	0.00	1.53	-	-	74.111%	-
Camouflage 86.63M states	0.00	0.00	0.74	1.00	0.26	0.00	1.00	1.00	1.00	0.61	0.00	0.61	0.00	1.00	0.00	1.00	1.00	0.26	0.00	0.00	1.53	-	-	74.111%	-
Hunting 81.79M states	0.00	0.00	0.74	1.00	0.26	0.00	1.00	1.00	1.00	0.61	0.00	0.61	0.00	1.00	0.00	1.00	1.00	0.26	0.00	0.00	1.53	-	-	74.111%	-
SixthSense 67.42M states	0.00	0.00	0.81	0.00	0.19	0.19	1.00	1.00	1.00	0.83	0.00	0.00	0.00	1.00	0.00	1.00	0.00	0.00	0.00	2.00	1.64	-	-	80.944%	19.056%
Camouflage 100.11M states	0.00	0.00	0.81	0.00	0.19	0.19	1.00	1.00	1.00	0.83	0.00	0.00	0.00	1.00	0.00	1.00	0.00	0.00	0.00	2.00	1.64	-	-	80.944%	19.056%
Hunting 94.47M states	0.00	0.00	0.81	0.00	0.19	0.19	1.00	1.00	1.00	0.83	0.00	0.00	0.00	1.00	0.00	1.00	0.00	0.00	0.00	2.00	1.64	-	-	80.944%	19.056%
Tracking 92.38M states	0.00	0.00	0.81	0.00	0.19	0.19	1.00	1.00	1.00	0.83	0.00	0.00	0.00	1.00	0.00	1.00	0.00	0.00	0.00	2.00	1.64	-	-	80.944%	19.056%
Tracking 79.54M states	0.00	0.00	0.74	1.00	0.26	0.00	1.00	1.00	1.00	0.60	0.00	0.58	0.00	1.00	0.00	1.00	1.00	0.26	0.00	0.00	1.49	-	-	74.111%	-
MindShield 81.90M states	0.00	0.00	0.73	1.00	0.27	0.00	1.00	1.00	1.00	0.72	0.00	0.66	0.00	1.00	0.00	1.00	0.00	0.27	0.00	0.00	1.57	100.000%	-	73.321%	-
MindShield 80.01M states	0.00	0.00	0.82	1.00	0.18	0.00	1.00	1.00	0.00	0.78	0.00	0.88	0.00	1.00	0.00	1.00	1.00	0.18	0.00	0.00	1.76	-	-	82.116%	-
SixthSense 65.98M states	0.00	0.00	0.78	1.00	0.22	0.00	1.00	1.00	1.00	0.78	0.00	0.76	0.00	1.00	0.00	1.00	0.00	0.22	0.00	0.00	1.69	-	-	78.243%	-
Hunting 92.04M states	0.00	0.00	0.78	1.00	0.22	0.00	1.00	1.00	1.00	0.78	0.00	0.76	0.00	1.00	0.00	1.00	0.00	0.22	0.00	0.00	1.69	-	-	78.243%	-
Camouflage 101.66M states	0.00	0.00	0.78	1.00	0.22	0.00	1.00	1.00	1.00	0.75	0.00	0.71	0.00	1.00	0.00	1.00	0.00	0.22	0.00	0.00	1.63	-	-	78.243%	-
Tracking 87.70M states	0.00	0.00	0.78	1.00	0.22	0.00	1.00	1.00	1.00	0.77	0.00	0.75	0.00	1.00	0.00	1.00	0.00	0.22	0.00	0.00	1.68	-	-	78.243%	-
MindOverMatter 59.27M states	0.00	0.00	0.80	1.00	0.20	0.00	0.93	0.93	0.00	0.69	0.00	0.69	0.00	1.00	0.07	1.00	0.93	0.20	0.15	1.07	1.40	100.000%	7.473%	79.846%	-
MindOverMatter 67.34M states	0.00	0.00	0.80	1.00	0.20	0.00	0.92	0.92	0.92	0.68	0.00	0.54	0.00	1.00	0.08	1.00	0.00	0.20	0.15	1.08	1.46	100.000%	7.503%	79.876%	-
MindShield 74.14M states	0.00	0.00	0.79	1.00	0.21	0.00	1.00	1.00	1.00	0.61	0.00	0.61	0.00	1.00	0.00	1.00	1.00	0.21	0.00	0.00	1.53	-	-	79.376%	-
MindOverMatter 60.24M states	0.00	0.00	0.76	1.00	0.24	0.00	0.92	0.92	0.92	0.53	0.00	0.49	0.00	1.00	0.08	1.00	0.92	0.24	0.16	1.12	1.22	-	7.785%	76.127%	-
MindShield 84.29M states	0.00	0.00	0.92	0.00	0.08	0.00	1.00	1.00	0.00	0.00	0.00	0.00	0.00	1.00	0.00	1.00	1.00	0.08	0.00	10.00	0.64	100.000%	-	92.254%	-
MindShield 84.29M states	0.00	0.00	0.92	0.00	0.08	0.00	1.00	1.00	0.00	0.00	0.00	0.00	0.00	1.00	0.00	1.00	1.00	0.08	0.00	10.00	0.64	-	-	92.254%	-
MindOverMatter 70.84M states	0.00	0.04	0.77	0.04	0.18	0.18	0.96	0.96	0.96	0.79	0.00	0.00	0.04	0.96	0.00	1.00	0.00	0.00	0.09	2.09	1.61	100.000%	4.494%	77.307%	18.200%
MindOverMatter 70.84M states	0.00	0.05	0.77	0.05	0.18	0.18	0.95	0.95	0.95	0.79	0.00	0.00	0.05	0.95	0.00	1.00	0.00	0.00	0.09	2.09	1.61	-	4.505%	77.298%	18.197%
MindOverMatter 67.34M states	0.00	0.00	0.80	1.00	0.20	0.00	0.92	0.92	0.92	0.63	0.00	0.48	0.00	1.00	0.08	1.00	0.00	0.20	0.16	1.18	1.38	-	8.193%	80.026%	-
MindShield 73.19M states	0.00	0.00	0.82	1.00	0.18	0.00	1.00	1.00	0.00	0.77	0.00	0.86	0.00	1.00	0.00	1.00	1.00	0.18	0.00	0.00	1.75	100.000%	-	82.116%	-
SixthSense 64.07M states	0.00	0.00	0.73	1.00	0.27	0.00	1.00	1.00	0.00	0.84	0.00	0.86	0.00	1.00	0.00	1.00	0.00	0.27	0.00	0.00	1.80	100.000%	-	72.767%	-
Camouflage 98.73M states	0.00	0.00	0.73	1.00	0.27	0.00	1.00	1.00	0.00	0.84	0.00	0.86	0.00	1.00	0.00	1.00	0.00	0.27	0.00	0.00	1.80	100.000%	-	72.767%	-
Hunting 89.11M states	0.00	0.00	0.73	1.00	0.27	0.00	1.00	1.00	0.00	0.84	0.00	0.86	0.00	1.00	0.00	1.00	0.00	0.27	0.00	0.00	1.80	100.000%	-	72.767%	-
Tracking 85.81M states	0.00	0.00	0.73	1.00	0.27	0.00	1.00	1.00	0.00	0.84	0.00	0.85	0.00	1.00	0.00	1.00	0.00	0.27	0.00	0.00	1.79	100.000%	-	72.767%	-
SixthSense 58.13M states	0.00	0.00	0.78	1.00	0.22	0.00	1.00	1.00	0.00	0.77	0.00	0.85	0.00	1.00	0.00	1.00	1.00	0.22	0.00	0.00	1.74	-	-	78.218%	-
Camouflage 85.16M states	0.00	0.00	0.78	1.00	0.22	0.00	1.00	1.00	0.00	0.77	0.00	0.85	0.00	1.00	0.00	1.00	1.00	0.22	0.00	0.00	1.74	-	-	78.218%	-
Hunting 80.32M states	0.00	0.00	0.78	1.00	0.22	0.00	1.00	1.00	0.00	0.77	0.00	0.85	0.00	1.00	0.00	1.00	1.00	0.22	0.00	0.00	1.74	-	-	78.218%	-
Tracking 78.60M states	0.00	0.00	0.78	1.00	0.22	0.00	1.00	1.00	0.00	0.75	0.00	0.82	0.00	1.00	0.00	1.00	1.00	0.22	0.00	0.00	1.71	-	-	78.218%	-
SixthSense 59.07M states	0.00	0.00	0.78	1.00	0.22	0.00	1.00	1.00	1.00	0.77	0.00	0.75	0.00	1.00	0.00	1.00	0.00	0.22	0.00	0.00	1.68	100.000%	-	78.243%	-
Hunting 81.75M states	0.00	0.00	0.78	1.00	0.22	0.00	1.00	1.00	1.00	0.77	0.00	0.75	0.00	1.00	0.00	1.00	0.00	0.22	0.00	0.00	1.68	100.000%	-	78.243%	-
Camouflage 86.59M states	0.00	0.00	0.78	1.00	0.22	0.00	1.00	1.00	1.00	0.72	0.00	0.66	0.00	1.00	0.00	1.00	0.00	0.22	0.00	0.00	1.57	100.000%	-	78.243%	-
Tracking 79.52M states	0.00	0.00	0.78	1.00	0.22	0.00	1.00	1.00	1.00	0.76	0.00	0.74	0.00	1.00	0.00	1.00	0.00	0.22	0.00	0.00	1.66	100.000%	-	78.243%	-
MindShield 80.00M states	0.00	0.00	0.76	1.00	0.24	0.00	1.00	1.00	0.00	0.84	0.00	0.86	0.00	1.00	0.00	1.00	0.00	0.24	0.00	0.00	1.80	100.000%	-	76.454%	-
SixthSense 66.41M states	0.00	0.00	0.85	0.00	0.15	0.15	1.00	1.00	0.00	0.88	0.00	0.00	0.00	1.00	0.00	1.00	0.00	0.00	0.00	2.00	1.73	100.000%	-	84.841%	15.159%
Camouflage 98.60M states	0.00	0.00	0.85	0.00	0.15	0.15	1.00	1.00	0.00	0.88	0.00	0.00	0.00	1.00	0.00	1.00	0.00	0.00	0.00	2.00	1.73	100.000%	-	84.841%	15.159%
Hunting 92.97M states	0.00	0.00	0.85	0.00	0.15	0.15	1.00	1.00	0.00	0.88	0.00	0.00	0.00	1.00	0.00	1.00	0.00	0.00	0.00	2.00	1.73	100.000%	-	84.841%	15.159%
Tracking 91.39M states	0.00	0.00	0.85	0.00	0.15	0.15	1.00	1.00	0.00	0.88	0.00	0.00	0.00	1.00	0.00	1.00	0.00	0.00	0.00	2.00	1.73	100.000%	-	84.841%	15.159%
MindOverMatter 59.27M states	0.00	0.00	0.80	1.00	0.20	0.00	0.90	0.90	0.00	0.67	0.00	0.64	0.00	1.00	0.10	1.00	0.90	0.20	0.20	1.45	1.36	-	10.119%	80.423%	-
MindOverMatter 65.41M states	0.00	0.00	0.75	1.00	0.25	0.00	0.90	0.90	0.00	0.71	0.00	0.66	0.00	1.00	0.10	1.00	0.00	0.24	0.20	1.46	1.32	100.000%	10.138%	75.194%	-
SixthSense 66.41M states	0.00	0.00	0.85	0.00	0.15	0.15	1.00	1.00	0.00	0.88	0.00	0.00	0.00	1.00	0.00	1.00	0.00	0.00	0.00	2.00	1.73	-	-	84.841%	15.159%
Camouflage 98.60M states	0.00	0.00	0.85	0.00	0.15	0.15	1.00	1.00	0.00	0.88	0.00	0.00	0.00	1.00	0.00	1.00	0.00	0.00	0.00	2.00	1.73	-	-	84.841%	15.159%
Hunting 92.97M states	0.00	0.00	0.85	0.00	0.15	0.15	1.00	1.00	0.00	0.88	0.00	0.00	0.00	1.00	0.00	1.00	0.00	0.00	0.00	2.00	1.73	-	-	84.841%	15.159%
Tracking 91.39M states	0.00	0.00	0.85	0.00	0.15	0.15	1.00	1.00	0.00	0.88	0.00	0.00	0.00	1.00	0.00	1.00	0.00	0.00	0.00	2.00	1.73	-	-	84.841%	15.159%
MindShield 81.90M states	0.00	0.00	0.80	1.00	0.20	0.00	1.00	1.00	1.00	0.75	0.00	0.71	0.00	1.00	0.00	1.00	0.00	0.20	0.00	0.00	1.63	-	-	80.131%	-
MindOverMatter 60.22M states	0.00	0.00	0.80	1.00	0.20	0.00	0.90	0.90	0.90	0.59	0.00	0.42	0.00	1.00	0.10	1.00	0.00	0.20	0.21	1.49	1.30	100.000%	10.361%	80.498%	-
SixthSense 64.07M states	0.00	0.00	0.79	1.00	0.21	0.00	1.00	1.00	0.00	0.85	0.00	0.87	0.00	1.00	0.00	1.00	0.00	0.21	0.00	0.00	1.81	-	-	78.943%	-
Camouflage 98.73M states	0.00	0.00	0.79	1.00	0.21	0.00	1.00	1.00	0.00	0.85	0.00	0.87	0.00	1.00	0.00	1.00	0.00	0.21	0.00	0.00	1.81	-	-	78.943%	-
Hunting 89.11M states	0.00	0.00	0.79	1.00	0.21	0.00	1.00	1.00	0.00	0.85	0.00	0.87	0.00	1.00	0.00	1.00	0.00	0.21	0.00	0.00	1.81	-	-	78.943%	-
Tracking 85.81M states	0.00	0.00	0.79	1.00	0.21	0.00	1.00	1.00	0.00	0.85	0.00	0.87	0.00	1.00	0.00	1.00	0.00	0.21	0.00	0.00	1.81	-	-	78.943%	-
MindShield 73.19M states	0.00	0.00	0.82	1.00	0.18	0.00	1.00	1.00	0.00	0.77	0.00	0.85	0.00	1.00	0.00	1.00	1.00	0.18	0.00	0.00	1.74	-	-	82.116%	-
MindShield 85.31M states	0.00	0.00	0.93	0.00	0.07	0.00	1.00	1.00	1.00	0.00	0.00	0.00	0.00	1.00	0.00	1.00	0.00	0.07	0.00	10.00	0.60	100.000%	-	93.119%	-
MindShield 85.31M states	0.00	0.00	0.93	0.00	0.07	0.00	1.00	1.00	1.00	0.00	0.00	0.00	0.00	1.00	0.00	1.00	0.00	0.07	0.00	10.00	0.60	-	-	93.119%	-
MindOverMatter 69.81M states	0.00	0.07	0.79	0.07	0.14	0.14	0.93	0.93	0.00	0.82	0.00	0.00	0.07	0.93	0.00	1.00	0.00	0.00	0.13	2.13	1.68	100.000%	6.524%	79.305%	14.170%
MindOverMatter 69.81M states	0.00	0.07	0.79	0.07	0.14	0.14	0.93	0.93	0.00	0.82	0.00	0.00	0.07	0.93	0.00	1.00	0.00	0.00	0.13	2.13	1.67	-	6.659%	79.191%	14.150%
MindOverMatter 65.41M states	0.00	0.00	0.82	1.00	0.18	0.00	0.88	0.88	0.00	0.59	0.00	0.63	0.00	1.00	0.12	1.00	0.00	0.18	0.25	1.77	1.47	-	12.330%	81.539%	-
MindShield 74.13M states	0.00	0.00	0.80	1.00	0.20	0.00	1.00	1.00	1.00	0.72	0.00	0.66	0.00	1.00	0.00	1.00	0.00	0.20	0.00	0.00	1.57	100.000%	-	80.131%	-
SixthSense 59.07M states	0.00	0.00	0.78	1.00	0.22	0.00	1.00	1.00	1.00	0.76	0.00	0.74	0.00	1.00	0.00	1.00	0.00	0.22	0.00	0.00	1.66	-	-	78.243%	-
Hunting 81.75M states	0.00	0.00	0.78	1.00	0.22	0.00	1.00	1.00	1.00	0.76	0.00	0.74	0.00	1.00	0.00	1.00	0.00	0.22	0.00	0.00	1.66	-	-	78.243%	-
Tracking 79.52M states	0.00	0.00	0.78	1.00	0.22	0.00	1.00	1.00	1.00	0.75	0.00	0.72	0.00	1.00	0.00	1.00	0.00	0.22	0.00	0.00	1.64	-	-	78.243%	-
Camouflage 86.59M states	0.00	0.00	0.78	1.00	0.22	0.00	1.00	1.00	1.00	0.68	0.00	0.58	0.00	1.00	0.00	1.00	0.00	0.22	0.00	0.00	1.47	-	-	78.243%	-
SixthSense 58.12M states	0.00	0.00	0.79	1.00	0.21	0.00	1.00	1.00	0.00	0.84	0.00	0.86	0.00	1.00	0.00	1.00	0.00	0.21	0.00	0.00	1.80	100.000%	-	78.943%	-
Camouflage 85.13M states	0.00	0.00	0.79	1.00	0.21	0.00	1.00	1.00	0.00	0.84	0.00	0.86	0.00	1.00	0.00	1.00	0.00	0.21	0.00	0.00	1.80	100.000%	-	78.943%	-
Hunting 80.29M states	0.00	0.00	0.79	1.00	0.21	0.00	1.00	1.00	0.00	0.84	0.00	0.86	0.00	1.00	0.00	1.00	0.00	0.21	0.00	0.00	1.80	100.000%	-	78.943%	-
Tracking 78.58M states	0.00	0.00	0.79	1.00	0.21	0.00	1.00	1.00	0.00	0.84	0.00	0.85	0.00	1.00	0.00	1.00	0.00	0.21	0.00	0.00	1.79	100.000%	-	78.943%	-
MindOverMatter 60.22M states	0.00	0.00	0.82	1.00	0.18	0.00	0.84	0.84	0.84	0.51	0.00	0.34	0.00	1.00	0.16	1.00	0.00	0.18	0.32	2.32	1.19	-	16.174%	81.762%	-
MindOverMatter 59.25M states	0.00	0.00	0.82	1.00	0.18	0.00	0.84	0.84	0.00	0.56	0.00	0.57	0.00	1.00	0.16	1.00	0.00	0.18	0.33	2.36	1.40	100.000%	16.426%	82.401%	-
Hunting 80.29M states	0.00	0.00	1.00	1.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	1.00	1.00	1.00	0.00	0.00	2.00	14.35	0.98	-	100.000%	100.000%	-
Tracking 78.58M states	0.00	0.00	1.00	1.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	1.00	1.00	1.00	0.00	0.00	2.00	14.35	0.98	-	100.000%	100.000%	-
SixthSense 58.12M states	0.00	0.00	1.00	1.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	1.00	1.00	1.00	0.00	0.00	2.00	14.34	0.98	-	100.000%	100.000%	-
MindShield 80.00M states	0.00	0.00	0.83	1.00	0.17	0.00	1.00	1.00	0.00	0.85	0.00	0.87	0.00	1.00	0.00	1.00	0.00	0.17	0.00	0.00	1.81	-	-	82.621%	-
MindOverMatter 59.25M states	0.00	0.00	1.00	1.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	1.00	1.00	1.00	0.00	0.00	2.00	14.35	0.64	-	100.000%	100.000%	-
MindShield 74.13M states	0.00	0.00	0.80	1.00	0.20	0.00	1.00	1.00	1.00	0.68	0.00	0.58	0.00	1.00	0.00	1.00	0.00	0.20	0.00	0.00	1.47	-	-	80.131%	-
MindShield 84.30M states	0.00	0.00	0.94	0.00	0.06	0.00	1.00	1.00	0.00	0.67	0.00	0.80	0.00	1.00	0.00	1.00	0.00	0.06	0.00	0.00	1.68	100.000%	-	93.825%	-
Camouflage 85.13M states	0.03	0.11	0.86	1.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.11	0.86	0.89	1.00	0.00	0.00	2.00	18.33	0.90	-	100.000%	85.618%	-
MindShield 84.30M states	0.00	0.00	0.94	0.00	0.06	0.00	1.00	1.00	0.00	0.67	0.00	0.80	0.00	1.00	0.00	1.00	0.00	0.06	0.00	0.00	1.68	-	-	93.825%	-
MindShield 73.18M states	0.00	0.00	0.83	1.00	0.17	0.00	1.00	1.00	0.00	0.84	0.00	0.86	0.00	1.00	0.00	1.00	0.00	0.17	0.00	0.00	1.80	100.000%	-	82.621%	-
MindShield 73.18M states	0.00	0.00	0.83	1.00	0.17	0.00	1.00	1.00	0.00	0.84	0.00	0.86	0.00	1.00	0.00	1.00	0.00	0.17	0.00	0.00	1.80	-	-	82.621%	-

]]> Dark souls 1 Remastered armor stats https://hbtvl.banquise.net/posts/2024-08-13-ds1r-armor-stats.html 2024-08-13T00:00:00Z

I compiled some statistics about the best armor combinations for a given weight budget, and a comparison of each armor pieces. It has a chart for each damage type plus poise (where all is a combination of all the damage types, plus poise).

You can find graphs for best armor combination, and a representation of each individual armor piece. To get details, just click on an image, and hover over the part you are interested about.

Damage	Best combination	Individual comparison
all
Physical
Magic
Poise
Fire
Lightning
Pierce
Slash
Strike
Poise

]]> Big updates to the Elden Ring tools https://hbtvl.banquise.net/posts/2024-08-02-eldenring-updates.html 2024-06-23T00:00:00Z

First of all, I finally wrote a tool that extracts all relevant game data directly from the Elden Ring files. The [ParamDex](https://github.com/soulsmods/Paramdex) repository has been essential for this. I used to have to find the data from external sources (such as parsing fextralife), with problems of reliability, and, more importantly, freshness. The tool is here. Special thanks to TheChewanater and TarnishedSpreadsheet for explaining the missing pieces.

So, the new stuff is:

updated armor stats, and score formula (also for the companion app). When extracting data, I realized that the armor stats are not stored in the same way that are displayed. For example, if a piece of armor has 10.0 strike resistance, it will be stored as 0.90. This made even more obvious the fact that it is an absorption value, and you are not supposed to sum them, but to multiply them together. New tools should now reflect the actual formula.
precomputed minimaxed stats for all weapons. It takes about half an hour to compute them all, thanks to the new Smithscript weapons in Shadow of the Erdtree that can scale on all five stats.

I will probably publish a quick guide that explains how to parse the game files.

Fun fact, the august patch updated Death Knight's Twin Axes with no notes.

]]> Shadow of the Erdtree armor stats https://hbtvl.banquise.net/posts/2024-06-23-erdtree-armor-stats.html 2024-06-23T00:00:00Z

I updated the companion and graphes with armor from Shadow of the Erdtree. I compiled some statistics about the best armor combinations for a given weight budget, and a comparison of each armor pieces. It has a chart for each damage type plus poise (where all is a combination of all the damage types, plus poise).

You can find graphs for best armor combination, and a representation of each individual armor piece. To get details, just click on an image, and hover over the part you are interested about.

updates

2024/08/02 - updated data for regulation 11310027, changed the score calculation formula so that it better matches in game effects

Damage	Best combination	Individual comparison
all
Physical
Magic
Poise
Fire
Holy
Lightning
Pierce
Slash
Strike
Poise

]]> Dark souls 3 armor stats https://hbtvl.banquise.net/posts/2023-05-29-ds3-armor-stats.html 2023-05-29T00:00:00Z

You can find graphs for best armor combination, and a representation of each individual armor piece. To get details, just click on an image, and hover over the part you are interested about.

Damage	Best combination	Individual comparison
all
Physical
Magic
Poise
Fire
Holy
Lightning
Pierce
Slash
Strike
Poise

]]> Elden Ring armor stats https://hbtvl.banquise.net/posts/2023-05-16-eldenring-armor-stats.html 2023-05-16T00:00:00Z

You can find graphes for best armor combination, and a representation of each individual armor piece. To get details, just click on an image, and hover over the part you are interested about.

Here are some observations:

for a given weight budget, chest pieces are usually more efficient ;
for elemental/magical damage, light armors are very efficient ;
as with damage scaling, there are diminishing returns.

Damage	Best combination	Individual comparison
all
Physical
Magic
Poise
Fire
Holy
Lightning
Pierce
Slash
Strike
Poise

]]> Eldenring companion, initial release https://hbtvl.banquise.net/posts/2023-05-14-eldenring-companion.html 2023-05-14T00:00:00Z

I just published an initial version of an Elden Ring companion application. The goal is to use it as a checklist, a bit like this one, with the following two features:

a search bar to quickly write the stuff you grabbed while playing
a real time armor optimizer, that will reflect updates in load capacity and picked armors

Do not hesitate leaving notes on the GitHub repo.

Note: you might notice there is also a Rust project. I wanted to write the armor optimizer in Rust and embed it as a WASM module, but it turns Vite doesn't like that, and the JavaScript is fast enough, so it is on hold for now.

]]> Solving a gamebook : Lone Wolf book 04, The Chasm of Doom https://hbtvl.banquise.net/posts/2022-03-01-GBS-chapter4.html 2022-03-01T00:00:00Z

What should you know?

First of all, having the +4 Strength Potion from the previous book is essential to maximizing the probability of winning the next book. So, if you start with it, you should not use it. In the computed solutions, there is only one case that would require using it, and it is when the following conditions are met:

missing disciplines are Mind Over Matter and Camouflage;
silver helm, but no silver sword
and some extra condition I could not decipher :)

The Flask of Holy Water is really important in order to avoid the hardest fight in the book. The first time you can grab it is at section 78, but there is a 50% chance you can end up there, because of the random fork earlier in section 319. This means that if you do not own it when reaching section 118, you should ascend the stairs, in order to secure it later in section 302. This however deprives you from the chance to fight the Elix, which will be useful in book 05.

Talking about section 302, there are four good items to get:

2 potions au Laumspur
a strength potion
the flask of holy water

You should drop extra meals in order to get these items, except when you have the Sommerwerd. In that case, you can ignore the strength potion.

Initial conditions

The initial conditions are:

start with:
- a Sword, or the Sommerswerd
- the Silver Helmet, or not
- the +4 Strength Potion, or not
- one potion of Laumspur
- a Shield
- 4 meals
- 15 gold (although having more or less gold has no influence on the win rate)
max endurance: 20
combat skill: 10
all disciplines, except the listed discipline (weapon specialization is Sword)

Note that I preselected the missing disciplines to ignore those that are really necessary.

Mega solution!

What does all of this mean?

Missing Discs: the disciplines that have not been selected in this run; under it, the number of states that have been computed for this solution;
Win rate: the probability of winning the fifth book when starting from this state;
Raw rate: the probability of winning this book. Note that there might be a better route if you are playing this book standalone, as this is the route that includes what's happening till book 05;
SS: start with the Sommerswerd;
SH: start with the Silver Helmet;
+4: start with the +4 Strength Potion;
Fought Elix: probability of fighting an Elix in this route;
Laumspur collect: probability of collecting Laumspur in chapters 12, 268 and 302;
End with +4 Strength Potion: probability of holding the +4 Strength potion at the end of the run.

Missing disc #states	Win rate	Raw rate	SS	SH	+4	Fought Elix	Laumspur collect	Final gold	End with +2 Strength Potion	End with +4 Strength Potion
Camouflage / SixthSense 20.23M states	79.669%	90.000%	yes	yes	yes	50.000%	90.000% / 100.000% / 50.000%	4.00	-	100.000%
Hunting / SixthSense 24.50M states	79.669%	90.000%	yes	yes	yes	50.000%	90.000% / 100.000% / 50.000%	5.50	-	100.000%
Camouflage / Hunting 19.39M states	79.669%	90.000%	yes	yes	yes	50.000%	90.000% / 100.000% / 50.000%	4.00	-	100.000%
SixthSense / MindOverMatter 25.30M states	79.669%	90.000%	yes	yes	yes	50.000%	90.000% / 100.000% / 50.000%	5.50	-	100.000%
Camouflage / MindOverMatter 25.43M states	79.669%	90.000%	yes	yes	yes	50.000%	90.000% / 100.000% / 50.000%	5.50	-	100.000%
Hunting / MindOverMatter 24.50M states	79.669%	90.000%	yes	yes	yes	50.000%	90.000% / 100.000% / 50.000%	5.50	-	100.000%
SixthSense / MindShield 25.16M states	79.669%	90.000%	yes	yes	yes	50.000%	90.000% / 100.000% / 50.000%	5.50	-	100.000%
Camouflage / MindShield 20.13M states	79.669%	90.000%	yes	yes	yes	50.000%	90.000% / 100.000% / 50.000%	4.00	-	100.000%
Hunting / MindShield 24.36M states	79.669%	90.000%	yes	yes	yes	50.000%	90.000% / 100.000% / 50.000%	5.50	-	100.000%
SixthSense / Tracking 25.48M states	79.669%	90.000%	yes	yes	yes	50.000%	90.000% / 100.000% / 50.000%	5.50	-	100.000%
Camouflage / Tracking 35.69M states	79.669%	90.000%	yes	yes	yes	50.000%	90.000% / 100.000% / 50.000%	4.00	-	100.000%
Hunting / Tracking 24.93M states	79.669%	90.000%	yes	yes	yes	50.000%	90.000% / 100.000% / 50.000%	5.50	-	100.000%
Tracking / MindOverMatter 25.48M states	79.669%	90.000%	yes	yes	yes	50.000%	90.000% / 100.000% / 50.000%	5.50	-	100.000%
Tracking / MindShield 25.34M states	79.669%	90.000%	yes	yes	yes	50.000%	90.000% / 100.000% / 50.000%	5.50	-	100.000%
Camouflage / SixthSense 20.35M states	77.041%	90.000%	yes	no	yes	50.000%	90.000% / 100.000% / 50.000%	4.00	-	100.000%
Hunting / SixthSense 24.66M states	77.041%	90.000%	yes	no	yes	50.000%	90.000% / 100.000% / 50.000%	5.50	-	100.000%
Camouflage / Hunting 19.51M states	77.041%	90.000%	yes	no	yes	50.000%	90.000% / 100.000% / 50.000%	4.00	-	100.000%
SixthSense / MindOverMatter 25.47M states	77.041%	90.000%	yes	no	yes	50.000%	90.000% / 100.000% / 50.000%	5.50	-	100.000%
Camouflage / MindOverMatter 25.59M states	77.041%	90.000%	yes	no	yes	50.000%	90.000% / 100.000% / 50.000%	5.50	-	100.000%
Hunting / MindOverMatter 24.66M states	77.041%	90.000%	yes	no	yes	50.000%	90.000% / 100.000% / 50.000%	5.50	-	100.000%
SixthSense / MindShield 25.20M states	77.041%	90.000%	yes	no	yes	50.000%	90.000% / 100.000% / 50.000%	5.50	-	100.000%
Camouflage / MindShield 20.16M states	77.041%	90.000%	yes	no	yes	50.000%	90.000% / 100.000% / 50.000%	4.00	-	100.000%
Hunting / MindShield 24.39M states	77.041%	90.000%	yes	no	yes	50.000%	90.000% / 100.000% / 50.000%	5.50	-	100.000%
SixthSense / Tracking 25.65M states	77.041%	90.000%	yes	no	yes	50.000%	90.000% / 100.000% / 50.000%	5.50	-	100.000%
Camouflage / Tracking 35.87M states	77.041%	90.000%	yes	no	yes	50.000%	90.000% / 100.000% / 50.000%	4.00	-	100.000%
Hunting / Tracking 25.09M states	77.041%	90.000%	yes	no	yes	50.000%	90.000% / 100.000% / 50.000%	5.50	-	100.000%
Tracking / MindOverMatter 25.65M states	77.041%	90.000%	yes	no	yes	50.000%	90.000% / 100.000% / 50.000%	5.50	-	100.000%
Tracking / MindShield 25.38M states	77.041%	90.000%	yes	no	yes	50.000%	90.000% / 100.000% / 50.000%	5.50	-	100.000%
MindShield / MindOverMatter 25.16M states	75.466%	90.000%	yes	yes	yes	50.000%	90.000% / 100.000% / 50.000%	5.50	-	100.000%
SixthSense / Tracking 27.51M states	73.796%	89.492%	no	yes	yes	49.783%	89.670% / 99.944% / 50.000%	5.20	28.159%	100.000%
SixthSense / MindOverMatter 27.31M states	73.796%	89.492%	no	yes	yes	49.783%	89.670% / 99.944% / 50.000%	5.26	28.159%	100.000%
SixthSense / MindShield 26.85M states	73.796%	89.492%	no	yes	yes	49.783%	89.670% / 99.944% / 50.000%	5.26	28.159%	100.000%
Hunting / Tracking 27.16M states	73.796%	89.492%	no	yes	yes	49.783%	89.670% / 99.944% / 50.000%	5.35	28.159%	100.000%
Hunting / SixthSense 26.59M states	73.796%	89.492%	no	yes	yes	49.783%	89.670% / 99.944% / 50.000%	4.96	28.159%	100.000%
Hunting / MindOverMatter 26.59M states	73.796%	89.492%	no	yes	yes	49.783%	89.670% / 99.944% / 50.000%	4.96	28.159%	100.000%
Hunting / MindShield 26.14M states	73.796%	89.492%	no	yes	yes	49.783%	89.670% / 99.944% / 50.000%	4.96	28.159%	100.000%
Tracking / MindOverMatter 27.49M states	73.786%	89.492%	no	yes	yes	49.783%	89.670% / 99.944% / 50.000%	5.20	28.159%	100.000%
Tracking / MindShield 27.03M states	73.786%	89.492%	no	yes	yes	49.783%	89.670% / 99.944% / 50.000%	5.20	28.159%	100.000%
Camouflage / SixthSense 24.02M states	73.600%	89.255%	no	yes	yes	49.726%	89.670% / 99.944% / 50.000%	4.24	12.100%	100.000%
Camouflage / MindShield 23.65M states	73.600%	89.255%	no	yes	yes	49.726%	89.670% / 99.944% / 50.000%	4.24	12.100%	100.000%
Camouflage / Tracking 40.85M states	73.600%	89.255%	no	yes	yes	49.726%	89.670% / 99.944% / 50.000%	4.24	12.100%	100.000%
Camouflage / Hunting 23.25M states	73.600%	89.255%	no	yes	yes	49.726%	89.670% / 99.944% / 50.000%	4.11	12.100%	100.000%
Camouflage / MindOverMatter 27.48M states	73.599%	89.254%	no	yes	yes	49.726%	89.670% / 99.944% / 50.000%	5.13	12.100%	99.999%
Camouflage / SixthSense 10.08M states	71.090%	90.000%	yes	yes	no	50.000%	90.000% / 100.000% / 50.000%	4.00	-	-
Hunting / SixthSense 12.21M states	71.090%	90.000%	yes	yes	no	50.000%	90.000% / 100.000% / 50.000%	5.50	-	-
Camouflage / Hunting 9.66M states	71.090%	90.000%	yes	yes	no	50.000%	90.000% / 100.000% / 50.000%	4.00	-	-
SixthSense / MindOverMatter 12.62M states	71.090%	90.000%	yes	yes	no	50.000%	90.000% / 100.000% / 50.000%	5.50	-	-
Camouflage / MindOverMatter 12.68M states	71.090%	90.000%	yes	yes	no	50.000%	90.000% / 100.000% / 50.000%	5.50	-	-
Hunting / MindOverMatter 12.21M states	71.090%	90.000%	yes	yes	no	50.000%	90.000% / 100.000% / 50.000%	5.50	-	-
SixthSense / MindShield 12.55M states	71.090%	90.000%	yes	yes	no	50.000%	90.000% / 100.000% / 50.000%	5.50	-	-
Camouflage / MindShield 10.03M states	71.090%	90.000%	yes	yes	no	50.000%	90.000% / 100.000% / 50.000%	4.00	-	-
Hunting / MindShield 12.15M states	71.090%	90.000%	yes	yes	no	50.000%	90.000% / 100.000% / 50.000%	5.50	-	-
SixthSense / Tracking 12.70M states	71.090%	90.000%	yes	yes	no	50.000%	90.000% / 100.000% / 50.000%	5.50	-	-
Camouflage / Tracking 17.81M states	71.090%	90.000%	yes	yes	no	50.000%	90.000% / 100.000% / 50.000%	4.00	-	-
Hunting / Tracking 12.43M states	71.090%	90.000%	yes	yes	no	50.000%	90.000% / 100.000% / 50.000%	5.50	-	-
Tracking / MindOverMatter 12.71M states	71.090%	90.000%	yes	yes	no	50.000%	90.000% / 100.000% / 50.000%	5.50	-	-
Tracking / MindShield 12.64M states	71.090%	90.000%	yes	yes	no	50.000%	90.000% / 100.000% / 50.000%	5.50	-	-
MindShield / MindOverMatter 26.85M states	67.768%	89.492%	no	yes	yes	49.783%	89.670% / 99.944% / 50.000%	5.26	28.159%	100.000%
MindShield / MindOverMatter 25.20M states	67.219%	90.000%	yes	no	yes	50.000%	90.000% / 100.000% / 50.000%	5.50	-	100.000%
SixthSense / Tracking 13.71M states	59.315%	89.492%	no	yes	no	49.783%	89.670% / 99.944% / 50.000%	5.20	28.159%	-
SixthSense / MindOverMatter 13.61M states	59.315%	89.492%	no	yes	no	49.783%	89.670% / 99.944% / 50.000%	5.26	28.159%	-
SixthSense / MindShield 13.38M states	59.315%	89.492%	no	yes	no	49.783%	89.670% / 99.944% / 50.000%	5.26	28.159%	-
Hunting / Tracking 13.53M states	59.315%	89.492%	no	yes	no	49.783%	89.670% / 99.944% / 50.000%	5.35	28.159%	-
Hunting / SixthSense 13.25M states	59.315%	89.492%	no	yes	no	49.783%	89.670% / 99.944% / 50.000%	4.96	28.159%	-
Hunting / MindOverMatter 13.25M states	59.315%	89.492%	no	yes	no	49.783%	89.670% / 99.944% / 50.000%	4.96	28.159%	-
Hunting / MindShield 13.03M states	59.315%	89.492%	no	yes	no	49.783%	89.670% / 99.944% / 50.000%	4.96	28.159%	-
Tracking / MindOverMatter 13.70M states	59.274%	89.492%	no	yes	no	49.783%	89.670% / 99.944% / 50.000%	5.20	28.159%	-
Tracking / MindShield 13.47M states	59.274%	89.492%	no	yes	no	49.783%	89.670% / 99.944% / 50.000%	5.20	28.159%	-
Camouflage / SixthSense 11.96M states	59.149%	89.255%	no	yes	no	49.726%	89.670% / 99.944% / 50.000%	4.24	12.100%	-
Camouflage / MindShield 11.78M states	59.149%	89.255%	no	yes	no	49.726%	89.670% / 99.944% / 50.000%	4.24	12.100%	-
Camouflage / Tracking 20.38M states	59.149%	89.255%	no	yes	no	49.726%	89.670% / 99.944% / 50.000%	4.24	12.100%	-
Camouflage / Hunting 11.58M states	59.149%	89.255%	no	yes	no	49.726%	89.670% / 99.944% / 50.000%	4.11	12.100%	-
Camouflage / MindOverMatter 13.70M states	59.149%	89.254%	no	yes	no	49.726%	89.670% / 99.944% / 50.000%	5.13	12.100%	-
MindShield / MindOverMatter 12.55M states	58.898%	90.000%	yes	yes	no	50.000%	90.000% / 100.000% / 50.000%	5.50	-	-
MindShield / MindOverMatter 13.38M states	52.758%	89.492%	no	yes	no	49.783%	89.670% / 99.944% / 50.000%	5.26	28.159%	-
Camouflage / SixthSense 10.15M states	51.666%	90.000%	yes	no	no	50.000%	90.000% / 100.000% / 50.000%	4.00	-	-
Hunting / SixthSense 12.30M states	51.666%	90.000%	yes	no	no	50.000%	90.000% / 100.000% / 50.000%	5.50	-	-
Camouflage / Hunting 9.73M states	51.666%	90.000%	yes	no	no	50.000%	90.000% / 100.000% / 50.000%	4.00	-	-
SixthSense / MindOverMatter 12.71M states	51.666%	90.000%	yes	no	no	50.000%	90.000% / 100.000% / 50.000%	5.50	-	-
Camouflage / MindOverMatter 12.77M states	51.666%	90.000%	yes	no	no	50.000%	90.000% / 100.000% / 50.000%	5.50	-	-
Hunting / MindOverMatter 12.30M states	51.666%	90.000%	yes	no	no	50.000%	90.000% / 100.000% / 50.000%	5.50	-	-
SixthSense / MindShield 12.57M states	51.666%	90.000%	yes	no	no	50.000%	90.000% / 100.000% / 50.000%	5.50	-	-
Camouflage / MindShield 10.05M states	51.666%	90.000%	yes	no	no	50.000%	90.000% / 100.000% / 50.000%	4.00	-	-
Hunting / MindShield 12.17M states	51.666%	90.000%	yes	no	no	50.000%	90.000% / 100.000% / 50.000%	5.50	-	-
SixthSense / Tracking 12.80M states	51.666%	90.000%	yes	no	no	50.000%	90.000% / 100.000% / 50.000%	5.50	-	-
Camouflage / Tracking 17.91M states	51.666%	90.000%	yes	no	no	50.000%	90.000% / 100.000% / 50.000%	4.00	-	-
Hunting / Tracking 12.52M states	51.666%	90.000%	yes	no	no	50.000%	90.000% / 100.000% / 50.000%	5.50	-	-
Tracking / MindOverMatter 12.80M states	51.666%	90.000%	yes	no	no	50.000%	90.000% / 100.000% / 50.000%	5.50	-	-
Tracking / MindShield 12.66M states	51.666%	90.000%	yes	no	no	50.000%	90.000% / 100.000% / 50.000%	5.50	-	-
Hunting / SixthSense 26.81M states	49.128%	89.020%	no	no	yes	-	89.615% / 100.000% / 100.000%	4.37	56.616%	100.000%
SixthSense / MindOverMatter 27.53M states	49.128%	89.020%	no	no	yes	-	89.615% / 100.000% / 100.000%	4.16	56.616%	100.000%
Hunting / MindOverMatter 26.81M states	49.128%	89.020%	no	no	yes	-	89.615% / 100.000% / 100.000%	4.37	56.616%	100.000%
SixthSense / MindShield 27.08M states	49.128%	89.020%	no	no	yes	-	89.615% / 100.000% / 100.000%	4.16	56.616%	100.000%
Hunting / MindShield 26.36M states	49.128%	89.020%	no	no	yes	-	89.615% / 100.000% / 100.000%	4.37	56.616%	100.000%
SixthSense / Tracking 27.73M states	49.128%	89.020%	no	no	yes	-	89.615% / 100.000% / 100.000%	4.16	56.616%	100.000%
Hunting / Tracking 27.39M states	49.128%	89.020%	no	no	yes	-	89.615% / 100.000% / 100.000%	4.37	56.616%	100.000%
Tracking / MindOverMatter 27.71M states	49.022%	89.020%	no	no	yes	-	89.615% / 100.000% / 100.000%	4.16	56.616%	100.000%
Tracking / MindShield 27.26M states	49.022%	89.020%	no	no	yes	-	89.615% / 100.000% / 100.000%	4.16	56.616%	100.000%
Camouflage / SixthSense 24.24M states	48.690%	88.227%	no	no	yes	-	89.615% / 100.000% / 100.000%	4.33	24.482%	100.000%
Camouflage / Hunting 23.46M states	48.690%	88.227%	no	no	yes	-	89.615% / 100.000% / 100.000%	4.00	24.482%	100.000%
Camouflage / MindOverMatter 27.71M states	48.648%	88.227%	no	no	yes	-	89.615% / 100.000% / 100.000%	4.33	24.482%	100.000%
Camouflage / MindShield 23.87M states	48.648%	88.227%	no	no	yes	-	89.615% / 100.000% / 100.000%	4.33	24.482%	100.000%
Camouflage / Tracking 41.10M states	48.648%	88.227%	no	no	yes	-	89.615% / 100.000% / 100.000%	4.33	24.482%	100.000%
MindShield / MindOverMatter 12.57M states	43.359%	90.000%	yes	no	no	50.000%	90.000% / 100.000% / 50.000%	5.50	-	-
MindShield / MindOverMatter 27.08M states	43.045%	89.020%	no	no	yes	-	89.615% / 100.000% / 100.000%	4.16	56.616%	100.000%
SixthSense / Tracking 13.80M states	27.678%	82.577%	no	no	no	46.099%	83.753% / 98.213% / 50.000%	5.48	30.517%	-
SixthSense / MindOverMatter 13.71M states	27.678%	82.577%	no	no	no	46.099%	83.752% / 98.213% / 50.000%	5.48	30.517%	-
SixthSense / MindShield 13.48M states	27.678%	82.577%	no	no	no	46.099%	83.752% / 98.213% / 50.000%	5.48	30.517%	-
Hunting / Tracking 13.63M states	27.678%	82.577%	no	no	no	46.099%	83.752% / 98.213% / 50.000%	5.60	30.517%	-
Hunting / SixthSense 13.35M states	27.678%	82.576%	no	no	no	46.098%	83.752% / 98.213% / 50.000%	5.60	30.517%	-
Hunting / MindOverMatter 13.35M states	27.678%	82.576%	no	no	no	46.098%	83.752% / 98.213% / 50.000%	5.60	30.517%	-
Hunting / MindShield 13.13M states	27.678%	82.576%	no	no	no	46.098%	83.752% / 98.213% / 50.000%	5.60	30.517%	-
Tracking / MindOverMatter 13.80M states	27.559%	82.577%	no	no	no	46.099%	83.753% / 98.213% / 50.000%	5.48	30.517%	-
Tracking / MindShield 13.57M states	27.559%	82.577%	no	no	no	46.099%	83.753% / 98.213% / 50.000%	5.48	30.517%	-
Camouflage / SixthSense 12.06M states	27.091%	81.044%	no	no	no	45.569%	83.760% / 98.213% / 50.000%	4.21	13.326%	-
Camouflage / Hunting 11.67M states	27.090%	81.042%	no	no	no	45.567%	83.760% / 98.213% / 50.000%	4.02	13.326%	-
Camouflage / MindShield 11.87M states	27.016%	81.044%	no	no	no	45.569%	83.760% / 98.213% / 50.000%	4.21	13.326%	-
Camouflage / Tracking 20.49M states	27.016%	81.044%	no	no	no	45.569%	83.760% / 98.213% / 50.000%	4.21	13.326%	-
Camouflage / MindOverMatter 13.79M states	26.978%	80.955%	no	no	no	45.508%	83.727% / 98.213% / 50.000%	5.49	13.341%	-
MindShield / MindOverMatter 13.48M states	25.578%	82.577%	no	no	no	46.099%	83.752% / 98.213% / 50.000%	5.48	30.517%	-

]]> Solving a gamebook : Lone Wolf book 05, Shadow on the Sand mega solution (edit) https://hbtvl.banquise.net/posts/2022-02-02-GBS-chapter5.html 2022-02-02T00:00:00Z

Yay! I finally computed stats for the last book, depending on what you did in previous books. This walk-through still lacks the armor items, which have not yet been modeled, but it should be quite good in other aspects.

First of all, the following items that can be taken from other books are useless in this book:

the Fire Sphere
the Prism
the Blue Stone Triangle
the Onyx Medallion

And the following special items are to be avoided:

the Brass Whistle, the towel, as they are useless
the Black Cube, as it is a trap

However, the following previous items or experiences are indeed useful:

having fought an Elix in book 04
having the Sommerswerd, or the Silver Helm
having the potent (+4) strength potion

Random notes:

~~If you visit the shop at chapter 27, buy the 6Hp potion, it is worth it!~~ With the fixed shopping code, it is actually the 2Hp potion that is worth it! This is probably an artifact from the fact that I added an heuristic that says you can only use a potion if it won't be wasted (you have to have lost at least 6Hp before using the +6Hp potion for example).
Starting conditions are sensitive to starting gold, but only if you start with less than 4 gold, so that you can buy the potion at chapter 27. As the minimum starting gold is 10, it doesn't matter.
You don't have to buy the Black Sash!
Almost all routes include getting and curing Limbdeath.

Edit

2022-02-04 -- added the number of states, and potent strength potion option
2022-02-20 -- added body armor statistics
2022-02-21 -- fix strength potion usage (again)
2022-03-01 -- many bug fixes, checked money sensitivity

Mega solution!

This are all the routes you should take, depending on the following starting conditions:

Missing disc: if you have done all the previous books, you should have 9 disciplines, which means you lack one. This column shows the discipline that is missing. It means that the character has all the other remaining disciplines available! Note that clicking on that column will let you show a visualisation of the related solution!.
SS: did you start with the Sommerswerd or with a Sword?
SH: did you start with the Silver Helm?
+4: did you start with the Potent Strength Potion?
BA: did you start with the +4HP body armor?
EX: have you fought an Elix previously?

The the following data is available:

Win rate: what is the probability of winning this book?
Imprisoned: probability of going through a route that leads to prison.
Offer Oede: probability of offering Oede to the poor waxeler (nice!).
Limbdeath: does the route leads to Limbdeath?
Prism and Sash: probability of following a path that leads to the acquisition of the Sash or the Prism.
c27 +2HP: probability of buying the +2Hp potion at section 27.
Dhorgaan: final fight against Dhorgaan (instead of the Crypt Spawn and Dark Lord).
Keep SS: probability of finishing the book WITH the Sommerswerd, as you can lose it in the end in order to get the easier fight.

The starting conditions are:

start with:
- a Sword, or the Sommerswerd
- one potion of Laumspur
- a Shield
- 4 meals
max endurance: 20
combat skill: 10
all disciplines, except the listed discipline (weapon specialization is Sword)

Missing disc #states	SS	SH	+4	BA	EX	Win rate	Imprisoned	Offer Oede	Limbdeath	Prism	Sash	c27 +2HP	Dhorgaan	Keep SS
SixthSense 67.68M states	yes	yes	yes	yes	yes	88.521%	-	63.000%	100.000%	-	-	100.000%	63.000%	28.837%
Camouflage 102.70M states	yes	yes	yes	yes	yes	88.521%	-	63.000%	100.000%	-	-	100.000%	63.000%	28.837%
Hunting 92.70M states	yes	yes	yes	yes	yes	88.521%	-	63.000%	100.000%	-	-	100.000%	63.000%	28.837%
Tracking 89.09M states	yes	yes	yes	yes	yes	88.521%	-	63.000%	100.000%	0.010%	-	100.000%	63.000%	28.837%
SixthSense 67.68M states	yes	yes	yes	yes	no	88.521%	-	63.000%	100.000%	-	-	100.000%	63.000%	28.837%
Camouflage 102.70M states	yes	yes	yes	yes	no	88.521%	-	63.000%	100.000%	-	-	100.000%	63.000%	28.837%
Hunting 92.70M states	yes	yes	yes	yes	no	88.521%	-	63.000%	100.000%	-	-	100.000%	63.000%	28.837%
Tracking 89.09M states	yes	yes	yes	yes	no	88.521%	-	63.000%	100.000%	0.010%	-	100.000%	63.000%	28.837%
SixthSense 67.68M states	yes	no	yes	yes	yes	85.712%	-	63.000%	100.000%	-	-	100.000%	63.000%	26.625%
Camouflage 102.71M states	yes	no	yes	yes	yes	85.712%	-	63.000%	100.000%	-	-	100.000%	63.000%	26.625%
Hunting 92.72M states	yes	no	yes	yes	yes	85.712%	-	63.000%	100.000%	-	-	100.000%	63.000%	26.625%
Tracking 89.09M states	yes	no	yes	yes	yes	85.712%	-	63.000%	100.000%	0.010%	-	100.000%	63.000%	26.625%
SixthSense 67.68M states	yes	no	yes	yes	no	85.491%	-	62.837%	100.000%	-	-	100.000%	62.837%	26.625%
Camouflage 102.71M states	yes	no	yes	yes	no	85.491%	-	62.837%	100.000%	-	-	100.000%	62.837%	26.625%
Hunting 92.72M states	yes	no	yes	yes	no	85.491%	-	62.837%	100.000%	-	-	100.000%	62.837%	26.625%
Tracking 89.09M states	yes	no	yes	yes	no	85.491%	-	62.837%	100.000%	0.010%	-	100.000%	62.837%	26.625%
SixthSense 66.26M states	yes	yes	yes	no	yes	84.788%	-	62.997%	100.000%	-	-	-	89.996%	-
SixthSense 69.91M states	yes	yes	yes	no	yes	84.788%	-	62.997%	100.000%	-	-	-	89.996%	-
Camouflage 100.36M states	yes	yes	yes	no	yes	84.788%	-	62.997%	100.000%	-	-	-	89.996%	-
Hunting 90.36M states	yes	yes	yes	no	yes	84.788%	-	62.997%	100.000%	-	-	-	89.996%	-
Tracking 87.68M states	yes	yes	yes	no	yes	84.788%	-	62.997%	100.000%	0.010%	-	-	89.996%	-
SixthSense 66.26M states	yes	yes	yes	no	no	84.695%	-	62.928%	100.000%	-	-	-	89.898%	-
Camouflage 100.36M states	yes	yes	yes	no	no	84.695%	-	62.928%	100.000%	-	-	-	89.898%	-
Hunting 90.36M states	yes	yes	yes	no	no	84.695%	-	62.928%	100.000%	-	-	-	89.898%	-
Tracking 87.68M states	yes	yes	yes	no	no	84.695%	-	62.928%	100.000%	0.010%	-	-	89.898%	-
MindShield 83.71M states	yes	yes	yes	yes	yes	83.852%	-	63.000%	100.000%	-	-	-	90.000%	-
MindShield 83.71M states	yes	yes	yes	yes	no	83.851%	-	63.000%	100.000%	-	-	-	90.000%	-
SixthSense 59.63M states	no	yes	yes	yes	yes	83.033%	-	62.483%	100.000%	100.000%	-	100.000%	89.262%	-
Camouflage 91.00M states	no	yes	yes	yes	yes	83.033%	-	62.483%	100.000%	100.000%	-	100.000%	89.262%	-
Hunting 82.17M states	no	yes	yes	yes	yes	83.033%	-	62.483%	100.000%	100.000%	-	100.000%	89.262%	-
Tracking 78.37M states	no	yes	yes	yes	yes	83.026%	-	62.478%	100.000%	100.000%	-	100.000%	89.254%	-
SixthSense 59.63M states	no	yes	yes	yes	no	81.893%	-	61.625%	100.000%	100.000%	-	100.000%	88.036%	-
Camouflage 91.00M states	no	yes	yes	yes	no	81.893%	-	61.625%	100.000%	100.000%	-	100.000%	88.036%	-
Hunting 82.17M states	no	yes	yes	yes	no	81.893%	-	61.625%	100.000%	100.000%	-	100.000%	88.036%	-
Tracking 78.37M states	no	yes	yes	yes	no	81.878%	-	61.614%	100.000%	100.000%	-	100.000%	88.020%	-
SixthSense 66.26M states	yes	no	yes	no	yes	80.237%	-	62.928%	100.000%	-	-	100.000%	62.928%	22.538%
Camouflage 100.38M states	yes	no	yes	no	yes	80.237%	-	62.928%	100.000%	-	-	100.000%	62.928%	22.538%
Hunting 90.39M states	yes	no	yes	no	yes	80.237%	-	62.928%	100.000%	-	-	100.000%	62.928%	22.538%
Tracking 87.69M states	yes	no	yes	no	yes	80.237%	-	62.928%	100.000%	0.010%	-	100.000%	62.928%	22.538%
SixthSense 61.47M states	yes	yes	no	yes	yes	78.989%	-	63.000%	100.000%	-	-	100.000%	63.000%	22.612%
Camouflage 88.54M states	yes	yes	no	yes	yes	78.989%	-	63.000%	100.000%	-	-	100.000%	63.000%	22.612%
Hunting 83.50M states	yes	yes	no	yes	yes	78.989%	-	63.000%	100.000%	-	-	100.000%	63.000%	22.612%
Tracking 81.62M states	yes	yes	no	yes	yes	78.989%	-	63.000%	100.000%	0.010%	-	100.000%	63.000%	22.612%
SixthSense 61.47M states	yes	yes	no	yes	no	78.989%	-	63.000%	100.000%	-	-	100.000%	63.000%	22.612%
Camouflage 88.54M states	yes	yes	no	yes	no	78.989%	-	63.000%	100.000%	-	-	100.000%	63.000%	22.612%
Hunting 83.50M states	yes	yes	no	yes	no	78.989%	-	63.000%	100.000%	-	-	100.000%	63.000%	22.612%
Tracking 81.62M states	yes	yes	no	yes	no	78.989%	-	63.000%	100.000%	0.010%	-	100.000%	63.000%	22.612%
MindShield 74.30M states	no	yes	yes	yes	yes	78.636%	-	56.213%	100.000%	100.000%	-	100.000%	80.304%	-
SixthSense 66.26M states	yes	no	yes	no	no	78.505%	-	61.569%	100.000%	-	-	100.000%	61.569%	22.538%
Camouflage 100.38M states	yes	no	yes	no	no	78.505%	-	61.569%	100.000%	-	-	100.000%	61.569%	22.538%
Hunting 90.39M states	yes	no	yes	no	no	78.505%	-	61.569%	100.000%	-	-	100.000%	61.569%	22.538%
Tracking 87.69M states	yes	no	yes	no	no	78.505%	-	61.569%	100.000%	0.010%	-	100.000%	61.569%	22.538%
MindShield 83.71M states	yes	no	yes	yes	yes	74.784%	-	63.000%	100.000%	100.000%	-	-	90.000%	-
SixthSense 54.24M states	no	yes	no	yes	yes	74.701%	-	56.213%	100.000%	100.000%	-	100.000%	80.304%	-
Camouflage 78.29M states	no	yes	no	yes	yes	74.701%	-	56.213%	100.000%	100.000%	-	100.000%	80.304%	-
Hunting 73.85M states	no	yes	no	yes	yes	74.701%	-	56.213%	100.000%	100.000%	-	100.000%	80.304%	-
Tracking 71.88M states	no	yes	no	yes	yes	74.663%	-	56.184%	100.000%	100.000%	-	100.000%	80.263%	-
MindShield 83.71M states	yes	no	yes	yes	no	74.592%	-	62.837%	100.000%	100.000%	-	-	89.768%	-
MindShield 82.30M states	yes	yes	yes	no	yes	73.663%	-	62.997%	100.000%	-	-	-	89.996%	-
MindShield 82.30M states	yes	yes	yes	no	no	73.582%	-	62.928%	100.000%	-	-	-	89.898%	-
SixthSense 58.36M states	no	yes	yes	no	yes	73.578%	-	60.259%	100.000%	100.000%	-	100.000%	86.084%	-
Camouflage 88.79M states	no	yes	yes	no	yes	73.578%	-	60.259%	100.000%	100.000%	-	100.000%	86.084%	-
Hunting 79.97M states	no	yes	yes	no	yes	73.578%	-	60.259%	100.000%	100.000%	-	100.000%	86.084%	-
Tracking 77.11M states	no	yes	yes	no	yes	73.571%	-	60.253%	100.000%	100.000%	-	100.000%	86.076%	-
MindOverMatter 68.83M states	yes	yes	yes	yes	yes	73.068%	20.000%	54.480%	80.000%	100.000%	-	100.000%	56.228%	27.949%
MindOverMatter 68.83M states	yes	yes	yes	yes	no	73.068%	20.000%	54.480%	80.000%	100.000%	-	100.000%	56.228%	27.949%
MindShield 74.30M states	no	yes	yes	yes	no	72.840%	-	61.625%	100.000%	100.000%	-	100.000%	88.036%	-
SixthSense 58.36M states	no	yes	yes	no	no	71.062%	-	58.198%	100.000%	100.000%	-	100.000%	83.140%	-
Camouflage 88.79M states	no	yes	yes	no	no	71.062%	-	58.198%	100.000%	100.000%	-	100.000%	83.140%	-
Hunting 79.97M states	no	yes	yes	no	no	71.062%	-	58.198%	100.000%	100.000%	-	100.000%	83.140%	-
Tracking 77.11M states	no	yes	yes	no	no	71.038%	-	58.179%	100.000%	100.000%	-	100.000%	83.113%	-
MindOverMatter 68.83M states	yes	no	yes	yes	yes	70.820%	20.000%	54.480%	80.000%	100.000%	-	100.000%	56.228%	25.779%
MindOverMatter 59.31M states	no	yes	yes	yes	yes	70.710%	20.000%	57.747%	80.000%	100.000%	-	100.000%	82.496%	-
MindOverMatter 68.83M states	yes	no	yes	yes	no	70.644%	20.000%	54.350%	80.000%	100.000%	-	100.000%	56.098%	25.777%
SixthSense 60.76M states	yes	yes	no	no	yes	70.307%	-	62.997%	100.000%	-	-	100.000%	62.997%	18.654%
Camouflage 87.37M states	yes	yes	no	no	yes	70.307%	-	62.997%	100.000%	-	-	100.000%	62.997%	18.654%
Hunting 82.33M states	yes	yes	no	no	yes	70.307%	-	62.997%	100.000%	-	-	100.000%	62.997%	18.654%
Tracking 80.92M states	yes	yes	no	no	yes	70.307%	-	62.997%	100.000%	0.010%	-	100.000%	62.997%	18.654%
SixthSense 60.76M states	yes	yes	no	no	no	70.230%	-	62.928%	100.000%	-	-	100.000%	62.928%	18.654%
Camouflage 87.37M states	yes	yes	no	no	no	70.230%	-	62.928%	100.000%	-	-	100.000%	62.928%	18.654%
Hunting 82.33M states	yes	yes	no	no	no	70.230%	-	62.928%	100.000%	-	-	100.000%	62.928%	18.654%
Tracking 80.92M states	yes	yes	no	no	no	70.230%	-	62.928%	100.000%	0.010%	-	100.000%	62.928%	18.654%
MindOverMatter 67.41M states	yes	yes	yes	no	yes	70.082%	20.000%	54.478%	80.000%	100.000%	-	-	77.825%	-
MindOverMatter 67.41M states	yes	yes	yes	no	no	70.007%	20.000%	54.423%	80.000%	100.000%	-	-	77.747%	-
MindOverMatter 59.31M states	no	yes	yes	yes	no	69.799%	20.000%	57.062%	80.000%	100.000%	-	100.000%	81.518%	-
MindShield 68.11M states	no	yes	no	yes	yes	66.443%	-	56.213%	100.000%	100.000%	-	100.000%	80.304%	-
MindOverMatter 67.41M states	yes	no	yes	no	yes	66.441%	20.000%	54.422%	80.000%	100.000%	-	100.000%	56.171%	21.775%
MindOverMatter 62.49M states	yes	yes	no	yes	yes	65.442%	20.000%	54.480%	80.000%	100.000%	-	100.000%	56.228%	21.834%
MindOverMatter 62.49M states	yes	yes	no	yes	no	65.442%	20.000%	54.480%	80.000%	100.000%	-	100.000%	56.228%	21.834%
MindShield 73.04M states	no	yes	yes	no	yes	65.433%	-	49.615%	100.000%	100.000%	-	100.000%	70.879%	-
MindOverMatter 67.41M states	yes	no	yes	no	no	65.055%	20.000%	53.335%	80.000%	100.000%	-	100.000%	55.083%	21.759%
MindOverMatter 53.88M states	no	yes	no	yes	yes	64.048%	20.000%	52.734%	80.000%	100.000%	-	100.000%	75.335%	-
MindShield 82.30M states	yes	no	yes	no	yes	63.286%	-	62.928%	100.000%	-	-	100.000%	89.897%	-
MindOverMatter 58.03M states	no	yes	yes	no	yes	63.139%	20.000%	55.963%	80.000%	100.000%	-	100.000%	79.947%	-
MindShield 82.30M states	yes	no	yes	no	no	61.920%	-	61.569%	100.000%	-	-	100.000%	87.956%	-
MindShield 76.75M states	yes	yes	no	yes	yes	61.290%	-	63.000%	100.000%	-	-	-	90.000%	-
MindShield 76.75M states	yes	yes	no	yes	no	61.289%	-	63.000%	100.000%	-	-	-	90.000%	-
MindOverMatter 58.03M states	no	yes	yes	no	no	61.120%	20.000%	54.310%	80.000%	100.000%	-	100.000%	77.585%	-
SixthSense 59.62M states	no	no	yes	yes	yes	60.623%	-	59.090%	100.000%	100.000%	-	100.000%	84.414%	-
Hunting 82.16M states	no	no	yes	yes	yes	60.623%	-	59.090%	100.000%	100.000%	-	100.000%	84.414%	-
Camouflage 90.98M states	no	no	yes	yes	yes	60.607%	-	59.074%	100.000%	100.000%	-	100.000%	84.391%	-
SixthSense 53.61M states	no	yes	no	no	yes	60.582%	-	49.615%	100.000%	100.000%	-	100.000%	70.879%	-
Camouflage 77.19M states	no	yes	no	no	yes	60.582%	-	49.615%	100.000%	100.000%	-	100.000%	70.879%	-
Hunting 72.75M states	no	yes	no	no	yes	60.582%	-	49.615%	100.000%	100.000%	-	100.000%	70.879%	-
Tracking 78.36M states	no	no	yes	yes	yes	60.575%	-	59.043%	100.000%	100.000%	-	100.000%	84.347%	-
Tracking 71.25M states	no	yes	no	no	yes	60.527%	-	49.571%	100.000%	100.000%	-	100.000%	70.815%	-
MindOverMatter 61.78M states	yes	yes	no	no	yes	58.497%	20.000%	54.478%	80.000%	100.000%	-	100.000%	56.226%	17.936%
MindOverMatter 61.78M states	yes	yes	no	no	no	58.435%	20.000%	54.423%	80.000%	100.000%	-	100.000%	56.171%	17.936%
SixthSense 54.24M states	no	yes	no	yes	no	57.931%	-	43.594%	100.000%	100.000%	-	100.000%	62.277%	-
Camouflage 78.29M states	no	yes	no	yes	no	57.931%	-	43.594%	100.000%	100.000%	-	100.000%	62.277%	-
Hunting 73.85M states	no	yes	no	yes	no	57.931%	-	43.594%	100.000%	100.000%	-	100.000%	62.277%	-
Tracking 71.88M states	no	yes	no	yes	no	57.878%	-	43.554%	100.000%	100.000%	-	100.000%	62.220%	-
SixthSense 61.46M states	yes	no	no	yes	yes	57.481%	-	63.000%	100.000%	-	-	100.000%	63.000%	18.364%
Camouflage 88.54M states	yes	no	no	yes	yes	57.481%	-	63.000%	100.000%	-	-	100.000%	63.000%	18.364%
Hunting 83.50M states	yes	no	no	yes	yes	57.481%	-	63.000%	100.000%	-	-	100.000%	63.000%	18.364%
Tracking 81.62M states	yes	no	no	yes	yes	57.481%	-	63.000%	100.000%	0.010%	-	100.000%	63.000%	18.364%
SixthSense 61.46M states	yes	no	no	yes	no	57.333%	-	62.837%	100.000%	-	-	100.000%	62.837%	18.364%
Camouflage 88.54M states	yes	no	no	yes	no	57.333%	-	62.837%	100.000%	-	-	100.000%	62.837%	18.364%
Hunting 83.50M states	yes	no	no	yes	no	57.333%	-	62.837%	100.000%	-	-	100.000%	62.837%	18.364%
Tracking 81.62M states	yes	no	no	yes	no	57.333%	-	62.837%	100.000%	0.010%	-	100.000%	62.837%	18.364%
MindShield 74.29M states	no	no	yes	yes	yes	56.336%	-	41.541%	100.000%	100.000%	-	100.000%	59.345%	-
MindShield 73.04M states	no	yes	yes	no	no	55.686%	-	58.198%	100.000%	100.000%	-	100.000%	83.140%	-
SixthSense 59.62M states	no	no	yes	yes	no	55.187%	-	53.791%	100.000%	100.000%	-	100.000%	76.844%	-
Hunting 82.16M states	no	no	yes	yes	no	55.187%	-	53.791%	100.000%	100.000%	-	100.000%	76.844%	-
Camouflage 90.98M states	no	no	yes	yes	no	55.139%	-	53.744%	100.000%	100.000%	-	100.000%	76.778%	-
Tracking 78.36M states	no	no	yes	yes	no	55.068%	-	53.675%	100.000%	100.000%	-	100.000%	76.679%	-
MindOverMatter 59.30M states	no	no	yes	yes	yes	52.748%	20.000%	55.000%	80.000%	100.000%	-	100.000%	78.572%	-
MindOverMatter 53.24M states	no	yes	no	no	yes	52.724%	20.000%	47.433%	80.000%	100.000%	-	100.000%	67.762%	-
MindShield 68.11M states	no	yes	no	yes	no	51.527%	-	43.594%	100.000%	100.000%	-	100.000%	62.277%	-
MindOverMatter 53.88M states	no	yes	no	yes	no	50.631%	20.000%	42.638%	80.000%	100.000%	-	100.000%	60.911%	-
MindShield 76.04M states	yes	yes	no	no	yes	48.627%	-	62.997%	100.000%	-	-	-	89.996%	-
MindShield 76.04M states	yes	yes	no	no	no	48.573%	-	62.928%	100.000%	-	-	-	89.898%	-
MindOverMatter 59.30M states	no	no	yes	yes	no	48.354%	20.000%	50.717%	80.000%	100.000%	-	100.000%	72.453%	-
MindOverMatter 62.49M states	yes	no	no	yes	yes	48.236%	20.000%	54.480%	80.000%	100.000%	-	100.000%	56.228%	17.507%
MindOverMatter 62.49M states	yes	no	no	yes	no	48.117%	20.000%	54.350%	80.000%	100.000%	-	100.000%	56.098%	17.505%
MindShield 67.47M states	no	yes	no	no	yes	47.473%	-	49.615%	100.000%	100.000%	-	100.000%	70.879%	-
SixthSense 58.36M states	no	no	yes	no	yes	43.776%	-	31.972%	100.000%	100.000%	-	100.000%	45.674%	-
Camouflage 88.78M states	no	no	yes	no	yes	43.776%	-	31.972%	100.000%	100.000%	-	100.000%	45.674%	-
Hunting 79.96M states	no	no	yes	no	yes	43.776%	-	31.972%	100.000%	100.000%	-	100.000%	45.674%	-
Tracking 77.11M states	no	no	yes	no	yes	43.521%	-	31.786%	100.000%	100.000%	-	100.000%	45.409%	-
SixthSense 53.61M states	no	yes	no	no	no	43.472%	-	35.603%	100.000%	100.000%	-	100.000%	50.861%	-
Camouflage 77.19M states	no	yes	no	no	no	43.472%	-	35.603%	100.000%	100.000%	-	100.000%	50.861%	-
Hunting 72.75M states	no	yes	no	no	no	43.472%	-	35.603%	100.000%	100.000%	-	100.000%	50.861%	-
Tracking 71.25M states	no	yes	no	no	no	43.417%	-	35.558%	100.000%	100.000%	-	100.000%	50.797%	-
SixthSense 54.23M states	no	no	no	yes	yes	42.704%	-	41.624%	100.000%	100.000%	-	100.000%	59.462%	-
Hunting 73.83M states	no	no	no	yes	yes	42.704%	-	41.624%	100.000%	100.000%	-	100.000%	59.462%	-
Camouflage 78.28M states	no	no	no	yes	yes	42.619%	-	41.541%	100.000%	100.000%	-	100.000%	59.345%	-
Tracking 71.87M states	no	no	no	yes	yes	42.551%	-	41.475%	100.000%	100.000%	-	100.000%	59.249%	-
MindShield 73.03M states	no	no	yes	no	yes	39.210%	-	31.972%	100.000%	100.000%	-	100.000%	45.674%	-
MindOverMatter 53.24M states	no	yes	no	no	no	39.038%	20.000%	36.225%	80.000%	100.000%	-	100.000%	51.750%	-
MindOverMatter 58.03M states	no	no	yes	no	yes	39.001%	20.000%	33.117%	80.000%	100.000%	-	100.000%	47.310%	-
SixthSense 60.76M states	yes	no	no	no	yes	38.862%	-	62.928%	100.000%	-	-	100.000%	62.928%	14.679%
Camouflage 87.38M states	yes	no	no	no	yes	38.862%	-	62.928%	100.000%	-	-	100.000%	62.928%	14.679%
Hunting 82.35M states	yes	no	no	no	yes	38.862%	-	62.928%	100.000%	-	-	100.000%	62.928%	14.679%
Tracking 80.92M states	yes	no	no	no	yes	38.862%	-	62.928%	100.000%	0.010%	-	100.000%	62.928%	14.679%
MindOverMatter 53.87M states	no	no	no	yes	yes	38.324%	20.000%	40.941%	80.000%	100.000%	-	100.000%	58.488%	-
MindShield 74.29M states	no	no	yes	yes	no	38.150%	-	53.744%	100.000%	100.000%	-	100.000%	76.778%	-
SixthSense 60.76M states	yes	no	no	no	no	38.023%	-	61.569%	100.000%	-	-	100.000%	61.569%	14.679%
Camouflage 87.38M states	yes	no	no	no	no	38.023%	-	61.569%	100.000%	-	-	100.000%	61.569%	14.679%
Hunting 82.35M states	yes	no	no	no	no	38.023%	-	61.569%	100.000%	-	-	100.000%	61.569%	14.679%
Tracking 80.92M states	yes	no	no	no	no	38.023%	-	61.569%	100.000%	0.010%	-	100.000%	61.569%	14.679%
SixthSense 58.36M states	no	no	yes	no	no	35.059%	-	44.598%	100.000%	100.000%	-	100.000%	63.711%	-
Camouflage 88.78M states	no	no	yes	no	no	35.059%	-	44.598%	100.000%	100.000%	-	100.000%	63.711%	-
Hunting 79.96M states	no	no	yes	no	no	35.059%	-	44.598%	100.000%	100.000%	-	100.000%	63.711%	-
Tracking 77.11M states	no	no	yes	no	no	34.921%	-	44.423%	100.000%	100.000%	-	100.000%	63.462%	-
MindShield 67.47M states	no	yes	no	no	no	34.066%	-	35.603%	100.000%	100.000%	-	100.000%	50.861%	-
MindShield 76.74M states	yes	no	no	yes	yes	33.578%	-	63.000%	100.000%	100.000%	-	-	90.000%	-
MindShield 76.74M states	yes	no	no	yes	no	33.491%	-	62.837%	100.000%	100.000%	-	-	89.768%	-
MindOverMatter 61.78M states	yes	no	no	no	yes	33.341%	20.000%	54.422%	80.000%	100.000%	-	100.000%	56.171%	13.688%
MindOverMatter 61.78M states	yes	no	no	no	no	32.669%	20.000%	53.335%	80.000%	100.000%	-	100.000%	55.083%	13.667%
MindOverMatter 58.03M states	no	no	yes	no	no	32.136%	20.000%	43.194%	80.000%	100.000%	-	100.000%	61.705%	-
MindShield 68.10M states	no	no	no	yes	yes	29.488%	-	41.541%	100.000%	100.000%	-	100.000%	59.345%	-
SixthSense 54.23M states	no	no	no	yes	no	25.662%	-	25.013%	100.000%	100.000%	-	100.000%	35.733%	-
Hunting 73.83M states	no	no	no	yes	no	25.662%	-	25.013%	100.000%	100.000%	-	100.000%	35.733%	-
Camouflage 78.28M states	no	no	no	yes	no	25.562%	-	24.916%	100.000%	100.000%	-	100.000%	35.594%	-
Tracking 71.87M states	no	no	no	yes	no	25.525%	-	24.880%	100.000%	100.000%	-	100.000%	35.542%	-
SixthSense 53.60M states	no	no	no	no	yes	25.133%	-	31.972%	100.000%	100.000%	-	100.000%	45.674%	-
Camouflage 77.18M states	no	no	no	no	yes	25.133%	-	31.972%	100.000%	100.000%	-	100.000%	45.674%	-
Hunting 72.73M states	no	no	no	no	yes	25.133%	-	31.972%	100.000%	100.000%	-	100.000%	45.674%	-
Tracking 71.25M states	no	no	no	no	yes	24.987%	-	31.786%	100.000%	100.000%	-	100.000%	45.409%	-
MindOverMatter 53.87M states	no	no	no	yes	no	24.690%	20.000%	27.651%	80.000%	100.000%	-	100.000%	39.502%	-
MindOverMatter 53.23M states	no	no	no	no	yes	24.215%	20.000%	33.117%	80.000%	100.000%	-	100.000%	47.310%	-
Hunting 72.73M states	no	no	no	no	no	22.146%	100.000%	40.136%	-	-	-	-	57.337%	-
Tracking 71.25M states	no	no	no	no	no	22.146%	100.000%	40.136%	-	0.898%	-	-	57.337%	-
SixthSense 53.60M states	no	no	no	no	no	22.121%	100.000%	40.091%	-	-	-	-	57.273%	-
MindShield 73.03M states	no	no	yes	no	no	21.552%	-	44.598%	100.000%	100.000%	-	100.000%	63.711%	-
MindOverMatter 53.23M states	no	no	no	no	no	21.408%	100.000%	38.799%	-	100.000%	-	-	55.427%	-
MindShield 68.10M states	no	no	no	yes	no	17.686%	-	24.916%	100.000%	100.000%	-	100.000%	35.594%	-
MindShield 76.04M states	yes	no	no	no	yes	17.326%	-	62.928%	100.000%	-	-	100.000%	89.897%	-
Camouflage 77.18M states	no	no	no	no	no	16.958%	100.000%	32.183%	-	11.415%	-	-	45.976%	-
MindShield 76.04M states	yes	no	no	no	no	16.952%	-	61.569%	100.000%	-	-	100.000%	87.956%	-
MindShield 67.47M states	no	no	no	no	yes	15.451%	-	31.972%	100.000%	100.000%	-	100.000%	45.674%	-
MindShield 67.47M states	no	no	no	no	no	7.693%	-	15.919%	100.000%	100.000%	-	100.000%	22.741%	-

Concluding remarks

If you have to choose between the Silver Helm or the Sommerswerd, it is better to have the Helm. The reason is that in the end, you can avoid the harder fight more easily, and still get a good magic weapon, that gives a +5 bonus. With the Helm, it is equivalent to what the Sommerswerd provides (+8).

Edit 2022/03/01: depending on the situation, the Sommerswerd is actually stronger, as it let you choose an easier fight. The potent strength potion can be used for several fights, depending on the configuration. Please look at the specific solution visualisation for details!

Also, with that new code, purchases are actually properly tracked, so you can see that it is sometimes optimal to purchase the +2HP Potion at chapter 154.

In terms of technical progress, the code has been optimized a bit, and the route extractor rewritten in Rust. I might be able to convert the whole solver to Rust to increase solving speed, who knows?

The resulting data is quite large, as it weights about 60GB in compressed form. This makes the creation of an online solution explorer quite difficult too. It might be possible to write a compressed solution by creating decision rules instead of storing all the states, but that is for another time :)

]]> Solving a gamebook : can we solve five books at once? The 2022 update. https://hbtvl.banquise.net/posts/2022-01-15-GBS-newyearupdate.html 2022-01-15T00:00:00Z

The Lone Wolf series starts with a five books arc. I now believe it might be possible to solve all of them at once, and compute the ultimate walkthrough!

The reason why it might make sense to solve them all at once instead of in isolation is that there are elements that are kept between books, namely selected disciplines and items. This means that some high-risk items might not be worth it in a book, but very much worth it in a subsequent book.

Since last time, the following items were improved:

I finally implemented the healing discipline properly ... at least kind of properly. This changes everything with regards to balancing, which means that all previous statistics are to be discarded :)
I also implemented book 05. This book has several specific mechanisms. There is poisoning, disease, and equipment confiscation. The latter is especially costly with regards to performance, as a copy of the inventory must be stored ...
... but now that I can't fit the state in 96 bits, there is more space for extra items :)

Notes on solving Book 05

Note that everything here is to be considered with suspicion, as there might be bugs lurking in the way I encoded the books (I just found one this morning where I forgot to encode a shop).

Book 05 has 400 chapters instead of 350, and is noticeably more complex than the others, which makes computation much more taxing. As a result, I decided to compute the stats only for hard mode Lone Wolf, that is with a weak (10 skill, 20 endurance) character. This is offset by the fact that you wield the Sommerswerd, but well, it is still hard :)

The other consequence is that I had to buy an extra 64GB of RAM for some computations to eventually complete. Of course, I later optimized it and everything fits in 20GB, but at least I can run more jobs in parallel :/

There is an interesting part in this book, where you can get imprisoned, and your equipment is seized. Later, there is the opportunity to retrieve it.

It turns out that it is never worth retrieving it!

if you wield the Sommerswerd, it makes more sense to avoid this episode entirely, as the weapon is too valuable to be lost, and strong enough that you can withstand the harder route ;
if you do not wield it, then the equipment you find is plenty sufficient, and it is not worth it trying to get your previous stuff back.

Contrast this with this walkthrough, where it recommends recovering the equipment! Also in this walkthrough it recommends items from previous books, but I did not find they made any difference :) I still have to compute if having previously fought an Elix , or having received a Crystal Star Pendant makes a difference!

But perhaps for a stronger character, or a character with only 5 disciplines, it is very much worth it.

Notes on healing

Healing changed everything. In book 05, section 98, if you go to chapter 118, you have 1/10 chance of dying, and 9/10 chance of ending up at the same place you would be if you chose the other path. It seems like it is strictly better to avoid this path, but my solver sometimes chose the "wrong" path anyways!

It turns out that there is a fight just after, and the "wrong" path, being longer, let you heal some more, which makes a difference for this fight!

Next time

Hopefully, next time I will have a solution for the whole five books :) I still have to model book 03 (book 04 is done but needs debugging), and to write a way to stitch them together. I am however confident this is computable, so let's check it again next time!

Bonus

click here for a solution visualization of the following starting state:

Max endurance 20, skill 10
Sommerswerd, 1 potion of Laumspur, shield, 2 meals

]]> Solving a gamebook : revenge of the Haskell solver https://hbtvl.banquise.net/posts/2021-12-01-GBS-revenge-of-haskell.html 2021-12-01T00:00:00Z

Last time I rewrote the Haskell solver in Rust, and it was so much faster that it became possible to solve the second book from zero. That was very satisfying, but I had misgivings.

First, the Haskell version was better tested, and supported more features (such as the hunting discipline). Secondly, for Haskell I used a standard memoizer, whereas in Rust I wrote my own memoization system. I decided to do the same with the Haskell version, and it turned out it runs sufficiently fast to be usable! It is also much more confortable to add features in such a high level language.

I also tweaked the book description a bit to remove graph loops (unfortunately, this is currently a manual process), and introduced a topological sort approach to state exploration that prevents exploration loops.

To cut things short, here are updated numbers, that have been checked more toroughly than in the previous post, and should be much more accurate. In particular, a lot of bugs have been found in the chapters descriptions, and the combat escape mechanism did not work. This means that win probabilities are now much better, as fights can actually be avoided!

Starting conditions

In the following examples, the starting conditions are:

a short sword (yes, even when having other weapon disciplines)
the Seal of Hammerdal
a shield
15 gold coins

In the previous instalments, I chose the pair of meals over the shield, but the shield is in fact the most effective item to choose at the beginning of the story. All other items would be quickly lost, but for some reason the shield can be kept.

It would be better to choose a weapon that matches selected weapon specialties, but this will be for another episode :)

Number of states

First of all, what is interesting is how many game states need to be traversed in order to find the solution, depending on the starting disciplines. The time it takes to compute a solution is generally proportional to the amount of states that must be searched. Here is a quick chart that should be illustrative:

	Tracking	Spear	ShortSword	Healing	MindBlast	MindShield	AnimalKinship	Camouflage	MindOverMatter	SixthSense	Hunting
Tracking	2985557	2625361	2711710	2902417	2985249	3099854	3150696	3204877	3198669	3198145	9314661
Spear	2625361	5236316	5819251	5060427	5230867	5323102	5519090	5512594	6073541	6171112	12612684
ShortSword	2711710	5819251	5297698	5123241	5293613	5420292	5561858	5567158	5860453	5942910	13092512
Healing	2902417	5060427	5123241	5514335	5510899	5705421	5812238	5861880	5862985	5917761	14926638
MindBlast	2985249	5230867	5293613	5510899	5654395	5845481	5957920	5983669	6102573	6174766	15263336
MindShield	3099854	5323102	5420292	5705421	5845481	5847823	6142386	6188875	6303044	6456377	15507739
AnimalKinship	3150696	5519090	5561858	5812238	5957920	6142386	5954460	6300863	6405337	6438119	15776351
Camouflage	3204877	5512594	5567158	5861880	5983669	6188875	6300863	5986011	6460004	6530189	16144572
MindOverMatter	3198669	6073541	5860453	5862985	6102573	6303044	6405337	6460004	6105302	6177566	15883371
SixthSense	3198145	6171112	5942910	5917761	6174766	6456377	6438119	6530189	6177566	6177567	15826247
Hunting	9314661	12612684	13092512	14926638	15263336	15507739	15776351	16144572	15883371	15826247	15129259

It turns out hunting makes the number of states explode. I believe it is because it allows you to keep more meals, creating more decisions when the character does not have to hunt.

On the other hand, tracking does reduce the amount of states generated. The reason is that at chapter 231, you get to choose between two huge branches, but tracking forces you into a given path.

Success probability

Strong character

A strong character starts with maxed out endurance and combat skill (respectively 29 and 19).

	AnimalKinship	MindShield	Hunting	Spear	Healing	MindBlast	ShortSword	Camouflage	MindOverMatter	SixthSense	Tracking
AnimalKinship	89.29%	89.29%	89.55%	89.49%	89.56%	89.54%	89.54%	89.29%	89.29%	84.83%	89.29%
MindShield	89.29%	50.24%	78.73%	67.31%	64.28%	52.69%	52.20%	50.24%	50.24%	47.96%	49.34%
Hunting	89.55%	78.73%	42.43%	63.95%	55.95%	46.79%	44.86%	42.43%	42.43%	40.31%	42.43%
Spear	89.49%	67.31%	63.95%	29.24%	43.05%	31.47%	31.37%	29.24%	29.24%	27.89%	28.51%
Healing	89.56%	64.28%	55.95%	43.05%	24.54%	24.54%	24.34%	24.54%	24.54%	23.52%	24.18%
MindBlast	89.54%	52.69%	46.79%	31.47%	24.54%	17.26%	17.91%	17.26%	17.26%	16.42%	15.62%
ShortSword	89.54%	52.20%	44.86%	31.37%	24.34%	17.91%	16.79%	16.79%	16.79%	16.00%	15.62%
Camouflage	89.29%	50.24%	42.43%	29.24%	24.54%	17.26%	16.79%	15.89%	15.89%	15.11%	14.10%
MindOverMatter	89.29%	50.24%	42.43%	29.24%	24.54%	17.26%	16.79%	15.89%	15.89%	15.11%	14.10%
SixthSense	84.83%	47.96%	40.31%	27.89%	23.52%	16.42%	16.00%	15.11%	15.11%	15.11%	13.40%
Tracking	89.29%	49.34%	42.43%	28.51%	24.18%	15.62%	15.62%	14.10%	14.10%	13.40%	14.10%

Average character

A strong character starts with everage endurance and combat skill (respectively 25 and 15).

	AnimalKinship	MindShield	Hunting	Spear	Healing	MindBlast	ShortSword	Camouflage	MindOverMatter	Tracking	SixthSense
AnimalKinship	84.92%	84.92%	85.38%	84.78%	85.22%	87.09%	86.94%	84.92%	84.92%	84.53%	80.73%
MindShield	84.92%	6.91%	22.92%	16.40%	13.77%	7.42%	7.40%	6.91%	6.91%	6.91%	6.56%
Hunting	85.38%	22.92%	3.96%	11.10%	7.82%	4.25%	4.24%	3.96%	3.96%	3.96%	3.76%
Spear	84.78%	16.40%	11.10%	2.41%	5.28%	2.61%	2.56%	2.41%	2.41%	2.29%	2.29%
Healing	85.22%	13.77%	7.82%	5.28%	1.66%	1.70%	1.70%	1.66%	1.66%	1.66%	1.57%
MindBlast	87.09%	7.42%	4.25%	2.61%	1.70%	0.73%	0.85%	0.73%	0.73%	0.73%	0.69%
ShortSword	86.94%	7.40%	4.24%	2.56%	1.70%	0.85%	0.73%	0.73%	0.73%	0.73%	0.69%
Camouflage	84.92%	6.91%	3.96%	2.41%	1.66%	0.73%	0.73%	0.67%	0.67%	0.67%	0.64%
MindOverMatter	84.92%	6.91%	3.96%	2.41%	1.66%	0.73%	0.73%	0.67%	0.67%	0.67%	0.64%
Tracking	84.53%	6.91%	3.96%	2.29%	1.66%	0.73%	0.73%	0.67%	0.67%	0.67%	0.64%
SixthSense	80.73%	6.56%	3.76%	2.29%	1.57%	0.69%	0.69%	0.64%	0.64%	0.64%	0.64%

Effect of skill and endurance on outcome

This table displays the effect of starting skill and endurance on the winning probability, for the animal kinship and mindblast choice of disciplines.

	10	11	12	13	14	15	16	17	18	19
20	68.63%	71.42%	77.10%	78.11%	80.31%	81.80%	85.20%	85.65%	86.88%	87.04%
21	70.60%	73.48%	79.06%	80.04%	82.11%	83.46%	86.36%	86.70%	87.69%	87.82%
22	72.35%	75.27%	80.69%	81.61%	83.53%	84.73%	87.14%	87.39%	88.26%	88.34%
23	73.96%	76.87%	82.09%	82.93%	84.67%	85.72%	87.69%	87.87%	88.67%	88.72%
24	75.42%	78.26%	83.25%	84.01%	85.59%	86.47%	88.10%	88.21%	88.97%	89.01%
25	76.80%	79.53%	84.25%	84.92%	86.35%	87.09%	88.42%	88.49%	89.18%	89.21%
26	78.10%	80.69%	85.10%	85.68%	86.99%	87.59%	88.69%	88.74%	89.34%	89.35%
27	79.33%	81.76%	85.81%	86.31%	87.53%	88.01%	88.92%	88.96%	89.43%	89.45%
28	80.46%	82.71%	86.40%	86.82%	87.99%	88.36%	89.12%	89.14%	89.49%	89.50%
29	81.51%	83.57%	86.90%	87.25%	88.37%	88.66%	89.27%	89.29%	89.53%	89.54%

Discussion

As always, fair bit of warning : all those statistics might be completely wrong. They sure were previously, and while I reviewed the chapter descriptions toroughly, there still might be mistakes, especially in the decision heuristics.

As was well known, the animal kinship discipline is almost obligatory if one wishes to survive this book. If you do not have this discipline, you are forced through a route that involves acquiring the magic spear, which is a very dangerous fight. A descent substitute is the hunting and mindshield combination, but only for strong characters.

One huge surprise was that some disciplines decreases your probability of winning.

The first is sixth sense. This discipline is useful for a human, as it gives hints throughout the adventure, but not to a solver that already knowns the optimal path. Its drawback is observed in chapter 254 where it forces you thought a path that leads to chapter 183, where you have a 10% chance of instadeath.
The other low tier discipline is tracking, which is used in four chapters in order to give you a hint about where you should go next, and only opens a new path in chapter 334, but this chapter is not on the optimal solution path anyway. As previously discussed, it also forces you into a bad path, at chapter 231.

Chapter 200, and the choice of assassin

In chapter 200, you get to choose who you will attack. As noted in other walkthough, attacking Halvorc the merchant seems preferable as it is the easier fight. However, it also has a weak reward, so I wondered if it might make sense, sometimes, to have a harder fight in exchange for a better reward.

It turns out that this is indeed that case, but in a completely counter-intuitive way. It is sometimes better to attack the priest Parsion. I did a tiny bit of correlation, and it turns out that it is inversely correlated to the amount of meals one have (except for characters with the hunting discipline), and to the probability of actually winning that fight!

I suppose that the stronger characters do not need that reward to survive the rest of the adventure, which makes the harder fight only interesting for weaker characters.

For those interested, here are the raw correlation results between all tracked variables for average character with mindblast and animal kinship. Win chances and expected endurance are computed for the priest fight.

                    chapter  allmeals  winchance  expectedendurance  Laumspur      Meal
chapter            1.000000 -0.226461  -0.413484          -0.259957 -0.195328 -0.141111
allmeals          -0.226461  1.000000  -0.014969          -0.059701  0.640212  0.797655
winchance         -0.413484 -0.014969   1.000000           0.858334 -0.139976  0.090432
expectedendurance -0.259957 -0.059701   0.858334           1.000000 -0.183343  0.066315
Laumspur          -0.195328  0.640212  -0.139976          -0.183343  1.000000  0.047357
Meal              -0.141111  0.797655   0.090432           0.066315  0.047357  1.000000

Conclusion

I am much more confident in the current version numbers, but I will need to audit the choice heuristics to make sure nothing is wrong. The current software has a solution explorer, that let you run through the adventure and tells you your probability of winning at each choice, but it is still very crude. A browser version would be much nicer, but I can't really reuse all the Haskell machinery, as this project is GHC 9+ and GHCJs doesn't seem to be supported for this version :(

2021-12-07 : fixed a bug in the way combat escapes were handled. Should work properly now!

]]> Solving a gamebook : the Rust rewrite, teaser https://hbtvl.banquise.net/posts/2021-10-24-GBS-rust-rewrite.html 2021-10-24T00:00:00Z

This is a teaser post, for the ongoing work on solving the second book of the Lone Wolf series.

Last year I decided to rewrite the gamebook solver in Rust. It turned out to be much, much more performant, but also a lot more annoying to write. As a result, it does not yet work in all situations the Haskell version works, but it turns out I can finally solve book 02 in many situations!

What is done

The book is fully modeled, and can be solved in less than 2 minutes in most configurations on my computer. It can use an awful lot of memory though (around 6Gb in extreme cases).

What is to be done

Unfortunately, there are still a lot of things to work on:

publishing the source code for the Rust version ;
documenting and verifying that the heuristics I chose to reduce the search space are sound ;
extending the way the game state is searched to account for non-looping intersecting states (this is the cause of the 6th sense bug below) ;
as it is written in Rust, the code could be compiled to WASM, and an interactive solution explorer could be written very easily!

And in the long term, solving other books, or even solving books from other series.

Unfortunately, I am not that motivated to do that. So if you are actually interested in seeing any of that, please let me know!

Results

Weak character

Probability of winning, with skill 10, endurance 20, and the following pairs of disciplines:

	Camouflage	Hunting	Sixth sense	Tracking	Healing	Weaponskill: sword	Weaponskill: spear	Mind shield	Mind blast	Animal kinship	Mind over matter
Camouflage	-	0.00%	BUG	0.00%	0.00%	0.00%	0.00%	0.00%	0.00%	0.30%	0.00%
Hunting	0.00%	-	BUG	0.00%	0.00%	0.00%	0.00%	0.00%	0.00%	0.53%	0.00%
Sixth sense	BUG	BUG	-	BUG	BUG	BUG	BUG	BUG	BUG	BUG	BUG
Tracking	0.00%	0.00%	BUG	-	0.00%	0.00%	0.00%	0.00%	0.00%	0.14%	0.00%
Healing	0.00%	0.00%	BUG	0.00%	-	0.00%	0.00%	0.00%	0.00%	0.41%	0.00%
Weaponskill: sword	0.00%	0.00%	BUG	0.00%	0.00%	-	0.00%	0.00%	0.00%	3.75%	0.00%
Weaponskill: spear	0.00%	0.00%	BUG	0.00%	0.00%	0.00%	-	0.00%	0.00%	3.57%	0.00%
Mind shield	0.00%	0.00%	BUG	0.00%	0.00%	0.00%	0.00%	-	0.00%	0.30%	0.00%
Mind blast	0.00%	0.00%	BUG	0.00%	0.00%	0.00%	0.00%	0.00%	-	4.15%	0.00%
Animal kinship	0.30%	0.53%	BUG	0.14%	0.41%	3.75%	3.57%	0.30%	4.15%	-	0.30%
Mind over matter	0.00%	0.00%	BUG	0.00%	0.00%	0.00%	0.00%	0.00%	0.00%	0.30%	-

Average character

Probability of winning, with skill 15, endurance 25, and the following pairs of disciplines:

	Camouflage	Hunting	Sixth sense	Tracking	Healing	Weaponskill: sword	Weaponskill: spear	Mind shield	Mind blast	Animal kinship	Mind over matter
Camouflage	-	0.00%	BUG	0.00%	0.00%	0.00%	0.01%	0.03%	0.00%	46.73%	0.00%
Hunting	0.00%	-	BUG	0.00%	0.00%	0.00%	0.03%	0.05%	0.00%	53.61%	0.00%
Sixth sense	BUG	BUG	-	BUG	BUG	BUG	BUG	BUG	BUG	BUG	BUG
Tracking	0.00%	0.00%	BUG	-	0.00%	0.00%	0.00%	0.01%	0.00%	40.97%	0.00%
Healing	0.00%	0.00%	BUG	0.00%	-	0.00%	0.03%	0.05%	0.01%	50.86%	0.00%
Weaponskill: sword	0.00%	0.00%	BUG	0.00%	0.00%	-	0.02%	0.05%	0.00%	50.71%	0.00%
Weaponskill: spear	0.01%	0.03%	BUG	0.00%	0.03%	0.02%	-	0.22%	0.03%	49.83%	0.01%
Mind shield	0.03%	0.05%	BUG	0.01%	0.05%	0.05%	0.22%	-	0.07%	46.73%	0.03%
Mind blast	0.00%	0.00%	BUG	0.00%	0.01%	0.00%	0.03%	0.07%	-	67.80%	0.00%
Animal kinship	46.73%	53.61%	BUG	40.97%	50.86%	50.71%	49.83%	46.73%	67.80%	-	46.73%
Mind over matter	0.00%	0.00%	BUG	0.00%	0.00%	0.00%	0.01%	0.03%	0.00%	46.73%	-

Strong character

Probability of winning, with skill 19, endurance 29, and the following pairs of disciplines:

	Camouflage	Hunting	Sixth sense	Tracking	Healing	Weaponskill: sword	Weaponskill: spear	Mind shield	Mind blast	Animal kinship	Mind over matter
Camouflage	-	8.88%	BUG	3.11%	8.09%	5.77%	25.90%	24.61%	7.08%	86.28%	5.47%
Hunting	8.88%	-	BUG	8.88%	11.40%	9.97%	40.43%	34.63%	11.17%	87.43%	8.88%
Sixth sense	BUG	BUG	-	BUG	BUG	BUG	BUG	BUG	BUG	BUG	BUG
Tracking	3.11%	8.88%	BUG	-	4.73%	3.31%	17.61%	16.49%	4.32%	85.90%	3.11%
Healing	8.09%	11.40%	BUG	4.73%	-	8.43%	35.29%	32.13%	9.47%	87.24%	8.09%
Weaponskill: sword	5.77%	9.97%	BUG	3.31%	8.43%	-	26.13%	25.23%	7.17%	86.38%	5.77%
Weaponskill: spear	25.90%	40.43%	BUG	17.61%	35.29%	26.13%	-	59.92%	31.56%	86.48%	25.90%
Mind shield	24.61%	34.63%	BUG	16.49%	32.13%	25.23%	59.92%	-	29.69%	86.28%	24.61%
Mind blast	7.08%	11.17%	BUG	4.32%	9.47%	7.17%	31.56%	29.69%	-	89.07%	7.08%
Animal kinship	86.28%	87.43%	BUG	85.90%	87.24%	86.38%	86.48%	86.28%	89.07%	-	86.28%
Mind over matter	5.47%	8.88%	BUG	3.11%	8.09%	5.77%	25.90%	24.61%	7.08%	86.28%	-

Effect of skill/max endurance

Probability of winning, with animal kinship + mind blast, for various levels of endurance and skill.

Endurance:	20	21	22	23	24	25	26	27	28	29
sk: 10	4.15%	4.92%	5.69%	6.48%	7.34%	8.33%	9.44%	10.65%	11.87%	13.07%
sk: 11	13.40%	15.01%	16.65%	18.41%	20.31%	22.26%	24.16%	25.92%	27.55%	29.13%
sk: 12	23.04%	25.22%	27.22%	29.18%	31.21%	33.26%	35.21%	36.93%	38.39%	39.65%
sk: 13	35.68%	38.43%	40.76%	42.81%	44.79%	46.73%	48.51%	50.02%	51.22%	52.20%
sk: 14	51.46%	53.96%	56.02%	57.78%	59.35%	60.82%	62.13%	63.29%	64.29%	65.20%
sk: 15	58.71%	61.11%	63.12%	64.86%	66.38%	67.80%	69.16%	70.50%	71.74%	72.97%
sk: 16	67.25%	69.36%	71.16%	72.84%	74.39%	75.90%	77.37%	78.78%	80.06%	81.23%
sk: 17	74.26%	76.30%	78.05%	79.61%	81.08%	82.49%	83.73%	84.79%	85.62%	86.29%
sk: 18	79.88%	81.57%	82.99%	84.28%	85.39%	86.32%	87.04%	87.58%	87.98%	88.30%
sk: 19	83.24%	84.63%	85.82%	86.77%	87.47%	87.99%	88.37%	88.67%	88.90%	89.07%

]]> Site revamp https://hbtvl.banquise.net/posts/2020-12-05-site-revamp.html 2020-12-05T00:00:00Z

I just revamped this website, replacing Hakyll by a shake-based build system. The Dark Souls weapon optimizer is now built with the website, and most files are brotli-compressed.

Other than that, not much news :)

]]> New release of stateWriter, v0.2.9 https://hbtvl.banquise.net/posts/2017-08-26-new-release-stateWriter.html 2017-08-26T00:00:00Z

The stateWriter package exposes a monad that has a similar API to the familiar RWST transformer, but where the Writer instance is implemented like the state part. As a result, this implementation does not leak memory, and is noticeably faster.

Unfortunately, there was a huge bug in the implementation of listen, which is now fixed. All users are strongly encouraged to upgrade now.

This bug was found while proving that the transformer was sound. You might like reading the coq proof that proves compliance with the Functor, Applicative, Monad and State laws. Unfortunately, I could not find good laws for the Writer and Reader parts. An ad-hoc test for the following law found the bug:

lift act = fmap (\v -> (v,mempty)) (lift act)

I will try to add a few more theorems in later versions!

]]> A quick update on the gamebook solver project https://hbtvl.banquise.net/posts/2017-07-08-quickupdate.html 2017-07-08T00:00:00Z

As some of you might have noticed, it has been a couple of month since the last update. While the symbolic interpreter part is almost certainly stalled until the next SSTIC challenge(french), and I did not work much on the gamebook solver, a couple things have happened:

I had non-determinism issues with the solver logic, which turned to be a pretty serious bug. I had no clue what the problem was, but Cale found the exact point where it was triggered on IRC (#haskell is still awesome), I opened an issue on the unordered-containers tracker, and it should be solved in 8.2.1.
I have written a quick package that finds supperbubbles, based on this article. This will hopefully be instrumental in simplifying gamebook problems.

]]> Parametric types, and turning a concrete interpreter to a symbolic one without modifying it https://hbtvl.banquise.net/posts/2017-05-29-01-parametricmonad.html 2017-04-29T00:00:00Z

TL;DR;

By making the intermediate language interpreter parametric in the type of monad, I could write a concrete interpreter, and then turn it into a fully symbolic one without modifying it.

While I am quite proud of this trick, this will certainly not be released, as it is just a toy. But keep a eye on BinCAT, which should be released soon, and really looks promising (with an IDA plugin!).

Introduction

I just spend quite a few hours in the previous weeks toying with symbolic execution and automated constraint solving for compiled programs. That was a lot of fun, and I can't deny that throwing all the constraints to the SMT solver and getting the solution of a (very simple) crackme felt like magic.

I found an extremely cute way to implement symbolic execution by completely reusing the code of a concrete interpreter, which is described at the end of this post.

As an introduction, here is a toy example for symbolic execution, using the IL I developed:

Set (Rg EAX) 0
If (EReg EBX :== 0x12345678) [Set (Rg EAX) 1] []
If (EReg ECX :== 0xabcde)    [Set (Rg EAX) (EReg EAX :+ 1)] []

In this case, the EAX register will only be equal to 2 if EBX == 0x12345678 and ECX == 0xabcde when the program starts. I wrote a toy symbolic execution engine that converts such problems to a set of constraints that is then fed to an SMT solver.

How it works is approximately:

Decode a binary program into a set of instructions
Write a function that describe the semantics of each instruction, using an intermediate language (IL)
Write an interpreter for this IL, that is able to collect constraints as it runs through the program
Find the desired end-state of your program, along with the related constraints, and try to solve them!

Intermediate representation

In order to get started, I had to define an intermediate representation (IL), which is a language that is used to describe the semantics of individual instructions. A well known language is REIL, but I really don't like how it is designed, especially with regards to its imperative feeling.

For example, here is how they define mov ah, al:

00000000.00     AND         R_EAX:32,             ff:8,           V_00:8
00000000.01     AND         R_EAX:32,      ffff00ff:32,          V_01:32
00000000.02      OR           V_00:8,             0:32,          V_02:32
00000000.03     SHL          V_02:32,             8:32,          V_03:32
00000000.04      OR          V_01:32,          V_03:32,         R_EAX:32

Note how temporary registers are created to hold the result of each step of computation. With my IR, which is heavily borrowed from BinCAT, it looks like:

Set (Rg EAX)
  ( EReg EAX :& 0xffff00ff :| ( (EReg EAX :<< Constant 8) :& 0xff00 ) )

The statements of the IL are listed here:

data Statement arch where
    Set    :: Extend width (RegType arch) => Lval arch width -> Expr arch width -> Statement arch
    If     :: Expr arch Bool -> [Statement arch] -> [Statement arch] -> Statement arch
    Jmp    :: Expr arch (RegType arch) -> SegmentType arch -> Statement arch
    Call   :: Expr arch (RegType arch) -> SegmentType arch -> Statement arch
    Return :: Statement arch
    Nop    :: Statement arch
    Halt   :: Statement arch
    Debug  :: String -> Statement arch -> Statement arch

As for the Expr arch a data type, I went for a large GADT. Most examples I have seen are very weakly typed, where all the values are unsigned words of some width. For this reason, they require specific IMUL, IDIV and IMOD instructions for signed operations.

I decided to go full GADTs, with stuff like:

data Expr arch a where
  (:+)     :: Num a
           => Expr arch a -> Expr arch a -> Expr arch a
  (:*)     :: Num a
           => Expr arch a -> Expr arch a -> Expr arch a
  (:/)     :: (Integral a, SDivisible (SBV a))
           => Expr arch a -> Expr arch a -> Expr arch a
  (:>)     :: (SymWord a, Integral a)
           => Expr arch a -> Expr arch a -> Expr arch Bool
  ...

And a typeclass-based explicit type conversion system:

  SignExt    :: (Extend a b, Integral a)        => Expr arch a -> Expr arch b
  ZeroExt    :: (Extend a b, Integral a)        => Expr arch a -> Expr arch b
  Truncate   :: (Truncate a b, Integral a)      => Expr arch a -> Expr arch b
  ToSigned   :: (Signop a, Integral a)          => Expr arch a -> Expr arch (Signed a)
  FromSigned :: (Signop a, Integral (Signed a)) => Expr arch (Signed a) -> Expr arch a

For example, here is the translation of the EBPF LsAbsB rg imm instructions, which loads a packet byte into a 64bit register:

let byte :: Expr EBPF Word8
    byte = Addr ( ZeroExt (Constant (Resolved imm)) ) Packet
in  Set (Rg rg) (ZeroExt byte)

Not how the size of the memory access is defined by its type, and so is everything else. This is however not all roses here, as the following prevents a proper Eq instance:

    (:>)     :: (SymWord a, Integral a) => Expr arch a -> Expr arch a -> Expr arch Bool

There is no way to write it:

instance Eq a => Eq (Expr arch a) where
   a == b = case (a,b) of
              (x1 :+ y1, x2 :+ y2) -> x1 == x2 && y1 == y2 -- OK
              (x1 :> y1, x2 :> y2) -> -- can't write an implementation! The types of x1 and x2 are not known to be identical :(

That might be solved by writing specific versions of the comparison operators, such as:

    GT8  :: Expr arch Word8  -> Expr arch Word8  -> Expr arch Bool
    GT16 :: Expr arch Word16 -> Expr arch Word16 -> Expr arch Bool
    GT32 :: Expr arch Word32 -> Expr arch Word32 -> Expr arch Bool
    GT64 :: Expr arch Word64 -> Expr arch Word64 -> Expr arch Bool

But it is clearly not very elegant ...

Interpreting the IL

I decided to have the following milestones when interpreting the IL:

A basic interpreter for concrete programs and memory. That means that the while program state is known when the program starts, and it is possible to explore that way a single possible execution path.
An interactive interpreter, that would work a bit like a debugger. This proved useful, not to debug the analyzed program, but my own program ;)
Symbolic values, introduced in place of the concrete ones. The interpreter should work as before.
Start abstracting some program state, and have the debugger interactively ask the user how to translate symbolic expressions into concrete values.
Finally, run all possible execution paths, collecting constraints.

Step 1 - concrete values

There isn't much to say about this. The following data structures are required in order to emulate a program:

The EmulInfo arch structure holds data such as which register is the instruction pointer, how to handle a jump, etc.
The AState arch structure holds the program state, that is registers and memory.

Once this was done, it was trivial to write an interpreter for the IL. I also wrote enough support for the EBPF architecture in order to solve one of the SSTIC 2017 challenge steps.

Step 2 - interactive interpreter

I used haskeline with optparse-applicative. The latter is kind of well suited for the task of defining commands and parsing parameters, even though the help text really looks too much like that of a command line utility. I still have to integrate tab-completion.

Step 3 - symbolic values

This is how my symbolic value type looks like:

data MemR arch a
  = Unresolved (Expr arch a)
  | Resolved a
  | Unk (SegmentType arch) (RegType arch)

So a symbolic value of type a can either be:

an arbitrary expression of type Expr arch a
a known, concrete value of type a
a reference to a memory address, representing an unknown zone of memory at program start

For now, all registers hold concrete values when the program starts, so the UnkReg constructor is not necessary. Now that values can be unresolved, the expression evaluation function can't have the following type:

evalExpression :: ...  => Expr arch a -> m a

The actual type becomes:

evalExpression :: ...  => Expr arch a -> Eval arch (Either (Expr arch a) a)

This means this can return either a fully evaluated value, or an unevaluated (but perhaps simplified) expression.

Symbolic memory was the biggest headache, as even memory reads could alter the program state! Indeed, a memory read that is 8bits wide at address 5 and another that is 32 bits wide at address 4 are related. In order to make sure this would not go unnoticed, each memory read of a new unknown memory address alters the program state so that subsequent reads could reference the same variable.

Another big pain point was having all the unaligned / endianity changing accesses right.

Step 4 - abstracting program state

This is where things get interesting, but problems start to arise. The most illustrative example is the conditional statement:

    If     :: Expr arch Bool -> [Statement arch] -> [Statement arch] -> Statement arch

The first argument represents a boolean condition. The second is the list of statement to be executed if that condition is true (the then branch), and the third if it is false (the else branch).

Unfortunately, resolving the boolean condition could result in an unevaluated expression! In order to be able to decide which branch must be executed, it must be reduced to a concrete Bool. For this reason, I made the following functions primitives of the Eval monad:

solveBool :: Expr arch Bool -> Eval arch Bool
solveAddr :: Expr arch (RegType arch) -> Eval arch (RegType arch)

(solveAddr is for jumps, so that the interpreter could determine where to go next.)

And now the If statement can be evaluated like:

evalStatement :: forall arch. (Ord (Register arch), Eq (RegType arch)) => Statement arch -> Eval arch ()
evalStatement s = do
   ...
   case s of
      If b th el -> do
          x <- evalExpression b >>= either solveBool pure
          mapM_ evalStatement $ if x then th else el
      ...

Here, the evalExpression b expression has type Eval arch (Either (Expr arch Bool) Bool)). With the help of the solveBool function, it is possible to turn the dreaded unresolved expression into a proper boolean.

But now, how to run the Eval monad?

This is the signature of the runEval function, which translates an Eval arch a expression into an arbitrary monad:

runEval :: Monad m
  => (Expr arch Bool -> m (Eval arch Bool))                     -- ^ resolving boolean choices
  -> (Expr arch (RegType arch) -> m (Eval arch (RegType arch))) -- ^ resolving addresses
  -> EmulInfo arch                                              -- ^ architecture specific data
  -> AState arch                                                -- ^ program state
  -> Eval arch a                                                -- ^ Eval action to run
  -> m ( Either (EmulError arch) (a, AState arch)
       , [EmulationTrace arch]
       )

The return result is a bit obscure at first. It includes a list of emulation traces (in particular, the state of the emulation at each instruction), and the result of the evaluation. The evaluation could fail (and return an EmulError arch), or succeed (and return the expected a, along with an updated program state).

As can be guessed, the first two arguments are used to implement the solveBool and solveAddr functions.

In the case of a simple interpreter, where all program state is know and evalExpression always returns the Right result, the following function can be used:

trivialSolve :: Monad m
             => Expr arch x
             -> m (Eval arch x)
trivialSolve e =
  return $ case e ^? _RConstant of
    Just x -> return x
    Nothing -> throwError (Misc "trivialSolve failed :(")

Note that the runEval function is parametric in the type of monad, and related to the type of the provided solveBool and solveAddr implementations.

Step 5 - actual symbolic execution

Let's look back at the introductory example:

Set (Rg EAX) 0
If (EReg EBX :== 0x12345678) [Set (Rg EAX) 1] []
If (EReg ECX :== 0xabcde)    [Set (Rg EAX) (EReg EAX :+ 1)] []

Symbolic execution would work that way:

Update the program state, setting EAX to 0.
Create two states:
- Set EAX to 1, assorted with the constraint EReg EBX :== 0x12346789, and branch again:
  - Set EAX to 2, assorted with the constraint EReg ECX :== 0xabcde.
  - Set EAX to 1, assorted with the constraint EReg ECX :!= 0xabcde.
- Set EAX to 0, assorted with the constraint EReg EBX :!= 0x12346789, and branch again:
  - Set EAX to 1, assorted with the constraint EReg ECX :== 0xabcde.
  - Set EAX to 0, assorted with the constraint EReg ECX :!= 0xabcde.

So if I am interested in the outcome where EAX is equal to 2, I should solve the following constraints set:

EReg EBX :== 0x12346789
EReg ECX :== 0xabcde

Now how to implement this without changing the interpreter? It turned out to be surprisingly easy! The branching can be handled using the list monad, and the constraint collection using the writer monad:

symbEvalBool :: Expr arch Bool -> LT.ListT (Writer [Expr arch Bool]) (Eval arch Bool)
symbEvalBool e = case e ^? _RConstant of
      Just b -> return (return b)
      Nothing ->  (lift (tell [e])     >> return (return True))
              <|> (lift (tell [Not e]) >> return (return False))

Conclusion

My toy program currently works as long as there are not too many branches. It solves the first (easy) part of the "don't let him escape" problem of the 2017 SSTIC challenge:

$ ./sstic2017-symbolicsolve
Initial packet content:
P/00000000 ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? 02 ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ??    ??????????????.?????????????????
P/00000020 ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ??    ????????????????????????????????
Symbolic exectution of all paths ...
10 possible executions
Interesting states:
  * ffffffff
    - not(zeroExt(P/0000000c,[Word16->Word64]) != 0x800)
    - not(zeroExt(P/00000017,[Word8->Word64]) != 0x11)
    - not(zeroExt(P/00000024,[Word16->Word64]) != 0x539)
    - not((zeroExt(P/00000010,[Word16->Word64]) + 0xfffffff8 - 0x8) << 32 >> 32 != 0x10)
    - not(zeroExt(P/0000001e,[Word8->Word64]) & 0xff != 0x4c)
    - not(zeroExt(P/0000001f,[Word8->Word64]) & 0xff != 0x55)
    - not(zeroExt(P/00000026,[Word32->Word64]) << 32 | zeroExt(P/0000002a,[Word32->Word64]) != 0x456443724d66417d)
    - not(zeroExt(P/00000022,[Word32->Word64]) << 32 >> 32 != 0x42765751)
    - not(zeroExt(P/00000020,[Word16->Word64]) & 0xffff != 0x4d7b)
Solution 1:
Updated packet content:
P/00000000 ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? 08 00 02 ?? 00 20 ?? ?? ?? ?? ?? 11 ?? ?? ?? ?? ?? ?? 4c 55    ????????????...?. ?????.??????LU
P/00000020 4d 7b xx xx xx xx xx xx xx xx xx xx xx 7d ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ??    M{xxxxxxxxxxx}??????????????????
Rerun, with the updated packet content, and retrieve the new EBPF map content:
map[0] = 00000001
map[1] = bd89a8ae
map[2] = ba9bbc8d

Note that the initial packet is not fully unknown, as I had to set a concrete value to a field that is used for indexed memory accesses. However, the second part of the problem involves a lot of conditional expressions, resulting is something like $2^115$ possible states.

What is amusing is that my current plan for fighting state explosion will have a direct application in the gamebook solving series. More to come soon!

]]> Solving a gamebook : a smarter solver https://hbtvl.banquise.net/posts/2017-03-23-GBS-smarter-solver.html 2017-03-22T00:00:00Z

In the previous episode, a "simple" solver was presented, along with ideas to make it go much faster (namely memoization and parallelism). It is however possible to do much better, with only a little bit of work.

A simple observation

In order to compare two possible choices, the previously presented algorithm scores them, and chooses the solution with the highest score. The score is equal to the score of all possible sub-solutions (corresponding to all possible outcomes of a choice), weighted by their probability.

This means that all possible states and all possible configurations are checked. Memoization saves us from checking the same state twice, but it might be possible to do better!

In particular, it is often obvious to the human that some choices are better than others:

It is sometimes possible to avoid a fight.
Some choices lead to certain death.
Some choices lead to free items.

I am currently thinking of a general solution to that sort of apparently "simple" decisions, but there is a simple observation that leads to a good speedup, with minimal complexity. Many random events are only occuring with a fairly small probability. Here is an example with the first fight, with some default starting values for our character:

Remaining HP	Probability	Cummulated probability
20	12.80%	12.80%
19	10.97%	23.77%
21	9.60%	33.37%
22	7.70%	41.07%
18	7.03%	48.10%
15	6.21%	54.31%
25	6.20%	60.51%
14	6.05%	66.56%
13	5.34%	71.90%
16	5.24%	77.14%
23	5.10%	82.24%
17	4.73%	86.97%
12	3.74%	90.71%
24	3.60%	94.31%
11	1.55%	95.87%
8	0.82%	96.69%
9	0.76%	97.45%
7	0.72%	98.17%
10	0.72%	98.88%
6	0.55%	99.43%
5	0.33%	99.76%
4	0.12%	99.88%
0	0.05%	99.93%
3	0.03%	99.96%
1	0.02%	99.98%
2	0.02%	100.00%

There are 26 possible outcomes, so 26 branches that span from this node, each of them having branches. But just exploring 6 of these branches (less than a quarter) gives 54% of the total score of this node. Exploring half of the outcomes gives 87% of the total score!

Lazy evaluation to the rescue

The idea, when comparing multiple choices, is to only explore as many outcomes as necessary to find the best. It could be possible to do that explicitely, but I decided to use lazy evaluation to make the algorithm a bit simpler.

Let's start with the following data structure:

data PScore = Approximate { _proba  :: !Proba
                          , _pscore :: !Rational
                          , _next   ::  PScore }
            | Certain !Rational
            deriving (Show, Eq, Generic)

It is a recursive data structure, a bit like a list, except the last constructor always contains a value. For representing the expected roll of a six sided dice, once completely forced, it will look like:

Approximate { _proba = 1 % 6, _pscore = 1 % 6, _next =
  Approximate {_proba = 1 % 3, _pscore = 1 % 2, _next =
    Approximate {_proba = 1 % 2, _pscore = 1 % 1, _next =
      Approximate {_proba = 2 % 3, _pscore = 5 % 3, _next =
        Approximate {_proba = 5 % 6, _pscore = 5 % 2, _next =
          Certain (7 % 2)
}}}}}

As can be seen, the _proba field represents how certain the _pscore field is. As the structure is traversed, the _proba field value increases, until the Certain constructor is reached (meaning the probability is 1).

Now, if a maximum possible score is known, it is possible to compare solutions that way:

scoreCompare :: Rational -> PScore -> PScore -> Ordering
scoreCompare _ (Certain sa) (Certain sb) = compare sa sb
scoreCompare smax na@(Certain sa) (Approximate pb sb nb)
  | sa < sb = LT
  | sa > sb / pb = GT
  | otherwise = scoreCompare smax na nb
scoreCompare smax (Approximate pa sa na) nb@(Certain sb)
  | sb < sa = GT
  | sb > sa / pa = LT
  | otherwise = scoreCompare smax na nb
scoreCompare smax ca@(Approximate pa sa na) cb@(Approximate pb sb nb)
  | sa / pa > sb + (1 - pb) * smax = GT
  | sb / pb > sa + (1 - pa) * smax = LT
  | pa > pb   = scoreCompare smax ca nb
  | otherwise = scoreCompare smax na cb

If constructed properly, forcing the evaluation of the next link in the PScore chain should only force the evaluation of the related outcome. It means that when comparing two PScores, only the minimum number of underlying outcomes should be evaluated.

But does it make a difference?

I have to admit I ran a few test really hoping this would help. During my previous tries, it only gave marginal speedups (around 5%). But this time, with the work that has been put into optimizing the game state representation, the speedups are much more substantive.

When computing the probability to reach chapter 150, the "simple" solver took 4 hours and 26 minutes, whereas with this optimization it only took 47 minutes! That is a huge speedup, and I can tell I am quite proud of it.

What's next?

Now that this optimization is out, and working even better than I expected, I am entering the terra incognita. I have only a few optimization ideas left, and the program is far from being efficient enough to run the book in one run.

During previous tries, I attained an approximation of the solution by combining manual patching of the chapter's description (to force some choices I knew to be better), and a "checkpoint" system. There are natural "choke points" in game books, that the player has to go through. My idea was that I could compute partial solutions between each pair of subsequent "choke points". While this makes the problem computationally easy to solve, it is also very approximate. Some choices have consequences only after the next choke point, so doing it blindly is not good.

This time, I will try to automatically simplify the chapter's structures by removing branches that offer no advantages compared to others. I am still unclear on how this should be done, but hopefuly this will turn out well!

]]> Solving a gamebook : a simple solver https://hbtvl.banquise.net/posts/2017-03-22-GBS-simple-solver.html 2017-03-22T00:00:00Z

In this episode I will quickly describe how to write a generic solver for games having the following properties:

single player (shouldn't be hard to generalize though)
can include an element of randomness
no loops in game states (can't go back to a previous situation)
relatively simple games

The cause of the last constraint is that the algorithm that is being described here is very simple. Techniques for making it faster will be describe at the end of this post, and in further episodes. This is a template Haskell file than can be copy-pasted and reused, so let's start with the imports:

module SimpleSolver where

import Data.Ord (comparing)
import Data.List
import qualified Data.Map.Strict as M
import qualified Data.List.NonEmpty as NE

A quick reminder of the problem and base types

A probability is represented as a rational value, and a choice is a description coupled with a list of states weighted by probability:

type Proba = Rational
type Choice description state = (description, Probably state)
type Probably a = [(a, Proba)]

Obviously, an event that is certain has probability 1:

certain :: a -> Probably a
certain a = [(a,1)]

And the regroup function, that is used to group together identical events. I tried discrimination, but it was too slow, so I moved to unordered-containers. Unfortunately, under heavy stress I encountered a nasty bug that got me scared. I complained about my program behaving randomly on the #haskell IRC channel, only to have Cale find the culprit in about two minutes. That was cool! It is now using containers, and it is just about as fast as with the HashMap.

regroup :: Ord a => Probably a -> Probably a
regroup = M.toList . M.fromListWith (+)

This function is not used in this post, but is nevertheless extremely critical for performance!

What is a solution ?

The basic structure of the game is, for each chapter:

The player chooses one of the possible outcomes,
and stuff happens to him, possibly randomly.

Solving the game means deciding which choice is the best, depending on the circumstances. In the case of the Lone Wolf series, the circumstances could be the current player health, equipment, or choice of disciplines. The solution can be thought of as some sort of tree, where each node describes what the best choice is, and the childrens are the possible outcomes. The tree leaves are the "game over" situations.

data Solution state description
    = Node { _desc    :: description
           , _stt     :: state
           , _score   :: Rational
           , _outcome :: Probably (Solution state description)
           }
    | Leaf { _score :: Rational
           , _stt   :: state
           }
    deriving (Show, Eq)

Finding the solution

We will need the following type, that is used to possibly assign a score to a situation. In the Lone Wolf books, it will give a score of Known 1 to chapter 350, Known 0 to all situations where the hero dies, and Unknown for all other states.

data Score = Known Rational | Unknown

The function that solves a game has the following type:

solve :: (state -> NE.NonEmpty (Choice description state))
      -> (state -> Score)
      -> state
      -> Solution state description

That is, it requires the following arguments:

A function that, given a game state, can give a non-empty list of choices to the player can decide to choose from.
A function that can assign a Score to any game state, as described above.
The current state.

The algorithm is fairly obvious, and as a result the function body is smaller than the Solution type declaration:

solve getChoice score stt =
    let scored = do
          (cdesc, pstates) <- getChoice stt

Given the current state, the possible choices are enumerated (using the list monad), where cdesc is the description of a choice, and pstates the list of possible outcomes, along with their respective probabilities of occurence.

          let ptrees = map (\(o,p) -> (solve getChoice score o, p)) pstates

Here, pstates has type Probably state. The ptrees variable contains the solution associated with each of the possible states, and has type Probably (Solution state description).

          return $ Node cdesc stt (sum (map (\(o, p) -> p * _score o) ptrees)) ptrees

A solution is built, with a score equal to the sum of the scores of the possible outcomes, weighted by their probability of occurence.

    in  case score stt of
          Known x -> Leaf x stt
          Unknown -> maximumBy (comparing _score) scored

Finally, if the current state can be scored, then the "solution" is known. If it can't be, the best choice, based on its score, is kept.

So, problem solved?

Unfortunately, this simple algorithm will not cut it. It is way too slow, but can be improved upon fairly easily with two simple changes:

The solve function can be memoized,
the computations in the solve function can be parallelized.

With these two simple changes, the solver will look like that. It is reasonnably efficient, and will solve the book and report statistics up to chapter 200, which is roughly half the book, in about 30 seconds on my computer. Unfortunately, from there is gets slower fairly quickly!

In the next chapter, I will describe a technique for smarter scoring that will decrease this time to about 13 seconds.

As a teaser, here is the output of the solver for now (chapter 247 really is a killer!):

Solving for the following initial state:
CharacterConstant {_maxendurance = Endurance {getEndurance = 25}, _combatSkill = CombatSkill {getCombatSkill = 15}
, _discipline = [Hunting,WeaponSkill ShortSword,MindBlast,SixthSense,MindShield]}
(Endurance {getEndurance = 25},(inventoryFromList [(Gold,15),(Meal,2),(SealHammerdalVol2,1),(Weapon ShortSword,1)]))
Winning probability: 0.8731436491536936 [62916630607511833328443669116357 % 72057594037927936000000000000000]
0.1968 NewChapter 200 (Endurance {getEndurance = 23},(inventoryFromList [(Meal,1),(TicketVol2,1),(SealHammerdalVol2,1)])) Didn'tFight
0.1968 NewChapter 200 (Endurance {getEndurance = 25},(inventoryFromList [(Meal,1),(TicketVol2,1),(SealHammerdalVol2,1)])) Didn'tFight
0.0974 HasLost 247
0.0612 NewChapter 200 (Endurance {getEndurance = 15},(inventoryFromList [(Gold,12),(TicketVol2,1),(Weapon Mace,1)])) Didn'tFight
0.0610 NewChapter 200 (Endurance {getEndurance = 16},(inventoryFromList [(Gold,12),(TicketVol2,1),(Weapon Mace,1)])) Didn'tFight
0.0564 NewChapter 200 (Endurance {getEndurance = 17},(inventoryFromList [(Gold,12),(TicketVol2,1),(Weapon Mace,1)])) Didn'tFight
0.0481 NewChapter 200 (Endurance {getEndurance = 18},(inventoryFromList [(Gold,12),(TicketVol2,1),(Weapon Mace,1)])) Didn'tFight
0.0480 NewChapter 200 (Endurance {getEndurance = 14},(inventoryFromList [(Gold,12),(TicketVol2,1),(Weapon Mace,1)])) Didn'tFight
0.0409 NewChapter 200 (Endurance {getEndurance = 13},(inventoryFromList [(Gold,12),(TicketVol2,1),(Weapon Mace,1)])) Didn'tFight
0.0332 NewChapter 200 (Endurance {getEndurance = 12},(inventoryFromList [(Gold,12),(TicketVol2,1),(Weapon Mace,1)])) Didn'tFight
0.0263 NewChapter 200 (Endurance {getEndurance = 19},(inventoryFromList [(Gold,12),(TicketVol2,1),(Weapon Mace,1)])) Didn'tFight
0.0233 HasLost 345
0.0232 NewChapter 200 (Endurance {getEndurance = 11},(inventoryFromList [(Gold,12),(TicketVol2,1),(Weapon Mace,1)])) Didn'tFight
0.0168 NewChapter 200 (Endurance {getEndurance = 10},(inventoryFromList [(Gold,12),(TicketVol2,1),(Weapon Mace,1)])) Didn'tFight
0.0132 NewChapter 200 (Endurance {getEndurance = 9},(inventoryFromList [(Gold,12),(TicketVol2,1),(Weapon Mace,1)])) Didn'tFight
0.0124 NewChapter 200 (Endurance {getEndurance = 20},(inventoryFromList [(Gold,12),(TicketVol2,1),(Weapon Mace,1)])) Didn'tFight
0.0083 NewChapter 200 (Endurance {getEndurance = 8},(inventoryFromList [(Gold,12),(TicketVol2,1),(Weapon Mace,1)])) Didn'tFight
0.0056 NewChapter 200 (Endurance {getEndurance = 23},(inventoryFromList [(Gold,12),(Meal,1),(TicketVol2,1),(SealHammerdalVol2,1)])) Didn'tFight
0.0056 NewChapter 200 (Endurance {getEndurance = 25},(inventoryFromList [(Gold,12),(Meal,1),(TicketVol2,1),(SealHammerdalVol2,1)])) Didn'tFight
0.0046 NewChapter 200 (Endurance {getEndurance = 21},(inventoryFromList [(Gold,12),(TicketVol2,1),(Weapon Mace,1)])) Didn'tFight
0.0043 NewChapter 200 (Endurance {getEndurance = 7},(inventoryFromList [(Gold,12),(TicketVol2,1),(Weapon Mace,1)])) Didn'tFight
0.0040 NewChapter 200 (Endurance {getEndurance = 24},(inventoryFromList [(Meal,1),(TicketVol2,1),(SealHammerdalVol2,1)])) Didn'tFight
0.0040 NewChapter 200 (Endurance {getEndurance = 26},(inventoryFromList [(Meal,1),(TicketVol2,1),(SealHammerdalVol2,1)])) Didn'tFight
0.0031 HasLost 186
0.0020 NewChapter 200 (Endurance {getEndurance = 6},(inventoryFromList [(Gold,12),(TicketVol2,1),(Weapon Mace,1)])) Didn'tFight
0.0016 HasLost 326
0.0010 HasLost 146
0.0005 HasLost 268
0.0001 NewChapter 200 (Endurance {getEndurance = 24},(inventoryFromList [(Gold,12),(Meal,1),(TicketVol2,1),(SealHammerdalVol2,1)])) Didn'tFight
0.0001 NewChapter 200 (Endurance {getEndurance = 26},(inventoryFromList [(Gold,12),(Meal,1),(TicketVol2,1),(SealHammerdalVol2,1)])) Didn'tFight
0.0001 HasLost 34

]]> Solving a gamebook : solving the cartwheel game https://hbtvl.banquise.net/posts/2017-03-19-GBS-cartwheel.html 2017-03-19T00:00:00Z

The "Cartwheel" game is described in chapter 238, and is a gambling game that is unline any real world gambling game. It goes like this:

The player decides on an (integral) amount of money he would like to bet.
The players chooses a number between 0 and 9.
A random number between 0 and 9 is rolled, with the following outcomes:
- If the player guessed what the rolled number was, he wins 8 times his wager.
- If the player chose a number that is just before or after (0 and 9 are adjacent for this purpose) the rolled number, he wins 5 times his wager.
- Otherwise, he loses his wager.

The game is very much biased for the player, as the expected win for a wager of $w$ is $8w/10 + 2×5w/10 = 1.8w$! So, unless it is a money laundering scheme, the gambling house that is described will soon go out out business.

The reason this game is placed here is that the player will soon after need at least 20 gold coins in order to buy a ticket, and some extra change depending on the circumstances. In order to convince yourself, you might want to take a look at the following picture, and search for chapter 238:

From here, one can see the only way to get a ticket (TKT) is to buy one for 20 coins at chapter 136. Then, depending on random rolls, one could be forced to leave another coin at chapters 195 or 339. If the player arrives at chapter 39 without tickets, he will be sent to his death. Same thing if he doesn't have an extra coin in the next chapter (346).

So the player needs at least 22 coins before buying a ticket in order to make sure he makes it. But then, he might want to have extra money lining up his pocket for the rest of the adventure!

Optimal strategy, and model

I asked for help solving this problem, but did not get a good answer yet. Fortunately, one of the answers confirmed my intuition that the optimal strategy would be to bet as little as possible, so one coin for each round.

That makes it possible to model the game, for a target of $t$ coins:

if the player owns $t$ or more coins, the game ends
if the player owns 0 coins, the game ends too!
otherwise, the next step is:
- the player loses 1 coin, with a probability of $7/10$
- the player gains 4 coin, with a probability of $2/10$
- the player gains 7 coin, with a probability of $1/10$

That makes it possible to write a transition matrix, mapping the three cases:

transition :: Fractional a => Int -> Matrix a
transition n = foldl' ins (emptyMatrix n) (transitions ++ zero ++ fixedWon)
  where
     ins :: IM.IntMap (IM.IntMap a) -> (Int, Int, a) -> IM.IntMap (IM.IntMap a)
     ins mtx (x,y,r) = mtx & ix y . ix x .~ r
     zero = [(0,0,1)]
     fixedWon = do
        x <- [n .. n + 8]
        return (x,x,1)
     transitions = do
         x <- [1 .. n - 1]
         [ (x, max 0 (x - 1), 7 / 10), (x, x + 7, 1 / 10), (x, x + 4, 2 / 10) ]

For example, if the target is 10 coins (which is a bad idea, as the player will lose later):

1.0	0.7
		0.7
			0.7
				0.7
					0.7
	0.2					0.7
		0.2					0.7
			0.2					0.7
	0.1			0.2					0.7
		0.1			0.2
			0.1			0.2				1.0
				0.1			0.2				1.0
					0.1			0.2				1.0
						0.1			0.2				1.0
							0.1							1.0
								0.1							1.0
									0.1							1.0

So, the probability of a single step, if one start with 5 coins is:

LoneWolf.Cartwheel> transition  10 !* IM.singleton 5 (1 :: Double) & IM.filter (/= 0)
fromList [(4,0.7),(9,0.2),(12,0.1)]

Solving

In order to solve this problem, one would need to compute, for a transition matrix $M$, the value of $\lim_{x\to\infty} M^x$. In order to do that, one would need to diagonalize the $M$ matrix. I don't know how to do that symbolically on large matrices (for our purpose, the smallest will be 28×28), and sage doesn't seem to either, so I will go for a numerical approximation. In all cases, I suppose the eigenvalues have irrational factors, so even with the exact symbolic formula the end result would have been approximated anyway, because all my probabilities are Rationals.

Instead of doing all the linear algebra, I just opted for squaring the matrix 20 times, using Doubles, and then converting it back to Rational.

The result, for the same target of 10 coins, looks like:

1.0	0.75	0.56	0.41	0.30	0.22	0.15	0.11	0.07	0.05









	0.04	0.10	0.21	0.17	0.14	0.30	0.21	0.14	0.10	1.0
	0.05	0.07	0.11	0.21	0.18	0.12	0.28	0.20	0.14		1.0
	0.07	0.10	0.09	0.12	0.22	0.15	0.10	0.27	0.19			1.0
	0.02	0.08	0.08	0.07	0.10	0.17	0.12	0.08	0.25				1.0
	0.01	0.02	0.04	0.04	0.04	0.03	0.12	0.08	0.05					1.0
	0.02	0.02	0.02	0.04	0.04	0.03	0.02	0.11	0.08						1.0
	0.00	0.02	0.01	0.01	0.03	0.02	0.01	0.01	0.10							1.0

And the probability of a potentially infinite amount of steps, if one start with 5 coins is:

LoneWolf.Cartwheel> approx  10 !* IM.singleton 5 (1 :: Double) & IM.filter (/= 0)
fromList [(0,0.22607400081196705)
         ,(10,0.14606094132925695)
         ,(11,0.18229417264972955)
         ,(12,0.22275693955176407)
         ,(13,0.10310731161277857)
         ,(14,4.406795246148139e-2)
         ,(15,4.4122564256762906e-2)
         ,(16,3.151611732625923e-2)]

And interesting observation is that it makes little difference to aim for 22 or 50 coins if you start from a low amount, if you look at the relative distribution!

What about the other special chapter

The other special chapter, the portholes chapter, is completely random, and biased against the player (the two-zeros win is decided for the opponents before the player plays). It makes no sense to play this game, and it should be avoided. This means the only ratio

Next stop

There is now an implementation of the approximation code, along with code that was used to produce the charts and table in this article. There is also an implementation of the special chapter! The player gets to choose the amount of money he is aiming for, which is much better than the previous solutions I had (which only tried to get 22 gold, with a worse approximation).

The next stop is writing a solver, and trying to solve the first few chapters of the game. Then, it will be time to optimize the heck of it.

]]> Solving a gamebook : game state and probability https://hbtvl.banquise.net/posts/2017-03-18-GBS-rules.html 2017-03-18T00:00:00Z

I wrote a whole lot of logic since the last episode, and the game can now be played on the console, until you reach one of the special chapters. You can try the game (look at the end of this post). Decision flattening code is complete, and so is the combat code, even though it's really slow.

I tried several times describing the implementation in this post, but it's much too tedious to read, so it will focus only on a few key points.

The hidden optimization story

Many of the representation choices I'll take are motivated by the knowledge that solving the game book will involve some sort of memoization. As a result, it is crucial to make the game state as simple as possible :

it should hold as little information as possible
it should be possible to quickly turn this information into a "simple" data type, such as an integer type

The player character

All of what the player has to remember, when playing, is kept on the action chart. That's where all the game state resides, and it can be separated in two parts.

First of all, most of the data is constant during a play session, as it is rolled at the beginning of the story, or is imported from a previous book.

data CharacterConstant = CharacterConstant
      { _maxendurance :: Endurance
      , _combatSkill  :: CombatSkill
      , _discipline   :: [Discipline]
      } deriving (Generic, Eq, Show, Read)

Endurance represent how much damage a player can withstand, whereas the combat skill represents ... its skill in combat. The disciplines are many, and changing them can make each session unique! This data doesn't need to be particularly optimized for now, even though an obvious change would be to make the _discipline field a Set.

Then, there is the part that varies during the game, which needs to be optimized:

data CharacterVariable = CharacterVariable
      { _curendurance :: Endurance
      , _equipment    :: Inventory
      } deriving (Generic, Eq, Show, Read)

This is mainly the inventory, and the current health of the player.

A note on the inventory

A straightforward implementation of the inventory would be something like:

type Inventory = Map Item Int

This will however be much too inefficient for our future optimizations, so I decided to preemptively optimize it:

data Inventory = Inventory
     { _singleItems :: Word32
     , _gold        :: Word8
     , _meals       :: Word8
     } deriving (Generic, Eq, Show, Read)

I did not currently add the ! and {-# UNPACK #-} annotations, but inventory will most likely look like that up until the end. As can be seen, unique items are handled differently from meals and gold. This is an approximation, as the player can have a pair of identical weapons, but as it serves no purpose (except when selling weapons), I thought it was alright.

The singleItems field is a bitmap, where the first 25 bits each represent an unique item, just like it would be done in C.

Here is a function that works on the Inventory:

addItem :: Item -> Int -> Inventory -> Inventory
addItem i count inv
  | count < 0 = delItem i (negate count) inv
  | count == 0 = inv
  | otherwise = case i of
                  Gold -> inv & gold %~ \curgold -> min 50 (curgold + fromIntegral count)
                  Meal -> inv & meals +~ fromIntegral count
                  _    -> inv & singleItems %~ flip setBit (fromEnum i)

As can be seen, it treats gold and meals specifically, and sets bits in the singleItems field according to each item fromEnum. It ensures that the player can't own more than 50 gold coins, but is otherwise pretty brittle:

adding more than one item that's not gold or a meal will only result in the item being owned once by the player
it doesn't enforce the backpack rule of 8 items at most (including meals)
it doesn't enforce the fact that the player can't own more than two weapons

All of this logic is handled in the part that handles player decisions, and that will be presented, if ever, in another episode (this logic lives in the LoneWolf.Choices module).

A possible future change would be to handle weapons differently, and have for example two slots reserved for them in the inventory. That would make a lot of the logic in the program a tad simpler, but I'll wait for the profiling data to decide.

The `Probably` type

Several outcomes involve an element of randomness, more specifically the Randomly and Fight ones. For this reason, the function that computes the updated game state from the ChapterOutcome has the following type:

data NextStep = NewChapter ChapterId CharacterVariable HadCombat
              | HasLost
              | HasWon CharacterVariable
              deriving (Show, Eq, Generic)

update :: CharacterConstant
       -> CharacterVariable
       -> ChapterOutcome
       -> Probably NextStep
update = ...

The Probably a type represents the probability distribution of events of type a happening. I went for this simple representation:

type Probably a = [(a, Proba)]
type Proba = Rational

So, for example, a six-sided dice roll will be:

d6 :: Probably Int
d6 = [ (n, 1/6) | n <- [1..6] ]

However, there are cases when transforming a distribution will require simplification. For example, if we make a game where we need to roll two six sided dices:

twod6 :: Probably Int
twod6 = do
  (roll1, p1) <- d6
  (roll2, p2) <- d6
  return (roll1 + roll2, p1 * p2)

This will return a list with 36 distinct elements:

[(2,1 % 36),(3,1 % 36),(4,1 % 36),(5,1 % 36),(6,1 % 36),(7,1 % 36),(3,1 % 36),(4,1 % 36)
,(5,1 % 36),(6,1 % 36),(7,1 % 36),(8,1 % 36),(4,1 % 36),(5,1 % 36),(6,1 % 36),(7,1 % 36)
,(8,1 % 36),(9,1 % 36),(5,1 % 36),(6,1 % 36),(7,1 % 36),(8,1 % 36),(9,1 % 36),(10,1 % 36)
,(6,1 % 36),(7,1 % 36),(8,1 % 36),(9,1 % 36),(10,1 % 36),(11,1 % 36),(7,1 % 36),(8,1 % 36)
,(9,1 % 36),(10,1 % 36),(11,1 % 36),(12,1 % 36)]

While this will technically work with all algorithms I will use to solve the game book, this will increase the work necessary to solve the game, as it will require exploring more game states. In the past, I used this maps to ensure the unicity of game states in the Probably type:

type Probably a = HashMap a Proba

twod6 :: Probably Int
twod6 = HM.fromListWith (+) $ do
  (roll1, p1) <- HM.toList d6
  (roll2, p2) <- HM.toList d6
  return (roll1 + roll2, p1 * p2)

But this time I decided to give discrimination a spin! I might benchmark it against other solutions in the next installments.

regroup :: D.Grouping a => Probably a -> Probably a
regroup = map (\( (a,s): as ) -> (a, s + sum (map snd as)) ) . D.groupWith fst

twod6 :: Probably Int
twod6 = regroup $ do
  (roll1, p1) <- d6
  (roll2, p2) <- d6
  return (roll1 + roll2, p1 * p2)

Playing the game

It's now possible to play the game, unless you arrive to either the cartwheel or portholes chapters. It's pretty ugly, but is there mainly to validate that things seem to work. If you want to try it, just clone the repo, and do:

$ stack build && .stack-work/install/x86_64-linux/lts-8.4/8.0.2/bin/gamebooksolver-consoleplay

]]> Solving a gamebook : implementing the combat system, and first optimizations https://hbtvl.banquise.net/posts/2017-03-18-GBS-fight.html 2017-03-18T00:00:00Z

The combat system of the Lone Wolf series is pretty simple in the absence of special rules. The problem is, there are quite a few special rules, that are modelized as FightModifiers:

data FightModifier = Undead
                   | MindblastImmune
                   | Timed Rounds FightModifier
                   | CombatBonus CombatSkill
                   | BareHanded
                   | EnemyMindblast
                   | PlayerInvulnerable
                   | DoubleDamage
                   | FakeFight ChapterId
                   | Evaded Rounds ChapterId

Here is a brief description of these modifiers:

Undead: undead creatures take double damage from the Sommerswerd.
MindblastImmune: the opponent is not affected by the MindBlast ability of the player.
CombatBonus: the player receives a combat bonus for this fight.
BareHanded: the player is forced to fight bare handed, reducing his combat skill by 4 points.
FakeFight: this isn't a real fight, and the player will have his endurance restored at the end of the fight. The ChapterId is provided as the destination if he loses the fight. This happens in chapter 276.
EnemyMindblast: the enemy has the MindBlast ability!
PlayerInvulnerable the player cannot be harmed.
DoubleDamage: happens in chapter 306, where the player inflicts double damage.
Timed: the effect only lasts a given number of rounds.
Evaded: the player will evade the fight after this round.

Player abilities also interact with the combat rules, resulting in an annoyingly complex function that I hope I got right for resolving a single round of combat, and another one that is just as annoying for computing the combat ratio.

In the end, the top level function takes all player data, the combat specific data, and returns a distribution of the player's Endurance statistic after the fight:

fight :: CharacterConstant -> CharacterVariable -> FightDetails -> Probably Endurance
fight cconstant cvariable fdetails = regroup $ do

It starts by computing the result of a single combat round.

  ((hpLW, hpOpponent), p) <- fightRound cconstant cvariable fdetails

Then checks if the combat is over.

  let evaded = fdetails ^? fightMod . traverse . _Evaded . _1 == Just 0
      outcome
        | hpLW <= 0 = return (-1, p) -- player is dead
        | hpOpponent <= 0 || evaded = return (hpLW, p) -- player escaped

Otherwise, the endurance of each opponent is updated, timed effects counters are decremented, and another round is started:

        | otherwise = let nvariable = cvariable & curendurance .~ hpLW
                          ndetails = fdetails & fendurance .~ hpOpponent
                                              & fightMod %~ mapMaybe decrementTimed
                      in  fmap (*p) <$> fight cconstant nvariable ndetails
  outcome

But it is so slow!

The sample fight that's in the tests makes the tests complete in about 3.4s. That is a lot of time, and is a showstopper, as many, many fights will be evaluated in order to solve the game.

The reason it's so slow is that each round produce ten possible outcomes, resulting in an exponential increase in the number of evaluated possibilities with the number of rounds. Hopefully, thanks to the way the combat system works, one of the fighters always receives damage, which means that the fight always finishes in a finite amount of time.

Unfortunately, as it is implemented, most nodes are traversed, and evaluated, several times, as can be seen in the following picture:

The cause is easy to understand: several distinct fighting situations can produce the same result. For example, the two following situations result in the player having 15 Endurance:

The player has 17 Endurance, and receives two points of damage ;
the player has 16 Endurance, and receives one point of damage.

In both cases, the algorithm, as it is written right now, will keep on computing starting from the same state, effectively computing twice the exact same expression. If it was possible to compute each expression exactly once, the situation would be like that:

There are several strategies for reducing the complexity of this problem, which would fall under the name "dynamic programming". The simplest way to do that in Haskell is memoization, for which I have often successfully used the data-memocombinators package.

Let's memoize!

Memoizing is about mapping the results of a function to a persistent structure. There must be a relationship between the arguments of the function and the position the result is stored in the structure. That way, in order to evaluate the result of a function, one only has to traverse the structure to the proper place, and read the stored value. But do not worry, data-memocombinators takes care of everything!

However, there is a major drawback to memoization: the structure that stores the results must be persistent, and will never be garbage collected. It is crucial to memoize functions with the smallest set of inputs as possible, in order to minimize the memory footprint (this is an approximation, as it depends on the properties of the lazy data structure, but it is a good rule of thumb).

Let's go back to the original function, that has type:

fight :: CharacterConstant -> CharacterVariable -> FightDetails -> Probably Endurance

This is far from ideal, as all three arguments are complex data structures, containing a lot of information irrelevant to the fighting situation. After a bit of research about compressing the input space of the memoized combat function, I made the following observations:

The Timed modifier, which models effects that only last for certain duration, keeps effects working for at most 2 rounds after the fight has started. As the effects do not last long, I decided to keep the original (slow) combat logic as long as a Timed effect was present in the list of CombatModifier. This decision simplifies the handling of PlayerInvulnerable, which only ever happens to be timed.
Almost all other modifiers / disciplines can be reduced to CombatSkill or Endurance modifications that are constant for the fight (player having WeaponSkill or Mindblast, combat modifiers such as MindblastImmune, CombatBonus, BareHanded, or Undead and DoubleDamage which both can be modelled as if the opponent had half the life).
The only remaining special case is EnemyMindblast!

The resulting code is a pair of functions, with their memoized equivalents:

fightVanillaM :: CombatSkill -> Endurance -> Endurance -> Probably (Endurance, Endurance)
fightVanillaM = Memo.memo3 Memo.integral Memo.integral Memo.integral fightVanilla

fightVanilla :: CombatSkill -> Endurance -> Endurance -> Probably (Endurance, Endurance)
fightVanilla ratio php ohp
  | php <= 0 || ohp <= 0 = certain (max 0 php, max 0 ohp)
  | otherwise = regroup $ do
      (odmg, pdmg) <- hits ratio
      fmap (/10) <$> fightVanillaM ratio (php - pdmg) (ohp - odmg)

fightMindBlastedM :: CombatSkill -> Endurance -> Endurance -> Probably (Endurance, Endurance)
fightMindBlastedM = Memo.memo3 Memo.integral Memo.integral Memo.integral fightMindBlasted

fightMindBlasted :: CombatSkill -> Endurance -> Endurance -> Probably (Endurance, Endurance)
fightMindBlasted ratio php ohp
  | php <= 0 || ohp <= 0 = certain (max 0 php, max 0 ohp)
  | otherwise = regroup $ do
      (odmg, pdmg) <- hits ratio
      fmap (/10) <$> fightVanillaM ratio (php - pdmg - 2) (ohp - odmg)

As can be seen, using data-memocombinators is really straightforward for simple types. The result is a performance gain of 100x to 200x, depending on the fight complexity. The test suite now runs in less than 0.1s, which is fast enough to keep on working! I will try factoring the two functions into one, as they share so much code, but only after there is enough code written for having meaningful performance data.

What's left?

From the roadmap, I skipped the part about solving the special chapters, so that will be the next stop.

]]> Solving a gamebook : chapters https://hbtvl.banquise.net/posts/2017-03-14-GBS-chapters.html 2017-03-14T00:00:00Z

Chapter modelisation

This article is a literate Haskell representation of the LoneWolf.Chapter module at the time of the writing.

It doesn't include any code, only type definitions, and will certainly be the least interesting post of this serie. It however shows a possible encoding of the chapter's descriptions, which the reader might find interesting on its own.

The last part discusses how the chapter definitions were partially generated.

Extensions and imports

Clearly not the best part :)

{-# LANGUAGE GADTs #-}
{-# LANGUAGE TemplateHaskell #-}
{-# LANGUAGE DeriveDataTypeable #-}
module LoneWolf.Chapter where

import LoneWolf.Character
import Control.Lens
import Data.Data
import Data.Data.Lens

Basic types

The following types should get a newtype, but to be honest, it is much more handy that way right now, because of the vicious code generation that is happening later ...

-- represents a probability
type Proba = Rational
-- each chapter has an identifier, with a value ranging from 1 to 350
type ChapterId = Int
-- some combat effects are timed, for example some fights can be evaded after a certain number of rounds
type Rounds = Int
-- price of stuff, in gold coins
type Price = Int

What's in a chapter?

Lone Wolf chapters usually have a descriptive text, some figures, a description of what happens to the payer, and a list of choices. The title of the chapter serves no real purpose here, as it is identical (in String form) to the associated ChapterId.

data Chapter = Chapter { _title   :: String
                       , _desc    :: String
                       , _pchoice :: Decision
                       } deriving (Show, Eq)

Here is what an actual chapter looks like:

 Chapter "346"
         "The driver nods and hands back the ticket. The inn is warm but poorly furnished..."
         (Decisions
          [ ( "Buy a meal, and a room"
            , Conditional
                (HasItem Gold 2)
                (NoDecision (LoseItem Gold 2 (Goto 280))))
          , ( "Just the room"
            , Conditional
                (HasItem Gold 1)
                (NoDecision (LoseItem Gold 1 (MustEat NoHunt (Goto 280)))))
          , ("Nothing", NoDecision (MustEat NoHunt (Goto 280)))
          ])

The important part of this model is that the Decision type encodes the player choices, and contains ChapterOutcome fields that encode the outcome of the choices. This is a critical distinction, as it is structural to how the solver works.

The Decision type

In details, the Decision types has the following constructors:

data Decision
   = Decisions [(String, Decision)]
   | CanTake Item Int Decision
   | Canbuy Item Price Decision
   | Cansell Item Price Decision
   | Conditional BoolCond Decision
   | Special SpecialChapter
   | NoDecision ChapterOutcome
   deriving (Show, Eq, Typeable, Data)

data SpecialChapter = Cartwheel
                    | Portholes
                    deriving (Show, Eq, Typeable, Data)

The Decisions constructor represents a list of possible choices for the player, with, for each case, a textual and programmatic description.
The CanTake and CanBuy constructors represent situations where the player can take, or buy, items. CanSell serves an identical purpose, for selling items. These constructors are not necessary, as they could be encoded using other, simpler, primitives. The following are equivalent, with regards to the choices they encode:
```
gaingold = CanTake Gold 10 (NoDecision (Goto 20))

gaingold' = Decisions
  [ ("Take the gold", NoDecision (GainItem Gold 10 (Goto 20)))
  , ("Take 9 coins", NoDecision (GainItem Gold 9 (Goto 20)))
  , ("Take 8 coins", NoDecision (GainItem Gold 8 (Goto 20)))
  , ...
  , ("Leave the gold", NoDecision (Goto 20)
  ]
```
Having specialized constructors helps a lot. First it makes it easier to write the chapter definitions, and makes them clearer. But, more importantly, it will help reducing the computational complexity by deciding locally with specific heuristics.

Some decisions are only accessible in certain conditions, for example:

(Decisions
 [ ( "If you have the Kai Discipline of Tracking, turn to 13."
     , Conditional (HasDiscipline Tracking) (NoDecision (Goto 13)))
 , ( "If you wish to take the left fork, turn to 155."
     , Conditional
         (Not (HasDiscipline Tracking))
         (NoDecision (Goto 155)))
 , ( "If you wish to take the right fork, turn to 293."
     , Conditional
         (Not (HasDiscipline Tracking))
         (NoDecision (Goto 293)))
 ])

In this example, if the player has the Tracking discipline, he will be forced to jump to chapter 13. If not, he will be able to choose between chapters 155 and 293.

Some chapters are special, and will be handled specifically. More on this later!
The NoDecision constructor indicates that there is no further choice for the player, and is at the leaf of most Decision structures. I currently have flagged two chapters as special, but this might evolve in the future.

The ChapterOutcome type

A ChapterOutcome describe what happens to the player once he has made a decision.

data ChapterOutcome
        = Fight FightDetails ChapterOutcome
        | EvadeFight Rounds ChapterId FightDetails ChapterOutcome
        | DamagePlayer Int ChapterOutcome
        | HealPlayer Int ChapterOutcome
        | FullHeal ChapterOutcome
        | HalfHeal ChapterOutcome
        | GainItem Item Int ChapterOutcome
        | LoseItem Item Int ChapterOutcome
        | LoseItemKind [Slot] ChapterOutcome
        | MustEat CanHunt ChapterOutcome

Most of the constructors are built like lists, in the sense that their last argument is a ChapterOutcome.

I know I will have trouble with the EvadeFight element. It represents a fight that the player can evade after a certain number of rounds. It is really problematic, because such fights can have a lot of outcomes during each round, and will make the game unsolvable if implemented naïvely. This will certainly require some form of approximation, but I'll take my time to commit to one.

Another constructor of interest is the MustEat constructor. It represents chapters where the player loses some hit points if he doesn't have a meal in his backpack. In the Lone Wolf series, the Hunting discipline let the player ignore these events. But in this specific book, there are a couple of chapters where Hunting can't be used! In a previous tentative, I used a flag in the game state to indicate whether or not hunting was permitted (it's explicitely told when it stops and when it starts). In general, it is better to make the game state as simple as possible, so as to get good memoization performance, so I decided to manually track the chapters where hunting was disabled, and mark them as such.

        | Goto ChapterId
        | GameLost
        | GameWon

Those three constructors are the "leaves" of this type, as they conclude an outcome. They work in the same way as the [] constructor that concludes a list.

        | Randomly [(Proba, ChapterOutcome)]
        | Conditionally [(BoolCond, ChapterOutcome)]
        deriving (Show, Eq, Typeable, Data)

Finally, those work like trees. They represent random outcomes, or cases where multiple choices are provided to the player, but in reality only one is ever possible depending on the situation. Here are example of the latter, taken from chapter 36:

Conditionally
 [ (HasItem Laumspur 1, Goto 145)
 , (HasDiscipline Healing, Goto 210)
 , (Always True, Goto 275)
 ]

Note how it works like Haskell's guards, and ends up with the equivalent of otherwise, which I called ... botherwise:

botherwise :: BoolCond
botherwise = Always True

data CanHunt = Hunt | NoHunt
             deriving (Show, Eq, Typeable, Data)

data BoolCond = HasDiscipline Discipline
              | Not BoolCond
              | HasEndurance Int
              | COr BoolCond BoolCond
              | CAnd BoolCond BoolCond
              | HasItem Item Int
              | Always Bool
              deriving (Show, Eq, Typeable, Data)

(.&&.) :: BoolCond -> BoolCond -> BoolCond
(.&&.) = CAnd

The boolean conditions type (BoolCond) is pretty self-explanatory.

Combat description

The three main parts of a combat descriptions are the name of the opponent, its combat skill, and its endurance. There are however many combat situations that involve special modifiers which are detailled here. Fights against consecutive opponents are handled by chaining the Fight and EvadeFight constructors.

data FightDetails = FightDetails
          { _opponent     :: String
          , _fcombatSkill :: CombatSkill
          , _fendurance   :: Endurance
          , _fightMod     :: [FightModifier]
          } deriving (Show, Eq, Typeable, Data)

data FightModifier = Undead
                   | MindblastImmune
                   | Timed Int FightModifier
                   | CombatBonus CombatSkill
                   | BareHanded
                   | FakeFight ChapterId
                   | EnemyMindblast
                   | PlayerInvulnerable
                   | DoubleDamage -- chapter 306
                   deriving (Show, Eq, Typeable, Data)

Undead: undead creatures take double damage from the Sommerswerd.
MindblastImmune: the opponent is not affected by the MindBlast ability of the player.
CombatBonus: the player receives a combat bonus for this fight.
BareHanded: the player is forced to fight bare handed, reducing his combat skill by 4 points.
FakeFight: this isn't a real fight, and the player will have his endurance restored at the end of the fight. The ChapterId is provided as the destination if he loses the fight. This happens in chapter 276.
EnemyMindblast: the enemy has the MindBlast ability!
PlayerInvulnerable the player cannot be harmed.
DoubleDamage: happens in chapter 306, where the player inflicts double damage.
Timed: the effect only lasts a given number of rounds.

Lenses, plates and utilities

makePrisms ''ChapterOutcome
makePrisms ''Decision
makeLenses ''FightDetails

moneyCond :: Int -> ChapterOutcome -> Decision
moneyCond price = Conditional (HasItem Gold price) . NoDecision . LoseItem Gold price

outcomePlate :: Traversal' Decision ChapterOutcome
outcomePlate = biplate

All of these are mainly useful in the LoneWolf.XML module.

The actual chapters data

The chapters are described in the LoneWolf.Book02 module, but it's all generated code! The LoneWolf.XML module contains all the code that is necessary to parse the XML file provided by project Aon, compute a baseline, adapt all chapters that have special rules and generate code. The code is horrendous, as it's a one-use program.

In previous iterations of this project, I used to manually generate all the chapter's definition, and chapter 346, that could be seen at the beginning of this article, would look like that:

(346, UnsetFlag CanHunt
    ( Decisions [ ("buy meal", ItemPay Gold 1 (Outcome (Goto 1104)))
                , ("eat own meal", ItemPay Meal 1 (Outcome (Goto 1104)))
                , ("starve", Outcome (ODamage 3 (Goto 1104))) ] ))

Compared to the current version, the chapter's text is missing, and the choice descriptions are not as nice. It was also harder to debug, and one can see the UnsetFlag CanHunt situation I mentioned earlier.

The current situation is a net gain, as it has all the original text content, and also wrote most of the code for me. It was however still a lot of work to read all the chapters and make sure they were properly encoded. I also realized my previous encoding was imperfect in parts (I never realized there was a chapter where you could sell your weapons!), and I have no illusions there are mistakes in the current definitions.

Next time will have actual code, with a rules interpreter!

]]> Solving a gamebook : roadmap https://hbtvl.banquise.net/posts/2017-03-13-GBS-roadmap.html 2017-03-13T00:00:00Z

I will try to achieve the following goals during this serie:

Model the book in the most accurate way. This part is already done, and is kind of failed already, as I removed several elements from the original story : the warm blanket that can be bought/grabbed but serves no actual purpose, and all references to a star pendant that is given to the player in the previous book. This will be detailed in the next post, and in my experience is one of the hardest part to get right, because of how tedious and error-prone it is.
Model the rules in the most accurate way, and provide a console interpreter of the game.
Write a generic solver for single player games, and try to run it on reduced versions of the game.
Optimize and memoize the combat solver. This part is quite tricky, and I found it hard to get the rules right and optimize for efficient memoization at the same time.
Solve the "special chapters" (such as the cartwheel game). They provide a way for the player to gain some money, which is required in later parts of the adventure. Solving this part will require careful analysis of the game structure in order to determine the proper "win conditions" for these games.
Manually alter the book description to get rid of loops, and to reduce the complexity of the player state.
Improve the efficiency of the solver. This part includes writing heuristics about player choice, using checkpoints, or other approximations. I will try to find a good balance between computability and precision.
Solve the game!
???
Generate pretty graphs and provide analysis.

The linked picture comes from a previous try at solving this story. The redder the node, the most likely the player is to traverse it, if he plays optimally. White nodes are "checkpoints", which means the solver would try to solve subsets of the game that lie between two consecutive checkpoints, which greatly helps with the performance. It can however destroy the accuracy of the analysis, as some paths are more risky, but provide the player with items that could make the game easier after the next checkpoint ... The combat solver would also not take into account timed effects (more on that in the next episode!), and some other parts were too approximate.

I hope this serie of posts will prove to be a fun read, and force me to create an accurate solver!

]]> Solving a gamebook : introduction https://hbtvl.banquise.net/posts/2017-03-11-GBS-intro.html 2017-03-11T00:00:00Z

I will start a series about writing a gamebook solver, more specifically the Fire on the Water episode in the Lone Wolf series. There are several advantages in choosing this book: it is freely available online, it is relatively linear (see the graph), and I liked this series when I was much younger ...

For those who did not grow up in the 80's, a gamebook is a book, divided in small chapters. The book tells an interactive story, where the player can make decisions affecting the outcome of the adventure. Each of the chapters describe the current situation, and give a choice to the player, which usually result in jumping to another chapter. As an example, take a look at the first chapter of the book.

In this context, solving the book means writing a program that will be able to decide what the optimal choice is at each chapter. However, most gamebooks include dice rolls, coin tosses or other random effects. This is the case in the Lone Wolf series, where the player is asked to close his eyes, and place a pencil at random on a numbered grid to "roll" a random number ranging from 0 to 9. The solver I will develop should also be able to give the exact success probably associated with each choice.

I have tried twice already to solve a Lone Wolf gamebook, so this blog posts serie will include a few tricks I have learned from the previous failures. Hopefully, the third time's a charm!

]]> Efficient parsing of large text files (4) https://hbtvl.banquise.net/posts/2015-12-14-fastParsing03.html 2015-12-14T00:00:00Z

More careful measurements

The golden standard for benchmarking in Haskell is the criterion library. For this series I did not bother with writing smaller test cases that would play well with it, and my measurement have not be rigorous. For this post, I have run each sample ten times and computed an average. This is still not ideal, but should suffice to give a feeling of the performance.

However, I don't have the measurements for the last iteration of my program. I waited till I had time to collect these, but this never happened for various reasons. Gauging the performance gains of the last version is left as an exercise to the reader!

It turned out that the state of the program at the end of the previous episode was much better than I expected! It performed its task in 2.712 seconds, which is just as fast as the discarding C program. Now let's beat it!

Playing with compiler options

Finding the precise set of compiler options that will produce a fast program is more art than science. People are using genetic algorithms to find the best set of compiler flags, but I am only going to focus on a single flag here : -fllvm. It can do wonder for tight loops and numerical code, but it also can slow down the compilation process and the resulting program.

I have found it best to just test it on performance critical code, and keep it if there was a significant gain.

In this case, the gain is about 3%, which would not be sufficient for me to keep in usual circumstances. But this series is about performance, so I will keep this nice "free" speedup.

Alternative encodings of the monad

An optimization technique that is fun and sometimes very efficient is to move the critical parts of the computation to a continuation passing like implementation. In our case, that would be the Parser monad.

I am used to call this process Church encoding, but it seems from the responses from Reddit that it might not be correct. I have been pointed to a paper describing how to systematically encode all the stuff in what is called Boehm-Berarducci encoding. Despite my claims I did not read it (yet), so there might be inaccuracies in my terminology.

As an example, here is how Maybe and Either are usually encoded:

newtype MaybeChurch a = MaybeChurch
      { runMaybe :: forall x. x -> (a -> x) -> x }
newtype EitherChurch l r = EitherChurch
      { runEither :: forall x. (l -> x) -> (r -> x) -> x }

Those types look a lot like the following functions:

maybe :: b -> (a -> b) -> Maybe a -> b
either :: (a -> c) -> (b -> c) -> Either a b -> c

And it is not surprising, for simple ADT like those you "just" have to write their fold to get their Church encodings (by the way, it is a fun exercise to newtype them and try to write the usual instances).

But our Parser type is a bit more complicated, as it really is a function:

newtype Parser a = Parser
      { runParser :: ByteString -> Maybe (a, ByteString) }

I don't know how to mechanically advance here, so I am going to try to explain my intuition. The original parser takes a ByteString and stops the computation with an empty result (Nothing), or a result along with the remaining input.

Once transformed, it will not directly return anything, but will pass the result of the current computation to a pair of continuation functions, one for each case:

newtype Parser a = Parser
      { runParser :: forall r. ByteString -> nothing -> just -> r }

Now, both the nothing and just part should be functions that must return r. The nothing part handles errors. The just part keeps on parsing, based on the result of the current function. It must be fed with the remaining input and said result:

newtype Parser a = Parser
      { runParser :: forall r. ByteString -> (error -> r) -> (ByteString -> a -> r) -> r }

But as a final optimisation, we might see that we don't care about fancy error messages for the "failure" case, so the final form will be:

newtype Parser a = Parser
      { runParser :: forall r. ByteString -> r -> (ByteString -> a -> r) -> r }
      deriving Functor

For a parser with verbose error messages, a suitable type could be:

newtype Parser a = Parser
      { runParser :: forall r. ByteString
                  -> (String -> r) -- ^ Failing case with a String error message
                  -> (ByteString -> a -> r)
                  -> r }
      deriving Functor

Now, most of the combinators are trivial to write:

takeWhile1 :: (Char -> Bool) -> Parser ByteString
takeWhile1 prd = Parser $ \s failure success -> case BS8.span prd s of
                                                    ("", _) -> failure
                                                    (a,b) -> success b a

char :: Char -> Parser ()
char c = Parser $ \input failure success ->
      if BS.null input
          then failure
          else if BS8.head input == c
                   then success (BS.tail input) ()
                   else failure

And running the parser is easy too:

parseOnly :: Parser a -> ByteString -> Maybe a
parseOnly (Parser p) s = p s Nothing $ \b a -> if BS.null b
                                                   then Just a
                                                   else Nothing

Now, one of the fun properties of Haskell is that deriving the Applicative and Monad instances can actually be done as an exercise in type tetris, where you have no idea of what the code you write actually means, but where it works fine once it typechecks:

instance Applicative Parser where
    pure a = Parser $ \b _ s -> s b a
    Parser pf <*> Parser px = Parser $ \input failure success ->
        let succ' input' f = px input' failure (\i a -> success i (f a))
        in  pf input failure succ'

While this gives no speed gain with the usual compiler options, the performance under llvm is slightly better (2.56s vs 2.63s). This technique can do wonders in many use cases, but is clearly not that great here.

Oh well, this was fun anyway :)

Parallel execution

In the previous episode I moved from a parsing function that would handle the whole input to a parsing function that would only handle a single line. This function being pure, it should be very easy to parallelize its execution.

In practice, I had to change this part of the program:

mapM findline

into:

traverse id . withStrategy (parListChunk 2000 rseq) . map findline

Note that traverse id is sequence, but for some reason I didn't go with it ... This strategy stuff is very well covered in the Marlow book which I recommend even if you don't believe you really care about these topics. It is a really good book!

Basically this will cut the list in chunks of 2000 items and spark a thread for each of them. Each of these threads can be processed on an individual processor.

In practice, the gain is highly dependent on the sequential parts of a program and the complexity of the parallel chunks of work. The latter part means that sometimes it is more work to handle the complexities of the parallel runtime than to do the work in the first place. That means that algorithms that look like they should parallelize well really don't, and that you should benchmark your programs with many values of -N (which is the runtime option that governs how many cores are going to be used).

For this program, I found out that a value of -N6 gave the best results. The program runs in 1.95s with this value.

There isn't much speedup, and it is probably related to the fact that the program does the following sequential operations:

File loading
Parsing
Map creation
Printing the map size

I tried going with lazy IO to increase the productivity of the program, but this worsened the performance. I did not try a streaming library because I suspect the increased complexity will not be worth it.

Performance summary

Here is a breakdown of all that has been discussed in this series. It should be quite obvious that the alternative encoding of the parser could have been left out. It would also be interesting to get a metric of how fast would the final version be while keeping attoparsec. Introducing a custom parser did wonders for the performance of the program, but it might be technical debt if more complex features were to be required.

In the end, the Haskell program could be made faster than a truncated C program that wouldn't do as much work. You will have to take my word for it, as I will not release the C program, but it wasn't a bad program I selected for an easy win.

Program version	Time spent (seconds)	Memory usage (MB)	Gain over first version (CPU / RAM)
parsec	95.741	14542	n/a
attoparsec	19.609	3154	4.88x / 4.61x
thyme	18.302	1975	5.23x / 7.36x
bytestring	13.890	1959	6.89x / 7.42x
BangPatterns	8.705	406	11x / 35.82x
parseDTime	7.7	406	12.43x / 35.82x
Custom parser	5.628	382	17.01x / 38.07x
Break	4.754	396	20.14x / 36.72x
Inlining	2.712	396	35.3x / 36.72x
With LLVM	2.63		36.4x
Church	2.71		35.3x
Church LLVM	2.561		37.4x
Parallel -N2	2.51		38.1x
Parallel -N4	2.13		44.9x
Parallel -N6	1.95		49.1x
original C (mem)	12
original C (discard)	2.7

Conclusion

This series was meant as a way to showcase some simple and effective optimization techniques, such as

Using efficient libraries. For parsing I suggested using attoparsec, but I would suggest using something like the parsers package, as it has a great interface and is really a front-end to several other parsers (attoparsec, parsec and trifecta). That way you can have a "slow" version with good error messages and a "fast" one powered by attoparsec (or your own parser!) using the same code base.
Using BangPatterns, even if a bit randomly, while roughly benchmarking.
Profiling to find the hot spots. This one is obvious, but it is not the first thing I usually do when optimizing. Also be warned that the cost centers might accounted as you might expect when using aggressive optimizations or continuation passing style.
Replacing some well used library with custom, more specialized, code. This is something that should be done only as last resort in production code, as this will reduce its maintainability (compared to a well known library with many useful features).
Experiment with the compiler flags. This can give some "free" speedups, but can also negatively impact compilation times.
Embarrassingly parallel tasks are embarrassingly easy to parallelize in Haskell.

One technique that didn't work was the alternative encoding of the custom parser. This is a technique I use a lot, but I suspect mainly because I like the soothing feeling of successful type-Tetris.

Happy optimizing!

]]> Efficient parsing of large text files (3) https://hbtvl.banquise.net/posts/2015-11-29-fastParsing02.html 2015-11-29T00:00:00Z

Estimating the expected memory usage

The UnixFile type is made of a single constructor containing five Ints, one sum type, three UTCTime, three ByteString and a Maybe ByteString. Given this source, we can compute the size of a "typical" entry, where the user and group names are less than 8 characters long, there is no link and the file name is something like has an average length of 57 bytes:

5 Int = 5 * 8 = 40 bytes
3 UTCTime = 3 Int64 = 24 bytes
2 "small" ByteString = 2 * (9 * 8 + 8) = 160 bytes
1 "large" ByteString = 9 * 8 + 64 = 136 bytes
1 simple sum type = 8 bytes
1 Nothing = 8 bytes
1 tag = 8 bytes

Total: 384 bytes, because all the fields are strict. Note that I didn't check that the sizes for my sum type, Nothing and tag are accurate.

As there are around 570k entries, we can expect a baseline of 213MB of live data, so a total memory usage of 427MB to account for GC copying (if I understand this correctly).

When we left our study, the program was consuming about 2GB of memory, so that's something like five times the expected memory usage!

How to fix the memory leak?

The Haskell runtime has built in options to trace the memory usage of a program, and more importantly the source of the memory allocations. It usually is a good idea to start running in profiling mode, at least with -p to find the hot spots.

I did that, and realized that a lot of time was spent in the timestamp function, along with a large part of the memory allocation.

This function can be divided in several parts : parsing and representing the day, parsing and representing the time, parsing the timezone and adjusting the UTC time. I decided to extract the two first parts in order to have finer grained performance metrics. In the process, I made them stricter by adding bangs. I really wish I could act all wise and knowledgeable in the ways of GHC and its core language, but what I often do when I hit a performance problem is to generously pepper the code with bangs. This is cargo-cult optimization, but it tends to work quite well.

In this particular case, this gave a 37% reduction in execution time, and, more importantly, reduced the memory usage to 406MB, which is very close to the previous guestimation.

This means that the memory leak was successfully discovered and plugged. In retrospect, it was very obvious that this function was part of the problem as this is the only place where calculations are being performed, so this is the only place where thunks could be piling up.

Anyway, now that I am happy with the memory usage, it is time to gun for the execution speed.

Using the simplest alternative

Now that the function was separated in three parts, I decided to run another profiling campaign. Note that profiling isn't accurate in the presence of optimizations or CPS/Church encodings! This example exhibits both sources of profiling inaccuracy: aggressive compilation options (-O2) and the attoparsec implementation of the Parser. Nevertheless, here was the result:

parseDTime     Main       57.0   46.4
findline.t     Main       22.0   33.3
parseYMD       Main        9.3   11.6
main.resultmap Main        4.2    3.5
timestamp      Main        2.5    1.7
myOctal        Main        2.1    2.0
findline       Main        1.4    0.1

What was surprising was the parseDTime was much more expensive than parseYMD. It should be obvious that this was caused by the fact that parseYMD only deals with decimal numbers, whereas I wastefuly used the scientific function to parse the hour and minutes fields. Using decimal on them gave a good (13%) speed increase for little effort.

State of the code and performance

Here is a recap of the performance so far:

Program version	Time spent (seconds)	Memory usage (MB)	Gain over first version (CPU / RAM)
parsec	95.741	14542	n/a
attoparsec	19.609	3154	4.88x / 4.61x
thyme	18.302	1975	5.23x / 7.36x
bytestring	13.890	1959	6.89x / 7.42x
BangPatterns	8.705	406	11x / 35.82x
parseDTime	7.7	406	12.43x / 35.82x
original C (mem)	12
original C (discard)	2.7

At this point of our optimization journey, the code is still safe, concise and easy to understand for a haskeller that knows about applicative functors, and much faster than the original C code!

Can we do better? Let's try!

Writing a custom parser

Writing a custom parser is a simple yet rewarding kata-like exercise. It is even featured in the NICTA course.

I do not suggest doing it for production code though, but I will do it here anyway, because I thought this would be fun (and it will allow for even more fun in the next episode) and really quick.

In this use case, there are only very few features that are required of the parser. No backtracking or error messages are required, and only a few combinators are used. We can start with this:

newtype Parser a = Parser { getParser :: BS.ByteString
                                      -> Maybe (a, BS.ByteString)
                          } deriving Functor

Deriving the required instance is trivial, and just a few primitives, all of them one-liners where useful to me:

This much simpler parser resulted in a nice speed increase (it runs 5.6s, which places the Haskell program as twice as slow as the "discarding" C program!). It turned out later that the Alternative instance wasn't even useful.

The performance gain is nice, but it comes at a cost in code readability: a quarter of the code is now made of stuff really belongs in a library.

Not parsing everything

This technique is one I really would like to emphasize here, as it is really simple, gives good speed boosts and can bring other benefits.

First of all, it could be observed that this file format is line-based. Splitting them and parsing each of them individually can be beneficial in conjunction with parsers that give poor error messages. In this particular case, knowing there is a parse error somewhere in the source file is next to useless, but knowing which line is affected can be sufficient to diagnose the problem.

Another benefit is that instead of running the parsing function on a single large piece of data, it is run on many smaller pieces. This makes parallelization trivial, which will be described in the next episode.

Finally, in this case it is also possible to split each field in the line very easily, which will give a nice speed boost.

The resulting implementation isn't difficult to parse or understand. The main takeaway would be the use of the pattern matching on the result of the split function on line 103.

The code is not as safe as before, as the getInt function doesn't check that it parses actual digits anymore. This check could be added back, but at this point I felt that the performance of the C "discard" code was within reach, and I started being more aggressive in the optimization techniques.

Inlining all the things

A final optimization step was to add INLINE pragmas to the instances of the custom parser and the simple combinators. This resulted in a dramatic speed increase:

Program version	Time spent (seconds)	Memory usage (MB)	Gain over first version (CPU / RAM)
parsec	95.741	14542	n/a
attoparsec	19.609	3154	4.88x / 4.61x
thyme	18.302	1975	5.23x / 7.36x
bytestring	13.890	1959	6.89x / 7.42x
BangPatterns	8.705	406	11x / 35.82x
parseDTime	7.7	406	12.43x / 35.82x
Custom parser	5.628	382	17.01x / 38.07x
Break	4.754	396	20.14x / 36.72x
Inlining	2.871	396	33.34x / 36.72x
original C (mem)	12
original C (discard)	2.7

As a reminder, the "discard" C implementation does parse the file and converts each fields in a manageable format, but doesn't do anything with them and just discards the results. The Haskell programs store all the files in a Map for further processing.

In the next episode, I will perform more accurate benchmarks and introduce a pair of advanced techniques that will let the program run in less than 2 seconds on my sample input, namely Church-encoding the custom parser and using simple parallelism primitives.

]]> Efficient parsing of large text files (2) https://hbtvl.banquise.net/posts/2015-11-28-fastParsing01Attoparsec.html 2015-11-28T00:00:00Z

The simplest way to increase the performance of a program is to use more efficient libraries. Well, at least in Haskell, where refactoring is easy and strong common abstractions are shared by most of the ecosystem. In this post, I will make trivial changes to the first implementation and measure the associated gains.

Measuring performance

I used the RTS option -s to measure performance for this section. Here is the output for the original (parsec) implementation:

 215,523,656,344 bytes allocated in the heap
  33,301,747,648 bytes copied during GC
   7,227,434,968 bytes maximum residency (15 sample(s))
     209,338,048 bytes maximum slop
           14542 MB total memory in use (0 MB lost due to fragmentation)
                                     Tot time (elapsed)  Avg pause  Max pause
  Gen  0     417618 colls,     0 par   24.950s  24.940s     0.0001s    0.0016s
  Gen  1        15 colls,     0 par   23.081s  23.115s     1.5410s    10.3409s
  INIT    time    0.000s  (  0.000s elapsed)
  MUT     time   43.614s  ( 43.598s elapsed)
  GC      time   48.031s  ( 48.055s elapsed)
  EXIT    time    4.061s  (  4.087s elapsed)
  Total   time   95.708s  ( 95.741s elapsed)
  %GC     time      50.2%  (50.2% elapsed)
  Alloc rate    4,941,573,670 bytes per MUT second
  Productivity  49.8% of total user, 49.8% of total elapsed

The most important metrics here are the total memory use and total time elapsed. I ran the programs several times to warm the file cache and picked one output.

Speed and memory usage were pretty stable, but these are not very precise benchmarks. They should be more than sufficient for this study though!

A faster parser combinator libraries : attoparsec

Everybody knows parsec isn't fast, and attoparsec is much faster.

On the top of my head, the most notable difference between the two are:

Parsec has actual error messages. With attoparsec it is almost always something like "satisfy failed" without the error location. This is the reason I almost always write a new parser with parsec, and only switch to attoparsec when bugs have been ironed out. If error messages are paramount for your application, I suggest taking a look at megaparsec or trifecta.
Attoparsec is much faster than parsec, and uses a lot less memory. By the way, if you are in the never-ending intermediate phase of Haskell proficiency, I really suggest that you learn about Church encoding like transformations. They are at work behind the scenes. We will make use of them in a subsequent episode.
Attoparsec is always backtracking. This means you don't need to use the try keyword.
With attoparsec you can grab consecutive characters efficiently. With parsec, you will get a stream of Char that will need to be repacked, whereas attoparsec can extract slices of the parsed text.
Attoparsec has a modern interface. Code is generally more idiomatic, mostly thanks to the use of the standard Applicative operators.

Other alternatives are trifecta (quite slow but great features and error messages) and megaparsec (a fork of parsec with a focus on modernizing its interface and with better error messages). We recently ported language-puppet to megaparsec without any hurdle.

As expected with attoparsec, the source is a bit shorter and tidier. You can find the result here, and the diff here.

There is a questionable choice where a pair of parseInt have been turned into scientific. We will see in subsequent episodes that this had a negative performance impact.

On the performance side, this change divided the execution time by 4.88 and the memory usage by 4.61. That is, with my sample output, 19.61s of clock time and 3154MB of memory.

Reducing the memory footprint : thyme

The other library that I knew could make a big difference was the thyme library. It is rewrite of time with a focus on performance, alledgedly at the expense of correctness.

The change is very localized, but resulted in a small speed boost, and 1GB of memory saved! The reason thyme is more memory efficient than time is that it stores the UTCTime as a newtype of Int64 instead of the more involved structure that time uses.

With this change, the program currently uses 1975MB of memory with the sample input. This sample input only weights about 120MB for 570k entries, so I expected a much higher efficiency at that stage. Of course, unevaluated thunks are crippling the program, but this will be handled in the next episode!

From Text to ByteString

The input is readable text, so the correct data type to represent it is Text. However, as I do not need to perform any text manipulation and am not too concerned about properly handling non-ASCII characters, it might make sense to move to bytestring performance-wise.

The change is trivial, and gives a nice speed boost.

In the actual program I wrote the UnixFileGen was parametrized for the user and group names, file path and link target fields. That way, the parsing and initial analysis would use ByteString for efficiency, and the files flagged as problematic would be converted to Text for further processing.

A note about the original C module

The original C module did the same parsing, but stored all data in a Sqlite database for further processing with Ruby. In order to capture the performance difference between my Haskell implementation and the C program, I modified it with two variants:

The in-memory database version (dubbed mem): this would do the parsing, and insert all the files in an in-memory database. This is a bit unfair compared to my program, as I don't need to serialize to external storage.
The discard version: this version would just parse the file and not do anything about it. This is also a bit unfair, as my program stores the actual results in a Map.

Optimization summary

Program version	Time spent (seconds)	Memory usage (MV)	Gain over first version (CPU / RAM)
parsec	95.741	14542	n/a
attoparsec	19.609	3154	4.88x / 4.61x
thyme	18.302	1975	5.23x / 7.36x
bytestring	13.890	1959	6.89x / 7.42x
original C (mem)	12
original C (discard)	2.7

As we can see, with a sensible choice of libraries we approach the execution speed of the in-memory original C program, at the expense of trivial changes.

In the next episode, I will introduce actual optimisations to the program that will only slightly affect its readability and maintainability.

Can we beat the discard version of the C program? Can we optimize much more without sacrificing readability and maintainability? Without rummaging in the core? All of these questions will be answered in the final episode!

]]> Efficient parsing of large text files (1) https://hbtvl.banquise.net/posts/2015-11-19-fastParsing00Parsec.html 2015-11-19T00:00:00Z

Efficiently parsing large files is one of those exercises that is not obvious for beginner haskellers, especially if ones goal is to keep the source code concise and readable.

In this serie, I will describe an actual use case where I had to greatly optimize my initial implementation. This is a litterate Haskell file, so you can copy and paste to experiment.

The problem is analyzing the output of the following command:

$ find / -fstype nfs -prune -o -path '/proc' -prune -o -path '/sys' -prune -o -printf '%i %n %A++%AZ %T++%TZ %C++%CZ %u %g %k %y %#m %s %p -> %l\n'
2 28 2015-11-19+17:49:59.7719267730+CET 2015-11-10+08:38:41.9129851680+CET 2015-11-10+08:38:41.9129851680+CET root root 4 d 0755 4096 / -> 
15 1 2015-11-19+18:35:11.4559739950+CET 2015-11-10+08:38:41.8849851690+CET 2015-11-10+08:38:41.8849851690+CET root root 0 l 0777 33 /initrd.img -> boot/initrd.img-3.13.0-68-generic
1025 16 2015-11-19+16:38:15.0775743760+CET 2015-11-19+16:38:14.8055743710+CET 2015-11-19+16:38:14.8055743710+CET root root 0 d 0755 4300 /dev -> 
14749 1 2015-11-19+16:38:21.0655744800+CET 2015-11-19+16:38:21.0055744790+CET 2015-11-19+16:38:21.0055744790+CET root disk 0 b 0660 0 /dev/dm-0 -> 
9479 3 2015-11-19+16:38:01.7495741430+CET 2015-11-19+16:38:01.7535741430+CET 2015-11-19+16:38:01.7535741430+CET root vboxusers 0 d 0750 60 /dev/vboxusb -> 
10839 1 2015-11-19+16:38:01.4615741380+CET 2015-11-19+16:38:01.4615741380+CET 2015-11-19+16:39:44.2358533610+CET root root 0 c 0660 0 /dev/kvm -> 
8957 1 2015-11-19+17:35:12.8439113270+CET 2015-11-19+16:38:01.4415741380+CET 2015-11-19+16:38:01.4415741380+CET root root 0 l 0777 3 /dev/dvdrw -> sr0
8954 1 2015-11-19+17:35:12.8439113270+CET 2015-11-19+16:38:01.4415741380+CET 2015-11-19+16:38:01.4415741380+CET root root 0 l 0777 3 /dev/dvd -> sr0
8950 1 2015-11-19+17:35:12.8439113270+CET 2015-11-19+16:38:01.4415741380+CET 2015-11-19+16:38:01.4415741380+CET root root 0 l 0777 3 /dev/cdrw -> sr0
...

This will output the list of all files on the current system if run as root. This list must then be stored in a key-value map where the key is the file name for further analyzis.

I knew this task would be performance sensitive, as I implemented it when porting a Ruby program to Haskell. The Ruby program used a C module to perform the parsing and storage (in a sqlite database), while the analyzis was performed in native Ruby. It ended up a time and memory intensive task.

In this first post, I will write a very direct implementation of the parser and expose the data structures that will be used throughout the serie. In subsequent posts, I will introduce basic optimization strategies and benchmark them.

The import fest

{-# LANGUAGE GeneralizedNewtypeDeriving #-}
{-# LANGUAGE OverloadedStrings #-}
module Main (main) where

import Data.Bits
import Data.Text (Text)
import qualified Data.Text as T
import qualified Data.Text.IO as T
import Data.Time
import System.Environment
import qualified Data.Map.Strict as M
import Control.Applicative
import Data.Char (isSpace, digitToInt)
import Data.Functor.Identity

import Text.Parsec.Text
import Text.Parsec.Char
import qualified Text.Parsec.Token as TOK
import Text.Parsec hiding (many, (<|>), optional)

As can be seen here, the venerable parsec library is used. We recently moved language-puppet to megaparsec for its much nicer error messages and imports (notice how I had to hide functions here), but I am used to parsec ...

I would recommend writing the inital version of performance sensitive parsers for text-based data with parsec, or even better megaparsec, as they give much better error messages than, say, attoparsec. This can prove invaluable when debugging the first versions, and it is usually straightforward to "upgrade" to attoparsec.

The main data structure

This is the record type that represents the meta-data that is available for a file. Note that the records are presented in the exact same order as in the parsed file, which helps write nicer code in applicative style.

I chose Text to represent user and group names, and let the file pathes as FilePath, which is just an alias to String. The reason is that most functions in base that deal with file pathes use this type (yes, this sucks). Time is represented with the time library. File permissions are stored in an Int, and bit operations are used to query them, instead of a more idiomatic data structure that could look like:

data Permissions = Permissions { _special :: Special
                               , _user    :: Perms
                               , _group   :: Perms
                               , _others  :: Perms
                               }

data AtomicPerm = R | W | X
                deriving (Eq, Ord)

newtype Perms = Perms (Set AtomicPerm)

The reason is that memory usage must be kept low, and manipulating the permissions with lenses is just as convenient as with a more explicit data structure (because you can write a Prism between the two representations).

data UnixFile = UnixFileGen { _fileInode     :: Int
                            , _fileHardLinks :: Int
                            , _fileAtime     :: UTCTime
                            , _fileMtime     :: UTCTime
                            , _fileCtime     :: UTCTime
                            , _fileUser      :: Text
                            , _fileGroup     :: Text
                            , _fileBlocks    :: Int
                            , _fileType      :: FileType
                            , _filePerms     :: FPerms
                            , _fileSize      :: Int
                            , _filePath      :: FilePath
                            , _fileTarget    :: Maybe FilePath
                            } deriving (Show, Eq)

data FileType = TFile
              | TDirectory
              | TLink
              | TPipe
              | TSocket
              | TBlock
              | TChar
              | TDoor
              deriving (Eq, Show)

newtype FPerms = FPerms Int
               deriving (Show, Ord, Eq, Num, Bits)

The token declaration

With parsec, when you want to use built-in parsers such as integer, you have to generate a GenTokenParser structure by using a language definition.

This is quite cumbersome in our use case, as we will only be parsing very simple types. Also the pre-defined language definitions only work for String parsers, whereas I would like to use a Text parser! The following code is just copy-pasta from the default language definition.

tok :: TOK.GenTokenParser Text () Identity
tok = TOK.makeTokenParser TOK.LanguageDef
        { TOK.commentStart   = ""
        , TOK.commentEnd     = ""
        , TOK.commentLine    = ""
        , TOK.nestedComments = True
        , TOK.identStart     = letter <|> char '_'
        , TOK.identLetter    = alphaNum <|> oneOf "_'"
        , TOK.opStart        = alphaNum
        , TOK.opLetter       = oneOf ":!#$%&*+./<=>?@\\^|-~"
        , TOK.reservedOpNames= []
        , TOK.reservedNames  = []
        , TOK.caseSensitive  = True
        }

It is now possible to write an arbitrary numeric parser this way:

parseInt :: Num a => Parser a
parseInt = fromIntegral <$> TOK.integer tok

I had to write my own octal parser as the parsec one expected those literals to start with 0o.

myOctal :: Parser Int
myOctal = foldl (\acc n -> acc * 8 + digitToInt n) 0 <$> some digit

Parsing the file type

char2ft :: Char -> Maybe FileType
char2ft x = case x of
                '-' -> Just TFile
                'f' -> Just TFile
                'd' -> Just TDirectory
                'l' -> Just TLink
                'p' -> Just TPipe
                's' -> Just TSocket
                'b' -> Just TBlock
                'c' -> Just TChar
                'D' -> Just TDoor
                _ -> Nothing

filetype :: Parser FileType
filetype = anyChar >>= maybe (fail "invalid file type") return . char2ft

Pretty straightfoward. The interested reader might notice the door file type, which isn't that common.

Parsing the time stamp

I don't really know how to manipulate timezones, so you can see I did something terrible in the timestamp function. I would be interested in a correction!

timestamp :: Parser UTCTime
timestamp = do
    y <- parseInt <* char '-'
    m <- parseInt <* char '-'
    d <- parseInt <* char '+'
    h <- parseInt <* char ':'
    mi <- parseInt <* char ':'
    s <- realToFrac <$> TOK.float tok <* char '+'
    let day = fromGregorian y m d
        difftime = h * 3600 + mi * 60 + s
        tm = UTCTime day difftime
    tz <- some upper <* spaces
    return $ case tz of
                 "CEST" -> addUTCTime (-7200) tm
                 "CET" -> addUTCTime (-3600) tm
                 _ -> tm

Parsing the actual file description

There is a trick here to parse the final part, as it could look as the following:

/foo -> 
/foo -> bar
/foo bar -> 
/foo bar -> baz
/foo bar -> baz qux

The end of the line is split using the words function and broken around the -> token to separate the file path from the link target. This is definitely not correct, as file names containing multiple spaces, tabs, or the -> sequence will be incorrectly decoded. This will however suffice for this discussion.

findline :: Parser UnixFile
findline = do
    let t :: Parser a -> Parser a
        t parser = parser <* spaces
    meta <- UnixFileGen <$> parseInt
                        <*> parseInt
                        <*> timestamp
                        <*> timestamp
                        <*> timestamp
                        <*> t (T.pack <$> some (satisfy (not . isSpace)))
                        <*> t (T.pack <$> some (satisfy (not . isSpace)))
                        <*> parseInt
                        <*> t filetype
                        <*> (FPerms <$> myOctal)
                        <*> parseInt
    rst <- words <$> t (some (satisfy ( /= '\n' )))
    return $ case break (== "->") rst of
                 (a, []) -> meta (unwords a) Nothing
                 (a, ["->"]) -> meta (unwords a) Nothing
                 (a, b) -> meta (unwords a) (Just (unwords b))

Annnnnnnnnd the actual parsing

The file is loaded at once using the readFile function, resulting in a strict Text. This means that the processing will only start once the file has been completely read.

parseFile :: FilePath -> IO [UnixFile]
parseFile fp = either (error . show) id . parse (many findline <* eof) fp <$> T.readFile fp

The resulting list of files is then stored in a strict map (where the key is the file path), and its size computed to make sure all entries have been inserted in the map.

main :: IO ()
main = do
    parsed <- getArgs >>= mapM parseFile
    let resultmap = M.fromList $ map (\f -> (_filePath f, f)) $  concat parsed
    print $ M.size resultmap

Performance goals

The file list that needs to be parsed can be quite large (usually around 100mb, but I had gigabyte specimens), and must be processed on a typical laptop. I would like the end program to take at most 1GB of memory for a 100MB source file, and the parsing and map creation to be as fast as possible, and under 5 seconds.

On my workstation, and with my test input of 570k entries, this program consumes 14.5GB of memory and takes around 95 seconds to finish.

In the following episodes, I will expose common optimization techniques and the impact they will have on program performance. The final program will consume 400MB of memory and run in less than 3 seconds, while remaining concise.

]]> Servant, persistent, and DSLs https://hbtvl.banquise.net/posts/2015-06-28-servantintegration.html 2015-06-28T00:00:00Z

A few days ago I hinted on how I used servant with persistent on reddit. My approach is about using a few DSLs to simplify the definition of the web application, and to make sure that all important side effects (authentication, access control, database failures, etc.) are handled in a single place. In this post I will define a DSL for access right management and another for writing the webservice itself.

I tried to extract the gist of my engine in that post, so this might seem a bit overengineered for such a simple example. Also I just made sure it compiled, and did not test it, so it might be buggy :)

This post is a literate Haskell file. It assumes you are already knowledgeable about servant, persistent and operational (for the last one, being familiar with free will be enough).

{-# LANGUAGE KitchenSink #-}

I am using servant, persistent and lens in this example, so there is quite a bit of boilerplate, as expected.

{-# LANGUAGE ConstraintKinds            #-}
{-# LANGUAGE DataKinds                  #-}
{-# LANGUAGE DeriveGeneric              #-}
{-# LANGUAGE FlexibleContexts           #-}
{-# LANGUAGE FlexibleInstances          #-}
{-# LANGUAGE GADTs                      #-}
{-# LANGUAGE GeneralizedNewtypeDeriving #-}
{-# LANGUAGE MultiParamTypeClasses      #-}
{-# LANGUAGE OverloadedStrings          #-}
{-# LANGUAGE PolyKinds                  #-}
{-# LANGUAGE QuasiQuotes                #-}
{-# LANGUAGE TemplateHaskell            #-}
{-# LANGUAGE TypeFamilies               #-}
{-# LANGUAGE TypeOperators              #-}

module Main where

import           Control.Lens
import           Control.Monad
import           Control.Monad.Error.Class
import           Control.Monad.IO.Class
import           Control.Monad.Logger
import qualified Control.Monad.Operational  as O
import           Control.Monad.Operational  hiding (view)
import           Control.Monad.Reader       (ask)
import           Control.Monad.Trans.Either
import           Data.Aeson
import           Data.Int
import qualified Data.Foldable              as F
import           Database.Persist.Sql
import           Database.Persist.Sqlite
import           Database.Persist.TH
import           Data.Text.Lens
import           Data.Text                  (Text)
import qualified Network.Wai.Handler.Warp
import           Servant

import AccessType

The AccessType module just exports the following definition, along with all the instances required to use it with servant and persistent :

data AccessType = NoAccess | ReadOnly | ReadWrite | Owner

Persistent model

A person is a user of the blog. They can have a special attribute that marks them as administrators, meaning they have all rights.

share [ mkPersist sqlSettings { mpsGenerateLenses = True }
      , mkMigrate "migrateAll"]
      [persistLowerCase|
Person json
    name Text
    isAdmin Bool
    UniqueName name
    deriving Show

The BlogPost definition is pretty basic.

BlogPost json
    title Text
    content Text
    deriving Show

Finally we have a table that associates an access right to a person. When no matching record exists, the access right is equivalent to NoAccess.

PostRights json
    person PersonId
    post BlogPostId
    access AccessType
    UniqueRight person post
    deriving Show
|]

API definition

I use a few tricks to define the API. The first one was stolen from this reddit comment:

type CRUD a = DN :> ReqBody '[JSON] a :> Post '[JSON] (MKey a) -- create
         :<|> DN :> Capture "id" (MKey a) :> Get '[JSON] a -- read
         :<|> DN :> Capture "id" (MKey a) :> ReqBody '[JSON] a :> Put '[JSON] () -- update
         :<|> DN :> Capture "id" (MKey a) :> Delete '[JSON] () -- delete

It lets you factor the four endpoints you will need for a basic CRUD interface, and will prove handy during the API definition. It uses some helper types:

type DN = Header "dn" Text

DN is a header that is provided by the web server. There is a web-facing server that handles the TLS authentication, adds the dn header when it succeeds or drops the connection when it fails. This field contains the distinguished name of the certificate the user presented. I can trust this value as much as I can trust my PKI, and will not verify it in the application code.

newtype MKey a = MKey { getMKey :: Int64 }
               deriving (FromJSON, ToJSON, FromText)

_MKey :: ToBackendKey SqlBackend a => Iso' (MKey a) (Key a)
_MKey = iso (toSqlKey . getMKey) (MKey . fromSqlKey)

The _MKey is a newtype that is isomorphic to persistent's Key. It is needed because Key does not have the instances I need to use it with servant. In this case the user-facing id of the entities will be their primary key in the database. Don't try this at home!

The API itself is just made of CRUD enpoints for Person and BlogPost :

type MyApi = "person" :> CRUD Person
        :<|> "post"   :> CRUD BlogPost

myApi :: Proxy MyApi
myApi = Proxy

Permission checking DSL

The first DSL that I am going to introduce is for permission checking. This will look overengineered in this toy program, but it turns out I can have funky permissions for my use case.

I use the operational package, mostly because I am used to it. It is overkill as this particular DSL probably only requires an Applicative instance. But as the next DSL requires a full Monad instance, both will be defined using the same machinery to keep things consistent.

Let's start with defining the actions :

type PermProgram = Program PermCheck
data PermCheck a where
    IsAdmin       :: PermCheck Bool
    BlogPostRight :: Key BlogPost -> PermCheck AccessType

I just have a pair of them here, but you should add one each time there is a new kind of access type you will need to enforce.

Now here are a few helper functions :

-- checks that the current user is an administrator
isAdmin :: PermProgram Bool
isAdmin = singleton IsAdmin

-- returns the access right of the current user on a particular blog post
blogPostRight :: Key BlogPost -> PermProgram AccessType
blogPostRight = singleton . BlogPostRight

-- checks that an `AccessType` is at least `ReadOnly`
ro :: PermProgram AccessType -> PermProgram Bool
ro = fmap (>= ReadOnly)

-- checks that an `AccessType` is at least `ReadWrite`
rw :: PermProgram AccessType -> PermProgram Bool
rw = fmap (>= ReadWrite)

-- checks that an `AccessType` is at least `Owner`
owner :: PermProgram AccessType -> PermProgram Bool
owner = fmap (>= Owner)

-- helper function for actions that everyone can perform
always :: PermProgram Bool
always = pure True

-- operator for "or-ing" two permission checking actions
(.||) :: PermProgram Bool -> PermProgram Bool -> PermProgram Bool
(.||) = liftM2 (||)

-- operator for "and-ing" two permission checking actions
(.&&) :: PermProgram Bool -> PermProgram Bool -> PermProgram Bool
(.&&) = liftM2 (&&)

And here is the kind of expression you can write :

isAdmin .|| (ro (blogPostRight postid) .&& owner (commentRight commentid))

Web application DSL

This DSL is a bit more complicated. It exposes primitives for interacting safely with the database and a way to cancel the computation and throw errors at the user. We expect the error throwing part to cancel the current SQL transaction and spit an error message to the webservice user.

The instructions are a direct mapping of their counterpart in persistent. The WebService monad is not an instance of MonadError ServantErr because "catching" an exception would not make sense (it would require nested SQL transactions, which I know nothing about). As all sort of constraints on our values are required to make persistent happy, the PC constraint synonym as been created.

In the real life, you will probably need a more versatile database access (arbitrary selects and deletes using esqueleto), logging and other goodies. They are not required for our simple example.

type WebService = Program WebAction
type PC val = (PersistEntityBackend val ~ SqlBackend, PersistEntity val)
data WebAction a where
    Throw :: ServantErr               -> WebAction a
    Get   :: PC val => Key val        -> WebAction (Maybe val)
    Del   :: PC val => Key val        -> WebAction ()
    GetBy :: PC val => Unique val     -> WebAction (Maybe (Entity val))
    New   :: PC val =>            val -> WebAction (Key val)
    Upd   :: PC val => Key val -> val -> WebAction ()

-- throws an error
throw :: ServantErr -> WebService a
throw = singleton . Throw

-- dual of `persistent`'s `get`
mget :: PC val => Key val -> WebService (Maybe val)
mget = singleton . Get

-- dual of `persistent`'s `getBy`
mgetBy :: PC val => Unique val ->  WebService (Maybe (Entity val))
mgetBy = singleton . GetBy

-- dual of `persistent`'s `insert`
mnew :: PC val => val ->  WebService (Key val)
mnew = singleton . New

-- dual of `persistent`'s `update`
mupd :: PC val => Key val -> val -> WebService ()
mupd k v = singleton (Upd k v)

-- dual of `persistent`'s `delete`
mdel :: PC val => Key val -> WebService ()
mdel = singleton . Del

-- like `mget` but throws a 404 if it could not find the corresponding record
mgetOr404 :: PC val => Key val -> WebService val
mgetOr404 = mget >=> maybe (throw err404) return

-- like `mgetBy` but throws a 404 if it could not find the corresponding record
mgetByOr404 :: PC val => Unique val -> WebService (Entity val)
mgetByOr404 = mgetBy >=> maybe (throw err404) return

Evaluating the permissions checking DSL

Now that we have defined the web application DSL, it's trivial to evaluate our expression checking DSL into it.

-- Given the current user, runs a `PermProgram` in the `WebService` monad.
checkPerms :: Entity Person -> PermProgram a -> WebService a
checkPerms ent cnd = eval (O.view cnd)
    where
        usr = entityVal ent
        userkey = entityKey ent
        eval :: ProgramView PermCheck a -> WebService a
        eval (Return a) = return a
        eval (IsAdmin :>>= nxt) =
              checkPerms ent (nxt (_personIsAdmin usr))

There might be no matching record when retrieving the access rights a given user has on a given blog post. In that case, NoAccess should be used.

        eval (BlogPostRight k :>>= nxt) =
              mgetBy (UniqueRight userkey k)
                  >>= checkPerms ent
                    . nxt
                    . maybe NoAccess (_postRightsAccess . entityVal)

Evaluating the web application DSL

Now we are going to turn the WebService type into something that can be used with persistent :

type ServantIO a = SqlPersistT (LoggingT (EitherT ServantErr IO)) a

The conversion practically writes itself, except for handling rollbacks. There is only one thing you should take care of: never use a bare throwError, except in the Throw handler, as the whole point of this exercise is to ensure that sessions are properly rollbacked in case of errors.

runServant :: WebService a -> ServantIO a
runServant ws = case O.view ws of
                  Return a -> return a
                  a :>>= f -> runM a f

runM :: WebAction a -> (a -> WebService b) -> ServantIO b
runM x f = case x of
    Throw rr@(ServantErr c rs _ _) -> do
                  conn <- ask
                  liftIO $ connRollback conn (getStmtConn conn)
                  logOtherNS "WS" LevelError (show (c,rs) ^. packed)
                  throwError rr
    Get k    -> get k       >>= tsf
    New v    -> insert v    >>= tsf
    Del v    -> delete v    >>= tsf
    GetBy u  -> getBy u     >>= tsf
    Upd k v  -> replace k v >>= tsf
  where
      tsf = runServant . f

Implementing the API

I start with defining a record that will hold the permission checking functions for my four CRUD actions:

data PermsFor a = PermsFor { _newPerms :: PermProgram Bool
                           , _getPerms :: Key a -> PermProgram Bool
                           , _updPerms :: Key a -> PermProgram Bool
                           , _delPerms :: Key a -> PermProgram Bool
                           }

adminOnly :: PermsFor a
adminOnly = PermsFor isAdmin (const isAdmin) (const isAdmin) (const isAdmin)

The runCrud function handles most of the logic. It needs as arguments a connection pool, the previously defined record, and two optional functions.

The first optional function is used when creating a new item in the database. It is here to setup appropriate rights for the object that was just created (set its owner).

The second optional function is used to perform additional cleanup before object deletion. Its main use is to remove cross references.

As a result you get all four crud actions defined and ready to use by the serve function!

runCrud :: (PersistEntity a, ToBackendKey SqlBackend a, PC b)
        => ConnectionPool -- ^ Connection pool
        -> PermsFor a -- ^ Permission checking record
        -> Maybe (Key Person -> Key a -> AccessType -> b)
           -- ^ Extra actions after creation
        -> Maybe (Key a -> WebService ()) -- ^ Extra actions after deletion
        -> (      Maybe Text           -> a -> EitherT ServantErr IO (MKey a))
           :<|> ((Maybe Text -> MKey a      -> EitherT ServantErr IO a)
           :<|> ((Maybe Text -> MKey a -> a -> EitherT ServantErr IO ())
           :<|> ( Maybe Text -> MKey a      -> EitherT ServantErr IO ()))
           )
runCrud pool (PermsFor pnew pget pupd pdel) rightConstructor predelete =
          runnew :<|> runget :<|> runupd :<|> rundel
    where
        auth Nothing _ = throw err401
        auth (Just dn) perm = do
            user <- mgetBy (UniqueName dn) >>= maybe (throw err403) return
            check <- checkPerms user perm
            unless check (throw err403)
            return user
        runget dn mk = runQuery $ do
            let k = mk ^. _MKey
            void $ auth dn (pget k)
            mgetOr404 k
        runnew dn val = runQuery $ do
            usr <- auth dn pnew
            k <- mnew val
            F.mapM_ (\c -> mnew (c (entityKey usr) k Owner)) rightConstructor
            return (k ^. from _MKey)
        runupd dn mk val = runQuery $ do
            let k = mk ^. _MKey
            void $ auth dn (pupd k)
            mupd k val
        rundel dn mk = runQuery $ do
            let k = mk ^. _MKey
            void $ auth dn (pdel k)
            F.mapM_ ($ k) predelete
            mdel k
        runQuery :: WebService a -> EitherT ServantErr IO a
        runQuery ws = runStderrLoggingT $ runSqlPool (runServant ws) pool

The last touch is a default null action that is going to be used when you do not need to setup extra stuff after object creation. You cannot just throw a Nothing as the type inference will fail (in the definition of runCrud you can see there is a b that needs to be known, even if it is not used). I picked randomly one of the available types that satisfy the PersistEntity constraint.

-- A default action for when you need not run additional actions after creation
noCreateRightAdjustment :: Maybe (Key Person -> Key a -> AccessType -> PostRights)
noCreateRightAdjustment = Nothing

Serving the API

The server function takes a ConnectionPool and can be directly served by ... the serve function from servant. It should be painless to add additional CRUD endpoints.

server :: ConnectionPool -> Server MyApi
server pool =
      runCrud pool adminOnly noCreateRightAdjustment Nothing
 :<|> defaultCrud blogPostRight PostRights Nothing
   where
     editRights c cid = rw (c cid) .|| isAdmin
     delRights c cid = owner (c cid) .|| isAdmin
     defaultPermissions c =
          PermsFor always
                  (const always)
                  (editRights c)
                  (delRights c)
     defaultCrud c r d = runCrud pool (defaultPermissions c)
                                 (Just r) d

Finally, the main function creates the connection pool, run the migration scripts and starts the web service.

main :: IO ()
main = do
  pool <- runStderrLoggingT $ do
      p <- createSqlitePool ":memory:" 1
      runSqlPool (runMigration migrateAll) p
      return p
  Network.Wai.Handler.Warp.run 8080 (serve myApi (server pool))

Conclusion

I am not sure how this post went. When I started it, I expected it to be just a few lines long, but it turned out I had to add all kind of parts to have a complete example that somebody else could compile and run. Hopefully it demonstrates how easy it is to integrate different libraries with the help of DSL glue.

What it didn't show is how useful this might be for testing, as you "just" need to write a pure variant of runServant that will mock the database endpoints to verify your logic.

This might also look overengineered, but I wrote all this machinery for a webservice that has around 100 endpoints, so the ability to concisely describe the endpoint logic and expected access rights pays off well.

]]> 7 Startups - part 5 - the XMPP backend https://hbtvl.banquise.net/posts/2014-08-09-7-startups-part-5-the-xmpp-backend.html 2014-08-09T00:00:00Z

Note: I ran out of time weeks ago. I could never finish this serie as I envisionned, and I don't see much free time on the horizon. Instead of letting this linger forever, here is a truncated conclusion.

In the previous episode, I added a ton of STM code and helper functions in several 15 minutes sessions. The result was not pretty, and left me dissatisfied.

For this episode, I decided to release my constraints. For now, I am only going to support the following :

The backend list will not be dynamic : a bunch of backends are going to be registered once, and it will be not be possible to remove an existing or add a previous backend once this is done.
The backends will be text-line based (XMPP and IRC are good protocols for this). This will unfortunately make it harder to write a nice web interface for the game too, but given how much time I can devote to this side-project this doesn't matter much ...

The MVC paradigm

A great man once said that "if you have category theory, everything looks like a pipe. Or a monad. Or a traversal. Or perhaps it's a cosomething". With the previously mentionned restrictions, I was able to shoehorn my problem in the shape of the mvc package, which I wanted to try for a while. It might be a bit different that what people usually expect when talking about the model - view - controller pattern, and is basically :

Some kind of pollable input (the controllers),
a pure stream based computation (the model), sporting an internal state and transforming the data coming from the inputs into something that is passed to ...
... IO functions that run the actual effects (the views).

Each of these components can be reasoned about separately, and combined together in various ways.

There is however one obvious problem with this pattern, due to the way the game is modeled. Currently, the game is supposed to be able to receive data from the players, and to send data to them. It would need to live entirely in the model for this to work as expected, but the way it is currently written doesn't make it obvious.

It might be possible to have the game be explicitely CPS, so that the pure part would run the game until communication with the players is required, which would translate nicely in an output that could be consumed by a view.

This would however require some refactoring and a lot of thinking, which I currently don't have time for, so here is instead how the information flows :

Information flow

Here PInput and GInput are the type of the inputs (respectively from player and games). The blue boxes are two models that will be combined together. The pink ones are the type of outputs emitted from the models. The backends serve as drivers for player communication. The games run in their respective threads, and the game manager spawns and manages the game threads.

Comparison with the "bunch of STM functions" model

I originally started with a global TVar containing the state information of each players (for example if they are part of a game, still joining, due to answer to a game query, etc.). There were a bunch of "helper functions" that would manipulate the global state in a way that would ensure its consistency. The catch is that the backends were responsible for calling these helper functions at appropriate times and for not messing with the global state.

The MVC pattern forces the structure of your program. In my particular case, it means a trick is necessary to integrate it with the current game logic (that will be explained later). The "boSf" pattern is more flexible, but carries a higher cognitive cost.

With the "boSf" pattern, response to player inputs could be :

Messages to players, which fits well with the model, as it happened over STM channels, so the whole processing / state manipulation / player output could be of type Input -> STM ().
Spawning a game. This time we need forkIO and state manipulation. This means a type like c :: Input -> STM (IO ()), with a call like join (atomically (c input)).

Now there are helper functions that return an IO action, and some that don't. When some functionnality is added, some functions need to start returning IO actions. This is ugly and makes it harder to extend.

Conclusion of the serie

Unfortunately I ran out of time for working on this serie a few weeks ago. The code is out, the game works and it's fun. My original motivation for writing this post was as an exposure on basic type-directed design to my non-Haskeller friends, but I think it's not approachable to non Haskellers, so I never shown them.

The main takeaways are :

Game rules

The game rules have first been written with an unspecified monad that exposed several functions required for user interaction. That's the reason I started with defining a typeclass, that way I wouldn't have to worry about implementing the "hard" part and could concentrate on writing the rules instead. For me, this was the fun part, and it was also the quickest.

As of the implementation of the aforementionned functions, I then used the operational package, that would let me write and "interpreter" for my game rules. One of them is pure, and used in tests. There are two other interpreters, one of them for the console version of the game, the other for the multi-backends system.

Backend system

The backends are, I think, easy to expand. Building the core of the multi-game logic with the mvc package very straightforward. It would be obvious to add an IRC backend to the XMPP one, if there weren't that many IRC packages to choose from on hackage ...

A web backend doesn't seem terribly complicated to write, until you want to take into account some common web application constraints, such as having several redundant servers. In order to do so, the game interpreter should be explicitely turned into an explicit continuation-like system (with the twist it only returns on blocking calls) and the game state serialized in a shared storage system.

Bugs

My main motivation was to show it was possible to eliminate tons of bug classes by encoding of the invariants in the type system. I would say this was a success.

The area where I expected to have a ton of problems was the card list. It's a tedious manual process, but some tests weeded out most of the errors (it helps that there are some properties that can be verified on the deck). The other one was the XMPP message processing in its XML horror. It looks terrible.

The area where I wanted this process to work well was a success. I wrote the game rules in one go, without any feedback. Once they were completed, I wrote the backends and tested the game. It turned out they were very few bugs, especially when considering the fact that the game is a moderately complicated board game :

One of the special capabilities was replaced with another, and handled at the wrong moment in the game. This was quickly debugged.
I used traverse instead of both for tuples. I expected them to have the same result, and it "typechecked" because my tuple was of type (a,a), but the Applicative instance for tuples made it obvious this wasn't the case. That took a bit longer to find out, as it impacted half of the military victory points, which are distributed only three times per game.
I didn't listen to my own advice, and didn't take the time to properly encode that some functions only worked with nonempty lists as arguments. This was also quickly found out, using quickcheck.

The game seems to run fine now. There is a minor rule bugs identified (the interaction between card-recycling abilities and the last turn for example), but I don't have time to fix it.

There might be some interest with the types of the Hub, as they also encode a lot of invariants.

Also off-topic, but I really like using the lens vocabulary to encode the relationship between types these days. A trivial example can be found here.

The game

That might be the most important part. I played a score of games, and it was a lot of fun. The game is playable, and just requires a valid account on an XMPP server. Have fun !

]]> 7 Startups - part 4 - Adding an asynchronous backend https://hbtvl.banquise.net/posts/2014-05-22-7-startups-part-4-adding-an-asynchronous-backend.html 2014-05-22T00:00:00Z

This one will be an experimental post, as I have just added a ton of new code. I did not have enough time to do this properly, so this looks like a giant tangle of STM functions now. I will probably rewrite a large part of it, but I still think the though process that led me to this could be interesting to others, so here we are.

TLDR: I wrote a lot of code, it sucks, and I will rewrite a large part of it for the next episode.

Stuff that was refactored since last time

A pair of minor items :

I fixed the PrettyElement problem here
I changed the type of playerActionsDialog so that it only accepts NonEmpty lists.
I did the same for allowableActions. I also rendered the function partial by mistake, can you spot the bug ? :)

The big change came from the fact that I realized my operation types were wrong. In particular this one :

AskCard :: Age -> PlayerId -> NonEmpty Card -> Message -> GameInstr Card

This type seemed right for writing the game rules, and the console version. However, it does suck for a multiplayer version, as this code will ask the second player for his choice only after the first one has answered. This will slow down the game considerably. We should query all players at once, and wait for their answers after that. I decided to model this as an abstract promise, ie. a value container that will eventually be filled. There is a new type parameter p for the GameInstr type, along with a new GetPromise instruction.

Now, all players are consulted at the same time, and the game then waits for them to answer (code).

This is all very abstract, but in practice things are not that simple, and the promise might not get fulfilled. One problem is a player disconnecting from a backend. One way to do this would be to make the return value of the getPromise function be an Either of some sort. But the only sensible option in the game logic would be to throw an error, so instead the interpreter's getPromise can fail, but not the version that's exposed to the game.

For the pure backend, the promise type is just Identity, as seen here.

An ambitious step

I decided to get a bit more ambitious for this episode. I wanted to implement code that would be able to run several games at once, over several medias at once, with player being connected on any media combination. I did not write a simple backend to start with, to get a feel of where the actual problems were, and decided to write the code top-down.

So here is what I had in mind :

The layers

So basically, there would be a "Hub" that would hold the player list, who is playing which game, and that would also run the games. Backends would interact with it to provide IO with the players. As IRC and XMPP have the same kind of communication model, they would be merged in a single "virtual backend" that would communicate with a pair of "real backends". Now how to do that ?

Asynchronous communication

Both backends need to listen to two kinds of events :

Requests from the game, such as asking a specific player to choose a card.
Instant messages from a server, a channel, or a person.

From the point of view of a game, the messages from the players are usually of no interest. It just needs them to choose a card, or an action to play from times to times. The backends, however, will need to watch out for administrative commands. This means there should be a lot of filtering.

The main problem resides in asking something to a player, and get his answer. An extremely naive way to go about this would be something along the lines of:

sendTextContent session (RUser playerid) message
resp <- getMessage session

This would be completely wrong because we are not assured the next message will be the one we are expecting. So instead, we need to implement some sort of callbacks, so that when a message arrives, the backend would check if we were expecting something from the sender, and react accordingly. This means that we need an extra thread that will just wait for messages, and handle administrative commands and callbacks. So something like :

forkIO $ forever $ do
    msg <- getMessage session
    let pid = getPlayerId msg
    case callbacks ^. at pid of
        Just cb -> cb msg
        Nothing -> case getContent msg of
            "xxx" -> adminCommandXxx
            "yyy" -> adminCommandYyy
            _     -> return ()
runGame

Where the game would do something like this to implement, for example, the "ask card" function:

askCard :: PlayerId -> NonEmpty Card -> m (promise Card)
askCard pid necards = do
    let msg = buildAskCardMessage necards
    sendMessage session pid msg
    p <- newPromise
    addCallback pid $ \msg ->
        case decodeMsg of
            Just x -> fulfill p x
            Nothing -> askAgain
    return p

Multiple backends

So that was cool, but what if there are multiple backends ? All of them must be able to fulfill that promise ! What I would like to do is to be able to return a promise that will be fulfilled at a later date in another part of my program. Usually, this would be something like that :

promise :: IO a -> IO (Promise a)
getPromise :: Promise a -> IO a

But I decided most of my function will live in the STM (I did not document this choice, but the STM is so useful it's a no brainer), and I wanted to write code anyway.

Because of "not invented here", I wrote my own implementation, gave it a bad name (PubSub) and wrote buggy Monoid instances. Code is here and is wrong on many levels. It exposes this set of functions :

newPubSub    :: STM (PubFP e a, SubFP e a)
fulfillPub   :: PubFP e a -> a -> STM ()
failPub      :: PubFP e a -> e -> STM ()
getResult    :: SubFP e a -> STM (Either e a)
addPubAction :: STM () -> PubFP e a -> PubFP e a

The newPubSub gives you a pair of values, one of them you can publish to (using fulfillPub for success and failPub for failure), and one of them you can get results from (with a blocking getResult).

The name is wrong because getResult will always return the same value, so this does not behave at all like a stream, which could be implied by the PubSub name.

The addPubAction is horrible too. I only had a sketchy idea that I needed callbacks at some point when I wrote this module, and that these callbacks should be lifted as soon as possible, so probably as soon as a response is published. This is wrong because :

The type here let you do a ton of stuff, so it's not obvious what this function does or is supposed to be used for.
It is not actually useful. I realized later this was not needed.

The Monoid instances suffer the same problem, as they are probably not useful. Even worse, one of them doesn't even work !

instance Monoid (PubFP e a) where
    mempty = PubFP (const (return ()))
    PubFP a `mappend` PubFP b = PubFP (a >> b)

It actually used the monad instance of (->) and not STM, which, if I desugar it properly, does something like that :

    a >> b
    a >>= \_ -> b
    \z -> (\_ -> b) (a z) z
    \z -> b z
    b

So it basically only used the last PubFP. The correct implementation for mappend should have been :

instance Monoid (PubFP e a) where
    PubFP a `mappend` PubFP b = PubFP (\e -> a e >> b e)

Abstracting a backend

Now that I have decided to have a "hub" connecting several backends, I need to find a way to abstract them. I will need to keep a list of them, and it must be possible to add and remove them dynamically, so I need some way to identify each backend. I also need a way to tell the backend that it is to be removed, so that it can clean things up and say goodbye. Finally, I need a way to talk to a backend, and a way for the backend to interact with the hub.

Here is the set of threads I should need without undue multiplexing :

One thread per active game. I don't think it's possible to combine them in a single thread due to the API I exposed, and I don't think it's worth worrying about this anyway.
One thread per backend, that will listen to messages from the outside world.

That is probably all I need, but because I started writing code before thinking about my architecture, I introduced a bad abstraction. I decided I would talk to the backends using a TChan, which means :

One additional thread per backend, listening to messages from the games.

So backends are defined here. A better abstraction for backendChan :: TChan Interaction could be backendTell :: Interaction -> STM (). You might notice that the comments are talking about a set of function, which was my first shot, and which was a better idea indeed.

Abstracting communication

Communications from the game and to the player are currently of three types :

Asking the player what card he would like to play during normal play. This returns a PlayerAction and Exchange couple.
Asking the player to choose between several cards. This returns a Card.
Telling various things to the player. There is no return value expected from the player.

For all cases, we need to contact each backends and tell them to contact a player or broadcast some message.

For the first two cases we need also to set some callback machinery and wait for them to answer. The function performing this should return quickly some kind of promise object that will be filled once a player has answered.

We would like to write a function with a type looking somehow like :

communicate :: TVar GlobalState -> a -> STM (Promise b)

Where a is one of the three message types, and b the corresponding answer. Obviously, we can't write such a function with this type. But we can use the TypeFamilies extension to write it :

data IAskingAction   = IAskingAction PlayerId Age (NonEmpty Card) GameState Turn
data IAskingCard     = IAskingCard   PlayerId Age (NonEmpty Card) GameState Message
data ISimpleMessage  = ISimpleMessage CommunicationType GameId

type family InteractionResult a :: *
type instance InteractionResult IAskingAction     = (PlayerAction, Exchange)
type instance InteractionResult IAskingCard       = Card
type instance InteractionResult ISimpleMessage    = ()

communicate :: TVar GlobalState -> a -> STM (Promise (InteractionResult a))

Now the type of a and of the Promise are statically linked, which will be useful for writing generic functions.

Conclusion

This episode was about hasty decisions and code quickly written. I was not exactly in my comfort zone with the multiple backends concept, and should probably have aimed lower to get a feel of the problem first.

I will rework all of this before the next episode, which will be about concurrency, the STM, and how to mix IO and STM actions.

]]> 7 Startups - part 3 - An interpreter for the rules https://hbtvl.banquise.net/posts/2014-05-16-7-startups-part-3-an-interpreter-for-the-rules.html 2014-05-16T00:00:00Z

In the previous episode, I implemented the game rules, but did not test them. I also had some reservations about some code I wrote, but predicted it would be mostly right, even without tests. Today's episode is about pretty printing and operational !

Minor modifications since last time

I refactored the getCardFunding and getCardVictory functions so that they are now pure. I toyed with the idea of having a monad morphism (I learned today it was called like that to integrate Reader GameState actions in the MonadState GameState functions, but this was not warranted as the functions are so simple.
I refactored neighborhood relationship so that it encodes more invariants. A player now must have a left and right neighbor. They might be invalid though.
I refactored the type of the interfaces between the game rules and the players, so that you can't pass empty lists where they are forbidden. I was later told this type already existed in semigroups.

Why pretty printing ?

I hinted heavily last time that there would be a dedicated pretty printer. An example of such an implementation is in the ansi-wl-pprint package. It introduces functions and combinators that let you easily create a Doc value that will look neat on your screen.

Unfortunately, in order to properly support all text-based backends (IRC, XMPP, email, console) it doesn't seem to be possible to reuse an existing printer. For example, the color set between all these backends is quite distinct, and some are even capable of printing pictures. I tried to engineer one that would be at the same time flexible, easy to use and good-looking an all backends. Time will tell if this was a success.

I will not give a dissertation on the subject, and have copied the interface from other pretty printing libraries. I will just give some implementation details here.

Basic pretty printing types

Speaking of stealing from other pretty printers, I really should have looked at their code too ! Here are my basic types:

newtype PrettyDoc = PrettyDoc { getDoc :: Seq PrettyElement }
    deriving (Eq, Monoid)

data PrettyElement = RawText T.Text
                   | NewLine
                   | Space
                   | ...

So you basically have all "elements" in PrettyElement, and they can be appended in a monoidal fashion in a PrettyDoc, which is just a newtype for Seq PrettyElement. This is a very inelegant decision, and I will be sure to refactor it for the next episode ! Looking at another implementation, it is clear that a single type was required, and that the Monoidal structure could be achieved by adding Empty and Cat constructors. There is a reason I wrote my type like this though, and it is related to how I intended to solve the problem of backends with poor or no support for multiline messages, but this will featured in another episode !

Specific design choices

I decided to directly encode the game entities as part of the pretty printing types. That should be obvious from the list of elements. A VictoryPoint, a Neighbor or even a CardType are directly representable, so that the backends can optimize their rendering.

Other than that, the code is pretty boring.

A pretty-pretty printer ?

My first backend will be the console, as it will not have any networking or concurrency problems to solve. I used the aforementioned ansi-wl-pprint package, and wrote a pretty instance for PrettyElement and PrettyDoc. This leads to strange code such as print (PP.pretty (pe something)).

Implementing the GameMonad

During the last episode, I wrote all the rules in an abstract monad that is an instance of GameMonad, meaning it featured a few functions for interacting with the players. I took a typeclass approach so that I could start writing the rules without worrying about the actual implementation of this abstract monad.

Now that the rules are written, it is time to give them a try. In order to do so, I ditched the typeclass, and expressed it in terms of ProgramT, from the operational package. It only takes a few steps to refactor :

The instructions GADT

You must start by writing all the operations that must be supported as a GADT.

We previously had :

type NonInteractive m = (MonadState GameState m,
                         Monad m,
                         MonadError Message m,
                         Functor m,
                         Applicative m)

class NonInteractive m => GameMonad m where
    playerDecision    :: Age -> Turn -> PlayerId -> [Card] -> GameState -> m (PlayerAction, Exchange)
    askCard           :: Age -> PlayerId -> [Card] -> GameState -> Message -> m Card
    tellPlayer        :: PlayerId -> Message -> m ()
    generalMessage    :: Message -> m ()

And now have :

data GameInstr a where
    PlayerDecision :: Age -> Turn -> PlayerId -> NonEmpty Card -> GameInstr (PlayerAction, Exchange)
    AskCard        :: Age -> PlayerId -> NonEmpty Card -> Message -> GameInstr Card
    TellPlayer     :: PlayerId -> Message -> GameInstr ()
    GeneralMessage :: Message -> GameInstr ()
    ActionsRecap   :: M.Map PlayerId (PlayerAction, Exchange) -> GameInstr ()
    ThrowError     :: Message -> GameInstr a
    CatchError     :: GameMonad a -> (Message -> GameMonad a) -> GameInstr a

So ... there have been some choices going on here. First of all, we need to support all the features we previously had, namely MonadState, MonadError and four game-specific functions. You can spot these four functions quite easily (along with a new one, which will be covered in a minute). We get MonadState and MonadError in the following way :

type GameMonad = ProgramT GameInstr (State GameState)

instance MonadError PrettyDoc (ProgramT GameInstr (State GameState)) where
    throwError = singleton . ThrowError
    catchError a handler = singleton (CatchError a handler)

I decided to use the monad transformer ProgramT over a base State GameState monad, but encode the error part with the provided instructions. It would have been easier to encode the state part that way, except I don't know how to write an instance for ProgramT (see this post comment).

The interaction functions no longer have a GameState in their types, because the interpreter will have access to the state when decoding this instruction, so it is not necessary to pass it here too.

Mechanically refactor all mentions of GameMonad

Now all you have to do is to replace all type signatures that looked like :

GameMonad m => a -> b -> m c

Into :

a -> b -> GameMonad c

Write an interpreter

I decided to write a generic interpreter, that takes a group of functions in some monad m, a base GameState, and gives you a function that computes any GameMonad a expression in the m monad. The implementation is pretty obvious, and not very interesting, but it should be easy to write backends now.

Perhaps of interest is the fact that the game state is explicitly passed as a parameter all over the place, so it can be passed to the backends at the interpreter level.

A pure backend

The easiest backend to write is a pure one, with no real player interaction. I could have used Identity as the base monad, but instead opted for State StdGen. That way, I can easily have the "players" play random cards, which will help with testing.

The implementation is also nothing special, but made me write a lot of code to support it. In particular, the allowableActions function is pretty tricky, and is not entirely satisfying. Given a game state, a player name and a list of cards in his hands, it gives a list of all the non obviously stupid legal actions that are available. It does so in the most direct way, enumerating all possible combinations of resources, neighbor resources, exchanges, etc. that would work. Then it removes all duplicates, and the actions that are obviously too expensive.

Fortunately, all this code will also be used by the other backends.

So ... are there bugs yet ?

I wrote a simple test that checks for errors. Theoretically, the pure backend should always result in games that end well (we should get a Right ... instead of a Left rr. So I wrote a simple property-based test that gets an arbitrary seed and number of players (between 3 and 7), runs a pure game and checks its result.

And there were runtime errors !

The Monoid instance for AddMap had an infinite loop.
The allowableActions function sometimes returned no valid actions. I forgot to always add the possibility to drop a card ...

To prevent the second case from happening again, I wrote the "prepend drop actions" before the big case statement, and modified the type of the askCardSafe function so that it can't accept an empty list. This means that if I introduce another bug in allowableActions, I should get a Left ... instead of a runtime exception.

There also was a "rule" bug, due to the fact that I had not understood a rule correctly. Basically, I use a fictional 7Th round to emulate the efficiency capability, but there should be no "hand rotation" before that turn. I fixed it wrong once, and then properly. However, I did not discover nor fix this bug because of tests.

The console backend

Before writing the console backend I needed a bit of code for pretty printing stuff. Once this was done, the backend was quickly written.

The opponents still play randomly, which explains the kind of results depicted below, but it is a genuine pleasure to finally play !

Crushing victory

I also realized when using the console backends that the messaging functions, while generic, would probably not work well on all backends. I decided to include more specialized functions, such as ActionsRecap, which can be passed a map of all the actions the players undertook in a turn. The current version also lacks a way of getting the results of the poacher wars between the ages, but that should be trivial to add.

Next time

Next time should get more interesting, as I will try to write an interesting backend. It will be a bit harder to design because I want players using distinct backends to be able to participate in the same game.

]]> 7 Startups - part 2 - Game rules definition https://hbtvl.banquise.net/posts/2014-05-12-7-startups-part-2-game-rules-definition.html 2014-05-12T00:00:00Z

Whew, I just added a big pile of code to what was done previously. I wrote all the missing game types and rules. It took about 4 or 5 hours.

In this post, I will describe how I decided to define the main game types, and some various details of interest.

Choosing the rules monad

I will describe the rules using a monad, mainly because I am used to work with them, and because they are mighty convenient in Haskell, with the do notation and the numerous libraries. As is often the case with games, there will be a state, containing the game state at a given time. But while I will just write the rules, I need to graft user interaction at some point. The goal of this project is to write a 7 Wonders clone that might work with multiple backends. To achieve this, I will try not to constraint my implementation any more than necessary.

Player identification

The first important type is to find a way to identify each players. I wrote this :

type PlayerId = T.Text

I currently am not sure this is sufficient / precise enough, but the backends I have in mind (IRC, XMPP, console and email) all have string based identifiers, so it should work for at least those three. Anyway, the backends will probably have to keep a relationship between a player nickname and his actual identity in the system, so this will probably turn out OK.

Game state

data GameState = GameState { _playermap   :: M.Map PlayerId PlayerState
                           , _discardpile :: [Card]
                           , _rnd         :: StdGen
}

data PlayerState = PlayerState { _pCompany         :: CompanyProfile
                               , _pCompanyStage    :: CompanyStage
                               , _pCards           :: [Card]
                               , _pFunds           :: Funding
                               , _pNeighborhood    :: M.Map Neighbor PlayerId
                               , _pPoachingResults :: [PoachingOutcome]
}

makeLenses ''GameState
makeLenses ''PlayerState

This might look pretty obvious, and it might be (as it is my first version), but this model has several shortcomings, the worst of them being the way that neighboring information is encoded. This is originally a tabletop game, and each player has two neighbors : on his left and on his right. Unfortunately, the Map Neighbor PlayerId only means that a player can have up to two neighbors (there are only two constructors in the Neighbor type), and it doesn't even garantee they have a corresponding state in the GameState.

A type that would properly model this property would be to store [(PlayerId, PlayerState)] in GameState, interpreted as a circular list (the first player in the list being the right neighbor of the last one). But this would be a major PITA to manipulate.

Another idea would be to store the neighboring information in a read-only structure. That way, we can make sure that no invariants are being violated, as the structure can't be modified, but this also might be too much of a hassle. I will probably refactor some of this for the next episode with something less challenging : a simple pair.

And now, the monad !

As we have seen, we will need a MonadState GameState to model most of the rules. Some parts of the game might also throw errors, so it might be a good idea to have our monad be an instance of MonadError. Finally, we need some user interaction. In order to be able to write any backend, I decided to keep it abstract for now :

type GameStateOnly m = ( MonadState GameState m
                       , Monad m
                       , Functor m
                       , Applicative m)
type NonInteractive m = (MonadState GameState m
                        , Monad m
                        , MonadError Message m
                        , Functor m
                        , Applicative m)

class NonInteractive m => GameMonad m where
    playerDecision    :: Age -> Turn -> PlayerId -> [Card] -> GameState
                             -> m (PlayerAction, Exchange)
    askCard           :: Age -> PlayerId -> [Card] -> GameState -> Message -> m Card
    -- | Tell some information to a specific player
    tellPlayer        :: PlayerId -> Message -> m ()
    generalMessage    :: Message -> m () -- ^ Broadcast some information

First of all are two constraints synonyms :

GameStateOnly : basically MonadState State with all the implied constaints, which will be used in all the functions that can't fail and that don't require user interaction.
NonInteractive : just like the previous constraint, but for functions that can throw errors.

Finally, a GameMonad typeclass. The monad our game will work in must implement these four functions, which are all I found was needed for player communication :

playerDecision: this is the main interaction. Given all kinds of data, it asks the player to decide what he will do in the current turn.
askCard: there are two effects where a player must chose a card over a list (copy community, and play for free a card from the discard pile). This is what this function is about, at least for now.
tellPlayer: tells a specific message to a given player.
generalMessage: tells a message to all players. This might not be necessary, as we could just iterate over the list of players and use tellPlayer. On the other hand, for IRC or XMPP backends, it might make sense to display this information on a conversation channel, so that watchers can follow the game.

The reason why it might make sense to have such granularity (pure, GameStateOnly, NonInteractive, GameMonad) is twofold :

It is easier to reason about the functions.
The less "powerful" a function is, the easier it is to test.

What is important to note is that I can't write arbitrary effects with just the GameMonad constraint. Even better, I know I should be careful only when using the first two functions, as they are the only ones where user input can creep in. This explains why the part of the code that deals with playerDecision is so full of checks.

The choice of a typeclass is debatable, as there probably will only be a single implementation. I chose to do so because it will let me write code without worrying about how the monad itself will be implemented. I will probably ditch the typeclass later.

One problem so far is that these functions don't have the proper type. Indeed, what happens when I pass askCard an empty list ? How is the player supposed to provide a card ? The other problem now is what to do with this Message type. Right now, it's a type synonym to String, but it will change for the next episode !

Various notes

No error recovery

I decided not to have error recovery in the game rules description. This is the responsability of the "driver" (which will be described in a later post) to make sure sore losers can't DoS the game. The game will just end on the first error it encounters.

Lenses everywhere

This code uses the lens library all over the place. This is not surprising, as it involves a lot of mangling of nested structures in the State monad. But the prisms are even better ! Here is an example :

-- | Compute the money that a card gives to a player
getCardFunding :: GameStateOnly m => PlayerId -> Card -> m Funding
getCardFunding pid card = do
    stt <- use playermap
    -- note how we exploit the fact that ^. behaves like foldMap here
    let funding = card ^. cEffect . traverse . _GainFunding . to computeFunding
        computeFunding (n, cond) = countConditionTrigger pid cond stt * n
    return funding

The choice of writing this option in GameStateOnly is debatable, as it just needs a read only access to the state once, and might just have been like that :

getCardFunding :: GameState -> PlayerId -> Card -> Funding

However, what is interesting is how it is working. Here is an anotated of how the funding function is composed :

cEffect                                               :: Traversal' Card [Effect]
cEffect . traverse                                    :: Traversal' Card Effect
cEffect . traverse . _GainFunding                     :: Traversal' Card (Funding, Condition)
cEffect . traverse . _GainFunding . to computeFunding :: Fold Card Funding

So basically we wrote a traversal that goes through all effects of a card, keeping those with the GainFunding constructor, extracting its arguments, and finally using them to compute a Funding.

Now, if I had written funding = card ^.. ..., I would have obtained a [Funding], that I could add with sum. But remember that we made sure that our numerical newtypes, such as Funding and Victory, had a monoid instance for addition. In that case, ^. (or view) will make a monoidal summary, meaning it will give me 0 if there were no matches, or the sum of these matches, which is exactly what I wanted.

Order of execution

In this game, order of execution is really important, as most actions are supposed to happen simultaneously, and some only at very specific steps. In particular, a players can "trade" a resource belonging to a neighbor in exchange for money. A naïve implementation would be something like :

playermap . ix playerid . cFunds -= x
playermap . ix neighbor . cFunds += x

But this would create a (risky) exploit : namely declaring that you want to trade more resource than what you have money for, hoping somebody else will trade with you and that this transaction will be processed before yours.

In order to fix this, the resolveExchange function only removes money from the current player, returning the set of bought resources and an AddMap PlayerId Funding, listing the money that needs to be given to the neighbors.

The AddMap newtype

The resolveAction function also returns this AddMap PlayerId Funding, and the payouts are only processed after all actions are resolved. In order to make the code nicer, we need this AddMap k v newtype to be Traversable and have a Monoid instance that does unionWith (+).

The code is here and is an example on how this is done. I also derived the Ix and At instances, even though I didn't end up using them. Strangely, someone asked on the cafe mailing list how to do this.

The 7th turn

There are only 6 turns for each age. But there is a company stage that let players use the 7th card, at the end of an age. Instead of having a special case, this is done by having an optional 7th turn.

No tests

Despite my claim that my rules are easy to test, tests are horrible to write, as they need a big setup. For this reason I postponed writing them ;) This will be a good test of the "Haskell code almost works the first time it runs" theory.

Next time

I will refactor a bit, and introduce a custom pretty-printer that will work with multiple backends, so that it is possible to have a nice view of what is going on during play.

]]> 7 Startups - part 1 - Introduction and types https://hbtvl.banquise.net/posts/2014-05-10-7-startups-part-1-introduction-and-types.html 2014-05-10T00:00:00Z

There have been recently complaints that there wasn't any resource available for bridging the gap between beginner and experimented Haskeller, and some posts on "Haskell program architecture" have been written to help with this transition. I have found these posts to be pretty interesting, and while I can hardly be called an expert, I would like to contribute to this effort by documenting a few advanced Haskell features, as well as my design decisions, applied to a simple, yet fun, project.

Now that this is out of the way, let's start !

The project

In this series of posts, I will describe how to model the rules of a well known board game, and how to turn them in an enjoyable program. If time permits, quite a few topics should be discussed, including key design decisions, how to interface a pure description of the rules with multiple backends, concurrency with the STM, and the advantage of always pretty printing your data structures.

The game itself is a shameless clone of the excellent 7 Wonders game (you can find the rules on the official web site), but with Internet giants instead of antique wonders. The theming took me a long time, and I am not particularly satisfied with it, so if you feel like contributing, please give me better names for the cards and resources.

All the code is on github. I will document my decisions and actions as I go, and will tag the repository accordingly. The relevant version for this article is tag Step1.1.

The types

Startups.Base

The Startups.Base module contains all the base elements of the game, with the relationship with the original game written the comments. While all the types are more or less directly transcribed from the rules book, the newtyped numerical types might not be obvious :

newtype Poacher = Poacher { getPoacher :: Integer }
    deriving (Ord, Eq, Num, Integral, Real, Enum, Show)

newtype VictoryPoint = VictoryPoint { getVictory :: Integer }
    deriving (Ord, Eq, Num, Integral, Real, Enum, Show)

newtype Funding = Funding { getFunding :: Integer }
    deriving (Ord, Eq, Num, Integral, Real, Enum, Show)

newtype PlayerCount = PlayerCount { getPlayerCount :: Integer }
    deriving (Ord, Eq, Num, Integral, Real, Enum, Show)

All the derived instances let you use them just like a standard Integer in your code, and the newtype prevents you from mixing them. But the main advantage is that it will make functions type signatures a lot more informative.

Startups.Cards

I usually would have merged this module with the previous one, but for the sake of blogging about it I separated the two. This module is all about modeling the cards. Fortunately, the cards have an obvious representation. But what about the Effect type ?

Modeling the effects

With a functional language, there are several ways to go :

Have some big case statements all over the code that depend on the card names, the effects being encoded where they are needed. This is obviously bad, as it will lead to a lot of verbose code, and it will be a pain to refactor the code.
Have the effect described as a state-changing function (ie. type Effect = PlayerId -> GameState -> GameState). This is the most versatile option, as it lets you add new cards with funky effects without modifying other parts of the code. Unfortunately, your program no longer have an easy way to "observe" the effect, so you will need to write a human-readable description for each card. It might be hard to write an AI for this game too (this point is debatable). There is also the problem of reasoning about new effects, especially concerning the order of application of the effects. A common workaround is to add a "priority" field, so that the order of application is known.
Fully describe all effects with a data type. This is the approach we are going to take, as it has obvious advantages in this particular case : most cards can be described with a handful of distinct "effect components", where the components are orthogonal. This means they should be implemented in the part of the code that are relevant. It will be quite easy to describe arbitrary effects to the user too.

All the possible effects components can be seen here. Some components have no parameters (such as Recycling), meaning they model a specific rule. But what is nice about this data type is that it models the effects of the cards, but also of the company building stages.

Precise types

The following types are not as obvious as they appear :

data Neighbor = NLeft
              | NRight
              deriving (Ord, Eq, Show)

data EffectDirection = Neighboring Neighbor
                     | Own
                     deriving (Ord, Eq, Show)

My first version was something like :

data EffectDirection = NLeft
                     | Own
                     | NRight
                     deriving (Ord, Eq, Show)

This was simpler, but some effects have no meaning when applied to the current player (such as reduced exchange rates). This will make pattern matching a bit more cumbersome, but it will probably prevent some mistakes.

Modeling the cost

What is more interesting is the Cost data type.

data Cost = Cost (MS.MultiSet Resource) Funding
          deriving (Ord, Eq, Show)

A MultiSet is a collection of objects that can be repeated but for which order is not important (you can also think of it as a sorted list). It perfectly models a resource cost, such as "3 operations, and a marketing", and it provides us with a isSubsetOf operation that can directly tell whether a player has enough resources to play some card. There is an obvious Monoid instance for it :

instance Monoid Cost where
    mempty = Cost mempty 0
    Cost r1 f1 `mappend` Cost r2 f2 = Cost (r1 <> r2) (f1 + f2)

I don't think this instance will be too useful, except for writing this cleanly :

instance IsString Cost where
    fromString = F.foldMap toCost
        where
            toCost 'Y' = Cost (MS.singleton Youthfulness) 0
            toCost 'V' = Cost (MS.singleton Vision) 0
            toCost 'A' = Cost (MS.singleton Adoption) 0
            toCost 'D' = Cost (MS.singleton Development) 0
            toCost 'O' = Cost (MS.singleton Operations) 0
            toCost 'M' = Cost (MS.singleton Marketing) 0
            toCost 'F' = Cost (MS.singleton Finance) 0
            toCost '$' = Cost mempty 1
            toCost _ = error "Invalid cost string"

When the OverloadedStrings extension is enabled, the compiler will accept strings in places where another type is expected, by adding a call to the fromString function. For example, "YYY" :: Cost will be replaced by fromString "YYY" :: Cost.

I don't think this is good practice to advise others to write partial IsString instances, but it greatly helped with writing the card list, speaking of which ...

Tests

Writing the first card list was the most tedious and error prone part of this endeavor. In order to make sure I did not introduce a typo, I performed a couple of tests on the card list :

All cards are distinct (got a bug).
For every number of players and ages, there are 7 cards for each player (there were three errors).

What could have been better

Ord instances

Most data types now have Ord instances that are not particularly meaningful. They are here so that the data structures can be used in the standard containers types, such as Data.Map.Strict and Data.Set. It might have been a better idea to use unordered-containers, but this would have meant more boilerplate (for deriving all the Hashable instances).

Why not use an external DSL for describing the cards ?

This indeed would have been a good idea, and wouldn't have been particularly hard to write. I don't think it would have added much to the project at this stage though.

Modeling the "Opportunity" effect

This effect currently looks like this : Opportunity (S.Set Age). It is used to describe the fact that a given player can build for free any card, once per age. The Set will contain the set of Ages for which this capacity has not been used. This means that when the player decides to use this capacity, this effect will need to be updated. If this wasn't for this effect, a player card list would only be modified by adding a card to it, which would have been more pleasant.

Card and Company stages

When I started writing this post, the Card type had a single constructor, and there was a CardType that was not part of the rules used to describe a company stage. I did that because I thought it was more elegant to unify cards and company stages, as they were pretty similar (both have costs and effects that work the same way).

It turned out that I had to enter dummy values for player count, age, card name, etc. for these "cards". Now there is an additional constructor for company building stages, as can be seen in this commit.

Next time

In the next episode, I will start writing the game rules, starting with choosing (or not) the proper abstraction for describing them. In the meantime, do not hesitate commenting (reddit link) !

]]>

Original Source | Taken Source

1.0	0.7
		0.7
			0.7
				0.7
					0.7
	0.2					0.7
		0.2					0.7
			0.2					0.7
	0.1			0.2					0.7
		0.1			0.2
			0.1			0.2				1.0
				0.1			0.2				1.0
					0.1			0.2				1.0
						0.1			0.2				1.0
							0.1							1.0
								0.1							1.0
									0.1							1.0

1.0	0.7
		0.7
			0.7
				0.7
					0.7
	0.2					0.7
		0.2					0.7
			0.2					0.7
	0.1			0.2					0.7
		0.1			0.2
			0.1			0.2				1.0
				0.1			0.2				1.0
					0.1			0.2				1.0
						0.1			0.2				1.0
							0.1							1.0
								0.1							1.0
									0.1							1.0

updates

Initial conditions

Mega solution!

Edit

Mega solution!

Concluding remarks

Notes on solving Book 05

Notes on healing

Next time

Bonus

Starting conditions

Number of states

Success probability

Strong character

Average character

Effect of skill and endurance on outcome

Discussion

Chapter 200, and the choice of assassin

Conclusion

What is done

What is to be done

Results

Weak character

Average character

Strong character

Effect of skill/max endurance

Introduction

Intermediate representation

Interpreting the IL

Step 1 - concrete values

Step 2 - interactive interpreter

Step 3 - symbolic values

Step 4 - abstracting program state

Step 5 - actual symbolic execution

Conclusion

A simple observation

Lazy evaluation to the rescue

But does it make a difference?

What's next?

A quick reminder of the problem and base types

What is a solution ?

Finding the solution

So, problem solved?

Optimal strategy, and model

Solving

What about the other special chapter

Next stop

The hidden optimization story

The player character

A note on the inventory

The Probably type

Playing the game

But it is so slow!

Let's memoize!

What's left?

Extensions and imports

Basic types

What's in a chapter?

The Decision type

The ChapterOutcome type

Combat description

Lenses, plates and utilities

The actual chapters data

Playing with compiler options

Alternative encodings of the monad

Parallel execution

Performance summary

Conclusion

How to fix the memory leak?

Using the simplest alternative

State of the code and performance

Writing a custom parser

Not parsing everything

Inlining all the things

Measuring performance

A faster parser combinator libraries : attoparsec

Reducing the memory footprint : thyme

From Text to ByteString

A note about the original C module

Optimization summary

The `Probably` type

1.0	0.7
		0.7
			0.7
				0.7
					0.7
	0.2					0.7
		0.2					0.7
			0.2					0.7
	0.1			0.2					0.7
		0.1			0.2
			0.1			0.2				1.0
				0.1			0.2				1.0
					0.1			0.2				1.0
						0.1			0.2				1.0
							0.1							1.0
								0.1							1.0
									0.1							1.0