|
|
| CARVIEW |
Vim emulation in Yi is far from perfect. It would be interesting to have a test suite to quantify what portion of Vim one or the other editor implements and have a leaderboard and a healthy competition. The most useful metric for the quality of Vim emulation that I have is saying to a person new to Yi to try and use it like Vim and counting the seconds until something doesn’t work as expected.
Nevertheless, Yi has some little bits and pieces in its emulation that are improvements over the real Vim. Just today I finally realized that I often go from insert mode to normal mode just to go back a character or two. But there already is a very popular shortcut for going back a character: <C-b>! It works in a shell, in many commandline programs, in text fields in macOS, in emacs of course.
What does it do in Vim? Inserts ^B, which is probably less useful thing. So as of today <C-f> and <C-b> move the cursor instead of doing nothing in insert mode. Going through other standard readline bindings and porting some to insert mode in Yi would be a good beginner project.
Of course, there are other ways in Vim (and Yi) to move the cursor one character left, like <C-o>h and the left arrow (no judging). I just find <C-b> to be the easiest.
Another tiny bit where Yi improves on Vim is blockwise visual insertion. While in Vim you have to type the whole thing, press Escape and then observe the result, Yi updates all rows live. This one is best described with a screen recording:
For some more (probably more obscure) points where Yi is intentionally incompatible with Vim, see yi-keymap-vim readme
]]>Following previous post it is a good moment to talk about ongoing modularization of Yi.
Rewind to a moment I started contributing to Yi: 2012. After two years of using vim I found a bug in it. It wasn’t anything too serious like a crash or corruption of user text, just some undocumented inconsistency in behavior. I thought, well, it’s open source why not try to fix it? This was the first time I looked at vim’s source code and was completely overwhelmed. Hundreds of thousands of lines of C. It’s not unprecedented, of course, but it’s A LOT. At that point I’ve only seen a codebase comparable in size one time at work, but that was, while larger, much more modular.
After some hours of trying to find a relevant place in vim, I was sufficiently lost to arrive at a question “Is there a vim-like editor that is written simpler?”. I remember looking at Kate, Yzis and Yi. Was I going through a list of vi emulations in reverse lexicographical order? Probably. Was I using KDE at the time? Definitely. Anyway, Yi seemed interesting because it had about 20 thousands lines of code and had multiple frontends (Terminal, Gtk and Cocoa) and multiple keymaps (vim and emacs, naturally).
This is how Yi was split into packages, or rather into a library part and an executable part within one cabal project at the time:
> cloc yi/src/library
158 text files.
158 unique files.
29 files ignored.
github.com/AlDanial/cloc v 1.72 T=0.49 s (262.7 files/s, 57554.5 lines/s)
-------------------------------------------------------------------------------
Language files blank comment code
-------------------------------------------------------------------------------
Haskell 129 4514 4647 19103
-------------------------------------------------------------------------------
> cloc yi/src/executable
1 text file.
1 unique file.
0 files ignored.
github.com/AlDanial/cloc v 1.72 T=0.01 s (95.5 files/s, 1241.3 lines/s)
-------------------------------------------------------------------------------
Language files blank comment code
-------------------------------------------------------------------------------
Haskell 1 5 3 5
-------------------------------------------------------------------------------So basically one package. Interestingly, Yi had a custom prelude and that gathered lots of complaints over the following years until Mateusz finally removed it in 2014. And yet these days custom preludes seem to be all the rage? By contrast, top result in google for “custom prelude” from 2012 is this answer by Don Stewart saying not to do it.
Of course, I was not the first person to come up with the idea to split emacs emulation from vim one and terminal interface from GUI. In fact, here is the quote from 2012 README:
We also want to simplify the core Yi package to make it more accessible, splitting some parts into several packages.Back then it was significantly harder to do just because of the tooling. Not only there was no stack at the time, cabal sandboxes were not a thing until late 2013. I fondly remember the character-building days of nuking your global ghc and cabal directories.
But thanks to cabal and later also stack folks things were steadily improving since.
The tooling situation was not the only difficulty, Yi had several circular references between modules. One by one, we untangled these and split some libraries potentially useful outside of yi.
yi-rope is actually used by our friendly competitor rasa and haskell-lsp.
oo-prototypes still blows my mind five years later. Yi was my intro to Haskell after a HelloWorld and I was basically greeted by a module saying “here we implement OOP inheritance in 7 lines out of thin air”.
yi-language I’m not too happy about, because we split it not because it’s a self-contained thing, but just to isolate alex-related stuff that was killing incremental compilation. I’m hoping to reshuffle this part, so that pieces of yi-language like Yi.Buffer.Basic and Yi.Region end up in yi-core and everything alex-related lives in yi-alex-utils-or-something and becomes entirely optional, that is you will be able assemble an editor without a dependency on alex.
Some time later stack came about and made it easy to work with multiproject repos and we finally split all the frontends and keymaps into separate projects.
So this is how project structure looks now:
Project Lines of haskell code
yi-core 10335
yi-dynamic-configuration 81
yi-frontend-pango 1566
yi-frontend-vty 407
yi-fuzzy-open 214
yi-intero 140
yi-ireader 124
yi-keymap-cua 147
yi-keymap-emacs 643
yi-keymap-vim 4669
yi-language 803
yi-misc-modes 449
yi-mode-haskell 1246
yi-mode-javascript 601
yi-snippet 375So in my mind the next thing in modularization of Yi is moving alex-powered highlighting into a plugin while making yi-core expose some general interface. It is already possible to make syntax highlighting without alex, e.g. I have rainbow parens mode in my config where actual parsing is done by regex-applicative, but it doesn’t feel like a first class citizen.
Finally, if this story was interesting to you, you’re very welcome to join the development!
Do you care about how pretty does editor look? Make a new shiny frontend!
Have an idea about a new crazy ergonomic control scheme? Try it out as a new keymap for Yi.
Maybe you’re interested in optimizing haskell code? Yi has plenty of that.
In any case, don’t hesitate to file an issue, make a PR or chat.
]]>The 0.14 section in the CHANGELOG has only one entry:
* yi-core now doesn't depend on dyre.
Dynamic configuration support is now provided by yi-dynamic-configuration package.(This is also a good moment to plug a post about why you should have a CHANGELOG file)
See the previous post on what dyre is and how it’s used in Yi.
The benefits of this change are:
- Users of static configurations have a leaner editor.
- Improved separation of concerns helps current developers to stay sane while helping potential developers to comprehend the codebase piece by piece
I intend to write another post about ongoing modularization of Yi and how it went from one package to a dozen in the last five years and what’s next in this direction.
There are still areas where user experience is much worse than it should be (in both static and dynamic configurations):
- tweaking per-file indentation settings, like switching from spaces to tabs or different amount of spaces
- tweaking GUI settings like colorscheme or (for pango frontend) font size
- .. or in general when you’re just changing some config values like numbers, strings and colors
If you feel any of it can be an interesting task, you’re very welcome to join! Here are some relevant tickets (but don’t let these limit you):
- #1006 Ex command for changing a colorscheme
- #1005 Ex commands for changing indentation settings
- #988 Support vim modelines for per-file indentation settings
- #987 Support EditorConfig files for per-project indentation settings
I will explain how the compilation of yi works and how the dynamic configurations tie into this.
Dynamic configurations
Yi is basically a library which you can use to build your own editor. The configuration file is a perfectly valid haskell file which must be compiled like any other haskell file. In this file you can specify what you want your Yi to look like. You can even completely change the editor to the point that it would not be recognizable as Yi.
For this to work you need to recompile your configuration file (and also dependencies like Yi itself) for every change you make. The dynamic configurations use dyre to try to automate this process.
When a dynamic configuration starts it checks if ~/.config/yi/yi.hs has been changed and recompiles it if it has been changed. It is also possible to manually trigger recompilation by running the reload command (M-x-reload in the emacs and :reload in the vim keymap), this will recompile ~/.config/yi/yi.hs and then transfer the current state to the newly compiled Yi.
To achieve dynamic recompilation you have to kickstart the process by compiling a dynamic configuration manually first. The default yi package is a static configuration so it won’t check for changes out of the box.
Static configurations
The static configurations are more obvious in that there is no automatic recompilation so you always know exactly what version of Yi you are using. Static configurations also allow for a complete cabal project to be your configuration. The advantages of a complete cabal project is that you can have dependencies other than just Yi.
Why switch?
The dyre package that is used to provide this dynamic recompilation is derived from the way that xmonad handles recompilation. Xmonad is a window manager so it’s not restarted very often and it is inconvenient to restart so I think dynamic recompilation is very useful in that case. But Yi is a text editor and I personally restart my text editor very often so I think that Yi should not use a dynamic recompilations system unless it is very intuitive and stable.
The future
I think dynamic configurations are awesome and should be used in the future, but right now there are a few issues:
- You can’t use a complete cabal project as configuration.
- Errors in the compilation process are not handeled smoothly and can even leave Yi in a broken state.
- There’s no user friendly way to configure the dynamic recompilation (e.g. where the configuration file is located).
It would also be nice to have a subset of the configuration in a DSL to allow for seamless configuration. It should be possible to change for example the font size without recompiling the entire editor. We could use dhall for this.
]]>In this post I will sketch an encoding for OO-style inheritance in Haskell, and show how this is used to in Yi to write code that can be customized.
This can also serve as an introduction to the concepts defined in module Data.Prototype (currently found in Yi sources)
Inheritance
Inheritance can create structures which are difficult to understand. Since a given method call can call dispatch to a number of methods at run-time, tracking what is going on might be tricky. Sometimes however, inheritance is exactly the construct we need.
Imagine you have the following piece of code:
a :: A
a = fa b c
b :: B
b = fb a c
c :: C
c = fc a bThat is, a, b and c are values defined in terms of each other.
You would like users to be able to customize a’s value. However, if the change actually occurs in the definition of c, you don’t want them to copy-paste the whole set of definitions. It would be preferable to amend only the definition for c and reuse the rest. Unfortunately, a’s value is closed, so this is not possible.
This situation seems to cry for inheritance. In an object oriented language, the solution is obvious: make a, b and c methods of a class. The user can then inherit it and override the definition of c.
In Yi, color themes have a similar structure: specific styles are defined in terms of base styles. If a user changes a base style, the change should be reflected automatically in all the styles that derive from it. As in the toy example above, we do not want the user to redefine everything from the ground up.
So, what can we do, since Haskell lacks inheritance?
Encoding prototypes
All is not lost! Pierce (TAPL, paragraph 18.10) has taught us that inheritance can be encoded as open recursion. The trick is to make the reference to the self object explicit. We can do so in Haskell by putting the definitions in a record and a lambda.
data Proto = Proto {a :: A, b :: B, c :: C}
proto = \self -> Proto {
a = fa (b self) (c self),
b = fb (a self) (c self),
c = fc (a self) (b self)
}We can retrieve our original definitions by taking the fix-point:
abc = fix protoOf course, this works only because Haskell is lazy (and because the original definition did not introduce an infinite recursion in the first place). If the fields of the record are marked strict, this ceases to work.
Given that definition, it is easy to customize the value as follows:
customizedProto = \self -> proto self {
c = userFunction (a self) (b self)
}
customizedABC = fix customizedProtoThe Data.Prototype module generalizes this example, and defines types and functions to corresponding to the prototype and inheritance abstractions.
Conclusion
Yi is intended to be highly customizable. In many instances, we can use compositional abstractions to provide customization. In some other instances, we prefer to provide a prototype that user can patch.
Despite Haskell lacking inheritance, we see that the basic concepts of lambda expression and lazy evaluation can be combined to provide a very lightweight encoding for prototypes, and we take advantage of this in Yi.
]]>In this post I will motivate Yi’s incremental parsing library and describe the main ideas behind it.
Why another parsing library?
Why bothering developing another parsing framework while there exist plenty already?
First, since we want to parse many languages, in many flavors, we want to be able to reuse pieces of grammars. Since we are using Haskell, the easiest way to achieve this is to through a parser-combinator library.
Second, we want to give timely feedback to users. Therefore, the parser has to be efficient. In particular, responsivity should not depend on the length of file. On the other hand, the input file will change incrementally (as the user edits it), and the parser should take advantage of this. It should reuse previous results to speed up the parse.
Third, the parser must gracefully handle errors in the input: while files are being edited, they will inevitably contain syntactically incorrect programs at several moments.
No parsing framework combining these characteristics exists.
Approach
Hughes and Swierstra have shown how online parsers can be constructed. An online parser takes advantage of lazy evaluation: the input is analyzed only as far as needed to produce the part of the result that is forced.
Effectively, this solves part of the incremental parsing problem: since the user can see only one page of text at a time, the editor will only force so much of the result tree, and only the corresponding part of the input will be analyzed, making the parser response time independent of the length of the input.
This does not completely solve the problem though. If the user edits the end of the file, the whole input will have to be analyzed at each key-press.
Caching results
The proposed solution is to cache intermediate parsing results. For given positions in the input (say every half-page), we will store a partially evaluated state of the parsing automaton. Whenever the input is modified, the new parsing result will be computed by using the most relevant cached state, and applying the new input to it. The cached states that became invalidated will also be recomputed on the basis of the most relevant state.
Of course, the cached states will only be computed lazily, so that no cost is paid for cached states that will be discarded.
Conclusion
The strategy sketched above has several advantages:
The design is relatively straightforward, and adds only a hundred lines of code compared to the polish parsers of Hughes and Swierstra.
There is no start-up cost. A non-online approach would need to parse the whole file the first time it is loaded. In our approach loading is instantaneous, and parsing proceeds as the user scrolls down the file.
The caching strategy is independent of the underlying parsing automaton. We only require it to accept partial inputs.
Its notable that the design has a strong functional flavor:
We never update a parse tree (no references, no zipper) and still achieve incremental parsing.
We take advantage of lazy evaluation in a cool way.
The main drawback is that the user code must use the parse tree lazily, and there is no way to enforce this in any current implementation of Haskell.
]]>