Libass supports some libass-specific extension to ASS. Those extensions aren't all meant for use in ditributed softsubs, but e.g. on-the-fly converted subs. As such they do not have as strong stability guarantees as VSFilter-compatible ASS and may sometimes be tweaked. Of course using any of them will also lead to incompatibilities with other renderers, so before using them in distributed softsubs carefully consider if the extensions bring you a significant benefit you are unable to achieve by standard means, justifying the reduced portability. In – hopefully rare – cases these incompatibilities can also go the other way around, with a *VSFilter-tailored script accidentally triggering a libass extension.
*If you want to write portable scripts, you might also want to take a look at Libass-VSFilter differences.

Here's the short overview of extensions:

1. Literal Curly Brackets
2. Kerning
3. Ligatures
4. Language
5. BorderStyle=4
6. Encoding=-1
7. Custom Format Lines

We will categorise the detriment of an extension, when script is not displayed with libass into:

soft-break:

Typesets may be off, text may not look as nice, but the general text content/meaning should still get across. Albeit be it with potentially some annoyances for the reader.

hard-break:

Significant text portions go missing.

usually [soft-break], but [hard-break] possible if care is not taken

In regular ASS literal opening and closing brackets {, } can only be output in specific circumstances. { can only be printed if after it the event's text does not have any closing bracket in it. } is only printed if it is not preceded by any unterminated opening curly bracket.
In libass only you can produce literal curly brackets independent of context by escaping them with backslashes.
Take a look at how the following will render in libass and *VSFilter:

Did you know\{: some people can't see this text\} ?

libass:

Did you know{: Some people can't see this text} ?

*VSFilter:

Did you know\ ?

To avoid this, provide a closing cb in form of an empty command block to *VSFilter after all escaped opening cbs, like this \{{}Hi. As for closing cb: If you take care you don't need to escape them in either *VSFilter or libass, as without a matching opening cb they are not treated special. An unmatched closing cb will just render as an literal c }.
Let's take a look at the following example:

\{{}Hi} \{{}Hi!\}

This is still not portable (as expected), but much less bad than before:

libass:

{Hi} {Hi!}

*VSFilter:

\Hi} \Hi!\}

[soft-break]

With libass you can use the special header Kerning: yes in [Script Info] to enable text kerning. *VSFilter will simply ignore this and apply GDI's poorly documented heuristic of not kerning simple scripts but sometimes kerning text involving complex scripts. If your file contains typesets some things may break with/without kerning.
In absence of typesets, this should probably be the safe to use.

HACK: You can enable ligatures and GPOS kerning in *VSFilter by using a subset font that lacks any glyph for U+0022, the straight quotation mark. Use curved quotation marks instead, which will make your text look better anyway (assuming a good font).

[soft-break]

Libass (currently) supports automatic ligatures, unlike *VSFilter. Ligatures can be useful for eg handwriting-fonts or any more complex font. Caution: When rendered with *VSFilter no ligatures will be displayed, this may lessen the intended effect or just look very strange. The text content will still carry across though.
Problems may arise the other way around, when broken fonts are being used. If the font has invalid or nonsensical ligatures the text may display ""fine"" in VSFilter but mangled in libass. See eg here for an example.

NOTE: It is being discussed if this behaviour should be able to be toggled off with, or even guarded behind a [Script Info] header like with Kerning (requiring explicit enabling). If you are relying on this now, your script may soft-break in future libass versions should this header be opt-in instead of opt-out.

HACK: You can enable ligatures and GPOS kerning in *VSFilter by using a subset font that lacks any glyph for U+0022, the straight quotation mark. Use curved quotation marks instead, which will make your text look better anyway (assuming a good font).

[soft-break]

By adding a Language: header to [Script Info] you can enable some langage specific character variants. Currently only ISO 639-1 two letter codes are accepted.
VSFilter will ignore this.

In the future libass may enable support for the full range of IETF BCP 47 codes as supported by Harfbuzz. However, it is still advisable to use ISO 639-1 codes instead of the three letter variants whenever possible as libraries involved in possible future uses of the Language tag are more likely to understand ISO 639-1 then more complex language tags.

[soft-break]

Despite being called a "Border"Style, this actually changes the Shadow. Similar to BorderStyle=3, the shadow will be rendered as a rectangular box around the text. Unlike BoderStyle=3 the box is not created for each line separately, but just one box for the event as a whole. (This is especially noticeable with semi-transparent boxes.) The border will still be drawn as with BorderStyle=1, just with an additional background box instead of a shadow. This background box will cover the whole event text, including borders.
As this BorderStyle=4-box replaces the shadow, its colour and transparency is not changed with \3c and \3a, but \4c and \4a. Since libass 0.13.7 it is possible to adjust the size of the BorderStyle=4 box's padding using \shad. Since libass 0.15.0 the padding is calculated from glyph border instead of glyph fill (so borders can no longer overflow the background box).

*VSFilter will render BorderStyle=4 lines as regular BorderStyle=1.

See the following image for illustration. Notice how the colour of the box differs between BorderStyle=3 and BorderStyle=4 and that there is a visible overlap of boxes between the lines with Borderstyle=3.

The style used corresponds to {\bord2.5\3c&H&28DB00\3a&H77}{\shad4\4c&H000000\4a&H99} with only the BorderStyle field changing between the images..

[hard-break] possible
(text doesn't go missing, but the reordering and other changes can potentially make it very hard to understand and break signs)

Setting Encoding=-1 (either in the Style or through \fe) enables auto-detection of the text's base direction; VSFilter always uses left-to-right and rejects -1 as an invalid value. For right-to-left languages and mixed RTL and LTR text this will yield “more intuitive” results matching what most other text renderers produce. Setting Encoding: -1 also enables some other VSFilter incompatibilities like bidi-reordering and shaping the whole line instead of each ASS-run individually to achieve the “more intuitive” ordering.

Like with all extensions, you should not use it if the file might be played back by VSFilter or some other non-libass renderer. As Encoding: -1's text layout gives less stability guarantees (to e.g. account for future changes to Unicode) you should also not use this for complex typesetting even when it's guaranteed to use libass; only use this for (possibly lightly styled) dialogue.

very very [hard break]

*VSFilter doesn't parse the [Style] and [Events] format lines and always assumes the following layout:

[V4+ Styles]
Format: Name, Fontname, Fontsize, PrimaryColour, SecondaryColour, OutlineColour, BackColour, Bold, Italic, Underline, Strikeout, ScaleX, ScaleY, Spacing, Angle, BorderStyle, Outline, Shadow, Alignment, MarginL, MarginR, MarginV, Encoding
...
[Events]
Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text

Libass allows to shuffle or shorten the format lines like this. Note that Text must always be the last column and that if the Style column in the Event field is missing you will always get the default ASS-Style, not the Default style of the script ! (RFC: Is this intended or a bug ?)

[V4+ Styles]
Format: Name, Fontname, Fontsize, PrimaryColour, SecondaryColour, OutlineColour, BackColour, Bold, Italic, Underline, BorderStyle, Outline, Shadow, Alignment, MarginL, MarginR, MarginV
...
[Events]
Format: Start, End, Style, Text

As using this libass extension renders all other renderers unable to do anything with your file, the use of it is especially discouraged. The benefit of shorter lines, are outweight by the detriment of complete non-portability in almost all cases.
^{Please don't use this unless you are absolutely sure no renderer other than libass will see your script.}

NOTE: Using custom format lines will also change the default value of ScaledBorderAndShadow to yes instead of no, if not set explicitly. This is done for backwards compatibility.