2016-07-23 by Vladimir Schneider

It's Alive!

The new parser has been integrated into Markdown Navigator and an early preview version is available on the EAP update channel and for direct download.

The preview pane HTML is still generated by pegdown so you need to select the "Editor Only" split layout to disable pegdown HTML rendering from slowing down the typing response. This can be done in Languages & Frameworks > Markdown or with the toolbar button .

Everything is now done by the new parser. The typing response is simply amazing!

I rushed to replace pegdown, even without having the HTML rendering in the new parser integrated, just to be able to edit the new parser's test files. pegdown parsing of these was creating a several second lag in typing and became intolerable. I am now using the preview version in all my IDEs.

I received a file sent by a user complaining that he cannot edit the file due to pegdown timeout errors. The file is mostly text but is 500k+ in size and consists of almost 12,000 lines. The user asked if I could up the timeout limit for pegdown so he could edit the file. I upped the limit but told him that it will not help since the two second parsing will introduce delays that make editing the file impossible. I upped the limit anyway and confirmed the prediction.

Well, this file can now be edited smoothly, even with syntax highlighting set to "Lexer", which causes the file to be parsed on every keystroke. When syntax highlighting is set to "Annotator", which re-parses the file after a short pause in typing, typing is as smooth as if you were editing a short plain text document.

On the implementation side the benefits are as significant. The new flexmark-java parser is based on commonmark-java and I did my best to preserve its easy to understand, maintain and extend parser architecture. Having a year's worth of experience with the markdown plugin helped.

I knew what was needed to make implementation of plugin features easier and made the necessary changes to the AST generated by the parser to eliminate additional parsing of text on the plugin side. The latter was error prone and difficult to maintain in sync with parser changes. Along with pegdown bugs and limitations, post pegdown parsing was the number one cause of problems reported by users, including exceptions and IDE hangups.

Getting a clean AST from the parser cleans up the plugin code in a major way. Due to pegdown's AST bugs and idiosyncrasies the plugin implementation was peppered with special condition handling. I spent a good deal of time tracking and removing them after integrating the new parser because they now caused bugs.

I will be making flexmark-java available on Maven Central as soon as I have completed rudimentary java docs and figured out all the intricacies of releasing to Maven Central. I will also be adding a migration guide and some helper classes to convert pegdown Extensions flags to flexmark-java options to make the configuration for anyone using pegdown easier.

Overall, I would highly recommend to anyone currently using pegdown to switch to commonmark-java if they don't need source position tracking nor the built-in pegdown extensions. On the other hand flexmark-java will be ideal if tracking source position or the extra extensions or there is a need to turn off some core processing because custom extensions handle the constructs.

I cannot wait to complete all the tweaks of the new parser and release Markdown Navigator 2.0 with spectacular typing response and stable markdown parsing characteristics.


Constructive comments and suggestions are welcome. I can also be reached me at vladimir@vladsch.com