Lazy dictionaries, parse-tree surgery, and a faster compile-check cycle

General
1

Busy week on the engine side — six point releases in five days, from v3.2.0 up through v3.6.0, plus the usual mirrored builds in the Windows/Linux/Mac packages and a couple of housekeeping releases in vscode-nlp. Let me walk through what actually changed and why.

Big dictionaries, loaded lazily

The headline item this week is lazy loading for *-full dictionaries. Previously, opening one of the big “full” dictionary files meant reading the whole thing into memory up front — fine for small vocabularies, painful once you’re working with full-scale lexicons. Now the engine loads words one at a time, on demand, as the analyzer actually needs them (v3.2.0).

Right after that went in, I hit a bug where you couldn’t have more than one *-full dictionary open simultaneously — the lazy loader was stepping on itself. Fixed in v3.2.1. If you’re using multiple large dictionaries in the same analyzer, that fix matters.

Parse-tree manipulation: pnpush

Added a new NLP++ function, pnpush (v3.5.0), which lets a rule push a parse node onto a new parent directly from code, rather than relying purely on the grammar’s tree-building as it walks the input. Shortly after, I extended it (v3.5.1) so that when pnpush moves a node under a new parent, the node’s existing attributes come along with it instead of getting dropped. This was a “found it while using it” fix — nothing worse than restructuring a tree and losing your annotations in the process.

Compile without running

v3.6.0 adds -COMPILEANA, a mode that compiles an analyzer without actually running it against text. If you’ve got a big passage or a large test corpus and you just want to know “does this grammar compile cleanly,” you no longer have to pay the cost of a full analysis pass to find out. Useful for CI-style sanity checks on grammar changes.

Smaller NLP++ language additions

  • loadkbb and loaddict — new functions to load knowledge-base and dictionary files directly from NLP++ code, rather than only via analyzer configuration (v3.4.0).
  • Token names prefixed with an underscore can now include digits (v3.3.0) — a small grammar-authoring annoyance fixed, no more working around _tok1 not being legal.

VS Code extension: help goes markdown-first

Over on vscode-nlp (v3.1.22–3.1.24), I switched the built-in help system to prefer markdown help files over the old HTML/browser-based lookup, and dropped the browser help menu items entirely. Simpler to maintain, and it means help content lives right alongside the docs I’m already writing in markdown instead of a separate HTML pipeline.

What’s next

With lazy dictionaries and pnpush attribute handling settled, I want to spend some time actually stress-testing -COMPILEANA on a couple of the larger conceptual grammars to see how much time it saves in practice — I’ll report back with numbers next week.