Unicode string fixes, python passes go anywhere, and a much smoother release pipeline

General
1

This was a heavy engine week — 113 commits, no PRs (I’ve been working mostly solo on these fixes), and a wall of releases. Let me explain why there are so many version bumps, because it looks alarming in the log but it’s actually the release pipeline doing its job correctly for once.

Engine stability: the good kind of boring fixes

A cluster of nasty bugs got squashed in nlp-engine this week:

  • Unicode string functions — strwrap, strtrim, and friends were operating on bytes instead of UTF-8 characters, which meant multi-byte characters could get mangled or dropped mid-string. Fixed, and I added the missing strtrim-on-all-spaces and strwrap-drops-chars cases that had been open a while (#544, #506).
  • Re-entrant analyze() bugs — if you called analyze() from within a callback (Python passes, embedding scenarios), the NLP object could get orphaned or freed out from under itself. Two separate use-after-free / rebuild bugs here, both fixed (#658, #659).
  • Infinite hang on certain inputs when sorting characters past the last full-dictionary word — now terminates correctly instead of spinning forever.
  • Missing input file no longer SIGSEGVs — you get a real error message instead.
  • Quieted some noisy con_add_root stderr output that wasn’t telling you anything useful.

None of these are glamorous, but if you’ve hit a crash or a hang running the engine on real-world text, one of these is probably why.

Python passes are now first-class, anywhere in the pipeline

The bigger conceptual change this week: I simplified the python pass type down to a single, position-aware type that can be placed anywhere in the analyzer sequence — including before the tokenizer runs. That’s new. Previously python passes were locked to specific spots; now you can pre-process raw input in Python before NLP++ ever tokenizes it. VS Code got a matching “Insert Python Pass Before Tokenizer” menu item, and the rule compiler now accepts unquoted _xVAR(attr) syntax cleanly, which was tripping people up.

Compile Analyzer Only

Added a -COMPILEANA / analyzer-only compile mode across the board — engine, VS Code, Windows/Linux/Mac wrapper scripts, and the Python and TypeScript packages all got this in sync. It compiles just the analyzer without a full rebuild, which should speed up iteration when you’re only touching rule files.

Why so many releases

The CI now auto-bumps the nlp-engine submodule and publishes the Python package (and the platform binary repos) on every engine release, gated on the last released engine rather than in-run deltas. That’s why you see a steady staircase of v3.5.x releases this week — each engine fix percolated automatically through nlp-engine-windows/linux/mac and py-package-nlpengine without me having to manually sync anything. It’s more releases in the log, but it means the binaries and packages never drift from the engine again. Landed at v3.6.2 for the engine, v2.0.21 for the Python package.

Next

I want to spend some time actually writing analyzers with the pre-tokenizer python pass now that it’s in — curious what people build with raw-text preprocessing before NLP++ tokenization kicks in. If you try it, tell me how it breaks.