I would guess your best bet is to do what fixed the problem, sadly we're depending on the tokenizer based on the locale