Background
The product recently needed to read LaTeX formulas
aloud, but the current TTS engine does not understand LaTeX
syntax. If we feed raw LaTeX directly into TTS, there will
be many pronunciation errors.
A quick survey found a few existing solutions:
Neither project was designed around Chinese mathematical speech, and
integrating a web-oriented solution into a native client would add
unnecessary runtime overhead. We therefore added a translation layer
before TTS: parse the LaTeX source, build a syntax tree,
turn that tree into natural Chinese speech, and pass the result to the
speech engine.
Since the LaTeX rendering engine already parses the syntax and builds the tree, the main job is to walk that existing tree and assemble text according to rules. Because LaTeX has many commands and symbols, the workload is large. This article explains the implementation with both math and chemistry examples, and also covers the tricky parts we hit in practice.
All LaTeX reading logic in this article is for Chinese only.
Current result
The video includes spoken audio.
How it works
Architecture
If you know LaTeX, you probably know that the raw LaTeX text is
parsed into an Atom tree, then that Atom tree
recursively generates a Box tree, and Box
objects are finally rendered through a platform-independent
Graphics2D interface.
Because speech generation does not need visual layout, it can operate
directly on the existing Atom tree. Each node contributes
spoken text according to a set of rules and lookup tables. This pipeline
has no platform-specific UI dependencies, so the implementation can be
shared across platforms.
Syntax tree nodes
Because Atom nodes can be nested recursively, the speech
assembly process is also recursive. That means every Atom
subclass needs to implement the same interface.
Interface definition
1 | class Atom : public std::enable_shared_from_this<Atom> { |
Interface implementation
Take \sqrt[3]{(x-y)^3}=x-y as an example. Its Atom tree
looks like this:
Five Atom subclasses are involved here, and each of them
needs to implement getSpeechText. The examples below show
some simple assembly rules.
NthRootAtom
1 | std::wstring NthRootAtom::getSpeechText() { |
ScriptsAtom
1 | std::wstring ScriptsAtom::getSpeechText() { |
CharAtom
1 | std::wstring CharAtom::getSpeechText() { |
SymbolAtom
1 | std::wstring SymbolAtom::getSpeechText() { |
RowAtom
1 | std::wstring RowAtom::getSpeechText() { |
Once these nodes implement getSpeechText, the engine can
traverse the node tree in preorder and assemble the final TTS text:
1 | 括号 x减 y括号 的立方的立方根 等于 x减 y |
Math rules
Unicode
LaTeX contains many Unicode symbols used to express mathematical
notation. Many of them cannot be recognized by the TTS
engine, so we maintain a dedicated hash table to translate Unicode
values into spoken text.
1 | static std::unordered_map<wchar_t, std::wstring> unicodeMap = { |
Special symbols
LaTeX also has many dedicated mathematical symbol expressions. The
TTS engine either cannot recognize them or pronounces them
inaccurately, so we maintain another hash table to translate those
dedicated symbols.
1 | static std::unordered_map<std::wstring, std::wstring> speechMap = { |
Command combinations
Sometimes two neighboring commands must be combined to produce the correct rendering result and the correct TTS text. For example:
1 | \root{A} \of{B} |
The final rendering result is:
Its TTS text is “B 的 A 次方根”.
When parsing root, the parser cannot yet know that an
of command will follow, so it initially creates two
independent commands: CommandRoot and
CommandOf. During initialization, CommandOf
checks the preceding command. If it finds CommandRoot, it
takes over that command’s argument, removes the standalone root command,
and creates a single NthRootAtom from the combined
arguments.
1 | bool CommandOf::init(const std::shared_ptr<TeXParser>& tp) { |
The same combined node can then generate the correct spoken form without adding a second special case to the TTS layer.
Early truncation for inference
Some symbols look visually simple. For example, the
\equalparallel command renders like this:
Its generated Atom tree, however, is surprisingly
complex:
1 | shared_ptr<Atom> Commands::CommandEqualParallel::newI(const std::shared_ptr<TeXParser> &tp) { |
As shown in the code, it is difficult to infer the TTS text
“平行且等于” from the final TypedAtom node. To solve this,
the engine defines a special node, ContainerAtom, which
captures the command name from the beginning without affecting the
original rendering logic. Its getSpeechText method simply
returns the name, without any inference process.
1 | class ContainerAtom : public Atom { |
The earlier code can then be rewritten as:
1 | shared_ptr<Atom> Commands::CommandEqualParallel::newI(const std::shared_ptr<TeXParser> &tp) { |
Parameter stealing
Consider this LaTeX text:
1 | 37^\circ\rm{C} |
Its final rendering is $37^\circ\rm{C}$, while the
speech text should be “37摄氏度”.
For rendering, this is simple: draw three adjacent nodes in sequence.
But inferring the TTS text is troublesome because the
related nodes do not live in the same subtree. During LaTeX parsing, a
“command stack” is generated. Normally, commands just pop from the stack
in order. But here commands have semantic relationships, so special
handling is needed.
When CommandRm closes, it performs several checks:
- Pop the top
CommandRmand check whether its content isC. - Check whether the current stack top is
SubSupCom, with emptysubandsupequal to°. - If both conditions hold, steal the
supfromSubSupCom, combine it with the current contentC, generate a newSubSupCom, place it in aContainerAtom, and name it “摄氏度”. - If either condition fails, use the normal pop logic.
1 | bool CommandRm::close(const std::shared_ptr<TeXParser>& tp) { |
Stack push/pop rules
The standard absolute-value notation is easy to distinguish because the left and right delimiters are explicit:
1 | \lvert xxx \rvert |
But sometimes LaTeX uses | directly:
1 | |a-1|=0 |
Because left and right cannot be distinguished directly, a push/pop
stack strategy is needed to determine whether it represents absolute
value. The content inside the delimiters is aggregated, and a
FencedAtom is generated.
1 | std::vector<std::shared_ptr<Atom>> RowAtom::getElementsByAggregatingABSSymbols() { |
The TTS generation logic for FencedAtom
is:
1 | std::wstring FencedAtom::getSpeechText() { |
This also shows that |xxx| can express not only absolute
value, but also “modulus” and “determinant”.
Ambiguous readings
Negative or minus?
Some symbols have different readings in different scenarios, so we need rules.
For -, in -b it should be read as
“negative”; in (-b) it should also be “negative”; while in
a-b it should be read as “minus”. The rule is:
- If the previous symbol is empty, read “negative”.
- If the previous symbol is a
SymbolAtom:- if its content is
(or[, read “negative”; - otherwise read “minus”.
- if its content is
- In all other cases, read “minus”.
Similar symbols include:
±, which has ambiguous readings between plus-minus and positive-negative.+, which can mean positive or plus.
Dot, multiplication, or dot product?
For ·, sometimes it should be read as “dot”, sometimes
as “multiplied by”, and sometimes as “dot product”. The rule is:
- If the previous or next symbol is empty, read “dot”.
- If both previous and next symbols are vectors, read “dot product”.
- Otherwise, read “multiplied by”.
In practice, · can also be combined to form different
readings, such as ellipsis or remainder.
Invisible symbols
Sometimes a symbol is missing visually, but the TTS text
must still include it.
Take multiplication as an example. For (a+b)(c+d) or
2(a+b), the speech needs to add “multiplied by” between the
two parts. The rule is:
- If the current symbol is a
FencedAtomin the form(xxx):- if the previous symbol is
CharAtom, add “multiplied by”; - if the previous symbol is also a
FencedAtomin the form(xxx), add “multiplied by”; - otherwise do nothing.
- if the previous symbol is
- Otherwise do nothing.
Improper fractions
Some structures are logically a whole, but can be rendered
independently. For example, the mixed number
$3\cfrac{1}{2}$ has LaTeX text 3\frac{1}{2}.
The rendering engine parses it into two Atoms: one
CharAtom for 3, and one FracAtom
for 1/2. For rendering, they can simply be drawn one after
another.
But for TTS, they are semantically related. Therefore,
when the current Atom is a FracAtom, the
engine needs to combine it with the previous Atom and
determine whether they may form a mixed number.
Mispronunciation
During testing, some pronunciations were inaccurate. In those cases,
we can use the TTS engine’s SSML
support to correct pronunciation.
1 | // a and A |
Chemistry rules
Flattening a subtree locally
The TTS reading of chemistry is completely different
from math. Chemistry involves recognition of chemical names, chemical
equations, ions, and similar concepts, so the existing tree traversal
rules are basically not reusable. This means that to support chemical
TTS, a completely different framework from the math one is
needed.
Consider this chemical formula:
1 | \ce{H2SO4} |
The generated node tree is shown above. If we still read it with tree traversal, it is difficult to associate it with the name “硫酸”. This conflict mainly comes from the natural mismatch between tree structure and linear split-and-lookup.
Most molecular formulas have a shallow structure and use
ScriptsAtom primarily for subscripts. We can therefore
flatten that part of the tree locally. Instead of recursively reading a
ScriptsAtom, the chemistry reader converts it into a single
normalized token.
The rule is: ScriptsAtom has three elements:
base, subscript, and superscript.
Convert each element to string, prefix subscripts with _,
prefix superscripts with ^, and concatenate them.
1 | class ScriptsAtom : public Atom, public HasTrueBase { |
With this rule, the tree above is flattened into this form:
Recognizing chemical formulas
After flattening, the TTS traversal result for the
formula above is H_2, S, O_4. How
do we turn these scattered texts into a chemical formula name?
Dictionary mapping is one option, but there are several problems:
- During traversal, how do we determine that
H_2SO_4is a group? - How do we know whether
H_2SO_4has a corresponding chemical name? - If it has no name, how do we fall back? After all,
H_2Salso has a name, “硫化氢”.
This problem is very similar to search engines. When you type some characters, search engines can show a list of associated suggestions.
This is a prefix-matching problem, so a trie is a natural fit for recognizing chemical formula names.
The figure shows a constructed Trie tree of chemical
formula names. Every node stores a name, if any, representing the
chemical formula on the path from the root to the current node. The
search process uses a greedy algorithm:
- Input
H_2and reach the red node. Since it has a name, record “氢气”. - Continue input
Sand reach the green node. It also has a name, so overwrite the name as “硫化氢”. - Continue input
O_4and reach the blue node. It also has a name, so overwrite the name as “硫酸”.
The search ends and the final chemical formula name is “硫酸”. Core code:
1 | std::wstring ChemTTSEngine::getSpeechText(const std::shared_ptr<RowAtom> &atom) { |
Building the chemical formula table
Since chemical formula recognition requires chemical names, a chemical formula name database is naturally needed.
Some data tables were built with reference to Wikipedia, and the final scale is expected to be around 3000 to 4000 entries, or even more.
Keys in the table follow several rules:
- Separate characters with
#. - Keep consecutive digits together.
- For
ScriptsAtom, prefix subscripts with_, prefix superscripts with^, and keep the whole part as one unit.
1 | std::unordered_map<std::wstring, std::wstring>& ChemTTSDatabase::getChemicalNameMap() { |
This manual work can be handed to ChatGPT: teach it the
rules and let it help build the database.
Loading the chemical formula table
The chemical-formula trie is built by inserting each normalized token sequence from left to right. For every token, the builder follows an existing child node or creates one when necessary; the final node stores the compound name.
1 | struct TrieNode { |
Recognizing chemical equations
Once chemical formulas can be recognized, we can further recognize chemical equations. Compared with chemical formulas, chemical equations contain the following concepts:
The following data structure is defined, and the existence of
Equation is used as the fundamental criterion for
determining whether something is a chemical equation.
1 | enum class ChemElementType { |
Normally, traversed elements are pushed onto a stack. If an
XArrowAtom or XLongequalAtom is encountered,
the current expression is considered a chemical equation. When popping
the stack, chemical formulas without coefficients get a default
coefficient of 1. If it is not a chemical equation, there
is no need to add coefficients.
The core code is:
1 | for (int i = 0; i < elements.size(); ++i) { |
The final TTS output for the above chemical equation
is:
1 | 2个过氧化钠和2个水在加热的条件下反应生成4个氢氧化钠和1个氧气气体 |
Reading concentrations
Ion concentration is read differently from a chemical equation. The
equals sign in the middle simply means “equals”, so we only need to
recognize concentration in the form c(xxx). Similar to
absolute value, a stack push/pop method is used to determine whether the
text expresses concentration. Core code:
1 | std::vector<std::shared_ptr<Atom>> ChemTTSEngine::getElementsByAggregatingMolaritySymbols(const std::vector<std::shared_ptr<Atom>>& elements) { |
The reading is:
1 | 次氯酸的浓度 加 次氯酸根离子的浓度 等于 碳酸氢根离子的浓度 加 碳酸的浓度 |
Hydrates
In chemistry, many substances crystallize from aqueous solution and
combine with a certain number of water molecules. These water molecules
are called waters of crystallization. Their reading is usually “x
hydrate y”, where x is the number of water molecules and
y is the compound name before ·.
CuSO4·5H2O: 五水合硫酸铜Na2CO3·10H2O: 十水合碳酸钠
If all hydrates are stored in the table, there are two problems:
- They cannot be fully enumerated, so omissions are inevitable.
- There is a lot of waste.
Both problems disappear if the parser extracts the number of water
molecules separately. It can look up the compound before ·,
then construct the hydrate name from the coefficient and the matched
compound.
The idea is: during Trie traversal, if the · symbol is
encountered, check whether the following substance is
number + H2O or just H2O. If so, stop the loop
and combine the previously found chemical symbol name y
into the final name: x hydrate y.
1 | for (int i = 0; i < elements.size(); ++i) { |
There are also cases where this rule does not fit naturally. For
example, NH3·H2O would become “一水合氨气” under the rule
above, which sounds awkward. The correct name is “一水合氨”. For this
kind of case, we can add a whitelist: when encountering ·,
look up the full form in the table first, and only fall back to the rule
if the lookup fails.
1 | {L"N#H_3#·#H_2#O" , L"一水合氨"}, |
Summary
Converting LaTeX into natural Chinese speech is not a one-time parsing task. Every new notation, ambiguous reading, and chemistry exception adds another language rule to test. The tree-based pipeline gives those rules a stable foundation, but pronunciation quality will continue to depend on real examples and careful human review.