Teaching a TTS Engine to Read LaTeX in Chinese

Background

The product recently needed to read LaTeX formulas aloud, but the current TTS engine does not understand LaTeX syntax. If we feed raw LaTeX directly into TTS, there will be many pronunciation errors.

A quick survey found a few existing solutions:

Neither project was designed around Chinese mathematical speech, and integrating a web-oriented solution into a native client would add unnecessary runtime overhead. We therefore added a translation layer before TTS: parse the LaTeX source, build a syntax tree, turn that tree into natural Chinese speech, and pass the result to the speech engine.

Since the LaTeX rendering engine already parses the syntax and builds the tree, the main job is to walk that existing tree and assemble text according to rules. Because LaTeX has many commands and symbols, the workload is large. This article explains the implementation with both math and chemistry examples, and also covers the tricky parts we hit in practice.

All LaTeX reading logic in this article is for Chinese only.

Current result

The video includes spoken audio.

How it works

Architecture

If you know LaTeX, you probably know that the raw LaTeX text is parsed into an Atom tree, then that Atom tree recursively generates a Box tree, and Box objects are finally rendered through a platform-independent Graphics2D interface.

Because speech generation does not need visual layout, it can operate directly on the existing Atom tree. Each node contributes spoken text according to a set of rules and lookup tables. This pipeline has no platform-specific UI dependencies, so the implementation can be shared across platforms.

Syntax tree nodes

Because Atom nodes can be nested recursively, the speech assembly process is also recursive. That means every Atom subclass needs to implement the same interface.

Interface definition

1
2
3
4
class Atom : public std::enable_shared_from_this<Atom> {
public:
virtual std::wstring getSpeechText() = 0;
};

Interface implementation

Take \sqrt[3]{(x-y)^3}=x-y as an example. Its Atom tree looks like this:

Teaching a TTS Engine to Read LaTeX in Chinese: Interface implementation (figure 1)

Five Atom subclasses are involved here, and each of them needs to implement getSpeechText. The examples below show some simple assembly rules.

  • NthRootAtom
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
std::wstring NthRootAtom::getSpeechText() {
std::wstring result = L"";
std::wstring baseSpeech = base->getSpeechText();
if (root != nullptr) {
std::wstring rootSpeech = root->getSpeechText();
if (rootSpeech == L"3") {
result = baseSpeech + L"的立方根";
} else {
result = rootSpeech + L"次根号下" + baseSpeech;
}
} else {
result = baseSpeech.size() > 1 ? L"根号下" + baseSpeech : L"根号" + baseSpeech;
}
return L" " + result + L" ";
}
  • ScriptsAtom
1
2
3
4
5
6
7
8
9
10
11
std::wstring ScriptsAtom::getSpeechText() {
std::wstring baseText = base != nullptr ? base->getSpeechText() : L"";
if (superscript->getSpeechText() == L"2") {
return baseText+ subscript->getSpeechText() + L"的平方";
} else if (superscript->getSpeechText() == L"3") {
return baseText + subscript->getSpeechText() + L"的立方";
} else {
std::wstring superscriptStr = superscript->getSpeechText();
return baseText + subscript->getSpeechText() + L"的" + superscriptStr + L"次方";
}
}
  • CharAtom
1
2
3
std::wstring CharAtom::getSpeechText() {
return StringHelper::toString(c);
}
  • SymbolAtom
1
2
3
4
5
6
7
8
std::wstring SymbolAtom::getSpeechText() {
std::wstring speechText = Configuration::Instance().getSpeechText(unicode);
if (speechText.length() > 0) {
return speechText + L" ";
}
std::wstring symbleStr = StringHelper::toString(unicode);
return symbleStr;
}
  • RowAtom
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
std::wstring RowAtom::getSpeechText()  {
std::wstring result = L"";

// If RowAtom itself has a mapped name, prefer that name.
if (name.length() > 0) {
return Configuration::Instance().getSpeechText(name);
}

// Absolute value recognition and aggregation.
auto newElements = getElementsByAggregatingABSSymbols();

std::wstring s;
int index = 0;
for (auto& e : newElements) {
std::shared_ptr<Atom> previousAtomPtr = nullptr;
std::shared_ptr<Atom> nexAtomPtr = nullptr;

if (index > 0) {
previousAtomPtr = newElements[index -1];
}
if (index < newElements.size() - 1) {
nexAtomPtr = newElements[index + 1];
}

s += Configuration::Instance().rectifySpeechText(
e,
previousAtomPtr,
nexAtomPtr);
++index;
}
return s;
}

Once these nodes implement getSpeechText, the engine can traverse the node tree in preorder and assemble the final TTS text:

1
括号 x减 y括号 的立方的立方根 等于 x减 y

Math rules

Unicode

LaTeX contains many Unicode symbols used to express mathematical notation. Many of them cannot be recognized by the TTS engine, so we maintain a dedicated hash table to translate Unicode values into spoken text.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
static std::unordered_map<wchar_t, std::wstring> unicodeMap = {
// Shapes
{L'▱', L"平行四边形"},
{L'⊙', L"圆"},
{L'\u25B3', L"三角形"},
{L'\u25B5', L"三角形"},
{L'\u2220', L"角"},
{L'\u22A5', L"垂直于"},
{L'\u2225', L"平行于"},
{L'\u22D5', L"等于和平行于"},
{L'\ue363', L"垂直于"},

// Sets
{L'\u222A', L"并集"}, // ∪
{L'\u2229', L"交集"}, // ∩
{L'\u2282', L"真包含于"}, // ⊂
{L'\u2286', L"包含于"}, // ⊆
{L'\u2284', L"非真包含于"}, // ⊄
{L'\u2288', L"非包含于"}, // ⊈
{L'\ue2ae', L"非包含于"}, // ⊈

......
}

Special symbols

LaTeX also has many dedicated mathematical symbol expressions. The TTS engine either cannot recognize them or pronounces them inaccurately, so we maintain another hash table to translate those dedicated symbols.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
static std::unordered_map<std::wstring, std::wstring> speechMap = {
// Trigonometric functions
{L"sin", L"sine"},
{L"cos", L"cosine"},
{L"tan", L"tangent"},
{L"cot", L"cotangent"},
{L"sec", L"secant"},
{L"csc", L"cosecant"},
{L"arcsin", L"arc sine"},
{L"arccos", L"arc cosine"},
{L"arctan", L"arc tangent"},
{L"arccot", L"arc cotangent"},
{L"sinh", L"shine"},
{L"cosh", L"cosh"},
{L"tanh", L"tanch"},
{L"coth", L"coth"},
{L"sech", L"setch"},
{L"csch", L"cosetch"},

// Units
{L"km" , L"千米"},
{L"m" , L"米"},
{L"dm" , L"分米"},
{L"cm" , L"厘米"},
{L"mm" , L"毫米"},
{L"μm" , L"微米"},
{L"nm" , L"纳米"},
......
}

Command combinations

Sometimes two neighboring commands must be combined to produce the correct rendering result and the correct TTS text. For example:

1
\root{A} \of{B}

The final rendering result is:

Its TTS text is “B 的 A 次方根”.

When parsing root, the parser cannot yet know that an of command will follow, so it initially creates two independent commands: CommandRoot and CommandOf. During initialization, CommandOf checks the preceding command. If it finds CommandRoot, it takes over that command’s argument, removes the standalone root command, and creates a single NthRootAtom from the combined arguments.

1
2
3
4
5
6
7
bool CommandOf::init(const std::shared_ptr<TeXParser>& tp) {
std::shared_ptr<RowAtom> at = tp->steal();
std::shared_ptr<NthRootAtom> root = std::dynamic_pointer_cast<NthRootAtom>(at->getLastAtom());
this->root = root != nullptr ? root->getTrueBase() : EmptyAtom::get();
base = std::make_shared<RowAtom>();
return true;
}

The same combined node can then generate the correct spoken form without adding a second special case to the TTS layer.

Early truncation for inference

Some symbols look visually simple. For example, the \equalparallel command renders like this:

Teaching a TTS Engine to Read LaTeX in Chinese: Early truncation for inference (figure 2)

Its generated Atom tree, however, is surprisingly complex:

1
2
3
4
5
6
7
8
shared_ptr<Atom> Commands::CommandEqualParallel::newI(const std::shared_ptr<TeXParser> &tp) {
std::shared_ptr<RowAtom> ra = make_shared_from_list<RowAtom>({std::make_shared<SpaceAtom>(Unit::MU, 8.0, 0.0, 0.0), SymbolAtom::get(L"shortparallel")});
auto raiseAtom = std::make_shared<RaiseAtom>(ra, std::make_shared<TeXLength>(Unit::MU, 3.0), nullptr, nullptr);
auto lapAtom = std::make_shared<LapedAtom>(raiseAtom, L'r');
auto raiseAtom2 = std::make_shared<RaiseAtom>(SymbolAtom::get(L"equals"), std::make_shared<TeXLength>(Unit::MU, -3.0), nullptr, nullptr);
auto ra2 = make_shared_from_list<RowAtom>({lapAtom, raiseAtom2});
auto result = std::make_shared<TypedAtom>(TYPE_RELATION, ra2);
return result;

As shown in the code, it is difficult to infer the TTS text “平行且等于” from the final TypedAtom node. To solve this, the engine defines a special node, ContainerAtom, which captures the command name from the beginning without affecting the original rendering logic. Its getSpeechText method simply returns the name, without any inference process.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
class ContainerAtom : public Atom {
private:
std::shared_ptr<Atom> base;
std::wstring description;
public:
ContainerAtom(const std::shared_ptr<Atom>& atom, const std::wstring& desctiption = L"");
virtual std::wstring getSpeechText() override;
......
}

std::wstring ContainerAtom::getSpeechText() {
std::wstring speechText = Configuration::Instance().getSpeechText(description);
return speechText;
}

The earlier code can then be rewritten as:

1
2
3
4
5
6
7
8
9
shared_ptr<Atom> Commands::CommandEqualParallel::newI(const std::shared_ptr<TeXParser> &tp) {
std::shared_ptr<RowAtom> ra = make_shared_from_list<RowAtom>({std::make_shared<SpaceAtom>(Unit::MU, 8.0, 0.0, 0.0), SymbolAtom::get(L"shortparallel")});
auto raiseAtom = std::make_shared<RaiseAtom>(ra, std::make_shared<TeXLength>(Unit::MU, 3.0), nullptr, nullptr);
auto lapAtom = std::make_shared<LapedAtom>(raiseAtom, L'r');
auto raiseAtom2 = std::make_shared<RaiseAtom>(SymbolAtom::get(L"equals"), std::make_shared<TeXLength>(Unit::MU, -3.0), nullptr, nullptr);
auto ra2 = make_shared_from_list<RowAtom>({lapAtom, raiseAtom2});
auto result = std::make_shared<TypedAtom>(TYPE_RELATION, ra2);
return std::make_shared<ContainerAtom>(std::make_shared<TypedAtom>(TYPE_RELATION, result), L"平行且等于");
}

Parameter stealing

Consider this LaTeX text:

1
37^\circ\rm{C}

Its final rendering is $37^\circ\rm{C}$, while the speech text should be “37摄氏度”.

For rendering, this is simple: draw three adjacent nodes in sequence. But inferring the TTS text is troublesome because the related nodes do not live in the same subtree. During LaTeX parsing, a “command stack” is generated. Normally, commands just pop from the stack in order. But here commands have semantic relationships, so special handling is needed.

Teaching a TTS Engine to Read LaTeX in Chinese: Parameter stealing (figure 3)

When CommandRm closes, it performs several checks:

  1. Pop the top CommandRm and check whether its content is C.
  2. Check whether the current stack top is SubSupCom, with empty sub and sup equal to °.
  3. If both conditions hold, steal the sup from SubSupCom, combine it with the current content C, generate a new SubSupCom, place it in a ContainerAtom, and name it “摄氏度”.
  4. If either condition fails, use the normal pop logic.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
bool CommandRm::close(const std::shared_ptr<TeXParser>& tp) {
auto a = size->simplify();
tp->pop();

auto secondTop = tp->peek();
if (std::dynamic_pointer_cast<CharAtom>(a) != nullptr &&
std::static_pointer_cast<CharAtom>(a)->getCharacter() == L'C' &&
std::dynamic_pointer_cast<SubSupCom>(secondTop) &&
std::static_pointer_cast<SubSupCom>(secondTop)->getSup()) {
auto sup = std::static_pointer_cast<SubSupCom>(secondTop)->getSup();
if (std::dynamic_pointer_cast<SymbolAtom>(sup) != nullptr && std::static_pointer_cast<SymbolAtom>(sup)->getUnicode() == L'∘') {
std::static_pointer_cast<SubSupCom>(secondTop)->setSup(nullptr);
auto subsupAtom = SubSupCom::get(EmptyAtom::get(), nullptr, sup);
std::vector<std::shared_ptr<Atom>> atoms{subsupAtom, std::make_shared<RomanAtom>(a)};
auto ra = std::make_shared<RowAtom>(atoms);
auto result = std::make_shared<ContainerAtom>(ra, L"摄氏度");
tp->addToConsumer(result);
}
return true;
}

std::shared_ptr<Atom> at = newI(tp, a);
auto last = tp->getLastAtom();
tp->closeConsumer(at);
return true;
}

Stack push/pop rules

The standard absolute-value notation is easy to distinguish because the left and right delimiters are explicit:

1
\lvert xxx \rvert

But sometimes LaTeX uses | directly:

1
|a-1|=0

Because left and right cannot be distinguished directly, a push/pop stack strategy is needed to determine whether it represents absolute value. The content inside the delimiters is aggregated, and a FencedAtom is generated.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
std::vector<std::shared_ptr<Atom>> RowAtom::getElementsByAggregatingABSSymbols() {
// extract abs
std::stack<std::shared_ptr<Atom>> stack;
std::vector<std::shared_ptr<Atom>> atoms;
std::vector<std::shared_ptr<Atom>> newElements;
for (auto& e : elements) {
if (std::dynamic_pointer_cast<SymbolAtom>(e) != nullptr && std::static_pointer_cast<SymbolAtom>(e)->getUnicode() == L'|') {
if (stack.empty()) {
// push sentinel
stack.push(e);
} else {
// make a FencedAtom
std::shared_ptr<RowAtom> ra = std::make_shared<RowAtom>(atoms);
auto fenceAtom = std::make_shared<FencedAtom>(ra->simplify(), std::static_pointer_cast<SymbolAtom>(e), std::static_pointer_cast<SymbolAtom>(e));
atoms.clear();
newElements.push_back(fenceAtom);

// pop sentinel
stack.pop();
}
} else {
if (stack.empty()) {
newElements.push_back(e);
} else {
atoms.push_back(e);
}
}
}

if (!stack.empty()) {
newElements.push_back(stack.top());
for (auto& ele : atoms) {
newElements.push_back(ele);
}
}
return newElements;
}

The TTS generation logic for FencedAtom is:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
std::wstring FencedAtom::getSpeechText() {
std::wstring speechText = L"";
std::wstring leftSpeechText = left ? left->getSpeechText() : L"";
std::wstring rightSpeechText = right ? right->getSpeechText() : L"";
std::wstring baseSpeechText = base->getSpeechText();

bool isArrayAtom = std::dynamic_pointer_cast<ArrayAtom>(base) != nullptr;

if (leftSpeechText == L"|" && rightSpeechText == L"|") {
if (isArrayAtom) {
return L"行列式 " + baseSpeechText;
} else {
if (std::dynamic_pointer_cast<UnderOverArrowAtom>(base) != nullptr) {
return baseSpeechText + L"的模";
} else {
return baseSpeechText + L"的绝对值";
}
}
}
......
}

This also shows that |xxx| can express not only absolute value, but also “modulus” and “determinant”.

Ambiguous readings

Negative or minus?

Some symbols have different readings in different scenarios, so we need rules.

For -, in -b it should be read as “negative”; in (-b) it should also be “negative”; while in a-b it should be read as “minus”. The rule is:

  1. If the previous symbol is empty, read “negative”.
  2. If the previous symbol is a SymbolAtom:
    1. if its content is ( or [, read “negative”;
    2. otherwise read “minus”.
  3. In all other cases, read “minus”.

Similar symbols include:

  1. ±, which has ambiguous readings between plus-minus and positive-negative.
  2. +, which can mean positive or plus.

Dot, multiplication, or dot product?

For ·, sometimes it should be read as “dot”, sometimes as “multiplied by”, and sometimes as “dot product”. The rule is:

  1. If the previous or next symbol is empty, read “dot”.
  2. If both previous and next symbols are vectors, read “dot product”.
  3. Otherwise, read “multiplied by”.

In practice, · can also be combined to form different readings, such as ellipsis or remainder.

Invisible symbols

Sometimes a symbol is missing visually, but the TTS text must still include it.

Take multiplication as an example. For (a+b)(c+d) or 2(a+b), the speech needs to add “multiplied by” between the two parts. The rule is:

  1. If the current symbol is a FencedAtom in the form (xxx):
    1. if the previous symbol is CharAtom, add “multiplied by”;
    2. if the previous symbol is also a FencedAtom in the form (xxx), add “multiplied by”;
    3. otherwise do nothing.
  2. Otherwise do nothing.

Improper fractions

Some structures are logically a whole, but can be rendered independently. For example, the mixed number $3\cfrac{1}{2}$ has LaTeX text 3\frac{1}{2}. The rendering engine parses it into two Atoms: one CharAtom for 3, and one FracAtom for 1/2. For rendering, they can simply be drawn one after another.

But for TTS, they are semantically related. Therefore, when the current Atom is a FracAtom, the engine needs to combine it with the previous Atom and determine whether they may form a mixed number.

Mispronunciation

During testing, some pronunciations were inaccurate. In those cases, we can use the TTS engine’s SSML support to correct pronunciation.

1
2
3
4
5
6
7
8
// a and A
<phoneme alphabet=\"ipa\" ph=\"eɪ\">a</phoneme>g

// x and X
<phoneme alphabet=\"ipa\" ph=\"eɪks\">x</phoneme>

// 2
It is often read as “两”, so we can directly translate it as “二”.

Chemistry rules

Flattening a subtree locally

The TTS reading of chemistry is completely different from math. Chemistry involves recognition of chemical names, chemical equations, ions, and similar concepts, so the existing tree traversal rules are basically not reusable. This means that to support chemical TTS, a completely different framework from the math one is needed.

Consider this chemical formula:

1
\ce{H2SO4}
Teaching a TTS Engine to Read LaTeX in Chinese: Flattening a subtree locally (figure 4)

The generated node tree is shown above. If we still read it with tree traversal, it is difficult to associate it with the name “硫酸”. This conflict mainly comes from the natural mismatch between tree structure and linear split-and-lookup.

Most molecular formulas have a shallow structure and use ScriptsAtom primarily for subscripts. We can therefore flatten that part of the tree locally. Instead of recursively reading a ScriptsAtom, the chemistry reader converts it into a single normalized token.

The rule is: ScriptsAtom has three elements: base, subscript, and superscript. Convert each element to string, prefix subscripts with _, prefix superscripts with ^, and concatenate them.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
class ScriptsAtom : public Atom, public HasTrueBase {
protected:
std::shared_ptr<Atom> base;
std::shared_ptr<Atom> subscript;
std::shared_ptr<Atom> superscript;
......
}

std::wstring ScriptsAtom::toString() {
std::wstring key = L"";
if (base != nullptr) key += base->toString();
if (subscript != nullptr) key += L"_" + subscript->toString();
if (superscript != nullptr) key += L"^" + superscript->toString();
return key;
}

With this rule, the tree above is flattened into this form:

Teaching a TTS Engine to Read LaTeX in Chinese: Flattening a subtree locally (figure 5)

Recognizing chemical formulas

After flattening, the TTS traversal result for the formula above is H_2, S, O_4. How do we turn these scattered texts into a chemical formula name? Dictionary mapping is one option, but there are several problems:

  1. During traversal, how do we determine that H_2SO_4 is a group?
  2. How do we know whether H_2SO_4 has a corresponding chemical name?
  3. If it has no name, how do we fall back? After all, H_2S also has a name, “硫化氢”.

This problem is very similar to search engines. When you type some characters, search engines can show a list of associated suggestions.

Teaching a TTS Engine to Read LaTeX in Chinese: Recognizing chemical formulas (figure 6)

This is a prefix-matching problem, so a trie is a natural fit for recognizing chemical formula names.

Teaching a TTS Engine to Read LaTeX in Chinese: Recognizing chemical formulas (figure 7)

The figure shows a constructed Trie tree of chemical formula names. Every node stores a name, if any, representing the chemical formula on the path from the root to the current node. The search process uses a greedy algorithm:

  1. Input H_2 and reach the red node. Since it has a name, record “氢气”.
  2. Continue input S and reach the green node. It also has a name, so overwrite the name as “硫化氢”.
  3. Continue input O_4 and reach the blue node. It also has a name, so overwrite the name as “硫酸”.

The search ends and the final chemical formula name is “硫酸”. Core code:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
std::wstring ChemTTSEngine::getSpeechText(const std::shared_ptr<RowAtom> &atom) {
std::wstring result;
TrieNode *root = Trie::Instance().root;
for (int i = 0; i < elements.size(); ++i) {
while (root != NULL && i + step < elements.size()) {
......
std::wstring key = elements[i + step]->toString();
root = root->find(key);
step++;
if(root != NULL && root->name != NULL) {
validStep = step;
temp = *root->name;
}
}

root = Trie::Instance().root;
step = 0;
}
......
}

Building the chemical formula table

Since chemical formula recognition requires chemical names, a chemical formula name database is naturally needed.

Some data tables were built with reference to Wikipedia, and the final scale is expected to be around 3000 to 4000 entries, or even more.

  1. List of inorganic compounds
  2. List of organic compounds

Keys in the table follow several rules:

  1. Separate characters with #.
  2. Keep consecutive digits together.
  3. For ScriptsAtom, prefix subscripts with _, prefix superscripts with ^, and keep the whole part as one unit.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
std::unordered_map<std::wstring, std::wstring>& ChemTTSDatabase::getChemicalNameMap() {
static std::unordered_map<std::wstring, std::wstring> nameMap = {
// Elements
{L"H", L"氢"},
{L"H#e", L"氦"},
{L"L#i", L"锂"},
{L"B#e", L"铍"},
{L"B", L"硼"},
{L"C", L"碳"},
......

// Simple substances
{L"H_2", L"氢气"},
{L"N_2", L"氮气"},
{L"O_2", L"氧气"},
{L"O_3", L"臭氧"},
{L"F_2", L"氟气"},
{L"C#l_2", L"氯气"},
{L"B#r_2", L"溴单质"},
{L"I_2", L"碘单质"},
{L"P_4", L"白磷"},
......

// Compounds
{L"K#F" , L"氟化钾"},
{L"K#C#l" , L"氯化钾"},
{L"K#B#r" , L"溴化钾"},
{L"K#I" , L"碘化钾"},
{L"K#O#H" , L"氢氧化钾"},
{L"K#H#C#O_3" , L"碳酸氢钾"},
......
}

This manual work can be handed to ChatGPT: teach it the rules and let it help build the database.

Loading the chemical formula table

The chemical-formula trie is built by inserting each normalized token sequence from left to right. For every token, the builder follows an existing child node or creates one when necessary; the final node stores the compound name.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
struct TrieNode {
std::wstring *name;
std::unordered_map<std::wstring, TrieNode *> children;

TrieNode();
void insert(const std::wstring& s, const std::wstring& name);
TrieNode *find(const std::wstring& s);
};

struct Trie {
TrieNode *root;
Trie();
static Trie& Instance();
};


Trie::Trie() {
root = new TrieNode();
auto& map = ChemTTSDatabase::getChemicalNameMap();
for (auto& kv : map) {
root->insert(kv.first, kv.second);
}
}

void TrieNode::insert(const std::wstring& s, const std::wstring& name) {
TrieNode *cur = this;
std::vector<std::wstring> elements = StringHelper::split(s, L'#');
for (auto& e : elements) {
auto& children = cur->children;
if (children.find(e) == children.end()) {
children[e] = new TrieNode();
}
cur = children[e];
}
cur->name = new std::wstring(name);
}

Recognizing chemical equations

Once chemical formulas can be recognized, we can further recognize chemical equations. Compared with chemical formulas, chemical equations contain the following concepts:

Teaching a TTS Engine to Read LaTeX in Chinese: Recognizing chemical equations (figure 8)

The following data structure is defined, and the existence of Equation is used as the fundamental criterion for determining whether something is a chemical equation.

1
2
3
4
5
6
7
8
9
enum class ChemElementType {
Molecular, // chemical formula
Coefficient, // coefficient
StateOfMatter, // state of matter
Combine, // +
Equation, // =, ->
Condition, // heating, high temperature, catalyst, etc.
Undefined,
};

Normally, traversed elements are pushed onto a stack. If an XArrowAtom or XLongequalAtom is encountered, the current expression is considered a chemical equation. When popping the stack, chemical formulas without coefficients get a default coefficient of 1. If it is not a chemical equation, there is no need to add coefficients.

The core code is:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
for (int i = 0; i < elements.size(); ++i) {
// chemical formula recognition logic
......

// recognize chemical-equation elements
auto& e = elements[start];
if (std::dynamic_pointer_cast<SymbolAtom>(e)) {
std::shared_ptr<SymbolAtom> sa = std::static_pointer_cast<SymbolAtom>(e);
switch (sa->getUnicode()) {
case L'+' :
stack.push(std::make_shared<ChemElement>(L"和", ChemElementType::Combine));
break;
case L'-':
stack.push(std::make_shared<ChemElement>(L"失去", ChemElementType::Combine));
break;
case L'↑':
stack.push(std::make_shared<ChemElement>(L"气体", ChemElementType::StateOfMatter));
break;
case L'↓':
stack.push(std::make_shared<ChemElement>(L"沉淀", ChemElementType::StateOfMatter));
break;
default:
stack.push(std::make_shared<ChemElement>(e->getSpeechText(), ChemElementType::Undefined));
break;
}
} else if (std::dynamic_pointer_cast<XArrowAtom>(e) || std::dynamic_pointer_cast<XLongequalAtom>(e)) {
stack.push(std::make_shared<ChemElement>(e->getSpeechText(), ChemElementType::Equation));
isChemEquation = true;
} else {
std::wstring text = e->getSpeechText();
if (text.length() > 0) {
if (std::all_of(text.begin(), text.end(), ::isdigit)) {
stack.push(std::make_shared<ChemElement>(text, ChemElementType::Coefficient));
} else {
stack.push(std::make_shared<ChemElement>(text, ChemElementType::Undefined));
}
}
}
}

// coefficient insertion logic
std::shared_ptr<ChemElement> top = nullptr;
while (!stack.empty()) {
top = stack.top();
stack.pop();
if (isChemEquation && top->type == ChemElementType::Molecular && !stack.empty() && stack.top()->type != ChemElementType::Coefficient) {
if (stack.top()->name.find(L"分之") != std::wstring::npos) {
result = L"个" + top->name + result;
} else {
result = L"1个" + top->name + result;
}
} else if (isChemEquation && top->type == ChemElementType::Coefficient) {
std::wstring temp = top->name;
while (!stack.empty() && stack.top()->type == ChemElementType::Coefficient) {
temp = stack.top()->name + temp;
stack.pop();
}
result = temp + L"个" + result;
} else {
result = top->name + result;
}
}
if (isChemEquation && top->type == ChemElementType::Molecular) {
result = L"1个" + result;
}

The final TTS output for the above chemical equation is:

1
2个过氧化钠和2个水在加热的条件下反应生成4个氢氧化钠和1个氧气气体

Reading concentrations

Teaching a TTS Engine to Read LaTeX in Chinese: Reading concentrations (figure 9)

Ion concentration is read differently from a chemical equation. The equals sign in the middle simply means “equals”, so we only need to recognize concentration in the form c(xxx). Similar to absolute value, a stack push/pop method is used to determine whether the text expresses concentration. Core code:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
std::vector<std::shared_ptr<Atom>> ChemTTSEngine::getElementsByAggregatingMolaritySymbols(const std::vector<std::shared_ptr<Atom>>& elements) {
// extract c(xxx)
std::stack<std::shared_ptr<Atom>> stack;
std::vector<std::shared_ptr<Atom>> atoms;
std::vector<std::shared_ptr<Atom>> newElements;
for (int i = 0; i < elements.size(); ++i) {
auto& e = elements[i];
if (std::dynamic_pointer_cast<CharAtom>(e) != nullptr &&
std::static_pointer_cast<CharAtom>(e)->getCharacter() == L'c' &&
i + 1 < elements.size() &&
std::dynamic_pointer_cast<SymbolAtom>(elements[i+1]) != nullptr &&
std::static_pointer_cast<SymbolAtom>(elements[i+1])->getUnicode() == L'(') {
stack.push(elements[i+1]);
stack.push(elements[i]);
i += 1;
} else if(std::dynamic_pointer_cast<SymbolAtom>(e) != nullptr && std::static_pointer_cast<SymbolAtom>(e)->getUnicode() == L')') {
if (!stack.empty()) {
stack.pop();
stack.pop();

std::shared_ptr<RowAtom> ra = std::make_shared<RowAtom>(atoms);
ra->setMathMode(false);
newElements.push_back(std::make_shared<ContainerAtom>(ra->simplify(), ra->simplify()->getSpeechText() + L"的浓度"));
atoms.clear();
} else {
newElements.push_back(e);
}
} else {
if (stack.empty()) {
newElements.push_back(e);
} else {
atoms.push_back(e);
}
}
}

if (!stack.empty()) {
while (!stack.empty()) {
newElements.push_back(stack.top());
stack.pop();
}
for (auto& ele : atoms) {
newElements.push_back(ele);
}
}
return newElements;
}

The reading is:

1
次氯酸的浓度 加 次氯酸根离子的浓度 等于  碳酸氢根离子的浓度 加 碳酸的浓度

Hydrates

In chemistry, many substances crystallize from aqueous solution and combine with a certain number of water molecules. These water molecules are called waters of crystallization. Their reading is usually “x hydrate y”, where x is the number of water molecules and y is the compound name before ·.

  • CuSO4·5H2O: 五水合硫酸铜
  • Na2CO3·10H2O: 十水合碳酸钠

If all hydrates are stored in the table, there are two problems:

  1. They cannot be fully enumerated, so omissions are inevitable.
  2. There is a lot of waste.

Both problems disappear if the parser extracts the number of water molecules separately. It can look up the compound before ·, then construct the hydrate name from the coefficient and the matched compound.

The idea is: during Trie traversal, if the · symbol is encountered, check whether the following substance is number + H2O or just H2O. If so, stop the loop and combine the previously found chemical symbol name y into the final name: x hydrate y.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
for (int i = 0; i < elements.size(); ++i) {
while (root != NULL && i + step < elements.size()) {

// If we encounter ·
if (std::dynamic_pointer_cast<SymbolAtom>(elements[i+step]) && std::static_pointer_cast<SymbolAtom>(elements[i+step])->getUnicode() == L'·' && temp.length() > 0) {
j = i + step + 1;

// Determine the number after ·
while (j < elements.size() && std::dynamic_pointer_cast<CharAtom>(elements[j]) && std::isdigit( std::static_pointer_cast<CharAtom>(elements[j])->getCharacter())) {
unit += std::static_pointer_cast<CharAtom>(elements[j])->getCharacter();
j++;
}
// And the chemical name after the number
root = Trie::Instance().root;
while (root != NULL && j < elements.size()) {
std::wstring key = elements[j]->toString();
root = root->find(key);
if (root != NULL) {
j++;
if( root->name != NULL) postName = *root->name;
} else {
break;
}
}
// Break the loop
break;
}

// Normal chemical molecular-name lookup
std::wstring key = elements[i + step]->toString();
root = root->find(key);
step++;
if(root != NULL && root->name != NULL) {
validStep = step;
temp = *root->name;
}
}

......

if (validStep != 0) {
if (postName.length() > 0 && postName == L"水") {
// If the part after · is number + water, it is a hydrate and should be renamed as "x hydrate y".
i = j - 1;
temp = (unit.length() > 0 ? unit : L"一") + "水合" + temp;
postName = L"";
unit = L"";
} else {
i = start + validStep - 1;
}
if (temp.length() > 0) {
stack.push(std::make_shared<ChemElement>(temp));
}
}

......

There are also cases where this rule does not fit naturally. For example, NH3·H2O would become “一水合氨气” under the rule above, which sounds awkward. The correct name is “一水合氨”. For this kind of case, we can add a whitelist: when encountering ·, look up the full form in the table first, and only fall back to the rule if the lookup fails.

1
{L"N#H_3#·#H_2#O" , L"一水合氨"},

Summary

Converting LaTeX into natural Chinese speech is not a one-time parsing task. Every new notation, ambiguous reading, and chemistry exception adds another language rule to test. The tree-based pipeline gives those rules a stable foundation, but pronunciation quality will continue to depend on real examples and careful human review.

-------------本文结束 感谢您的阅读-------------
If this article helped you, your support encourages me to keep writing.

Title:Teaching a TTS Engine to Read LaTeX in Chinese

Author:lingyun

Published:2024-02-02 23:06

Updated:2026-10-08 00:23

Original link:https://tsuijunxi.github.io/en/2024/02/02/latex-tts-reading-in-chinese/

License: Attribution-NonCommercial-NoDerivatives 4.0 International Please keep the original link and author when sharing.