Skip to content
Tết −50%Đoàn Viên, only 99,500 VND first yearĐoàn Viên, first year only 99,500 VND, 1 personal reading a yearSee the offer

Blog

Build an AI assistant that cites classical books

Three passages from the Complete Book of Tử Vi Đẩu Số, each numbered

An AI assistant that answers with citations to classical books needs three things: a store of passages labelled with book and chapter, retrieval that finds the right passages for a question, and an instruction to answer only from numbered passages. Nhat Nguyet's readings set already provides the first: every passage in store/passages carries the book title, author, chapter, lookup key (passage_key: star, palace, pattern) and the original text (classical_text), and each chart points to it by id in sections[].passages.

The datasets are on Hugging Face: nhatnguyet/ziwei-readings (Tử Vi), nhatnguyet/bazi-readings and nhatnguyet/numerology-readings. Folder names, field names and key codes are English.

Steps

  1. Load the book passage store store/passages/ziwei.jsonl.zst. The store has no repeats; each passage has one id.
  2. Filter by key: a question about Tham Lang in the Life palace only looks at passages keyed star=tan-lang and palace=life.
  3. Rank what remains by closeness to the question. The example counts shared character pairs; in production, use vectors.
  4. Number the passages and put them in the prompt with the instruction: use only numbered passages and mark each point [n].
  5. Check the answer before showing it: citation numbers must exist and quotes must appear verbatim in the passage.

Retrieval and prompt code

The files are zstd compressed (extension .jsonl.zst), so run pip install zstandard first, download the passage store (for example hf download nhatnguyet/ziwei-readings store/passages/ziwei.jsonl.zst --repo-type dataset --local-dir .), then run the code in that folder.

import io, json
from collections import Counter
import zstandard  # pip install zstandard

def read(path):
    """Read a .jsonl.zst file, one JSON object per line."""
    with open(path, "rb") as f:
        for line in io.TextIOWrapper(zstandard.ZstdDecompressor().stream_reader(f), encoding="utf-8"):
            yield json.loads(line)

# Step 1: load the book passage store. One id per passage, no repeats; one file for the whole set.
passages = list(read("store/passages/ziwei.jsonl.zst"))
print("Distinct book passages:", len(passages))

# Steps 2 and 3: filter by key (star, palace), rank by character pairs shared with the question
def key(p):
    return dict(f.split("=", 1) for f in p["passage_key"].split(";"))

def pairs(s):
    return Counter(s[i:i + 2] for i in range(len(s) - 1))

def search(question, star, palace=None, k=3):
    candidates = [p for p in passages if key(p).get("star") == star and (palace is None or key(p).get("palace") == palace)]
    q = pairs(question)
    found, seen = [], set()  # one line of a book can carry several keys: take it once
    for p in sorted(candidates, key=lambda p: -sum((q & pairs(p["classical_text"])).values())):
        if p["classical_text"][:40] not in seen:
            seen.add(p["classical_text"][:40])
            found.append(p)
    return found[:k]

# Step 4: build a prompt with numbered sources
found = search("贪狼在命宫遇化权", star="tan-lang", palace="life")
block = "\n".join(f"[{i}] {p['book_title']}, {p['chapter']}: {p['classical_text'].splitlines()[0][:48]}" for i, p in enumerate(found, 1))
print(block)
print("---")
print("Answer only from the numbered passages above and mark each point [n]. If they do not say it, say the books do not say it.")
Distinct book passages: 1015
[1] 紫微斗数全书 (Tử Vi Đẩu Số Toàn Thư), 卷二 · 命宫 · 贪狼 · 贪狼入男命吉凶诀: 四墓宫中福气浓,提兵指日立边功,火星拱会诚为贵,名震诸夷定有封。
[2] 紫微斗数全书 (Tử Vi Đẩu Số Toàn Thư), 卷二 · 命宫 · 贪狼: 贪狼属木,化桃花杀。贪狼入庙长耸肥胖,陷宫形小声高而量大,性格不常,心多计较,作事急速不耐静,作
[3] 紫微斗数全书 (Tử Vi Đẩu Số Toàn Thư), 卷二 · 命宫 · 贪狼 · 贪狼入女命吉凶诀: 四墓宫中多吉利,更逢左右方为贵,禄财丰富旺夫君,性格刚强多志气。
---
Answer only from the numbered passages above and mark each point [n]. If they do not say it, say the books do not say it.

The Tử Vi book store holds 1,015 distinct passages: 1,004 from the common edition of the Complete Book of Tử Vi Đẩu Số and 11 from the Ming Nanyang Tang woodblock edition. The 8,760 charts of 1990 alone cite 984 of them. Tradition attributes the book to Chen Tuan, with a preface by Luo Hongxian dated 1550; modern scholarship does not treat that attribution as proven, and the author_vi and author_en fields say exactly that.

Three passages found, each with a citation number and chapter
Three passages found, each with a citation number and chapter

Check the answer before showing it

Models sometimes cite numbers that do not exist or "quote" lines the book never says. The function below catches both before the text reaches the reader.

import re

def check_answer(answer, passages):
    """Every [n] must point to a real passage; a quoted line must appear verbatim in that passage."""
    errors = []
    for n in map(int, re.findall(r"\[(\d+)\]", answer)):
        if not 1 <= n <= len(passages):
            errors.append(f"[{n}] is not in the passage list")
    for quote, n in re.findall(r"「(.+?)」\[(\d+)\]", answer):
        n = int(n)
        if 1 <= n <= len(passages) and quote not in passages[n - 1]["classical_text"]:
            errors.append(f"[{n}] does not contain 「{quote}」")
    return errors

passages = [{"classical_text": "贪狼属木,化桃花杀。贪狼入庙长耸肥胖"}]
print(check_answer("The book describes Tham Lang as 「贪狼属木」[1].", passages))
print(check_answer("The book says 「贪狼主富贵」[1], see also [4].", passages))
[]
['[4] is not in the passage list', '[1] does not contain 「贪狼主富贵」']

If an answer fails, drop that sentence or ask the model again; do not show it.

What to keep

  • List book and chapter at the end of the answer and keep only [n] in the text.
  • The original text is Chinese. If the model translates it, label it as the assistant's translation, not the book's words.
  • Separate what the book says from interpretation: what a book says is a sourced fact; whether its rules hold for a person's life has not been tested, so do not let the assistant state it as certain.
  • To answer about one specific chart, take the ids in that chart's sections[].passages in charts/ as the candidate set instead of the whole store.
  • The readings (store/readings) and further reading (store/further-reading, with each article's summary, faq and url) can go into the store too; keep the id to link back to the chart section.

Licence

The data is CC BY 4.0 and the code MIT. An assistant built on this set should show "Data: Nhat Nguyet (nhatnguyet.org)" where users can see it. The code repository and test cases are on the open source page.

Classical textsThe books and classical texts Nhat Nguyet draws on.Try it

More from the blog

All blog posts