From Substring to NDCG

1.1 The definition

Start with a small experiment from the first lecture: type the same query, say coffee near me, into two different search services and compare the top three results. They won't match. Both engines read the same web, yet they disagree on what's "best". That disagreement is what this course is about.

Each of the three bold words carries weight:

• Unstructured. A sentence has no columns. "Budget guesthouses in Phnom Penh offer clean rooms" doesn't have a price field you can sort by.
• Information need. The thing in the person's head. It is not the same as the words they type.
• Large collections. Millions or billions of documents, so anything that reads every one per query is too slow.

Search is everywhere: the web, email, your phone, online shops, code search, legal and medical papers, company files. Even chatbots that "read your documents" begin with a retrieval step. That pattern is called RAG (retrieval-augmented generation), and we'll build up to it in Sessions 9–10.

1.2 Core vocabulary

The lecture gives one chain of words you should be able to say in your sleep:

1.3 Relevance

Relevance sounds obvious, but it shifts. It depends on the user, the task, the context and even the time (yesterday's bus timetable is useless today).

• Topical relevance means "it's about the right subject". Usefulness means "it actually helps me". A Wikipedia page about Siem Reap is topical; a page listing tonight's buses is useful.
• Judgments can be binary (relevant or not) or graded (how relevant, e.g. 0–3). We'll need graded ones in Chapter 4.
• Judgments are made by people. A computer can't decide relevance for us; that's the whole problem.

1.4 Why is it hard?

The biggest single enemy has a name: vocabulary mismatch. People use different words for the same thing, and the same word for different things.

1.5 IR vs database queries

You already know SQL, so it's tempting to think SELECT … WHERE text LIKE '%cat%' is a search engine. Here's the lecture's comparison:

IR shows up in web search, recommender systems, question answering and RAG.

1.6 Anatomy of a search system

Every search engine splits its work into two halves:

A third piece closes the loop: feedback from logs, clicks and evaluation tells you what to improve. The course follows this pipeline: Session 2 builds the index, Session 3 the ranking, Session 4 the evaluation, Sessions 5–6 data collection, Session 8 links, Sessions 9–10 neural search and RAG.

1.7 A first look at precision and recall

How do we know a search engine works? The lecture's example: the collection holds 20 relevant documents. Your system returns 10 results, and 6 of them are relevant.

Precision is about the quality of what you returned. Recall is about what you missed.

1.8 As demonstrated in Lab 1: three naive searches

Lab 1 builds the simplest search engines possible and measures where they break. There are three versions.

for id, text in docs.items():
    if query.lower() in text.lower():
        ids.append(id)

Query: 'cat' -> 4 result(s)
d06 … enterprise applications …    d16 Her cat sleeps all day …
d17 … catalog … category …      d18 … concatenate two strings …

Try all three yourself. Press Run and watch the scan read every single document, one after another, to answer one query. Matches are highlighted so you can see why each document came back. Documents tagged rel are the ones Lab 1 judged relevant to the information need.

What the evaluation table showed

Lab 1 Part 5 judged six information needs by hand and computed precision and recall for every method. Three rows tell the whole story:

   1,000 docs: scan   0.2 ms      100,000 docs: scan  14.3 ms
  10,000 docs: scan   1.7 ms      500,000 docs: scan  74.9 ms

2.1 Today's pipeline

The second lecture is one assembly line. Each arrow is one part of Lab 2:

One rule matters more than any single step: the query goes through exactly the same pipeline as the documents. If documents are stemmed and the query isn't, "running" will never meet "run".

2.2 Tokenization

Tokenization splits raw text into tokens: usually words, usually lowercased.

• "Cheap hotel deals" → ["cheap", "hotel", "deals"]
• Khmer needs a real segmenter: "និស្សិតសាលាស៊ីអេឌីធី" → ['និស្សិត', 'សាលា', 'ស៊ី', 'អេឌីធី']. Tools: NLTK for English; khmer-nltk or khmer-segmenter for Khmer.

Tokenization looks mechanical, but it forces decisions:

2.3 Stop words

Stop words are very frequent words that carry little meaning on their own: the, is, of, and, to. Removing them shrinks the index and speeds up ranking. In Lab 2 Part 2, removing a 45-word stop list cut the collection from 405 tokens to 284, about 30% smaller.

The lecture walks one sentence through it:

For Khmer there's a community stop-word dictionary (KSWv2) in the lecture notes.

2.4 Stemming and lemmatization

Both are forms of normalization: making different forms of a word look the same so they match.

The stemming trade-off

• Over-stemming: unrelated words collapse together. university and universe both become univers.
• Under-stemming: related words stay apart. ran and running never meet, because "ran" has no suffix to cut.

from nltk.stem import PorterStemmer
porter = PorterStemmer()
porter.stem("running")   # 'run'

   running -> run         farms -> farm        categories -> categori
       ran -> ran       farming -> farm          category -> categori
   runners -> runner    farmers -> farmer            cats -> cat

2.5 Try the pipeline

Type any sentence or load a document, then press Run. The pipeline goes one stage at a time: it checks every token against the stop list (stop words are highlighted, and the matching word lights up in the list below), then shows which suffix rule the stemmer cuts from each word.

2.6 The inverted index

Here's the idea that makes search engines fast. Instead of storing "document → words", flip it around and store "word → documents".

Building it is simple and paid once: for every document, preprocess it, add the document id to the postings list of each unique term, then sort every list.

• A common term (e.g. "language") has a long list.
• A rare term (e.g. "pagerank") has a short, informative list. If a document contains it, that tells you a lot. Hold that thought; it becomes IDF in Chapter 3.

def build_index(docs):
    index = {}
    for doc_id, text in docs.items():
        for term in set(preprocess(text)):      # set(): one entry per doc
            index.setdefault(term, []).append(doc_id)
    for term in index:
        index[term].sort()                      # sorted lists are the key
    return index

Vocabulary size: 229 terms (from 405 raw tokens)
'python' -> ['d04', 'd05', 'd18']     'car' -> []
'cat'    -> ['d16']                   'automobile' (stem 'automobil') -> ['d15']

2.7 Boolean retrieval and the merge algorithm

Boolean retrieval combines postings lists with AND, OR and NOT into one answer set:

• A AND B: documents containing both terms (intersection).
• A OR B: documents containing either (union).
• NOT A: every document without A (complement).

Because the lists are sorted, we can intersect two of them in a single left-to-right pass, like zipping two sorted decks of cards. Keep one finger on each list. If the ids match, keep it and move both fingers. Otherwise move the finger on the smaller id. That costs \(O(|p_1| + |p_2|)\), and it never touches document text.

def intersect(p1, p2):
    answer, i, j = [], 0, 0
    while i < len(p1) and j < len(p2):
        if p1[i] == p2[j]:  answer.append(p1[i]); i += 1; j += 1
        elif p1[i] < p2[j]: i += 1
        else:               j += 1
    return answer

2.8 As demonstrated in Lab 2: is the index worth it?

An index costs time to build. Lab 2 Part 7 measured whether that pays off at 500,000 documents for python AND language:

When does building once beat scanning every time? Set the totals equal for \(q\) queries:

After about eleven hundred queries the index has paid for itself. After that, every query is roughly 8× cheaper. A real engine answers millions of queries a day, so it's an easy win. A collection you search only once, or one that changes constantly, might not be worth indexing.

Lab 2 also re-ran Lab 1's six needs through the index. Lowercasing fixed N2 (Python with a capital P now matches), tokenization fixed N5 (cat), and budget accommodation OR found d01 and d30. Two needs are still broken:

• N4 car: recall 0. This is a vocabulary problem.
• N3 travel Phnom Penh Siem Reap: AND gives nothing, OR gives 4 unordered docs with precision 0.25. This is a ranking problem, and Chapter 3 fixes it.

3.1 Why ranking matters

Recall where Lab 2 left us. Boolean OR for travel Phnom Penh Siem Reap returned d01, d02, d12, d29, and only d12 is really about that trip. An unordered set can't tell you which one to read first.

• Scale: real collections return thousands or millions of Boolean matches.
• Behaviour: people rarely look past the first few results.
• So the core IR problem is: order documents by estimated relevance to the information need.

Ranked retrieval is simple to describe: score every document, sort by score, return the top k. No more hard yes/no matching; every document gets a real-valued score. The rest of this chapter is about how to compute that score.

3.2 TF, DF and IDF

Term frequency

\(tf(t,d)\) is how many times term \(t\) appears in document \(d\). Intuition: if doc A says "python" once and doc B says it 8 times, B is probably more about Python.

The problem with raw TF

Some words are everywhere: stop words, or generic words like "offer" and "system". A high count of a word that's in every document isn't special. We need to scale each term by how rare it is across the collection.

Document frequency and inverse document frequency

\(df(t)\) is the number of documents that contain \(t\) at least once (not the total count). Then:

IDF is "log of N over df". Rare term, small df, big IDF. Common term, big df, small IDF. A term that's in every document gets \(\log 1 = 0\): it can't help you tell documents apart.

A high TF-IDF weight needs both: the term is frequent in this document and rare across the collection.

def idf(term):
    d = df(term)
    return 0.0 if d == 0 else math.log(N / d)     # natural log, N = 30

3.3 Documents as vectors, compared by cosine

In the vector space model, every unique term in the collection is one dimension. A document's vector holds its TF-IDF weight in each dimension (0 if the term is absent). A query is treated as a very short document and gets a vector the same way.

To compare a query with a document, measure the angle between their vectors, not their lengths:

Dot product on top, the two lengths underneath. 1.0 means same direction (very similar). 0.0 means they share no terms.

'travel Phnom Penh Siem Reap' ->
   d12: 0.440  Buses run every hour from Phnom Penh to Siem Reap…
   d02: 0.280  Cheap hotel deals in Siem Reap…
   d29: 0.226  The National Museum in Phnom Penh…
   d01: 0.198  Budget guesthouses in Phnom Penh…

3.4 BM25: saturation and length normalization

TF-IDF grows linearly with term frequency: 10 mentions count 10× as much as 1. In practice, repetition matters less and less after the first few mentions. Otherwise a spam page that repeats a keyword 100 times would win. BM25 fixes two things.

Fix 1: term saturation (parameter \(k_1\))

The score still grows with tf, but levels off. With length ignored, a term contributes

\(k_1\) (typically 1.2–2.0) controls how fast it saturates. Higher \(k_1\) means slower saturation, so repetition keeps counting for longer.

Fix 2: length normalization (parameter \(b\))

A match in a short document is stronger evidence than the same match in a long one. BM25 compares the document's length \(|d|\) to the collection average \(avgdl\). \(b = 0\) turns this off; \(b = 1\) normalizes fully; the default is 0.75.

The full formula

For each query term: a rarity weight, times a saturating tf. The bracket under \(k_1\) grows for long documents (dragging the score down) and shrinks for short ones. The "+1" inside the log keeps the IDF positive even for a term in every document.

p1: raw tf =  1  |  BM25 saturation = 1.00
p2: raw tf =  4  |  BM25 saturation = 1.82
p3: raw tf = 10  |  BM25 saturation = 2.17     (ceiling k1+1 = 2.5)

score += idf_bm25(term) * (f * (k1 + 1)) / (f + k1 * (1 - b + b * doc_len[doc_id] / AVGDL))

3.5 Try both rankers

This runs the exact Lab 3 pipeline in your browser: the same tokenizer, stop list, suffix stemmer, TF-IDF cosine and BM25. With the defaults (\(k_1=1.5, b=0.75\)) you'll get the notebook's numbers.

3.6 The "python" disagreement

Here's the puzzle Lab 3 Part 5 ends on. For the one-word query python, the two models disagree on who's first:

All three mention "python" exactly once, so tf can't separate them. Each model falls back on something else:

• BM25 uses length. Shorter than average (9.5) means a stronger match, so d18 (7) > d05 (8) > d04 (9). Set \(b = 0\) and all three tie at 2.181. Try it with the sliders above.
• Cosine divides by the document's vector length, and that comes from the IDF weights of its other words. d04 contains fairly common words (data, science, language), so its vector is shorter and "python" takes up a bigger share of it.

3.7 Zipf's law and the limits

Zipf's law: in any language, a few terms are extremely frequent and most are rare. Rank the words by frequency, and the frequency falls roughly as 1/rank. That long tail of rare words is why IDF works at all: most terms are rare enough to be informative. Very short documents and queries, like ours, make length normalization (\(b\)) especially important.

4.1 Why we need evaluation

Lab 3 left a disagreement: TF-IDF ranked d04 first for "python", BM25 ranked d18 first. We compared two lists by eye, which isn't evidence. Every design choice (stemming rules, \(k_1\), \(b\), stop list) needs a way to compare before and after. Without measurement, tuning is guessing.

4.2 Test collections and relevance judgments

A test collection has three parts:

1. Documents: our 30.
2. Queries: the six needs N1–N6.
3. Relevance judgments (qrels): which docs satisfy each need, decided by a human.

For "python": d04 relevant (a useful Python introduction), d18 relevant (a brief code snippet), d05 not relevant (a snake).

Binary vs graded

Binary says yes or no. Graded says how relevant, for example on a 0–3 scale. Graded judgments let you reward putting the best document first, not just any relevant one.

Where judgments come from: in small projects, your own team or a few volunteers. In large campaigns like TREC, trained annotators follow detailed guidelines. Either way, judgments are data, and data has noise.

4.3 Precision, recall and F1

TP = relevant and returned. FP = returned but not relevant. FN = relevant but missed. F1 is the harmonic mean: it punishes a system that's great at only one of the two.

Why harmonic and not a plain average? A system that returns all 30 documents has recall 1.0 and precision maybe 0.07. The plain average is 0.53, which looks okay. F1 is 0.13, which is honest.

4.4 Cutoffs: P@k and R@k

Users look at the top of the list, so we evaluate only the first \(k\) results. \(k\) stands for "how far a user is likely to look".

Lecture example: ranking d04 ✓, d18 ✓, d15 ✗, with relevant set {d04, d18}.

At \(k=3\): \(F_1@3 = 2\cdot\frac{0.67\times1.00}{0.67+1.00} = 0.80\). Changing \(k\) changes the score, so which \(k\) matches real behaviour? And a single cutoff hides where the relevant documents sit. P@1 rewards a system that finds just one good result; P@10 counts results many people never see.

def precision_at_k(ranked_ids, relevant, k):
    hits = sum(1 for d in ranked_ids[:k] if d in relevant)
    return hits / k                # divide by k, even if fewer than k results

4.5 Average Precision and MAP

Average Precision avoids picking a single \(k\). Walk down the ranking. Every time you hit a relevant document, write down the precision at that rank. Then average those numbers over all relevant documents (missed ones count as 0).

\(rel(k)\) is 1 if the document at rank \(k\) is relevant, 0 otherwise. \(R\) is the total number of relevant documents. MAP is the mean AP over all queries: one number for a whole system.

Lecture example: ranks d04 ✓, d18 ✗, d05 ✓ with 2 relevant documents.

Relevant documents at early ranks push AP up. Lab 4 makes this concrete. For "python" (relevant = {d04, d18}):

• TF-IDF: d04 ✓, d18 ✓, d05 → AP = (1/1 + 2/2)/2 = 1.000
• BM25: d18 ✓, d05 ✗, d04 ✓ → AP = (1/1 + 2/3)/2 = 0.833

4.6 Graded relevance and NDCG

NDCG (Normalized Discounted Cumulative Gain) combines three ideas:

1. Graded gain: a highly relevant doc earns more credit, \(2^{r}-1\).
2. Rank discount: the same doc earns less lower down, divided by \(\log_2(i+1)\).
3. Normalize against the ideal ranking, so the score sits between 0 and 1.

Step 3: normalize. The ideal order is d04 (3), d18 (1), d05 (0): \(IDCG@3 = \frac71 + \frac1{1.58} + 0 = 7.63\). So \(NDCG@3 = 5.43 / 7.63 = 0.71\).

A puts the best document first. Only the graded metric notices.

4.7 Metrics bench

Reorder the ranking and change grades to see every metric update. The working is printed underneath, so you can check your hand calculations against it. "Relevant" for P, R and AP means grade ≥ 1.

4.8 As demonstrated in Lab 4: TF-IDF or BM25?

Lab 4 runs both rankers over N1–N6 with Lab 4's judgments (binary for AP, graded 0–3 for NDCG). These are the results once the TODO functions are filled in (see the explained notebook):

• Both metrics prefer TF-IDF, and only N2 decides it. The other five queries tie exactly.
• The gap is bigger under NDCG (0.045 vs 0.028), because NDCG also punishes BM25 for putting the grade-2 d18 above the grade-3 d04. AP only sees "a non-relevant doc at rank 2".
• N4 is 0 for both. Evaluation can't fix vocabulary mismatch, but it makes the failure visible as a number.

When metrics disagree: sweeping b

Lab 4 Part 5 varies BM25's \(b\) on "python":

AP doesn't move: either way, one relevant doc sits at rank 1 and the other at rank 3. NDCG jumps when d04 (grade 3) reaches the top. Be careful with the \(b=0\) row, though. That win comes from a tie broken by document id, not from BM25 knowing anything. Ties are a real trap in evaluation.

4.9 Choosing a metric, and trusting it

5.1 Where the documents come from

For four chapters our 30 documents were simply handed to us. A real search engine doesn't get that luxury. Before it can tokenize, index or rank anything, someone has to collect the documents first. Lesson 1's anatomy diagram had a spider crawling the web on the offline side. This chapter is that spider.

The lecture separates four jobs that people often lump together as "scraping":

5.2 The pipeline, and what a URL is made of

Each arrow is one part of Lab 5. Everything starts with a URL, so it pays to know its parts:

Links inside a page are usually relative: page-3.html, not the whole address. A relative link means "relative to the page I'm on", so you can't just glue it onto the site's root. Try it:

5.3 HTTP: a request and a response

The scraper is the client. It sends a request: a method (almost always GET, "give me this page"), a URL and some headers, such as its User-Agent name. The server sends back a response: a status code, headers (like Content-Type) and the body, which for a web page is HTML. POST, PUT and the rest exist, but scrapers rarely need them.

The status code is the first thing to read, because it tells the crawler what to do next:

HEADERS = {"User-Agent": "CADT-IR-Lab5/1.0 (educational sandbox crawler)"}
response = requests.get(BASE, headers=HEADERS, timeout=20)   # always set a timeout

Status code:  200
Content-Type: text/html
Final URL:    https://books.toscrape.com/
Size:         51294 bytes

5.4 HTML, the DOM and CSS selectors

HTML is text, but a browser (or Beautiful Soup) turns it into a tree called the DOM (Document Object Model). Every tag is a node, and tags inside it are its children. The lecture's example:

<html>
  <head>…</head>
  <body>
    <article class="product">
      <h3>
        <a title="My book" href="detail.html">my book</a>
      </h3>
      <p class="price">$12.50</p>
    </article>
  </body>
</html>

html
├── head
└── body
    └── article.product
        ├── h3
        │   └── a  (title, href)
        └── p.price  "$12.50"

Tags, classes and ids are the hooks a scraper grabs onto. To find them on a real site, open it and use View page source, or the Elements panel in the browser's developer tools. To grab them in code, use CSS selectors: the same syntax a stylesheet uses to style a page can select from it.

5.5 Beautiful Soup by hand

The lecture's live code is the whole idea in two lines: requests downloads the page, and Beautiful Soup parses it so you can select from it.

soup = BeautifulSoup(listing_response.content, "html.parser")
cards = soup.select("article.product_pod")              # 20 cards

def parse_card(card, page_url):
    link = card.select_one("h3 a")
    title = link.get("title") or link.get_text(strip=True) # the text is cut short: "A Light in the ..."
    url = urljoin(page_url, link.get("href"))             # relative -> absolute
    price = re.search(r"\d+(?:\.\d+)?", card.select_one("p.price_color").get_text())
    classes = card.select_one("p.star-rating").get("class", [])   # ["star-rating", "Three"]
    rating = next((RATING_MAP[c] for c in classes if c in RATING_MAP), None)
    ...

The £ that turned into £

HTTP Content-Type: text/html                 (no charset!)
requests encoding: ISO-8859-1
apparent encoding: utf-8
Price from bytes / Beautiful Soup: £51.77
Price from requests.text:          £51.77

The server didn't say which character set it used, so requests guessed Latin-1. The page is really UTF-8, where £ is two bytes. Read as Latin-1, those two bytes become two characters: £. That garbage is called mojibake. Passing the raw bytes (response.content) lets Beautiful Soup find the page's own <meta charset="UTF-8"> and decode it correctly.

Following the "next" link

current, visited = start_url, set()
while current and current not in visited and len(visited) < max_pages:   # stop rules
    visited.add(current)
    page = BeautifulSoup(polite_get(current).content, "html.parser")
    ...                                                  # parse_card each card, skip duplicate URLs
    nxt = page.select_one("li.next a")
    current = urljoin(current, nxt["href"]) if nxt else None

Listing pages visited: 2 | listing records: 40
Saved 3 Beautiful Soup detail records to books_bs_sample.json

5.6 What static scraping can't see

The quick test: View page source. If the text you want isn't in the source, the page builds it with JavaScript. The better fix is usually to find the API the JavaScript itself calls, and use it if you're permitted. Browser tools such as Selenium and Playwright can run the JavaScript for you, but they're beyond this lesson.

5.7 robots.txt: instructions for crawlers

A site publishes rules for bots at one fixed address, the root: https://example.com/robots.txt. The rules say which user agents (bots) may visit which paths.

User-agent: *            # these rules are for every bot
Disallow: /private/      # don't crawl anything under /private/
Allow: /public/          # /public/ is fine
Crawl-delay: 2           # optional, non-standard: wait 2 s between requests

Two things robots.txt is not: it isn't a login (nothing enforces it, a bot can ignore it), and it isn't a content licence (being allowed to crawl a page says nothing about what you may do with the text). Crawl-delay isn't part of the standard, so support varies. For a big real example, look at en.wikipedia.org/robots.txt.

rp = robotparser.RobotFileParser()
r = requests.get(urljoin(BASE, "robots.txt"), headers=HEADERS, timeout=20)
if r.status_code == 200:
    rp.parse(r.text.splitlines())
can = rp.can_fetch(HEADERS["User-Agent"], url)      # True / False for this bot and URL
delay = max(0.5, rp.crawl_delay(HEADERS["User-Agent"]) or 0)

robots.txt HTTP status: 404
No robots rules published at this path; still review terms and crawl politely.
Request delay in this notebook: 0.5 seconds

5.8 Politeness, and why the real web is hard

Obeying robots.txt is the minimum. A polite crawler also:

1. Limits its scope: 2 listing pages, not 50.
2. Waits between requests and keeps concurrency low: one request at a time.
3. Says who it is with a descriptive User-Agent.
4. Honours 429 and stops, instead of evading the limit.
5. Prefers an official API or already-published data when there is one.

def polite_get(url):
    if not url.startswith(BASE) or not is_allowed(url):        # scope + robots
        raise PermissionError(f"Out-of-scope or disallowed URL: {url}")
    if _last_request[0] is not None:                           # wait before *each* next fetch
        time.sleep(max(0.0, effective_delay - (time.monotonic() - _last_request[0])))
    r = requests.get(url, headers=HEADERS, timeout=20)
    ...

Even a polite crawler meets a messy web. Each problem on the lecture's list has a counterpart in Lab 5:

5.9 Scrapy: a real crawler project

Beautiful Soup only parses, and requests only downloads. Everything else in Part A (the queue, the visited set, the delay, robots) was code we wrote ourselves, and each piece was a chance to get it wrong. Scrapy is a framework that does those jobs for you: a request scheduler, asynchronous downloading, robots handling, retries, CSS selectors and exporters.

scrapy startproject bookcrawler
cd bookcrawler
scrapy genspider books books.toscrape.com
scrapy crawl books -a max_pages=2 -O books_scrapy.json -O books_scrapy.csv

# bookcrawler/settings.py
USER_AGENT = "CADT-IR-Lab5/1.0 (educational sandbox crawler)"
ROBOTSTXT_OBEY = True
CONCURRENT_REQUESTS_PER_DOMAIN = 1
DOWNLOAD_DELAY = 0.5
DOWNLOAD_TIMEOUT = 20
FEED_EXPORT_ENCODING = "utf-8"

def parse(self, response):                    # a listing page
    self.pages_seen += 1
    for href in response.css("article.product_pod h3 a::attr(href)").getall():
        yield response.follow(href, callback=self.parse_book)
    next_href = response.css("li.next a::attr(href)").get()
    if next_href and self.pages_seen < self.max_pages:
        yield response.follow(next_href, callback=self.parse)

def parse_book(self, response):               # a book's page: one IR-ready record
    yield {"doc_id": hashlib.sha1(response.url.encode("utf-8")).hexdigest()[:12],
           "title": title, "text": " ".join(p for p in (title, description) if p), ...}

'downloader/request_count': 43,              # 1 robots.txt + 2 listing pages + 40 books
'downloader/response_status_count/200': 42,
'downloader/response_status_count/404': 1,   # robots.txt
'elapsed_time_seconds': 25.4,
'item_scraped_count': 40,

Rows: 40 | unique doc_id: 40 | unique url: 40
Empty title: 0   Empty description: 0   Empty text: 0
Shared URLs with Beautiful Soup: 3 of 3      (same title, same doc_id, same description)
Removed the '...more' teaser from 39 of 40 texts
Wrote 40 documents to ir_books_corpus.jsonl

A page, an item, a document

Why put the title into text? It's the only field the engine reads, the title holds the most important words, and people often search by title. And a book with no description would otherwise be unfindable. An empty description isn't automatically a crawler bug, either: some books just don't have one. A real failure looks different, like every row empty (a wrong selector).

5.10 Permission, copyright and privacy

Three different things, often confused: robots.txt is a technical request about which paths to crawl. The terms of service are the legal side, and can ban scraping even with no robots file. Privacy is about the people in the data: names, emails and photos have their own rules, whatever robots and the terms say. Books to Scrape is fine because it was built for practice and its data is fake. That doesn't carry over to real sites.