Engineering Overview
A pnpm monorepo for AI-powered semantic code search, built as a pipeline rather than a demo. Source files are chunked at function/class boundaries by walking the TypeScript compiler API — fixed-size chunks tore function bodies in half and produced embeddings that did not cluster usefully — with a character-based fallback for everything else. BullMQ workers on Redis drive chunking and embedding: the embedding pass is idempotent (only PENDING/FAILED chunks are picked up, and a batch is flagged IN_PROGRESS up front so two workers cannot race) and retries transient Gemini errors with exponential backoff while failing fast on 4xx. Vectors land in a vector(768) column on Postgres via pgvector, and search is hybrid — cosine similarity, full-text and a reranker — behind a NestJS 11 API with JWT + GitHub OAuth and a global 60 req/min throttler. The Next.js 16 front end calls that API through a typed client rather than a mock dataset. What is NOT done: there is no hosted deployment, and no integration test against a real Postgres — the end-to-end pipeline test runs in memory against a mocked embedder.
The Problem
grep and file search break down on large codebases — you need to know the exact term. New engineers and AI tools struggle to navigate unfamiliar code when the vocabulary is unknown. A semantic layer over the codebase could turn 'where is auth handled?' into a ranked list of file + line locations.
The Solution
A monorepo whose ingestion pipeline chunks source at function/class boundaries rather than at fixed line counts, embeds each chunk, and stores the vectors in pgvector. Queries are embedded at request time and matched by cosine similarity, then combined with full-text results and reranked, so a query that shares no vocabulary with the code can still land on the right file and line range.
Key Engineering Challenges
Two decisions carry the project. Chunking is syntactic, not size-based: fixed-size chunks split function bodies mid-logic and produced embeddings that did not cluster, so the chunker walks the TypeScript compiler API and emits one chunk per top-level function, method or class with its real line range preserved. And the embedding pass is written to be re-run: it claims only PENDING or FAILED chunks and flags the batch IN_PROGRESS before any network call, so a crash mid-batch costs a retry rather than duplicate vectors, and two workers cannot embed the same chunk. What is still open is the honest part — nothing is deployed, and the end-to-end test proves the chain in memory against a mocked embedder, not against a real Postgres.
Core Capabilities
System Architecture
System Architecture
Ingestion
Embedding
Retrieval
Frontend & infrastructure
"Discovery and ingestion feed BullMQ queues; the chunker segments each file at syntactic boundaries; the embedding worker fills a pgvector column idempotently; the search API combines cosine similarity with full-text and reranks. Everything runs on `docker compose up` — and only there, because nothing is deployed."