Back to seenX
In development
Learning

Learning

The build journal, experiments, concepts, and notes. One small step at a time.

Learning in progressBuild notes updated
01 / RIGHT NOWExperiment not started

One question to begin with.

“Can I find an image just by describing what I remember?”

We’ll test that in Python with my own images, before connecting a model to the app. First ten images, then a bigger collection once the pipeline works.

NEXT BUILD STEP · AFTER REVIEW

Load and inspect ten images.

The isolated Python environment is configured. Choose a practice folder, then load images and inspect thumbnails before introducing a model.

Our paceExplain Make one change Inspect Review & approve
02 / THE BUILD JOURNAL

What we decided. And why.

A record of our discussions so far, with plans kept separate from completed work.

The idea

Find the image I remember seeing.

seenX is a local-first Mac app for finding screenshots and saved images through visible text, visual descriptions, and available metadata. Selected folders stay under the user’s control.

Keep in mindText in a screenshot, what an image looks like, and where it was taken are different kinds of evidence.

Architecture decision

Give each part one clear responsibility.

SwiftUI is the intended native interface and owns macOS permissions and platform APIs. Rust is the reusable catalog and retrieval engine. This Next.js website explains the project. Python is for experiments, not a required app runtime.

Keep in mindThe website’s sample search is an interaction preview. It does not run an embedding model or search personal files.

Implemented baseline

Try semantic search in Python first.

Python now has file readers, OCR, cached extraction, and a pinned CLIP baseline. One command supports text, visual, and hybrid search. Visual candidates must pass a configurable similarity cutoff. Rust implementation follows measured results; a local interface comes after useful retrieval.

Keep in mindA useful experiment answers a specific question. It does not need the desktop app, a database, or a Rust bridge.

Evaluation decision

Use my own images, starting with ten.

The ignored practice-images folder contains our own images and a PDF. There is no need to generate sample images. The initial collection checks the pipeline; a larger labeled collection is still needed to measure retrieval quality.

Keep in mindWrite descriptions and acceptable matches before inspecting search results. Include difficult screenshot queries that may need OCR.

Working agreement

Understand one step before adding the next.

The full plan felt like too much at once. The agreed pace is to explain the next idea, make one small change, inspect it together, and pause for review and approval before continuing.

Keep in mindUnderstanding the change is part of finishing the step. The Python README explains the current commands, caching, logging, and accuracy limits; the walkthrough follows each implementation step.

03 / THE SMALL STEPS

The whole path. Only when I need it.

Open one step at a time. These are planned milestones, not things we’ve already built.

01Set up the Python projectA separate place for the experiment and its dependencies.

The Python project lives in python/, with source in src/seenx and separate tests. Run uv sync --project python --all-extras from the repository root. uv creates the isolated environment and installs the versions recorded in uv.lock. No manual activation is needed.

Done whenImplemented: Python 3.13.7, package entry points, and a locked uv environment. The Python README has the commands and folder map.

02Load ten of my imagesSee the inputs before asking a model to interpret them.

seenx-inspect reads supported images and PDFs from the ignored practice-images folder. It prints dimensions, PDF page counts, and text availability without creating thumbnails. Unreadable files are reported and skipped.

Done whenImplemented and checked with eight images and a two-page PDF. Original files are only read.

03Load one pretrained modelUse a model that already knows how to relate images and text.

The baseline is openai/clip-vit-base-patch32 with its matching image processor and text tokenizer at a pinned revision. Assets live in ignored models/. The current baseline runs on CPU and was checked offline after download. This is inference, not training.

Done whenThe model loads locally and we know which revision and device we are using.

04Create the image embeddingsRepresent each image as a list of numbers.

The image encoder produces a normalized vector for each image. Optional SQLite caching saves vectors between runs, checking file timestamps, size, and reader identity before reuse. Cached extraction does not change the model weights.

Done whenEach readable image has a normalized embedding connected to the correct file path.

05Search with a descriptionCompare the query vector with every image vector.

The matching text encoder embeds the description. Cosine similarity ranks images, then a configurable cutoff rejects weaker suggestions. --top-k is a maximum rather than a requirement to fill five slots. The command prints paths and scores; a score is not confidence.

Done whenImplemented: text, visual, and hybrid modes with source-method explanations. Cached image vectors can be reused while each query is encoded again.

06Measure the resultsCheck usefulness across queries, not just one good example.

Expand to 50–200 mixed images. Label 10–15 descriptions and acceptable matches before running them. Record the percentage of queries with an acceptable result in the top five, indexing time, and warm query latency. Include objects, colors, scenery, screenshot appearance, and exact-text cases.

Done whenWe have recorded successes, failures, and timings on the same labeled collection.

07Rerun offline and write the findingsLeave evidence that another run can reproduce.

After the initial downloads, rerun with networking disabled. Record hardware, device, dependency versions, the exact model revision, collection size, successful queries, and failures. Keep personal images, model assets, and notebook outputs out of commits.

Done whenA short report explains what worked, what did not, and which retrieval step to improve next before porting proven behavior to Rust.

Later: OCR, metadata filters, databases, Rust integration, and the native app. They each get their own learning steps.

04 / THINGS TO UNDERSTAND

A little less jargon.

The basic idea: turn images and a description into numbers we can compare.

  1. My imagesImage encoder → vectors
  2. My descriptionText encoder → vector
  3. Compare & rankThe five closest matches
Ontology

Explicit concepts and relationships, such as Receipt, Trip, and associated-with. Facts populate this model; evidence establishes whether an association is justified.

Model-assisted learning

A pretrained model proposes facts and people correct them. Stored knowledge changes without retraining model weights. Keep source evidence and confirmation status.

Python-to-Rust parity

Check the production implementation against the prototype using the same inputs, preprocessing, model revision, and labeled queries.

Virtual environment

A separate set of Python packages for one project, so its dependencies do not interfere with other projects.

Notebook

A document with explanations and runnable code cells. Run a small piece, inspect its output, then continue. Restarting and running all cells checks that it works in order.

Embedding

A vector: a list of numbers representing information extracted from an image or text. It is not a written caption. Compatible encoders let us compare the two.

Preprocessing

Preparing inputs in the format the model expects, such as resizing and normalizing image pixels or turning query text into tokens.

Inference

Running an already trained model on a new input. Training changes model weights; this experiment only uses the existing weights.

Cosine similarity

A comparison of vector directions. Once both vectors have length one, their dot product gives this score. It helps rank matches; it is not a probability.

Exact search

Compare the query with every image embedding. It is a straightforward baseline for this small collection, with no vector database needed.

OCR vs. semantic search

OCR extracts visible words. Semantic search compares learned image and text representations. An image can match a description without containing those words; exact screenshot text may still need OCR.

Metadata

Recorded details such as a date, filename, or GPS location. Missing metadata stays unknown. A model’s visual guess is different from a recorded fact.

Top-five hit rate

The percentage of labeled queries with at least one acceptable image in the first five results. Expected matches are chosen before inspecting the ranking.

The CLIP baseline we’ll try
05 / MY OWN NOTES

Leave a thought for future me.

What clicked? What is still confusing? What would I change?

Saved only in this browser on this site address. Export a copy to keep your notes if you change browsers, change the site address, or clear browser data.

Use Save note to keep this draft. Unsaved text is lost if you leave or reload the page.

Loading your notes…

Small changes. Real understanding.Back to the project docs