Provenance Pursuit

Have I seen this label before? Labels, stamps and marks on the backs of works in the SKKG collection

How we got here

In a day and a half, a folder of spreadsheets, PDFs and photographs became a results site, a workbook and a handful of tools: a box viewer, an entry form, a review page and a search by photo. This page retraces the steps in order, with the numbers at each one: what we tried and why, what worked, what didn't, and what we decided.

GLAMhack 2026, challenge 304 «Provenance Pursuit», SKKG Stiftung für Kunst, Kultur und Geschichte, Winterthur, 13–14 November 2026. The team built it with an AI coding assistant (how). Times are Swiss time, taken from the log of the coding sessions and from the publication history of the site. Several steps ran side by side; they are listed in the order their main result arrived. Numbers are as of Saturday 14 November, 14:00.

← Back to the resultsAbout this site: what each part does

872photographs of backs, searched on SKKG's IIIF server
733labels and marks from SKKG's records placed as boxes, 594 of them reliably
1,913further labels no record mentions, found by a detector (to check)
135times a label turned up on another work; 101 backed by SKKG's records
2,015events in 594 provenance timelines; 533 documented by a label
19publications of the site, from Friday 13:20 to Saturday 13:48

The routes in one picture

Three kinds of evidence, three routes, one result. SKKG's records describe a label in one of two ways: with a picture of it (SKKG's photos of single labels, the pictures in its label register) or only in words (the provenance data sheets). Many labels are not recorded at all.

Each kind got its own route, and every route ends in the same place: a box in pixels of the original photograph of the back. That box is all SKKG's IIIF server needs to cut the label out, so no picture was ever copied. Reliable boxes were then searched on all other backs, and their hits checked against SKKG's records of exhibitions and acquisitions.

The numbers are today's; the steps below say how each one came about.

How labels were placed on the backs Three routes. A picture of the label: DINOv2 finds the most similar windows of the 872 backs, LightGlue and MAGSAC count the points that agree; 187 reliable boxes. A description: OCR looks for the quoted text in the area the data sheet names, else a template, else the vision-language model, unverified; 498 by text, 19 by template, 896 model leads. Nothing yet: OWLv2 marks everything that looks like a label, then OCR and rules sort SKKG's own, known and new labels; 1,913 new labels, 1,407 with text. All three end in a box in pixels of the original photograph, an IIIF region. From there: labels found again on all other backs, checked with SKKG's records; and the site, workbook, viewer, entry form and review page. A picture of it 197 label photos, 75 register pictures A description 1,906 marks from 236 data sheets Nothing yet labels that no record mentions DINOv2: the most similar windows of 872 backs OCR: the quoted text, in the area the sheet names OWLv2 marks all that looks like a label LightGlue and MAGSAC count the points that agree else a template, else the model (unverified) OCR and rules: SKKG's own, known or new 187 reliable boxes 498 by text, 19 by template, 896 leads 1,913 new labels, 1,407 with text A box in pixels of the original photograph …/iiif/3/{image}/x,y,w,h/max/0/default.jpg is the label, cut out by SKKG's IIIF server Found again searched on all other backs, checked with SKKG's records Shown and checked site, workbook, viewer, entry form, review page
Each route uses the most precise method that its evidence allows; the less precise ones get only what is left. Colours as on the results site: red label photos, purple data sheets, teal label detector.

Friday 13 November Steps 1–13

  1. 1The raw material, and what it lacked

    –

    Why 304The challenge asks «Have I seen this label before?». That is an image-search question, and the team knew a method for it from earlier work on placing aerial photographs on maps: an image model proposes candidates, a geometric matcher confirms them.

    What we hadFrom the challenge: SKKG's draft label register (75 entries on 49 works), an object list (237 works), 236 provenance data sheets (PDF) and full-resolution photographs (200 backs, 3.3 GB). Openly available: SKKG's knowledge graph (SPARQL) and its IIIF image server, with about 11,700 photographs of backs and 4,421 images typed as labels.

    What it lackedNothing said where on a back a label sits: no boxes, no coordinates. The register is a data model with 75 examples rather than a training set: 51 of its pictures are only 105 px wide, Sotheby's is spelled five ways, positions come in 55 free-text variants, and its column «other SKKG works with this label» is filled in once. That column is the gap a tool should close. Of the 4,421 label images only 356 are photos of a label on the object; 4,065 are A4 scans of detached tags, mostly of weapons, locks and sculptures. The object list cuts the provenance short for 170 of 237 works («[...]»).

    DecidedWork from SKKG's IIIF server, with copies of the backs at 1,600 px and full resolution only for the regions that need it, instead of downloading the full-size files; take the provenance from the knowledge graph rather than the object list.

  2. 2Candidate lists from three sources

    TriedOne script joins the register, the object list and the knowledge graph. The SPARQL endpoint answers quick series of queries with «429 Too Many Requests», so the script pauses between them.

    Worked705 works; 2,036 IIIF images, among them 872 photographs of backs of 622 works; a test set of 197 label photos of 150 works whose back is photographed too; 93 pairs of works with labels of the same issuer; 405 provenance stations (auction houses, dealers, estates) whose label might be on a back; 382 provenance actors with their spellings.

    LearnedA provenance station does not guarantee a label. On Inv. 1098 the provenance names six dealers and auction houses; the back shows the label of one of those dealers, but none from Koller.

  3. 3An image index of the backs

    –

    TriedLabels are small, so a whole back would drown them. Every back was cut into overlapping square windows of 20 % and 35 % of its shorter side, and DINOv2 turned each window into a vector of 1,536 values: 143,963 windows, about 100 a second on the laptop's GPU. The run pauses with Enter, for when the laptop gets too loud, and resumes where it stopped.

    WorkedAgainst all 872 backs, the right back is the top hit for 37 % of the 197 test photos and among the first 20 for 59 % (by chance: 2 %).

    Didn't workThe first evaluation, at 10:21, said 63 %, but only 181 backs and 16 photos were ready then; small galleries flatter. Most misses are labels that are not visible on the back photo (inserted slips, marks under a vessel) or too small for the windows. A second set of windows at 10 % (459,377 of them) moved such backs up the list (Inv. 1745 from rank 32 to 12), not to the top, so it was computed only for 86 backs.

  4. 4A geometric check

    , full evaluation

    TriedFor the 30 best backs, SuperPoint and LightGlue match the photo against the most similar windows, and a robust homography (MAGSAC) counts the points that agree in one perspective. The same physical label gives dozens to hundreds of such inliers, a merely similar one hardly any; from 15 on, a back moves up.

    WorkedTop hit right: 37 % → 56 %; mean reciprocal rank 0.42 → 0.57. The whole test set took 39 minutes.

    Didn't workMatching image pairs in batches gave wrong counts: the same pair had 105 inliers on its own and 9 in a batch of eight. Every pair is matched on its own since.

  5. 5Boxes in original pixels

    –

    DecidedA box for every label, relative to the original IIIF image, so that no new pictures of labels have to be made and everything refers to the original. (More under Decisions.)

    HowThe homography from step 4 maps the outline of a label photo onto the back. The models worked on 1,600 px copies with a known scale factor, so every box converts exactly into pixels of the original photograph: x, y, w, h is an IIIF region. Reliable boxes are also exported as W3C Web Annotations.

    WorkedLabel photos: 135 of 197 placed reliably (median 101 agreeing points; boxes were right even at 30), 9 uncertain, 53 not found, in 4 minutes. Register: 52 of 75 reliable, 33 of them from the 105 px pictures; 6 uncertain, 14 only roughly placed, and 3 sit on the front of the frame and were not searched.

    The back of Inv. 595, with a red box around a framer's label at the upper left The whole back of Inv. 595 and the box: 1643,1625,1677,1724, in pixels of the 4,972 × 5,563 original.
    The framer's label, cut out of the original photograph by SKKG's IIIF server The same box as an IIIF region: SKKG's server cuts the label out of the original.
    SKKG's photo of this framer's label was placed on the back with 754 agreeing points. Whole back, zoomable · The work on the results site

    LearnedThe label photos and the register hardly overlap: only 3 of the register's 49 works have such a photo. That is why the site shows more labels than the register lists.

  6. 6Found again on other works

    , evidence from exhibitions and sales

    TriedEvery reliably placed label (187) was cut from its back and searched on all other backs; LightGlue confirms a hit from 25 agreeing points.

    Worked40 labels turned up on other works, in 135 hits.

    A typed label on the back of Inv. 198, cut out by SKKG's IIIF server Inv. 198: SKKG's label photo, placed on its back.
    A label typed the same way on the back of Inv. 196, cut out by SKKG's IIIF server Inv. 196: found with 88 agreeing points.
    Two labels typed the same way for two related works (only the title and size differ), matched with 88 agreeing points. Both works came to SKKG from the same seller, which backs the hit.

    Didn't workAt first only 3 of the 135 hits were explained by the provenance chains, mostly because the provenance of most works is still open.

    FixedThe exhibitions and acquisitions of the 705 works, read from the knowledge graph following the graph model of skkg-mcp: 746 loans of 211 works to 281 exhibitions, 657 acquisitions by SKKG (532 of them from institutions). Now 101 of 135 hits are backed: 87 by an exhibition both works were lent to, 18 by the same seller, 3 by the provenance.

    LearnedHits are leads, not proof. One exhibition, «Sesam öffne dich!» (Bern and Martigny, 2014/15), showed about 70 of these works, and SKKG's own loan strips recur on many backs: they document a loan, not an earlier owner.

  7. 71,906 marks that exist only as text

    –

    What we hadThe 236 data sheets (206 MB) describe the provenance marks of each work in a section «Provenienzmerkmale». Parsed into a table: 1,906 marks (804 labels, 259 stamps, 1,140 inscriptions; one mark can be several), each with its side, position («o. r. Zierrahmen» = top right of the frame) and the quoted text («№ 1143»). Ten times as many as in the register, but without pictures.

    TriedApple's text recognition, running on the laptop, read the backs of these works at 3,000 px from IIIF. The position gives a search area; the quote is looked for in what was read, digit sequences exactly and words fuzzily.

    Worked235 marks found in the area the data sheet names, 50 elsewhere on the back.

    Didn't workFor 1,295 marks only the area was known: handwriting, worn labels, short codes. 279 marks are on the front and were not searched; 47 belong to works without a photographed back.

  8. 8A vision-language model, cut back

    –; review page

    TriedQwen3-VL-4B, a vision-language model running on the laptop, for the 1,295 marks: it sees the area plus a margin and the data sheet's description, and answers with a box. In a pilot on 40 marks whose text OCR had found, 33 of its boxes contained that text (82.5 %), at 4.6 seconds a mark.

    Didn't workIt always answers with a box, even when the mark is not visible. In a spot check of 20 boxes, 9 were right, 2 plausible, 6 wrong and 3 unclear; four of the wrong ones sat on SKKG's own inventory label, the most conspicuous label on a back.

    DecidedAt 13:45 the team asked for something more precise: the model should only be used where nothing better exists. The run stopped at 481 of 1,295.

    TriedTemplate matching, as for the aerial photographs: 328 labels with a known position served as templates. In the area the data sheet names, DINOv2 picks the most similar templates and all templates of the same issuer; LightGlue and MAGSAC must agree on at least 25 points.

    Didn't workMost labels are one-offs, with no second copy to serve as a template. In a pilot on 141 marks with a known position, only 15 matched any template; 10 got a box, 6 of them on the right spot. In all, 19 of 733 labels and stamps: 5 by a template of the same issuer, 14 by a similar label; 25 times another issuer's label sat in the area and was not assigned.

    WorkedThe model then ran only on the remaining 803 marks, for about an hour: 1,107 boxes, all marked unverified; 145 marks «not visible»; 24 boxes on SKKG's own inventory labels dropped.

    The 1,295 marks that reading the whole back did not find · how they are placed, on Friday evening and on Saturday (step 16)
    • Found by their text (OCR)
    • Template match
    • Model proposal, unverified
    • Area only, no box
    Friday 15:23, after the model run
    19 1,107 169
    Saturday 13:21, after reading each area on its own and with image filters
    213 19 896 167
    As a table
    Placed byFriday 15:23Saturday 13:21
    Their text (OCR)0213
    Template match1919
    Model proposal, unverified1,107896
    Area only, no box169167
    All1,2951,295

    ReviewA review page (16:04) shows one proposal at a time: correct (Y), wrong (N) or unsure (U), and since 17:52 redraw (R) on the whole back. Decisions stay in the reviewer's browser and are downloaded as CSV for the team, which merges them into the workbook and the site. No decisions have been merged yet.

  9. 9Workbook, box viewer and the first site

    –

    TriedAt 12:26 a first workbook went into the team's shared folder, with a small preview picture of every box.

    Didn't workThe team did not want pictures in the workbook, but the whole image from IIIF with the box drawn on it (13:18).

    WorkedA box viewer: one page that loads the whole photograph from SKKG's IIIF server with deep zoom and draws the boxes given in its address. Every one of 1,931 boxes in the workbook got a «view» link, and the file shrank from 3.4 to 1.1 MB. The viewer went live at 13:20. GitHub Pages would not serve the private repository, so it was made public at 13:32; it holds only what the site shows.

    WorkedAt 15:09 the workbook's first sheet got a guide: what each sheet, row, status, colour and link means. At 15:27 the results site went live: every label as a picture cut out by SKKG's server, the works with their boxes, the labels found again with their evidence, and the workbook to download. 93 of 144 label photos got their own text, read once by OCR, as a title.

    Live13:20 box viewer · 15:27 results site and workbook

  10. 10What next: ten ideas, each with a probe

    –

    TriedA side session tested ten ideas with small probes (a few requests or a short local run each) and ranked them by what they add to «Have I seen this label before?».

    Built nextThe comparison with SIK-ISEA (step 11); a label detector (OWLv2 found 70 % of 44 known labels on 12 backs in 17 seconds; step 14); and the search by photo in the browser (a 4.6 MB index instead of 442 MB searched as well; step 12).

    Explored, not builtAn IIIF export for the Mirador viewer (SKKG's manifest server took 95 seconds per manifest; small static manifests worked), an agent on skkg-mcp, Heidelberg University Library's digitised auction catalogues (3 of 7 lots before 1946 found), the 4,065 scans of detached tags as a reference collection, a direct look-up of SIK-ISEA's control numbers, and network or map views.

    Dead endGrouping label crops into families by DINOv2 similarity: labels of the same issuer were hardly closer to each other than to others (AUC 0.63). The vectors capture «white paper on wood», not the design.

  11. 11SIK-ISEA: links and counts only

    –, counts updated Saturday

    TriedSIK-ISEA's research portal has a public interface. 347 of its records belong to SKKG; 298 were linked to SKKG objects (237 by the inventory number, 61 by artist and title), 120 of them among the works with a photographed back.

    Worked536 exhibition venues and ownership stations that SIK-ISEA records and SKKG's knowledge graph lacks, on 97 works; 51 of them match a label on the back, 28 strongly (36 before the label detector's text was added on Saturday).

    Decided(17:57) SIK-ISEA's legal notice does not allow its content to be republished without consent. So the public site and workbook link to SIK-ISEA's records and count what they add, next to our own label evidence; no exhibitions, venues, years, lots, titles or dates from the portal. The details stay in the team's internal workbook, and the raw responses never leave the internal folder. Since 17:59 there are two workbooks: the internal one and a public copy.

    Live17:59 public workbook · 18:02 What SIK-ISEA adds

  12. 12Entry form and search by photo

    –

    WorkedAt 17:28 a teammate added an entry form for columns F to L of SKKG's label register. It was connected to every box: it opens prefilled with what the data already say, marked as such; the box can be confirmed or redrawn on the whole back; entries stay in the browser and are downloaded as CSV for the team.

    WorkedSearch by photo: DINOv2 runs on the visitor's device (a 174 MB model, downloaded once) against an index of 4.6 MB (256 dimensions, one bit each); the photo never leaves the device. The top hit is right for 36 % of the test photos: the coarse search only, since the 56 % of step 4 need LightGlue on a GPU. Hits can be confirmed or redrawn too.

    Didn't workA smaller, quantised model gave unusable vectors (similarity 0.18 to the original). On WebGPU the model computes in half precision and its vectors drift (0.88–0.96), so the page computes on the processor instead: 2 to 6 seconds a photo.

    Live18:21 entry form behind every box, search by photo

  13. 13Private owners: masked, then named

    , tightened Saturday , reversed for names

    FoundBefore the publication at 18:21, a scan of the public files found label transcriptions from the data sheets with names and addresses of former private owners. They had been in the site's data since its first publication.

    DecidedMask them in everything public: on the site 187 names became «[private owner]» and 66 addresses «[address]». Sellers who are persons are not named. The internal workbook keeps the full text for the team.

    Didn't holdOn Saturday, while linking labels to timeline events, about 17 published label texts turned up in which a private owner's name had slipped through: owner fields in capitals or after a «|» («EIGENTUM VON | …»), «SAMMLUNG …», «reserviert für …», lender fields.

    Fixed(12:32) The masking then ignored case and «|» and covered inclusive forms; a name that a data sheet's description gives as owner was hidden in the label's quoted text too. The names left in label texts were those SKKG itself publishes as provenance actors; in the timelines, persons appeared only by their role. At 13:47 the team confirmed that the label detector's text stays public, masked in the same way: the search and the entry form rely on it.

    Decided(14:36) Reversed for names. On 14 November the team decided to show names from public sources: they are already published there, and for provenance research the names matter most. Addresses stay masked («[address]»). Since then owners, heirs and sellers appear by name in the timelines, and label texts show the names the labels give.

Saturday 14 November Steps 14–19

  1. 14A label detector on every back

    Friday → Saturday

    WhyEvery route so far finds only labels that some record documents.

    TriedOWLv2, an open-vocabulary object detector, looked for «a paper label», «a sticker», «a label with printed text», «a rubber stamp» and «a handwritten note on paper» on all 872 backs, in chunks of under five minutes: 9,711 detections; threshold 0.15. Each box was read by OCR. Boxes with little text were read once more from a 1,000 px IIIF region: 1,887 requests, one a second, about 49 minutes; 464 of 972 boxes without text became readable. SKKG's own inventory labels, QR labels and colour charts (1,680), makers' stamps (66) and boxes on documented labels were set aside.

    WorkedIt finds 73 % of the 428 documented labels (data sheets 92 %, register 77 %, label photos 61 %), and 1,913 labels no record mentions, 1,407 of them with readable text. On Inv. 4999 it read exhibition labels from Baden-Baden and Montréal that SKKG's record does not list.

    LimitsOf 21 checked finds with text, 20 were labels or marks (15 provenance labels); of 15 finds without text, 7 were not labels at all. So every find is marked «check», and those without text appear only with «include unverified boxes». The issuer is a guess from the text read.

    Live10:55 on the site, in the entry form and in the workbook (sheet «Detected labels»)

  2. 15English, except the data

    Friday ; Saturday –

    Decided(Friday 11:52) The team is English-speaking, so every page, the workbook, the documentation and the scripts' messages are English. What SKKG delivered stays as it is: titles, descriptions, positions, file and column names, with a glossary in the project's README.

    WorkedThe entry form's interface became English at 11:07. What it writes into SKKG's German register stays German, with English glosses in the lists. The comment it prefills in column L was English for twelve minutes and went back to German at 11:19, because it lands in the same German register row.

  3. 16Four ideas from another prototype for this challenge

    –

    TriedAnother prototype for this challenge, built on a single photograph with classical image processing, had four ideas we did not have. Each was tried on SKKG's data in its own side session; none of its code or pictures was copied.

    1 · The back, exploded, with a provenance timeline: worked

    Each work's labels lift off its back, beside a timeline from SKKG's records: provenance, exhibitions, acquisition. Clicking an event lights up the labels that document it, and the other way round. 594 works have a timeline; 533 of 2,015 events are documented by a label (252 strongly), through 805 links from 575 labels on 291 works. Live at 12:32, with persons by their role only; since the decision on names (step 13), by name.

    2 · Its mark finder as a second detector: added almost nothing

    As shipped it finds 9 % of the 428 documented labels, tuned 23 %, against 73 % for OWLv2; both together 77 %. A frame or stretcher forms one closed outline around everything inside it, so the labels within are never proposed, and on busy wood labels merge with the grain. Of its 104 new boxes, two are genuinely new marks (an inscription on Inv. 1447, a strip of worn labels on Inv. 4769); in a random sample of 30, one was. Not imported.

    Share of the 428 documented labels each detector finds
    OWLv2 label detector73 %
    The other prototype's mark finder, as shipped9 %
    The same, tuned on SKKG's labels23 %
    Both together77 %
    As a table
    DetectorLabels foundShare
    OWLv2 label detector (score ≥ 0.15)31473 %
    Other prototype's mark finder, as shipped399 %
    The same, tuned9823 %
    Both together33077 %

    3 · Its image filters before OCR: worked

    Reading each data-sheet area on its own, and again after image filters (single colour channels, flat field, local contrast), found 213 more marks by their quoted text: 155 from the area's own crop, 58 only after a filter, mostly pencil and chalk on wood. All 15 filter-only finds checked by eye sat on the quoted mark. Reliable boxes on the site rose from 427 to 594, model proposals fell from 1,107 to 896 (see the chart in step 8). Live at 13:21. The same filters became «Ways of seeing» in the entry form (12:21): computed in the browser from the IIIF region, with nothing stored.

    4 · Shared storage for the team's entries: not possible here

    Hosted pages with a built-in database may only load their own files, so SKKG's IIIF images and the site's data would be blocked; links into them cannot carry a query, and everyone who writes needs an account. The entry form and the review page keep their entries in the browser and pass them on as CSV files.

  4. 17Feedback without a GitHub account

    –

    TriedAt 11:56 a «Feedback» button on every page opened GitHub's form for a new issue, prefilled with the page, the work, the label's key and the box.

    Didn't workThat needs a GitHub account, and most of the people whose feedback matters have none. A static page cannot open an issue by itself: that takes a token, and a token cannot sit in a public page. The first relay needed a personal GitHub token pasted into its settings; a way without any token replaced it within the hour.

    Worked(12:19–12:21) The form on the page sends the message to a small relay, a Google Apps Script web app that only keeps it for a limited time (six hours at first, 48 since Saturday afternoon) and holds no secret. Every 10 minutes a GitHub Action in the site's repository fetches new messages and opens one public issue each, with the short-lived token GitHub gives every run. No personal token is stored anywhere. A test message went through end to end, and the test issue was deleted.

    How feedback becomes an issue Form on the page: no account needed, page, work and box go along. Relay, a Google Apps Script web app: keeps messages for at most 48 hours and holds no secret. GitHub Action, every 10 minutes: uses the token GitHub gives each run. Public issue in the site's repository, when the Action next runs. Form on the page no account; page, work and box go along Relay: a Google Apps Script web app keeps messages ≤ 48 hours, holds no secret GitHub Action, every 10 minutes uses the token GitHub gives each run Public issue in the site's repository when the Action next runs
    No step holds a personal token. Those with a GitHub account can still open the prefilled issue on GitHub directly.

    DecidedThe price is a wait: 10 to 20 minutes by design (the schedule, plus GitHub's own delays). The team kept the 10-minute schedule (13:53).

    Didn't work yetBy Saturday 14:50 GitHub's timer had not started the Action by itself once. Messages are safe meanwhile: they wait in the relay and become issues as soon as the Action runs, by timer or started by hand.

    Also «Report a problem» links on labels, boxes, matches and timeline events carry the same context.

    Live11:56 button · 12:19 relay and Action · 12:21 form without an account

  5. 18Four backs with swapped sizes

    –

    FoundWhile testing the other prototype's filters (step 16): SKKG's image list gives width and height swapped for four portrait backs (Inv. 71, 4965, 4991 and 5826). The IIIF server's image information and the photographs themselves are portrait.

    What the swapped size did Left: the index took the back to be 7,952 pixels wide and 5,304 high, so its scale factor was 4.97. Right: the photograph is 5,304 wide and 7,952 high, scale factor 3.315. The search windows covered only its top 44 to 47 percent, and a box scaled with 4.97 lands down and to the right of the label, partly outside the image. What the index said 7,952 × 5,304 px, so a scale factor of 4.97 The photograph really 5,304 × 7,952, factor 3.315; windows only on the top 44–47 % where a label is its box, scaled × 4.97
    The models worked on 1,600 px copies. With the swapped size, the index expected a copy 1,067 px high instead of 2,399 px, laid its windows over the top part only, and scaled boxes back by the wrong factor.

    EffectBoxes on these backs were scaled with 4.97 instead of 3.315 and lay partly outside the image: seven model boxes and 17 of the detector's (three IIIF requests had been refused with «400»). The search windows of these backs covered only their top 44–47 %.

    FixedEverywhere: the index, the data-sheet areas and model boxes (13:21); the detector's boxes, re-read over IIIF (13:40); the search windows (736 replaced by 664), their vectors and the browser's index of 143,891 windows, where the top hit is now right for 37 % of the test photos instead of 36 % (13:48).

    DecidedFix it at the source too: the script that builds the index now compares every size with the cached image (13:55). It found the four backs and 16 label photos with the same swap.

  6. 19Writing it down

    The About tab describes what the site and its tools do; this page, how they came about. Both were written by side sessions of the coding assistant, from the same sources (below).

Decisions and why

Boxes, not copies

Every label is a box in pixels of SKKG's original photograph, and so an IIIF region: …/1643,1625,1677,1724/max/0/default.jpg is the label.

Why: the originals and their rights stay with SKKG; a box takes a few bytes instead of a picture; SKKG can take the boxes over as they are, also as W3C Web Annotations. The site's pictures are all cut out by SKKG's server, and the workbook links to the viewer instead of holding pictures (apart from the register's own).

The price: everything depends on SKKG's servers, which all teams shared, so bulk requests went one at a time, a second apart.

Step 5 · step 9

SIK-ISEA: links and counts

Why: SIK-ISEA's legal notice does not allow its portal's content to be republished without consent.

So the public site and workbook link to SIK-ISEA's records and say how much they add (536 items, 51 of them matched by a label on the back). The details stay in the team's internal workbook.

Step 11 · What SIK-ISEA adds

Names shown, addresses masked

Why: the names are already published in the sources we use, and for provenance research they matter most. Addresses are not needed.

At first everything public masked both the names and the addresses of former private owners. On Saturday afternoon the team decided to show names from public sources; addresses stay masked («[address]»). SIK-ISEA's content stays out for another reason: its terms of use.

Step 13

Feedback without an account

Why: most of the people whose feedback matters have no GitHub account, and a token must not sit in a public page.

So feedback goes through a form on the page, a relay that holds no secret and a scheduled GitHub Action. The price is a wait until the scheduled Action runs.

Step 17

The most precise method first

Why: a model that always answers with a box makes leads, not facts.

Text before templates, templates before the model. The model's boxes are marked unverified and hidden by default until the team confirms them on the review page.

Step 8 · Review page

English interface, German data

Why: the team is English-speaking, and SKKG's register is German.

Pages, workbook and documentation are English; SKKG's data and the register's own terms stay as delivered, with English glosses.

Step 15

Local, pausable, gentle

The models that processed the collection, six neural networks in five roles, all ran on one laptop (Apple M2 Max): DINOv2, SuperPoint and LightGlue, Apple's text recognition, Qwen3-VL-4B and OWLv2. One GPU job at a time, every long run pausable, and few parallel requests to SKKG's servers. The coding assistant worked in the cloud, on code, results and samples.

Step 3 · step 14

What didn't work

  • A score that flattered63 % against 181 backs became 37 % against all 872. Step 3
  • Small windows aloneThey moved tiny labels up the list, not to the top. Step 3
  • Matching in batchesWrong counts of agreeing points; every pair is matched on its own. Step 4
  • The model for everythingIt always answers with a box; cut back to the marks nothing else finds. Step 8
  • Templates for one-off labels19 of 733 labels and stamps: most have no second copy. Step 8
  • Pictures in the workbookReplaced by links to the whole image with the box. Step 9
  • Label families by image similarityAUC 0.63; the vectors see paper on wood, not the design. Step 10
  • Smaller or faster models in the browserThe quantised model was unusable, WebGPU drifted. Step 12
  • Masking that missed fieldsAbout 17 label texts with owners' names slipped through; tightened, then dropped for names. Step 13
  • A second detector73 % became 77 % together, with two genuinely new marks. Step 16
  • Shared storage in hosted pagesThey cannot load SKKG's images or the site's data. Step 16
  • Feedback through GitHub's own formIt needs an account most people don't have. Step 17
  • Swapped image sizesFour backs; fixed everywhere, and guarded against. Step 18

Publication history

Each time the site changed, from the history of its public repository (on GitHub), in Swiss time.

Friday 13 November

  1. Box viewer for IIIF images
  2. Results site and workbook
  3. Review page for the model's proposals
  4. A teammate's entry form for the label register
  5. Review page: redraw a box
  6. Public workbook, SIK-ISEA as links and counts
  7. The SIK-ISEA page
  8. Entry form behind every box, search by photo, private owners masked

Saturday 14 November

  1. The label detector's 1,913 finds
  2. Entry form and start page in English
  3. Column L comment in German again
  4. Feedback button (an issue on GitHub)
  5. Feedback button on the SIK-ISEA page
  6. Feedback relay and Action
  7. Feedback without an account; «Ways of seeing» in the form
  8. Exploded back and timeline; tighter masking
  9. 213 more marks by their text; four backs' sizes fixed
  10. Detector boxes on the four backs corrected
  11. Search index over the whole of the four backs

How it was made

With an AI coding assistant. The team worked with Claude Code, Anthropic's AI coding assistant. A main session wrote most of the scripts, pages and analyses. Side sessions took on separate tasks in parallel: the ten ideas, the comparison with SIK-ISEA, the label detector, the search by photo, connecting the entry form, the English interface, the timeline, the other prototype's ideas, the About tab and this page. The team chose the challenge, set the direction (boxes on the originals, a method like the one for the aerial photographs, the model only for what is left, SIK-ISEA as links and counts, feedback without an account) and took the decisions described on this page; the sessions' commits name Claude Opus 5.5 as co-author. The spot checks on this page (the model's boxes, the detector's finds, the filters' finds) were made by the assistant looking at the IIIF crops; the team's own review of the model's boxes is still open.

Where these facts come from. This page was written by one of the side sessions, from the main session's log (the team's requests and decisions and the results, with their times), the side sessions' hand-over notes, the publication history of the site's repository, the project's documentation and exploration notes, and the data behind the site. Every number on it was checked against one of them. Where two sources disagreed, the current data won.

Data and images: SKKG Stiftung für Kunst, Kultur und Geschichte, Winterthur (SKKG Sammlung digital, knowledge graph and IIIF image server). Object metadata and photographs of backs CC0; provenance data sheets CC BY 4.0, SKKG provenance research team. All pictures on this page are cut out live by SKKG's IIIF server; none is stored here.

← Back to the results · About this site