A LoRA fine-tune trained on all 100 plates of Kunstformen der Natur (1899-1904): exploring what a small, tight visual domain teaches a generative model that a broad one can't, and whether it can dream up specimens Haeckel never drew, in a style he'd recognize as his own.
Kunstformen der Natur is public domain, ~100 plates, and every plate follows the same recipe: one or more biological specimens, often radially symmetric, arranged in a multi-panel grid and rendered as a lithograph with fine line work and saturated ink, on either a pale cream or a stark black background, split roughly evenly across the set (52 black, 48 pale, measured directly from the scans, not eyeballed). That signature is strong enough that an output either looks like Haeckel or obviously doesn't, which makes a model's progress genuinely inspectable rather than a vibes-based "looks cool to me" call.
Haeckel was also drawing things (deep-sea radiolaria, siphonophores) that nobody had seen in this kind of detail before. A model that interpolates between plates should produce specimens that don't exist but feel like they could: a version of what Haeckel was doing a century earlier, only now the "field notes" are weights instead of microscope slides.
300dpi scans of all 100 plates plus cover from the Internet Archive / BioLib.de, verified complete with no missing or corrupt files, then checked for near-duplicates via perceptual hashing. Zero found, expected for 100 distinct illustrations rather than a scraped photo corpus.
Every plate's own printed genus, order, and German name transcribed directly off the plate, more accurate than the roughly 51% Wikimedia Commons categorizes (one of which is outright wrong), plus 1,102 individual specimens documented from Haeckel's own companion explanatory volume. A genuinely bug-prone pipeline, OCR page-boundary errors and two caught LLM-fabrication incidents, with an honest incident writeup kept in the repo rather than quietly fixed and forgotten.
A rank-16 LoRA on SD1.5's UNet cross-attention layers (to_q/to_k/to_v/to_out.0), designed around a real 6GB VRAM ceiling: bf16 autocast, gradient checkpointing, batch size 1 with gradient accumulation. Started as a 50-epoch probe, kept running to 124 epochs and 3,100 steps; loss dropped for the first ~50 epochs, then plateaued.
A fixed, seeded prompt set scored two ways: faithfulness (real caption vs. generated, side by side against the exact training plate) and novel recombinations, so results stay comparable across checkpoints instead of a one-off cherry-picked sample.
The full eval surfaced a real bug: captions never mentioned background color, so the model guessed. Fixed by measuring each plate's actual background from its pixels (median luminance of a central crop) and tagging every caption. The fix worked on some plates and not others; a follow-up run built specifically to close the gap moved nothing net, reported as an accepted, understood limitation rather than a target for more compute.
Haeckel drew far beyond radiolaria and jellyfish: bats, hummingbirds, orchids, antelope. The dataset's breadth is part of what makes the training run interesting: 100 images spread across roughly 90 distinct taxonomic orders is a hard generalization problem for a single small adapter.
Training loss (per-epoch mean) ran from 0.188 to roughly 0.15, flat from about epoch 50 onward across all 124 epochs. Diffusion MSE loss is inherently noisy per timestep; the reported curve is the honest per-epoch mean, not smoothed to look cleaner than it is. It drops for the first third of the run, then plateaus for the remaining two-thirds: more epochs on the same 100-image dataset stopped buying anything further.
Correct black background, correct multi-panel grid of geometric diatom shells, close enough to pass at a glance. Style, layout, and the background-color fix all land together here.
The background-color fix worked, pale background, correctly, but the jellyfish itself rendered as an abstract, cage-like structure rather than anything jellyfish-shaped. Background color and genus-specific anatomy turned out to be separate problems, solved on separate timelines.
Caption says pale background, output is teal, and two separate rounds of fine-tuning (including one that specifically doubled pale-example exposure) didn't move it. Aquatic subjects appear to have "teal" baked in stronger than the literal instruction.
Prompts that don't exist anywhere in the training set: a genus invented within a known order, or two orders blended together. There's no ground truth to compare against here, so the question is plausibility, not accuracy. Six such generations were kept: a discomedusae "aurelia nova," a siphonophorae described as a newly discovered deep-sea species, a larger colonial radiolaria, a hexacoralla × radiolaria hybrid, an ophiodea × siphonophorae hybrid, and a calcispongiae "undiscovered species" (the six specimens shown in the image above).
Each part maps to a real, current generative-ML skill, not just a pretty output gallery.
Trained a rank-16 LoRA adapter on SD 1.5's cross-attention layers under an actual 6GB VRAM ceiling (bf16 autocast, gradient checkpointing, gradient accumulation), not a scaled-up cloud run with no resource pressure.
Traced a background-color bug to its real source (captions never mentioned it) by measuring pixel luminance directly, then reported honestly when a targeted follow-up run built to fix it further moved nothing net.
Built a fixed, seeded eval script and published a genuine miss (a boxfish that stays teal through two rounds of fine-tuning) alongside a genuine win, instead of only showing best-case outputs.
Documented 1,102 individual specimens from Haeckel's own companion volume, catching and disclosing two LLM-fabrication incidents along the way rather than shipping them silently.
See the full training run, evaluation, and generated specimens:
↗ Open the Haeckel LoRA Write-Up