deepnoodletracker
no project

Welcome to deepnoodletracker

deepnoodletracker teaches a computer-vision model to find and trace elongated, line-like animal body parts it calls noodles — a sea-star arm, an octopus arm, a hydra tentacle, or any similar appendage. You trace a handful of examples by hand, and the model learns to trace the rest for you, including across long videos. You can also mark single-point pose keypoints and outline whole mask regions.

The cards below are ordered the way you actually work — Config, then Data, then Annotate, then Train, with Network and Hotkeys as helpers along the way. Open Quick steps on any card for a short checklist, or In detail for the full story.

Quick start — the whole workflow in five steps
  1. Log in. Enter a username and password and click Log in, or click Create account to register. Anyone can sign up; the first account becomes the admin.
  2. Config. Open an existing project, or create a new one and give it a schema (the list of body parts you’ll label).
  3. Data. Add the images and/or videos you want to label. Videos are sampled automatically so you only label a diverse, representative handful of frames.
  4. Annotate. Claim a batch of images and trace your animals. Every image saves on its own as you go.
  5. Export & train. Click Export index in the top bar to write index.csv, then train the model from the command line.

Network and Hotkeys are optional helpers you can visit any time: Network shows a picture of the model your schema builds; Hotkeys lets you change the keyboard shortcuts.

1 · Config

Open an existing project, or create a new one and choose what it will label.

Quick steps
  1. To continue earlier work: under Load a config, click a project to open it.
  2. To start fresh: under Set a config, type a Project name, pick or build a schema, set Images per task, and click Create project.
In detail

A project bundles a schema, your data, and your annotations. The tab has two cards:

  • Load a config lists every project on the server with its schema name and progress. Click one to open it.
  • Set a config creates a new project. Type a name, then choose Use existing schema (pick one from the dropdown) or Author new schema.

When authoring a schema, give it a name and click + Add class for each body part. Every class has a colour, a name, and a kind:

  • Noodle (line) — a traceable polyline appendage (arm, tentacle). At least one noodle class is required. Tick single when only one of that class ever appears per image.
  • Pose (point) — a single keypoint, such as an eye or mouth.
  • Mask (region) — a whole-animal or region polygon. Drawn with SAM2 when it’s available on the server, otherwise by hand.

Two more settings: Embedding dim (default 8) controls how well the model can tell apart overlapping individuals of the same noodle class, and Images per task (default 20) is just how many images are grouped into each labelling batch — it doesn’t change the model. Click Create project to finish.

2 · Data

Add the pictures and videos you want to label in two steps: upload, then extract frames.

Quick steps
  1. Upload — drag files from your computer into the upload area (or choose files) and click Upload. They appear under Uploaded data as Pending.
  2. Extract frames — click Extract frames to process everything still pending. Images are added as-is; videos are sampled into a diverse set automatically.
In detail

Upload (step 1): drag and drop, or choose files. You can mix images and videos freely. Large files are sent in 25 MB chunks and stored under the project’s uploads/ folder. Uploading works the same whether the browser is on the server or a remote machine, and nothing is processed yet — each file is listed under Uploaded data as Pending. You can remove a pending file before extracting it.

Extract frames (step 2): click Extract frames to ingest every file that is still pending. Extracted files stay listed for reference so you can see what has already been processed.

Why videos get sampled: a video has thousands of nearly identical frames, so the app picks a small, varied subset to label. It samples frames, describes each one with a frozen DINOv3 model, and selects the most different-looking frames so rare moments stay represented. Under Video sampling options you can set Images to annotate (default 100), Max frames extracted (default 5000), and Stride (auto). The Inventory card shows what’s currently in the project.

3 · Annotate

Trace your animals to build the labelled examples the model learns from.

Quick steps
  1. Pick a task (a batch of images) from the dropdown on the left and click Claim.
  2. Move through images with D (previous) and F (next), or the nav bar.
  3. Choose a class tool on the right, then draw on the image.
  4. Draw noodles base → tip: the first point is the base, the last is the tip.
  5. There’s no save button — each image saves itself as you edit.
In detail

The workspace has three panels; the two side panels can be collapsed (the ◀/▶ tabs) and resized (drag their inner edge).

  • Left — images. Pick a task (batch), Claim it so others know it’s yours (your first edit also claims it automatically; the button becomes Release, and a lock means someone else owns it and it’s read-only). The ⤓ button downloads the task as a zip, the search box filters by filename, and a filled dot marks an already-annotated image.
  • Centre — canvas. Scroll to zoom, hold Space and drag to pan, and use the floating bar (First / Prev / Next / Last / Fit, or F1 to fit).
  • Right — tools. Select / edit (v) plus one tool per class in your schema, a Shapes list of everything on the current image, and a Hotkeys reference.

Drawing:

  • Noodle — click to place points base→tip, or hold Shift and drag to trace freehand. Finish with N, Enter, or a double-click; Esc cancels; right-click removes the last point.
  • Pose — click once to drop the keypoint.
  • Mask — with SAM2 on the server, click foreground (Ctrl-click background, or drag a box) and press Enter to accept the outline; otherwise draw the polygon by hand.
  • Ignore — outline regions to leave out of training (e.g. blurry or unlabelled areas).

Editing (with Select / edit): click a shape to select it, drag a point to move it, click an edge to insert a point, Ctrl-click a point to delete it, r reverses a noodle’s base↔tip, Del removes the shape, and Ctrl+Z undoes.

Network (optional)

See a picture of the model your schema builds — just for understanding.

Quick steps
  1. Open it once your schema has at least one noodle class — the diagram draws itself.
  2. Scroll to zoom, drag to pan, double-click to fit.
  3. Use Download PNG or Download SVG to save a copy.
In detail

This tab is read-only — there’s nothing to fill in. The diagram is built from your current schema and updates as you change it (it shows “Add a noodle class to preview the network” until one exists).

It walks left to right through the model: a pretrained backbone that reads the image, a neck that fuses detail with context, small heads that predict one set of maps per body part, and decoders that turn those maps back into traced lines and points. It’s purely to help you understand what training will produce.

Hotkeys (optional)

Change the keyboard shortcuts to suit you.

Quick steps
  1. Click the key button next to an action, then press the key you want (Esc cancels).
  2. Click Reset to defaults any time to restore everything.
In detail

There are two groups. Tool hotkeys activate each class tool (these default to the number keys 1–9 in schema order). Drawing functions are the editing actions — select/edit, start/finish drawing, cancel, delete, reverse base↔tip, fit view, and previous/next image.

Each key can only do one thing, so assigning a key that’s already in use moves it from the old action. Your choices are saved in this browser (per browser), so they stick between sessions on this machine.

4 · Train

When you’ve labelled enough images, request a training run right here — an administrator approves it and the server runs it for you.

Quick steps
  1. Open the Train tab, set the hyperparameters (or keep the defaults), and click Submit training request. Your request appears as Pending.
  2. An administrator reviews pending requests and approves or rejects them.
  3. Approved runs execute one at a time on the server’s GPU. Watch progress and the live log under My training runs; you can cancel your own run any time.
In detail

Submitting snapshots the project’s currently annotated images into a training index, so later annotation edits won’t change a run that’s already queued. Hyperparameters are clamped to safe ranges on the server, and runs execute as isolated processes (one at a time) so a heavy job can’t destabilize the app or collide on the GPU.

Training works with any project schema — the project’s body plan is snapshotted and drives the model’s channels automatically. You only need a small minimum of annotated images; the tab tells you if that isn’t met yet. Full training finetunes the DINOv3 backbone and can take hours. Administrators see an Approval queue of everyone’s pending requests.

Load a config

Open an existing project (schema + data + annotations).

    Set a config

    Create a new project and choose its body plan (schema).

    What a config defines (body plan & model channels)

    A config binds an organism schema — a DeepLabCut-style declarative body plan [1] — to your data and annotations. The schema is the single source of truth for what the network predicts: it is compiled into an ordered set of dense output channels that the model, target rasterizer, loss, and decoders all read from. Each field below changes that channel contract:

    1. Noodle class (traceable polyline appendage: sea-star arm, octopus arm, hydra tentacle…). Each one adds 7 dense channels that let the model trace an elongated structure end-to-end: a support mask + a centerline heatmap (Gaussian heatmap regression [2]), a unit dx/dy direction field running along the centerline (a part-affinity-field style vector map [3]), a progress scalar (0→1 arc-length from base to tip), and base/tip endpoint heatmaps. At least one noodle class is required. Tick single when a class only ever has one instance per image (e.g. a body outline): it then gets no identity embedding and the annotator locks it to a single shape.
    2. Pose class (point keypoint). Each adds a single Gaussian-heatmap channel — standard keypoint heatmap regression [1, 2].
    3. Mask class (whole-animal / region polygon). Annotation-only: it adds no dense channels and never touches the noodle/pose head. When SAM2 [5] is installed on the server, these masks are drawn interactively — click a point (or drag a box) and SAM2 returns a polygon outline you accept with Enter; otherwise you draw the polygon by hand. The resulting polygons are stored alongside the other shapes so they can later train a separate instance-segmentation model.
    4. Embedding dim. Per-pixel identity width. Every multi-instance noodle class gets its own block of this many channels (single-instance classes get none), trained with a discriminative instance loss that pulls pixels of one individual toward a shared mean and pushes different individuals’ means apart [4], so overlapping/touching individuals of a class can be separated. Giving each appendage type its own embedding space keeps identities apart through occlusion without a cross-class term. Default 8; larger gives more separating capacity at some compute cost.
    5. Images per task. Batch size for loading/annotating in the UI — a workflow knob only; it does not affect the schema or the trained model.

    [1] Mathis et al. (2018), DeepLabCut: markerless pose estimation of user-defined body parts with deep learning, Nature Neuroscience.
    [2] Newell et al. (2016), Stacked Hourglass Networks for Human Pose Estimation, ECCV (arXiv:1603.06937).
    [3] Cao et al. (2017), Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields, CVPR (arXiv:1611.08050).
    [4] De Brabandere et al. (2017), Semantic Instance Segmentation with a Discriminative Loss Function, arXiv:1708.02551.
    [5] Ravi et al. (2024), SAM 2: Segment Anything in Images and Videos, arXiv:2408.00714.

    Network diagram

    Scroll to zoom · drag to pan · double-click to fit.
    What each part of the network does (and why)

    DeepNoodleTracker uses a single fully-convolutional hydra model: a frozen vision-foundation backbone extracts features, a shared FPN neck fuses them into one high-resolution map, and several lightweight heads each predict a stack of per-pixel maps. Those maps are concatenated into the exact channel contract your schema defines (the diagram above), then decoders turn them into traced appendages and keypoints. The components, left to right:

    1. Input. An RGB image scaled to 0–1; ImageNet mean/std normalization is applied inside the model so the foundation backbone sees the distribution it was pre-trained on.
    2. DINOv3 ConvNeXt-Base backbone. A hierarchical convolutional encoder [2] pre-trained with self-supervised DINOv3 [1]. It emits four feature maps at strides 4/8/16/32 (128/256/512/1024 channels). Why: self-supervised foundation features carry strong, transferable shape and texture priors, so the model learns from tens–hundreds of labels instead of thousands. By default the backbone is fine-tuned end-to-end at a low, optionally layer-wise-decayed learning rate (~10× below the heads) so it adapts to your organism while preserving its pre-trained priors; pass --freeze-backbone to keep it frozen for very small datasets or faster/cheaper training. (The separate, always-frozen DINOv3 pass used to pick diverse video frames during ingestion is a different, inference-only use described in the Data tab.)
    3. FPN neck (256-ch, stride 4). A feature-pyramid network [3]: 1×1 lateral convolutions project every backbone stage to 256 channels, a top-down pathway upsamples and adds them coarse→fine, and a small refine block cleans the result into one shared stride-4 map. Why: deep stages know what is present but are spatially coarse, while shallow stages are sharp but semantically weak; fusing them gives every head one feature map that is both semantically rich and precise enough to localize thin appendages and small keypoints.
    4. Dense prediction heads (GroupNorm residual conv). Every head shares the same tiny design: a 1×1 projection, a residual 3×3 block, and a 1×1 output convolution. They use GroupNorm [8] rather than BatchNorm so training is stable at the very small batch sizes typical of microscopy/field video, and SiLU activations [9]. The heads are:
      • Support + Centerline — per noodle class, a support mask and a centerline heatmap (Gaussian heatmap regression [4, 5]). Why: the support says “this appendage is here,” the centerline gives the 1-D curve to trace.
      • Geometry (dx, dy, progress) — per noodle class, a unit direction field tangent to the centerline (a part-affinity-field-style vector map [6]) plus a progress scalar (0→1 arc length from base to tip). Why: orientation + arc-length ordering let the decoder walk the curve the right way and know where the base and tip are.
      • Pose keypoints (only if the schema has pose classes) — one Gaussian heatmap per keypoint [4]. Why: landmarks (eyes, mouth, madreporite…) that are points, not curves.
      • Endpoints (base, tip) — per noodle class, heatmaps for the two ends. Why: they anchor the start/finish of tracing so curves are walked base→tip consistently.
      • Identity embedding (only if a class can have multiple instances) — a per-pixel embedding vector, with a separate block per multi-instance noodle class, trained with a discriminative instance loss that pulls a single individual’s pixels together and pushes different individuals apart [7]. Why: when several arms/tentacles of the same type touch or overlap, the embedding separates which pixel belongs to which individual, holding identities apart through occlusion. Giving each appendage type its own embedding space removes the need for a cross-class term.
    5. Output (concatenated dense map). The heads are concatenated in the schema’s channel order and bilinearly upsampled to the input resolution, so the model, the target rasterizer, the loss, and the decoders all share one channel contract.
    6. Decoders (inference, not trained). Heatmap peaks give endpoints and keypoints; appendages are then traced from base to tip along the centerline using the direction field as a geodesic/shortest-path cost [10], with the identity embedding added as a tie-breaking cost so a trace stays on one individual. The result is the same polyline/point structure you annotate.

    [1] Siméoni et al. (2025), DINOv3, arXiv:2508.10104.
    [2] Liu et al. (2022), A ConvNet for the 2020s, CVPR (arXiv:2201.03545).
    [3] Lin et al. (2017), Feature Pyramid Networks for Object Detection, CVPR (arXiv:1612.03144).
    [4] Mathis et al. (2018), DeepLabCut, Nature Neuroscience.
    [5] Newell et al. (2016), Stacked Hourglass Networks for Human Pose Estimation, ECCV (arXiv:1603.06937).
    [6] Cao et al. (2017), Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields, CVPR (arXiv:1611.08050).
    [7] De Brabandere et al. (2017), Semantic Instance Segmentation with a Discriminative Loss Function, arXiv:1708.02551.
    [8] Wu & He (2018), Group Normalization, ECCV (arXiv:1803.08494).
    [9] Elfwing et al. (2018) / Ramachandran et al. (2017), SiLU/Swish activation, arXiv:1710.05941.
    [10] Dijkstra (1959), A note on two problems in connexion with graphs, Numerische Mathematik.

    Add a noodle class to preview the network.

    1. Upload data

    Add images and/or videos from your computer. Files are uploaded to the project first; nothing is processed until you extract frames in step 2. This works the same whether the browser is on the server or a remote machine.

    Drag & drop images or videos here, or .

    Large files are sent in 25 MB chunks (so they pass the Cloudflare tunnel's body limit) and stored under the project's uploads/ folder.

      Uploaded data

      Files you have uploaded to this project. Pending files are extracted in step 2; extracted files stay listed for reference.

      • Nothing uploaded yet.

      2. Extract frames

      Process every uploaded file not yet extracted. Images are added to the annotation queue as-is; videos are sampled, DINOv3-embedded, and a diverse subset of frames is selected to annotate.

      How frames are extracted for annotation

      Each path you enter is resolved recursively. Image files are queued for annotation as-is. Video files go through a four-stage diversity pipeline so you label a small, representative set instead of thousands of near-duplicate frames:

      1. Uniform frame extraction. Every video is decoded with OpenCV and sampled at a fixed stride. The stride is chosen automatically as ceil(total_frames / max_frames) so that no more than Max frames extracted (default 5000) frames are pulled across all videos combined; you can override it with the Stride option.
      2. DINOv3 embedding. Each sampled frame is resized to 224×224 and passed through a frozen, pre-finetune DINOv3 ConvNeXt-Base backbone [1, 2]. The frame’s descriptor is the global-average-pooled feature map of the deepest stage (a 1024-d appearance vector).
      3. Cosine normalization. Embeddings are L2-normalized so distances reflect appearance similarity under cosine geometry, which is the standard regime for self-supervised visual features [1].
      4. Diverse subset selection. A target of Images to annotate (default 100) frames is selected by k-means++ / D² seeding [3]: an initial frame is drawn, then each subsequent frame is sampled with probability proportional to its squared distance to the nearest already-chosen frame. This randomized form of farthest-point sampling [4] spreads picks across the whole appearance manifold so rare states stay represented, rather than oversampling common frames the way density-based clustering would.

      The selected frames are re-decoded at full resolution, saved as JPEGs, and added to the annotation queue. Selection is deterministic for a fixed seed.

      [1] Siméoni et al. (2025), DINOv3, arXiv:2508.10104.
      [2] Liu et al. (2022), A ConvNet for the 2020s, CVPR (arXiv:2201.03545).
      [3] Arthur & Vassilvitskii (2007), k-means++: The Advantages of Careful Seeding, ACM-SIAM SODA.
      [4] Eldar et al. (1997), The Farthest Point Strategy for Progressive Image Sampling, IEEE Trans. Image Processing.

      Video sampling options

      Inventory

      No project open.

      Images + annotations for the open project, as a single zip streamed straight to your computer.

      – / –
      Open a project and add data to start annotating.

      Tool hotkeys

      Keys to activate each class tool. Click a key, then press the desired key (Esc to cancel capture). Keys are unique and saved in this browser.

        Drawing functions

        Keys for the drawing/editing actions.

          Overall leaderboard

          Active annotation time across all projects. Time counts only while you are interacting on the Annotate tab.

            Request a training run

            Configure a training run for the open project and submit it to the queue. An administrator reviews and approves pending runs; approved runs execute one at a time on the server's GPU.

            How training works

            When you submit, the server snapshots the project's currently annotated images into a training index (so later edits won't change this run) and adds a Pending request to a shared queue. An admin then approves or rejects it. Approved runs are executed one at a time as isolated processes on the server so a heavy run can't destabilize the web app or collide on the GPU.

            Full training finetunes the DINOv3 backbone and can take hours. You can watch progress and the live log below, and cancel your own run at any time. Hyperparameters are clamped to safe ranges on the server.

            Hyperparameters

            My training runs

            No project open.