LocateAnything-3B on Jetson Orin Nano
Running NVIDIA's LocateAnything-3B open-vocabulary detection VLM live on-device on an 8 GB Jetson Orin Nano — ~0.3–0.6 inferences/sec at 640 px, video at ~25 fps via optical-flow tracking. Full deep dive on model architecture, board limits, and every fix with its root cause.
TABLE OF CONTENTS
Milestone project — running NVIDIA's LocateAnything-3B open-vocabulary detection VLM live, on-device, on an 8 GB Jetson Orin Nano.
Status: working ✅ · ~0.3–0.6 inferences/sec at 640 px, video renders at ~25 fps.
What this is
A vision-language model that does open-vocabulary object detection and grounding — type what you want (
person, cup, laptop, or a phrase like the laptop screen) and it draws boxes on a live camera feed. Because the model does the detection directly, the vocabulary is open at runtime: change the prompt and the next frame uses it. No retraining. No recompiling.Quick reference
ㅤ | ㅤ |
Model | LocateAnything-3B — MoonViT vision encoder + Qwen2-3B decoder |
Board | Jetson Orin Nano (Super) 8 GB — JetPack 6, CUDA 12.6, sm_87 |
Throughput | ~0.3–0.6 inferences/sec @ 640 px; video ~25 fps via optical-flow tracking |
Code | local repo locateanything-jetson/ (VLM-only, GitHub-ready) |
The "understand it well enough to answer any question" companion to the setup guide. Covers (1) the model architecture, (2) what the Orin Nano can and can't do, (3) the performance work, and (4) every workaround and why it was needed.
TL;DR — the one-paragraph answer
LocateAnything-3B is a ~3-billion-parameter vision–language model that does open-vocabulary detection and grounding: give it an image and a text query, it replies with
<ref>label</ref><box>...</box> tokens. It's built from a MoonViT vision encoder feeding a Qwen2-3B text decoder that emits boxes using Parallel Box Decoding (a multi-token-prediction trick so it doesn't decode boxes one token at a time). Getting it onto an 8 GB Orin Nano meant 4-bit quantization (the bf16 weights are ~7.7 GB and don't fit), swapping the default MagiAttention kernel (Hopper-only) for sdpa / flash-attention-2, and a stack of Jetson-specific memory and packaging fixes. It runs at ~0.3–0.6 inferences/sec while the video renders at ~25 fps, because an optical-flow tracker carries the boxes forward between the slow inferences.Part 1 — The model architecture
1.1 What problem it solves
Traditional detectors (YOLO, etc.) have a fixed class list baked in at training/compile time. LocateAnything is a VLM detector: the "classes" are just text in the prompt, so the vocabulary is open — you can ask for
phillips screwdriver, the person in the red shirt, or the laptop screen (a relational phrase a small detector cannot express) and it localises it. That open-at-runtime property is the entire reason to run the VLM directly instead of compiling a detector.Three task modes (see
build_prompt() in la_common.py):Mode | Prompt template | Use |
detect | "Locate all the instances that matches the following description: {cat1}‹/c›{cat2}…" | comma-separated categories |
ground | "Locate all the instances that match the following description: {phrase}." | one natural-language phrase |
point | "Point to {query}." | returns a point instead of a box |
1.2 The two-tower structure
flowchart LR img["Image"] --> vit["MoonViT vision encoder<br>(2x2 patch merge)"] prompt["Text prompt"] --> dec vit --> dec["Qwen2-3B decoder<br>(Parallel Box Decoding)"] dec --> out["text + box tokens"]
MoonViT (the vision tower). A native-resolution Vision Transformer (the same family used in Kimi-VL / Moonshot's work). "Native resolution" means it doesn't force every image to a fixed square — it tiles the image into patches and processes them at their real aspect ratio, which matters for detection precision. Its output patches are 2x2-merged before going to the LLM, so the visual tokens the decoder sees = patch count / 4. On this device the vision tower runs in fp16 (it is not quantized) and costs ~190 ms at 448 px, ~310 ms at 640 px.
Qwen2-3B (the text decoder). A standard Qwen2 causal LM, but with a custom head and decoding scheme for boxes. This is the part we quantize to 4-bit. It is where ~68% of the runtime goes.
Coordinate system. The model emits box coordinates as integers in [0, 1000] (normalised), regardless of image size.
parse_boxes() scales them back to pixels: x_px = x_int / 1000 * width. The regex it parses:(?:<ref>(.*?)</ref>)?<box><(\d+)><(\d+)><(\d+)><(\d+)></box>
1.3 Parallel Box Decoding (why it isn't as slow as a normal 3B chat model)
A naive VLM would emit a box as a sequence of tokens, one autoregressive step each. This model uses Parallel Box Decoding — a Multi-Token-Prediction (MTP) style scheme where a block of tokens (a whole box) is proposed per step, using a block attention mask rather than a strict causal one. Two things I observed directly:
- Latency comes in quanta: generating 4 tokens and 8 tokens cost the same ~1358 ms; steps jump in ~675 ms increments. That's the block being emitted per step.
- The mode is selected with
generation_mode="fast"ininfer(). There's a slower exact mode too, but "fast" (MTP) is what makes real-time-ish operation plausible.
This is also why the attention backend choice is load-bearing (Part 4.2): the block mask needs a kernel that supports it.
1.4 The generation call, annotated
From
la_common.py::infer() — the non-obvious arguments:response = model.generate( pixel_values=inputs["pixel_values"].to(torch.bfloat16), # vision tower is bf16/fp16 input_ids=..., attention_mask=..., image_grid_hws=..., # native-res grid metadata tokenizer=tokenizer, # the model's custom sampler needs the tokenizer itself generation_mode="fast", # Parallel Box Decoding (MTP), not plain autoregression temperature=0.0, do_sample=False, # deterministic detection (guarded: >0 => sampling) repetition_penalty=1.1, # nudges it away from the box-repetition failure mode max_new_tokens=1024, )
image_grid_hws carries the native-resolution tiling shape — the decoder needs it to map visual tokens back to image geometry. temperature=0 is safe because the custom sampler guards it (if temperature > 0 else greedy).Part 2 — The Jetson Orin Nano, and what it can/can't do
2.1 The device
ㅤ | ㅤ |
Board | Jetson Orin Nano (Super) 8 GB |
Software | JetPack 6 / L4T R36.4.3, CUDA 12.6, Python 3.10.12 |
GPU | Ampere, compute capability sm_87, 1024 CUDA cores, up to 1020 MHz (MAXN_SUPER) |
Memory | 7.4 GB RAM unified (CPU+GPU share it) + ~3.7 GB zram swap |
Storage | 937 GB NVMe |
The single most important fact: memory is unified and there is ~6.5 GB usable after the OS. Every decision below is downstream of that.
2.2 Hard limits (things the hardware simply cannot do)
- No FP8. Needs Hopper/Ada. So no FP8 weight or KV-cache tricks.
- No DLA. The Orin Nano has no Deep Learning Accelerator (the AGX/NX Orins do). Can't offload anything to it.
- No MagiAttention. The model's default attention kernel is Hopper/Blackwell-only.
- No INA3221 power rails exposed on this board → can't read wattage in telemetry.
- ~6.5 GB memory ceiling. The bf16 model (7.7 GB) does not fit. Full stop. This forces quantization and dominates everything.
2.3 Soft limits (possible but slow / fragile)
- 4-bit LLM decode is bandwidth-bound at ~60% efficiency. ~21 ms/token is the memory roofline (1.5 GB of 4-bit weights read per step / ~70 GB/s effective); measured ~54 ms/token. The gap is dequantization overhead.
- Model load is non-deterministic (~2 in 3 succeed) even at identical free memory.
- Hardware NVJPEG decode can't run with the VLM resident — no memory for its ring buffers. CPU decode instead (fine at <1 fps).
- ~3–5 fps is the honest ceiling for this 3B VLM here. 30 fps is impossible.
2.4 The memory budget, concretely
7.99 GB total RAM -1.1 GB desktop (gdm) <- stopped before running -~1 GB CUDA context + cuBLAS/kernels --------- ~6.4 GB free needed at load time -2.2 GB 4-bit weights (nf4) - rest KV cache + activations (grows with visual tokens & output length)
If page cache (
buff/cache) is sitting at ~3 GB from an earlier model read, the ~6.4 GB isn't actually available and the load dies — hence the mandatory drop_caches (Part 4.1).Part 3 — Performance: where the time goes
All numbers measured on-device at the 640 px, 4096-patch, MAXN_SUPER operating point.
3.1 The latency breakdown (the map for all optimization)
Component | Cost | Share of a ~3.1 s / 3-object query |
CPU preprocessing | 14 ms | negligible — never optimize this |
Vision (MoonViT) | ~310 ms | ~10% |
Prefill | ~660 ms | ~21% |
Decode (per-token × tokens) | ~54 ms/token | ~68% |
So: fixed cost ≈ 1.0 s (vision + prefill), then ~54 ms per generated token. A 3-box query ≈ 1.0 s fixed + ~1.6 s decode ≈ 3.1 s. (In MAXN_SUPER a simpler scene lands closer to ~1.7 s.)
The lesson: decode is the prize (68%), and output length is the dominant cost driver — measured slope ~0.33 s per extra box. Fewer categories per query, tighter prompts = directly faster. Vision is only 10%, so accelerating it barely moves the total.
3.2 What I tried, and what the numbers said
Several "obvious" wins were wrong, proven by measurement:
Idea | Expectation | Reality |
Cut visual tokens ( max_patches 4096→1024) | prefill scales with tokens → faster | 5× SLOWER. Starved of detail the model degenerates into a repetition loop, emitting ~42 junk boxes until it hits the token cap. 3.11 s → 15.95 s. |
fp16 compute dtype for the 4-bit kernels | fp16 is the classic Ampere fast path | 1.8× slower than bf16 on sm_87 (95 ms vs 54 ms/token). The opposite of usual advice. Current bf16 config is already optimal. |
TensorRT the MoonViT vision tower | it's a whole ViT, must be worth it | Vision is only ~10% end-to-end. Even an optimistic 2.5× export saves ~6%. Not worth the ONNX/plugin effort. |
Smaller input size (640→384 px) | fewer pixels → faster | Only 1.13× (3.09 s → 2.73 s), and object count gets noisy. Input size is not the lever. |
expandable_segments:True | saves ~1.1 GB (it does, for small models) | Makes the 4-bit VLM unloadable. See Part 4.1. |
Where the real speed is (unclaimed): better W4 kernels (AWQ/GPTQ instead of bitsandbytes) could give ~1.5× on decode → ~27% end-to-end; CUDA graphs attack the ~40% of per-token time that isn't bandwidth. Both need aarch64 + sm_87 kernels and an architecture mapping for this custom model — real work, not done yet.
3.3 The tracking trick — decoupling video fps from inference fps
The single biggest perceived performance win isn't in the model at all. The naive loop encoded one frame per inference, so the picture froze for seconds. The current app splits it:
- A render_loop thread publishes annotated JPEG frames at
--render-fps(default 25), drawing whatever boxes the model last produced. - The inference loop runs independently at ~0.3–0.6 fps and calls
set_result(). box_tracker.FlowTrackercarries the boxes forward between inferences using sparse Lucas–Kanade optical flow + a per-object MOSSE correlation filter, so a box actually follows its object during the ~2 s until the next inference instead of sitting frozen.
Result: video at ~25 fps with detections at ~0.2/s. The boxes lag the picture by up to one inference, and the overlay honestly reports
boxes are X.Xs old.Why MOSSE (from
bench_trackers.py, 3 boxes on this board): MOSSE 12.5 ms vs CSRT 88.9 ms vs KCF 508 ms. Only MOSSE (spread across idle CPU cores) fits a 25 fps budget. Optical flow follows the scene; a detection box always contains background that drags it, so the correlation filter locks onto the object's own appearance instead.Part 4 — Every workaround, and why
Each of these cost real debugging time; most are Jetson-specific and invert an intuition.
4.1 Memory & loading (the hardest category)
🚨drop_cachesbefore loading — not optional. Loading needs ~6.4 GB free. After any earlier model read, Linux page cache holds ~3 GB and the kernel won't reclaim it fast enough for the CUDA/NvMap allocation. The load then dies withNVML_SUCCESS == r INTERNAL ASSERT FAILED— which is not a torch bug; it's PyTorch's OOM message-building path calling NVML, which Tegra doesn't support, masking the real "out of memory". Whenever you see that NVML assert anywhere: you are out of memory.
Every successful load followed
sudo sh -c 'sync; echo 3 > /proc/sys/vm/drop_caches'; every failure did not. start_vlm_prompt.sh does this + systemctl stop gdm (~1 GB) every launch.Load is non-deterministic — retry as a fresh process. Measured 3 clean attempts at identical 6.53 GB free: OK, OK, FAILED. A failed attempt leaks ~2.6 GB that
gc.collect() + empty_cache() cannot reclaim, so in-process retries start with less memory each time — worse than useless. Fix: the launcher loops up to 3 fresh process launches (~96% success). Note in la_common.load_model(): the cleanup runs outside the except block on purpose — while the except is live, the traceback still references the half-built model's frames, pinning its GPU memory.expandable_segments:True makes it unloadable. PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True saves ~1.1 GB and helped a small model earlier — but proven back-to-back at the same free memory: with it the 4-bit load FAILS, without it it loads in ~9.8 s. Tegra's unified memory doesn't get along with the allocator's virtual-memory segments for allocations this large. Removed everywhere; there's an explicit "do NOT set this" comment at the top of live_camera_prompt.py. If loads start failing again, grep for it first.Pre-quantize once (
prequantize.py). Quantizing at load holds a 4.7 GB bf16 shard while the quantized weights accumulate → a peak > 5.5 GB that only fits on an otherwise-idle box. Instead quantize once and save_pretrained the ~2.2 GB nf4 checkpoint to ~/locate-anything/la3b-4bit, then load that. Load: 10 s vs 24 s, far lower peak. la_common prefers the local checkpoint and falls back to quantize-from-hub if it's absent.Never pass
dtype= when loading the pre-quantized checkpoint. It's already nf4 and carries its compute dtype in quantization_config. Passing dtype= makes transformers run a conversion pass that materialises bf16 copies of every weight (~7.7 GB) → instant OOM. Only device_map={"": 0}.Never
.to(device) a quantized model. bitsandbytes places weights during from_pretrained via device_map. Calling .to() raises under 4-bit. (The upstream repo's own worker calls .to() — that's why we wrote our own loader.)4.2 Attention backends (the model defaults are wrong for Ampere)
The saved
config.json bakes "magi" (MagiAttention, Hopper/Blackwell-only) at the top level and leaves text_config._attn_implementation unset, so relying on from_pretrained(attn_implementation=...) to propagate is fragile. Fix in _load(): load AutoConfig first and set the backend explicitly on all three configs:- text decoder →
sdpa. Verified in the model code: sdpa is a first-class path for the parallel-box-decoding block mask (_prepare_block_mask_for_inference), not a degraded fallback. MTP fast decoding works under it. - vision (MoonViT) →
flash_attention_2(falls back to sdpa internally if flash-attn is missing). The Jetson AI Lab index has a prebuiltflash-attn==2.8.3for sm_87. Counter-intuitively, forcing vision tosdpareproducibly OOMs at load — its mask path allocates a large buffer.
4.3 The visual-token budget (in_token_limit)
The image processor's default
in_token_limit=25600 (pre-merge patches) means a large photo (the first test image was 12 MP) produces a huge visual-token sequence whose KV cache OOM-kills generation. Fix: set processor.image_processor.in_token_limit = max_patches (default 4096 → ~1024 visual tokens after 2x2 merge, ~1 MP — plenty for detection). But — see Part 3.2 — this is not a free speed knob: push it too low (1024) and the model degenerates. 4096 is the sweet spot.4.4 Packaging & environment (Jetson aarch64 pain)
Problem | Fix |
Upstream pip install -e . pulls deepspeed, liger_kernel, gradio, decord, MagiAttention… mostly unbuildable on aarch64 | Install --no-deps and hand-pick inference-only deps |
Generic PyPI aarch64 torch is not Jetson-compatible (different CUDA/cuDNN ABI) | Use the Jetson AI Lab index pypi.jetson-ai-lab.io/jp6/cu126 (torch 2.11, torchvision, triton, CUDA-enabled bitsandbytes 0.48-dev, flash-attn 2.8.3) |
import torch fails: libcudss.so.0: cannot open shared object file | The JAL torch wheel links cuDSS, absent from JetPack. pip install --no-deps nvidia-cudss-cu12 • add its .so dir to LD_LIBRARY_PATH |
decord (video) has no aarch64 wheel anywhere, but is hard-imported | Ship a stub decord.VideoReader that raises; fetch_video() catches it and falls back to torchvision — so video still works |
import cv2 fails: JAL opencv links libnvcuvid.so.1 (absent on Orin); PyPI headless has no GStreamer (no CSI camera) | Symlink JetPack's system cv2 ( /usr/lib/python3/dist-packages/cv2…so, 4.5.4 with GStreamer) into the venv. Surgical — one .so. Never pip-install opencv into the venv |
python3 -m venv fails: ensurepip missing | sudo apt-get install -y python3.10-venv |
4.5 Power & operations
⚡nvpmodel -m 2, not-m 0. On the Orin Nano Super, mode 0 is the 15 W profile (GPU capped 612 MHz); mode 2 = MAXN_SUPER (1020 MHz). Measured: 4.72 s → 1.73 s (2.7×) just from the mode switch. It resets to mode 0 on reboot, so the launcher sets it every time.
- Hardware MJPEG decode fails with the VLM resident (
nvv4l2decodercan't allocate ring buffers) → use--cpu-decode(default in the launcher). Free at <1 fps. - A reboot is cheap and helps — crashed CUDA processes leave the box ~1.4 GB worse off.
pkill -f <name>over SSH is dangerous — if the command string contains that name it kills your own SSH shell. Usetmux kill-sessioninstead.
Part 5 — The working app, end to end
live_camera_prompt.py is the whole thing (~640 lines). Flow:- Startup:
start_vlm_prompt.sh→ power mode, free memory, retry-launch. - Load:
la_common.load_model()— pre-quantized nf4, sdpa text / FA2 vision. - Camera thread: grabs frames, overwriting rather than queueing, so inference always resumes on the newest frame (reports
frames_skipped— typically ~50 per inference at 25 fps). - Render thread (
render_loop, ~25 fps): draws the last boxes (advanced byFlowTracker) onto the live frame, encodes JPEG, serves the MJPEG stream. - Inference loop (~0.3–0.6 fps): grabs newest frame →
infer()→parse_boxes()→set_result(). - Web server (
:8080): MJPEG stream, a prompt form (with atoucheddirty-flag so the 1 s stats poll can't clobber your typing), the model's raw answer text (to tell "found nothing" from "parser didn't match"), and a hardware telemetry panel fromtelemetry.py(per-core CPU, GPU %, GPU MHz, RAM, swap, thermals from sysfs; no wattage on this board).
Typical loaded reading: GPU 98.9% @ 1020 MHz, RAM ~7.0/8.0 GB, tj ~61 °C.
Part 6 — Quick Q&A drill (interview-style)
Why not just run the model normally?
7.7 GB bf16 weights don't fit in 8 GB unified memory. 4-bit nf4 (~2.2 GB) does.
Why is it slow, and where would you optimize?
~68% of runtime is 4-bit LLM decode, which is bandwidth-bound (~60% efficiency). Output length is the dominant driver. The real lever is better W4 kernels (AWQ/GPTQ) + CUDA graphs — not the vision tower (only 10%) and not shrinking the input (backfires).
How do you get 25 fps video from a <1 fps model?
Decouple them: a render thread draws boxes at 25 fps while the inference thread runs independently, and an optical-flow + MOSSE tracker carries the boxes forward between inferences.
What was the hardest bug?
The load failing with
NVML_SUCCESS == r INTERNAL ASSERT — it looks like a torch bug but is really OOM (NVML unsupported on Tegra masks the message). Root causes were page cache not dropped, and expandable_segments:True.What did you learn that contradicts standard advice?
On sm_87: bf16 compute beats fp16 for 4-bit kernels (1.8×); cutting visual tokens makes it slower not faster; sdpa is first-class here, not a fallback; and
nvpmodel -m 0 is the slow mode on the Super board.What's the honest performance ceiling?
~3–5 fps for this 3B VLM on this board. 30 fps is not physically reachable given the memory bandwidth.
🗓️ Written 2026-07-21. Device: Jetson Orin Nano Super 8 GB, JetPack 6, CUDA 12.6. Code snapshot: local repolocateanything-jetson/.
Made with Bullet