I shipped one of these tools. It removes the background from an image entirely on-device: no upload, works offline after the first load. Getting it running on a desktop GPU took an afternoon. Getting the same idea to survive contact with actual phones took weeks, and the fix was never "optimize the model." It was: stop pretending one engine fits every device.

That's the part the demos skip. So here's the real routing, why each branch exists, and the one specific combination of headers and threads that reliably hangs an iPhone.

The afternoon version (and why it lies)

The happy path really is short. transformers.js will load an ONNX segmentation model and run it on WebGPU with fp16 weights in about six lines:

import { AutoModel, AutoProcessor, env } from "@huggingface/transformers";

env.allowLocalModels = false;

const MODEL_ID = "studioludens/birefnet-lite-512"; // ~98MB fp16, Apache-2.0

const processor = await AutoProcessor.from_pretrained(MODEL_ID);
const model = await AutoModel.from_pretrained(MODEL_ID, {
  dtype: "fp16",
  device: "webgpu",
});

On a desktop with a real GPU this flies. The fp16 weights halve both the download and the VRAM, the mask comes out clean, and you feel like a genius. Ship it. Tweet about it.

Then the bug reports land. They're all from phones, and no two of them describe the same failure. One phone loads the model and freezes the entire UI mid-inference. Another never finishes booting the worker at all. A third goes white and reloads itself, like it's embarrassed. None of it reproduces on the machine you built it on. That's the worst flavor of bug, the kind where your test setup physically cannot see what's wrong.

A few things turn out to be true at the same time, and you don't get to ignore any of them:

• Most phones don't have usable WebGPU in the browser. iOS Safari is the cruel one here: it claims support, then hangs WebKit/ONNX inference instead of failing loudly. A hang is worse than a crash, because a crash you can at least catch.

• The fallback, multi-threaded WASM, needs SharedArrayBuffer. Which needs cross-origin isolation. Which is the exact thing that makes some phones refuse to start the worker in the first place.

• A mid-range phone has a sliver of your laptop's RAM and memory bandwidth, so the tempting "just run it on the main thread" shortcut turns a 4-second job into a 40-second frozen tab.

So the real architecture isn't "the browser version" of anything. It's a router that hands out a different model, a different runtime, a different threading model, and different HTTP headers depending on what's holding the page open.

Routing, not optimizing

The decision happens once, at worker-spawn time, off a user-agent sniff. UA sniffing is deeply unfashionable and I'd take a cleaner signal in a heartbeat. But the failure modes here are device-class-specific (iOS WebKit, Android RAM ceilings), and there is no feature flag for "this phone is about to OOM." So, user-agent it is. I'm not proud, I'm just shipping.

function getSharedWorker(): Worker {
  const ua = navigator.userAgent;
  const isPhone = /iPhone|iPod|Android/i.test(ua);

  // iPad reports a Mac-class UA, so detect it by touch + Mac UA, not /iPad/ alone.
  const isIPad =
    /iPad/i.test(ua) ||
    (/Macintosh|Mac OS X/i.test(ua) && navigator.maxTouchPoints > 1);

  const useMobileEngine = isPhone || isIPad;

  return useMobileEngine
    ? new Worker(new URL("./bg-remover-mobile-worker.ts", import.meta.url))
    : new Worker(new URL("./bg-remover-worker.ts", import.meta.url));
}

That iPad branch earns its rent. Modern iPads ship a desktop "Macintosh" user-agent, so /iPad/ matches nothing and /Macintosh/ matches a real Mac and an iPad. The tiebreaker is maxTouchPoints: a real Mac reports 0, an iPad reports more than 1. A touchscreen Windows laptop has a non-Mac UA, so it won't false-positive. It takes three conditions to answer the question "is this an iPad," which sounds insane until you remember Apple is the one who decided iPads should lie about being Macs. I'm just cleaning up after that decision.

Two engines fall out of this.

Desktop: transformers.js on WebGPU

Desktop runs the transformers.js worker and pulls birefnet-lite-512 at fp16, preferring WebGPU and dropping to multi-threaded WASM only on a machine with no usable GPU. The device preference is, literally, a list with a fallback:

function preferredDevices(): ("webgpu" | "wasm")[] {
  if (iosForceWasm) return ["wasm"];           // iOS: skip WebGPU, it hangs
  return hasWebGPU() ? ["webgpu", "wasm"] : ["wasm"];
}

iosForceWasm is a belt-and-suspenders flag: iOS and iPadOS Safari say yes to WebGPU and then hang. That hang is the whole reason Apple's touch devices never run this worker at all. They get their own engine, which is the next section.

iPad and phones: onnxruntime-web 1.18, one thread, no transformers.js

iPads and phones get their own worker, their own runtime, and their own pair of models. Not transformers.js. Plain onnxruntime-web, pinned to 1.18, aliased in package.json so two versions of the runtime can live in the same project without fighting:

import * as ort from "ort-stable"; // onnxruntime-web@1.18, NON-threaded build

ort.env.wasm.wasmPaths = "/ort-stable/";
ort.env.wasm.numThreads = 1;        // single thread. on purpose.

const session = await ort.InferenceSession.create(modelUrl, {
  executionProviders: ["wasm"],
  graphOptimizationLevel: "all",
});

numThreads = 1 is not a TODO I forgot to come back to. It's load-bearing. One thread means no SharedArrayBuffer, which means no cross-origin isolation, which means the worker actually boots on the exact phones that rejected the isolated build. The model still runs inside a Web Worker, so the UI never locks up. It just grinds away on a single CPU core instead of several. Slower per image, sure. But it finishes, and it doesn't take the whole tab down with it, which turns out to be the bar that matters.

The model itself also forks by phone:

const MODEL: MobileModel = /iPhone|iPod/i.test(UA) ? "isnet" : "birefnet";

iPhone gets imgly's MIT-licensed ISNet (fp16, ~88MB). Android and iPad get BiRefNet-Lite (fp16, ~94MB), since the iPad reports a "Macintosh" UA and falls to the non-iPhone branch. This split is empirical, not some elegant principle: those were the pairings that gave the cleanest masks per platform without blowing the memory budget. The iPad landed on this lean engine for a different reason than phones: a phone needs it because the isolated, threaded build won't even boot there, while an iPad will boot the heavier transformers.js engine and then run itself out of memory mid-batch. Same destination, two different cliffs. One gotcha I'll write down so you don't lose an afternoon to it like I did: ISNet runs a sigmoid inside the graph. If you sigmoid the output again on the way out, you get a washed-out, half-transparent mask, and you will swear up and down the model is broken. The model is fine. You did the sigmoid twice. (I did the sigmoid twice.)

The combination that hangs iOS

Here's the trap that ate the most time, because it's the one every tutorial walks you cheerfully into.

To run multi-threaded WASM you need SharedArrayBuffer. To get SharedArrayBuffer you need the page cross-origin isolated, which means serving these two headers:

Cross-Origin-Opener-Policy: same-origin
Cross-Origin-Embedder-Policy: credentialless

Every "run ONNX in the browser with threads" guide tells you to set them. The guides are right, for desktop. The trouble starts when you set them globally and a phone wanders in. Isolation plus a threaded worker is precisely the cocktail that hangs iOS Safari and helps Android workers decide not to boot. It isn't one header. It isn't one thread. It's the combination of isolation and a threaded worker running on a memory-pinched WebKit or Chrome-on-phone runtime.

So the headers are conditional, set in the proxy by user-agent, and only for the workspace route:

const ua = request.headers.get("user-agent") ?? "";
const isPhone = /iPhone|iPod|Android/i.test(ua);

if (!isPhone) {
  // Desktop isolates so threaded WASM / SharedArrayBuffer works. (iPad matches this branch
  // and gets the headers too, but it runs single-threaded, so for it they're a no-op.)
  res.headers.set("Cross-Origin-Opener-Policy", "same-origin");
  res.headers.set("Cross-Origin-Embedder-Policy", "credentialless");
}
// Phones: send NO isolation headers. The single-thread 1.18 build
// needs no SharedArrayBuffer and only boots when un-isolated.

That's the whole secret, and it's a genuinely annoying one: the same headers that switch on the desktop engine break the phone engine. There is no single config that satisfies both. You serve isolation to the devices that need it and you withhold it from the devices it kills, and you make your peace with the asymmetry.

I had to carve this rule into my own notes, because I tried to "simplify" it back into one config more than once and re-broke every phone each time: phones and iPad stay on ONNX 1.18, single-thread, in a worker. Phones stay non-isolated; the iPad gets the isolation headers but doesn't use them. Only the desktop runs the threaded / WebGPU transformers.js engine. Do not merge them. Future me, this means you.

What I'd do differently

A few honest ones, because the write-ups that only list wins aren't useful to anybody.

I'd stop trusting supportsWebGPU-style checks a lot sooner. The entire iOS detour exists because Safari answers "yes" to WebGPU and then hangs, which makes a capability check that returns true strictly more dangerous than one that returns false. I treat iOS as WASM-only by policy now, not by feature detection. I paid for that lesson in evenings.

I'd also accept earlier that there's a memory ceiling I simply cannot see from JavaScript. A low-end Android can OOM on cold model load, and there's no error to catch, no event, no nothing. The OS just takes the tab. And I removed navigator.deviceMemory on purpose, because it's a fingerprinting signal and the entire point of this tool is privacy, so I can't even read the RAM to bow out gracefully. The mitigation is unglamorous: smaller fp16 models, one image at a time, free the WASM-heap tensors the instant inference returns. Nothing clever. Just stop being greedy with memory you can't measure.

And the limit nobody puts on the landing page: these lightweight models find a salient subject. Aim one at a clean portrait and it nails it. Aim it at a cluttered gym-mirror selfie and it might hand you a near-empty mask, because it genuinely can't decide who the subject is among the towels, mirrors, and three other people. A bigger model would handle that better. A bigger model is also the exact thing that won't load on the phone you're bending over backwards to support. That tradeoff doesn't resolve. You just choose where to stand on it and say so out loud, which is most of what honest engineering is.

The takeaway

"AI in the browser" isn't one feature. It's a fan-out. The model is the easy 10%. The other 90% is that a WebGPU desktop and a lean single-thread engine for the iPad, iPhone, and Android each want a different runtime version and model, and the desktop and phones even want mutually exclusive HTTP headers. The only way to know which device you're dealing with is to sniff the user-agent and accept that you are, in fact, treating no two of them the same.

If your in-browser ML demo has exactly one code path, it doesn't run everywhere. It runs on your machine. Those are not the same claim, and the gap between them is most of your users.

Built by Swathik, solo indie developer. The tool lives at pdfandimagetools.com/background-remover if you want to open your own Network tab and watch nothing get uploaded.

Then someone uploads a photo straight off their iPhone and your converter just shrugs, because it's a .heic and the browser has never heard of it. Then someone wants a favicon with five sizes baked into one .ico. Then a .svg that's secretly 16px wide because nobody set a width. Then a multi-page .tiff off a scanner. That's the part nobody tweets about, and it's the part this post is about.

I built a converter that handles 20 conversion routes across about ten formats, entirely client-side. No server, no upload, the file never leaves the tab. Most of those pairs really are two boring lines of canvas. A handful made me read binary format specs at an hour I'd rather not put in writing. Here's where the line sits, format by format, and which ones fought back.

The boring (good) news: most of it is canvas

Browsers already decode JPG, PNG, WebP, BMP, and GIF natively. Hand any of those to an <img>, wait for onload, draw it onto a canvas, re-encode. For two-thirds of the formats, that's the entire "engine":

function loadHTMLImage(blob: Blob): Promise<HTMLImageElement> {
  return new Promise((resolve, reject) => {
    const img = new Image();
    const url = URL.createObjectURL(blob);
    img.onload = () => {
      // decode off the main thread BEFORE we resolve, so the later
      // drawImage doesn't trigger a synchronous main-thread decode
      if (typeof img.decode === "function") img.decode().then(done, done);
      else done();
      function done() { URL.revokeObjectURL(url); resolve(img); }
    };
    img.onerror = () => { URL.revokeObjectURL(url); reject(new Error("decode failed")); };
    img.src = url;
  });
}

Encoding is the mirror image: draw the bitmap onto a sized canvas, then canvas.toBlob(mime, quality). The one genuinely useful detail here, and it's a footgun people hit constantly, is JPG transparency. JPG has no alpha channel, so if your source PNG has transparent pixels, a lot of browsers render them as solid black. The fix is to make the canvas opaque from birth:

const ctx = canvas.getContext("2d", { alpha: false });
ctx.fillStyle = backgroundColor ?? "#ffffff";
ctx.fillRect(0, 0, w, h);
ctx.drawImage(bitmap, 0, 0, w, h);

Belt and suspenders. Fill a white background and ask for a context with no alpha channel at all. Now transparent pixels can't leak through, even on the browsers that quietly ignore the fill order. Worth knowing whether or not you ever go near HEIC.

So if everything were JPG, PNG, and WebP, this would be a 40-line library and a much shorter post. It isn't. Five formats refuse to play along.

First: don't trust the file extension

Before you decode anything you have to know what it actually is, and .jpg lies all the time. People rename files. Screenshots get saved with the wrong extension. iOS will occasionally hand you a HEIC wearing a .jpg name, apparently for sport.

So detection sniffs magic bytes, not the extension:

const head = new Uint8Array(await file.slice(0, 32).arrayBuffer());

if (head[0] === 0xff && head[1] === 0xd8 && head[2] === 0xff) return "jpeg";
if (head[0] === 0x89 && head[1] === 0x50) return "png";            // ‰PNG
if (head[0] === 0x42 && head[1] === 0x4d) return "bmp";            // BM
// TIFF little-endian "II*\0", big-endian "MM\0*"
// ICO is 00 00 01 00
// HEIC/AVIF share ISO-BMFF: bytes 4-7 spell "ftyp", brand at 8-11

HEIC and AVIF are the sneaky pair. They're both ISO Base Media containers (same family as MP4), so they both open with an ftyp box. The magic number alone can't tell them apart. You have to read the brand string that follows: heic, heix, mif1 and friends mean HEIC, while avif and avis mean AVIF. Get this wrong and you'll route an AVIF straight into the HEIC decoder and burn a 600KB download for nothing.

SVG is the odd one out, because it's text. No binary signature at all, so the last check reads the first 256 bytes as a string and looks for <svg or an XML preamble. Only when all of that fails do we fall back to the extension and MIME type, strictly as a last resort.

HEIC: the format people actually search for

This is the headline act. "Convert HEIC to JPG" has real search volume, because every iPhone shoots HEIC by default and almost nothing else opens it. Doing that without a server is the whole point. You shouldn't have to upload your private photos to a stranger's box just to escape Apple's container format.

No browser decodes HEIC natively. Safari can display a HEIC the OS already understands, but you can't reliably pull pixels back out of it through canvas. So this one needs a real decoder. I lazy-load heic-to, a WASM build of libheif, and only on the first HEIC a user actually touches:

async function decodeHeic(file: File) {
  const { heicTo } = await import("heic-to");        // ~600KB, fetched once, cached
  const jpegBlob = await heicTo({ blob: file, type: "image/jpeg", quality: 1 });
  return loadHTMLImage(jpegBlob);                     // now it's a normal image
}

Two decisions worth calling out. First, it's a dynamic import(), so the 99% of people who only ever convert PNGs never pay for the HEIC WASM. Second, I ask libheif for quality: 1 (max), because this is only the decode step. The real quality knob lives downstream in the encode step, and squashing quality twice would tax the same pixels twice.

Honest limit: it's a big module, and on an old phone the very first HEIC of a session has a visible "thinking" pause while it downloads and warms up. After that it's cached and quick. I decided a one-time pause beats uploading someone's camera roll to a server, but it's a genuine tradeoff and I'm not going to pretend the WASM is free.

ICO: where I stopped feeling clever

A favicon .ico is not one image. It's a tiny archive: a 6-byte header, then a 16-byte directory entry per size, then the image payloads packed at the end. Each entry can be a PNG or an old-school DIB (BMP-ish) bitmap. Modern toolchains write PNG. Plenty of files still in the wild carry DIB.

Decoding means parsing that directory by hand:

const count = view.getUint16(4, true);          // how many sizes are packed in
for (let i = 0; i < count; i++) {
  const entry = 6 + i * 16;
  let w = view.getUint8(entry) || 256;          // a stored 0 means 256, naturally
  const size   = view.getUint32(entry + 8, true);
  const offset = view.getUint32(entry + 12, true);
  const bytes  = new Uint8Array(buffer, offset, size);
  // PNG payload? decode via <img>. DIB payload? synthesize a BMP header and decode that.
}

The "0 means 256" rule is a real spec quirk. The width and height fields are a single byte each, so 256 (the largest icon size) won't fit and gets stored as 0. Miss it and your 256px icon decodes as a 0px nothing.

The DIB entries were the actual fight. A DIB inside an ICO stores its height doubled, because an XOR color plane sits stacked on top of an AND mask plane, and it carries no file header, so the browser won't touch the raw bytes. I prepend a synthetic 14-byte BMP header and patch the height field back to its true value so the browser renders just the color plane. The transparency mask gets dropped, and I've made my peace with that: nearly every ICO that matters today is PNG-encoded anyway, and the common case stays clean.

Encoding back to ICO is the reverse, and it's where the favicon use-case finally earns its keep. Render the source at each requested size (16, 32, 48, 192, 256, and so on), PNG-encode each one, then assemble the directory and concatenate the payloads:

dv.setUint16(0, 0, true);   // reserved
dv.setUint16(2, 1, true);   // type 1 = icon
dv.setUint16(4, sizes.length, true);
// then one 16-byte entry per size, then all the PNG blobs back to back
out[entryOffset + 0] = size === 256 ? 0 : size;   // and back to the 0-means-256 dance

One upload, one click, a real multi-size favicon out the far end, and no "favicon generator" site that emails you a zip afterward. Multi-size ICO is the kind of thing that's trivial to describe and surprisingly fiddly to get byte-correct, and it never once made it into the easy two-line version of this story.

TIFF: multi-page, and a library earns its keep

TIFF is the one where I happily handed the problem to someone smarter. It supports multiple pages, a pile of compression schemes, big and little-endian, and a whole zoo of color layouts. Reimplementing that is a multi-week tarpit with my name on it. utif is small, ships no WASM, and is multi-page aware, so I lazy-load it and walk the pages:

const UTIF = (await import("utif")).default;
const ifds = UTIF.decode(buffer);               // one IFD per page
const frames = ifds.map((ifd) => {
  UTIF.decodeImage(buffer, ifd);
  const rgba = UTIF.toRGBA8(ifd);               // raw pixels
  const ctx = canvas.getContext("2d");
  ctx.putImageData(new ImageData(new Uint8ClampedArray(rgba), ifd.width, ifd.height), 0, 0);
  return canvas;                                 // becomes one output frame
});

The design call: a multi-page TIFF comes back as multiple frames, and the UI surfaces an "extract all pages" option. Reaching for a battle-tested decoder here, instead of hand-rolling one to match the ICO chapter, was the right level of stubbornness. The whole skill is knowing when to write the parser and when to just install it.

AVIF: native, until it isn't

AVIF is the optimist of the bunch. Modern Chrome, Firefox, and Safari all decode it natively, so for most people it rides the same <img> path as JPG. No library, no WASM, lovely.

The catch is the long tail. Older browsers and some embedded webviews can't decode AVIF, and there's no graceful canvas fallback to lean on. The browser just fails. So rather than spin a loader forever, the decoder fails loudly and helpfully:

async function decodeAvif(file: File) {
  try {
    return await loadHTMLImage(file);
  } catch {
    throw new Error("Your browser can't decode this AVIF. Try a newer browser, or convert it on a different device.");
  }
}

There's a WASM AVIF decoder (@jsquash/avif) I could lazy-load as a fallback, the same trick HEIC uses. I haven't wired it in yet, because the browsers that fail are rare and I'd rather ship an honest error than a megabyte of WASM that 99.9% of visitors download and never use. It's an obvious "what I'd do differently" candidate. If I revisit it, AVIF gets the HEIC treatment.

SVG: a vector pretending to have a size

SVG converts fine. The gotcha is dimensions. Vectors have no intrinsic pixels, so "rasterize this SVG" is meaningless until you decide how big. Worse, plenty of SVGs leave out width and height entirely and carry only a viewBox, and a few carry nothing useful at all.

So before rasterizing I parse the opening <svg> tag and try, in order: explicit width/height attributes, then the last two numbers of the viewBox, then a 1024×1024 fallback so a sizeless SVG still produces something sane instead of a 1px speck or a crash. After that it's a normal canvas draw with smoothing cranked up, and the user can override the output size in the advanced options.

const w = parsePx(attrWidth) ?? viewBoxW ?? 1024;   // attr, then viewBox, then fallback

Small thing. But "convert SVG to PNG" silently producing a 16px image because the source had no explicit width is exactly the kind of papercut that makes a tool feel broken when the code is, technically, completely correct.

One bug that only happens on iPhones

A field note, because it cost me an afternoon. iOS Safari can reject a Blob.arrayBuffer() read for the second file in a batch with "The I/O read operation failed." The handle quietly decays between picks. So format detection has to survive a failed byte read and fall through to extension/MIME detection instead of throwing, and every decode path holds onto the bytes it needs rather than re-reading the Blob later. Test only on desktop Chrome and you'll never see it. Your users on phones will, and they'll hate you in silence.

The cheat-sheet

Print it, tape it near your monitor, skip the spec-reading I didn't:

And the rules that hold no matter the format:

• Sniff magic bytes, not the extension. Files lie.

• Decode off the main thread (img.decode()) before drawing, or you stall the scroll.

• For JPG output, make the canvas opaque from birth.

• Lazy-load the heavy decoders so common conversions stay weightless.

• Free the canvas (canvas.width = 0) after encoding, or batches OOM on mobile.

Why bother doing it client-side at all

Because none of it needs a server. Every format above decodes and encodes inside the tab using the browser's own image pipeline plus two small WASM/JS libraries that download only when their format shows up. The image never gets uploaded. You can prove that by opening the Network tab and watching zero bytes leave, or by killing your Wi-Fi after the page loads and converting anyway.

That's the actual reason I went down the ICO-directory rabbit hole instead of POSTing files to an endpoint and clocking off early. Your camera roll, your scans, your client mockups: none of that should have to take a round trip through someone else's hardware just to change a file extension. Most of the work really is two lines of canvas. The interesting 20% is the five formats that fight back, and now you've got the map.

If you want to poke at the working version, the converters live at pdfandimagetools.com under image tools. Open the Network tab while you use one. That part's the whole point.

Built by Swathik.