I personally shove Transmission into Docker:

services:
  wireguard:
    image: ghcr.io/linuxserver/wireguard
    container_name: wireguard
    cap_add:
      - NET_ADMIN
      - SYS_MODULE
    environment:
      - PUID=1000
      - PGID=1000
      - TZ=Europe/Stockholm
    ports:
      - 9091:9091/tcp
    volumes:
      - ./config:/config
      - /lib/modules:/lib/modules
    sysctls:
      - net.ipv6.conf.all.disable_ipv6=0
      - net.ipv4.conf.all.src_valid_mark=1
    restart: unless-stopped
  transmission:
    image: ghcr.io/linuxserver/transmission
    container_name: transmission
    ulimits:
      nofile: 1048576
    environment:
      - PUID=1000
      - PGID=996
      - TZ=Europe/Stockholm
      - USER=azurediamond
      - PASS=hunter2
    volumes:
      - ./config:/config
      - /data:/data
      - /data/Torrents/dl:/downloads
      - /data/Torrents/inbox/start:/watch
    network_mode: "service:wireguard"
    depends_on: [ "wireguard" ]
    restart: unless-stopped

Make sure your mullvad config is called wg0.conf in ./config.

Consider moving from mullvad if you are into torrenting. Mullvad doesnt support port forwarding anymore. But no matter what provider you chose, use docker container gluetun to route traffic from any other container (like transmission). I like transmission, but most people use qbittorrent because its more advanced. You can also set up VPN in qbittorrent settings

I highly recommend moving that to a docker or podman setup. Gluetun is the go-to for VPN traffic. Set up a little container and you can link other containers to it to route all their traffic through.

i use this https://github.com/haugene/docker-transmission-openvpn

You might want your program and the vpn interface to be in a separate namespace? Somewhat opposite this: https://www.wireguard.com/netns/#the-new-namespace-solution

You don’t want all traffic routed, you want only this one app. More info here: https://linuxhint.com/use-linux-network-namespace/

https://github.com/Safing/portmaster

You could use something like portmaster to route only that program to the VPN.

Probably safer to configure your program to use the mullvad proxy. That way if mullvad is offline it just won't work.

The gold standard for what you want to do is qubes, where you would set up a VM that can only talk to a VM that's routing to the VPN. But that's a lot of work. You might be able to set something similar with containers.

This other post may be interesting as well https://slrpnk.net/post/2096570

If transmission is running in a container, my latest blog post is actually about that: https://www.nicoco.fr/blog/2023/09/10/wireguard/