That is filesystem-level. Btrfs and i think ZFS? have deduplication built in.

Btrfs gave me 150 GB on my 2 TB gaming disk that way.

Not recommending software. As you mentioned old hard disks, it is better to copy the files or better dd them on a ssd. That way making index and finding duplicates will be faster cause you've to access files once and not care about fragmentation if you dd.

I've had great success with restic. It will handle your 4TB just fine, here's some stats of mine:

Total File Count: 78374
Total Size: 13.324 TiB

and another one, not as large but with lots of files

Total File Count: 1295210
Total Size: 2.717 TiB

Restic will automatically deduplicate your data so your duplicates won't waste storage at your backup location.

I've recently learned about backrest which can serve as a restic UI if you're not comfortable with the cli, but I haven't used it myself.

To clean your duplicates at the source I would look into Czkawka as another lemming already suggested.

fdupes to find duplicate files, freefilesync to back it up.

Nightly rsync job in crontab works well enough, if its an external hard drive.

If you're going over a network, syncthing.

I've had success using Czkawka (hiccup) for deduplicating

There's BeyondCompare and Meld if you want a GUI, but, if I understand this correctly, rmlint and fdupes might be helpful here

I've done similar in the past - I prefer commandline for this...

What I'd do is create a "final destination" folder on the 4TB drive and then other working folders for each hdd / cd / dvd that you're working through

Ie

/mnt/4TB/finaldestination /mnt/4TB/source1 /mnt/4TB/source2 ...

Obviously finaldestination is empty to start with so it could just be a direct copy of your first hdd - so make that the largest drive.

(I'm saying copy here, presuming you want to keep the old drives for now, just in case you accidentally delete the wrong stuff on the 4TB drive)

Maybe clean up any obvious stuff

Remove that first drive

Mount the next and copy the data to /mnt/4TB/source2

Now use rmlint or fdupes and do a dry-run between source2 and finaldestination and get a feel whether they're similar or not, so then you'll know whether to just move it all to finaldestination or maybe then use the gui tools.

You might completely empty /mnt4TB/source2, or it might still have something in, depends on how you feel it's going.

Repeat for the rest, working on smaller & smaller drives, comparing with the finaldestination first and then moving the data.

Slow? Yep. Satisfying that you know there's only 1 version there? Yep.

Then do a backup 😉

For duplicates: Czkawka. Also, you get a gold ⭐ if you can figure out how to pronounce it 😉

Take a look into borg backup.

I'm using Borg and it's fine at that scale. I don't know if it would still be viable with 100TB or whatever. The initial backup will be kind of slow but it encrypts everything, and deduplicates it too if I'm not mistaken. In any case, it deduplicates the common situation where you back up another snapshot later. Only the differences get written in the second backup. So you can save new snapshots fairly quickly and without much additional space.

So a lot of backup solutions do deduplication on the block level, so if you use a backup software that does this, you don't need to dedup files.

Personally, my toolkit includes Jdupes for duplication scanning and Rsync for directory merging and file transfer.

I have had good luck with Dupeguru

Honestly I maintain a list of file types I care about and copy those off. It’s mostly things I’ve created or specifically accumulated. Things like mp3, mkv, gcode, stl, jpeg, doc, txt, etc. Find all of those and copy them off. I also find any files over a certain size and copy them off unless they are things like library files, dlls, that sorta thing. Am I possible going to kiss something, yeah. But I’ll get most of the things I care about.

Do you need any of it? Usually I've not even thought about what might be on an old drive.

If I was worried about the slim chance there's something of critical importance I'd need later, I'd just look over each device and pick out individual files I might want, and dump the rest.

If you're extremely paranoid, I'd take a block-level backup of each device and archive it.

Deduping only works for a single target or context at a time, so if you're working with many drives, you'll need to sort your data into unified locations on the backup target first, THEN run dedupe tools against it all.

Second, if all of your data from these drives fits uncompressed on the target drive, rsync will be the fastest to get the data from A to B.