You do not NEED to defrag, but you should. When you delete files it leaves gaps. When writing new files they can be filled in where ever there is space available. That's how they get fragmented.
What you want is Volume Optimization. This is NOT to be done on SSD drives, but on spinning disks it is very good practice. I scan slides, usually 140 in a carousel. They are on a G-RAID in mirror mode. Every time I finish one I will back it up to a LaCie drive. Then run File Defrag, which takes seconds, then run Volume Optimization, then run Volume Rebuild. I have run this routine hundreds of times with no issues. Your drive is likely to have many GB data that is very scattered. The first time you do this may take hours. But after that all data is collected into one contagious block.
Go to Micromat.com Buy TechToolPro. They have been having discounts lately. It is the best and oldest utility for Mac only. I believe they share info back and forth with Apple engineers.