Two perspectives.
1. It depends
Whether deadlifts are worth it if you're already doing heavy swings depends on what you're training for. The swings get more of your arms, but I don't deadlift for my arms much. I don't mind that they increase grip strength and require some bicep activation but that's not the point.
Are you trying to sweaty that works the whole body? Is your goal to get tired fast? Then swing. If you're trying to develop maximal strength in your posterior chain, lats, spinal erectors, glutes, hamstrings, then deadlift.
Use the tool which helps you achieve your goals.
2. Deadlifts are worth it
Maximal strength
Swings are great, but the deadlift is something else. It's definitely possible to avoid the deadlift and still build incredible maximal strength, but the people who do tend to either train like madmen or be genetic outliers. People who do cleans, snatches, Atlas stone lifts, and very heavy swings but not deadlift tend to train with high frequency and intensity. They can trust their back strength for those other lifts. If you're not particularly athletic and strength-trained, I recommend some caution with lifts that involve a lot of back strain in positions that can easily become compromised, as the fast and odd lifts do.
Parts of the body
Surely the deadlift let's you lift more weight, but what's the point if it only works my back and lower body
The point is working the back and lower body against greater resistance? That's why most people deadlift--for maximal strength in their back and lower body. Those parts of the body are much stronger than the arms, so if one does an exercise that taxes the arms, it might not challenge the muscles behind the hinge movement. Consider the fact that you maybe should want to train maximal strength in your back and legs rather than use a "cool exercise" to get a whole-body pump going. Some people call this the distinction between "training" (improving yourself systematically) and "exercising" (getting sweaty & tired without a concrete plan).
Drawbacks of only doing explosive exercises
Say you don't care about maximal strength. That's valid! But the swing is a fast, dynamic, explosive exercise. The deadlift can improve and use those attributes, but is much more slow and controlled. This makes it a good weapon to have in your arsenal for when you're feeling sluggish or beat up but still want to train. Truly heavy swings require more coordination to stay injury-free than a similarly challenging deadlift, because there's less room for error while hitting reps at a fast clip.
I wonder if the people you see swinging 300 pounds were able to do so by deadlifting too. It's likely they did.
The deadlift is the deadlift
I also firmly believe that people who have never pushed their deadlift don't have the experience necessary to understand its benefits. It's just not possible to explain, the same way words are insufficient to communicate the feeling of cardio fitness or the feeling of properly executing highly technical gymnastic or dance movements. To legitimately compare the deadlift to other exercises, you must train it to at least a baseline level, which I consider to be double bodyweight for men who are at least semi-athletic.
