This of course has to do with your swimming technique, and it is quite normal.
Unlike running, swimming requires that part of your training is specifically devoted to learning the technique. That is, your workouts cannot be composed alone by breast stroke, front crawl and so on, but also you have to devote certain amount of time to specific swimming drills aimed at focusing only on a single component of swimming at a time (e.g. the leg kicks or how to breathe). Here is an example youtube list
Swimming workouts usually combine sessions with different kinds of swimming drills. You may choose to ignore the drills and simply keep on trying, but then it is very difficult that you achieve a decently efficient front crawl style. Furthermore, the more you swim with bad form, the more difficult to correct your swimming patterns in the future.
Additional Warning: You seem to have been swimming quite a lot by means of breaststroke only. Improper breaststroke technique may be eventually bad for your knees. The whipkick used in the breaststroke easily leads to problems. This is the well known "Swimmer's Knee" or "Breaststroker's Knee". A good workout should combine different swimming styles. Try focusing your attention on your technique rather than on your mileage.