I think we should start with what building muscular endurance and/or inducing hypertrophy means. They are not mutually exclusive and are different in their objective.
Muscular endurance is the ability of a muscle or group of muscles to repeatedly exert force against resistance. Performing multiple repetitions of an exercise is a form of muscular endurance, as is running or swimming. If your muscles have to contract in a similar pattern more than one time you are using muscular endurance.
Many factors contribute to muscular endurance, including strength, fiber type, training and diet. A larger, stronger muscle can perform the same task under load more times than a weaker muscle. If you can bench press 300 pounds you will be able to perform more repetitions with 100 pounds than if you could only bench 200 pounds. A larger muscle also holds more glycogen, the sugar that you use for energy, so it will be able to sustain a series of contractions -- or perform work -- for a longer period.
Muscle hypertrophy involves an increase in size of skeletal muscle through a growth in size of its component cells.
Training variables, in the context of strength training, such as frequency, intensity, and total volume also directly affect the increase of muscle hypertrophy. A gradual increase in all of these training variables will yield the muscular hypertrophy.
Microtrauma, which is tiny damage to the fibers, may play a significant role in muscle growth. When microtrauma occurs (from weight training or other strenuous activities), the body responds by overcompensating, replacing the damaged tissue and adding more, so that the risk of repeat damage is reduced.
Now in regards specifically to doing planks.
The plank strengthens the abdominals, back, and shoulders. Muscles involved in the front plank include:
- Primary muscles: erector spinae, rectus abdominis (abs), and transverse abdominus.
- Secondary muscles (synergists/segmental stabilizers): trapezius (traps), rhomboids, rotator cuff, the anterior, medial, and posterior deltoid muscles (delts), pectorals (pecs), serratus anterior, gluteus maximus (glutes), quadriceps (quads), and gastrocnemius.
Now for planks, this is a type of static or isometric exercise that is definitely great to do as all of us could use a stronger more stable core. But when talking about overloading I understand your confusion as usually it's in reference to non-isometric exercises.
If you wanted specifically to use progressive overload techniques with planks you could simply gradually increase the volume (add weight to your back), intensity (less rest between sets), frequency (more sets), or time (most obvious one).
So based on all this it'd stand to reason that planks can build both these factors. I'm not super sure if your goal is to increase your plank endurance but I assume you are more leaning towards the muscle growth.
There's a lot of going on in this question and you could literally write a Ph.D. thesis on these subjects.
While planks are great, if you're looking to build muscle you'll also want to incorporate some more full range of motion exercises.