Based on what I've read, we do have a genetic potential, but what exactly that is for any person is nearly impossible to determine.
Scientific Validity - Dr. Casey Butt did a study based on the top bodybuilder athletes that provided some formulas to estimate your genetic potential. Now, as he describes in the article, the study was based on observing the body building community, and the numbers are not hard limits. However, you would have to train harder than the people he used in the study. The formulas are fairly accurate when applied back on the complete population that the formulas were generated from.
Plateau - Plateaus are an indication that either the type of stress you are applying is not enough to cause adaptation, or that you do not have enough recovery to allow adaptation to complete. As you become more and more of an advanced athlete, you will have to vary the types of stress (endurance rep ranges, strength rep ranges, and changing assistance exercises) to stimulate the type of adaptation you want. Practical Programming for Strength Training by Dr. Kilgore and Rippetoe have a lot to say about that--whether you are interested in strength or physique the same principles apply.
Hard Limit - Using the same reference in the last bullet point (Practical Programming), there are some graphs that help illustrate what you suspect. Essentially, the graph of results over effort is logarithmic. In short as a beginner, your adaptation is quick and you can make incredible gains very quickly. The closer you get to your genetic potential (the limit in calculus terms), the more work you have to put in and the more carefully you have to manipulate all the variables to help you progress. An elite lifter will only be able to make an increase once or twice a year.
Sarcopenia - Getting old sucks, and one of the reasons for people who use weights is Sarcopenia. Essentially the older you get the more muscle mass you lose. For sedentary people it can be up to 15% per decade after 30-35. For weightlifters (and by extension bodybuilders) will slow the loss down to 5% per decade (per Practical Programming). Theoretically, if you hit your genetic peak before turning 30, the capacity of what you can do will diminish as you get older. Lifting weights is one way of slowing down those changes.