The downside is you'll never fix the problem to where you can remove the brace, you'll be dependent on it. The goal is to fix the muscle asymmetries so that your shoulders pull themselves back on their own so you can live your life brace free. Think of someone using a walker.. sure it helps them walk, but your leg muscles are weak from being dependent on it that when you remove the walker your leg muscles will have a hard time walking on their own. You want your legs to be strong enough that you don't need a walker. So walking longer and longer distance is usually the cure for getting rid of the walker. Over long term, braces and walkers will actually weaken your muscles and make it worse where youre muscles are so used to it that they can't support you hardly.
In short, a brace is great for short time periods, when you're in pain, or when you might be at work slumped in a chair all day. But not for long term. Try to work on using good posture 24/7, lie flat on your back, and work on strengthening your rhomboids, lower traps, middle traps, rear deltoids, rotator cuffs, and stretching other muscles such as your pecs, upper traps, and anterior deltoids. I'd suggest seeing a physical therapist and he/she will give you a workout plan. But if you can live without it don't rely on body assistance pieces such as shoulder or back braces, walkers, etc..
If you wear a brace all day, that cues your muscles to stop activating, so after long term, you take it off and your muscles will fail and you won't be able to hardly stand. But if you just wear a brace while you're at work lifting heavy equipment to save your back, or wear if while moving houses, etc.. then it's fine. Short term is great, long term is very bad. A majority of back/shoulder problems can be fixed by fixing muscle asymmetries. You can also look into kyphosis workouts, forward neck workouts, or even lower back workouts which focus on strengthening your abs and glutes.