I was reading a Chabad article on this very topic: Why do they wear black robes? One of the answers is to be inner-directed rather than being directed by "external norms." I find that to be hypocritical because it is the same exact thing when people of the Goth subculture say they're non-conformist, but at the same time are conforming within their own niche. The author states that they are not worried about the dictates of fashion, but within the Haredi community, wearing black robes is the dictate of fashion. The minhag (custom) of wearing black is relatively new because I can tell you right now that Moshe Rabbeinu (Moses) was not walking around in the desert for forty years with a black robe on. This custom actually comes from an edict from 18th-century Ashkenazi rabbis because if they were wearing a "plain color" such as black, it wouldn't arouse envy amongst their non-Jewish neighbors. It's that simple--it was a practical measure in attempts to mitigate the anti-Semitism they were facing. Needless to say, the notion that wearing black robes is the ultimate Jewish expression took on a life of its own.
What is modest dress for a Jewish man? For starters, a man needs to wear tzitzit on the corners of his garment (Numbers 15:38). A man cannot wear women's clothing (Deuteronomy 22:5), and vice versa. This is merely a matter of what is written in the first five books of Moses. Wearing a kippah, although one of the most famous of Jewish practices, ironically enough has the least amount of halachic backing. As a matter of fact, a man such as the Vilna Gaon takes issue with calling it a requirement. Many, however, see the kippah as a universal minhag of a binding nature. Aside from head covering, we have to look at other texts to see what is considered "modest." Orach Chaim (2:3) states that you must be dressed in a dignified manner. We have to keep in mind since we are created in His image, we should dress with that sort of dignity. The Talmud (Berachot 62b) states that when King David showed the slightest irreverence for King Saul's clothing, he was punished. Dress, like everything else, should be conducted in a distinguished manner because we are, after all, doing everything for His sake.
For a man, dressing modestly seems easy. Wear tzitzit, don't wear shatnez, wear man's clothing (as opposed to women's clothing), keep your head covered, and wear clothing with dignity. Wearing different colors are not an issue because our ancestors wore different colors (Hint, hint, Joseph wore a "technicolor dream coat"). As long as you're not wearing it to imitate your non-Jewish neighbor (See discussion on Thanskgiving and imitating non-Jews), it is acceptable to wear anything you buy at your local department store. There is one area of clothing that we have not discussed yet, and this is whether it is acceptable for a man to wear shorts and short-sleeved shirts. For those of you who have not noticed, Haredim do not wear shorts, and very few will be wearing short-sleeved shirts. Natrually, they will state that any man who does is dressing immodestly. But is he? Let's take a look at a Jewish text. The Mishnah Beruah (75:2) states that the minimum standard for a female to be modestly dressed is to have the upper arm covered and to be covered from the knee downward. In the Orthodox community, tzniut (modesty) is much more emphasized with the women because the belief is that women are more likely to tempt men. Just look at the fact that the concept of kol isha exists, yet there is no prohibition of a woman listening to a man's voice. As a result, the Orthodox community is more strict with how women dress than how men dress. There is a halachic inference that can be made. In the name of R. Ishmael, the principle of מקל וחמר applies. Essentially, מקל וחמר states that an inference can be made from a lenient law to strict law, and vice versa. For example, your child's bedtime during the week is 9:30 because "it's a school night." On the weekend, one can infer that the child can stay up a little later, such as 10:30 or 11:30, because "it's the weekend." Although clothing has nothing to do with bedtime, the principle still applies. If a women's modesty is considered more strict and worth guarding than that of a man's, surely a man's minimum requirements cannot exceed that of a woman's. In short, if a woman's minimum requirement is to have the upper arm covered, as well as be covered from the knee downward, the man's halachic expectation cannot go beyond her's. Therefore, it is permissible for a man to wear shorts and short-sleeved shirts.
Conclusion: From a minimalist perspective, there is nothing wrong with wearing shorts or short-sleeved shirts. If you like wearing a black robe all the time, that is your perogative. It certainly does not constitute as immodest dress. However, do not tell me that my dress is immodest because halachically speaking, it isn't. During the normal week, I choose to dress modestly. For Shabbos and Yom Tov, I elevate my dress to a nice suit and a tie because I want to be able to elevate as much as possible for a chag, thereby showing reverence for a special occassion. This is a preferential approach that is, as I already stated, within the dictates of proper attire. As long as one keeps modest attire as a part of their self-presentation, what you wear beyond that is your choice, and I'll respect it.