For one, it needs to be kept in mind that shirt dresses come in all shapes and styles. There is no limit as to what such a dress can look like. There can be variation in fabric, in how the dress is stitched and even in colors. This means that you could buy a light color and get it stitched in a very formal way. There is no way such a dress could look casual!