How can I dress this down?

Try getting a nice, brightly colored belt or other colorful accessory that makes the outfit look less monochrome (and thus a little less fancy). Wearing a bright, casual, open button-down shirt works similarly.