Remix Round Five: Wrap Dress

I almost went two weeks without a remix!  Argh!  So sorry about that; I've been feeling a bit sluggish and fat these days and didn't really want to try on or photograph any clothes--I'm sure you understand.  It's that odd weather time of year where it looks like fall, but occasionally still gets pretty warm.  I'm yearning to wear nothing but sweaters, but then I'd sweat constantly.  Basically, this is a tough time for me, wardrobewise.

I came up with an experimental outfit that I'm quite on the fence about, so would appreciate any and all feedback my lovely readers have for me.

Back Story:
I hate wearing button-down shirts.  I find them uncomfortable because the collars always seem to sit really high on me, they gap in the front, they always ride up my waist, and it always seems like small is slightly too small, but medium is huge and shapeless.  The problem is, I love to wear sweatervests, and I love the way a button-down looks under a v-neck sweater, and I feel like if I'm going to be "professional" I have to own some of these damn shirts, so I keep buying the damn things and then never wear them.

This morning, I was rooting through my closet and I espied these two, short-sleeved button down shirts I bought last year.

Another problem is that I always get really excited and buy two
of everything, and then when I discover the item is imperfect for me
I have two of it.

When I wear these shirts (which is rarely--I think I've worn each once), I usually put a cami underneath and leave the top button undone.  This solves the collar and the gaping problem, but may look a bit sexier than I want to at the public library.  So I thought I might try putting one of these shirts under a basic black wrap dress that I have.

Here's how I usually wear the wrap dress:

Nude or fun-colored cami underneath--easy peasy

Here it is with the silver button-down:

Gah! That collar is grazing my ears--so uncomfortable!

It looks a bit like a romper, which I don't mind, but I also feel like it makes me look a bit fat, though that may be the pizza I had last night and not the outfit.

What's the verdict on this one?  Weird?  Good?  I should just stop trying with the button-downs and let the dream die?