Elevator? Going... down?

The buildings that I work in are respectively, 16 and 66 floors high. There are a lot of elevators, and I've used them all. I get the "going up/down?" question a lot. And yet, I get an odd urge to laugh whenever someone says "going down?" to me (especially when they're attractive business attired guys, as just happened). I think it was the video for 'love in an elevator' by aerosmith that did this to me...