board short] or infamous [speedo] "dude" here is a middle ground that will suit you well. They are very similar to the boxer brief.
The speedo is such a harsh look, too close to a thong for comfort and it forces to expose the male camel toe. The board short has been worn by the majority, but it can be unflattering as well because there is so much fabric and they are baggy.
Concluding, the brief swim trunk is the way to go for a tailored, classy look.