An electronic device such as a handheld device may have a rectangular housing with a rectangular periphery. A conductive peripheral housing member may run along the rectangular periphery and may surround the rectangular housing. Radio-frequency transceiver circuitry within the electronic device may be coupled to antenna structures for transmitting and receiving radio-frequency signals. The conductive peripheral housing member may form part of the antenna structures. A gap in the conductive peripheral housing member may be filled with dielectric. The conductive peripheral housing member may be configured to form a recess. The recess may have the shape of a rectangle, oval, diamond, or other shape that overlaps and is bisected by the gap. The recess may also have the shape of a groove that extends around the entire periphery of the housing. The dielectric in the recess may include one or more different materials such as clear and opaque polymers.