Disclosed herein are system, apparatus, article of manufacture, method, and/or computer program product embodiments for selection and alignment of video segments for adaptive streaming. An embodiment operates by receiving one or more variant segments, the one or more variant segments including a first variant segment having a first bitrate and a second variant segment having a second bitrate for decoding and playback and generating a first packetized elementary stream (PES) collection based on the first variant segment and a second PES collection based on the second variant segment. The embodiment further operates by storing the generated first PES collection and second PES collection in a processing buffer; and determining a first out-of-tolerance splicing location and associated first out-of-tolerance offset between the first PES collection and the second PES collection based on a maximum tolerance offset.