An accessory device architecture is described. In one or more implementations, data is received from an accessory device at an intermediate processor of a computing device, the data usable to enumerate functionality of the accessory device for operation as part of a computing device that includes the intermediate processor. The data is passed by the intermediate processor to an operating system executed on processor of the computing device to enumerate the functionality of the accessory device as part of the intermediate processor.