An arithmetic processing apparatus includes: an instruction controller; a first level cache and a second level cache. The instruction controller, for a memory access instruction to be speculatively executed that is executed while a branch destination of a branch instruction is undetermined, adds a valid speculation flag and an instruction identifier of the branch instruction to the memory access instruction and issues to the first level cache. The first level cache controller interrupts execution of the memory access instruction when a virtual address of the memory access instruction hits in a TLB of the first level cache, the speculation flag of the memory access instruction is valid and an entry having a virtual address matching the virtual address of the memory access instruction stores a speculative access prohibition flag prohibiting speculative access.