A pointing controller enables interaction with virtual objects in a virtual or augmented reality environment. The pointing controller comprises a form factor that enables it to be securely positioned in the user's fingers during a pointing gesture. The pointing controller tracks a pointing direction in three-dimensional space based on motion of the pointing controller. The pointing controller furthermore includes one or more control elements such as buttons or touch interfaces to enable additional interactions.