Techniques are provided for foreground extraction from a point cloud (e.g., a LiDAR point cloud) using surface fitting. In an embodiment, one or more processors of a vehicle can receive point cloud data from one or more vehicle sensors. The one or more processors can identify points in the point cloud data as foreground points or ground points using a spatial Kalman filter to capture changes in the terrain. A route or trajectory in a driving area for the vehicle can be generated using the identified foreground points and ground points. A vehicle controller can control the vehicle while the vehicle is traveling on the route or trajectory in the driving area.