There are three major factors: magnitude, your distance from the fault, and local soil conditions. 
Although most of San Francisco escaped serious damage in Loma Prieta, those with unconsolidated landfill or soft soils (such as the Marina District) suffered serious damage. The ground motion in such areas was 10 times stronger than at neighboring sites on rock. 

[Information from: "Frequently Asked Earthquake Questions" - "Earthquakes - A Whole Lotta Shakin’ Goin’ On" - State of California - Department of Conservation]