Academy of Music was designed by Thomas W. Lamb and built in 1927 across the street from the site of an earlier venue of the same name. Opened as a deluxe movie palace by movie mogul William Fox, the Academy operated as a movie theater through the early 1970s.

Beginning in 1971, it was also utilized as a rock concert venue, sometimes referred to as Howard Stein's Academy of Music, to fill a gap left by the closure of the Fillmore East. It was rechristened The Palladium in 1976, and continued to serve as a concert hall into the following decade and a nightclub run by Steve Rubell and Ian Schrager, of Studio 54 fame.