Location. Inn at the Black Olive is located in Baltimore's Inner Harbor - Downtown Baltimore neighborhood close to Pier Six Concert Pavilion American Visionary Art Museum and National Aquarium in Baltimore.