Born in the Bronx to Puerto Rican immigrants, Spanish was Haslip's first language and he learned to speak English in kindergarten. His father, James Joseph (Jaime) Haslip (1915–1999) served in the United States Customs Service, beginning as a Merchant Marine until moving to patrolman and eventually deputy commissioner, marrying Jimmy's mother Virginia (Viera) Haslip (1912-2009) in 1937.

Haslip moved to Huntington, New York when he was four years old. At age seven, he began playing drums and then moved onto other instruments such as trumpet and tuba until playing bass at age 15. Although he took music lessons and went to a private music school, he considers himself self-taught. He has said that he went to a local music shop with his father and purchased a right-handed bass (he is left-handed) and learned to play it upside down.

In an interview with magazine JazzTimes, Haslip said he was surrounded by music as a young boy, from visiting nightclubs and concert venues to his peers. He explains that there was music in his house as well, from his older brother Gabriel listening to classic jazz, his father listening to Latin and orchestra jazz and his aunt listening to "sappy stuff like Jerry Vale and Johnny Mathis". In high school, Haslip created his first band called Soul Mine with his high school classmates, playing soul music at school dances and parties.

In the early 1970s Haslip played with New York glam band Street Punk, he toured alongside musicians, and moved to Los Angeles in 1976, where he played with guitarists Tommy Bolin and Harvey Mandel. in addition to the Yellowjackets, he has worked with Jeff Lorber, Eric Marienthal, Bruce Hornsby, Rita Coolidge, Gino Vannelli, Kiss, Tommy Bolin, Allan Holdsworth, Marilyn Scott, Chaka Khan, Al Jarreau, Donald Fagen, and Anita Baker.

Haslip has released two solo albums: Arc, and Red Heat, with Joe Vannelli, in 2000. He remains active with the Yellowjackets, and was part of a combo with Allan Holdsworth, Alan Pasqua, and Chad Wackerman.[10] Other collaborations include Jing Chi (with Robben Ford and Vinnie Colaiuta) and Modereko.

Haslip was a member of the rock group Blackjack from 1979–1980 with Bruce Kulick, Sandy Gennaro, and Michael Bolton. He also toured with guitarist Allan Holdsworth and drummer Virgil Donati.

In 2012, Haslip took a year-long break—which later turned permanent—from the Yellowjackets to spend more time with his family and focus on other musical projects. He was replaced by the son of Jaco Pastorius, Felix. Regarding this topic, he stated: "The primary reason for my hiatus is so that I can spend more quality time with my family", says Haslip. "I spent ten months on the road last year. The break will give me an opportunity to spend more time at home as well as work on other artistic endeavors, such as independently producing projects". Haslip adds, "This was not a rash decision and there is no animosity by any means. Everybody is still friends and I'm excited for the band and their upcoming plans with Felix"