James Sylvester attended two primary schools in London, then his secondary schooling was at the Royal Institution in Liverpool. In 1833, he became a student at St. John's College, Cambridge. At this time it was necessary for a student to sign a religious oath to the Church of England before graduating. Sylvester, being Jewish, refused to take the oath, so could not graduate.

From 1838 Sylvester taught physics for 3 years at the University of London, one of the few places which did not bar him because of his religion. His former teacher De Morgan was one of his colleagues. At the age of 27 he was appointed to a chair in the University of Virginia but he resigned after a few months. A student who had been reading a newspaper in one of Sylvester's lectures insulted him and Sylvester struck him with a sword stick. The student collapsed in shock and Sylvester thought that he had killed him. He fled to New York boarding the first available ship back to England.

On his return Sylvester worked as an actuary, lawyer, and math tutor. His pupils included Florence Nightingale. By good fortune, Cayley was also a lawyer. The two of them discussed mathematics as they walked around the courts, and they became life long friends. Sylvester tried hard to return to being a professional mathematician, and eventually he became professor of mathematics at Woolwich.

Sylvester did important work on matrix theory. In 1851 he discovered the discriminant of a cubic equation and first used the name "discriminant" for such expressions of quadratic equations and those of higher order. He used matrix theory to study higher dimensional geometry. He also contributed to the creation of the theory of elementary divisors of lambda matrices.

Sylvester became the second president of the London Mathematical Society, after De Morgan. He was the first recipient of the gold medal which the Society awarded to honour De Morgan.

Being at a military academy, Sylvester had to retire at age 55. At first it looked as though he might give up mathematics since he published his only book at this time and it was on poetry. For 3 years Sylvester appears to have done no mathematical research, but then Chebyshev visited London and the two had many discussions. He lectured on the topic, and this got Kempe interested in the topic. Kempe and Sylvester worked jointly on linkages and made important discoveries.

In 1877, Sylvester accepted a chair at Johns Hopkins University, and founded in 1878 the American Journal of Mathematics, the first mathematical journal in the United States. In 1883, Sylvester was appointed to the Savilian chair of Geometry at Oxford. However, he only liked to lecture on his own research and this was not well liked at Oxford where students wanted only to do well in examinations. In 1892, he retired.