The blaze started in the early hours of Wednesday morning local time in Block C of Tangerang Prison, in Banten province, according to Jakarta police spokesman Yusri Yunus.
Yusri said firefighters stopped the fire from spreading to other blocks in the prison and battled the flames for two hours before it was extinguished.
An investigation into the cause of the fire is ongoing but Yusri said they initially suspect an electrical short circuit is to blame.
Rika Aprianti, spokeswoman of the prison department of the law and human rights ministry, said authorities were still evacuating the facility and "the cause is under investigation."
The block housed inmates being held for drug-related offenses and had a capacity for 122 people, she said. She did not say how many people were present when the fire broke out, but confirmed the jail was overcrowded.
The prison in Tangerang, an industrial and manufacturing hub near Jakarta, housed more than 2,000 inmates, far more than its 600 people capacity, according to government data as of September.
Kompas TV showed footage of firefighters trying to put out huge flames from the top of a building. The broadcaster reported 41 people had died and eight had been seriously injured.
Indonesia's Metro TV cited a police report saying that 73 people also had light injuries.