The BJP won 99 of the 182 assembly seats available in the key state election, with the main opposition Congress Party claiming 77 seats, according to election authorities.
In 2012, the BJP won 115 seats while Congress secured 61.
"This is a positive message for the Congress. They have been able to increase their vote share, their seat tally has gone up and the notion of invincibility of the BJP, that no party can defeat them, that has not broken but at least been punctured," Sanjay Kumar, Director of the Center for the Study of Developing Societies, told CNN.
The BJP has controlled Gujarat since 1995, with Modi ruling the western Indian state for more than a decade before becoming India's leader.
In a tweet, Modi thanked voters and vowed to continue to serve them.
Monday's result marks the BJP's worst performance at a state election in Gujarat in more than two decades, with analysts pointing to widespread dissatisfaction over Modi's recent economic policies as a primary reason for the drop in poll numbers.
"The BJP has come through, but it's definitely the opposition that's going to see this as a small victory," Neelanjan Sircar, Senior Fellow at the Delhi-based Center for Public Policy Research, told CNN. "Modi came in with his motto, 'I am development, I am Gujarat,' and they claimed they were going to win 150 seats. That was just a few months ago so it shows you the ground that was made up by the opposition."
Modi, who swept to office in 2014 with promises to boost income levels, has seen his record challenged in recent months as a slowdown in the country's economy begins to bite.
Although growth in the most recent quarter recovered to 6.3% after slumping to a three-year low, it remains below levels seen last year. This leaves India trailing China, which grew at 6.8% over the same period.
Many blame the slowdown on two of Modi's signature policies: last year's sudden ban on high-value currency notes, and the introduction, earlier this year, of a new national goods and services tax to replace the patchwork of local levies. Many small businesses in states like Gujarat have struggled to adapt to the new systems, fueling anger against the BJP.
Meanwhile, in the northern state of Himachal Pradesh, which held assembly elections on November 9, a total of 68 seats were up for grabs. The BJP won 44 seats while the opposition Congress secured 21.
Next year will see polls take place in Karnataka in the south; Mizoram, Meghalaya, Nagaland and Tripura in the northeast; Madhya Pradesh and Chhattisgarh in the country's center; and Rajasthan -- a state that has had alternating governments since 1990 and is currently ruled by the BJP -- in the west.
State elections are often seen as key political tests for the ruling party, with the next general election scheduled for 2019.
"For the rural poor, the working class, the BJP hasn't been working for them. This is a narrative that, if it's felt strongly in Gujarat, it will be felt strongly in Rajasthan, Madhya Pradesh and Chhattisgarh," Sircar said. "So in that sense the Congress really has an opening because this sort of class-based disenchantment is something they can really define its political identity around."
Monday's results come as Rahul Gandhi takes over as leader of the Congress Party. Gandhi, the scion of India's most influential political dynasty -- the son, grandson and great-grandson of former Prime Ministers, officially took over as leader of the opposition on December 16, replacing his mother, Sonia.
"My Congress brothers and sisters, you have made me very proud. You are different than those you fought because you fought anger with dignity. You have demonstrated to everyone that the Congress's greatest strength is its decency and courage," Gandhi wrote in a later tweet.
The numbers out of Gujarat and Himachal Pradesh are likely to provide a much-needed boost to the Congress Party, and help position Gandhi as a credible alternative to Modi on the national stage in the run-up to 2019.