Swiss No campaign poster against banning firearms
It means that the voters decided during the referendum to retain the current system, which allows army-issued weapons to be kept at home.
Supporters of the tighter curbs wanted to have weapons kept in armouries and demanded stricter checks on gun owners.
Opponents said the move would undermine trust in the army. The final result of the vote is expected soon.
Near-complete results show at least 14 out of 26 Swiss cantons rejected the proposal in Sunday's vote.
National institution
For the proposal to succeed, it required the support of the majority of Swiss citizens and the cantons.
Overall, about 57% of the Swiss voters rejected the plan, while 43% backed it.
Geneva and Basel both bucked the trend by approving it, according to the Swissinfo website.
The result is a blow to gun-control groups in Switzerland, the BBC's Imogen Foulkes in Geneva reports.
There are an estimated two to three million guns circulating in Switzerland.
No-one knows the exact number because there is no national firearms register, our correspondent adds.
In addition to the semi-automatic assault rifle that all those serving in the army store at home, there are thousands of hunting rifles and pistols.
Serving and former soldiers have been allowed to keep their weapons at home since World War II.
The proposal to end that custom was backed by a coalition of doctors, women's groups and police associations.
Although Switzerland's overall crime rate is low by European standards, the country has the highest rate of gun suicide in Europe.
The proposal's backers had argued that keeping soldiers' firearms locked up in armouries would reduce the suicide rate.
A number of high-profile killings in recent years - such as the shooting of ski star Corinne Rey-Bellet by her estranged husband in 2006 - have also boosted support for greater gun control.
But the Swiss army is a national institution, and changing anything about it is controversial, says our correspondent.
Opponents of the proposals say taking soldiers' guns away would undermine the military and could open the door to abolishing Switzerland's citizen army all together.
"If the 'Yes' goes through, it really risks destroying the country," Xavier Schwitzguebel, an army reservist officer, was earlier quoted as saying by the AFP news agency.
"If we take away the weapon, which represents this trust, that means that we are breaking the sacred union between democracy and citizen."
Anyone familiar with the Swiss would realize that they have the largest citizen army in the world. They are neutral, but they are well equipped to keep the wolves at bay. I wish more countries were like this.
It is well established in countries that have strict gun control that it does not prevent suicide. People will jump from bridges, slash themselves or find one of a thousand other ways to do it if they choose to.
All that gun control really accomplishes is that when you are in harms way, and you really need to protect yourself, or your family, you have nothing. Sure, you can call the police, and in 4 or 5 hours they might show up to arrest you for defending yourself.
I for one applaud the Swiss for doing what ANY rational and free country would do.