Ihefu FC, which features in the National Second Division League (SDL), against all odds, put a spirited showing throughout the regular 90 minutes, which ended with the two teams level on 1-1.
In the penalty shootout, Mbeya City scored four penalties, whilst Ihefu FC netted five to progress to the third round of the tournament.
Ihefu FC's keeper, Andrew Kayuni, saved a free kick by one of Mbeya City's players from about 23 meters two minutes after kick off as the latter went in search of an early goal.
Striker Mohamed Mkopi put Mbeya City 1-0 up on the 29th minute, connecting home a free kick by midfielder Mohamed Samatta from outside the penalty area.
City pushed for more goals later in the half, only to be frustrated by Ihefu keeper, Kayuni.
In one of City's opening half exploits, striker Eliud Ambokile dribbled past two Ihefu defenders a few meters from the latter's area and unleashed a right foot shot which was saved by Kayuni.
The agile Kayuni, as well, palmed away a deep cross whipped in by City's right fullback John Kabanda, for an abortive corner kick.
Ihefu striker Ibrahim Kalenzo levelled matters for his team on the 50th minute, beating City's keeper Fikirini Bakari with a right foot shot inside the area.
City's defenders had failed to clear a cross from the right and Kalenzo made the most of the blunder.
The regular 90 minutes ended with the two teams level on 1-1 and the game had to be settled by a penalty shootout.
The day also saw Kiluvya United of Coast Region make it to the third round after cruising to a 3-2 victory over African Lyon of Dar es Salaam at the Uhuru Stadium.
Both teams are in group A of the 2017/18 National First Division League (FDL), which also consists of JKT Ruvu, Mshikamano, Ashanti United, Mvuvumwa FC and Friends Rangers.
In the event's other fixture, slated for the same day, Makanyagio FC took on Mbao FC of Mwanza, Shinyanga's Mwadui FC played Pepsi of Arusha at the former's venue, with Majimaji FC hosting New Generation in Songea.