Diego Simeone's side dominated the opening 45 minutes and the visitors had goalkeeper Yoel Rodriguez to thank for saving a Costa penalty on 25 minutes and pulling off a number of other saves to keep the home side at bay early on.
However, Atleti's dominance continued and on 42 minutes they took the lead when Costa bundled home from close range to put them 1-0 in front at the half-time interval.
The striker then doubled his and los Rojiblancos' lead in the just past the hour mark with a well taken finish and though Nolito pulled a goal back for Celta with 20 minutes remaining, the hosts held on to pick up a deserved victory.
The win sees Atleti draw level with Barcelona - who beat Valladolid 4-1 on Saturday - at the top of La Liga, with both sides continuing their 100% starts to the season.
The Copa del Rey holders are next in league action when they take on Espanyol on October 19 following the international break.