Talk to him, as soon as you can! You need to be clear and understand what happened. Was it an accident… does he think you’re using it too early or too much? Whatever it is, it’s a breach of trust to ignore it so you need to talk and get that sorted.
But, how often are you actually having to use safewords? In my opinion they should be pretty rare, it’s the doms job to learn from each one so they can better manage scenes as time goes on. Occasionally I’ll push someone to use it because I know it’s what they need but that’s very seldom.
And when you do use it, it has to be responded to immediately, for safety’s sake if nothing else.
This is why i like to use the yellow and red combo, because yellow is there to just flag up you’re close, so they can ideally not get to red. If you are using both, well red should happen virtually never.
So yes, safe, sane and consensual relies on safewords being agreed and respected, don’t shirk talking about it, you both need to be clear about what’s going on.