Still I don't see an issue - basically you either pay the armed coast gard cutter that stands in your way or you don't go through the straight. If you don't cause any trouble, the other cutter on the other end will pay you back. No money, no transit - unless you really like being boarded.
Regardless of what specific rules could be set you have to consider rules of engagement and potential escalation. What happens if a Russian merchant vessel (either legitimately flagged or shadow fleet) refuses to cooperate? Do you use force to stop them? What if they're being escorted by a Russian warship or combat aircraft?
Put mines where? How do you prevent neutral vessels from hitting them? What happens when they inevitably break loose in a storm and drift away? Naval mines are quite effective for closing down a body of water in an unrestricted hot war but we haven't reached that stage yet. EU and NATO countries still want to be able to use the Baltic Sea and Gulf of Finland for their own purposes.
Huh? What are you even talking about? Which models specifically? And how sure are you that they can reliably discriminate between Russian vessels versus others that look and sound identical?
No, smart mines won't solve that. I can't fathom where this fantasy is coming from. It's totally disconnected from the reality of current mine technology.