It sounds like what you're really saying is that you want aggressive players to get so annoyed by AI threats that they eventually just leave instead of taking on other players. Is the reason players are aggro towards other crews in Adventure JUST because they crave PVP? If that was the case they'd just play Arena. The thrill of PVP combat in Adventure is the mystery of what the other crew is going to do/how they will react/what treasure they may or may not have. They don't just want any old thing to shoot at.
But honestly, how would the game accurately judge whether the actions of some players are over the line. Just sinks alone? Repeated sinks? Let's say a player in a sloop repeatedly rolls up on a crew and fires on them or spams text chat and that crew kills/sinks the guy multiple times but he keeps coming back for whatever reason. (This has happened to me, btw, with a guy who claimed he 'just wanted to give us his cursed balls' and sailed up to us 3x times) Is that crew now labeled as the griefer cuz they sank the same person 3 times? Let's say putting a bounty on a crew is a user-initiated action - the guy who got sunk 3x can now label the crew that did it as griefers even if they weren't the initiator?
What you'd really be doing is encouraging a war of attrition where you can be labeled as the bad guy by a crew who got repeatedly sunk, or passive/reverse griefing where the end goal is making things more difficult for another crew instead of settling things with cannons.
What makes SoT special is that you cannot anticipate exactly what another crew is going to do (unless of course they are black hull, black sails and jolly roger or reaper's mark) or their skill when they roll up to you. You absolutely have to be watching out for other crews and expect treachery. I recently had a crew roll up to us and claim friendly, want an alliance and then shot at us, claiming they were "just kidding." Because they were streaming on mixer I later watched their VOD and their whole plan as they were heading towards us was to pretend to be friendly and sink us. The guy came back once more afterward and we sank him again. If the game had the bounty system you proposed the game could determine US as aggressive players (the guy was a horrible cannoneer and the new damage system allowed me to anchor/sink them fairly easily).
First of all, no I have no intention of discouraging PVP, and this system isn't about that (but yes, this was the tenor of many posts). First and foremost, if you're engaging in PVP with crews at your level, you may never even notice this system. If you regularly die by other players, your bounty/aggressiveness will never appreciably go up, even if you're being aggressive. However, if you're aggressive towards one crew over and over and successfully getting away with it, I think it's fair to ramp that up to make it physically more difficult to do. In no way do I want normal PVP to be less fun or rewarding, or to take away that feeling of never knowing what the other party will do. In fact, I would suggest that for many crews, this system actually encourages light PVP by rewarding you with a more intense and full world with all the associated benefits of that. Sure, some players want to avoid PVE altogether and would hate this system that would "annoy" them with PVE. I don't know why would elevate their preferences over the people who want to avoid PVP altogether. I think this system forces a natural balance where you HAVE to take breaks if you're too aggressive. (There's actually also the opposite problem of people farming the system - which, I'm actually ok with - it's expanding the sandbox and encouraging both PVP and PVE).
How do you measure? How do you measure in Arena? We know where every cannonball comes from. We can track every strike and shot. We know when you steal another player's treasure, we know when you just drop it off the back in transit. If you want to hang out on an enemy ship and threaten them with your presence alone, I am 100% for that and the system would never spike. If you want to wait until they finally trust you and leave the ship to steal 3 moderately valuable chests, I am 100% for that. The system would barely notice this. What if you stole, say an Athena's or 30 chests? Well, then, the Sea of Thieves would get interested in you, wouldn't it? Maybe you'll get surprised by a skelly ship - NOT to actively prevent the turn in, just to make the journey back a little bit more exciting. I'm 100% ok with this, and I would never want to discourage this.
Unless.. you were to gank more crews straight after this. If you and your crew manage to do it again successfully and without dying more than once or twice, I'm not mad at you. But I do want to challenge you more. Not sink you. Not ban you. Not shame you. That's even true if you maliciously go back and sink the other ship repeatedly. I want to use the tools of the world and sandbox to force you to consider how you're playing and adjust meaningfully on the fly. Maybe you learn to take more breaks in between where you work off your bounty, so to speak. Maybe you learn to incite PVP and learn to avoid the same crews just enough to get the additional spawns you want without being overwhelmed.
In theory, this is all a pirate game whatever blah blah, but in practice, it's super frustrating because the troll crew risks nearly nothing. They have no treasure to lose. They have all the resources to win a fight after a respawn. Maybe they're exactly the crew that is bored with the random nature of the PVE simulation. That doesn't feel fun. I won't pretend this is a perfect system, but I do think it's time to implement ideas that work within the sandbox that Rare has created to guide gameplay that is fun for everyone. I agree, not knowing the intentions and actions of other players is fun. Griefing is the opposite of that, isn't it? I know EXACTLY what that ship is doing every time they pop up on the horizon, and we all know it makes a lot of players shut off their game and leave. We don't want that.