I really think any game that mixes different mechanics - via minigames or other means - should have difficulty settings like System Shock. I'm good at shooting, but I'm too impatient for stealth? Make enemies look away in those sections. The other way around? Make me a tank in actions scenes, but let me fail when I'm spotted by eagle-eyed enemies.
But what actually bothers me more if those minigames kind of come out of the blue, and aren't really connected to the game in question. There are games where it works really well, like the Telltale games, TWD, Batman, Wolf, or Edith Finch then there are games where it distracts and annoys a bit, but might be a welcome change of pace like Bioshock, or the later Bloober Team games, and then there are games where it really feels out place - the earliest example for me being Lure of the Temptress with the annoying fight scenes, and later in part FF7 or Anachronox.
I think one problem is that people who are great designing something - like RPG mechanics - are not so great at designing an action game - but think they are. The next problem is that players thinking they bought a certain things are often not willing to get into another - and that's talking from experience - if you started a strategy game, you simply might not be up to shooting game, even if you are a decent action gamer otherwise. And lastly, often the engines aren't up to the task, making the minigames a slog - you're basic AGS game won't compete with R-Type, no matter how much the dev thinks that spaceship sequence is important for the story - hell no, it's bloody frustration and ugly crammed into that engine.
So the TL;DR is know what you, the dev, are doing, and give me, the player, a choice.