Many books can rightly justify things by the phrase “In my world” as in “In my world, people can readily wield swords 3x their size” or “In my world, men have next to no refractory period.” This is not an invalid form of writing – if we wanted to live in real life all the time, we’d get a second job.
The key to it, though, is twofold.
Number one, you need to make that “In My World” plausible enough that it becomes reasonable. It’s difficult to pull off something like “In my world, people can make a fortune overnight with nothing but their pure brilliance.” Yes, we can envision a world in which that might be true, but building a further mythology around it in which the world functions more or less like the normal one such that your Mary-Sue begins to make sense becomes challenging. Not impossible, but challenging to the point that it becomes a real flaw in your writing. After all, if getting rich were easy, as they say, everyone would do it. Once everyone did, where where would the resources be coming from to give everyone the lifestyle of the rich and famous?
Secondly, you must make certain that you establish that unlikely fact of your universe, either tacitly or explicitly. Final Fantasy makes it clear that the giant sword wielding is a thing simply by giving every-fucking-body a giant sword. We see them doing it, so we can only assume that’s the case. This is a tacit explication of that fact of their universe. We don’t see one guy who is otherwise normal doing it and no one else, leaving us to assume the author didn’t think things through, we see it universally such that it becomes apparent what is possible in that world. Conversely, you can make it such that your character specifically is, in your world, able to develop that level of strength. He can be the only one, but only in a world which allows that.
On the other side, you can make it explicit what is going on. For example, the Dune series is predicated on the idea that (ancient, dusty spoiler alert) the Fremen can isolate water into one central spring (or set of springs) and have that somehow improve their environment’s manifest lack of water. This at first makes no sense, but once it’s explained that free-roaming water is inevitably isolated by the sandfish and kept from the ecosystem, it becomes apparent that what the world needs is one big burst of water, not just a free rein (not to be confused with rain, which doesn’t exist In His World) of small bits. This is a world which is more or less the same as our own in principle, but which has creatures in it whose behavior changes things. In His World, there are sandworms that hate water.
In both of these cases, something is odd about that world which, to first glance, makes no sense. If we were just given that in isolation, it would be implausible, and it would ruin immersion and throw people out of the plot to a sufficient degree that it would function similarly to a plot hole.
Thus, we see two things. Firstly, your “In My World” must make sense, it must have a world built around it that embraces that fact and coherently addresses all the externalities of it. Secondly, it must be made clear enough to the reader that they understand your world is different than theirs and you are not simply an idiot.
In your world, things make sense, just… differently.