Unfortunately, it's not just the two of them.
In general, I always start with simplest ("most readable" in the sense of easiest to understand) implementation that I have the time to implement (time is another variable).
If a performant version is a no-brainer in terms of cost to implement, and at worst only slightly reduces readability, I'd probably go with that just because I like to be proud of my code (satisfaction in a job is yet another variable).
Ultimately, I rarely find that readability suffers with performant code.
Unreadable code is usually non-performant because it is hard to improve.
Focus on performance sometimes simply reduces how reusable/generic code is, but I don't find readable and performant to be in opposition at all.
Then profile your code and determine where to rewrite for performance.
In my experience, big performance optimizations are usually small changes and don't affect readability much.
(Systems-level C/C++ can be exceptions to the above, of course.)