Glad to hear the initial setup was easy and went well.
I haven't had any control problems and wonder if those are signal issues. Let's see if it's an Alexa issue first. When you speak the Alexa commands does the device respond in the Dashboard or device list and the LED on the PLM flicker? If yes, then that eliminates Alexa as an issue. Does the light respond when controlling from the app? On (yes)? Off? Dim? If 'yes' to on and 'no' to off or dim then it sounds like a signal issue. Are these LED bulbs? I've had LED bulbs turn on ok but not off because they create noise when on. A hash choke at the switch or light can solve that or sometimes a different brand of bulb. See:
http://jvde.us/x10/x10_cfl.pdf for more info. The chokes can be ordered at ebay.to/39yWLYC
If you're using an LED bulb can you replace it with a 40W or higher incandescent bulb for testing? What type of switch/module are you using? The best way to test would be to plug an incandescent lamp into a lamp module at the same outlet as the 2412U PLM for testing. That would prove or disprove the noise issue. Do you have any signal bridges/repeaters like an XTB-IIR?
There's nothing special you have to set up for dimming (other than setting it up as an 'X10 Dimmer' when adding devices) so if it at least turns on then you've set it up correctly.
Not that it should matter for this issue but make sure you have the latest f/w update by tapping the arrow in the upper right corner of the Smartenit-Harmony P2 device icon and clicking 'Updates' and it'll tell you if you're up to date or there's a new one. The latest is 3.15.0-6. You can also set up auto updates.