Since he will still be remembering the existence of Nami and Buggy, just not how they met, it's not quite there for a 4 point memory loss. 4 point memory loss would be more along the lines of him forgetting Nami entirely.

However I think if you want to just keep it to him losing memories of these couple days, we could reduce the luck boost down to the 2-point level one. This means he will still have higher luck, he'd just get a +2 on quest rolls instead of a +4.

How does that sound?