Does anyone knows how to do that?
This is default behavior for indirect dependencies.
If project P4 doesn't actually contain any code that explicitly uses reference project P3 and nuget package N1, VS/MSBuild doesn't detect that P3, N1 is required by P2, so MSBuild does not think its necessary to copy the P3, N1 to the output folder of P2. That is the reason why only add P4 and not the dependencies (P3 and N1) into P2's bin directory.
To resolve this issue, you can directly reference P3, N1 to project P2 or add a copy command line in the P2`s build event to copy those dll to the bin folder.
Besides, you can also add dummy code to a file in project P2 that uses project reference P3, N1.
See MSBuild doesn't copy references (DLL files) if using project dependencies in solution for some more details.
Hope this helps.