Fix environment with force-purged modules

Also specify `torchvision` version. It's not perfect since it's still
prone to breakage, but at least we forbid known not working versions.

Thanks to @khalfaoui1 for reporting it.