Did it work and scale with amd_iommu turned off?
关闭 amd_iommu 后，它是否有效并能够扩展？

Hello @abchauhan – How do I receive an update for this issue? I tried sending email to PSIRT@nvidia.com about issue 3902559, but haven’t hear back yet. Is there a portal I can sign in to see the trace of updates?
你好@abchauhan - 如何获取此问题的更新？我尝试向 PSIRT@nvidia.com 发送电子邮件报告问题 3902559，但尚未收到回复。是否有我可以登录以查看更新追踪的门户？

Thanks 谢谢

@abchauhan This is a very serious issue and has already been reproduced by a number of people: Problems With RTX4090 MultiGPU and AMD vs Intel vs RTX6000Ada or RTX3090 | Puget Systems 75
@abchauhan 这是一个非常严重的问题，已经被多人复现：RTX4090 多 GPU 和 AMD vs Intel vs RTX6000Ada 或 RTX3090 的问题 | Puget Systems75

It seems new A6000 Ada cards are also affected on AMD CPUs. I have ordered 8 such cards, so I hope this will be fixed soon.
看起来新的 A6000 Ada 卡在 AMD 处理器上也受到影响。我订购了 8 张这样的卡，所以希望这个问题能尽快得到解决。

I was able to reproduce with my 4090s too, on AMD EPYC. The only thing that prevents hang is NCCL_P2P_DISABLE=1, but the performance is subpar.
我使用 AMD EPYC 的 4090 也可以复现，唯一防止挂起的方法是设置 NCCL_P2P_DISABLE=1，但性能不佳。

We can not confirm that the RTX 6000 Ada GPUs have this problem on AMD EPYC or WRX80 based CPUs. P2P copy has not to be disabled when using RTX 6000 Ada GPUs.
我们无法确认 RTX 6000 Ada GPU 在 AMD EPYC 或 WRX80 基于的 CPU 上是否存在此问题。使用 RTX 6000 Ada GPU 时，不需要禁用 P2P 复制。

More important: the transfered data is correct. Multi RTX 6000 Ada setups seem to work without problems.
更重要的是：传输的数据是正确的。多块 RTX 6000 Ada 配置似乎可以无问题地工作。

Some findings for the multi RTX 4090 setups:
多块 RTX 4090 的设置的一些发现：

• When disable P2P copy with NCCL_P2P_DISABLE on AMD EPYC/WRX80 the locking problem can be by-passed, but then the transfered data between the GPUs is not copied correct! (destination data is all 0 or all NaN). This can be tested with for example:
  当在 AMD EPYC/WRX80 上禁用 NCCL_P2P_DISABLE 的 P2P 复制时，可以绕过锁定问题，但此时 GPU 之间的传输数据不会被正确复制！（目标数据全为 0 或全为 NaN）。这可以通过例如以下方式进行测试：

• The multi GPU RTX 4090 problem is not specific to AMD CPUs on the Intel CPUs we tested (for example XEON Silver 4309Y) the transfer is not blocked (NCCL_P2P_DISABLE has no effect) but the data is also not copied correct (destination all 0 or NaN)! This is independed of if NCCL_P2P_DISABLE is set or not (which of course should have no effect, as above example uses directly CUDA and not the higher level NCCL library).
  多 GPU RTX 4090 问题并非仅限于 AMD CPU，我们在测试的 Intel CPU（例如 XEON Silver 4309Y）上发现传输并未阻塞（NCCL_P2P_DISABLE 无效），但数据也没有正确复制（目标全为 0 或 NaN）！这与是否设置 NCCL_P2P_DISABLE 无关（当然，这应该没有影响，因为上述示例直接使用 CUDA 而不是高级的 NCCL 库）。

The RTX 4090 is currently not useable for multi GPU usage, neither on Intel nor AMD. The reason from our analysis seems to be a broken? CUDA UVA implementation.
RTX 4090 目前在 Intel 或 AMD 平台上都无法用于多 GPU 使用，根据我们的分析，原因似乎是损坏的 CUDA UVA 实现。

Is this a hardware or software/firmware issue?
这是硬件问题还是软件/固件问题？

I have just run into this issue as well using a 2X 4090 setup with i9-10980XE CPU. I too have been using the simpleP2P test which is broken when p2p is enabled. Doing a bit of hacking, it does appear that cudaMemcpyPeer works as expected. In my case the cudaMemcpyPeer gave 10.54GB/s vs 12.5 Gb/s when the p2paccess is enabled.
我也有同样的问题，使用的是 2X 4090 配置和 i9-10980XE CPU。我也一直在使用 simpleP2P 测试，当启用 p2p 时它会损坏。进行一些破解后，看起来 cudaMemcpyPeer 按预期工作。在我的情况下，cudaMemcpyPeer 提供的是 10.54GB/s，而启用 p2paccess 时是 12.5GB/s。

@abchauhan Any idea on when we should expect a fix? Can also confirm the same issue with p2p on a multi RTX 4090 setup with AMD WRX80.
@abchauhan 你有什么想法，我们什么时候可以期待修复？我还可以确认，在带有 AMD WRX80 的多 RTX 4090 设置上存在相同的问题。

@abchauhan Did the Engineering team provide any reasoning as to why P2P is not supported on the 4090?
@abchauhan 工程团队有没有提供为什么不支持 4090 上的 P2P 的解释？

Are there any plans to add support for P2P in the future?
未来有计划增加对 P2P 的支持吗？

The 3090 also does not support P2P, which is not a problem. It can give the same results about simple P2P as 3090, I think it is no problem
3090 也不支持 P2P，这倒不是问题，我认为它在简单的 P2P 方面可以给出同样的结果，没有问题。

Th UVA not working is a big problem for an application I’m working on too. After debugging it for several days I posted about it just a few days ago when I tracked it down to UVA failing in a simple test program.
对于我正在开发的应用程序来说，UVA 无法正常工作是一个大问题。经过几天的调试，我前几天在追踪到 UVA 在一个简单的测试程序中失败时，刚刚发布了相关帖子。

I tested it and it works on Windows 11 with the 528 driver, but I’m running into other problems with that. I’m keeping my fingers crossed that the next Linux driver release will resolve the problem. It would be great if we could get some feedback from Nvidia about what to expect on that though.
我测试了，它在 Windows 11 上与 528 驱动程序一起工作，但我遇到了其他问题。我希望下一个 Linux 驱动程序发布能解决这个问题。然而，如果能从 Nvidia 那里得到一些关于此的反馈，那就太好了。

I can confirm that dual 4090s work on Windows 10 too with pytorch (latest driver 528.49, CUDA 12.0), it however freezes on Ubuntu 22.04 (latest driver 525.89.02, CUDA 12.0). So this is not a hardware limitation, but a driver/software issue. Please fix asap. Thank you!
我确认在 Windows 10 上，使用 pytorch（最新驱动 528.49，CUDA 12.0）的两个 4090 也可以工作，但在 Ubuntu 22.04 上会冻结（最新驱动 525.89.02，CUDA 12.0）。所以这不是硬件限制，而是驱动程序/软件问题。请尽快修复，谢谢！

Hello @abchauhan-- Do you have a target dates for fix of the issue? There are many modules and packages out there that leverage DDP and are currently broken for multi-gpu use. Thank you!
你好@abchauhan--这个问题的修复有目标日期吗？有很多模块和包利用了 DDP，目前在多 GPU 使用中都出现了问题。谢谢！

Hi, @abchauhan 你好，@abchauhan
I’ve upgrade to the latest cuda 12.1 with driver 530.30.02.
我已升级到最新的 cuda 12.1 和驱动程序 530.30.02。
simpleP2P still failed with report that checking P2P is YES.
简单 P2P 仍然失败，报告显示 P2P 检查为是。

is there an estimate of when the fix will be available?
修复程序预计何时可用？

Thanks for the update @abchauhan
感谢更新@abchauhan

I understand that P2P is not and won’t be supported on Ada GeForce cards. Do you know if the problems with Unified memory will be fixed with this release? Or I guess if that is even supported? It works for me with two 4080’s on Windows and I don’t see anything in the documentation to suggest it shouldn’t work. I’m assuming it should be supported, but someone correct me if I’m wrong.
我明白 P2P 在 Ada GeForce 卡上不支持，也不会支持。你知道这次发布是否会修复统一内存的问题吗？或者我想知道这是否 even 支持？在 Windows 上，我使用两块 4080 时它可以工作，我在文档中没有看到任何表明它不应该工作的地方。我假设它应该被支持，但如果我错了，请有人纠正我。

Indeed, it would be realy appreciated if NVIDIA communicates if multi-GPU setups with Geforce GPUs are officially supported or not.
确实，如果 NVIDIA 能沟通一下 GeForce 显卡的多 GPU 配置是否官方支持，那将非常感谢。
This we have introduced a mysteries bug which make GPUs stall or calculate wrong values and we don’t know if it is solvable situation is unprofessional.
我们引入了一个神秘的 bug，它会导致 GPU 挂起或计算出错误的值，我们不知道这种情况是否可以解决，这很不专业。

I take it that the new 530.41.03 driver doesn’t solve this problem yet?
我猜新的 530.41.03 驱动程序还没有解决这个问题？

From the tone of the discussion it seems 4090s will have P2P locked. In my case, it renders the 4090 worthless.
从讨论的语气来看，4090 似乎会锁定 P2P。在我这种情况下，4090 就变得毫无价值了。

I ended up having an internal debate between A100 40G / A5500 24G x2 + opted for 2x A5500. It is in the ballpark of a pair of 4090s + it packs the performance of the A100 40G that, I have a feeling we both may have been aiming for.
我最终在 A100 40G/A5500 24G x2 之间进行了内部辩论，选择了 2x A5500。它的性能大约相当于一对 4090，并且包含了 A100 40G 的性能，我感觉这可能是我们共同的目标。

A consideration if you are able to make the hardware switch.
如果你能进行硬件切换的话，这是一个需要考虑的因素。

For HPC we’re stuck with using appropriate hardware. Not necessarily a bad thing. Tools tend to be more functional than toys.
对于 HPC，我们 stuck with 使用合适的硬件。不一定是坏事。工具往往比玩具更实用。

An A5500 is nearly as fast as a RTX 3090 … for 4090 performance one has to go for a RTX 6000 Ada.
A5500 的性能几乎和 RTX 3090 一样快……要达到 4090 的性能，就需要选择 RTX 6000 Ada。

That’s nice and all, but it’s useless. 24G of ram is nothing. 5500 has P2P/SLI 3090 has neither.
挺好的，但是没用，24GB 的内存根本不够。5500 有 P2P/SLI，3090 都没有。

With the poor capabilities of the 4090 it’s worthless as anything but a toy. I returned 2 of them. At least I can use 100% of an A5500.
4090 的能力很差，除了当玩具外毫无价值，我已经退回了两台。至少我可以使用 A5500 的全部性能。

Sure, may be faster but if it’s all blocked from use what good does the speed at playing games do for training and inference?
当然，可能会更快，但如果所有东西都被阻止使用，那么游戏中的速度对训练和推理有什么好处呢？

It did nothing for me. Gave me gibberish & unreliable results. Seems a few of us in this thread have shared experiences.
它对我来说毫无作用。给我乱码和不可靠的结果。似乎我们这个帖子中有几个人分享了类似的经历。

We can confirm that with driver 525.105.17 the locking problem is fixed and UVA (Unified Virtual Adressing) also seems to work with 2x RTX 4090 without error on AMD EPYC and Threadripper. With working UVA also NCCL works → multi GPU training with Pytorch and Tensorflow work.
我们确认，使用驱动程序 525.105.17，锁定问题已得到修复，同时在 AMD EPYC 和 Threadripper 上，UVA（统一虚拟地址）似乎也能在 2 张 RTX 4090 上正常工作而不会出错。随着 UVA 的正常工作，NCCL 也能够运行，从而实现 Pytorch 和 Tensorflow 的多 GPU 训练。

Finally 2x RTX 4090 can be used for deep learning training, the missing P2P performance probably hinders scaling with more than two RTX 4090 because of bad all gather and all reduce performance… but at least now they are useable.
终于可以使用 2 张 RTX 4090 进行深度学习训练了，缺失的 P2P 性能可能由于较差的全收集和全减少性能，在使用超过两张 RTX 4090 时会阻碍扩展...但至少现在它们可以使用了。

I said that it is useable (it does not crash or hang anymore) and it calculates correct results (which was also not the case in older versions). I did not say that it is good or fast and already doubted that it makes sense to use more than two RTX 4090s. What are your experiences with the performance?
我说过它是可用的（不再崩溃或挂起），并且它可以计算出正确的结果（这在旧版本中也不是这样）。我并没有说它好或快，而且我早就怀疑使用超过两个 RTX 4090 是否有意义。你对性能有什么经验？

Can you explain why it’s not good?
你能解释一下为什么不好吗？

The missing P2P performance probably hinders scaling with more than two RTX 4090 because of bad all gather and all reduce performance.
丢失的 P2P 性能可能由于糟糕的全收集和全减少性能，阻碍了使用两个以上 RTX 4090 进行扩展。

The missing functionality makes it as slow as CPU.
缺失的功能使其速度慢如 CPU。

If you have the $ for 2x 4090, you should pick up 2x A5500 instead. The memory > the processing for inference.
如果你有 2 个 4090 的钱，你应该选择 2 个 A5500 instead。内存比推理处理更重要。

20230314_1022461868×2314 353 KB 1868x2314 353 KB

20230501_0920041868×4000 783 KB

The After pic destroys the performance of 2x4090. Not even close.
2x4090 的性能被 After 图摧毁了，完全不在一个档次。

You need a completely un-hobbled GPU, including p2p to do anything. Unless you are using multiple systems and building your own p2p, 4090 is a total waste of time.
你需要一个完全未被限制的 GPU，包括点对点通信，才能做任何事情。除非你正在使用多个系统并构建自己的点对点通信，否则 4090 完全是浪费时间。

• note on the bracket: I changed cases later + bracket is no longer necessary. Old case needed it. LianLi O11XL fits everything without issue.
  括号内的注释：我后来改了案例+括号不再需要。旧案例需要它。LianLi O11XL 可以无问题地容纳所有东西。

Has there been any progress on this?
这个有什么进展吗？

I can get 2x 4090s working via NCCL_P2P_DISABLE=1. But moving to 3x 4090s barely improves throughput over 2x 4090s.
我可以通过 NCCL_P2P_DISABLE=1 使两个 4090 正常工作。但是，移动到 3 个 4090 几乎不会比 2 个 4090 提高吞吐量。

Hi, 你好，

This is really interesting, I can confirm that running different training jobs we are having the same issues described here.
这真的很有趣，我可以确认运行不同的训练作业时，我们遇到了这里描述的相同问题。

One thing I noticed is that when I train for example a LLM models or fine-tuning them, it scale fine between GPUs using multi GPUs 4090, we have tested with 4x GPUs, but when running a vision model like resnet50 the system crash or lose performance, for example: It start the resnet50 training and after 2 minutes performance goes down like 80%, the GPUs are being used at 100% at the beginning using all the power of the GPU and then after 2 minutes wattage consumption goes to 120w or less, and this is not related to hardware since we test multiple system and scenarios, it seems that is all related to the P2P being locked on Gforce cards.
我注意到的一点是，当我在例如训练一个LLM模型或者进行微调时，使用多 GPU（4090）进行扩展可以很好地在 GPU 之间进行，我们已经用 4 块 GPU 进行过测试。但是，当运行像 resnet50 这样的视觉模型时，系统会崩溃或者性能下降，例如：它开始训练 resnet50，然后在 2 分钟后性能下降了大约 80%。GPU 在开始时的使用率是 100%，充分利用了 GPU 的性能，然后在 2 分钟后，功耗下降到 120w 或更低。这与硬件无关，因为我们已经在多个系统和场景中进行了测试，似乎所有问题都与 GeForce 显卡上的 P2P 被锁定有关。

Can you please @abchauhan let us know if this is permanent and what it means for workstations then with multi GPUs, so we cannot train DL models using multi 4090s and we must buy Quadro GPUs?
请问@abchauhan，这是否永久的？对于多 GPU 的工作站意味着什么？那么我们不能使用多张 4090 训练深度学习模型，必须购买 Quadro 显卡吗？

Also in Windows 11 then for 3D rendering this P2P will be enable and it will not be affected or how it works in that case.
同时，在 Windows 11 中，对于 3D 渲染，这个 P2P 将启用，不会受到影响，或者在这种情况下是如何工作的。

Please @abchauhan confirming that is important for us to make the right business decision.
请@abchauhan 确认，对我们来说做出正确的商业决策非常重要。

Thanks 谢谢

By running following tests:
通过运行以下测试：

They are not comprehensive, but did not work at all when this thread was started.
它们并不全面，但在这个线程开始时根本不起作用。

Thank you! 谢谢！

Has there been any updates to this?
这个有更新吗？
On my multi-4090 AMD system, I can train with multi-gpu. But when deploying an LLM across multiple gpus, it raises device-side index assertion error. If I turn off all virtualizations, the system hangs when attempting to infer from multiple gpus.
在我的多 4090 AMD 系统上，我可以使用多 GPU 进行训练。但是，当在多个 GPU 上部署LLM时，它会引发设备端索引断言错误。如果关闭所有虚拟化，那么在尝试从多个 GPU 进行推理时，系统会挂起。
I believe this is related to the issue in this thread as another system of mine with multi-3090 has no issues at all running the exact same code. I’m on ubuntu22.04, python310, cuda12.1. P2P test seems to run without errors.
我相信这与本帖中的问题有关，因为我另一个配备多张 3090 的系统运行完全相同的代码时完全没有问题。我使用的是 ubuntu22.04、python310 和 cuda12.1。P2P 测试似乎没有错误。

@kj140717 FWIW it seems the latest drivers do correctly report p2p as disabled for 4090, though I’ve since switched to tinygrad’s fork since it implements p2p.
@kj140717 顺便说一下，最新的驱动程序似乎确实正确报告了 4090 的 p2p 被禁用，尽管我后来已经切换到了 tinygrad 的分支，因为它实现了 p2p。