1.1.1. MPS#

The Multi-Process Service (MPS) is an alternative, binary-compatible implementation of the CUDA Application Programming Interface (API). The MPS runtime architecture is designed to transparently enable cooperative multi-process CUDA applications, typically MPI jobs, to utilize Hyper-Q capabilities on the latest NVIDIA (Kepler and later) GPUs. Hyper-Q allows CUDA kernels to be processed concurrently on the same GPU; this can benefit performance when the GPU compute capacity is underutilized by a single application process.
多进程服务（MPS）是 CUDA 应用程序编程接口（API）的一种替代性、二进制兼容的实现。MPS 运行时架构旨在透明地支持协作式多进程 CUDA 应用程序（通常是 MPI 作业），以便在最新的 NVIDIA（Kepler 及以后版本）GPU 上利用 Hyper-Q 功能。Hyper-Q 允许 CUDA 内核在同一 GPU 上并发处理；当单个应用程序进程未充分利用 GPU 计算能力时，这可以提升性能。

1.1.2. Volta MPS#

The Volta architecture introduced new MPS capabilities. Compared to MPS on pre-Volta GPUs, Volta MPS provides a few key improvements:
Volta 架构引入了新的 MPS 功能。与之前的 Volta GPU 上的 MPS 相比，Volta MPS 提供了一些关键改进：

• Volta MPS clients submit work directly to the GPU without passing through the MPS server.
  Volta MPS 客户端直接向 GPU 提交工作，而无需通过 MPS 服务器。

• Each Volta MPS client owns its own GPU address space instead of sharing GPU address space with all other MPS clients.
  每个 Volta MPS 客户端都拥有自己的 GPU 地址空间，而不是与所有其他 MPS 客户端共享 GPU 地址空间。

• Volta MPS supports limited execution resource provisioning for Quality of Service (QoS).
  Volta MPS 支持为服务质量（QoS）提供有限的执行资源配置。

This document will introduce the new capabilities and note the differences between Volta MPS and MPS on pre-Volta GPUs. Running MPS on Volta will automatically enable the new capabilities.
本文档将介绍新的功能，并指出 Volta MPS 与 Volta 之前 GPU 上的 MPS 之间的区别。在 Volta 上运行 MPS 将自动启用这些新功能。

1.1.3. Intended Audience#
1.1.3. 目标读者 #

This document is a comprehensive guide to MPS capabilities and usage. It is intended to be read by application developers and users who will be running GPU calculations and intend to achieve the greatest level of execution performance. It is also intended to be read by system administrators who will be enabling the MPS capability in a user-friendly way, typically on multi-node clusters.
本文档是关于 MPS 功能和使用的全面指南。目标读者是应用程序开发者和用户，他们将运行 GPU 计算并希望实现最高水平的执行性能。同时也适用于系统管理员，他们将以用户友好的方式启用 MPS 功能，通常是在多节点集群上。

1.1.4. Organization of This Document#
1.1.4. 本文档的组织结构 #

The order of the presentation is as follows:
演示的顺序如下：

• Introduction and Concepts – describes why MPS is needed and how it enables Hyper-Q for multi-process applications.
  介绍与概念 – 阐述了为什么需要 MPS，以及它如何为多进程应用程序启用 Hyper-Q。

• When to Use MPS – describes what factors to consider when choosing to run an application with or choosing to deploy MPS for your users.
  何时使用 MPS – 描述在选择是否运行应用程序或为用户部署 MPS 时需要考虑的因素。

• Architecture – describes the client-server architecture of MPS in detail and how it multiplexes clients onto the GPU.
  架构 – 详细描述了 MPS 的客户端-服务器架构以及它如何将客户端复用到 GPU 上。

• Appendices – Reference information for the tools and interfaces used by the MPS system and guidance for common use-cases.
  附录 – 有关 MPS 系统使用的工具和接口的参考信息，以及常见用例的指导。

1.3.1. Why MPS is Needed#
1.3.1. 为什么需要 MPS #

To balance workloads between CPU and GPU tasks, MPI processes are often allocated individual CPU cores in a multi-core CPU machine to provide CPU-core parallelization of potential Amdahl bottlenecks. As a result, the amount of work each individual MPI process is assigned may underutilize the GPU when the MPI process is accelerated using CUDA kernels. While each MPI process may end up running faster, the GPU is being used inefficiently. The Multi-Process Service takes advantage of the inter-MPI rank parallelism, increasing the overall GPU utilization.
为了平衡 CPU 和 GPU 任务之间的工作负载，MPI 进程通常在多核 CPU 机器中被分配到单独的 CPU 核心，以实现 CPU 核心并行化，从而解决潜在的 Amdahl 瓶颈问题。因此，每个单独的 MPI 进程分配的工作量在使用 CUDA 内核加速时可能会导致 GPU 利用不足。虽然每个 MPI 进程的运行速度可能会更快，但 GPU 的使用效率却不高。多进程服务利用了 MPI 进程间的并行性，提高了 GPU 的整体利用率。

1.3.2. What MPS Is#
1.3.2. MPS 是什么 #

MPS is a binary-compatible client-server runtime implementation of the CUDA API which consists of several components:
MPS 是一个与二进制兼容的客户端-服务器运行时 CUDA API 实现，它由以下几个组件构成：

• Control Daemon Process – The control daemon is responsible for starting and stopping the server, as well as coordinating connections between clients and servers.
  控制守护进程 – 控制守护进程负责启动和停止服务器，以及协调客户端与服务器之间的连接。

• Client Runtime – The MPS client runtime is built into the CUDA Driver library and may be used transparently by any CUDA application.
  客户端运行时 – MPS 客户端运行时内置于 CUDA 驱动程序库中，任何 CUDA 应用程序都可以透明地使用它。

• Server Process – The server is the clients’ shared connection to the GPU and provides concurrency between clients.
  服务器进程 – 服务器是客户端与 GPU 的共享连接，并为客户端提供并发性。

2.1.1. GPU Utilization#  2.1.1. GPU 利用率 #

A single process may not utilize all the compute and memory-bandwidth capacity available on the GPU. MPS allows kernel and memcopy operations from different processes to overlap on the GPU, achieving higher utilization and shorter running times.
单一进程可能无法充分利用 GPU 上可用的全部计算和内存带宽容量。MPS 允许来自不同进程的内核和内存复制操作在 GPU 上重叠，从而实现更高的利用率和更短的运行时间。

2.1.2. Reduced On-GPU Context Storage#
2.1.2. 减少 GPU 上的上下文存储 #

Without MPS, each CUDA processes using a GPU allocates separate storage and scheduling resources on the GPU. In contrast, the MPS server allocates one copy of GPU storage and scheduling resources shared by all its clients. Volta MPS supports increased isolation between MPS clients, so the resource reduction is to a much lesser degree.
如果不使用 MPS，每个使用 GPU 的 CUDA 进程会在 GPU 上分配独立的存储和调度资源。相比之下，MPS 服务器会分配一份 GPU 存储和调度资源，供所有客户端共享。Volta MPS 支持 MPS 客户端之间更高的隔离性，因此资源的减少程度要小得多。

2.1.3. Reduced GPU Context Switching#
2.1.3. 减少 GPU 上下文切换 #

Without MPS, when processes share the GPU their scheduling resources must be swapped on and off the GPU. The MPS server shares one set of scheduling resources between all of its clients, eliminating the overhead of swapping when the GPU is scheduling between those clients.
如果没有 MPS，当多个进程共享 GPU 时，它们的调度资源必须在 GPU 上进行交换。MPS 服务器在所有客户端之间共享一组调度资源，消除了在 GPU 在这些客户端之间调度时交换的开销。

2.3.1. System Considerations#
2.3.1. 系统考量 #

2.3.2. Application Considerations#
2.3.2. 应用注意事项 #

• The NVIDIA Codec SDK: https://developer.nvidia.com/nvidia-video-codec-sdk is not supported under MPS on pre-Volta MPS clients.
  NVIDIA Codec SDK：https://developer.nvidia.com/nvidia-video-codec-sdk 在 Volta 之前的 MPS 客户端上不受 MPS 支持。

• Only 64-bit applications are supported. The MPS server will fail to start if the CUDA application is not 64-bit. The MPS client will fail CUDA initialization.
  仅支持 64 位应用程序。如果 CUDA 应用程序不是 64 位的，MPS 服务器将无法启动。MPS 客户端将无法完成 CUDA 初始化。

• If an application uses the CUDA driver API, then it must use headers from CUDA 4.0 or later (that is, it must not have been built by setting CUDA_FORCE_API_VERSION to an earlier version). Context creation in the client will fail if the context version is older than 4.0.
  如果一个应用程序使用 CUDA 驱动程序 API，那么它必须使用 CUDA 4.0 或更高版本的头文件（也就是说，它不能通过将 CUDA_FORCE_API_VERSION 设置为较早版本来构建）。如果上下文版本低于 4.0，客户端中的上下文创建将会失败。

• Dynamic parallelism is not supported. CUDA module load will fail if the module uses dynamic parallelism features.
  不支持动态并行。如果模块使用了动态并行功能，CUDA 模块加载将会失败。

• MPS server only supports clients running with the same UID as the server. The client application will fail to initialize if the server is not running with the same UID. Volta MPS may be launched in -multiuser-server mode to allow clients under different UIDs to connect to a single MPS server launched under the root user while dropping isolation between users. Refer to Server for details regarding -multiuser-server mode.
  MPS 服务器仅支持与服务器具有相同 UID 的客户端运行。如果服务器未以相同 UID 运行，客户端应用程序将无法初始化。Volta MPS 可以以 -multiuser-server 模式启动，允许不同 UID 下的客户端连接到以 root 用户启动的单个 MPS 服务器，同时取消用户之间的隔离。有关 -multiuser-server 模式的详细信息，请参阅服务器部分。

• Stream callbacks are not supported on pre-Volta MPS clients. Calling any stream callback APIs will return an error.
  在 Volta 之前的 MPS 客户端上不支持流回调。调用任何流回调 API 都将返回错误。

• CUDA graphs with host nodes are not supported under MPS on pre-Volta MPS clients.
  在预 Volta MPS 客户端上，MPS 不支持带有主机节点的 CUDA 图。

• The amount of page-locked host memory that pre-Volta MPS client applications can allocate is limited by the size of the tmpfs filesystem (/dev/shm for Linux and /dev/shmem for QNX). Attempting to allocate more page-locked memory than the allowed size using any of relevant CUDA APIs will fail.
  pre-Volta MPS 客户端应用程序可以分配的页面锁定主机内存量受到 tmpfs 文件系统大小的限制（在 Linux 上为 /dev/shm ，在 QNX 上为 /dev/shmem ）。尝试使用任何相关的 CUDA API 分配超出允许大小的页面锁定内存将会失败。

• Terminating an MPS client without synchronizing with all outstanding GPU work (via Ctrl-C / program exception such as segfault / signals, etc.) can leave the MPS server and other MPS clients in an undefined state, which may result in hangs, unexpected failures, or corruptions.
  如果在未与所有未完成的 GPU 工作同步的情况下终止 MPS 客户端（通过 Ctrl-C/程序异常如段错误/信号等），可能会使 MPS 服务器和其他 MPS 客户端处于未定义状态，从而可能导致挂起、意外失败或数据损坏。

• CUDA IPC between CUDA contexts which are created by processes running as MPS clients and CUDA contexts which are created by processes not running as MPS clients is supported under Volta MPS. CUDA IPC is not supported on Tegra platforms.
  在 Volta MPS 下，支持在作为 MPS 客户端运行的进程创建的 CUDA 上下文与非 MPS 客户端运行的进程创建的 CUDA 上下文之间进行 CUDA IPC。Tegra 平台不支持 CUDA IPC。

• Launching cooperative group kernel with MPS is not supported on Tegra platforms.
  在 Tegra 平台上，不支持使用 MPS 启动协作组内核。

2.3.3. Memory Protection and Error Containment#
2.3.3. 内存保护与错误遏制 #

MPS is only recommended for running cooperative processes effectively acting as a single application, such as multiple ranks of the same MPI job, such that the severity of the following memory protection and error containment limitations is acceptable.
MPS 仅推荐用于运行协作进程，以有效充当单个应用程序，例如同一 MPI 作业的多个 rank，从而使以下内存保护和错误遏制限制的严重性可以接受。

2.3.4. MPS on Multi-GPU Systems#
2.3.4. 多 GPU 系统上的 MPS #

The MPS server supports using multiple GPUs. On systems with more than one GPU, you can use CUDA_VISIBLE_DEVICES to enumerate the GPUs you would like to use. Refer to Environment Variables for more details.
MPS 服务器支持使用多个 GPU。在拥有多个 GPU 的系统上，您可以使用 CUDA_VISIBLE_DEVICES 来枚举您希望使用的 GPU。更多详情请参考环境变量。

On systems with a mix of Volta / pre-Volta GPUs, if the MPS server is set to enumerate any Volta GPU, it will discard all pre-Volta GPUs. In other words, the MPS server will either operate only on the Volta GPUs and expose Volta capabilities or operate only on pre-Volta GPUs.
在混合使用 Volta 和 pre-Volta GPU 的系统上，如果 MPS 服务器设置为枚举任何 Volta GPU，它将忽略所有 pre-Volta GPU。换句话说，MPS 服务器要么仅在 Volta GPU 上运行并展示 Volta 的功能，要么仅在 pre-Volta GPU 上运行。

2.3.5. Performance#  2.3.5. 性能 #

2.3.6. Interaction with Tools#
2.3.6. 与工具的交互 #

2.3.7. Client Early Termination#
2.3.7. 客户端提前终止 #

Terminating an MPS client via CTRL-C or signals is not supported and will lead to undefined behavior. The user must guarantee that the MPS client is idle, by calling either cudaDeviceSynchronize or cudaStreamSynchronize on all streams, before the MPS client can be terminated. Early termination of a MPS client without synchronizing all outstanding GPU work may leave the MPS server in an undefined state and result in unexpected failures, corruptions, or hangs; as a result, the affected MPS server and all its clients must be restarted.
通过 CTRL-C 或信号终止 MPS 客户端是不被支持的，并且会导致未定义的行为。用户必须确保 MPS 客户端处于空闲状态，通过在所有流上调用 cudaDeviceSynchronize 或 cudaStreamSynchronize ，然后才能终止 MPS 客户端。如果在未同步所有未完成的 GPU 工作的情况下提前终止 MPS 客户端，可能会使 MPS 服务器处于未定义状态，并导致意外的故障、数据损坏或挂起；因此，受影响的 MPS 服务器及其所有客户端必须重新启动。

On Volta MPS, user can instruct the MPS server to terminate the CUDA contexts of an MPS client process, regardless of whether the CUDA contexts are idle or not, by using the control command terminate_client <server PID> <client PID>. This mechanism enables user to terminate the CUDA contexts of a given MPS client process, even when the CUDA contexts are non-idle, without affecting the MPS server or its other MPS clients. The control command terminate_client sends a request to the MPS server which terminates the CUDA contexts of the target MPS client process on behalf of the user and returns after the MPS server has completed the request. The return value is CUDA_SUCCESS if the CUDA contexts of the target MPS client process have been successfully terminated; otherwise, a CUDA error describing the failure state. When the MPS server starts handling the request, each MPS client context running in the target MPS client process becomes INACTIVE; the status changes will be logged by the MPS server. Upon successful completion of the client termination, the target MPS client process will observe a sticky error CUDA_ERROR_MPS_CLIENT_TERMINATED, and it becomes safe to kill the target MPS client process with signals such as SIGKILL without affecting the rest of the MPS server and its MPS clients. Note that the MPS server is not responsible for killing the target MPS client process after the sticky error is set because the target MPS client process might want to:
在 Volta MPS 上，用户可以通过控制命令 terminate_client <server PID> <client PID> 指示 MPS 服务器终止某个 MPS 客户端进程的 CUDA 上下文，无论这些 CUDA 上下文是否处于空闲状态。这一机制允许用户终止指定 MPS 客户端进程的 CUDA 上下文，即使这些上下文处于非空闲状态，也不会影响 MPS 服务器或其他 MPS 客户端。控制命令 terminate_client 向 MPS 服务器发送请求，代表用户终止目标 MPS 客户端进程的 CUDA 上下文，并在 MPS 服务器完成请求后返回。如果目标 MPS 客户端进程的 CUDA 上下文被成功终止，返回值为 CUDA_SUCCESS ；否则返回描述失败状态的 CUDA 错误。当 MPS 服务器开始处理请求时，目标 MPS 客户端进程中运行的每个 MPS 客户端上下文将变为 INACTIVE ；状态变化将被 MPS 服务器记录。 在客户端终止成功完成后，目标 MPS 客户端进程将观察到一个粘性错误 CUDA_ERROR_MPS_CLIENT_TERMINATED ，此时可以安全地使用诸如 SIGKILL 的信号终止目标 MPS 客户端进程，而不会影响 MPS 服务器及其余 MPS 客户端。请注意，MPS 服务器不负责在设置粘性错误后终止目标 MPS 客户端进程，因为目标 MPS 客户端进程可能希望：

• Perform clean-up of its GPU or CPU state. This may include a device reset. Continue remaining CPU work.
  清理其 GPU 或 CPU 状态。这可能包括设备重置。继续剩余的 CPU 工作。

• Continue remaining CPU work.
  继续剩余的 CPU 工作。

If the user wants to terminate the GPU work of a MPS client process that is running inside a PID namespace different from the MPS control’s PID namespace, such as an MPS client process inside a container, the user must use the PID of the target MPS client process translated into the MPS control’s PID namespace. For example, the PID of an MPS client process inside the container is 6, and the PID of this MPS client process in the host PID namespace is 1024; the user must use 1024 to terminate the GPU work of the target MPS client process.
如果用户想要终止一个运行在与 MPS 控制进程不同的 PID 命名空间中的 MPS 客户端进程的 GPU 工作，例如容器内的 MPS 客户端进程，用户必须使用目标 MPS 客户端进程转换到 MPS 控制进程的 PID 命名空间中的 PID。例如，容器内某个 MPS 客户端进程的 PID 是 6，而在主机 PID 命名空间中该 MPS 客户端进程的 PID 是 1024；用户必须使用 1024 来终止目标 MPS 客户端进程的 GPU 工作。

The common workflow for terminating the client application nbody:
终止客户端应用程序的常见工作流程 nbody :

Use the control command ps to get the status of the current active MPS clients.
使用控制命令 ps 获取当前活跃的 MPS 客户端的状态。

$ echo "ps" | nvidia-cuda-mps-control

PID ID SERVER DEVICE NAMESPACE COMMAND

9741 0 6472 GPU-cb1213a3-d6a4-be7f 4026531836 ./nbody

9743 0 6472 GPU-cb1213a3-d6a4-be7f 4026531836 ./matrixMul

Terminate using the PID of nbody in the host PID namespace as reported by ps:
使用主机 PID 命名空间中由 ps 报告的 nbody 的 PID 进行终止：

$ echo "terminate_client 6472 9741" | nvidia-cuda-mps-control

#wait until terminate_client to return

#upon successful termination 0 is returned

0

Now it is safe to kill nbody:
现在可以安全地终止 nbody ：

$ kill -9 9741

MPS client termination is not supported on Tegra platforms.
在 Tegra 平台上不支持 MPS 客户端终止。

2.3.8. Client Priority Level Control#
2.3.8. 客户端优先级控制 #

Users are normally only able to control the GPU priority level of their kernels by using the cudaStreamCreateWithPriority() API while the program is being written. On Volta MPS, the user can use the control command set_default_client_priority <Priority Level> to map the stream priorities of a given client to a different range of internal CUDA priorities. Changes to this setting do not take effect until the next client connection to the server is opened. The user can also set the CUDA_MPS_CLIENT_PRIORITY environment variable before starting the control daemon or any given client process to set this value.
用户通常只能在编写程序时通过使用 cudaStreamCreateWithPriority() API 来控制其内核的 GPU 优先级。在 Volta MPS 上，用户可以使用控制命令 set_default_client_priority <Priority Level> 将给定客户端的流优先级映射到不同的内部 CUDA 优先级范围。此设置的更改直到下一次客户端与服务器建立连接时才会生效。用户还可以在启动控制守护进程或任何给定客户端进程之前设置 CUDA_MPS_CLIENT_PRIORITY 环境变量来设定此值。

In this release, the allowed priority level values are 0 (normal) and 1 (below normal). Lower numbers map to higher priorities to match the behavior of the Linux kernel scheduler.
在此版本中，允许的优先级值是 0 （正常）和 1 （低于正常）。较小的数字对应较高的优先级，以匹配 Linux 内核调度程序的行为。

For example:  例如：

• Process A is launched at Normal priority and only uses the default CUDA Stream, which has the lowest priority of 0.
  进程 A 以普通优先级启动，并且仅使用默认的 CUDA 流，默认流的优先级最低，为 0。

• Process B is launched at Below Normal priority and uses streams with custom Stream priority values, such as -3.
  进程 B 以低于正常的优先级启动，并使用具有自定义流优先级值的流，例如 -3。

Without this feature, the streams from Process B would be executed first by the CUDA driver. However, with the Client Priority Level feature, the streams from Process A will take precedence.
如果没有此功能，来自进程 B 的流将首先由 CUDA 驱动程序执行。然而，有了客户端优先级功能，来自进程 A 的流将优先执行。

3.3.1. Server#  3.3.1. 服务器 #

The MPS control daemon is responsible for the startup and shutdown of MPS servers. The control daemon allows at most one MPS server to be active at a time. When an MPS client connects to the control daemon, the daemon launches an MPS server if there is no server active. The MPS server is launched with the same user id as that of the MPS client.
MPS 控制守护进程负责 MPS 服务器的启动和关闭。控制守护进程最多允许一个 MPS 服务器同时处于活动状态。当 MPS 客户端连接到控制守护进程时，如果没有活动的服务器，守护进程会启动一个 MPS 服务器。MPS 服务器将以与 MPS 客户端相同的用户 ID 启动。

If there is an MPS server already active and the user ID of the server and client match, then the control daemon allows the client to proceed to connect to the server. If there is an MPS server already active, but the server and client were launched with different user IDs, the control daemon requests the existing server to shutdown once all its clients have disconnected. Once the existing server has shutdown, the control daemon launches a new server with the same user ID as that of the new user’s client process. This is shown in the figure above where user Bob starts client C’ before a server is available. Only once user Alice’s clients exit is a server created for user Bob and client C’.
如果已经有一个活跃的 MPS 服务器，并且服务器和客户端的用户 ID 匹配，那么控制守护进程会允许客户端继续连接到服务器。如果已经有一个活跃的 MPS 服务器，但服务器和客户端是以不同的用户 ID 启动的，控制守护进程会要求现有服务器在所有客户端断开连接后关闭。一旦现有服务器关闭，控制守护进程会启动一个新的服务器，其用户 ID 与新用户客户端进程的用户 ID 相同。这在上图中有所展示，用户 Bob 在服务器可用之前启动了客户端 C’。只有当用户 Alice 的客户端退出后，才会为用户 Bob 和客户端 C’创建一个服务器。

The MPS control daemon does not shutdown the active server if there are no pending client requests. This means that the active MPS server process will persist even if all active clients exit. The active server is shutdown when either a new MPS client, launched with a different user id than the active MPS server, connects to the control daemon or when the work launched by the clients has caused a fault. This is shown in the example above, where the control daemon issues a server exit request to Alice’s server only once user Bob starts client C, even though all of Alice’s clients have exited.
MPS 控制守护进程在没有待处理客户端请求时不会关闭活动服务器。这意味着即使所有活动客户端都退出，活动的 MPS 服务器进程仍将持续存在。活动服务器会在以下情况下被关闭：要么有一个新的 MPS 客户端（其用户 ID 与活动 MPS 服务器的不同）连接到控制守护进程，要么客户端启动的工作导致了故障。如上例所示，即使 Alice 的所有客户端都已退出，控制守护进程也只有在用户 Bob 启动客户端 C 时才会向 Alice 的服务器发出退出请求。

On Volta MPS, the restriction of one Linux user per MPS server may berelaxed to avoid reprovisioning the MPS server on each new user request. Under this mode, clients from all Linux users will appear as clients from the root user and connect to the root MPS server. It is important to make sure that isolation between different users (including the root user) can be safely disregarded before enabling this mode. Clients from all users will share the same MPS log files. The same error containment rules (refer to Memory Protection and Error Containment) also apply in this mode across clients from all users. For example, a fatal fault from one client may bring down a different user’s client that shares any GPU with the faulting client. To allow multiple Linux users share one MPS server, start the control daemon under superuser with the -multiuser-server option. This option is not supported on Tegra platforms.
在 Volta MPS 上，可能会放宽每个 MPS 服务器仅限一个 Linux 用户的限制，以避免在每次新用户请求时重新配置 MPS 服务器。在这种模式下，来自所有 Linux 用户的客户端将被视为来自 root 用户的客户端，并连接到 root MPS 服务器。在启用此模式之前，务必确保可以安全地忽略不同用户（包括 root 用户）之间的隔离。来自所有用户的客户端将共享相同的 MPS 日志文件。相同的错误遏制规则（参见内存保护和错误遏制）在此模式下也适用于来自所有用户的客户端。例如，来自一个客户端的致命故障可能会导致与该故障客户端共享任何 GPU 的其他用户的客户端宕机。为了允许多个 Linux 用户共享一个 MPS 服务器，请以超级用户身份启动控制守护进程，并使用 -multiuser-server 选项。此选项在 Tegra 平台上不受支持。

An MPS server may be in one of the following states: INITIALIZING, ACTIVE or FAULT. The INITIALIZING state indicates that the MPS server is busy initializing and the MPS control will hold the new client requests in its queue. The ACTIVE state indicates the MPS server is able to process new client requests. The FAULT state indicates that the MPS server is blocked on a fatal fault caused by a client. Any new client requests will be rejected with error CUDA_ERROR_MPS_SERVER_NOT_READY.
MPS 服务器可能处于以下状态之一： INITIALIZING 、 ACTIVE 或 FAULT 。 INITIALIZING 状态表示 MPS 服务器正忙于初始化，MPS 控制将新的客户端请求保留在其队列中。 ACTIVE 状态表示 MPS 服务器能够处理新的客户端请求。 FAULT 状态表示 MPS 服务器因客户端引发的致命故障而被阻塞。任何新的客户端请求都将被拒绝，并返回错误 CUDA_ERROR_MPS_SERVER_NOT_READY 。

A newly launched MPS server will be in the INITIALIZING state first. After successful initialization, the MPS server goes into the ACTIVE state. When a client encounters a fatal fault, the MPS server will transition from ACTIVE to FAULT. On pre-Volta MPS, the MPS server shuts down after encountering a fatal fault. On Volta MPS, the MPS server becomes ACTIVE again after all faulting clients have disconnected.
新启动的 MPS 服务器将首先处于 INITIALIZING 状态。初始化成功后，MPS 服务器会进入 ACTIVE 状态。当客户端遇到致命故障时，MPS 服务器将从 ACTIVE 状态转换到 FAULT 状态。在预 Volta MPS 上，MPS 服务器在遇到致命故障后会关闭。在 Volta MPS 上，MPS 服务器在所有故障客户端断开连接后会再次变为 ACTIVE 状态。

The control daemon executable also supports an interactive mode where a user with sufficient permissions can issue commands, for example to see the current list of servers and clients or startup and shutdown servers manually.
控制守护进程可执行文件还支持交互模式，在此模式下，拥有足够权限的用户可以发出命令，例如查看当前的服务器和客户端列表，或手动启动和关闭服务器。

3.3.2. Client Attach/Detach#
3.3.2. 客户端连接/断开 #

When CUDA is first initialized in a program, the CUDA driver attempts to connect to the MPS control daemon. If the connection attempt fails, the program continues to run as it normally would without MPS. If however, the connection attempt succeeds, the MPS control daemon proceeds to ensure that an MPS server, launched with same user ID as that of the connecting client, is active before returning to the client. The MPS client then proceeds to connect to the server.
当程序首次初始化 CUDA 时，CUDA 驱动程序会尝试连接到 MPS 控制守护进程。如果连接尝试失败，程序将照常运行，不使用 MPS。然而，如果连接尝试成功，MPS 控制守护进程会确保在返回客户端之前，启动一个与连接客户端具有相同用户 ID 的 MPS 服务器。然后，MPS 客户端会继续连接到该服务器。

All communication between the MPS client, the MPS control daemon, and the MPS server is done using named pipes and UNIX domain sockets. The MPS server launches a worker thread to receive commands from the client. Successful client connection will be logged by the MPS server as the client status becomes ACTIVE. Upon client process exit, the server destroys any resources not explicitly freed by the client process and terminates the worker thread. The client exit event will be logged by the MPS server.
MPS 客户端、MPS 控制守护进程和 MPS 服务器之间的所有通信都使用命名管道和 UNIX 域套接字进行。MPS 服务器会启动一个工作线程来接收来自客户端的命令。客户端成功连接后，MPS 服务器会记录客户端状态为 ACTIVE 。当客户端进程退出时，服务器会销毁客户端进程未明确释放的任何资源，并终止工作线程。客户端退出事件将被 MPS 服务器记录。

4.1.1. nvidia-cuda-mps-control#

Typically stored under /usr/bin on Linux and QNX systems and typically run with superuser privileges, this control daemon is used to manage the nvidia-cuda-mps-server described in the following section. These are the relevant use cases:
通常在 Linux 和 QNX 系统上存储在 /usr/bin 下，并且通常以超级用户权限运行，此控制守护进程用于管理下一节中描述的 nvidia-cuda-mps-server 。以下是相关的用例：

man nvidia-cuda-mps-control          # Describes usage of this utility.

nvidia-cuda-mps-control -d           # Start daemon in background process.

ps -ef | grep mps                    # Check if the MPS daemon is running, for Linux.

pidin  | grep mps                    # See if the MPS daemon is running, for QNX.

echo quit | nvidia-cuda-mps-control  # Shut the daemon down.

nvidia-cuda-mps-control -f           # Start daemon in foreground.

nvidia-cuda-mps-control -v           # Print version of control daemon executable (applicable on Tegra platforms only).

The control daemon creates a nvidia-cuda-mps-control.pid file that contains the PID of the control daemon process in the CUDA_MPS_PIPE_DIRECTORY. When there are multiple instances of the control daemon running in parallel, one can target a specific instance by looking up its PID in the corresponding CUDA_MPS_PIPE_DIRECTORY. If CUDA_MPS_PIPE_DIRECTORY is not set, the nvidia-cuda-mps-control.pid file will be created at the default pipe directory at /tmp/nvidia-mps.
控制守护进程会在 CUDA_MPS_PIPE_DIRECTORY 中创建一个包含控制守护进程 PID 的 nvidia-cuda-mps-control.pid 文件。当有多个控制守护进程实例并行运行时，可以通过在相应的 CUDA_MPS_PIPE_DIRECTORY 中查找其 PID 来定位特定实例。如果未设置 CUDA_MPS_PIPE_DIRECTORY ，则 nvidia-cuda-mps-control.pid 文件将在默认管道目录 /tmp/nvidia-mps 处创建。

When used in interactive mode, the available commands are:
在交互模式下，可用的命令有：

• get_server_list – prints out a list of all PIDs of server instances.
  get_server_list – 打印出所有服务器实例的 PID 列表。

• get_server_status <PID> – this will print out the status of the server with the given <PID>.
  get_server_status <PID> – 这将打印出给定 的服务器状态。

• start_server - uid <user id> – manually starts a new instance of nvidia-cuda-mps-server with the given user ID.
  start_server - uid <user id> – 手动启动一个新的 nvidia-cuda-mps-server 实例，并使用指定的用户 ID。

• get_client_list <PID> – lists the PIDs of client applications connected to a server instance assigned to the given PID.
  get_client_list <PID> – 列出连接到分配给指定 PID 的服务器实例的客户端应用程序的 PID。

• quit – terminates the nvidia-cuda-mps-control daemon.
  quit – 终止 nvidia-cuda-mps-control 守护进程。

Commands available to Volta MPS control:
Volta MPS 控制可用的命令：

• get_device_client_list [<PID>] – lists the devices and PIDs of client applications that enumerated this device. It optionally takes the server instance PID.
  get_device_client_list [<PID>] – 列出枚举此设备的客户端应用程序的设备和 PID。可选地接受服务器实例 PID。

• set_default_active_thread_percentage <percentage> – overrides the default active thread percentage for MPS servers. If there is already a server spawned, this command will only affect the next server. The set value is lost if a quit command is executed. The default is 100.
  set_default_active_thread_percentage <percentage> – 覆盖 MPS 服务器的默认活动线程百分比。如果已经启动了一个服务器，此命令将仅影响下一个服务器。如果执行了 quit 命令，设置的值将会丢失。默认值为 100。

• get_default_active_thread_percentage – queries the current default available thread percentage.
  get_default_active_thread_percentage – 查询当前默认可用线程百分比。

• set_active_thread_percentage <PID> <percentage> – overrides the active thread percentage for the MPS server instance of the given PID. All clients created with that server afterwards will observe the new limit. Existing clients are not affected.
  set_active_thread_percentage <PID> <percentage> – 覆盖指定 PID 的 MPS 服务器实例的活动线程百分比。之后使用该服务器创建的所有客户端都将遵守新的限制。现有客户端不受影响。

• get_active_thread_percentage <PID> – queries the current available thread percentage of the MPS server instance of the given PID.
  get_active_thread_percentage <PID> – 查询给定 PID 的 MPS 服务器实例当前可用的线程百分比。

• set_default_device_pinned_mem_limit <dev> <value> – sets the default device pinned memory limit for each MPS client. If there is already a server spawned, this command will only affect the next server. The set value is lost if a quit command is executed. The dev argument may be a device UUID string or an integer ordinal. The value must be in the form of an integer followed by a qualifier, either “G” or “M” that specifies the value in Gigabyte or Megabyte respectively. For example, to set a limit of 10 gigabytes for device 0, use the following command:
  set_default_device_pinned_mem_limit <dev> <value> – 为每个 MPS 客户端设置默认设备固定内存限制。如果已经有一个服务器启动，此命令将仅影响下一个服务器。如果执行了 quit 命令，设置的值将会丢失。dev 参数可以是设备 UUID 字符串或整数序号。值必须是整数后跟限定符的形式，限定符可以是“G”或“M”，分别表示以千兆字节或兆字节为单位。例如，要为设备 0 设置 10 千兆字节的限制，请使用以下命令：

  set_default_device_pinned_mem_limit 0 10G

  By default, there is no memory limit set.
  默认情况下，没有设置内存限制。

  Note that for this command, the dev argument is not validated against available devices in the MPS server. Therefore, it is possible to set two memory limits for the same device: one by device UUID and another by ordinal. When an MPS server is started, whichever limit was set last will take effect. A limit set with an invalid device UUID or ordinal will be ignored when starting the MPS server.
  请注意，对于此命令，dev 参数不会与 MPS 服务器中的可用设备进行验证。因此，可以为同一设备设置两个内存限制：一个通过设备 UUID 设置，另一个通过序号设置。当 MPS 服务器启动时，最后设置的限制将生效。如果使用无效的设备 UUID 或序号设置的限制，在启动 MPS 服务器时将被忽略。

• get_default_device_pinned_mem_limit <dev> – queries the current default pinned memory limit for the device. The dev argument may be device UUID string or an integer ordinal.
  get_default_device_pinned_mem_limit <dev> – 查询设备当前的默认固定内存限制。 dev 参数可以是设备 UUID 字符串或整数序号。

  Note that this command does not translate between device UUIDs or ordinals and will return the limit that was set for each device identifier via the set_default_device_pinned_mem_limit command.
  请注意，此命令不会在设备 UUID 或序号之间进行转换，并且将返回通过 set_default_device_pinned_mem_limit 命令为每个设备标识符设置的限制。

• set_device_pinned_mem_limit <PID> <dev> <value> - overrides the device pinned memory limit for MPS servers. This sets the device pinned memory limit for each client of MPS server instance of the given PID for the device dev. All clients created with that server afterwards will observe the new limit. Existing clients are not affected. The dev argument may be a device UUID string or an integer ordinal. For example, to set a limit of 900MB for the server with pid 1024 for device 0, use the following command:
  set_device_pinned_mem_limit <PID> <dev> <value> - 覆盖 MPS 服务器的设备固定内存限制。此设置针对给定 PID 的 MPS 服务器实例为设备 dev 设置设备固定内存限制。之后与该服务器创建的所有客户端都将遵守新的限制。现有客户端不受影响。 dev 参数可以是设备 UUID 字符串或整数序号。例如，要为 pid 为 1024 的服务器在设备 0 上设置 900MB 的限制，请使用以下命令：

  set_device_pinned_mem_limit 1024 0 900M

• get_device_pinned_mem_limit <PID> <dev> – queries the current device pinned memory limit of the MPS server instance of the given PID for the device dev. The dev argument may be a device UUID string or an integer ordinal.
  get_device_pinned_mem_limit <PID> <dev> – 查询给定 PID 的 MPS 服务器实例在设备 dev 上的当前设备固定内存限制。 dev 参数可以是设备 UUID 字符串或整数序号。

• terminate_client <server PID> <client PID> – terminates all the outstanding GPU work of the MPS client process <client PID> running on the MPS server denoted by <server PID>. For example, to terminate the outstanding GPU work for an MPS client process with PID 1024 running on an MPS server with PID 123, use the following command:
  terminate_client <server PID> <client PID> – 终止在由 <server PID> 指定的 MPS 服务器上运行的 MPS 客户端进程 <client PID> 的所有未完成的 GPU 工作。例如，要终止在 PID 为 123 的 MPS 服务器上运行的 PID 为 1024 的 MPS 客户端进程的未完成 GPU 工作，请使用以下命令：

  terminate_client 123 1024

• ps [-p PID] – reports a snapshot of the current client processes. It optionally takes the server instance PID. It displays the PID, the unique identifier assigned by the server, the partial UUID of the associated device, the PID of the connected server, the namespace PID, and the command line of the client.
  ps [-p PID] – 报告当前客户端进程的快照。可选地接受服务器实例的 PID。它显示 PID、服务器分配的唯一标识符、关联设备的部分 UUID、连接服务器的 PID、命名空间 PID 以及客户端的命令行。

• set_default_client_priority [priority] – sets the default client priority that will be used for new clients. The value is not applied to existing clients. Priority values should be considered as hints to the CUDA Driver, not guarantees. Allowed values are 0 [NORMAL] and 1 [BELOW NORMAL]. The set value is lost if a quit command is executed. The default is 0 [NORMAL].
  set_default_client_priority [priority] – 设置新客户端将使用的默认客户端优先级。该值不适用于现有客户端。优先级值应被视为对 CUDA 驱动程序的提示，而非保证。允许的值为 0 [NORMAL] 和 1 [BELOW NORMAL] 。如果执行了 quit 命令，设置的值将会丢失。默认值为 0 [NORMAL] 。

• get_default_client_priority – queries the current priority value that will be used for new clients.
  get_default_client_priority – 查询将用于新客户端的当前优先级值。

4.1.2. nvidia-cuda-mps-server#

Typically stored under /usr/bin on Linux and QNX systems, this daemon is run under the same $UID as the client application running on the node. The nvidia-cuda-mps-server instances are created on-demand when client applications connect to the control daemon. The server binary should not be invoked directly, and instead the control daemon should be used to manage the startup and shutdown of servers.
在 Linux 和 QNX 系统上，此守护进程通常存储在 /usr/bin 下，并以与节点上运行的客户端应用程序相同的$UID 运行。当客户端应用程序连接到控制守护进程时，会按需创建 nvidia-cuda-mps-server 实例。服务器二进制文件不应直接调用，而应使用控制守护进程来管理服务器的启动和关闭。

The nvidia-cuda-mps-server process owns the CUDA context on the GPU and uses it to execute GPU operations for its client application processes. Due to this, when querying active processes via nvidia-smi (or any NVML-based application) nvidia-cuda-mps-server will appear as the active CUDA process rather than any of the client processes.
nvidia-cuda-mps-server 进程拥有 GPU 上的 CUDA 上下文，并使用它为客户端应用程序进程执行 GPU 操作。因此，在通过 nvidia-smi （或任何基于 NVML 的应用程序）查询活动进程时， nvidia-cuda-mps-server 将显示为活动的 CUDA 进程，而不是任何客户端进程。

The version of the nvidia-cuda-mps-server executable can be printed with:
nvidia-cuda-mps-server 可执行文件的版本可以通过以下方式打印：

nvidia-cuda-mps-server -v

4.1.3. nvidia-smi#

Typically stored under /usr/bin on Linux systems, this is used to configure GPUs on a node. The following use cases are relevant to managing MPS:
在 Linux 系统上通常存储在 /usr/bin 下，用于配置节点上的 GPU。以下使用场景与管理 MPS 相关：

man nvidia-smi                        # Describes usage of this utility.

nvidia-smi -L                         # List the GPU's on node.

nvidia-smi -q                         # List GPU state and configuration information.

nvidia-smi -q -d compute              # Show the compute mode of each GPU.

nvidia-smi -i 0 -c EXCLUSIVE_PROCESS  # Set GPU 0 to exclusive mode, run as root.

nvidia-smi -i 0 -c DEFAULT            # Set GPU 0 to default mode, run as root. (SHARED_PROCESS)

nvidia-smi -i 0 -r                    # Reboot GPU 0 with the new setting.

4.2.1. CUDA_VISIBLE_DEVICES#

CUDA_VISIBLE_DEVICES is used to specify which GPU’s should be visible to a CUDA application. Only the devices whose index or UUID is present in the sequence are visible to CUDA applications and they are enumerated in the order of the sequence.
CUDA_VISIBLE_DEVICES 用于指定哪些 GPU 对 CUDA 应用程序可见。只有序列中包含其索引或 UUID 的设备对 CUDA 应用程序可见，并且它们会按照序列的顺序进行枚举。

When CUDA_VISIBLE_DEVICES is set before launching the control daemon, the devices will be remapped by the MPS server. This means that if your system has devices 0, 1 and 2, and if CUDA_VISIBLE_DEVICES is set to 0,2, then when a client connects to the server it will see the remapped devices – device 0 and a device 1. Therefore, keeping CUDA_VISIBLE_DEVICES set to 0,2 when launching the client would lead to an error.
在启动控制守护进程之前设置 CUDA_VISIBLE_DEVICES 时，设备将被 MPS 服务器重新映射。这意味着如果您的系统有设备 0、1 和 2，并且 CUDA_VISIBLE_DEVICES 设置为 0,2 ，那么当客户端连接到服务器时，它将看到重新映射的设备——设备 0 和设备 1。因此，在启动客户端时保持 CUDA_VISIBLE_DEVICES 设置为 0,2 会导致错误。

The MPS control daemon will further filter-out any pre-Volta devices, if any visible device is Volta+.
MPS 控制守护进程将进一步过滤掉任何 pre-Volta 设备，如果有任何可见设备是 Volta+ 的话。

To avoid this ambiguity, we recommend using UUIDs instead of indices. These can be viewed by launching nvidia-smi -q. When launching the server, or the application, you can set CUDA_VISIBLE_DEVICES to UUID_1,UUID_2, where UUID_1 and UUID_2 are the GPU UUIDs. It will also work when you specify the first few characters of the UUID (including GPU-) rather than the full UUID.
为了避免这种歧义，我们建议使用 UUID 而不是索引。可以通过启动 nvidia-smi -q 来查看这些 UUID。在启动服务器或应用程序时，您可以将 CUDA_VISIBLE_DEVICES 设置为 UUID_1,UUID_2 ，其中 UUID_1 和 UUID_2 是 GPU 的 UUID。如果您只指定 UUID 的前几个字符（包括 GPU- ），而不是完整的 UUID，也同样有效。

The MPS server will fail to start if incompatible devices are visible after the application of CUDA_VISIBLE_DEVICES.
如果在应用 CUDA_VISIBLE_DEVICES 后发现不兼容的设备，MPS 服务器将无法启动。

4.2.2. CUDA_MPS_PIPE_DIRECTORY#

The MPS control daemon, the MPS server, and the associated MPS clients communicate with each other via named pipes and UNIX domain sockets. The default directory for these pipes and sockets is /tmp/nvidia-mps. The environment variable, CUDA_MPS_PIPE_DIRECTORY, can be used to override the location of these pipes and sockets. The value of this environment variable should be consistent across all MPS clients sharing the same MPS server, and the MPS control daemon.
MPS 控制守护进程、MPS 服务器以及相关的 MPS 客户端通过命名管道和 UNIX 域套接字相互通信。这些管道和套接字的默认目录是 /tmp/nvidia-mps 。环境变量 CUDA_MPS_PIPE_DIRECTORY 可用于覆盖这些管道和套接字的位置。该环境变量的值应在共享同一 MPS 服务器的所有 MPS 客户端以及 MPS 控制守护进程中保持一致。

The recommended location for the directory containing these named pipes and domain sockets is local folders such as /tmp. If the specified location exists in a shared, multi-node filesystem, the path must be unique for each node to prevent multiple MPS servers or MPS control daemons from using the same pipes and sockets. When provisioning MPS on a per-user basis, the directory should be set to a location such that different users will not end up using the same directory.
建议将包含这些命名管道和域套接字的目录放置在本地文件夹中，例如 /tmp 。如果指定的位置存在于共享的多节点文件系统中，则每个节点的路径必须是唯一的，以防止多个 MPS 服务器或 MPS 控制守护进程使用相同的管道和套接字。在按用户分配 MPS 时，应将目录设置为一个位置，以确保不同用户不会最终使用同一个目录。

On Tegra platforms, there is no default directory setting for pipes and sockets. Users must set this environment variable such that only intended users have access to this location.
在 Tegra 平台上，管道和套接字没有默认的目录设置。用户必须设置此环境变量，以确保只有预定的用户可以访问此位置。

4.2.3. CUDA_MPS_LOG_DIRECTORY#

The MPS control daemon maintains a control.log file which contains the status of its MPS servers, user commands issued and their result, and startup and shutdown notices for the daemon. The MPS server maintains a server.log file containing its startup and shutdown information and the status of its clients.
MPS 控制守护进程维护着一个 control.log 文件，其中包含其 MPS 服务器的状态、用户发出的命令及其结果，以及守护进程的启动和关闭通知。MPS 服务器维护着一个 server.log 文件，其中包含其启动和关闭信息以及其客户端的状态。

By default these log files are stored in the directory /var/log/nvidia-mps. The CUDA_MPS_LOG_DIRECTORY environment variable can be used to override the default value. This environment variable should be set in the MPS control daemon’s environment and is automatically inherited by any MPS servers launched by that control daemon.
默认情况下，这些日志文件存储在目录 /var/log/nvidia-mps 中。可以使用 CUDA_MPS_LOG_DIRECTORY 环境变量来覆盖默认值。此环境变量应在 MPS 控制守护进程的环境中设置，并由该控制守护进程启动的任何 MPS 服务器自动继承。

On Tegra platforms, there is no default directory setting for storing the log files. MPS will remain operational without the user setting this environment variable; however, in such instances, MPS logs will not be available. If logs are required to be captured, then the user must set this environment variable such that only intended users have access to this location.
在 Tegra 平台上，没有用于存储日志文件的默认目录设置。MPS 可以在用户未设置此环境变量的情况下继续运行；然而，在这种情况下，MPS 日志将不可用。如果需要捕获日志，则用户必须设置此环境变量，以确保只有预期的用户可以访问该位置。

4.2.4. CUDA_DEVICE_MAX_CONNECTIONS#

When encountered in the MPS client’s environment, CUDA_DEVICE_MAX_CONNECTIONS sets the preferred number of compute and copy engine concurrent connections (work queues) from the host to the device for that client. The number actually allocated by the driver may differ from what is requested based on hardware resource limitations or other considerations. Under MPS, each server’s clients share one pool of connections, whereas without MPS each CUDA context would be allocated its own separate connection pool. Volta MPS clients exclusively owns the connections set aside for the client in the shared pool, so setting this environment variable under Volta MPS may reduce the number of available clients. The default value is 2 for Volta MPS clients.
在 MPS 客户端的环境中， CUDA_DEVICE_MAX_CONNECTIONS 设置了从主机到设备的最佳计算和复制引擎并发连接（工作队列）数量，适用于该客户端。驱动程序实际分配的数量可能会根据硬件资源限制或其他考虑因素与请求的数量有所不同。在 MPS 下，每个服务器的客户端共享一个连接池，而在没有 MPS 的情况下，每个 CUDA 上下文都会分配一个独立的连接池。Volta MPS 客户端独占共享池中为该客户端预留的连接，因此在 Volta MPS 下设置此环境变量可能会减少可用客户端的数量。Volta MPS 客户端的默认值为 2。

4.2.5. CUDA_MPS_ACTIVE_THREAD_PERCENTAGE#

On Volta GPUs, this environment variable sets the portion of the available threads that can be used by the client contexts. The limit can be configured at different levels.
在 Volta GPU 上，此环境变量设置了客户端上下文可用的线程部分。限制可以在不同级别进行配置。

4.2.6. CUDA_MPS_ENABLE_PER_CTX_DEVICE_MULTIPROCESSOR_PARTITIONING#

By default, users can only partition the available threads uniformly. An explicit opt-in via this environment variable is required to enable non-uniform partitioning capability. To enable non-uniform partitioning capability, this environment variable must be set before the client process starts.
默认情况下，用户只能均匀分配可用线程。要启用非均匀分配功能，必须通过此环境变量明确选择启用。要启用非均匀分配功能，必须在客户端进程启动前设置此环境变量。

When non-uniform partitioning capability is enabled in an MPS client’s environment, client CUDA contexts can have different active thread percentages within the same client process via setting CUDA_MPS_ACTIVE_THREAD_PERCENTAGE before context creations. The device attribute cudaDevAttrMultiProcessorCount will reflect the active thread percentage and return the portion of available SMs that can be used by the client CUDA context current to the calling thread.
当在 MPS 客户端的环境中启用非均匀分区功能时，客户端的 CUDA 上下文可以在同一客户端进程内通过在上下文创建前设置 CUDA_MPS_ACTIVE_THREAD_PERCENTAGE 来拥有不同的活跃线程百分比。设备属性 cudaDevAttrMultiProcessorCount 将反映活跃线程百分比，并返回当前调用线程的客户端 CUDA 上下文可用的 SMs 部分。

4.2.7. CUDA_MPS_PINNED_DEVICE_MEM_LIMIT#

The pinned memory limit control limits the amount of GPU memory that is allocatable by CUDA APIs by the client process. On Volta GPUs, this environment variable sets a limit on pinned device memory that can be allocated by the client contexts. Setting this environment variable in an MPS client’s environment will set the device’s pinned memory limit when the client process starts. The new limit will only further constrain the limit set by the control daemon (via set_default_device_pinned_mem_limit or set_device_pinned_mem_limit control daemon commands or this environment variable at the MPS control daemon level). If the control daemon has a lower value, the control daemon setting will be obeyed by the client process instead. This environment variable will have the same semantics as CUDA_VISIBLE_DEVICES i.e. the value string can contain comma-separated device ordinals and/or device UUIDs with per device memory limit separated by an equals. Example usage:
固定内存限制控制限制了客户端进程通过 CUDA API 可分配的 GPU 内存量。在 Volta GPU 上，此环境变量为客户端上下文可分配的固定设备内存设置了限制。在 MPS 客户端的环境中设置此环境变量，将在客户端进程启动时设置设备的固定内存限制。新的限制将进一步约束由控制守护进程设置的限制（通过 set_default_device_pinned_mem_limit 或 set_device_pinned_mem_limit control 守护进程命令，或在 MPS 控制守护进程级别设置此环境变量）。如果控制守护进程设置的值较低，则客户端进程将遵守控制守护进程的设置。此环境变量将具有与 CUDA_VISIBLE_DEVICES 相同的语义，即值字符串可以包含以逗号分隔的设备序号和/或设备 UUID，并以等号分隔的每设备内存限制。例如：

$ export CUDA_MPS_PINNED_DEVICE_MEM_LIMIT=''0=1G,1=512MB''

The following example highlights the hierarchy and usage of the MPS memory limiting functionality.
以下示例展示了 MPS 内存限制功能的层次结构和用法。

# Set the default device pinned mem limit to 3G for device 0. The default limit constrains the memory allocation limit of all the MPS clients of future MPS servers to 3G on device 0.

$ nvidia-cuda-mps-control set_default_device_pinned_mem_limit 0 3G

# Start daemon in background process

$ nvidia-cuda-mps-control -d

# Set device pinned mem limit to 2G for device 0 for the server instance of the
# given PID. All the MPS clients on this server will observe this new limit of 2G
# instead of the default limit of 3G when allocating pinned device memory on device 0.
# Note -- users are allowed to specify a server limit (via set_device_pinned_mem_limit)
# greater than the default limit previously set by set_default_device_pinned_mem_limit.

$ nvidia-cuda-mps-control set_device_pinned_mem_limit <pid> 0 2G

# Further constrain the device pinned mem limit for a particular MPS client to 1G for
# device 0. This ensures the maximum amount of memory allocated by this client is capped
# at 1G.
# Note - setting this environment variable to a value greater than value observed by the
# server for its clients (through set_default_device_pinned_mem_limit/ set_device_pinned_mem_limit)
* will not set the limit to the higher value and thus will be ineffective and the eventual
# limit observed by the client will be that observed by the server.

$ export CUDA_MPS_DEVICE_MEM_LIMIT="0=1G"

4.2.8. CUDA_MPS_CLIENT_PRIORITY#

The client priority level variable controls the initial default server value for the MPS Control Daemon if used to launch that, or the client priority level value for a given client if used in a client launch. The following examples demonstrate both usages.
客户端优先级变量控制 MPS 控制守护进程的初始默认服务器值（如果用于启动该守护进程），或者在客户端启动时控制特定客户端的优先级值。以下示例展示了两种用法。

# Set the default client priority level for new servers and clients to Below Normal

$ export CUDA_MPS_CLIENT_PRIORITY=1

$ nvidia-cuda-mps-control -d

# Set the client priority level for a single program to Normal without changing the priority level for future clients

$ CUDA_MPS_CLIENT_PRIORITY=0 <program>

4.3.1. Control Log#  4.3.1. 控制日志 #

Some of the example messages logged by the control daemon:
控制守护进程记录的一些示例消息：

• Startup and shutdown of MPS servers identified by their process IDs and the user ID with which they are being launched.
  MPS 服务器的启动和关闭通过其进程 ID 以及启动时使用的用户 ID 来识别。

  [2013-08-05 12:50:23.347 Control 13894] Starting new server 13929 for user 500

  [2013-08-05 12:50:24.870 Control 13894] NEW SERVER 13929: Ready

  [2013-08-05 13:02:26.226 Control 13894] Server 13929 exited with status 0

• New MPS client connections identified by the client process ID and the user ID of the user that launched the client process.
  新的 MPS 客户端连接通过客户端进程 ID 和启动客户端进程的用户的用户 ID 来识别。

  [2013-08-05 13:02:10.866 Control 13894] NEW CLIENT 19276 from user 500: Server already exists

  [2013-08-05 13:02:10.961 Control 13894] Accepting connection...

• User commands issued to the control daemon and their result.
  用户向控制守护进程发出的命令及其结果。

  [2013-08-05 12:50:23.347 Control 13894] Starting new server 13929 for user 500

  [2013-08-05 12:50:24.870 Control 13894] NEW SERVER 13929: Ready

• Error information such as failing to establish a connection with a client.
  错误信息，例如无法与客户端建立连接。

  [2013-08-05 13:02:10.961 Control 13894] Accepting connection...

  [2013-08-05 13:02:10.961 Control 13894] Unable to read new connection type information

4.3.2. Server Log#  4.3.2. 服务器日志 #

Some of the example messages logged by the MPS server:
MPS 服务器记录的一些示例消息：

• New MPS client connections and disconnections identified by the client process ID.
  新的 MPS 客户端连接和断开由客户端进程 ID 识别。

  [2013-08-05 13:00:09.269 Server 13929] New client 14781 connected

  [2013-08-05 13:00:09.270 Server 13929] Client 14777 disconnected

• Error information such as the MPS server failing to start due to system requirements not being met.
  由于系统要求未满足，MPS 服务器启动失败等错误信息。

  [2013-08-06 10:51:31.706 Server 29489] MPS server failed to start

  [2013-08-06 10:51:31.706 Server 29489] MPS is only supported on 64-bit Linux platforms, with an SM 3.5 or higher GPU.

• Information about fatal GPU error containment on Volta+ MPS
  关于 Volta+ MPS 上致命 GPU 错误遏制的信息

  [2022-04-28 15:56:07.410 Other 11570] Volta MPS: status of client {11661, 1} is ACTIVE

  [2022-04-28 15:56:07.468 Other 11570] Volta MPS: status of client {11663, 1} is ACTIVE

  [2022-04-28 15:56:07.518 Other 11570] Volta MPS: status of client {11643, 2} is ACTIVE

  [2022-04-28 15:56:08.906 Other 11570] Volta MPS: Server is handling a fatal GPU error.

  [2022-04-28 15:56:08.906 Other 11570] Volta MPS: status of client {11641, 1} is INACTIVE

  [2022-04-28 15:56:08.906 Other 11570] Volta MPS: status of client {11643, 1} is INACTIVE

  [2022-04-28 15:56:08.906 Other 11570] Volta MPS: status of client {11643, 2} is INACTIVE

  [2022-04-28 15:56:08.906 Other 11570] Volta MPS: The following devices

  [2022-04-28 15:56:08.906 Other 11570] 0

  [2022-04-28 15:56:08.907 Other 11570] 1

  [2022-04-28 15:56:08.907 Other 11570] Volta MPS: The following clients have a sticky error set:

  [2022-04-28 15:56:08.907 Other 11570] 11641

  [2022-04-28 15:56:08.907 Other 11570] 11643

  [2022-04-28 15:56:09.200 Other 11570] Client {11641, 1} exit

  [2022-04-28 15:56:09.244 Other 11570] Client {11643, 1} exit

  [2022-04-28 15:56:09.244 Other 11570] Client {11643, 2} exit

  [2022-04-28 15:56:09.245 Other 11570] Volta MPS: Destroy server context on device 0

  [2022-04-28 15:56:09.269 Other 11570] Volta MPS: Destroy server context on device 1

  [2022-04-28 15:56:10.310 Other 11570] Volta MPS: Creating server context on device 0

  [2022-04-28 15:56:10.397 Other 11570] Volta MPS: Creating server context on device 1

5.1.1. On a Multi-user System#
5.1.1. 在多用户系统上 #

To cause all users of the system to run CUDA applications via MPS you will need to set up the MPS control daemon to run when the system starts.
要让系统中的所有用户通过 MPS 运行 CUDA 应用程序，您需要在系统启动时设置 MPS 控制守护进程。

export CUDA_VISIBLE_DEVICES=0           # Select GPU 0.

nvidia-smi -i 0 -c EXCLUSIVE_PROCESS    # Set GPU 0 to exclusive mode.

nvidia-cuda-mps-control -d              # Start the daemon.

echo quit | nvidia-cuda-mps-control

5.1.2. On a Single-User System#
5.1.2. 在单用户系统上 #

When running as a single user, the control daemon must be launched with the same user ID as that of the client process.
以单一用户身份运行时，控制守护进程必须以与客户端进程相同的用户 ID 启动。

echo quit | nvidia-cuda-mps-control

$CUDA_MPS_LOG_DIRECTORY/control.log

$CUDA_MPS_LOG_DIRECTORY/server.log

5.1.3. Scripting a Batch Queuing System#
5.1.3. 批处理队列系统的脚本编写 #

# Activate MPS if requested by user

USER=$2
ACCTSTR=$7
echo $ACCTSTR | grep -i "MPS=true"
if [ $? -eq 0 ]; then
   nvidia-smi -c 3
   USERID=`id -u $USER`
   export CUDA_VISIBLE_DEVICES=0
   nvidia-cuda-mps-control -d && echo "MPS control daemon started"
   sleep 1
   echo "start_server -uid $USERID" | nvidia-cuda-mps-control && echo "MPS server started for $USER"
fi

# Reset compute mode to default
nvidia-smi -c 0

# Quit cuda MPS if it's running
ps aux | grep nvidia-cuda-mps-control | grep -v grep > /dev/null
if [ $? -eq 0 ]; then
   echo quit | nvidia-cuda-mps-control
fi

# Test for presence of MPS zombie
ps aux | grep nvidia-cuda-mps | grep -v grep > /dev/null
if [ $? -eq 0 ]; then
   logger "`hostname` epilogue: MPS refused to quit! Marking offline"
   pbsnodes -o -N "Epilogue check: MPS did not quit" `hostname`
fi

# Check GPU sanity, simple check
nvidia-smi > /dev/null
if [ $? -ne 0 ]; then
   logger "`hostname` epilogue: GPUs not sane! Marking `hostname` offline"
   pbsnodes -o -N "Epilogue check: nvidia-smi failed" `hostname`
fi

5.3.1. MPS Server Not Accepting New Client Requests#
5.3.1. MPS 服务器不接受新的客户端请求 #

Description This error indicates that the MPS (Multi-Process Service) server is currently unable to accept new client connections. This is typically a temporary condition caused by one of the following known issues.
描述 此错误表明 MPS（多进程服务）服务器当前无法接受新的客户端连接。这通常是由以下已知问题之一引起的临时状况。

Possible Causes and Recommended Actions Server Recovery or Automatic Restart
可能的原因及推荐措施 服务器恢复或自动重启

Cause: The MPS server is recovering from a previous error or automatically restarting due to a kernel-level application failure.
原因：MPS 服务器正在从之前的错误中恢复，或由于内核级应用程序故障而自动重启。

Action: Wait for the server to complete its recovery. If these errors occur repeatedly, investigate the client application for underlying issues.
操作：等待服务器完成恢复。如果这些错误反复发生，请检查客户端应用程序是否存在潜在问题。

Server Initialization in Progress
服务器初始化正在进行中

Cause: The server is performing initialization operations equivalent to cuInit() --> cuCtxCreate(). On systems with multiple GPUs and no persistence daemon, this can take a significant amount of time. Any client connections attempted during this period will receive this error.
原因：服务器正在执行相当于 cuInit() --> cuCtxCreate() 的初始化操作。在拥有多个 GPU 且没有持久性守护进程的系统上，这可能需要相当长的时间。在此期间尝试的任何客户端连接都将收到此错误。

Action: Allow the initialization to complete. To speed up future startups, consider enabling the NVIDIA Persistence Daemon.
操作：允许初始化完成。为了加快未来的启动速度，请考虑启用 NVIDIA Persistence Daemon。

Client Termination Cleanup
客户端终止清理

Cause: A terminate_client operation is in progress. During this time, the server restricts new connections to clean up resources from a specific client.
原因：当前正在进行 terminate_client 操作。在此期间，服务器会限制新的连接，以清理特定客户端的资源。

Action: Wait for the cleanup process to finish. This does not affect other connected clients.
操作：等待清理过程完成。这不会影响其他已连接的客户端。

User ID Mismatch Without Multi-User Mode
在没有多用户模式下的用户 ID 不匹配

Cause: A client with a different user ID is attempting to connect while the server is not started with the -multi-user flag.
原因：一个具有不同用户 ID 的客户端正在尝试连接，而服务器未以 -multi-user 标志启动。

Action: Either start the MPS server with the -multi-user option or ensure that clients are using the same user ID. The default MPS behavior is to restart the server for each unique user ID.
操作：要么使用 -multi-user 选项启动 MPS 服务器，要么确保客户端使用相同的用户 ID。MPS 的默认行为是为每个唯一的用户 ID 重启服务器。