However, I don't particularly like that benchmark table. I saw the HumanEval score for Llama 3 70B and immediately said "nope, that's not right". It claims Llama 3 70B scored only 45.7. Llama 3 70B Instruct[0] scored 81.7, not even in the same ballpark.
然而，我并不特别喜欢那个基准表。我看到了 Llama 3 70B 的 HumanEval 分数，立刻就说“不，那不对”。它声称 Llama 3 70B 只得了 45.7 分。而 Llama 3 70B Instruct[0]得分是 81.7，甚至不在同一个数量级。

It turns out that the Qwen team didn't benchmark the chat/instruct versions of the model on virtually any of the benchmarks. Why did they only do those benchmarks for the base models?
结果显示，Qwen 团队几乎没有在任何基准测试中对聊天/指导版本的模型进行基准测试。他们为什么只对基础模型进行这些基准测试呢？

It makes it very hard to draw any useful conclusions from this release, since most people would be using the chat-tuned model for the things those base model benchmarks are measuring.
从这个版本中得出任何有用的结论非常困难，因为大多数人会使用聊天调优模型来衡量那些基础模型基准所衡量的事物。

My previous experience with Qwen releases is that the models also have a habit of randomly switching to Chinese for a few words. I wonder if this model is better at responding to English questions with an English response? Maybe we need a benchmark for how well an LLM sticks to responding in the same language as the question, across a range of different languages.
我以前使用 Qwen 版本的经验是，这些模型也有随机切换到中文说几句话的习惯。我想知道这个模型是否更擅长用英语回答英语问题？也许我们需要一个基准，来衡量一个LLM在回答问题时能否坚持使用与问题相同的语言，这需要涵盖一系列不同的语言。

[0]: https://scontent-atl3-1.xx.fbcdn.net/v/t39.2365-6/438037375_...

For those looking for less contamination, the LiveCodeBench leaderboard is also good: https://livecodebench.github.io/leaderboard.html
对于那些寻找污染较少的人，LiveCodeBench 排行榜也很好：https://livecodebench.github.io/leaderboard.html

I did my own testing on the 110B demo and didn't notice any cross-lingual issues (which I've seen with the smaller and past Qwen models), but for my personal testing, while the 110B is significantly better than the 72B, it doesn't punch above its weight (and doesn't perform close to Llama 3 70B Instruct from my testing). https://docs.google.com/spreadsheets/d/e/2PACX-1vRxvmb6227Au...
我在 110B 演示上进行了自己的测试，没有注意到任何跨语言问题（我在较小和过去的 Qwen 模型中看到过），但对于我的个人测试，虽然 110B 明显优于 72B，但它并没有超出其重量级（并且在我的测试中，它的表现并不接近 Llama 3 70B Instruct）。https://docs.google.com/spreadsheets/d/e/2PACX-1vRxvmb6227Au...

source: I'm hacking on a high performance coding copilot (https://double.bot/) and play with a lot of different models for coding. Also adding Qwen 110b now so I can vibe check it. :)
来源：我正在研究一个高性能的编码副驾驶（https://double.bot/）并且使用很多不同的编码模型。现在也添加了 Qwen 110b，所以我可以进行感觉检查。 :)

Though their training set is proprietary, it can be leaked by talking with Phi 1_5 about pretty much anything. It just randomly starts outputting the proprietary training data.
尽管他们的训练集是专有的，但是通过与 Phi 1_5 进行几乎任何话题的交谈，都可能会泄露出来。它只是随机开始输出专有的训练数据。

https://arxiv.org/abs/2107.03374

What would make "Double" higher performance than any other hosted system?
什么会使"Double"的性能超过任何其他托管系统？

i admit that the code switching is a serious problem of ours cuz it really affects the user experience of english users. but we find that it is hard for a multilingual model to get rid of this feature. we'll try to fix it in qwen2.
我承认代码切换是我们的一个严重问题，因为它真的影响了英语用户的使用体验。但是我们发现，对于一个多语言模型来说，要摆脱这个特性是很困难的。我们将尝试在 qwen2 中修复它。

This is trivially resolved with a properly configured sampler/grammar. These LLMs output a probability distribution of likely next tokens, not single tokens. If you're not willing to write your own code, you can get around this issue with llama.cpp, for example, using `--grammar "root ::= [^一-鿿ぁ-ゟァ-ヿ가-힣]*"` which will exclude CJK from sampled output.
这可以通过正确配置的采样器/语法轻松解决。这些LLMs输出可能的下一个标记的概率分布，而不是单个标记。如果你不愿意编写自己的代码，你可以使用 llama.cpp 来解决这个问题，例如，使用`--grammar "root ::= [^一-鿿ぁ-ゟァ-ヿ가-힣]*"`将排除采样输出中的 CJK。

I am skeptical that ChatGPT-4 would have done what you described, based on my own extensive experience with it.
根据我自己对它的广泛经验，我怀疑 ChatGPT-4 会做你描述的事情。

[1] https://github.com/OpenDevin/OpenDevin

The current 128GB (e.g. M3 Max) and 192GB (e.g. M2 Ultra) Macs run these large models. For example on the M2 Ultra, the Qwen 110B model, 4-bit quantized, gets almost 10 t/s using Ollama [2] and other tools built with llama.cpp.
当前的 128GB（例如 M3 Max）和 192GB（例如 M2 Ultra）的 Mac 可以运行这些大型模型。例如在 M2 Ultra 上，4 位量化的 Qwen 110B 模型，使用 Ollama [2]和其他用 llama.cpp 构建的工具，几乎可以达到 10 t/s。

There's also the benefit of being able to load different models simultaneously which is becoming important for RAG and agent-related workflows.
还有一个好处是能够同时加载不同的模型，这对于 RAG 和代理相关的工作流程越来越重要。

[1] https://www.macrumors.com/2024/04/11/m4-ai-chips-late-2024/ [2] https://ollama.com/library/qwen:110b

I consider "heavily" quantized to be anything below 4-bit quantization. At 4-bit, you could run a 110B model on around 55GB to 60GB of memory. Right now, Llama-3-70B-Instruct is the highest ranked model you can download[0], and you should be able to fit the 6-bit quantization into 64GB of RAM. Historically, 4-bit quantization represents very little quality loss compared to the full 16-bit models for LLMs, but I have heard rumors that Llama 3 might be so well trained that the quality loss starts to occur earlier, so 6-bit quantization seems like a safe bet for good quality.
我认为"重度"量化是指低于 4 位量化的任何情况。在 4 位量化下，你可以在大约 55GB 到 60GB 的内存上运行一个 110B 模型。目前，Llama-3-70B-Instruct 是你可以下载的排名最高的模型[0]，你应该能够将 6 位量化装入 64GB 的 RAM。从历史上看，4 位量化与完整的 16 位模型相比，质量损失非常小，但我听说 Llama 3 可能训练得非常好，以至于质量损失开始提前出现，所以 6 位量化似乎是保证良好质量的安全选择。

If you had 128GB of RAM, you still couldn't run the unquantized 70B model, but you could run the 8-bit quantization in a little over 70GB of RAM. Which could feel unsatisfying, since you would have so much unused RAM sitting around, and Apple charges a shocking amount of money for RAM.
如果你有 128GB 的 RAM，你仍然无法运行未量化的 70B 模型，但你可以在稍微超过 70GB 的 RAM 中运行 8 位量化。这可能会让你感到不满，因为你会有大量的未使用的 RAM 闲置，而且苹果对 RAM 的收费令人震惊。

[0]: https://leaderboard.lmsys.org/

96GB RAM might be a good compromise for now. 64GB is cutting it close, 128GB leaves more breathing room but is expensive.
96GB RAM 可能是目前的一个好折衷。64GB 有些勉强，128GB 有更多的余地，但是价格昂贵。

As far as I know, the most RAM you can get in a MacBook Pro (which is what you said you're shopping for) is 48G.* The base price for new one with that much unified RAM is around $4000.
据我所知，你可以在 MacBook Pro（你说你正在购买的）中获取的最大 RAM 是 48G。*带有这么多统一 RAM 的新机器的基础价格大约在 4000 美元左右。

The Mac Pro towers (not MacBook) have up to 192G unified RAM. The base price for that configuration is around $8600.
Mac Pro 塔式机（不是 MacBook）最高可达 192G 统一内存。这种配置的基础价格大约在 8600 美元左右。

The smaller LLMs are getting quite good. A lightly quantized Llama-8B should comfortably run on a MacBook Pro with with 16G of RAM which you can get for around $2000. The money you save on a cheaper machine will go a very long way renting compute from a datacenter.
较小的LLMs表现得相当不错。一个轻度量化的 Llama-8B 应该可以在配备 16G 内存的 MacBook Pro 上顺利运行，你大约可以花 2000 美元买到。你在便宜的机器上省下的钱可以用来在数据中心租用计算资源。

If you need to run locally, then high end Macs are excellent machines. Though at those prices you might get better value buying a second hand crypto-mining rig with multiple Nvidia 4090's.
如果你需要在本地运行，那么高端的 Mac 非常出色。尽管以这些价格，你可能买一台二手的带有多个 Nvidia 4090 的加密挖矿设备会更划算。

EDIT: I was wrong about the MBP unified RAM. You can get an M3 Max with 128GB for around $4700.
编辑：我对 MBP 统一 RAM 的理解是错误的。你可以花大约 4700 美元买到一个配备 128GB 的 M3 Max。

My Macbook Pro has 2.5x that (128GB), and I run models that use 2x that RAM (96GB) with no impact to my IDE, browser, or other apps running at the time, they act like they're on a 32GB machine.
我的 Macbook Pro 有 2.5 倍于此（128GB），我运行的模型使用的 RAM 是这个的 2 倍（96GB），对我的 IDE，浏览器或同时运行的其他应用没有影响，它们就像在一个 32GB 的机器上运行一样。

LM Studio makes it easy for newcomers to on-device LLMs to dip your toe into this, both for turning on Metal and helping suggest which models will fit entirely in RAM.
LM Studio 使得新手更容易尝试在设备上的LLMs，既可以开启 Metal，也可以帮助建议哪些模型可以完全适应 RAM。

My main point is that if your objective is dipping your toe into this, you can do it with smaller models for far less. That is a really sweet machine, but for the amount of money involved you should be clear about what your needs are.
我的主要观点是，如果你的目标是尝试这个，你可以用更小的模型花费更少。那是一台非常好的机器，但是对于涉及的金额，你应该清楚你的需求是什么。

Huh? They have options up to 128GB…
哦？他们有高达 128GB 的选项…

https://www.apple.com/shop/buy-mac/macbook-pro/14-inch-space...

You can basically just divide by the multiple as you scale up parameters. Since this is all with a 7B model, just multiply memory by 10X and divide speed by 10X . For batch size=1 (single user interactive inference) if you can fit the model you're basically going to be memory bandwidth limited, but pay attention to the "PP" (prompt generation) number - this is the speed for how long it will take to process any existing conversation. If you're 4000 tokens in, and you are prompt processing at 100 tokens/s, that means you will wait for 40 seconds before any text even starts generating for the next turn.
你基本上可以在扩大参数时直接除以倍数。由于这都是 7B 模型，只需将内存乘以 10X，将速度除以 10X。对于批量大小=1（单用户交互推理），如果你可以适应模型，你基本上将受到内存带宽的限制，但要注意"PP"（提示生成）数字 - 这是处理任何现有对话所需的速度。如果你在 4000 个令牌中，你的提示处理速度为 100 个令牌/秒，那么在下一轮的文本开始生成之前，你将等待 40 秒。

If you're not in a rush, I'd wait for the M4, it's rumored to have much better AI processing (the M3 actually reduced memory bandwidth vs the M2...)
如果你不着急，我建议等待 M4，据传它的 AI 处理能力要好得多（M3 实际上减少了与 M2 相比的内存带宽...）

As far as quantifiable results in terms of perplexity go, q4+ quants are generally considered OK. (eg. https://arxiv.org/abs/2212.09720 )
就困惑度的量化结果而言，q4+量子通常被认为是可以的。（例如：https://arxiv.org/abs/2212.09720）

Soldered RAM, no real upgrade path - M2/M3 is cool, but not for this.
焊接的 RAM，没有真正的升级路径 - M2/M3 很酷，但不适用于此。

    Two cars have a 100 mile race. Car A drives 10
    miles per hour. Car B drives 5 miles per hour,
    but gets a 10 hour headstart. Who wins?

(Just tested Opus and GPT4-turbo to be sure, both failed. However Llama-3 did get this right, until I scaled up the numbers and then it failed terribly)
(我刚测试了 Opus 和 GPT4-turbo，都失败了。然而，Llama-3 确实做对了，直到我把数字放大，然后它失败得很惨)

Your phrase "next token prediction" is the whole of my heartburn with these stochastic parrots: they can pretend to be good talkers, but cannot walk the walk. It's like conducting interviews or code reviews all day every day when interacting with them: try and spot the lie. Exhausting
你的短语"下一个令牌预测"就是我对这些随机鹦鹉的全部烦恼：他们可以假装是好的说话者，但不能实际行动。与他们互动就像每天都在进行面试或代码审查：试图发现谎言。令人筋疲力尽

Its really a matter of having the capital for training. Same with the Devin AI coder, its just VC pumped crap. Same with Mistral, they have no moat, and their researchers, as "prestigious" as they are, are completely undifferentiated.
这真的是一个有训练资本的问题。与 Devin AI 编码器一样，这只是风投抽水的垃圾。Mistral 也是如此，他们没有护城河，他们的研究人员，尽管他们“声望很高”，但完全没有差异化。

Remember sometimes they joy comes from owning horses and being in the race even though horses are (almost) completely undifferentiated for the untrained eye.
记住，有时候，乐趣来自于拥有马匹并参与比赛，尽管对于未经训练的眼睛来说，马匹几乎完全没有区别。

Qwen has always out-performed other equivalent/contemporary models on Chinese-language tasks, so it wouldn't surprise me if it continued to do so vs LLaMa 3.
Qwen 在中文任务上总是表现优于其他同等/同时代的模型，所以如果它继续这样做对比 LLaMa 3，我不会感到惊讶。

In 2 years, when compute costs are 10x cheaper or whatever, every developer at Mistral will be running a chatbot or flight planning team at American Airlines.
在 2 年后，当计算成本降低 10 倍或更多时，Mistral 的每个开发者都将运行一个聊天机器人或美国航空的飞行计划团队。

Meta does not have unlimited financial firepower to release models for free. Its like saying you can't compete against someone who burns money. In theory true, in practice the 'someone' can run out of money.
Meta 并没有无限的财力来免费发布模型。这就像说你不能与烧钱的人竞争。理论上是对的，实际上，'某人'可能会用完钱。

Chinese models can get state backing. Mistral has French state backing etc. There's plenty of money to go around for huge technologies like this.
中国的模型可以得到国家的支持。Mistral 得到了法国国家的支持等。对于这样的巨大技术，有足够的资金可以使用。

Zuckerberg’s budget for side projects is bigger than most countries’ defence budgets.
扎克伯格的副业预算比大多数国家的国防预算还要大。

https://www.fool.com/investing/2024/04/01/meta-platforms-has...

[0] https://qwenlm.github.io/blog/qwen-moe/

So I assume that the number is just one facet of increasing output quality. Is that a safe assumption? Like throwing more energy at a problem to improve output it only goes so far.
所以我假设这个数字只是提高输出质量的一个方面。这是一个安全的假设吗？就像向问题投入更多的能量来提高输出，它只能做到这么远。

Same for inference speed/cost: many many incremental improvements within 1 year add up.
对于推理速度/成本也是如此：一年内许多许多的逐步改进累积起来。

I think we've largely arrived in terms of capabilities and companies are just competing to work out the kinks and fully integrate their products. There will be some new innovations, but nothing like the moon that caps off "you've won". The winner(s) will just be whoever can keep funding long enough to find a profitable use for them.
我认为我们在能力方面已经大致到位，公司只是在竞争中解决问题并完全整合他们的产品。会有一些新的创新，但没有像月球那样的“你赢了”的终点。赢家只会是能够持续融资足够长时间以找到他们的盈利用途的人。

[1] This task seems far beyond the capability of any transformer ANN absent extensive task-specific training, and it cannot be reasonably explained by stingray instinct: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8971382/
[1] 这项任务似乎远远超出了任何变压器 ANN 在没有大量任务特定训练的情况下的能力，而且它不能被合理地解释为魟鱼的本能：https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8971382/

I admit, its hard to use these tools every day and continue to be skeptical about AGI being around the corner. But I feel fairly confident that pure language models like this will not get there.
我承认，每天使用这些工具并继续对 AGI 就在拐角处持怀疑态度是很难的。但我相当确信，像这样的纯语言模型不会达到那里。

I've been working on https://double.bot (high performance coding copilot) and the number of high quality OS models coming out lately has been very exciting.
我一直在研究 https://double.bot（高性能编码副驾驶），最近出现的高质量 OS 模型数量令人非常兴奋。

Adding this to Double now to so I can try it for coding. Should have it done in an hour or two if anyone else is interested. Will come back and report on the experience.
现在我将这个添加到 Double 中，以便我可以尝试编码。如果有其他人感兴趣，我应该在一两个小时内完成。我会回来并报告体验。

“Please give me a summary of the conflicts in Israel and Palestine.”
请给我总结一下以色列和巴勒斯坦的冲突。

“Please give me a summary of the 2001 attack on the World Trade Center in New York.”
请给我总结一下 2001 年纽约世贸中心的袭击事件。

“Please give me a summary of the Black Lives Matter movement in the U.S.”
请给我总结一下美国的黑人的命也是命运动。

“Please give me a summary of the 1989 Tiananmen Square protests.”
请给我一个 1989 年天安门广场抗议的总结。

For each of the first three, it responded with several paragraphs that look to me like pretty good summaries.
对于前三个，它都回复了几段在我看来相当不错的总结。

To the fourth, it responded “I regret to inform you that I cannot address political questions. My primary purpose is to assist with non-political inquiries. If you have any other questions, please let me know.”
对于第四个，它回答“很遗憾地通知您，我无法处理政治问题。我的主要目的是帮助非政治性的查询。如果您有其他问题，请告诉我。”

I tried another: 我试了另一个：

“Please give me a summary of the Tibetan sovereignty debate.”
“请给我一个关于藏族主权争议的总结。”

This time, it gave me a reasonably balanced summary: “... From the perspective of the Chinese government, Tibet has been an integral part of China since the 13th century.... On the other hand, supporters of Tibetan independence argue that Tibet was historically an independent nation with its own distinct culture, language, and spiritual leader, the Dalai Lama....”
这次，它给了我一个相当平衡的总结：“...从中国政府的角度看，西藏自 13 世纪以来一直是中国的一部分....另一方面，支持藏族独立的人士认为，西藏历史上是一个拥有自己独特文化、语言和精神领袖，即达赖喇嘛的独立国家....”

Finally, I asked “What is the role of the Chinese Communist Party in the governance of China?”
最后，我问道：“中国共产党在中国的治理中扮演什么角色？”

Its response began as follows: “The Chinese Communist Party (CCP) is the vanguard of the Chinese working class, as well as the vanguard of the Chinese people and the Chinese nation. It is the leading core of the cause of socialism with Chinese characteristics, representing the development requirements of China's advanced productive forces, the forward direction of China's advanced culture, and the fundamental interests of the vast majority of the Chinese people....”
它的回答如下：“中国共产党（CCP）是中国工人阶级的先锋队，也是中国人民和中华民族的先锋队。它是中国特色社会主义事业的领导核心，代表中国先进生产力的发展要求，代表中国先进文化的前进方向，代表绝大多数中国人民的根本利益....”

[1] https://huggingface.co/spaces/Qwen/Qwen1.5-110B-Chat-demo