Rendered at 13:00:52 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
locust_p 4 days ago [-]
I have 2 HP DL380s with 2 Xeon Gold 6252 CPU @ 2.10GHz and 256gb of DDR4 memory that I bought for $700 a piece off ebay. I run 4 gpt-oss:120b models on CPU using 120GB memory for each VM. I get about 10-15 tokens/second for each virtual machine running ollama in cpu mode. I run these over night for complicated tasks. I have a multi agent system that will kick out the hard tasks the RTX 5070ti 16 and macbook pro m1 max 32gb can't do to the CPU farm. Works pretty well.
tarassenkov 19 hours ago [-]
Hi everyone, here’s a link to a local AI chat that you can test out on an iPhone, if you’re interested.
There are only 3 models available right now. I tested them for a couple of days, and on my iPhone 13 Pro, GigaChat was the one that worked the most reliably.
eqvinox 3 days ago [-]
Does this translate to older generation 16GB GPUs, e.g. RTX 3080 / RX 6950? (Assuming BF16 is available, or possibly even FP16?)
diabolic1 18 hours ago [-]
I haven't tried a 3080, but I rented a 16GB 4060 Ti to see how these MoEs behave. gpt-oss 20B actually fits, and I got around 100 tok/s. Qwen3.6 35B doesn't fit, and with part of it in system RAM I was still around 40.
tarassenkov 4 days ago [-]
A small update: I've also managed to port the engine to iOS.
I'm testing it on my old iPhone 13 Pro (A15 Bionic, 6GB RAM) with Ling-mini-2.0, a 16B MoE model. The first tests are showing around 13 tokens/sec, which is pretty encouraging for a phone from 2021.
It's still early, and I need to test memory usage, longer generations, and how performance changes as the phone heats up.
I'll keep posting updates here as I run more tests and see how far I can push it.
sebastienburel 3 days ago [-]
[flagged]
catalinbraescu2 3 days ago [-]
How about you post a real repo before making any claims?
hex4def6 4 days ago [-]
Cool, I guess.
I guess I'm not really sure what the point of this (100% AI generated) article is.
You have a magic engine with zero details, that you're comparing against a (to me at least) obscure github project.
I guess I'm not really sure what the takeaway for me is supposed to be. The benchmark setup is not detailed enough to make meaningful conclusions from it, and I guess the big takeway seems to be that a bounded cache that an engine process owns behaves better than relying on the system cache?
Paste this into the claude session:
"Add a section explaining the architecture of this engine, and how it compares to TUFF, EdgeMOE, etc SOTA implementations. Explain the novelty of this implementation. Show how experts are imported / evicted, and how that decision works. Also add more detail in a benchmark section about the exact testing you ran. Give test runs, upload to a git repo that links to this article. Include some charts showing test to test variance. "
tarassenkov 3 days ago [-]
[dead]
raver1975 3 days ago [-]
I'm running Deepseek 4.1 Flash in 4 gigabytes, but I'm not going to tell you how.
JPLeRouzic 3 days ago [-]
In how many hours per token?
fouc 3 days ago [-]
I believe he's joking, he's saying the OP article doesn't actually tell us the how.
https://github.com/MiTLN/OwnOwl-iPhone
There are only 3 models available right now. I tested them for a couple of days, and on my iPhone 13 Pro, GigaChat was the one that worked the most reliably.
I'm testing it on my old iPhone 13 Pro (A15 Bionic, 6GB RAM) with Ling-mini-2.0, a 16B MoE model. The first tests are showing around 13 tokens/sec, which is pretty encouraging for a phone from 2021.
It's still early, and I need to test memory usage, longer generations, and how performance changes as the phone heats up.
I'll keep posting updates here as I run more tests and see how far I can push it.
I guess I'm not really sure what the point of this (100% AI generated) article is.
You have a magic engine with zero details, that you're comparing against a (to me at least) obscure github project.
I guess I'm not really sure what the takeaway for me is supposed to be. The benchmark setup is not detailed enough to make meaningful conclusions from it, and I guess the big takeway seems to be that a bounded cache that an engine process owns behaves better than relying on the system cache?
Paste this into the claude session: "Add a section explaining the architecture of this engine, and how it compares to TUFF, EdgeMOE, etc SOTA implementations. Explain the novelty of this implementation. Show how experts are imported / evicted, and how that decision works. Also add more detail in a benchmark section about the exact testing you ran. Give test runs, upload to a git repo that links to this article. Include some charts showing test to test variance. "