Published on JAN 21 2025 by James Smith

Can you cut the cord on AI subscriptions? Not so fast

Is that $25 / month susbscription to an AI chat service really worth it? What if you could do the same thing with just your computer and free software? Enter OLLAMA.

OLLAMA, short for Open Large Language Model Access, is a framework designed to make the use of large language models (LLMs) more accessible, flexible, and private. It emphasizes running LLMs locally on personal or organizational hardware, allowing users to avoid reliance on cloud-based services. By leveraging open-source tools and efficient implementations, OLLAMA enables users to deploy and fine-tune models directly on their own machines, offering benefits such as improved data privacy, reduced latency, and greater control over workflows. This approach caters to developers, researchers, and enthusiasts who value autonomy and seek cost-effective alternatives to subscription-based AI services.

Mac Users

Older Macs may struggle with the weight of running LLMs localy, but progressive improvements to the ARM based M chips have been helpful.

If you have a brand new MAC: Apple’s M4 chip series, including the M4, M4 Pro, and M4 Max, represents a significant leap in AI capabilities for Mac devices. Central to this advancement is the enhanced Neural Engine, capable of performing up to 38 trillion operations per second (TOPS), more than double the performance of its predecessor, the M3. This substantial increase enables more efficient handling of complex AI tasks, such as natural language processing and machine learning model training, directly on the device.

The M4 Pro and M4 Max chips further augment AI performance with increased memory bandwidth and capacity. The M4 Pro supports up to 64GB of unified memory with 273GB/s bandwidth, while the M4 Max supports up to 128GB of unified memory with 546GB/s bandwidth. This expanded memory facilitates the processing of large datasets and complex AI models, allowing developers to interact with models containing nearly 200 billion parameters.

Additionally, the M4 chip family introduces hardware-accelerated ray tracing and mesh shading, enhancing graphics rendering and AI-driven visual effects. These features, combined with the powerful Neural Engine, position the M4 chips as a robust platform for AI applications, providing users with advanced capabilities for both professional and personal use.

PC users

Gamers with NVIDIA graphics cards already have a leg up. You will be able to use local LLMs efficiently without much delay. For those with machines not optimized for gaming, open LLMs can be painfully slow.

Pro users

The new kid in town: Projet Digits from NVIDIA, a $3000 AI supercomputer running on a customized version of Ubuntu Linux. While Linux may not be for everybody, if you are serious about computing it should be your goto OS. Not only does this machine come with a full suite of specialized software from NVIDIA (local and cloud), you can use it for anything else, including OLLAMA.

Bottom Line

Unless you are a very heavy user of AI or you already own decent hardware, an investment in something like Digits is probably not as cost effective as just keeping your AI subscription alive. There are no worries about your rig becoming obsolete. However, if you’re in the market for a new machine, it may make sense to pay a bit more and get something than can handle running AI workloads. At the end of the day, choose whatever option best suits your budget.

Written by James Smith