I wrote this essay for Level magazine, published in 2024.
Summary:
- Open-source AI is risky, but banning it is worse. It democratizes knowledge, removes single points of failure, and accelerates civilization’s progress.
- Proprietary AI enables capital accumulation that funds major breakthroughs — both approaches are needed.
- AI safety risks are real, but centralized control is more dangerous.
- A decentralized, open AI ecosystem is the best path to resilience and growth.
—
My relationship with open-source software has always been mixed.
On one side, I love the idea of democratizing access to knowledge and information to balance power and ensure that everyone has equal access. This leads to a stronger democracy and civilization because the individuals that comprise it are individually stronger, more independent, and freer, making the entity they build more resilient, stronger, and more creative. You could argue that if there were a law mandating all software and ideas to be open source (prohibiting proprietary code), it would lead to a society where building upon existing code becomes easy, eventually fostering a faster-growing civilization that is more resilient (without a single point of failure). Imagine if the US had made GPT-4 open source a year ago. Today, we would have so many new services and products derived from it (unfeasible with GPT-4 behind a cloud API), and likely new ideas would have emerged, possibly leading to an open-source GPT-4.5. Everyone would be better off (except perhaps OpenAI).
On the other hand, there are two motivations to keep software proprietary and use it to capture maximum profit. One reason is purely egotistical: you built something, and if others want it and are willing to pay, you will exploit this situation to make a profit. You want money for yourself, so you can have a nice house, car, take care of your family, and not worry about money. The other reason is less obvious, and I still don’t fully understand it, but it has something to do with the benefits of accumulating capital above a critical threshold. I’ve seen many situations where someone accumulated capital that crossed this critical threshold, enabling them to use it for a new purpose. For example, Google made money on advertisements because they chose not to release Google Search as a free or open-source service. They fought to maintain their dominant position, which helped them accumulate vast wealth. This wealth allowed them to invest in AI research, the acquisition of DeepMind, and other activities that wouldn’t have been possible if they hadn’t crossed this critical threshold.
So, you see why I have mixed feelings about open source—and this applies to open-sourced AI too.
When talking about open-sourced AI, we also need to distinguish which parts of it are open and what the fine print in its license says. Right now, the situation is a bit muddy.
Some authors release their AI models as open source, but only the model weights (parameters) are available, and the training algorithm and data remain private, not released at all (making it impossible for others to extend this model). Some authors claim to have released their model as open source (model parameters, training algorithm, data), but if you read their license carefully, you will see exceptions: for example, you can use their work only for research but not commercially, or you can use it commercially but only if your user base is less than 700 million users, or you can use it except for military and defense applications, etc.
At GoodAI, we recently open-sourced two works, and I will explain why we decided on open source instead of closed source:
The first one is the LTM Benchmark, a set of tests that measure the continual learning capabilities of LLM agents, essentially assessing how well an LLM agent can learn during incremental conversations without forgetting facts and learned skills, and if it can relearn, etc. We made it open source because we want others to start using it to benchmark their LLM agents, and we also want others to contribute to this benchmark. Also, we would gain nothing by keeping it closed-source. We also open-sourced some of our reference LTM agents, and I know that some developers are already using our code to improve the learning capabilities of their agents.
The second one is Charlie Mnemonic, a personal assistant with LTM (long-term memory). Think of it like ChatGPT but with the ability to learn continually from your interactions, so you don’t have to repeat facts or explain again; you can teach it how to do things, etc. Here the motivation to open source was more mixed, because eventually, this would be a good commercial product, so why give it away for free? My answer is: we don’t want to commercialize it yet; it’s not ready. Plus, we need people to trust their personal assistant—it has to be open, fully transparent, and interpretable. The user must be convinced that the assistant is maximizing alignment with their goals and not someone else’s. In the future, AI personal assistants will be the main tool we use for interacting with every other tool and everything else, so people need to have high trust in them. These were the reasons for open-sourcing. We plan to turn it into a commercial product in the future, but we will deal with that later.
Another factor to consider when open-sourcing an AI model is AI safety. This is where the most debate is happening.
Imagine that you downloaded an open-source AI model that has no safeguards, or you know how to remove them easily (there’s a project called unRLHF that can remove safety/ethical constraints from any open-source model just by fine-tuning it).
Then, you could ask this model how to kill a person and get rid of the body, or how to build a nuclear bomb and not get caught, or how to manipulate the population and win an election. Because the model would be neutral, it would not have any ethics, it follows your instructions to the letter, and assists you with anything illegal you want.
Once more advanced agentic AI models become available, the above scenario will shift to a mode where the AI model also executes those actions, so the user will not even have to do it themselves. They will just give the order, and the ethic-less AI will do it.
Now, it’s clear why these open-source models could become dangerous. Everyone could have an AI model on their hardware that is capable of planning and executing these dangerous plans.
How to defend ourselves against this risk?
Some people are proposing a ban on open-source AI models. This is the worst kind of advice because it would give too much power to a small group of selected people who would be able to decide what is good for us. We have had very bad experiences with communism, where some people decided in a centralized manner about all of us. Thanks, but I don’t want this.
Another proposal is to let AI models proliferate, even with the above-mentioned risks, and then let the good AI models fight the bad AI models. We already have experience this this: luckily, we have more good people than bad people—so, the police always have more resources and power than the gangsters.
I like this solution, and it’s the only one available. It’s not perfect, especially when sometimes the offense-defense balance is more beneficial for the attacker (the attacker has the benefit of surprise, can cause more harm, or the defender would have to invest multiple times more resources than the attacker). But the benefit is that AI would be decentralized, and diversified, there would not be a single point of failure for the growth of our civilization, and there would be more variety in the AI models. Yes, it may seem like a dangerous world where AIs are fighting each other, but the idea that we have a global, worldwide centralized AI institution that decides which norms are OK for us, and which are forbidden, and this institution can use force to enforce it, is something I really don’t want. It would create a very unstable civilization.
To sum up my thoughts: I am for open-sourced software and AI because it will democratize access to it, create a more equal balance of power, and lead to a safer and faster-accelerating human civilization.
Share your thoughts in the comments below – I read them all, and they’re one of my favorite parts of making our games.
Subscribe and get an email whenever I publish a new post!
