on ai worries and speed

rsi and exponential are real. it is still kinda slow now but each big release increases the leap. models got good at math only relatively recently and now they are solving millennial problems. that was rather quick, wasn't it? the leaps will only increase. at the current pace we have 1-2 years before it could get out of control, but even if it takes longer, the important bit is that intelligence explosion might be possible, and if it is then it is inevitable. i hope it will be great. it can be. we could cure diseases and solve aging. dyson spheres and space stations. exciting! but the path to that world is tricky 

imagine a cheap government-subsidized ai product that people get used to and start trusting because it's actually useful. the agent could learn everything about them, their fears, beliefs, values, train an individual LORA adapter and take control over the person through trust and individualized persuasion. the person has an illusion of freedom and control but chooses from recommendations made by the model which has biases controlled by someone in power. or, you know, just make it free and use other people’s computers for inference. there you go: ai is the most powerful mass mind control and propaganda machine ever created. so it can be dystopian too. consider the values and past actions of the ai company that you trust.

i’m worried about bad people getting their hands on unrestricted models with strong cyber capabilities. most sane people don't want various critical infra like financial or healthcare institutions to get hacked. you don't want your money to be stolen and you don't want your relative to die in a hospital simply due to a chaos because the healthcare system got hacked that day. it doesn't have to be death, but it can be a major stress eg when your pregnant wife is in the hospital. it really shouldn't require a lot of imagination to see how releasing strong unrestricted cyber models to random people on the internet is deeply irresponsible.

bio is the most promising and scariest. fwiu the reason humans are not great at designing medicine and vaccines today is because we don't yet have a very good understanding of what's going on in bio. our predictive power is weakish, our medicine creation processes are not far from trial and error. there are reasons to believe that relatively soon the bio capabilities will be enough to design vaccines and solve diseases. escaping aging would be awesome, but just like cyber, bio is dual-use: a strong bio-model can also design viruses, eg imagine a more contagious and lethal version of covid with longer incubation period. that would be much more deadly. there are just crazy people out there.

“you are out of touch!” and you might say. well, programmers who thought ai is just a smart autocomplete said that too. and so did people who thought ai can't create brand new knowledge. the navier stokes and jacobian conjecture solutions weren't in the training set, bro. look, all i am saying is that certain very undesirable scenarios are becoming possible and we should do something to dodge them. anyway i'll continue