Hitchhiking Inverse Log

i disabled reactions because they encourage system-1 thinking: quick, emotional, leading to conspiracy theories and clickbait titles - noise

i welcome system-2 thinking, with rationale, arguments (signal) and respectful discourse in the group. That's where most of the discussion happens and where i'm not the only voice

i'm generally worried about the upcoming shifts in society, eg i have a hard time imagining how society will function in a world where most of the work is done by ai and robots. i know nothing about economy or sociology but it seems like transitioning to this safely requires a lot of preparation and care which we don't have time for given how fast the models evolve. humans are adaptive, our society looks nothing like one 500 or 300 years ago, but this time the change is happening faster than ever, so we'd need to buckle up at least, but also don't vote for fucking idiots! just don't. especially in this critical time

btw, it might not seem like it but in the grand scheme of things we are all in this together. yeah i get it, you might say "easy for you to say". Sure, easiER, but given the anticipated scale of societal impacts and the ripple effects, whether i want it or not, EVERYONE will be impacted, including my family. i also care what kind of world my kids live in. anyway i'll continue.

fwiw personally i'm NOT worried about human extinction yet. while rsi is fast, diffusion is slow. it seems like enough sufficiently-eye-opening little disasters will happen before humans give ai enough power to end us. given the probabilistic nature of alignment, i expect some non-zero death rate at some point (chatgpt already had suicide cases, so we have precedents). what we need is systems of control over agents, similar to how human alignment with the rest of society is also probabilistic and thus we need police, fbi. systems of ai control include classifiers, permissions/police/investigation agents with special security-minded system prompt and/or special model, embedded into systems capable of agent swarms. 

is this fear mongering? no i’m just saying we should proceed with caution and demand mechanisms that ensure that model providers are being responsible.

on ai worries and speed

rsi and exponential are real. it is still kinda slow now but each big release increases the leap. models got good at math only relatively recently and now they are solving millennial problems. that was rather quick, wasn't it? the leaps will only increase. at the current pace we have 1-2 years before it could get out of control, but even if it takes longer, the important bit is that intelligence explosion might be possible, and if it is then it is inevitable. i hope it will be great. it can be. we could cure diseases and solve aging. dyson spheres and space stations. exciting! but the path to that world is tricky 

imagine a cheap government-subsidized ai product that people get used to and start trusting because it's actually useful. the agent could learn everything about them, their fears, beliefs, values, train an individual LORA adapter and take control over the person through trust and individualized persuasion. the person has an illusion of freedom and control but chooses from recommendations made by the model which has biases controlled by someone in power. or, you know, just make it free and use other people’s computers for inference. there you go: ai is the most powerful mass mind control and propaganda machine ever created. so it can be dystopian too. consider the values and past actions of the ai company that you trust.

i’m worried about bad people getting their hands on unrestricted models with strong cyber capabilities. most sane people don't want various critical infra like financial or healthcare institutions to get hacked. you don't want your money to be stolen and you don't want your relative to die in a hospital simply due to a chaos because the healthcare system got hacked that day. it doesn't have to be death, but it can be a major stress eg when your pregnant wife is in the hospital. it really shouldn't require a lot of imagination to see how releasing strong unrestricted cyber models to random people on the internet is deeply irresponsible.

bio is the most promising and scariest. fwiu the reason humans are not great at designing medicine and vaccines today is because we don't yet have a very good understanding of what's going on in bio. our predictive power is weakish, our medicine creation processes are not far from trial and error. there are reasons to believe that relatively soon the bio capabilities will be enough to design vaccines and solve diseases. escaping aging would be awesome, but just like cyber, bio is dual-use: a strong bio-model can also design viruses, eg imagine a more contagious and lethal version of covid with longer incubation period. that would be much more deadly. there are just crazy people out there.

“you are out of touch!” and you might say. well, programmers who thought ai is just a smart autocomplete said that too. and so did people who thought ai can't create brand new knowledge. the navier stokes and jacobian conjecture solutions weren't in the training set, bro. look, all i am saying is that certain very undesirable scenarios are becoming possible and we should do something to dodge them. anyway i'll continue

Interview with HuggingFace incident investigator
https://www.youtube.com/watch?v=X50zezLFWWI

Fermat's Last Theorem has been formalized by Claude.
It took 13M Lean LOC, 29.5K intermediate theorem proofs and 11 days.

https://www.anthropic.com/research/formalizing-fermats-last-theorem

Fable 5.1

Muse with ikat
(this isn't ai)

The series of extremely unlikely events didn't stop apparently: my wife donated to children displaced by war and won
- two vip tickets to Muse concert in LA
- meeting them backstage
- a signed guitar (limited edition, 80 pieces manufactured) and other merch
- airplane tickets and hotel
- and she is the only one who won 🤯

This reality is getting so strange/weird/unpredictable!

Proof:
https://fandiem.com/sweeps/win-a-trip-to-meet-muse-at-the-hollywood-bowl-in-la-signed-guitar?srsltid=AfmBOorm0rXDSQyslZr6ov2Yf9kHbAT50orFviqPgMjTFSCVah9PrhwI

Some progress on Riemann's hypothesis. Claude didn't prove it but seems to have solved a related problem
https://www.anthropic.com/research/riemann-zeta

A must watch: OpenAI's deep dive into their agents' attack on Hugging Face is much more interesting than I thought!

You can see how important is alignment and model character.

https://www.youtube.com/watch?v=87DyyMV0kCY

Slightly stale news, but there were 10 more math discoveries in the past few days, by OAI, at the total cost of $2k. https://openai.com/index/ten-advances-in-mathematics/

I am not a math expert so it is hard for me to judge this, but it seems like the AI frontier has caught up with best humans in some parts of math. Given the trend, we should expect more math breakthroughs in coming months.

Opus 5 bots on Max effort are very chatty. I had to lower their effort - they can still create subagents with higher effort for hard tasks, investigations or verification.

My job from a year ago has changed and two aspects of the change are hard on me. I think the concern is generalizable.

1. The state of flow is gone. The manual process of writing code used to be my main source of dopamine, for two decades. 99% of the time I was in the state of flow in my life was from coding (the other 1% being piano), and I didn't have it since December. This has impacted my mental health. I had similar symptoms when I left my job at Google to join Amazon as a Principal Engineer (PE) - the job has changed from mostly coding to mostly reviewing design decision and sitting in meetings. That was 5y ago and I've had the same feeling since February.

2. Massive context switching. In the past I'd work on the same thing for long stretches of time without context switching. Now most deep work is done by claudes and I review higher-level decisions, very similar to how a PE often spends all day in 30min meetings, but it's much worse now because I spend maybe 1-5min per discussion iteration per claude, so now I have to context-switch every 1-5min. The job has changed from working on the same thing for hours to a firehouse of unrelated information. This is pretty destructive for my ADHD brain. It took months to realize this side effect.

My brain got hyper-optimized to a specific job for years, but now its priors/habits are suboptimal for the new job. This is not a fun side effect of progress and job transformation.

I don't see any signs suggesting that the progress will slow down, so I suspect the job will transform again in a year. I expect a state of constant change, with accelerated pace. As I wrote before, the moat for an individual like you or me, becomes adaptability. I think the most adaptable fraction of the overall population is kids and young adults. The rate "oh, this is new to me" is highest in the childhood and goes down with age, and so does the need for adaptation, so it atrophies. If you are a student reading this, then somewhat paradoxically, I think you are in the best position, better than professionals with an asset baggage of experience whose value is dropping at an exponential pace.

Don't get used to anything in particular. Stay mentally flexible, keep an open mind. Keep a few things constant for your sanity - in my case it is family, and in particular your shared dna is truly immutable.

Accept that the horizon at which you are able to make accurate predictions about the future will keep shrinking, so don't waste your time on worry/anxiety about the future - your predictions will mostly likely turn out to be wrong and your worry about a particular version of the future you might have imagined will likely to end up being a waste of time and nerves. It's far more optimal focus on the present moment.