Principal Research Scientist, CORAL
Trying to figure out how to make Artificial Superintelligence aligned with human preferences. I believe that this will require a very different approach to designing AI systems, rooted in deep theoretical understanding from the ground up.
The foundation will be the emerging computational agent theory: a field that builds on computational learning theory, control theory, decision theory and other disciplines to answer questions such as:
- What does it mean for a system to be an autonomous intelligent agent?
- What does it mean for an agent to possess preferences or beliefs?
- How can algorithms produce intelligent agency in the face of information-theoretic and complexity-theoretic constraints?
Research
Compositional learning theory is trying to explain how machine learning algorithms can exploit compositional structure in the data to learn sophisticated models while retaining statistical and computational efficiency. Previous approaches in AI and machine learning all fall into one of three categories: (i) works in practice, but we don’t understand it (deep learning), (ii) works in theory, but is completely impractical to implement (e.g. Solomonoff induction), (iii) works in theory and practice, but can only learn simplistic models (e.g. linear classifiers, kernel methods, finite-state methods). The goal of this research direction is finding the missing center of this Venn diagram: a theory of generalization for realistic and powerful learning methods.
Formal computational realism is a new approach to understanding agency, drawing on ideas from Functional Decision Theory and Structural Realism in analytic philosophy to create a rigorous mathematical framework that can deal with conceptual challenges such as:
- The complexity of the real world means that no algorithm will be able to perfectly learn and simulate it (nonrealizability, misspecification).
- The real world contains irreversible events which rule out learning in the usual sense of RL theory.
- Agent preferences and beliefs should be grounded in a realist ontology rather than in the agent’s own action-observation stream, and so should the inductive bias.
Selected work
- Stringological Sequence Prediction II — arXiv, 2026
- Stringological Sequence Prediction I — arXiv, 2026
- Regret Bounds for Robust Online Decision Making, with Alexander Appel — COLT, 2025
- Imprecise Multi-Armed Bandits — JMLR, 2025
- The Learning-Theoretic Agenda: Status 2023 — Alignment Forum, 2023
- Infra-Bayesian Physicalism, with Alexander Appel — Alignment Forum, 2021
Full list of papers on Google Scholar and posts on the Alignment Forum.
Hobbies
In my free time, I enjoy reading books, visiting art galleries and playing board-games with friends (even though I can be a sore loser). My favorite authors include Brandon Sanderson, Kate Elliot, Sharon Shinn, Diana Gabaldon and Jacqueline Carey. I enjoy all sorts of scifi/fantasy, but romantasy will always be my guilty pleasure!
Contact
I’m always interested in discussing AI safety, including professional advice and possible collaborations, so don’t be shy to reach out. Swapping book recommendations is also a solid reason ;)
PGP key
Fingerprint: 9B8B 9C42 C44E 5B92 9A4B DA3D 88C3 7785 C7AA BCC8
-----BEGIN PGP PUBLIC KEY BLOCK-----
mDMEaqvRDhYJKwYBBAHaRw8BAQdAPVldwTgYL3axTaKQUEiMSAQSwrRfHty5Fgmj
79UlafO0JFZhbmVzc2EgS29zb3kgPHZhbmVzc2FAYWx0ZXIub3JnLmlsPoi1BBMW
CgBdFiEEm4ucQsROW5KaS9o9iMN3hceqvMgFAmqr0Q4bFIAAAAAABAAObWFudTIs
Mi41KzEuMTIsMiwxAhsDBQlLMAwABQsJCAcCAiICBhUKCQgLAgQWAgMBAh4HAheA
AAoJEIjDd4XHqrzIm04A/jP60/PK0vAG3iUjEQI8qZ2v9Li52OxpVWVOuazKy1U0
AQCc2BN0RLwEbxpPmI0Gsc5VzXCv5YgIjbFw2/WCzVx9B7g4BGqr0Q4SCisGAQQB
l1UBBQEBB0Cf3TLz0Aan1MhUPKjzQ1rncdn0VYT1AmONUL1ZeFu6TgMBCAeImgQY
FgoAQhYhBJuLnELETluSmkvaPYjDd4XHqrzIBQJqq9EOGxSAAAAAAAQADm1hbnUy
LDIuNSsxLjEyLDIsMQIbDAUJSzAMAAAKCRCIw3eFx6q8yE2VAQDEPSeqDbP277Dz
dIAs5Cqt5goyEXBdwB7P+plfa7hhaQD/YkgMhozKxl+cWBgPA2yYnDwuj/XObERK
lwIavEnN2wI=
=X/s5
-----END PGP PUBLIC KEY BLOCK-----
Signed commitment
The following statement, originally posted on LessWrong, is signed with the key above.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512
I precommit that if superintelligent AI is created in the future which is partially or fully aligned to me (Vanessa Kosoy), then it will, on my behalf, fairly compensate anyone, human or AI, who helped steer probability towards AI outcomes favorable to me, and doesn't already have proportional representation inside superintelligence. This applies to the extent that the helper has coherent preferences s.t. the helper can be meaningfully compensated.
This policy is not special to ASI creation - I do my best to deal fairly with everyone in all contexts. It just seemed worth underscoring in this context specifically.
- -- Vanessa Kosoy
Original public statement: https://www.lesswrong.com/posts/dPmmuaz9szk26BkmD?commentId=ZLGLCgPb6JYibsQHu
-----BEGIN PGP SIGNATURE-----
iJEEARYKADkWIQSbi5xCxE5bkppL2j2Iw3eFx6q8yAUCaqvYEhsUgAAAAAAEAA5t
YW51MiwyLjUrMS4xMiwyLDEACgkQiMN3hceqvMj4vwD9ESehack7N/yMnjjXS5mw
oJ0gRpgccbNf+2dN8EtACMEA/j6V/JCQ+HPHxNs3E818X/LiABTpxulYVdzKcVFh
AyoO
=KFhu
-----END PGP SIGNATURE-----
using WordPress and