Peter Norvig wrote today: “I am now done comparing three LLMs to my own coding on the Advent of Code problems. The LLMs did great! They couldn’t have done it last year.” Here are his main conclusions after asking 3 different LLMs to solve 12 puzzles:
- Overall, the LLMs did very well, producing code that gives the correct answer to every puzzle.
- I’m beginning to think I should use an LLM as an assistant for all my coding, not just as an experiment like this.
- This is a huge improvement over just one year ago, when LLMs could not perform anywhere near this level.
- The three LLMS seemed to be roughly equal in quality.
- The LLMs knew the things you would want an experienced software engineer to know
See the full analysis at https://lnkd.in/gCc2iuPK