How to overhaul the broken review system

TLDR: After briefly outlining the problems with the current review ecosystem, I detail in-depth a research & review platform which addresses these points and seems plausible to me. I further illustrate 4 imaginary researchers to better understand what this would mean in practice and try answering some additional questions the reader might have, like how this would actually be implemented. I don't know a single person who would argue that the current AI research ecosystem is working out. Conferences are filled to the brim with meaningless work, AI generated slop is all over the place, hardly anything is fully reproducible, reviewers have no clue what they are talking about, authors have no clue what they are talking about, and the list goes on. There are many things I believe need to be fixed, ranked by their importance: People must take responsibility for their own work. This includes facing meaningful backlash when producing bad or simply meaningless work - and the same applies to reviews! People must be incentivized to favor quality over quantity. The rating of a work should be weighted towards the judgment of people who have already shown their own expertise (and keep displaying it in recent works) Even though ML lends itself extremely well to reproducibility, as opposed to other sciences like Biology, the current review process never tries verifying any claims. Especially now, where LLMs are generally at a point that they can reproduce a lot of work, it seems absurd to not make use of this and try to automatically verify claims. ML is a very fast-paced space and yet there are only a few dates per year when one can submit his work for review. Good science is not done on the clock and yet when it is finished, it should quickly be accessible & judged. I believe hardly any of these points are very controversial - so how is it that we are somehow still stuck with the current system? I think for the most part, there haven't been any reasonable attempts to drastically supersede the established system - the fact of the matter is that careers in academia are often still bound to these few special conferences and going against the flow isn't realistic. Which is exactly why I think that the current situation might be perfect for it: if you believe there is a decent chance doom is coming, it doesn't take a hero to not care all that much about how many citations you have and which 4-letter combination you won in the conference lottery, but instead to focus on your work making an actual positive impact. In the rest of this post I'll explain and illustrate a system that I believe addresses the previous points and is plausible to achieve. Overview Posts are scored using a universal scale from 1 to 10. This might roughly equate to: 1 - terrible, shouldn't be on this platform 3 - requires significant overhaul / uninteresting 5 - interesting but not significant 7 - significant for a small space 9 - significant for a big space Alike to a review, people can leave a rating with an explanation, or instead simply discuss in the comments. A post's rating then becomes the weighted mean of all current reviews. This weight is composed of a self-confidence assessment of one's review and one's expertise. The expertise is quantified by how well one's own posts did. A specific example of what this could look like: Let mjx-container[jax="CHTML"] { line-height: 0; } mjx-container [space="1"] { margin-left: .111em; } mjx-container [space="2"] { margin-left: .167em; } mjx-container [space="3"] { margin-left: .222em; } mjx-container [space="4"] { margin-left: .278em; } mjx-container [space="5"] { margin-left: .333em; } mjx-container [rspace="1"] { margin-right: .111em; } mjx-container [rspace="2"] { margin-right: .167em; } mjx-container [rspace="3"] { margin-right: .222em; } mjx-container [rspace="4"] { margin-right: .278em; } mjx-container [rspace="5"] { margin-right: .333em; } mjx-container [size="s"] { font-size: 70.7%; } mjx-container [size="ss"] { font-size: 50%; } mjx-container [size="Tn"] { font-size: 60%; } mjx-container [size="sm"] { font-size: 85%; } mjx-container [size="lg"] { font-size: 120%; } mjx-container [size="Lg"] { font-size: 144%; } mjx-container [size="LG"] { font-size: 173%; } mjx-container [size="hg"] { font-size: 207%; } mjx-container [size="HG"] { font-size: 249%; } mjx-container [width="full"] { width: 100%; } mjx-box { display: inline-block; } mjx-block { display: block; } mjx-itable { display: inline-table; } mjx-row { display: table-row; } mjx-row > * { display: table-cell; } mjx-mtext { display: inline-block; text-align: left; } mjx-mstyle { display: inline-block; } mjx-merror { display: inline-block; color: red; background-color: yellow; } mjx-mphantom { visibility: hidden; } _::-webkit-full-page-media, _:future, :root mjx-container { will-change: opacity; } mjx-math { display: inline-block; text-alig