• 1 Post
  • 32 Comments
Joined 3 years ago
cake
Cake day: June 12th, 2023

help-circle

  • This “benchmark”, gdpval, does not at all evaluate the effective real-world use of ai across 44 occupations.

    Lawyers, police/detective supervisors, social workers are some of those jobs “evaluated”, jobs which we know ai currently hallucinate datasets and context for

    "GDPval is an early step that doesn’t reflect the full nuance of many economic tasks. While it spans 44 occupations and hundreds of knowledge work tasks, it is limited to one-shot evaluations, so it doesn’t capture cases where a model would need to build context or improve through multiple drafts. "

    Openai uses expertly constructed and meticulously crafted questions suitable to ai and tests a single job task a single time specifically to avoid the problem of low ai accuracy and consistency.

    Gdpval doesn’t even grade those 44 jobs, it grades a single task within one of those jobs and then extrapolates that single task “success” as proficiency across an entire occupation.

    This is more “despite the contrary evidence of real-world data, if you let openai cherrypick data within extremely constrained question-response conditions, it is possible to view single, non-holistic, hypothetical answers as a success of ai”.

    or

    “If you pretend our limited, invalid data is comprehensive and valid, then you can pretend our ai is successful.”


  • Varyk@sh.itjust.workstoComic Strips@lemmy.world•Aging 🫠
    link
    fedilink
    arrow-up
    0
    arrow-down
    1
    ·
    1 year ago

    that’s really interesting. I listened to a podcast with one of the writers, or maybe the creator, I can’t remember right now, and he talked about how sincere everybody was about creating the superhero who could actually change the cultural lack of concern about climate change.

    and what’s super cool is like statistically and culturally, it looks like the show worked.

    like yeah everything is still fucked, but it would have been way worse a generation earlier if a bunch of awesome nerds didn’t make Captain planet for generation x so they would become horrified over climate change.

    [here it is] (https://share.google/H4LaAuoFDf6AS01Al)