https://www.theregister.com/2024/01/10/top_large_language_
models_struggle/
Very little here that wasn't discussed in ulm passim but may be of
interest. Some of the BTL comments are worth a read (but some of them
are not!)
Nick
On 10/01/2024 16:32, Pamela wrote:
On 16:20 10 Jan 2024, Nick Odell said:
https://www.theregister.com/2024/01/10/top_large_language_
models_struggle/
Very little here that wasn't discussed in ulm passim but may be of
interest. Some of the BTL comments are worth a read (but some of them
are not!)
"Last year, when OpenAI showed GPT-4 was capable of passing the Bar
Exam, it was heralded as a breakthrough in AI and led some people to
question whether the technology could soon replace lawyers."
What score is required to pass the Bar Exam?
It varies. The Uniform Bar Exam (UBE) has a maximum score of 400. In
some states a UBE score of 266 would be considered a pass whilst in
other states it needs to be 280 or above.
In some professions, such as accountancy, the pass mark is 50% to 55%
depending on which paper you are. That leaves a lot of room for
ignorance.
GPT-4 didn't just "pass" the UBE, it was in the 90th percentile.
However, it must be borne in mind that ChatGPT uses the GPT-3.5 series
model which failed the UBE.
On 16/01/2024 00:52, Jon Ribbens wrote:
On 2024-01-16, Simon Parker <simonparkerulm@gmail.com> wrote:
On 10/01/2024 16:32, Pamela wrote:
What score is required to pass the Bar Exam?
It varies. The Uniform Bar Exam (UBE) has a maximum score of 400. In
some states a UBE score of 266 would be considered a pass whilst in
other states it needs to be 280 or above.
In some professions, such as accountancy, the pass mark is 50% to 55%
depending on which paper you are. That leaves a lot of room for
ignorance.
GPT-4 didn't just "pass" the UBE, it was in the 90th percentile.
However, it must be borne in mind that ChatGPT uses the GPT-3.5 series
model which failed the UBE.
Of course, answering a series of questions which are highly likely to
have been asked and answered many times in the past, i.e. an exam, is
exactly the sort of thing that can be answered by anyone or anything
with excellent recall and little to no intelligence.
Well, GPT3.5 (currently in use in ChatGPT) didn't just fail the UBE but was in the
bottom 10th percentile.
I can't help but wonder if specific very targetted work was done between the series 3.5
and 4 models to assist GPT-4 in passing the UBE thus enabling this to be announced to
much fanfare.
By chance I happen to have earlier today asked Google's and Microsoft's
AI implementations some questions regarding a trivia question which
I suspect has been seldom asked before: "what English football team has
the name with the largest number of words?".
They both said that the answer was "AFC Bournemouth" - Google because
that's 4 words, and Microsoft because that's 3 words. Other teams such
as "Arsenal Football Club" and "Blackburn Rovers Football Club" are out
because clearly they have only 2 words each in their respective names.
Pedantically, both AIs are correct but they don't know why and so cannot provide
context or explain their answer which leaves you none the wiser.
In 1923 Boscombe Football Club was renamed to "Bournemouth & Boscombe Athletic Football
Club" which is 7 words and therefore the English Football team with the largest number
of words in its name.
However, in 1971 it adopted the name "AFC Bournemouth" so the answer was correct only
between 1923 and 1971, but as you didn't ask "which *current* English football team..."
I will propose that as a good candidate for the correct answer.
If considering only current club names, thereby excluding historic ones, I would
suggest that the answer is "Brighton & Hove Albion Football Club" with six words.
Tangentially, Arsenal is an interesting club name as it is only one of two professional
clubs in England not named after a specific area. A bonus point if you know the other.
:-)
After I pointed out to Microsoft's AI that its answer was nonsense,
it apologised and said the correct answer was "AFC Wimbledon", because
that is obviously longer than "AFC Bournemouth".
Oh dear!
Regards
S.P.
Tangentially, Arsenal is an interesting club name as it is only one of
two professional clubs in England not named after a specific area. A
bonus point if you know the other. :-)
On 17/01/2024 09:19, Simon Parker wrote:
Tangentially, Arsenal is an interesting club name as it is only one of two >> professional clubs in England not named after a specific area. A bonus point if you
know the other. :-)
I always assumed Arsenal was named after Woolwich Arsenal? Have I been wrong all these
years?
If the question asked was really "what English football team has
the name with the largest number of words?", I don't understand why some local team
wasn't mentioned, for example:
Wingate & Finchley (Youth) Under 16 Eastern Junior Alliance Team
On 17/01/2024 09:19, Simon Parker wrote:
Tangentially, Arsenal is an interesting club name as it is only one
of two professional clubs in England not named after a specific
area. A bonus point if you know the other. :-)
I always assumed Arsenal was named after Woolwich Arsenal? Have I been
wrong all these years?
If the question asked was really "what English football team has
the name with the largest number of words?", I don't understand why
some local team wasn't mentioned, for example:
Wingate & Finchley (Youth) Under 16 Eastern Junior Alliance Team
On 17/01/2024 09:19, Simon Parker wrote:
Tangentially, Arsenal is an interesting club name as it is only one of
two professional clubs in England not named after a specific area. A
bonus point if you know the other. :-)
I always assumed Arsenal was named after Woolwich Arsenal? Have I been
wrong all these years?
If the question asked was really "what English football team has
the name with the largest number of words?", I don't understand why some local team wasn't mentioned, for example:
Wingate & Finchley (Youth) Under 16 Eastern Junior Alliance Team
On 17/01/2024 09:19, Simon Parker wrote:
Tangentially, Arsenal is an interesting club name as it is only one of
two professional clubs in England not named after a specific area. A
bonus point if you know the other. :-)
I always assumed Arsenal was named after Woolwich Arsenal? Have I been
wrong all these years?
If the question asked was really "what English football team has
the name with the largest number of words?", I don't understand why some
local team wasn't mentioned, for example:
Wingate & Finchley (Youth) Under 16 Eastern Junior Alliance Team
Most people asking trivia questions about football teams will be talking about some specific subset of all conceivable teams that I don't know
enough about football to describe, e.g. Premier League plus Football
League, or something. So "AI", which is merely mixing and regurgitating things that people have already said, will be working on the same basis.
On 17/01/2024 10:07, billy bookcase wrote:
Brighton & Hove Albion Football Club ?
https://en.wikipedia.org/wiki/Brighton_%26_Hove_Albion_F.C.
If only I hadn't mentioned that in the post, in the very next
paragraph even...
If considering only current club names, thereby excluding historic ones, >>> I would suggest that the answer is "Brighton & Hove Albion Football Club" >>> with six words.
Tangentially, Arsenal is an interesting club name as it is only one of two >>> professional
clubs in England not named after a specific area. A bonus point if you know >>> the
other.
:-)
Port Vale is usually given as the answer although according to Wiki they're >> named after the valley of ports on the Trent and Mersey Canal.
The club's official web-site [1] disagrees and I expect the club to
know the origin of the club better than Wikipedia, therefore I'll
stick with what the club say, (which for anyone interested is):
"Port Vale Football Club was formed in 1876 and took its name
from the venue of the inaugural meeting at 'Port Vale House' situated
in a suburb of Stoke-on-Trent. Upon moving to Burslem in 1884 the Club changed its name to 'Burslem Port Vale' and after several seasons in
the Midland League became founder members of the Football League Division
Two in 1892."
[1] https://www.port-vale.co.uk/history/
quote:
The accepted theory comes from respected Potteries historian Jeff
Kent whose research found that the club was probably formed in 1879
as an offshoot of Porthill Victoria FC and took its name from the
valley of canal ports where the team played.
:unquote
| Sysop: | Keyop |
|---|---|
| Location: | Huddersfield, West Yorkshire, UK |
| Users: | 300 |
| Nodes: | 16 (2 / 14) |
| Uptime: | 34:41:44 |
| Calls: | 6,707 |
| Files: | 12,239 |
| Messages: | 5,353,334 |