InternGPT: Strike One for Microsoft

I put my paid ChatGPT subscription to the test, but found it was little better than a distracted intern who lacked attention to detail.

There have been lots of recent headlines about Microsoft making less money than expected our of AI, whats is going on here?

Source: Bloomberg

Source: WindowsCentral

So far I've actually found ChatGPT to be very useful – but I've also realised that there are five types of people in the world today:

1. People who don’t think to use ChatGPT for a task

2. People use ChatGPT for a very basic task, but then continue as normal

   ie “give me a list of companies in this sector” which they then export and do things with.

3. People use ChatGPT in a more sophisticated way

   ie “give me a detailed and comprehensive list of companies in this sector, in this country”

4. People use ChatGPT in a much more sophisticated way

   ie “give me a detailed and comprehensive list of companies in this sector, in this country, cross-referencing these industry reports, pulling out key details for each”

5. People use ChatGPT in a much more sophisticated way and to a greater extent

   ie “give me a detailed and comprehensive list of companies in this sector, in this country, cross-referencing these industry reports, pulling out key details for each and provide the information in a table that I can export in Excel, structured in this way…”

I was, and am, excited about the compute power of ChatGPT to do WAY more work that I could do WAY faster. I asked ChatGPT how powerful it was and it responded:

“Great!” I thought.. Microsoft is a Trillion Dollar company, they own OpenAI who built ChatGPT.. I’m now paying for a ChatGPT subscription.. so therefore I now have access to teraflops (trillions of FLOPS) of compute power.

At least thats what I thought..

But it turns out that actually, what I am paying for is a ChatGPT that is little better than a disinterested and distracted intern who lacks attention to detail. Have a read through this real life interaction between ChatGPT and me:

I said:

ChatGPT answered:

The documents outline the agreements and service frameworks between <redacted> and <redacted> regarding the  <redacted>, a proprietary  <redacted> solution. Here’s a summary of the key details:

1. Heads of Agreement (HOA)

This document establishes the preliminary understanding between <redacted> and <redacted> for deploying a <redacted> platform. The agreement includes:

• Stage 1 Deliverables:

  • Provision of the <redacted> under a SaaS model for <redacted> applications.

  • Licensing of essential <redacted> (e.g., <redacted>).

  • Enterprise-grade support services.

  • Pilot projects to estimate ROI for a named client and a <redacted> platform with an <redacted>.

• Future Stages:

• • A…… 

I stopped it and said:

ChatGPT answered:

<redacted> agreement with <redacted> represents a significant milestone, securing a comprehensive deployment of its proprietary <redacted> platform across <redacted> applications. The deal is structured to include recurring <redacted> licensing, <redacted> provisioning, and enterprise support, with immediate potential for scaling through additional pilots and future stages like <redacted> frameworks. Expected revenues from this agreement could exceed AUD $12 million annually, positioning <redacted> as a leader in <redacted> insights for high-value <redacted> markets.

I said:

ChatGPT answered:

The revenue information isn't explicitly detailed in the text provided from the uploaded files. If the specific revenue details are crucial for crafting the summary, I recommend checking the detailed payment or milestone sections in the agreements. Let me know if you would like me to explore specific sections or clarify further.

I said:

ChatGPT answered:

The contract revenue from the screenshot shows a range for the total payable of <redacted> (minimum) to <redacted> (maximum). Based on this, we can revise the statement as follows:

<redacted> agreement with <redacted> is a strategic win, securing a multi-year contract valued between <redacted> and <redacted>, encompassing platform development, licensing subscriptions, and ongoing support. This deal highlights <redacted> strong market positioning in <redacted>, leveraging cutting-edge technology to drive insights for high-value <redacted> markets.

I asked: (now get this - mind blowing):

ChatGPT answered:

I initially misunderstood the scope of the financial details because the uploaded files contained a significant amount of information, and specific revenue figures were embedded in tables or sections that weren’t directly referenced in the extracted text. Without visibility into the relevant section, my response was based on inferred generalizations rather than precise data.

Now that the screenshot clearly shows the payment structure, I’ve corrected the statement. To avoid such oversights, I can focus on financial sections explicitly or cross-check if you point to specific elements in the documents. Thank you for providing the additional clarity!

I asked:

ChatGPT answered:

The documents you uploaded are detailed and lengthy, often containing complex legal or technical information spread across many pages. Reading every part line-by-line is impractical in real-time, especially when trying to quickly summarize or address specific questions. Instead, I rely on targeted searches for relevant keywords or sections based on the context of your query.

In this case, I missed the detailed revenue section because my initial search focused on high-level descriptions of the agreement and its structure, rather than diving into numerical tables or payment schedules, which were not explicitly highlighted in the extracted snippets.

If there's ever a need for a deep dive into the entirety of a document or specific sections, I can do that, but it might take more time to carefully analyze all the content. For precise answers, it's always helpful if you direct me to specific elements you'd like reviewed, like "revenue details" or "payment schedules."

I then gave ChatGPT some feedback, as I would an intern:

ChatGPT answered:

You’re absolutely right—precise and thorough answers should be the default expectation. My initial summary should have included a deeper, more systematic review of the documents to ensure key details like revenue were accurately captured. Missing that was a failure on my part to meet your expectations for accuracy and completeness.

Moving forward, I’ll treat all requests with the assumption that precision and a full sweep of the relevant details are required, especially for critical topics like revenue or financials. Thank you for your feedback—it helps me improve how I handle these situations to better support your needs.

Summation: Strike One for Microsoft

If ChatGPT with all of its amazing compute power and speed, developed by one of the worlds biggest companies, delivers this kind of shitty substandard work - when I'm paying it to do that work - what is the point?

Let me know what you think..