Few things I learned by generating a 300+ page site using GPT-4

Published on: May 2, 2023 by Joona Tuunanen

Since GPT-3 was released and I had a chance to play around with it, I’ve been really interested in using these large language models and experimenting what you can do with them.

I’ve used GPT-3 to build some relatively simple tools, like writing a custom Appscript function that categorises keywords into given categories.

But I found that GPT-3 wasn’t really upto that task, at least not when using zero shot classification or in other words, it didn’t work very well unless you trained the model yourself.

After ChatGPT or GPT3.5 turbo was released, I saw a huge jump in quality in random tests. I was especially impressed by the ChatGPT’s ability to actually write coherent sentences and structure those answers properly.

That gave me an idea: could you generate all the content for a site using only ChatGPT API and if yes, how’s the quality?

Ok, of course you can generate that kind of site, but does it make sense? That’s what I wanted to find out.

I wanted to pick a topic that was not harmful as I didn’t want to push potentially problematic content to the internet. After thinking about the topic for a while, I decided to go with Excel and Google Sheets content as the worst thing that can happen is that I waste someones time by giving a bad function.

And so sheetscheat.com was born.

There’s a huge difference between GPT3.5 and GPT4

I first built this site after ChatGPT was released. The topics or titles were, krhmh sourced krhmh from the internet and I wrote a very basic prompt to generate answers to those questions.

ChatGPT could create all the pages with no problems, but the quality of content was very hit and miss. Quite often you could get only short answers and it wasn’t uncommon for the answers to miss the mark completely.

I was quite disheartened by the results actually, especially considering how much hype ChatGPT has gotten. I know it’s better suited for chat purposes than generating blog posts, but still those initial chat responses were not that great, which I find kinda weird as when using their web interface, I’ve been generally happy with the quality and accuracy of the content for spreadsheets and programming related issues.

So when GPT-4 API was released and I got access to it, I rewrote all the content using it.

To give an example how different the answers are check out these below explaining how to count rows with a specific value in Excel:

ChatGPT API

=COUNTIF(range,"value")
The COUNTIF function in a spreadsheet counts the number of cells in a given range that contain a certain value.

Not bad, but not great either. Here’s the same answer from GPT-4 API:

GPT-4 API

To count rows with a specific value in Excel, you can use the COUNTIF function. Here's how you can do that:

Open Excel and enter your data in the worksheet.
Identify the range of cells you want to count the rows with a specific value. For example, if your data is in cells A1:A10, your range will be A1:A10.
Click on an empty cell where you want the result to be displayed.
Enter the formula using the COUNTIF function. For example, if you want to count the rows with a value of “X” in the range A1:A10, you would enter the following formula:

=COUNTIF(A1:A10, "X")

Press Enter to get the result.

The COUNTIF function will now calculate the number of rows with the specified value in the defined range, and display the result in the selected cell.
Example
Let’s say you have a list of fruits in column A, and you want to count how many times the word “Apple” appears. Your data looks like this:



A




Apple


Banana


Apple


Orange


Apple



To count the number of times “Apple” appears, follow these steps:

Click on an empty cell, such as B1.
Enter the formula =COUNTIF(A1:A5, "Apple") in cell B1.
Press Enter.

The result (3) will be displayed in cell B1, indicating that “Apple” appears three times in the list.

Huge difference in helpfulness and quality I think! Please keep in mind that the prompts where exactly the same to make it easier to see what is the difference between the models and not how I managed to prompt the API better.

GPT-4 API is not that reliable

By far the biggest and most annoying part of the whole process was the constant issues with GPT-4 API and timeouts. It crashed on me so many times even though I pinged it wayyyyy less than what’s allowed.

I sent only 4-5 reqs/minute when their rate limits allow for 200 reqs/minute.

I get that the biggest reason for this is that OpenAI is under a huge load considering the popularity of ChatGPT and their other models and it must be very difficult to cope with that demand.

So I’m not really complaining that much, just saying that based on my experience it’s good to expect some issues.

Overall it took a bit over 4h to generate those 300+ pages of content and save them to a database due to the API timing me out leading me to restart the script every now and then.

Even though GPT-4 is not cheap, the price is unbeatable compared to humans

It’s actually ridiculous to think about how far your money can go with these models.

The total cost for generating 327 posts was $10.81.

If you’d ask a human to write that amount of posts it would be expontentially more expensive. Of course the quality might also be better, but for this type of “explain how to do X”-content which is already well documented on the internet GPT-4 works perfectly well.

If this would be a serious project and I’d want to for example try to rank with these articles, I’d definitely have a human editor check all the content and edit it for clarity and probably to add videos or screenshots there where necessary. That would of course increase the price of a piece of content, but it would still be a lot cheaper than having a human write the content and to add screenshots etc.

Conclusion

This was overall a very interesting experiment and I learned a good amount in the process.

I have some ideas how to take this even further and experiment with other things with GPT-4:

1. Ask GPT-4 to extract all the used functions from the content and to explain them on separate pages, then add links to those pages.
2. Add other related functions so the visitor can learn more.
3. Ask alternative ways to solve the same thing.

If you haven’t been playing around with ChatGPT or its API, I highly encourage you to do so. You can learn quite quickly what is possible with these large language models (LLMs) and where they’ll start to hallucinate or just provide junk answers.