AI tip of the week
Here’s an AI tip we use nearly every day – you can use GenAI to navigate complex websites and figure out how to do things. For example, to run 3rd Rodeo AI, we use a bunch of services and products on the backend – things like SamCart, Kit, Stripe, probably 20 others. Often times, the FAQs and “regular” search don’t work – so we’ll turn to one of our AI tools to help us figure something out.
In a recent example, Jeremiah accidentally turned on a random white border in one of our blogposts, and it really bothered him. He wanted to make it go away – so he used this prompt (with an image of the white border in question): “how do i get rid of the white border in this WordPress blogpost?”
He’d previously spent more time than he’d like to admit, playing with WordPress trying to figure it out. 3 minutes after typing this prompt into ChatGPT, he was on to his next task.
GenAI as “how do I do something?” assistant – Give it a try!
Our theory is that the more you use GenAI, the more you’ll supercharge your career.
SOME OF THE TIME!
We’ve found so many scenarios where you can use GenAI. You’ll save hours – or even days of work. With GenAI you can knock out high-quality finished images, documents, slides, presentations, emails, even janky t-shirts!
But you have to be careful because power = responsibility. Spider-Man, etc.
As with any other powerful tool, there are things GenAI is good at, things it’s bad at, and things it’s, err, weird at. Like, a hammer is really bad at cutting things.
Anyway, we’ve found in our own work that GenAI is awesome at SOME things. Try handing a frontier model a “good enough” prompt, and you can unlock helpers like these:
- a coach, advisor, or collaborator to help you create, offering new ideas or improving existing ones
- a “research assistant,” helping you get up-to-speed quickly on new and unfamiliar concepts – and also help you understand them by explaining them in helpful ways
- a “ghostwriter,” helping you draft emails, landing pages, white papers, strategy docs, poems, sonnets, music, marketing copy, and many other types of written materials
- an “analyst”, to summarize, remix,, and tease insights out of longer texts
- a “co-creator,” to come up with new talks, presentations, code snippets, scripts, prototypes, working games, and even t-shirt designs
There are many more – we find new ones every day. (Let us know if we’ve missed your favorites!)
Today, though, it might be fun to look things that GenAI is NOT good at.
We’ll try to explore WHY it can fall down within these scenarios.
Rule #1: Generative AI GENERATES so…here are some of the domains where GenAI faces systematic limitations:
(1) Be very careful using GenAI if you absolutely need the response to be 100% factual, safe, and CORRECT.
We’ve said it a million times. Always check your work. Verify the sources, the quotes. Read the primary materials. Remember – GenAI is designed to produce responses that “seem” right. Not to produce responses that ARE right.
In high-stakes situations, (like legal cases and medical diagnoses), responses from GenAI are often unreliable without expert human oversight.
GenAI has no problem with making up authoritative-sounding sources. We’ve all seen high-profile court cases where a lawyer used GenAI to prepare legal filings – that came up with fake citations or bogus statutes.
Medicine is another area where we’d best be super careful with taking GenAI responses as fact. Studies have shown that GenAI can often get diagnoses correct that expert doctors might get wrong. But are you ready yet to put your own life into the hands of a model whose main goal is to persuade you with convincing-sounding information? NOT to give you the correct diagnosis?
(2) “No-repro” is a thing. Since GenAI’s responses are “Non-Deterministic,” you’re going to get different results every time
“Non-deterministic” is just a fancy word that means: we never know what response we’re going to get. The response is a mystery every time, even if we provide EXACTLY the same input over and over. This is a feature, not a bug.
You’ll get different answers to even similar-looking input prompts – and, especially frustrating, even THE EXACT SAME INPUT PROMPT. Even our spaces, punctuation, and politeness can matter to the response we get.
This can be a blessing (say, you’re trying to spark your own creativity by trying out wholly new sentence & para formulations). But it can ALSO be a curse, especially for testing that the system is working as you intended it to work.
It’s not fun to try to automate testing your AI system if you can’t even define what you’re testing for. (The most common way to do this is to have lots and lots of humans run models through their paces with lots and lots of tests, even simulating attacks on the system, aka “red teaming.”)
And your definition of “soup,” say, might be different from someone else’s. True story.
(3) GenAI isn’t really good for up-to-date news. Your data is going to be stale.
GenAI is trained on enormous, static datasets. It has a hard time incorporating real time data. Out of the box, most AI models can’t adapt to massive streams of incoming new information. To make these updates, models would need re-training or fine-tuning. Both of these are expensive and slow.
For example, good luck trying to get an AI model to read second-by-second stock market data! The use case to do this would be stunning – and many have tried – but the engineering work to get this running is just staggering.
You’d be much better off using a “traditional” stream-processing platform like Kafka, something custom-designed for this sort of real time data.
(4) Please, please check your math! GenAI has a hard time with doing precise computations
Even the newest, fanciest Gen AI models fall down on precise numerical computations. Things like counting the number of “r”s in the word “strawberry” or doing basic math.
This happens “by design” – GenAI prioritizes generating plausible-sounding responses over computational accuracy.
Want to do real math or logic? There are already apps for that.
Don’t use GenAI – use one of those!
(6) GenAI can produce results that look so good – you might need to be an actual expert to catch a mistake in some domains.
There are some topics that are super nuanced. This is a slightly different point than “check your sources.” This one is more – potentially the SOURCES are correct and the right ones – but the CONCLUSIONS you might draw from them might be slightly off.
General GenAI models lack the specialized knowledge to succeed in ALL domains. Again, we’ll pick on taxation, legal document drafting, medical diagnosis. GenAI can be wildly wrong, in hard-to-easily-catch ways.
In such domains, often time you will need precise and context-specific knowledge that a general-purpose AI just can’t deliver. You might need to be an actual expert to catch a flaw in nuances, terminology, or implication. And, let’s face it, even experts can mess up if they’re pressed for time or reviewing a very long document – which is exactly when we’d be most likely to turn to an AI assistant.
Gaps are narrowing in many fields with nearly every release – but PLEASE be aware that they still exist!!
(7) GenAI can raise privacy & security concerns you might not have seen coming
It’s an old InfoSec saying that “security through obscurity is not really security at all.” That is especially true in this AI era, where you have to be careful exactly what data is making it into the training datasets!
Since much of the GenAI goodness comes from the datasets fed to it in training, fine-tuning, and even enhancement/augmentation (like RAG), its models can expose sensitive or confidential information.
For example, you might want to train your on internal company emails (seems like a good idea, right?) However, if you’re not careful – you could share details of that top-secret tented CEO initiative. Oopsies.
There are a few other things you’re going to want to watch out for, when using GenAI, but 7 is a lucky number and we’ll leave it here for now.
Hit us back if you think we missed any!
Enjoy the rodeo, cowpokes!! See you out there.
Dona & Jeremiah
If you need help with any of these concepts OR want to meet others who are also doing the thing, our “Become an AI Power User workshop” on December 13th might be useful to you. In any case, it should be fun!
You can find out more here: https://shop.3rdrodeoai.com/products/find-your-career-niche-in-the-ai-verse-dec-13-2024
