<p>I’m a big believer in “Local AI”, i.e. running an AI model embedded on your phone or other device, instead of accessing it through the cloud.</p>
<p>Why?</p>
<ul>
<li>🔌 Availability: Works offline, not dependent on internet connectivity.</li>
<li>🔒 Privacy: sensitive data never leaves your device.</li>
<li>⚡️ “Speed”: less latency by processing data directly on the device, BUT a smaller device is also slower at loading and running a large language model.</li>
<li>☘️ Environment: less power usage, less CO₂.</li>
</ul>
<p>As an example, compare:</p>
<ul>
<li>Humane AI Pin (cloud): <a href="https://www.youtube.com/watch?v=_w1vv7_dU2Y">https://www.youtube.com/watch?v=_w1vv7_dU2Y</a></li>
<li>Rabbit R1 (local): <a href="https://www.youtube.com/watch?v=aqb3u5mXcl4">https://www.youtube.com/watch?v=aqb3u5mXcl4</a></li>
</ul>
<p><a href="https://www.theverge.com/2023/9/6/23861763/apple-ai-language-models-ajax-gpt-training-spending">Apple is working on their “Ajax” model</a> and <a href="https://www.xda-developers.com/openai-make-own-ai-hardware">OpenAI are working on their own hardware</a>. Exciting times ahead!</p>
<p>If you want to try a Local AI on iPhone, try <a href="https://apps.apple.com/app/id6448106860">Private LLM</a>.</p>


I’m a big believer in “Local AI”, i.e. running an AI model embedded on your phone or other device, instead of accessing it through the cloud.

Local AI

<p>In the vast expanse of digital content, the role of AI is becoming increasingly central, enabling what can be called not just the <a href="https://en.m.wikipedia.org/wiki/Long_tail">long tail</a>, but the “infinite tail” of content. This concept uses AI to extend the reach of content and also redefine how content aligns with user interests and trends.</p>
<h2 id="personalization-at-scale">Personalization at scale</h2>
<p>The core of this approach lies in personalization at scale. AI algorithms are adept at mining through extensive datasets to pinpoint niches and micro-trends. This capability enables the creation of content that’s not just broadly appealing but precisely tailored to meet the varied interests of individual users.</p>
<p>Furthermore, AI’s knack for analyzing user behavior brings to light the demand for content in areas previously overlooked or deemed unprofitable, thereby uncovering new content niches ripe for exploration.</p>
<h2 id="letting-clicks-and-views-decide">Letting clicks and views decide</h2>
<p>The decision-making process about what content to produce next is increasingly data-driven, relying on real-time metrics such as clicks and views. This marks a shift from guesswork to a more strategic approach where content strategies dynamically evolve based on direct feedback from the audience. The result is content that’s continually adapting and expanding in response to user engagement.</p>
<h2 id="ai-drafts-humans-refine">AI drafts, humans refine</h2>
<p>This method harnesses the power of artificial intelligence to produce initial versions of articles, blog posts, or any other form of content quickly and efficiently, addressing the demand for timely and relevant material. Such AI-generated drafts serve as a foundation upon which human creativity and insight can be applied, allowing for refinement and enhancement.</p>
<p>This collaborative process between human and machine not only streamlines content production but also ensures that the final output retains the necessary depth, nuance, and personal touch that resonates with audiences.</p>
<h2 id="a-demo-stylopedia">A demo: Stylopedia</h2>
<p>An example of these ideas can be seen through the demo <strong>Stylopedia</strong> (<a href="https://stylopedia.tomorroworld.com/">https://stylopedia.tomorroworld.com/</a>). This project demonstrates AI’s potential in generating on-demand content. By analyzing what is clicked on, Stylopedia dynamically produces new, relevant content pages on the fly. The site started with just a few content pages, and then quickly grew as users started clicking on related words.</p>
<p><img src="/blog/images/stylopedia.png" alt="Stylopedia"></p>
<p>The journey towards an “Infinite Tail” of content facilitated by AI is transforming the internet into a more diverse, personalized, and creative space. This evolution promises a future where digital content is not only more accessible and engaging but also where the lines between human and machine creativity are increasingly converging.</p>


Using AI to extend the long tail of content by creating new content based on user interactions.

AI and the Infinite Tail of Content

<p>Want to build AI-powered voice applications?</p>
<p>The <a href="https://platform.openai.com/docs/api-reference/audio/createTranscription">Whisper model by OpenAI</a> is great for transcribing voice into text – even supporting multiple languages. It supports a variety of audio formats like FLAC, MP3, MP4, MPEG, MPGA, M4A, OGG, WAV, WEBM.</p>
<p>Unfortunately, typical audio uploads from smartphones are often in CAF format for iOS, or 3GP for Android. Whisper will respond with “Invalid file format. Supported formats: [&#39;flac&#39;, &#39;m4a&#39;, &#39;mp3&#39;, &#39;mp4&#39;, &#39;mpeg&#39;, &#39;mpga&#39;, &#39;oga&#39;, &#39;ogg&#39;, &#39;wav&#39;, &#39;webm’]”.</p>
<p>To overcome this, we need to convert the audio first. This means we have to package <a href="https://ffmpeg.org/">FFmpeg</a>, a full Linux binary, with our backend code.</p>
<p>Luckily, <a href="https://aws.amazon.com/lambda/">AWS Lambda</a> serverless functions supports this.</p>
<h2 id="creating-the-lambda-function">Creating the Lambda function</h2>
<p>I like to have a <code>[MY_PROJECT]/lambda</code> folder, where each Lambda function gets its own subfolder. This is how this <code>transcribeAudio</code> function is organized:</p>
<pre><code>📁 [MY_PROJECT]/lambda/transcribeAudio
 ∟ 📁 bin
     ∟ 📄 ffmpeg // this is the FFmpeg binary
  ∟ 📄 index.js // main function handler
 ∟ 📄 package.json
 ∟ 📄 yarn.lock
 ∟ 📁 node_modules
</code></pre>
<h2 id="writing-the-lambda-function-handler">Writing the Lambda function handler</h2>
<p>The code inside <code>index.js</code>:</p>
<pre><code>const ffmpeg = require(&#39;fluent-ffmpeg&#39;)
const OpenAI = require(&#39;openai&#39;)
const fs = require(&#39;fs&#39;)
const path = require(&#39;path&#39;)

module.exports.handler = async (event) =&gt; {
  try {
    // Initialize FFmpeg
    const ffmpegPath = path.join(__dirname, &#39;./bin/ffmpeg&#39;)
    ffmpeg.setFfmpegPath(ffmpegPath)
    // Assume audio data POSTed as base64
    const { audioFormat = &#39;mp4&#39;, audioData } = JSON.parse(event.body)
    if (!audioData) throw new Error(&#39;No audio data provided&#39;)
    // Save audio to temporary storage in Lambda
    const inputPath = `/tmp/input.${audioFormat}`
    const outputPath = &#39;/tmp/output.mp3&#39;
    const audioBuffer = Buffer.from(audioData, &#39;base64&#39;)
    fs.writeFileSync(inputPath, audioBuffer)
    // Convert audio to MP3
    await convertToMp3(inputPath, outputPath, audioFormat)
    // Use OpenAI Whisper to transcribe audio
    const readStream = fs.createReadStream(outputPath)
    const text = await getAudioTranscription(readStream) ?? &#39;&#39;
    // Return results
    return {
      statusCode: 200,
      body: JSON.stringify({ text })
    }
  } catch (err) {
    // Handle error
    return {
      statusCode: 500,
      body: JSON.stringify({ message: `Server error: ${err.message}` })
    }
  }
}

const convertToMp3 = async (inputPath, outputPath, inputFormat = &#39;mp4&#39;, outputFormat = &#39;mp3&#39;) =&gt; {
  return await new Promise((resolve, reject) =&gt; {
    ffmpeg(inputPath)
      .inputFormat(inputFormat)
      .output(outputPath)
      .outputFormat(outputFormat)
      .on(&#39;end&#39;, () =&gt; {
        resolve()
      })
      .on(&#39;error&#39;, (err, stdout, stderr) =&gt; {
        console.error(&#39;* FFmpeg error:&#39;, err.message)
        console.error(&#39;stdout:\n&#39; + stdout)
        console.error(&#39;stderr:\n&#39; + stderr)
        reject(err)
      })
      .run()
  })
}

// FLAC, MP3, MP4, MPEG, MPGA, M4A, OGG, WAV, WEBM
// Expo: CAF for iOS, 3GP for Android
const getAudioTranscription = async (readStream) =&gt; {
  const openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY })
  const transcription = await openai.audio.transcriptions.create({
    file: readStream,
    model: &#39;whisper-1&#39;
  })
  return transcription.text
}
</code></pre>
<h2 id="getting-the-ffmpeg-binary">Getting the FFmpeg binary</h2>
<p>In <code>[MY_PROJECT]</code> root folder, I did this:</p>
<pre><code>mkdir ffmpeg-temp &amp;&amp; cd ffmpeg-temp
wget https://johnvansickle.com/ffmpeg/releases/ffmpeg-release-amd64-static.tar.xz
tar xvf ffmpeg-release-amd64-static.tar.xz
mkdir -p lambda/transcribeAudio/bin
cp ffmpeg-6.0-amd64-static/ffmpeg ../lambda/transcribeAudio/bin
cd ..
</code></pre>
<h2 id="deploying-the-lambda-function-to-aws">Deploying the Lambda function to AWS</h2>
<p>You can upload a ZIP file with your Lambda function to the <a href="https://console.aws.amazon.com/lambda">AWS web console</a>, but an easier way is to use the <a href="https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html">AWS command-line tool</a>.</p>
<p>So in <code>[MY_PROJECT]</code> folder, I did this:</p>
<pre><code>cd lambda
echo Creating ZIP archive...
rm transcribeAudio.zip
cd transcribeAudio &amp;&amp; zip -r ../transcribeAudio.zip * &amp;&amp; cd ..
echo Uploading to AWS Lambda...
aws lambda update-function-code --function-name transcribeAudio --zip-file fileb://transcribeAudio.zip
cd ..
</code></pre>
<p><strong>Tip:</strong> put these commands in the <code>scripts</code> section in <code>package.json</code>, so you can run:</p>
<pre><code>yarn lambda-deploy  # Deploy Lambda function to AWS
yarn lambda-ffmpeg  # Get latest FFmpeg
</code></pre>
<h2 id="connecting-a-rest-api-to-the-lambda-function">Connecting a REST API to the Lambda function</h2>
<p>Lambda functions can be used for cron jobs etc, but AWS also provides an easy way of wrapping it in a REST API: the <a href="https://aws.amazon.com/api-gateway/">API Gateway</a>.</p>
<p>Go to your <a href="https://console.aws.amazon.com/lambda">Lambda function on AWS Console</a>, and press the “Add trigger” button to set it up:</p>
<p><img src="/blog/images/transcribeaudio-aws.png" alt="Lambda on AWS Console"></p>
<p>You will then get a long URL that you can use to <code>POST</code> your data to.</p>
<h2 id="more">More</h2>
<h3 id="configuration">Configuration</h3>
<ul>
<li>You will most likely need to increase the <strong>timeout of your Lambda function</strong> to 1-2 minutes (default: 3 seconds), see the “Configuration” tab on your Lambda’s page on <a href="https://console.aws.amazon.com/lambda">AWS Console</a>.</li>
<li>You can enter <strong>environment variables</strong>, e.g. your secret OpenAI API key, again see the “Configuration” tab.</li>
</ul>
<h3 id="testing--debugging">Testing &amp; debugging</h3>
<ul>
<li>AWS Lambda has a built in tool for testing your function, see the “Test” tab on your Lambda’s page.</li>
<li><strong>Note:</strong> when using a real client, the <code>POST</code>:ed data will come in <code>event.body</code> as a string, not an object like the Test tab provides.</li>
<li>Detailed logging (including <code>console.log</code> statements) is captured in AWS CloudWatch. A shortcut is found if you use the Test feature mentioned above.</li>
</ul>


Want to build AI-powered voice applications? Learn how to upload audio from a smartphone, processing it with FFmpeg, and transcribing voice to text with OpenAI’s Whisper model.

Transcribing audio with OpenAI Whisper on AWS Lambda

<p>Here’s how you can use ChatGPT and Midjourney to create Hollywood style movie treatments, scripts, and storyboards.</p>
<blockquote class="tiktok-embed" cite="https://www.tiktok.com/@tomsoderlund/video/7227229641587592474" data-video-id="7227229641587592474" style="max-width: 605px;min-width: 325px;" > <section> <a target="_blank" title="@tomsoderlund" href="https://www.tiktok.com/@tomsoderlund?refer=embed">@tomsoderlund</a> Making movies using AI: tomsoderlund.com&#47;ai <a title="ai" target="_blank" href="https://www.tiktok.com/tag/ai?refer=embed">#AI</a> <a title="ml" target="_blank" href="https://www.tiktok.com/tag/ml?refer=embed">#ML</a> <a title="artificialintelligence" target="_blank" href="https://www.tiktok.com/tag/artificialintelligence?refer=embed">#artificialintelligence</a> <a title="machinelearning" target="_blank" href="https://www.tiktok.com/tag/machinelearning?refer=embed">#machinelearning</a> <a title="chatgpt" target="_blank" href="https://www.tiktok.com/tag/chatgpt?refer=embed">#chatgpt</a> <a title="midjourney" target="_blank" href="https://www.tiktok.com/tag/midjourney?refer=embed">#midjourney</a> <a title="moviemaking" target="_blank" href="https://www.tiktok.com/tag/moviemaking?refer=embed">#moviemaking</a> <a title="movies" target="_blank" href="https://www.tiktok.com/tag/movies?refer=embed">#movies</a> <a target="_blank" title="♬ original sound - Tom Söderlund" href="https://www.tiktok.com/music/original-sound-7227229635909225243?refer=embed">♬ original sound - Tom Söderlund</a> </section> </blockquote> <script async src="https://www.tiktok.com/embed.js"></script>

<h2 id="priming-chatgpt">Priming ChatGPT</h2>
<p><strong>ChatGPT prompt:</strong> “Pretend that you’re a world class movie director”</p>
<h2 id="get-ideas-for-movie-plots">Get ideas for movie plots</h2>
<p><strong>ChatGPT prompt:</strong> “List 3 ideas for [Tarantino style cowboy movie set in Stockholm in 1860]. Max 80 words per idea.”</p>
<p>Results:</p>
<ol>
<li><strong>Title: “The Stockholm Standoff”.</strong> In 1860 Stockholm, two rival gangs of outlaws find themselves in a tense standoff over a stolen shipment of gold. As the tension escalates, loyalties are tested, and betrayals are unveiled. Filled with razor-sharp dialogue, dark humor, and intense shootouts, the film explores the blurred lines between heroes and villains in a lawless city.</li>
<li><strong>Title: “Gallows of Gamla Stan”.</strong> A notorious gunslinger is captured and awaits execution in Stockholm&#39;s old town, Gamla Stan. A motley group of outlaws and misfits bands together to free him, leading to a series of daring heists, double-crosses, and violent confrontations. Tarantino&#39;s signature style shines in this gritty tale of redemption and revenge.</li>
<li><strong>Title: “The Baltic Bounty”.</strong> A hardened bounty hunter navigates the seedy underworld of 1860 Stockholm, seeking the infamous outlaw known as “The Baltic Butcher.” As the hunter delves deeper into the city&#39;s dark corners, he uncovers a complex web of corruption and intrigue. This action-packed, dialogue-driven film masterfully intertwines Stockholm&#39;s history with classic cowboy themes.</li>
</ol>
<h2 id="writing-a-film-treatment">Writing a film treatment</h2>
<p><strong>ChatGPT prompt:</strong> “Write a 100 word treatment for a [Tarantino style cowboy movie set in Stockholm in 1860]”</p>
<h2 id="creating-the-movie-poster">Creating the movie poster</h2>
<p><strong>ChatGPT prompt:</strong> Describe the movie poster for [“Gallows of Gamla Stan”]</p>
<h2 id="identifying-key-story-beats-or-scenes">Identifying key story beats or scenes</h2>
<p><strong>ChatGPT prompt:</strong> “Write the main story beats for [“Gallows of Gamla Stan”], as a bullet list. Aim for a 30 minute movie runtime.”</p>
<p>Results:</p>
<ol>
<li>Opening scene: Notorious gunslinger, Gustav &quot;The Ghost&quot; Svensson, is captured and imprisoned in Gamla Stan, Stockholm&#39;s old town, awaiting execution.</li>
<li>Introduction of a motley group of outlaws and misfits, each with their own unique skills, who decide to band together to rescue Gustav.</li>
<li>The group devises a daring plan to infiltrate the heavily guarded prison where Gustav is held.</li>
<li>Tension builds as the group carries out a series of smaller heists to gather resources and intelligence needed for the prison break.</li>
<li>Unexpected complications arise when one of the outlaws betrays the group to the authorities, forcing the remaining members to adapt their plan on the fly.</li>
<li>A thrilling action sequence ensues as the group infiltrates the prison, battling their way through guards and navigating the labyrinthine prison corridors.</li>
<li>The group successfully reaches Gustav&#39;s cell, but faces a final confrontation with the prison warden, who has a personal vendetta against Gustav.</li>
<li>With the warden defeated, the group frees Gustav, and they make a daring escape from the prison just as reinforcements arrive.</li>
<li>The group, now united by their shared experience, parts ways, having learned valuable lessons about trust, loyalty, and redemption.</li>
<li>Closing scene: Gustav gazes out over the city of Stockholm, contemplating his newfound freedom and the uncertain future that lies ahead.</li>
</ol>
<h2 id="casting-the-main-actors">Casting the main actors</h2>
<p><strong>ChatGPT prompt:</strong> “What actors should play the main roles in this movie? Present one actor per role as a bullet list.”</p>
<p>Results:</p>
<ul>
<li>Gustav &quot;The Ghost&quot; Svensson: Mads Mikkelsen</li>
<li>Stern-faced Guard: Stellan Skarsgård</li>
</ul>
<h2 id="creating-a-visual-storyboard">Creating a visual storyboard</h2>
<p><strong>ChatGPT prompt:</strong> “Write a 80 word description for a concept artist, what a key moment of Story Beat 1 above looks like. Include location￼, year, season, time of day, and shot type, lighting, color grading details. Also describe face, body, clothes details of characters in the scene.”</p>
<p>Results:</p>
<blockquote>
<p><strong>Title: “The Outlaws Assemble”</strong>. Visual Setting: A dimly lit, smoke-filled tavern in Gamla Stan, Stockholm, 1860, on a rainy autumn night. A diverse group of outlaws gathers around a wooden table, lit by flickering candlelight. The camera captures a wide shot, using a warm color grading to emphasize the camaraderie. Each character, dressed in period attire, showcases unique facial features and expressions reflecting their individual personalities. The scene conveys a sense of determination and unity, as they devise a plan to rescue Gustav.</p>
</blockquote>
<p>Then take this text, strip away any forbidden words (e.g. violence or sex related) and use it with Midjourney:</p>
<p><strong>Midjourney prompt:</strong> /imagine Title: “The Outlaws Assemble”. Visual Setting: A dimly lit, smoke-filled tavern in Gamla Stan, Stockholm, 1860, on a rainy autumn night. A diverse group of outlaws gathers around a wooden table, lit by flickering candlelight. The camera captures a wide shot, using a warm color grading to emphasize the camaraderie. Each character, dressed in period attire, showcases unique facial features and expressions reflecting their individual personalities. The scene conveys a sense of determination and unity, as they devise a plan to rescue Gustav. cinematic photorealistic Kodak Vision3 anamorphic 8k --ar 9:16</p>
<p>Midjourney results:</p>
<p><img src="/blog/images/AI_The_Outlaws_Assemble.png" alt="The Outlaws Assemble"></p>
<h3 id="alternative-with-separate-setting-and-character-descriptions">Alternative with separate setting and character descriptions</h3>
<p><strong>ChatGPT prompt:</strong> “Write two 50 word descriptions for a concept artist, what a key moment of Scene 1 looks like. The first description should focus on the setting and ￼include place, year, season, time of day, and cinematography details. The second description should be a medium shot of main character(s) in the scene, including￼ face, body, clothes, and cinematography details.”</p>
<p>Results:</p>
<blockquote>
<ol>
<li><strong>Setting Focus:</strong> Gamla Stan, Stockholm, 1860, during a chilly autumn twilight. Fading light casts long shadows on the cobblestone alley, surrounded by medieval buildings. Gas lanterns flicker, creating a moody atmosphere. The camera captures a low-angle shot, enhancing the sense of tension and danger in this key moment.</li>
</ol>
</blockquote>
<p><img src="/blog/images/AI_Gamla_Stan_Stockholm_1860.png" alt="Gamla Stan, Stockholm, 1860"></p>
<blockquote>
<ol start="2">
<li><strong>Character Close-up:</strong> Gustav “The Ghost” Svensson, a rugged gunslinger with piercing eyes and a scarred face, stands defiantly. Beads of sweat on his brow and a clenched jaw reveal his determination. The camera focuses on his twitching hands, subtly hinting at his readiness for a violent confrontation with the armed guards.</li>
</ol>
</blockquote>
<p><img src="/blog/images/AI_Gustav_The_Ghost_Svensson.png" alt="Gustav “The Ghost” Svensson"></p>


How to use ChatGPT and Midjourney to create Hollywood style movie treatments, scripts, and storyboards.

Making movies using AI

<p>⚠️ = extra important resources for beginners.</p>
<ul>
<li>My own AI/ML resources: <a href="https://www.tomsoderlund.com/ai/ai-resources">https://www.tomsoderlund.com/ai/ai-resources</a></li>
<li>Tweet: <a href="https://twitter.com/tomsoderlund/status/1583894481408905216"><em>“Ok, I feel I need to learn AI/ML now.”</em></a></li>
</ul>
<p>Thank you to <a href="https://twitter.com/spimescape">@spimescape</a> for mentoring me!</p>
<h2 id="introduction">Introduction</h2>
<p>I’m bullish about Artificial Intelligence (AI).</p>
<p>I’ve lived through three computing revolutions: the home PC, internet, and smartphones. I firmly believe that AI/ML is a 4th revolution.</p>
<blockquote>
<p>I firmly believe that AI/ML is a 4th computing revolution</p>
</blockquote>
<p>AI/ML is a new computing paradigm shift. Ignoring AI in 2022 is like ignoring the iPhone in 2010.</p>
<p>It requires a new way of thinking and approaching problems. My background as an entrepreneur and a software developer is not completely helpful; some things need to be unlearned and relearned.</p>
<h3 id="ai-is-not-magic">AI is not magic</h3>
<p>The more I learn about artificial intelligence, the less magical it seems. The Singularity is far way, artificial general intelligence (AGI) is nowhere near.</p>
<p>Instead, machine learning is all about math, identifying/generating patterns.</p>
<blockquote>
<p>The more I learn about artificial intelligence, the less magical it seems</p>
</blockquote>
<p>But at the same time <em>it is</em> magical. Example: making a blurry image sharp is <em>creating information</em> out of <em>nothing</em>. Tasks that previously required impossible number of man hours are now very feasible. <strong>AI will change how we work.</strong></p>
<h3 id="ai-or-ml">“AI” or “ML”?</h3>
<p>What’s the difference between Artificial Intelligence (AI) vs. Machine Learning (ML)? Today AI and ML refers to similar things, but AI is more used by media and outsiders, whereas ML is the more correct term used by people who research and build the technology.</p>
<p>More formally:</p>
<ul>
<li><strong>Artificial Intelligence</strong> refers to the general ability of computers to emulate human thought and perform tasks in real-world environments.</li>
<li><strong>Machine Learning</strong> refers to the technologies that enable systems to identify patterns, make decisions, and improve themselves through experience.</li>
</ul>
<h3 id="use-cases-for-aiml">Use cases for AI/ML</h3>
<p>Broadly speaking, we today use Machine Learning (ML) for:</p>
<ol>
<li>Classification &amp; Prediction (e.g. spam detection, find objects/people in images)</li>
<li>Generation &amp; Conversion (e.g. text, images, video, audio, speech)</li>
<li>Search (more advanced search vs pure text search)</li>
<li>Recommendations (e.g. music, film, products, social feeds)</li>
</ol>
<p><img src="/blog/images/ai_use_cases.jpg" alt="Use cases for AI/ML"></p>
<h2 id="learning-resources">Learning resources</h2>
<p>⚠️ Two great free courses for ML beginners: <a href="https://developers.google.com/machine-learning/crash-course/ml-intro">Google’s (theoretical)</a> and <a href="https://course.fast.ai/">FastAI’s (practical)</a>.</p>
<ul>
<li><a href="https://developers.google.com/machine-learning">Google ML Foundational Courses</a><ul>
<li>⚠️ <a href="https://developers.google.com/machine-learning/crash-course/ml-intro">ML Crash Course with TensorFlow APIs</a></li>
<li><a href="https://developers.google.com/machine-learning/problem-framing/problem">Problem Framing</a></li>
<li><a href="https://playground.tensorflow.org/">TensorFlow Playground</a></li>
</ul>
</li>
<li>⚠️ <a href="https://course.fast.ai/">FastAI: “Practical Deep Learning for Coders”</a><ul>
<li><a href="https://www.youtube.com/playlist?list=PLfYUBJiXbdtSvpQjSnJJ_PmDQB_VyT5iU">YouTube video lessons</a></li>
<li>Book: “Deep Learning for Coders with FastAI and PyTorch: AI Applications Without a PhD”: <a href="https://github.com/fastai/fastbook">free on GitHub</a> or <a href="https://www.amazon.com/Deep-Learning-Coders-fastai-PyTorch/dp/1492045527">on Amazon</a></li>
</ul>
</li>
<li><a href="https://www.kaggle.com/learn/intro-to-machine-learning">Kaggle course “Intro to Machine Learning”</a></li>
<li>O’Reilly:<ul>
<li>Courses: <a href="https://learning.oreilly.com/courses/dataml-engineer/">Data/ML engineer</a></li>
<li>Book: <a href="https://learning.oreilly.com/library/view/python-machine-learning/9781789955750/Text/Chapter_1.xhtml#_idParaDest-23">“Python Machine Learning”</a></li>
<li><a href="https://learning.oreilly.com/playlists/907fa947-4eba-4c1c-bcbd-9cc5167568d7/">Playlist</a></li>
</ul>
</li>
<li><a href="https://code.visualstudio.com/docs/datascience/data-science-tutorial">VS Code: “Data Science in VS Code tutorial”</a></li>
<li><a href="https://www.w3schools.com/python/python_ml_getting_started.asp">W3schools Python course</a></li>
<li><a href="https://twitter.com/1littlecoder/status/1580555066791972866">“Build your AI Product MVP in less than 24 hours”</a></li>
</ul>
<h2 id="people">People</h2>
<ul>
<li><a href="https://twitter.com/i/lists/1577855041150541826">Tom’s Twitter list “AI &amp; ML”</a></li>
</ul>
<h2 id="tools">Tools</h2>
<ul>
<li><a href="#programming-language">Programming language</a></li>
<li><a href="#%E2%9A%A0%EF%B8%8F-notebook-software">⚠️ Notebook software</a></li>
<li><a href="#%E2%9A%A0%EF%B8%8F-ml-code-examples">⚠️ ML code examples</a></li>
<li><a href="#frameworks--libraries">Frameworks &amp; libraries</a></li>
<li><a href="#models">Models</a></li>
<li><a href="#datasets">Datasets</a></li>
</ul>
<h3 id="programming-language">Programming language</h3>
<p><a href="https://github.com/tomsoderlund/toms-guidebooks/blob/master/Development/Python.md">Python</a> is the main programming language for AI/ML.</p>
<h3 id="⚠️-notebook-software">⚠️ Notebook software</h3>
<p>For writing/documenting/sharing ML code – essentially word processors with embedded (Python) code blocks. Uses <code>.ipynb</code> files, which are JSON structures containing Python code blocks and Markdown text blocks:</p>
<ul>
<li><a href="https://jupyter.org/">Jupyter Notebook</a>: the original (see below)</li>
<li><a href="https://code.visualstudio.com/docs/datascience/jupyter-notebooks">VS Code</a> has built in Jupyter support</li>
<li><a href="https://colab.research.google.com/">Google Colab</a>: work with files on Google Drive and GitHub</li>
<li><a href="https://www.kaggle.com/">Kaggle</a></li>
<li><a href="https://gradio.app/">Gradio</a> for building UI’s inside notebooks</li>
<li>IPython: interactive Python and Jupyter.</li>
<li>REPL: interactive Python session. Read, Evaluate, Print and Loop.</li>
</ul>
<h4 id="jupyter-notebook">Jupyter Notebook</h4>
<ul>
<li>Press Tab to get autocompletion suggestions</li>
<li>Inside the parentheses of a function, pressing Shift + Tab will display signature and description of the function</li>
<li><code>?[function]</code> to get help on function: <code>?verify_images</code>, or <code>??verify_images</code> for full source code</li>
<li><code>doc([function])</code> to get documentation page</li>
<li><code>%debug</code>: inspect every variable</li>
</ul>
<h3 id="⚠️-ml-code-examples">⚠️ ML code examples</h3>
<ul>
<li><a href="https://huggingface.co/spaces">HuggingFace Spaces</a>: discover ML apps</li>
<li><a href="https://www.kaggle.com/code">Kaggle Code</a>: explore, run, share ML code</li>
</ul>
<h3 id="frameworks--libraries">Frameworks &amp; libraries</h3>
<ul>
<li><a href="https://pytorch.org/">PyTorch</a> – open source machine learning framework</li>
<li><a href="https://www.tensorflow.org/">TensorFlow</a> – Google<ul>
<li><a href="https://keras.io/">Keras</a> for deep learning</li>
</ul>
</li>
<li><a href="https://jax.readthedocs.io/en/latest/">JAX</a></li>
<li><a href="https://docs.fast.ai/">FastAI</a></li>
</ul>
<h4 id="pytorch">PyTorch</h4>
<p>Libraries:</p>
<ul>
<li><strong>torch</strong>: main library<ul>
<li><a href="https://pytorch.org/tutorials/beginner/basics/intro.html">Introductory neural network tutorial</a></li>
</ul>
</li>
<li><strong>torchaudio</strong>: sound</li>
<li><strong>torchtext</strong>: text/NLP</li>
<li><strong>torchvision</strong>: images and video</li>
<li><strong>torcharrow</strong>: data preprocessing in deep learning</li>
<li><strong>TorchData</strong>: data loading primitives for easily constructing flexible and performant data pipelines</li>
<li><strong>TorchRec</strong>: recommendations</li>
<li><strong>TorchServe</strong>: server</li>
<li><strong>TorchX</strong>: job launcher for PyTorch applications</li>
</ul>
<h3 id="models">Models</h3>
<p>Collections of models:</p>
<ul>
<li><a href="https://huggingface.co/models">HuggingFace</a>: collection of models, “GitHub for AI models”</li>
<li><a href="https://www.eleuther.ai/">EleutherAI</a>: creator of open source AI models</li>
<li><a href="https://openai.com/">OpenAI</a>: creator of private/closed source – GPT-3, DALL·E</li>
</ul>
<p>Some models:</p>
<ul>
<li>Text-to-Text:<ul>
<li>GPT-3: OpenAI</li>
<li>GPT-Neo 1.3B: a transformer model designed using EleutherAI’s replication of the GPT-3 architecture.</li>
<li>GPT-J 6B: a transformer model trained using Ben Wang&#39;s Mesh Transformer JAX.</li>
</ul>
</li>
<li>Text-to-Image:<ul>
<li>DALL·E: OpenAI</li>
<li>Midjourney</li>
<li>Stable Diffusion: Stability AI</li>
<li>Imagen: Google</li>
<li>Parti: Google</li>
</ul>
</li>
<li>Text-to-Video:<ul>
<li>Imagen Video: Google</li>
<li>Phenaki: Google</li>
<li>Make-a-Video: Meta</li>
</ul>
</li>
</ul>
<h3 id="datasets">Datasets</h3>
<ul>
<li><a href="https://www.kaggle.com/datasets">Kaggle Datasets</a></li>
<li><a href="https://toolbox.google.com/datasetsearch">Google Dataset Search</a></li>
<li><a href="https://scholar.google.com/">Google Scholar</a></li>
<li><a href="http://mlr.cs.umass.edu/ml/">UCI Machine Learning Repository</a></li>
<li><a href="https://www.visualdata.io/">VisualData</a></li>
<li><a href="https://guides.library.cmu.edu/machine-learning/datasets">CMU Libraries</a></li>
<li><a href="https://datasets.quantumstat.com/">The Big Bad NLP Database</a></li>
<li>Lists of datasets:<ul>
<li><a href="https://pub.towardsai.net/best-datasets-for-machine-learning-data-science-computer-vision-nlp-ai-c9541058cf4f">TowardsAI</a></li>
<li><a href="http://yulanguo.me/dataset.html">Image/video/3D datasets (Yulan Guo)</a></li>
</ul>
</li>
</ul>
<h2 id="machine-learning-theory">Machine Learning Theory</h2>
<p>Machine learning algorithm:</p>
<pre><code>Inputs + Weights → Model → Results + Loss
…then adjust Weights
</code></pre>
<p>⚠️ ML input data is <strong>always in number format</strong> (with decimals), even though it came from text/image/video/sound. See “Feature engineering” below.</p>
<h3 id="terminology">Terminology</h3>
<ul>
<li><strong>Model</strong>: what does the actual predicting; a file that has been trained to recognize certain types of patterns.</li>
<li><strong>Features</strong>: model inputs, <code>x₁</code>, <code>x₂</code>, etc.</li>
<li><strong>Label</strong>: model output, <code>y</code>, what we want to predict.</li>
<li><strong>Example</strong>: one piece of data (features, labeled or unlabeled).</li>
<li><strong>Dataset</strong>: a whole collection of examples.</li>
<li><strong>Batch</strong>: the set of examples used in one training iteration.</li>
<li><strong>Epoch</strong>: number of passes of the entire training dataset the machine learning algorithm has completed.</li>
<li><strong>Vector</strong>: an array of numbers.</li>
<li><strong>Matrix</strong>: a 2D array of numbers, rows × columns, e.g. 150×4.</li>
<li><strong>Tensor</strong>: a specialized data structure that are very similar to arrays and matrices. A tensor interacts with other entities in a system and changes its values when other values change.</li>
<li><strong>Slope</strong>: <code>m</code>, angle of the line in linear regression.</li>
<li><strong>Weight</strong>: <code>w</code>, the importance of a feature</li>
<li><strong>Bias</strong>: <code>b</code>, how well a model matches the training set.</li>
<li><strong>Loss</strong>: mismatch with model, <em>mean square error</em>.</li>
<li><strong>Learning rate</strong>: granularity when training a model</li>
<li><strong>Gradient descent</strong>: algorithm for finding minima.</li>
<li><strong>Hyperparameters</strong>: the “knobs” that programmers tweak in machine learning algorithms.</li>
<li><strong>Deep learning</strong>: artificial neural networks with representation learning.</li>
<li><strong>Data augmentation</strong>: creating random variations of our input data so they appear different, but do not actually change the meaning of the data. E.g. image rotation and flipping.</li>
<li><strong>Generalization</strong>: a model’s ability to adapt properly to new, previously unseen data.</li>
<li><strong>Feature engineering</strong>: transform inputs so they can be used in model.</li>
<li><strong>One-hot encoding</strong>: as 1 or 0.</li>
<li><strong>Binning</strong>: put category values into “buckets”.</li>
<li><strong>Feature crosses</strong>: combine features to create synthetic features.</li>
<li><strong>Decision boundaries</strong>: a surface that separates data points belonging to different class labels.</li>
<li><strong>Regularization</strong>: reducing complexity of model (e.g. L2).</li>
<li><strong>Logistic regression</strong>: output is a <em>probability</em> (0.0-1.0).</li>
</ul>
<h3 id="learning-types">Learning types</h3>
<ul>
<li><strong>Supervised learning</strong>: labeled data/output</li>
<li><strong>Reinforcement learning</strong>: reward signal, e.g. chess engine</li>
<li><strong>Unsupervised learning</strong>: hidden structures – clustering, dimensionality reduction</li>
</ul>
<p>Also: feature learning/representation learning</p>
<h3 id="neural-networks">Neural networks</h3>
<p>Neural networks definitely shine where you have data that is not easily transformable into features – for example images used to be hard to transform into good features.</p>
<p>NN math, simplified: multiply, add them up, replace negatives with zeroes.</p>
<ul>
<li><strong>CNN: Convolutional Neural Network</strong>, a deep learning neural network sketched for processing structured arrays of data such as portrayals.</li>
<li><strong>RNN: Recurrent Neural Network</strong>, a class of artificial neural networks where connections between nodes can create a cycle, allowing output from some nodes to affect subsequent input to the same nodes.</li>
<li><strong>Transformer</strong>: is a deep learning model that adopts the mechanism of self-attention, differentially weighting the significance of each part of the input data. It is used primarily in the fields of natural language processing (NLP) and computer vision (CV).</li>
</ul>
<h3 id="text-processing-with-large-language-models-llms">Text processing with Large Language Models (LLM’s)</h3>
<ul>
<li><strong>LLM</strong>: Large Language Model</li>
<li><strong>NER</strong>: Named Entity Recognition</li>
</ul>
<h3 id="vector-search-and-semantic-search">Vector search and Semantic search</h3>
<ul>
<li><strong>Vector search</strong>: matching search key’s vectors’ proximity in vector space with other items.</li>
<li><strong>Semantic search</strong>: NLP for transforming search phrase into vectors, then vector search for matching their proximity in vector space with other items. These vectors are usually called <strong>embeddings</strong>.</li>
<li><strong>Embeddings</strong>: special type of vector that is produced as a result of running an input through an ML model - usually a neural network.</li>
<li><strong>Dense retrieval</strong>: using embeddings to search.</li>
</ul>
<h2 id="deployment">Deployment</h2>
<p>There are two main use cases for deploying ML code: <strong>training</strong> and <strong>inference</strong>.</p>
<p>For inference, there are several options:</p>
<ol>
<li>Use your own cloud compute/GPU and have your backend use that.</li>
<li>AWS SageMaker Inference service and ping that from your backend.</li>
<li>GCP’s ML Inference service (Vertex AI) and ping that from your backend.</li>
<li>HuggingFace Inference Endpoints: <a href="https://huggingface.co/inference-endpoints">https://huggingface.co/inference-endpoints</a></li>
</ol>


What problems Artificial Intelligence (AI) can solve today. What tools to use. How to integrate machine learning (ML) in your websites and apps.

Beginners introduction to Artificial Intelligence (AI) and Machine Learning (ML)

<p>My own resources for Artificial Intelligence (AI) and Machine Learning (ML):</p>
<ul>
<li><a href="/ai/ai-introduction">Beginners introduction to AI/ML with notes &amp; links</a></li>
<li>“AI for Mere Mortals” course (coming): <a href="https://tomorroworld.podia.com/ai-for-mere-mortals">https://tomorroworld.podia.com/ai-for-mere-mortals</a></li>
<li>Code:<ul>
<li>REST API with Gradio and Huggingface Spaces: <a href="https://www.tomsoderlund.com/ai/building-ai-powered-rest-api">blog post</a>, <a href="https://huggingface.co/spaces/tomsoderlund/swedish-entity-recognition">code</a></li>
<li>ML model with a REST API (local only): <a href="https://github.com/tomsoderlund/ml-on-rest-api">https://github.com/tomsoderlund/ml-on-rest-api</a></li>
<li>Some ML code snippets: <a href="https://github.com/tomsoderlund/toms-guidebooks/blob/master/Development/AI%20and%20Machine%20Learning.md">https://github.com/tomsoderlund/toms-guidebooks/blob/master/Development/AI%20and%20Machine%20Learning.md</a></li>
</ul>
</li>
</ul>


My own resources for Artificial Intelligence (AI) and Machine Learning (ML):
	Beginners introduction to AI/ML with notes

My AI/ML resources

<p>I want to use machine learning models in my web and mobile apps, but to do so I must host my ML app somewhere.</p>
<p>Hosting a pre-trained ML model is called <em>inference</em>. I just want to drop in some Python ML code and quickly get a REST API, but finding a simple way to do this has proven more difficult than I expected.</p>
<blockquote>
<p>“I just want to drop in some Python ML code and get a REST API”</p>
</blockquote>
<p>There are plenty of hosting providers, including the big ones like Amazon AWS and Google GCP, but the process of using these are complex and often requires building your own <a href="https://flask.palletsprojects.com/">Flask</a> REST API.</p>
<p>Luckily, <a href="https://twitter.com/prundin/status/1597595173562859523?s=20&amp;t=L72CeSiIQCVcgom0_cMAJA">@prundin and @abidlabs had a solution</a>: Gradio and Hugging Face “Spaces”.</p>
<h2 id="new-space-on-hugging-face">New Space on Hugging Face</h2>
<p><a href="https://huggingface.co/">Hugging Face</a> is like GitHub but for ML models and apps. A “Space” on Hugging Face is an ML app that you can update via Git. Spaces are <a href="https://huggingface.co/pricing#spaces">priced based on CPU type</a>, and the simplest one is free!</p>
<p><a href="https://gradio.app/">Gradio</a> is a library of user interface components (e.g. input fields) for ML apps.</p>
<p>Create a new Space by:</p>
<ul>
<li>Go to <a href="https://huggingface.co/spaces">https://huggingface.co/spaces</a> and click <strong>Create new Space</strong>.</li>
<li>Select the <a href="https://huggingface.co/blog/open_rail">OpenRAIL license</a> if you’re unsure.</li>
<li>Select <strong>Gradio</strong> as Space SDK.</li>
</ul>
<p><img src="/blog/images/huggingface-new-space.png" alt="Create a new Hugging Face Space"></p>
<h2 id="local-development-setup">Local development setup</h2>
<p>Go to your development folder and clone your space:</p>
<pre><code>git clone https://huggingface.co/spaces/USER_NAME/SPACE_NAME

cd SPACE_NAME
</code></pre>
<p>Set up Python, Gradio, etc:</p>
<pre><code># Create and activate a “safe” virtual Python environment (exit with command “deactivate”)
python3 -m venv env
source env/bin/activate

# Create a .gitignore file to exclude the packages in `env` folder
echo &quot;env/&quot; &gt;&gt; .gitignore

# Install Gradio
pip3 install gradio

# Update the list of required packages (do this every time you add packages)
pip3 freeze &gt; requirements.txt
</code></pre>
<h2 id="settings-in-the-readme-file">Settings in the README file</h2>
<p>The <code>README.md</code> contains some crucial settings for your app.</p>
<p>I struggled with some errors in my live environment on Hugging Face, until I realized I had to lock down the Python version to be the same as I use locally. By running:</p>
<pre><code>python3 --version
</code></pre>
<p>locally, and then adding the version number to <code>README.md</code> like this:</p>
<pre><code>python_version: 3.9.13
</code></pre>
<h2 id="creating-a-simple-user-interface-and-rest-api">Creating a simple user interface and REST API</h2>
<p>Create a blank app.py file:</p>
<pre><code>touch app.py
</code></pre>
<p>Edit <code>app.py</code>:</p>
<pre><code>import gradio

def my_inference_function(name):
  return &quot;Hello &quot; + name + &quot;!&quot;

gradio_interface = gradio.Interface(
  fn = my_inference_function,
  inputs = &quot;text&quot;,
  outputs = &quot;text&quot;
)
gradio_interface.launch()
</code></pre>
<p>Upload the files to Hugging Face using Git:</p>
<pre><code>git add .
git commit -m &quot;Creating app.py&quot;
git push
</code></pre>
<h2 id="testing-your-app-on-hugging-face">Testing your app on Hugging Face</h2>
<p>Believe it or not, but you now have a functional app with a REST API!</p>
<p>Open <a href="https://huggingface.co/spaces/USER_NAME/SPACE_NAME">https://huggingface.co/spaces/USER_NAME/SPACE_NAME</a> in your browser.</p>
<p>See an example here: <a href="https://huggingface.co/spaces/tomsoderlund/rest-api-with-gradio">https://huggingface.co/spaces/tomsoderlund/rest-api-with-gradio</a></p>
<p><img src="/blog/images/huggingface-space-running.png" alt="Hugging Face space when running"></p>
<p><strong>User interface:</strong> You can enter something in the <code>name</code> box and press <strong>Submit</strong>.</p>
<p><strong>REST API interface:</strong> At the bottom of the page you have a link called “Use via API”. Click it for instructions, but you can now call your app with REST:</p>
<pre><code>curl -X POST -H &#39;Content-type: application/json&#39; --data &#39;{ &quot;data&quot;: [&quot;Jill&quot;] }&#39; https://USER_NAME-SPACE_NAME.hf.space/run/predict
</code></pre>
<p>The REST API would return:</p>
<pre><code>{
  &quot;data&quot;:[
    &quot;Hello Jill!&quot;
  ],
  &quot;is_generating&quot;:false,
  &quot;duration&quot;:0.00015354156494140625,
  &quot;average_duration&quot;:0.00015354156494140625
}
</code></pre>
<h2 id="testing-your-app-locally">Testing your app locally</h2>
<p>Locally, you can run:</p>
<pre><code>python3 app.py
</code></pre>
<p>You can now test your app interactively on: <a href="http://127.0.0.1:7860/">http://127.0.0.1:7860/</a>
and access the REST API on <a href="http://127.0.0.1:7860/run/predict">http://127.0.0.1:7860/run/predict</a></p>
<p>(You need to stop (Ctrl+C) and restart your app when you modify <code>app.py</code>)</p>
<h2 id="going-further">Going further</h2>
<p>You can now <a href="https://huggingface.co/models">explore all the models on Hugging Face</a>, including <a href="https://huggingface.co/stabilityai/stable-diffusion-2">Stable Diffusion 2</a> and <a href="https://huggingface.co/EleutherAI/gpt-neo-2.7B">GPT-Neo</a>, and add them to your Spaces app.</p>
<p>See a complete ML example on: <a href="https://huggingface.co/spaces/tomsoderlund/swedish-entity-recognition">https://huggingface.co/spaces/tomsoderlund/swedish-entity-recognition</a> – you can see how few lines of code are actually needed to invoke the ML model.</p>
<p><img src="/blog/images/huggingface-rest-api-running.png" alt="A complete ML example with REST API"></p>


“I just want to drop in some Python ML code and get a REST API”

Building an AI powered REST API with Gradio and Hugging Face Spaces – for free!

<p><img src="/blog/images/AI_for_Mere_Mortals.jpg" alt="AI for Mere Mortals"></p>
<p>I’m bullish about Artificial Intelligence (AI).</p>
<p>I firmly believe that it is a new computing paradigm shift, on the same magnitude as smartphones and the internet. Ignoring AI in 2022 is like ignoring the iPhone in 2010.</p>
<p>It requires a new way of thinking and approaching problems. My background as an entrepreneur and a software developer is not completely helpful; some things need to be unlearned and relearned.</p>
<p>Wanna learn with me?</p>
<p>I’m collecting my learnings into the curriculum I was missing when I got started learning AI/ML:</p>
<p><a href="https://tomorroworld.podia.com/ai-for-mere-mortals">https://tomorroworld.podia.com/ai-for-mere-mortals</a></p>


AI for Mere Mortals

I’m bullish about Artificial Intelligence (AI).

I firmly believe that it is a new computing paradigm shift, on the same magnitude as smartphones and the internet. Ignoring AI in

AI for Mere Mortals

<p><img src="/blog/images/06070-1btnyhbyjmu6v650a3ngcoq.png" alt=""></p>
<p>It might sound like science-fiction, but it makes sense. AI’s are becoming increasingly powerful, but creativity is not their strongest suite. But they’re great at performing tasks such as organizing, counting KPI’s, and doing administrative tasks. Tasks which creative entrepreneurs are less adept for.</p>
<p>Paired with a creative entrepreneur, they could form a strong team. Man and machine, working as one.</p>
<p>Additionally, AI is red hot among VC’s, and I bet some of them would be excited to fund the first hybrid human/AI founder teams.</p>


It might sound like science-fiction, but it makes sense. AI’s are becoming increasingly powerful, but creativity is not their strongest suite. But they’re great at performing tasks such as organizi

When will we see AI’s as part of startup founder teams?

Read Tom Söderlund’s articles about artificial intelligence & machine learning.

Artificial Intelligence & Machine Learning