This is invaluable because it allows developers to test the API experience before they write any real code. With a mock server, developers can spot potential issues early in the API design process and quickly iterate by modifying a YAML or JSON document instead of having to make expensive code changes.

Here's how to get Prism set up on your machine.

First, we install Prism using npm or yarn.

npm install -g @stoplight/prism-cli

# OR

yarn global add @stoplight/prism-cli

Next, we'll need an OpenAPI document. Use one you have already, get a copy of the standard sample OpenAPI document of a Petstore API, or use the one I created in Creating an OpenAPI Specification the Hard Way.

Now it's as easy as running Prism and referencing your OpenAPI file.

prism mock my_open_api_doc.yaml

If all goes well, your output should be a list of all your endpoints with local server paths.

[2:59:43 PM] › [CLI] …  awaiting  Starting Prism…
[2:59:43 PM] › [CLI] ℹ  info      GET        http://127.0.0.1:4010/definitions/tortellini
[2:59:43 PM] › [CLI] ℹ  info      PUT        http://127.0.0.1:4010/definitions/tortellini
[2:59:43 PM] › [CLI] ℹ  info      DELETE     http://127.0.0.1:4010/definitions/tortellini
[2:59:43 PM] › [CLI] ℹ  info      POST       http://127.0.0.1:4010/definitions
[2:59:43 PM] › [CLI] ▶  start     Prism is listening on http://127.0.0.1:4010

If the command fails, it's likely due to minor syntax issues in your document. Use a YAML or JSON validator to check for those. Or check out one of these OpenAPI linters.

Assuming everything is working, let's do a simple GET to our local server at http://127.0.0.1:4010.

curl -i http://127.0.0.1:4010/definitions/tagliatelle

Ideally, we get a response like:

curl -i http://127.0.0.1:4010/definitions/tagliatelle
HTTP/1.1 200 OK
Access-Control-Allow-Origin: *
Access-Control-Allow-Headers: *
Access-Control-Allow-Credentials: true
Access-Control-Expose-Headers: *
Content-type: application/json
Content-Length: 225
Date: Mon, 29 Jan 2024 14:18:59 GMT
Connection: keep-alive
Keep-Alive: timeout=5

{"id":40125,"word":"tagliatelle","created_at":"2023-12-06T04:55:36.887Z","updated_at":"2022-12-06T04:55:36.887Z","meanings":[{"id":333,"part_of_speech":"noun","meaning":"A type of pasta shaped into long, thin, flat strips"}]}

The response body is constructed by whatever we put in the responses child object of any API path. In this case, our GET endpoint references the definitions schema object which includes the above sample data.

Your mock server will also give you server logs notifying you of any validation issues. Here's what a good GET event should look like:

[3:18:59 PM] › [HTTP SERVER] get /definitions/tortellini ℹ  info      Request received
[3:18:59 PM] ›     [NEGOTIATOR] ℹ  info      Request contains an accept header: */*
[3:18:59 PM] ›     [VALIDATOR] ✔  success   The request passed the validation rules. Looking for the best response
[3:18:59 PM] ›     [NEGOTIATOR] ✔  success   Found a compatible content for */*
[3:18:59 PM] ›     [NEGOTIATOR] ✔  success   Responding with the requested status code 200

And if we try a route like /v2/definitions/tagliatelle that doesn't exist in our OpenAPI spec, we see Prism handles that as expected with a 404 Not Found response.

curl -i http://127.0.0.1:4010/v2/definitions/tagliatelle
HTTP/1.1 404 Not Found
Access-Control-Allow-Origin: *
Access-Control-Allow-Headers: *
Access-Control-Allow-Credentials: true
Access-Control-Expose-Headers: *
content-type: application/problem+json
Content-Length: 218
Date: Tue, 30 Jan 2024 09:01:27 GMT
Connection: keep-alive
Keep-Alive: timeout=5

{"type":"https://stoplight.io/prism/errors#NO_PATH_MATCHED_ERROR","title":"Route not resolved, no path matched","status":404,"detail":"The route /v2/definitions/tagliatelle hasn't been found in the specification file"}

We can also test a POST request protected with Basic Auth. If you've protected your API with Basic Auth, Prism will expect to see a Basic Auth header included in the request. Of course it can't actually validate the token, but it's helpful enough in the early development phase of an API

curl -i --request POST \
  --url http://127.0.0.1:4010/definitions \
  --header 'Authorization: Basic dXNlcm5hbWU6cGFzc3dvcmQ=' \
  --header 'Content-Type: application/json' \
  --data '{
  "word": "tagliatelle",
  "meanings": [
    {
      "part_of_speech": "noun",
      "meaning": "A type of pasta shaped into long, thin, flat strips"
    }
  ]
}'
HTTP/1.1 201 Created
Access-Control-Allow-Origin: *
Access-Control-Allow-Headers: *
Access-Control-Allow-Credentials: true
Access-Control-Expose-Headers: *
Content-type: application/json
Content-Length: 225
Date: Mon, 29 Jan 2024 14:36:02 GMT
Connection: keep-alive
Keep-Alive: timeout=5

{"id":40125,"word":"tagliatelle","created_at":"2022-12-06T04:55:36.887Z","updated_at":"2022-12-06T04:55:36.887Z","meanings":[{"id":333,"part_of_speech":"noun","meaning":"A type of pasta shaped into long, thin, flat strips"}]}

Prism will also handle the various erroneous requests we might want to test. You can try removing the Basic Auth header, and you'll get a 401 Unauthorized.

curl -i --request POST \
  --url http://127.0.0.1:4010/definitions \
  --header 'Content-Type: application/json' \
  --data '{
  "word": "tagliatelle",
  "meanings": [
    {
      "part_of_speech": "noun",
      "meaning": "A type of pasta shaped into long, thin, flat strips"
    }
  ]
}'
HTTP/1.1 401 Unauthorized
Access-Control-Allow-Origin: *
Access-Control-Allow-Headers: *
Access-Control-Allow-Credentials: true
Access-Control-Expose-Headers: *
sl-violations: [{"location":["request"],"severity":"Error","code":401,"message":"Invalid security scheme used"}]
Date: Mon, 29 Jan 2024 14:40:17 GMT
Connection: keep-alive
Keep-Alive: timeout=5
Content-Length: 0

Changing the Content-Type header to application/xml when our API doesn't support XML returns a 422 Unprocessable Entity.

curl -i --request POST \
  --url http://127.0.0.1:4010/definitions \
  --header 'Authorization: Basic dXNlcm5hbWU6cGFzc3dvcmQ=' \
  --header 'Content-Type: application/xml' \
  --data '{
  "word": "tagliatelle",
  "meanings": [
    {
      "part_of_speech": "noun",
      "meaning": "A type of pasta shaped into long, thin, flat strips"
    }
  ]
}'
HTTP/1.1 422 Unprocessable Entity
Access-Control-Allow-Origin: *
Access-Control-Allow-Headers: *
Access-Control-Allow-Credentials: true
Access-Control-Expose-Headers: *
sl-violations: [{"location":["request"],"severity":"Error","code":415,"message":"Supported content types: application/json"}]
Date: Mon, 29 Jan 2024 14:41:51 GMT
Connection: keep-alive
Keep-Alive: timeout=5
Content-Length: 0

Adding a pronunciation field to our request body that doesn't exist in our schema returns the expected 400 Bad Request error.

curl -i --request POST \
  --url http://127.0.0.1:4010/definitions \
  --header 'Authorization: Basic dXNlcm5hbWU6cGFzc3dvcmQ=' \
  --header 'Content-Type: application/json' \
  --data '{
  "pronunciation: "taʎʎaˈtɛlle",
  "word": "tagliatelle",
  "meanings": [
    {
      "part_of_speech": "noun",
      "meaning": "A type of pasta shaped into long, thin, flat strips"
    }
  ]
}'
HTTP/1.1 400 Bad Request
Access-Control-Allow-Origin: *
Access-Control-Allow-Headers: *
Access-Control-Allow-Credentials: true
Access-Control-Expose-Headers: *
Content-Type: application/json; charset=utf-8
Content-Length: 58
Date: Mon, 29 Jan 2024 14:42:42 GMT
Connection: keep-alive
Keep-Alive: timeout=5

{"error":{"code":"invalid_json","message":"Invalid JSON"}}

A Prism mock server is no replacement for a fully working prototype, but it is an invaluable tool for the early design phase of an API as stakeholders are negotiating over how the API should function. It's certainly worth a whirl.

This gives businesses a powerful tool for monitoring search results that can help optimize their SEO, track news, or train AI models. Ultimately, the only limitation is one's own creativity.

My idea was to wire up SerpApi with OpenAI to get ChatGPT to analyze search results based on user-generated prompts. Here's how I built it.

Initial Setup

rails new search-results-ai-analyzer
cd search-results-ai-analyzer
rails db:migrate

I'm also going to install Tailwind CSS to make it a slightly easier on the eyes. I suggest following the installation steps on the Tailwind website.

Check that everything is working by firing up the server. With Tailwind installed, we start the server by running bin/dev.

Next, we'll create a SearchAnalysis model using Rails' scaffold generator to save us some time. The model needs the following fields:

• query: what we want to search
• engine: enum of which search engine to use
• prompt: what we want to ask ChatGPT to do
• search_results: the results from SerpApi
• chat_response: the response from ChatGPT
• status: enum of where we are in the process (e.g. pending, failed, complete)

Run the scaffold generator and migrate the database with the commands below.

rails g scaffold SearchAnalysis query:text engine:integer prompt:text search_results:text chat_response:text status:integer

The status field should have a default value of 0. This must be manually set in the migration file before running the migration. 0 will be defined as pending_search in the model file.

# db/migrate/20240122133152_create_search_analyses.rb

class CreateSearchAnalyses < ActiveRecord::Migration[7.0]
  def change
    create_table :search_analyses do |t|
      t.text :query
      t.integer :engine
      t.text :prompt
      t.text :search_results
      t.text :chat_response
      t.integer :status, default: 0

      t.timestamps
    end
  end
end

Now we run our migration.

rails db:migrate

Add our enum values and some very basic validations for user submitted data.

# app/models/search_analysis.rb

class SearchAnalysis < ApplicationRecord
  enum :engine, { google: 0, bing: 1, duckduckgo: 2, yahoo: 3, yandex: 4, baidu: 5 }
  enum :status, { pending_search: 0, pending_chat: 1, complete: 2, failed: 3 }

  validates :query, presence: true
  validates :engine, presence: true
  validates :prompt, presence: true
end

Change the root path to map to search_analyses#index in your config/routes.rb file.

# config/routes.rb

Rails.application.routes.draw do
  resources :search_analyses

  root "search_analyses#index"
end

We don't want our forms to ever accept user submitted data for the status, search_results and chat_response fields so we'll remove those from our strong params method.

# app/controllers/search_analyses_controller.rb

    # Only allow a list of trusted parameters through.
    def search_analysis_params
      params.require(:search_analysis).permit(:query, :engine, :prompt)
    end

We should also remove those fields from our form so the user doesn't even see them.

While we're editing our form, let's also improve the field labels to be more clear and also have the search engine field use a select dropdown.

Your form should look like this:

# app/views/search_analyses/_form.html.erb

<%= form_with(model: search_analysis, class: "contents") do |form| %>
  <% if search_analysis.errors.any? %>
    <div id="error_explanation" class="bg-red-50 text-red-500 px-3 py-2 font-medium rounded-lg mt-3">
      <h2><%= pluralize(search_analysis.errors.count, "error") %> prohibited this search_analysis from being saved:</h2>

      <ul>
        <% search_analysis.errors.each do |error| %>
          <li><%= error.full_message %></li>
        <% end %>
      </ul>
    </div>
  <% end %>

  <div class="my-5">
    <%= form.label :query, "Search Query" %>
    <%= form.text_field :query, rows: 4, class: "block shadow rounded-md border border-gray-200 outline-none px-3 py-2 mt-2 w-full" %>
  </div>

  <div class="my-5">
      <%= form.label :engine, "Search Engine" %>
      <%= form.select(:engine, ['google', 'bing', 'duckduckgo', 'yahoo', 'yandex', 'baidu'], { :class => "block shadow rounded-md border border-gray-200 outline-none px-3 py-2 mt-2 w-full" }) %>
  </div>

  <div class="my-5">
    <%= form.label :prompt, "Chat Prompt" %>
    <%= form.text_area :prompt, rows: 3, class: "block shadow rounded-md border border-gray-200 outline-none px-3 py-2 mt-2 w-full" %>
  </div>

  <div class="inline">
    <%= form.submit class: "rounded-lg py-3 px-5 bg-blue-600 text-white inline-block font-medium cursor-pointer" %>
  </div>
<% end %>

As an optional but recommended step, you can remove the following things the scaffold generated that we don't need.

1. The edit and update actions in the SearchAnalyses controller.
2. The entire edit view file views/search_analyses/edit.html.erb
3. Remove the "Edit this search analysis" buttons from views/search_analyses/_search_analysis.html.erb and views/search_analyses/show.html.erb

Give your form a quick test before moving on.

Connecting to OpenAI/ChatGPT

Now we'll wire up our app to talk to OpenAI's API. If you don't have an OpenAI API account yet, create one here first and then find the API Keys page. Create a key and store it somewhere safe. OpenAI will only show you the key once.

Hopefully OpenAI will give you some free credit to play around with the API. If not, you'll need to add at least $5 credit on the Billing page. $5 will go very far for just hobby usage.

We'll be using the ruby-openai gem. It's not an official library made by OpenAI, but it more than does the job and OpenAI recommends it in their API documentation.

Add gem "ruby-openai" to your Gemfile and run bundle install.

We'll also want to create an initializer file to configure the gem with our OpenAI API key.

touch config/initializers/openai.rb

Add the following to the file.

# config/initializers/openai.rb

OpenAI.configure do |config|
  config.access_token = ENV.fetch("OPENAI_ACCESS_TOKEN")
end

We'll also need to set this environment variable in our command line to get this to work.

export OPENAI_ACCESS_TOKEN=your_api_key_here

We can test all this is working by opening up the rails console. You should be able to run client = OpenAI::Client.new to create a client and then client.models.list to get a list of available AI models.

user@host:~/search-results-ai-analyzer$ rails c
Loading development environment (Rails 7.0.8)
irb(main):001> client = OpenAI::Client.new
=> 
#<OpenAI::Client:0x00007fa6404fa6f8
...
irb(main):002> client.models.list
=> 
{"object"=>"list",
 "data"=>
  [{"id"=>"curie-search-query",
    "object"=>"model",
    "created"=>1651172509,
    "owned_by"=>"openai-dev"},
   {"id"=>"babbage-search-query",
    "object"=>"model",
    "created"=>1651172509,
    "owned_by"=>"openai-dev"},
   {"id"=>"dall-e-3",
    "object"=>"model",
    "created"=>1698785189,
    "owned_by"=>"system"},
   {"id"=>"babbage-search-document",
    "object"=>"model",
    "created"=>1651172510,
    "owned_by"=>"openai-dev"},
   {"id"=>"dall-e-2",
...

If the above fails, the most likely reason is because of an issue configuring your API key. Ensure your environment variable is set and the initializer file references it correctly.

We can hit the OpenAI API technically from nearly anywhere in our Rails app, but I prefer dropping this logic into a service. We'll create two directories and two files.

mkdir app/services
mkdir app/services/openai_services
touch app/services/openai_services.rb
touch app/services/openai_services/chat_service.rb

In app/services/openai_services.rb, we can initialize the module with the following code.

# app/services/openai_services.rb

module OpenAIServices
end

To be able to reference this module as OpenAIServices, we need to make an addition to our config/initializers/inflections.rb file. Otherwise, our Rails app will be expecting OpenaiServices because the file name is openai_services.rb and Rails doesn't know the ai part should read as AI. We'll encounter the same problem with SerpApi. We'll fix both with the below code.

# config/initializers/inflections.rb

ActiveSupport::Inflector.inflections(:en) do |inflect|
  inflect.acronym "OpenAI"
  inflect.acronym "SerpApi"
end

Open up our ChatService file and let's get to work on hitting the OpenAI API. We'll start by creating a ChatService class that is initialized with a search_analysis object. Our SearchAnalysis model includes everything we might need for our AI prompt.

# app/services/openai_services/chat_service.rb

module OpenAIServices
  class ChatService
    attr_reader :search_analysis
    
    def initialize(search_analysis)
      @search_analysis = search_analysis
    end
  end
end

Next, we'll create a call method that will handle the logic of sending a message to the Chat API.

# app/services/openai_services/chat_service.rb

module OpenAIServices
  class ChatService
    attr_reader :search_analysis
    
    def initialize(search_analysis)
      @search_analysis = search_analysis
    end
    
    def call
      client = OpenAI::Client.new
      response = client.chat(
        parameters: {
          model: "gpt-3.5-turbo",
          messages: [{ role: "user", content: @search_analysis.prompt}],
          temperature: 0,
        })
      chat_response = response.dig("choices", 0, "message", "content")
    end
  end
end

We ought to test this connection in rails console before continuing.

user@host:~/search-results-ai-analyzer$ rails c
Loading development environment (Rails 7.0.8)
irb(main):001> search_analysis = SearchAnalysis.new(prompt: "Hey, can you hea
r me?")
=> 
#<SearchAnalysis:0x00007f6684c5a7f0
...
irb(main):002> OpenAIServices::ChatService.new(search_analysis).call
=> "Yes, I can hear you. How can I assist you today?"

Great, we got a working response back from ChatGPT. Now we'll improve the logic in our call method to follow typical Rails service object conventions of returning an OpenStruct.

# app/services/openai_services/chat_service.rb

module OpenAIServices
  class ChatService
    attr_reader :search_analysis
    
    def initialize(search_analysis)
      @search_analysis = search_analysis
    end
    
    def call
      begin
        client = OpenAI::Client.new
        response = client.chat(
          parameters: {
            model: "gpt-3.5-turbo",
            messages: [{ role: "user", content: @search_analysis.prompt}],
            temperature: 0,
          })
        chat_response = response.dig("choices", 0, "message", "content")
      rescue StandardError => e
        OpenStruct.new(success?: false, error: e)
      else
        OpenStruct.new({success?: true, openai_response: chat_response})
      end
    end
  end
end

The prompt we're sending to ChatGPT is currently just what the user put in the form, but we actually need to append some additional context to their prompt. Remember this is a search results analyzer so we'll need to include the search results in the prompt. It'd also be a good idea to tell ChatGPT what our search query was. Let's add a private method called generate_prompt to handle this logic.

# app/services/openai_services/chat_service.rb

    private
    
      def generate_prompt
        <<~PROMPT
          #{@search_analysis.prompt}

          My search query: #{@search_analysis.query}"

          The search results were:
          #{@search_analysis.search_results}
        PROMPT
      end

We also need to update the content value in the chat parameters to call our method messages: [{ role: "user", content: generate_prompt}].

# app/services/openai_services/chat_service.rb
    
    def call
      begin
        client = OpenAI::Client.new
        response = client.chat(
          parameters: {
              model: "gpt-3.5-turbo",
              messages: [{ role: "user", content: generate_prompt}],
              temperature: 0,
          })
        chat_response = response.dig("choices", 0, "message", "content")
      rescue StandardError => e
        OpenStruct.new(success?: false, error: e)
      else
        OpenStruct.new({success?: true, results: chat_response})
      end
    end

We'll do our rails console check again.

user@host:~/search-results-ai-analyzer$ rails cLoading development environment (Rails 7.0.8)
irb(main):001* search_analysis = SearchAnalysis.new(
irb(main):002*   query: "italian city with best food",
irb(main):003*   prompt: "Analyze the search results below and tell me which city I should visit if I'm a foodie. Tell me why as well",
irb(main):004*   search_results: "Surveys confirm Bologna as the food capital of Italy thanks to its wonderful food products like mortadella."
irb(main):005> )
=> 
#<SearchAnalysis:0x00007f537cd93c40
...
irb(main):006> OpenAIServices::ChatService.new(search_analysis).call
=> #<OpenStruct success?=true, results="Based on the search results, it is suggested that you should visit Bologna if you are a foodie. Bologna is known as the food capital of Italy, and it offers wonderful food products like mortadella. This implies that Bologna has a rich culinary tradition and is renowned for its delicious Italian cuisine. As a foodie, you can expect to indulge in a wide variety of authentic and high-quality Italian dishes in Bologna, making it an ideal destination for your culinary exploration.">

This all looks good. We'll leave this for now and move on to our SerpApi connection.

Connecting to SerpApi

First, create a SerpApi account and then locate your API key.

SerpApi's free plan grants 100 searches per month. For our hobby purposes, this will be enough, but we should be judicious with how we use them. Fortunately, SerpApi caches recent searches so if you re-run the same search several times over the course of an afternoon, you'll get cached results that don't consume your free search allowance.

It's also worth noting your search history is stored in your dashboard with the complete JSON response as well as a UI rendering of what your search results would have looked like in the browser.

From what I can tell from job postings, SerpApi uses Ruby on Rails as a major part of their tech stack. They must be Rubyists at heart so course they have an official Ruby gem that we'll be using.

Add gem "google_search_results" to your Gemfile and run bundle install.

The following steps we'll be very similar to what we did for the OpenAI connection. First, we'll create an initializer to set our API key.

touch config/initializers/serpapi.rb

Set the key using an environment variable again.

export SERPAPI_API_KEY=your_api_key_here

And reference that environment variable in the initializer. We'll use the generic SerpApi class SerpApiSearch which allows us to search any engine as long as we specify the engine in the request.

If you only wanted to use a particular search engine, you can set the key on classes like GoogleSearch, DuckDuckgoSearch, YandexSearch, etc. By doing this, you can omit the engine in the API request since the engine is already set via the class. See the docs for more on this.

# config/initializers/serpapi.rb

SerpApiSearch.api_key = ENV.fetch("SERPAPI_API_KEY")

We'll test our setup in rails console.

user@host:~/search-results-ai-analyzer$ rails c
Loading development environment (Rails 7.0.8)
irb(main):001* query = {
irb(main):002*   q: "italian city with best food",
irb(main):003*   engine: "google"
irb(main):004> }
=> {:q=>"italian city with best food", :engine=>"google"}
irb(main):005> search = SerpApiSearch.new(query)
irb(main):006> organic_results = search.get_hash[:organic_results]
=> 
[{:position=>1,
...
irb(main):007> organic_results
=> 
[{:position=>1,
  :title=>"9 Best Food Cities in Italy",
  :link=>"https://www.celebritycruises.com/blog/best-food-cities-in-italy",
  :redirect_link=>
   "https://www.google.comhttps://www.celebritycruises.com/blog/best-food-cities-in-italy",
  :displayed_link=>
   "https://www.celebritycruises.com › blog › best-food-ci...",
  :thumbnail=>
   "https://serpapi.com/searches/65af9352fdca3e33d38b4de0/images/5c56f82085de25fb7b377b8923ee596b106bddcb2a9676ce6af1dc1ce6a4662b.jpeg",
  :favicon=>
   "https://serpapi.com/searches/65af9352fdca3e33d38b4de0/images/5c56f82085de25fb7b377b8923ee596b5a12458ea0949e9ab510eb6af83484f2.png",
  :date=>"Nov 29, 2023",
  :snippet=>
   "9 Best Food Cities in Italy · Bologna · Palermo · Rome · Florence · Cagliari · Sorrento · Genoa · Parma. Street view of Parma. Parma. Often ...",
  :snippet_highlighted_words=>["Best Food Cities", "Italy"],
...

This all looks good. Now onto our service object. Let's create the module and directory.

mkdir app/services/serpapi_services
touch app/services/serpapi_services.rb
touch app/services/serpapi_services/search_service.rb

# app/services/serpapi_services.rb

module SerpApiServices
end

Our initial setup looks exactly the same as our ChatService where we initialize an instance of the SearchService with a search_analysis.

# app/services/serpapi_services/search_service.rb

module SerpApiServices
  class SearchService
    attr_reader :search_analysis
    
    def initialize(search_analysis)
      @search_analysis = search_analysis
    end
  end
end

We'll then create a basic call method.

# app/services/serpapi_services/search_service.rb

module SerpApiServices
  class SearchService
    attr_reader :search_analysis
    
    def initialize(search_analysis)
      @search_analysis = search_analysis
    end
    
    def call
      search = SerpApiSearch.new(
        q: @search_analysis.query, 
        engine: @search_analysis.engine
      )
      search.get_hash[:organic_results]
    end
  end
end

We could theoretically take the whole JSON response from SerpApi and send it to ChatGPT for analysis. I've tested this and ChatGPT understands the content of the JSON well enough to analyze it. Trouble is the JSON body is very verbose with a lot of characters there are irrelevant to ChatGPT. This is an issue because OpenAI charges more for prompts with a lot of text. We'll need to cut down how much of the response body we use.

We'll simplify the search results to only use organic results and only use page titles, snippets, and links. We'll also save just the content and none of the JSON syntax. This will make it more readable for both humans and ChatGPT as well as keep the character count down.

We'll handle this logic in a private textify_results method. I'll also add the OpenStruct responses to our call method.

# app/services/serpapi_services/search_service.rb

  private
  
    def textify_results(results)
      concatenated_text = ""

      results.each do |result|
        concatenated_text += <<~RESULT
          Title: #{result[:title]}
          Link: #{result[:link]}
          Snippet: #{result[:snippet]}
        RESULT
      end

      return concatenated_text
    end

Test it in rails console.

user@host:~/search-results-ai-analyzer$ rails c
Loading development environment (Rails 7.0.8)
irb(main):001* search_analysis = SearchAnalysis.new(
irb(main):002*   query: "italian city with best food",
irb(main):003*   engine: "google"
irb(main):004> )
irb(main):005> test_search = SerpApiServices::SearchService.new(search_analysis).call
=> #<OpenStruct success?=true, results="Title: 9 Best Food Cities in Ita...
irb(main):006> test_search.results
=> "Title: 9 Best Food Cities in Italy\nLink: https://www.celebritycruises.com/blog/best-food-cities-in-italy\nSnippet: 9 Best Food Cities in Italy · Bologna · Palermo · Rome · Florence · Cagliari · Sorrento · Genoa · Parma. Street view of Parma. Parma. Often ...\n\nTitle: 10 of the best Italian cities for food - Times Travel\nLink: https://www.thetimes.co.uk/travel/destinations/europe/italy/italy-food-guide\nSnippet: Italy's best cities for food · 1. Florence · 2. Venice · 3. Naples · 4. Rome · 5. Milan · 6. Bologna · 7. Genoa · 8. Parma.\n\n
...

All good here. Onto the next task.

Linking It All Together

We finally have both APIs wired up and a model ready to go. Now we need to link all these pieces together.

We'll start in our SearchAnalysis model and create methods that will call our API service objects.

First, a simplerun_search method. This will call our SerpApi service and if the response is successful, it will store the results in search_results.

If it fails, we'll record the error message and mark the record as failed. Ideally, the error message would be cleaned up to present something user friendly, but that can be left aside for now.

# app/models/search_analysis.rb

  def run_search
    # We only run the search if the status is "pending_search"
    return unless self.status == "pending_search"
    
    search_response = SerpApiServices::SearchService.new(self).call
    
    if search_response.success?
      self.search_results = search_response.results
      self.status = "pending_chat"
    else
      self.search_results = "Failed due to error: #{search_response.error.to_s}"
      self.status = "failed"
    end
    
    self.save
  end

A quick test in rails console confirms this works and our search results get saved.

user@host:~/search-results-ai-analyzer$ rails c
Loading development environment (Rails 7.0.8)
irb(main):001* search_analysis = SearchAnalysis.new(
irb(main):002*   query: "italian city with best food",
irb(main):003*   engine: "google",
irb(main):004> prompt: "Analyze the search results below and tell me which city I should visit if I'm a foodie. Tell me why as well.")
=> 
#<SearchAnalysis:0x00007fbc9744f470
...
irb(main):005> search_analysis.run_search
  TRANSACTION (0.1ms)  begin transaction
  SearchAnalysis Create (0.4ms)  INSERT INTO "search_analyses" ("query", "engine", "prompt", "search_results", "chat_response", "status", "created_at", "updated_at") VALUES (?, ?, ?, ?, ?, ?, ?, ?)  [["query", "italian city with best food"], ["engine", 0], ["prompt", "Based solely on these search results, where should I go in Italy for a delicious vacation?"], ["search_results", "Title: 9 Best Food Cities in Italy\nLink: https://www.celebritycruises.com/blog/best-food-cities-in-italy\nSnippet: 9 Best Food Cities in Italy · Bologna · Palermo · Rome · Florence · Cagliari · Sorrento · Genoa · Parma. Street view of Parma. Parma. Often ...\n\n...
  TRANSACTION (3.4ms)  commit transaction
=> true

Next, we'll create a run_chat method. The logic will effectively be identical to the run_search method.

# app/models/search_analysis.rb

  def run_chat
    # We only run the chat if the status is "pending_chat"
    return unless self.status == "pending_chat"
    
    chat_response = OpenAIServices::ChatService.new(self).call
    
    if chat_response.success?
      self.chat_response = chat_response.results
      self.status = "complete"
    else
      self.chat_response = "Failed due to error: #{chat_response.error.to_s}"
      self.status = "failed"
    end
    self.save
  end

And now the test.

user@host:~/search-results-ai-analyzer$ rails c
Loading development environment (Rails 7.0.8)
irb(main):001> search_analysis = SearchAnalysis.last
  SearchAnalysis Load (0.2ms)  SELECT "search_analyses".* FROM "search_analyses" ORDER BY "search_analyses"."id" DESC LIMIT ?  [["LIMIT", 1]]
=> 
#<SearchAnalysis:0x00007f9d61599d30
...
irb(main):002> search_analysis.run_chat
  TRANSACTION (0.1ms)  begin transaction
  SearchAnalysis Update (0.4ms)  UPDATE "search_analyses" SET "chat_response" = ?, "updated_at" = ? WHERE "search_analyses"."id" = ?  [["chat_response", "Based on the search results, the city you should visit if you're a foodie is Bologna. Bologna consistently appears in multiple search results and is mentioned in titles such as \"9 Best Food Cities in Italy,\" \"10 of the best Italian cities for food,\" and \"Our Favorite Food Cities In Italy.\" Bologna is known for its rich culinary traditions, including dishes like tortellini, ragu, and mortadella. It is also praised for its local markets, food festivals, and vibrant food scene. Therefore, if you want to experience authentic Italian cuisine and immerse yourself in a city with a strong food culture, Bologna is the ideal choice."], ["updated_at", "2024-01-24 12:14:40.214658"], ["id", 6]]
  TRANSACTION (2.8ms)  commit transaction
=> true

Beautiful. We got a helpful response from ChatGPT telling us we should go to Bologna for our Italian foodie vacation.

Thanks to our scaffolding, we know our views will already display all this information to the user. We also already tested the create action of our controller.

We could call the search and chat services from our controller, but I think a cleaner way would be to handle all this in the model. We'll always save a search analysis request with valid data so we don't need the controller to worry about any of that. Instead, we can use our run_search and run_chat methods as callbacks that fire after the record is created.

# app/models/search_analysis.rb

class SearchAnalysis < ApplicationRecord
  enum :engine, { google: 0, bing: 1, duckduckgo: 2, yahoo: 3, yandex: 4, baidu: 5 }
  enum :status, { pending_search: 0, pending_chat: 1, complete: 2, failed: 3 }
  
  validates :query, presence: true
  validates :engine, presence: true
  validates :prompt, presence: true

  after_create :run_search, :run_chat
  
...
end

One potential concern here would be the risk of the search failing and then our chat would also fail (or not be useful) because we wouldn't have any search results to send ChatGPT. Fortunately, we already handled that when we set it so run_chat would fully execute only if the status was pending_chat which is only set if the run_search method was successful.

Now we can try requesting a search analysis through our form. Rails will automatically fire our methods to call SerpApi and ChatGPT. The form submission will take a few seconds to process because it has to make two API calls. With a little patience we finally see that it all works.

Rails renders the search results and chat response by default without any newlines so it's worth adding simple_format to these.

# app/views/_search_analysis.html.erb

  <p class="my-5">
    <strong class="block font-medium mb-1">Search results:</strong>
    <%= simple_format(search_analysis.search_results) %>
  </p>

  <p class="my-5">
    <strong class="block font-medium mb-1">Chat response:</strong>
    <%= simple_format(search_analysis.chat_response) %>
  </p>

It's rather ugly, but everything is now working. Perhaps we can pass this off to some designers to pretty up.

Areas to Improve

What we have is a good starting point, but it could be a lot better. Some possible improvements:

* Process API calls in a background job.
* Display user friendly error messages in case calls to SerpApi or OpenAI fail.
* Pretty up the UI.
* Allows users to store prompts for future use.
* Add support for all the other search engines SerpApi supports like Google News, Finance, Maps, Jobs, etc.
* Allow users to send additional chat messages to ChatGPT and present this in a chatbox.
* Re-try failed search analyses.

Source Code

The full source code of this application is available on Github.

Chatbase is a tool to simplify building a custom chatbot trained with whatever content you want. You can train it with content from files, raw text, or a website. Chatbase makes it trivial to train a chatbot and then embed it on your website.

For more technical users, Chatbase offers an API allowing you to create, train, and chat with Chatbase chatbots programmatically. This is invaluable for developers who don't want to maintain an integration with OpenAI, but are technical enough to integrate with Chatbase and customize the end user experience in ways that Chatbase doesn't support yet.

For example, you might want to customize the look and feel of a chatbot in ways Chatbase doesn't support. Or you might want to integrate Chatbase into other tools your organization uses such as support or internal company chat.

In this article, we'll go over what the API can do and cover some of its limitations. You'll even be able to chat with this very article.

API Access

The API is restricted to paying users of Chatbase. The lowest paid membership is $19/month and scale up from there offering more chatbots, messages/month, and characters/chatbot. Check out their pricing page for more information.

Once you've signed up for a subscription, navigate to your "Account" page and scroll to the bottom to create a new API key.

I suggest copying your API key and pasting it in your terminal as an environment variable to most easily follow along with this guide.

export CHATBASE_API_KEY=your_api_key_here

Otherwise, replace where I use $CHATBASE_API_KEY with your API key in the sample API calls I include below.

Crawl a Website

Chatbase has an API to crawl a website and get all the site's URLs for you. This is a cool feature that they didn't really need to offer, but it's certainly nice to have. Using the response from this endpoint, you can drop in the list of links as source training data in the create chatbot call we'll look at next.

Note that for big websites the crawler may crash and not collect everything. In this case, you're better off creating your own crawler or referring to OpenAI's tutorial on creating a website Q&A that includes a website crawler and scraper.

You can also slim down the list of URLs included by passing a source URL to the API that is targeted on a particular path. For example, example.com/developers to only get URLs within the /developers part of a website.

This API is available through /api/v1/fetch-links and accepts one URL parameter of sourceURL for the URL of the website you want to scrape.

curl 'https://www.chatbase.co/api/v1/fetch-links?sourceURL=https://www.apimaestro.com' \
    --request "GET" \
    -H "Authorization: Bearer $CHATBASE_API_KEY"

The response body will contain an array called fetchedLinks with objects containing a url and size. The size field contains the number of characters present on the page. This is important to keep in mind because Chatbase limits how many characters you can train each chatbot on. The base $19/month plan is limited to 2 million characters per chatbot. The limit scales up along with the more premium plans. Refer to the Chatbase pricing page for more details.

The response body will also contain a stoppingReason field indicating why the crawler may have stopped. According to Chatbase's docs, this most commonly happens when the crawler takes more than 60 seconds. The crawler will stop if it exceeds 60 seconds and will return only what it found in those 60 seconds. Their API docs don't currently indicate what else you might see in this field.

{
  "fetchedLinks": [
    {
      "url": "https://www.apimaestro.com/",
      "size": 4048
    },
    {
      "url": "https://www.apimaestro.com/about/",
      "size": 1405
    },
    {
      "url": "https://www.apimaestro.com/what-is-openapi/",
      "size": 6729
    },
    {
      "url": "https://www.apimaestro.com/getting-started-with-pocket-bitcoins-api/",
      "size": 15527
    },
    {
      "url": "https://www.apimaestro.com/creating-an-openapi-specification-the-hard-way/",
      "size": 18034
    }
  ],
  "stoppingReason": ""
}

Create a Chatbot

Now we'll create our first chatbot through the API. This is done making a POST request to the api/v1/create-chatbot endpoint.

The request body should contain a chatbotName and some source materials to train the bot. The source materials can be sent in 2 different fields:

1. urlsToScrape to scrape text from web pages. This field should contain an array of URLs.
2. sourceText to use raw text to train the bot. As of my testing today, this field required at least 100 characters. The maximum number of characters depends on your plan.

You must provide at least one of the above fields. You can include both if you have a mix of URLs and raw text.

curl https://www.chatbase.co/api/v1/create-chatbot \
  -H 'Content-Type: application/json' \
  -H "Authorization: Bearer $CHATBASE_API_KEY" \
  -d '{"sourceText": "risus sed vulputate odio ut enim blandit volutpat maecenas volutpat blandit aliquam etiam erat velit scelerisque in dictum non consectetur a erat nam at lectus urna duis convallis convallis tellus id interdum velit laoreet id donec ultrices tincidunt arcu non sodales neque sodales ut etiam sit amet nisl purus in", "urlsToScrape": ["https://apimaestro.com/getting-started-with-pocket-bitcoins-api/", "https://apimaestro.com/what-is-openapi/"], "chatbotName": "API Maestro"}'

The response body will contain a chatbotIdfield with the ID of your chatbot.

{"chatbotId": "exampleId-123"}

You'll need this ID for all API calls chatting with or modifying your chatbot. If you fail to save the ID from the API response, you can always find it again in the Chatbase UI under your chatbot's settings page.

Possible Errors Creating a Chatbot

The Chatbase API docs don't explain anything about possible errors, but I was able to find a few playing around with this endpoint.

• If you don't pass any training material, you'll get a 500 error with message "Cannot read properties of undefined (reading 'replace')"
• If you're already at your limit for how many chatbots your plan allows, you'll get a 403 error with message "Limit to 5 chatbots for your plan. Delete existing chatbots or upgrade plan to increase the limit."
• If you pass too few characters as training material, you'll get a 403 error with message "Only 51 characters were extracted. undefined"
• If you pass too many characters as training material, you'll get a 403 with message "Exceeded character limit for your plan of 2000000 characters"

Updating a Chatbot

The Chatbase API also has an endpoint to update a chatbot via POST /api/v1/update-chatbot-data. Currently, this endpoint only allows you to change the name of your chatbot and replace the training material. Whatever material you include in the update call will get rid of all the past training material you gave your bot

It's important to keep this replace function in mind because it means you must store on your side a list of all URLs and text you gave your chatbot and pass them again through this endpoint to ensure you don't tell your chatbot to forget anything. More on this in the challenges and limitations section of this guide.

This endpoint takes in all the same fields as the create call except you must also pass in a chatbotId field.

curl https://www.chatbase.co/api/v1/update-chatbot-data \
  -H 'Content-Type: application/json' \
  -H "Authorization: Bearer $CHATBASE_API_KEY" \
  -d '{"chatbotId": "exampleId-123", "sourceText": "Chatbase offers an API allowing you to create, train, and chat with Chatbase chatbots programmatically. This is invaluable for developers who don't want to maintain an integration with OpenAI, but are technical enough to integrate with Chatbase and customize the end user experience in ways that Chatbase doesn't support yet.", "chatbotName": "API Maestro new name"}'

The response body is the same as creating a chatbot. The possible errors are also the same from what I can tell.

Chatting with Your Chatbot

Finally, the moment you've been waiting for; chatting with your Chatbase chatbot. This is done through the POST /api/v1/chat endpoint.

The request body should contain the following:

• messages which is an array containing each message in the chat. A message object contains a role and content. Supports up to a maximum of 7 messages.
  • role can be assistant to refer to the chatbot and user to refer the human user
  • content is the message content from either the chatbot or human user
• stream which is a boolean where if true will stream the chatbot's response to you word by word as raw text. If false, it will wait until the full response is generated before sending you the response as JSON.
• chatbotId which is a string of your chatbot's ID

curl https://www.chatbase.co/api/v1/chat \
  -H "Authorization: Bearer $CHATBASE_API_KEY" \
  -d '{"messages":[{"content":"How can I help you?","role":"assistant"},{"content":"What is API Maestro?","role":"user"}],"chatId":"exampleId-123","stream":true}'

The response body, as mentioned previously, will be raw text with the chatbot's response if you set stream as true. If you put stream as false, you'll get the following JSON body back.

{
    "text": "API Maestro helps API businesses grow by creating delightful developer experiences and documentation."
}

To continue chatting with the same context, you append the chatbot's last message to the messages object and send another chat call. You can continue doing this until you have 7 messages in the messages object according to their docs; although in practice I found I could go beyond 7 messages. This will probably be patched up eventually.

curl https://www.chatbase.co/api/v1/chat \
  -H "Authorization: Bearer $CHATBASE_API_KEY" \
  -d '{"messages":[{"content":"How can I help you?","role":"assistant"},{"content":"What is API Maestro?","role":"user"},{"content":"API Maestro helps API businesses grow by creating delightful developer experiences and documentation.","role":"assistant"},{"content":"Who runs this website?","role":"user"}],"chatId":"exampleId-123","stream":true}'

Challenges and Limitations

The Chatbase API has the basics down well, but it's very apparent the API is lacking in features compared to the UI experience.

For example, you can't set or modify anything for your chatbot except the name and training material through the API whereas the UI allows you to customize the prompt and GPT version.

You also can't see what training materials you've trained your chatbot with. Ideally, there would be a GET chatbot or GET chatbot training materials endpoint that would return the URLs and raw text you've used to train a bot.

Frustratingly, if you create a chatbot through the API, you also can't look up what you've trained it with through the UI either. It's all a black box whereas chatbots created through the UI will show you in the UI what source materials it's been trained with.

This introduces challenges particularly when updating a chatbot's training materials through the API. Unless you saved the list of URLs and text you sent to Chatbase when you created the chatbot, you run the risk of accidentally deleting old training material when you update your chatbot with new material.

With the lack of a GET chatbot endpoint, you also can't check in on your limits to see if you're near your cap of characters or messages.

There's a lot of promise here, but the API needs some work before it's ready for prime time. Nevertheless, it's amazing what Chatbase's creator Yasser has done by himself in just 2 months. I'm sure great things are still to come for both the UI and API experience.

What Next?

1. Consider signing up for Chatbase.
2. Check out the official docs.
3. Follow the Chatbase founder's Twitter for product updates and follow how his business is growing.
4. Try chatting with this article:

OpenAPI documents are most commonly created as YAML files. You can also use JSON. The resulting file serves as your fundamental text that streamlines production of documentation and client libraries. This saves time and minimizes errors from manually creating reference docs and API wrappers.

There are numerous ways to create an OpenAPI spec. OpenAPI.tools is probably the most comprehensive list of tools to help you. Spotlight and Postman are two of the best UI tools to create your spec.

You can also, of course, do it the hard way. In this guide, we'll take inspiration from Zed Shaw over at Learn Code The Hard Way by writing an OpenAPI YAML file by hand. This is obviously more time-consuming and error prone, but sometimes if you want to understand something, doing it the hard, manual way is the best way. Once you understand it at a deeper level, you become a better conductor of the tools that abstract some of the inner workings.

What We're Building

We need an idea for a simple API before we can get started. Let's imagine a simple, CRUD dictionary API that allows user to look up, create, and modify definitions of words. We'll need the following paths:

• POST /definitions
• GET /definitions/{word}
• PUT /definitions/{word}
• DELETE /definitions/{word}

The API will use JSON in request and response bodies.

For security, we'll use Basic Authentication for all endpoints except GET /definitions.

Initial Setup

Create a file called definitions.yaml. We'll start by defining the version of OpenAPI we're using (3.0.0) and same basic description info of our API. The info.version field refers to the version of our API.

# definitions.yaml
openapi: 3.0.0
info:
  title: Dictionary REST API
  description: An API to look up definitions of words.
  version: 1.0.0

Next we'll add our API's base URL under servers. You can add multiple servers if you want to support production and development instances. We'll stick to one for now.

We'll also define our security scheme of Basic Auth here as well under components. Doing this defines a security scheme that we can then apply to each endpoint that needs this security scheme.

# definitions.yaml
openapi: 3.0.0
info:
  title: Dictionary REST API
  description: An API to look up definitions of words.
  version: 1.0.0
servers:
  - url: https://sample-api.apimaestro.com
components:
  securitySchemes:
    basicAuth:
      type: http
      scheme: basic

If we wanted to set Basic Auth as our global security scheme and apply it to all endpoints, we could do that under security like the example below. We'll leave this out of our spec to keep GET /definitions unprotected.

components:
  securitySchemes:
    basicAuth:
      type: http
      scheme: basic
security:
  - basicAuth: []

Schemas

Data schemas are defined under components. You can set a title, description, and type. This content will appear in generated reference documentation.

The type field can be a string, number, integer, boolean, array, or object. You can also define multiple types for a single field.

We'll create a definition schema of type object.

# definitions.yaml
openapi: 3.0.0
...

components:
  securitySchemes:
    basicAuth:
      type: http
      scheme: basic
  schemas:
    Definition:
    	title: Definition
        description: A definition is a statement that explains the meaning of a word or phrase.
        type: object

The object's schema will be referenced when we're setting request and response bodies for our endpoints. We can define the fields that belong to our object under properties. In the example below, I'll define the following:

• id as a read-only integer
• word as a string
• created_at and updated_at as read-only strings in the date-time format
• meanings as an array of objects

The meanings array will allow us to handle words with multiple definitions. We could split out meanings as its own schema model, but for now, we can define its schema inline with its parent object.

Our meanings object will have the following properties:

• id as a read-only integer
• part_of_speech as a string
• meaning as a string

# definitions.yaml
openapi: 3.0.0
...

components:
  securitySchemes:
    basicAuth:
      type: http
      scheme: basic
  schemas:
    Definition:
      title: Definition
      description: A definition is a statement that explains the meaning of a word or phrase.
      type: object
      properties:
        id:
          type: integer
          example: 40125
          readOnly: true
        word:
          type: string
          example: tagliatelle
        created_at:
          type: string
          format: date-time
          example: 2022-12-06T04:55:36.887Z
          readOnly: true
        updated_at:
          type: string
          format: date-time
          example: 2022-12-06T04:55:36.887Z
          readOnly: true
        meanings:
          type: array
          items:
            type: object
            properties:
              id:
                type: integer
                example: 333
                readOnly: true
              part_of_speech:
                type: string
                example: noun
              meaning:
                type: string
                example: A type of pasta shaped into long, thin, flat strips

All set here. Now we can start writing our endpoints.

Endpoints

All of our endpoints will go under the paths section. For each path, we can set multiple methods. Our simple CRUD API has only 2 paths; /definitions for POST and /definitions{word} for GET, PUT, and DELETE.

# definitions.yaml
openapi: 3.0.0
...

paths:
  /definitions/{word}:
    get:
    put:
    delete:
  /definitions:
    post:

Get a Definition

First, we'll add our word look up endpoint.

Similar to schemas, you can add summary and description content here that will appear in documentation.

I also recommend setting an operationId. This is used to set the method name in auto-generated client libraries. We'll set operationId: get_definition. We can also use camel case operationId: getDefinition.

OpenAPI is smart enough to accept either snake or camel case and convert appropriately. In Python or Ruby, the method would remain in snake case, api.get_definition(word). In camel case languages like Javascript, the method would be api.getDefinition(word).

# definitions.yaml
openapi: 3.0.0
...

paths:
  /definitions/{word}:
    get:
      summary: Get a word's definition.
      operationId: get_definition
      description: > # We can write a multi-line description using the `<` character.
        The GET /definitions endpoint accepts a word and returns its meaning.
        The endpoint returns multiple meanings if available.

Our GET /definitions endpoint will accept one path parameter. This will be the word we're looking up. We define this under parameters.

# definitions.yaml
openapi: 3.0.0
...

paths:
  /definitions/{word}:
    get:
      summary: Get a word's definition.
      operationId: get_definition
      description: > # We can write a multi-line description using the `<` character.
        The GET /definitions endpoint accepts a word and returns its meaning.
        The endpoint returns multiple meanings if available.
      parameters:
        - name: word
          in: path
          description: Word to look up
          schema:
            type: string
          required: true
          example: tortellini

Next, we define how this endpoint responds under responses. We can define multiple responses depending on the response code.

We need to define the response description and content. We set the schema of the response body by referencing the definition schema we created previously.

paths:
  /definitions/{word}:
    get:
      summary: Get a word's definition.
      operationId: get_definition
      description: > # We can write a multi-line description using the `<` character.
        The GET /definitions endpoint accepts a word and returns its meaning.
        The endpoint returns multiple meanings if available.
      parameters:
        - name: word
          in: path
          description: Word to look up
          schema:
            type: string
          required: true
          example: tortellini
      responses:
        '200':
          description: OK
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/Definition'

Let's also add a 404 Not Found response.

paths:
  /definitions/{word}:
    get:
      summary: Gets a definition of a word.
      operationId: get_definition
      description: > # We can write a multi-line description using the `<` character.
        The GET /definitions endpoint accepts a word and returns its meaning.
        The endpoint returns multiple meanings if available.
      parameters:
        - name: word
          in: path
          description: Word to look up
          schema:
            type: string
          required: true
          example: tortellini
      responses:
        '200':
          description: OK
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/Definition'
        '404':
          description: No definition found for the specified word.

Create a Definition

The structure for creating a POST request doesn't require much more than our GET request. We start again with the basic description information.

  /definitions:
    post:
      summary: Create a word's definition
      operationId: create_definition
      description: >
        POST /definitions creates a definition of a word.
        Requests to create a duplicate definitions for a word will be rejected with a 409 response code.
        Requests must be authorized with Basic Auth.

Remember we want to protect this endpoint with Basic Auth to prevent anyone from creating definitions. We can do that by adding a security specification set to basicAuth.

  /definitions:
    post:
      summary: Create a word's definition
      operationId: create_definition
      description: >
        POST /definitions creates a definition of a word.
        Requests to create a duplicate definitions for a word will be rejected with a 409 response code.
        Requests must be authorized with Basic Auth.
      security:
        - basicAuth: []

This time we don't need to set any parameters. Instead, we'll accept a request body. Similar to the response body we set in the GET request, we can reference our schema definition in the same way to set the request body.

  /definitions:
    post:
      summary: Create a word's definition
      operationId: create_definition
      description: >
        POST /definitions creates a definition of a word. 
        Requests to create a duplicate definitions for a word will be rejected with a 409 response code. 
        Requests must be authorized with Basic Auth.
      security:
        - basicAuth: []
      requestBody:
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/Definition'
        required: true
      responses:

Finally, we'll add a few response codes and specify that successful creations respond with the full definition.

  /definitions:
    post:
      summary: Create a word's definition
      operationId: create_definition
      description: >
        POST /definitions creates a definition of a word.
        Requests to create a duplicate definitions for a word will be rejected with a 409 response code.
        Requests must be authorized with Basic Auth.
      security:
        - basicAuth: []
      requestBody:
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/Definition'
        required: true
      responses:
        '201':
          description: Word created successfully.
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/Definition'
        '400':
          description: Invalid definition data provided.
        '401':
          description: Unauthorized request.
        '409':
          description: Word already has a definition.
        '422':
          description: Unprocessable entity.

Update a Definition

Things get easy here because we don't need anything new. We can re-use what we set in the GET and POST requests to create our PUT request. We'll copy the POST request and then adjust the following:

1. Change the method.
2. Update labels and descriptions.
3. Copy the parameter specs from the GET request.
4. Update response codes.

Here's what the full PUT request should look like.

  /definitions/{word}:
    put:
      summary: Update a word's definition
      operationId: update_definition
      description: >
        PUT /definitions updates a definition of a word.
        Requests must be authorized with Basic Auth.
      security:
        - basicAuth: []
      parameters:
        - name: word
          in: path
          description: Word to look up
          schema:
            type: string
          required: true
          example: tortellini
      requestBody:
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/Definition'
        required: true
      responses:
        '204':
          description: Word updated successfully.
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/Definition'
        '400':
          description: Invalid definition data provided.
        '401':
          description: Unauthorized request.
        '404':
          description: Word not found.
        '422':
          description: Unprocessable entity.

Delete a Definition

Setting the DELETE request is even more trivial than the PUT request. In this case, we copy the GET request and update the following:

1. Change the method.
2. Update labels and descriptions.
3. Add Basic Auth.
4. Update response codes.

Here's what we're left with.

  /definitions/{word}:
    delete:
      summary: Delete a word's definition
      operationId: delete_definition
      description: >
        DELETE /definitions/{word} deletes a definition of a word.
        Requests must be authorized with Basic Auth.
      security:
        - basicAuth: []
      parameters:
        - name: word
          in: path
          description: Word to look up
          schema:
            type: string
          required: true
          example: tortellini
      responses:
        '204'
          description: Word deleted successfully.
        '401':
          description: Unauthorized request.
        '404':
          description: Word not found.

Conclusion

Congratulations. You've created an OpenAPI document the hard way. Of course, you're not going to want to do this a regular exercise. Now that you have a foundation of how OpenAPI works, you're better prepared to use tools like Stoplight to simplify generating an OpenAPI file while also benefitting from their linter.

Finally, here's the full spec we created in a single code block:

openapi: 3.0.0
info:
  title: Dictionary REST API
  description: An API to look up definitions of words.
  version: 1.0.0
servers:
  - url: 'https://sample-api.apimaestro.com'
components:
  securitySchemes:
    basicAuth:
      type: http
      scheme: basic
  schemas:
    Definition:
      title: Definition
      description: A definition is a statement that explains the meaning of a word or phrase.
      type: object
      properties:
        id:
          type: integer
          example: 40125
          readOnly: true
        word:
          type: string
          example: tagliatelle
        created_at:
          type: string
          format: date-time
          example: '2022-12-06T04:55:36.887Z'
          readOnly: true
        updated_at:
          type: string
          format: date-time
          example: '2022-12-06T04:55:36.887Z'
          readOnly: true
        meanings:
          type: array
          items:
            type: object
            properties:
              id:
                type: integer
                example: 333
                readOnly: true
              part_of_speech:
                type: string
                example: noun
              meaning:
                type: string
                example: 'A type of pasta shaped into long, thin, flat strips'
paths:
  /definitions/{word}:
    get:
      summary: Get a word's definition.
      operationId: get_definition
      description: |
        GET /definitions/{word} accepts a word and returns its meaning. The endpoint returns multiple meanings if available.
      parameters:
        - name: word
          in: path
          description: Word to look up.
          schema:
            type: string
          required: true
          example: tortellini
      responses:
        '200':
          description: OK
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/Definition'
        '404':
          description: No definition found for the specified word.
    put:
      summary: Update a word's definition
      operationId: update_definition
      description: |
        PUT /definitions updates a definition of a word. Requests must be authorized with Basic Auth.
      security:
        - basicAuth: []
      parameters:
        - name: word
          in: path
          description: Word to look up
          schema:
            type: string
          required: true
          example: tortellini
      requestBody:
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/Definition'
        required: true
      responses:
        '204':
          description: Word updated successfully.
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/Definition'
        '400':
          description: Invalid definition data provided.
        '401':
          description: Unauthorized request.
        '404':
          description: Word not found.
        '422':
          description: Unprocessable entity.
    delete:
      summary: Delete a word's definition
      operationId: delete_definition
      description: |
        DELETE /definitions/{word} deletes a definition of a word. Requests must be authorized with Basic Auth.
      security:
        - basicAuth: []
      parameters:
        - name: word
          in: path
          description: Word to look up
          schema:
            type: string
          required: true
          example: tortellini
      responses:
        '204':
          description: Word deleted successfully.
        '401':
          description: Unauthorized request.
        '404':
          description: Word not found.
  /definitions:
    post:
      summary: Create a word's definition
      operationId: create_definition
      description: >
        POST /definitions creates a definition of a word. 
        Requests to create a duplicate definitions for a word will be rejected with a 409 response code. 
        Requests must be authorized with Basic Auth.
      security:
        - basicAuth: []
      requestBody:
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/Definition'
        required: true
      responses:
        '201':
          description: Word created successfully.
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/Definition'
        '400':
          description: Invalid definition data provided.
        '401':
          description: Unauthorized request.
        '409':
          description: Word already has a definition.
        '422':
          description: Unprocessable entity.
    parameters: []

If you've never used Pocket before, I recommend trying a purchase through their site. The purchase flow through the API is similar to the UI flow and it's helpful to have a foundation of how their product works.

Access

Pocket requires developers authenticate their API calls with a client ID so your first step is to get a client ID. Head over to Pocket's Contact page and explain why you want API access.

The team will reply back soon with your client ID or additional questions about your request. Assuming they approve your request, they will send credentials and configure a temporary webhooks receiver. I'll explain more about webhooks later in this guide.

Before we move on, make sure your client ID is working with your first request to their API. Their API requires an 'Authorization' header with the value containing the prefix 'client_id' followed by your client ID.

curl "https://api.pocketbitcoin.com" \
  -H "Authorization: client_id YOUR_CLIENT_ID"

I recommend adding your client ID to an environment variable which will allow you to copy and paste the curl commands in this guide without having to manually add your client ID each time. Run the following in your terminal, adding your actual client ID where YOUR_CLIENT_ID is.

export POCKET_CLIENT_ID=YOUR_CLIENT_ID

Test your environment variable.

echo $POCKET_CLIENT_ID

If your terminal responds with your client ID, you can move on to testing an API call.

curl "https://api.pocketbitcoin.com" \
    -H "Authorization: client_id "$POCKET_CLIENT_ID""

If you get a 401 Unauthorized error, double check your environment variable was set correctly. Carry on if you can't figure it out and add the client ID manually.

Resources

The API has three key resources to understand.

1. Challenge: A randomly generated token used in a signed message confirming ownership over a Bitcoin address. Challenges can be only used once and expire after 24 hours. These can be created and retrieved through the API.
2. Order: A set of payment and payout instructions to be used in a Bitcoin and fiat exchange. Orders can be used indefinitely for multiple purchases. These can be created, updated (with limitations), and retrieved through the API.
3. Exchange: A swap event of Bitcoin and fiat currency. Exchanges are created automatically when a user follows the payment instructions in an Order. Thus, these can only be retrieved through the API.  

High Level Flow

Hopefully you had a chance to try a purchase on their site. Here's a high level overview of how purchases work through the API. Some details are left out to avoid too much complexity too soon.

1. App creates a challenge token with Pocket Bitcoin.
2. App passes challenge token to User.
3. User provides the following to App:
• Signed message with challenge token
• User's Bitcoin address
• User's IBAN
• User's currency; EUR or CHF
4. App sends the above details to Pocket Bitcoin to create an order.
5. Pocket Bitcoin responds with an order containing the following:
• Pocket Bitcoin's IBAN
• Payment reference number
6. App provides order details to User and requests them to make payment to Pocket Bitcoin.
7. User sends payment to Pocket Bitcoin and includes the payment reference number in the payment description.
8. Pocket Bitcoin creates an exchange once it sees the payment and begins sending status webhooks to App.
9. Pocket Bitcoin sends Bitcoin to User at the end of the day.

Creating a Challenge

The first call we'll make is a POST to /v1/challenges to generate a random token. This POST request does not take any data in its request body. You can leave it out of your request if you want.

curl -X POST "https://api.pocketbitcoin.com/v1/challenges" \
    -H "Authorization: client_id "$POCKET_CLIENT_ID"" \
    -H "Content-Type: application/json" \
    --data-raw '{}'

We should receive the following response:

{
    "id": "d7a2907b-3921-4bdc-9922-67e0bafa8187",
    "token": "qf3GFbmi",
    "expires_on": "2022-12-07T15:31:53.259Z",
    "completed_on": null
}

Great. We have our token and now we can create a signed message.

Signing Message

Before we can create an order, we need to create a signed message using the token from the previous step. The message can say anything, but it must contain a valid token. Pocket's docs provide this sample message that will work for our purposes too: "I confirm my bitcoin wallet. [qf3GFbmi]"

For testing purposes, I recommend setting up a wallet in Sparrow or Electrum. These wallets make it simple to sign a message with a simple UI tool. See my guide on signing messages in Sparrow if you need help. My guide assumes you're using a hardware wallet as a signing device, but if you're just testing, you can create a hot wallet through Sparrow or Electrum to sign messages.

Navigate to the 'Sign/Verify Message' option under 'Tools' and paste the Bitcoin address you want to receive to and your messsage.

Hit the 'Sign' button and the 'Signature' field will be populated with the signature we'll need to create an order. Keep this window open. We'll need the address, message, and signature in the next step.

Creating an Order

In addition to the 3 items mentioned above, you'll also need to decide the following:

1. A fee rate ranging 1.2% to 4.5% in float form (e.g. 0.015 for 1.5%). I've set the fee to 1.2% in the sample below.
2. Currency you'll pay with; EUR or CHF only.
3. IBAN for the bank account you'll pay Pocket with.

Replace the values in the sample below with yours. All fields you need to add are prefixed with YOUR_....

curl -X POST "https://api.pocketbitcoin.com/v1/orders" \
    -H "Authorization: client_id "$POCKET_CLIENT_ID"" \
    -H "Content-Type: application/json" \
    --data-raw '{
        "active": true,
        "fee_rate": 0.012,
        "payment_method": {
            "currency": "YOUR_CURRENCY",
            "debitor_iban": "YOUR_IBAN"
        },
        "payout_method": {
            "bitcoin_address": "YOUR_BITCOIN_ADDRESS",
            "message": "I confirm my bitcoin wallet. [YOUR_TOKEN]",
            "signature": "YOUR_SIGNATURE"
        }
    }'

If you become a Pocket affiliate partner, you would also include an affiliate_id field to track the customers you send their way.

A successful response will return 201 Created and a response body containing the details to make payment.

{
  "id": "64decab6-4129-4fe7-9f6e-1db68283f5ce",
  "active": true,
  "created_on": "2022-12-07T15:32:59.211Z",
  "fee_rate": 0.012,
  "payment_method": {
    "currency": "eur",
    "debitor_iban": "DE75512108001245126199",
    "creditor_reference": "RF18GW8K79",
    "creditor_iban": "CH9400778214768302002",
    "creditor_bank_name": "Luzerner Kantonalbank",
    "creditor_bank_street": "Pilatusstrasse 12",
    "creditor_bank_postal_code": "6002",
    "creditor_bank_town": "Luzern",
    "creditor_bank_country": "CH",
    "creditor_bank_bic": "LUKBCH2260A",
    "creditor_name": "Pocket App",
    "creditor_street": "Industriestrasse 33",
    "creditor_postal_code": "5242",
    "creditor_town": "Lupfig",
    "creditor_country": "CH",
    "swiss_qr_bill_payload": null
  },
  "payout_method": {
    "bitcoin_address": "bc1q5a7uxjq6z5l2wguzu77rzmzk6tyx24yvq5r4fj",
    "message": "I confirm my bitcoin wallet. [qf3GFbmi]",
    "signature": "IDoGQ1yyj257K5c/eroZcn1zWg1QULuqo/ALqjkRFEfZE8CA8ANp0+IZ2KpQz2PUhn+ryUy4JWI7J+VbYz/aXOM="
  }
}

You can look up this order at any time through the get order endpoint. You can also update the order's affiliate ID, fee rate, and active status through the patch endpoint. See Pocket's docs for examples of those.

Making Payment

Payment must be done from your banking app to Pocket's bank account provided in the order response body. You can theoretically send any amount but it's advisable to send a small amount first. For a large amount, Pocket suggests contacting them first.

It's crucial you include the payment reference number in your payment description to Pocket. This is found in the creditor_reference field. RF18GW8K79 in the sample response above. This field is used by Pocket to link your fiat payment to the order and send the payout to the right Bitcoin address.

Exchanges

Webhooks

Pocket uses webhooks to keep developers up to date on exchanges. Listening for webhooks is crucial to any integration because there is no other way to get updates on exchanges. There is an endpoint to fetch exchanges, but if you're not listening for exchange webhooks, you won't know the exchange ID needed to look it up. Once you have the ID, you can of course query the exchange endpoint. Hopefully Pocket will soon add an endpoint to search for exchanges with query params such as date ranges or by order ID.

Pocket will provision a temporary webhooks endpoint for you when you first get API credentials. See the webhooks.site link they sent you to watch for webhooks.

Contact the Pocket team once you have your own webhook endpoint configured to rotate where you receive notifications.

Pocket will send a signature header along with every webhook. If you're building a real production application, use this to verify the webhook really came from Pocket. More information on this is available here.

Executed Exchange

Pocket will send an executed exchange webhook once they receive your fiat payment. See the rate field for the Bitcoin price you got.

{
  "id": "a2d75998-5ee5-4501-a4cf-4e3788732b7a",
  "order_id": "64decab6-4129-4fe7-9f6e-1db68283f5ce",
  "fee_rate": 0.012,
  "pair": "btc/eur",
  "type": "buy",
  "cost": 100.00,
  "executed_on": "2022-12-07T15:32:59.211Z",
  "amount": 0.00494000,
  "rate": 20000.00,
  "fee": 1.20,
  "action": null,
  "reason": null,
  "payout": null
}

Settled Exchange

Pocket will send a settled exchange webhook when they have paid out to your Bitcoin address. Exchanges settle at the end of the day in which Pocket received your fiat payment. This is usually around 10pm Central European Time.

The settled exchange webhook will be the same as above except the payout field will be populated with details on the payout transaction.

{
  "id": "a2d75998-5ee5-4501-a4cf-4e3788732b7a",
  "order_id": "64decab6-4129-4fe7-9f6e-1db68283f5ce",
  "fee_rate": 0.012,
  "pair": "btc/eur",
  "type": "buy",
  "cost": 100.00,
  "executed_on": "2022-12-07T15:32:59.211Z",
  "amount": 0.00494000,
  "rate": 20000.00,
  "fee": 1.20,
  "action": null,
  "reason": null,
  "payout": {
    "txid": "c01c7a0fd270aa3876119098c0e2d51687a795efab99787e214d8a93ab9f8342",
    "outpoint": 1,
    "bitcoin_address": "bc1q5a7uxjq6z5l2wguzu77rzmzk6tyx24yvq5r4fj",
    "derivation_path": null,
    "block_height": null,
    "confirmed": false,
    "fee": 0.00000190,
    "amount": 0.00049381
  }
}

Interrupted Exchange

An interrupted exchange means the exchange is blocked and you must take a particular action in order to unblock it. For example, if you try to buy a very large amount of Bitcoin, you may trigger a request for identification. See the action field.

{
  "event": "exchange.interrupted",
  "payload": {
    "exchange_id": "a2d75998-5ee5-4501-a4cf-4e3788732b7a",
    "order_id": "64decab6-4129-4fe7-9f6e-1db68283f5ce",
    "fee_rate": 0.012,
    "pair": "btc/eur",
    "type": "buy",
    "cost": 100.00,
    "executed_on": null,
    "amount": null,
    "rate": null,
    "fee": null,
    "action": {
      "code": "identification_required",
      "identification_id": "7bf967b5-3d34-4cd6-936c-6631811401d2"
    },
    "reason": null,
    "payout": null
  }
}

Refunded Exchange

This one is fairly simple. You'll get this notification if a fiat payment had to be refunded. There are several potential refund reasons. The example below shows a refund due to threshold_exceeded in the reason field. The full list is available in Pocket's documentation.

{
  "event": "exchange.refunded",
  "payload": {
    "exchange_id": "a2d75998-5ee5-4501-a4cf-4e3788732b7a",
    "order_id": "64decab6-4129-4fe7-9f6e-1db68283f5ce",
    "fee_rate": 0.012,
    "pair": "btc/eur",
    "type": "buy",
    "cost": 100.00,
    "executed_on": null,
    "amount": null,
    "rate": null,
    "fee": null,
    "action": null,
    "reason": {
      "code": "threshold_exceeded"
    },
    "payout": null
  }
}

Fetching an Exchange

As mentioned previously, you can retrieve exchanges through the API if you have the exchange's ID.

curl -X GET "https://api.pocketbitcoin.com/v1/exchanges/a2d75998-5ee5-4501-a4cf-4e3788732b7a" \
    -H "Authorization: client_id "$POCKET_CLIENT_ID""

The response won't explicitly tell you the status of the exchange. Based on what fields are populated, you can understand the state of it.

• Executed if the following fields are null: action, reason, and payout.
• Settled if payout is populated.
• Interrupted if action is populated.
• Refunded if reason is populated.

{
  "id": "a2d75998-5ee5-4501-a4cf-4e3788732b7a",
  "order_id": "64decab6-4129-4fe7-9f6e-1db68283f5ce",
  "fee_rate": 0.012,
  "pair": "btc/eur",
  "type": "buy",
  "cost": 100.00,
  "executed_on": "2022-12-07T15:32:59.211Z",
  "amount": 0.00494000,
  "rate": 20000.00,
  "fee": 1.20,
  "action": null,
  "reason": null,
  "payout": null
}

What Next?

That wraps up my guide on Pocket's API. Check out the following resources if you'd like to learn more or start playing around with their API.

1. Pocket's API Docs
2. Pocket's Postman Collection
3. My OpenAPI Config for Pocket (use this to auto-generate an API client library in any language you want)

OpenAPI is a standardized technical specification to make describing the capabilities of the API unambiguous for both humans and machines. Why this is desirable requires a brief tangent on APIs and the problem of asymmetric information.

APIs are like contracts. The terms of the contract must be clear and unambiguous to all interested parties. An API provider creates these terms that dictate how a consumer interacts with the API. If the consumer follows the terms correctly, they'll get what they want. If they don't, someone will end up disappointed.

The trouble for the API consumer is source code is generally not accessible to them. Even if it was, it wouldn't necessarily be easy to understand. Instead, they depend on documentation or client libraries that are more like proxy contracts of the real contract. It's like asking your lawyer for a contract and their assistant copies it out by hand for you.

This problem is exacerbated as more parties become involved in managing and using the API; both internally and externally.

APIs encounter numerous problems trying to keep everyone on the same page.

• The engineer didn't implement the code and tests according to the design doc.
• The technical writer overlooked something while preparing documentation.
• The API consumer didn't integrate with the API correctly.

The poor support engineer is now stuck trying to understand if there's a bug in the code, the documentation, or the client/partner's code.

How the OpenAPI Specification Helps

The OpenAPI Specification (formerly Swagger) aims to solve all these problems and more by offering a standard, unambiguous way to describe REST APIs in a machine-readable YAML or JSON file.

This file is called an OpenAPI document and acts as your API's contract and source of truth for all interested parties to streamline development, documentation, and reduce errors. Using this document, you can do all of the following without having to write any code:

1. Auto-generate documentation in multiple file formats.
2. Auto-generate SDKs/client libraries in any programming language.
3. Create mock servers.
4. Generate and run test suites of your API.

Focusing your development efforts on an OpenAPI document encourages a design-first approach. OpenAPI documents allow you to design your API in a simple way right from the start without getting lost in code or a lengthy design doc no one wants to read.

Instead, you write a bit of YAML or JSON to define your API endpoints, schemas, requests, and responses in mere minutes. Then use that source file to generate mock servers and documentation to review with all your API stakeholders.

As someone who has read a lot of confusing API design docs, this is a game changer. There's no comparison to being able to actually see what the documentation will look like so early on and to make mock requests during the early design phase.

Since OpenAPI documents are not, strictly speaking, code, it also enables less technical stakeholders to more fully contribute as reviewers and to propose changes.

OpenAPI Ecosystem & Tools

The OpenAPI ecosystem is very active and supported by many of the largest companies in the space through the OpenAPI Initiative. It is a Linux Foundation Collaborative Project and strong proponent of open source.

The developer community around OpenAPI is also excellent. The number of tools available to support usage of OpenAPI is truly staggering.

OpenAPI.Tools has the most comprehensive list of OpenAPI tools as far as I can tell. They're an excellent repository to find tools such as

• Auto Generators: Convert your code into OpenAPI documents.
• Converters: Convert to and from OpenAPI and other API formats.
• Description Validators: Validate your OpenAPI document follows the OpenAPI Specification.
• GUI Editors: Visual editors to create OpenAPI documents.
• Mock Servers: Fake servers that respond to requests with examples set in your OpenAPI document.
• SDK Generators: Generate full client libraries in any language to make requests to your API.
• Text Editors: Get visual feedback while you write an OpenAPI document.

Where to Start with OpenAPI

OpenAPI can do so much it's not always clear where to start. I recommend starting by downloading the classic OpenAPI example document of a pet store API. Or if you already have a project in Postman, try converting it to OpenAPI with this Node package.

Then try generating HTML docs or a client library in your favorite programming language using the OpenAPI command line generator.

Otherwise, go through the long list of OpenAPI tools and see what catches your attention.

In this tutorial, I'll walk through how to add a simple Bitcoin donation button to your website in only five minutes. This implementation can also be used with minor tweaks to accept payments for freelance work or even selling digital or physical products. Recurring subscription payments won't easily fit this model.

We'll be using two services to implement our payment interface.

1. Voltage: Simple to use, hosted Bitcoin infrastructure with plug-and-play, open source software. Available as a 7 day free trial and then $0.012/hour which works out to about $8.76/month.
2. BTCPay: A self-hosted, open source cryptocurrency payment processor. It's secure, private, censorship-resistant and free.

Set Up Voltage

First, we need to create an account and node on Voltage. Once you've registered, you'll be presented with the following screen. Hit 'Nodes'.

And then 'Create Node'.

We'll piggyback off of Voltage's Bitcoin Core node allowing us to go straight to choosing BTCPay Server.

For the moment, BTCPay Server is only available in the US West region. Hopefully they'll add other regions soon. For now, confirm the cost looks acceptable to you and then carry on to deploy your BTCPay Server.

Give your store a name. Ignore the text about Lightning nodes unless you're running a Lightning node on Voltage.

Easy. Your server has been created and deployed. Store your credentials presented here, but note that the first thing we'll do once we're in the BTCPay dashboard is change your password. Proceed to login to continue.

Before we get to the fun part, let's change that default password. Navigate to the bottom left corner of the sidebar and click 'Account' and then 'Manage Account'.

Pop over to 'Password' and pick a long, secure password.

In the sidebar, click 'Bitcoin' under 'Wallets' to create or link an existing wallet. It's more secure to connect an existing wallet because creating a new wallet on BTCPay creates a hot wallet that can more easily be hacked. We'll be following the steps to connect an existing wallet.

Several import methods exist. In my opinion, entering an extended public key is the simplest if you're already using a hardware wallet with software that exposes the xpub/zpub easily such as Sparrow. This will route your funds directly to cold storage. Consider creating a new account in your wallet software with a descriptive name related to BTCPay or your website. Make sure you give BTCPay the xpub/zpub from the newly created account instead of your other accounts.

My second choice would be connecting a hardware wallet. This will require a bit of extra time in order to download and setup BTCPay's Vault software. Follow the steps for that here.

Paste your xpub/zpub if you're following that option.

Assuming you copied your xpub/zpub correctly, the next page will list your first 10 addresses calculated off that public key. Be sure to check this against your wallet software. This is very important. Badly entered public keys will cause Bitcoin to  go to addresses you don't control.

If you use Sparrow, check the 'Addresses' menu to compare that list to this one. Other wallet applications should have similar lists with your addresses.

If not, you can check the addresses from past transactions(must be the first 10 ever used). If it's a new wallet or account, use the receive functionality of your hardware wallet to generate an address and ensure it's in the list BTCPay generated.

Next, we'll create an app that will function as our payment receiver. In the sidebar, click 'New App'.

Keep the default 'App Type' of 'Point of Sale' and add a name for this app. This name doesn't get displayed to the user.

Enter a 'Display Title' which will be displayed to user and a description.

You can also adjust the fiat currency your store will use. It looks like a dropdown, but clicking the dropdown arrow, won't show all the options if USD is still populated. Clear the text and type in the 3 letter code for the currency you want if you're not using USD.

There will be several products auto-generated under 'Products'. You should delete all but one of these unless you really like tea or something.

Edit the product you didn't delete. Put 'Donate' or whatever you want as 'Title'. Set Price to 'Minimum' and '0'.

The remaining fields are optional. Do as you wish to spruce up your point of sale. Click 'Save' once you're done.

Leave everything under 'Appearance' as is and make sure 'Custom Payments' is disabled. Users can still input whatever payment amount they want the way we set it up.

Under 'Additional Options', you can optionally add a redirect URL after the user completes payment. For example, if you want to send them to a thank you page.

Once you're done with all these options, click 'Save' in the upper right corner. It's easy to miss.

Now, you can finally add this to your website.  The iframe option will be the easiest in my opinion. Copy the whole iframe HTML under 'Embed Point of Sale via iframe' and paste it into the appropriate page of your website. Make sure you've enabled raw HTML access for whatever content management system you're using.

If the store doesn't fit the iframe well(it didn't for me), you need to adjust the dimensions. In my case, I had to add height by updating the style field. See example below.

<iframe src='https://btcpay0.voltageapp.io/apps/XKrNPjtnLCq4kuKRpYgzAn4PAxo/pos' style='height:1000px;max-width: 100%; border: 0;'></iframe>

You can also link to the payment page using the payment button HTML BTCPay Server generates for you.

Or you can simply send people the link to your payment page if you don't want it on your own site. Grab the link from the iframe HTML after where it says 'src'.

Success! You can now accept donations/payments in Bitcoin. Give the flow a test with a small amount and make sure the payment gets routed to an address you control.

Voltage Billing

If you intend to keep using Voltage beyond the free trial or free credit(I received $10 free credit just for creating an account), you'll need to add credit to your account.

Head back to your Voltage account page. Click 'Billing'.

On this page, you'll see a full overview of the costs of running your node and how much credit you have left. Add credit to keep the satoshis flowing in. Voltage accepts Bitcoin, Lightning, and credit card payments.

Know Your Customer(KYC) laws claim to make the world a safer place because it hinders criminals' ability to transact with cryptocurrencies or traditional banks anonymously. That would be true for only the most inept criminals. Any slightly more advanced criminal has already figured out how to stay anonymous and are generally several steps ahead of regulators. The only people actually suffering from the obstacles of KYC are ordinary people like you and me.

Exchanges demand our sensitive data like IDs and tax numbers and store it in perpetuity in the service of KYC compliance. Their databases with this information are the perfect honeypot for criminals to hack into and steal our identities to be used for identity theft. Thus, criminals can continue to use financial platforms requiring KYC by impersonating you and me until the identity is burnt and they move onto the next victim. We pay the price while legislators get to campaign on cracking down on money launderers and criminals get off free. If you're interested in understanding the threats of KYC more, see this Bitcoin Magazine article.

So what do we do about this problem? Well, we must use no-KYC or KYC-light platforms instead. No-KYC is the most anonymous and generally means peer-to-peer trading by buying Bitcoin through a person you know already or meet through an online platform like Bisq or Hodl Hodl. The buyers and sellers negotiate among themselves how much personal information they want to reveal. Buyers pay sellers directly through payment channels unconnected to the platform that introduced them.

The other category is KYC-light which is the model Pocket Bitcoin uses. In this model, you give them a Bitcoin address, your IBAN number, and an email address(you can spin up an anonymous one). You buy Bitcoin by creating an order with an open-ended amount and then you send funds to their bank account with a reference number tied to your order. Funds can be sent on a one-off or recurring basis with the same reference number. When you pay their bank account, the wire transfer will reveal your name in their records.

You obviously reveal some personal information in this model so if you need perfect privacy, this isn't the best option for you. For most people, this model is a great option for revealing minimal sensitive data and only information that can't be used for identity theft.

Additionally, their model is entirely non-custodial meaning the Bitcoin you buy gets sent directly to whatever Bitcoin address(es) you want. It eliminates the extra step of having to withdraw Bitcoin from an exchange because Pocket sends your coins directly to your own wallet.

How to Buy on Pocket

Prerequisites:

1. A Bitcoin wallet(hardware or software)
2. An email address(ideally an anonymous one tied to no other platforms)
3. A bank account in one of the following countries: Andorra, Austria, Belgium, Bulgaria, Croatia, Czech Republic, Denmark, Estonia, Finland, France, Germany, Greece, Hungary, Italy, Latvia, Liechtenstein, Lithuania, Luxembourg, Moldova, Montenegro, Netherlands, North Macedonia, Norway, Poland, Portugal, Romania, San Marino, Serbia, Slovakia, Slovenia, Spain, Sweden, Switzerland, Turkey, United Kingdom
4. Euro or Swiss Franc fiat money

Assuming you have all of the above, you're ready to make your first purchase. Let's go.

Wallet Setup

Head over to Pocket Bitcoin and click 'Set up now'.

The next page will briefly summarize the steps at a high-level. We'll go through each in detail.

Your first action is to select how you want to receive your Bitcoin. Pocket Bitcoin sells hardware wallets like Nano Ledgers and BitBoxes. They also offer Opendimes which are a simple device that stores small amounts of Bitcoin. For this guide, we're only interested in using an existing wallet so select that.

Here you'll choose which wallet you're using.

In reality, you're less picking where your Bitcoin will be sent, but rather how you want to provide Pocket a receive address and sign a message confirming that you own that address. If you use your hardware wallet with software like Sparrow or any other wallet that allows signing messages, you can choose 'Other'. Otherwise, select your device.

I'll be walking through the steps with a hardware wallet like the Ledger and then showing you the steps for the 'Other' option.

Connecting Pocket to a Hardware Wallet

I'll be using a Ledger, but the steps for BitBox, Trezor, and Coldcard should be the same. Once you select your device, you'll be presented with the following screen. Connect your wallet to your computer at this point and make sure it is unlocked and you're using the appropriate Bitcoin app, if applicable. Then click 'Determine address'.

Your browser will ask permission to let pocketbitcoin.com connect to your wallet. Go ahead and hit 'Connect'.

Once it connects, your wallet will walk you through sharing a Bitcoin address with Pocket and signing a message confirming you own that address. Simply follow the steps on the device until you see the following screen.

Connecting with 'Other'

Here we'll walk through the same steps but through the 'Other' path. This can also be done with a hardware wallet connected to wallet software like Sparrow. The main prerequisite here is you need a way to sign a message with your wallet. Ledger Live won't do the trick unfortunately. Check out my guide on signing messages with Sparrow if you're stuck.

Pocket will prompt you to paste in a Bitcoin address you own. Get this through the Receive function of whatever wallet software you use.

Next you will confirm your Bitcoin address by signing a message. I go through the full steps of that in the article linked above.

The message you need to sign will be given to you by Pocket. Simply copy that into your signing tool and then copy the resulting signature back here under 'Message Signature'.

You'll be presented with this success message if you did everything right.

A Note on xpubs

In the steps above, we shared only one address with Pocket. All future purchases through Pocket will be sent to that address unless you repeat the setup process from zero. This is convenient, but it's generally bad practice from a privacy perspective to reuse Bitcoin addresses.

Pocket gives you the option to share your extended public key(xpub) which Pocket can use to send each purchase to a new Bitcoin address. This solves the reuse problem and is very convenient, but introduces a different privacy problem. Giving your xpub to someone allows that entity to see all your addresses associated with that wallet. This means that even if you stopped using Pocket, they could monitor transactions on addresses you never used to receive Bitcoin from them.

You have to decide for yourself what trade offs you can live with. Here are two options if you're keen to optimize for privacy over convenience.

1. Repeat the setup process with a new address every time you buy Bitcoin with Pocket.
2. Use a separate wallet only for Pocket buys, share the xpub with Pocket, and then periodically transfer coins to your cold storage. Pocket could track those subsequent transactions if they wanted, but they wouldn't have access to all your addresses in your cold storage.

If you decide to share your xpub, you can do so once you reach the success page after confirming one address. Click the link to 'share your entire wallet'.

If you use Sparrow, you can get your xpub from the 'Settings' tab of your wallet. If you use any other wallet, look up the steps in this article or do an Internet search for "how to find xpub for your_wallet_name".

Payment Setup

We're on the home stretch now. Next you need to provide an email address to Pocket. This is only used to send you updates on your status of your orders. There is no account creation in Pocket. I would encourage you to use an anonymous email here if you want to optimize for privacy. Or at least use a unique address so if there was a data leak, it would be harder to match the email provided to any of your accounts elsewhere.

Here you'll enter the IBAN of your EUR or CHF account and select what currency you want to pay in. Your IBAN will be available through your bank and is outside of the scope of this article to walk you through. Be sure to choose the appropriate currency because this will change where you send payment to.

Success! You're now ready to buy. Pocket will display a bunch of information you'll need to complete payment. All this information will also be emailed to you.

Add Pocket's bank details as a payee in your bank account using the details from the personal Pocket invoice. Then create a transfer for any amount you want. Make sure you put your payment reference(listed in the Pocket invoice) in the transfer description field before you send the payment. This is crucial to ensure your payment is appropriately assigned for pay out to your Bitcoin address.

Depending on how much you send, Pocket will use Kraken's API's to pull the latest price and calculate the amount of Bitcoin they owe you. The price you get is what the price was when your money hits Pocket's account. Use instant SEPA payments if you're anxious about future price movements.

Note that Pocket takes a 1.5% cut. Compared to the premiums one pays on decentralized marketplaces like Hodl Hodl and Bisq, Pocket's 1.5% cut is a bargain.

For very large one-time orders, they advise contacting them first.

Receive Your Coins

Once your bank transfer lands in Pocket's account, you'll receive a confirmation of your purchase via email.

Pocket pays out once a day at about 10pm Central European time. This allows them to batch all their pay outs into fewer transactions and reduce transaction fees for everyone involved. Use the 'Track your payout' link in the confirmation email to check the status of your pay out.

If you managed to follow all that, you're now the proud owner of some KYC-light Bitcoin. Don't spend it all on place. In fact, don't spend it at all. Just hodl.

There are many reasons to use more feature rich wallets like Sparrow, Electrum, and others instead of the apps shipped alongside hardware wallets(e.g. Ledger Live). One of the particularly nice features is a simple signing tool is baked into these wallets I listed which allow you to create cryptographically signed messages proving to the world that a message was truly written by you.

In reality, the signature only proves that the message was written by someone who holds a particular private key so you must keep your private key secure at all times to avoid impersonation. Cryptographic signatures work by encrypting a message with your private key that can be verified using your public key. The goal here is not to pass a secret, but rather it is to prove that no one else could have written a message besides the owner of the private key associated with the public key used to verify the message.

This is most interesting to us as Bitcoiners because this functionality is the basis for how you send Bitcoin to someone else. Your wallet signs a message with transaction data that tells the network that your Bitcoin now belongs to someone else. That's not all though.

You can also use signatures to prove to someone you control a particular Bitcoin address. This is useful if you want to prove you hold a certain amount of Bitcoin or prove to a Bitcoin exchange that you own an address to reduce the risk of withdrawing Bitcoin to an address you don't control. My favorite platform to buy Bitcoin, Pocket Bitcoin, uses this model to send purchased Bitcoin directly to your cold storage.

The math behind signing messages may be very complex, but Sparrow makes signing a message as simple as copy, paste, and click . First, open up Sparrow Wallet. Check out my guide on setting up a Nano Ledger on Sparrow if you haven't moved to Sparrow yet.

Navigate to the 'Receive' tab to get a Bitcoin address. Copy this to your clipboard.

Open 'Tools' from the Sparrow menu and select 'Sign/Verify Message'.

This will open a window where you paste the address you copied previously and enter the message you want to sign. If you're doing this to buy from Pocket Bitcoin, this is where you enter the message they ask you to sign. Leave the 'Signature' box empty as this is where Sparrow will deposit your signature.

At this point, make sure your hardware wallet is connected, unlocked, and if applicable, its Bitcoin app is open.

Next, hit the 'Sign' button and you'll be prompted to choose your signing device.

Click 'Sign Message' and follow the instructions on your device's screen. For example, Ledgers will ask you to confirm the address and a hash of the message you're signing. Once you confirm everything, the 'Signature' box will show your signed message. Done!

Note that this same tool can also verify a signature. While all three fields are still filled, try clicking the 'Verify' button. You should get confirmation of a good signature.

Finally, try making a change to either the message or the signature such as adding or removing a character. Try verifying again and you'll see it fails.

I've recently been taking my Bitcoin custody and privacy more seriously. That means different things to different people, but for me it means moving towards services that better embody the ethos of Bitcoin; open sourced, decentralized, and self-custodied. Like many of you, I started my crypto journey holding coins on exchanges. Soon I learned hardware wallets were a better tool to secure your coins so I got myself a Nano Ledger S.

Knowing what I know now, I probably would have purchased a different hardware wallet, but my Ledger has nevertheless performed well. For a new user, a Ledger is a great hardware wallet. Their software Ledger Live is also perfectly fine for a new user with its simplicity, but if you're looking to take your Bitcoin custody more seriously, there are better options such as open source and privacy focused Sparrow.

In the developer's words, "Sparrow is a Bitcoin wallet for those who value financial self sovereignty. Sparrow’s emphasis is on security, privacy and usability." It's also designed for both beginners and advanced users in that it does not hide any data from you the way tools like Ledger Live does. If you want to micromanage your UTXOs(which you should), it empowers you to do that. On the other hand, if you don't even know what a UTXO is, that is fine and you can still use Sparrow.

For our purposes here, the important thing is that it integrates cleanly with Ledger devices. Here's the simple process to switch away from Ledger Live to Sparrow. These steps are virtually the same even if you're using another hardware wallet.

Download & Verify Sparrow

Head over to Sparrow's download page and follow the download and verification instructions. I know you'll be tempted to skip the verification steps. Take it seriously and learn something. You'll encounter this challenge again in your Bitcoin journey so give it a go.

Connect Sparrow to the Bitcoin Network

You'll be presented with a welcome screen when you first open Sparrow that explains a bit about Sparrow's offline and online modes. Offline mode is primarily for more serious privacy and security focused use cases. For most users who want to simply receive and send transactions without too much friction, leaving Sparrow in online mode will be fine.

Online mode requires connecting to the Bitcoin network via a node that will be your relay for downloading and broadcasting your transactions. Sparrow will show you three server options: Public Server, Bitcoin Core node, and Private Electrum Server.

In an ideal world, you have your own node, but that is a more advanced setup to consider in the future. The easiest option is to connect to a public server. There are some privacy downsides to using a public server, but if you're coming from Ledger Live, you've already revealed the same information through their servers. Moving to Sparrow is a good first step to better privacy. Take the next step with your own node or Electrum server when you're ready.

Once you go through the connection options, you'll be asked to configure your connection. Choose 'Public Server' and click 'Create New Wallet'.

Create Your Wallet

Give your wallet a short and descriptive name.

Now you'll need to choose a 'Keystore' for your wallet. Select 'Connected Hardware Wallet' to configure your Ledger.

Your Ledger device must be unlocked and the Bitcoin app must be open to be discoverable by Sparrow. If your Ledger is not in this state, you'll get a 'No hardware wallets found' error. Assuming the scan is a success, you'll see your Ledger pop up in the list and you can click 'Import Keystore'. This imports your public keys and Bitcoin addresses into Sparrow. Private keys never leave hardware wallets.

You can set a password for additional security.

If all goes well, you'll see the following screen. Hit 'Apply' to complete the import.

Done! Sparrow will use your public key to calculate all your addresses and then look up past transactions which will be displayed in the 'Transactions' and 'UTXOs' tabs.

Add Accounts (Optional)

If you've ever created additional Bitcoin accounts in Ledger Live, you'll need to import these as well. This is very simple. Go to the 'Settings' tab and look for the 'Add Account' button.

Sparrow will show you a list of account numbers. 'Account #1' will be the second account you created in Ledger Live. 'Account #2' will be the third and so on. Adding each account works the same as the initial wallet setup you did in the previous step.

What Next?

Take a look around Sparrow and take note of how much more detailed information it provides you compared to Ledger Live. This information, when used correctly, can significantly improve your Bitcoin management and general understanding of how Bitcoin works under the hood. Check out Sparrow's documentation if you'd like to learn more.