Web Scraping User Reviews from Amazon.com

Learn in this post how to scrape user reviews from the Amazon.com website using .NET 6 and Playwright step-by-step.

Pssst… over here… In the world of e-commerce, user reviews play a crucial role in influencing other people’s purchasing decisions. Besides that, we can harness this data to possibly train some neural network to classify reviews or even create pseudo reviews for a product. Using tools like Playwright and .NET Core, we can create a web scraper to collect these review data. In this blog post, I’ll show how to extract data from user reviews of books on Amazon.com.

Setting Up Your Environment

Before diving into the procedure, make sure you have .NET Core 6 SDK installed. You can download it from the official .NET website (I’m using Ubuntu Linux. If you are too, check the instructions here). After that, create a new C# console application project using your favorite IDE or by using the command line. I’ll use Visual Studio Code and the .NET command line tool.

mkdir AmazonScraper && cd AmazonScraper && dotnet new console --use-program-main

This command will create a directory called AmazonScraper, cd into it and create a new .NET console application with the old program style (with the main method).

Installing Playwright

Playwright is an automation library like Selenum or Puppeteer that allows you to control web browsers programmatically. You can install the Playwright NuGet package using the following command in your project directory:

dotnet add package Microsoft.Playwright

This will install Playwright and its dependencies into your project. You can run Playwright with Chrome/Chromium or Firefox browser that you have installed on your system but it is also possible to install an “embedded” browser to ship it with your program. If it rings a bell, check how to install it here.

First Test, Navigate to Amazon.com

This program can be divided into two main functions: navigation and exporting data. As you may have guessed, the Navigation function navigates through pages and collects all data needed into a Review list. Also, this method saves the current state of the execution. The Export method does precisely that, it exports data as a JSON file.

As mentioned before, I’m using the system-installed Chromium browser just by defining the ExecutablePath property of BrowserTypeLaunchOptions and passing it to the method LaunchAsync. It is worth mentioning that if you want to run the browser without rendering the main window, you have to set Headless = true (or by not defining the Headless property, once Headless = true is the default option):

namespace AmazonScraper;

using Microsoft.Playwright;
using System.Threading.Tasks;

public class Program {

    public static async Task Navigate() {
        using var playwright = await Playwright.CreateAsync();
        await using var browser = await playwright.Chromium.LaunchAsync(
                this is important if you did not installed the "embedded"
                browser. here I'am pointing the installed location of 
            new BrowserTypeLaunchOptions()
                    Headless = true if you want to run the browser 
                    without rendring the main window
                Headless = false,
                    system browser executable path
                ExecutablePath = "/snap/bin/chromium" 

        var page = await browser.NewPageAsync();
        await page.GotoAsync("https://www.amazon.com.br/s?k=livros");

    public static async Task Export() {

    public static async Task Main(string[] args) {
        await Navigate();
        await Export();

If you run this code, you should see a browser window and it will navigate automatically to the Amazon books page.

The goal is to scrape as many reviews as possible so I built an automatic navigation to recognize all the book elements in the root category, and then navigate to its review page.

Playwright provides a test generator that can generate C# code while you navigate manually through the site. The generator will recognize mouse clicks, the text you put on some textbox, read the XPath/Selector for each element and build all the jazz for you. You can find more about this kind of profanity here.

I went for the funny way and panned the XPath/Selectors for each element using the browser dev tools/console.

Review Object

This class has properties to identify the review.

public class Review
    /// <summary>
    /// review unique id
    /// </summary>
    public string Id { get; set; }

    /// <summary>
    /// product unique id
    /// </summary>
    public string ProductId { get; set; }

    /// <summary>
    /// review title
    /// </summary>
    public string Title { get; set; }

    /// <summary>
    /// review rating 1 to 5 stars
    /// </summary>
    public decimal Rating { get; set; }

    /// <summary>
    /// review body
    /// </summary>
    public string Comment { get; set; }

Execution State and Configuration Object

I created the CurrentState object to give some flexibility, track how is going the process and add some failure handling. It has properties like current URL, next URL, max pages to navigate, etc.

public class CurrentState
    /// <summary>
    /// current product url
    /// </summary>
    public string ProductsUrl { get; set; }

    /// <summary>
    /// all product ids on the current page
    /// </summary>
    public List<string> ProductList { get; set; }

    /// <summary>
    /// the product id that is currently being processed
    /// </summary>
    public string CurrentProduct { get; set; }

    /// <summary>
    /// the next product page url
    /// </summary>
    public string NextUrl { get; set; }

    /// <summary>
    /// the maximum number of pages to be read
    /// </summary>
    public int MaxPages { get; set; }

    /// <summary>
    /// delay time in seconds between url navigations
    /// </summary>
    public int Delay { get; set; }

    /// <summary>
    /// the number of the page that is currently being processed
    /// </summary>
    public int CurrentPage { get; set; }

    /// <summary>
    /// the review list
    /// </summary>
    public List<Review> Reviews { get; set; }

    /// <summary>
    /// amazon store language (pt-BR, en-GB, en-US...)
    /// </summary>
    public string StoreLanguage { get; set; }

    /// <summary>
    /// amazon product review base url
    /// </summary>
    public string ProductReviewBaseUrl { get; set; }

    /// <summary>
    /// amazon product base url
    /// </summary>
    public string AmazonBaseUrl { get; set; }

Also, I added some functions and variables to the file Program.cs to load and save the state.

Define a static variable to hold the name of the state:

private const string CurrentStateFileName = "current_state.json";

Initialize the CurrentState object with default values.

private static CurrentState currentState = new()
    MaxPages = 1,
    AmazonBaseUrl = "https://www.amazon.com.br",
    ProductReviewBaseUrl = "https://www.amazon.com.br/product-reviews",
    ProductsUrl = "https://www.amazon.com.br/s?k=livros",
    Delay = 5,
    ProductList = new List<string>(),
    Reviews = new List<Review>(),
    StoreLanguage = "pt-BR"

And write the functions to read and write the current state.

private async Task SaveCurrentState()
    Console.WriteLine($"Save Current State: {currentState}");
    await File.WriteAllTextAsync(CurrentStateFileName, JsonConvert.SerializeObject(currentState));

private async Task LoadCurrentState()
    if (!File.Exists(CurrentStateFileName)) return;
    Console.WriteLine($"Load Current State: {currentState}");
    var tFile = await File.ReadAllTextAsync(CurrentStateFileName);
    currentState = JsonConvert.DeserializeObject<CurrentState>(tFile);

I used Newtonsoft Json to handle the serialization and deserialization of objects. Add it to the project by running:

dotnet add package Newtonsoft.Json

Improving Navigation

Once the book page is loaded, use Playwright methods like QuerySelectorAllAsync, QuerySelectorAsync, GetAttributeAsync and InnerTextAsync to interact with page elements and extract the relevant information.

Here is an example:

var productId = await productContainer.GetAttributeAsync("data-asin");

This line of code is getting the product ID from a data attribute defined on an HTML tag, from a div in this particular case.

The navigation procedure is straightforward forward and it follows this diagram.

At the start point “next URL” is the product URL defined via the current state JSON file or at the CurrentState object initialization, if there is no previous JSON file.

Navigation method flow diagram
Flow diagram for the navigation function

Is worth mentioning that each URL navigation is preceded by a delay with a bit of time drift, a small wait to simulate a random person navigating through products and reviews. I do not know if Amazon will block requests or ask for a captcha resolution but keep in mind this kind of trickery.

The histogram mentioned on the flow diagram is a star gauge widget that groups reviews by ratings. Like the one below.

Example of a review histogram
Review histogram

Data Filtering and Exporting

At this point, you should have a pretty neat collection with hundreds of reviews yet, filtering and manipulating this data is made easy by using Linq.

First of all, let’s exclude duplicates if any.

var reviews = currentState.Reviews
    .GroupBy(x => x.Id)
    .Select(x => x.First()) //exclude duplicates
    .OrderBy(x => x.Rating)

Now let’s remove double spaces, new lines and some special characters that appeared somehow. Also, I changed the ratings according to my needs.

I’ll use this data to train an AI to categorize short sentences into sentiments that could be negative, neutral or positive. So to achieve that goal, I translated the ratings from a five-level (five stars) sentiment to a three-level one. That is, ratings less than three are now recognized as negative and with a value of zero, ratings equal to three are now recognized as neutral and with a value of 1 and the rest are recognized as positive and with a value of 2.

reviews.ForEach(r =>
    if (Regex.IsMatch(r.Comment, @"\s{2,}"))
        r.Comment = Regex.Replace(r.Comment, @"\s{2,}", " ", RegexOptions.Multiline);
    if (Regex.IsMatch(r.Comment, @"\xA0"))
        r.Comment = Regex.Replace(r.Comment, @"\xA0", "", RegexOptions.Multiline);
    if (Regex.IsMatch(r.Comment, @"\n"))
        r.Comment = Regex.Replace(r.Comment, @"\n+", "", RegexOptions.Multiline);
    if(r.Rating < 3) r.Rating = 0;
    if(r.Rating == 3) r.Rating = 1;
    if(r.Rating > 3) r.Rating = 2;

After that, I saved it to a JSON file with a sample list containing the same number of each one of the sentiments.


As you extract reviews, you can store them in a suitable data structure or save them to a database. Afterward, you can use the reviews to play with sentiment analysis or maybe gain insights into the book’s reception.

In conclusion, web scraping user reviews of books on Amazon opens up a world of possibilities for extracting valuable information. From setting up the environment to navigating web pages, extracting reviews, and analyzing the data, this guide has introduced you to the fundamentals of the process.

Note: Remember to adhere to ethical guidelines while scraping data from websites.

Get the sources: https://github.com/raffsalvetti/AmazonScraper

Until the next one!

ATtiny85 Gamepad

Build your own gamepad from literally scratch using the ATtiny85 microcontroller, the V-USB driver and a lot of creativity.

Hey listen… this project is from 2020, the pandemic boom. From when people were spreading COVID-19 all over the world.

I’ll show how I built an HID gamepad using the Atmel ATtiny85 microcontroller, shift registers, some passive components and firmware based on the V-USB library.

The ATtiny85 is a small and powerful microcontroller that, despite its low pin count, is perfect for this kind of project.

HID Basics

HID stands for Human Interface Device. These USB devices are used to communicate to the computer using drivers shipped with the operating system, so you don’t need to install additional drivers. Some common examples of HID devices include keyboards, mice and game controllers. HID devices use the USB protocol to communicate with the computer and the devices are “configured” by the HID report descriptor.

The HID device descriptor is used to inform the host device (mostly your desktop or laptop but can be any device that can operate in USB “host mode”), how data sent (reports) by the device will be interpreted. It is a byte array defined by you in your program, configured according to the needs of your device. You can find great information at the usb.org website. Also on the page, you can find a tool that helps to create the device descriptor of your HID device. The device descriptor configures things like the device class, how many devices (yes, you can use just one USB device and report multiple gamepads =]) and how big are your data packages. For example, the descriptor of my gamepad is configured with 8 buttons and 2 axis but it could be configured with more buttons or axis by changing the descriptor byte array.

Circuit design

The hardware setup for this project is relatively simple. We will need an ATtiny85 microcontroller, a USB connector, two shift registers and some simple components.

The ATtiny85 (datasheet) is a great microcontroller but has a few pins, two for power and ground, one for reset and five multi-function pins. To extend its I/O pins to my needs, I used two shift registers connected in series (daisy chained). A shift register takes a clocked serial input and latches its output pins to a low state (ground), high state (5 volts for this project) and sometimes (depending on the shift register) high impedance state (disconnect the pin). For this project, I used the shift register 74HC595 (datasheet).

Then I defined the microcontroller pins for the following tasks:

  • Driver the shift registers (three pins: data, clock and latch)
  • Read the state of each button and axis (one pin: state sense)
  • USB communication (two pins: D+ and D-)

To drive the 74HC595 we need to LATCH a CLOCKED serial DATA generated by the microcontroller, which means that at least three pins of the five available I/O pins will be used. We could program the RESET pin to work as an ordinary I/O pin and get all the six pins we need, but after that, the RESET pin could not be used to program the microcontroller. This is not a good idea in the early stages development of a circuit. Instead, we will use the CLOCK line to also drive an NPN transistor and charge a capacitor. This capacitor holds the shift registers LATCH pin in a high state for enough time, so the shift registers can activate their output pins.

The following image shows the LATCH driver.

Latch circuit
Latch driver

I selected the capacitor value by trial and error. For my circuit, a 100nF ceramic capacitor was good enough.

My gamepad has 8 buttons and 2 axes (two pins for the X-axis and two pins for the Y-axis). To read all the possible states (pressed or not pressed) for all these inputs, at least 12 pins are needed out of the 16 pins provided by the two 74HC595 connected in series. Each output has a diode to prevent shadowed reads when multiple buttons are pressed at the same time.

Shift register circuit
Daisy chained shift registers

Shifting one bit thru the shift registers will activate each button individually and the microcontroller will read thru the SIGNAL line for any button press.

The SIGNAL line must have a pull-down resistor otherwise parasitic capacitance may cause wrong reads or activate permanently all buttons and axis at the same time. I used a 1k resistor for the pull-down, but values between 1k and 4k7 are acceptable.

Signal line circuit
SIGNAL pull-down resistor

According to the V-USB library documentation, we must use an interruption pin for the D- line. At this point, we can define all the microcontroller physical pins for each function as shown below.

Microcontroller pins and their roles/functions

The V-USB library documentation provides some examples of circuits to the USB data lines, as shown in the next figure. It must follow the USB standards otherwise the device may not function as expected.

USB connection circuit

I used three of the free pins of the shift registers to drive an RGB LED.

RGB LED circuit

I build the circuit on a breadboard and started to write the firmware for it.

Gamepad circuit on a breadboard

Firmware development

Most of the hard work of this firmware is done by the V-USB library (lucky me!) which is also the most time-consuming for the microcontroller itself. The firmware can be divided into five parts:

  • V-USB configuration
  • HID report descriptor definition
  • Shift register driver
  • LED driver
  • Main loop

V-USB configuration

The V-USB configuration is made easy because of the usbconfig.h header is well documented. We just follow the comment instructions on each define line according to our needs. Here are some important configurations for this project:


#define USB_CFG_DMINUS_BIT      3

#define USB_CFG_DPLUS_BIT       4

#define USB_INTR_CFG            PCMSK
#define USB_INTR_CFG_SET        (1 << USB_CFG_DPLUS_BIT)
#define USB_INTR_CFG_CLR        0
#define USB_INTR_ENABLE         GIMSK
#define USB_INTR_PENDING        GIFR
#define USB_INTR_VECTOR         PCINT0_vect

#define  USB_CFG_VENDOR_ID      0xc0, 0x16
#define  USB_CFG_DEVICE_ID      0xdc, 0x27

#define USB_CFG_VENDOR_NAME     'A', 'R', 'E', 'N', 'A', '6', '4'

#define USB_CFG_DEVICE_NAME     'F', 'i', 'n', 'g', 'e', 'r', 'G', 'r', 'i', 'n','d', 'e', 'r'

You should put the name of your gamepad in USB_CFG_DEVICE_NAME. It is the name that will show up on the Windows tray when you plug in your gamepad. Mine is FingerGrinder.

HID report descriptor definition

I used the “HID usage table” document which can be found here to create my descriptor. This file gets updates from time to time. You can find the update here. Also, you can use the “HID descriptor tool” to build the device descriptor.

0x05, 0x01,                    // USAGE_PAGE (Generic Desktop)
0x09, 0x05,                    // USAGE (Game Pad)
0xa1, 0x01,                    // COLLECTION (Application)
0x09, 0x01,                    //   USAGE (Pointer)
0xa1, 0x00,                    //   COLLECTION (Physical)
0x09, 0x30,                    //     USAGE (X)
0x09, 0x31,                    //     USAGE (Y)
0x15, 0x00,                    //   LOGICAL_MINIMUM (0)
0x26, 0xff, 0x00,              //   LOGICAL_MAXIMUM (255)
0x75, 0x08,                    //   REPORT_SIZE (8)
0x95, 0x02,                    //   REPORT_COUNT (2)
0x81, 0x02,                    //   INPUT (Data, Var, Abs)
0xc0,                          // END_COLLECTION
0x05, 0x09,                    // USAGE_PAGE (Button)
0x19, 0x01,                    //   USAGE_MINIMUM (Button 1)
0x29, 0x08,                    //   USAGE_MAXIMUM (Button 8)
0x15, 0x00,                    //   LOGICAL_MINIMUM (0)
0x25, 0x01,                    //   LOGICAL_MAXIMUM (1)
0x75, 0x01,                    // REPORT_SIZE (1)
0x95, 0x08,                    // REPORT_COUNT (8)
0x81, 0x02,                    // INPUT (Data, Var, Abs)
0xc0                           // END_COLLECTION

Shift register driver

The ATtiny85 has a USI (Universal Serial Interface) which can be configured to operate in SPI mode. We will use this interface to generate the CLOCK signal and send the serial DATA to the shift registers.

The latch part of this serial driver has to be put into the main loop because we have to latch the signal before trying to read each button.

SPI_PORT |= (1 << USCK_PIN); //enable clock (and charge latch cap)
_delay_us(1); //waiting for charge
SPI_PORT &= ~(1 << USCK_PIN); //disable clock

If you do not understand how to expand the microcontroller I/O with shift registers or read the state of buttons using a single input pin, I suggest you watch this great video.

LED driver

The LED driver was based on a simple and nice code that you can find at Łukasz Podkalicki

Main loop

At the main loop, we poll the USB and “wait” to send the data relative to the current state of the gamepad, if any button is pressed. It is important to sort/order the data according to the descriptor that was defined earlier.

if(usbInterruptIsReady()) { //send data when the usb is ready to receive
	if(has_changed) { //and when any change occurred
		buildReport(); //build the report according descriptor
		has_changed = 0; //clean change flag
	usbSetInterrupt(report_buffer, sizeof(report_buffer)); //send data

After that, we have a loop to read all buttons and mark if it was pressed or not.

while(!(mask & 0x1000)) {
	// placing bit on right place to shift into shifters
	b2 = (uchar)((mask & 0xff00) >> 8);
	b1 = (uchar)(mask & 0x00ff);
	b2 |= (rgb_led_state() << 5);

	SPI_PORT |= (1 << USCK_PIN); //enable clock (and charge latch cap)
	_delay_us(1); //waiting for charge
	SPI_PORT &= ~(1 << USCK_PIN); //disable clock

	// loading serial data (shifting bits to the shift register)

	_delay_us(666); //waiting for button settle down (debouncing)

	if(PINB & 0x01) { //reading if the button is pressed
		signal |= mask; //add the state of each button to the buffer
	mask = (mask << 1); //set the next button to be read

Building and flashing the firmware

To write the code I used the VSCode with C/CPP extensions. The build process is facilitated with Makefiles. Here’s an example.

# -------------- start of configurtion --------------

PROJ_NAME = firmware
DEVICE = attiny85
CLOCK = 16500000L

# https://www.engbedded.com/fusecalc/
FUSE_LOW = 0x62
FUSE_HIGH = 0xdd

INCLUDES = -I/usr/lib/avr/include -I./src -I./src/usbdrv
CFLAGS = -std=c11 -Wl,-Map,$(PROJ_NAME).map -mmcu=$(DEVICE) -DF_CPU=$(CLOCK) $(INCLUDES)

# -------------- end of configuration --------------

AVRDUDE = avrdude
OBJCOPY = avr-objcopy
OBJDUMP = avr-objdump
AVRSIZE = avr-size
CC = avr-gcc

H_SOURCE = $(wildcard ./src/*.h)
H_SOURCE += $(wildcard ./src/usbdrv/*.h)
C_SOURCE = $(wildcard ./src/*.c)
C_SOURCE += $(wildcard ./src/usbdrv/*.c)
S_SOURCE = $(wildcard ./src/usbdrv/*.S)
CPP_SOURCE = $(wildcard ./src/*.cpp)

OBJ = $(C_SOURCE:.c=.o) $(S_SOURCE:.cpp=.o) $(CPP_SOURCE:.cpp=.o)

RM = rm -rf

all: objFolder hex eep size

$(PROJ_NAME).elf: $(OBJ)
	@ echo 'Linking: $@'
	$(CC) $(CFLAGS) $(CPPFLAGS) $(LFLAGS) $(LIBS) $^ -o $@
	@ echo 'Finished linking: $@'
	@ echo ' '

./obj/%.o: ./src/%.c ./src/%.cpp ./src/%.h
	@ echo 'Building objects: $<'
	$(CC) -c $(CFLAGS) $(CPPFLAGS) $< -o $@
	@ echo ' '

./obj/main.o: ./src/main.c $(H_SOURCE)
	@ echo 'Building main: $<'
	$(CC) -c $(CFLAGS) $(CPPFLAGS) $< -o $@
	@ echo ' '

hex: $(PROJ_NAME).elf
	$(OBJCOPY) -R .eeprom -R .fuse -R .lock -R .signature -O ihex $(PROJ_NAME).elf $(PROJ_NAME).hex

eep: $(PROJ_NAME).elf
	$(OBJCOPY) -j .eeprom --no-change-warnings --change-section-lma .eeprom=0 -O ihex $(PROJ_NAME).elf $(PROJ_NAME).eep

size: $(PROJ_NAME).elf
	$(AVRSIZE) --format=avr --mcu=$(DEVICE) $(PROJ_NAME).elf

disasm: $(PROJ_NAME).elf
	$(OBJDUMP) -d $(PROJ_NAME).elf

	@ mkdir -p obj

	@ $(RM) ./obj/*.o ./src/*.o $(PROJ_NAME).elf $(PROJ_NAME).hex $(PROJ_NAME).eep $(PROJ_NAME).map 


flash: all
	$(AVRDUDE) -c $(PROGRAMMER) -p $(DEVICE) -U flash:w:$(PROJ_NAME).hex:i

	$(AVRDUDE) -c $(PROGRAMMER) -p $(DEVICE) -U flash:r:dump_$(shell date +'%y%m%d%H%M%S').hex:i

	$(AVRDUDE) -c $(PROGRAMMER) -p $(DEVICE) -U lfuse:w:$(FUSE_LOW):m -U hfuse:w:$(FUSE_HIGH):m -U efuse:w:$(FUSE_EXTENDED):m	

.PHONY: all clean

# DO NOT DELETE THIS LINE -- make depend needs it

To build just type:

make clean && make

To flash the firmware I use AVRDUDE (also in the makefile). Just run this command:

make flash

The fuses were configured for this particular project. Before running make fuse read the ATtiny85 datasheet to understand its meanings.

Building the gamepad case

This task was the most time-consuming one. I do not have a laser cutter or a 3D printer so all the work was hand-made using prehistoric tools such as hand saws, files and sandpaper. The job wasn’t worse because, thank god, I have a Dremel and an electric drill.

I guessed that all the work would be done on a weekend but I was so wrong! It took almost fifteen f*#&ing days to finish the whole project. The electronics and programming were finished at the weekend, but all the structural handwork took me ten days. Most of the time was spent searching for the right part, cutting, sanding, gluing and making it all fit together.

I used a tuna can as the case, scraped plexiglass to hold the keys in place and the cover, random plastic bits like buttons (yes sir, the one used in sewing), scraped pocket calculator parts to make the key contacts and some random stuff to hold everything in place.

I ate all the tuna into this can, washed it for real, pimped my can with a nice’n shiny blue vinyl film to the outside, cut some cardboard and placed to the bottom of the can to ensure no short circuit, marked the place to drill the hole for the USB port and then finished it with a small file. In the end, I cut skinny pieces of black EVA foam and glued them to the border to make it look good!

Testing if the USB port fits well into the tuna can

Scared tuna can (eyes are markings for drilling the shoulder buttons holes and mouth is the USB port)

Assembling and hand-wiring the circuit also took some time but it was done in the weekend.

Bottom of the perfboard with my beautiful hand-wire job

The key contacts board was built using scraped plexiglass, pieces of a PCB from an old solar calculator, magnet wire and flat cable from an old floppy driver.

Key contacts board

The button holes were made with a drill and the Dremel sanding drum. I cut the D-PAD slot with the Dremel cut disk and finished it with a file.

Top view of the partially assembled gamepad

The directional control (D-PAD) was made with epoxy putty and the gamepad buttons with colored buttons.

D-PAD made of white epoxy putty

Colored buttons glued with epoxy glue to two boring white buttons.

Buttons made of buttons

The shoulder buttons and their holders were custom-made using random plexiglass scrap and bits of some retractable pens.

Custom-made shoulder button, holder and my nails in need of care

The structure is stacked together with epoxy glue, mounting posts, that are used to hold motherboards and screws.

Side view showing the circuit board, key contacts holder, cover and the mounting posts


The gamepad worked very well, I was able to play games that need speed and accuracy in movements, however, the D-PAD and the buttons have some gaps that could be fixed if I use more accurate methods or tools during its production. Maybe I would use epoxy resin and models for the next time.

Building a HID gamepad using the Atmel ATtiny85 microcontroller and the V-USB library is not as difficult as it seems and while challenging, the process of building something useful out of scraped materials is a very enjoyable one. I really need to buy a 3D printer and a laser cutter LOL!

Get the source: https://github.com/raffsalvetti/FingerGrinder

See ya!

Deploying a docker service stack

Hey listen… docker is an open-source platform for building, shipping, and running distributed applications. It allows developers to package their applications and dependencies into lightweight containers that can run on other machine with Docker installed. Docker Swarm is Docker’s native clustering and orchestration feature that allows you to deploy and manage multiple Docker containers across a cluster of machines.

I’ll show the process of installing Docker on a Linux machine, configuring Swarm mode, create a docker image of a net core app service and deploying a simple stack service containing RabbitMQ, Redis and the created .NET Core API image.

Install Docker on a Linux Machine

The first step is to install Docker on your Linux. This can be done by updating your Linux machine’s package index, and then installing Docker using the appropriate package manager command. Once installed, you can verify that Docker is running by executing a command that checks its status.

I’m using Ubuntu 20.04, you’ll have to adapt commands to a different distro. Also, I’m using the official Docker package not the Docker package shipped with Ubuntu!

You MUST uninstall any docker related package shipped with your system. To do that simple run:

sudo apt remove docker.io docker-doc docker-compose podman-docker containerd runc

Now install some tools necessary to download and install repository security keys:

sudo apt update && sudo apt install ca-certificates curl gnupg tee

Add Docker’s official GPG keys:

sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
sudo chmod a+r /etc/apt/keyrings/docker.gpg

Setup Docker’s official repositories:

echo "deb [arch="$(dpkg --print-architecture)" signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
  "$(. /etc/os-release && echo "$VERSION_CODENAME")" stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

Update your packages definitions:

sudo apt update

Install Docker using the following command:

sudo apt install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

After installation, verify that Docker is running by executing the following command:

$ sudo systemctl status docker

You should see an output that shows Docker is active and running.

Now test Docker installation with:

sudo docker run hello-world

This command downloads a test image and runs it in a container. When the container runs, it prints a confirmation message and exits.

The Docker daemon binds to a Unix socket, not a TCP port. By default it’s the root user that owns the Unix socket, and other users can only access it using sudo. The Docker daemon always runs as the root user.

If you don’t want to preface the docker command with sudo, create a Unix group called docker and add users to it. When the Docker daemon starts, it creates a Unix socket accessible by members of the docker group. On some Linux distributions, the system automatically creates this group when installing Docker Engine using a package manager (like Ubuntu). In that case, there is no need for you to manually create the group.

The docker group grants root-level privileges to the user. For details on how this impacts security in your system, see Docker Daemon Attack Surface.

If you don’t want to use sudo when dealing with docker, add your user to the docker group:

sudo usermod -aG docker $MYUSER

You have to log out and log back in so that your group membership is re-evaluated.

Configure Docker Swarm Mode

Swarm mode is Docker’s native clustering and orchestration feature that allows you to deploy and manage multiple Docker containers across a cluster of machines. Initializing Docker Swarm mode is a simple process that involves running a command to initialize the swarm.

Initialize Docker Swarm mode by executing the following command:

$ docker swarm init

You can add machines to the swarm (the cluster) by asking a token to the main manager (this machine where you just installed docker and enabled the swarm mode) with one of these commands:

To add a manager node (a machine which will manage and run services):

docker swarm join-token manager

To add a worker (a machine which will only run services):

docker swarm join-token worker

A command will be generated as output. You should run this command on the target machine that will join the cluster.

Create the DotNet Core Api Docker Image

This is a simple example and it will not communicate with rabbitmq nor redis. I’ll use the Weather Service (WeatherForecast) provided as API template by the dotnet new command.

To create a new net core project run:

mkdir -p MySolution/MyService && cd MySolution/MyService && dotnet new webapi -f netcoreapp3.1

This will create a functional API service and you can test it by running the command:

dotnet run

The command will output the port (usually 5000) and spawn the service.

In order to create a Docker image for the API service, we need to create a Dockerfile file, in the same directory of your code, with the following content.

FROM mcr.microsoft.com/dotnet/aspnet:3.1 AS base

FROM mcr.microsoft.com/dotnet/sdk:3.1 AS build
COPY ["MyService/MyService.csproj", "MyService/"]
RUN dotnet restore "MyService/MyService.csproj"
COPY . .
WORKDIR "/src/MyService"
RUN dotnet build "MyService.csproj" -c Release -o /app/build

FROM build AS publish
RUN dotnet publish "MyService.csproj" -c Release -o /app/publish

FROM base AS final
COPY --from=publish /app/publish .
ENTRYPOINT ["dotnet", "MyService.dll"]

Now build the image by running this command at the same folder where you created the Dockerfile file:

docker build -f Dockerfile -t myservice ../

This command will create a docker image with your service and tag it with the name myservice.

To run the image execute this command:

docker run --rm -it -p 5000:80 myservice

This will create a container running your service at TCP port 5000.

To exit the container press CTRL+C (this will also remove the container due to the --rm parameter)

To test your service execute this command:

curl -k "http://localhost:5000/WeatherForecast"

The output should be a JSON array like this one:


Deploy the Stack Service

Now that Swarm mode is configured and we have a functional docker image of our API service, we can deploy the stack service containing RabbitMQ, Redis, and the NetCore Forecast Service API. This involves creating a docker-compose.yml file that defines the services and then deploying the stack service using docker command line tools.

Create a file named docker-compose.yml and add the following content:

version: '3.7'

    image: rabbitmq:3-management
      - my-net
      - "15672:15672"
      - "5672:5672"
      replicas: 1

    image: redis
      - my-net
      - "6379:6379"
      replicas: 1

    image: myservice
      - my-net
      - "5000:80"
      replicas: 2

    driver: overlay

This file describe all the services docker will create and how they interact with each other. The deploy.replicas property define how many instances of each service will be created. The networks section defines the overlay network to be used by the services.

The overlay network driver creates a distributed network among multiple Docker daemon hosts. This network sits on top of (overlays) the host-specific networks, allowing containers connected to it (including swarm service containers) to communicate securely when encryption is enabled. Docker transparently handles routing of each packet to and from the correct Docker daemon host and the correct destination container.

Deploy the stack service using the following command:

docker stack deploy -c docker-compose.yml my-stack

Verify that the stack service is running by executing the following command:

docker stack ps my-stack

You should see an output that shows the status of each service in the stack.

After deploying the stack service, you may want to scale the .NET Core application to handle increased traffic. Scaling the service in Swarm mode is a simple process that does not require restarting the stack. Here’s how you can do it:

Check the current number of replicas for the api service by executing the following command:

docker service ls

You should see an output that shows the number of replicas currently running for each service in the stack.

Scale the api service by executing the following command:

docker service scale my-stack_api=4

This command scales the api service to four replicas. You can replace 4 with any number of replicas you want to run.

Verify that the api service has been scaled by executing the following command:

docker service ls

You should see an output that shows the updated number of replicas for the api service.

Scaling the service in Swarm mode is as simple as that! Docker Swarm will automatically distribute the workload among the available replicas, allowing you to handle increased traffic without having to restart the stack.


Docker and Docker Swarm make it easy to build, ship, and run distributed applications. Scaling services in Swarm mode is a simple process that does not require restarting the stack. By following the steps outlined in this article, you can easily install docker, configure the swarm mode, create images, create service stacks and scale up or down services to handle demand, balance cost and response time.

That’s all folks!