RainbowOS — Building a Touch-Based Multi-App Operating Environment on ESP32-S3

Building My Own Operating Environment on a Microcontroller

The idea behind RainbowOS started with a simple question:

How far could I push a relatively inexpensive ESP32 microcontroller beyond its usual role as a sensor controller or IoT device?

ESP32 boards are commonly used for automation, wireless communication, sensor monitoring and small displays. However, modern microcontrollers have become powerful enough to support considerably more sophisticated graphical interfaces.

I wanted to explore whether I could turn an ESP32-S3 and a square touchscreen into something resembling a miniature smart-device operating environment. And the idea was basically inspired by my friend Md Ali Haider Ontor.

Not a full desktop operating system running Linux, but a custom firmware platform capable of managing several independent applications through a unified graphical interface.

My goal was to create a device where users could launch apps, swipe between screens, access connectivity settings, view information and interact with animated graphics without needing a keyboard, mouse or external computer.

This became RainbowOS.

Rather than developing isolated programs that each took complete control of the display, I began combining them into a shared application environment.

The project has gradually evolved from a basic Hello World display test into a multi-application embedded interface.

Starting with the Hardware

RainbowOS is being developed on the Waveshare ESP32-S3-Touch-LCD-4B development board.

The device combines an ESP32-S3 microcontroller, a 480×480 RGB LCD and a GT911 capacitive touchscreen.

The hardware provides several important capabilities:

• A dual-core ESP32-S3 processor running at up to 240 MHz.

• External PSRAM for graphics buffers and application data.

• A 480×480-pixel colour display.

• Capacitive multitouch input.

• Integrated Wi-Fi and Bluetooth Low Energy connectivity.

• Flash storage for firmware, graphical assets and saved application data.

The square display became an interesting design constraint.

Unlike a smartphone, the screen offers limited vertical and horizontal space. A normal desktop-style interface would waste too much room, while a fixed single-purpose dashboard would limit future expansion.

RainbowOS therefore required an interface designed specifically for a compact square display.

The development environment uses the Arduino framework with the ESP32 board package and the Arduino_GFX graphics library.

The LCD is driven through the ESP32-S3 RGB peripheral using the ST7701 display controller, while the touchscreen communicates over I²C.

An I/O expander also participates in the display initialization sequence.

One important decision was to preserve the original, verified display configuration throughout development.

This reduced the risk of introducing hardware initialization problems while the graphical interface became more complex.

From Hello World to a Graphical Interface

The first successful milestone was straightforward: initializing the LCD and displaying text.

The original program showed Hello World and drew random text at different positions.

Although simple, this established the essential hardware communication path.

The next challenge was much more significant.

A static message is easy to render. An animated interface must repeatedly update the screen while also reading touch events, maintaining time, processing application logic and handling network connections.

On an RGB LCD, directly clearing and redrawing visible graphics can produce flickering or screen tearing.

This became particularly noticeable when I started developing an analog clock.

The first implementations could display the clock correctly, but the moving hands visibly blinked during updates.

Solving that problem led to one of the most important architectural improvements in RainbowOS: separating graphical composition from visible screen updates.

Developing the Rainbow Analog Clock

The first application was a colourful analog clock inspired by modern smartwatch interfaces.

I wanted it to feel visually dynamic rather than display a conventional static dial.

The clock includes a multicoloured circular dial, moving hour and minute hands, a continuously sweeping second hand, digital time information and a date display.

The animation is calculated from elapsed time rather than simply advancing the second hand once per second.

This allows the hands to move progressively between positions.

However, smooth movement requires more than calculating accurate angles.

The graphics must also be updated efficiently.

Initially, redrawing the clock produced visible blinking because large portions of the framebuffer were being replaced while the display was scanning the same memory.

To improve the rendering process, the clock was redesigned around off-screen drawing buffers and selective updates.

The static dial can be stored separately from the moving elements.

Instead of redrawing the entire background on every frame, the renderer restores and recomposes only the regions affected by the moving hands.

This reduces unnecessary drawing operations and helps maintain visual continuity.

The clock targets a smooth animation rate while using NTP to synchronize with real-world time when Wi-Fi is available.

An optional buzzer can also produce a traditional ticking sound.

This application became the starting point for understanding the relationship between animation quality, framebuffer memory and display refresh behaviour on an embedded device.

Building Pixel Hopper — A Game Running Inside RainbowOS

The second application was Pixel Hopper PLUS, a side-scrolling platform game developed specifically for the touchscreen.

I wanted RainbowOS to demonstrate that the ESP32-S3 could do more than display information.

It should also support interactive applications requiring continuous input, animation and game logic.

Pixel Hopper contains a scrolling game world with platforms, collectible coins, enemies, shooting mechanics, jumping, lives, scoring and a finish line.

The game includes animated effects such as particles, hit effects and victory fireworks.

A major technical decision was to separate the game's physics simulation from its graphics rendering.

The physics engine uses a fixed timestep targeting 60 updates per second.

Movement, gravity, jumping, enemy behaviour and projectile calculations therefore do not depend directly on how quickly the display finishes drawing each frame.

This makes the gameplay more consistent when rendering time varies.

The game also uses a camera system that follows the player across a world larger than the physical display.

Multitouch Controls

A touchscreen platform game requires a different input system from a game controlled by physical buttons.

The GT911 touch controller can report multiple independent contacts.

Pixel Hopper uses these contacts to support simultaneous actions, such as moving and jumping or moving and firing.

Each finger is tracked separately using its contact identifier.

Additional mechanics include jump buffering and a short grace period after leaving a platform, helping make the controls feel more responsive.

High scores are saved using the ESP32 Preferences storage system.

When the game was integrated into RainbowOS, its application state was retained while switching to another app.

This allows the operating environment to manage the game without necessarily restarting the current run.

Designing the RainbowOS Home Screen

Once the clock and game were working as separate programs, I wanted a common home screen from which both could be launched.

This was the point where RainbowOS began evolving into an application platform rather than a collection of standalone sketches.

The launcher uses a dark visual theme, colourful application cards and a personalized greeting:

Hello Showaib

Each application has a dedicated icon, name and visual identity.

A centralized application registry connects each launcher card to its corresponding application.

The launcher was initially designed for two applications, but this quickly introduced a scalability problem.

If every new app required another fixed button, the screen would eventually become overcrowded.

To solve this, I introduced a horizontally scrollable application carousel.

Users can swipe left and right to browse the available apps.

A horizontal scrollbar provides a second way to navigate the collection.

The launcher distinguishes between a tap and a horizontal drag, preventing an application from opening accidentally during scrolling.

The scrollable design allows new applications to be registered without redesigning the entire home screen.

Creating a Swipe-Down Control Center

One of the most important elements of RainbowOS is its system-level Control Center.

I wanted connectivity and time information to remain accessible without requiring users to leave their current application.

The interface includes a status bar displaying the current time, date, Wi-Fi status, Bluetooth status and navigation controls.

A downward swipe starting at the top edge opens an expanded Control Center.

The panel provides larger Wi-Fi and Bluetooth controls, connection-state indicators, date and time information, and navigation back to the home screen.

The Control Center is implemented as an overlay managed by the system rather than by each individual application.

This means new applications do not need to implement their own connectivity interface.

Solving Touch Gesture Conflicts

Getting the gesture system working reliably was one of the more challenging parts of development.

An early implementation caused the control panel to open when the user touched almost anywhere on the display.

Another revision became too restrictive and prevented normal touch interaction.

These bugs exposed an important issue: touch coordinates, gesture recognition and application input must be handled as separate concerns.

The input system was revised to track the physical touch coordinates correctly, identify when a contact begins and distinguish between tapping, dragging and swiping.

The Control Center now requires a deliberate downward gesture beginning near the top edge.

It also supports an upward dismissal gesture and a dedicated Close button.

The working touchscreen foundation was validated on the actual device before expanding the system with additional applications.

This debugging process was an important lesson in building user interfaces around asynchronous hardware input.

Managing Graphics and Memory Efficiently

Rendering a graphical environment on an ESP32-S3 requires careful memory management.

A single 480×480 RGB565 framebuffer occupies:

480 × 480 × 2 bytes = 460,800 bytes

That is approximately 450 KiB for one complete screen image.

Multiple graphics buffers, cached backgrounds and application surfaces therefore consume substantial memory.

RainbowOS uses external PSRAM to accommodate these larger graphical structures.

The rendering architecture uses different approaches depending on the application's requirements.

The clock benefits from selective updates around its moving hands.

Pixel Hopper uses a dedicated playfield canvas while keeping its touchscreen controls in a separate area.

The launcher redraws its scrolling card region without unnecessarily repainting the entire interface.

The Control Center prepares its contents in a separate canvas and reveals or restores display regions during animation.

These techniques aim to reduce flickering, unnecessary memory transfers and visible rendering artifacts.

The project also uses a larger firmware application partition to accommodate the expanding codebase and graphical assets.

This became necessary as networking, Bluetooth functionality and additional applications increased firmware size.

Adding Skycast — A Live Weather Application

The third application introduced internet-based data into RainbowOS.

Skycast is a weather dashboard powered by the Open-Meteo forecast API.

The application retrieves weather information for a configured geographic location, initially London.

It provides current temperature, apparent temperature, humidity, wind speed, weather conditions, a five-hour forecast and a four-day outlook.

Weather conditions are represented using graphics drawn directly with Arduino_GFX primitives.

This avoids requiring a large collection of external image files.

Separating Network Requests from the Interface

An important challenge with a connected application is network latency.

A weather request can take considerably longer than a graphics frame.

If an HTTPS request runs directly inside the main interface loop, the touchscreen and animations may appear frozen while the system waits for a response.

To reduce this problem, the integrated Skycast version uses a separate FreeRTOS worker for network requests and JSON processing.

The background worker retrieves and parses weather data.

The graphical application then reads prepared forecast information and updates the display.

This separation allows network operations and interface rendering to be managed more independently.

The application caches successful weather results and schedules periodic refreshes.

The system also distinguishes between disconnected, loading and failed-request conditions.

This application demonstrates how RainbowOS can connect external services to an embedded graphical interface without allowing normal network delays to dominate the user experience.

Adding iPhone Notifications Through Bluetooth

The fourth application extends RainbowOS into smartphone connectivity.

The iPhone Notification Dashboard is designed to receive notification information from a nearby iPhone using Bluetooth Low Energy.

The integration uses Apple's Notification Center Service, commonly known as ANCS.

Unlike a typical web notification system, ANCS allows a compatible Bluetooth accessory to receive notification events and request selected information from an iPhone.

The application is designed to display details such as the originating app, notification title, message preview and connection status.

It also supports small RGB565 emoji sprites for common emoji characters.

The notification dashboard maintains a limited in-memory history, allowing recent notifications to appear as cards.

Integrating a Shared Bluetooth System

The original RainbowOS Bluetooth controls and the standalone notification application used different Bluetooth software stacks.

Combining both implementations directly would create conflicts and unnecessarily complicate memory and radio management.

The newer architecture therefore uses NimBLE-Arduino as a shared BLE implementation for the operating environment and the iPhone notification application.

The system-level Bluetooth control can enable or disable the BLE service.

The notification application uses the same connection manager and ANCS event-processing logic.

BLE callbacks queue incoming events for processing outside the callback context, avoiding heavy graphics operations inside Bluetooth event handlers.

The notification application also uses encrypted bonding, although the current prototype's Just Works pairing mode does not provide man-in-the-middle protection.

Notification content remains in volatile application memory and is not intentionally published through the weather or Wi-Fi interface.

This fourth application is implemented in the latest development branch, but reliable iPhone pairing, ANCS service discovery and notification delivery still require physical-device validation.

Making the System Interface Compatible with Every App

As more applications were added, another interface problem became important.

A persistent status bar can hide application content if each program assumes it owns the entire 480×480 screen.

This is particularly problematic when integrating standalone applications that were originally designed to render from the first pixel at the top.

RainbowOS addresses this through application-specific layout management.

For the iPhone notification application, the system reserves the top 48 pixels for the RainbowOS status bar.

The dashboard places its title, connection indicators, notification cards and footer within the remaining usable area.

Other applications use their own rendering arrangements so that system navigation and application-specific controls can coexist.

This is an important part of turning separate programs into a shared operating environment.

Applications must cooperate with the system interface rather than independently assuming complete control of the display.

Designing for Future Applications

One of the main objectives of RainbowOS is extensibility.

The intention is not to stop after four applications.

The firmware is organized around application identifiers, shared device services and application-specific drawing and update functions.

The common system layer is responsible for navigation, touch gestures, connectivity controls and display coordination.

Individual applications are responsible for their own logic and graphics.

This architecture makes it easier to add another application without rewriting the complete home screen.

The same approach could eventually support utilities such as calendars, sensor dashboards, music controls, home automation panels, system monitors and other interactive tools.

However, additional applications will also create new engineering challenges.

Firmware size, PSRAM usage, radio scheduling, responsiveness and application lifecycle management will become increasingly important as the project grows.

Current Development Progress

RainbowOS has developed through several firmware iterations.

The project began with successful hardware display initialization and a standalone analog clock.

A platform game was then integrated into a basic launcher.

Subsequent revisions addressed touchscreen gesture problems, improved the interface's navigation and added a swipe-down Control Center.

The working v1.2 touchscreen foundation was confirmed on the physical device.

Later versions introduced horizontal application scrolling, a connected weather dashboard and an iPhone notification application.

The current development codebase contains four registered apps:

1. Rainbow Clock

2. Pixel Hopper PLUS

3. Skycast Weather

4. iPhone Notification Dashboard

The code also includes a shared launcher, Control Center, Wi-Fi time synchronization and common Bluetooth management.

Not all features have reached the same validation stage.

The core navigation and touch system have been tested on the device. The latest four-application integration, especially iPhone pairing and notification delivery, still needs complete hardware-level testing.

This distinction is important because RainbowOS is an actively developing embedded prototype rather than a finished commercial operating system.

Challenges and Lessons Learned

RainbowOS has introduced several problems that are less noticeable when developing software for a desktop computer.

Limited graphics memory makes efficient frame composition essential.

An RGB display can reveal tearing even when the application appears to render complete frames.

Touchscreen communication must be interpreted carefully to distinguish simultaneous contacts, taps and gestures.

Network requests must be managed so they do not block the user interface.

Bluetooth services must share the same radio and software architecture.

Applications must also respect common display regions and system-level navigation.

Perhaps the most important lesson has been that integrating several working programs is not the same as building a reliable application platform.

Each standalone application can function correctly on its own while introducing new problems when combined with other components.

RainbowOS is therefore as much an experiment in software architecture and resource management as it is a graphical interface project.

Next Stage of Development

The next stage will focus on reliability and making the application system more modular.

Planned improvements include more extensive hardware testing, refining animations, improving app transitions and reducing unnecessary framebuffer transfers.

Other priorities include more robust iPhone connectivity, better application lifecycle management, clearer system diagnostics and continued optimization of memory usage.

As more applications are added, I also want to explore organizing them into independent source modules rather than maintaining an increasingly large single Arduino sketch.

This would make the codebase easier to maintain, test and extend.

Longer term, RainbowOS could become a reusable framework for small touchscreen devices powered by ESP32-class microcontrollers.

The same software foundation could potentially support different applications and connected hardware with comparatively small changes to the user interface.

Long-Term Goal

The long-term objective of RainbowOS is to explore how much functionality and visual quality can be achieved from an affordable microcontroller-based device.

I want to create a small but practical embedded application environment that feels responsive, visually consistent and straightforward to extend.

The project is intended to demonstrate several engineering concepts together:

• Embedded graphics and framebuffer optimization.

• Real-time input and application-state management.

• Touchscreen gesture recognition and multitouch interaction.

• Wireless communication using Wi-Fi and Bluetooth Low Energy.

• Internet API integration and asynchronous data processing.

• Resource-conscious firmware architecture.

• Modular application development on constrained hardware.

RainbowOS is not intended to compete directly with Android or iOS.

Instead, it explores a different question:

How much of the experience of a modern smart device can be recreated on a microcontroller when its software is designed around the hardware's actual capabilities?

By combining embedded C++, custom graphics, interactive applications and wireless connectivity, RainbowOS is gradually transforming a simple ESP32-S3 display board into a personalized, expandable smart-device platform.

And as development continues, every new application introduces another opportunity to push the capabilities of the hardware further.