The Sunday Blog: Multimedia on Nextion

A first stage piano attempt

Didn’t I tell you 3 weeks ago about the new small Nextion Intelligent HMI displays? Guess what, I’ve fallen in love with them and I decided to have a look into how far one can push them, multimedia-wise. Finally, I discovered that my initial approach was consuming too much RAM (ha, I got the 512kB more than full), so I had to explore a different path. Today, it is still monophonic, which means that it can for the moment play only one note at a time. We’ll add polyphony and multi-touch next week.

But before, let’s insert a short paragraph about some…

Nextion News

Active Nextioners are already seen it: There was again a Nextion Editor update, we are now at v1.62.2.
This version fixes two regressions in the very advanced programming domain. Both affected very specific and rarely used functionalities in protocol reparse mode: indirect addressing using the u[] array, and CRC calculation from the serial input buffer. Thanks to attentive Nextion Forum users, these problems were documented, verified, reported, and rapidly fixed since these functions had been working correctly, before.

As most electronics manufacturers, Nextion and its suppliers are suffering from Covid19 late effects, and it is actually not easy to maintain the availability of all display formats. That’s for example why there is a temporary shortage in 3.2″ displays not only for Nextion, where Standard (T) and Enhanced (K) models are affected, just because a supplier is out of raw materials for LCD glass substrates. My Chinese colleagues have informed all registered customers by email and have asked them to communicate their needs for the upcoming months, so that the remaining stock can be fairly distributed.

Other recurring temporary but worldwide shortages of components challenge the Nextion staff to find creative ways to maintain deliverability of the other display sizes. Although the situation remains very thrilling, many things move behind the scenes, stay tuned!

Now to our Multimedia project

At the beginning, there is the GUI design, and I can tell you from my experience, that designing a keyboard is not at all trivial. In the upper part, where black and white keys are side by side, 12 equally spaced areas for the 12 halftones of an octave have to fit pixel-perfect with the lower part, where just 7 wider, but also equally spaced parts of the white keys have to be drawn. I decided to go with the keys on a full size page background image, so that I had to do all the maths only once when setting the guid lines in my graphics program (Gimp 2.10). After creating and importing it as a picture resource, the fiddly part of the artwork was almost done. For this first approach, I decided then to lay transparent hotspots first over all the white keys, and then, above, over all the black keys. This way, the hotspots for the white keys (m0 to m13) get the IDs 1 to 14 (since the page itself has always the ID 0), and the hotspots for the black keys (m14 to m23) get the IDs 15 to 24. It’s just important to remember that due to this visual order, we can’t address our audio samples not in the usual chromatic way, but (if we do not want to write individual event code for each.single.hotspot) to order these in the audio pane in the same way: The samples for the white keys (F3, G3, A3, …D5, E5),first, and those for the black keys (Gb3, Ab3,Bb3, …Db5, Eb5) afterwards.

The audio samples

Thanks to the great work of the University of Iowa, I was able to find and download very high quality grand piano samples here. Then, using Audacity, I converted the aiff files to the wav format and truncated them from ~40s down to 2.5s. 40 seconds of sustain are very nice, but that makes really no sense to fill up the Nextion’s flash memory with that, especially not in the monophonic version. After this treatment, I could upload the wav resources in the audio pane.

The code

There isn’t much code. There is one single touchCap component, which captures and handles the touch press and release events for all the 24 hotspots. For the touch press event determining the ID of the just pressed hotspot, calculating the ID of the corresponding Audio resource by subtracting 1 (hotspot IDs go from 1 to 24, audio IDs from 0 to 23), attributing the result to the .vid attribute of the audio component and start playing the note by setting its .en attribute to 1 (Start).

sys0=tc0.val-1
wav0.vid=sys0
wav0.en=1

The release event is still simpler: Just set the audio component’s .en attribute to 0 (Stop).

wav0.en=0

Just for convenience, I added a line to the page postInitialize event to set the volume, because the default value (60) seemed not loud enough to me.

volume=90

That’s all. 5 lines of code for 24 keys!

Next time, we’ll look into improving this somewhat, by adding multi-touch and polyphony. Ideally this would be solved with as many audio components (24) as there are keys (that’s the approach which I tried first), but an audio component eats so much RAM that you can’t place more than a handful of these on a page. So, we’ll have to develop an algorithm to re-attribute these dynamically.

BTW: My readers’ remarks and suggestions are always welcome! To let me hear your voice, just sign up to the Nextion forums and start a topic.

Ah, and before I forget it, here is the HMI project file: stage_piano.HMI

Because of the numerous audio samples, this .hmi file is somewhat big. Make sure to select a high baud rate (512000 or 921600) before you start the upload to your NX4827P043-011C-Y. If you don’t have the boxed version, make sure to connect a 16Ohm speaker to hear your piano playing.

Happy Nextioning!