The Sunday Blog

Protocols and parsing – Part 1

As we are all more or less experienced Nextion HMI developers, we are familiar with the default operational mode of our displays: Without any additional effort, data which comes in over the serial port is parsed and interpreted as if it were one or more instructions in Nextion language, for example in a local TouchPress or Timer event handler. Furthermore, when preceded by commands like addt or wept, we can transmit data over serial bulk-wise towards a Waveform component or towards the internal EEPROM. Finally, when using the appropriate protocol, our Nextion handles even a complete firmware update without requiring us any action on the Nextion side. But what if you want your Nextion HMI to handle data streams which do not follow the Nextion protocol by terminating every sequence by the terminator 0xFF 0xFF 0xFF? What if bulk data arrives and is not intended to land either in a Waveform component or in the EEPROM?

As always, many paths lead to Rome. You might feel inspired to write code for an external MCU (for example an Arduino or a Raspberry Pi) to receive or generate these “unsuitable” data streams, to filter and process them, and to send them out in an appropriate form towards a Nextion HMI. This approach is probably efficient and meaningful for complex data streams like CAN bus data where much information arrives at high speed but you need to display only a small subset of these.

For simpler data structures, our Nextion HMIs have a less know functionality, called active parsing mode or protocol reparse mode as my Canadian colleague Patrick prefers to name it. It allows to handle and to interpret incoming data your own way, even in a Nextion standalone setup, without external MCU. With this short series of articles, I want to shed some light on this mode, show what can be (meaningfully) done with it and where it is (probably) better to avoid it.

To buffer or not to buffer

When it comes to handling data streams, buffering is essential. That allows to let multiple bytes in without being forced to process each byte as soon as it arrives by fear of losing it. It allows thus to let a full command (in default mode) in, like n0.val=4711ÿÿÿ and to process all the 11 useful bytes in one step only, as soon as the termination sequence has been detected. To prevent accidental overflow and to allow more data or the next command to come in while a first part is being processed, the buffer should be as large as possible, but without eating up too much system RAM. Thus, the Nextion firmware developers decided to make the serial buffer 1024 bytes for the Basic, Enhanced, and Discovery Series and 4096 bytes for the Intelligent series.

A technical detail for those interested in such: The Nextion command buffer is internally realized as a ring buffer which has a set of advantages over a linear buffer. Ring or circular buffer means that reading and writing to the buffer is circular. Once the end is reached, it will automatically continue again from the beginning. That means that if the 1024 bytes are “full”, the 1025th will overwrite the first, the 1026th the second and so on. This is the first advantage: instead of blocking further receiving and losing recent bytes, only very old bytes are in danger to be overwritten before they could be processed. Second advantage: Imagine, you’d need to delete the first 11 bytes (after processing). This would mean that you had to copy the remaining 1015 bytes over one by one to be again in the “pole position”. Instead, on a ring buffer, it is sufficient to add 11 to the read pointer which means instead of moving all the bytes, you move simply the start line. One more advantage: At every moment, by subtracting the read position from the write position, you get the number of yet unprocessed bytes in the buffer. And when the write buffer reaches the read position minus one, you know that the buffer is full and that further writes will overwrite unprocessed data. That’s the moment when your Nextion sends the 0x24 0xFF 0xFF 0xFF error code. All that happens behind the scenes and has no visible impact but being quick and efficient.

The other side of the mirror

As Steve Jobs would probably have similarly said, “There is a system variable for that!”. The recmod system variable is 0 by default and at every boot. As long as it remains 0, our Nextion HMI will act as usual: Every command which comes in over serial is processed as documented in the Nextion Instruction Set (NIS). Issuing the command recmod=1, either over serial or from internal event code, will cause the Nextion HMI to parse and to interpret incoming data and let you handle this by your code. At this very moment, a whole set of new commands, functions, and operators becomes available and allows your code to take full control over the serial buffer.

The most important information, the number of bytes waiting to be processed is available in a read-only variable, usize. Since it’s incremented each time a new byte arrives, it’s in constant movement and might be already higher by 91 within a millisecond when receiving at the highest possible data rate. Thus, it is highly recommended to save it temporarily in another variable, for example sys0=usize, to process the number of bytes in sys0, and then to free up the corresponding space with udelete sys0. Because if you’d issue udelete usize instead, you risk not only to delete your processed bytes but also those which came in during processing! Beware of using the code_c command since it will delete all pending bytes in the buffer, processed or unprocessed.

The “u” space

In the previous paragraph, we have already seen that the letter “u” plays a central role in everything around the active parsing mode. Beyond the usize variable which indicates the number of pending bytes and the udelete <n> command which allows to delete the fixed number n of oldest bytes in the buffer, we need naturally some functionality to access the data in the buffer. The most elementary way to do so is the u[] array  where u[0] holds the most ancient byte and u[usize-1] the most recent byte. While this allows a highly efficient processing of byte oriented data, you’d need to write several lines of code to assemble (for example) u[0] to u[3] into a 32bit integer or (if your data stream contains text) copy the ASCII data from u[4] to u[8] into a text variable.

To ease the developers life, use the  ucopy command. It can work for numbers and for text. To copy 4 bytes starting at u[0] “as is” into the n0 number component,  use ucopy n0.val,0,4,0 where the first “0” indicates starting at u[0], the “4” means to copy 4 bytes, and the second “0” means that we copy the 4 bytes “as is” into the 32bit integer without shifting. Now imagine a situation where we know that we have a 2byte (16bit) number like 0x7FFF. To copy this into n0 “as is”, we’d use ucopy n0.val,0,2,0 after which we’d have 32767 (0x7FF) in n0. But the ucopy command is more mighty – the last parameter allows us to set the start position inside the target variable which corresponds to shift the data byte-wise. Thus, ucopy n0.val,0,2,1 sets n0.val to 0x7FFF00 or 8388352. Although the use cases for such byte shifting are rather rare, it’s good to know that it’s possible. For text, the usage is similar: ucopy t0.txt,4,5,0 will copy 5 character bytes starting from u[4] into t0 without shifting. Modifying the last parameter will again allow to right-shift the copied character chain.

One more thing

Once we are in the active parsing mode, how to switch back to the classic command processing mode? Since for the moment, the normal command parsing isn’t active, sending recmod=0ÿÿÿ over serial will have no effect. But since the internal command processing remains active, a screen “Exit” button with recmod=0 in the TouchPress event will do the trick. And if you absolutely need to exit the active parsing mode from outside, there is another trick: send the string “DRAKJHSUYDGBNCJHGJKSHBDN” (all capitals) followed by the classic terminator 0xFF 0xFF 0xFF over serial.

… interesting information, but in practice… ?!?

You might be interested in knowing what all this is good for, especially after this rather lengthy and heavily theoretical article. Wait a week for part 2 where we’ll use this knowledge to build an example project which transforms a Nextion HMI into a MIDI protocol sniffer and decoder!

Happy Nextioning!