Welcome back to the uORB Explained Part 2! In Part 1, we covered the basic uORB concepts like messages, topics and modules, and how to use them. In this post, we will uncover how uORB actually works, specifically, the topics in uORB. Be aware that we will go deep into the codebase and it helps to have some experience in C/C++ programming to understand, but I will try my best to describe it on a basic level!
Agenda
We are going to cover the following topics :
- What is a uORB Topic, really?
- How are uORB topics created? (How the “# TOPIC” comment in the .msg file turns into uORB topics)
What is a uORB Topic, really?
The most basic and the most essential concept in uORB is a ‘topic’. Topics are at the heart of all the interfaces to uORB. So how are the topics actually defined and used?
The essence of uORB
Since topic is the most fundamental concept in uORB, let’s first learn more about what uORB actually is.
Here’s one secret that no one told you:
“uORB stands for ‘u (Micro) Object Request Broker’”
Let’s break that down.
uORB works like a librarian who keeps track of information, while communicating with people. Customers have a notion of ‘Topic’ as the book they are writing / reading from. But only the librarian can actually touch, write and read the book. Therefore customers never actually get to interact with the physical book itself.
Publishers request the librarian to write down the data into a book, and Subscribers can then request that data. All through the help of the librarian 🙂
We can generalize this as “Object Request Broker (ORB)” since a book is just an object in general terms and customers will be requesting the data to either be published or subscribed. You can learn more about the concept of ORB in this Wikipedia article.
Because PX4 is running on an embedded system like a micro-controller on a flight control board, the ORB needs to be small and lightweight. That’s why it’s named u-ORB, since it’s a micro (symbolized as u) sized Object Request Broker.
In fact the uORB was an essential component that was designed into the PX4 system architecture since the early days. If you are curious, check out this paper written by the founder of PX4 : Link is Here.
ORB_ID( ) Macro
We usually use the ORB_ID(topic_name) macro to refer to the topic (If you don’t know about this Macro, I strongly suggest you to read the Part 1!) Modules can publish, subscribe and interface freely to a topic using this handle. It’s like a ‘title’ of a book for the customers!
In reality, it’s just a pointer to the topic’s metadata instance. This is defined in the platforms/common/uORB/uORB.h.
/**
* Generates a pointer to the uORB metadata structure for
* a given topic.
*
* The topic must have been declared previously in scope
* with ORB_DECLARE().
*
* @param _name The name of the topic.
*/
#define ORB_ID(_name) &__orb_##_name
For example, whenever you refer to “ORB_ID(pasta_cook)”, you would be accessing the “__orb_pasta_cook” metadata instance. So what is this metadata struct?
ORB Metadata struct
The metadata, which includes information about the specific topic is like this :
/**
* Object metadata.
*/
struct orb_metadata {
const char *o_name; /**< unique object name */
const uint16_t o_size; /**< object size */
const uint16_t o_size_no_padding; /**< object size w/o padding at the end (for logger) */
const char *o_fields; /**< semicolon separated list of fields (with type) */
uint8_t o_id; /**< ORB_ID enum */
};
https://github.com/PX4/PX4-Autopilot/blob/bb2ea574aa82f8c18bf6eeb0a37a3d5a7cd497a0/platforms/common/uORB/uORB.h#L46-L55
Note that it doesn’t actually include any data (physical book) itself. This is rather a catalog that the librarian and customers can use to refer to a specific book. It includes a lot of information, but what we are most interested in for now is the o_id, which is the unique Identifier of that topic.
Keep in mind that the ORB_ID identifier enum (number) is a totally different concept from the ORB_ID macro (a pointer)!\
You can think of ‘o_id’ as the label number that the librarian uses to know which book you are talking about. This is mostly hidden from the user, since if you did use the ID, it would be like asking the librarian to bring “803.11b-21” instead of “Catcher in the Rye”. But users don’t need to know that, which is why the ORB_ID( ) macro exists to provide the layer of abstraction for the user.
Additionally, the ‘o_name’ is where the name of the topic is stored, like “pasta_cook”. And o_size, o_size_no_padding, o_fields contain more information about the topic, which will be discussed more on Part 3 when we discuss the uLog logging system!
How are uORB Topics generated?
You probably noticed that we never actually explained how the Metadata structs are initialized, that is : “Where are instances like __orb_pasta_cook defined?”
If you remember from Part 1, we created .msg files to define a message with internal fields and custom topic names. And they were somehow magically translated into the header files that we could use in the code. In fact, this is where the metadata instances are initialized as well!
.msg file to .h and .cpp conversion Magic
When we build PX4, all the uORB topic headers (.h) and source files (.cpp) are generated in the build folder in ‘uORB/topics/’ and ‘msg/topics_sources/’ respectively. How does that happen?
There is a “CMakeLists.txt” file in each folder that defines what to do during the build. The uORB messages are a great example for that, since we build a ton of files out of them! Since they are in the “msg/” folder, we can check the “msg/CMakeLists.txt” to uncover the magic:
# Generate uORB headers
add_custom_command(OUTPUT ${uorb_headers}
COMMAND ${PYTHON_EXECUTABLE} tools/px_generate_uorb_topic_files.py
--headers
-f ${msg_files}
-i ${CMAKE_CURRENT_SOURCE_DIR}
-o ${msg_out_path}
-e templates/uorb
-t ${CMAKE_CURRENT_BINARY_DIR}/tmp/headers
-q
${added_arguments}
DEPENDS
${msg_files}
templates/uorb/msg.h.em
templates/uorb/uORBTopics.hpp.em
tools/px_generate_uorb_topic_files.py
tools/px_generate_uorb_topic_helper.py
COMMENT "Generating uORB topic headers"
WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}
VERBATIM
)
add_custom_target(uorb_headers DEPENDS ${uorb_headers})
This is an example of how message header files like pasta_information.h get generated. The python script ‘px_generate_uorb_topic_files.py’ reads the message definitions (.msg) and formats it according to the predefined EmPy template.
The script does a lot of things, including multi topic registration. Remember the custom topics declared like this at the end of the .msg file in Part 1 for the pasta_information message?
# TOPICS pasta_cook pasta_order
The python script then parses this custom topic line from the pasta_information.msg file like this:
def get_topics(filename, msg_name):
"""
Get TOPICS names from a "# TOPICS" line. If there are no multi topics defined,
set topic name same as the message name, since the user doesn't expect any new
custom topic names.
"""
ofile = open(filename, 'r')
text = ofile.read()
result = []
for each_line in text.split('\n'):
if each_line.startswith(TOPICS_TOKEN):
topic_names_str = each_line.strip()
topic_names_str = topic_names_str.replace(TOPICS_TOKEN, "")
result.extend(topic_names_str.split(" "))
ofile.close()
if len(result) == 0:
result.append(msg_name)
return result
This topic list in python script then gets used in the EmPy template to take care of multi topic registration. Let’s go more in depth into the generation of the header and learn how these custom topics are handled.
Message header (.h) generation
In msg/templates/uorb/msg.h.em, we find these interesting lines where the custom topics are referenced.
/* register this as object request broker structure */
@[for multi_topic in topics]@
ORB_DECLARE(@multi_topic);
@[end for]
This part essentially includes the topic registration in the header file pasta_information.h like this:
/* register this as object request broker structure */
ORB_DECLARE(pasta_cook);
ORB_DECLARE(pasta_order);
build/px4_sitl_default/uORB/topics/pasta_information.h
So what does ORB_DECLARE do?
/**
* Declare (prototype) the uORB metadata for a topic (used by code generators).
*
* @param _name The name of the topic.
*/
#if defined(__cplusplus)
# define ORB_DECLARE(_name) extern "C" const struct orb_metadata __orb_##_name __EXPORT
It basically tells the compiler that the orb_metadata instances for the topics exist; it would therefore ‘declare’ __orb_pasta_cook and __orb_pasta_order. This allows you to use the ORB_ID( ) macro in the code (which basically references the metadata instances), since the compiler already knows that these instances exist.
Remember that this is a small part of the header file! You can discover the full header template here.
Message source (.cpp) generation
But the header only declares the orb_metadata instance, which means the instance doesn’t actually exist yet. This is like having a knowledge of the catalog of the books in the library, but not having a physical catalog. This is where the source file comes in.
Source file generation is also defined in the msg/CMakeLists.txt. It uses the msg.cpp.em template, which has a slightly different format from the header template.
@# join all msg files in one line e.g: "float[3] position;float[3] velocity;bool armed"
@# This is used for the logger
constexpr char __orb_@(topic_name)_fields[] = "@( ";".join(topic_fields) );";
@[for multi_topic in topics]@
ORB_DEFINE(@multi_topic, struct @uorb_struct, @(struct_size-padding_end_size), __orb_@(topic_name)_fields, static_cast<uint8_t>(ORB_ID::@multi_topic));
@[end for]
First we notice that it has the ‘fields’ strings, and a bunch of ORB_DEFINE macro calls for the topics in the list. This ORB_DEFINE macro actually creates the orb_metadata struct and places it in the memory!
This is what the resulting pasta_information.cpp output from my build looks like :
constexpr char __orb_pasta_information_fields[] = "\x89 timestamp;\x8a pasta_temperature;\x87 customer_table_id;\x86 menu_name;\x86 cooked_texture;\x86 pasta_type;\x86[7] _padding0;";
ORB_DEFINE(pasta_cook, struct pasta_information_s, 17, __orb_pasta_information_fields, static_cast<uint8_t>(ORB_ID::pasta_cook));
ORB_DEFINE(pasta_order, struct pasta_information_s, 17, __orb_pasta_information_fields, static_cast<uint8_t>(ORB_ID::pasta_order));
build/px4_sitl_default/msg/topics_sources/pasta_information.cpp
There we find the definitions for the orb_metadata instances for the topics we wanted to register. It includes all the information the struct needs like the name of the topic, struct for the message definition, ORB_ID enum and so on.
By the way, the fields string just lists the names of the inner fields of pasta_information. This will be very relevant in Part 3 of the uORB Blog series where we will talk about the uLog data logging system!
But first let’s find out what ORB_DEFINE( ) macro does:
/**
* Define (instantiate) the uORB metadata for a topic.
*
* The uORB metadata is used to help ensure that updates and
* copies are accessing the right data.
*
* Note that there must be no more than one instance of this macro
* for each topic.
*
* @param _name The name of the topic.
* @param _struct The structure the topic provides.
* @param _size_no_padding Struct size w/o padding at the end
* @param _fields All fields in a semicolon separated list e.g: "float[3] position;bool armed"
* @param _orb_id_enum ORB ID enum e.g.: ORB_ID::vehicle_status
*/
#define ORB_DEFINE(_name, _struct, _size_no_padding, _fields, _orb_id_enum) \
const struct orb_metadata __orb_##_name = { \
#_name, \
sizeof(_struct), \
_size_no_padding, \
_fields, \
_orb_id_enum \
}; struct hack
It instantiates the uORB metadata as expected. If we apply this macro in the source file, the pasta_cook topic definition would then look like this (I added comments):
const struct orb_metadata __orb_pasta_cook = {
"Pasta_cook", /**< unique object name */
sizeof(struct pasta_information_s), /**< object size */
17, /**< object size w/o padding at the end (for logger) */
"\x89 timestamp;\x8a pasta_temperature;\x87 customer_table_id;\x86 menu_name;\x86 cooked_texture;\x86 pasta_type;\x86[7] _padding0;", /**< semicolon separated list of fields (with type) */
static_cast<uint8_t>(ORB_ID::pasta_cook) /**< ORB_ID enum */
};
Voila, our topic metadata has been instantiated. Now this struct will be the place where the ORB_ID(pasta_cook) will refer to!
Extra : ORB_ID enum
I thought it would help to include how the ORB_ID enum (the librarian’s ID for the books : “803.11b-21”) gets generated as well. They get generated in a separate file along with message header and sources, called “uORBTopics.hpp”. The part where this is generated in the python script is here:
The script also uses the separate templates to generate uORBTopics.hpp and uORBTopics.cpp. The ORB_ID enum definition then lies in the generated header file:
Feel free to check out the uORBTopics.hpp.em template to help your understanding here.
static constexpr size_t ORB_TOPICS_COUNT{216};
static constexpr size_t orb_topics_count() { return ORB_TOPICS_COUNT; }
/*
* Returns array of topics metadata
*/
extern const struct orb_metadata *const *orb_get_topics() __EXPORT;
enum class ORB_ID : uint8_t {
action_request = 0,
// ...
pasta_cook = 118,
pasta_order = 119,
// ...
yaw_estimator_status = 215,
INVALID
};
build/px4_sitl_default/uORB/topics/uORBTopics.hpp
It’s basically an indexed list of all the 216 topics used in this build. Note that when you build your own target, you may have different ORB_TOPICS_COUNT and topics list. This is because uORB Topics (.msg files) are constantly getting modified in the codebase! You can check the full list of edit history on uORB messages / topics in the msg/ folder!
One last thing: the function ‘orb_get_topics()’ declared in the same header returns a list of all the metadata struct pointers to the uORB topics. This is especially helpful for the functions that need to find metadata using only the topic’s name. Forr us developers, we can easily call “ORB_ID(pasta_cook)” to get the metadata for the topic ‘pasta_cook’, but once the target is built, the software itself can’t use macros like ORB_ID since it’s used only during compile time.
Therefore when you use the ‘listener’ module that listens to the specific topic, PX4 first reads in what you typed : (e.g. Your command will be : listener pasta_cook), parses to extract the name ‘pasta_cook’ as the topic you are interested in, then goes through the full list of metadata to find the one with the ‘o_name’ property equal to ‘pasta_cook’.
Not so simple, right? But this will be utilized in the uLog system discussed in Part 3 as well, so keep it in mind!
Feel free to visit this User Doc to learn more about listener module and other uORB debugging methods!
Thank you
With this post, we have covered all about uORB topics. In Part 3, we will cover:
- What happens when you publish a uORB topic?
- What happens when you subscribe to an uORB topic?
- How are uORB topics logged? (.ulg Files)
For more Tips & Tutorials
This post is part of the ongoing series explaining various confusing/complicated concepts in PX4. If you haven’t already, check out our previous blog posts!
Have any questions or feedback for the team?
Did this blog post explain the concept clearly? Did I miss anything important? Do you have suggestions on how to improve this blog post? Or have any suggested future topics?
If so, please give us your valuable feedback in the form below. I’m eager to read your feedback and thoughts on this blog post.
Meet you next time!
-Junwoo
PS: You can find me on Dronecode Discord Server (click to get an invite)