Your robot has tons of useful information, but using that data can be difficult. However, if you can analyze your robot data, debugging can be super simple and you can find issues very quickly.
My team has started using Shuffleboard Recordings, as well as Microsoft's Power BI Desktop, to effectively analyze our autonomous development.
Your Driver Station probably has an application called Shuffleboard.jar on it. Shuffleboard is the key to being able to analyze your data.
If you are not familiar with it, Shuffleboard is the successor to the SmartDashboard application. It can take data from NetworkTables and display it on a computer. The main reason why Shuffleboard can help us, while SmartDashboard was unable to, is because Shuffleboard can record data and encode it as a CSV (Comma Separated Values) file. The recordings are designed to be run in the Shuffleboard.jar application, but it can super easily be brought into better applications for analysis.
Why Not Just Use Shuffleboard Replay?
It is laggy.
I love Shuffleboard. It is so close to being the perfect application for robot debugging. But across every computer I have tested it on (including very powerful workstations) has not been able to have a smooth experience. It is frustrating having to fight a laggy piece of software when debugging. It also is not designed for analysis, so other software (such as Power BI) is better able to handle more complex analysis.
What is Power BI Desktop?
Power BI Desktop is a free app from Microsoft that allows for analysis of data from a variety of sources, including CSV files. It is designed for local data analysis.
It is hugely popular in business. I actually first heard of Power BI from my Uncle who is a professor of a graduate school data analytics program. Many companies use it for a wide variety of employees, not just data science employees, so learning it could prove to be very helpful in the future.
Power BI projects consist of two parts, data and visuals. The visuals can be split into separate pages (which are similar to a Powerpoint slide), and when you update your data, the visuals will automatically update. It also is easy to share, either as a project or as a PDF export, which makes collaboration easy with other people on your team. You can have a variety of data sources, but for robot debugging, CSV data is by far the best input.
Power BI Desktop can be connected to a Power BI subscription, though that is not necessary and probably not worth the investment for an FRC team. The subscription allows you to publish reports online, but for the way that my team uses Power BI, the free PDF export is more than enough.
Getting data into Power BI is fairly simple, if you follow the right process. This is the process that my team has found to work well.
At the bottom of Shuffleboard, there is a record button. Click this button before running your robot, and then click it again at the end. This generates a .sbr file that can then be converted to a CSV file.
Your recording will be exported to the configured folder.
Mine (which is based on the default), is set to:
Convert to CSV
The .sbr file is great, but Power BI does not know how to read it. Fortunately, there is a tool built into Shuffleboard that allows us to convert a sbr file to a CSV file. In the menu, go to Recordings, followed by Convert Recordings.
On this window, you can add files, specify an output folder, and then convert.
Optimize CSV (Optional)
The default generated CSV has lots of data that crowds everything. The biggest issue is that the variable names are insanely long, which makes viewing the data quickly much more difficult down the line.
I wrote a very quick, not very sophisticated, and by no means well-written script that can help us resolve this problem. Basically what it does is strips away some of the repetitive text that is not helpful to us.
This script can then be copied into the directory, and the conversion can happen with:
python ConvertFile.py recording-14.59.02.csv
Create a New Power BI Document
When you open up Power BI, a window will appear.
You can click Get Data to start a new document. Navigate to your converted CSV file, and the data will be added.
Create a Chart
Power BI Desktop has four main panes. The first one is your document, and that is where your graph is shown. The second one is filters, and that can help you narrow down data. Visualizations is where you can add graphs and specify what data you want to show. The last pane, fields, contains all of your data that you can add in.
To add data, just drag a field from the Fields pane into the respective boxes on the Visualizations tab.
For most graphs, you will want Timestamp to be on your axis, since much of the plotting is for data over time.
Values At a Point
When you have your graph, you can mouse over it to see the values of the different data at the given time.
The nice thing about Power BI is that you can easily update data.
1) Follow the steps to get your data into the formatted CSV file.
2) Go to Edit Queries -> Data Source Settings
4) Click Change Source
5) Choose a new file
6) Click Apply Changes on the main screen
7) You now have new data
Send Your Data as a PDF
In the File menu, you can export as a PDF. This makes it significantly easier to share with your team on Slack or on a shared drive.
Our Visualization Gallery
We started using the graphs because it was a huge help with autonomous. Here is a gallery of our current visualizations.
Has it helped?
Yes! We have significantly reduced our debugging time through the use of Power BI, and we are now able to collaboratively debug.
We are so much more efficient now that we can easily analyze data and don't have to deal with Shuffleboard's severe lag. The combination of data means that we can easily analyze relationships between different types of data, such as why we experience significant offset at a certain position. We have brought our getting started time (from untested code to confident path navigation) from about a week to under two worksessions, and Power BI was one of the most important parts of us being able to get working so quickly.
All of the data in the gallery above is actual data that we used for analysis (from one of our autonomous tests).
Analyzing Data on a Large Scale
When we have our PDF (which is what created those images you saw above), we immediately post it on our #programming channel on Slack. That way, we can have everyone (regardless of team role), help look for potential problems. We usually have software students and mentors looking at it, since the process was developed by the software sub-team, and usually we are debugging code. However, it is nice to easily show people on other sub-teams how the robot is reacting in the real world.
We also put it on our Google Drive for archival purposes. By putting it in both places, we can have both instantaneous and long-term analysis, and stay organized throughout the entire process. Depending on our needs, we may put the Power BI file or the CSV and SBR files on Slack and/org Google Drive, so that more intense analysis can happen. The nice thing about PDF analysis is that it can be opened virtually anywhere. We have had people discover the problems with robot code not just in our shop, but while hiking, in the car, and many other places. Especially with major problems, you may realize the solution at any time, and having easy, visual, portable data allows you to take advantage of those opportunities.
We are still very new to Power BI, so there is a lot to learn
We started using Power BI a couple of days ago, but it has completely changed how we debug, which is why I decided to write this article. Microsoft has tons of articles and tutorials for how to effectively use Power BI, so I highly recommend checking those out as a way to improve.
Two Weeks On
I have been busy and haven't gotten around to posting this for the last two weeks. So, I think I will share a little bit more about my experience with Power BI.
Overall: Very Good
Often times, using new systems can become more annoying as time goes on. This has not been my experience with Power BI. We have figured out how to effectively capture data and analyze quickly, which has dramatically reduced the amount of time that we spend debugging.
My One Annoyance
I wish that Power BI had better support for graphing different scales on the same graph. One of our graphs had to be superimposed on the other to have everything be on the same scale, and it seems to be even worse than Excel (to the best of my knowledge, you can only have one scale). This makes it much more difficult to compare data across data types (e.g. ticks/100ms vs m/s for auto).