GSoC 2019 – Continuing Working on Web App

So it is August and frankly speaking I am a bit slacking as I am back to my home country and it is pretty hard to focus on the work when there is great summer weather, friends and bicycle in place šŸ˜›. In this post, I will talk about new features of the Web App and at the end of the week, I will publish a final (šŸ˜¢) GSoC-related post, describing everything I’ve accomplished so far.

As I have already mentioned August was not the most productive month of mine. Apart from constant distractions I also had to have JS and HTML refreshers. Thus, I have finished 2 major components that lay a foundation for future work on Web App.

Interactive Task List

File Types

First of all – PMR-Web now supports all types of files:

  1. Videos – .mp4, .avi etc.
  2. Images – either one image or a .zip archive with a pack of images
  3. YouTube video – video will be downloaded in the background using PyTube library and AJAX technology.
Downloading YT videos has never been so easy

Task List

Do you remember times when you had to hit F5 to check whether your GF sent you a message? Nowadays, your browser is fetching updates from the server in the background using JavaScript and AJAX and that is what I try to do basically everywhere in PMR-Web. With new Task List, you don’t need to press F5 to check whether the processing of your file has finished or not.

Using JavaScript and AJAX Task List is fetching the data from the server every time there is some user event (you added a new file, hit the next page, pressed refresh task list button).

Checking the progress of a task execution

Also, task list supports pagination as you can see from the GIF above. It took me a while to figure out how to handle it – basically, every time you change a page, your browser sends a request to a server for additional tasks. In the future, I might add caching so that we don’t request pages with tasks that didn’t change.

Task Page

You might have noticed in the GIF above that task name is now a link. Let’s check out what is hidden behind it!

Task Page

Let’s look at it step by step:

  1. Video Player with overlay for bounding boxes along with names
  2. Person’s characteristics like age, gender, facial expression.
  3. Task details

I had to spend quite a lot of time to come up with this page as there were some non-trivial problems that I have encountered:

Video Player

One of the most “digestible” types of information is a visual one – that is how humans work. That is why right from the beginning I came up with VideoOutputHandlerElem to draw bounding boxes with detected faces on video. However, it takes quite a time to loop over all the frames in a video and that is why I decided to create an HTML overlay for a video that could do the same thing without creating a new video.

First Try

However, it turned up to be not an easy task. My initial try was to simply create new bounding boxes as <div> elements and put it as an overlay on a video. I was able to create new boxes but then I thought that it would be terrible performance-wise – just think about creating and deleting new divs every 100-200 ms? Turns out there was a better solution.

HTML5 Canvas

As Mozilla Developers Website states – “TheĀ Canvas APIĀ provides a means for drawing graphics viaĀ JavaScriptĀ and theĀ HTMLĀ <canvas>Ā element.” That was exactly what we needed. I was able to lay canvas over <video> and draw and clear boxes on it.

Tip: don’t set the width or height of <canvas> via CSS, use them as attributes. I spent a lot of time figuring out why my <canvas> was so blurry.

Canvas over <video>

Full-Screen Problems

I was happy with the canvas solution until I encountered the problem with watching a video in a full-screen mode – nothing helped to lay <canvas> over <video> in this mode!

Turns out that full-screen mode is a very special thing in newest versions of HTML – you can get almost every HTML element into full-screen mode but nothing else can overlay it and you can’t get to full-screen unless a user specifically clicked or interacted somehow else with that element!

I spent a lot of time thinking about how to handle this issue and the only thing that came up to my mind was to get the whole <div> with <video> and <canvas> into full-screen mode. Long story short – I was able to do that but I had to create custom controls for the player in order to overwrite behavior of “Full-Screen” button.

Custom video player

Big Data Ain’t a Joke

Everything was great, I was happy with my new video player until I didn’t notice that pages with long videos were taking a lot of time to load (and that was happening on localhost!). Sure, the longer the video, the more time it takes to send from server to client, but that was not it – obviously JSON files with recognition data were quite large for long videos (some 4 minutes video had 10.5 MB large JSON file).

At that moment recognition data for each video was stored in MySQL database as JSON type along with other entries. Turns out that MySQL is not particularly good when it comes to handling loosely structured data such as the one we had here. Thus, I decided to move this already growing part of the data to MongoDB.

Introducing MongoDB

MongoDB is a document-oriented database that is much less strict to data structure than MySQL. I was thinking about using it right from the beginning but it turned out that a combination of MySQL for user data and MongoDB for recognition-related data works the best.

With MongoDB’s aggregations, I no longer had to fetch the whole JSON file from the database as it allows to query only a part of an array from the database (e.g. JSON data for frames from 10th to 25th second).

Every 15 seconds video player queries PMR’s backend for more JSON data. PMR’s backend, in turn, calculates the range of array’s slice that corresponds to the requested 15 seconds timeframe. This way JSON data is loaded asynchronously, effectively lifting up from the server the burden of querying and sending JSON data all at once.

Typical MongoDB document (entry in the database)

Summing up, MongoDB is a valuable addition to the project as I am planning to use it for other types of data too.

Demo of Video Player

Enough talking, let’s see how all these actions perform in real life!

Video Player Demonstration

In the beginning, I was really afraid that <canvas> won’t have good performance while updating it several times a second, but as you can see it works pretty well. If you hover over a bounding box you can see all the recognized data in the box on the left from the player. Finally, as I have already mentioned it works in Full-Screen mode too (but without showing extra recognized data).

Conclusion

Thanks for reading yet another PMR-related blogpost! In the coming days, I will work on the blogpost that will summarize all the things I have done during GSoC 2019. I will also work on preparing PMR-Web for demonstration and work on readmes for both PMR-Web and PMR.

See you soon!