This project aims to develop resources sharing platform that allows users to share any type of files with each other. The platform provides a set of characteristics that improve the way of how humans interact with computers.
The resources sharing platform enables users to perform an image-based search to find resources published through the platform and provides the ability to view digital information such as images, audios and videos using augmented reality technology in real-time using a smartphone application.
The system itself divided into distributed sub-systems that communicate with each other to perform multiple tasks, this approach helps to achieve efficiency and increase the performance speed.
System architecture
Upload Content To Platform Video
AR Experience Video
Image Search Experience Video
The AR engine is a separate software that performs an image-based search by matching a query image sent from the client application with the images published by the users through the platform and return a resource if a match occurred.
AR Engine data flow diagram in level 0
The published resources through the platform consist of the following files: -
- Template image: is the image that gets matched with the query image in the search process.
- Preview file: could be an image, audio or video that is returned to the client to be viewed if the published resource supports augmented reality experience.
- Resource file: the actual file that the client seeks.
The AR engine uses ORB (Oriented FAST and Rotated BRIEF) algorithm and performs the matching process between keypoints to find a resource published by a user through the platform. The process begins when the client application sends a query image to the AR engine, the AR engine receives the image and prepare it for the search process by detecting the keypoints and calculating their descriptors, the search process takes the descriptors of the query image and match it with pre-calculated descriptors of the recourse’s template image that has been stored in the memory, if a match occurred, the resource file and preview file will be sent to the client application along with metadata used for handling the preview file [10].
AR Engine data flow diagram in level 1
It’s necessary to take the following points into consideration during the implementation process of the AR engine: -
- Programming language: the AR engine must be developed using a compiled language such as Java or C++ to achieve good performance.
- Search speed: it’s important to avoid any unnecessary operation in the search process to get results as quickly as possible.
- Search accuracy: the search process must return accurate results as much as possible.
AR engine uses an application-level protocol to achieve real-time communication between the client application and the AR engine.
Communication between smartphone and AR engine
The session begins when a client initiates a request by establishing a Transmission Control Protocol (TCP) connection to port 1232 on the AR server. After the connection successfully established, the client application will begin sending the camera frames to be processed on the server and waits for a response. Upon receiving the response, the client will handle the response message and send another frame to be processed.
Communication sequence
The connection remains open until one of the pairs disconnect.
Request message specification
Response message specification
AR engine connection states
AR engine class diagram
Image Search sequence diagram
The AR engine will be tested by filling its storage with a dataset consisting of 1000 unique images that will be used in the search process, each image in the dataset will be associated with a unique value representing the resource ID, a set of random images will be taken from the test images to be used as a query images and test whether the AR engine will be able to return the resource ID associated with them or not, the test cases will focus on the performance and accuracy of the AR engine.
The test process will be executed on a laptop computer with the following specifications: -
- Processor: Intel CORE i7 6700 2.60GHz.
- RAM: 16GB.
- OS: Windows 10 64-bit
AR engine test cases
Number of successful test cases: 21
Number of failed test cases: 4
Average execution time: 1.291 seconds.
Some of the test cases were failed when using a query image contains only text due to the inability to recognize characters in the feature matching approach that is used in the search process.